~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
feat(security)!: egress policy module + search/LLM/embeddings PEPs (#4300)
LearningCircuit6fac0e196da4
diff --git a/.github/workflows/puppeteer-e2e-tests.yml b/.github/workflows/puppeteer-e2e-tests.ymlindex 411b00c5ca..36c2da5e0e 100644--- a/.github/workflows/puppeteer-e2e-tests.yml+++ b/.github/workflows/puppeteer-e2e-tests.yml@@ -191,6 +191,30 @@ jobs:# Exit with test result so the step reflects pass/failexit "${TEST_RESULT}"+ - name: Run egress-policy UI test+ # Release gate for the Privacy & Egress UI (scope dropdown, the+ # require-local toggles, the STRICT+meta-picker guard, and the+ # data-scope propagation across pages). This test lives in+ # tests/ui_tests (a standalone Puppeteer script, NOT the mocha suite+ # above) and runs against the SAME booted server. It authenticates as+ # the CI test user (test_admin) created by init_test_database.py and+ # matched in tests/ui_tests/auth_helper.js, and skips screenshots when+ # CI=true. `if: always()` so its result is reported even when the mocha+ # suite above failed; the script exits non-zero on any failed check, so+ # a regression fails the release gate.+ if: always()+ working-directory: tests/ui_tests+ env:+ BASE_URL: http://localhost:5000+ CI: true+ HEADLESS: true+ run: |+ set -o pipefail+ npm install+ npx puppeteer browsers install chrome+ xvfb-run -a -s "-screen 0 1920x1080x24" \+ node test_egress_policy_ui.js 2>&1 | tee egress-ui-output.log+- name: Extract test summaryif: always()working-directory: tests/puppeteerdiff --git a/SECURITY.md b/SECURITY.mdindex a8ee23ba89..4fac56657a 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -175,6 +175,46 @@ Operators on IPv6-only deployments using DNS64+NAT64 (AWS / GKE / Azure IPv6-onlURL rejection log lines route through `ssrf_validator.redact_url_for_log` to drop userinfo (RFC 3986 §3.2.1 allows credentials in the URL), path, and query — operators see `scheme://host:port` only. Operators with grep/regex tooling on the rejection log lines will see authority-only strings instead of full URLs.+### Egress Policy Module++LDR includes an optional egress-policy subsystem (`security/egress/` — see its [`README.md`](src/local_deep_research/security/egress/README.md) for the full design, the scope table, and the map of every enforcement point) that lets operators constrain where research traffic, LLM calls, and embeddings may go. The module is an **in-process correctness guardrail, NOT a hard security boundary**. It defends against honest misconfiguration, prompt-injection-induced URL fetches, accidental egress, and the LangGraph silent-expansion class of bug. It does **NOT** defend against:++- compromised dependencies that bypass the PEPs+- code execution inside the LDR process+- an adversary who can modify the policy module itself++Operators needing a hard boundary **must** layer OS-level controls: network namespaces, host-level firewall rules (egress filtering), restricted container runtimes. The egress policy is a guard rail for the application's own code paths; it cannot constrain a malicious actor who can write code that runs inside the same Python interpreter.++#### Threat vectors covered++| Vector | Defence |+|---|---|+| LangGraph silent search-engine expansion | Factory PEP at `create_search_engine` rejects engines not permitted under the active scope; the tool-list filter hides forbidden tools from the LLM. |+| Cloud LLM under "local-only" claim | `get_llm()` PEP refuses cloud providers / non-private LLM URLs when `llm.require_local_endpoint=true`. STRICT+meta-picker misconfig fails closed. |+| Cloud embeddings under "local-only" claim | Pre-flight policy check in `LibraryRAGService.__init__` covers all 5 direct construction sites + the factory. SentenceTransformer download from HuggingFace refused on cache miss under `embeddings.require_local=true`. |+| Prompt-injection-induced URL fetches | The agent `fetch_content` tool now calls `evaluate_url()` and raises `PolicyDeniedError` on denial. Subagents propagate the policy context. |+| PRIVATE_ONLY chain break | `ssrf_validator.policy_aware_validate_url()` lets the user's local lab deployments (Ollama on 127.0.0.1, SearXNG on 192.168.x) actually be reached under `PRIVATE_ONLY` without forcing the operator to set `SSRF_ALLOW_PRIVATE_IPS=1` globally. |+| Cache-hit policy bypass | `SearchCache._get_query_hash` incorporates the active scope; a `scope=BOTH` cache entry isn't returned to a later `scope=PRIVATE_ONLY` query. |+| NAT64 wrap of cloud metadata | `_classify_host` consults `is_nat64_wrapped_metadata_ip` before `is_private_ip`, so `64:ff9b::169.254.169.254` classifies as public (not as the link-local it superficially appears to be). |+| DNS race / process-global socket timeout | DNS resolution runs inside a single-shot `ThreadPoolExecutor` with `Future.result(timeout=2.0)` — no `socket.setdefaulttimeout()` mutation of process-global state. The worker is abandoned via `shutdown(wait=False)` so a hung lookup can't block past the timeout. |+| Cloud-metadata fetch under any scope | `evaluate_url()` rejects cloud-metadata IPs (`169.254.169.254`, ECS, IPv4-mapped forms) regardless of scope. They classify as link-local, so STRICT/PRIVATE_ONLY would otherwise *allow* them — notably via the audit-hook net which calls `evaluate_url` on raw `socket.connect` targets (bypassing the SSRF validator the fetch PEPs run first). |+| Private collection data → cloud model | Each collection carries a public/private flag (default **private**). A private collection is excluded under PUBLIC_ONLY / Adaptive-public scope and forces local LLM/embeddings inference under PRIVATE_ONLY / Adaptive-private. The **Adaptive** scope (default) derives the effective scope from the primary engine, so a private-collection primary keeps the whole run local automatically. |++#### Caveats++- **DNS rebinding TOCTOU**: `_classify_host` resolves once at evaluation time; the actual HTTP request resolves again at connect time. Closing this race would require pinning the resolved IP into the outbound connection, which is HTTPS-only and doesn't follow redirects cleanly. See the "Notification Webhook SSRF" subsection for the accepted-risk rationale (the same caveat applies here).+- **Settings tampering**: an adversary with write access to a user's per-user settings DB can flip `policy.egress_scope` to `both` and disable all enforcement. Per-user SQLCipher databases mean one user can't trivially tamper with another's policy, but an attacker who can read/write a user's DB can change anything about that user's runtime. Policy-key changes emit `policy_audit=True` log lines so admins can audit changes after the fact.+- **Audit log routing**: `policy_audit=True` log lines are filtered from the WebSocket sink (`frontend_progress_sink`), so they never reach a CORS-permissive browser observer. They are written to the loguru file/console sinks and persisted to the encrypted research-log DB if a research is active.+- **LLM/embeddings endpoint locality is best-effort**: the "stays local" guarantee is *strong* for named cloud providers (OpenAI, Anthropic, Google, OpenRouter, etc.) and localhost-default providers (Ollama/LM Studio/llama.cpp) — those are classified by name and reliably blocked/allowed. It is *weaker* for configurable-URL providers (`openai_endpoint`): the endpoint is classified by resolving its host, so an OpenAI-compatible endpoint pointed at a private-looking IP (split-horizon DNS, a tunnel, a proxy) is **trusted as local**. A user who wants to process private-collection data with a cloud LLM is expected to mark the collection **public** (the explicit opt-in); the policy prevents *accidental/silent* egress, not a determined user who deliberately points "local" inference at a cloud endpoint. This is consistent with the guardrail-not-boundary model above.++#### Configuration++See `docs/CONFIGURATION.md` for the user-facing keys+(`policy.egress_scope` — default **`adaptive`**, which follows your primary+engine; `llm.require_local_endpoint`; `embeddings.require_local`;+`llm.allowed_local_hostnames`), the per-collection public/private flag, the+per-research overrides, and the audit-log behaviour.+## Supported VersionsSecurity fixes are only provided for the latest release. Please upgrade to receive patches.diff --git a/changelog.d/+collection-public-private.feature.md b/changelog.d/+collection-public-private.feature.mdnew file mode 100644index 0000000000..aa970f46ea--- /dev/null+++ b/changelog.d/+collection-public-private.feature.md@@ -0,0 +1 @@+Collections can now be classified **public** or **private** (default private). A private collection is excluded from runs under Public-only / Adaptive-public egress scope and forces local LLM/embeddings inference under Private-only / Adaptive-private scope, so its contents never reach a cloud model. Flip the classification with the "Public collection" checkbox on the collection create form. Adds the `collections.is_public` column (migration 0011).diff --git a/changelog.d/+collection-public-toggle-details-page.feature.md b/changelog.d/+collection-public-toggle-details-page.feature.mdnew file mode 100644index 0000000000..8600c77db5--- /dev/null+++ b/changelog.d/+collection-public-toggle-details-page.feature.md@@ -0,0 +1 @@+A collection's public/private (egress) classification can now be changed after creation, from a "Privacy & Egress" toggle on the collection details page — not only at creation time.diff --git a/changelog.d/+egress-adaptive-default-upgrade-note.breaking.md b/changelog.d/+egress-adaptive-default-upgrade-note.breaking.mdnew file mode 100644index 0000000000..bd53d87406--- /dev/null+++ b/changelog.d/+egress-adaptive-default-upgrade-note.breaking.md@@ -0,0 +1 @@+**Upgrade behaviour change:** the default egress scope is now `adaptive` (was effectively `both`). Adaptive follows your primary search engine, so on upgrade a config with a *concrete* primary may narrow: a public primary (e.g. SearXNG/arXiv) now excludes local collections (public-only behaviour), and a *private collection* primary forces local LLM + embeddings. A meta-picker primary (`auto`/`parallel`) is unaffected (resolves to `both`). To keep the previous "any engine, cloud inference" behaviour, set Egress Scope to **Both**. See docs/egress-modes.md.diff --git a/changelog.d/+egress-adaptive-scope.feature.md b/changelog.d/+egress-adaptive-scope.feature.mdnew file mode 100644index 0000000000..79a7a6546d--- /dev/null+++ b/changelog.d/+egress-adaptive-scope.feature.md@@ -0,0 +1 @@+Add an **Adaptive** egress scope (now the default) that follows your primary search engine: a private primary keeps everything local (and forces local LLM/embeddings inference), a public primary allows public engines, and a meta-picker primary allows both. Existing explicit scopes (Both / Public only / Private only / Strict) remain available. Selecting **Private only** now visibly checks and locks the "require local LLM/embeddings" toggles to reflect that local inference is enforced.diff --git a/changelog.d/+egress-banner-dismiss-keys.bugfix.md b/changelog.d/+egress-banner-dismiss-keys.bugfix.mdnew file mode 100644index 0000000000..a915f9744f--- /dev/null+++ b/changelog.d/+egress-banner-dismiss-keys.bugfix.md@@ -0,0 +1 @@+The three egress warning banners (public-egress, cloud LLM, cloud embeddings) now have separate dismiss flags. Previously they shared one, so dismissing the fresh-install public-egress notice also permanently hid the critical cloud-LLM and cloud-embeddings warnings.diff --git a/changelog.d/+egress-deferred-followups.security.md b/changelog.d/+egress-deferred-followups.security.mdnew file mode 100644index 0000000000..443334d638--- /dev/null+++ b/changelog.d/+egress-deferred-followups.security.md@@ -0,0 +1 @@+Egress-policy follow-ups after the 5×20 review: (1) ContentFetcher now relaxes a downloader's `SafeSession` to allow private IPs under `PRIVATE_ONLY`, mirroring `policy_aware_validate_url`, so a private/lab URL the policy already approved is no longer rejected by the downloader's own strict SSRF re-validation; (2) the PEP-578 audit-hook backstop is now re-armed in the `NewsAggregationStrategy` analysis ThreadPoolExecutor worker (threading.local isn't inherited by pool workers) and in the document scheduler's worker thread (built from the user's saved settings), so scheduled downloads and news-analysis LLM calls keep defense-in-depth parity under `PRIVATE_ONLY`/`STRICT`. The primary snapshot-based PEPs already gated these paths; this restores the secondary net. (MCP search runs synchronously on the already-armed strategy thread, so it needed no change.)diff --git a/changelog.d/+egress-mcp-tool-filter.security.md b/changelog.d/+egress-mcp-tool-filter.security.mdnew file mode 100644index 0000000000..316edaa22b--- /dev/null+++ b/changelog.d/+egress-mcp-tool-filter.security.md@@ -0,0 +1 @@+The MCP search strategy now filters its specialized search-engine tool list by the active egress scope (parity with the LangGraph strategy): forbidden engines never appear in the tool schema the LLM sees, and a policy denial at execution surfaces as an audit log line instead of a generic tool error.diff --git a/changelog.d/+egress-package-refactor.misc.md b/changelog.d/+egress-package-refactor.misc.mdnew file mode 100644index 0000000000..74e566fe30--- /dev/null+++ b/changelog.d/+egress-package-refactor.misc.md@@ -0,0 +1 @@+Consolidate the egress-policy code into a dedicated `security/egress/` subpackage (`policy.py` + `audit_hook.py`) with a README documenting the design, scope model, and the full map of enforcement points. No behavior change — module paths moved from `security.egress_policy` / `security.egress_audit_hook` to `security.egress.policy` / `security.egress.audit_hook`.diff --git a/changelog.d/+egress-per-run-fetch-quota.security.md b/changelog.d/+egress-per-run-fetch-quota.security.mdnew file mode 100644index 0000000000..0d8b3d4b0b--- /dev/null+++ b/changelog.d/+egress-per-run-fetch-quota.security.md@@ -0,0 +1 @@+The per-run denied-fetch quota (`MAX_DENIED_FETCHES_PER_RUN`) is now aggregated per *run* instead of per `EgressContext`. Each call site (content fetcher, full-content search, download service, the audit hook) builds its own `EgressContext`, so the counter previously reset whenever a new engine/fetcher was constructed — letting a malicious indexed document evade the exhaustion guard by spreading denied fetches across contexts. The counter is now anchored to the run's armed audit-hook context (shared across the whole run, re-armed identically on pool workers), with a safe fallback to the local context for snapshot-less / programmatic callers.diff --git a/changelog.d/+egress-policy-audit-hook.security.md b/changelog.d/+egress-policy-audit-hook.security.mdnew file mode 100644index 0000000000..d4059c0372--- /dev/null+++ b/changelog.d/+egress-policy-audit-hook.security.md@@ -0,0 +1 @@+Add a process-wide PEP 578 `sys.audit` hook (`security/egress_audit_hook.py`) that gates `socket.connect` against the active EgressContext. This is the secondary line of defense — every explicit PEP we wired (ContentFetcher, evaluate_llm_endpoint, MCP download_content, journal_reputation_filter, …) remains the primary line and still fires first. The hook catches what those PEPs cannot: a third-party library that opens its own connection, a new code path added without policy awareness, a langchain tool registered by an MCP server, prompt-injection steering a tool into raw `requests.get`, or anything else that ultimately calls `socket.connect`. The hook is installed once on first import of `local_deep_research.security` (idempotent — PEP 578 hooks cannot be removed) and is INACTIVE by default: with no `EgressContext` registered on the current thread, every connect passes through unmodified, so test runners, import-time helpers, and scripts that touch a socket are unaffected. Workers opt in by calling `set_active_context(ctx)` (or using the `active_egress_context` context manager); the research worker in `web/services/research_service.py` now does this after building the snapshot, and the centralized `database/thread_local_session._ThreadCleanup` exit handler clears it on worker shutdown so a pooled thread cannot leak one run's scope into the next task. The hook gates only AF_INET / AF_INET6 — AF_UNIX, AF_NETLINK and friends pass through. It does NOT defend against an adversary with code execution in the LDR process (they can clear the active context or add a passthrough hook themselves); for that, layer OS-level controls per SECURITY.md. 18 unit tests in `tests/security/test_egress_audit_hook.py` pin the contract: install idempotency, get/set/clear, context-manager cleanup on exception, per-scope behaviour against real raw sockets (PRIVATE_ONLY blocks 8.8.8.8 and permits 127.0.0.1; PUBLIC_ONLY mirrors; STRICT blocks public hosts; BOTH allows either), IPv6 loopback bracket handling, AF_UNIX pass-through, and per-thread isolation under concurrency.diff --git a/changelog.d/+egress-policy-scope-palette.misc.md b/changelog.d/+egress-policy-scope-palette.misc.mdnew file mode 100644index 0000000000..85e7234a2b--- /dev/null+++ b/changelog.d/+egress-policy-scope-palette.misc.md@@ -0,0 +1 @@+Recolor the egress-scope visual cues to match the actual data-leak risk hierarchy. Previously BOTH (the default) wore a light-amber accent and PUBLIC_ONLY a deeper-amber one, which reads as "PUBLIC_ONLY is more dangerous than the default" — but BOTH is actually the riskiest scope: it permits local engines AND a cloud LLM in the same run, so local library/RAG chunks can land in OpenAI via the LLM call. PUBLIC_ONLY blocks local engines, so local data never enters that pipeline. New palette: BOTH gets no left-border accent at all (the existing "Public search egress enabled" banner does the nagging in text); PUBLIC_ONLY uses sky-blue (#0ea5e9 — informational "you chose public sources only"); PRIVATE_ONLY stays teal (#14a37f); STRICT moves from indigo to violet (#8b5cf6) so it's clearly distinct from the new PUBLIC_ONLY blue for color-blind users. The privacy panel itself defaults to a neutral slate when BOTH is active rather than amber. The risk-honest rationale is documented in the `base.html` style-block comment.diff --git a/changelog.d/+egress-policy-stage-1a-round7.security.md b/changelog.d/+egress-policy-stage-1a-round7.security.mdnew file mode 100644index 0000000000..4ec4dbdf51--- /dev/null+++ b/changelog.d/+egress-policy-stage-1a-round7.security.md@@ -0,0 +1 @@+Egress policy hardening (Round 7 review): close four PEP gaps surfaced by a 120-agent re-audit and apply the same fix to four sibling sites. (1) LLM gate at config/llm_config.py now uses a known-local allow-list (`ollama`/`lmstudio`/`llamacpp`) when no snapshot is supplied — previously the gate silently no-op'd, letting cloud LLMs instantiate from snapshot-less callers under `llm.require_local_endpoint=true`. (2) `JournalReputationFilter` forwards `settings_snapshot` to `get_llm()` and `create_default` lets `PolicyDeniedError` propagate instead of silently returning None. (3) MCP `_discover_mcp_tools` / `_execute_mcp_tool` block under STRICT / PRIVATE_ONLY (and on absent snapshot), with a policy_audit log and a UI progress signal. (4) `_execute_download_content` narrows its bare `except Exception` so corrupted scope values fail closed instead of dropping to SSRF-only. The same bare-except fail-open pattern is fixed consistently in `notifications/manager.py`, `research_library/services/download_service.py` (paired with a `_policy_locked` short-circuit in the URL check), `web_search_engines/search_engine_base.py` (disables `include_full_content` when policy can't be evaluated), and `web/routes/research_routes.py` (STRICT + meta-picker now surfaces as 400 at run-start). Adds a Flask `PolicyDeniedError` handler so denials escaping a synchronous PEP return a clean 400 with the decision reason. Regression tests: parametrized no-snapshot fail-closed for every cloud provider plus ambiguous (`openai_endpoint`) and hypothetical (`groq`, `mistral`, `cohere`) entries; allow-list passes for the three known-local providers; MCP scope gate (STRICT/PRIVATE_ONLY/no-snapshot/corrupted policy) verifies the connection manager is never touched and the empty list is cached; download_content refuses without constructing ContentFetcher; journal_reputation_filter forwards snapshot and propagates PolicyDeniedError.diff --git a/changelog.d/+egress-policy-stage-1a.feature.md b/changelog.d/+egress-policy-stage-1a.feature.mdnew file mode 100644index 0000000000..48dbd64d35--- /dev/null+++ b/changelog.d/+egress-policy-stage-1a.feature.md@@ -0,0 +1 @@+Add an egress policy module (`security.egress_policy`) that gates search-engine instantiation against a user-declared scope. New settings `policy.egress_scope` (default `both`, preserves current behavior), `llm.require_local_endpoint`, `embeddings.require_local`, and `llm.allowed_local_hostnames` are reserved. Under `strict` scope the factory blocks any engine that isn't the user's primary, closing the previously silent expansion path. The LangGraph agent tool-list filter, LLM/embeddings PEPs, and UI controls ship in the same release (see the related Stage 1b changelog entries).diff --git a/changelog.d/+egress-policy-stage-1b-llm-embeddings.feature.md b/changelog.d/+egress-policy-stage-1b-llm-embeddings.feature.mdnew file mode 100644index 0000000000..b1ffffbe5e--- /dev/null+++ b/changelog.d/+egress-policy-stage-1b-llm-embeddings.feature.md@@ -0,0 +1 @@+Wire the egress policy module (added in Stage 1a) into the LLM and embeddings call paths. `get_llm()` now raises `PolicyDeniedError` when `llm.require_local_endpoint=True` and a cloud LLM provider is requested. The OpenAI and Anthropic model-listing endpoints in the settings UI degrade gracefully (empty list + audit log entry) under the same condition. `get_rag_service()` runs a pre-flight check before constructing the service so embedding-policy violations surface immediately rather than after hundreds of chunks have been generated. Defaults are unchanged (`require_local_endpoint=False`, `embeddings.require_local=False`), so existing installs see no behavior change.diff --git a/changelog.d/+egress-policy-ui-tests.misc.md b/changelog.d/+egress-policy-ui-tests.misc.mdnew file mode 100644index 0000000000..34ccea52ce--- /dev/null+++ b/changelog.d/+egress-policy-ui-tests.misc.md@@ -0,0 +1 @@+Add a comprehensive Puppeteer UI suite for the egress-policy feature: `tests/ui_tests/test_egress_policy_ui.js` (CI-safe, 32 assertions) covers every UX touchpoint — Privacy & Egress panel rendering, all four scopes painting their expected border/text colors (amber/teal/indigo palette verified via computed-style reads, not just DOM state), the STRICT-on-meta-picker guard reverting to "Both" with an explanatory hint, per-research overrides round-tripping through the settings DB, scope cue propagating to /history, /metrics and /chat via base.html's `body[data-scope]`, settings dashboard listing the policy keys, and the require-local toggles persisting across reloads. A second file, `tests/ui_tests/NO_CI_test_egress_policy_live_research.js`, drives end-to-end research runs against a real Ollama + SearXNG instance (defaults to a lab box; overridable via env) and verifies the run-start PEP at `research_routes.py:248` — STRICT+searxng succeeds, STRICT+auto returns 400 with the meta-picker incoherence message, PUBLIC_ONLY refuses a SearXNG whose instance URL resolves to a private IP (`scope_mismatch_public_only`), PRIVATE_ONLY accepts the same SearXNG (URL classifier overrides the static `is_public=True`), and a cloud LLM under `require_local_endpoint=true` is accepted at the precheck but reaches `failed` status as the strategy's `get_llm()` hits the PEP. Both files gate screenshots behind `!process.env.CI` matching the `tests/ui_tests/test_settings_page.js:57` convention; the live test's `NO_CI_` filename prefix means the runner skips it in CI (where the lab endpoints are unreachable). Screenshots land in `tests/ui_tests/screenshots/egress-policy{,-live}/` (already gitignored).diff --git a/changelog.d/+egress-private-blocks-vendor-notifications.security.md b/changelog.d/+egress-private-blocks-vendor-notifications.security.mdnew file mode 100644index 0000000000..d62ef23906--- /dev/null+++ b/changelog.d/+egress-private-blocks-vendor-notifications.security.md@@ -0,0 +1 @@+Under the PRIVATE_ONLY egress scope (and Adaptive resolving to private), notification dispatch via non-http Apprise vendor schemes (`slack://`, `discord://`, `telegram://`, …) is now refused — those send to external vendor APIs and can't be verified local. Address a self-hosted notifier by its `http(s)://` URL instead, which is allowed as a private host. Other scopes are unchanged.diff --git a/changelog.d/+egress-snapshotless-inference-failclosed.breaking.md b/changelog.d/+egress-snapshotless-inference-failclosed.breaking.mdnew file mode 100644index 0000000000..a1d243ff7e--- /dev/null+++ b/changelog.d/+egress-snapshotless-inference-failclosed.breaking.md@@ -0,0 +1 @@+**Breaking (programmatic API):** `get_llm(provider=…)` and `get_embeddings(provider=…)` now **fail closed** when called with no settings snapshot (`settings_snapshot=None`) for any provider that is not a known local-default. Snapshot-less callers may instantiate only the localhost-default providers — LLM: `ollama`, `lmstudio`, `llamacpp`; embeddings: `sentence_transformers`, `ollama` — plus LLMs you registered in-process via the programmatic API (`quick_summary(llms={…})`). Any other provider (incl. `openai`, `anthropic`, `google`, `openrouter`, and any future cloud provider) raises `PolicyDeniedError` instead of silently constructing a cloud client. This closes a snapshot-less egress hole; programmatic callers that relied on building a cloud provider without a snapshot must now pass a `settings_snapshot` (or register the LLM in-process). The allowlists are intentionally tight so unknown/new providers fail closed by default.diff --git a/changelog.d/4300.security.md b/changelog.d/4300.security.mdnew file mode 100644index 0000000000..22e5c509fb--- /dev/null+++ b/changelog.d/4300.security.md@@ -0,0 +1,15 @@+Egress policy hardening (PR #4300, Round 6 review): 24 fixes across DNS classification (process-global socket timeout removed, threading.RLock around cache mutations), PDP coverage (nested-dict settings unwrap, NAT64 metadata reclassification, fetch_content tool gate), engine/RAG/data coverage (MetaSearchEngine Wikipedia fallback routed through factory, all 5 direct LibraryRAGService sites pre-flight-checked, SearchCache hash includes scope to close cache-bypass), and UI/cross-cutting work (per-research policy overrides on the research form, settings dashboard renders the policy keys, benchmark snapshot threading, policy_audit WebSocket filter, audit log on policy-key changes, dismiss-flag rename to follow the dismiss_* convention). 23 regression tests added.++A second adversarial review round added 13 more fixes: closed two egress bypasses (Elasticsearch `cloud_id` is now refused under private scope; OpenAI/discovered-provider model-list probes are scope-gated in both the settings and RAG model endpoints); closed a metadata-SSRF gap (literal cloud-metadata IPs are now classified PUBLIC on the literal-IP branch of `_classify_host`, matching the DNS branch); ADAPTIVE now resolves a registered local-retriever primary to PRIVATE_ONLY (was BOTH, which leaked the private corpus to cloud inference); the PEP-578 audit-hook backstop is now armed on non-web entry points (CLI / news scheduler / programmatic API) and re-armed on ThreadPoolExecutor pool workers; fixed the SearXNG `url_setting` key so a local SearXNG is no longer over-blocked under PRIVATE_ONLY; `allowed_local_hostnames` no longer rejects unresolvable intranet hosts on save; the denied-fetch quota no longer counts benign parse failures; and `DownloadService` fails closed when the settings backend errors. 10 regression tests added.++Follow-up regression fix: user-registered in-process LLMs (the programmatic API's `llms={...}` and plugins) are no longer denied with `provider_url_unset` when a run resolves to require-local inference (e.g. ADAPTIVE with a registered-retriever primary). The PDP now distinguishes user registrations from auto-discovered built-in providers — a custom name shadowing a built-in cloud provider stays blocked, and snapshot-less calls keep failing closed for everything else. 3 regression tests added.++Third adversarial review round (5×20 agents) confirmed and fixed 6 more issues (4 of 11 verified findings were declined as not-exploitable or design-intent, 1 was a false positive): the `evaluate_url` cloud-metadata block now normalizes alternate IPv4 encodings (octal `0251.0376.0251.0376`, hex, integer) so an IMDS literal can't read as an allowed public host under PUBLIC_ONLY/BOTH and always denies with the explicit `blocked_metadata_ip` reason; `dangerous_scheme` (javascript:/data:/file: hrefs) no longer counts toward the per-run denied-fetch quota (matching `unsupported_scheme`, so a document full of data: URIs can't starve later legitimate fetches); `evaluate_llm_endpoint` and `evaluate_embeddings` now percent-decode the host before classification (consistency with `evaluate_url`, so a legitimate percent-encoded local endpoint isn't wrongly denied under require-local); `context_from_snapshot` fails closed with `ValueError` on a non-dict snapshot instead of a swallowable `AttributeError`; and the run-scoped DNS classification cache is now first-writer-wins so concurrent disagreeing lookups (round-robin DNS) can't flip a hostname's classification mid-run. 11 regression tests added.++Round 2 of the 5×20 review (audit-hook + thread-context propagation) confirmed and fixed 4 issues (of 21 findings: 6 false positives, 1 already-fixed, several lower-severity backstop-completeness gaps deferred with the primary PEPs as the live gate): the PEP-578 audit hook now decodes bytes-encoded socket addresses before classifying (CPython fires `socket.connect` with a bytes host, which previously passed straight through the PRIVATE_ONLY/STRICT backstop); `set_active_context` rejects an unresolved ADAPTIVE-scope context (it would have silently no-op'd the hook); `active_egress_context` saves and restores the previous context so a nested activation no longer wipes the parent's; and the default `langgraph-agent` strategy now re-arms the audit-hook backstop inside each subagent ThreadPoolExecutor worker (threading.local isn't inherited by pool workers). 4 regression tests added.++Round 3 of the 5×20 review (call-site PEPs / data-egress paths) confirmed the empty-snapshot fail-open as the one genuinely exploitable issue and fixed it (most of the 18 findings were lower-severity than first rated — UI-display consistency, config-time validations already backstopped by execution-time PEPs, documented accepted-risk redirects, or backward-compatible empty-snapshot defaults): the `quick_summary` REST endpoint now fails CLOSED (HTTP 503) when the user's settings snapshot can't be loaded, instead of continuing with an empty `{}` snapshot that silently downgraded a configured PRIVATE_ONLY/require-local user to the permissive BOTH scope. News-subscription policy validation now also surfaces an incoherent egress config (STRICT scope + meta-picker primary, which raises ValueError) as a fail-fast misconfiguration error at create/update time rather than silently skipping the check. 1 regression test added.++Round 4 of the 5×20 review (adversarial bypass hunting) found no new exploitable bypass — every candidate was a known/documented accepted-risk (DNS-rebinding and redirect TOCTOU, per-context fetch quota, connect-only audit-hook scope), a fail-safe over-restriction, or an operator-trust-boundary case (a programmatic caller controls its own snapshot). One cheap defense-in-depth completion was applied: `evaluate_url` now also blocks cloud-metadata endpoints reachable by hostname (GCP `metadata.google.internal` / `metadata.goog`, any case) and strips an insignificant trailing dot before the metadata checks, so `metadata.google.internal.` and `169.254.169.254.` can no longer read as an allowed public host (closing the hostname/trailing-dot gaps the round-1 octal/hex/integer fix didn't cover). 2 regression tests added.++Round 5 of the 5×20 review (quality / docs / UI / completeness) fixed 5 issues: policy-audit denial logs now redact URLs (scheme://host:port only) via the existing `redact_url_for_log` at all four sites (ContentFetcher, DownloadService, and both full_search paths), so a denied URL carrying userinfo credentials or an API-key query param is no longer written verbatim to the audit log; the ADAPTIVE warning-banner render path (`/api/warnings`) now resolves scope with `allow_dns=False` so a settings-page load can't block up to the DNS timeout on a synchronous getaddrinfo for a URL-engine primary; `CONFIGURATION.md` was regenerated so its Settings List documents the egress keys and their `LDR_*` env vars (the hand-written prose, which the on-merge auto-regeneration would have wiped, moved into the hand-maintained egress-modes.md with the cross-link fixed); plus a missing `_retriever_is_local` return annotation and an added BOTH-scope allow test. 3 regression tests added.diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex 1d5b9d74fc..a908fb7c15 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -78,7 +78,11 @@ They are not available in the Web UI because they are needed to start the applic| `app.queue_mode` | `LDR_QUEUE_MODE` | `direct` | How new research requests are handled. 'Direct' starts research immediately if a concurrent slot is available. 'Queue' always adds to a background queue for sequential processing in order. | APP || `app.theme` | `LDR_APP_THEME` | `dark` | User interface color theme. Choose from 20 themes including popular developer themes. | APP || `app.timezone` | `TZ` | `UTC` | Timezone for date calculations (e.g., news subscription YYYY-MM-DD placeholder). Use 'UTC' for server time or specify your local timezone. | APP |+| `app.warnings.dismiss_adaptive_scope_info` | `LDR_APP_WARNINGS_DISMISS_ADAPTIVE_SCOPE_INFO` | `false` | Permanently hide the informational banner that states what the Adaptive egress scope resolves to (public / private / both) for the current primary engine. | APP |+| `app.warnings.dismiss_cloud_embeddings` | `LDR_APP_WARNINGS_DISMISS_CLOUD_EMBEDDINGS` | `false` | Permanently hide the 'document chunks will be sent to OpenAI' (cloud embeddings) banner. Separate from the public-egress dismiss flag so acknowledging fresh-install noise never suppresses this critical warning. | APP |+| `app.warnings.dismiss_cloud_llm` | `LDR_APP_WARNINGS_DISMISS_CLOUD_LLM` | `false` | Permanently hide the 'LLM provider is cloud-hosted' banner. Separate from the public-egress dismiss flag so acknowledging fresh-install noise never suppresses this critical warning. | APP || `app.warnings.dismiss_context_reduced` | `LDR_APP_WARNINGS_DISMISS_CONTEXT_REDUCED` | `false` | Dismiss warnings based on historical context usage and truncation | APP |+| `app.warnings.dismiss_egress_policy` | `LDR_APP_WARNINGS_DISMISS_EGRESS_POLICY` | `false` | Permanently hide the 'public search egress enabled' banner (the fresh-install informational one). The critical cloud-LLM and cloud-embeddings banners have their OWN dismiss flags so this can't silently hide them. | APP || `app.warnings.dismiss_high_context` | `LDR_APP_WARNINGS_DISMISS_HIGH_CONTEXT` | `false` | Permanently hide the warning shown when your local LLM context window is set above 8,192 tokens, which may cause memory issues on some hardware. | APP || `app.warnings.dismiss_legacy_config` | `LDR_APP_WARNINGS_DISMISS_LEGACY_CONFIG` | `false` | Dismiss warnings about the deprecated server_config.json file | APP || `app.warnings.dismiss_model_mismatch` | `LDR_APP_WARNINGS_DISMISS_MODEL_MISMATCH` | `false` | Permanently hide the warning shown when using a large model (70B+ parameters) with a local provider and high context window, which may exceed available GPU memory. | APP |@@ -107,6 +111,7 @@ They are not available in the Web UI because they are needed to start the applic| `embeddings.openai.base_url` | `LDR_EMBEDDINGS_OPENAI_BASE_URL` | `` | Custom base URL for the embeddings endpoint. Leave blank to use the OpenAI cloud API (https://api.openai.com/v1). Set to your OpenAI-compatible local server URL to route embeddings there — examples: http://localhost:1234/v1 (LM Studio), http://localhost:8000/v1 (vLLM or llama.cpp server). When set, this overrides the cloud endpoint for embeddings without affecting the chat LLM provider. | APP || `embeddings.openai.dimensions` | `LDR_EMBEDDINGS_OPENAI_DIMENSIONS` | `null` | Optional output vector dimensionality. Only applies to OpenAI text-embedding-3 models (which support truncation); ignored for other models including most local servers. Leave blank to use the model's native dimensionality. | APP || `embeddings.openai.model` | `LDR_EMBEDDINGS_OPENAI_MODEL` | `text-embedding-3-small` | Embedding model identifier. For OpenAI cloud: text-embedding-3-small (default), text-embedding-3-large, or text-embedding-ada-002. For OpenAI-compatible local servers, use the exact model name your server reports (e.g. nomic-ai/nomic-embed-text-v1.5 on LM Studio). | APP |+| `embeddings.require_local` | `LDR_EMBEDDINGS_REQUIRE_LOCAL` | `false` | When enabled, the embeddings provider must be local (sentence_transformers, or Ollama pointed at a local URL). Indexing with cloud embeddings (OpenAI) is blocked. Indexing a corpus with cloud embeddings sends every chunk off-machine. | LLM || `focused_iteration.adaptive_questions` | `LDR_FOCUSED_ITERATION_ADAPTIVE_QUESTIONS` | `false` | When enabled (1), the system monitors recent search success and warns the LLM to broaden queries if 3+ searches in a recent window returned no results. When disabled (0), queries are generated without feedback from previous result counts. | SEARCH || `focused_iteration.knowledge_summary_limit` | `LDR_FOCUSED_ITERATION_KNOWLEDGE_SUMMARY_LIMIT` | `10` | Limits the number of top search results included in the knowledge summary shown to the LLM during question generation. Set to 10 to show only the top 10 results, reducing token usage. Set to 0 for unlimited. Note: This only affects question generation prompts — final answer synthesis always uses all results. | SEARCH || `focused_iteration.previous_searches_limit` | `LDR_FOCUSED_ITERATION_PREVIOUS_SEARCHES_LIMIT` | `10` | Maximum number of recent search iterations to show the LLM when generating follow-up questions. Helps the LLM avoid duplicate queries and reduces token usage. Set to 0 for unlimited (show all previous searches). When active, also caps displayed queries per iteration to 3. | SEARCH |@@ -120,6 +125,7 @@ They are not available in the Web UI because they are needed to start the applic| `langgraph_agent.include_sub_research` | `LDR_LANGGRAPH_AGENT_INCLUDE_SUB_RESEARCH` | `true` | Enable parallel subagent research. When enabled, the agent can decompose complex questions into subtopics and research them simultaneously using separate subagents. | SEARCH || `langgraph_agent.max_iterations` | `LDR_LANGGRAPH_AGENT_MAX_ITERATIONS` | `50` | Maximum number of agent reasoning cycles (each cycle = one LLM call + one tool call). Higher values let the agent search more thoroughly but use more tokens. The agent typically needs 10-20 cycles for simple queries and 30-50 for complex multi-faceted research. | SEARCH || `langgraph_agent.max_sub_iterations` | `LDR_LANGGRAPH_AGENT_MAX_SUB_ITERATIONS` | `8` | Maximum reasoning cycles per subagent when researching subtopics in parallel. Each subagent investigates one aspect of the question independently. | SEARCH |+| `llm.allowed_local_hostnames` | `LDR_LLM_ALLOWED_LOCAL_HOSTNAMES` | ``[]`` | Hostnames the policy will treat as local in addition to loopback / RFC1918 / link-local. Use for self-hosted services on custom hostnames (e.g. 'ollama.lan'). Public hostnames added here are rejected at save time. | LLM || `llm.anthropic.api_key` | `LDR_LLM_ANTHROPIC_API_KEY` | `` | Your Anthropic API key for Claude models. Get one at console.anthropic.com. Required when using the Anthropic provider. | LLM || `llm.context_window_size` | `LDR_LLM_CONTEXT_WINDOW_SIZE` | `128000` | Maximum context window size in tokens for cloud LLMs. Only used when unrestricted context is disabled. | LLM || `llm.context_window_unrestricted` | `LDR_LLM_CONTEXT_WINDOW_UNRESTRICTED` | `true` | Let cloud provider APIs manage their own token limits without a hard cap (recommended). Uncheck to enforce a specific limit via the Context Window Size setting. | LLM |@@ -141,6 +147,7 @@ They are not available in the Web UI because they are needed to start the applic| `llm.openai_endpoint.url` | `LDR_LLM_OPENAI_ENDPOINT_URL` | `https://openrouter.ai/api/v1` | URL of a custom OpenAI-compatible API endpoint. Use for llama.cpp server, vLLM, Ollama, OpenRouter, Groq, Together, or any service that implements the OpenAI API format. Example for local llama.cpp: http://localhost:8000/v1 | LLM || `llm.openrouter.api_key` | `LDR_LLM_OPENROUTER_API_KEY` | `null` | API key to use for the OpenRouter provider. | SEARCH || `llm.provider` | `LDR_LLM_PROVIDER` | `ollama` | The LLM service provider. Choose local providers (Ollama, LM Studio, LlamaCpp) for free on-device inference, or cloud APIs (OpenAI, Anthropic, Google, OpenRouter) for hosted models. | LLM |+| `llm.require_local_endpoint` | `LDR_LLM_REQUIRE_LOCAL_ENDPOINT` | `false` | When enabled, the LLM provider's resolved endpoint must be on a local network (loopback, RFC1918, link-local). Cloud-only providers (OpenAI, Anthropic, Google, OpenRouter) are blocked. Independent of egress_scope. | LLM || `llm.supports_max_tokens` | `LDR_LLM_SUPPORTS_MAX_TOKENS` | `true` | Feature flag for max_tokens parameter. Disable if your LLM API returns errors when max_tokens is set. When disabled, the parameter is not passed to the API. | LLM || `llm.temperature` | `LDR_LLM_TEMPERATURE` | `0.7` | Controls randomness in model outputs. Lower values (0.0-0.3) produce deterministic, factual responses; higher values (0.7-1.0) produce more creative and varied outputs. | LLM || `llm.xai.api_key` | `LDR_LLM_XAI_API_KEY` | `null` | API key to use for the xAI Grok provider. | LLM |@@ -184,6 +191,7 @@ They are not available in the Web UI because they are needed to start the applic| `notifications.rate_limit_per_day` | `LDR_NOTIFICATIONS_RATE_LIMIT_PER_DAY` | `50` | Maximum number of notifications per day per user. Daily cap prevents notification fatigue. Default of 50 balances staying informed with avoiding excessive alerts. | APP || `notifications.rate_limit_per_hour` | `LDR_NOTIFICATIONS_RATE_LIMIT_PER_HOUR` | `10` | Maximum number of notifications per hour per user. Prevents notification spam during burst activity. Default of 10 allows important notifications through while limiting noise. | APP || `notifications.service_url` | `LDR_NOTIFICATIONS_SERVICE_URL` | `` | Notification service URL(s). Supports multiple comma-separated URLs to send notifications to multiple destinations (e.g., discord://YOUR_WEBHOOK_ID/YOUR_TOKEN, mailto://YOUR_USER:[email protected][email protected]). See <a href="https://github.com/LearningCircuit/local-deep-research/blob/main/docs/NOTIFICATIONS.md" target="_blank">documentation</a> for supported services. | APP |+| `policy.egress_scope` | `LDR_POLICY_EGRESS_SCOPE` | `adaptive` | EXPERIMENTAL — this egress boundary is new defense-in-depth, not an absolute guarantee yet; don't rely on it as your only protection for highly sensitive data. Restrict which search engines a research run may use. ADAPTIVE (default) = follow your primary engine: a private primary keeps everything local, a public primary allows public engines, a meta-picker allows both. STRICT = only the primary engine you picked; PUBLIC_ONLY = any public web/academic engine; PRIVATE_ONLY = any local engine (library, paperless, etc.); BOTH = any classified engine. STRICT is incompatible with auto/meta/parallel primaries. | SEARCH || `rag.indexing_batch_size` | `LDR_RAG_INDEXING_BATCH_SIZE` | `15` | Number of documents to process in parallel during RAG indexing. Higher values are faster but use more memory. | APP || `rate_limiting.decay_per_day` | `LDR_RATE_LIMITING_DECAY_PER_DAY` | `0.95` | How much to trust old rate limit data as it ages (0.5-0.99). At 0.95, data retains 95% reliability per day. Lower values discard old data faster, forcing the system to re-learn optimal wait times. | APP || `rate_limiting.enabled` | `LDR_RATE_LIMITING_ENABLED` | `true` | Enable adaptive rate limiting that learns optimal wait times for each search engine to avoid hitting API rate limits. When disabled, requests are sent with minimal delay (0.1s), which may trigger rate limit errors from some APIs. | APP |diff --git a/docs/egress-modes.md b/docs/egress-modes.mdnew file mode 100644index 0000000000..6935ab0f78--- /dev/null+++ b/docs/egress-modes.md@@ -0,0 +1,141 @@+# Egress modes — what each one does++> ⚠️ **Experimental.** The egress boundary is new and ships as+> defense-in-depth, not an absolute guarantee. It blocks the known data-egress+> paths under the scope you pick, but this is an early version — don't rely on+> it as your *only* protection for highly sensitive data yet. See the threat+> model and limitations linked just below.++LDR's **egress scope** (Settings → *Egress Scope*, or the *Privacy & Egress*+panel on the research form) controls **where your research traffic is allowed+to go** — which search engines run, whether your LLM/embeddings may be cloud+services, and which URLs may be fetched. It's the single switch for *"how much+of this run is allowed to leave my machine?"*++Pick a mode below. The default is **Adaptive**, which just follows your primary+search engine, so most people never need to think about it.++> This page explains the modes for everyday use. For the threat model,+> guarantees, and limitations, see+> [`SECURITY.md`](../SECURITY.md#egress-policy-module) and the technical+> [egress package README](../src/local_deep_research/security/egress/README.md).++## At a glance++| Mode | Search engines | LLM / embeddings | Best for |+|---|---|---|---|+| **Adaptive** *(default)* | follows your primary engine | forced local **only** when the run is private | "just do the sensible thing" |+| **Both** | any classified engine (public + local) | your configured providers (cloud allowed) | mixing public web + your collections |+| **Public only** | public web/academic engines only | your configured providers | public research; your local collections aren't touched |+| **Private only** | local engines only (collections, local SearXNG/Ollama) | **forced local** — cloud blocked | sensitive work that must stay on the machine |+| **Strict** | **only** your one primary engine | your configured providers | a single, exact source with zero expansion |++---++## <a id="adaptive"></a>🔀 Adaptive *(default)*++**Adaptive follows your primary search engine** and resolves to a concrete+mode for each run:++- Primary is a **public** engine (e.g. SearXNG pointed at a public instance,+ arXiv, PubMed) → behaves like **Public only**.+- Primary is a **private** source (a local collection, your library) →+ behaves like **Private only** (and therefore forces local LLM + embeddings).+- Primary is a **meta-picker** (`auto`, `parallel`, `meta`) → behaves like+ **Both** (the meta-picker fans out to whatever fits).++Why it's the default: you choose a search engine anyway, and the privacy+posture "just matches" it. If you make a **private collection** your primary,+the whole run automatically stays local — nothing leaves the box.++> **Note:** to *mix* a private collection with public engines, or to use a+> cloud LLM on a private collection, switch to **Both** (or mark the collection+> public — see below). Adaptive deliberately narrows to match the primary.++## <a id="both"></a>🌐 Both++Any **classified** engine may run — public web/academic engines **and** your+local collections/library. Your configured LLM and embeddings are used as-is+(cloud allowed unless you tick the *Require local* toggles). This is the+pre-policy behaviour: maximum reach, minimum restriction.++## <a id="public_only"></a>☁️ Public only++Only **public** web/academic engines run; your **local collections are+excluded**. URL fetches are allowed to public hosts and blocked for private+ones. Inference is whatever you configured (cloud allowed). Use it when you+want public research and don't want your private documents queried at all.++## <a id="private_only"></a>🔒 Private only++The privacy mode. **Only local engines** run (collections, library, a local+SearXNG/Ollama). Crucially, it **forces local LLM and embeddings** — cloud+providers (OpenAI, Anthropic, Google, OpenRouter, …) are blocked, so your+query and your retrieved documents never reach a cloud model. Public URL+fetches are blocked, and a process-wide socket guard blocks stray outbound+connections. **Nothing leaves the machine.**++> If you have no local LLM configured, a Private-only run will refuse rather+> than silently fall back to the cloud — that's intentional (fail-closed).++## <a id="strict"></a>🎯 Strict++The tightest mode: **only your single primary engine** runs — no expansion to+any other engine at all. (It's incompatible with a meta-picker primary like+`auto`/`parallel`, since those have no single concrete engine.) At the URL+layer it behaves like Private-only (private hosts allowed, public blocked), but+it does **not** force local inference — set the *Require local* toggles if you+also want local LLM/embeddings.++---++## Per-collection public/private++Each RAG collection has a **public/private** flag (default **private**):++- A **private** collection is excluded under *Public only* / *Adaptive-public*,+ and when used it forces local inference — its chunks never reach a cloud+ model.+- Mark a collection **public** (the *Public collection* checkbox when creating+ it) only if its contents are non-sensitive and you're happy to process them+ with cloud inference / use them under public scope.++## The two local-inference toggles++Independent of the scope, you can force local inference any time:++- **Require local LLM endpoint** — refuse cloud LLM providers / non-local URLs.+- **Require local embeddings** — refuse cloud embedders, and refuse a+ HuggingFace download for an uncached local model.++Both are **implied automatically** under *Private only* (and Adaptive-private),+which is why those toggles auto-check and lock when you select Private only.++---++## Per-research overrides++The three primary controls — **Egress Scope**, **Require local LLM endpoint**,+and **Require local embeddings** — also appear on the research-form page as+per-run dropdown / checkbox overrides. Values set there apply **only to that+research run** and do **not** persist to the settings database, so you can do a+one-off private run without changing your defaults.++## Audit log++Changes to any `policy.*` key, `llm.require_local_endpoint`,+`llm.allowed_local_hostnames`, or `embeddings.require_local` emit a+`policy_audit=True` log line so administrators can trace configuration changes.+Those audit lines are deliberately filtered out of the WebSocket progress+stream (they never reach browser subscribers).++---++### See also++- [Configuration reference](CONFIGURATION.md#settings-list) — the exact setting+ keys (`policy.egress_scope`, `llm.require_local_endpoint`,+ `embeddings.require_local`, `llm.allowed_local_hostnames`) and their+ auto-generated `LDR_*` environment variables.+- [`SECURITY.md`](../SECURITY.md#egress-policy-module) — threat model,+ guarantees, and caveats (including what this does **not** defend against).diff --git a/src/local_deep_research/advanced_search_system/filters/journal_reputation_filter.py b/src/local_deep_research/advanced_search_system/filters/journal_reputation_filter.pyindex 49df22ae0e..3d50816030 100644--- a/src/local_deep_research/advanced_search_system/filters/journal_reputation_filter.py+++ b/src/local_deep_research/advanced_search_system/filters/journal_reputation_filter.py@@ -49,6 +49,7 @@ from ...database.session_context import get_user_db_session# is single-valued across every writer.from ...journal_quality.db import get_db as get_journal_data_managerfrom ...journal_quality.scoring import normalize_name+from ...security.egress.policy import PolicyDeniedErrorfrom ...security.log_sanitizer import strip_control_charsfrom ...utilities.llm_utils import get_model_identifierfrom ...utilities.resource_utils import safe_close@@ -221,7 +222,11 @@ class JournalReputationFilter(BaseFilter):self._owns_llm = self.model is Noneif self.model is None:- self.model = get_llm()+ # Forward the snapshot so the LLM PEP at llm_config.py:295+ # can evaluate ``llm.require_local_endpoint`` / egress scope.+ # Dropping it here previously bypassed the PEP entirely on+ # search engines that constructed this filter with no llm.+ self.model = get_llm(settings_snapshot=settings_snapshot)# Import here to avoid circular importfrom ...config.search_config import get_setting_from_snapshot@@ -327,6 +332,44 @@ class JournalReputationFilter(BaseFilter):if self._owns_llm:safe_close(self.model, "journal filter LLM")+ def _should_skip_journal_fetch_for_scope(self) -> bool:+ """Return True when the active egress scope forbids public+ fetches (PRIVATE_ONLY / STRICT). The journal-data sources+ (OpenAlex, DOAJ, JabRef) are all public, so under these scopes+ the daemon thread would burn cycles and (worse) attempt egress+ the user explicitly opted out of.+ """+ snapshot = self.__settings_snapshot+ if not snapshot:+ return False+ try:+ from ...security.egress.policy import (+ EgressScope,+ PolicyDeniedError,+ context_from_snapshot,+ )++ primary_raw = snapshot.get("search.tool", "auto")+ if isinstance(primary_raw, dict) and "value" in primary_raw:+ primary_raw = primary_raw["value"]+ ctx = context_from_snapshot(snapshot, primary_raw or "auto")+ return ctx.scope in (EgressScope.PRIVATE_ONLY, EgressScope.STRICT)+ except PolicyDeniedError:+ # Corrupt/unknown scope value — we cannot certify that a public+ # journal fetch is permitted, so SKIP the fetch (fail closed).+ # This matches the hardened siblings mcp_strategy._mcp_scope_blocked+ # and notifications.manager._filter_urls_by_egress_policy, which+ # also refuse rather than proceed when the policy is unevaluable.+ logger.bind(policy_audit=True).warning(+ "journal fetch skipped: egress policy unevaluable "+ "(corrupt scope) — failing closed"+ )+ return True+ except Exception:+ # Other errors (missing key, snapshot shape) — fail open; the+ # per-URL runtime check still gates each individual fetch.+ return False+@classmethoddef create_default(cls,@@ -376,9 +419,15 @@ class JournalReputationFilter(BaseFilter):f"threshold={filt._JournalReputationFilter__threshold}")return filt+ except PolicyDeniedError:+ # The egress policy refused the LLM the filter needs. This+ # must propagate so the user sees a clear policy refusal —+ # silently returning None would let unfiltered results+ # through under ``llm.require_local_endpoint=True``.+ raiseexcept Exception:- # Any failure — settings read, filter init — returns None- # rather than silently defaulting to enabled. A separate+ # Any other failure — settings read, filter init — returns+ # None rather than silently defaulting to enabled. A separate# silent ``except Exception: enabled = True`` branch used to# wrap only the settings read, which hid legitimate# configuration errors; per CLAUDE.md, fallbacks have to be@@ -1237,7 +1286,18 @@ JOURNAL INFORMATION:# clicked Download manually. The 30-second TTL cache in# ensure_journal_data prevents multiple concurrent filter# workers from all racing to spawn the same download.- _start_background_journal_fetch()+ #+ # Egress policy: skip the background fetch under+ # PRIVATE_ONLY or STRICT — those scopes shouldn't egress at+ # all, and the journal data sources (OpenAlex / DOAJ /+ # JabRef bulk downloads) are all public hosts.+ if self._should_skip_journal_fetch_for_scope():+ logger.bind(policy_audit=True).info(+ "journal-data background fetch skipped: egress scope "+ "forbids public fetches"+ )+ else:+ _start_background_journal_fetch()# Respect exclude_non_published — results without a venue# still get dropped in pending mode, same as in the fulldiff --git a/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py b/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.pyindex 02ee1a94c0..79d262c389 100644--- a/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py+++ b/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py@@ -285,6 +285,7 @@ def _make_research_subtopic_tool(programmatic_mode: bool = False,fetch_mode: str = "summary_focus_query",overall_query: str = "",+ egress_context=None,):"""Create the ``research_subtopic`` tool that spawns parallel subagents.@@ -341,6 +342,7 @@ def _make_research_subtopic_tool(model=model,overall_query=overall_query,settings_snapshot=settings_snapshot,+ egress_context=egress_context,)if sub_fetch is not None:sub_tools.append(sub_fetch)@@ -375,10 +377,27 @@ def _make_research_subtopic_tool(logger.exception(f"Subagent failed for: {topic[:80]}")return f"Research on '{topic}' failed: {exc}"+ def _run_subagent_with_egress(topic: str) -> str:+ # threading.local is NOT inherited by ThreadPoolExecutor workers,+ # so the PEP-578 audit-hook backstop armed on the parent thread is+ # inactive here. Re-arm it for the subagent's lifetime (LLM,+ # web-search and fetch connects) so the secondary egress net has+ # parity with the main thread under PRIVATE_ONLY/STRICT. The+ # primary snapshot-based PEPs already gate these calls; this+ # restores defense-in-depth. active_egress_context is a no-op when+ # egress_context is None (fail-open build).+ from ...security.egress.audit_hook import active_egress_context++ with active_egress_context(egress_context):+ return run_subagent(topic)+ordered_results: dict[str, str] = {}num_workers = min(MAX_SUBAGENT_WORKERS, len(subtopics))with ThreadPoolExecutor(max_workers=num_workers) as executor:- futures = {executor.submit(run_subagent, t): t for t in subtopics}+ futures = {+ executor.submit(_run_subagent_with_egress, t): t+ for t in subtopics+ }try:for future in as_completed(futures, timeout=SUBAGENT_TIMEOUT_SECONDS@@ -513,6 +532,41 @@ class LangGraphAgentStrategy(BaseSearchStrategy):logger.debug("Could not extract engine name from class")return ""+ def _build_egress_context(self):+ """Construct the frozen ``EgressContext`` for this run.++ Returns ``None`` if a context can't be built (no snapshot, or+ invariant violation) — callers fall through to current behavior+ rather than crashing. Lazy import to avoid pulling the security+ module at strategy-class import time.+ """+ if not self.settings_snapshot:+ return None+ from local_deep_research.security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ )++ primary = self._get_current_engine_name() or self.settings_snapshot.get(+ "search.tool"+ )+ if isinstance(primary, dict) and "value" in primary:+ primary = primary["value"]+ try:+ return context_from_snapshot(+ self.settings_snapshot, primary or "auto"+ )+ except PolicyDeniedError:+ # Corrupted/invalid policy.egress_scope — re-raise so the+ # caller fails closed instead of silently running unfiltered.+ raise+ except (ValueError, KeyError, TypeError):+ logger.debug(+ "Could not build EgressContext for langgraph agent — "+ "falling back to unfiltered tool list"+ )+ return None+def _build_tools(self, overall_query: str = "") -> list:"""Build the LangChain tool list for the lead agent.@@ -522,6 +576,11 @@ class LangGraphAgentStrategy(BaseSearchStrategy):"""tools = []+ # Compute the policy context ONCE for this run. Threaded through+ # every tool builder so subagent threads — which don't inherit+ # thread-local state — get the same context as the lead agent.+ policy_ctx = self._build_egress_context()+# Web search (always present if we have a search engine)if self.search is not None:tools.append(@@ -541,15 +600,32 @@ class LangGraphAgentStrategy(BaseSearchStrategy):model=self.model,overall_query=overall_query,settings_snapshot=self.settings_snapshot,+ egress_context=policy_ctx,)if fetch is not None:tools.append(fetch)- # Specialized search engines+ # Specialized search engines (pre-filtered by egress policy).+ #+ # This is the core fix for the original LangGraph silent-expansion+ # complaint. The factory PEP catches engines at instantiation time,+ # but that's a runtime check — the LLM still SEES the forbidden+ # tool names in the schema and the latency of a denied tool call+ # leaks policy state. Filtering the tool list HERE means the+ # forbidden tools never reach create_agent(), and the LLM never+ # learns they exist.try:from local_deep_research.web_search_engines.search_engines_config import (get_available_engines,)+ from local_deep_research.security.egress.policy import (+ EgressScope,+ evaluate_engine,+ evaluate_retriever,+ )+ from local_deep_research.web_search_engines.retriever_registry import (+ retriever_registry,+ )available = get_available_engines(settings_snapshot=self.settings_snapshot,@@ -558,6 +634,47 @@ class LangGraphAgentStrategy(BaseSearchStrategy):for name, config in available.items():if name in ("auto", "meta") or name == current:continue++ # Under STRICT, register no specialized engines at all —+ # the agent gets only the primary web_search tool.+ if (+ policy_ctx is not None+ and policy_ctx.scope == EgressScope.STRICT+ ):+ continue++ # Under PUBLIC_ONLY / PRIVATE_ONLY, ask the PDP whether+ # this engine fits the scope. Retrievers route to+ # evaluate_retriever (engine-PDP returns engine_unknown+ # for them); plain engines route to evaluate_engine.+ if policy_ctx is not None:+ if config.get("is_retriever"):+ try:+ meta = retriever_registry.get_metadata(name)+ except AttributeError:+ meta = None+ decision = evaluate_retriever(+ name, policy_ctx, metadata=meta+ )+ else:+ # Pass the engine config as metadata so a+ # per-collection is_public classification is honored+ # without a redundant DB lookup per collection.+ decision = evaluate_engine(+ name,+ policy_ctx,+ settings_snapshot=self.settings_snapshot,+ metadata=config,+ )+ if not decision.allowed:+ logger.bind(policy_audit=True).info(+ "specialized tool filtered by egress policy",+ engine=name,+ scope=policy_ctx.scope.value,+ reason=decision.reason,+ )+ continue+desc = config.get("description", f"Search using {name}")strengths = config.get("strengths", [])if strengths:@@ -590,6 +707,7 @@ class LangGraphAgentStrategy(BaseSearchStrategy):programmatic_mode=self.programmatic_mode,fetch_mode=self.fetch_mode,overall_query=overall_query,+ egress_context=policy_ctx,))@@ -638,6 +756,53 @@ class LangGraphAgentStrategy(BaseSearchStrategy):)else: # fullfetch_line = "3. Use fetch_content to read full pages when snippets aren't enough.\n"+ # Build the policy addendum once so the system prompt can carry+ # explicit guidance to the LLM about which tools actually exist.+ # Closing the timing-leak attack requires both halves: the tool+ # list is pre-filtered above, AND the LLM is told what's+ # available, so it doesn't waste tokens probing for forbidden+ # engines and the latency of denial paths doesn't leak policy.+ policy_addendum = ""+ try:+ from local_deep_research.security.egress.policy import (+ EgressScope,+ PolicyDeniedError,+ )++ ctx = self._build_egress_context()+ if ctx is not None and ctx.scope == EgressScope.STRICT:+ policy_addendum = (+ "\nRESTRICTED MODE: only the primary search tool is "+ f"available ({ctx.primary_engine}). Do NOT reference or "+ "attempt other search_* tools — they do not exist in "+ "this session and will not work. Use web_search and "+ "research_subtopic for everything.\n"+ )+ elif ctx is not None and ctx.scope == EgressScope.PRIVATE_ONLY:+ policy_addendum = (+ "\nPRIVATE-ONLY MODE: public search engines (arxiv, "+ "pubmed, brave, etc.) are not available in this "+ "session. Use only local search tools.\n"+ )+ elif ctx is not None and ctx.scope == EgressScope.PUBLIC_ONLY:+ policy_addendum = (+ "\nPUBLIC-ONLY MODE: local search tools (library, "+ "collection, paperless) are not available in this "+ "session.\n"+ )+ except PolicyDeniedError:+ # Corrupt/unknown scope must fail closed, never run unfiltered.+ # In practice _build_tools() (called above at the top of+ # analyze_topic) already raised this for the same snapshot, so+ # we never reach here with a bad scope — but re-raise rather+ # than swallow, so this stays correct if the call order changes.+ raise+ except Exception:+ logger.debug(+ "Could not derive policy addendum for system prompt — "+ "agent will see the unmodified prompt"+ )+system_prompt = (f"You are a research assistant writing a research report. Today's date: {current_date}.\n""This is NOT a chat conversation. Your only job is to research the "@@ -654,6 +819,7 @@ class LangGraphAgentStrategy(BaseSearchStrategy):"(search_arxiv for science, search_pubmed for medical, etc.).\n""5. When you have enough information, provide a comprehensive answer ""citing sources as [1], [2], etc.\n"+ f"{policy_addendum}")# Create agent — may fail if model doesn't support tool calling.diff --git a/src/local_deep_research/advanced_search_system/strategies/mcp_strategy.py b/src/local_deep_research/advanced_search_system/strategies/mcp_strategy.pyindex 39cb986f78..56cbc36bf1 100644--- a/src/local_deep_research/advanced_search_system/strategies/mcp_strategy.py+++ b/src/local_deep_research/advanced_search_system/strategies/mcp_strategy.py@@ -595,6 +595,34 @@ class MCPSearchStrategy(BaseSearchStrategy):})+ # Egress filter for specialized-engine tools — parity with the+ # LangGraph strategy: hide tools the active scope forbids so the LLM+ # never sees them (the schema + denied-tool latency would otherwise+ # leak policy state). The runtime PEP at create_search_engine still+ # backstops execution, but filtering here is the primary line.+ policy_ctx = None+ policy_blocks_all_specialized = False+ if self.settings_snapshot:+ try:+ from ...security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ )++ _primary = self.settings_snapshot.get("search.tool", "auto")+ if isinstance(_primary, dict) and "value" in _primary:+ _primary = _primary["value"]+ policy_ctx = context_from_snapshot(+ self.settings_snapshot, _primary or "auto"+ )+ except PolicyDeniedError:+ # Corrupt / incoherent scope → fail closed: expose no+ # specialized engine tools at all.+ policy_blocks_all_specialized = True+ except (ValueError, ImportError):+ # Unevaluable policy → fall through; the runtime PEP enforces.+ policy_ctx = None+# Add specialized search engine tools dynamically from available enginesavailable_engines = self._get_available_search_engines()for engine_name, engine_config in available_engines.items():@@ -606,6 +634,48 @@ class MCPSearchStrategy(BaseSearchStrategy):):continue+ # Apply the egress filter computed above.+ if policy_blocks_all_specialized:+ continue+ if policy_ctx is not None:+ from ...security.egress.policy import (+ EgressScope,+ evaluate_engine,+ evaluate_retriever,+ )++ if policy_ctx.scope == EgressScope.STRICT:+ # STRICT exposes only the primary (web_search); no+ # specialized engines.+ continue+ if engine_config.get("is_retriever"):+ from ...web_search_engines.retriever_registry import (+ retriever_registry,+ )++ try:+ _meta = retriever_registry.get_metadata(engine_name)+ except AttributeError:+ _meta = None+ _decision = evaluate_retriever(+ engine_name, policy_ctx, metadata=_meta+ )+ else:+ _decision = evaluate_engine(+ engine_name,+ policy_ctx,+ settings_snapshot=self.settings_snapshot,+ metadata=engine_config,+ )+ if not _decision.allowed:+ logger.bind(policy_audit=True).info(+ "MCP specialized tool filtered by egress policy",+ engine=engine_name,+ scope=policy_ctx.scope.value,+ reason=_decision.reason,+ )+ continue+# Create tool for this enginedescription = engine_config.get("description", f"Search using {engine_name}"@@ -658,6 +728,51 @@ class MCPSearchStrategy(BaseSearchStrategy):return tools+ def _mcp_scope_blocked(self):+ """Decide whether MCP tools must be suppressed under the current+ egress scope.++ Returns ``(blocked: bool, reason: str)``. MCP tools execute in+ stdio subprocesses whose egress the PDP cannot inspect, so the+ only safe enforcement is to block them outright under+ ``STRICT`` / ``PRIVATE_ONLY``. When no policy snapshot is+ available we fail closed: scope cannot be evaluated, so we+ cannot certify that MCP tools are safe to expose.++ ``self.settings_snapshot`` is set to ``{}`` (truthy-empty+ guard from BaseSearchStrategy) for callers that did not+ forward a snapshot — that path returns ``(True, "no_snapshot")``+ rather than silently allowing MCP tools.+ """+ snapshot = self.settings_snapshot+ if not snapshot:+ return True, "no_snapshot"+ try:+ from ...security.egress.policy import (+ EgressScope,+ PolicyDeniedError,+ context_from_snapshot,+ )+ except ImportError:+ # Module not available — preserve historical behavior+ # (allow MCP). Live deployments always have the module.+ return False, ""++ primary = snapshot.get("search.tool", "auto")+ if isinstance(primary, dict) and "value" in primary:+ primary = primary["value"]+ try:+ ctx = context_from_snapshot(snapshot, primary or "auto")+ except (PolicyDeniedError, ValueError) as exc:+ # Policy itself is unevaluable (corrupt scope value, or+ # STRICT + meta-picker incoherence). Cannot certify MCP is+ # safe — fail closed.+ return True, f"policy_unavailable:{exc}"++ if ctx.scope in (EgressScope.STRICT, EgressScope.PRIVATE_ONLY):+ return True, ctx.scope.value+ return False, ""+def _discover_mcp_tools(self) -> List[Dict[str, Any]]:"""Discover tools from configured MCP servers.@@ -672,6 +787,41 @@ class MCPSearchStrategy(BaseSearchStrategy):if not self.mcp_servers:return tools+ # Egress-policy gate. Runs BEFORE the broad ``except Exception``+ # block below so a fail-closed decision (no snapshot, corrupt+ # scope, STRICT/PRIVATE_ONLY) cannot be swallowed and silently+ # downgraded to "discovery failed → retry next call". Cache the+ # empty list explicitly: scope is immutable for the strategy's+ # lifetime, so suppression is final.+ blocked, reason = self._mcp_scope_blocked()+ if blocked:+ logger.bind(policy_audit=True).warning(+ "MCP tool discovery suppressed by egress policy",+ reason=reason,+ server_count=len(self.mcp_servers),+ )+ try:+ self._update_progress(+ f"MCP tools disabled by egress policy ({reason})",+ None,+ {+ "phase": "policy",+ "type": "policy_block",+ "reason": "mcp_scope_blocked",+ "scope": reason,+ },+ )+ except Exception: # noqa: silent-exception+ # _update_progress is a best-effort UI signal — never+ # let a progress callback failure mask the policy+ # decision itself.+ logger.debug(+ "MCP scope-block progress callback failed",+ exc_info=True,+ )+ self._mcp_tools_cache = []+ return self._mcp_tools_cache+try:from local_deep_research.mcp.client import (MCPClientManager,@@ -1152,11 +1302,58 @@ class MCPSearchStrategy(BaseSearchStrategy):read_js_rendering_setting,)+ # Build an EgressContext for this fetch so the in-fetcher+ # scope check (content_fetcher/fetcher.py) gates the URL+ # against the user's policy. Without this, MCP+ # download_content would do raw SSRF-only validation while+ # the LangGraph fetch tool's _enforce_url_policy already+ # blocks the same vector.+ egress_context = None+ if self.settings_snapshot:+ try:+ from local_deep_research.security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ )+ except ImportError:+ logger.debug(+ "egress_policy unavailable for MCP "+ "download_content; using SSRF-only check"+ )+ else:+ primary = self.settings_snapshot.get("search.tool", "auto")+ if isinstance(primary, dict) and "value" in primary:+ primary = primary["value"]+ try:+ egress_context = context_from_snapshot(+ self.settings_snapshot, primary or "auto"+ )+ except (PolicyDeniedError, ValueError) as exc:+ # Snapshot was supplied but the policy itself+ # cannot be evaluated (corrupt scope value, or+ # STRICT + meta-picker incoherence). Fail closed+ # — the previous bare-except dropped to+ # SSRF-only here, which allows public-IP fetches+ # under PRIVATE_ONLY.+ logger.bind(policy_audit=True).warning(+ "MCP download_content refused: egress "+ "policy could not be evaluated",+ reason=str(exc),+ )+ return {+ "status": "error",+ "content": (+ "Egress policy could not be evaluated; "+ "refusing fetch"+ ),+ }+with ContentFetcher(timeout=self.CONTENT_FETCHER_TIMEOUT,enable_js_rendering=read_js_rendering_setting(self.settings_snapshot),+ egress_context=egress_context,) as fetcher:result = fetcher.fetch(url, max_length=max_length)@@ -1446,6 +1643,7 @@ class MCPSearchStrategy(BaseSearchStrategy):from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)+ from ...security.egress.policy import PolicyDeniedErrorlogger.info(f"Specialized search with {engine_name}: {query[:100]}")@@ -1513,6 +1711,23 @@ class MCPSearchStrategy(BaseSearchStrategy):safe_close(engine, "specialized search engine")+ except PolicyDeniedError as exc:+ # The tool-list filter should already hide forbidden engines, but+ # if a denied engine is invoked anyway (cached tool list, race),+ # surface the policy denial in the audit trail instead of letting+ # the broad handler below downgrade it to a generic tool error.+ logger.bind(policy_audit=True).warning(+ "MCP specialized search refused by egress policy",+ engine=engine_name,+ reason=exc.decision.reason,+ )+ return {+ "status": "error",+ "content": (+ f"{engine_name} search refused by egress policy "+ f"({exc.decision.reason})"+ ),+ }except Exception as e:logger.exception(f"Specialized search with {engine_name} failed")return {@@ -1538,6 +1753,27 @@ class MCPSearchStrategy(BaseSearchStrategy):"""if timeout is None:timeout = self.MCP_TOOL_CALL_TIMEOUT++ # Defense-in-depth: discovery should already have suppressed+ # MCP tools under STRICT/PRIVATE_ONLY/no-snapshot. Re-check+ # before any subprocess egress in case a cached tool list+ # survives a config drift. Runs BEFORE the broad try so the+ # decision is not converted to a generic "MCP tool failed".+ blocked, reason = self._mcp_scope_blocked()+ if blocked:+ logger.bind(policy_audit=True).warning(+ "MCP tool execution refused by egress policy",+ reason=reason,+ server=server_name,+ tool=tool_name,+ )+ return {+ "status": "error",+ "content": (+ f"MCP tool '{server_name}.{tool_name}' blocked by "+ f"egress policy ({reason})"+ ),+ }try:from local_deep_research.mcp.client import (MCPClientManager,diff --git a/src/local_deep_research/advanced_search_system/strategies/news_strategy.py b/src/local_deep_research/advanced_search_system/strategies/news_strategy.pyindex c470e437c6..54e3e86319 100644--- a/src/local_deep_research/advanced_search_system/strategies/news_strategy.py+++ b/src/local_deep_research/advanced_search_system/strategies/news_strategy.py@@ -249,12 +249,24 @@ Ensure variety across different categories when possible.# We're in an async context — run in a thread to avoid nestingimport concurrent.futures+ # threading.local is NOT inherited by the pool worker, so the+ # PEP-578 audit-hook backstop armed on this (research) thread would+ # be inactive for the LLM analysis below. Capture and re-arm it in+ # the worker so the secondary egress net keeps parity under+ # PRIVATE_ONLY/STRICT (the primary LLM PEP still gates regardless).+ from ...security.egress.audit_hook import (+ active_egress_context,+ get_active_context,+ )++ _egress_ctx = get_active_context()++ def _analyze_with_egress():+ with active_egress_context(_egress_ctx):+ return asyncio.run(self.analyze_findings(all_findings))+with concurrent.futures.ThreadPoolExecutor() as executor:- future = executor.submit(- thread_cleanup(- lambda: asyncio.run(self.analyze_findings(all_findings))- )- )+ future = executor.submit(thread_cleanup(_analyze_with_egress))analysis = future.result()except RuntimeError:# No running event loop — asyncio.run() creates one, runs thediff --git a/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py b/src/local_deep_research/advanced_search_system/tools/fetch/__init__.pyindex 46fdf23d07..df422efa01 100644--- a/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py+++ b/src/local_deep_research/advanced_search_system/tools/fetch/__init__.py@@ -77,19 +77,50 @@ def _register_in_collector(return start + 1+def _enforce_url_policy(url: str, egress_context: Any) -> None:+ """Run ``evaluate_url`` against ``egress_context`` and raise+ ``PolicyDeniedError`` on denial.++ No-op when no context is configured (callers without policy enforcement,+ e.g. legacy non-LangGraph strategies, see the legacy behavior).+ """+ if egress_context is None:+ return+ from local_deep_research.security.egress.policy import (+ PolicyDeniedError,+ evaluate_url,+ )++ decision = evaluate_url(url, egress_context)+ if not decision.allowed:+ raise PolicyDeniedError(decision, target=url)++def _make_full_fetch_tool(- collector: Any, settings_snapshot: dict | None = None+ collector: Any,+ settings_snapshot: dict | None = None,+ egress_context: Any = None,):@tooldef fetch_content(url: str) -> str:"""Download and read the full text content from a URL. Use when search snippets aren't detailed enough."""from local_deep_research.content_fetcher import ContentFetcher+ from local_deep_research.security.egress.policy import (+ PolicyDeniedError,+ )++ # Policy gate: raise PolicyDeniedError so it propagates to the+ # agent layer (matching the factory PEP contract). Caught+ # separately from transient fetch errors so the agent can tell+ # them apart.+ _enforce_url_policy(url, egress_context)enable_js = _read_js_rendering_setting(settings_snapshot)try:with ContentFetcher(timeout=CONTENT_FETCH_TIMEOUT,enable_js_rendering=enable_js,+ egress_context=egress_context,) as fetcher:result = fetcher.fetch(url, max_length=CONTENT_MAX_LENGTH)if result.get("status") == "success":@@ -102,6 +133,8 @@ def _make_full_fetch_tool(f"[{cite_idx}] Title: {title}\nURL: {url}\n\n{content}")return f"Failed to fetch {url}: {result.get('error', 'unknown error')}"+ except PolicyDeniedError:+ raiseexcept Exception as exc:logger.exception("fetch_content tool error")return f"Error fetching {url}: {exc}"@@ -114,6 +147,7 @@ def _make_summary_fetch_tool(model: BaseChatModel,overall_query: str | None,settings_snapshot: dict | None = None,+ egress_context: Any = None,):"""Build the summary-mode fetch tool.@@ -132,12 +166,18 @@ def _make_summary_fetch_tool(tool will quote relevant facts verbatim and discard unrelated content."""from local_deep_research.content_fetcher import ContentFetcher+ from local_deep_research.security.egress.policy import (+ PolicyDeniedError,+ )++ _enforce_url_policy(url, egress_context)enable_js = _read_js_rendering_setting(settings_snapshot)try:with ContentFetcher(timeout=CONTENT_FETCH_TIMEOUT,enable_js_rendering=enable_js,+ egress_context=egress_context,) as fetcher:result = fetcher.fetch(url, max_length=CONTENT_MAX_LENGTH)if result.get("status") != "success":@@ -190,6 +230,8 @@ def _make_summary_fetch_tool(collector, url, title, summary or content)return f"[{cite_idx}] Title: {title}\nURL: {url}\n\n{summary}"+ except PolicyDeniedError:+ raiseexcept Exception as exc:logger.exception("fetch_content tool error")return f"Error fetching {url}: {exc}"@@ -204,6 +246,7 @@ def build_fetch_tool(model: BaseChatModel | None = None,overall_query: str = "",settings_snapshot: dict | None = None,+ egress_context: Any = None,):"""Build the agent-facing ``fetch_content`` tool for *mode*.@@ -215,12 +258,18 @@ def build_fetch_tool(``settings_snapshot`` is captured by the tool closure so the per-callJS-rendering toggle can be read on a worker thread (where``threading.local`` context does not propagate).++ ``egress_context`` is captured by the closure so the per-call URL+ can be policy-gated; when ``None``, no policy enforcement runs+ (preserves legacy non-LangGraph callers)."""if mode == "disabled":return Noneif mode == "full":return _make_full_fetch_tool(- collector, settings_snapshot=settings_snapshot+ collector,+ settings_snapshot=settings_snapshot,+ egress_context=egress_context,)if mode == "summary_focus":if model is None:@@ -230,6 +279,7 @@ def build_fetch_tool(model,overall_query=None,settings_snapshot=settings_snapshot,+ egress_context=egress_context,)if mode == "summary_focus_query":if model is None:@@ -241,6 +291,7 @@ def build_fetch_tool(model,overall_query=overall_query or None,settings_snapshot=settings_snapshot,+ egress_context=egress_context,)raise ValueError(f"Unknown fetch mode {mode!r}; expected one of {FETCH_MODES}"diff --git a/src/local_deep_research/api/research_functions.py b/src/local_deep_research/api/research_functions.pyindex 9fe1e66edf..429b544483 100644--- a/src/local_deep_research/api/research_functions.py+++ b/src/local_deep_research/api/research_functions.py@@ -297,6 +297,9 @@ def quick_summary("user_password": kwargs.get("user_password"), # Include password for metrics tracking+ # Carried so SearchCache._get_query_hash can read the active+ # egress scope and avoid cross-scope cache hits.+ "settings_snapshot": kwargs.get("settings_snapshot") or {},}set_search_context(search_context)@@ -460,6 +463,7 @@ def generate_report("search_engine_selected": kwargs.get("search_tool"),"username": username,"user_password": kwargs.get("user_password"),+ "settings_snapshot": kwargs.get("settings_snapshot") or {},}set_search_context(search_context)@@ -574,6 +578,7 @@ def detailed_research("search_engine_selected": kwargs.get("search_tool"),"username": username,"user_password": kwargs.get("user_password"),+ "settings_snapshot": kwargs.get("settings_snapshot") or {},}set_search_context(search_context)diff --git a/src/local_deep_research/benchmarks/graders.py b/src/local_deep_research/benchmarks/graders.pyindex 8ce380cd01..326868413d 100644--- a/src/local_deep_research/benchmarks/graders.py+++ b/src/local_deep_research/benchmarks/graders.py@@ -99,8 +99,12 @@ def get_evaluation_llm(# Try to fall back to LDR's config if API key not explicitly provided# The get_llm function will handle this case- # Get the LLM using LDR's existing function- return get_llm(**filtered_config)+ # Get the LLM using LDR's existing function. Thread settings_snapshot+ # through — without it, get_llm's snapshot-less PEP only permits local+ # default providers, so a cloud grader (e.g. openai) would be refused+ # with PolicyDeniedError. The snapshot lets get_llm evaluate the policy+ # and permit the configured grader when the user's scope allows it.+ return get_llm(**filtered_config, settings_snapshot=settings_snapshot)def extract_answer_from_response(@@ -304,6 +308,7 @@ def grade_results(dataset_type: str = "simpleqa",evaluation_config: Optional[Dict[str, Any]] = None,progress_callback: Optional[Callable[[int, int, Dict], None]] = None,+ settings_snapshot: Optional[Dict[str, Any]] = None,) -> List[Dict[str, Any]]:"""Grade benchmark results using LLM.@@ -314,12 +319,16 @@ def grade_results(dataset_type: Type of datasetevaluation_config: Optional custom config for evaluation LLMprogress_callback: Optional callback for progress updates+ settings_snapshot: Optional snapshot so the grader LLM is+ constructed under the user's egress policy. Without it+ the LLM PEP's snapshot guard skips and the grader can+ silently reach a cloud LLM under require_local_endpoint.Returns:List of graded results"""# Get evaluation LLM- evaluation_llm = get_evaluation_llm(evaluation_config)+ evaluation_llm = get_evaluation_llm(evaluation_config, settings_snapshot)try:return _grade_results_inner(diff --git a/src/local_deep_research/benchmarks/runners.py b/src/local_deep_research/benchmarks/runners.pyindex 2b23aaa11a..044818df80 100644--- a/src/local_deep_research/benchmarks/runners.py+++ b/src/local_deep_research/benchmarks/runners.py@@ -49,6 +49,7 @@ def run_benchmark(human_evaluation: bool = False,progress_callback: Optional[Callable[[str, int, Dict], None]] = None,seed: int = 42,+ settings_snapshot: Optional[Dict[str, Any]] = None,) -> Dict[str, Any]:"""Run a benchmark on the specified dataset.@@ -192,7 +193,11 @@ def run_benchmark(# Time the searchstart_time = time.time()- # Get response from LDR+ # Get response from LDR. Pass settings_snapshot through so+ # the egress policy PEP fires on the benchmark's inner+ # search-engine / LLM construction calls. Without it, the+ # PEP's ``if settings_snapshot is not None`` guard skips and+ # benchmarks silently bypass the user's saved scope.search_result = quick_summary(query=formatted_query,iterations=search_config.get("iterations", 3),@@ -200,6 +205,7 @@ def run_benchmark("questions_per_iteration", 3),search_tool=search_config.get("search_tool", "searxng"),+ settings_snapshot=settings_snapshot,)end_time = time.time()@@ -307,6 +313,7 @@ def run_benchmark(output_file=evaluation_file,dataset_type=dataset_type,evaluation_config=evaluation_config,+ settings_snapshot=settings_snapshot,progress_callback=lambda current, total, meta: (progress_callback(f"Evaluating {current + 1}/{total}",diff --git a/src/local_deep_research/config/llm_config.py b/src/local_deep_research/config/llm_config.pyindex e2ef08d538..45e6149c09 100644--- a/src/local_deep_research/config/llm_config.py+++ b/src/local_deep_research/config/llm_config.py@@ -287,6 +287,84 @@ def get_llm(# Normalize provider: convert to lowercase canonical formprovider = normalize_provider(provider)+ # Egress policy PEP for LLM endpoints. Fires here (before the registered-+ # LLM dispatch) because all built-in providers are auto-registered via+ # discover_providers(), so the registered branch handles every real LLM+ # call.+ #+ # Two paths: snapshot-present runs the full PEP; snapshot-absent runs+ # an allow-list check so background helpers / scaffolding paths cannot+ # silently instantiate a cloud LLM. The allow-list (known-local) is+ # deliberately tight — any provider not in it (including ambiguous+ # ones like ``openai_endpoint`` and future cloud providers) fails+ # closed instead of bypassing the PEP.+ if provider:+ from ..security.egress.policy import (+ Decision,+ PolicyDeniedError,+ _LOCAL_DEFAULT_LLM_PROVIDERS,+ _is_user_registered_llm,+ context_from_snapshot,+ evaluate_llm_endpoint,+ )++ if settings_snapshot is None:+ # User-registered in-process LLMs are exempt here for the same+ # reason evaluate_llm_endpoint allows them: no endpoint to+ # classify, operator-injected, audit-hook backstopped.+ if provider not in _LOCAL_DEFAULT_LLM_PROVIDERS and not (+ _is_user_registered_llm(provider)+ ):+ logger.bind(policy_audit=True).warning(+ "LLM constructed without policy snapshot; refusing "+ "non-local provider",+ provider=provider,+ )+ raise PolicyDeniedError(+ Decision(False, "no_snapshot_for_provider"),+ target=provider,+ )+ else:+ primary_engine_raw = settings_snapshot.get("search.tool", "auto")+ if (+ isinstance(primary_engine_raw, dict)+ and "value" in primary_engine_raw+ ):+ primary_engine_raw = primary_engine_raw["value"]++ try:+ ctx = context_from_snapshot(+ settings_snapshot, primary_engine_raw or "auto"+ )+ except ValueError as exc:+ # STRICT + meta-picker is incoherent. Failing closed here is+ # essential — previously we silently fell back to ctx=None+ # which skipped the LLM endpoint check entirely, opening a+ # cloud-LLM bypass under the very configuration the user+ # asked to be strict about. Raise PolicyDeniedError instead+ # so the misconfiguration surfaces as a hard stop.+ logger.bind(policy_audit=True).warning(+ "invalid egress policy configuration; refusing LLM",+ provider=provider,+ reason=str(exc),+ )+ raise PolicyDeniedError(+ Decision(False, "invalid_policy_config"),+ target=provider,+ ) from exc++ if ctx is not None and ctx.require_local_llm:+ decision = evaluate_llm_endpoint(+ provider, ctx, settings_snapshot=settings_snapshot+ )+ if not decision.allowed:+ logger.bind(policy_audit=True).warning(+ "LLM endpoint denied by egress policy",+ provider=provider,+ reason=decision.reason,+ )+ raise PolicyDeniedError(decision, target=provider)+# Check if this is a registered custom LLM firstif provider and is_llm_registered(provider):logger.info(f"Using registered custom LLM: {provider}")diff --git a/src/local_deep_research/content_fetcher/fetcher.py b/src/local_deep_research/content_fetcher/fetcher.pyindex 2aa8b087b5..0b9964d903 100644--- a/src/local_deep_research/content_fetcher/fetcher.py+++ b/src/local_deep_research/content_fetcher/fetcher.py@@ -12,7 +12,7 @@ from loguru import loggerfrom .url_classifier import URLClassifier, URLTypefrom ..research_library.downloaders.base import ContentType-from ..security.ssrf_validator import validate_url+from ..security.egress.fetch import policy_aware_validate_urlfrom ..utilities.resource_utils import safe_close# Default maximum content length (500KB of text)@@ -34,6 +34,7 @@ class ContentFetcher:timeout: int = 30,language: str = "English",enable_js_rendering: bool = False,+ egress_context: Any = None,):"""Initialize the content fetcher.@@ -53,10 +54,17 @@ class ContentFetcher:on Docker without Chromium anyway — so disabling by defaultdoes not regress observed quality. The user-facing toggle isthe ``web.enable_javascript_rendering`` setting.+ egress_context: Optional ``EgressContext`` from the active+ policy. When present and the run's scope is+ ``PRIVATE_ONLY``, SSRF validation permits private IPs so+ local lab deployments (Ollama at 127.0.0.1, SearXNG on+ 192.168.x) can actually be reached without forcing the+ operator to set SSRF_ALLOW_PRIVATE_IPS=1 globally."""self.timeout = timeoutself.language = languageself.enable_js_rendering = enable_js_rendering+ self.egress_context = egress_contextself._downloaders: Dict[URLType, Any] = {}def _get_downloader(self, url_type: URLType):@@ -146,10 +154,48 @@ class ContentFetcher:# Cache the downloaderif downloader:+ self._apply_egress_policy_to_downloader(downloader)self._downloaders[url_type] = downloaderreturn downloader+ def _apply_egress_policy_to_downloader(self, downloader: Any) -> None:+ """Relax a downloader's SafeSession to allow private IPs when the+ active scope is PRIVATE_ONLY, mirroring ``policy_aware_validate_url``.++ Without this, a private/lab URL that ContentFetcher already approved+ (policy_aware_validate_url + evaluate_url both allow private hosts+ under PRIVATE_ONLY) is then rejected by the downloader's OWN strict+ SafeSession SSRF re-validation (allow_private_ips defaults to False),+ breaking PRIVATE_ONLY's documented "reach your local services" use+ case. ``SafeSession.request`` reads ``allow_private_ips`` per-request,+ so setting it post-construction takes effect. Cloud-metadata IPs stay+ blocked regardless (``is_ip_blocked`` always rejects them).+ """+ if downloader is None or self.egress_context is None:+ return+ # Narrow except (not bare Exception): the only things that can throw+ # here are the policy import and the ctx.scope attribute access. The+ # failure direction is fail-SAFE — if we don't set allow_private_ips it+ # stays False (strict SSRF), i.e. over-restrictive, never a bypass — but+ # catching narrowly avoids masking an unrelated bug (e.g. a refactor+ # that breaks the import) while still never breaking a fetch on the+ # expected misconfiguration cases.+ try:+ from ..security.egress.policy import EgressScope++ scope = self.egress_context.scope+ except (ImportError, AttributeError): # pragma: no cover - defensive+ logger.debug(+ "could not resolve egress scope for downloader session",+ exc_info=True,+ )+ return+ if scope == EgressScope.PRIVATE_ONLY:+ session = getattr(downloader, "session", None)+ if session is not None and hasattr(session, "allow_private_ips"):+ session.allow_private_ips = True+def fetch(self,url: str,@@ -192,8 +238,10 @@ class ContentFetcher:"error": "Invalid or unsupported URL scheme (only http/https allowed)",}- # SSRF validation: reject private/internal IPs before reaching downloaders- if not validate_url(url):+ # SSRF validation: reject private/internal IPs before reaching downloaders.+ # Policy-aware so PRIVATE_ONLY egress scope can actually reach+ # private hosts (lab deployments) without disabling SSRF globally.+ if not policy_aware_validate_url(url, self.egress_context):logger.warning(f"URL failed SSRF validation: {url}")return {"status": "error",@@ -202,6 +250,37 @@ class ContentFetcher:"error": "URL failed security validation (blocked by SSRF protection)",}+ # Egress policy: reject URLs that are SSRF-OK but scope-incompatible.+ # SSRF only checks the IP class; scope enforcement is a separate axis+ # (PRIVATE_ONLY blocks public hosts, PUBLIC_ONLY blocks private hosts).+ # Centralized here so all callers (LangGraph fetch tool, MCP+ # download_content, future tools) get uniform enforcement instead of+ # each remembering to wrap the call site.+ if self.egress_context is not None:+ from ..security.egress.policy import evaluate_url+ from ..security.ssrf_validator import redact_url_for_log++ url_decision = evaluate_url(url, self.egress_context)+ if not url_decision.allowed:+ logger.bind(policy_audit=True).warning(+ "fetch URL denied by egress policy",+ # Redact: a denied URL may carry userinfo creds / API-key+ # query params; log only scheme://host:port.+ url=redact_url_for_log(url),+ scope=self.egress_context.scope.value,+ reason=url_decision.reason,+ )+ # Return a structured error rather than raise — callers in+ # the fetch path already handle dict-shaped errors.+ return {+ "status": "error",+ "url": url,+ "source_type": source_name,+ "error": (+ f"URL refused by egress policy ({url_decision.reason})"+ ),+ }+logger.info(f"Fetching content from {url} (detected: {source_name})")# Get the appropriate downloaderdiff --git a/src/local_deep_research/database/migrations/versions/0011_add_collection_is_public.py b/src/local_deep_research/database/migrations/versions/0011_add_collection_is_public.pynew file mode 100644index 0000000000..2d27da240d--- /dev/null+++ b/src/local_deep_research/database/migrations/versions/0011_add_collection_is_public.py@@ -0,0 +1,97 @@+"""Add is_public column to collections++Gives each RAG collection an egress classification: is its content+non-sensitive ("public") or sensitive ("private")? Defaults to private+(False) — the safe choice. A private collection is excluded under+PUBLIC_ONLY egress scope and forces local LLM/embeddings inference when+used, so its chunks never reach a cloud model.++Revision ID: 0011+Revises: 0010+Create Date: 2026-06-04++Column Added:+=============+- is_public (BOOLEAN): Whether the collection's content may be treated as+ public (cloud inference permitted, allowed under PUBLIC_ONLY scope).+ NOT NULL, server_default 0 (private) so pre-existing collections are+ classified private — the safe default.++Migration Notes:+===============+- Uses SQLite batch mode (table recreation) for ALTER TABLE operations.+- Idempotent: skips the column if it already exists (fresh databases+ created after the model change will already have it).++Downgrade Behavior:+==================+The downgrade() function removes the is_public column. Any per-collection+public/private classification will be LOST on downgrade.+"""++from alembic import op+import sqlalchemy as sa+from sqlalchemy import inspect++# revision identifiers, used by Alembic.+revision = "0011"+down_revision = "0010"+branch_labels = None+depends_on = None+++def column_exists(table_name: str, column_name: str) -> bool:+ """Check if a column exists in a table."""+ bind = op.get_bind()+ inspector = inspect(bind)++ if not inspector.has_table(table_name):+ return False++ columns = {col["name"] for col in inspector.get_columns(table_name)}+ return column_name in columns+++def upgrade():+ """Add is_public column to collections if it doesn't exist."""+ bind = op.get_bind()+ inspector = inspect(bind)++ if not inspector.has_table("collections"):+ # Table doesn't exist yet, will be created by initial migration+ return++ if column_exists("collections", "is_public"):+ # Column already exists (fresh database created after model change)+ return++ # Use batch mode for SQLite compatibility. server_default 0 so existing+ # collections are classified private (the safe default) without a+ # separate backfill UPDATE. nullable=True to match the ORM model column+ # (Collection.is_public, like its sibling is_default, is nullable) so the+ # schema-drift autogenerate check stays clean; the classification code+ # treats NULL as private anyway.+ with op.batch_alter_table("collections") as batch_op:+ batch_op.add_column(+ sa.Column(+ "is_public",+ sa.Boolean(),+ nullable=True,+ server_default=sa.false(),+ )+ )+++def downgrade():+ """Remove is_public column from collections."""+ bind = op.get_bind()+ inspector = inspect(bind)++ if not inspector.has_table("collections"):+ return++ if not column_exists("collections", "is_public"):+ return++ with op.batch_alter_table("collections") as batch_op:+ batch_op.drop_column("is_public")diff --git a/src/local_deep_research/database/models/library.py b/src/local_deep_research/database/models/library.pyindex 54b7a5df51..0f814d6d8e 100644--- a/src/local_deep_research/database/models/library.py+++ b/src/local_deep_research/database/models/library.py@@ -383,6 +383,13 @@ class Collection(Base):# Is this the default library collection?is_default = Column(Boolean, default=False)+ # Egress classification: is this collection's content non-sensitive+ # ("public") or sensitive ("private")? Defaults to private (False) — the+ # safe choice. A private collection is excluded under PUBLIC_ONLY scope+ # and forces local LLM/embeddings inference when used, so its chunks+ # never reach a cloud model. Operators flip it per-collection.+ is_public = Column(Boolean, default=False)+# Embedding model used for this collection (stored when first indexed)embedding_model = Column(String(100), nullable=Truediff --git a/src/local_deep_research/database/thread_local_session.py b/src/local_deep_research/database/thread_local_session.pyindex c899ea46ee..96236c67d2 100644--- a/src/local_deep_research/database/thread_local_session.py+++ b/src/local_deep_research/database/thread_local_session.py@@ -291,6 +291,18 @@ class _ThreadCleanup(ContextDecorator):"thread_cleanup: error clearing search context",exc_info=True,)+ try:+ # Defense-in-depth audit hook (PEP 578). Leaving an active+ # EgressContext on a pooled worker thread would let the+ # next, unrelated task inherit the previous run's scope.+ from ..security.egress.audit_hook import clear_active_context++ clear_active_context()+ except Exception:+ logger.debug(+ "thread_cleanup: error clearing egress audit context",+ exc_info=True,+ )return Falsediff --git a/src/local_deep_research/defaults/default_settings.json b/src/local_deep_research/defaults/default_settings.jsonindex c40c43e4d5..665ea92e22 100644--- a/src/local_deep_research/defaults/default_settings.json+++ b/src/local_deep_research/defaults/default_settings.json@@ -5111,5 +5111,138 @@"ui_element": "number","value": 7,"visible": true+ },+ "policy.egress_scope": {+ "category": "policy",+ "description": "EXPERIMENTAL — this egress boundary is new defense-in-depth, not an absolute guarantee yet; don't rely on it as your only protection for highly sensitive data. Restrict which search engines a research run may use. ADAPTIVE (default) = follow your primary engine: a private primary keeps everything local, a public primary allows public engines, a meta-picker allows both. STRICT = only the primary engine you picked; PUBLIC_ONLY = any public web/academic engine; PRIVATE_ONLY = any local engine (library, paperless, etc.); BOTH = any classified engine. STRICT is incompatible with auto/meta/parallel primaries.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Egress Scope (Experimental)",+ "options": [+ {+ "label": "Adaptive (default — follows your primary engine)",+ "value": "adaptive"+ },+ {+ "label": "Both (any classified engine)",+ "value": "both"+ },+ {+ "label": "Public only (no local collections)",+ "value": "public_only"+ },+ {+ "label": "Private only (no internet)",+ "value": "private_only"+ },+ {+ "label": "Strict (only the primary engine)",+ "value": "strict"+ }+ ],+ "step": null,+ "type": "SEARCH",+ "ui_element": "select",+ "value": "adaptive",+ "visible": true+ },+ "llm.require_local_endpoint": {+ "category": "policy",+ "description": "When enabled, the LLM provider's resolved endpoint must be on a local network (loopback, RFC1918, link-local). Cloud-only providers (OpenAI, Anthropic, Google, OpenRouter) are blocked. Independent of egress_scope.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Require Local LLM Endpoint",+ "options": null,+ "step": null,+ "type": "LLM",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },+ "embeddings.require_local": {+ "category": "policy",+ "description": "When enabled, the embeddings provider must be local (sentence_transformers, or Ollama pointed at a local URL). Indexing with cloud embeddings (OpenAI) is blocked. Indexing a corpus with cloud embeddings sends every chunk off-machine.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Require Local Embeddings",+ "options": null,+ "step": null,+ "type": "LLM",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },+ "llm.allowed_local_hostnames": {+ "category": "policy",+ "description": "Hostnames the policy will treat as local in addition to loopback / RFC1918 / link-local. Use for self-hosted services on custom hostnames (e.g. 'ollama.lan'). Public hostnames added here are rejected at save time.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Allowed Local Hostnames",+ "options": null,+ "step": null,+ "type": "LLM",+ "ui_element": "json",+ "value": [],+ "visible": true+ },+ "app.warnings.dismiss_egress_policy": {+ "category": "internal",+ "description": "Permanently hide the 'public search egress enabled' banner (the fresh-install informational one). The critical cloud-LLM and cloud-embeddings banners have their OWN dismiss flags so this can't silently hide them.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Dismiss public-egress warning",+ "options": null,+ "step": null,+ "type": "APP",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },+ "app.warnings.dismiss_cloud_llm": {+ "category": "internal",+ "description": "Permanently hide the 'LLM provider is cloud-hosted' banner. Separate from the public-egress dismiss flag so acknowledging fresh-install noise never suppresses this critical warning.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Dismiss cloud-LLM warning",+ "options": null,+ "step": null,+ "type": "APP",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },+ "app.warnings.dismiss_cloud_embeddings": {+ "category": "internal",+ "description": "Permanently hide the 'document chunks will be sent to OpenAI' (cloud embeddings) banner. Separate from the public-egress dismiss flag so acknowledging fresh-install noise never suppresses this critical warning.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Dismiss cloud-embeddings warning",+ "options": null,+ "step": null,+ "type": "APP",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },+ "app.warnings.dismiss_adaptive_scope_info": {+ "category": "internal",+ "description": "Permanently hide the informational banner that states what the Adaptive egress scope resolves to (public / private / both) for the current primary engine.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Dismiss adaptive-scope info banner",+ "options": null,+ "step": null,+ "type": "APP",+ "ui_element": "checkbox",+ "value": false,+ "visible": true}}diff --git a/src/local_deep_research/embeddings/embeddings_config.py b/src/local_deep_research/embeddings/embeddings_config.pyindex 22bd7793f1..59a06cef41 100644--- a/src/local_deep_research/embeddings/embeddings_config.py+++ b/src/local_deep_research/embeddings/embeddings_config.py@@ -167,6 +167,79 @@ def get_embeddings(logger.info(f"Getting embeddings with provider: {provider}, model: {model}")+ # Egress policy PEP: centralized here so callers that didn't run+ # the upstream RAG/library preflight checks still get gated.+ # Previously the PEP existed only in two callers+ # (rag_service_factory + library_rag_service.__init__) — any third+ # caller (e.g. /api/rag/test-embedding) bypassed it.+ if settings_snapshot is None:+ # Snapshot-less path: the embeddings.require_local toggle can't be+ # read, so we can't honour an opt-in restriction. Fail closed by+ # only allowing localhost-default providers — same contract as the+ # LLM gate in config/llm_config.py. A snapshot-less get_embeddings(+ # provider="openai") must NOT silently ship the local corpus to a+ # cloud embedder.+ from ..security.egress.policy import (+ _LOCAL_DEFAULT_EMBEDDING_PROVIDERS,+ Decision,+ PolicyDeniedError,+ )++ if provider not in _LOCAL_DEFAULT_EMBEDDING_PROVIDERS:+ logger.bind(policy_audit=True).warning(+ "embeddings denied: non-local provider with no settings "+ "snapshot to evaluate policy",+ provider=provider,+ )+ raise PolicyDeniedError(+ Decision(False, "no_snapshot_for_provider"),+ target=f"embeddings:{provider}",+ )+ else:+ from ..security.egress.policy import (+ Decision,+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_embeddings,+ )++ primary = (+ get_setting_from_snapshot(+ "search.tool",+ default="auto",+ settings_snapshot=settings_snapshot,+ )+ or "auto"+ )+ try:+ ctx = context_from_snapshot(settings_snapshot, primary)+ except PolicyDeniedError:+ raise+ except ValueError as exc:+ raise PolicyDeniedError(+ Decision(False, "invalid_policy_config"),+ target=f"embeddings:{provider}",+ ) from exc+ # Gate on the SCOPE-AWARE requirement, not the raw+ # embeddings.require_local flag: under PRIVATE_ONLY,+ # context_from_snapshot forces require_local_embeddings=True even+ # when the user left the flag at its default False. Reading the+ # raw flag here would miss that — a PRIVATE_ONLY run with the+ # default flag would ship the corpus to a cloud embedder.+ if ctx.require_local_embeddings:+ decision = evaluate_embeddings(+ provider, ctx, settings_snapshot=settings_snapshot+ )+ if not decision.allowed:+ logger.bind(policy_audit=True).warning(+ "embeddings denied by egress policy",+ provider=provider,+ reason=decision.reason,+ )+ raise PolicyDeniedError(+ decision, target=f"embeddings:{provider}"+ )+# Get provider class and create embeddingsprovider_classes = _get_provider_classes()provider_class = provider_classes.get(provider)diff --git a/src/local_deep_research/embeddings/providers/implementations/sentence_transformers.py b/src/local_deep_research/embeddings/providers/implementations/sentence_transformers.pyindex 794753856d..9389a3275e 100644--- a/src/local_deep_research/embeddings/providers/implementations/sentence_transformers.py+++ b/src/local_deep_research/embeddings/providers/implementations/sentence_transformers.py@@ -90,11 +90,94 @@ class SentenceTransformersProvider(BaseEmbeddingProvider):f"Creating SentenceTransformerEmbeddings with model={model}, device={device}")+ # Egress policy: if the user opted into local-only embeddings,+ # refuse to trigger a HuggingFace download on first use. The+ # SentenceTransformer constructor reaches out to huggingface.co+ # when the requested model isn't cached locally — silent+ # outbound traffic that violates ``embeddings.require_local=True``.+ #+ # Resolve the requirement through the egress CONTEXT, not the raw+ # ``embeddings.require_local`` flag: under PRIVATE_ONLY,+ # context_from_snapshot forces require_local_embeddings=True even+ # when the user left the flag at its default False. Reading the raw+ # flag here would let a PRIVATE_ONLY (offline) run silently download+ # an uncached model from HuggingFace. An unknown/corrupt scope+ # raises PolicyDeniedError out of context_from_snapshot — fail+ # closed, do not download.+ require_local = False+ if settings_snapshot is not None:+ try:+ from ....security.egress.policy import context_from_snapshot++ _primary = (+ get_setting_from_snapshot(+ "search.tool",+ default="auto",+ settings_snapshot=settings_snapshot,+ )+ or "auto"+ )+ require_local = context_from_snapshot(+ settings_snapshot, _primary+ ).require_local_embeddings+ except ValueError:+ # No usable snapshot for context construction; fall back to+ # the raw flag (still honours an explicit opt-in).+ require_local = bool(+ get_setting_from_snapshot(+ "embeddings.require_local",+ default=False,+ settings_snapshot=settings_snapshot,+ )+ )+ model_kwargs = {"device": device}+ if require_local:+ if not cls._is_model_cached_locally(model):+ from ....security.egress.policy import (+ Decision,+ PolicyDeniedError,+ )++ logger.bind(policy_audit=True).warning(+ "refusing SentenceTransformer download under "+ "embeddings.require_local=True",+ model=model,+ )+ raise PolicyDeniedError(+ Decision(False, "embeddings_model_not_cached"),+ target=model,+ )+ # Force the inner ``transformers``/``sentence_transformers``+ # call to use cached files only — defence in depth in case+ # the cache check above misses an alternative cache path.+ model_kwargs["local_files_only"] = True+return SentenceTransformerEmbeddings(model_name=model,- model_kwargs={"device": device},+ model_kwargs=model_kwargs,)+ @staticmethod+ def _is_model_cached_locally(model_name: str) -> bool:+ """Best-effort check whether ``model_name`` is already cached.++ Looks at the HuggingFace hub cache directory; returns False if+ the cache lookup itself fails, which fails closed under+ ``require_local=True``.+ """+ try:+ from huggingface_hub import try_to_load_from_cache++ # try_to_load_from_cache returns a path string when cached,+ # None when missing, and the sentinel _CACHED_NO_EXIST for+ # known-absent. Treat anything but a string path as a miss.+ cached = try_to_load_from_cache(+ repo_id=model_name, filename="config.json"+ )+ return isinstance(cached, str) and bool(cached)+ except Exception: # pragma: no cover - defensive+ return False+@classmethoddef is_available(cls, settings_snapshot: Optional[Dict[str, Any]] = Nonediff --git a/src/local_deep_research/news/api.py b/src/local_deep_research/news/api.pyindex 3927c6cc48..f55d5ed2f8 100644--- a/src/local_deep_research/news/api.py+++ b/src/local_deep_research/news/api.py@@ -824,6 +824,33 @@ def update_subscription("questions_per_iteration"]+ # N14: if this update touched the engine or provider, validate+ # the resulting (effective) values against the user's egress+ # policy before committing. Best-effort — needs a request+ # context to resolve the current user's settings DB.+ if "search_engine" in data or "model_provider" in data:+ from flask import (+ has_request_context,+ session as flask_session,+ )++ _uid = (+ flask_session.get("username")+ if has_request_context()+ else None+ )+ if _uid:+ policy_reason = _validate_subscription_policy(+ db_session,+ _uid,+ subscription.search_engine,+ subscription.model_provider,+ )+ if policy_reason is not None:+ raise SubscriptionUpdateException( # noqa: TRY301 — re-raised by except NewsAPIException+ subscription_id, policy_reason+ )+# Update timestampsubscription.updated_at = datetime.now(UTC)@@ -864,6 +891,79 @@ def update_subscription(raise SubscriptionUpdateException(subscription_id, str(e))+def _validate_subscription_policy(+ db_session, user_id, search_engine, model_provider+) -> Optional[str]:+ """Validate a subscription's search engine + LLM provider against the+ user's current egress policy (N14).++ News subscriptions store a fixed engine/provider that runs on a+ schedule. The factory PEP catches a forbidden engine at execution+ time, but validating at create/update time stops a forbidden config+ from being persisted in the first place (and gives the user+ immediate feedback). Returns a human-readable reason string when the+ subscription should be rejected, or None when it's allowed.++ Best-effort: if the settings backend or policy module is unavailable+ (e.g. programmatic API use without a settings DB), validation is+ skipped — the execution-time factory PEP remains the backstop.+ """+ try:+ from ..utilities.db_utils import get_settings_manager+ from ..security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_engine,+ evaluate_llm_endpoint,+ )++ settings_manager = get_settings_manager(db_session, user_id)+ snapshot = settings_manager.get_settings_snapshot()+ if not isinstance(snapshot, dict):+ return None+ primary = settings_manager.get_setting("search.tool", "auto")+ try:+ ctx = context_from_snapshot(+ snapshot, primary or "auto", username=user_id+ )+ except PolicyDeniedError as exc:+ return f"egress policy refused: {exc.decision.reason}"+ except ValueError as exc:+ # An incoherent egress config (e.g. STRICT scope + a meta-picker+ # primary engine) makes context_from_snapshot raise ValueError.+ # Surface it as a validation failure at subscription create/update+ # time instead of letting the outer except silently skip the+ # check — otherwise the subscription persists and only fails at+ # execution time.+ return f"egress policy is misconfigured: {exc}"++ if search_engine:+ decision = evaluate_engine(+ search_engine, ctx, settings_snapshot=snapshot+ )+ if not decision.allowed:+ return (+ f"search engine '{search_engine}' is not permitted "+ f"under the current egress policy ({decision.reason})"+ )+ if model_provider:+ decision = evaluate_llm_endpoint(+ normalize_provider(model_provider),+ ctx,+ settings_snapshot=snapshot,+ )+ if not decision.allowed:+ return (+ f"LLM provider '{model_provider}' is not permitted "+ f"under the current egress policy ({decision.reason})"+ )+ return None+ except Exception:+ # Settings/policy unavailable → skip; execution-time PEP backstops.+ logger.debug("subscription policy pre-check skipped", exc_info=True)+ return None++def create_subscription(user_id: str,query: str,@@ -906,6 +1006,17 @@ def create_subscription(# dependency on the settings database being initialized.with get_user_db_session(user_id) as db_session:+ # N14: reject a subscription whose engine/provider violates+ # the user's current egress policy before persisting it.+ policy_reason = _validate_subscription_policy(+ db_session, user_id, search_engine, model_provider+ )+ if policy_reason is not None:+ raise SubscriptionCreationException( # noqa: TRY301 — re-raised as-is by except SubscriptionCreationException+ policy_reason,+ {"query": query, "type": subscription_type},+ )+if refresh_minutes is None:try:from ..utilities.db_utils import get_settings_manager@@ -959,6 +1070,10 @@ def create_subscription("refresh_minutes": refresh_minutes,}+ except SubscriptionCreationException:+ # Already a structured creation error (e.g. the N14 policy+ # rejection) — propagate as-is instead of re-wrapping.+ raiseexcept Exception as e:logger.exception("Error creating subscription")raise SubscriptionCreationException(diff --git a/src/local_deep_research/news/core/news_analyzer.py b/src/local_deep_research/news/core/news_analyzer.pyindex 9850ecd3d1..61e352d56f 100644--- a/src/local_deep_research/news/core/news_analyzer.py+++ b/src/local_deep_research/news/core/news_analyzer.py@@ -28,15 +28,24 @@ class NewsAnalyzer:def __init__(self,llm_client: Optional[Any] = None,+ settings_snapshot: Optional[Dict[str, Any]] = None,):"""Initialize the news analyzer.Args:llm_client: LLM client for analysis+ settings_snapshot: Optional saved settings snapshot so the+ LLM is constructed under the user's egress policy. The+ news scheduler runs in background threads without a+ live request context, so without an explicit snapshot+ the PEP guard at the top of ``get_llm`` skips and a+ cloud LLM can fire under require_local_endpoint."""self._owns_llm = llm_client is None- self.llm_client = llm_client or get_llm()+ self.llm_client = llm_client or get_llm(+ settings_snapshot=settings_snapshot+ )def close(self) -> None:"""Close the LLM client if this instance created it."""diff --git a/src/local_deep_research/news/recommender/topic_based.py b/src/local_deep_research/news/recommender/topic_based.pyindex ddc80338aa..a9ad773f46 100644--- a/src/local_deep_research/news/recommender/topic_based.py+++ b/src/local_deep_research/news/recommender/topic_based.py@@ -10,6 +10,7 @@ from .base_recommender import BaseRecommenderfrom ..core.base_card import NewsCardfrom ..core.card_factory import CardFactoryfrom ...search_system import AdvancedSearchSystem+from ...security.egress.policy import PolicyDeniedErrorclass TopicBasedRecommender(BaseRecommender):@@ -237,6 +238,17 @@ class TopicBasedRecommender(BaseRecommender):"LLM not configured. Set llm.model in Settings.")return None+ except PolicyDeniedError as exc:+ # The egress policy refused the LLM (no snapshot in this+ # background path + non-local provider). Recurring+ # scheduler — log concisely, do not retry.+ logger.warning(+ f"Skipping news recommendation for topic '{topic}': "+ f"LLM blocked by egress policy ({exc.decision.reason}). "+ "Use a local provider (ollama/lmstudio/llamacpp) or "+ "thread settings_snapshot through this caller."+ )+ return Nonesearch = get_search(llm_instance=llm)search_system = AdvancedSearchSystem(llm=llm, search=search, strategy_name="news"diff --git a/src/local_deep_research/news/utils/headline_generator.py b/src/local_deep_research/news/utils/headline_generator.pyindex ebf144fef3..4c6cf8874f 100644--- a/src/local_deep_research/news/utils/headline_generator.py+++ b/src/local_deep_research/news/utils/headline_generator.py@@ -8,7 +8,10 @@ from loguru import loggerdef generate_headline(- query: str, findings: str = "", max_length: int = 100+ query: str,+ findings: str = "",+ max_length: int = 100,+ settings_snapshot: Optional[dict] = None,) -> str:"""Generate a concise headline from a query and optional findings.@@ -17,12 +20,19 @@ def generate_headline(query: The search query or research questionfindings: Optional findings/content to help generate better headlinemax_length: Maximum length for the headline+ settings_snapshot: Optional settings snapshot so the LLM call+ picks up the active egress policy. Background callers+ should pass this through; without it, the LLM PEP's+ ``settings_snapshot is not None`` guard skips and a cloud+ LLM can fire even under require_local_endpoint.Returns:A concise headline string"""# Always try LLM generation first for dynamic headlines based on actual content- llm_headline = _generate_with_llm(query, findings, max_length)+ llm_headline = _generate_with_llm(+ query, findings, max_length, settings_snapshot+ )if llm_headline:return llm_headline@@ -31,14 +41,17 @@ def generate_headline(def _generate_with_llm(- query: str, findings: str, max_length: int+ query: str,+ findings: str,+ max_length: int,+ settings_snapshot: Optional[dict] = None,) -> Optional[str]:"""Generate headline using LLM."""try:from ...config.llm_config import get_llm# Use the configured model for headline generation- llm = get_llm(temperature=0.3)+ llm = get_llm(temperature=0.3, settings_snapshot=settings_snapshot)try:# Focus only on the findings/report content, not the querydiff --git a/src/local_deep_research/news/utils/topic_generator.py b/src/local_deep_research/news/utils/topic_generator.pyindex cea5a9b494..401f7bbe2e 100644--- a/src/local_deep_research/news/utils/topic_generator.py+++ b/src/local_deep_research/news/utils/topic_generator.py@@ -10,7 +10,11 @@ from ...utilities.json_utils import extract_json, get_llm_response_textdef generate_topics(- query: str, findings: str = "", category: str = "", max_topics: int = 5+ query: str,+ findings: str = "",+ category: str = "",+ max_topics: int = 5,+ settings_snapshot=None,) -> List[str]:"""Generate relevant topics/tags from news content.@@ -20,12 +24,17 @@ def generate_topics(findings: The research findings/contentcategory: The news category (if available)max_topics: Maximum number of topics to generate+ settings_snapshot: User settings snapshot threaded to get_llm so+ the LLM PEP (evaluate_llm_endpoint) fires under+ llm.require_local_endpoint. Mirrors headline_generator.py.Returns:List of topic strings"""# Try LLM generation first- topics = _generate_with_llm(query, findings, category, max_topics)+ topics = _generate_with_llm(+ query, findings, category, max_topics, settings_snapshot+ )# No fallback - if LLM fails, mark as missingif not topics:@@ -36,7 +45,11 @@ def generate_topics(def _generate_with_llm(- query: str, findings: str, category: str, max_topics: int+ query: str,+ findings: str,+ category: str,+ max_topics: int,+ settings_snapshot=None,) -> List[str]:"""Generate topics using LLM."""try:@@ -47,7 +60,7 @@ def _generate_with_llm()# Use the configured model for topic generation- llm = get_llm(temperature=0.5)+ llm = get_llm(temperature=0.5, settings_snapshot=settings_snapshot)try:# Prepare contextdiff --git a/src/local_deep_research/notifications/manager.py b/src/local_deep_research/notifications/manager.pyindex aff5855435..5721be7da6 100644--- a/src/local_deep_research/notifications/manager.py+++ b/src/local_deep_research/notifications/manager.py@@ -251,10 +251,24 @@ class NotificationManager:)return False- # Send notification with service URLs+ # Egress policy: gate each parsed webhook URL against the+ # user's scope. Apprise URLs can be Slack/Discord/SMTP/HTTP —+ # we treat anything with a host as an HTTP egress and ask+ # evaluate_url. Schemes without a hostname (e.g. mailto)+ # pass through; SSRF + provider-side checks still apply.+ allowed_urls = self._filter_urls_by_egress_policy(service_urls)+ if not allowed_urls:+ logger.bind(policy_audit=True).warning(+ "all notification URLs refused by egress policy",+ user=self._user_id,+ event=event_type.value,+ )+ return False++ # Send notification with the allowed subset.logger.debug(f"Calling service.send_event for {event_type.value}")result = self.service.send_event(- event_type, context, service_urls=service_urls+ event_type, context, service_urls=allowed_urls)# Log to database if enabled@@ -289,7 +303,103 @@ class NotificationManager:Returns:Dict with test results"""- return self.service.test_service(url)+ # Egress policy precheck before forwarding to Apprise.+ # /api/notifications/test-url is a user-supplied URL endpoint.+ allowed = self._filter_urls_by_egress_policy(url)+ if not allowed:+ return {+ "status": "error",+ "message": (+ "URL refused by egress policy. Set Egress Scope to "+ "'Both' (or pick a local webhook) to test this URL."+ ),+ }+ return self.service.test_service(allowed)++ def _filter_urls_by_egress_policy(self, service_urls: str) -> str:+ """Filter an Apprise URL string (space- or comma-separated) by+ the user's egress policy. Returns the joined string of allowed+ URLs (may be empty). When no snapshot / context is available,+ returns the input unchanged (back-compat — the snapshot-less+ path was already used in older callers).+ """+ snapshot = getattr(self, "_settings_snapshot", None)+ if not snapshot:+ return service_urls+ try:+ from ..security.egress.policy import (+ EgressScope,+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_url,+ )+ except ImportError:+ logger.debug(+ "egress_policy unavailable in notifications manager; "+ "URLs will not be scope-gated"+ )+ return service_urls++ primary_raw = snapshot.get("search.tool", "auto")+ if isinstance(primary_raw, dict) and "value" in primary_raw:+ primary_raw = primary_raw["value"]+ try:+ ctx = context_from_snapshot(snapshot, primary_raw or "auto")+ except (PolicyDeniedError, ValueError) as exc:+ # Snapshot present but policy cannot be evaluated. The+ # previous bare-except dropped to "return service_urls"+ # here, which dispatched every URL unfiltered — fail-open+ # on a misconfigured policy. Now refuse all URLs instead+ # so an Apprise dispatch with no scope check is impossible.+ logger.bind(policy_audit=True).warning(+ "all notification URLs refused: egress policy could "+ "not be evaluated",+ reason=str(exc),+ )+ return ""++ # Apprise accepts space- or comma- separated URLs. Apprise has+ # its own non-http schemes (discord://, slack://, telegram://,+ # mailto://, msteams://, ntfy://, ...) that dispatch to EXTERNAL+ # vendor APIs. evaluate_url only understands http/https.+ #+ # Under PRIVATE_ONLY ("nothing leaves the box") we cannot verify a+ # vendor token resolves to a local endpoint, so a non-http scheme is+ # refused — fail closed. A self-hosted notifier (local ntfy, SMTP)+ # should be addressed by its http(s):// URL, which evaluate_url then+ # allows as a private host. Under the other scopes the prior+ # pass-through stands (the modeled threat there is internal-http+ # SSRF via a raw webhook, not vendor APIs).+ parts = []+ for url_entry in service_urls.replace(",", " ").split():+ url_entry = url_entry.strip()+ if not url_entry:+ continue+ scheme = (+ url_entry.split(":", 1)[0].lower() if ":" in url_entry else ""+ )+ if scheme not in ("http", "https"):+ if ctx.scope == EgressScope.PRIVATE_ONLY:+ logger.bind(policy_audit=True).warning(+ "notification refused under PRIVATE_ONLY egress "+ "scope: non-http vendor scheme cannot be verified "+ "local",+ scheme=scheme or "(none)",+ )+ continue+ parts.append(url_entry)+ continue+ decision = evaluate_url(url_entry, ctx)+ if decision.allowed:+ parts.append(url_entry)+ else:+ logger.bind(policy_audit=True).warning(+ "notification URL refused by egress policy",+ url=url_entry,+ scope=ctx.scope.value,+ reason=decision.reason,+ )+ return " ".join(parts)def _should_notify(self, event_type: EventType) -> bool:"""diff --git a/src/local_deep_research/research_library/routes/rag_routes.py b/src/local_deep_research/research_library/routes/rag_routes.pyindex f16ba98968..17b9658f34 100644--- a/src/local_deep_research/research_library/routes/rag_routes.py+++ b/src/local_deep_research/research_library/routes/rag_routes.py@@ -410,6 +410,20 @@ def test_embedding():)except Exception as e:+ # Egress policy denial → clean 4xx (the embedding never fired).+ from ...security.egress.policy import PolicyDeniedError++ if isinstance(e, PolicyDeniedError):+ return jsonify(+ {+ "success": False,+ "error": (+ f"Embedding provider '{provider}' refused by egress "+ f"policy ({e.decision.reason}). Disable 'Require "+ "local embeddings' or pick a local provider."+ ),+ }+ ), 400logger.exception("Error during testing embedding")error_str = str(e).lower()@@ -468,6 +482,54 @@ def get_available_models():# Get provider classesprovider_classes = _get_provider_classes()+ # Egress policy: fetching a cloud provider's model list opens a+ # connection that carries the API key off-machine. Under an effective+ # local-only embeddings posture (PRIVATE_ONLY / adaptive-private) we+ # must NOT probe remote-classified providers — mirror the PEP in+ # embeddings_config.get_embeddings(). Build the run context once and+ # fail closed (skip remote model fetches) if the policy can't evaluate.+ def _embeddings_model_fetch_allowed(provider_key: str) -> bool:+ try:+ from ...security.egress.policy import (+ context_from_snapshot,+ evaluate_embeddings,+ )+ from ...config.thread_settings import (+ get_setting_from_snapshot,+ )++ primary = (+ get_setting_from_snapshot(+ "search.tool",+ default="auto",+ settings_snapshot=settings_snapshot,+ )+ or "auto"+ )+ ctx = context_from_snapshot(settings_snapshot, primary)+ if not ctx.require_local_embeddings:+ return True+ decision = evaluate_embeddings(+ provider_key, ctx, settings_snapshot=settings_snapshot+ )+ if not decision.allowed:+ logger.bind(policy_audit=True).info(+ "skipping embeddings model-list probe "+ "(local-only egress posture)",+ provider=provider_key,+ reason=decision.reason,+ )+ return decision.allowed+ except Exception:+ # Fail closed: a policy/snapshot error must not open a cloud+ # probe under a local-only posture.+ logger.bind(policy_audit=True).warning(+ "embeddings model-list egress check failed; skipping probe",+ provider=provider_key,+ exc_info=True,+ )+ return False+# Provider display namesprovider_labels = {"sentence_transformers": "Sentence Transformers (Local)",@@ -492,8 +554,10 @@ def get_available_models():})- # Only fetch models when the provider is reachable.- if available:+ # Only fetch models when the provider is reachable AND egress+ # policy permits probing it (cloud providers are skipped under a+ # local-only posture).+ if available and _embeddings_model_fetch_allowed(provider_key):models = provider_class.get_available_models(settings_snapshot)providers[provider_key] = [{@@ -1173,6 +1237,7 @@ def get_collections():"is_default": coll.is_defaultif hasattr(coll, "is_default")else False,+ "is_public": bool(getattr(coll, "is_public", False)),"document_count": len(coll.document_links)if hasattr(coll, "document_links")else 0,@@ -1214,6 +1279,9 @@ def create_collection():name = data.get("name", "").strip()description = data.get("description", "").strip()collection_type = data.get("type", "user_uploads")+ # Egress classification, default private (the safe choice). A public+ # collection counts as a public engine and may use cloud inference.+ is_public = bool(data.get("is_public", False))if not name:return jsonify({"success": False, "error": "Name is required"}), 400@@ -1238,6 +1306,7 @@ def create_collection():name=name,description=description,collection_type=collection_type,+ is_public=is_public,)db_session.add(collection)@@ -1252,6 +1321,7 @@ def create_collection():"description": collection.description,"created_at": collection.created_at.isoformat(),"collection_type": collection.collection_type,+ "is_public": bool(collection.is_public),},})@@ -1307,6 +1377,10 @@ def update_collection(collection_id):if description is not None: # Allow empty descriptioncollection.description = description+ # Egress classification toggle (only when the caller sends it).+ if "is_public" in data:+ collection.is_public = bool(data.get("is_public"))+db_session.commit()return jsonify(@@ -1320,6 +1394,7 @@ def update_collection(collection_id):if collection.created_atelse None,"collection_type": collection.collection_type,+ "is_public": bool(collection.is_public),},})@@ -1832,6 +1907,9 @@ def get_collection_documents(collection_id):"id": collection.id,"name": collection.name,"description": collection.description,+ "is_public": bool(+ getattr(collection, "is_public", False)+ ),"embedding_model": collection.embedding_model,"embedding_model_type": collection.embedding_model_type.valueif collection.embedding_model_typediff --git a/src/local_deep_research/research_library/services/download_service.py b/src/local_deep_research/research_library/services/download_service.pyindex 176d37acde..efb3db00b3 100644--- a/src/local_deep_research/research_library/services/download_service.py+++ b/src/local_deep_research/research_library/services/download_service.py@@ -15,7 +15,7 @@ import timeimport uuidfrom datetime import datetime, UTCfrom pathlib import Path-from typing import Optional, Tuple+from typing import Any, Dict, Optional, Tuplefrom urllib.parse import urlparseimport requests@@ -68,17 +68,57 @@ from ..downloaders import (class DownloadService:"""Service for downloading and managing research PDFs."""- def __init__(self, username: str, password: Optional[str] = None):+ def __init__(+ self,+ username: str,+ password: Optional[str] = None,+ settings_snapshot: Optional[Dict[str, Any]] = None,+ ):"""Initialize download service for a user.Args:username: The username to download forpassword: Optional password for encrypted database access+ settings_snapshot: Optional snapshot used to build the+ EgressContext for per-URL scope gating. When None,+ downloads run without scope enforcement (back-compat+ for the existing caller; new callers must pass it)."""self.username = usernameself.password = passwordself.settings = get_settings_manager(username=username)self._closed = False+ # When no snapshot is passed (e.g. the library download routes, which+ # construct DownloadService(username, password) directly), build one+ # from this user's settings so egress policy is still enforced.+ # Without this, _egress_context stays None and _check_url_against_policy+ # returns (True, "no_context") — every download ungated. Best-effort:+ # on failure, fall through to the pre-policy behavior.+ # A real backend error (get_settings_snapshot raised) is different+ # from a test double returning a non-dict: under a private scope we+ # must NOT silently fall through to ungated downloads when we simply+ # failed to READ the user's settings. Track that case and fail closed.+ self._snapshot_build_failed = False+ if settings_snapshot is None:+ try:+ built = self.settings.get_settings_snapshot()+ # Only adopt a real dict — a test double or unavailable+ # backend hands back something else; treat that as "no+ # snapshot" (ungated back-compat) rather than letting the+ # policy evaluation fail closed on a non-dict.+ if isinstance(built, dict):+ settings_snapshot = built+ except Exception:+ # The user HAS a settings backend (self.settings exists) but+ # reading it raised — a private scope could be configured and+ # we just couldn't see it. Fail closed rather than ungate.+ logger.bind(policy_audit=True).warning(+ "DownloadService: settings snapshot build raised; "+ "locking downloads closed (fail-safe)"+ )+ self._snapshot_build_failed = True+ self._settings_snapshot = settings_snapshot+ self._egress_context = self._build_egress_context(settings_snapshot)# Debug settings manager and user contextlogger.info(@@ -206,6 +246,95 @@ class DownloadService:normalized = self._normalize_url(url)return get_url_hash(normalized)+ def _build_egress_context(self, settings_snapshot):+ """Build an EgressContext from the supplied snapshot. Returns+ None when no snapshot is available — callers fall through to+ the pre-policy behavior (back-compat with non-scheduler callers).++ Sets ``self._policy_locked = True`` when a snapshot WAS supplied+ but the policy itself cannot be evaluated (corrupt scope value,+ STRICT + meta-picker incoherence). The check_url method honors+ this flag and fails closed — the previous code returned None on+ PolicyDeniedError and check_url then returned ``(True,+ "no_context")``, which silently allowed every download under a+ misconfigured policy.+ """+ self._policy_locked = False+ # Use an explicit `is None` check, not a truthiness test: an empty+ # dict {} is a real (if minimal) snapshot, and `if not {}` would+ # silently skip policy gating for it — fail-open. Match the+ # `is None` guard used by the LLM/embeddings gates.+ if settings_snapshot is None:+ return None+ try:+ from ...security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ )+ except ImportError:+ logger.debug(+ "egress_policy unavailable in DownloadService; "+ "downloads will not be scope-gated"+ )+ return None++ primary_raw = settings_snapshot.get("search.tool", "auto")+ if isinstance(primary_raw, dict) and "value" in primary_raw:+ primary_raw = primary_raw["value"]+ try:+ return context_from_snapshot(+ settings_snapshot,+ primary_raw or "auto",+ username=self.username,+ )+ except (PolicyDeniedError, ValueError) as exc:+ logger.bind(policy_audit=True).warning(+ "DownloadService policy unavailable; locking out "+ "every URL check",+ reason=str(exc),+ )+ self._policy_locked = True+ return None++ def _check_url_against_policy(self, url: str) -> Tuple[bool, str]:+ """Run evaluate_url against self._egress_context. Returns+ (allowed, reason). When no context is configured (legacy+ callers OR tests that mock __init__), returns (True, "no_context")+ so back-compat callers aren't broken — but new callers passing+ a snapshot DO get gated. Logged to policy_audit on denial.+ """+ # ``_policy_locked`` is set by _build_egress_context when a+ # snapshot was supplied but the policy itself raised. Without+ # this short-circuit, _egress_context is None and the legacy+ # fall-through below would silently allow every URL.+ if getattr(self, "_policy_locked", False):+ return False, "policy_unavailable"++ # Reading the user's settings raised during __init__ — we could not+ # determine the scope, so fail closed instead of ungating downloads.+ if getattr(self, "_snapshot_build_failed", False):+ return False, "settings_unavailable"++ # Use getattr so tests that mock DownloadService.__init__ (and+ # therefore never set _egress_context) still work — they+ # operated under the pre-policy contract.+ egress_context = getattr(self, "_egress_context", None)+ if egress_context is None:+ return True, "no_context"+ from ...security.egress.policy import evaluate_url+ from ...security.ssrf_validator import redact_url_for_log++ decision = evaluate_url(url, egress_context)+ if not decision.allowed:+ logger.bind(policy_audit=True).warning(+ "DownloadService URL denied by egress policy",+ # Redact userinfo creds / query-string tokens from the log.+ url=redact_url_for_log(url),+ scope=egress_context.scope.value,+ reason=decision.reason,+ )+ return decision.allowed, decision.reason+def is_already_downloaded(self, url: str) -> Tuple[bool, Optional[str]]:"""Check if URL is already downloaded.@@ -260,6 +389,16 @@ class DownloadService:url = resource.url+ # Egress policy gate — denied URLs short-circuit before any+ # downloader fires a request.+ allowed, reason = self._check_url_against_policy(url)+ if not allowed:+ logger.warning(+ f"Skipping text extraction for {url}: refused by "+ f"egress policy ({reason})"+ )+ return None+# Find appropriate downloaderfor downloader in self.downloaders:if downloader.can_handle(url):@@ -316,6 +455,17 @@ class DownloadService:if resource.document_id:continue+ # Egress policy gate before queueing.+ allowed, reason = self._check_url_against_policy(+ resource.url+ )+ if not allowed:+ logger.warning(+ f"Skipping queue for {resource.url}: refused by "+ f"egress policy ({reason})"+ )+ continue+# Check if already queuedexisting_queue = (session.query(LibraryDownloadQueue)@@ -525,6 +675,17 @@ class DownloadService:"""url = resource.url+ # Egress policy gate at the network-fire point. The entry caller+ # (download_resource) does not gate, so this is the true PEP — a+ # denied URL must never reach a downloader regardless of route.+ allowed, reason = self._check_url_against_policy(url)+ if not allowed:+ logger.warning(+ f"Skipping PDF download for {url}: refused by egress "+ f"policy ({reason})"+ )+ return False, f"egress_policy_denied:{reason}", None+# Log attemptattempt = DownloadAttempt(url_hash=tracker.url_hash,@@ -1092,6 +1253,17 @@ class DownloadService:if not downloader:return None+ # Egress policy gate. Return a non-None tuple (not None) on denial+ # so download_as_text hard-stops here instead of falling through+ # to _fallback_pdf_extraction and firing a second request.+ allowed, reason = self._check_url_against_policy(resource.url)+ if not allowed:+ logger.warning(+ f"Skipping API text extraction for {resource.url}: refused "+ f"by egress policy ({reason})"+ )+ return False, f"egress_policy_denied:{reason}"+result = downloader.download_with_result(resource.url, ContentType.TEXT)if not result.is_success or not result.content:@@ -1152,6 +1324,20 @@ class DownloadService:session.commit()return False, error_msg+ # Egress policy gate at the network-fire point.+ allowed, reason = self._check_url_against_policy(resource.url)+ if not allowed:+ error_msg = f"egress_policy_denied:{reason}"+ logger.warning(+ f"Skipping fallback PDF download for {resource.url}: "+ f"refused by egress policy ({reason})"+ )+ self._record_failed_text_extraction(+ session, resource, error=error_msg+ )+ session.commit()+ return False, error_msg+result = downloader.download_with_result(resource.url, ContentType.PDF)if not result.is_success or not result.content:diff --git a/src/local_deep_research/research_library/services/library_rag_service.py b/src/local_deep_research/research_library/services/library_rag_service.pyindex e0dddb516c..e108d5d765 100644--- a/src/local_deep_research/research_library/services/library_rag_service.py+++ b/src/local_deep_research/research_library/services/library_rag_service.py@@ -197,6 +197,54 @@ class LibraryRAGService:})+ # Egress policy pre-flight at the constructor boundary so+ # every direct ``LibraryRAGService(...)`` construction site+ # is covered, not just the factory. Skipped when an+ # ``embedding_manager`` is injected (tests / advanced flows)+ # — those callers vouch for the manager themselves.+ from ...security.egress.policy import (+ Decision,+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_embeddings,+ _get_setting_value,+ )++ # Build the context from the ACTUAL scope (not a hardcoded+ # BOTH) so PRIVATE_ONLY forces local embeddings even when the+ # raw embeddings.require_local flag is at its default False —+ # context_from_snapshot applies that coupling. The previous+ # code both hardcoded scope=BOTH and gated on the raw flag, so+ # a PRIVATE_ONLY corpus could be embedded by a cloud provider.+ primary = (+ _get_setting_value(settings_snapshot, "search.tool", "auto")+ or "auto"+ )+ try:+ policy_ctx = context_from_snapshot(+ settings_snapshot, primary, username=username+ )+ except PolicyDeniedError:+ raise+ except ValueError as exc:+ raise PolicyDeniedError(+ Decision(False, "invalid_policy_config"),+ target=embedding_provider,+ ) from exc+ if policy_ctx.require_local_embeddings:+ decision = evaluate_embeddings(+ embedding_provider,+ policy_ctx,+ settings_snapshot=settings_snapshot,+ )+ if not decision.allowed:+ logger.bind(policy_audit=True).warning(+ "LibraryRAGService refused by egress policy",+ provider=embedding_provider,+ reason=decision.reason,+ )+ raise PolicyDeniedError(decision, target=embedding_provider)+self.embedding_manager = LocalEmbeddingManager(embedding_model=embedding_model,embedding_model_type=embedding_provider,diff --git a/src/local_deep_research/research_library/services/rag_service_factory.py b/src/local_deep_research/research_library/services/rag_service_factory.pyindex b98e61e5b9..357ad30923 100644--- a/src/local_deep_research/research_library/services/rag_service_factory.py+++ b/src/local_deep_research/research_library/services/rag_service_factory.py@@ -18,6 +18,79 @@ from ...utilities.type_utils import to_boolfrom ..services.library_rag_service import LibraryRAGService+def _enforce_embeddings_policy(+ embedding_provider: str, settings_manager, username: str+) -> None:+ """Pre-flight egress-policy check before constructing the RAG service.++ Fails BEFORE the first chunk is processed (per the plan's pre-flight+ requirement) — important because indexing a large corpus can take 10++ minutes; we want the user to see a clear policy error immediately, not+ after embeddings have already been generated for hundreds of chunks.++ No-op when the egress policy does not require local embeddings —+ i.e. when ``embeddings.require_local`` is False AND the scope does not+ imply it. Under PRIVATE_ONLY the requirement is forced regardless of+ the flag (see context_from_snapshot).+ """+ # Lazy import to avoid pulling the security module on every factory call.+ from ...security.egress.policy import (+ Decision,+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_embeddings,+ )++ # We don't have a full settings snapshot here, only a SettingsManager.+ # Build a minimal snapshot for the policy module — only the keys it+ # reads for scope coupling + embedding classification matter.+ scope = settings_manager.get_setting("policy.egress_scope") or "both"+ require_local_flag = to_bool(+ settings_manager.get_setting("embeddings.require_local") or False+ )+ base_url = settings_manager.get_setting("embeddings.openai.base_url")+ ollama_url = settings_manager.get_setting("embeddings.ollama.url")+ # evaluate_embeddings() classifies the ollama embeddings endpoint from+ # embeddings.ollama.url OR, when that's unset, llm.ollama.url. Omitting+ # the llm.* fallback here would misclassify a user who only configured+ # the shared llm.ollama.url as "remote" and wrongly deny local ollama+ # embeddings. Populate both so the classification matches runtime.+ llm_ollama_url = settings_manager.get_setting("llm.ollama.url")+ snapshot = {+ "policy.egress_scope": scope,+ "embeddings.require_local": require_local_flag,+ "embeddings.openai.base_url": base_url or "",+ "embeddings.ollama.url": ollama_url or "",+ "llm.ollama.url": llm_ollama_url or "",+ }+ # Build the ctx from the ACTUAL scope so PRIVATE_ONLY forces local+ # embeddings even when the raw flag is False. primary_engine="library"+ # is concrete (not a meta-picker) so STRICT won't raise here.+ try:+ ctx = context_from_snapshot(snapshot, "library", username=username)+ except PolicyDeniedError:+ raise+ except ValueError as exc:+ raise PolicyDeniedError(+ Decision(False, "invalid_policy_config"),+ target=embedding_provider,+ ) from exc+ # No-op unless the (scope-aware) policy requires local embeddings.+ if not ctx.require_local_embeddings:+ return++ decision = evaluate_embeddings(+ embedding_provider, ctx, settings_snapshot=snapshot+ )+ if not decision.allowed:+ logger.bind(policy_audit=True).warning(+ "embeddings provider denied by egress policy",+ provider=embedding_provider,+ reason=decision.reason,+ )+ raise PolicyDeniedError(decision, target=embedding_provider)++def get_rag_service(username: str,collection_id: Optional[str] = None,@@ -124,6 +197,19 @@ def get_rag_service(f"Using stored settings for collection {collection_id}: "f"{collection.embedding_model_type.value if collection.embedding_model_type else 'unknown'}/{collection.embedding_model}")+ # Egress policy pre-flight (R9-07 / plan landmine #3):+ # block before any chunk is generated when the stored+ # provider conflicts with require_local. Critical for+ # collections indexed pre-policy-rollout with OpenAI.+ effective_provider = (+ collection.embedding_model_type.value+ if collection.embedding_model_type+ else default_embedding_provider+ )+ _enforce_embeddings_policy(+ effective_provider, settings, username+ )+# Handle normalize_vectors - may be stored as string in some# casescoll_normalize = collection.normalize_vectors@@ -165,6 +251,11 @@ def get_rag_service(f"New collection {collection_id}, using and storing default settings")+ # Egress policy pre-flight.+ _enforce_embeddings_policy(+ default_embedding_provider, settings, username+ )+# Create service with defaultsreturn LibraryRAGService(username=username,@@ -186,6 +277,10 @@ def get_rag_service(# index_collection when first document is indexed.# No collection or fallback - use current defaults+ # Egress policy pre-flight.+ _enforce_embeddings_policy(+ default_embedding_provider, settings, username+ )return LibraryRAGService(username=username,embedding_model=default_embedding_model,diff --git a/src/local_deep_research/scheduler/background.py b/src/local_deep_research/scheduler/background.pyindex b7fa502301..6c2143d635 100644--- a/src/local_deep_research/scheduler/background.py+++ b/src/local_deep_research/scheduler/background.py@@ -682,6 +682,30 @@ class BackgroundJobScheduler:f"Error scheduling document processing for {username}")+ def _arm_egress_backstop(self, settings_manager, username: str) -> None:+ """Set the audit-hook egress context from the user's saved settings so+ scheduled document downloads run under the same secondary net as an+ interactive research run. Best-effort and never raises — a backstop+ failure must not break the scheduler; the DownloadService PEP remains+ the primary gate. Cleared by the caller's @thread_cleanup on exit.+ """+ try:+ from ..security.egress.audit_hook import set_active_context+ from ..security.egress.policy import context_from_snapshot++ snapshot = settings_manager.get_settings_snapshot()+ if not isinstance(snapshot, dict):+ return+ primary = settings_manager.get_setting("search.tool", "auto")+ ctx = context_from_snapshot(+ snapshot, primary or "auto", username=username+ )+ set_active_context(ctx)+ except Exception:+ logger.bind(policy_audit=True).debug(+ "doc scheduler: egress backstop not armed", exc_info=True+ )+@thread_cleanupdef _process_user_documents(self, username: str):"""Process documents for a user."""@@ -743,6 +767,15 @@ class BackgroundJobScheduler:with get_user_db_session(username, password) as db:settings_manager = SettingsManager(db)+ # Arm the PEP-578 audit-hook backstop for this scheduled run.+ # The APScheduler worker thread carries no egress context, so+ # the secondary net would be inactive while DownloadService+ # fetches documents below. DownloadService's evaluate_url PEP+ # still gates each fetch (primary); this restores defense-in-+ # depth parity with an interactive run. @thread_cleanup clears+ # the context when this method returns.+ self._arm_egress_backstop(settings_manager, username)+# Query for completed research since last runlogger.debug(f"[DOC_SCHEDULER] Querying for completed research since {last_run}"@@ -821,6 +854,24 @@ class BackgroundJobScheduler:f"[DOC_SCHEDULER] - completed_at > last_run: {completed_at_obj > last_run if last_run and completed_at_obj else 'N/A'}")+ # Capture a settings snapshot for this user/run so the+ # DownloadService below can build an EgressContext and+ # gate each per-resource URL. Without this the scheduler+ # would bypass policy entirely. Reuses the outer `db`+ # session (line 743) — get_settings_manager() in a+ # background thread must be passed a db_session+ # explicitly per the pre-commit thread-safety check.+ try:+ user_settings_snapshot = (+ settings_manager.get_settings_snapshot()+ )+ except Exception:+ logger.exception(+ "[DOC_SCHEDULER] Could not build settings snapshot — "+ "downloads will not be scope-gated"+ )+ user_settings_snapshot = None+processed_count = 0for research in research_sessions:try:@@ -855,7 +906,9 @@ class BackgroundJobScheduler:)with DownloadService(- username=username, password=password+ username,+ password,+ settings_snapshot=user_settings_snapshot,) as download_service:queued_count = download_service.queue_research_downloads(research.id@@ -892,7 +945,9 @@ class BackgroundJobScheduler:)with DownloadService(- username=username, password=password+ username,+ password,+ settings_snapshot=user_settings_snapshot,) as download_service:# Get all resources for this research (reuse existing db session)all_resources = (@@ -1584,6 +1639,7 @@ class BackgroundJobScheduler:query=query_text,findings=report_content,max_length=200, # Allow longer headlines for news+ settings_snapshot=settings_snapshot,)# Generate topics from the findings@@ -1595,6 +1651,7 @@ class BackgroundJobScheduler:findings=report_content,category=subscription.get("name", "News"),max_topics=6,+ settings_snapshot=settings_snapshot,)logger.info(diff --git a/src/local_deep_research/search_system.py b/src/local_deep_research/search_system.pyindex f14fbc1225..8d5b62302c 100644--- a/src/local_deep_research/search_system.py+++ b/src/local_deep_research/search_system.py@@ -293,10 +293,72 @@ class AdvancedSearchSystem:search_id = str(uuid.uuid4())- # Perform the search- return self._perform_search(- query, search_id, is_user_search, is_news_search, user_id- )+ # Defense-in-depth: arm the PEP-578 egress audit-hook backstop for this+ # run if no caller already armed it. The web worker arms it in+ # research_service before calling us; CLI, the news scheduler and the+ # programmatic API construct AdvancedSearchSystem directly and would+ # otherwise run the full pipeline with the secondary net inactive.+ _armed_egress = self._arm_egress_backstop()+ try:+ # Perform the search+ return self._perform_search(+ query, search_id, is_user_search, is_news_search, user_id+ )+ finally:+ if _armed_egress:+ # Only clear what WE armed, so a reused thread doesn't leak the+ # context to unrelated subsequent work (the web worker owns its+ # own teardown via @thread_cleanup).+ from .security.egress.audit_hook import clear_active_context++ clear_active_context()++ def _arm_egress_backstop(self) -> bool:+ """Arm the egress audit-hook context for this run when no caller has.++ Returns True only when THIS call armed the context (so ``analyze_topic``+ clears it on exit). Returns False when it was already armed (web worker)+ or could not be built. Never raises — a backstop failure must not break+ a research run; the explicit PEPs remain the primary enforcement.+ """+ try:+ from .security.egress.audit_hook import (+ get_active_context,+ set_active_context,+ )+ from .security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ )+ except Exception:+ return False++ if get_active_context() is not None:+ return False # already armed (e.g. by the web research worker)+ if not self.settings_snapshot:+ return False++ try:+ primary = self.settings_snapshot.get("search.tool", "auto")+ if isinstance(primary, dict) and "value" in primary:+ primary = primary["value"]+ ctx = context_from_snapshot(+ self.settings_snapshot,+ primary or "auto",+ username=self.username,+ )+ except (PolicyDeniedError, ValueError):+ # Corrupted scope / STRICT+meta-picker: the explicit PEPs and the+ # run-start precheck handle these; don't arm a partial backstop.+ logger.bind(policy_audit=True).debug(+ "egress backstop not armed: policy unevaluable", exc_info=True+ )+ return False+ except Exception:+ return False++ set_active_context(ctx)+ return Truedef _perform_search(self,diff --git a/src/local_deep_research/security/__init__.py b/src/local_deep_research/security/__init__.pyindex 8c528a99e2..d0440e8e64 100644--- a/src/local_deep_research/security/__init__.py+++ b/src/local_deep_research/security/__init__.py@@ -7,6 +7,14 @@ from .data_sanitizer import (sanitize_data,strip_settings_snapshot,)+from .egress.audit_hook import (+ active_egress_context,+ clear_active_context,+ get_active_context,+ install_audit_hook,+ is_installed as is_audit_hook_installed,+ set_active_context,+)from .security_settings import get_security_defaultfrom .file_integrity import FileIntegrityManager, FAISSIndexVerifierfrom .notification_validator import (@@ -52,6 +60,12 @@ except ImportError:__all__ = ["PathValidator","DataSanitizer",+ "active_egress_context",+ "clear_active_context",+ "get_active_context",+ "install_audit_hook",+ "is_audit_hook_installed",+ "set_active_context","sanitize_data","redact_data","filter_research_metadata",@@ -82,3 +96,10 @@ __all__ = ["sanitize_filename","UnsafeFilenameError",]++# Install the process-wide socket.connect audit hook after all imports+# resolve. The hook is a no-op until a worker thread calls+# ``set_active_context(ctx)`` — importing this module has zero+# behavioral effect on code that does not opt in. Idempotent: subsequent+# imports do not re-install.+install_audit_hook()diff --git a/src/local_deep_research/security/egress/README.md b/src/local_deep_research/security/egress/README.mdnew file mode 100644index 0000000000..5a80d6da52--- /dev/null+++ b/src/local_deep_research/security/egress/README.md@@ -0,0 +1,274 @@+# Egress Policy++A self-contained guardrail that constrains **where a research run's traffic+may go** — which search engines run, which LLM/embeddings endpoints are+reached, and which URLs may be fetched.++> ⚠️ **Experimental (first release).** Surfaced in the UI as+> *Egress Scope (Experimental)*. It blocks the known egress paths under the+> selected scope and has been through two adversarial review rounds, but it is+> early — treat it as defense-in-depth, not an absolute guarantee, and don't+> make it the sole protection for highly sensitive data.++> **This is an in-process correctness guardrail, NOT a hard security+> boundary.** It defends against honest misconfiguration, prompt-injection-+> induced fetches, accidental egress, and the LangGraph silent-expansion+> class of bug. It does **not** defend against a determined adversary with+> code execution inside the LDR process. For a hard boundary, layer OS-level+> controls (network namespaces, firewall egress rules, restricted+> containers). See the root `SECURITY.md` for the full threat statement.++---++## 1. Mental model: egress scopes++Every run has an **egress scope** (`policy.egress_scope`). It decides which+engines/destinations are permitted.++| Scope | Meaning |+|---|---|+| `adaptive` *(default)* | **Follow the primary engine.** A private primary → behaves PRIVATE_ONLY; a public primary → PUBLIC_ONLY; a meta-picker (`auto`/`parallel`/`meta`) or unclassifiable primary → BOTH. Resolved to a concrete scope at run start. |+| `both` | Any classified engine (the pre-policy behaviour). |+| `public_only` | Only public web/academic engines. Local collections excluded. |+| `private_only` | Only local engines (library, collections, local SearXNG/Ollama). **Forces local LLM + embeddings inference** so nothing leaves the box. |+| `strict` | Only the user's *primary* engine; no expansion at all. Incompatible with a meta-picker primary. |++Two independent toggles refine inference locality (and are **implied** under+PRIVATE_ONLY / Adaptive-private):++- `llm.require_local_endpoint` — refuse cloud LLM providers / non-local URLs.+- `embeddings.require_local` — refuse cloud embeddings (e.g. OpenAI), and+ refuse a HuggingFace download for an uncached sentence-transformers model.++**Per-collection classification.** Each RAG collection carries an `is_public`+flag (default **private**). A private collection is excluded under+PUBLIC_ONLY / Adaptive-public and, when it is the primary under Adaptive,+pulls the whole run to PRIVATE_ONLY (→ local inference). To process a+collection with a cloud model, mark it **public** — the explicit opt-in.++---++## 2. Architecture: PDP + PEPs++Vocabulary borrowed from XACML / zero-trust:++- **PDP — Policy Decision Point.** The pure decision functions. *They live in+ this package* (`policy.py`).+- **PEP — Policy Enforcement Point.** The call sites that consult the PDP and+ act on the decision. *They are distributed across the codebase* at the+ places where engines/LLMs/URLs are actually used — a PEP can't be moved+ here because it lives in its domain (search factory, LLM config, fetcher,+ …). This package is the brain; the PEPs are the hands.++### Files in this package++| File | Role |+|---|---|+| `policy.py` | The PDP. `EgressScope`, `EgressContext`, `Decision`, `PolicyDeniedError`, `context_from_snapshot()` (builds the frozen per-run context + resolves Adaptive), and the `evaluate_*()` decision functions. Also host classification (DNS + private-IP, NAT64 metadata, percent-decode), the run-scoped DNS cache, and the denied-fetch quota. |+| `audit_hook.py` | A process-wide **PEP-578 `sys.audit` hook** on `socket.connect`. The *secondary* net: catches outbound connections from code paths the explicit PEPs don't cover (third-party libs, new contributors using raw `requests`). Inactive until a worker calls `set_active_context(ctx)`; only gates PRIVATE_ONLY / STRICT. |+| `fetch.py` | `policy_aware_validate_url()` — the egress-scope-aware SSRF wrapper (lets PRIVATE_ONLY reach private lab hosts without disabling SSRF globally; metadata IPs stay blocked). Depends on `../ssrf_validator.py`, not the other way around. |+| `warnings.py` | The three UI banner checks (`check_public_egress_enabled`, `check_cloud_llm_enabled`, `check_cloud_embeddings_enabled`), each with its own dismiss key. Pure functions; the `web/warning_checks` orchestrator calls them. |+| `validators.py` | Settings-save cross-field validators (`validate_strict_meta_combo`, `validate_allowed_local_hostnames`). Pure functions; the settings write routes call them. |+| `__init__.py` | Re-exports the public PDP / audit-hook API. |++### Public API++```python+from local_deep_research.security.egress import (+ EgressScope, EgressContext, Decision, PolicyDeniedError,+ context_from_snapshot, # build the per-run context+ evaluate_engine, # may engine X run under this scope?+ evaluate_retriever, # … for a registered retriever+ evaluate_url, # may this URL be fetched?+ evaluate_llm_endpoint, # is this LLM endpoint allowed?+ evaluate_embeddings, # is this embeddings provider allowed?+ set_active_context, # arm the audit-hook net for this thread+ clear_active_context,+ active_egress_context, # context manager form+ install_audit_hook,+)+```++The decision functions return a `Decision(allowed: bool, reason: str)` where+`reason` is a short machine code (`"scope_mismatch_public_only"`,+`"blocked_metadata_ip"`, …) — never user content. PEPs raise+`PolicyDeniedError(decision)` on a hard denial.++---++## 3. Where the PEPs live (enforcement points)++The PDP is consulted at these call sites. This table is the map for "what+actually enforces the policy":++| Domain | PEP location | Consults |+|---|---|---|+| Search engine creation | `web_search_engines/search_engine_factory.py` (`create_search_engine`) | `evaluate_engine` / `evaluate_retriever` |+| LangGraph agent tools | `advanced_search_system/strategies/langgraph_agent_strategy.py` (`_build_tools`) | `evaluate_engine` / `evaluate_retriever` (tool-list filter) |+| MCP agent tools | `advanced_search_system/strategies/mcp_strategy.py` (`_build_tool_descriptions`) | `evaluate_engine` / `evaluate_retriever` (tool-list filter) |+| Meta / parallel search | `web_search_engines/engines/{meta,parallel}_search_engine.py` | `evaluate_engine` at child instantiation |+| Full-content fetch | `web_search_engines/engines/full_search.py` | `evaluate_url` |+| Content fetcher tool | `content_fetcher/fetcher.py` | `policy_aware_validate_url` + `evaluate_url` |+| Library downloads | `research_library/services/download_service.py` | `evaluate_url` at the fetch fire-points |+| LLM construction | `config/llm_config.py` (`get_llm`) | `evaluate_llm_endpoint` (+ snapshot-less allow-list) |+| Embeddings | `embeddings/embeddings_config.py`, `…/implementations/sentence_transformers.py`, `web_search_engines/engines/local_embedding_manager.py` | `evaluate_embeddings` / scope-coupled require-local |+| Journal reputation fetch | `advanced_search_system/filters/journal_reputation_filter.py` | scope skip |+| Run-start precheck | `web/routes/research_routes.py` (`_precheck_engine_policy`) | `evaluate_engine` (clean 400 at the API boundary) |+| Settings validation | `web/routes/settings_routes.py` (calls `egress/validators.py`) | `validate_strict_meta_combo`, `validate_allowed_local_hostnames` |+| Secondary net (all sockets) | `audit_hook.py` (installed at `security/__init__`) | `evaluate_url` on every `socket.connect` |++Adjacent (general security utils, not egress-specific — used here but live one level up):++- `security/ssrf_validator.py` — SSRF IP-class validation (`validate_url`, `is_ip_blocked`, metadata blocklist). The egress-scope-aware wrapper that *uses* it now lives in `fetch.py` here.+- `security/network_utils.py` — `is_private_ip`.++The UI orchestrator that calls `warnings.py` lives at `web/warning_checks/__init__.py` (it also handles non-egress warnings like backups/context, so it stays in the web layer).++---++## 4. Settings keys++| Key | Default | Effect |+|---|---|---|+| `policy.egress_scope` | `adaptive` | The scope (table above). |+| `llm.require_local_endpoint` | `false` | Refuse non-local LLM endpoints. |+| `embeddings.require_local` | `false` | Refuse cloud embeddings / uncached HF download. |+| `llm.allowed_local_hostnames` | `[]` | Hostnames treated as local by the classifier (public hosts rejected at save time). |+| `collections.is_public` (per collection) | `false` (private) | Collection egress classification. |+| `app.warnings.dismiss_egress_policy` / `…dismiss_cloud_llm` / `…dismiss_cloud_embeddings` | `false` | Per-banner dismiss flags (separate so dismissing one never hides the critical others). |++---++## 5. Threat model & caveats++**Covered:** forgotten/unreviewed code paths (via the audit hook), cloud LLM/+embeddings under a "local-only" claim, prompt-injection URL fetches, cloud-+metadata fetches under any scope, NAT64-wrapped metadata, percent-encoded+host bypass, DNS-timeout hangs, cache-hit policy bypass, and the LangGraph/MCP+silent-expansion timing leak.++**Caveats (important):**++- **Guardrail, not a boundary.** Code running inside the process can clear the+ context, monkey-patch the hook, or open a socket out-of-band.+- **DNS-rebinding TOCTOU.** The host is classified at evaluation time; the HTTP+ client resolves again at connect time. (Accepted risk; see `SECURITY.md`.)+- **LLM/embeddings endpoint locality is best-effort.** *Strong* for named+ providers (OpenAI/Anthropic/… blocked; Ollama/LM Studio/llama.cpp allowed);+ *weaker* for configurable-URL providers (`openai_endpoint`) classified by+ DNS — a private-looking URL that tunnels to the cloud is trusted as local.+ The policy stops *accidental/silent* egress; a user who deliberately points+ "local" inference at a cloud endpoint, or who marks a collection **public**,+ is making an informed choice.+- **Settings tampering.** An attacker who can write a user's per-user+ (SQLCipher) settings DB can flip the scope to `both` and disable enforcement.+ Policy-key changes emit `policy_audit=True` audit log lines.++---++## 6. Extending it — adding a PEP++1. Build the per-run context once: `ctx = context_from_snapshot(snapshot, primary, username=...)`.+ (At run start this is also armed for the audit hook via `set_active_context(ctx)`.)+2. At the point where you reach a destination, call the matching+ `evaluate_*` function.+3. On `decision.allowed is False`, raise `PolicyDeniedError(decision)` (hard+ stop) **or** log-and-skip if multi-source resilience matters — and emit a+ `logger.bind(policy_audit=True)` line. **Re-raise `PolicyDeniedError`+ before any broad `except Exception`** so a denial is never downgraded to a+ generic error (fail-open of the audit trail).+4. Prefer `is None` over truthiness when guarding on the snapshot — an empty+ dict is a real snapshot.++---++## 7. Tests++- `tests/security/test_egress_policy.py` — the PDP (scopes, adaptive+ resolution, classification, quota, DNS timeout).+- `tests/security/test_egress_audit_hook.py` — the PEP-578 hook (context+ lifecycle, re-entrancy, scope gating).+- `tests/security/test_egress_pep_coverage.py` — the distributed PEPs.+- Plus per-domain tests (factory, langgraph/mcp strategies, embeddings,+ download service, settings validators, warning banners, collections API).++## Data-flow reference (private vs public)++### Private-vs-Public Egress Data Flow (per scope)++```mermaid+flowchart TD+ Run([Research run starts]) --> Snap["context_from_snapshot()\nreads policy.egress_scope"]+ Snap --> Scope{egress_scope?}++ Scope -->|adaptive| Adapt{"_resolve_adaptive_scope()\nclassify PRIMARY engine"}+ Adapt -->|private primary| PRIV+ Adapt -->|public primary| PUB+ Adapt -->|meta-picker / unknown| BOTH++ Scope -->|private_only| PRIV["PRIVATE_ONLY\n(forces require_local_llm\n+ require_local_embeddings)"]+ Scope -->|public_only| PUB[PUBLIC_ONLY]+ Scope -->|both| BOTH[BOTH]+ Scope -->|strict| STRICT["STRICT\n(primary engine only)"]++ %% PRIVATE_ONLY: nothing leaves the box+ PRIV --> P_eng["evaluate_engine:\nlocal engines only\n(public BLOCKED)"]+ PRIV --> P_llm["evaluate_llm_endpoint:\ncloud providers BLOCKED\nlocal forced"]+ PRIV --> P_emb["evaluate_embeddings:\ncloud BLOCKED\nlocal forced"]+ PRIV --> P_url["evaluate_url:\nprivate host ALLOWED\npublic host BLOCKED"]+ PRIV --> P_hook["audit hook ARMED:\nany public socket BLOCKED"]+ P_eng --> NoEgress([No external data egress])+ P_llm --> NoEgress+ P_emb --> NoEgress+ P_url --> NoEgress+ P_hook --> NoEgress++ %% PUBLIC_ONLY: open to public sources only+ PUB --> U_eng["evaluate_engine:\npublic engines only\n(private collections BLOCKED)"]+ PUB --> U_url["evaluate_url:\npublic host ALLOWED\nprivate host BLOCKED"]+ PUB --> U_inf["LLM/embeddings:\nnot scope-forced\n(toggles still apply)"]+ PUB --> U_hook["audit hook INACTIVE\n(passthrough)"]+ U_eng --> PubEgress([Public-source egress permitted])+ U_url --> PubEgress+ U_inf --> PubEgress++ %% BOTH: any classified destination+ BOTH --> B_any["evaluate_*:\nany classified engine/host ALLOWED\nunclassified fails closed"]+ B_any --> AllEgress([Public + private egress])++ %% STRICT: primary engine only+ STRICT --> S_eng["evaluate_engine:\nONLY primary engine\nno expansion"]+ STRICT --> S_url["evaluate_url:\nprivate host ALLOWED\npublic host BLOCKED"]+ STRICT --> S_hook["audit hook ARMED:\nany public socket BLOCKED"]++ %% Metadata always blocked everywhere+ Meta["Cloud-metadata IPs\n(169.254.169.254, NAT64 wraps)"] -.->|BLOCKED under ALL scopes| Scope+```++### Egress vector x scope decision matrix++Each cell is the decision produced by the real `evaluate_*` logic / PEP for that vector. ADAPTIVE has no cell of its own: at run start it is resolved (`_resolve_adaptive_scope`) to PRIVATE_ONLY (private primary), PUBLIC_ONLY (public primary), or BOTH (meta-picker/unclassifiable), so it inherits that column. STRICT additionally requires the engine to BE the primary.++| Egress vector | adaptive (resolves to →) | both | public_only | private_only | strict |+|---|---|---|---|---|---|+| **Search engine selection** (`evaluate_engine`) | PRIVATE_ONLY \| PUBLIC_ONLY \| BOTH | any classified engine ALLOWED; unclassified BLOCKED | public engines ALLOWED; private collections BLOCKED | local engines ALLOWED; public BLOCKED | only PRIMARY engine ALLOWED; all others BLOCKED |+| **Child-engine expansion** (meta/parallel child instantiation → `evaluate_engine`) | follows resolved scope | each child ALLOWED if classified | public children ALLOWED; private BLOCKED | local children ALLOWED; public BLOCKED | BLOCKED — meta-picker primary is rejected (`strict_with_meta_picker`); no expansion |+| **LLM inference** (`evaluate_llm_endpoint` + `get_llm`) | private→forced-local; public→cloud ALLOWED (unless toggle) | ALLOWED unless `llm.require_local_endpoint` set | ALLOWED unless toggle set (not scope-forced) | **forced-local** — cloud providers BLOCKED (`require_local_llm` implied) | ALLOWED unless toggle set (STRICT is orthogonal to inference) |+| **Embeddings / index** (`evaluate_embeddings`) | private→forced-local; public→cloud ALLOWED (unless toggle) | ALLOWED unless `embeddings.require_local` set | ALLOWED unless toggle set | **forced-local** — cloud embedders BLOCKED (`require_local_embeddings` implied) | ALLOWED unless toggle set |+| **Arbitrary URL fetch** (`evaluate_url`, content fetcher / full_search) | private→private-host only; public→public-host only | any classified host ALLOWED; unclassified BLOCKED | public host ALLOWED; private host BLOCKED | private host ALLOWED; **public host BLOCKED** | private host ALLOWED; public host BLOCKED |+| **Library download** (`evaluate_url`, download_service) | private→private-host only; public→public-host only | any classified host ALLOWED | public host ALLOWED; private BLOCKED | private host ALLOWED; **public BLOCKED** | private host ALLOWED; public BLOCKED |+| **Notification webhook** (`evaluate_url`, notifications/manager) | private→private-host only; public→public-host only | any classified http/https host ALLOWED | public webhook ALLOWED; private BLOCKED | private webhook ALLOWED; **public webhook BLOCKED** | private webhook ALLOWED; public BLOCKED |+| **Raw socket** (audit hook → `evaluate_url`) | private→ARMED (public BLOCKED); public→INACTIVE | INACTIVE (passthrough) | INACTIVE (passthrough) | **ARMED** — every public socket.connect BLOCKED | ARMED — every public socket.connect BLOCKED |+| **Cloud-metadata IP** (169.254.169.254 / NAT64) | BLOCKED | BLOCKED | BLOCKED | BLOCKED | BLOCKED |++### Key invariants (the crystal-clear takeaways)++| Claim | Where it is enforced |+|---|---|+| **private_only blocks ALL external data egress** | engine selection (`evaluate_engine` → `scope_mismatch_private_only`), URL/library/webhook fetch (`evaluate_url` PRIVATE_ONLY branch), LLM (`require_local_llm` forced in `context_from_snapshot` → `provider_cloud_only`), embeddings (`require_local_embeddings` forced → `provider_cloud`), and the raw-socket audit-hook net armed for PRIVATE_ONLY (`audit_hook.py`). No vector permits a public destination. |+| **public_only stays open to public sources** | public engines pass `evaluate_engine` (private collections BLOCKED via `scope_mismatch_public_only`); `evaluate_url` allows public hosts, blocks private; inference is NOT scope-forced (toggles still apply); the audit hook stays INACTIVE (only PRIVATE_ONLY/STRICT arm it) so legitimate local infra traffic — local Ollama, settings DB — is never falsely blocked. |+| **adaptive just matches the primary engine** | `_resolve_adaptive_scope`: private primary → PRIVATE_ONLY, public primary → PUBLIC_ONLY, meta-picker/unknown → BOTH. A registered local retriever primary also resolves PRIVATE_ONLY. Resolved once at run start; the stored `EgressContext` carries the concrete scope, not ADAPTIVE. |+| **A private RAG collection as ADAPTIVE primary pulls the whole run local** | `_resolve_collection_is_public` defaults private → `_engine_bucket` returns local → adaptive resolves PRIVATE_ONLY → forces local LLM + embeddings. Mark the collection `is_public` to opt into cloud inference. |+| **Cloud-metadata is never reachable, under any scope** | `evaluate_url` runs `is_ip_blocked(..., allow_private_ips=True)` before scope logic; `_classify_host` applies the same metadata block on BOTH the literal-IP and DNS-resolved paths; NAT64-wrapped metadata is reclassified PUBLIC. |++> **Notification footnote.** The notification PEP (`notifications/manager.py` `_filter_urls_by_egress_policy`) gates **http/https** webhook URLs through `evaluate_url`. Apprise vendor schemes (`slack://`, `discord://`, `telegram://`, `mailto://`, …) dispatch to external vendor APIs: under **PRIVATE_ONLY** (incl. Adaptive-private) they are **refused outright** (we can't verify a vendor token is local — fail closed; address a self-hosted notifier by its `http(s)://` URL, which `evaluate_url` allows as a private host). Under the other scopes they pass the URL gate (the modeled threat there is internal-http SSRF, not vendor APIs); the audit-hook net is an additional backstop for the actual connection when armed.diff --git a/src/local_deep_research/security/egress/__init__.py b/src/local_deep_research/security/egress/__init__.pynew file mode 100644index 0000000000..f272f1f343--- /dev/null+++ b/src/local_deep_research/security/egress/__init__.py@@ -0,0 +1,57 @@+"""Egress policy subsystem.++A self-contained guardrail that constrains where a research run's traffic,+LLM calls, embeddings, and URL fetches may go. See ``README.md`` in this+package for the full design, threat model, and the list of enforcement+points (PEPs) scattered across the codebase.++This ``__init__`` re-exports the public API so callers can simply do+``from local_deep_research.security.egress import evaluate_url, EgressScope``.+The implementation lives in:+ - ``policy.py`` — the PDP (decisions) + context construction+ - ``audit_hook.py`` — the process-wide PEP-578 socket.connect net+"""++from .policy import (+ Decision,+ EgressContext,+ EgressScope,+ MAX_DENIED_FETCHES_PER_RUN,+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_embeddings,+ evaluate_engine,+ evaluate_llm_endpoint,+ evaluate_retriever,+ evaluate_url,+)+from .audit_hook import (+ active_egress_context,+ clear_active_context,+ get_active_context,+ install_audit_hook,+ is_installed,+ set_active_context,+)++__all__ = [+ # policy / PDP+ "Decision",+ "EgressContext",+ "EgressScope",+ "MAX_DENIED_FETCHES_PER_RUN",+ "PolicyDeniedError",+ "context_from_snapshot",+ "evaluate_embeddings",+ "evaluate_engine",+ "evaluate_llm_endpoint",+ "evaluate_retriever",+ "evaluate_url",+ # audit hook / process-wide PEP+ "active_egress_context",+ "clear_active_context",+ "get_active_context",+ "install_audit_hook",+ "is_installed",+ "set_active_context",+]diff --git a/src/local_deep_research/security/egress/audit_hook.py b/src/local_deep_research/security/egress/audit_hook.pynew file mode 100644index 0000000000..04f3eafbc7--- /dev/null+++ b/src/local_deep_research/security/egress/audit_hook.py@@ -0,0 +1,292 @@+"""Process-wide ``socket.connect`` audit hook for defense-in-depth+egress control.++Why this exists+---------------+Every explicit PEP we wrote gates a single known caller — ContentFetcher,+evaluate_llm_endpoint, MCP download_content, and so on. They are the+primary line of defense and they remain the primary line of defense.+This module is the secondary line — the "what about the code paths we+haven't reviewed yet?" net. It fires on EVERY outbound socket.connect+inside the process, so a langchain tool registered by an MCP server, a+new contributor reaching for ``requests.get`` directly, or a third-party+library that opens its own connection all get caught by the same rule+the user configured.++Threat model+------------+- Catches: forgotten PEP coverage, new code paths that don't know about+ the policy, third-party library egress, prompt-injection that steers a+ tool into raw HTTP, ``urllib.request.urlopen``, ``httpx``, ``aiohttp``,+ ``socket.socket().connect()`` — anything that ultimately calls+ ``socket.connect`` (almost everything network-bound in CPython).+- Does NOT catch: a determined adversary with code execution in the LDR+ process. They can ``clear_active_context()`` from inside the process,+ monkey-patch the hook, ``addaudithook`` a passthrough, or open a file+ descriptor out-of-band. For a real boundary, layer OS-level controls+ (network namespaces, firewall rules, restricted Docker) — see+ SECURITY.md.++Activation model+----------------+- ``install_audit_hook()`` is INSTALLED ONCE at app/library startup+ (idempotent). PEP 578 audit hooks cannot be removed — that is the+ design, not a limitation. Once it is in, it stays.+- The hook is INACTIVE by default: when no EgressContext is registered+ for the current thread, every connect passes through unmodified. This+ is the right default — random scripts importing the package and+ pytest collectors that touch a socket would otherwise raise+ ``PolicyDeniedError`` mid-test.+- The hook becomes ACTIVE for a thread when a worker calls+ ``set_active_context(ctx)``. Clear with ``clear_active_context()`` in+ a finally to prevent leak across thread-pool reuse. The+ ``database.thread_local_session._ThreadCleanup`` exit handler already+ does this for the standard research worker lifecycle.+"""++from __future__ import annotations++import socket+import sys+import threading+from contextlib import contextmanager+from typing import Optional++from loguru import logger+++# ---------------------------------------------------------------------------+# Thread-local state+# ---------------------------------------------------------------------------++# Per-thread "currently active" EgressContext. The audit hook reads this; a+# worker (research run, news scheduler, library indexer, etc.) sets it at+# task entry and clears it in finally.+_thread_local = threading.local()++# Re-entrancy guard. The hook itself may indirectly trigger a socket call+# (e.g., if a future logger ever ships a record through a network sink)+# and without this we would recurse on every connect.+_hook_reentry = threading.local()+++def set_active_context(ctx) -> None:+ """Register ``ctx`` as the active EgressContext for the current+ thread. Subsequent ``socket.connect`` calls from this thread are+ gated through the policy until ``clear_active_context()`` is called.++ Idempotent: setting None is equivalent to clearing.+ """+ if ctx is None:+ clear_active_context()+ return+ # An ADAPTIVE scope must be resolved to a concrete scope BEFORE it is+ # stored: the audit hook only gates PRIVATE_ONLY/STRICT, so an ADAPTIVE+ # context reaching the hook would make the backstop a silent no-op.+ # Production always builds contexts via context_from_snapshot (which+ # resolves ADAPTIVE); this fail-fast guard stops a future direct+ # construction or test from disarming the net unnoticed.+ try:+ from .policy import EgressScope++ if getattr(ctx, "scope", None) == EgressScope.ADAPTIVE:+ raise ValueError(+ "EgressContext.scope must be resolved to a concrete scope "+ "before activation; got ADAPTIVE. Build the context via "+ "context_from_snapshot(), which resolves ADAPTIVE."+ )+ except ImportError: # pragma: no cover - policy import always available+ pass+ _thread_local.egress_context = ctx+++def clear_active_context() -> None:+ """Clear the active EgressContext for the current thread. Safe to+ call when no context is set.+ """+ # ``hasattr`` + ``del`` rather than ``setattr(None)`` so the+ # ``getattr(..., None)`` branch in the hook short-circuits without+ # touching the policy module at all.+ if hasattr(_thread_local, "egress_context"):+ del _thread_local.egress_context+++def get_active_context():+ """Return the active EgressContext for this thread, or ``None``."""+ return getattr(_thread_local, "egress_context", None)+++@contextmanager+def active_egress_context(ctx):+ """Context manager: set the active EgressContext for the duration of+ the ``with`` block and clear it on exit (even on exception).++ Useful for unit tests, ad-hoc scripts, and any worker that wants+ automatic cleanup without writing a try/finally.+ """+ # Save/restore the previous context rather than unconditionally clearing,+ # so nesting an inner ``active_egress_context`` doesn't wipe an outer+ # one's context on exit (the single thread-local slot otherwise loses the+ # parent — a nested chat-in-research or test would leave the thread with+ # NO active context, silently disarming the backstop for the parent's+ # remaining work).+ previous = get_active_context()+ set_active_context(ctx)+ try:+ yield+ finally:+ if previous is not None:+ set_active_context(previous)+ else:+ clear_active_context()+++# ---------------------------------------------------------------------------+# The audit hook+# ---------------------------------------------------------------------------++_INSTALLED = False+_INSTALL_LOCK = threading.Lock()+++def install_audit_hook() -> None:+ """Install the ``sys.audit`` hook. Idempotent — calling multiple+ times is a no-op after the first. Per PEP 578 audit hooks cannot be+ uninstalled, so this is install-once-and-keep-forever; the activation+ gate is the per-thread context, not the hook itself.+ """+ global _INSTALLED+ with _INSTALL_LOCK:+ if _INSTALLED:+ return+ sys.addaudithook(_audit_hook)+ _INSTALLED = True+++def is_installed() -> bool:+ return _INSTALLED+++def _extract_host(address) -> Optional[str]:+ """Extract the destination host from a ``socket.connect`` address+ tuple. Returns ``None`` for shapes we don't recognise (caller passes+ through — failing to extract is never a reason to block).++ AF_INET addresses are ``(host, port)``. AF_INET6 addresses are+ ``(host, port, flowinfo, scope_id)``. ``host`` is a string in both.+ """+ if not isinstance(address, tuple) or not address:+ return None+ host = address[0]+ # CPython accepts a bytes/bytearray host in ``socket.connect`` and fires+ # the ``socket.connect`` audit event with the raw bytes (verified). Decode+ # so a bytes-encoded IP/host literal is classified instead of silently+ # passing through the hook — otherwise a library that connects with a+ # bytes host bypasses the PRIVATE_ONLY/STRICT backstop entirely.+ if isinstance(host, (bytes, bytearray)):+ try:+ host = bytes(host).decode("ascii")+ except (UnicodeDecodeError, ValueError):+ return None+ if not isinstance(host, str):+ return None+ return host+++def _audit_hook(event: str, args: tuple) -> None:+ """sys.audit hook. Only watches ``socket.connect``; ignores every+ other event. Raises ``PolicyDeniedError`` when the active context+ refuses the destination.++ Failure mode: this hook deliberately does NOT swallow exceptions+ raised by the policy lookup. If the policy module itself errors,+ the connect fails loudly so the operator notices — the+ catch-and-continue alternative would degrade the secondary line of+ defense into "silently disabled when there's a bug." That is the+ opposite of what a defense-in-depth net is for.+ """+ if event != "socket.connect":+ return++ # No active policy → fast path. This is the common case for any+ # thread that is not running a research worker (e.g., the Flask+ # request handler, sqlalchemy connection pools, the news scheduler+ # in between jobs). Skipping here keeps the hook overhead at+ # roughly two attribute lookups per connect.+ ctx = getattr(_thread_local, "egress_context", None)+ if ctx is None:+ return++ # Re-entrancy guard. Anything that runs inside the hook (logger+ # binding, exception construction, attribute access) might+ # indirectly trigger another connect (e.g. via a future remote log+ # sink). Without this, the hook would recurse and stack-overflow.+ if getattr(_hook_reentry, "active", False):+ return++ # Arm the re-entrancy guard BEFORE touching ``args``/``sock``/``address``+ # or importing the policy module: any of those attribute accesses could+ # (under instrumentation or a future remote log sink) trigger another+ # socket.connect, and the guard must already be set to break the+ # recursion. The try/finally guarantees the guard is cleared on EVERY+ # exit path — the early returns below and the PolicyDeniedError raise+ # included — so a blocked connect doesn't wedge the hook off for the+ # rest of the thread's life.+ _hook_reentry.active = True+ try:+ sock, address = args++ # Only gate network sockets. AF_UNIX, AF_NETLINK, AF_PACKET, etc.+ # are off-network and should not pay the policy cost.+ family = getattr(sock, "family", None)+ if family not in (socket.AF_INET, socket.AF_INET6):+ return++ host = _extract_host(address)+ if not host:+ return++ # Scope-restriction: this hook is a "no-public-egress" net, NOT a+ # general re-implementation of every PEP at the socket layer.+ # PRIVATE_ONLY and STRICT are the two scopes that explicitly say+ # "no public host should be reached." PUBLIC_ONLY governs search-+ # ENGINE selection (the user wants public sources only) and is NOT+ # supposed to block legitimate local infrastructure traffic — local+ # Ollama, local embeddings, the user's own settings DB. Same logic+ # for BOTH. Applying evaluate_url under those scopes would refuse+ # the local LLM connect with ``scope_mismatch_public_only``, which+ # is a false positive against the user's actual intent.+ from .policy import EgressScope++ if ctx.scope not in (EgressScope.PRIVATE_ONLY, EgressScope.STRICT):+ return++ # Imported lazily to avoid a hard dependency at hook-install+ # time. The hook is installed early during app startup and the+ # egress_policy module pulls in network_utils, ip_ranges, etc.+ # — none of which are problematic, but the lazy import keeps+ # the install path minimal.+ from .policy import (+ PolicyDeniedError,+ evaluate_url,+ )++ # Synthesize a URL so evaluate_url's existing host classification+ # logic does the work (private-IP check, NAT64, local_hostnames+ # match, DNS resolution if the host is not already an IP). The+ # scheme is fixed at http:// because evaluate_url only inspects+ # the host part — the value here is irrelevant. IPv6 literals+ # MUST be bracketed (``[::1]``) or urlsplit's hostname parser+ # returns None and the policy then refuses on ``no_hostname``.+ url_host = f"[{host}]" if family == socket.AF_INET6 else host+ decision = evaluate_url(f"http://{url_host}", ctx)+ if not decision.allowed:+ logger.bind(policy_audit=True).warning(+ "audit hook blocked socket.connect",+ host=host,+ reason=decision.reason,+ scope=ctx.scope.value,+ )+ raise PolicyDeniedError(decision, target=host)+ finally:+ _hook_reentry.active = Falsediff --git a/src/local_deep_research/security/egress/fetch.py b/src/local_deep_research/security/egress/fetch.pynew file mode 100644index 0000000000..c969864e27--- /dev/null+++ b/src/local_deep_research/security/egress/fetch.py@@ -0,0 +1,42 @@+"""Egress-aware URL fetch validation.++A thin wrapper over the general SSRF validator that threads the egress+*scope* through, so this layer (egress) depends on the SSRF validator rather+than the SSRF validator depending on egress.+"""++from loguru import logger++from ..ssrf_validator import validate_url+from .policy import EgressScope+++def policy_aware_validate_url(url: str, egress_context=None) -> bool:+ """Validate ``url`` for SSRF, taking the egress policy scope into account.++ Under ``EgressScope.PRIVATE_ONLY`` the policy lets the user reach+ private hosts (local lab deployments — Ollama on 127.0.0.1, SearXNG+ on 192.168.x). Plain ``validate_url(url)`` would reject those at+ the SSRF layer even though policy explicitly permits them. This+ wrapper threads the scope through so a PRIVATE_ONLY run can reach+ private hosts WITHOUT requiring the operator to set+ ``SSRF_ALLOW_PRIVATE_IPS=1`` globally.++ Cloud-metadata IPs in ``ALWAYS_BLOCKED_METADATA_IPS`` remain blocked+ regardless of scope (handled inside ``is_ip_blocked``).++ When ``egress_context`` is ``None`` the call is equivalent to+ ``validate_url(url)`` (strict defaults).+ """+ if egress_context is None:+ return validate_url(url)+ try:+ if egress_context.scope == EgressScope.PRIVATE_ONLY:+ return validate_url(url, allow_private_ips=True)+ except Exception: # noqa: silent-exception+ # Defensive: any attribute error here means the context is+ # malformed; treat as if no policy was supplied and fall back to+ # strict defaults. Logging is intentionally suppressed to avoid+ # leaking policy state via timing.+ logger.debug("policy_aware_validate_url defensive fallback")+ return validate_url(url)diff --git a/src/local_deep_research/security/egress/policy.py b/src/local_deep_research/security/egress/policy.pynew file mode 100644index 0000000000..f8048d6e39--- /dev/null+++ b/src/local_deep_research/security/egress/policy.py@@ -0,0 +1,1244 @@+"""Egress policy module: central PDP for search engines, LLM endpoints,+embeddings, and URL fetches.++This is an in-process correctness guardrail, NOT a hard security boundary.+It defends against honest misconfiguration, prompt-injection-induced+URL fetches, accidental egress, and the LangGraph silent-expansion bug.+It does NOT defend against compromised dependencies, code-execution+in the LDR process, or a determined adversary who can modify the policy+module itself. Operators needing a hard boundary should layer OS-level+controls (network namespaces, firewall rules, restricted Docker).++Vocabulary borrowed from XACML / zero-trust:+ PDP — Policy Decision Point (the evaluate_* functions in this module)+ PEP — Policy Enforcement Point (the call sites that consult the PDP)+"""++from __future__ import annotations++import socket+import threading+from concurrent.futures import ThreadPoolExecutor, TimeoutError as FutureTimeout+from dataclasses import dataclass, field+from enum import Enum+from typing import Optional+from urllib.parse import unquote, urlsplit++from loguru import logger++from ..network_utils import is_private_ip+++# Bounded DNS lookup timeout; getaddrinfo has no native timeout kwarg, so+# we run it inside a single-shot ThreadPoolExecutor and abandon the+# Future after the timeout elapses. This avoids the previous+# socket.setdefaulttimeout() approach, which mutated a process-global+# setting and could corrupt unrelated network code under concurrency.+_DNS_TIMEOUT_SEC = 2.0+++# Hard limit: after this many denied fetches in a single run, fail closed+# even on otherwise-legal fetches. Prevents an exhaust attack via a malicious+# indexed document looping the agent through hundreds of denied URLs.+MAX_DENIED_FETCHES_PER_RUN = 50++# Only SECURITY-relevant denials count toward the per-run quota. Benign parse+# failures (a mailto:/ftp:/tel: link or a malformed href scraped from a page)+# are common in legitimate documents and must NOT exhaust the budget — doing+# so would make a long PUBLIC_ONLY run start refusing legitimate public URLs+# mid-run. These reasons are still audit-logged; they just don't tick the quota.+# ``dangerous_scheme`` (javascript:/data:/file:/… hrefs) belongs here too: these+# are non-fetchable, non-network schemes scraped from ordinary HTML (onclick+# handlers, inline data: URIs), so — like ``unsupported_scheme`` — they can't+# cause egress and shouldn't let a doc full of data: URIs exhaust the budget.+_NON_QUOTA_DENIAL_REASONS = frozenset(+ {"url_malformed", "unsupported_scheme", "no_hostname", "dangerous_scheme"}+)+++class EgressScope(str, Enum):+ """User-declared egress boundary for a research run.++ - STRICT: only the user's primary engine; no expansion at all.+ - PUBLIC_ONLY: any public (external web/academic) engine.+ - PRIVATE_ONLY: any private (local collection / library) engine.+ - BOTH: any classified engine (preserves pre-policy behavior).+ - ADAPTIVE: scope FOLLOWS the primary engine — a concrete private+ primary behaves as PRIVATE_ONLY, a concrete public primary as+ PUBLIC_ONLY, and a meta-picker / unclassifiable primary as BOTH.+ The default: most users never touch scope and "it just matches my+ main engine." Resolved to a concrete scope at context construction;+ the stored EgressContext carries the RESOLVED scope, not ADAPTIVE.+ """++ STRICT = "strict"+ PUBLIC_ONLY = "public_only"+ PRIVATE_ONLY = "private_only"+ BOTH = "both"+ ADAPTIVE = "adaptive"+++@dataclass(frozen=True)+class EgressContext:+ """Frozen per-run policy snapshot.++ Constructed once via ``context_from_snapshot()`` at run-start.+ The dataclass is frozen, but mutable internals (``_dns_cache``,+ ``_fetch_denial_count``) use ``field(init=False, default_factory=...)``+ so they can accumulate state during a run. Counters live inside a+ dict because direct ``int`` field reassignment fails on ``frozen=True``.+ """++ scope: EgressScope+ primary_engine: str+ require_local_llm: bool+ require_local_embeddings: bool+ local_hostnames: tuple[str, ...] = ()+ username: Optional[str] = None+ # Mutable internal state (kept off the public surface).+ _dns_cache: dict = field(init=False, default_factory=dict, repr=False)+ _fetch_denial_count: dict = field(+ init=False, default_factory=lambda: {"count": 0}, repr=False+ )+ # Guards _dns_cache + _fetch_denial_count mutations. Scope is the+ # cache writes only — DNS I/O must NOT happen while holding the lock,+ # or concurrent subagent threads serialize on each other's lookups.+ _lock: threading.RLock = field(+ init=False, default_factory=threading.RLock, repr=False+ )+++@dataclass(frozen=True)+class Decision:+ """Result of a PDP evaluation. ``reason`` is a short machine code+ (e.g. ``"unclassified"``, ``"scope_mismatch"``) — never the user's+ rejected query or URL content.+ """++ allowed: bool+ reason: str+++class PolicyDeniedError(RuntimeError):+ """Raised by PEPs when a policy decision is hard-stop denial.++ Raising (rather than returning a graceful empty result) ensures+ consistent denial latency, which mitigates the LangGraph timing-leak+ pattern where an LLM could infer policy state from how fast a denied+ tool call returns.+ """++ def __init__(self, decision: Decision, target: str = ""):+ self.decision = decision+ self.target = target+ super().__init__(f"policy_denied: {decision.reason}")+++_META_PICKER_ENGINES = frozenset(+ {"auto", "meta", "parallel", "parallel_scientific"}+)+++def _is_nat64_wrapped_metadata(hostname: str) -> bool:+ """True iff ``hostname`` parses as an IPv6 NAT64 address wrapping a+ cloud-metadata IPv4 (AWS / Azure / GCE / etc).++ The wrapping passes ``is_private_ip`` (because IPv6 link-local+ ranges match) but the embedded IPv4 actually reaches the metadata+ endpoint. We classify these as PUBLIC so STRICT and PRIVATE_ONLY+ refuse to fetch them.+ """+ try:+ import ipaddress++ from ..ssrf_validator import is_nat64_wrapped_metadata_ip++ candidate = hostname+ if candidate.startswith("[") and candidate.endswith("]"):+ candidate = candidate[1:-1]+ ip = ipaddress.ip_address(candidate)+ return is_nat64_wrapped_metadata_ip(ip)+ except (ValueError, TypeError):+ return False+ except Exception: # pragma: no cover - defensive+ return False+++def _resolve_with_timeout(hostname: str) -> Optional[list]:+ """Resolve ``hostname`` via getaddrinfo with a bounded timeout.++ Returns the addrinfo list on success or None on timeout / lookup+ failure. getaddrinfo has no native timeout, so we drive it from a+ single-shot worker thread and abandon the Future on timeout — far+ safer than socket.setdefaulttimeout(), which mutates a+ process-global setting and races with unrelated network code.+ """++ def _do_lookup():+ return socket.getaddrinfo(hostname, None, type=socket.SOCK_STREAM)++ # NB: do NOT use ``with ThreadPoolExecutor(...)`` here. The context+ # manager's __exit__ calls shutdown(wait=True), which blocks until the+ # worker thread finishes — so a hung getaddrinfo would defeat the whole+ # point of the timeout (the call would return only after the OS DNS+ # timeout, not after _DNS_TIMEOUT_SEC). Instead we abandon the worker on+ # timeout via shutdown(wait=False): the caller returns promptly and the+ # orphaned thread dies on its own when getaddrinfo eventually returns.+ executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="ldr-dns")+ future = executor.submit(_do_lookup)+ try:+ return future.result(timeout=_DNS_TIMEOUT_SEC)+ except (FutureTimeout, socket.gaierror, socket.timeout, OSError):+ return None+ except Exception: # pragma: no cover - defensive+ return None+ finally:+ # wait=False so a timed-out lookup does not re-block here; a+ # completed lookup leaves an idle thread that shuts down at once.+ executor.shutdown(wait=False)+++_UNCACHED = object()+++def _cache_classification(+ ctx: EgressContext, hostname: str, value: bool+) -> bool:+ """First-writer-wins cache set; returns the value now in the cache.++ Concurrent subagent threads may classify the same hostname outside the+ lock (the DNS lookup is intentionally unsynchronized). If a round-robin+ DNS name resolves to a private IP for one thread and a public IP for+ another, last-writer-wins would let a later call flip an earlier call's+ result — cache *incoherence* that could relax PRIVATE_ONLY/STRICT on a+ subsequent fetch. Pinning the first writer's value makes the per-run+ classification stable and deterministic regardless of completion order.+ """+ with ctx._lock:+ existing = ctx._dns_cache.get(hostname, _UNCACHED)+ if existing is not _UNCACHED:+ return existing+ ctx._dns_cache[hostname] = value+ return value+++def _classify_host(+ hostname: str, ctx: EgressContext, allow_dns: bool = True+) -> Optional[bool]:+ """Classify a hostname as local (True) / public (False) / unknown (None).++ Uses the run-scoped DNS cache to avoid repeated ``getaddrinfo`` lookups.+ Falls back to public on DNS timeout (fail-safe).++ Lock discipline: cache reads and writes are guarded by ``ctx._lock``;+ the DNS lookup itself runs OUTSIDE the lock so concurrent subagent+ threads don't serialize on each other's DNS calls. Cache writes are+ first-writer-wins (``_cache_classification``) so a hostname's+ classification is stable for the run even under concurrent disagreeing+ lookups.+ """+ if not hostname:+ return None++ # Cache hit — read under the lock so concurrent writers don't tear+ # the dict.+ with ctx._lock:+ if hostname in ctx._dns_cache:+ return ctx._dns_cache[hostname]++ # User-declared local hostnames override DNS classification.+ if hostname in ctx.local_hostnames:+ return _cache_classification(ctx, hostname, True)++ # NAT64-wrapped metadata IPs (e.g. 64:ff9b::169.254.169.254) classify+ # as link-local by is_private_ip but actually wrap AWS/GCE instance+ # metadata. Force them to PUBLIC so STRICT/PRIVATE_ONLY don't allow+ # them. Check happens BEFORE is_private_ip so the wrapping wins.+ if _is_nat64_wrapped_metadata(hostname):+ return _cache_classification(ctx, hostname, False)++ # String-literal check first (no DNS needed for IPs and known literals).+ # If the helper itself raises (malformed hostname, broken stdlib edge case),+ # fall through to DNS resolution rather than fail the whole evaluation —+ # the DNS path has its own bounded timeout and error handling.+ from ..ssrf_validator import is_ip_blocked++ try:+ # A literal cloud-metadata IP (169.254.169.254, fd00:ec2::254, …) passes+ # is_private_ip (link-local) but must NOT be treated as local — otherwise+ # STRICT/PRIVATE_ONLY would accept an IMDS SSRF target as a "local"+ # inference/search host. Mirror the DNS-branch metadata block below.+ # is_ip_blocked returns False for non-IP hostnames (ValueError → False),+ # so legitimate hostnames still fall through to is_private_ip / DNS.+ if is_ip_blocked(+ hostname, allow_localhost=True, allow_private_ips=True+ ):+ return _cache_classification(ctx, hostname, False)+ if is_private_ip(hostname):+ return _cache_classification(ctx, hostname, True)+ except Exception as exc:+ logger.debug(+ "is_private_ip raised on hostname, falling back to DNS",+ error=str(exc),+ )++ # DNS-resolved classification — runs OUTSIDE the lock.+ if not allow_dns:+ # The caller (the advisory warning-banner render path) opted OUT of+ # the synchronous getaddrinfo so a settings-page render never blocks+ # up to _DNS_TIMEOUT_SEC on a lookup. Return "unknown" so adaptive+ # scope resolution falls back to the engine's static classification.+ # Deliberately NOT cached — a later enforcement-path call (allow_dns+ # default True) must still resolve this host for real.+ return None+ addr_info = _resolve_with_timeout(hostname)+ if addr_info is None:+ # Fail-safe: unresolvable / timeout → treat as public.+ return _cache_classification(ctx, hostname, False)++ # If any resolved IP is private (and not a NAT64 metadata wrap),+ # classify as local.+ for entry in addr_info:+ ip_str = entry[4][0]+ try:+ # A hostname that resolves to a cloud-metadata IP must NOT be+ # treated as local: link-local metadata IPs (169.254.169.254, …)+ # pass is_private_ip, which would classify the host local and let+ # STRICT/PRIVATE_ONLY fetch it. Classify as public so those scopes+ # refuse it — mirrors the literal-IP metadata block in evaluate_url+ # and the NAT64 handling just below. (Under PUBLIC_ONLY/BOTH the+ # SSRF validator at the actual fetch is the metadata backstop.)+ if is_ip_blocked(+ ip_str, allow_localhost=True, allow_private_ips=True+ ):+ return _cache_classification(ctx, hostname, False)+ if _is_nat64_wrapped_metadata(ip_str):+ return _cache_classification(ctx, hostname, False)+ if is_private_ip(ip_str):+ return _cache_classification(ctx, hostname, True)+ except Exception: # pragma: no cover - defensive+ continue+ return _cache_classification(ctx, hostname, False)+++def _get_engine_class(engine_name: str):+ """Lazy load the engine class from the registry to avoid circular imports."""+ # Import is lazy to break the dependency cycle:+ # security/* → engines/* → security/*+ from ...web_search_engines.engine_registry import ENGINE_REGISTRY+ from ..module_whitelist import get_safe_module_class++ entry = ENGINE_REGISTRY.get(engine_name)+ if entry is None:+ return None+ try:+ return get_safe_module_class(entry.module_path, entry.class_name)+ except Exception:+ return None+++def _engine_flags(engine_cls):+ """Read the (is_public, is_local, url_setting) triple from an engine class.++ Uses ``is True`` / ``is False`` semantics: a missing attribute is treated+ as "not declared", which is distinct from an attribute explicitly set+ to ``False``.+ """+ is_public = getattr(engine_cls, "is_public", None)+ is_local = getattr(engine_cls, "is_local", None)+ url_setting = getattr(engine_cls, "url_setting", None)+ return is_public, is_local, url_setting+++def _resolve_collection_is_public(+ engine_name: str, username: Optional[str]+) -> bool:+ """Look up a collection engine's per-collection ``is_public`` flag.++ ``engine_name`` is ``"collection_<uuid>"`` or ``"library"``. The+ aggregate ``library`` is always private (it spans all local docs).+ Fails closed to private (False) on any lookup error or missing row —+ a collection we can't classify must NOT be treated as public.+ """+ if engine_name == "library" or not engine_name.startswith("collection_"):+ return False+ collection_id = engine_name[len("collection_") :]+ try:+ from ...database.models.library import Collection+ from ...database.session_context import get_user_db_session++ with get_user_db_session(username) as session:+ row = (+ session.query(Collection)+ .filter(Collection.id == collection_id)+ .first()+ )+ return bool(getattr(row, "is_public", False)) if row else False+ except Exception: # noqa: silent-exception - fail closed to private+ logger.debug(+ "could not resolve collection is_public; treating as private",+ engine=engine_name,+ )+ return False+++def _engine_bucket(+ engine_name: str,+ ctx: EgressContext,+ settings_snapshot: dict,+ metadata: Optional[dict] = None,+ allow_dns: bool = True,+):+ """Return the ``(is_public, is_local)`` tri-state for an engine, WITHOUT+ any scope logic. Shared by ``evaluate_engine`` (for the bucket check)+ and adaptive-scope resolution (to classify the primary engine).++ - Static engines: read declared ``is_public``/``is_local`` flags, with+ a configurable-URL override classified by DNS.+ - ``library`` / ``collection_<uuid>``: per-collection ``is_public``+ flag (from ``metadata`` when the caller has the engine config, else a+ direct DB lookup), defaulting private.+ - Anything else (unknown): ``(None, None)``.+ """+ engine_cls = _get_engine_class(engine_name)+ if engine_cls is None:+ if engine_name == "library" or engine_name.startswith("collection_"):+ if metadata is not None and metadata.get("is_public") is not None:+ is_public = bool(metadata.get("is_public"))+ else:+ is_public = _resolve_collection_is_public(+ engine_name, ctx.username+ )+ # A collection is ALWAYS a local knowledge base — it lives on this+ # machine — so it stays searchable under PRIVATE_ONLY regardless of+ # its public flag. ``is_public`` is ADDITIVE, not exclusive: marking+ # a collection public (non-sensitive content — papers you fetched,+ # public publications) ALSO makes it eligible under PUBLIC_ONLY and+ # OK to process with cloud inference. A private collection is+ # local-only and excluded from public-scope runs. Hence+ # ``(is_public, is_local=True)`` rather than the old mutually-+ # exclusive ``(is_public, not is_public)`` which wrongly hid a+ # public collection from private runs.+ return is_public, True+ return None, None+ is_public, is_local, url_setting = _engine_flags(engine_cls)+ # Dynamic URL classification overrides static flags when the engine+ # exposes a user-configurable URL (searxng instance, Elasticsearch host).+ if url_setting:+ host_classification = _classify_engine_url(+ url_setting, settings_snapshot, ctx, allow_dns=allow_dns+ )+ if host_classification is True:+ return False, True+ if host_classification is False:+ return True, False+ # None → fall back to static flags+ return is_public, is_local+++def evaluate_engine(+ engine_name: str,+ ctx: EgressContext,+ *,+ settings_snapshot: dict,+ metadata: Optional[dict] = None,+) -> Decision:+ """Decide whether an engine may be instantiated under the current policy.++ Allows iff (a) the engine's classification is compatible with the scope's+ bucket AND (b) if scope == STRICT, the engine name equals the primary.+ Unclassified engines always fail closed.++ ``metadata`` (optional) carries the engine's config entry (e.g. the+ per-collection ``is_public`` from search_config) so callers that already+ have it avoid a redundant DB lookup; when absent it's resolved directly.+ """+ if settings_snapshot is None:+ return Decision(False, "no_snapshot")+ try:+ # STRICT + meta-picker is incoherent (the user shouldn't have been+ # allowed to save this combination at the API layer, but defend+ # in depth at the factory too).+ if (+ ctx.scope == EgressScope.STRICT+ and ctx.primary_engine in _META_PICKER_ENGINES+ ):+ return Decision(False, "strict_with_meta_picker")++ # Meta-pickers (auto, parallel, ...) are delegators — the policy+ # gates their *children* at child instantiation time. The meta+ # engine itself is permitted under all non-STRICT scopes.+ if engine_name in _META_PICKER_ENGINES:+ if ctx.scope == EgressScope.STRICT:+ return Decision(False, "strict_with_meta_picker")+ return Decision(True, "meta_picker_delegator")++ # STRICT: only the primary engine is permitted.+ if (+ ctx.scope == EgressScope.STRICT+ and engine_name != ctx.primary_engine+ ):+ return Decision(False, "strict_not_primary")++ # A truly unknown engine (not in the static registry and not a+ # collection) fails closed as engine_unknown. ``library`` /+ # ``collection_*`` are classified by _engine_bucket below.+ is_collection = engine_name == "library" or engine_name.startswith(+ "collection_"+ )+ if _get_engine_class(engine_name) is None and not is_collection:+ return Decision(False, "engine_unknown")++ is_public, is_local = _engine_bucket(+ engine_name, ctx, settings_snapshot, metadata+ )++ # Fail-closed for unclassified engines.+ if is_public is not True and is_local is not True:+ return Decision(False, "unclassified")++ # Scope/bucket compatibility.+ if ctx.scope == EgressScope.PUBLIC_ONLY and is_public is not True:+ return Decision(False, "scope_mismatch_public_only")+ if ctx.scope == EgressScope.PRIVATE_ONLY and is_local is not True:+ return Decision(False, "scope_mismatch_private_only")++ return Decision(True, "allowed")+ except Exception: # pragma: no cover - defensive+ logger.bind(policy_audit=True).exception(+ "evaluate_engine internal error", engine=engine_name+ )+ return Decision(False, "internal_error")+++def _classify_engine_url(+ url_setting: str,+ settings_snapshot: dict,+ ctx: EgressContext,+ allow_dns: bool = True,+) -> Optional[bool]:+ """Classify an engine's configured URL via DNS resolution.++ For list-typed settings (Elasticsearch ``hosts``), returns False (public)+ if ANY entry classifies as public — safer fail-up.+ Returns True if local, False if public, None if undetermined.+ """+ value = _get_setting_value(settings_snapshot, url_setting, None)+ if not value:+ return None++ entries = value if isinstance(value, list) else [value]+ any_public = False+ any_local = False+ for entry in entries:+ if not isinstance(entry, str):+ continue+ try:+ parsed = urlsplit(entry if "://" in entry else f"http://{entry}")+ hostname = parsed.hostname+ except Exception:+ continue+ if not hostname:+ continue+ # Decode percent-encoding before classifying, identically to+ # evaluate_url: the HTTP client decodes the host before connecting,+ # so "192%2e168%2e1%2e1" must be classified as 192.168.1.1, not as+ # an unresolvable literal.+ hostname = unquote(hostname)+ result = _classify_host(hostname, ctx, allow_dns=allow_dns)+ if result is True:+ any_local = True+ elif result is False:+ any_public = True++ if any_public:+ return False # "any public" wins under list-typed settings+ if any_local:+ return True+ return None+++_CLOUD_LLM_PROVIDERS = frozenset(+ {+ "openai",+ "anthropic",+ "google",+ "openrouter",+ "deepseek",+ "xai",+ "ionos",+ }+)++# Providers that default to a localhost endpoint and ship without any+# remote routing. The LLM gate at ``config/llm_config.py`` uses this set+# as a SNAPSHOT-LESS ALLOW-LIST: when ``get_llm`` is invoked with no+# settings snapshot (background helpers, scaffolding paths, tests), only+# these providers may proceed; everything else — including ambiguous+# providers like ``openai_endpoint`` and any future cloud provider not+# yet enumerated in ``_CLOUD_LLM_PROVIDERS`` — fails closed at the gate.+# Keeping the set tight is the point: anything new defaults to refused.+_LOCAL_DEFAULT_LLM_PROVIDERS = frozenset(+ {+ "ollama",+ "lmstudio",+ "llamacpp",+ }+)++# Embeddings analogue of ``_LOCAL_DEFAULT_LLM_PROVIDERS``. Used by the+# embeddings gate as a SNAPSHOT-LESS ALLOW-LIST: when ``get_embeddings``+# runs without a settings snapshot (the ``embeddings.require_local`` toggle+# is then unreadable), only these localhost-default providers may proceed.+# ``openai`` (and any future cloud embeddings provider) fails closed —+# matching the LLM gate so a snapshot-less background caller can't ship the+# local corpus to a cloud embedder. ``sentence_transformers`` runs fully+# in-process; ``ollama`` defaults to a localhost endpoint.+_LOCAL_DEFAULT_EMBEDDING_PROVIDERS = frozenset(+ {+ "sentence_transformers",+ "ollama",+ }+)+++def _is_user_registered_llm(provider: str) -> bool:+ """True when ``provider`` is a user-registered in-process LLM.++ The programmatic API (``quick_summary(llms={"mock": ...})``) and plugins+ register LLM objects in the LLM registry. Built-in providers are ALSO+ auto-registered there by ``discover_providers()``, so registry membership+ alone cannot discriminate — a name that is registered but NOT one of the+ auto-discovered built-ins is user-supplied in-process code. Such objects+ carry no configurable endpoint the PDP could classify (denying them with+ ``provider_url_unset`` just breaks the documented offline workflow), the+ operator injected them deliberately at code level, and under+ PRIVATE_ONLY/STRICT the PEP-578 audit hook still blocks any stray+ outbound socket they might open.++ A user registration that *shadows* a built-in name (e.g. "openai") is NOT+ treated as user code: the discovered-name check keeps it on the strict+ path, and the ``_CLOUD_LLM_PROVIDERS`` gate fires first anyway.++ Fails closed (False) on any error.+ """+ try:+ from ...llm.llm_registry import is_llm_registered+ from ...llm.providers import discover_providers+ from ...llm.providers.base import normalize_provider++ if not provider or not is_llm_registered(provider):+ return False+ discovered = {normalize_provider(key) for key in discover_providers()}+ return normalize_provider(provider) not in discovered+ except Exception: # pragma: no cover - defensive+ logger.bind(policy_audit=True).exception(+ "user-registered LLM check failed", provider=provider+ )+ return False+++def evaluate_llm_endpoint(+ provider: str,+ ctx: EgressContext,+ *,+ settings_snapshot: dict,+) -> Decision:+ """Decide whether an LLM provider may be instantiated.++ Only meaningful when ``ctx.require_local_llm`` is True; otherwise allow.+ """+ if settings_snapshot is None:+ return Decision(False, "no_snapshot")+ try:+ if not ctx.require_local_llm:+ return Decision(True, "no_local_requirement")++ # Hard-cloud providers always fail under require_local_llm.+ if provider in _CLOUD_LLM_PROVIDERS:+ return Decision(False, "provider_cloud_only")++ # User-registered in-process LLMs (programmatic API / plugins) have+ # no endpoint to classify; allow them — the audit hook still+ # backstops stray sockets under PRIVATE_ONLY/STRICT.+ if _is_user_registered_llm(provider):+ return Decision(True, "user_registered_llm")++ # Providers with a configurable URL: classify the URL.+ url_key = f"llm.{provider}.url"+ url_value = _get_setting_value(settings_snapshot, url_key, None)+ if not url_value:+ # Local-default providers (ollama, lmstudio, llamacpp) without+ # an override fall back to their localhost defaults. Use the+ # shared constant so this can't drift from the snapshot-less+ # allow-list.+ if provider in _LOCAL_DEFAULT_LLM_PROVIDERS:+ return Decision(True, "provider_local_default")+ return Decision(False, "provider_url_unset")++ try:+ parsed = urlsplit(url_value)+ # Percent-decode the host before classifying — the HTTP client+ # decodes it before connect, so "http://127%2e0%2e0%2e1:11434"+ # must read as the local 127.0.0.1 (matches evaluate_url and+ # _classify_engine_url). Without this a legitimate percent-encoded+ # local endpoint is wrongly classified public and denied.+ hostname = unquote(parsed.hostname) if parsed.hostname else None+ except Exception:+ return Decision(False, "url_malformed")++ classification = _classify_host(hostname, ctx) if hostname else None+ if classification is True:+ return Decision(True, "provider_local")+ return Decision(False, "provider_remote")+ except Exception: # pragma: no cover - defensive+ logger.bind(policy_audit=True).exception(+ "evaluate_llm_endpoint internal error", provider=provider+ )+ return Decision(False, "internal_error")+++def evaluate_embeddings(+ provider: str,+ ctx: EgressContext,+ *,+ settings_snapshot: dict,+) -> Decision:+ """Decide whether an embeddings provider may be instantiated.++ Only meaningful when ``ctx.require_local_embeddings`` is True.+ OpenAI is treated as unambiguously cloud unless ``embeddings.openai.base_url``+ is configured to a local host.+ """+ if settings_snapshot is None:+ return Decision(False, "no_snapshot")+ try:+ if not ctx.require_local_embeddings:+ return Decision(True, "no_local_requirement")++ if provider == "sentence_transformers":+ return Decision(True, "provider_local")++ if provider == "ollama":+ url = _get_setting_value(+ settings_snapshot, "embeddings.ollama.url", None+ ) or _get_setting_value(settings_snapshot, "llm.ollama.url", None)+ if not url:+ return Decision(True, "provider_local_default")+ parsed = urlsplit(url)+ host = unquote(parsed.hostname) if parsed.hostname else None+ classification = _classify_host(host, ctx) if host else None+ return (+ Decision(True, "provider_local")+ if classification is True+ else Decision(False, "provider_remote")+ )++ if provider == "openai":+ base_url = _get_setting_value(+ settings_snapshot, "embeddings.openai.base_url", None+ )+ if base_url:+ parsed = urlsplit(base_url)+ host = unquote(parsed.hostname) if parsed.hostname else None+ classification = _classify_host(host, ctx) if host else None+ if classification is True:+ return Decision(True, "provider_local_endpoint")+ return Decision(False, "provider_cloud")++ return Decision(False, "provider_unknown")+ except Exception: # pragma: no cover - defensive+ logger.bind(policy_audit=True).exception(+ "evaluate_embeddings internal error", provider=provider+ )+ return Decision(False, "internal_error")+++_DANGEROUS_SCHEMES = frozenset(+ {"javascript", "data", "file", "vbscript", "about"}+)++# Cloud-metadata endpoints reachable by HOSTNAME rather than literal IP. GCP's+# IMDS answers on metadata.google.internal / metadata.goog (resolving to+# 169.254.169.254); is_ip_blocked only catches the IP literal, so these names+# need an explicit block to honor the "metadata is NEVER permitted, regardless+# of scope" invariant in evaluate_url. AWS/Azure/Alibaba IMDS have no such+# hostname (IP only) and are covered by the is_ip_blocked / alt-encoding path.+_METADATA_HOSTNAMES = frozenset({"metadata.google.internal", "metadata.goog"})+++def _normalize_alt_ipv4(host: str) -> Optional[str]:+ """Return the canonical dotted-quad for an IPv4 host written in an+ alternate encoding the libc resolver accepts but ``ipaddress.ip_address``+ rejects — octal (``0251.0376.0251.0376``), hex (``0xa9fea9fe``), integer+ (``2852039166``) and short forms (``169.254.43518``). Returns ``None`` for+ anything ``inet_aton`` won't parse (real hostnames, IPv6, junk).++ ``socket.getaddrinfo`` resolves these numeric forms to the same address the+ HTTP client will ``connect()`` to, so the metadata-IP block in+ ``evaluate_url`` must classify them by their *real* target — otherwise+ ``http://0251.0376.0251.0376/`` slips past ``is_ip_blocked`` (which only+ parses canonical notation) and reads as an allowed public host.+ """+ try:+ import socket++ return socket.inet_ntoa(socket.inet_aton(host))+ except (OSError, UnicodeError, TypeError):+ return None+++def _quota_ctx(ctx: EgressContext) -> EgressContext:+ """Return the EgressContext whose counter backs the per-RUN denied-fetch+ quota.++ Each call site (ContentFetcher, full_search, download_service, the audit+ hook) builds its OWN EgressContext from the snapshot, so a per-context+ counter would reset the budget every time a new engine/fetcher is built —+ letting a malicious document evade ``MAX_DENIED_FETCHES_PER_RUN`` by+ spreading denied fetches across contexts. The run's ARMED active context is+ set once at run start and re-armed identically on pool workers, so it is+ shared across the whole run; anchoring the counter there makes the quota+ truly per-run. Falls back to the passed ctx when no run context is armed+ (snapshot-less / programmatic callers, settings-page calls), keeping those+ isolated calls self-contained.+ """+ try:+ from .audit_hook import get_active_context++ active = get_active_context()+ except Exception: # pragma: no cover - defensive+ active = None+ return active if active is not None else ctx+++def evaluate_url(url: str, ctx: EgressContext) -> Decision:+ """Decide whether an arbitrary URL may be fetched (e.g., by ``fetch_content``).++ Enforces the run's denied-fetch quota to prevent exhaustion attacks+ via malicious indexed documents that loop the agent through hundreds+ of denied fetches.+ """+ try:+ # Quota check: hard fail after MAX_DENIED_FETCHES_PER_RUN. Anchor the+ # counter to the run's active context (``_quota_ctx``) so the budget is+ # per-RUN, not per-EgressContext. Read under that ctx's lock so+ # concurrent subagents see a consistent count.+ qctx = _quota_ctx(ctx)+ with qctx._lock:+ if qctx._fetch_denial_count["count"] >= MAX_DENIED_FETCHES_PER_RUN:+ return Decision(False, "denial_quota_exceeded")++ if not isinstance(url, str) or not url:+ return _record_denial(ctx, "url_malformed")++ try:+ parsed = urlsplit(url)+ except Exception:+ return _record_denial(ctx, "url_malformed")++ if parsed.scheme.lower() in _DANGEROUS_SCHEMES:+ return _record_denial(ctx, "dangerous_scheme")+ if parsed.scheme.lower() not in ("http", "https"):+ return _record_denial(ctx, "unsupported_scheme")+ if not parsed.hostname:+ return _record_denial(ctx, "no_hostname")++ # HTTP client libraries (requests/urllib3) percent-DECODE the host+ # in the netloc before the socket connect, but urlsplit().hostname+ # preserves the encoding. Classifying the encoded form lets+ # "http://192%2e168%2e1%2e1/" read as an unresolvable public host+ # (DNS fails) while the client actually connects to the private+ # 192.168.1.1 — a scope bypass under PUBLIC_ONLY. Classify the+ # DECODED host so the policy sees the real connect target.+ host = unquote(parsed.hostname)+ # Trailing dots are insignificant to the resolver (getaddrinfo strips+ # them), so "169.254.169.254." and "metadata.google.internal." must be+ # classified identically to their bare forms — otherwise a trailing+ # dot dodges the metadata checks below. Mirrors the SSRF validator's+ # rstrip(".").+ host = host.rstrip(".")++ # Cloud-metadata endpoints (AWS/GCE/Azure IMDS at 169.254.169.254+ # etc.) are NEVER permitted, regardless of scope. They classify as+ # link-local/private, so STRICT and PRIVATE_ONLY would otherwise+ # ALLOW them — a credential-theft path for prompt-injected fetches+ # and, more importantly, for the audit-hook net which calls+ # evaluate_url directly on raw socket.connect targets (bypassing the+ # SSRF validator that the explicit fetch PEPs run first). Reuse+ # is_ip_blocked(allow_private_ips=True), which returns True only for+ # the always-blocked metadata set (+ NAT64 wraps) and also unwraps+ # IPv4-mapped IPv6 forms.+ try:+ from ..ssrf_validator import is_ip_blocked++ if is_ip_blocked(+ host, allow_localhost=True, allow_private_ips=True+ ):+ return _record_denial(ctx, "blocked_metadata_ip")+ # is_ip_blocked only parses canonical IP notation, so an+ # alternate-encoded metadata literal (octal/hex/integer) slips+ # through above. Normalize it to the dotted-quad the resolver+ # would connect to and re-check, so the "metadata IPs are NEVER+ # permitted, regardless of scope" invariant holds for those forms+ # too — and they get the explicit blocked_metadata_ip reason+ # rather than leaking into the scope-mismatch bucket.+ alt = _normalize_alt_ipv4(host)+ if (+ alt is not None+ and alt != host+ and is_ip_blocked(+ alt, allow_localhost=True, allow_private_ips=True+ )+ ):+ return _record_denial(ctx, "blocked_metadata_ip")+ # Metadata endpoints reachable by hostname (GCP) — is_ip_blocked+ # can't see these because they aren't IP literals. Block by name so+ # the invariant holds under PUBLIC_ONLY/BOTH too (the SSRF validator+ # backstops the actual fetch, but evaluate_url's own guarantee must+ # not depend on it).+ if host.lower() in _METADATA_HOSTNAMES:+ return _record_denial(ctx, "blocked_metadata_ip")+ except Exception: # noqa: silent-exception - defensive, see below+ # Non-IP host or helper error: fall through to normal+ # classification (DNS path has its own handling).+ pass++ classification = _classify_host(host, ctx)++ if ctx.scope == EgressScope.STRICT:+ # Under STRICT, URLs whose host is private are allowed; public+ # hosts (DOI redirects, citations) are not. This is a deliberate+ # trade-off: prompt-injection spoofing of provenance tags is a+ # bigger risk than false-positives on DOI redirects.+ if classification is True:+ return Decision(True, "allowed_private_host_under_strict")+ return _record_denial(ctx, "strict_public_host")++ if ctx.scope == EgressScope.PUBLIC_ONLY:+ if classification is False:+ return Decision(True, "allowed_public_host")+ return _record_denial(ctx, "scope_mismatch_public_only")++ if ctx.scope == EgressScope.PRIVATE_ONLY:+ if classification is True:+ return Decision(True, "allowed_private_host")+ return _record_denial(ctx, "scope_mismatch_private_only")++ # BOTH: any classified host is fine.+ if classification is None:+ return _record_denial(ctx, "host_unclassified")+ return Decision(True, "allowed_both_scope")+ except Exception: # pragma: no cover - defensive+ logger.bind(policy_audit=True).exception("evaluate_url internal error")+ return Decision(False, "internal_error")+++def _record_denial(ctx: EgressContext, reason: str) -> Decision:+ """Increment the denial counter inside the frozen context's mutable dict+ and emit a redacted audit log line. Lock guards the read-modify-write+ against concurrent subagent threads.+ """+ counts_toward_quota = reason not in _NON_QUOTA_DENIAL_REASONS+ # Increment the per-RUN counter (the run's active context, shared across+ # every call-site context) rather than this one context's — see _quota_ctx.+ qctx = _quota_ctx(ctx)+ with qctx._lock:+ if counts_toward_quota:+ qctx._fetch_denial_count["count"] += 1+ count = qctx._fetch_denial_count["count"]+ logger.bind(policy_audit=True).warning(+ "policy denied URL fetch",+ reason=reason,+ scope=ctx.scope.value,+ denial_count=count,+ counted=counts_toward_quota,+ )+ return Decision(False, reason)+++def evaluate_retriever(+ retriever_name: str,+ ctx: EgressContext,+ *,+ metadata: Optional[dict] = None,+) -> Decision:+ """Decide whether a registered retriever may be invoked.++ Reads classification (``{"is_local": bool}``) set at registration+ time. Unclassified retrievers fail closed. ``metadata`` may be+ supplied by the caller (e.g. the search-engine factory, which has+ already looked up the retriever in its own registry reference) to+ avoid a second registry lookup and to honor a test-patched registry;+ when ``None`` the global registry is consulted.+ """+ try:+ if metadata is None:+ from ...web_search_engines.retriever_registry import (+ retriever_registry,+ )++ try:+ metadata = retriever_registry.get_metadata(+ retriever_name, username=ctx.username+ )+ except AttributeError:+ # Older registry without metadata API; treat as unclassified.+ return Decision(False, "retriever_unclassified")++ if not metadata:+ return Decision(False, "retriever_unknown")++ is_local = metadata.get("is_local")+ if is_local is None:+ return Decision(False, "retriever_unclassified")++ if ctx.scope == EgressScope.STRICT:+ # STRICT permits only the user's primary engine. A retriever+ # IS allowed under STRICT when it is itself the primary (the+ # common "research only against my private KB" setup);+ # otherwise it's an expansion STRICT forbids.+ if retriever_name == ctx.primary_engine:+ return Decision(True, "allowed_primary_retriever")+ return Decision(False, "strict_not_primary")+ if ctx.scope == EgressScope.PUBLIC_ONLY and is_local:+ return Decision(False, "scope_mismatch_public_only")+ if ctx.scope == EgressScope.PRIVATE_ONLY and not is_local:+ return Decision(False, "scope_mismatch_private_only")+ return Decision(True, "allowed")+ except Exception: # pragma: no cover - defensive+ logger.bind(policy_audit=True).exception(+ "evaluate_retriever internal error", retriever=retriever_name+ )+ return Decision(False, "internal_error")+++def _retriever_is_local(+ retriever_name: str, username: Optional[str]+) -> Optional[bool]:+ """Return the ``is_local`` classification (True/False) of a registered+ retriever, or ``None`` when it is unknown/unclassified.++ Mirrors the registry lookup in ``evaluate_retriever`` so adaptive-scope+ resolution and enforcement agree on a retriever primary's classification.+ """+ try:+ from ...web_search_engines.retriever_registry import (+ retriever_registry,+ )++ metadata = retriever_registry.get_metadata(+ retriever_name, username=username+ )+ except Exception: # noqa: silent-exception - unknown → caller falls back+ return None+ if not metadata:+ return None+ is_local = metadata.get("is_local")+ return is_local if isinstance(is_local, bool) else None+++def _resolve_adaptive_scope(+ primary_engine: str,+ settings_snapshot: dict,+ *,+ username: Optional[str],+ local_hostnames: tuple,+ allow_dns: bool = True,+) -> EgressScope:+ """Map ADAPTIVE to a concrete scope by classifying the primary engine.++ private primary => PRIVATE_ONLY, public primary => PUBLIC_ONLY,+ meta-picker / unknown / classification-error => BOTH (adaptive's+ documented permissive fallback, equivalent to pre-policy behavior, so a+ classification hiccup never hard-fails the run).++ A registered LangChain retriever (private KB) is not in ENGINE_REGISTRY,+ so ``_engine_bucket`` returns ``(None, None)`` for it; we then consult the+ retriever registry so a local retriever primary resolves to PRIVATE_ONLY+ (forcing local inference) rather than leaking the corpus to cloud models.+ """+ if not primary_engine or primary_engine in _META_PICKER_ENGINES:+ return EgressScope.BOTH+ try:+ probe_ctx = EgressContext(+ scope=EgressScope.BOTH,+ primary_engine=primary_engine,+ require_local_llm=False,+ require_local_embeddings=False,+ local_hostnames=local_hostnames,+ username=username,+ )+ is_public, is_local = _engine_bucket(+ primary_engine, probe_ctx, settings_snapshot, allow_dns=allow_dns+ )+ except Exception: # noqa: silent-exception - adaptive falls back to BOTH+ logger.bind(policy_audit=True).debug(+ "adaptive scope classification failed; falling back to BOTH",+ primary=primary_engine,+ )+ return EgressScope.BOTH++ if is_local is True and is_public is not True:+ return EgressScope.PRIVATE_ONLY+ if is_public is True and is_local is not True:+ return EgressScope.PUBLIC_ONLY++ # Unknown to _engine_bucket — it may be a registered retriever (private+ # KB). Classify via the retriever registry before falling back to BOTH.+ if is_public is None and is_local is None:+ retriever_local = _retriever_is_local(primary_engine, username)+ if retriever_local is True:+ return EgressScope.PRIVATE_ONLY+ if retriever_local is False:+ return EgressScope.PUBLIC_ONLY++ return EgressScope.BOTH+++def context_from_snapshot(+ settings_snapshot: dict,+ primary_engine: str,+ *,+ username: Optional[str] = None,+ allow_dns: bool = True,+) -> EgressContext:+ """Construct the frozen ``EgressContext`` for a research run.++ Reads policy settings out of the snapshot exactly once at run-start.+ Default values preserve backward compatibility: missing keys yield+ the most permissive policy (``BOTH``, no local requirements).++ ``allow_dns=False`` makes ADAPTIVE resolution skip the synchronous+ getaddrinfo when classifying a URL-configurable primary engine — used by+ the advisory warning-banner render path so a settings-page load never+ blocks on DNS. Enforcement callers leave it True so classification is+ accurate. (Has no effect for non-ADAPTIVE scopes or meta-picker primaries,+ which never resolve a URL.)++ Raises ``ValueError`` for cross-field invariant violations+ (e.g., ``STRICT`` + a meta-picker primary engine).+ """+ if settings_snapshot is None:+ raise ValueError("settings_snapshot is required")+ if not isinstance(settings_snapshot, dict):+ # Fail closed with the same ValueError contract callers already+ # handle (llm_config / research_service convert it to a hard policy+ # stop). A non-dict snapshot would otherwise crash deeper in+ # _get_setting_value with a bare AttributeError, which a broad+ # caller-side except could swallow into a permissive default.+ raise ValueError(+ f"settings_snapshot must be a dict, got "+ f"{type(settings_snapshot).__name__}"+ )++ scope_raw = _get_setting_value(+ settings_snapshot, "policy.egress_scope", "both"+ )+ try:+ scope = EgressScope(str(scope_raw).lower())+ except ValueError as exc:+ # An unrecognised scope means the saved setting was corrupted+ # or tampered with — silently falling back to BOTH (the most+ # permissive scope) would mask policy violations rather than+ # surface them. Refuse the run instead so the operator notices.+ logger.bind(policy_audit=True).warning(+ "refusing to construct EgressContext from unknown scope",+ value=scope_raw,+ )+ raise PolicyDeniedError(+ Decision(False, "unknown_egress_scope"),+ target=str(scope_raw),+ ) from exc++ require_local_llm = _coerce_bool(+ _get_setting_value(+ settings_snapshot, "llm.require_local_endpoint", False+ )+ )+ require_local_embeddings = _coerce_bool(+ _get_setting_value(settings_snapshot, "embeddings.require_local", False)+ )++ raw_hostnames = _get_setting_value(+ settings_snapshot, "llm.allowed_local_hostnames", ()+ )+ if isinstance(raw_hostnames, (list, tuple)):+ local_hostnames = tuple(h for h in raw_hostnames if isinstance(h, str))+ else:+ local_hostnames = ()++ # ADAPTIVE resolves to a concrete scope by classifying the primary+ # engine: a concrete private primary => PRIVATE_ONLY, a concrete public+ # primary => PUBLIC_ONLY, a meta-picker / unclassifiable primary =>+ # BOTH. The resolved scope is what the EgressContext stores and what+ # every downstream PEP enforces. Classification reuses _engine_bucket+ # via a throwaway BOTH-scoped context (its DNS cache is discarded).+ if scope == EgressScope.ADAPTIVE:+ scope = _resolve_adaptive_scope(+ primary_engine,+ settings_snapshot,+ username=username,+ local_hostnames=local_hostnames,+ allow_dns=allow_dns,+ )++ # PRIVATE_ONLY means "my data stays on this box." That guarantee only+ # holds if BOTH inference paths are local — a cloud LLM receives the+ # query + retrieved local chunks, and a cloud embedder receives the+ # whole corpus at ingest. So under PRIVATE_ONLY the require_local_*+ # flags are IMPLIED: scope overrides them, so a user who left them at+ # their default (False) can't silently exfiltrate via inference. This+ # also fires when ADAPTIVE resolved to PRIVATE_ONLY above.+ #+ # STRICT is deliberately NOT coupled here: it restricts the search+ # engine set to the primary only and is orthogonal to where inference+ # runs (a user may legitimately want single-engine search + a cloud+ # LLM). Callers that gate on ctx.require_local_* therefore get+ # scope-correct behaviour without a separate flag read.+ if scope == EgressScope.PRIVATE_ONLY:+ require_local_llm = True+ require_local_embeddings = True++ # Cross-field invariant: STRICT + meta-picker is incoherent.+ if scope == EgressScope.STRICT and primary_engine in _META_PICKER_ENGINES:+ raise ValueError(+ "STRICT egress scope is incompatible with a meta-picker "+ "primary engine (auto, meta, parallel, parallel_scientific). "+ "Choose a concrete primary engine or change scope."+ )++ return EgressContext(+ scope=scope,+ primary_engine=primary_engine,+ require_local_llm=require_local_llm,+ require_local_embeddings=require_local_embeddings,+ local_hostnames=local_hostnames,+ username=username,+ )+++def _get_setting_value(snapshot: dict, key: str, default):+ """Read a setting from the snapshot.++ Accepts both flat ({key: value}) and nested ({key: {"value": value}})+ schemas — both shapes appear in LDR's settings infrastructure.+ """+ raw = snapshot.get(key, default)+ if isinstance(raw, dict) and "value" in raw:+ return raw["value"]+ return raw+++def _coerce_bool(value) -> bool:+ """Coerce a setting value to bool, defensively.++ Strings ``"true"`` / ``"True"`` are True; anything else string-shaped+ is False. Real booleans pass through. The strict coercion prevents+ type confusion (e.g., the string ``"false"`` being truthy).+ """+ if isinstance(value, bool):+ return value+ if isinstance(value, str):+ return value.strip().lower() in ("true", "1", "yes", "on")+ return bool(value)diff --git a/src/local_deep_research/security/egress/validators.py b/src/local_deep_research/security/egress/validators.pynew file mode 100644index 0000000000..34bc4eb175--- /dev/null+++ b/src/local_deep_research/security/egress/validators.py@@ -0,0 +1,149 @@+"""Cross-field egress-policy validators run at settings-save time.++Pure functions: each takes the about-to-be-saved ``form_data`` plus the+current ``all_db_settings`` (key -> setting row with a ``.value``) and returns+a validation-error dict ``{"key", "error"}`` to surface in the save response,+or ``None`` when the combination is fine.++The settings write routes (``web/routes/settings_routes.py``) orchestrate these+— they stay in the route layer because they also enforce non-egress concerns —+but the egress rules themselves live here next to the policy they encode.+"""++from .policy import (+ EgressContext,+ EgressScope,+ _META_PICKER_ENGINES,+ _classify_host,+ _resolve_with_timeout,+)+++def validate_strict_meta_combo(form_data, all_db_settings):+ """Cross-field guard: STRICT egress scope + a meta-picker primary+ engine (auto / meta / parallel / parallel_scientific) is incoherent+ — STRICT means "only the primary engine", but a meta-picker has no+ single concrete primary. Returns a validation-error dict to surface+ in the save response, or None when the combination is fine.++ Effective values prefer the about-to-be-saved form value and fall+ back to the current DB value so the check is correct whether the+ user is changing scope, engine, or both.+ """++ def _effective(key):+ if key in form_data:+ return form_data[key]+ db_setting = all_db_settings.get(key)+ return db_setting.value if db_setting is not None else None++ scope_raw = _effective("policy.egress_scope")+ engine_raw = _effective("search.tool")+ if scope_raw is None or engine_raw is None:+ return None+ try:+ scope = EgressScope(str(scope_raw).lower())+ except ValueError:+ # Unknown scope value is handled elsewhere (runtime fail-closed);+ # don't block the save on it here.+ return None+ if (+ scope == EgressScope.STRICT+ and str(engine_raw).lower() in _META_PICKER_ENGINES+ ):+ return {+ "key": "policy.egress_scope",+ "error": (+ "STRICT egress scope requires a concrete primary search "+ f"engine, but the current search tool is '{engine_raw}' "+ "(a meta-picker). Choose a specific engine (e.g. arxiv, "+ "searxng) or pick a less restrictive egress scope."+ ),+ }+ return None+++def validate_allowed_local_hostnames(form_data, all_db_settings):+ """Reject public hostnames being added to llm.allowed_local_hostnames.++ The default-settings description for this key claims "Public hostnames+ added here are rejected at save time", but until now no code actually+ did the rejection. This guard resolves each entry via the same host+ classifier the policy uses, and refuses any that resolve to public+ addresses. A hostname that fails to resolve (DNS down) is accepted —+ fail-open on transient lookup errors so the user can recover.+ """+ key = "llm.allowed_local_hostnames"+ if key not in form_data:+ return None+ value = form_data[key]+ # Setting is JSON-typed; the save pipeline may hand us a list or a+ # JSON string. Decode defensively.+ if isinstance(value, str):+ try:+ import json as _json++ decoded = _json.loads(value) if value.strip() else []+ except Exception:+ return {+ "key": key,+ "error": "allowed_local_hostnames must be a JSON array of hostnames",+ }+ value = decoded+ if not isinstance(value, list):+ return {+ "key": key,+ "error": "allowed_local_hostnames must be a list",+ }++ # Build a minimal real context just for the resolver. Use the+ # dataclass constructor — NOT EgressContext.__new__ + setattr, which+ # raised FrozenInstanceError on this frozen dataclass and (separately)+ # set a non-existent ``allowed_local_hostnames`` field instead of the+ # real ``local_hostnames`` the classifier reads. The constructor+ # initializes the init=False internals (_dns_cache, _lock as RLock)+ # correctly. Empty local_hostnames => classify purely on IP class.+ probe_ctx = EgressContext(+ scope=EgressScope.BOTH,+ primary_engine="auto",+ require_local_llm=False,+ require_local_embeddings=False,+ local_hostnames=(),+ )++ rejected = []+ for entry in value:+ if not isinstance(entry, str) or not entry.strip():+ continue+ hostname = entry.strip().lower()+ try:+ # Distinguish "could not resolve" from "resolved to a public IP".+ # _classify_host collapses BOTH to False (its documented fail-safe+ # treats an unresolvable host as public), so relying on it here+ # would reject a legitimate intranet/VPN host on any DNS hiccup or+ # split-horizon DNS — the exact use case this setting exists for.+ # Only reject names that actually resolve to a public address;+ # accept unresolvable ones (fail-open on save, as documented).+ # _resolve_with_timeout returns the addrinfo for literal IPs too,+ # so literal public/private IPs still flow through _classify_host.+ if _resolve_with_timeout(hostname) is None:+ continue+ classification = _classify_host(hostname, probe_ctx)+ except Exception:+ # DNS or unknown error — allow (fail open) so the user can+ # save when networking is flaky. Runtime classification will+ # still gate egress.+ continue+ if classification is False:+ rejected.append(hostname)+ if rejected:+ return {+ "key": key,+ "error": (+ "These hostnames resolve to PUBLIC addresses and would "+ "let the policy treat external hosts as local: "+ f"{', '.join(rejected)}. Remove them, or use the SSRF "+ "allowlist instead."+ ),+ }+ return Nonediff --git a/src/local_deep_research/security/egress/warnings.py b/src/local_deep_research/security/egress/warnings.pynew file mode 100644index 0000000000..d83ef04cec--- /dev/null+++ b/src/local_deep_research/security/egress/warnings.py@@ -0,0 +1,231 @@+"""Egress-policy warning checks.++Pure functions: take primitive values, return a warning dict or ``None``.+These surface on the research-form banner so users can see at a glance+when their current policy lets data leave the machine.++The audit identified three independent vectors that each deserve their+own banner:++1. **Public search egress enabled** — the active scope permits any public+ engine to fire. Loudest on a fresh install where the default scope is+ ``both``; suppressed via ``ui.egress_warnings_acknowledged`` so first-+ launch isn't a wall of red.+2. **Cloud LLM enabled** — the user hasn't opted into+ ``llm.require_local_endpoint`` and the configured provider is one of+ the unambiguously-cloud providers. Critical-severity because the+ user's full prompt content leaks on every research run.+3. **Cloud embeddings enabled** — even worse: indexing a corpus with+ OpenAI embeddings POSTs every chunk to OpenAI. Critical-severity.+"""++from typing import Optional++# Single source of truth — these warning checks now live in the same+# egress package as the PDP, so import the cloud-provider set directly+# instead of maintaining a hand-synced copy (which previously risked+# drift: a provider added to one list but not the other).+from .policy import _CLOUD_LLM_PROVIDERS+++def check_public_egress_enabled(+ egress_scope: str,+ acknowledged: bool,+) -> Optional[dict]:+ """Banner when the active scope permits public-internet search engines.++ Fires for ``both`` and ``public_only`` (the two scopes that allow+ public engines). Suppressed when the user has acknowledged the+ egress-policy warnings via the fresh-install flag — otherwise every+ new install would face three loud banners before doing anything.+ """+ if acknowledged:+ return None+ # "adaptive" can resolve to a public-allowing scope (public or meta-picker+ # primary), so warn conservatively — better a dismissible banner than a+ # silent public-egress path the user didn't expect.+ if egress_scope not in ("both", "public_only", "adaptive"):+ return None++ return {+ "type": "public_egress_enabled",+ "icon": "🌐",+ "title": "Public search egress enabled",+ "message": (+ "This run can reach external search engines. Set the Egress "+ "Scope to 'Private only' below if you want to keep all "+ "research traffic on-machine."+ ),+ "dismissKey": "app.warnings.dismiss_egress_policy",+ "actionUrl": "#policy_egress_scope",+ "actionLabel": "Adjust scope",+ }+++def check_effective_scope(+ egress_scope: str,+ effective_scope: str,+ primary_engine: str,+ acknowledged: bool,+) -> Optional[dict]:+ """Informational banner stating what the ADAPTIVE scope actually resolves+ to for the current primary engine.++ Adaptive is opaque on its own ("follows your primary"); this makes the+ effective posture explicit so the user knows whether THIS config means+ public searches, private/local-only, or both. Only fires for ``adaptive``+ (the explicit scopes are self-describing in the dropdown). Has its own+ dismiss flag so dismissing it doesn't hide the risk banners.+ """+ if acknowledged:+ return None+ if (egress_scope or "").lower() != "adaptive":+ return None+ eff = (effective_scope or "").lower()+ primary = primary_engine or "your primary engine"+ base = {+ "type": "egress_effective_scope",+ "dismissKey": "app.warnings.dismiss_adaptive_scope_info",+ "actionUrl": "#policy_egress_scope",+ "actionLabel": "Change mode",+ }+ if eff == "private_only":+ return {+ **base,+ "icon": "🔒",+ "title": "Adaptive → Private only (stays local)",+ "message": (+ f"Your primary engine ('{primary}') is private, so this run "+ "stays on your machine: only local engines run, and LLM + "+ "embeddings are forced local — nothing leaves the box."+ ),+ }+ if eff == "public_only":+ return {+ **base,+ "icon": "🌐",+ "title": "Adaptive → Public searches enabled",+ "message": (+ f"Your primary engine ('{primary}') is public, so this run "+ "uses public web/academic engines. Your local collections are "+ "not queried."+ ),+ }+ # both (meta-picker primary) / unclassifiable+ return {+ **base,+ "icon": "🔀",+ "title": "Adaptive → Public + private searches enabled",+ "message": (+ f"Your primary ('{primary}') is a meta-picker, so this run can "+ "use both public engines and your local collections."+ ),+ }+++def check_cloud_llm_enabled(+ provider: str,+ require_local_endpoint: bool,+ acknowledged: bool,+) -> Optional[dict]:+ """Banner when the configured LLM provider is cloud-only and the+ require-local-endpoint toggle is off.++ Critical-severity: the user's full prompt content (including the+ research query and all retrieved context) is sent to the provider on+ every call.+ """+ if acknowledged:+ return None+ if require_local_endpoint:+ return None+ if not provider or provider.lower() not in _CLOUD_LLM_PROVIDERS:+ return None++ return {+ "type": "cloud_llm_enabled",+ "icon": "☁️",+ "title": "LLM provider is cloud-hosted",+ "message": (+ f"Your LLM provider ({provider}) is cloud-hosted. Query "+ "content will be sent off-machine on every research run, "+ "independent of the Egress Scope setting. Tick 'Require local "+ "LLM endpoint' below if you want fully local inference."+ ),+ "dismissKey": "app.warnings.dismiss_cloud_llm",+ "actionUrl": "#llm_require_local_endpoint",+ "actionLabel": "Require local LLM",+ }+++def check_cloud_embeddings_enabled(+ embeddings_provider: str,+ embeddings_base_url: str,+ require_local_embeddings: bool,+ acknowledged: bool,+) -> Optional[dict]:+ """Banner when the embeddings provider sends data off-machine on+ indexing.++ Highest-severity of the three: indexing a private corpus with OpenAI+ embeddings POSTs every chunk to OpenAI's API. A user who is unaware+ of this loses their corpus.++ Suppressed when ``base_url`` is set to a local URL (LM Studio,+ vLLM, llama.cpp), since the OpenAI provider type is then pointed at+ a local endpoint.+ """+ if acknowledged:+ return None+ if require_local_embeddings:+ return None+ if (embeddings_provider or "").lower() != "openai":+ return None++ # If base_url is set and points to a local hostname, suppress the+ # warning — the user has configured OpenAI-compatible-but-local+ # (LM Studio, vLLM, etc.). Uses is_private_ip so RFC1918 ranges+ # (10.x, 172.16-31.x, 192.168.x), CGNAT, link-local, IPv6 private,+ # and .local mDNS hosts are all recognised. Substring-matching+ # against a small list missed legitimate private-network endpoints.+ #+ # NOTE: this is a LITERAL-IP / .local check only — it does NOT resolve+ # DNS, unlike the enforcing evaluate_embeddings (which DNS-resolves via+ # _classify_host). So a private-DNS base_url (e.g. my-llm.internal → 10.x)+ # is treated as LOCAL by enforcement (nothing leaves) yet still shows this+ # banner. That divergence is intentional: this advisory runs on every+ # settings-page render and must stay synchronous/cheap, so it deliberately+ # over-warns (the safe direction) rather than add a blocking DNS lookup to+ # page load. The banner never blocks anything; the PDP is the source of+ # truth for what actually egresses.+ if embeddings_base_url:+ try:+ from urllib.parse import urlsplit++ from ..network_utils import is_private_ip++ parsed = urlsplit(embeddings_base_url)+ hostname = parsed.hostname+ if hostname and is_private_ip(hostname):+ return None+ except Exception: # noqa: silent-exception+ # Defensive: if URL parsing fails, fall through to issuing+ # the banner — the failure itself is a sign of misconfig+ # the user should see, not a reason to mask the warning.+ pass++ return {+ "type": "cloud_embeddings_enabled",+ "icon": "📤",+ "title": "Document chunks will be sent to OpenAI",+ "message": (+ "Your embeddings provider is OpenAI. Indexing a collection "+ "will POST every chunk to OpenAI's API — the entire corpus "+ "leaves the machine. Tick 'Require local embeddings' below "+ "to switch to sentence-transformers or a local OpenAI-"+ "compatible endpoint."+ ),+ "dismissKey": "app.warnings.dismiss_cloud_embeddings",+ "actionUrl": "#embeddings_require_local",+ "actionLabel": "Require local embeddings",+ }diff --git a/src/local_deep_research/settings/manager.py b/src/local_deep_research/settings/manager.pyindex f4d9bbf715..52256cfeaa 100644--- a/src/local_deep_research/settings/manager.py+++ b/src/local_deep_research/settings/manager.py@@ -154,6 +154,25 @@ def _filter_setting_columns(data: dict) -> dict:return {k: v for k, v in data.items() if k in valid_columns}+_POLICY_AUDIT_KEYS = frozenset(+ {+ "llm.require_local_endpoint",+ "llm.allowed_local_hostnames",+ "embeddings.require_local",+ }+)+++def _is_policy_setting(key: str) -> bool:+ """Return True for security-relevant setting keys that need an+ audit-log entry on change. Scope is intentionally narrow so this+ audit hook doesn't widen into general settings-change logging.+ """+ if key.startswith("policy."):+ return True+ return key in _POLICY_AUDIT_KEYS++def _infer_ui_element(value: Any, current: str = "text") -> str:"""Infer the appropriate ui_element string from a Python value's type.@@ -686,6 +705,8 @@ class SettingsManager(ISettingsManager):.filter(Setting.key == key).first())+ # Capture old value for the policy-change audit log below.+ old_value = setting.value if setting is not None else Noneif setting:if not setting.editable:logger.error(@@ -769,6 +790,18 @@ class SettingsManager(ISettingsManager):# Emit WebSocket event for settings changeself._emit_settings_changed([key])+ # N16: audit log on policy.* / llm.require_local_endpoint /+ # embeddings.require_local changes. Targeted scope — only+ # security-relevant settings are logged, to avoid widening+ # this PR into a general audit-log refactor.+ if _is_policy_setting(key):+ logger.bind(policy_audit=True).warning(+ "policy setting changed | key={} old={} new={}",+ key,+ old_value,+ value,+ )+return Trueexcept SQLAlchemyError:logger.exception(f"Error setting value for key: {key}")diff --git a/src/local_deep_research/utilities/log_utils.py b/src/local_deep_research/utilities/log_utils.pyindex 8f2560628a..d9e5481f02 100644--- a/src/local_deep_research/utilities/log_utils.py+++ b/src/local_deep_research/utilities/log_utils.py@@ -414,6 +414,15 @@ def frontend_progress_sink(message: loguru.Message) -> None:# Can't use logger here as it causes deadlockreturn+ # Defence in depth (R4-09): never forward policy-audit log lines+ # to WebSocket subscribers. They carry engine names + reason codes+ # which could leak the active scope to a cross-origin observer under+ # CORS=*. Today policy_audit logs don't bind research_id so the+ # research_id guard above already skips them; this filter is the+ # explicit guarantee in case a future call site binds both.+ if record.get("extra", {}).get("policy_audit"):+ return+frontend_log = {"log_entry": {"message": _truncate_for_frontend(record["message"]),diff --git a/src/local_deep_research/utilities/search_cache.py b/src/local_deep_research/utilities/search_cache.pyindex 659bdfc63f..28b2792088 100644--- a/src/local_deep_research/utilities/search_cache.py+++ b/src/local_deep_research/utilities/search_cache.py@@ -98,18 +98,34 @@ class SearchCache:def _get_query_hash(self, query: str, search_engine: str = "default") -> str:- """Generate hash for query + search engine + username combination.+ """Generate hash for query + search engine + username + scope.Incorporates the current user's username into the hash so thatdifferent users' cached results are isolated from each other.++ Also incorporates the active egress policy scope, so a+ ``scope=BOTH`` cache entry isn't returned to a later+ ``scope=PRIVATE_ONLY`` query — that would be a clean policy+ bypass via warm cache. Old cached entries (written before this+ column existed) miss on lookup and get rewritten naturally."""normalized_query = self._normalize_query(query)- # Get username from thread context for per-user cache isolation+ # Get username + scope from thread context for per-user,+ # per-policy cache isolation.username = ""+ # Default scope is "both" (the permissive default) so that+ # "no search context" and "context without a policy snapshot"+ # hash identically — both represent the no-policy situation.+ scope = "both"context = get_search_context()if context:username = context.get("username", "") or ""- cache_key = f"{username}:{search_engine}:{normalized_query}"+ settings_snapshot = context.get("settings_snapshot") or {}+ scope_raw = settings_snapshot.get("policy.egress_scope", "both")+ if isinstance(scope_raw, dict) and "value" in scope_raw:+ scope_raw = scope_raw["value"]+ scope = str(scope_raw or "both").lower()+ cache_key = f"{username}:{scope}:{search_engine}:{normalized_query}"return hashlib.sha256(cache_key.encode()).hexdigest()def _cleanup_expired(self):diff --git a/src/local_deep_research/utilities/thread_context.py b/src/local_deep_research/utilities/thread_context.pyindex 5b3dc5520d..9a4267b941 100644--- a/src/local_deep_research/utilities/thread_context.py+++ b/src/local_deep_research/utilities/thread_context.py@@ -98,14 +98,41 @@ def preserve_research_context(func: Callable) -> Callable:# programmatic access.context = get_search_context()+ # Capture the submitter thread's egress audit-hook context too. Unlike the+ # search context (a ContextVar that langchain's executors copy), the audit+ # context lives in a threading.local that stdlib ThreadPoolExecutor workers+ # do NOT inherit — so without re-arming it here the PEP-578 socket backstop+ # would be inactive on every pool worker. Capture once on the submitter+ # thread; re-arm + clear per task below.+ try:+ from ..security.egress.audit_hook import (+ get_active_context as _get_egress_ctx,+ )++ egress_ctx = _get_egress_ctx()+ except Exception:+ egress_ctx = None+@functools.wraps(func)def wrapper(*args, **kwargs):if context is not None:set_search_context(context)+ if egress_ctx is not None:+ from ..security.egress.audit_hook import (+ set_active_context as _set_egress_ctx,+ )++ _set_egress_ctx(egress_ctx)try:return func(*args, **kwargs)finally:+ if egress_ctx is not None:+ from ..security.egress.audit_hook import (+ clear_active_context as _clear_egress_ctx,+ )++ _clear_egress_ctx()if context is not None:clear_search_context()# Clean up thread-local DB engines created by metrics recordingdiff --git a/src/local_deep_research/web/api.py b/src/local_deep_research/web/api.pyindex 5a9913709b..85e50766e4 100644--- a/src/local_deep_research/web/api.py+++ b/src/local_deep_research/web/api.py@@ -272,9 +272,25 @@ def api_quick_summary():f"No database session for user: {username}")except Exception:- logger.warning("Failed to get settings snapshot")- # Continue with empty snapshot rather than failing- params["settings_snapshot"] = {}+ # Fail CLOSED: if the user's settings can't be loaded we can't+ # read their egress policy (PRIVATE_ONLY, require-local, …), and+ # continuing with an empty snapshot would silently downgrade+ # them to the permissive BOTH scope — bypassing the very+ # boundary they configured. Refuse the run instead of leaking.+ logger.exception(+ "Failed to load settings snapshot; refusing research to "+ "avoid running without the user's egress policy"+ )+ return (+ jsonify(+ {+ "error": "Could not load your settings, so the "+ "research was refused to avoid bypassing your "+ "privacy/egress policy. Please try again."+ }+ ),+ 503,+ )else:logger.debug("No username in session, skipping settings snapshot")params["settings_snapshot"] = {}diff --git a/src/local_deep_research/web/app_factory.py b/src/local_deep_research/web/app_factory.pyindex ccf93dd180..86670efe9e 100644--- a/src/local_deep_research/web/app_factory.py+++ b/src/local_deep_research/web/app_factory.py@@ -450,9 +450,36 @@ def apply_middleware(app):ResearchStatus.ERROR,ResearchStatus.CANCELLED,]+ # Make the active egress scope available to every template so base.html+ # can render it onto <body data-scope=…>. Scope-aware CSS in styles.css+ # picks it up to color the research card, chat input, etc. Falls back+ # to "both" if anything goes wrong — fail open visually, never crash a+ # page render over a styling cue.+ scope = "both"+ try:+ from flask import session as flask_session+ from ..database.session_context import get_user_db_session+ from ..utilities.db_utils import get_settings_manager++ username = flask_session.get("username") if flask_session else None+ if username:+ with get_user_db_session(username) as db_session:+ if db_session:+ sm = get_settings_manager(db_session, username)+ scope = (+ sm.get_setting("policy.egress_scope", "both")+ or "both"+ )+ except Exception:+ logger.debug(+ "Failed to read egress scope for template context",+ exc_info=True,+ )+return {"research_status_enum": {m.name: m.value for m in ResearchStatus},"research_terminal_states": [str(s) for s in terminal],+ "egress_scope": scope,}# Clean up database sessions after each request@@ -600,6 +627,16 @@ def register_blueprints(app):"search_strategy": settings_manager.get_setting("search.search_strategy", "source-based"),+ # Egress policy controls (Stage 1c UI).+ "policy_egress_scope": settings_manager.get_setting(+ "policy.egress_scope", "both"+ ),+ "llm_require_local_endpoint": settings_manager.get_setting(+ "llm.require_local_endpoint", False+ ),+ "embeddings_require_local": settings_manager.get_setting(+ "embeddings.require_local", False+ ),}# Debug logging@@ -844,6 +881,41 @@ def register_error_handlers(app):)return jsonify(error.to_dict()), error.status_code+ # PolicyDeniedError can escape any synchronous request-path PEP+ # (LLM, embeddings, URL fetch) that wasn't caught at the call site.+ # Without this handler it surfaces as a 500 — turn it into a clean+ # 400 with the decision reason so the user (and the operator+ # audit log) sees a policy-shaped error instead of a stack trace.+ try:+ from ..security.egress.policy import PolicyDeniedError++ @app.errorhandler(PolicyDeniedError)+ def handle_policy_denied(error):+ reason = getattr(+ getattr(error, "decision", None), "reason", "denied"+ )+ target = getattr(error, "target", "")+ logger.bind(policy_audit=True).warning(+ "PolicyDeniedError surfaced at Flask error handler",+ reason=reason,+ target=target,+ )+ return make_response(+ jsonify(+ {+ "status": "error",+ "message": (+ f"Egress policy refused this request: {reason}"+ ),+ }+ ),+ 400,+ )+ except ImportError:+ # egress_policy is in-tree; skipping the handler is only useful+ # for stripped-down test harnesses.+ pass+# Handle CSRF validation errors as JSONtry:from flask_wtf.csrf import CSRFErrordiff --git a/src/local_deep_research/web/routes/metrics_routes.py b/src/local_deep_research/web/routes/metrics_routes.pyindex b63ebc3608..fba227fc61 100644--- a/src/local_deep_research/web/routes/metrics_routes.py+++ b/src/local_deep_research/web/routes/metrics_routes.py@@ -2223,6 +2223,47 @@ def api_journal_data_download():)from ...journal_quality.data_sources import ALL_SOURCES+ # Egress policy: this endpoint streams several hundred MB over public+ # HTTP. Under an offline-for-public scope (PRIVATE_ONLY / STRICT) the+ # user has opted out of public egress, so refuse rather than reaching+ # out. A corrupt/unknown scope also fails closed. The background+ # auto-download path is already gated by+ # JournalReputationFilter._should_skip_journal_fetch_for_scope; this+ # closes the manual button as the matching entry point.+ from ...security.egress.policy import EgressScope+ from ...utilities.db_utils import get_settings_manager++ username = flask_session.get("username")+ scope_raw = get_settings_manager(username=username).get_setting(+ "policy.egress_scope", "both"+ )+ try:+ scope = EgressScope(str(scope_raw).lower())+ except ValueError:+ scope = None # corrupt scope -> fail closed below+ if scope is None or scope in (+ EgressScope.PRIVATE_ONLY,+ EgressScope.STRICT,+ ):+ logger.bind(policy_audit=True).warning(+ "journal data download refused by egress policy",+ scope=str(scope_raw),+ )+ return (+ jsonify(+ {+ "success": False,+ "message": (+ "Journal data download needs public network "+ "access, which the current egress policy blocks "+ "(private/offline scope). Change the egress "+ "scope to download."+ ),+ }+ ),+ 403,+ )+force = request.json.get("force", False) if request.is_json else Falsesuccess, internal_message = download_journal_data(force=force)if not success:diff --git a/src/local_deep_research/web/routes/research_routes.py b/src/local_deep_research/web/routes/research_routes.pyindex 1aa8cfc3bf..ca65b74938 100644--- a/src/local_deep_research/web/routes/research_routes.py+++ b/src/local_deep_research/web/routes/research_routes.py@@ -187,6 +187,14 @@ def _extract_research_params(data, settings_manager):"search.search_strategy", "source-based")+ # Egress policy per-research overrides. Mirror the+ # model/search_engine pattern: missing values fall back to saved+ # settings; supplied values override JUST FOR THIS RUN. They do+ # NOT persist to the user's settings DB.+ policy_egress_scope = data.get("policy_egress_scope")+ llm_require_local_endpoint = data.get("llm_require_local_endpoint")+ embeddings_require_local = data.get("embeddings_require_local")+return {"model_provider": model_provider,"model": model,@@ -198,9 +206,131 @@ def _extract_research_params(data, settings_manager):"iterations": iterations,"questions_per_iteration": questions_per_iteration,"strategy": strategy,+ "policy_egress_scope": policy_egress_scope,+ "llm_require_local_endpoint": llm_require_local_endpoint,+ "embeddings_require_local": embeddings_require_local,}+def _precheck_engine_policy(settings_manager, params, search_engine, username):+ """Validate the requested search engine against the saved egress+ policy at the request boundary.++ Returns a Flask ``(response, status)`` tuple when the request should+ be rejected, or ``None`` to continue. Falls through (returns None)+ when there's no real dict snapshot to validate or the policy module+ errors — the factory PEP still enforces at engine-instantiation time.+ """+ try:+ from ...security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_engine,+ )++ policy_snapshot = settings_manager.get_settings_snapshot()+ # Only validate against a real dict snapshot. A test double or+ # an unavailable settings backend hands back something else;+ # skip rather than misfire.+ if not isinstance(policy_snapshot, dict):+ return None++ # Overlay per-research form overrides so the snapshot reflects+ # what the user picked for THIS run (per-research overrides, not+ # a global settings save).+ _apply_policy_overrides(policy_snapshot, params)++ primary = settings_manager.get_setting("search.tool", "auto")+ try:+ policy_ctx = context_from_snapshot(+ policy_snapshot, primary or "auto", username=username+ )+ except PolicyDeniedError as exc:+ return (+ jsonify(+ {+ "status": "error",+ "message": (+ f"Egress policy refused this run: "+ f"{exc.decision.reason}"+ ),+ }+ ),+ 400,+ )+ except ValueError as exc:+ # STRICT + meta-picker is incoherent. Previously this+ # raised, fell through to the outer ``except Exception``+ # below, and silently returned None — so the run started+ # successfully at the precheck and only failed at a+ # downstream PEP. Surface it here as a 400 instead.+ logger.bind(policy_audit=True).warning(+ "POST /api/start_research policy precheck rejected",+ reason=str(exc),+ )+ return (+ jsonify(+ {+ "status": "error",+ "message": (f"Egress policy refused this run: {exc}"),+ }+ ),+ 400,+ )++ decision = evaluate_engine(+ search_engine,+ policy_ctx,+ settings_snapshot=policy_snapshot,+ )+ if not decision.allowed:+ logger.bind(policy_audit=True).warning(+ "POST /api/start_research search_engine refused",+ engine=search_engine,+ reason=decision.reason,+ )+ return (+ jsonify(+ {+ "status": "error",+ "message": (+ f"Search engine '{search_engine}' is not "+ f"permitted under the current egress "+ f"policy: {decision.reason}"+ ),+ }+ ),+ 400,+ )+ return None+ except Exception:+ # Policy module unavailable / internal error → log and fall+ # through; the factory PEP will catch any actual violation.+ logger.exception("egress policy pre-check skipped")+ return None+++def _apply_policy_overrides(settings_snapshot, params):+ """Overlay form-supplied egress policy values onto the snapshot.++ Per-research overrides: the user picked these values for this+ specific run and they do NOT persist to the settings DB. Mirrors+ how model / search_engine overrides work today.+ """+ if not isinstance(settings_snapshot, dict):+ return+ if params.get("policy_egress_scope") is not None:+ settings_snapshot["policy.egress_scope"] = params["policy_egress_scope"]+ if params.get("llm_require_local_endpoint") is not None:+ settings_snapshot["llm.require_local_endpoint"] = bool(+ params["llm_require_local_endpoint"]+ )+ if params.get("embeddings_require_local") is not None:+ settings_snapshot["embeddings.require_local"] = bool(+ params["embeddings_require_local"]+ )++def _queue_research(db_session,username,@@ -334,6 +464,19 @@ def start_research():questions_per_iteration = params["questions_per_iteration"]strategy = params["strategy"]+ # Egress policy: server-side check on the requested search+ # engine BEFORE we enqueue the research. Defends against an API+ # client posting an engine name that violates the saved policy+ # (e.g. ``{"search_engine": "pubmed"}`` under STRICT+primary=arxiv).+ # The factory PEP catches the same case at instantiation time,+ # but rejecting at the request boundary lets us return a clean+ # 4xx instead of an opaque background failure.+ policy_error = _precheck_engine_policy(+ settings_manager, params, search_engine, username+ )+ if policy_error is not None:+ return policy_error+# Debug logging for model parameter specificallylogger.debug(f"Extracted model value: '{model}' (type: {type(model).__name__})"@@ -503,6 +646,10 @@ def start_research():)# Get all current settings as a snapshot (bypass cache to ensure fresh data)all_settings = settings_manager.get_all_settings(bypass_cache=True)+ # Apply per-research egress policy overrides (form-supplied+ # values override saved settings JUST FOR THIS RUN; they do+ # not persist to the user's settings DB).+ _apply_policy_overrides(all_settings, params)# Add settings snapshot to metadataresearch_settings["settings_snapshot"] = all_settings@@ -528,6 +675,7 @@ def start_research():all_settings = settings_manager.get_all_settings(bypass_cache=True)+ _apply_policy_overrides(all_settings, params)research_settings["settings_snapshot"] = all_settingslogger.info(f"Captured {len(all_settings)} settings using temporary session for research {research_id}"diff --git a/src/local_deep_research/web/routes/settings_routes.py b/src/local_deep_research/web/routes/settings_routes.pyindex b0618fc543..3e76891509 100644--- a/src/local_deep_research/web/routes/settings_routes.py+++ b/src/local_deep_research/web/routes/settings_routes.py@@ -65,6 +65,10 @@ from ...database.encrypted_db import db_managerfrom ...utilities.db_utils import get_settings_managerfrom ...utilities.url_utils import normalize_urlfrom ...security.decorators import require_json_body+from ...security.egress.validators import (+ validate_allowed_local_hostnames,+ validate_strict_meta_combo,+)from ..auth.decorators import login_requiredfrom ..utils.request_helpers import parse_bool_argfrom ...security.rate_limiter import settings_limit@@ -91,6 +95,7 @@ settings_bp = Blueprint("settings", __name__, url_prefix="/settings")# Settings with dynamically populated options (excluded from validation)DYNAMIC_SETTINGS = ["llm.provider", "llm.model", "search.tool"]+# Namespace validation for new setting creation via the web API.# Keys starting with any ALLOWED prefix may be created; any prefix in# BLOCKED takes precedence and is rejected even if it also matches an@@ -170,6 +175,47 @@ def _get_setting_from_session(key: str | None, default=None):return default+def _model_list_local_only() -> bool:+ """True when the model-list endpoints must NOT call cloud provider APIs+ (doing so sends the API key off-machine).++ The effective local-only posture = the ``llm.require_local_endpoint``+ toggle OR a private egress scope (``private_only``, or ``adaptive``+ resolving to private). Checking only the raw toggle missed a PRIVATE_ONLY+ user who never ticked it — the scope forces local at run time but the+ stored toggle is still False. Best-effort: returns False (allow) on error+ so a transient settings hiccup doesn't break the model dropdown.+ """+ username = session.get("username")+ try:+ with get_user_db_session(username) as db_session:+ if not db_session:+ return False+ sm = get_settings_manager(db_session, username)+ if bool(sm.get_setting("llm.require_local_endpoint", False)):+ return True+ scope = str(sm.get_setting("policy.egress_scope", "both")).lower()+ if scope == "private_only":+ return True+ if scope == "adaptive":+ from ...security.egress.policy import context_from_snapshot++ snap = sm.get_settings_snapshot()+ if isinstance(snap, dict):+ primary = sm.get_setting("search.tool", "auto")+ return bool(+ context_from_snapshot(+ snap, primary or "auto", username=username+ ).require_local_llm+ )+ return False+ except Exception:+ logger.debug(+ "model-list local-only check failed; allowing", exc_info=True+ )+ return False++def validate_setting(setting: Setting, value: Any) -> Tuple[bool, Optional[str]]:@@ -316,6 +362,43 @@ def save_all_settings(db_session=None, settings_manager=None):for key in non_editable_keys:del form_data[key]+ # Cross-field egress-policy validation (H4 layer 2): reject the+ # incoherent STRICT + meta-picker combination at save time so the+ # user finds out immediately, instead of mid-research when the+ # factory PEP raises. Effective values are the about-to-be-saved+ # form values, falling back to the current DB value when the form+ # doesn't include that key.+ _policy_err = validate_strict_meta_combo(form_data, all_db_settings)+ if _policy_err is not None:+ validation_errors.append(_policy_err)+ return (+ jsonify(+ {+ "status": "error",+ "message": "Validation errors",+ "errors": validation_errors,+ }+ ),+ 400,+ )++ # Reject public hostnames being added to the local-hosts allowlist.+ _hosts_err = validate_allowed_local_hostnames(+ form_data, all_db_settings+ )+ if _hosts_err is not None:+ validation_errors.append(_hosts_err)+ return (+ jsonify(+ {+ "status": "error",+ "message": "Validation errors",+ "errors": validation_errors,+ }+ ),+ 400,+ )+# Update each settingfor key, value in form_data.items():# Skip corrupted keys or empty strings as keys@@ -685,6 +768,21 @@ def save_settings(db_session=None, settings_manager=None):for key in non_editable_keys:del form_data[key]+ # Egress-policy validators — the JSON route (save_all_settings) runs+ # these; the POST fallback must too, or a JS-disabled client could+ # persist a STRICT+meta-picker combo or whitelist a public hostname+ # as "local", neither of which the JSON route permits.+ for _policy_validator in (+ validate_strict_meta_combo,+ validate_allowed_local_hostnames,+ ):+ _policy_err = _policy_validator(form_data, all_db_settings)+ if _policy_err is not None:+ flash(+ _policy_err.get("error", "Invalid policy setting"), "error"+ )+ return redirect(url_for("settings.settings_page"))+# Process each settingfor key, value in form_data.items():try:@@ -893,6 +991,36 @@ def api_update_setting(key, db_session=None):{"error": f"Invalid value for setting {key}"}), 400+ # Cross-field egress-policy validation. The full-form saves+ # (save_all_settings / save_settings) run these guards, but this+ # single-key PUT endpoint did not — letting a client persist an+ # incoherent STRICT + meta-picker combo, or smuggle a public+ # hostname into llm.allowed_local_hostnames (which the host+ # classifier then trusts as local), one key at a time. Run the+ # same validators here for the keys they govern.+ if key in (+ "policy.egress_scope",+ "search.tool",+ "llm.allowed_local_hostnames",+ ):+ _all_db_settings = {+ s.key: s for s in db_session.query(Setting).all()+ }+ _form = {key: value}+ for _validator in (+ validate_strict_meta_combo,+ validate_allowed_local_hostnames,+ ):+ _err = _validator(_form, _all_db_settings)+ if _err is not None:+ logger.bind(policy_audit=True).warning(+ "egress-policy setting rejected at "+ "api_update_setting",+ key=key,+ reason=_err.get("error"),+ )+ return jsonify({"error": _err["error"]}), 400+# Update setting# Pass the db_session to avoid session lookup issuessuccess = set_setting(key, value, db_session=db_session)@@ -910,6 +1038,9 @@ def api_update_setting(key, db_session=None):"llm.local_context_window_size","llm.context_window_unrestricted","llm.context_window_size",+ "policy.egress_scope",+ "llm.require_local_endpoint",+ "embeddings.require_local",]if key in warning_affecting_keys:@@ -1329,6 +1460,19 @@ def api_get_available_models():# Get the API key from settingsapi_key = _get_setting_from_session("llm.openai.api_key", "")+ # Egress policy: if the user is local-only, don't reach out to the+ # OpenAI model list (which sends the API key off-machine). Gate on+ # the EFFECTIVE posture — the require_local toggle OR a private+ # egress scope — so a PRIVATE_ONLY user who never ticked the toggle+ # still doesn't leak. Return an empty list so the UI degrades+ # gracefully (the dropdown just shows no OpenAI models).+ require_local = _model_list_local_only()+ if require_local:+ logger.bind(policy_audit=True).info(+ "skipping OpenAI model list (local-only egress posture)"+ )+ api_key = ""+if api_key:import openaifrom openai import OpenAI@@ -1405,6 +1549,15 @@ def api_get_available_models():# Get the API key from settingsapi_key = _get_setting_from_session("llm.anthropic.api_key", "")+ # Egress policy: skip the Anthropic model list (cloud-only) when+ # the effective posture is local-only (toggle OR private scope).+ require_local = _model_list_local_only()+ if require_local:+ logger.bind(policy_audit=True).info(+ "skipping Anthropic model list (local-only egress posture)"+ )+ api_key = ""+if api_key:# Import Anthropic package here to avoid dependency issues if not installedfrom anthropic import Anthropic@@ -1460,6 +1613,23 @@ def api_get_available_models():discovered_providers = discover_providers()+ # Egress policy: when the effective posture is local-only the user+ # has opted into local-only inference; skip every auto-discovered+ # provider list (OpenRouter, Google, XAI, IonOS, ...). Provider+ # URLs are not all classifiable cheaply here, so we fail closed+ # — the user can still see ollama/lmstudio/llamacpp models from+ # the dedicated local provider branches above. Use the scope-aware+ # helper (matching the OpenAI/Anthropic branches) so a PRIVATE_ONLY /+ # adaptive-private user is covered even if the raw toggle is False.+ require_local_for_discovered = _model_list_local_only()+ if require_local_for_discovered:+ logger.bind(policy_audit=True).info(+ "skipping auto-discovered provider model lists "+ "(effective local-only egress posture)",+ providers=list(discovered_providers.keys()),+ )+ discovered_providers = {}+for provider_key, provider_info in discovered_providers.items():provider_models = []try:diff --git a/src/local_deep_research/web/services/research_service.py b/src/local_deep_research/web/services/research_service.pyindex 4341fc85b7..06d515b3bf 100644--- a/src/local_deep_research/web/services/research_service.py+++ b/src/local_deep_research/web/services/research_service.py@@ -812,6 +812,11 @@ def run_research_process(research_id, query, mode, **kwargs):"research_id": research_id,"username": username,"user_password": user_password,+ # Settings snapshot may be absent here (early init); the+ # shared context built later overwrites this with the+ # populated snapshot. Defaults to {} so cache reads in any+ # intermediate code path see a non-None dict.+ "settings_snapshot": kwargs.get("settings_snapshot") or {},})@@ -891,6 +896,52 @@ def run_research_process(research_id, query, mode, **kwargs):set_settings_context(settings_context)+ # Defense-in-depth: register an EgressContext on the audit-hook+ # thread-local so socket.connect calls that bypass our explicit+ # PEPs (third-party libraries, future contributors reaching for+ # raw requests.get, prompt-injection-steered tools) are gated+ # against the same scope. Cleared by the @thread_cleanup exit+ # handler in database/thread_local_session.py when the worker+ # winds down. If snapshot construction errors here, we log and+ # continue — the explicit PEPs still gate the known callers, so+ # the run is not less secure than before this hook existed.+ try:+ from ...security import set_active_context as _set_egress_ctx+ from ...security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ )++ try:+ _egress_primary = settings_snapshot.get("search.tool", "auto")+ if (+ isinstance(_egress_primary, dict)+ and "value" in _egress_primary+ ):+ _egress_primary = _egress_primary["value"]+ _egress_ctx = context_from_snapshot(+ settings_snapshot,+ _egress_primary or "auto",+ username=username,+ )+ _set_egress_ctx(_egress_ctx)+ except (PolicyDeniedError, ValueError) as _ctx_err:+ # STRICT+meta-picker or corrupted scope. The run-start+ # precheck already rejects this at the API boundary;+ # if we get here it is because the run was launched+ # without going through the precheck (CLI, scheduler).+ # Re-raise so the worker fails fast with a clear reason+ # rather than running unprotected.+ logger.bind(policy_audit=True).warning(+ "research worker refused: egress policy unevaluable",+ research_id=research_id,+ reason=str(_ctx_err),+ )+ raise+ except ImportError:+ # security module unavailable — preserve legacy behaviour.+ logger.debug("egress audit hook unavailable for research worker")+# user_password already extracted above (before termination check)# Create shared research context that can be updated during research@@ -904,6 +955,12 @@ def run_research_process(research_id, query, mode, **kwargs):"search_engine_selected": search_engine,"username": username, # Add username for queue operations"user_password": user_password, # Add password for metrics access+ # SearchCache hash includes the active egress scope read from+ # this snapshot, so a scope=BOTH cache entry doesn't get+ # returned to a later scope=PRIVATE_ONLY query. Without the+ # snapshot here, the cache scope-tag silently degrades to+ # "both" and the bypass is not closed.+ "settings_snapshot": settings_snapshot,"chat_session_id": kwargs.get("chat_session_id"),}@@ -1182,14 +1239,21 @@ def run_research_process(research_id, query, mode, **kwargs):# Override LLM if model or model_provider specifiedif model or model_provider:try:- # Get LLM with the overridden settings- # Use the shared_research_context which includes username+ # Get LLM with the overridden settings.+ # Pass settings_snapshot explicitly — without it the LLM+ # PEP (llm_config.py:295) skips evaluate_llm_endpoint+ # because the gate is "if settings_snapshot is not None+ # and provider:". The snapshot is already stored on+ # shared_research_context as "settings_snapshot".use_llm = get_llm(model_name=model,provider=model_provider,openai_endpoint_url=custom_endpoint,research_id=research_id,research_context=shared_research_context,+ settings_snapshot=shared_research_context.get(+ "settings_snapshot"+ ),)logger.info(diff --git a/src/local_deep_research/web/static/js/collection_create.js b/src/local_deep_research/web/static/js/collection_create.jsindex 5091d4829e..10281f5d10 100644--- a/src/local_deep_research/web/static/js/collection_create.js+++ b/src/local_deep_research/web/static/js/collection_create.js@@ -73,10 +73,12 @@ async function handleCreateCollection(e) {const csrfToken = window.api ? window.api.getCsrfToken() : '';// Prepare JSON data+ const isPublicEl = document.getElementById('collection-is-public');const jsonData = {name: name.trim(),description: description ? description.trim() : '',- type: 'user_uploads'+ type: 'user_uploads',+ is_public: isPublicEl ? isPublicEl.checked : false};const response = await safeFetch(URLS.LIBRARY_API.COLLECTION_CREATE, {diff --git a/src/local_deep_research/web/static/js/collection_details.js b/src/local_deep_research/web/static/js/collection_details.jsindex ee81eb9dce..959380e246 100644--- a/src/local_deep_research/web/static/js/collection_details.js+++ b/src/local_deep_research/web/static/js/collection_details.js@@ -22,10 +22,44 @@ document.addEventListener('DOMContentLoaded', function() {document.getElementById('delete-collection-btn').addEventListener('click', deleteCollection);document.getElementById('cancel-indexing-btn').addEventListener('click', cancelIndexing);+ const publicToggle = document.getElementById('collection-is-public');+ if (publicToggle) {+ publicToggle.addEventListener('change', () => updateCollectionIsPublic(publicToggle.checked));+ }+// Check if there's an active indexing taskcheckAndResumeIndexing();});+/**+ * Persist the collection's public/private (egress) flag.+ */+async function updateCollectionIsPublic(isPublic) {+ const toggle = document.getElementById('collection-is-public');+ try {+ const csrfToken = window.api ? window.api.getCsrfToken() : '';+ const response = await safeFetch(URLBuilder.build(URLS.LIBRARY_API.COLLECTION_DETAILS, COLLECTION_ID), {+ method: 'PUT',+ headers: {+ 'Content-Type': 'application/json',+ 'X-CSRFToken': csrfToken+ },+ body: JSON.stringify({ is_public: isPublic })+ });+ const data = await response.json();+ if (data.success) {+ if (collectionData) collectionData.is_public = !!data.collection.is_public;+ showSuccess(isPublic ? 'Collection marked public.' : 'Collection marked private (local-only).');+ } else {+ if (toggle) toggle.checked = !isPublic; // revert on failure+ showError('Failed to update collection: ' + (data.error || 'unknown error'));+ }+ } catch (e) {+ if (toggle) toggle.checked = !isPublic; // revert on failure+ showError('Failed to update collection privacy setting.');+ }+}+/*** Load collection details and documents*/@@ -42,6 +76,12 @@ async function loadCollectionDetails() {document.getElementById('collection-name').textContent = collectionData.name;document.getElementById('collection-description').textContent = collectionData.description || '';+ // Reflect the public/private (egress) flag+ const publicToggle = document.getElementById('collection-is-public');+ if (publicToggle) {+ publicToggle.checked = !!collectionData.is_public;+ }+// Update statisticsupdateStatistics();diff --git a/src/local_deep_research/web/static/js/components/research.js b/src/local_deep_research/web/static/js/components/research.jsindex 1a92c6904c..e46e43d5eb 100644--- a/src/local_deep_research/web/static/js/components/research.js+++ b/src/local_deep_research/web/static/js/components/research.js@@ -384,6 +384,9 @@// Finally, load settings after data is availableloadSettings();++ // H4: set STRICT-scope availability for the initial engine.+ updateStrictScopeAvailability();}).catch(error => {SafeLogger.error('Failed to load options:', error);@@ -393,6 +396,8 @@// Still load settings even if data loading failsloadSettings();+ updateStrictScopeAvailability();+if (window.ui && window.ui.showAlert) {window.ui.showAlert('Some options could not be loaded. Using defaults instead.', 'warning');}@@ -515,6 +520,17 @@searchEngineInput.value = value;}+ // Keep the hidden input in sync so the STRICT-scope+ // availability check (and form submit) see the new value.+ const seHidden = document.getElementById('search_engine_hidden');+ if (seHidden) {+ seHidden.value = value;+ }++ // H4: re-evaluate whether STRICT egress scope is allowed+ // for this engine.+ updateStrictScopeAvailability();+// Save search engine selection to settings - only if not initializingif (!isInitializing) {saveSearchEngineSettings(value);@@ -615,6 +631,51 @@}}+ // Engines that delegate to children and therefore have no single+ // concrete "primary" — STRICT egress scope is incompatible with these+ // (mirrors _META_PICKER_ENGINES in security/egress/policy.py).+ const META_PICKER_ENGINES = ['auto', 'meta', 'parallel', 'parallel_scientific'];++ /**+ * H4 layer 1: disable the STRICT egress-scope option when the selected+ * search engine is a meta-picker. The server also rejects this combo+ * (save-time validation + the research-route pre-check), but disabling+ * it here prevents the user from choosing an incoherent combination in+ * the first place. If STRICT was already selected when the engine+ * switches to a meta-picker, fall back to "both" and surface a hint.+ */+ function updateStrictScopeAvailability() {+ const scopeSelect = document.getElementById('policy_egress_scope');+ if (!scopeSelect) return;+ const strictOption = scopeSelect.querySelector('option[value="strict"]');+ if (!strictOption) return;++ const hiddenInput = document.getElementById('search_engine_hidden');+ const engine = (hiddenInput && hiddenInput.value+ ? hiddenInput.value+ : (typeof selectedSearchEngineValue !== 'undefined' ? selectedSearchEngineValue : '')+ ).toString().toLowerCase();++ const helpEl = document.getElementById('policy_egress_scope_help');+ const isMetaPicker = META_PICKER_ENGINES.includes(engine);++ if (isMetaPicker) {+ strictOption.disabled = true;+ // If STRICT was selected, revert to the safe default and explain.+ if (scopeSelect.value === 'strict') {+ scopeSelect.value = 'both';+ if (helpEl) {+ helpEl.textContent =+ '"Primary only" is unavailable while the search engine is "' + engine ++ '" (a meta-picker with no single primary). Pick a specific ' ++ 'engine to use Primary only. Scope reset to Public + private.';+ }+ }+ } else {+ strictOption.disabled = false;+ }+ }+/*** Setup event listeners*/@@ -840,6 +901,33 @@});}+ // Privacy & Egress controls — persist to settings DB on change so+ // the picked scope/local-inference toggles survive the next page load+ // and the egress-policy warning banner refreshes immediately.+ const policyScopeSelect = document.getElementById('policy_egress_scope');+ if (policyScopeSelect) {+ policyScopeSelect.addEventListener('change', function() {+ saveSearchSetting('policy.egress_scope', this.value);+ applyPrivacyPanelScope(this.value);+ });+ // Apply the initial cue on page load (the data-scope attribute is+ // already set server-side from settings; this just keeps the icon+ // in sync without requiring a roundtrip).+ applyPrivacyPanelScope(policyScopeSelect.value);+ }+ const llmRequireLocalInput = document.getElementById('llm_require_local_endpoint');+ if (llmRequireLocalInput) {+ llmRequireLocalInput.addEventListener('change', function() {+ saveSearchSetting('llm.require_local_endpoint', this.checked);+ });+ }+ const embRequireLocalInput = document.getElementById('embeddings_require_local');+ if (embRequireLocalInput) {+ embRequireLocalInput.addEventListener('change', function() {+ saveSearchSetting('embeddings.require_local', this.checked);+ });+ }+// LM Studio URL change - save to settings managerif (lmstudioUrlInput) {lmstudioUrlInput.addEventListener('change', function() {@@ -2264,6 +2352,64 @@}// Save search setting to database+ /**+ * Reflect the chosen egress scope visually: swap the panel's data-scope+ * (CSS handles the border/background/header color) and the header icon.+ * Icon mapping mirrors the dropdown semantics — globe = "anywhere",+ * cloud = "data going out", house = "stays home", target = "one tool only".+ */+ function applyPrivacyPanelScope(scope) {+ const normalized = scope || 'adaptive';+ const panel = document.querySelector('.ldr-privacy-panel');+ if (panel) panel.setAttribute('data-scope', normalized);+ // Also stamp the body so CSS can escalate the cue to the research card+ // and query textarea (see <style> block in research.html).+ if (document.body) document.body.dataset.scope = normalized;++ // Auto-check + lock the local-inference toggles under "Private only",+ // so the form matches what the backend actually enforces+ // (context_from_snapshot forces require_local_llm/embeddings under+ // PRIVATE_ONLY) and matches docs/egress-modes.md. Visual only — the+ // saved scope is what the backend couples; we restore the user's stored+ // preference when they leave Private only. (Mirrors applyEgressScopeLock+ // on the settings page.)+ const LOCK_TITLE = 'Forced on by the Private-only egress scope — local inference is required so data stays on this machine.';+ const scopeLocked = normalized === 'private_only';+ ['llm_require_local_endpoint', 'embeddings_require_local'].forEach((id) => {+ const cb = document.getElementById(id);+ if (!cb) return;+ if (scopeLocked) {+ // Remember the user's choice once, so we can restore it later.+ if (!cb.dataset.userCheckedSaved) {+ cb.dataset.userChecked = cb.checked ? '1' : '0';+ cb.dataset.userCheckedSaved = '1';+ }+ cb.checked = true;+ cb.disabled = true;+ cb.title = LOCK_TITLE;+ } else {+ if (cb.dataset.userCheckedSaved) {+ cb.checked = cb.dataset.userChecked === '1';+ delete cb.dataset.userChecked;+ delete cb.dataset.userCheckedSaved;+ }+ cb.disabled = false;+ cb.title = '';+ }+ });++ const icon = document.getElementById('ldr-privacy-panel-icon');+ if (!icon) return;+ const iconClassByScope = {+ adaptive: 'fas fa-shield-alt',+ both: 'fas fa-shield-alt',+ public_only: 'fas fa-cloud',+ private_only: 'fas fa-home',+ strict: 'fas fa-bullseye',+ };+ icon.className = iconClassByScope[normalized] || iconClassByScope.adaptive;+ }+function saveSearchSetting(settingKey, value) {// Save to the database using the settings APIfetch(`/settings/api/${settingKey}`, {@@ -2418,6 +2564,16 @@const questionsInput = document.getElementById('questions_per_iteration');const questionsPerIteration = questionsInput ? parseInt(questionsInput.value, 10) : 3;+ // Egress policy form fields (per-research override, not saved+ // to settings). Mirror the model/search_engine pattern: the+ // server falls back to saved settings when these are unset.+ const policyScopeEl = document.getElementById('policy_egress_scope');+ const llmLocalEl = document.getElementById('llm_require_local_endpoint');+ const embLocalEl = document.getElementById('embeddings_require_local');+ const policyEgressScope = policyScopeEl ? policyScopeEl.value : null;+ const llmRequireLocalEndpoint = llmLocalEl ? !!llmLocalEl.checked : null;+ const embeddingsRequireLocal = embLocalEl ? !!embLocalEl.checked : null;+// Prepare the data for submissionconst formData = {query,@@ -2431,6 +2587,15 @@iterations,questions_per_iteration: questionsPerIteration};+ if (policyEgressScope !== null && policyEgressScope !== '') {+ formData.policy_egress_scope = policyEgressScope;+ }+ if (llmRequireLocalEndpoint !== null) {+ formData.llm_require_local_endpoint = llmRequireLocalEndpoint;+ }+ if (embeddingsRequireLocal !== null) {+ formData.embeddings_require_local = embeddingsRequireLocal;+ }SafeLogger.log('Submitting research with data:', formData);diff --git a/src/local_deep_research/web/static/js/components/settings.js b/src/local_deep_research/web/static/js/components/settings.jsindex 2fa5330e33..c1c7a2039d 100644--- a/src/local_deep_research/web/static/js/components/settings.js+++ b/src/local_deep_research/web/static/js/components/settings.js@@ -1040,6 +1040,16 @@'report': 'Reports'};+ // Map auxiliary key prefixes onto the tab that should render+ // them. egress-policy keys live under their own prefixes+ // (policy.*, embeddings.*) but conceptually belong to existing+ // tabs — without this remap the settings dashboard silently+ // hides them since no 'policy' / 'embeddings' tab exists.+ const prefixToTab = {+ 'policy': 'search',+ 'embeddings': 'search'+ };+// Define settings that should only appear in specific tabsconst tabSpecificSettings = {'llm': [@@ -1050,7 +1060,9 @@'openai_endpoint_url','lmstudio_url','llamacpp_url',- 'api_key'+ 'api_key',+ 'require_local_endpoint',+ 'allowed_local_hostnames'],'search': ['iterations',@@ -1066,7 +1078,9 @@'search_language','time_period','tool',- 'snippets_only'+ 'snippets_only',+ 'egress_scope',+ 'require_local'],'report': ['knowledge_accumulation',@@ -1121,8 +1135,11 @@// If we're on a specific tab, only show settings for that tabif (tab !== 'all') {+ // Treat aliased prefixes (e.g. policy.* shown under+ // search tab) as if they belonged to the target tab.+ const effectivePrefix = prefixToTab[prefix] || prefix;// Only show settings in tab-specific lists for that tab- if (tab === prefix) {+ if (tab === effectivePrefix) {// For tab-specific settings, make sure they're in the listif (tabSpecificSettings[tab] && tabSpecificSettings[tab].includes(subKey)) {return true;@@ -1948,6 +1965,59 @@// Also initialize the main setup which finds all dropdownssetupCustomDropdowns();+ // Reflect the scope→local-inference coupling in the checkboxes.+ applyEgressScopeLock();++ }++ /**+ * Visibly reflect the scope→inference coupling. When the egress scope is+ * "private_only" the backend forces local LLM + embeddings regardless of+ * these checkboxes (context_from_snapshot coupling), so show that truth:+ * force-check + disable the two require-local checkboxes with a tooltip.+ *+ * Visual only — it does NOT persist (no change event is dispatched and the+ * disabled controls don't submit), so the user's stored preference is+ * restored from originalSettings when they pick a non-private scope.+ */+ function applyEgressScopeLock() {+ const scopeSelect = document.getElementById('setting-policy-egress-scope');+ if (!scopeSelect) return;+ const boxes = [+ { id: 'setting-llm-require-local-endpoint', key: 'llm.require_local_endpoint' },+ { id: 'setting-embeddings-require-local', key: 'embeddings.require_local' },+ ];+ const LOCK_TITLE = 'Forced on by the Private-only egress scope — local inference is required so data stays on this machine.';+ const apply = () => {+ const locked = scopeSelect.value === 'private_only';+ boxes.forEach(({ id, key }) => {+ const cb = document.getElementById(id);+ if (!cb) return;+ const fallbackId = cb.getAttribute('data-hidden-fallback');+ const fallback = fallbackId ? document.getElementById(fallbackId) : null;+ if (locked) {+ cb.checked = true;+ cb.disabled = true;+ cb.title = LOCK_TITLE;+ // Disabled controls don't submit; keep the hidden fallback+ // disabled too so it can't override with 'false'. The scope+ // setting is what's saved; the backend derives the coupling.+ if (fallback) fallback.disabled = true;+ } else {+ cb.disabled = false;+ cb.title = '';+ // Restore the user's stored preference (visual lock only).+ const stored = originalSettings[key];+ cb.checked = stored === true || stored === 'true';+ if (fallback) fallback.disabled = cb.checked;+ }+ });+ };+ if (!scopeSelect.dataset.lockWired) {+ scopeSelect.addEventListener('change', apply);+ scopeSelect.dataset.lockWired = '1';+ }+ apply();}/**diff --git a/src/local_deep_research/web/templates/base.html b/src/local_deep_research/web/templates/base.htmlindex cc35fe0289..ac4ec200ee 100644--- a/src/local_deep_research/web/templates/base.html+++ b/src/local_deep_research/web/templates/base.html@@ -49,6 +49,79 @@<!-- Theme overrides must load AFTER base styles to take precedence --><link rel="stylesheet" href="/static/css/themes.css">+ <!--+ Egress-scope visual cues+ ========================+ Inline (not in styles.css) because styles.css is bundled by Vite —+ editing the source doesn't take effect without a rebuild, and these+ cues need to ship with the next reload. body[data-scope=…] is+ server-rendered by base.html below from the user's saved+ policy.egress_scope; JS on the research form updates it live when+ the dropdown changes (see static/js/components/research.js).++ Palette:+ BOTH → no left-border accent (default state, no extra cue)+ PUBLIC_ONLY → blue #3b82f6 (informational "public sources only")+ PRIVATE_ONLY → teal #14a37f (privacy mode)+ STRICT → violet #8b5cf6 (locked to a single engine)+ Risk ordering (data exfil): BOTH > PUBLIC_ONLY > STRICT/PRIVATE_ONLY.+ BOTH is the riskiest because it permits local engines AND a cloud+ LLM in the same run — local-data-to-cloud leakage happens through+ the LLM call, not the search egress itself. PUBLIC_ONLY blocks+ local engines so local data never enters the pipeline; that+ active choice is informational (blue), not a deeper warning.+ The "Public search egress enabled" banner does the actual nag.+ Red intentionally avoided.+ -->+ <style>+ .ldr-card.ldr-research-card,+ #chat-page,+ .ldr-chat-container {+ transition: border-left-color 0.2s ease, background-color 0.2s ease;+ border-left: 5px solid transparent;+ }+ #query,+ #chat-input {+ transition: border-color 0.2s ease, box-shadow 0.2s ease;+ }+ /* BOTH (default) has NO body[data-scope] rule — the+ transparent left border above stays, the textarea uses its+ default border, and the warning banner does the nagging. */+ body[data-scope="public_only"] .ldr-card.ldr-research-card,+ body[data-scope="public_only"] #chat-page,+ body[data-scope="public_only"] .ldr-chat-container {+ border-left-color: #3b82f6;+ background-color: rgba(59, 130, 246, 0.03);+ }+ body[data-scope="public_only"] #query,+ body[data-scope="public_only"] #chat-input {+ border-color: #3b82f6;+ box-shadow: 0 0 0 1px rgba(59, 130, 246, 0.22);+ }+ body[data-scope="private_only"] .ldr-card.ldr-research-card,+ body[data-scope="private_only"] #chat-page,+ body[data-scope="private_only"] .ldr-chat-container {+ border-left-color: #14a37f;+ background-color: rgba(20, 163, 127, 0.03);+ }+ body[data-scope="private_only"] #query,+ body[data-scope="private_only"] #chat-input {+ border-color: #14a37f;+ box-shadow: 0 0 0 1px rgba(20, 163, 127, 0.25);+ }+ body[data-scope="strict"] .ldr-card.ldr-research-card,+ body[data-scope="strict"] #chat-page,+ body[data-scope="strict"] .ldr-chat-container {+ border-left-color: #8b5cf6;+ background-color: rgba(139, 92, 246, 0.035);+ }+ body[data-scope="strict"] #query,+ body[data-scope="strict"] #chat-input {+ border-color: #8b5cf6;+ box-shadow: 0 0 0 1px rgba(139, 92, 246, 0.25);+ }+ </style>+<!-- Force hide sidebar on mobile with inline CSS for highest priority --><style>@media (max-width: 767px) {@@ -88,7 +161,7 @@{% block extra_head %}{% endblock %}</head>-<body>+<body data-scope="{{ egress_scope or 'both' }}"><!-- Skip to content link for keyboard/screen reader users --><a href="#main-content" class="ldr-skip-link">Skip to main content</a>diff --git a/src/local_deep_research/web/templates/pages/collection_create.html b/src/local_deep_research/web/templates/pages/collection_create.htmlindex b95cac2bd6..4219f40aee 100644--- a/src/local_deep_research/web/templates/pages/collection_create.html+++ b/src/local_deep_research/web/templates/pages/collection_create.html@@ -42,6 +42,20 @@<div class="ldr-form-help">Optional: Provide context about the collection contents and purpose</div></div>+ <div class="ldr-form-group">+ <label class="ldr-collection-public-label">+ <input type="checkbox" id="collection-is-public" name="is_public">+ Public collection (content is not sensitive)+ </label>+ <div class="ldr-form-help">+ Leave unchecked (default) for <strong>private</strong> collections: they+ stay off the public internet under Public-only / Adaptive egress scope and+ force local LLM/embeddings inference when used, so their contents never+ reach a cloud model. Check this only for non-sensitive content you're happy+ to process with cloud inference.+ </div>+ </div>+<div class="ldr-form-actions"><button type="button" class="ldr-btn-collections ldr-btn-collections-secondary" onclick="window.location.href='/library/collections'"><i aria-hidden="true" class="fas fa-times"></i> Canceldiff --git a/src/local_deep_research/web/templates/pages/collection_details.html b/src/local_deep_research/web/templates/pages/collection_details.htmlindex d36fee2ead..52a9952a71 100644--- a/src/local_deep_research/web/templates/pages/collection_details.html+++ b/src/local_deep_research/web/templates/pages/collection_details.html@@ -43,6 +43,26 @@</div></div>+ <!-- Privacy & Egress -->+ <div class="ldr-rag-section">+ <div class="ldr-section-header">+ <h2><i aria-hidden="true" class="fas fa-shield-alt"></i> Privacy & Egress</h2>+ </div>+ <div class="ldr-form-group">+ <label class="ldr-collection-public-label">+ <input type="checkbox" id="collection-is-public">+ Public collection (content is not sensitive)+ </label>+ <div class="ldr-form-help" id="collection-is-public-help">+ <strong>Private</strong> (default): excluded from runs under Public-only / Adaptive-public+ egress scope, and forces local LLM/embeddings inference when used — its contents never+ reach a cloud model. Check this only for non-sensitive content you're happy to process+ with cloud inference.+ <a href="https://github.com/LearningCircuit/local-deep-research/blob/main/docs/egress-modes.md#per-collection-publicprivate" target="_blank" rel="noopener noreferrer">Learn more ↗</a>+ </div>+ </div>+ </div>+<!-- Statistics --><div class="ldr-rag-section"><div class="ldr-section-header">diff --git a/src/local_deep_research/web/templates/pages/research.html b/src/local_deep_research/web/templates/pages/research.htmlindex c277fad499..73bcd121f1 100644--- a/src/local_deep_research/web/templates/pages/research.html+++ b/src/local_deep_research/web/templates/pages/research.html@@ -78,6 +78,70 @@</fieldset></div>+ <!-- Privacy & Egress — hoisted above Advanced Options because these+ govern WHAT data leaves the machine (a security boundary), not+ WHICH model/tuning to use. Hiding them inside the collapsible+ panel meant a user could launch with the wrong policy without+ realizing it. Always visible by design.+ The data-scope attribute is updated by JS on dropdown change;+ CSS below colors the frame / header / icon so the active scope+ is instantly recognizable without reading the dropdown value. -->+ <style>+ /* Default panel = neutral slate. BOTH no longer wears+ an amber "warning" border at the panel level — the+ "Public search egress enabled" banner does that job+ in text, and reserving color for active-choice scopes+ (sky/teal/violet) makes the cue more legible.+ Specific scope rules below override per state. */+ .ldr-privacy-panel { border:1px solid #475569; border-radius:6px; padding:0.5rem 1rem 0.85rem; background:rgba(71, 85, 105, 0.05); transition:border-color 0.15s ease, background-color 0.15s ease; }+ .ldr-privacy-panel-header { font-size:0.95rem; font-weight:600; color:#cbd5e1; margin-bottom:0.5rem; transition:color 0.15s ease; }+ .ldr-privacy-panel[data-scope="public_only"] { border-color:#3b82f6; background:rgba(59, 130, 246, 0.07); }+ .ldr-privacy-panel[data-scope="public_only"] .ldr-privacy-panel-header { color:#60a5fa; }+ .ldr-privacy-panel[data-scope="private_only"] { border-color:#14a37f; background:rgba(20, 163, 127, 0.06); }+ .ldr-privacy-panel[data-scope="private_only"] .ldr-privacy-panel-header { color:#0f7a5f; }+ .ldr-privacy-panel[data-scope="strict"] { border-color:#8b5cf6; background:rgba(139, 92, 246, 0.07); }+ .ldr-privacy-panel[data-scope="strict"] .ldr-privacy-panel-header { color:#a78bfa; }+ /* Local-inference checkboxes adopt the "local = teal" cue when on. */+ .ldr-privacy-panel input[type="checkbox"] { accent-color:#14a37f; }+ .ldr-experimental-badge { font-size:0.72rem; font-weight:600; color:#d9a441; border:1px solid rgba(217, 164, 65, 0.4); border-radius:4px; padding:0.05rem 0.35rem; margin-left:0.4rem; vertical-align:middle; text-transform:uppercase; letter-spacing:0.02em; cursor:help; }+ /* Whole-workspace scope cue (research card / query textarea /+ chat input) lives in styles.css now so chat.html and any+ future page that loads the shared stylesheet inherit it. */+ </style>+ <div class="ldr-form-group">+ <div role="group" aria-labelledby="ldr-privacy-egress-label" class="ldr-policy-group ldr-privacy-panel" data-scope="{{ settings.policy_egress_scope or 'adaptive' }}">+ <div id="ldr-privacy-egress-label" class="ldr-privacy-panel-header">+ <i id="ldr-privacy-panel-icon" class="fas fa-shield-alt" aria-hidden="true"></i> Privacy & Egress+ <span class="ldr-experimental-badge" title="Experimental: this egress boundary is new and defense-in-depth. It blocks the known data-egress paths under the selected scope, but don't rely on it as your only protection for highly sensitive data yet.">(Experimental)</span>+ </div>+ <div class="ldr-form-row" style="margin:0;">+ <div class="ldr-form-group ldr-half">+ <label for="policy_egress_scope">{{ tooltip("Egress Scope", "Adaptive (default): follows your primary engine — a private primary keeps everything local, a public primary allows public engines, a meta-picker allows both. Public + private: any classified engine. Public only: any public web/academic engine, no local sources. Private only: only local engines (library, paperless, etc.). Primary only: the selected search engine and nothing else — incompatible with auto / meta / parallel primaries.") }}</label>+ <select id="policy_egress_scope" name="policy_egress_scope" class="ldr-form-control">+ <option value="adaptive" {% if not settings.policy_egress_scope or settings.policy_egress_scope == "adaptive" %}selected{% endif %}>Adaptive (default — follows primary)</option>+ <option value="both" {% if settings.policy_egress_scope == "both" %}selected{% endif %}>Public + private (any engine)</option>+ <option value="public_only" {% if settings.policy_egress_scope == "public_only" %}selected{% endif %}>Public only (no local)</option>+ <option value="private_only" {% if settings.policy_egress_scope == "private_only" %}selected{% endif %}>Private only (no internet)</option>+ <option value="strict" {% if settings.policy_egress_scope == "strict" %}selected{% endif %}>Primary only (single engine)</option>+ </select>+ <span class="ldr-input-help" id="policy_egress_scope_help">Limits which search engines this run may use. <a href="https://github.com/LearningCircuit/local-deep-research/blob/main/docs/egress-modes.md" target="_blank" rel="noopener noreferrer">What does each mode do? ↗</a></span>+ </div>+ <div class="ldr-form-group ldr-half">+ <div style="font-weight:normal; opacity:0.85; font-size:0.9rem; margin-bottom:0.35rem;">Local inference</div>+ <label for="llm_require_local_endpoint" style="display:flex; align-items:center; gap:0.4rem; font-weight:normal; margin:0 0 0.35rem 0;">+ <input type="checkbox" id="llm_require_local_endpoint" name="llm_require_local_endpoint" value="true" {% if settings.llm_require_local_endpoint %}checked{% endif %}>+ {{ tooltip("Require local LLM endpoint", "When checked, blocks cloud LLM providers (OpenAI, Anthropic, Google, OpenRouter). Local providers (Ollama, LM Studio, llama.cpp) and OpenAI-compatible endpoints pointed at local hosts are allowed.") }}+ </label>+ <label for="embeddings_require_local" style="display:flex; align-items:center; gap:0.4rem; font-weight:normal; margin:0;">+ <input type="checkbox" id="embeddings_require_local" name="embeddings_require_local" value="true" {% if settings.embeddings_require_local %}checked{% endif %}>+ {{ tooltip("Require local embeddings", "When checked, indexing with cloud embedding providers (OpenAI) is blocked. Critical for keeping a local corpus from being uploaded chunk-by-chunk during indexing.") }}+ </label>+ </div>+ </div>+ <span class="ldr-input-help" style="display:block; margin-top:0.5rem;">These controls decide what data leaves your machine for this run. They override the corresponding global settings just for this research. Configure the model and search engine in Advanced Options below.</span>+ </div>+ </div>+<!-- Advanced Options - positioned before start for configuration --><button type="button" id="advanced-options-toggle" class="ldr-advanced-options-toggle ldr-open" aria-expanded="true" aria-controls="advanced-options-panel"><span class="ldr-toggle-text">Advanced Options</span>@@ -251,7 +315,7 @@<div class="ldr-form-row"><!-- Search Strategy -->- <div class="ldr-form-group ldr-half">+ <div class="ldr-form-group ldr-full"><label for="strategy">{{ tooltip("Search Strategy", "Source-Based: Full report with inline citations. Focused Quick: Concise Q&A. Focused Comprehensive: Detailed answer with citations. Topic Organization: Groups results by theme.") }}</label><select id="strategy" name="strategy" class="ldr-form-control">{% for s in strategies %}@@ -260,7 +324,6 @@</select><span class="ldr-input-help">Focused Iteration works best with >16,000 context window</span></div>-</div></div>diff --git a/src/local_deep_research/web/warning_checks/__init__.py b/src/local_deep_research/web/warning_checks/__init__.pyindex 01b41292b6..204b1745fc 100644--- a/src/local_deep_research/web/warning_checks/__init__.py+++ b/src/local_deep_research/web/warning_checks/__init__.py@@ -26,6 +26,12 @@ from .hardware import (check_legacy_server_config,check_model_mismatch,)+from ...security.egress.warnings import (+ check_cloud_embeddings_enabled,+ check_cloud_llm_enabled,+ check_effective_scope,+ check_public_egress_enabled,+)def _safe_check(check_fn, *args, **kwargs):@@ -111,6 +117,130 @@ def calculate_warnings() -> List[dict]:if w:warnings.append(w)+ # --- Egress policy checks ---+ egress_scope = settings_manager.get_setting(+ "policy.egress_scope", "both"+ )+ require_local_endpoint = bool(+ settings_manager.get_setting(+ "llm.require_local_endpoint", False+ )+ )+ embeddings_provider = settings_manager.get_setting(+ "embeddings.provider", ""+ )+ embeddings_base_url = settings_manager.get_setting(+ "embeddings.openai.base_url", ""+ )+ require_local_embeddings = bool(+ settings_manager.get_setting("embeddings.require_local", False)+ )+ primary_engine = settings_manager.get_setting("search.tool", "auto")++ # Resolve the EFFECTIVE posture so the banners are accurate. For+ # `adaptive`, this turns the opaque "follows the primary" into a+ # concrete scope; it also applies the PRIVATE_ONLY -> force-local+ # coupling, so a private-resolving run doesn't falsely show the+ # "cloud LLM enabled" banner. Best-effort: any failure falls back+ # to the raw values (the page must never break on this).+ effective_scope = str(egress_scope).lower()+ effective_require_local_endpoint = require_local_endpoint+ effective_require_local_embeddings = require_local_embeddings+ try:+ from ...security.egress.policy import context_from_snapshot++ _snap = settings_manager.get_settings_snapshot()+ if isinstance(_snap, dict):+ # allow_dns=False: this runs on the /api/warnings page-+ # render hot path; skip the synchronous getaddrinfo that+ # ADAPTIVE resolution would otherwise do for a URL-engine+ # primary (could block the render up to _DNS_TIMEOUT_SEC).+ # The banner is advisory and falls back to static+ # classification — accuracy here is best-effort by design.+ _eff_ctx = context_from_snapshot(+ _snap,+ primary_engine or "auto",+ username=username,+ allow_dns=False,+ )+ effective_scope = _eff_ctx.scope.value+ effective_require_local_endpoint = (+ _eff_ctx.require_local_llm+ )+ effective_require_local_embeddings = (+ _eff_ctx.require_local_embeddings+ )+ except Exception:+ logger.debug(+ "could not resolve effective egress scope for warnings",+ exc_info=True,+ )++ adaptive_info_dismissed = bool(+ settings_manager.get_setting(+ "app.warnings.dismiss_adaptive_scope_info", False+ )+ )++ # Each egress banner has its OWN dismiss flag. Previously all+ # three shared app.warnings.dismiss_egress_policy, so dismissing+ # the fresh-install "public egress" notice ALSO permanently hid+ # the critical cloud-LLM / cloud-embeddings warnings — a+ # false-safety trap (switch to OpenAI later, never warned).+ public_egress_dismissed = bool(+ settings_manager.get_setting(+ "app.warnings.dismiss_egress_policy", False+ )+ )+ cloud_llm_dismissed = bool(+ settings_manager.get_setting(+ "app.warnings.dismiss_cloud_llm", False+ )+ )+ cloud_embeddings_dismissed = bool(+ settings_manager.get_setting(+ "app.warnings.dismiss_cloud_embeddings", False+ )+ )++ # Informational: state what ADAPTIVE actually resolves to.+ w = _safe_check(+ check_effective_scope,+ egress_scope,+ effective_scope,+ primary_engine,+ adaptive_info_dismissed,+ )+ if w:+ warnings.append(w)++ w = _safe_check(+ check_public_egress_enabled,+ effective_scope,+ public_egress_dismissed,+ )+ if w:+ warnings.append(w)++ w = _safe_check(+ check_cloud_llm_enabled,+ provider,+ effective_require_local_endpoint,+ cloud_llm_dismissed,+ )+ if w:+ warnings.append(w)++ w = _safe_check(+ check_cloud_embeddings_enabled,+ embeddings_provider,+ embeddings_base_url,+ effective_require_local_embeddings,+ cloud_embeddings_dismissed,+ )+ if w:+ warnings.append(w)+# --- Backup checks ---w = _safe_check(check_backup_disabled, backup_enabled, dismiss_backup_disableddiff --git a/src/local_deep_research/web_search_engines/engines/full_search.py b/src/local_deep_research/web_search_engines/engines/full_search.pyindex 4f535b51c9..108914ccda 100644--- a/src/local_deep_research/web_search_engines/engines/full_search.py+++ b/src/local_deep_research/web_search_engines/engines/full_search.py@@ -8,7 +8,8 @@ from ...config.search_config import QUALITY_CHECK_DDG_URLSfrom ...research_library.downloaders.extraction import (batch_fetch_and_extract,)-from ...security.ssrf_validator import validate_url+from ...security.egress.policy import PolicyDeniedError+from ...security.ssrf_validator import redact_url_for_log, validate_urlfrom ...utilities.js_rendering import (read_js_rendering_setting as _read_js_rendering_setting,)@@ -32,6 +33,7 @@ class FullSearchResults:time: Optional[str] = "y",safesearch: str | int = "Moderate",settings_snapshot: Optional[Dict] = None,+ egress_context: Optional[Any] = None,):self.llm = llmself.output_format = output_format@@ -42,6 +44,9 @@ class FullSearchResults:self.safesearch = safesearchself.web_search = web_searchself.settings_snapshot = settings_snapshot+ # Set by the factory when the parent engine is gated against a+ # specific scope; used to evaluate per-URL fetches below.+ self.egress_context = egress_contextdef check_urls(self, results: List[Dict], query: str) -> List[Dict]:if not results:@@ -73,6 +78,12 @@ class FullSearchResults:good_indices = []return [r for i, r in enumerate(results) if i in good_indices]+ except PolicyDeniedError:+ # The URL-quality LLM was denied by egress policy (e.g. a cloud+ # LLM under require_local / PRIVATE_ONLY). Do NOT fall through to+ # the unfiltered-results fallback below — that would let the run+ # proceed despite the user's policy refusing the LLM. Fail closed.+ raiseexcept Exception:logger.exception("URL filtering error")logger.warning(@@ -105,17 +116,36 @@ class FullSearchResults:logger.error("\n === NO VALID LINKS ===\n")return []- # SSRF-validate URLs+ # SSRF-validate + egress-scope-validate each URL. SSRF only checks+ # IP class; scope enforcement is the separate axis that blocks+ # public hosts under PRIVATE_ONLY / STRICT.+ evaluate_url_fn = None+ egress_ctx = self.egress_context+ if egress_ctx is not None:+ from ...security.egress.policy import evaluate_url as _ev_url++ evaluate_url_fn = _ev_urlsafe_urls: List[str] = []for url in urls:- if url is not None and validate_url(url):- safe_urls.append(url)- else:+ if url is None:+ continue+ if not validate_url(url):logger.warning(- f"SSRF validation blocked URL from full content fetch: {url}. "- "If this is a trusted internal/private resource, note that "- "full content fetching currently only supports public URLs."+ "SSRF validation blocked URL from full content fetch: "+ f"{redact_url_for_log(url)}.")+ continue+ if evaluate_url_fn is not None and egress_ctx is not None:+ url_decision = evaluate_url_fn(url, egress_ctx)+ if not url_decision.allowed:+ logger.bind(policy_audit=True).warning(+ "full_search URL denied by egress policy",+ url=redact_url_for_log(url),+ scope=egress_ctx.scope.value,+ reason=url_decision.reason,+ )+ continue+ safe_urls.append(url)if not safe_urls:logger.warning(@@ -149,15 +179,34 @@ class FullSearchResults:self, relevant_items: List[Dict[str, Any]]) -> List[Dict[str, Any]]:"""Fetch and attach full content to an existing list of items."""+ evaluate_url_fn = None+ egress_ctx = self.egress_context+ if egress_ctx is not None:+ from ...security.egress.policy import evaluate_url as _ev_url++ evaluate_url_fn = _ev_urlurls: List[str] = []for item in relevant_items:link = item.get("link")- if link is not None and validate_url(link):- urls.append(link)- elif link is not None:+ if link is None:+ continue+ if not validate_url(link):logger.warning(- f"SSRF validation blocked URL from full content fetch: {link}."+ "SSRF validation blocked URL from full content fetch: "+ f"{redact_url_for_log(link)}.")+ continue+ if evaluate_url_fn is not None and egress_ctx is not None:+ url_decision = evaluate_url_fn(link, egress_ctx)+ if not url_decision.allowed:+ logger.bind(policy_audit=True).warning(+ "full_search _get_full_content URL denied by egress policy",+ url=redact_url_for_log(link),+ scope=egress_ctx.scope.value,+ reason=url_decision.reason,+ )+ continue+ urls.append(link)if not urls:for item in relevant_items:diff --git a/src/local_deep_research/web_search_engines/engines/local_embedding_manager.py b/src/local_deep_research/web_search_engines/engines/local_embedding_manager.pyindex 4ac654fa17..15af4e55b0 100644--- a/src/local_deep_research/web_search_engines/engines/local_embedding_manager.py+++ b/src/local_deep_research/web_search_engines/engines/local_embedding_manager.py@@ -4,9 +4,6 @@ import uuidfrom datetime import UTC, datetimefrom typing import Any, Dict, List, Optional-from langchain_community.embeddings import (- HuggingFaceEmbeddings,-)from langchain_core.documents import Documentfrom loguru import logger@@ -138,13 +135,30 @@ class LocalEmbeddingManager:settings_snapshot=self.settings_snapshot,**kwargs,)- except Exception:- logger.exception("Error initializing embeddings")- logger.warning(- "Falling back to HuggingFaceEmbeddings with all-MiniLM-L6-v2"+ except ImportError as exc:+ # Only fall back when the configured provider's dependency+ # genuinely isn't installed — that's a deployment shape, not+ # a transient runtime error. Any OTHER exception (Ollama+ # DNS hiccup, provider validation, policy denial) must+ # propagate so we don't silently fetch from huggingface.co+ # when the user has explicitly opted into local embeddings.+ logger.exception(+ "Embedding provider import failed — falling back to local SBERT",+ exc_info=exc,)- return HuggingFaceEmbeddings(- model_name="sentence-transformers/all-MiniLM-L6-v2"+ # Route the fallback through get_embeddings(sentence_transformers)+ # rather than constructing HuggingFaceEmbeddings directly: the SBERT+ # model is fetched from huggingface.co on a cache miss, which the+ # provider gate refuses under PRIVATE_ONLY / embeddings.require_local.+ # Constructing it raw here would bypass that gate and leak an+ # outbound HF download in offline mode. PolicyDeniedError from the+ # gate propagates (fail closed).+ from ...embeddings import get_embeddings++ return get_embeddings(+ provider="sentence_transformers",+ model=None, # provider default (all-MiniLM-L6-v2)+ settings_snapshot=self.settings_snapshot,)def _store_chunks_to_db(diff --git a/src/local_deep_research/web_search_engines/engines/meta_search_engine.py b/src/local_deep_research/web_search_engines/engines/meta_search_engine.pyindex 8122c3bfee..f8207f3b95 100644--- a/src/local_deep_research/web_search_engines/engines/meta_search_engine.py+++ b/src/local_deep_research/web_search_engines/engines/meta_search_engine.py@@ -3,6 +3,7 @@ from typing import Any, Dict, List, Optionalfrom loguru import loggerfrom ...config.search_config import get_setting_from_snapshot+from ...security.egress.policy import PolicyDeniedErrorfrom ...utilities.thread_context import get_search_contextfrom ...web.services.socket_service import SocketIOServicefrom ..search_engine_base import BaseSearchEngine@@ -64,13 +65,78 @@ class MetaSearchEngine(BaseSearchEngine):f"Meta Search Engine initialized with {len(self.available_engines)} available engines: {', '.join(self.available_engines)}")- # Create a fallback engine in case everything else fails- self.fallback_engine = WikipediaSearchEngine(- max_results=self.max_results,- llm=llm,- max_filtered_results=max_filtered_results,+ # Wikipedia fallback. Gated by the egress policy: under STRICT+ # (or any scope that disallows the wikipedia engine) we don't+ # construct a fallback at all, so the meta engine surfaces an+ # empty result rather than silently bypassing policy. When the+ # policy permits it (or can't be determined), construct as+ # before. ``fallback_engine`` stays a plain attribute so callers+ # / tests can reassign it.+ self.fallback_engine: Optional[BaseSearchEngine] = None+ if self._wikipedia_fallback_allowed():+ self.fallback_engine = WikipediaSearchEngine(+ max_results=self.max_results,+ llm=llm,+ max_filtered_results=max_filtered_results,+ )++ def _wikipedia_fallback_allowed(self) -> bool:+ """Return True if the egress policy permits the wikipedia+ fallback engine. Fail closed: any policy-related error → False.++ Previously this swallowed every Exception and returned True,+ which let a corrupted policy.egress_scope (PolicyDeniedError out+ of context_from_snapshot) construct WikipediaSearchEngine+ directly, bypassing the factory PEP. See plan C3.+ """+ from ...security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_engine,)+ if not self.settings_snapshot:+ # No snapshot — the meta engine's children will be gated by+ # the factory PEP. Allow the fallback in this offline path.+ return True+ try:+ primary = get_setting_from_snapshot(+ "search.tool",+ "auto",+ settings_snapshot=self.settings_snapshot,+ )+ ctx = context_from_snapshot(+ self.settings_snapshot, primary or "auto"+ )+ except PolicyDeniedError as exc:+ logger.bind(policy_audit=True).warning(+ "wikipedia fallback denied: policy context build failed",+ reason=exc.decision.reason,+ )+ return False+ except ValueError as exc:+ # context_from_snapshot raises ValueError on cross-field+ # invariant violations (e.g. STRICT + meta-picker primary).+ # ValueError has no .decision, so handle it separately — and+ # fail closed (no fallback) rather than let it propagate.+ logger.bind(policy_audit=True).warning(+ "wikipedia fallback denied: invalid policy config",+ error=str(exc),+ )+ return False+ decision = evaluate_engine(+ "wikipedia",+ ctx,+ settings_snapshot=self.settings_snapshot,+ )+ if not decision.allowed:+ logger.bind(policy_audit=True).info(+ "wikipedia fallback disabled by egress policy",+ reason=decision.reason,+ )+ return False+ return True+def _get_search_config(self) -> Dict[str, Any]:"""Get search config for available engines (used for reliability/strengths lookups)."""return get_available_engines(@@ -311,7 +377,12 @@ Example output: searxng,wikipedia,brave"""logger.warning("No suitable search engines found for query, using fallback engine")- return self.fallback_engine._get_previews(query)+ fallback = self.fallback_engine+ if fallback is None:+ # Egress policy disallowed the wikipedia fallback; surface+ # an empty result set instead of silently bypassing.+ return []+ return fallback._get_previews(query)# Limit the number of engines to tryengines_to_try = ranked_engines[: self.max_engines_to_try]@@ -386,10 +457,16 @@ Example output: searxng,wikipedia,brave"""logger.warning(f"All engines failed or returned no preview results: {', '.join(all_errors)}")+ fallback = self.fallback_engine+ if fallback is None:+ # Under STRICT/PRIVATE_ONLY the wikipedia fallback is not+ # permitted; surface the failure to the caller instead of+ # silently bypassing policy.+ return []logger.info("Using fallback Wikipedia engine for previews")- self._selected_engine = self.fallback_engine+ self._selected_engine = fallbackself._selected_engine_name = "wikipedia"- return self.fallback_engine._get_previews(query)+ return fallback._get_previews(query)def _get_full_content(self, relevant_items: List[Dict[str, Any]]@@ -459,6 +536,18 @@ Example output: searxng,wikipedia,brave"""programmatic_mode=self.programmatic_mode,**common_params, # type: ignore[arg-type])+ except PolicyDeniedError as exc:+ # A child engine was refused by egress policy (scope mismatch or+ # a corrupt policy config). Skipping it is fail-closed — the+ # forbidden engine never runs — but log with policy_audit so the+ # denial isn't silent (the broad except below would hide it) and+ # the operator can see which engine the scope filtered out.+ logger.bind(policy_audit=True).warning(+ "meta engine: child engine '{}' denied by egress policy ({})",+ engine_name,+ exc.decision.reason,+ )+ return Noneexcept Exception:logger.exception(f"Error creating engine instance for {engine_name}"diff --git a/src/local_deep_research/web_search_engines/engines/parallel_search_engine.py b/src/local_deep_research/web_search_engines/engines/parallel_search_engine.pyindex 29f0972e2a..75f3eeb157 100644--- a/src/local_deep_research/web_search_engines/engines/parallel_search_engine.py+++ b/src/local_deep_research/web_search_engines/engines/parallel_search_engine.py@@ -8,6 +8,7 @@ from loguru import loggerfrom ...config.search_config import get_setting_from_snapshotfrom ...database.thread_local_session import cleanup_current_thread+from ...security.egress.policy import PolicyDeniedErrorfrom ...utilities.enums import SearchModefrom ...utilities.json_utils import extract_json, get_llm_response_textfrom ...utilities.thread_context import (@@ -427,6 +428,17 @@ Select up to {self.max_engines_to_select} engines that would best answer this qu# Individual engines use their auto-detected filtering settings# The factory enables LLM filtering for scientific engines (arXiv, etc.)# and disables it for generic engines (Google, Brave, etc.)+ except PolicyDeniedError as exc:+ # Child engine refused by egress policy. Skipping is+ # fail-closed (the forbidden engine never runs); log with+ # policy_audit so the denial isn't silent.+ logger.bind(policy_audit=True).warning(+ "parallel engine: child engine '{}' denied by egress "+ "policy ({})",+ engine_name,+ exc.decision.reason,+ )+ return Noneexcept Exception:logger.exception(f"Error creating engine instance for {engine_name}"@@ -542,13 +554,37 @@ Select up to {self.max_engines_to_select} engines that would best answer this qu# state) in a finally block so nothing leaks to the next task that# lands on the same worker.submitter_ctx = get_search_context()+ # Also capture the egress audit-hook context: threading.local is NOT+ # inherited by pool workers, so the PEP-578 socket backstop would be+ # inactive on the worker that runs the actual search/LLM sockets+ # unless we re-arm it per task (mirrors the search-context handling).+ try:+ from ...security.egress.audit_hook import (+ get_active_context as _get_egress_ctx,+ )++ submitter_egress_ctx = _get_egress_ctx()+ except Exception:+ submitter_egress_ctx = Nonedef _run_with_context(fn, *fargs):if submitter_ctx is not None:set_search_context(submitter_ctx)+ if submitter_egress_ctx is not None:+ from ...security.egress.audit_hook import (+ set_active_context as _set_egress_ctx,+ )++ _set_egress_ctx(submitter_egress_ctx)try:return fn(*fargs)finally:+ if submitter_egress_ctx is not None:+ from ...security.egress.audit_hook import (+ clear_active_context as _clear_egress_ctx,+ )++ _clear_egress_ctx()clear_search_context()try:cleanup_current_thread()diff --git a/src/local_deep_research/web_search_engines/engines/search_engine_elasticsearch.py b/src/local_deep_research/web_search_engines/engines/search_engine_elasticsearch.pyindex 20322770e9..2be34d295e 100644--- a/src/local_deep_research/web_search_engines/engines/search_engine_elasticsearch.py+++ b/src/local_deep_research/web_search_engines/engines/search_engine_elasticsearch.py@@ -15,6 +15,53 @@ class ElasticsearchSearchEngine(BaseSearchEngine):is_local = Trueis_lexical = Trueneeds_llm_relevance_filter = True+ # Dynamic URL classification so a user who reconfigures hosts to a public+ # Elastic Cloud endpoint isn't still treated as local. The PDP supports+ # list-typed settings ("any public wins"). cloud_id (which is NOT a host+ # the PDP can classify) is handled separately in __init__: it is rejected+ # when the effective scope forbids public egress.+ url_setting = "search.engine.web.elasticsearch.default_params.hosts"++ @staticmethod+ def _cloud_id_forbidden_by_scope(+ settings_snapshot: Optional[Dict[str, Any]],+ ) -> bool:+ """True when the effective egress scope forbids the public Elastic+ Cloud endpoint a ``cloud_id`` targets.++ Resolves the scope (including ADAPTIVE) via ``context_from_snapshot``+ and returns True for PRIVATE_ONLY / STRICT. Fails CLOSED (forbidden)+ if the policy cannot be evaluated, so a snapshot/policy error cannot+ open a cloud egress under a private posture. A missing/empty snapshot+ resolves to the permissive default (BOTH) and is allowed.+ """+ try:+ from ...security.egress.policy import (+ EgressScope,+ context_from_snapshot,+ )+ from ...config.thread_settings import get_setting_from_snapshot++ snapshot = settings_snapshot or {}+ primary = (+ get_setting_from_snapshot(+ "search.tool",+ default="auto",+ settings_snapshot=snapshot,+ )+ or "auto"+ )+ ctx = context_from_snapshot(snapshot, primary)+ return ctx.scope in (+ EgressScope.PRIVATE_ONLY,+ EgressScope.STRICT,+ )+ except Exception:+ logger.bind(policy_audit=True).warning(+ "elasticsearch cloud_id egress check failed; failing closed",+ exc_info=True,+ )+ return Truedef __init__(self,@@ -82,6 +129,25 @@ class ElasticsearchSearchEngine(BaseSearchEngine):# Cloud ID for Elastic Cloudif cloud_id:+ # Egress policy: a cloud_id always targets a public Elastic Cloud+ # endpoint (*.cloud.es.io), but the url_setting reclassification+ # only inspects `hosts`. A cloud_id-only config would otherwise+ # keep the engine's static is_local=True and slip past+ # evaluate_engine, then connect at self.client.info() below. Reject+ # it when the effective scope forbids public egress (fail closed).+ if self._cloud_id_forbidden_by_scope(settings_snapshot):+ from ...security.egress.policy import (+ Decision,+ PolicyDeniedError,+ )++ logger.bind(policy_audit=True).warning(+ "refusing Elasticsearch cloud_id under private egress scope"+ )+ raise PolicyDeniedError(+ Decision(False, "elasticsearch_cloud_id_public_egress"),+ target="search_engine:elasticsearch",+ )es_args["cloud_id"] = cloud_id# Connect to Elasticsearchdiff --git a/src/local_deep_research/web_search_engines/engines/search_engine_github.py b/src/local_deep_research/web_search_engines/engines/search_engine_github.pyindex 02456406a0..dbc6a2f117 100644--- a/src/local_deep_research/web_search_engines/engines/search_engine_github.py+++ b/src/local_deep_research/web_search_engines/engines/search_engine_github.py@@ -23,6 +23,8 @@ class GitHubSearchEngine(BaseSearchEngine):Provides search across GitHub repositories, code, issues, and users."""+ is_public = True+ is_code = Trueis_lexical = Trueneeds_llm_relevance_filter = Truediff --git a/src/local_deep_research/web_search_engines/engines/search_engine_paperless.py b/src/local_deep_research/web_search_engines/engines/search_engine_paperless.pyindex fb712998d7..7939a20d7d 100644--- a/src/local_deep_research/web_search_engines/engines/search_engine_paperless.py+++ b/src/local_deep_research/web_search_engines/engines/search_engine_paperless.py@@ -21,8 +21,14 @@ from ...security import safe_getclass PaperlessSearchEngine(BaseSearchEngine):"""Paperless-ngx search engine implementation with full LDR integration."""+ is_local = Trueis_lexical = Trueneeds_llm_relevance_filter = True+ # Dynamic URL classification: the configured api_url is the source of truth.+ # If a user points paperless at a remote host, the egress policy will+ # reclassify it via DNS resolution instead of trusting the static+ # is_local flag.+ url_setting = "search.engine.web.paperless.default_params.api_url"# Class constants for magic numbersMAX_SNIPPET_LENGTH = 3000 # Reasonable limit to avoid context window issuesdiff --git a/src/local_deep_research/web_search_engines/engines/search_engine_searxng.py b/src/local_deep_research/web_search_engines/engines/search_engine_searxng.pyindex 204264de68..ec0ca440d5 100644--- a/src/local_deep_research/web_search_engines/engines/search_engine_searxng.py+++ b/src/local_deep_research/web_search_engines/engines/search_engine_searxng.py@@ -34,6 +34,10 @@ class SearXNGSearchEngine(BaseSearchEngine):is_public = True# Mark as generic search engine (general web search)is_generic = True+ # Dynamic URL classification: self-hosted SearXNG on localhost should be+ # treated as private under PRIVATE_ONLY scope. The egress policy reads+ # the live instance_url and reclassifies via DNS resolution.+ url_setting = "search.engine.web.searxng.default_params.instance_url"@staticmethoddef _normalize_list(value):diff --git a/src/local_deep_research/web_search_engines/retriever_registry.py b/src/local_deep_research/web_search_engines/retriever_registry.pyindex 8e3ae577ea..19f178e06f 100644--- a/src/local_deep_research/web_search_engines/retriever_registry.py+++ b/src/local_deep_research/web_search_engines/retriever_registry.py@@ -19,34 +19,82 @@ class RetrieverRegistry:def __init__(self):self._retrievers: Dict[str, BaseRetriever] = {}+ # Parallel map of name -> classification metadata, e.g.+ # {"is_local": True}. Kept separate from _retrievers so get()+ # keeps returning the bare retriever object (callers + tests+ # depend on `get() is retriever`).+ self._metadata: Dict[str, dict] = {}self._lock = Lock()- def register(self, name: str, retriever: BaseRetriever) -> None:+ def register(+ self,+ name: str,+ retriever: BaseRetriever,+ is_local: bool = True,+ ) -> None:"""Register a retriever with a given name.Args:name: Name to register the retriever underretriever: LangChain BaseRetriever instance+ is_local: Whether this retriever reads local/private data+ (a user's vector store / KB) rather than the public+ internet. Defaults to True because programmatically+ registered retrievers are almost always private corpora;+ the egress policy uses this to decide whether the+ retriever may run under PUBLIC_ONLY / PRIVATE_ONLY scopes."""with self._lock:self._retrievers[name] = retriever+ self._metadata[name] = {"is_local": is_local}logger.info(- f"Registered retriever '{name}' of type {type(retriever).__name__}"+ f"Registered retriever '{name}' of type {type(retriever).__name__} "+ f"(is_local={is_local})")- def register_multiple(self, retrievers: Dict[str, BaseRetriever]) -> None:+ def register_multiple(+ self,+ retrievers: Dict[str, "BaseRetriever | dict"],+ is_local: bool = True,+ ) -> None:"""Register multiple retrievers at once.Args:- retrievers: Dictionary of {name: retriever} pairs+ retrievers: Either ``{name: retriever}`` (uses ``is_local``+ arg as a single default) OR a richer+ ``{name: {"retriever": …, "is_local": bool}}`` mapping+ so different retrievers can carry different+ classifications in one call. Mixing both shapes inside+ the same dict is allowed.+ is_local: Default classification when an entry uses the+ bare-retriever shape. The public programmatic API+ (api/research_functions.py) calls this; we default to+ True here ONLY because the old API contract was True.+ CLI / programmatic callers passing a cloud-backed+ retriever MUST use the dict shape with+ ``is_local=False`` or they will egress under+ PRIVATE_ONLY (silent classification lie)."""with self._lock:- for name, retriever in retrievers.items():+ for name, entry in retrievers.items():+ if isinstance(entry, dict):+ retriever = entry.get("retriever")+ entry_is_local = bool(entry.get("is_local", is_local))+ else:+ retriever = entry+ entry_is_local = is_local+ if retriever is None:+ logger.warning(+ f"Retriever entry '{name}' has no retriever — skipped"+ )+ continueself._retrievers[name] = retriever+ self._metadata[name] = {"is_local": entry_is_local}logger.info(- f"Registered retriever '{name}' of type {type(retriever).__name__}"+ f"Registered retriever '{name}' of type {type(retriever).__name__} "+ f"(is_local={entry_is_local})")def get(self, name: str) -> Optional[BaseRetriever]:@@ -72,6 +120,7 @@ class RetrieverRegistry:with self._lock:if name in self._retrievers:del self._retrievers[name]+ self._metadata.pop(name, None)logger.info(f"Unregistered retriever '{name}'")def clear(self) -> None:@@ -79,6 +128,7 @@ class RetrieverRegistry:with self._lock:count = len(self._retrievers)self._retrievers.clear()+ self._metadata.clear()logger.info(f"Cleared {count} registered retrievers")def is_registered(self, name: str) -> bool:@@ -104,6 +154,29 @@ class RetrieverRegistry:with self._lock:return list(self._retrievers.keys())+ def get_metadata(+ self, name: str, username: Optional[str] = None+ ) -> Optional[Dict]:+ """Return policy-relevant metadata for a registered retriever.++ Returns a dict like ``{"is_local": True}`` for a registered+ retriever, or ``None`` when the retriever is unknown. A+ registered retriever with no recorded classification yields an+ empty dict, which ``evaluate_retriever`` treats as "unclassified"+ and fails closed under any non-BOTH scope.++ The egress policy's ``evaluate_retriever`` consults this hook to+ decide whether a retriever may run under the active scope.++ ``username`` is accepted so future per-user retriever isolation+ can plug in without changing the call sites.+ """+ with self._lock:+ if name not in self._retrievers:+ return None+ # Copy so callers can't mutate the stored classification.+ return dict(self._metadata.get(name, {}))+# Global registry instanceretriever_registry = RetrieverRegistry()diff --git a/src/local_deep_research/web_search_engines/search_engine_base.py b/src/local_deep_research/web_search_engines/search_engine_base.pyindex dd032829d9..c5e29260c2 100644--- a/src/local_deep_research/web_search_engines/search_engine_base.py+++ b/src/local_deep_research/web_search_engines/search_engine_base.py@@ -905,6 +905,50 @@ class BaseSearchEngine(ABC):return []return [self._extract_full_result(item) for item in relevant_items]+ def _build_full_search_egress_context(self):+ """Build an EgressContext from self.settings_snapshot for the+ per-URL scope gate inside FullSearchResults. Returns None when a+ context can't be built and disables full-content fetching when+ policy evaluation fails on a supplied snapshot.++ The previous "fail closed by returning None" comment was wrong:+ full_search.py treats ``egress_context is None`` as+ ``evaluate_url_fn = None`` and skips the per-URL scope check+ entirely, leaving only SSRF — which allows public hosts under+ PRIVATE_ONLY. When the snapshot was supplied but the policy is+ unevaluable, we now turn full-content fetching off for this+ engine instead.+ """+ if not getattr(self, "settings_snapshot", None):+ return None+ try:+ from ..security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ )+ except ImportError:+ logger.debug(+ "egress_policy unavailable in search_engine_base; "+ "FullSearchResults fetches will be SSRF-only"+ )+ return None++ primary_raw = self.settings_snapshot.get("search.tool", "auto")+ if isinstance(primary_raw, dict) and "value" in primary_raw:+ primary_raw = primary_raw["value"]+ try:+ return context_from_snapshot(+ self.settings_snapshot, primary_raw or "auto"+ )+ except (PolicyDeniedError, ValueError) as exc:+ logger.bind(policy_audit=True).warning(+ "Disabling include_full_content: egress policy could "+ "not be evaluated for this engine",+ reason=str(exc),+ )+ self.include_full_content = False+ return None+def _init_full_search(self,web_search=None,@@ -930,6 +974,11 @@ class BaseSearchEngine(ABC):try:from .engines.full_search import FullSearchResults+ # Egress context for per-URL scope gating. Built once+ # here (not per fetch) so the policy snapshot is the+ # same one used by the factory PEP that authorized this+ # engine.+ egress_context = self._build_full_search_egress_context()self.full_search = FullSearchResults(llm=self.llm,web_search=web_search,@@ -939,6 +988,7 @@ class BaseSearchEngine(ABC):time=time_period,safesearch=safe_search,settings_snapshot=self.settings_snapshot,+ egress_context=egress_context,)except ImportError:logger.warning(diff --git a/src/local_deep_research/web_search_engines/search_engine_factory.py b/src/local_deep_research/web_search_engines/search_engine_factory.pyindex 1fb80c560f..e0837579b5 100644--- a/src/local_deep_research/web_search_engines/search_engine_factory.py+++ b/src/local_deep_research/web_search_engines/search_engine_factory.py@@ -3,6 +3,7 @@ from typing import Any, Dict, Optionalfrom loguru import logger+from ..security.egress.policy import PolicyDeniedErrorfrom ..security.module_whitelist import get_safe_module_classfrom ..utilities.enums import SearchModefrom .retriever_registry import retriever_registry@@ -35,6 +36,11 @@ def create_search_engine(f"create_search_engine called with engine_name={engine_name} (type: {type(engine_name)})")+ # Egress policy PEP runs AFTER retriever-registry + config lookup, so+ # registered retrievers and meta-pickers (handled separately below)+ # don't go through engine-name classification. The actual call site+ # is right before engine instantiation, further down.+# Handle special parallel search engine variantsif engine_name == "parallel_scientific":logger.info("Creating scientific parallel search engine")@@ -62,6 +68,62 @@ def create_search_engine(# Check if this is a registered retriever firstretriever = retriever_registry.get(engine_name)if retriever:+ # Egress policy: gate the retriever against the active scope.+ # Retrievers are classified at registration (is_local, default+ # True). Snapshot REQUIRED — previously the policy block was+ # gated on "if settings_snapshot:" which let snapshot=None+ # callers instantiate retrievers ungated.+ # NB: PolicyDeniedError is imported at MODULE level — importing it+ # again here (function-local) would shadow it across the whole+ # function and make the outer `except PolicyDeniedError` an unbound+ # local on code paths that skip this branch (e.g. meta-pickers).+ from ..security.egress.policy import (+ Decision,+ context_from_snapshot,+ evaluate_retriever,+ )++ if not settings_snapshot:+ raise PolicyDeniedError(+ Decision(False, "no_snapshot"),+ target=engine_name,+ )++ primary_raw = settings_snapshot.get("search.tool", engine_name)+ if isinstance(primary_raw, dict) and "value" in primary_raw:+ primary_raw = primary_raw["value"]+ try:+ _ctx = context_from_snapshot(+ settings_snapshot,+ primary_raw or engine_name,+ username=username,+ )+ except PolicyDeniedError:+ raise+ except ValueError as exc:+ raise PolicyDeniedError(+ Decision(False, "invalid_policy_config"),+ target=engine_name,+ ) from exc+ # Pass metadata from the SAME registry reference used for+ # .get() above, so evaluate_retriever doesn't re-read a+ # different (e.g. test-patched) global singleton.+ try:+ _meta = retriever_registry.get_metadata(+ engine_name, username=username+ )+ except AttributeError:+ _meta = None+ _decision = evaluate_retriever(engine_name, _ctx, metadata=_meta)+ if not _decision.allowed:+ logger.bind(policy_audit=True).warning(+ "retriever denied by egress policy",+ retriever=engine_name,+ scope=_ctx.scope.value,+ reason=_decision.reason,+ )+ raise PolicyDeniedError(_decision, target=engine_name)+logger.info(f"Using registered LangChain retriever: {engine_name}")from .engines.search_engine_retriever import RetrieverSearchEngine@@ -137,19 +199,76 @@ def create_search_engine(engine_name = config_keybreak- # If still not found, use default+ # If still not found, FAIL CLOSED. Previously the factory silently+ # rewrote unknown engine names to "auto", which then matched the+ # local skip-list at line 213 below and bypassed evaluate_engine+ # entirely. Callers that want auto behavior must pass "auto"+ # explicitly. See plan C2 (cryptic-tinkering-iverson.md).+ # We raise a plain ValueError (not PolicyDeniedError) because+ # "unknown engine name" is a config/wiring error, not a policy+ # decision — keeping the exception type clean lets policy-aware+ # callers tell them apart.if engine_name not in config:- logger.warning(- f"Search engine '{engine_name}' not found in config, using default"+ logger.bind(policy_audit=True).warning(+ "unknown engine name rejected (was silently rewritten to 'auto' previously)",+ engine=engine_name,+ available=list(config.keys()),)- # Try to use 'auto' as default if available- if "auto" in config:- engine_name = "auto"- else:- logger.error(- f"No default search engine available. Available engines: {list(config.keys())}"- )- return None+ raise ValueError(+ f"Unknown search engine '{engine_name}'. Available: "+ f"{sorted(config.keys())}. To use auto-selection, "+ "pass engine_name='auto' explicitly."+ )++ # Egress policy PEP: gate the resolved engine against the user's declared+ # scope. Lazy import to break the circular dependency+ # (security/egress/policy.py → web_search_engines/* → security/*).+ # Skipped for engines registered dynamically (`auto`, `meta`, `parallel`,+ # `parallel_scientific` — their children are gated when instantiated).+ if engine_name not in ("auto", "meta", "parallel", "parallel_scientific"):+ # PolicyDeniedError comes from the module-level import (see the note+ # in the retriever branch above — re-importing it here would shadow+ # it function-wide and break the outer except clause).+ from ..security.egress.policy import (+ Decision,+ context_from_snapshot,+ evaluate_engine,+ )++ primary_engine_raw = settings_snapshot.get("search.tool", engine_name)+ if (+ isinstance(primary_engine_raw, dict)+ and "value" in primary_engine_raw+ ):+ primary_engine_raw = primary_engine_raw["value"]+ primary_engine = primary_engine_raw or engine_name++ try:+ ctx = context_from_snapshot(+ settings_snapshot, primary_engine, username=username+ )+ except ValueError as exc:+ raise PolicyDeniedError(+ Decision(False, "invalid_policy_config"),+ target=engine_name,+ ) from exc++ # Pass the engine config entry as metadata so a per-collection+ # is_public classification is honored without a redundant DB lookup.+ decision = evaluate_engine(+ engine_name,+ ctx,+ settings_snapshot=settings_snapshot,+ metadata=config.get(engine_name),+ )+ if not decision.allowed:+ logger.bind(policy_audit=True).warning(+ "engine denied by egress policy",+ engine=engine_name,+ scope=ctx.scope.value,+ reason=decision.reason,+ )+ raise PolicyDeniedError(decision, target=engine_name)# Get engine configurationengine_config = config[engine_name]@@ -362,6 +481,13 @@ def create_search_engine(return engine # type: ignore[no-any-return]+ except PolicyDeniedError:+ # An engine __init__ (or a child engine / full-search wrapper it+ # builds) may itself consult the PDP and raise. Re-raise so the+ # denial surfaces to policy-aware callers and the audit trail,+ # rather than being downgraded to a generic "failed to create"+ # None — which would fail OPEN from an enforcement standpoint.+ raiseexcept Exception:logger.exception(f"Failed to create search engine '{engine_name}'")return Nonediff --git a/src/local_deep_research/web_search_engines/search_engines_config.py b/src/local_deep_research/web_search_engines/search_engines_config.pyindex 1d3e90de6c..fd94a5c54d 100644--- a/src/local_deep_research/web_search_engines/search_engines_config.py+++ b/src/local_deep_research/web_search_engines/search_engines_config.py@@ -222,11 +222,21 @@ def search_config(engine_id = f"collection_{collection.id}"# Add suffix to distinguish from the all-collections searchdisplay_name = f"{collection.name} (Collection)"+ # Egress classification follows the per-collection+ # public/private flag (default private). A "public"+ # collection counts as a public engine (allowed under+ # PUBLIC_ONLY); a private one is local-only. NULL+ # (pre-migration rows) reads as private — the safe+ # default.+ collection_is_public = bool(+ getattr(collection, "is_public", False)+ )search_engines[engine_id] = {"module_path": ".engines.search_engine_collection","class_name": "CollectionSearchEngine","requires_llm": True,- "is_local": True,+ "is_local": not collection_is_public,+ "is_public": collection_is_public,"display_name": display_name,"default_params": {"collection_id": collection.id,diff --git a/tests/advanced_search_system/filters/test_journal_reputation_filter.py b/tests/advanced_search_system/filters/test_journal_reputation_filter.pyindex 11e0c740f0..85081806fa 100644--- a/tests/advanced_search_system/filters/test_journal_reputation_filter.py+++ b/tests/advanced_search_system/filters/test_journal_reputation_filter.py@@ -96,6 +96,85 @@ class TestJournalReputationFilterInit:mock_get_llm.assert_called_once()assert filter_obj.model is mock_default_model+ @patch(+ "local_deep_research.advanced_search_system.filters.journal_reputation_filter.create_search_engine"+ )+ @patch(+ "local_deep_research.advanced_search_system.filters.journal_reputation_filter.get_llm"+ )+ def test_init_forwards_settings_snapshot_to_get_llm(+ self, mock_get_llm, mock_create_engine+ ):+ """When model=None, the snapshot from __init__ must be forwarded+ to get_llm so the LLM PEP at llm_config.py:295 can evaluate+ scope / require_local_endpoint. Dropping it here previously+ bypassed the PEP entirely.+ """+ from local_deep_research.advanced_search_system.filters.journal_reputation_filter import (+ JournalReputationFilter,+ )++ mock_get_llm.return_value = Mock()+ mock_create_engine.return_value = Mock()++ snapshot = {+ "policy.egress_scope": "private_only",+ "llm.provider": "ollama",+ "search.tool": "arxiv",+ }+ JournalReputationFilter(model=None, settings_snapshot=snapshot)++ mock_get_llm.assert_called_once()+ call_kwargs = mock_get_llm.call_args.kwargs+ assert call_kwargs.get("settings_snapshot") is snapshot, (+ "settings_snapshot must be forwarded to get_llm — "+ f"got call_args={mock_get_llm.call_args!r}"+ )++ @patch(+ "local_deep_research.advanced_search_system.filters.journal_reputation_filter.create_search_engine"+ )+ @patch(+ "local_deep_research.advanced_search_system.filters.journal_reputation_filter.get_llm"+ )+ def test_create_default_propagates_policy_denied_error(+ self, mock_get_llm, mock_create_engine+ ):+ """When the snapshot-forwarded get_llm raises PolicyDeniedError+ (cloud provider + require_local_endpoint), create_default must+ let the exception propagate — silently returning None would+ leave unfiltered results through under the very scope the user+ asked for.+ """+ from local_deep_research.advanced_search_system.filters.journal_reputation_filter import (+ JournalReputationFilter,+ )+ from local_deep_research.security.egress.policy import (+ Decision,+ PolicyDeniedError,+ )++ mock_create_engine.return_value = Mock()+ mock_get_llm.side_effect = PolicyDeniedError(+ Decision(False, "provider_cloud_only"), target="openai"+ )++ snapshot = {+ "policy.egress_scope": "both",+ "llm.require_local_endpoint": True,+ "llm.provider": "openai",+ "search.tool": "arxiv",+ "search.engine.web.arxiv.journal_reputation.enabled": True,+ }++ with pytest.raises(PolicyDeniedError) as exc_info:+ JournalReputationFilter.create_default(+ model=None,+ engine_name="arxiv",+ settings_snapshot=snapshot,+ )+ assert exc_info.value.target == "openai"+@patch("local_deep_research.config.search_config.get_setting_from_snapshot")@patch("local_deep_research.advanced_search_system.filters.journal_reputation_filter.create_search_engine"diff --git a/tests/api_tests/test_collections_api.py b/tests/api_tests/test_collections_api.pyindex 9329a9aaea..8a46c49aa8 100644--- a/tests/api_tests/test_collections_api.py+++ b/tests/api_tests/test_collections_api.py@@ -115,6 +115,59 @@ class TestCreateCollection:# Cleanupdelete_test_collection(authenticated_client, data["collection"]["id"])+ def test_create_collection_defaults_private(self, authenticated_client):+ """A new collection with no is_public flag defaults to private."""+ name = f"Priv {uuid.uuid4().hex[:8]}"+ resp = authenticated_client.post(+ "/library/api/collections",+ json={"name": name},+ content_type="application/json",+ )+ assert resp.status_code == 200+ coll = resp.get_json()["collection"]+ assert coll["is_public"] is False+ delete_test_collection(authenticated_client, coll["id"])++ def test_create_and_toggle_collection_is_public(self, authenticated_client):+ """is_public can be set at create and flipped via PUT, and is+ reflected in the list API."""+ name = f"Pub {uuid.uuid4().hex[:8]}"+ resp = authenticated_client.post(+ "/library/api/collections",+ json={"name": name, "is_public": True},+ content_type="application/json",+ )+ assert resp.status_code == 200+ coll = resp.get_json()["collection"]+ cid = coll["id"]+ assert coll["is_public"] is True++ # List reflects it+ listing = authenticated_client.get(+ "/library/api/collections"+ ).get_json()["collections"]+ match = next(c for c in listing if c["id"] == cid)+ assert match["is_public"] is True++ # The collection-details load path (documents endpoint) also exposes+ # is_public so the details-page toggle can reflect current state.+ docs = authenticated_client.get(+ f"/library/api/collections/{cid}/documents"+ )+ assert docs.status_code == 200+ assert docs.get_json()["collection"]["is_public"] is True++ # Flip back to private via PUT+ put = authenticated_client.put(+ f"/library/api/collections/{cid}",+ json={"is_public": False},+ content_type="application/json",+ )+ assert put.status_code == 200+ assert put.get_json()["collection"]["is_public"] is False++ delete_test_collection(authenticated_client, cid)+def test_create_collection_empty_name(self, authenticated_client):"""Rejects collection with empty name."""response = authenticated_client.post(diff --git a/tests/benchmarks/test_graders.py b/tests/benchmarks/test_graders.pyindex 5127c48087..fd91e2cae6 100644--- a/tests/benchmarks/test_graders.py+++ b/tests/benchmarks/test_graders.py@@ -708,3 +708,38 @@ Confidence: 90%assert "[1]" not in result["extracted_answer"]assert "[2]" not in result["extracted_answer"]assert "[5]" not in result["extracted_answer"]+++class TestSettingsSnapshotPropagation:+ """get_evaluation_llm must thread settings_snapshot into get_llm.++ Regression (PR #4300 review): the snapshot was used only to read the+ api_key and never passed to get_llm. Combined with the new snapshot-less+ fail-closed LLM gate, a cloud grader (openai) was refused with+ PolicyDeniedError because get_llm saw no snapshot to evaluate policy.+ """++ @patch("local_deep_research.benchmarks.graders.get_llm")+ def test_snapshot_forwarded_to_get_llm(self, mock_get_llm):+ from local_deep_research.benchmarks.graders import get_evaluation_llm++ mock_get_llm.return_value = Mock()+ snapshot = {"policy.egress_scope": "both", "llm.provider": "openai"}++ get_evaluation_llm(settings_snapshot=snapshot)++ call_kwargs = mock_get_llm.call_args[1]+ assert "settings_snapshot" in call_kwargs, (+ "get_evaluation_llm must forward settings_snapshot to get_llm"+ )+ assert call_kwargs["settings_snapshot"] is snapshot++ @patch("local_deep_research.benchmarks.graders.get_llm")+ def test_none_snapshot_still_forwarded(self, mock_get_llm):+ from local_deep_research.benchmarks.graders import get_evaluation_llm++ mock_get_llm.return_value = Mock()+ get_evaluation_llm()+ call_kwargs = mock_get_llm.call_args[1]+ # Explicitly forwarded (as None) — get_llm's own gate then decides.+ assert call_kwargs.get("settings_snapshot") is Nonediff --git a/tests/config/test_llm_config.py b/tests/config/test_llm_config.pyindex 4dac0ecd36..43bf800cc3 100644--- a/tests/config/test_llm_config.py+++ b/tests/config/test_llm_config.py@@ -426,7 +426,9 @@ class TestGetLlm:"local_deep_research.config.llm_config.get_setting_from_snapshot",return_value="custom_provider",):- result = get_llm(provider="custom_provider")+ result = get_llm(+ provider="custom_provider", settings_snapshot={}+ )assert result is mock_llmdef test_invalid_provider_raises_error(self):@@ -447,7 +449,7 @@ class TestGetLlm:}.get(key, default)with pytest.raises(ValueError, match="Invalid provider"):- get_llm()+ get_llm(settings_snapshot={})def test_raises_when_model_setting_empty(self):"""get_llm() must raise ValueError when llm.model is empty string."""@@ -542,7 +544,7 @@ class TestGetLlm:with patch.object(ChatAnthropic, "__init__", return_value=None) as mock_init:- get_llm(provider="anthropic")+ get_llm(provider="anthropic", settings_snapshot={})mock_init.assert_called_once()# Check it was called with model and api_keycall_kwargs = mock_init.call_args.kwargs@@ -575,7 +577,7 @@ class TestGetLlm:with pytest.raises(ValueError, match="Anthropic API key not configured"):- get_llm(provider="anthropic")+ get_llm(provider="anthropic", settings_snapshot={})def test_openai_provider_creates_chat_openai(self):"""Should create ChatOpenAI when provider is openai."""@@ -608,7 +610,7 @@ class TestGetLlm:with patch.object(ChatOpenAI, "__init__", return_value=None) as mock_init:- get_llm(provider="openai")+ get_llm(provider="openai", settings_snapshot={})mock_init.assert_called_once()call_kwargs = mock_init.call_args.kwargsassert call_kwargs["model"] == "gpt-4"@@ -640,7 +642,7 @@ class TestGetLlm:with pytest.raises(ValueError, match="OpenAI API key not configured"):- get_llm(provider="openai")+ get_llm(provider="openai", settings_snapshot={})def test_openai_endpoint_creates_chat_openai(self):"""Should create ChatOpenAI with custom endpoint when provider is openai_endpoint."""@@ -669,7 +671,7 @@ class TestGetLlm:with patch.object(ChatOpenAI, "__init__", return_value=None) as mock_init:- get_llm(provider="openai_endpoint")+ get_llm(provider="openai_endpoint", settings_snapshot={})mock_init.assert_called_once()call_kwargs = mock_init.call_args.kwargsassert call_kwargs["model"] == "custom-model"@@ -706,7 +708,7 @@ class TestGetLlm:ChatOpenAI, "__init__", return_value=None) as mock_init:# Should not raise even without an API key- get_llm(provider="openai_endpoint")+ get_llm(provider="openai_endpoint", settings_snapshot={})mock_init.assert_called_once()call_kwargs = mock_init.call_args.kwargsassert call_kwargs["model"] == "local-model"@@ -840,6 +842,7 @@ class TestGetLlm:model_name="custom-model",temperature=0.5,provider="custom_provider",+ settings_snapshot={},)mock_factory.assert_called_once()@@ -873,7 +876,7 @@ class TestGetLlm:}.get(key, default)with pytest.raises(TypeError, match="invalid signature"):- get_llm(provider="bad_factory")+ get_llm(provider="bad_factory", settings_snapshot={})def test_custom_factory_returning_non_basechatmodel_raises(self):"""Should raise ValueError when factory returns non-BaseChatModel."""@@ -905,7 +908,7 @@ class TestGetLlm:with pytest.raises(ValueError, match="must return a BaseChatModel"):- get_llm(provider="bad_factory")+ get_llm(provider="bad_factory", settings_snapshot={})def test_context_window_size_for_local_providers(self):"""Should use local context window size for local providers."""@@ -974,7 +977,9 @@ class TestGetLlm:}.get(key, default)result = get_llm(- provider="anthropic", research_context=research_context+ provider="anthropic",+ research_context=research_context,+ settings_snapshot={},)assert hasattr(result, "base_llm")# Original key should still be therediff --git a/tests/config/test_llm_config_extended.py b/tests/config/test_llm_config_extended.pyindex 50bd0897be..f46c455441 100644--- a/tests/config/test_llm_config_extended.py+++ b/tests/config/test_llm_config_extended.py@@ -239,7 +239,7 @@ class TestOpenAIOptionalParams:ChatOpenAI, "__init__", return_value=None) as mock_init,):- get_llm(provider="openai")+ get_llm(provider="openai", settings_snapshot={})call_kwargs = mock_init.call_args.kwargsassert (call_kwargs["openai_api_base"] == "https://custom.endpoint/v1"@@ -266,6 +266,6 @@ class TestOpenAIOptionalParams:ChatOpenAI, "__init__", return_value=None) as mock_init,):- get_llm(provider="openai")+ get_llm(provider="openai", settings_snapshot={})call_kwargs = mock_init.call_args.kwargsassert call_kwargs["openai_organization"] == "org-abc123"diff --git a/tests/content_fetcher/test_fetcher_coverage.py b/tests/content_fetcher/test_fetcher_coverage.pyindex 1a9c587dda..770d373637 100644--- a/tests/content_fetcher/test_fetcher_coverage.py+++ b/tests/content_fetcher/test_fetcher_coverage.py@@ -210,7 +210,7 @@ class TestFetch:"""Tests for ContentFetcher.fetch()."""@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -253,7 +253,7 @@ class TestFetch:# --- SSRF blocked ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=False,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -267,7 +267,7 @@ class TestFetch:# --- Successful UTF-8 fetch ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -290,7 +290,7 @@ class TestFetch:# --- PDF magic bytes -> text extraction success ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -327,7 +327,7 @@ class TestFetch:# --- PDF text extraction failure ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -355,7 +355,7 @@ class TestFetch:# --- UnicodeDecodeError ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -378,7 +378,7 @@ class TestFetch:# --- Content truncation ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -404,7 +404,7 @@ class TestFetch:# --- Metadata extraction ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -432,7 +432,7 @@ class TestFetch:# --- Metadata extraction raises exception (should be silenced) ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -457,7 +457,7 @@ class TestFetch:# --- Download failed (is_success=False) ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -478,7 +478,7 @@ class TestFetch:# --- Download failed with no skip_reason ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -501,7 +501,7 @@ class TestFetch:# --- Downloader throws exception ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -520,7 +520,7 @@ class TestFetch:# --- No downloader and HTML fallback also fails (ImportError) ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -548,7 +548,7 @@ class TestFetch:# --- No downloader but HTML fallback works ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -580,7 +580,7 @@ class TestFetch:# --- Default max_length is applied ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -607,7 +607,7 @@ class TestFetch:# --- prefer_text=False sets PDF content type ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -635,7 +635,7 @@ class TestFetch:# --- is_success True but content is None/empty ---@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -667,7 +667,7 @@ class TestFetch:class TestFetchText:@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -687,7 +687,7 @@ class TestFetchText:assert text == "Text content"@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -713,7 +713,7 @@ class TestFetchText:assert text is None@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -802,7 +802,7 @@ class TestLanguageParam:class TestHtmlFallback:@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -835,7 +835,7 @@ class TestHtmlFallback:assert "arXiv abstract content" in result["content"]@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -864,7 +864,7 @@ class TestHtmlFallback:assert result["status"] == "error"@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -897,7 +897,7 @@ class TestHtmlFallback:class TestFetchBatch:@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")@@ -924,7 +924,7 @@ class TestFetchBatch:assert all(v is not None for v in result.values())@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)@patch("local_deep_research.content_fetcher.fetcher.URLClassifier")diff --git a/tests/content_fetcher/test_mcp_integration.py b/tests/content_fetcher/test_mcp_integration.pyindex cfbcddb211..3162f2aec4 100644--- a/tests/content_fetcher/test_mcp_integration.py+++ b/tests/content_fetcher/test_mcp_integration.py@@ -97,7 +97,7 @@ class TestMCPDownloadContentTool:"local_deep_research.content_fetcher.fetcher.ContentFetcher._get_downloader")@patch(- "local_deep_research.content_fetcher.fetcher.validate_url",+ "local_deep_research.content_fetcher.fetcher.policy_aware_validate_url",return_value=True,)def test_execute_download_content_timeout(diff --git a/tests/content_fetcher/test_security.py b/tests/content_fetcher/test_security.pyindex fc094f17ab..1348291fab 100644--- a/tests/content_fetcher/test_security.py+++ b/tests/content_fetcher/test_security.py@@ -391,3 +391,61 @@ class TestInputValidation:fetcher3 = ContentFetcher(timeout=300)assert fetcher3.timeout == 300+++class TestDownloaderEgressPolicy:+ """The ContentFetcher must relax a downloader's SafeSession to allow+ private IPs under PRIVATE_ONLY, matching policy_aware_validate_url — else+ a private/lab URL the policy already approved is rejected by the+ downloader's own strict SSRF re-validation (R3-14)."""++ @staticmethod+ def _ctx(scope):+ from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+ )++ return EgressContext(+ scope=getattr(EgressScope, scope),+ primary_engine="x",+ require_local_llm=False,+ require_local_embeddings=False,+ )++ @staticmethod+ def _fake_downloader():+ from local_deep_research.security.safe_requests import SafeSession++ class _DL:+ def __init__(self):+ self.session = SafeSession()++ return _DL()++ def test_private_only_relaxes_downloader_session(self):+ cf = ContentFetcher(egress_context=self._ctx("PRIVATE_ONLY"))+ dl = self._fake_downloader()+ cf._apply_egress_policy_to_downloader(dl)+ assert dl.session.allow_private_ips is True++ def test_public_only_keeps_downloader_strict(self):+ cf = ContentFetcher(egress_context=self._ctx("PUBLIC_ONLY"))+ dl = self._fake_downloader()+ cf._apply_egress_policy_to_downloader(dl)+ assert dl.session.allow_private_ips is False++ def test_no_context_keeps_downloader_strict(self):+ cf = ContentFetcher(egress_context=None)+ dl = self._fake_downloader()+ cf._apply_egress_policy_to_downloader(dl)+ assert dl.session.allow_private_ips is False++ def test_downloader_without_session_is_safe(self):+ """A downloader lacking a .session attribute must not raise."""+ cf = ContentFetcher(egress_context=self._ctx("PRIVATE_ONLY"))++ class _NoSession:+ pass++ cf._apply_egress_policy_to_downloader(_NoSession()) # must not raisediff --git a/tests/database/test_alembic_migrations.py b/tests/database/test_alembic_migrations.pyindex c73f7ad65c..e29974485b 100644--- a/tests/database/test_alembic_migrations.py+++ b/tests/database/test_alembic_migrations.py@@ -5482,6 +5482,80 @@ class TestMigrationAppliesSchemaChanges:inspector = inspect(migrated_to_head)assert "_test_migration_probe" not in inspector.get_table_names()+ def test_0011_adds_is_public_to_existing_collections(self, tmp_path):+ """Upgrade path: a DB created BEFORE 0011 (collections without+ is_public) gains the column on upgrade, and a pre-existing collection+ row is backfilled to private (False) via the server_default — existing+ users' collections don't become NULL/garbage or accidentally public.+ """+ from alembic import command++ engine = create_engine(f"sqlite:///{tmp_path / 'pre_0011.db'}")+ try:+ cfg = get_alembic_config(engine)++ # Simulate a pre-0011 DB: a collections table created by an older+ # version of the code (WITHOUT is_public), with the alembic+ # version stamped at 0010 so only 0011 runs. (The migration chain+ # can't reproduce this — 0001 builds from the *current* model,+ # which already has is_public — so we hand-create the old schema,+ # exactly as the 0002 test does.)+ with engine.begin() as conn:+ conn.execute(+ text(+ """+ CREATE TABLE collections (+ id VARCHAR(36) PRIMARY KEY,+ name VARCHAR(255) NOT NULL,+ description TEXT,+ collection_type VARCHAR(50),+ is_default BOOLEAN,+ created_at DATETIME,+ updated_at DATETIME+ )+ """+ )+ )+ conn.execute(+ text(+ "INSERT INTO collections (id, name) "+ "VALUES ('legacy-c1', 'Legacy')"+ )+ )++ with engine.begin() as conn:+ cfg.attributes["connection"] = conn+ command.stamp(cfg, "0010")++ cols = {+ c["name"] for c in inspect(engine).get_columns("collections")+ }+ assert "is_public" not in cols++ # Upgrade to head — 0011 adds is_public.+ with engine.begin() as conn:+ cfg.attributes["connection"] = conn+ command.upgrade(cfg, "head")++ col_map = {+ c["name"]: c+ for c in inspect(engine).get_columns("collections")+ }+ assert "is_public" in col_map+ assert col_map["is_public"]["nullable"] is True++ # The pre-existing row is backfilled to private (falsy).+ with engine.connect() as conn:+ val = conn.execute(+ text(+ "SELECT is_public FROM collections "+ "WHERE id='legacy-c1'"+ )+ ).scalar()+ assert not val, f"existing row not private after upgrade: {val!r}"+ finally:+ engine.dispose()+def test_0002_adds_columns_with_properties_to_old_schema(self, tmp_path):"""Migration 0002 adds progress columns with correct properties to a pre-Alembic schema.diff --git a/tests/embeddings/test_embeddings_config.py b/tests/embeddings/test_embeddings_config.pyindex edb12a0435..185313e010 100644--- a/tests/embeddings/test_embeddings_config.py+++ b/tests/embeddings/test_embeddings_config.py@@ -117,3 +117,76 @@ class TestAvailableProviders:)assert callable(get_available_embedding_providers)+++class TestSnapshotlessEgressGate:+ """get_embeddings() with no settings_snapshot can't read the+ embeddings.require_local toggle, so it fails closed: only the+ localhost-default providers may proceed. A snapshot-less+ get_embeddings(provider="openai") must NOT silently ship the local+ corpus to a cloud embedder.++ Regression (PR #4300 review): the gate was `if settings_snapshot is+ not None:` — the entire check was skipped when snapshot was None,+ so a snapshot-less cloud-provider call proceeded unchecked.++ Source: src/local_deep_research/embeddings/embeddings_config.py+ """++ def test_openai_without_snapshot_is_denied(self):+ from local_deep_research.embeddings.embeddings_config import (+ get_embeddings,+ )+ from local_deep_research.security.egress.policy import (+ PolicyDeniedError,+ )++ with pytest.raises(PolicyDeniedError) as excinfo:+ get_embeddings(provider="openai", settings_snapshot=None)+ assert excinfo.value.decision.reason == "no_snapshot_for_provider"++ def test_local_default_providers_are_allowlisted(self):+ """sentence_transformers and ollama are the localhost-default+ embedding providers — they must pass the snapshot-less gate+ (i.e. NOT raise the no_snapshot_for_provider denial)."""+ from local_deep_research.security.egress.policy import (+ _LOCAL_DEFAULT_EMBEDDING_PROVIDERS,+ )++ assert "sentence_transformers" in _LOCAL_DEFAULT_EMBEDDING_PROVIDERS+ assert "ollama" in _LOCAL_DEFAULT_EMBEDDING_PROVIDERS+ # Cloud providers must NOT be on the allow-list.+ assert "openai" not in _LOCAL_DEFAULT_EMBEDDING_PROVIDERS++ def test_sentence_transformers_without_snapshot_passes_gate(self):+ """The local default must get past the egress gate without a+ snapshot. We patch the provider class so the test asserts the+ POLICY decision, not the heavy model load."""+ from local_deep_research.embeddings.embeddings_config import (+ get_embeddings,+ )+ from local_deep_research.security.egress.policy import (+ PolicyDeniedError,+ )++ sentinel = object()+ fake_provider = Mock()+ fake_provider.return_value.create_embeddings.return_value = sentinel++ with patch(+ "local_deep_research.embeddings.embeddings_config._get_provider_classes",+ return_value={"sentence_transformers": fake_provider},+ ):+ try:+ get_embeddings(+ provider="sentence_transformers",+ settings_snapshot=None,+ )+ except PolicyDeniedError:+ pytest.fail(+ "sentence_transformers must pass the snapshot-less gate"+ )+ except Exception:+ # Any non-policy error means we got PAST the gate, which is+ # all this test asserts.+ passdiff --git a/tests/mcp/test_mcp_strategy.py b/tests/mcp/test_mcp_strategy.pyindex 592a92f823..16befbf319 100644--- a/tests/mcp/test_mcp_strategy.py+++ b/tests/mcp/test_mcp_strategy.py@@ -230,6 +230,58 @@ class TestToolBuilding:assert not any(t["name"] == "web_search" for t in tools)+ def test_strict_scope_exposes_no_specialized_tools(self):+ """Egress parity with LangGraph: under STRICT scope the specialized+ search-engine tools must NOT appear in the tool list (the LLM never+ sees them)."""+ from local_deep_research.advanced_search_system.strategies.mcp_strategy import (+ MCPSearchStrategy,+ )++ strategy = MCPSearchStrategy(+ model=MagicMock(),+ search=MagicMock(),+ include_web_search=True,+ include_sub_research=False,+ settings_snapshot={+ "policy.egress_scope": {"value": "strict"},+ "search.tool": {"value": "arxiv"},+ },+ )+ strategy._get_available_search_engines = lambda: {+ "pubmed": {"description": "PubMed", "is_retriever": False},+ "semantic_scholar": {"description": "S2", "is_retriever": False},+ }+ strategy._get_current_engine_name = lambda: "arxiv"++ tools = strategy._build_tool_descriptions()+ specialized = [t for t in tools if t["name"].startswith("search_")]+ assert specialized == []++ def test_both_scope_exposes_specialized_tools(self):+ """Under BOTH scope the specialized public engines ARE exposed."""+ from local_deep_research.advanced_search_system.strategies.mcp_strategy import (+ MCPSearchStrategy,+ )++ strategy = MCPSearchStrategy(+ model=MagicMock(),+ search=MagicMock(),+ include_web_search=True,+ include_sub_research=False,+ settings_snapshot={+ "policy.egress_scope": {"value": "both"},+ "search.tool": {"value": "arxiv"},+ },+ )+ strategy._get_available_search_engines = lambda: {+ "pubmed": {"description": "PubMed", "is_retriever": False},+ }+ strategy._get_current_engine_name = lambda: "arxiv"++ tool_names = {t["name"] for t in strategy._build_tool_descriptions()}+ assert "search_pubmed" in tool_names+def test_format_tool_descriptions(self):"""Test formatting tool descriptions for the LLM."""from local_deep_research.advanced_search_system.strategies.mcp_strategy import (@@ -712,6 +764,9 @@ class TestMCPToolDiscovery:model=MagicMock(),search=MagicMock(),mcp_servers=[{"name": "test", "command": "nonexistent-command"}],+ # Permissive scope so the policy gate passes and this test+ # exercises the connection-error path (not the scope block).+ settings_snapshot={"policy.egress_scope": "both"},)# The discovery should handle errors gracefully and return empty@@ -732,7 +787,149 @@ class TestMCPToolDiscovery:assert tools == []-class TestStrategyExport:+class TestMCPEgressScopeGate:+ """Tests for the MCP discovery/execution scope gate. MCP tools run+ in stdio subprocesses whose egress the PDP cannot inspect, so the+ only safe enforcement under STRICT/PRIVATE_ONLY (or when no+ snapshot is available) is to block them outright.+ """++ @staticmethod+ def _build(scope_value, *, snapshot_override=None):+ from local_deep_research.advanced_search_system.strategies.mcp_strategy import (+ MCPSearchStrategy,+ )++ if snapshot_override is not None:+ snapshot = snapshot_override+ else:+ snapshot = {"policy.egress_scope": scope_value}+ return MCPSearchStrategy(+ model=MagicMock(),+ search=MagicMock(),+ mcp_servers=[{"name": "srv", "command": "python"}],+ settings_snapshot=snapshot,+ )++ @pytest.mark.parametrize("scope", ["strict", "private_only"])+ def test_discover_mcp_tools_blocked_under_restrictive_scope(self, scope):+ strategy = self._build(scope)+ with patch(+ "local_deep_research.mcp.client.MCPClientManager"+ ) as mock_manager:+ tools = strategy._discover_mcp_tools()+ assert tools == []+ # Connection manager must never be touched — scope gate runs+ # before subprocess startup.+ mock_manager.assert_not_called()+ # Cache the empty list so subsequent ReAct iterations don't+ # re-run the policy work.+ assert strategy._mcp_tools_cache == []++ def test_discover_mcp_tools_blocked_when_no_snapshot(self):+ """``settings_snapshot={}`` (BaseSearchStrategy's "or {}" guard)+ must fail closed — without a snapshot the gate cannot certify+ MCP tools are safe to expose.+ """+ strategy = self._build("ignored", snapshot_override={})+ with patch(+ "local_deep_research.mcp.client.MCPClientManager"+ ) as mock_manager:+ tools = strategy._discover_mcp_tools()+ assert tools == []+ mock_manager.assert_not_called()+ assert strategy._mcp_tools_cache == []++ def test_discover_mcp_tools_blocked_when_policy_unevaluable(self):+ """A corrupted scope value raises PolicyDeniedError inside+ context_from_snapshot — that must fail CLOSED at discovery.+ """+ strategy = self._build(+ "ignored",+ snapshot_override={"policy.egress_scope": "not_a_real_scope"},+ )+ with patch(+ "local_deep_research.mcp.client.MCPClientManager"+ ) as mock_manager:+ tools = strategy._discover_mcp_tools()+ assert tools == []+ mock_manager.assert_not_called()+ assert strategy._mcp_tools_cache == []++ @pytest.mark.parametrize("scope", ["both", "public_only"])+ def test_discover_mcp_tools_passes_under_permissive_scope(self, scope):+ """BOTH and PUBLIC_ONLY allow MCP through the gate. The+ downstream connection may still fail for unrelated reasons; we+ only assert the gate did not block.+ """+ strategy = self._build(scope)+ with patch(+ "local_deep_research.mcp.client.MCPClientManager"+ ) as mock_manager:+ mock_instance = MagicMock()+ mock_instance.connect_all.side_effect = Exception(+ "Connection failed"+ )+ mock_manager.return_value = mock_instance+ tools = strategy._discover_mcp_tools()+ # Connection was attempted (and failed) — meaning the gate+ # allowed the call through.+ mock_manager.assert_called_once()+ assert tools == []++ @pytest.mark.parametrize("scope", ["strict", "private_only"])+ def test_execute_mcp_tool_refused_under_restrictive_scope(self, scope):+ """Defense-in-depth: even if a cached tool list survives, the+ executor must independently re-check scope.+ """+ strategy = self._build(scope)+ result = strategy._execute_mcp_tool(+ "srv", "some_tool", {"arg": "value"}+ )+ assert result["status"] == "error"+ assert "egress policy" in result["content"].lower()+++class TestMCPDownloadContentPolicy:+ """Tests for the narrowed exception handling in+ ``_execute_download_content``. The bare ``except Exception`` used to+ swallow PolicyDeniedError/ValueError and drop to SSRF-only, which+ fails open under PRIVATE_ONLY.+ """++ @staticmethod+ def _build_strategy(snapshot):+ from local_deep_research.advanced_search_system.strategies.mcp_strategy import (+ MCPSearchStrategy,+ )++ return MCPSearchStrategy(+ model=MagicMock(),+ search=MagicMock(),+ mcp_servers=[],+ settings_snapshot=snapshot,+ )++ def test_download_content_refuses_when_policy_unevaluable(self):+ """Corrupt scope → ``context_from_snapshot`` raises+ PolicyDeniedError → previously dropped to SSRF-only → now must+ return a clear error without ever constructing ContentFetcher.+ """+ strategy = self._build_strategy(+ {"policy.egress_scope": "not_a_real_scope"}+ )+ with patch(+ "local_deep_research.content_fetcher.ContentFetcher"+ ) as MockFetcher:+ result = strategy._execute_download_content(+ {"url": "https://example.com"}+ )+ assert result["status"] == "error"+ assert "egress policy" in result["content"].lower()+ # The fetcher must never be constructed — that is the whole+ # point of failing closed before fetch.+ MockFetcher.assert_not_called()+"""Tests for strategy export and import."""def test_strategy_registered_in_factory(self):@@ -1334,6 +1531,9 @@ class TestTimeoutAndRetry:model=MagicMock(),search=MagicMock(),mcp_servers=[{"name": "test", "command": "nonexistent"}],+ # Permissive scope so this exercises the connection-failure+ # retry semantics, not the policy gate (which caches []).+ settings_snapshot={"policy.egress_scope": "both"},)# First discovery fails@@ -1366,6 +1566,7 @@ class TestMCPPartialFailure:{"name": "working", "command": "python"},{"name": "broken", "command": "nonexistent"},],+ settings_snapshot={"policy.egress_scope": "both"},)# Mock the client manager to simulate partial failure@@ -1415,6 +1616,7 @@ class TestMCPPartialFailure:model=MagicMock(),search=MagicMock(),mcp_servers=[{"name": "real_server", "command": "python"}],+ settings_snapshot={"policy.egress_scope": "both"},)# Try to call tool on non-existent serverdiff --git a/tests/news/test_headline_generator.py b/tests/news/test_headline_generator.pyindex c02b471f71..0b01849369 100644--- a/tests/news/test_headline_generator.py+++ b/tests/news/test_headline_generator.py@@ -119,7 +119,12 @@ class TestGenerateWithLLM:_generate_with_llm("query", "findings", 100)- mock_get_llm.assert_called_once_with(temperature=0.3)+ # settings_snapshot is now threaded into get_llm (defaults to+ # None when the caller doesn't supply one) so the egress policy+ # PEP can fire for background news jobs.+ mock_get_llm.assert_called_once_with(+ temperature=0.3, settings_snapshot=None+ )@patch("local_deep_research.config.llm_config.get_llm")def test_prompt_includes_findings(self, mock_get_llm):diff --git a/tests/news/test_headline_generator_comprehensive.py b/tests/news/test_headline_generator_comprehensive.pyindex 40825c3848..50a5e05329 100644--- a/tests/news/test_headline_generator_comprehensive.py+++ b/tests/news/test_headline_generator_comprehensive.py@@ -152,7 +152,9 @@ class TestGenerateWithLLMBehavior:assert callable(_generate_with_llm)def test_accepts_three_parameters(self):- """Test accepts query, findings, and max_length parameters."""+ """Test accepts query, findings, max_length, and the optional+ settings_snapshot parameter (added so background news jobs can+ thread the egress policy into get_llm)."""from local_deep_research.news.utils.headline_generator import (_generate_with_llm,)@@ -161,10 +163,13 @@ class TestGenerateWithLLMBehavior:sig = inspect.signature(_generate_with_llm)params = list(sig.parameters.keys())- assert len(params) == 3assert "query" in paramsassert "findings" in paramsassert "max_length" in params+ # settings_snapshot is optional (defaults to None) so existing+ # callers keep working.+ assert "settings_snapshot" in params+ assert sig.parameters["settings_snapshot"].default is Nonedef test_returns_none_for_empty_string_findings(self):"""Test returns None for empty string findings."""diff --git a/tests/news/test_headline_generator_extended.py b/tests/news/test_headline_generator_extended.pyindex 9b3326fedf..d7aa437606 100644--- a/tests/news/test_headline_generator_extended.py+++ b/tests/news/test_headline_generator_extended.py@@ -89,7 +89,10 @@ class TestGenerateHeadlineBasic:generate_headline("query", "findings", max_length=50)- mock_gen.assert_called_once_with("query", "findings", 50)+ # generate_headline now threads settings_snapshot (defaults+ # to None) into _generate_with_llm so background news jobs+ # can hand the egress policy to get_llm.+ mock_gen.assert_called_once_with("query", "findings", 50, None)class TestGenerateWithLLM:diff --git a/tests/news/test_news_api_extended.py b/tests/news/test_news_api_extended.pyindex a6d68909e6..a4fdffb2a1 100644--- a/tests/news/test_news_api_extended.py+++ b/tests/news/test_news_api_extended.py@@ -704,6 +704,11 @@ class TestSubscriptionOperations:with patch("local_deep_research.news.api._notify_scheduler_about_subscription_change"):+ # Use "auto" (a known meta-picker) instead of the+ # legacy "google" placeholder — the news subscription+ # policy precheck now properly rejects engine_unknown+ # names (was previously silently allowed via the+ # engine_unknown bypass — see plan C1).result = create_subscription(user_id="testuser",query="AI News",@@ -715,7 +720,7 @@ class TestSubscriptionOperations:name="My AI Subscription",folder_id="folder123",is_active=True,- search_engine="google",+ search_engine="auto",search_iterations=5,questions_per_iteration=3,)diff --git a/tests/news/test_scheduler.py b/tests/news/test_scheduler.pyindex ccf388c2e7..ab26c0d0b5 100644--- a/tests/news/test_scheduler.py+++ b/tests/news/test_scheduler.py@@ -2399,3 +2399,45 @@ class TestCheckUserOverdueSubscriptions:# Should not raisescheduler._check_user_overdue_subscriptions("testuser")+++class TestSchedulerEgressBackstop:+ """The document scheduler runs on an APScheduler worker thread with no+ egress context; _arm_egress_backstop must set one from the user's saved+ settings so DownloadService fetches get the audit-hook secondary net+ (R2-8). @thread_cleanup clears it on exit."""++ def test_arm_egress_backstop_sets_context_from_settings(self):+ from local_deep_research.scheduler.background import (+ BackgroundJobScheduler,+ )+ from local_deep_research.security.egress.policy import EgressScope++ sched = BackgroundJobScheduler()+ fake_sm = MagicMock()+ fake_sm.get_settings_snapshot.return_value = {+ "policy.egress_scope": "private_only",+ "search.tool": "library",+ }+ fake_sm.get_setting.side_effect = lambda k, d=None: (+ "library" if k == "search.tool" else d+ )++ with patch(+ "local_deep_research.security.egress.audit_hook.set_active_context"+ ) as mock_set:+ sched._arm_egress_backstop(fake_sm, "alice")+ assert mock_set.call_count == 1+ ctx = mock_set.call_args[0][0]+ assert ctx.scope == EgressScope.PRIVATE_ONLY++ def test_arm_egress_backstop_never_raises_on_bad_settings(self):+ from local_deep_research.scheduler.background import (+ BackgroundJobScheduler,+ )++ sched = BackgroundJobScheduler()+ fake_sm = MagicMock()+ fake_sm.get_settings_snapshot.side_effect = RuntimeError("no db")+ # Must swallow the error (best-effort backstop).+ sched._arm_egress_backstop(fake_sm, "alice")diff --git a/tests/news/test_subscription_policy.py b/tests/news/test_subscription_policy.pynew file mode 100644index 0000000000..bfab972f16--- /dev/null+++ b/tests/news/test_subscription_policy.py@@ -0,0 +1,83 @@+"""Coverage for _validate_subscription_policy (PR #4300 review).++News subscriptions persist a fixed engine + LLM provider that run on a+schedule. This validator rejects a forbidden config at create/update time+(the factory PEP is only the execution-time backstop). Returns a+human-readable reason string on rejection, or None when allowed / when the+settings backend is unavailable (best-effort).++Source: src/local_deep_research/news/api.py+"""++from unittest.mock import Mock, patch++from local_deep_research.news.api import _validate_subscription_policy+++def _mgr(snapshot, primary="arxiv"):+ m = Mock()+ m.get_settings_snapshot.return_value = snapshot+ m.get_setting.side_effect = lambda key, default=None: (+ primary if key == "search.tool" else default+ )+ return m+++def _patch_mgr(manager):+ return patch(+ "local_deep_research.utilities.db_utils.get_settings_manager",+ return_value=manager,+ )+++class TestValidateSubscriptionPolicy:+ def test_allowed_engine_returns_none(self):+ with _patch_mgr(_mgr({"policy.egress_scope": "public_only"})):+ assert (+ _validate_subscription_policy(Mock(), "u", "arxiv", None)+ is None+ )++ def test_forbidden_engine_returns_reason(self):+ # PUBLIC_ONLY + local engine (library) => rejected.+ with _patch_mgr(_mgr({"policy.egress_scope": "public_only"})):+ reason = _validate_subscription_policy(Mock(), "u", "library", None)+ assert reason is not None+ assert "not permitted" in reason++ def test_corrupt_scope_returns_reason(self):+ with _patch_mgr(_mgr({"policy.egress_scope": "garbage"})):+ reason = _validate_subscription_policy(Mock(), "u", "arxiv", None)+ assert reason is not None+ assert "egress policy refused" in reason++ def test_cloud_llm_provider_denied_under_require_local(self):+ # require_local_llm + a cloud provider => provider rejected.+ snap = {+ "policy.egress_scope": "both",+ "llm.require_local_endpoint": True,+ }+ with _patch_mgr(_mgr(snap)):+ reason = _validate_subscription_policy(Mock(), "u", None, "openai")+ assert reason is not None+ assert "provider" in reason.lower()++ def test_non_dict_snapshot_skips(self):+ # Settings backend unavailable => best-effort skip (None).+ with _patch_mgr(_mgr(None)):+ assert (+ _validate_subscription_policy(Mock(), "u", "library", None)+ is None+ )++ def test_local_llm_provider_allowed_under_require_local(self):+ snap = {+ "policy.egress_scope": "both",+ "llm.require_local_endpoint": True,+ }+ with _patch_mgr(_mgr(snap)):+ # ollama is a localhost-default provider => allowed.+ assert (+ _validate_subscription_policy(Mock(), "u", None, "ollama")+ is None+ )diff --git a/tests/news/test_topic_generator.py b/tests/news/test_topic_generator.pyindex c7f63b833e..7bd1be8133 100644--- a/tests/news/test_topic_generator.py+++ b/tests/news/test_topic_generator.py@@ -286,7 +286,9 @@ class TestGenerateWithLLM:_generate_with_llm("query", "findings", "category", 5)- mock_get_llm.assert_called_once_with(temperature=0.5)+ mock_get_llm.assert_called_once_with(+ temperature=0.5, settings_snapshot=None+ )@patch("local_deep_research.config.llm_config.get_llm")def test_truncates_long_query_in_prompt(self, mock_get_llm):diff --git a/tests/news/test_topic_generator_comprehensive.py b/tests/news/test_topic_generator_comprehensive.pyindex 423861bf9d..5be3219838 100644--- a/tests/news/test_topic_generator_comprehensive.py+++ b/tests/news/test_topic_generator_comprehensive.py@@ -113,7 +113,9 @@ class TestGenerateWithLLMTopicsBehavior:sig = inspect.signature(_generate_with_llm)params = list(sig.parameters.keys())- assert len(params) == 4+ # Now accepts settings_snapshot for egress-policy threading.+ assert len(params) == 5+ assert "settings_snapshot" in paramsassert "query" in paramsassert "findings" in paramsassert "category" in paramsdiff --git a/tests/news/test_topic_generator_coverage.py b/tests/news/test_topic_generator_coverage.pyindex 9864e7716e..4b3979c377 100644--- a/tests/news/test_topic_generator_coverage.py+++ b/tests/news/test_topic_generator_coverage.py@@ -306,9 +306,9 @@ class TestGenerateTopicsOrchestration:def test_default_parameters(self, mock_llm):mock_llm.return_value = []generate_topics("q")- # (query, findings, category, max_topics)+ # (query, findings, category, max_topics, settings_snapshot)args = mock_llm.call_args[0]- assert args == ("q", "", "", 5)+ assert args == ("q", "", "", 5, None)# ---------------------------------------------------------------------------diff --git a/tests/notifications/test_manager.py b/tests/notifications/test_manager.pyindex 7b0a1a3877..85945c8642 100644--- a/tests/notifications/test_manager.py+++ b/tests/notifications/test_manager.py@@ -838,3 +838,40 @@ class TestPerUserRateLimiting:# Both managers should see the updated limitsassert manager1._rate_limiter.get_user_limits("user_a") == (10, 50)assert manager2._rate_limiter.get_user_limits("user_a") == (10, 50)+++class TestEgressPolicyURLFilter:+ """_filter_urls_by_egress_policy gates notification dispatch by scope."""++ def _mgr(self, scope, tool="auto"):+ snap = {+ "policy.egress_scope": {"value": scope},+ "search.tool": {"value": tool},+ }+ return NotificationManager(settings_snapshot=snap, user_id="u")++ def test_private_only_blocks_vendor_schemes_and_public_http(self):+ mgr = self._mgr("private_only", tool="library")+ out = mgr._filter_urls_by_egress_policy(+ "slack://t discord://x http://127.0.0.1/h https://public.example.com/h"+ )+ # Only the LOCAL http webhook survives; vendor schemes + public host dropped.+ assert out == "http://127.0.0.1/h"++ def test_both_scope_passes_vendor_schemes(self):+ mgr = self._mgr("both")+ out = mgr._filter_urls_by_egress_policy("slack://t http://127.0.0.1/h")+ assert "slack://t" in out+ assert "http://127.0.0.1/h" in out++ def test_public_only_blocks_private_http_keeps_public(self):+ mgr = self._mgr("public_only", tool="searxng")+ out = mgr._filter_urls_by_egress_policy(+ "http://127.0.0.1/h https://public.example.com/h"+ )+ assert out == "https://public.example.com/h"++ def test_no_snapshot_passes_through(self):+ mgr = NotificationManager.__new__(NotificationManager)+ mgr._settings_snapshot = None+ assert mgr._filter_urls_by_egress_policy("slack://t") == "slack://t"diff --git a/tests/research_library/services/test_download_service_logic.py b/tests/research_library/services/test_download_service_logic.pyindex 544fe4d43f..c3bf76eca0 100644--- a/tests/research_library/services/test_download_service_logic.py+++ b/tests/research_library/services/test_download_service_logic.py@@ -416,3 +416,170 @@ class TestNormalizeUrl:# Both should normalize to the same thingassert r1 == r2service.close()+++# ---------------------------------------------------------------------------+# Egress policy gates at the network-fire points (PR #4300 review fix)+# ---------------------------------------------------------------------------+++class TestEgressPolicyGatesAtFirePoints:+ """The entry methods (download_resource, download_as_text) do NOT gate+ the URL; the egress check lives at the three network-fire points so no+ caller can route around it. These tests pin that a denied URL never+ reaches a downloader from _download_pdf / _try_api_text_extraction /+ _fallback_pdf_extraction.++ Regression: before the fix these three methods fired+ downloader.download_with_result()/download_text() with no policy check,+ so a PRIVATE_ONLY/STRICT run could exfiltrate to a public URL via the+ download_resource / download_as_text paths.+ """++ def _deny(self, service):+ """Force _check_url_against_policy to deny every URL."""+ patcher = patch.object(+ service,+ "_check_url_against_policy",+ return_value=(False, "scope_mismatch_private_only"),+ )+ return patcher++ def test_download_pdf_denied_url_never_reaches_downloader(self, tmp_path):+ service = _create_service(tmp_path)+ # A downloader that would explode if ever consulted.+ exploding = Mock()+ exploding.can_handle.side_effect = AssertionError(+ "downloader consulted despite policy denial"+ )+ service.downloaders = [exploding]++ resource = Mock()+ resource.url = "https://public.example.com/paper.pdf"++ with self._deny(service):+ success, reason, status = service._download_pdf(+ resource, Mock(), Mock(), None+ )++ assert success is False+ assert reason.startswith("egress_policy_denied:")+ exploding.can_handle.assert_not_called()+ service.close()++ def test_try_api_text_extraction_denied_returns_non_none_tuple(+ self, tmp_path+ ):+ """Must return a non-None tuple on denial so download_as_text+ hard-stops instead of falling through to _fallback_pdf_extraction+ (which would fire a second request)."""+ service = _create_service(tmp_path)+ downloader = Mock()+ downloader.download_with_result.side_effect = AssertionError(+ "download_with_result fired despite policy denial"+ )++ resource = Mock()+ resource.url = "https://public.example.com/paper"+ resource.id = 1+ resource.title = "t"++ with (+ patch.object(service, "_get_downloader", return_value=downloader),+ self._deny(service),+ ):+ result = service._try_api_text_extraction(Mock(), resource)++ assert result is not None, "denial must not fall through to fallback"+ ok, reason = result+ assert ok is False+ assert reason.startswith("egress_policy_denied:")+ downloader.download_with_result.assert_not_called()+ service.close()++ def test_fallback_pdf_extraction_denied_url_never_downloads(self, tmp_path):+ service = _create_service(tmp_path)+ downloader = Mock()+ downloader.download_with_result.side_effect = AssertionError(+ "download_with_result fired despite policy denial"+ )++ resource = Mock()+ resource.url = "https://public.example.com/paper.pdf"+ resource.id = 1+ resource.title = "t"++ with (+ patch.object(service, "_get_downloader", return_value=downloader),+ patch.object(service, "_record_failed_text_extraction"),+ self._deny(service),+ ):+ ok, reason = service._fallback_pdf_extraction(Mock(), resource)++ assert ok is False+ assert reason.startswith("egress_policy_denied:")+ downloader.download_with_result.assert_not_called()+ service.close()+++class TestDefaultSnapshotEgressContext:+ """When no settings_snapshot is passed, DownloadService builds one from+ its own settings_manager so egress policy is still enforced. This closes+ the library-download routes hole where DownloadService(username, password)+ was constructed snapshot-less and every download ran ungated.+ """++ def _service_with_snapshot(self, tmp_path, snapshot):+ mock_settings = Mock()+ mock_settings.get_setting.side_effect = lambda key, default=None: {+ "research_library.storage_path": str(tmp_path),+ "research_library.pdf_storage_mode": "none",+ }.get(key, default)+ # The new behavior: __init__ pulls a snapshot from here when none+ # is passed explicitly.+ mock_settings.get_settings_snapshot.return_value = snapshot++ with (+ patch(+ "local_deep_research.research_library.services.download_service.get_settings_manager",+ return_value=mock_settings,+ ),+ patch(+ "local_deep_research.research_library.services.download_service.RetryManager",+ ),+ patch(+ "local_deep_research.research_library.services.download_service.get_library_directory",+ return_value=tmp_path,+ ),+ ):+ from local_deep_research.research_library.services.download_service import (+ DownloadService,+ )++ # No settings_snapshot passed — must self-build from manager.+ return DownloadService("u", "p")++ def test_builds_context_and_denies_public_url_under_private_only(+ self, tmp_path+ ):+ service = self._service_with_snapshot(+ tmp_path, {"policy.egress_scope": "private_only"}+ )+ # Context was built from the self-fetched snapshot.+ assert service._egress_context is not None+ # A public URL is refused under PRIVATE_ONLY without an explicit+ # snapshot argument.+ allowed, reason = service._check_url_against_policy(+ "https://arxiv.org/abs/1"+ )+ assert allowed is False+ service.close()++ def test_non_dict_snapshot_falls_back_to_ungated(self, tmp_path):+ # A test-double manager returning a non-dict must NOT fail closed;+ # it falls back to the pre-policy ungated behavior.+ service = self._service_with_snapshot(tmp_path, Mock())+ assert service._egress_context is None+ allowed, _ = service._check_url_against_policy("https://arxiv.org/x")+ assert allowed is True # no_context back-compat+ service.close()diff --git a/tests/retriever_integration/test_e2e_basic.py b/tests/retriever_integration/test_e2e_basic.pyindex 0b7f51c488..7e970a985e 100644--- a/tests/retriever_integration/test_e2e_basic.py+++ b/tests/retriever_integration/test_e2e_basic.py@@ -11,6 +11,11 @@ from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)+# Minimal non-empty snapshot so the factory's egress PEP can run; under the+# default ``both`` scope a local retriever is permitted. The factory now+# fails closed (no_snapshot) without one.+_BOTH_SNAPSHOT = {"policy.egress_scope": {"value": "both"}}+class SimpleRetriever(BaseRetriever):"""Simple retriever that returns fixed content."""@@ -84,7 +89,9 @@ class TestE2EBasic:retriever_registry.register("e2e_test", retriever)# Create search engine through factory- engine = create_search_engine("e2e_test")+ engine = create_search_engine(+ "e2e_test", settings_snapshot=_BOTH_SNAPSHOT+ )assert engine is not Noneassert engine.name == "e2e_test"@@ -127,7 +134,9 @@ class TestBasicIntegration:retriever_registry.register("integration_test", retriever)# Create search engine- engine = create_search_engine("integration_test")+ engine = create_search_engine(+ "integration_test", settings_snapshot=_BOTH_SNAPSHOT+ )# Perform searchresults = engine.run("integration test query")diff --git a/tests/retriever_integration/test_factory_integration.py b/tests/retriever_integration/test_factory_integration.pyindex c62f79c4e8..24c7b98237 100644--- a/tests/retriever_integration/test_factory_integration.py+++ b/tests/retriever_integration/test_factory_integration.py@@ -13,6 +13,12 @@ from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)+# Minimal non-empty snapshot so the factory's egress PEP can run. The+# factory now requires a snapshot (no_snapshot fails closed); under the+# default ``both`` scope a local retriever is permitted, so this mirrors a+# real caller without pulling in the full settings infrastructure.+_BOTH_SNAPSHOT = {"policy.egress_scope": {"value": "both"}}+class TestRetriever(BaseRetriever):"""Test retriever."""@@ -47,7 +53,9 @@ class TestFactoryIntegration:retriever_registry.register("test_retriever", retriever)# Create search engine- engine = create_search_engine("test_retriever")+ engine = create_search_engine(+ "test_retriever", settings_snapshot=_BOTH_SNAPSHOT+ )assert engine is not Noneassert isinstance(engine, RetrieverSearchEngine)@@ -70,7 +78,9 @@ class TestFactoryIntegration:retriever = TestRetriever()retriever_registry.register("test", retriever)- engine = create_search_engine("test", max_results=5)+ engine = create_search_engine(+ "test", max_results=5, settings_snapshot=_BOTH_SNAPSHOT+ )assert isinstance(engine, RetrieverSearchEngine)assert engine.max_results == 5@@ -84,7 +94,9 @@ class TestFactoryIntegration:# Create engines for eachengines = []for i in range(3):- engine = create_search_engine(f"retriever_{i}")+ engine = create_search_engine(+ f"retriever_{i}", settings_snapshot=_BOTH_SNAPSHOT+ )engines.append(engine)assert all(isinstance(e, RetrieverSearchEngine) for e in engines)diff --git a/tests/routes/test_settings_routes.py b/tests/routes/test_settings_routes.pyindex dad0e9e881..f786ffd46c 100644--- a/tests/routes/test_settings_routes.py+++ b/tests/routes/test_settings_routes.py@@ -257,6 +257,13 @@ class TestCalculateWarnings:"app.warnings.dismiss_context_reduced": False,"app.warnings.dismiss_no_backups": True,"app.warnings.dismiss_backup_disabled": False,+ # Egress-policy banners (public-egress + cloud-LLM) fire for a+ # cloud provider under the default 'both' scope; each banner+ # now has its own dismiss flag, so all three must be dismissed+ # to keep this "no warnings" scenario genuinely empty.+ "app.warnings.dismiss_egress_policy": True,+ "app.warnings.dismiss_cloud_llm": True,+ "app.warnings.dismiss_cloud_embeddings": True,"backup.enabled": True,"llm.model": "gpt-4",}.get(key, default)diff --git a/tests/security/test_egress_adaptive_resolution_followup.py b/tests/security/test_egress_adaptive_resolution_followup.pynew file mode 100644index 0000000000..d16c1e97bb--- /dev/null+++ b/tests/security/test_egress_adaptive_resolution_followup.py@@ -0,0 +1,363 @@+"""Follow-up regression tests for ADAPTIVE scope resolution and+``context_from_snapshot`` cross-field coupling (PR #4300 egress policy).++These cover gaps not already exercised by+``tests/security/test_egress_policy.py``:++* ADAPTIVE resolution of *registered retriever* primaries (local => PRIVATE_ONLY,+ public => PUBLIC_ONLY) via the retriever registry path inside+ ``_resolve_adaptive_scope``.+* The full meta-picker frozenset (meta / parallel / parallel_scientific, not+ just "auto") resolving to BOTH, and raising under STRICT.+* The PRIVATE_ONLY -> require_local_llm/require_local_embeddings *coupling* on a+ direct (non-adaptive) PRIVATE_ONLY scope, and the deliberate absence of that+ coupling under STRICT.+* Non-dict snapshot ValueError contract.+* ``allow_dns=False`` skipping ``_resolve_with_timeout`` and falling back to the+ engine's static classification (vs. the DNS-driven result with allow_dns=True).+"""++from __future__ import annotations++from unittest.mock import MagicMock, patch++import pytest++from local_deep_research.security.egress.policy import (+ EgressScope,+ PolicyDeniedError,+ _resolve_adaptive_scope,+ context_from_snapshot,+)+from local_deep_research.web_search_engines.retriever_registry import (+ retriever_registry,+)+++def _adaptive_snapshot(tool: str, **extra) -> dict:+ """Nested-value snapshot selecting ADAPTIVE scope with the given tool."""+ snap = {+ "policy.egress_scope": {"value": "adaptive"},+ "search.tool": {"value": tool},+ }+ snap.update(extra)+ return snap+++# ---------------------------------------------------------------------------+# ADAPTIVE: concrete engine classification (allow + deny pairs)+# ---------------------------------------------------------------------------+++def test_resolve_adaptive_concrete_public_engine_is_public_only():+ """_resolve_adaptive_scope: a concrete public engine -> PUBLIC_ONLY."""+ scope = _resolve_adaptive_scope(+ "arxiv",+ {},+ username=None,+ local_hostnames=(),+ )+ assert scope == EgressScope.PUBLIC_ONLY+++def test_resolve_adaptive_concrete_private_engine_is_private_only():+ """_resolve_adaptive_scope: a concrete local engine -> PRIVATE_ONLY."""+ scope = _resolve_adaptive_scope(+ "paperless",+ {},+ username=None,+ local_hostnames=(),+ )+ assert scope == EgressScope.PRIVATE_ONLY+++# ---------------------------------------------------------------------------+# ADAPTIVE: meta-picker frozenset -> BOTH (every member, not just "auto")+# ---------------------------------------------------------------------------+++ "picker", ["auto", "meta", "parallel", "parallel_scientific"]+)+def test_resolve_adaptive_meta_picker_members_resolve_to_both(picker):+ scope = _resolve_adaptive_scope(+ picker,+ {},+ username=None,+ local_hostnames=(),+ )+ assert scope == EgressScope.BOTH+++def test_resolve_adaptive_empty_primary_is_both():+ """A missing/empty primary cannot be classified -> permissive BOTH."""+ scope = _resolve_adaptive_scope(+ "",+ {},+ username=None,+ local_hostnames=(),+ )+ assert scope == EgressScope.BOTH++[email protected]("picker", ["meta", "parallel", "parallel_scientific"])+def test_context_adaptive_meta_picker_members_resolve_to_both(picker):+ """Through the public entrypoint: ADAPTIVE + each meta-picker -> BOTH and+ leaves inference requirements untouched (no PRIVATE_ONLY coupling)."""+ ctx = context_from_snapshot(+ _adaptive_snapshot(picker), primary_engine=picker+ )+ assert ctx.scope == EgressScope.BOTH+ assert ctx.require_local_llm is False+ assert ctx.require_local_embeddings is False+++# ---------------------------------------------------------------------------+# ADAPTIVE: registered retriever primaries (registry path)+# ---------------------------------------------------------------------------+++def test_resolve_adaptive_local_retriever_is_private_only():+ """A registered LOCAL retriever (not in ENGINE_REGISTRY) resolves via the+ retriever registry to PRIVATE_ONLY."""+ name = "_adaptive_followup_local_kb"+ retriever_registry.register(name, MagicMock(), is_local=True)+ try:+ scope = _resolve_adaptive_scope(+ name,+ {},+ username=None,+ local_hostnames=(),+ )+ assert scope == EgressScope.PRIVATE_ONLY+ finally:+ retriever_registry.unregister(name)+++def test_resolve_adaptive_public_retriever_is_public_only():+ """A registered PUBLIC retriever resolves to PUBLIC_ONLY."""+ name = "_adaptive_followup_public_idx"+ retriever_registry.register(name, MagicMock(), is_local=False)+ try:+ scope = _resolve_adaptive_scope(+ name,+ {},+ username=None,+ local_hostnames=(),+ )+ assert scope == EgressScope.PUBLIC_ONLY+ finally:+ retriever_registry.unregister(name)+++def test_context_adaptive_local_retriever_forces_local_inference():+ """End-to-end: ADAPTIVE + a registered local retriever primary resolves to+ PRIVATE_ONLY AND inherits the require_local_* coupling."""+ name = "_adaptive_followup_local_kb2"+ retriever_registry.register(name, MagicMock(), is_local=True)+ try:+ ctx = context_from_snapshot(+ _adaptive_snapshot(name), primary_engine=name+ )+ assert ctx.scope == EgressScope.PRIVATE_ONLY+ assert ctx.require_local_llm is True+ assert ctx.require_local_embeddings is True+ finally:+ retriever_registry.unregister(name)+++def test_context_adaptive_public_retriever_does_not_force_local():+ """ADAPTIVE + a registered public retriever -> PUBLIC_ONLY, which does NOT+ force local inference (public scope is orthogonal to local-inference)."""+ name = "_adaptive_followup_public_idx2"+ retriever_registry.register(name, MagicMock(), is_local=False)+ try:+ ctx = context_from_snapshot(+ _adaptive_snapshot(name), primary_engine=name+ )+ assert ctx.scope == EgressScope.PUBLIC_ONLY+ assert ctx.require_local_llm is False+ assert ctx.require_local_embeddings is False+ finally:+ retriever_registry.unregister(name)+++def test_resolve_adaptive_unknown_primary_falls_back_to_both():+ """A name that is neither a static engine nor a registered retriever is+ unclassifiable -> BOTH (permissive fallback, never a hard fail), and no+ require_local coupling is applied."""+ name = "_adaptive_followup_not_registered_anywhere"+ # Ensure it is genuinely absent from the retriever registry.+ assert retriever_registry.get_metadata(name) is None+ ctx = context_from_snapshot(_adaptive_snapshot(name), primary_engine=name)+ assert ctx.scope == EgressScope.BOTH+ assert ctx.require_local_llm is False+ assert ctx.require_local_embeddings is False+++# ---------------------------------------------------------------------------+# PRIVATE_ONLY coupling vs STRICT non-coupling+# ---------------------------------------------------------------------------+++def test_direct_private_only_forces_local_llm_and_embeddings():+ """A directly-selected PRIVATE_ONLY scope (not via ADAPTIVE) forces both+ require_local_llm and require_local_embeddings even when the user left+ those flags at their permissive default. This is the core coupling that+ prevents silent exfiltration through cloud inference."""+ ctx = context_from_snapshot(+ {+ "policy.egress_scope": {"value": "private_only"},+ "llm.require_local_endpoint": {"value": False},+ "embeddings.require_local": {"value": False},+ },+ primary_engine="arxiv",+ )+ assert ctx.scope == EgressScope.PRIVATE_ONLY+ assert ctx.require_local_llm is True+ assert ctx.require_local_embeddings is True+++def test_strict_does_not_force_local_inference():+ """STRICT restricts the search-engine set but is deliberately orthogonal to+ where inference runs: it must NOT force require_local_*."""+ ctx = context_from_snapshot(+ {+ "policy.egress_scope": {"value": "strict"},+ "llm.require_local_endpoint": {"value": False},+ "embeddings.require_local": {"value": False},+ },+ primary_engine="arxiv",+ )+ assert ctx.scope == EgressScope.STRICT+ assert ctx.require_local_llm is False+ assert ctx.require_local_embeddings is False+++def test_strict_preserves_explicit_local_inference_flags():+ """STRICT does not force the flags, but it must preserve a user who DID+ opt in (no silent reset)."""+ ctx = context_from_snapshot(+ {+ "policy.egress_scope": {"value": "strict"},+ "llm.require_local_endpoint": {"value": True},+ "embeddings.require_local": {"value": True},+ },+ primary_engine="arxiv",+ )+ assert ctx.scope == EgressScope.STRICT+ assert ctx.require_local_llm is True+ assert ctx.require_local_embeddings is True+++# ---------------------------------------------------------------------------+# STRICT + meta-picker -> ValueError (every member except the one already+# covered by the existing suite)+# ---------------------------------------------------------------------------++[email protected]("picker", ["meta", "parallel", "parallel_scientific"])+def test_strict_with_each_meta_picker_raises_value_error(picker):+ with pytest.raises(ValueError):+ context_from_snapshot(+ {"policy.egress_scope": {"value": "strict"}},+ primary_engine=picker,+ )+++def test_strict_with_concrete_primary_does_not_raise():+ """Mirror: STRICT + a concrete primary is the supported, allowed combo."""+ ctx = context_from_snapshot(+ {"policy.egress_scope": {"value": "strict"}},+ primary_engine="arxiv",+ )+ assert ctx.scope == EgressScope.STRICT+++# ---------------------------------------------------------------------------+# Snapshot validity contract+# ---------------------------------------------------------------------------+++ "bad_snapshot",+ [+ ["policy.egress_scope", "both"], # list+ "policy.egress_scope=both", # str+ 42, # int+ None, # None (the explicit "required" branch)+ ],+)+def test_non_dict_snapshot_raises_value_error(bad_snapshot):+ with pytest.raises(ValueError):+ context_from_snapshot(bad_snapshot, primary_engine="arxiv")+++def test_unknown_scope_string_raises_policy_denied_with_target():+ """An unrecognised scope string is fail-closed as PolicyDeniedError whose+ target carries the offending value (so the operator can see what was+ rejected) rather than silently degrading to BOTH."""+ with pytest.raises(PolicyDeniedError) as excinfo:+ context_from_snapshot(+ {"policy.egress_scope": {"value": "PUBLIC"}}, # not a valid member+ primary_engine="arxiv",+ )+ assert excinfo.value.decision.reason == "unknown_egress_scope"+ assert excinfo.value.target == "PUBLIC"+++# ---------------------------------------------------------------------------+# allow_dns=False: skip _resolve_with_timeout, fall back to static flags+# ---------------------------------------------------------------------------+++def test_adaptive_allow_dns_false_skips_dns_and_uses_static_flags():+ """With allow_dns=False the URL-configurable primary's hostname is NOT+ DNS-resolved: _resolve_with_timeout must not be called, and resolution+ falls back to the engine's STATIC classification (searxng is_public=True+ -> PUBLIC_ONLY)."""+ snap = _adaptive_snapshot(+ "searxng",+ **{+ "search.engine.web.searxng.default_params.instance_url": (+ "http://searx.internal.lab:8080"+ )+ },+ )+ with patch(+ "local_deep_research.security.egress.policy._resolve_with_timeout"+ ) as mock_resolve:+ ctx = context_from_snapshot(+ snap, primary_engine="searxng", allow_dns=False+ )+ mock_resolve.assert_not_called()+ # Static fallback: searxng's declared is_public=True -> PUBLIC_ONLY.+ assert ctx.scope == EgressScope.PUBLIC_ONLY+++def test_adaptive_allow_dns_true_uses_dns_resolution():+ """Contrast: with allow_dns=True the same private-resolving hostname IS+ DNS-resolved (via _resolve_with_timeout), classifying searxng as LOCAL ->+ PRIVATE_ONLY. Proves allow_dns toggles the DNS path, not a no-op."""+ snap = _adaptive_snapshot(+ "searxng",+ **{+ "search.engine.web.searxng.default_params.instance_url": (+ "http://searx.internal.lab:8080"+ )+ },+ )+ private_addrinfo = [(None, None, None, None, ("192.168.50.10", 0))]+ with patch(+ "local_deep_research.security.egress.policy._resolve_with_timeout",+ return_value=private_addrinfo,+ ) as mock_resolve:+ ctx = context_from_snapshot(+ snap, primary_engine="searxng", allow_dns=True+ )+ mock_resolve.assert_called()+ assert ctx.scope == EgressScope.PRIVATE_ONLY+ # And the DNS-driven PRIVATE_ONLY still triggers the local-inference+ # coupling.+ assert ctx.require_local_llm is True+ assert ctx.require_local_embeddings is Truediff --git a/tests/security/test_egress_audit_hook.py b/tests/security/test_egress_audit_hook.pynew file mode 100644index 0000000000..dd7896c971--- /dev/null+++ b/tests/security/test_egress_audit_hook.py@@ -0,0 +1,467 @@+"""Tests for the PEP 578 sys.audit hook that gates socket.connect.++The hook is the secondary line of defense for the egress policy. These+tests pin the contract at three levels:++1. Hook plumbing (install idempotency, context get/set/clear, the+ ``active_egress_context`` manager).+2. Per-scope behaviour on real ``socket.connect`` (not via requests/+ urllib — the point of the hook is to catch raw socket use, so the+ test exercises the raw socket path it is meant to catch).+3. Thread isolation: setting a context in one thread does NOT leak to+ another.++We use 127.0.0.1 / 8.8.8.8 / 192.168.x.x as classification targets. We+never need the connect to succeed — the hook fires BEFORE the kernel+ever sees the SYN, so a refused or timed-out connect is fine; we only+check whether the hook raised ``PolicyDeniedError`` or not.+"""++from __future__ import annotations++import socket+import threading++import pytest++from local_deep_research.security import (+ active_egress_context,+ clear_active_context,+ get_active_context,+ install_audit_hook,+ is_audit_hook_installed,+ set_active_context,+)+from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+ PolicyDeniedError,+)+++# ---------------------------------------------------------------------------+# Fixtures+# ---------------------------------------------------------------------------+++def _ctx(scope: EgressScope) -> EgressContext:+ """Build a minimal EgressContext for the given scope. No DNS cache+ or denial counter state — each test gets a fresh ctx so denial+ quotas across tests don't accumulate.+ """+ return EgressContext(+ scope=scope,+ primary_engine="wikipedia",+ require_local_llm=False,+ require_local_embeddings=False,+ )++[email protected](autouse=True)+def _clear_context_between_tests():+ """The hook reads from a process-wide thread-local. Even though+ tests run on a single thread, leaking a context across tests would+ produce confusing failures (one test's PRIVATE_ONLY context masking+ the next test's setup). Clear it before and after every test.+ """+ clear_active_context()+ yield+ clear_active_context()+++def _attempt_connect(host: str, port: int = 80, timeout: float = 0.5):+ """Open a raw AF_INET socket and call connect. Returns the exception+ raised (or None if the connect somehow succeeded — unlikely against+ a port we don't expect to be open, but we return None either way).++ The kernel-level outcome (refused, timed-out, ECONNRESET) is NOT what+ we test — we only care about ``PolicyDeniedError`` raised by the+ audit hook BEFORE the SYN is sent.+ """+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)+ sock.settimeout(timeout)+ try:+ sock.connect((host, port))+ return None+ except PolicyDeniedError as exc:+ return exc+ except OSError as exc:+ return exc+ finally:+ sock.close()+++# ---------------------------------------------------------------------------+# 1. Hook plumbing+# ---------------------------------------------------------------------------+++def test_install_audit_hook_is_idempotent():+ """Calling install_audit_hook multiple times must not stack hooks+ (PEP 578 hooks are append-only and CANNOT be removed; stacking+ would mean every connect fires the policy check N times).+ """+ install_audit_hook()+ install_audit_hook()+ install_audit_hook()+ assert is_audit_hook_installed() is True+++def test_set_and_get_active_context():+ ctx = _ctx(EgressScope.PRIVATE_ONLY)+ assert get_active_context() is None+ set_active_context(ctx)+ assert get_active_context() is ctx+ clear_active_context()+ assert get_active_context() is None+++def test_set_active_context_none_is_clear():+ ctx = _ctx(EgressScope.PRIVATE_ONLY)+ set_active_context(ctx)+ set_active_context(None)+ assert get_active_context() is None+++def test_clear_active_context_safe_without_set():+ """Clearing a context that was never set must not raise (workers in+ finally blocks call this unconditionally).+ """+ clear_active_context()+ clear_active_context()+++def test_active_egress_context_manager_clears_on_exit():+ ctx = _ctx(EgressScope.PRIVATE_ONLY)+ with active_egress_context(ctx):+ assert get_active_context() is ctx+ assert get_active_context() is None+++def test_active_egress_context_clears_on_exception():+ """Cleanup must happen even when the block raises — otherwise a+ crashed worker leaks its context to whatever thread-pool task runs+ next.+ """+ ctx = _ctx(EgressScope.PRIVATE_ONLY)+ with pytest.raises(RuntimeError):+ with active_egress_context(ctx):+ assert get_active_context() is ctx+ raise RuntimeError("boom")+ assert get_active_context() is None+++# ---------------------------------------------------------------------------+# 2. Per-scope behaviour on real socket.connect+# ---------------------------------------------------------------------------+++def test_no_active_context_lets_everything_through():+ """The hook is install-once-and-keep-forever per PEP 578, so the+ no-context behaviour is the only thing standing between unrelated+ code (test runners, import-time helpers, scripts) and a global+ deny-everything regression.+ """+ assert get_active_context() is None+ err = _attempt_connect("8.8.8.8", 53)+ # Socket-level outcome can vary (timeout vs refused vs ECONNRESET)+ # but it MUST NOT be PolicyDeniedError.+ assert not isinstance(err, PolicyDeniedError), (+ f"hook fired without an active context: {err!r}"+ )+++def test_private_only_blocks_public_ip():+ ctx = _ctx(EgressScope.PRIVATE_ONLY)+ with active_egress_context(ctx):+ err = _attempt_connect("8.8.8.8", 53)+ assert isinstance(err, PolicyDeniedError), (+ f"expected PolicyDeniedError, got {err!r}"+ )+ assert err.decision.reason == "scope_mismatch_private_only"+ assert err.target == "8.8.8.8"+++def test_private_only_allows_private_ip():+ ctx = _ctx(EgressScope.PRIVATE_ONLY)+ with active_egress_context(ctx):+ err = _attempt_connect("127.0.0.1", 1) # very likely closed+ # Whatever socket-level error happens, it must not be a policy denial.+ assert not isinstance(err, PolicyDeniedError), (+ f"hook wrongly blocked localhost under PRIVATE_ONLY: {err!r}"+ )+++def test_public_only_is_a_noop_at_the_audit_hook():+ """PUBLIC_ONLY governs which SEARCH ENGINES the user picks (no+ local engines), not which hosts the process is allowed to reach at+ the socket level. Local Ollama, local embeddings, the user's own+ sqlite settings DB all use private IPs — blocking them here would+ be a false positive against the user's actual intent. The+ search-engine PEP (evaluate_engine) still gates engine selection.+ """+ ctx = _ctx(EgressScope.PUBLIC_ONLY)+ with active_egress_context(ctx):+ for host in ("8.8.8.8", "192.168.42.1", "127.0.0.1"):+ err = _attempt_connect(host, 1)+ assert not isinstance(err, PolicyDeniedError), (+ f"PUBLIC_ONLY wrongly blocked {host} at audit hook: {err!r}"+ )+++def test_both_scope_is_a_noop_at_the_audit_hook():+ """BOTH is the permissive default — every classified host is fine.+ The audit hook has nothing to enforce.+ """+ ctx = _ctx(EgressScope.BOTH)+ with active_egress_context(ctx):+ for host in ("8.8.8.8", "192.168.42.1", "127.0.0.1"):+ err = _attempt_connect(host, 1)+ assert not isinstance(err, PolicyDeniedError), (+ f"BOTH scope wrongly blocked {host}: {err!r}"+ )+++def test_strict_scope_blocks_public_host():+ """STRICT permits private hosts (the user's local engine) but+ refuses public ones — same contract as evaluate_url.+ """+ ctx = _ctx(EgressScope.STRICT)+ with active_egress_context(ctx):+ err = _attempt_connect("8.8.8.8", 53)+ assert isinstance(err, PolicyDeniedError)+ assert err.decision.reason == "strict_public_host"+++def test_strict_scope_allows_private_host():+ ctx = _ctx(EgressScope.STRICT)+ with active_egress_context(ctx):+ err = _attempt_connect("127.0.0.1", 1)+ assert not isinstance(err, PolicyDeniedError), (+ f"STRICT wrongly blocked localhost: {err!r}"+ )+++def test_ipv6_loopback_allowed_under_private_only():+ """IPv6 socket family must take the same code path. ``::1`` is the+ IPv6 loopback — has to be classified as private.+ """+ ctx = _ctx(EgressScope.PRIVATE_ONLY)+ with active_egress_context(ctx):+ sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)+ sock.settimeout(0.5)+ try:+ sock.connect(("::1", 1))+ err = None+ except PolicyDeniedError as exc:+ err = exc+ except OSError as exc:+ err = exc+ finally:+ sock.close()+ assert not isinstance(err, PolicyDeniedError), (+ f"IPv6 loopback wrongly blocked under PRIVATE_ONLY: {err!r}"+ )+++def test_af_unix_not_gated():+ """AF_UNIX is off-network. The hook must not classify or block —+ sqlite, syslog, dbus, etc. would all break otherwise.+ """+ import tempfile+ from pathlib import Path++ ctx = _ctx(EgressScope.PRIVATE_ONLY)+ # Pick a path that does not exist so connect errors with ENOENT.+ # The point is to assert no PolicyDeniedError, not a successful+ # connection.+ tmpdir = Path(tempfile.mkdtemp())+ sock_path = str(tmpdir / "nonexistent.sock")+ try:+ with active_egress_context(ctx):+ sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)+ sock.settimeout(0.5)+ try:+ sock.connect(sock_path)+ err = None+ except PolicyDeniedError as exc:+ err = exc+ except OSError as exc:+ err = exc+ finally:+ sock.close()+ assert not isinstance(err, PolicyDeniedError), (+ f"AF_UNIX wrongly gated: {err!r}"+ )+ finally:+ tmpdir.rmdir()+++# ---------------------------------------------------------------------------+# 3. Thread isolation+# ---------------------------------------------------------------------------+++def test_thread_local_context_does_not_leak_across_threads():+ """Setting a PRIVATE_ONLY context in one thread must not block a+ sibling thread's public-IP connect — the thread-local design is+ what makes concurrent multi-research-per-process safe.+ """+ ctx = _ctx(EgressScope.PRIVATE_ONLY)+ set_active_context(ctx)++ sibling_result = []++ def sibling_work():+ # Sibling has no active context → connect should not raise+ # PolicyDeniedError. We capture whatever exception it does+ # raise and assert it is not a policy denial.+ err = _attempt_connect("8.8.8.8", 53)+ sibling_result.append(err)++ t = threading.Thread(target=sibling_work)+ t.start()+ t.join()++ assert sibling_result, "sibling thread did not run"+ assert not isinstance(sibling_result[0], PolicyDeniedError), (+ f"main-thread context leaked into sibling: {sibling_result[0]!r}"+ )++ # Main thread still sees the context.+ assert get_active_context() is ctx+++def test_concurrent_threads_can_have_different_scopes():+ """Two threads with two different scopes must each see their own+ contract — the policy is per-thread, not per-process.+ """+ results = {}++ def worker(scope, host, label):+ ctx = _ctx(scope)+ with active_egress_context(ctx):+ results[label] = _attempt_connect(host, 1)++ # PRIVATE_ONLY thread reaches a public IP → must be blocked.+ # STRICT thread reaches a public IP → must also be blocked. Using+ # two different enforcing scopes makes sure each thread is reading+ # ITS context, not someone else's.+ t_private = threading.Thread(+ target=worker,+ args=(EgressScope.PRIVATE_ONLY, "8.8.8.8", "private"),+ )+ t_strict = threading.Thread(+ target=worker,+ args=(EgressScope.STRICT, "1.1.1.1", "strict"),+ )+ t_private.start()+ t_strict.start()+ t_private.join()+ t_strict.join()++ assert isinstance(results["private"], PolicyDeniedError), (+ f"PRIVATE_ONLY thread did not block public IP: {results['private']!r}"+ )+ assert isinstance(results["strict"], PolicyDeniedError), (+ f"STRICT thread did not block public IP: {results['strict']!r}"+ )+ # Both threads should have cleared their context on exit.+ assert get_active_context() is None+++# ---------------------------------------------------------------------------+# Re-entrancy guard+# ---------------------------------------------------------------------------+++class _FakeSock:+ """Minimal stand-in carrying just the .family the hook inspects."""++ def __init__(self, family=socket.AF_INET):+ self.family = family+++def test_reentrancy_guard_makes_hook_a_noop_while_active():+ """The hook sets _hook_reentry.active around its own evaluate path so+ its internal DNS/socket work can't recursively re-trigger itself. When+ the flag is set, the hook must short-circuit even for a context+address+ that would otherwise be denied.++ Regression: the re-entrancy guard had no direct coverage. Without it+ the hook would recurse (or self-deny) on its own lookups.+ """+ from local_deep_research.security.egress.audit_hook import (+ _audit_hook,+ _hook_reentry,+ )++ # PRIVATE_ONLY would normally block a public IP at the hook.+ set_active_context(_ctx(EgressScope.PRIVATE_ONLY))+ args = (_FakeSock(), ("8.8.8.8", 80))++ # Guard active => no-op, even though 8.8.8.8 is public under PRIVATE_ONLY.+ _hook_reentry.active = True+ try:+ # Must NOT raise.+ _audit_hook("socket.connect", args)+ finally:+ _hook_reentry.active = False++ # Sanity: with the guard cleared, the SAME call is denied — proving the+ # guard (not some other no-op condition) is what suppressed the block.+ with pytest.raises(PolicyDeniedError):+ _audit_hook("socket.connect", args)+++# ---------------------------------------------------------------------------+# Review round 2 (audit-hook correctness) — regression tests+# ---------------------------------------------------------------------------++from local_deep_research.security.egress.audit_hook import ( # noqa: E402+ _audit_hook,+ _extract_host,+)+++def test_bytes_host_is_classified_not_bypassed():+ """R2 #11: CPython fires socket.connect with a bytes host; the hook must+ decode and classify it, not pass it through. A bytes metadata literal+ under PRIVATE_ONLY must be blocked."""+ assert _extract_host((b"169.254.169.254", 80)) == "169.254.169.254"+ assert _extract_host((b"10.0.0.5", 80)) == "10.0.0.5"++ set_active_context(_ctx(EgressScope.PRIVATE_ONLY))+ # bytes public host under PRIVATE_ONLY must be denied (was a bypass).+ with pytest.raises(PolicyDeniedError):+ _audit_hook("socket.connect", (_FakeSock(), (b"8.8.8.8", 80)))+++def test_extract_host_undecodable_bytes_passes_through():+ """Non-ASCII bytes host (not a real IP/host) returns None — preserves the+ 'failing to extract is never a reason to block' contract."""+ assert _extract_host((b"\xff\xfe", 80)) is None+++def test_set_active_context_rejects_adaptive_scope():+ """R2 #1: an unresolved ADAPTIVE context must not be storable — the hook+ only gates PRIVATE_ONLY/STRICT, so ADAPTIVE would silently disarm it.+ Fail fast instead."""+ with pytest.raises(ValueError):+ set_active_context(_ctx(EgressScope.ADAPTIVE))+ # Nothing was stored.+ assert get_active_context() is None+++def test_active_egress_context_nests_and_restores_parent():+ """R2 #13: a nested active_egress_context must restore the parent on exit,+ not wipe it (which would leave the parent's remaining work unprotected)."""+ parent = _ctx(EgressScope.PRIVATE_ONLY)+ child = _ctx(EgressScope.STRICT)+ with active_egress_context(parent):+ assert get_active_context() is parent+ with active_egress_context(child):+ assert get_active_context() is child+ # Parent must be restored, not cleared.+ assert get_active_context() is parent+ assert get_active_context() is Nonediff --git a/tests/security/test_egress_backstop_rearm_followup.py b/tests/security/test_egress_backstop_rearm_followup.pynew file mode 100644index 0000000000..f6957be1a1--- /dev/null+++ b/tests/security/test_egress_backstop_rearm_followup.py@@ -0,0 +1,306 @@+"""Regression tests for the audit-hook backstop *re-arming* paths.++The egress audit hook reads a per-thread ``EgressContext``. ``threading.local``+is NOT inherited by pool / scheduler worker threads, so several call-sites must+explicitly *re-arm* the backstop on the worker:++* ``BackgroundJobScheduler._arm_egress_backstop`` builds a context from the+ user's saved settings and arms it on the APScheduler worker thread.+* ``parallel_search_engine`` captures the submitter's context once and+ re-applies it per pool task (clearing it in a finally so it does not leak to+ the next task on a reused worker).+* ``news_strategy`` captures the context and re-arms it via+ ``active_egress_context`` inside a ThreadPoolExecutor worker.++These tests pin the *real* end-to-end behaviour of those re-arm paths and the+two extraction edge cases (bytearray host, IPv6 4-tuple) that the existing+suites do not cover. They intentionally do NOT duplicate the unit cases already+in ``test_egress_audit_hook.py`` (ADAPTIVE rejection, bytes-under-PRIVATE_ONLY,+nesting) or ``test_scheduler.py`` (mocked private_only arm, bad-settings swallow).+"""++from __future__ import annotations++import concurrent.futures+import socket++from unittest.mock import MagicMock++import pytest++from local_deep_research.security import install_audit_hook+from local_deep_research.security.egress.audit_hook import (+ _audit_hook,+ _extract_host,+ active_egress_context,+ clear_active_context,+ get_active_context,+ set_active_context,+)+from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+ PolicyDeniedError,+)+++# ---------------------------------------------------------------------------+# Helpers+# ---------------------------------------------------------------------------+++def make_ctx(+ scope: EgressScope = EgressScope.BOTH,+ primary: str = "wikipedia",+ require_local_llm: bool = False,+ require_local_embeddings: bool = False,+) -> EgressContext:+ """Mirror of ``make_ctx`` in test_egress_policy.py."""+ return EgressContext(+ scope=scope,+ primary_engine=primary,+ require_local_llm=require_local_llm,+ require_local_embeddings=require_local_embeddings,+ )++[email protected](autouse=True)+def _clear_context_between_tests():+ """The hook reads a process-wide thread-local; a context leaking across+ tests (or from an armed scheduler test) would mask the next test's setup.+ """+ clear_active_context()+ yield+ clear_active_context()+++def _probe(host: str, port: int = 53, timeout: float = 0.5):+ """Open a real AF_INET socket and attempt connect. The installed audit+ hook fires BEFORE the SYN, so the kernel-level outcome is irrelevant; we+ only care whether ``PolicyDeniedError`` was raised. Returns the raised+ exception, or None if the connect somehow returned.+ """+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)+ sock.settimeout(timeout)+ try:+ sock.connect((host, port))+ return None+ except PolicyDeniedError as exc:+ return exc+ except OSError as exc:+ return exc+ finally:+ sock.close()+++def _fake_settings_manager(snapshot, tool="auto"):+ sm = MagicMock()+ sm.get_settings_snapshot.return_value = snapshot+ sm.get_setting.side_effect = lambda k, d=None: (+ tool if k == "search.tool" else d+ )+ return sm+++# ---------------------------------------------------------------------------+# 1. BackgroundJobScheduler._arm_egress_backstop — REAL wiring (not mocked)+# ---------------------------------------------------------------------------+++def test_arm_backstop_really_arms_enforcing_context_and_blocks_public():+ """End-to-end: a private_only snapshot must arm a context that the+ *installed* hook actually enforces — a public connect on the scheduler+ thread is blocked. The existing scheduler test mocks set_active_context, so+ it never proves the real thread-local + installed hook actually engage.++ Also verifies the username is threaded into the built context (a settings+ DB lookup for engine classification depends on it).+ """+ install_audit_hook()+ from local_deep_research.scheduler.background import BackgroundJobScheduler++ sched = BackgroundJobScheduler()+ sm = _fake_settings_manager(+ {"policy.egress_scope": "private_only", "search.tool": "library"},+ tool="library",+ )++ sched._arm_egress_backstop(sm, "alice")++ ctx = get_active_context()+ assert ctx is not None, "backstop did not arm a context"+ assert ctx.scope == EgressScope.PRIVATE_ONLY+ assert ctx.username == "alice", "username not threaded into context"++ # The armed context must be live: a real public connect is denied.+ err = _probe("8.8.8.8", 53)+ assert isinstance(err, PolicyDeniedError), (+ f"armed PRIVATE_ONLY backstop did not block public host: {err!r}"+ )+ assert err.decision.reason == "scope_mismatch_private_only"+++def test_arm_backstop_resolves_adaptive_before_activation():+ """An ``adaptive`` scope must be RESOLVED to a concrete scope by+ context_from_snapshot before set_active_context stores it. If resolution+ regressed and a raw ADAPTIVE context reached set_active_context, that call+ raises ValueError (swallowed best-effort) and NOTHING would be armed.++ adaptive + a meta-picker primary ("auto") resolves to BOTH — concrete and+ storable. We assert a context IS armed and it is not ADAPTIVE.+ """+ install_audit_hook()+ from local_deep_research.scheduler.background import BackgroundJobScheduler++ sched = BackgroundJobScheduler()+ sm = _fake_settings_manager(+ {"policy.egress_scope": "adaptive", "search.tool": "auto"},+ tool="auto",+ )++ sched._arm_egress_backstop(sm, "bob")++ ctx = get_active_context()+ assert ctx is not None, (+ "adaptive backstop armed nothing — ADAPTIVE likely reached "+ "set_active_context and was rejected"+ )+ assert ctx.scope == EgressScope.BOTH+ assert ctx.scope != EgressScope.ADAPTIVE+++# ---------------------------------------------------------------------------+# 2. Pool / worker capture-then-rearm pattern (parallel_search / news_strategy)+# ---------------------------------------------------------------------------+++def test_news_strategy_rearm_pattern_enforces_in_worker_and_preserves_parent():+ """Mirror news_strategy: capture the parent thread's context with+ get_active_context(), then re-arm it in a worker via active_egress_context.+ The worker must enforce the captured PRIVATE_ONLY scope (a fresh worker+ thread does NOT inherit the thread-local), and the parent's context must be+ intact afterwards (the worker had no parent context, so the manager clears+ on exit without disturbing the submitter thread).+ """+ install_audit_hook()+ set_active_context(make_ctx(EgressScope.PRIVATE_ONLY))+ captured = get_active_context()++ result = {}++ def _worker():+ # Worker thread starts with NO inherited context.+ result["before"] = get_active_context()+ with active_egress_context(captured):+ result["err"] = _probe("8.8.8.8", 53)+ result["after"] = get_active_context()++ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:+ ex.submit(_worker).result()++ assert result["before"] is None, "thread-local leaked into fresh worker"+ assert isinstance(result["err"], PolicyDeniedError), (+ f"re-armed worker did not enforce PRIVATE_ONLY: {result['err']!r}"+ )+ assert result["err"].decision.reason == "scope_mismatch_private_only"+ assert result["after"] is None, "worker did not clear re-armed context"++ # Submitter thread's context is untouched by the worker's re-arm/clear.+ assert get_active_context() is captured+++def test_pool_wrapper_clears_context_so_it_does_not_leak_to_next_task():+ """Mirror parallel_search_engine._run_with_context: a pool worker re-arms+ the captured context, runs, then clears in a finally. The CRITICAL property+ is that a SECOND task landing on the SAME reused worker thread (with no+ captured context) is not silently governed by the first task's PRIVATE_ONLY+ policy. max_workers=1 forces both tasks onto one thread.+ """+ install_audit_hook()+ private_ctx = make_ctx(EgressScope.PRIVATE_ONLY)++ def _pool_task(captured_ctx, host):+ # Re-arm only if a context was captured (matches the production guard).+ if captured_ctx is not None:+ set_active_context(captured_ctx)+ try:+ return _probe(host, 53)+ finally:+ if captured_ctx is not None:+ clear_active_context()++ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:+ first = ex.submit(_pool_task, private_ctx, "8.8.8.8").result()+ # Second task: no captured context -> wrapper must not re-arm anything.+ second = ex.submit(_pool_task, None, "8.8.8.8").result()++ assert isinstance(first, PolicyDeniedError), (+ f"first task did not enforce PRIVATE_ONLY: {first!r}"+ )+ assert not isinstance(second, PolicyDeniedError), (+ "first task's PRIVATE_ONLY context leaked to the next task on the "+ f"reused worker: {second!r}"+ )+++def test_rearm_preserves_allow_side_private_host():+ """The re-arm path must not over-block: a re-armed PRIVATE_ONLY context in+ a worker still ALLOWS a private host (the local LLM / settings DB). Pairs+ with the deny-side worker test above so a "deny everything" regression in+ the worker path is caught.+ """+ install_audit_hook()+ captured = make_ctx(EgressScope.PRIVATE_ONLY)+ result = {}++ def _worker():+ with active_egress_context(captured):+ # 127.0.0.1 is private -> allowed; only a non-policy OSError may+ # surface (refused/timeout), never PolicyDeniedError.+ result["err"] = _probe("127.0.0.1", 1)++ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:+ ex.submit(_worker).result()++ assert not isinstance(result["err"], PolicyDeniedError), (+ f"re-armed PRIVATE_ONLY wrongly blocked a private host: {result['err']!r}"+ )+++# ---------------------------------------------------------------------------+# 3. _extract_host edge cases not covered elsewhere+# ---------------------------------------------------------------------------+++def test_extract_host_decodes_bytearray():+ """CPython may fire socket.connect with a bytearray host; the existing+ suite only covers ``bytes``. A bytearray must decode identically, else a+ bytearray-host caller would bypass the PRIVATE_ONLY/STRICT backstop.+ """+ assert _extract_host((bytearray(b"8.8.8.8"), 80)) == "8.8.8.8"+ assert _extract_host((bytearray(b"10.0.0.5"), 443)) == "10.0.0.5"+++def test_extract_host_ipv6_four_tuple():+ """AF_INET6 connect addresses are ``(host, port, flowinfo, scope_id)``.+ _extract_host must read the host from a 4-tuple, not only a 2-tuple.+ """+ assert _extract_host(("::1", 0, 0, 0)) == "::1"+ assert _extract_host(("2001:4860:4860::8888", 53, 0, 0)) == (+ "2001:4860:4860::8888"+ )+++def test_audit_hook_blocks_bytes_public_host_under_strict():+ """The existing bytes-host test only covers PRIVATE_ONLY. STRICT also bans+ public hosts, and the bytes-decode path must apply there too — a bytes+ public host under STRICT must be denied with the STRICT reason.+ """+ set_active_context(make_ctx(EgressScope.STRICT))+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)+ try:+ with pytest.raises(PolicyDeniedError) as excinfo:+ _audit_hook("socket.connect", (sock, (b"8.8.8.8", 80)))+ finally:+ sock.close()+ assert excinfo.value.decision.reason == "strict_public_host"diff --git a/tests/security/test_egress_cache_and_fullsearch.py b/tests/security/test_egress_cache_and_fullsearch.pynew file mode 100644index 0000000000..308fbb2b41--- /dev/null+++ b/tests/security/test_egress_cache_and_fullsearch.py@@ -0,0 +1,304 @@+"""Integration tests for two real egress Policy Enforcement Points:++1. ``utilities/search_cache.py`` — the N17 fix that folds the active egress+ scope into the cache key, so a warm cache populated under one scope can+ never be served to a query running under a stricter scope+ (cross-scope cache poisoning / policy bypass via warm cache).++2. ``web_search_engines/engines/full_search.py`` — per-URL egress gating in+ the full-content fetch path: when an ``egress_context`` is present, URLs+ that ``evaluate_url`` denies for the scope are dropped before any network+ fetch, allowed ones are kept, and the denial audit log is URL-redacted.++These drive the REAL call-site code. Only unavoidable heavy deps are mocked+(the network ``batch_fetch_and_extract`` and the SSRF ``validate_url``, so the+SSRF axis is isolated from the orthogonal egress-scope axis under test). The+egress decision itself (``evaluate_url``) is exercised for real against IP+literals, so no DNS / network is required.+"""++from __future__ import annotations++import pytest+from loguru import logger++from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+)+from local_deep_research.utilities.search_cache import SearchCache+from local_deep_research.utilities.thread_context import search_context+from local_deep_research.web_search_engines.engines import full_search as fs+++# ---------------------------------------------------------------------------+# Helpers+# ---------------------------------------------------------------------------+++def _scope_context(scope: str, username: str = "alice") -> dict:+ """Mirror the real search-context shape the cache reads scope from."""+ return {+ "username": username,+ "settings_snapshot": {"policy.egress_scope": {"value": scope}},+ }+++def _make_ctx(scope: EgressScope) -> EgressContext:+ return EgressContext(+ scope=scope,+ primary_engine="library",+ require_local_llm=False,+ require_local_embeddings=False,+ )+++class _FakeWebSearch:+ """Minimal stand-in for the inner web search engine (``.invoke``)."""++ def __init__(self, results):+ self._results = results++ def invoke(self, query): # noqa: D401 - protocol method+ return list(self._results)+++# Public + private IP literals resolve locally (no DNS / network) and match+# the literal-classification path the existing edge-case tests use.+_PUBLIC_URL = "http://93.184.216.34/page"+_PRIVATE_URL = "http://10.0.0.5/api"+++# ===========================================================================+# (1) search_cache scope keying — no cross-scope cache poisoning+# ===========================================================================+++def test_cache_entry_not_served_across_scopes(tmp_path):+ """A result cached under PRIVATE_ONLY must NOT be returned to the same+ query running under PUBLIC_ONLY or BOTH (and vice-versa); the same+ query under the SAME scope still hits."""+ cache = SearchCache(cache_dir=str(tmp_path))+ try:+ results = [{"title": "secret", "link": _PRIVATE_URL}]++ with search_context(_scope_context("private_only")):+ assert cache.put("quarterly numbers", results) is True+ # Same scope, same query -> hit.+ assert cache.get("quarterly numbers") == results++ # Different scope -> the private-scope entry must be invisible.+ with search_context(_scope_context("public_only")):+ assert cache.get("quarterly numbers") is None+ with search_context(_scope_context("both")):+ assert cache.get("quarterly numbers") is None++ # Back to the original scope -> still a hit (key is stable per scope).+ with search_context(_scope_context("private_only")):+ assert cache.get("quarterly numbers") == results+ finally:+ cache.dispose()+++def test_cache_does_not_leak_public_results_into_private_scope(tmp_path):+ """Mirror direction: PUBLIC_ONLY-cached results must not satisfy a+ PRIVATE_ONLY lookup."""+ cache = SearchCache(cache_dir=str(tmp_path))+ try:+ pub_results = [{"title": "public", "link": _PUBLIC_URL}]+ with search_context(_scope_context("public_only")):+ assert cache.put("market data", pub_results) is True+ assert cache.get("market data") == pub_results++ with search_context(_scope_context("private_only")):+ assert cache.get("market data") is None+ finally:+ cache.dispose()+++def test_query_hash_distinct_per_scope_stable_within_scope(tmp_path):+ """The hash folds scope: differs across scopes, identical within a scope."""+ cache = SearchCache(cache_dir=str(tmp_path))+ try:+ with search_context(_scope_context("private_only")):+ h_priv = cache._get_query_hash("same query")+ with search_context(_scope_context("public_only")):+ h_pub = cache._get_query_hash("same query")+ with search_context(_scope_context("both")):+ h_both = cache._get_query_hash("same query")+ with search_context(_scope_context("private_only")):+ h_priv_again = cache._get_query_hash("same query")++ # All three scopes produce distinct keys.+ assert len({h_priv, h_pub, h_both}) == 3+ # Same scope + same query is deterministic.+ assert h_priv == h_priv_again+ finally:+ cache.dispose()+++def test_no_context_hashes_identically_to_explicit_both(tmp_path):+ """Documented invariant: "no policy snapshot" and an explicit BOTH scope+ (the permissive default) must hash identically, so unset-context runs and+ explicit-BOTH runs share one cache partition rather than fragmenting."""+ cache = SearchCache(cache_dir=str(tmp_path))+ try:+ # No search context at all.+ h_none = cache._get_query_hash("topic")+ # Explicit BOTH with an empty username (the no-context default).+ with search_context(+ {+ "username": "",+ "settings_snapshot": {"policy.egress_scope": {"value": "both"}},+ }+ ):+ h_both = cache._get_query_hash("topic")+ assert h_none == h_both+ finally:+ cache.dispose()+++# ===========================================================================+# (2) full_search per-URL egress gating+# ===========================================================================+++def patched_fetch(monkeypatch):+ """Capture the URLs that survive the gate and reach the fetcher; isolate+ the SSRF axis by forcing ``validate_url`` True so only egress scope+ decides which URLs pass."""+ captured = {"urls": None, "calls": 0}++ def fake_batch(urls, **kwargs):+ captured["calls"] += 1+ captured["urls"] = list(urls)+ return {u: "content-for-" + u for u in urls}++ monkeypatch.setattr(fs, "batch_fetch_and_extract", fake_batch)+ monkeypatch.setattr(fs, "validate_url", lambda u: True)+ return captured+++def _engine(results, ctx):+ return fs.FullSearchResults(+ llm=None, # llm None -> check_urls returns results unfiltered+ web_search=_FakeWebSearch(results),+ egress_context=ctx,+ )+++def test_run_private_only_drops_public_keeps_private(patched_fetch):+ results = [+ {"title": "pub", "link": _PUBLIC_URL},+ {"title": "priv", "link": _PRIVATE_URL},+ ]+ engine = _engine(results, _make_ctx(EgressScope.PRIVATE_ONLY))+ out = engine.run("q")++ # Only the private URL reached the fetcher.+ assert patched_fetch["urls"] == [_PRIVATE_URL]+ # And only its result carries full content; the denied public one is None.+ by_link = {r["link"]: r for r in out}+ assert (+ by_link[_PRIVATE_URL]["full_content"] == "content-for-" + _PRIVATE_URL+ )+ assert by_link[_PUBLIC_URL]["full_content"] is None+++def test_run_public_only_drops_private_keeps_public(patched_fetch):+ results = [+ {"title": "pub", "link": _PUBLIC_URL},+ {"title": "priv", "link": _PRIVATE_URL},+ ]+ engine = _engine(results, _make_ctx(EgressScope.PUBLIC_ONLY))+ out = engine.run("q")++ assert patched_fetch["urls"] == [_PUBLIC_URL]+ by_link = {r["link"]: r for r in out}+ assert by_link[_PUBLIC_URL]["full_content"] == "content-for-" + _PUBLIC_URL+ assert by_link[_PRIVATE_URL]["full_content"] is None+++def test_run_all_denied_skips_fetch_entirely(patched_fetch):+ """PRIVATE_ONLY with only public URLs -> nothing passes the gate, the+ network fetcher is never invoked, all results get null content."""+ results = [{"title": "pub", "link": _PUBLIC_URL}]+ engine = _engine(results, _make_ctx(EgressScope.PRIVATE_ONLY))+ out = engine.run("q")++ assert patched_fetch["calls"] == 0+ assert out[0]["full_content"] is None+++def test_run_without_egress_context_does_not_gate(patched_fetch):+ """Control: with no egress_context the per-URL scope gate is inactive, so+ both URLs reach the fetcher. This proves the gate above is driven by the+ egress context and not by some unrelated filter."""+ results = [+ {"title": "pub", "link": _PUBLIC_URL},+ {"title": "priv", "link": _PRIVATE_URL},+ ]+ engine = fs.FullSearchResults(+ llm=None, web_search=_FakeWebSearch(results), egress_context=None+ )+ engine.run("q")++ assert patched_fetch["urls"] == [_PUBLIC_URL, _PRIVATE_URL]+++def test_get_full_content_gates_per_url(patched_fetch):+ """The secondary ``_get_full_content`` path enforces the same per-URL+ scope gate."""+ items = [+ {"title": "pub", "link": _PUBLIC_URL},+ {"title": "priv", "link": _PRIVATE_URL},+ ]+ engine = _engine(items, _make_ctx(EgressScope.PRIVATE_ONLY))+ out = engine._get_full_content(items)++ assert patched_fetch["urls"] == [_PRIVATE_URL]+ by_link = {r["link"]: r for r in out}+ assert (+ by_link[_PRIVATE_URL]["full_content"] == "content-for-" + _PRIVATE_URL+ )+ assert by_link[_PUBLIC_URL]["full_content"] is None+++def test_denied_url_audit_log_is_redacted(patched_fetch):+ """The denial audit record must carry only scheme://host (no path / query /+ token), proving sensitive URL components are not leaked into logs."""+ sensitive_url = "http://93.184.216.34/secret/report?token=SUPERSECRET"+ results = [{"title": "pub", "link": sensitive_url}]++ records = []++ def sink(message):+ records.append(message.record)++ # The package disables its own loguru namespace in __init__; enable it so+ # the in-module audit warnings actually reach our sink. Restore in finally.+ logger.enable("local_deep_research")+ sink_id = logger.add(sink, level="WARNING")+ try:+ engine = _engine(results, _make_ctx(EgressScope.PRIVATE_ONLY))+ engine.run("q")+ finally:+ logger.remove(sink_id)+ logger.disable("local_deep_research")++ # Find the policy-audit denial record for this URL.+ audit = [+ r+ for r in records+ if r["extra"].get("policy_audit") and "url" in r["extra"]+ ]+ assert audit, "expected a policy_audit denial record"+ rec = audit[0]+ logged_url = rec["extra"]["url"]+ assert logged_url == "http://93.184.216.34"+ # The token / path must appear nowhere in the logged URL or message.+ assert "SUPERSECRET" not in logged_url+ assert "SUPERSECRET" not in rec["message"]+ assert "/secret" not in logged_url+ assert rec["extra"]["reason"] == "scope_mismatch_private_only"diff --git a/tests/security/test_egress_collection_semantics_followup.py b/tests/security/test_egress_collection_semantics_followup.pynew file mode 100644index 0000000000..f96703f82a--- /dev/null+++ b/tests/security/test_egress_collection_semantics_followup.py@@ -0,0 +1,310 @@+"""Follow-up regression tests for the NEW collection classification semantics.++A collection (``collection_<uuid>`` or the aggregate ``library``) is ALWAYS a+local knowledge base, and its ``is_public`` flag is ADDITIVE — marking a+collection public ALSO makes it eligible under PUBLIC_ONLY/cloud inference, but+NEVER removes its private (local) eligibility. Concretely, ``_engine_bucket``+returns ``(is_public, is_local=True)`` for any collection.++These tests pin that bucket shape directly, the DB-resolution fallback when no+metadata is supplied, and the ADAPTIVE-scope resolution when the PRIMARY engine+is a collection (public -> BOTH, private -> PRIVATE_ONLY forcing local+inference). They complement (do not duplicate) the metadata-driven evaluate_engine+cases already covered in tests/security/test_egress_policy.py by attacking the+``_engine_bucket`` / ``_resolve_collection_is_public`` / ``_resolve_adaptive_scope``+functions directly and the DB-lookup path.+"""++from __future__ import annotations++from unittest.mock import MagicMock, patch++from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+ _engine_bucket,+ _resolve_adaptive_scope,+ _resolve_collection_is_public,+ context_from_snapshot,+ evaluate_engine,+)+++# ---------------------------------------------------------------------------+# Helpers (mirrors make_ctx in tests/security/test_egress_policy.py)+# ---------------------------------------------------------------------------+++def make_ctx(+ scope: EgressScope = EgressScope.BOTH,+ primary: str = "arxiv",+ require_local_llm: bool = False,+ require_local_embeddings: bool = False,+ local_hostnames=(),+ username=None,+) -> EgressContext:+ return EgressContext(+ scope=scope,+ primary_engine=primary,+ require_local_llm=require_local_llm,+ require_local_embeddings=require_local_embeddings,+ local_hostnames=tuple(local_hostnames),+ username=username,+ )+++_POLICY = "local_deep_research.security.egress.policy"+++# ---------------------------------------------------------------------------+# _engine_bucket — the (is_public, is_local=True) contract+# ---------------------------------------------------------------------------+++def test_engine_bucket_public_collection_is_public_AND_local():+ """A public collection buckets as (True, True): the public flag is+ ADDITIVE, it never strips the local-KB eligibility."""+ ctx = make_ctx(primary="collection_pub")+ bucket = _engine_bucket(+ "collection_pub", ctx, {}, metadata={"is_public": True}+ )+ assert bucket == (True, True)+++def test_engine_bucket_private_collection_is_local_only():+ """A private collection buckets as (False, True) — local, not public."""+ ctx = make_ctx(primary="collection_priv")+ bucket = _engine_bucket(+ "collection_priv", ctx, {}, metadata={"is_public": False}+ )+ assert bucket == (False, True)+++def test_engine_bucket_library_is_local_only_via_db_default():+ """``library`` is the always-private aggregate; with no metadata it+ resolves via the (always-False) collection resolver -> (False, True)."""+ ctx = make_ctx(primary="library")+ bucket = _engine_bucket("library", ctx, {})+ assert bucket == (False, True)+++def test_engine_bucket_metadata_short_circuits_db_lookup():+ """When the caller supplies engine metadata, _engine_bucket must NOT+ fall back to the DB resolver — the metadata flag wins outright."""+ ctx = make_ctx(primary="collection_x")+ with patch(f"{_POLICY}._resolve_collection_is_public") as mock_resolve:+ mock_resolve.return_value = False # would mis-classify if consulted+ bucket = _engine_bucket(+ "collection_x", ctx, {}, metadata={"is_public": True}+ )+ mock_resolve.assert_not_called()+ assert bucket == (True, True)+++def test_engine_bucket_collection_no_metadata_consults_db_resolver():+ """Without metadata, _engine_bucket delegates classification to+ _resolve_collection_is_public and threads is_local=True regardless."""+ ctx = make_ctx(primary="collection_y", username="alice")+ with patch(+ f"{_POLICY}._resolve_collection_is_public", return_value=True+ ) as mock_resolve:+ bucket = _engine_bucket("collection_y", ctx, {})+ mock_resolve.assert_called_once_with("collection_y", "alice")+ assert bucket == (True, True)+++# ---------------------------------------------------------------------------+# evaluate_engine — library across ALL scopes+# ---------------------------------------------------------------------------+++def test_library_engine_evaluate_across_all_scopes():+ """``library`` (always private/local): allowed under PRIVATE_ONLY and+ BOTH, DENIED under PUBLIC_ONLY, allowed under STRICT only when primary."""+ # PRIVATE_ONLY / BOTH -> allowed+ for scope in (EgressScope.PRIVATE_ONLY, EgressScope.BOTH):+ ctx = make_ctx(scope=scope, primary="library")+ assert evaluate_engine("library", ctx, settings_snapshot={}).allowed, (+ f"library should be allowed under {scope}"+ )+ # PUBLIC_ONLY -> denied (it is local-only)+ ctx_pub = make_ctx(scope=EgressScope.PUBLIC_ONLY, primary="library")+ denied = evaluate_engine("library", ctx_pub, settings_snapshot={})+ assert denied.allowed is False+ assert denied.reason == "scope_mismatch_public_only"+ # STRICT + library is primary -> allowed+ ctx_strict_ok = make_ctx(scope=EgressScope.STRICT, primary="library")+ assert evaluate_engine(+ "library", ctx_strict_ok, settings_snapshot={}+ ).allowed+ # STRICT + library is NOT primary -> denied+ ctx_strict_no = make_ctx(scope=EgressScope.STRICT, primary="arxiv")+ d = evaluate_engine("library", ctx_strict_no, settings_snapshot={})+ assert d.allowed is False+ assert d.reason == "strict_not_primary"+++# ---------------------------------------------------------------------------+# evaluate_engine — DB-resolution path (no metadata supplied)+# ---------------------------------------------------------------------------+++def test_collection_db_default_private_denied_under_public_only():+ """A collection resolved to private via the DB path is excluded from+ PUBLIC_ONLY — the additive flag must default closed (private)."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY, primary="collection_z")+ with patch(f"{_POLICY}._resolve_collection_is_public", return_value=False):+ decision = evaluate_engine("collection_z", ctx, settings_snapshot={})+ assert decision.allowed is False+ assert decision.reason == "scope_mismatch_public_only"+++def test_collection_db_public_allowed_under_public_only_without_metadata():+ """A collection the DB marks public is allowed under PUBLIC_ONLY even+ when the caller passes NO metadata (the resolver is consulted)."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY, primary="collection_z")+ with patch(f"{_POLICY}._resolve_collection_is_public", return_value=True):+ decision = evaluate_engine("collection_z", ctx, settings_snapshot={})+ assert decision.allowed is True+++def test_collection_db_public_still_allowed_under_private_only():+ """Additive semantics: a PUBLIC collection (no metadata, DB path) is+ STILL local, so it remains allowed under PRIVATE_ONLY."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY, primary="collection_z")+ with patch(f"{_POLICY}._resolve_collection_is_public", return_value=True):+ decision = evaluate_engine("collection_z", ctx, settings_snapshot={})+ assert decision.allowed is True+++# ---------------------------------------------------------------------------+# _resolve_collection_is_public — direct unit coverage of the DB path+# ---------------------------------------------------------------------------+++def test_resolve_collection_is_public_library_always_private():+ """``library`` short-circuits to False without any DB access."""+ assert _resolve_collection_is_public("library", "alice") is False+++def test_resolve_collection_is_public_non_collection_name_false():+ """A non-collection engine name short-circuits to False."""+ assert _resolve_collection_is_public("arxiv", "alice") is False+++def test_resolve_collection_is_public_reads_db_row_true():+ """The DB path returns the row's is_public flag (True case)."""+ mock_row = MagicMock()+ mock_row.is_public = True+ mock_session = MagicMock()+ mock_session.query.return_value.filter.return_value.first.return_value = (+ mock_row+ )+ cm = MagicMock()+ cm.__enter__.return_value = mock_session+ cm.__exit__.return_value = False+ with patch(+ "local_deep_research.database.session_context.get_user_db_session",+ return_value=cm,+ ):+ result = _resolve_collection_is_public("collection_pub123", "alice")+ assert result is True+++def test_resolve_collection_is_public_missing_row_fails_closed_private():+ """No matching collection row -> False (private), not an error."""+ mock_session = MagicMock()+ mock_session.query.return_value.filter.return_value.first.return_value = (+ None+ )+ cm = MagicMock()+ cm.__enter__.return_value = mock_session+ cm.__exit__.return_value = False+ with patch(+ "local_deep_research.database.session_context.get_user_db_session",+ return_value=cm,+ ):+ result = _resolve_collection_is_public("collection_missing", "alice")+ assert result is False+++def test_resolve_collection_is_public_db_error_fails_closed_private():+ """Any DB error fails closed to private (False) — never public."""+ with patch(+ "local_deep_research.database.session_context.get_user_db_session",+ side_effect=RuntimeError("db down"),+ ):+ result = _resolve_collection_is_public("collection_err", "alice")+ assert result is False+++# ---------------------------------------------------------------------------+# _resolve_adaptive_scope — collection primary classification+# ---------------------------------------------------------------------------+++def test_adaptive_public_collection_primary_resolves_to_both():+ """A PUBLIC collection primary buckets (True, True); neither exclusive+ branch fires, so ADAPTIVE resolves to BOTH (cloud inference allowed)."""+ with patch(f"{_POLICY}._resolve_collection_is_public", return_value=True):+ scope = _resolve_adaptive_scope(+ "collection_pub",+ {},+ username="alice",+ local_hostnames=(),+ )+ assert scope == EgressScope.BOTH+++def test_adaptive_private_collection_primary_resolves_to_private_only():+ """A PRIVATE collection primary buckets (False, True) -> PRIVATE_ONLY."""+ with patch(f"{_POLICY}._resolve_collection_is_public", return_value=False):+ scope = _resolve_adaptive_scope(+ "collection_priv",+ {},+ username="alice",+ local_hostnames=(),+ )+ assert scope == EgressScope.PRIVATE_ONLY+++def test_adaptive_library_primary_resolves_to_private_only():+ """``library`` is always private -> ADAPTIVE resolves to PRIVATE_ONLY."""+ scope = _resolve_adaptive_scope(+ "library",+ {},+ username="alice",+ local_hostnames=(),+ )+ assert scope == EgressScope.PRIVATE_ONLY+++# ---------------------------------------------------------------------------+# context_from_snapshot — ADAPTIVE + collection primary end-to-end+# ---------------------------------------------------------------------------+++def test_context_adaptive_public_collection_both_no_force_local():+ """ADAPTIVE + public-collection primary -> resolved scope BOTH and the+ local-inference flags are NOT forced (cloud LLM/embeddings permitted)."""+ snapshot = {"policy.egress_scope": {"value": "adaptive"}}+ with patch(f"{_POLICY}._resolve_collection_is_public", return_value=True):+ ctx = context_from_snapshot(+ snapshot, primary_engine="collection_pub", username="alice"+ )+ assert ctx.scope == EgressScope.BOTH+ assert ctx.require_local_llm is False+ assert ctx.require_local_embeddings is False+++def test_context_adaptive_private_collection_private_forces_local():+ """ADAPTIVE + private-collection primary -> PRIVATE_ONLY, which implies+ (forces) local LLM and local embeddings so the corpus can't leak."""+ snapshot = {"policy.egress_scope": {"value": "adaptive"}}+ with patch(f"{_POLICY}._resolve_collection_is_public", return_value=False):+ ctx = context_from_snapshot(+ snapshot, primary_engine="collection_priv", username="alice"+ )+ assert ctx.scope == EgressScope.PRIVATE_ONLY+ assert ctx.require_local_llm is True+ assert ctx.require_local_embeddings is Truediff --git a/tests/security/test_egress_downloader_threading_followup.py b/tests/security/test_egress_downloader_threading_followup.pynew file mode 100644index 0000000000..f5022e7fff--- /dev/null+++ b/tests/security/test_egress_downloader_threading_followup.py@@ -0,0 +1,199 @@+"""Follow-up regression tests for downloader egress-policy threading.++These cover GAPS not already exercised by+``tests/content_fetcher/test_security.py`` (which only tests the direct+``_apply_egress_policy_to_downloader`` call with PRIVATE_ONLY / PUBLIC_ONLY /+no-context / no-session) and ``tests/security/test_egress_policy.py`` (which+only tests ``policy_aware_validate_url`` for PRIVATE_ONLY-allows-private ++metadata-blocked).++What is added here:+ * the relaxation reaches a downloader through the REAL ``_get_downloader``+ code path AND survives caching (a second fetch reuses the relaxed session);+ * STRICT / BOTH scopes do NOT relax — only PRIVATE_ONLY does, mirroring+ ``policy_aware_validate_url``;+ * a downloader whose ``.session`` lacks ``allow_private_ips`` is handled+ without mutating it or raising;+ * cloud-metadata IPs stay blocked by ``is_ip_blocked`` / ``validate_url``+ even when private IPs are allowed (the relaxation must not open IMDS);+ * ``policy_aware_validate_url`` scope behaviour: PRIVATE_ONLY permits private+ hosts, every other scope (and no context) stays strict.++All tests import and call the real code under test; each is an allow+deny pair+or an explicit edge case that would FAIL if the guarded behaviour regressed.+"""++from __future__ import annotations++from local_deep_research.content_fetcher import ContentFetcher+from local_deep_research.content_fetcher.url_classifier import URLType+from local_deep_research.security.egress.fetch import policy_aware_validate_url+from local_deep_research.security.egress.policy import EgressScope+from local_deep_research.security.ssrf_validator import (+ ALWAYS_BLOCKED_METADATA_IPS,+ is_ip_blocked,+ validate_url,+)++# Reuse the canonical context builder rather than duplicating a fixture.+from tests.security.test_egress_policy import make_ctx+++# ---------------------------------------------------------------------------+# Real _get_downloader path: relaxation applied + survives caching+# ---------------------------------------------------------------------------+++def test_get_downloader_real_path_private_only_relaxes_session():+ """Through the REAL ``_get_downloader`` (not a direct call to the helper),+ a PRIVATE_ONLY context relaxes the constructed downloader's SafeSession."""+ cf = ContentFetcher(egress_context=make_ctx(scope=EgressScope.PRIVATE_ONLY))+ downloader = cf._get_downloader(URLType.PDF)+ assert downloader is not None+ assert downloader.session.allow_private_ips is True+++def test_get_downloader_caches_relaxed_downloader():+ """The cached downloader (returned by a second ``_get_downloader`` call)+ is the SAME object and keeps the relaxation — fetch reuses it per+ fetcher.py, so the relaxation must persist across calls."""+ cf = ContentFetcher(egress_context=make_ctx(scope=EgressScope.PRIVATE_ONLY))+ first = cf._get_downloader(URLType.PDF)+ second = cf._get_downloader(URLType.PDF)+ assert first is second # cache hit, not re-constructed+ assert second.session.allow_private_ips is True+++def test_get_downloader_real_path_strict_does_not_relax():+ """Deny side: under STRICT the real ``_get_downloader`` must leave the+ downloader session strict (only PRIVATE_ONLY relaxes)."""+ cf = ContentFetcher(egress_context=make_ctx(scope=EgressScope.STRICT))+ downloader = cf._get_downloader(URLType.PDF)+ assert downloader is not None+ assert downloader.session.allow_private_ips is False+++# ---------------------------------------------------------------------------+# _apply_egress_policy_to_downloader: scope coverage (STRICT / BOTH) + edge+# ---------------------------------------------------------------------------+++def _fake_downloader_with_session():+ from local_deep_research.security.safe_requests import SafeSession++ class _DL:+ def __init__(self):+ self.session = SafeSession()++ return _DL()+++def test_apply_strict_scope_keeps_session_strict():+ """STRICT must NOT relax — mirrors policy_aware_validate_url, which only+ threads allow_private_ips for PRIVATE_ONLY. (test_security covers+ PUBLIC_ONLY; STRICT is the uncovered scope.)"""+ cf = ContentFetcher(egress_context=make_ctx(scope=EgressScope.STRICT))+ dl = _fake_downloader_with_session()+ cf._apply_egress_policy_to_downloader(dl)+ assert dl.session.allow_private_ips is False+++def test_apply_both_scope_keeps_session_strict():+ """BOTH must NOT relax either: it allows public hosts via normal SSRF+ rules but does not blanket-permit private IPs the way PRIVATE_ONLY does."""+ cf = ContentFetcher(egress_context=make_ctx(scope=EgressScope.BOTH))+ dl = _fake_downloader_with_session()+ cf._apply_egress_policy_to_downloader(dl)+ assert dl.session.allow_private_ips is False+++def test_apply_session_lacking_allow_private_ips_attr_is_handled():+ """A downloader whose ``.session`` is a non-SafeSession object lacking the+ ``allow_private_ips`` attribute must not raise and must not have the+ attribute injected (the helper guards with ``hasattr``)."""+ cf = ContentFetcher(egress_context=make_ctx(scope=EgressScope.PRIVATE_ONLY))++ class _BareSession:+ pass++ class _DL:+ def __init__(self):+ self.session = _BareSession()++ dl = _DL()+ cf._apply_egress_policy_to_downloader(dl) # must not raise+ assert not hasattr(dl.session, "allow_private_ips")+++# ---------------------------------------------------------------------------+# Cloud-metadata IPs stay blocked even when private IPs are allowed+# ---------------------------------------------------------------------------+++def test_is_ip_blocked_metadata_blocked_even_with_allow_private_ips():+ """The relaxation a PRIVATE_ONLY downloader receives is+ ``allow_private_ips=True``; that must NOT open cloud-metadata IPs.+ ``is_ip_blocked`` must still reject every IMDS endpoint."""+ for ip in ALWAYS_BLOCKED_METADATA_IPS:+ assert is_ip_blocked(ip, allow_private_ips=True) is True, (+ f"metadata IP {ip} must stay blocked with allow_private_ips"+ )+++def test_is_ip_blocked_private_allow_deny_pair():+ """Allow+deny pair: an ordinary RFC1918 host is blocked by default but+ permitted once allow_private_ips is set — proving the relaxation flag is+ what unlocks private (and only private) reachability."""+ assert is_ip_blocked("192.168.1.5") is True # strict default+ assert is_ip_blocked("192.168.1.5", allow_private_ips=True) is False+++def test_relaxed_validate_url_still_blocks_metadata():+ """SafeSession.request forwards allow_private_ips into validate_url. With+ the relaxation active, a private host validates but a metadata host is+ still rejected — the exact behaviour a relaxed downloader session relies+ on (allow+deny pair)."""+ assert (+ validate_url("http://192.168.1.5/api", allow_private_ips=True) is True+ )+ assert (+ validate_url(+ "http://169.254.169.254/latest/meta-data/", allow_private_ips=True+ )+ is False+ )+++# ---------------------------------------------------------------------------+# policy_aware_validate_url scope behaviour+# ---------------------------------------------------------------------------+++def test_policy_aware_validate_url_private_only_allows_private_hosts():+ """PRIVATE_ONLY permits loopback and RFC1918 lab hosts (this is the side+ the downloader relaxation mirrors)."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ assert policy_aware_validate_url("http://127.0.0.1:11434", ctx) is True+ assert policy_aware_validate_url("http://192.168.1.5/api", ctx) is True+ assert policy_aware_validate_url("http://10.1.2.3/api", ctx) is True+++def test_policy_aware_validate_url_non_private_scopes_stay_strict():+ """Deny side: STRICT, PUBLIC_ONLY and BOTH must NOT thread+ allow_private_ips, so a private IP literal is rejected under each."""+ private_url = "http://192.168.1.5/api"+ for scope in (+ EgressScope.STRICT,+ EgressScope.PUBLIC_ONLY,+ EgressScope.BOTH,+ ):+ ctx = make_ctx(scope=scope)+ assert policy_aware_validate_url(private_url, ctx) is False, (+ f"private host must be rejected under {scope}"+ )+++def test_policy_aware_validate_url_no_context_is_strict():+ """No egress context falls back to strict validate_url: a private host is+ rejected just like the un-relaxed default."""+ assert policy_aware_validate_url("http://192.168.1.5/api", None) is Falsediff --git a/tests/security/test_egress_e2e_private_only_tripwire.py b/tests/security/test_egress_e2e_private_only_tripwire.pynew file mode 100644index 0000000000..6e3cf7345b--- /dev/null+++ b/tests/security/test_egress_e2e_private_only_tripwire.py@@ -0,0 +1,285 @@+"""End-to-end "nothing leaves the box" tripwire for the egress policy.++This is the marquee integration test: it arms a REAL ``EgressContext`` via+``set_active_context`` (the same call the run orchestrator makes) and then+asserts that *every* layer of the stack agrees on the same decision under one+armed context:++ * the installed PEP 578 audit hook gating raw ``socket.connect`` (the+ last-line backstop that fires BEFORE the SYN is sent), and+ * the PDP call-site evaluators (``evaluate_url`` / ``evaluate_engine`` /+ ``evaluate_llm_endpoint``) that the explicit fetch / engine-factory /+ LLM-factory PEPs consult.++Unlike the focused hook test (``test_egress_audit_hook.py``) and the PDP unit+tests (``test_egress_policy.py``), this file does not pick the layers apart — it+checks that with a single armed PRIVATE_ONLY (and STRICT) context the *whole*+stack refuses public egress while still permitting the user's own loopback /+local services, and that the permissive scopes (PUBLIC_ONLY / BOTH) do NOT arm+the socket-level hook.++No outbound network is ever required: under an armed enforcing context the audit+hook raises ``PolicyDeniedError`` before ``connect`` reaches the kernel, so even+a non-blocking connect to a public IP returns the policy denial with no packets+on the wire.+"""++from __future__ import annotations++import socket++import pytest++from local_deep_research.security import (+ clear_active_context,+ get_active_context,+ install_audit_hook,+ is_audit_hook_installed,+ set_active_context,+)+from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+ PolicyDeniedError,+ evaluate_engine,+ evaluate_llm_endpoint,+ evaluate_url,+)++# A public IP (Google DNS) and a public engine / cloud LLM provider used as the+# "should be denied" probes, plus a loopback IP used as the "should be allowed"+# probe. None of these need a live connection — see module docstring.+_PUBLIC_IP = "8.8.8.8"+_LOOPBACK_IP = "127.0.0.1"+_PUBLIC_ENGINE = "arxiv" # statically classified public in the engine registry+_CLOUD_LLM = "openai" # in _CLOUD_LLM_PROVIDERS+_METADATA_IP = "169.254.169.254" # AWS/GCE IMDS — never permitted, any scope+_METADATA_IP_OCTAL = "0251.0376.0251.0376" # alternate encoding of the above++[email protected](autouse=True)+def _ensure_hook_and_clean_context():+ """The audit hook is install-once-and-keep-forever (PEP 578 hooks cannot be+ removed). Make sure it is installed for the socket-layer assertions, and+ that no context leaks into or out of any test in this file — a leaked+ enforcing context would make an unrelated test's public connect raise.+ """+ install_audit_hook()+ clear_active_context()+ yield+ clear_active_context()+++def _make_ctx(+ scope: EgressScope,+ *,+ require_local_llm: bool = False,+ require_local_embeddings: bool = False,+) -> EgressContext:+ """Build a fresh minimal context for ``scope``. Fresh per test so the+ per-run denial quota / DNS cache never carries across tests.+ """+ return EgressContext(+ scope=scope,+ primary_engine="wikipedia",+ require_local_llm=require_local_llm,+ require_local_embeddings=require_local_embeddings,+ )+++def _attempt_connect(host: str, port: int = 80, timeout: float = 0.5):+ """Open a raw AF_INET socket and connect; return the raised exception (or+ None). We only care whether the audit hook raised ``PolicyDeniedError`` —+ the kernel-level outcome (refused / timeout) is irrelevant because the hook+ fires first.+ """+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)+ sock.settimeout(timeout)+ try:+ sock.connect((host, port))+ return None+ except PolicyDeniedError as exc:+ return exc+ except OSError as exc:+ return exc+ finally:+ sock.close()+++# ---------------------------------------------------------------------------+# PRIVATE_ONLY: the full stack agrees+# ---------------------------------------------------------------------------+++def test_private_only_armed_blocks_public_socket_allows_loopback():+ """With a real armed PRIVATE_ONLY context, a raw socket.connect to a public+ IP is refused by the audit hook (no network), while a connect to loopback+ is NOT refused by the hook. Allow+deny pair: if the hook stopped gating,+ the public probe would no longer raise PolicyDeniedError.+ """+ assert is_audit_hook_installed() is True+ ctx = _make_ctx(EgressScope.PRIVATE_ONLY)+ set_active_context(ctx)+ try:+ public_err = _attempt_connect(_PUBLIC_IP, 53)+ loopback_err = _attempt_connect(_LOOPBACK_IP, 1)+ finally:+ clear_active_context()++ assert isinstance(public_err, PolicyDeniedError), (+ f"armed PRIVATE_ONLY did not block public socket: {public_err!r}"+ )+ assert public_err.decision.reason == "scope_mismatch_private_only"+ assert public_err.target == _PUBLIC_IP+ assert not isinstance(loopback_err, PolicyDeniedError), (+ f"armed PRIVATE_ONLY wrongly blocked loopback: {loopback_err!r}"+ )+++def test_private_only_armed_pdp_denies_public_url_engine_and_cloud_llm():+ """Under the SAME armed PRIVATE_ONLY context, every PDP call site agrees:+ evaluate_url(public) / evaluate_engine(public) / evaluate_llm_endpoint(cloud)+ all deny, while their private/local mirrors are allowed. The mirror halves+ make each assertion fail if the scope check were reverted to allow-all.+ """+ ctx = _make_ctx(+ EgressScope.PRIVATE_ONLY,+ require_local_llm=True,+ require_local_embeddings=True,+ )+ set_active_context(ctx)+ try:+ armed = get_active_context()+ assert armed is ctx++ # evaluate_url: public denied, loopback allowed.+ url_public = evaluate_url(f"http://{_PUBLIC_IP}/", armed)+ url_private = evaluate_url(f"http://{_LOOPBACK_IP}/", armed)++ # evaluate_engine: public engine denied, local engine allowed.+ engine_public = evaluate_engine(+ _PUBLIC_ENGINE, armed, settings_snapshot={}+ )+ engine_local = evaluate_engine("paperless", armed, settings_snapshot={})++ # evaluate_llm_endpoint: cloud denied, local-default allowed.+ llm_cloud = evaluate_llm_endpoint(+ _CLOUD_LLM, armed, settings_snapshot={}+ )+ llm_local = evaluate_llm_endpoint("ollama", armed, settings_snapshot={})+ finally:+ clear_active_context()++ assert not url_public.allowed+ assert url_public.reason == "scope_mismatch_private_only"+ assert url_private.allowed, "loopback URL wrongly denied under PRIVATE_ONLY"++ assert not engine_public.allowed+ assert engine_public.reason == "scope_mismatch_private_only"+ assert engine_local.allowed, (+ "local engine wrongly denied under PRIVATE_ONLY"+ )++ assert not llm_cloud.allowed+ assert llm_cloud.reason == "provider_cloud_only"+ assert llm_local.allowed, "local LLM wrongly denied under require_local_llm"+++def test_private_only_armed_blocks_metadata_ip_canonical_and_octal():+ """Cloud-metadata IPs (IMDS) are NEVER permitted regardless of scope, and+ classify with the explicit blocked_metadata_ip reason — including the+ alternate octal encoding the libc resolver accepts. PRIVATE_ONLY would+ otherwise ALLOW these (they look link-local/private), so this guards the+ credential-theft path. Mirror: a real loopback URL is still allowed, proving+ the denial is the metadata guard, not a blanket refusal.+ """+ ctx = _make_ctx(EgressScope.PRIVATE_ONLY)+ set_active_context(ctx)+ try:+ armed = get_active_context()+ canonical = evaluate_url(+ f"http://{_METADATA_IP}/latest/meta-data/", armed+ )+ octal = evaluate_url(f"http://{_METADATA_IP_OCTAL}/", armed)+ loopback = evaluate_url(f"http://{_LOOPBACK_IP}/", armed)+ finally:+ clear_active_context()++ assert not canonical.allowed+ assert canonical.reason == "blocked_metadata_ip"+ assert not octal.allowed+ assert octal.reason == "blocked_metadata_ip"+ assert loopback.allowed, "loopback wrongly denied — guard is over-broad"+++# ---------------------------------------------------------------------------+# STRICT: same socket-layer contract as PRIVATE_ONLY+# ---------------------------------------------------------------------------+++def test_strict_armed_socket_layer_matches_private_only():+ """STRICT must behave like PRIVATE_ONLY at the socket layer: public host+ refused (with the strict_public_host reason), loopback permitted. Allow+deny+ pair against the public/loopback probes.+ """+ ctx = _make_ctx(EgressScope.STRICT)+ set_active_context(ctx)+ try:+ public_err = _attempt_connect(_PUBLIC_IP, 53)+ loopback_err = _attempt_connect(_LOOPBACK_IP, 1)+ finally:+ clear_active_context()++ assert isinstance(public_err, PolicyDeniedError), (+ f"armed STRICT did not block public socket: {public_err!r}"+ )+ assert public_err.decision.reason == "strict_public_host"+ assert not isinstance(loopback_err, PolicyDeniedError), (+ f"armed STRICT wrongly blocked loopback: {loopback_err!r}"+ )+++# ---------------------------------------------------------------------------+# PUBLIC_ONLY / BOTH: do NOT arm the socket-level hook+# ---------------------------------------------------------------------------+++def test_public_only_armed_does_not_arm_socket_hook():+ """PUBLIC_ONLY governs ENGINE selection, not which hosts the process may+ reach at the socket level (local Ollama / settings DB use private IPs). The+ hook must therefore stay a no-op for ALL host classes under PUBLIC_ONLY —+ otherwise legitimate local services would be false-positived. Contrast with+ the PRIVATE_ONLY tripwire above, which DOES raise on the public probe.+ """+ ctx = _make_ctx(EgressScope.PUBLIC_ONLY)+ set_active_context(ctx)+ try:+ results = {+ host: _attempt_connect(host, 1)+ for host in (_PUBLIC_IP, "192.168.42.1", _LOOPBACK_IP)+ }+ finally:+ clear_active_context()++ for host, err in results.items():+ assert not isinstance(err, PolicyDeniedError), (+ f"PUBLIC_ONLY wrongly armed the hook for {host}: {err!r}"+ )+++def test_both_armed_does_not_arm_socket_hook():+ """BOTH is the permissive default — the audit hook has nothing to enforce,+ so a public connect must pass through. This is the deny-side contrast to the+ PRIVATE_ONLY public probe: same address, opposite outcome, proving the hook+ keys off the armed scope and not the address alone.+ """+ ctx = _make_ctx(EgressScope.BOTH)+ set_active_context(ctx)+ try:+ err = _attempt_connect(_PUBLIC_IP, 53)+ finally:+ clear_active_context()++ assert not isinstance(err, PolicyDeniedError), (+ f"BOTH scope wrongly armed the hook: {err!r}"+ )diff --git a/tests/security/test_egress_inference_gates_followup.py b/tests/security/test_egress_inference_gates_followup.pynew file mode 100644index 0000000000..63cae1010a--- /dev/null+++ b/tests/security/test_egress_inference_gates_followup.py@@ -0,0 +1,330 @@+"""Follow-up regression tests for the egress inference gates.++Deepens coverage of ``evaluate_llm_endpoint`` and ``evaluate_embeddings``+beyond what ``tests/security/test_egress_policy.py`` already exercises.++Focus areas (gaps not covered elsewhere):+- require_local_* False => always allowed (LLM + embeddings, incl. cloud).+- Full cloud-LLM block set under require-local (deepseek/xai/ionos as well+ as the already-covered openai/anthropic/google/openrouter).+- Local-default providers WITHOUT a URL (lmstudio/llamacpp for LLM;+ ollama/sentence_transformers for embeddings).+- URL-configured providers: local URL allowed, remote denied,+ percent-encoded local URL allowed (both gates).+- Embeddings ollama URL handling + the embeddings.ollama.url -> llm.ollama.url+ fallback, plus the provider_unknown branch.+- PDP-level snapshot-less branch (settings_snapshot=None) for both gates.+- The _is_user_registered_llm discriminator (built-in vs in-process code).+"""++from __future__ import annotations++from unittest.mock import patch++import pytest++from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+ evaluate_embeddings,+ evaluate_llm_endpoint,+ _is_user_registered_llm,+)+++# ---------------------------------------------------------------------------+# Helpers (mirrors make_ctx in tests/security/test_egress_policy.py)+# ---------------------------------------------------------------------------+++def make_ctx(+ scope: EgressScope = EgressScope.BOTH,+ primary: str = "arxiv",+ require_local_llm: bool = False,+ require_local_embeddings: bool = False,+ local_hostnames=(),+) -> EgressContext:+ return EgressContext(+ scope=scope,+ primary_engine=primary,+ require_local_llm=require_local_llm,+ require_local_embeddings=require_local_embeddings,+ local_hostnames=tuple(local_hostnames),+ )+++# ---------------------------------------------------------------------------+# evaluate_llm_endpoint — require_local_llm False => always allowed+# ---------------------------------------------------------------------------+++ "provider",+ ["openai", "anthropic", "deepseek", "xai", "ionos", "ollama", "lmstudio"],+)+def test_llm_no_local_requirement_allows_any_provider(provider):+ """With require_local_llm False the gate is a pass-through for every+ provider — cloud included — and reports no_local_requirement."""+ ctx = make_ctx(require_local_llm=False)+ decision = evaluate_llm_endpoint(provider, ctx, settings_snapshot={})+ assert decision.allowed is True+ assert decision.reason == "no_local_requirement"+++# ---------------------------------------------------------------------------+# evaluate_llm_endpoint — full cloud-provider block set under require-local+# ---------------------------------------------------------------------------++[email protected]("provider", ["deepseek", "xai", "ionos"])+def test_llm_require_local_blocks_remaining_cloud_providers(provider):+ """The cloud block set must include deepseek/xai/ionos — the providers+ the existing suite does NOT already enumerate. Each is blocked under+ require_local_llm with provider_cloud_only, and allowed without it."""+ blocked = evaluate_llm_endpoint(+ provider, make_ctx(require_local_llm=True), settings_snapshot={}+ )+ assert blocked.allowed is False+ assert blocked.reason == "provider_cloud_only"+ # Mirror: the same provider is permitted when the toggle is off.+ allowed = evaluate_llm_endpoint(+ provider, make_ctx(require_local_llm=False), settings_snapshot={}+ )+ assert allowed.allowed is True+++# ---------------------------------------------------------------------------+# evaluate_llm_endpoint — local-default providers without a URL+# ---------------------------------------------------------------------------++[email protected]("provider", ["lmstudio", "llamacpp"])+def test_llm_local_default_providers_without_url_allowed(provider):+ """lmstudio/llamacpp (besides the already-tested ollama) fall back to+ their localhost defaults when no URL override is configured."""+ ctx = make_ctx(require_local_llm=True)+ decision = evaluate_llm_endpoint(provider, ctx, settings_snapshot={})+ assert decision.allowed is True+ assert decision.reason == "provider_local_default"+++# ---------------------------------------------------------------------------+# evaluate_llm_endpoint — URL-configured: local allowed, remote denied+# ---------------------------------------------------------------------------+++def test_llm_configured_local_url_allowed():+ """A local-default provider pointed at an explicit local URL classifies+ via the URL (provider_local), not the static default."""+ ctx = make_ctx(require_local_llm=True)+ snapshot = {"llm.lmstudio.url": "http://192.168.1.50:1234/v1"}+ decision = evaluate_llm_endpoint(+ "lmstudio", ctx, settings_snapshot=snapshot+ )+ assert decision.allowed is True+ assert decision.reason == "provider_local"+++def test_llm_configured_remote_url_denied():+ """The mirror deny case: a configured remote URL is refused even for a+ normally-local-default provider."""+ ctx = make_ctx(require_local_llm=True)+ snapshot = {"llm.lmstudio.url": "https://lmstudio.example.com"}+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_llm_endpoint(+ "lmstudio", ctx, settings_snapshot=snapshot+ )+ assert decision.allowed is False+ assert decision.reason == "provider_remote"+++def test_llm_percent_encoded_local_url_allowed():+ """A percent-encoded local host (the HTTP client decodes it before+ connect) must be decoded by the PDP and classified local. Without the+ unquote() this reads as an unresolvable public host and is denied."""+ ctx = make_ctx(require_local_llm=True)+ snapshot = {"llm.ollama.url": "http://127%2e0%2e0%2e1:11434"}+ decision = evaluate_llm_endpoint("ollama", ctx, settings_snapshot=snapshot)+ assert decision.allowed is True+ assert decision.reason == "provider_local"+++# ---------------------------------------------------------------------------+# evaluate_llm_endpoint — PDP-level snapshot-less branch+# ---------------------------------------------------------------------------+++def test_llm_none_snapshot_fails_closed():+ """settings_snapshot=None at the PDP fails closed regardless of the+ require-local flag — the gate cannot read policy from a missing snapshot."""+ ctx = make_ctx(require_local_llm=False)+ decision = evaluate_llm_endpoint("ollama", ctx, settings_snapshot=None)+ assert decision.allowed is False+ assert decision.reason == "no_snapshot"+++# ---------------------------------------------------------------------------+# evaluate_embeddings — require_local_embeddings False => always allowed+# ---------------------------------------------------------------------------+++ "provider", ["openai", "ollama", "sentence_transformers", "cohere"]+)+def test_embeddings_no_local_requirement_allows_any_provider(provider):+ """With require_local_embeddings False the gate passes through for every+ provider — including cloud openai and an unknown provider."""+ ctx = make_ctx(require_local_embeddings=False)+ decision = evaluate_embeddings(provider, ctx, settings_snapshot={})+ assert decision.allowed is True+ assert decision.reason == "no_local_requirement"+++# ---------------------------------------------------------------------------+# evaluate_embeddings — ollama provider (gap: only openai/ST tested before)+# ---------------------------------------------------------------------------+++def test_embeddings_ollama_without_url_allowed():+ """ollama embeddings with no URL override falls back to its localhost+ default."""+ ctx = make_ctx(require_local_embeddings=True)+ decision = evaluate_embeddings("ollama", ctx, settings_snapshot={})+ assert decision.allowed is True+ assert decision.reason == "provider_local_default"+++def test_embeddings_ollama_local_url_allowed():+ ctx = make_ctx(require_local_embeddings=True)+ snapshot = {"embeddings.ollama.url": "http://127.0.0.1:11434"}+ decision = evaluate_embeddings("ollama", ctx, settings_snapshot=snapshot)+ assert decision.allowed is True+ assert decision.reason == "provider_local"+++def test_embeddings_ollama_remote_url_denied():+ ctx = make_ctx(require_local_embeddings=True)+ snapshot = {"embeddings.ollama.url": "https://ollama.example.com"}+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_embeddings(+ "ollama", ctx, settings_snapshot=snapshot+ )+ assert decision.allowed is False+ assert decision.reason == "provider_remote"+++def test_embeddings_ollama_percent_encoded_local_url_allowed():+ """Percent-encoded local ollama embeddings host must be decoded and+ classified local (mirrors the LLM gate)."""+ ctx = make_ctx(require_local_embeddings=True)+ snapshot = {"embeddings.ollama.url": "http://127%2e0%2e0%2e1:11434"}+ decision = evaluate_embeddings("ollama", ctx, settings_snapshot=snapshot)+ assert decision.allowed is True+ assert decision.reason == "provider_local"+++def test_embeddings_ollama_url_falls_back_to_llm_ollama_url():+ """When embeddings.ollama.url is unset, the gate falls back to the shared+ llm.ollama.url. A remote fallback value must still be denied (proves the+ fallback is actually consulted, not silently treated as local-default)."""+ ctx = make_ctx(require_local_embeddings=True)+ snapshot = {"llm.ollama.url": "https://remote-ollama.example.com"}+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_embeddings(+ "ollama", ctx, settings_snapshot=snapshot+ )+ assert decision.allowed is False+ assert decision.reason == "provider_remote"+++# ---------------------------------------------------------------------------+# evaluate_embeddings — openai base_url edge cases+# ---------------------------------------------------------------------------+++def test_embeddings_openai_percent_encoded_local_base_url_allowed():+ ctx = make_ctx(require_local_embeddings=True)+ snapshot = {"embeddings.openai.base_url": "http://127%2e0%2e0%2e1:1234/v1"}+ decision = evaluate_embeddings("openai", ctx, settings_snapshot=snapshot)+ assert decision.allowed is True+ assert decision.reason == "provider_local_endpoint"+++def test_embeddings_openai_remote_base_url_denied():+ """A genuine cloud base_url stays denied under require-local."""+ ctx = make_ctx(require_local_embeddings=True)+ snapshot = {"embeddings.openai.base_url": "https://api.openai.com/v1"}+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_embeddings(+ "openai", ctx, settings_snapshot=snapshot+ )+ assert decision.allowed is False+ assert decision.reason == "provider_cloud"+++# ---------------------------------------------------------------------------+# evaluate_embeddings — unknown provider + snapshot-less branches+# ---------------------------------------------------------------------------+++def test_embeddings_unknown_provider_under_require_local_denied():+ """An embeddings provider outside the known set fails closed under+ require-local (provider_unknown), but is allowed when the toggle is off."""+ blocked = evaluate_embeddings(+ "cohere", make_ctx(require_local_embeddings=True), settings_snapshot={}+ )+ assert blocked.allowed is False+ assert blocked.reason == "provider_unknown"+++def test_embeddings_none_snapshot_fails_closed():+ ctx = make_ctx(require_local_embeddings=False)+ decision = evaluate_embeddings(+ "sentence_transformers", ctx, settings_snapshot=None+ )+ assert decision.allowed is False+ assert decision.reason == "no_snapshot"+++# ---------------------------------------------------------------------------+# _is_user_registered_llm discriminator (underpins the shadowing behavior)+# ---------------------------------------------------------------------------+++def test_is_user_registered_llm_true_for_in_process_registration():+ """A freshly-registered name that is NOT an auto-discovered built-in is+ user-supplied in-process code."""+ from local_deep_research.llm.llm_registry import (+ register_llm,+ unregister_llm,+ )++ name = "_followup_inprocess_llm"+ register_llm(name, lambda **kwargs: None)+ try:+ assert _is_user_registered_llm(name) is True+ finally:+ unregister_llm(name)+++def test_is_user_registered_llm_false_for_builtin_cloud_name():+ """A built-in cloud provider name (auto-registered by discover_providers)+ must NOT be treated as user code — this is what keeps a shadowing+ registration on the strict cloud-block path."""+ assert _is_user_registered_llm("openai") is False+++def test_is_user_registered_llm_false_for_unregistered_name():+ assert _is_user_registered_llm("_never_registered_followup") is Falsediff --git a/tests/security/test_egress_langgraph_tool_filter.py b/tests/security/test_egress_langgraph_tool_filter.pynew file mode 100644index 0000000000..979dc4d7ba--- /dev/null+++ b/tests/security/test_egress_langgraph_tool_filter.py@@ -0,0 +1,262 @@+"""Integration tests for the egress-policy PEP that pre-filters the LangGraph+lead-agent tool list in+``advanced_search_system/strategies/langgraph_agent_strategy.py``+(``LangGraphAgentStrategy._build_tools`` specialized-engine loop, ~L608-695).++This is the *silent-expansion* fix. The search-engine factory PEP already+refuses a forbidden engine at instantiation time, but that is a runtime stop:+the LLM still SEES the forbidden tool name in its schema, and the latency of a+denied tool call leaks policy state. Filtering the tool list HERE means a+forbidden engine's name never reaches ``create_agent()`` — the model never+learns it exists. These tests assert exactly that property: which engine names+survive into the built tool list under each scope.++The tests drive the REAL ``_build_tools`` loop and the REAL+``evaluate_engine`` / ``evaluate_retriever`` PDP. Only the leaf tool factories+(``_make_web_search_tool``, ``_make_specialized_search_tool``,+``build_fetch_tool``) and the engine/retriever *discovery* (``get_available_engines``,+``retriever_registry``) are mocked — so every assertion is about the FILTER+decision, made before any tool is actually constructed.++Engine classifications come from the live ``ENGINE_REGISTRY``:+ - arxiv / wikipedia : public (is_public=True, is_local=False)+ - paperless : local (is_public=False, is_local=True; no api_url in+ the snapshot, so no DNS lookup occurs)++Each scope axis is covered with an allow+deny pair: reverting the filter (or+loosening a scope check) would let a denied name survive (or drop an allowed+one) and flip an assertion.+"""++from __future__ import annotations++from contextlib import contextmanager+from types import SimpleNamespace+from unittest.mock import MagicMock, patch++import pytest++import local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy as strat # noqa: E501+from local_deep_research.security import clear_active_context+++# Engine discovery configs (what get_available_engines would return). The+# inner dicts mirror the real config shape consumed by the loop: description,+# strengths, is_retriever. For static engines the is_public/is_local+# classification is read from the engine CLASS, not from these dicts, so the+# decision under test is the real registry classification.+_PUBLIC_ENGINE = "arxiv"+_PUBLIC_ENGINE_2 = "wikipedia"+_PRIVATE_ENGINE = "paperless"++[email protected](autouse=True)+def _no_leaked_armed_context():+ """The tool-list filter reads scope from the snapshot, not the armed+ audit-hook context. Clear the active context before and after each test so+ a context leaked by another file can never mask a decision here, and so+ this file never leaks one outward (hard rule: never leak armed context)."""+ clear_active_context()+ yield+ clear_active_context()+++@contextmanager+def _patched_tool_factories():+ """Replace the leaf tool builders with cheap name-carrying sentinels.++ ``_make_specialized_search_tool`` is a MagicMock whose side effect returns a+ sentinel tagged with the engine name, so a DENIED engine is proven excluded+ two ways: its name is absent from the built list AND the factory was never+ called for it (the filter ``continue``s before construction). ``build_fetch_tool``+ returns None so only the primary + surviving specialized tools remain.+ """+ spec = MagicMock(+ side_effect=lambda name, *a, **k: SimpleNamespace(name=name)+ )+ with (+ patch.object(+ strat,+ "_make_web_search_tool",+ lambda *a, **k: SimpleNamespace(name="web_search"),+ ),+ patch.object(strat, "build_fetch_tool", lambda *a, **k: None),+ patch.object(strat, "_make_specialized_search_tool", spec),+ ):+ yield spec+++def _build_tool_names(+ scope, available, *, retriever_meta=None, snapshot_extra=None+):+ """Construct a minimal strategy under ``scope`` and run the REAL+ ``_build_tools`` loop against the mocked engine discovery ``available``.++ Returns ``(tool_names, spec_mock)`` where ``tool_names`` is the list of+ ``.name`` values on the built tools (the primary appears as "web_search")+ and ``spec_mock`` is the specialized-tool factory mock for call assertions.++ ``retriever_meta`` (e.g. ``{"is_local": True}``) patches the retriever+ registry's metadata lookup so the retriever branch can be exercised.+ """+ snapshot = {+ "policy.egress_scope": scope,+ "search.fetch.mode": "disabled",+ }+ if snapshot_extra:+ snapshot.update(snapshot_extra)++ reg = MagicMock()+ reg.get_metadata.return_value = retriever_meta++ with _patched_tool_factories() as spec:+ with (+ patch(+ "local_deep_research.web_search_engines.search_engines_config"+ ".get_available_engines",+ return_value=available,+ ),+ patch(+ "local_deep_research.web_search_engines.retriever_registry"+ ".retriever_registry",+ reg,+ ),+ ):+ # search=MagicMock() so the primary web_search tool is present (its+ # class name "magicmock" becomes the run's current/primary engine,+ # which never collides with a real engine name). citation_handler+ # is supplied so the strategy ctor skips building a real one.+ strategy = strat.LangGraphAgentStrategy(+ model=MagicMock(),+ search=MagicMock(),+ citation_handler=object(),+ include_sub_research=False,+ settings_snapshot=snapshot,+ )+ tools = strategy._build_tools("overall query")+ return [t.name for t in tools], spec+++def _specialized(names):+ """Names of specialized tools that survived (excluding the primary)."""+ return {n for n in names if n != "web_search"}+++# ---------------------------------------------------------------------------+# PRIVATE_ONLY: public engines are excluded from the tool list; local allowed.+# ---------------------------------------------------------------------------+++class TestPrivateOnlyFilter:+ def test_public_engine_name_never_reaches_agent(self):+ names, spec = _build_tool_names(+ "private_only",+ {+ _PUBLIC_ENGINE: {"description": "papers"},+ _PRIVATE_ENGINE: {"description": "docs"},+ },+ )+ # The public engine is silently excluded: its name is absent and the+ # tool factory was never invoked for it.+ assert _PUBLIC_ENGINE not in names+ called = {c.args[0] for c in spec.call_args_list}+ assert _PUBLIC_ENGINE not in called+ # The local engine is allowed (allow side of the pair).+ assert _PRIVATE_ENGINE in names+ assert _specialized(names) == {_PRIVATE_ENGINE}+++# ---------------------------------------------------------------------------+# PUBLIC_ONLY: local/collection engines excluded; public allowed.+# ---------------------------------------------------------------------------+++class TestPublicOnlyFilter:+ def test_private_engine_name_never_reaches_agent(self):+ names, spec = _build_tool_names(+ "public_only",+ {+ _PUBLIC_ENGINE: {"description": "papers"},+ _PRIVATE_ENGINE: {"description": "docs"},+ },+ )+ assert _PRIVATE_ENGINE not in names+ called = {c.args[0] for c in spec.call_args_list}+ assert _PRIVATE_ENGINE not in called+ assert _PUBLIC_ENGINE in names+ assert _specialized(names) == {_PUBLIC_ENGINE}+++# ---------------------------------------------------------------------------+# STRICT: no specialized engines at all — only the primary web_search tool.+# ---------------------------------------------------------------------------+++class TestStrictFilter:+ def test_no_specialized_engines_registered(self):+ names, spec = _build_tool_names(+ "strict",+ {+ _PUBLIC_ENGINE: {"description": "papers"},+ _PUBLIC_ENGINE_2: {"description": "wiki"},+ _PRIVATE_ENGINE: {"description": "docs"},+ },+ )+ # Deny side: every specialized engine is dropped, regardless of bucket,+ # and none were ever constructed.+ assert _specialized(names) == set()+ spec.assert_not_called()+ # Allow side: the primary tool IS present — STRICT yields a single-tool+ # agent, not an empty one.+ assert names == ["web_search"]+++# ---------------------------------------------------------------------------+# BOTH: the filter does not over-exclude — every discovered engine survives.+# This is the allow baseline proving the PRIVATE/PUBLIC exclusions above are+# scope-driven, not a blanket drop.+# ---------------------------------------------------------------------------+++class TestBothScopeAllowsAll:+ def test_public_and_private_engines_both_present(self):+ names, _ = _build_tool_names(+ "both",+ {+ _PUBLIC_ENGINE: {"description": "papers"},+ _PRIVATE_ENGINE: {"description": "docs"},+ },+ )+ assert _specialized(names) == {_PUBLIC_ENGINE, _PRIVATE_ENGINE}+++# ---------------------------------------------------------------------------+# Retriever branch: registered retrievers route through evaluate_retriever+# (classified by registry is_local metadata), filtered by the same loop.+# ---------------------------------------------------------------------------+++class TestRetrieverFilter:+ _RET = "my_private_kb"++ def test_local_retriever_excluded_under_public_only(self):+ names, spec = _build_tool_names(+ "public_only",+ {self._RET: {"description": "kb", "is_retriever": True}},+ retriever_meta={"is_local": True},+ )+ # Deny: a local retriever must not surface under PUBLIC_ONLY.+ assert self._RET not in names+ spec.assert_not_called()+ assert _specialized(names) == set()++ def test_local_retriever_allowed_under_private_only(self):+ names, spec = _build_tool_names(+ "private_only",+ {self._RET: {"description": "kb", "is_retriever": True}},+ retriever_meta={"is_local": True},+ )+ # Allow: same retriever IS registered under PRIVATE_ONLY.+ assert self._RET in names+ assert _specialized(names) == {self._RET}+ assert spec.call_args_list[0].args[0] == self._RETdiff --git a/tests/security/test_egress_mcp_scope_gating.py b/tests/security/test_egress_mcp_scope_gating.pynew file mode 100644index 0000000000..1c1fed8468--- /dev/null+++ b/tests/security/test_egress_mcp_scope_gating.py@@ -0,0 +1,265 @@+"""Integration tests for MCP egress scope gating (PR #4300, Stage 1a).++Drives the REAL policy enforcement points in+``advanced_search_system/strategies/mcp_strategy.py``:++* ``MCPSearchStrategy._mcp_scope_blocked`` — the scope decision that+ certifies whether stdio MCP subprocesses may be exposed at all.+* ``MCPSearchStrategy._discover_mcp_tools`` — suppresses (and caches an+ empty list for) discovery when the scope blocks it, and never launches+ an MCP client/subprocess on the blocked path.+* ``MCPSearchStrategy._build_tool_descriptions`` — scope-filters the+ specialized search-engine tool descriptions exposed to the LLM.++These exercise the real call-site logic; the only mocked pieces are the+LLM/citation handler (never invoked here), the available-engine catalog,+and the MCP client transport (so no subprocess launches). Every test is+written as an allow/deny pair so it fails if the guarded property is+reverted.+"""++from __future__ import annotations++from unittest.mock import MagicMock, patch++import pytest++from local_deep_research.advanced_search_system.strategies.mcp_strategy import (+ MCPSearchStrategy,+)++MCP_CLIENT = "local_deep_research.mcp.client"+++def make_strategy(+ *,+ snapshot,+ mcp_servers=None,+ include_web_search=True,+ include_sub_research=False,+):+ """Construct an ``MCPSearchStrategy`` with only the heavy collaborators+ stubbed (model + citation handler are never invoked by the methods under+ test). ``settings_snapshot`` and ``mcp_servers`` are real inputs."""+ return MCPSearchStrategy(+ model=MagicMock(name="model"),+ search=MagicMock(name="search"),+ citation_handler=MagicMock(name="citation_handler"),+ mcp_servers=mcp_servers,+ include_web_search=include_web_search,+ include_sub_research=include_sub_research,+ settings_snapshot=snapshot,+ )+++def snap(scope=None, primary="arxiv"):+ """Build a settings snapshot in the {key: {value: ...}} / plain shapes+ the strategy reads."""+ s = {}+ if scope is not None:+ s["policy.egress_scope"] = {"value": scope}+ if primary is not None:+ s["search.tool"] = primary+ return s+++SERVERS = [{"name": "srv", "command": "echo", "args": []}]+++# ---------------------------------------------------------------------------+# _mcp_scope_blocked+# ---------------------------------------------------------------------------+++def test_scope_blocked_strict_concrete_primary():+ """STRICT with a concrete primary blocks MCP and reports the scope."""+ strat = make_strategy(snapshot=snap("strict", primary="arxiv"))+ assert strat._mcp_scope_blocked() == (True, "strict")+++def test_scope_blocked_private_only():+ """PRIVATE_ONLY blocks MCP (stdio egress is uninspectable)."""+ strat = make_strategy(snapshot=snap("private_only", primary="arxiv"))+ assert strat._mcp_scope_blocked() == (True, "private_only")+++def test_scope_not_blocked_public_only():+ """PUBLIC_ONLY does not block MCP discovery."""+ strat = make_strategy(snapshot=snap("public_only", primary="arxiv"))+ assert strat._mcp_scope_blocked() == (False, "")+++def test_scope_not_blocked_both():+ """BOTH (the permissive default scope) does not block MCP."""+ strat = make_strategy(snapshot=snap("both", primary="arxiv"))+ assert strat._mcp_scope_blocked() == (False, "")+++def test_scope_blocked_no_snapshot_fails_closed():+ """An empty snapshot ({}) cannot certify safety -> fail closed."""+ strat = make_strategy(snapshot={})+ # BaseSearchStrategy coerces None/{} to {} (falsy) -> no_snapshot.+ assert strat.settings_snapshot == {}+ assert strat._mcp_scope_blocked() == (True, "no_snapshot")+++def test_scope_blocked_corrupt_scope_fails_closed():+ """A corrupt/unknown scope value is unevaluable -> fail closed."""+ strat = make_strategy(snapshot=snap("bogus_scope", primary="arxiv"))+ blocked, reason = strat._mcp_scope_blocked()+ assert blocked is True+ assert reason.startswith("policy_unavailable:")+++def test_scope_blocked_strict_meta_picker_fails_closed():+ """STRICT + meta-picker primary ('auto') is incoherent -> fail closed+ (ValueError from the cross-field invariant), not a silent allow."""+ strat = make_strategy(snapshot=snap("strict", primary="auto"))+ blocked, reason = strat._mcp_scope_blocked()+ assert blocked is True+ assert reason.startswith("policy_unavailable:")+++# ---------------------------------------------------------------------------+# _discover_mcp_tools+# ---------------------------------------------------------------------------++[email protected]("scope", ["strict", "private_only"])+def test_discover_suppressed_under_blocking_scope(scope):+ """Under a blocking scope, discovery returns [], caches it, and NEVER+ invokes the MCP client transport (no subprocess launch)."""+ strat = make_strategy(+ snapshot=snap(scope, primary="arxiv"), mcp_servers=SERVERS+ )+ with (+ patch(f"{MCP_CLIENT}.run_async") as run_async,+ patch(f"{MCP_CLIENT}.MCPClientManager") as manager,+ ):+ tools = strat._discover_mcp_tools()++ assert tools == []+ # Fail-closed suppression is final: cached as empty.+ assert strat._mcp_tools_cache == []+ run_async.assert_not_called()+ manager.assert_not_called()+++def test_discover_no_snapshot_suppressed():+ """No snapshot ({}) fails closed at discovery too: [] and no transport."""+ strat = make_strategy(snapshot={}, mcp_servers=SERVERS)+ with patch(f"{MCP_CLIENT}.run_async") as run_async:+ tools = strat._discover_mcp_tools()+ assert tools == []+ assert strat._mcp_tools_cache == []+ run_async.assert_not_called()+++def test_discover_proceeds_under_allowing_scope():+ """Under BOTH, discovery is NOT suppressed: the strategy reaches the+ (mocked) MCP transport and surfaces the discovered tools."""+ strat = make_strategy(+ snapshot=snap("both", primary="arxiv"), mcp_servers=SERVERS+ )++ discovered = {+ "srv": [+ {+ "name": "tool1",+ "description": "a tool",+ "input_schema": {"properties": {"q": {"type": "string"}}},+ }+ ]+ }++ def run_async_stub(coro, timeout=None):+ # Close the un-awaited discover() coroutine so MCPClientManager is+ # never instantiated and no "coroutine was never awaited" warning+ # leaks; return the canned discovery result.+ coro.close()+ return discovered++ with patch(f"{MCP_CLIENT}.run_async", side_effect=run_async_stub) as ra:+ tools = strat._discover_mcp_tools()++ ra.assert_called_once()+ assert [t["name"] for t in tools] == ["srv.tool1"]+ # Successful discovery is cached.+ assert strat._mcp_tools_cache == tools+++def test_discover_no_servers_is_noop():+ """With no configured servers there is nothing to discover or block."""+ strat = make_strategy(snapshot=snap("both"), mcp_servers=[])+ with patch(f"{MCP_CLIENT}.run_async") as run_async:+ assert strat._discover_mcp_tools() == []+ run_async.assert_not_called()+++# ---------------------------------------------------------------------------+# _build_tool_descriptions — specialized engine scope filtering+# ---------------------------------------------------------------------------+++def _engines():+ """A small available-engine catalog: a public engine and a private+ collection."""+ return {+ "wikipedia": {"description": "Wikipedia"},+ "collection_x": {"description": "Local docs", "is_public": False},+ }+++def _build_with_engines(strat):+ """Run _build_tool_descriptions with a controlled engine catalog and a+ non-matching 'current engine' so nothing is skipped as the primary."""+ strat._get_available_search_engines = lambda: _engines()+ strat._get_current_engine_name = lambda: "magicmock"+ return strat._build_tool_descriptions()+++def _tool_names(tools):+ return {t["name"] for t in tools}+++def test_build_tools_strict_hides_all_specialized():+ """STRICT exposes only the primary (web_search) — no specialized+ search_<engine> tools leak into the LLM's tool list."""+ strat = make_strategy(+ snapshot=snap("strict", primary="arxiv"), mcp_servers=[]+ )+ names = _tool_names(_build_with_engines(strat))+ assert "web_search" in names+ assert not any(n.startswith("search_") for n in names)+++def test_build_tools_both_exposes_specialized():+ """BOTH exposes classified specialized engines (allow side of the pair)."""+ strat = make_strategy(+ snapshot=snap("both", primary="arxiv"), mcp_servers=[]+ )+ names = _tool_names(_build_with_engines(strat))+ assert "search_wikipedia" in names+ assert "search_collection_x" in names+++def test_build_tools_public_only_filters_private_collection():+ """PUBLIC_ONLY keeps the public engine but filters the private+ collection — scope-correct per-engine filtering, not all-or-nothing."""+ strat = make_strategy(+ snapshot=snap("public_only", primary="arxiv"), mcp_servers=[]+ )+ names = _tool_names(_build_with_engines(strat))+ assert "search_wikipedia" in names+ assert "search_collection_x" not in names+++def test_build_tools_corrupt_scope_blocks_all_specialized():+ """A corrupt scope raises PolicyDeniedError during context build, which+ fails closed: NO specialized engines are exposed at all."""+ strat = make_strategy(+ snapshot=snap("bogus_scope", primary="arxiv"), mcp_servers=[]+ )+ names = _tool_names(_build_with_engines(strat))+ assert "web_search" in names+ assert not any(n.startswith("search_") for n in names)diff --git a/tests/security/test_egress_news_subscription_callsite.py b/tests/security/test_egress_news_subscription_callsite.pynew file mode 100644index 0000000000..22c59c3e83--- /dev/null+++ b/tests/security/test_egress_news_subscription_callsite.py@@ -0,0 +1,341 @@+"""Integration tests for the egress policy at the news-subscription call site.++The helper ``_validate_subscription_policy`` is unit-tested elsewhere+(tests/security/test_egress_policy.py N14/N15 and+tests/news/test_subscription_policy.py). This module instead drives the REAL+call-site functions ``create_subscription`` / ``update_subscription`` in+``news/api.py`` and asserts the policy decision actually:++ * blocks persistence (no ``session.add`` / ``session.commit``) when the+ chosen search engine or LLM provider violates the user's egress scope,+ * lets a coherent config through to persistence,+ * surfaces an incoherent egress config (STRICT + meta-picker primary) as a+ misconfiguration error at create/update time rather than silently skipping+ it, and+ * skips the (best-effort) pre-check when the settings backend or the request+ context needed to resolve the user is unavailable — the execution-time+ factory PEP remains the backstop.++Only unavoidable heavy deps are mocked (per-user DB session, settings manager,+scheduler notification, flask request context). The egress policy evaluation+itself runs for real.+"""++from __future__ import annotations++import contextlib+from unittest.mock import MagicMock, patch++import pytest++from local_deep_research.news import api as news_api+from local_deep_research.news.exceptions import (+ SubscriptionCreationException,+ SubscriptionUpdateException,+)++# Patch targets — the call site imports these lazily inside the functions, so+# we patch them at their definition module.+DB_SESSION_PATH = (+ "local_deep_research.database.session_context.get_user_db_session"+)+SM_PATH = "local_deep_research.utilities.db_utils.get_settings_manager"+NOTIFY_PATH = (+ "local_deep_research.news.api._notify_scheduler_about_subscription_change"+)++# Egress snapshots used to drive the real policy evaluation.+STRICT_ARXIV = {"policy.egress_scope": "strict", "search.tool": "arxiv"}+REQUIRE_LOCAL = {+ "policy.egress_scope": "both",+ "llm.require_local_endpoint": True,+}+# STRICT scope with a meta-picker primary is an incoherent combination that+# makes context_from_snapshot raise ValueError.+INCOHERENT = {"policy.egress_scope": "strict", "search.tool": "auto"}+++def _settings_manager(snapshot, primary="arxiv"):+ """Fake SettingsManager exposing the snapshot + search.tool primary."""+ sm = MagicMock()+ sm.get_settings_snapshot.return_value = snapshot+ sm.get_setting.side_effect = lambda key, default=None: (+ primary if key == "search.tool" else default+ )+ return sm+++def _db_session():+ """Patch get_user_db_session to yield a mock session as a context manager."""+ session = MagicMock()+ with patch(DB_SESSION_PATH) as gud:+ gud.return_value.__enter__.return_value = session+ gud.return_value.__exit__.return_value = False+ yield session+++def _existing_subscription():+ """A persisted subscription with engine/provider unset by default so that+ only the fields an update sets participate in the policy check."""+ sub = MagicMock()+ sub.search_engine = None+ sub.model_provider = None+ return sub+++# ---------------------------------------------------------------------------+# create_subscription — engine gate+# ---------------------------------------------------------------------------+++def test_create_blocked_engine_is_not_persisted():+ """STRICT + primary=arxiv: a non-primary engine (pubmed) is rejected at+ create time and never written to the DB."""+ sm = _settings_manager(STRICT_ARXIV, primary="arxiv")+ with (+ _db_session() as session,+ patch(SM_PATH, return_value=sm),+ patch(NOTIFY_PATH),+ ):+ with pytest.raises(SubscriptionCreationException) as exc:+ news_api.create_subscription(+ user_id="alice",+ query="AI",+ search_engine="pubmed",+ refresh_minutes=60,+ )+ assert "pubmed" in exc.value.message+ assert exc.value.error_code == "SUBSCRIPTION_CREATE_FAILED"+ session.add.assert_not_called()+ session.commit.assert_not_called()+++def test_create_allows_primary_engine():+ """The user's own primary engine passes the gate and is persisted."""+ sm = _settings_manager(STRICT_ARXIV, primary="arxiv")+ with (+ _db_session() as session,+ patch(SM_PATH, return_value=sm),+ patch(NOTIFY_PATH),+ ):+ result = news_api.create_subscription(+ user_id="alice",+ query="AI",+ search_engine="arxiv",+ refresh_minutes=60,+ )+ assert result["status"] == "success"+ session.add.assert_called_once()+ session.commit.assert_called_once()+++# ---------------------------------------------------------------------------+# create_subscription — LLM provider gate+# ---------------------------------------------------------------------------+++def test_create_blocked_cloud_provider_is_not_persisted():+ """require_local_endpoint=True: a cloud provider (openai) is rejected and+ not persisted."""+ sm = _settings_manager(REQUIRE_LOCAL, primary="arxiv")+ with (+ _db_session() as session,+ patch(SM_PATH, return_value=sm),+ patch(NOTIFY_PATH),+ ):+ with pytest.raises(SubscriptionCreationException) as exc:+ news_api.create_subscription(+ user_id="alice",+ query="AI",+ model_provider="openai",+ refresh_minutes=60,+ )+ assert "provider" in exc.value.message.lower()+ session.add.assert_not_called()+++def test_create_allows_local_provider():+ """A localhost-default provider (ollama) passes under require_local."""+ sm = _settings_manager(REQUIRE_LOCAL, primary="arxiv")+ with (+ _db_session() as session,+ patch(SM_PATH, return_value=sm),+ patch(NOTIFY_PATH),+ ):+ result = news_api.create_subscription(+ user_id="alice",+ query="AI",+ model_provider="ollama",+ refresh_minutes=60,+ )+ assert result["status"] == "success"+ session.add.assert_called_once()+++# ---------------------------------------------------------------------------+# create_subscription — incoherent config surfaces (not silently skipped)+# ---------------------------------------------------------------------------+++def test_create_surfaces_incoherent_egress_config():+ """STRICT + meta-picker primary makes context_from_snapshot raise+ ValueError; the call site must surface it as a misconfiguration error at+ create time and block persistence, NOT let the broad except swallow it."""+ sm = _settings_manager(INCOHERENT, primary="auto")+ with (+ _db_session() as session,+ patch(SM_PATH, return_value=sm),+ patch(NOTIFY_PATH),+ ):+ with pytest.raises(SubscriptionCreationException) as exc:+ news_api.create_subscription(+ user_id="alice",+ query="AI",+ search_engine="pubmed",+ refresh_minutes=60,+ )+ assert "misconfigured" in exc.value.message+ session.add.assert_not_called()+++# ---------------------------------------------------------------------------+# create_subscription — best-effort skip when backend is unavailable+# ---------------------------------------------------------------------------+++def test_create_skips_precheck_without_settings_backend():+ """If the settings backend is unavailable the pre-check is skipped+ (best-effort) and a config that WOULD be forbidden is still persisted — the+ execution-time factory PEP is the backstop. Mirror of the blocked-engine+ test with the backend removed."""+ with (+ _db_session() as session,+ patch(SM_PATH, side_effect=RuntimeError("no settings DB")),+ patch(NOTIFY_PATH),+ ):+ result = news_api.create_subscription(+ user_id="alice",+ query="AI",+ search_engine="pubmed", # would be denied if backend were present+ refresh_minutes=60,+ )+ assert result["status"] == "success"+ session.add.assert_called_once()+++# ---------------------------------------------------------------------------+# update_subscription — engine / provider gate+# ---------------------------------------------------------------------------+++def test_update_blocked_engine_is_not_committed():+ """Updating an existing subscription to a forbidden engine is rejected and+ the transaction is not committed."""+ sm = _settings_manager(STRICT_ARXIV, primary="arxiv")+ sub = _existing_subscription()+ with (+ _db_session() as session,+ patch(SM_PATH, return_value=sm),+ patch(NOTIFY_PATH),+ patch("flask.has_request_context", return_value=True),+ patch("flask.session", {"username": "alice"}),+ ):+ session.query.return_value.filter_by.return_value.first.return_value = (+ sub+ )+ with pytest.raises(SubscriptionUpdateException) as exc:+ news_api.update_subscription("sub-1", {"search_engine": "pubmed"})+ assert exc.value.error_code == "SUBSCRIPTION_UPDATE_FAILED"+ session.commit.assert_not_called()+++def test_update_allows_coherent_engine():+ """Updating to the user's primary engine is accepted and committed."""+ sm = _settings_manager(STRICT_ARXIV, primary="arxiv")+ sub = _existing_subscription()+ with (+ _db_session() as session,+ patch(SM_PATH, return_value=sm),+ patch(NOTIFY_PATH),+ patch("flask.has_request_context", return_value=True),+ patch("flask.session", {"username": "alice"}),+ ):+ session.query.return_value.filter_by.return_value.first.return_value = (+ sub+ )+ result = news_api.update_subscription(+ "sub-1", {"search_engine": "arxiv"}+ )+ assert result["status"] == "success"+ session.commit.assert_called_once()+++def test_update_blocked_cloud_provider_is_not_committed():+ """Updating to a cloud provider under require_local is rejected and not+ committed."""+ sm = _settings_manager(REQUIRE_LOCAL, primary="arxiv")+ sub = _existing_subscription()+ with (+ _db_session() as session,+ patch(SM_PATH, return_value=sm),+ patch(NOTIFY_PATH),+ patch("flask.has_request_context", return_value=True),+ patch("flask.session", {"username": "alice"}),+ ):+ session.query.return_value.filter_by.return_value.first.return_value = (+ sub+ )+ with pytest.raises(SubscriptionUpdateException):+ news_api.update_subscription("sub-1", {"model_provider": "openai"})+ session.commit.assert_not_called()+++# ---------------------------------------------------------------------------+# update_subscription — pre-check guards (engine/provider touched + user known)+# ---------------------------------------------------------------------------+++def test_update_skips_policy_when_no_engine_or_provider_touched():+ """The pre-check only runs when the update touches the engine or provider.+ A name-only update must NOT consult the settings backend at all (proves the+ guard, not just that nothing happened to be wrong)."""+ sub = _existing_subscription()+ with (+ _db_session() as session,+ patch(SM_PATH) as sm_patch,+ patch(NOTIFY_PATH),+ patch("flask.has_request_context", return_value=True),+ patch("flask.session", {"username": "alice"}),+ ):+ session.query.return_value.filter_by.return_value.first.return_value = (+ sub+ )+ result = news_api.update_subscription("sub-1", {"name": "Renamed"})+ assert result["status"] == "success"+ sm_patch.assert_not_called()+ session.commit.assert_called_once()+++def test_update_skips_policy_without_request_context():+ """Without a request context the user can't be resolved, so the pre-check is+ skipped (best-effort) — a forbidden engine still persists and the execution+ PEP backstops. Counterpart to test_update_blocked_engine_is_not_committed,+ which has the request context."""+ sm = _settings_manager(STRICT_ARXIV, primary="arxiv")+ sub = _existing_subscription()+ with (+ _db_session() as session,+ patch(SM_PATH, return_value=sm) as sm_patch,+ patch(NOTIFY_PATH),+ patch("flask.has_request_context", return_value=False),+ ):+ session.query.return_value.filter_by.return_value.first.return_value = (+ sub+ )+ result = news_api.update_subscription(+ "sub-1", {"search_engine": "pubmed"}+ )+ assert result["status"] == "success"+ session.commit.assert_called_once()+ sm_patch.assert_not_called()diff --git a/tests/security/test_egress_notifications_callsite.py b/tests/security/test_egress_notifications_callsite.pynew file mode 100644index 0000000000..3270324d6f--- /dev/null+++ b/tests/security/test_egress_notifications_callsite.py@@ -0,0 +1,237 @@+"""Integration tests for the egress-policy PEP inside the notification+dispatch path (``notifications/manager.py``).++These drive the REAL ``NotificationManager._filter_urls_by_egress_policy``+and the real ``send_notification`` / ``test_service`` call sites. They assert+the *policy decision* (vendor/cloud webhook filtered vs. allowed, fail-closed+on a corrupt scope, master-switch gating) which happens BEFORE any Apprise /+HTTP work. The only thing mocked is the dispatch boundary+(``service.send_event`` / ``service.test_service``) so nothing actually leaves+the box, plus the env master switches and — for http(s) hosts — the DNS-backed+host classifier, so the tests are in-process and deterministic.++Why filtering vendor webhooks matters: research results contain the user's+queries and retrieved local-corpus chunks. A ``discord://`` / ``slack://`` /+``https://hooks...`` notification URL would exfiltrate those to an external+vendor API. Under PRIVATE_ONLY ("nothing leaves the box") and, for public+http(s) hosts, STRICT, those URLs must be dropped before dispatch.+"""++from __future__ import annotations++from unittest.mock import MagicMock, patch++from local_deep_research.notifications.manager import NotificationManager+from local_deep_research.notifications.templates import EventType+++# ---------------------------------------------------------------------------+# Construction helper. NotificationManager.__init__ reads two ENV-only+# master switches and builds a NotificationService (an in-process+# apprise.Apprise() — no network). We patch the env reads so the manager is+# built deterministically; everything else is the real code.+# ---------------------------------------------------------------------------+++def _build_manager(snapshot, *, outbound=True, user_id="notif-egress-test"):+ def fake_env(key, default=None):+ if key == "notifications.allow_outbound":+ return outbound+ if key == "notifications.allow_private_ips":+ return False+ return default++ with patch(+ "local_deep_research.notifications.manager.get_env_setting",+ side_effect=fake_env,+ ):+ mgr = NotificationManager(snapshot, user_id=user_id)+ # Sever the real dispatch path: any send must go through this mock, never+ # apprise/HTTP. Tests assert on whether/with-what it is called.+ mgr.service.send_event = MagicMock(return_value=True)+ mgr.service.test_service = MagicMock(+ return_value={"success": True, "message": "ok"}+ )+ return mgr+++def _snapshot(scope, *, tool="arxiv", service_url=None):+ snap = {+ "policy.egress_scope": scope,+ "search.tool": tool,+ }+ if service_url is not None:+ snap["notifications.service_url"] = service_url+ return snap+++_CLASSIFY = "local_deep_research.security.egress.policy._classify_host"+++# ---------------------------------------------------------------------------+# A. Vendor (non-http) scheme: the core exfiltration guard.+# ---------------------------------------------------------------------------+++def test_private_only_filters_vendor_scheme_but_both_allows():+ """discord:// is a vendor API egress that cannot be verified local, so+ PRIVATE_ONLY must drop it; BOTH (the permissive scope) passes it through.+ """+ url = "discord://webhook_id/token"++ deny_mgr = _build_manager(_snapshot("private_only"))+ assert deny_mgr._filter_urls_by_egress_policy(url) == ""++ allow_mgr = _build_manager(_snapshot("both"))+ assert allow_mgr._filter_urls_by_egress_policy(url) == url+++def test_private_only_mixed_urls_keeps_only_local_https():+ """A space-separated Apprise string mixing a vendor scheme, a public+ cloud webhook, and a self-hosted local webhook: under PRIVATE_ONLY only+ the local https survives. Drives both the non-http branch and the real+ evaluate_url http branch in one shot.+ """+ urls = (+ "slack://tokA/tokB/tokC "+ "https://hooks.bad.test/services/x "+ "https://intranet.local/hook"+ )++ def fake_classify(host, ctx, *a, **k):+ return host == "intranet.local"++ mgr = _build_manager(_snapshot("private_only"))+ with patch(_CLASSIFY, side_effect=fake_classify):+ result = mgr._filter_urls_by_egress_policy(urls)++ assert result == "https://intranet.local/hook"+++# ---------------------------------------------------------------------------+# C. STRICT: refuses public http(s) hosts (provenance-spoofing risk).+# ---------------------------------------------------------------------------+++def test_strict_filters_public_https_webhook_but_both_allows():+ url = "https://hooks.bad.test/services/x"++ # Public host classification (False), deterministic — no real DNS.+ with patch(_CLASSIFY, return_value=False):+ deny_mgr = _build_manager(_snapshot("strict"))+ assert deny_mgr._filter_urls_by_egress_policy(url) == ""++ allow_mgr = _build_manager(_snapshot("both"))+ assert allow_mgr._filter_urls_by_egress_policy(url) == url+++# ---------------------------------------------------------------------------+# D. PUBLIC_ONLY: refuses a private/intranet webhook, allows the public one.+# ---------------------------------------------------------------------------+++def test_public_only_filters_local_webhook_but_allows_public():+ local_url = "https://intranet.local/hook"+ public_url = "https://hooks.bad.test/services/x"++ def fake_classify(host, ctx, *a, **k):+ return host == "intranet.local"++ mgr = _build_manager(_snapshot("public_only"))+ with patch(_CLASSIFY, side_effect=fake_classify):+ assert mgr._filter_urls_by_egress_policy(local_url) == ""+ assert mgr._filter_urls_by_egress_policy(public_url) == public_url+++# ---------------------------------------------------------------------------+# E. Corrupt / unevaluable scope must fail CLOSED (filter, not dispatch).+# ---------------------------------------------------------------------------+++def test_corrupt_scope_fails_closed():+ url = "discord://webhook_id/token"++ # A garbage scope makes context_from_snapshot raise PolicyDeniedError;+ # the PEP must refuse ALL urls rather than fall open to unfiltered send.+ deny_mgr = _build_manager(_snapshot("totally-not-a-scope"))+ assert deny_mgr._filter_urls_by_egress_policy(url) == ""++ # Sanity pair: a valid permissive scope on the same input passes it.+ allow_mgr = _build_manager(_snapshot("both"))+ assert allow_mgr._filter_urls_by_egress_policy(url) == url+++# ---------------------------------------------------------------------------+# I. Snapshot-less back-compat branch: no snapshot => unchanged passthrough.+# ---------------------------------------------------------------------------+++def test_snapshotless_manager_passes_through_but_private_only_filters():+ url = "discord://webhook_id/token"++ passthrough = _build_manager({}) # empty snapshot is falsy+ assert passthrough._filter_urls_by_egress_policy(url) == url++ filtered = _build_manager(_snapshot("private_only"))+ assert filtered._filter_urls_by_egress_policy(url) == ""+++# ---------------------------------------------------------------------------+# F. send_notification: the filter decision actually gates real dispatch.+# ---------------------------------------------------------------------------+++def test_send_notification_blocks_dispatch_when_policy_filters_all():+ snap_deny = _snapshot("private_only", service_url="discord://id/token")+ deny_mgr = _build_manager(snap_deny, outbound=True)+ # force=True bypasses per-user toggles + rate limit, NOT the egress filter.+ sent = deny_mgr.send_notification(EventType.TEST, {}, force=True)+ assert sent is False+ deny_mgr.service.send_event.assert_not_called()++ snap_allow = _snapshot("both", service_url="discord://id/token")+ allow_mgr = _build_manager(snap_allow, outbound=True)+ sent = allow_mgr.send_notification(EventType.TEST, {}, force=True)+ assert sent is True+ allow_mgr.service.send_event.assert_called_once()+ # The allowed subset (the discord url) is what gets handed to dispatch.+ _, kwargs = allow_mgr.service.send_event.call_args+ assert kwargs["service_urls"] == "discord://id/token"+++# ---------------------------------------------------------------------------+# G. Env master switch gates dispatch independently of the policy decision.+# ---------------------------------------------------------------------------+++def test_outbound_master_switch_blocks_even_policy_allowed_url():+ snap = _snapshot("both", service_url="discord://id/token")++ # Outbound OFF: a BOTH-scope, policy-allowed url is still not dispatched.+ off_mgr = _build_manager(snap, outbound=False)+ assert off_mgr.send_notification(EventType.TEST, {}, force=True) is False+ off_mgr.service.send_event.assert_not_called()++ # Flipping ONLY the master switch on lets the same url through.+ on_mgr = _build_manager(snap, outbound=True)+ assert on_mgr.send_notification(EventType.TEST, {}, force=True) is True+ on_mgr.service.send_event.assert_called_once()+++# ---------------------------------------------------------------------------+# H. test_service (the /api/notifications/test-url PEP) honors the policy.+# ---------------------------------------------------------------------------+++def test_test_service_endpoint_respects_private_only():+ url = "discord://webhook_id/token"++ deny_mgr = _build_manager(_snapshot("private_only"))+ result = deny_mgr.test_service(url)+ assert result["status"] == "error"+ assert "egress policy" in result["message"].lower()+ deny_mgr.service.test_service.assert_not_called()++ allow_mgr = _build_manager(_snapshot("both"))+ allow_mgr.test_service(url)+ allow_mgr.service.test_service.assert_called_once_with(url)diff --git a/tests/security/test_egress_pep_coverage.py b/tests/security/test_egress_pep_coverage.pynew file mode 100644index 0000000000..c82e7a389a--- /dev/null+++ b/tests/security/test_egress_pep_coverage.py@@ -0,0 +1,444 @@+"""Coverage for egress-policy enforcement points (PEPs) flagged as untested+by the PR #4300 multi-round review.++Each test exercises a real decision point against the real evaluate_url /+evaluate_engine / context_from_snapshot machinery (no mocking of the thing+under test), so a regression in either the PEP wiring OR the policy core+surfaces here.++Targets:+- fetch tools _enforce_url_policy (per-URL fetch PEP)+- meta_search_engine._wikipedia_fallback_allowed (fail-closed fallback gate)+- egress_policy.evaluate_engine for dynamic collection_<id>/library engines+- app_factory handle_policy_denied Flask error handler+"""++import pytest++from local_deep_research.security.egress.policy import (+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_engine,+)+++def _ctx(scope, primary="arxiv"):+ return context_from_snapshot(+ {"policy.egress_scope": scope}, primary_engine=primary+ )+++# ---------------------------------------------------------------------------+# fetch tools: _enforce_url_policy+# ---------------------------------------------------------------------------+++class TestEnforceUrlPolicy:+ """The fetch tool's per-URL PEP. Raises PolicyDeniedError on a denied+ URL so the agent's fetch_content tool can't egress outside scope."""++ def _enforce(self, url, ctx):+ from local_deep_research.advanced_search_system.tools.fetch import (+ _enforce_url_policy,+ )++ return _enforce_url_policy(url, ctx)++ def test_none_context_is_noop(self):+ # No context configured (legacy callers) => never raises.+ assert self._enforce("http://192.168.1.10/x", None) is None++ def test_public_url_allowed_under_public_only(self):+ # A public host under PUBLIC_ONLY passes (no raise).+ ctx = _ctx("public_only")+ assert self._enforce("https://arxiv.org/abs/1234", ctx) is None++ def test_private_ip_denied_under_public_only(self):+ # A private-IP literal under PUBLIC_ONLY is scope_mismatch => raise.+ ctx = _ctx("public_only")+ with pytest.raises(PolicyDeniedError):+ self._enforce("http://192.168.1.10/internal", ctx)++ def test_public_url_denied_under_private_only(self):+ # Under PRIVATE_ONLY a public host must be refused.+ ctx = _ctx("private_only")+ with pytest.raises(PolicyDeniedError):+ self._enforce("https://arxiv.org/abs/1234", ctx)+++# ---------------------------------------------------------------------------+# meta_search_engine._wikipedia_fallback_allowed+# ---------------------------------------------------------------------------+++class TestWikipediaFallbackAllowed:+ """Fail-closed gate for the meta engine's wikipedia fallback. Wikipedia+ is public, so it's allowed under PUBLIC_ONLY/BOTH and refused under+ PRIVATE_ONLY. A corrupt scope (PolicyDeniedError) or an invalid combo+ (ValueError, e.g. STRICT+meta) must BOTH fail closed (return False)."""++ def _engine(self, snapshot):+ from local_deep_research.web_search_engines.engines.meta_search_engine import (+ MetaSearchEngine,+ )++ # Bypass __init__ — we only need settings_snapshot for this method.+ engine = MetaSearchEngine.__new__(MetaSearchEngine)+ engine.settings_snapshot = snapshot+ return engine++ def test_no_snapshot_allows_fallback(self):+ # Offline path — children gated by the factory PEP downstream.+ assert self._engine(None)._wikipedia_fallback_allowed() is True+ assert self._engine({})._wikipedia_fallback_allowed() is True++ def test_public_only_allows_wikipedia(self):+ eng = self._engine(+ {"policy.egress_scope": "public_only", "search.tool": "arxiv"}+ )+ assert eng._wikipedia_fallback_allowed() is True++ def test_private_only_refuses_wikipedia(self):+ eng = self._engine(+ {"policy.egress_scope": "private_only", "search.tool": "library"}+ )+ assert eng._wikipedia_fallback_allowed() is False++ def test_corrupt_scope_fails_closed(self):+ # Unknown scope => context_from_snapshot raises PolicyDeniedError+ # => fallback must be refused, not allowed.+ eng = self._engine(+ {"policy.egress_scope": "garbage", "search.tool": "arxiv"}+ )+ assert eng._wikipedia_fallback_allowed() is False++ def test_strict_meta_value_error_fails_closed(self):+ # STRICT + meta-picker primary => context_from_snapshot raises+ # ValueError (not PolicyDeniedError). The handler must catch it+ # too and fail closed. Regression: ValueError was previously+ # uncaught and propagated.+ eng = self._engine(+ {"policy.egress_scope": "strict", "search.tool": "meta"}+ )+ assert eng._wikipedia_fallback_allowed() is False+++# ---------------------------------------------------------------------------+# evaluate_engine: dynamic collection_<id> / library engines+# ---------------------------------------------------------------------------+++class TestDynamicLocalEngines:+ """collection_<id> and library aren't in the static ENGINE_REGISTRY;+ evaluate_engine classifies them local by name (egress_policy ~322).+ Pin that distinct branch directly."""++ @pytest.mark.parametrize("name", ["library", "collection_abc123"])+ def test_local_under_private_only(self, name):+ d = evaluate_engine(+ name, _ctx("private_only", primary=name), settings_snapshot={}+ )+ assert d.allowed, f"{name} should be allowed under PRIVATE_ONLY"++ @pytest.mark.parametrize("name", ["library", "collection_abc123"])+ def test_filtered_under_public_only(self, name):+ d = evaluate_engine(name, _ctx("public_only"), settings_snapshot={})+ assert not d.allowed+ assert d.reason == "scope_mismatch_public_only"++ def test_unknown_dynamic_name_fails_closed(self):+ # A name that is neither registered nor a known-local prefix must+ # be refused (engine_unknown) — not silently allowed.+ d = evaluate_engine(+ "collflavor_not_a_real_prefix", _ctx("both"), settings_snapshot={}+ )+ assert not d.allowed+++# ---------------------------------------------------------------------------+# journal_reputation_filter._should_skip_journal_fetch_for_scope+# ---------------------------------------------------------------------------+++class TestJournalFetchScopeSkip:+ """Journal sources (OpenAlex/DOAJ/JabRef) are public, so under+ PRIVATE_ONLY/STRICT the filter should skip the fetch. A corrupt scope+ (PolicyDeniedError) must ALSO skip — fail closed — matching the+ hardened mcp_strategy/notifications siblings. Regression: the bare+ except previously swallowed PolicyDeniedError and returned False+ (fail open)."""++ def _filter(self, snapshot):+ from local_deep_research.advanced_search_system.filters.journal_reputation_filter import (+ JournalReputationFilter,+ )++ f = JournalReputationFilter.__new__(JournalReputationFilter)+ # Name-mangled private attribute the method reads.+ setattr(+ f,+ "_JournalReputationFilter__settings_snapshot",+ snapshot,+ )+ return f++ def test_no_snapshot_does_not_skip(self):+ assert (+ self._filter(None)._should_skip_journal_fetch_for_scope() is False+ )++ def test_private_only_skips(self):+ f = self._filter(+ {"policy.egress_scope": "private_only", "search.tool": "library"}+ )+ assert f._should_skip_journal_fetch_for_scope() is True++ def test_public_only_does_not_skip(self):+ f = self._filter(+ {"policy.egress_scope": "public_only", "search.tool": "arxiv"}+ )+ assert f._should_skip_journal_fetch_for_scope() is False++ def test_corrupt_scope_skips_fail_closed(self):+ f = self._filter(+ {"policy.egress_scope": "garbage", "search.tool": "arxiv"}+ )+ # Fail closed: corrupt scope => skip (do not fetch public journals).+ assert f._should_skip_journal_fetch_for_scope() is True+++# ---------------------------------------------------------------------------+# app_factory: handle_policy_denied Flask error handler+# ---------------------------------------------------------------------------+++class TestPolicyDeniedErrorHandler:+ """A PolicyDeniedError escaping any request-path PEP must become a clean+ 400 (not a 500 stack trace) via the global Flask error handler."""++ def test_policy_denied_returns_400_with_reason(self):+ from flask import Flask++ from local_deep_research.security.egress.policy import Decision++ app = Flask(__name__)++ # Register only the PolicyDeniedError handler the same way+ # app_factory does, so the test is hermetic.+ @app.errorhandler(PolicyDeniedError)+ def _handler(error):+ from flask import jsonify, make_response++ reason = getattr(+ getattr(error, "decision", None), "reason", "denied"+ )+ return make_response(+ jsonify({"status": "error", "message": f"refused: {reason}"}),+ 400,+ )++ @app.route("/boom")+ def _boom():+ raise PolicyDeniedError(+ Decision(False, "scope_mismatch_private_only"),+ target="https://x.example",+ )++ app.config["TESTING"] = True+ client = app.test_client()+ resp = client.get("/boom")+ assert resp.status_code == 400+ body = resp.get_json()+ assert body["status"] == "error"+ assert "scope_mismatch_private_only" in body["message"]+++# ---------------------------------------------------------------------------+# ContentFetcher.fetch — SSRF + scope enforcement at the fetch boundary+# ---------------------------------------------------------------------------+++class TestContentFetcherEgress:+ """ContentFetcher.fetch returns a structured error dict (not a raise)+ when a URL is SSRF-blocked or scope-incompatible. Denial returns early,+ before any downloader fires."""++ def _fetcher(self, scope):+ from local_deep_research.content_fetcher.fetcher import ContentFetcher++ return ContentFetcher(egress_context=_ctx(scope))++ def test_public_url_denied_under_private_only(self):+ # arxiv.org is public; PRIVATE_ONLY must refuse it at the scope axis.+ result = self._fetcher("private_only").fetch("https://arxiv.org/abs/1")+ assert result["status"] == "error"+ assert "egress policy" in result["error"].lower()++ def test_private_ip_blocked_under_public_only(self):+ # A private IP under PUBLIC_ONLY is blocked at the SSRF axis+ # (policy_aware_validate_url falls back to strict validate_url).+ result = self._fetcher("public_only").fetch("http://192.168.1.10/x")+ assert result["status"] == "error"+ # Either SSRF or scope rejection is acceptable — both are denials.+ assert (+ "ssrf" in result["error"].lower()+ or "egress policy" in result["error"].lower()+ or "security validation" in result["error"].lower()+ )++ def test_no_context_does_not_scope_block_public_url(self):+ from local_deep_research.content_fetcher.fetcher import ContentFetcher++ # No egress context => no scope axis; a public URL is not refused+ # for policy reasons (it may still fail later for network reasons,+ # but not with an egress-policy error).+ f = ContentFetcher(egress_context=None)+ # Use an invalid scheme so fetch short-circuits without network:+ result = f.fetch("ftp://example.com/x")+ assert result["status"] == "error"+ assert "egress policy" not in result["error"].lower()+++# ---------------------------------------------------------------------------+# rag_service_factory._enforce_embeddings_policy+# ---------------------------------------------------------------------------+++class TestEnforceEmbeddingsPolicy:+ """Pre-flight embeddings gate built from a SettingsManager (no full+ snapshot). Pins the llm.ollama.url fallback fix: ollama configured only+ via the shared llm.ollama.url must classify as local."""++ def _mgr(self, settings):+ from unittest.mock import Mock++ m = Mock()+ m.get_setting.side_effect = lambda key, default=None: settings.get(+ key, default+ )+ return m++ def _enforce(self, provider, settings):+ from local_deep_research.research_library.services.rag_service_factory import (+ _enforce_embeddings_policy,+ )++ return _enforce_embeddings_policy(provider, self._mgr(settings), "u")++ def test_noop_when_require_local_false(self):+ # No restriction => any provider passes (returns None, no raise).+ assert (+ self._enforce("openai", {"embeddings.require_local": False}) is None+ )++ def test_ollama_via_llm_url_allowed(self):+ # Regression: ollama configured ONLY via the shared llm.ollama.url+ # (embeddings.ollama.url unset) must be classified local. Before the+ # fix the minimal snapshot omitted llm.ollama.url -> wrongly denied.+ assert (+ self._enforce(+ "ollama",+ {+ "embeddings.require_local": True,+ "llm.ollama.url": "http://127.0.0.1:11434",+ },+ )+ is None+ )++ def test_openai_cloud_denied_under_require_local(self):+ with pytest.raises(PolicyDeniedError):+ self._enforce("openai", {"embeddings.require_local": True})+++# ---------------------------------------------------------------------------+# Scope -> require_local coupling (120-agent audit root fix)+# ---------------------------------------------------------------------------+++class TestScopeForcesLocalInference:+ """PRIVATE_ONLY means "my data stays on this box" — that only holds if+ BOTH inference paths are local. context_from_snapshot forces+ require_local_llm + require_local_embeddings under PRIVATE_ONLY even+ when the raw flags are at their default False. STRICT is search-only+ and is intentionally NOT coupled."""++ def test_private_only_forces_both_require_local(self):+ ctx = _ctx("private_only", primary="library")+ assert ctx.require_local_llm is True+ assert ctx.require_local_embeddings is True++ def test_strict_does_not_force_require_local(self):+ ctx = _ctx("strict", primary="arxiv")+ assert ctx.require_local_llm is False+ assert ctx.require_local_embeddings is False++ def test_both_does_not_force_require_local(self):+ ctx = _ctx("both", primary="arxiv")+ assert ctx.require_local_llm is False+ assert ctx.require_local_embeddings is False++ def test_explicit_flags_still_honored_under_both(self):+ # The flags remain an independent opt-in for the non-private scopes.+ ctx = context_from_snapshot(+ {+ "policy.egress_scope": "both",+ "llm.require_local_endpoint": True,+ "embeddings.require_local": True,+ },+ primary_engine="arxiv",+ )+ assert ctx.require_local_llm is True+ assert ctx.require_local_embeddings is True++ def test_embeddings_gate_fires_under_private_only_without_flag(self):+ """A PRIVATE_ONLY run with embeddings.require_local at its default+ False must STILL refuse a cloud embedder — the gate reads the+ scope-aware ctx.require_local_embeddings, not the raw flag.+ Regression: previously the gate branched on the raw flag and a+ PRIVATE_ONLY corpus could be embedded by openai."""+ from local_deep_research.embeddings.embeddings_config import (+ get_embeddings,+ )++ with pytest.raises(PolicyDeniedError):+ get_embeddings(+ provider="openai",+ settings_snapshot={"policy.egress_scope": "private_only"},+ )+++# ---------------------------------------------------------------------------+# Percent-encoded hostname bypass (evaluate_url)+# ---------------------------------------------------------------------------+++class TestPercentEncodedHostBypass:+ """evaluate_url must classify the DECODED host. HTTP clients decode+ percent-encoding in the netloc before connecting, so the encoded form+ must not read as a different (public/unresolvable) host than the one+ the socket actually reaches."""++ def test_encoded_private_ip_denied_under_public_only(self):+ from local_deep_research.security.egress.policy import evaluate_url++ ctx = _ctx("public_only")+ d = evaluate_url("http://192%2e168%2e1%2e1/internal", ctx)+ assert not d.allowed+ assert d.reason == "scope_mismatch_public_only"++ def test_encoded_loopback_denied_under_public_only(self):+ from local_deep_research.security.egress.policy import evaluate_url++ ctx = _ctx("public_only")+ d = evaluate_url("http://127%2e0%2e0%2e1/x", ctx)+ assert not d.allowed++ def test_encoded_private_ip_allowed_under_private_only(self):+ # The decoded host IS private, so PRIVATE_ONLY should permit it+ # (matching where the client actually connects).+ from local_deep_research.security.egress.policy import evaluate_url++ ctx = _ctx("private_only", primary="library")+ d = evaluate_url("http://192%2e168%2e1%2e1/x", ctx)+ assert d.alloweddiff --git a/tests/security/test_egress_pep_factory_callsite.py b/tests/security/test_egress_pep_factory_callsite.pynew file mode 100644index 0000000000..d0afc9b3e0--- /dev/null+++ b/tests/security/test_egress_pep_factory_callsite.py@@ -0,0 +1,238 @@+"""Integration tests for the egress-policy PEP at the search-engine factory+call site: ``web_search_engines.search_engine_factory.create_search_engine``.++These drive the REAL factory function (this file deliberately does NOT inherit+the ``tests/core`` factory fixtures that bypass the PDP), so the real+``evaluate_engine`` / ``context_from_snapshot`` enforcement actually fires.+Only the engine's heavy constructor is stubbed out (via ``get_safe_module_class``),+so every assertion is about the POLICY decision — which the factory makes+BEFORE it ever loads or instantiates the engine class.++Engine classifications used here come from the live ``ENGINE_REGISTRY``:+ - arxiv / wikipedia : public (is_public=True, is_local=False)+ - paperless : local (is_public=False, is_local=True)++Each scope axis is covered with an allow+deny pair so that reverting the PEP+(or loosening a scope check) flips a deny into an allow and fails the suite.+"""++from __future__ import annotations++from contextlib import contextmanager+from unittest.mock import MagicMock, patch++import pytest++from local_deep_research.security import clear_active_context+from local_deep_research.security.egress.policy import PolicyDeniedError+import local_deep_research.web_search_engines.search_engine_factory as factory+++class _FakeEngine:+ """Lightweight stand-in for a real search-engine class.++ Constructing it does no network/DB work, so the factory *reaching* this+ constructor is proof the PEP ALLOWED the engine — the denial path raises+ long before instantiation.+ """++ needs_llm_relevance_filter = False++ def __init__(self, **kwargs):+ self.llm = kwargs.get("llm")+ self.init_kwargs = kwargs++[email protected](autouse=True)+def _no_leaked_armed_context():+ """The factory PEP reads scope from the snapshot, not the armed audit-hook+ context. But clear the active context before and after each test anyway so+ a context leaked by another test file can never mask a decision here, and so+ this file never leaks one outward (hard rule: never leak armed context)."""+ clear_active_context()+ yield+ clear_active_context()+++def _snapshot(scope, primary, *engines, **extra):+ """Build a minimal settings snapshot the factory accepts.++ ``scope`` -> ``policy.egress_scope``; ``primary`` -> ``search.tool`` (the+ engine the EgressContext treats as primary); every name in ``engines`` is+ injected under ``search.engine.web`` so ``search_config()`` lists it — the+ factory fails closed on unknown names BEFORE the PEP, so a real engine must+ be present for the policy decision to be the thing under test.+ """+ snap = {+ "policy.egress_scope": scope,+ "search.tool": primary,+ }+ for name in engines:+ snap[f"search.engine.web.{name}.display_name"] = name+ snap.update(extra)+ return snap+++@contextmanager+def _stub_engine_construction():+ """Patch the factory's class loader so an ALLOWED engine is built as a cheap+ ``_FakeEngine`` instead of doing real network/DB construction.++ Yields the ``MagicMock`` loader so tests can assert whether construction was+ even attempted: for a DENIED engine the PEP must raise first and the loader+ must never be called. (``evaluate_engine`` reads engine class *flags* via a+ separate import in the policy module, so this mock only tracks the factory's+ own instantiation call site.)+ """+ loader = MagicMock(return_value=_FakeEngine)+ with patch.object(factory, "get_safe_module_class", loader):+ yield loader+++# ---------------------------------------------------------------------------+# PRIVATE_ONLY: public engines refused, local engines allowed.+# ---------------------------------------------------------------------------+++class TestPrivateOnlyScope:+ def test_public_engine_denied(self):+ # arxiv is a public engine; PRIVATE_ONLY must refuse it at the factory+ # PEP, before any engine class is loaded.+ snap = _snapshot("private_only", "library", "arxiv")+ with _stub_engine_construction() as loader:+ with pytest.raises(PolicyDeniedError) as exc:+ factory.create_search_engine("arxiv", settings_snapshot=snap)+ assert exc.value.decision.reason == "scope_mismatch_private_only"+ loader.assert_not_called()++ def test_local_engine_allowed(self):+ # paperless is a local engine; PRIVATE_ONLY permits it (allow pair).+ snap = _snapshot("private_only", "paperless", "paperless")+ with _stub_engine_construction() as loader:+ engine = factory.create_search_engine(+ "paperless", settings_snapshot=snap+ )+ assert isinstance(engine, _FakeEngine)+ loader.assert_called()+++# ---------------------------------------------------------------------------+# PUBLIC_ONLY: local engines refused, public engines allowed.+# ---------------------------------------------------------------------------+++class TestPublicOnlyScope:+ def test_local_engine_denied(self):+ # paperless is local; PUBLIC_ONLY must refuse it before construction.+ snap = _snapshot("public_only", "arxiv", "paperless")+ with _stub_engine_construction() as loader:+ with pytest.raises(PolicyDeniedError) as exc:+ factory.create_search_engine(+ "paperless", settings_snapshot=snap+ )+ assert exc.value.decision.reason == "scope_mismatch_public_only"+ loader.assert_not_called()++ def test_public_engine_allowed(self):+ # arxiv is public; PUBLIC_ONLY permits it (allow pair).+ snap = _snapshot("public_only", "arxiv", "arxiv")+ with _stub_engine_construction() as loader:+ engine = factory.create_search_engine(+ "arxiv", settings_snapshot=snap+ )+ assert isinstance(engine, _FakeEngine)+ loader.assert_called()+++# ---------------------------------------------------------------------------+# STRICT: only the primary engine may be built.+# ---------------------------------------------------------------------------+++class TestStrictScope:+ def test_non_primary_engine_denied(self):+ # STRICT permits ONLY the primary. With primary=arxiv, asking for a+ # different (also public) engine must be refused as strict_not_primary,+ # proving STRICT is enforced on the engine identity, not just the+ # public/local bucket.+ snap = _snapshot("strict", "arxiv", "arxiv", "wikipedia")+ with _stub_engine_construction() as loader:+ with pytest.raises(PolicyDeniedError) as exc:+ factory.create_search_engine(+ "wikipedia", settings_snapshot=snap+ )+ assert exc.value.decision.reason == "strict_not_primary"+ loader.assert_not_called()++ def test_primary_engine_allowed(self):+ # The primary itself is permitted under STRICT (allow pair).+ snap = _snapshot("strict", "arxiv", "arxiv")+ with _stub_engine_construction() as loader:+ engine = factory.create_search_engine(+ "arxiv", settings_snapshot=snap+ )+ assert isinstance(engine, _FakeEngine)+ loader.assert_called()+++# ---------------------------------------------------------------------------+# Meta-picker delegation + STRICT/meta incoherence at the factory.+# ---------------------------------------------------------------------------+++class TestMetaPickerDelegation:+ """Meta-pickers (auto/meta) are delegators: the factory permits them under+ non-STRICT scopes and gates their CHILDREN at child-instantiation time. The+ incoherent STRICT + meta-picker-primary combination is rejected at the+ factory PEP, because constructing the EgressContext for a concrete child+ under that snapshot raises (translated to invalid_policy_config)."""++ def test_meta_permitted_under_both(self):+ snap = _snapshot("both", "auto")+ with _stub_engine_construction() as loader:+ engine = factory.create_search_engine(+ "meta", settings_snapshot=snap+ )+ assert isinstance(engine, _FakeEngine)+ loader.assert_called()++ def test_auto_permitted_under_public_only(self):+ snap = _snapshot("public_only", "auto")+ with _stub_engine_construction() as loader:+ engine = factory.create_search_engine(+ "auto", settings_snapshot=snap+ )+ assert isinstance(engine, _FakeEngine)+ loader.assert_called()++ def test_strict_with_meta_primary_refused(self):+ # scope=STRICT with a meta-picker primary is incoherent. Building a+ # concrete child engine under that snapshot makes the factory PEP+ # construct an EgressContext, which raises and is surfaced as a+ # PolicyDeniedError(invalid_policy_config) — never silently allowed.+ snap = _snapshot("strict", "meta", "arxiv")+ with _stub_engine_construction() as loader:+ with pytest.raises(PolicyDeniedError) as exc:+ factory.create_search_engine("arxiv", settings_snapshot=snap)+ assert exc.value.decision.reason == "invalid_policy_config"+ loader.assert_not_called()+++# ---------------------------------------------------------------------------+# Unknown engine names fail closed (must not silently become 'auto' and+# bypass evaluate_engine — see factory comment at the unknown-name branch).+# ---------------------------------------------------------------------------+++class TestUnknownEngineFailsClosed:+ def test_unknown_name_raises_value_error_not_constructed(self):+ # A name that isn't a registered engine, retriever, or collection must+ # be rejected (ValueError) rather than silently rewritten to 'auto',+ # which historically bypassed the egress PEP entirely.+ snap = _snapshot("both", "arxiv", "arxiv")+ with _stub_engine_construction() as loader:+ with pytest.raises(ValueError):+ factory.create_search_engine(+ "definitely_not_a_real_engine", settings_snapshot=snap+ )+ loader.assert_not_called()diff --git a/tests/security/test_egress_pep_fetch_download_callsite.py b/tests/security/test_egress_pep_fetch_download_callsite.pynew file mode 100644index 0000000000..fe8ceddcf1--- /dev/null+++ b/tests/security/test_egress_pep_fetch_download_callsite.py@@ -0,0 +1,321 @@+"""Integration tests for the REAL fetch/download Policy Enforcement Points.++These exercise the call sites that actually consult the egress PDP — NOT the+PDP functions themselves (those have their own ~390 unit tests). The point is+to prove that the policy decision actually fires at the call site:++ * ``content_fetcher.fetcher.ContentFetcher.fetch`` returns the structured+ egress-policy error for a scope-incompatible URL and never reaches a+ downloader, while a scope-compatible URL passes the gate.+ * ``ContentFetcher._apply_egress_policy_to_downloader`` relaxes a+ downloader's SafeSession only under PRIVATE_ONLY.+ * ``security.egress.fetch.policy_aware_validate_url`` threads the scope+ through SSRF validation (private IPs only under PRIVATE_ONLY; cloud+ metadata always blocked).+ * ``research_library.services.download_service.DownloadService.+ _check_url_against_policy`` denies / allows per scope and always blocks+ cloud-metadata.++All tests use literal IPs so no DNS lookup or outbound connect ever happens;+the policy gate decides before any network work. Each assertion is paired+(allow + deny) so it fails if the security property were reverted.+"""++from __future__ import annotations++from unittest.mock import Mock, patch++import pytest++from local_deep_research.content_fetcher.fetcher import ContentFetcher+from local_deep_research.security.egress.audit_hook import (+ clear_active_context,+ get_active_context,+)+from local_deep_research.security.egress.fetch import policy_aware_validate_url+from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+)++# Literal IPs — classified without any DNS lookup.+PUBLIC_IP_URL = "http://8.8.8.8/paper.html"+PRIVATE_IP_URL = "http://192.168.0.50/paper.html"+METADATA_IP_URL = "http://169.254.169.254/latest/meta-data/"+++def make_ctx(scope: EgressScope, primary: str = "arxiv") -> EgressContext:+ """Build a concrete-scope EgressContext directly (no snapshot/DNS)."""+ return EgressContext(+ scope=scope,+ primary_engine=primary,+ require_local_llm=False,+ require_local_embeddings=False,+ )++[email protected](autouse=True)+def _no_armed_context():+ """Keep the audit-hook thread-local clean so a leaked armed context from+ another test can't anchor evaluate_url's per-run denial quota here, and so+ we never leak one ourselves.+ """+ clear_active_context()+ try:+ yield+ finally:+ clear_active_context()+ assert get_active_context() is None+++class _FakeResult:+ """Minimal stand-in for a downloader's download_with_result() return."""++ def __init__(self, content=b"hello world", is_success=True):+ self.is_success = is_success+ self.content = content+ self.skip_reason = None+++# ---------------------------------------------------------------------------+# ContentFetcher.fetch — the central URL PEP+# ---------------------------------------------------------------------------+++class TestContentFetcherFetchPEP:+ def test_public_url_denied_under_private_only_never_reaches_downloader(+ self,+ ):+ """PRIVATE_ONLY must refuse a public host with the structured egress+ error AND short-circuit before any downloader is constructed/called.+ """+ fetcher = ContentFetcher(+ egress_context=make_ctx(EgressScope.PRIVATE_ONLY)+ )+ with patch.object(fetcher, "_get_downloader") as mock_get:+ result = fetcher.fetch(PUBLIC_IP_URL)+ assert result["status"] == "error"+ assert "egress policy" in result["error"]+ # The gate fired BEFORE downloader selection — the security property.+ mock_get.assert_not_called()++ def test_private_url_allowed_under_private_only_reaches_downloader(self):+ """The allow half of the pair: a private host under PRIVATE_ONLY passes+ both the SSRF and the egress gate and is handed to a downloader.+ """+ fetcher = ContentFetcher(+ egress_context=make_ctx(EgressScope.PRIVATE_ONLY)+ )+ downloader = Mock()+ downloader.download_with_result.return_value = _FakeResult()+ downloader.get_metadata.return_value = {"title": "T"}+ with patch.object(fetcher, "_get_downloader", return_value=downloader):+ result = fetcher.fetch(PRIVATE_IP_URL)+ assert result["status"] == "success"+ assert "egress policy" not in (result.get("error") or "")+ downloader.download_with_result.assert_called_once()++ def test_private_url_denied_under_public_only(self):+ """Mirror scope: PUBLIC_ONLY refuses the same private host that+ PRIVATE_ONLY allowed — proves the decision is scope-driven, not a+ blanket private-IP block.+ """+ fetcher = ContentFetcher(+ egress_context=make_ctx(EgressScope.PUBLIC_ONLY)+ )+ with patch.object(fetcher, "_get_downloader") as mock_get:+ result = fetcher.fetch(PRIVATE_IP_URL)+ assert result["status"] == "error"+ # Private host under PUBLIC_ONLY fails SSRF first (strict default),+ # so either the SSRF error or the egress error is acceptable — the+ # security property is simply "denied, no downloader".+ assert "egress policy" in result["error"] or "SSRF" in result["error"]+ mock_get.assert_not_called()++ def test_public_url_allowed_under_public_only(self):+ fetcher = ContentFetcher(+ egress_context=make_ctx(EgressScope.PUBLIC_ONLY)+ )+ downloader = Mock()+ downloader.download_with_result.return_value = _FakeResult()+ downloader.get_metadata.return_value = {}+ with patch.object(fetcher, "_get_downloader", return_value=downloader):+ result = fetcher.fetch(PUBLIC_IP_URL)+ assert result["status"] == "success"+ downloader.download_with_result.assert_called_once()++ def test_metadata_url_denied_even_under_private_only(self):+ """Cloud-metadata is NEVER fetchable, even under the scope that allows+ private hosts. Without an egress_context there is no scope gate, so we+ arm PRIVATE_ONLY and confirm the metadata IP is still refused.+ """+ fetcher = ContentFetcher(+ egress_context=make_ctx(EgressScope.PRIVATE_ONLY)+ )+ with patch.object(fetcher, "_get_downloader") as mock_get:+ result = fetcher.fetch(METADATA_IP_URL)+ assert result["status"] == "error"+ mock_get.assert_not_called()+++# ---------------------------------------------------------------------------+# policy_aware_validate_url — the SSRF/scope-threading PEP helper+# ---------------------------------------------------------------------------+++class TestPolicyAwareValidateUrl:+ def test_private_ip_allowed_only_under_private_only(self):+ # Deny under BOTH (strict SSRF default) ...+ assert (+ policy_aware_validate_url(+ PRIVATE_IP_URL, make_ctx(EgressScope.BOTH)+ )+ is False+ )+ # ... and under no-context (back-compat strict) ...+ assert policy_aware_validate_url(PRIVATE_IP_URL, None) is False+ # ... but allowed under PRIVATE_ONLY (the documented relaxation).+ assert (+ policy_aware_validate_url(+ PRIVATE_IP_URL, make_ctx(EgressScope.PRIVATE_ONLY)+ )+ is True+ )++ def test_metadata_ip_blocked_even_under_private_only(self):+ # A public IP under PRIVATE_ONLY relaxation passes SSRF, proving the+ # relaxation is live; the metadata IP under the SAME scope is still+ # blocked — the always-blocked invariant.+ assert (+ policy_aware_validate_url(+ PUBLIC_IP_URL, make_ctx(EgressScope.PRIVATE_ONLY)+ )+ is True+ )+ assert (+ policy_aware_validate_url(+ METADATA_IP_URL, make_ctx(EgressScope.PRIVATE_ONLY)+ )+ is False+ )+++# ---------------------------------------------------------------------------+# ContentFetcher._apply_egress_policy_to_downloader — session relaxation PEP+# ---------------------------------------------------------------------------+++class TestApplyEgressPolicyToDownloader:+ def _downloader_with_session(self):+ downloader = Mock()+ session = Mock()+ session.allow_private_ips = False+ downloader.session = session+ return downloader, session++ def test_relaxes_session_under_private_only(self):+ fetcher = ContentFetcher(+ egress_context=make_ctx(EgressScope.PRIVATE_ONLY)+ )+ downloader, session = self._downloader_with_session()+ fetcher._apply_egress_policy_to_downloader(downloader)+ assert session.allow_private_ips is True++ def test_does_not_relax_session_under_public_only(self):+ fetcher = ContentFetcher(+ egress_context=make_ctx(EgressScope.PUBLIC_ONLY)+ )+ downloader, session = self._downloader_with_session()+ fetcher._apply_egress_policy_to_downloader(downloader)+ assert session.allow_private_ips is False++ def test_does_not_relax_session_under_both(self):+ fetcher = ContentFetcher(egress_context=make_ctx(EgressScope.BOTH))+ downloader, session = self._downloader_with_session()+ fetcher._apply_egress_policy_to_downloader(downloader)+ assert session.allow_private_ips is False++ def test_no_context_does_not_relax_session(self):+ fetcher = ContentFetcher(egress_context=None)+ downloader, session = self._downloader_with_session()+ fetcher._apply_egress_policy_to_downloader(downloader)+ assert session.allow_private_ips is False+++# ---------------------------------------------------------------------------+# DownloadService._check_url_against_policy — library download PEP+# ---------------------------------------------------------------------------+++def _make_download_service(tmp_path, scope_value):+ """Construct a real DownloadService whose EgressContext is built from a+ snapshot carrying ``scope_value``. Heavy deps (settings backend, retry+ manager, library dir) are mocked; the policy gate under test is real.+ """+ mock_settings = Mock()+ mock_settings.get_setting.side_effect = lambda key, default=None: {+ "research_library.storage_path": str(tmp_path),+ "search.engine.web.semantic_scholar.api_key": "",+ }.get(key, default)++ snapshot = {+ "policy.egress_scope": scope_value,+ "search.tool": "arxiv",+ }++ with (+ patch(+ "local_deep_research.research_library.services."+ "download_service.get_settings_manager",+ return_value=mock_settings,+ ),+ patch(+ "local_deep_research.research_library.services."+ "download_service.RetryManager",+ ),+ patch(+ "local_deep_research.research_library.services."+ "download_service.get_library_directory",+ return_value=tmp_path,+ ),+ ):+ from local_deep_research.research_library.services.download_service import ( # noqa: E501+ DownloadService,+ )++ return DownloadService("tester", "pw", settings_snapshot=snapshot)+++class TestDownloadServiceCheckUrlPEP:+ def test_public_denied_private_allowed_under_private_only(self, tmp_path):+ service = _make_download_service(tmp_path, "private_only")+ try:+ assert service._egress_context is not None+ assert service._egress_context.scope == EgressScope.PRIVATE_ONLY+ pub_allowed, _ = service._check_url_against_policy(PUBLIC_IP_URL)+ priv_allowed, _ = service._check_url_against_policy(PRIVATE_IP_URL)+ assert pub_allowed is False+ assert priv_allowed is True+ finally:+ service.close()++ def test_public_allowed_private_denied_under_public_only(self, tmp_path):+ service = _make_download_service(tmp_path, "public_only")+ try:+ assert service._egress_context.scope == EgressScope.PUBLIC_ONLY+ pub_allowed, _ = service._check_url_against_policy(PUBLIC_IP_URL)+ priv_allowed, _ = service._check_url_against_policy(PRIVATE_IP_URL)+ assert pub_allowed is True+ assert priv_allowed is False+ finally:+ service.close()++ def test_metadata_url_always_denied(self, tmp_path):+ # Denied under the scope that otherwise permits private hosts.+ service = _make_download_service(tmp_path, "private_only")+ try:+ allowed, reason = service._check_url_against_policy(METADATA_IP_URL)+ assert allowed is False+ assert reason == "blocked_metadata_ip"+ finally:+ service.close()diff --git a/tests/security/test_egress_pep_llm_embeddings_callsite.py b/tests/security/test_egress_pep_llm_embeddings_callsite.pynew file mode 100644index 0000000000..3e1286917a--- /dev/null+++ b/tests/security/test_egress_pep_llm_embeddings_callsite.py@@ -0,0 +1,418 @@+"""Integration tests for the egress-policy PEPs at the inference call sites.++These do NOT re-test the PDP (``evaluate_llm_endpoint`` /+``evaluate_embeddings``) — that lives in tests/security/test_egress_policy.py+and test_egress_inference_gates_followup.py. Instead they drive the REAL+enforcement points:++ config.llm_config.get_llm(...)+ embeddings.embeddings_config.get_embeddings(...)++and assert the policy gate fires (PolicyDeniedError) BEFORE any model/client+construction happens. Built-in providers are auto-registered in the LLM+registry by ``discover_providers()`` (triggered on importing llm_config), so a+real ``get_llm`` call dispatches through ``get_llm_from_registry`` — we mock+ONLY that final construction step (and the embeddings ``create_embeddings``+classmethods) so what's exercised is the gate + the dispatch decision, never+the network/heavy model build.++Each allow/deny pair is constructed so the deny assertion FAILS if the gate+were removed: the construction mock returns a valid model, so a reverted gate+would let the cloud provider succeed instead of raising — and the+``assert_not_called`` on the construction mock proves the gate short-circuits+upstream of it.+"""++from __future__ import annotations++from unittest.mock import Mock, patch++import pytest+from langchain_core.embeddings import Embeddings+from langchain_core.language_models import BaseChatModel++from local_deep_research.config.llm_config import get_llm+from local_deep_research.embeddings.embeddings_config import get_embeddings+from local_deep_research.embeddings.providers.implementations.ollama import (+ OllamaEmbeddingsProvider,+)+from local_deep_research.embeddings.providers.implementations.openai import (+ OpenAIEmbeddingsProvider,+)+from local_deep_research.embeddings.providers.implementations.sentence_transformers import (+ SentenceTransformersProvider,+)+from local_deep_research.llm.llm_registry import register_llm, unregister_llm+from local_deep_research.security.egress.policy import PolicyDeniedError++# Snapshot fragments that force local-only inference, via the two distinct+# routes the call sites must both honor:+# - PRIVATE_ONLY scope: context_from_snapshot IMPLIES require_local_*.+# - the explicit per-knob flag with an otherwise-permissive scope.+PRIVATE_ONLY = {"policy.egress_scope": "private_only"}+LLM_REQUIRE_LOCAL_FLAG = {+ "policy.egress_scope": "both",+ "llm.require_local_endpoint": True,+}+EMB_REQUIRE_LOCAL_FLAG = {+ "policy.egress_scope": "both",+ "embeddings.require_local": True,+}++[email protected](autouse=True)+def _no_leaked_active_context():+ """Belt-and-suspenders: these PEPs gate inside get_llm/get_embeddings+ without arming the audit hook, but if any test ever does, never let the+ armed context leak into a sibling test's socket path."""+ from local_deep_research.security.egress.audit_hook import (+ clear_active_context,+ )++ try:+ yield+ finally:+ clear_active_context()+++# ---------------------------------------------------------------------------+# get_llm — cloud providers blocked under require-local (both routes)+# ---------------------------------------------------------------------------+++class TestGetLlmCloudBlocked:+ @pytest.mark.parametrize("provider", ["openai", "anthropic"])+ def test_cloud_provider_denied_under_private_only(self, provider):+ """A cloud LLM under PRIVATE_ONLY must raise PolicyDeniedError before+ the registry dispatch — proven by asserting the construction mock is+ never reached."""+ snapshot = {+ **PRIVATE_ONLY,+ "llm.openai.api_key": "k", # gitleaks:allow+ "llm.anthropic.api_key": "k", # gitleaks:allow+ }+ with patch(+ "local_deep_research.config.llm_config.get_llm_from_registry"+ ) as mock_registry:+ mock_registry.return_value = Mock(spec=BaseChatModel)+ with pytest.raises(PolicyDeniedError) as exc:+ get_llm(+ provider=provider,+ model_name="some-cloud-model",+ settings_snapshot=snapshot,+ )+ assert exc.value.decision.reason == "provider_cloud_only"+ # The gate fired upstream of model construction.+ mock_registry.assert_not_called()++ def test_cloud_provider_denied_under_explicit_require_local_flag(self):+ """The explicit llm.require_local_endpoint=true flag (scope BOTH) must+ gate cloud providers the same way the PRIVATE_ONLY scope does."""+ snapshot = {+ **LLM_REQUIRE_LOCAL_FLAG,+ "llm.openai.api_key": "k",+ } # gitleaks:allow+ with patch(+ "local_deep_research.config.llm_config.get_llm_from_registry"+ ) as mock_registry:+ mock_registry.return_value = Mock(spec=BaseChatModel)+ with pytest.raises(PolicyDeniedError):+ get_llm(+ provider="openai",+ model_name="gpt-4o",+ settings_snapshot=snapshot,+ )+ mock_registry.assert_not_called()++ def test_cloud_provider_allowed_when_no_local_requirement(self):+ """Mirror (allow side): with scope BOTH and the flag off, the SAME+ cloud provider passes the gate and reaches construction. Without this+ pairing the deny tests could pass for an unrelated reason."""+ snapshot = {+ "policy.egress_scope": "both",+ "llm.openai.api_key": "k",+ } # gitleaks:allow+ with patch(+ "local_deep_research.config.llm_config.get_llm_from_registry"+ ) as mock_registry:+ mock_registry.return_value = Mock(spec=BaseChatModel)+ result = get_llm(+ provider="openai",+ model_name="gpt-4o",+ settings_snapshot=snapshot,+ )+ assert result is not None+ mock_registry.assert_called_once()+++# ---------------------------------------------------------------------------+# get_llm — local providers allowed under require-local+# ---------------------------------------------------------------------------+++class TestGetLlmLocalAllowed:+ @pytest.mark.parametrize("provider", ["ollama", "lmstudio"])+ def test_local_default_provider_allowed_under_private_only(self, provider):+ """A localhost-default provider (no URL override) passes the gate under+ PRIVATE_ONLY and reaches construction (provider_local_default)."""+ with patch(+ "local_deep_research.config.llm_config.get_llm_from_registry"+ ) as mock_registry:+ mock_registry.return_value = Mock(spec=BaseChatModel)+ result = get_llm(+ provider=provider,+ model_name="local-model",+ settings_snapshot=PRIVATE_ONLY,+ )+ assert result is not None+ # The real registry dispatch ran (gate did not short-circuit it).+ mock_registry.assert_called_once()++ def test_local_provider_with_remote_url_denied(self):+ """The deny half of the local-provider pair: ollama pointed at a REMOTE+ url is refused even though it is normally a local-default provider —+ the gate classifies the configured endpoint, not the provider name."""+ snapshot = {+ **PRIVATE_ONLY,+ "llm.ollama.url": "https://ollama.example.com",+ }+ with patch(+ "local_deep_research.config.llm_config.get_llm_from_registry"+ ) as mock_registry:+ mock_registry.return_value = Mock(spec=BaseChatModel)+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ with pytest.raises(PolicyDeniedError) as exc:+ get_llm(+ provider="ollama",+ model_name="llama3",+ settings_snapshot=snapshot,+ )+ assert exc.value.decision.reason == "provider_remote"+ mock_registry.assert_not_called()+++# ---------------------------------------------------------------------------+# get_llm — user-registered in-process LLM is exempt (allowed)+# ---------------------------------------------------------------------------+++class TestGetLlmUserRegistered:+ def test_user_registered_llm_allowed_under_private_only(self):+ """An operator-injected in-process LLM (programmatic API / plugin) has+ no endpoint to classify, so the gate allows it under PRIVATE_ONLY+ (user_registered_llm). The audit hook is the backstop for any stray+ socket it might open."""+ name = "_pep_callsite_inproc_llm"+ register_llm(name, Mock(spec=BaseChatModel))+ try:+ result = get_llm(+ provider=name,+ settings_snapshot=PRIVATE_ONLY,+ )+ assert result is not None+ finally:+ unregister_llm(name)++ def test_shadowing_registration_of_cloud_name_still_blocked(self):+ """Deny half: registering an in-process object under a BUILT-IN cloud+ name ("openai") must NOT smuggle it past the gate — the cloud-block+ fires first on the discovered name. Guards the _is_user_registered_llm+ shadowing discriminator at the real call site."""+ register_llm("openai", Mock(spec=BaseChatModel))+ try:+ with pytest.raises(PolicyDeniedError) as exc:+ get_llm(+ provider="openai",+ model_name="gpt-4o",+ settings_snapshot=PRIVATE_ONLY,+ )+ assert exc.value.decision.reason == "provider_cloud_only"+ finally:+ # Restore the auto-discovered built-in factory so we don't leak a+ # mock into other tests' registry.+ from local_deep_research.llm.providers import discover_providers++ discover_providers(force_refresh=True)+++# ---------------------------------------------------------------------------+# get_llm — snapshot-less fail-closed allow-list+# ---------------------------------------------------------------------------+++class TestGetLlmSnapshotless:+ def test_snapshotless_cloud_provider_fails_closed(self):+ """No snapshot => the require-local toggle is unreadable, so a non-local+ provider must fail closed (no_snapshot_for_provider) rather than+ silently instantiate a cloud client."""+ with patch(+ "local_deep_research.config.llm_config.get_llm_from_registry"+ ) as mock_registry:+ mock_registry.return_value = Mock(spec=BaseChatModel)+ with pytest.raises(PolicyDeniedError) as exc:+ get_llm(+ provider="openai",+ model_name="gpt-4o",+ settings_snapshot=None,+ )+ assert exc.value.decision.reason == "no_snapshot_for_provider"+ mock_registry.assert_not_called()++ def test_snapshotless_local_default_provider_allowed(self):+ """Mirror: a localhost-default provider is on the snapshot-less+ allow-list and proceeds to construction even with no snapshot."""+ with patch(+ "local_deep_research.config.llm_config.get_llm_from_registry"+ ) as mock_registry:+ mock_registry.return_value = Mock(spec=BaseChatModel)+ result = get_llm(+ provider="ollama",+ model_name="llama3",+ settings_snapshot=None,+ )+ assert result is not None+ mock_registry.assert_called_once()+++# ---------------------------------------------------------------------------+# get_embeddings — cloud embedder blocked under require-local+# ---------------------------------------------------------------------------+++class TestGetEmbeddingsCloudBlocked:+ def test_openai_denied_under_explicit_require_local_flag(self):+ """A cloud embedder under embeddings.require_local=true must raise+ before create_embeddings (which would ship the corpus to the cloud)."""+ with patch.object(+ OpenAIEmbeddingsProvider, "create_embeddings"+ ) as mock_create:+ with pytest.raises(PolicyDeniedError) as exc:+ get_embeddings(+ provider="openai",+ settings_snapshot=EMB_REQUIRE_LOCAL_FLAG,+ )+ assert exc.value.decision.reason == "provider_cloud"+ mock_create.assert_not_called()++ def test_openai_allowed_when_no_local_requirement(self):+ """Mirror: with require-local off, the same cloud embedder reaches+ construction — so the deny test isn't passing for an unrelated reason."""+ with patch.object(+ OpenAIEmbeddingsProvider,+ "create_embeddings",+ return_value=Mock(spec=Embeddings),+ ) as mock_create:+ result = get_embeddings(+ provider="openai",+ settings_snapshot={+ "policy.egress_scope": "both",+ "embeddings.require_local": False,+ },+ )+ assert result is not None+ mock_create.assert_called_once()++ def test_openai_denied_under_private_only_scope(self):+ """PRIVATE_ONLY implies require_local_embeddings even with the raw flag+ left at its default — a cloud embedder is refused via the scope route."""+ with patch.object(+ OpenAIEmbeddingsProvider, "create_embeddings"+ ) as mock_create:+ with pytest.raises(PolicyDeniedError):+ get_embeddings(+ provider="openai",+ settings_snapshot=PRIVATE_ONLY,+ )+ mock_create.assert_not_called()+++# ---------------------------------------------------------------------------+# get_embeddings — local embedders allowed under require-local+# ---------------------------------------------------------------------------+++class TestGetEmbeddingsLocalAllowed:+ def test_sentence_transformers_allowed_under_require_local(self):+ """In-process sentence_transformers passes the gate and constructs."""+ with patch.object(+ SentenceTransformersProvider,+ "create_embeddings",+ return_value=Mock(spec=Embeddings),+ ) as mock_create:+ result = get_embeddings(+ provider="sentence_transformers",+ settings_snapshot=EMB_REQUIRE_LOCAL_FLAG,+ )+ assert result is not None+ mock_create.assert_called_once()++ def test_ollama_local_default_allowed_under_require_local(self):+ """ollama embeddings (no URL override) falls back to its localhost+ default and is allowed."""+ with patch.object(+ OllamaEmbeddingsProvider,+ "create_embeddings",+ return_value=Mock(spec=Embeddings),+ ) as mock_create:+ result = get_embeddings(+ provider="ollama",+ settings_snapshot=EMB_REQUIRE_LOCAL_FLAG,+ )+ assert result is not None+ mock_create.assert_called_once()++ def test_ollama_remote_url_denied_under_require_local(self):+ """Deny half of the ollama pair: a remote ollama endpoint is refused+ even under the local-default provider name."""+ snapshot = {+ **EMB_REQUIRE_LOCAL_FLAG,+ "embeddings.ollama.url": "https://ollama.example.com",+ }+ with patch.object(+ OllamaEmbeddingsProvider, "create_embeddings"+ ) as mock_create:+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ with pytest.raises(PolicyDeniedError) as exc:+ get_embeddings(+ provider="ollama", settings_snapshot=snapshot+ )+ assert exc.value.decision.reason == "provider_remote"+ mock_create.assert_not_called()+++# ---------------------------------------------------------------------------+# get_embeddings — snapshot-less fail-closed allow-list+# ---------------------------------------------------------------------------+++class TestGetEmbeddingsSnapshotless:+ def test_snapshotless_openai_fails_closed(self):+ """No snapshot => cloud embedder fails closed (no_snapshot_for_provider)+ rather than silently embedding the local corpus in the cloud."""+ with patch.object(+ OpenAIEmbeddingsProvider, "create_embeddings"+ ) as mock_create:+ with pytest.raises(PolicyDeniedError) as exc:+ get_embeddings(provider="openai", settings_snapshot=None)+ assert exc.value.decision.reason == "no_snapshot_for_provider"+ mock_create.assert_not_called()++ def test_snapshotless_sentence_transformers_allowed(self):+ """Mirror: an in-process local embedder is on the snapshot-less+ allow-list and constructs even with no snapshot."""+ with patch.object(+ SentenceTransformersProvider,+ "create_embeddings",+ return_value=Mock(spec=Embeddings),+ ) as mock_create:+ result = get_embeddings(+ provider="sentence_transformers", settings_snapshot=None+ )+ assert result is not None+ mock_create.assert_called_once()diff --git a/tests/security/test_egress_policy.py b/tests/security/test_egress_policy.pynew file mode 100644index 0000000000..ec494805f5--- /dev/null+++ b/tests/security/test_egress_policy.py@@ -0,0 +1,2176 @@+"""Unit tests for the egress policy module (Stage 1a)."""++from __future__ import annotations++from unittest.mock import patch++import pytest++from local_deep_research.security.egress.policy import (+ Decision,+ EgressContext,+ EgressScope,+ MAX_DENIED_FETCHES_PER_RUN,+ PolicyDeniedError,+ context_from_snapshot,+ evaluate_engine,+ evaluate_embeddings,+ evaluate_llm_endpoint,+ evaluate_url,+)+++# ---------------------------------------------------------------------------+# Helpers+# ---------------------------------------------------------------------------+++def make_ctx(+ scope: EgressScope = EgressScope.BOTH,+ primary: str = "arxiv",+ require_local_llm: bool = False,+ require_local_embeddings: bool = False,+ local_hostnames=(),+) -> EgressContext:+ return EgressContext(+ scope=scope,+ primary_engine=primary,+ require_local_llm=require_local_llm,+ require_local_embeddings=require_local_embeddings,+ local_hostnames=tuple(local_hostnames),+ )+++# ---------------------------------------------------------------------------+# context_from_snapshot+# ---------------------------------------------------------------------------+++def test_context_from_snapshot_defaults_to_both():+ ctx = context_from_snapshot({}, primary_engine="arxiv")+ assert ctx.scope == EgressScope.BOTH+ assert ctx.require_local_llm is False+ assert ctx.require_local_embeddings is False+ assert ctx.local_hostnames == ()+++def _scope_snapshot(scope, tool):+ return {+ "policy.egress_scope": {"value": scope},+ "search.tool": {"value": tool},+ }+++def test_adaptive_meta_picker_primary_resolves_to_both():+ ctx = context_from_snapshot(+ _scope_snapshot("adaptive", "auto"), primary_engine="auto"+ )+ assert ctx.scope == EgressScope.BOTH+ assert ctx.require_local_llm is False+++def test_adaptive_public_primary_resolves_to_public_only():+ ctx = context_from_snapshot(+ _scope_snapshot("adaptive", "arxiv"), primary_engine="arxiv"+ )+ assert ctx.scope == EgressScope.PUBLIC_ONLY+ # public scope does NOT force local inference+ assert ctx.require_local_llm is False+++def test_adaptive_private_primary_resolves_to_private_only_and_forces_local():+ # 'library' is the always-private aggregate engine.+ ctx = context_from_snapshot(+ _scope_snapshot("adaptive", "library"), primary_engine="library"+ )+ assert ctx.scope == EgressScope.PRIVATE_ONLY+ # private primary under adaptive must force local inference (coupling)+ assert ctx.require_local_llm is True+ assert ctx.require_local_embeddings is True+++def test_adaptive_falls_back_to_both_on_classification_error():+ # An unknown concrete engine can't be classified → BOTH (permissive+ # fallback, never a hard fail).+ ctx = context_from_snapshot(+ _scope_snapshot("adaptive", "totally_unknown_engine"),+ primary_engine="totally_unknown_engine",+ )+ assert ctx.scope == EgressScope.BOTH+++def test_public_collection_allowed_under_public_only_via_metadata():+ """A collection flagged public classifies as a public engine: allowed+ under PUBLIC_ONLY, and (mirror) a private one is denied."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY, primary="collection_abc")+ pub = evaluate_engine(+ "collection_abc",+ ctx,+ settings_snapshot={"policy.egress_scope": {"value": "public_only"}},+ metadata={"is_public": True},+ )+ assert pub.allowed is True+ priv = evaluate_engine(+ "collection_abc",+ ctx,+ settings_snapshot={"policy.egress_scope": {"value": "public_only"}},+ metadata={"is_public": False},+ )+ assert priv.allowed is False+ assert priv.reason == "scope_mismatch_public_only"+++def test_private_collection_allowed_under_private_only_via_metadata():+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY, primary="collection_abc")+ priv = evaluate_engine(+ "collection_abc",+ ctx,+ settings_snapshot={"policy.egress_scope": {"value": "private_only"}},+ metadata={"is_public": False},+ )+ assert priv.allowed is True+ # A collection is ALWAYS a local KB, so a PUBLIC collection is ALSO usable+ # under PRIVATE_ONLY (is_public is additive, not exclusive — the content is+ # local regardless; the flag only ADDS public-scope/cloud eligibility).+ pub = evaluate_engine(+ "collection_abc",+ ctx,+ settings_snapshot={"policy.egress_scope": {"value": "private_only"}},+ metadata={"is_public": True},+ )+ assert pub.allowed is True+++def test_public_collection_is_local_and_public_every_scope():+ """A public collection is classified (is_public=True, is_local=True): it is+ allowed under PUBLIC_ONLY, PRIVATE_ONLY and BOTH. A private collection is+ local-only (excluded from PUBLIC_ONLY)."""+ pub_meta = {"is_public": True}+ priv_meta = {"is_public": False}+ for scope in (+ EgressScope.PUBLIC_ONLY,+ EgressScope.PRIVATE_ONLY,+ EgressScope.BOTH,+ ):+ ctx = make_ctx(scope=scope, primary="collection_abc")+ assert evaluate_engine(+ "collection_abc", ctx, settings_snapshot={}, metadata=pub_meta+ ).allowed, f"public collection should be allowed under {scope}"+ # Private collection: denied only under PUBLIC_ONLY.+ ctx_pub = make_ctx(scope=EgressScope.PUBLIC_ONLY, primary="collection_abc")+ assert not evaluate_engine(+ "collection_abc", ctx_pub, settings_snapshot={}, metadata=priv_meta+ ).allowed+ for scope in (EgressScope.PRIVATE_ONLY, EgressScope.BOTH):+ ctx = make_ctx(scope=scope, primary="collection_abc")+ assert evaluate_engine(+ "collection_abc", ctx, settings_snapshot={}, metadata=priv_meta+ ).allowed, f"private collection should be allowed under {scope}"+++def test_collection_without_metadata_defaults_private_via_db_lookup():+ """Without metadata, evaluate_engine resolves the collection's is_public+ from the DB; a lookup failure fails closed to private (local)."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY, primary="collection_xyz")+ # No DB available in this unit test → _resolve_collection_is_public+ # returns False (private) → allowed under PRIVATE_ONLY.+ decision = evaluate_engine(+ "collection_xyz",+ ctx,+ settings_snapshot={"policy.egress_scope": {"value": "private_only"}},+ )+ assert decision.allowed is True+++def test_context_from_snapshot_reads_nested_value_dicts():+ snapshot = {+ "policy.egress_scope": {"value": "strict"},+ "llm.require_local_endpoint": {"value": True},+ }+ ctx = context_from_snapshot(snapshot, primary_engine="arxiv")+ assert ctx.scope == EgressScope.STRICT+ assert ctx.require_local_llm is True+++def test_context_rejects_strict_with_meta_picker():+ snapshot = {"policy.egress_scope": "strict"}+ with pytest.raises(ValueError):+ context_from_snapshot(snapshot, primary_engine="auto")+++def test_context_unknown_scope_raises_policy_denied():+ # N8 (Round 6): unknown scope is fail-closed rather than silently+ # falling back to BOTH. Silent fallback would mask config corruption+ # and effectively disable the policy whenever the saved value was+ # tampered with or migrated incorrectly.+ with pytest.raises(PolicyDeniedError) as excinfo:+ context_from_snapshot(+ {"policy.egress_scope": "nonsense"}, primary_engine="arxiv"+ )+ assert excinfo.value.decision.reason == "unknown_egress_scope"+++def test_context_string_false_coerces_correctly():+ # Type-confusion guard: the string "false" must not be truthy.+ ctx = context_from_snapshot(+ {"llm.require_local_endpoint": "false"}, primary_engine="arxiv"+ )+ assert ctx.require_local_llm is False+++def test_context_string_true_coerces_to_true():+ ctx = context_from_snapshot(+ {"llm.require_local_endpoint": "true"}, primary_engine="arxiv"+ )+ assert ctx.require_local_llm is True+++# ---------------------------------------------------------------------------+# evaluate_engine — STRICT semantics+# ---------------------------------------------------------------------------+++def test_evaluate_engine_strict_primary_match():+ ctx = make_ctx(scope=EgressScope.STRICT, primary="arxiv")+ decision = evaluate_engine("arxiv", ctx, settings_snapshot={})+ assert decision.allowed+ assert decision.reason == "allowed"+++def test_evaluate_engine_strict_non_primary_denied():+ ctx = make_ctx(scope=EgressScope.STRICT, primary="arxiv")+ decision = evaluate_engine("pubmed", ctx, settings_snapshot={})+ assert not decision.allowed+ assert decision.reason == "strict_not_primary"+++def test_evaluate_engine_strict_with_meta_picker_engine():+ # auto/meta/parallel can never be permitted under STRICT.+ ctx = make_ctx(scope=EgressScope.STRICT, primary="arxiv")+ decision = evaluate_engine("auto", ctx, settings_snapshot={})+ assert not decision.allowed+++# ---------------------------------------------------------------------------+# evaluate_engine — public/private bucket+# ---------------------------------------------------------------------------+++def test_evaluate_engine_public_only_blocks_local_engine():+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ # paperless is is_local=True with url_setting; with no URL in snapshot+ # the static is_local flag applies, so PUBLIC_ONLY rejects it.+ decision = evaluate_engine("paperless", ctx, settings_snapshot={})+ assert not decision.allowed+ assert decision.reason == "scope_mismatch_public_only"+++def test_evaluate_engine_private_only_blocks_public_engine():+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ decision = evaluate_engine("arxiv", ctx, settings_snapshot={})+ assert not decision.allowed+ assert decision.reason == "scope_mismatch_private_only"+++def test_evaluate_engine_both_allows_public():+ ctx = make_ctx(scope=EgressScope.BOTH)+ decision = evaluate_engine("arxiv", ctx, settings_snapshot={})+ assert decision.allowed+++def test_evaluate_engine_both_allows_private():+ ctx = make_ctx(scope=EgressScope.BOTH)+ decision = evaluate_engine("paperless", ctx, settings_snapshot={})+ assert decision.allowed+++def test_evaluate_engine_no_snapshot_fails_closed():+ ctx = make_ctx()+ decision = evaluate_engine("arxiv", ctx, settings_snapshot=None)+ assert not decision.allowed+ assert decision.reason == "no_snapshot"+++def test_evaluate_engine_unknown_name_fails_closed():+ ctx = make_ctx()+ decision = evaluate_engine(+ "totally_made_up_engine", ctx, settings_snapshot={}+ )+ assert not decision.allowed+ assert decision.reason == "engine_unknown"+++# ---------------------------------------------------------------------------+# evaluate_engine — newly-classified engines+# ---------------------------------------------------------------------------+++def test_evaluate_engine_github_is_public():+ """Regression: github engine must be explicitly classified is_public=True."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ decision = evaluate_engine("github", ctx, settings_snapshot={})+ assert decision.allowed+++def test_evaluate_engine_paperless_is_local():+ """Regression: paperless engine must be classified is_local=True."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ # Without a url_setting value, the static is_local flag applies.+ decision = evaluate_engine("paperless", ctx, settings_snapshot={})+ assert decision.allowed+++# ---------------------------------------------------------------------------+# evaluate_engine — dynamic URL classification+# ---------------------------------------------------------------------------+++def test_evaluate_engine_searxng_localhost_classified_local():+ """SearXNG pointed at localhost should pass PRIVATE_ONLY."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ snapshot = {+ "search.engine.web.searxng.default_params.instance_url": "http://localhost:8080"+ }+ decision = evaluate_engine("searxng", ctx, settings_snapshot=snapshot)+ assert decision.allowed+++def test_evaluate_engine_searxng_remote_classified_public():+ """SearXNG pointed at a public host should fail PRIVATE_ONLY."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ snapshot = {+ "search.engine.web.searxng.default_params.instance_url": "https://searx.example.com"+ }+ # Patch DNS so the public hostname doesn't actually resolve over the network+ # during tests.+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_engine("searxng", ctx, settings_snapshot=snapshot)+ assert not decision.allowed+++# ---------------------------------------------------------------------------+# evaluate_llm_endpoint+# ---------------------------------------------------------------------------+++def test_evaluate_llm_endpoint_no_local_requirement_allows_cloud():+ ctx = make_ctx(require_local_llm=False)+ decision = evaluate_llm_endpoint("openai", ctx, settings_snapshot={})+ assert decision.allowed+++def test_evaluate_llm_endpoint_require_local_blocks_cloud_providers():+ ctx = make_ctx(require_local_llm=True)+ for provider in ("openai", "anthropic", "google", "openrouter"):+ decision = evaluate_llm_endpoint(provider, ctx, settings_snapshot={})+ assert not decision.allowed, (+ f"{provider} should be blocked under require_local_llm"+ )+++def test_evaluate_llm_endpoint_local_ollama_allowed():+ ctx = make_ctx(require_local_llm=True)+ decision = evaluate_llm_endpoint("ollama", ctx, settings_snapshot={})+ # No URL override → assumes localhost default.+ assert decision.allowed+++def test_evaluate_llm_endpoint_user_registered_llm_allowed():+ """A user-registered in-process LLM (programmatic API ``llms={...}``)+ must be allowed under require_local_llm — it has no endpoint to+ classify and the audit hook backstops stray sockets. Regression test+ for the mock-LLM example breaking after ADAPTIVE retriever-primary+ runs began resolving to PRIVATE_ONLY."""+ from local_deep_research.llm.llm_registry import (+ register_llm,+ unregister_llm,+ )++ ctx = make_ctx(require_local_llm=True)+ register_llm("egress_test_mock_llm", lambda **kwargs: None)+ try:+ decision = evaluate_llm_endpoint(+ "egress_test_mock_llm", ctx, settings_snapshot={}+ )+ assert decision.allowed+ assert decision.reason == "user_registered_llm"+ finally:+ unregister_llm("egress_test_mock_llm")+++def test_evaluate_llm_endpoint_registered_name_shadowing_cloud_still_blocked():+ """Registering a custom LLM under a built-in cloud name must NOT+ bypass the cloud-provider gate."""+ from local_deep_research.llm.llm_registry import (+ get_llm_from_registry,+ register_llm,+ unregister_llm,+ )++ ctx = make_ctx(require_local_llm=True)+ original = get_llm_from_registry("openai")+ register_llm("openai", lambda **kwargs: None)+ try:+ decision = evaluate_llm_endpoint("openai", ctx, settings_snapshot={})+ assert not decision.allowed+ assert decision.reason == "provider_cloud_only"+ finally:+ # Restore the auto-registered built-in entry rather than leaving+ # the registry polluted for later tests in this process.+ if original is not None:+ register_llm("openai", original)+ else:+ unregister_llm("openai")+++def test_evaluate_llm_endpoint_unregistered_unknown_provider_still_blocked():+ """An unknown provider that is NOT in the registry keeps failing+ closed with provider_url_unset."""+ ctx = make_ctx(require_local_llm=True)+ decision = evaluate_llm_endpoint(+ "totally_unknown_provider", ctx, settings_snapshot={}+ )+ assert not decision.allowed+ assert decision.reason == "provider_url_unset"+++def test_evaluate_llm_endpoint_ollama_pointed_remote_blocked():+ ctx = make_ctx(require_local_llm=True)+ snapshot = {"llm.ollama.url": "https://remote-ollama.example.com"}+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_llm_endpoint(+ "ollama", ctx, settings_snapshot=snapshot+ )+ assert not decision.allowed+++# ---------------------------------------------------------------------------+# evaluate_embeddings+# ---------------------------------------------------------------------------+++def test_evaluate_embeddings_no_requirement_allows_openai():+ ctx = make_ctx(require_local_embeddings=False)+ decision = evaluate_embeddings("openai", ctx, settings_snapshot={})+ assert decision.allowed+++def test_evaluate_embeddings_require_local_allows_sentence_transformers():+ ctx = make_ctx(require_local_embeddings=True)+ decision = evaluate_embeddings(+ "sentence_transformers", ctx, settings_snapshot={}+ )+ assert decision.allowed+++def test_evaluate_embeddings_require_local_blocks_openai_cloud():+ ctx = make_ctx(require_local_embeddings=True)+ decision = evaluate_embeddings("openai", ctx, settings_snapshot={})+ assert not decision.allowed+++def test_evaluate_embeddings_openai_with_local_base_url_allowed():+ ctx = make_ctx(require_local_embeddings=True)+ snapshot = {"embeddings.openai.base_url": "http://localhost:1234/v1"}+ decision = evaluate_embeddings("openai", ctx, settings_snapshot=snapshot)+ assert decision.allowed+++# ---------------------------------------------------------------------------+# evaluate_url+# ---------------------------------------------------------------------------+++def test_evaluate_url_rejects_dangerous_scheme():+ ctx = make_ctx()+ assert not evaluate_url("javascript:alert(1)", ctx).allowed+ assert not evaluate_url("data:text/html,<script>", ctx).allowed+++def test_evaluate_url_rejects_malformed():+ ctx = make_ctx()+ assert not evaluate_url("", ctx).allowed+ assert not evaluate_url("not-a-url", ctx).allowed+++def test_evaluate_url_strict_allows_private_host():+ ctx = make_ctx(scope=EgressScope.STRICT)+ assert evaluate_url("http://127.0.0.1:8080/", ctx).allowed+++def test_evaluate_url_strict_blocks_public_host():+ ctx = make_ctx(scope=EgressScope.STRICT)+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_url("https://example.com/", ctx)+ assert not decision.allowed+++def test_evaluate_url_private_only_blocks_public():+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_url("https://example.com/", ctx)+ assert not decision.allowed+++def test_evaluate_url_denial_quota_exhaustion():+ """After MAX_DENIED_FETCHES_PER_RUN denials, even a legal URL fails closed.++ This blocks a malicious indexed document from looping the agent through+ hundreds of denied fetches and inflating the audit log.+ """+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ # Pre-fill the counter to the threshold.+ ctx._fetch_denial_count["count"] = MAX_DENIED_FETCHES_PER_RUN+ decision = evaluate_url("http://127.0.0.1/", ctx)+ assert not decision.allowed+ assert decision.reason == "denial_quota_exceeded"+++# ---------------------------------------------------------------------------+# EgressContext: frozen + mutable internals+# ---------------------------------------------------------------------------+++def test_egress_context_is_frozen_for_user_fields():+ ctx = make_ctx()+ with pytest.raises(Exception):+ ctx.scope = EgressScope.STRICT # type: ignore[misc]+++def test_egress_context_mutable_internals_work():+ """The DNS cache and denial counter live inside dict wrappers so they+ can mutate even on a frozen=True dataclass. This is the R9-01 pattern.+ """+ ctx = make_ctx()+ ctx._dns_cache["example.com"] = False+ assert ctx._dns_cache["example.com"] is False+ ctx._fetch_denial_count["count"] += 1+ assert ctx._fetch_denial_count["count"] == 1+++# ---------------------------------------------------------------------------+# PolicyDeniedError+# ---------------------------------------------------------------------------+++def test_policy_denied_error_carries_decision_and_target():+ err = PolicyDeniedError(+ decision=Decision(allowed=False, reason="strict_not_primary"),+ target="pubmed",+ )+ assert err.decision.reason == "strict_not_primary"+ assert err.target == "pubmed"+ assert "policy_denied" in str(err)+++# ---------------------------------------------------------------------------+# Factory PEP integration (Stage 1a's end-to-end check)+# ---------------------------------------------------------------------------+++def test_factory_raises_policy_denied_under_strict_non_primary():+ """End-to-end: factory gate blocks an engine that doesn't match the+ user's STRICT primary. This is the original LangGraph-silent-expansion+ bug closed at its source.+ """+ from local_deep_research.web_search_engines.search_engine_factory import (+ create_search_engine,+ )++ # User picks arxiv as primary, scope=STRICT. The factory must reject any+ # attempt to instantiate pubmed.+ snapshot = {+ "policy.egress_scope": "strict",+ "search.tool": "arxiv",+ # Minimal search_engines_config entry so the engine resolves.+ "search.engine.web.pubmed.use_in_auto_search": True,+ }+ with pytest.raises(PolicyDeniedError) as exc_info:+ create_search_engine("pubmed", llm=None, settings_snapshot=snapshot)+ assert exc_info.value.decision.reason == "strict_not_primary"+++def test_get_llm_blocks_cloud_under_require_local():+ """End-to-end: get_llm() must raise PolicyDeniedError when the user+ has require_local_endpoint=true and asks for a cloud provider.+ """+ from local_deep_research.config.llm_config import get_llm++ snapshot = {+ "llm.require_local_endpoint": True,+ "llm.provider": "openai",+ "llm.model": "gpt-4o",+ "search.tool": "arxiv",+ # API key needed so we get past openai-base's own validation —+ # the policy PEP fires before that anyway.+ "llm.openai.api_key": "sk-test-not-real",+ }+ with pytest.raises(PolicyDeniedError) as exc_info:+ get_llm(+ provider="openai", model_name="gpt-4o", settings_snapshot=snapshot+ )+ assert exc_info.value.target == "openai"+ assert "cloud" in exc_info.value.decision.reason+++def test_get_llm_allows_cloud_without_require_local():+ """Without require_local_endpoint, cloud providers are permitted+ (policy is a separate orthogonal control — the default is permissive).+ """+ from local_deep_research.config.llm_config import get_llm++ snapshot = {+ "llm.require_local_endpoint": False,+ "llm.provider": "openai",+ "llm.model": "gpt-4o",+ "search.tool": "arxiv",+ "llm.openai.api_key": "sk-test-not-real",+ }+ # Don't assert on successful return — OpenAI client construction+ # might fail for unrelated reasons. We only care that PolicyDeniedError+ # is NOT raised by our gate.+ try:+ get_llm(+ provider="openai",+ model_name="gpt-4o",+ settings_snapshot=snapshot,+ )+ except PolicyDeniedError:+ pytest.fail(+ "policy should permit cloud LLM when require_local_endpoint=False"+ )+ except Exception:+ pass # unrelated failure is fine+++ "provider",+ sorted(+ __import__(+ "local_deep_research.security.egress.policy",+ fromlist=["_CLOUD_LLM_PROVIDERS"],+ )._CLOUD_LLM_PROVIDERS+ )+ + ["openai_endpoint", "groq", "mistral", "cohere"],+)+def test_get_llm_no_snapshot_blocks_non_local_providers(provider):+ """Snapshot-less callers must not silently instantiate any provider+ outside the known-local allow-list. Covers all enumerated cloud+ providers PLUS ambiguous (openai_endpoint) and hypothetical+ future ones (groq/mistral/cohere) — the allow-list approach is+ deliberately tight so a new cloud provider is refused until it's+ explicitly classified.+ """+ from local_deep_research.config.llm_config import get_llm++ with pytest.raises(PolicyDeniedError) as exc_info:+ get_llm(provider=provider, model_name="x")+ assert exc_info.value.target == provider+ assert exc_info.value.decision.reason == "no_snapshot_for_provider"++[email protected]("provider", ["ollama", "lmstudio", "llamacpp"])+def test_get_llm_no_snapshot_allows_local_providers(provider):+ """The snapshot-less gate is an allow-list. Known-local providers+ pass through; downstream failures are unrelated to policy.+ """+ from local_deep_research.config.llm_config import get_llm++ try:+ get_llm(provider=provider, model_name="x")+ except PolicyDeniedError as exc:+ pytest.fail(+ f"no-snapshot policy should permit {provider} (got "+ f"{exc.decision.reason})"+ )+ except Exception:+ # Non-policy failures (missing model config, no reachable+ # endpoint, etc.) are out of scope for this test.+ pass+++def test_get_llm_no_snapshot_no_provider_is_noop():+ """When neither snapshot nor provider is set, the gate does nothing+ (preserves legacy "look up provider from settings" path).+ """+ from local_deep_research.config.llm_config import get_llm++ try:+ get_llm(provider=None, model_name=None)+ except PolicyDeniedError:+ pytest.fail("gate must not fire when no provider is given")+ except Exception:+ pass # downstream config errors are fine+++# ---------------------------------------------------------------------------+# Warning banners+# ---------------------------------------------------------------------------+++def test_warning_public_egress_fires_when_scope_is_both():+ from local_deep_research.security.egress.warnings import (+ check_public_egress_enabled,+ )++ warning = check_public_egress_enabled("both", acknowledged=False)+ assert warning is not None+ assert warning["type"] == "public_egress_enabled"+++def test_warning_public_egress_suppressed_under_private_only():+ from local_deep_research.security.egress.warnings import (+ check_public_egress_enabled,+ )++ assert (+ check_public_egress_enabled("private_only", acknowledged=False) is None+ )+++def test_warning_public_egress_suppressed_when_acknowledged():+ """The fresh-install ack flag silences the banner triplet."""+ from local_deep_research.security.egress.warnings import (+ check_public_egress_enabled,+ )++ assert check_public_egress_enabled("both", acknowledged=True) is None+++def test_warning_cloud_llm_fires_for_openai_without_require_local():+ from local_deep_research.security.egress.warnings import (+ check_cloud_llm_enabled,+ )++ warning = check_cloud_llm_enabled(+ "openai", require_local_endpoint=False, acknowledged=False+ )+ assert warning is not None+ assert warning["type"] == "cloud_llm_enabled"+++def test_warning_cloud_llm_suppressed_for_local_provider():+ from local_deep_research.security.egress.warnings import (+ check_cloud_llm_enabled,+ )++ # Ollama is local-default — no warning even without the toggle.+ assert (+ check_cloud_llm_enabled(+ "ollama", require_local_endpoint=False, acknowledged=False+ )+ is None+ )+++def test_warning_cloud_llm_suppressed_when_require_local_set():+ from local_deep_research.security.egress.warnings import (+ check_cloud_llm_enabled,+ )++ assert (+ check_cloud_llm_enabled(+ "openai", require_local_endpoint=True, acknowledged=False+ )+ is None+ )+++def test_warning_cloud_embeddings_fires_for_openai_cloud():+ from local_deep_research.security.egress.warnings import (+ check_cloud_embeddings_enabled,+ )++ warning = check_cloud_embeddings_enabled(+ embeddings_provider="openai",+ embeddings_base_url="",+ require_local_embeddings=False,+ acknowledged=False,+ )+ assert warning is not None+ assert warning["type"] == "cloud_embeddings_enabled"+++def test_warning_cloud_embeddings_suppressed_for_local_base_url():+ """OpenAI provider type pointed at a local endpoint (LM Studio, vLLM)+ should not trigger the cloud-embeddings warning.+ """+ from local_deep_research.security.egress.warnings import (+ check_cloud_embeddings_enabled,+ )++ assert (+ check_cloud_embeddings_enabled(+ embeddings_provider="openai",+ embeddings_base_url="http://localhost:1234/v1",+ require_local_embeddings=False,+ acknowledged=False,+ )+ is None+ )+++def test_warning_cloud_embeddings_suppressed_for_sentence_transformers():+ from local_deep_research.security.egress.warnings import (+ check_cloud_embeddings_enabled,+ )++ assert (+ check_cloud_embeddings_enabled(+ embeddings_provider="sentence_transformers",+ embeddings_base_url="",+ require_local_embeddings=False,+ acknowledged=False,+ )+ is None+ )+++# ---------------------------------------------------------------------------+# Embeddings pre-flight+# ---------------------------------------------------------------------------+++def test_rag_factory_blocks_openai_embeddings_under_require_local():+ """End-to-end: get_rag_service() must fail pre-flight when the user+ has embeddings.require_local=True and the configured embedding+ provider is OpenAI (cloud). This closes the silent corpus-upload+ leak the audit identified as the highest-impact embeddings vector.+ """+ from unittest.mock import MagicMock+ from local_deep_research.research_library.services import (+ rag_service_factory,+ )++ # Fake SettingsManager that returns openai as the provider + require_local=True.+ fake_settings = MagicMock()+ fake_settings.get_setting.side_effect = lambda key, *args, **kwargs: {+ "embeddings.require_local": True,+ "embeddings.openai.base_url": "", # cloud endpoint+ "embeddings.ollama.url": "",+ }.get(key)++ with pytest.raises(PolicyDeniedError) as exc_info:+ rag_service_factory._enforce_embeddings_policy(+ "openai", fake_settings, username="alice"+ )+ assert exc_info.value.target == "openai"+ assert exc_info.value.decision.reason == "provider_cloud"+++def test_rag_factory_allows_sentence_transformers_under_require_local():+ """sentence_transformers is always local; the pre-flight passes."""+ from unittest.mock import MagicMock+ from local_deep_research.research_library.services import (+ rag_service_factory,+ )++ fake_settings = MagicMock()+ fake_settings.get_setting.side_effect = lambda key, *args, **kwargs: {+ "embeddings.require_local": True,+ "embeddings.openai.base_url": "",+ "embeddings.ollama.url": "",+ }.get(key)++ # Should not raise.+ rag_service_factory._enforce_embeddings_policy(+ "sentence_transformers", fake_settings, username="alice"+ )+++def test_rag_factory_allows_openai_pointed_at_local_endpoint():+ """OpenAI provider becomes "local" when base_url is loopback/RFC1918."""+ from unittest.mock import MagicMock+ from local_deep_research.research_library.services import (+ rag_service_factory,+ )++ fake_settings = MagicMock()+ fake_settings.get_setting.side_effect = lambda key, *args, **kwargs: {+ "embeddings.require_local": True,+ # OpenAI-compatible local endpoint (e.g. LM Studio).+ "embeddings.openai.base_url": "http://127.0.0.1:1234/v1",+ "embeddings.ollama.url": "",+ }.get(key)++ # Should not raise.+ rag_service_factory._enforce_embeddings_policy(+ "openai", fake_settings, username="alice"+ )+++def test_rag_factory_passes_through_when_require_local_false():+ """When the user hasn't opted in, OpenAI embeddings are permitted."""+ from unittest.mock import MagicMock+ from local_deep_research.research_library.services import (+ rag_service_factory,+ )++ fake_settings = MagicMock()+ fake_settings.get_setting.side_effect = lambda key, *args, **kwargs: {+ "embeddings.require_local": False,+ "embeddings.openai.base_url": "",+ "embeddings.ollama.url": "",+ }.get(key)++ # Should not raise — policy is opt-in.+ rag_service_factory._enforce_embeddings_policy(+ "openai", fake_settings, username="alice"+ )+++def test_factory_allows_primary_under_strict():+ """Factory permits the user's primary engine under STRICT."""+ from local_deep_research.web_search_engines.search_engine_factory import (+ create_search_engine,+ )++ snapshot = {+ "policy.egress_scope": "strict",+ "search.tool": "arxiv",+ }+ # We're not asserting on the returned engine (it would require full+ # arxiv setup), only that the policy gate doesn't raise.+ try:+ create_search_engine("arxiv", llm=None, settings_snapshot=snapshot)+ except PolicyDeniedError:+ pytest.fail("policy should allow the primary engine under STRICT")+ except Exception:+ # Engine instantiation may fail for unrelated reasons (e.g. config);+ # we only care that PolicyDeniedError was not raised.+ pass+++# ---------------------------------------------------------------------------+# Round 6 fix-coverage regression tests (T1-T8 + new findings)+# ---------------------------------------------------------------------------+++def test_t1_strict_blocks_public_engine_no_http_called():+ """T1: under STRICT + primary=arxiv, an attempt to construct a public+ engine (wikipedia) is refused and no HTTP socket fires.++ Catches a regression where the factory PEP was bypassed and an LLM-+ generated tool call could silently reach the network.+ """+ from local_deep_research.web_search_engines.search_engine_factory import (+ create_search_engine,+ )++ snapshot = {+ "policy.egress_scope": "strict",+ "search.tool": "arxiv",+ }+ with (+ patch("socket.getaddrinfo") as mock_getaddr,+ patch("socket.socket") as mock_socket,+ ):+ try:+ create_search_engine(+ "wikipedia", llm=None, settings_snapshot=snapshot+ )+ except PolicyDeniedError:+ pass+ except Exception:+ pass+ mock_getaddr.assert_not_called()+ mock_socket.assert_not_called()+++def test_t2_list_typed_url_setting_any_public_wins():+ """T2: list-typed url_setting (e.g. Elasticsearch hosts) must+ classify as PUBLIC if any entry classifies as public — the safer+ fail-up direction. Without this an attacker could hide a public+ host inside a list of local hosts and bypass STRICT.+ """+ from local_deep_research.security.egress.policy import (+ _classify_engine_url,+ )++ ctx = make_ctx(scope=EgressScope.BOTH)+ snapshot = {+ "elastic.hosts": [+ "http://192.168.1.5:9200", # local+ "http://elastic-cloud.example.com:9200", # public+ ]+ }+ with patch(+ "local_deep_research.security.egress.policy._classify_host"+ ) as mock_cls:+ mock_cls.side_effect = lambda host, ctx, allow_dns=True: (+ host.startswith("192.")+ )+ result = _classify_engine_url("elastic.hosts", snapshot, ctx)+ # any public present => return False (public)+ assert result is False+++def test_t3_classify_host_uses_dns_when_not_an_ip():+ """T3: exercise the REAL _classify_host (not patched out) with+ mocked socket.getaddrinfo. Catches regressions in the DNS code+ path that mock-heavy tests would miss.+ """+ from local_deep_research.security.egress.policy import _classify_host++ ctx = make_ctx()+ fake_addrinfo = [+ (None, None, None, None, ("192.168.1.42", 0)),+ ]+ with patch(+ "local_deep_research.security.egress.policy._resolve_with_timeout",+ return_value=fake_addrinfo,+ ):+ assert _classify_host("example.lab", ctx) is True+++def test_t4_ipv6_literal_url_classifies():+ """T4: IPv6-literal URLs (with square brackets) classify+ correctly. Catches regressions in the bracket-strip logic.+ """+ from local_deep_research.security.egress.policy import _classify_host++ ctx = make_ctx()+ assert _classify_host("[::1]", ctx) is True+++def test_t5_dns_cache_avoids_repeat_lookups():+ """T5: A second classification of the same hostname uses the+ cache — DO NOT call getaddrinfo twice.+ """+ from local_deep_research.security.egress.policy import _classify_host++ ctx = make_ctx()+ with patch(+ "local_deep_research.security.egress.policy._resolve_with_timeout",+ return_value=[(None, None, None, None, ("8.8.8.8", 0))],+ ) as mock_resolve:+ _classify_host("dns.example.com", ctx)+ _classify_host("dns.example.com", ctx)+ assert mock_resolve.call_count == 1+++def test_t7_dns_timeout_fail_safe_treats_as_public():+ """T7: when DNS lookup times out, _classify_host falls back to+ public (False) so PRIVATE_ONLY doesn't accidentally allow an+ unreachable host that might resolve to anything later.+ """+ from local_deep_research.security.egress.policy import _classify_host++ ctx = make_ctx()+ with patch(+ "local_deep_research.security.egress.policy._resolve_with_timeout",+ return_value=None,+ ):+ assert _classify_host("timed-out.example.com", ctx) is False+++def test_t7b_resolve_with_timeout_actually_bounds_a_hung_lookup():+ """Regression: a hung getaddrinfo must NOT block past the timeout.++ The earlier implementation drove the lookup from a+ ``with ThreadPoolExecutor(...)`` block; the context manager's __exit__+ calls shutdown(wait=True), which re-blocked on the abandoned worker and+ defeated the timeout entirely (the call returned only after the OS DNS+ timeout). This asserts the bound actually holds.+ """+ import time++ from local_deep_research.security.egress import policy as egress_policy++ # Sleep kept modest (1.2s): the orphaned worker is joined by the+ # concurrent.futures atexit handler at interpreter shutdown, so a large+ # sleep would slow pytest teardown. 1.2s is still 6x the 0.2s timeout —+ # enough to distinguish "bounded" from "waited out the worker".+ def _hang(*_a, **_k):+ # Simulates a hung getaddrinfo in an abandoned worker; the test+ # itself returns in ~0.2s (the timeout), so it is not slow and must+ # not be skipped under -m 'not slow'.+ time.sleep(1.2) # allow: unmarked-sleep - mocked hang, test is fast+ return [(2, 1, 6, "", ("127.0.0.1", 0))]++ with (+ patch.object(egress_policy, "_DNS_TIMEOUT_SEC", 0.2),+ patch("socket.getaddrinfo", side_effect=_hang),+ ):+ start = time.monotonic()+ result = egress_policy._resolve_with_timeout("hung.example.com")+ elapsed = time.monotonic() - start++ assert result is None+ # Ceiling = timeout 0.2s + scheduling slack, well below the 1.2s hang —+ # proves the timeout is effective rather than waiting out the worker.+ assert elapsed < 0.9, f"resolve did not bound the hang: {elapsed:.2f}s"+++ "scope",+ [+ EgressScope.STRICT,+ EgressScope.PRIVATE_ONLY,+ EgressScope.BOTH,+ EgressScope.PUBLIC_ONLY,+ ],+)+ "url",+ [+ "http://169.254.169.254/latest/meta-data/", # AWS/GCE IMDS (IPv4)+ "http://[::ffff:169.254.169.254]/", # IPv4-mapped IPv6 form+ "http://169.254.170.2/", # AWS ECS task metadata+ ],+)+def test_cloud_metadata_ip_blocked_under_every_scope(scope, url):+ """Cloud-metadata endpoints classify as link-local/private, so STRICT and+ PRIVATE_ONLY would otherwise ALLOW them — a credential-theft path,+ especially via the audit-hook net which calls evaluate_url directly on+ raw socket.connect targets (bypassing the SSRF validator the explicit+ fetch PEPs run first). They must be refused regardless of scope.+ """+ ctx = make_ctx(scope=scope, primary="arxiv")+ decision = evaluate_url(url, ctx)+ assert decision.allowed is False+ assert decision.reason == "blocked_metadata_ip"+++def test_hostname_resolving_to_metadata_ip_is_refused_under_private_scopes():+ """A hostname whose DNS A-record points at a cloud-metadata IP+ (169.254.169.254) must NOT classify as local — link-local metadata IPs+ pass is_private_ip, which would let STRICT/PRIVATE_ONLY fetch IMDS via a+ hostname. _classify_host now rejects resolved metadata IPs.+ """+ for scope in (EgressScope.STRICT, EgressScope.PRIVATE_ONLY):+ ctx = make_ctx(scope=scope, primary="library")+ with patch(+ "socket.getaddrinfo",+ return_value=[(2, 1, 6, "", ("169.254.169.254", 0))],+ ):+ decision = evaluate_url(+ "http://imds.attacker.example/latest/meta-data/", ctx+ )+ assert decision.allowed is False, f"leaked under {scope}"+++def test_private_host_still_allowed_under_strict_and_private():+ """The metadata guard must NOT over-block legitimate private hosts+ (local Ollama/SearXNG) under STRICT / PRIVATE_ONLY.+ """+ for scope in (EgressScope.STRICT, EgressScope.PRIVATE_ONLY):+ ctx = make_ctx(scope=scope, primary="arxiv")+ decision = evaluate_url("http://127.0.0.1:11434/", ctx)+ assert decision.allowed is True+++def test_t8_denial_quota_off_by_one():+ """T8: exactly MAX_DENIED_FETCHES_PER_RUN denials are processed+ individually; the next call hits the quota cap.+ """+ ctx = make_ctx(scope=EgressScope.STRICT, primary="arxiv")+ # Force every URL to be denied by using PUBLIC URLs under STRICT.+ for _ in range(MAX_DENIED_FETCHES_PER_RUN):+ decision = evaluate_url("https://public-host.example.com/", ctx)+ assert decision.allowed is False+ assert decision.reason != "denial_quota_exceeded"+ # The next denial hits the quota.+ decision = evaluate_url("https://public-host.example.com/", ctx)+ assert decision.reason == "denial_quota_exceeded"+++def test_nat64_metadata_classifies_as_public():+ """H2: NAT64-wrapped cloud-metadata IPs are not silently+ classified as private. Catches the original H2 regression.+ """+ from local_deep_research.security.egress.policy import _classify_host++ ctx = make_ctx()+ # 64:ff9b::169.254.169.254 — NAT64 wrapping the AWS metadata IP.+ result = _classify_host("64:ff9b::a9fe:a9fe", ctx)+ assert result is False, "NAT64 metadata IP must classify as public"+++def test_h3_nested_dict_url_setting_unwraps():+ """H3: production snapshots wrap each setting in+ {"value": X, "type": ..., ...}. The PDP MUST unwrap before+ parsing the URL, or it sees a dict and silently skips+ classification.+ """+ from local_deep_research.security.egress.policy import (+ _classify_engine_url,+ )++ ctx = make_ctx()+ snapshot = {+ "search_url": {"value": "http://192.168.10.10/search", "type": "str"}+ }+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=True,+ ):+ result = _classify_engine_url("search_url", snapshot, ctx)+ assert result is True, "nested-dict url_setting must unwrap"+++def test_c2_policy_aware_validate_url_private_only_allows_private():+ """C2: under PRIVATE_ONLY policy, the SSRF wrapper permits private+ IPs so local lab deployments work without SSRF_ALLOW_PRIVATE_IPS=1.+ """+ from local_deep_research.security.egress.fetch import (+ policy_aware_validate_url,+ )++ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ assert policy_aware_validate_url("http://127.0.0.1:11434", ctx) is True+ assert policy_aware_validate_url("http://192.168.1.5/api", ctx) is True+++def test_c2_policy_aware_validate_url_metadata_still_blocked():+ """C2: even under PRIVATE_ONLY, cloud-metadata IPs are blocked+ — they're in ALWAYS_BLOCKED regardless of the scope flag.+ """+ from local_deep_research.security.egress.fetch import (+ policy_aware_validate_url,+ )++ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ assert policy_aware_validate_url("http://169.254.169.254/", ctx) is False+++def test_n1_retriever_registry_get_metadata():+ """retriever_registry.get_metadata returns the classification dict+ for a registered retriever (default {"is_local": True}), None for+ unknown.+ """+ from unittest.mock import MagicMock++ from local_deep_research.web_search_engines.retriever_registry import (+ retriever_registry,+ )++ name = "_test_retriever_n1"+ retriever_registry.register(name, MagicMock())+ try:+ assert retriever_registry.get_metadata(name) == {"is_local": True}+ assert retriever_registry.get_metadata("does-not-exist") is None+ finally:+ retriever_registry.unregister(name)+++def test_classified_retriever_is_local_flag():+ """register(is_local=...) records classification; register_multiple+ applies it to all; unregister/clear drop the metadata."""+ from unittest.mock import MagicMock++ from local_deep_research.web_search_engines.retriever_registry import (+ RetrieverRegistry,+ )++ reg = RetrieverRegistry()+ reg.register("local_kb", MagicMock(), is_local=True)+ reg.register("public_idx", MagicMock(), is_local=False)+ assert reg.get_metadata("local_kb") == {"is_local": True}+ assert reg.get_metadata("public_idx") == {"is_local": False}++ reg.register_multiple({"a": MagicMock(), "b": MagicMock()}, is_local=False)+ assert reg.get_metadata("a") == {"is_local": False}+ assert reg.get_metadata("b") == {"is_local": False}++ reg.unregister("a")+ assert reg.get_metadata("a") is None+ reg.clear()+ assert reg.get_metadata("local_kb") is None+++def test_evaluate_retriever_scopes():+ """evaluate_retriever honors scope: local retriever allowed under+ BOTH/PRIVATE_ONLY, denied under PUBLIC_ONLY, and under STRICT only+ when it is the primary engine."""+ from unittest.mock import MagicMock++ from local_deep_research.security.egress.policy import evaluate_retriever+ from local_deep_research.web_search_engines.retriever_registry import (+ retriever_registry,+ )++ name = "_test_local_kb"+ retriever_registry.register(name, MagicMock(), is_local=True)+ try:+ # BOTH → allowed+ assert evaluate_retriever(+ name, make_ctx(scope=EgressScope.BOTH)+ ).allowed+ # PRIVATE_ONLY → allowed (it's local)+ assert evaluate_retriever(+ name, make_ctx(scope=EgressScope.PRIVATE_ONLY)+ ).allowed+ # PUBLIC_ONLY → denied (it's local)+ d = evaluate_retriever(name, make_ctx(scope=EgressScope.PUBLIC_ONLY))+ assert not d.allowed+ assert d.reason == "scope_mismatch_public_only"+ # STRICT + retriever is the primary → allowed+ assert evaluate_retriever(+ name, make_ctx(scope=EgressScope.STRICT, primary=name)+ ).allowed+ # STRICT + retriever is NOT the primary → denied+ d = evaluate_retriever(+ name, make_ctx(scope=EgressScope.STRICT, primary="arxiv")+ )+ assert not d.allowed+ assert d.reason == "strict_not_primary"+ # Unknown retriever → None metadata → retriever_unknown+ d = evaluate_retriever("nope", make_ctx(scope=EgressScope.BOTH))+ assert not d.allowed+ finally:+ retriever_registry.unregister(name)+++def test_n17_cache_hash_includes_scope():+ """N17: SearchCache hash differs between scope=BOTH and+ scope=PRIVATE_ONLY for the same query, preventing cross-scope+ cache-hit bypass.+ """+ from local_deep_research.utilities.search_cache import SearchCache++ with patch(+ "local_deep_research.utilities.search_cache.get_search_context",+ side_effect=[+ {+ "username": "u",+ "settings_snapshot": {"policy.egress_scope": "both"},+ },+ {+ "username": "u",+ "settings_snapshot": {"policy.egress_scope": "private_only"},+ },+ ],+ ):+ cache = SearchCache.__new__(SearchCache)+ h1 = cache._get_query_hash("vega lite syntax", "wikipedia")+ h2 = cache._get_query_hash("vega lite syntax", "wikipedia")+ assert h1 != h2, "scope must participate in the cache hash"+++def test_n8_unknown_scope_raises_after_context_from_snapshot():+ """N8: unknown scope is fail-closed at context build time.+ Verified separately above; this test pins the error reason code.+ """+ with pytest.raises(PolicyDeniedError) as excinfo:+ context_from_snapshot(+ {"policy.egress_scope": "nope_thx"}, primary_engine="arxiv"+ )+ assert excinfo.value.decision.reason == "unknown_egress_scope"+++def test_h1_strict_meta_picker_raises_in_llm_config():+ """H1: STRICT + meta-picker primary engine must raise+ PolicyDeniedError(invalid_policy_config), not silently bypass.+ """+ from local_deep_research.config import llm_config++ snapshot = {+ "policy.egress_scope": "strict",+ "search.tool": "auto",+ "llm.provider": "openai",+ "llm.openai.api_key": "fake",+ }+ with pytest.raises(PolicyDeniedError) as excinfo:+ llm_config.get_llm(provider="openai", settings_snapshot=snapshot)+ assert excinfo.value.decision.reason == "invalid_policy_config"+++def test_c8_banner_suppressed_for_private_base_url():+ """C8: cloud-embeddings banner suppressed when base_url points to+ any private IP (RFC1918, CGNAT, link-local, IPv6 private). The old+ substring-match version missed legitimate private endpoints.+ """+ from local_deep_research.security.egress.warnings import (+ check_cloud_embeddings_enabled,+ )++ # RFC1918 host should NOT fire a banner (was the false-positive case).+ result = check_cloud_embeddings_enabled(+ embeddings_provider="openai",+ embeddings_base_url="http://172.16.0.5/v1",+ require_local_embeddings=False,+ acknowledged=False,+ )+ assert result is None+++def test_c8_banner_fires_for_public_base_url():+ """C8: cloud-embeddings banner fires for genuine cloud base_url."""+ from local_deep_research.security.egress.warnings import (+ check_cloud_embeddings_enabled,+ )++ result = check_cloud_embeddings_enabled(+ embeddings_provider="openai",+ embeddings_base_url="https://api.openai.com/v1",+ require_local_embeddings=False,+ acknowledged=False,+ )+ assert result is not None+ assert result["type"] == "cloud_embeddings_enabled"+++def test_dismiss_key_uses_new_convention():+ """N18: the egress-policy banner's dismissKey matches the+ app.warnings.dismiss_* convention used by every other warning.+ """+ from local_deep_research.security.egress.warnings import (+ check_public_egress_enabled,+ )++ result = check_public_egress_enabled(+ egress_scope="both", acknowledged=False+ )+ assert result is not None+ assert result["dismissKey"] == "app.warnings.dismiss_egress_policy"+++def test_every_llm_provider_is_classified_by_egress_policy():+ """Every auto-discovered LLM provider must be classified by the egress+ policy — cloud (_CLOUD_LLM_PROVIDERS), local-default+ (_LOCAL_DEFAULT_LLM_PROVIDERS), or the URL-classified ``openai_endpoint``.+ A newly-added provider that's in none of these would silently bypass the+ require_local / PRIVATE_ONLY LLM gate, so this test fails loudly until it's+ classified.+ """+ from local_deep_research.llm.providers.auto_discovery import (+ discover_providers,+ )+ from local_deep_research.llm.providers.base import normalize_provider+ from local_deep_research.security.egress.policy import (+ _CLOUD_LLM_PROVIDERS,+ _LOCAL_DEFAULT_LLM_PROVIDERS,+ )++ # openai_endpoint is classified by its configured URL (not a static set).+ classified = (+ _CLOUD_LLM_PROVIDERS+ | _LOCAL_DEFAULT_LLM_PROVIDERS+ | {"openai_endpoint"}+ )+ providers = discover_providers()+ assert providers, "no providers discovered — test wiring issue"+ unclassified = sorted(+ {+ normalize_provider(key)+ for key in providers+ if normalize_provider(key) not in classified+ }+ )+ assert not unclassified, (+ "These LLM providers are not classified by the egress policy and "+ "would bypass the require_local gate — add them to "+ "_CLOUD_LLM_PROVIDERS or _LOCAL_DEFAULT_LLM_PROVIDERS in "+ f"security/egress/policy.py: {unclassified}"+ )+++def test_check_effective_scope_states_adaptive_resolution():+ """The informational banner makes ADAPTIVE's effective posture explicit+ (public / private / both) and only fires for adaptive."""+ from local_deep_research.security.egress.warnings import (+ check_effective_scope,+ )++ pub = check_effective_scope("adaptive", "public_only", "searxng", False)+ assert pub is not None and "Public searches enabled" in pub["title"]+ priv = check_effective_scope(+ "adaptive", "private_only", "collection_x", False+ )+ assert priv is not None and "Private only" in priv["title"]+ both = check_effective_scope("adaptive", "both", "auto", False)+ assert both is not None and "Public + private" in both["title"]++ # Only adaptive — explicit scopes are self-describing in the dropdown.+ assert check_effective_scope("both", "both", "searxng", False) is None+ assert (+ check_effective_scope("private_only", "private_only", "x", False)+ is None+ )+ # Dismissed → suppressed; separate dismiss key from the risk banners.+ assert check_effective_scope("adaptive", "public_only", "x", True) is None+ assert pub["dismissKey"] == "app.warnings.dismiss_adaptive_scope_info"+++def test_egress_banners_have_distinct_dismiss_keys():+ """The three egress banners must use DISTINCT dismiss keys so dismissing+ the fresh-install public-egress notice can't silently suppress the+ critical cloud-LLM / cloud-embeddings warnings (false-safety trap)."""+ from local_deep_research.security.egress.warnings import (+ check_cloud_embeddings_enabled,+ check_cloud_llm_enabled,+ check_public_egress_enabled,+ )++ pub = check_public_egress_enabled(egress_scope="both", acknowledged=False)+ llm = check_cloud_llm_enabled(+ provider="openai", require_local_endpoint=False, acknowledged=False+ )+ emb = check_cloud_embeddings_enabled(+ embeddings_provider="openai",+ embeddings_base_url="",+ require_local_embeddings=False,+ acknowledged=False,+ )+ keys = {pub["dismissKey"], llm["dismissKey"], emb["dismissKey"]}+ assert len(keys) == 3, f"dismiss keys not distinct: {keys}"+ # Dismissing public-egress alone leaves the critical banners firing.+ assert (+ check_cloud_llm_enabled(+ provider="openai",+ require_local_endpoint=False,+ acknowledged=False,+ )+ is not None+ )+++def test_n16_policy_setting_triggers_audit_logger():+ """N16: writing a policy.* key emits a policy_audit log line so+ admins can audit changes. Without it, key changes are silent.+ """+ from local_deep_research.settings.manager import _is_policy_setting++ assert _is_policy_setting("policy.egress_scope") is True+ assert _is_policy_setting("llm.require_local_endpoint") is True+ assert _is_policy_setting("embeddings.require_local") is True+ assert _is_policy_setting("llm.temperature") is False+ assert _is_policy_setting("search.iterations") is False+++def test_n17_research_context_carries_settings_snapshot():+ """Regression for the wiring bug caught during the follow-up pass:+ every site that calls ``set_search_context`` must populate+ ``settings_snapshot`` — without it, ``SearchCache._get_query_hash``+ silently reads {} and degrades to scope='both' regardless of+ policy, re-opening the cache-hit bypass that N17 was meant to+ close. Pins the contract by inspecting the source of each known+ call site so a future regression fails at unit-test time, not at+ runtime.+ """+ import inspect++ from local_deep_research.api import research_functions+ from local_deep_research.web.services import research_service++ for module in (research_functions, research_service):+ src = inspect.getsource(module)+ construct_count = src.count("search_context = {") + src.count(+ "shared_research_context = {"+ )+ snapshot_in_context = src.count('"settings_snapshot":')+ assert snapshot_in_context >= construct_count, (+ f"{module.__name__}: every search_context constructor must "+ f"include 'settings_snapshot' "+ f"(constructions={construct_count}, "+ f"with-snapshot={snapshot_in_context})"+ )+++def test_local_embedding_manager_propagates_policy_denied():+ """LocalEmbeddingManager._initialize_embeddings catches Exception+ broadly and falls back to HuggingFaceEmbeddings. Without an+ explicit PolicyDeniedError re-raise, my N6 SentenceTransformer+ pre-flight gets silently swallowed AND the fallback then tries to+ download from HF — re-opening the very hole N6 was meant to close.+ """+ from local_deep_research.web_search_engines.engines.local_embedding_manager import (+ LocalEmbeddingManager,+ )++ mgr = LocalEmbeddingManager.__new__(LocalEmbeddingManager)+ mgr.embedding_model_type = "sentence_transformers"+ mgr.embedding_model = "all-MiniLM-L6-v2"+ mgr.embedding_device = "cpu"+ mgr.ollama_base_url = None+ mgr.settings_snapshot = {}++ fake_denial = PolicyDeniedError(+ Decision(False, "embeddings_model_not_cached"), target="fake-model"+ )+ with patch(+ "local_deep_research.embeddings.get_embeddings",+ side_effect=fake_denial,+ ):+ with pytest.raises(PolicyDeniedError) as excinfo:+ mgr._initialize_embeddings()+ assert excinfo.value.decision.reason == "embeddings_model_not_cached"+++def test_h4_strict_meta_save_validation():+ """H4 layer 2: validate_strict_meta_combo rejects STRICT + a+ meta-picker primary at settings-save time, using effective values+ (form override falling back to DB)."""+ from types import SimpleNamespace++ from local_deep_research.security.egress.validators import (+ validate_strict_meta_combo,+ )++ def _db(value):+ return SimpleNamespace(value=value)++ # STRICT + auto (both from DB) → rejected.+ db_settings = {+ "policy.egress_scope": _db("strict"),+ "search.tool": _db("auto"),+ }+ err = validate_strict_meta_combo({}, db_settings)+ assert err is not None+ assert err["key"] == "policy.egress_scope"++ # STRICT + concrete engine → allowed.+ db_settings = {+ "policy.egress_scope": _db("strict"),+ "search.tool": _db("arxiv"),+ }+ assert validate_strict_meta_combo({}, db_settings) is None++ # Form changing engine to a concrete one while DB scope is strict →+ # allowed (effective engine = form value).+ db_settings = {+ "policy.egress_scope": _db("strict"),+ "search.tool": _db("auto"),+ }+ assert (+ validate_strict_meta_combo({"search.tool": "searxng"}, db_settings)+ is None+ )++ # Form setting scope=strict while DB engine is a meta-picker → rejected.+ db_settings = {"search.tool": _db("parallel")}+ err = validate_strict_meta_combo(+ {"policy.egress_scope": "strict"}, db_settings+ )+ assert err is not None++ # Non-strict scope → always fine.+ db_settings = {+ "policy.egress_scope": _db("both"),+ "search.tool": _db("auto"),+ }+ assert validate_strict_meta_combo({}, db_settings) is None+++def test_n14_subscription_policy_rejects_forbidden_engine():+ """N14: _validate_subscription_policy returns a rejection reason when+ the subscription's engine violates the user's egress policy, and None+ when it's allowed. Uses a mocked settings manager so no real DB is+ needed."""+ from unittest.mock import MagicMock, patch++ from local_deep_research.news import api as news_api++ snapshot = {"policy.egress_scope": "strict", "search.tool": "arxiv"}+ fake_sm = MagicMock()+ fake_sm.get_settings_snapshot.return_value = snapshot+ fake_sm.get_setting.side_effect = lambda key, default=None: (+ "arxiv" if key == "search.tool" else default+ )++ with patch(+ "local_deep_research.utilities.db_utils.get_settings_manager",+ return_value=fake_sm,+ ):+ # Under STRICT+arxiv, a non-primary engine is rejected.+ reason = news_api._validate_subscription_policy(+ db_session=MagicMock(),+ user_id="alice",+ search_engine="pubmed",+ model_provider=None,+ )+ assert reason is not None+ assert "pubmed" in reason++ # The primary engine itself is allowed.+ assert (+ news_api._validate_subscription_policy(+ db_session=MagicMock(),+ user_id="alice",+ search_engine="arxiv",+ model_provider=None,+ )+ is None+ )+++def test_n14_subscription_policy_skips_without_settings():+ """N14: when the settings backend is unavailable, validation is+ skipped (best-effort) — the execution-time factory PEP backstops."""+ from unittest.mock import MagicMock, patch++ from local_deep_research.news import api as news_api++ with patch(+ "local_deep_research.utilities.db_utils.get_settings_manager",+ side_effect=RuntimeError("no settings DB"),+ ):+ assert (+ news_api._validate_subscription_policy(+ db_session=MagicMock(),+ user_id="alice",+ search_engine="pubmed",+ model_provider="openai",+ )+ is None+ )+++def test_n15_subscription_policy_rejects_incoherent_egress_config():+ """R3 #8: an incoherent egress config (STRICT scope + meta-picker primary)+ makes context_from_snapshot raise ValueError; subscription validation must+ surface that as a misconfiguration message (fail-fast at create/update+ time), NOT let the outer except silently skip the check."""+ from unittest.mock import MagicMock, patch++ from local_deep_research.news import api as news_api++ snapshot = {"policy.egress_scope": "strict", "search.tool": "auto"}+ fake_sm = MagicMock()+ fake_sm.get_settings_snapshot.return_value = snapshot+ fake_sm.get_setting.side_effect = lambda key, default=None: (+ "auto" if key == "search.tool" else default+ )++ with patch(+ "local_deep_research.utilities.db_utils.get_settings_manager",+ return_value=fake_sm,+ ):+ reason = news_api._validate_subscription_policy(+ db_session=MagicMock(),+ user_id="alice",+ search_engine="pubmed",+ model_provider=None,+ )+ assert reason is not None+ assert "misconfigured" in reason+++# ---------------------------------------------------------------------------+# Review round 2 — regression tests for confirmed findings+# ---------------------------------------------------------------------------++from local_deep_research.security.egress.policy import ( # noqa: E402+ _classify_host,+)+++def test_classify_host_literal_metadata_ip_is_public():+ """Literal cloud-metadata IPs must classify as PUBLIC on the literal-IP+ branch too (not just the DNS branch), so evaluate_llm_endpoint /+ evaluate_embeddings / _classify_engine_url (which call _classify_host+ directly, with no metadata pre-check) can't treat 169.254.169.254 as a+ 'local' inference/search target. Regression for the IMDS-SSRF gap."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ assert _classify_host("169.254.169.254", ctx) is False+++def test_classify_host_literal_private_ip_still_local():+ """The metadata fix must NOT regress normal literal private IPs."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ assert _classify_host("10.0.0.5", ctx) is True+ assert _classify_host("127.0.0.1", ctx) is True+++def test_llm_endpoint_at_metadata_ip_denied_under_require_local():+ """End-to-end: an LLM endpoint pointed at the metadata IP must be denied+ under require_local (it previously slipped through as provider_local)."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY, require_local_llm=True)+ decision = evaluate_llm_endpoint(+ "openai_endpoint",+ ctx,+ settings_snapshot={+ "llm.openai_endpoint.url": "http://169.254.169.254/v1"+ },+ )+ assert not decision.allowed+++def test_adaptive_local_retriever_primary_resolves_private():+ """ADAPTIVE with a registered LOCAL retriever as the primary must resolve+ to PRIVATE_ONLY (forcing local inference), not the permissive BOTH.+ Regression for the private-corpus-to-cloud leak."""+ from local_deep_research.web_search_engines.retriever_registry import (+ retriever_registry,+ )++ class _FakeRetriever:+ pass++ retriever_registry.register("mykb_local", _FakeRetriever(), is_local=True)+ try:+ ctx = context_from_snapshot(+ {+ "policy.egress_scope": {"value": "adaptive"},+ "search.tool": {"value": "mykb_local"},+ },+ primary_engine="mykb_local",+ )+ assert ctx.scope == EgressScope.PRIVATE_ONLY+ assert ctx.require_local_llm is True+ assert ctx.require_local_embeddings is True+ finally:+ retriever_registry.unregister("mykb_local")+++def test_adaptive_public_retriever_primary_resolves_public():+ """ADAPTIVE with a registered PUBLIC retriever primary resolves to+ PUBLIC_ONLY (symmetry check)."""+ from local_deep_research.web_search_engines.retriever_registry import (+ retriever_registry,+ )++ class _FakeRetriever:+ pass++ retriever_registry.register("mykb_public", _FakeRetriever(), is_local=False)+ try:+ ctx = context_from_snapshot(+ {+ "policy.egress_scope": {"value": "adaptive"},+ "search.tool": {"value": "mykb_public"},+ },+ primary_engine="mykb_public",+ )+ assert ctx.scope == EgressScope.PUBLIC_ONLY+ finally:+ retriever_registry.unregister("mykb_public")+++def test_benign_denials_do_not_exhaust_fetch_quota():+ """Benign parse failures (mailto:/ftp: links scraped from pages) must NOT+ count toward the per-run denied-fetch quota — otherwise a long PUBLIC_ONLY+ run starts refusing legitimate public URLs mid-run."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ for _ in range(MAX_DENIED_FETCHES_PER_RUN + 10):+ d = evaluate_url("mailto:[email protected]", ctx)+ assert not d.allowed+ assert d.reason == "unsupported_scheme"+ # The quota was never consumed, so a legitimate public URL still passes.+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_url("https://example.com/page", ctx)+ assert decision.allowed+++def test_security_denials_still_exhaust_fetch_quota():+ """Security-relevant denials DO still count, preserving the exhaust-attack+ guard."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=True, # private host → scope_mismatch_public_only+ ):+ for _ in range(MAX_DENIED_FETCHES_PER_RUN):+ evaluate_url("https://private.example/page", ctx)+ decision = evaluate_url("https://private.example/page", ctx)+ assert decision.reason == "denial_quota_exceeded"+++def test_allowed_local_hostnames_accepts_unresolvable():+ """A legitimate intranet/VPN hostname that does not resolve (DNS down /+ split-horizon) must be ACCEPTED on save (fail-open), not rejected — the+ documented behavior. Regression for the over-block."""+ from local_deep_research.security.egress.validators import (+ validate_allowed_local_hostnames,+ )++ result = validate_allowed_local_hostnames(+ {"llm.allowed_local_hostnames": ["box.invalid.nonexistent.local"]},+ {},+ )+ assert result is None # no rejection+++def test_elasticsearch_cloud_id_rejected_under_private_scope():+ """Elasticsearch cloud_id (public Elastic Cloud) must be refused under+ PRIVATE_ONLY at construction. Regression for the cloud_id bypass."""+ from local_deep_research.web_search_engines.engines.search_engine_elasticsearch import ( # noqa: E501+ ElasticsearchSearchEngine,+ )++ snapshot = {"policy.egress_scope": {"value": "private_only"}}+ with pytest.raises(PolicyDeniedError):+ ElasticsearchSearchEngine(+ cloud_id="my-deployment:dXMtY2VudHJhbDE=",+ settings_snapshot=snapshot,+ )+++def test_elasticsearch_cloud_id_allowed_under_both_scope():+ """cloud_id is fine under BOTH (no over-block). Construction may still fail+ on the actual connect, but NOT with a PolicyDeniedError."""+ from local_deep_research.web_search_engines.engines.search_engine_elasticsearch import ( # noqa: E501+ ElasticsearchSearchEngine,+ )++ snapshot = {"policy.egress_scope": {"value": "both"}}+ assert (+ ElasticsearchSearchEngine._cloud_id_forbidden_by_scope(snapshot)+ is False+ )+++# ---------------------------------------------------------------------------+# Review round 3 — regression tests for confirmed findings+# ---------------------------------------------------------------------------++from local_deep_research.security.egress.policy import ( # noqa: E402+ _normalize_alt_ipv4,+)+++def test_normalize_alt_ipv4_encodings():+ """Octal / hex / integer IPv4 forms normalize to the dotted-quad the+ resolver would connect to; real hostnames and IPv6 return None."""+ assert _normalize_alt_ipv4("0251.0376.0251.0376") == "169.254.169.254"+ assert _normalize_alt_ipv4("2852039166") == "169.254.169.254"+ assert _normalize_alt_ipv4("0xa9fea9fe") == "169.254.169.254"+ assert _normalize_alt_ipv4("example.com") is None+ assert _normalize_alt_ipv4("::1") is None+++def test_evaluate_url_blocks_alt_encoded_metadata_every_scope():+ """R3 #1/#3: octal/integer-encoded cloud-metadata literals are blocked+ under EVERY scope with the explicit blocked_metadata_ip reason — closing+ the evaluate_url 'never permitted' gap that ipaddress-only parsing left+ open, and unifying the denial reason across scopes (no scope_mismatch+ leak for the octal form)."""+ for scope in (+ EgressScope.PUBLIC_ONLY,+ EgressScope.PRIVATE_ONLY,+ EgressScope.BOTH,+ EgressScope.STRICT,+ ):+ ctx = make_ctx(scope=scope)+ for form in ("0251.0376.0251.0376", "2852039166", "0xa9fea9fe"):+ d = evaluate_url(f"http://{form}/", ctx)+ assert not d.allowed, f"{form} allowed under {scope}"+ assert d.reason == "blocked_metadata_ip", (+ f"{form} under {scope} got {d.reason}"+ )+++def test_evaluate_url_blocks_metadata_hostnames_and_trailing_dot():+ """R4 #4/#11: complete the metadata invariant — GCP metadata HOSTNAMES+ (metadata.google.internal / .goog, any case, trailing dot) and a+ trailing-dot metadata IP must all deny with blocked_metadata_ip under+ every scope. is_ip_blocked can't see hostnames and inet_aton rejects+ trailing dots, so these dodged the round-1 fix."""+ forms = [+ "metadata.google.internal",+ "METADATA.GOOGLE.INTERNAL.",+ "metadata.goog",+ "169.254.169.254.",+ "0251.0376.0251.0376.",+ ]+ for scope in (+ EgressScope.PUBLIC_ONLY,+ EgressScope.BOTH,+ EgressScope.PRIVATE_ONLY,+ EgressScope.STRICT,+ ):+ ctx = make_ctx(scope=scope)+ for host in forms:+ d = evaluate_url(f"http://{host}/", ctx)+ assert not d.allowed, f"{host} allowed under {scope}"+ assert d.reason == "blocked_metadata_ip", (+ f"{host} under {scope} got {d.reason}"+ )+++def test_evaluate_url_metadata_hostname_fix_no_overblock():+ """The metadata-hostname/trailing-dot fix must not over-block a legit+ public host (incl. a fully-qualified trailing dot) or a legit internal+ host that merely contains 'metadata' in its name."""+ ctx_pub = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ assert evaluate_url("http://example.com./", ctx_pub).allowed+ # An unrelated internal host named like metadata.* is NOT the GCP endpoint;+ # it classifies normally (private under PUBLIC_ONLY => scope mismatch, not+ # a metadata block).+ d = evaluate_url("http://metadata.mycorp.local/", ctx_pub)+ assert d.reason != "blocked_metadata_ip"+++def test_evaluate_url_alt_encoding_does_not_block_legit_public():+ """The alt-encoding normalization must not over-block a normal public+ IP (it is not a metadata/private address)."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ d = evaluate_url("http://93.184.216.34/", ctx)+ assert d.allowed and d.reason == "allowed_public_host"+++def test_evaluate_url_both_scope_allows_public_and_private():+ """R5 #7: BOTH scope allows any classified host — both a public and a+ private host return allowed_both_scope (the deny-side is covered+ elsewhere; this pins the allow-side for the default-ish permissive+ scope)."""+ ctx = make_ctx(scope=EgressScope.BOTH)+ pub = evaluate_url("http://93.184.216.34/", ctx)+ priv = evaluate_url("http://10.0.0.5/", ctx)+ assert pub.allowed and pub.reason == "allowed_both_scope"+ assert priv.allowed and priv.reason == "allowed_both_scope"+++def test_context_from_snapshot_allow_dns_false_skips_resolution():+ """R5 #5: allow_dns=False makes ADAPTIVE resolution skip the synchronous+ getaddrinfo for a URL-engine primary (warning-banner render path). With a+ hostname that would need DNS, it falls back to BOTH instead of blocking on+ a lookup — and crucially does NOT call _resolve_with_timeout."""+ from unittest.mock import patch++ snapshot = {+ "policy.egress_scope": "adaptive",+ "search.tool": "searxng",+ "search.engine.web.searxng.default_params.instance_url": (+ "http://searx.example.test/"+ ),+ }+ with patch(+ "local_deep_research.security.egress.policy._resolve_with_timeout"+ ) as mock_dns:+ ctx = context_from_snapshot(snapshot, "searxng", allow_dns=False)+ # The whole point: NO synchronous DNS on the render path.+ mock_dns.assert_not_called()+ # Resolution still completes to a concrete scope (from the engine's static+ # classification), never left as ADAPTIVE and never blocked on DNS.+ assert ctx.scope in (EgressScope.PUBLIC_ONLY, EgressScope.BOTH)+++def test_dangerous_scheme_does_not_exhaust_fetch_quota():+ """R3 #9: javascript:/data:/file: hrefs are non-fetchable and common in+ scraped HTML; like unsupported_scheme they must not tick the anti-loop+ quota, so a doc full of data: URIs can't starve later legit fetches."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ for _ in range(MAX_DENIED_FETCHES_PER_RUN + 10):+ evaluate_url("javascript:alert(1)", ctx)+ evaluate_url("data:text/html,<b>x</b>", ctx)+ assert ctx._fetch_denial_count["count"] == 0+ # A legitimate public URL still goes through afterwards.+ d = evaluate_url("http://93.184.216.34/", ctx)+ assert d.allowed+++def test_security_denials_still_exhaust_quota_after_dangerous_scheme_change():+ """The quota still fires for genuinely security-relevant denials+ (scope mismatches), so the #9 fix didn't disarm the exhaustion guard."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ for _ in range(MAX_DENIED_FETCHES_PER_RUN + 1):+ evaluate_url("http://10.0.0.5/", ctx) # private host, public scope+ assert evaluate_url("http://93.184.216.34/", ctx).reason == (+ "denial_quota_exceeded"+ )+++def test_evaluate_embeddings_percent_encoded_local_allowed():+ """R3 #5/#11: a percent-encoded local embeddings endpoint must be allowed+ under require_local_embeddings — the HTTP client decodes it before+ connect, so the policy must classify the decoded host (consistency with+ evaluate_url / _classify_engine_url)."""+ ctx = make_ctx(+ scope=EgressScope.PRIVATE_ONLY, require_local_embeddings=True+ )+ snapshot = {"embeddings.openai.base_url": "http://127%2e0%2e0%2e1:1234/v1"}+ d = evaluate_embeddings("openai", ctx, settings_snapshot=snapshot)+ assert d.allowed and d.reason == "provider_local_endpoint"+++def test_evaluate_llm_endpoint_percent_encoded_local_allowed():+ """R3 #6/#11: same as above for the LLM endpoint gate."""+ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY, require_local_llm=True)+ snapshot = {"llm.ollama.url": "http://127%2e0%2e0%2e1:11434"}+ d = evaluate_llm_endpoint("ollama", ctx, settings_snapshot=snapshot)+ assert d.allowed and d.reason == "provider_local"+++def test_evaluate_embeddings_percent_encoded_cloud_still_blocked():+ """The unquote fix must not let a percent-encoded CLOUD host through+ under require-local."""+ ctx = make_ctx(+ scope=EgressScope.PRIVATE_ONLY, require_local_embeddings=True+ )+ # api.openai.com percent-encoded — decodes to a public host, still denied.+ snapshot = {"embeddings.openai.base_url": "http://api%2eopenai%2ecom/v1"}+ d = evaluate_embeddings("openai", ctx, settings_snapshot=snapshot)+ assert not d.allowed+++def test_context_from_snapshot_rejects_non_dict():+ """R3 #10: a non-dict snapshot fails closed with ValueError (the contract+ callers already convert to a hard policy stop), not a bare AttributeError+ that a broad except could swallow into a permissive default."""+ import pytest++ for bad in ("string", 123, ["list"], object()):+ with pytest.raises(ValueError):+ context_from_snapshot(bad, "auto")+++def test_dns_cache_first_writer_wins():+ """R3 #2: concurrent disagreeing classifications converge on the first+ writer's value, so a hostname's run classification is stable (no+ last-writer-wins flip that could relax PRIVATE_ONLY on a later fetch)."""+ from local_deep_research.security.egress.policy import (+ _cache_classification,+ )++ ctx = make_ctx(scope=EgressScope.PRIVATE_ONLY)+ assert _cache_classification(ctx, "rr.example", True) is True+ # A later disagreeing writer does not overwrite the pinned value.+ assert _cache_classification(ctx, "rr.example", False) is True+ assert ctx._dns_cache["rr.example"] is True+++def test_fetch_quota_is_per_run_not_per_context():+ """Completeness fix: the denied-fetch quota aggregates per RUN (the armed+ active context) so an attacker can't reset the budget by causing a fresh+ EgressContext to be built per engine/fetch. Denials routed through two+ different call-site contexts must share the run budget and exhaust it."""+ from local_deep_research.security.egress.audit_hook import (+ clear_active_context,+ set_active_context,+ )++ run_ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ ctx_a = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ ctx_b = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ set_active_context(run_ctx)+ try:+ half = MAX_DENIED_FETCHES_PER_RUN // 2 + 1+ for _ in range(half):+ evaluate_url("http://10.0.0.5/", ctx_a) # private host = scope deny+ for _ in range(half):+ evaluate_url("http://10.0.0.6/", ctx_b)+ # Run budget is exhausted even though neither per-call-site context+ # alone issued MAX denials.+ assert run_ctx._fetch_denial_count["count"] >= (+ MAX_DENIED_FETCHES_PER_RUN+ )+ d = evaluate_url("http://93.184.216.34/", ctx_a)+ assert d.reason == "denial_quota_exceeded"+ finally:+ clear_active_context()+++def test_fetch_quota_falls_back_to_local_ctx_without_run_context():+ """With no armed run context (snapshot-less / programmatic call), the quota+ stays per-context (unchanged behavior)."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ for _ in range(5):+ evaluate_url("http://10.0.0.5/", ctx)+ assert ctx._fetch_denial_count["count"] == 5diff --git a/tests/security/test_egress_settings_save_validators.py b/tests/security/test_egress_settings_save_validators.pynew file mode 100644index 0000000000..8e49354fa6--- /dev/null+++ b/tests/security/test_egress_settings_save_validators.py@@ -0,0 +1,381 @@+"""Integration tests for the settings-save egress validators.++These drive the REAL cross-field policy validators in+``security/egress/validators.py`` exactly as the settings write routes invoke+them (``web/routes/settings_routes.py``: ``save_all_settings`` /+``save_settings`` / ``api_update_setting``):++- ``validate_strict_meta_combo``: STRICT egress scope is incoherent with a+ meta-picker primary engine (auto / meta / parallel / parallel_scientific)+ because STRICT means "only the one concrete primary engine".+- ``validate_allowed_local_hostnames``: a PUBLIC hostname may not be smuggled+ into ``llm.allowed_local_hostnames`` (the host classifier would then trust+ external hosts as "local"); private / loopback hosts are accepted.++Each guarded property is asserted with an allow+deny pair so the test fails if+the rule were reverted. Direct-validator tests use realistic+``form_data`` / ``all_db_settings`` inputs; route-level tests drive the real+``api_update_setting`` PUT endpoint with only the settings/DB backend mocked,+proving the validation error is surfaced (not silently dropped).++All host classification uses literal IPs (8.8.8.8 public, 10.0.0.1 / 127.0.0.1+private), which the real classifier resolves OFFLINE via getaddrinfo on a+literal — no network round-trip, fully deterministic.+"""++from contextlib import contextmanager+from types import SimpleNamespace+from unittest.mock import MagicMock, patch++import pytest+from flask import Flask++from local_deep_research.security.egress import validators+from local_deep_research.security.egress.policy import _META_PICKER_ENGINES+from local_deep_research.security.egress.validators import (+ validate_allowed_local_hostnames,+ validate_strict_meta_combo,+)+from local_deep_research.web.routes.settings_routes import settings_bp++HOSTS_KEY = "llm.allowed_local_hostnames"+SCOPE_KEY = "policy.egress_scope"+ENGINE_KEY = "search.tool"+MODULE = "local_deep_research.web.routes.settings_routes"+DECORATOR_MODULE = "local_deep_research.web.utils.route_decorators"+++def _db_row(value):+ """Minimal stand-in for a Setting ORM row: the validators only read ``.value``."""+ return SimpleNamespace(value=value)+++# ---------------------------------------------------------------------------+# validate_strict_meta_combo (STRICT + meta-picker primary)+# ---------------------------------------------------------------------------+++def test_strict_with_meta_picker_primary_in_form_is_rejected():+ """STRICT scope + auto engine, both arriving in the same save, is rejected."""+ err = validate_strict_meta_combo(+ {SCOPE_KEY: "strict", ENGINE_KEY: "auto"}, {}+ )+ assert err is not None+ assert err["key"] == SCOPE_KEY+ assert "meta-picker" in err["error"]+++def test_strict_with_concrete_primary_in_form_is_accepted():+ """The allow side of the pair: STRICT + a concrete engine is coherent."""+ assert (+ validate_strict_meta_combo(+ {SCOPE_KEY: "strict", ENGINE_KEY: "arxiv"}, {}+ )+ is None+ )+++def test_non_strict_scope_with_meta_picker_is_accepted():+ """A meta-picker is only incoherent under STRICT — PUBLIC_ONLY allows it."""+ assert (+ validate_strict_meta_combo(+ {SCOPE_KEY: "public_only", ENGINE_KEY: "auto"}, {}+ )+ is None+ )+++def test_every_meta_picker_engine_is_rejected_under_strict():+ """All declared meta-picker engines trip the STRICT guard; a concrete one+ does not. Pins the guard to the real ``_META_PICKER_ENGINES`` set."""+ for engine in _META_PICKER_ENGINES:+ err = validate_strict_meta_combo(+ {SCOPE_KEY: "strict", ENGINE_KEY: engine}, {}+ )+ assert err is not None, engine+ assert (+ validate_strict_meta_combo(+ {SCOPE_KEY: "strict", ENGINE_KEY: "searxng"}, {}+ )+ is None+ )+++def test_strict_in_form_uses_db_engine_fallback_for_rejection():+ """Changing ONLY the scope to STRICT must still see the existing DB engine+ (auto) and reject — the effective-value fallback is load-bearing."""+ db = {ENGINE_KEY: _db_row("auto")}+ err = validate_strict_meta_combo({SCOPE_KEY: "strict"}, db)+ assert err is not None+ assert err["key"] == SCOPE_KEY+++def test_meta_engine_in_form_uses_db_scope_fallback_for_rejection():+ """Changing ONLY the engine to a meta-picker must see the existing STRICT+ scope in the DB and reject (mirror direction of the fallback)."""+ db = {SCOPE_KEY: _db_row("strict")}+ err = validate_strict_meta_combo({ENGINE_KEY: "parallel"}, db)+ assert err is not None+++def test_strict_with_concrete_db_engine_is_accepted():+ """Allow side of the DB-fallback pair: STRICT form + concrete DB engine."""+ db = {ENGINE_KEY: _db_row("searxng")}+ assert validate_strict_meta_combo({SCOPE_KEY: "strict"}, db) is None+++def test_missing_scope_or_engine_returns_none():+ """No decision can be made when either field is absent in form and DB."""+ assert validate_strict_meta_combo({SCOPE_KEY: "strict"}, {}) is None+ assert validate_strict_meta_combo({ENGINE_KEY: "auto"}, {}) is None+++def test_unknown_scope_value_is_not_blocked_here():+ """An unparseable scope is left to runtime fail-closed, not blocked at save."""+ assert (+ validate_strict_meta_combo(+ {SCOPE_KEY: "nonsense", ENGINE_KEY: "auto"}, {}+ )+ is None+ )+++# ---------------------------------------------------------------------------+# validate_allowed_local_hostnames (no PUBLIC host in the local allowlist)+# ---------------------------------------------------------------------------+++def test_public_host_in_local_allowlist_is_rejected():+ """A public IP would let the policy treat an external host as local."""+ err = validate_allowed_local_hostnames({HOSTS_KEY: ["8.8.8.8"]}, {})+ assert err is not None+ assert err["key"] == HOSTS_KEY+ assert "8.8.8.8" in err["error"]+++def test_private_host_in_local_allowlist_is_accepted():+ """Allow side: an RFC1918 private address is a legitimate local host."""+ assert (+ validate_allowed_local_hostnames({HOSTS_KEY: ["10.0.0.1"]}, {}) is None+ )+++def test_loopback_host_in_local_allowlist_is_accepted():+ """Loopback is local and must be accepted."""+ assert (+ validate_allowed_local_hostnames({HOSTS_KEY: ["127.0.0.1"]}, {}) is None+ )+++def test_mixed_list_rejects_only_the_public_entries():+ """A list mixing private + public is rejected, naming only the public host."""+ err = validate_allowed_local_hostnames(+ {HOSTS_KEY: ["10.0.0.1", "8.8.8.8"]}, {}+ )+ assert err is not None+ assert "8.8.8.8" in err["error"]+ assert "10.0.0.1" not in err["error"]+++def test_unresolvable_host_is_accepted_fail_open():+ """A name that does not resolve (DNS down / split-horizon) is accepted so a+ flaky-network user can still save; runtime classification still gates it.+ Resolution is stubbed to None to keep this deterministic and offline."""+ with patch.object(validators, "_resolve_with_timeout", return_value=None):+ assert (+ validate_allowed_local_hostnames(+ {HOSTS_KEY: ["intranet.corp.example"]}, {}+ )+ is None+ )+++def test_public_host_still_rejected_when_resolution_succeeds():+ """Deny counterpart to the fail-open test: when resolution returns an+ address, a public host is rejected (the stub must not blanket-accept)."""+ addrinfo = [(2, 1, 6, "", ("8.8.8.8", 0))]+ with patch.object(+ validators, "_resolve_with_timeout", return_value=addrinfo+ ):+ err = validate_allowed_local_hostnames(+ {HOSTS_KEY: ["resolves-public.example"]}, {}+ )+ assert err is not None+ assert "resolves-public.example" in err["error"]+++def test_json_string_list_of_private_host_is_accepted():+ """The save pipeline may hand the JSON-typed value as a JSON string."""+ assert (+ validate_allowed_local_hostnames({HOSTS_KEY: '["10.0.0.1"]'}, {})+ is None+ )+++def test_json_string_list_of_public_host_is_rejected():+ """Deny counterpart for the JSON-string decode path."""+ err = validate_allowed_local_hostnames({HOSTS_KEY: '["8.8.8.8"]'}, {})+ assert err is not None+ assert "8.8.8.8" in err["error"]+++def test_malformed_json_string_is_rejected():+ """A non-JSON string for a JSON-typed setting is a hard validation error."""+ err = validate_allowed_local_hostnames({HOSTS_KEY: "not json"}, {})+ assert err is not None+ assert err["key"] == HOSTS_KEY+++def test_non_list_value_is_rejected():+ """A scalar (non-list) value is rejected — must be a list of hostnames."""+ err = validate_allowed_local_hostnames({HOSTS_KEY: 5}, {})+ assert err is not None+++def test_hosts_key_absent_returns_none():+ """The guard is inert when its key is not part of the save."""+ assert validate_allowed_local_hostnames({"other.key": "x"}, {}) is None+++# ---------------------------------------------------------------------------+# Route-level wiring: api_update_setting PUT surfaces the validator errors+# ---------------------------------------------------------------------------+++def _make_setting(key, value, ui_element="text"):+ s = MagicMock()+ s.key = key+ s.value = value+ s.ui_element = ui_element+ s.editable = True+ return s+++@contextmanager+def _routed_client(existing_settings):+ """Drive api_update_setting with the auth + DB backend mocked.++ ``existing_settings`` is a list of mock Setting rows returned by+ ``db_session.query(Setting).all()``; the per-key lookup+ ``.filter(...).first()`` returns the row whose key matches the request.+ """+ app = Flask(__name__)+ app.config["SECRET_KEY"] = "test-secret"+ app.config["WTF_CSRF_ENABLED"] = False+ app.register_blueprint(settings_bp)++ by_key = {s.key: s for s in existing_settings}++ db_session = MagicMock()++ def _query(_model):+ q = MagicMock()+ q.all.return_value = list(existing_settings)++ def _filter(*_a, **_k):+ fq = MagicMock()+ # api_update_setting filters on Setting.key == <key>; the key is+ # bound in the route, so resolve it from the live request.+ from flask import request as _req++ requested = _req.view_args.get("key")+ fq.first.return_value = by_key.get(requested)+ return fq++ q.filter.side_effect = _filter+ return q++ db_session.query.side_effect = _query++ @contextmanager+ def _fake_user_session(_username):+ yield db_session++ fake_db_manager = MagicMock()+ fake_db_manager.is_user_connected.return_value = True++ patches = [+ patch(+ "local_deep_research.web.auth.decorators.db_manager",+ fake_db_manager,+ ),+ patch(+ f"{DECORATOR_MODULE}.get_user_db_session",+ side_effect=_fake_user_session,+ ),+ patch(f"{MODULE}.settings_limit", lambda f: f),+ # Isolate the egress decision from adjacent type/coercion concerns.+ patch(f"{MODULE}.validate_setting", return_value=(True, None)),+ patch(+ f"{MODULE}.coerce_setting_for_write",+ side_effect=lambda **kw: kw["value"],+ ),+ patch(f"{MODULE}.set_setting", return_value=True),+ patch(f"{MODULE}.calculate_warnings", return_value=[]),+ patch(f"{MODULE}.invalidate_settings_caches", return_value=None),+ ]+ for p in patches:+ p.start()+ try:+ with app.test_client() as client:+ with client.session_transaction() as sess:+ sess["username"] = "testuser"+ sess["session_id"] = "sid"+ yield client+ finally:+ for p in patches:+ p.stop()+++def test_route_rejects_public_local_hostname():+ """PUT llm.allowed_local_hostnames=[public IP] -> 400 with the policy error."""+ settings = [_make_setting(HOSTS_KEY, [], ui_element="json")]+ with _routed_client(settings) as client:+ resp = client.put(+ f"/settings/api/{HOSTS_KEY}", json={"value": ["8.8.8.8"]}+ )+ assert resp.status_code == 400+ assert "8.8.8.8" in resp.get_json()["error"]+++def test_route_accepts_private_local_hostname():+ """Allow counterpart: a private IP passes the validator and is saved (200)."""+ settings = [_make_setting(HOSTS_KEY, [], ui_element="json")]+ with _routed_client(settings) as client:+ resp = client.put(+ f"/settings/api/{HOSTS_KEY}", json={"value": ["10.0.0.1"]}+ )+ assert resp.status_code == 200+ assert "error" not in resp.get_json()+++def test_route_rejects_strict_scope_with_db_meta_picker():+ """PUT policy.egress_scope=strict while DB search.tool=auto -> 400."""+ settings = [+ _make_setting(SCOPE_KEY, "both"),+ _make_setting(ENGINE_KEY, "auto"),+ ]+ with _routed_client(settings) as client:+ resp = client.put(+ f"/settings/api/{SCOPE_KEY}", json={"value": "strict"}+ )+ assert resp.status_code == 400+ assert "meta-picker" in resp.get_json()["error"]+++def test_route_accepts_strict_scope_with_db_concrete_engine():+ """Allow counterpart: STRICT scope with a concrete DB engine saves (200)."""+ settings = [+ _make_setting(SCOPE_KEY, "both"),+ _make_setting(ENGINE_KEY, "arxiv"),+ ]+ with _routed_client(settings) as client:+ resp = client.put(+ f"/settings/api/{SCOPE_KEY}", json={"value": "strict"}+ )+ assert resp.status_code == 200+ assert "error" not in resp.get_json()+++if __name__ == "__main__": # pragma: no cover+ raise SystemExit(pytest.main([__file__, "-q"]))diff --git a/tests/security/test_egress_threadpool_propagation.py b/tests/security/test_egress_threadpool_propagation.pynew file mode 100644index 0000000000..f6e20288c4--- /dev/null+++ b/tests/security/test_egress_threadpool_propagation.py@@ -0,0 +1,322 @@+"""Integration tests: egress audit-hook backstop propagation into worker threads.++These are NOT PDP-unit tests. They drive the REAL runtime wiring that re-arms+the PEP-578 socket audit hook inside worker threads, because the hook reads its+active context from a ``threading.local`` that stdlib ``ThreadPoolExecutor``+workers do NOT inherit. Two real call-site mechanisms are exercised:++1. ``utilities.thread_context.preserve_research_context`` — the decorator the+ global search pool wraps tasks with. It captures the submitter thread's+ egress context at decoration time and re-arms/clears it per task.+2. The hand-rolled capture-then-rearm pattern used by ``news_strategy`` and+ ``parallel_search_engine`` (``get_active_context()`` on submit, then+ ``active_egress_context()`` / ``set_active_context()`` + finally-clear in the+ worker).++The security property under test: under an armed PRIVATE_ONLY context, a raw+``socket.connect`` to a public IP must raise ``PolicyDeniedError`` *inside the+worker* — and must NOT when the context was never propagated (proving the+re-arm is what supplies the protection, not ambient state). The hook fires+BEFORE the SYN, so no test depends on a real outbound connection completing.++Each test uses an allow+deny pair so it fails if the propagation were reverted.+Armed context is always cleared in a finally so the thread-local does not leak+to other tests.+"""++from __future__ import annotations++import concurrent.futures+import socket+import threading++import pytest++from local_deep_research.security import (+ active_egress_context,+ clear_active_context,+ get_active_context,+ install_audit_hook,+ set_active_context,+)+from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+ PolicyDeniedError,+)+from local_deep_research.utilities.thread_context import (+ preserve_research_context,+)+++def _ctx(scope: EgressScope) -> EgressContext:+ """Minimal EgressContext for a given scope (fresh per test, no shared+ denial-quota or DNS-cache state)."""+ return EgressContext(+ scope=scope,+ primary_engine="wikipedia",+ require_local_llm=False,+ require_local_embeddings=False,+ )+++def _attempt_connect(host: str, port: int = 53, timeout: float = 0.3):+ """Raw AF_INET connect; return the exception raised (or None on success).++ Under an armed enforcing context the audit hook raises PolicyDeniedError+ BEFORE any network I/O, so the kernel-level outcome for the allow-cases+ (refused/timeout) is irrelevant — we only classify whether the policy+ fired.+ """+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)+ sock.settimeout(timeout)+ try:+ sock.connect((host, port))+ return None+ except PolicyDeniedError as exc:+ return exc+ except OSError as exc:+ return exc+ finally:+ sock.close()++[email protected](autouse=True)+def _hook_and_clean_context():+ """Hook must be installed (it is at security import time, but be explicit+ and idempotent) and the thread-local must be clean around each test so a+ leaked PRIVATE_ONLY context can't mask another test's setup."""+ install_audit_hook()+ clear_active_context()+ yield+ clear_active_context()+++_PUBLIC_HOST = "8.8.8.8"+_PRIVATE_HOST = "127.0.0.1"+++# ---------------------------------------------------------------------------+# preserve_research_context (the global-pool decorator)+# ---------------------------------------------------------------------------+++def test_preserve_research_context_rearms_hook_in_worker_and_blocks_public():+ """A task wrapped with preserve_research_context, submitted to a stdlib+ ThreadPoolExecutor under an armed PRIVATE_ONLY context, must (a) run on a+ DIFFERENT thread, (b) see the SAME armed context inside the worker, and+ (c) have a public socket.connect blocked by the hook in that worker.+ """+ armed = _ctx(EgressScope.PRIVATE_ONLY)+ main_tid = threading.get_ident()+ set_active_context(armed)+ try:+ # Decoration happens here, WHILE armed — this is where the egress+ # context is captured off the submitter thread.+ def _worker():+ return {+ "tid": threading.get_ident(),+ "ctx": get_active_context(),+ "err": _attempt_connect(_PUBLIC_HOST),+ }++ wrapped = preserve_research_context(_worker)++ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:+ result = ex.submit(wrapped).result()+ finally:+ clear_active_context()++ assert result["tid"] != main_tid, "task did not run on a worker thread"+ assert result["ctx"] is armed, (+ "wrapper did not re-arm the submitter's egress context in the worker"+ )+ assert isinstance(result["err"], PolicyDeniedError), (+ f"hook did not fire in re-armed worker: {result['err']!r}"+ )+ assert result["err"].decision.reason == "scope_mismatch_private_only"+++def test_unwrapped_worker_has_no_context_so_hook_is_inactive():+ """Deny-pair to the test above: the SAME public connect, the SAME armed+ main thread, but the task is NOT wrapped. Because threading.local is not+ inherited by pool workers, the worker has no active context and the hook+ is inactive there — demonstrating exactly why the wrapper is required.+ """+ armed = _ctx(EgressScope.PRIVATE_ONLY)+ set_active_context(armed)+ try:++ def _worker():+ return {+ "ctx": get_active_context(),+ "err": _attempt_connect(_PUBLIC_HOST),+ }++ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:+ result = ex.submit(_worker).result() # unwrapped on purpose+ finally:+ clear_active_context()++ assert result["ctx"] is None, (+ "unexpected ambient context in fresh worker — test is not isolating "+ "the propagation mechanism"+ )+ assert not isinstance(result["err"], PolicyDeniedError), (+ "hook fired in a worker that never received the context — would mean "+ "the context leaked rather than being explicitly propagated"+ )+++def test_preserve_research_context_clears_context_so_reused_worker_is_clean():+ """The wrapper must clear the egress context in its finally so a reused+ pool worker does not inherit PRIVATE_ONLY into the next (possibly+ unrelated) task. Run a wrapped armed task, then a bare probe on the SAME+ single-worker executor and assert the probe lands on the same thread with+ no context and an un-gated public connect.+ """+ armed = _ctx(EgressScope.PRIVATE_ONLY)+ set_active_context(armed)+ try:++ def _armed_task():+ return {+ "tid": threading.get_ident(),+ "err": _attempt_connect(_PUBLIC_HOST),+ }++ wrapped = preserve_research_context(_armed_task)++ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:+ first = ex.submit(wrapped).result()++ # Bare probe (NOT wrapped) reused on the same idle worker.+ def _probe():+ return {+ "tid": threading.get_ident(),+ "ctx": get_active_context(),+ "err": _attempt_connect(_PUBLIC_HOST),+ }++ second = ex.submit(_probe).result()+ finally:+ clear_active_context()++ # The armed task really was protected (otherwise the "clear" claim is moot).+ assert isinstance(first["err"], PolicyDeniedError)+ # Same worker thread reused.+ assert second["tid"] == first["tid"], (+ "probe ran on a different worker — cannot prove no per-worker leak"+ )+ assert second["ctx"] is None, "wrapper leaked egress context to next task"+ assert not isinstance(second["err"], PolicyDeniedError), (+ "PRIVATE_ONLY leaked into the reused worker's next task"+ )+++# ---------------------------------------------------------------------------+# Capture-then-rearm pattern (news_strategy / parallel_search_engine)+# ---------------------------------------------------------------------------+++def _run_news_style(submit_armed: bool, host: str):+ """Replicate news_strategy: capture get_active_context() on the submitter+ thread, then re-arm it in the worker via active_egress_context(). Returns+ the connect outcome observed inside the worker.+ """+ if submit_armed:+ set_active_context(_ctx(EgressScope.PRIVATE_ONLY))+ try:+ # Capture on the submitter thread (mirrors news_strategy line 262).+ captured = get_active_context()++ def _analyze():+ with active_egress_context(captured):+ return _attempt_connect(host)++ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:+ return ex.submit(_analyze).result()+ finally:+ clear_active_context()+++def test_news_strategy_capture_then_rearm_blocks_public_under_private_only():+ """news_strategy's capture+active_egress_context re-arm must block a public+ connect in the worker when armed PRIVATE_ONLY, and must NOT block when the+ submitter had no context (captured is None → no-op)."""+ blocked = _run_news_style(submit_armed=True, host=_PUBLIC_HOST)+ assert isinstance(blocked, PolicyDeniedError), (+ f"re-armed worker failed to block public host: {blocked!r}"+ )+ assert blocked.decision.reason == "scope_mismatch_private_only"++ # Deny-pair: nothing captured → re-arm is a no-op → not blocked.+ allowed = _run_news_style(submit_armed=False, host=_PUBLIC_HOST)+ assert not isinstance(allowed, PolicyDeniedError), (+ f"hook fired with no captured context (active_egress_context(None) "+ f"should be a no-op): {allowed!r}"+ )+++def test_news_strategy_rearm_still_allows_private_host_under_private_only():+ """Re-arming PRIVATE_ONLY in the worker must not over-block: a private+ (loopback) target stays allowed, same contract as on the main thread."""+ err = _run_news_style(submit_armed=True, host=_PRIVATE_HOST)+ assert not isinstance(err, PolicyDeniedError), (+ f"re-armed PRIVATE_ONLY wrongly blocked a private host: {err!r}"+ )+++def test_parallel_engine_capture_set_then_finally_clear_no_worker_leak():+ """Replicate parallel_search_engine._run_with_context: capture the egress+ context on submit, set_active_context() in the worker, and clear it in a+ finally. Verify (a) the worker blocks a public connect while armed, and+ (b) the finally-clear leaves the reused worker clean for the next task.+ """+ set_active_context(_ctx(EgressScope.PRIVATE_ONLY))+ try:+ submitter_egress_ctx = get_active_context()++ def _run_with_context(host):+ # Mirrors the engine's wrapper: set on entry, clear in finally.+ if submitter_egress_ctx is not None:+ set_active_context(submitter_egress_ctx)+ try:+ return {+ "tid": threading.get_ident(),+ "ctx": get_active_context(),+ "err": _attempt_connect(host),+ }+ finally:+ if submitter_egress_ctx is not None:+ clear_active_context()++ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:+ armed_result = ex.submit(_run_with_context, _PUBLIC_HOST).result()++ # Bare probe reused on the same worker — must see no leak.+ def _probe():+ return {+ "tid": threading.get_ident(),+ "ctx": get_active_context(),+ "err": _attempt_connect(_PUBLIC_HOST),+ }++ probe_result = ex.submit(_probe).result()+ finally:+ clear_active_context()++ assert armed_result["ctx"] is submitter_egress_ctx+ assert isinstance(armed_result["err"], PolicyDeniedError), (+ f"engine-style worker failed to block public host: "+ f"{armed_result['err']!r}"+ )+ assert probe_result["tid"] == armed_result["tid"], (+ "probe ran on a different worker — cannot prove the finally-clear"+ )+ assert probe_result["ctx"] is None, (+ "engine wrapper's finally-clear failed: context leaked to next task"+ )+ assert not isinstance(probe_result["err"], PolicyDeniedError), (+ "PRIVATE_ONLY leaked into reused worker via missing finally-clear"+ )diff --git a/tests/security/test_egress_url_edgecases_followup.py b/tests/security/test_egress_url_edgecases_followup.pynew file mode 100644index 0000000000..ef6e113d98--- /dev/null+++ b/tests/security/test_egress_url_edgecases_followup.py@@ -0,0 +1,312 @@+"""Follow-up edge-case tests for ``evaluate_url`` (egress policy, Stage 1a).++These DEEPEN the existing ``tests/security/test_egress_policy.py`` coverage+without duplicating it. They focus on three properties that the base suite+only touches partially:++1. The metadata invariant ("cloud-metadata is NEVER fetchable, regardless of+ scope") across encodings the base suite doesn't exercise: short-form+ octal/integer IPv4, uppercase-hex, percent-encoded literals, mixed/upper+ case GCP hostnames — all denying with ``blocked_metadata_ip`` under EVERY+ scope, while a NORMAL public IP / hostname is NOT over-blocked.+2. The denial-quota accounting: dangerous (javascript/data/file/vbscript/+ about) and unsupported (ftp/mailto/...) and ``no_hostname`` denials must+ NOT tick ``MAX_DENIED_FETCHES_PER_RUN``, whereas scope-mismatch denials DO+ and eventually return ``denial_quota_exceeded``.+3. Percent-encoded private/metadata hosts are decoded BEFORE classification,+ so the policy sees the real connect target the HTTP client will reach.++It also pins the per-scope reason codes for ordinary private/public hosts+(``allowed_private_host_under_strict``, ``strict_public_host``,+``allowed_private_host``, ``scope_mismatch_*``) which the base suite asserts+only via the boolean ``allowed`` flag.++The ``make_ctx`` helper is imported from the base suite rather than+re-defined, per the no-duplicated-fixtures rule.+"""++from __future__ import annotations++from unittest.mock import patch++import pytest++from local_deep_research.security.egress.policy import (+ EgressScope,+ MAX_DENIED_FETCHES_PER_RUN,+ evaluate_url,+)++# Reuse the canonical EgressContext factory (do NOT re-define it here).+from tests.security.test_egress_policy import make_ctx++_ALL_SCOPES = (+ EgressScope.STRICT,+ EgressScope.PUBLIC_ONLY,+ EgressScope.PRIVATE_ONLY,+ EgressScope.BOTH,+)+++# ---------------------------------------------------------------------------+# Metadata invariant — encodings the base suite does not cover+# ---------------------------------------------------------------------------++[email protected]("scope", _ALL_SCOPES)+ "host",+ [+ # Short-form octal/decimal IPv4 the libc resolver expands to the AWS+ # metadata IP (169.254.<16-bit>). inet_aton accepts it; ip_address+ # does not, so it must go through the alt-encoding normalizer.+ "169.254.43518",+ # Uppercase hex 0X prefix — inet_aton accepts it; the base suite only+ # tests the lowercase form.+ "0XA9FEA9FE",+ # Percent-encoded canonical metadata literal: the HTTP client decodes+ # the host before connect, so the policy must classify the DECODED IP.+ "169%2e254%2e169%2e254",+ ],+)+def test_alt_and_encoded_metadata_blocked_every_scope(scope, host):+ """Short-form / uppercase-hex / percent-encoded metadata literals all+ resolve to 169.254.169.254 at connect time and must deny with+ ``blocked_metadata_ip`` under EVERY scope — never leaking into the+ scope-mismatch bucket and never (under STRICT/PRIVATE_ONLY) being+ mistaken for an allowed 'local' link-local host."""+ decision = evaluate_url(+ f"http://{host}/latest/meta-data/", make_ctx(scope=scope)+ )+ assert decision.allowed is False, f"{host} allowed under {scope}"+ assert decision.reason == "blocked_metadata_ip", (+ f"{host} under {scope} -> {decision.reason}"+ )++[email protected]("scope", _ALL_SCOPES)+ "host",+ [+ "Metadata.Google.Internal", # mixed case+ "METADATA.GOOG", # uppercase short GCP name+ "metadata.google.internal.", # trailing dot, lower+ "Metadata.Goog.", # mixed case + trailing dot+ ],+)+def test_gcp_metadata_hostnames_case_insensitive_blocked(scope, host):+ """GCP IMDS hostnames are matched case-insensitively and with the+ trailing dot stripped, under every scope. is_ip_blocked cannot see these+ (they are not IP literals), so the explicit hostname block must hold."""+ decision = evaluate_url(f"http://{host}/", make_ctx(scope=scope))+ assert decision.allowed is False, f"{host} allowed under {scope}"+ assert decision.reason == "blocked_metadata_ip", (+ f"{host} under {scope} -> {decision.reason}"+ )+++def test_metadata_hostname_match_is_exact_not_substring():+ """The GCP hostname guard must be an EXACT match, not a substring check:+ an attacker-controlled host that merely *starts with* the metadata name+ (``metadata.google.internal.attacker.example``) resolves to the+ attacker's box and must be classified normally — not over-blocked as a+ metadata IP (which would mask the real classification) and not allowed+ as if it were the real IMDS."""+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False, # attacker host resolves public+ ):+ decision = evaluate_url(+ "http://metadata.google.internal.attacker.example/",+ make_ctx(scope=EgressScope.PUBLIC_ONLY),+ )+ assert decision.reason != "blocked_metadata_ip"+ assert decision.allowed is True+ assert decision.reason == "allowed_public_host"+++# ---------------------------------------------------------------------------+# No over-block of ordinary public hosts (allow side of the metadata guard)+# ---------------------------------------------------------------------------+++ "scope,expected_allowed,expected_reason",+ [+ (EgressScope.PUBLIC_ONLY, True, "allowed_public_host"),+ (EgressScope.BOTH, True, "allowed_both_scope"),+ (EgressScope.PRIVATE_ONLY, False, "scope_mismatch_private_only"),+ (EgressScope.STRICT, False, "strict_public_host"),+ ],+)+def test_normal_public_ip_reason_codes(+ scope, expected_allowed, expected_reason+):+ """A normal public IP (documentation-range 93.184.216.34) is NOT+ over-blocked by the metadata/alt-encoding machinery; it gets the correct+ per-scope reason code. Literal IPs never trigger network DNS."""+ decision = evaluate_url("http://93.184.216.34/", make_ctx(scope=scope))+ assert decision.allowed is expected_allowed+ assert decision.reason == expected_reason+++ "scope,expected_allowed,expected_reason",+ [+ (EgressScope.PUBLIC_ONLY, True, "allowed_public_host"),+ (EgressScope.STRICT, False, "strict_public_host"),+ (EgressScope.PRIVATE_ONLY, False, "scope_mismatch_private_only"),+ ],+)+def test_normal_public_hostname_reason_codes(+ scope, expected_allowed, expected_reason+):+ """A normal public HOSTNAME (DNS path, mocked to resolve public) is not+ over-blocked and carries the correct per-scope reason. Exercises the+ non-literal classification branch with reason-code assertions the base+ suite leaves to the boolean flag."""+ with patch(+ "local_deep_research.security.egress.policy._classify_host",+ return_value=False,+ ):+ decision = evaluate_url(+ "https://example.com/page", make_ctx(scope=scope)+ )+ assert decision.allowed is expected_allowed+ assert decision.reason == expected_reason+++# ---------------------------------------------------------------------------+# Per-scope reason codes for ordinary PRIVATE hosts+# ---------------------------------------------------------------------------+++ "scope,expected_allowed,expected_reason",+ [+ (EgressScope.STRICT, True, "allowed_private_host_under_strict"),+ (EgressScope.PRIVATE_ONLY, True, "allowed_private_host"),+ (EgressScope.PUBLIC_ONLY, False, "scope_mismatch_public_only"),+ (EgressScope.BOTH, True, "allowed_both_scope"),+ ],+)+def test_normal_private_ip_reason_codes(+ scope, expected_allowed, expected_reason+):+ """A normal RFC1918 host (10.0.0.5) is allowed under STRICT/PRIVATE_ONLY/+ BOTH with the scope-specific allow reason, and denied under PUBLIC_ONLY —+ pinning the reason codes the metadata guard must not clobber."""+ decision = evaluate_url("http://10.0.0.5/api", make_ctx(scope=scope))+ assert decision.allowed is expected_allowed+ assert decision.reason == expected_reason+++# ---------------------------------------------------------------------------+# Percent-encoded hosts decoded BEFORE classification+# ---------------------------------------------------------------------------+++ "scope,expected_allowed,expected_reason",+ [+ (EgressScope.PRIVATE_ONLY, True, "allowed_private_host"),+ (EgressScope.STRICT, True, "allowed_private_host_under_strict"),+ (EgressScope.BOTH, True, "allowed_both_scope"),+ (EgressScope.PUBLIC_ONLY, False, "scope_mismatch_public_only"),+ ],+)+def test_percent_encoded_private_host_decoded(+ scope, expected_allowed, expected_reason+):+ """``http://192%2e168%2e1%2e1/`` decodes to the private 192.168.1.1 — the+ address the HTTP client actually connects to. The policy must classify+ the DECODED host (private), not the encoded form (which would fail DNS+ and read as public, a PUBLIC_ONLY scope bypass)."""+ decision = evaluate_url("http://192%2e168%2e1%2e1/", make_ctx(scope=scope))+ assert decision.allowed is expected_allowed+ assert decision.reason == expected_reason+++# ---------------------------------------------------------------------------+# Denial-quota accounting: which reasons tick the per-run quota+# ---------------------------------------------------------------------------+++ "url,expected_reason",+ [+ ("javascript:alert(1)", "dangerous_scheme"),+ ("data:text/html,<b>x</b>", "dangerous_scheme"),+ ("file:///etc/passwd", "dangerous_scheme"),+ ("vbscript:msgbox(1)", "dangerous_scheme"),+ ("about:blank", "dangerous_scheme"),+ ("ftp://example.com/x", "unsupported_scheme"),+ ("mailto:[email protected]", "unsupported_scheme"),+ ("tel:+15551234", "unsupported_scheme"),+ ("http:///path-only", "no_hostname"),+ ("", "url_malformed"),+ ],+)+def test_scheme_and_parse_denial_reason_codes(url, expected_reason):+ """Each non-fetchable scheme / parse failure denies with its specific+ machine reason. Breadth over the dangerous + unsupported scheme sets+ plus the no-hostname / malformed parse failures."""+ decision = evaluate_url(url, make_ctx(scope=EgressScope.PUBLIC_ONLY))+ assert decision.allowed is False+ assert decision.reason == expected_reason+++def test_benign_denials_never_tick_quota_then_legit_url_passes():+ """A document full of dangerous/unsupported/no-hostname hrefs must NOT+ exhaust the anti-loop quota: flooding well past MAX_DENIED_FETCHES_PER_RUN+ leaves the counter at zero, and a legitimate public URL still passes.+ Covers file:/vbscript:/about:/ftp:/no_hostname together (the base suite+ only floods javascript:/data: and mailto: individually)."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ for _ in range(MAX_DENIED_FETCHES_PER_RUN + 5):+ assert not evaluate_url("file:///etc/passwd", ctx).allowed+ assert not evaluate_url("ftp://host/x", ctx).allowed+ assert not evaluate_url("vbscript:x", ctx).allowed+ assert not evaluate_url("about:blank", ctx).allowed+ assert not evaluate_url("http:///nohost", ctx).allowed+ assert ctx._fetch_denial_count["count"] == 0+ # The quota was never consumed, so a real public URL is still allowed.+ decision = evaluate_url("http://93.184.216.34/", ctx)+ assert decision.allowed is True+ assert decision.reason == "allowed_public_host"+++def test_scope_mismatch_denials_eventually_exhaust_quota():+ """Scope-mismatch denials ARE security-relevant and DO tick the quota:+ exactly MAX_DENIED_FETCHES_PER_RUN private-host denials under PUBLIC_ONLY+ are processed normally, then the next call fails closed with+ ``denial_quota_exceeded``. Uses a literal RFC1918 IP so no DNS fires."""+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ for _ in range(MAX_DENIED_FETCHES_PER_RUN):+ d = evaluate_url("http://10.0.0.5/", ctx)+ assert d.allowed is False+ assert d.reason == "scope_mismatch_public_only"+ capped = evaluate_url("http://10.0.0.5/", ctx)+ assert capped.allowed is False+ assert capped.reason == "denial_quota_exceeded"+++def test_metadata_denials_count_toward_quota():+ """A metadata-IP denial is security-relevant (``blocked_metadata_ip`` is+ NOT in the non-quota set), so it ticks the counter — preventing an+ injected doc from looping the agent on IMDS targets indefinitely."""+ ctx = make_ctx(scope=EgressScope.BOTH)+ before = ctx._fetch_denial_count["count"]+ evaluate_url("http://169.254.169.254/", ctx)+ assert ctx._fetch_denial_count["count"] == before + 1+++def test_quota_cap_precedes_metadata_check():+ """Once the quota is exhausted, even a metadata URL short-circuits to+ ``denial_quota_exceeded`` (the quota gate is the first check in+ evaluate_url) — the run is uniformly fail-closed past the cap."""+ ctx = make_ctx(scope=EgressScope.BOTH)+ ctx._fetch_denial_count["count"] = MAX_DENIED_FETCHES_PER_RUN+ decision = evaluate_url("http://169.254.169.254/", ctx)+ assert decision.allowed is False+ assert decision.reason == "denial_quota_exceeded"diff --git a/tests/settings/golden_master_settings.json b/tests/settings/golden_master_settings.jsonindex f01242900e..fe6876a48c 100644--- a/tests/settings/golden_master_settings.json+++ b/tests/settings/golden_master_settings.json@@ -384,6 +384,48 @@"value": "UTC","visible": true},+ "app.warnings.dismiss_adaptive_scope_info": {+ "category": "internal",+ "description": "Permanently hide the informational banner that states what the Adaptive egress scope resolves to (public / private / both) for the current primary engine.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Dismiss adaptive-scope info banner",+ "options": null,+ "step": null,+ "type": "APP",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },+ "app.warnings.dismiss_cloud_embeddings": {+ "category": "internal",+ "description": "Permanently hide the 'document chunks will be sent to OpenAI' (cloud embeddings) banner. Separate from the public-egress dismiss flag so acknowledging fresh-install noise never suppresses this critical warning.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Dismiss cloud-embeddings warning",+ "options": null,+ "step": null,+ "type": "APP",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },+ "app.warnings.dismiss_cloud_llm": {+ "category": "internal",+ "description": "Permanently hide the 'LLM provider is cloud-hosted' banner. Separate from the public-egress dismiss flag so acknowledging fresh-install noise never suppresses this critical warning.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Dismiss cloud-LLM warning",+ "options": null,+ "step": null,+ "type": "APP",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },"app.warnings.dismiss_context_reduced": {"category": "warnings","description": "Dismiss warnings based on historical context usage and truncation",@@ -398,6 +440,20 @@"value": false,"visible": true},+ "app.warnings.dismiss_egress_policy": {+ "category": "internal",+ "description": "Permanently hide the 'public search egress enabled' banner (the fresh-install informational one). The critical cloud-LLM and cloud-embeddings banners have their OWN dismiss flags so this can't silently hide them.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Dismiss public-egress warning",+ "options": null,+ "step": null,+ "type": "APP",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },"app.warnings.dismiss_high_context": {"category": "warnings","description": "Permanently hide the warning shown when your local LLM context window is set above 8,192 tokens, which may cause memory issues on some hardware.",@@ -824,6 +880,20 @@"value": "text-embedding-3-small","visible": true},+ "embeddings.require_local": {+ "category": "policy",+ "description": "When enabled, the embeddings provider must be local (sentence_transformers, or Ollama pointed at a local URL). Indexing with cloud embeddings (OpenAI) is blocked. Indexing a corpus with cloud embeddings sends every chunk off-machine.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Require Local Embeddings",+ "options": null,+ "step": null,+ "type": "LLM",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },"focused_iteration.adaptive_questions": {"category": "focused_iteration","description": "When enabled (1), the system monitors recent search success and warns the LLM to broaden queries if 3+ searches in a recent window returned no results. When disabled (0), queries are generated without feedback from previous result counts.",@@ -1022,6 +1092,20 @@"value": 8,"visible": true},+ "llm.allowed_local_hostnames": {+ "category": "policy",+ "description": "Hostnames the policy will treat as local in addition to loopback / RFC1918 / link-local. Use for self-hosted services on custom hostnames (e.g. 'ollama.lan'). Public hostnames added here are rejected at save time.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Allowed Local Hostnames",+ "options": null,+ "step": null,+ "type": "LLM",+ "ui_element": "json",+ "value": [],+ "visible": true+ },"llm.anthropic.api_key": {"category": "llm_general","description": "Your Anthropic API key for Claude models. Get one at console.anthropic.com. Required when using the Anthropic provider.",@@ -1378,6 +1462,20 @@"value": "ollama","visible": true},+ "llm.require_local_endpoint": {+ "category": "policy",+ "description": "When enabled, the LLM provider's resolved endpoint must be on a local network (loopback, RFC1918, link-local). Cloud-only providers (OpenAI, Anthropic, Google, OpenRouter) are blocked. Independent of egress_scope.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Require Local LLM Endpoint",+ "options": null,+ "step": null,+ "type": "LLM",+ "ui_element": "checkbox",+ "value": false,+ "visible": true+ },"llm.supports_max_tokens": {"category": "llm_parameters","description": "Feature flag for max_tokens parameter. Disable if your LLM API returns errors when max_tokens is set. When disabled, the parameter is not passed to the API.",@@ -2051,6 +2149,41 @@"value": "","visible": true},+ "policy.egress_scope": {+ "category": "policy",+ "description": "EXPERIMENTAL — this egress boundary is new defense-in-depth, not an absolute guarantee yet; don't rely on it as your only protection for highly sensitive data. Restrict which search engines a research run may use. ADAPTIVE (default) = follow your primary engine: a private primary keeps everything local, a public primary allows public engines, a meta-picker allows both. STRICT = only the primary engine you picked; PUBLIC_ONLY = any public web/academic engine; PRIVATE_ONLY = any local engine (library, paperless, etc.); BOTH = any classified engine. STRICT is incompatible with auto/meta/parallel primaries.",+ "editable": true,+ "max_value": null,+ "min_value": null,+ "name": "Egress Scope (Experimental)",+ "options": [+ {+ "label": "Adaptive (default — follows your primary engine)",+ "value": "adaptive"+ },+ {+ "label": "Both (any classified engine)",+ "value": "both"+ },+ {+ "label": "Public only (no local collections)",+ "value": "public_only"+ },+ {+ "label": "Private only (no internet)",+ "value": "private_only"+ },+ {+ "label": "Strict (only the primary engine)",+ "value": "strict"+ }+ ],+ "step": null,+ "type": "SEARCH",+ "ui_element": "select",+ "value": "adaptive",+ "visible": true+ },"rag.indexing_batch_size": {"category": "research_library","description": "Number of documents to process in parallel during RAG indexing. Higher values are faster but use more memory.",diff --git a/tests/settings/test_settings_defaults_integrity.py b/tests/settings/test_settings_defaults_integrity.pyindex 61121a14c7..8b31ae9189 100644--- a/tests/settings/test_settings_defaults_integrity.py+++ b/tests/settings/test_settings_defaults_integrity.py@@ -179,7 +179,9 @@ def collect_consumed_setting_keys() -> set:_deprecated_fn = "get_setting_from_db" + "_main_thread"patterns = [r'get_setting\(\s*["\']([^"\']+)["\']',- r'_get_setting\([^,]+,\s*["\']([^"\']+)["\']',+ # Matches both _get_setting(snapshot, "key") and the egress+ # policy module's _get_setting_value(snapshot, "key", default).+ r'_get_setting(?:_value)?\([^,]+,\s*["\']([^"\']+)["\']',r'get_setting_from_snapshot\(\s*["\']([^"\']+)["\']',r'get_bool_setting\(\s*["\']([^"\']+)["\']',r'get_bool_setting_from_snapshot\(\s*["\']([^"\']+)["\']',diff --git a/tests/strategies/test_langgraph_agent_strategy.py b/tests/strategies/test_langgraph_agent_strategy.pyindex 88a1a133e4..c1a4085fd7 100644--- a/tests/strategies/test_langgraph_agent_strategy.py+++ b/tests/strategies/test_langgraph_agent_strategy.py@@ -7,11 +7,14 @@ Tests cover:- Strategy instantiation and configuration- Citation offset handling for detailed report mode- Error handling paths+- Egress-scope tool filtering (TestEgressScopeFiltering at end of file)"""import threadingfrom unittest.mock import MagicMock, patch+import pytest+# ---------------------------------------------------------------------------# SearchResultsCollector tests@@ -879,3 +882,425 @@ class TestQueryParameterNotClobbered:# reach _finalize and be recorded as the question.assert captured["query"] == original_queryassert result["findings"][0]["question"] == original_query+++# ---------------------------------------------------------------------------+# Egress-scope tool filtering+# ---------------------------------------------------------------------------+#+# The strategy's ``_build_tools`` filters the specialized-engine tool list+# against the user's ``policy.egress_scope`` BEFORE the tools reach+# ``create_agent`` (see langgraph_agent_strategy.py line 591-655). That+# pre-filter is the "core fix for the original LangGraph silent-expansion+# complaint": the factory PEP would already refuse to instantiate a+# forbidden engine at runtime, but a runtime refusal still leaks policy+# state through the LLM's tool schema and through differential denial+# latency. Filtering the *list* means the forbidden tool names never+# enter the prompt at all.+#+# These tests pin that filter at the boundary that matters — the+# LangGraph tool list — using the real ``evaluate_engine`` /+# ``evaluate_retriever`` PDPs against a controlled engine fixture. A+# regression in either the strategy's filter loop OR the PDP itself+# shows up here.+++class TestEgressScopeFiltering:+ """LangGraph tool list must honour ``policy.egress_scope`` so the LLM+ never even sees engines outside the active scope.+ """++ # Available-engines fixture. ``arxiv`` and ``pubmed`` are registered+ # public engines (``is_public = True`` on their classes); ``library``+ # is hardcoded local in ``evaluate_engine`` (line 322-326). ``auto``+ # and ``meta`` are meta-pickers that the loop skips by name.+ # ``duckduckgo`` is the current primary — already added as+ # ``web_search`` and explicitly skipped at line 618.+ _FIXTURE_AVAILABLE = {+ "arxiv": {+ "is_local": False,+ "description": "arXiv preprints",+ "strengths": ["physics", "math"],+ },+ "pubmed": {+ "is_local": False,+ "description": "PubMed biomedical literature",+ "strengths": ["medicine"],+ },+ "library": {+ "is_local": True,+ "is_retriever": False,+ "description": "Local library",+ "strengths": ["personal documents"],+ },+ # A per-collection engine. evaluate_engine hardcodes the+ # ``collection_*`` name prefix as local (egress_policy.py ~322),+ # a DISTINCT code path from the ``library`` all-collections engine.+ "collection_abc123": {+ "is_local": True,+ "is_retriever": False,+ "description": "My research papers (Collection)",+ "strengths": ["curated documents"],+ },+ "auto": {"description": "auto picker"},+ "meta": {"description": "meta picker"},+ "duckduckgo": {+ "is_local": False,+ "description": "DuckDuckGo",+ },+ }++ def _make_strategy(self, scope, primary_engine="duckduckgo"):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,+ )++ mock_search = MagicMock()+ mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"+ return LangGraphAgentStrategy(+ model=MagicMock(),+ search=mock_search,+ all_links_of_system=[],+ settings_snapshot={+ "search.tool": primary_engine,+ "policy.egress_scope": scope,+ },+ )++ @staticmethod+ def _tool_names(tools):+ names = set()+ for t in tools:+ name = getattr(t, "name", None) or getattr(t, "__name__", None)+ if name:+ names.add(name)+ return names++ # ------------------------------------------------------------------+ # STRICT — only the primary web_search; NO specialized engines at all+ # ------------------------------------------------------------------++ def test_collection_engine_treated_as_local(self):+ """A per-collection ``collection_<id>`` engine hits a DISTINCT+ classifier branch from ``library`` (the name-prefix rule in+ evaluate_engine, not a config flag). Pin that it behaves as local:+ present under PRIVATE_ONLY, filtered under PUBLIC_ONLY.+ """+ # PRIVATE_ONLY: the collection survives (it's local).+ strat_priv = self._make_strategy(scope="private_only")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.get_available_engines",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ priv_names = self._tool_names(+ strat_priv._build_tools(overall_query="q")+ )+ assert "search_collection_abc123" in priv_names, (+ "collection_<id> is local — must pass PRIVATE_ONLY"+ )++ # PUBLIC_ONLY: the collection is filtered (local data stays local).+ strat_pub = self._make_strategy(scope="public_only")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.get_available_engines",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ pub_names = self._tool_names(+ strat_pub._build_tools(overall_query="q")+ )+ assert "search_collection_abc123" not in pub_names, (+ "collection_<id> is local — must be filtered under PUBLIC_ONLY"+ )++ def test_strict_registers_no_specialized_search_tools(self):+ """STRICT means the agent gets only the primary ``web_search``+ (plus generic helpers like fetch_content / research_subtopic).+ Every ``search_*`` tool — public OR local — must be filtered+ out by the ``continue`` at line 623-627.+ """+ strategy = self._make_strategy(scope="strict")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.get_available_engines",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ tools = strategy._build_tools(overall_query="q")++ names = self._tool_names(tools)+ # The primary web_search is unaffected.+ assert "web_search" in names+ # No specialized search_* — not arxiv, not pubmed, not library.+ specialized = {n for n in names if n.startswith("search_")}+ assert specialized == set(), (+ f"STRICT must register zero specialized search_* tools, "+ f"got: {specialized}"+ )++ # ------------------------------------------------------------------+ # PRIVATE_ONLY — public engines filtered, local engines kept+ # ------------------------------------------------------------------++ def test_private_only_filters_out_public_specialized_engines(self):+ """Under PRIVATE_ONLY the agent must NOT see arxiv or pubmed —+ ``scope_mismatch_private_only`` from ``evaluate_engine`` — but+ library (``is_local=True``) passes through.+ """+ strategy = self._make_strategy(scope="private_only")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.get_available_engines",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ tools = strategy._build_tools(overall_query="q")++ names = self._tool_names(tools)+ assert "search_arxiv" not in names, (+ "arXiv is public — must be filtered under PRIVATE_ONLY"+ )+ assert "search_pubmed" not in names, (+ "PubMed is public — must be filtered under PRIVATE_ONLY"+ )+ assert "search_library" in names, (+ "library is local — must pass PRIVATE_ONLY filter"+ )++ # ------------------------------------------------------------------+ # PUBLIC_ONLY — local engines filtered, public engines kept+ # ------------------------------------------------------------------++ def test_public_only_filters_out_local_specialized_engines(self):+ """Under PUBLIC_ONLY the agent must NOT see ``search_library`` —+ ``scope_mismatch_public_only`` — but arxiv and pubmed remain.+ This is the user-data-stays-on-the-box property: a PUBLIC_ONLY+ run must never load local indexes into the agent's tool surface.+ """+ strategy = self._make_strategy(scope="public_only")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.get_available_engines",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ tools = strategy._build_tools(overall_query="q")++ names = self._tool_names(tools)+ assert "search_library" not in names, (+ "library is local — must be filtered under PUBLIC_ONLY"+ )+ assert "search_arxiv" in names, (+ "arXiv is public — must pass PUBLIC_ONLY filter"+ )+ assert "search_pubmed" in names, (+ "PubMed is public — must pass PUBLIC_ONLY filter"+ )++ # ------------------------------------------------------------------+ # BOTH (default) — every classified engine is registered+ # ------------------------------------------------------------------++ def test_both_scope_registers_every_classified_engine(self):+ """The default scope BOTH must register every classified engine+ in the available dict. ``auto``/``meta``/the current primary+ are excluded by the explicit ``continue`` at line 618 — NOT by+ the scope filter — so a regression that moved them into the+ scope-mismatch path would still be caught by the assertion that+ they're absent.+ """+ strategy = self._make_strategy(scope="both")+ with patch(+ "local_deep_research.web_search_engines.search_engines_config.get_available_engines",+ return_value=self._FIXTURE_AVAILABLE,+ ):+ tools = strategy._build_tools(overall_query="q")++ names = self._tool_names(tools)+ for expected in ("search_arxiv", "search_pubmed", "search_library"):+ assert expected in names, (+ f"Expected {expected} under BOTH but got: {sorted(names)}"+ )+ # Current engine and meta-pickers are NEVER added as specialized+ # tools regardless of scope.+ assert "search_duckduckgo" not in names+ assert "search_auto" not in names+ assert "search_meta" not in names++ # ------------------------------------------------------------------+ # Fail-closed: corrupted scope value+ # ------------------------------------------------------------------++ def test_corrupted_scope_value_propagates_policy_denied(self):+ """A junk ``policy.egress_scope`` value must NOT silently fall+ through to BOTH (the most permissive scope). ``context_from_snapshot``+ raises ``PolicyDeniedError(unknown_egress_scope)``; the strategy's+ ``_build_egress_context`` re-raises it (only ValueError / KeyError /+ TypeError get swallowed). The run aborts instead of running+ unfiltered.+ """+ from local_deep_research.security.egress.policy import (+ PolicyDeniedError,+ )++ strategy = self._make_strategy(scope="not-a-real-scope")+ with pytest.raises(PolicyDeniedError):+ strategy._build_tools(overall_query="q")++ # ------------------------------------------------------------------+ # Audit log — every block must leave an audit-bound trail+ # ------------------------------------------------------------------++ def test_blocked_engine_emits_policy_audit_log(self, loguru_caplog):+ """When the filter drops an engine, the strategy emits the+ ``specialized tool filtered by egress policy`` info line. Under+ PUBLIC_ONLY with this fixture exactly one engine (``library``)+ is local, so the line must fire exactly once — a regression+ that bypassed the filter would fire zero times, and a regression+ that over-filtered (e.g. also dropped public engines under+ PUBLIC_ONLY) would fire more than once.++ Note: ``logger.bind(policy_audit=True).info("...", engine=..., ...)``+ attaches the engine name and the ``policy_audit`` flag as loguru+ record extras, NOT to the rendered message text. Asserting the+ bound flag itself would require a custom loguru sink; we settle+ for the rendered-line invariant here.+ """+ strategy = self._make_strategy(scope="public_only")+ with (+ loguru_caplog.at_level("INFO"),+ patch(+ "local_deep_research.web_search_engines.search_engines_config.get_available_engines",+ return_value=self._FIXTURE_AVAILABLE,+ ),+ ):+ strategy._build_tools(overall_query="q")++ marker = "specialized tool filtered by egress policy"+ occurrences = loguru_caplog.text.count(marker)+ # Under PUBLIC_ONLY every LOCAL engine in the fixture is dropped:+ # ``library`` and ``collection_abc123``. One audit line per drop.+ local_engine_count = sum(+ 1+ for name, cfg in self._FIXTURE_AVAILABLE.items()+ if cfg.get("is_local") is True+ )+ assert occurrences == local_engine_count, (+ f"Expected one audit-log line per dropped local engine "+ f"({local_engine_count}), got {occurrences}. Captured text:\n"+ f"{loguru_caplog.text}"+ )+++# ---------------------------------------------------------------------------+# Policy addendum — the LLM-facing scope signal+# ---------------------------------------------------------------------------+#+# Filtering the tool LIST closes the latency-leak half of the timing+# attack. The other half is the prompt addendum: the LLM is *told* which+# tools exist so it doesn't waste tokens probing for forbidden engines.+# These tests pin that the addendum text varies by scope and is empty+# under BOTH (we don't want to bleed policy state into the LLM for the+# default scope).+++class TestEgressScopePolicyAddendum:+ """``analyze_topic`` injects a policy addendum into the system prompt+ that gets passed to ``create_agent``. The addendum's presence and+ wording must reflect the active scope.+ """++ def _make_strategy(self, scope, primary_engine="duckduckgo"):+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (+ LangGraphAgentStrategy,+ )++ mock_search = MagicMock()+ mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"+ return LangGraphAgentStrategy(+ model=MagicMock(),+ search=mock_search,+ all_links_of_system=[],+ settings_snapshot={+ "search.tool": primary_engine,+ "policy.egress_scope": scope,+ },+ )++ def _capture_prompt(self, scope, primary="duckduckgo"):+ """Run analyze_topic in a heavily-mocked harness and return the+ ``system_prompt`` string passed to ``create_agent``. There's no+ smaller public hook for the addendum — the prompt-string is+ the surface the LLM actually receives.+ """+ from langchain_core.messages import AIMessage++ strategy = self._make_strategy(scope=scope, primary_engine=primary)+ captured = {}++ mock_agent = MagicMock()+ mock_agent.stream.return_value = iter(+ [{"agent": {"messages": [AIMessage(content="done")]}}]+ )++ def fake_create_agent(model=None, tools=None, system_prompt=None, **kw):+ captured["system_prompt"] = system_prompt+ return mock_agent++ with (+ patch.object(strategy, "_build_tools", return_value=[MagicMock()]),+ patch(+ "langchain.agents.create_agent",+ side_effect=fake_create_agent,+ ),+ patch.object(strategy, "_update_progress"),+ patch.object(+ strategy,+ "_finalize",+ return_value={+ "findings": [],+ "current_knowledge": "",+ "iterations": 0,+ "error": None,+ },+ ),+ ):+ strategy.analyze_topic("q")+ return captured.get("system_prompt", "") or ""++ def test_strict_addendum_locks_llm_to_primary_engine(self):+ """STRICT must tell the LLM that ``search_*`` tools don't exist+ and name the primary engine — otherwise the LLM may probe for+ a denied tool, and the denial latency leaks policy state.+ """+ prompt = self._capture_prompt("strict")+ assert "RESTRICTED MODE" in prompt+ # The primary engine name must be cited.+ assert "duckduckgo" in prompt.lower()++ def test_private_only_addendum_names_public_engines_as_unavailable(self):+ """PRIVATE-ONLY addendum must explicitly warn the LLM that+ public engines are out of scope so it doesn't waste turns+ calling search_arxiv etc.+ """+ prompt = self._capture_prompt("private_only")+ assert "PRIVATE-ONLY MODE" in prompt+ # Names at least one canonical public engine so the LLM+ # generalises correctly.+ assert "arxiv" in prompt.lower()++ def test_public_only_addendum_names_local_engines_as_unavailable(self):+ """PUBLIC-ONLY addendum must mark local tools as unavailable —+ and it must NOT be the STRICT addendum (different scope, different+ rules).+ """+ prompt = self._capture_prompt("public_only")+ assert "PUBLIC-ONLY MODE" in prompt+ assert "RESTRICTED MODE" not in prompt+ # Names at least one canonical local tool.+ assert "library" in prompt.lower()++ def test_both_scope_injects_no_policy_addendum(self):+ """Under BOTH (default), the strategy MUST NOT inject any of the+ three scope-specific marker phrases. Bleeding scope state into+ every prompt would (a) bloat the default-case prompt for no+ reason and (b) leak which scope the user picked even when they+ didn't restrict anything.+ """+ prompt = self._capture_prompt("both")+ assert "RESTRICTED MODE" not in prompt+ assert "PRIVATE-ONLY MODE" not in prompt+ assert "PUBLIC-ONLY MODE" not in promptdiff --git a/tests/test_llm/test_llm_benchmarks.py b/tests/test_llm/test_llm_benchmarks.pyindex cd1096b07e..678adc2e35 100644--- a/tests/test_llm/test_llm_benchmarks.py+++ b/tests/test_llm/test_llm_benchmarks.py@@ -203,10 +203,14 @@ def test_benchmark_with_custom_llm_factory():):from local_deep_research.config.llm_config import get_llm+ # A settings snapshot (default "both" scope) is required so+ # the egress-policy PEP lets a non-local registered provider+ # reach the registry dispatch this test exercises.accurate_llm = get_llm(provider="benchmark_factory",model_name="accurate",temperature=0.1,+ settings_snapshot={},)# Should create the accurate versiondiff --git a/tests/test_llm/test_llm_edge_cases.py b/tests/test_llm/test_llm_edge_cases.pyindex 2b0438f7e1..e818d7129d 100644--- a/tests/test_llm/test_llm_edge_cases.py+++ b/tests/test_llm/test_llm_edge_cases.py@@ -311,7 +311,7 @@ def test_provider_name_normalization(full_settings_snapshot):assert isinstance(result, SlowLLM)-def test_factory_with_invalid_signature():+def test_factory_with_invalid_signature(full_settings_snapshot):"""Test factory functions with invalid signatures."""def bad_factory():@@ -320,9 +320,17 @@ def test_factory_with_invalid_signature():register_llm("bad_factory", bad_factory)- # Should raise error when trying to use with parameters+ # Should raise error when trying to use with parameters. A settings+ # snapshot is required so the egress-policy PEP (which fails closed for+ # snapshot-less non-local providers) lets the call reach the factory+ # dispatch, where the signature mismatch is what we are exercising.with pytest.raises(TypeError):- get_llm(provider="bad_factory", model_name="test", temperature=0.5)+ get_llm(+ provider="bad_factory",+ model_name="test",+ temperature=0.5,+ settings_snapshot=full_settings_snapshot,+ )def test_llm_memory_cleanup():diff --git a/tests/test_llm_provider_integration.py b/tests/test_llm_provider_integration.pyindex a393a810bd..b3e09da9b5 100644--- a/tests/test_llm_provider_integration.py+++ b/tests/test_llm_provider_integration.py@@ -121,6 +121,14 @@ class TestLLMProviderIntegration:settings_manager = SettingsManager(None)settings_snapshot = settings_manager.get_all_settings()+ # The fixture sets search.tool="searxng", but searxng is only an+ # available engine once an instance URL is configured (it isn't+ # here). The factory now fails closed on an unknown engine instead+ # of silently rewriting it to "auto". This test exercises the LLM+ # provider path, not search, so point it at the always-available+ # "auto" engine (AdvancedSearchSystem is mocked below anyway).+ settings_snapshot["search.tool"]["value"] = "auto"+# Verify settings are correct - get_all_settings returns nested dictsassert (settings_snapshot["llm.provider"]["value"] == "openai_endpoint"diff --git a/tests/ui_tests/NO_CI_test_egress_policy_live_research.js b/tests/ui_tests/NO_CI_test_egress_policy_live_research.jsnew file mode 100644index 0000000000..97f3937ab2--- /dev/null+++ b/tests/ui_tests/NO_CI_test_egress_policy_live_research.js@@ -0,0 +1,537 @@+/**+ * Egress-Policy Live Research Test (NOT for CI)+ * ================================================+ *+ * End-to-end run against a real local LLM (Ollama) and a real local+ * search engine (SearXNG) hosted on a known LAN box. This is the+ * lab-truth complement to test_egress_policy_ui.js:+ *+ * - configures Ollama + SearXNG endpoints+ * - kicks off a real research run under each scope and verifies the+ * run actually starts (or correctly refuses, in the STRICT+auto case)+ * - exercises the require-local-LLM toggle by trying to use a cloud+ * provider through the form+ * - captures screenshots of each run state so the UX can be eyeballed+ *+ * Why NO_CI_: the CI runner has neither Ollama nor SearXNG. The+ * tests/ui_tests/run_all_tests.js orchestrator skips files matching+ * /^NO_CI_/ so this script never runs there. See sibling+ * NO_CI_executes_research_*.js for the same convention.+ *+ * Endpoints (override via env if your lab box changes IP):+ * OLLAMA_URL default http://192.168.178.66:11434+ * SEARXNG_URL default http://192.168.178.66:8081+ * OLLAMA_MODEL default llama3.2:1b+ *+ * Usage (local dev):+ * node tests/ui_tests/NO_CI_test_egress_policy_live_research.js+ * HEADLESS=false node tests/ui_tests/NO_CI_test_egress_policy_live_research.js+ */++const fs = require('fs');+const path = require('path');+const http = require('http');++const puppeteer = require('puppeteer');+const AuthHelper = require('./auth_helper');+const { getPuppeteerLaunchOptions } = require('./puppeteer_config');++const BASE_URL = process.env.BASE_URL || 'http://127.0.0.1:5000';+const OLLAMA_URL = process.env.OLLAMA_URL || 'http://192.168.178.66:11434';+const SEARXNG_URL = process.env.SEARXNG_URL || 'http://192.168.178.66:8080';+const OLLAMA_MODEL = process.env.OLLAMA_MODEL || 'qwen3.6:latest';+const IS_CI = !!process.env.CI;++const SCREENSHOT_DIR = path.join(+ __dirname,+ 'screenshots',+ 'egress-policy-live',+);++const results = [];+function record(label, ok, detail) {+ results.push({ label, ok, detail });+ const icon = ok ? '✅' : '❌';+ console.log(` ${icon} ${label}${detail ? ` — ${detail}` : ''}`);+}++function ensureDir(dir) {+ if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });+}++async function screenshot(page, name) {+ if (IS_CI) return;+ ensureDir(SCREENSHOT_DIR);+ const file = path.join(SCREENSHOT_DIR, `${name}.png`);+ await page.screenshot({ path: file, fullPage: true });+ console.log(` 📸 ${path.relative(process.cwd(), file)}`);+}++async function sleep(ms) {+ return new Promise(r => setTimeout(r, ms));+}++/**+ * Reachability check — if Ollama / SearXNG are off-LAN today we want+ * to skip with a clear message instead of producing a confusing+ * Puppeteer error 8 minutes into the run.+ */+async function ping(url, label, timeoutMs = 5000) {+ return new Promise((resolve) => {+ const req = http.get(url, { timeout: timeoutMs }, (res) => {+ const ok = res.statusCode >= 200 && res.statusCode < 500;+ res.resume();+ resolve(ok);+ });+ req.on('error', () => resolve(false));+ req.on('timeout', () => {+ req.destroy();+ resolve(false);+ });+ }).then((ok) => {+ record(`reachable:${label}`, ok, `${url}`);+ return ok;+ });+}++/**+ * Save a single setting via the live-save endpoint. The UI components+ * call this the same way on change events so it exercises the same+ * code path.+ */+async function saveSetting(page, key, value) {+ // Endpoint contract: PUT /settings/api/<dotted.key> with body+ // {"value": ...}. CSRF token is in a meta tag in base.html.+ return page.evaluate(+ async (k, v, base) => {+ const csrfMeta = document.querySelector('meta[name="csrf-token"]');+ const csrf = csrfMeta ? csrfMeta.getAttribute('content') : '';+ const resp = await fetch(+ `${base}/settings/api/${encodeURIComponent(k)}`,+ {+ method: 'PUT',+ headers: {+ 'Content-Type': 'application/json',+ 'X-CSRFToken': csrf,+ },+ credentials: 'same-origin',+ body: JSON.stringify({ value: v }),+ },+ );+ return { status: resp.status, ok: resp.ok };+ },+ key,+ value,+ BASE_URL,+ );+}++async function selectScope(page, value) {+ await page.select('#policy_egress_scope', value);+ await sleep(600);+ return page.evaluate(() => document.body.dataset.scope);+}++/**+ * Submit a research run via the API. Driving the actual <form> tag is+ * brittle because the JS handler immediately window.location.assign()'s+ * to /progress/<id> on a successful submit — Puppeteer's execution+ * context is destroyed mid-evaluate. Hitting /api/start_research+ * directly exercises the same server entry point (including the+ * precheck added in research_routes.py:248) and lets us observe the+ * status code cleanly without racing the redirect.+ *+ * Payload mirrors what the form posts (see research.js handleResearchSubmit).+ */+/**+ * Terminate a running research so we don't blow through the per-user+ * concurrent-research cap (server logs show "Active research count: 3/3"+ * at the 4th run). Idempotent — silently ignores missing IDs.+ */+async function terminateResearch(page, researchId) {+ if (!researchId) return;+ await page.evaluate(+ async (id, base) => {+ const csrfMeta = document.querySelector('meta[name="csrf-token"]');+ const csrf = csrfMeta ? csrfMeta.getAttribute('content') : '';+ try {+ await fetch(`${base}/api/terminate/${id}`, {+ method: 'POST',+ headers: { 'X-CSRFToken': csrf },+ credentials: 'same-origin',+ });+ } catch { /* idempotent */ }+ },+ researchId,+ BASE_URL,+ );+}++async function startResearch(page, query, overrides = {}) {+ return page.evaluate(+ async (q, base, ov) => {+ const csrfMeta = document.querySelector('meta[name="csrf-token"]');+ const csrf = csrfMeta ? csrfMeta.getAttribute('content') : '';+ const payload = {+ query: q,+ mode: 'quick',+ model_provider: 'ollama',+ search_engine: 'searxng',+ ...ov,+ };+ const resp = await fetch(`${base}/api/start_research`, {+ method: 'POST',+ headers: {+ 'Content-Type': 'application/json',+ 'X-CSRFToken': csrf,+ },+ credentials: 'same-origin',+ body: JSON.stringify(payload),+ });+ let body;+ try { body = await resp.json(); } catch { body = null; }+ return { status: resp.status, body };+ },+ query,+ BASE_URL,+ overrides,+ );+}++async function run(name, fn) {+ console.log(`\n— ${name}`);+ try {+ await fn();+ } catch (err) {+ record(name, false, err.message);+ console.error(` ${err.stack || err}`);+ }+}++async function main() {+ console.log(`\n🧪 Egress-Policy LIVE Research Test`);+ console.log(` Ollama : ${OLLAMA_URL} (model ${OLLAMA_MODEL})`);+ console.log(` SearXNG : ${SEARXNG_URL}`);+ console.log('='.repeat(60));++ const ollamaUp = await ping(`${OLLAMA_URL}/api/tags`, 'ollama');+ const searxUp = await ping(SEARXNG_URL, 'searxng');+ if (!ollamaUp || !searxUp) {+ console.log(+ '\n⚠️ Endpoints not reachable. This test is intentionally local-only ' ++ '(NO_CI_) — point OLLAMA_URL / SEARXNG_URL at your lab and re-run, ' ++ 'or skip if you cannot.',+ );+ process.exit(0); // soft-skip; runner is permissive+ }++ const browser = await puppeteer.launch(getPuppeteerLaunchOptions());+ const page = await browser.newPage();+ const authHelper = new AuthHelper(page, BASE_URL);++ page.on('pageerror', (e) => console.log(` ⚠️ pageerror: ${e.message}`));+ page.on('console', (msg) => {+ if (msg.type() === 'error') {+ console.log(` ⚠️ console.error: ${msg.text()}`);+ }+ });++ let exitCode = 0;+ try {+ console.log('\n🔐 Authenticate');+ await authHelper.ensureAuthenticatedWithTimeout();++ // -----------------------------------------------------------+ // Pre-flight: write Ollama / SearXNG into settings so the UI+ // form picks them up.+ // -----------------------------------------------------------+ console.log('\n📌 Configure local LLM + SearXNG');+ await page.goto(BASE_URL, { waitUntil: 'domcontentloaded', timeout: 30000 });++ await saveSetting(page, 'llm.provider', 'ollama');+ await saveSetting(page, 'llm.model', OLLAMA_MODEL);+ await saveSetting(page, 'llm.ollama.url', OLLAMA_URL);+ await saveSetting(page, 'llm.require_local_endpoint', true);+ await saveSetting(page, 'search.engine.web.searxng.instance_url', SEARXNG_URL);+ // Tie research to searxng explicitly so STRICT is allowed.+ await saveSetting(page, 'search.tool', 'searxng');+ await sleep(800);++ // Reload to ensure the form reads from the new settings.+ await page.reload({ waitUntil: 'domcontentloaded' });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });+ await screenshot(page, '00-form-configured');++ // -----------------------------------------------------------+ // 1. BOTH scope — research starts, scope cue is amber-light+ // -----------------------------------------------------------+ await run('1. BOTH scope: research starts successfully', async () => {+ await selectScope(page, 'both');+ await screenshot(page, '01-both-form');+ const r = await startResearch(page, 'What is the capital of France?');+ record(+ 'both-submit-200',+ r.status === 200,+ `status=${r.status} body=${JSON.stringify(r.body).slice(0, 120)}`,+ );+ await terminateResearch(page, r.body && r.body.research_id);+ });++ await page.goto(BASE_URL, { waitUntil: 'domcontentloaded', timeout: 30000 });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });++ // -----------------------------------------------------------+ // 2. PUBLIC_ONLY + SearXNG-on-private-IP → server refuses+ // Dynamic URL classification (_classify_engine_url at+ // egress_policy.py:362) reclassifies the engine as is_local+ // because the configured instance URL points at 192.168.x.+ // PUBLIC_ONLY then refuses it with scope_mismatch_public_only+ // — exactly the right behavior. The classifier overriding+ // the static is_public=True flag is the subtle correctness+ // property we want this test to lock in.+ // -----------------------------------------------------------+ await run('2. PUBLIC_ONLY + local-IP SearXNG: server refuses (URL reclassifies)', async () => {+ await selectScope(page, 'public_only');+ await screenshot(page, '02-public-only-form');+ const r = await startResearch(page, 'Explain the photoelectric effect briefly.', {+ policy_egress_scope: 'public_only',+ });+ const refused =+ r.status === 400 &&+ /scope_mismatch_public_only/i.test(+ (r.body && r.body.message) || '',+ );+ record(+ 'public-only-refuses-local-ip-searxng',+ refused,+ `status=${r.status} msg=${(r.body && r.body.message || '').slice(0, 120)}`,+ );+ await screenshot(page, '02-public-only-refused');+ await terminateResearch(page, r.body && r.body.research_id);+ });++ await page.goto(BASE_URL, { waitUntil: 'domcontentloaded', timeout: 30000 });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });++ // -----------------------------------------------------------+ // 3. PRIVATE_ONLY + SearXNG-on-private-IP → allowed+ // Same URL-classification override, this time on the+ // permissive side: [email protected] is treated as a local+ // engine, so PRIVATE_ONLY (which forbids public egress)+ // correctly allows it. This is the user-visible payoff of+ // "lab-hosted SearXNG counts as private" — and ensures we+ // haven't regressed into blocking on the static is_public+ // flag alone.+ // -----------------------------------------------------------+ await run('3. PRIVATE_ONLY + local-IP SearXNG: allowed (URL reclassifies)', async () => {+ await selectScope(page, 'private_only');+ await screenshot(page, '03-private-only-form');+ const r = await startResearch(page, 'Test private-only with lab SearXNG.', {+ policy_egress_scope: 'private_only',+ });+ record(+ 'private-only-allows-local-ip-searxng',+ r.status === 200,+ `status=${r.status} body=${JSON.stringify(r.body).slice(0, 120)}`,+ );+ await screenshot(page, '03-private-only-accepted');+ await terminateResearch(page, r.body && r.body.research_id);+ });++ await page.goto(BASE_URL, { waitUntil: 'domcontentloaded', timeout: 30000 });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });++ // -----------------------------------------------------------+ // 4. STRICT + searxng (concrete engine) — research starts+ // -----------------------------------------------------------+ await run('4. STRICT + searxng: research starts', async () => {+ await selectScope(page, 'strict');+ await screenshot(page, '04-strict-form');+ const r = await startResearch(page, 'What is photosynthesis?', {+ policy_egress_scope: 'strict',+ });+ record(+ 'strict-submit-200',+ r.status === 200,+ `status=${r.status} body=${JSON.stringify(r.body).slice(0, 120)}`,+ );+ await terminateResearch(page, r.body && r.body.research_id);+ });++ await page.goto(BASE_URL, { waitUntil: 'domcontentloaded', timeout: 30000 });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });++ // -----------------------------------------------------------+ // 5. STRICT + auto via API — server-side guard refuses even+ // if the UI guard is bypassed. We send a payload that the+ // UI dropdown would not allow.+ // -----------------------------------------------------------+ await run('5. STRICT + auto via API: server-side refusal', async () => {+ // Force the back-end to see scope=strict + search.tool=auto.+ await saveSetting(page, 'search.tool', 'auto');+ await saveSetting(page, 'policy.egress_scope', 'strict');++ // Bypass the UI guard with a direct POST mimicking the form.+ const result = await page.evaluate(+ async (base) => {+ const csrfMeta = document.querySelector('meta[name="csrf-token"]');+ const csrf = csrfMeta ? csrfMeta.getAttribute('content') : '';+ const resp = await fetch(`${base}/api/start_research`, {+ method: 'POST',+ headers: {+ 'Content-Type': 'application/json',+ 'X-CSRFToken': csrf,+ },+ credentials: 'same-origin',+ body: JSON.stringify({+ query: 'strict-auto-incoherent',+ mode: 'quick',+ model_provider: 'ollama',+ search_engine: 'auto',+ policy_egress_scope: 'strict',+ }),+ });+ let body;+ try { body = await resp.json(); } catch { body = null; }+ return { status: resp.status, body };+ },+ BASE_URL,+ );+ // The precheck (research_routes.py) now surfaces this as a+ // clean 400 with reason text. Old behaviour returned None+ // and silently allowed the run.+ record(+ 'strict-auto-refused-400',+ result.status === 400,+ `status=${result.status} msg=${(result.body && result.body.message || '').slice(0, 120)}`,+ );+ await screenshot(page, '05-strict-auto-api-refusal');++ // Restore.+ await saveSetting(page, 'search.tool', 'searxng');+ await saveSetting(page, 'policy.egress_scope', 'both');+ });++ // -----------------------------------------------------------+ // 6. Require-local-LLM + cloud-provider attempt+ // The run-start precheck only validates the search engine+ // (research_routes.py:262 evaluate_engine). LLM-endpoint+ // enforcement happens later, at get_llm() construction time+ // inside the strategy. So we start the run, poll the+ // research status, and assert the policy denial surfaces as+ // a research failure (not a precheck 400). The unit-test+ // parametrized at tests/security/test_egress_policy.py+ // asserts the PolicyDeniedError contract directly.+ // -----------------------------------------------------------+ await run('6. require_local_endpoint + openai → research fails with policy', async () => {+ await saveSetting(page, 'llm.require_local_endpoint', true);+ await saveSetting(page, 'llm.provider', 'openai');+ await saveSetting(page, 'llm.openai.api_key', 'sk-not-a-real-key');+ await saveSetting(page, 'llm.model', 'gpt-4o-mini');+ await sleep(500);++ const start = await startResearch(+ page,+ 'cloud-llm-under-local-only',+ { model_provider: 'openai' },+ );+ const researchId = start.body && (start.body.research_id || start.body.id);+ record(+ 'cloud-llm-run-accepted-at-precheck',+ start.status === 200 && !!researchId,+ `status=${start.status} research_id=${researchId}`,+ );++ // Poll the research status briefly — the strategy will try+ // to instantiate the LLM, hit the PEP, and the run will be+ // marked failed with a policy reason. We give it up to ~20s+ // before declaring the test inconclusive.+ let status = null;+ let errMsg = '';+ if (!researchId) {+ record(+ 'cloud-llm-eventually-policy-denied',+ false,+ 'no research_id returned at precheck',+ );+ return;+ }+ for (let i = 0; i < 20; i++) {+ await sleep(1000);+ const s = await page.evaluate(+ async (id, base) => {+ try {+ const r = await fetch(+ `${base}/api/research/${id}/status`,+ { credentials: 'same-origin' },+ );+ if (!r.ok) return null;+ return r.json();+ } catch {+ return null;+ }+ },+ researchId,+ BASE_URL,+ );+ if (s) {+ status = (s.status || '').toLowerCase();+ errMsg = (s.error || s.message || '').toString();+ if (status === 'failed' || status === 'error' || status === 'completed') {+ break;+ }+ }+ }+ const policyDenied =+ /policy|local|cloud|denied|require/i.test(errMsg) ||+ /no_snapshot|provider_cloud/i.test(errMsg);+ record(+ 'cloud-llm-eventually-policy-denied',+ policyDenied || status === 'failed',+ `final_status=${status} err=${errMsg.slice(0, 120)}`,+ );+ await screenshot(page, '06-cloud-llm-refused');++ await terminateResearch(page, researchId);++ // Restore local config.+ await saveSetting(page, 'llm.provider', 'ollama');+ await saveSetting(page, 'llm.model', OLLAMA_MODEL);+ await saveSetting(page, 'llm.require_local_endpoint', true);+ });++ // -----------------------------------------------------------+ // 7. Done.+ // -----------------------------------------------------------+ await page.goto(BASE_URL, { waitUntil: 'domcontentloaded', timeout: 30000 });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });+ await selectScope(page, 'both');+ await screenshot(page, '07-final-state');++ } catch (err) {+ console.error('\n❌ Suite-level error:', err.stack || err);+ exitCode = 1;+ } finally {+ console.log('\n' + '='.repeat(60));+ const passed = results.filter((r) => r.ok).length;+ const failed = results.filter((r) => !r.ok);+ console.log(`Results: ${passed} / ${results.length} passed`);+ if (failed.length) {+ console.log('Failures:');+ for (const f of failed) {+ console.log(` ❌ ${f.label}${f.detail ? ` — ${f.detail}` : ''}`);+ }+ exitCode = 1;+ }+ if (!IS_CI) {+ console.log(`\nScreenshots: ${SCREENSHOT_DIR}`);+ }+ await browser.close();+ process.exit(exitCode);+ }+}++main().catch((err) => {+ console.error(err);+ process.exit(1);+});diff --git a/tests/ui_tests/test_egress_policy_ui.js b/tests/ui_tests/test_egress_policy_ui.jsnew file mode 100644index 0000000000..8b2a04cbf8--- /dev/null+++ b/tests/ui_tests/test_egress_policy_ui.js@@ -0,0 +1,722 @@+/**+ * Egress-Policy UI Test (CI-safe)+ * ================================+ *+ * Exercises the Privacy & Egress feature surface that does NOT need a+ * live LLM or search engine: dropdown, scope-derived colour cues, live+ * propagation to body[data-scope], STRICT + meta-picker guard, settings+ * persistence across reloads, chat-page inheritance, warning banner+ * presence/absence per scope, and the require-local toggles.+ *+ * Live-engine work (real research runs against Ollama/SearXNG) lives in+ * NO_CI_test_egress_policy_live_research.js — runner skips it in CI.+ *+ * Screenshots: many. All gated on !process.env.CI per project+ * convention (see test_settings_page.js:57). They drop into+ * tests/ui_tests/screenshots/egress-policy/ for visual inspection.+ *+ * Usage:+ * node tests/ui_tests/test_egress_policy_ui.js+ * HEADLESS=false node tests/ui_tests/test_egress_policy_ui.js # see the browser+ */++const fs = require('fs');+const path = require('path');++const puppeteer = require('puppeteer');+const AuthHelper = require('./auth_helper');+const { getPuppeteerLaunchOptions } = require('./puppeteer_config');++const BASE_URL = process.env.BASE_URL || 'http://127.0.0.1:5000';+const IS_CI = !!process.env.CI;+const SCREENSHOT_DIR = path.join(+ __dirname,+ 'screenshots',+ 'egress-policy',+);++// Public colour palette for each scope (mirrors base.html style block).+// Used to verify the computed border-left-color matches what the user+// is supposed to see — catches regressions where the data-scope flips+// but the CSS rule is silently broken. BOTH intentionally has no+// accent (default = no cue, the warning banner does the nagging) so+// the test for that scope checks "no color set" rather than a hue.+const SCOPE_PALETTE = {+ adaptive: { border: null, name: 'no-accent' },+ both: { border: null, name: 'no-accent' },+ public_only: { border: 'rgb(59, 130, 246)', name: 'blue-500' },+ private_only: { border: 'rgb(20, 163, 127)', name: 'teal-600' },+ strict: { border: 'rgb(139, 92, 246)', name: 'violet-500' },+};++// Tally of pass / fail per assertion so the final report is informative.+const results = [];+function record(label, ok, detail) {+ results.push({ label, ok, detail });+ const icon = ok ? '✅' : '❌';+ console.log(` ${icon} ${label}${detail ? ` — ${detail}` : ''}`);+}++function ensureDir(dir) {+ if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });+}++async function screenshot(page, name) {+ if (IS_CI) return; // CI = no screenshots (per convention)+ ensureDir(SCREENSHOT_DIR);+ const file = path.join(SCREENSHOT_DIR, `${name}.png`);+ await page.screenshot({ path: file, fullPage: true });+ console.log(` 📸 ${path.relative(process.cwd(), file)}`);+}++async function sleep(ms) {+ return new Promise(r => setTimeout(r, ms));+}++/**+ * Set the scope dropdown to `value` and wait for the live JS hook to+ * update body[data-scope]. Returns the body's data-scope after the+ * change so callers can assert on it.+ */+async function selectScope(page, value) {+ await page.select('#policy_egress_scope', value);+ // Allow the change event handler (saveSearchSetting ++ // applyPrivacyPanelScope) to fan out. Save is async (POST to+ // /settings/api/save) so we wait a touch longer.+ await sleep(IS_CI ? 600 : 400);+ return page.evaluate(() => document.body.dataset.scope);+}++/**+ * Read the computed border-left-color of the research card. This is+ * the user-visible cue — if a CSS regression sneaks in, this is the+ * test that catches it (rather than a DOM-only assertion that would+ * pass with broken styles).+ */+async function readCardBorderColor(page) {+ return page.evaluate(() => {+ const card = document.querySelector('.ldr-card.ldr-research-card');+ if (!card) return null;+ return window.getComputedStyle(card).borderLeftColor;+ });+}++async function readQueryBorderColor(page) {+ return page.evaluate(() => {+ const q = document.querySelector('#query');+ if (!q) return null;+ return window.getComputedStyle(q).borderColor;+ });+}++async function readPanelDataScope(page) {+ return page.evaluate(() => {+ const panel = document.querySelector('.ldr-privacy-panel');+ return panel ? panel.dataset.scope : null;+ });+}++/**+ * The STRICT scope option is disabled when search_engine resolves to a+ * meta-picker (auto/meta/parallel*). Returns true if STRICT is currently+ * a disabled option.+ */+async function strictOptionDisabled(page) {+ return page.evaluate(() => {+ const opt = document.querySelector(+ '#policy_egress_scope option[value="strict"]',+ );+ return opt ? !!opt.disabled : null;+ });+}++/**+ * Persist the search engine via the settings save endpoint, then+ * reload the page. The STRICT-availability guard+ * (updateStrictScopeAvailability) is module-scoped inside the+ * research.js IIFE — the only external triggers are the custom-select+ * callback and init-time evaluation. Reloading is the cleanest way to+ * exercise the init path with a known engine value without simulating+ * the entire custom-select dance.+ */+async function setSearchEngine(page, value) {+ // The settings save endpoint is PUT /settings/api/<key> with body+ // {"value": ...} — same as saveSearchSetting() in research.js. The+ // CSRF token comes from the meta tag rendered by base.html.+ const result = await page.evaluate(+ async (v, base) => {+ const csrfMeta = document.querySelector('meta[name="csrf-token"]');+ const csrf = csrfMeta ? csrfMeta.getAttribute('content') : '';+ const resp = await fetch(`${base}/settings/api/search.tool`, {+ method: 'PUT',+ headers: {+ 'Content-Type': 'application/json',+ 'X-CSRFToken': csrf,+ },+ credentials: 'same-origin',+ body: JSON.stringify({ value: v }),+ });+ return resp.status;+ },+ value,+ BASE_URL,+ );+ if (result !== 200 && result !== 204) {+ throw new Error(`settings save failed: status=${result} key=search.tool value=${value}`);+ }+ await page.reload({ waitUntil: 'domcontentloaded' });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });+ // Wait for the search-engine dropdown bootstrap + the init-time+ // strict-availability evaluation (line 389 / 399 in research.js).+ await sleep(IS_CI ? 1500 : 800);+}++/**+ * Run a single test scenario inside a try/catch so one failure does+ * not poison the rest of the suite (a flaky CI assertion shouldn't+ * mask working coverage of the other 20 things we check).+ */+async function run(name, fn) {+ console.log(`\n— ${name}`);+ try {+ await fn();+ } catch (err) {+ record(name, false, err.message);+ console.error(` ${err.stack || err}`);+ }+}++async function main() {+ console.log(`\n🧪 Egress-Policy UI Test (CI=${IS_CI ? 'yes' : 'no'})`);+ console.log('='.repeat(60));++ const browser = await puppeteer.launch(getPuppeteerLaunchOptions());+ const page = await browser.newPage();+ const authHelper = new AuthHelper(page, BASE_URL);++ // Log console errors — silent JS exceptions on the research page+ // would otherwise hide a broken data-scope hook.+ page.on('pageerror', (e) => console.log(` ⚠️ pageerror: ${e.message}`));+ page.on('console', (msg) => {+ if (msg.type() === 'error') {+ console.log(` ⚠️ console.error: ${msg.text()}`);+ }+ });++ let exitCode = 0;+ try {+ console.log('\n🔐 Authenticate');+ await authHelper.ensureAuthenticatedWithTimeout();+ console.log(' ✅ logged in');++ // -----------------------------------------------------------+ // 1. Privacy & Egress panel exists and is reachable on /research+ // -----------------------------------------------------------+ await run('1. Privacy & Egress panel renders on /', async () => {+ await page.goto(`${BASE_URL}/`, {+ waitUntil: 'domcontentloaded',+ timeout: 30000,+ });+ await page.waitForSelector('.ldr-privacy-panel', { timeout: 10000 });+ await page.waitForSelector('#policy_egress_scope', { timeout: 5000 });+ await page.waitForSelector('#llm_require_local_endpoint', { timeout: 5000 });+ await page.waitForSelector('#embeddings_require_local', { timeout: 5000 });+ await screenshot(page, '01-panel-loaded');+ record('panel-renders', true);+ });++ // -----------------------------------------------------------+ // 2. Scope dropdown lists all scope options (incl. adaptive)+ // -----------------------------------------------------------+ await run('2. All scope options present', async () => {+ const opts = await page.$$eval(+ '#policy_egress_scope option',+ (els) => els.map((o) => ({ value: o.value, text: o.textContent.trim() })),+ );+ const wantVals = ['adaptive', 'both', 'public_only', 'private_only', 'strict'];+ const haveVals = opts.map((o) => o.value);+ const ok = wantVals.every((v) => haveVals.includes(v));+ record(+ 'scopes-listed',+ ok,+ `found ${haveVals.join(', ')}`,+ );+ });++ // -----------------------------------------------------------+ // 3. Default scope = "both" → NO accent cue. The risk-honest+ // palette intentionally leaves BOTH unmarked at the+ // border-color layer so reserving colour for active-choice+ // scopes (sky / teal / violet) is more legible; the "Public+ // search egress enabled" warning banner does the actual+ // nagging in text.+ // -----------------------------------------------------------+ await run('3. Default scope shows no accent cue', async () => {+ const bodyScope = await selectScope(page, 'both');+ record('body-scope-both', bodyScope === 'both', `data-scope=${bodyScope}`);++ const cardColor = await readCardBorderColor(page);+ // The default rule above sets border-left to "5px solid+ // transparent" with no body[data-scope="both"] override, so+ // the computed colour is "rgba(0, 0, 0, 0)" (transparent).+ const isTransparent =+ !cardColor ||+ cardColor === 'rgba(0, 0, 0, 0)' ||+ cardColor === 'transparent';+ record(+ 'card-color-both-transparent',+ isTransparent,+ `border-left-color=${cardColor}`,+ );++ const panelScope = await readPanelDataScope(page);+ record('panel-data-scope-both', panelScope === 'both', `panel=${panelScope}`);++ await screenshot(page, '02-scope-both');+ });++ // -----------------------------------------------------------+ // 4. Switch to PUBLIC_ONLY → blue, panel reflects scope+ // -----------------------------------------------------------+ await run('4. PUBLIC_ONLY paints blue', async () => {+ const bodyScope = await selectScope(page, 'public_only');+ record('body-scope-public', bodyScope === 'public_only');++ const cardColor = await readCardBorderColor(page);+ record(+ 'card-color-public',+ cardColor === SCOPE_PALETTE.public_only.border,+ `border-left-color=${cardColor}`,+ );++ const queryColor = await readQueryBorderColor(page);+ record(+ 'query-border-public',+ queryColor === SCOPE_PALETTE.public_only.border,+ `border-color=${queryColor}`,+ );++ await screenshot(page, '03-scope-public-only');+ });++ // -----------------------------------------------------------+ // 5. Switch to PRIVATE_ONLY → teal+ // -----------------------------------------------------------+ await run('5. PRIVATE_ONLY paints teal', async () => {+ const bodyScope = await selectScope(page, 'private_only');+ record('body-scope-private', bodyScope === 'private_only');++ const cardColor = await readCardBorderColor(page);+ record(+ 'card-color-private',+ cardColor === SCOPE_PALETTE.private_only.border,+ `border-left-color=${cardColor}`,+ );++ const queryColor = await readQueryBorderColor(page);+ record(+ 'query-border-private',+ queryColor === SCOPE_PALETTE.private_only.border,+ `border-color=${queryColor}`,+ );++ await screenshot(page, '04-scope-private-only');+ });++ // -----------------------------------------------------------+ // 6. Switch to STRICT (with a concrete engine first so it's+ // selectable) → violet+ // -----------------------------------------------------------+ await run('6. STRICT paints violet when engine is concrete', async () => {+ // Pin the engine to "wikipedia" so STRICT is allowed. The+ // hidden field is what the guard inspects (see+ // updateStrictScopeAvailability in research.js).+ await setSearchEngine(page, 'wikipedia');+ const stillDisabled = await strictOptionDisabled(page);+ record(+ 'strict-enabled-on-concrete',+ stillDisabled === false,+ `option.disabled=${stillDisabled}`,+ );++ const bodyScope = await selectScope(page, 'strict');+ record('body-scope-strict', bodyScope === 'strict');++ const cardColor = await readCardBorderColor(page);+ record(+ 'card-color-strict',+ cardColor === SCOPE_PALETTE.strict.border,+ `border-left-color=${cardColor}`,+ );++ await screenshot(page, '05-scope-strict');+ });++ // -----------------------------------------------------------+ // 7. STRICT + meta-picker guard: switching engine to "auto"+ // must disable the STRICT option AND fall back to "both"+ // if STRICT was selected.+ // -----------------------------------------------------------+ await run('7. STRICT + meta-picker guard (client disables, server rejects)', async () => {+ // Persist a COHERENT combo first: the scope must not be STRICT+ // before we persist a meta-picker engine, otherwise the server's+ // cross-field validator (validate_strict_meta_combo) correctly+ // 400s the incoherent STRICT+meta-picker save — which is exactly+ // the server-side guarantee asserted at the end of this step. We+ // drive the save endpoints directly (not the custom-select UI+ // dance), so set scope=both, then engine=auto, then reload to+ // exercise the init-time guard.+ await selectScope(page, 'both');+ await setSearchEngine(page, 'auto');++ // Layer 1 — client guard (init path): the STRICT option is+ // disabled while the engine is a meta-picker, so the user can't+ // pick the incoherent combo in the first place.+ const disabled = await strictOptionDisabled(page);+ record(+ 'strict-disabled-on-auto',+ disabled === true,+ `option.disabled=${disabled}`,+ );++ // The scope select must not be sitting on STRICT now that the+ // engine is a meta-picker.+ const scopeNow = await page.$eval(+ '#policy_egress_scope',+ (el) => el.value,+ );+ record(+ 'scope-not-strict-under-meta-picker',+ scopeNow !== 'strict',+ `scope=${scopeNow}`,+ );++ // Layer 2 — server guard: even if the client guard were bypassed,+ // persisting STRICT while the engine is a meta-picker is refused+ // (HTTP 400, fail-closed). This is the real defense.+ const strictSaveStatus = await page.evaluate(async (base) => {+ const csrf =+ document+ .querySelector('meta[name="csrf-token"]')+ ?.getAttribute('content') || '';+ const resp = await fetch(+ `${base}/settings/api/policy.egress_scope`,+ {+ method: 'PUT',+ headers: {+ 'Content-Type': 'application/json',+ 'X-CSRFToken': csrf,+ },+ credentials: 'same-origin',+ body: JSON.stringify({ value: 'strict' }),+ },+ );+ return resp.status;+ }, BASE_URL);+ record(+ 'server-rejects-strict-meta-combo',+ strictSaveStatus === 400,+ `status=${strictSaveStatus}`,+ );++ await screenshot(page, '06-strict-disabled-on-auto');+ });++ // -----------------------------------------------------------+ // 8. Require-local toggles persist to settings DB+ // -----------------------------------------------------------+ await run('8. Local-inference checkboxes toggle + persist', async () => {+ const llmInitial = await page.$eval('#llm_require_local_endpoint', (el) => el.checked);+ await page.click('#llm_require_local_endpoint');+ await sleep(IS_CI ? 500 : 300);+ const llmAfter = await page.$eval('#llm_require_local_endpoint', (el) => el.checked);+ record(+ 'llm-toggle-flips',+ llmAfter !== llmInitial,+ `${llmInitial} → ${llmAfter}`,+ );++ const embInitial = await page.$eval('#embeddings_require_local', (el) => el.checked);+ await page.click('#embeddings_require_local');+ await sleep(IS_CI ? 500 : 300);+ const embAfter = await page.$eval('#embeddings_require_local', (el) => el.checked);+ record(+ 'emb-toggle-flips',+ embAfter !== embInitial,+ `${embInitial} → ${embAfter}`,+ );++ await screenshot(page, '07-toggles-flipped');++ // Reload — toggles must persist (proves settings save fired).+ await page.reload({ waitUntil: 'domcontentloaded' });+ await page.waitForSelector('#llm_require_local_endpoint', { timeout: 10000 });+ const llmReloaded = await page.$eval('#llm_require_local_endpoint', (el) => el.checked);+ const embReloaded = await page.$eval('#embeddings_require_local', (el) => el.checked);+ record(+ 'llm-persists-reload',+ llmReloaded === llmAfter,+ `reloaded=${llmReloaded}`,+ );+ record(+ 'emb-persists-reload',+ embReloaded === embAfter,+ `reloaded=${embReloaded}`,+ );++ await screenshot(page, '08-toggles-after-reload');++ // Restore to original to leave the env clean for repeat runs.+ if (llmAfter !== llmInitial) await page.click('#llm_require_local_endpoint');+ if (embAfter !== embInitial) await page.click('#embeddings_require_local');+ await sleep(IS_CI ? 500 : 300);+ });++ // -----------------------------------------------------------+ // 9. Settings dashboard exposes the egress-policy keys+ // -----------------------------------------------------------+ await run('9. Settings dashboard lists policy keys', async () => {+ await page.goto(`${BASE_URL}/settings/`, {+ waitUntil: 'domcontentloaded',+ timeout: 30000,+ });+ await sleep(2000); // wait for JS-rendered settings list++ const html = await page.content();+ const haveScope = html.includes('policy.egress_scope') || html.includes('policy_egress_scope');+ const haveLocalLLM = html.includes('llm.require_local_endpoint') || html.includes('require_local_endpoint');+ record(+ 'settings-has-scope-key',+ haveScope,+ haveScope ? 'found' : 'missing',+ );+ record(+ 'settings-has-local-llm',+ haveLocalLLM,+ haveLocalLLM ? 'found' : 'missing',+ );++ await screenshot(page, '09-settings-page');+ });++ // -----------------------------------------------------------+ // 10. base.html scope cue propagates to OTHER pages+ // -----------------------------------------------------------+ await run('10. Scope cue carries to history & metrics pages', async () => {+ // Set a memorable scope first on the research page.+ await page.goto(`${BASE_URL}/`, {+ waitUntil: 'domcontentloaded',+ timeout: 30000,+ });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });+ // Make sure the search engine is concrete so STRICT is allowed+ // — otherwise selectScope('strict') falls back to "both" via+ // the guard and our cross-page check would assert the wrong thing.+ await setSearchEngine(page, 'wikipedia');+ await selectScope(page, 'private_only');++ const pages = [+ { name: 'History', path: '/history/' },+ { name: 'Metrics', path: '/metrics/' },+ ];+ for (const p of pages) {+ await page.goto(`${BASE_URL}${p.path}`, {+ waitUntil: 'domcontentloaded',+ timeout: 30000,+ });+ await sleep(500);+ const bodyScope = await page.evaluate(+ () => document.body.dataset.scope,+ );+ record(+ `${p.name.toLowerCase()}-inherits-scope`,+ bodyScope === 'private_only',+ `data-scope=${bodyScope}`,+ );+ await screenshot(page, `10-cross-page-${p.name.toLowerCase()}`);+ }++ // Reset to "both" so a follow-up test run is deterministic.+ await page.goto(`${BASE_URL}/`, {+ waitUntil: 'domcontentloaded',+ timeout: 30000,+ });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });+ await selectScope(page, 'both');+ });++ // -----------------------------------------------------------+ // 11. Warning banner appears/disappears with scope+ // -----------------------------------------------------------+ await run('11. Public-egress warning banner reflects scope', async () => {+ // Force scope=both, ack=false → banner SHOULD show. We don't+ // have a clean handle for the ack flag from JS so we+ // approximate by looking for the icon + text. This test is+ // a smoke-check, not a contract test.+ await page.goto(`${BASE_URL}/`, {+ waitUntil: 'domcontentloaded',+ timeout: 30000,+ });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });+ await selectScope(page, 'both');++ const bannerText = await page.evaluate(() => document.body.innerText);+ const hasEgressBanner =+ /Public search egress/i.test(bannerText)+ || /egress/i.test(bannerText);+ record(+ 'egress-banner-present-on-both',+ true, // soft assertion — banner may be acked on this user+ hasEgressBanner ? 'visible' : 'absent (likely already acked)',+ );++ await screenshot(page, '11-warning-banners-both');++ // Switch to private_only and verify the warning text is gone+ // for the public-egress banner (other banners may remain).+ await selectScope(page, 'private_only');+ await sleep(500);+ await screenshot(page, '11-warning-banners-private');+ record('warning-screenshot-private', true);+ });++ // -----------------------------------------------------------+ // 12. Privacy panel header icon adopts the scope cue+ // -----------------------------------------------------------+ await run('12. Privacy panel header reflects scope colour', async () => {+ await selectScope(page, 'private_only');+ const headerColor = await page.evaluate(() => {+ const el = document.querySelector('.ldr-privacy-panel-header');+ if (!el) return null;+ return window.getComputedStyle(el).color;+ });+ record(+ 'header-color-set',+ !!headerColor && headerColor !== 'rgb(0, 0, 0)',+ `header.color=${headerColor}`,+ );+ await screenshot(page, '12-panel-header-private');+ });++ // -----------------------------------------------------------+ // 13. Tooltip explains each scope+ // -----------------------------------------------------------+ await run('13. Scope dropdown carries an explanatory tooltip', async () => {+ const tipText = await page.evaluate(() => {+ const lbl = document.querySelector('label[for="policy_egress_scope"]');+ if (!lbl) return null;+ return lbl.innerText || lbl.textContent;+ });+ const informative = !!tipText && tipText.length > 10;+ record(+ 'tooltip-label-present',+ informative,+ `label="${(tipText || '').slice(0, 60)}…"`,+ );+ });++ // -----------------------------------------------------------+ // 14. Per-research overrides survive a form-state roundtrip+ // -----------------------------------------------------------+ await run('14. Per-research override values are read back by the form', async () => {+ // Set a deliberate combination, save by changing each+ // control, then reload and verify the form reflects them.+ await selectScope(page, 'public_only');+ const llmStart = await page.$eval('#llm_require_local_endpoint', (el) => el.checked);+ if (!llmStart) await page.click('#llm_require_local_endpoint');+ await sleep(IS_CI ? 500 : 300);++ await page.reload({ waitUntil: 'domcontentloaded' });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });++ const scopeReloaded = await page.$eval(+ '#policy_egress_scope',+ (el) => el.value,+ );+ const llmReloaded = await page.$eval(+ '#llm_require_local_endpoint',+ (el) => el.checked,+ );+ record(+ 'scope-roundtrips-via-DB',+ scopeReloaded === 'public_only',+ `value=${scopeReloaded}`,+ );+ record(+ 'require-local-llm-roundtrips',+ llmReloaded === true,+ `checked=${llmReloaded}`,+ );++ await screenshot(page, '14-roundtrip-public-only-require-local');++ // Clean up.+ if (llmReloaded) await page.click('#llm_require_local_endpoint');+ await selectScope(page, 'both');+ });++ // -----------------------------------------------------------+ // 15. Chat page inherits scope cue (only if chat is enabled)+ // -----------------------------------------------------------+ await run('15. Chat page inherits data-scope', async () => {+ await selectScope(page, 'strict');+ const chatRes = await page.goto(`${BASE_URL}/chat/`, {+ waitUntil: 'domcontentloaded',+ timeout: 30000,+ });+ if (!chatRes || chatRes.status() >= 400) {+ record(+ 'chat-page-reachable',+ false,+ `status=${chatRes ? chatRes.status() : 'no-response'}`,+ );+ return;+ }+ const bodyScope = await page.evaluate(+ () => document.body.dataset.scope,+ );+ record(+ 'chat-inherits-scope',+ bodyScope === 'strict',+ `data-scope=${bodyScope}`,+ );+ await screenshot(page, '15-chat-page-strict');++ // Reset.+ await page.goto(`${BASE_URL}/`, {+ waitUntil: 'domcontentloaded',+ timeout: 30000,+ });+ await page.waitForSelector('#policy_egress_scope', { timeout: 10000 });+ await selectScope(page, 'both');+ });++ } catch (err) {+ console.error('\n❌ Suite-level error:', err.stack || err);+ exitCode = 1;+ } finally {+ // Final summary+ console.log('\n' + '='.repeat(60));+ console.log('Results:');+ const passed = results.filter((r) => r.ok).length;+ const failed = results.filter((r) => !r.ok);+ console.log(` ${passed} / ${results.length} passed`);+ if (failed.length) {+ console.log(' Failures:');+ for (const f of failed) {+ console.log(` ❌ ${f.label}${f.detail ? ` — ${f.detail}` : ''}`);+ }+ exitCode = 1;+ }+ if (!IS_CI) {+ console.log(`\n Screenshots: ${SCREENSHOT_DIR}`);+ }+ await browser.close();+ process.exit(exitCode);+ }+}++main().catch((err) => {+ console.error(err);+ process.exit(1);+});diff --git a/tests/utilities/test_log_utils_extended.py b/tests/utilities/test_log_utils_extended.pyindex 12a4c973ed..d01e2c7385 100644--- a/tests/utilities/test_log_utils_extended.py+++ b/tests/utilities/test_log_utils_extended.py@@ -361,6 +361,49 @@ class TestFrontendProgressSinkExtended:call_kwargs = mock_instance.emit_to_subscribers.call_args[1]assert call_kwargs["enable_logging"] is False+ def test_policy_audit_lines_never_reach_websocket(self):+ """SECURITY.md guarantee: ``policy_audit=True`` log lines must NOT be+ forwarded to WebSocket subscribers, even when a research_id is bound.++ policy_audit lines carry engine names + reason codes that would leak+ the active egress scope to a cross-origin observer under CORS=*.+ Today these lines don't bind a research_id (so the research_id guard+ already skips them), but the SECURITY.md doc promises the filter holds+ even if a future call site binds BOTH — this pins that promise.+ """+ from local_deep_research.utilities.log_utils import (+ frontend_progress_sink,+ )+ import local_deep_research.utilities.log_utils as module++ mock_time = Mock()+ mock_time.isoformat.return_value = "2026-06-04T00:00:00"+ mock_level = Mock()+ mock_level.name = "WARNING"++ mock_message = Mock()+ mock_message.record = {+ "message": "engine denied by egress policy",+ "level": mock_level,+ "time": mock_time,+ # The dangerous case: a policy_audit line that ALSO carries a+ # research_id, so it survives the research_id guard and reaches+ # the policy_audit filter.+ "extra": {"policy_audit": True, "research_id": "audit-uuid"},+ }++ with patch.object(+ module, "_get_research_id", return_value="audit-uuid"+ ):+ with patch.object(module, "SocketIOService") as mock_socket_cls:+ mock_instance = Mock()+ mock_socket_cls.return_value = mock_instance++ frontend_progress_sink(mock_message)++ # Must be dropped before any WebSocket emit.+ mock_instance.emit_to_subscribers.assert_not_called()+class TestFlushLogQueueExtended:"""Extended tests for flush_log_queue."""diff --git a/tests/utilities/test_search_cache_high_value.py b/tests/utilities/test_search_cache_high_value.pyindex 74573251db..ddba248584 100644--- a/tests/utilities/test_search_cache_high_value.py+++ b/tests/utilities/test_search_cache_high_value.py@@ -117,11 +117,13 @@ class TestGetQueryHashHighValue:assert all(c in "0123456789abcdef" for c in result)def test_hash_matches_manual_sha256(self):- """Hash matches manually computed SHA-256 of 'username:engine:normalized_query'."""+ """Hash matches manually computed SHA-256 of+ 'username:scope:engine:normalized_query'. The scope component+ (N17) defaults to 'both' when no policy snapshot is in context."""cache = _make_cache()result = cache._get_query_hash("Hello World", search_engine="google")- # No search context set, so username is empty string- expected_input = ":google:hello world"+ # No search context set: username empty, scope defaults to "both".+ expected_input = ":both:google:hello world"expected = hashlib.sha256(expected_input.encode()).hexdigest()assert result == expected@@ -129,7 +131,7 @@ class TestGetQueryHashHighValue:"""Default search_engine parameter is 'default'."""cache = _make_cache()result = cache._get_query_hash("test")- expected_input = ":default:test"+ expected_input = ":both:default:test"expected = hashlib.sha256(expected_input.encode()).hexdigest()assert result == expecteddiff --git a/tests/web/routes/conftest.py b/tests/web/routes/conftest.pynew file mode 100644index 0000000000..70accd09f7--- /dev/null+++ b/tests/web/routes/conftest.py@@ -0,0 +1,34 @@+"""Shared fixtures for web route tests.++The Flask-Limiter ``limiter`` is a module-level singleton whose in-memory+storage is attached once at ``create_app()`` time and never reset between+tests. Without intervention the per-scope request counters (notably the+shared ``settings`` scope, "30 per minute") accumulate across every test+in the process that hits a rate-limited route — so unrelated tests that+make their 31st settings request in a 60-second window get an unexpected+HTTP 429 instead of the status they assert. The failing set is+non-deterministic because it depends on how many sibling requests landed+in the trailing window.++Resetting the storage before each test gives every test a clean bucket+without disabling enforcement, so rate-limiting tests+(test_settings_routes_rate_limiting.py) still exercise real limits from a+known-clean starting point.+"""++import pytest++[email protected](autouse=True)+def _reset_rate_limiter_storage():+ """Clear the global limiter's storage before each route test."""+ try:+ from local_deep_research.security.rate_limiter import limiter++ storage = getattr(limiter, "_storage", None)+ if storage is not None and hasattr(storage, "reset"):+ storage.reset()+ except Exception:+ # Limiter not importable / no storage yet → nothing to reset.+ pass+ yielddiff --git a/tests/web/routes/test_metrics_routes.py b/tests/web/routes/test_metrics_routes.pyindex 0e0e8f0edb..fe08bdfc65 100644--- a/tests/web/routes/test_metrics_routes.py+++ b/tests/web/routes/test_metrics_routes.py@@ -1126,6 +1126,53 @@ class TestApiJournalDataDownload:# Structured counts reach the user-facing message.assert "42" in data["message"]+ def test_refused_under_private_only_egress_scope(+ self, authenticated_client+ ):+ """Under an offline-for-public scope (PRIVATE_ONLY) the manual+ journal-data download must be refused (403) BEFORE any public HTTP+ fetch — the user opted out of public egress. The downloader is+ intentionally NOT mocked: if the gate works it is never reached.+ """+ mgr = MagicMock()+ mgr.get_setting.side_effect = lambda key, default=None: (+ "private_only" if key == "policy.egress_scope" else default+ )+ with patch(+ "local_deep_research.utilities.db_utils.get_settings_manager",+ return_value=mgr,+ ):+ response = authenticated_client.post(+ f"{METRICS_PREFIX}/api/journal-data/download",+ json={"force": False},+ )+ if response.status_code in (302, 400, 429):+ pytest.skip(+ f"Route gated by CSRF/rate-limit (status={response.status_code})"+ )+ assert response.status_code == 403, response.status_code+ assert response.get_json()["success"] is False++ def test_refused_under_strict_egress_scope(self, authenticated_client):+ """STRICT is likewise an offline-for-public scope here."""+ mgr = MagicMock()+ mgr.get_setting.side_effect = lambda key, default=None: (+ "strict" if key == "policy.egress_scope" else default+ )+ with patch(+ "local_deep_research.utilities.db_utils.get_settings_manager",+ return_value=mgr,+ ):+ response = authenticated_client.post(+ f"{METRICS_PREFIX}/api/journal-data/download",+ json={"force": False},+ )+ if response.status_code in (302, 400, 429):+ pytest.skip(+ f"Route gated by CSRF/rate-limit (status={response.status_code})"+ )+ assert response.status_code == 403, response.status_code+def test_response_up_to_date_message_when_counts_none(self, authenticated_client):diff --git a/tests/web/routes/test_research_routes_extracted_helpers.py b/tests/web/routes/test_research_routes_extracted_helpers.pyindex 7ca3d503aa..aee465ac84 100644--- a/tests/web/routes/test_research_routes_extracted_helpers.py+++ b/tests/web/routes/test_research_routes_extracted_helpers.py@@ -219,6 +219,10 @@ class TestExtractResearchParams:"iterations","questions_per_iteration","strategy",+ # Per-research egress policy overrides (N11).+ "policy_egress_scope",+ "llm_require_local_endpoint",+ "embeddings_require_local",}assert set(result.keys()) == expected_keysdiff --git a/tests/web/routes/test_research_routes_policy.py b/tests/web/routes/test_research_routes_policy.pynew file mode 100644index 0000000000..d02371a896--- /dev/null+++ b/tests/web/routes/test_research_routes_policy.py@@ -0,0 +1,121 @@+"""Coverage for the request-boundary egress precheck in research_routes+(flagged untested by the PR #4300 review).++Targets:+- _apply_policy_overrides: overlays per-research form overrides onto the+ snapshot (pure dict logic).+- _precheck_engine_policy: rejects a forbidden engine / corrupt scope /+ STRICT+meta at /api/start_research with a 400, or returns None to+ continue. Needs a Flask app context for jsonify.+"""++from unittest.mock import Mock++import pytest+from flask import Flask++from local_deep_research.web.routes.research_routes import (+ _apply_policy_overrides,+ _precheck_engine_policy,+)+++# ---------------------------------------------------------------------------+# _apply_policy_overrides (pure)+# ---------------------------------------------------------------------------+++class TestApplyPolicyOverrides:+ def test_scope_override_applied(self):+ snap = {"policy.egress_scope": "both"}+ _apply_policy_overrides(snap, {"policy_egress_scope": "private_only"})+ assert snap["policy.egress_scope"] == "private_only"++ def test_bool_overrides_coerced(self):+ snap = {}+ _apply_policy_overrides(+ snap,+ {+ "llm_require_local_endpoint": "1",+ "embeddings_require_local": "",+ },+ )+ assert snap["llm.require_local_endpoint"] is True+ # Empty string is falsy => coerced to False.+ assert snap["embeddings.require_local"] is False++ def test_absent_params_leave_snapshot_untouched(self):+ snap = {"policy.egress_scope": "both"}+ _apply_policy_overrides(snap, {})+ assert snap == {"policy.egress_scope": "both"}++ def test_non_dict_snapshot_is_noop(self):+ # Must not raise on a non-dict snapshot.+ assert (+ _apply_policy_overrides(None, {"policy_egress_scope": "x"}) is None+ )+++# ---------------------------------------------------------------------------+# _precheck_engine_policy (needs app context for jsonify)+# ---------------------------------------------------------------------------+++def _mgr(snapshot, primary="arxiv"):+ m = Mock()+ m.get_settings_snapshot.return_value = snapshot+ m.get_setting.side_effect = lambda key, default=None: (+ primary if key == "search.tool" else default+ )+ return m+++def app_ctx():+ app = Flask(__name__)+ with app.app_context():+ yield+++class TestPrecheckEnginePolicy:+ def test_allowed_engine_returns_none(self, app_ctx):+ # PUBLIC_ONLY + a public engine (arxiv) => allowed => continue.+ mgr = _mgr({"policy.egress_scope": "public_only"})+ result = _precheck_engine_policy(mgr, {}, "arxiv", "user")+ assert result is None++ def test_forbidden_engine_returns_400(self, app_ctx):+ # PUBLIC_ONLY + a local engine (library) => refused => 400.+ mgr = _mgr({"policy.egress_scope": "public_only"})+ result = _precheck_engine_policy(mgr, {}, "library", "user")+ assert result is not None+ _resp, status = result+ assert status == 400++ def test_corrupt_scope_returns_400(self, app_ctx):+ mgr = _mgr({"policy.egress_scope": "garbage"})+ result = _precheck_engine_policy(mgr, {}, "arxiv", "user")+ assert result is not None+ assert result[1] == 400++ def test_strict_meta_value_error_returns_400(self, app_ctx):+ # STRICT + meta-picker primary => ValueError => surfaced as 400+ # (regression: previously fell through to None and started the run).+ mgr = _mgr({"policy.egress_scope": "strict"}, primary="meta")+ result = _precheck_engine_policy(mgr, {}, "meta", "user")+ assert result is not None+ assert result[1] == 400++ def test_non_dict_snapshot_returns_none(self, app_ctx):+ # No real snapshot => skip precheck (factory PEP backstops).+ mgr = _mgr(None)+ assert _precheck_engine_policy(mgr, {}, "library", "user") is None++ def test_per_research_override_tightens_scope(self, app_ctx):+ # Saved scope is permissive (both) but the form overrides to+ # public_only for THIS run => a local engine must be refused.+ mgr = _mgr({"policy.egress_scope": "both"})+ params = {"policy_egress_scope": "public_only"}+ result = _precheck_engine_policy(mgr, params, "library", "user")+ assert result is not None+ assert result[1] == 400diff --git a/tests/web/routes/test_settings_policy_validators.py b/tests/web/routes/test_settings_policy_validators.pynew file mode 100644index 0000000000..5fd2ee5281--- /dev/null+++ b/tests/web/routes/test_settings_policy_validators.py@@ -0,0 +1,75 @@+"""Regression tests for the egress-policy validators in settings_routes.++Covers validate_allowed_local_hostnames, which rejects public hostnames+from being whitelisted as "local". The validator was previously dead code:+it built its probe context via ``EgressContext.__new__`` + attribute+assignment, which raises ``FrozenInstanceError`` on the frozen dataclass+(and also set a non-existent ``allowed_local_hostnames`` field instead of+the real ``local_hostnames``). The guard therefore never ran — every save+that touched ``llm.allowed_local_hostnames`` either crashed or, if the+exception were swallowed, silently accepted public hostnames.++Source: src/local_deep_research/web/routes/settings_routes.py+"""++from local_deep_research.security.egress.validators import (+ validate_allowed_local_hostnames,+)++KEY = "llm.allowed_local_hostnames"+++class TestValidateAllowedLocalHostnames:+ def test_missing_key_returns_none(self):+ """No hostnames in the payload => nothing to validate."""+ assert validate_allowed_local_hostnames({}, {}) is None++ def test_does_not_raise_frozeninstanceerror(self):+ """The core regression: building the probe context must not crash.++ Before the fix this raised dataclasses.FrozenInstanceError. The+ whole point is that the function returns a value (None or an error+ dict) rather than throwing.+ """+ # Should simply return (not raise) for any well-formed input.+ result = validate_allowed_local_hostnames({KEY: '["127.0.0.1"]'}, {})+ assert result is None or isinstance(result, dict)++ def test_local_hostnames_accepted(self):+ """Private/loopback addresses classify as local => accepted."""+ result = validate_allowed_local_hostnames(+ {KEY: '["192.168.1.50", "127.0.0.1", "10.0.0.1"]'}, {}+ )+ assert result is None++ def test_public_hostname_rejected(self):+ """A public IP must be rejected with an error dict naming the key."""+ result = validate_allowed_local_hostnames({KEY: '["8.8.8.8"]'}, {})+ assert isinstance(result, dict)+ assert result["key"] == KEY+ assert "PUBLIC" in result["error"]+ assert "8.8.8.8" in result["error"]++ def test_invalid_json_returns_error(self):+ result = validate_allowed_local_hostnames({KEY: "not-valid-json["}, {})+ assert isinstance(result, dict)+ assert result["key"] == KEY++ def test_non_list_value_returns_error(self):+ result = validate_allowed_local_hostnames({KEY: '"single"'}, {})+ assert isinstance(result, dict)+ assert "list" in result["error"].lower()++ def test_empty_and_whitespace_entries_skipped(self):+ """Blank entries are ignored, not classified — so a list of only+ blanks is accepted."""+ result = validate_allowed_local_hostnames(+ {KEY: '["", " ", "127.0.0.1"]'}, {}+ )+ assert result is None++ def test_list_value_passed_directly(self):+ """The save pipeline may hand a Python list (already decoded)+ rather than a JSON string — both shapes must work."""+ result = validate_allowed_local_hostnames({KEY: ["192.168.1.1"]}, {})+ assert result is Nonediff --git a/tests/web/services/test_research_service_core.py b/tests/web/services/test_research_service_core.pyindex 9582257f4d..4691187b4a 100644--- a/tests/web/services/test_research_service_core.py+++ b/tests/web/services/test_research_service_core.py@@ -319,8 +319,11 @@ class TestLLMInstantiation:"llm.provider": "invalid_provider",}.get(key, default)+ # A (default-scope) snapshot is required so the egress-policy PEP —+ # which fails closed for snapshot-less non-local providers — lets the+ # call reach the provider-name validation this test exercises.with pytest.raises(ValueError) as exc_info:- get_llm(provider="invalid_provider")+ get_llm(provider="invalid_provider", settings_snapshot={})assert "Invalid provider" in str(exc_info.value)diff --git a/tests/web/warning_checks/test_orchestrator.py b/tests/web/warning_checks/test_orchestrator.pyindex 408c62cc48..73f58007eb 100644--- a/tests/web/warning_checks/test_orchestrator.py+++ b/tests/web/warning_checks/test_orchestrator.py@@ -290,6 +290,10 @@ class TestCalculateWarningsMultipleWarnings:"context_below_history","context_truncation_history","no_backups",+ # The egress policy defaults to scope="both" (unacknowledged),+ # so the public-egress banner correctly fires alongside the+ # hardware/context warnings.+ "public_egress_enabled",}diff --git a/tests/web_search_engines/engines/test_embedding_manager_coverage.py b/tests/web_search_engines/engines/test_embedding_manager_coverage.pyindex 0b20a0bd4a..3acf76564d 100644--- a/tests/web_search_engines/engines/test_embedding_manager_coverage.py+++ b/tests/web_search_engines/engines/test_embedding_manager_coverage.py@@ -213,22 +213,81 @@ class TestInitializeEmbeddings:_, kwargs = mock_get.call_argsassert "base_url" not in kwargs- def test_fallback_on_exception(self):- """When get_embeddings raises, should fall back to HuggingFaceEmbeddings."""+ def test_fallback_only_on_importerror(self):+ """Fallback ONLY on ImportError — other exceptions (RuntimeError,+ ConnectionError, PolicyDeniedError) must propagate so we don't+ silently fetch from huggingface.co when the user has opted into+ local embeddings. The fallback itself routes through the GATED+ get_embeddings(sentence_transformers) path (not a raw+ HuggingFaceEmbeddings construction), so PRIVATE_ONLY /+ embeddings.require_local still blocks an uncached HF download.+ See plan Wave D + the egress-policy review.+ """+ sentinel = Mock()+ calls = []++ def fake_get_embeddings(provider=None, **kwargs):+ calls.append(provider)+ # First call (configured provider) fails with ImportError; the+ # fallback re-invokes for sentence_transformers, which succeeds.+ if len(calls) == 1:+ raise ImportError("missing provider deps")+ return sentinel+with patch("local_deep_research.embeddings.get_embeddings",- side_effect=RuntimeError("boom"),+ side_effect=fake_get_embeddings,):- with patch(- "local_deep_research.web_search_engines.engines.local_embedding_manager.HuggingFaceEmbeddings"- ) as mock_hf:- mock_hf.return_value = Mock()- mgr = LocalEmbeddingManager()- result = mgr._initialize_embeddings()- mock_hf.assert_called_once_with(- model_name="sentence-transformers/all-MiniLM-L6-v2"+ mgr = LocalEmbeddingManager()+ result = mgr._initialize_embeddings()++ # Routed through the gated provider, not a raw HF build.+ assert calls[-1] == "sentence_transformers"+ assert result is sentinel++ def test_fallback_propagates_policy_denial(self):+ """A PolicyDeniedError from the gated SBERT fallback must propagate —+ the fallback must NOT silently download from huggingface.co under+ PRIVATE_ONLY / embeddings.require_local.+ """+ from local_deep_research.security.egress.policy import (+ Decision,+ PolicyDeniedError,+ )++ def fake_get_embeddings(provider=None, **kwargs):+ if provider == "sentence_transformers":+ raise PolicyDeniedError(+ Decision(False, "embeddings_model_not_cached"),+ target="all-MiniLM-L6-v2",)- assert result is mock_hf.return_value+ raise ImportError("missing provider deps")++ with patch(+ "local_deep_research.embeddings.get_embeddings",+ side_effect=fake_get_embeddings,+ ):+ mgr = LocalEmbeddingManager()+ with pytest.raises(PolicyDeniedError):+ mgr._initialize_embeddings()++ def test_runtime_error_propagates(self):+ """RuntimeError must propagate — no silent HF fetch, no fallback."""+ calls = []++ def fake_get_embeddings(provider=None, **kwargs):+ calls.append(provider)+ raise RuntimeError("boom")++ with patch(+ "local_deep_research.embeddings.get_embeddings",+ side_effect=fake_get_embeddings,+ ):+ mgr = LocalEmbeddingManager()+ with pytest.raises(RuntimeError, match="boom"):+ mgr._initialize_embeddings()+ # No fallback attempt: only the original configured-provider call.+ assert len(calls) == 1# ---------------------------------------------------------------------------diff --git a/tests/web_search_engines/test_search_engine_factory.py b/tests/web_search_engines/test_search_engine_factory.pyindex bf212b373b..81ebe0893b 100644--- a/tests/web_search_engines/test_search_engine_factory.py+++ b/tests/web_search_engines/test_search_engine_factory.py@@ -101,6 +101,28 @@ class TestCreateSearchEngineRetriever:assert call_kwargs["name"] == "custom_retriever"[email protected](autouse=True)+def _module_wide_bypass_engine_pdp():+ """Auto-applied to every test in this module — bypasses the PEP so+ factory tests using mock engine names don't get rejected by the+ egress policy. The PEP itself is exercised in+ tests/security/test_egress_policy.py.+ """+ from local_deep_research.security.egress.policy import Decision++ with (+ patch(+ "local_deep_research.security.egress.policy.evaluate_engine",+ return_value=Decision(True, "test_bypass"),+ ),+ patch(+ "local_deep_research.security.egress.policy.evaluate_retriever",+ return_value=Decision(True, "test_bypass"),+ ),+ ):+ yield++class TestCreateSearchEngineRequirements:"""Tests for API key and LLM requirements."""@@ -542,9 +564,30 @@ def _make_engine_class(*param_names):return _Eng+def _bypass_engine_pdp():+ """Patch evaluate_engine to allow unknown mock engine names through.+ The PEP is exercised separately in tests/security/test_egress_policy.py;+ factory tests below use mock names (myengine, eng) that aren't in the+ static ENGINE_REGISTRY and would otherwise be rejected as engine_unknown.+ """+ from local_deep_research.security.egress.policy import Decision++ with patch(+ "local_deep_research.security.egress.policy.evaluate_engine",+ return_value=Decision(True, "test_bypass"),+ ):+ yield++class TestCreateEngineInstantiation:"""Tests for the real engine instantiation path (lines 194-339)."""+ @pytest.fixture(autouse=True)+ def _auto_bypass(self, _bypass_engine_pdp):+ """Auto-apply the PEP bypass to every test in this class."""+ yield+def _make_engine_config(self, **overrides):"""Build a minimal engine config dict."""base = {@@ -727,8 +770,8 @@ class TestCreateEngineInstantiation:result = create_search_engine("eng", settings_snapshot={"x": 1})assert result is None- def test_unknown_engine_falls_back_to_auto(self):- """Unknown engine name falls back to 'auto' config."""+ def test_unknown_engine_raises_value_error(self):+ """Unknown engine name FAILS CLOSED (was silent rewrite to 'auto' — see plan C2)."""from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)@@ -750,13 +793,11 @@ class TestCreateEngineInstantiation:mock_sc.return_value = {"auto": self._make_engine_config()}mock_gsmc.return_value = EngCls- result = create_search_engine(- "nonexistent", settings_snapshot={"x": 1}- )- assert result is not None+ with pytest.raises(ValueError, match="Unknown search engine"):+ create_search_engine("nonexistent", settings_snapshot={"x": 1})- def test_no_config_returns_none(self):- """Missing engine in config (and no 'auto') returns None."""+ def test_no_config_raises_value_error(self):+ """Missing engine in config (and no 'auto') raises ValueError (was silent None — plan C2)."""from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)@@ -772,10 +813,8 @@ class TestCreateEngineInstantiation:mock_reg.get.return_value = Nonemock_sc.return_value = {"other": {}}- result = create_search_engine(- "nonexistent", settings_snapshot={"x": 1}- )- assert result is None+ with pytest.raises(ValueError, match="Unknown search engine"):+ create_search_engine("nonexistent", settings_snapshot={"x": 1})def test_default_params_merged(self):"""default_params from config are merged with kwargs."""diff --git a/tests/web_search_engines/test_search_engine_factory_branch_coverage.py b/tests/web_search_engines/test_search_engine_factory_branch_coverage.pyindex 2aa10ae78d..6bb9fa09f3 100644--- a/tests/web_search_engines/test_search_engine_factory_branch_coverage.py+++ b/tests/web_search_engines/test_search_engine_factory_branch_coverage.py@@ -73,10 +73,20 @@ def _patches(registry_return=None, config_return=None, class_return=None):@contextlib.contextmanagerdef _ctx():+ # Mock evaluate_engine so factory branch-coverage tests using+ # fake engine names (not in ENGINE_REGISTRY) aren't rejected by+ # the egress PEP. The PEP is exercised in+ # tests/security/test_egress_policy.py.+ from local_deep_research.security.egress.policy import Decision+with (patch(f"{_FACTORY_MOD}.retriever_registry") as mock_reg,patch(f"{_FACTORY_MOD}.search_config") as mock_sc,patch(f"{_FACTORY_MOD}.get_safe_module_class") as mock_gsmc,+ patch(+ "local_deep_research.security.egress.policy.evaluate_engine",+ return_value=Decision(True, "test_bypass"),+ ),):mock_reg.get.return_value = registry_returnif config_return is not None:@@ -98,24 +108,21 @@ class TestDisplayLabelNoIconPrefix:extraction is skipped and the engine falls through to the 'auto' fallback.Covers the else branch at line 107 of search_engine_factory.py."""- def test_label_without_icon_prefix_falls_back_to_auto(self):+ def test_label_without_icon_prefix_raises_value_error(self):+ """Label with no icon prefix is still an unknown engine name → FAIL CLOSED.+ Plan C2 removed the silent rewrite to 'auto'."""from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)EngCls = _make_engine_class("max_results")- # Label has parenthesised category but no icon prefix:- # rsplit(" (", 1) produces ["NoIcon", "Cat)"],- # space_idx on "NoIcon" is -1 -> base_name extraction is skipped.config = {"auto": _engine_config()}with _patches(config_return=config, class_return=EngCls):- result = create_search_engine(- "NoIcon (Category)", settings_snapshot={"x": 1}- )-- # Should have fallen through to 'auto' and succeeded- assert result is not None+ with pytest.raises(ValueError, match="Unknown search engine"):+ create_search_engine(+ "NoIcon (Category)", settings_snapshot={"x": 1}+ )# ---------------------------------------------------------------------------diff --git a/tests/web_search_engines/test_search_engine_factory_coverage.py b/tests/web_search_engines/test_search_engine_factory_coverage.pyindex 230fdb1b52..9050af2c17 100644--- a/tests/web_search_engines/test_search_engine_factory_coverage.py+++ b/tests/web_search_engines/test_search_engine_factory_coverage.py@@ -80,6 +80,12 @@ def _patches(registry_return=None, config_return=None, class_return=None):@contextlib.contextmanagerdef _ctx():+ # Mock evaluate_engine so these factory tests (which use fake+ # engine names like "myeng") don't get rejected by the egress+ # PEP for not being in ENGINE_REGISTRY. The PEP is exercised+ # separately in tests/security/test_egress_policy.py.+ from local_deep_research.security.egress.policy import Decision+with (patch("local_deep_research.web_search_engines.search_engine_factory.retriever_registry"@@ -90,6 +96,10 @@ def _patches(registry_return=None, config_return=None, class_return=None):patch("local_deep_research.web_search_engines.search_engine_factory.get_safe_module_class") as mock_gsmc,+ patch(+ "local_deep_research.security.egress.policy.evaluate_engine",+ return_value=Decision(True, "test_bypass"),+ ),):mock_reg.get.return_value = registry_returnif config_return is not None:@@ -443,8 +453,9 @@ class TestDisplayLabelFallback:assert result is not None- def test_unresolvable_label_falls_back_to_auto(self):- """Display label that doesn't match any config falls back to 'auto'."""+ def test_unresolvable_label_raises_value_error(self):+ """Display label that doesn't match any config FAILS CLOSED.+ Previously fell back to 'auto' silently — see plan C2."""from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)@@ -458,15 +469,15 @@ class TestDisplayLabelFallback:},class_return=EngCls,):- result = create_search_engine(- "\U0001f50d Unknown (Category)",- settings_snapshot={"x": 1},- )-- assert result is not None+ with pytest.raises(ValueError, match="Unknown search engine"):+ create_search_engine(+ "\U0001f50d Unknown (Category)",+ settings_snapshot={"x": 1},+ )- def test_plain_unknown_engine_falls_back_to_auto(self):- """Plain unknown engine name (no label format) falls back to 'auto'."""+ def test_plain_unknown_engine_raises_value_error(self):+ """Plain unknown engine name FAILS CLOSED (no more silent rewrite to 'auto').+ See plan C2."""from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)@@ -477,12 +488,11 @@ class TestDisplayLabelFallback:config_return={"auto": _engine_config()},class_return=EngCls,):- result = create_search_engine(- "nonexistent_plain_name",- settings_snapshot={"x": 1},- )-- assert result is not None+ with pytest.raises(ValueError, match="Unknown search engine"):+ create_search_engine(+ "nonexistent_plain_name",+ settings_snapshot={"x": 1},+ )# ---------------------------------------------------------------------------@@ -1292,11 +1302,27 @@ class TestProgrammaticModePostConstructionPatch:retriever = _DummyRetriever()- with patch(- "local_deep_research.web_search_engines.search_engine_factory.retriever_registry"- ) as mock_reg:+ # Retriever PEP now requires a snapshot (plan H3). Pass a+ # minimal snapshot + mock evaluate_retriever to allow the+ # registered retriever through so we can test programmatic_mode.+ from local_deep_research.security.egress.policy import Decision++ with (+ patch(+ "local_deep_research.web_search_engines.search_engine_factory.retriever_registry"+ ) as mock_reg,+ patch(+ "local_deep_research.security.egress.policy.evaluate_retriever",+ return_value=Decision(True, "test_bypass"),+ ),+ ):mock_reg.get.return_value = retriever- result = create_search_engine("my_rag", programmatic_mode=True)+ mock_reg.get_metadata.return_value = {"is_local": True}+ result = create_search_engine(+ "my_rag",+ programmatic_mode=True,+ settings_snapshot={"policy.egress_scope": "both"},+ )assert result is not Noneassert result.programmatic_mode is Truediff --git a/tests/web_search_engines/test_search_engine_factory_deep_coverage.py b/tests/web_search_engines/test_search_engine_factory_deep_coverage.pyindex aec9138980..b30f17b59f 100644--- a/tests/web_search_engines/test_search_engine_factory_deep_coverage.py+++ b/tests/web_search_engines/test_search_engine_factory_deep_coverage.py@@ -15,6 +15,8 @@ Focuses on gaps not addressed by test_search_engine_factory_coverage.py:import inspect as _inspectfrom unittest.mock import MagicMock, patch+import pytest+# ---------------------------------------------------------------------------# helpers@@ -144,13 +146,20 @@ class TestRegisteredRetrieverPath:# ----------------------------------------------------------------------------class TestUnknownEngineFallback:- def test_unknown_engine_falls_back_to_auto_when_available(self):+class TestUnknownEngineFailsClosed:+ """Plan C2: unknown engine_name FAILS CLOSED with a clear error.++ Previously the factory silently rewrote unknown names to 'auto',+ which then matched the local skip-list in the PEP block and+ bypassed evaluate_engine entirely. The fix raises ValueError on+ unknown names; tests now assert that contract.+ """++ def test_unknown_engine_raises_value_error_when_auto_available(self):from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)- EngClass = _make_engine_class("max_results")config = {"auto": {"module_path": "fake.module",@@ -161,24 +170,17 @@ class TestUnknownEngineFallback:}}- with (- patch(- "local_deep_research.web_search_engines.search_engine_factory.search_config",- return_value=config,- ),- patch(- "local_deep_research.web_search_engines.search_engine_factory.get_safe_module_class",- return_value=EngClass,- ),+ with patch(+ "local_deep_research.web_search_engines.search_engine_factory.search_config",+ return_value=config,):- result = create_search_engine(- "nonexistent_engine",- settings_snapshot=_make_snapshot(),- )- # Fell back to 'auto', engine created- assert result is not None+ with pytest.raises(ValueError, match="Unknown search engine"):+ create_search_engine(+ "nonexistent_engine",+ settings_snapshot=_make_snapshot(),+ )- def test_unknown_engine_without_auto_returns_none(self):+ def test_unknown_engine_raises_value_error_without_auto(self):from local_deep_research.web_search_engines.search_engine_factory import (create_search_engine,)@@ -197,11 +199,11 @@ class TestUnknownEngineFallback:"local_deep_research.web_search_engines.search_engine_factory.search_config",return_value=config,):- result = create_search_engine(- "totally_unknown",- settings_snapshot=_make_snapshot(),- )- assert result is None+ with pytest.raises(ValueError, match="Unknown search engine"):+ create_search_engine(+ "totally_unknown",+ settings_snapshot=_make_snapshot(),+ )# ---------------------------------------------------------------------------