Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

refactor(search): remove direct DuckDuckGo integration (#7310)

LearningCircuit8f2cfce760e3
diff --git a/changelog.d/7310.removal.md b/changelog.d/7310.removal.md
new file mode 100644
index 0000000000..fc4fe82a74
--- /dev/null
+++ b/changelog.d/7310.removal.md
@@ -0,0 +1 @@
+Remove LDR's dormant DuckDuckGo adapter and direct search dependency. Wayback no longer silently queries another provider to discover URLs. It keeps its lookup of the host of each lowercase `http://` or `https://` URL in a query, or of a query that is only a bare domain, and tells research agents to discover URLs through a configured provider first. Any other query now returns no results, including a dotted topic, a domain with a path but no scheme, a URL with an uppercase scheme or a bracketed IPv6 host, and a bare domain with a punycode top-level domain such as `.xn--p1ai` (give it as a full URL); Wayback used to look a dotted query up as a URL when discovery found no URLs. A URL with a user name or password before the host is not supported: the start of the user name is looked up instead of the host. SearXNG's upstream engine configuration remains available.
diff --git a/community_benchmark_results/benchmark_template.yaml b/community_benchmark_results/benchmark_template.yaml
index 11b21e758b..08700ac73e 100644
--- a/community_benchmark_results/benchmark_template.yaml
+++ b/community_benchmark_results/benchmark_template.yaml
@@ -8,7 +8,7 @@ model_provider: # ollama, llamacpp, openai_endpoint, etc.
quantization: # e.g., Q4_K_M, Q5_K_S, FP16, etc.
# Search Engine (critical for benchmark reproducibility)
-search_engine: # e.g., searxng, tavily, duckduckgo, google
+search_engine: # e.g., searxng, tavily, wikipedia, google_pse
search_provider_version: # if known, e.g., "latest", "2024.1.0"
average_results_per_query: # optional, e.g., 10
diff --git a/docs/architecture.md b/docs/architecture.md
index 67d81d1053..60714c99f6 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -44,7 +44,6 @@ flowchart TB
WEB_SEARCH[/"🌐 Web Search"\]
TAVILY[Tavily]
BRAVE[Brave]
- DDG[DuckDuckGo]
ACADEMIC[/"📚 Academic"\]
ARXIV[ArXiv]
PUBMED[PubMed]
@@ -78,7 +77,7 @@ flowchart TB
EXEC <--> OPENAI & ANTHROPIC & GEMINI & OPENROUTER
EXEC <--> SEARXNG & ELASTIC & LIBRARY
- EXEC <--> TAVILY & BRAVE & DDG
+ EXEC <--> TAVILY & BRAVE
EXEC <--> ARXIV & PUBMED & SEMANTIC
SYNTH --> SQLCIPHER
@@ -184,7 +183,6 @@ mindmap
SearXNG
Tavily
Brave
- DuckDuckGo
Academic
ArXiv
PubMed
diff --git a/docs/search-engines.md b/docs/search-engines.md
index 7e7e9b2a7f..32fb0c434b 100644
--- a/docs/search-engines.md
+++ b/docs/search-engines.md
@@ -75,12 +75,6 @@ result = quick_summary(
`LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true` — only one is needed — see
[SearXNG-Setup](SearXNG-Setup.md)
-#### DuckDuckGo
-- **Specialization**: Privacy-focused web search
-- **Best for**: General web queries without tracking
-- **Results**: Web pages, instant answers
-- **Rate Limit**: Strict - use SearXNG for better reliability
-
### Technical Search
#### GitHub
@@ -101,6 +95,23 @@ result = quick_summary(
- **Best for**: Finding deleted content, tracking changes over time
- **Results**: Archived web pages with timestamps
- **Rate Limit**: Moderate
+- **Input**: A query that contains full `http://` or `https://` URLs, or a
+ query that is only a bare domain, such as `example.com`. For these, only
+ the host is looked up; the port, path, query string and fragment are
+ ignored. Not looked up: a domain inside a sentence, a domain with a path
+ but without `http://` or `https://`, a URL with an uppercase scheme such
+ as `HTTPS://`, a URL with a bracketed IPv6 host, and a bare domain with a
+ punycode top-level domain such as `.xn--p1ai` (give it as a full
+ `http://` or `https://` URL instead). A URL with a user name or password
+ before the host is not supported: Wayback looks up the start of the user
+ name instead of the host, so give the plain URL. A host name is read only
+ as far as letters, digits, hyphens, underscores, dots and percent-escapes
+ run, so one written with other characters, such as the combining marks of
+ some scripts, is cut short (as a bare domain it is not recognised); use
+ its `xn--` form in a full URL. For a topic query, first find page URLs
+ with your configured search provider, such as SearXNG, then pass those
+ URLs to Wayback, which looks up their hosts. Wayback does not send
+ queries to another search provider.
### News Search
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
index 89f281b6ed..25f603a9e5 100644
--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -97,26 +97,21 @@ This guide covers common issues and their solutions.
## Search Engine Issues
-### DuckDuckGo Returning No Results
-
-**Symptoms:**
-- Empty search results
-- "No results found" consistently
-
-**Solutions:**
-
-1. **Rate limiting:** DuckDuckGo aggressively rate limits. Solutions:
- - Switch to SearXNG or another engine
- - Increase wait time in rate limiting settings
- - Use `rate_limiting.profile = conservative`
-
-2. **Check network:** Verify you can access DuckDuckGo directly
-
-3. **Try alternative engines:**
- ```python
- # In settings
- search.tool = "searxng" # or "brave", "tavily", etc.
- ```
+### Wayback Returning No Results
+
+Wayback looks up archived versions of the host of each `http://` or
+`https://` URL, or of a bare domain. It does not discover page URLs through
+another search provider. Search for the topic with your configured provider,
+such as SearXNG, and pass a resulting URL to Wayback, which looks up its
+host. An empty archive result can also mean that the archive has no capture
+of that host.
+
+Wayback also returns nothing for a URL with an uppercase scheme or a
+bracketed IPv6 host, and for a bare domain with a punycode top-level domain
+such as `.xn--p1ai`; give a punycode domain as a full `http://` or `https://`
+URL. A URL with a user name or password before the host is not supported:
+Wayback looks up the start of the user name instead of the host, so remove
+it.
### SearXNG Setup Issues
diff --git a/pdm.lock b/pdm.lock
index e03a2a0311..49a382c437 100644
--- a/pdm.lock
+++ b/pdm.lock
@@ -5,7 +5,7 @@
groups = ["default", "dev", "mcp"]
strategy = ["inherit_metadata"]
lock_version = "4.5.0"
-content_hash = "sha256:4fead3891d3376d1334013c74a30c63d9f084aa68bdba90cd7e80765a5cb768f"
+content_hash = "sha256:6c63ba0723772a8ca46b78c8a4dbe1ef828657edcf8c07cd259b27b9eeaba4a6"
[[metadata.targets]]
requires_python = ">=3.12,<3.15"
@@ -1277,22 +1277,6 @@ files = [
{file = "docstring_parser-0.18.0.tar.gz", hash = "sha256:292510982205c12b1248696f44959db3cdd1740237a968ea1e2e7a900eeb2015"},
]
-[[package]]
-name = "duckduckgo-search"
-version = "8.1.1"
-requires_python = ">=3.9"
-summary = "Search for words, documents, images, news, maps and text translation using the DuckDuckGo.com search engine."
-groups = ["default"]
-dependencies = [
- "click>=8.1.8",
- "lxml>=5.3.0",
- "primp>=0.15.0",
-]
-files = [
- {file = "duckduckgo_search-8.1.1-py3-none-any.whl", hash = "sha256:f48adbb06626ee05918f7e0cef3a45639e9939805c4fc179e68c48a12f1b5062"},
- {file = "duckduckgo_search-8.1.1.tar.gz", hash = "sha256:9da91c9eb26a17e016ea1da26235d40404b46b0565ea86d75a9f78cc9441f935"},
-]
-
[[package]]
name = "dynaconf"
version = "3.3.5"
@@ -4319,46 +4303,6 @@ files = [
{file = "pre_commit-4.6.2.tar.gz", hash = "sha256:8f5d7bfb021ecdbcd9d49d89847082dd24172ccde534390081a679ad046e2441"},
]
-[[package]]
-name = "primp"
-version = "2.0.1"
-requires_python = ">=3.10"
-summary = "HTTP client that can impersonate web browsers"
-groups = ["default"]
-files = [
- {file = "primp-2.0.1-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:4296ae43a8660bcb1dfc1570ed07dc9f8ab64f11fdebf3272532411b7fe321ef"},
- {file = "primp-2.0.1-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:ca21c764f17ba42dde38c29d6a1f01970d39fa5f4793b0fe702006bd71b7a16a"},
- {file = "primp-2.0.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:aa85a55b1c53ef8c2d14f1c61f0b7ab0ff300b349b2768a52d6c2f3f3f9ca80c"},
- {file = "primp-2.0.1-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:9edc6d2f2fd30d2ddf8c093bf533a3e99ae1385a1d2af15baaa299fb55310fb8"},
- {file = "primp-2.0.1-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:24d3ffeb054c23c7588d0240b5488d960f669398d1ab4cb3873158322bdf821d"},
- {file = "primp-2.0.1-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0beec080cb61044bd8b2eb0e3ca60f15f2df8ced9f2fbc6f1760855c20aed42a"},
- {file = "primp-2.0.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9a7be373adfded677a9092ae2743873d5c8a9573148d617a189d26715f7d8ea5"},
- {file = "primp-2.0.1-cp310-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:acc8b31f7fcc241b8ecb94069efea611f4f451d90f1798684c379d05e108b2c0"},
- {file = "primp-2.0.1-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:e5f8d170c69b4afbe61d854b3f0cf27a0c0e557f0e54ebe595dad4db0f72127d"},
- {file = "primp-2.0.1-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:706c843c86162d431c5a2051b8b41e16c7c51bca6bd60d139d7614609463a6c5"},
- {file = "primp-2.0.1-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:415aa6bb1b998ace5a456734df95755b5342b73fa0a6babbcb3ca471c83b9dbe"},
- {file = "primp-2.0.1-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:7f494686da2991212f5607d125c32cb8da72a92e179d497c7d2ffeba21abff1e"},
- {file = "primp-2.0.1-cp310-abi3-win32.whl", hash = "sha256:1cb429afd3a5ea98c625292f3c6581b0ccc0d398d3307603cce25ec140dfd671"},
- {file = "primp-2.0.1-cp310-abi3-win_amd64.whl", hash = "sha256:0e27f3e233cf34cae6cbc8158af58b93fb0a87ceb1802b4611c7a14a2d822cdc"},
- {file = "primp-2.0.1-cp310-abi3-win_arm64.whl", hash = "sha256:dea9370fcf6624725f564f9b9cf4fec3c127f86b7494196d03343a835fe3dee4"},
- {file = "primp-2.0.1-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:656ea4ff0d45bbd119394a6834a367e34192d75cb7a5945cfc2f1cbb266b1be4"},
- {file = "primp-2.0.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:31b3cd957f02dc49e5e9d9cbadd809747872caebfaf093dc34775a7f866a3e65"},
- {file = "primp-2.0.1-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0b8af35eb64d61291b105479245c89ed1231a5fff1e9b75870515892ffabf054"},
- {file = "primp-2.0.1-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4f659e6479073c4693195f0f43b7a96ae0afd353f6f057754f80be016eea6f20"},
- {file = "primp-2.0.1-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:9ba278a9af63981f2aece0d98fec9d6cc5a918943eb56e4f4df2b3ca90dab787"},
- {file = "primp-2.0.1-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:bedb42ea9188dd571db46d93f0b994d2e8e9271d2ddd55ee0ab7ac2844742bed"},
- {file = "primp-2.0.1-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:888ba0708e518acad84bfc0a8bb274ec7ae48655a95afdb9c8184bd88cc6d7ab"},
- {file = "primp-2.0.1-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:71fa07394c0084940d86c9f441bc2f05d7f8951a944bde315bebb6eec9b718d7"},
- {file = "primp-2.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c8f2b2eaacddf2bdff7b1b5f4219249d06246e577432fcb9b9babbc65146ff32"},
- {file = "primp-2.0.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:934696c74b8a88a7dbb40b3bb54a182b0c9782427f036035cb21bfc0cf7e24a9"},
- {file = "primp-2.0.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:e5f2e7b9fe557a440a929746a318074fd9989be318ce75411d01f1f3ed7bc85d"},
- {file = "primp-2.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:2b4970ab274deaa13224777fb52b8745523293c23566a6c44fa3fbd61e47c183"},
- {file = "primp-2.0.1-cp314-cp314t-win32.whl", hash = "sha256:0440d84854d1f9218277eef2c688a1774408e0d8a3805077eb6db432a4fa7970"},
- {file = "primp-2.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:f0806c7653ee05bbe3c7b28f97f19bd5d763d7da1366026d5cb85cd8713f5c33"},
- {file = "primp-2.0.1-cp314-cp314t-win_arm64.whl", hash = "sha256:a26651747b21efdff1ff986ee3e98ec3b349ce84b01b22226ce0b7a967041f01"},
- {file = "primp-2.0.1.tar.gz", hash = "sha256:82ba17b077bef19a189d9ec8d77ca632496cb444e0f4fa37e27e90041cf0da8f"},
-]
-
[[package]]
name = "propcache"
version = "0.5.4"
diff --git a/pyproject.toml b/pyproject.toml
index 2d613ea788..eb2c4de289 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -28,7 +28,6 @@ dependencies = [
"langchain-anthropic~=1.3", # lock pulls 1.4.6: GHSA-gr75-jv2w-4656 fix
"openai>=2", # direct dep: error_handling/openai_compat_errors.py imports exception classes
"httpx>=0.27", # direct dep: error_handling/openai_compat_errors.py imports ConnectError/ReadTimeout
- "duckduckgo-search~=8.1",
"python-dateutil~=2.9",
"typing-extensions~=4.15",
"justext~=3.0",
diff --git a/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py b/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py
index 5da1ed6874..16b58fd1e9 100644
--- a/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py
+++ b/src/local_deep_research/advanced_search_system/strategies/langgraph_agent_strategy.py
@@ -1345,7 +1345,6 @@ _TOOL_DISPLAY_NAMES = {
"search_searxng": "the web (SearXNG)",
"search_google_scholar": "Google Scholar",
"search_brave": "Brave Search",
- "search_duckduckgo": "DuckDuckGo",
"search_serper": "Google (Serper)",
"search_scaleserp": "Google (ScaleSERP)",
"search_wikipedia": "Wikipedia",
@@ -1436,6 +1435,20 @@ def _format_results(results: list[dict], start_idx: int) -> str:
# ``_make_fetch_progress_adapter`` for backwards compatibility with
# existing tests.
+# Appended to both Wayback tool descriptions. The engine looks up the host
+# of each lowercase http(s) URL in its input, or an input that is only a bare
+# domain; it does not discover URLs or pick a domain out of a sentence. The
+# "Not supported" sentence names the inputs its patterns miss or misread.
+_WAYBACK_INPUT_HINT = (
+ " Input must contain a full http:// or https:// URL, or be only a bare"
+ " domain such as example.com. Only the host is looked up, so a URL's"
+ " port, path, query and fragment are ignored. Not supported: an"
+ " uppercase scheme, a bracketed IPv6 host, a user name or password"
+ " before the host, and a bare domain with a punycode (xn--) top-level"
+ " domain. Use a configured search provider to discover URLs before"
+ " looking up their archives."
+)
+
def _make_web_search_tool(
search_engine_name: str,
@@ -1449,6 +1462,8 @@ def _make_web_search_tool(
):
"""Create a primary search tool with research-scoped availability."""
availability = availability or SearchEngineAvailability()
+ if search_engine_name == "wayback":
+ description += _WAYBACK_INPUT_HINT
@tool
def web_search(query: str) -> str:
@@ -1517,6 +1532,8 @@ def _make_specialized_search_tool(
):
"""Create a specialized search tool with research-scoped availability."""
availability = availability or SearchEngineAvailability()
+ if engine_name == "wayback":
+ description += _WAYBACK_INPUT_HINT
@tool
def specialized_search(query: str) -> str:
@@ -2462,9 +2479,8 @@ class LangGraphAgentStrategy(BaseSearchStrategy):
``semantic_scholar`` rather than the class-derived
``semanticscholar``). The class-name fallback historically returned
a lowercased, ``SearchEngine``-stripped variant that was almost
- but not quite canonical — e.g. ``DuckDuckGoSearchEngine`` →
- ``"duckduckgo"`` instead of the registry key ``"ddg"``, and
- ``SemanticScholarSearchEngine`` → ``"semanticscholar"`` instead of
+ but not quite canonical — e.g. ``SemanticScholarSearchEngine`` →
+ ``"semanticscholar"`` instead of
``"semantic_scholar"``. That mismatch leaked the configured primary
engine back into the specialised-tools loop as a redundant
``search_<engine>`` tool alongside ``web_search``. Reverse-lookup
@@ -2484,7 +2500,7 @@ class LangGraphAgentStrategy(BaseSearchStrategy):
return tool_setting
# Fall back to the engine CLASS, but resolve via the registry so
# we land on the canonical id (``semantic_scholar``,
- # ``ddg``, ...) rather than the class-derived heuristic. The
+ # ``google_pse``, ...) rather than the class-derived heuristic. The
# registry is the single source of truth for which Python class
# implements which canonical id.
if self.search is not None and hasattr(self.search, "__class__"):
@@ -2508,7 +2524,7 @@ class LangGraphAgentStrategy(BaseSearchStrategy):
``web_search`` is a generic wrapper around the user's configured
engine. Resolve it through the same curated ``_TOOL_DISPLAY_NAMES``
map as the specialized search tools (keyed by ``search_<engine>``)
- so the UI shows brand-correct names like "DuckDuckGo" or
+ so the UI shows brand-correct names like "Brave Search" or
"the web (SearXNG)" instead of the raw lowercase engine id
(e.g. "searxng"). ``search_collection_<id>`` tools (including
``web_search`` remapped onto a collection primary engine) resolve
diff --git a/src/local_deep_research/error_handling/report_generator.py b/src/local_deep_research/error_handling/report_generator.py
index 218ed018a9..5d86371301 100644
--- a/src/local_deep_research/error_handling/report_generator.py
+++ b/src/local_deep_research/error_handling/report_generator.py
@@ -378,7 +378,7 @@ We're here to help you get this working:
"Your search query is too long for GitHub's API (max 256 characters).\n\n"
"**Try this:**\n"
"- Shorten your research query\n"
- "- Use a different search engine (DuckDuckGo, Searx, etc.)\n"
+ "- Use a different search engine (SearXNG, Wikipedia, etc.)\n"
"- Break your research into smaller, focused queries"
),
"No module named.*local_deep_research": (
@@ -402,7 +402,7 @@ We're here to help you get this working:
"- **Start SearXNG:** `docker run -d -p 8080:8080 searxng/searxng` then set URL to `http://localhost:8080`\n"
"- **Try different search terms:** Use broader, more general keywords\n"
"- **Check network connection:** Ensure you can access the internet\n"
- "- **Switch search engines:** Try DuckDuckGo, Brave, or Google (if API key configured)\n"
+ "- **Switch search engines:** Try SearXNG, Brave, or Google (if API key configured)\n"
"- **Check for typos** in your research query"
),
"TypeError.*Context.*Size|'<' not supported between instances of .* and 'NoneType'": (
diff --git a/src/local_deep_research/mcp/server.py b/src/local_deep_research/mcp/server.py
index b9f6fa7799..0693191752 100644
--- a/src/local_deep_research/mcp/server.py
+++ b/src/local_deep_research/mcp/server.py
@@ -688,6 +688,11 @@ def search(
query: The search query string.
engine: Search engine to use (e.g., "arxiv", "wikipedia", "searxng", "brave").
This is required — use list_search_engines() to see available options.
+ For "wayback", the query must contain a full http:// or https://
+ URL, or be only a bare domain such as example.com; only the
+ host is looked up. Not supported: an uppercase scheme, a
+ bracketed IPv6 host, a user name or password before the host,
+ and a bare domain with a punycode (xn--) top-level domain.
max_results: Maximum number of results to return (1-100). Default is 10.
Returns:
diff --git a/src/local_deep_research/security/module_whitelist.py b/src/local_deep_research/security/module_whitelist.py
index 3186ec1767..39e0cf1cb0 100644
--- a/src/local_deep_research/security/module_whitelist.py
+++ b/src/local_deep_research/security/module_whitelist.py
@@ -31,7 +31,6 @@ ALLOWED_MODULE_PATHS: frozenset[str] = frozenset(
".engines.search_engine_arxiv",
".engines.search_engine_brave",
".engines.search_engine_collection",
- ".engines.search_engine_ddg",
".engines.search_engine_elasticsearch",
".engines.search_engine_exa",
".engines.search_engine_federal_register",
@@ -81,7 +80,6 @@ ALLOWED_CLASS_NAMES: frozenset[str] = frozenset(
"BaseSearchEngine",
"BraveSearchEngine",
"CollectionSearchEngine",
- "DuckDuckGoSearchEngine",
"ElasticsearchSearchEngine",
"GutenbergSearchEngine",
"ExaSearchEngine",
diff --git a/src/local_deep_research/web_search_engines/engine_registry.py b/src/local_deep_research/web_search_engines/engine_registry.py
index e885b300d0..a37454c55f 100644
--- a/src/local_deep_research/web_search_engines/engine_registry.py
+++ b/src/local_deep_research/web_search_engines/engine_registry.py
@@ -168,12 +168,6 @@ ENGINE_REGISTRY: Dict[str, EngineEntry] = {
class_name="ZenodoSearchEngine",
),
# --- Engines implemented but without settings files ---
- "ddg": EngineEntry(
- module_path=".engines.search_engine_ddg",
- class_name="DuckDuckGoSearchEngine",
- full_search_module=".engines.full_search",
- full_search_class="FullSearchResults",
- ),
"guardian": EngineEntry(
module_path=".engines.search_engine_guardian",
class_name="GuardianSearchEngine",
diff --git a/src/local_deep_research/web_search_engines/engines/search_engine_ddg.py b/src/local_deep_research/web_search_engines/engines/search_engine_ddg.py
deleted file mode 100644
index 6b5a9db132..0000000000
--- a/src/local_deep_research/web_search_engines/engines/search_engine_ddg.py
+++ /dev/null
@@ -1,133 +0,0 @@
-from typing import Any, Dict, List, Optional
-
-from langchain_community.utilities import DuckDuckGoSearchAPIWrapper
-from langchain_core.language_models import BaseLLM
-
-from ...security.secure_logging import logger
-from ..rate_limiting import RateLimitError
-from ..search_engine_base import BaseSearchEngine, Exposure, Sensitivity
-
-
-class DuckDuckGoSearchEngine(BaseSearchEngine):
- """DuckDuckGo search engine implementation with two-phase retrieval"""
-
- # Mark as public search engine
- is_public = True
- egress_sensitivity = Sensitivity.NON_SENSITIVE
- egress_exposure = Exposure.EXPOSING
- # Mark as generic search engine (general web search)
- is_generic = True
-
- def __init__(
- self,
- max_results: int = 10,
- region: str = "us",
- safe_search: bool = True,
- llm: Optional[BaseLLM] = None,
- language: str = "English",
- include_full_content: bool = False,
- max_filtered_results: Optional[int] = None,
- settings_snapshot: Optional[Dict[str, Any]] = None,
- ):
- """
- Initialize the DuckDuckGo search engine.
-
- Args:
- max_results: Maximum number of search results
- region: Region code for search results
- safe_search: Whether to enable safe search
- llm: Language model for relevance filtering
- language: Language for content processing
- include_full_content: Retained for backward compatibility; full-content
- retrieval is handled via the factory wrapper.
- """
- max_results = min(max_results, 50)
- # Initialize the BaseSearchEngine with LLM, max_filtered_results, and max_results
- super().__init__(
- llm=llm,
- max_filtered_results=max_filtered_results,
- max_results=max_results,
- include_full_content=include_full_content,
- settings_snapshot=settings_snapshot,
- )
- self.region = region
- self.safe_search = safe_search
- self.language = language
-
- # Initialize the DuckDuckGo wrapper
- self.engine = DuckDuckGoSearchAPIWrapper(
- region=region,
- max_results=max_results,
- safesearch="moderate" if safe_search else "off",
- )
-
- # Note: full-content retrieval is the factory wrapper's job.
- # The wrapper fetches pages via ``batch_fetch_and_extract`` and
- # populates ``full_content`` on each result; the inner engine
- # only emits snippets from ``_get_previews``.
-
- def _get_previews(self, query: str) -> List[Dict[str, Any]]:
- """
- Get preview information (titles and snippets) for initial search results.
-
- Args:
- query: The search query
-
- Returns:
- List of preview dictionaries with 'id', 'title', and 'snippet' keys
- """
- try:
- # Get search results from DuckDuckGo
- results = self.engine.results(query, max_results=self.max_results)
-
- if not isinstance(results, list):
- return []
-
- # Process results to get previews
- previews = []
- for i, result in enumerate(results):
- preview = {
- "id": result.get("link"), # Use URL as ID for DDG
- "title": result.get("title", ""),
- "snippet": result.get("snippet", ""),
- "link": result.get("link", ""),
- }
-
- previews.append(preview)
-
- return previews
-
- except Exception as e:
- error_msg = str(e)
- sanitized = self._sanitize_error_message(error_msg)
- logger.exception(
- f"Error getting DuckDuckGo previews ({type(e).__name__}): {sanitized}"
- )
-
- # Check for known rate limit patterns
- if "202 Ratelimit" in error_msg or "ratelimit" in error_msg.lower():
- raise RateLimitError(f"DuckDuckGo rate limit hit: {sanitized}")
- if "403" in error_msg or "forbidden" in error_msg.lower():
- raise RateLimitError(
- f"DuckDuckGo access forbidden (possible rate limit): {sanitized}"
- )
- if (
- "timeout" in error_msg.lower()
- or "timed out" in error_msg.lower()
- ):
- # Timeouts can sometimes indicate rate limiting
- raise RateLimitError(
- f"DuckDuckGo timeout (possible rate limit): {sanitized}"
- )
-
- return []
-
- def _get_full_content(
- self, relevant_items: List[Dict[str, Any]]
- ) -> List[Dict[str, Any]]:
- """Return items unchanged.
-
- The factory wrapper populates ``full_content`` after running the
- engine; the inner engine never fetches pages itself.
- """
- return relevant_items
diff --git a/src/local_deep_research/web_search_engines/engines/search_engine_wayback.py b/src/local_deep_research/web_search_engines/engines/search_engine_wayback.py
index f4c2fd4bd9..ea80086078 100644
--- a/src/local_deep_research/web_search_engines/engines/search_engine_wayback.py
+++ b/src/local_deep_research/web_search_engines/engines/search_engine_wayback.py
@@ -15,8 +15,9 @@ from ..search_engine_base import BaseSearchEngine, Exposure, Sensitivity
class WaybackSearchEngine(BaseSearchEngine):
"""
- Internet Archive Wayback Machine search engine implementation
- Provides access to historical versions of web pages
+ Look up historical versions of supplied URLs or domains in the
+ Internet Archive Wayback Machine. This engine does not discover URLs
+ through another search provider.
"""
# Mark as public search engine
@@ -62,8 +63,23 @@ class WaybackSearchEngine(BaseSearchEngine):
def _extract_urls_from_query(self, query: str) -> List[str]:
"""
- Extract URLs from a query string or interpret as an URL if possible.
- For non-URL queries, use a DuckDuckGo search to find relevant URLs.
+ Extract the scheme and host of each lowercase ``http://`` or
+ ``https://`` URL in a query, or read a query that is only a bare
+ domain as a URL. After the scheme, a URL match keeps letters and
+ digits of any script, underscores, hyphens, dots and percent-escapes
+ and ends at the first other character, so a port, path, query
+ string or fragment is dropped and only the host is looked up. The
+ same rule cuts short a host written with other characters, such as
+ combining marks, and keeps a bare domain written with them from
+ matching. Not read: a URL with an uppercase scheme or a bracketed
+ IPv6 host, which does not match, and the host of a URL with a user
+ name or password before it, where the match ends at or before the
+ ``@`` and so looks up the start of the user name instead. The last
+ label of a bare domain may hold only letters, digits and
+ underscores, so a punycode top-level domain such as ``.xn--p1ai`` is
+ not read as a bare domain, though the same domain in a full URL is.
+ Topic queries need URL discovery through a configured search
+ provider before they can be looked up in the Wayback Machine.
Args:
query: The search query or URL
@@ -85,47 +101,11 @@ class WaybackSearchEngine(BaseSearchEngine):
logger.info("Query appears to be a domain")
return [f"http://{query}"]
- # For non-URL queries, use DuckDuckGo to find relevant URLs
- logger.info(
- "Query is not a URL, using DuckDuckGo to find relevant URLs"
- )
- try:
- # Import DuckDuckGo search engine
- from langchain_community.utilities import DuckDuckGoSearchAPIWrapper
-
- # Use max_results from parent class, but limit to 5 for URL discovery
- url_search_limit = min(5, self.max_results)
- ddg = DuckDuckGoSearchAPIWrapper(max_results=url_search_limit)
- # Pass max_results as a positional argument
- results = ddg.results(query, url_search_limit)
-
- # Extract URLs from results
- ddg_urls = [
- str(result.get("link"))
- for result in results
- if result.get("link")
- ]
- if ddg_urls:
- logger.info(
- f"Found {len(ddg_urls)} URLs from DuckDuckGo search"
- )
- return ddg_urls
- except Exception as e:
- safe_msg = self._scrub_error(e)
- logger.exception(
- f"Error using DuckDuckGo for URL discovery ({type(e).__name__}): {safe_msg}"
- )
-
- # Fallback: treat the query as a potential domain or path
- if "/" in query and "." in query:
- logger.info("Treating query as a partial URL")
- return [f"http://{query}"]
- if "." in query:
- logger.info("Treating query as a domain")
- return [f"http://{query}"]
-
# Return empty list if nothing worked
- logger.warning("Could not extract any URLs from the query")
+ logger.warning(
+ "Wayback needs an http(s) URL, or a query that is only a bare "
+ "domain; discover URLs with a configured search provider first"
+ )
return []
def _format_timestamp(self, timestamp: str) -> str:
diff --git a/src/local_deep_research/web_search_engines/search_engine_factory.py b/src/local_deep_research/web_search_engines/search_engine_factory.py
index 477042b09e..2dbb98edf5 100644
--- a/src/local_deep_research/web_search_engines/search_engine_factory.py
+++ b/src/local_deep_research/web_search_engines/search_engine_factory.py
@@ -26,7 +26,7 @@ TIME_PERIOD_ENGINES = ("serpapi", "tavily", "wikinews")
_NON_LOCATION_REGIONS = frozenset({"wt-wt", "all", "none", "worldwide", ""})
# Engines that accept a language code (ISO 639-1) in their ``language`` parameter.
-# Engines like OpenLibrary (MARC code or None) or Wayback/DDG (language name
+# Engines like OpenLibrary (MARC code or None) or Wayback (language name
# for Justext) are excluded so that global language settings do not corrupt them.
LANGUAGE_CODE_ENGINES = frozenset(
{"searxng", "mojeek", "wikipedia", "tinyfish", "scaleserp"}
@@ -443,7 +443,7 @@ def create_search_engine(
# MCP server never set that flag (langgraph-agent tools explicitly set
# use_full_search=False), so without this fallback the engine fetches
# full content (search_snippets_only=False) but the wrapper path that
- # actually hydrates ``full_content`` for searxng/ddg/serpapi/etc. never
+ # actually hydrates ``full_content`` for searxng/serpapi/etc. never
# runs.
if "use_full_search" not in kwargs and "search_snippets_only" in kwargs:
kwargs["use_full_search"] = not kwargs["search_snippets_only"]
@@ -1079,7 +1079,6 @@ def get_search(
# Add engine-specific parameters
if search_tool in [
- "duckduckgo",
"serpapi",
"google_pse",
"brave",
diff --git a/tests/api_tests/populate_search_engines.py b/tests/api_tests/populate_search_engines.py
index 3c42543d36..f63470ffca 100644
--- a/tests/api_tests/populate_search_engines.py
+++ b/tests/api_tests/populate_search_engines.py
@@ -36,11 +36,6 @@ def populate_search_engines():
"description": "Microsoft Bing search engine",
"strengths": ["Good for technical queries", "Image search", "News"],
},
- "duckduckgo": {
- "display_name": "DuckDuckGo",
- "description": "Privacy-focused search engine",
- "strengths": ["Privacy", "No tracking", "Instant answers"],
- },
"wikipedia": {
"display_name": "Wikipedia",
"description": "Wikipedia search",
diff --git a/tests/api_tests/test_search_engines_api.py b/tests/api_tests/test_search_engines_api.py
index a8fec49e13..331a5961cf 100644
--- a/tests/api_tests/test_search_engines_api.py
+++ b/tests/api_tests/test_search_engines_api.py
@@ -53,7 +53,7 @@ class TestSearchEnginesAPI:
# Check if search engine settings exist
endpoints = [
"/settings/api/search.engine.web.searxng.display_name",
- "/settings/api/search.engine.web.duckduckgo.display_name",
+ "/settings/api/search.engine.web.wikipedia.display_name",
"/settings/api/search.engine.web.google.display_name",
"/settings/api/search.engine.web.bing.display_name",
]
diff --git a/tests/fixtures/search_engine_mocks.py b/tests/fixtures/search_engine_mocks.py
index df72cff125..5f49d424af 100644
--- a/tests/fixtures/search_engine_mocks.py
+++ b/tests/fixtures/search_engine_mocks.py
@@ -4,7 +4,7 @@ Search engine mock fixtures - Based on scottvr's patterns but updated for curren
This module provides mock responses for various search engines.
"""
-from typing import Any, Dict, List
+from typing import Any, Dict
from urllib.parse import urlparse
import pytest
@@ -135,22 +135,6 @@ class SearchEngineMocks:
],
}
- @staticmethod
- def ddg_response() -> List[Dict[str, Any]]:
- """Mock DuckDuckGo search response."""
- return [
- {
- "title": "Artificial Intelligence - Wikipedia",
- "link": "https://en.wikipedia.org/wiki/Artificial_intelligence",
- "snippet": "Artificial intelligence (AI) is intelligence demonstrated by machines...",
- },
- {
- "title": "What is AI? Artificial Intelligence Explained",
- "link": "https://example.com/what-is-ai",
- "snippet": "AI refers to the simulation of human intelligence in machines...",
- },
- ]
-
@staticmethod
def error_responses() -> Dict[str, Any]:
"""Collection of error responses for testing error handling."""
diff --git a/tests/mcp/test_mcp_protocol.py b/tests/mcp/test_mcp_protocol.py
index 7f3313139c..345ff7a27b 100644
--- a/tests/mcp/test_mcp_protocol.py
+++ b/tests/mcp/test_mcp_protocol.py
@@ -121,6 +121,20 @@ class TestMCPToolRegistration:
assert get_configuration.__doc__ is not None
assert "config" in get_configuration.__doc__.lower()
+ def test_search_docstring_states_wayback_input_contract(self):
+ """The wayback hint names the inputs the engine cannot read."""
+ from local_deep_research.mcp.server import search
+
+ doc = " ".join(search.__doc__.split())
+ assert "full http:// or https:// URL" in doc
+ for unsupported in (
+ "an uppercase scheme",
+ "a bracketed IPv6 host",
+ "a user name or password before the host",
+ "a bare domain with a punycode (xn--) top-level domain",
+ ):
+ assert unsupported in doc
+
class TestMCPLogging:
"""Tests for MCP logging configuration (critical for STDIO)."""
diff --git a/tests/search_engines/conftest.py b/tests/search_engines/conftest.py
index 385d38a09a..a4e8a5d3a5 100644
--- a/tests/search_engines/conftest.py
+++ b/tests/search_engines/conftest.py
@@ -351,33 +351,6 @@ def mock_github_code_search_response(mock_response_factory):
)
-# ============================================================================
-# DuckDuckGo-specific fixtures
-# ============================================================================
-
-
-def mock_ddg_response():
- """Mock DuckDuckGo search results (from duckduckgo_search library format)."""
- return [
- {
- "title": "Introduction to Machine Learning | Example Site",
- "href": "https://example.com/ml-intro",
- "body": "A comprehensive introduction to machine learning concepts and algorithms.",
- },
- {
- "title": "Machine Learning Tutorial - Learn ML",
- "href": "https://learn-ml.example.org/tutorial",
- "body": "Step-by-step tutorial on building your first machine learning model.",
- },
- {
- "title": "ML Best Practices",
- "href": "https://blog.example.com/ml-best-practices",
- "body": "Learn the best practices for implementing machine learning in production.",
- },
- ]
-
-
# ============================================================================
# Brave Search-specific fixtures
# ============================================================================
diff --git a/tests/search_engines/test_search_engine_ddg.py b/tests/search_engines/test_search_engine_ddg.py
deleted file mode 100644
index 645744e324..0000000000
--- a/tests/search_engines/test_search_engine_ddg.py
+++ /dev/null
@@ -1,358 +0,0 @@
-"""
-Comprehensive tests for the DuckDuckGo search engine.
-Tests initialization, search functionality, error handling, and result parsing.
-
-Note: These tests require the 'duckduckgo_search' package to be installed.
-"""
-
-import pytest
-from unittest.mock import Mock, patch
-
-# Check if DuckDuckGo search is actually importable (requires both
-# langchain_community and the ddgs package to be installed)
-try:
- from langchain_community.utilities import DuckDuckGoSearchAPIWrapper
-
- DuckDuckGoSearchAPIWrapper()
- DDGS_AVAILABLE = True
-except (ImportError, Exception):
- DDGS_AVAILABLE = False
-
-
[email protected](not DDGS_AVAILABLE, reason="ddgs package not installed")
-class TestDuckDuckGoSearchEngineInit:
- """Tests for DuckDuckGo search engine initialization."""
-
- def test_init_default_parameters(self):
- """Test initialization with default parameters."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- engine = DuckDuckGoSearchEngine()
-
- assert engine.max_results >= 10
- assert engine.is_public is True
-
- def test_init_custom_max_results(self):
- """Test initialization with custom max results."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- engine = DuckDuckGoSearchEngine(max_results=25)
- assert engine.max_results >= 25
-
- def test_init_with_region(self):
- """Test initialization with specific region."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- engine = DuckDuckGoSearchEngine(region="us-en")
- assert engine.region == "us-en"
-
-
[email protected](not DDGS_AVAILABLE, reason="ddgs package not installed")
-class TestDuckDuckGoSearchExecution:
- """Tests for DuckDuckGo search execution."""
-
- @pytest.fixture
- def ddg_engine(self):
- """Create a DuckDuckGo engine instance."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- return DuckDuckGoSearchEngine(max_results=10)
-
- def test_engine_initialization(self, ddg_engine):
- """Test that engine is properly initialized."""
- assert ddg_engine is not None
- assert ddg_engine.max_results >= 10
-
-
[email protected](not DDGS_AVAILABLE, reason="ddgs package not installed")
-class TestDuckDuckGoRegionSupport:
- """Tests for DuckDuckGo region/locale support."""
-
- def test_default_region(self):
- """Test default region configuration."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- engine = DuckDuckGoSearchEngine()
- # Default region should be set or None
- assert hasattr(engine, "region")
-
- def test_custom_region(self):
- """Test custom region configuration."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- engine = DuckDuckGoSearchEngine(region="de-de")
- assert engine.region == "de-de"
-
-
[email protected](not DDGS_AVAILABLE, reason="ddgs package not installed")
-class TestDuckDuckGoEngineType:
- """Tests for DuckDuckGo engine type identification."""
-
- def test_engine_type_set(self):
- """Test that engine type is properly set."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- engine = DuckDuckGoSearchEngine()
- assert "duckduckgo" in engine.engine_type.lower()
-
- def test_engine_is_public(self):
- """Test that engine is marked as public."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- engine = DuckDuckGoSearchEngine()
- assert engine.is_public is True
-
- def test_engine_is_generic(self):
- """Test that engine is marked as generic (not specialized)."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- engine = DuckDuckGoSearchEngine()
- assert engine.is_generic is True
-
-
-# Tests that can run without the ddgs package (just testing mock structure)
-class TestDDGResponseFixtures:
- """Tests for DuckDuckGo response fixture structure."""
-
- def test_mock_response_structure(self, mock_ddg_response):
- """Test that mock response has correct structure."""
- assert isinstance(mock_ddg_response, list)
- assert len(mock_ddg_response) == 3
-
- def test_mock_response_has_required_fields(self, mock_ddg_response):
- """Test that mock response items have required fields."""
- for result in mock_ddg_response:
- assert "title" in result
- assert "href" in result
- assert "body" in result
-
- def test_mock_response_urls_valid(self, mock_ddg_response):
- """Test that URLs in mock response are valid."""
- for result in mock_ddg_response:
- assert result["href"].startswith("http")
-
-
[email protected](not DDGS_AVAILABLE, reason="ddgs package not installed")
-class TestDuckDuckGoGetPreviews:
- """Tests for the _get_previews method and error handling."""
-
- @pytest.fixture
- def ddg_engine(self):
- """Create a DuckDuckGo engine instance."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- return DuckDuckGoSearchEngine(max_results=10)
-
- def test_get_previews_returns_correct_format(self, ddg_engine):
- """Test that _get_previews returns properly formatted previews."""
- with patch.object(ddg_engine.engine, "results") as mock_results:
- mock_results.return_value = [
- {
- "title": "Test Title",
- "link": "https://example.com",
- "snippet": "Test snippet",
- },
- {
- "title": "Title 2",
- "link": "https://example2.com",
- "snippet": "Snippet 2",
- },
- ]
-
- previews = ddg_engine._get_previews("test query")
-
- assert len(previews) == 2
- assert previews[0]["title"] == "Test Title"
- assert previews[0]["link"] == "https://example.com"
- assert previews[0]["snippet"] == "Test snippet"
- assert previews[0]["id"] == "https://example.com"
-
- def test_get_previews_handles_empty_results(self, ddg_engine):
- """Test that _get_previews handles empty results list."""
- with patch.object(ddg_engine.engine, "results") as mock_results:
- mock_results.return_value = []
-
- previews = ddg_engine._get_previews("test query")
-
- assert previews == []
-
- def test_get_previews_handles_non_list_results(self, ddg_engine):
- """Test that _get_previews handles non-list results."""
- with patch.object(ddg_engine.engine, "results") as mock_results:
- mock_results.return_value = None
-
- previews = ddg_engine._get_previews("test query")
-
- assert previews == []
-
- def test_get_previews_rate_limit_202_raises_error(self, ddg_engine):
- """Test that 202 Ratelimit error raises RateLimitError."""
- from local_deep_research.web_search_engines.rate_limiting import (
- RateLimitError,
- )
-
- with patch.object(ddg_engine.engine, "results") as mock_results:
- mock_results.side_effect = Exception("202 Ratelimit hit")
-
- with pytest.raises(RateLimitError) as exc_info:
- ddg_engine._get_previews("test query")
-
- assert "rate limit" in str(exc_info.value).lower()
-
- def test_get_previews_403_forbidden_raises_error(self, ddg_engine):
- """Test that 403 forbidden error raises RateLimitError."""
- from local_deep_research.web_search_engines.rate_limiting import (
- RateLimitError,
- )
-
- with patch.object(ddg_engine.engine, "results") as mock_results:
- mock_results.side_effect = Exception("403 Forbidden response")
-
- with pytest.raises(RateLimitError) as exc_info:
- ddg_engine._get_previews("test query")
-
- assert "forbidden" in str(exc_info.value).lower()
-
- def test_get_previews_timeout_raises_error(self, ddg_engine):
- """Test that timeout error raises RateLimitError."""
- from local_deep_research.web_search_engines.rate_limiting import (
- RateLimitError,
- )
-
- with patch.object(ddg_engine.engine, "results") as mock_results:
- mock_results.side_effect = Exception("Connection timeout")
-
- with pytest.raises(RateLimitError) as exc_info:
- ddg_engine._get_previews("test query")
-
- assert "timeout" in str(exc_info.value).lower()
-
- def test_get_previews_generic_exception_returns_empty(self, ddg_engine):
- """Test that generic exceptions return empty list."""
- with patch.object(ddg_engine.engine, "results") as mock_results:
- mock_results.side_effect = ValueError("Some generic error")
-
- previews = ddg_engine._get_previews("test query")
-
- assert previews == []
-
- def test_get_previews_handles_missing_fields(self, ddg_engine):
- """Test that _get_previews handles results with missing fields."""
- with patch.object(ddg_engine.engine, "results") as mock_results:
- mock_results.return_value = [
- {"title": "Title Only"}, # Missing link and snippet
- {"link": "https://example.com"}, # Missing title and snippet
- ]
-
- previews = ddg_engine._get_previews("test query")
-
- assert len(previews) == 2
- assert previews[0]["title"] == "Title Only"
- assert previews[0]["snippet"] == ""
- assert previews[1]["title"] == ""
-
-
[email protected](not DDGS_AVAILABLE, reason="ddgs package not installed")
-class TestDuckDuckGoGetFullContent:
- """Tests for the _get_full_content method."""
-
- @pytest.fixture
- def ddg_engine_with_full_content(self):
- """Create a DuckDuckGo engine with full content enabled."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- mock_llm = Mock()
- return DuckDuckGoSearchEngine(
- max_results=10,
- llm=mock_llm,
- include_full_content=True,
- )
-
- @pytest.fixture
- def ddg_engine_snippets_only(self):
- """Create a DuckDuckGo engine without full content."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- return DuckDuckGoSearchEngine(
- max_results=10,
- include_full_content=False,
- )
-
- def test_get_full_content_passes_through(
- self, ddg_engine_with_full_content
- ):
- """DDG no longer self-wraps, so ``_get_full_content`` is a
- pass-through. The factory wrapper populates ``full_content``
- after the inner engine runs.
- """
- items = [{"title": "Test", "snippet": "Snippet"}]
- assert not hasattr(ddg_engine_with_full_content, "full_search")
- assert ddg_engine_with_full_content._get_full_content(items) is items
-
-
[email protected](not DDGS_AVAILABLE, reason="ddgs package not installed")
-class TestDuckDuckGoRunMethod:
- """Tests for the run() method which uses the two-phase approach."""
-
- @pytest.fixture
- def ddg_engine(self):
- """Create a DuckDuckGo engine instance."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- return DuckDuckGoSearchEngine(max_results=5)
-
- def test_run_returns_results(self, ddg_engine):
- """Test that run() returns search results."""
- with patch.object(ddg_engine, "_get_previews") as mock_previews:
- mock_previews.return_value = [
- {
- "id": "1",
- "title": "Test",
- "snippet": "Test snippet",
- "link": "https://example.com",
- },
- ]
-
- results = ddg_engine.run("test query")
-
- # Should return filtered results (or all if no filtering)
- assert isinstance(results, list)
-
- def test_run_accepts_research_context(self, ddg_engine):
- """Test that run() accepts research_context parameter."""
- with patch.object(ddg_engine, "_get_previews") as mock_previews:
- mock_previews.return_value = []
-
- # Should not raise
- results = ddg_engine.run(
- "test query",
- research_context={"previous_findings": "some info"},
- )
-
- assert isinstance(results, list)
diff --git a/tests/search_engines/test_search_engine_wayback.py b/tests/search_engines/test_search_engine_wayback.py
index 9c94cb3a38..5c87930017 100644
--- a/tests/search_engines/test_search_engine_wayback.py
+++ b/tests/search_engines/test_search_engine_wayback.py
@@ -122,8 +122,7 @@ class TestWaybackURLExtraction:
"""Test extraction of full HTTP URL (domain portion)."""
urls = engine._extract_urls_from_query("https://example.com/page")
# The regex extracts the domain portion of the URL
- assert len(urls) >= 1
- assert any("example.com" in url for url in urls)
+ assert urls == ["https://example.com"]
def test_extract_url_from_multiple_urls(self, engine):
"""Test extraction of multiple URLs."""
@@ -138,10 +137,10 @@ class TestWaybackURLExtraction:
urls = engine._extract_urls_from_query("example.com")
assert "http://example.com" in urls
- def test_extract_url_from_partial_path(self, engine):
- """Test extraction from partial URL path."""
+ def test_partial_path_without_scheme_is_not_extracted(self, engine):
+ """A domain with a path but no scheme is not an archive target."""
urls = engine._extract_urls_from_query("example.com/page/test")
- assert "http://example.com/page/test" in urls
+ assert urls == []
class TestWaybackTimestampFormatting:
diff --git a/tests/search_engines/test_wayback_archive_only.py b/tests/search_engines/test_wayback_archive_only.py
new file mode 100644
index 0000000000..aaec84be7a
--- /dev/null
+++ b/tests/search_engines/test_wayback_archive_only.py
@@ -0,0 +1,226 @@
+"""Wayback looks up the hosts of supplied targets without another provider."""
+
+from unittest.mock import Mock
+
+import pytest
+import requests
+
+from local_deep_research.web_search_engines.engines.search_engine_wayback import (
+ WaybackSearchEngine,
+)
+
+
+def discovery_guards(monkeypatch):
+ import langchain_community.utilities as utilities
+
+ discovery = Mock()
+ discovery.return_value.results.return_value = [
+ {"link": "https://unexpected.example"}
+ ]
+ monkeypatch.setattr(utilities, "DuckDuckGoSearchAPIWrapper", discovery)
+ request = Mock(side_effect=AssertionError("Unexpected outbound request"))
+ monkeypatch.setattr(requests.sessions.Session, "request", request)
+ return discovery, request
+
+
+ ("query", "expected"),
+ [
+ ("https://example.com", ["https://example.com"]),
+ ("example.com", ["http://example.com"]),
+ ("sub.example.co.uk", ["http://sub.example.co.uk"]),
+ pytest.param(
+ "Check https://example.com/page archive",
+ ["https://example.com"],
+ id="running_text_path",
+ ),
+ pytest.param(
+ "https://example.com:8443/page?token=secret#access_token=x",
+ ["https://example.com"],
+ id="port_query_and_fragment_dropped",
+ ),
+ pytest.param(
+ "[source](https://example.com:8443/page?x=1).",
+ ["https://example.com"],
+ id="markdown_link",
+ ),
+ pytest.param(
+ "[source: https://example.com] for details",
+ ["https://example.com"],
+ id="bracket_closer_after_host",
+ ),
+ pytest.param(
+ "Citations: [1: https://example.com], [2: https://example.org]",
+ ["https://example.com", "https://example.org"],
+ id="bracketed_citations",
+ ),
+ pytest.param(
+ "Archived versions of [https://a.example, https://b.example]",
+ ["https://a.example", "https://b.example"],
+ id="bracketed_list",
+ ),
+ pytest.param(
+ "Pages (https://a.example/about, https://b.example/about) in 2010",
+ ["https://a.example", "https://b.example"],
+ id="parenthesised_list",
+ ),
+ pytest.param(
+ "What did https://example.com's homepage say in 2005?",
+ ["https://example.com"],
+ id="ascii_possessive",
+ ),
+ pytest.param(
+ "The bot said 'see https://example.com' twice",
+ ["https://example.com"],
+ id="closing_quote_without_adjacent_opener",
+ ),
+ pytest.param(
+ "**https://example.com**'s homepage",
+ ["https://example.com"],
+ id="emphasis_then_possessive",
+ ),
+ pytest.param(
+ "https://example.com(see note)",
+ ["https://example.com"],
+ id="parenthesis_after_host",
+ ),
+ pytest.param(
+ "see https://example.com/a(x)https://example.org/b now",
+ ["https://example.com", "https://example.org"],
+ id="urls_jammed_without_space",
+ ),
+ pytest.param(
+ "[read https://example.com/label-url][1]",
+ ["https://example.com"],
+ id="reference_style_label",
+ ),
+ pytest.param(
+ "请查看https://example.com/about的历史版本",
+ ["https://example.com"],
+ id="cjk_text_around_url",
+ ),
+ pytest.param(
+ "参见https://example.com。",
+ ["https://example.com"],
+ id="cjk_full_stop_after_host",
+ ),
+ pytest.param(
+ "https://example.com,https://example.org",
+ ["https://example.com", "https://example.org"],
+ id="fullwidth_comma_list",
+ ),
+ ("https://", []),
+ ("example.com/page", []),
+ ("Find example.com", []),
+ ("This sentence has a period.", []),
+ ("topic.with.dots and words", []),
+ ("topic/with.a.path and words", []),
+ ("historical information about a topic", []),
+ ("", []),
+ # Inputs the tool descriptions and docs name as unsupported: userinfo
+ # reads the user name instead of the host, and an uppercase scheme, a
+ # bracketed IPv6 host and a punycode TLD in a bare domain match
+ # nothing, while the same punycode domain in a full URL is read. A
+ # combining mark is not a host character and ends the host early.
+ ("https://user:[email protected]/?api_key=x", ["https://user"]),
+ ("HTTPS://EXAMPLE.COM/page", []),
+ ("https://[2001:db8::1]/page", []),
+ ("xn--e1afmkfd.xn--p1ai", []),
+ ("http://xn--e1afmkfd.xn--p1ai/", ["http://xn--e1afmkfd.xn--p1ai"]),
+ ("https://bu\u0308cher.de/x", ["https://bu"]),
+ ("bu\u0308cher.de", []),
+ ],
+)
+def test_extract_archive_hosts_without_discovery(
+ query, expected, discovery_guards
+):
+ discovery, request = discovery_guards
+ engine = WaybackSearchEngine()
+
+ assert engine._extract_urls_from_query(query) == expected
+ discovery.assert_not_called()
+ request.assert_not_called()
+
+
+ "query",
+ [
+ "historical information about a topic",
+ "This sentence has a period.",
+ "topic/with.a.path and words",
+ "example.com/page",
+ "",
+ ],
+)
+def test_plain_text_previews_do_not_search_or_request_archives(
+ query, discovery_guards
+):
+ discovery, request = discovery_guards
+ engine = WaybackSearchEngine()
+ engine._get_wayback_snapshots = Mock(return_value=[])
+
+ assert engine._get_previews(query) == []
+ discovery.assert_not_called()
+ engine._get_wayback_snapshots.assert_not_called()
+ request.assert_not_called()
+
+
[email protected]("closest_only", [False, True])
+def test_archive_previews_look_up_only_the_host(
+ closest_only, discovery_guards, monkeypatch
+):
+ from local_deep_research.web_search_engines.engines import (
+ search_engine_wayback as wayback,
+ )
+
+ discovery, request = discovery_guards
+ query = (
+ "See https://example.com:8443/page?token=secret#access_token=x "
+ "and https://example.org/a"
+ )
+ hosts = ["https://example.com", "https://example.org"]
+ timestamp = "20240101000000"
+
+ def archive_response(_endpoint, *, params):
+ target = params["url"]
+ response = Mock(status_code=200, headers={})
+ if closest_only:
+ response.json.return_value = {
+ "archived_snapshots": {
+ "closest": {
+ "available": True,
+ "timestamp": timestamp,
+ "url": f"https://web.archive.org/web/{timestamp}/{target}",
+ }
+ }
+ }
+ else:
+ response.json.return_value = [
+ ["timestamp", "original", "statuscode", "mimetype"],
+ [timestamp, target, "200", "text/html"],
+ ]
+ return response
+
+ archive_lookup = Mock(side_effect=archive_response)
+ monkeypatch.setattr(wayback, "safe_get", archive_lookup)
+ engine = WaybackSearchEngine(closest_only=closest_only)
+ engine.rate_tracker = Mock()
+
+ previews = engine._get_previews(query)
+
+ endpoint = engine.available_api if closest_only else engine.cdx_api
+ calls = archive_lookup.call_args_list
+ assert [call.args[0] for call in calls] == [endpoint] * len(hosts)
+ assert [call.kwargs["params"]["url"] for call in calls] == hosts
+ assert [preview["original_url"] for preview in previews] == hosts
+ assert [preview["link"] for preview in previews] == [
+ f"https://web.archive.org/web/{timestamp}/{host}" for host in hosts
+ ]
+ # Nothing after a host reaches the archive or the stored results.
+ for preview in previews:
+ for value in (preview["title"], preview["link"], preview["id"]):
+ assert "8443" not in value
+ assert "token" not in value
+ discovery.assert_not_called()
+ request.assert_not_called()
diff --git a/tests/security/test_egress_policy.py b/tests/security/test_egress_policy.py
index d55a67a2b1..8035cf8586 100644
--- a/tests/security/test_egress_policy.py
+++ b/tests/security/test_egress_policy.py
@@ -353,11 +353,16 @@ def test_evaluate_engine_no_snapshot_fails_closed():
assert decision.reason == "no_snapshot"
-def test_evaluate_engine_unknown_name_fails_closed():
- ctx = make_ctx()
- decision = evaluate_engine(
- "totally_made_up_engine", ctx, settings_snapshot={}
- )
+ "engine_name", ["totally_made_up_engine", "ddg", "duckduckgo"]
+)
+ "scope",
+ [EgressScope.PRIVATE_ONLY, EgressScope.PUBLIC_ONLY, EgressScope.BOTH],
+)
+def test_evaluate_engine_unknown_name_fails_closed(engine_name, scope):
+ ctx = make_ctx(scope=scope)
+ decision = evaluate_engine(engine_name, ctx, settings_snapshot={})
assert not decision.allowed
assert decision.reason == "engine_unknown"
@@ -434,7 +439,6 @@ def test_evaluate_engine_searxng_remote_classified_public():
# Engines that can be loaded in CI (no optional deps needed).
_PUBLIC_ENGINES = [
"brave",
- "ddg",
"exa",
"federal_register",
"github",
diff --git a/tests/security/test_module_whitelist.py b/tests/security/test_module_whitelist.py
index 52e6d5863e..af1dc9a371 100644
--- a/tests/security/test_module_whitelist.py
+++ b/tests/security/test_module_whitelist.py
@@ -44,7 +44,6 @@ class TestAllowedModulePaths:
"""Whitelist should contain expected search engine modules."""
expected_engines = [
".engines.search_engine_brave",
- ".engines.search_engine_ddg",
".engines.search_engine_tavily",
".engines.search_engine_wikipedia",
]
@@ -65,7 +64,6 @@ class TestAllowedClassNames:
"""Whitelist should contain expected search engine classes."""
expected_classes = [
"BraveSearchEngine",
- "DuckDuckGoSearchEngine",
"TavilySearchEngine",
"WikipediaSearchEngine",
"BaseSearchEngine",
@@ -101,6 +99,25 @@ class TestValidateModuleImport:
)
assert result is True
+ @pytest.mark.parametrize(
+ "module_path,class_name",
+ [
+ (".engines.search_engine_ddg", "BraveSearchEngine"),
+ (".engines.search_engine_brave", "DuckDuckGoSearchEngine"),
+ (".engines.search_engine_ddg", "DuckDuckGoSearchEngine"),
+ ],
+ )
+ def test_removed_duckduckgo_adapter_cannot_be_imported(
+ self, module_path, class_name
+ ):
+ assert validate_module_import(module_path, class_name) is False
+ with patch(
+ "local_deep_research.security.module_whitelist.importlib.import_module"
+ ) as import_module:
+ with pytest.raises(SecurityError):
+ get_safe_module_class(module_path, class_name)
+ import_module.assert_not_called()
+
def test_rejects_absolute_path(self):
"""Should reject absolute module paths."""
# Absolute path attempt
diff --git a/tests/security/test_module_whitelist_high_value.py b/tests/security/test_module_whitelist_high_value.py
index cb1eaf9543..5b1fd9345b 100644
--- a/tests/security/test_module_whitelist_high_value.py
+++ b/tests/security/test_module_whitelist_high_value.py
@@ -38,7 +38,6 @@ class TestAllowedModulePaths:
def test_known_engine_paths_present(self):
expected = [
".engines.search_engine_brave",
- ".engines.search_engine_ddg",
".engines.search_engine_arxiv",
".engines.search_engine_wikipedia",
".engines.search_engine_tavily",
@@ -64,7 +63,6 @@ class TestAllowedClassNames:
def test_known_classes_present(self):
expected = [
"BraveSearchEngine",
- "DuckDuckGoSearchEngine",
"ArXivSearchEngine",
"WikipediaSearchEngine",
"TavilySearchEngine",
diff --git a/tests/strategies/test_langgraph_agent_strategy.py b/tests/strategies/test_langgraph_agent_strategy.py
index e2959323ad..8b78ff797d 100644
--- a/tests/strategies/test_langgraph_agent_strategy.py
+++ b/tests/strategies/test_langgraph_agent_strategy.py
@@ -1845,7 +1845,7 @@ class TestSearchToolMakers:
return_value=mock_engine,
):
tool_fn = _make_web_search_tool(
- search_engine_name="duckduckgo",
+ search_engine_name="brave",
model=MagicMock(),
settings_snapshot={},
collector=collector,
@@ -1908,7 +1908,7 @@ class TestSearchToolMakers:
return_value=mock_engine,
) as mock_create:
tool_fn = _make_web_search_tool(
- search_engine_name="duckduckgo",
+ search_engine_name="brave",
model=MagicMock(),
settings_snapshot=snapshot,
collector=collector,
@@ -2236,7 +2236,7 @@ class TestFetchProgressAdapter:
return_value=mock_engine,
) as mock_create:
tool_fn = _make_web_search_tool(
- search_engine_name="duckduckgo",
+ search_engine_name="brave",
model=MagicMock(),
settings_snapshot={},
collector=collector,
@@ -2453,7 +2453,7 @@ class TestLangGraphAgentStrategy:
"model": MagicMock(),
"search": MagicMock(),
"all_links_of_system": [],
- "settings_snapshot": {"search.tool": {"value": "duckduckgo"}},
+ "settings_snapshot": {"search.tool": {"value": "brave"}},
}
defaults.update(overrides)
return LangGraphAgentStrategy(**defaults)
@@ -2784,14 +2784,9 @@ class TestLangGraphAgentStrategy:
def test_engine_name_fallback_to_class(self):
mock_search = MagicMock()
- mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"
+ mock_search.__class__.__name__ = "GooglePSESearchEngine"
strategy = self._make_strategy(search=mock_search, settings_snapshot={})
- # Registry reverse-lookup yields the canonical id (``ddg``),
- # NOT the class-derived ``duckduckgo`` the previous heuristic
- # produced — that mismatch let ``search_duckduckgo`` slip into
- # the agent's specialized tool list alongside ``web_search`` when
- # DuckDuckGo was the configured primary.
- assert strategy._search_engine_name == "ddg"
+ assert strategy._search_engine_name == "google_pse"
def test_engine_name_semantic_scholar_resolves_to_canonical(self):
"""``SemanticScholarSearchEngine`` -> ``semantic_scholar`` (canonical),
@@ -2809,9 +2804,9 @@ class TestLangGraphAgentStrategy:
display-name map, with brand-correct casing rather than the raw
lowercase id."""
strategy = self._make_strategy(
- settings_snapshot={"search.tool": {"value": "duckduckgo"}}
+ settings_snapshot={"search.tool": {"value": "brave"}}
)
- assert strategy._display_tool_name("web_search") == "DuckDuckGo"
+ assert strategy._display_tool_name("web_search") == "Brave Search"
def test_display_tool_name_web_search_searxng(self):
strategy = self._make_strategy(
@@ -3493,7 +3488,7 @@ class TestBuildLibraryResolver:
"model": MagicMock(),
"search": MagicMock(),
"all_links_of_system": [],
- "settings_snapshot": {"search.tool": {"value": "duckduckgo"}},
+ "settings_snapshot": {"search.tool": {"value": "brave"}},
}
defaults.update(overrides)
return LangGraphAgentStrategy(**defaults)
@@ -3508,7 +3503,7 @@ class TestBuildLibraryResolver:
strategy = self._make_strategy(
settings_snapshot={
- "search.tool": {"value": "duckduckgo"},
+ "search.tool": {"value": "brave"},
"_username": "alice",
}
)
@@ -3523,7 +3518,7 @@ class TestBuildLibraryResolver:
the pre-A3 behaviour so the egress policy rejects library URLs
unchanged."""
strategy = self._make_strategy(
- settings_snapshot={"search.tool": {"value": "duckduckgo"}}
+ settings_snapshot={"search.tool": {"value": "brave"}}
)
assert strategy._build_library_resolver() is None
@@ -3564,7 +3559,7 @@ class TestToolCallProgressFormatting:
model=MagicMock(),
search=MagicMock(),
all_links_of_system=[],
- settings_snapshot={"search.tool": {"value": "duckduckgo"}},
+ settings_snapshot={"search.tool": {"value": "brave"}},
)
def _tc(self, name, **args):
@@ -3685,9 +3680,9 @@ class TestToolCallProgressFormatting:
strategy = self._make_strategy()
out = strategy._format_tool_call_progress(
self._tc("web_search", url="https://example.org"),
- "DuckDuckGo",
+ "Brave Search",
)
- assert out == '🔍 Searching DuckDuckGo: "https://example.org"'
+ assert out == '🔍 Searching Brave Search: "https://example.org"'
def test_unknown_tool_uses_search_prefix(self):
"""A tool that isn't fetch_content or research_subtopic gets the
@@ -5150,7 +5145,7 @@ class TestFetchModeSettingResolution:
"model": MagicMock(),
"search": MagicMock(),
"all_links_of_system": [],
- "settings_snapshot": {"search.tool": "duckduckgo"},
+ "settings_snapshot": {"search.tool": "brave"},
}
defaults.update(overrides)
return LangGraphAgentStrategy(**defaults)
@@ -5161,7 +5156,7 @@ class TestFetchModeSettingResolution:
"""
strategy = self._make_strategy(
settings_snapshot={
- "search.tool": "duckduckgo",
+ "search.tool": "brave",
"search.fetch.mode": "summary_focus",
}
)
@@ -5179,7 +5174,7 @@ class TestFetchModeSettingResolution:
with loguru_caplog.at_level("WARNING"):
strategy = self._make_strategy(
settings_snapshot={
- "search.tool": "duckduckgo",
+ "search.tool": "brave",
"search.fetch.mode": "definitely-not-a-real-mode",
}
)
@@ -5196,7 +5191,7 @@ class TestFetchModeSettingResolution:
"""
strategy = self._make_strategy(
settings_snapshot={
- "search.tool": "duckduckgo",
+ "search.tool": "brave",
"search.fetch.mode": "disabled",
}
)
@@ -5513,7 +5508,7 @@ class TestProgressMetadataKeepsStableId:
model=MagicMock(),
search=MagicMock(),
all_links_of_system=[],
- settings_snapshot={"search.tool": {"value": "duckduckgo"}},
+ settings_snapshot={"search.tool": {"value": "brave"}},
)
def test_tool_call_metadata_keeps_id_label_in_message(self):
@@ -5578,7 +5573,7 @@ class TestProgressMetadataKeepsStableId:
# metadata keeps the stable id ...
assert metadata["tool"] == "web_search"
# ... while the user sees the brand label in the message text.
- assert "DuckDuckGo" in message
+ assert "Brave Search" in message
# ---------------------------------------------------------------------------
@@ -5610,7 +5605,7 @@ class TestEgressScopeFiltering:
# Available-engines fixture. ``arxiv`` and ``pubmed`` are registered
# public engines (``is_public = True`` on their classes); ``library``
# is hardcoded local in ``evaluate_engine`` (line 322-326).
- # ``duckduckgo`` is the current primary — already added as
+ # ``brave`` is the current primary — already added as
# ``web_search`` and explicitly skipped at line 618.
_FIXTURE_AVAILABLE = {
"arxiv": {
@@ -5638,19 +5633,19 @@ class TestEgressScopeFiltering:
"description": "My research papers (Collection)",
"strengths": ["curated documents"],
},
- "duckduckgo": {
+ "brave": {
"is_local": False,
- "description": "DuckDuckGo",
+ "description": "Brave Search",
},
}
- def _make_strategy(self, scope, primary_engine="duckduckgo"):
+ def _make_strategy(self, scope, primary_engine="brave"):
from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (
LangGraphAgentStrategy,
)
mock_search = MagicMock()
- mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"
+ mock_search.__class__.__name__ = "BraveSearchEngine"
return LangGraphAgentStrategy(
model=MagicMock(),
search=mock_search,
@@ -5785,17 +5780,20 @@ class TestEgressScopeFiltering:
)
# ------------------------------------------------------------------
- # BOTH (default) — every classified engine is registered
+ # Legacy BOTH resolves through adaptive primary classification
# ------------------------------------------------------------------
- def test_both_scope_registers_every_classified_engine(self):
- """The default scope BOTH must register every classified engine
- in the available dict. The current primary is excluded by the
- explicit ``continue`` at line 618 — NOT by the scope filter — so
- a regression that moved it into the scope-mismatch path would
- still be caught by the assertion that it's absent.
+ @pytest.mark.parametrize(
+ "primary,include_local",
+ [("brave", False), ("unclassified_test_primary", True)],
+ )
+ def test_legacy_both_scope_follows_primary_classification(
+ self, primary, include_local
+ ):
+ """Known public primaries resolve to PUBLIC_ONLY; an unknown
+ primary retains the resolved BOTH fallback.
"""
- strategy = self._make_strategy(scope="both")
+ strategy = self._make_strategy(scope="both", primary_engine=primary)
with patch(
"local_deep_research.web_search_engines.search_engines_config.list_eligible_engine_configs",
return_value=self._FIXTURE_AVAILABLE,
@@ -5803,13 +5801,15 @@ class TestEgressScopeFiltering:
tools = strategy._build_tools(overall_query="q")
names = self._tool_names(tools)
- for expected in ("search_arxiv", "search_pubmed", "search_library"):
+ for expected in ("search_arxiv", "search_pubmed"):
assert expected in names, (
- f"Expected {expected} under BOTH but got: {sorted(names)}"
+ f"Expected {expected} but got: {sorted(names)}"
)
# The current engine is NEVER added as a specialized tool
# regardless of scope.
- assert "search_duckduckgo" not in names
+ assert ("search_library" in names) is include_local
+ assert ("search_collection_abc123" in names) is include_local
+ assert ("search_brave" in names) is (primary != "brave")
# ------------------------------------------------------------------
# Fail-closed: corrupted scope value
@@ -5883,9 +5883,8 @@ class TestEgressScopeFiltering:
# Filtering the tool LIST closes the latency-leak half of the timing
# attack. The other half is the prompt addendum: the LLM is *told* which
# tools exist so it doesn't waste tokens probing for forbidden engines.
-# These tests pin that the addendum text varies by scope and is empty
-# under BOTH (we don't want to bleed policy state into the LLM for the
-# default scope).
+# These tests pin that the addendum text varies by resolved scope and is
+# empty under the unclassified-primary BOTH fallback.
class TestEgressScopePolicyAddendum:
@@ -5894,13 +5893,13 @@ class TestEgressScopePolicyAddendum:
wording must reflect the active scope.
"""
- def _make_strategy(self, scope, primary_engine="duckduckgo"):
+ def _make_strategy(self, scope, primary_engine="brave"):
from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (
LangGraphAgentStrategy,
)
mock_search = MagicMock()
- mock_search.__class__.__name__ = "DuckDuckGoSearchEngine"
+ mock_search.__class__.__name__ = "BraveSearchEngine"
return LangGraphAgentStrategy(
model=MagicMock(),
search=mock_search,
@@ -5911,7 +5910,7 @@ class TestEgressScopePolicyAddendum:
},
)
- def _capture_prompt(self, scope, primary="duckduckgo"):
+ def _capture_prompt(self, scope, primary="brave"):
"""Run analyze_topic in a heavily-mocked harness and return the
``system_prompt`` string passed to ``create_agent``. There's no
smaller public hook for the addendum — the prompt-string is
@@ -5960,7 +5959,7 @@ class TestEgressScopePolicyAddendum:
prompt = self._capture_prompt("strict")
assert "RESTRICTED MODE" in prompt
# The primary engine name must be cited.
- assert "duckduckgo" in prompt.lower()
+ assert "brave" in prompt.lower()
def test_private_only_addendum_names_public_engines_as_unavailable(self):
"""PRIVATE-ONLY addendum must explicitly warn the LLM that
@@ -5984,17 +5983,17 @@ class TestEgressScopePolicyAddendum:
# Names at least one canonical local tool.
assert "library" in prompt.lower()
- def test_both_scope_injects_no_policy_addendum(self):
- """Under BOTH (default), the strategy MUST NOT inject any of the
- three scope-specific marker phrases. Bleeding scope state into
- every prompt would (a) bloat the default-case prompt for no
- reason and (b) leak which scope the user picked even when they
- didn't restrict anything.
- """
- prompt = self._capture_prompt("both")
+ @pytest.mark.parametrize(
+ "primary,public_addendum",
+ [("brave", True), ("unclassified_test_primary", False)],
+ )
+ def test_legacy_both_scope_addendum_follows_primary(
+ self, primary, public_addendum
+ ):
+ prompt = self._capture_prompt("both", primary=primary)
assert "RESTRICTED MODE" not in prompt
assert "PRIVATE-ONLY MODE" not in prompt
- assert "PUBLIC-ONLY MODE" not in prompt
+ assert ("PUBLIC-ONLY MODE" in prompt) is public_addendum
# ---------------------------------------------------------------------------
@@ -6028,9 +6027,9 @@ class TestResearchSubtopicToolOverflow:
else {}
)
tool = _make_research_subtopic_tool(
- search_engine_name="duckduckgo",
+ search_engine_name="brave",
model=MagicMock(),
- settings_snapshot={"search.tool": {"value": "duckduckgo"}},
+ settings_snapshot={"search.tool": {"value": "brave"}},
collector=collector,
max_sub_iterations=8,
progress_callback=progress_callback,
@@ -6468,7 +6467,7 @@ class TestFinalizeCitationLogging:
model=MagicMock(),
search=MagicMock(),
all_links_of_system=all_links if all_links is not None else [],
- settings_snapshot={"search.tool": {"value": "duckduckgo"}},
+ settings_snapshot={"search.tool": {"value": "brave"}},
citation_handler=citation_handler,
)
@@ -7184,7 +7183,7 @@ class TestFinalizeSourcesPayload:
model=MagicMock(),
search=MagicMock(),
all_links_of_system=all_links if all_links is not None else [],
- settings_snapshot={"search.tool": {"value": "duckduckgo"}},
+ settings_snapshot={"search.tool": {"value": "brave"}},
citation_handler=MagicMock(),
)
@@ -8532,3 +8531,47 @@ class TestSearchToolThreadCleanup:
assert "t" in result
mock_cleanup.assert_not_called()
+
+
[email protected]("specialized", [False, True])
+def test_wayback_tool_description_states_accepted_input(specialized):
+ from local_deep_research.advanced_search_system.strategies.langgraph_agent_strategy import (
+ _make_specialized_search_tool,
+ _make_web_search_tool,
+ )
+ from local_deep_research.web_search_engines.engines.search_engine_wayback import (
+ WaybackSearchEngine,
+ )
+
+ if specialized:
+ search = _make_specialized_search_tool(
+ "wayback", "Search archives.", MagicMock(), {}, MagicMock()
+ )
+ else:
+ search = _make_web_search_tool("wayback", MagicMock(), {}, MagicMock())
+
+ assert "full http:// or https:// URL" in search.description
+ assert "or be only a bare domain" in search.description
+ assert "Only the host is looked up" in search.description
+ assert "configured search provider" in search.description
+ # It also names the inputs the patterns miss or misread; each form is
+ # pinned in test_wayback_archive_only.py.
+ for unsupported in (
+ "an uppercase scheme",
+ "a bracketed IPv6 host",
+ "a user name or password before the host",
+ "a bare domain with a punycode (xn--) top-level domain",
+ ):
+ assert unsupported in search.description
+ # The description is the parser's contract: the host of a full URL
+ # anywhere in the input, but a bare domain only when it is the whole
+ # input.
+ engine = WaybackSearchEngine()
+ assert engine._extract_urls_from_query("example.com") == [
+ "http://example.com"
+ ]
+ assert engine._extract_urls_from_query("Find example.com") == []
+ assert engine._extract_urls_from_query("example.com/page") == []
+ assert engine._extract_urls_from_query("Find https://example.com/page") == [
+ "https://example.com"
+ ]
diff --git a/tests/test_advertised_but_dead_sweep.py b/tests/test_advertised_but_dead_sweep.py
index e44a8d0c59..dd8743e6bb 100644
--- a/tests/test_advertised_but_dead_sweep.py
+++ b/tests/test_advertised_but_dead_sweep.py
@@ -1073,9 +1073,9 @@ def test_fallback_guard_rejects_unreadable_provider_id(monkeypatch):
reason=(
"KNOWN DEFECT, pre-existing. ENGINE_REGISTRY calls itself the "
"single source of truth for which class implements each engine, "
- "but 'ddg' and 'guardian' have no settings block, so "
+ "but 'guardian' has no settings block, so "
"search_config()'s 'if name in search_engines' guard never fires "
- "and the factory rejects both names. The classes are correct and "
+ "and the factory rejects that name. The class is correct and "
"importable; it is the offer that cannot be taken up. "
),
)
@@ -1083,10 +1083,10 @@ def test_engine_registry_entries_are_reachable():
"""Registry entries that name a real class but can never be selected.
engine_registry.py calls itself "the single source of truth for which
- Python module/class implements each search engine". Two of its entries —
- ``ddg`` and ``guardian`` — have no ``search.engine.web.<name>.*`` settings,
+ Python module/class implements each search engine". Its ``guardian`` entry
+ has no ``search.engine.web.guardian.*`` settings,
so ``search_engines_config``'s ``if name in search_engines`` guard never
- fires for them and the factory rejects both names.
+ fires for it and the factory rejects that name.
Category: caller-facing dead registry entry. The classes themselves are
correct and importable; it is the offer that cannot be taken up.
diff --git a/tests/ui_tests/NO_CI_executes_research_full_research_completion.js b/tests/ui_tests/NO_CI_executes_research_full_research_completion.js
index 3776815404..191cc14b7f 100644
--- a/tests/ui_tests/NO_CI_executes_research_full_research_completion.js
+++ b/tests/ui_tests/NO_CI_executes_research_full_research_completion.js
@@ -69,7 +69,7 @@ async function testCompleteResearch() {
model_provider: 'OLLAMA',
model: 'llama3.1:8b', // Use available model
custom_endpoint: '',
- search_engine: 'duckduckgo', // Use search engine that doesn't need API key
+ search_engine: 'wikipedia', // Use search engine that doesn't need API key
iterations: 1,
questions_per_iteration: 2,
strategy: 'source-based',
diff --git a/tests/ui_tests/test_check_search_engines.js b/tests/ui_tests/test_check_search_engines.js
index 7948cc27e8..80dea6a0d6 100644
--- a/tests/ui_tests/test_check_search_engines.js
+++ b/tests/ui_tests/test_check_search_engines.js
@@ -70,8 +70,8 @@ async function checkSearchEngines() {
});
}
- // Try to use duckduckgo which should work without API key
- console.log('\n🧪 Testing research with DuckDuckGo...');
+ // Wikipedia works without an API key.
+ console.log('\n🧪 Testing research with Wikipedia...');
const testResult = await page.evaluate(async () => {
const response = await fetch('/api/start_research', {
@@ -82,7 +82,7 @@ async function checkSearchEngines() {
mode: 'quick',
model_provider: 'OLLAMA',
model: 'llama3.1:8b',
- search_engine: 'duckduckgo', // Try duckduckgo
+ search_engine: 'wikipedia',
iterations: 1,
questions_per_iteration: 1
})
diff --git a/tests/web_search_engines/engines/test_search_engine_ddg.py b/tests/web_search_engines/engines/test_search_engine_ddg.py
deleted file mode 100644
index d382f059fb..0000000000
--- a/tests/web_search_engines/engines/test_search_engine_ddg.py
+++ /dev/null
@@ -1,381 +0,0 @@
-"""
-Tests for the DuckDuckGoSearchEngine class.
-
-Tests cover:
-- Initialization and configuration
-- Preview generation
-- Full content retrieval
-- Rate limit error handling
-- Run method
-"""
-
-from unittest.mock import Mock, patch
-import pytest
-
-
-class TestDuckDuckGoSearchEngineInit:
- """Tests for DuckDuckGoSearchEngine initialization."""
-
- def test_init_with_defaults(self):
- """Initialize with default values."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper:
- engine = DuckDuckGoSearchEngine()
-
- assert engine.max_results == 10
- assert engine.region == "us"
- assert engine.safe_search is True
- assert engine.language == "English"
- assert engine.include_full_content is False
- mock_wrapper.assert_called_once()
-
- def test_init_with_custom_max_results(self):
- """Initialize with custom max_results."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ):
- engine = DuckDuckGoSearchEngine(max_results=25)
-
- assert engine.max_results == 25
-
- def test_init_with_custom_region(self):
- """Initialize with custom region."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper:
- engine = DuckDuckGoSearchEngine(region="uk")
-
- assert engine.region == "uk"
- mock_wrapper.assert_called_with(
- region="uk", max_results=10, safesearch="moderate"
- )
-
- def test_init_with_safe_search_disabled(self):
- """Initialize with safe_search disabled."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper:
- engine = DuckDuckGoSearchEngine(safe_search=False)
-
- assert engine.safe_search is False
- mock_wrapper.assert_called_with(
- region="us", max_results=10, safesearch="off"
- )
-
- def test_init_with_llm(self):
- """Initialize with LLM."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- mock_llm = Mock()
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ):
- engine = DuckDuckGoSearchEngine(llm=mock_llm)
-
- assert engine.llm is mock_llm
-
- def test_init_does_not_create_full_search(self):
- """DDG no longer self-wraps in ``__init__``. The factory wrapper
- is the canonical fetcher; the inner engine only emits snippets."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- mock_llm = Mock()
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ):
- with patch(
- "local_deep_research.web_search_engines.engines.full_search.FullSearchResults"
- ) as mock_full_search:
- engine = DuckDuckGoSearchEngine(
- llm=mock_llm, include_full_content=True
- )
-
- assert not hasattr(engine, "full_search")
- mock_full_search.assert_not_called()
-
-
-class TestGetPreviews:
- """Tests for _get_previews method."""
-
- def test_get_previews_returns_results(self):
- """Get previews returns formatted results."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- mock_results = [
- {
- "title": "Result 1",
- "snippet": "Snippet 1",
- "link": "https://example1.com",
- },
- {
- "title": "Result 2",
- "snippet": "Snippet 2",
- "link": "https://example2.com",
- },
- ]
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper_class:
- mock_wrapper = Mock()
- mock_wrapper.results.return_value = mock_results
- mock_wrapper_class.return_value = mock_wrapper
-
- engine = DuckDuckGoSearchEngine()
- previews = engine._get_previews("test query")
-
- assert len(previews) == 2
- assert previews[0]["title"] == "Result 1"
- assert previews[0]["snippet"] == "Snippet 1"
- assert previews[0]["link"] == "https://example1.com"
- assert previews[0]["id"] == "https://example1.com"
-
- def test_get_previews_empty_results(self):
- """Get previews handles empty results."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper_class:
- mock_wrapper = Mock()
- mock_wrapper.results.return_value = []
- mock_wrapper_class.return_value = mock_wrapper
-
- engine = DuckDuckGoSearchEngine()
- previews = engine._get_previews("test query")
-
- assert previews == []
-
- def test_get_previews_non_list_results(self):
- """Get previews handles non-list results."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper_class:
- mock_wrapper = Mock()
- mock_wrapper.results.return_value = None
- mock_wrapper_class.return_value = mock_wrapper
-
- engine = DuckDuckGoSearchEngine()
- previews = engine._get_previews("test query")
-
- assert previews == []
-
- def test_get_previews_rate_limit_error(self):
- """Get previews raises RateLimitError on rate limit."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
- from local_deep_research.web_search_engines.rate_limiting import (
- RateLimitError,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper_class:
- mock_wrapper = Mock()
- mock_wrapper.results.side_effect = Exception(
- "202 Ratelimit exceeded"
- )
- mock_wrapper_class.return_value = mock_wrapper
-
- engine = DuckDuckGoSearchEngine()
-
- with pytest.raises(RateLimitError):
- engine._get_previews("test query")
-
- def test_get_previews_forbidden_error(self):
- """Get previews raises RateLimitError on 403 forbidden."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
- from local_deep_research.web_search_engines.rate_limiting import (
- RateLimitError,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper_class:
- mock_wrapper = Mock()
- mock_wrapper.results.side_effect = Exception("403 Forbidden")
- mock_wrapper_class.return_value = mock_wrapper
-
- engine = DuckDuckGoSearchEngine()
-
- with pytest.raises(RateLimitError):
- engine._get_previews("test query")
-
- def test_get_previews_timeout_error(self):
- """Get previews raises RateLimitError on timeout."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
- from local_deep_research.web_search_engines.rate_limiting import (
- RateLimitError,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper_class:
- mock_wrapper = Mock()
- mock_wrapper.results.side_effect = Exception("Connection timed out")
- mock_wrapper_class.return_value = mock_wrapper
-
- engine = DuckDuckGoSearchEngine()
-
- with pytest.raises(RateLimitError):
- engine._get_previews("test query")
-
- def test_get_previews_general_error(self):
- """Get previews returns empty on general error."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ) as mock_wrapper_class:
- mock_wrapper = Mock()
- mock_wrapper.results.side_effect = Exception("Some other error")
- mock_wrapper_class.return_value = mock_wrapper
-
- engine = DuckDuckGoSearchEngine()
- previews = engine._get_previews("test query")
-
- assert previews == []
-
-
-class TestGetFullContent:
- """Tests for _get_full_content method.
-
- Engines no longer self-wrap, so ``_get_full_content`` is a
- pass-through. The factory wrapper populates ``full_content`` after
- the inner engine runs.
- """
-
- def test_get_full_content_passes_through(self):
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ):
- engine = DuckDuckGoSearchEngine()
-
- items = [{"title": "Result", "link": "https://example.com"}]
- assert engine._get_full_content(items) is items
-
- def test_engine_does_not_self_wrap(self):
- """DDG must not carry a ``full_search`` attribute; the factory
- wrapper handles full-content retrieval."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ):
- engine = DuckDuckGoSearchEngine(include_full_content=True)
-
- assert not hasattr(engine, "full_search")
-
-
-class TestRun:
- """Tests for run method."""
-
- def test_run_calls_parent(self):
- """Run calls parent class run method."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ):
- engine = DuckDuckGoSearchEngine()
-
- with patch.object(
- engine,
- "_get_previews",
- return_value=[
- {
- "id": "https://example.com",
- "title": "Result",
- "snippet": "Snippet",
- "link": "https://example.com",
- }
- ],
- ):
- with patch.object(
- engine,
- "_get_full_content",
- return_value=[{"title": "Result"}],
- ):
- results = engine.run("test query")
-
- assert len(results) == 1
-
- def test_run_handles_empty_results(self):
- """Run handles empty results."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- with patch(
- "local_deep_research.web_search_engines.engines.search_engine_ddg.DuckDuckGoSearchAPIWrapper"
- ):
- engine = DuckDuckGoSearchEngine()
-
- with patch.object(engine, "_get_previews", return_value=[]):
- results = engine.run("test query")
-
- assert results == []
-
-
-class TestClassAttributes:
- """Tests for class attributes."""
-
- def test_is_public(self):
- """DuckDuckGoSearchEngine is marked as public."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- assert DuckDuckGoSearchEngine.is_public is True
-
- def test_is_generic(self):
- """DuckDuckGoSearchEngine is marked as generic."""
- from local_deep_research.web_search_engines.engines.search_engine_ddg import (
- DuckDuckGoSearchEngine,
- )
-
- assert DuckDuckGoSearchEngine.is_generic is True
diff --git a/tests/web_search_engines/engines/test_search_engine_wayback.py b/tests/web_search_engines/engines/test_search_engine_wayback.py
index a6d38b19a9..47be632d94 100644
--- a/tests/web_search_engines/engines/test_search_engine_wayback.py
+++ b/tests/web_search_engines/engines/test_search_engine_wayback.py
@@ -89,7 +89,7 @@ class TestExtractUrlsFromQuery:
# Note: The regex only captures domain, not path
urls = engine._extract_urls_from_query(
- "Check https://example.com archive"
+ "Check https://example.com/page archive"
)
assert len(urls) == 1
@@ -122,8 +122,8 @@ class TestExtractUrlsFromQuery:
assert len(urls) == 1
assert urls[0] == "http://example.com"
- def test_extract_partial_url(self):
- """Extract partial URL with path."""
+ def test_partial_url_without_scheme_is_not_extracted(self):
+ """A domain with a path but no scheme is not an archive target."""
from local_deep_research.web_search_engines.engines.search_engine_wayback import (
WaybackSearchEngine,
)
@@ -132,8 +132,7 @@ class TestExtractUrlsFromQuery:
urls = engine._extract_urls_from_query("example.com/page")
- assert len(urls) == 1
- assert urls[0] == "http://example.com/page"
+ assert urls == []
def test_extract_no_urls_found(self):
"""Return empty when no URLs found."""
@@ -143,12 +142,7 @@ class TestExtractUrlsFromQuery:
engine = WaybackSearchEngine()
- # Patch at the langchain_community location since it's imported inside the method
- with patch(
- "langchain_community.utilities.DuckDuckGoSearchAPIWrapper"
- ) as mock_ddg:
- mock_ddg.return_value.results.return_value = []
- urls = engine._extract_urls_from_query("just some text")
+ urls = engine._extract_urls_from_query("just some text")
assert urls == []
diff --git a/tests/web_search_engines/engines/test_search_engine_wayback_coverage.py b/tests/web_search_engines/engines/test_search_engine_wayback_coverage.py
index 8f67d5e470..0d1f9663ad 100644
--- a/tests/web_search_engines/engines/test_search_engine_wayback_coverage.py
+++ b/tests/web_search_engines/engines/test_search_engine_wayback_coverage.py
@@ -3,8 +3,8 @@ Coverage tests for WaybackSearchEngine targeting statements not hit by
the existing test_search_engine_wayback.py.
Covers:
-- _extract_urls_from_query: DuckDuckGo returns URLs; DuckDuckGo raises; query
- contains dot+slash (partial URL branch); query has dot but no slash (domain branch)
+- _extract_urls_from_query: plain text and partial URLs without a scheme
+ return no targets; a bare domain is accepted without provider discovery
- _format_timestamp: exception path returns raw timestamp
- _get_wayback_snapshots: CDX rate-limit (429); CDX response with only header row;
generic exception is swallowed and returns []
@@ -51,51 +51,16 @@ def _mock_response(status=200, json_body=None, text=""):
class TestExtractUrlsFromQueryExtra:
- def test_ddg_returns_urls(self):
- engine = _engine()
- ddg_results = [
- {"link": "https://found.com"},
- {"link": "https://also.com"},
+ def test_plain_text_has_no_archive_target(self):
+ assert _engine()._extract_urls_from_query("some plain text query") == []
+
+ def test_partial_url_without_scheme_has_no_archive_target(self):
+ assert _engine()._extract_urls_from_query("example.com/some/path") == []
+
+ def test_domain_without_protocol(self):
+ assert _engine()._extract_urls_from_query("example.org") == [
+ "http://example.org"
]
- with patch(
- "langchain_community.utilities.DuckDuckGoSearchAPIWrapper"
- ) as MockDDG:
- MockDDG.return_value.results.return_value = ddg_results
- urls = engine._extract_urls_from_query("some plain text query")
- assert "https://found.com" in urls
- assert "https://also.com" in urls
-
- def test_ddg_raises_falls_through_to_empty(self):
- engine = _engine()
- with patch(
- "langchain_community.utilities.DuckDuckGoSearchAPIWrapper"
- ) as MockDDG:
- MockDDG.return_value.results.side_effect = RuntimeError("ddg down")
- urls = engine._extract_urls_from_query("no dot here just words")
- assert urls == []
-
- def test_partial_url_dot_and_slash_branch(self):
- """query with '.' and '/' but no http prefix hits the dot+slash branch."""
- engine = _engine()
- with patch(
- "langchain_community.utilities.DuckDuckGoSearchAPIWrapper"
- ) as MockDDG:
- MockDDG.return_value.results.return_value = []
- urls = engine._extract_urls_from_query("example.com/some/path")
- assert urls == ["http://example.com/some/path"]
-
- def test_domain_only_dot_no_slash_branch(self):
- """query with '.' but no '/' hits the domain branch after DuckDuckGo fails."""
- engine = _engine()
- with patch(
- "langchain_community.utilities.DuckDuckGoSearchAPIWrapper"
- ) as MockDDG:
- # Results list has no 'link' values → ddg_urls will be empty
- MockDDG.return_value.results.return_value = [{"snippet": "no link"}]
- urls = engine._extract_urls_from_query("example.org")
- # The domain-without-http regex should have matched first and returned early
- # OR the dot-no-slash fallback fires. Either way the result includes the domain.
- assert any("example.org" in u for u in urls)
# ---------------------------------------------------------------------------
diff --git a/tests/web_search_engines/test_engine_success_path.py b/tests/web_search_engines/test_engine_success_path.py
index 777de16fb0..378a4dfa10 100644
--- a/tests/web_search_engines/test_engine_success_path.py
+++ b/tests/web_search_engines/test_engine_success_path.py
@@ -18,10 +18,10 @@ What this file pins that the per-engine unit tests do not:
1. **The whole registry, not a favourite subset.** Every entry in
``ENGINE_REGISTRY`` must resolve to an importable ``BaseSearchEngine``
- subclass, and must be constructible without a live backend. Three
- entries are not (``ddg``, ``elasticsearch``, ``google_pse``), one more
- constructs into a permanently empty engine (``searxng``), and two are
- unreachable through the factory at all (``ddg``, ``guardian``) --
+ subclass, and must be constructible without a live backend. Two
+ entries are not (``elasticsearch``, ``google_pse``), one more
+ constructs into a permanently empty engine (``searxng``), and one is
+ unreachable through the factory at all (``guardian``) --
each pinned below.
2. **"Usable", not "a list".** For each engine the first preview is
asserted field by field against the stubbed upstream document: the
@@ -77,9 +77,6 @@ from local_deep_research.web_search_engines.engines import (
from local_deep_research.web_search_engines.engines import (
search_engine_collection as collection_engine,
)
-from local_deep_research.web_search_engines.engines import (
- search_engine_ddg as ddg_engine,
-)
from local_deep_research.web_search_engines.engines import (
search_engine_elasticsearch as es_engine,
)
@@ -1924,9 +1921,9 @@ CASES_BY_NAME = {case.name: case for case in CASES}
# ``search.engine.web.*`` settings and only *decorates* the entries it finds
# with the registry's module/class, so an engine with no settings never
# enters the dict and ``create_search_engine`` fails closed on it with
-# "Unknown search engine". These two are therefore unreachable from the UI,
+# "Unknown search engine". This entry is therefore unreachable from the UI,
# the API and the LangGraph tool list alike.
-REGISTRY_ENGINES_WITHOUT_SETTINGS = {"ddg", "guardian"}
+REGISTRY_ENGINES_WITHOUT_SETTINGS = {"guardian"}
def _shipped_engine_names():
@@ -1984,15 +1981,15 @@ class TestRegistryIsWiredUp:
"case in this file"
)
- def test_ddg_and_guardian_are_unreachable_through_the_factory(self):
- """DEFECT: two registry entries no settings file defines.
+ def test_guardian_is_unreachable_through_the_factory(self):
+ """DEFECT: a registry entry no settings file defines.
``search_config()`` is settings-derived: it reads
``search.engine.web.*`` and only *decorates* the names it finds
with the registry's module/class. An engine with no shipped
settings therefore never enters the dict, and
``create_search_engine`` fails closed on it with "Unknown search
- engine". ``ddg`` and ``guardian`` are dead registry rows --
+ engine". ``guardian`` is a dead registry row --
unreachable from the UI, the API and the agent's tool list. A fix
either ships settings for them or drops the rows.
"""
@@ -2008,23 +2005,6 @@ class TestRegistryIsWiredUp:
class TestEnginesThatCannotBeConstructed:
"""Registry rows whose class cannot be built without a live backend."""
- def test_ddg_cannot_be_constructed_at_all(self):
- """DEFECT: the DuckDuckGo engine is dead on a stock install.
-
- ``pyproject.toml`` pins ``duckduckgo-search~=8.1``, but
- ``langchain-community~=0.4`` (0.4.2 here) migrated
- ``DuckDuckGoSearchAPIWrapper`` onto the renamed ``ddgs``
- distribution, which nothing declares. Constructing the wrapper --
- which ``DuckDuckGoSearchEngine.__init__`` does eagerly -- raises
- ImportError, so the engine can never run. The blast radius is
- wider than the ``ddg`` row itself: ``WaybackSearchEngine`` uses
- the same wrapper to turn a plain question into URLs (see below).
- """
- with pytest.raises(ImportError, match="ddgs"):
- ddg_engine.DuckDuckGoSearchEngine(
- max_results=3, llm=None, settings_snapshot=SNAPSHOT
- )
-
def test_elasticsearch_raises_when_the_cluster_is_down(self, inert_tracker):
"""Elasticsearch probes the cluster inside ``__init__``.
@@ -2372,15 +2352,10 @@ class TestDownstreamSilentDrops:
def test_wayback_returns_nothing_for_a_plain_research_question(
self, inert_tracker
):
- """DEFECT: Wayback is unusable for ordinary queries.
-
- ``_extract_urls_from_query`` needs a URL. For a normal question
- it falls back to ``DuckDuckGoSearchAPIWrapper`` to discover
- URLs -- the wrapper that cannot even be constructed on this
- install (see ``test_ddg_cannot_be_constructed_at_all``). The
- remaining heuristics only fire on strings containing a dot, so a
- research question yields zero snapshots. Wayback ships with
- settings and is offered in the UI regardless.
+ """Wayback looks up hosts of supplied URLs; topics need discovery.
+
+ The same adapter and archive payload produce usable results for
+ explicit URLs, while plain topics make no archive request.
"""
engine = _wire(
wayback_engine.WaybackSearchEngine(
diff --git a/tests/web_search_engines/test_search_engine_factory.py b/tests/web_search_engines/test_search_engine_factory.py
index fb3d0fc554..1f8cb302a2 100644
--- a/tests/web_search_engines/test_search_engine_factory.py
+++ b/tests/web_search_engines/test_search_engine_factory.py
@@ -342,7 +342,7 @@ class TestGetSearch:
)
result = get_search(
- search_tool="duckduckgo",
+ search_tool="brave",
llm_instance=mock_llm,
max_results=10,
settings_snapshot={"test": "value"},
@@ -351,8 +351,8 @@ class TestGetSearch:
mock_create.assert_called_once()
assert result == mock_engine
- def test_get_search_with_duckduckgo_params(self):
- """Test get_search with DuckDuckGo specific params."""
+ def test_get_search_with_brave_params(self):
+ """Test get_search with Brave specific params."""
mock_llm = Mock()
mock_engine = Mock()
@@ -366,7 +366,7 @@ class TestGetSearch:
)
get_search(
- search_tool="duckduckgo",
+ search_tool="brave",
llm_instance=mock_llm,
max_results=20,
region="uk",
@@ -699,7 +699,7 @@ class TestGetSearch:
mock_create.return_value = Mock()
get_search(
- search_tool="duckduckgo",
+ search_tool="brave",
llm_instance=mock_llm,
max_results=10,
region="uk",