~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Search Engines Guide
Local Deep Research integrates with multiple search engines to provide comprehensive research capabilities. This guide covers all available search engines, their specializations, and configuration details.
Note: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via GitHub Issues.
Overview
LDR supports three categories of search engines:
- Free Search Engines - No API key required
- Premium Search Engines - Require API keys but offer enhanced features
- Custom Sources - Your own documents and databases
Search Engine Selection
Dynamic Engine Selection (Recommended)
The default langgraph-agent strategy selects the most appropriate engines dynamically per query: every enabled query-search engine (and any registered retriever or local collection) is exposed to the research agent as a tool, and the agent decides which to call for each sub-question. Wayback is available separately through fetch_archive for known URLs. You only pick a primary engine (searxng is the recommended default):
result = quick_summary(
query="What are the latest advances in quantum computing?",
search_tool="searxng" # Primary engine; the langgraph-agent strategy
# can still pull in other enabled engines per query
)
Note: The former
autoandparallelmeta engines were removed — the langgraph-agent strategy replaces them. Stored settings are migrated automatically; explicitsearch_tool="auto"callers should switch to a concrete engine likesearxng.
Free Search Engines
Academic Search Engines
arXiv
- Specialization: Scientific papers and preprints
- Best for: Physics, mathematics, computer science, biology
- Results: Direct access to research papers
- Rate Limit: Moderate - automatic retry on limits
PubMed
- Specialization: Biomedical and life science literature
- Best for: Medical research, clinical studies, biology
- Results: Abstracts and links to full papers
- Rate Limit: Generous - rarely hits limits
Semantic Scholar
- Specialization: Academic literature across all fields
- Best for: Cross-disciplinary research, citation networks
- Results: Paper summaries with citation context
- Rate Limit: Moderate - adaptive rate limiting handles this
General Purpose
Wikipedia
- Specialization: General knowledge and encyclopedic information
- Best for: Background information, concepts, facts
- Results: Well-structured article content
- Rate Limit: Very generous
SearXNG (Highly Recommended)
- Specialization: Meta-search engine aggregating multiple sources
- Best for: Comprehensive web search with privacy
- Results: Aggregated results from Google, Bing, DuckDuckGo, etc.
- Setup:
docker pull searxng/searxng docker run -d -p 8080:8080 --name searxng searxng/searxng - Configuration: Set URL to
http://localhost:8080in settings - Note (v1.10.3+): a localhost/LAN URL additionally requires operator
approval in the server environment — its origin listed in
LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST=http://localhost:8080(recommended, v1.10.5+), env-locking the URL (as the bundled docker-compose does), or the blanketLDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true— only one is needed — see SearXNG-Setup
Technical Search
GitHub
- Specialization: Code repositories and documentation
- Best for: Finding code examples, libraries, technical solutions
- Results: Repository information, code snippets, issues
- Rate Limit: Moderate when unauthenticated
Elasticsearch
- Specialization: Custom search within your Elasticsearch cluster
- Best for: Searching your own indexed data
- Configuration: See Elasticsearch Setup Guide
Archive Fetching
Wayback Machine
- Specialization: Historical web content
- Best for: Finding deleted content, tracking changes over time
- Results: Archived web pages with timestamps
- Rate Limit: Moderate
- Input: A page URL and an optional date in
YYYY-MM-DDformat - Agent tool:
fetch_archive, available to the lead LangGraph research agent and its subagents when archive fetching is enabled
For a topic query, first find page URLs with your configured search provider, such as SearXNG, then fetch the historical version of a chosen URL:
# Full mode, after the configured search provider finds this URL:
fetch_archive("https://example.org/policy", date="2020-01-01")
fetch_archive("https://example.org/policy") # Latest available capture
# Summary modes require a focus question:
fetch_archive("https://example.org/policy", date="2020-01-01",
focus="What did the policy require at that time?")
These are agent tool calls. A requested date selects the nearest available
capture, which may fall before or after that date. The result preserves the
original URL, archive URL, and actual capture timestamp in UTC; cite the actual
capture date rather than the requested date. With the date omitted or None, the tool selects
the latest available capture. An unarchived URL may have no capture.
Archive fetch reads saved content statically and refuses redirects to live pages or outside the selected capture, keeping the returned metadata tied to the capture actually read.
Enable or disable this tool with search.engine.web.wayback.agent_enabled
(Enable Archive Fetch in Settings). It uses the same search.fetch.mode
as fetch_content: full returns extracted text without a focus argument,
summary_focus and summary_focus_query require focus and return focused
summaries, and disabled removes both
fetch tools. search.fetch.content_max_length also applies to archive text.
For each lead agent or subagent, archive and live-page fetch calls share the
search.fetch.max_per_topic budget for the topic. Archive requests still
follow the configured egress policy.
Wayback is excluded from the primary search picker and topic/auto search.
Saved search.tool=wayback values are not rewritten: choose a configured search
provider in Settings for topic queries. The research form keeps an unlisted saved
choice visible with a notice and requires you to choose a configured provider
before starting research. Existing Wayback settings remain available, and the legacy
WaybackSearchEngine class and engine factory registration still support
programmatic URL or domain lookups, optionally with a path. Its
search.engine.web.wayback.default_params.* settings configure that legacy
adapter; they do not control fetch_archive. Wayback does not discover URLs
through another search provider.
The legacy adapter looks up each http:// or https:// URL in its query,
with the scheme in any case, or a query that is only a bare domain,
optionally with a port, path, query string or fragment. It sends the
complete URL, including its port, path, query string and fragment, to the
Internet Archive and keeps it in its results. A comma or semicolon, ASCII
or fullwidth, or an ideographic comma right before another URL separates
the two, except in a query string, in a fragment, or in the path of a URL
in straight quotes, parentheses, square or angle brackets, or backticks. A
query that is only one URL, and a URL in straight quotes, angle brackets or
backticks, keep their punctuation. Curly quotes and guillemets do not
delimit a URL: it follows the running-text rules below, and a German
closing quote stays part of it, as in „https://example.com/a“.
Otherwise, in running text the adapter drops what ends a URL rather than
belongs to it: sentence punctuation (ASCII, CJK or fullwidth, and an
ellipsis), Markdown * emphasis, straight and curly closing quotes and
closing guillemets, and a closing bracket without its opening partner in
the URL, whether ASCII (), ] or }), fullwidth (such as )) or CJK
(such as 】 or 」). A balanced pair stays, as in
https://ja.wikipedia.org/wiki/【推しの子】 or
https://zh.wikipedia.org/wiki/苹果(水果). A URL right after ( or [
ends at its matching closer and drops only trailing commas and semicolons.
Known limits: words that follow a URL without a space stay part of it, such
as a possessive 's, CJK text, a parenthetical as in
https://example.com(see note), a second URL, or the label of a
reference-style link as in [text https://example.com/a][1]; put a space
after the URL. In running text, a URL that ends in 。, …, * or '
loses that character, as in https://ja.wikipedia.org/wiki/君の名は。, a
query ending in ?a=*, or a Wayback-style
https://web.archive.org/web/*/example.com/*; give such a URL alone or in
straight quotes, angle brackets or backticks. Bracket balance is counted over
the whole URL, not by position, so a stray closer earlier in a URL makes a
balanced pair at its end look unmatched: in running text
https://example.com/a)(b) becomes https://example.com/a)(b. A URL right
after ( or [ keeps a fullwidth or CJK closer at its end, as in
[https://example.com/a], because only the matching ASCII closer ends it
there. A bare domain written with combining marks is not recognised; give
it as a full URL.
Like fetch_archive, the adapter skips a URL with a user name or password
before its host, an invalid port or no host; a credential inside a query
value is sent as written. It looks up at most max_results URLs per query.
The configured egress policy allows or blocks the adapter as a whole;
unlike fetch_archive, it does not also check each target URL against that
policy.
With Wayback saved as the primary engine, the LangGraph agent omits the
generic web_search tool for Wayback; it can still use enabled archive
fetching and other allowed configured tools.
News Search
The Guardian
- Specialization: News articles and journalism
- Best for: Current events, news analysis
- Results: Recent news articles
- Note: Requires API key (free tier available at https://open-platform.theguardian.com/)
Wikinews
- Specialization: Open and collaboratively-written news articles on a wide range of topics
- Best for: Historical and recent news, general news coverage, quick overviews
- Results: News articles written by volunteers with verified sources
Premium Search Engines
Tavily
- Specialization: AI-optimized search for LLM applications
- Best for: High-quality, relevant results for AI research
- Pricing: Free tier available, paid plans for higher volume
- Configuration:
# In the server environment or via the web interface LDR_SEARCH_ENGINE_TAVILY_API_KEY=your-key-here
Google (via SerpAPI)
- Specialization: Comprehensive web search
- Best for: Most current and comprehensive results
- Pricing: Paid service with free trial
- Configuration:
LDR_SEARCH_ENGINE_WEB_SERPAPI_API_KEY=your-key-here
Google Programmable Search Engine
- Specialization: Customizable Google search
- Best for: Searching specific sites or topics
- Pricing: Free tier with limits
- Configuration:
LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_API_KEY=your-key-here LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_ENGINE_ID=your-engine-id
Brave Search
- Specialization: Independent search index with privacy focus
- Best for: Web search without big tech tracking
- Pricing: Free tier available
- Configuration:
LDR_SEARCH_ENGINE_WEB_BRAVE_API_KEY=your-key-here
Custom Sources
Local Documents
- Specialization: Search your private documents
- Supported formats: PDF, TXT, MD, DOCX, CSV, and more
- Configuration: See Configuring Local Search
- Setup:
- Go to Settings → Search for "local"
- Add document collection paths
- Choose embedding model (CPU or Ollama)
- First search will index documents
LangChain Retrievers
- Specialization: Any vector store or database
- Supported: FAISS, Chroma, Pinecone, Weaviate, Elasticsearch
- Configuration: See LangChain Integration Guide
Search Performance Comparison
| Engine | Speed | Quality | Privacy | Rate Limits |
|---|---|---|---|---|
| SearXNG | ★★★★★ | ★★★★☆ | ★★★★★ | ★★★★★ |
| Wikipedia | ★★★★☆ | ★★★★☆ | ★★★★★ | ★★★★★ |
| arXiv | ★★★★☆ | ★★★★★ | ★★★★★ | ★★★☆☆ |
| PubMed | ★★★★☆ | ★★★★★ | ★★★★★ | ★★★★☆ |
| Tavily | ★★★★☆ | ★★★★★ | ★★★☆☆ | ★★★★☆ |
| Google (SerpAPI) | ★★★★☆ | ★★★★★ | ★★☆☆☆ | ★★★★★ |
| Local Documents | ★★★☆☆ | ★★★★★ | ★★★★★ | ★★★★★ |
Rate Limiting and Reliability
LDR includes intelligent adaptive rate limiting that:
- Learns optimal wait times for each engine
- Automatically retries failed requests
- Prevents your IP from being blocked
- Maintains high reliability
Managing Rate Limits
Rate-limit statistics (per-engine wait times and success rates) are shown
in the Rate Limiting section of the metrics dashboard
(/metrics). The learned wait times can be tuned with the rate limiting
profile setting (conservative, balanced, aggressive).
Search Strategies
LDR supports multiple search strategies that determine how queries are processed:
- langgraph-agent: Agentic research that picks engines dynamically per query (default)
- source-based: Single query, fast results
- focused_iteration: Iterative refinement for accuracy
Best Practices
- For General Research: Use
searxngwith the default langgraph-agent strategy - For Academic Research: Combine
arxiv,pubmed, andsemantic_scholar - For Technical Questions: Use
githubwithsearxng - For Maximum Privacy: Use
searxngwith local Ollama models - For Best Quality: Use
tavilyor Google withfocused_iterationstrategy
Troubleshooting
SearXNG Not Working
- Check the LDR logs for
SearXNG engine disabled: instance URL … is a private / loopback / link-local address— if present, add the URL origin toLDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST(or setLDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true) in the server environment and restart (see SearXNG-Setup) - Verify container is running:
docker ps | grep searxng - Check URL in settings:
http://localhost:8080 - Test directly:
curl http://localhost:8080 - Check the logs:
docker logs searxngor view them in the LDR web UI
Rate Limit Errors
- Wait a few minutes and retry
- Use the langgraph-agent strategy, which can route around rate-limited engines
- Consider adding premium engines for higher limits
Temporarily Disabled Engines
During LangGraph research, an engine that reports a failure is removed from subsequent model tool choices. The lead agent and its subagents share this state for the research run. Calls already proposed are checked again before engine creation, so a disabled engine is not repeatedly initialized or queried. Engines on cooldown are also excluded from discovery-time service probes. Subagent instructions refer to the current tool schema rather than a fixed list of engine names that could become unavailable mid-run. Searches are never serialized: parallel searches of one engine, including its first calls and retries after a cooldown, run concurrently, and the first failure disables the engine for all of them. Searches already in flight may finish.
When every search engine the agent (or a subagent) could use is cooling down, the next model call waits for the soonest cooldown to end, checking for cancellation every second. Tools come back as soon as their cooldown ends, at any step of the run, not only when a run or report section starts. The research log says how long the next wait is expected to take.
Cooldown waits in one run share a one-hour budget by default; parallel
subagents waiting at the same time spend it once. Set
langgraph_agent.cooldown_wait_budget_seconds to allow longer waits (or to
zero to skip waiting). This budget covers only waiting for search engines; it
does not limit time spent searching, running the LLM, or writing the report.
Subagents have a separate two-hour wall-clock deadline by default, adjustable
with langgraph_agent.subagent_timeout_seconds. Its effective value is at
least the cooldown wait budget plus 30 minutes, so a subagent can complete a
full cooldown wait and still have time to work. Cancelling the research stops
the lead agent waiting for running subagents within a minute, not at their
deadline, also when it waits for several subagent batches at once. Each of
those subagents stops before its next model call, so one in the middle of a
model call or a tool call (such as a slow search or page fetch) can take until
that call returns. A subagent already past its deadline is no longer waited
for and may keep running until it finishes, even after a cancel.
For a local model that is silent for more than 30 minutes on one request,
also raise llm.request_timeout; that setting is separate from research
cooldown waits and subagent deadlines.
Temporary failures and provider Retry-After values keep their cooldowns,
even after repeated failures. When the next cooldown cannot finish within the
remaining wait budget, the cooling engines are disabled for the rest of the
run. A Retry-After longer than a day (or longer than the wait budget, if that
is larger) is not treated as a cooldown at all: the engine is disabled for the
rest of the run at once, with that reason.
Once that leaves no search engine, the rules below for engines disabled for the rest of the run apply.
If every search engine is disabled for the rest of the run but page fetching
is enabled, the agent keeps only fetch_content: it can still read URLs from
the query or from earlier results, but it cannot find new sources and may
answer without searching (it is told to say when evidence is insufficient). If
every research tool is disabled for the rest of the run, the model is not
called without tools: the research (or the current detailed-report section)
stops, and the answer is written from the sources gathered so far. A section
stopped before gathering any sources is rendered as “Limited information was
found for this subsection.”; the research log records the stop as an error
line, while overall report progress continues.
Availability also propagates through the optional full-content search wrapper.
The shared search boundary classifies primary request failures for web search providers, registered retrievers, and local document-store engines. Adapters propagate failures to this boundary instead of converting them into successful empty results. Initialization failures also retain their classification when the agent creates an engine, so an access refusal is not converted to a generic retry. Successful searches with no matches and optional enrichment failures retain their existing behavior. The web worker carries a failed primary initialization into the tracker, so that engine is excluded before the first model call as well as later calls.
The research log records the reason, disable time, and retry time. The
“selecting next action” status lists the available tools and explains disabled
engines separately. A search skipped because its engine is disabled or cooling
down is recorded in search metrics with the status skipped_unavailable and
the reason: the Metrics dashboard and the research details page show these
skips per engine, separately from searches that were sent, so they do not
change call counts, response times or success rates. Cooldowns expire automatically; the engine then becomes
eligible for another search, and the log records that retry is available.
The shared tracker reports each availability transition once across the lead
agent and subagents. Repeated skipped calls log at debug level. A cooldown
expiry permits a retry; it does not claim the provider has recovered.
- Rate limits: use the provider's
Retry-Aftervalue when reported, otherwise wait five minutes after the engine's retries are exhausted. ARetry-Afterlonger than a day (or than the wait budget, if that is larger), here or for other temporary failures below, disables the engine for the rest of the run instead. Retries may happen in the HTTP client rather than the search pipeline: Semantic Scholar's session retries HTTP 429 itself, honoringRetry-Afterup to 30 seconds per retry (at most five retries), and when those retries run out the engine is disabled for five minutes once its fallback queries are also used up. Exhausted retries on HTTP 5xx count as an ordinary search failure. Any non-JSON Wikipedia response (normally its HTML 429 page) is treated as a rate limit, because its client library discards the HTTP status; a non-JSON server error therefore also waits five minutes. - Connection failures, timeouts, and other search failures: use the provider's
Retry-Afterwhen reported, otherwise wait one minute. - A request URL refused by security validation: wait five minutes. The validator also refuses hosts that cannot be resolved, so a DNS outage is retried rather than treated as a configuration error.
- HTTP 403: use the provider's
Retry-Afterwhen reported, otherwise wait five minutes, because it is often a temporary bot or firewall block. A 403 whose reply is clearly about credentials (it asks for authentication or mentions an API key, token, subscription or billing) is treated as denied credentials below. So is any 403 from Semantic Scholar when an API key is configured: it rejects an invalid key with a plain{"message":"Forbidden"}reply. SerpAPI's account permission refusals are also run-long failures. Google PSE's structured quota-limit responses stay temporary and retain their retry delay; credential refusals remain disabled for the run. - Denied credentials/access (HTTP 401/402, or such a 403) or a private/loopback SearXNG instance URL without server approval, when the engine reports such a failure: disable for the rest of the research. Correct the configuration before starting another run. This does not change saved engine settings.
- A successful search with no matches is still available. Usable partial results also keep an engine available, and so does a search whose results all arrived but were then dropped by the relevance filter (or failed a later filtering step): the provider answered.
- Errors specific to one query do not disable the engine: a query the provider rejects as malformed (HTTP 400, 414 or 422) is reported back to the agent to rephrase, and a non-JSON Wayback Machine reply for one URL (such as its "blocked site" page) counts only against that URL. SearXNG listing unresponsive backends alongside no results counts as no matches, because it does not report which backends were queried.
Availability tracking lives in web_search_engines/engine_availability.py and
has no LangGraph dependency. advanced_search_system/tools/search_availability.py
adapts it to the agent. BaseSearchEngine._search_previews is the common
boundary for provider retrieval: it calls the adapter's _get_previews,
classifies exceptions, preserves retry metadata, and records a safe reason.
Public search methods use this boundary before relevance filtering or optional
content enrichment. Outcomes belong to each invocation, so concurrent calls
and nested searches on one engine cannot overwrite one another's failures.
When implementing an adapter:
- Return a list from
_get_previews. Return[]only when a successful search has no matches. Propagate transport, provider, and malformed-response errors; do not catch them just to return[]or manually set availability state. RaiseSearchQueryErrorfor local query validation failures so other queries can still use the engine. - Preserve usable partial results if a later optional request fails. For
fallback sequences,
self._search_attempts()providesrun(operation, ...)andfinish(results): a successful empty attempt supersedes an earlier error, usable results survive a later error, and a final failure without results propagates to the common boundary. - Preserve the original HTTP/SDK exception when it carries status or retry
timing. The common boundary reads responses and SDK metadata. If an SDK
discards that information, translate its provider-specific refusal into
SearchEngineError(SearchFailure(...))before it is lost. Safe reasons must not contain response bodies, request URLs, or credentials. Structured HTTP statuses take priority over text-only SDK rate-limit messages. - Keep optional enrichment failures separate from primary search health. Policy denials, invalid settings, and research cancellation must propagate.
Constructor probes can still record an initial failure through
_record_search_failure; it remains visible when the engine is handed to
another worker. Primary adapters do not need this manual reporting hook.
Agent callers use create_search_engine(..., raise_on_error=True) to preserve
safe, structured initialization failures for the same tracker. Direct factory
callers retain the default None result for failed construction.
No Results Found
- Try different search engines
- Broaden your query
- Check internet connectivity
- Verify API keys for premium engines
Advanced Configuration
Configuring Search Engines
You can enable/disable specific search engines and adjust their reliability parameters in the settings. This affects which engines the langgraph-agent strategy can choose from and how the system handles rate limiting.
Multi-Engine Research
The former auto and parallel meta engines (which fanned a query out over several engines) have been removed. To research across multiple engines, use the default langgraph-agent strategy: it calls any enabled engine as a tool, in parallel where useful, and picks per sub-question which engines to query.