Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

26.4 KB
Raw Blame

Search Engines Guide

Local Deep Research integrates with multiple search engines to provide comprehensive research capabilities. This guide covers all available search engines, their specializations, and configuration details.

Note: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via GitHub Issues.

Overview

LDR supports three categories of search engines:

  • Free Search Engines - No API key required
  • Premium Search Engines - Require API keys but offer enhanced features
  • Custom Sources - Your own documents and databases

Search Engine Selection

Dynamic Engine Selection (Recommended)

The default langgraph-agent strategy selects the most appropriate engines dynamically per query: every enabled query-search engine (and any registered retriever or local collection) is exposed to the research agent as a tool, and the agent decides which to call for each sub-question. Wayback is available separately through fetch_archive for known URLs. You only pick a primary engine (searxng is the recommended default):

result = quick_summary(
    query="What are the latest advances in quantum computing?",
    search_tool="searxng"  # Primary engine; the langgraph-agent strategy
                           # can still pull in other enabled engines per query
)

Note: The former auto and parallel meta engines were removed — the langgraph-agent strategy replaces them. Stored settings are migrated automatically; explicit search_tool="auto" callers should switch to a concrete engine like searxng.

Free Search Engines

Academic Search Engines

arXiv

  • Specialization: Scientific papers and preprints
  • Best for: Physics, mathematics, computer science, biology
  • Results: Direct access to research papers
  • Rate Limit: Moderate - automatic retry on limits

PubMed

  • Specialization: Biomedical and life science literature
  • Best for: Medical research, clinical studies, biology
  • Results: Abstracts and links to full papers
  • Rate Limit: Generous - rarely hits limits

Semantic Scholar

  • Specialization: Academic literature across all fields
  • Best for: Cross-disciplinary research, citation networks
  • Results: Paper summaries with citation context
  • Rate Limit: Moderate - adaptive rate limiting handles this

General Purpose

Wikipedia

  • Specialization: General knowledge and encyclopedic information
  • Best for: Background information, concepts, facts
  • Results: Well-structured article content
  • Rate Limit: Very generous

SearXNG (Highly Recommended)

  • Specialization: Meta-search engine aggregating multiple sources
  • Best for: Comprehensive web search with privacy
  • Results: Aggregated results from Google, Bing, DuckDuckGo, etc.
  • Setup:
    docker pull searxng/searxng
    docker run -d -p 8080:8080 --name searxng searxng/searxng
    
  • Configuration: Set URL to http://localhost:8080 in settings
  • Note (v1.10.3+): a localhost/LAN URL additionally requires operator approval in the server environment — its origin listed in LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST=http://localhost:8080 (recommended, v1.10.5+), env-locking the URL (as the bundled docker-compose does), or the blanket LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true — only one is needed — see SearXNG-Setup

Technical Search

GitHub

  • Specialization: Code repositories and documentation
  • Best for: Finding code examples, libraries, technical solutions
  • Results: Repository information, code snippets, issues
  • Rate Limit: Moderate when unauthenticated

Elasticsearch

  • Specialization: Custom search within your Elasticsearch cluster
  • Best for: Searching your own indexed data
  • Configuration: See Elasticsearch Setup Guide

Archive Fetching

Wayback Machine

  • Specialization: Historical web content
  • Best for: Finding deleted content, tracking changes over time
  • Results: Archived web pages with timestamps
  • Rate Limit: Moderate
  • Input: A page URL and an optional date in YYYY-MM-DD format
  • Agent tool: fetch_archive, available to the lead LangGraph research agent and its subagents when archive fetching is enabled

For a topic query, first find page URLs with your configured search provider, such as SearXNG, then fetch the historical version of a chosen URL:

# Full mode, after the configured search provider finds this URL:
fetch_archive("https://example.org/policy", date="2020-01-01")
fetch_archive("https://example.org/policy")  # Latest available capture

# Summary modes require a focus question:
fetch_archive("https://example.org/policy", date="2020-01-01",
              focus="What did the policy require at that time?")

These are agent tool calls. A requested date selects the nearest available capture, which may fall before or after that date. The result preserves the original URL, archive URL, and actual capture timestamp in UTC; cite the actual capture date rather than the requested date. With the date omitted or None, the tool selects the latest available capture. An unarchived URL may have no capture.

Archive fetch reads saved content statically and refuses redirects to live pages or outside the selected capture, keeping the returned metadata tied to the capture actually read.

Enable or disable this tool with search.engine.web.wayback.agent_enabled (Enable Archive Fetch in Settings). It uses the same search.fetch.mode as fetch_content: full returns extracted text without a focus argument, summary_focus and summary_focus_query require focus and return focused summaries, and disabled removes both fetch tools. search.fetch.content_max_length also applies to archive text. For each lead agent or subagent, archive and live-page fetch calls share the search.fetch.max_per_topic budget for the topic. Archive requests still follow the configured egress policy.

Wayback is excluded from the primary search picker and topic/auto search. Saved search.tool=wayback values are not rewritten: choose a configured search provider in Settings for topic queries. The research form keeps an unlisted saved choice visible with a notice and requires you to choose a configured provider before starting research. Existing Wayback settings remain available, and the legacy WaybackSearchEngine class and engine factory registration still support programmatic URL or domain lookups, optionally with a path. Its search.engine.web.wayback.default_params.* settings configure that legacy adapter; they do not control fetch_archive. Wayback does not discover URLs through another search provider.

The legacy adapter looks up each http:// or https:// URL in its query, with the scheme in any case, or a query that is only a bare domain, optionally with a port, path, query string or fragment. It sends the complete URL, including its port, path, query string and fragment, to the Internet Archive and keeps it in its results. A comma or semicolon, ASCII or fullwidth, or an ideographic comma right before another URL separates the two, except in a query string, in a fragment, or in the path of a URL in straight quotes, parentheses, square or angle brackets, or backticks. A query that is only one URL, and a URL in straight quotes, angle brackets or backticks, keep their punctuation. Curly quotes and guillemets do not delimit a URL: it follows the running-text rules below, and a German closing quote stays part of it, as in „https://example.com/a“. Otherwise, in running text the adapter drops what ends a URL rather than belongs to it: sentence punctuation (ASCII, CJK or fullwidth, and an ellipsis), Markdown * emphasis, straight and curly closing quotes and closing guillemets, and a closing bracket without its opening partner in the URL, whether ASCII (), ] or }), fullwidth (such as )) or CJK (such as 】 or 」). A balanced pair stays, as in https://ja.wikipedia.org/wiki/【推しの子】 or https://zh.wikipedia.org/wiki/苹果(水果). A URL right after ( or [ ends at its matching closer and drops only trailing commas and semicolons.

Known limits: words that follow a URL without a space stay part of it, such as a possessive 's, CJK text, a parenthetical as in https://example.com(see note), a second URL, or the label of a reference-style link as in [text https://example.com/a][1]; put a space after the URL. In running text, a URL that ends in 。, …, * or ' loses that character, as in https://ja.wikipedia.org/wiki/君の名は。, a query ending in ?a=*, or a Wayback-style https://web.archive.org/web/*/example.com/*; give such a URL alone or in straight quotes, angle brackets or backticks. Bracket balance is counted over the whole URL, not by position, so a stray closer earlier in a URL makes a balanced pair at its end look unmatched: in running text https://example.com/a)(b) becomes https://example.com/a)(b. A URL right after ( or [ keeps a fullwidth or CJK closer at its end, as in [https://example.com/a], because only the matching ASCII closer ends it there. A bare domain written with combining marks is not recognised; give it as a full URL.

Like fetch_archive, the adapter skips a URL with a user name or password before its host, an invalid port or no host; a credential inside a query value is sent as written. It looks up at most max_results URLs per query. The configured egress policy allows or blocks the adapter as a whole; unlike fetch_archive, it does not also check each target URL against that policy.

With Wayback saved as the primary engine, the LangGraph agent omits the generic web_search tool for Wayback; it can still use enabled archive fetching and other allowed configured tools.

News Search

The Guardian

  • Specialization: News articles and journalism
  • Best for: Current events, news analysis
  • Results: Recent news articles
  • Note: Requires API key (free tier available at https://open-platform.theguardian.com/)

Wikinews

  • Specialization: Open and collaboratively-written news articles on a wide range of topics
  • Best for: Historical and recent news, general news coverage, quick overviews
  • Results: News articles written by volunteers with verified sources

Premium Search Engines

Tavily

  • Specialization: AI-optimized search for LLM applications
  • Best for: High-quality, relevant results for AI research
  • Pricing: Free tier available, paid plans for higher volume
  • Configuration:
    # In the server environment or via the web interface
    LDR_SEARCH_ENGINE_TAVILY_API_KEY=your-key-here
    

Google (via SerpAPI)

  • Specialization: Comprehensive web search
  • Best for: Most current and comprehensive results
  • Pricing: Paid service with free trial
  • Configuration:
    LDR_SEARCH_ENGINE_WEB_SERPAPI_API_KEY=your-key-here
    

Google Programmable Search Engine

  • Specialization: Customizable Google search
  • Best for: Searching specific sites or topics
  • Pricing: Free tier with limits
  • Configuration:
    LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_API_KEY=your-key-here
    LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_ENGINE_ID=your-engine-id
    

Brave Search

  • Specialization: Independent search index with privacy focus
  • Best for: Web search without big tech tracking
  • Pricing: Free tier available
  • Configuration:
    LDR_SEARCH_ENGINE_WEB_BRAVE_API_KEY=your-key-here
    

Custom Sources

Local Documents

  • Specialization: Search your private documents
  • Supported formats: PDF, TXT, MD, DOCX, CSV, and more
  • Configuration: See Configuring Local Search
  • Setup:
    1. Go to Settings → Search for "local"
    2. Add document collection paths
    3. Choose embedding model (CPU or Ollama)
    4. First search will index documents

LangChain Retrievers

  • Specialization: Any vector store or database
  • Supported: FAISS, Chroma, Pinecone, Weaviate, Elasticsearch
  • Configuration: See LangChain Integration Guide

Search Performance Comparison

Engine Speed Quality Privacy Rate Limits
SearXNG ★★★★★ ★★★★☆ ★★★★★ ★★★★★
Wikipedia ★★★★☆ ★★★★☆ ★★★★★ ★★★★★
arXiv ★★★★☆ ★★★★★ ★★★★★ ★★★☆☆
PubMed ★★★★☆ ★★★★★ ★★★★★ ★★★★☆
Tavily ★★★★☆ ★★★★★ ★★★☆☆ ★★★★☆
Google (SerpAPI) ★★★★☆ ★★★★★ ★★☆☆☆ ★★★★★
Local Documents ★★★☆☆ ★★★★★ ★★★★★ ★★★★★

Rate Limiting and Reliability

LDR includes intelligent adaptive rate limiting that:

  • Learns optimal wait times for each engine
  • Automatically retries failed requests
  • Prevents your IP from being blocked
  • Maintains high reliability

Managing Rate Limits

Rate-limit statistics (per-engine wait times and success rates) are shown in the Rate Limiting section of the metrics dashboard (/metrics). The learned wait times can be tuned with the rate limiting profile setting (conservative, balanced, aggressive).

Search Strategies

LDR supports multiple search strategies that determine how queries are processed:

  • langgraph-agent: Agentic research that picks engines dynamically per query (default)
  • source-based: Single query, fast results
  • focused_iteration: Iterative refinement for accuracy

Best Practices

  1. For General Research: Use searxng with the default langgraph-agent strategy
  2. For Academic Research: Combine arxiv, pubmed, and semantic_scholar
  3. For Technical Questions: Use github with searxng
  4. For Maximum Privacy: Use searxng with local Ollama models
  5. For Best Quality: Use tavily or Google with focused_iteration strategy

Troubleshooting

SearXNG Not Working

  • Check the LDR logs for SearXNG engine disabled: instance URL … is a private / loopback / link-local address — if present, add the URL origin to LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST (or set LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true) in the server environment and restart (see SearXNG-Setup)
  • Verify container is running: docker ps | grep searxng
  • Check URL in settings: http://localhost:8080
  • Test directly: curl http://localhost:8080
  • Check the logs: docker logs searxng or view them in the LDR web UI

Rate Limit Errors

  • Wait a few minutes and retry
  • Use the langgraph-agent strategy, which can route around rate-limited engines
  • Consider adding premium engines for higher limits

Temporarily Disabled Engines

During LangGraph research, an engine that reports a failure is removed from subsequent model tool choices. The lead agent and its subagents share this state for the research run. Calls already proposed are checked again before engine creation, so a disabled engine is not repeatedly initialized or queried. Engines on cooldown are also excluded from discovery-time service probes. Subagent instructions refer to the current tool schema rather than a fixed list of engine names that could become unavailable mid-run. Searches are never serialized: parallel searches of one engine, including its first calls and retries after a cooldown, run concurrently, and the first failure disables the engine for all of them. Searches already in flight may finish.

When every search engine the agent (or a subagent) could use is cooling down, the next model call waits for the soonest cooldown to end, checking for cancellation every second. Tools come back as soon as their cooldown ends, at any step of the run, not only when a run or report section starts. The research log says how long the next wait is expected to take.

Cooldown waits in one run share a one-hour budget by default; parallel subagents waiting at the same time spend it once. Set langgraph_agent.cooldown_wait_budget_seconds to allow longer waits (or to zero to skip waiting). This budget covers only waiting for search engines; it does not limit time spent searching, running the LLM, or writing the report. Subagents have a separate two-hour wall-clock deadline by default, adjustable with langgraph_agent.subagent_timeout_seconds. Its effective value is at least the cooldown wait budget plus 30 minutes, so a subagent can complete a full cooldown wait and still have time to work. Cancelling the research stops the lead agent waiting for running subagents within a minute, not at their deadline, also when it waits for several subagent batches at once. Each of those subagents stops before its next model call, so one in the middle of a model call or a tool call (such as a slow search or page fetch) can take until that call returns. A subagent already past its deadline is no longer waited for and may keep running until it finishes, even after a cancel. For a local model that is silent for more than 30 minutes on one request, also raise llm.request_timeout; that setting is separate from research cooldown waits and subagent deadlines. Temporary failures and provider Retry-After values keep their cooldowns, even after repeated failures. When the next cooldown cannot finish within the remaining wait budget, the cooling engines are disabled for the rest of the run. A Retry-After longer than a day (or longer than the wait budget, if that is larger) is not treated as a cooldown at all: the engine is disabled for the rest of the run at once, with that reason.

Once that leaves no search engine, the rules below for engines disabled for the rest of the run apply.

If every search engine is disabled for the rest of the run but page fetching is enabled, the agent keeps only fetch_content: it can still read URLs from the query or from earlier results, but it cannot find new sources and may answer without searching (it is told to say when evidence is insufficient). If every research tool is disabled for the rest of the run, the model is not called without tools: the research (or the current detailed-report section) stops, and the answer is written from the sources gathered so far. A section stopped before gathering any sources is rendered as “Limited information was found for this subsection.”; the research log records the stop as an error line, while overall report progress continues. Availability also propagates through the optional full-content search wrapper.

The shared search boundary classifies primary request failures for web search providers, registered retrievers, and local document-store engines. Adapters propagate failures to this boundary instead of converting them into successful empty results. Initialization failures also retain their classification when the agent creates an engine, so an access refusal is not converted to a generic retry. Successful searches with no matches and optional enrichment failures retain their existing behavior. The web worker carries a failed primary initialization into the tracker, so that engine is excluded before the first model call as well as later calls.

The research log records the reason, disable time, and retry time. The “selecting next action” status lists the available tools and explains disabled engines separately. A search skipped because its engine is disabled or cooling down is recorded in search metrics with the status skipped_unavailable and the reason: the Metrics dashboard and the research details page show these skips per engine, separately from searches that were sent, so they do not change call counts, response times or success rates. Cooldowns expire automatically; the engine then becomes eligible for another search, and the log records that retry is available. The shared tracker reports each availability transition once across the lead agent and subagents. Repeated skipped calls log at debug level. A cooldown expiry permits a retry; it does not claim the provider has recovered.

  • Rate limits: use the provider's Retry-After value when reported, otherwise wait five minutes after the engine's retries are exhausted. A Retry-After longer than a day (or than the wait budget, if that is larger), here or for other temporary failures below, disables the engine for the rest of the run instead. Retries may happen in the HTTP client rather than the search pipeline: Semantic Scholar's session retries HTTP 429 itself, honoring Retry-After up to 30 seconds per retry (at most five retries), and when those retries run out the engine is disabled for five minutes once its fallback queries are also used up. Exhausted retries on HTTP 5xx count as an ordinary search failure. Any non-JSON Wikipedia response (normally its HTML 429 page) is treated as a rate limit, because its client library discards the HTTP status; a non-JSON server error therefore also waits five minutes.
  • Connection failures, timeouts, and other search failures: use the provider's Retry-After when reported, otherwise wait one minute.
  • A request URL refused by security validation: wait five minutes. The validator also refuses hosts that cannot be resolved, so a DNS outage is retried rather than treated as a configuration error.
  • HTTP 403: use the provider's Retry-After when reported, otherwise wait five minutes, because it is often a temporary bot or firewall block. A 403 whose reply is clearly about credentials (it asks for authentication or mentions an API key, token, subscription or billing) is treated as denied credentials below. So is any 403 from Semantic Scholar when an API key is configured: it rejects an invalid key with a plain {"message":"Forbidden"} reply. SerpAPI's account permission refusals are also run-long failures. Google PSE's structured quota-limit responses stay temporary and retain their retry delay; credential refusals remain disabled for the run.
  • Denied credentials/access (HTTP 401/402, or such a 403) or a private/loopback SearXNG instance URL without server approval, when the engine reports such a failure: disable for the rest of the research. Correct the configuration before starting another run. This does not change saved engine settings.
  • A successful search with no matches is still available. Usable partial results also keep an engine available, and so does a search whose results all arrived but were then dropped by the relevance filter (or failed a later filtering step): the provider answered.
  • Errors specific to one query do not disable the engine: a query the provider rejects as malformed (HTTP 400, 414 or 422) is reported back to the agent to rephrase, and a non-JSON Wayback Machine reply for one URL (such as its "blocked site" page) counts only against that URL. SearXNG listing unresponsive backends alongside no results counts as no matches, because it does not report which backends were queried.

Availability tracking lives in web_search_engines/engine_availability.py and has no LangGraph dependency. advanced_search_system/tools/search_availability.py adapts it to the agent. BaseSearchEngine._search_previews is the common boundary for provider retrieval: it calls the adapter's _get_previews, classifies exceptions, preserves retry metadata, and records a safe reason. Public search methods use this boundary before relevance filtering or optional content enrichment. Outcomes belong to each invocation, so concurrent calls and nested searches on one engine cannot overwrite one another's failures.

When implementing an adapter:

  • Return a list from _get_previews. Return [] only when a successful search has no matches. Propagate transport, provider, and malformed-response errors; do not catch them just to return [] or manually set availability state. Raise SearchQueryError for local query validation failures so other queries can still use the engine.
  • Preserve usable partial results if a later optional request fails. For fallback sequences, self._search_attempts() provides run(operation, ...) and finish(results): a successful empty attempt supersedes an earlier error, usable results survive a later error, and a final failure without results propagates to the common boundary.
  • Preserve the original HTTP/SDK exception when it carries status or retry timing. The common boundary reads responses and SDK metadata. If an SDK discards that information, translate its provider-specific refusal into SearchEngineError(SearchFailure(...)) before it is lost. Safe reasons must not contain response bodies, request URLs, or credentials. Structured HTTP statuses take priority over text-only SDK rate-limit messages.
  • Keep optional enrichment failures separate from primary search health. Policy denials, invalid settings, and research cancellation must propagate.

Constructor probes can still record an initial failure through _record_search_failure; it remains visible when the engine is handed to another worker. Primary adapters do not need this manual reporting hook. Agent callers use create_search_engine(..., raise_on_error=True) to preserve safe, structured initialization failures for the same tracker. Direct factory callers retain the default None result for failed construction.

No Results Found

  • Try different search engines
  • Broaden your query
  • Check internet connectivity
  • Verify API keys for premium engines

Advanced Configuration

Configuring Search Engines

You can enable/disable specific search engines and adjust their reliability parameters in the settings. This affects which engines the langgraph-agent strategy can choose from and how the system handles rate limiting.

Multi-Engine Research

The former auto and parallel meta engines (which fanned a query out over several engines) have been removed. To research across multiple engines, use the default langgraph-agent strategy: it calls any enabled engine as a tool, in parallel where useful, and picks per sub-question which engines to query.

Related Documentation