Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

16.4 KB
Raw Blame

SearXNG Integration for Local Deep Research

This document explains how to configure and use the SearXNG integration with Local Deep Research.

⚠️ Self-hosting on localhost or your LAN? Read this first.

Since v1.10.3, a SearXNG instance URL that points at a private, loopback, or link-local address (including the default http://localhost:8080) is blocked by default: it is rejected when saved in the web UI, and the engine is skipped for the rest of the research run with a warning in the logs. This is SSRF protection — the URL is editable by any authenticated user, and without the block it could be pointed at internal services on your network.

Self-hosted local instances remain fully supported; the server operator just has to approve them once, in one of three ways (then restart LDR):

# Option A (recommended, v1.10.5+; on older versions use Option B or C):
# allow ONLY specific URL origins
LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST=http://localhost:8080

# Option B: pin the URL itself via the environment. An env-locked URL is
# treated as operator-provisioned and trusted; it becomes read-only in the
# web UI. The bundled docker-compose.yml does this.
LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL=http://localhost:8080

# Option C (broadest): allow ALL private/localhost/LAN engine URLs
LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true

Cloud-metadata addresses (e.g. 169.254.169.254) stay blocked under all three options. Details in Network Security below.

Configuring SearXNG Access

The SearXNG search engine is disabled by default until you provide a reachable instance URL. This ensures the system doesn't attempt to use public instances without explicit configuration.

Setting Up Access

(The three options in the callout above answer a different question — how a private URL gets approved. The two ways below are simply how the URL value is set; note that option 1 doubles as approval, while a private URL set via option 2 still needs one of the approvals above.)

You have two ways to configure the SearXNG instance URL:

  1. Environment Variable (Recommended for self-hosted instances):

    # Set in the server process's environment (export / docker -e / compose
    # environment: / systemd EnvironmentFile). LDR's own code does not load a
    # .env file for its LDR_* settings; hand such a file to whatever starts LDR
    # instead.
    LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL=http://localhost:8080
    
    # Optional: Set custom delay between requests (in seconds)
    LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_DELAY_BETWEEN_REQUESTS=2.0
    

    Environment variables override and lock the corresponding web UI setting, and an env-locked instance URL counts as operator-provisioned — so this single line also satisfies the private-URL protection described above.

  2. Web UI: Settings → Search → SearXNG → Endpoint URL. Note that a private/localhost URL entered here additionally requires operator approval in the server environment — either its origin listed in LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST or the blanket LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true (see the callout at the top of this page).

Restricting to Specific SearXNG Engines

By default SearXNG queries all engines enabled on your instance. If you want LDR to only use a subset (for example, only kagi, brave, wikipedia, and arxiv), set the engines parameter via the web UI Settings → SearXNG → Engines or via environment variable:

# Comma-separated engine names matching your SearXNG instance's /engines list
LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_ENGINES='["kagi", "brave", "wikipedia", "arxiv"]'

LDR forwards this list as SearXNG's engines query parameter for every search. Leave it empty (the default) to use all engines configured on the instance. Engine names must match those listed at <your-searxng-instance>/engines; unknown names are ignored by SearXNG and yield no results from that engine.

This is useful for splitting research across engine profiles (e.g. a free-only profile for routine queries and a premium-engines profile for important research) — change the setting before starting a research session to switch profiles.

Self-Hosting SearXNG (Recommended)

For the most ethical usage, we strongly recommend self-hosting your own SearXNG instance:

Using Docker (easiest method)

# Pull the SearXNG Docker image
docker pull searxng/searxng

# Run SearXNG (will be available at http://localhost:8080)
docker run -d -p 8080:8080 --name searxng searxng/searxng

Using Docker Compose (recommended for production)

  1. Create a file named docker-compose.yml with the following content:
version: '3'
services:
  searxng:
    container_name: searxng
    image: searxng/searxng
    ports:
      - "8080:8080"
    volumes:
      - ./searxng:/etc/searxng
    environment:
      - SEARXNG_BASE_URL=http://localhost:8080/
    restart: unless-stopped
  1. Run with Docker Compose:
docker-compose up -d

Using Public Instances

If you must use a public instance:

  1. Get Permission: Always contact the administrator of any public instance
  2. Respect Resources: Use a longer delay (4-5 seconds minimum) between requests
  3. Limited Usage: Keep your research volume reasonable

Example configuration for a public instance:

LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL=https://instance.example.com
LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_DELAY_BETWEEN_REQUESTS=5.0

Note that most public instances do not enable the JSON API (format=json) and aggressively rate-limit automated clients, so a self-hosted instance is both the more reliable and the more ethical choice.

Checking Configuration

To verify if SearXNG is properly configured:

from web_search_engines.search_engine_factory import create_search_engine

# Create the engine
engine = create_search_engine("searxng")

# Check if available
if engine and getattr(engine, "_is_available", False):
    print(f"SearXNG configured with instance: {engine.instance_url}")
    print(f"Delay between requests: {engine.delay_between_requests} seconds")
else:
    print("SearXNG is not properly configured or is disabled")

Network Security

SearXNG is designed for self-hosting, and running it on a private address is the normal setup:

  • Localhost: http://127.0.0.1:8080 or http://localhost:8080
  • LAN IPs: http://192.168.1.100:8080, http://10.0.0.5:8080, http://172.16.0.2:8080
  • Docker networks: http://172.17.0.2:8080 or http://searxng:8080
  • Local hostnames: http://searxng.local:8080 (if configured in DNS/hosts)

However, since v1.10.3 such private / loopback / link-local URLs are not fetched by default and are rejected when saved through the web UI. The instance URL is an editable setting, so without this protection any authenticated user could point a "public web search" engine at internal services and use research runs to probe your private network (SSRF). Peer projects in this space have accumulated a series of SSRF CVEs through exactly this kind of URL handling, so LDR ships strict-by-default.

A private URL becomes allowed when the server operator (someone with access to the server environment, not just the web UI) approves it in one of three ways:

  1. Origin allowlist (recommended) — set LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST to a comma-separated list of exact URL origins, e.g. http://localhost:8080,http://192.168.1.5:8888. Private fetching is enabled for the engine only when its configured URL matches a listed origin; a non-listed private URL stays blocked. Matching is exact scheme://host:port (default ports 80/443 implied, host case-insensitive, path/query ignored) — no wildcards or CIDR ranges, and hostnames are matched as text, so localhost and 127.0.0.1 are distinct entries (list both if unsure). Prefer IP-literal entries over hostnames where practical: a listed hostname grants whatever private address it resolves to at fetch time, while an IP entry pins the destination. List only origins you control — the grant also covers redirects served by the listed origin.
  2. Env-locked URL — set LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL to the exact URL. The setting becomes read-only in the UI and is trusted as operator-provisioned; this is what the bundled docker-compose.yml does (http://searxng:8080).
  3. Blanket opt-in — LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true. Allows all private/localhost/LAN engine URLs, including ones entered in the web UI. Broadest grant — prefer the allowlist when you can.

Regardless of which option is used:

  • Cloud metadata endpoints (AWS IMDS / ECS / EKS Pod Identity, Azure IMDS and WireServer, GCP, OCI, Scaleway, Akamai / Linode, OpenStack / Hetzner, DigitalOcean, AlibabaCloud, Tencent Cloud, Oracle Compute Classic — see ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS — plus the GCP and IBM Cloud classic metadata hostnames in ssrf_validator.ALWAYS_BLOCKED_METADATA_HOSTNAMES) are always blocked to prevent credential theft in cloud environments. The IBM entry is a hostname block only: another name for the same private address is not covered
  • Only http/https URLs are accepted — other schemes are always refused
  • The approval covers the instance URL only. It does not let full-content fetching follow the result URLs SearXNG returns into your private network; that needs the separate opt-in below.

Full-content fetching of private result URLs

With full content enabled (search.snippets_only off), LDR fetches the page behind each result through the full-search wrapper the search-engine factory builds around the engine. Private / loopback result URLs (say an internal wiki at http://192.168.1.50/wiki that your instance indexes) are blocked by default even on an approved instance: SearXNG proxies the public web, so a poisoned or malicious index could otherwise turn result fetching into a scan of the LDR host's LAN, loopback included. To fetch their full content, the server operator must additionally set:

LDR_SEARCH_ALLOW_PRIVATE_RESULT_FETCH=true

This is required in addition to one of the three instance-URL options above (the opt-in alone grants nothing), is environment-only (not settable from the web UI), defaults to off, and is not set by the bundled docker-compose.yml. Even with it enabled, link-local addresses (169.254.0.0/16, fe80::/10) and the cloud-metadata endpoints stay blocked.

The flag takes effect only under the unprotected egress scope. The search-engine factory hands the full-search wrapper the run's egress context, and the wrapper applies the flag only when that context's scope places no address-class restriction on hosts. Under every other scope the flag is ignored and private result URLs are refused exactly as if it were unset, by the SSRF check or, for private addresses the SSRF check lets through, by the wrapper itself (see below):

  • the default adaptive scope never resolves to unprotected: with SearXNG (or another public engine) as the primary it resolves to public_only, which does not permit private hosts, and on a classification fallback to both, covered next;
  • private_only, strict and both retain the policy that full-content result URLs must classify as public. The private-result grant does not override that rule. Static downloads also enforce the run scope on every redirect and pinned connect address.

Under every scope except unprotected the wrapper refuses every result URL whose address the egress policy cannot classify as public. This includes the ambiguous 198.18.0.0/15 range handed out by fake-IP DNS resolvers, and private ranges the SSRF check itself lets through (the documentation ranges, 240.0.0.0/4 and 2001:db8::/32). It asks the policy's own classifier before the counted scope check, so this also holds under strict, private_only and both. Refused non-public result URLs do not count toward the run's denied-fetch quota, so they cannot use it up and block later public fetches. The snippets of those results are still returned, only their full content is missing. These checks apply to the result URL itself. Static redirects and pinned connect addresses also receive the run-scope check. Browser fallback is disabled under public_only, private_only and strict; when available under both or unprotected, its requests retain SSRF checks. Under unprotected without the flag only the SSRF check applies, so the ranges it lets through are fetched there.

Known limitation: fake-IP DNS. Proxy tools in fake-IP mode (Clash, sing-box, Surge in TUN mode) answer public names with 198.18.0.0/15 addresses, which the egress policy cannot classify as public or private. Full-content fetching is then unavailable under the default adaptive scope (which resolves to public_only with SearXNG) and every other scope except unprotected. Other policy-checked fetch paths and the strict / private_only socket guard refuse these addresses too. See Known limitations in the egress modes guide.

Upgrade note. Released v1.x versions have no such flag and always block private result URLs, so upgrading from them changes nothing: the flag is new in v2.0.0 and works as described above. Development builds from #6196 up to #6535 gave the wrapper no run context and applied the flag under any scope that let SearXNG run, the default adaptive scope included, which bypassed the run's egress scope. If you ran such a build and relied on the flag for intranet full content, the research must now use the Unprotected scope (which needs the operator gate described under Unprotected in docs/egress-modes.md); under any other scope those results arrive without full content. When the flag is set but the scope keeps it from applying, LDR logs an INFO line naming the scope. See security/egress/README.md.

IPv6-only deployments (NAT64)

The "private IPs allowed" exception above does not cover IPv6 transition prefixes. On IPv6-only Kubernetes / cloud deployments (AWS / GKE / Azure IPv6-only nodes) where outbound IPv4 traffic is synthesized through NAT64 (64:ff9b::/96 RFC 6052 well-known or 64:ff9b:1::/48 RFC 8215 local-use), reaching a SearXNG instance through these prefixes is blocked by default. To opt in, set:

LDR_SECURITY_ALLOW_NAT64=true

The opt-in is scoped strictly to the two NAT64 prefixes — 6to4 (2002::/16), Teredo (2001::/32), the discard prefix (100::/64), the deprecated IPv4-Compatible IPv6 form (::/96, except ::1 loopback which follows the loopback/private-IPs flags), and the IPv4-Translated SIIT form (::ffff:0:0:0/96, RFC 2765) remain blocked, and cloud-metadata IPs stay unreachable through any NAT64 wrap. See SECURITY.md for the full rationale.

Troubleshooting

SearXNG suddenly returns no results after upgrading to v1.10.3+

This is almost always the private-URL protection described above: a localhost/LAN instance URL is no longer fetched by default. When SearXNG is your selected search engine, the research form shows a dismissible warning banner ("SearXNG is disabled: private URL not approved") naming the exact remedies; the logs carry the same information as a warning saying SearXNG engine disabled: instance URL … is a private / loopback / link-local address. SearXNG is skipped for the rest of that research run; other available engines can still be used. Fix it by adding the URL origin to LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST (or env-locking the URL, or setting LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true) in the server environment, then restarting. Note the allowlist variable requires v1.10.5 or newer — on v1.10.3/v1.10.4 use one of the other two options.

Other errors

  1. Check that your instance is running
  2. Verify the URL is correct in your environment variables
  3. Ensure you can access the instance in your browser
  4. Check firewall settings and network connectivity

Resources