Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

18.7 KB
Raw Blame

💥 Breaking Changes

  • search.time_period default changed from "Past year" to "All time". Default searches are no longer silently restricted to the last year. Migration 0030 updates every stored "Past year" value to "All time" — including deliberately selected ones, since they are indistinguishable from the old default; if you want year-restricted results, re-select Past year in Settings after upgrading (choices of Past 24 hours / week / month are preserved). The global setting now reaches Tavily, serpapi, and Wikinews on every search path — get_search()-based flows (including news subscriptions) as well as direct create_search_engine() callers such as the default langgraph-agent strategy and the MCP server — and takes precedence over the per-engine search.engine.web.<engine>.default_params.time_period settings, which apply only when the global setting is absent (rare, programmatic snapshots). Brave is not covered by the global setting and remains governed by its own per-engine setting. Wikinews changes: its built-in default is now "All time" (previously an implicit one-year window), and unrecognized values apply no filter instead of silently falling back to one year — much older articles can appear unless a narrower period is explicitly selected. (#4978)

🔒 Security

  • Live WebSocket connections are now torn down whenever the session behind them ends — logging out, changing your password, a session expiring, or an idle connection being reclaimed — instead of only at handshake. A socket can no longer keep receiving your events (including settings_changed events that carry secret values) after its session is gone. Teardown is scoped per session, so logging out of one tab no longer disconnects your other tabs or devices, and every connect/subscribe/unsubscribe is re-checked against a still-valid session as defense in depth. In particular, a stale or captured cookie can no longer open a brand-new socket and join your event room after its own session was logged out or expired, even while another of your sessions (or an in-flight research run) keeps your database connection open. (#5535)

  • SSRF hardening for the JavaScript-rendering fetch path. The headless-browser downloader (Crawl4AI and plain Playwright) now validates or blocks every subrequest it makes — the initial navigation, each redirect hop, and every subresource — against the same SSRF rules as the static fetch path, so a public page that redirects to a cloud-metadata endpoint or a private/internal host is blocked before the browser connects. This also closes several sibling bypasses on the browser path: robots.txt is now fetched through the SSRF-guarded session instead of an unchecked client, Service Workers are blocked, and WebSocket connections are rejected. Cloud-metadata IPs stay blocked in all cases; the PRIVATE_ONLY egress scope still reaches local lab hosts. On both browser engines (Crawl4AI and plain Playwright), cookies set during a redirect chain are now re-scoped per hop from the browser cookie jar, so a cookie belonging to one domain is no longer forwarded to a different domain a later redirect lands on (matching normal browser behavior); same-domain cookie persistence is unchanged, and a cookie the server clears mid-chain (Max-Age=0 / past Expires) is not resurrected.

  • Confined the Sentence Transformers embedding-model setting to the app's models directory. Previously any authenticated user could set local_search_embedding_model to an arbitrary absolute server path, turning it into a filesystem existence/structure probe and an arbitrary local-model loader. A value is now treated as a local model only when it resolves under the models directory; anything else is handled as a HuggingFace model id, and a path that escapes the models directory is rejected.

  • Harden multi-tenant defense-in-depth: refuse symlinked/out-of-root deletes in the research library cascade helper, fail closed on empty usernames at per-user hash chokepoints, and cap per-user research concurrency against the server-wide limit -- including chat-initiated research starts (send_message and retry_attempt), not just the main research routes and queue processor. Note: the per-user cap corrects accounting against the global ceiling, not per-user fairness -- a single user can still legitimately occupy the whole global limit; anti-monopoly scheduling is a separate follow-up.

  • Harden user-editable public search-engine instance URLs against SSRF. A public engine URL (e.g. the SearXNG instance_url) that points at a private, loopback, or link-local address is now rejected at save time, and by default such an instance is no longer fetched at runtime. Whether the engine may reach a private instance is an operator decision only (the opt-in / env-locked cases below); it is deliberately never granted by the user-editable policy.egress_scope setting. Operators who self-host SearXNG on localhost/LAN can opt back in with LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true; Docker deployments that env-lock the URL are unaffected, and cloud-metadata endpoints remain blocked regardless.

    Upgrade note: the default http://localhost:8080 SearXNG instance is a loopback URL, so after this change it is refused unless you opt in. If your self-hosted SearXNG stops returning results, set LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true (or env-lock the URL). The engine now logs a warning naming that flag when it disables itself for this reason.

  • Hardened background log persistence so it can no longer reopen a user's encrypted database or reinstate their connected state after logout. Queued log entries no longer carry the user's password, the log-queue drain now writes only while the user's database is already open (post-logout backlog entries are dropped), and the drain thread clears any cached credential on every iteration. In-flight logging for active research is unaffected.

    Logout also no longer closes a user's database while their research is still running — closing it mid-run made the log-queue drain silently drop that job's logs. The connection now stays open for the rest of the run, plus up to one idle-connection sweep afterward (a few minutes), which is also when is_user_connected stops reporting the account as reachable. This is safe only together with the server-side session-id revocation on the request/socket auth path (#5532); this PR must ship with, or after, that change, not before it.

    Changing your password no longer pauses other users' in-progress research. The database re-encryption is now gated per-user rather than process-wide, so only your own new research is briefly held back while the re-encryption runs; everyone else's active research keeps progressing. The queue processor additionally re-checks your session immediately before reopening your encrypted database, so a logout that lands in that instant can no longer bring the decrypted database back or resume a queued research.

  • Hardened outbound HTTP fetches (RAG source and web-scraping requests via safe_get/safe_post/SafeSession) so the connection targets the exact IP address the SSRF guard validated, instead of letting the HTTP client re-resolve the hostname independently at connect time. The validated address is now pinned for the connection and re-checked at every redirect hop, while the original hostname is preserved for TLS SNI and certificate verification.

  • Hardened the outbound notification (Apprise) path against send-time DNS rebinding. Notification service URLs were validated once at configuration time, but Apprise re-resolves the hostname (and follows redirects) when it actually sends, so a rebinding host could serve a public address to the validator and a private/cloud-metadata address to the sender. The delivery path now runs synchronously in-thread with HTTP redirect-following disabled (closing redirect-to-internal and redirect-to-exfil outright, even against a user-supplied ?redirect=yes) and, for the duration of each send, pins every raw-webhook host to the address just validated and refuses any unpinned lookup that resolves to a private/loopback/link-local/cloud-metadata IP — blocking it at the socket layer before a connection is made. Send-time DNS resolution is bounded by a timeout (a slow/hostile resolver cannot hang the sending thread), and the send fails closed if the pinning shim is not the active resolver or if delivery would fan out to worker threads the thread-local guards do not cover. Per-scheme policy matches the validator (http/https block private unless opted in; plugin/self-hosted schemes allow LAN targets but always block cloud-metadata), the original hostname is kept for TLS SNI and certificate verification, and the pin/block are thread-local so concurrent requests are unaffected. The LDR_NOTIFICATIONS_ALLOW_OUTBOUND gate remains as defense in depth.

    A follow-up review round closed further gaps: AWS's native IPv6 instance-metadata endpoint (fd00:ec2::254, a ULA that the IPv4 metadata list and the NAT64 embedded-IPv4 check did not cover) is now always blocked, including under the private-IP opt-in; notification service URLs whose authority contains more than one @ are rejected before the host is extracted (the validator/pin parser and Apprise's own parser pick different hosts for such authorities); a confirmed send-time SSRF block is now recognised and fails fast in a single attempt instead of being retried; and the bounded send-time DNS lookup runs on a daemon thread so a hung hostile resolver can never block graceful process shutdown.

    A further round closed a residual and tightened defense in depth: the notification plugin/raw-webhook partition now blocks the entire link-local range (IPv4 169.254.0.0/16, IPv6 fe80::/10) even under the private-IP opt-in — cloud-provider metadata is served across link-local beyond the always-blocked literals (e.g. Scaleway's 169.254.42.42), while self-hosted RFC1918 / loopback / non-link-local ULA notifiers keep working; and notification URLs with an empty // authority but an in-path host (e.g. json:///169.254.169.254/path, where the validator's parser sees no host but Apprise dials the path segment) are rejected before host extraction.

  • Logging out now revokes the session cookie immediately. The auth path (HTTP and WebSocket) validates the cookie's server-side session id, so a session that has been logged out can no longer be reused — even after the same user logs back in. Idle/expired sessions are now also rejected (sliding-expiry).

  • Programmatically registered search retrievers and custom LLMs are now scoped to the user who registered them. Built-in providers and engines remain shared for everyone, but one user's registration can no longer shadow another user's provider/engine resolution or appear in another user's engine list.

    The run's user is now threaded through every LLM-resolution seam that had been relying on the settings snapshot alone. When a user's primary search source is their own private (local-only) retriever, the local-only inference policy is now enforced for the model/provider override path of a research run, for domain classification, and for the news scheduler's post-run headline/topic generation — closing paths where those calls could otherwise fall back to a cloud LLM for a private-corpus run.

    As a safety net, when ADAPTIVE egress cannot classify a run's primary engine in any visible namespace it still resolves to the permissive (cloud-capable) scope, but now emits a policy-audit warning naming the primary and reporting whether a username was threaded — so a future missed username thread surfaces in the logs instead of failing open silently.

    Note for programmatic/SDK callers: registering a None retriever or LLM (via register / register_multiple) now raises ValueError instead of being silently stored, and register_multiple is atomic — a batch containing any None value registers none of its entries.

  • Unencrypted filesystem storage for library-downloaded PDFs is now an environment-only operator gate (LDR_RESEARCH_LIBRARY_ALLOW_FILESYSTEM_PDF_STORAGE) and disabled by default. Previously any user could select it and have fetched third-party PDFs written as plaintext to a shared library directory. When the gate is off, the option is withheld from the settings UI and any stored or environment value of filesystem is coerced to the encrypted database mode before a PDF is written; a migration rewrites existing filesystem selections to database. Previously-written plaintext files remain readable. The default (database, encrypted) and none (text-only) modes are unchanged.

    Library-downloaded PDFs are now stored under a per-user subdirectory (<library>/<username>/) instead of a single shared directory (#5521). Previously two users' documents could collide on the same on-disk filename (per-user autoincrement resource ids) in the shared directory, letting one user's PDF overwrite or expose another's. Reads can fall back to the legacy shared location so PDFs downloaded before this change still load — no files are moved — but that fallback is now operator-gated and off by default (see below). Shared-library mode (research_library.shared_library) now requires the operator opt-in LDR_RESEARCH_LIBRARY_ALLOW_SHARED_LIBRARY: because both that flag and storage_path are user-editable, honoring a user's shared_library=true would let a tenant drop the per-user boundary and point storage at another user's directory, so the setting is ignored unless the operator enables the env gate. Upgrade note: existing shared-library deployments must set LDR_RESEARCH_LIBRARY_ALLOW_SHARED_LIBRARY=true to keep sharing a single directory; otherwise new downloads go to per-user subdirectories and previously-downloaded shared PDFs stop resolving until the operator also enables the legacy read fallback (LDR_RESEARCH_LIBRARY_ALLOW_LEGACY_READ_FALLBACK=true).

    The legacy shared-root fallback is now read-only: document deletion resolves only within the deleting user's own per-user directory and never follows the fallback into the shared root. Because pre-isolation files were never moved into per-user directories, an ordinary self-service delete of a legacy document could otherwise resolve a colliding relative path (e.g. pdfs/5.pdf) to another tenant's file and unlink it. Deletes now leave shared-root files untouched.

    The legacy shared-root read fallback is now operator-gated (LDR_RESEARCH_LIBRARY_ALLOW_LEGACY_READ_FALLBACK) and disabled by default, closing a cross-tenant read: because the fallback base is derived from the user-editable research_library.storage_path, a user could point their storage_path at another user's directory and — since per-user resource ids collide by construction — have a read of their own document id resolve to the other user's PDF. With the gate off (default), library PDF reads resolve strictly within the requesting user's own per-user root; operators who need pre-isolation files to keep loading can opt in on single-user or trusted deployments.

🐛 Bug Fixes

  • Decode OpenAlex snapshot partitions record by record so the journal-quality download no longer holds a whole partition in memory, which could exhaust RAM on smaller hosts and leave the container thrashing in swap. (#4383)
  • The Tavily search engine now honors the configured time period. When a specific period (Past 24 hours / week / month / year) is selected under Settings → Search, it is forwarded to the Tavily API as time_range, so that selection actually narrows results instead of being silently ignored. The default is "All time" (no filter) — select a narrower period if you want time-restricted results. The setting now reaches engines on every search path, including the default langgraph-agent strategy and the MCP server, which previously bypassed it. (#4936)
  • SearXNG search engine rate limiting (delay_between_requests) is now shared process-wide per instance URL, ensuring request throttling applies across per-call engine instances constructed during research agent tool calls. (#5351)
  • Detect leaked active-research state after connected tests. (#5466)
  • Lock the provider-specific validation boundary for explicit OpenAI endpoint URLs. (#5488)
  • Handle list-form LLM content blocks in news analysis, GitHub query optimisation and topic organisation. Topic text synthesis raised a TypeError, the GitHub search query became the block list's repr, and the news summaries came back empty. These call sites now also strip <think> reasoning tags from model output, matching the sibling call already in each file.
  • Handle list-form LLM content blocks in the question-generation family and the evidence evaluator. With a provider that returns .content as a list of blocks (Anthropic extended thinking or tool use), follow-up question generation, sub-question decomposition, atomic-fact decomposition, gap-filling questions, BrowseComp entity extraction and progressive searches, and evidence extraction all raised AttributeError. Since StandardQuestionGenerator is the default generator, this broke the main research loop on every iteration. These call sites now also strip <think> reasoning tags from model output, which is a visible change for an LLM that was not obtained through get_llm().
  • Handle list-form LLM content when parsing advanced-search constraints.
  • Handle list-form LLM content when parsing fallback journal reputation scores.
  • Invalidate stale wiki-link autocomplete requests and results immediately when the query changes.
  • Keep aggregate RAG chunk counts accurate when reindexing a document with a different number of chunks.
  • Prevent an older Unified Search response from rendering stale clickable results after the query changes.
  • Prevent stale document and research note-panel requests from overwriting newer refresh results.
  • Queue bounded LangGraph subtopic overflow instead of dropping it, and reject oversized batches before starting partial work. Progress metadata reports queued overflow through overflow_strategy="queued" and overflow_queued_count; legacy truncated_from is omitted because no topics are truncated.
  • Reject non-string collection names and descriptions with a 400 response instead of returning an internal server error.
  • Sort Unified Search semantic results using full similarity precision before rounding response values.