~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
docs(adr): research runs keep their daemon thread; async is I/O fan-out on one shared loop (#6450)
LearningCircuit1ce4ed6ca38e
diff --git a/docs/decisions/0013-research-runs-keep-daemon-thread.md b/docs/decisions/0013-research-runs-keep-daemon-thread.mdnew file mode 100644index 0000000000..dbe7e2fa68--- /dev/null+++ b/docs/decisions/0013-research-runs-keep-daemon-thread.md@@ -0,0 +1,354 @@+# ADR-0013: Research runs keep their dedicated daemon thread; async is for I/O fan-out only++**Date:** 2026-09-13+**Status:** Accepted++## Context++### How a research run executes today++`web/services/research_service.py::start_research_process` acquires the+server-wide `_global_research_semaphore` (a `threading.Semaphore` sized by+`server.max_concurrent_research`, default 10) in the caller's own thread+with a non-blocking `acquire(blocking=False)`. If no slot is free it raises+`SystemAtCapacityError`, which the web routes turn into HTTP 429 (the queue+processor re-queues the run instead), so capacity is enforced before any+thread is spawned. It then+copies the calling request's contextvars with `contextvars.copy_context()`+(on default GIL builds a bare `threading.Thread` starts with an empty+context, which would break `get_current_username()` and metrics attribution+inside the worker; free-threaded builds enable+`sys.flags.thread_inherit_context` by default, and the explicit copy keeps+the behaviour the same on both) and+starts one dedicated `threading.Thread` with `daemon = True` per run.+`check_and_start_research` performs an atomic check-and-start keyed by+`research_id` so a retry cannot double-spawn the same run.++Inside that thread, `run_research_process` calls `set_search_context(...)`+first, before any other work, establishing the research id, username and+password so the run's early log lines can be attributed. The settings+snapshot in that first call may still be empty+(`kwargs.get("settings_snapshot") or {}`); a second+`set_search_context(shared_research_context)` later in the run installs the+populated snapshot. Settings reads inside the run go through+`config/thread_settings.py`, which keeps a+`_thread_local = threading.local()` settings context+(`set_settings_context` / `clear_settings_context` / `get_settings_context`).+`get_setting_from_snapshot` and `get_bool_setting_from_snapshot` resolve+against that thread-local context whenever the key is not found in the+`settings_snapshot` they are handed, whether because no snapshot was+passed or because the snapshot passed lacks that key. (`get_typed_setting_value` in `settings/manager.py` is+only the pure type-coercion helper those getters call; it reads no context.)+Database access goes+through `database/thread_local_session.py`'s `ThreadLocalSessionManager`,+which hands out one session per thread from `get_session(username,+password)`, backed by its own `threading.local()`, tracked per thread id in+`_thread_credentials` for logout/idle cleanup, and scoped by+`enter_scope` / `exit_scope` around each `get_user_db_session` block.+Progress reporting and cancellation live in the same thread: a+`progress_callback(message, progress_percent, metadata)` closure checks+`is_termination_requested(research_id)` on every call and, whenever the step+reports a progress value, calls `update_progress_and_check_active(...)`+(otherwise `is_research_active(research_id)`). It is wired into the+strategy with `system.set_progress_callback(progress_callback)`.++In short, the run's state is bound to its dedicated thread in two different+ways. The settings context, the DB session and the egress audit context+(`security/egress/audit_hook.py`, `_thread_local = threading.local()`, set+by `set_active_context`) are `threading.local` state. The search context+(research id, username, password, settings snapshot;+`utilities/thread_context.py`'s `_search_context_var`) and the request+username (`utilities/request_context.py`'s `_username_var`) are+`ContextVar`s, carried into the thread by the copied context. Progress and+cancellation run on that same thread. All of it lives and dies with one+research run's dedicated thread. The settings snapshot that+`run_research_process` builds once at run start and installs with+`set_settings_context` and `set_search_context` is what guarantees a+research run keeps exactly the settings it started with, no matter what+settings edits are made while the run is in progress; that guarantee is a+design requirement this decision codifies, not an accident of the current+implementation.++### What #5854 is actually about, and the misreading it invites++#5854 measured 89 call sites doing synchronous `.invoke()` / `.stream()`+against an AnyIO worker pool of 40 threads, process-wide. A `sync def`+FastAPI route handler occupies one of those 40 pool threads for the whole+duration of the call underneath it; an LLM call held one for seconds to+minutes. The fix #5854 asks for is converting those calls to+`ainvoke` / `astream` so the pool thread is free while the model is called.++Research runs started from the web UI or the queue never went through that+pool. They run on their own dedicated `threading.Thread`, started directly+by `start_research_process`, not by FastAPI's `run_in_threadpool`. Reading+#5854 as "move research execution onto the event loop so it stops holding+threads" is a misreading: those runs were never taking a thread from the+pool #5854 is about, so converting research itself to loop tasks would not+free anything #5854 cares about. It would only give up the per-run thread+isolation described above, for no matching benefit.++Not every research run takes that path, though. REST v1 `quick_summary` /+`generate_report` (`web/routers/api_v1.py`) call the library functions+through `run_db_sync`, which wraps `asyncio.to_thread`, so the whole run+executes inline on a pooled default-executor thread. Scheduled subscription+runs (`scheduler/background.py::_trigger_subscription_research_sync`) call+`quick_summary` directly on an APScheduler worker thread. Neither path+acquires `_global_research_semaphore`. The web benchmark runner+(`benchmarks/web_api/benchmark_service.py`) is a third shape: it starts its+own daemon thread (`_run_benchmark_thread`), not through+`start_research_process`, and calls `quick_summary` per task+(`_process_benchmark_task`); it does count against the global semaphore,+but with a blocking `_global_research_semaphore.acquire()` per task rather+than the non-blocking acquire and 429 of the web routes. Library callers of+`api/research_functions.py` run in the caller's own thread and have no+server event loop at all.++#6232 is the case #5854 is actually aimed at: it converted the notes AI+routes to `async def` and awaits `llm.ainvoke` directly, because those+routes genuinely ran as `sync def` handlers on the AnyIO pool. That is a+different execution model from a research run, which is why the same fix+does not apply to research.++### The #6293 constraint on where async I/O can live++#6293 found that `langchain_openai` and `langchain_anthropic` each cache+`httpx.AsyncClient` instances per process with `functools.lru_cache`, keyed+with no event-loop or thread component. In `langchain_openai` the cache is+`_cached_async_httpx_client`, keyed by `(base_url, timeout,+socket_options)`. In `langchain_anthropic` the cached function is+`_get_default_async_httpx_client`, keyed by `(base_url, timeout,+anthropic_proxy)`. The cached client's keep-alive connections are bound to+whichever event loop first uses them. Any code that awaits from a different+loop than the one that made the previous call reuses a connection from the+wrong loop and fails with `RuntimeError: Event loop is closed`, with the+provider SDK's retry logic sometimes turning that into a silently duplicated+(and double-billed) request. This happens whether the second loop is created+by a fresh `asyncio.run(...)` per call or by a second long-lived loop+running on a different thread; a per-research-thread event loop reproduces+the same failure, only less often. #6293 names the safe shape: exactly one+long-lived event loop in the process ever touches these clients (the+uvicorn request loop, or one dedicated loop that other code reaches via+`asyncio.run_coroutine_threadsafe`), unless LDR takes over supplying and+closing its own per-loop `http_async_client` instances, which it does not+do today.++### #5857 was a related but separate problem++#5857 reported that `DatabaseMiddleware`'s cleanup was `async def` and ran+on the event-loop thread, while the session it was cleaning up lived in+`threading.local()` on the AnyIO worker thread that served the request, so+a sync route's DB session was never returned to the pool. #6095 and its+follow-up #6207 addressed that mechanism: `web/fastapi_app.py` installs+`app.router.route_class = WorkerCleanupAPIRoute`+(`web/dependencies/threadpool.py`), which returns each sync route's+connections on the worker thread when that request finishes, and+`run_db_sync` does the same for explicit `asyncio.to_thread` work. That+failure mode required a database session to cross the worker-thread /+event-loop boundary. Research runs open and close their DB session on the+same daemon thread for the run's lifetime and never hand it to loop-side+code, so they were never subject to it.++### The async twins already added under #5854++Several tranches have already added an async counterpart next to an+existing synchronous call, without switching the production caller:++- #6249 (merged): `citation_handlers/base_citation_handler.py` gained+ `_invoke_text_async` / `_invoke_with_streaming_async` beside the+ synchronous `_invoke_text`; `web_search_engines/engines/full_search.py`+ gained `_check_urls_async` beside `check_urls`.+- #6268 (merged): `advanced_search_system/strategies/topic_organization_strategy.py`+ consolidated its model calls onto `_invoke_model` (sync) and added+ `_ainvoke_model` (async); production strategy calls stayed synchronous.+- #6246 (merged): news generators (`generate_headline_async`,+ `generate_topics_async`) and benchmark graders (`grade_single_result_async`,+ `grade_results_async`) gained async entry points beside their synchronous+ originals; no production caller has been converted yet.+- #6277 (merged): `utilities/llm_utils.py` gained `invoke_llm_sync`, the one+ call site the research pipeline's question generators, filters,+ explorers, context managers and evaluators go through, and its async core+ `ainvoke_llm`, which nothing in production awaits yet.++`RateLimitedLLMWrapper.ainvoke` (`web_search_engines/rate_limiting/llm/wrapper.py`,+hardened in #6347) and `ProcessingLLMWrapper.ainvoke`+(`config/llm_config.py`) both already `await` the wrapped provider's own+async client rather than bridging through a freshly created loop. That is+the correct shape for a fan-out interface and is the pattern this decision+generalizes.++## Decision++1. Each research run started through `start_research_process` (the web UI+ and the queue) keeps its dedicated daemon thread. That thread owns the+ run's settings context, database session, egress/search context,+ progress reporting and cancellation checks for the run's entire+ lifetime, exactly as today. The REST v1 and scheduler paths, which run+ research inline on a pooled or scheduler thread without the global+ semaphore, are noted exceptions to this rule. This decision does not+ change them, and whether to bring them under it is left to a follow-up+ (#6862). The web benchmark runner, which runs `quick_summary` on its own+ daemon thread and blocks on the global semaphore per task, is likewise+ outside `start_research_process` and unchanged by this decision.+ Library callers have no server loop; for them rules 2 and 3 do not+ apply and the synchronous twins remain the only entry point.+2. The process holds exactly one long-lived event loop for LLM and HTTP+ async I/O. Today that is the FastAPI/uvicorn loop. A dedicated I/O+ thread with its own loop is an acceptable alternative, but only ever+ one such loop, never one per research thread. Which of the two hosts+ the loop is not fixed by this decision and is left open until the code+ commits to one.+3. Daemon code that needs an async result submits the coroutine to that+ shared loop with `asyncio.run_coroutine_threadsafe` and waits on the+ returned future, with a timeout and a cancel path tied to+ `is_termination_requested`. Daemon code does not open its own event+ loop and does not call `asyncio.run` from inside the daemon thread.+4. Everything a submitted coroutine needs is passed in explicitly as+ arguments. The settings snapshot is taken once on the daemon thread at+ run start and is the only settings source for that run: loop-side code+ receives it as a read-only argument, never calls the thread-local+ settings getters (`get_settings_context`, or `get_setting_from_snapshot`+ / `get_bool_setting_from_snapshot` for a key the passed+ `settings_snapshot` does not contain, including when none is passed,+ since both then fall back to the thread-local context), never calls+ `SettingsManager.get_setting`, and+ never re-reads settings from the database, so a settings edit made+ while a run is in progress cannot change what that run does.+ Fan-out passes the same frozen snapshot to every coroutine it submits,+ together with the username, the scoped credential the call needs, and+ any database rows read ahead of time. A coroutine never reaches back+ into the daemon thread's state to fetch something it was not handed.+ This includes ambient contextvars: `asyncio.run_coroutine_threadsafe`+ schedules the task with a copy of the submitting thread's contextvars,+ so a coroutine submitted from the daemon inherits the daemon's search+ context, including the user's password, unless the submit helper runs+ it in an explicit minimal context. The helper must do so. The egress+ audit context is `threading.local` and does not follow the coroutine+ at all: it must be passed in and re-armed explicitly on the loop side.+5. Loop-side code, meaning anything running as a task on the shared loop,+ must never touch `threading.local` settings or session state, must not+ read the search context through `get_search_context()`, and must never+ open a database session. If a coroutine needs data from the database,+ the daemon reads it first and passes it in per rule 4.+6. The synchronous twin of each of these calls remains the canonical+ production entry point running on the daemon thread. The async twin+ exists to be submitted to the shared loop for fan-out, meaning several+ concurrent calls within one run, not to replace the sync path as the+ default caller.+7. A fresh `asyncio.run(...)` per call and a persistent event loop per+ thread are both forbidden, per #6293.++## Consequences++- #5854's measure of done changes. It is no longer "fewer research+ threads", because research threads were never the AnyIO pool threads+ #5854 measured. It becomes: no AnyIO worker-pool thread blocks on model+ I/O, and fan-out inside a single research run (parallel citation, search+ or topic calls) costs no extra OS thread because it is submitted to the+ one shared loop instead of spawning a thread or a loop of its own. The+ wiring is tracked in #6467.+- #5857 was never on the critical path for research concurrency, since a+ research run's DB session never leaves its daemon thread. Its mechanism+ for sync routes was addressed by #6095 and #6207.+- LLM rate limiting (`web_search_engines/rate_limiting/llm/wrapper.py`,+ currently disabled by `_should_rate_limit` returning `False`) is not+ loop-safe in two places. `AdaptiveLLMWait` calls `tracker.get_wait_time`+ synchronously, which reads `get_search_context()` and can open the+ user's encrypted database to load estimates; the wrapper's own comment+ warns not to enable rate limiting before that is fixed. And+ `_acall_rate_limited` also calls `tracker.record_outcome` synchronously+ on the loop after each call. `record_outcome`+ (`web_search_engines/rate_limiting/tracker.py`) reads+ `get_settings_context()`, and its `_update_estimate` reads+ `get_search_context()` for the username and password and persists+ through `metrics_writer.set_user_password` / `metrics_writer.get_session`.+ Under this decision both the wait strategy and `record_outcome` must+ become loop-safe (async, or with their reads and writes done on the+ daemon and the values passed in) or stay daemon-side before rate+ limiting can be turned on for calls made through the shared loop.+- `TokenCountingCallback.on_llm_end` (`metrics/token_counter.py`) writes+ each call's usage to the user's encrypted database through `_save_to_db`+ (`metrics_writer` off the main thread, `get_user_db_session` on it).+ Under `ainvoke` it no longer runs on the daemon thread: langchain-core+ dispatches a synchronous handler that is not `run_inline` (the+ `BaseCallbackHandler` default, which this callback keeps) to the loop's+ default executor with a copy of the task's contextvars, so the write+ would open a database session from loop-side code on a pooled thread,+ against rule 5. It must become loop-safe (or hand its row back to the+ daemon) before any async twin is driven in+ production; #6294 tracks the callback's related attribution gap on the+ async path. Rule 4's minimal submit context does not contain this: the+ callback carries the credential on its own `self.research_context`, not+ in a contextvar (`password = self.research_context.get("user_password")`+ followed by `metrics_writer.set_user_password(username, password)`), so+ the password would be cached in the pooled executor thread's+ `metrics_writer` thread-local, and nothing on that dispatch path calls+ `clear_passwords()`.+- The codebase already has `asyncio.run` sites that rule 7 does not yet+ reflect: `advanced_search_system/strategies/news_strategy.py` (two calls+ running `analyze_findings`), `utilities/llm_utils.py::_close_base_llm`+ (closing a model's async httpx client), and+ `research_library/downloaders/playwright_html.py` (running a Playwright+ coroutine, on a helper thread when a loop is already running). None of+ them awaits a provider's async client for a model call+ (`analyze_findings` reaches the model through the synchronous+ `invoke_llm_sync`). Each is still to be reviewed against rule 7 when+ the shared loop is wired, and the pattern must not spread.+- The Socket.IO emit path is the in-tree precedent for rule 3:+ `web/services/socketio_asgi.py::_schedule_coroutine_threadsafe` hands+ emits from worker threads to the captured uvicorn loop with+ `asyncio.run_coroutine_threadsafe`. Those emits inherit the calling+ thread's contextvars too, which that module relies on deliberately for+ `_logging_enabled_var`; when a research daemon emits, its search context+ rides along into the emit task as well.+- Rule 5 is easy to violate without noticing, because code that works on+ the daemon thread keeps running when it is moved onto the loop but+ changes meaning there. The loop thread has no research settings context,+ so a snapshot-less settings getter silently falls through to its default+ (or raises), and a thread-local DB session opened on the loop thread+ would be one session shared by every interleaved request and task on+ that thread.+ An AST-based or import-boundary lint check should be added to catch+ `threading.local` and DB-session access, importing the settings-context+ module, or calling a settings getter (`get_settings_context`,+ `get_setting_from_snapshot` / `get_bool_setting_from_snapshot` without+ a snapshot that is guaranteed to contain the key,+ `SettingsManager.get_setting`) or `get_search_context` from+ code that is only ever reached through+ the async fan-out path, the same way `check-layer-imports` already+ catches a different boundary violation.+- Estimated remaining work to finish wiring the shared loop and move the+ remaining sync call sites onto submitting their async twins is roughly+ 3-5 engineer-weeks, against roughly 13-16 engineer-weeks for the+ rejected alternative of moving research execution itself onto the loop.++## Alternatives considered++**(a) Move research execution onto the event loop as `asyncio` tasks.**+Rejected. This requires re-basing the settings context, the database+session and the egress audit context off `threading.local` (the search+context and request username are already `ContextVar`s, which tasks do+inherit), rewriting progress reporting and cancellation to be async-aware,+and converting all 33 search engines and every strategy+(`advanced_search_system/strategies/`) to cooperate with a shared loop. It+also gives up part of the per-run isolation the current design has for+free: today one run's blocking call or hung step cannot stall another+run's steps, whereas a blocking call inside one task on a shared loop+stalls every task on it. Separate threads do not isolate CPU work, since+CPU-heavy steps in one run still slow sibling runs through the GIL.++**(b) One event loop per research thread.** Rejected by #6293. LangChain's+process-cached `httpx.AsyncClient` is keyed without any loop or thread+component, so several long-lived loops in the same process still hand+each other's connections around and reproduce the same+`RuntimeError: Event loop is closed` failure as per-call loops, just less+frequently and therefore harder to diagnose.++**(c) `asyncio.run(...)` per model call from inside the daemon thread.**+Rejected by #6293 for the same underlying reason: a fresh loop per call+still shares the one process-cached async client, so the second call+onward either fails outright or is silently retried and double-sent.++## Related++#5854, #6293, #5857, #6095, #6207, #6232, #6347, #6249, #6268, #6246,+#6277, #6294, #6467, #6862