Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

docs(adr): research runs keep their daemon thread; async is I/O fan-out on one shared loop (#6450)

LearningCircuit1ce4ed6ca38e
diff --git a/docs/decisions/0013-research-runs-keep-daemon-thread.md b/docs/decisions/0013-research-runs-keep-daemon-thread.md
new file mode 100644
index 0000000000..dbe7e2fa68
--- /dev/null
+++ b/docs/decisions/0013-research-runs-keep-daemon-thread.md
@@ -0,0 +1,354 @@
+# ADR-0013: Research runs keep their dedicated daemon thread; async is for I/O fan-out only
+
+**Date:** 2026-09-13
+**Status:** Accepted
+
+## Context
+
+### How a research run executes today
+
+`web/services/research_service.py::start_research_process` acquires the
+server-wide `_global_research_semaphore` (a `threading.Semaphore` sized by
+`server.max_concurrent_research`, default 10) in the caller's own thread
+with a non-blocking `acquire(blocking=False)`. If no slot is free it raises
+`SystemAtCapacityError`, which the web routes turn into HTTP 429 (the queue
+processor re-queues the run instead), so capacity is enforced before any
+thread is spawned. It then
+copies the calling request's contextvars with `contextvars.copy_context()`
+(on default GIL builds a bare `threading.Thread` starts with an empty
+context, which would break `get_current_username()` and metrics attribution
+inside the worker; free-threaded builds enable
+`sys.flags.thread_inherit_context` by default, and the explicit copy keeps
+the behaviour the same on both) and
+starts one dedicated `threading.Thread` with `daemon = True` per run.
+`check_and_start_research` performs an atomic check-and-start keyed by
+`research_id` so a retry cannot double-spawn the same run.
+
+Inside that thread, `run_research_process` calls `set_search_context(...)`
+first, before any other work, establishing the research id, username and
+password so the run's early log lines can be attributed. The settings
+snapshot in that first call may still be empty
+(`kwargs.get("settings_snapshot") or {}`); a second
+`set_search_context(shared_research_context)` later in the run installs the
+populated snapshot. Settings reads inside the run go through
+`config/thread_settings.py`, which keeps a
+`_thread_local = threading.local()` settings context
+(`set_settings_context` / `clear_settings_context` / `get_settings_context`).
+`get_setting_from_snapshot` and `get_bool_setting_from_snapshot` resolve
+against that thread-local context whenever the key is not found in the
+`settings_snapshot` they are handed, whether because no snapshot was
+passed or because the snapshot passed lacks that key. (`get_typed_setting_value` in `settings/manager.py` is
+only the pure type-coercion helper those getters call; it reads no context.)
+Database access goes
+through `database/thread_local_session.py`'s `ThreadLocalSessionManager`,
+which hands out one session per thread from `get_session(username,
+password)`, backed by its own `threading.local()`, tracked per thread id in
+`_thread_credentials` for logout/idle cleanup, and scoped by
+`enter_scope` / `exit_scope` around each `get_user_db_session` block.
+Progress reporting and cancellation live in the same thread: a
+`progress_callback(message, progress_percent, metadata)` closure checks
+`is_termination_requested(research_id)` on every call and, whenever the step
+reports a progress value, calls `update_progress_and_check_active(...)`
+(otherwise `is_research_active(research_id)`). It is wired into the
+strategy with `system.set_progress_callback(progress_callback)`.
+
+In short, the run's state is bound to its dedicated thread in two different
+ways. The settings context, the DB session and the egress audit context
+(`security/egress/audit_hook.py`, `_thread_local = threading.local()`, set
+by `set_active_context`) are `threading.local` state. The search context
+(research id, username, password, settings snapshot;
+`utilities/thread_context.py`'s `_search_context_var`) and the request
+username (`utilities/request_context.py`'s `_username_var`) are
+`ContextVar`s, carried into the thread by the copied context. Progress and
+cancellation run on that same thread. All of it lives and dies with one
+research run's dedicated thread. The settings snapshot that
+`run_research_process` builds once at run start and installs with
+`set_settings_context` and `set_search_context` is what guarantees a
+research run keeps exactly the settings it started with, no matter what
+settings edits are made while the run is in progress; that guarantee is a
+design requirement this decision codifies, not an accident of the current
+implementation.
+
+### What #5854 is actually about, and the misreading it invites
+
+#5854 measured 89 call sites doing synchronous `.invoke()` / `.stream()`
+against an AnyIO worker pool of 40 threads, process-wide. A `sync def`
+FastAPI route handler occupies one of those 40 pool threads for the whole
+duration of the call underneath it; an LLM call held one for seconds to
+minutes. The fix #5854 asks for is converting those calls to
+`ainvoke` / `astream` so the pool thread is free while the model is called.
+
+Research runs started from the web UI or the queue never went through that
+pool. They run on their own dedicated `threading.Thread`, started directly
+by `start_research_process`, not by FastAPI's `run_in_threadpool`. Reading
+#5854 as "move research execution onto the event loop so it stops holding
+threads" is a misreading: those runs were never taking a thread from the
+pool #5854 is about, so converting research itself to loop tasks would not
+free anything #5854 cares about. It would only give up the per-run thread
+isolation described above, for no matching benefit.
+
+Not every research run takes that path, though. REST v1 `quick_summary` /
+`generate_report` (`web/routers/api_v1.py`) call the library functions
+through `run_db_sync`, which wraps `asyncio.to_thread`, so the whole run
+executes inline on a pooled default-executor thread. Scheduled subscription
+runs (`scheduler/background.py::_trigger_subscription_research_sync`) call
+`quick_summary` directly on an APScheduler worker thread. Neither path
+acquires `_global_research_semaphore`. The web benchmark runner
+(`benchmarks/web_api/benchmark_service.py`) is a third shape: it starts its
+own daemon thread (`_run_benchmark_thread`), not through
+`start_research_process`, and calls `quick_summary` per task
+(`_process_benchmark_task`); it does count against the global semaphore,
+but with a blocking `_global_research_semaphore.acquire()` per task rather
+than the non-blocking acquire and 429 of the web routes. Library callers of
+`api/research_functions.py` run in the caller's own thread and have no
+server event loop at all.
+
+#6232 is the case #5854 is actually aimed at: it converted the notes AI
+routes to `async def` and awaits `llm.ainvoke` directly, because those
+routes genuinely ran as `sync def` handlers on the AnyIO pool. That is a
+different execution model from a research run, which is why the same fix
+does not apply to research.
+
+### The #6293 constraint on where async I/O can live
+
+#6293 found that `langchain_openai` and `langchain_anthropic` each cache
+`httpx.AsyncClient` instances per process with `functools.lru_cache`, keyed
+with no event-loop or thread component. In `langchain_openai` the cache is
+`_cached_async_httpx_client`, keyed by `(base_url, timeout,
+socket_options)`. In `langchain_anthropic` the cached function is
+`_get_default_async_httpx_client`, keyed by `(base_url, timeout,
+anthropic_proxy)`. The cached client's keep-alive connections are bound to
+whichever event loop first uses them. Any code that awaits from a different
+loop than the one that made the previous call reuses a connection from the
+wrong loop and fails with `RuntimeError: Event loop is closed`, with the
+provider SDK's retry logic sometimes turning that into a silently duplicated
+(and double-billed) request. This happens whether the second loop is created
+by a fresh `asyncio.run(...)` per call or by a second long-lived loop
+running on a different thread; a per-research-thread event loop reproduces
+the same failure, only less often. #6293 names the safe shape: exactly one
+long-lived event loop in the process ever touches these clients (the
+uvicorn request loop, or one dedicated loop that other code reaches via
+`asyncio.run_coroutine_threadsafe`), unless LDR takes over supplying and
+closing its own per-loop `http_async_client` instances, which it does not
+do today.
+
+### #5857 was a related but separate problem
+
+#5857 reported that `DatabaseMiddleware`'s cleanup was `async def` and ran
+on the event-loop thread, while the session it was cleaning up lived in
+`threading.local()` on the AnyIO worker thread that served the request, so
+a sync route's DB session was never returned to the pool. #6095 and its
+follow-up #6207 addressed that mechanism: `web/fastapi_app.py` installs
+`app.router.route_class = WorkerCleanupAPIRoute`
+(`web/dependencies/threadpool.py`), which returns each sync route's
+connections on the worker thread when that request finishes, and
+`run_db_sync` does the same for explicit `asyncio.to_thread` work. That
+failure mode required a database session to cross the worker-thread /
+event-loop boundary. Research runs open and close their DB session on the
+same daemon thread for the run's lifetime and never hand it to loop-side
+code, so they were never subject to it.
+
+### The async twins already added under #5854
+
+Several tranches have already added an async counterpart next to an
+existing synchronous call, without switching the production caller:
+
+- #6249 (merged): `citation_handlers/base_citation_handler.py` gained
+ `_invoke_text_async` / `_invoke_with_streaming_async` beside the
+ synchronous `_invoke_text`; `web_search_engines/engines/full_search.py`
+ gained `_check_urls_async` beside `check_urls`.
+- #6268 (merged): `advanced_search_system/strategies/topic_organization_strategy.py`
+ consolidated its model calls onto `_invoke_model` (sync) and added
+ `_ainvoke_model` (async); production strategy calls stayed synchronous.
+- #6246 (merged): news generators (`generate_headline_async`,
+ `generate_topics_async`) and benchmark graders (`grade_single_result_async`,
+ `grade_results_async`) gained async entry points beside their synchronous
+ originals; no production caller has been converted yet.
+- #6277 (merged): `utilities/llm_utils.py` gained `invoke_llm_sync`, the one
+ call site the research pipeline's question generators, filters,
+ explorers, context managers and evaluators go through, and its async core
+ `ainvoke_llm`, which nothing in production awaits yet.
+
+`RateLimitedLLMWrapper.ainvoke` (`web_search_engines/rate_limiting/llm/wrapper.py`,
+hardened in #6347) and `ProcessingLLMWrapper.ainvoke`
+(`config/llm_config.py`) both already `await` the wrapped provider's own
+async client rather than bridging through a freshly created loop. That is
+the correct shape for a fan-out interface and is the pattern this decision
+generalizes.
+
+## Decision
+
+1. Each research run started through `start_research_process` (the web UI
+ and the queue) keeps its dedicated daemon thread. That thread owns the
+ run's settings context, database session, egress/search context,
+ progress reporting and cancellation checks for the run's entire
+ lifetime, exactly as today. The REST v1 and scheduler paths, which run
+ research inline on a pooled or scheduler thread without the global
+ semaphore, are noted exceptions to this rule. This decision does not
+ change them, and whether to bring them under it is left to a follow-up
+ (#6862). The web benchmark runner, which runs `quick_summary` on its own
+ daemon thread and blocks on the global semaphore per task, is likewise
+ outside `start_research_process` and unchanged by this decision.
+ Library callers have no server loop; for them rules 2 and 3 do not
+ apply and the synchronous twins remain the only entry point.
+2. The process holds exactly one long-lived event loop for LLM and HTTP
+ async I/O. Today that is the FastAPI/uvicorn loop. A dedicated I/O
+ thread with its own loop is an acceptable alternative, but only ever
+ one such loop, never one per research thread. Which of the two hosts
+ the loop is not fixed by this decision and is left open until the code
+ commits to one.
+3. Daemon code that needs an async result submits the coroutine to that
+ shared loop with `asyncio.run_coroutine_threadsafe` and waits on the
+ returned future, with a timeout and a cancel path tied to
+ `is_termination_requested`. Daemon code does not open its own event
+ loop and does not call `asyncio.run` from inside the daemon thread.
+4. Everything a submitted coroutine needs is passed in explicitly as
+ arguments. The settings snapshot is taken once on the daemon thread at
+ run start and is the only settings source for that run: loop-side code
+ receives it as a read-only argument, never calls the thread-local
+ settings getters (`get_settings_context`, or `get_setting_from_snapshot`
+ / `get_bool_setting_from_snapshot` for a key the passed
+ `settings_snapshot` does not contain, including when none is passed,
+ since both then fall back to the thread-local context), never calls
+ `SettingsManager.get_setting`, and
+ never re-reads settings from the database, so a settings edit made
+ while a run is in progress cannot change what that run does.
+ Fan-out passes the same frozen snapshot to every coroutine it submits,
+ together with the username, the scoped credential the call needs, and
+ any database rows read ahead of time. A coroutine never reaches back
+ into the daemon thread's state to fetch something it was not handed.
+ This includes ambient contextvars: `asyncio.run_coroutine_threadsafe`
+ schedules the task with a copy of the submitting thread's contextvars,
+ so a coroutine submitted from the daemon inherits the daemon's search
+ context, including the user's password, unless the submit helper runs
+ it in an explicit minimal context. The helper must do so. The egress
+ audit context is `threading.local` and does not follow the coroutine
+ at all: it must be passed in and re-armed explicitly on the loop side.
+5. Loop-side code, meaning anything running as a task on the shared loop,
+ must never touch `threading.local` settings or session state, must not
+ read the search context through `get_search_context()`, and must never
+ open a database session. If a coroutine needs data from the database,
+ the daemon reads it first and passes it in per rule 4.
+6. The synchronous twin of each of these calls remains the canonical
+ production entry point running on the daemon thread. The async twin
+ exists to be submitted to the shared loop for fan-out, meaning several
+ concurrent calls within one run, not to replace the sync path as the
+ default caller.
+7. A fresh `asyncio.run(...)` per call and a persistent event loop per
+ thread are both forbidden, per #6293.
+
+## Consequences
+
+- #5854's measure of done changes. It is no longer "fewer research
+ threads", because research threads were never the AnyIO pool threads
+ #5854 measured. It becomes: no AnyIO worker-pool thread blocks on model
+ I/O, and fan-out inside a single research run (parallel citation, search
+ or topic calls) costs no extra OS thread because it is submitted to the
+ one shared loop instead of spawning a thread or a loop of its own. The
+ wiring is tracked in #6467.
+- #5857 was never on the critical path for research concurrency, since a
+ research run's DB session never leaves its daemon thread. Its mechanism
+ for sync routes was addressed by #6095 and #6207.
+- LLM rate limiting (`web_search_engines/rate_limiting/llm/wrapper.py`,
+ currently disabled by `_should_rate_limit` returning `False`) is not
+ loop-safe in two places. `AdaptiveLLMWait` calls `tracker.get_wait_time`
+ synchronously, which reads `get_search_context()` and can open the
+ user's encrypted database to load estimates; the wrapper's own comment
+ warns not to enable rate limiting before that is fixed. And
+ `_acall_rate_limited` also calls `tracker.record_outcome` synchronously
+ on the loop after each call. `record_outcome`
+ (`web_search_engines/rate_limiting/tracker.py`) reads
+ `get_settings_context()`, and its `_update_estimate` reads
+ `get_search_context()` for the username and password and persists
+ through `metrics_writer.set_user_password` / `metrics_writer.get_session`.
+ Under this decision both the wait strategy and `record_outcome` must
+ become loop-safe (async, or with their reads and writes done on the
+ daemon and the values passed in) or stay daemon-side before rate
+ limiting can be turned on for calls made through the shared loop.
+- `TokenCountingCallback.on_llm_end` (`metrics/token_counter.py`) writes
+ each call's usage to the user's encrypted database through `_save_to_db`
+ (`metrics_writer` off the main thread, `get_user_db_session` on it).
+ Under `ainvoke` it no longer runs on the daemon thread: langchain-core
+ dispatches a synchronous handler that is not `run_inline` (the
+ `BaseCallbackHandler` default, which this callback keeps) to the loop's
+ default executor with a copy of the task's contextvars, so the write
+ would open a database session from loop-side code on a pooled thread,
+ against rule 5. It must become loop-safe (or hand its row back to the
+ daemon) before any async twin is driven in
+ production; #6294 tracks the callback's related attribution gap on the
+ async path. Rule 4's minimal submit context does not contain this: the
+ callback carries the credential on its own `self.research_context`, not
+ in a contextvar (`password = self.research_context.get("user_password")`
+ followed by `metrics_writer.set_user_password(username, password)`), so
+ the password would be cached in the pooled executor thread's
+ `metrics_writer` thread-local, and nothing on that dispatch path calls
+ `clear_passwords()`.
+- The codebase already has `asyncio.run` sites that rule 7 does not yet
+ reflect: `advanced_search_system/strategies/news_strategy.py` (two calls
+ running `analyze_findings`), `utilities/llm_utils.py::_close_base_llm`
+ (closing a model's async httpx client), and
+ `research_library/downloaders/playwright_html.py` (running a Playwright
+ coroutine, on a helper thread when a loop is already running). None of
+ them awaits a provider's async client for a model call
+ (`analyze_findings` reaches the model through the synchronous
+ `invoke_llm_sync`). Each is still to be reviewed against rule 7 when
+ the shared loop is wired, and the pattern must not spread.
+- The Socket.IO emit path is the in-tree precedent for rule 3:
+ `web/services/socketio_asgi.py::_schedule_coroutine_threadsafe` hands
+ emits from worker threads to the captured uvicorn loop with
+ `asyncio.run_coroutine_threadsafe`. Those emits inherit the calling
+ thread's contextvars too, which that module relies on deliberately for
+ `_logging_enabled_var`; when a research daemon emits, its search context
+ rides along into the emit task as well.
+- Rule 5 is easy to violate without noticing, because code that works on
+ the daemon thread keeps running when it is moved onto the loop but
+ changes meaning there. The loop thread has no research settings context,
+ so a snapshot-less settings getter silently falls through to its default
+ (or raises), and a thread-local DB session opened on the loop thread
+ would be one session shared by every interleaved request and task on
+ that thread.
+ An AST-based or import-boundary lint check should be added to catch
+ `threading.local` and DB-session access, importing the settings-context
+ module, or calling a settings getter (`get_settings_context`,
+ `get_setting_from_snapshot` / `get_bool_setting_from_snapshot` without
+ a snapshot that is guaranteed to contain the key,
+ `SettingsManager.get_setting`) or `get_search_context` from
+ code that is only ever reached through
+ the async fan-out path, the same way `check-layer-imports` already
+ catches a different boundary violation.
+- Estimated remaining work to finish wiring the shared loop and move the
+ remaining sync call sites onto submitting their async twins is roughly
+ 3-5 engineer-weeks, against roughly 13-16 engineer-weeks for the
+ rejected alternative of moving research execution itself onto the loop.
+
+## Alternatives considered
+
+**(a) Move research execution onto the event loop as `asyncio` tasks.**
+Rejected. This requires re-basing the settings context, the database
+session and the egress audit context off `threading.local` (the search
+context and request username are already `ContextVar`s, which tasks do
+inherit), rewriting progress reporting and cancellation to be async-aware,
+and converting all 33 search engines and every strategy
+(`advanced_search_system/strategies/`) to cooperate with a shared loop. It
+also gives up part of the per-run isolation the current design has for
+free: today one run's blocking call or hung step cannot stall another
+run's steps, whereas a blocking call inside one task on a shared loop
+stalls every task on it. Separate threads do not isolate CPU work, since
+CPU-heavy steps in one run still slow sibling runs through the GIL.
+
+**(b) One event loop per research thread.** Rejected by #6293. LangChain's
+process-cached `httpx.AsyncClient` is keyed without any loop or thread
+component, so several long-lived loops in the same process still hand
+each other's connections around and reproduce the same
+`RuntimeError: Event loop is closed` failure as per-call loops, just less
+frequently and therefore harder to diagnose.
+
+**(c) `asyncio.run(...)` per model call from inside the daemon thread.**
+Rejected by #6293 for the same underlying reason: a fresh loop per call
+still shares the one process-cached async client, so the second call
+onward either fails outright or is silently retried and double-sent.
+
+## Related
+
+#5854, #6293, #5857, #6095, #6207, #6232, #6347, #6249, #6268, #6246,
+#6277, #6294, #6467, #6862