~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""Guard for ADR-0013 decision 1: one daemon thread per research run.
ADR-0013 (``docs/decisions/0013-research-runs-keep-daemon-thread.md``)
decides that each research run started through ``start_research_process``
(the web UI and the queue) keeps its own dedicated daemon thread, with the
server-wide research semaphore enforced before that thread is spawned.
This file pins that shape of the spawn path so a regression (a second
thread per run, a non-daemon thread, a double spawn, a leaked permit) fails
loudly.
This is NOT #5854's measure of done. ADR-0013 redefines that as "no AnyIO
worker-pool thread blocks on model I/O, and fan-out inside a single run
costs no extra OS thread"; research daemon threads were never the AnyIO
pool threads #5854 measured. That measurement needs a different instrument
and is tracked in #6467.
What this file pins
-------------------
The spawn path, exactly:
1. ``start_research_process`` (``web/services/research_service.py``)
creates exactly one ``_ctx_wrapped`` daemon thread per run, and
2. at the sampled instant while N callbacks are parked, no additional
``_ctx_wrapped`` thread is live in the process.
Both are measured by thread identity, not by a process-global count.
``threading.enumerate()`` is filtered to threads whose ``_target``
qualname is ``start_research_process.<locals>._ctx_wrapped`` (or whose
CPython-assigned name carries that same target), and the set that
appeared during the run must equal the set of thread objects the spawn
calls returned. The assertion is exact: ``== runs``.
``threading.active_count()`` is recorded as a diagnostic only — it is
process-global and would pick up unrelated threads.
What this file does NOT pin
---------------------------
The research runner itself. ``run_research_callback`` is substituted with
a stub that parks, so ``run_research_process`` — the LLM calls, the
search engines, the encrypted-database work — never executes. It cannot:
it requires a username, a user password and a real per-user DB, and
``research_service.py`` contains no executor / ``to_thread`` / AnyIO seam
that could dispatch it without them.
Consequently this guard does not observe async work inside the run.
It will not fail merely because parts of the run submit coroutines to the
shared loop (ADR-0013 rules 2-3), as long as the spawn path still creates
one ``_ctx_wrapped`` thread per run. If a future decision replaces the
per-run thread, ADR-0013 must be superseded first and this exact count
updated with it.
Process state
-------------
The benchmark raises the server-wide research concurrency semaphore to
its own run count. ``start_research_process``'s worker releases the
MODULE GLOBAL ``_global_research_semaphore`` resolved at exit time, not a
captured object, so the substituted semaphore is restored only after
every spawned thread has been joined — on the error path too. If a thread
is still alive, the substitution is deliberately left in place and the
run fails, because restoring first would let that thread release a permit
into the production semaphore it never took.
Run directly for a standalone measurement report::
pdm run python tests/web/test_research_thread_occupancy.py --runs 12
"""
import argparse
import sys
import threading
import time
import pytest
# ``start_research_process`` spawns its worker as (research_service.py)::
#
# def _ctx_wrapped(*a, **kw):
# return _ctx.run(_release_semaphore_on_exit, *a, **kw)
#
# thread = threading.Thread(target=_ctx_wrapped, args=(...), kwargs=kwargs)
# thread.daemon = True
#
# No ``name=`` is passed, so CPython names it ``"Thread-N (_ctx_wrapped)"``
# (``threading.Thread.__init__`` appends ``" (<target.__name__>)"`` when a
# target with a ``__name__`` is given), and ``Thread._target`` holds the
# closure until the target returns. The qualname is the precise identity;
# the name is the fallback that survives the target's return, so a thread
# that has finished its callback but not yet exited still counts as
# occupancy.
_SPAWN_TARGET_QUALNAME = "start_research_process.<locals>._ctx_wrapped"
_SPAWN_THREAD_NAME_MARKER = "(_ctx_wrapped)"
def _is_research_spawn_thread(thread: threading.Thread) -> bool:
target = getattr(thread, "_target", None)
if getattr(target, "__qualname__", None) == _SPAWN_TARGET_QUALNAME:
return True
return _SPAWN_THREAD_NAME_MARKER in thread.name
def _research_spawn_threads() -> set:
"""Live threads belonging to the research spawn path, by identity."""
return {t for t in threading.enumerate() if _is_research_spawn_thread(t)}
def _permits(semaphore) -> int:
"""Permit count of a ``threading.Semaphore`` (its private counter)."""
value = getattr(semaphore, "_value", None)
assert isinstance(value, int), (
f"cannot read the permit count of {semaphore!r}; the semaphore "
"implementation changed and this instrument needs updating"
)
return value
def _run_benchmark(
runs: int,
*,
park_timeout: float = 10.0,
release_timeout: float = 30.0,
join_timeout: float = 10.0,
) -> dict:
"""Drive *runs* stubbed research runs through the real spawn path.
Every run is parked inside its callback until all of them have
arrived, so the measurement sees them simultaneously. Only the spawn
wrapper is exercised; see the module docstring for the scope.
"""
from local_deep_research.web.services.research_service import (
start_research_process,
)
import local_deep_research.web.services.research_service as rs
from local_deep_research.web.research_state import remove_active_research
entered_lock = threading.Lock()
entered: list[str] = []
release = threading.Event()
def stub_research_run(research_id, query, mode, **kwargs):
with entered_lock:
entered.append(threading.current_thread().name)
# Park exactly as a model response parks the thread today. The
# hold length is not a parameter: it is the measurement window,
# ended by ``release`` once the count has been taken.
release.wait(timeout=release_timeout)
ids = [f"bench-{i}" for i in range(runs)]
spawned: list[threading.Thread] = []
started_ids: list[str] = []
production_sem = rs._global_research_semaphore
# Headroom above the run count on purpose: if the spawn path ever
# regressed to two threads per run, a semaphore sized exactly to
# ``runs`` would mask it as SystemAtCapacityError instead of letting
# the occupancy assertion below name it.
bench_sem = threading.Semaphore(max(runs, 1) * 2 + 2)
rs._global_research_semaphore = bench_sem
try:
baseline_all = set(threading.enumerate())
baseline_spawn = _research_spawn_threads()
baseline_active = threading.active_count()
for i in range(runs):
# The exact spawn path every research entry point uses:
# HTTP route, queue processor, chat.
spawned.append(
start_research_process(
research_id=ids[i],
query=f"benchmark query {i}",
mode="quick",
run_research_callback=stub_research_run,
)
)
started_ids.append(ids[i])
# Pins the "daemon" leg of the module docstring's claim 1 — no
# test elsewhere asserts on ``.daemon``, so a regression of
# ``thread.daemon = True`` in ``start_research_process`` would
# otherwise go uncaught.
assert all(t.daemon for t in spawned), (
"start_research_process returned a non-daemon thread; "
"research_service.py's `thread.daemon = True` regressed"
)
deadline = time.monotonic() + park_timeout
while len(entered) < runs and time.monotonic() < deadline:
time.sleep(0.05)
during_all = set(threading.enumerate())
during_spawn = _research_spawn_threads()
during_active = threading.active_count()
expected = set(spawned)
new_spawn = during_spawn - baseline_spawn
return {
"runs": runs,
"entered": len(entered),
"spawn_calls_returned_threads": len(spawned),
"occupied": len(new_spawn),
"per_run": round(len(new_spawn) / runs, 2) if runs else None,
# A returned thread that is not live during the measurement.
"missing": sorted(t.name for t in expected - new_spawn),
# A research-path thread nobody here asked for.
"extra": sorted(t.name for t in new_spawn - expected),
# Diagnostics only — process-global, never asserted on.
"diag_active_count_baseline": baseline_active,
"diag_active_count_during": during_active,
"diag_active_count_delta": during_active - baseline_active,
"diag_other_new_threads": sorted(
t.name
for t in during_all - baseline_all
if not _is_research_spawn_thread(t)
),
}
finally:
# 1. Wake every parked run BEFORE touching module state, so no
# worker can reach its release with the production semaphore
# reinstalled.
release.set()
# 2. Join everything that was actually spawned. The worker's
# release resolves ``_global_research_semaphore`` as a module
# global at exit time, so the substitute must still be
# installed while any of them can still run.
leftover = []
for thread in spawned:
thread.join(timeout=join_timeout)
if thread.is_alive():
leftover.append(thread.name)
try:
# 3. Remove only registrations this invocation started. A refused
# duplicate belongs to its existing owner. Use the locked accessor.
for research_id in started_ids:
remove_active_research(research_id)
if leftover:
# Restoring now would hand the production semaphore a permit
# release from a thread that took its permit from the
# substitute. Leave the substitute installed and fail.
raise AssertionError(
f"{len(leftover)} spawned research thread(s) still alive "
f"after join(timeout={join_timeout}): {leftover}. The "
"benchmark semaphore is intentionally left installed; "
"restoring it would inflate the production semaphore."
)
# Every spawned thread has now been joined, so every permit this
# benchmark handed out must be back. This guards against a leak —
# e.g. a deleted ``finally: _global_research_semaphore.release()``
# inside ``_release_semaphore_on_exit``
# — which would otherwise slip past silently: the occupancy
# assertions above never inspect the semaphore, only thread
# identity.
assert _permits(bench_sem) == max(runs, 1) * 2 + 2, (
f"the benchmark semaphore has {_permits(bench_sem)} permits "
f"after every spawned thread was joined, expected "
f"{max(runs, 1) * 2 + 2} (its starting count); a spawned "
"thread did not release the permit it acquired"
)
finally:
# Once every worker has stopped, restore the original object even
# if registry cleanup or permit validation raises. Live workers
# must keep releasing into the substitute they acquired from.
if not leftover:
rs._global_research_semaphore = production_sem
@pytest.mark.slow
@pytest.mark.fd_canary
def test_each_research_run_occupies_exactly_one_spawn_thread():
"""N parked runs occupy exactly N research spawn threads, and no more.
This pins the spawn wrapper's arithmetic — one named daemon per run,
no double-spawn, no leak across runs. It does not measure the
research runner (stubbed) and does not change when fan-out moves to
the shared loop; see the module docstring (ADR-0013 decision 1).
"""
runs = 6
pre_existing = _research_spawn_threads()
results = _run_benchmark(
runs=runs, park_timeout=10.0, release_timeout=30.0, join_timeout=10.0
)
print(f"\n[ADR-0013] research spawn occupancy: {results}")
assert results["entered"] == runs, (
f"{results['entered']} callback entries for {runs} runs — the "
f"measurement premise (exactly {runs} runs, all parked "
"simultaneously) was not established"
)
assert results["spawn_calls_returned_threads"] == runs, (
f"{runs} spawn calls returned "
f"{results['spawn_calls_returned_threads']} threads"
)
assert results["occupied"] == runs, (
f"{runs} parked runs occupied {results['occupied']} research "
f"spawn threads, expected exactly {runs} (one per run). "
f"missing={results['missing']} extra={results['extra']}. "
f"'extra' means a double-spawn or a leaked research thread; "
f"'missing' means a run's thread was not live during the "
f"measurement."
)
assert results["missing"] == [] and results["extra"] == [], (
f"the live research spawn threads are not exactly the ones the "
f"spawn calls returned: missing={results['missing']} "
f"extra={results['extra']}"
)
leftover = sorted(t.name for t in _research_spawn_threads() - pre_existing)
assert leftover == [], (
f"research spawn threads outlived their runs: {leftover}"
)
@pytest.mark.slow
@pytest.mark.fd_canary
def test_spawn_failure_leaves_the_production_semaphore_untouched():
"""A mid-spawn failure must not inflate the server-wide capacity.
``start_research_process``'s worker releases the module-global
``_global_research_semaphore`` resolved at exit time. If the
benchmark restored the production semaphore while already-parked
workers were still running, each of them would release a permit into
a semaphore it never took one from — and because the production
object is a plain ``threading.Semaphore``, not a
``BoundedSemaphore``, that over-release is silent and permanent.
Against the original pre-PR revert (restore the production semaphore
first, then wake, with no join at all) the assertion that fires first
is ``stragglers == []`` below, not the permit count; the
permit-count assertion inside ``_run_benchmark`` is what catches the
narrower variant where only the restore is moved earlier and the
wake/join ordering is otherwise kept.
"""
import local_deep_research.web.services.research_service as rs
pre_existing = _research_spawn_threads()
production_sem = rs._global_research_semaphore
permits_before = _permits(production_sem)
real_start = rs.start_research_process
attempts = []
def exploding_start(*args, **kwargs):
attempts.append(1)
if len(attempts) == 3:
raise RuntimeError("simulated spawn failure on run 3")
return real_start(*args, **kwargs)
# ``_run_benchmark`` imports the symbol at call time, so patching the
# module attribute reaches it.
rs.start_research_process = exploding_start
try:
with pytest.raises(RuntimeError, match="simulated spawn failure"):
_run_benchmark(
runs=6,
park_timeout=2.0,
release_timeout=5.0,
join_timeout=10.0,
)
finally:
rs.start_research_process = real_start
assert len(attempts) == 3, (
f"expected the failure on spawn call 3 of 6, saw {len(attempts)} calls"
)
assert rs._global_research_semaphore is production_sem, (
"the production semaphore was not restored after the failed run"
)
# The two runs that did start have already been joined by
# _run_benchmark's finally, so nothing can still be holding a permit.
stragglers = sorted(
t.name for t in _research_spawn_threads() - pre_existing
)
assert stragglers == [], (
f"research spawn threads survived the failed run: {stragglers} — "
"they would release into the restored production semaphore"
)
assert _permits(production_sem) == permits_before, (
f"the production semaphore went from {permits_before} to "
f"{_permits(production_sem)} permits across a failed benchmark "
"run; the server-wide concurrency cap has been corrupted"
)
@pytest.mark.slow
@pytest.mark.fd_canary
def test_post_admission_failure_restores_semaphore_after_validation(
monkeypatch,
):
"""A failed start can leave an acquired permit; validation must still restore."""
import local_deep_research.web.services.research_service as rs
production_sem = rs._global_research_semaphore
permits_before = _permits(production_sem)
pre_existing = _research_spawn_threads()
def fail_after_admission(*args, **kwargs):
assert rs._global_research_semaphore.acquire(blocking=False)
raise RuntimeError("simulated failure after admission")
try:
with monkeypatch.context() as patch:
patch.setattr(rs, "start_research_process", fail_after_admission)
with pytest.raises(AssertionError, match="benchmark semaphore"):
_run_benchmark(runs=1)
assert rs._global_research_semaphore is production_sem
assert _permits(production_sem) == permits_before
assert _research_spawn_threads() == pre_existing
finally:
# The start callable raised before any worker existed, so restoring here
# also keeps a failed regression assertion isolated from later cases.
rs._global_research_semaphore = production_sem
@pytest.mark.slow
@pytest.mark.fd_canary
def test_refused_duplicate_keeps_the_existing_registry_entry():
"""A failed start does not transfer ownership of an existing registration."""
import local_deep_research.web.services.research_service as rs
from local_deep_research.exceptions import DuplicateResearchError
from local_deep_research.web.research_state import (
get_research_field,
is_research_active,
remove_active_research,
set_active_research,
)
class ExistingWorker:
def is_alive(self):
return True
research_id = "bench-0"
assert not is_research_active(research_id)
owner = object()
set_active_research(
research_id, {"thread": ExistingWorker(), "owner": owner}
)
production_sem = rs._global_research_semaphore
permits_before = _permits(production_sem)
try:
with pytest.raises(DuplicateResearchError):
_run_benchmark(runs=1)
assert get_research_field(research_id, "owner") is owner
assert rs._global_research_semaphore is production_sem
assert _permits(production_sem) == permits_before
finally:
remove_active_research(research_id)
# Duplicate admission was refused, so this case started no worker.
rs._global_research_semaphore = production_sem
@pytest.mark.parametrize("runs", ["0", "-1"])
def test_cli_rejects_non_positive_run_count_before_starting(
monkeypatch, capsys, runs
):
monkeypatch.setattr(sys, "argv", ["thread-occupancy", "--runs", runs])
def unexpected_benchmark(**kwargs):
pytest.fail("invalid counts must not start a benchmark")
monkeypatch.setattr(
sys.modules[__name__], "_run_benchmark", unexpected_benchmark
)
with pytest.raises(SystemExit) as error:
main()
assert error.value.code == 2
assert "--runs must be greater than zero" in capsys.readouterr().err
def test_cli_reports_incomplete_measurement_without_success_conclusion(
monkeypatch, capsys
):
monkeypatch.setattr(sys, "argv", ["thread-occupancy", "--runs", "2"])
observed = {
"runs": 2,
"entered": 1,
"spawn_calls_returned_threads": 1,
"occupied": 1,
"missing": ["worker"],
"extra": [],
}
monkeypatch.setattr(
sys.modules[__name__], "_run_benchmark", lambda **kwargs: observed
)
assert main() == 1
captured = capsys.readouterr()
output = captured.out
assert "INCOMPLETE" in captured.err
assert "INCOMPLETE" not in output
assert "entered: 1" in output
assert "occupied: 1" in output
assert "missing: ['worker']" in output
assert "exactly one dedicated thread per run" not in output
assert "at the sampled instant" in output
def test_cli_exits_zero_for_complete_measurement(monkeypatch, capsys):
monkeypatch.setattr(sys, "argv", ["thread-occupancy", "--runs", "2"])
observed = {
"runs": 2,
"entered": 2,
"spawn_calls_returned_threads": 2,
"occupied": 2,
"missing": [],
"extra": [],
}
monkeypatch.setattr(
sys.modules[__name__], "_run_benchmark", lambda **kwargs: observed
)
assert main() == 0
assert "INCOMPLETE" not in capsys.readouterr().err
def _measurement_complete(results: dict) -> bool:
runs = results["runs"]
return (
results["entered"] == runs
and results["spawn_calls_returned_threads"] == runs
and results["occupied"] == runs
and not results["missing"]
and not results["extra"]
)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--runs", type=int, default=12)
args = parser.parse_args()
if args.runs <= 0:
parser.error("--runs must be greater than zero")
results = _run_benchmark(runs=args.runs)
print("[ADR-0013] research-run spawn-thread occupancy")
for key, value in results.items():
print(f" {key}: {value}")
print(
"\nThese values describe the spawn wrapper at the sampled instant. "
"The research callback is stubbed; measuring occupancy savings "
"inside the research runner requires a separate instrument. "
"#5854's measure of done (no AnyIO pool thread blocked on model "
"I/O, fan-out without extra OS threads) is tracked in #6467."
)
if not _measurement_complete(results):
print(
"\nINCOMPLETE: entered/spawned/occupied do not all equal runs, "
"or missing/extra is non-empty.",
file=sys.stderr,
)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())