~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""Runtime detection of event-loop stalls, at any call depth.
The AST guards in ``tests/web/routers/test_async_handler_blocking_guard.py``
catch blocking calls written literally in an ``async def`` handler, and one
level of indirection through a sync helper: a same-file one (since the
pre-merge readiness audit), or one imported from another module of this
package. All are static, so all share a ceiling: blocking reached more than
one call deep, through an async helper or a method, or through a callee the
scanner cannot resolve is invisible to them unless one of their lists names a
function on the way. The audit named blocking through a helper in ANOTHER
module as its top residual gap and proposed ``PYTHONASYNCIODEBUG=1`` with a
lowered ``slow_callback_duration`` as the experiment that would close it.
This is that experiment, as a test rather than a manual procedure.
How it works: asyncio's debug mode times every callback the loop executes and
logs ``Executing <...> took N seconds`` above ``slow_callback_duration``. A
coroutine that blocks — for any reason, at any depth, through any number of
modules — runs as one long callback and trips it. That is the property the
static guards can only approximate.
Deliberately NOT run against the lifespan: startup does heavy synchronous work
by design (engine bootstrap, scheduler start), and the repo permits exactly one
lifespan test per process anyway. The subject here is request handling.
"""
import asyncio
import pytest
# Threshold. Generous on purpose: this is a stall detector, not a benchmark.
# A handler doing ordinary work stays far below it; one doing a synchronous
# network call, a SQLCipher open, or a large pure-Python parse blows past it.
# CI runners are noisy and shared, so a tight bound would flake — the failure
# this is built to catch (the TOML parse the audit found) was ~5.5 SECONDS.
SLOW_CALLBACK_SECONDS = 1.0
# Routes reachable without authentication. Unauthenticated requests still
# traverse the entire middleware stack -- security headers, secure cookies,
# CSRF, rate limiting, the database middleware, path-scoped CORS -- which is
# where a stall would hurt every route at once rather than just one.
PUBLIC_PATHS = [
"/api/v1/health",
"/auth/login",
"/",
]
def _slow_callback_messages(caplog):
"""asyncio's slow-callback warnings, out of pytest's stdlib capture.
``caplog`` is the right tool here even though this repo's own logging goes
through loguru and is invisible to it: the subject is asyncio's OWN warning,
emitted by ``asyncio.base_events`` through the standard library. Using
caplog also avoids importing ``logging`` in a test file, which the repo's
custom-checks hook reserves for the few modules that bridge the two systems.
"""
return [
record.getMessage()
for record in caplog.records
if "took" in record.getMessage() and "seconds" in record.getMessage()
]
@pytest.mark.asyncio
async def test_no_request_stalls_the_event_loop(caplog):
"""Drive the real app under asyncio debug and assert nothing stalls."""
import httpx
from local_deep_research.web.fastapi_app import app
loop = asyncio.get_running_loop()
previous_debug = loop.get_debug()
previous_threshold = loop.slow_callback_duration
caplog.set_level("WARNING", logger="asyncio")
loop.set_debug(True)
loop.slow_callback_duration = SLOW_CALLBACK_SECONDS
try:
# Yield once before doing anything timed. asyncio records a
# callback's start time inside Handle._run(), reading get_debug() at
# that moment -- so work that continues the step which was ALREADY
# running when debug was switched on is never timed. Awaiting here
# ends that step; everything below runs in steps that will be.
await asyncio.sleep(0)
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(
transport=transport, base_url="http://testserver"
) as client:
for path in PUBLIC_PATHS:
# Status is not the subject -- a 401/403/404 still means the
# request traversed the middleware stack, which is what is
# being timed. Only a stall fails this test.
await client.get(path)
finally:
loop.set_debug(previous_debug)
loop.slow_callback_duration = previous_threshold
assert not _slow_callback_messages(caplog), (
"asyncio reported callbacks exceeding "
f"{SLOW_CALLBACK_SECONDS}s while serving ordinary requests -- "
"something on the request path is blocking the event loop:\n "
+ "\n ".join(_slow_callback_messages(caplog))
+ "\n\nThis catches what the AST guards miss: blocking reached more "
"than one call deep, through an async helper or a method, or "
"through a callee the scanner cannot resolve, when none of their "
"lists names a function on the way. Find the call and offload it "
"(asyncio.to_thread / run_db_sync), or make it async."
)
@pytest.mark.asyncio
async def test_the_stall_detector_actually_detects_a_stall(caplog):
"""Anti-vacuity: the test above asserts an empty list, which is also what
a detector wired up wrongly produces.
Blocks the loop deliberately and confirms asyncio reports it through the
same capture path. If this stops failing-when-it-should, the test above is
no longer evidence of anything.
"""
import time
loop = asyncio.get_running_loop()
previous_debug = loop.get_debug()
previous_threshold = loop.slow_callback_duration
caplog.set_level("WARNING", logger="asyncio")
loop.set_debug(True)
loop.slow_callback_duration = 0.05
try:
# See the sibling test: yield first so the blocking work below runs in
# a step that started with debug already enabled. Without this the
# block is invisible and this control passes for the wrong reason --
# which is exactly what it did on the first attempt.
await asyncio.sleep(0)
# Not @pytest.mark.slow and not freezegun-able: the thing under
# test is asyncio measuring real elapsed wall-clock inside a single
# callback, so the block has to actually happen. 0.2s against a
# 0.05s threshold is the smallest margin that is not flaky.
time.sleep(
0.2
) # allow: unmarked-sleep — blocking the loop IS the assertion
# The warning is logged when the step completes and the loop times it,
# so yield again before inspecting.
await asyncio.sleep(0)
finally:
loop.set_debug(previous_debug)
loop.slow_callback_duration = previous_threshold
assert _slow_callback_messages(caplog), (
"asyncio debug mode did not report a deliberate 0.2s block against a "
"0.05s threshold -- the capture above is not wired to asyncio's "
"slow-callback logging, so the sibling test proves nothing"
)