Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

41.7 KB
Raw Blame
"""Concurrency behaviour of the FastAPI app under ``workers=1``.

``web/app.py`` hardcodes ``workers=1`` (Socket.IO needs a single process
without a Redis message queue). Threaded Werkzeug used to give every request
its own OS thread; there is now exactly one event loop plus one AnyIO
worker pool, in one process, for the whole instance. Every prior audit of
this migration checked *correctness* under that model. These tests check
*behaviour under concurrent load*, which nothing else does.

Five properties, in rising order of how much a regression would cost:

1. **Concurrent callers really are served concurrently.** ~248 ``def``
   routes run in the AnyIO threadpool; ~65 ``async def`` routes run on the
   loop. Both must overlap. A ``threading.Barrier`` / ``asyncio.Barrier``
   with N parties can only trip if N handlers are genuinely in flight, so
   it proves overlap without timing anything.

2. **A blocking sync handler must not stall the loop.** This is the core
   hazard of the model: if Starlette's threadpool offload ever stopped
   applying (a route accidentally becoming ``async def`` around blocking
   work is the realistic version), one slow request would freeze every
   other request, every Socket.IO event and the Docker healthcheck.

3. **Threadpool exhaustion must queue, not fail.** The AnyIO limiter is a
   hard ceiling; requests past it must wait and then succeed, and async
   routes must keep answering while it is saturated.
   ``warn_if_threadpool_exceeds_db_pool`` reconciles that ceiling against
   the per-user DB pool (POOL_SIZE 20 + MAX_OVERFLOW 40 = 60), because a
   sync route's DB session is pinned to its worker thread.

4. **Per-request state must not leak between concurrent requests.** The
   username/session_id contextvar, the thread-local DB session and the
   thread-local settings context are all per-request. Under one shared
   loop and a *pooled* threadpool, a leak here is a cross-user data
   disclosure, not a slowdown. This is the highest-value section: tests
   marked "control" beside it deliberately break the property to show the
   harness would notice.

5. **Nothing bounds concurrency except the AnyIO limiter.**
   ``uvicorn.run()`` sets no ``limit_concurrency``, so async routes have no
   ceiling at all and the sync ceiling is a knob most operators never see.

What breaks if this regresses: the instance serves one request at a time,
or serves them concurrently but hands one user another user's identity.
"""

from __future__ import annotations

import ast
import asyncio
import itertools
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from contextlib import asynccontextmanager, contextmanager
from pathlib import Path
from unittest.mock import MagicMock, patch

import anyio.to_thread
import httpx
import pytest
from fastapi import FastAPI, Request

from local_deep_research.config.thread_settings import (
    clear_settings_context,
    get_settings_context,
    set_settings_context,
)
from local_deep_research.database.pool_config import MAX_OVERFLOW, POOL_SIZE
from local_deep_research.utilities.request_context import (
    get_current_session_id,
    get_current_username,
)
from local_deep_research.web import app as web_app_module
from local_deep_research.web.dependencies.threadpool import run_db_sync
from local_deep_research.web.fastapi_app import (
    DatabaseMiddleware,
    warn_if_threadpool_exceeds_db_pool,
)

# Bounds on every blocking wait in this file. These are give-up deadlines,
# never a measurement: each is either reached instantly (the property holds)
# or burned in full (the property is broken and the test fails). Nothing here
# asserts on elapsed time, so CI noise cannot flip a result.
TRIP_TIMEOUT = 10.0
# The one wait that is *expected* to expire, in the serialisation control.
# Kept short because it is always paid in full.
NO_TRIP_TIMEOUT = 0.5

DB_POOL_CAPACITY = POOL_SIZE + MAX_OVERFLOW


# ---------------------------------------------------------------------------
# Harness
# ---------------------------------------------------------------------------


@asynccontextmanager
async def _drive(app):
    """An httpx client wired straight to the ASGI callable.

    ``ASGITransport`` calls the app in the caller's event loop, so N
    ``asyncio.gather``-ed requests are N real concurrent ASGI invocations —
    the same shape uvicorn produces, minus the socket.
    """
    async with httpx.AsyncClient(
        transport=httpx.ASGITransport(app=app), base_url="http://testserver"
    ) as client:
        yield client


@contextmanager
def _threadpool_limit(tokens: int):
    """Resize the AnyIO worker pool for the duration of the block.

    ``current_default_thread_limiter()`` is the exact object the lifespan
    resizes for ``LDR_WEB_THREADPOOL_MAX_THREADS``, so this exercises the
    real ceiling rather than a stand-in. Restored unconditionally: it is
    per-event-loop, but a shared loop would otherwise carry the value into
    the next test.
    """
    limiter = anyio.to_thread.current_default_thread_limiter()
    previous = limiter.total_tokens
    limiter.total_tokens = tokens
    try:
        yield limiter
    finally:
        limiter.total_tokens = previous


async def _yield_until(predicate, *, timeout: float = TRIP_TIMEOUT) -> None:
    """Spin the event loop until ``predicate()`` is true.

    Deliberately ``asyncio.sleep(0)`` rather than a delay: this both waits
    for the condition *and* proves the loop was free to run other callbacks
    while waiting. If the loop were stalled this never gets a turn and the
    ``wait_for`` deadline fires.
    """

    async def _spin():
        while not predicate():
            await asyncio.sleep(0)

    await asyncio.wait_for(_spin(), timeout=timeout)


# ---------------------------------------------------------------------------
# 1. Concurrent requests are actually served concurrently
# ---------------------------------------------------------------------------


def _barrier_app(parties: int, timeout: float):
    """App whose sync route can only answer if ``parties`` callers overlap."""
    app = FastAPI()
    barrier = threading.Barrier(parties, timeout=timeout)

    @app.get("/gated")
    def gated():
        try:
            barrier.wait()
        except threading.BrokenBarrierError:
            # Not an assertion swallow: the barrier expiring is a *result*,
            # reported to the caller so the test can assert on it. Raising
            # here would surface as a transport error instead.
            return {"tripped": False, "thread": threading.get_ident()}
        return {"tripped": True, "thread": threading.get_ident()}

    return app


@pytest.mark.asyncio
async def test_concurrent_sync_handlers_run_simultaneously():
    """N plain ``def`` routes must be in flight at once, on N threads.

    Starlette runs every ``def`` handler through ``anyio.to_thread.run_sync``.
    The barrier is the assertion: with the offload working, all N arrive and
    it trips; if handlers ran one at a time the first would wait alone until
    its deadline. See the control below, which shows exactly that.
    """
    parties = 6
    app = _barrier_app(parties, TRIP_TIMEOUT)

    async with _drive(app) as client:
        responses = await asyncio.gather(
            *[client.get("/gated") for _ in range(parties)]
        )

    payloads = [r.json() for r in responses]
    assert all(r.status_code == 200 for r in responses), (
        f"statuses: {[r.status_code for r in responses]}"
    )
    assert all(p["tripped"] for p in payloads), (
        f"{parties} concurrent requests to a sync route did not all reach the "
        "barrier within "
        f"{TRIP_TIMEOUT}s — sync handlers are being serialised. Starlette's "
        "anyio.to_thread offload is the only thing giving this app "
        "concurrency on its ~248 sync routes under workers=1."
    )
    assert len({p["thread"] for p in payloads}) == parties, (
        "each concurrent sync request must get its own worker thread; got "
        f"{len({p['thread'] for p in payloads})} distinct threads for "
        f"{parties} requests"
    )


@pytest.mark.asyncio
async def test_a_serialised_threadpool_fails_the_barrier_check():
    """Control for the test above: prove the barrier detects serialisation.

    "The requests interleaved" means nothing unless a non-interleaving app
    would have been caught. Squeezing the AnyIO limiter to a single token
    reproduces exactly the failure mode the previous test rules out — one
    sync handler at a time — and the same barrier then never trips.
    """
    parties = 3
    app = _barrier_app(parties, NO_TRIP_TIMEOUT)

    with _threadpool_limit(1) as limiter:
        assert limiter.total_tokens == 1, "limiter did not accept the resize"
        async with _drive(app) as client:
            responses = await asyncio.gather(
                *[client.get("/gated") for _ in range(parties)]
            )

    payloads = [r.json() for r in responses]
    assert not any(p["tripped"] for p in payloads), (
        "with a one-token threadpool the handlers CANNOT overlap, so the "
        "barrier must never trip. It did, which means the barrier is not "
        "actually measuring concurrency and the sibling test proves nothing."
    )
    assert len({p["thread"] for p in payloads}) == 1, (
        "a one-token limiter must funnel every sync request through a single "
        f"worker thread; saw {len({p['thread'] for p in payloads})}"
    )


@pytest.mark.asyncio
async def test_concurrent_async_handlers_run_simultaneously():
    """``async def`` routes overlap on the single loop.

    Separate property from the sync case and a separate mechanism: these
    never touch the threadpool, so the limiter cannot help them and cannot
    hurt them. ``asyncio.Barrier`` needs all N coroutines suspended in it
    simultaneously, which is only reachable if the loop is interleaving them.
    """
    parties = 6
    app = FastAPI()
    barrier = asyncio.Barrier(parties)

    @app.get("/gated")
    async def gated():
        await asyncio.wait_for(barrier.wait(), timeout=TRIP_TIMEOUT)
        return {"ok": True}

    async with _drive(app) as client:
        responses = await asyncio.gather(
            *[client.get("/gated") for _ in range(parties)]
        )

    assert [r.status_code for r in responses] == [200] * parties, (
        f"{parties} concurrent async requests did not all reach the barrier; "
        "the loop is not interleaving them"
    )


# ---------------------------------------------------------------------------
# 2. A blocking sync handler must not stall the event loop
# ---------------------------------------------------------------------------


@pytest.mark.asyncio
async def test_a_blocking_sync_handler_does_not_stall_the_event_loop():
    """The threadpool offload is what keeps one slow route from freezing all.

    Fully deterministic — no sleeps. ``/slow-sync`` parks on a
    ``threading.Event`` that only the test can set, so it is provably still
    executing while ``/fast-async`` is served. The completion order is then
    a fact, not a race: ``fast`` is recorded before ``slow`` because ``slow``
    is physically unable to finish until the test releases it.
    """
    app = FastAPI()
    completions: list[str] = []
    entered = asyncio.Event()
    release = threading.Event()
    loop = asyncio.get_running_loop()

    @app.get("/slow-sync")
    def slow_sync():
        # Signal from the worker thread back onto the loop. Reaching the
        # loop at all already demonstrates the loop is not the thing blocked.
        loop.call_soon_threadsafe(entered.set)
        released = release.wait(timeout=TRIP_TIMEOUT)
        completions.append("slow")
        return {"released": released}

    @app.get("/fast-async")
    async def fast_async():
        completions.append("fast")
        return {"ok": True}

    async with _drive(app) as client:
        slow = asyncio.create_task(client.get("/slow-sync"))
        await asyncio.wait_for(entered.wait(), timeout=TRIP_TIMEOUT)

        fast_response = await asyncio.wait_for(
            client.get("/fast-async"), timeout=TRIP_TIMEOUT
        )
        assert fast_response.status_code == 200
        assert completions == ["fast"], (
            "the fast async route answered, but the blocking sync route had "
            f"already completed too — completions={completions}. The test is "
            "no longer proving overlap."
        )

        release.set()
        slow_response = await asyncio.wait_for(slow, timeout=TRIP_TIMEOUT)

    assert slow_response.status_code == 200
    assert slow_response.json()["released"] is True, (
        "the blocking handler timed out instead of being released; it was "
        "not actually parked on the event the test controls"
    )
    assert completions == ["fast", "slow"], (
        "a fast async route must be served while a sync route is blocked. "
        f"Got {completions}: the sync handler is running ON the event loop "
        "instead of in the AnyIO threadpool, so one slow request freezes "
        "every other request, every Socket.IO event and the healthcheck."
    )


@pytest.mark.asyncio
async def test_blocking_the_loop_really_does_reverse_that_order():
    """Control: the ordering assertion above has teeth.

    ``completions == ["fast", "slow"]`` is only evidence if the broken
    arrangement produces something else. Here the slow route is ``async
    def`` and blocks the loop directly — the exact mistake the offload
    protects against — and the fast route cannot be served until it
    finishes, inverting the order.
    """
    app = FastAPI()
    completions: list[str] = []
    started = threading.Event()

    @app.get("/blocking-async")
    async def blocking_async():
        started.set()
        # Real wall-clock blocking is the subject: it must happen on the
        # loop for the control to mean anything, so it cannot be faked or
        # frozen. Duration is irrelevant to the assertion (the loop simply
        # cannot run anything during it), so this is the smallest value
        # that is unambiguous.
        time.sleep(
            0.3
        )  # allow: unmarked-sleep — stalling the loop IS the control
        completions.append("blocked")
        return {"ok": True}

    @app.get("/fast-async")
    async def fast_async():
        completions.append("fast")
        return {"ok": True}

    async with _drive(app) as client:
        blocking = asyncio.create_task(client.get("/blocking-async"))
        await _yield_until(started.is_set)
        await asyncio.wait_for(client.get("/fast-async"), timeout=TRIP_TIMEOUT)
        await asyncio.wait_for(blocking, timeout=TRIP_TIMEOUT)

    assert completions == ["blocked", "fast"], (
        "an async handler that blocks the loop must delay every other "
        f"request until it returns. Got {completions} — if the fast route "
        "came first, the ordering assertion in the sibling test is not "
        "sensitive to a stalled loop and proves nothing."
    )


# ---------------------------------------------------------------------------
# 3. Threadpool exhaustion behaves sanely
# ---------------------------------------------------------------------------


@pytest.mark.asyncio
async def test_saturated_threadpool_queues_requests_instead_of_failing():
    """Past the ceiling, requests wait and then succeed.

    Two properties at once: the limiter is a real ceiling (concurrency never
    exceeds it, so a raised value cannot silently pin more DB connections
    than the operator sanctioned), and overflow is *queued* rather than
    dropped or 500'd.
    """
    limit = 4
    overflow = 3
    total = limit + overflow

    app = FastAPI()
    barrier = threading.Barrier(limit, timeout=TRIP_TIMEOUT)
    release = threading.Event()
    arrivals = itertools.count()
    lock = threading.Lock()
    state = {"live": 0, "peak": 0}

    @app.get("/hold")
    def hold():
        index = next(arrivals)
        with lock:
            state["live"] += 1
            state["peak"] = max(state["peak"], state["live"])
        # Only the first `limit` callers meet at the barrier — that is what
        # proves the pool really runs `limit` handlers at once. Later
        # arrivals skip it (the barrier has already been consumed) and just
        # hold their slot, which is what makes the queueing observable.
        if index < limit:
            barrier.wait()
        release.wait(timeout=TRIP_TIMEOUT)
        with lock:
            state["live"] -= 1
        return {"index": index}

    with _threadpool_limit(limit):
        async with _drive(app) as client:
            tasks = [
                asyncio.create_task(client.get("/hold")) for _ in range(total)
            ]
            await _yield_until(lambda: state["peak"] >= limit)

            queued = [t for t in tasks if not t.done()]
            assert len(queued) == total, (
                "no request may complete while the pool is saturated and "
                f"held; {total - len(queued)} already finished"
            )

            release.set()
            responses = await asyncio.gather(*tasks)

    assert [r.status_code for r in responses] == [200] * total, (
        "requests beyond the threadpool ceiling must queue and then succeed, "
        f"not fail: {[r.status_code for r in responses]}"
    )
    assert sorted(r.json()["index"] for r in responses) == list(range(total)), (
        "every queued request must eventually enter the handler"
    )
    assert state["peak"] == limit, (
        f"the AnyIO limiter was set to {limit} but {state['peak']} sync "
        "handlers ran at once. The ceiling is not being enforced, so "
        "LDR_WEB_THREADPOOL_MAX_THREADS does not bound how many worker "
        "threads can each pin a per-user DB connection."
    )


@pytest.mark.asyncio
async def test_async_routes_still_answer_while_the_threadpool_is_saturated():
    """A jammed threadpool must not take the async routes down with it.

    The lifespan comment warns the pool is *shared* with async dependency
    solving and response validation, so this is not obvious: it is the
    difference between "sync routes are slow right now" and "/api/v1/health
    times out and the container is marked unhealthy".
    """
    limit = 3
    app = FastAPI()
    release = threading.Event()
    lock = threading.Lock()
    state = {"live": 0}

    @app.get("/hold")
    def hold():
        with lock:
            state["live"] += 1
        release.wait(timeout=TRIP_TIMEOUT)
        with lock:
            state["live"] -= 1
        return {"ok": True}

    @app.get("/ping")
    async def ping():
        return {"pong": True}

    with _threadpool_limit(limit):
        async with _drive(app) as client:
            held = [
                asyncio.create_task(client.get("/hold")) for _ in range(limit)
            ]
            # Every worker occupied and parked: saturation is a counted fact,
            # not an inference from timing.
            await _yield_until(lambda: state["live"] == limit)

            ping_response = await asyncio.wait_for(
                client.get("/ping"), timeout=TRIP_TIMEOUT
            )
            assert not any(t.done() for t in held), (
                "the holding requests released early; the pool was not "
                "actually saturated when /ping was served"
            )

            release.set()
            held_responses = await asyncio.gather(*held)

    assert ping_response.status_code == 200, (
        "an async route must keep answering with every threadpool worker "
        f"occupied; got {ping_response.status_code}"
    )
    assert ping_response.json() == {"pong": True}
    assert [r.status_code for r in held_responses] == [200] * limit


@pytest.mark.parametrize(
    ("worker_threads", "expected"),
    [
        (1, False),
        (40, False),  # AnyIO's own default — must never warn
        (DB_POOL_CAPACITY - 1, False),
        (DB_POOL_CAPACITY, False),  # boundary: at capacity is still fine
        (DB_POOL_CAPACITY + 1, True),  # boundary: one over warns
        (1000, True),
    ],
)
def test_threadpool_vs_db_pool_reconciliation_at_the_boundary(
    worker_threads, expected
):
    """``warn_if_threadpool_exceeds_db_pool`` fires strictly above capacity.

    The boundary is the whole point. A sync route's DB session lives in a
    ``threading.local()`` on the AnyIO worker that served it, while
    ``DatabaseMiddleware``'s cleanup runs ``async def`` on the loop thread —
    so it cannot reach that worker, and checked-out connections track the
    *worker count*, not the request count. Sizing the pool at or below
    ``POOL_SIZE + MAX_OVERFLOW`` keeps that harmless; above it, one user's
    own concurrency exhausts their own pool and every later request of
    theirs waits out ``pool_timeout``.
    """
    assert warn_if_threadpool_exceeds_db_pool(worker_threads) is expected, (
        f"{worker_threads} worker threads against a DB pool capacity of "
        f"{DB_POOL_CAPACITY} (POOL_SIZE {POOL_SIZE} + MAX_OVERFLOW "
        f"{MAX_OVERFLOW}) should{'' if expected else ' not'} warn"
    )


def test_the_reconciliation_warning_is_actually_emitted(loguru_caplog):
    """The return value is for tests; the *log line* is for the operator.

    A refactor that kept the boolean and dropped the ``logger.warning``
    would leave every parametrised case above green while the misconfigured
    operator learned nothing. Uses the shared ``loguru_caplog`` fixture
    because ``local_deep_research/__init__.py`` calls
    ``logger.disable("local_deep_research")`` — without re-enabling, a sink
    added here receives nothing and this test would pass vacuously against a
    silent function.
    """
    with loguru_caplog.at_level("WARNING"):
        warned = warn_if_threadpool_exceeds_db_pool(DB_POOL_CAPACITY + 1)
        below = warn_if_threadpool_exceeds_db_pool(DB_POOL_CAPACITY)

    assert warned is True and below is False
    # Filtered rather than counted outright: background threads in this
    # process may log their own warnings while these two calls run.
    relevant = [
        record.getMessage()
        for record in loguru_caplog.records
        if "web.threadpool_max_threads" in record.getMessage()
    ]
    assert len(relevant) == 1, (
        "exactly one WARNING naming web.threadpool_max_threads should have "
        "been emitted — for the over-capacity call only. Captured "
        f"{len(relevant)}: {relevant}. A refactor that kept the boolean and "
        "dropped the logger.warning leaves the operator with no signal."
    )
    assert str(DB_POOL_CAPACITY) in relevant[0], (
        "the warning must tell the operator what the ceiling is; got: "
        f"{relevant[0]!r}"
    )


@pytest.mark.asyncio
async def test_run_db_sync_does_not_use_the_pool_the_operator_can_size():
    """FINDING: ``LDR_WEB_THREADPOOL_MAX_THREADS`` does not bound
    ``run_db_sync`` / ``asyncio.to_thread``.

    The lifespan resizes AnyIO's default thread limiter, which governs
    Starlette's ``def``-handler offload. ``run_db_sync`` is built on
    ``asyncio.to_thread``, which uses the *event loop's default
    ThreadPoolExecutor* — a different pool, with its own
    ``min(32, cpu_count + 4)`` ceiling, that neither the knob nor
    ``warn_if_threadpool_exceeds_db_pool`` knows about. There are ~100
    ``run_db_sync`` call sites in the web layer, plus
    ``DatabaseMiddleware``'s own ``ensure_user_database`` offload, and every
    one of them opens a per-user DB session.

    Consequence: the real number of worker threads that can each pin a DB
    connection is (AnyIO limit + default-executor size), so the
    reconciliation against ``POOL_SIZE + MAX_OVERFLOW`` can report "fine"
    while the effective total is well above capacity — and the second pool
    is the one an operator has no supported way to shrink.

    This test pins the fact, so the gap cannot be closed silently or
    forgotten. It asserts nothing about which behaviour is *correct*.
    """
    parties = 4
    barrier = threading.Barrier(parties, timeout=TRIP_TIMEOUT)

    def _rendezvous():
        try:
            barrier.wait()
        except threading.BrokenBarrierError:
            # Reported as a value so the assertion below explains it, rather
            # than surfacing as a raised exception from gather().
            return None
        return threading.get_ident()

    with _threadpool_limit(1) as limiter:
        assert limiter.total_tokens == 1
        threads = await asyncio.gather(
            *[run_db_sync(_rendezvous) for _ in range(parties)]
        )

    assert None not in threads, (
        "run_db_sync started honouring the AnyIO limiter — with one token "
        f"only one of {parties} tasks could run, so the barrier expired. "
        "That would be an improvement, but this test and the DB pool "
        "reconciliation (which counts only the AnyIO pool) must be updated "
        "together with it."
    )
    assert len(set(threads)) == parties, (
        f"expected {parties} distinct threads for {parties} concurrent "
        f"run_db_sync calls, saw {len(set(threads))}"
    )


# ---------------------------------------------------------------------------
# 4. Per-request state must not leak between concurrent requests
# ---------------------------------------------------------------------------


@pytest.fixture
def registered_users():
    """Real server-side sessions for four users.

    ``DatabaseMiddleware`` runs ``_enforce_session_revocation``, which clears
    any session whose ``session_id`` does not resolve to the claimed
    username. ``session_manager`` is an in-memory dict, so registering here
    is cheap and keeps the middleware on its real code path rather than a
    patched one.
    """
    from local_deep_research.web.auth.session_manager import session_manager

    users = ["alice", "bob", "carol", "dave"]
    sessions = {u: session_manager.create_session(u) for u in users}
    yield sessions
    for session_id in sessions.values():
        session_manager.destroy_session(session_id)


class _SessionInjector:
    """Stands in for ``SessionMiddleware``, which needs a signed cookie.

    ``DatabaseMiddleware``'s entire contract with the layer above it is
    ``scope["session"]`` being a dict, so populating it from a header
    exercises the real middleware without a login round trip. Ordering
    matches production: session outside, database inside.
    """

    def __init__(self, app, sessions):
        self.app = app
        self.sessions = sessions

    async def __call__(self, scope, receive, send):
        if scope["type"] == "http":
            headers = dict(scope["headers"])
            username = headers.get(b"x-test-user", b"").decode()
            scope["session"] = (
                {"username": username, "session_id": self.sessions[username]}
                if username
                else {}
            )
        await self.app(scope, receive, send)


def _identity_app(sessions, *, is_async: bool, parties: int):
    """Route that reports who the request-scoped context thinks it is."""
    app = FastAPI()
    sync_barrier = threading.Barrier(parties, timeout=TRIP_TIMEOUT)
    async_barrier = asyncio.Barrier(parties)

    @app.get("/whoami-sync")
    def whoami_sync(request: Request):
        claimed = request.headers["x-test-user"]
        # Overlap first, read identity second: a leak only becomes visible
        # once several requests are inside the handler at the same time.
        sync_barrier.wait()
        return {
            "claimed": claimed,
            "context_username": get_current_username(),
            "context_session_id": get_current_session_id(),
        }

    @app.get("/whoami-async")
    async def whoami_async(request: Request):
        claimed = request.headers["x-test-user"]
        await asyncio.wait_for(async_barrier.wait(), timeout=TRIP_TIMEOUT)
        return {
            "claimed": claimed,
            "context_username": get_current_username(),
            "context_session_id": get_current_session_id(),
        }

    path = "/whoami-async" if is_async else "/whoami-sync"
    return _SessionInjector(DatabaseMiddleware(app), sessions), path


async def _collect_identities(sessions, *, is_async: bool):
    stack, path = _identity_app(
        sessions, is_async=is_async, parties=len(sessions)
    )
    # ensure_user_database opens a SQLCipher database; irrelevant to identity
    # propagation and far too heavy for a concurrency test. Everything else in
    # DatabaseMiddleware — expiry, revocation, set/reset_request_user, the
    # cleanup in `finally` — runs for real.
    with patch(
        "local_deep_research.web.dependencies.auth.ensure_user_database",
        lambda request: None,
    ):
        async with _drive(stack) as client:
            responses = await asyncio.gather(
                *[
                    client.get(path, headers={"x-test-user": user})
                    for user in sessions
                ]
            )
    assert [r.status_code for r in responses] == [200] * len(sessions), (
        f"requests did not all succeed: {[r.status_code for r in responses]}"
    )
    return [r.json() for r in responses]


@pytest.mark.asyncio
@pytest.mark.parametrize("is_async", [False, True], ids=["sync", "async"])
async def test_request_identity_does_not_leak_between_concurrent_users(
    registered_users, is_async
):
    """Four different users, all inside the handler simultaneously.

    ``DatabaseMiddleware`` publishes username and session_id into a
    ``contextvars.ContextVar``. Under one shared event loop that is the only
    thing keeping the identities apart — and for the sync case the value has
    to survive AnyIO's hop onto a *pooled* worker thread as well, which is a
    genuinely different mechanism (context propagation into the thread)
    rather than the same test twice.

    A failure here is a cross-user disclosure: service-layer code calls
    ``get_current_username()`` to decide whose database to open.
    """
    payloads = await _collect_identities(registered_users, is_async=is_async)

    mismatches = [
        f"request claiming {p['claimed']!r} saw {p['context_username']!r}"
        for p in payloads
        if p["context_username"] != p["claimed"]
    ]
    assert not mismatches, (
        "the request-scoped username contextvar leaked between concurrent "
        "requests for different users — service code reads it to choose "
        "which user's database to open:\n  " + "\n  ".join(mismatches)
    )

    session_mismatches = [
        p["claimed"]
        for p in payloads
        if p["context_session_id"] != registered_users[p["claimed"]]
    ]
    assert not session_mismatches, (
        "the request-scoped session_id contextvar leaked between concurrent "
        f"requests for users: {session_mismatches}"
    )
    assert len(payloads) == len(registered_users) >= 2, (
        "premise guard: the check above is vacuous without several "
        "overlapping users"
    )


@pytest.mark.asyncio
async def test_the_identity_harness_would_catch_a_leak(registered_users):
    """Control: swap the contextvar for process-global storage.

    ``context_username == claimed`` for every request is also what a test
    that never really overlapped would produce. This runs the *same* drive
    against a middleware identical to ``DatabaseMiddleware`` except that it
    stores the username in a module-level global — the classic wrong answer,
    and what a ``threading.local()`` would degrade to on a pooled worker.
    With four users overlapping at a barrier, at least one request must read
    somebody else's name.
    """
    users = list(registered_users)
    parties = len(users)
    leaked = {"username": None}

    class _LeakyMiddleware:
        def __init__(self, app):
            self.app = app

        async def __call__(self, scope, receive, send):
            if scope["type"] == "http":
                leaked["username"] = scope.get("session", {}).get("username")
            await self.app(scope, receive, send)

    app = FastAPI()
    barrier = threading.Barrier(parties, timeout=TRIP_TIMEOUT)

    @app.get("/whoami-sync")
    def whoami_sync(request: Request):
        claimed = request.headers["x-test-user"]
        barrier.wait()
        return {"claimed": claimed, "context_username": leaked["username"]}

    stack = _SessionInjector(_LeakyMiddleware(app), registered_users)
    async with _drive(stack) as client:
        responses = await asyncio.gather(
            *[
                client.get("/whoami-sync", headers={"x-test-user": user})
                for user in users
            ]
        )

    payloads = [r.json() for r in responses]
    mismatches = [p for p in payloads if p["context_username"] != p["claimed"]]
    assert mismatches, (
        "globally-stored per-request identity did NOT cross-contaminate "
        f"across {parties} overlapping users ({payloads}). The harness is "
        "not exercising real concurrency, so the sibling test's clean result "
        "is not evidence that the contextvar isolates anything."
    )


@pytest.mark.asyncio
async def test_settings_context_leaks_on_a_pooled_worker_unless_cleaned():
    """Why ``run_db_sync`` exists, demonstrated on one pinned thread.

    ``config.thread_settings`` keeps the settings context in a bare
    ``threading.local()``. Worker threads are POOLED and reused across users,
    so a task that sets a context and returns leaves it attached to the
    thread for whoever lands there next. ``run_db_sync``'s ``finally`` calls
    ``clear_settings_context()`` on the worker — the only place it can be
    done, since ``DatabaseMiddleware``'s cleanup runs on the loop thread.

    Forcing a single-worker executor makes "the next task on the same thread"
    deterministic instead of a race: both halves below provably share a
    thread. The ``asyncio.to_thread`` half is the positive control — it shows
    the leak is real, so the ``run_db_sync`` half is evidence of a fix rather
    than of nothing having happened.
    """
    marker = {"settings": "user-a-secret"}
    loop = asyncio.get_running_loop()
    single = ThreadPoolExecutor(max_workers=1, thread_name_prefix="pinned")
    previous_executor = getattr(loop, "_default_executor", None)
    loop.set_default_executor(single)
    try:

        def _set_context():
            set_settings_context(marker)
            return threading.get_ident()

        def _read_context():
            return threading.get_ident(), get_settings_context()

        raw_setter_thread = await asyncio.to_thread(_set_context)
        raw_reader_thread, raw_seen = await asyncio.to_thread(_read_context)

        # Clear before the second half so the two are independent.
        await asyncio.to_thread(clear_settings_context)

        safe_setter_thread = await run_db_sync(_set_context)
        safe_reader_thread, safe_seen = await run_db_sync(_read_context)
    finally:
        loop.set_default_executor(previous_executor or ThreadPoolExecutor())
        single.shutdown(wait=True)

    assert raw_setter_thread == raw_reader_thread == safe_setter_thread, (
        "the single-worker executor did not pin every task to one thread, so "
        "neither half below is testing reuse"
    )
    assert safe_reader_thread == safe_setter_thread
    assert raw_seen == marker, (
        "positive control: a bare asyncio.to_thread task MUST leave its "
        "settings context on the pooled worker for the next task to find. It "
        f"did not (saw {raw_seen!r}), so the run_db_sync assertion below "
        "proves nothing."
    )
    assert safe_seen is None, (
        "run_db_sync must clear the thread-local settings context on the "
        f"worker before returning; the next task on that thread saw "
        f"{safe_seen!r}. On a pooled worker that is one user's settings "
        "snapshot handed to the next user's request."
    )


def test_a_cached_db_session_is_never_handed_to_a_different_user():
    """The last line of defence for the pooled-worker DB session.

    ``DatabaseMiddleware`` cannot clean up a worker thread's session (its
    ``finally`` runs on the loop thread), so a session opened by one user's
    sync request stays cached on that worker. The guard inside
    ``ThreadLocalSessionManager.get_session`` is what stops the next request
    on that worker — possibly a different user — from being handed it.

    Both directions asserted: same user reuses the cache (otherwise the
    guard could be "always discard", which would pass a one-sided test while
    destroying the caching this design depends on).
    """
    from local_deep_research.database import thread_local_session as tls

    manager = tls.ThreadLocalSessionManager()
    cached = MagicMock(name="alice_session")
    manager._local.session = cached
    manager._local.username = "alice"

    same_user = manager.get_session("alice", "pw")
    assert same_user is cached, (
        "the same user on the same worker thread must reuse the cached "
        "session; discarding it would reopen SQLCipher on every request"
    )

    with patch.object(
        tls.db_manager, "open_user_database", return_value=None
    ) as opener:
        other_user = manager.get_session("bob", "pw")

    assert other_user is not cached, (
        "SECURITY: a worker thread's cached session was handed to a "
        "different user. Worker threads are pooled and serve every user in "
        "turn under workers=1, so this is a cross-user database handle."
    )
    assert other_user is None, (
        "with open_user_database stubbed out there is no session to hand "
        f"back; got {other_user!r}"
    )
    assert opener.call_args.args[0] == "bob", (
        "the guard must go on to open the REQUESTING user's database, not "
        f"reuse the cached user's; opened for {opener.call_args.args[0]!r}"
    )
    assert cached.close.called, (
        "the stale cross-user session must be closed, not merely dropped — "
        "otherwise its DB connection never returns to the pool"
    )
    assert manager._local.username is None, (
        "the thread-local user must be cleared alongside the session"
    )


# ---------------------------------------------------------------------------
# 5. Nothing bounds concurrency except the AnyIO limiter
# ---------------------------------------------------------------------------


def _uvicorn_run_kwargs() -> dict[str, str]:
    """Keyword arguments of the single ``uvicorn.run(...)`` call."""
    source = Path(web_app_module.__file__).resolve().read_text(encoding="utf-8")
    tree = ast.parse(source)
    calls = [
        node
        for node in ast.walk(tree)
        if isinstance(node, ast.Call)
        and ast.unparse(node.func) == "uvicorn.run"
    ]
    assert len(calls) == 1, (
        "premise guard: expected exactly one uvicorn.run call in web/app.py, "
        f"found {len(calls)} — this scan is looking at the wrong thing"
    )
    return {
        kw.arg: ast.unparse(kw.value)
        for kw in calls[0].keywords
        if kw.arg is not None
    }


def test_uvicorn_runs_one_worker_with_no_concurrency_limit():
    """Documents the ceiling situation, and fails if anyone changes it.

    ``workers=1`` is mandatory (Socket.IO without a Redis message queue), so
    the whole instance is one loop. ``uvicorn.run`` sets no
    ``limit_concurrency`` and no ``backlog``, which means:

    * accepted connections are unbounded — uvicorn will not shed load, it
      will queue it, and the queue is only bounded by memory;
    * the ONLY ceiling in the process is the AnyIO thread limiter, and that
      applies to ``def`` routes alone (see the sibling test: async routes
      have no ceiling at all);
    * ``timeout_keep_alive=5`` is what keeps idle connections from
      accumulating, so it is doing more work here than it looks.

    If a ``limit_concurrency`` is ever added it must be at least the AnyIO
    limit, or uvicorn will reject requests while worker threads sit idle —
    hence the failure here, to force that decision to be made explicitly.
    """
    kwargs = _uvicorn_run_kwargs()

    assert kwargs.get("workers") == "1", (
        "Socket.IO without a Redis message queue requires a single worker; "
        f"got workers={kwargs.get('workers')!r}. Everything these tests "
        "assert about a shared loop and a shared threadpool assumes it."
    )
    assert "limit_concurrency" not in kwargs, (
        "a limit_concurrency was added to uvicorn.run. That is a real "
        "ceiling on accepted requests and it must be reconciled with the "
        "AnyIO thread limiter (LDR_WEB_THREADPOOL_MAX_THREADS) and the "
        "per-user DB pool capacity of "
        f"{DB_POOL_CAPACITY}. Update this test with the reasoning."
    )
    assert kwargs.get("timeout_keep_alive") == "5", (
        "with no limit_concurrency, timeout_keep_alive is the only thing "
        "bounding accumulated idle connections; it must stay set"
    )


@pytest.mark.asyncio
async def test_async_routes_have_no_concurrency_ceiling_at_all():
    """The threadpool knob does not bound async routes — by a wide margin.

    Worth pinning because the opposite is a natural assumption: the lifespan
    comment notes the AnyIO pool is shared with async dependency solving and
    response validation, which makes it sound like a global ceiling. It is
    not one. With the limiter at 2, forty concurrent async requests still all
    reach a forty-party barrier, so async concurrency is bounded only by
    memory and by whatever the operator puts in front of the process.
    """
    parties = 40
    app = FastAPI()
    barrier = asyncio.Barrier(parties)

    @app.get("/gated")
    async def gated():
        await asyncio.wait_for(barrier.wait(), timeout=TRIP_TIMEOUT)
        return {"ok": True}

    with _threadpool_limit(2):
        async with _drive(app) as client:
            responses = await asyncio.gather(
                *[client.get("/gated") for _ in range(parties)]
            )

    assert [r.status_code for r in responses] == [200] * parties, (
        f"only some of {parties} concurrent async requests completed with a "
        "2-token threadpool. If the AnyIO limiter has started bounding async "
        "routes too, it is now the app's global concurrency ceiling and the "
        "DB-pool reconciliation needs revisiting."
    )