Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

45.0 KB
Raw Blame
"""
Follow-up LONG, multi-step integration flows for the Flask -> FastAPI
migration branch. Companion to ``test_long_integration_flows.py`` — read
that file's module docstring first; this one does not repeat its survey and
does not re-cover anything it already pins.

SURVEY — what test_long_integration_flows.py already covers
--------------------------------------------------------------
Full register -> login -> settings -> collection -> upload -> logout ->
re-login -> cross-user-invisibility lifecycle; session lifecycle including
replaying a cookie captured before logout; two concurrent users interleaved
(sequential deterministic + one genuinely-concurrent slice via
``threading.Barrier``); research submission -> ResearchHistory row ->
cross-user isolation of that row; a setting written via ``/settings/api``
observed via a DIFFERENT router (``/research/api/settings/current-config``).
Its own docstring additionally surveys ``test_state_changing_flows.py``,
``test_authenticated_flows.py``, ``test_session_cookie_behavior.py``,
``test_cross_user_isolation_invariants.py`` and
``test_collection_upload_http.py`` — none of those are re-covered here
either.

NEW GROUND COVERED IN THIS FILE
--------------------------------
1. ``TestPasswordChangeLifecycle`` — change password mid-session over REAL
   HTTP with the REAL (unmocked) SQLCipher rekey, then verifies five
   different subsystems all agree the old password is dead: the submitting
   session, a SECOND concurrent session for the same user, the encrypted DB
   itself (old password can no longer open it, new password can), a
   simulated pooled-worker thread-credential cache entry, and that data
   created before the change is still readable after logging back in with
   the new password. ``tests/web/routers/test_auth_flow_gaps.py`` already
   pins the router's session-cleanup CONTRACT for this flow but explicitly
   mocks ``db_manager.change_password`` and the backup service at the
   boundary ("The SQLCipher rekey itself ... and the backup refresh are
   mocked at their boundary"); this file is the one that drives the REAL
   rekey and REAL data-survives-relogin path end to end.
2. ``TestQueueLifecycleAcrossUsers`` — submits enough research to exceed a
   user's ``app.max_concurrent_researches`` cap so rows land QUEUED, checks
   two independent read paths, and proves a second user's cap/queue is
   untouched. Also documents (see class docstring) a REAL BUG found while
   building this: the queue processor's dispatch-time slot accounting is
   blind to research started via the ordinary direct (non-queued) path.
3. ``TestCollectionDocumentCascadeDeletion`` — collection -> document
   upload -> collection delete, then verifies via the REAL per-user
   encrypted DB (not just the HTTP surface) that Collection/Document/
   DocumentCollection rows are gone, no plaintext trace of the document's
   content survives anywhere on disk under ``LDR_DATA_DIR``, and a second
   user's identically-NAMED collection is untouched by the first user's
   deletion.
4. ``TestSettingsChangeBehaviorAcrossRouters`` — writes ``policy.egress_scope``
   through the settings router and proves the research router's
   request-boundary egress precheck (a completely different file,
   ``web/routers/research.py``) refuses a request on the very next call
   that it accepted before the write.
5. ``TestRecoveryAfterSimulatedRestart`` — builds real state for two users,
   drops every in-memory singleton a process restart would wipe (session
   manager, both password stores, the thread-credential cache, the auth-DB
   engine), and verifies both users are forced to re-authenticate and, once
   they do, see their own persisted data and never each other's.

RULES FOLLOWED (see test_long_integration_flows.py for the fuller
rationale on the same points):
* Real HTTP via ``TestClient``, real CSRF, real per-user encrypted DBs.
* Every test takes the ``app`` fixture from ``tests/conftest.py`` (fresh
  ``LDR_DATA_DIR`` per test) — never a bare
  ``from local_deep_research.web.fastapi_app import app`` import.
* No bare ``sleep``. No test here needs one: every real background research
  thread spawned below uses a nonexistent LLM provider name so it fails
  fast and in-process (see ``_UNREACHABLE_PROVIDER`` below) and every
  assertion that depends on a DB write happening before spawn is backed by
  reading the source (the write is committed synchronously before the
  thread starts, mirrored from the sibling file's documented pattern).
"""

import threading
from unittest.mock import patch

from local_deep_research.web.queue.processor_v2 import (
    QueueProcessorV2,
)
import uuid

from fastapi.testclient import TestClient

from local_deep_research.constants import ResearchStatus

TEST_PASSWORD = "LongFlowPass123!"  # noqa: S105
_UNREACHABLE_PROVIDER = "no_such_llm_provider_xyz"
_ACCEPTED_ENGINE = "searxng"


# ---------------------------------------------------------------------------
# Shared helpers — identical in spirit to test_long_integration_flows.py's
# (kept local rather than imported so this file has no import-order
# dependency on its sibling).
# ---------------------------------------------------------------------------


def _new_client(app) -> TestClient:
    client = TestClient(app, raise_server_exceptions=False)
    fwd_ip = f"10.{uuid.uuid4().int % 254 + 1}.{uuid.uuid4().int % 254 + 1}.1"
    client.headers.update({"X-Forwarded-For": fwd_ip})
    return client


def _csrf(client: TestClient) -> str:
    client.get("/auth/login")
    resp = client.get("/auth/csrf-token")
    return resp.json().get("csrf_token", "") if resp.status_code == 200 else ""


def _drop_stale_csrf_header(client: TestClient) -> None:
    """See test_long_integration_flows.py::_drop_stale_csrf_header for the
    full rationale: a persistent X-CSRFToken default header shadows a fresh
    csrf_token form field once the session rotates."""
    for name in ("X-CSRFToken", "X-CSRF-Token"):
        client.headers.pop(name, None)


def _register(client: TestClient, username: str, password: str = TEST_PASSWORD):
    _drop_stale_csrf_header(client)
    return client.post(
        "/auth/register",
        data={
            "username": username,
            "password": password,
            "confirm_password": password,
            "acknowledge": "true",
            "csrf_token": _csrf(client),
        },
        follow_redirects=False,
    )


def _login(
    client: TestClient,
    username: str,
    password: str = TEST_PASSWORD,
    remember: str = "false",
):
    _drop_stale_csrf_header(client)
    return client.post(
        "/auth/login",
        data={
            "username": username,
            "password": password,
            "remember": remember,
            "csrf_token": _csrf(client),
        },
        follow_redirects=False,
    )


def _attach_csrf(client: TestClient) -> None:
    resp = client.get("/auth/csrf-token")
    if resp.status_code == 200:
        token = resp.json().get("csrf_token")
        if token:
            client.headers.update({"X-CSRFToken": token})


def _register_and_login(
    client: TestClient, username: str, password: str = TEST_PASSWORD
) -> None:
    r = _register(client, username, password)
    assert r.status_code == 302, (
        f"register failed for {username!r}: {r.status_code} {r.text[:300]}"
    )
    r = _login(client, username, password)
    assert r.status_code == 302, (
        f"login failed for {username!r}: {r.status_code} {r.text[:300]}"
    )
    _attach_csrf(client)


def _unique_name(prefix: str) -> str:
    return f"{prefix}_{uuid.uuid4().hex[:10]}"


def _submit_research(
    client: TestClient, query: str, engine: str = _ACCEPTED_ENGINE
):
    """POST /api/start_research with a deliberately-unreachable LLM
    provider (fails fast, in-process, no network — see module docstring)
    and a real, policy-accepted search engine name."""
    return client.post(
        "/api/start_research",
        json={
            "query": query,
            "mode": "quick",
            "model_provider": _UNREACHABLE_PROVIDER,
            "model": "no-such-model-xyz",
            "search_engine": engine,
            "iterations": 1,
            "questions_per_iteration": 1,
        },
    )


# ===========================================================================
# 1. Password change as a lifecycle event
# ===========================================================================


def _seed_worker_credential(username: str, password: str):
    """Register a (username, password) entry on a *different* thread, as a
    pooled AnyIO worker would after serving an authenticated request for
    this user, and keep that thread alive so cleanup_dead_threads() cannot
    mask the behaviour under test by reaping it first.

    Mirrors tests/security/test_logout_clears_thread_credentials.py's
    ``_seed_credentials_on_worker`` helper, wired to a REAL end-to-end HTTP
    change-password call instead of a direct ``clear_user_credentials()``
    call — this is the "between files" link: does the router's
    change-password handler actually reach into
    ``database/thread_local_session.py`` and evict a live worker's cached
    plaintext password, or does it only clear the session-scoped store?

    Returns (thread_id, release_event); caller MUST set the event when done.
    """
    from local_deep_research.database.thread_local_session import (
        thread_session_manager,
    )

    holder = {}
    ready = threading.Event()
    release = threading.Event()

    def worker():
        holder["id"] = threading.get_ident()
        with thread_session_manager._lock:
            thread_session_manager._thread_credentials[holder["id"]] = (
                username,
                password,
            )
        ready.set()
        release.wait(timeout=10)

    t = threading.Thread(target=worker, daemon=True)
    t.start()
    assert ready.wait(timeout=5), "seed worker did not register in time"
    return holder["id"], release


class TestPasswordChangeLifecycle:
    """Change password mid-session; verify every subsystem that cached the
    old credential agrees it is dead, and that data survives into the new
    key.

    Crosses: auth router (change-password handler) -> session_manager ->
    session_passwords store -> thread_local_session credential cache ->
    encrypted_db (the actual SQLCipher rekey) -> library/rag router (data
    written under the OLD password, read back under the NEW one).
    """

    def test_change_password_rekeys_db_kills_other_sessions_and_preserves_data(
        self, app
    ):
        username = _unique_name("pwchange_user")
        old_password = TEST_PASSWORD
        new_password = "NewLongFlowPass456!"  # noqa: S105

        client_a = _new_client(app)
        _register_and_login(client_a, username, old_password)

        # A second, independent session for the SAME user — this is the
        # "all OTHER sessions for that user destroyed" side of the
        # assertion below.
        client_b = _new_client(app)
        r = _login(client_b, username, old_password)
        assert r.status_code == 302, r.text
        _attach_csrf(client_b)

        # --- Data created BEFORE the password change, under the OLD key.
        collection_name = _unique_name("pwchange-collection")
        create_resp = client_a.post(
            "/library/api/collections", json={"name": collection_name}
        )
        assert create_resp.status_code == 200, create_resp.text
        collection_id = create_resp.json()["collection"]["id"]

        doc_filename = f"pwchange-{uuid.uuid4().hex[:6]}.txt"
        upload_resp = client_a.post(
            f"/library/api/collections/{collection_id}/upload",
            files={
                "files": (
                    doc_filename,
                    b"password change lifecycle document\n",
                    "text/plain",
                )
            },
            data={"storage_mode": "database"},
        )
        assert upload_resp.status_code == 200, upload_resp.text
        assert upload_resp.json().get("summary", {}).get("successful") == 1

        # --- Simulate a pooled AnyIO worker holding the OLD plaintext
        # password in the thread-credential cache (see helper docstring).
        seeded_tid, release_seed = _seed_worker_credential(
            username, old_password
        )
        try:
            from local_deep_research.database.thread_local_session import (
                thread_session_manager,
            )

            def _seeded_entry_present() -> bool:
                with thread_session_manager._lock:
                    entry = thread_session_manager._thread_credentials.get(
                        seeded_tid
                    )
                return entry is not None and entry[0] == username

            assert _seeded_entry_present(), "seed did not take"

            # Baseline: B's session genuinely works before the change (so
            # the lockout asserted below is caused BY the change, not some
            # unrelated failure).
            pre = client_b.get("/settings/api")
            assert pre.status_code == 200, pre.text

            # --- Change password via client_a. REAL rekey — nothing here
            # is mocked.
            change_resp = client_a.post(
                "/auth/change-password",
                data={
                    "current_password": old_password,
                    "new_password": new_password,
                    "confirm_password": new_password,
                    "csrf_token": _csrf(client_a),
                },
                follow_redirects=False,
            )
            assert change_resp.status_code == 302, (
                f"password change failed: {change_resp.status_code} "
                f"{change_resp.text[:400]}"
            )
            assert change_resp.headers.get("location") == "/auth/login"

            # --- 1. The submitting session is rejected (not just
            # redirected — /settings/api is an /api/ path).
            post_change_a = client_a.get("/settings/api")
            assert post_change_a.status_code == 401, (
                f"submitting session still authenticated: "
                f"{post_change_a.status_code}"
            )

            # --- 2. The OTHER, concurrent session for the same user is
            # ALSO rejected — destroy_all_user_sessions + the session
            # password store being cleared for the whole user, not just
            # one session_id.
            post_change_b = client_b.get("/settings/api")
            assert post_change_b.status_code == 401, (
                "a second, concurrent session for the same user survived "
                f"a password change: {post_change_b.status_code}"
            )

            # --- 3. The thread-credential cache entry seeded above (the
            # OLD password, on a simulated pooled worker) is gone.
            assert not _seeded_entry_present(), (
                "change-password left a pooled worker's cached OLD "
                "plaintext password in thread_local_session"
            )
        finally:
            release_seed.set()

        # --- 4. The encrypted DB now only opens with the NEW password —
        # login with the OLD password must fail...
        client_c = _new_client(app)
        old_login = _login(client_c, username, old_password)
        assert old_login.status_code == 401, (
            f"login with the OLD (rekeyed-away) password unexpectedly "
            f"succeeded: {old_login.status_code}"
        )

        # ...and login with the NEW password must succeed.
        new_login = _login(client_c, username, new_password)
        assert new_login.status_code == 302, (
            f"login with the NEW password failed: {new_login.status_code} "
            f"{new_login.text[:400]}"
        )
        _attach_csrf(client_c)

        # --- 5. Data created under the OLD key is still readable once the
        # DB is reopened under the NEW key.
        collections = client_c.get("/library/api/collections").json()
        assert collections.get("success") is True
        ids = {c["id"] for c in collections["collections"]}
        assert collection_id in ids, (
            f"collection missing after password change + relogin: {ids!r}"
        )

        docs = client_c.get(
            f"/library/api/collections/{collection_id}/documents"
        ).json()
        assert docs.get("success") is True, docs
        filenames = {d["filename"] for d in docs["documents"]}
        assert doc_filename in filenames, (
            f"uploaded document missing after password change + relogin: "
            f"{filenames!r}"
        )


# ===========================================================================
# 2. Queue lifecycle
# ===========================================================================


class TestQueueLifecycleAcrossUsers:
    """Submit past a user's ``app.max_concurrent_researches`` cap so rows
    QUEUE, check the queued state through two independent read paths, and
    verify a second user's cap/queue is completely unaffected.

    Crosses: settings router (writes the per-user cap) -> research router
    (reads it back and makes the queue/direct-start decision) ->
    QueuedResearch / ResearchHistory tables -> history router (both read
    paths) -> queue/processor_v2.py (notified, though its background
    thread is never started here — see class docstring below and
    ``TestQueueProcessorDispatchAccounting`` for why).
    """

    def test_submissions_beyond_cap_queue_and_second_user_is_unaffected(
        self, app
    ):
        client_a = _new_client(app)
        user_a = _unique_name("queue_a")
        _register_and_login(client_a, user_a)

        cap_resp = client_a.put(
            "/settings/api/app.max_concurrent_researches", json={"value": 1}
        )
        assert cap_resp.status_code == 200, cap_resp.text

        # --- Submission 1: under cap (active_count 0 < 1) -> starts
        # immediately.
        #
        # Keep its UserActiveResearch row for the duration of the test, so it
        # occupies the cap slot the way a genuinely-running research would.
        # These submissions use a deliberately-unreachable provider and so
        # reach a terminal state almost immediately, at which point the
        # terminal notification correctly DELETES the row and frees the slot.
        # (This test previously passed only because that row leaked -- rows
        # were never deleted on completion; fixing that leak is what exposed
        # the test's dependency on it.)
        # Hold the cap slot for the whole scenario. Both patches span BOTH
        # submissions because the terminal notification that would release
        # the slot fires on a background thread, after the request returns.
        #
        #  * _drop_active_research_row: research 1 uses a deliberately
        #    unreachable provider and so reaches a terminal state almost
        #    immediately, at which point its UserActiveResearch row is
        #    correctly DELETED and the slot frees. This test previously
        #    passed only because those rows leaked -- they were never deleted
        #    on completion. Fixing that leak is what exposed the dependency.
        #  * is_research_thread_alive: the admission check first reclaims
        #    rows whose worker thread is dead, which would also free the slot.
        #
        # Both simulate what a genuinely long-running research provides for
        # free, isolating what this test is actually about: the capacity
        # decision.
        _drop_patch = patch.object(
            QueueProcessorV2, "_drop_active_research_row", return_value=None
        )
        _alive_patch = patch(
            "local_deep_research.web.research_state.is_research_thread_alive",
            return_value=True,
        )
        _drop_patch.start()
        _alive_patch.start()
        try:
            r1 = _submit_research(
                client_a, f"queue test one {uuid.uuid4().hex[:6]}"
            )
            assert r1.status_code == 200, r1.text
            body1 = r1.json()
            assert body1.get("status") == "success", body1
            research_id_1 = body1["research_id"]

            status_1 = client_a.get(f"/history/status/{research_id_1}").json()
            assert status_1["status"] == ResearchStatus.IN_PROGRESS, status_1

            # --- Submission 2: at cap (active_count 1 >= 1) -> QUEUED.
            r2 = _submit_research(
                client_a, f"queue test two {uuid.uuid4().hex[:6]}"
            )
            assert r2.status_code == 200, r2.text
            body2 = r2.json()
            assert body2.get("status") == ResearchStatus.QUEUED, body2
        finally:
            _alive_patch.stop()
            _drop_patch.stop()

        assert body2.get("queue_position") == 1, body2
        research_id_2 = body2["research_id"]

        # Read path 1: status-by-id.
        status_2 = client_a.get(f"/history/status/{research_id_2}").json()
        assert status_2["status"] == ResearchStatus.QUEUED, status_2

        # Read path 2: the history listing (a different query entirely).
        history_items = client_a.get("/history/api").json().get("items", [])
        matching = {
            item["id"]: item["status"]
            for item in history_items
            if item["id"] in (research_id_1, research_id_2)
        }
        assert matching.get(research_id_1) == ResearchStatus.IN_PROGRESS
        assert matching.get(research_id_2) == ResearchStatus.QUEUED

        # NOTE: a third submission was deliberately NOT added here. The
        # provider name is unreachable specifically so the spawned
        # background thread for submission 1 fails FAST (no network); once
        # it has died, ``start_research()``'s own
        # ``reclaim_stale_user_active_research`` self-heals the "stale"
        # UserActiveResearch row on the NEXT submission (this is real,
        # correct, intentional production behaviour — see
        # ``web/research_state.py::reclaim_stale_user_active_research``).
        # That makes "is submission N still over cap" a wall-clock race
        # against that background thread's death for any N >= 3, which
        # would make this test non-deterministic. Two submissions are
        # enough to prove the cap -> QUEUED transition and are verified
        # directly against the DB immediately below, before any further
        # HTTP round-trip can let that race window open.

        # --- Directly verify the QueuedResearch row exists in A's own
        # encrypted DB (real DB, not just the HTTP surface).
        from local_deep_research.database.models import QueuedResearch
        from local_deep_research.database.session_context import (
            get_user_db_session,
        )

        with get_user_db_session(user_a, TEST_PASSWORD) as db_session:
            queued_ids = {
                row.research_id
                for row in db_session.query(QueuedResearch).filter_by(
                    username=user_a
                )
            }
        assert queued_ids == {research_id_2}, queued_ids

        # --- A second, unrelated user: default cap (3), nothing queued.
        # Their FIRST submission must start immediately even though user A
        # already has two QUEUED rows sitting around — the cap and the
        # queue are per-user.
        client_b = _new_client(app)
        user_b = _unique_name("queue_b")
        _register_and_login(client_b, user_b)

        rb = _submit_research(client_b, f"queue test b {uuid.uuid4().hex[:6]}")
        assert rb.status_code == 200, rb.text
        body_b = rb.json()
        assert body_b.get("status") == "success", (
            "user B's first submission was queued even though B has no "
            f"active research and a default cap: {body_b!r}"
        )

        with get_user_db_session(user_b, TEST_PASSWORD) as db_session:
            b_queued_count = (
                db_session.query(QueuedResearch)
                .filter_by(username=user_b)
                .count()
            )
        assert b_queued_count == 0, (
            "user B has QueuedResearch rows despite never exceeding their "
            "own cap — user A's queue leaked into user B's database (it "
            "shouldn't be possible, since these are separate per-user "
            "encrypted DBs, but assert it explicitly)"
        )

        # --- SCOPED DOWN, on purpose (see task brief): we stop here and do
        # NOT attempt to observe QUEUED -> IN_PROGRESS for research_id_2/3.
        # The queue processor's background dispatch thread is stopped by
        # the autouse `reset_all_singletons` fixture (tests/conftest.py)
        # before AND after every test, specifically to keep the suite
        # deterministic. The next test exercises one dispatch tick directly
        # and pins the shared active-slot accounting contract.

    def test_dispatch_slot_accounting_includes_directly_started_research(
        self, app
    ):
        """Queue dispatch counts directly started research against the cap.

        Setup: cap = 1. Submission 1 starts directly (the ordinary,
        non-queued path in ``research.py``'s ``start_research()`` — this is
        what happens on every submission until a user is already at their
        cap). Submission 2 is correctly queued by the SAME route, because
        the cap check there queries the real ``UserActiveResearch`` table.

        Queue replay now derives capacity from the same ``UserActiveResearch``
        rows as direct submission, under their shared per-user admission lock.
        The queued research must therefore remain queued while research #1 is
        still active.

        This calls the real ``QueueProcessorV2._process_user_queue`` method
        directly — the exact method the (normally-disabled-under-pytest)
        background thread calls every tick — rather than starting that
        thread. The only mocking pins research #1's ``UserActiveResearch``
        row open for the duration (see the comment at the two ``patch``
        calls below); it stands in for a genuinely long-running research
        and touches nothing about the accounting logic under test.

        This used to be a strict xfail. The shared accounting fix makes its
        existing correct-behaviour assertions pass, so the marker must not
        remain or pytest reports the fix itself as an XPASS failure.
        """
        from local_deep_research.database.models import UserActiveResearch
        from local_deep_research.database.session_context import (
            get_user_db_session,
        )
        from local_deep_research.web.queue.processor_v2 import queue_processor

        client = _new_client(app)
        username = _unique_name("queue_bug")
        _register_and_login(client, username)

        cap_resp = client.put(
            "/settings/api/app.max_concurrent_researches", json={"value": 1}
        )
        assert cap_resp.status_code == 200, cap_resp.text

        r1 = _submit_research(client, f"bug repro one {uuid.uuid4().hex[:6]}")
        assert r1.status_code == 200, r1.text
        assert r1.json().get("status") == "success", r1.json()

        # Pin research #1's slot open for the rest of the scenario, the same
        # way the sibling test above does and for the same reason.
        # `_submit_research` uses a deliberately-unreachable provider, so
        # research #1's worker thread dies almost immediately. That death
        # races TWO independent cleanup paths against the dispatch tick this
        # test is trying to observe:
        #   * the admission-time reclaim in research.py's start_research
        #     (`reclaim_stale_user_active_research` flips a dead-thread row
        #     to FAILED), and
        #   * the terminal-notification row deletion in QueueProcessorV2
        #     (`notify_research_failed` -> `_drop_active_research_row`).
        # Under CI load thread #1 frequently loses that race, so the row is
        # gone before the tick runs, `available_slots` reads as expected and
        # the accounting bug does not fire — an XPASS, which `strict=True`
        # turns into a failed job even though the product bug is untouched.
        # Pinning holds the slot open exactly as a genuinely long-running
        # research would for free. Neither patch touches
        # `_process_user_queue`'s `available_slots` computation, research.py's
        # direct-start branch, or the UserQueueService/TaskMetadata gap — the
        # mechanism under test runs completely unmocked.
        _drop_patch = patch.object(
            QueueProcessorV2, "_drop_active_research_row", return_value=None
        )
        _alive_patch = patch(
            "local_deep_research.web.research_state.is_research_thread_alive",
            return_value=True,
        )
        _drop_patch.start()
        _alive_patch.start()
        try:
            r2 = _submit_research(
                client, f"bug repro two {uuid.uuid4().hex[:6]}"
            )
            assert r2.status_code == 200, r2.text
            assert r2.json().get("status") == ResearchStatus.QUEUED, r2.json()

            session_id = client.cookies.get("session")
            # Decode the real session_id out of the signed cookie payload
            # (same technique as test_auth_flow_gaps.py::_session_dict).
            import base64
            import json as _json

            b64_part = session_id.split(".")[0]
            padded = b64_part + "=" * (-len(b64_part) % 4)
            sid = _json.loads(base64.b64decode(padded)).get("session_id")
            assert sid

            # Call the real production dispatch method directly, once.
            queue_processor._process_user_queue(username, sid)

            with get_user_db_session(username, TEST_PASSWORD) as db_session:
                in_progress_count = (
                    db_session.query(UserActiveResearch)
                    .filter_by(
                        username=username, status=ResearchStatus.IN_PROGRESS
                    )
                    .count()
                )
        finally:
            _alive_patch.stop()
            _drop_patch.stop()

        assert in_progress_count <= 1, (
            "dispatch exceeded the configured cap of 1: "
            f"{in_progress_count} researches are IN_PROGRESS after a single "
            "dispatch tick"
        )


# ===========================================================================
# 3. Collection -> document -> deletion cascade
# ===========================================================================


class TestCollectionDocumentCascadeDeletion:
    """Create a collection, upload a document, delete the collection, then
    verify the cascade is complete: DB rows gone, no plaintext trace left
    on disk, and a second user's identically-named collection untouched.

    SCOPED DOWN: this environment has no LLM/embedding provider and no
    network, so a real FAISS/RAG index cannot be built (sentence-transformer
    weights would need a HuggingFace download; a fake-provider-name trick
    like the LLM one doesn't work here because indexing needs to SUCCEED,
    not fail fast, to have anything to cascade-delete). This test therefore
    verifies the Collection/Document/DocumentCollection/RAGIndex/
    DocumentChunk row-level cascade and the on-disk footprint, not actual
    FAISS file cleanup. The RAGIndex/DocumentChunk row assertions are real
    but not very informative on their own here (nothing indexed this
    document, so they're empty before AND after) — included anyway since a
    non-empty leak in the "after" state would still be a real bug even
    though a positive ("something existed and got cleaned up") case isn't
    exercised. Real FAISS-file cleanup already has dedicated (non-HTTP)
    coverage in tests/research_library/test_document_full_lifecycle.py and
    tests/research_library/zotero/test_faiss_cleanup.py.
    """

    def test_delete_collection_cascades_and_second_user_untouched(self, app):
        from local_deep_research.database.models.library import (
            Collection,
            Document,
            DocumentChunk,
            DocumentCollection,
            RAGIndex,
        )
        from local_deep_research.database.session_context import (
            get_user_db_session,
        )

        client_a = _new_client(app)
        user_a = _unique_name("cascade_a")
        _register_and_login(client_a, user_a)

        client_b = _new_client(app)
        user_b = _unique_name("cascade_b")
        _register_and_login(client_b, user_b)

        shared_name = _unique_name("cascade-collection")

        create_a = client_a.post(
            "/library/api/collections", json={"name": shared_name}
        )
        assert create_a.status_code == 200, create_a.text
        collection_id_a = create_a.json()["collection"]["id"]

        # B creates a collection with the SAME name — proves the deletion
        # below is scoped to A's row, not name-matched globally.
        create_b = client_b.post(
            "/library/api/collections", json={"name": shared_name}
        )
        assert create_b.status_code == 200, create_b.text
        collection_id_b = create_b.json()["collection"]["id"]
        assert collection_id_b != collection_id_a

        marker = f"CASCADEMARKER{uuid.uuid4().hex[:10]}"
        doc_filename = f"cascade-{uuid.uuid4().hex[:6]}.txt"
        upload_resp = client_a.post(
            f"/library/api/collections/{collection_id_a}/upload",
            files={
                "files": (
                    doc_filename,
                    f"cascade deletion test document {marker}\n".encode(),
                    "text/plain",
                )
            },
            data={"storage_mode": "database"},
        )
        assert upload_resp.status_code == 200, upload_resp.text
        upload_body = upload_resp.json()
        assert upload_body.get("summary", {}).get("successful") == 1
        document_id = upload_body["uploaded"][0]["id"]

        # --- Appears in the collection listing before deletion.
        docs_before = client_a.get(
            f"/library/api/collections/{collection_id_a}/documents"
        ).json()
        assert docs_before.get("success") is True
        assert document_id in {d["id"] for d in docs_before["documents"]}

        collections_before = client_a.get("/library/api/collections").json()
        by_id = {c["id"]: c for c in collections_before["collections"]}
        assert by_id[collection_id_a]["document_count"] == 1

        # --- Delete via the real, unmocked CollectionDeletionService.
        del_resp = client_a.delete(
            f"/library/api/collections/{collection_id_a}"
        )
        assert del_resp.status_code == 200, del_resp.text
        del_body = del_resp.json()
        assert del_body.get("deleted") is True, del_body
        assert del_body.get("documents_unlinked") == 1
        assert del_body.get("orphaned_documents_deleted") == 1

        # --- HTTP surface: collection and its documents are gone.
        collections_after = client_a.get("/library/api/collections").json()
        assert collection_id_a not in {
            c["id"] for c in collections_after["collections"]
        }
        docs_after = client_a.get(
            f"/library/api/collections/{collection_id_a}/documents"
        )
        assert docs_after.status_code == 404

        # --- Real per-user encrypted DB: every related row is gone.
        with get_user_db_session(user_a, TEST_PASSWORD) as db_session:
            assert (
                db_session.query(Collection)
                .filter_by(id=collection_id_a)
                .first()
                is None
            ), "Collection row survived deletion"
            assert (
                db_session.query(Document).filter_by(id=document_id).first()
                is None
            ), "orphaned Document row survived deletion"
            assert (
                db_session.query(DocumentCollection)
                .filter_by(collection_id=collection_id_a)
                .count()
                == 0
            ), "DocumentCollection link row(s) survived deletion"
            assert (
                db_session.query(RAGIndex)
                .filter_by(collection_name=f"collection_{collection_id_a}")
                .count()
                == 0
            )
            assert (
                db_session.query(DocumentChunk)
                .filter_by(source_id=document_id)
                .count()
                == 0
            )

        # --- No plaintext trace of the document's content survives
        # anywhere on disk under this user's data directory (everything
        # that DID exist lived inside the encrypted SQLCipher file).
        from local_deep_research.config.paths import get_data_directory

        needle = marker.encode()
        leaks = [
            str(f)
            for f in get_data_directory().rglob("*")
            if f.is_file() and needle in f.read_bytes()
        ]
        assert leaks == [], (
            f"plaintext trace of deleted content found: {leaks!r}"
        )

        # --- B's identically-named collection is completely untouched.
        collections_b = client_b.get("/library/api/collections").json()
        b_ids = {c["id"] for c in collections_b["collections"]}
        assert collection_id_b in b_ids, (
            "B's collection was affected by A's deletion of a "
            "same-named collection"
        )
        docs_b = client_b.get(
            f"/library/api/collections/{collection_id_b}/documents"
        )
        assert docs_b.status_code == 200
        with get_user_db_session(user_b, TEST_PASSWORD) as db_session:
            assert (
                db_session.query(Collection)
                .filter_by(id=collection_id_b)
                .first()
                is not None
            ), "B's collection row was deleted by A's cascade"


# ===========================================================================
# 4. Settings that change behaviour, end to end
# ===========================================================================


class TestSettingsChangeBehaviorAcrossRouters:
    """Write ``policy.egress_scope`` through the settings router and prove
    the research router's request-boundary egress precheck — a completely
    different file — behaves differently on the very next call.

    Crosses: settings router (persists the value) -> SettingsManager / DB
    -> research router's ``_precheck_engine_policy`` (reads a fresh
    snapshot on every request, no caching layer in between) -> the egress
    policy decision engine.
    """

    def test_egress_scope_write_changes_research_router_precheck(self, app):
        client = _new_client(app)
        username = _unique_name("egress_settings_user")
        _register_and_login(client, username)

        # --- Baseline: default scope (adaptive) accepts a public engine
        # (searxng). Uses the same fail-fast-provider trick as elsewhere in
        # this file/its sibling — no network, the row is what we assert on.
        baseline = _submit_research(
            client, f"egress baseline {uuid.uuid4().hex[:6]}", engine="searxng"
        )
        assert baseline.status_code == 200, baseline.text
        assert baseline.json().get("status") == "success", baseline.json()

        # --- Write policy.egress_scope=private_only via the SETTINGS
        # router.
        put_resp = client.put(
            "/settings/api/policy.egress_scope", json={"value": "private_only"}
        )
        assert put_resp.status_code == 200, put_resp.text

        # Confirm the write round-trips through the SAME router (sanity —
        # not the interesting assertion).
        got = client.get("/settings/api/policy.egress_scope").json()
        assert got.get("value") == "private_only", got

        # --- The RESEARCH router's precheck, on the very next call,
        # refuses the exact same public engine it accepted a moment ago —
        # proving it reads the freshly-written value rather than a stale
        # snapshot cached from before the write.
        after = _submit_research(
            client,
            f"egress after write {uuid.uuid4().hex[:6]}",
            engine="searxng",
        )
        assert after.status_code == 400, (
            f"research router did not observe the settings-router write: "
            f"{after.status_code} {after.text[:400]}"
        )
        after_body = after.json()
        assert after_body.get("reason") == "scope_mismatch_private_only", (
            after_body
        )
        assert after_body.get("field") == "policy_egress_scope", after_body


# ===========================================================================
# 5. Recovery after a simulated restart
# ===========================================================================


class TestRecoveryAfterSimulatedRestart:
    """Build real state for two users, drop every in-memory singleton a
    process restart would wipe, and verify both users are forced to
    re-authenticate and — once they do — see only their own persisted data.

    Crosses: session_manager, session_passwords store, thread_local_session
    credential cache, and the auth-DB SQLAlchemy engine (all in-memory,
    all reset here) versus the durable per-user encrypted SQLCipher files
    on disk (untouched by the reset, exactly as a real process restart
    would leave them).
    """

    def test_state_survives_singleton_reset_and_forces_relogin(self, app):
        from local_deep_research.database.auth_db import dispose_auth_engine
        from local_deep_research.database.encrypted_db import db_manager
        from local_deep_research.database.session_passwords import (
            session_password_store,
        )
        from local_deep_research.database.thread_local_session import (
            thread_session_manager,
        )
        from local_deep_research.web.auth.session_manager import (
            session_manager,
        )

        client_a = _new_client(app)
        user_a = _unique_name("restart_a")
        _register_and_login(client_a, user_a)
        marker_a = f"restart-a-{uuid.uuid4().hex[:8]}"
        assert (
            client_a.put(
                "/settings/api/llm.model", json={"value": marker_a}
            ).status_code
            == 200
        )
        coll_a = client_a.post(
            "/library/api/collections",
            json={"name": _unique_name("restart-coll-a")},
        )
        assert coll_a.status_code == 200, coll_a.text
        collection_id_a = coll_a.json()["collection"]["id"]

        client_b = _new_client(app)
        user_b = _unique_name("restart_b")
        _register_and_login(client_b, user_b)
        marker_b = f"restart-b-{uuid.uuid4().hex[:8]}"
        assert (
            client_b.put(
                "/settings/api/llm.model", json={"value": marker_b}
            ).status_code
            == 200
        )
        coll_b = client_b.post(
            "/library/api/collections",
            json={"name": _unique_name("restart-coll-b")},
        )
        assert coll_b.status_code == 200, coll_b.text
        collection_id_b = coll_b.json()["collection"]["id"]

        # Both sessions genuinely work pre-restart.
        assert client_a.get("/settings/api").status_code == 200
        assert client_b.get("/settings/api").status_code == 200

        # --- Simulate a process restart: drop every in-memory singleton
        # the running server would lose. The per-user encrypted DB FILES
        # on disk are untouched (that's the whole point).
        db_manager.close_all_databases()
        session_manager.sessions.clear()
        with session_password_store._lock:
            session_password_store._store.clear()
        with thread_session_manager._lock:
            thread_session_manager._thread_credentials.clear()
        dispose_auth_engine()

        # --- Both OLD cookies are now rejected — genuinely (JSON 401 on
        # an /api/ path), forcing re-login. Neither client has logged out;
        # this is purely the effect of the in-memory reset.
        post_restart_a = client_a.get("/settings/api")
        assert post_restart_a.status_code == 401, (
            f"user A's pre-restart session survived the singleton reset: "
            f"{post_restart_a.status_code}"
        )
        post_restart_b = client_b.get("/settings/api")
        assert post_restart_b.status_code == 401, (
            f"user B's pre-restart session survived the singleton reset: "
            f"{post_restart_b.status_code}"
        )

        # --- Re-login (real HTTP, against the real on-disk encrypted DB
        # files, which the reset above never touched).
        relogin_a = _login(client_a, user_a, TEST_PASSWORD)
        assert relogin_a.status_code == 302, relogin_a.text
        _attach_csrf(client_a)

        relogin_b = _login(client_b, user_b, TEST_PASSWORD)
        assert relogin_b.status_code == 302, relogin_b.text
        _attach_csrf(client_b)

        # --- Persisted data is intact and correctly attributed to each
        # user (never swapped or merged).
        setting_a = client_a.get("/settings/api/llm.model").json()
        assert setting_a.get("value") == marker_a, setting_a
        setting_b = client_b.get("/settings/api/llm.model").json()
        assert setting_b.get("value") == marker_b, setting_b

        collections_a = client_a.get("/library/api/collections").json()
        a_ids = {c["id"] for c in collections_a["collections"]}
        assert collection_id_a in a_ids
        assert collection_id_b not in a_ids

        collections_b = client_b.get("/library/api/collections").json()
        b_ids = {c["id"] for c in collections_b["collections"]}
        assert collection_id_b in b_ids
        assert collection_id_a not in b_ids