Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

34.4 KB
Raw Blame
"""
Long, multi-step INTEGRATION flows for the Flask -> FastAPI migration branch.

Unlike the per-endpoint tests scattered across ``tests/web/routers/``, every
test in this file is a genuine SEQUENCE with state carried from one step to
the next — register survives into login, a setting written through one
router is read back through a different router, a session captured before
logout is replayed after it. The migration's worst defects were reportedly
behaviours lost BETWEEN files (a route moved from Flask to FastAPI but a
side effect of the old ``before_request``/session-teardown chain didn't come
with it); only a path that actually crosses those files exercises that.

SURVEY — what already exists (not duplicated here)
----------------------------------------------------
* ``tests/web/routers/test_state_changing_flows.py`` — settings PUT/GET/DELETE
  round-trip on a single router, bulk-endpoint cache coherency *within*
  settings, slowapi login-burst 429, Socket.IO subscription bookkeeping,
  single-shot cross-user settings isolation (A writes, B reads default).
* ``tests/web/routers/test_authenticated_flows.py`` — one GET per endpoint
  smoke coverage, security headers on error responses, the
  DatabaseInitializationError -> 503 regression, the ``?next=`` encoding
  regression.
* ``tests/web/test_session_cookie_behavior.py`` — Set-Cookie attribute
  correctness (Max-Age/Expires/SameSite/HttpOnly/Secure) for login with and
  without "remember me", at the header level, for a single request/response
  pair. Does NOT replay an old cookie after logout — that gap is what
  ``TestSessionLifecycle`` below covers.
* ``tests/security/test_cross_user_isolation_invariants.py`` — pins five
  *internal* isolation invariants (engine cache, thread-local session
  self-heal, ``thread_specific_cache`` keying, settings-context identity
  guard, contextvar propagation across the shared AnyIO threadpool) by
  calling the underlying Python functions directly with real threads/tasks.
  It preserves the corrected username-aware cache-key behavior and adjacent
  isolation defenses.
* ``tests/web/routers/test_collection_upload_http.py`` — pins the
  "starlette UploadFile vs fastapi.UploadFile" multipart regression for a
  single collection/upload call pair (no lifecycle around it).

None of the above chains registration through logout/re-login, replays a
captured cookie, drives two users through an interleaved multi-step
sequence via real HTTP, or submits research end-to-end through the real
per-user encrypted DB and checks the resulting row's cross-user visibility.
That gap is what this file fills.

FIXTURE ISOLATION
-----------------
Every test below takes the function-scoped ``app`` fixture from
``tests/conftest.py``. The fixture sets ``LDR_DATA_DIR`` to a fresh temporary
directory. ``DatabaseManager.data_dir`` resolves its override or
``get_data_directory()`` on each access, so the fixture's isolation remains
effective throughout the test. No test in this file imports a global app
before that fixture is established.

Tests elsewhere that bypass the shared app fixture can resolve the platform
default application data directory when ``LDR_DATA_DIR`` is unset during a
single-file run. This file does not. Broader test-harness isolation remains
follow-up work tracked in issue #6014.

DROPPED / SCOPED-DOWN CANDIDATES
---------------------------------
* True multi-THREAD concurrency for the "two users, interleaved" flow
  (candidate #2) is used only for a narrow, single-mutation-per-user
  slice (``TestTwoConcurrentUsersInterleaved.test_concurrent_http_requests_do_not_cross_users``).
  A broader, many-step concurrent dance is intentionally omitted because it
  would add timing sensitivity without strengthening the cache-key regression
  invariant. The bulk of flow #2 is instead a deterministic sequential
  interleave (A mutates, B mutates, A reads, B reads, ...) that still drives
  the full HTTP, CSRF, per-user database, and session stack for both users.
* Waiting for a submitted research run to reach a terminal state
  (candidate #4) is not attempted. ``POST /api/start_research`` commits the
  ``ResearchHistory``/``UserActiveResearch`` rows and returns before the
  background thread runs, so the row's existence/attribution is already
  observable at response time — no LLM, no network, and (per the task
  brief) no requirement to observe completion. The submission deliberately
  uses a nonexistent LLM provider name (``_UNREACHABLE_PROVIDER``) so
  that, if the background thread ever runs far enough to reach the LLM
  factory, it fails FAST and in-process with a ``ValueError`` before
  opening any socket. The search engine must be a real registered name
  (``searxng`` / ``_ACCEPTED_ENGINE``): an unrecognized engine name is
  rejected by the request-boundary egress-policy precheck with a 400
  *before* the ResearchHistory row is even created, which would make the
  row-existence assertions this test is actually chartered to make
  untestable.
* No test waits on anything with a bare ``sleep``. The one place with
  genuine concurrency (``test_concurrent_http_requests_do_not_cross_users``)
  uses a ``threading.Barrier`` to force simultaneity and bounded
  ``Thread.join(timeout=...)`` calls to bound the wait — never a
  fixed-length sleep.
"""

import threading
import uuid

from fastapi.testclient import TestClient

# Used to assert the submitted-research row lands in a genuine lifecycle
# status right after submission (TestResearchSubmissionQueueHistory) — not
# just "some string", without pinning to a specific terminal state we
# deliberately never wait for (see the module docstring's "DROPPED /
# SCOPED-DOWN CANDIDATES" section on why completion is out of scope here).
from local_deep_research.constants import ResearchStatus

TEST_PASSWORD = "LongFlowPass123!"  # noqa: S105

# _UNREACHABLE_PROVIDER is not registered with any LLM provider factory, so
# the background research thread fails with a synchronous ValueError
# ("Unknown provider ...") before it ever opens a socket for an LLM call —
# fast, deterministic, no network/LLM needed. The search engine, by
# contrast, MUST be a real registered name: the request-boundary egress
# policy precheck in start_research (``_precheck_engine_policy`` ->
# ``evaluate_engine``) fail-closed rejects any unrecognized engine name
# with a 400 *before* the ResearchHistory row is even created — an
# unknown engine name never reaches the background thread at all, so it
# can't be used to keep the DB-row-creation part of this test network-free.
# "searxng" is DEFAULT_SEARCH_TOOL and is accepted by the policy precheck;
# the background thread still never completes without a real instance, but
# (as documented in the module docstring) we don't wait for or assert on
# that outcome.
_UNREACHABLE_PROVIDER = "no_such_llm_provider_xyz"
_ACCEPTED_ENGINE = "searxng"


# ---------------------------------------------------------------------------
# Shared helpers (module-level functions, not fixtures: every test below
# takes the function-scoped ``app`` fixture from tests/conftest.py, which
# hands back a fresh, isolated LDR_DATA_DIR per test — see the module
# docstring's "FIXTURE ISOLATION" section.
# ---------------------------------------------------------------------------


def _new_client(app) -> TestClient:
    """Fresh TestClient with its own cookie jar and rate-limit bucket.

    A unique X-Forwarded-For keeps this client's registration attempts out
    of any other client's slowapi bucket (register is capped per-IP); the
    TestClient peer address is treated as private, so X-Forwarded-For is
    honored.
    """
    client = TestClient(app, raise_server_exceptions=False)
    fwd_ip = f"10.{uuid.uuid4().int % 254 + 1}.{uuid.uuid4().int % 254 + 1}.1"
    client.headers.update({"X-Forwarded-For": fwd_ip})
    return client


def _csrf(client: TestClient) -> str:
    """Stamp the session with a CSRF token and return it."""
    client.get("/auth/login")
    resp = client.get("/auth/csrf-token")
    return resp.json().get("csrf_token", "") if resp.status_code == 200 else ""


def _drop_stale_csrf_header(client: TestClient) -> None:
    """Remove any previously-attached X-CSRFToken default header.

    CSRFMiddleware prefers the header over the ``csrf_token`` form field
    (it only falls back to the form field when the header is absent) —
    see src/local_deep_research/web/dependencies/csrf.py. ``_attach_csrf``
    sets a PERSISTENT default header on the client; if the session's CSRF
    token later rotates (e.g. logout clears the session, so the next GET
    stamps a brand new one), that old header shadows the fresh
    ``csrf_token`` form field this module's ``_register``/``_login`` pass,
    and every subsequent form POST 403s with "CSRF token missing or
    invalid" even though the form field itself is correct. Call this
    before posting a form with an explicit ``csrf_token`` field so the
    header (if any) never wins over it.
    """
    for name in ("X-CSRFToken", "X-CSRF-Token"):
        client.headers.pop(name, None)


def _register(client: TestClient, username: str, password: str = TEST_PASSWORD):
    _drop_stale_csrf_header(client)
    return client.post(
        "/auth/register",
        data={
            "username": username,
            "password": password,
            "confirm_password": password,
            "acknowledge": "true",
            "csrf_token": _csrf(client),
        },
        follow_redirects=False,
    )


def _login(
    client: TestClient,
    username: str,
    password: str = TEST_PASSWORD,
    remember: str = "false",
):
    _drop_stale_csrf_header(client)
    return client.post(
        "/auth/login",
        data={
            "username": username,
            "password": password,
            "remember": remember,
            "csrf_token": _csrf(client),
        },
        follow_redirects=False,
    )


def _attach_csrf(client: TestClient) -> None:
    """Fetch the session's CSRF token and set it as a default header so
    subsequent JSON PUT/POST/DELETE calls pass CSRFMiddleware."""
    resp = client.get("/auth/csrf-token")
    if resp.status_code == 200:
        token = resp.json().get("csrf_token")
        if token:
            client.headers.update({"X-CSRFToken": token})


def _register_and_login(
    client: TestClient, username: str, password: str = TEST_PASSWORD
) -> None:
    r = _register(client, username, password)
    assert r.status_code == 302, (
        f"register failed for {username!r}: {r.status_code} {r.text[:300]}"
    )
    r = _login(client, username, password)
    assert r.status_code == 302, (
        f"login failed for {username!r}: {r.status_code} {r.text[:300]}"
    )
    _attach_csrf(client)


def _unique_name(prefix: str) -> str:
    return f"{prefix}_{uuid.uuid4().hex[:10]}"


# ---------------------------------------------------------------------------
# 1. Full user lifecycle
# ---------------------------------------------------------------------------


class TestFullUserLifecycle:
    """register -> login -> change a setting -> create a collection ->
    upload a document -> log out -> log back in -> everything persisted
    and still scoped to that user (and NOT visible to a second user).

    Crosses: auth (register/login/logout), CSRF middleware, per-user
    encrypted-DB provisioning, the settings router, and the library/rag
    router — five files' worth of state that all have to agree the same
    user is involved at every step.
    """

    def test_lifecycle_persists_across_relogin_and_stays_scoped(self, app):
        client = _new_client(app)
        username = _unique_name("lifecycle_user")
        _register_and_login(client, username)

        # --- Step: change a setting.
        model_marker = f"lifecycle-model-{uuid.uuid4().hex[:8]}"
        put_resp = client.put(
            "/settings/api/llm.model", json={"value": model_marker}
        )
        assert put_resp.status_code == 200, put_resp.text

        # --- Step: create a collection.
        collection_name = _unique_name("lifecycle-collection")
        create_resp = client.post(
            "/library/api/collections", json={"name": collection_name}
        )
        assert create_resp.status_code == 200, create_resp.text
        create_body = create_resp.json()
        assert create_body.get("success") is True, create_body
        collection_id = create_body["collection"]["id"]

        # --- Step: upload a document into that collection.
        doc_filename = f"lifecycle-{uuid.uuid4().hex[:6]}.txt"
        upload_resp = client.post(
            f"/library/api/collections/{collection_id}/upload",
            files={
                "files": (
                    doc_filename,
                    b"full lifecycle integration test document\n",
                    "text/plain",
                )
            },
            data={"storage_mode": "database"},
        )
        assert upload_resp.status_code == 200, upload_resp.text
        upload_body = upload_resp.json()
        assert upload_body.get("success") is True, upload_body
        assert upload_body.get("summary", {}).get("successful") == 1, (
            upload_body
        )

        # --- Step: log out.
        logout_resp = client.post("/auth/logout", follow_redirects=False)
        assert logout_resp.status_code == 302, logout_resp.text

        # Confirm the session is actually gone (not just a page redirect) —
        # an authenticated JSON API must now 401.
        settings_after_logout = client.get("/settings/api")
        assert settings_after_logout.status_code == 401, (
            f"session still authenticated after logout: "
            f"{settings_after_logout.status_code}"
        )

        # --- Step: log back in (SAME client, i.e. same underlying user).
        relogin_resp = _login(client, username)
        assert relogin_resp.status_code == 302, relogin_resp.text
        _attach_csrf(client)

        # --- Assert: the setting change persisted (it's in the DB, not
        # just session/session-cookie state that logout would have wiped).
        got_setting = client.get("/settings/api/llm.model").json()
        assert got_setting.get("value") == model_marker, (
            f"setting did not survive logout/login: {got_setting!r}"
        )

        # --- Assert: the collection persisted and is listed.
        collections = client.get("/library/api/collections").json()
        assert collections.get("success") is True
        ids = {c["id"] for c in collections["collections"]}
        assert collection_id in ids, (
            f"collection {collection_id} missing after relogin: {ids!r}"
        )

        # --- Assert: the uploaded document persisted inside the collection.
        docs = client.get(
            f"/library/api/collections/{collection_id}/documents"
        ).json()
        assert docs.get("success") is True, docs
        filenames = {d["filename"] for d in docs["documents"]}
        assert doc_filename in filenames, (
            f"uploaded document missing after relogin: {filenames!r}"
        )

        # --- Assert: none of this is visible to a second, unrelated user
        # (each user has their own encrypted DB — a completely different
        # user must never see this collection, let alone its documents).
        other_client = _new_client(app)
        other_username = _unique_name("lifecycle_other")
        _register_and_login(other_client, other_username)

        other_collections = other_client.get("/library/api/collections").json()
        other_ids = {c["id"] for c in other_collections["collections"]}
        assert collection_id not in other_ids, (
            "cross-user leak: second user's collection list contains the "
            "first user's collection id"
        )

        cross_docs_resp = other_client.get(
            f"/library/api/collections/{collection_id}/documents"
        )
        assert cross_docs_resp.status_code == 404, (
            "cross-user leak: second user could fetch documents for a "
            f"collection_id belonging to the first user "
            f"(got {cross_docs_resp.status_code})"
        )

        # And the setting marker must not leak to the second user either —
        # they should see whatever their own (default) llm.model is.
        other_setting = other_client.get("/settings/api/llm.model").json()
        assert other_setting.get("value") != model_marker, (
            "cross-user leak: second user's llm.model equals the first "
            "user's custom value"
        )


# ---------------------------------------------------------------------------
# 2. Session lifecycle
# ---------------------------------------------------------------------------


class TestSessionLifecycle:
    """login -> use an authenticated route -> log out -> the CURRENT
    client's session is rejected (not merely redirected) -> log back in ->
    a NEW session works.

    ``test_cookie_captured_before_logout_is_rejected_after_logout``
    additionally replays a cookie CAPTURED before logout — see its
    docstring; it pins a migration regression that has since been fixed.
    """

    def test_new_login_after_logout_creates_working_session(self, app):
        client = _new_client(app)
        username = _unique_name("session_user")
        _register_and_login(client, username)

        # Authenticated route works before logout.
        pre_logout = client.get("/settings/api")
        assert pre_logout.status_code == 200, pre_logout.text

        # Log out through the SAME client.
        logout_resp = client.post("/auth/logout", follow_redirects=False)
        assert logout_resp.status_code == 302, logout_resp.text

        # The CLIENT'S OWN (current) session is rejected on the same route
        # — genuinely, not merely redirected: /settings/api is an /api/
        # path, so the exception handler must return JSON 401, never a
        # Location header.
        post_logout = client.get("/settings/api")
        assert post_logout.status_code == 401, (
            f"session still authenticated after logout: "
            f"{post_logout.status_code} {post_logout.text[:300]}"
        )
        assert "location" not in {
            k.lower() for k in post_logout.headers.keys()
        }, "post-logout request was redirected instead of cleanly rejected"

        # Log back in with the SAME client -> a genuinely new session.
        old_cookie_value = client.cookies.get("session")
        relogin_resp = _login(client, username)
        assert relogin_resp.status_code == 302, relogin_resp.text
        _attach_csrf(client)

        new_session_cookie = client.cookies.get("session")
        assert new_session_cookie, "no session cookie after relogin"
        assert new_session_cookie != old_cookie_value

        post_relogin = client.get("/settings/api")
        assert post_relogin.status_code == 200, (
            f"new session does not work after relogin: "
            f"{post_relogin.status_code} {post_relogin.text[:300]}"
        )
        check = client.get("/auth/check").json()
        assert check.get("authenticated") is True
        assert check.get("username") == username

    def test_cookie_captured_before_logout_is_rejected_after_logout(self, app):
        """Logout must invalidate a cookie captured before it.

        Regression test for a migration defect. ``ensure_user_database()``
        gained an ``if db_manager.is_user_connected(username): return``
        fast path at the TOP of the function, above the block that consumes
        ``temp_auth_token``. Login opens the connection, so that fast path
        hit on every subsequent request and the one-time token was never
        consumed: it stayed live in ``temp_auth_store`` for its full 10s TTL
        (``database/temp_auth.py``) and stayed embedded in the client's
        already-issued signed cookie.

        ``logout()`` closes the DB connection and calls
        ``session_password_store.clear_all_for_user()`` — its own comments
        cite exactly this replay scenario — but it cannot reach into a
        cookie that has already been handed out. So a cookie captured within
        10s of login still re-authenticated after logout via the Source-1
        fallback. Worse, that fallback then wrote the recovered password
        into ``session_password_store``, promoting a 10-second window into a
        24-hour session.

        Flask consumed the token unconditionally and checked
        ``is_user_connected()`` only at the point of opening the DB
        (``web/auth/database_middleware.py``), so this was a regression, not
        an inherited bug. The fix restores that ordering; reverting it fails
        this test.
        """
        client = _new_client(app)
        username = _unique_name("session_replay_user")
        _register_and_login(client, username)

        # Authenticated route works before logout.
        pre_logout = client.get("/settings/api")
        assert pre_logout.status_code == 200, pre_logout.text

        old_cookie_value = client.cookies.get("session")
        assert old_cookie_value, "no session cookie was set after login"

        # Log out through the SAME client.
        logout_resp = client.post("/auth/logout", follow_redirects=False)
        assert logout_resp.status_code == 302, logout_resp.text

        new_cookie_after_logout = client.cookies.get("session")
        assert new_cookie_after_logout != old_cookie_value, (
            "logout did not rotate the client's session cookie"
        )

        # Replay the CAPTURED PRE-LOGOUT cookie on a brand new client (a
        # stolen/cached cookie scenario) against the exact same route.
        replay_client = _new_client(app)
        replay_client.cookies.set("session", old_cookie_value)
        replay_resp = replay_client.get("/settings/api")

        assert replay_resp.status_code == 401, (
            "a session cookie captured BEFORE logout still authenticates "
            f"AFTER logout (got {replay_resp.status_code}, "
            f"body={replay_resp.text[:300]!r})"
        )


# ---------------------------------------------------------------------------
# 3. Two concurrent users, interleaved
# ---------------------------------------------------------------------------


class TestTwoConcurrentUsersInterleaved:
    """End-to-end analogue of test_cross_user_isolation_invariants.py: two
    real users, real per-user encrypted DBs, driven through real HTTP
    calls against the SAME shared ``app``/``db_manager`` singletons that
    every real request shares under uvicorn ``workers=1``.
    """

    def test_sequential_interleave_never_crosses(self, app):
        """A and B alternate settings + collection mutations; after EVERY
        single step, both users' own reads are checked against both what
        they wrote and what the other user wrote, so a leak introduced at
        any point in the sequence is caught at that point (not just at
        the end)."""
        client_a = _new_client(app)
        client_b = _new_client(app)
        user_a = _unique_name("interleave_a")
        user_b = _unique_name("interleave_b")
        _register_and_login(client_a, user_a)
        _register_and_login(client_b, user_b)

        marker_a = f"interleave-a-{uuid.uuid4().hex[:8]}"
        marker_b = f"interleave-b-{uuid.uuid4().hex[:8]}"

        # Step 1: A writes, B has not written yet -> B must still see its
        # own (unrelated) default, never A's marker.
        assert (
            client_a.put(
                "/settings/api/llm.model", json={"value": marker_a}
            ).status_code
            == 200
        )
        b_after_a_write = client_b.get("/settings/api/llm.model").json()
        assert b_after_a_write.get("value") != marker_a, (
            "B observed A's setting write before B ever wrote anything"
        )

        # Step 2: B writes its own marker.
        assert (
            client_b.put(
                "/settings/api/llm.model", json={"value": marker_b}
            ).status_code
            == 200
        )

        # Step 3: both read back — each must see only its own marker.
        a_value = client_a.get("/settings/api/llm.model").json().get("value")
        b_value = client_b.get("/settings/api/llm.model").json().get("value")
        assert a_value == marker_a, f"A's own write did not stick: {a_value!r}"
        assert b_value == marker_b, f"B's own write did not stick: {b_value!r}"

        # Step 4: A creates a collection; B immediately lists collections
        # and must not see it.
        coll_a_name = _unique_name("interleave-coll-a")
        create_a = client_a.post(
            "/library/api/collections", json={"name": coll_a_name}
        )
        assert create_a.status_code == 200, create_a.text
        coll_a_id = create_a.json()["collection"]["id"]

        b_collections = client_b.get("/library/api/collections").json()
        assert coll_a_id not in {
            c["id"] for c in b_collections["collections"]
        }, (
            "B's collection list contains A's collection right after A created it"
        )

        # Step 5: B creates its own collection; A must not see it either.
        coll_b_name = _unique_name("interleave-coll-b")
        create_b = client_b.post(
            "/library/api/collections", json={"name": coll_b_name}
        )
        assert create_b.status_code == 200, create_b.text
        coll_b_id = create_b.json()["collection"]["id"]

        a_collections = client_a.get("/library/api/collections").json()
        a_coll_ids = {c["id"] for c in a_collections["collections"]}
        assert coll_a_id in a_coll_ids, "A lost its own collection"
        assert coll_b_id not in a_coll_ids, (
            "A's collection list contains B's collection"
        )

        # Step 6: cross-user direct-object access on the collection
        # endpoints must 404, not leak/500.
        assert (
            client_a.get(
                f"/library/api/collections/{coll_b_id}/documents"
            ).status_code
            == 404
        )
        assert (
            client_b.get(
                f"/library/api/collections/{coll_a_id}/documents"
            ).status_code
            == 404
        )

    def test_concurrent_http_requests_do_not_cross_users(self, app):
        """A narrower, genuinely-concurrent slice: both users are already
        registered/logged in (sequentially, to keep registration/login
        itself out of the concurrent surface), then BOTH fire one settings
        write at the same moment via a ``threading.Barrier`` and real OS
        threads against two independent TestClients sharing the same
        ``app`` — the actual uvicorn workers=1 + shared-AnyIO-threadpool
        shape. Verification happens after both threads join, sequentially,
        so a failure here is unambiguous (not a race in the assertions
        themselves).
        """
        client_a = _new_client(app)
        client_b = _new_client(app)
        user_a = _unique_name("concurrent_a")
        user_b = _unique_name("concurrent_b")
        _register_and_login(client_a, user_a)
        _register_and_login(client_b, user_b)

        marker_a = f"concurrent-a-{uuid.uuid4().hex[:8]}"
        marker_b = f"concurrent-b-{uuid.uuid4().hex[:8]}"

        barrier = threading.Barrier(2, timeout=30)
        errors: list[BaseException] = []
        responses: dict[str, object] = {}

        def _write(client, marker, key):
            try:
                barrier.wait()
                resp = client.put(
                    "/settings/api/llm.model", json={"value": marker}
                )
                responses[key] = resp
            except BaseException as exc:  # noqa: BLE001 - surfaced below
                errors.append(exc)

        t_a = threading.Thread(target=_write, args=(client_a, marker_a, "a"))
        t_b = threading.Thread(target=_write, args=(client_b, marker_b, "b"))
        t_a.start()
        t_b.start()
        t_a.join(timeout=30)
        t_b.join(timeout=30)

        assert not t_a.is_alive() and not t_b.is_alive(), (
            "concurrent settings writes did not complete within 30s"
        )
        assert not errors, f"concurrent writes raised: {errors!r}"
        assert responses["a"].status_code == 200, responses["a"].text
        assert responses["b"].status_code == 200, responses["b"].text

        a_value = client_a.get("/settings/api/llm.model").json().get("value")
        b_value = client_b.get("/settings/api/llm.model").json().get("value")
        assert a_value == marker_a, (
            f"A's concurrent write was overwritten by B's: got {a_value!r}, "
            f"expected {marker_a!r}"
        )
        assert b_value == marker_b, (
            f"B's concurrent write was overwritten by A's: got {b_value!r}, "
            f"expected {marker_b!r}"
        )


# ---------------------------------------------------------------------------
# 4. Research submission -> queue/history row -> cross-user isolation
# ---------------------------------------------------------------------------


class TestResearchSubmissionQueueHistory:
    """Submit research through the real route, real per-user encrypted DB,
    and real (but deliberately unreachable-provider) background thread.
    Assert the ResearchHistory row exists and is attributed to the
    submitting user via two different read paths (history list + history
    status-by-id), and that a second user gets 404 / an empty list for it.

    Does NOT wait for the research to finish — the row is committed to the
    DB before the HTTP response is returned (the background thread is
    spawned afterward), so existence/attribution are already observable
    at response time. See the module docstring for why the provider/engine
    names are deliberately nonexistent (fast, in-process, network-free
    failure in the background thread we never wait on).
    """

    def test_research_row_attributed_and_isolated_across_users(self, app):
        client_a = _new_client(app)
        client_b = _new_client(app)
        user_a = _unique_name("research_a")
        user_b = _unique_name("research_b")
        _register_and_login(client_a, user_a)
        _register_and_login(client_b, user_b)

        query = f"integration test query {uuid.uuid4().hex[:8]}"
        start_resp = client_a.post(
            "/api/start_research",
            json={
                "query": query,
                "mode": "quick",
                "model_provider": _UNREACHABLE_PROVIDER,
                "model": "no-such-model-xyz",
                "search_engine": _ACCEPTED_ENGINE,
                "iterations": 1,
                "questions_per_iteration": 1,
            },
        )
        assert start_resp.status_code == 200, start_resp.text
        start_body = start_resp.json()
        assert start_body.get("status") == "success", start_body
        research_id = start_body["research_id"]
        assert research_id

        # --- Attribution, via the status-by-id route.
        status_resp = client_a.get(f"/history/status/{research_id}")
        assert status_resp.status_code == 200, status_resp.text
        status_body = status_resp.json()
        assert status_body["id"] == research_id
        assert status_body["query"] == query
        # A genuine lifecycle status, not completion — we never wait for
        # the (deliberately doomed) background thread to reach a terminal
        # state; QUEUED/IN_PROGRESS is expected here, FAILED is also
        # accepted in case the thread already lost the race to fail fast.
        assert status_body["status"] in (
            ResearchStatus.QUEUED,
            ResearchStatus.IN_PROGRESS,
            ResearchStatus.FAILED,
        ), status_body["status"]

        # --- Attribution, via the OTHER read path (the history list) —
        # a different query/route than status-by-id, so this also proves
        # the row isn't visible-by-id-only due to some caching quirk.
        history_resp = client_a.get("/history/api")
        assert history_resp.status_code == 200, history_resp.text
        history_items = history_resp.json().get("items", [])
        matching = [item for item in history_items if item["id"] == research_id]
        assert len(matching) == 1, (
            f"expected exactly one history row for {research_id}, "
            f"found {len(matching)}"
        )
        assert matching[0]["query"] == query

        # --- Cross-user isolation: user B must not be able to see this
        # research by id...
        cross_status = client_b.get(f"/history/status/{research_id}")
        assert cross_status.status_code == 404, (
            f"second user could read first user's research status "
            f"(got {cross_status.status_code}): {cross_status.text[:300]}"
        )

        # ...nor find it in their own history listing.
        b_history = client_b.get("/history/api").json().get("items", [])
        assert research_id not in {item["id"] for item in b_history}, (
            "second user's history listing contains the first user's "
            "research_id"
        )


# ---------------------------------------------------------------------------
# 5. Settings propagation across routers (no stale cache between layers)
# ---------------------------------------------------------------------------


class TestSettingsPropagationAcrossRouters:
    """Write a setting through ``/settings/api/{key}`` (settings.py) and
    read it back through ``/research/api/settings/current-config``
    (api.py) — a different router module that builds its own fresh
    ``SettingsManager`` over its own fresh DB session. If either layer
    cached a settings snapshot across requests, this is where it would
    show up as a stale value on the READING side, which is the side that
    would actually bite a user (research would start with a
    provider/model that no longer matches the Settings page).
    """

    def test_setting_written_via_settings_router_observed_via_research_router(
        self, app
    ):
        client = _new_client(app)
        username = _unique_name("propagation_user")
        _register_and_login(client, username)

        # Baseline: read the value the "other" router sees before we touch
        # anything, so the assertion below proves a CHANGE was observed
        # rather than coincidentally matching a shared default.
        baseline = client.get("/research/api/settings/current-config").json()
        assert baseline.get("success") is True, baseline

        marker = f"propagation-{uuid.uuid4().hex[:8]}"
        assert baseline["config"]["model"] != marker

        put_resp = client.put("/settings/api/llm.model", json={"value": marker})
        assert put_resp.status_code == 200, put_resp.text

        updated = client.get("/research/api/settings/current-config").json()
        assert updated.get("success") is True, updated
        assert updated["config"]["model"] == marker, (
            "settings write via /settings/api was not observed via "
            f"/research/api/settings/current-config: {updated['config']!r}"
        )