Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

35.3 KB
Raw Blame
"""The research execution boundary: owner, terminal state, cancellation.

``run_research_process`` is the seam where an HTTP request (or a queue
tick) becomes a long-running background thread. Under Flask it ran inside
an app context, so ``username``, the settings snapshot, the DB session and
socket emission were all ambient. The FastAPI port has no app context:
every one of those has to be threaded through the call explicitly. This
module covers the four properties that threading has to preserve, driving
the REAL (undecorated) worker via the shared harness in
``tests/web/services/helpers.py``. Nothing here re-implements service
logic; every assertion reads a value ``research_service`` produced.

WHAT IS ALREADY COVERED ELSEWHERE (not repeated here)
-----------------------------------------------------
* the username *gate* (missing/blank ``username`` raises) and the shortest
  post-gate path -- ``tests/security/test_research_service_isolation_fastapi
  .py::TestRunResearchProcessUsernameGate``;
* contextvar propagation into the spawned thread and semaphore accounting
  -- ``test_research_service_start_process.py``;
* the two-stage error *classification* and the ``solution`` hint a
  classified failure persists -- ``test_research_service_error_typing.py``;
* ``ResearchTerminatedException``'s BaseException ancestry as a *class
  property*, and per-strategy cancellation checkpoints --
  ``tests/strategies/test_cancellation_checks.py`` and
  ``tests/advanced_search_system/strategies/
  test_strategy_cancellation_regression.py``;
* ``search.max_results`` / ``search.time_period`` precedence between
  explicit kwargs and the effective snapshot --
  ``test_research_service_search_limit_precedence.py``;
* the emit-then-remove ordering inside ``cleanup_research_resources`` and
  the socket fanout -- ``tests/web/services/test_socketio_asgi_*.py``.

WHAT IS NEW HERE
----------------
1. A *full-run* owner sweep. The isolation module pins two sinks
   (``set_search_context``, ``cleanup_research_resources``) on a run that
   terminates before it starts. Nothing pinned the eight sinks a run that
   actually reaches the strategy touches -- the LLM factory, the search
   factory, ``AdvancedSearchSystem``, the shared research context, every
   ``get_user_db_session`` open, the report storage, the socket owner key
   and the queue processor. Plus a hostile control: an ambient
   ``request_context`` username for a DIFFERENT user must reach none of
   them. That contextvar is the FastAPI port's replacement for
   ``flask_session``, so a sink that reads it instead of the kwarg is
   exactly the regression this port could introduce, and it is invisible
   -- the run succeeds, it just belongs to the wrong person.

2. Terminal state from failure sites the classification tests never
   reach. ``test_research_service_error_typing.py`` only ever fails inside
   ``analyze_topic``. A failure BEFORE the strategy is built, and one
   AFTER it returned (the report save), take different routes to the same
   outer handler; if either one leaves the run without a queued terminal
   status the row stays ``IN_PROGRESS`` forever and the UI spins.

3. Cancellation driven through the REAL ``research_state``. Every
   existing worker-level termination test patches
   ``is_termination_requested`` with a canned side-effect list, which
   proves the worker reacts to a boolean but not that ``cancel_research``
   sets one the worker can see. Here the real flag is set by the real
   ``cancel_research`` from another thread while a strategy is mid-loop,
   and the assertion is that the strategy STOPS -- not merely that a row
   was marked. The fake strategy wraps its callback in
   ``except Exception``, the way real strategies do, so the run also
   proves cancellation is not swallowable in situ.

4. The settings context the run installs is the owner's. Its
   ``username`` attribute is the input to the cross-user identity guard in
   ``config/thread_settings.py`` (pinned from the other side by
   ``tests/security/test_cross_user_isolation_invariants.py::
   TestSettingsContextIdentityGuard``); nothing pinned that the worker
   actually stamps it.

POSITIVE CONTROLS. "The error was recorded" is worthless unless the run
reached the failing code, so every failure class here is paired with a
success run through the same machinery that asserts the terminal SUCCESS
marker instead: ``ResearchHistory.status = COMPLETED`` written on the
row the run itself fetched. ``_run`` also refuses to return if the
strategy was never constructed.
"""

import threading
from contextlib import ExitStack, contextmanager
from dataclasses import dataclass, field
from typing import Any, Callable
from unittest.mock import MagicMock, patch

import pytest
from loguru import logger

from local_deep_research.constants import ResearchStatus
from local_deep_research.settings.manager import SnapshotSettingsContext
from local_deep_research.utilities.request_context import (
    reset_request_user,
    set_request_user,
)
from local_deep_research.web import research_state
from local_deep_research.web.services import research_service
from tests.web.services.helpers import (
    MODULE,
    QUEUE_PROC_MOD,
    RESEARCH_STATE_MOD,
    GLOBALS_MOD,
    THREAD_SETTINGS_MOD,
    _base_run_patches,
    _egress_and_search_patches,
    _get_raw_run_research_process,
    _make_mock_research,
)

# The production progress callback logs at the custom "MILESTONE" level,
# registered by log_utils.init_loguru, which tests do not run.
try:
    logger.level("MILESTONE", no=26)
except (ValueError, TypeError):
    pass

OWNER = "alice_owner"
INTRUDER = "mallory_intruder"


#: A snapshot whose values are all distinguishable, so an assertion that a
#: value arrived at a consumer cannot be satisfied by a default.
def _snapshot():
    return {
        # deliberately NOT one of the openai-compatible providers: that
        # branch rewrites failures ahead of the classification chain.
        "llm.provider": "ollama",
        "llm.model": "owner-private-model",
        "search.tool": "owner-private-engine",
        "search.iterations": 4,
        "search.questions_per_iteration": 6,
        "search.search_strategy": "owner-private-strategy",
    }


#: Drives quick mode straight through to a saved report.
HEALTHY_RESULTS = {
    "findings": [{"content": "A finding", "phase": "Final synthesis"}],
    "formatted_findings": "# Report\n\nA finding",
    "iterations": 1,
    "current_knowledge": "",
}


#: ``apply_environment_overrides_to_snapshot`` runs for real inside the
#: worker (operator env policy is reapplied at dispatch). Clearing the
#: overrides for exactly the keys this module asserts on keeps the run
#: faithful without making the assertions depend on the developer's shell.
_SNAPSHOT_ENV_OVERRIDES = (
    "LDR_LLM_PROVIDER",
    "LDR_LLM_MODEL",
    "LDR_SEARCH_TOOL",
    "LDR_SEARCH_ITERATIONS",
    "LDR_SEARCH_QUESTIONS_PER_ITERATION",
    "LDR_SEARCH_SEARCH_STRATEGY",
)


@pytest.fixture(autouse=True)
def _no_env_override_of_asserted_settings(monkeypatch):
    for name in _SNAPSHOT_ENV_OVERRIDES:
        monkeypatch.delenv(name, raising=False)


@pytest.fixture(autouse=True)
def _clear_worker_egress_context():
    """The worker registers an EgressContext on a thread-local that the
    real ``@thread_cleanup`` decorator (stripped here) would clear."""
    from local_deep_research.security.egress.audit_hook import (
        clear_active_context,
    )

    clear_active_context()
    yield
    clear_active_context()


class _FakeSystem:
    """Stand-in for ``AdvancedSearchSystem``.

    Real enough to hold the progress callback the worker installs, so a
    test can make the "strategy" emit progress and observe what the
    worker's closure does back to it. Never runs an LLM or a search.
    """

    def __init__(self, analyze: Callable[[Callable, str], Any]):
        self._analyze = analyze
        self.progress_callback = None
        self.all_links_of_system = []
        self.closed = False

    def set_progress_callback(self, callback):
        self.progress_callback = callback

    def analyze_topic(self, query):
        assert self.progress_callback is not None, (
            "the worker installed no progress callback, so this strategy "
            "has no way to observe cancellation"
        )
        return self._analyze(self.progress_callback, query)

    def close(self):
        self.closed = True


@dataclass
class RunProbe:
    """Everything the run handed to a user-scoped or terminal-state sink."""

    research_id: int
    research_row: MagicMock
    system_cls: MagicMock
    system: _FakeSystem
    get_llm: MagicMock
    get_search: MagicMock
    storage: MagicMock
    queue_processor: MagicMock
    sio_emit: MagicMock
    cleanup: MagicMock
    handle_termination: MagicMock
    set_search_context: MagicMock
    set_settings_context: MagicMock
    db_opens: list = field(default_factory=list)
    raised: BaseException | None = None

    # -- terminal-state readers -------------------------------------
    @property
    def row_status(self):
        return self.research_row.status

    @property
    def error_update(self):
        calls = self.queue_processor.queue_error_update.call_args_list
        assert len(calls) <= 1, f"more than one terminal update: {calls}"
        return calls[0].kwargs if calls else None

    @property
    def cleanup_final_status(self):
        assert self.cleanup.called, "cleanup_research_resources never ran"
        return self.cleanup.call_args.kwargs.get(
            "final_status", ResearchStatus.COMPLETED
        )

    # -- owner readers ----------------------------------------------
    def owners_seen(self) -> dict:
        """Map sink name -> the username that sink was given."""
        seen = {
            "set_search_context": self.set_search_context.call_args.args[0][
                "username"
            ],
            "settings_context": self.set_settings_context.call_args.args[
                0
            ].username,
            "get_llm": self.get_llm.call_args.kwargs["username"],
            "get_search": self.get_search.call_args.kwargs["username"],
            "AdvancedSearchSystem": self.system_cls.call_args.kwargs[
                "username"
            ],
            "shared_research_context": self.system_cls.call_args.kwargs[
                "research_context"
            ]["username"],
            "report_storage": self.storage.save_report.call_args.kwargs[
                "username"
            ],
            "cleanup_research_resources": self.cleanup.call_args.args[1],
        }
        seen["db_session"] = sorted(set(self.db_opens))
        seen["socket_owner"] = sorted(
            {c.kwargs["owner"] for c in self.sio_emit.call_args_list}
        )
        seen["queue_progress"] = sorted(
            {
                c.args[0]
                for c in self.queue_processor.queue_progress_update.call_args_list
            }
        )
        return seen


def _run(
    *,
    username: str = OWNER,
    research_id: int,
    analyze: Callable | None = None,
    system_side_effect: BaseException | None = None,
    search_side_effect: BaseException | None = None,
    save_report_returns: Any = True,
    queue_error_raises: bool = False,
    ambient_user: str | None = None,
    use_real_research_state: bool = False,
    expect_reached_strategy: bool = True,
    snapshot: dict | None = None,
    run_kwargs: dict | None = None,
) -> RunProbe:
    """Drive the real ``run_research_process`` in quick mode.

    Only genuine external boundaries are stubbed: the LLM factory, the
    search factory, the strategy system, the report storage, the DB
    session, the socket and the queue processor. Everything the
    assertions read is produced by ``research_service`` itself.
    """
    if analyze is None:

        def analyze(_callback, _query):
            return HEALTHY_RESULTS

    research_row = _make_mock_research(
        status=ResearchStatus.IN_PROGRESS, research_meta={}
    )
    mock_session = MagicMock()
    mock_session.query.return_value.filter_by.return_value.first.return_value = research_row

    db_opens: list = []

    @contextmanager
    def _session_ctx(user=None, *_a, **_kw):
        db_opens.append(user)
        yield mock_session

    system = _FakeSystem(analyze)
    system_cls = MagicMock(return_value=system)
    if system_side_effect is not None:
        system_cls.side_effect = system_side_effect

    formatter = MagicMock()
    formatter.format_document_split.return_value = ("answer", [], False)
    formatter.apply_inline_hyperlinks.return_value = "answer"

    storage = MagicMock()
    storage.save_report.return_value = save_report_returns

    get_llm = MagicMock(return_value=MagicMock())
    get_search = MagicMock(return_value=MagicMock())
    if search_side_effect is not None:
        get_search.side_effect = search_side_effect

    patches = _base_run_patches(mock_session)
    patches[f"{MODULE}.get_user_db_session"] = _session_ctx
    patches[f"{MODULE}.get_llm"] = get_llm
    patches[f"{MODULE}.get_search"] = get_search
    patches[f"{MODULE}.AdvancedSearchSystem"] = system_cls
    patches[f"{MODULE}.get_citation_formatter"] = MagicMock(
        return_value=formatter
    )
    if use_real_research_state:
        # Let the genuine flag store decide, so cancel_research and the
        # worker have to agree about a real piece of shared state.
        for target in (
            f"{RESEARCH_STATE_MOD}.is_termination_requested",
            f"{RESEARCH_STATE_MOD}.is_research_active",
            f"{RESEARCH_STATE_MOD}.update_progress_and_check_active",
            f"{GLOBALS_MOD}.is_termination_requested",
        ):
            patches.pop(target)

    queue_processor = patches[f"{QUEUE_PROC_MOD}.queue_processor"]
    if queue_error_raises:
        queue_processor.queue_error_update.side_effect = RuntimeError(
            "queue processor is wedged"
        )

    raised: BaseException | None = None
    ambient_tokens = None
    with ExitStack() as stack:
        for cm in _egress_and_search_patches():
            stack.enter_context(cm)
        stack.enter_context(
            patch(
                "local_deep_research.storage.get_report_storage",
                MagicMock(return_value=storage),
            )
        )
        for target, mock_obj in patches.items():
            stack.enter_context(patch(target, mock_obj))
        if ambient_user is not None:
            ambient_tokens = set_request_user(ambient_user, "sess-x")
        try:
            _get_raw_run_research_process()(
                research_id,
                "a query",
                "quick",
                username=username,
                settings_snapshot=snapshot or _snapshot(),
                **(run_kwargs or {}),
            )
        except BaseException as exc:  # noqa: BLE001 -- recorded, not hidden
            raised = exc
        finally:
            if ambient_tokens is not None:
                reset_request_user(ambient_tokens)

    if expect_reached_strategy:
        assert system_cls.called, (
            "the run never built a strategy -- it failed earlier than the "
            "test intends, so nothing below is being tested"
        )

    return RunProbe(
        research_id=research_id,
        research_row=research_row,
        system_cls=system_cls,
        system=system,
        get_llm=get_llm,
        get_search=get_search,
        storage=storage,
        queue_processor=queue_processor,
        sio_emit=patches[f"{MODULE}._sio_emit"],
        cleanup=patches[f"{MODULE}.cleanup_research_resources"],
        handle_termination=patches[f"{MODULE}.handle_termination"],
        set_search_context=patches[f"{MODULE}.set_search_context"],
        set_settings_context=patches[
            f"{THREAD_SETTINGS_MOD}.set_settings_context"
        ],
        db_opens=db_opens,
        raised=raised,
    )


# =====================================================================
# 1. The run owner reaches every user-scoped sink
# =====================================================================


class TestTheRunOwnerReachesEveryUserScopedSink:
    """Under Flask these sinks read the app/session context. Under FastAPI
    each one has to be handed the username explicitly, and a run that
    reaches the strategy touches eight of them."""

    def test_a_completed_run_marks_the_row_completed(self):
        """POSITIVE CONTROL for this class and the terminal-state class
        below: the harness really carries a run to its successful end, so
        "the sink got the owner" is a statement about a run that happened
        and "no error was queued" is a statement about a run that did not
        fail."""
        probe = _run(research_id=9101)

        assert probe.raised is None, f"the run raised {probe.raised!r}"
        assert probe.row_status == ResearchStatus.COMPLETED, (
            "the success path never wrote the terminal COMPLETED status "
            f"(row is {probe.row_status!r})"
        )
        assert probe.storage.save_report.called, "no report was saved"
        assert probe.error_update is None, (
            "a successful run queued a terminal error update"
        )
        assert probe.cleanup_final_status == ResearchStatus.COMPLETED

    def test_every_user_scoped_sink_receives_the_run_owner(self):
        probe = _run(research_id=9102)
        seen = probe.owners_seen()

        assert seen["db_session"] == [OWNER], (
            f"a DB session was opened for {seen['db_session']!r}"
        )
        assert seen["socket_owner"] == [OWNER], (
            "progress was emitted under the wrong subscription key: "
            f"{seen['socket_owner']!r}"
        )
        assert seen["queue_progress"] == [OWNER]
        for sink in (
            "set_search_context",
            "settings_context",
            "get_llm",
            "get_search",
            "AdvancedSearchSystem",
            "shared_research_context",
            "report_storage",
            "cleanup_research_resources",
        ):
            assert seen[sink] == OWNER, (
                f"{sink} was scoped to {seen[sink]!r}, not the run owner"
            )

    def test_an_ambient_request_user_never_displaces_the_run_owner(self):
        """HOSTILE CONTROL. ``request_context``'s contextvar is the port's
        replacement for ``flask_session['username']``; a sink that falls
        back to it instead of using the kwarg would attribute alice's run
        to whoever's request thread happened to spawn it. The run below
        happens with mallory in the ambient contextvar throughout."""
        probe = _run(research_id=9103, ambient_user=INTRUDER)

        assert probe.row_status == ResearchStatus.COMPLETED, (
            "the run did not complete, so the sweep below proves nothing"
        )
        flat = []
        for value in probe.owners_seen().values():
            flat.extend(value if isinstance(value, list) else [value])
        assert INTRUDER not in flat, (
            f"the ambient request user leaked into a sink: {flat!r}"
        )
        assert set(flat) == {OWNER}


# =====================================================================
# 2. A failure anywhere in the run reaches a recorded terminal state
# =====================================================================


class TestAnyFailureReachesARecordedTerminalState:
    """A run whose worker dies without queueing a terminal status leaves
    ``ResearchHistory.status`` at IN_PROGRESS forever: the UI spins, and
    the partial unique in-progress index blocks the user from retrying.

    ``test_research_service_error_typing.py`` proves this for failures
    raised inside ``analyze_topic``. The two sites below are the failure
    *sites* it never visits -- one before the strategy exists, one after
    it has already returned -- and they take different routes into the
    same outer handler."""

    def test_a_failure_before_the_strategy_is_built_still_records_failed(
        self,
    ):
        probe = _run(
            research_id=9201,
            system_side_effect=RuntimeError("strategy registry unavailable"),
            expect_reached_strategy=False,
        )

        assert probe.raised is None, (
            "the worker let the exception escape the thread; nothing "
            "would ever record a terminal status"
        )
        assert probe.system_cls.called, (
            "the run failed before it even tried to build a strategy"
        )
        assert probe.row_status != ResearchStatus.COMPLETED
        assert probe.error_update is not None, (
            "no terminal status was queued -- the row stays IN_PROGRESS"
        )
        assert probe.error_update["status"] == ResearchStatus.FAILED
        assert probe.error_update["username"] == OWNER
        assert probe.error_update["research_id"] == 9201
        assert probe.error_update["completed_at"], (
            "a terminal row with no completed_at reads as still running"
        )
        assert probe.cleanup_final_status == ResearchStatus.FAILED

    def test_a_failure_after_the_strategy_returned_still_records_failed(
        self,
    ):
        """The report save is the last thing between a finished strategy
        and the COMPLETED write. If it fails, the run must not be left
        looking either finished or running."""
        probe = _run(research_id=9202, save_report_returns=False)

        assert probe.raised is None
        assert probe.system.progress_callback is not None, (
            "the strategy never ran, so this is not the post-search path"
        )
        assert probe.storage.save_report.called
        assert probe.row_status != ResearchStatus.COMPLETED, (
            "a run whose report was never stored was marked COMPLETED"
        )
        assert probe.error_update is not None
        assert probe.error_update["status"] == ResearchStatus.FAILED
        assert probe.cleanup_final_status == ResearchStatus.FAILED

    def test_a_broken_error_handler_does_not_escape_the_worker(self):
        """DEFECT PROBE, recorded rather than asserted away.

        If the queue processor itself raises while recording the failure,
        the inner ``except Exception: logger.exception("Error in error
        handler")`` swallows it and the run proceeds to cleanup. The
        worker stays alive and the socket/room teardown still happens --
        which is what this test pins -- but NOTHING has written a terminal
        status to ``ResearchHistory``, so the row is left IN_PROGRESS
        permanently. See the module report; asserting the current
        behaviour here is a description of that hole, not an endorsement.
        """
        probe = _run(
            research_id=9203,
            system_side_effect=RuntimeError("boom"),
            queue_error_raises=True,
            expect_reached_strategy=False,
        )

        assert probe.raised is None, (
            "a wedged queue processor took the whole worker thread down"
        )
        assert probe.queue_processor.queue_error_update.called
        assert probe.cleanup_final_status == ResearchStatus.FAILED, (
            "cleanup must still report FAILED, not a spurious completion"
        )
        # The hole: no terminal status ever landed on the row.
        assert probe.row_status == ResearchStatus.IN_PROGRESS


# =====================================================================
# 3. Cancellation stops the strategy, not just the row
# =====================================================================


@pytest.fixture
def _real_research_state():
    """Register/deregister a real active-research entry.

    ``cleanup_research_resources`` (which would normally call
    ``cleanup_research``) is stubbed by the harness, so the entry has to
    be removed here or it leaks into every later test in the process.
    """
    registered: list = []

    def register(research_id, username):
        research_state.set_active_research(
            research_id,
            {
                "thread": None,
                "progress": 0,
                "status": ResearchStatus.IN_PROGRESS,
                "log": [],
                "settings": {"username": username},
            },
        )
        registered.append(research_id)

    yield register
    for research_id in registered:
        research_state.cleanup_research(research_id)


def _looping_strategy(steps: list, entered=None, released=None, n=4):
    """A strategy that emits progress in a loop and SWALLOWS exceptions
    from its own callback -- the shape real strategies have.

    ``steps`` records what actually executed, so a test can tell "the
    loop stopped" from "the loop finished and something else complained".
    """

    def analyze(progress_callback, _query):
        for i in range(n):
            steps.append(f"work-{i}")
            # Hand off at the START of the SECOND iteration, so the run
            # has already driven one full progress round-trip through the
            # worker's closure before the cancel lands. "The loop stopped"
            # then means it stopped, not that it never got going.
            if i == 1 and entered is not None:
                entered.set()
                assert released.wait(timeout=30), (
                    "the cancelling thread never released the strategy"
                )
            try:
                progress_callback(
                    f"iteration {i}", 10 + i, {"phase": "iteration"}
                )
            except Exception:  # noqa: BLE001 -- deliberate: see docstring
                steps.append(f"swallowed-{i}")
        steps.append("finished")
        return HEALTHY_RESULTS

    return analyze


class TestCancellationStopsTheStrategyNotJustTheRow:
    """Driven through the REAL ``research_state`` flag store and the REAL
    ``cancel_research``. Every other worker-level termination test hands
    the worker a canned ``is_termination_requested`` side-effect list,
    which shows the worker reacts to a boolean but not that the cancel
    entry point sets one it can see."""

    def test_an_uncancelled_run_completes_every_iteration(
        self, _real_research_state
    ):
        """POSITIVE CONTROL. Without it, "the strategy stopped" below is
        equally satisfied by a strategy that never ran at all."""
        _real_research_state(9301, OWNER)
        steps: list = []

        probe = _run(
            research_id=9301,
            analyze=_looping_strategy(steps),
            use_real_research_state=True,
        )

        assert steps == [
            "work-0",
            "work-1",
            "work-2",
            "work-3",
            "finished",
        ], f"the uncancelled strategy did not run to completion: {steps}"
        assert probe.row_status == ResearchStatus.COMPLETED
        assert not probe.handle_termination.called

    def test_cancelling_mid_strategy_halts_the_loop(self, _real_research_state):
        """The worker runs in its own thread, as it does in production;
        the cancel arrives from another thread, as it does from the HTTP
        route. Handoff is by Event, so the interleaving is fixed and no
        wall-clock waiting is involved."""
        _real_research_state(9302, OWNER)
        steps: list = []
        entered = threading.Event()
        released = threading.Event()
        result: dict = {}

        def worker():
            result["probe"] = _run(
                research_id=9302,
                analyze=_looping_strategy(steps, entered, released),
                use_real_research_state=True,
            )

        thread = threading.Thread(target=worker, daemon=True)
        thread.start()
        try:
            assert entered.wait(timeout=30), "the strategy never started"
            assert steps == ["work-0", "work-1"], (
                "the strategy is not at the handoff point, so the cancel "
                f"below is not landing mid-loop: {steps}"
            )

            cancelled = research_service.cancel_research(9302, OWNER)
            assert cancelled is True, (
                "cancel_research refused an active research"
            )
            assert research_state.is_termination_requested(9302), (
                "cancel_research set no flag the worker could observe"
            )
        finally:
            released.set()
            thread.join(timeout=30)
        assert not thread.is_alive(), "the worker thread never finished"

        probe = result["probe"]
        # One full iteration ran, the second was cut off at its progress
        # callback, and iterations 3-4 never happened. No "swallowed-1"
        # either: the strategy's own ``except Exception`` could not catch
        # it -- the whole point of the BaseException choice.
        assert steps == ["work-0", "work-1"], (
            "cancellation did not stop the strategy in place; the loop "
            f"continued: {steps}"
        )
        assert "finished" not in steps

        # ...and the run ended as a termination, not as a failure.
        assert probe.raised is None
        # cancel_research makes one call, the worker's progress callback
        # the other; both must name the run and its owner.
        assert probe.handle_termination.called
        assert all(
            call.args[:2] == (9302, OWNER)
            for call in probe.handle_termination.call_args_list
        ), probe.handle_termination.call_args_list
        assert probe.error_update is None, (
            "a user cancellation was recorded as a research FAILURE"
        )
        assert probe.row_status != ResearchStatus.COMPLETED, (
            "a cancelled run was marked COMPLETED"
        )
        assert not probe.storage.save_report.called, (
            "a cancelled run still wrote a report"
        )


# =====================================================================
# 4. Settings resolve from the owner's snapshot
# =====================================================================


class TestSettingsResolveFromTheOwnersSnapshot:
    """No app context means no ambient ``SettingsManager``: the run gets a
    snapshot and must resolve everything from it."""

    def test_the_installed_settings_context_is_stamped_with_the_owner(self):
        """``config/thread_settings.py`` refuses to serve a thread-local
        settings context whose ``username`` differs from the current
        request's (pinned from the consumer side in
        ``tests/security/test_cross_user_isolation_invariants.py``). That
        guard is only as good as the stamp the worker puts on."""
        probe = _run(research_id=9401)

        ctx = probe.set_settings_context.call_args.args[0]
        assert isinstance(ctx, SnapshotSettingsContext)
        assert ctx.username == OWNER, (
            "the run installed an unstamped settings context; the "
            "cross-user identity guard has nothing to compare against"
        )
        assert ctx.get_setting("llm.model") == "owner-private-model"

    def test_unspecified_run_parameters_come_from_the_owners_snapshot(self):
        """No ``model`` / ``search_engine`` / ``strategy`` kwargs are
        passed, so every one of these had to be resolved from the
        snapshot the caller handed the run -- not from a process-wide
        settings singleton."""
        probe = _run(research_id=9402)

        assert probe.get_llm.call_args.kwargs["model_name"] == (
            "owner-private-model"
        )
        assert probe.get_llm.call_args.kwargs["provider"] == "ollama"
        assert probe.get_search.call_args.kwargs["search_tool"] == (
            "owner-private-engine"
        )
        system_kwargs = probe.system_cls.call_args.kwargs
        assert system_kwargs["strategy_name"] == "owner-private-strategy"
        assert system_kwargs["max_iterations"] == 4
        assert system_kwargs["questions_per_iteration"] == 6
        assert (
            system_kwargs["settings_snapshot"]["llm.model"]
            == "owner-private-model"
        )

    def test_a_foreign_snapshot_value_never_wins(self):
        """NEGATIVE CONTROL for the test above: change the snapshot and
        every consumer changes with it, so the assertions there are
        reading the snapshot rather than a coincidence of defaults."""
        other = dict(_snapshot())
        other["llm.model"] = "second-users-model"
        other["search.tool"] = "second-users-engine"
        other["search.iterations"] = 11

        probe = _run(research_id=9403, snapshot=other)

        assert probe.get_llm.call_args.kwargs["model_name"] == (
            "second-users-model"
        )
        assert probe.get_search.call_args.kwargs["search_tool"] == (
            "second-users-engine"
        )
        assert probe.system_cls.call_args.kwargs["max_iterations"] == 11


# =====================================================================
# 5. A follow-up's delegate strategy reaches the strategy system
# =====================================================================


class TestTheFollowUpDelegateReachesTheStrategySystem:
    """``AdvancedSearchSystem`` picks a follow-up's delegate from
    ``research_context["delegate_strategy"]``. The chat router writes that
    key; the follow-up router passes the user's strategy as the
    ``delegate_strategy=`` keyword instead, and the run has to carry it
    into the context or the user's choice is lost (#7565)."""

    def test_the_delegate_keyword_lands_in_the_strategys_context(self):
        probe = _run(
            research_id=9501,
            run_kwargs={
                "delegate_strategy": "focused-iteration",
                "research_context": {"past_findings": "prior"},
            },
        )

        context = probe.system_cls.call_args.kwargs["research_context"]
        assert context["delegate_strategy"] == "focused-iteration"
        assert context["past_findings"] == "prior"

    def test_the_keyword_wins_over_a_delegate_already_in_the_context(self):
        probe = _run(
            research_id=9502,
            run_kwargs={
                "delegate_strategy": "focused-iteration",
                "research_context": {"delegate_strategy": "source-based"},
            },
        )

        context = probe.system_cls.call_args.kwargs["research_context"]
        assert context["delegate_strategy"] == "focused-iteration"

    def test_a_context_delegate_is_kept_when_no_keyword_is_passed(self):
        """The chat router's path: the delegate arrives in the context."""
        probe = _run(
            research_id=9503,
            run_kwargs={"research_context": {"delegate_strategy": "iterative"}},
        )

        context = probe.system_cls.call_args.kwargs["research_context"]
        assert context["delegate_strategy"] == "iterative"


@pytest.mark.parametrize(
    "run_kwargs",
    [
        {"strategy": "langgraph-agent"},
        {"strategy": "agentic"},
        {
            "strategy": "enhanced-contextual-followup",
            "delegate_strategy": "langgraph-agent",
        },
        {
            "strategy": "contextual-followup",
            "research_context": {"delegate_strategy": "langgraph-agent"},
        },
    ],
)
def test_agent_startup_carries_failed_primary_into_strategy_availability(
    run_kwargs,
):
    from local_deep_research.web_search_engines.engine_availability import (
        SearchEngineInitializationError,
        SearchFailure,
    )

    failure = SearchFailure("Search request timed out", 60)
    probe = _run(
        research_id=9510,
        run_kwargs=run_kwargs,
        search_side_effect=SearchEngineInitializationError(failure),
    )
    assert probe.raised is None
    assert probe.row_status == ResearchStatus.COMPLETED
    assert probe.get_search.call_args.kwargs["raise_on_error"] is True
    system_args = probe.system_cls.call_args.kwargs
    assert system_args["search"] is None
    assert system_args["initial_search_failure"] == failure
    assert probe.error_update is None