Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

26.4 KB
Raw Blame
"""Connection-lifecycle semantics of the ASGI Socket.IO layer.

Flask-SocketIO's ``SocketIOService`` became python-socketio's ``AsyncServer``
mounted at ``/ws``. Flask-SocketIO gave us rooms, a room registry that
survived across a reconnect only because the client re-joined, and a
``message_queue=`` knob that made multi-process deployment a config change.
The replacement keeps *all* of its routing state in module-level dicts
(``_subscriptions``, ``_sid_users``, ``_sid_sessions``, ``_main_loop``) plus
an in-process ``AsyncManager``. That is a deliberate trade, and it has
consequences that only show up at connection boundaries: a reconnect, a tab
closing mid-fanout, a run finishing, a second uvicorn worker.

This file covers the four of those that nothing else reaches:

1. **Reconnection and resubscription.** A reconnect is a *new sid*. Nothing
   server-side carries the old sid's subscriptions across, so the JS client's
   ``socket.on('connect', ...)`` re-subscribe is load-bearing, and the window
   between the drop and the re-subscribe delivers to nobody rather than
   falling back to a broadcast.

2. **The reconnect window itself**, where the old sid has not been reaped by
   engine.io's ping timeout yet, so a dead sid and a live sid sit in the same
   subscription set and the dead one's emit raises.

3. **Teardown landing mid-fanout** — a tab closing, or a run's own
   ``remove_subscriptions_for_research`` arriving, while the terminal
   ``research_progress`` message is still being delivered to the user's other
   tabs. ``research_service`` schedules those two back to back
   (``_sio_emit(...)`` then ``_sio_remove(...)``, both onto the same loop), so
   this interleaving is the normal completion path, not a corner case.

4. **Why ``workers=1`` is mandatory.** Two other files already pin the literal
   ``workers=1`` in ``web/app.py`` by AST. Neither pins the *reason*, so
   nothing tells the person who eventually adds a Redis manager that they may
   now raise it -- or tells the person who raises it first what breaks.

Deliberately NOT covered here (already pinned elsewhere, do not duplicate):

* mount path / CORS ``None``-vs-``[]`` / ``(username, research_id)`` keying /
  emit fail-closed / ``_async_emit`` transport-failure containment
  -- ``test_socketio_asgi_contracts.py``
* handshake cookie crypto and the connect auth gate
  -- ``test_socketio_handshake_auth.py``, ``test_socketio_connect_gate.py``
* revoked-session gating at connect and on subscribe/unsubscribe
  -- ``test_socket_connect_session_gate.py``,
  ``tests/security/test_socket_ownership_edges_fastapi.py``,
  ``tests/chat/test_chat_socket_events.py``
* cross-*user* delivery scoping -- ``test_subscription_owner_scoping.py``,
  ``test_socketio_asgi_user_scoping.py``
* the ``get_active_research_snapshot`` replay on subscribe
  -- ``tests/chat/test_chat_socket_events.py``

Everything here drives the real coroutines to completion under
``asyncio.run``; no wall-clock sleeps and no polling.
"""

import asyncio
import importlib.util
import sys
from unittest.mock import patch

import pytest
from socketio.async_pubsub_manager import AsyncPubSubManager

from local_deep_research.web.auth.session_manager import session_manager
from local_deep_research.web.services import socketio_asgi as sio_mod

USER = "ws_semantics_user"
OTHER = "ws_semantics_other"
RID = "research-abc"


@pytest.fixture
def socket_state(monkeypatch):
    """Isolated socket state plus a recording ``sio.emit``.

    A fresh ``asyncio.Lock`` per test: the real one is created once at
    lifespan startup, and a lock left bound to a previous test's closed loop
    raises on the next acquire.
    """
    monkeypatch.setattr(sio_mod, "_lock", asyncio.Lock())
    monkeypatch.setattr(sio_mod, "_subscriptions", {})
    monkeypatch.setattr(sio_mod, "_sid_users", {})
    monkeypatch.setattr(sio_mod, "_sid_sessions", {})

    delivered = []

    async def _capture(event, data, room=None):
        delivered.append((room, event, data))

    monkeypatch.setattr(sio_mod.sio, "emit", _capture)
    return delivered


def _seed(username, research_id, sid, session_id=None):
    """Register ``sid`` as an authenticated subscriber, as connect+subscribe
    would have left it."""
    sio_mod._sid_users[sid] = username
    if session_id is not None:
        sio_mod._sid_sessions[sid] = session_id
    sio_mod._subscriptions.setdefault((username, research_id), set()).add(sid)


def _fanout(research_id=RID, owner=USER, payload=None):
    """Run one real ``_async_emit_to_subscribers`` fanout to completion."""
    asyncio.run(
        sio_mod._async_emit_to_subscribers(
            "research_progress",
            research_id,
            {"progress": 50} if payload is None else payload,
            owner,
        )
    )


def _rooms(delivered):
    return [room for room, _, _ in delivered]


# ---------------------------------------------------------------------------
# 1. Reconnection and resubscription
# ---------------------------------------------------------------------------


@pytest.mark.real_session_check
class TestReconnectMustResubscribe:
    """A reconnect is a brand-new sid with no inherited subscriptions.

    Under Flask-SocketIO the same was true, but the failure mode was visible:
    ``join_room`` was explicit. Here the subscription set is keyed by sid and
    the disconnect handler purges it, so after a reconnect the server has no
    memory of what the socket was watching. The only thing that restores
    delivery is the client re-issuing ``subscribe_to_research`` from its
    ``connect`` handler.

    The important half is the *negative*: during the gap the fanout must find
    no bucket and deliver nothing. It must never fall back to a roomless
    ``sio.emit(event, data)``, which would hand one user's research progress
    to every connected client.
    """

    def _connect(self, sid, session_id, username=USER):
        """Drive the real ``connect`` coroutine with a live server-side
        session, as a reconnecting browser would.

        ``is_user_connected`` is forced True so the lazy-DB-open branch (which
        has its own tests) stays out of the way.
        """
        with (
            patch.object(
                sio_mod,
                "_decode_session_cookie",
                return_value={
                    "username": username,
                    "session_id": session_id,
                },
            ),
            patch(
                "local_deep_research.database.encrypted_db.db_manager"
                ".is_user_connected",
                return_value=True,
            ),
        ):
            return asyncio.run(
                sio_mod.connect(sid, {"HTTP_COOKIE": "session=whatever"})
            )

    def test_the_gap_between_disconnect_and_resubscribe_reaches_nobody(
        self, socket_state
    ):
        """The full round trip, with the real connect/disconnect coroutines.

        Subscribed -> transport drops -> reconnects on a new sid -> a progress
        event fires before the client has re-subscribed -> it reaches nobody
        and is not broadcast -> the client re-subscribes -> delivery resumes
        on the new sid only.
        """
        token = session_manager.create_session(USER, remember_me=False)
        try:
            _seed(USER, RID, "sid-old", session_id=token)
            _fanout()
            assert _rooms(socket_state) == ["sid-old"], (
                "premise: the original socket was receiving"
            )
            socket_state.clear()

            # Transport drops. engine.io fires the disconnect handler.
            asyncio.run(sio_mod.disconnect("sid-old"))
            assert sio_mod._subscriptions == {}, (
                "disconnect left the old sid's subscription behind"
            )

            # The browser reconnects: new handshake, new sid, same cookie.
            assert self._connect("sid-new", token) is True
            assert sio_mod._sid_users == {"sid-new": USER}

            # A progress event fires in the window before the client's
            # connect handler has re-subscribed.
            _fanout(payload={"progress": 60})
            assert socket_state == [], (
                "an event in the reconnect window was delivered to somebody; "
                "the only two ways that happens are a resurrected sid or a "
                "roomless broadcast, and the second leaks to every user"
            )

            # The client re-subscribes (socket.js does this from its
            # 'connect' handler). Delivery resumes -- on the new sid only.
            with patch.object(
                sio_mod, "_user_owns_research", return_value=True
            ):
                with patch.object(
                    sio_mod, "get_active_research_snapshot", return_value=None
                ):
                    asyncio.run(
                        sio_mod.on_subscribe("sid-new", {"research_id": RID})
                    )

            assert sio_mod._subscriptions == {(USER, RID): {"sid-new"}}
            socket_state.clear()
            _fanout(payload={"progress": 70})
            assert _rooms(socket_state) == ["sid-new"]
        finally:
            session_manager.destroy_session(token)

    def test_a_reconnect_does_not_inherit_the_previous_sids_subscriptions(
        self, socket_state
    ):
        """Same user, same session, same research -- still nothing inherited.

        Pins that the carry-over is keyed by sid and nothing else. If someone
        ever "helpfully" re-keys ``_subscriptions`` by username or by session
        so a reconnect resumes automatically, the socket would start receiving
        a run it never asked for on *this* connection, and the ownership check
        in ``on_subscribe`` -- the only place ownership is ever verified --
        would be bypassed for it.
        """
        token = session_manager.create_session(USER, remember_me=False)
        try:
            _seed(USER, RID, "sid-old", session_id=token)
            asyncio.run(sio_mod.disconnect("sid-old"))

            assert self._connect("sid-new", token) is True

            assert sio_mod._subscriptions == {}, (
                "connect() restored subscriptions for the reconnecting "
                "session; subscriptions must only ever be created by "
                "on_subscribe, which is where ownership is checked"
            )
            _fanout()
            assert socket_state == []
        finally:
            session_manager.destroy_session(token)


class TestReconnectWindowWithAStaleSid:
    """Both sids are subscribed at once, briefly.

    A dropped WebSocket is not noticed instantly: the server holds
    ``ping_timeout=20`` / ``ping_interval=5``, so up to ~25s can pass before
    the disconnect handler runs and purges the old sid. socket.io's client
    reconnects after ``reconnectionDelay: 1000``, i.e. long before that. So
    for most of that window the subscription set holds a *dead* sid alongside
    the freshly reconnected one, and emitting to the dead one raises.
    """

    def test_a_dead_sid_does_not_starve_the_reconnected_one(
        self, socket_state, monkeypatch
    ):
        """The per-sid try/except around ``sio.emit`` is what makes this safe.

        ``_subscriptions`` holds a plain ``set``, so iteration order is not
        defined -- this must hold whichever sid the fanout reaches first, and
        it does: each delivery is independently guarded. Without the guard the
        exception escapes the coroutine in *both* orders (it is not a
        50/50 flake), which is what makes this a real control.
        """
        delivered = []

        async def _emit(event, data, room=None):
            if room == "sid-dead":
                raise ConnectionError("client is gone")
            delivered.append(room)

        monkeypatch.setattr(sio_mod.sio, "emit", _emit)

        _seed(USER, RID, "sid-dead")
        _seed(USER, RID, "sid-reconnected")

        _fanout()

        assert delivered == ["sid-reconnected"], (
            "a stale socket left over from the reconnect window swallowed "
            "the delivery to the socket that actually replaced it"
        )

    def test_the_stale_sid_is_purged_when_the_timeout_finally_fires(
        self, socket_state
    ):
        """And the live sid survives that purge.

        ``disconnect`` walks *every* subscription bucket discarding the sid
        and deletes buckets that empty. The bucket here must not be deleted:
        the reconnected socket is still in it. This is the same code path as
        two tabs on one run, one of which closes.
        """
        _seed(USER, RID, "sid-dead")
        _seed(USER, RID, "sid-reconnected")

        asyncio.run(sio_mod.disconnect("sid-dead"))

        assert sio_mod._subscriptions == {(USER, RID): {"sid-reconnected"}}, (
            "one socket disconnecting tore down a sibling's subscription to "
            "the same run"
        )
        _fanout()
        assert _rooms(socket_state) == ["sid-reconnected"]


# ---------------------------------------------------------------------------
# 2. Teardown landing in the middle of a fanout
# ---------------------------------------------------------------------------


class TestTeardownDuringTheTerminalFanout:
    """What subscribers get when a research completes or errors.

    ``research_service._cleanup_research_resources`` ends every run --
    COMPLETED, FAILED or SUSPENDED -- with two calls back to back::

        _sio_emit("research_progress", research_id, final_message, owner=...)
        _sio_remove(research_id, username)

    Both are sync wrappers that schedule a coroutine onto the one uvicorn
    loop, so the cleanup coroutine is queued while the terminal message is
    still being fanned out to the user's tabs. On top of that, a user watching
    a run finish very often closes the tab at that exact moment, firing
    ``disconnect`` into the same window.

    ``_async_emit_to_subscribers`` copies the subscription set under the lock
    and then releases the lock before delivering. These tests pin what that
    copy buys.
    """

    def test_a_tab_closing_mid_fanout_does_not_abort_the_terminal_message(
        self, socket_state, monkeypatch
    ):
        """The user's other tabs must still be told the run ended.

        ``disconnect`` mutates the subscription set in place
        (``sids.discard(sid)``). The fanout has already released the lock, so
        a disconnect *can* land between two deliveries. If the fanout were
        iterating the live set rather than a copy, that in-place discard
        raises ``RuntimeError: Set changed size during iteration`` out of the
        coroutine, and every subscriber after the closing tab -- including the
        one still watching -- is left on a progress bar frozen at 99%.
        """
        delivered = []

        async def _emit(event, data, room=None):
            delivered.append(room)
            if len(delivered) == 1:
                # The *other* tab closes while this delivery is in flight.
                other = {"sid-tab-1", "sid-tab-2"} - {room}
                await sio_mod.disconnect(other.pop())

        monkeypatch.setattr(sio_mod.sio, "emit", _emit)

        _seed(USER, RID, "sid-tab-1")
        _seed(USER, RID, "sid-tab-2")

        _fanout(payload={"status": "completed", "progress": 100})

        assert sorted(delivered) == ["sid-tab-1", "sid-tab-2"], (
            "the terminal message stopped at the tab that closed; the "
            f"remaining subscriber never learned the run ended: {delivered}"
        )

    def test_the_runs_own_cleanup_cannot_cancel_the_message_announcing_it(
        self, socket_state, monkeypatch
    ):
        """``_sio_remove`` is scheduled one line after ``_sio_emit``.

        Modelled at the point where it actually bites: the cleanup coroutine
        runs while the terminal fanout is mid-flight. Every subscriber must
        still receive the final status, and the bucket must be gone
        afterwards. Both halves matter -- delivering but leaking the bucket,
        or reaping the bucket but dropping the message, are each a bug, and
        only asserting them together pins the ordering.
        """
        delivered = []

        async def _emit(event, data, room=None):
            delivered.append((room, data))
            if len(delivered) == 1:
                # research_service's very next line, landing early.
                await sio_mod._async_remove_subscriptions(RID, USER)

        monkeypatch.setattr(sio_mod.sio, "emit", _emit)

        _seed(USER, RID, "sid-tab-1")
        _seed(USER, RID, "sid-tab-2")

        final = {"status": "failed", "message": "Research was failed"}
        _fanout(payload=final)

        assert sorted(room for room, _ in delivered) == [
            "sid-tab-1",
            "sid-tab-2",
        ], (
            "the completion cleanup cancelled delivery of the very message "
            f"that reports the completion: {delivered}"
        )
        assert all(data == final for _, data in delivered)
        assert sio_mod._subscriptions == {}, (
            "the finished run's subscription bucket outlived the run"
        )

    def test_cleanup_leaves_the_socket_connected_for_the_next_run(
        self, socket_state
    ):
        """A run ending is not a logout.

        ``_async_remove_subscriptions`` pops one ``(owner, research_id)``
        bucket and nothing else. The identity maps must survive, or the user
        would have to reconnect before starting another research -- and
        ``on_subscribe`` would reject them, since it reads identity from
        ``_sid_users``.
        """
        _seed(USER, RID, "sid-tab-1", session_id="sess-1")
        _seed(USER, "research-other", "sid-tab-1", session_id="sess-1")

        asyncio.run(sio_mod._async_remove_subscriptions(RID, USER))

        assert sio_mod._sid_users == {"sid-tab-1": USER}
        assert sio_mod._sid_sessions == {"sid-tab-1": "sess-1"}
        assert sio_mod._subscriptions == {
            (USER, "research-other"): {"sid-tab-1"}
        }, "cleanup for one run tore down the socket's other subscription"

    def test_a_late_event_after_cleanup_reaches_nobody_not_everybody(
        self, socket_state
    ):
        """Emits genuinely do arrive after cleanup.

        The log-queue processor and the search threads drain asynchronously,
        so a stray ``research_progress`` for a finished run is routine. With
        the bucket gone the lookup misses, and the miss must be a drop --
        ``_async_emit_to_subscribers`` has no ``else`` that broadcasts, and a
        second connected user proves it stays that way.
        """
        _seed(USER, RID, "sid-owner")
        _seed(OTHER, "another-run", "sid-bystander")

        asyncio.run(sio_mod._async_remove_subscriptions(RID, USER))
        _fanout(payload={"progress": 100})

        assert socket_state == [], (
            "a post-cleanup event was delivered; the only reachable "
            "recipient set is 'everyone', which is the cross-user leak"
        )


# ---------------------------------------------------------------------------
# 3. Why workers=1 is mandatory
# ---------------------------------------------------------------------------


def _load_second_instance():
    """Load a second, independent instance of ``socketio_asgi``.

    This stands in for a second uvicorn worker. It is the same source file
    executed a second time, so it is an honest model of what a second *process*
    gets: its own module globals and its own ``AsyncServer``. It is not a
    re-implementation -- the delivery path exercised below is the production
    coroutine, reached through the second instance.

    The name keeps the real package prefix so the module's relative imports
    (``from ...constants import ...``) still resolve.
    """
    name = sio_mod.__name__ + "_second_worker"
    spec = importlib.util.spec_from_file_location(name, sio_mod.__file__)
    module = importlib.util.module_from_spec(spec)
    sys.modules[name] = module
    try:
        spec.loader.exec_module(module)
    except BaseException:
        sys.modules.pop(name, None)
        raise
    return name, module


class TestSingleWorkerConstraint:
    """``web/app.py`` passes ``workers=1`` with the comment "Required for
    Socket.IO without Redis message queue".

    ``tests/web/test_concurrency_under_single_worker.py`` and
    ``tests/web/test_lifespan_startup_shutdown.py`` both pin that literal by
    AST. Neither pins the premise it rests on, which leaves two ways to get it
    wrong:

    * someone raises ``workers``, the AST tests fail, and the failure message
      asserts a reason nobody has verified is still true;
    * someone *adds* a Redis manager -- which is exactly what would make
      raising ``workers`` legitimate -- and nothing tells them the constraint
      can now be lifted, so the single worker stays forever.

    These tests pin the causal chain instead of the number.
    """

    def test_the_server_has_no_message_queue_so_state_cannot_span_processes(
        self,
    ):
        """The premise, asserted against the live server object.

        python-socketio fans an emit out across processes only through a
        ``AsyncPubSubManager`` subclass (``AsyncRedisManager``,
        ``AsyncKombuManager``, ...) passed as ``client_manager``. The default
        ``AsyncManager`` keeps its room registry in this process's memory.
        ``socketio_asgi`` passes no ``client_manager``, so there is no medium
        by which worker 2 could learn about worker 1's sockets.
        """
        assert not isinstance(sio_mod.sio.manager, AsyncPubSubManager), (
            "a Socket.IO message queue manager "
            f"({type(sio_mod.sio.manager).__name__}) was added. Emits can now "
            "cross processes, so the `workers=1` in web/app.py may no longer "
            "be required -- but `_subscriptions`, `_sid_users`, "
            "`_sid_sessions` and `_main_loop` are still plain module globals "
            "and do NOT cross processes, so a multi-worker deploy still "
            "misroutes unless those move too. Decide that explicitly and "
            "update this test with the reasoning."
        )

    def test_the_routing_state_that_decides_delivery_is_process_local(self):
        """The consequence, executed.

        A subscriber registered in instance A is invisible to instance B, so
        an emit issued by B for that research reaches nobody -- silently, with
        no error anywhere. Under ``--workers 2`` that is a progress bar that
        freezes for roughly half of all users, depending on which worker
        accepted the WebSocket versus which one ran the research thread.

        Instance A doubles as the positive control in the same test: the
        identical fanout, same seeded state, delivers through A. So "reaches
        nobody" cannot be an artefact of a mis-seeded harness.
        """
        name, worker2 = _load_second_instance()
        try:
            assert worker2.__file__ == sio_mod.__file__, (
                "premise: the second instance must be the same source file"
            )
            assert worker2.sio is not sio_mod.sio
            assert worker2.sio.manager is not sio_mod.sio.manager
            assert worker2._subscriptions == {}, (
                "a freshly loaded instance already sees subscriptions, so "
                "this is not modelling a separate process"
            )

            a_delivered = []
            b_delivered = []

            async def _capture_a(event, data, room=None):
                a_delivered.append(room)

            async def _capture_b(event, data, room=None):
                b_delivered.append(room)

            with (
                patch.object(sio_mod, "_lock", asyncio.Lock()),
                patch.object(sio_mod, "_subscriptions", {}),
                patch.object(sio_mod.sio, "emit", _capture_a),
                patch.object(worker2, "_lock", asyncio.Lock()),
                patch.object(worker2.sio, "emit", _capture_b),
            ):
                # Worker 1 accepted the WebSocket and the subscribe.
                sio_mod._subscriptions[(USER, RID)] = {"sid-on-worker-1"}

                # Worker 2 is running the research thread and emits progress.
                asyncio.run(
                    worker2._async_emit_to_subscribers(
                        "research_progress", RID, {"progress": 50}, USER
                    )
                )
                # Positive control: the same call on worker 1 does deliver.
                asyncio.run(
                    sio_mod._async_emit_to_subscribers(
                        "research_progress", RID, {"progress": 50}, USER
                    )
                )

            assert a_delivered == ["sid-on-worker-1"], (
                "control failed: the subscriber was not reachable even from "
                "the instance that registered it, so the assertion below "
                "would prove nothing"
            )
            assert b_delivered == [], (
                "a second worker reached worker 1's subscriber; if that is "
                "genuinely possible now, the module no longer relies on "
                "process-local state and workers=1 can be revisited"
            )
        finally:
            sys.modules.pop(name, None)

    def test_the_captured_event_loop_is_process_local_too(self):
        """The other half of the same constraint, and the quieter half.

        Every emit from a background thread goes through ``_get_main_loop()``,
        which returns the loop captured by ``set_main_loop`` during *this*
        process's lifespan startup. A second worker captures its own. So even
        with a shared subscription store, a worker could only ever dispatch
        into its own loop -- fixing the state sharing alone would not be
        enough, which is why the fix is "add a message queue", not "move the
        dicts to Redis".
        """
        name, worker2 = _load_second_instance()
        saved_loop = sio_mod._main_loop
        try:
            loop = asyncio.new_event_loop()
            try:
                sio_mod.set_main_loop(loop)
                assert sio_mod._get_main_loop() is loop
                assert worker2._main_loop is None, (
                    "set_main_loop reached beyond its own module instance"
                )
                # With no loop of its own, the second worker's sync emit
                # wrapper fails closed rather than dispatching anywhere.
                assert (
                    worker2.emit_to_subscribers(
                        "research_progress", RID, {}, owner=USER
                    )
                    is False
                )
            finally:
                loop.close()
                sio_mod.set_main_loop(saved_loop)
        finally:
            sys.modules.pop(name, None)