~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""Connection-lifecycle semantics of the ASGI Socket.IO layer.
Flask-SocketIO's ``SocketIOService`` became python-socketio's ``AsyncServer``
mounted at ``/ws``. Flask-SocketIO gave us rooms, a room registry that
survived across a reconnect only because the client re-joined, and a
``message_queue=`` knob that made multi-process deployment a config change.
The replacement keeps *all* of its routing state in module-level dicts
(``_subscriptions``, ``_sid_users``, ``_sid_sessions``, ``_main_loop``) plus
an in-process ``AsyncManager``. That is a deliberate trade, and it has
consequences that only show up at connection boundaries: a reconnect, a tab
closing mid-fanout, a run finishing, a second uvicorn worker.
This file covers the four of those that nothing else reaches:
1. **Reconnection and resubscription.** A reconnect is a *new sid*. Nothing
server-side carries the old sid's subscriptions across, so the JS client's
``socket.on('connect', ...)`` re-subscribe is load-bearing, and the window
between the drop and the re-subscribe delivers to nobody rather than
falling back to a broadcast.
2. **The reconnect window itself**, where the old sid has not been reaped by
engine.io's ping timeout yet, so a dead sid and a live sid sit in the same
subscription set and the dead one's emit raises.
3. **Teardown landing mid-fanout** — a tab closing, or a run's own
``remove_subscriptions_for_research`` arriving, while the terminal
``research_progress`` message is still being delivered to the user's other
tabs. ``research_service`` schedules those two back to back
(``_sio_emit(...)`` then ``_sio_remove(...)``, both onto the same loop), so
this interleaving is the normal completion path, not a corner case.
4. **Why ``workers=1`` is mandatory.** Two other files already pin the literal
``workers=1`` in ``web/app.py`` by AST. Neither pins the *reason*, so
nothing tells the person who eventually adds a Redis manager that they may
now raise it -- or tells the person who raises it first what breaks.
Deliberately NOT covered here (already pinned elsewhere, do not duplicate):
* mount path / CORS ``None``-vs-``[]`` / ``(username, research_id)`` keying /
emit fail-closed / ``_async_emit`` transport-failure containment
-- ``test_socketio_asgi_contracts.py``
* handshake cookie crypto and the connect auth gate
-- ``test_socketio_handshake_auth.py``, ``test_socketio_connect_gate.py``
* revoked-session gating at connect and on subscribe/unsubscribe
-- ``test_socket_connect_session_gate.py``,
``tests/security/test_socket_ownership_edges_fastapi.py``,
``tests/chat/test_chat_socket_events.py``
* cross-*user* delivery scoping -- ``test_subscription_owner_scoping.py``,
``test_socketio_asgi_user_scoping.py``
* the ``get_active_research_snapshot`` replay on subscribe
-- ``tests/chat/test_chat_socket_events.py``
Everything here drives the real coroutines to completion under
``asyncio.run``; no wall-clock sleeps and no polling.
"""
import asyncio
import importlib.util
import sys
from unittest.mock import patch
import pytest
from socketio.async_pubsub_manager import AsyncPubSubManager
from local_deep_research.web.auth.session_manager import session_manager
from local_deep_research.web.services import socketio_asgi as sio_mod
USER = "ws_semantics_user"
OTHER = "ws_semantics_other"
RID = "research-abc"
@pytest.fixture
def socket_state(monkeypatch):
"""Isolated socket state plus a recording ``sio.emit``.
A fresh ``asyncio.Lock`` per test: the real one is created once at
lifespan startup, and a lock left bound to a previous test's closed loop
raises on the next acquire.
"""
monkeypatch.setattr(sio_mod, "_lock", asyncio.Lock())
monkeypatch.setattr(sio_mod, "_subscriptions", {})
monkeypatch.setattr(sio_mod, "_sid_users", {})
monkeypatch.setattr(sio_mod, "_sid_sessions", {})
delivered = []
async def _capture(event, data, room=None):
delivered.append((room, event, data))
monkeypatch.setattr(sio_mod.sio, "emit", _capture)
return delivered
def _seed(username, research_id, sid, session_id=None):
"""Register ``sid`` as an authenticated subscriber, as connect+subscribe
would have left it."""
sio_mod._sid_users[sid] = username
if session_id is not None:
sio_mod._sid_sessions[sid] = session_id
sio_mod._subscriptions.setdefault((username, research_id), set()).add(sid)
def _fanout(research_id=RID, owner=USER, payload=None):
"""Run one real ``_async_emit_to_subscribers`` fanout to completion."""
asyncio.run(
sio_mod._async_emit_to_subscribers(
"research_progress",
research_id,
{"progress": 50} if payload is None else payload,
owner,
)
)
def _rooms(delivered):
return [room for room, _, _ in delivered]
# ---------------------------------------------------------------------------
# 1. Reconnection and resubscription
# ---------------------------------------------------------------------------
@pytest.mark.real_session_check
class TestReconnectMustResubscribe:
"""A reconnect is a brand-new sid with no inherited subscriptions.
Under Flask-SocketIO the same was true, but the failure mode was visible:
``join_room`` was explicit. Here the subscription set is keyed by sid and
the disconnect handler purges it, so after a reconnect the server has no
memory of what the socket was watching. The only thing that restores
delivery is the client re-issuing ``subscribe_to_research`` from its
``connect`` handler.
The important half is the *negative*: during the gap the fanout must find
no bucket and deliver nothing. It must never fall back to a roomless
``sio.emit(event, data)``, which would hand one user's research progress
to every connected client.
"""
def _connect(self, sid, session_id, username=USER):
"""Drive the real ``connect`` coroutine with a live server-side
session, as a reconnecting browser would.
``is_user_connected`` is forced True so the lazy-DB-open branch (which
has its own tests) stays out of the way.
"""
with (
patch.object(
sio_mod,
"_decode_session_cookie",
return_value={
"username": username,
"session_id": session_id,
},
),
patch(
"local_deep_research.database.encrypted_db.db_manager"
".is_user_connected",
return_value=True,
),
):
return asyncio.run(
sio_mod.connect(sid, {"HTTP_COOKIE": "session=whatever"})
)
def test_the_gap_between_disconnect_and_resubscribe_reaches_nobody(
self, socket_state
):
"""The full round trip, with the real connect/disconnect coroutines.
Subscribed -> transport drops -> reconnects on a new sid -> a progress
event fires before the client has re-subscribed -> it reaches nobody
and is not broadcast -> the client re-subscribes -> delivery resumes
on the new sid only.
"""
token = session_manager.create_session(USER, remember_me=False)
try:
_seed(USER, RID, "sid-old", session_id=token)
_fanout()
assert _rooms(socket_state) == ["sid-old"], (
"premise: the original socket was receiving"
)
socket_state.clear()
# Transport drops. engine.io fires the disconnect handler.
asyncio.run(sio_mod.disconnect("sid-old"))
assert sio_mod._subscriptions == {}, (
"disconnect left the old sid's subscription behind"
)
# The browser reconnects: new handshake, new sid, same cookie.
assert self._connect("sid-new", token) is True
assert sio_mod._sid_users == {"sid-new": USER}
# A progress event fires in the window before the client's
# connect handler has re-subscribed.
_fanout(payload={"progress": 60})
assert socket_state == [], (
"an event in the reconnect window was delivered to somebody; "
"the only two ways that happens are a resurrected sid or a "
"roomless broadcast, and the second leaks to every user"
)
# The client re-subscribes (socket.js does this from its
# 'connect' handler). Delivery resumes -- on the new sid only.
with patch.object(
sio_mod, "_user_owns_research", return_value=True
):
with patch.object(
sio_mod, "get_active_research_snapshot", return_value=None
):
asyncio.run(
sio_mod.on_subscribe("sid-new", {"research_id": RID})
)
assert sio_mod._subscriptions == {(USER, RID): {"sid-new"}}
socket_state.clear()
_fanout(payload={"progress": 70})
assert _rooms(socket_state) == ["sid-new"]
finally:
session_manager.destroy_session(token)
def test_a_reconnect_does_not_inherit_the_previous_sids_subscriptions(
self, socket_state
):
"""Same user, same session, same research -- still nothing inherited.
Pins that the carry-over is keyed by sid and nothing else. If someone
ever "helpfully" re-keys ``_subscriptions`` by username or by session
so a reconnect resumes automatically, the socket would start receiving
a run it never asked for on *this* connection, and the ownership check
in ``on_subscribe`` -- the only place ownership is ever verified --
would be bypassed for it.
"""
token = session_manager.create_session(USER, remember_me=False)
try:
_seed(USER, RID, "sid-old", session_id=token)
asyncio.run(sio_mod.disconnect("sid-old"))
assert self._connect("sid-new", token) is True
assert sio_mod._subscriptions == {}, (
"connect() restored subscriptions for the reconnecting "
"session; subscriptions must only ever be created by "
"on_subscribe, which is where ownership is checked"
)
_fanout()
assert socket_state == []
finally:
session_manager.destroy_session(token)
class TestReconnectWindowWithAStaleSid:
"""Both sids are subscribed at once, briefly.
A dropped WebSocket is not noticed instantly: the server holds
``ping_timeout=20`` / ``ping_interval=5``, so up to ~25s can pass before
the disconnect handler runs and purges the old sid. socket.io's client
reconnects after ``reconnectionDelay: 1000``, i.e. long before that. So
for most of that window the subscription set holds a *dead* sid alongside
the freshly reconnected one, and emitting to the dead one raises.
"""
def test_a_dead_sid_does_not_starve_the_reconnected_one(
self, socket_state, monkeypatch
):
"""The per-sid try/except around ``sio.emit`` is what makes this safe.
``_subscriptions`` holds a plain ``set``, so iteration order is not
defined -- this must hold whichever sid the fanout reaches first, and
it does: each delivery is independently guarded. Without the guard the
exception escapes the coroutine in *both* orders (it is not a
50/50 flake), which is what makes this a real control.
"""
delivered = []
async def _emit(event, data, room=None):
if room == "sid-dead":
raise ConnectionError("client is gone")
delivered.append(room)
monkeypatch.setattr(sio_mod.sio, "emit", _emit)
_seed(USER, RID, "sid-dead")
_seed(USER, RID, "sid-reconnected")
_fanout()
assert delivered == ["sid-reconnected"], (
"a stale socket left over from the reconnect window swallowed "
"the delivery to the socket that actually replaced it"
)
def test_the_stale_sid_is_purged_when_the_timeout_finally_fires(
self, socket_state
):
"""And the live sid survives that purge.
``disconnect`` walks *every* subscription bucket discarding the sid
and deletes buckets that empty. The bucket here must not be deleted:
the reconnected socket is still in it. This is the same code path as
two tabs on one run, one of which closes.
"""
_seed(USER, RID, "sid-dead")
_seed(USER, RID, "sid-reconnected")
asyncio.run(sio_mod.disconnect("sid-dead"))
assert sio_mod._subscriptions == {(USER, RID): {"sid-reconnected"}}, (
"one socket disconnecting tore down a sibling's subscription to "
"the same run"
)
_fanout()
assert _rooms(socket_state) == ["sid-reconnected"]
# ---------------------------------------------------------------------------
# 2. Teardown landing in the middle of a fanout
# ---------------------------------------------------------------------------
class TestTeardownDuringTheTerminalFanout:
"""What subscribers get when a research completes or errors.
``research_service._cleanup_research_resources`` ends every run --
COMPLETED, FAILED or SUSPENDED -- with two calls back to back::
_sio_emit("research_progress", research_id, final_message, owner=...)
_sio_remove(research_id, username)
Both are sync wrappers that schedule a coroutine onto the one uvicorn
loop, so the cleanup coroutine is queued while the terminal message is
still being fanned out to the user's tabs. On top of that, a user watching
a run finish very often closes the tab at that exact moment, firing
``disconnect`` into the same window.
``_async_emit_to_subscribers`` copies the subscription set under the lock
and then releases the lock before delivering. These tests pin what that
copy buys.
"""
def test_a_tab_closing_mid_fanout_does_not_abort_the_terminal_message(
self, socket_state, monkeypatch
):
"""The user's other tabs must still be told the run ended.
``disconnect`` mutates the subscription set in place
(``sids.discard(sid)``). The fanout has already released the lock, so
a disconnect *can* land between two deliveries. If the fanout were
iterating the live set rather than a copy, that in-place discard
raises ``RuntimeError: Set changed size during iteration`` out of the
coroutine, and every subscriber after the closing tab -- including the
one still watching -- is left on a progress bar frozen at 99%.
"""
delivered = []
async def _emit(event, data, room=None):
delivered.append(room)
if len(delivered) == 1:
# The *other* tab closes while this delivery is in flight.
other = {"sid-tab-1", "sid-tab-2"} - {room}
await sio_mod.disconnect(other.pop())
monkeypatch.setattr(sio_mod.sio, "emit", _emit)
_seed(USER, RID, "sid-tab-1")
_seed(USER, RID, "sid-tab-2")
_fanout(payload={"status": "completed", "progress": 100})
assert sorted(delivered) == ["sid-tab-1", "sid-tab-2"], (
"the terminal message stopped at the tab that closed; the "
f"remaining subscriber never learned the run ended: {delivered}"
)
def test_the_runs_own_cleanup_cannot_cancel_the_message_announcing_it(
self, socket_state, monkeypatch
):
"""``_sio_remove`` is scheduled one line after ``_sio_emit``.
Modelled at the point where it actually bites: the cleanup coroutine
runs while the terminal fanout is mid-flight. Every subscriber must
still receive the final status, and the bucket must be gone
afterwards. Both halves matter -- delivering but leaking the bucket,
or reaping the bucket but dropping the message, are each a bug, and
only asserting them together pins the ordering.
"""
delivered = []
async def _emit(event, data, room=None):
delivered.append((room, data))
if len(delivered) == 1:
# research_service's very next line, landing early.
await sio_mod._async_remove_subscriptions(RID, USER)
monkeypatch.setattr(sio_mod.sio, "emit", _emit)
_seed(USER, RID, "sid-tab-1")
_seed(USER, RID, "sid-tab-2")
final = {"status": "failed", "message": "Research was failed"}
_fanout(payload=final)
assert sorted(room for room, _ in delivered) == [
"sid-tab-1",
"sid-tab-2",
], (
"the completion cleanup cancelled delivery of the very message "
f"that reports the completion: {delivered}"
)
assert all(data == final for _, data in delivered)
assert sio_mod._subscriptions == {}, (
"the finished run's subscription bucket outlived the run"
)
def test_cleanup_leaves_the_socket_connected_for_the_next_run(
self, socket_state
):
"""A run ending is not a logout.
``_async_remove_subscriptions`` pops one ``(owner, research_id)``
bucket and nothing else. The identity maps must survive, or the user
would have to reconnect before starting another research -- and
``on_subscribe`` would reject them, since it reads identity from
``_sid_users``.
"""
_seed(USER, RID, "sid-tab-1", session_id="sess-1")
_seed(USER, "research-other", "sid-tab-1", session_id="sess-1")
asyncio.run(sio_mod._async_remove_subscriptions(RID, USER))
assert sio_mod._sid_users == {"sid-tab-1": USER}
assert sio_mod._sid_sessions == {"sid-tab-1": "sess-1"}
assert sio_mod._subscriptions == {
(USER, "research-other"): {"sid-tab-1"}
}, "cleanup for one run tore down the socket's other subscription"
def test_a_late_event_after_cleanup_reaches_nobody_not_everybody(
self, socket_state
):
"""Emits genuinely do arrive after cleanup.
The log-queue processor and the search threads drain asynchronously,
so a stray ``research_progress`` for a finished run is routine. With
the bucket gone the lookup misses, and the miss must be a drop --
``_async_emit_to_subscribers`` has no ``else`` that broadcasts, and a
second connected user proves it stays that way.
"""
_seed(USER, RID, "sid-owner")
_seed(OTHER, "another-run", "sid-bystander")
asyncio.run(sio_mod._async_remove_subscriptions(RID, USER))
_fanout(payload={"progress": 100})
assert socket_state == [], (
"a post-cleanup event was delivered; the only reachable "
"recipient set is 'everyone', which is the cross-user leak"
)
# ---------------------------------------------------------------------------
# 3. Why workers=1 is mandatory
# ---------------------------------------------------------------------------
def _load_second_instance():
"""Load a second, independent instance of ``socketio_asgi``.
This stands in for a second uvicorn worker. It is the same source file
executed a second time, so it is an honest model of what a second *process*
gets: its own module globals and its own ``AsyncServer``. It is not a
re-implementation -- the delivery path exercised below is the production
coroutine, reached through the second instance.
The name keeps the real package prefix so the module's relative imports
(``from ...constants import ...``) still resolve.
"""
name = sio_mod.__name__ + "_second_worker"
spec = importlib.util.spec_from_file_location(name, sio_mod.__file__)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
try:
spec.loader.exec_module(module)
except BaseException:
sys.modules.pop(name, None)
raise
return name, module
class TestSingleWorkerConstraint:
"""``web/app.py`` passes ``workers=1`` with the comment "Required for
Socket.IO without Redis message queue".
``tests/web/test_concurrency_under_single_worker.py`` and
``tests/web/test_lifespan_startup_shutdown.py`` both pin that literal by
AST. Neither pins the premise it rests on, which leaves two ways to get it
wrong:
* someone raises ``workers``, the AST tests fail, and the failure message
asserts a reason nobody has verified is still true;
* someone *adds* a Redis manager -- which is exactly what would make
raising ``workers`` legitimate -- and nothing tells them the constraint
can now be lifted, so the single worker stays forever.
These tests pin the causal chain instead of the number.
"""
def test_the_server_has_no_message_queue_so_state_cannot_span_processes(
self,
):
"""The premise, asserted against the live server object.
python-socketio fans an emit out across processes only through a
``AsyncPubSubManager`` subclass (``AsyncRedisManager``,
``AsyncKombuManager``, ...) passed as ``client_manager``. The default
``AsyncManager`` keeps its room registry in this process's memory.
``socketio_asgi`` passes no ``client_manager``, so there is no medium
by which worker 2 could learn about worker 1's sockets.
"""
assert not isinstance(sio_mod.sio.manager, AsyncPubSubManager), (
"a Socket.IO message queue manager "
f"({type(sio_mod.sio.manager).__name__}) was added. Emits can now "
"cross processes, so the `workers=1` in web/app.py may no longer "
"be required -- but `_subscriptions`, `_sid_users`, "
"`_sid_sessions` and `_main_loop` are still plain module globals "
"and do NOT cross processes, so a multi-worker deploy still "
"misroutes unless those move too. Decide that explicitly and "
"update this test with the reasoning."
)
def test_the_routing_state_that_decides_delivery_is_process_local(self):
"""The consequence, executed.
A subscriber registered in instance A is invisible to instance B, so
an emit issued by B for that research reaches nobody -- silently, with
no error anywhere. Under ``--workers 2`` that is a progress bar that
freezes for roughly half of all users, depending on which worker
accepted the WebSocket versus which one ran the research thread.
Instance A doubles as the positive control in the same test: the
identical fanout, same seeded state, delivers through A. So "reaches
nobody" cannot be an artefact of a mis-seeded harness.
"""
name, worker2 = _load_second_instance()
try:
assert worker2.__file__ == sio_mod.__file__, (
"premise: the second instance must be the same source file"
)
assert worker2.sio is not sio_mod.sio
assert worker2.sio.manager is not sio_mod.sio.manager
assert worker2._subscriptions == {}, (
"a freshly loaded instance already sees subscriptions, so "
"this is not modelling a separate process"
)
a_delivered = []
b_delivered = []
async def _capture_a(event, data, room=None):
a_delivered.append(room)
async def _capture_b(event, data, room=None):
b_delivered.append(room)
with (
patch.object(sio_mod, "_lock", asyncio.Lock()),
patch.object(sio_mod, "_subscriptions", {}),
patch.object(sio_mod.sio, "emit", _capture_a),
patch.object(worker2, "_lock", asyncio.Lock()),
patch.object(worker2.sio, "emit", _capture_b),
):
# Worker 1 accepted the WebSocket and the subscribe.
sio_mod._subscriptions[(USER, RID)] = {"sid-on-worker-1"}
# Worker 2 is running the research thread and emits progress.
asyncio.run(
worker2._async_emit_to_subscribers(
"research_progress", RID, {"progress": 50}, USER
)
)
# Positive control: the same call on worker 1 does deliver.
asyncio.run(
sio_mod._async_emit_to_subscribers(
"research_progress", RID, {"progress": 50}, USER
)
)
assert a_delivered == ["sid-on-worker-1"], (
"control failed: the subscriber was not reachable even from "
"the instance that registered it, so the assertion below "
"would prove nothing"
)
assert b_delivered == [], (
"a second worker reached worker 1's subscriber; if that is "
"genuinely possible now, the module no longer relies on "
"process-local state and workers=1 can be revisited"
)
finally:
sys.modules.pop(name, None)
def test_the_captured_event_loop_is_process_local_too(self):
"""The other half of the same constraint, and the quieter half.
Every emit from a background thread goes through ``_get_main_loop()``,
which returns the loop captured by ``set_main_loop`` during *this*
process's lifespan startup. A second worker captures its own. So even
with a shared subscription store, a worker could only ever dispatch
into its own loop -- fixing the state sharing alone would not be
enough, which is why the fix is "add a message queue", not "move the
dicts to Redis".
"""
name, worker2 = _load_second_instance()
saved_loop = sio_mod._main_loop
try:
loop = asyncio.new_event_loop()
try:
sio_mod.set_main_loop(loop)
assert sio_mod._get_main_loop() is loop
assert worker2._main_loop is None, (
"set_main_loop reached beyond its own module instance"
)
# With no loop of its own, the second worker's sync emit
# wrapper fails closed rather than dispatching anywhere.
assert (
worker2.emit_to_subscribers(
"research_progress", RID, {}, owner=USER
)
is False
)
finally:
loop.close()
sio_mod.set_main_loop(saved_loop)
finally:
sys.modules.pop(name, None)