Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(ws): stop a client disconnect from cancelling database teardown mid-rollback
trohitg914afa633ec3
diff --git a/CLAUDE.md b/CLAUDE.mdindex f7b41d3e..e41331a8 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -2745,6 +2745,7 @@ This function:- **writeTodos Tool Node**: Dedicated AI tool for task planning connecting to any agent's `input-tools` handle. `TodoService` singleton (`server/services/todo_service.py`) stores JSON-based per-session todo state keyed by workflow_id. Handler broadcasts `phase: "todo_update"` via WebSocket on each update for real-time UI. (The former checklist rendering — `formatTodoOutput()` — lived only in the legacy `ui/OutputDisplayPanel.tsx`, which was deleted as dead code; the active `components/output/OutputPanel.tsx` shows todo results as regular JSON.) Schema uses `TodoItem`/`TodoStatus` Pydantic enum. Skill at `server/skills/assistant/write-todos-skill/SKILL.md` teaches the plan-work-update loop.- **Temporal Activity Heartbeats**: Activities send `activity.heartbeat()` on every non-matching WebSocket broadcast inside the read loop in `services/temporal/activities.py`. This keeps long-running browser and claude_code_agent operations alive past the 2-minute `heartbeat_timeout`. Start/end heartbeats alone were causing `TIMEOUT_TYPE_HEARTBEAT` failures on ops taking 5-10 minutes. Connection config: `heartbeat=30`, `receive_timeout=None` (each receive is bounded by a 30 s `asyncio.wait_for`; liveness comes from heartbeats, and per-node activities run under a 24 h `start_to_close_timeout`).- **WebSocket `_safe_send` Guard**: `server/routers/websocket.py` checks `websocket.client_state.name != "CONNECTED"` before sending and logs at `debug` level (not `error`) on failure. Prevents "ASGI message after websocket.close" errors when broadcasts race with disconnects.+- **WebSocket disconnect must not cancel database teardown**: a client that disconnects within ~100 ms of connecting (Strict Mode double mount, a health probe) used to make the `/ws/status` endpoint cancel its in-flight init-burst handlers mid-statement, so SQLAlchemy's reset-on-return rollback took the `CancelledError` and logged `Exception during reset or similar` with a traceback per handler. Three guards, locked by `tests/test_ws_disconnect_cancellation.py`: `core/session_teardown.py` runs rollback + close under `asyncio.shield` for both `Database.get_session` and `CredentialsDatabase.get_session`; `_drain_handler_tasks` in `routers/websocket.py` waits one second for handlers before cancelling stragglers; `StatusBroadcaster.broadcast` skips and prunes sockets that are no longer `CONNECTED` on either side. Any new session helper must reuse `teardown_session`, and a new `core` module that `core/database.py` imports must also be file-loaded in `tests/conftest.py` (the stubbed `core` package has an empty `__path__`).- **Claude Code CLI Flag**: the Claude Code agent (`nodes/agent/claude_code_agent/`) uses `--max-budget-usd <amount>` (previously incorrectly `--max-cost`, which caused "unknown option" errors on every run).- **Persisted-prefix cache contract**: `PersistQueryClientProvider` hydrates entries with the QueryClient's *default* options, so any prefix in `PERSISTED_KEY_PREFIXES` (`client/src/lib/queryPersist.ts`) MUST also have a matching `queryClient.setQueryDefaults(['<prefix>'], { staleTime: FOREVER, gcTime: FOREVER })` declaration in `client/src/lib/queryClient.ts`. Per-call options don't apply on hydration. Canonical set: `nodeSpec`, `nodeGroups`, `skillContent`. `credentialValues` was previously persisted here but was removed per OWASP HTML5 Security Cheat Sheet / ASVS V9.9 — decrypted API keys must never live in `localStorage`. The in-memory TanStack Query cache (`gcTime: ∞`) keeps the credentials form populated for the session lifetime; on reload the panel refetches via WS. `credentialCatalogue` has its own `idb-keyval` warm-start path so it is intentionally not in either list.- **WebSocket `sendRequest` replay queue**: when the socket is not open, requests enqueue with `AbortController`-backed per-request timeouts (default 30s) and replay on reconnect inside `ws.onopen` before `setIsReady(true)`. Queue capped at 200 with FIFO eviction. Intentional close (`event.code === 1000`) drops the queue; transient closes preserve it. Eliminates indefinite spinners during the 3s reconnect window. Implementation in `client/src/contexts/WebSocketContext.tsx` (`pendingSendQueueRef` + `drainPendingSends`).diff --git a/docs-internal/errors.md b/docs-internal/errors.mdindex 17f9f76c..87fe2b14 100644--- a/docs-internal/errors.md+++ b/docs-internal/errors.md@@ -384,3 +384,13 @@ python -c "import ntpath; print(ntpath.join(r'D:\ws\AI_Employee_1', '2:processMa```**Fix** (`21fb1a9a`): `core.paths.safe_path_component` sanitizes a wire identifier before it becomes a path component; `processManager` and `cli_agent/session.py` route the node id through it. The guardrail error now names the rejected resolved path and says to omit `cwd` (the designed happy path — the process-manager skill never passes one), and `httpRequest` maps `ConnectError` / `TimeoutException` to `NodeUserError`. Rule: never join a node id, model-supplied name, or any other wire identifier into a path raw — go through `safe_path_component`. Locked by `tests/nodes/test_process_manager_cwd.py`.++---++## 13. `Exception during reset or similar` / `CancelledError` in aiosqlite rollback on WebSocket disconnect++**Symptom**: Right after `[StatusBroadcaster] Client connected`, the operator log shows one or more SQLAlchemy tracebacks ending in `asyncio.exceptions.CancelledError` from `aiosqlite/core.py ... rollback`, then `[StatusBroadcaster] Send failed: Cannot call "send" once a close message has been sent`, then `Client disconnected`. Typical trigger: a client that disconnects within ~100 ms of connecting (React Strict Mode double mount in `company dev`, a health probe), i.e. while the connect-time init burst (chat / console / terminal history, credential probes) is still running.++**Root cause**: the `/ws/status` endpoint's cleanup cancelled every in-flight handler task the instant the socket closed. A handler cancelled while inside a database call receives `CancelledError` at the aiosqlite await; the session teardown then returns the connection to the pool, whose reset-on-return rollback is itself interrupted by the pending cancellation. SQLAlchemy's `_finalize_fairy` logs the traceback at ERROR, invalidates the connection, and re-raises, so nothing was actually broken (the pool rebuilt the connection) but every reconnect produced a traceback per cancelled handler. The `Send failed` warning came from a broadcast racing the close frame.++**Fix**: `core/session_teardown.py` runs rollback + close under `asyncio.shield` in both `Database.get_session` and `CredentialsDatabase.get_session`, so a cancelled caller cannot interrupt the pool reset; `_drain_handler_tasks` in `routers/websocket.py` gives in-flight handlers a one-second grace to finish before cancelling stragglers; `StatusBroadcaster.broadcast` skips sockets that are no longer `CONNECTED` on either side and prunes them quietly. Locked by `tests/test_ws_disconnect_cancellation.py`.diff --git a/docs-internal/status_broadcaster.md b/docs-internal/status_broadcaster.mdindex 537d78f7..a75d6e6c 100644--- a/docs-internal/status_broadcaster.md+++ b/docs-internal/status_broadcaster.md@@ -219,7 +219,8 @@ Android service nodes (`batteryMonitor`, `wifiAutomation`, etc.) check `androidS- Responds to `ping` with `{"type": "pong"}`.- On `get_status`, returns the full current status snapshot.-- On disconnect, removes the WebSocket from `StatusBroadcaster._connections`.+- On disconnect, waits up to one second for the socket's in-flight handler tasks to finish (`_drain_handler_tasks`), cancels any still running, then removes the WebSocket from `StatusBroadcaster._connections`. The grace exists because a client that reconnects ~100 ms after connecting would otherwise cancel the init-burst handlers inside a database call; session teardown is additionally shielded from cancellation in `core/session_teardown.py` (see errors.md entry 13).+- `broadcast()` skips and prunes sockets that are no longer `CONNECTED` on either side, so a broadcast racing a close frame does not log a warning per message.## Error Handlingdiff --git a/server/core/credentials_database.py b/server/core/credentials_database.pyindex eaba77f7..d4ba1bf1 100644--- a/server/core/credentials_database.py+++ b/server/core/credentials_database.py@@ -12,7 +12,10 @@ The database is separate from the main application database (workflow.db) to:import hashlibimport logging+import asynciofrom contextlib import asynccontextmanager++from core.session_teardown import teardown_sessionfrom datetime import datetime, timezonefrom typing import Any, Dict, List, Optional@@ -151,9 +154,24 @@ class CredentialsDatabase:@asynccontextmanagerasync def get_session(self):- """Get async database session."""- async with self._session_factory() as session:+ """Get async database session.++ Same cancellation-safe teardown as ``core.database.Database``:+ rollback + close run under ``asyncio.shield`` so a cancelled+ caller (a WebSocket client disconnecting during the connect-time+ credential probes) cannot interrupt the pool's reset rollback.+ """+ session = self._session_factory()+ try:yield session+ except asyncio.CancelledError:+ await asyncio.shield(teardown_session(session, rollback=True))+ raise+ except Exception:+ await asyncio.shield(teardown_session(session, rollback=True))+ raise+ else:+ await asyncio.shield(teardown_session(session, rollback=False))# --- Metadata Operations ---diff --git a/server/core/database.py b/server/core/database.pyindex 7b0a7759..9e8d8234 100644--- a/server/core/database.py+++ b/server/core/database.py@@ -1,5 +1,6 @@"""Modern async database service with SQLModel and SQLAlchemy 2.0."""+import asyncioimport jsonimport inspectimport secrets@@ -13,6 +14,8 @@ from sqlalchemy.exc import IntegrityErrorfrom sqlalchemy import case, text, func, update, or_from contextlib import asynccontextmanager+from core.session_teardown import teardown_session as _teardown_session+from core.config import Settingsfrom models.database import (NodeParameter,@@ -375,18 +378,31 @@ class Database:@asynccontextmanagerasync def get_session(self):- """Get async database session."""+ """Get async database session.++ Teardown (rollback + close) runs under ``asyncio.shield``. When the+ calling task is cancelled mid-statement -- a WebSocket client that+ disconnects while its connect-time handlers are still reading --+ the cancellation would otherwise land inside the pool's+ reset-on-return rollback; SQLAlchemy then logs "Exception during+ reset or similar" with a full traceback and discards the+ connection. Shielding lets the reset finish while the+ CancelledError still propagates to the caller.+ """if not self.async_session:raise RuntimeError("Database not initialized")- async with self.async_session() as session:- try:- yield session- except Exception:- await session.rollback()- raise- finally:- await session.close()+ session = self.async_session()+ try:+ yield session+ except asyncio.CancelledError:+ await asyncio.shield(_teardown_session(session, rollback=True))+ raise+ except Exception:+ await asyncio.shield(_teardown_session(session, rollback=True))+ raise+ else:+ await asyncio.shield(_teardown_session(session, rollback=False))async def _migrate_workflow_controls(self):"""Backfill control-plane columns when upgrading an early preview DB."""diff --git a/server/core/session_teardown.py b/server/core/session_teardown.pynew file mode 100644index 00000000..28e8b63c--- /dev/null+++ b/server/core/session_teardown.py@@ -0,0 +1,26 @@+"""Cancellation-safe teardown for async SQLAlchemy sessions.++Shared by ``core.database`` and ``core.credentials_database``. Both wrap+the call in ``asyncio.shield`` so a task cancelled while a session is+open (a WebSocket client disconnecting mid-request) cannot interrupt+the pool's reset-on-return rollback. Without the shield SQLAlchemy's+``_finalize_fairy`` logs "Exception during reset or similar" with a+traceback, invalidates the connection, and re-raises the+CancelledError; the connection is then rebuilt on the next checkout.+"""++from __future__ import annotations++from typing import Any+++async def teardown_session(session: Any, *, rollback: bool) -> None:+ """Roll back (optionally) and close ``session``, always closing."""+ try:+ if rollback:+ await session.rollback()+ finally:+ await session.close()+++__all__ = ["teardown_session"]diff --git a/server/routers/websocket.py b/server/routers/websocket.pyindex d4953c1f..d4c9ab57 100644--- a/server/routers/websocket.py+++ b/server/routers/websocket.py@@ -56,6 +56,27 @@ async def _safe_send(websocket: WebSocket, data: dict):logger.debug("[WebSocket] Send skipped (connection closed): %s", e)+# How long a disconnecting socket waits for its in-flight handlers before+# cancelling them. The connect-time init burst is a handful of short DB+# reads; a client that reconnects ~100 ms later (React Strict Mode double+# mount, a health probe) used to cancel those mid-statement, which+# SQLAlchemy reported as "Exception during reset or similar" with a+# traceback per handler. Long handlers (execute_node parked on an event+# waiter) are still cancelled once the grace expires.+_HANDLER_DRAIN_GRACE_SECONDS = 1.0+++async def _drain_handler_tasks(handler_tasks: Set[asyncio.Task], grace: float = _HANDLER_DRAIN_GRACE_SECONDS) -> None:+ """Let in-flight handler tasks finish for ``grace`` seconds, then cancel the rest."""+ tasks = [task for task in handler_tasks if not task.done()]+ if not tasks:+ return+ _done, pending = await asyncio.wait(tasks, timeout=grace)+ for task in pending:+ task.cancel()+ await asyncio.gather(*tasks, return_exceptions=True)++# Type for message handlersMessageHandler = Callable[[Dict[str, Any], WebSocket], Awaitable[Dict[str, Any]]]@@ -1708,14 +1729,9 @@ async def websocket_status_endpoint(websocket: WebSocket):if not isinstance(exc, (WebSocketDisconnect, asyncio.CancelledError, KeyboardInterrupt, SystemExit)):logger.error(f"[WebSocket] TaskGroup error: {exc}")finally:- # Cancel any running handler tasks on disconnect- for task in list(handler_tasks):- if not task.done():- task.cancel()-- # Wait for tasks to finish cancellation- if handler_tasks:- await asyncio.gather(*handler_tasks, return_exceptions=True)+ # Let short handlers finish, then cancel stragglers (see+ # _drain_handler_tasks for why this is not an immediate cancel).+ await _drain_handler_tasks(handler_tasks)# Cleanup_handler_tasks.pop(websocket, None)@@ -1806,12 +1822,7 @@ async def websocket_internal_endpoint(websocket: WebSocket):if not isinstance(exc, (WebSocketDisconnect, asyncio.CancelledError, KeyboardInterrupt, SystemExit)):logger.error(f"[WebSocket Internal] TaskGroup error: {exc}")finally:- for task in list(handler_tasks):- if not task.done():- task.cancel()-- if handler_tasks:- await asyncio.gather(*handler_tasks, return_exceptions=True)+ await _drain_handler_tasks(handler_tasks)@router.get("/ws/info")diff --git a/server/services/status_broadcaster.py b/server/services/status_broadcaster.pyindex 2ac0233d..2be8198c 100644--- a/server/services/status_broadcaster.py+++ b/server/services/status_broadcaster.py@@ -8,6 +8,7 @@ import asyncioimport orjsonfrom typing import Set, Dict, Any, Optional, Listfrom fastapi import WebSocket+from starlette.websockets import WebSocketStatefrom opentelemetry import tracefrom core.logging import get_logger@@ -59,6 +60,14 @@ def _elide_for_cache(output: Any) -> Any:tracer = trace.get_tracer(__name__)+def _socket_open(connection: WebSocket) -> bool:+ """True while both sides of the socket are still CONNECTED."""+ return (+ getattr(connection, "client_state", WebSocketState.CONNECTED) is WebSocketState.CONNECTED+ and getattr(connection, "application_state", WebSocketState.CONNECTED) is WebSocketState.CONNECTED+ )++class StatusBroadcaster:"""Manages WebSocket connections and broadcasts status updates."""@@ -232,10 +241,19 @@ class StatusBroadcaster:async def send_to_client(connection: WebSocket):"""Send message to a single client."""+ if not _socket_open(connection):+ # The client already sent (or received) a close frame; the+ # endpoint's disconnect() has not run yet. Drop it silently+ # instead of warning once per broadcast in the gap.+ disconnected.add(connection)+ returntry:await connection.send_text(message_bytes)except Exception as e:- logger.warning(f"[StatusBroadcaster] Send failed: {e}")+ if _socket_open(connection):+ logger.warning(f"[StatusBroadcaster] Send failed: {e}")+ else:+ logger.debug("[StatusBroadcaster] Send skipped, client closed: %s", e)disconnected.add(connection)# Execute all sends concurrently with TaskGroupdiff --git a/server/tests/conftest.py b/server/tests/conftest.pyindex 977f0a81..4bd9c48a 100644--- a/server/tests/conftest.py+++ b/server/tests/conftest.py@@ -145,6 +145,19 @@ sys.modules["core.encryption"] = _encryption_mod_encryption_spec.loader.exec_module(_encryption_mod)setattr(_core_pkg, "encryption", _encryption_mod)+# core.session_teardown is stdlib-only (the shielded rollback + close both+# databases run under a cancelled caller); the real core/database.py and+# core/credentials_database.py import it at module load, and tests that+# file-load those modules need it resolvable under the stubbed package.+_session_teardown_spec = _importlib_util.spec_from_file_location(+ "core.session_teardown",+ SERVER_DIR / "core" / "session_teardown.py",+)+_session_teardown_mod = _importlib_util.module_from_spec(_session_teardown_spec)+sys.modules["core.session_teardown"] = _session_teardown_mod+_session_teardown_spec.loader.exec_module(_session_teardown_mod)+setattr(_core_pkg, "session_teardown", _session_teardown_mod)+# core.paths — central path resolution. Stub the public surface with# tmpdir-rooted Paths so plugin module imports don't trip over the# real ``Path.home()`` lookup during test collection. Tests thatdiff --git a/server/tests/test_ws_disconnect_cancellation.py b/server/tests/test_ws_disconnect_cancellation.pynew file mode 100644index 00000000..27c08355--- /dev/null+++ b/server/tests/test_ws_disconnect_cancellation.py@@ -0,0 +1,232 @@+"""A WebSocket disconnect must not interrupt database teardown.++Cancelling a handler task while a session is open used to deliver the+``CancelledError`` inside SQLAlchemy's reset-on-return rollback, which the+pool reports as "Exception during reset or similar" with a traceback.+Three guards fix it: session teardown is shielded, the endpoint drains+handlers briefly before cancelling them, and broadcasts skip sockets+that already saw a close frame.+"""++from __future__ import annotations++import asyncio+import importlib.util+import sys+import uuid+from pathlib import Path+from types import SimpleNamespace+from typing import List, Optional+from unittest.mock import AsyncMock++import pytest+from starlette.websockets import WebSocketState++pytestmark = pytest.mark.unit++_SERVER = Path(__file__).resolve().parents[1]+_LOADED: dict = {}+++def _load(rel: str):+ """Load a real ``core`` module privately; the root conftest stubs ``core.database``.++ Memoised per path: executing ``credentials_database.py`` twice would+ redefine its SQLModel tables on the shared metadata.+ """+ if rel in _LOADED:+ return _LOADED[rel]+ target = (_SERVER / rel).resolve()+ # tests/credentials/conftest.py replaces the stubbed ``core`` package with+ # the real modules at collection time; reuse that module rather than+ # executing the file a second time against the same SQLModel metadata.+ existing = sys.modules.get(f"core.{Path(rel).stem}")+ existing_file = getattr(existing, "__file__", None)+ if existing_file and Path(existing_file).resolve() == target:+ _LOADED[rel] = existing+ return existing+ name = f"tests._ws_cancel_{uuid.uuid4().hex}"+ spec = importlib.util.spec_from_file_location(name, target)+ assert spec is not None and spec.loader is not None+ module = importlib.util.module_from_spec(spec)+ sys.modules[name] = module+ spec.loader.exec_module(module)+ _LOADED[rel] = module+ return module+++class _FakeSession:+ def __init__(self, *, close_gate: Optional[asyncio.Event] = None):+ self.calls: List[str] = []+ self.close_gate = close_gate+ self.closed = False++ async def rollback(self) -> None:+ self.calls.append("rollback")++ async def close(self) -> None:+ self.calls.append("close")+ if self.close_gate is not None:+ await self.close_gate.wait()+ self.closed = True+++def _database(session: _FakeSession):+ module = _load("core/database.py")+ database = module.Database(+ SimpleNamespace(+ database_url="sqlite+aiosqlite:///:memory:",+ database_echo=False,+ database_pool_size=1,+ database_max_overflow=0,+ )+ )+ database.async_session = lambda: session+ return database+++def _credentials_database(session: _FakeSession):+ module = _load("core/credentials_database.py")+ database = object.__new__(module.CredentialsDatabase)+ database._session_factory = lambda: session+ return database+++async def _settle() -> None:+ for _ in range(5):+ await asyncio.sleep(0)+++class TestSessionTeardownIsShielded:+ @pytest.mark.parametrize("factory", [_database, _credentials_database], ids=["workflow-db", "credentials-db"])+ async def test_cancel_inside_the_body_still_rolls_back_and_closes(self, factory):+ session = _FakeSession()+ database = factory(session)+ entered = asyncio.Event()++ async def work():+ async with database.get_session():+ entered.set()+ await asyncio.Event().wait()++ task = asyncio.create_task(work())+ await entered.wait()+ task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await task++ assert session.calls == ["rollback", "close"]+ assert session.closed is True++ @pytest.mark.parametrize("factory", [_database, _credentials_database], ids=["workflow-db", "credentials-db"])+ async def test_cancel_during_close_lets_the_close_finish(self, factory):+ """The cancellation arrives while the pool reset is in progress; the+ reset must complete instead of being torn out from under SQLAlchemy."""+ gate = asyncio.Event()+ session = _FakeSession(close_gate=gate)+ database = factory(session)++ async def work():+ async with database.get_session():+ pass++ task = asyncio.create_task(work())+ while "close" not in session.calls:+ await asyncio.sleep(0)+ task.cancel()+ with pytest.raises(asyncio.CancelledError):+ await task++ assert session.closed is False, "close is still running, shielded from the cancel"+ gate.set()+ await _settle()+ assert session.closed is True++ async def test_exception_in_body_rolls_back_and_closes(self):+ session = _FakeSession()+ database = _database(session)++ with pytest.raises(RuntimeError):+ async with database.get_session():+ raise RuntimeError("boom")++ assert session.calls == ["rollback", "close"]++ async def test_clean_exit_closes_without_rollback(self):+ session = _FakeSession()+ database = _database(session)++ async with database.get_session():+ pass++ assert session.calls == ["close"]+++class TestHandlerDrain:+ async def test_short_handlers_finish_and_long_ones_are_cancelled(self):+ from routers.websocket import _drain_handler_tasks++ fast = asyncio.create_task(asyncio.sleep(0.01))+ slow = asyncio.create_task(asyncio.Event().wait())++ await _drain_handler_tasks({fast, slow}, grace=0.3)++ assert fast.done() and not fast.cancelled(), "a handler that finishes inside the grace is never cancelled"+ assert slow.cancelled()++ async def test_no_tasks_is_a_no_op(self):+ from routers.websocket import _drain_handler_tasks++ await _drain_handler_tasks(set(), grace=0.01)++ def test_endpoints_use_the_drain(self):+ import inspect++ import routers.websocket as ws++ source = inspect.getsource(ws)+ assert source.count("await _drain_handler_tasks(handler_tasks)") == 2, (+ "both /ws/status and /ws/internal must drain instead of cancelling immediately"+ )+++class _Socket:+ """Hashable stand-in for a Starlette WebSocket (SimpleNamespace is not hashable)."""++ def __init__(self, *, client=WebSocketState.CONNECTED, application=WebSocketState.CONNECTED, send=None):+ self.client_state = client+ self.application_state = application+ self.send_text = send or AsyncMock()+++def _socket(**kwargs) -> _Socket:+ return _Socket(**kwargs)+++class TestBroadcastSkipsClosedSockets:+ async def test_closed_socket_is_pruned_without_a_send(self):+ from services.status_broadcaster import StatusBroadcaster++ broadcaster = StatusBroadcaster()+ closing = _socket(client=WebSocketState.DISCONNECTED)+ closed_by_server = _socket(application=WebSocketState.DISCONNECTED)+ live = _socket()+ broadcaster._connections = {closing, closed_by_server, live}++ await broadcaster.broadcast({"type": "node_status"})++ closing.send_text.assert_not_awaited()+ closed_by_server.send_text.assert_not_awaited()+ live.send_text.assert_awaited_once()+ assert broadcaster._connections == {live}++ async def test_send_failure_prunes_the_socket(self):+ from services.status_broadcaster import StatusBroadcaster++ broadcaster = StatusBroadcaster()+ failing = _socket(send=AsyncMock(side_effect=RuntimeError('Cannot call "send" once a close message has been sent.')))+ broadcaster._connections = {failing}++ await broadcaster.broadcast({"type": "node_status"})++ assert broadcaster._connections == set()