Skip to content

Self-improving AI Employees for everyone

fix(ws): stop a client disconnect from cancelling database teardown mid-rollback

trohitg914afa633ec3
diff --git a/CLAUDE.md b/CLAUDE.md
index f7b41d3e..e41331a8 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -2745,6 +2745,7 @@ This function:
- **writeTodos Tool Node**: Dedicated AI tool for task planning connecting to any agent's `input-tools` handle. `TodoService` singleton (`server/services/todo_service.py`) stores JSON-based per-session todo state keyed by workflow_id. Handler broadcasts `phase: "todo_update"` via WebSocket on each update for real-time UI. (The former checklist rendering — `formatTodoOutput()` — lived only in the legacy `ui/OutputDisplayPanel.tsx`, which was deleted as dead code; the active `components/output/OutputPanel.tsx` shows todo results as regular JSON.) Schema uses `TodoItem`/`TodoStatus` Pydantic enum. Skill at `server/skills/assistant/write-todos-skill/SKILL.md` teaches the plan-work-update loop.
- **Temporal Activity Heartbeats**: Activities send `activity.heartbeat()` on every non-matching WebSocket broadcast inside the read loop in `services/temporal/activities.py`. This keeps long-running browser and claude_code_agent operations alive past the 2-minute `heartbeat_timeout`. Start/end heartbeats alone were causing `TIMEOUT_TYPE_HEARTBEAT` failures on ops taking 5-10 minutes. Connection config: `heartbeat=30`, `receive_timeout=None` (each receive is bounded by a 30 s `asyncio.wait_for`; liveness comes from heartbeats, and per-node activities run under a 24 h `start_to_close_timeout`).
- **WebSocket `_safe_send` Guard**: `server/routers/websocket.py` checks `websocket.client_state.name != "CONNECTED"` before sending and logs at `debug` level (not `error`) on failure. Prevents "ASGI message after websocket.close" errors when broadcasts race with disconnects.
+- **WebSocket disconnect must not cancel database teardown**: a client that disconnects within ~100 ms of connecting (Strict Mode double mount, a health probe) used to make the `/ws/status` endpoint cancel its in-flight init-burst handlers mid-statement, so SQLAlchemy's reset-on-return rollback took the `CancelledError` and logged `Exception during reset or similar` with a traceback per handler. Three guards, locked by `tests/test_ws_disconnect_cancellation.py`: `core/session_teardown.py` runs rollback + close under `asyncio.shield` for both `Database.get_session` and `CredentialsDatabase.get_session`; `_drain_handler_tasks` in `routers/websocket.py` waits one second for handlers before cancelling stragglers; `StatusBroadcaster.broadcast` skips and prunes sockets that are no longer `CONNECTED` on either side. Any new session helper must reuse `teardown_session`, and a new `core` module that `core/database.py` imports must also be file-loaded in `tests/conftest.py` (the stubbed `core` package has an empty `__path__`).
- **Claude Code CLI Flag**: the Claude Code agent (`nodes/agent/claude_code_agent/`) uses `--max-budget-usd <amount>` (previously incorrectly `--max-cost`, which caused "unknown option" errors on every run).
- **Persisted-prefix cache contract**: `PersistQueryClientProvider` hydrates entries with the QueryClient's *default* options, so any prefix in `PERSISTED_KEY_PREFIXES` (`client/src/lib/queryPersist.ts`) MUST also have a matching `queryClient.setQueryDefaults(['<prefix>'], { staleTime: FOREVER, gcTime: FOREVER })` declaration in `client/src/lib/queryClient.ts`. Per-call options don't apply on hydration. Canonical set: `nodeSpec`, `nodeGroups`, `skillContent`. `credentialValues` was previously persisted here but was removed per OWASP HTML5 Security Cheat Sheet / ASVS V9.9 — decrypted API keys must never live in `localStorage`. The in-memory TanStack Query cache (`gcTime: ∞`) keeps the credentials form populated for the session lifetime; on reload the panel refetches via WS. `credentialCatalogue` has its own `idb-keyval` warm-start path so it is intentionally not in either list.
- **WebSocket `sendRequest` replay queue**: when the socket is not open, requests enqueue with `AbortController`-backed per-request timeouts (default 30s) and replay on reconnect inside `ws.onopen` before `setIsReady(true)`. Queue capped at 200 with FIFO eviction. Intentional close (`event.code === 1000`) drops the queue; transient closes preserve it. Eliminates indefinite spinners during the 3s reconnect window. Implementation in `client/src/contexts/WebSocketContext.tsx` (`pendingSendQueueRef` + `drainPendingSends`).
diff --git a/docs-internal/errors.md b/docs-internal/errors.md
index 17f9f76c..87fe2b14 100644
--- a/docs-internal/errors.md
+++ b/docs-internal/errors.md
@@ -384,3 +384,13 @@ python -c "import ntpath; print(ntpath.join(r'D:\ws\AI_Employee_1', '2:processMa
```
**Fix** (`21fb1a9a`): `core.paths.safe_path_component` sanitizes a wire identifier before it becomes a path component; `processManager` and `cli_agent/session.py` route the node id through it. The guardrail error now names the rejected resolved path and says to omit `cwd` (the designed happy path — the process-manager skill never passes one), and `httpRequest` maps `ConnectError` / `TimeoutException` to `NodeUserError`. Rule: never join a node id, model-supplied name, or any other wire identifier into a path raw — go through `safe_path_component`. Locked by `tests/nodes/test_process_manager_cwd.py`.
+
+---
+
+## 13. `Exception during reset or similar` / `CancelledError` in aiosqlite rollback on WebSocket disconnect
+
+**Symptom**: Right after `[StatusBroadcaster] Client connected`, the operator log shows one or more SQLAlchemy tracebacks ending in `asyncio.exceptions.CancelledError` from `aiosqlite/core.py ... rollback`, then `[StatusBroadcaster] Send failed: Cannot call "send" once a close message has been sent`, then `Client disconnected`. Typical trigger: a client that disconnects within ~100 ms of connecting (React Strict Mode double mount in `company dev`, a health probe), i.e. while the connect-time init burst (chat / console / terminal history, credential probes) is still running.
+
+**Root cause**: the `/ws/status` endpoint's cleanup cancelled every in-flight handler task the instant the socket closed. A handler cancelled while inside a database call receives `CancelledError` at the aiosqlite await; the session teardown then returns the connection to the pool, whose reset-on-return rollback is itself interrupted by the pending cancellation. SQLAlchemy's `_finalize_fairy` logs the traceback at ERROR, invalidates the connection, and re-raises, so nothing was actually broken (the pool rebuilt the connection) but every reconnect produced a traceback per cancelled handler. The `Send failed` warning came from a broadcast racing the close frame.
+
+**Fix**: `core/session_teardown.py` runs rollback + close under `asyncio.shield` in both `Database.get_session` and `CredentialsDatabase.get_session`, so a cancelled caller cannot interrupt the pool reset; `_drain_handler_tasks` in `routers/websocket.py` gives in-flight handlers a one-second grace to finish before cancelling stragglers; `StatusBroadcaster.broadcast` skips sockets that are no longer `CONNECTED` on either side and prunes them quietly. Locked by `tests/test_ws_disconnect_cancellation.py`.
diff --git a/docs-internal/status_broadcaster.md b/docs-internal/status_broadcaster.md
index 537d78f7..a75d6e6c 100644
--- a/docs-internal/status_broadcaster.md
+++ b/docs-internal/status_broadcaster.md
@@ -219,7 +219,8 @@ Android service nodes (`batteryMonitor`, `wifiAutomation`, etc.) check `androidS
- Responds to `ping` with `{"type": "pong"}`.
- On `get_status`, returns the full current status snapshot.
-- On disconnect, removes the WebSocket from `StatusBroadcaster._connections`.
+- On disconnect, waits up to one second for the socket's in-flight handler tasks to finish (`_drain_handler_tasks`), cancels any still running, then removes the WebSocket from `StatusBroadcaster._connections`. The grace exists because a client that reconnects ~100 ms after connecting would otherwise cancel the init-burst handlers inside a database call; session teardown is additionally shielded from cancellation in `core/session_teardown.py` (see errors.md entry 13).
+- `broadcast()` skips and prunes sockets that are no longer `CONNECTED` on either side, so a broadcast racing a close frame does not log a warning per message.
## Error Handling
diff --git a/server/core/credentials_database.py b/server/core/credentials_database.py
index eaba77f7..d4ba1bf1 100644
--- a/server/core/credentials_database.py
+++ b/server/core/credentials_database.py
@@ -12,7 +12,10 @@ The database is separate from the main application database (workflow.db) to:
import hashlib
import logging
+import asyncio
from contextlib import asynccontextmanager
+
+from core.session_teardown import teardown_session
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
@@ -151,9 +154,24 @@ class CredentialsDatabase:
@asynccontextmanager
async def get_session(self):
- """Get async database session."""
- async with self._session_factory() as session:
+ """Get async database session.
+
+ Same cancellation-safe teardown as ``core.database.Database``:
+ rollback + close run under ``asyncio.shield`` so a cancelled
+ caller (a WebSocket client disconnecting during the connect-time
+ credential probes) cannot interrupt the pool's reset rollback.
+ """
+ session = self._session_factory()
+ try:
yield session
+ except asyncio.CancelledError:
+ await asyncio.shield(teardown_session(session, rollback=True))
+ raise
+ except Exception:
+ await asyncio.shield(teardown_session(session, rollback=True))
+ raise
+ else:
+ await asyncio.shield(teardown_session(session, rollback=False))
# --- Metadata Operations ---
diff --git a/server/core/database.py b/server/core/database.py
index 7b0a7759..9e8d8234 100644
--- a/server/core/database.py
+++ b/server/core/database.py
@@ -1,5 +1,6 @@
"""Modern async database service with SQLModel and SQLAlchemy 2.0."""
+import asyncio
import json
import inspect
import secrets
@@ -13,6 +14,8 @@ from sqlalchemy.exc import IntegrityError
from sqlalchemy import case, text, func, update, or_
from contextlib import asynccontextmanager
+from core.session_teardown import teardown_session as _teardown_session
+
from core.config import Settings
from models.database import (
NodeParameter,
@@ -375,18 +378,31 @@ class Database:
@asynccontextmanager
async def get_session(self):
- """Get async database session."""
+ """Get async database session.
+
+ Teardown (rollback + close) runs under ``asyncio.shield``. When the
+ calling task is cancelled mid-statement -- a WebSocket client that
+ disconnects while its connect-time handlers are still reading --
+ the cancellation would otherwise land inside the pool's
+ reset-on-return rollback; SQLAlchemy then logs "Exception during
+ reset or similar" with a full traceback and discards the
+ connection. Shielding lets the reset finish while the
+ CancelledError still propagates to the caller.
+ """
if not self.async_session:
raise RuntimeError("Database not initialized")
- async with self.async_session() as session:
- try:
- yield session
- except Exception:
- await session.rollback()
- raise
- finally:
- await session.close()
+ session = self.async_session()
+ try:
+ yield session
+ except asyncio.CancelledError:
+ await asyncio.shield(_teardown_session(session, rollback=True))
+ raise
+ except Exception:
+ await asyncio.shield(_teardown_session(session, rollback=True))
+ raise
+ else:
+ await asyncio.shield(_teardown_session(session, rollback=False))
async def _migrate_workflow_controls(self):
"""Backfill control-plane columns when upgrading an early preview DB."""
diff --git a/server/core/session_teardown.py b/server/core/session_teardown.py
new file mode 100644
index 00000000..28e8b63c
--- /dev/null
+++ b/server/core/session_teardown.py
@@ -0,0 +1,26 @@
+"""Cancellation-safe teardown for async SQLAlchemy sessions.
+
+Shared by ``core.database`` and ``core.credentials_database``. Both wrap
+the call in ``asyncio.shield`` so a task cancelled while a session is
+open (a WebSocket client disconnecting mid-request) cannot interrupt
+the pool's reset-on-return rollback. Without the shield SQLAlchemy's
+``_finalize_fairy`` logs "Exception during reset or similar" with a
+traceback, invalidates the connection, and re-raises the
+CancelledError; the connection is then rebuilt on the next checkout.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+async def teardown_session(session: Any, *, rollback: bool) -> None:
+ """Roll back (optionally) and close ``session``, always closing."""
+ try:
+ if rollback:
+ await session.rollback()
+ finally:
+ await session.close()
+
+
+__all__ = ["teardown_session"]
diff --git a/server/routers/websocket.py b/server/routers/websocket.py
index d4953c1f..d4c9ab57 100644
--- a/server/routers/websocket.py
+++ b/server/routers/websocket.py
@@ -56,6 +56,27 @@ async def _safe_send(websocket: WebSocket, data: dict):
logger.debug("[WebSocket] Send skipped (connection closed): %s", e)
+# How long a disconnecting socket waits for its in-flight handlers before
+# cancelling them. The connect-time init burst is a handful of short DB
+# reads; a client that reconnects ~100 ms later (React Strict Mode double
+# mount, a health probe) used to cancel those mid-statement, which
+# SQLAlchemy reported as "Exception during reset or similar" with a
+# traceback per handler. Long handlers (execute_node parked on an event
+# waiter) are still cancelled once the grace expires.
+_HANDLER_DRAIN_GRACE_SECONDS = 1.0
+
+
+async def _drain_handler_tasks(handler_tasks: Set[asyncio.Task], grace: float = _HANDLER_DRAIN_GRACE_SECONDS) -> None:
+ """Let in-flight handler tasks finish for ``grace`` seconds, then cancel the rest."""
+ tasks = [task for task in handler_tasks if not task.done()]
+ if not tasks:
+ return
+ _done, pending = await asyncio.wait(tasks, timeout=grace)
+ for task in pending:
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+
+
# Type for message handlers
MessageHandler = Callable[[Dict[str, Any], WebSocket], Awaitable[Dict[str, Any]]]
@@ -1708,14 +1729,9 @@ async def websocket_status_endpoint(websocket: WebSocket):
if not isinstance(exc, (WebSocketDisconnect, asyncio.CancelledError, KeyboardInterrupt, SystemExit)):
logger.error(f"[WebSocket] TaskGroup error: {exc}")
finally:
- # Cancel any running handler tasks on disconnect
- for task in list(handler_tasks):
- if not task.done():
- task.cancel()
-
- # Wait for tasks to finish cancellation
- if handler_tasks:
- await asyncio.gather(*handler_tasks, return_exceptions=True)
+ # Let short handlers finish, then cancel stragglers (see
+ # _drain_handler_tasks for why this is not an immediate cancel).
+ await _drain_handler_tasks(handler_tasks)
# Cleanup
_handler_tasks.pop(websocket, None)
@@ -1806,12 +1822,7 @@ async def websocket_internal_endpoint(websocket: WebSocket):
if not isinstance(exc, (WebSocketDisconnect, asyncio.CancelledError, KeyboardInterrupt, SystemExit)):
logger.error(f"[WebSocket Internal] TaskGroup error: {exc}")
finally:
- for task in list(handler_tasks):
- if not task.done():
- task.cancel()
-
- if handler_tasks:
- await asyncio.gather(*handler_tasks, return_exceptions=True)
+ await _drain_handler_tasks(handler_tasks)
@router.get("/ws/info")
diff --git a/server/services/status_broadcaster.py b/server/services/status_broadcaster.py
index 2ac0233d..2be8198c 100644
--- a/server/services/status_broadcaster.py
+++ b/server/services/status_broadcaster.py
@@ -8,6 +8,7 @@ import asyncio
import orjson
from typing import Set, Dict, Any, Optional, List
from fastapi import WebSocket
+from starlette.websockets import WebSocketState
from opentelemetry import trace
from core.logging import get_logger
@@ -59,6 +60,14 @@ def _elide_for_cache(output: Any) -> Any:
tracer = trace.get_tracer(__name__)
+def _socket_open(connection: WebSocket) -> bool:
+ """True while both sides of the socket are still CONNECTED."""
+ return (
+ getattr(connection, "client_state", WebSocketState.CONNECTED) is WebSocketState.CONNECTED
+ and getattr(connection, "application_state", WebSocketState.CONNECTED) is WebSocketState.CONNECTED
+ )
+
+
class StatusBroadcaster:
"""Manages WebSocket connections and broadcasts status updates."""
@@ -232,10 +241,19 @@ class StatusBroadcaster:
async def send_to_client(connection: WebSocket):
"""Send message to a single client."""
+ if not _socket_open(connection):
+ # The client already sent (or received) a close frame; the
+ # endpoint's disconnect() has not run yet. Drop it silently
+ # instead of warning once per broadcast in the gap.
+ disconnected.add(connection)
+ return
try:
await connection.send_text(message_bytes)
except Exception as e:
- logger.warning(f"[StatusBroadcaster] Send failed: {e}")
+ if _socket_open(connection):
+ logger.warning(f"[StatusBroadcaster] Send failed: {e}")
+ else:
+ logger.debug("[StatusBroadcaster] Send skipped, client closed: %s", e)
disconnected.add(connection)
# Execute all sends concurrently with TaskGroup
diff --git a/server/tests/conftest.py b/server/tests/conftest.py
index 977f0a81..4bd9c48a 100644
--- a/server/tests/conftest.py
+++ b/server/tests/conftest.py
@@ -145,6 +145,19 @@ sys.modules["core.encryption"] = _encryption_mod
_encryption_spec.loader.exec_module(_encryption_mod)
setattr(_core_pkg, "encryption", _encryption_mod)
+# core.session_teardown is stdlib-only (the shielded rollback + close both
+# databases run under a cancelled caller); the real core/database.py and
+# core/credentials_database.py import it at module load, and tests that
+# file-load those modules need it resolvable under the stubbed package.
+_session_teardown_spec = _importlib_util.spec_from_file_location(
+ "core.session_teardown",
+ SERVER_DIR / "core" / "session_teardown.py",
+)
+_session_teardown_mod = _importlib_util.module_from_spec(_session_teardown_spec)
+sys.modules["core.session_teardown"] = _session_teardown_mod
+_session_teardown_spec.loader.exec_module(_session_teardown_mod)
+setattr(_core_pkg, "session_teardown", _session_teardown_mod)
+
# core.paths — central path resolution. Stub the public surface with
# tmpdir-rooted Paths so plugin module imports don't trip over the
# real ``Path.home()`` lookup during test collection. Tests that
diff --git a/server/tests/test_ws_disconnect_cancellation.py b/server/tests/test_ws_disconnect_cancellation.py
new file mode 100644
index 00000000..27c08355
--- /dev/null
+++ b/server/tests/test_ws_disconnect_cancellation.py
@@ -0,0 +1,232 @@
+"""A WebSocket disconnect must not interrupt database teardown.
+
+Cancelling a handler task while a session is open used to deliver the
+``CancelledError`` inside SQLAlchemy's reset-on-return rollback, which the
+pool reports as "Exception during reset or similar" with a traceback.
+Three guards fix it: session teardown is shielded, the endpoint drains
+handlers briefly before cancelling them, and broadcasts skip sockets
+that already saw a close frame.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import importlib.util
+import sys
+import uuid
+from pathlib import Path
+from types import SimpleNamespace
+from typing import List, Optional
+from unittest.mock import AsyncMock
+
+import pytest
+from starlette.websockets import WebSocketState
+
+pytestmark = pytest.mark.unit
+
+_SERVER = Path(__file__).resolve().parents[1]
+_LOADED: dict = {}
+
+
+def _load(rel: str):
+ """Load a real ``core`` module privately; the root conftest stubs ``core.database``.
+
+ Memoised per path: executing ``credentials_database.py`` twice would
+ redefine its SQLModel tables on the shared metadata.
+ """
+ if rel in _LOADED:
+ return _LOADED[rel]
+ target = (_SERVER / rel).resolve()
+ # tests/credentials/conftest.py replaces the stubbed ``core`` package with
+ # the real modules at collection time; reuse that module rather than
+ # executing the file a second time against the same SQLModel metadata.
+ existing = sys.modules.get(f"core.{Path(rel).stem}")
+ existing_file = getattr(existing, "__file__", None)
+ if existing_file and Path(existing_file).resolve() == target:
+ _LOADED[rel] = existing
+ return existing
+ name = f"tests._ws_cancel_{uuid.uuid4().hex}"
+ spec = importlib.util.spec_from_file_location(name, target)
+ assert spec is not None and spec.loader is not None
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[name] = module
+ spec.loader.exec_module(module)
+ _LOADED[rel] = module
+ return module
+
+
+class _FakeSession:
+ def __init__(self, *, close_gate: Optional[asyncio.Event] = None):
+ self.calls: List[str] = []
+ self.close_gate = close_gate
+ self.closed = False
+
+ async def rollback(self) -> None:
+ self.calls.append("rollback")
+
+ async def close(self) -> None:
+ self.calls.append("close")
+ if self.close_gate is not None:
+ await self.close_gate.wait()
+ self.closed = True
+
+
+def _database(session: _FakeSession):
+ module = _load("core/database.py")
+ database = module.Database(
+ SimpleNamespace(
+ database_url="sqlite+aiosqlite:///:memory:",
+ database_echo=False,
+ database_pool_size=1,
+ database_max_overflow=0,
+ )
+ )
+ database.async_session = lambda: session
+ return database
+
+
+def _credentials_database(session: _FakeSession):
+ module = _load("core/credentials_database.py")
+ database = object.__new__(module.CredentialsDatabase)
+ database._session_factory = lambda: session
+ return database
+
+
+async def _settle() -> None:
+ for _ in range(5):
+ await asyncio.sleep(0)
+
+
+class TestSessionTeardownIsShielded:
+ @pytest.mark.parametrize("factory", [_database, _credentials_database], ids=["workflow-db", "credentials-db"])
+ async def test_cancel_inside_the_body_still_rolls_back_and_closes(self, factory):
+ session = _FakeSession()
+ database = factory(session)
+ entered = asyncio.Event()
+
+ async def work():
+ async with database.get_session():
+ entered.set()
+ await asyncio.Event().wait()
+
+ task = asyncio.create_task(work())
+ await entered.wait()
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+
+ assert session.calls == ["rollback", "close"]
+ assert session.closed is True
+
+ @pytest.mark.parametrize("factory", [_database, _credentials_database], ids=["workflow-db", "credentials-db"])
+ async def test_cancel_during_close_lets_the_close_finish(self, factory):
+ """The cancellation arrives while the pool reset is in progress; the
+ reset must complete instead of being torn out from under SQLAlchemy."""
+ gate = asyncio.Event()
+ session = _FakeSession(close_gate=gate)
+ database = factory(session)
+
+ async def work():
+ async with database.get_session():
+ pass
+
+ task = asyncio.create_task(work())
+ while "close" not in session.calls:
+ await asyncio.sleep(0)
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+
+ assert session.closed is False, "close is still running, shielded from the cancel"
+ gate.set()
+ await _settle()
+ assert session.closed is True
+
+ async def test_exception_in_body_rolls_back_and_closes(self):
+ session = _FakeSession()
+ database = _database(session)
+
+ with pytest.raises(RuntimeError):
+ async with database.get_session():
+ raise RuntimeError("boom")
+
+ assert session.calls == ["rollback", "close"]
+
+ async def test_clean_exit_closes_without_rollback(self):
+ session = _FakeSession()
+ database = _database(session)
+
+ async with database.get_session():
+ pass
+
+ assert session.calls == ["close"]
+
+
+class TestHandlerDrain:
+ async def test_short_handlers_finish_and_long_ones_are_cancelled(self):
+ from routers.websocket import _drain_handler_tasks
+
+ fast = asyncio.create_task(asyncio.sleep(0.01))
+ slow = asyncio.create_task(asyncio.Event().wait())
+
+ await _drain_handler_tasks({fast, slow}, grace=0.3)
+
+ assert fast.done() and not fast.cancelled(), "a handler that finishes inside the grace is never cancelled"
+ assert slow.cancelled()
+
+ async def test_no_tasks_is_a_no_op(self):
+ from routers.websocket import _drain_handler_tasks
+
+ await _drain_handler_tasks(set(), grace=0.01)
+
+ def test_endpoints_use_the_drain(self):
+ import inspect
+
+ import routers.websocket as ws
+
+ source = inspect.getsource(ws)
+ assert source.count("await _drain_handler_tasks(handler_tasks)") == 2, (
+ "both /ws/status and /ws/internal must drain instead of cancelling immediately"
+ )
+
+
+class _Socket:
+ """Hashable stand-in for a Starlette WebSocket (SimpleNamespace is not hashable)."""
+
+ def __init__(self, *, client=WebSocketState.CONNECTED, application=WebSocketState.CONNECTED, send=None):
+ self.client_state = client
+ self.application_state = application
+ self.send_text = send or AsyncMock()
+
+
+def _socket(**kwargs) -> _Socket:
+ return _Socket(**kwargs)
+
+
+class TestBroadcastSkipsClosedSockets:
+ async def test_closed_socket_is_pruned_without_a_send(self):
+ from services.status_broadcaster import StatusBroadcaster
+
+ broadcaster = StatusBroadcaster()
+ closing = _socket(client=WebSocketState.DISCONNECTED)
+ closed_by_server = _socket(application=WebSocketState.DISCONNECTED)
+ live = _socket()
+ broadcaster._connections = {closing, closed_by_server, live}
+
+ await broadcaster.broadcast({"type": "node_status"})
+
+ closing.send_text.assert_not_awaited()
+ closed_by_server.send_text.assert_not_awaited()
+ live.send_text.assert_awaited_once()
+ assert broadcaster._connections == {live}
+
+ async def test_send_failure_prunes_the_socket(self):
+ from services.status_broadcaster import StatusBroadcaster
+
+ broadcaster = StatusBroadcaster()
+ failing = _socket(send=AsyncMock(side_effect=RuntimeError('Cannot call "send" once a close message has been sent.')))
+ broadcaster._connections = {failing}
+
+ await broadcaster.broadcast({"type": "node_status"})
+
+ assert broadcaster._connections == set()