Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Bring the multi-pod stack (#809, #863, #864, #865, #867) to main (#886)

Abel Dantas66bad451fbcc
diff --git a/CLAUDE.md b/CLAUDE.md
index 1891489f5..c411cc6fa 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -77,7 +77,8 @@ just test -k "test_name" # run specific test
- `stores/` — query methods and domain-specific data access
- `migrations/` — Alembic migrations (`env.py`, `versions/`)
- `room_service.py` / `rooms_yaml.py` — Room lifecycle, configuration, provisioning
-- `clients/` — room participants: `actor.py` (identity, membership, writes: `HumanActor`, `AgentActor`, `SystemActor`) and `consumer.py` (the delivery loop and hooks: `AgentConsumer`, `CommandConsumer`, `WorkspaceConsumer`)
+- `clients/` — room participants: `actor.py` (identity, membership, writes: `HumanActor`, `AgentActor`, `SystemActor`) and `consumer.py` (the delivery loop and hooks: `CommandConsumer`, `WorkspaceConsumer`)
+- `delivery/` decides who a row is for: `addressing.py` (the rules), `router.py` (`MessageRouter`, which decides every row for every agent in its room as the row is written, in the same transaction) and `recipient.py` (`AgentRecipient`, one agent's decisions and the replies posted on its behalf)
- `bridges/` — External integrations
- `agent/` — Agent Bridge (HTTP API, MCP server, server-side connectors); `protocol/agent_core.py` is `AgentCore`, `protocol/agent_connections.py` holds each `AgentConnection`
- `collaboration/` — Collaboration Bridge: one `CollaborationCore` per workspace, driving a `PlatformAdapter` (Slack, Mattermost, Discord, Teams, Telegram)
@@ -88,19 +89,24 @@ just test -k "test_name" # run specific test
- Async throughout: all I/O is async (DB, external APIs)
- Dependency injection: stores and services are injected, not global singletons
- Session management: API endpoints use middleware-provided sessions; background work creates sessions explicitly
-- Every room participant is an actor writing to the `messages` table through the transport port; only a participant that reads the room has a consumer. Human actors (one `HumanActor` per platform user) read nothing, because the bridge's `WorkspaceConsumer` reads for the whole workspace
+- Every room participant is an actor writing to the `messages` table through the transport port; only a participant that reads the room has a consumer. Human actors (one `HumanActor` per platform user) read nothing, because the bridge's `WorkspaceConsumer` reads for the whole workspace. Agents read nothing either: the router decides, as each row is written, which agents it is for and stores their own events with it
**The message bus is PostgreSQL** (`transport/postgres.py`). Switch once ran on a
Matrix homeserver; that is gone, and no Matrix server or client library is
involved. A send is an `INSERT` into `messages`, with a per-room advisory lock
assigning `seq` (`db/stores/message_store.py`). The `messages_notify` trigger
calls `pg_notify('switch_message', …)` with the room, `seq` and row id, delivered
-on commit (`db/notify_ddl.py`). One `MessageListener` (`messages/notify.py`)
-holds a dedicated `LISTEN` connection and wakes the subscribers for that room;
-each client then reads the rows after its own cursor, 200 at a time. The
-notification is a wake-up, never the payload. Invites and presence travel over
-in-process buses (`transport/invites.py`, `transport/ephemeral.py`), which is
-part of why switch-core runs as a single replica. Names such as
+on commit (`db/notify_ddl.py`). Every row is written by the router
+(`delivery/router.py`), which decides in the same transaction which agents in
+the room it is for and stores their own events (`agent_events`) with it. One
+`MessageListener` (`messages/notify.py`) holds a dedicated `LISTEN` connection
+and wakes the subscribers for that room; each client then reads the rows after
+its own cursor, 200 at a time. The notification is a wake-up, never the
+payload. Membership changes reach every
+process through a `NOTIFY` trigger on `client_rooms` and the signal bus
+(`signals.py`, `transport/membership_signals.py`); presence still travels over
+an in-process bus (`transport/ephemeral.py`), which is part of why switch-core
+runs as a single replica. Names such as
the `matrix_room_id` / `matrix_user_id` columns and wire fields, the
`@localpart:server` id shape and `m.room.message` content types are kept as
stable identifiers and wire shapes, not as a sign that Matrix is in use. In
@@ -112,12 +118,17 @@ Matrix homeserver; that is gone, and no Matrix server or client library is
involved. A send is an `INSERT` into `messages`, with a per-room advisory lock
assigning `seq` (`db/stores/message_store.py`). The `messages_notify` trigger
calls `pg_notify('switch_message', …)` with the room, `seq` and row id, delivered
-on commit (`db/notify_ddl.py`). One `MessageListener` (`messages/notify.py`)
-holds a dedicated `LISTEN` connection and wakes the subscribers for that room;
-each client then reads the rows after its own cursor, 200 at a time. The
-notification is a wake-up, never the payload. Invites and presence travel over
-in-process buses (`transport/invites.py`, `transport/ephemeral.py`), which is
-part of why switch-core runs as a single replica. Names such as
+on commit (`db/notify_ddl.py`). Every row is written by the router
+(`delivery/router.py`), which decides in the same transaction which agents in
+the room it is for and stores their own events (`agent_events`) with it. One
+`MessageListener` (`messages/notify.py`) holds a dedicated `LISTEN` connection
+and wakes the subscribers for that room; each client then reads the rows after
+its own cursor, 200 at a time. The notification is a wake-up, never the
+payload. Membership changes reach every
+process through a `NOTIFY` trigger on `client_rooms` and the signal bus
+(`signals.py`, `transport/membership_signals.py`); presence still travels over
+an in-process bus (`transport/ephemeral.py`), which is part of why switch-core
+runs as a single replica. Names such as
`matrix_room_id`, `matrix_user_id`, `MATRIX_SERVER_NAME`, the `@localpart:server`
id shape and `m.room.message` content types are kept as stable identifiers and
wire shapes, not as a sign that Matrix is in use.
diff --git a/core/switch_core/bridges/agent/api/handlers.py b/core/switch_core/bridges/agent/api/handlers.py
index 594c0c5f7..8c2b85850 100644
--- a/core/switch_core/bridges/agent/api/handlers.py
+++ b/core/switch_core/bridges/agent/api/handlers.py
@@ -86,6 +86,7 @@ from switch_core.bridges.agent.dependencies import (
from switch_core.bridges.agent.protocol import positions
from switch_core.bridges.agent.protocol.agent_connections import (
HEARTBEAT_INTERVAL_SECONDS,
+ MAX_ID_LENGTH,
AgentConnection,
ClientDeclaration,
Closure,
@@ -937,6 +938,12 @@ async def _open_connection(
"UUID and reuse it when reconnecting so the connection survives the "
"drop",
)
+ if len(connection_id) > MAX_ID_LENGTH:
+ raise HTTPException(
+ status_code=400,
+ detail=f"connection_id must be at most {MAX_ID_LENGTH} characters, and "
+ f"this one is {len(connection_id)}; generate a UUID",
+ )
if scope not in ("single", "all"):
raise HTTPException(
status_code=400, detail=f"scope must be 'single' or 'all', got {scope!r}"
@@ -956,18 +963,27 @@ async def _open_connection(
reads_conversation=event_filter == "all",
)
- conn = _register_connection(
- protocol=protocol,
- agent=agent,
- connection_id=connection_id,
- scope=scope,
- event_filter=event_filter,
- spawn_capable=spawn_capable,
- position=position,
- declaration=declaration,
- expected_generation=expected_generation,
- transport=transport,
- )
+ # Opened as a recorded move, so a reattach is fenced on the incarnation
+ # every process sees, and the connection is recorded for them before the
+ # client is told it is open. Recording it is refused, as opening it is,
+ # when the id is another agent's connection's.
+ try:
+ async with protocol.connections.slots(agent.id):
+ async with protocol.connections.recorded(agent.id):
+ conn = _register_connection(
+ protocol=protocol,
+ agent=agent,
+ connection_id=connection_id,
+ scope=scope,
+ event_filter=event_filter,
+ spawn_capable=spawn_capable,
+ position=position,
+ declaration=declaration,
+ expected_generation=expected_generation,
+ transport=transport,
+ )
+ except UnknownConnectionError as exc:
+ raise HTTPException(status_code=409, detail=str(exc)) from exc
if lost is not None:
conn.resume_lost = True
conn.resume_requested = lost
@@ -1006,7 +1022,8 @@ async def _open_connection(
# share connections keeps the slot, and the supervisor's restored
# stream 409s and retries forever.
async with protocol.connections.slots(agent.id):
- protocol.connections.claim_room(conn, room_id, takeover=True)
+ async with protocol.connections.recorded(agent.id):
+ protocol.connections.claim_room(conn, room_id, takeover=True)
# The room's unread count follows the slot: whoever is told how
# far behind the room is has to be the one whose reading clears
# it. Recorded with the slots still held, so a later claim of
@@ -1143,7 +1160,23 @@ async def connection_beat(
recovered by reopening.
"""
try:
- conn = _record_beat(protocol, agent.id, req.connection_id, req.generation)
+ try:
+ conn = _record_beat(protocol, agent.id, req.connection_id, req.generation)
+ except (
+ UnknownConnectionError,
+ SupersededConnectionError,
+ UnfencedBeatError,
+ ):
+ if _held_here(protocol, req.connection_id) and not _presents_later(
+ protocol, req.connection_id, req.generation
+ ):
+ raise
+ # The beat reached a process that does not hold the stream, or
+ # that held it before a later incarnation was opened elsewhere,
+ # and its copy of the connection may be a moment old: a refusal is
+ # final for the client, so it is given only on the recorded row.
+ await protocol.connections.refresh(agent.id)
+ conn = _record_beat(protocol, agent.id, req.connection_id, req.generation)
except (
NoStreamAttachedError,
SupersededConnectionError,
@@ -1348,6 +1381,51 @@ async def _receive_pongs(
return refusal
+def _held_here(protocol: AgentCore, connection_id: str) -> bool:
+ conn = protocol.connections.get(connection_id)
+ return conn is not None and conn.held_by is None
+
+
+def _presents_later(
+ protocol: AgentCore, connection_id: str, generation: int | None
+) -> bool:
+ """Whether the caller names a later incarnation than this process has.
+
+ Every incarnation of a connection is later than the ones before it, so a
+ later one presented was opened through another process after this one
+ last read the rows: this process's copy is the stale one, not the
+ caller's."""
+ conn = protocol.connections.get(connection_id)
+ return (
+ conn is not None
+ and generation is not None
+ and generation > conn.stream_generation
+ )
+
+
+async def _admitted_connection(
+ protocol: AgentCore,
+ agent: Agent,
+ req: ConnectionSubscribeRequest | ConnectionPlacementsRequest,
+) -> AgentConnection:
+ """The first check, before anything is read on the caller's behalf.
+
+ A connection held by another process is checked against this process's
+ copy, which may be a moment old, and so is one held here when the caller
+ names a later incarnation, so a refusal is checked again on the recorded
+ row before it is given. The check that decides is `_current_connection`,
+ made again inside the recorded move."""
+ try:
+ return _current_connection(protocol, agent, req)
+ except HTTPException:
+ if _held_here(protocol, req.connection_id) and not _presents_later(
+ protocol, req.connection_id, req.generation
+ ):
+ raise
+ await protocol.connections.refresh(agent.id)
+ return _current_connection(protocol, agent, req)
+
+
def _current_connection(
protocol: AgentCore,
agent: Agent,
@@ -1391,7 +1469,7 @@ async def connection_subscribe(
door names no session, so the connection is the whole of what it is, and
replacing is what it has always been promised.
"""
- conn = _current_connection(protocol, agent, req)
+ conn = await _admitted_connection(protocol, agent, req)
try:
await protocol.require_room_member(agent.id, req.room_id)
@@ -1401,30 +1479,34 @@ async def connection_subscribe(
raise HTTPException(status_code=403, detail=str(exc)) from exc
async with protocol.connections.slots(agent.id):
- departing = (
- frozenset(conn.rooms - {req.room_id})
- if conn.scope == "single"
- else frozenset()
- )
- # Named again now the wait for the slots is over, and with nothing
- # awaited between here and the write: a client displaced while it waited
- # would otherwise move a room on the connection its successor holds.
- conn = _current_connection(protocol, agent, req)
- try:
- evicted = protocol.connections.claim_room(
- conn, req.room_id, takeover=req.takeover
+ async with protocol.connections.recorded(agent.id):
+ # Named again now the wait for the slots is over, and with nothing
+ # awaited between here and the write: a client displaced while it
+ # waited would otherwise move a room on the connection its successor
+ # holds.
+ conn = _current_connection(protocol, agent, req)
+ departing = (
+ frozenset(conn.rooms - {req.room_id})
+ if conn.scope == "single"
+ else frozenset()
)
- except RoomOccupiedError as exc:
- raise HTTPException(status_code=409, detail=str(exc)) from exc
+ try:
+ evicted = protocol.connections.claim_room(
+ conn, req.room_id, takeover=req.takeover
+ )
+ except RoomOccupiedError as exc:
+ raise HTTPException(status_code=409, detail=str(exc)) from exc
- for departed in departing:
- protocol.connections.release_room(conn, departed)
+ for departed in departing:
+ protocol.connections.release_room(conn, departed)
# A room slot changes hands here as much as it does on the stream URL
# or in connect_to_room, and the room's unread count follows it: the
# holder being told how far behind the room is has to be the one whose
- # reading clears it. Recorded with the slots still held, so a later
- # claim of the room cannot have its count overwritten by this one.
+ # reading clears it. Handed over after the claim is recorded, so a
+ # claim that fails hands nothing over, and before the slots are let go,
+ # so a later claim of the room cannot have its count overwritten by
+ # this one.
await protocol.event_buffer.take_counting(
agent.tenant_id, agent.id, conn.id, req.room_id
)
@@ -1458,9 +1540,12 @@ async def connection_unsubscribe(
protocol: Annotated[AgentCore, Depends(get_protocol)],
) -> dict[str, Any]:
"""Release a room, returning coverage to any all-scope connection."""
- conn = _current_connection(protocol, agent, req)
+ conn = await _admitted_connection(protocol, agent, req)
- async with protocol.connections.slots(agent.id):
+ async with (
+ protocol.connections.slots(agent.id),
+ protocol.connections.recorded(agent.id),
+ ):
conn = _current_connection(protocol, agent, req)
protocol.connections.release_room(conn, req.room_id)
return {"ok": True, "rooms": sorted(conn.rooms)}
@@ -1479,15 +1564,22 @@ async def connection_placements(
here after each change and on every (re)connect, so a restart of either
side converges on what the watcher knows. A room another connection holds
is taken over, as `connect_to_room` takes it, and that connection is sent
- `room_released`. Every room must be one the agent belongs to; one that is
- not refuses the whole request and changes nothing.
+ `room_released`. Every room must be one the agent belongs to, and every
+ session id at most `MAX_ID_LENGTH` characters; a request that breaks
+ either is refused whole and changes nothing.
"""
if agent_id != agent.id:
raise HTTPException(
status_code=403,
detail=f"authenticated as agent {agent.id}, not {agent_id}",
)
- conn = _current_connection(protocol, agent, req)
+ if any(len(session_id) > MAX_ID_LENGTH for session_id in req.placements):
+ raise HTTPException(
+ status_code=400,
+ detail=f"a session id must be at most {MAX_ID_LENGTH} characters; "
+ "nothing was placed",
+ )
+ conn = await _admitted_connection(protocol, agent, req)
for room_id in sorted(set(req.placements.values())):
try:
@@ -1498,15 +1590,17 @@ async def connection_placements(
raise HTTPException(status_code=403, detail=str(exc)) from exc
async with protocol.connections.slots(agent.id):
- conn = _current_connection(protocol, agent, req)
- before = protocol.connections.connection_placements(conn)
- try:
- released = protocol.connections.replace_placements(conn, req.placements)
- except ValueError as exc:
- raise HTTPException(status_code=400, detail=str(exc)) from exc
+ async with protocol.connections.recorded(agent.id):
+ conn = _current_connection(protocol, agent, req)
+ before = protocol.connections.connection_placements(conn)
+ try:
+ released = protocol.connections.replace_placements(conn, req.placements)
+ except ValueError as exc:
+ raise HTTPException(status_code=400, detail=str(exc)) from exc
- # Each moved session takes its room's unread count, with the slots
- # still held, so a later claim of the room cannot have its count
+ # Each moved session takes its room's unread count: after the move is
+ # recorded, so a move that fails hands nothing over, and before the
+ # slots are let go, so a later claim of the room cannot have its count
# overwritten by this one.
for session_id, room_id in req.placements.items():
if before.get(session_id) != room_id:
diff --git a/core/switch_core/bridges/agent/api/operations.py b/core/switch_core/bridges/agent/api/operations.py
index 561b6f92c..0f7d472f7 100644
--- a/core/switch_core/bridges/agent/api/operations.py
+++ b/core/switch_core/bridges/agent/api/operations.py
@@ -33,7 +33,10 @@ from switch_core.bridges.agent.operations.callctx import (
CallerSession,
call_context,
)
-from switch_core.bridges.agent.protocol.agent_connections import UnknownConnectionError
+from switch_core.bridges.agent.protocol.agent_connections import (
+ MAX_ID_LENGTH,
+ UnknownConnectionError,
+)
from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.budgets import BudgetExceeded
from switch_core.db.models import Agent
@@ -175,19 +178,37 @@ async def resolve_caller(
agent, or one that has died, is refused.
"""
bound = connection_id
+ if session_id is not None and connection_id is None:
+ raise HTTPException(
+ status_code=400,
+ detail=(
+ "a session selector names the connection it calls over; "
+ "send X-Switch-Connection-Id with X-Switch-Session-Id"
+ ),
+ )
+ if session_id is not None and len(session_id) > MAX_ID_LENGTH:
+ # The session is recorded with the room it connects to, and its id
+ # travels in the signals that tell other processes it lost the room.
+ raise HTTPException(
+ status_code=400,
+ detail=f"X-Switch-Session-Id must be at most {MAX_ID_LENGTH} characters",
+ )
+ if bound is None:
+ return None, None
+
+ # The connection may be held by another process, and its row written a
+ # moment before this request; and the rooms it claimed, or the calling
+ # session's room, may have moved through another process whose answer the
+ # caller already has. So the rows are read again before the caller is
+ # resolved, rather than this process's copy trusted: a copy only a moment
+ # old would refuse the caller, or post into the room it left.
+ await protocol.connections.refresh(agent_id)
+
caller: CallerSession | None = None
if session_id is not None:
# Host and epoch are the host's own business now: Switch keeps no
# record of a session to check them against, only which room it
# connected to.
- if connection_id is None:
- raise HTTPException(
- status_code=400,
- detail=(
- "a session selector names the connection it calls over; "
- "send X-Switch-Connection-Id with X-Switch-Session-Id"
- ),
- )
caller = CallerSession(
id=session_id,
host_id=host_id or "",
@@ -195,9 +216,6 @@ async def resolve_caller(
room_id=protocol.connections.session_room(agent_id, session_id),
)
- if bound is None:
- return None, caller
-
# Derived, never taken on trust: a connection belonging to another agent,
# or to one that has already died, is refused rather than silently treated
# as no connection at all.
diff --git a/core/switch_core/bridges/agent/commands.py b/core/switch_core/bridges/agent/commands.py
index f05704a4a..e63a82a9d 100644
--- a/core/switch_core/bridges/agent/commands.py
+++ b/core/switch_core/bridges/agent/commands.py
@@ -27,10 +27,10 @@ from switch_core.gateway.known_agents import known_agent_for
from switch_core.transport import RoomRef
if TYPE_CHECKING:
- from switch_core.clients.agent_consumer import AgentConsumer
from switch_core.clients.command_consumer import CommandConsumer
from switch_core.clients.room_meta import RoomMeta
from switch_core.db.models import Agent
+ from switch_core.delivery.recipient import AgentRecipient
logger = logging.getLogger(__name__)
@@ -47,13 +47,13 @@ AGENT_GREETINGS = [
]
CommandHandler = Callable[
- ["AgentConsumer", RoomRef, CommandEvent, bool], Awaitable[None]
+ ["AgentRecipient", RoomRef, CommandEvent, bool], Awaitable[None]
]
# (client, args, room_id) -> whether THIS agent is addressed by the command.
# Targeting is a per-command policy (see Command.addressed) rather than a fixed
# rule in on_command, so a command like `run-cmd` can interpret its args its
# own way.
-CommandTargeting = Callable[["AgentConsumer", str, str], Awaitable[bool]]
+CommandTargeting = Callable[["AgentRecipient", str, str], Awaitable[bool]]
# (host, room, event, meta) -> None. An admin-side usage check run by the
# always-present admin client BEFORE agents act on a command. It posts a
# system message when the command is misused (bad/missing target) and does
@@ -61,10 +61,15 @@ CommandTargeting = Callable[["AgentConsumer", str, str], Awaitable[bool]]
CommandAdminCheck = Callable[
["CommandConsumer", RoomRef, CommandEvent, "RoomMeta"], Awaitable[None]
]
+# (client, room, event) -> what a command that queues itself for the agent
+# does for this agent, decided as the command's row is written.
+CommandDecider = Callable[
+ ["AgentRecipient", RoomRef, CommandEvent], Awaitable["ControlDecision"]
+]
async def _addressed_by_name_or_role(
- client: AgentConsumer, args: str, room_id: str
+ client: AgentRecipient, args: str, room_id: str
) -> bool:
"""Default command targeting.
@@ -83,7 +88,7 @@ async def _addressed_by_name_or_role(
async def _addressed_by_first_mention(
- client: AgentConsumer, args: str, room_id: str
+ client: AgentRecipient, args: str, room_id: str
) -> bool:
"""Targeting for `run-cmd`: only the FIRST `@token` addresses an agent (by
name or by a role it holds). Any further `@token` is data for the handler
@@ -99,7 +104,7 @@ async def _addressed_by_first_mention(
async def _addressed_by_required_first_mention(
- client: AgentConsumer, args: str, room_id: str
+ client: AgentRecipient, args: str, room_id: str
) -> bool:
"""Like `_addressed_by_first_mention`, but a missing target addresses NO
ONE. Used by `!reset`, where a bare `!reset` must not silently reset every
@@ -112,14 +117,14 @@ async def _addressed_by_required_first_mention(
return await _first_token_is_me(client, tokens[0], room_id)
-async def _addressed_everyone(client: AgentConsumer, args: str, room_id: str) -> bool:
+async def _addressed_everyone(client: AgentRecipient, args: str, room_id: str) -> bool:
"""Targeting for room-wide control commands (`!reset-all-agents`): every
agent is addressed regardless of args, so the command always fans out to
the whole room."""
return True
-async def _first_token_is_me(client: AgentConsumer, first: str, room_id: str) -> bool:
+async def _first_token_is_me(client: AgentRecipient, first: str, room_id: str) -> bool:
"""Whether `@first` names this agent — by name, room alias, or a held role."""
if client._args_tag_my_name(f"@{first}"):
return True
@@ -232,6 +237,11 @@ class Command:
# it. If False, the command is handled by the agents themselves (e.g.
# `!run-cmd`, `!reset`, `!agents-greet`), which answer in their own voice.
admin_owned: bool = False
+ # Set for a command that is queued for the agent (the session controls).
+ # The router decides it in the transaction that writes the command and
+ # stores what is queued with it, so the agent gets the command before
+ # anything written after it; `handler` runs it outside the router.
+ decide: CommandDecider | None = None
# ── Handlers ──────────────────────────────────────────────────────────────────
@@ -351,7 +361,7 @@ def _control_frame(
}
-async def _room_surface(client: AgentConsumer, room_id: str) -> str:
+async def _room_surface(client: AgentRecipient, room_id: str) -> str:
"""The platform a room is bridged to, as a contract surface."""
async with client.session_factory() as db:
room = await db.get(Room, room_id)
@@ -364,7 +374,7 @@ async def _room_surface(client: AgentConsumer, room_id: str) -> str:
async def _reply(
- client: AgentConsumer | CommandConsumer,
+ client: AgentRecipient | CommandConsumer,
room: RoomRef,
event: CommandEvent,
body: str,
@@ -388,7 +398,7 @@ async def _reply(
async def _cmd_help(
- client: AgentConsumer, room: RoomRef, event: CommandEvent, _is_direct: bool
+ client: AgentRecipient, room: RoomRef, event: CommandEvent, _is_direct: bool
) -> None:
lines = ["**Available commands:**"]
for cmd in COMMANDS:
@@ -398,17 +408,41 @@ async def _cmd_help(
await _reply(client, room, event, "\n".join(lines))
-async def _dispatch_control_command(
- client: AgentConsumer,
+@dataclass(frozen=True)
+class ControlTexts:
+ """What a session-control command says in the room for each outcome."""
+
+ ack: str
+ unsupported: str
+ # No live session in the room at all.
+ no_live_session: str
+ # A live session that cannot be controlled from here, e.g. one not
+ # started from Switch Console.
+ no_session: str
+
+
+@dataclass(frozen=True)
+class ControlDecision:
+ """What a session-control command does for one agent.
+
+ `queued` is the command to queue for the agent's controller, None when
+ nothing is queued. The router stores it as an own event in the
+ transaction that writes the command, so a message written after the
+ command reaches the agent after it. `answer` is what is said in the room,
+ or relayed to a session, once the command has committed.
+ """
+
+ queued: CommandPayload | None
+ answer: Callable[[], Awaitable[None]] | None
+
+
+async def _decide_control_command(
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
command: str,
- *,
- ack: str,
- unsupported_msg: str,
- no_live_session_msg: str,
- no_session_msg: str,
-) -> None:
+ texts: ControlTexts,
+) -> ControlDecision:
"""Shared logic for the session-control commands (reset/compact/interrupt).
Resolves the agent's declared capability level for `command` from its
@@ -417,15 +451,21 @@ async def _dispatch_control_command(
control capabilities). When actionable, acknowledges in the room and queues
a `command` event for the agent's controller (e.g. Switch Console) to execute.
- Two distinct "can't act" cases are reported separately: no live session in
- the room at all (`no_live_session_msg`) vs. a live session that can't be
- controlled from here — e.g. not started from Switch Console (`no_session_msg`).
+ Everything here is a read; what it decides to say or relay is left to
+ `ControlDecision.answer`.
"""
+
+ def say(body: str) -> ControlDecision:
+ async def answer() -> None:
+ await _reply(client, room, event, body)
+
+ return ControlDecision(queued=None, answer=answer)
+
async with client.session_factory() as session:
agent = await client._fresh_agent(session)
meta = await client._resolve_room_meta(room.room_id)
if meta is None:
- return
+ return ControlDecision(queued=None, answer=None)
# A session that connected to this room takes the command itself, over
# its agent's controller. Nothing is queued: with no controller to relay
# it to, the room is told so.
@@ -436,13 +476,9 @@ async def _dispatch_control_command(
placed = client._connections.session_in_room(agent.id, meta.room_id)
if placed is not None or controller_backed:
if not event.message_id:
- await _reply(
- client,
- room,
- event,
- f"Could not send {command}: the room command has no stable message ID.",
+ return say(
+ f"Could not send {command}: the room command has no stable message ID."
)
- return
frame = room_control_frame(
agent_id=agent.id,
session_id=placed,
@@ -454,27 +490,24 @@ async def _dispatch_control_command(
surface=await _room_surface(client, meta.room_id),
requester_name=event.user_name,
)
- if client._connections.relay_session_command(agent.id, frame):
- await _reply(client, room, event, ack)
- return
- if placed is not None or controller_backed:
- await _reply(
- client,
- room,
- event,
- f"Could not send {command}: the agent's controller is not connected to Switch.",
- )
- return
+
+ async def relay() -> None:
+ if client._connections.relay_session_command(agent.id, frame):
+ await _reply(client, room, event, texts.ack)
+ return
+ await _reply(
+ client,
+ room,
+ event,
+ f"Could not send {command}: the agent's controller is not connected to Switch.",
+ )
+
+ return ControlDecision(queued=None, answer=relay)
profile = agent.integration_profile or {}
level = (profile.get("command_capabilities") or {}).get(command, "unsupported")
if level == "unsupported":
- await _reply(client, room, event, unsupported_msg)
- return
-
- meta = await client._resolve_room_meta(room.room_id)
- if meta is None:
- return
+ return say(texts.unsupported)
deeplink: str | None = None
role: str | None = None
@@ -504,11 +537,9 @@ async def _dispatch_control_command(
live = statuses.get(agent.id) == AgentStatus.LIVE
caps = (runtime.control_capabilities or {}) if runtime else {}
if not live:
- await _reply(client, room, event, no_live_session_msg)
- return
+ return say(texts.no_live_session)
if not caps.get(command):
- await _reply(client, room, event, no_session_msg)
- return
+ return say(texts.no_session)
deeplink = runtime.deeplink_url if runtime else None
args = event.args
@@ -517,8 +548,41 @@ async def _dispatch_control_command(
# Mirror the runtime "working" surface: link back to the session in
# Switch Console when we know its deeplink.
- body = f"{ack} ([Open in Switch Console]({deeplink}))" if deeplink else ack
- await _reply(client, room, event, body)
+ body = (
+ f"{texts.ack} ([Open in Switch Console]({deeplink}))" if deeplink else texts.ack
+ )
+ decision = say(body)
+ return ControlDecision(
+ queued=CommandPayload(
+ command=command,
+ args=args,
+ user_id=event.user_id,
+ user_name=event.user_name,
+ thread_id=event.thread_id,
+ ),
+ answer=decision.answer,
+ )
+
+
+async def _dispatch_control_command(
+ client: AgentRecipient,
+ room: RoomRef,
+ event: CommandEvent,
+ command: str,
+ texts: ControlTexts,
+) -> None:
+ """Decide a session-control command and carry it out at once: answer in
+ the room, then queue the command for the agent in a transaction of its
+ own. For a command run outside the router; the router stores what is
+ queued with the command's row instead (`AgentRecipient.decide_command`)."""
+ decision = await _decide_control_command(client, room, event, command, texts)
+ if decision.answer is not None:
+ await decision.answer()
+ if decision.queued is None:
+ return
+ meta = await client._resolve_room_meta(room.room_id)
+ if meta is None:
+ return
await client._event_buffer.enqueue(
client.tenant_id,
client.agent.id,
@@ -528,82 +592,87 @@ async def _dispatch_control_command(
room_id=meta.room_id,
bridge_id=meta.bridge_id,
channel_type=meta.channel_type,
- payload=CommandPayload(
- command=command,
- args=args,
- user_id=event.user_id,
- user_name=event.user_name,
- thread_id=event.thread_id,
- ),
+ payload=decision.queued,
),
room_seq=event.room_seq,
message_key=event.message_id,
)
+_RESET = ControlTexts(
+ ack="Resetting my session — clearing context, then reconnecting to this room.",
+ unsupported="My session can't be reset — this isn't supported for me.",
+ no_live_session=(
+ "I have no active session in this room, so there's nothing to reset."
+ ),
+ no_session=(
+ "My session can't be reset from here — it wasn't started from "
+ "Switch Console, so my operator needs to reset it manually (e.g. by "
+ "restarting my session)."
+ ),
+)
+_COMPACT = ControlTexts(
+ ack="Compacting my session's context.",
+ unsupported="I can't compact my session — this isn't supported for me.",
+ no_live_session=(
+ "I have no active session in this room, so there's nothing to compact."
+ ),
+ no_session=(
+ "I can't compact from here — my session wasn't started from "
+ "Switch Console, so my operator needs to compact it manually."
+ ),
+)
+_INTERRUPT = ControlTexts(
+ ack="Interrupted my current turn.",
+ unsupported="I can't be interrupted — this isn't supported for me.",
+ no_live_session=(
+ "I have no active session in this room, so there's nothing to interrupt."
+ ),
+ no_session=(
+ "I can't be interrupted from here — my session wasn't started from "
+ "Switch Console, so my operator needs to interrupt it manually."
+ ),
+)
+
+
async def _cmd_reset(
- client: AgentConsumer, room: RoomRef, event: CommandEvent, _is_direct: bool
+ client: AgentRecipient, room: RoomRef, event: CommandEvent, _is_direct: bool
) -> None:
- await _dispatch_control_command(
- client,
- room,
- event,
- "reset",
- ack="Resetting my session — clearing context, then reconnecting to this room.",
- unsupported_msg="My session can't be reset — this isn't supported for me.",
- no_live_session_msg=(
- "I have no active session in this room, so there's nothing to reset."
- ),
- no_session_msg=(
- "My session can't be reset from here — it wasn't started from "
- "Switch Console, so my operator needs to reset it manually (e.g. by "
- "restarting my session)."
- ),
- )
+ await _dispatch_control_command(client, room, event, "reset", _RESET)
+
+
+async def _decide_reset(
+ client: AgentRecipient, room: RoomRef, event: CommandEvent
+) -> ControlDecision:
+ return await _decide_control_command(client, room, event, "reset", _RESET)
async def _cmd_compact(
- client: AgentConsumer, room: RoomRef, event: CommandEvent, _is_direct: bool
+ client: AgentRecipient, room: RoomRef, event: CommandEvent, _is_direct: bool
) -> None:
- await _dispatch_control_command(
- client,
- room,
- event,
- "compact",
- ack="Compacting my session's context.",
- unsupported_msg="I can't compact my session — this isn't supported for me.",
- no_live_session_msg=(
- "I have no active session in this room, so there's nothing to compact."
- ),
- no_session_msg=(
- "I can't compact from here — my session wasn't started from "
- "Switch Console, so my operator needs to compact it manually."
- ),
- )
+ await _dispatch_control_command(client, room, event, "compact", _COMPACT)
+
+
+async def _decide_compact(
+ client: AgentRecipient, room: RoomRef, event: CommandEvent
+) -> ControlDecision:
+ return await _decide_control_command(client, room, event, "compact", _COMPACT)
async def _cmd_interrupt(
- client: AgentConsumer, room: RoomRef, event: CommandEvent, _is_direct: bool
+ client: AgentRecipient, room: RoomRef, event: CommandEvent, _is_direct: bool
) -> None:
- await _dispatch_control_command(
- client,
- room,
- event,
- "interrupt",
- ack="Interrupted my current turn.",
- unsupported_msg="I can't be interrupted — this isn't supported for me.",
- no_live_session_msg=(
- "I have no active session in this room, so there's nothing to interrupt."
- ),
- no_session_msg=(
- "I can't be interrupted from here — my session wasn't started from "
- "Switch Console, so my operator needs to interrupt it manually."
- ),
- )
+ await _dispatch_control_command(client, room, event, "interrupt", _INTERRUPT)
+
+
+async def _decide_interrupt(
+ client: AgentRecipient, room: RoomRef, event: CommandEvent
+) -> ControlDecision:
+ return await _decide_control_command(client, room, event, "interrupt", _INTERRUPT)
async def _cmd_list_room_agents(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -631,7 +700,7 @@ async def _cmd_list_room_agents(
async def _cmd_list_aliases(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -663,7 +732,7 @@ async def _cmd_list_aliases(
async def _cmd_set_alias(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -733,7 +802,7 @@ async def _cmd_set_alias(
async def _cmd_remove_alias(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -785,7 +854,7 @@ async def _cmd_remove_alias(
async def _cmd_invite(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -904,7 +973,7 @@ def _format_status_lines(
async def _cmd_status(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -946,7 +1015,7 @@ async def _cmd_status(
async def _cmd_roles(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -984,7 +1053,7 @@ async def _cmd_roles(
async def _cmd_list_documents(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -1030,7 +1099,7 @@ async def _cmd_list_documents(
async def _cmd_list_references(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -1066,7 +1135,7 @@ async def _cmd_list_references(
async def _cmd_room_url(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -1093,7 +1162,7 @@ async def _cmd_room_url(
async def _cmd_list_all_agents(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -1125,7 +1194,7 @@ def _role_arg(args: str) -> str | None:
async def _cmd_run_cmd(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
_is_direct: bool,
@@ -1186,7 +1255,7 @@ async def _cmd_run_cmd(
async def _cmd_agents_greet(
- client: AgentConsumer,
+ client: AgentRecipient,
room: RoomRef,
event: CommandEvent,
is_direct: bool,
@@ -1219,6 +1288,7 @@ COMMANDS: list[Command] = [
"Usage: `!reset @agent-name` or `!reset @role` (the role's holder). "
"A target is required — use `!reset-all-agents` to reset everyone.",
_cmd_reset,
+ decide=_decide_reset,
args_spec=(
CommandArg("target", "Agent name, alias, or role to reset", required=True),
),
@@ -1234,6 +1304,7 @@ COMMANDS: list[Command] = [
"Reset EVERY agent's session in this room (clears context, then "
"reconnects each). Usage: `!reset-all-agents`.",
_cmd_reset,
+ decide=_decide_reset,
addressed=_addressed_everyone,
),
Command(
@@ -1242,6 +1313,7 @@ COMMANDS: list[Command] = [
"Usage: `!compact @agent-name` or `!compact @role` (the role's holder). "
"A target is required — use `!compact-all-agents` for everyone.",
_cmd_compact,
+ decide=_decide_compact,
args_spec=(
CommandArg(
"target", "Agent name, alias, or role to compact", required=True
@@ -1257,6 +1329,7 @@ COMMANDS: list[Command] = [
"Compact EVERY agent's session context in this room. "
"Usage: `!compact-all-agents`.",
_cmd_compact,
+ decide=_decide_compact,
addressed=_addressed_everyone,
),
Command(
@@ -1265,6 +1338,7 @@ COMMANDS: list[Command] = [
"Usage: `!interrupt @agent-name` or `!interrupt @role` (the role's holder). "
"A target is required — use `!interrupt-all-agents` for everyone.",
_cmd_interrupt,
+ decide=_decide_interrupt,
args_spec=(
CommandArg(
"target", "Agent name, alias, or role to interrupt", required=True
@@ -1278,6 +1352,7 @@ COMMANDS: list[Command] = [
"Interrupt EVERY agent's current turn in this room. "
"Usage: `!interrupt-all-agents`.",
_cmd_interrupt,
+ decide=_decide_interrupt,
addressed=_addressed_everyone,
),
Command(
@@ -1394,7 +1469,7 @@ COMMANDS_BY_NAME: dict[str, Command] = {cmd.name: cmd for cmd in COMMANDS}
async def dispatch_command(
- client: AgentConsumer, room: RoomRef, event: CommandEvent, is_direct: bool
+ client: AgentRecipient, room: RoomRef, event: CommandEvent, is_direct: bool
) -> bool:
"""Runs a command.
@@ -1444,4 +1519,4 @@ async def dispatch_admin_command(
is_direct = await host._is_direct_room(room.room_id)
# Admin-owned handlers use only the store/reply surface both clients share
# (never `.agent`), so the admin host satisfies the handler's contract.
- await cmd.handler(cast("AgentConsumer", host), room, event, is_direct)
+ await cmd.handler(cast("AgentRecipient", host), room, event, is_direct)
diff --git a/core/switch_core/bridges/agent/mcp/server.py b/core/switch_core/bridges/agent/mcp/server.py
index e914cb2c1..e145bf6d2 100644
--- a/core/switch_core/bridges/agent/mcp/server.py
+++ b/core/switch_core/bridges/agent/mcp/server.py
@@ -9,18 +9,27 @@ Switch hosting an MCP server is the legacy path: the supported direction is a
local runtime that serves the tool surface next to the agent and translates
each call onto its own connection. Because operations live in the registry
rather than here, retiring this door is removing a file, not a refactor.
+
+It holds no session in memory, so any switch-core process can answer any
+request of a client's. The MCP transport runs stateless, and `SessionIds`
+gives the client a session id of Switch's own on its first request, which the
+client sends back on every later one (the transport's `Mcp-Session-Id`). That
+id is all an operation needs to find the room the client connected to, and
+the binding it names is in Postgres (`operations/context.py`), not here.
"""
from __future__ import annotations
import logging
+import uuid
from typing import TYPE_CHECKING, Any
from fastmcp import FastMCP
from fastmcp.exceptions import ToolError
from fastmcp.server.dependencies import get_http_request
-from fastmcp.server.middleware import Middleware, MiddlewareContext, PingMiddleware
-from starlette.types import ASGIApp
+from fastmcp.server.middleware import Middleware, MiddlewareContext
+from starlette.datastructures import MutableHeaders
+from starlette.types import ASGIApp, Message, Receive, Scope, Send
from switch_core.bridges.agent.auth import BearerAuthMiddleware, OIDCTokenValidator
from switch_core.bridges.agent.operations import (
@@ -56,6 +65,10 @@ class CallContextMiddleware(Middleware):
each front door is responsible for binding it. Here that means the agent
from the authenticated request scope and the MCP transport session as the
key that owns the room binding.
+
+ The session id comes from the client (`SessionIds`), so the key is the
+ agent's id and the session's together: an agent naming another's session
+ id reaches only a binding of its own.
"""
async def on_call_tool(self, context: MiddlewareContext, call_next): # type: ignore[no-untyped-def]
@@ -69,7 +82,9 @@ class CallContextMiddleware(Middleware):
CallContext(
agent_id=agent_id,
session_key=(
- fastmcp_context.session_id if fastmcp_context is not None else None
+ session_key(agent_id, fastmcp_context.session_id)
+ if fastmcp_context is not None
+ else None
),
# An MCP transport session is not an SDK session: there is no
# host lease behind it, so it has no fenced room binding of its
@@ -80,6 +95,47 @@ class CallContextMiddleware(Middleware):
return await call_next(context)
+def session_key(agent_id: str, session_id: str) -> str:
+ """The key an MCP session's room binding is kept under."""
+ return f"{agent_id}/{session_id}"
+
+
+SESSION_HEADER = b"mcp-session-id"
+
+
+class SessionIds:
+ """Give each MCP client a session id that any switch-core process honours.
+
+ The transport runs stateless, so it issues no `Mcp-Session-Id` and checks
+ none. A request that arrives without one is given a fresh id here, both on
+ the request (where the call context reads it) and on the response, and a
+ client following the transport sends it on every later request, whichever
+ process those reach. A request that brings one keeps it.
+ """
+
+ def __init__(self, app: ASGIApp) -> None:
+ self.app = app
+
+ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+ if scope["type"] != "http" or any(
+ name == SESSION_HEADER for name, _ in scope["headers"]
+ ):
+ await self.app(scope, receive, send)
+ return
+ minted = uuid.uuid4().hex
+ scope = {
+ **scope,
+ "headers": [*scope["headers"], (SESSION_HEADER, minted.encode())],
+ }
+
+ async def send_with_id(message: Message) -> None:
+ if message["type"] == "http.response.start":
+ MutableHeaders(scope=message).append("Mcp-Session-Id", minted)
+ await send(message)
+
+ await self.app(scope, receive, send_with_id)
+
+
class ExistingOperationsOnly(Middleware):
"""Serve only the operations that exist now.
@@ -115,6 +171,12 @@ for _op in declared_operations().values():
logger.debug("Registered %d operations as MCP tools", len(declared_operations()))
+def mcp_http_app() -> Any:
+ """The MCP transport, stateless: no session lives in this process, so no
+ keepalive ping either, since there is no long-lived stream to keep."""
+ return mcp.http_app(path="/", stateless_http=True)
+
+
def create_mcp_app(
*,
agent_store: AgentStore,
@@ -125,8 +187,6 @@ def create_mcp_app(
"""Returns (asgi_app, lifespan). The lifespan must be wired into the parent app."""
init_mcp_protocol(protocol)
- mcp.add_middleware(PingMiddleware(interval_ms=30000))
-
oidc_validator = None
if config.oauth_issuer_url:
assert config.oauth_audience is not None # enforced by SwitchConfig
@@ -137,9 +197,9 @@ def create_mcp_app(
)
logger.info("OIDC auth enabled for MCP (issuer: %s)", config.oauth_issuer_url)
- starlette_app = mcp.http_app(path="/")
+ starlette_app = mcp_http_app()
app = BearerAuthMiddleware(
- starlette_app,
+ SessionIds(starlette_app),
agent_store=agent_store,
api_key_store=api_key_store,
api_key_cache=protocol.api_key_cache,
diff --git a/core/switch_core/bridges/agent/operations/definitions.py b/core/switch_core/bridges/agent/operations/definitions.py
index 6e1dc395a..df2e28de5 100644
--- a/core/switch_core/bridges/agent/operations/definitions.py
+++ b/core/switch_core/bridges/agent/operations/definitions.py
@@ -342,18 +342,23 @@ async def connect_to_room(
# then fails would send them somewhere the session is not — and the
# rooms to vacate are the caller's own, read under the bind's lock
# rather than from what it believed on arrival.
- if caller is not None:
- previous, displaced_session_id = protocol.connections.place_session(
- agent_id, caller.id, room.id, key
+ #
+ # Recorded, so the move is made on what every process holds and the
+ # process holding the caller's connection is told: its stream may
+ # be parked waiting for exactly this room.
+ async with protocol.connections.recorded(agent_id):
+ if caller is not None:
+ previous, displaced_session_id = protocol.connections.place_session(
+ agent_id, caller.id, room.id, key
+ )
+ else:
+ previous = rooms_on_caller_connection(protocol, agent_id, key)
+
+ evicted_connection_id = claim_room_on_caller_connection(
+ protocol, agent_id, key, room.id
)
- else:
- previous = rooms_on_caller_connection(protocol, agent_id, key)
-
- evicted_connection_id = claim_room_on_caller_connection(
- protocol, agent_id, key, room.id
- )
- for departed in previous - {room.id}:
- release_room_on_caller_connection(protocol, agent_id, key, departed)
+ for departed in previous - {room.id}:
+ release_room_on_caller_connection(protocol, agent_id, key, departed)
# Connecting is how an agent's occupancy of a room changes hands, and
# the occupant is the one whose reading clears that room's unread
diff --git a/core/switch_core/bridges/agent/protocol/agent_connections.py b/core/switch_core/bridges/agent/protocol/agent_connections.py
index 2b7b0457c..c42741c9b 100644
--- a/core/switch_core/bridges/agent/protocol/agent_connections.py
+++ b/core/switch_core/bridges/agent/protocol/agent_connections.py
@@ -15,9 +15,16 @@ Two properties matter most:
sleeping laptop leaves one behind for minutes. A connection is alive while
its client keeps ticking, and dead when it stops, whatever the socket says.
-Connections live in memory only. They cannot outlive the process (their
-sockets and buffers cannot), so persisting them would recreate the stale-state
-bug this design removes.
+A connection lives in the memory of the process holding its socket: the
+socket, its heartbeat and the frames it is owed cannot be anywhere else. What
+the other switch-core processes need to know about it is recorded as rows
+(`connection_rows.py`), on transitions only and never per heartbeat, and each
+process keeps a copy of every other process's live connections here, so the
+presence readers below answer for every process without a query. A copy is
+an `AgentConnection` whose `held_by` names the process holding it. Rows count
+only while the holding process's lease is held, so a connection never
+outlives its process, which is what keeping connections in memory alone used
+to guarantee.
"""
from __future__ import annotations
@@ -26,10 +33,15 @@ import asyncio
import logging
import secrets
import time
-from collections.abc import AsyncIterator, Callable, Iterable
-from contextlib import asynccontextmanager
+from collections.abc import AsyncIterator, Callable, Iterable, Iterator
+from contextlib import (
+ AbstractAsyncContextManager,
+ asynccontextmanager,
+ contextmanager,
+ nullcontext,
+)
from dataclasses import dataclass, field
-from typing import Any, Literal
+from typing import Any, Literal, Protocol, cast
from switch_core.artifacts import ARTIFACT_VERSIONS, contract_range
from switch_core.bridges.agent.protocol.controller_presence import ControllerPresence
@@ -46,6 +58,7 @@ from switch_core.bridges.agent.protocol.liveness import TAKEN_OVER as TAKEN_OVER
from switch_core.bridges.agent.protocol.liveness import CloseCode as CloseCode
from switch_core.bridges.agent.protocol.liveness import Closure as Closure
from switch_core.bridges.agent.protocol.positions import OWN_EVENTS, Position
+from switch_core.db.stores.agent_connection_store import AgentSlotRows, ConnectionRow
from switch_core.logging_context import log_context
from switch_core.observability.catalogue import (
AGENT_CONNECTIONS_EXPIRED,
@@ -104,6 +117,12 @@ POSITIONS_PROTOCOL_REVISION = 9
# visible error instead of quiet resource creep.
MAX_CONNECTIONS_PER_AGENT = 32
+# The longest connection id or session id a client may choose; clients use
+# UUIDs, which are 36. Both travel in the signals switch-core processes send
+# each other (`connection_rows.py`), and Postgres refuses one of 8000 bytes or
+# more, so an id with no bound could make a signal that cannot be sent.
+MAX_ID_LENGTH = 64
+
class ConnectionError_(Exception):
"""Base for connection faults that a client must be told about.
@@ -444,8 +463,19 @@ class AgentConnection:
# gap can report it.
resume_lost: bool = False
resume_requested: int = 0
+ # The switch-core process holding this connection, when it is another
+ # one: this is then a copy of that process's row, kept so presence, room
+ # claims and fences read the same here as there. A copy has no socket and
+ # no heartbeat of its own; it is live for as long as the row is, and is
+ # replaced whenever the row is read again.
+ held_by: str | None = None
+ # The tenant the connection's row is recorded in. Set once the row is
+ # written or read; None on a registry that records nothing.
+ tenant_id: str | None = None
def is_alive(self, now: float) -> bool:
+ if self.held_by is not None:
+ return self.closure is None
return self.closure is None and (now - self.last_beat) < HEARTBEAT_TTL_SECONDS
@property
@@ -488,8 +518,57 @@ def connection_transport(conn: AgentConnection) -> str:
return conn.stream_transport if conn.stream_attached else "detached"
+#: How a deleted agent's connections end, in whichever process holds them. A
+#: client that reopens is refused: its key names no agent any more.
+AGENT_DELETED = Closure(code="closed", message="the agent was deleted", room_id=None)
+
+
+class ConnectionRows(Protocol):
+ """Where a registry records its connections, so that every switch-core
+ process sees them (`connection_rows.SharedConnections`).
+
+ A registry without one records nothing and sees only its own process,
+ which is all a single process needs.
+ """
+
+ def next_incarnation(self) -> int:
+ """An incarnation no process has handed out, from the shared
+ sequence."""
+ ...
+
+ def recorded(self, agent_id: str) -> AbstractAsyncContextManager[None]:
+ """Make the agent's slot moves in the block on what every process has
+ recorded, and record them."""
+ ...
+
+ async def refresh(self, agent_id: str) -> None:
+ """Read the agent's connections again from the rows."""
+ ...
+
+ async def agent_deleted(self, agent_id: str) -> None:
+ """The agent was deleted, and its rows with it: end what this process
+ holds of it, and have every other process read its rows again."""
+ ...
+
+ def closed(self, conn: AgentConnection) -> None:
+ """A connection held here closed. Must not block."""
+ ...
+
+ def forward_beat(self, conn: AgentConnection, generation: int | None) -> None:
+ """A beat for a connection another process holds. Must not block."""
+ ...
+
+ def relay_session_command(
+ self, tenant_id: str, agent_id: str, process_id: str, frame: dict[str, Any]
+ ) -> None:
+ """A session command for a stream another process holds. Must not
+ block."""
+ ...
+
+
class AgentConnectionRegistry:
- """The live set of agent connections. Authoritative, in memory."""
+ """The live set of agent connections: those this process holds,
+ authoritative, and a copy of every other process's (`held_by`)."""
def __init__(self) -> None:
# Called with every AgentConnection this registry closes, whoever closed it.
@@ -527,14 +606,30 @@ class AgentConnectionRegistry:
# be counted as a return after a lapse rather than a first connect.
# Insertion-ordered and capped: only the counter reads it.
self._recently_closed: dict[str, str] = {}
+ # Where connections are recorded for the other processes, once wired.
+ self._rows: ConnectionRows | None = None
+ # The rooms taken off other connections while a recorded move runs,
+ # so the processes holding them can be told (`collect_released`).
+ self._released: list[Released] | None = None
+ # The tenant each agent's connections are recorded in, as last read
+ # or written, so a pass over one tenant's rows can find the agents it
+ # no longer names.
+ self._agent_tenants: dict[str, str] = {}
+
+ def use_rows(self, rows: ConnectionRows) -> None:
+ """Record connections for every process, and see theirs."""
+ self._rows = rows
def _new_incarnation(self) -> int:
"""The next never-before-used incarnation number.
- Monotonic so it is readable in a log and orderable in a comparison, and
- registry-wide rather than per-connection so that closing an id and
- opening it again cannot reissue a number a departed client still holds.
+ Registry-wide rather than per-connection so that closing an id and
+ opening it again cannot reissue a number a departed client still
+ holds. With rows wired it comes from the shared sequence, so no two
+ processes hand out the same number and any of them can check a fence.
"""
+ if self._rows is not None:
+ return self._rows.next_incarnation()
incarnation = self._next_incarnation
self._next_incarnation += 1
return incarnation
@@ -606,6 +701,8 @@ class AgentConnectionRegistry:
presented=expected_generation,
current=existing.stream_generation,
)
+ moved_from = existing.held_by
+ existing.held_by = None
existing.scope = scope
existing.delivery_filter = delivery_filter
existing.spawn_capable = spawn_capable
@@ -626,12 +723,13 @@ class AgentConnectionRegistry:
metrics().increment(AGENT_CONNECTIONS_OPENED, {"kind": "reattach"})
logger.info(
"[CONN] reattached agent=%s connection=%s scope=%s generation=%s "
- "transport=%s",
+ "transport=%s%s",
agent_id,
connection_id,
scope,
existing.stream_generation,
transport,
+ f" from process={moved_from}" if moved_from is not None else "",
)
return existing
@@ -702,7 +800,7 @@ class AgentConnectionRegistry:
"""
self._on_close = listener
- def close(self, connection_id: str, closure: Closure) -> AgentConnection | None:
+ def _forget(self, connection_id: str) -> AgentConnection | None:
conn = self._by_id.pop(connection_id, None)
if conn is None:
return None
@@ -711,6 +809,16 @@ class AgentConnectionRegistry:
owned.discard(connection_id)
if not owned:
self._by_agent.pop(conn.agent_id, None)
+ return conn
+
+ def close(self, connection_id: str, closure: Closure) -> AgentConnection | None:
+ held = self._by_id.get(connection_id)
+ if held is not None and held.held_by is not None:
+ # Another process holds it, and only that process can close it.
+ return None
+ conn = self._forget(connection_id)
+ if conn is None:
+ return None
conn.closure = closure
conn.stream_attached = False
conn.wake.set()
@@ -744,6 +852,8 @@ class AgentConnectionRegistry:
conn.beats,
time.monotonic() - conn.last_beat,
)
+ if self._rows is not None:
+ self._rows.closed(conn)
# After the bookkeeping and the log, so an observer sees the closed
# state. Guarded because the registry's own contract — the connection
# is closed and the caller gets it back — must not depend on whoever
@@ -760,7 +870,10 @@ class AgentConnectionRegistry:
return conn
def _close_for_controller(self, agent_id: str) -> None:
- """An agent now run by a controller keeps no connection of its own."""
+ """An agent now run by a controller keeps no connection of its own.
+
+ Only the connections held here: every process is told of the binding,
+ and each closes its own."""
for cid in list(self._by_agent.get(agent_id, ())):
self.close(
cid,
@@ -784,7 +897,7 @@ class AgentConnectionRegistry:
stale = [
conn
for conn in self._by_id.values()
- if (now - conn.last_beat) >= HEARTBEAT_TTL_SECONDS
+ if conn.held_by is None and (now - conn.last_beat) >= HEARTBEAT_TTL_SECONDS
]
closed = []
for conn in stale:
@@ -843,6 +956,13 @@ class AgentConnectionRegistry:
)
if not conn.stream_attached:
raise NoStreamAttachedError(connection_id)
+ if conn.held_by is not None:
+ # An old event-stream client's beat that reached a process other
+ # than the one holding its stream. Fenced here on the row, and
+ # handed on to the holder, which is the one that counts it.
+ if self._rows is not None:
+ self._rows.forward_beat(conn, generation)
+ return conn
conn.last_beat = time.monotonic()
conn.beats += 1
return conn
@@ -1017,15 +1137,21 @@ class AgentConnectionRegistry:
self._notify_released(loser, room_id, session_id)
return Released(connection_id=loser.id, room_id=room_id, session_id=session_id)
- @staticmethod
def _notify_released(
- conn: AgentConnection, room_id: str, session_id: str | None
+ self, conn: AgentConnection, room_id: str, session_id: str | None
) -> None:
"""Queue a `room_released` frame for a client that can take one.
A room released twice before the stream writes it is reported once,
- naming the session when either release knew it.
+ naming the session when either release knew it. A connection another
+ process holds is told by that process, from what was collected here.
"""
+ if self._released is not None:
+ self._released.append(
+ Released(connection_id=conn.id, room_id=room_id, session_id=session_id)
+ )
+ if conn.held_by is not None:
+ return
if (conn.declaration.speaks or 0) < ROOM_RELEASED_PROTOCOL_REVISION:
return
if session_id is not None or room_id not in conn.released_rooms:
@@ -1081,22 +1207,61 @@ class AgentConnectionRegistry:
Not stored anywhere else: a command relayed to nobody is refused to
whoever sent it, rather than held for a watcher that may not return.
A controller-backed agent's command goes to its controller's stream.
+ A stream another process holds is handed the command through that
+ process (`ConnectionRows.relay_session_command`).
"""
if self.controllers.is_bound(agent_id):
- return self.controllers.relay_session_command(agent_id, frame)
- else:
- watchers = [
- conn
- for conn in self.for_agent(agent_id)
- if conn.scope == "all"
- and conn.stream_attached
- and (conn.declaration.speaks or 0) >= SESSION_COMMAND_PROTOCOL_REVISION
- ]
+ if self.controllers.relay_session_command(agent_id, frame):
+ return True
+ holder = self.controllers.process_holding(agent_id)
+ binding = self.controllers.binding(agent_id)
+ if holder is None or binding is None or self._rows is None:
+ return False
+ self._rows.relay_session_command(binding.tenant_id, agent_id, holder, frame)
+ return True
+ watchers = self._watchers(agent_id)
+ relayed_to: set[str] = set()
for conn in watchers:
- conn.session_commands.append(frame)
- conn.wake.set()
+ if conn.held_by is None:
+ self.deliver_session_command(conn, frame)
+ elif (
+ self._rows is not None
+ and conn.tenant_id is not None
+ and conn.held_by not in relayed_to
+ ):
+ relayed_to.add(conn.held_by)
+ self._rows.relay_session_command(
+ conn.tenant_id, agent_id, conn.held_by, frame
+ )
return bool(watchers)
+ def deliver_relayed(self, agent_id: str, frame: dict[str, Any]) -> bool:
+ """A session command another process relayed to the streams held here.
+ False when none of them is still here to take it."""
+ if self.controllers.is_bound(agent_id):
+ return self.controllers.relay_session_command(agent_id, frame)
+ delivered = False
+ for conn in self._watchers(agent_id):
+ if conn.held_by is None:
+ self.deliver_session_command(conn, frame)
+ delivered = True
+ return delivered
+
+ def _watchers(self, agent_id: str) -> list[AgentConnection]:
+ return [
+ conn
+ for conn in self.for_agent(agent_id)
+ if conn.scope == "all"
+ and conn.stream_attached
+ and (conn.declaration.speaks or 0) >= SESSION_COMMAND_PROTOCOL_REVISION
+ ]
+
+ @staticmethod
+ def deliver_session_command(conn: AgentConnection, frame: dict[str, Any]) -> None:
+ """Queue a relayed session command on a watcher held here."""
+ conn.session_commands.append(frame)
+ conn.wake.set()
+
# ------------------------------------------------------------------
# Room slots
# ------------------------------------------------------------------
@@ -1129,6 +1294,239 @@ class AgentConnectionRegistry:
else:
self._slot_locks[agent_id] = (lock, remaining - 1)
+ def holds_slots(self, agent_id: str) -> bool:
+ """Whether something holds this agent's slots (`slots`) right now."""
+ held = self._slot_locks.get(agent_id)
+ return held is not None and held[0].locked()
+
+ def recorded(self, agent_id: str) -> AbstractAsyncContextManager[None]:
+ """Make the slot moves inside the block on what every process holds.
+
+ Taken inside `slots`, around moves only: the block must not await. On
+ entry the agent's connections, claims and placements are read again
+ under a lock every process takes, so a fence, a claimant or a cap is
+ judged on what is recorded rather than on a copy that may be a moment
+ old. On exit what the block changed is written under the same lock,
+ and the processes holding the connections it touched are told. A
+ registry that records nothing makes this a no-op.
+ """
+ if self._rows is None:
+ return nullcontext()
+ return self._rows.recorded(agent_id)
+
+ async def refresh(self, agent_id: str) -> None:
+ """Read this agent's connections again from what every process has
+ recorded. For a caller that named a connection or a session this
+ process has no copy of yet: its row may have been written a moment
+ ago by another process."""
+ if self._rows is not None:
+ await self._rows.refresh(agent_id)
+
+ async def agent_deleted(self, agent_id: str) -> None:
+ """End a deleted agent's connections, here and in every other process.
+
+ Deleting the agent deleted its connections' rows. Each process makes
+ the agent's connections what the rows now say, which closes those it
+ holds (`apply_rows`): this one at once, and the others when told. A
+ registry that records nothing closes the connections it holds.
+ """
+ if self._rows is not None:
+ await self._rows.agent_deleted(agent_id)
+ return
+ for cid in list(self._by_agent.get(agent_id, ())):
+ self.close(cid, AGENT_DELETED)
+
+ # ------------------------------------------------------------------
+ # Keeping the copies of other processes' connections
+ # ------------------------------------------------------------------
+
+ def held_here(self, agent_id: str) -> list[AgentConnection]:
+ """The agent's connections this process holds, live or lapsed."""
+ return [
+ conn
+ for cid in self._by_agent.get(agent_id, ())
+ if (conn := self._by_id.get(cid)) is not None and conn.held_by is None
+ ]
+
+ def agents_recorded_in(self, tenant_id: str) -> set[str]:
+ """The agents this process keeps anything of from the tenant's rows,
+ so a pass over those rows judges each, the ones they no longer name
+ included: those `apply_rows` last set from them, and those with a
+ connection recorded in the tenant (`tenant_id`), whose row this
+ process may have written since."""
+ return {
+ agent_id
+ for agent_id, tenant in self._agent_tenants.items()
+ if tenant == tenant_id
+ } | {
+ conn.agent_id
+ for conn in self._by_id.values()
+ if conn.tenant_id == tenant_id
+ }
+
+ def held_by_processes(self) -> set[str]:
+ """Every other process this registry knows to hold an agent's or a
+ controller's connection."""
+ return {
+ conn.held_by for conn in self._by_id.values() if conn.held_by is not None
+ } | self.controllers.processes_elsewhere()
+
+ def slot_state(
+ self, agent_id: str
+ ) -> tuple[dict[str, str], dict[str, tuple[str, str]]]:
+ """The agent's claims (room to connection) and placements (session to
+ room and the connection that placed it), as they stand here."""
+ claims: dict[str, str] = {}
+ now = time.monotonic()
+ # A live claimant first, so that a room a lapsed connection still
+ # holds and a live one has since claimed is recorded as the live
+ # one's: at most one connection of an agent may hold a room.
+ conns = sorted(
+ (
+ conn
+ for cid in self._by_agent.get(agent_id, ())
+ if (conn := self._by_id.get(cid)) is not None
+ ),
+ key=lambda conn: (not conn.is_alive(now), conn.id),
+ )
+ for conn in conns:
+ for room_id in sorted(conn.rooms):
+ if room_id in claims:
+ conn.rooms.discard(room_id)
+ else:
+ claims[room_id] = conn.id
+ owners = self._placement_owners.get(agent_id, {})
+ placements = {
+ session_id: (room_id, owners[session_id])
+ for session_id, room_id in self._session_rooms.get(agent_id, {}).items()
+ if session_id in owners
+ }
+ return claims, placements
+
+ def incarnations_held_here(self, agent_id: str) -> dict[str, int]:
+ return {conn.id: conn.stream_generation for conn in self.held_here(agent_id)}
+
+ @contextmanager
+ def collect_released(self) -> Iterator[list[Released]]:
+ """Collect every room taken off a connection inside the block."""
+ released: list[Released] = []
+ self._released = released
+ try:
+ yield released
+ finally:
+ self._released = None
+
+ def apply_rows(
+ self, agent_id: str, tenant_id: str, rows: AgentSlotRows, process_id: str
+ ) -> list[AgentConnection]:
+ """Make this agent's connections what the rows say.
+
+ A connection held here that another process (or a later incarnation)
+ has taken over is closed `taken_over`, so its stream tells its client.
+ One whose row was written and is gone is closed too: deleting the
+ agent deletes its rows. Copies of other processes' connections are
+ replaced by what is live now. Claims and placements are set from the
+ rows, and a connection held here whose rooms changed is woken, so its
+ stream reports the change and a session waiting for its first room
+ starts reading. Returns the connections closed as taken over.
+
+ The rows are never older than what this process holds of the agent
+ (`connection_rows.py` reads them under its slots, or leaves an agent
+ moved since its read alone), so a connection opened after the read is
+ not here to be judged on them.
+ """
+ self._agent_tenants[agent_id] = tenant_id
+ closed: list[AgentConnection] = []
+ for conn in self.held_here(agent_id):
+ row = rows.connections.get(conn.id)
+ if row is None:
+ # Recorded once (`tenant_id` is set when its row is written),
+ # and the row is gone since.
+ if conn.tenant_id is not None:
+ self.close(conn.id, AGENT_DELETED)
+ continue
+ if row.closed:
+ continue
+ if (
+ row.process_id != process_id
+ or row.incarnation != conn.stream_generation
+ ):
+ gone = self.close(conn.id, TAKEN_OVER)
+ if gone is not None:
+ closed.append(gone)
+ for cid in list(self._by_agent.get(agent_id, ())):
+ copy = self._by_id.get(cid)
+ if copy is not None and copy.held_by is not None:
+ self._forget(cid)
+ for row in rows.live_connections():
+ if row.process_id == process_id or row.id in self._by_id:
+ continue
+ self._add_copy(row, tenant_id)
+
+ claims = rows.claims()
+ for cid in self._by_agent.get(agent_id, ()):
+ conn = self._by_id[cid]
+ rooms = {room for room, holder in claims.items() if holder == cid}
+ if conn.rooms != rooms:
+ conn.rooms = rooms
+ conn.wake.set()
+
+ placements = rows.placements()
+ if placements:
+ self._session_rooms[agent_id] = {
+ session_id: room_id for session_id, (room_id, _) in placements.items()
+ }
+ self._placement_owners[agent_id] = {
+ session_id: owner for session_id, (_, owner) in placements.items()
+ }
+ else:
+ self._session_rooms.pop(agent_id, None)
+ self._placement_owners.pop(agent_id, None)
+ if not self._by_agent.get(agent_id) and not placements:
+ self._agent_tenants.pop(agent_id, None)
+ return closed
+
+ def _add_copy(self, row: ConnectionRow, tenant_id: str) -> None:
+ now = time.monotonic()
+ copy = AgentConnection(
+ id=row.id,
+ agent_id=row.agent_id,
+ scope=cast(Scope, row.scope),
+ delivery_filter=cast(DeliveryFilter, row.delivery_filter),
+ spawn_capable=row.spawn_capable,
+ position={},
+ last_beat=now,
+ opened_at=now,
+ stream_attached=True,
+ stream_transport=cast(Transport, row.transport),
+ stream_generation=row.incarnation,
+ declaration=ClientDeclaration(speaks=row.speaks, accepts=row.accepts),
+ held_by=row.process_id,
+ tenant_id=tenant_id,
+ )
+ self._by_id[row.id] = copy
+ self._by_agent.setdefault(row.agent_id, set()).add(row.id)
+
+ def drop_copies_held_by(self, process_ids: set[str]) -> set[str]:
+ """Forget the connections these processes held, agents' and
+ controllers': their leases lapsed. Returns the agents that lost one."""
+ agents: set[str] = set()
+ for conn in list(self._by_id.values()):
+ if conn.held_by in process_ids:
+ self._forget(conn.id)
+ agents.add(conn.agent_id)
+ for controller_id in self.controllers.forget_processes(process_ids):
+ agents |= self.controllers.agents_of(controller_id)
+ return agents
+
+ def notify_released(self, released: Iterable[Released]) -> None:
+ """Tell the connections held here that another process took a room
+ off them."""
+ for lost in released:
+ conn = self._by_id.get(lost.connection_id)
+ if conn is not None and conn.held_by is None:
+ self._notify_released(conn, lost.room_id, lost.session_id)
+
def claim_room(
self, conn: AgentConnection, room_id: str, *, takeover: bool = False
) -> AgentConnection | None:
@@ -1179,6 +1577,19 @@ class AgentConnectionRegistry:
conn.rooms.discard(room_id)
conn.wake.set()
+ def claims_held_here(self) -> dict[str, set[str]]:
+ """The rooms each agent's connections held by this process have
+ claimed, lapsed ones included, for checking them against the agents'
+ memberships. A copy of another process's connection is left out: that
+ process checks the connections it holds."""
+ claims: dict[str, set[str]] = {}
+ for agent_id, ids in self._by_agent.items():
+ for cid in ids:
+ conn = self._by_id.get(cid)
+ if conn is not None and conn.held_by is None and conn.rooms:
+ claims.setdefault(agent_id, set()).update(conn.rooms)
+ return claims
+
def claimant_of(self, agent_id: str, room_id: str) -> AgentConnection | None:
"""The connection that has explicitly claimed this room, if any.
@@ -1280,11 +1691,15 @@ class AgentConnectionRegistry:
against one agent is twenty connections and one agent. Telemetry
reports sessions, and a session is a connection. A controller's
stream is one connection, however many agents it carries.
+
+ Counted across every process: those held here, the copies of the
+ others', and the controllers whose socket another process holds.
"""
now = time.monotonic()
return (
sum(1 for conn in self._by_id.values() if conn.is_alive(now))
+ self.controllers.live_connection_count()
+ + self.controllers.held_elsewhere_count()
)
def live_agent_ids(self) -> set[str]:
@@ -1302,14 +1717,18 @@ class AgentConnectionRegistry:
once under each, so the labels can sum to more than the agent count.
An agent run by an agents controller has no connection of its own and
counts under `CONTROLLER_LABEL` while its controller is live.
+
+ Only what this process holds: each process reports its own, so the
+ gauge sums across processes rather than counting every agent once per
+ process.
"""
now = time.monotonic()
agents: dict[tuple[str, str], set[str]] = {}
for conn in self._by_id.values():
- if conn.is_alive(now):
+ if conn.held_by is None and conn.is_alive(now):
key = (connection_transport(conn), client_label(conn.declaration))
agents.setdefault(key, set()).add(conn.agent_id)
- controlled = self.controllers.live_agent_ids()
+ controlled = self.controllers.live_agent_ids_held_here()
if controlled:
agents.setdefault(CONTROLLER_LABEL, set()).update(controlled)
return {key: len(ids) for key, ids in agents.items()}
diff --git a/core/switch_core/bridges/agent/protocol/agent_core.py b/core/switch_core/bridges/agent/protocol/agent_core.py
index 466b0f0a6..0ce3626e3 100644
--- a/core/switch_core/bridges/agent/protocol/agent_core.py
+++ b/core/switch_core/bridges/agent/protocol/agent_core.py
@@ -726,6 +726,7 @@ class AgentCore:
await session.commit()
self.client_lifecycle.start_client(client_record)
+ self.client_lifecycle.announce(client_record.tenant_id)
return agent.id
async def _reregister_agent(
@@ -1027,17 +1028,28 @@ class AgentCore:
if self._agent_removal_listener is not None:
await self._agent_removal_listener(tenant_id, resolved_id)
await self.client_lifecycle.stop(client_id)
- self.event_buffer.remove(resolved_id)
+ try:
+ self.event_buffer.remove(resolved_id)
- await self._remove_bridge_identities(tenant_id, resolved_name)
+ await self._remove_bridge_identities(tenant_id, resolved_name)
- # One transaction for the agent and its client, stopped above; see
- # `ClientLifecycleService.delete_record`.
- async with self.session_factory() as session:
- await self.agent_store.delete(session, resolved_id)
- await self.client_lifecycle.delete_record(session, client_id)
- await session.commit()
+ # One transaction for the agent and its client, stopped above; see
+ # `ClientLifecycleService.delete_record`.
+ async with self.session_factory() as session:
+ await self.agent_store.delete(session, resolved_id)
+ await self.client_lifecycle.delete_record(session, client_id)
+ await session.commit()
+ except BaseException:
+ # A deletion that did not commit leaves the agent in place, and
+ # its client must not stay stopped here.
+ self.client_lifecycle.deletion_failed(client_id)
+ raise
+ self.client_lifecycle.announce(tenant_id)
self.api_key_cache.invalidate_agent(resolved_id)
+ # Its connections end with it, here and in whichever process holds
+ # one: their rows went with its row. After its key is dropped, so a
+ # client told cannot open another.
+ await self.connections.agent_deleted(resolved_id)
if self._room_agents_listener is not None:
for room_id in room_ids:
await self._room_agents_listener(tenant_id, room_id)
@@ -2369,10 +2381,10 @@ class AgentCore:
without the same check this call would hand over events from a room
the agent has since been removed from.
- `AgentConsumer.on_removed` already empties the buffer of a removed
- room's events, so this is the second of two answers. It is the
- authoritative one: that signal is in-process, and this reads the table
- the removal wrote.
+ The removal deletes the agent's events in the room it left
+ (`MessageRouter.remove`), so this is the second of two answers, and
+ the one that holds whatever wrote the events, an older release's
+ consumer included: it reads the membership the removal wrote.
"""
async with self.session_factory() as session:
await self.agent_session_store.touch_heartbeat(session, agent_id, None)
diff --git a/core/switch_core/bridges/agent/protocol/connection_rows.py b/core/switch_core/bridges/agent/protocol/connection_rows.py
new file mode 100644
index 000000000..a7d2af971
--- /dev/null
+++ b/core/switch_core/bridges/agent/protocol/connection_rows.py
@@ -0,0 +1,651 @@
+"""Agent connections recorded for every switch-core process.
+
+An agent's connection lives in the memory of the process holding its socket
+(`agent_connections.py`): the socket, the heartbeat and the frames it is owed
+cannot be anywhere else. Its other requests (tool calls, room claims,
+placements, an old client's heartbeat) may reach any process, and every
+process answers presence questions about it. So the connection is also
+recorded as rows (`AgentConnectionStore`), and each process keeps a copy of
+the other processes' live connections in its own registry, which is what lets
+every presence reader answer without a query.
+
+What is written, and when:
+
+- A connection's row is written when it opens or is reattached, inside a
+ recorded move (`recorded`), and closed when it closes, by a queued writer
+ (`closed`), fenced on the process and the incarnation so a late closing never
+ touches a newer row. Detaching a socket writes nothing: the heartbeat window
+ is the grace a client has to come back. Heartbeats write nothing at all.
+- Room claims and session placements are rewritten whenever a recorded move
+ changes them. Every recorded move takes a per-agent advisory lock and reads
+ the agent's rows again first, so a fence, a claimant or a cap is judged on
+ what every process has committed.
+
+After each write the other processes are told to look, over the `SignalBus`
+(`agent_connections.changed`), and read the agent's rows again. The process
+holding a connection that lost a room is told which, so it can send
+`room_released`. A connection whose row now names another process or a later
+incarnation was taken over, and its holder closes it `taken_over`. One whose
+row is gone was its agent's, deleted since, and its holder closes it too: told
+by the process that deleted the agent, or on its next read of every tenant.
+
+Liveness is the holding process's lease: a row counts while it is not closed
+and its process's lease is held. A lost signal costs a delay, never a wrong
+answer for long: on the signal bus's periodic re-check, and whenever the
+signal connection comes back, every tenant's rows are read again, and every
+lease check (`WATCH_SECONDS`) drops the copies of a process whose lease
+lapsed. What a signal asks this process to read goes through the bus's
+`PendingWork`, so a read that fails is kept and tried again.
+
+Two requests are handed to the holder rather than served where they land: a
+`POST /connection/beat` from an old event-stream client (forwarded in batches,
+`agent_connections.beats`) and a session command for a stream another process
+holds, the agent's watcher or its controller's (stored as a relay row for that
+process and signalled, `agent_connections.session_command`, since its content
+may not travel in a `NOTIFY`).
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+from dataclasses import dataclass
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from switch_core.bridges.agent.protocol.agent_connections import (
+ AgentConnection,
+ AgentConnectionRegistry,
+ Closure,
+ ConnectionError_,
+ Released,
+ UnknownConnectionError,
+)
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.agent_connection_store import (
+ AgentConnectionStore,
+ AgentSlotRows,
+)
+from switch_core.db.tenant_lookup import all_tenant_ids
+from switch_core.process_lease import (
+ LEASE_TTL,
+ PRUNE_AFTER,
+ RENEW_INTERVAL_SECONDS,
+ ProcessLease,
+)
+from switch_core.signals import MAX_PAYLOAD_BYTES, RETRY_SECONDS
+from switch_core.tenant_context import current_tenant_id, no_tenant
+
+if TYPE_CHECKING:
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+ from switch_core.signals import SignalBus
+
+logger = logging.getLogger(__name__)
+
+CHANGED_SIGNAL = "agent_connections.changed"
+BEATS_SIGNAL = "agent_connections.beats"
+SESSION_COMMAND_SIGNAL = "agent_connections.session_command"
+
+# How many incarnations are drawn from the sequence at a time. One draw per
+# this many opens, and a process restart wastes at most this many numbers out
+# of about nine quadrillion.
+INCARNATION_BATCH = 32
+
+# How often beats forwarded to their holder go out, at most. A client beats
+# every two seconds and lapses after six, so this leaves the holder plenty.
+BEAT_FORWARD_SECONDS = 0.5
+
+# How often every lease is checked, so a process that died stops counting as
+# the holder of its connections soon after its lease goes stale, rather than
+# at the bus's next re-check.
+WATCH_SECONDS = RENEW_INTERVAL_SECONDS
+
+# How long a session command waits for its watcher's process to take it.
+RELAY_KEPT = timedelta(seconds=60)
+
+# Room kept in every signal for what it carries beside its list of beats or
+# released rooms: its kind, the sending process's id, and the tenant and agent
+# a `changed` signal names, about 250 bytes at most.
+_SIGNAL_ENVELOPE_BYTES = 400
+
+_EMPTY = AgentSlotRows(connections={}, stored_claims={}, stored_placements={})
+
+
+@dataclass(frozen=True)
+class _Close:
+ tenant_id: str
+ agent_id: str
+ connection_id: str
+ incarnation: int
+ reason: str
+
+
+@dataclass(frozen=True)
+class _Relay:
+ tenant_id: str
+ agent_id: str
+ process_id: str
+ frame: dict[str, Any]
+
+
+class SharedConnections:
+ """One process's side of the recorded agent connections."""
+
+ def __init__(
+ self,
+ *,
+ registry: AgentConnectionRegistry,
+ session_factory: async_sessionmaker[AsyncSession],
+ store: AgentConnectionStore,
+ bus: SignalBus,
+ lease: ProcessLease,
+ ) -> None:
+ if lease.process_id != bus.process_id:
+ raise ValueError(
+ "The process lease and the signal bus must name the same process"
+ )
+ self.process_id = bus.process_id
+ self._registry = registry
+ self._session_factory = session_factory
+ self._store = store
+ self._bus = bus
+ self._lease = lease
+ self._incarnations: list[int] = []
+ # Bumped every time an agent's state here is set from the rows or
+ # written to them, so a full pass that read the rows before a move
+ # does not set them back to what they were before it.
+ self._versions: dict[str, int] = {}
+ self._closes: dict[str, _Close] = {}
+ self._relays: list[_Relay] = []
+ self._writes = asyncio.Event()
+ self._write_lock = asyncio.Lock()
+ self._beats: dict[str, list[Any]] = {}
+ self._beats_waiting = asyncio.Event()
+ # Agents, by (tenant id, agent id), whose connections another process
+ # changed and this one has still to read; and the rooms that process
+ # took off connections held here, to be told once the read is in.
+ self._changes_owed = bus.pending_work("agent connections", self._read_owed)
+ self._released_owed: dict[tuple[str, str], list[Released]] = {}
+ # Agents, by (tenant id, agent id), with session commands relayed to
+ # this process and not yet taken.
+ self._relays_owed = bus.pending_work(
+ "relayed session commands", self._take_relays
+ )
+ bus.handle(CHANGED_SIGNAL, self._on_changed)
+ bus.handle(BEATS_SIGNAL, self._on_beats)
+ bus.handle(SESSION_COMMAND_SIGNAL, self._on_session_command)
+ bus.on_resync(self.reload_all)
+ bus.on_recheck(self.reload_all)
+ registry.use_rows(self)
+
+ # ── What the registry asks of its rows ────────────────────────────────
+
+ def next_incarnation(self) -> int:
+ if not self._incarnations:
+ raise RuntimeError(
+ "No incarnation drawn: a connection is opened inside a recorded "
+ "move, which draws them"
+ )
+ return self._incarnations.pop(0)
+
+ @asynccontextmanager
+ async def recorded(self, agent_id: str) -> AsyncIterator[None]:
+ tenant_id = current_tenant_id()
+ if tenant_id is None:
+ raise RuntimeError(
+ f"Agent {agent_id}'s room slots are recorded in its tenant, and "
+ "none is bound"
+ )
+ if not self._registry.holds_slots(agent_id):
+ raise RuntimeError(
+ f"A recorded move for agent {agent_id} is made inside its slots"
+ )
+ registry = self._registry
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ await self._store.lock_agent(session, agent_id)
+ rows = await self._read_agent(session, agent_id)
+ # Every incarnation of the agent's connections is later than the
+ # ones recorded before it, so a process holding an older one
+ # knows a newer one presented is current and its copy is stale.
+ # Numbers drawn in an earlier batch than the latest recorded are
+ # passed over: the sequence's next ones are later than both.
+ latest = max(
+ (row.incarnation for row in rows.connections.values()), default=0
+ )
+ self._incarnations = [n for n in self._incarnations if n > latest]
+ if not self._incarnations:
+ self._incarnations.extend(
+ await self._store.incarnations(session, INCARNATION_BATCH)
+ )
+ self._apply(agent_id, tenant_id, rows)
+ # Compared with what is recorded rather than with the copy just
+ # set from it: a claim of a connection this process has closed,
+ # and not yet recorded as closed, is in the rows but not here,
+ # and a connection opening again on that id must not inherit it.
+ slots_recorded = (rows.claims(), rows.placements())
+ held_before = registry.incarnations_held_here(agent_id)
+ opened: list[AgentConnection] = []
+ try:
+ with registry.collect_released() as released:
+ yield
+ slots_after = registry.slot_state(agent_id)
+ opened = [
+ conn
+ for conn in registry.held_here(agent_id)
+ if held_before.get(conn.id) != conn.stream_generation
+ ]
+ for conn in opened:
+ conn.tenant_id = tenant_id
+ written = await self._store.record_open(
+ session,
+ tenant_id=tenant_id,
+ connection_id=conn.id,
+ agent_id=agent_id,
+ process_id=self.process_id,
+ incarnation=conn.stream_generation,
+ scope=conn.scope,
+ delivery_filter=conn.delivery_filter,
+ spawn_capable=conn.spawn_capable,
+ transport=conn.stream_transport,
+ speaks=conn.declaration.speaks,
+ accepts=conn.declaration.protocol_floor,
+ )
+ if not written:
+ # Another agent's connection has this id, opened
+ # through a process this one has not heard from.
+ raise UnknownConnectionError(conn.id)
+ if slots_after != slots_recorded:
+ claims, placements = slots_after
+ await self._store.replace_slots(
+ session,
+ tenant_id=tenant_id,
+ agent_id=agent_id,
+ claims=claims,
+ placements=placements,
+ )
+ await session.commit()
+ except BaseException:
+ # Nothing was recorded, so nothing the block did stands: a
+ # connection it opened is closed rather than left serving a row
+ # no other process can see, and the rest is set back to what is
+ # recorded.
+ for conn in registry.held_here(agent_id):
+ if held_before.get(conn.id) != conn.stream_generation:
+ conn.tenant_id = None
+ registry.close(
+ conn.id,
+ Closure(
+ code="closed",
+ message="the connection could not be recorded",
+ room_id=None,
+ ),
+ )
+ self._apply(agent_id, tenant_id, rows)
+ raise
+ self._bump(agent_id)
+ if opened or slots_after != slots_recorded or released:
+ self._publish_changed(tenant_id, agent_id, released)
+
+ async def refresh(self, agent_id: str) -> None:
+ tenant_id = current_tenant_id()
+ if tenant_id is None:
+ raise RuntimeError(
+ f"Agent {agent_id}'s connections are read in its tenant, and none "
+ "is bound"
+ )
+ await self._refresh(tenant_id, agent_id)
+
+ async def agent_deleted(self, agent_id: str) -> None:
+ tenant_id = current_tenant_id()
+ if tenant_id is None:
+ raise RuntimeError(
+ f"Agent {agent_id}'s connections are recorded in its tenant, and "
+ "none is bound"
+ )
+ # Its rows went with it, so there is nothing to read: what this
+ # process holds of the agent is made what no rows say, which closes
+ # the connections held here and drops the copies of the others'.
+ async with self._registry.slots(agent_id):
+ self._apply(agent_id, tenant_id, _EMPTY)
+ self._publish_changed(tenant_id, agent_id, [])
+
+ def closed(self, conn: AgentConnection) -> None:
+ if conn.tenant_id is None or conn.closure is None:
+ return
+ self._closes[conn.id] = _Close(
+ tenant_id=conn.tenant_id,
+ agent_id=conn.agent_id,
+ connection_id=conn.id,
+ incarnation=conn.stream_generation,
+ reason=conn.closure.code,
+ )
+ self._writes.set()
+
+ def forward_beat(self, conn: AgentConnection, generation: int | None) -> None:
+ self._beats[conn.id] = [conn.agent_id, conn.id, generation]
+ self._beats_waiting.set()
+
+ def relay_session_command(
+ self, tenant_id: str, agent_id: str, process_id: str, frame: dict[str, Any]
+ ) -> None:
+ self._relays.append(
+ _Relay(
+ tenant_id=tenant_id,
+ agent_id=agent_id,
+ process_id=process_id,
+ frame=frame,
+ )
+ )
+ self._writes.set()
+
+ # ── Reading the rows into the registry ────────────────────────────────
+
+ async def _read_agent(self, session: AsyncSession, agent_id: str) -> AgentSlotRows:
+ return await self._store.read_agent(
+ session, agent_id, process_id=self.process_id, lease_ttl=LEASE_TTL
+ )
+
+ def _apply(self, agent_id: str, tenant_id: str, rows: AgentSlotRows) -> None:
+ for conn in self._registry.apply_rows(
+ agent_id, tenant_id, rows, self.process_id
+ ):
+ logger.info(
+ "[CONN] agent=%s connection=%s generation=%s was taken over by "
+ "another process or a later incarnation; closed here",
+ agent_id,
+ conn.id,
+ conn.stream_generation,
+ )
+ self._bump(agent_id)
+
+ def _bump(self, agent_id: str) -> None:
+ self._versions[agent_id] = self._versions.get(agent_id, 0) + 1
+
+ async def _refresh(self, tenant_id: str, agent_id: str) -> None:
+ async with self._registry.slots(agent_id):
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ rows = await self._read_agent(session, agent_id)
+ self._apply(agent_id, tenant_id, rows)
+ # Told only once the read is in, so the rooms the stream reports
+ # and the rooms it is told it lost agree.
+ self._registry.notify_released(
+ self._released_owed.pop((tenant_id, agent_id), [])
+ )
+
+ async def reload_all(self) -> None:
+ """Read every tenant's rows again, and take any session commands
+ waiting for a stream held here. Run when the signal connection comes
+ back and on the bus's periodic re-check, so a tenant that cannot be
+ read is logged and left to the next run, and the others are read."""
+ with no_tenant():
+ for tenant_id in await all_tenant_ids(self._session_factory):
+ try:
+ await self._reload_tenant(tenant_id)
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ logger.error(
+ "Could not read tenant %s's agent connections again",
+ tenant_id,
+ exc_info=True,
+ )
+
+ async def _reload_tenant(self, tenant_id: str) -> None:
+ versions = dict(self._versions)
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ pruned = await self._store.prune(
+ session, tenant_id, closed_before=PRUNE_AFTER, relays_before=RELAY_KEPT
+ )
+ found = await self._store.read_tenant(
+ session, tenant_id, process_id=self.process_id, lease_ttl=LEASE_TTL
+ )
+ relayed = await self._store.take_relays(
+ session, tenant_id, process_id=self.process_id, agent_id=None
+ )
+ await session.commit()
+ if pruned:
+ logger.info(
+ "Pruned %d agent connection row(s) in tenant %s closed or held by a "
+ "process long gone",
+ pruned,
+ tenant_id,
+ )
+ for agent_id in set(found) | self._registry.agents_recorded_in(tenant_id):
+ async with self._registry.slots(agent_id):
+ if self._versions.get(agent_id, 0) != versions.get(agent_id, 0):
+ # Moved or read again since this pass read the rows, so
+ # what it holds is newer than what this pass found.
+ continue
+ self._apply(agent_id, tenant_id, found.get(agent_id, _EMPTY))
+ self._deliver(relayed)
+
+ def _deliver(self, relayed: list[tuple[str, dict[str, Any]]]) -> None:
+ for agent_id, frame in relayed:
+ if not self._registry.deliver_relayed(agent_id, frame):
+ logger.warning(
+ "A session command for agent %s arrived after the stream it "
+ "was relayed to went; dropped",
+ agent_id,
+ )
+
+ # ── Signals from the other processes ──────────────────────────────────
+
+ def _on_changed(self, signal: dict[str, Any]) -> None:
+ key = (signal["tenant_id"], signal["agent_id"])
+ self._released_owed.setdefault(key, []).extend(
+ Released(connection_id=entry[0], room_id=entry[1], session_id=entry[2])
+ for entry in signal.get("released", [])
+ )
+ self._changes_owed.add(key)
+
+ async def _read_owed(self, key: tuple[str, str]) -> None:
+ tenant_id, agent_id = key
+ await self._refresh(tenant_id, agent_id)
+
+ def _on_beats(self, signal: dict[str, Any]) -> None:
+ for agent_id, connection_id, generation in signal["beats"]:
+ conn = self._registry.get(connection_id)
+ if conn is None or conn.held_by is not None or conn.agent_id != agent_id:
+ continue
+ try:
+ self._registry.beat(agent_id, connection_id, generation)
+ except ConnectionError_ as refusal:
+ # Fenced already where the beat landed; a refusal here means the
+ # connection moved on in between, and the client is told at its
+ # next beat.
+ logger.debug(
+ "A forwarded beat for connection %s was refused: %s",
+ connection_id,
+ refusal,
+ )
+
+ def _on_session_command(self, signal: dict[str, Any]) -> None:
+ if signal["process_id"] != self.process_id:
+ return
+ self._relays_owed.add((signal["tenant_id"], signal["agent_id"]))
+
+ async def _take_relays(self, key: tuple[str, str]) -> None:
+ tenant_id, agent_id = key
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ relayed = await self._store.take_relays(
+ session, tenant_id, process_id=self.process_id, agent_id=agent_id
+ )
+ await session.commit()
+ self._deliver(relayed)
+
+ def _publish_changed(
+ self, tenant_id: str, agent_id: str, released: list[Released]
+ ) -> None:
+ entries = [[r.connection_id, r.room_id, r.session_id] for r in released]
+ # Sent with no rooms released too: the change is what is announced.
+ for chunk in _chunks(entries) or [[]]:
+ self._bus.publish(
+ CHANGED_SIGNAL, tenant_id=tenant_id, agent_id=agent_id, released=chunk
+ )
+
+ # ── Lifecycle ─────────────────────────────────────────────────────────
+
+ async def start(self) -> None:
+ """Read every other process's connections before this one serves.
+
+ A tenant that cannot be read fails the start, as reading the
+ controllers' connections does (`Management.load_connections`), rather
+ than being logged as a re-read is: this process would otherwise serve
+ and answer for that tenant's agents connected elsewhere as offline."""
+ with no_tenant():
+ for tenant_id in await all_tenant_ids(self._session_factory):
+ await self._reload_tenant(tenant_id)
+
+ async def run(self) -> None:
+ """Write closings and relays, forward beats and watch the leases, until
+ cancelled. What signals ask this process to read is worked by the
+ signal bus, which runs it (`PendingWork`)."""
+ # A task group rather than `gather`: cancelled, it waits for every
+ # loop to unwind, so none is left holding a database connection.
+ async with asyncio.TaskGroup() as group:
+ group.create_task(self._write_forever())
+ group.create_task(self._forward_beats_forever())
+ group.create_task(self._watch_forever())
+
+ async def flush(self) -> None:
+ """Write every queued closing and relay. Raises the first failure,
+ keeping what was not written."""
+ async with self._write_lock:
+ closes = list(self._closes.values())
+ self._closes.clear()
+ relays, self._relays = self._relays, []
+ unwritten_closes = list(closes)
+ unwritten_relays = list(relays)
+ try:
+ for close in closes:
+ async with tenant_session(
+ self._session_factory, close.tenant_id
+ ) as session:
+ written = await self._store.record_closed(
+ session,
+ tenant_id=close.tenant_id,
+ connection_id=close.connection_id,
+ incarnation=close.incarnation,
+ process_id=self.process_id,
+ reason=close.reason,
+ )
+ await session.commit()
+ unwritten_closes.remove(close)
+ if written:
+ self._publish_changed(close.tenant_id, close.agent_id, [])
+ for relay in relays:
+ async with tenant_session(
+ self._session_factory, relay.tenant_id
+ ) as session:
+ await self._store.add_relay(
+ session,
+ tenant_id=relay.tenant_id,
+ agent_id=relay.agent_id,
+ process_id=relay.process_id,
+ frame=relay.frame,
+ )
+ await session.commit()
+ unwritten_relays.remove(relay)
+ self._bus.publish(
+ SESSION_COMMAND_SIGNAL,
+ tenant_id=relay.tenant_id,
+ agent_id=relay.agent_id,
+ process_id=relay.process_id,
+ )
+ except BaseException:
+ for close in unwritten_closes:
+ self._closes.setdefault(close.connection_id, close)
+ self._relays[:0] = unwritten_relays
+ raise
+
+ async def _write_forever(self) -> None:
+ while True:
+ await self._writes.wait()
+ self._writes.clear()
+ try:
+ await self.flush()
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ logger.exception(
+ "Recording agent connection closings and session commands "
+ "failed; %d kept and retried in %.0fs",
+ len(self._closes) + len(self._relays),
+ RETRY_SECONDS,
+ )
+ await asyncio.sleep(RETRY_SECONDS)
+ self._writes.set()
+
+ async def _forward_beats_forever(self) -> None:
+ while True:
+ await self._beats_waiting.wait()
+ self._beats_waiting.clear()
+ beats, self._beats = list(self._beats.values()), {}
+ for chunk in _chunks(beats):
+ try:
+ self._bus.publish(BEATS_SIGNAL, beats=chunk)
+ except ValueError as error:
+ # Only a beat too large to send even alone is refused, and
+ # only a connection id recorded by a process that does not
+ # bound ids makes one. It is dropped rather than ending
+ # this loop, and with it the writer and the lease watcher
+ # that run beside it (`run`).
+ logger.error(
+ "A beat of agent %s could not be forwarded to its holder "
+ "and was dropped: %s",
+ chunk[0][0],
+ error,
+ )
+ await asyncio.sleep(BEAT_FORWARD_SECONDS)
+
+ async def _watch_forever(self) -> None:
+ while True:
+ await asyncio.sleep(WATCH_SECONDS)
+ try:
+ leases = await self._lease.read_all()
+ lapsed = {
+ process_id
+ for process_id in self._registry.held_by_processes()
+ if not leases.holds(process_id)
+ }
+ if not lapsed:
+ continue
+ agents = self._registry.drop_copies_held_by(lapsed)
+ logger.warning(
+ "Switch-core process(es) %s stopped holding their lease; "
+ "their connections of %d agent(s) no longer count",
+ ", ".join(sorted(lapsed)),
+ len(agents),
+ )
+ # The placements those processes held go with them too, which
+ # only a read of the rows can tell apart.
+ await self.reload_all()
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ logger.exception("Checking the other processes' connections failed")
+
+
+def _chunks(entries: list[list[Any]]) -> list[list[list[Any]]]:
+ """Split a signal's entries, forwarded beats or released rooms, into lists
+ that each fit in one payload under the `NOTIFY` limit. Measured as JSON,
+ which writes a character outside ASCII in up to twelve bytes. An entry
+ too large to fit even alone gets a list of its own, which the bus then
+ refuses."""
+ chunks: list[list[list[Any]]] = []
+ current: list[list[Any]] = []
+ size = _SIGNAL_ENVELOPE_BYTES
+ for entry in entries:
+ cost = len(json.dumps(entry)) + 1
+ if current and size + cost > MAX_PAYLOAD_BYTES:
+ chunks.append(current)
+ current, size = [], _SIGNAL_ENVELOPE_BYTES
+ current.append(entry)
+ size += cost
+ if current:
+ chunks.append(current)
+ return chunks
diff --git a/core/switch_core/bridges/agent/protocol/controller_presence.py b/core/switch_core/bridges/agent/protocol/controller_presence.py
index 58c5663c9..372e17d6f 100644
--- a/core/switch_core/bridges/agent/protocol/controller_presence.py
+++ b/core/switch_core/bridges/agent/protocol/controller_presence.py
@@ -11,20 +11,27 @@ all, is the controller's business.
Core owns this state and never reads Management's tables. Management fills it
through the narrow surface at the top of the class (`load`, `bind`, `unbind`,
-`revoke_controller`) at startup and after every change it commits, and each of
-those drops what the controller-token cache (`ControllerAuthCache`) holds for
-the controller or agent it touches. Everything
+`reconcile`, `revoke_controller`) at startup, after every change it commits,
+and when another process signals a change it committed (`management/signals.py`),
+and each of those drops what the controller-token cache (`ControllerAuthCache`)
+holds for the controller or agent it touches. Everything
else here is read by Core: the presence readers through `AgentConnectionRegistry`,
the bearer middleware's act-as check, and the controller stream.
-Like the connection registry it is memory only. A restart forgets every
-controller connection, and Management loads the bindings again before the
-agent bridge serves. Each controller connection's transitions, its socket
-attaching and its socket going (and why), are also handed to a ledger, which
-Management installs (`use_ledger`) and which persists them, so that whether
-a machine is connected can be read from any process. Heartbeats are not:
-they stay here. The ledger is told synchronously, as each transition happens,
-once per transition.
+A controller's connection lives in the memory of the process holding its
+socket. A restart forgets every controller connection, and Management loads
+the bindings again before the agent bridge serves. Each controller
+connection's transitions, its socket attaching and its socket going (and
+why), are also handed to a ledger, which Management installs (`use_ledger`)
+and which persists them, so that whether a machine is connected can be read
+from any process. Heartbeats are not: they stay here. The ledger is told
+synchronously, as each transition happens, once per transition.
+
+Every other process learns from those rows which process holds each
+controller's live socket (`held_elsewhere`), so an agent run by a controller
+connected anywhere reads as live everywhere, and a connection held here that
+another process has since taken over is closed. Management keeps that, and
+the bindings, in step across processes (`management/signals.py`).
"""
from __future__ import annotations
@@ -200,6 +207,11 @@ class ControllerPresence:
self._on_bound = on_bound
self._bindings: dict[str, Binding] = {}
self._by_controller: dict[str, set[str]] = {}
+ # A count of binding changes, and the count at each agent's last one,
+ # so a reconcile built from a read that started earlier can tell
+ # which agents changed under it (`change_mark`, `reconcile`).
+ self._changes = 0
+ self._changed_at: dict[str, int] = {}
self._connections: dict[str, ControllerConnection] = {}
self._superseded: dict[str, deque[str]] = {}
self._revoked: set[str] = set()
@@ -212,6 +224,29 @@ class ControllerPresence:
# Set as the process begins shutting down: a socket going after that
# went because the server closed it.
self._shutting_down = False
+ # Controllers whose live socket another process holds, and which one,
+ # as Management reads it from their rows. Dropped when that process's
+ # lease lapses (`forget_processes`) or its row says the socket went.
+ self._elsewhere: dict[str, str] = {}
+ # A count of changes to which process holds each controller's
+ # connection, and the count at each controller's last one, as for the
+ # bindings: so a read of the controllers' rows that started earlier
+ # can tell which controllers changed under it (`connection_mark`).
+ self._connection_changes = 0
+ self._connection_changed_at: dict[str, int] = {}
+
+ def connection_mark(self) -> int:
+ """Where the connection changes stand now. Taken before reading the
+ controllers' rows, so what is read is not applied over a change made
+ here since (`connection_changed_since`)."""
+ return self._connection_changes
+
+ def connection_changed_since(self, controller_id: str, mark: int) -> bool:
+ return self._connection_changed_at.get(controller_id, 0) > mark
+
+ def _note_connection_change(self, controller_id: str) -> None:
+ self._connection_changes += 1
+ self._connection_changed_at[controller_id] = self._connection_changes
def use_auth_cache(self, cache: ControllerAuthCache) -> None:
"""The cache of controller-token reads to keep in step with the
@@ -252,8 +287,45 @@ class ControllerPresence:
for binding in bindings:
self.bind(binding)
+ def change_mark(self) -> int:
+ """Where the binding changes stand now. Taken before reading
+ Management's tables for `reconcile`."""
+ return self._changes
+
+ def _note_change(self, agent_id: str) -> None:
+ self._changes += 1
+ self._changed_at[agent_id] = self._changes
+
+ def reconcile(
+ self, bindings: Iterable[Binding], *, agent_ids: Iterable[str], since: int
+ ) -> None:
+ """Make the bindings of `agent_ids` what Management's tables said,
+ changing only what differs: an agent in `agent_ids` with no binding
+ in `bindings` is unbound.
+
+ `bindings` was read after `since` was taken (`change_mark`). An agent
+ this process bound or unbound after that has a newer answer than the
+ read, so it is left as it is: otherwise a re-check that read the
+ tables a moment before a request placed an agent would unbind it
+ again until the next re-check.
+ """
+ wanted = {binding.agent_id: binding for binding in bindings}
+ for agent_id in agent_ids:
+ if self._changed_at.get(agent_id, 0) > since:
+ continue
+ binding = wanted.get(agent_id)
+ if binding is None:
+ if agent_id in self._bindings:
+ self.unbind(agent_id, DETACH_UNASSIGNED)
+ elif self._bindings.get(agent_id) != binding:
+ self.bind(binding)
+
+ def bound_agent_ids(self) -> set[str]:
+ return set(self._bindings)
+
def bind(self, binding: Binding) -> None:
"""Bind an agent to a controller, moving it off any other."""
+ self._note_change(binding.agent_id)
previous = self._bindings.get(binding.agent_id)
if previous is not None:
self._forget(previous)
@@ -286,6 +358,7 @@ class ControllerPresence:
def unbind(self, agent_id: str, reason: str) -> None:
"""The agent is no longer controller-backed."""
+ self._note_change(agent_id)
previous = self._bindings.pop(agent_id, None)
self._rooms.pop(agent_id, None)
self._invalidate_agent_auth(agent_id)
@@ -303,6 +376,7 @@ class ControllerPresence:
def rename_controller(self, controller_id: str, name: str) -> None:
"""The controller's owner renamed it: what its agents' bindings call it."""
for agent_id in self._by_controller.get(controller_id, set()):
+ self._note_change(agent_id)
binding = self._bindings[agent_id]
self._bindings[agent_id] = replace(binding, controller_name=name)
@@ -364,6 +438,12 @@ class ControllerPresence:
return None
return conn
+ def _connected(self, controller_id: str) -> bool:
+ """Whether the controller's socket is live here or in another process."""
+ return self._live_connection(controller_id) is not None or (
+ controller_id in self._elsewhere and controller_id not in self._revoked
+ )
+
def is_stopped(self, agent_id: str) -> bool:
"""Bound, and set to stopped by its owner."""
binding = self._bindings.get(agent_id)
@@ -374,12 +454,17 @@ class ControllerPresence:
return (
binding is not None
and binding.running
- and self._live_connection(binding.controller_id) is not None
+ and self._connected(binding.controller_id)
)
def rooms(self, agent_id: str) -> set[str]:
return set(self._rooms.get(agent_id, ()))
+ def rooms_kept(self) -> dict[str, set[str]]:
+ """The rooms this process keeps for each agent a controller streamed
+ from here, the ones `room_joined` and `room_left` change."""
+ return {agent_id: set(rooms) for agent_id, rooms in self._rooms.items()}
+
def live_in_room(self, agent_id: str, room_id: str) -> bool:
return self.is_live(agent_id) and room_id in self._rooms.get(agent_id, ())
@@ -392,6 +477,15 @@ class ControllerPresence:
def live_agent_ids(self) -> set[str]:
return {agent_id for agent_id in self._bindings if self.is_live(agent_id)}
+ def live_agent_ids_held_here(self) -> set[str]:
+ """The live agents whose controller's socket this process holds."""
+ return {
+ agent_id
+ for agent_id, binding in self._bindings.items()
+ if binding.running
+ and self._live_connection(binding.controller_id) is not None
+ }
+
def live_holder_ids(self) -> set[str]:
return {
self.holder_id(binding)
@@ -400,13 +494,91 @@ class ControllerPresence:
}
def live_connection_count(self) -> int:
+ """The controller connections this process holds that are live."""
now = time.monotonic()
return sum(1 for conn in self._connections.values() if conn.is_live(now))
+ # ------------------------------------------------------------------
+ # Controllers held by other processes
+ # ------------------------------------------------------------------
+
+ def held_elsewhere(self, controller_id: str, process_id: str | None) -> None:
+ """Which other process holds the controller's live socket, or None
+ when no other process does."""
+ self._note_connection_change(controller_id)
+ if process_id is None:
+ self._elsewhere.pop(controller_id, None)
+ else:
+ self._elsewhere[controller_id] = process_id
+
+ def holder_elsewhere(self, controller_id: str) -> str | None:
+ """The other process holding the controller's live socket, if any."""
+ return self._elsewhere.get(controller_id)
+
+ def process_holding(self, agent_id: str) -> str | None:
+ """The other process holding the live socket of the controller that
+ runs this agent, while the agent is set running."""
+ binding = self._bindings.get(agent_id)
+ if binding is None or not binding.running:
+ return None
+ return self._elsewhere.get(binding.controller_id)
+
+ def processes_elsewhere(self) -> set[str]:
+ return set(self._elsewhere.values())
+
+ def controllers_elsewhere(self) -> set[str]:
+ return set(self._elsewhere)
+
+ def held_elsewhere_count(self) -> int:
+ """How many controllers' live sockets other processes hold."""
+ return sum(
+ 1 for controller_id in self._elsewhere if controller_id not in self._revoked
+ )
+
+ def forget_processes(self, process_ids: set[str]) -> set[str]:
+ """These processes' leases lapsed: the controllers they held are not
+ connected. Returns those controllers."""
+ gone = {
+ controller_id
+ for controller_id, process_id in self._elsewhere.items()
+ if process_id in process_ids
+ }
+ for controller_id in gone:
+ self._note_connection_change(controller_id)
+ del self._elsewhere[controller_id]
+ return gone
+
+ def supersede(self, controller_id: str) -> None:
+ """The controller's connection held here was taken over through
+ another process: its stream ends with `taken_over`, and a late beat
+ from it is told so."""
+ self._note_connection_change(controller_id)
+ previous = self._connections.pop(controller_id, None)
+ if previous is not None:
+ self._take_over(previous)
+ logger.info(
+ "[CONTROLLER] controller=%s connection=%s taken over through "
+ "another process",
+ controller_id,
+ previous.id,
+ )
+
+ def _take_over(self, previous: ControllerConnection) -> None:
+ previous.closure = TAKEN_OVER
+ self._record_disconnected(previous, TAKEN_OVER.code)
+ previous.stream_attached = False
+ previous.wake.set()
+ remembered = self._superseded.setdefault(
+ previous.controller_id, deque(maxlen=_SUPERSEDED_REMEMBERED)
+ )
+ remembered.append(previous.id)
+
def relay_session_command(self, agent_id: str, frame: dict[str, Any]) -> bool:
- """Queue a session command on the agent's controller stream. False when
- its controller is not live or the agent is stopped, so the sender is
- told rather than the command held for a stream that may not return."""
+ """Queue a session command on the agent's controller stream, when this
+ process holds it. False when its controller is not live here or the
+ agent is stopped, so the sender is told rather than the command held
+ for a stream that may not return. A controller held by another process
+ is reached through the registry (`process_holding`)."""
binding = self._bindings.get(agent_id)
if binding is None or not binding.running:
return False
@@ -455,31 +627,102 @@ class ControllerPresence:
# The controller's connection
# ------------------------------------------------------------------
+ def new_connection(self) -> tuple[str, int]:
+ """An id and a generation for a connection the controller is opening.
+
+ Drawn before the connection is held (`open`), so the open can be
+ recorded first: a read of the controller's row taken between holding
+ it and recording it would find the connection it replaces still named
+ there, and close the new one as taken over."""
+ generation = self._next_generation
+ self._next_generation += 1
+ return str(uuid.uuid4()), generation
+
def open(
self,
*,
controller_id: str,
tenant_id: str,
resume_cursors: dict[str, int | None],
+ identity: tuple[str, int] | None = None,
+ ) -> ControllerConnection:
+ """Open a connection for the controller, taking over any it had.
+
+ `identity` is the id and generation drawn for it (`new_connection`);
+ a new one is drawn when none is given."""
+ connection_id, generation = identity or self.new_connection()
+ conn = self._hold(
+ controller_id=controller_id,
+ tenant_id=tenant_id,
+ connection_id=connection_id,
+ generation=generation,
+ resume_cursors=resume_cursors,
+ )
+ logger.info(
+ "[CONTROLLER] opened controller=%s connection=%s generation=%s agents=%d",
+ controller_id,
+ conn.id,
+ generation,
+ len(self.agents_of(controller_id)),
+ )
+ return conn
+
+ def adopt(
+ self,
+ *,
+ controller_id: str,
+ tenant_id: str,
+ connection_id: str,
+ generation: int,
+ resume_cursors: dict[str, int | None],
+ ) -> ControllerConnection:
+ """Hold here a connection another process opened.
+
+ The open and the socket attaching are two requests, and need not
+ reach the same process; nor need a socket reattaching after a drop.
+ The connection's id, generation and cursors are read from its row, so
+ the attach is fenced on what the open handed the controller wherever
+ it lands. The process that held it closes its copy when told.
+ """
+ conn = self._hold(
+ controller_id=controller_id,
+ tenant_id=tenant_id,
+ connection_id=connection_id,
+ generation=generation,
+ resume_cursors=resume_cursors,
+ )
+ logger.info(
+ "[CONTROLLER] adopted controller=%s connection=%s generation=%s from "
+ "another process",
+ controller_id,
+ connection_id,
+ generation,
+ )
+ return conn
+
+ def _hold(
+ self,
+ *,
+ controller_id: str,
+ tenant_id: str,
+ connection_id: str,
+ generation: int,
+ resume_cursors: dict[str, int | None],
) -> ControllerConnection:
- """Open a connection for the controller, taking over any it had."""
if controller_id in self._revoked:
raise ControllerRevokedError(controller_id)
+ self._note_connection_change(controller_id)
previous = self._connections.get(controller_id)
if previous is not None:
- previous.closure = TAKEN_OVER
- self._record_disconnected(previous, TAKEN_OVER.code)
- previous.stream_attached = False
- previous.wake.set()
- remembered = self._superseded.setdefault(
- controller_id, deque(maxlen=_SUPERSEDED_REMEMBERED)
- )
- remembered.append(previous.id)
+ self._take_over(previous)
+ # Held again here after another process held it: no longer one of
+ # the connections a late beat is told were taken over.
+ remembered = self._superseded.get(controller_id)
+ if remembered is not None and connection_id in remembered:
+ remembered.remove(connection_id)
now = time.monotonic()
- generation = self._next_generation
- self._next_generation += 1
conn = ControllerConnection(
- id=str(uuid.uuid4()),
+ id=connection_id,
controller_id=controller_id,
tenant_id=tenant_id,
generation=generation,
@@ -488,13 +731,6 @@ class ControllerPresence:
opened_at=now,
)
self._connections[controller_id] = conn
- logger.info(
- "[CONTROLLER] opened controller=%s connection=%s generation=%s agents=%d",
- controller_id,
- conn.id,
- generation,
- len(self.agents_of(controller_id)),
- )
return conn
def require(
@@ -521,9 +757,21 @@ class ControllerPresence:
and the client cannot beat before it has one. A connection that had
no socket attached is recorded as connected; displacing an attached
one changes nothing the ledger holds.
+
+ Refused when the connection is no longer the controller's current one
+ here, as `require` refuses it: the attach was decided on a read of the
+ controller's row, and in the wait the controller may have opened
+ another connection, or this one closed. Attaching then would hold a
+ connection that is over, and forget the process holding the newer one.
"""
+ if not self.is_current(conn):
+ if conn.closure is not None and conn.closure.code == TAKEN_OVER.code:
+ raise ControllerTakenOverError(conn.id)
+ raise UnknownControllerConnectionError(conn.id)
if not conn.stream_attached:
self._record_connected(conn)
+ self._note_connection_change(conn.controller_id)
+ self._elsewhere.pop(conn.controller_id, None)
conn.stream_token += 1
conn.stream_attached = True
conn.last_beat = time.monotonic()
@@ -560,6 +808,10 @@ class ControllerPresence:
if current is None or cursor > current:
conn.resume_cursors[agent_id] = cursor
+ def held_connections(self) -> list[ControllerConnection]:
+ """Every controller connection this process holds."""
+ return list(self._connections.values())
+
def current_connection(self, controller_id: str) -> ControllerConnection | None:
"""The controller's open connection, live or not yet swept."""
return self._connections.get(controller_id)
diff --git a/core/switch_core/bridges/agent/protocol/event_buffer.py b/core/switch_core/bridges/agent/protocol/event_buffer.py
index 55c8c70c9..fbef5c43a 100644
--- a/core/switch_core/bridges/agent/protocol/event_buffer.py
+++ b/core/switch_core/bridges/agent/protocol/event_buffer.py
@@ -4,18 +4,21 @@ The events an agent receives come from two tables.
**The agent's own events.** These are the events meant for the agent: a
message addressed to it, a room_join in a room where it listens for joins, a
-task it takes part in, a command sent to it. The agent's consumer
-(`clients/agent_consumer.py`) decides whether a room row is one of these when
-the row is delivered to it live. The decision cannot be repeated later, because
-the addressing policy, budgets and role leases it depended on may have changed
-by then, so the consumer stores it as a row in `agent_events`. These rows are
-called own events. Each agent numbers its own events in its own sequence
-(`agent_events.seq`).
+task it takes part in, a command sent to it. The router
+(`delivery/router.py`) decides whether a room row is one of these as the row
+is written, and stores each decision as a row in `agent_events` in the same
+transaction as the room row (`store`). The decision cannot be repeated later,
+because the addressing policy, budgets and role leases it depended on may have
+changed by then. These rows are called own events. Each agent numbers its own
+events in its own sequence (`agent_events.seq`).
**The rest of the conversation.** Everything else said in the agent's rooms is
already stored in `messages`, numbered per room (`messages.seq`), and is read
from there when a reader asks for it. Nothing is copied per agent, so a message
-in a busy room costs one write, not one write per agent in the room.
+in a busy room costs one write, not one write per agent in the room. A room row
+and the own events decided for it commit together, so a reader that can see
+the row can also see whether it is one of the agent's own events, and can read
+the conversation the moment it is written.
A reader's position (`positions.py`) records how far it has read: one number
for the agent's own events and one number per room for the conversation. Reading
@@ -33,7 +36,9 @@ When an own event is inserted, Postgres sends a notification
(`db/agent_event_notify_ddl.py`). A listener in every process receives it and
sets that agent's doorbell, an in-memory `asyncio.Event`, so the process that
holds the agent's connection wakes up and reads the new own event from the table.
-The doorbell only wakes readers; the table is the record.
+A room row rings no doorbell of an agent's: a reader of the conversation asks to
+be woken by the room's own announcement instead (`rooms`, the
+`MessageListener`). Neither carries anything; the tables are the record.
An unread count says how much unaddressed conversation in a room the agent has
not caught up on. It is counted from `messages` when asked: the room's messages
@@ -50,14 +55,24 @@ import time
from collections.abc import Iterable
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
-from typing import TYPE_CHECKING, Any, cast
-
-from sqlalchemy import BigInteger, bindparam, delete, func, literal, select, text
-from sqlalchemy.dialects.postgresql import ARRAY, insert
+from typing import TYPE_CHECKING, Any, Protocol, cast
+
+from sqlalchemy import (
+ BigInteger,
+ Boolean,
+ Text,
+ bindparam,
+ delete,
+ func,
+ literal,
+ select,
+ text,
+)
+from sqlalchemy.dialects.postgresql import ARRAY, JSONB, insert
from sqlalchemy.exc import DBAPIError, InterfaceError, OperationalError
from sqlalchemy.exc import TimeoutError as PoolTimeoutError
-from switch_core.attachments import parse_attachment_group
+from switch_core.attachments import ATTACHMENT_GROUP_KEY, parse_attachment_group
from switch_core.bridges.agent.protocol.message_events import (
attachment_ref,
message_event,
@@ -85,6 +100,7 @@ from switch_core.db.models import (
ClientRoom,
Message,
MessageAttachment,
+ PendingAttachmentGroup,
Room,
)
from switch_core.db.session_scope import tenant_session
@@ -98,10 +114,31 @@ from switch_core.transport.stored import to_inbound
from switch_core.transport.types import InboundMedia, InboundMembership, InboundMessage
if TYPE_CHECKING:
+ from collections.abc import Awaitable, Callable
+
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
logger = logging.getLogger(__name__)
+
+class RoomAnnouncements(Protocol):
+ """Who announces that a room has new rows: `messages.notify.MessageListener`.
+
+ A reader of the rooms' conversation is woken by it, since a room row rings
+ no doorbell of an agent's. The waker is called with the room's id and is
+ told nothing else: it reads from its own position, as every reader here
+ does.
+ """
+
+ def subscribe(
+ self, room_id: str, waker: Callable[[str], Awaitable[None]]
+ ) -> None: ...
+
+ def unsubscribe(
+ self, room_id: str, waker: Callable[[str], Awaitable[None]]
+ ) -> None: ...
+
+
# How long an own event stays in `agent_events` before `expire` deletes it. A client
# that reconnects within this window receives every own event it missed; a client
# that reconnects later is told it has a gap instead of being handed an
@@ -109,24 +146,11 @@ logger = logging.getLogger(__name__)
# conversation has no such window.
DEFAULT_RETENTION_SECONDS = 24 * 60 * 60
-# How old a row in `messages` must be before it is delivered as conversation.
-# The agent's consumer decides, when the row is delivered to it, whether the
-# row becomes an own event. If a reader delivered the row as conversation before
-# that decision and the consumer then made it an own event, the agent would receive
-# the same message twice. The wait is skipped when the agent already has a
-# own event for a later row in the same room, because the consumer handles a room's
-# rows in order and has therefore decided this one. The consumer normally
-# decides within milliseconds, so the duplicate only happens when it stalls
-# for longer than this; the agent then sees the message twice, never zero
-# times.
-CONVERSATION_SETTLE_SECONDS = 2.0
-
-# Several files posted as one message are stored as separate rows in
-# `messages` that share an attachment group. A reader waits this many seconds
-# beyond the settle time for the rest of the group, then delivers the files
-# that did arrive as one message marked incomplete. The agent's consumer waits
-# the same time (`ATTACHMENT_GROUP_TIMEOUT_SECONDS`) when it assembles a group.
-CONVERSATION_GROUP_GRACE_SECONDS = 5.0
+# How often a long poll that includes the rooms' conversation looks again while
+# it waits. A room row rings no agent's doorbell, and a long poll has no room
+# announcements to wait on, so without this it would sleep through its whole
+# timeout while the rooms it reads were being talked in.
+CONVERSATION_RECHECK_SECONDS = 2.0
# Some clients can store only one integer as their place in the stream. Such a
# client is sent a ticket: the id of a row in `agent_position_tickets` that
@@ -351,8 +375,8 @@ class EventBuffer:
session_factory: async_sessionmaker[AsyncSession],
*,
sequence_base: int,
+ rooms: RoomAnnouncements,
retention_seconds: float = DEFAULT_RETENTION_SECONDS,
- settle_seconds: float = CONVERSATION_SETTLE_SECONDS,
) -> None:
self._session_factory = session_factory
# `sequence_base` comes from a database sequence that advances on every
@@ -363,23 +387,199 @@ class EventBuffer:
# own event. Later own events are numbered from the agent's row in
# `agent_event_heads`, so their numbers survive restarts.
self.sequence_floor = sequence_base + 1
+ # Where a reader of the rooms' conversation asks to be woken when a
+ # room it reads has a new row.
+ self.rooms = rooms
self._retention_seconds = retention_seconds
- self._settle_seconds = settle_seconds
# The doorbells of this process: one `asyncio.Event` per agent, set to
# wake this process's readers of that agent when an own event is inserted.
- # `enqueue` sets it for own events written here, and the notification
+ # The writer sets it for own events written here, and the notification
# listener sets it for own events written by other processes. The events
# themselves are always read from `agent_events`.
self._notify: dict[str, asyncio.Event] = {}
- @property
- def settle_seconds(self) -> float:
- return self._settle_seconds
-
# ------------------------------------------------------------------
# Producing
# ------------------------------------------------------------------
+ async def store(
+ self,
+ session: AsyncSession,
+ *,
+ tenant_id: str,
+ agent_id: str,
+ client_id: str,
+ room_id: str,
+ event: AgentEvent,
+ room_seq: int,
+ message_key: str | None,
+ ) -> int | None:
+ """Number one own event and insert it in the caller's transaction, and
+ return its number, or None when nothing was stored.
+
+ This is how the router (`delivery/router.py`) writes an agent's own
+ events: in the transaction that writes the room row they come from, so
+ the row and the events decided for it commit together or not at all.
+ The number is claimed from the agent's row in `agent_event_heads`, and
+ that row stays locked until the caller commits, so an agent's numbers
+ are handed out in commit order whichever room each event comes from.
+ A caller that stores events for several agents has to claim them in a
+ fixed order (the router sorts by agent id), or two transactions each
+ holding one agent's row would wait for each other.
+
+ The work runs in a savepoint of its own, so a failure here leaves the
+ caller's transaction usable: the room row the caller wrote still
+ commits, and only this agent misses the event.
+
+ Nothing is stored, and the number taken is given back with the
+ savepoint, in two cases. The agent is no longer a member of the room:
+ membership is read after the agent's row is locked, so a removal that
+ committed while this waited for the lock is seen, and a removal that
+ comes later deletes the event under the same lock
+ (`forget_room`). Or an event from the same source is already stored:
+ an agent consumer of an older release, still running beside this one
+ during a rolling deploy, decides the same row and stores it itself.
+ """
+ savepoint = await session.begin_nested()
+ try:
+ seq = (await session.execute(self._claim(tenant_id, agent_id))).scalar_one()
+ member = (
+ select(ClientRoom.room_id)
+ .where(ClientRoom.client_id == client_id, ClientRoom.room_id == room_id)
+ .exists()
+ )
+ inserted = await session.scalar(
+ self._own_event_insert(
+ select(
+ literal(tenant_id, Text),
+ literal(agent_id, Text),
+ literal(seq, BigInteger),
+ literal(room_id, Text),
+ literal(room_seq, BigInteger),
+ literal(message_key, Text),
+ literal(event.type, Text),
+ literal(event.model_dump(mode="json"), JSONB),
+ literal(is_notifiable(event), Boolean),
+ ).where(member)
+ )
+ )
+ except BaseException:
+ await savepoint.rollback()
+ raise
+ if inserted is None:
+ await savepoint.rollback()
+ return None
+ await savepoint.commit()
+ return int(seq)
+
+ def _claim(self, tenant_id: str, agent_id: str) -> Any:
+ """The next number of the agent's own events, taken from its row in
+ `agent_event_heads`, which the statement locks until the transaction
+ ends. An agent with no row yet gets one, starting at `sequence_floor`."""
+ return (
+ insert(AgentEventHead)
+ .values(
+ tenant_id=tenant_id,
+ agent_id=agent_id,
+ first_seq=self.sequence_floor,
+ last_seq=self.sequence_floor,
+ )
+ .on_conflict_do_update(
+ index_elements=[AgentEventHead.agent_id],
+ set_={"last_seq": AgentEventHead.last_seq + 1},
+ )
+ .returning(AgentEventHead.last_seq)
+ )
+
+ @staticmethod
+ def _own_event_insert(values: Any) -> Any:
+ """An insert into `agent_events` of the row `values` selects, that
+ does nothing when an event from the same source is already stored
+ (`uq_agent_events_source`), and returns the stored number."""
+ return (
+ insert(AgentEventRow)
+ .from_select(
+ [
+ AgentEventRow.tenant_id,
+ AgentEventRow.agent_id,
+ AgentEventRow.seq,
+ AgentEventRow.room_id,
+ AgentEventRow.room_seq,
+ AgentEventRow.message_key,
+ AgentEventRow.type,
+ AgentEventRow.payload,
+ AgentEventRow.notifiable,
+ ],
+ values,
+ )
+ .on_conflict_do_nothing(
+ index_elements=[
+ AgentEventRow.agent_id,
+ AgentEventRow.message_key,
+ AgentEventRow.room_id,
+ AgentEventRow.room_seq,
+ AgentEventRow.type,
+ ],
+ index_where=AgentEventRow.message_key.is_not(None),
+ )
+ .returning(AgentEventRow.seq)
+ )
+
+ async def forget_room(
+ self, session: AsyncSession, *, agent_id: str, room_id: str
+ ) -> None:
+ """Delete the agent's own events in one room, in the caller's
+ transaction, which is the one that takes the agent out of the room.
+
+ Sequence numbers are untouched and no gap is recorded: a reader that
+ skips these has missed nothing it was entitled to, and saying
+ otherwise would send it to re-read the context of a room it is not in.
+
+ The agent's row in `agent_event_heads` is locked first, the lock
+ `store` takes to number an event. An event the router was storing for
+ the room while this waited has then committed, and is deleted here;
+ one it stores after this commits finds the membership gone and is not
+ stored. Without the lock a message written as the agent was removed
+ could leave an event behind in a room it is no longer in, which a
+ stream resuming from an old position would hand it.
+ """
+ await session.execute(
+ insert(AgentEventHead)
+ .from_select(
+ [
+ AgentEventHead.tenant_id,
+ AgentEventHead.agent_id,
+ AgentEventHead.first_seq,
+ AgentEventHead.last_seq,
+ ],
+ select(
+ Agent.tenant_id,
+ Agent.id,
+ literal(self.sequence_floor, BigInteger),
+ literal(self.sequence_floor - 1, BigInteger),
+ ).where(Agent.id == agent_id),
+ )
+ .on_conflict_do_update(
+ index_elements=[AgentEventHead.agent_id],
+ set_={"last_seq": AgentEventHead.last_seq},
+ )
+ )
+ result = await session.execute(
+ delete(AgentEventRow).where(
+ AgentEventRow.agent_id == agent_id,
+ AgentEventRow.room_id == room_id,
+ )
+ )
+ dropped = result.rowcount or 0 # type: ignore[attr-defined]
+ if dropped:
+ logger.info(
+ "[EVENT-BUF] dropped %s retained event(s) for agent=%s room=%s: "
+ "no longer a member",
+ dropped,
+ agent_id,
+ room_id,
+ )
+
async def enqueue(
self,
tenant_id: str,
@@ -390,7 +590,14 @@ class EventBuffer:
room_seq: int,
message_key: str | None,
) -> int:
- """Store an event meant for the agent as an own event, and return its number.
+ """Store an event meant for the agent as an own event, in a transaction
+ of its own, and return its number.
+
+ For an event decided after the room row it comes from has committed:
+ a room command's handler (`bridges/agent/commands.py`) queues the
+ command for the agent only once it has checked the agent's session and
+ answered in the room. Everything decided as the row is written goes
+ through `store` instead.
The number comes from the agent's row in `agent_event_heads`, which is
advanced in the same transaction as the insert into `agent_events`, so
@@ -400,9 +607,8 @@ class EventBuffer:
unread counts can recognise that row as already delivered.
A write that fails because the database connection or the pool did is
- retried after each of `ENQUEUE_RETRY_DELAYS`. The consumer has already
- moved past the room row by the time it calls this, so an event not
- stored here is never stored.
+ retried after each of `ENQUEUE_RETRY_DELAYS`. Nothing else will store
+ this event, so one not stored here is never stored.
"""
for delay in (*ENQUEUE_RETRY_DELAYS, None):
try:
@@ -445,14 +651,12 @@ class EventBuffer:
event from the same source is already stored, and return its number.
The source is `(message_key, room_id, room_seq, type)`
- (`uq_agent_events_source`):
- the consumer of another switch-core process makes the same decision
- for the same room row, and a write retried after a commit whose
- acknowledgement was lost repeats one that was made. Either would
+ (`uq_agent_events_source`): a write retried after a commit whose
+ acknowledgement was lost repeats one that was made, and would
otherwise give the agent the same event twice under two numbers. When
- the other write lands between the lookup and the insert, the insert
- does nothing and the transaction is rolled back, which also gives back
- the number it took.
+ another write of the same source lands between the lookup and the
+ insert, the insert does nothing and the transaction is rolled back,
+ which also gives back the number it took.
"""
source = (
select(AgentEventRow.seq).where(
@@ -470,45 +674,21 @@ class EventBuffer:
stored = await session.scalar(source)
if stored is not None:
return int(stored)
- claim = (
- insert(AgentEventHead)
- .values(
- tenant_id=tenant_id,
- agent_id=agent_id,
- first_seq=self.sequence_floor,
- last_seq=self.sequence_floor,
- )
- .on_conflict_do_update(
- index_elements=[AgentEventHead.agent_id],
- set_={"last_seq": AgentEventHead.last_seq + 1},
- )
- .returning(AgentEventHead.last_seq)
- )
- seq = (await session.execute(claim)).scalar_one()
+ seq = (await session.execute(self._claim(tenant_id, agent_id))).scalar_one()
inserted = await session.scalar(
- insert(AgentEventRow)
- .values(
- tenant_id=tenant_id,
- agent_id=agent_id,
- seq=seq,
- room_id=room_id,
- room_seq=room_seq,
- message_key=message_key,
- type=event.type,
- payload=event.model_dump(mode="json"),
- notifiable=is_notifiable(event),
- )
- .on_conflict_do_nothing(
- index_elements=[
- AgentEventRow.agent_id,
- AgentEventRow.message_key,
- AgentEventRow.room_id,
- AgentEventRow.room_seq,
- AgentEventRow.type,
- ],
- index_where=AgentEventRow.message_key.is_not(None),
+ self._own_event_insert(
+ select(
+ literal(tenant_id, Text),
+ literal(agent_id, Text),
+ literal(seq, BigInteger),
+ literal(room_id, Text),
+ literal(room_seq, BigInteger),
+ literal(message_key, Text),
+ literal(event.type, Text),
+ literal(event.model_dump(mode="json"), JSONB),
+ literal(is_notifiable(event), Boolean),
+ )
)
- .returning(AgentEventRow.seq)
)
if inserted is None:
await session.rollback()
@@ -524,7 +704,7 @@ class EventBuffer:
)
return int(stored)
await session.commit()
- return seq
+ return int(seq)
# ------------------------------------------------------------------
# The agent's own events
@@ -831,15 +1011,23 @@ class EventBuffer:
A row is delivered when it is a message or a join by someone other
than the agent, sent after the agent joined the room, and not already
- an own event. A row may only be delivered once the agent's consumer has
- decided whether it becomes an own event. The consumer handles a room's rows
- one at a time, in order, so every row up to the room row of the
- agent's newest own event in that room is decided. A later row is treated as
- decided once it is older than `CONVERSATION_SETTLE_SECONDS`. Each room
- is read in order and the read stops at the first undecided row.
- Several files posted as one message are delivered as one event, once
- all of them are present or `CONVERSATION_GROUP_GRACE_SECONDS` has
- passed.
+ an own event. Whether a row is one of the agent's own events is decided
+ as the row is written, and committed with it (`store`), so a row that
+ can be read here is already decided and is delivered at once.
+
+ The one row still undecided when it can be read is a file of a
+ multi-file post whose other files have not all been written: the post
+ is decided once, as a whole, when its last file arrives or when the
+ router gives up waiting for it (`PendingAttachmentGroup`). The read
+ stops at the first file of such a post, so the post is delivered by a
+ later read as one event rather than in pieces.
+
+ One transient case breaks the rule. During a rolling deploy a process
+ still on the release before the router writes a room row without
+ deciding it, and leaves the decision to its agent consumers a moment
+ later. A read that lands in between delivers the row as conversation,
+ and the agent receives it again as an own event if a consumer then
+ decides it is one.
"""
if not after:
return Conversation()
@@ -874,43 +1062,27 @@ class EventBuffer:
await session.scalars(select(Room).where(Room.id.in_(after)))
).all()
}
- # Computed with the database's clock, which is the clock that
- # stamped `sent_at`, so a difference between this process's clock
- # and the database's cannot shorten or lengthen the settle time.
- settled_before: datetime = cast(
- datetime,
- await session.scalar(
- select(func.now() - timedelta(seconds=self._settle_seconds))
- ),
- )
- group_deadline = settled_before - timedelta(
- seconds=CONVERSATION_GROUP_GRACE_SECONDS
- )
- # For each room, the room row of the agent's newest own event written by
- # its consumer. Only message and room_join own events count: command
- # own events are written elsewhere, so they say nothing about how far
- # the consumer has got in the room.
- decided_through: dict[str, int] = {
- room_id: seq
- for room_id, seq in (
- await session.execute(
- select(AgentEventRow.room_id, func.max(AgentEventRow.room_seq))
- .where(
- AgentEventRow.agent_id == agent_id,
- AgentEventRow.room_id.in_(after),
- AgentEventRow.type.in_(("message", "room_join")),
- )
- .group_by(AgentEventRow.room_id)
- )
- ).all()
- }
for room_id, after_seq in sorted(after.items()):
room = rooms.get(room_id)
if room is None or room_id not in joined:
continue
- fetched = (
- await session.scalars(
- select(Message)
+ # Whether a file's post is still pending is read in the same
+ # statement as the rows: the file that completes a post
+ # writes its row and deletes the pending one in one commit,
+ # so one snapshot shows a post either whole or pending, never
+ # a lone first file that is neither.
+ pending = (
+ select(PendingAttachmentGroup.group_id)
+ .where(
+ PendingAttachmentGroup.room_id == Message.room_id,
+ PendingAttachmentGroup.group_id
+ == Message.content[ATTACHMENT_GROUP_KEY]["id"].astext,
+ )
+ .exists()
+ )
+ found = (
+ await session.execute(
+ select(Message, pending)
.where(
Message.room_id == room_id,
Message.seq > max(after_seq, 0),
@@ -919,17 +1091,12 @@ class EventBuffer:
.limit(limit + 1)
)
).all()
- rows: list[Message] = []
- for row in fetched[:limit]:
- if (
- row.seq > decided_through.get(room_id, 0)
- and cast(datetime, row.sent_at) >= settled_before
- ):
- break
- rows.append(row)
- if not rows:
+ if not found:
continue
- if len(fetched) > limit and len(rows) == limit:
+ more = len(found) > limit
+ found = found[:limit]
+ rows = [row for row, _ in found]
+ if more:
cut[room_id] = _epoch(cast(datetime, rows[-1].sent_at))
room_events, room_through = await self._conversation_in(
session,
@@ -938,7 +1105,7 @@ class EventBuffer:
my_sender=my_sender,
joined_at=joined[room_id],
agent_id=agent_id,
- group_deadline=group_deadline,
+ pending_rows={row.id for row, held in found if held},
)
events.extend(room_events)
if room_through > after_seq:
@@ -955,10 +1122,12 @@ class EventBuffer:
my_sender: str,
joined_at: Any,
agent_id: str,
- group_deadline: datetime,
+ pending_rows: set[str],
) -> tuple[list[BufferedEvent], int]:
"""Turn one room's rows into conversation events, and return them with
- the `messages.seq` of the last row examined."""
+ the `messages.seq` of the last row examined. `pending_rows` are the
+ ids of the rows whose multi-file post was still pending when the rows
+ were read."""
# For each message key, the highest room row an own event for that message
# was delivered through. A row at or below it is already an own event and is
# skipped. A file of a multi-file message that arrives after the own event
@@ -1004,17 +1173,18 @@ class EventBuffer:
if group is not None:
groups.setdefault(group[0], []).append((group[1], group[2], row))
- # A group that is still incomplete and within its grace time stops the
- # read at its first file, so the whole group is delivered by a later
- # read instead of in pieces.
+ # A group the router has not decided yet, because its files are still
+ # arriving, stops the read at its first file, so the whole group is
+ # delivered by a later read instead of in pieces.
+ undecided = {
+ group_id
+ for group_id, parts in groups.items()
+ if any(row.id in pending_rows for _, _, row in parts)
+ }
held_from: int | None = None
- for parts in groups.values():
- total = parts[0][1]
- lead = min(parts, key=lambda part: part[0])[2]
- complete = len({index for index, _, _ in parts}) >= total
- if not complete and cast(datetime, lead.sent_at) >= group_deadline:
- first = min(row.seq for _, _, row in parts)
- held_from = first if held_from is None else min(held_from, first)
+ for group_id in undecided:
+ first = min(row.seq for _, _, row in groups[group_id])
+ held_from = first if held_from is None else min(held_from, first)
# A group goes out at its last file, so one with a file before the
# hold and its last file at or after it would not go out in this read,
# while the read's place moved past its earlier files. The next read
@@ -1134,8 +1304,8 @@ class EventBuffer:
The event is based on the lowest-numbered file present, which carries
the caption and the message id the sender was given. When files are
- missing, the body says how many arrived, as a live delivery by the
- consumer does.
+ missing, the body says how many arrived, as the own event the router
+ makes of an incomplete post does.
"""
ordered = sorted(parts, key=lambda part: part[0])
total = ordered[0][1]
@@ -1551,32 +1721,6 @@ class EventBuffer:
"""
self._notify.pop(agent_id, None)
- async def drop_room(self, tenant_id: str, agent_id: str, room_id: str) -> None:
- """Delete the agent's own events in one room.
-
- Called when the agent is removed from it. Sequence numbers are untouched
- and no gap is recorded: a reader that skips these has missed nothing it
- was entitled to, and saying otherwise would send it to re-read the
- context of a room it is not in.
- """
- async with tenant_session(self._session_factory, tenant_id) as session:
- result = await session.execute(
- delete(AgentEventRow).where(
- AgentEventRow.agent_id == agent_id,
- AgentEventRow.room_id == room_id,
- )
- )
- await session.commit()
- dropped = result.rowcount or 0 # type: ignore[attr-defined]
- if dropped:
- logger.info(
- "[EVENT-BUF] dropped %s retained event(s) for agent=%s room=%s: "
- "no longer a member",
- dropped,
- agent_id,
- room_id,
- )
-
# ------------------------------------------------------------------
# Retention
# ------------------------------------------------------------------
@@ -1730,11 +1874,11 @@ class EventBuffer:
return events
if conversation:
- # New conversation rows do not set the doorbell, and they are only
- # delivered once they have settled, so a poll that includes
- # conversation wakes after twice the settle time to look again
- # instead of sleeping through the whole timeout.
- await self.wait(agent_id, min(timeout, self._settle_seconds * 2))
+ # New conversation rows do not set the doorbell, so a poll that
+ # includes conversation looks again every
+ # `CONVERSATION_RECHECK_SECONDS` instead of sleeping through the
+ # whole timeout.
+ await self.wait(agent_id, min(timeout, CONVERSATION_RECHECK_SECONDS))
else:
await self.wait(agent_id, timeout)
diff --git a/core/switch_core/bridges/agent/protocol/message_events.py b/core/switch_core/bridges/agent/protocol/message_events.py
index b8c9fc0e9..cc7c5ed9e 100644
--- a/core/switch_core/bridges/agent/protocol/message_events.py
+++ b/core/switch_core/bridges/agent/protocol/message_events.py
@@ -1,8 +1,8 @@
"""Builds the agent event for a room message or join, for every code path.
-A room row reaches an agent in one of two ways. `AgentConsumer` builds an event
-when the row is delivered and is for the agent (an addressed message, a join it
-listens for), and writes it to the agent's own events (`agent_events`), the
+A room row reaches an agent in one of two ways. The router builds an event when
+the row is written and is for the agent (an addressed message, a join it listens
+for), and stores it as one of the agent's own events (`agent_events`), the
events meant for the agent, not ones it sent. `EventBuffer` builds one later
from the `messages` row itself, for a reader that asked for the rest of the
room's conversation too. Both call the functions here, so the agent receives
diff --git a/core/switch_core/bridges/agent/protocol/stream.py b/core/switch_core/bridges/agent/protocol/stream.py
index 03462caf3..66bbec376 100644
--- a/core/switch_core/bridges/agent/protocol/stream.py
+++ b/core/switch_core/bridges/agent/protocol/stream.py
@@ -242,6 +242,10 @@ async def _event_stream(
# `agent_room_reads`. Each pass of the loop calls it only for rooms not in
# this set, so it does not write to the database for every room each time.
counted: set[str] = set()
+ # A new row in a room whose conversation this stream reads rings no
+ # doorbell of the agent's, so the stream asks each such room's own
+ # announcement (`EventBuffer.rooms`) to wake it.
+ rooms = _ConversationBells(buffer)
last_keepalive = time.monotonic()
try:
if conn.reads_conversation:
@@ -348,6 +352,9 @@ async def _event_stream(
conversation_rooms = await _place_in_conversation(
conn, registry, buffer, tenant
)
+ # Before the read, so a row written while it runs wakes the wait
+ # below rather than being left for the keepalive.
+ rooms.watch(conversation_rooms)
try:
pending, looked = await buffer.read(
@@ -477,25 +484,13 @@ async def _event_stream(
):
continue
- # New room messages do not ring the doorbell; only the agent's own
- # events do. A room message newer than the newest of the agent's
- # own events in that room is read only once it is `settle_seconds`
- # old, which gives the agent's consumer time to add it to the
- # agent's own events first if it is for the agent. A stream that
- # receives every room message therefore wakes every
- # `settle_seconds` to read messages that have settled, instead of
- # sleeping for the whole keepalive interval.
- timeout = (
- buffer.settle_seconds
- if conn.reads_conversation
- else KEEPALIVE_INTERVAL_SECONDS
- )
- if await _wait_for_work(bell, conn, timeout):
+ if await _wait_for_work(bell, conn, rooms.moved):
continue
if time.monotonic() - last_keepalive >= KEEPALIVE_INTERVAL_SECONDS:
last_keepalive = time.monotonic()
yield KEEPALIVE
finally:
+ rooms.watch(set())
if unsubscribe_outcomes is not None:
unsubscribe_outcomes()
registry.detach_stream(conn, generation)
@@ -789,26 +784,54 @@ async def _wait_for_wake(conn: AgentConnection) -> bool:
async def _wait_for_work(
- bell: asyncio.Event,
- conn: AgentConnection,
- timeout: float = KEEPALIVE_INTERVAL_SECONDS,
+ bell: asyncio.Event, conn: AgentConnection, moved: asyncio.Event
) -> bool:
- """Wait for a new event or a change to the connection itself.
+ """Wait for a new event, a new row in a room whose conversation the
+ stream reads, or a change to the connection itself.
- Returns False when neither happened within `timeout`, so the caller can
- read again or send a keepalive.
+ Returns False when none happened within the keepalive interval, so the
+ caller can read again or send a keepalive.
"""
waiters = [
asyncio.ensure_future(bell.wait()),
asyncio.ensure_future(conn.wake.wait()),
+ asyncio.ensure_future(moved.wait()),
]
try:
done, _ = await asyncio.wait(
waiters,
- timeout=timeout,
+ timeout=KEEPALIVE_INTERVAL_SECONDS,
return_when=asyncio.FIRST_COMPLETED,
)
return bool(done)
finally:
for waiter in waiters:
waiter.cancel()
+
+
+class _ConversationBells:
+ """The rooms whose new rows wake one stream, kept to the rooms whose
+ conversation it reads.
+
+ `moved` is set when any of them is announced, and cleared each time the
+ rooms are given again, just before the stream reads: an announcement that
+ lands during the read leaves it set, so the stream reads once more rather
+ than sleeping on a row it has not seen. The rows themselves are read from
+ the stream's own position, as everything here is.
+ """
+
+ def __init__(self, buffer: EventBuffer) -> None:
+ self._rooms = buffer.rooms
+ self._watched: set[str] = set()
+ self.moved = asyncio.Event()
+
+ async def _moved(self, _room_id: str) -> None:
+ self.moved.set()
+
+ def watch(self, rooms: set[str]) -> None:
+ self.moved.clear()
+ for room_id in rooms - self._watched:
+ self._rooms.subscribe(room_id, self._moved)
+ for room_id in self._watched - rooms:
+ self._rooms.unsubscribe(room_id, self._moved)
+ self._watched = set(rooms)
diff --git a/core/switch_core/bridges/agent/server_connectors/lifecycle.py b/core/switch_core/bridges/agent/server_connectors/lifecycle.py
index feb5f6df9..2d3678dcd 100644
--- a/core/switch_core/bridges/agent/server_connectors/lifecycle.py
+++ b/core/switch_core/bridges/agent/server_connectors/lifecycle.py
@@ -1,8 +1,23 @@
+"""Every server-side connector, run by the one switch-core process that holds it.
+
+A server-side connector (OpenCode) discovers the agents on its server,
+registers them, and long-polls each one's events, answering through the
+platform. Run in two processes, every agent is polled twice and every turn
+answered twice. So a connector runs in the one process that holds it
+(`ownership.py`, kind `SERVER_CONNECTORS`), and is handed over when that
+process stops or another should hold it. A connector registered or removed
+through one process reaches the others as a `connector.changed` signal
+(`signals.py`): the holder of a removed connector stops it, and a new one is
+claimed by whichever process should hold it.
+"""
+
from __future__ import annotations
+import asyncio
import hashlib
import logging
import secrets
+from typing import TYPE_CHECKING
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -14,15 +29,25 @@ from switch_core.bridges.agent.server_connectors.base import (
from switch_core.bridges.agent.server_connectors.core import ConnectorCore
from switch_core.db.models import ApiKey, ServerConnector
from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.api_key_store import ApiKeyStore
from switch_core.db.stores.server_connector_store import ServerConnectorStore
from switch_core.db.tenant_lookup import all_tenant_ids, tenant_of_server_connector
from switch_core.keys import Keyring
from switch_core.outbound import OutboundPolicy
from switch_core.telemetry import TelemetryService, emit_safely
+from switch_core.tenant_context import no_tenant, tenant_scope
+
+if TYPE_CHECKING:
+ from switch_core.bridges.agent.server_connectors.signals import ConnectorSignals
+ from switch_core.ownership import Fence, Ownership
logger = logging.getLogger(__name__)
+# The kind of owned work a server-side connector is (`ownership.py`), keyed by
+# connector id.
+SERVER_CONNECTORS = "connector"
+
def _connector_kind(connector_type: str) -> str:
"""A connector type as the catalogue spells it.
@@ -48,8 +73,23 @@ class ServerSideConnectorLifecycleService:
telemetry: TelemetryService | None = None,
keyring: Keyring,
outbound_policy: OutboundPolicy,
+ ownership: Ownership,
) -> None:
self._connector_store = connector_store
+ self._agent_store = AgentStore()
+ self._owned = ownership.kind(
+ SERVER_CONNECTORS, start=self._take_over, stop=self._hand_over
+ )
+ self._signals: ConnectorSignals | None = None
+ # Why the last take-over of each connector failed to start it, for
+ # `register` to raise.
+ self._failed_take_overs: dict[str, Exception] = {}
+ # Connectors this process has taken over and is still starting. A
+ # start dials the connector's server and registers each agent it
+ # finds, so it runs in a task of its own: the ownership takes work
+ # over one key at a time, and a server slow to answer would hold up
+ # every other hand-over in this process meanwhile.
+ self._starting: dict[str, asyncio.Task[None]] = {}
self._api_key_store = api_key_store
self._protocol = protocol
self._session_factory = session_factory
@@ -60,11 +100,21 @@ class ServerSideConnectorLifecycleService:
self._connector_registry: dict[str, type[ServerSideConnector]] = {}
self._config_registry: dict[str, type[ServerSideConnectorConfig]] = {}
self._cores: dict[str, ConnectorCore] = {}
- # What this process means to be running. `_cores` alone cannot answer
- # "is anything missing": a connector that failed to start never
- # entered it, and `start_all` logs each failure and steps over it.
+ # What this process means to be running: the connectors it holds.
+ # `_cores` alone cannot answer "is anything missing": a connector that
+ # failed to start never entered it, and the ownership logs each
+ # failure and steps over it.
self._expected: set[str] = set()
+ def use_signals(self, signals: ConnectorSignals) -> None:
+ """Where a connector registered or removed here is announced, so every
+ other process applies it (`signals.py`)."""
+ self._signals = signals
+
+ def _announce(self, connector_id: str) -> None:
+ if self._signals is not None:
+ self._signals.changed(connector_id)
+
def register_connector_type(
self,
type_name: str,
@@ -98,13 +148,12 @@ class ServerSideConnectorLifecycleService:
if record.tenant_id == tenant_id
)
- logger.info("Starting %d server-side connectors", len(connectors))
+ logger.info("Wanting %d server-side connectors", len(connectors))
for record in connectors:
- self._expected.add(record.id)
- try:
- await self.start(record.id)
- except Exception:
- logger.exception("Failed to start connector %s", record.id)
+ self._owned.want(record.id)
+ # The ones this process holds start now, each failure logged and
+ # stepped over; the rest run in the processes that hold them.
+ await self._owned.settle()
async def register(
self,
@@ -162,7 +211,18 @@ class ServerSideConnectorLifecycleService:
{"connector_kind": _connector_kind(connector_type)},
)
- await self.start(record.id)
+ # Started where it is held: here, or in the process that claims it
+ # once it hears of it. A start here is waited for, so its failure
+ # reaches the caller as it did when registering started it directly.
+ self._owned.want(record.id)
+ await self._owned.settle()
+ self._announce(record.id)
+ starting = self._starting.get(record.id)
+ if starting is not None:
+ await asyncio.shield(starting)
+ failed = self._failed_take_overs.pop(record.id, None)
+ if failed is not None:
+ raise failed
logger.info(
"Registered server-side connector %s (%s): %s (owner: %s)",
@@ -219,9 +279,14 @@ class ServerSideConnectorLifecycleService:
registration_token=registration_token,
protocol=self._protocol,
)
- await core.start()
+ try:
+ await core.start()
+ except BaseException:
+ # Agents it registered before failing, or before being cancelled
+ # by a hand-over, are already being polled.
+ await core.stop()
+ raise
self._cores[connector_id] = core
- self._expected.add(connector_id)
async def stop(self, connector_id: str) -> None:
self._expected.discard(connector_id)
@@ -243,12 +308,18 @@ class ServerSideConnectorLifecycleService:
return sum(1 for connector_id in self._expected if connector_id in self._cores)
async def stop_all(self) -> None:
+ starting = list(self._starting.values())
+ self._starting.clear()
+ for task in starting:
+ task.cancel()
+ await asyncio.gather(*starting, return_exceptions=True)
logger.info("Stopping all %d server-side connectors", len(self._cores))
for connector_id in list(self._cores):
await self.stop(connector_id)
async def remove(self, connector_id: str) -> None:
- """Delete a connector: its agents, its running core, and its row.
+ """Delete a connector: its agents, its running core, and its row, and
+ tell the other processes, so the one holding it stops it.
Scoped to the caller, and checked before anything is torn down. The
session inherits the tenant the request bound (`gateway/auth.py`), so
@@ -272,12 +343,18 @@ class ServerSideConnectorLifecycleService:
core = self._cores.get(connector_id)
if core is not None:
await core.delete_agents()
+ else:
+ # Held by another process: its agents are found by their rows.
+ await self._delete_agents(connector_id, existing.tenant_id)
+ # Held here or not, let go of it: no process is to take it over.
+ await self._owned.drop(connector_id)
await self.stop(connector_id)
async with self._session_factory() as session:
await self._connector_store.delete(session, connector_id)
await session.commit()
+ self._announce(connector_id)
emit_safely(
self._telemetry,
@@ -298,8 +375,129 @@ class ServerSideConnectorLifecycleService:
def get_connector_ids(self) -> list[str]:
return list(self._cores.keys())
- def get_agent_names(self, connector_id: str) -> list[str]:
- core = self._cores.get(connector_id)
- if core is None:
- return []
- return core.get_agent_names()
+ async def get_agent_names(self, connector_id: str) -> list[str]:
+ """The names of the agents the connector registered, read from their
+ rows in the bound tenant: the same answer from every process, whichever
+ holds the connector."""
+ async with self._session_factory() as session:
+ agents = await self._agent_store.get_by_server_connector(
+ session, connector_id
+ )
+ return [agent.name for agent in agents]
+
+ async def _delete_agents(self, connector_id: str, tenant_id: str) -> None:
+ with tenant_scope(tenant_id):
+ async with self._session_factory() as session:
+ agents = await self._agent_store.get_by_server_connector(
+ session, connector_id
+ )
+ for agent in agents:
+ try:
+ await self._protocol.delete_agent(agent_id=agent.id)
+ except Exception:
+ logger.exception(
+ "Failed to delete agent %s during connector removal",
+ agent.id,
+ )
+
+ # ── Ownership ────────────────────────────────────────────────────────────
+
+ async def _take_over(self, connector_id: str, fence: Fence) -> None:
+ """This process now holds the connector: start it here, in the
+ background (`_starting`)."""
+ logger.info("Taking over server-side connector %s", connector_id)
+ self._failed_take_overs.pop(connector_id, None)
+ self._expected.add(connector_id)
+ self._starting[connector_id] = asyncio.create_task(
+ self._start_held(connector_id)
+ )
+
+ async def _start_held(self, connector_id: str) -> None:
+ try:
+ # Whoever's request led to the take-over, the connector acts as
+ # its own row's tenant (`start`).
+ with no_tenant():
+ await self.start(connector_id)
+ except asyncio.CancelledError:
+ raise
+ except Exception as exc:
+ self._failed_take_overs[connector_id] = exc
+ logger.exception("Failed to start connector %s", connector_id)
+ finally:
+ if self._starting.get(connector_id) is asyncio.current_task():
+ del self._starting[connector_id]
+
+ async def _hand_over(self, connector_id: str) -> None:
+ """This process no longer holds the connector: stop it here, starting
+ or started."""
+ logger.info("Handing server-side connector %s over", connector_id)
+ starting = self._starting.pop(connector_id, None)
+ if starting is not None and not starting.done():
+ starting.cancel()
+ await asyncio.gather(starting, return_exceptions=True)
+ await self.stop(connector_id)
+
+ def holds(self, connector_id: str) -> bool:
+ """Whether this process holds the connector and runs it."""
+ return self._owned.fence(connector_id) is not None
+
+ # ── Changes made in other processes ──────────────────────────────────────
+
+ async def reconcile_connector(self, connector_id: str) -> None:
+ """Bring this process in line with one connector's row, after another
+ process registered or removed it."""
+ tenant_id = await tenant_of_server_connector(
+ self._session_factory, connector_id
+ )
+ record: ServerConnector | None = None
+ if tenant_id is not None:
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ record = await self._connector_store.get(session, connector_id)
+ await self._reconcile_row(connector_id, record)
+
+ async def reconcile_all(self) -> None:
+ """`reconcile_connector` for every connector stored or wanted here,
+ reading the rows one tenant at a time."""
+ rows: dict[str, ServerConnector] = {}
+ for tenant_id in await all_tenant_ids(self._session_factory):
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ rows.update(
+ (record.id, record)
+ for record in await self._connector_store.get_active(session)
+ if record.tenant_id == tenant_id
+ )
+ for connector_id in sorted(set(rows) | self._owned.wanted() | set(self._cores)):
+ try:
+ await self._reconcile_row(connector_id, rows.get(connector_id))
+ except Exception:
+ logger.warning(
+ "Could not bring server-side connector %s in line with its "
+ "row; retrying at the next re-check",
+ connector_id,
+ exc_info=True,
+ )
+
+ async def _reconcile_row(
+ self, connector_id: str, record: ServerConnector | None
+ ) -> None:
+ """Wanted where it is new here, so the process that should hold it
+ claims it; stopped and let go of where it is gone or no longer
+ active."""
+ if self._owned.stopping:
+ return
+ if record is None or record.status != "active":
+ if self._owned.wants(connector_id) or connector_id in self._cores:
+ logger.info(
+ "Server-side connector %s was removed elsewhere; stopping it",
+ connector_id,
+ )
+ await self._owned.drop(connector_id)
+ await self.stop(connector_id)
+ return
+ if not self._owned.wants(connector_id):
+ logger.info(
+ "Server-side connector %s was added elsewhere; wanting it",
+ connector_id,
+ )
+ self._owned.want(connector_id)
+ await self._owned.settle()
diff --git a/core/switch_core/bridges/agent/server_connectors/signals.py b/core/switch_core/bridges/agent/server_connectors/signals.py
new file mode 100644
index 000000000..bcc709e5d
--- /dev/null
+++ b/core/switch_core/bridges/agent/server_connectors/signals.py
@@ -0,0 +1,48 @@
+"""Server-side connectors registered or removed in one process, reaching every other.
+
+A server-side connector runs in the one switch-core process that holds it
+(`lifecycle.py`). Registering or removing one is served by whichever process
+the request reached, and changed only that process: a connector removed there
+went on polling its agents in the holder, and one registered there was run
+nowhere else, so another process that should hold it never claimed it.
+
+So a registration or a removal, once committed, announces `connector.changed`
+naming the connector. Every other process re-reads the connector's row
+(`ServerSideConnectorLifecycleService.reconcile_connector`): wanting a new one,
+so the process that should hold it claims it, or stopping and letting go of
+one that is gone. A re-read that fails is kept and tried again, and every
+connector is re-read when the signal connection comes back and on each
+periodic re-check.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from switch_core.bridges.agent.server_connectors.lifecycle import (
+ ServerSideConnectorLifecycleService,
+ )
+ from switch_core.signals import SignalBus
+
+CONNECTOR_CHANGED_SIGNAL = "connector.changed"
+
+
+class ConnectorSignals:
+ def __init__(
+ self, *, lifecycle: ServerSideConnectorLifecycleService, bus: SignalBus
+ ) -> None:
+ self._bus = bus
+ self._owed = bus.pending_work("connectors", lifecycle.reconcile_connector)
+ bus.handle(CONNECTOR_CHANGED_SIGNAL, self._on_changed)
+ bus.on_resync(lifecycle.reconcile_all)
+ bus.on_recheck(lifecycle.reconcile_all)
+ lifecycle.use_signals(self)
+
+ def changed(self, connector_id: str) -> None:
+ """The connector's row changed and has committed: every other process
+ re-reads it. This one has already applied the change itself."""
+ self._bus.publish(CONNECTOR_CHANGED_SIGNAL, connector_id=connector_id)
+
+ def _on_changed(self, signal: dict[str, Any]) -> None:
+ self._owed.add(signal["connector_id"])
diff --git a/core/switch_core/bridges/collaboration/adapter.py b/core/switch_core/bridges/collaboration/adapter.py
index 381d1e673..ab6a2bb46 100644
--- a/core/switch_core/bridges/collaboration/adapter.py
+++ b/core/switch_core/bridges/collaboration/adapter.py
@@ -532,7 +532,19 @@ class PlatformAdapter(ABC):
#: withheld is the notice, not the request.
discloses_unconfirmed_posts: ClassVar[bool] = False
+ # Whether this adapter keeps quiet when told not to listen
+ # (`set_listening`): it opens nothing the platform delivers events over,
+ # no socket, no poll, no subscription it keeps alive, and only calls the
+ # platform's API. A bridge is listened to by the one switch-core process
+ # that holds it (`ownership.py`); every other runs it quietly to serve the
+ # requests and webhooks that reach it. An adapter that cannot keep quiet
+ # listens wherever it runs, so its platform delivers to every process.
+ keeps_quiet_unless_listening: ClassVar[bool] = False
+
def __init__(self) -> None:
+ # Whether this process holds the bridge: see
+ # `keeps_quiet_unless_listening`. Set by the lifecycle before `start`.
+ self._listening = True
self._on_message: Callable[[InboundMessage], Awaitable[None]] | None = None
self._on_command: Callable[[InboundCommand], Awaitable[None]] | None = None
self._on_agent_joined: Callable[[InboundAgentJoin], Awaitable[None]] | None = (
@@ -587,6 +599,12 @@ class PlatformAdapter(ABC):
def set_max_attachment_bytes(self, max_bytes: int) -> None:
self._max_attachment_bytes = max_bytes
+ def set_listening(self, listening: bool) -> None:
+ """Whether this process holds the bridge and listens to the platform
+ for it, or runs it quietly. Before `start`; an adapter that cannot
+ keep quiet ignores it (`keeps_quiet_unless_listening`)."""
+ self._listening = listening
+
def set_third_party_avatars(self, enabled: bool) -> None:
"""Whether this adapter may hand platforms an icon drawn by an outside
avatar service (`agent_icon.is_third_party_avatar`)."""
diff --git a/core/switch_core/bridges/collaboration/bridge_signals.py b/core/switch_core/bridges/collaboration/bridge_signals.py
new file mode 100644
index 000000000..7f10aaba9
--- /dev/null
+++ b/core/switch_core/bridges/collaboration/bridge_signals.py
@@ -0,0 +1,70 @@
+"""Bridges registered, edited or removed in one process, reaching every other.
+
+Every switch-core process runs its own copy of each collaboration bridge:
+listening in the one that holds it, quiet in the rest
+(`lifecycle_service.py`). A bridge registered, edited or removed through one
+process changed only that process's copy, so the holder elsewhere kept an old
+config, a removed bridge kept its socket and kept relaying, and a new one ran
+nowhere else until a restart.
+
+So a registration, an edit or a removal, once committed, announces
+`bridge.changed` naming the bridge. Every other process re-reads the bridge's
+row and brings its copy in line (`CollaborationBridgeLifecycleService.
+reconcile_bridge`): starting it, restarting it with the new config, or
+stopping it and letting go of its lease. A re-read that fails is kept and
+tried again, and every bridge is re-read when the signal connection comes back
+and on each periodic re-check, so a missed signal costs a delay rather than a
+stale bridge until a restart.
+
+A room bridged to a channel, or unbridged from one, is announced the same way,
+as `bridge.rooms_changed`: the processes running the bridge map its rooms
+again (`reload_rooms`), so the one listening to the platform stops carrying
+the channel into a room that no longer has it. A room bridged elsewhere is
+also found on the spot when a message reaches a process that has not mapped
+it yet (`CollaborationCore._channel_for_room`).
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from switch_core.bridges.collaboration.lifecycle_service import (
+ CollaborationBridgeLifecycleService,
+ )
+ from switch_core.signals import SignalBus
+
+BRIDGE_CHANGED_SIGNAL = "bridge.changed"
+BRIDGE_ROOMS_CHANGED_SIGNAL = "bridge.rooms_changed"
+
+
+class BridgeSignals:
+ def __init__(
+ self, *, lifecycle: CollaborationBridgeLifecycleService, bus: SignalBus
+ ) -> None:
+ self._bus = bus
+ self._owed = bus.pending_work("bridges", lifecycle.reconcile_bridge)
+ self._rooms_owed = bus.pending_work("bridge-rooms", lifecycle.reload_rooms)
+ bus.handle(BRIDGE_CHANGED_SIGNAL, self._on_changed)
+ bus.handle(BRIDGE_ROOMS_CHANGED_SIGNAL, self._on_rooms_changed)
+ bus.on_resync(lifecycle.reconcile_all)
+ bus.on_recheck(lifecycle.reconcile_all)
+ bus.on_resync(lifecycle.reload_all_rooms)
+ bus.on_recheck(lifecycle.reload_all_rooms)
+ lifecycle.use_signals(self)
+
+ def changed(self, bridge_id: str) -> None:
+ """The bridge's row changed and has committed: every other process
+ re-reads it. This one has already applied the change itself."""
+ self._bus.publish(BRIDGE_CHANGED_SIGNAL, bridge_id=bridge_id)
+
+ def rooms_changed(self, bridge_id: str) -> None:
+ """A room was bridged to the bridge or unbridged from it, and that has
+ committed: every other process maps the bridge's rooms again."""
+ self._bus.publish(BRIDGE_ROOMS_CHANGED_SIGNAL, bridge_id=bridge_id)
+
+ def _on_changed(self, signal: dict[str, Any]) -> None:
+ self._owed.add(signal["bridge_id"])
+
+ def _on_rooms_changed(self, signal: dict[str, Any]) -> None:
+ self._rooms_owed.add(signal["bridge_id"])
diff --git a/core/switch_core/bridges/collaboration/collaboration_core.py b/core/switch_core/bridges/collaboration/collaboration_core.py
index dc1249e35..26de36e3c 100644
--- a/core/switch_core/bridges/collaboration/collaboration_core.py
+++ b/core/switch_core/bridges/collaboration/collaboration_core.py
@@ -1,6 +1,8 @@
from __future__ import annotations
import asyncio
+import contextlib
+import hashlib
import logging
import re
import time
@@ -8,6 +10,7 @@ import uuid
from collections.abc import Awaitable, Callable, Iterator, Mapping
from contextlib import contextmanager
from dataclasses import dataclass, field
+from datetime import timedelta
from typing import TYPE_CHECKING, TypeVar
from sqlalchemy.exc import IntegrityError
@@ -41,7 +44,7 @@ from switch_core.clients.admin_messages import (
platform_on_behalf_of,
)
from switch_core.clients.mentions import mention_regex, strip_emphasis
-from switch_core.db.models import BridgeMessageMap, ExternalUser
+from switch_core.db.models import BridgeMessageMap, Client, ExternalUser
from switch_core.db.session_scope import tenant_session
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.bridge_message_map_store import BridgeMessageMapStore
@@ -57,6 +60,7 @@ from switch_core.observability.catalogue import (
BRIDGE_EVENTS_OUT,
)
from switch_core.observability.metrics import metrics
+from switch_core.process_lease import LEASE_TTL, RENEW_EVERY_SECONDS
from switch_core.provisioning import Provisioning
from switch_core.room_service import RoomCreateConfig
from switch_core.room_wide_mention import is_room_wide_mention
@@ -81,6 +85,7 @@ from switch_core.transport import (
from switch_core.transport import (
RoomRef,
TransportError,
+ writes_derived_from,
)
from switch_core.trust.client import (
NullTrustClient,
@@ -120,6 +125,22 @@ HUMAN_JOIN_TIMEOUT = 30.0
# parts that arrived, flagged as incomplete (see _schedule_outbound_group_flush).
OUTBOUND_GROUP_TIMEOUT_SECONDS = 5.0
+# How old a claim on a room message going out (`_claim_outbound`) must be,
+# still with no post, before it is taken to be from a holder that died
+# between claiming and posting, and claimed again. Under the soonest the next
+# holder can read the message after such a death: the process lease lapsing
+# (`LEASE_TTL` after its last renewal, which came at most `RENEW_EVERY_SECONDS`
+# before the claim). Only one process holds the bridge at a time, so no live
+# post in flight is ever read by another holder this soon.
+_UNPOSTED_CLAIM_LAPSES_AFTER = timedelta(
+ seconds=LEASE_TTL.total_seconds() - 2 * RENEW_EVERY_SECONDS
+)
+
+# How long to wait for the person behind a human actor's client row that
+# another process has just written, before giving up on telling whose it is.
+# That process writes the `ExternalUser` right after the client.
+_HUMAN_ACTOR_MINTING_SECONDS = 2.0
+
def _is_thread_reply(content: Mapping[str, object]) -> bool:
relates = content.get("m.relates_to")
@@ -131,9 +152,16 @@ class _PendingOutboundGroup:
"""Files of a multi-attachment message seen so far, keyed by group index."""
total: int
+ # The room the group is in (its transport id) and the lowest `messages.seq`
+ # among its files so far: the bridge's progress in the room is held back
+ # to before it until the group is posted (`held_from`), so whoever holds
+ # the bridge next posts it if this process stops first.
+ room_id: str
+ from_seq: int
parts: dict[int, OutboundAttachment] = field(default_factory=dict)
+ # The room event of each file, by group index.
+ event_ids: dict[int, str] = field(default_factory=dict)
caption: str | None = None
- first_event_id: str | None = None
_LOBBY_DEPRECATION_NOTICE = (
@@ -162,10 +190,25 @@ def _no_agents_notice(slash_hint: str | None) -> str:
)
+def _post_source(bridge_id: str, channel_id: str, message_ref: str) -> str:
+ """What a platform post's room rows are numbered from: the post's own
+ reference on its channel of this bridge, the same each time the platform
+ delivers it.
+
+ A platform may deliver one post twice. Slack sends a Socket Mode event
+ again when its acknowledgement did not arrive, as when the process holding
+ the bridge stopped between storing the post and acknowledging it, and a
+ webhook platform retries a delivery it saw fail. The second delivery finds
+ the rows written and writes nothing new."""
+ return f"{bridge_id}\x1f{channel_id}\x1f{message_ref}"
+
+
class CollaborationCore:
- # Tests assemble a CollaborationCore with `__new__`; these read as "not wired".
+ # Tests assemble a CollaborationCore with `__new__`; these read as "not wired",
+ # and the bridge as held here, as it is in a deployment of one process.
_activity_publisher: AgentSessionActivityPublisher | None = None
_approval_answers: ApprovalAnswers | None = None
+ _active: bool = True
def __init__(
self,
@@ -190,10 +233,16 @@ class CollaborationCore:
session_activity_listener: AgentSessionActivityListener,
session_activity_service: AgentSessionActivityService,
connections: AgentConnectionRegistry,
+ active: bool,
trust_client: TrustClient = NullTrustClient(),
gateway_public_url: str | None = None,
) -> None:
self._bridge_id = bridge_id
+ # Whether this process holds the bridge (`ownership.py`): it listens to
+ # the platform, relays the rooms out to it and draws session activity
+ # there. Every other process runs the bridge too, quietly, to serve
+ # the requests and webhooks that reach it.
+ self._active = active
# The tenant of this bridge's own row. Read once here because it is
# immutable, but never *bound* for the life of anything: each unit of
# work binds it at the point it acts, and room-scoped work binds the
@@ -234,7 +283,11 @@ class CollaborationCore:
# message. One-way, so a cached answer cannot go stale — see
# _repair_placeholder_username.
self._names_known_good: set[str] = set()
+ # Senders known to be people on this bridge, and senders known not to
+ # be: a person first seen by another process is found in the database
+ # once (`_is_human_sender`), and an agent is asked about once.
self._human_user_ids: set[str] = set()
+ self._not_human_user_ids: set[str] = set()
self._channel_locks: dict[str, asyncio.Lock] = {}
self._human_actor_locks: dict[str, asyncio.Lock] = {}
# Channels Switch is itself provisioning right now (outbound room
@@ -249,6 +302,10 @@ class CollaborationCore:
# never completes cannot leak.
self._outbound_groups: dict[str, _PendingOutboundGroup] = {}
self._outbound_group_timers: dict[str, asyncio.TimerHandle] = {}
+ # Incomplete groups whose timeout is posting what arrived. Out of
+ # `_outbound_groups`, so a late file of one starts a group of its own,
+ # and still holding the room's progress back until posted.
+ self._flushing_outbound_groups: list[_PendingOutboundGroup] = []
# Room-event -> external-post anchors written synchronously right after
# room_send, before the durable _record_message_map commit, so a fast
# command reply relayed during that DB await still resolves the command's
@@ -257,6 +314,11 @@ class CollaborationCore:
self._pending_message_maps: dict[str, str] = {}
# Identity provisioning runs in the background — see _create_agent_identities.
self._identity_task: asyncio.Task[None] | None = None
+ # Agents given an identity by this copy of the bridge, so one deleted
+ # by another process is let go of here too; and whether the agents
+ # changed while a provisioning run was going.
+ self._provisioned_agents: set[str] = set()
+ self._identities_stale = False
self._channel_type_refresh_task: asyncio.Task[None] | None = None
# Turns and request cards from the tables the host reports to, pushed
# as they change. Only where the platform draws session activity.
@@ -457,34 +519,56 @@ class CollaborationCore:
),
on_app_joined=self._traced("app_joined", self._handle_app_joined_channel),
)
- await self._ensure_channel_captures()
- if self._activity_publisher is not None:
- self._activity_publisher.start()
+ if self._active:
+ await self._ensure_channel_captures()
+ if self._activity_publisher is not None:
+ self._activity_publisher.start()
# Deliberately not awaited. Provisioning is one call per agent against
# the platform, and a rate-limited platform makes that minutes of
# mostly waiting — which would hold up the bridge coming online, and
# with it every other bridge behind it at startup and any request that
# restarts one. Messages do not depend on it: an agent is addressable
- # by name whether or not its platform identity exists yet.
+ # by name whether or not its platform identity exists yet. Every
+ # process provisions, the adapter deciding how much a quiet one needs.
self._identity_task = asyncio.create_task(self._run_agent_identities())
- # Not awaited either: one platform read per channel, and nothing waits on it.
- self._channel_type_refresh_task = asyncio.create_task(
- self._run_channel_type_refresh()
- )
+ # Not awaited either: one platform read per channel, and nothing waits
+ # on it. Only where the bridge is held: one process is enough.
+ if self._active:
+ self._channel_type_refresh_task = asyncio.create_task(
+ self._run_channel_type_refresh()
+ )
+
+ @property
+ def active(self) -> bool:
+ """Whether this process holds the bridge and listens for it."""
+ return self._active
+
+ def agents_changed(self) -> None:
+ """Agents were registered or deleted in this bridge's tenant, maybe by
+ another process: give the new ones their platform identities here and
+ let go of the deleted ones'. A run already going reads the agents
+ again once it is done, in case it read them before the change."""
+ self._identities_stale = True
+ self._provision_identities_on_attach()
async def stop(self) -> None:
- if self._activity_publisher is not None:
- await self._activity_publisher.stop()
- if self._identity_task and not self._identity_task.done():
- self._identity_task.cancel()
- self._identity_task = None
- if (
- self._channel_type_refresh_task
- and not self._channel_type_refresh_task.done()
- ):
- self._channel_type_refresh_task.cancel()
- self._channel_type_refresh_task = None
- await self._adapter.stop()
+ try:
+ if self._activity_publisher is not None:
+ await self._activity_publisher.stop()
+ finally:
+ # Closed whatever the publisher raised: the bridge's lease is let
+ # go of once this returns or raises, and the next holder's
+ # platform connection must not run beside this one.
+ if self._identity_task and not self._identity_task.done():
+ self._identity_task.cancel()
+ self._identity_task = None
+ if (
+ self._channel_type_refresh_task
+ and not self._channel_type_refresh_task.done()
+ ):
+ self._channel_type_refresh_task.cancel()
+ self._channel_type_refresh_task = None
+ await self._adapter.stop()
# ── Startup loading ──────────────────────────────────────────────────────
@@ -597,7 +681,11 @@ class CollaborationCore:
async def _run_agent_identities_unbound(self) -> None:
try:
- await self._create_agent_identities()
+ while True:
+ self._identities_stale = False
+ await self._create_agent_identities()
+ if not self._identities_stale:
+ break
except asyncio.CancelledError:
logger.info(
"%s identity provisioning cancelled before finishing",
@@ -615,10 +703,24 @@ class CollaborationCore:
) as session:
agents = await self._agent_store.get_all(session)
+ # Deleted since the last run here, wherever the deletion was served.
+ current = {agent.name for agent in agents}
+ for name in sorted(self._provisioned_agents - current):
+ try:
+ await self._adapter.remove_agent_identity(name)
+ except Exception:
+ logger.exception(
+ "Failed to remove the %s identity of deleted agent %s",
+ self._bridge_type,
+ name,
+ )
+ self._provisioned_agents.discard(name)
+
failed = 0
for agent in agents:
try:
await self._adapter.create_agent_identity(agent.name, agent.description)
+ self._provisioned_agents.add(agent.name)
except Exception:
failed += 1
logger.exception(
@@ -988,13 +1090,17 @@ class CollaborationCore:
annotation = trust_annotation(check)
if annotation is not None:
content = f"{content}\n\n{annotation}"
- event_id = await human_actor.send_message(
- transport_room_id,
- content,
- format="markdown",
- thread_root_id=thread_root_id,
- metered=True,
- )
+ with writes_derived_from(
+ human_actor.client_id,
+ _post_source(self._bridge_id, msg.channel_id, msg.message_ref),
+ ):
+ event_id = await human_actor.send_message(
+ transport_room_id,
+ content,
+ format="markdown",
+ thread_root_id=thread_root_id,
+ metered=True,
+ )
if event_id is None:
logger.error(
"[BRIDGE-IN] failed to relay message from %s into room %s — "
@@ -1042,7 +1148,11 @@ class CollaborationCore:
# A platform post hands us all its files at once, so the group is known
# up front — no waiting on the receiving side to learn how many to
# expect. The room carries them as `total` events sharing this id.
- group_id = str(uuid.uuid4()) if total > 1 else None
+ # Derived from the post, as the files' rows are: a second delivery of
+ # a post whose first stopped partway writes the files still missing
+ # into the same group, which then completes.
+ source = _post_source(self._bridge_id, msg.channel_id, msg.message_ref)
+ group_id = str(uuid.uuid5(uuid.NAMESPACE_URL, source)) if total > 1 else None
for index, attachment in enumerate(msg.attachments):
media_uri = await human_actor.upload_media(
attachment.data, attachment.mimetype, attachment.filename
@@ -1050,22 +1160,25 @@ class CollaborationCore:
msgtype = (
"m.image" if attachment.mimetype.startswith("image/") else "m.file"
)
- event_id = await human_actor.send_media(
- transport_room_id,
- media_uri,
- attachment.filename,
- attachment.mimetype,
- len(attachment.data),
- msgtype=msgtype,
- caption=caption if index == 0 else None,
- thread_root_id=thread_root_id,
- group=(
- {"id": group_id, "index": index, "total": total}
- if group_id is not None
- else None
- ),
- metered=True,
- )
+ # Numbered from the post and the file's place in it, as the text
+ # above is, so a second delivery of the post writes no file twice.
+ with writes_derived_from(human_actor.client_id, f"{source}\x1f{index}"):
+ event_id = await human_actor.send_media(
+ transport_room_id,
+ media_uri,
+ attachment.filename,
+ attachment.mimetype,
+ len(attachment.data),
+ msgtype=msgtype,
+ caption=caption if index == 0 else None,
+ thread_root_id=thread_root_id,
+ group=(
+ {"id": group_id, "index": index, "total": total}
+ if group_id is not None
+ else None
+ ),
+ metered=True,
+ )
if event_id is None:
logger.error(
"[BRIDGE-IN] failed to relay attachment %s from %s into room %s",
@@ -1157,10 +1270,22 @@ class CollaborationCore:
"event_id": existing_root,
}
- try:
- event_id = await human_actor.send_event(
- transport_room_id, "com.switch.command", content
+ # A command post the platform delivers again (`_post_source`) is
+ # written, and so run, once. One without a post id cannot be told
+ # apart from another, and is written each time.
+ derived_from = (
+ contextlib.nullcontext()
+ if cmd.message_ref is None
+ else writes_derived_from(
+ human_actor.client_id,
+ _post_source(self._bridge_id, cmd.channel_id, cmd.message_ref),
)
+ )
+ try:
+ with derived_from:
+ event_id = await human_actor.send_event(
+ transport_room_id, "com.switch.command", content
+ )
except TransportError as exc:
logger.error(
"Failed to bridge command %s into %s: %s",
@@ -1926,6 +2051,12 @@ class CollaborationCore:
existing = self._human_actors.get(external_user_id)
if existing is not None:
return existing
+ # Another switch-core process may have minted this person already:
+ # each process learns its bridge's people when the bridge starts, and
+ # someone who first spoke since then is known only in the database.
+ adopted = await self._adopt_human_actor(external_user_id)
+ if adopted is not None:
+ return adopted
# Defence in depth against agent/user misdetection: a bridged agent
# must never get a human actor as if it were an external user. If the
@@ -1946,12 +2077,38 @@ class CollaborationCore:
# type (e.g. two Slack workspaces) must map to distinct clients,
# since transport_user_id is globally unique but usernames are not.
localpart = f"switch-{self._bridge_type}-{self._bridge_id}-{sanitized}"
+ # Two people whose names sanitize alike (`ann.marie` and `ann-marie`)
+ # would share that localpart: the one who comes second gets it with
+ # their platform id's digest appended, the same in every process.
+ digest = hashlib.blake2b(external_user_id.encode(), digest_size=4).hexdigest()
+ candidates = (localpart, f"{localpart}-{digest}")
- client = await self._client_lifecycle.create_and_start(
- client_type="user",
- display_name=external_username,
- localpart=localpart,
- )
+ for localpart in candidates:
+ try:
+ client = await self._client_lifecycle.create_and_start(
+ client_type="user",
+ display_name=external_username,
+ localpart=localpart,
+ )
+ except IntegrityError:
+ # Taken: by this person, minted by another process between the
+ # read above and here, or by someone else whose name
+ # sanitizes the same.
+ adopted = await self._adopt_human_actor_client(
+ external_user_id,
+ external_username,
+ f"@{localpart}:{self._id_server_name}",
+ )
+ if adopted is not None:
+ return adopted
+ continue
+ break
+ else:
+ raise RuntimeError(
+ f"No client localpart is free for external user "
+ f"{external_user_id} ({external_username!r}) on bridge "
+ f"{self._bridge_id}"
+ )
ext_user = ExternalUser(
bridge_id=self._bridge_id,
@@ -1975,6 +2132,137 @@ class CollaborationCore:
)
return client.client_id
+ async def _adopt_human_actor(self, external_user_id: str) -> str | None:
+ """The human actor another process minted for this person, started
+ here and remembered, or None when there is none yet."""
+ async with self._session_factory() as session:
+ known = await self._external_user_store.get_by_external_id(
+ session, self._bridge_id, external_user_id
+ )
+ record = (
+ None
+ if known is None
+ else await self._client_store.get(session, known.client_id)
+ )
+ if record is None or known is None:
+ return None
+ self._run_human_actor(external_user_id, known.external_username, record)
+ return record.id
+
+ async def _adopt_human_actor_client(
+ self, external_user_id: str, external_username: str, transport_user_id: str
+ ) -> str | None:
+ """The human actor whose client row was written first, when it is this
+ person's; None when it stands for someone else."""
+ deadline = time.monotonic() + _HUMAN_ACTOR_MINTING_SECONDS
+ while True:
+ async with self._session_factory() as session:
+ record = await self._client_store.get_by_transport_user_id(
+ session, transport_user_id
+ )
+ person = (
+ None
+ if record is None
+ else await self._external_user_store.get_by_client_id(
+ session, record.id
+ )
+ )
+ if record is None:
+ raise RuntimeError(
+ f"Creating the human actor {transport_user_id} conflicted "
+ "with another, which cannot be read back"
+ )
+ if person is not None:
+ break
+ # Its process writes whose it is right after the client row.
+ if time.monotonic() >= deadline:
+ raise RuntimeError(
+ f"The human actor {transport_user_id} has no external user "
+ f"after {_HUMAN_ACTOR_MINTING_SECONDS:.0f}s, so whose it is "
+ "cannot be told"
+ )
+ await asyncio.sleep(0.1)
+ if (
+ person.bridge_id != self._bridge_id
+ or person.external_user_id != external_user_id
+ ):
+ return None
+ self._run_human_actor(external_user_id, external_username, record)
+ return record.id
+
+ def _run_human_actor(
+ self, external_user_id: str, external_username: str, record: Client
+ ) -> None:
+ client = self._client_lifecycle.get(record.id)
+ if client is None:
+ client = self._client_lifecycle.start_client(record)
+ self._human_actors[external_user_id] = record.id
+ self._human_user_ids.add(client.transport_user_id)
+ self._adapter.prime_mention_targets({external_username: external_user_id})
+ logger.info(
+ "Adopted human actor %s for external user %s on bridge %s, minted "
+ "by another process",
+ client.transport_user_id,
+ external_user_id,
+ self._bridge_id,
+ )
+
+ async def _channel_for_room(self, transport_room_id: str) -> str | None:
+ """The channel a room is bridged to, from memory or, on a miss, from
+ the room's row. A room bridged through another switch-core process
+ (created from the Console there, or adopted from a channel that
+ process heard of first) is mapped only there until this one reads it,
+ and this process may be the one relaying the room out."""
+ channel_id = self._find_channel(transport_room_id=transport_room_id)
+ if channel_id is not None:
+ return channel_id
+ async with self._session_factory() as session:
+ room = await self._room_store.get_by_transport_room_id(
+ session, transport_room_id
+ )
+ if (
+ room is None
+ or room.bridge_id != self._bridge_id
+ or not room.external_channel_id
+ ):
+ return None
+ self.add_room_mapping(
+ room.id, room.transport_room_id, room.external_channel_id, room.tenant_id
+ )
+ return room.external_channel_id
+
+ async def reload_channel_map(self) -> None:
+ """Map this bridge's rooms as their rows say now, after another
+ process bridged, unbridged or moved one. A mapping made here while
+ the rows were being read is newer than the read, and is kept."""
+ before = dict(self._room_to_channel)
+ async with tenant_session(
+ self._session_factory, self._bridge_tenant_id
+ ) as session:
+ rooms = await self._room_store.get_by_bridge(session, self._bridge_id)
+ stored = {
+ (room.id, room.transport_room_id): room
+ for room in rooms
+ if room.external_channel_id
+ }
+ for key, channel_id in before.items():
+ if key not in stored and self._room_to_channel.get(key) == channel_id:
+ logger.info(
+ "Room %s is no longer bridged to channel %s; unmapping it",
+ key[0],
+ channel_id,
+ )
+ self.remove_room_mapping(*key)
+ for key, room in stored.items():
+ if self._room_to_channel.get(key) != room.external_channel_id:
+ assert room.external_channel_id is not None
+ self.add_room_mapping(
+ room.id,
+ room.transport_room_id,
+ room.external_channel_id,
+ room.tenant_id,
+ )
+
def _find_channel(
self, room_id: str | None = None, transport_room_id: str | None = None
) -> str | None:
@@ -1996,16 +2284,16 @@ class CollaborationCore:
room.room_id,
event.body[:80] if event.body else "",
)
- if event.sender in self._human_user_ids:
+ if event.sender == self._workspace_consumer_transport_user_id:
+ logger.debug("[BRIDGE-OUT] skipping bridge client message")
+ return
+ if await self._is_human_sender(event.sender):
logger.debug(
"[BRIDGE-OUT] skipping human actor message from %s", event.sender
)
return
- if event.sender == self._workspace_consumer_transport_user_id:
- logger.debug("[BRIDGE-OUT] skipping bridge client message")
- return
- channel_id = self._find_channel(transport_room_id=room.room_id)
+ channel_id = await self._channel_for_room(room.room_id)
if channel_id is None:
logger.debug("[BRIDGE-OUT] no channel mapping for room %s", room.room_id)
return
@@ -2076,6 +2364,33 @@ class CollaborationCore:
event_content, channel_id
)
+ if not await self._claim_outbound(channel_id, event.event_id):
+ return
+ try:
+ message_ref = await self._post_outbound_message(
+ channel_id,
+ event,
+ sender_name,
+ thread_root_ref,
+ admin_marker=admin_marker if is_system else None,
+ is_system=is_system,
+ )
+ except (Exception, asyncio.CancelledError):
+ await asyncio.shield(self._release_outbound(event.event_id))
+ raise
+ await self._settle_outbound(event.event_id, message_ref)
+
+ async def _post_outbound_message(
+ self,
+ channel_id: str,
+ event: TransportMessage,
+ sender_name: str | None,
+ thread_root_ref: str | None,
+ *,
+ admin_marker: object,
+ is_system: bool,
+ ) -> str | None:
+ event_content = event.content
if is_system:
message_type = (
admin_marker.get("type") if isinstance(admin_marker, dict) else None
@@ -2096,35 +2411,27 @@ class CollaborationCore:
and person.label not in body
):
body = f"On behalf of {person.label}:\n\n{body}"
- message_ref = await self._adapter.admin_message(
+ return await self._adapter.admin_message(
channel_id,
body,
thread_root_ref,
message_type=message_type,
)
- else:
- assert sender_name is not None # guarded above
- # Only the marker pages a channel. An `@channel` an agent wrote
- # itself is defused by `translate_outbound` like any other text.
- room_wide = is_room_wide_mention(event_content)
- message_ref = await self._adapter.send_message(
- channel_id,
- sender_name,
- (
- self._adapter.render_room_wide_mention(event.body)
- if room_wide
- else self._adapter.translate_outbound(event.body)
- ),
- thread_root_id=thread_root_ref,
- room_wide_mention=room_wide,
- )
-
- if message_ref is not None:
- await self._record_message_map(
- external_channel_id=channel_id,
- transport_event_id=event.event_id,
- external_post_id=message_ref,
- )
+ assert sender_name is not None # guarded by the caller
+ # Only the marker pages a channel. An `@channel` an agent wrote
+ # itself is defused by `translate_outbound` like any other text.
+ room_wide = is_room_wide_mention(event_content)
+ return await self._adapter.send_message(
+ channel_id,
+ sender_name,
+ (
+ self._adapter.render_room_wide_mention(event.body)
+ if room_wide
+ else self._adapter.translate_outbound(event.body)
+ ),
+ thread_root_id=thread_root_ref,
+ room_wide_mention=room_wide,
+ )
async def _outbound_thread_root_ref(
self, event_content: dict[str, object], channel_id: str
@@ -2174,12 +2481,12 @@ class CollaborationCore:
room.room_id,
event.body[:80] if event.body else "",
)
- if event.sender in self._human_user_ids:
- return
if event.sender == self._workspace_consumer_transport_user_id:
return
+ if await self._is_human_sender(event.sender):
+ return
- channel_id = self._find_channel(transport_room_id=room.room_id)
+ channel_id = await self._channel_for_room(room.room_id)
if channel_id is None:
logger.debug("[BRIDGE-OUT] no channel mapping for room %s", room.room_id)
return
@@ -2187,10 +2494,34 @@ class CollaborationCore:
room_id, _ = self._channel_to_room[channel_id]
with tenant_scope(await self._room_tenant(room_id)):
with self._counted_outbound("media"):
- await self._relay_outbound_media(channel_id, event, client)
+ await self._relay_outbound_media(room, channel_id, event, client)
+
+ async def _is_human_sender(self, sender: str) -> bool:
+ """Whether a room message was written by a person on this bridge, so
+ relaying it out would echo them back to themselves.
+
+ Kept in memory both ways, and read from the database on a miss: a
+ person whose actor another process minted is known there first.
+ """
+ if sender in self._human_user_ids:
+ return True
+ if sender in self._not_human_user_ids:
+ return False
+ async with tenant_session(
+ self._session_factory, self._bridge_tenant_id
+ ) as session:
+ human = await self._external_user_store.is_bridge_user(
+ session, self._bridge_id, sender
+ )
+ (self._human_user_ids if human else self._not_human_user_ids).add(sender)
+ return human
async def _relay_outbound_media(
- self, channel_id: str, event: TransportMedia, client: WorkspaceConsumer
+ self,
+ room: RoomRef,
+ channel_id: str,
+ event: TransportMedia,
+ client: WorkspaceConsumer,
) -> None:
event_content = event.content
sender_name = event.sender_name
@@ -2221,43 +2552,29 @@ class CollaborationCore:
)
)
- message_ref: str | None
data = await self._download_media(client, event.uri, filename)
- if data is None or len(data) > self._max_attachment_bytes:
- if data is not None:
- logger.warning(
- "[BRIDGE-OUT] attachment %s is %d bytes, over the "
- "%d-byte relay cap",
- filename,
- len(data),
- self._max_attachment_bytes,
- )
- note = f"_sent an attachment that couldn't be relayed: {filename}_"
- body = f"{caption}\n{note}" if caption else note
- message_ref = await self._adapter.send_message(
- channel_id,
- sender_name,
- self._adapter.translate_outbound(body),
- thread_root_id=thread_root_ref,
- )
- else:
+ if data is not None and len(data) <= self._max_attachment_bytes:
# Part of a multi-file message? Hold it until the whole group has
# arrived, then relay all of it as one platform post.
group = parse_attachment_group(event_content)
if group is not None:
group_id, index, total = group
pending = self._outbound_groups.setdefault(
- group_id, _PendingOutboundGroup(total=total)
+ group_id,
+ _PendingOutboundGroup(
+ total=total, room_id=room.room_id, from_seq=event.seq
+ ),
)
+ pending.from_seq = min(pending.from_seq, event.seq)
pending.parts[index] = OutboundAttachment(
filename=filename, mimetype=mimetype, data=data
)
+ pending.event_ids[index] = event.event_id
if index == 0:
pending.caption = caption
- pending.first_event_id = event.event_id
if len(pending.parts) < total:
self._schedule_outbound_group_flush(
- group_id, channel_id, sender_name, thread_root_ref
+ group_id, channel_id, sender_name, thread_root_ref, client
)
return
self._cancel_outbound_group_flush(group_id)
@@ -2267,22 +2584,55 @@ class CollaborationCore:
)
return
- message_ref = await self._adapter.send_attachment(
- channel_id,
- sender_name,
- filename,
- mimetype,
- data,
- caption=caption,
- thread_root_id=thread_root_ref,
- )
-
- if message_ref is not None:
- await self._record_message_map(
- external_channel_id=channel_id,
- transport_event_id=event.event_id,
- external_post_id=message_ref,
- )
+ if not await self._claim_outbound(channel_id, event.event_id):
+ return
+ message_ref: str | None
+ try:
+ if data is None or len(data) > self._max_attachment_bytes:
+ if data is not None:
+ logger.warning(
+ "[BRIDGE-OUT] attachment %s is %d bytes, over the "
+ "%d-byte relay cap",
+ filename,
+ len(data),
+ self._max_attachment_bytes,
+ )
+ note = f"_sent an attachment that couldn't be relayed: {filename}_"
+ body = f"{caption}\n{note}" if caption else note
+ message_ref = await self._adapter.send_message(
+ channel_id,
+ sender_name,
+ self._adapter.translate_outbound(body),
+ thread_root_id=thread_root_ref,
+ )
+ else:
+ message_ref = await self._adapter.send_attachment(
+ channel_id,
+ sender_name,
+ filename,
+ mimetype,
+ data,
+ caption=caption,
+ thread_root_id=thread_root_ref,
+ )
+ except (Exception, asyncio.CancelledError):
+ await asyncio.shield(self._release_outbound(event.event_id))
+ raise
+ await self._settle_outbound(event.event_id, message_ref)
+
+ def held_from(self, room_id: str) -> int | None:
+ """The first file of a multi-file message in this room still waiting
+ for the rest: not posted yet, so the bridge's progress in the room
+ stops before it."""
+ held = [
+ pending.from_seq
+ for pending in [
+ *self._outbound_groups.values(),
+ *self._flushing_outbound_groups,
+ ]
+ if pending.room_id == room_id
+ ]
+ return min(held) if held else None
def _schedule_outbound_group_flush(
self,
@@ -2290,6 +2640,7 @@ class CollaborationCore:
channel_id: str,
sender_name: str,
thread_root_ref: str | None,
+ client: WorkspaceConsumer,
) -> None:
"""Arm the safety net for an incomplete outbound attachment group.
@@ -2306,7 +2657,7 @@ class CollaborationCore:
OUTBOUND_GROUP_TIMEOUT_SECONDS,
lambda: asyncio.create_task(
self._flush_incomplete_outbound_group(
- group_id, channel_id, sender_name, thread_root_ref
+ group_id, channel_id, sender_name, thread_root_ref, client
)
),
)
@@ -2322,11 +2673,39 @@ class CollaborationCore:
channel_id: str,
sender_name: str,
thread_root_ref: str | None,
+ client: WorkspaceConsumer,
) -> None:
self._outbound_group_timers.pop(group_id, None)
- pending = self._outbound_groups.pop(group_id, None)
- if pending is None:
+ pending = self._outbound_groups.get(group_id)
+ if pending is None or not client.running:
+ # A bridge that stopped leaves the group to whoever holds it next:
+ # its progress stops before the group's first file.
return
+ # A file of the group arriving while this posts would otherwise be
+ # added to a post already on its way without it, and never sent.
+ del self._outbound_groups[group_id]
+ self._flushing_outbound_groups.append(pending)
+ try:
+ await self._flush_outbound_group(
+ group_id, pending, channel_id, sender_name, thread_root_ref
+ )
+ finally:
+ self._flushing_outbound_groups = [
+ flushing
+ for flushing in self._flushing_outbound_groups
+ if flushing is not pending
+ ]
+ if client.transport is not None:
+ client.transport.held_back_released(pending.room_id)
+
+ async def _flush_outbound_group(
+ self,
+ group_id: str,
+ pending: _PendingOutboundGroup,
+ channel_id: str,
+ sender_name: str,
+ thread_root_ref: str | None,
+ ) -> None:
received = len(pending.parts)
logger.error(
"[BRIDGE-OUT] attachment group %s incomplete: %d of %d parts arrived "
@@ -2352,19 +2731,24 @@ class CollaborationCore:
sender_name: str,
thread_root_ref: str | None,
) -> None:
- message_ref = await self._adapter.send_attachments(
- channel_id,
- sender_name,
- [pending.parts[i] for i in sorted(pending.parts)],
- caption=pending.caption,
- thread_root_id=thread_root_ref,
- )
- if message_ref is not None and pending.first_event_id is not None:
- await self._record_message_map(
- external_channel_id=channel_id,
- transport_event_id=pending.first_event_id,
- external_post_id=message_ref,
+ # Claimed, and recorded, by the earliest file that arrived: the post
+ # stands for the whole group, and that file is the first of it a reply
+ # can thread under.
+ claim_event_id = pending.event_ids[min(pending.event_ids)]
+ if not await self._claim_outbound(channel_id, claim_event_id):
+ return
+ try:
+ message_ref = await self._adapter.send_attachments(
+ channel_id,
+ sender_name,
+ [pending.parts[i] for i in sorted(pending.parts)],
+ caption=pending.caption,
+ thread_root_id=thread_root_ref,
)
+ except (Exception, asyncio.CancelledError):
+ await asyncio.shield(self._release_outbound(claim_event_id))
+ raise
+ await self._settle_outbound(claim_event_id, message_ref)
async def _download_media(
self, client: WorkspaceConsumer, media_uri: str | None, filename: str
@@ -2407,7 +2791,7 @@ class CollaborationCore:
mapping = await self._bridge_message_map_store.get_by_transport_event_id(
session, self._bridge_id, event_id
)
- if mapping is None:
+ if mapping is None or mapping.external_post_id is None:
return
channel_id = mapping.external_channel_id
@@ -2441,6 +2825,71 @@ class CollaborationCore:
race."""
self._pending_message_maps[transport_event_id] = external_post_id
+ async def _claim_outbound(self, channel_id: str, transport_event_id: str) -> bool:
+ """Claim a room message for posting to the platform, before posting
+ it. False when it is claimed already: posted, or being posted, by
+ whoever held the bridge before. Claiming first is what keeps a
+ hand-over between processes from posting a message twice.
+
+ A post that fails, or is cancelled because the bridge stopped before
+ it finished, lets its claim go, so whoever holds the bridge next posts
+ it; one the platform took just before the cancel then shows twice. A
+ process that dies between the claim and the post leaves the claim,
+ which the next holder claims again once it is
+ `_UNPOSTED_CLAIM_LAPSES_AFTER` old and posts the message, twice if the
+ platform had taken it before the death."""
+ async with tenant_session(
+ self._session_factory, self._bridge_tenant_id
+ ) as session:
+ claimed = await self._bridge_message_map_store.claim(
+ session,
+ bridge_id=self._bridge_id,
+ external_channel_id=channel_id,
+ transport_event_id=transport_event_id,
+ unposted_after=_UNPOSTED_CLAIM_LAPSES_AFTER,
+ )
+ await session.commit()
+ if not claimed:
+ logger.info(
+ "[BRIDGE-OUT] %s was already posted to channel %s; not posting it "
+ "again",
+ transport_event_id,
+ channel_id,
+ )
+ return claimed
+
+ async def _settle_outbound(
+ self, transport_event_id: str, external_post_id: str | None
+ ) -> None:
+ """Record what a claimed message became on the platform, or let the
+ claim go when the platform took nothing."""
+ if external_post_id is None:
+ await self._release_outbound(transport_event_id)
+ return
+ async with tenant_session(
+ self._session_factory, self._bridge_tenant_id
+ ) as session:
+ await self._bridge_message_map_store.fill(
+ session,
+ bridge_id=self._bridge_id,
+ transport_event_id=transport_event_id,
+ external_post_id=external_post_id,
+ )
+ await session.commit()
+
+ async def _release_outbound(self, transport_event_id: str) -> None:
+ """Let a claim go after the platform refused the post, or the post
+ was cancelled before it finished."""
+ async with tenant_session(
+ self._session_factory, self._bridge_tenant_id
+ ) as session:
+ await self._bridge_message_map_store.release_claim(
+ session,
+ bridge_id=self._bridge_id,
+ transport_event_id=transport_event_id,
+ )
+ await session.commit()
+
async def _record_message_map(
self,
*,
@@ -2481,6 +2930,7 @@ class CollaborationCore:
mapping = await self._bridge_message_map_store.get_by_transport_event_id(
session, self._bridge_id, transport_event_id
)
+ # A claim not filled yet maps to nothing on the platform.
return mapping.external_post_id if mapping is not None else None
# ── Room mapping management ──────────────────────────────────────────────
diff --git a/core/switch_core/bridges/collaboration/install_service.py b/core/switch_core/bridges/collaboration/install_service.py
index dbd990040..bb228b62a 100644
--- a/core/switch_core/bridges/collaboration/install_service.py
+++ b/core/switch_core/bridges/collaboration/install_service.py
@@ -305,9 +305,6 @@ class MessagingInstallService:
self._lifecycle = lifecycle
self._public_origin = public_origin
self._keyring = keyring
- # Burnt claim states awaiting an answer: state id -> (chat, when). See
- # `propose`.
- self._proposals: dict[str, tuple[str, float]] = {}
def _redirect_uri(self, platform: str) -> str:
"""Where the platform sends the browser back to.
@@ -749,10 +746,9 @@ class MessagingInstallService:
The claim-based counterpart of `complete`: verify, then burn the state
and commit before anything else, so a code seen in the chat — in the
claim itself, or behind the proposal's buttons — is dead from here on.
- The burnt state is then held for this one chat, and only an answer from
- it can decide it (see `connect`). Held in memory: the app runs on one
- pod, and a restart costs an unanswered proposal, whose answer is then
- refused as expired.
+ The burnt state is held for this one chat, in its row, and only an
+ answer from it can decide it (see `connect`), whichever switch-core
+ process the answer reaches.
Everything `connect` would refuse is refused here first, so a chat is
only asked about a claim that can still succeed.
@@ -776,15 +772,14 @@ class MessagingInstallService:
f"the {platform} workspace {chat_id} is already connected to Switch"
)
try:
- burnt = await self._burn(state)
+ burnt = await self._burn(state, proposed_in=chat_id)
except MessagingInstallStateError:
- if self._proposed_chat(state.state_id) == chat_id:
+ if await self._proposed_chat(state) == chat_id:
raise InstallClaimRepeated(
f"the {platform} workspace {chat_id} was already asked "
"about this claim"
) from None
raise
- self._proposals[state.state_id] = (chat_id, time.monotonic())
async with tenant_session(
self._session_factory, state.tenant_id
@@ -916,16 +911,15 @@ class MessagingInstallService:
)
return attached
- def _proposed_chat(self, state_id: str) -> str | None:
+ async def _proposed_chat(self, state: InstallState) -> str | None:
"""The chat a burnt state was proposed in, while it can be answered."""
- held = self._proposals.get(state_id)
- if held is None:
- return None
- chat_id, proposed_at = held
- if time.monotonic() - proposed_at > CONFIRM_TTL.total_seconds():
- del self._proposals[state_id]
- return None
- return chat_id
+ async with tenant_session(self._session_factory, state.tenant_id) as session:
+ return await self._store.proposed_chat(
+ session,
+ state_id=state.state_id,
+ platform=state.platform,
+ window=CONFIRM_TTL,
+ )
async def _decide(
self,
@@ -940,7 +934,8 @@ class MessagingInstallService:
from another. Who answered is checked before the state is decided, so
a refused press leaves the proposal for an admin to answer.
"""
- if self._proposed_chat(state.state_id) != claim.grant.external_workspace_id:
+ chat_id = claim.grant.external_workspace_id
+ if await self._proposed_chat(state) != chat_id:
raise MessagingInstallStateError(
"this claim was not proposed in this chat, or its proposal has "
"expired. Start again from Switch."
@@ -952,9 +947,9 @@ class MessagingInstallService:
state_id=state.state_id,
platform=state.platform,
window=CONFIRM_TTL,
+ proposed_in=chat_id,
)
await session.commit()
- self._proposals.pop(state.state_id, None)
return decided
@staticmethod
@@ -1019,16 +1014,21 @@ class MessagingInstallService:
raise
await asyncio.sleep(_BRIDGE_START_POLL)
- async def _burn(self, state: InstallState) -> MessagingInstallState:
+ async def _burn(
+ self, state: InstallState, *, proposed_in: str | None = None
+ ) -> MessagingInstallState:
"""Redeem a verified state, and commit, before anything else happens.
Shared by both ways an install completes, and in its own transaction
for the reason the module docstring gives: a replay must fail even if
- everything after this does.
+ everything after this does. A claim names the chat it is put to.
"""
async with tenant_session(self._session_factory, state.tenant_id) as session:
burnt = await self._store.redeem_state(
- session, state_id=state.state_id, platform=state.platform
+ session,
+ state_id=state.state_id,
+ platform=state.platform,
+ proposed_in=proposed_in,
)
await session.commit()
return burnt
diff --git a/core/switch_core/bridges/collaboration/lifecycle_service.py b/core/switch_core/bridges/collaboration/lifecycle_service.py
index b3b3ff5b3..e67665dd6 100644
--- a/core/switch_core/bridges/collaboration/lifecycle_service.py
+++ b/core/switch_core/bridges/collaboration/lifecycle_service.py
@@ -1,6 +1,25 @@
+"""Every collaboration bridge: quiet in every process, listening in the one that holds it.
+
+A bridge is a connection to a messaging platform. Two halves of it must run
+once: listening to the platform (a socket, a poll) and relaying the bridge's
+rooms out to it (the `WorkspaceConsumer`) with the session-activity drawings
+beside it, or every event comes in, and every message goes out, once per
+process. Those run in the one switch-core process that holds the bridge
+(`ownership.py`, kind `BRIDGE_CONNECTIONS`). Every other process runs the
+bridge quietly, without them, because requests that touch a platform (create a
+channel, add an agent to one, answer a webhook or a button press) reach any
+process.
+
+Registering, editing and removing a bridge reach every process as a
+`bridge.changed` signal (`bridge_signals.py`), so the holder applies an edit
+made through another process, and a bridge removed anywhere stops everywhere.
+"""
+
from __future__ import annotations
import asyncio
+import hashlib
+import json
import logging
import re
import time
@@ -43,6 +62,7 @@ from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.bridge_message_map_store import BridgeMessageMapStore
from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
from switch_core.db.stores.external_user_store import ExternalUserStore
from switch_core.db.stores.room_store import RoomStore
from switch_core.db.tenant_lookup import all_tenant_ids, tenant_of_collaboration_bridge
@@ -58,19 +78,55 @@ from switch_core.telemetry.deployment import (
)
from switch_core.telemetry.snapshot import normalise_platform
from switch_core.tenant_context import current_tenant_id, no_tenant
+from switch_core.transport.cursors import ConsumerCursors, CursorWriter
from switch_core.trust.client import NullTrustClient, TrustClient
if TYPE_CHECKING:
from switch_core.bridges.agent.protocol.agent_connections import (
AgentConnectionRegistry,
)
+ from switch_core.bridges.collaboration.bridge_signals import BridgeSignals
from switch_core.clients.client_lifecycle_service import ClientLifecycleService
+ from switch_core.ownership import Fence, Ownership
from switch_core.room_service import RoomService
from switch_core.session_activity.listener import AgentSessionActivityListener
from switch_core.session_activity.service import AgentSessionActivityService
logger = logging.getLogger(__name__)
+# How long a bridge being stopped waits for its workspace consumer to finish
+# the message in hand. The transport gives it as long, then cancels the rest.
+STOP_GRACE_SECONDS = 5.0
+
+# The kind of owned work a bridge's platform connection is (`ownership.py`),
+# keyed by bridge id.
+BRIDGE_CONNECTIONS = "bridge"
+
+
+# What a running bridge writes into its own config as it learns it from
+# traffic (`_persist_service_url`, `_persist_channel_team`). Not an edit: the
+# copy that learned it already uses it, and any other reads it on its next
+# start or learns it from traffic itself.
+_LEARNED_CONFIG_KEYS = frozenset({"service_url", "channel_teams"})
+
+
+def _started_from(bridge: CollaborationBridge) -> str:
+ """What a running bridge was built from, as one value: an edit made by
+ another process changes it, and a re-check that finds it changed restarts
+ the bridge here."""
+ config = {
+ key: value
+ for key, value in (bridge.connection_config or {}).items()
+ if key not in _LEARNED_CONFIG_KEYS
+ }
+ return hashlib.sha256(
+ json.dumps(
+ [bridge.type, bridge.display_name, config],
+ sort_keys=True,
+ default=str,
+ ).encode()
+ ).hexdigest()
+
# Reaching the platform failed outright, in the transports Switch itself
# depends on: aiohttp (Slack's Socket Mode, Teams' inbound listener) and httpx
@@ -202,10 +258,36 @@ class CollaborationBridgeLifecycleService:
session_activity_listener: AgentSessionActivityListener,
session_activity_service: AgentSessionActivityService,
connections: AgentConnectionRegistry,
+ ownership: Ownership,
+ cursor_writer: CursorWriter,
telemetry: TelemetryService | None = None,
trust_client: TrustClient = NullTrustClient(),
) -> None:
self._bridge_store = bridge_store
+ self._cursor_writer = cursor_writer
+ self._cursor_store = DeliveryCursorStore()
+ self._owned = ownership.kind(
+ BRIDGE_CONNECTIONS, start=self._take_over, stop=self._hand_over
+ )
+ self._signals: BridgeSignals | None = None
+ # The running bridges' workspace consumers, here only where this
+ # process holds the bridge, with the cursors each runs under.
+ self._workspace_consumers: dict[str, WorkspaceConsumer] = {}
+ self._workspace_cursors: dict[str, ConsumerCursors] = {}
+ # What each running bridge was started from (`_started_from`).
+ self._started_rows: dict[str, str] = {}
+ # Quiet copies being started in the background after a hand-over.
+ self._quiet_starts: dict[str, asyncio.Task[None]] = {}
+ # One start or stop of a bridge at a time. A bridge is started and
+ # stopped from several places that interleave: the ownership taking
+ # it over or handing it on, an edit made here or announced from
+ # another process, a quiet copy starting in the background. Two
+ # starts at once would build two adapters, and the one overwritten
+ # would go on listening with nothing left to stop it.
+ self._bridge_locks: dict[str, asyncio.Lock] = {}
+ # Why the last take-over of each bridge failed to start it, for
+ # `register` to raise: the ownership logs the failure and goes on.
+ self._failed_take_overs: dict[str, Exception] = {}
self._external_user_store = external_user_store
self._bridge_message_map_store = bridge_message_map_store
self._room_store = room_store
@@ -290,15 +372,21 @@ class CollaborationBridgeLifecycleService:
# so two concurrent registrations would both see a free resource and
# both take it — the very collision the check exists to refuse.
#
- # A process-wide lock is sufficient *because* switch-core is a
- # singleton: the chart fails the render for replicaCount != 1, since
- # it keeps delivery state (event buffers, the invite and presence
- # buses, the message listener) in memory. If that ever changes, this
- # has to become a database constraint — the way the single-default
- # bridge invariant already is — because a lock in one process would
- # then be guarding nothing.
+ # A process-wide lock guards only this process's registrations. Two
+ # processes registering a conflicting bridge at the same moment are
+ # not kept apart by it; that needs a database constraint, as the
+ # single-default bridge invariant has.
self._register_lock = asyncio.Lock()
+ def use_signals(self, signals: BridgeSignals) -> None:
+ """Where a bridge registered, edited or removed here is announced, so
+ every other process applies it to its own copy (`bridge_signals.py`)."""
+ self._signals = signals
+
+ def _announce(self, bridge_id: str) -> None:
+ if self._signals is not None:
+ self._signals.changed(bridge_id)
+
def register_adapter(
self,
bridge_type: str,
@@ -630,12 +718,24 @@ class CollaborationBridgeLifecycleService:
if bridge.tenant_id == tenant_id
)
- logger.info("Starting %d collaboration bridges", len(bridges))
+ logger.info("Wanting %d collaboration bridges", len(bridges))
for bridge in bridges:
+ self._owned.want(bridge.id)
+
+ async def start_quiet_copies(self) -> None:
+ """Start, quietly, every wanted bridge this process does not hold.
+
+ After the ownership's first balance (`main`), which has started the
+ bridges this process holds: starting the others before it would build
+ every bridge twice at boot, quiet and then listening.
+ """
+ for bridge_id in sorted(self._owned.wanted()):
+ if self.holds(bridge_id) or bridge_id in self._bridges:
+ continue
try:
- await self.start(bridge.id)
+ await self.start(bridge_id)
except Exception:
- logger.exception("Failed to start bridge %s", bridge.id)
+ logger.exception("Failed to start bridge %s", bridge_id)
def reserve_resource(self, resource: str, holder: str) -> None:
"""Keep a resource the deployment itself uses away from every bridge.
@@ -881,7 +981,15 @@ class CollaborationBridgeLifecycleService:
)
try:
- await self.start(bridge.id)
+ # Which process holds it is settled first, so it starts once, in
+ # the form it will run here.
+ self._owned.want(bridge.id)
+ await self._owned.settle()
+ failed = self._failed_take_overs.pop(bridge.id, None)
+ if failed is not None:
+ raise failed
+ if not self.holds(bridge.id):
+ await self.start(bridge.id)
except Exception:
# Stored and not startable is a half state: a row that fails every
# boot, and for an install a second bridge on the next attempt. So
@@ -912,10 +1020,40 @@ class CollaborationBridgeLifecycleService:
bridge_type,
display_name,
)
+ self._announce(bridge.id)
+ return bridge
+
+ def _bridge_lock(self, bridge_id: str) -> asyncio.Lock:
+ lock = self._bridge_locks.get(bridge_id)
+ if lock is None:
+ lock = self._bridge_locks[bridge_id] = asyncio.Lock()
+ return lock
+
+ def _running(self, bridge_id: str) -> CollaborationCore | None:
+ bridge = self._bridges.get(bridge_id)
+ task = self._tasks.get(bridge_id)
+ if bridge is None or task is None or task.done():
+ return None
return bridge
async def start(self, bridge_id: str) -> None:
- """Start a bridge, reporting a failure even before one is ever run.
+ """Start a bridge here, listening if this process holds it and quietly
+ otherwise. One already running in that form is left as it is, and one
+ running in the other form is restarted in this one."""
+ async with self._bridge_lock(bridge_id):
+ await self._start_locked(bridge_id)
+
+ async def _start_locked(self, bridge_id: str) -> None:
+ running = self._running(bridge_id)
+ if running is not None and running.active == self.holds(bridge_id):
+ return
+ if running is not None or bridge_id in self._bridges:
+ await self._stop_locked(bridge_id, reason="restart")
+ await self._start_bridge(bridge_id)
+
+ async def _start_bridge(self, bridge_id: str) -> None:
+ """Build and run a bridge here, listening if this process holds it and
+ quietly otherwise, reporting a failure even before one is ever run.
Everything here runs before `_run_bridge` is scheduled, so nothing
downstream reports a raise from this method. Unreported it would be an
@@ -935,6 +1073,8 @@ class CollaborationBridgeLifecycleService:
# room with no bridge at all.
platform = "unknown"
adapter_cls: type[PlatformAdapter] | None = None
+ fence = self._owned.fence(bridge_id)
+ active = fence is not None
# Before the first statement that can fail, so a bridge that never gets
# past reading its own row still reports how long that took.
self._connect_started[bridge_id] = time.monotonic()
@@ -955,6 +1095,10 @@ class CollaborationBridgeLifecycleService:
bridge = await self._bridge_store.get(session, bridge_id)
if bridge is None:
raise ValueError(f"Bridge not found: {bridge_id}")
+ # Recorded before anything can fail: a re-check started from the
+ # same row does not try again what failed here, as one process
+ # tried a failing bridge once until it was edited or restarted.
+ self._started_rows[bridge_id] = _started_from(bridge)
platform = normalise_platform(bridge.type)
adapter_cls = self._adapter_registry.get(bridge.type)
@@ -1005,6 +1149,7 @@ class CollaborationBridgeLifecycleService:
connection_config=bridge.connection_config or {},
)
adapter = adapter_cls(config=typed_config) # type: ignore[call-arg]
+ adapter.set_listening(active)
adapter.set_tenant_id(tenant_id)
adapter.set_outbound_policy(self._config.outbound_policy)
adapter.set_service_url_persister(
@@ -1065,22 +1210,37 @@ class CollaborationBridgeLifecycleService:
session_activity_listener=self._session_activity_listener,
session_activity_service=self._session_activity_service,
connections=self._connections,
+ active=active,
trust_client=self._trust_client,
)
- workspace_consumer = WorkspaceConsumer(
- collaboration_core=collaboration_core,
- actor=Actor(
- client_id=workspace_consumer_record.id,
+ # Only where this process holds the bridge: everywhere else the
+ # bridge relays nothing out, and its rooms are read by the holder.
+ workspace_consumer: WorkspaceConsumer | None = None
+ if fence is not None:
+ workspace_consumer = WorkspaceConsumer(
+ collaboration_core=collaboration_core,
+ actor=Actor(
+ client_id=workspace_consumer_record.id,
+ tenant_id=workspace_consumer_record.tenant_id,
+ transport_user_id=workspace_consumer_record.transport_user_id,
+ display_name=workspace_consumer_record.display_name,
+ session_factory=self._session_factory,
+ client_store=self._client_store,
+ config=WorkspaceConsumerConfig(bridge_id=bridge_id),
+ transport_factory=self._client_factory.transport_for,
+ ),
+ )
+ self._workspace_consumers[bridge_id] = workspace_consumer
+ self._workspace_cursors[bridge_id] = ConsumerCursors(
+ writer=self._cursor_writer,
+ store=self._cursor_store,
tenant_id=workspace_consumer_record.tenant_id,
- transport_user_id=workspace_consumer_record.transport_user_id,
- display_name=workspace_consumer_record.display_name,
- session_factory=self._session_factory,
- client_store=self._client_store,
- config=WorkspaceConsumerConfig(bridge_id=bridge_id),
- transport_factory=self._client_factory.transport_for,
- ),
- )
+ client_id=workspace_consumer_record.id,
+ fence=fence,
+ actor_role=workspace_consumer.actor.role,
+ on_lost=self._owned.lost,
+ )
for listener in self._bridge_starting_listeners:
listener(adapter)
@@ -1107,8 +1267,19 @@ class CollaborationBridgeLifecycleService:
if workspace is not None:
self._running_workspaces[bridge_id] = workspace
- logger.info("Started collaboration bridge %s (%s)", bridge_id, bridge.type)
+ logger.info(
+ "Started collaboration bridge %s (%s), %s",
+ bridge_id,
+ bridge.type,
+ "listening" if active else "quiet: another process holds it",
+ )
except Exception as exc:
+ self._workspace_consumers.pop(bridge_id, None)
+ self._workspace_cursors.pop(bridge_id, None)
+ if not active:
+ # Reported by the process that holds the bridge, once.
+ self._connect_started.pop(bridge_id, None)
+ raise
self._note_connect_failure(bridge_id)
# Resolved before the call, never inside its argument list: an
# expression there is evaluated before `emit_safely` is entered, so
@@ -1190,9 +1361,11 @@ class CollaborationBridgeLifecycleService:
bridge_id: str,
tenant_id: str,
collaboration_core: CollaborationCore,
- workspace_consumer: WorkspaceConsumer,
+ workspace_consumer: WorkspaceConsumer | None,
) -> None:
- """The bridge's own long-lived task.
+ """The bridge's own long-lived task: until stopped, the workspace
+ consumer where this process holds the bridge, and nothing but the
+ quiet adapter elsewhere.
`no_tenant` first, because an `asyncio.Task` snapshots the contextvars
of whoever created it — and `start` is reached from an HTTP request as
@@ -1208,16 +1381,31 @@ class CollaborationBridgeLifecycleService:
with no_tenant():
connected = False
try:
- await self._record_bridge_memberships(
- bridge_id, tenant_id, workspace_consumer.client_id
- )
+ if workspace_consumer is not None:
+ await self._record_bridge_memberships(
+ bridge_id, tenant_id, workspace_consumer.client_id
+ )
await collaboration_core.start()
# The one point that means "connected": `start()` only
- # launched this task and `workspace_consumer.start()` never returns.
+ # launched this task, and what follows runs until the bridge
+ # is stopped.
connected = True
self._connected.add(bridge_id)
+ if workspace_consumer is None:
+ # Quiet: serving requests and webhooks until stopped.
+ await asyncio.Event().wait()
+ return
await self._report_connector_up(bridge_id, platform, configured_at)
- await workspace_consumer.start()
+ cursors = self._workspace_cursors.get(bridge_id)
+ if (
+ cursors is None
+ or self._workspace_consumers.get(bridge_id)
+ is not workspace_consumer
+ ):
+ # Stopped, or stopped and started again, while it was
+ # coming up: this consumer is not to read at all.
+ return
+ await workspace_consumer.start(cursors)
except Exception as exc:
logger.exception("Bridge %s crashed", bridge_id)
self._bridges.pop(bridge_id, None)
@@ -1240,7 +1428,10 @@ class CollaborationBridgeLifecycleService:
reason = _failure_reason(
exc, self._running_adapter_classes.pop(bridge_id, None)
)
- if connected:
+ if workspace_consumer is None:
+ # Reported by the process that holds the bridge, once.
+ self._connect_started.pop(bridge_id, None)
+ elif connected:
emit_safely(
self._telemetry,
"bridge_disconnected",
@@ -1386,6 +1577,31 @@ class CollaborationBridgeLifecycleService:
)
async def stop(self, bridge_id: str, *, reason: str = "shutdown") -> None:
+ async with self._bridge_lock(bridge_id):
+ await self._stop_locked(bridge_id, reason=reason)
+
+ async def _stop_locked(self, bridge_id: str, *, reason: str) -> None:
+ # The workspace consumer first, and gracefully: it finishes the
+ # message in hand and stores how far it got, so whoever holds the
+ # bridge next neither posts it again nor skips it.
+ workspace_consumer = self._workspace_consumers.pop(bridge_id, None)
+ cursors = self._workspace_cursors.pop(bridge_id, None)
+ if workspace_consumer is not None:
+ await workspace_consumer.stop()
+ running = self._tasks.get(bridge_id)
+ if running is not None:
+ await asyncio.wait({running}, timeout=STOP_GRACE_SECONDS)
+ if cursors is not None:
+ try:
+ await cursors.flush()
+ except Exception:
+ logger.warning(
+ "Could not store bridge %s's progress as it stopped; whoever "
+ "holds it next handles its last messages again, and posts "
+ "none of them twice",
+ bridge_id,
+ exc_info=True,
+ )
# Before the adapter goes, so a press in flight is answered as gone
# rather than handled by a bridge that is halfway shut down.
endpoint = self._callback_endpoints.pop(bridge_id, None)
@@ -1394,29 +1610,39 @@ class CollaborationBridgeLifecycleService:
collaboration_core = self._bridges.get(bridge_id)
was_connected = bridge_id in self._connected
self._connected.discard(bridge_id)
- if collaboration_core:
- await collaboration_core.stop()
-
- task = self._tasks.pop(bridge_id, None)
- if task and not task.done():
- task.cancel()
-
- self._bridges.pop(bridge_id, None)
- self._held_resources.pop(bridge_id, None)
- self._running_workspaces.pop(bridge_id, None)
- self._running_adapter_classes.pop(bridge_id, None)
- self._started.discard(bridge_id)
- # A bridge cancelled before it connected never reaches an outcome, and
- # `_run_bridge`'s handler does not catch `CancelledError`, so its
- # reading would otherwise sit here for the life of the process — and be
- # spent by the *next* attempt on the same id, which would then report a
- # duration measured from the one before it.
- self._connect_started.pop(bridge_id, None)
+ try:
+ if collaboration_core:
+ await collaboration_core.stop()
+ finally:
+ # Unregistered whatever the stop raised, as the lease is let go
+ # of all the same: left here, the copy would read as running and
+ # be stopped again at shutdown.
+ task = self._tasks.pop(bridge_id, None)
+ if task and not task.done():
+ task.cancel()
+
+ self._bridges.pop(bridge_id, None)
+ self._held_resources.pop(bridge_id, None)
+ self._running_workspaces.pop(bridge_id, None)
+ self._running_adapter_classes.pop(bridge_id, None)
+ self._started.discard(bridge_id)
+ # A bridge cancelled before it connected never reaches an outcome, and
+ # `_run_bridge`'s handler does not catch `CancelledError`, so its
+ # reading would otherwise sit here for the life of the process — and be
+ # spent by the *next* attempt on the same id, which would then report a
+ # duration measured from the one before it.
+ self._connect_started.pop(bridge_id, None)
+ self._started_rows.pop(bridge_id, None)
logger.info("Stopped collaboration bridge %s", bridge_id)
# `_bridges` membership is set before the connection is attempted, so
- # on its own it would report a bridge that never connected.
- if collaboration_core is not None and was_connected:
+ # on its own it would report a bridge that never connected. Reported
+ # where the bridge was held, once.
+ if (
+ collaboration_core is not None
+ and was_connected
+ and collaboration_core.active
+ ):
platform, _ = self._bridge_facts.get(bridge_id, ("none", None))
emit_safely(
self._telemetry,
@@ -1429,20 +1655,203 @@ class CollaborationBridgeLifecycleService:
)
async def restart(self, bridge_id: str) -> None:
- """Stop and start a bridge so it picks up its stored config.
+ """Stop and start a bridge so it picks up its stored config, here and
+ in every other process.
An adapter is built from the config it was given at start, so an edit
is inert until the bridge is rebuilt."""
await self.stop(bridge_id, reason="restart")
await self.start(bridge_id)
logger.info("Restarted collaboration bridge %s", bridge_id)
+ self._announce(bridge_id)
async def stop_all(self) -> None:
+ """Stop every bridge still running here, as the process shuts down.
+ The ones it held were stopped and handed over first
+ (`Ownership.stop`); these are its quiet copies. A quiet copy still
+ starting is given the grace period to finish, so it is stopped whole
+ rather than cancelled halfway through being built."""
+ starting = list(self._quiet_starts.values())
+ if starting:
+ _, unfinished = await asyncio.wait(starting, timeout=STOP_GRACE_SECONDS)
+ for task in unfinished:
+ task.cancel()
+ await asyncio.gather(*unfinished, return_exceptions=True)
logger.info("Stopping all %d collaboration bridges", len(self._bridges))
for bridge_id in list(self._bridges):
await self.stop(bridge_id)
await self._callback_ingress.stop()
+ # ── Ownership ────────────────────────────────────────────────────────────
+
+ async def _take_over(self, bridge_id: str, fence: Fence) -> None:
+ """This process now holds the bridge: rebuild its quiet copy here as
+ the listening one. Called by the ownership with the hold in place."""
+ logger.info("Taking over collaboration bridge %s", bridge_id)
+ self._failed_take_overs.pop(bridge_id, None)
+ try:
+ await self.start(bridge_id)
+ except Exception as exc:
+ self._failed_take_overs[bridge_id] = exc
+ raise
+
+ async def _hand_over(self, bridge_id: str) -> None:
+ """This process no longer holds the bridge: stop listening, finish the
+ message in hand, and run on as a quiet copy if the bridge is still
+ wanted here. The quiet copy starts in the background, so the next
+ holder is not kept waiting on this process's platform calls."""
+ logger.info("Handing collaboration bridge %s over", bridge_id)
+ await self.stop(bridge_id, reason="restart")
+ if self._owned.wants(bridge_id) and not self._owned.stopping:
+ self._start_quietly(bridge_id)
+
+ def _start_quietly(self, bridge_id: str) -> None:
+ async def run() -> None:
+ try:
+ with no_tenant():
+ async with self._bridge_lock(bridge_id):
+ # Removed, or the process began shutting down, since
+ # the hand-over: nothing is to run here any more.
+ if not self._owned.wants(bridge_id) or self._owned.stopping:
+ return
+ await self._start_locked(bridge_id)
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ logger.exception(
+ "Collaboration bridge %s could not start quietly here after "
+ "being handed over",
+ bridge_id,
+ )
+ finally:
+ if self._quiet_starts.get(bridge_id) is asyncio.current_task():
+ del self._quiet_starts[bridge_id]
+
+ self._quiet_starts[bridge_id] = asyncio.create_task(run())
+
+ def holds(self, bridge_id: str) -> bool:
+ """Whether this process holds the bridge and listens for it."""
+ return self._owned.fence(bridge_id) is not None
+
+ # ── Changes made in other processes ──────────────────────────────────────
+
+ async def reconcile_bridge(self, bridge_id: str) -> None:
+ """Bring this process's copy of one bridge in line with its row, after
+ another process registered, edited or removed it."""
+ tenant_id = await tenant_of_collaboration_bridge(
+ self._session_factory, bridge_id
+ )
+ bridge: CollaborationBridge | None = None
+ if tenant_id is not None:
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ bridge = await self._bridge_store.get(session, bridge_id)
+ await self._reconcile_row(bridge_id, bridge)
+
+ async def reconcile_all(self) -> None:
+ """`reconcile_bridge` for every bridge stored or running here, reading
+ the rows one tenant at a time."""
+ rows: dict[str, CollaborationBridge] = {}
+ for tenant_id in await all_tenant_ids(self._session_factory):
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ rows.update(
+ (bridge.id, bridge)
+ for bridge in await self._bridge_store.get_active(session)
+ if bridge.tenant_id == tenant_id
+ )
+ bridge_ids = set(rows) | set(self._bridges) | self._owned.wanted()
+ for bridge_id in sorted(bridge_ids):
+ try:
+ await self._reconcile_row(bridge_id, rows.get(bridge_id))
+ except Exception:
+ logger.warning(
+ "Could not bring collaboration bridge %s in line with its "
+ "row; retrying at the next re-check",
+ bridge_id,
+ exc_info=True,
+ )
+
+ async def _reconcile_row(
+ self, bridge_id: str, bridge: CollaborationBridge | None
+ ) -> None:
+ """Bring this process's copy of a bridge in line with its row.
+
+ Stopped, and let go of, where the row is gone or no longer active.
+ Wanted where it is new here, so the process that should hold it
+ claims it. Started, or restarted, where this process has not started
+ it from this version of the row. A bridge already started from this
+ row is left alone, running or not: one that crashed or failed to
+ start stays down until it is edited or restarted, as it does in a
+ deployment of one process.
+ """
+ if self._owned.stopping:
+ return
+ if bridge is None or bridge.status != "active":
+ if self._owned.wants(bridge_id) or bridge_id in self._bridges:
+ logger.info(
+ "Collaboration bridge %s was removed elsewhere; stopping it",
+ bridge_id,
+ )
+ await self._owned.drop(bridge_id)
+ await self.stop(bridge_id)
+ return
+ if not self._owned.wants(bridge_id):
+ # New here: which process holds it is settled first, so it starts
+ # once, in the form it runs in.
+ self._owned.want(bridge_id)
+ await self._owned.settle()
+ if bridge_id in self._quiet_starts:
+ # Starting in the background, from the row as it reads then.
+ return
+ started_from = self._started_rows.get(bridge_id)
+ if started_from == _started_from(bridge):
+ return
+ if started_from is None:
+ logger.info(
+ "Collaboration bridge %s was added elsewhere; starting it", bridge_id
+ )
+ else:
+ logger.info(
+ "Collaboration bridge %s was edited elsewhere; restarting it",
+ bridge_id,
+ )
+ await self.stop(bridge_id, reason="restart")
+ await self.start(bridge_id)
+
+ def rooms_changed(self, bridge_id: str) -> None:
+ """A room was bridged to `bridge_id` or unbridged from it, and the
+ change has committed: every other process maps the bridge's rooms
+ again, so none goes on carrying a channel into a room it no longer
+ serves. Called by whoever made the change, after it has mapped its own
+ copy."""
+ if self._signals is not None:
+ self._signals.rooms_changed(bridge_id)
+
+ async def reload_rooms(self, bridge_id: str) -> None:
+ """Map the rooms of one bridge running here as their rows say."""
+ bridge = self._bridges.get(bridge_id)
+ if bridge is not None:
+ await bridge.reload_channel_map()
+
+ async def reload_all_rooms(self) -> None:
+ """`reload_rooms` for every bridge running here."""
+ for bridge_id in sorted(self._bridges):
+ try:
+ await self.reload_rooms(bridge_id)
+ except Exception:
+ logger.warning(
+ "Could not map collaboration bridge %s's rooms again; "
+ "retrying at the next re-check",
+ bridge_id,
+ exc_info=True,
+ )
+
+ def agents_changed(self, tenant_id: str) -> None:
+ """Agents were registered or deleted in `tenant_id`, maybe by another
+ process: every bridge of the tenant running here provisions their
+ platform identities again."""
+ for bridge in self.bridges_for_tenant(tenant_id):
+ bridge.agents_changed()
+
async def remove(self, bridge_id: str) -> None:
"""Disconnect a messaging app and take its identities with it.
@@ -1481,6 +1890,9 @@ class CollaborationBridgeLifecycleService:
"what it holds on its platform itself; removing it anyway",
bridge_id,
)
+ # Held here or not, let go of it: no process is to take it over. A
+ # quiet copy waiting to start finds it no longer wanted.
+ await self._owned.drop(bridge_id)
await self.stop(bridge_id)
async with self._session_factory() as session:
bridge = await self._bridge_store.get(session, bridge_id)
@@ -1518,6 +1930,7 @@ class CollaborationBridgeLifecycleService:
bridge_id,
len(removed),
)
+ self._announce(bridge_id)
if bridge is not None:
emit_safely(
@@ -1541,6 +1954,10 @@ class CollaborationBridgeLifecycleService:
self._preconfigured.discard(bridge_id)
self._connect_failures.pop(bridge_id, None)
self._connect_started.pop(bridge_id, None)
+ self._failed_take_overs.pop(bridge_id, None)
+ lock = self._bridge_locks.get(bridge_id)
+ if lock is not None and not lock.locked():
+ del self._bridge_locks[bridge_id]
def get(self, bridge_id: str) -> CollaborationCore | None:
"""The running bridge for ``bridge_id``, if it is the bound tenant's.
diff --git a/core/switch_core/bridges/collaboration/mattermost/adapter.py b/core/switch_core/bridges/collaboration/mattermost/adapter.py
index aa46cb4e7..97b302113 100644
--- a/core/switch_core/bridges/collaboration/mattermost/adapter.py
+++ b/core/switch_core/bridges/collaboration/mattermost/adapter.py
@@ -6,7 +6,6 @@ import json
import logging
import re
import threading
-import time
import uuid
from collections import OrderedDict
from collections.abc import Awaitable, Callable
@@ -302,6 +301,11 @@ def _ephemeral(text: str) -> dict[str, Any]:
class MattermostAdapter(PlatformAdapter):
display_name: ClassVar[str] = "Mattermost"
docs_slug: ClassVar[str | None] = "mattermost"
+ # A quiet Mattermost bridge opens no websocket and mints no bot token: it
+ # knows each agent's bot by id, for mentions, and calls the API as the
+ # admin. Mattermost sends every event to every open socket, so a socket
+ # in each process would hand each process every post.
+ keeps_quiet_unless_listening: ClassVar[bool] = True
@classmethod
def classify_failure(cls, exc: BaseException) -> FailureReason | None:
@@ -408,6 +412,13 @@ class MattermostAdapter(PlatformAdapter):
self._agent_bots: dict[str, dict[str, str]] = {}
self._bot_drivers: dict[str, Driver] = {}
+ # Set to close an agent bot's socket for good: when the adapter stops,
+ # or the agent's identity is removed. Each socket runs in a thread of
+ # its own that reconnects whenever the connection drops, so without
+ # this a stopped bridge's sockets stayed open for the life of the
+ # process, handing it every post. A bridge handed from one process to
+ # another stops here and starts there.
+ self._ws_stops: dict[str, threading.Event] = {}
self._identity_locks: dict[str, asyncio.Lock] = {}
self._bot_id_to_username: dict[str, str] = {}
self._bridge_bot_ids: set[str] = set()
@@ -510,10 +521,23 @@ class MattermostAdapter(PlatformAdapter):
)
async def stop(self) -> None:
+ for agent_name in list(self._ws_stops):
+ self._close_bot_socket(agent_name)
self._admin_driver = None
self._bot_drivers.clear()
logger.info("Mattermost adapter stopped")
+ def _close_bot_socket(self, agent_name: str) -> None:
+ """Close an agent bot's socket and keep its thread from reconnecting.
+ The thread finishes once the driver notices, within its read timeout,
+ and hands nothing over meanwhile."""
+ stopped = self._ws_stops.pop(agent_name, None)
+ if stopped is not None:
+ stopped.set()
+ driver = self._bot_drivers.get(agent_name)
+ if driver is not None and getattr(driver, "websocket", None) is not None:
+ driver.disconnect()
+
# ── Callbacks ────────────────────────────────────────────────────────────
@property
@@ -2090,7 +2114,7 @@ class MattermostAdapter(PlatformAdapter):
existing = await self._find_existing_bot(agent_name)
if existing:
bot_id: str = str(existing["user_id"])
- if existing.get("display_name") != label:
+ if existing.get("display_name") != label and self._listening:
try:
await self._mm_api(
"put", f"/bots/{bot_id}", {"display_name": label}
@@ -2114,10 +2138,29 @@ class MattermostAdapter(PlatformAdapter):
)
bot_id = str(bot["user_id"])
except Exception as e:
- logger.exception(
- "Failed to create Mattermost bot %s: %s", agent_name, e
- )
- return
+ # Another switch-core process may have created it a moment ago:
+ # every process provisions its copy of the bridge.
+ created = await self._find_existing_bot(agent_name)
+ if created is None:
+ logger.exception(
+ "Failed to create Mattermost bot %s: %s", agent_name, e
+ )
+ return
+ bot_id = str(created["user_id"])
+
+ if not self._listening:
+ # Known by id, for mentions and for telling its posts from people's.
+ # The process holding the bridge posts as the bot and listens on
+ # its socket.
+ self._agent_bots[agent_name] = {
+ "bot_id": bot_id,
+ "user_id": bot_id,
+ "username": agent_name,
+ }
+ self._bridge_bot_ids.add(bot_id)
+ self._bot_id_to_username[bot_id] = agent_name
+ logger.info("Found Mattermost bot identity %s (quiet)", agent_name)
+ return
try:
token_resp = await self._mm_api(
@@ -2155,25 +2198,33 @@ class MattermostAdapter(PlatformAdapter):
except Exception:
pass
+ stopped = threading.Event()
+ self._ws_stops[agent_name] = stopped
+
def _run_ws(driver: Driver = bot_driver, name: str = agent_name) -> None:
async def _handler(event_data: str) -> None:
+ if stopped.is_set():
+ return
await self._ws_handler(event_data, name)
- time.sleep(1)
+ if stopped.wait(1):
+ return
backoff = 2
- while True:
+ while not stopped.is_set():
try:
ws_loop = asyncio.new_event_loop()
asyncio.set_event_loop(ws_loop)
ws_loop.run_until_complete(driver.init_websocket(_handler))
except Exception as e:
+ if stopped.is_set():
+ break
logger.debug(
"WebSocket for %s disconnected: %s, retrying in %ds",
name,
e,
backoff,
)
- time.sleep(backoff)
+ stopped.wait(backoff)
backoff = min(backoff * 2, 30)
else:
backoff = 2
@@ -2253,6 +2304,10 @@ class MattermostAdapter(PlatformAdapter):
existing = await self._find_existing_bot(username)
if existing:
bot_id = str(existing["user_id"])
+ elif not self._listening:
+ # Created by the process holding the bridge; this one posts no
+ # notices.
+ return
else:
try:
bot = await self._mm_api(
@@ -2268,6 +2323,10 @@ class MattermostAdapter(PlatformAdapter):
except Exception:
logger.exception("Failed to create Switch Admin bot")
return
+ if not self._listening:
+ self._admin_bot_id = bot_id
+ self._bridge_bot_ids.add(bot_id)
+ return
try:
token_resp = await self._mm_api(
"post",
@@ -2320,6 +2379,7 @@ class MattermostAdapter(PlatformAdapter):
pass
async def remove_agent_identity(self, agent_name: str) -> None:
+ self._close_bot_socket(agent_name)
self._agent_bots.pop(agent_name, None)
self._bot_drivers.pop(agent_name, None)
diff --git a/core/switch_core/bridges/collaboration/slack/adapter.py b/core/switch_core/bridges/collaboration/slack/adapter.py
index 0c1d39951..9b9366aae 100644
--- a/core/switch_core/bridges/collaboration/slack/adapter.py
+++ b/core/switch_core/bridges/collaboration/slack/adapter.py
@@ -94,6 +94,11 @@ _SLACK_MASS_MENTION_WORD = re.compile(
# reload can tell ours apart from the workspace's own groups.
_AGENT_GROUP_MARKER = "Switch agent — "
+# How long an event from Socket Mode may take to be stored before it is
+# acknowledged anyway. Slack sends an event again unless it is acknowledged
+# within three seconds; the rest is left for the acknowledgement to arrive.
+SOCKET_ACK_WITHIN_SECONDS = 2.0
+
# User group calls are rate-limited at roughly 20/minute, and the Slack client
# retries connection errors but not throttling.
_RATE_LIMIT_MAX_ATTEMPTS = 5
@@ -384,6 +389,9 @@ def _slack_failure_reason(exc: SlackApiError) -> FailureReason:
class SlackAdapter(PlatformAdapter):
display_name: ClassVar[str] = "Slack"
docs_slug: ClassVar[str | None] = "slack"
+ # A quiet Slack bridge opens no Socket Mode connection. One over the
+ # Events API opens none anywhere: Slack posts each event to one process.
+ keeps_quiet_unless_listening: ClassVar[bool] = True
@classmethod
def classify_failure(cls, exc: BaseException) -> FailureReason | None:
@@ -584,6 +592,13 @@ class SlackAdapter(PlatformAdapter):
"Slack adapter listening over HTTP; events arrive at the "
"messaging install endpoint rather than over Socket Mode"
)
+ elif not self._listening:
+ # Another switch-core process holds the bridge and keeps its
+ # Socket Mode connection. This one only calls the Web API.
+ logger.info(
+ "Slack adapter running quietly: another process holds this "
+ "bridge's Socket Mode connection"
+ )
else:
# Narrowing for the type checker; the config validator is what
# actually guarantees it.
@@ -2751,10 +2766,44 @@ class SlackAdapter(PlatformAdapter):
async def _handle_socket_event(
self, client: SocketModeClient, req: SocketModeRequest
) -> None:
- await client.send_socket_mode_response(
- SocketModeResponse(envelope_id=req.envelope_id)
+ """Handle one Socket Mode envelope and acknowledge it.
+
+ An event (a message, someone joining a channel) is stored first and
+ acknowledged after. Slack sends an event again when it is not
+ acknowledged within three seconds, so a bridge whose process stops in
+ between is sent the event again, by then on the process that holds the
+ bridge next, instead of losing it; that copy finds the post's rows
+ written and writes nothing (`collaboration_core._post_source`). Storing
+ that takes longer than `SOCKET_ACK_WITHIN_SECONDS` is acknowledged at
+ that point and finishes after, so Slack does not send again an event
+ still being stored. Storing that fails within it is not acknowledged,
+ so Slack sends the event again.
+
+ A slash command or a button press is acknowledged first, as before:
+ Slack never sends one again, and the person waits on the answer.
+ """
+ if req.type != "events_api":
+ await client.send_socket_mode_response(
+ SocketModeResponse(envelope_id=req.envelope_id)
+ )
+ await self.dispatch_event(envelope_type=req.type, payload=req.payload)
+ return
+ dispatch = asyncio.ensure_future(
+ self.dispatch_event(envelope_type=req.type, payload=req.payload)
)
- await self.dispatch_event(envelope_type=req.type, payload=req.payload)
+ try:
+ await asyncio.wait({dispatch}, timeout=SOCKET_ACK_WITHIN_SECONDS)
+ failed = (
+ dispatch.done()
+ and not dispatch.cancelled()
+ and dispatch.exception() is not None
+ )
+ if not failed:
+ await client.send_socket_mode_response(
+ SocketModeResponse(envelope_id=req.envelope_id)
+ )
+ finally:
+ await dispatch
async def dispatch_event(
self, *, envelope_type: str, payload: dict[str, Any]
@@ -2768,9 +2817,10 @@ class SlackAdapter(PlatformAdapter):
modes would drift into behaving differently for the same event.
Acknowledgement is *not* here, because the two transports acknowledge
- incompatibly: Socket Mode replies on the socket before dispatching,
- while HTTP acknowledges by returning 200 to the post. Both must do it
- promptly and neither can do it for the other.
+ incompatibly: Socket Mode replies on the socket once an event is
+ stored (`_handle_socket_event`), while HTTP acknowledges by returning
+ 200 to the post. Both must do it promptly and neither can do it for
+ the other.
"""
if envelope_type == "slash_commands":
await self._handle_slash_command(payload)
@@ -2925,7 +2975,24 @@ class SlackAdapter(PlatformAdapter):
self._seen_ts[message_ts] = None
if len(self._seen_ts) > self._seen_ts_max:
self._seen_ts.popitem(last=False)
+ try:
+ await self._deliver_message_event(event, channel_id, message_ts)
+ except BaseException:
+ # Not seen after all. An event whose handling raises within the
+ # acknowledgement window is left unacknowledged
+ # (`_handle_socket_event`), and the copy Slack sends comes back to
+ # this adapter, since only the process holding the bridge has its
+ # socket. Skipped as seen, the copy would be acknowledged and the
+ # event lost. It writes only what this attempt did not
+ # (`collaboration_core._post_source`).
+ self._seen_ts.pop(message_ts, None)
+ raise
+ async def _deliver_message_event(
+ self, event: dict[str, object], channel_id: str, message_ts: str
+ ) -> None:
+ """Route a post `_handle_message_event` let through to the command or
+ message callback."""
user_id = str(event.get("user", ""))
text = str(event.get("text", ""))
if not text.strip():
diff --git a/core/switch_core/bridges/collaboration/telegram/adapter.py b/core/switch_core/bridges/collaboration/telegram/adapter.py
index 91c68a3ed..78e4690f2 100644
--- a/core/switch_core/bridges/collaboration/telegram/adapter.py
+++ b/core/switch_core/bridges/collaboration/telegram/adapter.py
@@ -524,6 +524,11 @@ class TelegramAdapter(PlatformAdapter):
display_name: ClassVar[str] = "Telegram"
docs_slug: ClassVar[str | None] = "telegram"
+ # A quiet self-registered bridge does not poll: Telegram hands each update
+ # to a single getUpdates caller, so a poller in every process would split
+ # a chat's messages between them. A bridge on the shared app bot polls
+ # nowhere; its updates reach any process as webhooks.
+ keeps_quiet_unless_listening: ClassVar[bool] = True
@classmethod
def classify_failure(cls, exc: BaseException) -> FailureReason | None:
@@ -589,6 +594,9 @@ class TelegramAdapter(PlatformAdapter):
self._config = config
self._app: Application | None = None # type: ignore[type-arg]
self._bot: Any = None
+ # Whether `_app` was started and polls, which only the process holding
+ # the bridge does; a quiet one only initialises it to call the Bot API.
+ self._polling = False
self._bot_user_id: int = 0
self._bot_username = (config.bot_username or "").lstrip("@")
# A shared bridge polls nothing and is handed the deployment's bot by
@@ -715,12 +723,20 @@ class TelegramAdapter(PlatformAdapter):
"Bot Settings -> Allow Groups?",
self._bot_username,
)
+ if not self._listening:
+ logger.info(
+ "Telegram adapter for @%s running quietly: another process holds "
+ "this bridge and polls for it",
+ self._bot_username,
+ )
+ return
await app.start()
if app.updater is None:
raise RuntimeError("Telegram application was built without an updater")
await app.updater.start_polling(
allowed_updates=_ALLOWED_UPDATES, error_callback=self._on_polling_error
)
+ self._polling = True
logger.info("Telegram adapter connected as @%s (id %s)", me.username, me.id)
await self._publish_command_menu()
@@ -1007,10 +1023,11 @@ class TelegramAdapter(PlatformAdapter):
async def stop(self) -> None:
app = self._app
self._app = None
+ polling, self._polling = self._polling, False
if app is not None:
for shutdown in (
- app.updater.stop if app.updater else None,
- app.stop,
+ app.updater.stop if app.updater and polling else None,
+ app.stop if polling else None,
app.shutdown,
):
if shutdown is None:
diff --git a/core/switch_core/budgets.py b/core/switch_core/budgets.py
index 58bc94938..c9739583a 100644
--- a/core/switch_core/budgets.py
+++ b/core/switch_core/budgets.py
@@ -13,9 +13,21 @@ check stops the next piece of work, not the one in flight, and a budget can be
overshot by whatever was already running when it was reached.
"""
+from typing import Protocol
+
from sqlalchemy.ext.asyncio import AsyncSession
-from switch_core.db.stores.budget_store import BudgetStanding, BudgetStore
+from switch_core.db.stores.budget_store import BudgetStanding
+
+
+class BudgetStandings(Protocol):
+ """Where the guard reads budgets from: `BudgetStore`, or the router's
+ reads for one row, which ask for a tenant's budgets once and answer every
+ agent in the room from them."""
+
+ async def standings(
+ self, session: AsyncSession, *, tenant_id: str, agent_id: str | None
+ ) -> list[BudgetStanding]: ...
def _describe(standing: BudgetStanding) -> str:
@@ -43,7 +55,7 @@ class BudgetExceeded(Exception):
class BudgetGuard:
- def __init__(self, store: BudgetStore) -> None:
+ def __init__(self, store: BudgetStandings) -> None:
self._store = store
async def require_within(
diff --git a/core/switch_core/chats/service.py b/core/switch_core/chats/service.py
index e204924f1..3ab8a75bd 100644
--- a/core/switch_core/chats/service.py
+++ b/core/switch_core/chats/service.py
@@ -27,7 +27,7 @@ import weakref
from collections.abc import Callable
from dataclasses import dataclass
from datetime import UTC, datetime
-from typing import Literal
+from typing import TYPE_CHECKING, Literal
import markdown
from sqlalchemy import (
@@ -99,6 +99,9 @@ from switch_core.tenant_context import tenant_scope
from switch_core.transport.content import media_content, message_content
from switch_core.transport.types import MessageFormat
+if TYPE_CHECKING:
+ from switch_core.delivery.router import MessageRouter, Routed
+
logger = logging.getLogger(__name__)
# Fixed so the ids derived from it are stable across processes and releases.
@@ -153,6 +156,7 @@ class ChatService:
agent_store: AgentStore,
user_store: UserStore,
message_store: MessageStore,
+ router: MessageRouter,
media_store: MediaStore,
usage_store: UsageStore,
id_server_name: str,
@@ -173,6 +177,7 @@ class ChatService:
self._agent_store = agent_store
self._user_store = user_store
self._message_store = message_store
+ self._router = router
self._media_store = media_store
self._usage_store = usage_store
self._id_server_name = id_server_name
@@ -1110,6 +1115,7 @@ class ChatService:
),
)
posted: list[Message] = []
+ routed: list[Routed] = []
for index, content in enumerate(contents):
message = message_row(
room_id=room.id,
@@ -1121,11 +1127,17 @@ class ChatService:
content=content,
client_txn_id=f"{request_id}:{index}",
)
- posted.append(
- await self._message_store.create(
- session, message, attachments_in(content)
+ routed.append(
+ await self._router.store(
+ session,
+ tenant_id=tenant_id,
+ room_id=room.id,
+ transport_room_id=room.transport_room_id,
+ message=message,
+ attachments=attachments_in(content),
)
)
+ posted.append(message)
await self._usage_store.record(
session,
tenant_id=tenant_id,
@@ -1144,6 +1156,8 @@ class ChatService:
.values(result={"messageIds": [m.transport_event_id for m in posted]})
)
await session.commit()
+ for each in routed:
+ each.committed()
for content in contents:
metrics().increment(
MESSAGES_SENT,
@@ -1215,9 +1229,12 @@ class ChatService:
return str((operation.result or {})["commandId"])
command_id = new_event_id()
- await self._message_store.create(
+ routed = await self._router.store(
session,
- message_row(
+ tenant_id=tenant_id,
+ room_id=room.id,
+ transport_room_id=room.transport_room_id,
+ message=message_row(
room_id=room.id,
event_id=command_id,
sender_id=client.transport_user_id,
@@ -1232,7 +1249,7 @@ class ChatService:
},
client_txn_id=request_id,
),
- [],
+ attachments=[],
)
await session.execute(
update(ChatOperation)
@@ -1244,6 +1261,7 @@ class ChatService:
.values(result={"commandId": command_id})
)
await session.commit()
+ routed.committed()
return command_id
async def warm(
diff --git a/core/switch_core/clients/actor.py b/core/switch_core/clients/actor.py
index fb239c6e5..472600ed6 100644
--- a/core/switch_core/clients/actor.py
+++ b/core/switch_core/clients/actor.py
@@ -23,6 +23,8 @@ from switch_core.clients.admin_messages import (
OnBehalfOf,
admin_extra_content,
)
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.client_store import ClientStore
from switch_core.transport import MessageTransport, TransportError
@@ -208,9 +210,12 @@ class Actor[ConfigT: ClientConfig]:
self.mark_joined(room_id, self._connected_at)
return True
- async def join_room(self, room_id: str) -> None:
+ async def join_room(self, room_id: str, joined_at_ms: int) -> None:
+ """Join, and filter the room's rows from `joined_at_ms` on: when the
+ invitation came, or, for a membership another process wrote, the
+ membership row's own time."""
if await self._transport.join_room(room_id):
- self.mark_joined(room_id, int(time.time() * 1000))
+ self.mark_joined(room_id, joined_at_ms)
logger.info("Actor %s joined %s", self.transport_user_id, room_id)
# ── Writing ────────────────────────────────────────────────────────────────
@@ -329,8 +334,9 @@ class AgentActor(Actor[ClientConfig]):
"""An agent, as a member of Switch rooms (client type `agent`).
Carries the agent row it stands for: the bridge writes the agent's replies
- through it, found by agent id. The row is loaded by the agent's consumer
- when it starts, and refreshed there when it may have been edited.
+ through it, found by agent id. An agent reads no room, so it has no
+ consumer: what is for it is decided as each row is written
+ (`delivery/router.py`). The row is loaded when the actor connects.
"""
role: ClassVar[ActorRole] = "agent"
@@ -339,10 +345,40 @@ class AgentActor(Actor[ClientConfig]):
super().__init__(**kwargs)
self._agent: Agent | None = None
+ async def connect(self) -> None:
+ """Load the agent this actor is, then open the transport.
+
+ This runs at the top of the client's own task, which binds nothing, so
+ the tenant is carried rather than inherited: it comes from the
+ `clients` row this actor was built from. A guessed tenant would turn
+ "this client is an agent" into "no agent found for client" and fail
+ the client at boot over a row that exists, which is why nothing here
+ falls back to whatever is ambient.
+ """
+ if self._agent is None:
+ async with tenant_session(self.session_factory, self.tenant_id) as session:
+ agent = await AgentStore().get_by_client_id(session, self.client_id)
+ if agent is None:
+ raise RuntimeError(f"No agent found for client: {self.client_id}")
+ self._agent = agent
+ await super().connect()
+
+ def preload_agent(self, agent: Agent) -> None:
+ """Hand `connect` the agent row read in bulk at boot, so it needs no
+ query of its own."""
+ self._agent = agent
+
+ async def wait_receiving(self, timeout: float) -> bool:
+ """True at once. An agent reads no room: each row written to a room it
+ is in is decided for it as the row is written, whether or not its
+ client has started, so a room it is added to is one it hears from the
+ start (`delivery/router.py`)."""
+ return True
+
@property
def agent(self) -> Agent:
if self._agent is None:
- raise RuntimeError("Agent not loaded — its consumer has not started")
+ raise RuntimeError("Agent not loaded: the actor has not connected")
return self._agent
@property
diff --git a/core/switch_core/clients/admin_messages.py b/core/switch_core/clients/admin_messages.py
index 125ea3736..b2219fed4 100644
--- a/core/switch_core/clients/admin_messages.py
+++ b/core/switch_core/clients/admin_messages.py
@@ -5,7 +5,7 @@ from enum import StrEnum
from typing import NamedTuple
# Content flag stamped on the no-session / busy-elsewhere auto-reply (see
-# AgentConsumer.on_message). Its first job is to mark a message as itself an
+# AgentRecipient.decide_message). Its first job is to mark a message as itself an
# auto-reply so that another offline agent addressed by it does NOT emit a
# second auto-reply — two session-less agents tagging each other would
# otherwise ping-pong identical "no session" replies forever. It also lets the
diff --git a/core/switch_core/clients/client_factory.py b/core/switch_core/clients/client_factory.py
index 05d13c035..924e7d0e8 100644
--- a/core/switch_core/clients/client_factory.py
+++ b/core/switch_core/clients/client_factory.py
@@ -1,7 +1,7 @@
from __future__ import annotations
import logging
-from typing import Any
+from typing import TYPE_CHECKING, Any
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -22,6 +22,9 @@ from switch_core.transport.observer import ParticipantMessageObserver
from switch_core.transport.postgres import PostgresTransport
from switch_core.transport.room_cache import RoomDeliveryCache
+if TYPE_CHECKING:
+ from switch_core.delivery.router import MessageRouter
+
logger = logging.getLogger(__name__)
@@ -41,6 +44,7 @@ class ClientFactory:
ephemeral: EphemeralBus,
room_cache: RoomDeliveryCache,
message_observer: ParticipantMessageObserver,
+ router: MessageRouter,
) -> None:
self._client_store = client_store
self._session_factory = session_factory
@@ -55,6 +59,10 @@ class ClientFactory:
# One for the process, so every member of a room shares it.
self._room_cache = room_cache
self._message_observer = message_observer
+ # Every transport writes through it, and it posts agents' replies
+ # through transports built here, so it is handed how to build one.
+ self._router = router
+ router.use_transports(self.transport_for)
self._registry: dict[
str,
tuple[
@@ -80,15 +88,22 @@ class ClientFactory:
"""
self._registry[client_type] = (actor_cls, consumer_cls, consumer_kwargs)
- def create(
- self, record: Client
- ) -> tuple[Actor[ClientConfig], Consumer[Any] | None]:
+ def has_consumer(self, client_type: str) -> bool:
+ """Whether a client of this type has a consumer, which runs in one
+ process only, the one that owns the client (`ownership.py`)."""
+ entry = self._registry.get(client_type)
+ if entry is None:
+ raise ValueError(f"Unknown client type: {client_type!r}")
+ return entry[1] is not None
+
+ def create_actor(self, record: Client) -> Actor[ClientConfig]:
+ """The actor a `clients` row becomes."""
entry = self._registry.get(record.type)
if entry is None:
raise ValueError(f"Unknown client type: {record.type!r}")
- actor_cls, consumer_cls, consumer_kwargs = entry
+ actor_cls = entry[0]
config = actor_cls.config_class.model_validate(record.config or {})
- actor = actor_cls(
+ return actor_cls(
client_id=record.id,
# Straight off the row this client *is*. `create_client` writes
# that row inside a session with the tenant bound and the factory
@@ -103,9 +118,19 @@ class ClientFactory:
config=config,
transport_factory=self.transport_for,
)
+
+ def create_consumer(
+ self, client_type: str, actor: Actor[ClientConfig]
+ ) -> Consumer[Any] | None:
+ """A fresh consumer around `actor`, or None for a type that reads
+ nothing."""
+ entry = self._registry.get(client_type)
+ if entry is None:
+ raise ValueError(f"Unknown client type: {client_type!r}")
+ _, consumer_cls, consumer_kwargs = entry
if consumer_cls is None:
- return actor, None
- return actor, consumer_cls(actor=actor, **consumer_kwargs)
+ return None
+ return consumer_cls(actor=actor, **consumer_kwargs)
def transport_for(self, client: Actor[Any]) -> MessageTransport:
"""The transport every client in the process runs on.
@@ -131,4 +156,5 @@ class ClientFactory:
ephemeral=self._ephemeral,
room_cache=self._room_cache,
message_observer=self._message_observer,
+ router=self._router,
)
diff --git a/core/switch_core/clients/client_lifecycle_service.py b/core/switch_core/clients/client_lifecycle_service.py
index 9de330a73..0fc69c206 100644
--- a/core/switch_core/clients/client_lifecycle_service.py
+++ b/core/switch_core/clients/client_lifecycle_service.py
@@ -1,10 +1,22 @@
+"""Every client running in this process, and the consumers it reads with.
+
+Every switch-core process runs an actor for each client (`clients/actor.py`),
+because any process may be asked to write as any client: the admin answering
+in a room created there, an agent posting through the API. A consumer is
+different when what it does must happen once: the admin's `CommandConsumer`
+answers commands and posts warnings, so it runs in the one process that owns
+its client (`ownership.py`), reading from the client's stored cursors and
+stopped when the ownership moves. Every consumer runs that way.
+"""
+
from __future__ import annotations
import asyncio
import logging
import random
import uuid
-from typing import Any, Protocol, runtime_checkable
+from collections.abc import Callable
+from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
from sqlalchemy.exc import InterfaceError, OperationalError
from sqlalchemy.exc import TimeoutError as PoolTimeoutError
@@ -19,11 +31,21 @@ from switch_core.db.models import Agent, Client, Tenant
from switch_core.db.session_scope import tenant_session
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.client_store import ClientStore
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
from switch_core.db.stores.tenant_store import TenantStore
from switch_core.db.tenant_lookup import all_tenant_ids
from switch_core.logging_context import log_context
from switch_core.provisioning import Provisioning
from switch_core.tenant_context import no_tenant, tenant_scope
+from switch_core.transport.cursors import ConsumerCursors, CursorWriter
+
+if TYPE_CHECKING:
+ from switch_core.clients.client_signals import ClientSignals
+ from switch_core.ownership import Fence, Ownership
+
+# The kind of owned work a client's consumer is, where only one process may
+# run it (`ownership.py`), keyed by client id.
+CLIENT_CONSUMERS = "client"
logger = logging.getLogger(__name__)
@@ -50,7 +72,8 @@ CLIENT_RETRY_CAP_SECONDS = 60.0
@runtime_checkable
class _PreloadsAgent(Protocol):
- """A consumer that can start from an agent row read for it in bulk."""
+ """An agent's actor, which can connect with an agent row read for it in
+ bulk rather than reading its own."""
def preload_agent(self, agent: Agent) -> None: ...
@@ -70,8 +93,12 @@ class ClientLifecycleService:
session_factory: async_sessionmaker[AsyncSession],
config: SwitchConfig,
tenants_isolated: bool,
+ ownership: Ownership,
+ cursor_writer: CursorWriter,
) -> None:
self._provisioning = provisioning
+ self._cursor_writer = cursor_writer
+ self._cursor_store = DeliveryCursorStore()
self._tenants_isolated = tenants_isolated
self._client_store = client_store
self._tenant_store = tenant_store
@@ -91,6 +118,53 @@ class ClientLifecycleService:
# wants without reaching into the objects. See `get_by_type`.
self._client_tenants: dict[str, str] = {}
self._tasks: dict[str, asyncio.Task[None]] = {}
+ # The read loops of the consumers this process owns, by client id.
+ # Their actors' tasks are in `_tasks` like every other.
+ self._owned_tasks: dict[str, asyncio.Task[None]] = {}
+ # Clients stopped here on purpose (an agent being deleted stops its
+ # client before its row goes) or whose task crashed on a bug.
+ # `reconcile_tenant` leaves them stopped, as a crash always has, until
+ # something starts them on purpose; otherwise a re-check between a
+ # deletion's stop and its commit would start the client again. A
+ # deletion that does not commit lets go of its client
+ # (`deletion_failed`). Keyed by client id, to the client's tenant.
+ self._left_stopped: dict[str, str] = {}
+ self._signals: ClientSignals | None = None
+ self._owned = ownership.kind(
+ CLIENT_CONSUMERS,
+ start=self._start_owned_consumer,
+ stop=self._stop_owned_consumer,
+ )
+ # Told the tenant whenever a re-read starts or stops an agent client
+ # here, which is how an agent registered or deleted in another process
+ # reaches what keeps per-agent state of its own (a bridge's platform
+ # identities, `CollaborationBridgeLifecycleService.agents_changed`).
+ self._agents_changed_listeners: list[Callable[[str], None]] = []
+
+ def add_agents_changed_listener(self, listener: Callable[[str], None]) -> None:
+ self._agents_changed_listeners.append(listener)
+
+ def _agents_changed(self, tenant_id: str) -> None:
+ for listener in self._agents_changed_listeners:
+ try:
+ listener(tenant_id)
+ except Exception:
+ logger.exception(
+ "Telling a listener that tenant %s's agents changed failed",
+ tenant_id,
+ )
+
+ def use_signals(self, signals: ClientSignals) -> None:
+ """Where a client started or deleted here is announced, so every
+ other process starts or stops its own copy (`client_signals.py`)."""
+ self._signals = signals
+
+ def announce(self, tenant_id: str) -> None:
+ """A client of `tenant_id` was registered or deleted, and has
+ committed: every process reconciles its copies of the tenant's
+ clients with the table."""
+ if self._signals is not None:
+ self._signals.changed(tenant_id)
@property
def tenants_isolated(self) -> bool:
@@ -205,6 +279,8 @@ class ClientLifecycleService:
slug,
)
raise
+ # The admin client exists only as a row until something starts it.
+ self.announce(tenant.id)
return tenant
async def start_all(self) -> None:
@@ -248,13 +324,93 @@ class ClientLifecycleService:
logger.info("Starting %d clients", len(records))
for record in records:
- self._register(record)
+ actor = self._register(record)
# Before the task first runs: `_register` only schedules it, and
# nothing here yields until the loop is done.
- consumer = self._consumers.get(record.id)
agent = agents.get(record.id)
- if agent is not None and isinstance(consumer, _PreloadsAgent):
- consumer.preload_agent(agent)
+ if agent is not None and isinstance(actor, _PreloadsAgent):
+ actor.preload_agent(agent)
+
+ async def reconcile_all(self) -> None:
+ """`reconcile_tenant` for every tenant."""
+ for tenant_id in await all_tenant_ids(self._session_factory):
+ await self.reconcile_tenant(tenant_id)
+
+ async def reconcile_tenant(self, tenant_id: str) -> None:
+ """Run here exactly the tenant's clients that `start_all` would run.
+
+ Every process starts every non-collaboration client at boot, and a
+ client registered later used to start only in the process that served
+ the registration, while a deleted one kept running everywhere else
+ until a restart. This re-reads the tenant's rows: a client with no copy
+ here is started, and a copy whose row is gone is stopped.
+
+ An agent's client row commits before its agent row, so one without an
+ agent row yet is left alone: its actor would find no agent and crash.
+ The registration's own signal comes after both have committed.
+ Collaboration clients (bridges and the people they speak for) are the
+ bridges' to start and stop, and a client that crashed stays stopped.
+ """
+ # Only a client already running before the read can be judged deleted
+ # by it. One registered in this process while the read awaits has a
+ # row the read may not have seen, and stopping it would put it in
+ # `_left_stopped`, where later re-checks leave it.
+ running_before = [
+ client_id
+ for client_id, client_tenant in self._client_tenants.items()
+ if client_tenant == tenant_id
+ and self._client_types.get(client_id) not in self.COLLAB_CLIENT_TYPES
+ ]
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ records = [
+ record
+ for record in await self._client_store.get_all(session)
+ if record.tenant_id == tenant_id
+ and record.type not in self.COLLAB_CLIENT_TYPES
+ ]
+ present = {record.id for record in records}
+ # A client whose row is gone will not be started again: nothing is
+ # left to hold back.
+ self._left_stopped = {
+ client_id: client_tenant
+ for client_id, client_tenant in self._left_stopped.items()
+ if client_tenant != tenant_id or client_id in present
+ }
+ agents_changed = False
+ for client_id in running_before:
+ if client_id in self._client_tenants and client_id not in present:
+ logger.info("Client %s was deleted elsewhere; stopping it", client_id)
+ agents_changed |= self._client_types.get(client_id) == "agent"
+ await self.stop(client_id)
+ missing = [
+ record
+ for record in records
+ if record.id not in self._clients and record.id not in self._left_stopped
+ ]
+ if not missing:
+ if agents_changed:
+ self._agents_changed(tenant_id)
+ return
+ agents = await self._agents_by_client_id(missing)
+ for record in missing:
+ agent = agents.get(record.id)
+ if record.type == "agent" and agent is None:
+ continue
+ # Checked again: `stop` above and the agent read both yield, and a
+ # registration in this process may have started it meanwhile.
+ if record.id in self._clients:
+ continue
+ logger.info(
+ "Client %s (%s) was added elsewhere; starting it",
+ record.display_name,
+ record.transport_user_id,
+ )
+ actor = self._register(record)
+ if agent is not None and isinstance(actor, _PreloadsAgent):
+ actor.preload_agent(agent)
+ agents_changed |= agent is not None
+ if agents_changed:
+ self._agents_changed(tenant_id)
async def _agents_by_client_id(self, records: list[Client]) -> dict[str, Agent]:
"""The agent row behind each client, one query per tenant.
@@ -307,16 +463,23 @@ class ClientLifecycleService:
)
return actor
+ def _leave_stopped(self, client_id: str) -> None:
+ tenant_id = self._client_tenants.get(client_id)
+ if tenant_id is not None:
+ self._left_stopped[client_id] = tenant_id
+
def _register(self, record: Client) -> Actor[ClientConfig]:
- actor, consumer = self._client_factory.create(record)
+ self._left_stopped.pop(record.id, None)
+ actor = self._client_factory.create_actor(record)
self._clients[record.id] = actor
- if consumer is not None:
- self._consumers[record.id] = consumer
- else:
- self._consumers.pop(record.id, None)
self._client_types[record.id] = record.type
self._client_tenants[record.id] = record.tenant_id
- self._start_task(record.id, actor, consumer)
+ # The actor here, and the consumer, if the type has one, where this
+ # process owns the client.
+ self._consumers.pop(record.id, None)
+ self._start_task(record.id, actor)
+ if self._client_factory.has_consumer(record.type):
+ self._owned.want(record.id)
return actor
async def create_and_start(
@@ -340,11 +503,16 @@ class ClientLifecycleService:
if actor is None:
logger.warning("Cannot stop unknown client %s", client_id)
return
+ if self._owned.wants(client_id):
+ # Its consumer, where this process owns it, and its lease with it.
+ await self._owned.drop(client_id)
+ await self._stop_owned_consumer(client_id)
consumer = self._consumers.pop(client_id, None)
if consumer is not None:
await consumer.stop()
else:
await actor.close()
+ self._leave_stopped(client_id)
task = self._cancel_task(client_id)
if task is not None:
await asyncio.gather(task, return_exceptions=True)
@@ -355,6 +523,10 @@ class ClientLifecycleService:
async def stop_all(self) -> None:
logger.info("Stopping all %d clients", len(self._clients))
+ # The ownership has stopped and released the consumers it held
+ # (`Ownership.stop`); whatever still runs here goes the same way.
+ for client_id in list(self._owned_tasks):
+ await self._stop_owned_consumer(client_id)
for client_id, actor in self._clients.items():
consumer = self._consumers.get(client_id)
if consumer is not None:
@@ -387,22 +559,45 @@ class ClientLifecycleService:
Stopping the running client is the half that cannot join a
transaction, so callers do that first. A rollback then leaves a
- stopped client whose row survives, which the next start repairs — the
- opposite order leaves an orphan row that nothing repairs.
+ stopped client whose row survives, which `deletion_failed` or the
+ next start repairs; the opposite order leaves an orphan row that
+ nothing repairs.
"""
await self._client_store.delete(session, client_id)
+ def deletion_failed(self, client_id: str) -> None:
+ """Let a client stopped for a deletion that did not commit run again.
+
+ Its row is still there, so the client still exists, but `stop` left
+ it in `_left_stopped`, where every re-check leaves it, so it would
+ stay stopped here until the process restarts. Announced rather than
+ started here: the re-read that follows starts the client if its row
+ is there, and nothing if the deletion committed after all and failed
+ only on the way back.
+ """
+ tenant_id = self._left_stopped.pop(client_id, None)
+ if tenant_id is not None:
+ self.announce(tenant_id)
+
def get(self, client_id: str) -> Actor[ClientConfig] | None:
return self._clients.get(client_id)
def running_count(self) -> int:
- """Consumers believed to be running right now: the read loops.
-
- A crashed consumer removes itself and its actor, so this falling is the
- only signal. Actors with no consumer have no loop to crash.
+ """Consumers believed to be running in this process right now: the
+ read loops.
+
+ A crashed consumer removes itself, so this falling is the only signal.
+ Actors with no consumer have no loop to crash, and that includes every
+ agent's: what is for an agent is decided as each row is written, so
+ nothing reads a room for it. A consumer another process owns is
+ counted there.
"""
return len(self._consumers)
+ def owns(self, client_id: str) -> bool:
+ """Whether this process holds the client's consumer lease."""
+ return self._owned.fence(client_id) is not None
+
def get_by_agent_id(self, agent_id: str) -> AgentActor | None:
for actor in self._clients.values():
if isinstance(actor, AgentActor) and actor.agent_id == agent_id:
@@ -432,12 +627,104 @@ class ClientLifecycleService:
and self._client_tenants.get(client_id) == tenant_id
]
- def _start_task(
- self,
- client_id: str,
- actor: Actor[ClientConfig],
- consumer: Consumer[Any] | None,
+ async def _start_owned_consumer(self, client_id: str, fence: Fence) -> None:
+ """This process now owns the client: run its consumer here, a fresh
+ one for this hold, reading from the client's stored cursors."""
+ actor = self._clients.get(client_id)
+ client_type = self._client_types.get(client_id)
+ if actor is None or client_type is None:
+ logger.warning(
+ "Owning client %s, which is not running here; its consumer "
+ "waits for it",
+ client_id,
+ )
+ return
+ consumer = self._client_factory.create_consumer(client_type, actor)
+ if consumer is None:
+ return
+ cursors = ConsumerCursors(
+ writer=self._cursor_writer,
+ store=self._cursor_store,
+ tenant_id=actor.tenant_id,
+ client_id=client_id,
+ fence=fence,
+ actor_role=actor.role,
+ on_lost=self._owned.lost,
+ )
+ self._consumers[client_id] = consumer
+ self._owned_tasks[client_id] = asyncio.create_task(
+ self._run_owned_consumer(client_id, consumer, cursors)
+ )
+
+ async def _stop_owned_consumer(self, client_id: str) -> None:
+ """Stop the client's consumer here after the row in hand, and store
+ how far it got, so the process that owns the client next starts where
+ it ended. A progress write that fails is logged: the next owner then
+ handles the last rows again, and writes their replies once."""
+ task = self._owned_tasks.pop(client_id, None)
+ if task is None:
+ return
+ consumer = self._consumers.pop(client_id, None)
+ if consumer is not None:
+ await consumer.stop()
+ await asyncio.gather(task, return_exceptions=True)
+ try:
+ await self._cursor_writer.flush()
+ except Exception:
+ logger.warning(
+ "Could not store client %s's progress as its consumer stopped; "
+ "whoever reads for it next handles its last rows again",
+ client_id,
+ exc_info=True,
+ )
+
+ async def _run_owned_consumer(
+ self, client_id: str, consumer: Consumer[Any], cursors: ConsumerCursors
) -> None:
+ """An owned consumer's read loop, ambient-free for the reasons
+ `_run_client` gives. A crash ends the consumer and leaves its actor:
+ the client is still written as here, and is read nowhere until its
+ ownership moves or the process restarts, as a crashed client always
+ was."""
+ actor = consumer.actor
+ with no_tenant(), log_context(room_id=None):
+ try:
+ attempt = 0
+ while True:
+ try:
+ await consumer.start(cursors)
+ return
+ except _TRANSIENT_START_ERRORS:
+ attempt += 1
+ delay = random.uniform(
+ 0,
+ min(
+ CLIENT_RETRY_BASE_SECONDS * 2 ** (attempt - 1),
+ CLIENT_RETRY_CAP_SECONDS,
+ ),
+ )
+ logger.warning(
+ "Client %s (%s) failed on the database or network "
+ "(attempt %d); retrying in %.0fs",
+ actor.display_name,
+ actor.transport_user_id,
+ attempt,
+ delay,
+ exc_info=True,
+ )
+ await asyncio.sleep(delay)
+ except Exception:
+ logger.exception(
+ "Client %s (%s) crashed",
+ actor.display_name,
+ actor.transport_user_id,
+ )
+ if self._consumers.get(client_id) is consumer:
+ del self._consumers[client_id]
+ if self._owned_tasks.get(client_id) is asyncio.current_task():
+ del self._owned_tasks[client_id]
+
+ def _start_task(self, client_id: str, actor: Actor[ClientConfig]) -> None:
# A client row is one running task. Anything that starts a second for
# the same row — a boot sweep reaching a client that registration
# already started, a bridge restarting its own — replaces the first,
@@ -447,7 +734,7 @@ class ClientLifecycleService:
# finally reaches it, it runs the teardown for both: the live client
# for that user goes deaf to invitations it never saw arrive.
self._cancel_task(client_id)
- task = asyncio.create_task(self._run_client(client_id, actor, consumer))
+ task = asyncio.create_task(self._run_client(client_id, actor))
self._tasks[client_id] = task
def _cancel_task(self, client_id: str) -> asyncio.Task[None] | None:
@@ -456,12 +743,7 @@ class ClientLifecycleService:
task.cancel()
return task
- async def _run_client(
- self,
- client_id: str,
- actor: Actor[ClientConfig],
- consumer: Consumer[Any] | None,
- ) -> None:
+ async def _run_client(self, client_id: str, actor: Actor[ClientConfig]) -> None:
"""A client's long-lived task, deliberately ambient-free.
`no_tenant` because a task keeps the context of whoever created it,
@@ -490,10 +772,7 @@ class ClientLifecycleService:
attempt = 0
while True:
try:
- if consumer is not None:
- await consumer.start()
- else:
- await actor.connect()
+ await actor.connect()
return
except _TRANSIENT_START_ERRORS:
attempt += 1
@@ -520,6 +799,7 @@ class ClientLifecycleService:
actor.display_name,
actor.transport_user_id,
)
+ self._leave_stopped(client_id)
self._clients.pop(client_id, None)
self._consumers.pop(client_id, None)
self._client_types.pop(client_id, None)
diff --git a/core/switch_core/clients/client_signals.py b/core/switch_core/clients/client_signals.py
new file mode 100644
index 000000000..7ae6452e7
--- /dev/null
+++ b/core/switch_core/clients/client_signals.py
@@ -0,0 +1,46 @@
+"""Clients registered or deleted after boot, reaching every process.
+
+Every switch-core process runs its own copy of each agent and admin client:
+`ClientLifecycleService.start_all` starts them all at boot. A client
+registered later started only in the process that served the registration, so
+any other process answered that agent's requests with "not running" and read
+nothing for it; a client deleted kept its copies running everywhere else, on a
+row that no longer exists.
+
+So a registration or deletion, once committed, announces `client.changed` for
+its tenant. Every process, the announcing one included, then re-reads the
+tenant's clients and starts or stops its copies to match
+(`ClientLifecycleService.reconcile_tenant`). The announcing process has
+already started or stopped its own, so for it the re-read is a check that
+changes nothing. A re-read that fails is kept and tried again, and every
+tenant is re-read when the signal connection comes back and on each periodic
+re-check, so a missed signal costs a delay rather than a client missing from
+a process until it restarts.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from switch_core.clients.client_lifecycle_service import ClientLifecycleService
+ from switch_core.signals import SignalBus
+
+CLIENT_CHANGED_SIGNAL = "client.changed"
+
+
+class ClientSignals:
+ def __init__(self, *, lifecycle: ClientLifecycleService, bus: SignalBus) -> None:
+ self._bus = bus
+ self._owed = bus.pending_work("clients", lifecycle.reconcile_tenant)
+ bus.handle(CLIENT_CHANGED_SIGNAL, self._on_changed)
+ bus.on_resync(lifecycle.reconcile_all)
+ bus.on_recheck(lifecycle.reconcile_all)
+ lifecycle.use_signals(self)
+
+ def changed(self, tenant_id: str) -> None:
+ self._owed.add(tenant_id)
+ self._bus.publish(CLIENT_CHANGED_SIGNAL, tenant_id=tenant_id)
+
+ def _on_changed(self, signal: dict[str, Any]) -> None:
+ self._owed.add(signal["tenant_id"])
diff --git a/core/switch_core/clients/consumer.py b/core/switch_core/clients/consumer.py
index a58569f74..40099e86f 100644
--- a/core/switch_core/clients/consumer.py
+++ b/core/switch_core/clients/consumer.py
@@ -1,16 +1,20 @@
"""Who reads a room: the delivery loop and the hooks it calls.
A consumer is built around an `Actor`. It reads the rooms that actor belongs
-to from its own cursor, filters what that actor should not see (its own
-messages, anything from before it joined), and hands the rest to the hooks a
-subclass overrides. When a hook needs to answer, it writes through the actor.
+to from where its progress starts each room, filters out the actor's own
+messages, and hands the rest to the hooks a subclass overrides. When a hook
+needs to answer, it writes through the actor.
+
+Where reading starts is the cursor, not a clock: a consumer that resumes from
+its stored cursors is handed what its client has not handled yet however old
+it is, and one that reads from each room's head is handed nothing from before
+it started. Neither is handed anything from before its client joined a room.
"""
from __future__ import annotations
import asyncio
import logging
-import time
from typing import TYPE_CHECKING, Any
from switch_core.events import (
@@ -41,6 +45,7 @@ if TYPE_CHECKING:
from switch_core.clients.actor import Actor
from switch_core.db.stores.client_store import ClientStore
+ from switch_core.transport.cursors import DeliveryProgress
logger = logging.getLogger(__name__)
@@ -55,11 +60,9 @@ class Consumer[ActorT: Actor[Any]]:
# Rooms this consumer has already announced its actor's arrival in.
# Distinct from the actor's `room_join_times`, which is membership
# bookkeeping recorded as early as possible (an explicit join, a
- # membership lookup) so `wait_joined` and the `_should_ignore` cutoff
- # are accurate. Membership being known is not evidence the arrival was
- # announced.
+ # membership lookup) so `wait_joined` is accurate. Membership being
+ # known is not evidence the arrival was announced.
self._self_join_dispatched: set[str] = set()
- self._startup_ts: int = 0
self._running: bool = False
# The identity a consumer reads for is its actor's. Read-only views, so
@@ -93,6 +96,11 @@ class Consumer[ActorT: Actor[Any]]:
def transport(self) -> MessageTransport | None:
return self.actor.transport
+ @property
+ def running(self) -> bool:
+ """Whether this consumer is reading: started and not told to stop."""
+ return self._running
+
@property
def _transport(self) -> MessageTransport:
return self.actor._transport
@@ -102,8 +110,9 @@ class Consumer[ActorT: Actor[Any]]:
# ── Lifecycle ──────────────────────────────────────────────────────────────
- async def start(self) -> None:
- self._startup_ts = int(time.time() * 1000)
+ async def start(self, progress: DeliveryProgress) -> None:
+ """Read the actor's rooms until stopped, starting each room where
+ `progress` says and recording through it how far it got."""
self._running = True
await self.actor.connect()
@@ -113,7 +122,7 @@ class Consumer[ActorT: Actor[Any]]:
retries = 0
while self._running:
try:
- await self._transport.receive_forever()
+ await self._transport.receive_forever(progress)
retries = 0
except Exception:
if not self._running:
@@ -152,12 +161,20 @@ class Consumer[ActorT: Actor[Any]]:
on_custom_event=self._handle_custom_event,
on_invite=self._handle_invite,
on_removed=self._handle_removed,
+ held_from=self.held_from,
)
)
async def teardown(self) -> None:
pass
+ def held_from(self, room_id: str) -> int | None:
+ """The lowest position in the room this consumer was handed and has
+ not acted on yet, or None. Progress is recorded only up to before it,
+ so whoever reads for the client next handles it. Nothing by default:
+ a consumer acts on each row as it is handed."""
+ return None
+
# ── Internal event handlers (filtering + dispatch to hooks) ────────────────
async def _handle_message(self, room: RoomRef, event: InboundMessage) -> None:
@@ -200,12 +217,12 @@ class Consumer[ActorT: Actor[Any]]:
self.actor.mark_joined(room.room_id, event.timestamp)
# A membership-preserving update (display name, avatar) re-fires
# m.room.member with membership == "join"; only a transition into
- # membership is an arrival. Joins predating this process are not
- # ours to announce, and the room is recorded so a redelivery of
- # the same join does not announce it twice.
+ # membership is an arrival. A join is handed over only while the
+ # client has not handled it, however long ago it was, and the
+ # room is recorded so a redelivery of the same join does not
+ # announce it twice.
if (
event.prev_membership != "join"
- and event.timestamp >= self._startup_ts
and room.room_id not in self._self_join_dispatched
):
self._self_join_dispatched.add(room.room_id)
@@ -321,15 +338,9 @@ class Consumer[ActorT: Actor[Any]]:
# ── Filtering ──────────────────────────────────────────────────────────────
def _should_ignore(self, room: RoomRef, event: InboundEvent) -> bool:
- if event.sender == self.transport_user_id:
- return True
-
- if event.timestamp:
- join_time = self.actor.room_join_times.get(room.room_id, self._startup_ts)
- if event.timestamp < join_time:
- return True
-
- return False
+ """The actor's own rows. Rows from before it joined are never handed
+ over at all: its reading in a room starts at its arrival."""
+ return event.sender == self.transport_user_id
# ── Event hooks (subclasses override) ──────────────────────────────────────
@@ -354,7 +365,7 @@ class Consumer[ActorT: Actor[Any]]:
self.transport_user_id,
room.room_id,
)
- await self.actor.join_room(room.room_id)
+ await self.actor.join_room(room.room_id, event.timestamp)
async def on_removed(self, room: RoomRef, event: InboundMembership) -> None:
"""This consumer's actor has been taken out of a room while it was
diff --git a/core/switch_core/clients/workspace_consumer.py b/core/switch_core/clients/workspace_consumer.py
index b291b3f90..1aa685f4a 100644
--- a/core/switch_core/clients/workspace_consumer.py
+++ b/core/switch_core/clients/workspace_consumer.py
@@ -22,11 +22,12 @@ class WorkspaceConsumerConfig(ClientConfig):
class WorkspaceConsumer(Consumer[Actor[WorkspaceConsumerConfig]]):
"""Reads every room a collaboration bridge mirrors, for its workspace.
- One per bridge (client type `bridge`). Everything new in those rooms is
- handed to the bridge's `CollaborationCore`, which drops what came from the
- platform and posts the rest to it. Its actor is the bridge's own identity
- in the room; it writes nothing, because the people on the platform write
- through their own `HumanActor`s.
+ One per bridge (client type `bridge`), running in the one switch-core
+ process that holds the bridge (`ownership.py`). Everything new in those
+ rooms is handed to the bridge's `CollaborationCore`, which drops what came
+ from the platform and posts the rest to it. Its actor is the bridge's own
+ identity in the room; it writes nothing, because the people on the
+ platform write through their own `HumanActor`s.
"""
def __init__(
@@ -38,6 +39,12 @@ class WorkspaceConsumer(Consumer[Actor[WorkspaceConsumerConfig]]):
super().__init__(actor=actor)
self._collaboration_core = collaboration_core
+ def held_from(self, room_id: str) -> int | None:
+ """The first file of a multi-file message still waiting to be posted
+ as one: progress stops before it, so whoever holds the bridge next
+ posts the group."""
+ return self._collaboration_core.held_from(room_id)
+
async def on_message(self, room: RoomRef, event: InboundMessage) -> None:
logger.debug(
"[WORKSPACE-CONSUMER] on_message room=%s sender=%s",
diff --git a/core/switch_core/config.py b/core/switch_core/config.py
index cd112de25..a0f486330 100644
--- a/core/switch_core/config.py
+++ b/core/switch_core/config.py
@@ -356,6 +356,18 @@ class SwitchConfig(BaseSettings):
# (pilot, development, demo, public), not the machine.
service_name: str = "switch-core"
environment: str | None = None
+ # The Kubernetes pod this process runs in, from the downward API. It names
+ # the process everywhere one process has to be told from another: its
+ # lease row, the controller connections it holds, the signals it sends,
+ # its log lines and its metrics (`process_identity.py`). Unset outside
+ # Kubernetes, where the host name stands in.
+ pod_name: str | None = None
+ # How many switch-core processes the deployment runs against this
+ # database. Read for the connection budget checked at boot
+ # (`db/connection_budget.py`) and for whether a dead LISTEN connection
+ # takes this process out of service: with one replica there is no other
+ # to send traffic to, so it does not.
+ core_replicas: int = Field(default=1, ge=1)
# The placeholder a log line carries when no tenant is bound at all —
# deliberately not a tenant id, and it must not be set to one. An
diff --git a/core/switch_core/connections/flows.py b/core/switch_core/connections/flows.py
index 7a36625be..b3426fee5 100644
--- a/core/switch_core/connections/flows.py
+++ b/core/switch_core/connections/flows.py
@@ -16,18 +16,19 @@ Either way Console posts the code back with its completion secret, Core
exchanges it and reads the account, and Console confirms; only then is the
connection written, through `ServiceBroker.connect`.
-Flows live in memory, as GitHub's do: ten minutes each, at most 256, one per
-person.
+Flows are rows in `sign_in_flows` (`connections/sign_in_flows.py`), as
+GitHub's are, so any switch-core process can take any step: ten minutes each,
+at most 256 per workspace, one per person.
"""
from __future__ import annotations
import secrets
-import time
from dataclasses import dataclass
from urllib.parse import urlencode
from switch_core.connections.adapters import (
+ ConnectionSecret,
ReauthorizationRequiredError,
ServiceAdapterError,
)
@@ -43,9 +44,16 @@ from switch_core.connections.loader import (
RedirectMode,
)
from switch_core.connections.oauth_clients import core_callback, loopback_redirect
+from switch_core.connections.sign_in_flows import (
+ FlowGoneError,
+ FlowsBusyError,
+ FlowUsedError,
+ SignInFlows,
+ StoredFlow,
+ digest,
+)
-FLOW_SECONDS = 600
-MAX_FLOWS = 256
+KIND = "service"
# Some vendors issue codes as signed tokens: Atlassian's run past 2,000
# characters.
MAX_CODE_LENGTH = 8192
@@ -56,36 +64,70 @@ class FlowError(Exception):
"""The flow cannot go on; the message is for the person."""
-@dataclass
+@dataclass(frozen=True)
class ServiceFlow:
- service: str
- name: str
- tenant_id: str
- user_id: str
- expires_at: float
- mode: RedirectMode
- redirect_uri: str
- port: int
- completion_secret: str
- owner_label: str
- consent: AccessLevel
- scopes: list[str]
- read_scopes: list[str]
- write_scopes: list[str]
- verifier: str
- browser_nonce: str
- # pending → authorizing → (core: returning → delivered) → checking →
- # ready | failed.
- status: str = "pending"
- account: Identity | None = None
- signed_in: SignIn | None = None
- error: str | None = None
+ """A service sign-in as its row stands.
+
+ Status runs pending → authorizing → (core: returning → delivered) →
+ checking → ready | failed.
+ """
+
+ stored: StoredFlow
+
+ @property
+ def state(self) -> str:
+ return self.stored.state
+
+ @property
+ def service(self) -> str:
+ return self.stored.service
+
+ @property
+ def name(self) -> str:
+ return str(self.stored.details["name"])
+
+ @property
+ def mode(self) -> RedirectMode:
+ return "core" if self.stored.mode == "core" else "loopback"
+
+ @property
+ def status(self) -> str:
+ return self.stored.status
+
+ @property
+ def port(self) -> int:
+ return self.stored.port
+
+ @property
+ def owner_label(self) -> str:
+ return self.stored.owner_label
+
+ @property
+ def error(self) -> str | None:
+ return self.stored.error
+
+ @property
+ def account(self) -> Identity | None:
+ result = self.stored.result
+ if result is None:
+ return None
+ return Identity(account_id=result["account_id"], label=result["account_label"])
+
+ @property
+ def signed_in(self) -> SignIn | None:
+ result = self.stored.result
+ if result is None:
+ return None
+ return SignIn(
+ secret=ConnectionSecret(dict(result["secret"])),
+ granted_scopes=list(result["granted_scopes"]),
+ )
class ServiceFlows:
- def __init__(self, public_url: str | None) -> None:
+ def __init__(self, public_url: str | None, flows: SignInFlows) -> None:
self._public_url = public_url
- self.flows: dict[str, ServiceFlow] = {}
+ self._flows = flows
def mode(self, definition: ConnectionDefinition) -> RedirectMode | None:
"""The first way back the catalog allows that this server can offer."""
@@ -103,7 +145,7 @@ class ServiceFlows:
f"{service}/flows/authorize?" + urlencode({"state": state})
)
- def start(
+ async def start(
self,
*,
definition: ConnectionDefinition,
@@ -114,20 +156,6 @@ class ServiceFlows:
completion_secret: str,
owner_label: str,
) -> ServiceFlow:
- now = time.time()
- self.flows = {k: v for k, v in self.flows.items() if v.expires_at > now}
- if state in self.flows:
- raise FlowError("This sign-in has already been used. Connect again.")
- for key in [
- key
- for key, flow in self.flows.items()
- if (flow.tenant_id, flow.user_id) == (tenant_id, user_id)
- ]:
- del self.flows[key]
- if len(self.flows) >= MAX_FLOWS:
- raise FlowError(
- "Too many sign-ins are under way. Please try again shortly."
- )
mode = self.mode(definition)
if mode is None:
raise FlowError(
@@ -149,157 +177,220 @@ class ServiceFlows:
consent: AccessLevel = "write" if access.write is not None else "read"
level = access.write or access.read
assert level.scopes is not None and access.read.scopes is not None
- flow = ServiceFlow(
- service=definition.slug,
- name=definition.name,
- tenant_id=tenant_id,
- user_id=user_id,
- expires_at=now + FLOW_SECONDS,
- mode=mode,
- redirect_uri=(
- loopback_redirect(port)
- if mode == "loopback"
- else core_callback(self._public_url or "", definition.slug)
- ),
- port=port,
- completion_secret=completion_secret,
- owner_label=owner_label,
- consent=consent,
- scopes=list(level.scopes),
- read_scopes=list(access.read.scopes),
- write_scopes=list(level.scopes) if consent == "write" else [],
- verifier=secrets.token_urlsafe(48),
- browser_nonce=secrets.token_urlsafe(32),
- )
- self.flows[state] = flow
- return flow
+ try:
+ stored = await self._flows.start(
+ kind=KIND,
+ service=definition.slug,
+ tenant_id=tenant_id,
+ user_id=user_id,
+ state=state,
+ mode=mode,
+ port=port,
+ redirect_uri=(
+ loopback_redirect(port)
+ if mode == "loopback"
+ else core_callback(self._public_url or "", definition.slug)
+ ),
+ owner_label=owner_label,
+ details={
+ "name": definition.name,
+ "consent": consent,
+ "scopes": list(level.scopes),
+ "read_scopes": list(access.read.scopes),
+ "write_scopes": list(level.scopes) if consent == "write" else [],
+ },
+ completion_secret=completion_secret,
+ )
+ except FlowUsedError:
+ raise FlowError(
+ "This sign-in has already been used. Connect again."
+ ) from None
+ except FlowsBusyError:
+ raise FlowError(
+ "Too many sign-ins are under way. Please try again shortly."
+ ) from None
+ return ServiceFlow(stored)
- def flow(self, state: str) -> ServiceFlow:
- flow = self.flows.get(state)
- if flow is None or flow.expires_at <= time.time():
- self.flows.pop(state, None)
+ async def flow(self, state: str) -> ServiceFlow:
+ """The flow under `state`, for the browser's steps."""
+ try:
+ stored = await self._flows.find(state)
+ except FlowGoneError:
+ raise FlowError(INTERRUPTED) from None
+ if stored.kind != KIND:
raise FlowError(INTERRUPTED)
- return flow
+ return ServiceFlow(stored)
- def owned(self, state: str, tenant_id: str, user_id: str) -> ServiceFlow:
- flow = self.flow(state)
- if (flow.tenant_id, flow.user_id) != (tenant_id, user_id):
+ async def owned(self, state: str, tenant_id: str, user_id: str) -> ServiceFlow:
+ try:
+ stored = await self._flows.get(state, tenant_id)
+ except FlowGoneError:
+ raise FlowError(INTERRUPTED) from None
+ if stored.kind != KIND or stored.user_id != user_id:
raise FlowError(INTERRUPTED)
- return flow
+ return ServiceFlow(stored)
- async def vendor_url(self, state: str, adapter: OAuthAdapter) -> str:
- """Where the browser signs in at the vendor; once per flow."""
- flow = self.flow(state)
- if flow.status != "pending":
+ async def vendor_url(
+ self, flow: ServiceFlow, adapter: OAuthAdapter
+ ) -> tuple[str, str]:
+ """Where the browser signs in at the vendor, and the nonce that marks
+ the browser sent there; once per flow."""
+ nonce = secrets.token_urlsafe(32)
+ taken = await self._flows.advance(
+ flow.stored,
+ expect=("pending",),
+ to="authorizing",
+ changes={"browser_nonce_hash": digest(nonce)},
+ )
+ if taken is None:
raise FlowError("This sign-in has already been used. Connect again.")
- flow.status = "authorizing"
try:
- return await adapter.authorization_url(
- redirect_uri=flow.redirect_uri,
- state=state,
- code_challenge=s256(flow.verifier),
- scopes=flow.scopes,
+ url = await adapter.authorization_url(
+ redirect_uri=flow.stored.redirect_uri,
+ state=flow.state,
+ code_challenge=s256(flow.stored.verifier),
+ scopes=list(flow.stored.details["scopes"]),
)
except BaseException:
- self.flows.pop(state, None)
+ await self._flows.discard(flow.stored)
raise
+ return url, nonce
- def returning(self, state: str, code: str, nonce: str) -> ServiceFlow:
+ async def returning(self, flow: ServiceFlow, code: str, nonce: str) -> None:
"""The browser is back at Core's callback, in core mode."""
- flow = self.flow(state)
if (
flow.mode != "core"
or flow.status not in ("authorizing", "returning")
or len(code) > MAX_CODE_LENGTH
- or not secrets.compare_digest(flow.browser_nonce, nonce)
+ or not flow.stored.holds_nonce(nonce)
):
raise FlowError("The sign-in could not be verified.")
if not code:
- flow.status = "failed"
- flow.error = f"{flow.name} sign-in was cancelled."
- raise FlowError(flow.error)
- flow.status = "returning"
- return flow
+ error = f"{flow.name} sign-in was cancelled."
+ await self._flows.advance(
+ flow.stored,
+ expect=("authorizing", "returning"),
+ to="failed",
+ changes={"error": error},
+ )
+ raise FlowError(error)
+ taken = await self._flows.advance(
+ flow.stored, expect=("authorizing", "returning"), to="returning", changes={}
+ )
+ if taken is None:
+ raise FlowError("The sign-in could not be verified.")
- def deliver(self, state: str, code: str, nonce: str) -> str:
+ async def deliver(self, flow: ServiceFlow, code: str, nonce: str) -> str:
"""The person continued: where their browser takes the code to Console."""
- flow = self.flow(state)
if (
flow.mode != "core"
or flow.status != "returning"
or not code
or len(code) > MAX_CODE_LENGTH
- or not secrets.compare_digest(flow.browser_nonce, nonce)
+ or not flow.stored.holds_nonce(nonce)
):
raise FlowError("The sign-in could not be verified.")
- flow.status = "delivered"
+ taken = await self._flows.advance(
+ flow.stored, expect=("returning",), to="delivered", changes={}
+ )
+ if taken is None:
+ raise FlowError("The sign-in could not be verified.")
return f"{loopback_redirect(flow.port)}?" + urlencode(
- {"code": code, "state": state}
+ {"code": code, "state": flow.state}
)
async def complete(
self,
- state: str,
+ flow: ServiceFlow,
*,
completion_secret: str,
code: str,
adapter: OAuthAdapter,
- ) -> ServiceFlow:
+ ) -> None:
"""Console hands back the code; Core exchanges it and reads the account."""
- flow = self.flow(state)
ready_for = "authorizing" if flow.mode == "loopback" else "delivered"
+ refused = FlowError(
+ f"{flow.name} sign-in could not be verified. Connect again."
+ )
if (
- not secrets.compare_digest(flow.completion_secret, completion_secret)
+ not flow.stored.holds_secret(completion_secret)
or flow.status != ready_for
or not code
or len(code) > MAX_CODE_LENGTH
):
- raise FlowError(
- f"{flow.name} sign-in could not be verified. Connect again."
- )
- flow.status = "checking"
+ raise refused
+ checking = await self._flows.advance(
+ flow.stored, expect=(ready_for,), to="checking", changes={}
+ )
+ if checking is None:
+ raise refused
try:
signed_in = await adapter.exchange_code(
code=code,
- verifier=flow.verifier,
- redirect_uri=flow.redirect_uri,
- scopes=flow.scopes,
+ verifier=flow.stored.verifier,
+ redirect_uri=flow.stored.redirect_uri,
+ scopes=list(flow.stored.details["scopes"]),
)
account = await adapter.identify(signed_in.secret.access_token or "")
except ReauthorizationRequiredError:
- flow.status = "failed"
- flow.error = f"{flow.name} refused the sign-in. Connect again."
- raise FlowError(flow.error) from None
+ await self._fail(
+ checking, f"{flow.name} refused the sign-in. Connect again."
+ )
+ raise FlowError(
+ f"{flow.name} refused the sign-in. Connect again."
+ ) from None
except ServiceAdapterError as error:
- flow.status = "failed"
- flow.error = str(error)
- raise FlowError(flow.error) from None
+ await self._fail(checking, str(error))
+ raise FlowError(str(error)) from None
except BaseException:
- flow.status = "failed"
- flow.error = INTERRUPTED
+ await self._fail(checking, INTERRUPTED)
raise
- flow.signed_in = signed_in
- flow.account = account
- flow.status = "ready"
- return flow
+ ready = await self._flows.advance(
+ checking,
+ expect=("checking",),
+ to="ready",
+ changes={
+ "encrypted_result": self._flows.seal(
+ {
+ "secret": signed_in.secret.values,
+ "granted_scopes": list(signed_in.granted_scopes),
+ "account_id": account.account_id,
+ "account_label": account.label,
+ }
+ ),
+ "account_label": account.label,
+ },
+ )
+ if ready is None:
+ # The flow expired or was cancelled while the code was exchanged:
+ # what the vendor issued has nowhere to go.
+ raise FlowError(INTERRUPTED)
+
+ async def _fail(self, flow: StoredFlow, error: str) -> None:
+ await self._flows.advance(
+ flow, expect=("checking",), to="failed", changes={"error": error}
+ )
def consent(self, flow: ServiceFlow) -> AccessLevel:
"""What the person consented to, as the vendor granted it."""
- assert flow.signed_in is not None
- granted = set(flow.signed_in.granted_scopes)
- if flow.write_scopes and set(flow.write_scopes) <= granted:
+ signed_in = flow.signed_in
+ assert signed_in is not None
+ granted = set(signed_in.granted_scopes)
+ write_scopes = set(flow.stored.details["write_scopes"])
+ if write_scopes and write_scopes <= granted:
return "write"
- if set(flow.read_scopes) <= granted:
+ if set(flow.stored.details["read_scopes"]) <= granted:
return "read"
raise FlowError(
f"{flow.name} granted less than Switch needs to read. Connect again "
"and allow what it asks for."
)
- def ready(self, state: str, completion_secret: str) -> ServiceFlow:
- flow = self.flow(state)
- if not secrets.compare_digest(flow.completion_secret, completion_secret):
+ def ready(self, flow: ServiceFlow, completion_secret: str) -> None:
+ if not flow.stored.holds_secret(completion_secret):
raise FlowError(INTERRUPTED)
if flow.status != "ready":
raise FlowError(f"Finish signing in to {flow.name} before confirming.")
- return flow
+
+ async def discard(self, flow: ServiceFlow) -> None:
+ await self._flows.discard(flow.stored)
diff --git a/core/switch_core/connections/sign_in_flows.py b/core/switch_core/connections/sign_in_flows.py
new file mode 100644
index 000000000..7fd9afd23
--- /dev/null
+++ b/core/switch_core/connections/sign_in_flows.py
@@ -0,0 +1,297 @@
+"""Sign-ins under way, kept in Postgres so any switch-core process can take
+any step of one.
+
+A person connecting GitHub (`gateway/github_connections.py`) or a catalog
+service (`connections/flows.py`) from Switch Console goes through half a dozen
+requests: Console starts the flow, the browser leaves for the vendor and comes
+back, Console hands over the code, follows the flow and confirms it. With
+several processes behind one address each request may reach a different one,
+so the flow is a row in `sign_in_flows` rather than an entry in one process's
+memory.
+
+Every step is a conditional update from the status it expects to the next,
+so two requests racing for one step cannot both take it, and a step that
+finds the flow already past it is refused as before. The flow's tenant is
+bound for every read and write. Console's requests carry a signed-in person
+and so their tenant; the browser's carry only the flow's state, and find the
+tenant through `tenant_of_sign_in_flow` (`db/tenant_lookup.py`).
+
+What would be secret is never stored in the clear. The PKCE verifier and the
+vendor's credentials, once the code is exchanged, are keyring-encrypted; the
+completion secret Console holds and the nonce the browser carries in a cookie
+are stored as SHA-256 hashes and compared against what each presents.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import secrets
+from collections.abc import Collection, Mapping
+from dataclasses import dataclass
+from datetime import timedelta
+from typing import Any
+
+from sqlalchemy import delete, func, select, text, update
+from sqlalchemy.dialects.postgresql import insert
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import SignInFlow, require_tenant_id
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.tenant_lookup import tenant_of_sign_in_flow
+from switch_core.keys import Keyring
+
+FLOW_SECONDS = 600
+# Per workspace. A person holds at most one flow of each kind, so this bounds
+# what one workspace's members can have open at once, not a global queue.
+MAX_FLOWS = 256
+
+
+class FlowGoneError(Exception):
+ """No flow under this state: never started, finished, cancelled, or
+ expired."""
+
+
+class FlowUsedError(Exception):
+ """A flow under this state already exists or existed: a state is used once."""
+
+
+class FlowsBusyError(Exception):
+ """The workspace has as many sign-ins under way as it may."""
+
+
+def digest(value: str) -> str:
+ return hashlib.sha256(value.encode()).hexdigest()
+
+
+@dataclass(frozen=True)
+class StoredFlow:
+ """One flow as its row stands, with the verifier and any result opened."""
+
+ state: str
+ tenant_id: str
+ kind: str
+ service: str
+ user_id: str
+ status: str
+ mode: str
+ port: int
+ redirect_uri: str
+ owner_label: str
+ details: Mapping[str, Any]
+ verifier: str
+ completion_secret_hash: str
+ browser_nonce_hash: str | None
+ result: Mapping[str, Any] | None
+ account_label: str | None
+ error: str | None
+
+ def holds_secret(self, presented: str) -> bool:
+ """Whether `presented` is the completion secret Console started with."""
+ return secrets.compare_digest(self.completion_secret_hash, digest(presented))
+
+ def holds_nonce(self, presented: str) -> bool:
+ """Whether `presented` is the nonce set on the browser that left for
+ the vendor through Core."""
+ return self.browser_nonce_hash is not None and secrets.compare_digest(
+ self.browser_nonce_hash, digest(presented)
+ )
+
+
+class SignInFlows:
+ def __init__(
+ self, session_factory: async_sessionmaker[AsyncSession], keyring: Keyring
+ ) -> None:
+ self._session_factory = session_factory
+ self._keyring = keyring
+
+ async def start(
+ self,
+ *,
+ kind: str,
+ service: str,
+ tenant_id: str,
+ user_id: str,
+ state: str,
+ mode: str,
+ port: int,
+ redirect_uri: str,
+ owner_label: str,
+ details: Mapping[str, Any],
+ completion_secret: str,
+ ) -> StoredFlow:
+ """Record a new flow, replacing the person's earlier one of this kind.
+
+ Raises `FlowUsedError` when the state has been used before, and
+ `FlowsBusyError` when the workspace is at `MAX_FLOWS`. Expired flows
+ in the workspace are deleted first, so a flow nobody finished is gone
+ by the time anyone in its workspace starts another.
+ """
+ verifier = secrets.token_urlsafe(48)
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ # One start at a time per workspace, held to the commit. The delete,
+ # count and insert below see only committed rows, so two starts side
+ # by side would otherwise both keep a flow for one person, or both
+ # take the workspace's last place under `MAX_FLOWS`.
+ await session.execute(
+ text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
+ {"key": f"sign-in-flows:{tenant_id}"},
+ )
+ await session.execute(
+ delete(SignInFlow).where(
+ SignInFlow.tenant_id == tenant_id,
+ SignInFlow.expires_at <= func.now(),
+ )
+ )
+ if await session.scalar(
+ select(SignInFlow.id).where(SignInFlow.id == state)
+ ):
+ await session.rollback()
+ raise FlowUsedError
+ await session.execute(
+ delete(SignInFlow).where(
+ SignInFlow.tenant_id == tenant_id,
+ SignInFlow.user_id == user_id,
+ SignInFlow.kind == kind,
+ )
+ )
+ open_flows = await session.scalar(
+ select(func.count())
+ .select_from(SignInFlow)
+ .where(SignInFlow.tenant_id == tenant_id)
+ )
+ if (open_flows or 0) >= MAX_FLOWS:
+ await session.rollback()
+ raise FlowsBusyError
+ inserted = await session.execute(
+ insert(SignInFlow)
+ .values(
+ id=state,
+ tenant_id=tenant_id,
+ kind=kind,
+ service=service,
+ user_id=user_id,
+ status="pending",
+ mode=mode,
+ port=port,
+ redirect_uri=redirect_uri,
+ owner_label=owner_label,
+ details=dict(details),
+ encrypted_verifier=self._keyring.encrypt(verifier),
+ completion_secret_hash=digest(completion_secret),
+ expires_at=func.now() + timedelta(seconds=FLOW_SECONDS),
+ )
+ .on_conflict_do_nothing(index_elements=[SignInFlow.id])
+ .returning(SignInFlow)
+ )
+ row = inserted.scalar_one_or_none()
+ if row is None:
+ await session.rollback()
+ raise FlowUsedError
+ flow = self._open(row)
+ await session.commit()
+ return flow
+
+ async def get(self, state: str, tenant_id: str) -> StoredFlow:
+ """The flow under `state` in `tenant_id`. Raises `FlowGoneError`."""
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ row = await session.scalar(
+ select(SignInFlow).where(
+ SignInFlow.tenant_id == tenant_id,
+ SignInFlow.id == state,
+ SignInFlow.expires_at > func.now(),
+ )
+ )
+ if row is None:
+ raise FlowGoneError
+ return self._open(row)
+
+ async def find(self, state: str) -> StoredFlow:
+ """The flow under `state`, in whichever tenant holds it: for the
+ browser's steps, which carry no tenant. Raises `FlowGoneError`."""
+ if not state:
+ raise FlowGoneError
+ tenant_id = await tenant_of_sign_in_flow(self._session_factory, state)
+ if tenant_id is None:
+ raise FlowGoneError
+ return await self.get(state, tenant_id)
+
+ async def advance(
+ self,
+ flow: StoredFlow,
+ *,
+ expect: Collection[str],
+ to: str,
+ changes: Mapping[str, Any],
+ ) -> StoredFlow | None:
+ """Move the flow to status `to`, with the columns in `changes`, if it
+ is still unexpired and in one of `expect`. None when it is not: another
+ request took the step first, or the flow is gone."""
+ async with tenant_session(self._session_factory, flow.tenant_id) as session:
+ updated = await session.execute(
+ update(SignInFlow)
+ .where(
+ SignInFlow.tenant_id == flow.tenant_id,
+ SignInFlow.id == flow.state,
+ SignInFlow.status.in_(list(expect)),
+ SignInFlow.expires_at > func.now(),
+ )
+ .values(status=to, **changes)
+ .returning(SignInFlow)
+ )
+ row = updated.scalar_one_or_none()
+ await session.commit()
+ return None if row is None else self._open(row)
+
+ def seal(self, result: Mapping[str, Any]) -> str:
+ """`result` as `advance` stores it in `encrypted_result`."""
+ return self._keyring.encrypt(json.dumps(dict(result), separators=(",", ":")))
+
+ async def discard(self, flow: StoredFlow) -> None:
+ async with tenant_session(self._session_factory, flow.tenant_id) as session:
+ await session.execute(
+ delete(SignInFlow).where(
+ SignInFlow.tenant_id == flow.tenant_id,
+ SignInFlow.id == flow.state,
+ )
+ )
+ await session.commit()
+
+ async def discard_for_user(
+ self, session: AsyncSession, user_id: str, kind: str | None
+ ) -> None:
+ """Delete the person's flows in the caller's tenant: of one kind, or
+ of every kind when `kind` is None. In the caller's session, so it
+ commits with whatever removed the person's connection."""
+ query = delete(SignInFlow).where(
+ SignInFlow.tenant_id == require_tenant_id(),
+ SignInFlow.user_id == user_id,
+ )
+ if kind is not None:
+ query = query.where(SignInFlow.kind == kind)
+ await session.execute(query)
+
+ def _open(self, row: SignInFlow) -> StoredFlow:
+ return StoredFlow(
+ state=row.id,
+ tenant_id=row.tenant_id,
+ kind=row.kind,
+ service=row.service,
+ user_id=row.user_id,
+ status=row.status,
+ mode=row.mode,
+ port=row.port,
+ redirect_uri=row.redirect_uri,
+ owner_label=row.owner_label,
+ details=row.details,
+ verifier=self._keyring.decrypt(row.encrypted_verifier),
+ completion_secret_hash=row.completion_secret_hash,
+ browser_nonce_hash=row.browser_nonce_hash,
+ result=(
+ None
+ if row.encrypted_result is None
+ else json.loads(self._keyring.decrypt(row.encrypted_result))
+ ),
+ account_label=row.account_label,
+ error=row.error,
+ )
diff --git a/core/switch_core/db/connection_budget.py b/core/switch_core/db/connection_budget.py
new file mode 100644
index 000000000..9ab677643
--- /dev/null
+++ b/core/switch_core/db/connection_budget.py
@@ -0,0 +1,108 @@
+"""How many Postgres connections the deployment's switch-core processes can
+hold at once, checked against what the server allows.
+
+Each process holds its pool at worst (DB_POOL_SIZE plus DB_MAX_OVERFLOW) and,
+outside the pool, one connection per `LISTEN` it keeps open, plus one for the
+database-server sampler when metrics are exported. The pool is sized per
+process, so every replica added multiplies it, and a rolling deploy of
+several replicas starts one more before it stops an old one. A deployment that
+outgrows `max_connections` does not fail at once: it fails under load, when
+the last pools fill and new connections are refused.
+
+So the sum is checked at boot, against the server's `max_connections` less
+the slots it keeps for superusers. Over budget, a deployment of several
+replicas refuses to start, since it is a new configuration and should be
+sized before it serves. A single replica only warns: that is the
+configuration every existing deployment has, and it may share the server with
+other software whose connections are not counted here either way.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import AsyncEngine
+
+from switch_core.config import SwitchConfig
+
+logger = logging.getLogger(__name__)
+
+# The connections each process holds open outside its pool, one per LISTEN:
+# `MessageListener`, `AgentSessionActivityListener`, `AgentEventListener` and
+# `SignalBus`.
+LISTENING_CONNECTIONS = 4
+
+
+class ConnectionBudgetError(RuntimeError):
+ """The deployment's processes could hold more connections than Postgres
+ allows."""
+
+
+@dataclass(frozen=True)
+class ConnectionBudget:
+ per_process: int
+ processes: int
+ available: int
+
+ @property
+ def needed(self) -> int:
+ return self.per_process * self.processes
+
+ def describe(self) -> str:
+ return (
+ f"{self.processes} switch-core process(es) at once x "
+ f"{self.per_process} connection(s) each = {self.needed}, against "
+ f"{self.available} the database allows"
+ )
+
+
+def per_process(config: SwitchConfig) -> int:
+ """The most connections one process can hold."""
+ sampler = 1 if config.observability_enabled and config.otlp_metrics_enabled else 0
+ return (
+ config.db_pool_size + config.db_max_overflow + LISTENING_CONNECTIONS + sampler
+ )
+
+
+def processes_at_once(replicas: int) -> int:
+ """The most processes running at once: every replica, and during a rolling
+ deploy of several, the one started before an old one stops. A single
+ replica is replaced rather than rolled."""
+ return replicas + 1 if replicas > 1 else replicas
+
+
+async def check_connection_budget(config: SwitchConfig, engine: AsyncEngine) -> None:
+ async with engine.connect() as connection:
+ max_connections = int(
+ (await connection.execute(text("SHOW max_connections"))).scalar_one()
+ )
+ reserved = int(
+ (
+ await connection.execute(text("SHOW superuser_reserved_connections"))
+ ).scalar_one()
+ )
+ budget = ConnectionBudget(
+ per_process=per_process(config),
+ processes=processes_at_once(config.core_replicas),
+ available=max_connections - reserved,
+ )
+ if budget.needed <= budget.available:
+ logger.info("Database connection budget: %s.", budget.describe())
+ return
+ advice = (
+ "Lower DB_POOL_SIZE or DB_MAX_OVERFLOW, run fewer replicas, or raise "
+ "the server's max_connections."
+ )
+ if config.core_replicas > 1:
+ raise ConnectionBudgetError(
+ f"switch-core could hold more database connections than Postgres "
+ f"allows: {budget.describe()}. {advice}"
+ )
+ logger.warning(
+ "switch-core could hold more database connections than Postgres "
+ "allows: %s. Under load, new connections will be refused. %s",
+ budget.describe(),
+ advice,
+ )
diff --git a/core/switch_core/db/encrypted_json.py b/core/switch_core/db/encrypted_json.py
index fe1b50412..40b71344e 100644
--- a/core/switch_core/db/encrypted_json.py
+++ b/core/switch_core/db/encrypted_json.py
@@ -119,7 +119,12 @@ async def reencrypt_stale_values(
ids = list(rows.scalars())
if not ids:
return 0
- instances = await session.execute(select(model).where(model.id.in_(ids)))
+ # Locked, so a value another process writes between the select above and
+ # the flush below waits for this transaction, or, if it committed first,
+ # is the value read here: either way the newer value is the one kept.
+ instances = await session.execute(
+ select(model).where(model.id.in_(ids)).with_for_update()
+ )
for instance in instances.scalars():
attributes.flag_modified(instance, column)
await session.flush()
diff --git a/core/switch_core/db/key_rotation.py b/core/switch_core/db/key_rotation.py
index d5b964051..e07ae787e 100644
--- a/core/switch_core/db/key_rotation.py
+++ b/core/switch_core/db/key_rotation.py
@@ -5,6 +5,18 @@ key is added to the front of `SECRET_KEYS`, or on the first boot after
`SECRET_KEYS` replaces `JWT_SECRET_KEY`, values under any other key are
rewritten here, so the older key can be removed once this has run.
+With several processes serving at once, a new front key has to be rolled out
+in two phases. First every process is given the new key at the back of
+`SECRET_KEYS`, so all of them can read it while still writing under the old
+one. Only then is it moved to the front: the processes that start with it
+write, sign and re-encrypt under it, and the older ones still running can read
+all of that. Done in one step, the first new process would rewrite every
+stored secret under a key the old ones lack, and they could no longer open
+bridge configs, bot tokens or API keys until replaced.
+`require_front_key_readable` enforces the order: every live process records
+the ids of its keys on its lease, and a process whose front key one of them
+lacks refuses to start.
+
The columns are listed here rather than discovered: each is a different shape
(a JSON envelope, a bare string), and a new encrypted column is a deliberate
addition this list should be part of.
@@ -29,10 +41,13 @@ from switch_core.db.models import (
ServiceConnection,
ServiceOAuthClient,
ServiceTokenIssuance,
+ SignInFlow,
TrustSettings,
)
from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
from switch_core.keys import Keyring
+from switch_core.process_lease import read_leases
logger = logging.getLogger(__name__)
@@ -47,6 +62,8 @@ _ENCRYPTED_TEXT_COLUMNS: tuple[tuple[type[Any], str], ...] = (
(MachineWorkspace, "enrollment_code_encrypted"),
(ServiceConnection, "encrypted_secret"),
(ServiceTokenIssuance, "encrypted_token"),
+ (SignInFlow, "encrypted_verifier"),
+ (SignInFlow, "encrypted_result"),
)
# In tables no tenant holds (`db/rls_ddl.py`, GLOBAL_TABLES), one row set for
# the whole deployment (`trust_settings`, `service_oauth_clients`): rewritten
@@ -57,6 +74,51 @@ _GLOBAL_ENCRYPTED_TEXT_COLUMNS: tuple[tuple[type[Any], str], ...] = (
)
+class KeyRotationError(RuntimeError):
+ """A live process could not read what this one would write."""
+
+
+async def require_front_key_readable(
+ session_factory: async_sessionmaker[AsyncSession],
+ keyring: Keyring,
+ process_id: str,
+) -> None:
+ """Refuse to start when a live process lacks this one's front key.
+
+ Called under the boot lock, before anything is encrypted under that key.
+ A process from before keys were recorded is not checked, and is named in
+ a warning instead: there is no way to know what it holds.
+ """
+ async with session_factory() as session:
+ leases = await read_leases(session, SwitchCoreProcessStore())
+ front = keyring.current.id
+ lacking: list[str] = []
+ unknown: list[str] = []
+ for other, lease in leases.leases.items():
+ if other == process_id or not leases.holds(other):
+ continue
+ if lease.key_ids is None:
+ unknown.append(other)
+ elif front not in lease.key_ids:
+ lacking.append(other)
+ if lacking:
+ raise KeyRotationError(
+ f"SECRET_KEYS puts key {front!r} first, and switch-core process(es) "
+ f"{', '.join(sorted(lacking))} still running cannot read it, so what "
+ "this process writes would be unreadable to them. Roll the key out "
+ "in two phases: first add it at the end of SECRET_KEYS everywhere, "
+ "and only once every process has it, move it to the front."
+ )
+ if unknown:
+ logger.warning(
+ "Could not check that switch-core process(es) %s can read key %r: "
+ "they predate the check. If this deploy moves a new key to the "
+ "front of SECRET_KEYS, they cannot read what is written under it.",
+ ", ".join(sorted(unknown)),
+ front,
+ )
+
+
async def reencrypt_stale_text(
session: AsyncSession, keyring: Keyring, model: type[Any], column: str
) -> int:
@@ -82,17 +144,21 @@ async def reencrypt_stale_text(
)
count = 0
for *key_values, value in rows.all():
- await session.execute(
+ # Only while the value is still the one read: a request on another
+ # process may have written a new one since, already under the
+ # current key, and that is the one to keep.
+ result = await session.execute(
update(model)
.where(
*(
key == key_value
for key, key_value in zip(keys, key_values, strict=True)
- )
+ ),
+ stored == value,
)
.values({column: keyring.encrypt(keyring.decrypt(value))})
)
- count += 1
+ count += result.rowcount # type: ignore[attr-defined]
return count
diff --git a/core/switch_core/db/membership_notify_ddl.py b/core/switch_core/db/membership_notify_ddl.py
new file mode 100644
index 000000000..bb8d01291
--- /dev/null
+++ b/core/switch_core/db/membership_notify_ddl.py
@@ -0,0 +1,67 @@
+"""The trigger that announces each change to `client_rooms` as a signal.
+
+A client reads the rooms it is a member of, and every switch-core process runs
+its own copy of each client, so a membership written by one process has to
+reach the copies in all of them: the one added must start reading the room,
+and the one removed must stop. Removal is an access-control decision, so it
+cannot depend on every writer remembering to say so. Memberships are written
+from several places (an invitation, a join, room provisioning, a room or
+client deleted with its rows), and a trigger is the one place none of them can
+skip, as `notify_ddl.py` argues for messages.
+
+The notification goes out on the `SignalBus` channel as an ordinary signal,
+`membership.changed`, naming the tenant, the client and the room. It carries no
+sender process, so every process handles it, the writer's included, and each
+re-reads that client's rows rather than trusting the payload
+(`transport/membership_signals.py`). `NOTIFY` is delivered on commit, so a
+process never hears of a membership it could not then read.
+
+The migration that installs this carries its own copy, frozen as it was
+written; this copy is what `create_all` builds for the tests.
+"""
+
+from __future__ import annotations
+
+from switch_core.signals import SIGNAL_CHANNEL
+
+MEMBERSHIP_CHANGED_SIGNAL = "membership.changed"
+
+MEMBERSHIP_NOTIFY_FUNCTION_NAME = "switch_notify_membership"
+MEMBERSHIP_NOTIFY_TRIGGER_NAME = "client_rooms_notify"
+
+CREATE_MEMBERSHIP_NOTIFY_FUNCTION = f"""
+CREATE OR REPLACE FUNCTION {MEMBERSHIP_NOTIFY_FUNCTION_NAME}() RETURNS trigger AS $$
+DECLARE
+ changed client_rooms;
+BEGIN
+ IF TG_OP = 'DELETE' THEN
+ changed := OLD;
+ ELSE
+ changed := NEW;
+ END IF;
+ PERFORM pg_notify(
+ '{SIGNAL_CHANNEL}',
+ json_build_object(
+ 'kind', '{MEMBERSHIP_CHANGED_SIGNAL}',
+ 'tenant_id', changed.tenant_id,
+ 'client_id', changed.client_id,
+ 'room_id', changed.room_id
+ )::text
+ );
+ RETURN NULL;
+END;
+$$ LANGUAGE plpgsql
+"""
+
+CREATE_MEMBERSHIP_NOTIFY_TRIGGER = f"""
+CREATE TRIGGER {MEMBERSHIP_NOTIFY_TRIGGER_NAME}
+ AFTER INSERT OR DELETE ON client_rooms
+ FOR EACH ROW EXECUTE FUNCTION {MEMBERSHIP_NOTIFY_FUNCTION_NAME}()
+"""
+
+DROP_MEMBERSHIP_NOTIFY_TRIGGER = (
+ f"DROP TRIGGER IF EXISTS {MEMBERSHIP_NOTIFY_TRIGGER_NAME} ON client_rooms"
+)
+DROP_MEMBERSHIP_NOTIFY_FUNCTION = (
+ f"DROP FUNCTION IF EXISTS {MEMBERSHIP_NOTIFY_FUNCTION_NAME}()"
+)
diff --git a/core/switch_core/db/models.py b/core/switch_core/db/models.py
index 024724e97..3d6c12c68 100644
--- a/core/switch_core/db/models.py
+++ b/core/switch_core/db/models.py
@@ -24,7 +24,7 @@ from sqlalchemy import (
func,
text,
)
-from sqlalchemy.dialects.postgresql import JSONB
+from sqlalchemy.dialects.postgresql import ARRAY, JSONB
from sqlalchemy.orm import Mapped, declared_attr, mapped_column
from switch_core.db.agent_event_notify_ddl import (
@@ -34,6 +34,11 @@ from switch_core.db.agent_event_notify_ddl import (
)
from switch_core.db.base import Base
from switch_core.db.encrypted_json import EncryptedJSONB
+from switch_core.db.membership_notify_ddl import (
+ CREATE_MEMBERSHIP_NOTIFY_FUNCTION,
+ CREATE_MEMBERSHIP_NOTIFY_TRIGGER,
+ DROP_MEMBERSHIP_NOTIFY_TRIGGER,
+)
from switch_core.db.notify_ddl import (
CREATE_NOTIFY_FUNCTION,
CREATE_NOTIFY_TRIGGER,
@@ -616,6 +621,16 @@ class ClientRoom(TenantScoped, Base):
__tablename__ = "client_rooms"
__table_args__ = (
Index("ix_client_rooms_tenant_id", "tenant_id"),
+ # What a membership's delivery cursor refers to, tenant included, as
+ # every reference between scoped tables does (`DeliveryCursor`).
+ UniqueConstraint(
+ "tenant_id",
+ "client_id",
+ "room_id",
+ name="uq_client_rooms_tenant_client_room",
+ ),
+ # The router reads a room's members on every row it writes there.
+ Index("ix_client_rooms_room_id", "room_id"),
ForeignKeyConstraint(
["tenant_id", "client_id"],
["clients.tenant_id", "clients.id"],
@@ -1655,6 +1670,11 @@ class MessagingInstallState(TenantScoped, Base):
decided_at: Mapped[str | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
+ # The chat a claim-based install (Telegram) was put to when its state was
+ # redeemed; only an answer from that chat may decide it. In the row, not
+ # in the memory of the process that posted the proposal, because the
+ # admin's answer may reach any switch-core process.
+ proposed_chat_id: Mapped[str | None] = mapped_column(Text, nullable=True)
class MessagingEventReceipt(TenantScoped, Base):
@@ -2047,7 +2067,7 @@ class BridgeMessageMap(TenantScoped, Base):
bridging (resolving a Switch thread root to its external post and vice
versa) and the edit/delete sync that previously relied on a volatile
in-memory dict. Unique in both directions per bridge so either id resolves
- the other.
+ the other. A room message going out is claimed here before it is posted.
"""
__tablename__ = "bridge_message_map"
@@ -2067,7 +2087,11 @@ class BridgeMessageMap(TenantScoped, Base):
bridge_id: Mapped[str] = mapped_column(Text, nullable=False)
external_channel_id: Mapped[str] = mapped_column(Text, nullable=False)
transport_event_id: Mapped[str] = mapped_column(Text, nullable=False)
- external_post_id: Mapped[str] = mapped_column(Text, nullable=False)
+ # Null while a room message is claimed for posting and not posted yet: an
+ # outbound relay writes the row first and fills this in after the platform
+ # answers, so a second process handling the same message finds the claim
+ # and does not post it again (`CollaborationCore._claim_outbound`).
+ external_post_id: Mapped[str | None] = mapped_column(Text, nullable=True)
created_at: Mapped[str] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
@@ -2142,14 +2166,15 @@ class TelemetrySnapshotWatermark(Base):
class SwitchCoreProcess(Base):
- """A running switch-core process's lease (`management/process_lease.py`).
+ """A running switch-core process's lease (`process_lease.py`).
- Each process holding controller sockets claims a row at startup, renews
- `beat_at` every few seconds, and sets `stopped_at` as it shuts down. A
- controller connection whose holding process's lease is stale, stopped or
- gone reads as offline, which is how a process that died without writing
- its connections' closings is noticed. One write per process per renewal,
- however many machines it holds.
+ Every process claims a row at startup under its process id
+ (`process_identity.py`), renews `beat_at` every few seconds, and sets
+ `stopped_at` as it shuts down. An agent or controller connection whose
+ holding process's lease is stale, stopped or gone reads as offline, which
+ is how a process that died without writing its connections' closings is
+ noticed, and work it owned (`OwnershipLease`) may be taken over. One
+ write per process per renewal, however much it holds.
Not tenant-scoped: a process serves every tenant. Rows long stale are
pruned by the live processes.
@@ -2167,6 +2192,74 @@ class SwitchCoreProcess(Base):
stopped_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
+ # The ids of the `SECRET_KEYS` this process can read, so a process
+ # starting with a new front key can tell whether every live one could read
+ # what it writes (`db/key_rotation.py`). Ids only, never key material.
+ # NULL for a process from before the column.
+ key_ids: Mapped[list[str] | None] = mapped_column(ARRAY(Text), nullable=True)
+
+
+# ── Schema revisions ─────────────────────────────────────────────────────────
+
+
+class SchemaExpandOnlyRevision(Base):
+ """An applied schema revision that only added to the schema.
+
+ Written by `migrations/env.py` as Alembic applies a revision marked
+ `expand_only`, and deleted as it is undone, with the revision it follows.
+ A switch-core release older than the schema reads this chain back from the
+ database's head to its own (`db/schema_version.py`): if every revision in
+ between is here, it can serve on the newer schema, which is what lets two
+ releases run side by side during a rolling deploy. The older release has
+ no copy of those revisions' files, so the database is the only place it
+ can learn this.
+ """
+
+ __tablename__ = "schema_expand_only_revisions"
+
+ revision: Mapped[str] = mapped_column(Text, primary_key=True)
+ down_revision: Mapped[str] = mapped_column(Text, nullable=False)
+ applied_at: Mapped[datetime] = mapped_column(
+ DateTime(timezone=True), server_default=func.now(), nullable=False
+ )
+
+
+class OwnershipLease(Base):
+ """Which switch-core process runs a piece of work that must run once.
+
+ One row per piece of work: `kind` says what it is (a client's consumer, a
+ collaboration bridge's platform connection) and `key` which one (the
+ client's or the bridge's id). `process_id` is the holder, null once it
+ has let go. A holder counts only while its own lease in
+ `switch_core_processes` holds, so a process that dies loses everything it
+ held when that lease lapses, with no write of its own (`ownership.py`).
+
+ `token` grows by one with every claim, and the holder's writes that must
+ not outlive its hold name it (`DeliveryCursorStore.advance`), so a holder
+ that was taken over without knowing it is refused rather than obeyed.
+
+ Not tenant-scoped: one process holds work for every tenant at once, and a
+ row names only which process runs which id.
+ """
+
+ __tablename__ = "ownership_leases"
+ __table_args__ = (Index("ix_ownership_leases_process_id", "process_id"),)
+
+ kind: Mapped[str] = mapped_column(Text, primary_key=True)
+ key: Mapped[str] = mapped_column(Text, primary_key=True)
+ process_id: Mapped[str | None] = mapped_column(
+ Text,
+ ForeignKey(
+ "switch_core_processes.id",
+ name="fk_ownership_leases_process",
+ ondelete="SET NULL",
+ ),
+ nullable=True,
+ )
+ token: Mapped[int] = mapped_column(BigInteger, nullable=False)
+ changed_at: Mapped[datetime] = mapped_column(
+ DateTime(timezone=True), server_default=func.now(), nullable=False
+ )
# ── Trust settings ───────────────────────────────────────────────────────────
@@ -2330,48 +2423,47 @@ class MessageAttachment(TenantScoped, Base):
class DeliveryCursor(TenantScoped, Base):
- """How far one agent has been delivered in one room.
-
- The cursor it replaces lived in memory in the event buffer, so a restart
- resumed from wherever that buffer happened to be rather than from what the
- agent had actually been given. Persisting it makes "what has this agent
- seen" outlive the process, which is what lets delivery be driven from the
- table instead of from a live connection.
-
- `last_seq` is a position in the room, not a count: `seq` is a per-room
- total order, so "everything up to n" is unambiguous and re-reading from it
- is idempotent. It is only ever advanced, never rewound — a cursor that
- could go backwards would redeliver, and a redelivered message is
- indistinguishable to a reader from a new one.
+ """How far one client's consumer has handled one room it is a member of.
+
+ A consumer reads its rooms from these rows, so whichever switch-core
+ process owns the client (`ownership.py`) resumes where the last owner
+ stopped, after a restart or a hand-over, instead of at the room's newest
+ row. Every row the consumer reads moves its cursor, whether or not the
+ row did anything; that is what lets an agent's reader treat a room row as
+ decided once the cursor of the agent's consumer has passed it
+ (`EventBuffer.read_conversation`).
+
+ `last_seq` is a position in the room (`messages.seq`), only ever moved
+ forward: a cursor that went back would hand its consumer rows it had
+ already acted on. Only the owner moves it, checked against its lease
+ token in the same statement (`DeliveryCursorStore.advance`).
+
+ Keyed by the membership and deleted with it, so a client taken out of a
+ room and added back starts from its new arrival, never from where it had
+ got to before. `RoomStore.add_client` writes the first value, the room's
+ newest row, in the transaction that writes the membership.
"""
__tablename__ = "delivery_cursors"
__table_args__ = (
Index("ix_delivery_cursors_tenant_id", "tenant_id"),
- UniqueConstraint("agent_id", "room_id", name="uq_delivery_cursors_agent_room"),
- ForeignKeyConstraint(
- ["tenant_id", "agent_id"],
- ["agents.tenant_id", "agents.id"],
- name="fk_delivery_cursors_agent",
- ondelete="CASCADE",
- ),
ForeignKeyConstraint(
- ["tenant_id", "room_id"],
- ["rooms.tenant_id", "rooms.id"],
- name="fk_delivery_cursors_room",
+ ["tenant_id", "client_id", "room_id"],
+ [
+ "client_rooms.tenant_id",
+ "client_rooms.client_id",
+ "client_rooms.room_id",
+ ],
+ name="fk_delivery_cursors_membership",
ondelete="CASCADE",
),
)
- id: Mapped[str] = mapped_column(Text, primary_key=True, default=_uuid)
- agent_id: Mapped[str] = mapped_column(Text, nullable=False)
- room_id: Mapped[str] = mapped_column(Text, nullable=False)
- last_seq: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
- updated_at: Mapped[str] = mapped_column(
- DateTime(timezone=True),
- server_default=func.now(),
- onupdate=func.now(),
- nullable=False,
+ client_id: Mapped[str] = mapped_column(Text, primary_key=True)
+ room_id: Mapped[str] = mapped_column(Text, primary_key=True)
+ last_seq: Mapped[int] = mapped_column(BigInteger, nullable=False)
+ updated_at: Mapped[datetime] = mapped_column(
+ DateTime(timezone=True), server_default=func.now(), nullable=False
)
@@ -2379,13 +2471,14 @@ class AgentEventRow(TenantScoped, Base):
"""One of an agent's own events: an event meant for that agent, not one it
sent.
- The agent's consumer (`AgentConsumer`) writes a row when a room row it is
- handed turns out to be for the agent: a message addressed to it, a join it
- is set to listen for, a task event it is part of, or a command sent to it.
- Whether a row is for the agent depends on the room's state at the moment
- it is delivered, so the consumer records the decision here instead of
- leaving it to be worked out again at read time. The rest of a room's
- conversation is not copied here; a reader that wants it reads `messages`.
+ The router (`delivery/router.py`) writes a row, in the transaction that
+ writes the room row, when that room row is for the agent: a message
+ addressed to it, a join it is set to listen for, or a task event it is part
+ of. A room command's handler writes one for a command it passes on to the
+ agent. Whether a row is for the agent depends on the room's state at the
+ moment it is written, so the decision is recorded here instead of being
+ worked out again at read time. The rest of a room's conversation is not
+ copied here; a reader that wants it reads `messages`.
`seq` numbers the own events of one agent. The numbers are contiguous
and never reused (`AgentEventHead` hands them out). `payload` is the event
@@ -2401,12 +2494,12 @@ class AgentEventRow(TenantScoped, Base):
it has already received from here.
`message_key`, `room_id`, `room_seq` and `type` together name the source of
- an event,
- and an agent has at most one event per source (`uq_agent_events_source`).
- `EventBuffer.enqueue` hands back the number of the event already stored
- instead of storing a second copy. A file of a multi-file message that
- arrives after the message was delivered has a later `room_seq`, so it is
- stored as an event of its own.
+ an event, and an agent has at most one event per source
+ (`uq_agent_events_source`). The router stores nothing for a source already
+ stored, and `EventBuffer.enqueue` hands back the number of the event
+ already stored instead of storing a second copy. A file of a multi-file
+ message that arrives after the message was delivered has a later
+ `room_seq`, so it is stored as an event of its own.
The retention loop in `main.py` deletes rows older than
`agent_event_retention_seconds`. It never deletes from `messages`.
@@ -2423,9 +2516,10 @@ class AgentEventRow(TenantScoped, Base):
"room_id",
"room_seq",
),
- # One decision of the consumer is stored once. The same room row, read
- # by the consumer of a second process or written again after a commit
- # whose acknowledgement was lost, finds this row already there.
+ # One decision is stored once. The same room row decided again (by an
+ # agent consumer of an older release during a rolling deploy) or an
+ # event written again after a commit whose acknowledgement was lost
+ # finds this row already there.
Index(
"uq_agent_events_source",
"agent_id",
@@ -2519,6 +2613,54 @@ class AgentEventDrop(TenantScoped, Base):
dropped_through: Mapped[int] = mapped_column(BigInteger, nullable=False)
+class PendingAttachmentGroup(TenantScoped, Base):
+ """A multi-file post whose files have not all been written yet.
+
+ Several files posted as one message are written as one row each in
+ `messages`, sharing a group id (`attachments.py`). Whether the post is for
+ an agent, and the one own event it becomes, can only be decided once the
+ files are rows, so the router (`delivery/router.py`) records the group
+ here when a file arrives that does not complete it. The file that does
+ complete it is written in a transaction that also deletes this row and
+ stores the post's own events.
+
+ A group whose files stop arriving is decided with the files that did,
+ once `deadline` has passed (`MessageRouter.flush_due_groups`). Every
+ switch-core process runs that sweep, and each claims a row with `FOR
+ UPDATE SKIP LOCKED` and decides it in the transaction that deletes it, so
+ one process decides each group whichever processes are running.
+ `deadline` is set from the database's clock when the group's first file
+ is written and never moved, so the wait bounds the whole group rather
+ than the gap between two files.
+
+ `from_seq` is the room row this group's files are counted from: the
+ first file written since the group was last decided. A file that arrives
+ after its group was decided incomplete starts a new row here, from
+ itself, rather than completing a post that has already gone out.
+
+ A reader of a room's conversation holds a group back while it has a row
+ here, so the post is delivered as one message once it is decided.
+ """
+
+ __tablename__ = "pending_attachment_groups"
+ __table_args__ = (
+ # The sweep's question: which of a tenant's groups are due.
+ Index("ix_pending_attachment_groups_deadline", "tenant_id", "deadline"),
+ ForeignKeyConstraint(
+ ["tenant_id", "room_id"],
+ ["rooms.tenant_id", "rooms.id"],
+ name="fk_pending_attachment_groups_room",
+ ondelete="CASCADE",
+ ),
+ )
+
+ room_id: Mapped[str] = mapped_column(Text, primary_key=True)
+ group_id: Mapped[str] = mapped_column(Text, primary_key=True)
+ total: Mapped[int] = mapped_column(Integer, nullable=False)
+ from_seq: Mapped[int] = mapped_column(BigInteger, nullable=False)
+ deadline: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
+
+
class AgentRoomRead(TenantScoped, Base):
"""The baseline for an agent's unread count in one room, and who owns it.
@@ -2658,6 +2800,189 @@ class AgentPositionTicket(TenantScoped, Base):
)
+#: The first connection incarnation the database hands out. Before agent
+#: connections were recorded, each process drew incarnations from a counter
+#: seeded with a random 32-bit number, so every number a client may still hold
+#: from then is below 2**33 and none of them can match one drawn from here.
+CONNECTION_INCARNATION_FLOOR = 2**33
+
+#: The last one: every incarnation stays an integer a JavaScript client holds
+#: exactly.
+CONNECTION_INCARNATION_CEILING = 2**53 - 1
+
+agent_connection_incarnation_sequence = Sequence(
+ "agent_connection_incarnation",
+ metadata=Base.metadata,
+ start=CONNECTION_INCARNATION_FLOOR,
+ minvalue=CONNECTION_INCARNATION_FLOOR,
+ maxvalue=CONNECTION_INCARNATION_CEILING,
+ cycle=False,
+)
+
+
+class AgentConnectionRecord(TenantScoped, Base):
+ """An agent connection, as every switch-core process can see it.
+
+ The connection itself (its socket, its heartbeat, the frames it is owed)
+ lives in the memory of the process holding it (`agent_connections.py`).
+ This row is what the other processes need to know about it: which process
+ holds it (`process_id`, a `switch_core_processes` row), which incarnation
+ is current, and what the client declared. It is written when the
+ connection opens or is reattached and when it closes, never on a
+ heartbeat, and it reads as live only while `closed_at` is null and the
+ holding process's lease is held.
+
+ `incarnation` comes from `agent_connection_incarnation`, so the fence a
+ client names can be checked by any process, not only the one that issued
+ it.
+ """
+
+ __tablename__ = "agent_connections"
+ __table_args__ = (
+ Index("ix_agent_connections_tenant_id", "tenant_id"),
+ Index("ix_agent_connections_agent_id", "agent_id"),
+ UniqueConstraint("id", "tenant_id", name="uq_agent_connections_id_tenant"),
+ ForeignKeyConstraint(
+ ["tenant_id", "agent_id"],
+ ["agents.tenant_id", "agents.id"],
+ name="fk_agent_connections_agent",
+ ondelete="CASCADE",
+ ),
+ )
+
+ id: Mapped[str] = mapped_column(Text, primary_key=True)
+ agent_id: Mapped[str] = mapped_column(Text, nullable=False)
+ process_id: Mapped[str] = mapped_column(Text, nullable=False)
+ incarnation: Mapped[int] = mapped_column(BigInteger, nullable=False)
+ scope: Mapped[str] = mapped_column(Text, nullable=False)
+ delivery_filter: Mapped[str] = mapped_column(Text, nullable=False)
+ spawn_capable: Mapped[bool] = mapped_column(Boolean, nullable=False)
+ transport: Mapped[str] = mapped_column(Text, nullable=False)
+ speaks: Mapped[int | None] = mapped_column(Integer, nullable=True)
+ accepts: Mapped[int | None] = mapped_column(Integer, nullable=True)
+ attached_at: Mapped[datetime] = mapped_column(
+ DateTime(timezone=True), server_default=func.now(), nullable=False
+ )
+ closed_at: Mapped[datetime | None] = mapped_column(
+ DateTime(timezone=True), nullable=True
+ )
+ close_reason: Mapped[str | None] = mapped_column(Text, nullable=True)
+
+
+class AgentConnectionRoom(TenantScoped, Base):
+ """A room explicitly claimed on an agent connection.
+
+ The primary key is the rule that at most one connection of an agent acts
+ in a room, kept by the database so that two processes claiming the same
+ room for the same agent cannot both succeed. A claim counts only while its
+ connection's row is live.
+ """
+
+ __tablename__ = "agent_connection_rooms"
+ __table_args__ = (
+ Index("ix_agent_connection_rooms_tenant_id", "tenant_id"),
+ Index("ix_agent_connection_rooms_connection_id", "connection_id"),
+ ForeignKeyConstraint(
+ ["tenant_id", "agent_id"],
+ ["agents.tenant_id", "agents.id"],
+ name="fk_agent_connection_rooms_agent",
+ ondelete="CASCADE",
+ ),
+ ForeignKeyConstraint(
+ ["tenant_id", "room_id"],
+ ["rooms.tenant_id", "rooms.id"],
+ name="fk_agent_connection_rooms_room",
+ ondelete="CASCADE",
+ ),
+ ForeignKeyConstraint(
+ ["tenant_id", "connection_id"],
+ ["agent_connections.tenant_id", "agent_connections.id"],
+ name="fk_agent_connection_rooms_connection",
+ ondelete="CASCADE",
+ ),
+ )
+
+ agent_id: Mapped[str] = mapped_column(Text, primary_key=True)
+ room_id: Mapped[str] = mapped_column(Text, primary_key=True)
+ connection_id: Mapped[str] = mapped_column(Text, nullable=False)
+
+
+class AgentSessionPlacement(TenantScoped, Base):
+ """Which room one of an agent's sessions connected to, and over which
+ connection.
+
+ A session placed itself with `connect_to_room`, or its watcher stated it
+ with `/connection/placements`. One session of an agent acts in a room, so
+ a room appears once per agent. The placement belongs to the connection it
+ arrived on and goes with that connection's row.
+ """
+
+ __tablename__ = "agent_session_placements"
+ __table_args__ = (
+ Index("ix_agent_session_placements_tenant_id", "tenant_id"),
+ Index("ix_agent_session_placements_connection_id", "connection_id"),
+ UniqueConstraint(
+ "agent_id", "room_id", name="uq_agent_session_placements_agent_room"
+ ),
+ ForeignKeyConstraint(
+ ["tenant_id", "agent_id"],
+ ["agents.tenant_id", "agents.id"],
+ name="fk_agent_session_placements_agent",
+ ondelete="CASCADE",
+ ),
+ ForeignKeyConstraint(
+ ["tenant_id", "room_id"],
+ ["rooms.tenant_id", "rooms.id"],
+ name="fk_agent_session_placements_room",
+ ondelete="CASCADE",
+ ),
+ ForeignKeyConstraint(
+ ["tenant_id", "connection_id"],
+ ["agent_connections.tenant_id", "agent_connections.id"],
+ name="fk_agent_session_placements_connection",
+ ondelete="CASCADE",
+ ),
+ )
+
+ agent_id: Mapped[str] = mapped_column(Text, primary_key=True)
+ session_id: Mapped[str] = mapped_column(Text, primary_key=True)
+ room_id: Mapped[str] = mapped_column(Text, nullable=False)
+ connection_id: Mapped[str] = mapped_column(Text, nullable=False)
+
+
+class AgentSessionCommandRelay(TenantScoped, Base):
+ """A session command waiting for the process that can deliver it.
+
+ A command for one of an agent's sessions (from Switch Console, or a Stop
+ pressed on a bridge) can arrive at any process, while the stream it must
+ be written to (the agent's watcher, or its controller's) is held by one.
+ The command is stored here for that process (`process_id`) and the process
+ is told to look; it deletes the row as it takes the command. The signal
+ carries only ids, because a command's content is tenant data and a
+ `NOTIFY` is readable by any listener.
+ """
+
+ __tablename__ = "agent_session_command_relays"
+ __table_args__ = (
+ Index("ix_agent_session_command_relays_tenant_id", "tenant_id"),
+ Index("ix_agent_session_command_relays_process_id", "process_id"),
+ ForeignKeyConstraint(
+ ["tenant_id", "agent_id"],
+ ["agents.tenant_id", "agents.id"],
+ name="fk_agent_session_command_relays_agent",
+ ondelete="CASCADE",
+ ),
+ )
+
+ id: Mapped[str] = mapped_column(Text, primary_key=True, default=_uuid)
+ agent_id: Mapped[str] = mapped_column(Text, nullable=False)
+ process_id: Mapped[str] = mapped_column(Text, nullable=False)
+ frame: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
+ created_at: Mapped[datetime] = mapped_column(
+ DateTime(timezone=True), server_default=func.now(), nullable=False
+ )
+
+
class MediaBlob(TenantScoped, Base):
"""The bytes behind an attachment.
@@ -2832,6 +3157,20 @@ event.listen(
DDL(DROP_NOTIFY_TRIGGER).execute_if(dialect="postgresql"),
)
+# The trigger that signals every process when a client's membership changes,
+# so each process's copy of that client starts or stops reading the room. As
+# above, `create_all` builds it for the tests; real databases get it from a
+# migration. The SQL lives in `db/membership_notify_ddl.py`.
+for _ddl in (CREATE_MEMBERSHIP_NOTIFY_FUNCTION, CREATE_MEMBERSHIP_NOTIFY_TRIGGER):
+ event.listen(
+ ClientRoom.__table__, "after_create", DDL(_ddl).execute_if(dialect="postgresql")
+ )
+event.listen(
+ ClientRoom.__table__,
+ "before_drop",
+ DDL(DROP_MEMBERSHIP_NOTIFY_TRIGGER).execute_if(dialect="postgresql"),
+)
+
# These attach the trigger that sends a Postgres `NOTIFY` on channel
# `switch_agent_event` for every new `agent_events` row, so the server process
# waiting to deliver that agent's events wakes up. They run only when
@@ -3302,6 +3641,15 @@ class AgentController(TenantScoped, Base):
the current connection, or the last one once it has closed; a new one
replaces them. Every replica reads them, so whether the machine is
connected does not depend on which process holds its socket.
+
+ The `open_*` columns are the connection the controller last opened,
+ attached or not: its id, its generation, the process holding it and where
+ its stream resumes each agent. Opening and attaching the socket are two
+ requests that may reach two processes, as may a socket reattaching after a
+ drop, so the process the socket reaches reads them to hold the connection
+ itself, and records that it does. Written when the connection opens, when
+ another process takes it up and, for the cursors, when its socket goes;
+ never per heartbeat.
"""
__tablename__ = "agent_controllers"
@@ -3354,6 +3702,13 @@ class AgentController(TenantScoped, Base):
# `socket_closed`, `heartbeat_lapsed`, `taken_over`, `revoked` or
# `server_shutdown`.
disconnect_reason: Mapped[str | None] = mapped_column(Text, nullable=True)
+ open_connection_id: Mapped[str | None] = mapped_column(Text, nullable=True)
+ open_generation: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
+ # The process holding that connection: the one that opened it, or the one
+ # its socket reached since. Any other process holding a copy closes it.
+ open_process_id: Mapped[str | None] = mapped_column(Text, nullable=True)
+ # Agent id to the cursor its delivery resumes after, or null for the head.
+ open_cursors: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
@@ -3616,6 +3971,69 @@ class ServiceConnection(TenantScoped, Base):
)
+class SignInFlow(TenantScoped, Base):
+ """A person part-way through signing in to a service from Switch Console.
+
+ A sign-in is half a dozen requests, from Console and from the person's
+ browser, and with several switch-core processes behind one address each
+ can land on any of them, so the flow lives here rather than in the memory
+ of the process that started it (`connections/sign_in_flows.py`).
+
+ `id` is the state Console chose for the flow, which travels in the
+ vendor's redirect, so the browser's requests find the flow by it and the
+ flow's tenant through `tenant_of_sign_in_flow`. `kind` is `github` for the
+ GitHub App's own sign-in and `service` for the catalog's OAuth sign-ins:
+ a person has at most one flow of each kind at a time. `status` moves by
+ conditional update only, so of two requests racing for one step exactly
+ one takes it.
+
+ Nothing secret is stored in the clear: the PKCE verifier and, once the code
+ is exchanged, the vendor's credentials are keyring-encrypted, and the
+ completion secret and browser nonce are kept as SHA-256 hashes, compared
+ against what Console and the browser present. `details` holds what the
+ flow asked the vendor for (its name, scopes and consent), none of it
+ secret. Rows expire after ten minutes, and expired ones are deleted as the
+ next flow in the workspace starts.
+ """
+
+ __tablename__ = "sign_in_flows"
+ __table_args__ = (
+ CheckConstraint("kind IN ('github', 'service')", name="ck_sign_in_flows_kind"),
+ CheckConstraint(
+ "status IN ('pending', 'authorizing', 'returning', 'delivered', "
+ "'checking', 'ready', 'failed')",
+ name="ck_sign_in_flows_status",
+ ),
+ CheckConstraint("mode IN ('loopback', 'core')", name="ck_sign_in_flows_mode"),
+ Index("ix_sign_in_flows_user", "tenant_id", "user_id", "kind"),
+ )
+
+ id: Mapped[str] = mapped_column(Text, primary_key=True)
+ kind: Mapped[str] = mapped_column(Text, nullable=False)
+ service: Mapped[str] = mapped_column(Text, nullable=False)
+ user_id: Mapped[str] = mapped_column(
+ Text, ForeignKey("users.id", ondelete="CASCADE"), nullable=False
+ )
+ status: Mapped[str] = mapped_column(Text, nullable=False)
+ mode: Mapped[str] = mapped_column(Text, nullable=False)
+ port: Mapped[int] = mapped_column(Integer, nullable=False)
+ redirect_uri: Mapped[str] = mapped_column(Text, nullable=False)
+ owner_label: Mapped[str] = mapped_column(Text, nullable=False)
+ details: Mapped[dict] = mapped_column(JSONB, nullable=False)
+ encrypted_verifier: Mapped[str] = mapped_column(Text, nullable=False)
+ completion_secret_hash: Mapped[str] = mapped_column(Text, nullable=False)
+ browser_nonce_hash: Mapped[str | None] = mapped_column(Text, nullable=True)
+ encrypted_result: Mapped[str | None] = mapped_column(Text, nullable=True)
+ account_label: Mapped[str | None] = mapped_column(Text, nullable=True)
+ error: Mapped[str | None] = mapped_column(Text, nullable=True)
+ created_at: Mapped[datetime] = mapped_column(
+ DateTime(timezone=True), server_default=func.now(), nullable=False
+ )
+ expires_at: Mapped[datetime] = mapped_column(
+ DateTime(timezone=True), nullable=False
+ )
+
+
class ServiceOAuthClient(Base):
"""The OAuth client Core registered for itself at a service's vendor.
diff --git a/core/switch_core/db/rls_ddl.py b/core/switch_core/db/rls_ddl.py
index 8c476cb02..9947aa757 100644
--- a/core/switch_core/db/rls_ddl.py
+++ b/core/switch_core/db/rls_ddl.py
@@ -131,6 +131,12 @@ POLICY_NAME = "tenant_isolation"
# controller sockets for every tenant at once, so its lease belongs to none of
# them, and it holds no customer data, only that the process is alive.
#
+# `ownership_leases` says which of those processes runs each piece of work that
+# must run once, a client's consumer or a bridge's platform connection: a kind,
+# the work's id, the holding process and a counter. One process holds work for
+# every tenant at once and claims it in one statement across them, and a row
+# carries no customer data, only which process runs which id.
+#
# `cloud_machines` is a user's Switch cloud VM: one per person, whichever
# workspaces it serves, as the person is global. It holds the VM's lifecycle,
# disk and instance, and no workspace's data: what the machine runs for a
@@ -147,6 +153,11 @@ POLICY_NAME = "tenant_isolation"
# a vendor that offers no other way to have one. The vendor sees one Switch,
# so there is one client per deployment and service, shared by every tenant's
# sign-ins; it holds no person's account, only the deployment's own client.
+#
+# `schema_expand_only_revisions` records which applied schema revisions only
+# added to the schema (`db/schema_version.py`), as Alembic's own
+# `alembic_version` records which is applied. It describes the schema and
+# holds no data of anyone's.
GLOBAL_TABLES = frozenset(
{
"cloud_machines",
@@ -156,8 +167,10 @@ GLOBAL_TABLES = frozenset(
"telemetry_milestones",
"telemetry_snapshot_watermark",
"switch_core_processes",
+ "ownership_leases",
"trust_settings",
"service_oauth_clients",
+ "schema_expand_only_revisions",
}
)
diff --git a/core/switch_core/db/schema_version.py b/core/switch_core/db/schema_version.py
index 17a83d90e..26a968650 100644
--- a/core/switch_core/db/schema_version.py
+++ b/core/switch_core/db/schema_version.py
@@ -1,19 +1,77 @@
-"""Refuse to serve on a schema this build's migrations have not brought to head.
+"""Refuse to serve on a schema this build's migrations have not brought to head,
+unless the schema is ahead of it only by revisions it can serve on.
For a server that does not migrate at boot (`DB_MIGRATE_ON_BOOT=false`): the
migration ran somewhere else, as the schema owner, and this process holds only
the runtime role. The check is what stops a server whose migration step was
skipped or failed from serving against tables it does not expect.
+
+During a rolling deploy, though, the newer release migrates while the older
+one is still serving, and an older pod that restarts then (or a rollback)
+finds a head it has no file for. It accepts that head when every revision
+between it and its own head is recorded in `schema_expand_only_revisions`:
+each was marked `expand_only` (it only added to the schema) and recorded as
+Alembic applied it (`migrations/env.py`). Anything else is refused as before.
"""
from __future__ import annotations
+import logging
+from typing import Any
+
from alembic.config import Config as AlembicConfig
-from alembic.runtime.migration import MigrationContext
+from alembic.runtime.migration import MigrationContext, MigrationInfo
from alembic.script import ScriptDirectory
+from sqlalchemy import text
from sqlalchemy.engine import Connection
from sqlalchemy.ext.asyncio import AsyncEngine
+logger = logging.getLogger(__name__)
+
+# Far more revisions than one release ever runs ahead of another; a longer
+# chain is a cycle in the table, not a deploy.
+_MAX_CHAIN = 1000
+
+
+def record_expand_only(
+ ctx: MigrationContext,
+ step: MigrationInfo,
+ heads: set[str],
+ run_args: dict[str, Any],
+) -> None:
+ """Keep `schema_expand_only_revisions` in step with what is applied.
+
+ Runs after each revision, in its transaction. An upgrade through a
+ revision whose module sets `expand_only = True` records it with the
+ revision it follows; undoing one removes it. An older switch-core release,
+ which has no copy of the revision's file, reads these rows to decide
+ whether it can serve on the schema (`schema_ahead_through_expand_only`
+ below). Alembic calls it as `on_version_apply`, from `migrations/env.py`.
+ """
+ if step.is_stamp or ctx.connection is None:
+ return
+ connection = ctx.connection
+ if step.is_upgrade:
+ # A `Script` in practice, which carries the revision file's module.
+ script = step.up_revision
+ if not getattr(getattr(script, "module", None), "expand_only", False):
+ return
+ (down_revision,) = step.down_revision_ids
+ connection.execute(
+ text(
+ "INSERT INTO schema_expand_only_revisions (revision, down_revision) "
+ "VALUES (:revision, :down_revision) ON CONFLICT (revision) DO NOTHING"
+ ),
+ {"revision": step.up_revision_id, "down_revision": down_revision},
+ )
+ return
+ # The revision that created the table drops it as it is undone.
+ if connection.scalar(text("SELECT to_regclass('schema_expand_only_revisions')")):
+ connection.execute(
+ text("DELETE FROM schema_expand_only_revisions WHERE revision = :revision"),
+ {"revision": step.up_revision_id},
+ )
+
class SchemaNotAtHeadError(RuntimeError):
"""The database's Alembic revision is not this build's head."""
@@ -23,16 +81,82 @@ def _current_heads(connection: Connection) -> set[str]:
return set(MigrationContext.configure(connection).get_current_heads())
+def _expand_only_parents(connection: Connection) -> dict[str, str]:
+ """Each recorded expand-only revision, and the revision it follows. Empty
+ on a schema from before the table existed."""
+ if not connection.scalar(
+ text("SELECT to_regclass('schema_expand_only_revisions')")
+ ):
+ return {}
+ rows = connection.execute(
+ text("SELECT revision, down_revision FROM schema_expand_only_revisions")
+ )
+ return {revision: down for revision, down in rows.all()}
+
+
+def _ahead_through_expand_only(
+ current: set[str],
+ expected: set[str],
+ known: set[str],
+ parents: dict[str, str],
+) -> bool:
+ """Whether every head in `current` leads back to one of `expected` through
+ revisions this build does not know, each recorded as expand-only, and
+ every head in `expected` is reached."""
+ if not current:
+ return False
+ reached: set[str] = set()
+ for head in current:
+ revision = head
+ for _ in range(_MAX_CHAIN):
+ if revision in known:
+ break
+ parent = parents.get(revision)
+ if parent is None:
+ return False
+ revision = parent
+ else:
+ return False
+ if revision not in expected:
+ return False
+ reached.add(revision)
+ return reached == expected and current != expected
+
+
+async def schema_ahead_through_expand_only(
+ engine: AsyncEngine, alembic_cfg: AlembicConfig
+) -> bool:
+ """Whether the database is ahead of this build only by expand-only
+ revisions: this build can serve on it, and has nothing to migrate."""
+ script = ScriptDirectory.from_config(alembic_cfg)
+ expected = set(script.get_heads())
+ known = {revision.revision for revision in script.walk_revisions()}
+ async with engine.connect() as connection:
+ current = await connection.run_sync(_current_heads)
+ parents = await connection.run_sync(_expand_only_parents)
+ return _ahead_through_expand_only(current, expected, known, parents)
+
+
async def require_schema_at_head(
engine: AsyncEngine, alembic_cfg: AlembicConfig
) -> None:
expected = set(ScriptDirectory.from_config(alembic_cfg).get_heads())
async with engine.connect() as connection:
current = await connection.run_sync(_current_heads)
- if current != expected:
- raise SchemaNotAtHeadError(
- f"the database is at revision {sorted(current) or 'none'} but this "
- f"build expects {sorted(expected)}. DB_MIGRATE_ON_BOOT is false, so "
- "migrations must run before the server starts (switch-migrate, as "
- "the schema owner)."
+ if current == expected:
+ return
+ if await schema_ahead_through_expand_only(engine, alembic_cfg):
+ logger.warning(
+ "The database is at revision %s, ahead of this build's %s by "
+ "expand-only revisions only. Serving on it: a newer release has "
+ "migrated, and this one is still rolling out or being rolled back.",
+ sorted(current),
+ sorted(expected),
)
+ return
+ raise SchemaNotAtHeadError(
+ f"the database is at revision {sorted(current) or 'none'} but this "
+ f"build expects {sorted(expected)}. DB_MIGRATE_ON_BOOT is false, so "
+ "migrations must run before the server starts (switch-migrate, as "
+ "the schema owner)."
+ )
diff --git a/core/switch_core/db/stores/agent_connection_store.py b/core/switch_core/db/stores/agent_connection_store.py
new file mode 100644
index 000000000..870362e98
--- /dev/null
+++ b/core/switch_core/db/stores/agent_connection_store.py
@@ -0,0 +1,468 @@
+"""Agent connections as rows, so every switch-core process can see them.
+
+The process holding a connection's socket writes its row when the connection
+opens or is reattached and when it closes, and rewrites the agent's room claims
+and session placements whenever they move. Nothing here is written per
+heartbeat. A row is live while it is not closed and the process it names holds
+its lease in `switch_core_processes`, so a process that dies without writing
+its closings stops counting once its lease goes stale.
+
+Every read names the agent or the tenant as well as relying on the policy, for
+the reason `JoinDomainStore` gives: the owner connection the tests (and an
+unrestricted deployment) run on is exempt from row-level security.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import timedelta
+from typing import Any
+
+from sqlalchemy import (
+ ColumnElement,
+ and_,
+ delete,
+ exists,
+ func,
+ literal,
+ or_,
+ select,
+ text,
+ update,
+)
+from sqlalchemy.dialects.postgresql import insert
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from switch_core.db.models import (
+ AgentConnectionRecord,
+ AgentConnectionRoom,
+ AgentSessionCommandRelay,
+ AgentSessionPlacement,
+ SwitchCoreProcess,
+)
+
+
+@dataclass(frozen=True)
+class ConnectionRow:
+ """One `agent_connections` row, and whether its holder still holds it."""
+
+ id: str
+ agent_id: str
+ process_id: str
+ incarnation: int
+ scope: str
+ delivery_filter: str
+ spawn_capable: bool
+ transport: str
+ speaks: int | None
+ accepts: int | None
+ closed: bool
+ # The holding process's lease is held, or the holder is the process that
+ # asked. A process never reads its own rows as lost: its lease going stale
+ # means it could not reach the database, not that its sockets went.
+ held: bool
+
+ @property
+ def live(self) -> bool:
+ return self.held and not self.closed
+
+
+@dataclass(frozen=True)
+class AgentSlotRows:
+ """Everything recorded about one agent's connections."""
+
+ # Every row of the agent's connections, closed ones included.
+ connections: dict[str, ConnectionRow]
+ # Room -> the connection claiming it, as stored.
+ stored_claims: dict[str, str]
+ # Session -> (room, the connection that placed it), as stored.
+ stored_placements: dict[str, tuple[str, str]]
+
+ def live_connections(self) -> list[ConnectionRow]:
+ return [row for row in self.connections.values() if row.live]
+
+ def claims(self) -> dict[str, str]:
+ """The claims that count: those of a live connection."""
+ return {
+ room_id: connection_id
+ for room_id, connection_id in self.stored_claims.items()
+ if (row := self.connections.get(connection_id)) is not None and row.live
+ }
+
+ def placements(self) -> dict[str, tuple[str, str]]:
+ """The placements that count: those placed over a connection whose
+ process still holds its lease, closed or not. A session keeps its room
+ when the connection it arrived on lapses, as it did when this was kept
+ in memory, and loses it when that process goes, as a restart did."""
+ return {
+ session_id: placed
+ for session_id, placed in self.stored_placements.items()
+ if (row := self.connections.get(placed[1])) is not None and row.held
+ }
+
+
+def _held(process_id: str, lease_ttl: timedelta) -> ColumnElement[bool]:
+ return or_(
+ AgentConnectionRecord.process_id == process_id,
+ exists().where(
+ SwitchCoreProcess.id == AgentConnectionRecord.process_id,
+ SwitchCoreProcess.stopped_at.is_(None),
+ SwitchCoreProcess.beat_at > func.now() - literal(lease_ttl),
+ ),
+ )
+
+
+class AgentConnectionStore:
+ async def lock_agent(self, session: AsyncSession, agent_id: str) -> None:
+ """Hold the agent's room slots still until the transaction ends.
+
+ Every process that moves a claim or a placement, or opens a
+ connection, takes this first, so each move is made on what the others
+ have committed and none is lost between a read and a write."""
+ await session.execute(
+ select(func.pg_advisory_xact_lock(func.hashtext("agent-slots:" + agent_id)))
+ )
+
+ async def incarnations(self, session: AsyncSession, count: int) -> list[int]:
+ result = await session.execute(
+ text(
+ "SELECT nextval('agent_connection_incarnation') "
+ "FROM generate_series(1, :count)"
+ ),
+ {"count": count},
+ )
+ return [int(value) for value in result.scalars().all()]
+
+ async def read_agent(
+ self,
+ session: AsyncSession,
+ agent_id: str,
+ *,
+ process_id: str,
+ lease_ttl: timedelta,
+ ) -> AgentSlotRows:
+ found = await self._read(
+ session,
+ AgentConnectionRecord.agent_id == agent_id,
+ AgentConnectionRoom.agent_id == agent_id,
+ AgentSessionPlacement.agent_id == agent_id,
+ process_id=process_id,
+ lease_ttl=lease_ttl,
+ )
+ return found.get(
+ agent_id,
+ AgentSlotRows(connections={}, stored_claims={}, stored_placements={}),
+ )
+
+ async def read_tenant(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ *,
+ process_id: str,
+ lease_ttl: timedelta,
+ ) -> dict[str, AgentSlotRows]:
+ """Every agent of the tenant that has a connection row."""
+ return await self._read(
+ session,
+ AgentConnectionRecord.tenant_id == tenant_id,
+ AgentConnectionRoom.tenant_id == tenant_id,
+ AgentSessionPlacement.tenant_id == tenant_id,
+ process_id=process_id,
+ lease_ttl=lease_ttl,
+ )
+
+ async def _read(
+ self,
+ session: AsyncSession,
+ connections_where: ColumnElement[bool],
+ claims_where: ColumnElement[bool],
+ placements_where: ColumnElement[bool],
+ *,
+ process_id: str,
+ lease_ttl: timedelta,
+ ) -> dict[str, AgentSlotRows]:
+ rows = await session.execute(
+ select(
+ AgentConnectionRecord.id,
+ AgentConnectionRecord.agent_id,
+ AgentConnectionRecord.process_id,
+ AgentConnectionRecord.incarnation,
+ AgentConnectionRecord.scope,
+ AgentConnectionRecord.delivery_filter,
+ AgentConnectionRecord.spawn_capable,
+ AgentConnectionRecord.transport,
+ AgentConnectionRecord.speaks,
+ AgentConnectionRecord.accepts,
+ AgentConnectionRecord.closed_at.is_not(None),
+ _held(process_id, lease_ttl),
+ ).where(connections_where)
+ )
+ by_agent: dict[str, AgentSlotRows] = {}
+
+ def of(agent_id: str) -> AgentSlotRows:
+ return by_agent.setdefault(
+ agent_id,
+ AgentSlotRows(connections={}, stored_claims={}, stored_placements={}),
+ )
+
+ for (
+ connection_id,
+ agent_id,
+ holder,
+ incarnation,
+ scope,
+ delivery_filter,
+ spawn_capable,
+ transport,
+ speaks,
+ accepts,
+ closed,
+ held,
+ ) in rows.all():
+ of(agent_id).connections[connection_id] = ConnectionRow(
+ id=connection_id,
+ agent_id=agent_id,
+ process_id=holder,
+ incarnation=incarnation,
+ scope=scope,
+ delivery_filter=delivery_filter,
+ spawn_capable=spawn_capable,
+ transport=transport,
+ speaks=speaks,
+ accepts=accepts,
+ closed=bool(closed),
+ held=bool(held),
+ )
+ claims = await session.execute(
+ select(
+ AgentConnectionRoom.agent_id,
+ AgentConnectionRoom.room_id,
+ AgentConnectionRoom.connection_id,
+ ).where(claims_where)
+ )
+ for agent_id, room_id, connection_id in claims.all():
+ of(agent_id).stored_claims[room_id] = connection_id
+ placements = await session.execute(
+ select(
+ AgentSessionPlacement.agent_id,
+ AgentSessionPlacement.session_id,
+ AgentSessionPlacement.room_id,
+ AgentSessionPlacement.connection_id,
+ ).where(placements_where)
+ )
+ for agent_id, session_id, room_id, connection_id in placements.all():
+ of(agent_id).stored_placements[session_id] = (room_id, connection_id)
+ return by_agent
+
+ async def record_open(
+ self,
+ session: AsyncSession,
+ *,
+ tenant_id: str,
+ connection_id: str,
+ agent_id: str,
+ process_id: str,
+ incarnation: int,
+ scope: str,
+ delivery_filter: str,
+ spawn_capable: bool,
+ transport: str,
+ speaks: int | None,
+ accepts: int | None,
+ ) -> bool:
+ """The connection opened, or was reattached, on `process_id`. Returns
+ whether it was written: an id another agent's connection has is left
+ to that agent, whichever process recorded it."""
+ values: dict[str, Any] = {
+ "agent_id": agent_id,
+ "process_id": process_id,
+ "incarnation": incarnation,
+ "scope": scope,
+ "delivery_filter": delivery_filter,
+ "spawn_capable": spawn_capable,
+ "transport": transport,
+ "speaks": speaks,
+ "accepts": accepts,
+ "attached_at": func.now(),
+ "closed_at": None,
+ "close_reason": None,
+ }
+ result = await session.execute(
+ insert(AgentConnectionRecord)
+ .values(id=connection_id, tenant_id=tenant_id, **values)
+ .on_conflict_do_update(
+ index_elements=[AgentConnectionRecord.id],
+ set_=values,
+ where=AgentConnectionRecord.agent_id == agent_id,
+ )
+ .returning(AgentConnectionRecord.id)
+ )
+ return result.scalar_one_or_none() is not None
+
+ async def replace_slots(
+ self,
+ session: AsyncSession,
+ *,
+ tenant_id: str,
+ agent_id: str,
+ claims: dict[str, str],
+ placements: dict[str, tuple[str, str]],
+ ) -> None:
+ """Make these the agent's claims and placements. Called with the
+ agent locked (`lock_agent`), from a state read under the same lock."""
+ await session.execute(
+ delete(AgentConnectionRoom).where(AgentConnectionRoom.agent_id == agent_id)
+ )
+ await session.execute(
+ delete(AgentSessionPlacement).where(
+ AgentSessionPlacement.agent_id == agent_id
+ )
+ )
+ if claims:
+ await session.execute(
+ insert(AgentConnectionRoom),
+ [
+ {
+ "tenant_id": tenant_id,
+ "agent_id": agent_id,
+ "room_id": room_id,
+ "connection_id": connection_id,
+ }
+ for room_id, connection_id in claims.items()
+ ],
+ )
+ if placements:
+ await session.execute(
+ insert(AgentSessionPlacement),
+ [
+ {
+ "tenant_id": tenant_id,
+ "agent_id": agent_id,
+ "session_id": session_id,
+ "room_id": room_id,
+ "connection_id": connection_id,
+ }
+ for session_id, (room_id, connection_id) in placements.items()
+ ],
+ )
+
+ async def record_closed(
+ self,
+ session: AsyncSession,
+ *,
+ tenant_id: str,
+ connection_id: str,
+ incarnation: int,
+ process_id: str,
+ reason: str,
+ ) -> bool:
+ """The connection closed. Written only while the row still names this
+ process and this incarnation, so a process closing a connection that
+ has since been reattached elsewhere leaves the newer one standing.
+ Returns whether the row was this one to close."""
+ closed = await session.execute(
+ update(AgentConnectionRecord)
+ .where(
+ AgentConnectionRecord.tenant_id == tenant_id,
+ AgentConnectionRecord.id == connection_id,
+ AgentConnectionRecord.incarnation == incarnation,
+ AgentConnectionRecord.process_id == process_id,
+ AgentConnectionRecord.closed_at.is_(None),
+ )
+ .values(closed_at=func.now(), close_reason=reason)
+ .returning(AgentConnectionRecord.id)
+ )
+ if closed.first() is None:
+ return False
+ await session.execute(
+ delete(AgentConnectionRoom).where(
+ AgentConnectionRoom.tenant_id == tenant_id,
+ AgentConnectionRoom.connection_id == connection_id,
+ )
+ )
+ return True
+
+ async def add_relay(
+ self,
+ session: AsyncSession,
+ *,
+ tenant_id: str,
+ agent_id: str,
+ process_id: str,
+ frame: dict[str, Any],
+ ) -> None:
+ session.add(
+ AgentSessionCommandRelay(
+ tenant_id=tenant_id,
+ agent_id=agent_id,
+ process_id=process_id,
+ frame=frame,
+ )
+ )
+ await session.flush()
+
+ async def take_relays(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ *,
+ process_id: str,
+ agent_id: str | None,
+ ) -> list[tuple[str, dict[str, Any]]]:
+ """Delete and return the commands waiting for this process, for one
+ agent or for every agent of the tenant, oldest first, as (agent id,
+ frame)."""
+ where = [
+ AgentSessionCommandRelay.tenant_id == tenant_id,
+ AgentSessionCommandRelay.process_id == process_id,
+ ]
+ if agent_id is not None:
+ where.append(AgentSessionCommandRelay.agent_id == agent_id)
+ taken = await session.execute(
+ delete(AgentSessionCommandRelay)
+ .where(*where)
+ .returning(
+ AgentSessionCommandRelay.agent_id,
+ AgentSessionCommandRelay.frame,
+ AgentSessionCommandRelay.created_at,
+ )
+ )
+ rows = sorted(taken.all(), key=lambda row: row[2])
+ return [(relay_agent, frame) for relay_agent, frame, _ in rows]
+
+ async def prune(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ *,
+ closed_before: timedelta,
+ relays_before: timedelta,
+ ) -> int:
+ """Delete connections closed long ago or held by a process whose lease
+ is gone, with their claims and placements, and commands nobody took.
+ Returns how many connections."""
+ gone = await session.execute(
+ delete(AgentConnectionRecord)
+ .where(
+ AgentConnectionRecord.tenant_id == tenant_id,
+ or_(
+ AgentConnectionRecord.closed_at
+ < func.now() - literal(closed_before),
+ ~exists().where(
+ SwitchCoreProcess.id == AgentConnectionRecord.process_id
+ ),
+ ),
+ )
+ .returning(AgentConnectionRecord.id)
+ )
+ await session.execute(
+ delete(AgentSessionCommandRelay).where(
+ and_(
+ AgentSessionCommandRelay.tenant_id == tenant_id,
+ AgentSessionCommandRelay.created_at
+ < func.now() - literal(relays_before),
+ )
+ )
+ )
+ return len(gone.all())
diff --git a/core/switch_core/db/stores/agent_controller_store.py b/core/switch_core/db/stores/agent_controller_store.py
index b0fc90ab1..6df694d67 100644
--- a/core/switch_core/db/stores/agent_controller_store.py
+++ b/core/switch_core/db/stores/agent_controller_store.py
@@ -3,7 +3,7 @@ from __future__ import annotations
from datetime import datetime
from typing import Any
-from sqlalchemy import delete, func, or_, select, update
+from sqlalchemy import and_, delete, func, or_, select, update
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
@@ -279,15 +279,25 @@ class AgentControllerStore:
connected_at: datetime,
) -> bool:
"""Record that the controller's socket attached to this connection,
- held by this process. It replaces whatever the row held: the process
- holding the live socket is the authority. Returns whether the
- controller's row exists. `updated_at` is left alone: connecting is
- not an edit."""
+ held by this process. Written only while the row names this
+ connection as the one the controller opened last (`record_opened`),
+ held by this process, so an attach to an earlier connection whose
+ write lands late leaves the newer one standing. A row naming no
+ opened connection was never opened through a process that records
+ opens, and is written as it always was. Returns whether the row was
+ written. `updated_at` is left alone: connecting is not an edit."""
result = await session.execute(
update(AgentController)
.where(
AgentController.tenant_id == tenant_id,
AgentController.id == controller_id,
+ or_(
+ AgentController.open_connection_id.is_(None),
+ and_(
+ AgentController.open_connection_id == connection_id,
+ AgentController.open_process_id == process_id,
+ ),
+ ),
)
.values(
connection_id=connection_id,
@@ -309,13 +319,17 @@ class AgentControllerStore:
controller_id: str,
*,
connection_id: str,
+ process_id: str,
disconnected_at: datetime,
reason: str,
+ resume_cursors: dict[str, int | None],
) -> bool:
"""Record that the controller's socket on this connection went, and
- why. Written only while the row still names this connection, so a
- process closing a connection the controller has since replaced
- through another process leaves the replacement standing. Returns
+ why. Written only while the row still names this connection held by
+ this process, so a process closing a connection the controller has
+ since replaced, or reattached through another process, leaves the
+ newer one standing. Where the connection's stream resumes each agent is
+ kept with it, for whichever process the socket reattaches to. Returns
whether the row was written."""
result = await session.execute(
update(AgentController)
@@ -323,6 +337,7 @@ class AgentControllerStore:
AgentController.tenant_id == tenant_id,
AgentController.id == controller_id,
AgentController.connection_id == connection_id,
+ AgentController.connection_process_id == process_id,
)
.values(
disconnected_at=disconnected_at,
@@ -332,8 +347,94 @@ class AgentControllerStore:
.returning(AgentController.id)
.execution_options(synchronize_session=False)
)
+ written = result.scalar_one_or_none() is not None
+ if written:
+ await session.execute(
+ update(AgentController)
+ .where(
+ AgentController.tenant_id == tenant_id,
+ AgentController.id == controller_id,
+ AgentController.open_connection_id == connection_id,
+ )
+ .values(
+ open_cursors=resume_cursors,
+ updated_at=AgentController.updated_at,
+ )
+ .execution_options(synchronize_session=False)
+ )
+ return written
+
+ async def record_opened(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ controller_id: str,
+ *,
+ connection_id: str,
+ generation: int,
+ process_id: str,
+ resume_cursors: dict[str, int | None],
+ ) -> None:
+ """Record the connection the controller just opened through this
+ process, so whichever process its socket reaches can hold it. Its
+ socket attaching is recorded separately (`record_connected`): opened
+ is not connected."""
+ await session.execute(
+ update(AgentController)
+ .where(
+ AgentController.tenant_id == tenant_id,
+ AgentController.id == controller_id,
+ )
+ .values(
+ open_connection_id=connection_id,
+ open_generation=generation,
+ open_process_id=process_id,
+ open_cursors=resume_cursors,
+ updated_at=AgentController.updated_at,
+ )
+ .execution_options(synchronize_session=False)
+ )
+
+ async def record_taken_up(
+ self,
+ session: AsyncSession,
+ tenant_id: str,
+ controller_id: str,
+ *,
+ connection_id: str,
+ process_id: str,
+ ) -> bool:
+ """Record that this process now holds the connection the controller
+ last opened. Written only while it is still that connection. Returns
+ whether it was."""
+ result = await session.execute(
+ update(AgentController)
+ .where(
+ AgentController.tenant_id == tenant_id,
+ AgentController.id == controller_id,
+ AgentController.open_connection_id == connection_id,
+ )
+ .values(open_process_id=process_id, updated_at=AgentController.updated_at)
+ .returning(AgentController.id)
+ .execution_options(synchronize_session=False)
+ )
return result.scalar_one_or_none() is not None
+ async def list_with_connections(
+ self, session: AsyncSession, tenant_id: str
+ ) -> list[AgentController]:
+ """Every controller of the tenant that has opened a connection."""
+ result = await session.execute(
+ select(AgentController).where(
+ AgentController.tenant_id == tenant_id,
+ or_(
+ AgentController.connection_id.is_not(None),
+ AgentController.open_connection_id.is_not(None),
+ ),
+ )
+ )
+ return list(result.scalars().all())
+
async def set_credential(
self,
session: AsyncSession,
diff --git a/core/switch_core/db/stores/agent_store.py b/core/switch_core/db/stores/agent_store.py
index d9b647faa..aeab82932 100644
--- a/core/switch_core/db/stores/agent_store.py
+++ b/core/switch_core/db/stores/agent_store.py
@@ -110,6 +110,18 @@ class AgentStore:
result = await session.execute(select(Agent))
return list(result.scalars().all())
+ async def get_by_server_connector(
+ self, session: AsyncSession, connector_id: str
+ ) -> list[Agent]:
+ """The agents a server-side connector registered, in the bound tenant:
+ each carries the connector's id in its metadata."""
+ result = await session.execute(
+ select(Agent)
+ .where(Agent.metadata_["server_connector_id"].astext == connector_id)
+ .order_by(Agent.name)
+ )
+ return list(result.scalars().all())
+
async def get_by_owner(self, session: AsyncSession, owner_id: str) -> list[Agent]:
"""Every agent `owner_id` owns in the bound tenant.
diff --git a/core/switch_core/db/stores/bridge_message_map_store.py b/core/switch_core/db/stores/bridge_message_map_store.py
index a406fcce9..752ec63e2 100644
--- a/core/switch_core/db/stores/bridge_message_map_store.py
+++ b/core/switch_core/db/stores/bridge_message_map_store.py
@@ -1,10 +1,83 @@
-from sqlalchemy import delete, select
+from datetime import timedelta
+
+from sqlalchemy import delete, func, select, update
+from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
from switch_core.db.models import BridgeMessageMap
class BridgeMessageMapStore:
+ async def claim(
+ self,
+ session: AsyncSession,
+ *,
+ bridge_id: str,
+ external_channel_id: str,
+ transport_event_id: str,
+ unposted_after: timedelta,
+ ) -> bool:
+ """Claim a room message for posting to the platform: a row with no
+ external post yet. False when the message has a row already, posted,
+ or claimed within `unposted_after` and not posted yet. A claim older
+ than that with no post is claimed again, from now and for
+ `external_channel_id`, the channel the post goes to: the room may
+ have moved to another channel since the first claim."""
+ statement = insert(BridgeMessageMap).values(
+ bridge_id=bridge_id,
+ external_channel_id=external_channel_id,
+ transport_event_id=transport_event_id,
+ external_post_id=None,
+ )
+ result = await session.execute(
+ statement.on_conflict_do_update(
+ index_elements=[
+ BridgeMessageMap.bridge_id,
+ BridgeMessageMap.transport_event_id,
+ ],
+ set_={
+ "created_at": func.now(),
+ "external_channel_id": statement.excluded.external_channel_id,
+ },
+ where=(
+ BridgeMessageMap.external_post_id.is_(None)
+ & (BridgeMessageMap.created_at < func.now() - unposted_after)
+ ),
+ ).returning(BridgeMessageMap.id)
+ )
+ return result.scalar_one_or_none() is not None
+
+ async def fill(
+ self,
+ session: AsyncSession,
+ *,
+ bridge_id: str,
+ transport_event_id: str,
+ external_post_id: str,
+ ) -> None:
+ """Record the post a claimed message became on the platform."""
+ await session.execute(
+ update(BridgeMessageMap)
+ .where(
+ BridgeMessageMap.bridge_id == bridge_id,
+ BridgeMessageMap.transport_event_id == transport_event_id,
+ BridgeMessageMap.external_post_id.is_(None),
+ )
+ .values(external_post_id=external_post_id)
+ )
+
+ async def release_claim(
+ self, session: AsyncSession, *, bridge_id: str, transport_event_id: str
+ ) -> None:
+ """Drop a claim the platform never turned into a post."""
+ await session.execute(
+ delete(BridgeMessageMap).where(
+ BridgeMessageMap.bridge_id == bridge_id,
+ BridgeMessageMap.transport_event_id == transport_event_id,
+ BridgeMessageMap.external_post_id.is_(None),
+ )
+ )
+
async def create(
self, session: AsyncSession, mapping: BridgeMessageMap
) -> BridgeMessageMap:
diff --git a/core/switch_core/db/stores/delivery_cursor_store.py b/core/switch_core/db/stores/delivery_cursor_store.py
index 5d3bd45b1..4e374ecd2 100644
--- a/core/switch_core/db/stores/delivery_cursor_store.py
+++ b/core/switch_core/db/stores/delivery_cursor_store.py
@@ -1,13 +1,20 @@
-"""Where each agent has been delivered to in each room."""
+"""How far each client's consumer has handled each room (`delivery_cursors`).
+
+Read and written in a session bound to the client's tenant: the table is
+scoped, and every row belongs to a membership in that tenant.
+"""
from __future__ import annotations
+from dataclasses import dataclass
+from datetime import timedelta
from typing import TYPE_CHECKING
-from sqlalchemy import func, select
-from sqlalchemy.dialects.postgresql import insert
+from sqlalchemy import BigInteger, bindparam, func, literal, select, text
+from sqlalchemy.dialects.postgresql import ARRAY, insert
+from sqlalchemy.types import Text
-from switch_core.db.models import DeliveryCursor
+from switch_core.db.models import DeliveryCursor, Message
if TYPE_CHECKING:
from collections.abc import Collection
@@ -15,63 +22,186 @@ if TYPE_CHECKING:
from sqlalchemy.ext.asyncio import AsyncSession
+@dataclass(frozen=True)
+class CursorAdvance:
+ """Move one client's cursor in one room to `seq`, under the lease that
+ makes this process the client's reader: `kind` and `key` name the lease,
+ `token` is the one this process claimed it with."""
+
+ client_id: str
+ room_id: str
+ seq: int
+ kind: str
+ key: str
+ token: int
+
+
+# The advance and its fence in one statement. A cursor moves only while this
+# process holds the lease under the token it claimed, and only while the
+# membership exists. The lease row is locked for share, so a process taking
+# the lease over waits for this statement to commit and then reads the cursor
+# as moved; the membership row is locked against deletion for the same span,
+# so a removal cannot slip in between the check and the write. A cursor only
+# ever moves forward.
+_ADVANCE = text(
+ """
+ WITH wanted AS (
+ SELECT * FROM unnest(:client_ids, :room_ids, :seqs, :kinds, :keys, :tokens)
+ AS wanted(client_id, room_id, seq, kind, key, token)
+ ),
+ fenced AS (
+ SELECT member.tenant_id, wanted.client_id, wanted.room_id, wanted.seq
+ FROM wanted
+ JOIN ownership_leases lease
+ ON lease.kind = wanted.kind
+ AND lease.key = wanted.key
+ AND lease.token = wanted.token
+ AND lease.process_id = :process_id
+ JOIN client_rooms member
+ ON member.client_id = wanted.client_id
+ AND member.room_id = wanted.room_id
+ FOR SHARE OF lease FOR KEY SHARE OF member
+ )
+ INSERT INTO delivery_cursors AS cursor (tenant_id, client_id, room_id, last_seq)
+ SELECT tenant_id, client_id, room_id, seq FROM fenced
+ ON CONFLICT (client_id, room_id) DO UPDATE
+ SET last_seq = GREATEST(cursor.last_seq, EXCLUDED.last_seq),
+ updated_at = now()
+ RETURNING cursor.client_id, cursor.room_id
+ """
+).bindparams(
+ bindparam("client_ids", type_=ARRAY(Text)),
+ bindparam("room_ids", type_=ARRAY(Text)),
+ bindparam("seqs", type_=ARRAY(BigInteger)),
+ bindparam("kinds", type_=ARRAY(Text)),
+ bindparam("keys", type_=ARRAY(Text)),
+ bindparam("tokens", type_=ARRAY(BigInteger)),
+)
+
+# A cursor at its room's newest row for each of the client's memberships in
+# the given rooms that has none.
+_SEED_MEMBERSHIPS = text(
+ """
+ INSERT INTO delivery_cursors (tenant_id, client_id, room_id, last_seq)
+ SELECT member.tenant_id, member.client_id, member.room_id,
+ GREATEST(COALESCE(
+ (SELECT max(m.seq) FROM messages m WHERE m.room_id = member.room_id),
+ 0
+ ), 0)
+ FROM client_rooms member
+ WHERE member.client_id = :client_id AND member.room_id = ANY(:room_ids)
+ ON CONFLICT (client_id, room_id) DO NOTHING
+ """
+).bindparams(bindparam("room_ids", type_=ARRAY(Text)))
+
+# For each room, the newest row after the given place that is older than the
+# cut-off: a consumer resuming there skips through it.
+_BACKLOG_ENDS = text(
+ """
+ SELECT place.room_id, max(m.seq)
+ FROM unnest(:room_ids, :seqs) AS place(room_id, seq)
+ JOIN messages m ON m.room_id = place.room_id AND m.seq > place.seq
+ WHERE m.sent_at < now() - CAST(:older_than AS interval)
+ GROUP BY place.room_id
+ """
+).bindparams(
+ bindparam("room_ids", type_=ARRAY(Text)),
+ bindparam("seqs", type_=ARRAY(BigInteger)),
+)
+
+
+def _room_head(room_id: str) -> object:
+ """The room's newest live row, or 0 for a room with none. Reconstructed
+ history is numbered below zero and is never delivered."""
+ return func.greatest(
+ select(func.coalesce(func.max(Message.seq), 0))
+ .where(Message.room_id == room_id)
+ .scalar_subquery(),
+ literal(0, BigInteger),
+ )
+
+
class DeliveryCursorStore:
- async def position(self, session: AsyncSession, agent_id: str, room_id: str) -> int:
- """How far this agent has been delivered, 0 if it never has.
-
- Zero rather than None because live messages are numbered from 1: an
- agent with no cursor is behind the room's first live message, which is
- the same statement the number already makes. Reconstructed history sits
- below zero and is not delivered.
- """
+ async def positions(
+ self, session: AsyncSession, client_id: str, room_ids: Collection[str]
+ ) -> dict[str, int]:
+ """The client's cursor in each of `room_ids` that has one."""
+ if not room_ids:
+ return {}
result = await session.execute(
- select(DeliveryCursor.last_seq).where(
- DeliveryCursor.agent_id == agent_id,
- DeliveryCursor.room_id == room_id,
+ select(DeliveryCursor.room_id, DeliveryCursor.last_seq).where(
+ DeliveryCursor.client_id == client_id,
+ DeliveryCursor.room_id.in_(list(room_ids)),
)
)
- return result.scalar_one_or_none() or 0
+ return dict(result.tuples().all())
- async def positions_in(
- self, session: AsyncSession, room_id: str, agent_ids: Collection[str]
- ) -> dict[str, int]:
- """Several agents' positions in one room, for dispatching a room once.
-
- Agents with no cursor are absent from the map rather than present at 0,
- so a caller can tell "never delivered" from "delivered nothing".
- """
- if not agent_ids:
- return {}
- result = await session.execute(
- select(DeliveryCursor.agent_id, DeliveryCursor.last_seq).where(
- DeliveryCursor.room_id == room_id,
- DeliveryCursor.agent_id.in_(list(agent_ids)),
+ async def seed(self, session: AsyncSession, client_id: str, room_id: str) -> None:
+ """Start the client's cursor in the room at the room's newest row,
+ unless it has one. Written with the membership, in the caller's
+ transaction, so a consumer that learns of the membership later still
+ starts at the moment the client joined."""
+ await session.execute(
+ insert(DeliveryCursor)
+ .values(client_id=client_id, room_id=room_id, last_seq=_room_head(room_id))
+ .on_conflict_do_nothing(
+ index_elements=[DeliveryCursor.client_id, DeliveryCursor.room_id]
)
)
- return {agent_id: last_seq for agent_id, last_seq in result.all()}
- async def advance(
- self, session: AsyncSession, agent_id: str, room_id: str, seq: int
+ async def seed_memberships(
+ self, session: AsyncSession, client_id: str, room_ids: Collection[str]
) -> None:
- """Move the cursor forward to `seq`, never backward.
-
- `GREATEST` rather than an assignment because two deliveries for the
- same agent can finish out of order — a retry after a slow read, two
- workers on the same room — and the later write must not undo the
- further one. Rewinding a cursor redelivers, and a redelivered message
- is indistinguishable from a new one to whoever reads it.
- """
- statement = insert(DeliveryCursor).values(
- agent_id=agent_id, room_id=room_id, last_seq=seq
- )
+ """`seed` for each of `room_ids` the client is a member of and has no
+ cursor in, in one statement: for memberships written before cursors
+ were, or by something that writes none."""
+ if not room_ids:
+ return
await session.execute(
- statement.on_conflict_do_update(
- constraint="uq_delivery_cursors_agent_room",
- set_={
- "last_seq": func.greatest(
- DeliveryCursor.last_seq, statement.excluded.last_seq
- ),
- "updated_at": func.now(),
- },
- )
+ _SEED_MEMBERSHIPS, {"client_id": client_id, "room_ids": list(room_ids)}
+ )
+
+ async def advance(
+ self,
+ session: AsyncSession,
+ *,
+ process_id: str,
+ advances: Collection[CursorAdvance],
+ ) -> set[tuple[str, str]]:
+ """Move each cursor forward where this process still holds the lease
+ and the membership still exists. Returns the (client, room) pairs
+ written; the rest were refused. One (client, room) pair at most once
+ in `advances`."""
+ if not advances:
+ return set()
+ rows = list(advances)
+ result = await session.execute(
+ _ADVANCE,
+ {
+ "client_ids": [row.client_id for row in rows],
+ "room_ids": [row.room_id for row in rows],
+ "seqs": [row.seq for row in rows],
+ "kinds": [row.kind for row in rows],
+ "keys": [row.key for row in rows],
+ "tokens": [row.token for row in rows],
+ "process_id": process_id,
+ },
+ )
+ return {(client_id, room_id) for client_id, room_id in result.tuples().all()}
+
+ async def backlog_ends(
+ self, session: AsyncSession, places: dict[str, int], older_than: timedelta
+ ) -> dict[str, int]:
+ """For each room in `places` (room to cursor), the newest row past the
+ cursor that is older than `older_than`, where there is one."""
+ if not places:
+ return {}
+ result = await session.execute(
+ _BACKLOG_ENDS,
+ {
+ "room_ids": list(places),
+ "seqs": list(places.values()),
+ "older_than": older_than,
+ },
)
+ return dict(result.tuples().all())
diff --git a/core/switch_core/db/stores/external_user_store.py b/core/switch_core/db/stores/external_user_store.py
index 8c5767ed4..96467f230 100644
--- a/core/switch_core/db/stores/external_user_store.py
+++ b/core/switch_core/db/stores/external_user_store.py
@@ -5,7 +5,7 @@ from sqlalchemy import CursorResult, delete, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession
-from switch_core.db.models import ClientRoom, ExternalUser, ExternalUserClaim
+from switch_core.db.models import Client, ClientRoom, ExternalUser, ExternalUserClaim
class ExternalUserStore:
@@ -14,6 +14,22 @@ class ExternalUserStore:
await session.flush()
return user
+ async def is_bridge_user(
+ self, session: AsyncSession, bridge_id: str, transport_user_id: str
+ ) -> bool:
+ """Whether `transport_user_id` is the human actor of a person on this
+ bridge."""
+ result = await session.execute(
+ select(ExternalUser.id)
+ .join(Client, Client.id == ExternalUser.client_id)
+ .where(
+ ExternalUser.bridge_id == bridge_id,
+ Client.transport_user_id == transport_user_id,
+ )
+ .limit(1)
+ )
+ return result.scalar_one_or_none() is not None
+
async def get(self, session: AsyncSession, user_id: str) -> ExternalUser | None:
return await session.get(ExternalUser, user_id)
diff --git a/core/switch_core/db/stores/message_store.py b/core/switch_core/db/stores/message_store.py
index 5048210ee..2acd356ab 100644
--- a/core/switch_core/db/stores/message_store.py
+++ b/core/switch_core/db/stores/message_store.py
@@ -31,6 +31,33 @@ class MessageStore:
await session.flush()
return message
+ async def create_unless_stored(
+ self,
+ session: AsyncSession,
+ message: Message,
+ attachments: list[MessageAttachment],
+ ) -> bool:
+ """`create`, unless a message with the same `transport_event_id` is
+ already stored, in which case nothing is written. Returns whether the
+ message was written.
+
+ For writes whose id is derived from what caused them, so causing them
+ twice writes once. The check runs under the room's lock, which every
+ writer to the room holds to commit, so an earlier write of the same
+ message has either committed and is found, or has not started.
+ """
+ message.seq = await self._next_seq(session, message.room_id)
+ if await self.get_by_transport_event_id(session, message.transport_event_id):
+ return False
+ session.add(message)
+ await session.flush()
+ for position, attachment in enumerate(attachments):
+ attachment.message_id = message.id
+ attachment.position = position
+ session.add(attachment)
+ await session.flush()
+ return True
+
async def create_historical(
self,
session: AsyncSession,
diff --git a/core/switch_core/db/stores/messaging_install_store.py b/core/switch_core/db/stores/messaging_install_store.py
index e3046c7b3..b72f1d580 100644
--- a/core/switch_core/db/stores/messaging_install_store.py
+++ b/core/switch_core/db/stores/messaging_install_store.py
@@ -80,10 +80,19 @@ class MessagingInstallStore:
return state
async def redeem_state(
- self, session: AsyncSession, *, state_id: str, platform: str
+ self,
+ session: AsyncSession,
+ *,
+ state_id: str,
+ platform: str,
+ proposed_in: str | None = None,
) -> MessagingInstallState:
"""Burn a state, or raise. Exactly one caller can succeed.
+ `proposed_in` is the chat a claim-based install is put to as it is
+ burnt, recorded in the same statement: only an answer from that chat
+ may decide it (`decide_state`).
+
The check and the write are one statement on purpose. Reading the row,
deciding it is unused and then updating it leaves a window in which two
callbacks both read `consumed_at IS NULL` — and the whole reason this
@@ -104,7 +113,7 @@ class MessagingInstallStore:
MessagingInstallState.consumed_at.is_(None),
MessagingInstallState.expires_at > datetime.now(UTC),
)
- .values(consumed_at=datetime.now(UTC))
+ .values(consumed_at=datetime.now(UTC), proposed_chat_id=proposed_in)
.returning(MessagingInstallState)
)
state = result.scalars().one_or_none()
@@ -115,6 +124,28 @@ class MessagingInstallStore:
)
return state
+ async def proposed_chat(
+ self,
+ session: AsyncSession,
+ *,
+ state_id: str,
+ platform: str,
+ window: timedelta,
+ ) -> str | None:
+ """The chat a redeemed claim state was proposed in, while an answer
+ there can still decide it: redeemed within `window` and undecided."""
+ now = datetime.now(UTC)
+ chat_id: str | None = await session.scalar(
+ select(MessagingInstallState.proposed_chat_id).where(
+ MessagingInstallState.id == state_id,
+ MessagingInstallState.tenant_id == require_tenant_id(),
+ MessagingInstallState.platform == platform,
+ MessagingInstallState.consumed_at > now - window,
+ MessagingInstallState.decided_at.is_(None),
+ )
+ )
+ return chat_id
+
async def decide_state(
self,
session: AsyncSession,
@@ -122,14 +153,21 @@ class MessagingInstallStore:
state_id: str,
platform: str,
window: timedelta,
+ proposed_in: str | None = None,
) -> MessagingInstallState:
"""Record the approver's Connect or Cancel on a redeemed state, or raise.
One statement for the same reason as `redeem_state`: two submissions
of the same confirmation page must not both go through. Only a state
- redeemed within `window` can be decided.
+ redeemed within `window` can be decided, and with `proposed_in` only
+ one proposed in that chat.
"""
now = datetime.now(UTC)
+ proposed = (
+ ()
+ if proposed_in is None
+ else (MessagingInstallState.proposed_chat_id == proposed_in,)
+ )
result = await session.execute(
update(MessagingInstallState)
.where(
@@ -139,6 +177,7 @@ class MessagingInstallStore:
MessagingInstallState.consumed_at.is_not(None),
MessagingInstallState.consumed_at > now - window,
MessagingInstallState.decided_at.is_(None),
+ *proposed,
)
.values(decided_at=now)
.returning(MessagingInstallState)
diff --git a/core/switch_core/db/stores/ownership_lease_store.py b/core/switch_core/db/stores/ownership_lease_store.py
new file mode 100644
index 000000000..53d0fd10a
--- /dev/null
+++ b/core/switch_core/db/stores/ownership_lease_store.py
@@ -0,0 +1,153 @@
+"""Which switch-core process holds each piece of work (`ownership_leases`).
+
+A global table, read and written with no tenant bound: one process holds work
+for every tenant at once. Every time here is the database's own clock, the
+one `switch_core_processes` is renewed on, so whether a holder's process lease
+still holds is answered by the same clock that stamped it.
+"""
+
+from __future__ import annotations
+
+from datetime import timedelta
+from typing import TYPE_CHECKING
+
+from sqlalchemy import BigInteger, bindparam, text, update
+from sqlalchemy.dialects.postgresql import ARRAY
+from sqlalchemy.types import Text
+
+from switch_core.db.models import OwnershipLease
+
+if TYPE_CHECKING:
+ from collections.abc import Collection
+
+ from sqlalchemy.ext.asyncio import AsyncSession
+
+# A row may be taken when nobody holds it, when this process holds it from
+# before (so a stale hold of its own is fenced off by the new token), or when
+# its holder's process lease no longer holds: stopped, or not renewed within
+# the lease's lifetime. The new token is one more than the last, whoever held
+# it, so a holder that was taken over is refused by everything it still tries.
+_CLAIM = text(
+ """
+ INSERT INTO ownership_leases AS lease (kind, key, process_id, token, changed_at)
+ SELECT :kind, claimed.key, :process_id, 1, now()
+ FROM unnest(:keys) AS claimed(key)
+ ON CONFLICT (kind, key) DO UPDATE
+ SET process_id = EXCLUDED.process_id,
+ token = lease.token + 1,
+ changed_at = now()
+ WHERE lease.process_id IS NULL
+ OR lease.process_id = EXCLUDED.process_id
+ OR NOT EXISTS (
+ SELECT 1 FROM switch_core_processes holder
+ WHERE holder.id = lease.process_id
+ AND holder.stopped_at IS NULL
+ AND holder.beat_at > now() - CAST(:lease_ttl AS interval)
+ )
+ RETURNING lease.key, lease.token
+ """
+).bindparams(bindparam("keys", type_=ARRAY(Text)))
+
+
+class OwnershipLeaseStore:
+ async def claim(
+ self,
+ session: AsyncSession,
+ *,
+ kind: str,
+ keys: Collection[str],
+ process_id: str,
+ lease_ttl: timedelta,
+ ) -> dict[str, int]:
+ """Take every one of `keys` that may be taken, in one statement.
+
+ Returns the keys taken, each with its new token. A key held by a
+ process whose lease still holds is not returned and stays its.
+ """
+ if not keys:
+ return {}
+ result = await session.execute(
+ _CLAIM,
+ {
+ "kind": kind,
+ "keys": list(keys),
+ "process_id": process_id,
+ "lease_ttl": lease_ttl,
+ },
+ )
+ return {key: token for key, token in result.tuples().all()}
+
+ async def release(
+ self,
+ session: AsyncSession,
+ *,
+ kind: str,
+ keys: Collection[str],
+ process_id: str,
+ ) -> None:
+ """Let go of `keys`, where this process still holds them. The token
+ is kept, so the next claim moves past every one handed out before."""
+ if not keys:
+ return
+ await session.execute(
+ update(OwnershipLease)
+ .where(
+ OwnershipLease.kind == kind,
+ OwnershipLease.key.in_(list(keys)),
+ OwnershipLease.process_id == process_id,
+ )
+ .values(process_id=None, changed_at=text("now()"))
+ )
+
+ async def release_all(self, session: AsyncSession, process_id: str) -> None:
+ """Let go of everything this process holds, as it shuts down."""
+ await session.execute(
+ update(OwnershipLease)
+ .where(OwnershipLease.process_id == process_id)
+ .values(process_id=None, changed_at=text("now()"))
+ )
+
+ async def held(
+ self,
+ session: AsyncSession,
+ *,
+ kind: str,
+ tokens: dict[str, int],
+ process_id: str,
+ ) -> set[str]:
+ """Which of `tokens` this process still holds under the token given."""
+ if not tokens:
+ return set()
+ result = await session.execute(
+ text(
+ "SELECT lease.key FROM ownership_leases lease "
+ "JOIN unnest(:keys, :tokens) AS mine(key, token) "
+ "ON lease.key = mine.key AND lease.token = mine.token "
+ "WHERE lease.kind = :kind AND lease.process_id = :process_id"
+ ).bindparams(
+ bindparam("keys", type_=ARRAY(Text)),
+ bindparam("tokens", type_=ARRAY(BigInteger)),
+ ),
+ {
+ "kind": kind,
+ "keys": list(tokens),
+ "tokens": list(tokens.values()),
+ "process_id": process_id,
+ },
+ )
+ return set(result.scalars().all())
+
+ async def prune(self, session: AsyncSession, older_than: timedelta) -> int:
+ """Delete rows nobody has held for `older_than`. Work that still
+ exists is claimed again within seconds, so such a row belongs to
+ work that is gone; one that comes back is claimed afresh."""
+ result = await session.execute(
+ text(
+ "DELETE FROM ownership_leases "
+ "WHERE process_id IS NULL "
+ "AND changed_at < now() - CAST(:older_than AS interval) "
+ "RETURNING key"
+ ),
+ {"older_than": older_than},
+ )
+ return len(result.all())
diff --git a/core/switch_core/db/stores/room_role_store.py b/core/switch_core/db/stores/room_role_store.py
index c7e13e5b7..214121e17 100644
--- a/core/switch_core/db/stores/room_role_store.py
+++ b/core/switch_core/db/stores/room_role_store.py
@@ -278,6 +278,23 @@ class RoomRoleStore:
)
return result.scalar_one_or_none()
+ async def room_role_names(
+ self,
+ session: AsyncSession,
+ room_id: str,
+ live_connection_ids: Collection[str],
+ ) -> dict[str, str]:
+ """`agent_room_role` for every agent in one room, in one query: the role
+ name each agent live-holds there, keyed by agent id. An agent holding
+ none has no key."""
+ result = await session.execute(
+ select(RoleLease.agent_id, RoomRole.name)
+ .join(RoleLease, RoleLease.role_id == RoomRole.id)
+ .where(RoleLease.room_id == room_id)
+ .where(self._live(live_connection_ids))
+ )
+ return {agent_id: name for agent_id, name in result.all()}
+
async def agent_room_roles(
self,
session: AsyncSession,
diff --git a/core/switch_core/db/stores/room_store.py b/core/switch_core/db/stores/room_store.py
index cb1e5db5d..e19937e8c 100644
--- a/core/switch_core/db/stores/room_store.py
+++ b/core/switch_core/db/stores/room_store.py
@@ -1,4 +1,5 @@
from collections.abc import Collection
+from dataclasses import dataclass
from datetime import UTC, datetime
from sqlalchemy import delete, insert, or_, select, update
@@ -15,6 +16,16 @@ from switch_core.db.models import (
require_tenant_id,
room_agents,
)
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
+
+
+@dataclass(frozen=True)
+class Membership:
+ """A client's membership of one room, as `client_rooms` records it."""
+
+ room_id: str
+ transport_room_id: str
+ joined_at: datetime
class RoomStore:
@@ -420,8 +431,9 @@ class RoomStore:
async def add_client(
self, session: AsyncSession, client_id: str, room_id: str
- ) -> None:
- """Record a membership, tolerating one that is already there.
+ ) -> bool:
+ """Record a membership, tolerating one that is already there, and
+ say whether this call recorded it.
Membership has more than one writer. Over Matrix it effectively had
one: an invitation was accepted later, over sync, long after the
@@ -437,13 +449,24 @@ class RoomStore:
rather than a failure — the membership exists, which is what the
caller asked for — and this also closes the check-then-insert race
between two concurrent invitations.
+
+ Whoever recorded it writes the arrival that announces it, and nobody
+ else does: two writers racing for one membership must not greet the
+ room twice.
"""
- await session.execute(
+ recorded = await session.scalar(
pg_insert(ClientRoom)
.values(client_id=client_id, room_id=room_id)
.on_conflict_do_nothing(index_elements=["client_id", "room_id"])
+ .returning(ClientRoom.client_id)
)
+ # The client's consumer starts reading the room here, wherever and
+ # whenever it learns of the membership: the process that owns the
+ # client may not be this one. A membership that already existed keeps
+ # the cursor it has.
+ await DeliveryCursorStore().seed(session, client_id, room_id)
await session.flush()
+ return recorded is not None
async def remove_client(
self, session: AsyncSession, client_id: str, room_id: str
@@ -470,19 +493,39 @@ class RoomStore:
)
return list(result.scalars().all())
+ async def memberships_of(
+ self, session: AsyncSession, client_id: str
+ ) -> list[Membership]:
+ """Every room the client is a member of, with both its ids and when
+ the client joined, in one query."""
+ result = await session.execute(
+ select(ClientRoom.room_id, Room.transport_room_id, ClientRoom.joined_at)
+ .join(Room, Room.id == ClientRoom.room_id)
+ .where(ClientRoom.client_id == client_id)
+ )
+ return [
+ Membership(
+ room_id=room_id,
+ transport_room_id=transport_room_id,
+ joined_at=joined_at,
+ )
+ for room_id, transport_room_id, joined_at in result.all()
+ ]
+
async def member_room_ids(
self, session: AsyncSession, client_id: str, room_ids: list[str]
- ) -> set[str]:
- """Which of `room_ids` the client is still a member of, in one query."""
+ ) -> dict[str, datetime]:
+ """Which of `room_ids` the client is still a member of, each with when
+ it joined, in one query."""
if not room_ids:
- return set()
+ return {}
result = await session.execute(
- select(ClientRoom.room_id).where(
+ select(ClientRoom.room_id, ClientRoom.joined_at).where(
ClientRoom.client_id == client_id,
ClientRoom.room_id.in_(room_ids),
)
)
- return set(result.scalars().all())
+ return {room_id: joined_at for room_id, joined_at in result.all()}
async def get_client_ids(self, session: AsyncSession, room_id: str) -> list[str]:
result = await session.execute(
diff --git a/core/switch_core/db/stores/switch_core_process_store.py b/core/switch_core/db/stores/switch_core_process_store.py
index f2e799a5e..579068583 100644
--- a/core/switch_core/db/stores/switch_core_process_store.py
+++ b/core/switch_core/db/stores/switch_core_process_store.py
@@ -14,6 +14,8 @@ from switch_core.db.models import SwitchCoreProcess
class ProcessLeaseRow:
beat_at: datetime
stopped_at: datetime | None
+ # None for a process from before the column was written.
+ key_ids: tuple[str, ...] | None
class SwitchCoreProcessStore:
@@ -25,12 +27,19 @@ class SwitchCoreProcessStore:
hosts' clocks say.
"""
- async def renew(self, session: AsyncSession, process_id: str) -> None:
+ async def renew(
+ self, session: AsyncSession, process_id: str, key_ids: list[str]
+ ) -> None:
"""Claim the lease, or renew it: its beat is now. A row pruned while
this process could not reach the database is claimed again."""
await session.execute(
insert(SwitchCoreProcess)
- .values(id=process_id, started_at=func.now(), beat_at=func.now())
+ .values(
+ id=process_id,
+ started_at=func.now(),
+ beat_at=func.now(),
+ key_ids=key_ids,
+ )
.on_conflict_do_update(
index_elements=[SwitchCoreProcess.id],
set_={"beat_at": func.now(), "stopped_at": None},
@@ -64,9 +73,14 @@ class SwitchCoreProcessStore:
SwitchCoreProcess.id,
SwitchCoreProcess.beat_at,
SwitchCoreProcess.stopped_at,
+ SwitchCoreProcess.key_ids,
)
)
return now, {
- process_id: ProcessLeaseRow(beat_at=beat_at, stopped_at=stopped_at)
- for process_id, beat_at, stopped_at in rows.all()
+ process_id: ProcessLeaseRow(
+ beat_at=beat_at,
+ stopped_at=stopped_at,
+ key_ids=None if key_ids is None else tuple(key_ids),
+ )
+ for process_id, beat_at, stopped_at, key_ids in rows.all()
}
diff --git a/core/switch_core/db/tenant_lookup.py b/core/switch_core/db/tenant_lookup.py
index 638dd2528..899c26b69 100644
--- a/core/switch_core/db/tenant_lookup.py
+++ b/core/switch_core/db/tenant_lookup.py
@@ -1,4 +1,4 @@
-"""The whole exemption from row-level security, written out as twelve functions.
+"""The whole exemption from row-level security, written out as thirteen functions.
Row-level security is enforced by `require_tenant_id()` (`db/rls_ddl.py`),
which raises when no tenant is bound. That is the property everything else
@@ -251,7 +251,7 @@ class TenantLookup:
return f"{self.name}({', '.join('text' for _ in self.arguments)})"
-# The twelve of them. Ordered as the three shapes above: enumeration, then
+# The thirteen of them. Ordered as the three shapes above: enumeration, then
# credential resolution, then deriving a tenant from an identifier in hand.
TENANT_LOOKUPS: tuple[TenantLookup, ...] = (
TenantLookup(
@@ -380,6 +380,22 @@ TENANT_LOOKUPS: tuple[TenantLookup, ...] = (
"and its enrollment code, one bound tenant at a time."
),
),
+ TenantLookup(
+ name="tenant_of_sign_in_flow",
+ arguments=("state",),
+ query=(
+ "SELECT tenant_id FROM sign_in_flows "
+ "WHERE id = p_state AND expires_at > now()"
+ ),
+ purpose=(
+ "Which tenant a sign-in under way belongs to, from the state the "
+ "vendor hands back to the person's browser. The browser's steps "
+ "(leaving for the vendor, coming back, continuing to Switch "
+ "Console) carry no Switch session, only that state, and may reach "
+ "any switch-core process; the flow itself is read afterwards in "
+ "the tenant this names."
+ ),
+ ),
TenantLookup(
name="tenant_of_messaging_install",
arguments=("platform", "external_workspace_id"),
@@ -453,12 +469,12 @@ def attach_tenant_lookups(metadata: MetaData) -> None:
# ── Calling them ──────────────────────────────────────────────────────────────
# One `text()` per lookup, written out rather than assembled from `lookup.name`
-# at call time: the twelve names are fixed and known here, so there is nothing
+# at call time: the thirteen names are fixed and known here, so there is nothing
# for a call site to build. Each bind is named after the lookup's own argument,
# which is what lets `_call` zip them positionally against the dataclass and
# fail loudly on a mismatch rather than binding the workspace to the platform.
# The assertion below is what keeps this dict from quietly falling behind
-# `TENANT_LOOKUPS` — a thirteenth lookup with no entry here fails at import, not
+# `TENANT_LOOKUPS`: a fourteenth lookup with no entry here fails at import, not
# with a `KeyError` on whatever request reaches it first.
_LOOKUP_STATEMENTS: dict[str, TextClause] = {
"all_tenant_ids": text("SELECT tenant_id FROM all_tenant_ids() AS tenant_id"),
@@ -497,6 +513,9 @@ _LOOKUP_STATEMENTS: dict[str, TextClause] = {
"SELECT tenant_id FROM "
"tenant_of_messaging_install(:platform, :external_workspace_id) AS tenant_id"
),
+ "tenant_of_sign_in_flow": text(
+ "SELECT tenant_id FROM tenant_of_sign_in_flow(:state) AS tenant_id"
+ ),
}
assert _LOOKUP_STATEMENTS.keys() == TENANT_LOOKUPS_BY_NAME.keys(), (
@@ -631,3 +650,10 @@ async def tenants_of_cloud_machine(
return await _call(
session_factory, TENANT_LOOKUPS_BY_NAME["tenants_of_cloud_machine"], machine_id
)
+
+
+async def tenant_of_sign_in_flow(
+ session_factory: async_sessionmaker[AsyncSession], state: str
+) -> str | None:
+ lookup = TENANT_LOOKUPS_BY_NAME["tenant_of_sign_in_flow"]
+ return _at_most_one(lookup, await _call(session_factory, lookup, state))
diff --git a/core/switch_core/delivery/addressing.py b/core/switch_core/delivery/addressing.py
index cd442dacf..a846f2203 100644
--- a/core/switch_core/delivery/addressing.py
+++ b/core/switch_core/delivery/addressing.py
@@ -13,10 +13,10 @@ ordinary room chatter when so. Nothing here sends anything; deciding is all it
does, and the refusal wording travels back with the decision so the caller does
not have to re-derive why.
-This used to live on the agent client (now `AgentConsumer`), where every answer was reached through a
-live Matrix client. It takes a message as data instead — a sender, a body, a
-content dict — so the same rules decide for a message read out of the log as
-for one that arrived on the bus. That equivalence is the point: two code paths
+This used to live on the agent client, where every answer was reached through
+a live Matrix client. It takes a message as data instead (a sender, a body, a
+content dict), so the same rules decide for a message read out of the log as
+for one that arrived on the bus, and for every agent in a room at once. That equivalence is the point: two code paths
answering "is this for me?" differently would be a security bug waiting for a
rewrite to expose it.
"""
@@ -238,7 +238,7 @@ class AddressingResolver:
"""
if "@" not in message.body:
return False
- alias = await self._room_store.get_alias(session, room_id, agent.id)
+ alias = await self.alias_of(session, room_id, agent.id)
if not alias:
return False
return mention_regex(alias).search(message.body) is not None
@@ -263,13 +263,43 @@ class AddressingResolver:
"""
if "@" not in message.body:
return False
- role_name = await self._room_role_store.agent_room_role(
- session, room_id, agent.id, self._live_connection_ids()
- )
+ role_name = await self.role_of(session, room_id, agent.id)
if not role_name:
return False
return mention_regex(role_name).search(strip_emphasis(message.body)) is not None
+ # ── What the rules read ───────────────────────────────────────────────────
+ #
+ # The rules above and below ask the database these few questions and
+ # nothing else. Each is a method of its own so that a caller deciding one
+ # message for every agent in a room at once (the router) can answer them
+ # from reads it made once for the whole room, while every rule stays the
+ # one written here.
+
+ async def alias_of(
+ self, session: AsyncSession, room_id: str, agent_id: str
+ ) -> str | None:
+ """The name the agent goes under in this room, if it has one."""
+ return await self._room_store.get_alias(session, room_id, agent_id)
+
+ async def role_of(
+ self, session: AsyncSession, room_id: str, agent_id: str
+ ) -> str | None:
+ """The room role the agent live-holds in this room, if any."""
+ return await self._room_role_store.agent_room_role(
+ session, room_id, agent_id, self._live_connection_ids()
+ )
+
+ async def group_of(self, session: AsyncSession, room_id: str) -> str | None:
+ """The room group the room belongs to, which a policy can name."""
+ room = await self._room_store.get(session, room_id)
+ return room.group_id if room is not None else None
+
+ async def claimed_accounts(self, session: AsyncSession, user_id: str) -> list[str]:
+ """The ids of the platform accounts a Switch user has claimed."""
+ claimed = await self._external_user_store.get_by_user(session, user_id)
+ return [ext.id for ext in claimed]
+
# ── Permitted ─────────────────────────────────────────────────────────────
async def permitted(
@@ -302,8 +332,7 @@ class AddressingResolver:
return AddressingDecision(allowed=True, refusal="")
principal = await self.resolve_sender(session, sender, content)
- room = await self._room_store.get(session, room_id)
- group_id = room.group_id if room is not None else None
+ group_id = await self.group_of(session, room_id)
if principal is None:
if policy.is_open():
@@ -320,15 +349,14 @@ class AddressingResolver:
return AddressingDecision(allowed=False, refusal=ADDRESSING_DENIED_MESSAGE)
if principal.kind == "platform" and principal.on_behalf_of is not None:
- claimed = await self._external_user_store.get_by_user(
- session, principal.on_behalf_of
- )
allowed = allows_on_behalf_of(
policy,
room_id=room_id,
group_id=group_id,
user_id=principal.on_behalf_of,
- external_user_ids=[ext.id for ext in claimed],
+ external_user_ids=await self.claimed_accounts(
+ session, principal.on_behalf_of
+ ),
owner_user_id=agent.owner_id,
)
elif policy.is_open() and principal.kind != "platform":
diff --git a/core/switch_core/clients/agent_consumer.py b/core/switch_core/delivery/recipient.py
similarity index 57%
rename from core/switch_core/clients/agent_consumer.py
rename to core/switch_core/delivery/recipient.py
index 0ac95941b..d01c56b16 100644
--- a/core/switch_core/clients/agent_consumer.py
+++ b/core/switch_core/delivery/recipient.py
@@ -1,13 +1,31 @@
+"""One agent in a room, as the router decides a room row for it.
+
+The router (`delivery/router.py`) builds an `AgentRecipient` for every agent
+member of a room each time a row is written to that room, and asks it two
+things. While the row's transaction is open: is this row one of the agent's
+own events, and what event does the agent receive for it (`decide_*`)? Once
+the transaction has committed: what does the agent have left to say in the
+room, that it cannot say itself (`Delivery.after`, `greet`,
+`handle_command`)? That second part is the replies Switch posts on the
+agent's behalf: a refusal when the sender may not address it, a notice when
+it is offline, a greeting when it arrives, a command's answer.
+
+The rules are the ones the agent's consumer used to apply when each room row
+was delivered to it, moved here unchanged. What changed is when they run and
+how often: once per row, on the process that wrote it, rather than once per
+row on every process that ran a consumer for the agent.
+"""
+
from __future__ import annotations
import asyncio
import logging
import random
import re
-from collections.abc import Mapping
-from dataclasses import dataclass, field
+from collections.abc import Awaitable, Callable, Mapping
+from dataclasses import dataclass
from datetime import UTC, datetime
-from typing import TYPE_CHECKING, Literal, NamedTuple
+from typing import TYPE_CHECKING, Literal, NamedTuple, Protocol
from switch_core.attachments import parse_attachment_group
from switch_core.bridges.agent.commands import (
@@ -16,8 +34,6 @@ from switch_core.bridges.agent.commands import (
_addressed_by_name_or_role,
dispatch_command,
)
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
from switch_core.bridges.agent.protocol.message_events import (
attachment_ref,
message_event,
@@ -29,7 +45,6 @@ from switch_core.bridges.agent.protocol.presence import (
)
from switch_core.bridges.agent.protocol.types import (
AgentEvent,
- AttachmentRef,
CommandPayload,
TaskAcceptPayload,
TaskCancelPayload,
@@ -38,14 +53,12 @@ from switch_core.bridges.agent.protocol.types import (
TaskUpdatePayload,
)
from switch_core.budgets import BudgetExceeded, BudgetGuard
-from switch_core.clients.actor import AgentActor
from switch_core.clients.admin_messages import (
AUTO_REPLY_FLAG,
PLATFORM_MARKER,
platform_on_behalf_of,
platform_replies_in_channel,
)
-from switch_core.clients.consumer import Consumer
from switch_core.clients.mentions import (
NAME_CHAR as _NAME_CHAR,
)
@@ -58,16 +71,7 @@ from switch_core.clients.mentions import (
from switch_core.clients.room_meta import RoomMeta
from switch_core.db.models import Agent, CloudMachine
from switch_core.db.session_scope import tenant_session
-from switch_core.db.stores.agent_session_store import AgentSessionStore
-from switch_core.db.stores.agent_store import AgentStore
-from switch_core.db.stores.budget_store import BudgetStore
-from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
-from switch_core.db.stores.document_store import DocumentStore
-from switch_core.db.stores.external_user_store import ExternalUserStore
from switch_core.db.stores.hosted_machine_store import owner_stopped
-from switch_core.db.stores.reference_store import ReferenceStore
-from switch_core.db.stores.room_role_store import RoomRoleStore
-from switch_core.db.stores.room_store import RoomStore
from switch_core.delivery.addressing import (
ADDRESSING_DENIED_MESSAGE,
ADDRESSING_UNCLAIMED_MESSAGE,
@@ -77,6 +81,7 @@ from switch_core.delivery.addressing import (
)
from switch_core.events import (
CommandEvent,
+ SwitchEvent,
TaskAccept,
TaskCancel,
TaskDelegate,
@@ -89,6 +94,8 @@ from switch_core.observability.catalogue import OBSERVER_FAILURES
from switch_core.observability.metrics import metrics
from switch_core.observability.throttle import WarningThrottle
from switch_core.transport import (
+ InboundCustomEvent,
+ InboundEvent,
InboundMedia,
InboundMembership,
InboundMessage,
@@ -96,44 +103,108 @@ from switch_core.transport import (
)
if TYPE_CHECKING:
- from sqlalchemy.ext.asyncio import AsyncSession
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+ from switch_core.bridges.agent.protocol.agent_connections import (
+ AgentConnectionRegistry,
+ )
+ from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
+ from switch_core.clients.actor import AgentActor
+ from switch_core.db.models import Client
+ from switch_core.db.stores.agent_session_store import AgentSessionStore
+ from switch_core.db.stores.agent_store import AgentStore
+ from switch_core.db.stores.document_store import DocumentStore
+ from switch_core.db.stores.external_user_store import ExternalUserStore
+ from switch_core.db.stores.reference_store import ReferenceStore
+ from switch_core.db.stores.room_role_store import RoomRoleStore
+ from switch_core.db.stores.room_store import RoomStore
from switch_core.telemetry.messages import MessageTelemetry
logger = logging.getLogger(__name__)
-# One for the process rather than one per consumer: every agent has a
-# consumer, and a telemetry bug breaks the report for all of them.
+# One for the process rather than one per recipient: a recipient lives for one
+# row, and a telemetry bug breaks the report for every agent.
_report_failures = WarningThrottle(60.0)
-# How long to hold an incomplete multi-attachment group before delivering the
-# parts that did arrive, flagged as incomplete. Groups normally complete in
-# milliseconds (one sender, back-to-back events); this is a safety net so a
-# broken batch surfaces rather than being buffered indefinitely.
-ATTACHMENT_GROUP_TIMEOUT_SECONDS = 5.0
-
-
-@dataclass
-class _PendingAttachmentGroup:
- """Parts of a multi-attachment message seen so far, keyed by group index."""
-
- total: int
- parts: dict[int, AttachmentRef] = field(default_factory=dict)
- body: str = ""
- # The highest `messages.seq` among the parts received so far. The combined
- # message is written to the agent's own events with this as its `room_seq`,
- # the row of the part that completed the group.
- through_seq: int = 0
- # Whether any part of the group addressed this agent. Addressing is read off
- # the message text, and the text rides on part 0 alone — every other part's
- # body is its bare filename. Answering from whichever part completes the
- # group would therefore say "not addressed" for any group not finishing on
- # part 0, which is every group a bridge sends in order.
- addressed: bool = False
- # The index-0 event, kept so a group that has to be flushed incomplete is
- # still anchored on its canonical first part (message_id / timestamp /
- # sender) rather than on whichever part happened to arrive last.
- first_event: InboundMedia | None = None
+
+@dataclass(frozen=True)
+class RoomMember:
+ """An agent member of a room, as the router read it for one row.
+
+ `alias` is the name the agent goes under in this room, if it has one, and
+ `listening` whether it is set to receive `room_join` events here. Both come
+ from the agent's `room_agents` row, read with the membership itself.
+ `joined_at` is when the membership was written, on the database's clock.
+ """
+
+ agent: Agent
+ client_id: str
+ transport_user_id: str
+ display_name: str
+ alias: str | None
+ listening: bool
+ joined_at: datetime
+
+
+class OwnEvent(NamedTuple):
+ """One of an agent's own events, and the room row it comes from: its
+ place in the room (`room_seq`) and the key the row is known by
+ (`message_key`, the attachment group's id for a multi-file post)."""
+
+ event: AgentEvent
+ room_seq: int
+ message_key: str | None
+
+
+@dataclass(frozen=True)
+class Delivery:
+ """What was decided about one room row for one agent.
+
+ `own_event` builds the agent's own event from the row once the row is
+ written and has its place in the room, and is None when the row is not
+ one of the agent's own events. `after` is what is left to do once the
+ row has committed (a refusal or an offline notice to post, a report to
+ make), and is None when there is nothing. Both take the row as written.
+
+ `named` says whether a rule named the agent (its name, its alias here, a
+ role it holds, a direct room), before its addressing policy and budget
+ were applied: the router asks whether a message named anybody at all.
+ """
+
+ own_event: Callable[[InboundEvent], OwnEvent] | None
+ after: Callable[[InboundEvent], Awaitable[None]] | None
+ named: bool = False
+
+
+NOTHING = Delivery(own_event=None, after=None)
+
+
+@dataclass(frozen=True)
+class RecipientServices:
+ """What every recipient reads and writes through. One per process."""
+
+ session_factory: async_sessionmaker[AsyncSession]
+ event_buffer: EventBuffer
+ agent_store: AgentStore
+ room_store: RoomStore
+ document_store: DocumentStore
+ reference_store: ReferenceStore
+ agent_session_store: AgentSessionStore
+ room_role_store: RoomRoleStore
+ external_user_store: ExternalUserStore
+ connections: AgentConnectionRegistry
+ frontend_base_url: str | None
+ message_telemetry: MessageTelemetry
+
+
+class ClientNames(Protocol):
+ """Who a member of a room is, by the id they post under: the one read an
+ arrival's own event needs, which the router makes once for every agent in
+ the room."""
+
+ async def get_by_transport_user_id(
+ self, session: AsyncSession, transport_user_id: str
+ ) -> Client | None: ...
_UNAVAILABLE_MESSAGES = {
@@ -322,229 +393,118 @@ def _role_elsewhere_message(other_room_name: str) -> str:
return _elsewhere_message([other_room_name], holds_role_here=True)
-class AgentConsumer(Consumer[AgentActor]):
- """Reads the rooms an agent is in, and decides what reaches the agent.
+class AgentRecipient:
+ """One agent member of a room, deciding one room row.
+
+ Built by the router for a single row and dropped with it. `decide_*` run
+ inside the transaction that writes the row, read through the session they
+ are handed, and post nothing: what they decide is returned as a
+ `Delivery`. Everything that posts runs after the commit, through `actor`,
+ the agent's own identity.
- One per agent. Applies addressing, queues what should wake the agent in
- its event buffer for the SSE stream, and posts the replies an agent cannot
- give itself (greetings, offline and refusal notices) through its actor.
+ The reads that every agent in the room would otherwise repeat (the agent
+ rows, aliases, role leases, the sender, budgets) go through `addressing`,
+ `budget_guard` and `clients`, which the router answers from reads it made
+ once for the whole room.
"""
def __init__(
self,
*,
+ services: RecipientServices,
+ member: RoomMember,
+ tenant_id: str,
+ meta: RoomMeta,
+ transport_room_id: str,
actor: AgentActor,
- event_buffer: EventBuffer,
- agent_store: AgentStore,
- room_store: RoomStore,
- bridge_store: CollaborationBridgeStore,
- document_store: DocumentStore,
- reference_store: ReferenceStore,
- agent_session_store: AgentSessionStore,
- room_role_store: RoomRoleStore,
- external_user_store: ExternalUserStore,
- connections: AgentConnectionRegistry,
- frontend_base_url: str | None,
- message_telemetry: MessageTelemetry,
+ addressing: AddressingResolver,
+ budget_guard: BudgetGuard,
+ clients: ClientNames,
+ cloud_waking_notices: set[tuple[str, int]],
) -> None:
- super().__init__(actor=actor)
- self._event_buffer = event_buffer
- self._agent_store = agent_store
- self._preloaded_agent: Agent | None = None
- self._room_store = room_store
- self._bridge_store = bridge_store
- self._document_store = document_store
- self._reference_store = reference_store
- self._agent_session_store = agent_session_store
- self._room_role_store = room_role_store
- self._external_user_store = external_user_store
- self._cloud_waking_notices: set[tuple[str, int]] = set()
- self._connections = connections
- self._message_telemetry = message_telemetry
+ self.actor = actor
+ self.tenant_id = tenant_id
+ self.client_id = member.client_id
+ self.transport_user_id = member.transport_user_id
+ self.session_factory = services.session_factory
+ self._member = member
+ self._meta = meta
+ self._room = RoomRef(room_id=transport_room_id)
+ self._event_buffer = services.event_buffer
+ self._agent_store = services.agent_store
+ self._room_store = services.room_store
+ self._document_store = services.document_store
+ self._reference_store = services.reference_store
+ self._agent_session_store = services.agent_session_store
+ self._room_role_store = services.room_role_store
+ self._external_user_store = services.external_user_store
+ self._connections = services.connections
+ self._message_telemetry = services.message_telemetry
self._frontend_base_url = (
- frontend_base_url.rstrip("/") if frontend_base_url else None
- )
- # Addressing is decided from stores, not from this client, so the same
- # rules can decide for a message read out of the log.
- self._addressing = AddressingResolver(
- room_store=room_store,
- room_role_store=room_role_store,
- client_store=self.client_store,
- agent_store=agent_store,
- external_user_store=external_user_store,
- live_connection_ids=connections.live_connection_ids,
+ services.frontend_base_url.rstrip("/")
+ if services.frontend_base_url
+ else None
)
- self._budget_guard = BudgetGuard(BudgetStore())
- self._room_meta: dict[str, RoomMeta | None] = {}
- # In-flight multi-attachment groups, by group id, with their safety-net
- # timers. Both are cleared when a group completes or times out, so a
- # never-completed group cannot leak.
- self._attachment_groups: dict[str, _PendingAttachmentGroup] = {}
- self._attachment_group_timers: dict[str, asyncio.TimerHandle] = {}
+ self._addressing = addressing
+ self._budget_guard = budget_guard
+ self._clients = clients
+ # The "machine is waking" notices this process has posted for the
+ # agent, by room and machine start, so the messages that follow the
+ # first one wait with it instead of each saying so again.
+ self._cloud_waking_notices = cloud_waking_notices
@property
def agent(self) -> Agent:
- return self.actor.agent
+ """The agent's row, read for this row's routing."""
+ return self._member.agent
async def _fresh_agent(self, session: AsyncSession) -> Agent:
- """Re-read the agent row and refresh the cached snapshot.
-
- `self.agent` is loaded once in `start()`. An operator can edit the
- agent's `integration_profile` (e.g. flip `connection_model` to/from
- `auto_session`) via the gateway while this client runs, so any branch
- on `connection_model` must read fresh rather than trust the boot-time
- snapshot.
+ """The agent's row as it stands now.
- Reads through the caller's session: one inbound message asks several
- questions of this row, and each one opening a session of its own is
- several pool slots taken to answer a single event.
+ An operator can edit the agent (its `integration_profile`, its
+ addressing policy) through the gateway at any time, so every branch on
+ it reads the row rather than a copy kept from earlier. The router read
+ it for this row, in the transaction writing the row, which is as fresh
+ as it gets, so nothing is read again here.
"""
- fresh = await self._agent_store.get(session, self.agent.id)
- if fresh is not None:
- self.actor.set_agent(fresh)
return self.agent
- async def start(self) -> None:
- """Resolve the agent this client is, then run.
-
- This runs at the top of the client's own task, which binds nothing, so
- the tenant is carried rather than inherited: it comes from the
- `clients` row this client was built from. A *guessed* tenant would
- turn "this client is an agent" into "no agent found for client" and
- fail the client at boot over a row that exists, which is why nothing
- here falls back to whatever is ambient.
+ async def still_member(self) -> bool:
+ """Whether the agent is a member of the room now, read when the work
+ a row left it (`Delivery.after`, `greet`) is about to run.
+
+ The row was decided from the members read before it was written, and
+ a removal takes no lock of the room's, so the agent can be taken out
+ of the room after that read: before the row commits, or after it and
+ before this work's turn comes. Either way it keeps no own event for
+ the row (`EventBuffer.store` reads the membership again, and the
+ removal deletes what was stored). This asks the same question of the
+ work, so an agent that has left says nothing in the room as itself (a
+ refusal, an offline notice, a greeting, a command's answer), relays
+ nothing to its controller and wakes no machine.
+
+ In a session of its own, closed before the work posts anything.
"""
- agent = self._preloaded_agent
- self._preloaded_agent = None
- if agent is None:
- async with tenant_session(self.session_factory, self.tenant_id) as session:
- agent = await self._agent_store.get_by_client_id(
- session, self.client_id
- )
- if agent is None:
- raise RuntimeError(f"No agent found for client: {self.client_id}")
- self.actor.set_agent(agent)
- await super().start()
-
- def preload_agent(self, agent: Agent) -> None:
- """Hand `start` the agent row read in bulk at boot, so it needs no
- query of its own. Used once: a restart reads it fresh."""
- self._preloaded_agent = agent
-
- # ── Event hooks ───────────────────────────────────────────────────────────
-
- async def on_self_join(self, room: RoomRef, event: InboundMembership) -> None:
- is_direct = await self._is_direct_room(room.room_id)
- name = self.agent.name
- # The per-agent self-join greeting can be switched off per bridge
- # — e.g. on Mattermost, whose native "X joined the
- # channel" notice makes it redundant.
- meta = await self._resolve_room_meta(room.room_id)
- if meta is not None:
- self._connections.controllers.room_joined(self.agent.id, meta.room_id)
- if meta is not None and not meta.agent_greetings_enabled:
- logger.info(
- "Suppressing self-join greeting for %s in %s "
- "(agent greetings disabled on the room's bridge)",
- name,
- room.room_id,
+ room_id = self._meta.room_id
+ async with tenant_session(self.session_factory, self.tenant_id) as session:
+ members = await self._room_store.member_room_ids(
+ session, self.client_id, [room_id]
)
- return
- if is_direct:
- greeting = f"Hi! I'm {name} — how can I help?"
- else:
- greeting = random.choice(AGENT_GREETINGS).format(name=name)
- await self.actor.send_message(
- room.room_id, greeting, format="markdown", metered=False
- )
+ return room_id in members
- async def on_removed(self, room: RoomRef, event: InboundMembership) -> None:
- """Forget the room's events, everywhere this agent could still read them.
+ # ── Messages ──────────────────────────────────────────────────────────────
- Dropping the subscription only stops what has not been read yet. What
- has is in the event buffer for the whole retention window, and is
- served from there to a long poll, to the notification stream, and to an
- SSE reader resuming from an old cursor — the last of which has no
- membership of its own to apply.
- """
- meta = await self._resolve_room_meta(room.room_id)
- if meta is None:
- return
- await self._event_buffer.drop_room(self.tenant_id, self.agent.id, meta.room_id)
- self._connections.release_room_everywhere(self.agent.id, meta.room_id)
- self._connections.controllers.room_left(self.agent.id, meta.room_id)
+ async def decide_message(
+ self, session: AsyncSession, event: InboundMessage
+ ) -> Delivery:
+ """Whether a message is one of the agent's own events.
- async def _member_name(
- self, session: AsyncSession, event: InboundMembership
- ) -> str:
- """What to call the member who just arrived.
-
- The name Switch knows them by, not the one on the membership event: a
- profile is set from whatever the source platform calls someone, so
- taking it makes the same person read one way when they arrive and
- another way when they speak — and the log, which records the Switch
- name, disagree with what was delivered live.
+ It is when a rule names the agent and its addressing policy and budget
+ let the sender through. A sender the policy refuses, or an agent over
+ its budget, gets a refusal posted once the message has committed, and
+ the message stays the room's conversation.
"""
- client = await self.client_store.get_by_transport_user_id(
- session, event.state_key
- )
- if client is not None:
- return client.display_name
- fallback = event.display_name or event.state_key.split(":")[0].lstrip("@")
- logger.warning(
- "No Switch client owns %s; naming the arrival %r from the membership event",
- event.state_key,
- fallback,
- )
- return fallback
-
- async def on_member_event(self, room: RoomRef, event: InboundMembership) -> None:
- # Only forward genuine joins. A membership-preserving update (e.g. a
- # display-name or avatar change) re-fires m.room.member with
- # membership == "join" and prev_membership == "join" — exclude those, and
- # all leave/ban/knock transitions. The base handler already routes this
- # agent's own join to on_self_join and drops backfilled/self events, so
- # this fires once per other-member join (humans and agents alike).
- if event.membership != "join" or event.prev_membership == "join":
- return
- meta = await self._resolve_room_meta(room.room_id)
- if meta is None:
- return
- # `listening` tells each connector whether THIS receiving agent is
- # configured to react to join events in this room. The event is always
- # delivered; the connector decides whether to surface it.
- async with self.session_factory() as session:
- listening = await self._room_store.get_receives_join_events(
- session, meta.room_id, self.agent.id
- )
- if not listening:
- # The agent does not listen for joins in this room, so the join
- # is not written to its own events. A reader that wants the
- # whole conversation still sees it, read from `messages`.
- return
- member_name = await self._member_name(session, event)
- await self._event_buffer.enqueue(
- self.tenant_id,
- self.agent.id,
- meta.room_id,
- room_join_event(
- room_id=meta.room_id,
- bridge_id=meta.bridge_id,
- channel_type=meta.channel_type,
- member=event.state_key,
- member_name=member_name,
- timestamp=event.timestamp,
- listening=listening,
- ),
- room_seq=event.seq,
- message_key=event.event_id,
- )
-
- async def on_message(self, room: RoomRef, event: InboundMessage) -> None:
- meta = await self._resolve_room_meta(room.room_id)
- if meta is None:
- return
-
+ meta = self._meta
thread_id = event.thread_root_id
if platform_replies_in_channel(event.content):
# A threaded kickoff that wants its work in the channel: the agent
@@ -554,33 +514,16 @@ class AgentConsumer(Consumer[AgentActor]):
else:
reply_thread_root = thread_id if thread_id is not None else event.event_id
- # Every database read this message needs happens in one session, and
- # nothing is posted to the room while it is open. A busy room fans one
- # inbound event out to every agent client in it at once, so a client
- # that took a slot per question would multiply a single message into
- # dozens of concurrent checkouts.
- is_addressed = False
- agent_live = False
+ named = False
+ addressed = False
refusal: str | None = None
- unavailable: str | None = None
if self._addressed_without_lookup(event, meta) is not False:
- async with self.session_factory() as session:
- is_addressed = await self._compute_addressed(session, event, meta)
- if is_addressed:
- agent = await self._fresh_agent(session)
- gate = await self._gate_addressed(session, agent, event, meta)
- is_addressed = gate.addressed
- refusal = gate.refusal
- if is_addressed and not self._triggered_by_auto_reply(event):
- agent_live = await self._is_available(
- session, agent, meta.room_id
- )
- if not agent_live:
- unavailable = await self._reply_when_unavailable_here(
- session, agent, meta, self._sender_handle(event)
- )
-
- text = event.body
+ named = await self._compute_addressed(session, event, meta)
+ if named:
+ agent = await self._fresh_agent(session)
+ gate = await self._gate_addressed(session, agent, event, meta)
+ addressed = gate.addressed
+ refusal = gate.refusal
sender_name = event.sender_name
if not sender_name:
@@ -590,27 +533,82 @@ class AgentConsumer(Consumer[AgentActor]):
sender_name,
)
- agent_event = message_event(
- room_id=meta.room_id,
- bridge_id=meta.bridge_id,
- channel_type=meta.channel_type,
- event=event,
- addressed=is_addressed,
- sender_name=sender_name,
- body=text,
- thread_id=thread_id,
+ def own_event(row: InboundEvent) -> OwnEvent:
+ assert isinstance(row, InboundMessage)
+ agent_event = message_event(
+ room_id=meta.room_id,
+ bridge_id=meta.bridge_id,
+ channel_type=meta.channel_type,
+ event=row,
+ addressed=True,
+ sender_name=sender_name,
+ body=row.body,
+ thread_id=thread_id,
+ )
+ # Serialising runs once per agent per message, so only when it
+ # will be seen.
+ if logger.isEnabledFor(logging.DEBUG):
+ logger.debug(
+ "Enqueuing event %s", agent_event.model_dump_json(indent=2)
+ )
+ return OwnEvent(agent_event, room_seq=row.seq, message_key=row.event_id)
+
+ async def after(row: InboundEvent) -> None:
+ assert isinstance(row, InboundMessage)
+ await self._after_message(
+ row,
+ addressed=addressed,
+ refusal=refusal,
+ thread_id=thread_id,
+ reply_thread_root=reply_thread_root,
+ )
+
+ return Delivery(
+ own_event=own_event if addressed else None,
+ after=after if addressed or refusal is not None else None,
+ named=named,
)
- if is_addressed:
+ async def _after_message(
+ self,
+ event: InboundMessage,
+ *,
+ addressed: bool,
+ refusal: str | None,
+ thread_id: str | None,
+ reply_thread_root: str | None,
+ ) -> None:
+ """What a message leaves the agent to say once it has committed.
+
+ Whether the agent can take the message is read now, in a session of
+ its own that is closed before anything is posted: posting is a write
+ of its own, and holding a connection across it is how a pool slot
+ ends up idle in a transaction.
+ """
+ meta = self._meta
+ agent_live = False
+ unavailable: str | None = None
+ if addressed and not self._triggered_by_auto_reply(event):
+ async with self.session_factory() as session:
+ agent = await self._fresh_agent(session)
+ agent_live = await self._is_available(session, agent, meta.room_id)
+ if not agent_live:
+ unavailable = await self._reply_when_unavailable_here(
+ session, agent, meta, self._sender_handle(event)
+ )
+
+ if addressed:
self._report_addressed(
meta, event, agent_live=agent_live, has_attachment=False
)
if refusal is not None:
- await self._post_auto_reply(room.room_id, event, refusal, reply_thread_root)
+ await self._post_auto_reply(
+ self._room.room_id, event, refusal, reply_thread_root
+ )
if unavailable is not None:
await self._post_auto_reply(
- room.room_id,
+ self._room.room_id,
event,
unavailable,
# Where this lands depends on whether an answer follows it.
@@ -618,7 +616,7 @@ class AgentConsumer(Consumer[AgentActor]):
# "Starting a session" is a preamble: the real answer arrives
# after it, in the room the question was asked in. Threading it
# off the trigger buries the notice away from the answer it
- # introduces, so it goes where the question was — `thread_id`,
+ # introduces, so it goes where the question was: `thread_id`,
# which is None for a message at the root (CHOO-2173).
#
# Everything else here is the whole reply and nothing follows
@@ -630,247 +628,292 @@ class AgentConsumer(Consumer[AgentActor]):
else reply_thread_root,
)
- if not is_addressed:
- # An unaddressed message is not written to the agent's own events.
- # A reader that wants the whole conversation reads it from
- # `messages`.
- return
- # Serialising runs once per agent per message, so only when it will be seen.
- if logger.isEnabledFor(logging.DEBUG):
- logger.debug("Enqueuing event %s", agent_event.model_dump_json(indent=2))
- await self._event_buffer.enqueue(
- self.tenant_id,
- self.agent.id,
- meta.room_id,
- agent_event,
- room_seq=event.seq,
- message_key=event.event_id,
- )
+ # ── Files ─────────────────────────────────────────────────────────────────
- async def on_media(self, room: RoomRef, event: InboundMedia) -> None:
- meta = await self._resolve_room_meta(room.room_id)
- if meta is None:
- return
- is_addressed = await self._addressed(event, meta)
+ async def addressed_by(self, session: AsyncSession, event: InboundMessage) -> bool:
+ """Whether a rule names the agent in this row, before its policy and
+ budget are applied. Reads only when the text alone cannot say."""
+ decided = self._addressed_without_lookup(event, self._meta)
+ if decided is not None:
+ return decided
+ return await self._compute_addressed(session, event, self._meta)
- content = event.content
- attachment = attachment_ref(event)
+ async def decide_post(
+ self,
+ session: AsyncSession,
+ files: list[InboundMedia],
+ *,
+ named: bool,
+ lead: InboundMedia,
+ complete: bool,
+ message_key: str,
+ ) -> Delivery:
+ """Whether a post of one or more files is one of the agent's own
+ events, delivered as one message carrying every file.
+
+ `files` are the post's rows, in the order they were written; a file
+ still being written is given as it will be stored, and replaced by
+ the stored row when the event is built. `named` is whether any of them
+ named the agent: the text and its mentions ride on the first file
+ alone, so asking only the file that completed the post would say
+ "not addressed" for every post not finishing on its first file.
+
+ The message is anchored on the first file (index 0) when it arrived,
+ so its id and time are the ones the sender was given, whichever file
+ completed it. `lead` is the file the rest is taken from when the first
+ is missing, and whose thread and sender name the message carries: the
+ file that completed the post, or for a post given up on, the first of
+ its files written. A post given up on (`complete` False) says in its
+ body how many files arrived.
+ """
+ meta = self._meta
+ anchor = _first_file(files) or lead
+ gate = _GateOutcome(addressed=False, refusal=None)
+ if named:
+ agent = await self._fresh_agent(session)
+ gate = await self._gate_addressed(session, agent, anchor, meta)
- sender_name = event.sender_name
+ sender_name = lead.sender_name
if not sender_name:
- sender_name = event.sender
+ sender_name = lead.sender
logger.error(
"Media received without sender name, using the sender id: %s",
sender_name,
)
-
- # Mirror on_message: surface the thread this media belongs to (if any).
- thread_id = event.thread_root_id
-
- # Several files posted as one message arrive as separate media events
- # sharing a group marker (a media event carries one file). Hold
- # them until the group is complete, then emit ONE payload carrying all
- # of them, so the agent sees one message with N attachments.
- group = parse_attachment_group(content)
- if group is None:
- await self._emit_media(
- room,
- event,
- meta,
- is_addressed,
- sender_name,
- thread_id,
- [attachment],
- event.body,
- room_seq=event.seq,
- message_key=event.event_id,
+ thread_id = lead.thread_root_id
+ group = parse_attachment_group(lead.content)
+ total = group[2] if group is not None else 1
+
+ def written(row: InboundEvent) -> list[InboundMedia]:
+ return [
+ row
+ if isinstance(row, InboundMedia) and row.event_id == part.event_id
+ else part
+ for part in files
+ ]
+
+ def own_event(row: InboundEvent) -> OwnEvent:
+ parts = written(row)
+ first = _first_file(parts)
+ anchored = first or next(p for p in parts if p.event_id == lead.event_id)
+ body = (first.body if first is not None else "") or anchored.body
+ if not complete:
+ present = len({file_index(part) for part in parts})
+ notice = (
+ f"[incomplete attachment group: {present} of {total} files arrived]"
+ )
+ body = f"{body}\n{notice}" if body else notice
+ agent_event = message_event(
+ room_id=meta.room_id,
+ bridge_id=meta.bridge_id,
+ channel_type=meta.channel_type,
+ event=anchored,
+ addressed=True,
+ sender_name=sender_name,
+ body=body,
+ thread_id=thread_id,
+ attachments=[
+ attachment_ref(part) for part in sorted(parts, key=file_index)
+ ],
+ )
+ if logger.isEnabledFor(logging.DEBUG):
+ logger.debug(
+ "Enqueuing media event %s", agent_event.model_dump_json(indent=2)
+ )
+ return OwnEvent(
+ agent_event,
+ room_seq=max(part.seq for part in parts),
+ message_key=message_key,
)
- return
- group_id, index, total = group
- pending = self._attachment_groups.setdefault(
- group_id, _PendingAttachmentGroup(total=total)
- )
- pending.parts[index] = attachment
- pending.addressed = pending.addressed or is_addressed
- pending.through_seq = max(pending.through_seq, event.seq)
- if index == 0:
- pending.body = event.body
- pending.first_event = event
- if len(pending.parts) < total:
- self._schedule_attachment_group_flush(
- group_id, room, event, meta, sender_name, thread_id
+ async def after(row: InboundEvent) -> None:
+ parts = written(row)
+ anchored = _first_file(parts) or next(
+ p for p in parts if p.event_id == lead.event_id
)
- return
+ await self._after_post(anchored, gate=gate, thread_id=thread_id)
- self._cancel_attachment_group_flush(group_id)
- self._attachment_groups.pop(group_id, None)
- # Anchor the coalesced message on part 0, not on whichever part
- # happened to complete the group, so message_id / timestamp are stable.
- await self._emit_media(
- room,
- pending.first_event or event,
- meta,
- pending.addressed,
- sender_name,
- thread_id,
- [pending.parts[i] for i in sorted(pending.parts)],
- pending.body or event.body,
- room_seq=pending.through_seq,
- message_key=group_id,
+ return Delivery(
+ own_event=own_event if gate.addressed else None,
+ after=after if named else None,
+ named=named,
)
- def _schedule_attachment_group_flush(
- self,
- group_id: str,
- room: RoomRef,
- event: InboundMedia,
- meta: RoomMeta,
- sender_name: str,
- thread_id: str | None,
+ async def _after_post(
+ self, anchor: InboundMedia, *, gate: _GateOutcome, thread_id: str | None
) -> None:
- """Arm the safety-net timer for an incomplete attachment group.
+ """What a post leaves the agent to say once it has committed: a
+ refusal, and the report that it was asked something."""
+ meta = self._meta
+ reply_thread_root = thread_id if thread_id is not None else anchor.event_id
+ agent_live = False
+ # Only the report reads this here (a post gets no offline notice), so
+ # it is not worth a query while reporting is off.
+ if (
+ gate.addressed
+ and self._message_telemetry.enabled
+ and not self._triggered_by_auto_reply(anchor)
+ ):
+ async with self.session_factory() as session:
+ agent = await self._fresh_agent(session)
+ agent_live = await self._is_available(session, agent, meta.room_id)
+ if gate.refusal is not None:
+ await self._post_auto_reply(
+ self._room.room_id, anchor, gate.refusal, reply_thread_root
+ )
+ if gate.addressed:
+ self._report_addressed(
+ meta, anchor, agent_live=agent_live, has_attachment=True
+ )
+
+ # ── Arrivals ──────────────────────────────────────────────────────────────
- The group should normally complete within milliseconds — every event is
- sent back-to-back by one sender. The timer exists so a group that never
- completes (a failed send mid-batch, a dropped event) surfaces what did
- arrive, clearly flagged, instead of being buffered forever.
+ async def decide_arrival(
+ self, session: AsyncSession, event: InboundMembership
+ ) -> Delivery:
+ """Whether someone else's arrival in the room is one of the agent's
+ own events: it is when the agent listens for joins in this room.
- Armed once per group, NOT re-armed per part: the deadline bounds the
- whole group, so a batch dribbling in just under the timeout can't hold
- the buffer open indefinitely.
+ Only a genuine join counts. A membership-preserving update (a display
+ name or avatar change) re-fires the membership with `prev_membership`
+ "join", and leaving is never written as a row.
"""
- if group_id in self._attachment_group_timers:
- return
- self._attachment_group_timers[group_id] = asyncio.get_running_loop().call_later(
- ATTACHMENT_GROUP_TIMEOUT_SECONDS,
- lambda: asyncio.create_task(
- self._flush_incomplete_attachment_group(
- group_id, room, event, meta, sender_name, thread_id
- )
- ),
+ if event.membership != "join" or event.prev_membership == "join":
+ return NOTHING
+ if not self._member.listening:
+ # The agent does not listen for joins in this room, so the join
+ # is not written to its own events. A reader that wants the whole
+ # conversation still sees it, read from `messages`.
+ return NOTHING
+ meta = self._meta
+ member_name = await self._member_name(session, event)
+
+ def own_event(row: InboundEvent) -> OwnEvent:
+ assert isinstance(row, InboundMembership)
+ return OwnEvent(
+ room_join_event(
+ room_id=meta.room_id,
+ bridge_id=meta.bridge_id,
+ channel_type=meta.channel_type,
+ member=row.state_key,
+ member_name=member_name,
+ timestamp=row.timestamp,
+ listening=True,
+ ),
+ room_seq=row.seq,
+ message_key=row.event_id,
+ )
+
+ return Delivery(own_event=own_event, after=None)
+
+ async def _member_name(
+ self, session: AsyncSession, event: InboundMembership
+ ) -> str:
+ """What to call the member who just arrived.
+
+ The name Switch knows them by, not the one on the membership event: a
+ profile is set from whatever the source platform calls someone, so
+ taking it makes the same person read one way when they arrive and
+ another way when they speak, and the log, which records the Switch
+ name, disagree with what was delivered live.
+ """
+ client = await self._clients.get_by_transport_user_id(session, event.state_key)
+ if client is not None:
+ return client.display_name
+ fallback = event.display_name or event.state_key.split(":")[0].lstrip("@")
+ logger.warning(
+ "No Switch client owns %s; naming the arrival %r from the membership event",
+ event.state_key,
+ fallback,
)
+ return fallback
- def _cancel_attachment_group_flush(self, group_id: str) -> None:
- timer = self._attachment_group_timers.pop(group_id, None)
- if timer is not None:
- timer.cancel()
+ async def greet(self) -> None:
+ """Say hello in a room the agent has just arrived in.
- async def _flush_incomplete_attachment_group(
- self,
- group_id: str,
- room: RoomRef,
- event: InboundMedia,
- meta: RoomMeta,
- sender_name: str,
- thread_id: str | None,
- ) -> None:
- self._attachment_group_timers.pop(group_id, None)
- pending = self._attachment_groups.pop(group_id, None)
- if pending is None:
+ The per-agent greeting can be switched off per bridge: on Mattermost,
+ for one, whose native "X joined the channel" notice makes it
+ redundant.
+ """
+ meta = self._meta
+ name = self.agent.name
+ if not meta.agent_greetings_enabled:
+ logger.info(
+ "Suppressing self-join greeting for %s in %s "
+ "(agent greetings disabled on the room's bridge)",
+ name,
+ self._room.room_id,
+ )
return
- received = len(pending.parts)
- logger.error(
- "Attachment group %s incomplete: %d of %d parts arrived within %ss; "
- "delivering what arrived",
- group_id,
- received,
- pending.total,
- ATTACHMENT_GROUP_TIMEOUT_SECONDS,
- )
- body = pending.body or event.body
- notice = (
- f"[incomplete attachment group: {received} of {pending.total} "
- f"files arrived]"
- )
- # Anchor on part 0 when we have it, so the payload's message_id matches
- # the completed-group case and replies thread off the canonical event.
- anchor = pending.first_event or event
- await self._emit_media(
- room,
- anchor,
- meta,
- pending.addressed,
- sender_name,
- thread_id,
- [pending.parts[i] for i in sorted(pending.parts)],
- f"{body}\n{notice}" if body else notice,
- room_seq=pending.through_seq,
- message_key=group_id,
+ if meta.channel_type == "direct":
+ greeting = f"Hi! I'm {name} — how can I help?"
+ else:
+ greeting = random.choice(AGENT_GREETINGS).format(name=name)
+ await self.actor.send_message(
+ self._room.room_id, greeting, format="markdown", metered=False
)
- async def _emit_media(
- self,
- room: RoomRef,
- event: InboundMedia,
- meta: RoomMeta,
- is_addressed: bool,
- sender_name: str,
- thread_id: str | None,
- attachments: list[AttachmentRef],
- body: str,
- *,
- room_seq: int,
- message_key: str,
- ) -> None:
- reply_thread_root = thread_id if thread_id is not None else event.event_id
- agent_live = False
- if is_addressed:
- async with self.session_factory() as session:
- agent = await self._fresh_agent(session)
- gate = await self._gate_addressed(session, agent, event, meta)
- # Only telemetry reads this here — the media path posts no
- # unavailable notice — so it is not worth a query while off.
- if (
- gate.addressed
- and self._message_telemetry.enabled
- and not self._triggered_by_auto_reply(event)
- ):
- agent_live = await self._is_available(session, agent, meta.room_id)
- is_addressed = gate.addressed
- if gate.refusal is not None:
- await self._post_auto_reply(
- room.room_id, event, gate.refusal, reply_thread_root
+ # ── Commands ──────────────────────────────────────────────────────────────
+
+ async def decide_command(self, event: CommandEvent) -> Delivery:
+ """What a room command does for the agent, decided as its row is
+ written.
+
+ A session control (`!reset`, `!compact`, `!interrupt`) is queued for
+ the agent as an own event, stored with the command's row so the agent
+ gets it before anything written after it; what it says in the room
+ is said once it has committed. Every other command queues nothing and
+ is run by its handler once it has committed (`handle_command`).
+ """
+ cmd = COMMANDS_BY_NAME.get(event.command)
+ if cmd is None or cmd.admin_owned:
+ return NOTHING
+ if cmd.decide is None:
+
+ async def run(row: InboundEvent) -> None:
+ assert isinstance(row, InboundCustomEvent)
+ await self.handle_command(command_event(row))
+
+ return Delivery(own_event=None, after=run)
+ meta = self._meta
+ if not await cmd.addressed(self, event.args, meta.room_id):
+ return NOTHING
+ refusal = await self._command_refusal(self._room, event, meta)
+ if refusal is not None:
+ return Delivery(own_event=None, after=lambda _row: refusal())
+ decision = await cmd.decide(self, self._room, event)
+ answer = decision.answer
+ queued = decision.queued
+ own_event: Callable[[InboundEvent], OwnEvent] | None = None
+ if queued is not None:
+
+ def own_event(row: InboundEvent) -> OwnEvent:
+ return OwnEvent(
+ AgentEvent(
+ type="command",
+ room_id=meta.room_id,
+ bridge_id=meta.bridge_id,
+ channel_type=meta.channel_type,
+ payload=queued,
+ ),
+ room_seq=row.seq,
+ message_key=row.event_id,
)
- agent_event = message_event(
- room_id=meta.room_id,
- bridge_id=meta.bridge_id,
- channel_type=meta.channel_type,
- event=event,
- addressed=is_addressed,
- sender_name=sender_name,
- body=body,
- thread_id=thread_id,
- attachments=attachments,
- )
- if is_addressed:
- self._report_addressed(
- meta, event, agent_live=agent_live, has_attachment=True
- )
+ async def after(_row: InboundEvent) -> None:
+ if answer is not None:
+ await answer()
- if not is_addressed:
- return
- if logger.isEnabledFor(logging.DEBUG):
- logger.debug(
- "Enqueuing media event %s", agent_event.model_dump_json(indent=2)
- )
- await self._event_buffer.enqueue(
- self.tenant_id,
- self.agent.id,
- meta.room_id,
- agent_event,
- room_seq=room_seq,
- message_key=message_key,
- )
+ return Delivery(own_event=own_event, after=after)
- async def on_command(self, room: RoomRef, event: CommandEvent) -> None:
- meta = await self._resolve_room_meta(room.room_id)
- if meta is None:
- return
+ async def handle_command(self, event: CommandEvent) -> None:
+ """Run a room command for the agent, once the command has committed.
- # Admin-owned commands (room/info/alias) are handled by the admin
- # client, which is in every room; agents ignore them entirely.
+ Admin-owned commands (room, info, alias) are the admin client's, which
+ is in every room; agents ignore them entirely.
+ """
+ room = self._room
+ meta = self._meta
cmd = COMMANDS_BY_NAME.get(event.command)
if cmd is not None and cmd.admin_owned:
return
@@ -928,8 +971,8 @@ class AgentConsumer(Consumer[AgentActor]):
) -> bool:
"""Apply the addressing policy to a command aimed at this agent.
- Commands drive the agent as surely as a message does — `!reset` wipes
- its context, `!interrupt` stops it mid-task — so a sender who may not
+ Commands drive the agent as surely as a message does: `!reset` wipes
+ its context, `!interrupt` stops it mid-task. So a sender who may not
address the agent may not command it either.
A refusal is posted only when the command named this agent. A room-wide
@@ -938,23 +981,37 @@ class AgentConsumer(Consumer[AgentActor]):
would flood a room holding several restricted agents; those are declined
quietly, with a warning in the log.
"""
+ refusal = await self._command_refusal(room, event, meta)
+ if refusal is None:
+ return True
+ await refusal()
+ return False
+
+ async def _command_refusal(
+ self, room: RoomRef, event: CommandEvent, meta: RoomMeta
+ ) -> Callable[[], Awaitable[None]] | None:
+ """None when the sender may command this agent, and otherwise how the
+ command is declined (`_gate_command`)."""
async with self.session_factory() as session:
agent = await self._fresh_agent(session)
decision = await self._addressing_allowed(
session, agent, event.user_id, meta.room_id
)
if decision.allowed:
- return True
+ return None
targets_me = await self._command_targets_me_explicitly(
session, event.args, meta.room_id
)
- if targets_me:
- await self.reply_command(
- room.room_id,
- decision.refusal,
- thread_root_id=event.thread_id,
- )
- else:
+ refusal = decision.refusal
+
+ async def decline() -> None:
+ if targets_me:
+ await self.reply_command(
+ room.room_id,
+ refusal,
+ thread_root_id=event.thread_id,
+ )
+ return
logger.warning(
"Command %s from %s declined for %s in room %s: sender may not "
"address this agent",
@@ -963,9 +1020,8 @@ class AgentConsumer(Consumer[AgentActor]):
self.agent.name,
meta.room_id,
)
- return False
- # ── Command handling ──────────────────────────────────────────────────────
+ return decline
async def _handle_command(self, room: RoomRef, event: CommandEvent) -> bool:
is_direct = await self._is_direct_room(room.room_id)
@@ -990,35 +1046,110 @@ class AgentConsumer(Consumer[AgentActor]):
)
async def _resolve_room_meta(self, transport_room_id: str) -> RoomMeta | None:
- if transport_room_id in self._room_meta:
- return self._room_meta[transport_room_id]
+ """The room this recipient decides for, which the router read with the
+ row. A command handler asks by the room's transport id."""
+ if transport_room_id != self._room.room_id:
+ logger.error(
+ "Agent %s was asked about room %s while deciding a row in %s",
+ self.agent.name,
+ transport_room_id,
+ self._room.room_id,
+ )
+ return None
+ return self._meta
+
+ # ── Tasks ─────────────────────────────────────────────────────────────────
+
+ def decide_task(self, event: SwitchEvent) -> Delivery:
+ """Whether a task event is one of the agent's own events: it is for
+ the agent doing the task when it is delegated or cancelled, and for
+ the agent that asked when it is accepted, updated or finalised."""
+ meta = self._meta
+ me = self.agent.id
+ payload: (
+ TaskDelegatePayload
+ | TaskAcceptPayload
+ | TaskUpdatePayload
+ | TaskFinalisePayload
+ | TaskCancelPayload
+ )
+ if isinstance(event, TaskDelegate):
+ if event.performer_agent_id != me:
+ return NOTHING
+ kind = "task_delegate"
+ payload = TaskDelegatePayload(
+ task_id=event.task_id,
+ requester_agent_id=event.requester_agent_id,
+ performer_agent_id=event.performer_agent_id,
+ summary=event.summary,
+ description=event.description,
+ )
+ elif isinstance(event, TaskAccept):
+ if event.requester_agent_id != me:
+ return NOTHING
+ kind = "task_accept"
+ payload = TaskAcceptPayload(
+ task_id=event.task_id,
+ requester_agent_id=event.requester_agent_id,
+ performer_agent_id=event.performer_agent_id,
+ )
+ elif isinstance(event, TaskUpdate):
+ if event.requester_agent_id != me:
+ return NOTHING
+ kind = "task_update"
+ payload = TaskUpdatePayload(
+ task_id=event.task_id,
+ requester_agent_id=event.requester_agent_id,
+ performer_agent_id=event.performer_agent_id,
+ update=event.update,
+ )
+ elif isinstance(event, TaskFinalise):
+ if event.requester_agent_id != me:
+ return NOTHING
+ kind = "task_finalise"
+ payload = TaskFinalisePayload(
+ task_id=event.task_id,
+ requester_agent_id=event.requester_agent_id,
+ performer_agent_id=event.performer_agent_id,
+ outcome=event.outcome,
+ )
+ elif isinstance(event, TaskCancel):
+ if event.performer_agent_id != me:
+ return NOTHING
+ kind = "task_cancel"
+ payload = TaskCancelPayload(
+ task_id=event.task_id,
+ requester_agent_id=event.requester_agent_id,
+ performer_agent_id=event.performer_agent_id,
+ reason=event.reason,
+ )
+ else:
+ return NOTHING
+
+ def own_event(row: InboundEvent) -> OwnEvent:
+ return OwnEvent(
+ AgentEvent(
+ type=kind,
+ room_id=meta.room_id,
+ bridge_id=meta.bridge_id,
+ channel_type=meta.channel_type,
+ payload=payload,
+ ),
+ room_seq=row.seq,
+ message_key=row.event_id,
+ )
- async with tenant_session(self.session_factory, self.tenant_id) as session:
- room = await self._room_store.get_by_transport_room_id(
- session, transport_room_id
+ async def working_on_it(_row: InboundEvent) -> None:
+ await self.actor.send_message(
+ self._room.room_id, "Working on it.", format="markdown", metered=False
)
- if room is None:
- logger.error(
- "Room not found for transport room id: %s", transport_room_id
- )
- self._room_meta[transport_room_id] = None
- return None
- agent_greetings_enabled = True
- if room.bridge_id is not None:
- bridge = await self._bridge_store.get(session, room.bridge_id)
- if bridge is not None:
- agent_greetings_enabled = bridge.agent_greetings_enabled
-
- meta = RoomMeta(
- room_id=room.id,
- name=room.name,
- bridge_id=room.bridge_id,
- agent_greetings_enabled=agent_greetings_enabled,
- channel_type=room.channel_type,
+ return Delivery(
+ own_event=own_event,
+ after=working_on_it if isinstance(event, TaskDelegate) else None,
)
- self._room_meta[transport_room_id] = meta
- return meta
+
+ # ── Offline and refusal notices ───────────────────────────────────────────
async def _reply_when_unavailable_here(
self, session: AsyncSession, agent: Agent, meta: RoomMeta, asker_handle: str
@@ -1027,17 +1158,18 @@ class AgentConsumer(Consumer[AgentActor]):
None when there is nothing to post.
If the agent has live session(s) connected to OTHER rooms, name them so
- the asker knows where to find it — a session hops rooms while keeping
+ the asker knows where to find it: a session hops rooms while keeping
its heartbeat (and any role lease) alive, so the agent is reachable
there even though it has no session in the room it was addressed from.
When the agent also holds a role in *this* room, the wording notes that
the role-assuming session has stepped away. A live session in a
- genuinely different room wins — the asker is pointed there. Sessions in
- rooms that show the SAME name as this one are NOT offered (pointing the
- asker to a room they appear to already be in is confusing and useless).
+ genuinely different room wins, and the asker is pointed there. Sessions
+ in rooms that show the SAME name as this one are NOT offered (pointing
+ the asker to a room they appear to already be in is confusing and
+ useless).
session_passive agents have no heartbeat, so the elsewhere/role logic
- never applies — they fall through to the generic reply, whose wording
+ never applies: they fall through to the generic reply, whose wording
already covers pulling from an existing session or starting a new one.
For a session_addressable agent with a session bound to THIS room but
not live here (and no live session in a distinct room), the reply says
@@ -1057,7 +1189,7 @@ class AgentConsumer(Consumer[AgentActor]):
)
# auto_session: if a connector is actively watching (global heartbeat),
- # it will spin a session up to handle this — promise that rather than
+ # it will spin a session up to handle this, so promise that rather than
# the "elsewhere"/offline wording. With no watcher, fall through to the
# generic offline reply below.
# A live connection that declared itself spawn-capable and covers this
@@ -1073,7 +1205,7 @@ class AgentConsumer(Consumer[AgentActor]):
# The heartbeat arm only: a client still running the
# /watch/heartbeat loop declares no capability, and that loop meant
# willingness. A connection declares its own, which the check above
- # has already asked — reading a connection as willing because it
+ # has already asked; reading a connection as willing because it
# exists promises a session over a session worker that will never
# spawn one, or over a controller with auto-start switched off.
watching = await self._agent_session_store.get_live_agent_ids(
@@ -1142,7 +1274,7 @@ class AgentConsumer(Consumer[AgentActor]):
Never the terminal command or "open Switch Console": a managed agent
is started by its controller, so the reply names what stands in the
- way — the owner stopped it, or its machine was removed or is offline.
+ way: the owner stopped it, or its machine was removed or is offline.
None while it is connected: the message is delivered, and any notice
that a session is starting is the agent's own to post.
"""
@@ -1199,8 +1331,8 @@ class AgentConsumer(Consumer[AgentActor]):
Resolved from who owns the agent and which account that person has
claimed on this bridge, rather than from a handle configured on the
- agent. A handle only means anything on one platform — the same person
- is one name on Slack and another on Telegram — so a single stored
+ agent. A handle only means anything on one platform (the same person
+ is one name on Slack and another on Telegram), so a single stored
string was at best right in one room and inert everywhere else.
None when the agent has no owner, the room has no bridge, or the owner
@@ -1236,10 +1368,9 @@ class AgentConsumer(Consumer[AgentActor]):
has no known-agent spec or the spec returns None; `connected_not_live`
selects the "bound here but not live" fallback.
- `agent` is the freshly-read row supplied by the caller (via
- `_fresh_agent`), so option edits made through the gateway (e.g.
- `PATCH /agents/{id}/options`, `connection_model`) take effect without
- restarting this client — `self.agent` alone is a boot-time snapshot.
+ `agent` is the row read for this routing, so option edits made through
+ the gateway (e.g. `PATCH /agents/{id}/options`, `connection_model`)
+ take effect on the next message.
"""
connection_model = (agent.integration_profile or {}).get(
"connection_model", "session_passive"
@@ -1310,7 +1441,7 @@ class AgentConsumer(Consumer[AgentActor]):
[self.agent.id], room_id, self._connections
):
# A session in the room, or a claimed room slot no session of this
- # agent accounts for — not mere coverage: an `all`-scope watcher
+ # agent accounts for. Not mere coverage: an `all`-scope watcher
# covering this room is not a session that can answer.
return True
@@ -1324,146 +1455,6 @@ class AgentConsumer(Consumer[AgentActor]):
meta = await self._resolve_room_meta(transport_room_id)
return meta is not None and meta.channel_type == "direct"
- # ── Task event forwarding ────────────────────────────────────────────────
-
- async def on_task_delegate(self, room: RoomRef, event: TaskDelegate) -> None:
- if event.performer_agent_id != self.agent.id:
- return
- meta = await self._resolve_room_meta(room.room_id)
- agent_event = (
- None
- if meta is None
- else AgentEvent(
- type="task_delegate",
- room_id=meta.room_id,
- bridge_id=meta.bridge_id,
- channel_type=meta.channel_type,
- payload=TaskDelegatePayload(
- task_id=event.task_id,
- requester_agent_id=event.requester_agent_id,
- performer_agent_id=event.performer_agent_id,
- summary=event.summary,
- description=event.description,
- ),
- )
- )
- await self.actor.send_message(
- room.room_id, "Working on it.", format="markdown", metered=False
- )
- if meta is None or agent_event is None:
- return
- await self._event_buffer.enqueue(
- self.tenant_id,
- self.agent.id,
- meta.room_id,
- agent_event,
- room_seq=event.room_seq,
- message_key=event.source_event_id,
- )
-
- async def on_task_accept(self, room: RoomRef, event: TaskAccept) -> None:
- if event.requester_agent_id != self.agent.id:
- return
- meta = await self._resolve_room_meta(room.room_id)
- if meta is None:
- return
- await self._event_buffer.enqueue(
- self.tenant_id,
- self.agent.id,
- meta.room_id,
- AgentEvent(
- type="task_accept",
- room_id=meta.room_id,
- bridge_id=meta.bridge_id,
- channel_type=meta.channel_type,
- payload=TaskAcceptPayload(
- task_id=event.task_id,
- requester_agent_id=event.requester_agent_id,
- performer_agent_id=event.performer_agent_id,
- ),
- ),
- room_seq=event.room_seq,
- message_key=event.source_event_id,
- )
-
- async def on_task_update(self, room: RoomRef, event: TaskUpdate) -> None:
- if event.requester_agent_id != self.agent.id:
- return
- meta = await self._resolve_room_meta(room.room_id)
- if meta is None:
- return
- await self._event_buffer.enqueue(
- self.tenant_id,
- self.agent.id,
- meta.room_id,
- AgentEvent(
- type="task_update",
- room_id=meta.room_id,
- bridge_id=meta.bridge_id,
- channel_type=meta.channel_type,
- payload=TaskUpdatePayload(
- task_id=event.task_id,
- requester_agent_id=event.requester_agent_id,
- performer_agent_id=event.performer_agent_id,
- update=event.update,
- ),
- ),
- room_seq=event.room_seq,
- message_key=event.source_event_id,
- )
-
- async def on_task_finalise(self, room: RoomRef, event: TaskFinalise) -> None:
- if event.requester_agent_id != self.agent.id:
- return
- meta = await self._resolve_room_meta(room.room_id)
- if meta is None:
- return
- await self._event_buffer.enqueue(
- self.tenant_id,
- self.agent.id,
- meta.room_id,
- AgentEvent(
- type="task_finalise",
- room_id=meta.room_id,
- bridge_id=meta.bridge_id,
- channel_type=meta.channel_type,
- payload=TaskFinalisePayload(
- task_id=event.task_id,
- requester_agent_id=event.requester_agent_id,
- performer_agent_id=event.performer_agent_id,
- outcome=event.outcome,
- ),
- ),
- room_seq=event.room_seq,
- message_key=event.source_event_id,
- )
-
- async def on_task_cancel(self, room: RoomRef, event: TaskCancel) -> None:
- if event.performer_agent_id != self.agent.id:
- return
- meta = await self._resolve_room_meta(room.room_id)
- if meta is None:
- return
- await self._event_buffer.enqueue(
- self.tenant_id,
- self.agent.id,
- meta.room_id,
- AgentEvent(
- type="task_cancel",
- room_id=meta.room_id,
- bridge_id=meta.bridge_id,
- channel_type=meta.channel_type,
- payload=TaskCancelPayload(
- task_id=event.task_id,
- requester_agent_id=event.requester_agent_id,
- performer_agent_id=event.performer_agent_id,
- reason=event.reason,
- ),
- ),
- room_seq=event.room_seq,
- message_key=event.source_event_id,
- )
-
# ── Mention detection ─────────────────────────────────────────────────────
@staticmethod
@@ -1482,9 +1473,8 @@ class AgentConsumer(Consumer[AgentActor]):
"""The addressing answer that needs no database read, or None when the
room's aliases and role leases have to be consulted.
- Callers use it to decide whether to take a pool slot at all: most room
- chatter carries no `@` and is answered here, and a room fans every
- message out to all of its agent clients at once.
+ Most room chatter carries no `@` and is answered here, without asking
+ the reads the router makes for the whole room.
"""
return self._addressing.addressed_without_lookup(
agent=self.agent,
@@ -1493,14 +1483,6 @@ class AgentConsumer(Consumer[AgentActor]):
message=self._as_incoming(event),
)
- async def _addressed(self, event: InboundMessage, meta: RoomMeta) -> bool:
- """`_compute_addressed`, in a session of its own when it needs one."""
- decided = self._addressed_without_lookup(event, meta)
- if decided is not None:
- return decided
- async with self.session_factory() as session:
- return await self._compute_addressed(session, event, meta)
-
async def _compute_addressed(
self, session: AsyncSession, event: InboundMessage, meta: RoomMeta
) -> bool:
@@ -1524,9 +1506,9 @@ class AgentConsumer(Consumer[AgentActor]):
"""Whether `sender_user_id` may address this agent in `room_id`, per the
agent's scoped addressing policy.
- `agent` is the freshly-read row rather than the cached snapshot: the
- policy is the thing being enforced, and enforcing a stale copy of it is
- the one way this check can be wrong in the dangerous direction.
+ `agent` is the row read for this routing rather than an older copy:
+ the policy is the thing being enforced, and enforcing a stale copy of
+ it is the one way this check can be wrong in the dangerous direction.
`content` is the event's content when there is one: a platform message
says there whose authority it carries.
"""
@@ -1548,10 +1530,11 @@ class AgentConsumer(Consumer[AgentActor]):
"""Apply the scoped addressing policy to a message that tags this agent.
When the sender is not permitted by the agent's policy, or the agent
- has reached a budget covering it, the message is demoted to unaddressed room chatter and the caller is handed a refusal
- to post (once, guarded by AUTO_REPLY_FLAG so two agents can't
- ping-pong). Zero cost for the common case: only messages that already
- tag this agent are ever checked, and open policies short-circuit.
+ has reached a budget covering it, the message is demoted to unaddressed
+ room chatter and the caller is handed a refusal to post (once, guarded
+ by AUTO_REPLY_FLAG so two agents can't ping-pong). Zero cost for the
+ common case: only messages that already tag this agent are ever
+ checked, and open policies short-circuit.
"""
decision = await self._addressing_allowed(
session, agent, event.sender, meta.room_id, event.content
@@ -1589,14 +1572,15 @@ class AgentConsumer(Consumer[AgentActor]):
not the agent could take it, so an offline or stopped agent is still
asked. `agent_live` is the agent having a live session for the room.
Not an auto-reply: Switch's notice that another agent is offline or
- refused is not someone asking this one for something. Not the files of a multi-file post delivered without
- its first part, which happens when a group times out incomplete and
- the rest arrives later: the post is counted with its first part, as
- the transport counts it, so it is one request however it is split.
+ refused is not someone asking this one for something. Not the files of
+ a multi-file post delivered without its first part, which happens when
+ a post is decided incomplete and the rest arrives later: the post is
+ counted with its first part, as the transport counts it, so it is one
+ request however it is split.
Guarded like the transport's observer call, and counted the same way:
- this runs on the delivery path, ahead of the enqueue, and a telemetry
- bug must not cost the agent the message.
+ this runs after the message has committed, and a telemetry bug must
+ not cost the agent the replies that follow it.
"""
if self._triggered_by_auto_reply(event):
return
@@ -1701,3 +1685,32 @@ class AgentConsumer(Consumer[AgentActor]):
re.IGNORECASE,
)
return pattern.sub("", text).strip()
+
+
+def file_index(part: InboundMedia) -> int:
+ """A file's place in its post: its group index, or 0 for a lone file."""
+ group = parse_attachment_group(part.content)
+ return group[1] if group is not None else 0
+
+
+def _first_file(parts: list[InboundMedia]) -> InboundMedia | None:
+ """The file a post is anchored on, index 0, when it is among `parts`."""
+ return next((part for part in parts if file_index(part) == 0), None)
+
+
+def command_event(row: InboundCustomEvent) -> CommandEvent:
+ """A command as the row it was written in reads, with the ids the row
+ alone knows filled in.
+
+ Command results reply into the command's thread. When the command was
+ typed inside an existing thread the bridge relates it to that thread's
+ root; that root is used so the result stays in the thread. Otherwise the
+ command itself roots the thread and its own event id is used. The command
+ keeps its own id as well, so two commands in one thread are distinct.
+ """
+ command = CommandEvent(**row.content)
+ command.message_id = row.event_id
+ command.thread_id = row.thread_root_id or row.event_id
+ command.room_seq = row.seq
+ command.source_event_id = row.event_id
+ return command
diff --git a/core/switch_core/delivery/router.py b/core/switch_core/delivery/router.py
new file mode 100644
index 000000000..c3d417ebb
--- /dev/null
+++ b/core/switch_core/delivery/router.py
@@ -0,0 +1,1649 @@
+"""Deciding, as a room row is written, which agents it is for.
+
+Every row written to a room goes through `MessageRouter.store`, in the
+transaction that writes it: a message, a file, a command, a task event, an
+arrival. The router decides, for every agent member of the room, whether the
+row is one of that agent's own events (`delivery/recipient.py` holds the
+rules), and stores those events (`agent_events`) in the same transaction.
+What the decisions leave to say in the room (a refusal, an offline notice, a
+greeting, a command's answer) is posted once the transaction has committed,
+by the process that wrote the row.
+
+It replaced an agent consumer per agent in every process, each woken for
+every row of its rooms to ask "is this for me?" and to store its answer in a
+transaction of its own. Three things are better for it:
+
+- **Atomic.** A row and the own events decided for it commit together or not
+ at all. A consumer's place in a room lived in memory, so a process that
+ stopped between storing a row and deciding it lost the row for the agent:
+ a consumer started afterwards read each room from its head.
+- **Cheaper.** One row woke agents times processes consumers, each with its
+ own session; the router decides once, in the writer's session, with the
+ reads every agent would have made asked once for the whole room
+ (`RoomReads`).
+- **Nothing to wait for.** A reader of a room's conversation used to hold back
+ each new row for two seconds, so the consumer could decide first whether it
+ was an own event. A row is decided by the time anything can read it now.
+
+How the transaction runs, and why in this order:
+
+1. Reads first, before the room's lock: the room, its agent members and
+ whatever the decisions ask (aliases, role leases, the sender, budgets).
+ `MessageStore.create` takes the room's advisory lock to number the row,
+ and writers to one room wait on each other while it is held, so only
+ inserts happen inside it.
+2. The row is inserted and numbered (`MessageStore.create`, called from here
+ and nowhere else; a test holds the code to that). An agent whose arrival
+ committed between the reads and the lock comes before the row in the
+ room, so the members are checked once more and, if one joined, the row is
+ decided again from the members as they are.
+3. The own events are inserted, one savepoint per agent, in the order of the
+ agents' ids. Each claims the agent's next number from its row in
+ `agent_event_heads`, which stays locked to commit, so an agent's events are
+ numbered in commit order; a fixed order means two rows for the same two
+ agents cannot each hold one agent's row and wait for the other's. A
+ failure for one agent is logged and counted, and the row still commits.
+
+Multi-file posts are written one file per row and are decided once, as a
+whole: when the file that completes the post is written, or, when the rest
+never comes, by a sweep every process runs once `ATTACHMENT_GROUP_TIMEOUT_SECONDS`
+have passed since the first file (`PendingAttachmentGroup`, `flush_due_groups`).
+
+During a rolling deploy a process still on the release before this one writes
+rows without deciding them, and its agent consumers decide them a moment
+later. The rows this release writes are decided here and once more by those
+consumers, which find the events already stored (`uq_agent_events_source`)
+but post their replies a second time until the old processes are gone.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextvars
+import logging
+from collections import deque
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass, field
+from datetime import UTC, datetime, timedelta
+from typing import TYPE_CHECKING, Any, Literal, cast, overload
+
+from sqlalchemy import and_, delete, func, literal, select, update
+from sqlalchemy.dialects.postgresql import insert
+from sqlalchemy.exc import DBAPIError
+
+from switch_core.attachments import ATTACHMENT_GROUP_KEY, parse_attachment_group
+from switch_core.budgets import BudgetGuard
+from switch_core.clients.actor import AgentActor, ClientConfig
+from switch_core.clients.admin_messages import PLATFORM_MARKER
+from switch_core.clients.room_meta import RoomMeta
+from switch_core.db.models import (
+ Agent,
+ AgentConnectionRecord,
+ Client,
+ ClientRoom,
+ CollaborationBridge,
+ Message,
+ PendingAttachmentGroup,
+ RoleLease,
+ Room,
+ SwitchCoreProcess,
+ room_agents,
+)
+from switch_core.db.session_scope import tenant_session
+from switch_core.delivery.addressing import AddressingResolver, SenderPrincipal
+from switch_core.delivery.recipient import (
+ NOTHING,
+ AgentRecipient,
+ Delivery,
+ RecipientServices,
+ RoomMember,
+ command_event,
+ file_index,
+)
+from switch_core.events import (
+ SwitchEvent,
+ TaskAccept,
+ TaskCancel,
+ TaskDelegate,
+ TaskFinalise,
+ TaskUpdate,
+)
+from switch_core.logging_context import log_context
+from switch_core.observability.catalogue import ROUTING_FAILURES
+from switch_core.observability.metrics import metrics
+from switch_core.process_lease import LEASE_TTL
+from switch_core.tenant_context import tenant_scope
+from switch_core.transport.stored import to_inbound
+from switch_core.transport.types import (
+ InboundCustomEvent,
+ InboundEvent,
+ InboundMedia,
+ InboundMembership,
+ InboundMessage,
+)
+
+if TYPE_CHECKING:
+ from collections.abc import Mapping
+
+ from sqlalchemy.ext.asyncio import AsyncSession
+
+ from switch_core.clients.actor import Actor
+ from switch_core.db.models import MessageAttachment
+ from switch_core.db.stores.budget_store import BudgetStanding, BudgetStore
+ from switch_core.db.stores.client_store import ClientStore
+ from switch_core.db.stores.message_store import MessageStore
+ from switch_core.transport.port import MessageTransport
+
+logger = logging.getLogger(__name__)
+
+# How long a multi-file post waits for the rest of its files before it is
+# decided with the files that did arrive, flagged as incomplete. A post's files
+# normally arrive within milliseconds (one sender, back to back); this is a
+# safety net so a broken batch surfaces rather than being held indefinitely.
+# Counted from the post's first file and never extended by a later one.
+ATTACHMENT_GROUP_TIMEOUT_SECONDS = 5.0
+
+# How often every process looks for posts whose wait is over. The process
+# that wrote a post's first file also sets a timer for its deadline; this is
+# what decides a post whose writer has stopped since.
+GROUP_SWEEP_SECONDS = 5.0
+
+# The custom events an agent can be part of, by type. The rest (reports,
+# observations) are the platform's own bookkeeping and are nobody's own event.
+_TASK_EVENTS: dict[str, type[SwitchEvent]] = {
+ "com.switch.task.delegate": TaskDelegate,
+ "com.switch.task.accept": TaskAccept,
+ "com.switch.task.update": TaskUpdate,
+ "com.switch.task.finalise": TaskFinalise,
+ "com.switch.task.cancel": TaskCancel,
+}
+_COMMAND_EVENT = "com.switch.command"
+
+
+class RoomReads:
+ """What deciding one row reads about its room, read once and shared by
+ every agent the row is decided for.
+
+ The room and its agent members are read up front. Everything else is
+ read the first time an agent's decision asks and kept for the others:
+ the role each agent live-holds here, who the sender is, the tenant's
+ budgets, who a member is by the id they post under. The rules ask the
+ same questions they always did (`AddressingResolver`, `BudgetGuard`);
+ this only answers them for a whole room at once, so what a row costs in
+ reads does not grow with the number of agents in the room.
+ """
+
+ def __init__(
+ self,
+ *,
+ services: RecipientServices,
+ client_store: ClientStore,
+ budget_store: BudgetStore,
+ tenant_id: str,
+ room_id: str,
+ transport_room_id: str,
+ meta: RoomMeta,
+ group_id: str | None,
+ members: list[RoomMember],
+ ) -> None:
+ self.tenant_id = tenant_id
+ self.room_id = room_id
+ self.transport_room_id = transport_room_id
+ self.meta = meta
+ self.group_id = group_id
+ self.members = members
+ self._services = services
+ self._client_store = client_store
+ self._budget_store = budget_store
+ self._roles: dict[str, str] | None = None
+ self._senders: dict[tuple[str, str], SenderPrincipal | None] = {}
+ self._claims: dict[str, list[str]] = {}
+ self._standings: list[BudgetStanding] | None = None
+ self._clients: dict[str, Client | None] = {}
+ self.addressing = _RoomAddressing(
+ self, services=services, client_store=client_store
+ )
+ self.budget_guard = BudgetGuard(_RoomBudgets(self))
+
+ @classmethod
+ async def load(
+ cls,
+ session: AsyncSession,
+ *,
+ services: RecipientServices,
+ client_store: ClientStore,
+ budget_store: BudgetStore,
+ tenant_id: str,
+ room_id: str,
+ ) -> RoomReads | None:
+ """The room and its agent members, or None when the room is gone.
+
+ A member is one whose membership has committed by the time this
+ reads it. Its arrival was written under the room's lock and committed
+ with the membership, so it comes before the row being decided in the
+ room's order, whatever the clocks of the two transactions say.
+ """
+ found = (
+ await session.execute(
+ select(Room, CollaborationBridge.agent_greetings_enabled)
+ .outerjoin(
+ CollaborationBridge,
+ and_(
+ CollaborationBridge.tenant_id == Room.tenant_id,
+ CollaborationBridge.id == Room.bridge_id,
+ ),
+ )
+ .where(Room.id == room_id)
+ )
+ ).first()
+ if found is None:
+ return None
+ room, greetings = found
+ rows = (
+ await session.execute(
+ select(
+ Agent,
+ Client.id,
+ Client.transport_user_id,
+ Client.display_name,
+ room_agents.c.alias,
+ room_agents.c.receives_join_events,
+ ClientRoom.joined_at,
+ )
+ .select_from(ClientRoom)
+ .join(Client, Client.id == ClientRoom.client_id)
+ .join(Agent, Agent.client_id == Client.id)
+ .outerjoin(
+ room_agents,
+ and_(
+ room_agents.c.room_id == ClientRoom.room_id,
+ room_agents.c.agent_id == Agent.id,
+ ),
+ )
+ .where(ClientRoom.room_id == room_id)
+ )
+ ).all()
+ members = [
+ RoomMember(
+ agent=agent,
+ client_id=client_id,
+ transport_user_id=transport_user_id,
+ display_name=display_name,
+ alias=alias,
+ listening=bool(listening),
+ joined_at=joined_at,
+ )
+ for (
+ agent,
+ client_id,
+ transport_user_id,
+ display_name,
+ alias,
+ listening,
+ joined_at,
+ ) in rows
+ ]
+ return cls(
+ services=services,
+ client_store=client_store,
+ budget_store=budget_store,
+ tenant_id=tenant_id,
+ room_id=room.id,
+ transport_room_id=room.transport_room_id,
+ meta=RoomMeta(
+ room_id=room.id,
+ name=room.name,
+ bridge_id=room.bridge_id,
+ agent_greetings_enabled=True if greetings is None else greetings,
+ channel_type=room.channel_type,
+ ),
+ group_id=room.group_id,
+ members=sorted(members, key=lambda member: member.agent.id),
+ )
+
+ def member(self, client_id: str | None) -> RoomMember | None:
+ """The agent member posting as `client_id`, if it is one."""
+ return next(
+ (member for member in self.members if member.client_id == client_id),
+ None,
+ )
+
+ def alias_of(self, agent_id: str) -> str | None:
+ return next(
+ (m.alias for m in self.members if m.agent.id == agent_id),
+ None,
+ )
+
+ async def role_names(self, session: AsyncSession) -> dict[str, str]:
+ """The role each agent live-holds in the room, by agent id.
+
+ A seat taken over a connection is held while that connection is
+ live. This process's memory knows its own connections, but its copy
+ of another process's can be missing or late, and a message decided
+ without a live holder is never decided again. So the connections
+ recorded as holding a seat here are read as well: one counts while
+ its row is open and the process named on it holds its lease.
+ """
+ if self._roles is None:
+ live = self._services.connections.live_connection_ids()
+ live |= set(
+ (
+ await session.scalars(
+ select(AgentConnectionRecord.id).where(
+ AgentConnectionRecord.id.in_(
+ select(RoleLease.transport_session_id).where(
+ RoleLease.room_id == self.room_id
+ )
+ ),
+ AgentConnectionRecord.closed_at.is_(None),
+ select(SwitchCoreProcess.id)
+ .where(
+ SwitchCoreProcess.id
+ == AgentConnectionRecord.process_id,
+ SwitchCoreProcess.stopped_at.is_(None),
+ SwitchCoreProcess.beat_at
+ > func.now() - literal(LEASE_TTL),
+ )
+ .exists(),
+ )
+ )
+ ).all()
+ )
+ self._roles = await self._services.room_role_store.room_role_names(
+ session, self.room_id, live
+ )
+ return self._roles
+
+ async def sender(
+ self,
+ transport_user_id: str,
+ content: Mapping[str, object] | None,
+ resolve: Callable[[], Awaitable[SenderPrincipal | None]],
+ ) -> SenderPrincipal | None:
+ # Keyed on the platform marker as well as the sender: an admin client's
+ # message speaks for whoever its marker names.
+ marker = repr((content or {}).get(PLATFORM_MARKER))
+ key = (transport_user_id, marker)
+ if key not in self._senders:
+ self._senders[key] = await resolve()
+ return self._senders[key]
+
+ async def claimed_accounts(
+ self, user_id: str, read: Callable[[], Awaitable[list[str]]]
+ ) -> list[str]:
+ if user_id not in self._claims:
+ self._claims[user_id] = await read()
+ return self._claims[user_id]
+
+ async def standings(self, session: AsyncSession) -> list[BudgetStanding]:
+ if self._standings is None:
+ self._standings = await self._budget_store.standings(
+ session, tenant_id=self.tenant_id, agent_id=None
+ )
+ return self._standings
+
+ async def get_by_transport_user_id(
+ self, session: AsyncSession, transport_user_id: str
+ ) -> Client | None:
+ """Who posts as `transport_user_id`, for naming an arrival."""
+ if transport_user_id not in self._clients:
+ self._clients[
+ transport_user_id
+ ] = await self._client_store.get_by_transport_user_id(
+ session, transport_user_id
+ )
+ return self._clients[transport_user_id]
+
+
+class _RoomAddressing(AddressingResolver):
+ """The addressing rules, answered from one row's `RoomReads`."""
+
+ def __init__(
+ self,
+ reads: RoomReads,
+ *,
+ services: RecipientServices,
+ client_store: ClientStore,
+ ) -> None:
+ super().__init__(
+ room_store=services.room_store,
+ room_role_store=services.room_role_store,
+ client_store=client_store,
+ agent_store=services.agent_store,
+ external_user_store=services.external_user_store,
+ live_connection_ids=services.connections.live_connection_ids,
+ )
+ self._reads = reads
+
+ async def alias_of(
+ self, session: AsyncSession, room_id: str, agent_id: str
+ ) -> str | None:
+ if room_id != self._reads.room_id:
+ return await super().alias_of(session, room_id, agent_id)
+ return self._reads.alias_of(agent_id)
+
+ async def role_of(
+ self, session: AsyncSession, room_id: str, agent_id: str
+ ) -> str | None:
+ if room_id != self._reads.room_id:
+ return await super().role_of(session, room_id, agent_id)
+ return (await self._reads.role_names(session)).get(agent_id)
+
+ async def group_of(self, session: AsyncSession, room_id: str) -> str | None:
+ if room_id != self._reads.room_id:
+ return await super().group_of(session, room_id)
+ return self._reads.group_id
+
+ async def claimed_accounts(self, session: AsyncSession, user_id: str) -> list[str]:
+ read = super().claimed_accounts
+ return await self._reads.claimed_accounts(
+ user_id, lambda: read(session, user_id)
+ )
+
+ async def resolve_sender(
+ self,
+ session: AsyncSession,
+ transport_user_id: str,
+ content: Mapping[str, object] | None = None,
+ ) -> SenderPrincipal | None:
+ resolve = super().resolve_sender
+ return await self._reads.sender(
+ transport_user_id,
+ content,
+ lambda: resolve(session, transport_user_id, content),
+ )
+
+
+class _RoomBudgets:
+ """A tenant's budgets read once for one row, and narrowed to the ones
+ covering each agent the way `BudgetStore.standings` narrows them: the
+ agent's own and the tenant-wide ones, in the same order."""
+
+ def __init__(self, reads: RoomReads) -> None:
+ self._reads = reads
+
+ async def standings(
+ self, session: AsyncSession, *, tenant_id: str, agent_id: str | None
+ ) -> list[BudgetStanding]:
+ every = await self._reads.standings(session)
+ return [
+ standing
+ for standing in every
+ if agent_id is None
+ or standing.agent_id is None
+ or standing.agent_id == agent_id
+ ]
+
+
+@dataclass
+class _GroupStep:
+ """What writing one file does to its post's pending row."""
+
+ group_id: str
+ total: int
+ # The file the pending post's files are counted from, when the post was
+ # pending as this file was decided; None when this file starts it.
+ pending_from: int | None
+ complete: bool
+
+
+@dataclass
+class _Plan:
+ """What was decided for one row, before it is written."""
+
+ deliveries: list[tuple[AgentRecipient, Delivery]] = field(default_factory=list)
+ # The agent whose arrival this row is, when it is one.
+ arrival: AgentRecipient | None = None
+ group: _GroupStep | None = None
+
+
+class Routed:
+ """A row decided and written in its writer's transaction, and what is
+ left to do once that transaction commits.
+
+ The writer calls `committed()` after its commit and never before: a row
+ rolled back must not wake an agent, greet anybody or post a reply.
+ """
+
+ def __init__(
+ self,
+ router: MessageRouter,
+ reads: RoomReads | None,
+ plan: _Plan,
+ row: InboundEvent,
+ *,
+ seq: int,
+ stored: list[str],
+ deadline_in: float | None,
+ ) -> None:
+ self._router = router
+ self._reads = reads
+ self._plan = plan
+ self._row = row
+ # The row's place in its room.
+ self.seq = seq
+ self._stored = stored
+ self._deadline_in = deadline_in
+
+ def committed(self) -> None:
+ """Wake the agents the row is for, and leave the replies it owes to
+ run in the background, in the order the room's rows were written.
+
+ Never raises. The row is committed by now, and a writer told its send
+ failed would send it again: a failure here is logged and counted, and
+ costs only what it was about to do.
+ """
+ try:
+ self._committed()
+ except Exception:
+ metrics().increment(ROUTING_FAILURES, {"stage": "after_commit"})
+ logger.error(
+ "Could not finish routing row %s after its commit; it is written "
+ "and its own events are stored",
+ self._row.event_id,
+ exc_info=True,
+ )
+
+ def _committed(self) -> None:
+ router = self._router
+ for agent_id in self._stored:
+ router.services.event_buffer.ring(agent_id)
+ reads = self._reads
+ if reads is None:
+ return
+ plan = self._plan
+ if plan.arrival is not None:
+ router.services.connections.controllers.room_joined(
+ plan.arrival.agent.id, reads.room_id
+ )
+ if self._deadline_in is not None:
+ router.flush_groups_after(reads.tenant_id, self._deadline_in)
+ work: list[tuple[AgentRecipient, Callable[[], Awaitable[None]]]] = []
+ row = self._row
+ for recipient, delivery in plan.deliveries:
+ if delivery.after is not None:
+ work.append((recipient, _bind(recipient, delivery.after, row)))
+ if plan.arrival is not None:
+ arrival = plan.arrival
+ work.append((arrival, _bind(arrival, lambda _row: arrival.greet(), row)))
+ if work:
+ router.after_commit(reads.tenant_id, reads.room_id, work)
+
+
+def _bind(
+ recipient: AgentRecipient,
+ after: Callable[[InboundEvent], Awaitable[None]],
+ row: InboundEvent,
+) -> Callable[[], Awaitable[None]]:
+ """The work `row` left the agent, run as the agent once the row has
+ committed, and only while the agent is still a member of the room
+ (`AgentRecipient.still_member`)."""
+
+ async def run() -> None:
+ if not await recipient.still_member():
+ logger.info(
+ "Agent %s was taken out of room %s after row %s was decided; "
+ "what the row left it to say is not said",
+ recipient.agent.name,
+ row.room_id,
+ row.event_id,
+ )
+ return
+ await recipient.actor.connect()
+ await after(row)
+
+ return run
+
+
+class Removed:
+ """An agent taken out of a room, and what is let go of once the removal
+ has committed."""
+
+ def __init__(
+ self,
+ router: MessageRouter,
+ *,
+ tenant_id: str,
+ agent_id: str | None,
+ room_id: str,
+ ) -> None:
+ self._router = router
+ self._tenant_id = tenant_id
+ self._agent_id = agent_id
+ self._room_id = room_id
+
+ async def committed(self) -> None:
+ """Take the room off the agent's connections, whichever process holds
+ them, and off a controller streaming the agent from here
+ (`MessageRouter.release_room`).
+
+ Never raises. The removal is committed by now, and a caller told it
+ failed would remove the agent again: a failure here is logged and
+ counted, and the process holding a connection that still claims the
+ room lets it go when the membership signal reaches it.
+ """
+ if self._agent_id is None:
+ return
+ try:
+ await self._router.release_room(
+ self._tenant_id, self._agent_id, self._room_id
+ )
+ except Exception:
+ metrics().increment(ROUTING_FAILURES, {"stage": "after_commit"})
+ logger.error(
+ "Could not release room %s from agent %s's connections after "
+ "the agent was removed from it",
+ self._room_id,
+ self._agent_id,
+ exc_info=True,
+ )
+
+
+@dataclass(frozen=True)
+class _Unwritten:
+ """A row about to be written, as `to_inbound` reads one.
+
+ Everything but its place in the room and the time the database will
+ stamp on it, which nothing that decides a row reads: an own event is
+ built from the row once it is written (`Delivery.own_event`).
+ """
+
+ content: dict[str, Any]
+ transport_event_id: str
+ sender_id: str
+ sent_at: Any
+ event_type: str
+ seq: int
+ body: str | None
+ sender_name: str | None
+ formatted_body: str | None
+ msgtype: str | None
+ thread_root_event_id: str | None
+
+
+class _AfterCommit:
+ """The replies and reports routed rows owe, run once each row's
+ transaction has committed.
+
+ Off the writer's path, as the consumers it replaced ran them: whoever
+ wrote a message does not wait for an offline notice to be composed and
+ posted. One task per room at a time, taking that room's work in the order
+ its rows were written, so a room's notices and command answers come out
+ in the order of the rows that caused them.
+
+ Each piece of work runs in a context of its own, bound to its room's
+ tenant: a writer's request context (its log fields, anything it bound)
+ is not the context of the replies its message causes.
+ """
+
+ def __init__(self) -> None:
+ self._queues: dict[str, deque[tuple[str, Callable[[], Awaitable[None]]]]] = {}
+ self._runners: dict[str, asyncio.Task[None]] = {}
+
+ def add(
+ self,
+ tenant_id: str,
+ room_id: str,
+ work: list[tuple[AgentRecipient, Callable[[], Awaitable[None]]]],
+ ) -> None:
+ queue = self._queues.setdefault(room_id, deque())
+ for recipient, run in work:
+ queue.append((recipient.agent.name, run))
+ if room_id not in self._runners:
+ self._runners[room_id] = asyncio.get_running_loop().create_task(
+ self._drain(tenant_id, room_id), context=contextvars.Context()
+ )
+
+ async def _drain(self, tenant_id: str, room_id: str) -> None:
+ queue = self._queues[room_id]
+ try:
+ with tenant_scope(tenant_id), log_context(room_id=room_id):
+ while queue:
+ agent_name, run = queue.popleft()
+ try:
+ await run()
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ metrics().increment(ROUTING_FAILURES, {"stage": "after_commit"})
+ logger.error(
+ "A reply owed by agent %s in room %s could not be "
+ "posted; the message it answers was delivered",
+ agent_name,
+ room_id,
+ exc_info=True,
+ )
+ finally:
+ del self._queues[room_id]
+ del self._runners[room_id]
+
+ async def idle(self) -> None:
+ """Wait until every room's work, including work added meanwhile, is
+ done."""
+ while self._runners:
+ await asyncio.gather(*self._runners.values(), return_exceptions=True)
+
+
+class MessageRouter:
+ """Decides every room row as it is written. One per process."""
+
+ def __init__(
+ self,
+ *,
+ services: RecipientServices,
+ message_store: MessageStore,
+ client_store: ClientStore,
+ budget_store: BudgetStore,
+ ) -> None:
+ self.services = services
+ self._messages = message_store
+ self._client_store = client_store
+ self._budget_store = budget_store
+ self._transports: Callable[[Actor[ClientConfig]], MessageTransport] | None = (
+ None
+ )
+ # The identity each agent posts its replies under, one per agent,
+ # kept so its transport's room lookups are made once.
+ self._posters: dict[str, AgentActor] = {}
+ # The "machine is waking" notices posted for each agent's rooms.
+ self._cloud_waking: dict[str, set[tuple[str, int]]] = {}
+ self._after = _AfterCommit()
+ self._timers: set[asyncio.TimerHandle] = set()
+ self._flushes: set[asyncio.Task[None]] = set()
+
+ def use_transports(
+ self, transport_for: Callable[[Actor[ClientConfig]], MessageTransport]
+ ) -> None:
+ """How an agent's replies are posted: through a transport like every
+ other client's (`ClientFactory.transport_for`), so they are routed in
+ turn. Given by the client factory, which is built with this router."""
+ self._transports = transport_for
+
+ # ── Writing a row ─────────────────────────────────────────────────────────
+
+ @overload
+ async def store(
+ self,
+ session: AsyncSession,
+ *,
+ tenant_id: str,
+ room_id: str,
+ transport_room_id: str,
+ message: Message,
+ attachments: list[MessageAttachment],
+ unless_stored: Literal[False] = False,
+ ) -> Routed: ...
+
+ @overload
+ async def store(
+ self,
+ session: AsyncSession,
+ *,
+ tenant_id: str,
+ room_id: str,
+ transport_room_id: str,
+ message: Message,
+ attachments: list[MessageAttachment],
+ unless_stored: bool,
+ ) -> Routed | None: ...
+
+ async def store(
+ self,
+ session: AsyncSession,
+ *,
+ tenant_id: str,
+ room_id: str,
+ transport_room_id: str,
+ message: Message,
+ attachments: list[MessageAttachment],
+ unless_stored: bool = False,
+ ) -> Routed | None:
+ """Decide `message` for every agent in its room, write it, and store
+ the own events decided for it, all in the caller's transaction.
+
+ The caller commits, then calls `Routed.committed()`.
+
+ With `unless_stored`, a message whose `transport_event_id` is already
+ stored is neither written nor routed again, and None comes back: the
+ id was derived from what caused the write (`transport/derived_ids.py`),
+ and an earlier run of the same cause wrote it and owed what it owed.
+ """
+ reads = await RoomReads.load(
+ session,
+ services=self.services,
+ client_store=self._client_store,
+ budget_store=self._budget_store,
+ tenant_id=tenant_id,
+ room_id=room_id,
+ )
+ plan = _Plan()
+ if reads is not None:
+ unwritten = to_inbound(
+ # A row is decided before it is written; see `_Unwritten`.
+ cast("Message", _unwritten(message)),
+ attachments,
+ transport_room_id=transport_room_id,
+ )
+ plan = await self._decide(
+ session, reads, unwritten, sender_client_id=message.sender_client_id
+ )
+ if not unless_stored:
+ await self._messages.create(session, message, attachments)
+ elif not await self._messages.create_unless_stored(
+ session, message, attachments
+ ):
+ return None
+ if reads is not None and await self._joined_since(session, reads):
+ # An agent's arrival committed between the reads and the room's
+ # lock, so it comes before this row in the room and the row is
+ # for it as much as for the members read. Rare enough to decide
+ # the row again, inside the lock, from the members as they are.
+ reads = await RoomReads.load(
+ session,
+ services=self.services,
+ client_store=self._client_store,
+ budget_store=self._budget_store,
+ tenant_id=tenant_id,
+ room_id=room_id,
+ )
+ plan = _Plan()
+ if reads is not None:
+ plan = await self._decide(
+ session,
+ reads,
+ unwritten,
+ sender_client_id=message.sender_client_id,
+ )
+ row = to_inbound(message, attachments, transport_room_id=transport_room_id)
+ deadline_in: float | None = None
+ stored: list[str] = []
+ if reads is not None:
+ if plan.group is not None:
+ deadline_in = await self._step_group(session, reads, plan, row)
+ stored = await self._store_own_events(session, reads, plan.deliveries, row)
+ return Routed(
+ self,
+ reads,
+ plan,
+ row,
+ seq=message.seq,
+ stored=stored,
+ deadline_in=deadline_in,
+ )
+
+ @staticmethod
+ async def _joined_since(session: AsyncSession, reads: RoomReads) -> bool:
+ """Whether an agent became a member of the room after `reads` read
+ its members. Read with the room's lock held, when every arrival
+ numbered before the row being written has committed."""
+ known = {member.client_id for member in reads.members}
+ now = set(
+ (
+ await session.scalars(
+ select(ClientRoom.client_id)
+ .join(Agent, Agent.client_id == ClientRoom.client_id)
+ .where(ClientRoom.room_id == reads.room_id)
+ )
+ ).all()
+ )
+ return bool(now - known)
+
+ async def membership_changed(self, change: tuple[str, str, str]) -> None:
+ """Bring what this process holds in memory for an agent up to date
+ with a membership written by any process: `change` is the tenant, the
+ client and the room of a `membership.changed` signal.
+
+ The membership's own consequences (the greeting, the arrival handed
+ to the agents listening for joins, the agent's events in a room it
+ left, the room taken off the agent's connections wherever they are)
+ were taken care of once, by the process that wrote it. What is left
+ is per process: the rooms this process keeps for an agent run by a
+ controller, which say where the agent is present; and, should the
+ writer have failed to release them, the claims of the connections
+ this process holds. The writer did all of it as it wrote, so for it
+ this finds nothing to do.
+ """
+ tenant_id, client_id, room_id = change
+ async with tenant_session(self.services.session_factory, tenant_id) as session:
+ found = (
+ await session.execute(
+ select(
+ Agent.id,
+ select(ClientRoom.room_id)
+ .where(
+ ClientRoom.client_id == client_id,
+ ClientRoom.room_id == room_id,
+ )
+ .exists(),
+ ).where(Agent.client_id == client_id)
+ )
+ ).first()
+ if found is None:
+ return
+ agent_id, member = found
+ connections = self.services.connections
+ controllers = connections.controllers
+ if member:
+ if room_id not in controllers.rooms(agent_id):
+ controllers.room_joined(agent_id, room_id)
+ return
+ if room_id in controllers.rooms(agent_id):
+ controllers.room_left(agent_id, room_id)
+ if any(room_id in conn.rooms for conn in connections.held_here(agent_id)):
+ await self._release_claims(tenant_id, agent_id, room_id)
+
+ async def recheck_agent_rooms(self, tenant_id: str) -> None:
+ """Compare what this process holds in memory for the tenant's agents
+ with their memberships, and repair any difference: the periodic
+ re-check behind `membership_changed`, for a signal that never came.
+
+ A read and nothing more when nothing changed, and nothing at all when
+ this process holds no claim and keeps no rooms for a managed agent.
+ """
+ connections = self.services.connections
+ controllers = connections.controllers
+ claimed = connections.claims_held_here()
+ kept = controllers.rooms_kept()
+ agents = set(claimed) | set(kept)
+ if not agents:
+ return
+ async with tenant_session(self.services.session_factory, tenant_id) as session:
+ rows = (
+ await session.execute(
+ select(Agent.id, ClientRoom.room_id)
+ .outerjoin(ClientRoom, ClientRoom.client_id == Agent.client_id)
+ .where(Agent.id.in_(agents))
+ )
+ ).all()
+ memberships: dict[str, set[str]] = {}
+ for agent_id, room_id in rows:
+ rooms = memberships.setdefault(agent_id, set())
+ if room_id is not None:
+ rooms.add(room_id)
+ # Only the agents of this tenant come back; the rest are another
+ # tenant's to check.
+ for agent_id, rooms in memberships.items():
+ for room_id in sorted(claimed.get(agent_id, set()) - rooms):
+ await self._release_claims(tenant_id, agent_id, room_id)
+ if agent_id in kept:
+ for room_id in kept[agent_id] - rooms:
+ controllers.room_left(agent_id, room_id)
+ for room_id in rooms - controllers.rooms(agent_id):
+ controllers.room_joined(agent_id, room_id)
+
+ async def release_room(self, tenant_id: str, agent_id: str, room_id: str) -> None:
+ """Take a room the agent is no longer in off a controller streaming
+ the agent from here, and off its connections, whichever process holds
+ them. The controller first: it is this process's memory and cannot
+ fail, and the claims are a write that can."""
+ self.services.connections.controllers.room_left(agent_id, room_id)
+ await self._release_claims(tenant_id, agent_id, room_id)
+
+ async def _release_claims(
+ self, tenant_id: str, agent_id: str, room_id: str
+ ) -> None:
+ """Take the room off every connection of the agent's that claims it.
+
+ A claim outlives the membership it was checked against
+ (`require_room_member` runs when the room is claimed and never again).
+ Claims are recorded for every process, so this is a recorded move
+ (`AgentConnectionRegistry.recorded`): the agent's claims are read
+ again from the rows, the room is taken off whichever connection holds
+ it, here or in another process, and the change is written and
+ signalled to the process holding that connection.
+ """
+ connections = self.services.connections
+ with tenant_scope(tenant_id):
+ async with (
+ connections.slots(agent_id),
+ connections.recorded(agent_id),
+ ):
+ connections.release_room_everywhere(agent_id, room_id)
+
+ async def remove(
+ self, session: AsyncSession, *, tenant_id: str, client_id: str, room_id: str
+ ) -> Removed:
+ """Forget an agent's own events in a room it is being taken out of,
+ in the caller's transaction, the one that removes the membership.
+ Nothing to do for a client that is not an agent.
+
+ The caller commits, then calls `Removed.committed()`.
+ """
+ agent_id = await session.scalar(
+ select(Agent.id).where(Agent.client_id == client_id)
+ )
+ if agent_id is not None:
+ await self.services.event_buffer.forget_room(
+ session, agent_id=agent_id, room_id=room_id
+ )
+ return Removed(self, tenant_id=tenant_id, agent_id=agent_id, room_id=room_id)
+
+ # ── Deciding ──────────────────────────────────────────────────────────────
+
+ async def _decide(
+ self,
+ session: AsyncSession,
+ reads: RoomReads,
+ event: InboundEvent,
+ *,
+ sender_client_id: str | None,
+ ) -> _Plan:
+ # Nobody's own event is the row they wrote themselves.
+ recipients = [
+ self._recipient(reads, member)
+ for member in reads.members
+ if member.client_id != sender_client_id
+ ]
+ if isinstance(event, InboundMembership):
+ return await self._decide_arrival(
+ session, reads, recipients, event, sender_client_id
+ )
+ if isinstance(event, InboundMedia):
+ return await self._decide_file(session, reads, recipients, event)
+ if isinstance(event, InboundCustomEvent):
+ return await self._decide_custom(recipients, event)
+ assert isinstance(event, InboundMessage)
+ return await self._decide_message(session, recipients, event)
+
+ async def _decide_message(
+ self,
+ session: AsyncSession,
+ recipients: list[AgentRecipient],
+ event: InboundMessage,
+ ) -> _Plan:
+ deliveries = [
+ (
+ recipient,
+ await _decided(
+ recipient, event, lambda: recipient.decide_message(session, event)
+ ),
+ )
+ for recipient in recipients
+ ]
+ if not any(delivery.named for _, delivery in deliveries):
+ # No rule named any agent in the room (no name, alias, held role
+ # or direct room), so the message is the room's conversation:
+ # every agent that reads the room reads it from `messages`, and
+ # none is woken for it. This is the one place a message is found
+ # to address no agent by any explicit rule.
+ return _Plan()
+ return _Plan(deliveries=deliveries)
+
+ async def _decide_file(
+ self,
+ session: AsyncSession,
+ reads: RoomReads,
+ recipients: list[AgentRecipient],
+ event: InboundMedia,
+ ) -> _Plan:
+ group = parse_attachment_group(event.content)
+ if group is None:
+ files = [event]
+ key = event.event_id
+ step = None
+ else:
+ group_id, _, total = group
+ pending = await session.get(
+ PendingAttachmentGroup,
+ {"room_id": reads.room_id, "group_id": group_id},
+ )
+ earlier = (
+ []
+ if pending is None
+ else (await self._files_of(session, reads, group_id, pending.from_seq))[
+ 0
+ ]
+ )
+ files = [*earlier, event]
+ complete = len({file_index(part) for part in files}) >= total
+ step = _GroupStep(
+ group_id=group_id,
+ total=total,
+ pending_from=pending.from_seq if pending is not None else None,
+ complete=complete,
+ )
+ if not complete:
+ return _Plan(group=step)
+ key = group_id
+ return _Plan(
+ deliveries=await self._decide_post(
+ session, recipients, files, lead=event, complete=True, key=key
+ ),
+ group=step,
+ )
+
+ @staticmethod
+ async def _decide_post(
+ session: AsyncSession,
+ recipients: list[AgentRecipient],
+ files: list[InboundMedia],
+ *,
+ lead: InboundMedia,
+ complete: bool,
+ key: str,
+ ) -> list[tuple[AgentRecipient, Delivery]]:
+ deliveries = []
+ for recipient in recipients:
+
+ async def decide(recipient: AgentRecipient = recipient) -> Delivery:
+ named = False
+ for part in files:
+ if await recipient.addressed_by(session, part):
+ named = True
+ break
+ return await recipient.decide_post(
+ session,
+ files,
+ named=named,
+ lead=lead,
+ complete=complete,
+ message_key=key,
+ )
+
+ deliveries.append((recipient, await _decided(recipient, lead, decide)))
+ return deliveries
+
+ async def _decide_arrival(
+ self,
+ session: AsyncSession,
+ reads: RoomReads,
+ recipients: list[AgentRecipient],
+ event: InboundMembership,
+ sender_client_id: str | None,
+ ) -> _Plan:
+ plan = _Plan(
+ deliveries=[
+ (
+ recipient,
+ await _decided(
+ recipient,
+ event,
+ lambda: recipient.decide_arrival(session, event),
+ ),
+ )
+ for recipient in recipients
+ ]
+ )
+ joining = reads.member(sender_client_id)
+ if (
+ joining is not None
+ and event.membership == "join"
+ and event.prev_membership != "join"
+ ):
+ plan.arrival = self._recipient(reads, joining)
+ return plan
+
+ async def _decide_custom(
+ self, recipients: list[AgentRecipient], event: InboundCustomEvent
+ ) -> _Plan:
+ if event.event_type == _COMMAND_EVENT:
+ try:
+ command = command_event(event)
+ except Exception:
+ logger.exception(
+ "Failed to parse %s event in %s", event.event_type, event.room_id
+ )
+ return _Plan()
+ # A command that is queued for the agent is decided here, so it
+ # is numbered for the agent before anything written after it;
+ # the rest are decided by their handlers once they have committed.
+ return _Plan(
+ deliveries=[
+ (
+ recipient,
+ await _decided(
+ recipient,
+ event,
+ lambda: recipient.decide_command(command),
+ ),
+ )
+ for recipient in recipients
+ ]
+ )
+ kind = _TASK_EVENTS.get(event.event_type)
+ if kind is None:
+ return _Plan()
+ try:
+ typed = kind(**event.content)
+ except Exception:
+ logger.exception(
+ "Failed to parse %s event in %s", event.event_type, event.room_id
+ )
+ return _Plan()
+ return _Plan(
+ deliveries=[
+ (recipient, await _decided(recipient, event, _task(recipient, typed)))
+ for recipient in recipients
+ ]
+ )
+
+ def _recipient(self, reads: RoomReads, member: RoomMember) -> AgentRecipient:
+ return AgentRecipient(
+ services=self.services,
+ member=member,
+ tenant_id=reads.tenant_id,
+ meta=reads.meta,
+ transport_room_id=reads.transport_room_id,
+ actor=self._poster(reads.tenant_id, member),
+ addressing=reads.addressing,
+ budget_guard=reads.budget_guard,
+ clients=reads,
+ cloud_waking_notices=self._cloud_waking.setdefault(member.agent.id, set()),
+ )
+
+ def _poster(self, tenant_id: str, member: RoomMember) -> AgentActor:
+ """The identity the agent's replies are posted under. Connected only
+ when a reply is posted (`_AfterCommit`), so deciding a row needs no
+ transport at all."""
+ actor = self._posters.get(member.client_id)
+ if actor is None:
+ actor = AgentActor(
+ client_id=member.client_id,
+ tenant_id=tenant_id,
+ transport_user_id=member.transport_user_id,
+ display_name=member.display_name,
+ session_factory=self.services.session_factory,
+ client_store=self._client_store,
+ config=ClientConfig(),
+ transport_factory=self._transport_for,
+ )
+ self._posters[member.client_id] = actor
+ actor.set_agent(member.agent)
+ actor.display_name = member.display_name
+ return actor
+
+ def _transport_for(self, actor: Actor[ClientConfig]) -> MessageTransport:
+ if self._transports is None:
+ raise RuntimeError(
+ "The router has no transports to post agents' replies "
+ "through; the client factory built with it gives them"
+ )
+ return self._transports(actor)
+
+ # ── Writing the decisions ─────────────────────────────────────────────────
+
+ async def _store_own_events(
+ self,
+ session: AsyncSession,
+ reads: RoomReads,
+ deliveries: list[tuple[AgentRecipient, Delivery]],
+ row: InboundEvent,
+ ) -> list[str]:
+ """Store every own event decided for `row`, in the order of the
+ agents' ids, and return the agents whose events were stored."""
+ stored: list[str] = []
+ for recipient, delivery in sorted(
+ deliveries, key=lambda decided: decided[0].agent.id
+ ):
+ if delivery.own_event is None:
+ continue
+ agent_id = recipient.agent.id
+ try:
+ own = delivery.own_event(row)
+ seq = await self.services.event_buffer.store(
+ session,
+ tenant_id=reads.tenant_id,
+ agent_id=agent_id,
+ client_id=recipient.client_id,
+ room_id=reads.room_id,
+ event=own.event,
+ room_seq=own.room_seq,
+ message_key=own.message_key,
+ )
+ except Exception:
+ metrics().increment(ROUTING_FAILURES, {"stage": "store"})
+ logger.error(
+ "Could not store agent %s's own event for row %s in room %s; "
+ "the row is written and the agent misses it",
+ recipient.agent.name,
+ row.event_id,
+ reads.room_id,
+ exc_info=True,
+ )
+ continue
+ if seq is not None:
+ stored.append(agent_id)
+ return stored
+
+ async def _step_group(
+ self, session: AsyncSession, reads: RoomReads, plan: _Plan, row: InboundEvent
+ ) -> float | None:
+ """Bring the post `row` is a file of up to date in
+ `pending_attachment_groups`, inside the room's lock.
+
+ A file that completes its post deletes the pending row, and the post
+ is decided with it. If the row is gone already, the sweep decided the
+ post while this file was being written, and this file starts a new
+ post of its own instead: what has gone out does not go out again. A
+ file that does not complete its post leaves the pending row as it is,
+ or starts one. Returns how long a pending row this started has until
+ its deadline, so the writer can set its timer.
+ """
+ step = plan.group
+ assert step is not None
+ pending = and_(
+ PendingAttachmentGroup.room_id == reads.room_id,
+ PendingAttachmentGroup.group_id == step.group_id,
+ PendingAttachmentGroup.from_seq == step.pending_from,
+ )
+ if step.complete:
+ if step.pending_from is None:
+ return None
+ decided = await session.scalar(
+ delete(PendingAttachmentGroup)
+ .where(pending)
+ .returning(PendingAttachmentGroup.group_id)
+ )
+ if decided is not None:
+ return None
+ plan.deliveries = []
+ return await self._hold_group(session, reads, step, row.seq)
+ if step.pending_from is not None:
+ # Locks the pending row, so a sweep deciding it now is waited for
+ # and its outcome seen.
+ held = await session.scalar(
+ update(PendingAttachmentGroup)
+ .where(pending)
+ .values(total=PendingAttachmentGroup.total)
+ .returning(PendingAttachmentGroup.group_id)
+ )
+ if held is not None:
+ return None
+ return await self._hold_group(session, reads, step, row.seq)
+
+ async def _hold_group(
+ self, session: AsyncSession, reads: RoomReads, step: _GroupStep, from_seq: int
+ ) -> float | None:
+ """Record a post that has files to come, counted from `from_seq`, and
+ return the seconds until its deadline; None when another writer
+ recorded it first."""
+ seconds = await session.scalar(
+ insert(PendingAttachmentGroup)
+ .values(
+ tenant_id=reads.tenant_id,
+ room_id=reads.room_id,
+ group_id=step.group_id,
+ total=step.total,
+ from_seq=from_seq,
+ deadline=func.now()
+ + timedelta(seconds=ATTACHMENT_GROUP_TIMEOUT_SECONDS),
+ )
+ .on_conflict_do_nothing(
+ index_elements=[
+ PendingAttachmentGroup.room_id,
+ PendingAttachmentGroup.group_id,
+ ]
+ )
+ .returning(
+ func.extract("epoch", PendingAttachmentGroup.deadline - func.now())
+ )
+ )
+ return None if seconds is None else float(seconds)
+
+ async def _files_of(
+ self, session: AsyncSession, reads: RoomReads, group_id: str, from_seq: int
+ ) -> tuple[list[InboundMedia], str | None]:
+ """The files of a post written since `from_seq`, in the order they
+ were written, and the client that wrote them."""
+ rows = list(
+ (
+ await session.scalars(
+ select(Message)
+ .where(
+ Message.room_id == reads.room_id,
+ Message.seq >= from_seq,
+ Message.content[ATTACHMENT_GROUP_KEY]["id"].astext == group_id,
+ )
+ .order_by(Message.seq)
+ )
+ ).all()
+ )
+ attachments = await self._messages.attachments_for(
+ session, [row.id for row in rows]
+ )
+ files: list[InboundMedia] = []
+ for row in rows:
+ inbound = to_inbound(
+ row,
+ attachments.get(row.id, []),
+ transport_room_id=reads.transport_room_id,
+ )
+ if isinstance(inbound, InboundMedia):
+ files.append(inbound)
+ return files, rows[0].sender_client_id if rows else None
+
+ # ── Posts given up on ─────────────────────────────────────────────────────
+
+ def flush_groups_after(self, tenant_id: str, delay: float) -> None:
+ """Decide the tenant's posts whose wait is over, `delay` seconds from
+ now: the deadline of a post the caller just started."""
+ loop = asyncio.get_running_loop()
+
+ def due() -> None:
+ self._timers.discard(timer)
+ task = loop.create_task(
+ self._flush_logged(tenant_id), context=contextvars.Context()
+ )
+ self._flushes.add(task)
+ task.add_done_callback(self._flushes.discard)
+
+ timer = loop.call_later(max(delay, 0.0), due)
+ self._timers.add(timer)
+
+ async def _flush_logged(self, tenant_id: str) -> None:
+ try:
+ await self.flush_due_groups(tenant_id)
+ except Exception:
+ metrics().increment(ROUTING_FAILURES, {"stage": "sweep"})
+ logger.error(
+ "Could not decide the multi-file posts of tenant %s whose wait "
+ "is over; the next sweep tries again",
+ tenant_id,
+ exc_info=True,
+ )
+
+ async def flush_due_groups(self, tenant_id: str) -> int:
+ """Decide every post of `tenant_id` whose wait is over with the files
+ that did arrive, and return how many were decided.
+
+ Every process runs this. Each post is claimed with `FOR UPDATE SKIP
+ LOCKED` and decided in the transaction that deletes its pending row,
+ so it is decided once however many processes look at the same moment,
+ and the deadline is read on the database's clock, the one that set it.
+ """
+ decided = 0
+ while (outcome := await self._flush_one(tenant_id)) is not None:
+ decided += outcome
+ return decided
+
+ async def _flush_one(self, tenant_id: str) -> bool | None:
+ """Decide one post whose wait is over: True when it was decided,
+ False when it was put off, None when none is due.
+
+ A post that cannot be decided is put off for another wait and logged,
+ so the posts due after it, and the tenants swept after this one, are
+ not held up behind it, and it is tried again once that wait is over.
+ """
+ async with tenant_session(self.services.session_factory, tenant_id) as session:
+ due = await session.scalar(
+ select(PendingAttachmentGroup)
+ .where(PendingAttachmentGroup.deadline <= func.now())
+ .order_by(PendingAttachmentGroup.deadline)
+ .limit(1)
+ .with_for_update(skip_locked=True)
+ )
+ if due is None:
+ return None
+ room_id, group_id = due.room_id, due.group_id
+ try:
+ routed = await self._decide_due(session, tenant_id, due)
+ except Exception:
+ await session.rollback()
+ metrics().increment(ROUTING_FAILURES, {"stage": "sweep"})
+ logger.error(
+ "Could not decide multi-file post %s in room %s whose wait "
+ "is over; trying again in %ss",
+ group_id,
+ room_id,
+ ATTACHMENT_GROUP_TIMEOUT_SECONDS,
+ exc_info=True,
+ )
+ await session.execute(
+ update(PendingAttachmentGroup)
+ .where(
+ PendingAttachmentGroup.room_id == room_id,
+ PendingAttachmentGroup.group_id == group_id,
+ )
+ .values(
+ deadline=func.now()
+ + timedelta(seconds=ATTACHMENT_GROUP_TIMEOUT_SECONDS)
+ )
+ )
+ await session.commit()
+ return False
+ if routed is not None:
+ routed.committed()
+ return True
+
+ async def _decide_due(
+ self, session: AsyncSession, tenant_id: str, due: PendingAttachmentGroup
+ ) -> Routed | None:
+ """Decide the post `due` with the files that did arrive, delete its
+ pending row and commit; None when there was nothing left to decide."""
+ room_id, group_id = due.room_id, due.group_id
+ total, from_seq = due.total, due.from_seq
+ await session.delete(due)
+ reads = await RoomReads.load(
+ session,
+ services=self.services,
+ client_store=self._client_store,
+ budget_store=self._budget_store,
+ tenant_id=tenant_id,
+ room_id=room_id,
+ )
+ files: list[InboundMedia] = []
+ sender: str | None = None
+ if reads is not None:
+ files, sender = await self._files_of(session, reads, group_id, from_seq)
+ if reads is None or not files:
+ await session.commit()
+ return None
+ lead = files[0]
+ present = len({file_index(part) for part in files})
+ complete = present >= total
+ if not complete:
+ logger.error(
+ "Attachment group %s incomplete: %d of %d parts arrived within "
+ "%ss; delivering what arrived",
+ group_id,
+ present,
+ total,
+ ATTACHMENT_GROUP_TIMEOUT_SECONDS,
+ )
+ # Members from before the post began, as a member who joined after
+ # its first file would not have been handed that file.
+ began = lead.timestamp
+ recipients = [
+ self._recipient(reads, member)
+ for member in reads.members
+ if member.client_id != sender
+ and int(member.joined_at.timestamp() * 1000) <= began
+ ]
+ plan = _Plan(
+ deliveries=await self._decide_post(
+ session,
+ recipients,
+ files,
+ lead=lead,
+ complete=complete,
+ key=group_id,
+ )
+ )
+ stored = await self._store_own_events(session, reads, plan.deliveries, lead)
+ await session.commit()
+ return Routed(
+ self, reads, plan, lead, seq=lead.seq, stored=stored, deadline_in=None
+ )
+
+ # ── After the commit ──────────────────────────────────────────────────────
+
+ def after_commit(
+ self,
+ tenant_id: str,
+ room_id: str,
+ work: list[tuple[AgentRecipient, Callable[[], Awaitable[None]]]],
+ ) -> None:
+ self._after.add(tenant_id, room_id, work)
+
+ async def idle(self) -> None:
+ """Wait for the replies already owed, and posts already being decided,
+ to be done."""
+ while True:
+ await self._after.idle()
+ if not self._flushes:
+ return
+ await asyncio.gather(*list(self._flushes), return_exceptions=True)
+
+ def close(self) -> None:
+ """Stop the timers set for posts' deadlines. The sweep in every
+ process decides those posts."""
+ for timer in self._timers:
+ timer.cancel()
+ self._timers.clear()
+
+
+async def _decided(
+ recipient: AgentRecipient,
+ event: InboundEvent,
+ decide: Callable[[], Awaitable[Delivery]],
+) -> Delivery:
+ """What `decide` decides for one agent, or nothing for it when deciding
+ raises: a rule that cannot be applied to one agent must not fail the
+ write of a row every other member is owed, nor block every later row
+ that names that agent. A database error is raised all the same: the
+ transaction it broke cannot write the row, and the writer tries again.
+ """
+ try:
+ return await decide()
+ except DBAPIError:
+ raise
+ except Exception:
+ metrics().increment(ROUTING_FAILURES, {"stage": "decide"})
+ logger.error(
+ "Could not decide row %s in room %s for agent %s; the row is "
+ "written and the agent misses it",
+ event.event_id,
+ event.room_id,
+ recipient.agent.name,
+ exc_info=True,
+ )
+ return NOTHING
+
+
+def _task(
+ recipient: AgentRecipient, event: SwitchEvent
+) -> Callable[[], Awaitable[Delivery]]:
+ async def decide() -> Delivery:
+ return recipient.decide_task(event)
+
+ return decide
+
+
+def _unwritten(message: Message) -> _Unwritten:
+ return _Unwritten(
+ content=dict(message.content),
+ transport_event_id=message.transport_event_id,
+ sender_id=message.sender_id,
+ sent_at=_NOT_STAMPED,
+ event_type=message.event_type,
+ seq=0,
+ body=message.body,
+ sender_name=message.sender_name,
+ formatted_body=message.formatted_body,
+ msgtype=message.msgtype,
+ thread_root_event_id=message.thread_root_event_id,
+ )
+
+
+# The time on a row not written yet, which nothing deciding it reads.
+_NOT_STAMPED = datetime.fromtimestamp(0, UTC)
diff --git a/core/switch_core/events.py b/core/switch_core/events.py
index fc652c7e8..14c39d59f 100644
--- a/core/switch_core/events.py
+++ b/core/switch_core/events.py
@@ -7,9 +7,10 @@ class SwitchEvent(BaseModel):
model_config = ConfigDict(extra="allow")
# The `messages` row this event was parsed from: its `seq` in the room and
- # its event id. `Consumer` fills both when it dispatches the event, and
- # `AgentConsumer` stores them on the row in the agent's own events so a
- # reader can match that row to the room row. They are not part of the
+ # its event id. Whoever parses the event off its row fills both
+ # (`Consumer` as it dispatches the event, the router as it decides it),
+ # and the agent's own event is stored with them so a reader can match it
+ # to the room row. They are not part of the
# event's content, and stay 0 and None for an event that did not come from
# a `messages` row.
room_seq: int = 0
diff --git a/core/switch_core/gateway/app.py b/core/switch_core/gateway/app.py
index 84b46efcd..8fcda59a8 100644
--- a/core/switch_core/gateway/app.py
+++ b/core/switch_core/gateway/app.py
@@ -3,6 +3,7 @@ from __future__ import annotations
from pathlib import Path
from fastapi import FastAPI
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from starlette.middleware.sessions import SessionMiddleware
from switch_core.bridges.agent.protocol.agent_core import AgentCore
@@ -22,6 +23,7 @@ from switch_core.clients.client_lifecycle_service import ClientLifecycleService
from switch_core.config import SwitchConfig
from switch_core.connections.adapters.github import load_github_app
from switch_core.connections.flows import ServiceFlows
+from switch_core.connections.sign_in_flows import SignInFlows
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.api_key_store import ApiKeyStore
from switch_core.db.stores.budget_store import BudgetStore
@@ -91,7 +93,7 @@ def create_gateway_app(
connector_lifecycle: ServerSideConnectorLifecycleService,
connector_store: ServerConnectorStore,
event_buffer: EventBuffer,
- session_factory: object,
+ session_factory: async_sessionmaker[AsyncSession],
user_store: UserStore,
external_user_store: ExternalUserStore,
api_key_store: ApiKeyStore,
@@ -159,7 +161,12 @@ def create_gateway_app(
)
app.include_router(connection_catalog_router, tags=["provider-connections"])
app.include_router(providers_router, tags=["agent providers"])
- app.state.service_flows = ServiceFlows(config.gateway_public_url)
+ # Sign-ins under way, GitHub's and the catalog's, in Postgres so that any
+ # switch-core process can take any step of one.
+ app.state.sign_in_flows = SignInFlows(session_factory, config.keyring)
+ app.state.service_flows = ServiceFlows(
+ config.gateway_public_url, app.state.sign_in_flows
+ )
if config.service_connections_enabled:
app.include_router(service_connections_router, tags=["service-connections"])
app.include_router(service_flows_router, tags=["service-connections"])
diff --git a/core/switch_core/gateway/connectors.py b/core/switch_core/gateway/connectors.py
index a32b6b587..694597158 100644
--- a/core/switch_core/gateway/connectors.py
+++ b/core/switch_core/gateway/connectors.py
@@ -69,7 +69,7 @@ async def create_connector(
connector_type=record.type,
display_name=record.display_name,
status=record.status,
- agent_names=connector_lifecycle.get_agent_names(record.id),
+ agent_names=await connector_lifecycle.get_agent_names(record.id),
created_at=str(record.created_at),
)
@@ -90,7 +90,7 @@ async def list_connectors(
connector_type=r.type,
display_name=r.display_name,
status=r.status,
- agent_names=connector_lifecycle.get_agent_names(r.id),
+ agent_names=await connector_lifecycle.get_agent_names(r.id),
created_at=str(r.created_at),
)
for r in records
@@ -116,7 +116,7 @@ async def get_connector(
connector_type=record.type,
display_name=record.display_name,
status=record.status,
- agent_names=connector_lifecycle.get_agent_names(record.id),
+ agent_names=await connector_lifecycle.get_agent_names(record.id),
created_at=str(record.created_at),
)
diff --git a/core/switch_core/gateway/github_connections.py b/core/switch_core/gateway/github_connections.py
index e2d525115..e69eb9fc3 100644
--- a/core/switch_core/gateway/github_connections.py
+++ b/core/switch_core/gateway/github_connections.py
@@ -1,3 +1,4 @@
+import asyncio
import logging
import secrets
from html import escape
@@ -18,6 +19,14 @@ from switch_core.connections.broker import (
ServiceError,
get_service_broker,
)
+from switch_core.connections.sign_in_flows import (
+ FlowGoneError,
+ FlowsBusyError,
+ FlowUsedError,
+ SignInFlows,
+ StoredFlow,
+ digest,
+)
from switch_core.db.models import (
ServiceConnection,
Tenant,
@@ -32,7 +41,6 @@ from switch_core.providers.github import (
GitHubAuthorizationError,
GitHubConnections,
GitHubError,
- GitHubFlow,
)
logger = logging.getLogger(__name__)
@@ -41,6 +49,10 @@ router = APIRouter(prefix="/provider-connections/github")
STORE = ServiceConnectionStore()
+KIND = "github"
+INTERRUPTED = "Sign-in was interrupted. Start it again from Switch Console."
+
+
def service(request: Request) -> GitHubConnections:
value = request.app.state.github_connections
if value is None:
@@ -48,16 +60,34 @@ def service(request: Request) -> GitHubConnections:
return cast(GitHubConnections, value)
-def owned_flow(github: GitHubConnections, flow_id: str, user_id: str) -> GitHubFlow:
+def get_sign_in_flows(request: Request) -> SignInFlows:
+ flows = getattr(request.app.state, "sign_in_flows", None)
+ if not isinstance(flows, SignInFlows):
+ raise RuntimeError("No sign-in flow store is installed on this app.")
+ return flows
+
+
+async def owned_flow(flows: SignInFlows, flow_id: str, user_id: str) -> StoredFlow:
try:
- flow = github.flow(flow_id)
- except GitHubError as error:
- raise HTTPException(410, str(error)) from None
- if (flow.tenant_id, flow.user_id) != (require_tenant_id(), user_id):
+ flow = await flows.get(flow_id, require_tenant_id())
+ except FlowGoneError:
+ raise HTTPException(410, INTERRUPTED) from None
+ if flow.kind != KIND or flow.user_id != user_id:
raise HTTPException(404, "Authorization not found.")
return flow
+async def browser_flow(flows: SignInFlows, state: str) -> StoredFlow:
+ """The flow a browser step names by its state, whichever tenant holds it."""
+ try:
+ flow = await flows.find(state)
+ except FlowGoneError:
+ raise GitHubError(INTERRUPTED) from None
+ if flow.kind != KIND:
+ raise GitHubError(INTERRUPTED)
+ return flow
+
+
def page(message: str, status: int) -> HTMLResponse:
return HTMLResponse(
"<!doctype html><html lang=en><meta charset=utf-8><meta name=viewport content='width=device-width'>"
@@ -95,47 +125,67 @@ async def start(
session: Annotated[AsyncSession, Depends(get_session)],
user: Annotated[User, Depends(get_current_user)],
github: Annotated[GitHubConnections, Depends(service)],
+ flows: Annotated[SignInFlows, Depends(get_sign_in_flows)],
) -> dict:
+ tenant = await session.get(Tenant, require_tenant_id())
+ if tenant is None:
+ raise HTTPException(404, "Workspace not found.")
try:
- tenant = await session.get(Tenant, require_tenant_id())
- if tenant is None:
- raise HTTPException(404, "Workspace not found.")
- flow_id = github.start(
- require_tenant_id(),
- user.id,
- body.port,
- body.completion_secret,
- body.state,
- f"{user.name} ({user.email}) in {tenant.name}",
+ await flows.start(
+ kind=KIND,
+ service="github",
+ tenant_id=require_tenant_id(),
+ user_id=user.id,
+ state=body.state,
+ mode="core",
+ port=body.port,
+ redirect_uri=github.callback,
+ owner_label=f"{user.name} ({user.email}) in {tenant.name}",
+ details={},
+ completion_secret=body.completion_secret,
)
- except GitHubError as error:
- raise HTTPException(503, str(error)) from None
+ except FlowUsedError:
+ raise HTTPException(
+ 503, "This authorization has already been used. Connect again."
+ ) from None
+ except FlowsBusyError:
+ raise HTTPException(
+ 503, "GitHub connection is busy. Please try again shortly."
+ ) from None
return {
- "id": flow_id,
+ "id": body.state,
"url": github.origin
+ "/gateway/provider-connections/github/authorize?state="
- + flow_id,
+ + body.state,
}
@router.get("/authorize")
async def authorize(
- state: str, github: Annotated[GitHubConnections, Depends(service)]
+ state: str,
+ github: Annotated[GitHubConnections, Depends(service)],
+ flows: Annotated[SignInFlows, Depends(get_sign_in_flows)],
) -> Response:
+ nonce = secrets.token_urlsafe(32)
try:
- url = github.authorize_url(state)
- flow = github.flow(state)
- except GitHubError:
- return page(
- "Sign-in was interrupted. Start it again from Switch Console.",
- 400,
+ flow = await browser_flow(flows, state)
+ taken = await flows.advance(
+ flow,
+ expect=("pending",),
+ to="authorizing",
+ changes={"browser_nonce_hash": digest(nonce)},
)
+ if taken is None:
+ raise GitHubError("This authorization has already been used.")
+ except GitHubError:
+ return page(INTERRUPTED, 400)
response = RedirectResponse(
- url, headers={"Cache-Control": "no-store", "Referrer-Policy": "no-referrer"}
+ github.authorize_url(state, flow.verifier),
+ headers={"Cache-Control": "no-store", "Referrer-Policy": "no-referrer"},
)
response.set_cookie(
"switch_github_" + state,
- flow.browser_nonce,
+ nonce,
max_age=600,
secure=True,
httponly=True,
@@ -147,24 +197,35 @@ async def authorize(
@router.get("/callback")
async def callback(
- request: Request, github: Annotated[GitHubConnections, Depends(service)]
+ request: Request,
+ github: Annotated[GitHubConnections, Depends(service)],
+ flows: Annotated[SignInFlows, Depends(get_sign_in_flows)],
) -> Response:
state = request.query_params.get("state", "")
code = request.query_params.get("code", "")
try:
- flow = github.flow(state)
+ flow = await browser_flow(flows, state)
if (
flow.status not in {"authorizing", "returning"}
or len(code) > 512
- or not secrets.compare_digest(
- flow.browser_nonce, request.cookies.get("switch_github_" + state, "")
- )
+ or not flow.holds_nonce(request.cookies.get("switch_github_" + state, ""))
):
raise GitHubError("Authorization could not be verified.")
if not code:
- flow.status = "failed"
+ await flows.advance(
+ flow,
+ expect=("authorizing", "returning"),
+ to="failed",
+ changes={"error": "GitHub authorization was cancelled."},
+ )
raise GitHubError("GitHub authorization was cancelled.")
- flow.status = "returning"
+ if (
+ await flows.advance(
+ flow, expect=("authorizing", "returning"), to="returning", changes={}
+ )
+ is None
+ ):
+ raise GitHubError("Authorization could not be verified.")
response = page(
f"Linking GitHub to <strong>{escape(flow.owner_label)}</strong>. "
"Continue only if this is your Switch account. Use the same computer where you started the connection."
@@ -180,15 +241,14 @@ async def callback(
)
return response
except GitHubError:
- return page(
- "Sign-in was interrupted. Start it again from Switch Console.",
- 400,
- )
+ return page(INTERRUPTED, 400)
@router.post("/callback")
async def relay(
- request: Request, github: Annotated[GitHubConnections, Depends(service)]
+ request: Request,
+ github: Annotated[GitHubConnections, Depends(service)],
+ flows: Annotated[SignInFlows, Depends(get_sign_in_flows)],
) -> Response:
body = bytearray()
async for chunk in request.stream():
@@ -199,17 +259,19 @@ async def relay(
state = value.get("state", [""])[0]
code = value.get("code", [""])[0]
try:
- flow = github.flow(state)
+ flow = await browser_flow(flows, state)
if (
not code
or len(code) > 512
or flow.status != "returning"
- or not secrets.compare_digest(
- flow.browser_nonce, request.cookies.get("switch_github_" + state, "")
- )
+ or not flow.holds_nonce(request.cookies.get("switch_github_" + state, ""))
+ ):
+ raise GitHubError("Authorization could not be verified.")
+ if (
+ await flows.advance(flow, expect=("returning",), to="delivered", changes={})
+ is None
):
raise GitHubError("Authorization could not be verified.")
- flow.status = "delivered"
response = RedirectResponse(
f"http://127.0.0.1:{flow.port}/switch-github/callback?"
+ urlencode({"code": code, "state": state}),
@@ -225,7 +287,7 @@ async def relay(
)
return response
except GitHubError:
- return page("Sign-in was interrupted. Start it again from Switch Console.", 400)
+ return page(INTERRUPTED, 400)
@router.post("/flows/{flow_id}/complete", status_code=204)
@@ -234,12 +296,41 @@ async def complete(
body: CompleteFlow,
user: Annotated[User, Depends(get_current_user)],
github: Annotated[GitHubConnections, Depends(service)],
+ flows: Annotated[SignInFlows, Depends(get_sign_in_flows)],
) -> Response:
- owned_flow(github, flow_id, user.id)
+ flow = await owned_flow(flows, flow_id, user.id)
+ refused = "GitHub authorization could not be verified. Connect again."
+ if not flow.holds_secret(body.completion_secret) or flow.status != "delivered":
+ raise HTTPException(400, refused)
+ checking = await flows.advance(
+ flow, expect=("delivered",), to="checking", changes={}
+ )
+ if checking is None:
+ raise HTTPException(400, refused)
try:
- await github.complete(flow_id, body.completion_secret, body.code)
- except GitHubError as error:
- raise HTTPException(400, str(error)) from None
+ credentials = await github.sign_in(body.code, flow.verifier)
+ except (GitHubError, asyncio.CancelledError) as error:
+ await flows.advance(
+ checking,
+ expect=("checking",),
+ to="failed",
+ changes={"error": str(error) or INTERRUPTED},
+ )
+ if isinstance(error, GitHubError):
+ raise HTTPException(400, str(error)) from None
+ raise
+ ready = await flows.advance(
+ checking,
+ expect=("checking",),
+ to="ready",
+ changes={
+ "encrypted_result": flows.seal(credentials),
+ "account_label": credentials["login"],
+ },
+ )
+ if ready is None:
+ # The flow expired or was cancelled while the code was exchanged.
+ raise HTTPException(410, INTERRUPTED)
return Response(status_code=204)
@@ -248,13 +339,14 @@ async def flow_status(
flow_id: str,
user: Annotated[User, Depends(get_current_user)],
github: Annotated[GitHubConnections, Depends(service)],
+ flows: Annotated[SignInFlows, Depends(get_sign_in_flows)],
) -> dict:
- flow = owned_flow(github, flow_id, user.id)
+ flow = await owned_flow(flows, flow_id, user.id)
return {
"status": flow.status
if flow.status in {"checking", "ready", "failed"}
else "pending",
- "login": flow.login,
+ "login": flow.account_label or "",
}
@@ -263,9 +355,9 @@ async def cancel(
flow_id: str,
user: Annotated[User, Depends(get_current_user)],
github: Annotated[GitHubConnections, Depends(service)],
+ flows: Annotated[SignInFlows, Depends(get_sign_in_flows)],
) -> Response:
- owned_flow(github, flow_id, user.id)
- del github.flows[flow_id]
+ await flows.discard(await owned_flow(flows, flow_id, user.id))
return Response(status_code=204)
@@ -277,17 +369,19 @@ async def confirm(
session: Annotated[AsyncSession, Depends(get_session)],
broker: Annotated[ServiceBroker, Depends(get_service_broker)],
github: Annotated[GitHubConnections, Depends(service)],
+ flows: Annotated[SignInFlows, Depends(get_sign_in_flows)],
) -> Response:
- flow = owned_flow(github, flow_id, user.id)
- if not secrets.compare_digest(flow.completion_secret, body.completion_secret):
+ flow = await owned_flow(flows, flow_id, user.id)
+ if not flow.holds_secret(body.completion_secret):
raise HTTPException(404, "Authorization not found.")
- if flow.status != "ready":
+ if flow.status != "ready" or flow.result is None:
raise HTTPException(409, "Finish GitHub authorization before confirming.")
+ credentials = dict(flow.result)
await session.execute(
text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
{"key": f"github-link:{require_tenant_id()}"},
)
- account_id = str(flow.credentials["user_id"])
+ account_id = str(credentials["user_id"])
taken = await session.scalar(
select(ServiceConnection.user_id).where(
ServiceConnection.tenant_id == require_tenant_id(),
@@ -313,12 +407,12 @@ async def confirm(
consent="write",
granted_scopes=[],
account_id=account_id,
- external_identity=flow.credentials["login"],
- secret=ConnectionSecret(dict(flow.credentials)),
+ external_identity=credentials["login"],
+ secret=ConnectionSecret(credentials),
)
except ServiceError as error:
return service_refusal(error)
- github.flows.pop(flow_id, None)
+ await flows.discard(flow)
logger.info(
"GitHub account linked: tenant=%s user=%s github_user=%s",
require_tenant_id(),
@@ -379,6 +473,7 @@ async def disconnect(
session: Annotated[AsyncSession, Depends(get_session)],
broker: Annotated[ServiceBroker, Depends(get_service_broker)],
github: Annotated[GitHubConnections, Depends(service)],
+ flows: Annotated[SignInFlows, Depends(get_sign_in_flows)],
) -> Response:
"""Disconnect GitHub: its grants go, and what was issued is revoked."""
try:
@@ -387,7 +482,6 @@ async def disconnect(
if error.code != CONNECTOR_NOT_CONNECTED:
return service_refusal(error)
warning = None
- for key, flow in list(github.flows.items()):
- if (flow.tenant_id, flow.user_id) == (require_tenant_id(), user.id):
- del github.flows[key]
+ await flows.discard_for_user(session, user.id, KIND)
+ await session.commit()
return JSONResponse({"warning": warning})
diff --git a/core/switch_core/gateway/service_flows.py b/core/switch_core/gateway/service_flows.py
index 34ebdd482..fa694ce2b 100644
--- a/core/switch_core/gateway/service_flows.py
+++ b/core/switch_core/gateway/service_flows.py
@@ -82,9 +82,11 @@ def _page(name: str, message: str, status: int) -> HTMLResponse:
)
-def _owned(flows: ServiceFlows, service: str, flow_id: str, user: User) -> ServiceFlow:
+async def _owned(
+ flows: ServiceFlows, service: str, flow_id: str, user: User
+) -> ServiceFlow:
try:
- flow = flows.owned(flow_id, require_tenant_id(), user.id)
+ flow = await flows.owned(flow_id, require_tenant_id(), user.id)
except FlowError as error:
raise HTTPException(410, str(error)) from None
if flow.service != service:
@@ -126,7 +128,7 @@ async def start(
if tenant is None:
raise HTTPException(404, "Workspace not found.")
try:
- flow = flows.start(
+ flow = await flows.start(
definition=definition,
tenant_id=require_tenant_id(),
user_id=user.id,
@@ -141,7 +143,7 @@ async def start(
url = flows.authorize_url(service, body.state)
else:
try:
- url = await flows.vendor_url(body.state, adapter)
+ url, _ = await flows.vendor_url(flow, adapter)
except ServiceUnavailableError as error:
raise HTTPException(503, str(error)) from None
except ServiceAdapterError as error:
@@ -162,10 +164,10 @@ async def authorize(
except ServiceError:
return _page("Connections", INTERRUPTED, 400)
try:
- flow = flows.flow(state)
+ flow = await flows.flow(state)
if flow.service != service or flow.mode != "core":
raise FlowError(INTERRUPTED)
- url = await flows.vendor_url(state, adapter)
+ url, nonce = await flows.vendor_url(flow, adapter)
except (FlowError, ServiceAdapterError):
return _page(definition.name, INTERRUPTED, 400)
response = RedirectResponse(
@@ -173,7 +175,7 @@ async def authorize(
)
response.set_cookie(
_cookie(state),
- flow.browser_nonce,
+ nonce,
max_age=600,
secure=True,
httponly=True,
@@ -193,10 +195,10 @@ async def callback(
state = request.query_params.get("state", "")
code = request.query_params.get("code", "")
try:
- flow = flows.flow(state)
+ flow = await flows.flow(state)
if flow.service != service:
raise FlowError(INTERRUPTED)
- flows.returning(state, code, request.cookies.get(_cookie(state), ""))
+ await flows.returning(flow, code, request.cookies.get(_cookie(state), ""))
except FlowError:
return _page("Connections", INTERRUPTED, 400)
response = _page(
@@ -233,10 +235,12 @@ async def relay(
state = value.get("state", [""])[0]
code = value.get("code", [""])[0]
try:
- flow = flows.flow(state)
+ flow = await flows.flow(state)
if flow.service != service:
raise FlowError(INTERRUPTED)
- target = flows.deliver(state, code, request.cookies.get(_cookie(state), ""))
+ target = await flows.deliver(
+ flow, code, request.cookies.get(_cookie(state), "")
+ )
except FlowError:
return _page("Connections", INTERRUPTED, 400)
response = RedirectResponse(
@@ -264,14 +268,14 @@ async def complete(
flows: Annotated[ServiceFlows, Depends(get_service_flows)],
) -> Response:
"""Console hands back the code its listener received, with its secret."""
- _owned(flows, service, flow_id, user)
+ flow = await _owned(flows, service, flow_id, user)
try:
_, adapter = broker.sign_in(service)
except ServiceError as error:
return service_refusal(error)
try:
await flows.complete(
- flow_id,
+ flow,
completion_secret=body.completion_secret,
code=body.code,
adapter=adapter,
@@ -290,7 +294,7 @@ async def flow_status(
user: Annotated[User, Depends(get_current_user)],
flows: Annotated[ServiceFlows, Depends(get_service_flows)],
) -> dict:
- flow = _owned(flows, service, flow_id, user)
+ flow = await _owned(flows, service, flow_id, user)
return {
"status": (
flow.status if flow.status in ("checking", "ready", "failed") else "pending"
@@ -307,8 +311,7 @@ async def cancel(
user: Annotated[User, Depends(get_current_user)],
flows: Annotated[ServiceFlows, Depends(get_service_flows)],
) -> Response:
- _owned(flows, service, flow_id, user)
- flows.flows.pop(flow_id, None)
+ await flows.discard(await _owned(flows, service, flow_id, user))
return Response(status_code=204)
@@ -323,13 +326,14 @@ async def confirm(
flows: Annotated[ServiceFlows, Depends(get_service_flows)],
) -> dict | JSONResponse:
"""The person confirms the account: the connection is written."""
- _owned(flows, service, flow_id, user)
+ flow = await _owned(flows, service, flow_id, user)
try:
- flow = flows.ready(flow_id, body.completion_secret)
+ flows.ready(flow, body.completion_secret)
consent = flows.consent(flow)
except FlowError as error:
raise HTTPException(409, str(error)) from None
- assert flow.account is not None and flow.signed_in is not None
+ account, signed_in = flow.account, flow.signed_in
+ assert account is not None and signed_in is not None
await session.execute(
text("SELECT pg_advisory_xact_lock(hashtextextended(:key, 0))"),
{"key": f"service-link:{require_tenant_id()}:{service}"},
@@ -339,7 +343,7 @@ async def confirm(
ServiceConnection.tenant_id == require_tenant_id(),
ServiceConnection.service == service,
ServiceConnection.user_id != user.id,
- ServiceConnection.account_id == flow.account.account_id,
+ ServiceConnection.account_id == account.account_id,
)
)
if taken is not None:
@@ -360,14 +364,14 @@ async def confirm(
user_id=user.id,
service=service,
consent=consent,
- granted_scopes=flow.signed_in.granted_scopes,
- account_id=flow.account.account_id,
- external_identity=flow.account.label,
- secret=flow.signed_in.secret,
+ granted_scopes=signed_in.granted_scopes,
+ account_id=account.account_id,
+ external_identity=account.label,
+ secret=signed_in.secret,
)
except ServiceError as error:
return service_refusal(error)
- flows.flows.pop(flow_id, None)
+ await flows.discard(flow)
logger.info(
"Service account linked: tenant=%s user=%s service=%s",
require_tenant_id(),
diff --git a/core/switch_core/gateway/tenants.py b/core/switch_core/gateway/tenants.py
index 015353b90..9d2194781 100644
--- a/core/switch_core/gateway/tenants.py
+++ b/core/switch_core/gateway/tenants.py
@@ -1463,16 +1463,13 @@ async def remove_member(
"service_connections_deleted": len(removed_services),
},
)
+ # Sign-ins the member had under way in this workspace go with them.
+ await request.app.state.sign_in_flows.discard_for_user(session, user_id, None)
await session.delete(membership)
await session.commit()
for key_hash in revoked_key_hashes:
protocol.api_key_cache.invalidate(key_hash)
- github = getattr(request.app.state, "github_connections", None)
- if github is not None:
- for flow_id, flow in list(github.flows.items()):
- if (flow.tenant_id, flow.user_id) == (tenant_id, user_id):
- del github.flows[flow_id]
warning = await broker.finish_disconnect(session, user_id, removed_services)
return {"ok": True, "warning": warning}
diff --git a/core/switch_core/logging_config.py b/core/switch_core/logging_config.py
index ac79dc73c..c01079b22 100644
--- a/core/switch_core/logging_config.py
+++ b/core/switch_core/logging_config.py
@@ -52,29 +52,40 @@ def _context_fields(record: logging.LogRecord) -> dict[str, str]:
class TextFormatter(logging.Formatter):
- """Human-readable single line, with the context in brackets."""
+ """Human-readable single line, with the process id and the context in
+ brackets."""
- def __init__(self) -> None:
+ def __init__(self, process_id: str) -> None:
super().__init__(
fmt="%(asctime)s %(levelname)-5.5s [%(name)s]%(switch_context)s %(message)s",
datefmt="%Y-%m-%dT%H:%M:%S",
)
+ self._process_id = process_id
def format(self, record: logging.LogRecord) -> str:
- fields = _context_fields(record)
+ # Which switch-core process wrote the line comes first, under the same
+ # name as in the JSON format (`process_identity.py`).
+ fields = {"process_id": self._process_id, **_context_fields(record)}
rendered = " ".join(f"{name}={value}" for name, value in fields.items())
- record.switch_context = f" [{rendered}]" if rendered else ""
+ record.switch_context = f" [{rendered}]"
return super().format(record)
class JsonFormatter(logging.Formatter):
"""One JSON object per line."""
- def __init__(self, service: str, environment: str | None, version: str | None):
+ def __init__(
+ self,
+ service: str,
+ environment: str | None,
+ version: str | None,
+ process_id: str,
+ ):
super().__init__()
self._service = service
self._environment = environment
self._version = version
+ self._process_id = process_id
def format(self, record: logging.LogRecord) -> str:
payload: dict[str, Any] = {
@@ -85,6 +96,9 @@ class JsonFormatter(logging.Formatter):
"message": record.getMessage(),
"logger": {"name": record.name, "thread_name": record.threadName},
"service": self._service,
+ # Which switch-core process wrote the line, under the name its
+ # lease, its signals and its metrics carry (`process_identity.py`).
+ "process_id": self._process_id,
}
if self._environment:
payload["env"] = self._environment
@@ -128,7 +142,9 @@ class AccessQueryRedaction(logging.Filter):
return True
-def build_handler(config: SwitchConfig, version: str | None) -> logging.Handler:
+def build_handler(
+ config: SwitchConfig, version: str | None, process_id: str
+) -> logging.Handler:
"""The single stderr handler: format per config, context on every record."""
handler = logging.StreamHandler(sys.stderr)
if config.log_format == "json":
@@ -137,23 +153,26 @@ def build_handler(config: SwitchConfig, version: str | None) -> logging.Handler:
service=config.service_name,
environment=config.environment,
version=version,
+ process_id=process_id,
)
)
else:
- handler.setFormatter(TextFormatter())
+ handler.setFormatter(TextFormatter(process_id))
handler.addFilter(AccessQueryRedaction())
handler.addFilter(LogContextFilter(config.tenant_id))
return handler
-def configure_logging(config: SwitchConfig, version: str | None) -> None:
+def configure_logging(
+ config: SwitchConfig, version: str | None, process_id: str
+) -> None:
"""Install the root handler and levels. Call once, before anything logs."""
global _configured
root = logging.getLogger()
for existing in list(root.handlers):
root.removeHandler(existing)
- root.addHandler(build_handler(config, version))
+ root.addHandler(build_handler(config, version, process_id))
root.setLevel(config.log_level.upper())
logging.getLogger("switch_core").setLevel(config.switch_log_level.upper())
diff --git a/core/switch_core/main.py b/core/switch_core/main.py
index 5e7b5c7cd..e92973085 100644
--- a/core/switch_core/main.py
+++ b/core/switch_core/main.py
@@ -6,7 +6,7 @@ import logging
import os
import signal
import time
-from collections.abc import AsyncIterator, Coroutine
+from collections.abc import AsyncIterator, Awaitable, Callable, Coroutine
from contextlib import asynccontextmanager
from datetime import timedelta
from pathlib import Path
@@ -33,6 +33,7 @@ from switch_core.bridges.agent.protocol.agent_connections import (
)
from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.bridges.agent.protocol.agent_event_listener import AgentEventListener
+from switch_core.bridges.agent.protocol.connection_rows import SharedConnections
from switch_core.bridges.agent.protocol.controller_presence import ControllerPresence
from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
from switch_core.bridges.agent.registration_bootstrap import (
@@ -51,7 +52,9 @@ from switch_core.bridges.agent.server_connectors.opencode.connector import (
OpenCodeConnectionConfig,
OpenCodeConnector,
)
+from switch_core.bridges.agent.server_connectors.signals import ConnectorSignals
from switch_core.bridges.collaboration.adapter import PlatformAdapter
+from switch_core.bridges.collaboration.bridge_signals import BridgeSignals
from switch_core.bridges.collaboration.discord.adapter import (
DiscordAdapter,
)
@@ -92,9 +95,9 @@ from switch_core.bridges.collaboration.webhooks import (
from switch_core.bridges.resource.service import ResourceService
from switch_core.chats.service import ChatService
from switch_core.clients.actor import Actor, AgentActor, HumanActor, SystemActor
-from switch_core.clients.agent_consumer import AgentConsumer
from switch_core.clients.client_factory import ClientFactory
from switch_core.clients.client_lifecycle_service import ClientLifecycleService
+from switch_core.clients.client_signals import ClientSignals
from switch_core.clients.command_consumer import CommandConsumer
from switch_core.config import SwitchConfig, deprecated_env_names
from switch_core.connections.adapters.github import GitHubApp
@@ -107,12 +110,16 @@ from switch_core.connections.maintenance import (
from switch_core.connections.registry import ClientRegistration, build_adapters
from switch_core.db import encrypted_json
from switch_core.db.boot_lock import boot_lock
+from switch_core.db.connection_budget import check_connection_budget
from switch_core.db.engine import (
create_engine_from_config,
create_session_factory,
create_unpooled_engine,
)
-from switch_core.db.key_rotation import reencrypt_stored_secrets
+from switch_core.db.key_rotation import (
+ reencrypt_stored_secrets,
+ require_front_key_readable,
+)
from switch_core.db.models import (
TENANT_ZERO_ID,
ApiKey,
@@ -124,8 +131,12 @@ from switch_core.db.runtime_role import (
grant_runtime_role,
verify_restricted_role,
)
-from switch_core.db.schema_version import require_schema_at_head
+from switch_core.db.schema_version import (
+ require_schema_at_head,
+ schema_ahead_through_expand_only,
+)
from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.agent_connection_store import AgentConnectionStore
from switch_core.db.stores.agent_session_store import AgentSessionStore
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.api_key_store import ApiKeyStore
@@ -133,6 +144,7 @@ from switch_core.db.stores.bridge_message_map_store import BridgeMessageMapStore
from switch_core.db.stores.budget_store import BudgetStore
from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
from switch_core.db.stores.document_store import DocumentStore
from switch_core.db.stores.external_user_store import ExternalUserStore
from switch_core.db.stores.invitation_store import InvitationStore
@@ -141,6 +153,7 @@ from switch_core.db.stores.media_store import MediaStore
from switch_core.db.stores.message_store import MessageStore
from switch_core.db.stores.messaging_event_store import MessagingEventReceiptStore
from switch_core.db.stores.messaging_install_store import MessagingInstallStore
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
from switch_core.db.stores.package_store import PackageStore
from switch_core.db.stores.reference_store import ReferenceStore
from switch_core.db.stores.reference_type_store import ReferenceTypeStore
@@ -150,12 +163,15 @@ from switch_core.db.stores.room_role_store import RoomRoleStore
from switch_core.db.stores.room_store import RoomStore
from switch_core.db.stores.server_connector_store import ServerConnectorStore
from switch_core.db.stores.service_connection_store import ServiceConnectionStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
from switch_core.db.stores.task_store import TaskStore
from switch_core.db.stores.template_store import TemplateStore
from switch_core.db.stores.tenant_store import TenantStore
from switch_core.db.stores.usage_store import UsageStore
from switch_core.db.stores.user_store import UserStore
from switch_core.db.tenant_lookup import all_tenant_ids
+from switch_core.delivery.recipient import RecipientServices
+from switch_core.delivery.router import GROUP_SWEEP_SECONDS, MessageRouter
from switch_core.gateway.app import create_gateway_app
from switch_core.gateway.auth import hash_password
from switch_core.gateway.invite_mail import SmtpInviteMailer
@@ -175,6 +191,9 @@ from switch_core.observability.pool import (
)
from switch_core.observability.query import instrument_queries
from switch_core.outbound import guarded_async_client
+from switch_core.ownership import Ownership
+from switch_core.process_identity import new_process_id
+from switch_core.process_lease import ProcessLease
from switch_core.provisioning import Provisioning
from switch_core.provisioning.postgres import PostgresProvisioning
from switch_core.room_service import RoomService
@@ -184,13 +203,16 @@ from switch_core.session_activity.maintenance import (
)
from switch_core.session_activity.outcomes import ApprovalOutcomes
from switch_core.session_activity.service import AgentSessionActivityService
+from switch_core.signals import SignalBus
from switch_core.telemetry.messages import MessageTelemetry
from switch_core.telemetry.reporter import SnapshotReporter
from switch_core.telemetry.service import TelemetryService
from switch_core.telemetry.setup import build_telemetry
from switch_core.tenant_context import no_tenant
+from switch_core.transport.cursors import CursorWriter
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
+from switch_core.transport.membership_signals import MembershipSignals
from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
from switch_core.trust.setup import build_trust_client
from switch_core.version import switch_core_version
@@ -215,6 +237,15 @@ _CONNECTION_SWEEP_INTERVAL = 2.0
# `observability.logs.SHUTDOWN_FLUSH_SECONDS`.
_FORCED_EXIT_GRACE_SECONDS = 3.0
+# With several replicas, how long a process keeps serving after it is told to
+# stop and has taken itself out of readiness: long enough for the Service to
+# stop sending it new requests, so they go to another replica rather than
+# being refused. A single replica has nowhere to send them, and stops at once
+# as it always has. Inside the chart's 45-second grace period
+# (`switchCore.terminationGracePeriodSeconds`) with the bridge stops and
+# `_FORCED_EXIT_GRACE_SECONDS` after it.
+_DRAIN_SECONDS = 5.0
+
async def _agent_event_retention_loop(
session_factory: async_sessionmaker[AsyncSession], buffer: EventBuffer
@@ -231,6 +262,43 @@ async def _agent_event_retention_loop(
logger.exception("Agent event retention pass failed")
+async def _attachment_group_sweep_loop(
+ session_factory: async_sessionmaker[AsyncSession], router: MessageRouter
+) -> None:
+ """Decide the multi-file posts whose files stopped arriving, in every
+ tenant. The process that wrote a post's first file sets a timer for it;
+ this is what decides a post whose writer is gone, and every process runs
+ it, since a post is claimed by whoever gets to it first."""
+ with no_tenant():
+ while True:
+ await asyncio.sleep(GROUP_SWEEP_SECONDS)
+ try:
+ tenants = await all_tenant_ids(session_factory)
+ except Exception:
+ logger.exception("Attachment group sweep failed")
+ continue
+ # One tenant at a time, each on its own: a tenant whose sweep
+ # fails leaves the tenants after it to be swept all the same.
+ for tenant_id in tenants:
+ try:
+ await router.flush_due_groups(tenant_id)
+ except Exception:
+ logger.exception(
+ "Attachment group sweep failed for tenant %s", tenant_id
+ )
+
+
+async def _recheck_agent_rooms(
+ session_factory: async_sessionmaker[AsyncSession], router: MessageRouter
+) -> None:
+ """Check the room claims and managed agents this process holds against
+ the agents' memberships, in every tenant, for a membership signal that
+ never arrived (`MessageRouter.recheck_agent_rooms`)."""
+ with no_tenant():
+ for tenant_id in await all_tenant_ids(session_factory):
+ await router.recheck_agent_rooms(tenant_id)
+
+
async def _runtime_state_sweep_loop(protocol: AgentCore) -> None:
# `no_tenant` for the reason every other long-lived task does it: a task
# keeps the context of whoever created it, and nothing in here may depend
@@ -290,27 +358,73 @@ async def _connection_sweep_loop(protocol: AgentCore) -> None:
# Under `_FORCED_EXIT_GRACE_SECONDS`, and first in the teardown: what it
-# writes is what lets every other process see this one's machines go offline
-# now rather than when its lease lapses.
-_MANAGEMENT_STOP_SECONDS = 1.0
-
-
-async def _stop_management(management: Management) -> None:
- """Record the controller sockets that went and mark this process's lease
- stopped. Never raises: if it fails, the lease lapses on its own."""
+# writes is what lets every other process see this one's connections go
+# offline, and take over the work it owned, now rather than when its lease
+# lapses.
+_RELEASE_SECONDS = 1.0
+
+# How long a process shutting down gives the work it owns to stop and be
+# handed over, before the rest of its release. Each consumer finishes the row
+# in hand within a grace of its own (`transport/postgres.py`), so this is a
+# backstop for a stop that hangs on a platform: the lease then lapses on its
+# own and the work is taken over after it. Inside the chart's 45-second grace
+# period (`switchCore.terminationGracePeriodSeconds`) with the drain and the
+# rest of the shutdown after it.
+_HAND_OVER_SECONDS = 10.0
+
+
+async def _release_process(
+ ownership: Ownership,
+ connections: SharedConnections,
+ management: Management | None,
+ lease: ProcessLease,
+) -> None:
+ """Hand the work this process owns to the others, record the agent
+ connections and controller sockets that went, then mark this process's
+ lease stopped and tell the others, which take the work over on hearing
+ it. In that order: the lease must not read as stopped while
+ owned work still runs here, or another process would start it beside
+ this one. Once the owned work has stopped, the lease is marked stopped
+ even when the recording fails, so the connections this process held read
+ offline either way. Never raises: if it all fails, the lease lapses on
+ its own.
+
+ Run from the signal handler and again from the lifespan's teardown, in
+ either order: once one call has given up on the owned work, the other
+ leaves the lease to lapse as well (`Ownership.stop` raises)."""
try:
- async with asyncio.timeout(_MANAGEMENT_STOP_SECONDS):
- await management.stop()
+ async with asyncio.timeout(_HAND_OVER_SECONDS):
+ await ownership.stop()
+ except Exception:
+ logger.warning(
+ "Stopping this process's owned work at shutdown failed or took "
+ "longer than %.0fs; the other processes take it over once its "
+ "lease lapses.",
+ _HAND_OVER_SECONDS,
+ exc_info=True,
+ )
+ return
+ try:
+ async with asyncio.timeout(_RELEASE_SECONDS):
+ try:
+ await connections.flush()
+ if management is not None:
+ await management.flush()
+ finally:
+ await lease.stop()
+ ownership.gone()
except TimeoutError:
logger.warning(
- "Gave up recording controller disconnections after %.1fs; their "
- "machines read offline once this process's lease lapses.",
- _MANAGEMENT_STOP_SECONDS,
+ "Gave up releasing switch-core process %s after %.1fs; its "
+ "connections read offline once its lease lapses.",
+ lease.process_id,
+ _RELEASE_SECONDS,
)
except Exception:
logger.warning(
- "Recording controller disconnections at shutdown failed; their "
- "machines read offline once this process's lease lapses.",
+ "Releasing switch-core process %s at shutdown failed; its "
+ "connections read offline once its lease lapses.",
+ lease.process_id,
exc_info=True,
)
@@ -472,7 +586,7 @@ async def _check_tenant_isolation(config: SwitchConfig, engine: AsyncEngine) ->
return True
-async def run(config: SwitchConfig) -> None:
+async def run(config: SwitchConfig, process_id: str) -> None:
# ── Database ─────────────────────────────────────────────────────────────
# The migration and the runtime role's grant re-issue already ran, under
# Switch's boot-time advisory lock, before this coroutine was ever started
@@ -482,6 +596,7 @@ async def run(config: SwitchConfig) -> None:
encrypted_json.configure(config.keyring)
engine = create_engine_from_config(config, poolclass=WaitTimedQueuePool)
tenants_isolated = await _check_tenant_isolation(config, engine)
+ await check_connection_budget(config, engine)
session_factory = create_session_factory(engine)
# Wired here rather than inside the engine factory, so the database layer
# keeps knowing nothing about observability. Unconditional: the listeners
@@ -509,9 +624,50 @@ async def run(config: SwitchConfig) -> None:
session_activity_listener = AgentSessionActivityListener(
lambda: create_unpooled_engine(config)
)
+ # Signals between switch-core processes (see `signals.py`): the same
+ # arrangement again, carrying "who should look" rather than a room's rows.
+ signal_bus = SignalBus(
+ session_factory,
+ lambda: create_unpooled_engine(config),
+ process_id=process_id,
+ )
+ # This process's row in `switch_core_processes`, claimed before anything
+ # can record a connection against it and held until the process drains.
+ # What another process reads to know this one is alive.
+ process_lease = ProcessLease(
+ process_id=process_id,
+ session_factory=session_factory,
+ processes=SwitchCoreProcessStore(),
+ key_ids=[key.id for key in config.keyring.keys],
+ )
+ await process_lease.renew()
+ logger.info("Claimed switch-core process lease %s", process_id)
+ # Renewed from here on, not from when the process starts serving: the
+ # owned work started during boot stops once the lease is not sure
+ # (`ProcessLease.surely_held`), and a boot can take longer than that.
+ lease_task = asyncio.create_task(process_lease.run())
+ # Which of the work that must run once this process runs (`ownership.py`),
+ # held under that lease. With one process, all of it.
+ ownership = Ownership(
+ lease=process_lease,
+ session_factory=session_factory,
+ bus=signal_bus,
+ store=OwnershipLeaseStore(),
+ processes=SwitchCoreProcessStore(),
+ )
+ # Every owned consumer's progress, written in batches
+ # (`transport/cursors.py`).
+ cursor_writer = CursorWriter(
+ session_factory=session_factory,
+ cursors=DeliveryCursorStore(),
+ leases=OwnershipLeaseStore(),
+ process_id=process_id,
+ )
# Invitations for the Postgres transport, which has no durable one of its
# own. Built unconditionally: it is a dict until something registers.
+ # Membership changes made in other processes reach the clients here over
+ # the signal bus (`transport/membership_signals.py`).
invites = InviteBus()
ephemeral = EphemeralBus()
@@ -553,17 +709,17 @@ async def run(config: SwitchConfig) -> None:
# `_seed_agent_registration_bootstrap_key`'s docstring), so only a
# session-level lock, held for the whole span, actually serialises it.
async with boot_lock(config):
+ await _seed_admin_and_bootstrap_key(
+ session_factory, user_store, api_key_store, agent_store, config, process_id
+ )
async with engine.begin() as connection:
event_boot = await connection.scalar(agent_event_boot_sequence.next_value())
- await _seed_admin_user(session_factory, user_store, config)
- await _seed_agent_registration_bootstrap_key(
- session_factory, user_store, api_key_store, agent_store, config
- )
# ── Event queue + request trackers ───────────────────────────────────────
event_buffer = EventBuffer(
session_factory,
sequence_base=event_boot << 32,
+ rooms=message_listener,
retention_seconds=config.agent_event_retention_seconds,
)
# Listens for the `NOTIFY` that each insert into `agent_events` sends, and
@@ -614,26 +770,69 @@ async def run(config: SwitchConfig) -> None:
for tenant_id in tenant_ids:
async with tenant_session(session_factory, tenant_id) as session:
await resource_service.log_builtin_shadowing(session)
- await reencrypt_stored_secrets(session_factory, config.keyring, tenant_ids)
- # After rotation, which has opened every stored GitHub secret or stopped
- # the boot: the move reads them, once per tenant.
- await move_github_connections(session_factory, config.keyring, tenant_ids)
+ # Under the boot lock, so two processes starting together do not rewrite
+ # the same secrets at once, and after `_seed_admin_and_bootstrap_key` has
+ # checked that every process still running can read the key they are
+ # rewritten under.
+ async with boot_lock(config):
+ await reencrypt_stored_secrets(session_factory, config.keyring, tenant_ids)
+ # After rotation, which has opened every stored GitHub secret or
+ # stopped the boot: the move reads them, once per tenant.
+ await move_github_connections(session_factory, config.keyring, tenant_ids)
+
+ # One connection registry for the process. Created here rather than inside
+ # the agent bridge because the router is wired first and reads presence
+ # from it: an agent is reachable if it has a live connection OR a fresh
+ # heartbeat row (CHOO-1857 stage B).
+ connections = AgentConnectionRegistry()
+ # The agent connections this process holds, recorded for the others under
+ # this process's lease, and a copy of theirs kept in `connections`
+ # (`connection_rows.py`), so the router's presence questions are answered
+ # for every process.
+ shared_connections = SharedConnections(
+ registry=connections,
+ session_factory=session_factory,
+ store=AgentConnectionStore(),
+ bus=signal_bus,
+ lease=process_lease,
+ )
+
+ # ── Message router ───────────────────────────────────────────────────────
+ # Every row written to a room goes through it, which decides in the same
+ # transaction which agents the row is for (`delivery/router.py`).
+ router = MessageRouter(
+ services=RecipientServices(
+ session_factory=session_factory,
+ event_buffer=event_buffer,
+ agent_store=agent_store,
+ room_store=room_store,
+ document_store=document_store,
+ reference_store=reference_store,
+ agent_session_store=agent_session_store,
+ room_role_store=room_role_store,
+ external_user_store=external_user_store,
+ connections=connections,
+ frontend_base_url=config.frontend_base_url,
+ message_telemetry=message_telemetry,
+ ),
+ message_store=message_store,
+ client_store=client_store,
+ budget_store=budget_store,
+ )
+
+ MembershipSignals(invites=invites, bus=signal_bus, router=router)
+ signal_bus.on_resync(lambda: _recheck_agent_rooms(session_factory, router))
+ signal_bus.on_recheck(lambda: _recheck_agent_rooms(session_factory, router))
# ── Provisioning ─────────────────────────────────────────────────────────
provisioning: Provisioning = PostgresProvisioning(
session_factory=session_factory,
room_store=room_store,
client_store=client_store,
- message_store=message_store,
+ router=router,
invites=invites,
)
- # One connection registry for the process. Created here rather than inside
- # the agent bridge because the room clients are wired first and read
- # presence from it — an agent is reachable if it has a live connection OR a
- # fresh heartbeat row (CHOO-1857 stage B).
- connections = AgentConnectionRegistry()
-
# One shared read per room for every client in this process.
room_cache = RoomDeliveryCache(
session_factory=session_factory,
@@ -660,26 +859,12 @@ async def run(config: SwitchConfig) -> None:
ephemeral=ephemeral,
room_cache=room_cache,
message_observer=message_telemetry,
+ router=router,
)
- client_factory.register(
- "agent",
- AgentActor,
- AgentConsumer,
- event_buffer=event_buffer,
- agent_store=agent_store,
- room_store=room_store,
- bridge_store=bridge_store,
- document_store=document_store,
- reference_store=reference_store,
- agent_session_store=agent_session_store,
- room_role_store=room_role_store,
- external_user_store=external_user_store,
- connections=connections,
- frontend_base_url=config.frontend_base_url,
- message_telemetry=message_telemetry,
- )
- # Members that only write: a person on another platform, and a bridge's own
+ # Members that only write: an agent, whose rows are decided as they are
+ # written (the router), a person on another platform, and a bridge's own
# identity (its reader, the WorkspaceConsumer, is built by the bridge).
+ client_factory.register("agent", AgentActor)
client_factory.register("user", HumanActor)
client_factory.register("bridge", Actor)
@@ -692,7 +877,12 @@ async def run(config: SwitchConfig) -> None:
session_factory=session_factory,
config=config,
tenants_isolated=tenants_isolated,
+ ownership=ownership,
+ cursor_writer=cursor_writer,
)
+ # Clients registered or deleted in another process are started or stopped
+ # here too (`clients/client_signals.py`).
+ ClientSignals(lifecycle=client_lifecycle, bus=signal_bus)
# ── Collaboration bridge lifecycle ───────────────────────────────────────
collab_lifecycle = CollaborationBridgeLifecycleService(
@@ -711,9 +901,16 @@ async def run(config: SwitchConfig) -> None:
session_activity_listener=session_activity_listener,
session_activity_service=AgentSessionActivityService(session_factory),
connections=connections,
+ ownership=ownership,
+ cursor_writer=cursor_writer,
telemetry=telemetry,
trust_client=trust_client,
)
+ # Bridges registered, edited or removed in another process are applied to
+ # this process's copies too (`bridges/collaboration/bridge_signals.py`),
+ # and agents registered or deleted elsewhere reach their identities here.
+ BridgeSignals(lifecycle=collab_lifecycle, bus=signal_bus)
+ client_lifecycle.add_agents_changed_listener(collab_lifecycle.agents_changed)
# ── Room service ─────────────────────────────────────────────────────────
room_service = RoomService(
@@ -752,7 +949,9 @@ async def run(config: SwitchConfig) -> None:
# Built before the agent bridge app because its authenticator is the bearer
# middleware's controller branch; its routes are installed once both apps
# exist, below.
- management = create_management(config, session_factory, connections.controllers)
+ management = create_management(
+ config, session_factory, connections.controllers, signal_bus, process_lease
+ )
# ── FastAPI apps ─────────────────────────────────────────────────────────
agent_bridge_app, protocol = create_agent_bridge_app(
@@ -791,7 +990,9 @@ async def run(config: SwitchConfig) -> None:
keyring=config.keyring,
telemetry=telemetry,
outbound_policy=config.outbound_policy,
+ ownership=ownership,
)
+ ConnectorSignals(lifecycle=connector_lifecycle, bus=signal_bus)
connector_lifecycle.register_connector_type(
"opencode", OpenCodeConnector, OpenCodeConnectionConfig
)
@@ -890,6 +1091,7 @@ async def run(config: SwitchConfig) -> None:
agent_store=agent_store,
user_store=user_store,
message_store=message_store,
+ router=router,
media_store=media_store,
usage_store=usage_store,
id_server_name=config.id_server_name,
@@ -944,9 +1146,16 @@ async def run(config: SwitchConfig) -> None:
# Set by the lifespan; before that the honest answer is "no".
observability: Observability | None = None
+ # Set as shutdown begins, so the Service stops routing here first.
+ draining = asyncio.Event()
@agent_bridge_app.get("/health/ready")
async def readiness_check() -> JSONResponse:
+ if draining.is_set():
+ return JSONResponse(
+ {"status": "not ready", "checks": {"draining": {"healthy": False}}},
+ status_code=503,
+ )
if observability is None:
return JSONResponse(
{"status": "not ready", "checks": {"startup": {"healthy": False}}},
@@ -985,16 +1194,27 @@ async def run(config: SwitchConfig) -> None:
# Before the bridge serves: until Core knows which agents a controller
# runs, their own keys would be let in and their presence misread.
await management.load_bindings()
- await management.start()
+ # And which controllers the other processes hold, so their agents do
+ # not read offline here until the signal bus has connected.
+ await management.load_connections()
+
+ # Before the bridge serves, so the first presence question already knows
+ # the agents connected to the other processes.
+ await shared_connections.start()
agent_bridge_app.mount("/gateway", gateway_app)
# ── Ensure system clients exist ─────────────────────────────────────────
- await client_lifecycle.ensure_system_client("admin")
+ # Under the boot lock: it checks, then creates, and two processes starting
+ # on a fresh install would otherwise both create tenant zero's admin.
+ async with boot_lock(config):
+ await client_lifecycle.ensure_system_client("admin")
probes = RuntimeProbes(
listener_connected=message_listener.connected.is_set,
session_activity_listener_connected=session_activity_listener.connected.is_set,
+ agent_event_listener_connected=agent_event_listener.connected.is_set,
+ signal_bus_connected=signal_bus.connected.is_set,
bridges_running=collab_lifecycle.running_count,
bridges_running_by_platform=collab_lifecycle.running_by_platform,
bridges_configured=collab_lifecycle.expected_count,
@@ -1055,6 +1275,7 @@ async def run(config: SwitchConfig) -> None:
version=switch_core_version(),
session_factory=session_factory,
probes=probes,
+ process_id=process_id,
db_server_engine=lambda: create_unpooled_engine(config),
)
asyncio.create_task(connector_lifecycle.start_all())
@@ -1062,6 +1283,9 @@ async def run(config: SwitchConfig) -> None:
retention_task = asyncio.create_task(
_agent_event_retention_loop(session_factory, event_buffer)
)
+ attachment_group_task = asyncio.create_task(
+ _attachment_group_sweep_loop(session_factory, router)
+ )
session_activity_task = asyncio.create_task(
session_activity_maintenance_loop(session_factory)
)
@@ -1071,6 +1295,8 @@ async def run(config: SwitchConfig) -> None:
service_token_task = asyncio.create_task(
service_token_maintenance_loop(session_factory, service_broker)
)
+ shared_connections_task = asyncio.create_task(shared_connections.run())
+ cursor_writer_task = asyncio.create_task(cursor_writer.run())
management_task = (
asyncio.create_task(management.run())
if management is not None
@@ -1086,23 +1312,36 @@ async def run(config: SwitchConfig) -> None:
await message_listener.start()
await session_activity_listener.start()
await agent_event_listener.start()
+ await signal_bus.start()
try:
yield
finally:
sweep_task.cancel()
retention_task.cancel()
+ attachment_group_task.cancel()
+ router.close()
session_activity_task.cancel()
connection_sweep_task.cancel()
service_token_task.cancel()
if snapshot_task is not None:
snapshot_task.cancel()
- if management is not None:
- assert management_task is not None
+ shared_connections_task.cancel()
+ if management_task is not None:
management_task.cancel()
- await _stop_management(management)
+ ownership_task.cancel()
+ # The lease goes on being renewed while the owned work is
+ # handed over, which can take a few seconds: it must not lapse
+ # under work still running here. A renewal after the release
+ # does nothing.
+ await _release_process(
+ ownership, shared_connections, management, process_lease
+ )
+ lease_task.cancel()
+ cursor_writer_task.cancel()
await message_listener.stop()
await session_activity_listener.stop()
await agent_event_listener.stop()
+ await signal_bus.stop()
# Before the operational flush, and bounded: the whole
# teardown runs inside `_FORCED_EXIT_GRACE_SECONDS` and a
# product event is the least valuable thing in it.
@@ -1178,10 +1417,21 @@ async def run(config: SwitchConfig) -> None:
_run_telegram_app(telegram_app), name="telegram-app"
)
+ # Start the work this process owns, before it serves: with one process,
+ # all of it. Then the quiet copies of the bridges another process holds.
+ await ownership.start()
+ # Running from here, so the rest of the boot is covered: a lease that
+ # stops being sure stops the work it holds there and then.
+ ownership_task = asyncio.create_task(ownership.run())
+ await collab_lifecycle.start_quiet_copies()
+
# Backfill room membership: system clients (e.g. the admin client) added
# after a room was created, and any agent whose invite did not land. The
- # just-started clients accept the invites on their first sync.
- await room_service.reconcile_room_clients()
+ # just-started clients accept the invites on their first sync. Under the
+ # boot lock, so two processes starting together do not both invite the
+ # same missing client.
+ async with boot_lock(config):
+ await room_service.reconcile_room_clients()
logger.info(
"Switch is running on http://%s:%d", config.server_host, config.server_port
@@ -1207,6 +1457,11 @@ async def run(config: SwitchConfig) -> None:
lambda: asyncio.create_task(
_shutdown(
server,
+ draining.set,
+ lambda: _release_process(
+ ownership, shared_connections, management, process_lease
+ ),
+ _DRAIN_SECONDS if config.core_replicas > 1 else 0.0,
connections.controllers,
client_lifecycle,
collab_lifecycle,
@@ -1224,6 +1479,29 @@ async def run(config: SwitchConfig) -> None:
await server.serve()
+async def _seed_admin_and_bootstrap_key(
+ session_factory: async_sessionmaker[AsyncSession],
+ user_store: UserStore,
+ api_key_store: ApiKeyStore,
+ agent_store: AgentStore,
+ config: SwitchConfig,
+ process_id: str,
+) -> None:
+ """Seed the admin user and the agent-registration bootstrap key, after
+ checking that every live process can read this one's front key.
+
+ The check comes first because the bootstrap key is the first thing a boot
+ encrypts: it is written under the front key when it is created or
+ AGENT_REGISTRATION_TOKEN changes, and a process refused only after that
+ would already have stored a value the others cannot decrypt.
+ """
+ await require_front_key_readable(session_factory, config.keyring, process_id)
+ await _seed_admin_user(session_factory, user_store, config)
+ await _seed_agent_registration_bootstrap_key(
+ session_factory, user_store, api_key_store, agent_store, config
+ )
+
+
async def _seed_admin_user(
session_factory: async_sessionmaker[AsyncSession],
user_store: UserStore,
@@ -1692,6 +1970,9 @@ def _distributed_teams_app(
async def _shutdown(
server: uvicorn.Server,
+ begin_draining: Callable[[], None],
+ release: Callable[[], Awaitable[None]],
+ drain_seconds: float,
controllers: ControllerPresence,
client_lifecycle: ClientLifecycleService,
collab_lifecycle: CollaborationBridgeLifecycleService,
@@ -1706,9 +1987,18 @@ async def _shutdown(
telegram_app_task: asyncio.Task[None] | None = None,
) -> None:
logger.info("Shutting down...")
+ begin_draining()
# Before uvicorn closes the sockets, so the controllers' are recorded as
# closed by the server.
controllers.begin_shutdown()
+ # Also before the sockets close, and not left to the lifespan's teardown:
+ # uvicorn runs that only once every connection has closed, and a stream
+ # still open when the process is killed would mean it never runs. Every
+ # other process reads this one's machines offline from here, and has taken
+ # over the consumers, bridges and connectors it owned.
+ await release()
+ if drain_seconds:
+ await asyncio.sleep(drain_seconds)
server.should_exit = True
await connector_lifecycle.stop_all()
await collab_lifecycle.stop_all()
@@ -1779,6 +2069,17 @@ async def _migrate_and_grant(config: SwitchConfig) -> None:
"""
alembic_cfg = _alembic_config()
async with boot_lock(config):
+ if await _schema_ahead(config, alembic_cfg):
+ # A newer release has migrated, and this one is still rolling out
+ # or being rolled back. Alembic has no file for the database's
+ # head and would refuse; the newer release already granted what
+ # its tables need, and this release reads none of them.
+ logger.warning(
+ "The database is ahead of switch-core %s by expand-only "
+ "revisions only; nothing to migrate.",
+ switch_core_version() or "(version unknown)",
+ )
+ return
await asyncio.to_thread(alembic_command.upgrade, alembic_cfg, "head")
await _prepare_database(config)
logger.info(
@@ -1787,6 +2088,20 @@ async def _migrate_and_grant(config: SwitchConfig) -> None:
)
+async def _schema_ahead(config: SwitchConfig, alembic_cfg: AlembicConfig) -> bool:
+ """Whether the database is ahead of this build by expand-only revisions,
+ read on the connection the migration would run on."""
+ engine = create_async_engine(
+ config.owner_database_url or config.database_url,
+ poolclass=NullPool,
+ connect_args=config.db_connect_args,
+ )
+ try:
+ return await schema_ahead_through_expand_only(engine, alembic_cfg)
+ finally:
+ await engine.dispose()
+
+
async def _check_schema_at_head(config: SwitchConfig) -> None:
"""Stand in for `_migrate_and_grant` on a server that does not migrate."""
if config.db_owner_password is not None:
@@ -1801,7 +2116,7 @@ async def _check_schema_at_head(config: SwitchConfig) -> None:
await require_schema_at_head(engine, _alembic_config())
finally:
await engine.dispose()
- logger.info("Database schema is at head; migrations were applied before boot")
+ logger.info("Database schema is servable; migrations were applied before boot")
def _alembic_config() -> AlembicConfig:
@@ -1828,7 +2143,7 @@ def migrate() -> None:
"""
config = SwitchConfig()
running_version = switch_core_version()
- configure_logging(config, running_version)
+ configure_logging(config, running_version, new_process_id(config.pod_name))
logger.info("Migrating switch-core %s", running_version or "(version unknown)")
@@ -1838,9 +2153,14 @@ def migrate() -> None:
def main() -> None:
config = SwitchConfig()
running_version = switch_core_version()
- configure_logging(config, running_version)
+ process_id = new_process_id(config.pod_name)
+ configure_logging(config, running_version, process_id)
- logger.info("Starting switch-core %s", running_version or "(version unknown)")
+ logger.info(
+ "Starting switch-core %s as process %s",
+ running_version or "(version unknown)",
+ process_id,
+ )
for warning in deprecated_env_names():
logger.warning(warning)
@@ -1849,7 +2169,7 @@ def main() -> None:
else:
asyncio.run(_check_schema_at_head(config))
- asyncio.run(run(config))
+ asyncio.run(run(config, process_id))
if __name__ == "__main__":
diff --git a/core/switch_core/management/agent_operations.py b/core/switch_core/management/agent_operations.py
index 10d38610a..dbc207ed9 100644
--- a/core/switch_core/management/agent_operations.py
+++ b/core/switch_core/management/agent_operations.py
@@ -31,7 +31,6 @@ from switch_core.management import reason_codes
from switch_core.management.advanced_config import provider_fields
from switch_core.management.errors import ManagementError
from switch_core.management.placement import is_revoked, provider_auth
-from switch_core.management.process_lease import ProcessLeases
from switch_core.management.schemas import (
CreateManagedAgentRequest,
DefinitionV1,
@@ -41,6 +40,7 @@ from switch_core.management.schemas import (
workspaces_dir_of,
)
from switch_core.management.service import ManagementService
+from switch_core.process_lease import ProcessLeases
NOTHING_CREATED = "Nothing was created"
NOTHING_CHANGED = "Nothing was changed"
diff --git a/core/switch_core/management/bindings.py b/core/switch_core/management/bindings.py
index a5f2ca645..b5f82cba7 100644
--- a/core/switch_core/management/bindings.py
+++ b/core/switch_core/management/bindings.py
@@ -1,4 +1,4 @@
-"""Telling Core, at startup, which controller runs each managed agent.
+"""Telling Core which controller runs each managed agent.
Core keeps the bindings in memory (`ControllerPresence`), so a restart forgets
them, and they must be back before the agent bridge serves: until then a
@@ -7,6 +7,14 @@ would be let in. They are read from every tenant, which is a question no
tenant can be scoped to, so the tenants come from the exemption
(`db/tenant_lookup.py`) and each tenant's definitions are then read under its
own policy.
+
+The same reads keep every process's bindings current after startup. A
+placement changes in the process that served the request, and every other
+process holds the old binding until it reads the tables again: on a
+`controller.assignment_changed` or `controller.renamed` signal for one
+controller (`reconcile_controller_bindings`), and for all of them when the
+signal connection comes back and on each periodic re-check
+(`reconcile_bindings`). See `management/signals.py`.
"""
from __future__ import annotations
@@ -20,15 +28,29 @@ from switch_core.bridges.agent.protocol.controller_presence import (
ControllerPresence,
)
from switch_core.db.models import AgentController
+from switch_core.db.models import AgentDefinition as AgentDefinitionRow
from switch_core.db.session_scope import tenant_session
from switch_core.db.stores.agent_controller_store import AgentControllerStore
from switch_core.db.stores.agent_definition_store import AgentDefinitionStore
from switch_core.db.tenant_lookup import all_tenant_ids
-from switch_core.management.service import binding_of
logger = logging.getLogger(__name__)
+def binding_of(
+ tenant_id: str, row: AgentDefinitionRow, controller: AgentController
+) -> Binding:
+ """The binding Core keeps for a definition placed on `controller`."""
+ assert row.controller_id == controller.id
+ return Binding(
+ agent_id=row.agent_id,
+ controller_id=controller.id,
+ tenant_id=tenant_id,
+ controller_name=controller.name,
+ running=row.desired_state == "running",
+ )
+
+
async def load_bindings(
*,
session_factory: async_sessionmaker[AsyncSession],
@@ -38,6 +60,92 @@ async def load_bindings(
) -> int:
"""Load every placed definition into `presence`, and mark the controllers
already revoked that agents are still placed on. Returns how many."""
+ bindings, revoked = await _read_bindings(
+ session_factory=session_factory,
+ definitions=definitions,
+ controllers=controllers,
+ )
+ presence.load(bindings)
+ for controller_id in revoked:
+ presence.revoke_controller(controller_id)
+ logger.info("Loaded %d controller binding(s) into Core", len(bindings))
+ return len(bindings)
+
+
+async def reconcile_bindings(
+ *,
+ session_factory: async_sessionmaker[AsyncSession],
+ definitions: AgentDefinitionStore,
+ controllers: AgentControllerStore,
+ presence: ControllerPresence,
+) -> None:
+ """Bring every binding in `presence` in line with the tables, changing
+ only what differs, and close the controllers revoked meanwhile."""
+ since = presence.change_mark()
+ bindings, revoked = await _read_bindings(
+ session_factory=session_factory,
+ definitions=definitions,
+ controllers=controllers,
+ )
+ presence.reconcile(
+ bindings,
+ agent_ids=presence.bound_agent_ids() | {b.agent_id for b in bindings},
+ since=since,
+ )
+ for controller_id in revoked:
+ if not presence.is_revoked(controller_id):
+ presence.revoke_controller(controller_id)
+
+
+async def reconcile_controller_bindings(
+ *,
+ session_factory: async_sessionmaker[AsyncSession],
+ definitions: AgentDefinitionStore,
+ controllers: AgentControllerStore,
+ presence: ControllerPresence,
+ tenant_id: str,
+ controller_id: str,
+) -> None:
+ """Bring the bindings of the agents on one controller in line with the
+ tables: the agents placed on it now, and the ones this process still has
+ bound to it. One of those that moved to another controller is bound
+ there, so the order in which the two controllers' signals arrive does not
+ matter."""
+ since = presence.change_mark()
+ bindings: list[Binding] = []
+ async with tenant_session(session_factory, tenant_id) as session:
+ controller = await controllers.get(session, tenant_id, controller_id)
+ placed = (
+ await definitions.list_for_controller(session, tenant_id, controller_id)
+ if controller is not None
+ else []
+ )
+ for row, _agent in placed:
+ assert controller is not None
+ bindings.append(binding_of(tenant_id, row, controller))
+ placed_here = {binding.agent_id for binding in bindings}
+ for agent_id in presence.agents_of(controller_id) - placed_here:
+ moved = await definitions.get_for_agent(session, tenant_id, agent_id)
+ if moved is None or moved.controller_id is None:
+ continue
+ elsewhere = await controllers.get(session, tenant_id, moved.controller_id)
+ if elsewhere is not None:
+ bindings.append(binding_of(tenant_id, moved, elsewhere))
+ presence.reconcile(
+ bindings,
+ agent_ids=presence.agents_of(controller_id) | placed_here,
+ since=since,
+ )
+
+
+async def _read_bindings(
+ *,
+ session_factory: async_sessionmaker[AsyncSession],
+ definitions: AgentDefinitionStore,
+ controllers: AgentControllerStore,
+) -> tuple[list[Binding], set[str]]:
+ """Every placed definition's binding, and the revoked controllers that
+ agents are still placed on."""
bindings: list[Binding] = []
revoked: set[str] = set()
for tenant_id in await all_tenant_ids(session_factory):
@@ -62,8 +170,4 @@ async def load_bindings(
if controller.revoked_at is not None:
revoked.add(controller.id)
bindings.append(binding_of(tenant_id, row, controller))
- presence.load(bindings)
- for controller_id in revoked:
- presence.revoke_controller(controller_id)
- logger.info("Loaded %d controller binding(s) into Core", len(bindings))
- return len(bindings)
+ return bindings, revoked
diff --git a/core/switch_core/management/connection_ledger.py b/core/switch_core/management/connection_ledger.py
index a00b0ab9e..4fa9e4b6a 100644
--- a/core/switch_core/management/connection_ledger.py
+++ b/core/switch_core/management/connection_ledger.py
@@ -12,15 +12,20 @@ once:
- disconnected: `disconnected_at` and `disconnect_reason` (`socket_closed`,
`server_shutdown`, `heartbeat_lapsed`, `taken_over`, `revoked`).
-That row, and the lease of the process it names (`process_lease.py`), are
+That row, and the lease of the process it names (`switch_core/process_lease.py`), are
what a machine's state is read from (`placement.controller_state`).
-Which write wins, across processes: a connecting comes from the process
-holding the live socket, and replaces whatever the row holds. A closing is
-only written while the row still names that connection, so a process closing
-a connection the controller has since replaced through another process
-leaves the replacement standing. (Clocks are not compared: two processes'
-clocks need not agree.)
+Which write wins, across processes: each is written only while the row
+still names its connection, a connecting while that is the connection the
+controller opened last and this process holds it, a closing while this
+process holds its socket. So a write about a connection the controller has
+since replaced, or reattached through another process, leaves the newer one
+standing however late it lands. (Clocks are not compared: two processes'
+clocks need not agree.) A closing also keeps where the connection's stream
+resumes each agent, for a socket that reattaches through another process.
+
+Each transition that took effect is announced (`announce`), so every other
+process reads the row again and knows which process holds the socket now.
Writes are queued, latest per controller, and written by one task (`run`),
woken by each transition, so a socket attaching or going never waits on the
@@ -58,6 +63,7 @@ class _Transition:
at: datetime
# None for a connecting; why the socket went for a closing.
disconnect_reason: str | None
+ resume_cursors: dict[str, int | None]
class ControllerConnectionLedger:
@@ -70,8 +76,12 @@ class ControllerConnectionLedger:
session_factory: async_sessionmaker[AsyncSession],
controllers: AgentControllerStore,
clock: Callable[[], datetime],
+ announce: Callable[[str, str], None],
) -> None:
self._process_id = process_id
+ # Called with the tenant and the controller of each transition that
+ # took effect, once it has committed.
+ self._announce = announce
self._session_factory = session_factory
self._controllers = controllers
self._clock = clock
@@ -92,6 +102,7 @@ class ControllerConnectionLedger:
connection_id=conn.id,
at=self._clock(),
disconnect_reason=reason,
+ resume_cursors=dict(conn.resume_cursors),
)
self._wake.set()
@@ -132,20 +143,25 @@ class ControllerConnectionLedger:
unwritten = list(transitions)
try:
for tenant_id, batch in by_tenant.items():
+ written: list[_Transition] = []
async with tenant_session(self._session_factory, tenant_id) as session:
for transition in batch:
- await self._write_one(session, transition)
+ if await self._write_one(session, transition):
+ written.append(transition)
await session.commit()
for transition in batch:
unwritten.remove(transition)
+ for transition in written:
+ self._announce(transition.tenant_id, transition.controller_id)
except BaseException:
for transition in unwritten:
self._pending.setdefault(transition.controller_id, transition)
raise
- async def _write_one(self, session: AsyncSession, transition: _Transition) -> None:
+ async def _write_one(self, session: AsyncSession, transition: _Transition) -> bool:
+ """Write one transition. Returns whether it took effect."""
if transition.disconnect_reason is None:
- found = await self._controllers.record_connected(
+ written = await self._controllers.record_connected(
session,
transition.tenant_id,
transition.controller_id,
@@ -153,25 +169,30 @@ class ControllerConnectionLedger:
process_id=self._process_id,
connected_at=transition.at,
)
- if not found:
- logger.warning(
- "Controller %s connected, but has no row to record it on",
+ if not written:
+ logger.info(
+ "Controller %s connection %s attached, but its row names a "
+ "newer connection or another process; left as it is",
transition.controller_id,
+ transition.connection_id,
)
- return
+ return written
written = await self._controllers.record_disconnected(
session,
transition.tenant_id,
transition.controller_id,
connection_id=transition.connection_id,
+ process_id=self._process_id,
disconnected_at=transition.at,
reason=transition.disconnect_reason,
+ resume_cursors=transition.resume_cursors,
)
if not written:
logger.info(
"Controller %s connection %s went (%s), but its row names a newer "
- "connection; left as it is",
+ "connection or another process; left as it is",
transition.controller_id,
transition.connection_id,
transition.disconnect_reason,
)
+ return written
diff --git a/core/switch_core/management/controller_routes.py b/core/switch_core/management/controller_routes.py
index e0689c2ff..ebd5e93f6 100644
--- a/core/switch_core/management/controller_routes.py
+++ b/core/switch_core/management/controller_routes.py
@@ -43,6 +43,7 @@ from switch_core.bridges.agent.dependencies import get_protocol, get_session
from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.bridges.agent.protocol.controller_presence import (
ControllerConnectionError,
+ ControllerRevokedError,
)
from switch_core.bridges.agent.protocol.controller_stream import (
IDLE,
@@ -351,19 +352,38 @@ async def open_connection(
body: ControllerConnectionRequest,
principal: PathController,
protocol: Protocol,
+ management: Management,
+ session: Session,
) -> dict[str, Any]:
"""Open the controller's connection, taking over any it had.
The agents bound to it are attached on the stream, each from its cursor
here; the response names them. Opening again is a takeover: the earlier
- connection's socket ends with `taken_over`, and its beats are refused.
+ connection's socket ends with `taken_over`, and its beats are refused,
+ whichever process held it. The connection is recorded before the answer,
+ so its socket can attach through any process.
"""
presence = protocol.connections.controllers
+ resume_cursors = body.resume_cursors()
+ if presence.is_revoked(principal.controller_id):
+ raise _connection_refusal(ControllerRevokedError(principal.controller_id))
+ # Recorded before it is held: a read of the row in between would find the
+ # connection this one replaces still named there, and close this one.
+ identity = presence.new_connection()
+ connection_id, generation = identity
+ await management.record_opened(
+ session,
+ principal,
+ connection_id=connection_id,
+ generation=generation,
+ resume_cursors=resume_cursors,
+ )
try:
conn = presence.open(
controller_id=principal.controller_id,
tenant_id=principal.tenant_id,
- resume_cursors=body.resume_cursors(),
+ resume_cursors=resume_cursors,
+ identity=identity,
)
except ControllerConnectionError as exc:
raise _connection_refusal(exc) from exc
@@ -401,9 +421,18 @@ async def attach_stream(
presence = protocol.connections.controllers
nudges = management.notifier.subscribe(principal.controller_id)
try:
- conn = presence.require(principal.controller_id, connection_id, generation)
async with tenant_session(session_factory, principal.tenant_id) as session:
+ # Wherever it was opened or last held: decided on the
+ # controller's row, which says whether it is still the one the
+ # controller opened last.
+ conn = await management.hold_connection(
+ session, principal, connection_id, generation
+ )
state = await management.connection_state(session, principal)
+ # Attached once nothing more is awaited, and refused if the connection
+ # stopped being the controller's current one while the reads above
+ # waited.
+ token = presence.attach_stream(conn)
except ControllerConnectionError as exc:
nudges.close()
raise _connection_refusal(exc) from exc
@@ -416,7 +445,6 @@ async def attach_stream(
"generation": conn.generation,
"heartbeat_interval_s": HEARTBEAT_INTERVAL_SECONDS,
}
- token = presence.attach_stream(conn)
return controller_event_stream(
conn=conn,
stream_token=token,
diff --git a/core/switch_core/management/placement.py b/core/switch_core/management/placement.py
index 553e1f873..2d1559584 100644
--- a/core/switch_core/management/placement.py
+++ b/core/switch_core/management/placement.py
@@ -2,7 +2,7 @@
A controller's state is read from its connection's recorded transitions
(`connection_ledger.py`) and the lease of the switch-core process holding it
-(`process_lease.py`), so every process answers alike and a restart changes
+(`switch_core/process_lease.py`), so every process answers alike and a restart changes
nothing:
- `revoked`: its credential is gone.
@@ -35,7 +35,7 @@ from typing import Any, Literal
from switch_core.db.models import AgentController
from switch_core.management import reason_codes
from switch_core.management.errors import ManagementError
-from switch_core.management.process_lease import ProcessLeases
+from switch_core.process_lease import ProcessLeases
ControllerState = Literal["online", "offline", "unknown", "revoked"]
diff --git a/core/switch_core/management/process_lease.py b/core/switch_core/management/process_lease.py
deleted file mode 100644
index 9e3d8192f..000000000
--- a/core/switch_core/management/process_lease.py
+++ /dev/null
@@ -1,131 +0,0 @@
-"""This switch-core process's lease, and reading every process's.
-
-A controller's socket lives in the memory of the one process holding it, and
-that process writes the connection's transitions to the controller's row
-(`connection_ledger.py`). A process that dies writes no closing, so each
-process also holds a lease in `switch_core_processes`: it claims it at
-startup, renews it every `RENEW_INTERVAL`, and marks it stopped as it shuts
-down. A connection reads as online only while its holding process's lease is
-held: renewed within `LEASE_TTL` and not stopped. One write per process per
-renewal, however many machines it holds.
-
-Leases are written and compared on the database's clock, never a host's.
-Leases not renewed for `PRUNE_AFTER` are deleted by whichever process renews
-next; a connection naming a process with no lease reads as lost.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import logging
-from dataclasses import dataclass
-from datetime import datetime, timedelta
-
-from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-
-from switch_core.db.stores.switch_core_process_store import (
- ProcessLeaseRow,
- SwitchCoreProcessStore,
-)
-
-logger = logging.getLogger(__name__)
-
-RENEW_INTERVAL_SECONDS = 5.0
-
-# Three renewals: one missed write, or a slow one, does not put every machine
-# the process holds offline.
-LEASE_TTL = timedelta(seconds=3 * RENEW_INTERVAL_SECONDS)
-
-PRUNE_AFTER = timedelta(hours=1)
-
-# Why a connection the row still shows open is not, when its process is the
-# reason: the process shut down without writing it, or stopped renewing.
-SERVER_SHUTDOWN = "server_shutdown"
-SERVER_LOST = "server_lost"
-
-
-@dataclass(frozen=True)
-class ProcessLeases:
- """Every process's lease as the database held it at `read_at`, its own
- clock."""
-
- read_at: datetime
- leases: dict[str, ProcessLeaseRow]
-
- def holds(self, process_id: str | None) -> bool:
- """Whether the process is alive and holding its sockets."""
- lease = self.leases.get(process_id) if process_id is not None else None
- return (
- lease is not None
- and lease.stopped_at is None
- and self.read_at - lease.beat_at < LEASE_TTL
- )
-
- def ended(self, process_id: str | None) -> tuple[datetime | None, str]:
- """When and why a process not holding stopped: shut down at
- `stopped_at`, or lost after its last renewal (when not known once its
- lease is pruned)."""
- lease = self.leases.get(process_id) if process_id is not None else None
- if lease is None:
- return None, SERVER_LOST
- if lease.stopped_at is not None:
- return lease.stopped_at, SERVER_SHUTDOWN
- return lease.beat_at, SERVER_LOST
-
-
-async def read_leases(
- session: AsyncSession, processes: SwitchCoreProcessStore
-) -> ProcessLeases:
- """Every lease, in the caller's session; the table has no tenant policy,
- so any session reads it."""
- read_at, leases = await processes.read(session)
- return ProcessLeases(read_at=read_at, leases=leases)
-
-
-class ProcessLease:
- """This process's lease. The sessions are opened with no tenant bound:
- the table carries none."""
-
- def __init__(
- self,
- *,
- process_id: str,
- session_factory: async_sessionmaker[AsyncSession],
- processes: SwitchCoreProcessStore,
- ) -> None:
- self.process_id = process_id
- self._session_factory = session_factory
- self._processes = processes
-
- async def renew(self) -> None:
- """Claim or renew the lease, and prune leases long dead."""
- async with self._session_factory() as session:
- await self._processes.renew(session, self.process_id)
- pruned = await self._processes.prune(session, PRUNE_AFTER)
- await session.commit()
- if pruned:
- logger.info("Pruned %d switch-core process lease(s) long stale", pruned)
-
- async def stop(self) -> None:
- """Mark the lease stopped: every connection this process still shows
- open reads as offline from now."""
- async with self._session_factory() as session:
- await self._processes.stop(session, self.process_id)
- await session.commit()
- logger.info("Released switch-core process lease %s", self.process_id)
-
- async def run(self) -> None:
- """Renew every `RENEW_INTERVAL_SECONDS` until cancelled. A failed
- renewal is logged and retried at the next; three in a row and this
- process's machines read offline elsewhere."""
- while True:
- await asyncio.sleep(RENEW_INTERVAL_SECONDS)
- try:
- await self.renew()
- except Exception:
- logger.exception(
- "Renewing switch-core process lease %s failed; the machines "
- "it holds read offline once it is %s old",
- self.process_id,
- LEASE_TTL,
- )
diff --git a/core/switch_core/management/schemas.py b/core/switch_core/management/schemas.py
index 8c711e19a..a77d31623 100644
--- a/core/switch_core/management/schemas.py
+++ b/core/switch_core/management/schemas.py
@@ -28,8 +28,8 @@ from switch_core.db.models import (
)
from switch_core.db.models import AgentDefinition as AgentDefinitionRow
from switch_core.management.advanced_config import validate_advanced_config
-from switch_core.management.process_lease import ProcessLeases
from switch_core.management.sealed_logins import PublicKey, key_id
+from switch_core.process_lease import ProcessLeases
from switch_core.providers.registry import AgentProviderId
Provider = AgentProviderId
diff --git a/core/switch_core/management/service.py b/core/switch_core/management/service.py
index 761d00c72..83a22a03b 100644
--- a/core/switch_core/management/service.py
+++ b/core/switch_core/management/service.py
@@ -40,9 +40,14 @@ from switch_core.bridges.agent.protocol.agent_core import AgentCore, AgentExists
from switch_core.bridges.agent.protocol.controller_presence import (
DETACH_DELETED,
DETACH_UNASSIGNED,
- Binding,
+ DISCONNECT_REVOKED,
+ ControllerConnection,
ControllerPresence,
+ ControllerRevokedError,
+ ControllerTakenOverError,
+ UnknownControllerConnectionError,
)
+from switch_core.bridges.agent.protocol.liveness import HEARTBEAT_LAPSED, TAKEN_OVER
from switch_core.config import ControllerRelease
from switch_core.db.models import (
CONTROLLER_ENROLLMENT_KEY_TYPE,
@@ -70,15 +75,14 @@ from switch_core.gateway.cloud_controllers import (
)
from switch_core.gateway.known_agents import KnownAgent, provider_known_agent
from switch_core.management import reason_codes, tokens
+from switch_core.management.bindings import binding_of
from switch_core.management.errors import ManagementError, not_found
-from switch_core.management.notifier import ControllerNotifier
from switch_core.management.placement import (
ControllerState,
controller_state,
is_revoked,
require_placement,
)
-from switch_core.management.process_lease import ProcessLeases, read_leases
from switch_core.management.schemas import (
ControllerDescription,
CreateManagedAgentRequest,
@@ -94,6 +98,8 @@ from switch_core.management.schemas import (
workspaces_dir_of,
)
from switch_core.management.sealed_logins import PublicKey, SealedLogin, key_id
+from switch_core.management.signals import ControllerSignals
+from switch_core.process_lease import ProcessLeases, read_leases
from switch_core.providers.registry import provider_ids
logger = logging.getLogger(__name__)
@@ -138,7 +144,7 @@ class ManagementService:
self,
*,
settings: ManagementSettings,
- notifier: ControllerNotifier,
+ signals: ControllerSignals,
controllers: AgentControllerStore,
definitions: AgentDefinitionStore,
operations: AgentControllerOperationStore,
@@ -149,7 +155,8 @@ class ManagementService:
clock: Callable[[], datetime],
) -> None:
self.settings = settings
- self.notifier = notifier
+ self.signals = signals
+ self.notifier = signals.notifier
self.presence = presence
self.controllers = controllers
self.definitions = definitions
@@ -350,7 +357,7 @@ class ManagementService:
self._announce_revoked(replaced)
for row in moved:
await self._bind(session, tenant_id, row)
- self._nudge(revisions)
+ self._nudge(tenant_id, revisions)
logger.info(
"Cloud machine %s enrolled again; revoked its previous controller %s "
"and moved its %d agent(s) to %s",
@@ -485,6 +492,127 @@ class ManagementService:
),
}
+ async def record_opened(
+ self,
+ session: AsyncSession,
+ principal: ControllerPrincipal,
+ *,
+ connection_id: str,
+ generation: int,
+ resume_cursors: dict[str, int | None],
+ ) -> None:
+ """Record the connection the controller is opening and tell every
+ other process: its socket may attach through any of them, and a
+ connection another process holds for this controller has been taken
+ over.
+
+ Committed before the connection is held here, so a read of the row
+ that lands in between finds the new connection named there, or finds
+ the earlier one while this process still holds only that one."""
+ await self.controllers.record_opened(
+ session,
+ principal.tenant_id,
+ principal.controller_id,
+ connection_id=connection_id,
+ generation=generation,
+ process_id=self.signals.process_id,
+ resume_cursors=resume_cursors,
+ )
+ await session.commit()
+ self.signals.connection_changed(principal.tenant_id, principal.controller_id)
+
+ async def hold_connection(
+ self,
+ session: AsyncSession,
+ principal: ControllerPrincipal,
+ connection_id: str,
+ generation: int,
+ ) -> ControllerConnection:
+ """The connection a socket attaching names, held here, or the refusal.
+
+ Decided on the controller's row rather than on this process's memory,
+ which may be a moment behind what another process has done: only a
+ connection older than the one the row names is refused as taken
+ over. The row names the connection the controller opened last and the
+ process holding it:
+
+ - this connection, held by this process: the copy here is the one
+ that counts, and none here means it ended here (its heartbeat
+ lapsed, say), which is final, as it always was;
+ - another connection: the controller has opened a newer one, so a
+ copy of this one still here is closed `taken_over`, and the attach
+ refused as this process answers for it;
+ - this connection, held by another process: it is taken up here,
+ replacing any copy this process kept from holding it before.
+ """
+ presence = self.presence
+ controller_id = principal.controller_id
+ controller = await self.controllers.get(
+ session, principal.tenant_id, controller_id
+ )
+ if controller is None:
+ raise UnknownControllerConnectionError(connection_id)
+ if is_revoked(controller):
+ raise ControllerRevokedError(controller_id)
+ if controller.open_connection_id != connection_id:
+ local = presence.current_connection(controller_id)
+ if local is not None and local.id == connection_id:
+ presence.supersede(controller_id)
+ return presence.require(controller_id, connection_id, generation)
+ if controller.open_process_id == self.signals.process_id:
+ return presence.require(controller_id, connection_id, generation)
+ return await self._take_up(
+ session, principal, controller, connection_id, generation
+ )
+
+ async def _take_up(
+ self,
+ session: AsyncSession,
+ principal: ControllerPrincipal,
+ controller: AgentController,
+ connection_id: str,
+ generation: int,
+ ) -> ControllerConnection:
+ """Hold here the connection the controller opened last, which another
+ process holds or held, unless something has ended it. Recorded before
+ it is held, as an open is.
+
+ Its stream resumes each agent from the cursors recorded with it, which
+ are the open's until a socket on it goes and its holder writes them.
+ A socket attached to it whose going was never written (its holder
+ died, or has not written it yet) may have sent past them, and the
+ beats confirming how far went with that holder: resuming there would
+ skip what it sent after, so the controller is told to open again,
+ with its own cursors."""
+ if controller.open_generation != generation or (
+ controller.connection_id == connection_id
+ and (
+ controller.disconnected_at is None
+ or controller.disconnect_reason in _ENDING_REASONS
+ )
+ ):
+ raise UnknownControllerConnectionError(connection_id)
+ resume_cursors = _resume_cursors(controller.open_cursors)
+ if not await self.controllers.record_taken_up(
+ session,
+ principal.tenant_id,
+ principal.controller_id,
+ connection_id=connection_id,
+ process_id=self.signals.process_id,
+ ):
+ # The controller opened another connection since the row was read.
+ raise ControllerTakenOverError(connection_id)
+ await session.commit()
+ conn = self.presence.adopt(
+ controller_id=principal.controller_id,
+ tenant_id=principal.tenant_id,
+ connection_id=connection_id,
+ generation=generation,
+ resume_cursors=resume_cursors,
+ )
+ self.signals.connection_changed(principal.tenant_id, principal.controller_id)
+ return conn
+
async def connection_state(
self, session: AsyncSession, principal: ControllerPrincipal
) -> dict[str, Any]:
@@ -732,6 +860,7 @@ class ManagementService:
await session.commit()
if "name" in changes:
self.presence.rename_controller(controller_id, controller.name)
+ self.signals.renamed(tenant_id, controller_id)
logger.info(
"Updated agent controller %s (%s)",
controller_id,
@@ -812,8 +941,7 @@ class ManagementService:
return moved
def _announce_revoked(self, controller_id: str) -> None:
- self.notifier.credential_revoked(controller_id)
- self.presence.revoke_controller(controller_id)
+ self.signals.revoked(controller_id)
# ── Provider logins sealed to a controller ────────────────────────────────
@@ -879,7 +1007,7 @@ class ManagementService:
provider,
row.revision,
)
- self.notifier.operation_pending(
+ self.signals.operation_pending(
controller_id,
operation_id=operation.id,
kind="provider.login",
@@ -917,7 +1045,7 @@ class ManagementService:
logger.info(
"Withdrew agent controller %s's sealed %s login", controller_id, provider
)
- self.notifier.operation_pending(
+ self.signals.operation_pending(
controller_id,
operation_id=operation.id,
kind="provider.recheck",
@@ -1082,9 +1210,9 @@ class ManagementService:
)
return revisions
- def _nudge(self, revisions: dict[str, int]) -> None:
+ def _nudge(self, tenant_id: str, revisions: dict[str, int]) -> None:
for controller_id, revision in revisions.items():
- self.notifier.assignment_changed(controller_id, revision)
+ self.signals.assignment_changed(tenant_id, controller_id, revision)
async def _bind(
self, session: AsyncSession, tenant_id: str, row: AgentDefinitionRow
@@ -1186,7 +1314,7 @@ class ManagementService:
)
await session.commit()
await self._bind(session, tenant_id, row)
- self._nudge(revisions)
+ self._nudge(tenant_id, revisions)
logger.info(
"Created managed agent %s on controller %s",
result.agent_id,
@@ -1338,7 +1466,7 @@ class ManagementService:
revisions = await self._bump_and_collect(session, tenant_id, affected)
await session.commit()
await self._bind(session, tenant_id, row)
- self._nudge(revisions)
+ self._nudge(tenant_id, revisions)
agent = await self._owned_agent(session, owner_id, agent.id)
return await self._view(
session, tenant_id, row, agent, {}, await self.leases(session)
@@ -1362,7 +1490,7 @@ class ManagementService:
)
await session.commit()
self.presence.unbind(agent_id, DETACH_UNASSIGNED)
- self._nudge(revisions)
+ self._nudge(tenant_id, revisions)
async def forget_deleted_agent(
self, session: AsyncSession, tenant_id: str, agent_id: str
@@ -1387,7 +1515,7 @@ class ManagementService:
)
await session.commit()
self.presence.unbind(agent_id, DETACH_DELETED)
- self._nudge(revisions)
+ self._nudge(tenant_id, revisions)
logger.info("Stopped managing agent %s, which is being deleted", agent_id)
# ── Operations, owner side ────────────────────────────────────────────────
@@ -1458,7 +1586,7 @@ class ManagementService:
)
view = operation_view(operation)
await session.commit()
- self.notifier.operation_pending(
+ self.signals.operation_pending(
controller_id, operation_id=operation.id, kind=kind, agent_id=agent_id
)
return view
@@ -1486,18 +1614,21 @@ class ManagementService:
return [operation_view(operation) for operation in operations]
-def binding_of(
- tenant_id: str, row: AgentDefinitionRow, controller: AgentController
-) -> Binding:
- """The binding Core keeps for a definition placed on `controller`."""
- assert row.controller_id == controller.id
- return Binding(
- agent_id=row.agent_id,
- controller_id=controller.id,
- tenant_id=tenant_id,
- controller_name=controller.name,
- running=row.desired_state == "running",
- )
+# Why a connection ended for good: a socket going for any of these leaves
+# nothing to reattach to, and the controller opens another.
+_ENDING_REASONS = frozenset(
+ {HEARTBEAT_LAPSED.code, TAKEN_OVER.code, DISCONNECT_REVOKED}
+)
+
+
+def _resume_cursors(recorded: dict[str, Any] | None) -> dict[str, int | None]:
+ """The cursors recorded with a connection, as the presence holds them."""
+ if not recorded:
+ return {}
+ return {
+ agent_id: cursor if isinstance(cursor, int) else None
+ for agent_id, cursor in recorded.items()
+ }
def placement_from(
diff --git a/core/switch_core/management/signals.py b/core/switch_core/management/signals.py
new file mode 100644
index 000000000..ceaec7138
--- /dev/null
+++ b/core/switch_core/management/signals.py
@@ -0,0 +1,440 @@
+"""Management's nudges and bindings, kept current in whichever process holds
+the controller.
+
+A change commits in the process that handled the request; the controller it
+affects may be streaming from any process, and every process answers that
+controller's agents' requests. So every change is acted on here at once (the
+local subscriptions and bindings, and on revocation the local presence) and
+published on the `SignalBus` for the others, which do the same on receipt.
+
+Nudges were already wake-ups, never the record: a controller woken pulls its
+assignment or its operations. A binding is a copy of the record, which is why
+a process that hears `controller.assignment_changed` or `controller.renamed`
+reads the controller's placed definitions again rather than trusting the
+signal (`bindings.reconcile_controller_bindings`). A move bumps the revision of
+both controllers, so both are re-read.
+
+Which process holds each controller's socket is kept the same way. A
+controller opening a connection, another process taking one up, and a socket
+attaching or going are announced as `controller.connection_changed`, and each
+process told reads the controller's row again (`_read_connection`): an agent
+run by a controller connected to any process then reads as live in every one,
+and a connection this process holds that another process has since replaced
+or taken up is closed `taken_over`.
+
+Nothing here depends on a signal arriving. When the signal connection comes
+back, every binding and every controller's connection is read again and every
+controller held here is re-nudged from the database, so one that missed a
+signal while the connection was down pulls again, and one revoked meanwhile is
+closed. The periodic re-check does the same reads, but nudges a held
+controller only with what it has not been told yet (a newer revision, an
+operation it has not heard of), so a quiet controller is not made to pull
+every `RECHECK_SECONDS`.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from datetime import datetime
+from typing import TYPE_CHECKING, Any
+
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.tenant_lookup import all_tenant_ids
+from switch_core.management.bindings import (
+ reconcile_bindings,
+ reconcile_controller_bindings,
+)
+from switch_core.process_lease import ProcessLeases, read_leases
+from switch_core.tenant_context import no_tenant
+
+if TYPE_CHECKING:
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+ from switch_core.bridges.agent.protocol.controller_presence import (
+ ControllerPresence,
+ )
+ from switch_core.db.models import AgentController
+ from switch_core.db.stores.agent_controller_operation_store import (
+ AgentControllerOperationStore,
+ )
+ from switch_core.db.stores.agent_controller_store import AgentControllerStore
+ from switch_core.db.stores.agent_definition_store import AgentDefinitionStore
+ from switch_core.db.stores.room_store import RoomStore
+ from switch_core.db.stores.switch_core_process_store import (
+ SwitchCoreProcessStore,
+ )
+ from switch_core.management.notifier import ControllerNotifier
+ from switch_core.signals import SignalBus
+
+logger = logging.getLogger(__name__)
+
+ASSIGNMENT_CHANGED_SIGNAL = "controller.assignment_changed"
+OPERATION_PENDING_SIGNAL = "controller.operation_pending"
+REVOKED_SIGNAL = "controller.revoked"
+RENAMED_SIGNAL = "controller.renamed"
+CONNECTION_CHANGED_SIGNAL = "controller.connection_changed"
+
+# How many times a controller's row is read again when this process changed
+# that controller's connection while it was being read, before it is left to
+# the next re-check.
+_READ_ATTEMPTS = 3
+
+
+class ControllerSignals:
+ def __init__(
+ self,
+ *,
+ notifier: ControllerNotifier,
+ presence: ControllerPresence,
+ bus: SignalBus,
+ session_factory: async_sessionmaker[AsyncSession],
+ controllers: AgentControllerStore,
+ definitions: AgentDefinitionStore,
+ operations: AgentControllerOperationStore,
+ rooms: RoomStore,
+ processes: SwitchCoreProcessStore,
+ clock: Callable[[], datetime],
+ ) -> None:
+ self.notifier = notifier
+ # The process this is, as signals and the rows naming a holder call it.
+ self.process_id = bus.process_id
+ self._presence = presence
+ self._bus = bus
+ self._session_factory = session_factory
+ self._controllers = controllers
+ self._definitions = definitions
+ self._operations = operations
+ self._rooms = rooms
+ self._processes = processes
+ self._clock = clock
+ # What each controller has been nudged with from this process, so the
+ # periodic re-check tells a held controller only what is new to it.
+ self._nudged_revision: dict[str, int] = {}
+ self._nudged_operations: dict[str, set[str]] = {}
+ # Controllers, by (tenant id, controller id), whose bindings another
+ # process changed and this one has still to read.
+ self._bindings_owed = bus.pending_work(
+ "controller bindings", self._reconcile_controller
+ )
+ # Controllers, by (tenant id, controller id), whose connection another
+ # process changed and this one has still to read.
+ self._connections_owed = bus.pending_work(
+ "controller connections", self._read_connection_of
+ )
+ bus.handle(ASSIGNMENT_CHANGED_SIGNAL, self._on_assignment_changed)
+ bus.handle(OPERATION_PENDING_SIGNAL, self._on_operation_pending)
+ bus.handle(REVOKED_SIGNAL, self._on_revoked)
+ bus.handle(RENAMED_SIGNAL, self._on_renamed)
+ bus.handle(CONNECTION_CHANGED_SIGNAL, self._on_connection_changed)
+ bus.on_resync(self.resync)
+ bus.on_recheck(self.recheck)
+
+ # ── Announcing, from the process that made the change ─────────────────────
+
+ def assignment_changed(
+ self, tenant_id: str, controller_id: str, revision: int
+ ) -> None:
+ """The process that committed the change has already bound its agents."""
+ self._nudge_assignment(controller_id, revision)
+ self._bus.publish(
+ ASSIGNMENT_CHANGED_SIGNAL,
+ tenant_id=tenant_id,
+ controller_id=controller_id,
+ revision=revision,
+ )
+
+ def operation_pending(
+ self, controller_id: str, *, operation_id: str, kind: str, agent_id: str | None
+ ) -> None:
+ self._nudge_operation(
+ controller_id, operation_id=operation_id, kind=kind, agent_id=agent_id
+ )
+ self._bus.publish(
+ OPERATION_PENDING_SIGNAL,
+ controller_id=controller_id,
+ operation_id=operation_id,
+ operation_kind=kind,
+ agent_id=agent_id,
+ )
+
+ def revoked(self, controller_id: str) -> None:
+ self._revoke_here(controller_id)
+ self._bus.publish(REVOKED_SIGNAL, controller_id=controller_id)
+
+ def renamed(self, tenant_id: str, controller_id: str) -> None:
+ """The controller's owner renamed it. A rename bumps no assignment
+ revision, so it needs a signal of its own for the other processes to
+ re-read the name their bindings carry."""
+ self._bus.publish(
+ RENAMED_SIGNAL, tenant_id=tenant_id, controller_id=controller_id
+ )
+
+ def connection_changed(self, tenant_id: str, controller_id: str) -> None:
+ """The controller's connection row changed (an open, a connection
+ taken up, a socket attaching or going), and this process already acts
+ on it."""
+ self._bus.publish(
+ CONNECTION_CHANGED_SIGNAL, tenant_id=tenant_id, controller_id=controller_id
+ )
+
+ # ── Receiving, in every other process ─────────────────────────────────────
+
+ def _on_assignment_changed(self, signal: dict[str, Any]) -> None:
+ # Nudged at once, as before bindings were re-read here: the
+ # controller's pull may land on any process, so waiting for this
+ # one's read would only delay it.
+ self._nudge_assignment(signal["controller_id"], signal["revision"])
+ self._bindings_owed.add((signal["tenant_id"], signal["controller_id"]))
+
+ def _on_operation_pending(self, signal: dict[str, Any]) -> None:
+ self._nudge_operation(
+ signal["controller_id"],
+ operation_id=signal["operation_id"],
+ kind=signal["operation_kind"],
+ agent_id=signal["agent_id"],
+ )
+
+ def _on_revoked(self, signal: dict[str, Any]) -> None:
+ self._revoke_here(signal["controller_id"])
+
+ def _on_renamed(self, signal: dict[str, Any]) -> None:
+ self._bindings_owed.add((signal["tenant_id"], signal["controller_id"]))
+
+ def _on_connection_changed(self, signal: dict[str, Any]) -> None:
+ self._connections_owed.add((signal["tenant_id"], signal["controller_id"]))
+
+ async def _reconcile_controller(self, key: tuple[str, str]) -> None:
+ tenant_id, controller_id = key
+ await reconcile_controller_bindings(
+ session_factory=self._session_factory,
+ definitions=self._definitions,
+ controllers=self._controllers,
+ presence=self._presence,
+ tenant_id=tenant_id,
+ controller_id=controller_id,
+ )
+ # An agent just bound to a controller another process holds needs its
+ # rooms here, as the controller's other agents have them, to be
+ # present in them at once: reading the connection again loads them.
+ if self._presence.holder_elsewhere(controller_id) is not None:
+ await self._read_connection(tenant_id, controller_id)
+
+ # ── Which process holds each controller's socket ──────────────────────────
+
+ async def _read_connection_of(self, key: tuple[str, str]) -> None:
+ tenant_id, controller_id = key
+ await self._read_connection(tenant_id, controller_id)
+
+ async def _read_connection(self, tenant_id: str, controller_id: str) -> None:
+ """Read the controller's row again and act on it here."""
+ for _ in range(_READ_ATTEMPTS):
+ mark = self._presence.connection_mark()
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ controller = await self._controllers.get(
+ session, tenant_id, controller_id
+ )
+ if controller is None:
+ return
+ leases = await read_leases(session, self._processes)
+ rooms = await self._rooms_if_held_elsewhere(session, controller, leases)
+ if self._presence.connection_changed_since(controller_id, mark):
+ # Changed here while it was read: what was read may be older.
+ continue
+ self._apply_connection(controller, leases, rooms)
+ return
+
+ async def read_connections(self) -> None:
+ """Read every controller's connection row again, in every tenant, and
+ make this process's view of who holds each socket agree. Run at
+ startup, before the bridge serves, and on resync and each re-check."""
+ seen: set[str] = set()
+ elsewhere = self._presence.controllers_elsewhere()
+ with no_tenant():
+ tenant_ids = await all_tenant_ids(self._session_factory)
+ for tenant_id in tenant_ids:
+ mark = self._presence.connection_mark()
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ leases = await read_leases(session, self._processes)
+ found = [
+ controller
+ for controller in await self._controllers.list_with_connections(
+ session, tenant_id
+ )
+ if controller.tenant_id == tenant_id
+ ]
+ rooms = {
+ controller.id: await self._rooms_if_held_elsewhere(
+ session, controller, leases
+ )
+ for controller in found
+ }
+ for controller in found:
+ seen.add(controller.id)
+ if self._presence.connection_changed_since(controller.id, mark):
+ # Changed here while it was read; read it again on its own.
+ await self._read_connection(tenant_id, controller.id)
+ continue
+ self._apply_connection(controller, leases, rooms[controller.id])
+ for controller_id in elsewhere - seen:
+ self._presence.held_elsewhere(controller_id, None)
+
+ async def _rooms_if_held_elsewhere(
+ self,
+ session: AsyncSession,
+ controller: AgentController,
+ leases: ProcessLeases,
+ ) -> dict[str, set[str]]:
+ """The rooms of each agent the controller runs, when another process
+ holds its socket: they are what makes those agents present in a room
+ here, and the stream that loads them runs over there."""
+ if self._held_elsewhere(controller, leases) is None:
+ return {}
+ return {
+ agent_id: {
+ room.id
+ for room in await self._rooms.get_rooms_for_agent(
+ session, agent_id, include_archived=True
+ )
+ }
+ for agent_id in sorted(self._presence.agents_of(controller.id))
+ }
+
+ def _held_elsewhere(
+ self, controller: AgentController, leases: ProcessLeases
+ ) -> str | None:
+ """The other process holding the controller's live socket, if any."""
+ process_id = controller.connection_process_id
+ if (
+ controller.connection_id is None
+ or controller.disconnected_at is not None
+ or process_id is None
+ or process_id == self.process_id
+ or not leases.holds(process_id)
+ ):
+ return None
+ return process_id
+
+ def _apply_connection(
+ self,
+ controller: AgentController,
+ leases: ProcessLeases,
+ rooms: dict[str, set[str]],
+ ) -> None:
+ local = self._presence.current_connection(controller.id)
+ if local is not None and controller.open_connection_id is not None:
+ # The row names the connection the controller opened last and the
+ # process holding it. A connection held here that is not that one,
+ # or is that one but taken up by another process, is over here.
+ if (
+ controller.open_connection_id != local.id
+ or controller.open_process_id != self.process_id
+ ):
+ self._presence.supersede(controller.id)
+ self._presence.held_elsewhere(
+ controller.id, self._held_elsewhere(controller, leases)
+ )
+ for agent_id, agent_rooms in rooms.items():
+ self._presence.set_rooms(agent_id, agent_rooms)
+
+ # ── Acting here ───────────────────────────────────────────────────────────
+
+ def _nudge_assignment(self, controller_id: str, revision: int) -> None:
+ if revision > self._nudged_revision.get(controller_id, -1):
+ self._nudged_revision[controller_id] = revision
+ self.notifier.assignment_changed(controller_id, revision)
+
+ def _nudge_operation(
+ self, controller_id: str, *, operation_id: str, kind: str, agent_id: str | None
+ ) -> None:
+ self._nudged_operations.setdefault(controller_id, set()).add(operation_id)
+ self.notifier.operation_pending(
+ controller_id, operation_id=operation_id, kind=kind, agent_id=agent_id
+ )
+
+ def _revoke_here(self, controller_id: str) -> None:
+ """The stream ends, the controller cannot open another here, and this
+ process's token cache forgets it."""
+ self.notifier.credential_revoked(controller_id)
+ self._presence.revoke_controller(controller_id)
+
+ # ── Resync and re-check ───────────────────────────────────────────────────
+
+ async def resync(self) -> None:
+ """Re-read every binding and every controller's connection, then
+ re-nudge every controller this process holds from what the database
+ says now: its assignment revision and its open operations, or its
+ revocation."""
+ await self._reconcile_all_bindings()
+ await self._read_all_connections()
+ await self._renudge_held(only_new=False)
+
+ async def recheck(self) -> None:
+ """What `resync` does, but a held controller is nudged only with what
+ this process has not told it yet."""
+ await self._reconcile_all_bindings()
+ await self._read_all_connections()
+ await self._renudge_held(only_new=True)
+
+ async def _read_all_connections(self) -> None:
+ try:
+ await self.read_connections()
+ except Exception:
+ logger.error("Could not re-read controller connections", exc_info=True)
+
+ async def _reconcile_all_bindings(self) -> None:
+ try:
+ await reconcile_bindings(
+ session_factory=self._session_factory,
+ definitions=self._definitions,
+ controllers=self._controllers,
+ presence=self._presence,
+ )
+ except Exception:
+ logger.error("Could not re-read controller bindings", exc_info=True)
+
+ async def _renudge_held(self, *, only_new: bool) -> None:
+ held = self._presence.held_connections()
+ # What a controller held elsewhere was told is no use here: if it
+ # connects to this process later, its opening frame carries the state.
+ held_ids = {conn.controller_id for conn in held}
+ for remembered in (self._nudged_revision, self._nudged_operations):
+ for controller_id in set(remembered) - held_ids:
+ del remembered[controller_id]
+ for conn in held:
+ try:
+ async with tenant_session(
+ self._session_factory, conn.tenant_id
+ ) as session:
+ controller = await self._controllers.get(
+ session, conn.tenant_id, conn.controller_id
+ )
+ if controller is None or controller.revoked_at is not None:
+ self._revoke_here(conn.controller_id)
+ continue
+ offered = await self._operations.list_offered(
+ session, conn.tenant_id, conn.controller_id, self._clock()
+ )
+ revision = controller.assignment_revision
+ if not only_new or revision > self._nudged_revision.get(
+ conn.controller_id, -1
+ ):
+ self._nudge_assignment(conn.controller_id, revision)
+ told = self._nudged_operations.get(conn.controller_id, set())
+ for operation in offered:
+ if only_new and operation.id in told:
+ continue
+ self._nudge_operation(
+ conn.controller_id,
+ operation_id=operation.id,
+ kind=operation.kind,
+ agent_id=operation.agent_id,
+ )
+ # Only the operations still offered need remembering.
+ self._nudged_operations[conn.controller_id] = {
+ operation.id for operation in offered
+ }
+ except Exception:
+ logger.error(
+ "Could not resync controller %s", conn.controller_id, exc_info=True
+ )
diff --git a/core/switch_core/management/wiring.py b/core/switch_core/management/wiring.py
index c7fb114d1..5c6079a59 100644
--- a/core/switch_core/management/wiring.py
+++ b/core/switch_core/management/wiring.py
@@ -9,17 +9,17 @@ serves. `install` also hands Core the agent-facing side
(`ManagementAgentOperations`), which is what makes the agent operations on
machines and managed agents exist.
-Each process also holds a lease (`process_lease.py`) and writes its
-controllers' connection transitions (`connection_ledger.py`): `start` claims
-the lease before the bridge serves, `run` renews it and writes transitions
-until cancelled, and `stop` writes what is left and marks the lease stopped.
+Each process writes its controllers' connection transitions
+(`connection_ledger.py`), read against the lease the server holds for the
+process whether or not management is on (`switch_core/process_lease.py`): `run` writes
+transitions until cancelled, and `flush` writes what is left as the process
+drains, before the server marks its lease stopped. `load_connections` reads
+which controllers the other processes hold, before the bridge serves.
"""
from __future__ import annotations
-import asyncio
import logging
-import uuid
from collections.abc import Callable
from dataclasses import dataclass
from datetime import UTC, datetime
@@ -42,6 +42,7 @@ from switch_core.db.stores.agent_controller_store import AgentControllerStore
from switch_core.db.stores.agent_definition_store import AgentDefinitionStore
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.api_key_store import ApiKeyStore
+from switch_core.db.stores.room_store import RoomStore
from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
from switch_core.gateway.cloud_controllers import set_cloud_enrollment
from switch_core.management.agent_operations import ManagementAgentOperations
@@ -52,8 +53,10 @@ from switch_core.management.controller_routes import router as controller_router
from switch_core.management.dependencies import init_management_dependencies
from switch_core.management.gateway_routes import router as gateway_router
from switch_core.management.notifier import ControllerNotifier
-from switch_core.management.process_lease import ProcessLease
from switch_core.management.service import ManagementService, ManagementSettings
+from switch_core.management.signals import ControllerSignals
+from switch_core.process_lease import ProcessLease
+from switch_core.signals import SignalBus
logger = logging.getLogger(__name__)
@@ -94,25 +97,22 @@ class Management:
async with tenant_session(self.session_factory, tenant_id) as session:
await self.service.forget_deleted_agent(session, tenant_id, agent_id)
- async def start(self) -> None:
- """Claim this process's lease, so the connections it records read as
- held. Raises when the database refuses: a process that cannot hold a
- lease would show every machine it connects as offline."""
- await self.lease.renew()
- logger.info("Claimed switch-core process lease %s", self.lease.process_id)
-
async def run(self) -> None:
- """Renew the lease and write connection transitions, until cancelled."""
- await asyncio.gather(self.lease.run(), self.ledger.run())
-
- async def stop(self) -> None:
- """Write the transitions still queued, then mark the lease stopped.
- The lease is marked stopped even when the writes fail, so the
- machines this process held read offline either way."""
- try:
- await self.ledger.flush_all()
- finally:
- await self.lease.stop()
+ """Write connection transitions as they happen, until cancelled. The
+ process lease they are read against is the server's to renew."""
+ await self.ledger.run()
+
+ async def flush(self) -> None:
+ """Write the transitions still queued. Called as the process drains,
+ before its lease is marked stopped, so the machines it held read as
+ closed by the server rather than lost with it."""
+ await self.ledger.flush_all()
+
+ async def load_connections(self) -> None:
+ """Read which controllers' sockets the other processes hold, before
+ the bridge serves: until then every agent they run reads offline
+ here. The signal bus keeps it current from then on."""
+ await self.service.signals.read_connections()
async def load_bindings(self) -> int:
return await load_bindings(
@@ -134,16 +134,38 @@ def build_management(
presence: ControllerPresence,
auth_cache: ControllerAuthCache,
clock: Callable[[], datetime],
- process_id: str,
+ signal_bus: SignalBus,
+ lease: ProcessLease,
) -> Management:
+ process_id = lease.process_id
+ if signal_bus.process_id != process_id:
+ raise ValueError(
+ f"The signal bus speaks for process {signal_bus.process_id!r} and "
+ f"the lease for {process_id!r}; one process has one name."
+ )
controllers = AgentControllerStore()
+ definitions = AgentDefinitionStore()
+ operations = AgentControllerOperationStore()
processes = SwitchCoreProcessStore()
presence.use_auth_cache(auth_cache)
+ signals = ControllerSignals(
+ notifier=ControllerNotifier(),
+ presence=presence,
+ bus=signal_bus,
+ session_factory=session_factory,
+ controllers=controllers,
+ operations=operations,
+ definitions=definitions,
+ rooms=RoomStore(),
+ processes=processes,
+ clock=clock,
+ )
ledger = ControllerConnectionLedger(
process_id=process_id,
session_factory=session_factory,
controllers=controllers,
clock=clock,
+ announce=signals.connection_changed,
)
presence.use_ledger(ledger)
service = ManagementService(
@@ -154,10 +176,10 @@ def build_management(
third_party_avatars=third_party_avatars,
controller_release=controller_release,
),
- notifier=ControllerNotifier(),
+ signals=signals,
controllers=controllers,
- definitions=AgentDefinitionStore(),
- operations=AgentControllerOperationStore(),
+ definitions=definitions,
+ operations=operations,
api_keys=ApiKeyStore(),
agents=AgentStore(),
processes=processes,
@@ -180,11 +202,7 @@ def build_management(
service=service, session_factory=session_factory
),
session_factory=session_factory,
- lease=ProcessLease(
- process_id=process_id,
- session_factory=session_factory,
- processes=processes,
- ),
+ lease=lease,
ledger=ledger,
)
@@ -193,6 +211,8 @@ def create_management(
config: SwitchConfig,
session_factory: async_sessionmaker[AsyncSession],
presence: ControllerPresence,
+ signal_bus: SignalBus,
+ lease: ProcessLease,
) -> Management | None:
if not config.agent_management_enabled:
return None
@@ -217,5 +237,6 @@ def create_management(
max_entries=config.agent_auth_cache_max_entries,
),
clock=utc_now,
- process_id=str(uuid.uuid4()),
+ signal_bus=signal_bus,
+ lease=lease,
)
diff --git a/core/switch_core/migrations/env.py b/core/switch_core/migrations/env.py
index eea50e3cb..31cbf576a 100644
--- a/core/switch_core/migrations/env.py
+++ b/core/switch_core/migrations/env.py
@@ -11,6 +11,7 @@ from switch_core.db import (
)
from switch_core.db.base import Base
from switch_core.db.engine import migration_connect_args
+from switch_core.db.schema_version import record_expand_only
from switch_core.logging_config import logging_is_configured
config = context.config
@@ -49,7 +50,11 @@ def run_migrations_offline() -> None:
def do_run_migrations(connection): # type: ignore[no-untyped-def]
- context.configure(connection=connection, target_metadata=target_metadata)
+ context.configure(
+ connection=connection,
+ target_metadata=target_metadata,
+ on_version_apply=[record_expand_only],
+ )
with context.begin_transaction():
context.run_migrations()
diff --git a/core/switch_core/migrations/script.py.mako b/core/switch_core/migrations/script.py.mako
index fbc4b07dc..b825cea98 100644
--- a/core/switch_core/migrations/script.py.mako
+++ b/core/switch_core/migrations/script.py.mako
@@ -16,6 +16,14 @@ revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
+# True only when a switch-core release from before this revision can go on
+# serving on the schema it leaves: it adds tables, nullable or defaulted
+# columns, or indexes, and drops, renames or tightens nothing an older release
+# reads or writes. During a rolling deploy both releases serve at once, and an
+# older one restarted after this ran accepts it only if this is True
+# (`db/schema_version.py`). A contracting change ships as its own revision, a
+# release after the code stopped using what it removes.
+expand_only: bool = False
def upgrade() -> None:
diff --git a/core/switch_core/migrations/versions/1feaa7700723_claim_outbound_posts.py b/core/switch_core/migrations/versions/1feaa7700723_claim_outbound_posts.py
new file mode 100644
index 000000000..8ce51800f
--- /dev/null
+++ b/core/switch_core/migrations/versions/1feaa7700723_claim_outbound_posts.py
@@ -0,0 +1,41 @@
+"""claim a room message before posting it to a platform
+
+`bridge_message_map.external_post_id` may be null: a bridge relaying a room
+message out writes the row first, as a claim, and fills the platform's post id
+in once the post is made. A second switch-core process handling the same room
+message finds the claim and does not post it again, which is what keeps a
+bridge's hand-over between processes from posting a message twice. Existing
+rows all carry a post id and are unchanged.
+
+Revision ID: 1feaa7700723
+Revises: 340ead685ad6
+Create Date: 2026-10-14 13:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "1feaa7700723"
+down_revision: str | None = "340ead685ad6"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+ op.alter_column(
+ "bridge_message_map", "external_post_id", existing_type=sa.Text(), nullable=True
+ )
+
+
+def downgrade() -> None:
+ # Claims never turned into posts have nothing to keep.
+ op.execute("DELETE FROM bridge_message_map WHERE external_post_id IS NULL")
+ op.alter_column(
+ "bridge_message_map",
+ "external_post_id",
+ existing_type=sa.Text(),
+ nullable=False,
+ )
diff --git a/core/switch_core/migrations/versions/340ead685ad6_ownership_leases_and_delivery_cursors.py b/core/switch_core/migrations/versions/340ead685ad6_ownership_leases_and_delivery_cursors.py
new file mode 100644
index 000000000..f255ad2fa
--- /dev/null
+++ b/core/switch_core/migrations/versions/340ead685ad6_ownership_leases_and_delivery_cursors.py
@@ -0,0 +1,164 @@
+"""ownership leases, and a delivery cursor per membership
+
+`ownership_leases` records which switch-core process runs each piece of work
+that must run once when several processes share the database: a client's
+consumer, a collaboration bridge's platform connection, a server-side
+connector. A holder counts only while its own lease in
+`switch_core_processes` holds, and its row goes back to nobody when that
+lease is pruned. A global table: no tenant column and no row-level security
+policy (`db/rls_ddl.py`'s `GLOBAL_TABLES`).
+
+`delivery_cursors` was keyed by agent and never written. It becomes how far
+each client's consumer has handled each room it is a member of, keyed by the
+membership and deleted with it, so the process that owns a client resumes
+where the last owner stopped. `client_rooms` gains the unique key the cursor
+refers to. Every existing membership starts at its room's newest row, which
+is where a consumer started reading before this.
+
+Revision ID: 340ead685ad6
+Revises: be940bbaeaec
+Create Date: 2026-10-14 12:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "340ead685ad6"
+down_revision: str | None = "be940bbaeaec"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+_POLICY = (
+ "CREATE POLICY tenant_isolation ON delivery_cursors FOR ALL "
+ "USING (tenant_id = (SELECT require_tenant_id())) "
+ "WITH CHECK (tenant_id = (SELECT require_tenant_id()))"
+)
+
+# Where every existing membership's consumer starts: the newest row of its
+# room, or 0 for a room with none. Reconstructed history is numbered below
+# zero and is never delivered.
+_SEED_CURSORS = """
+INSERT INTO delivery_cursors (tenant_id, client_id, room_id, last_seq)
+SELECT m.tenant_id, m.client_id, m.room_id,
+ GREATEST(COALESCE(
+ (SELECT max(seq) FROM messages WHERE messages.room_id = m.room_id), 0
+ ), 0)
+FROM client_rooms m
+"""
+
+
+def upgrade() -> None:
+ op.create_table(
+ "ownership_leases",
+ sa.Column("kind", sa.Text(), nullable=False),
+ sa.Column("key", sa.Text(), nullable=False),
+ sa.Column("process_id", sa.Text(), nullable=True),
+ sa.Column("token", sa.BigInteger(), nullable=False),
+ sa.Column(
+ "changed_at",
+ sa.DateTime(timezone=True),
+ server_default=sa.func.now(),
+ nullable=False,
+ ),
+ sa.ForeignKeyConstraint(
+ ["process_id"],
+ ["switch_core_processes.id"],
+ name="fk_ownership_leases_process",
+ ondelete="SET NULL",
+ ),
+ sa.PrimaryKeyConstraint("kind", "key"),
+ )
+ op.create_index(
+ "ix_ownership_leases_process_id", "ownership_leases", ["process_id"]
+ )
+
+ op.create_unique_constraint(
+ "uq_client_rooms_tenant_client_room",
+ "client_rooms",
+ ["tenant_id", "client_id", "room_id"],
+ )
+
+ op.drop_table("delivery_cursors")
+ op.create_table(
+ "delivery_cursors",
+ sa.Column(
+ "tenant_id",
+ sa.Text(),
+ sa.ForeignKey("tenants.id", name="fk_delivery_cursors_tenant"),
+ nullable=False,
+ ),
+ sa.Column("client_id", sa.Text(), nullable=False),
+ sa.Column("room_id", sa.Text(), nullable=False),
+ sa.Column("last_seq", sa.BigInteger(), nullable=False),
+ sa.Column(
+ "updated_at",
+ sa.DateTime(timezone=True),
+ server_default=sa.func.now(),
+ nullable=False,
+ ),
+ sa.ForeignKeyConstraint(
+ ["tenant_id", "client_id", "room_id"],
+ [
+ "client_rooms.tenant_id",
+ "client_rooms.client_id",
+ "client_rooms.room_id",
+ ],
+ name="fk_delivery_cursors_membership",
+ ondelete="CASCADE",
+ ),
+ sa.PrimaryKeyConstraint("client_id", "room_id"),
+ )
+ op.create_index("ix_delivery_cursors_tenant_id", "delivery_cursors", ["tenant_id"])
+ op.execute("ALTER TABLE delivery_cursors ENABLE ROW LEVEL SECURITY")
+ op.execute(_POLICY)
+ op.execute(_SEED_CURSORS)
+
+
+def downgrade() -> None:
+ op.drop_table("delivery_cursors")
+ op.create_table(
+ "delivery_cursors",
+ sa.Column(
+ "tenant_id",
+ sa.Text(),
+ sa.ForeignKey("tenants.id", name="fk_delivery_cursors_tenant"),
+ nullable=False,
+ ),
+ sa.Column("id", sa.Text(), nullable=False),
+ sa.Column("agent_id", sa.Text(), nullable=False),
+ sa.Column("room_id", sa.Text(), nullable=False),
+ sa.Column("last_seq", sa.BigInteger(), nullable=False),
+ sa.Column(
+ "updated_at",
+ sa.DateTime(timezone=True),
+ server_default=sa.func.now(),
+ nullable=False,
+ ),
+ sa.ForeignKeyConstraint(
+ ["tenant_id", "agent_id"],
+ ["agents.tenant_id", "agents.id"],
+ name="fk_delivery_cursors_agent",
+ ondelete="CASCADE",
+ ),
+ sa.ForeignKeyConstraint(
+ ["tenant_id", "room_id"],
+ ["rooms.tenant_id", "rooms.id"],
+ name="fk_delivery_cursors_room",
+ ondelete="CASCADE",
+ ),
+ sa.PrimaryKeyConstraint("id"),
+ sa.UniqueConstraint(
+ "agent_id", "room_id", name="uq_delivery_cursors_agent_room"
+ ),
+ )
+ op.create_index("ix_delivery_cursors_tenant_id", "delivery_cursors", ["tenant_id"])
+ op.execute("ALTER TABLE delivery_cursors ENABLE ROW LEVEL SECURITY")
+ op.execute(_POLICY)
+ op.drop_constraint(
+ "uq_client_rooms_tenant_client_room", "client_rooms", type_="unique"
+ )
+ op.drop_index("ix_ownership_leases_process_id", table_name="ownership_leases")
+ op.drop_table("ownership_leases")
diff --git a/core/switch_core/migrations/versions/5f97c7c7407a_claim_proposals_in_the_database.py b/core/switch_core/migrations/versions/5f97c7c7407a_claim_proposals_in_the_database.py
new file mode 100644
index 000000000..94ffcedc0
--- /dev/null
+++ b/core/switch_core/migrations/versions/5f97c7c7407a_claim_proposals_in_the_database.py
@@ -0,0 +1,38 @@
+"""remember which chat a messaging install claim was proposed in
+
+`messaging_install_states.proposed_chat_id` names the chat a claim-based
+install (Telegram) was put to when its state was redeemed. Only an answer from
+that chat may decide the state. It was held in the memory of the process that
+posted the proposal, so an admin's Connect reaching another switch-core process
+was refused as expired. Existing rows have nothing proposed.
+
+Expand only: a process from before this revision holds its proposals in
+memory and never reads the column.
+
+Revision ID: 5f97c7c7407a
+Revises: 1feaa7700723
+Create Date: 2026-10-14 18:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "5f97c7c7407a"
+down_revision: str | None = "1feaa7700723"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+expand_only: bool = True
+
+
+def upgrade() -> None:
+ op.add_column(
+ "messaging_install_states",
+ sa.Column("proposed_chat_id", sa.Text(), nullable=True),
+ )
+
+
+def downgrade() -> None:
+ op.drop_column("messaging_install_states", "proposed_chat_id")
diff --git a/core/switch_core/migrations/versions/87ebf9517271_pending_attachment_groups.py b/core/switch_core/migrations/versions/87ebf9517271_pending_attachment_groups.py
new file mode 100644
index 000000000..f49d03b43
--- /dev/null
+++ b/core/switch_core/migrations/versions/87ebf9517271_pending_attachment_groups.py
@@ -0,0 +1,66 @@
+"""pending_attachment_groups: multi-file posts the router has not decided yet
+
+The router decides, as a room row is written, which agents it is for, and
+stores their own events in the same transaction. A post of several files is
+written as one row per file and can only be decided once its files are rows,
+so a file that does not complete its post records the post here. The file
+that completes it deletes the row and stores the post's own events; a post
+whose files stop arriving is decided by a sweep every process runs once
+`deadline` has passed, claiming the row with `FOR UPDATE SKIP LOCKED`.
+
+Expand only: a process from before this revision assembles posts in its agent
+consumers' memory and never reads this table.
+
+Revision ID: 87ebf9517271
+Revises: c8f2a6d4e1b9
+Create Date: 2026-10-14 18:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "87ebf9517271"
+down_revision: str | None = "c8f2a6d4e1b9"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+expand_only: bool = True
+
+
+def upgrade() -> None:
+ op.create_table(
+ "pending_attachment_groups",
+ sa.Column(
+ "tenant_id",
+ sa.Text(),
+ sa.ForeignKey("tenants.id", name="fk_pending_attachment_groups_tenant"),
+ nullable=False,
+ ),
+ sa.Column("room_id", sa.Text(), nullable=False),
+ sa.Column("group_id", sa.Text(), nullable=False),
+ sa.Column("total", sa.Integer(), nullable=False),
+ sa.Column("from_seq", sa.BigInteger(), nullable=False),
+ sa.Column("deadline", sa.DateTime(timezone=True), nullable=False),
+ sa.PrimaryKeyConstraint("room_id", "group_id"),
+ sa.ForeignKeyConstraint(
+ ["tenant_id", "room_id"],
+ ["rooms.tenant_id", "rooms.id"],
+ name="fk_pending_attachment_groups_room",
+ ondelete="CASCADE",
+ ),
+ )
+ op.create_index(
+ "ix_pending_attachment_groups_deadline",
+ "pending_attachment_groups",
+ ["tenant_id", "deadline"],
+ )
+ op.execute("ALTER TABLE pending_attachment_groups ENABLE ROW LEVEL SECURITY")
+ op.execute(
+ "CREATE POLICY tenant_isolation ON pending_attachment_groups FOR ALL USING (tenant_id = (SELECT require_tenant_id())) WITH CHECK (tenant_id = (SELECT require_tenant_id()))"
+ )
+
+
+def downgrade() -> None:
+ op.drop_table("pending_attachment_groups")
diff --git a/core/switch_core/migrations/versions/b3e9d4a1c7f2_agent_connections_in_postgres.py b/core/switch_core/migrations/versions/b3e9d4a1c7f2_agent_connections_in_postgres.py
new file mode 100644
index 000000000..edfc27c6e
--- /dev/null
+++ b/core/switch_core/migrations/versions/b3e9d4a1c7f2_agent_connections_in_postgres.py
@@ -0,0 +1,193 @@
+"""Agent connections in Postgres, so every switch-core process can see them
+
+Revision ID: b3e9d4a1c7f2
+Revises: e5956cbd3bdc
+Create Date: 2026-10-14
+
+An agent's connection (its socket, heartbeat and the frames it is owed) stays
+in the memory of the process holding it. What the other processes need to
+know about it is written here, on transitions only and never per heartbeat:
+
+- `agent_connections`: one row per connection, naming the process that holds
+ it and its current incarnation, written when it opens or is reattached and
+ when it closes. It reads as live only while the holding process's lease in
+ `switch_core_processes` is held.
+- `agent_connection_rooms`: the rooms claimed on each connection. The primary
+ key keeps at most one claimant per agent per room.
+- `agent_session_placements`: which room each of an agent's sessions
+ connected to, and over which connection.
+- `agent_session_command_relays`: a session command waiting for the process
+ holding the stream it must be written to (the agent's watcher, or its
+ controller's).
+- `agent_connection_incarnation`: the sequence incarnations are drawn from,
+ above 2**33 so no number matches one a client holds from the per-process
+ counters used before, and below 2**53 so a JavaScript client holds it
+ exactly.
+
+Expand only: a process from before this revision keeps its agent connections
+in memory and never reads these tables.
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects import postgresql
+
+revision: str = "b3e9d4a1c7f2"
+down_revision: str | None = "e5956cbd3bdc"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+expand_only: bool = True
+
+_POLICY = (
+ "CREATE POLICY tenant_isolation ON {table} FOR ALL "
+ "USING (tenant_id = (SELECT require_tenant_id())) "
+ "WITH CHECK (tenant_id = (SELECT require_tenant_id()))"
+)
+
+_TABLES = (
+ "agent_connections",
+ "agent_connection_rooms",
+ "agent_session_placements",
+ "agent_session_command_relays",
+)
+
+
+def _tenant(table: str) -> sa.Column: # type: ignore[type-arg]
+ return sa.Column(
+ "tenant_id",
+ sa.Text(),
+ sa.ForeignKey("tenants.id", name=f"fk_{table}_tenant"),
+ nullable=False,
+ )
+
+
+def _agent_fk(table: str) -> sa.ForeignKeyConstraint:
+ return sa.ForeignKeyConstraint(
+ ["tenant_id", "agent_id"],
+ ["agents.tenant_id", "agents.id"],
+ name=f"fk_{table}_agent",
+ ondelete="CASCADE",
+ )
+
+
+def _room_fk(table: str) -> sa.ForeignKeyConstraint:
+ return sa.ForeignKeyConstraint(
+ ["tenant_id", "room_id"],
+ ["rooms.tenant_id", "rooms.id"],
+ name=f"fk_{table}_room",
+ ondelete="CASCADE",
+ )
+
+
+def _connection_fk(table: str) -> sa.ForeignKeyConstraint:
+ return sa.ForeignKeyConstraint(
+ ["tenant_id", "connection_id"],
+ ["agent_connections.tenant_id", "agent_connections.id"],
+ name=f"fk_{table}_connection",
+ ondelete="CASCADE",
+ )
+
+
+def upgrade() -> None:
+ op.execute(
+ "CREATE SEQUENCE agent_connection_incarnation"
+ " START WITH 8589934592"
+ " MINVALUE 8589934592"
+ " MAXVALUE 9007199254740991"
+ " NO CYCLE"
+ )
+ op.create_table(
+ "agent_connections",
+ _tenant("agent_connections"),
+ sa.Column("id", sa.Text(), nullable=False),
+ sa.Column("agent_id", sa.Text(), nullable=False),
+ sa.Column("process_id", sa.Text(), nullable=False),
+ sa.Column("incarnation", sa.BigInteger(), nullable=False),
+ sa.Column("scope", sa.Text(), nullable=False),
+ sa.Column("delivery_filter", sa.Text(), nullable=False),
+ sa.Column("spawn_capable", sa.Boolean(), nullable=False),
+ sa.Column("transport", sa.Text(), nullable=False),
+ sa.Column("speaks", sa.Integer(), nullable=True),
+ sa.Column("accepts", sa.Integer(), nullable=True),
+ sa.Column(
+ "attached_at",
+ sa.DateTime(timezone=True),
+ server_default=sa.func.now(),
+ nullable=False,
+ ),
+ sa.Column("closed_at", sa.DateTime(timezone=True), nullable=True),
+ sa.Column("close_reason", sa.Text(), nullable=True),
+ sa.PrimaryKeyConstraint("id"),
+ sa.UniqueConstraint("id", "tenant_id", name="uq_agent_connections_id_tenant"),
+ _agent_fk("agent_connections"),
+ )
+ op.create_index("ix_agent_connections_agent_id", "agent_connections", ["agent_id"])
+ op.create_table(
+ "agent_connection_rooms",
+ _tenant("agent_connection_rooms"),
+ sa.Column("agent_id", sa.Text(), nullable=False),
+ sa.Column("room_id", sa.Text(), nullable=False),
+ sa.Column("connection_id", sa.Text(), nullable=False),
+ sa.PrimaryKeyConstraint("agent_id", "room_id"),
+ _agent_fk("agent_connection_rooms"),
+ _room_fk("agent_connection_rooms"),
+ _connection_fk("agent_connection_rooms"),
+ )
+ op.create_index(
+ "ix_agent_connection_rooms_connection_id",
+ "agent_connection_rooms",
+ ["connection_id"],
+ )
+ op.create_table(
+ "agent_session_placements",
+ _tenant("agent_session_placements"),
+ sa.Column("agent_id", sa.Text(), nullable=False),
+ sa.Column("session_id", sa.Text(), nullable=False),
+ sa.Column("room_id", sa.Text(), nullable=False),
+ sa.Column("connection_id", sa.Text(), nullable=False),
+ sa.PrimaryKeyConstraint("agent_id", "session_id"),
+ sa.UniqueConstraint(
+ "agent_id", "room_id", name="uq_agent_session_placements_agent_room"
+ ),
+ _agent_fk("agent_session_placements"),
+ _room_fk("agent_session_placements"),
+ _connection_fk("agent_session_placements"),
+ )
+ op.create_index(
+ "ix_agent_session_placements_connection_id",
+ "agent_session_placements",
+ ["connection_id"],
+ )
+ op.create_table(
+ "agent_session_command_relays",
+ _tenant("agent_session_command_relays"),
+ sa.Column("id", sa.Text(), nullable=False),
+ sa.Column("agent_id", sa.Text(), nullable=False),
+ sa.Column("process_id", sa.Text(), nullable=False),
+ sa.Column("frame", postgresql.JSONB(), nullable=False),
+ sa.Column(
+ "created_at",
+ sa.DateTime(timezone=True),
+ server_default=sa.func.now(),
+ nullable=False,
+ ),
+ sa.PrimaryKeyConstraint("id"),
+ _agent_fk("agent_session_command_relays"),
+ )
+ op.create_index(
+ "ix_agent_session_command_relays_process_id",
+ "agent_session_command_relays",
+ ["process_id"],
+ )
+ for table in _TABLES:
+ op.create_index(f"ix_{table}_tenant_id", table, ["tenant_id"])
+ op.execute(f"ALTER TABLE {table} ENABLE ROW LEVEL SECURITY")
+ op.execute(_POLICY.format(table=table))
+
+
+def downgrade() -> None:
+ for table in reversed(_TABLES):
+ op.drop_table(table)
+ op.execute("DROP SEQUENCE IF EXISTS agent_connection_incarnation")
diff --git a/core/switch_core/migrations/versions/be940bbaeaec_index_client_rooms_room_id.py b/core/switch_core/migrations/versions/be940bbaeaec_index_client_rooms_room_id.py
new file mode 100644
index 000000000..07aac1d32
--- /dev/null
+++ b/core/switch_core/migrations/versions/be940bbaeaec_index_client_rooms_room_id.py
@@ -0,0 +1,42 @@
+"""index client_rooms on room_id for the router
+
+The router reads a room's agent members on every row it writes to that room
+(`RoomReads.load`), and once more under the room's lock to find an agent
+that joined meanwhile. The primary key is (client_id, room_id), whose leading
+column is the client, so neither read could use it and each was a sequential
+scan of `client_rooms`.
+
+Not `CONCURRENTLY`, for the reason `b8f2d0c41e57` gives: revisions run in
+a transaction, and the table is small on every deployment today. A deployment
+where it is not can build the index by hand first, and this becomes a no-op:
+
+ CREATE INDEX CONCURRENTLY ix_client_rooms_room_id ON client_rooms (room_id);
+
+Expand only: a process from before this revision reads the table as it did.
+
+Revision ID: be940bbaeaec
+Revises: 87ebf9517271
+Create Date: 2026-10-15 00:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+from alembic import op
+
+revision: str = "be940bbaeaec"
+down_revision: str | None = "87ebf9517271"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+expand_only: bool = True
+
+_INDEX = "ix_client_rooms_room_id"
+
+
+def upgrade() -> None:
+ # `IF NOT EXISTS` so an index built by hand ahead of the upgrade is kept.
+ op.execute(f"CREATE INDEX IF NOT EXISTS {_INDEX} ON client_rooms (room_id)")
+
+
+def downgrade() -> None:
+ op.execute(f"DROP INDEX IF EXISTS {_INDEX}")
diff --git a/core/switch_core/migrations/versions/c8f2a6d4e1b9_controller_open_connection.py b/core/switch_core/migrations/versions/c8f2a6d4e1b9_controller_open_connection.py
new file mode 100644
index 000000000..544f945b9
--- /dev/null
+++ b/core/switch_core/migrations/versions/c8f2a6d4e1b9_controller_open_connection.py
@@ -0,0 +1,53 @@
+"""The controller connection last opened, so any process can hold it
+
+Revision ID: c8f2a6d4e1b9
+Revises: b3e9d4a1c7f2
+Create Date: 2026-10-14
+
+A controller opens its connection with one request and attaches its socket
+with another, and a socket that drops reattaches with a third. With more than
+one switch-core process these may reach different processes, so the
+connection the controller last opened is recorded on its row: its id, its
+generation, the process holding it and where its stream resumes each agent.
+The process the socket reaches reads them, holds the connection itself and
+records that it does, and any other process holding a copy closes it.
+
+Expand only: the columns are nullable, and a process from before this
+revision neither reads nor writes them.
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects import postgresql
+
+revision: str = "c8f2a6d4e1b9"
+down_revision: str | None = "b3e9d4a1c7f2"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+expand_only: bool = True
+
+
+def upgrade() -> None:
+ op.add_column(
+ "agent_controllers", sa.Column("open_connection_id", sa.Text(), nullable=True)
+ )
+ op.add_column(
+ "agent_controllers",
+ sa.Column("open_generation", sa.BigInteger(), nullable=True),
+ )
+ op.add_column(
+ "agent_controllers", sa.Column("open_process_id", sa.Text(), nullable=True)
+ )
+ op.add_column(
+ "agent_controllers",
+ sa.Column("open_cursors", postgresql.JSONB(), nullable=True),
+ )
+
+
+def downgrade() -> None:
+ op.drop_column("agent_controllers", "open_cursors")
+ op.drop_column("agent_controllers", "open_process_id")
+ op.drop_column("agent_controllers", "open_generation")
+ op.drop_column("agent_controllers", "open_connection_id")
diff --git a/core/switch_core/migrations/versions/e2b0df5be5ff_expand_only_revisions_and_lease_keys.py b/core/switch_core/migrations/versions/e2b0df5be5ff_expand_only_revisions_and_lease_keys.py
new file mode 100644
index 000000000..d89de1fed
--- /dev/null
+++ b/core/switch_core/migrations/versions/e2b0df5be5ff_expand_only_revisions_and_lease_keys.py
@@ -0,0 +1,58 @@
+"""schema_expand_only_revisions, and the keys each process can read
+
+During a rolling deploy two releases serve one database. Once the newer one
+has migrated, the older one found a schema head it did not know and refused
+to start, so a restarted old pod crash-looped until the roll finished, and a
+rollback across a migration could not start at all. A revision marked
+`expand_only` (it only adds: new tables, nullable columns, indexes) is now
+recorded here as it is applied, with the revision it follows, and an older
+release accepts a head it reaches back from through those rows alone.
+
+`switch_core_processes.key_ids` holds the ids of the keys in each process's
+`SECRET_KEYS`, so a process starting with a new front key can see whether
+every live process could read what it is about to write.
+
+Expand only, and the first revision recorded as such, ahead of the others: `migrations/env.py`
+records it once this revision has created the table.
+
+Revision ID: e2b0df5be5ff
+Revises: e5b2c8d1f7a3
+Create Date: 2026-10-14 15:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects.postgresql import ARRAY
+
+revision: str = "e2b0df5be5ff"
+down_revision: str | None = "e5b2c8d1f7a3"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+expand_only: bool = True
+
+
+def upgrade() -> None:
+ op.create_table(
+ "schema_expand_only_revisions",
+ sa.Column("revision", sa.Text(), nullable=False),
+ sa.Column("down_revision", sa.Text(), nullable=False),
+ sa.Column(
+ "applied_at",
+ sa.DateTime(timezone=True),
+ server_default=sa.func.now(),
+ nullable=False,
+ ),
+ sa.PrimaryKeyConstraint("revision"),
+ )
+ op.add_column(
+ "switch_core_processes",
+ sa.Column("key_ids", ARRAY(sa.Text()), nullable=True),
+ )
+
+
+def downgrade() -> None:
+ op.drop_column("switch_core_processes", "key_ids")
+ op.drop_table("schema_expand_only_revisions")
diff --git a/core/switch_core/migrations/versions/e5956cbd3bdc_sign_in_flows.py b/core/switch_core/migrations/versions/e5956cbd3bdc_sign_in_flows.py
new file mode 100644
index 000000000..d7d5c2e36
--- /dev/null
+++ b/core/switch_core/migrations/versions/e5956cbd3bdc_sign_in_flows.py
@@ -0,0 +1,109 @@
+"""sign_in_flows: a sign-in under way, readable by every switch-core process
+
+A person connecting GitHub or a catalog service from Switch Console makes
+half a dozen requests, from Console and from their browser. Until now the
+flow they belonged to lived in the memory of the process that started it, so
+with several processes behind one address any step that reached another
+process found nothing and was told the sign-in had been interrupted. The flow
+moves here, scoped to its tenant like every other per-workspace row, and
+`tenant_of_sign_in_flow` names that tenant for the browser's steps, which
+carry only the flow's state.
+
+Expand only: a process from before this revision keeps its flows in memory
+and never reads this table.
+
+The lookup's DDL is a frozen copy of `switch_core/db/tenant_lookup.py`'s, as
+every revision in this chain copies rather than imports.
+
+Revision ID: e5956cbd3bdc
+Revises: e2b0df5be5ff
+Create Date: 2026-10-14 12:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects.postgresql import JSONB
+
+revision: str = "e5956cbd3bdc"
+down_revision: str | None = "e2b0df5be5ff"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+expand_only: bool = True
+
+SECURE_SEARCH_PATH = "pg_catalog, public, pg_temp"
+
+CREATE_TENANT_OF_SIGN_IN_FLOW = f"""CREATE OR REPLACE FUNCTION tenant_of_sign_in_flow(p_state text)
+ RETURNS SETOF text
+ LANGUAGE sql STABLE SECURITY DEFINER
+ SET search_path = {SECURE_SEARCH_PATH}
+AS $$SELECT tenant_id FROM sign_in_flows WHERE id = p_state AND expires_at > now()$$"""
+
+DROP_TENANT_OF_SIGN_IN_FLOW = "DROP FUNCTION IF EXISTS tenant_of_sign_in_flow(text)"
+
+
+def upgrade() -> None:
+ op.create_table(
+ "sign_in_flows",
+ sa.Column(
+ "tenant_id",
+ sa.Text(),
+ sa.ForeignKey("tenants.id", name="fk_sign_in_flows_tenant"),
+ nullable=False,
+ ),
+ sa.Column("id", sa.Text(), nullable=False),
+ sa.Column("kind", sa.Text(), nullable=False),
+ sa.Column("service", sa.Text(), nullable=False),
+ sa.Column(
+ "user_id",
+ sa.Text(),
+ sa.ForeignKey("users.id", ondelete="CASCADE"),
+ nullable=False,
+ ),
+ sa.Column("status", sa.Text(), nullable=False),
+ sa.Column("mode", sa.Text(), nullable=False),
+ sa.Column("port", sa.Integer(), nullable=False),
+ sa.Column("redirect_uri", sa.Text(), nullable=False),
+ sa.Column("owner_label", sa.Text(), nullable=False),
+ sa.Column("details", JSONB(), nullable=False),
+ sa.Column("encrypted_verifier", sa.Text(), nullable=False),
+ sa.Column("completion_secret_hash", sa.Text(), nullable=False),
+ sa.Column("browser_nonce_hash", sa.Text(), nullable=True),
+ sa.Column("encrypted_result", sa.Text(), nullable=True),
+ sa.Column("account_label", sa.Text(), nullable=True),
+ sa.Column("error", sa.Text(), nullable=True),
+ sa.Column(
+ "created_at",
+ sa.DateTime(timezone=True),
+ server_default=sa.func.now(),
+ nullable=False,
+ ),
+ sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
+ sa.PrimaryKeyConstraint("id"),
+ sa.CheckConstraint(
+ "kind IN ('github', 'service')", name="ck_sign_in_flows_kind"
+ ),
+ sa.CheckConstraint(
+ "status IN ('pending', 'authorizing', 'returning', 'delivered', "
+ "'checking', 'ready', 'failed')",
+ name="ck_sign_in_flows_status",
+ ),
+ sa.CheckConstraint(
+ "mode IN ('loopback', 'core')", name="ck_sign_in_flows_mode"
+ ),
+ )
+ op.create_index(
+ "ix_sign_in_flows_user", "sign_in_flows", ["tenant_id", "user_id", "kind"]
+ )
+ op.execute("ALTER TABLE sign_in_flows ENABLE ROW LEVEL SECURITY")
+ op.execute(
+ "CREATE POLICY tenant_isolation ON sign_in_flows FOR ALL USING (tenant_id = (SELECT require_tenant_id())) WITH CHECK (tenant_id = (SELECT require_tenant_id()))"
+ )
+ op.execute(CREATE_TENANT_OF_SIGN_IN_FLOW)
+
+
+def downgrade() -> None:
+ op.execute(DROP_TENANT_OF_SIGN_IN_FLOW)
+ op.drop_table("sign_in_flows")
diff --git a/core/switch_core/migrations/versions/e5b2c8d1f7a3_client_rooms_notify.py b/core/switch_core/migrations/versions/e5b2c8d1f7a3_client_rooms_notify.py
new file mode 100644
index 000000000..1e7078298
--- /dev/null
+++ b/core/switch_core/migrations/versions/e5b2c8d1f7a3_client_rooms_notify.py
@@ -0,0 +1,66 @@
+"""client_rooms_notify: signal every switch-core process when a membership changes
+
+Each switch-core process runs its own copy of every client, and a membership
+was only ever announced to the copy in the process that wrote it, through an
+in-process bus. A client added to a room through one process read nothing
+there from any other, and one removed kept reading it. This trigger sends a
+`membership.changed` signal on the `switch_signal` channel for every row
+inserted into or deleted from `client_rooms`, so every process re-reads that
+client's memberships.
+
+Revision ID: e5b2c8d1f7a3
+Revises: c4d8e2f6a913
+Create Date: 2026-10-14 00:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+from alembic import op
+
+revision: str = "e5b2c8d1f7a3"
+down_revision: str | None = "c4d8e2f6a913"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+# Copies of the DDL in `db/membership_notify_ddl.py`, written out here so this
+# migration keeps creating the same objects if that module changes.
+CREATE_NOTIFY_FUNCTION = """
+CREATE OR REPLACE FUNCTION switch_notify_membership() RETURNS trigger AS $$
+DECLARE
+ changed client_rooms;
+BEGIN
+ IF TG_OP = 'DELETE' THEN
+ changed := OLD;
+ ELSE
+ changed := NEW;
+ END IF;
+ PERFORM pg_notify(
+ 'switch_signal',
+ json_build_object(
+ 'kind', 'membership.changed',
+ 'tenant_id', changed.tenant_id,
+ 'client_id', changed.client_id,
+ 'room_id', changed.room_id
+ )::text
+ );
+ RETURN NULL;
+END;
+$$ LANGUAGE plpgsql
+"""
+
+CREATE_NOTIFY_TRIGGER = """
+CREATE TRIGGER client_rooms_notify
+ AFTER INSERT OR DELETE ON client_rooms
+ FOR EACH ROW EXECUTE FUNCTION switch_notify_membership()
+"""
+
+
+def upgrade() -> None:
+ op.execute(CREATE_NOTIFY_FUNCTION)
+ op.execute(CREATE_NOTIFY_TRIGGER)
+
+
+def downgrade() -> None:
+ op.execute("DROP TRIGGER IF EXISTS client_rooms_notify ON client_rooms")
+ op.execute("DROP FUNCTION IF EXISTS switch_notify_membership()")
diff --git a/core/switch_core/observability/bootstrap.py b/core/switch_core/observability/bootstrap.py
index 043f4f3fe..7b3269cca 100644
--- a/core/switch_core/observability/bootstrap.py
+++ b/core/switch_core/observability/bootstrap.py
@@ -34,11 +34,13 @@ from switch_core.observability.db_server import DbServerSampler
from switch_core.observability.exporter import MetricsExporter
from switch_core.observability.health import (
HealthMonitor,
+ agent_event_listener_check,
bridges_check,
connectors_check,
database_check,
message_listener_check,
session_activity_listener_check,
+ signal_bus_check,
)
from switch_core.observability.logs import (
DEFAULT_BATCH_SIZE,
@@ -85,6 +87,8 @@ class RuntimeProbes:
listener_connected: Callable[[], bool]
session_activity_listener_connected: Callable[[], bool]
+ agent_event_listener_connected: Callable[[], bool]
+ signal_bus_connected: Callable[[], bool]
bridges_running: Callable[[], int]
bridges_running_by_platform: Callable[[], Mapping[str, int]]
bridges_configured: Callable[[], int]
@@ -173,19 +177,30 @@ def start_observability(
version: str | None,
session_factory: async_sessionmaker,
probes: RuntimeProbes,
+ process_id: str,
db_server_engine: Callable[[], AsyncEngine] | None = None,
) -> Observability:
"""Install the registry, start the loops, and hand back the handle.
Called once, from the server's lifespan. `db_server_engine` builds the
engine the database sampler takes its one connection from; it must not be
- the application's pooled engine.
+ the application's pooled engine. `process_id` names this process on
+ everything it reports (`process_identity.py`).
"""
+ # With another replica to take the traffic, a process that would answer
+ # late is better out of service; with one, it is the only one there is.
+ listeners_gate = config.core_replicas > 1
monitor = HealthMonitor(
checks=[
database_check(session_factory),
- message_listener_check(probes.listener_connected),
- session_activity_listener_check(probes.session_activity_listener_connected),
+ message_listener_check(probes.listener_connected, listeners_gate),
+ session_activity_listener_check(
+ probes.session_activity_listener_connected, listeners_gate
+ ),
+ agent_event_listener_check(
+ probes.agent_event_listener_connected, listeners_gate
+ ),
+ signal_bus_check(probes.signal_bus_connected, listeners_gate),
bridges_check(probes.bridges_running, probes.bridges_configured),
connectors_check(probes.connectors_running, probes.connectors_configured),
],
@@ -236,6 +251,7 @@ def start_observability(
deployment_id=str(config.deployment_id),
commit_sha=switch_core_commit(),
repository_url=switch_core_repository_url(),
+ instance_id=process_id,
)
if config.otlp_metrics_enabled:
diff --git a/core/switch_core/observability/catalogue.py b/core/switch_core/observability/catalogue.py
index 1930e03aa..25355d406 100644
--- a/core/switch_core/observability/catalogue.py
+++ b/core/switch_core/observability/catalogue.py
@@ -246,12 +246,36 @@ OBSERVER_FAILURES = _spec(
"sum",
"{failure}",
"Reports about a message that raised: the transport telling its observer "
- "after the commit, or an agent's consumer reporting a message addressed to "
- "it. The message is unaffected and only the report is lost, so these are "
+ "after the commit, or the router reporting a message addressed to an agent "
+ "once it has committed. The message is unaffected and only the report is lost, so these are "
"caught by design and invisible without a counter. `actor` is who was "
"writing or reading.",
"actor",
)
+DELIVERY_BACKLOG_SKIPPED = _spec(
+ "switch.messages.backlog_skipped",
+ "sum",
+ "{message}",
+ "Room rows a consumer resuming from its stored cursor stepped over because "
+ "they were too old to act on (`transport/cursors.py`, BACKLOG_MAX_AGE): "
+ "rows written while no process read for that client, past the cap. They "
+ "stay in the room's history; the command replies and platform posts they "
+ "would have caused were never made. `actor` is whose consumer it was.",
+ "actor",
+)
+ROUTING_FAILURES = _spec(
+ "switch.messages.routing_failures",
+ "sum",
+ "{failure}",
+ "Work the router could not do for one agent while the message it was "
+ "routing still went out: deciding whether the message is for the agent "
+ "(`stage` decide), storing the agent's own event for it "
+ "(`stage` store), a reply or report owed once it committed "
+ "(`stage` after_commit), or deciding a multi-file post given up on "
+ "(`stage` sweep). Each is caught so one agent cannot cost a room its "
+ "message, which is what makes it invisible without a counter.",
+ "stage",
+)
DELIVERY_LAG = _spec(
"switch.messages.delivery_lag",
"histogram",
@@ -626,6 +650,8 @@ CATALOGUE: dict[str, MetricSpec] = {
SEND_FAILURES,
DELIVERY_FAILURES,
OBSERVER_FAILURES,
+ DELIVERY_BACKLOG_SKIPPED,
+ ROUTING_FAILURES,
DELIVERY_LAG,
DELIVERY_CACHE_READS,
DELIVERY_CACHE_FILLS,
diff --git a/core/switch_core/observability/health.py b/core/switch_core/observability/health.py
index 3aa524320..cb24ddd92 100644
--- a/core/switch_core/observability/health.py
+++ b/core/switch_core/observability/health.py
@@ -5,10 +5,13 @@ probe, the gateway Deployment and the setup Job wait on it at boot, so anything
it checked would become a boot-ordering dependency for them.
``/health/ready`` is the real one, and **what gates it is deliberately narrow.**
-switch-core runs as a single replica with `Recreate`, so a failing readiness
-probe does not shift traffic to a healthy pod — it empties the Service. Only
-the database gates, because without it every request is an error anyway. A
-crashed bridge is reported and alerted on but never fatal.
+With one replica, a failing readiness probe does not shift traffic to a
+healthy pod: it empties the Service. So on one replica only the database
+gates, because without it every request is an error anyway. With several
+(CORE_REPLICAS above 1), a process whose LISTEN connections are down is also
+taken out of service, since another replica can take its traffic and this one
+would answer it late. A crashed bridge is reported and alerted on but never
+fatal.
The checks run on their own schedule; the kubelet and the metrics exporter read
one cached answer. The cache carries its age, and one nobody is refreshing
@@ -52,7 +55,7 @@ class CheckOutcome:
class HealthCheck:
name: str
# Whether a failure takes the server out of service. On a single-replica
- # deployment that is a bigger decision than it looks — see the docstring.
+ # deployment that is a bigger decision than it looks; see the docstring.
gates_readiness: bool
probe: Callable[[], Awaitable[CheckOutcome]]
@@ -110,7 +113,20 @@ def database_check(session_factory: async_sessionmaker) -> HealthCheck:
return HealthCheck(name="database", gates_readiness=True, probe=probe)
-def message_listener_check(is_connected: Callable[[], bool]) -> HealthCheck:
+def _listener_check(
+ name: str, is_connected: Callable[[], bool], gates_readiness: bool, detail: str
+) -> HealthCheck:
+ async def probe() -> CheckOutcome:
+ if is_connected():
+ return CheckOutcome(name=name, healthy=True, detail="")
+ return CheckOutcome(name=name, healthy=False, detail=detail)
+
+ return HealthCheck(name=name, gates_readiness=gates_readiness, probe=probe)
+
+
+def message_listener_check(
+ is_connected: Callable[[], bool], gates_readiness: bool
+) -> HealthCheck:
"""Whether room delivery is actually happening.
Every room's fan-out is woken through this `LISTEN`. When it is down the
@@ -118,49 +134,74 @@ def message_listener_check(is_connected: Callable[[], bool]) -> HealthCheck:
reconnects itself, so restarting the pod would not fix it faster and would
drop every live session to find out.
"""
-
- async def probe() -> CheckOutcome:
- if is_connected():
- return CheckOutcome(name="message_listener", healthy=True, detail="")
- return CheckOutcome(
- name="message_listener",
- healthy=False,
- detail=(
- "The Postgres LISTEN connection is down, so no room message is "
- "being delivered. It retries with backoff; if this persists, "
- "the database is refusing connections."
- ),
- )
-
- return HealthCheck(name="message_listener", gates_readiness=False, probe=probe)
+ return _listener_check(
+ "message_listener",
+ is_connected,
+ gates_readiness,
+ "The Postgres LISTEN connection is down, so no room message is being "
+ "delivered. It retries with backoff; if this persists, the database is "
+ "refusing connections.",
+ )
-def session_activity_listener_check(is_connected: Callable[[], bool]) -> HealthCheck:
+def session_activity_listener_check(
+ is_connected: Callable[[], bool], gates_readiness: bool
+) -> HealthCheck:
"""Whether session activity and approval answers are being pushed.
Approval answers reach agents, and activity reaches messaging platforms,
through this `LISTEN`. While it is down both are still recorded but nothing
is pushed; it reconnects itself and subscribers resync when it does.
"""
+ return _listener_check(
+ "session_activity_listener",
+ is_connected,
+ gates_readiness,
+ "The Postgres LISTEN connection for session activity is down, so "
+ "approval answers and activity are recorded but not pushed. It retries "
+ "with backoff; if this persists, the database is refusing connections.",
+ )
- async def probe() -> CheckOutcome:
- if is_connected():
- return CheckOutcome(
- name="session_activity_listener", healthy=True, detail=""
- )
- return CheckOutcome(
- name="session_activity_listener",
- healthy=False,
- detail=(
- "The Postgres LISTEN connection for session activity is down, so "
- "approval answers and activity are recorded but not pushed. It "
- "retries with backoff; if this persists, the database is "
- "refusing connections."
- ),
- )
- return HealthCheck(
- name="session_activity_listener", gates_readiness=False, probe=probe
+def agent_event_listener_check(
+ is_connected: Callable[[], bool], gates_readiness: bool
+) -> HealthCheck:
+ """Whether an agent's stream hears at once about events written elsewhere.
+
+ An agent's own events can be written by any process, and this `LISTEN` is
+ how the process holding the agent's stream learns of one. While it is down
+ the stream still finds them, but only at its next keepalive re-read, some
+ seconds late.
+ """
+ return _listener_check(
+ "agent_event_listener",
+ is_connected,
+ gates_readiness,
+ "The Postgres LISTEN connection for agent events is down, so agents "
+ "receive events written by another process only at their next "
+ "keepalive. It retries with backoff; if this persists, the database is "
+ "refusing connections.",
+ )
+
+
+def signal_bus_check(
+ is_connected: Callable[[], bool], gates_readiness: bool
+) -> HealthCheck:
+ """Whether signals from other switch-core processes are arriving.
+
+ A change made through another process (a controller given a new agent, a
+ controller revoked) reaches the client it concerns on this one through
+ this `LISTEN`. While it is down those changes wait for the reconnect, which
+ resyncs everything it missed.
+ """
+ return _listener_check(
+ "signal_bus",
+ is_connected,
+ gates_readiness,
+ "The Postgres LISTEN connection for signals between switch-core "
+ "processes is down, so changes made through another process reach "
+ "this one's clients only when it reconnects. It retries with backoff; "
+ "if this persists, the database is refusing connections.",
)
diff --git a/core/switch_core/observability/otlp.py b/core/switch_core/observability/otlp.py
index 831608d55..67a3142fd 100644
--- a/core/switch_core/observability/otlp.py
+++ b/core/switch_core/observability/otlp.py
@@ -137,6 +137,11 @@ class OtlpResource:
version, and is then omitted rather than placeheld — a dashboard filtered
by version should show the record missing, not attribute it to a release
nobody built. ``commit_sha`` and ``repository_url`` follow the same rule.
+
+ ``instance_id`` is the process (`process_identity.py`), so a deployment of
+ several pods reports one series per pod rather than one series every pod
+ overwrites. ``None`` for a resource that describes no single process, as
+ product telemetry's does.
"""
service_name: str
@@ -145,6 +150,7 @@ class OtlpResource:
deployment_id: str
commit_sha: str | None
repository_url: str | None
+ instance_id: str | None
def attributes(self) -> dict[str, AttributeValue]:
values: dict[str, AttributeValue] = {
@@ -155,6 +161,8 @@ class OtlpResource:
}
if self.service_version:
values["service.version"] = self.service_version
+ if self.instance_id:
+ values["service.instance.id"] = self.instance_id
if self.environment:
# Datadog maps this onto `env`.
values["deployment.environment"] = self.environment
diff --git a/core/switch_core/ownership.py b/core/switch_core/ownership.py
new file mode 100644
index 000000000..172cc82b9
--- /dev/null
+++ b/core/switch_core/ownership.py
@@ -0,0 +1,469 @@
+"""One owner for each piece of work that must run once, across switch-core processes.
+
+Several switch-core processes share one database, and some work must run in
+exactly one of them: a client's consumer, such as the admin's, which answers
+commands and posts warnings; a collaboration bridge's platform connection and
+the consumer that posts to the platform; a server-side connector polling its
+agents. Run in two processes, every such effect happens twice.
+
+Each piece of work is a kind and a key ("client" and a client id). A process
+wants the keys it could run, and runs those it owns. Which process owns a key
+is decided twice over:
+
+- **Who should.** Rendezvous hashing over the processes whose lease in
+ `switch_core_processes` holds: every process scores every key against every
+ live process and the highest score wins. All processes reading the same
+ leases agree, and a process coming or going moves only the keys it wins or
+ held, not everyone's.
+- **Who does.** A row in `ownership_leases`, claimed in one statement and only
+ while nobody holds it or its holder's process lease has lapsed. The hash says
+ who tries; the row makes sure only one succeeds, whatever two processes
+ briefly believe about each other.
+
+A key moves without overlap. The process that should no longer own it stops
+the work first and then releases the row, and only then can the new owner claim
+it. A process that dies holds nothing once its lease lapses, and its work is
+claimed by the next owner. A process that cannot renew its own lease stops all
+its work at half the lease's lifetime by its own clock, before anyone else may
+claim it (`ProcessLease.surely_held`).
+
+Every claim bumps the row's `token`, and the work is started with it (`Fence`).
+Writes that must not outlive the hold name the token in the same statement
+(`DeliveryCursorStore.advance`), so a holder taken over without knowing it is
+refused, and is stopped when it is.
+
+Changes reach the other processes over the signal bus: `ownership.changed`
+after a process releases keys, starts, or has marked its process lease stopped
+as it shuts down, so they rebalance at once. Every process also rebalances
+every `BALANCE_INTERVAL_SECONDS`, which is how a process that died is noticed:
+one read of the process leases, and no write unless something moved.
+
+With one process this is that process owning everything, claimed once at boot.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import hashlib
+import logging
+import time
+from collections.abc import Awaitable, Callable, Sequence
+from dataclasses import dataclass
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from switch_core.process_lease import (
+ LEASE_TTL,
+ RENEW_INTERVAL_SECONDS,
+ read_leases,
+)
+
+if TYPE_CHECKING:
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+ from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+ from switch_core.db.stores.switch_core_process_store import (
+ SwitchCoreProcessStore,
+ )
+ from switch_core.process_lease import ProcessLease
+ from switch_core.signals import SignalBus
+
+logger = logging.getLogger(__name__)
+
+OWNERSHIP_CHANGED_SIGNAL = "ownership.changed"
+
+# How often every process re-reads the process leases and rebalances, which
+# is how a process that stopped renewing is noticed. As often as leases are
+# renewed: a lapse cannot be seen sooner than that.
+BALANCE_INTERVAL_SECONDS = RENEW_INTERVAL_SECONDS
+
+# How long a released row is kept. Work that still exists is claimed again
+# within seconds of its release.
+PRUNE_AFTER = timedelta(hours=1)
+
+# How often a balance also prunes, so the delete is not a write on every pass.
+PRUNE_EVERY_SECONDS = 600.0
+
+StartWork = Callable[[str, "Fence"], Awaitable[None]]
+StopWork = Callable[[str], Awaitable[None]]
+
+
+@dataclass(frozen=True)
+class Fence:
+ """The hold a piece of work was started under: which lease, which process,
+ and the token the claim returned."""
+
+ kind: str
+ key: str
+ process_id: str
+ token: int
+
+
+def owner_of(kind: str, key: str, processes: Sequence[str]) -> str | None:
+ """The process that should own `key` among `processes`: the one whose
+ score for it is highest. None when there is no process."""
+ if not processes:
+ return None
+ return max(
+ processes, key=lambda process_id: (_score(kind, key, process_id), process_id)
+ )
+
+
+def _score(kind: str, key: str, process_id: str) -> int:
+ digest = hashlib.blake2b(
+ f"{kind}\x1f{key}\x1f{process_id}".encode(), digest_size=8
+ ).digest()
+ return int.from_bytes(digest, "big")
+
+
+class OwnedKind:
+ """The keys of one kind of work: which this process could run, and which
+ it runs. Built by `Ownership.kind`."""
+
+ def __init__(
+ self, ownership: Ownership, name: str, start: StartWork, stop: StopWork
+ ) -> None:
+ self.name = name
+ self._ownership = ownership
+ self._start = start
+ self._stop = stop
+ self._wanted: set[str] = set()
+ self._held: dict[str, Fence] = {}
+ # Fences the work reported taken over, kept until the next balance
+ # stops it.
+ self._lost: set[Fence] = set()
+
+ def want(self, key: str) -> None:
+ """This process could run `key`. It runs it once it owns it."""
+ if key in self._wanted:
+ return
+ self._wanted.add(key)
+ self._ownership.nudge()
+
+ def wanted(self) -> set[str]:
+ return set(self._wanted)
+
+ def wants(self, key: str) -> bool:
+ return key in self._wanted
+
+ @property
+ def stopping(self) -> bool:
+ """Whether the process is shutting down, so work stopped now is not
+ to be started again in any form."""
+ return self._ownership.stopping
+
+ async def settle(self) -> None:
+ """Balance now and return once done, for a caller that needs to know
+ whether this process holds a key it has just wanted."""
+ await self._ownership.balance()
+
+ def held(self) -> set[str]:
+ """The keys this process runs now."""
+ return set(self._held)
+
+ def fence(self, key: str) -> Fence | None:
+ return self._held.get(key)
+
+ async def drop(self, key: str) -> None:
+ """This process will not run `key` any more: stop it here now if it
+ runs, and let go of its lease. For work being deleted, which must be
+ stopped before its rows go."""
+ self._wanted.discard(key)
+ async with self._ownership.balancing():
+ if key in self._held:
+ await self._stop_held([key])
+ await self._ownership.release(self.name, [key])
+
+ def lost(self, fence: Fence) -> None:
+ """The work found its lease taken over: stop it at the next balance."""
+ if self._held.get(fence.key) == fence:
+ self._lost.add(fence)
+ self._ownership.nudge()
+
+ async def _balance(self, live: Sequence[str]) -> None:
+ me = self._ownership.process_id
+ lost = [fence.key for fence in self._lost if self._held.get(fence.key) == fence]
+ self._lost.clear()
+ if lost:
+ logger.warning(
+ "%d %s lease(s) were taken over by another process; stopping "
+ "them here: %s",
+ len(lost),
+ self.name,
+ ", ".join(sorted(lost)),
+ )
+ await self._stop_held(lost)
+ mine = {key for key in self._wanted if owner_of(self.name, key, live) == me}
+ leaving = [key for key in self._held if key not in mine]
+ if leaving:
+ logger.info(
+ "Handing %d %s(s) to the process that now owns them",
+ len(leaving),
+ self.name,
+ )
+ await self._stop_held(leaving)
+ await self._ownership.release(self.name, leaving)
+ wanted = [key for key in mine if key not in self._held]
+ if not wanted:
+ return
+ claimed = await self._ownership.claim(self.name, wanted)
+ for key, token in sorted(claimed.items()):
+ # Dropped while the claim was in flight: give it straight back.
+ if key not in self._wanted:
+ await self._ownership.release(self.name, [key])
+ continue
+ fence = Fence(kind=self.name, key=key, process_id=me, token=token)
+ self._held[key] = fence
+ try:
+ await self._start(key, fence)
+ except Exception:
+ logger.exception("Starting %s %s failed", self.name, key)
+
+ async def _stop_held(self, keys: Sequence[str]) -> None:
+ """Stop the work of `keys` here, all at once, each failure logged."""
+ stopping = [key for key in keys if self._held.pop(key, None) is not None]
+ results = await asyncio.gather(
+ *(self._stop(key) for key in stopping), return_exceptions=True
+ )
+ for key, result in zip(stopping, results, strict=True):
+ if isinstance(result, BaseException):
+ logger.error(
+ "Stopping %s %s failed",
+ self.name,
+ key,
+ exc_info=result,
+ )
+
+ async def _stop_all(self) -> None:
+ await self._stop_held(list(self._held))
+
+
+class Ownership:
+ """Which pieces of work this process runs, decided with the others."""
+
+ def __init__(
+ self,
+ *,
+ lease: ProcessLease,
+ session_factory: async_sessionmaker[AsyncSession],
+ bus: SignalBus,
+ store: OwnershipLeaseStore,
+ processes: SwitchCoreProcessStore,
+ ) -> None:
+ self._lease = lease
+ self._session_factory = session_factory
+ self._bus = bus
+ self._store = store
+ self._processes = processes
+ self._kinds: dict[str, OwnedKind] = {}
+ # One balance, drop or stop at a time: each reads what is held, then
+ # awaits stops and claims, and two interleaved would claim or release
+ # each other's keys.
+ self._lock = asyncio.Lock()
+ self._nudged = asyncio.Event()
+ self._suspended = False
+ self._stopped = False
+ # Whether `stop` saw every piece of work here stop, rather than being
+ # cancelled first.
+ self._work_stopped = False
+ self._pruned_at: float | None = None
+ bus.handle(OWNERSHIP_CHANGED_SIGNAL, self._on_changed)
+ bus.on_resync(self._resync)
+
+ @property
+ def process_id(self) -> str:
+ return self._lease.process_id
+
+ @property
+ def stopping(self) -> bool:
+ return self._stopped
+
+ def kind(self, name: str, *, start: StartWork, stop: StopWork) -> OwnedKind:
+ """Register a kind of work. `start(key, fence)` runs the work here and
+ returns once it is under way; `stop(key)` stops it and returns once it
+ has stopped."""
+ if name in self._kinds:
+ raise ValueError(f"Ownership kind {name!r} is already registered")
+ kind = OwnedKind(self, name, start, stop)
+ self._kinds[name] = kind
+ return kind
+
+ def nudge(self) -> None:
+ """Balance soon, rather than at the next interval."""
+ self._nudged.set()
+
+ @contextlib.asynccontextmanager
+ async def balancing(self) -> Any:
+ async with self._lock:
+ yield
+
+ # ── Lifecycle ────────────────────────────────────────────────────────────
+
+ async def start(self) -> None:
+ """The first balance, before the process serves, so what this process
+ owns is running as it would have at boot with one process. Then the
+ other processes are told this one is here.
+
+ The lease is renewed first, so the work starts with all the time the
+ lease is sure for, however long the boot took since it was claimed."""
+ await self._lease.renew()
+ await self.balance()
+ self._bus.publish(OWNERSHIP_CHANGED_SIGNAL)
+
+ async def run(self) -> None:
+ """Balance whenever nudged, every `BALANCE_INTERVAL_SECONDS`, and the
+ moment this process's own lease stops being sure, until cancelled."""
+ while not self._stopped:
+ timeout = BALANCE_INTERVAL_SECONDS
+ held_until = self._lease.held_until()
+ if held_until is not None and held_until > time.monotonic():
+ # Wake as the lease stops being sure, not up to an interval
+ # after: that is the moment this process's work has to stop.
+ timeout = min(timeout, held_until - time.monotonic())
+ with contextlib.suppress(TimeoutError):
+ await asyncio.wait_for(self._nudged.wait(), timeout)
+ self._nudged.clear()
+ try:
+ await self.balance()
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ logger.exception("Balancing owned work failed; retrying shortly")
+
+ async def stop(self) -> None:
+ """Stop everything this process runs, then let go of its leases. Called
+ as the process shuts down, before its process lease is marked stopped;
+ a second call waits for the first and does nothing. It raises instead
+ when the first was cancelled before all the work had stopped, as
+ `main` does with a stop that takes too long: that work may still run
+ here, so the process lease must be left to lapse, not marked stopped.
+
+ Nothing is signalled yet: while the process lease holds, the other
+ processes still hash this one's work to it and claim none of it. The
+ caller signals with `gone` once the lease is marked stopped."""
+ async with self._lock:
+ if self._stopped:
+ if not self._work_stopped:
+ raise RuntimeError(
+ "This process's owned work was not all stopped; its "
+ "leases pass to the other processes once its process "
+ "lease lapses"
+ )
+ return
+ self._stopped = True
+ for kind in self._kinds.values():
+ await kind._stop_all()
+ self._work_stopped = True
+ try:
+ async with self._session_factory() as session:
+ await self._store.release_all(session, self.process_id)
+ await session.commit()
+ except Exception:
+ logger.warning(
+ "Could not release this process's owned work at shutdown; "
+ "the other processes take it over once its lease lapses",
+ exc_info=True,
+ )
+ return
+
+ def gone(self) -> None:
+ """Tell the other processes this one's process lease is marked
+ stopped, so they rebalance at once and claim the work `stop` let go
+ of, rather than at their next interval."""
+ self._bus.publish(OWNERSHIP_CHANGED_SIGNAL)
+
+ # ── Balancing ────────────────────────────────────────────────────────────
+
+ async def balance(self) -> None:
+ """Run here exactly the wanted keys this process should own, as far
+ as the leases allow."""
+ async with self._lock:
+ if self._stopped:
+ return
+ if not self._lease.surely_held():
+ await self._suspend()
+ return
+ async with self._session_factory() as session:
+ leases = await read_leases(session, self._processes)
+ await self._prune_now_and_then()
+ live = sorted(
+ process_id for process_id in leases.leases if leases.holds(process_id)
+ )
+ if self.process_id not in live:
+ # Sure of its own lease by its own clock but not yet seen
+ # renewed by the database's: claim nothing until both agree.
+ logger.warning(
+ "This process's lease %s does not read as held; owning "
+ "nothing until it does",
+ self.process_id,
+ )
+ return
+ if self._suspended:
+ logger.warning(
+ "This process's lease %s holds again; claiming its share "
+ "of the work back",
+ self.process_id,
+ )
+ self._suspended = False
+ for kind in self._kinds.values():
+ await kind._balance(live)
+
+ async def _prune_now_and_then(self) -> None:
+ now = time.monotonic()
+ if self._pruned_at is not None and now - self._pruned_at < PRUNE_EVERY_SECONDS:
+ return
+ self._pruned_at = now
+ async with self._session_factory() as session:
+ pruned = await self._store.prune(session, PRUNE_AFTER)
+ await session.commit()
+ if pruned:
+ logger.info("Pruned %d ownership lease(s) long released", pruned)
+
+ async def _suspend(self) -> None:
+ """Stop all work here because this process can no longer be sure its
+ lease holds. Nothing is released: the database may not be reachable,
+ and the leases pass to the other processes once this one lapses."""
+ if not any(kind.held() for kind in self._kinds.values()):
+ self._suspended = True
+ return
+ logger.error(
+ "This process's lease %s was not renewed in time; stopping every "
+ "piece of work it owns before another process may take it over",
+ self.process_id,
+ )
+ self._suspended = True
+ for kind in self._kinds.values():
+ await kind._stop_all()
+
+ async def claim(self, kind: str, keys: Sequence[str]) -> dict[str, int]:
+ async with self._session_factory() as session:
+ claimed = await self._store.claim(
+ session,
+ kind=kind,
+ keys=keys,
+ process_id=self.process_id,
+ lease_ttl=LEASE_TTL,
+ )
+ await session.commit()
+ if claimed:
+ logger.info("Claimed %d %s(s) of %d wanted", len(claimed), kind, len(keys))
+ return claimed
+
+ async def release(self, kind: str, keys: Sequence[str]) -> None:
+ async with self._session_factory() as session:
+ await self._store.release(
+ session, kind=kind, keys=keys, process_id=self.process_id
+ )
+ await session.commit()
+ self._bus.publish(OWNERSHIP_CHANGED_SIGNAL)
+
+ # ── Signals ──────────────────────────────────────────────────────────────
+
+ def _on_changed(self, signal: dict[str, Any]) -> None:
+ self.nudge()
+
+ async def _resync(self) -> None:
+ """Signals sent while this process was not listening are gone; the
+ balance reads what they would have said."""
+ self.nudge()
diff --git a/core/switch_core/process_identity.py b/core/switch_core/process_identity.py
new file mode 100644
index 000000000..a74bb8a7a
--- /dev/null
+++ b/core/switch_core/process_identity.py
@@ -0,0 +1,25 @@
+"""The one name this switch-core process goes by.
+
+Several processes can serve one database, and each writes things the others
+read: a lease row in `switch_core_processes`, the controller connections it
+holds, the signals it sends on `signals.SignalBus`. An operator reading any of
+those, or a log line, or a metric series, needs to know which process it came
+from, and needs the same answer from all of them. So the name is made once, at
+startup, and handed to everything that records it.
+
+It is the pod's name (POD_NAME, from the Kubernetes downward API) or the host
+name, followed by a random suffix drawn at each start. The suffix is what keeps
+two lives of one pod apart: a pod restarted in place keeps its name, and a
+lease or a connection row left by the process before must never read as held
+by the process after.
+"""
+
+from __future__ import annotations
+
+import secrets
+import socket
+
+
+def new_process_id(pod_name: str | None) -> str:
+ """A name for this process: its pod or host, then a suffix of its own."""
+ return f"{pod_name or socket.gethostname()}-{secrets.token_hex(4)}"
diff --git a/core/switch_core/process_lease.py b/core/switch_core/process_lease.py
new file mode 100644
index 000000000..4594cbaa6
--- /dev/null
+++ b/core/switch_core/process_lease.py
@@ -0,0 +1,214 @@
+"""This switch-core process's lease, and reading every process's.
+
+Every process holds one, under its process id (`process_identity.py`): it is
+how one process knows another is alive. An agent's or a controller's socket
+lives in the memory of the one process holding it, and that process writes the
+connection's transitions to a row (`bridges/agent/protocol/connection_rows.py`
+for agents, `management/connection_ledger.py` for controllers). A process that
+dies writes no closing, so each process also holds a lease in
+`switch_core_processes`: it claims it at startup, renews it every
+`RENEW_EVERY_SECONDS`, and marks it stopped as it shuts down. A connection
+reads as online only while its holding process's lease is held: renewed
+within `LEASE_TTL` and not stopped. One write per process per renewal, however
+many connections it holds.
+
+The work that must run once (a client's consumer, a bridge's platform
+connection, a server-side connector) is owned through `ownership.py`, whose
+leases are valid only while their holder's process lease is held, and which
+stops its work when this process can no longer be sure its own is
+(`surely_held`).
+
+The lease also records the ids of the keys the process can read, which a
+process starting with a new front key checks (`db/key_rotation.py`).
+
+Leases are written and compared on the database's clock, never a host's.
+Leases not renewed for `PRUNE_AFTER` are deleted by whichever process renews
+next; a connection naming a process with no lease reads as lost.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import time
+from dataclasses import dataclass
+from datetime import datetime, timedelta
+
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.stores.switch_core_process_store import (
+ ProcessLeaseRow,
+ SwitchCoreProcessStore,
+)
+
+logger = logging.getLogger(__name__)
+
+# The unit the lease's lifetime is counted in, and how often the other
+# processes look at the leases (`ownership.BALANCE_INTERVAL_SECONDS`).
+RENEW_INTERVAL_SECONDS = 5.0
+
+# Several renewals: one missed write, or a slow one, does not put every
+# machine the process holds offline.
+LEASE_TTL = timedelta(seconds=3 * RENEW_INTERVAL_SECONDS)
+
+# How long after the last good renewal was sent the next is sent: a third of
+# the half lease `ProcessLease.surely_held` counts from. A renewal may then
+# take twice this, or fail once and be retried, before this process has to
+# stop its owned work (`ownership.py`).
+RENEW_EVERY_SECONDS = LEASE_TTL.total_seconds() / 6
+
+# How long after a failed renewal the next is tried.
+RETRY_RENEWAL_SECONDS = 1.0
+
+PRUNE_AFTER = timedelta(hours=1)
+
+# Why a connection the row still shows open is not, when its process is the
+# reason: the process shut down without writing it, or stopped renewing.
+SERVER_SHUTDOWN = "server_shutdown"
+SERVER_LOST = "server_lost"
+
+
+@dataclass(frozen=True)
+class ProcessLeases:
+ """Every process's lease as the database held it at `read_at`, its own
+ clock."""
+
+ read_at: datetime
+ leases: dict[str, ProcessLeaseRow]
+
+ def holds(self, process_id: str | None) -> bool:
+ """Whether the process is alive and holding its sockets."""
+ lease = self.leases.get(process_id) if process_id is not None else None
+ return (
+ lease is not None
+ and lease.stopped_at is None
+ and self.read_at - lease.beat_at < LEASE_TTL
+ )
+
+ def ended(self, process_id: str | None) -> tuple[datetime | None, str]:
+ """When and why a process not holding stopped: shut down at
+ `stopped_at`, or lost after its last renewal (when not known once its
+ lease is pruned)."""
+ lease = self.leases.get(process_id) if process_id is not None else None
+ if lease is None:
+ return None, SERVER_LOST
+ if lease.stopped_at is not None:
+ return lease.stopped_at, SERVER_SHUTDOWN
+ return lease.beat_at, SERVER_LOST
+
+
+async def read_leases(
+ session: AsyncSession, processes: SwitchCoreProcessStore
+) -> ProcessLeases:
+ """Every lease, in the caller's session; the table has no tenant policy,
+ so any session reads it."""
+ read_at, leases = await processes.read(session)
+ return ProcessLeases(read_at=read_at, leases=leases)
+
+
+class ProcessLease:
+ """This process's lease. The sessions are opened with no tenant bound:
+ the table carries none."""
+
+ def __init__(
+ self,
+ *,
+ process_id: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: SwitchCoreProcessStore,
+ key_ids: list[str],
+ ) -> None:
+ self.process_id = process_id
+ self._session_factory = session_factory
+ self._processes = processes
+ # The ids of the keys this process can read, recorded on its row so a
+ # process starting with a new front key can check them
+ # (`db/key_rotation.py`).
+ self._key_ids = key_ids
+ # Set by `stop`. A renewal clears `stopped_at`, so once the process
+ # has released its lease (as it starts draining, while it still runs)
+ # no renewal may claim it back.
+ self._stopped = False
+ # This process's monotonic clock when the last renewal that succeeded
+ # was sent, or None before the first and after `stop`. Sent, not
+ # answered: the database stamped the lease at some moment after this,
+ # so a deadline counted from here can only be early.
+ self._renewed_at: float | None = None
+
+ def surely_held(self) -> bool:
+ """Whether no other process can yet read this lease as lapsed.
+
+ Another process takes over this one's owned work once the lease is
+ `LEASE_TTL` old by the database's clock. This answers from half that,
+ on this process's own clock and from when the last good renewal was
+ sent, so the work stops here well before anyone else may start it,
+ however far the two clocks have drifted.
+ """
+ held_until = self.held_until()
+ return held_until is not None and time.monotonic() < held_until
+
+ def held_until(self) -> float | None:
+ """On this process's monotonic clock, when `surely_held` turns false
+ unless the lease is renewed first. None before the first renewal and
+ after `stop`."""
+ if self._renewed_at is None:
+ return None
+ return self._renewed_at + LEASE_TTL.total_seconds() / 2
+
+ async def renew(self) -> None:
+ """Claim or renew the lease, and prune leases long dead. Does nothing
+ once the lease is stopped: a draining process stays released."""
+ if self._stopped:
+ return
+ sent_at = time.monotonic()
+ async with self._session_factory() as session:
+ await self._processes.renew(session, self.process_id, self._key_ids)
+ pruned = await self._processes.prune(session, PRUNE_AFTER)
+ await session.commit()
+ self._renewed_at = sent_at
+ if pruned:
+ logger.info("Pruned %d switch-core process lease(s) long stale", pruned)
+
+ async def read_all(self) -> ProcessLeases:
+ """Every process's lease, as the database holds it now."""
+ async with self._session_factory() as session:
+ return await read_leases(session, self._processes)
+
+ async def stop(self) -> None:
+ """Mark the lease stopped: every connection this process still shows
+ open reads as offline from now. Only the first call writes."""
+ if self._stopped:
+ return
+ self._stopped = True
+ self._renewed_at = None
+ async with self._session_factory() as session:
+ await self._processes.stop(session, self.process_id)
+ await session.commit()
+ logger.info("Released switch-core process lease %s", self.process_id)
+
+ async def run(self) -> None:
+ """Renew `RENEW_EVERY_SECONDS` after the last good renewal was sent,
+ until cancelled. A failed renewal is logged and retried after
+ `RETRY_RENEWAL_SECONDS`; once the lease is `LEASE_TTL` old this
+ process's connections read offline elsewhere.
+
+ Counted from the last renewal, wherever it was made, so a loop started
+ late (after a slow boot, whose renewal came from `Ownership.start`)
+ renews at once rather than an interval after it starts."""
+ while True:
+ await asyncio.sleep(self._until_next_renewal())
+ try:
+ await self.renew()
+ except Exception:
+ logger.exception(
+ "Renewing switch-core process lease %s failed; the "
+ "connections it holds read offline once it is %s old",
+ self.process_id,
+ LEASE_TTL,
+ )
+ await asyncio.sleep(RETRY_RENEWAL_SECONDS)
+
+ def _until_next_renewal(self) -> float:
+ if self._renewed_at is None:
+ return 0.0 if not self._stopped else RENEW_EVERY_SECONDS
+ return max(0.0, self._renewed_at + RENEW_EVERY_SECONDS - time.monotonic())
diff --git a/core/switch_core/providers/github.py b/core/switch_core/providers/github.py
index 50fb471cc..caa8e85f6 100644
--- a/core/switch_core/providers/github.py
+++ b/core/switch_core/providers/github.py
@@ -1,13 +1,11 @@
from __future__ import annotations
-import asyncio
import base64
import hashlib
import json
import re
-import secrets
import time
-from dataclasses import dataclass, field
+from dataclasses import dataclass
from pathlib import Path
from urllib.parse import urlencode, urlsplit
@@ -97,21 +95,6 @@ class GitHubReach:
sso_hidden: list[int]
-@dataclass
-class GitHubFlow:
- tenant_id: str
- user_id: str
- expires_at: float
- verifier: str
- browser_nonce: str
- port: int
- completion_secret: str
- owner_label: str
- status: str = "pending"
- login: str = ""
- credentials: dict = field(default_factory=dict)
-
-
class GitHubConnections:
def __init__(self, config_path: str):
config = json.loads(Path(config_path).read_text())
@@ -135,7 +118,6 @@ class GitHubConnections:
raise ValueError("GitHub connection configuration is invalid")
self.callback = self.origin + "/gateway/provider-connections/github/callback"
self.install_url = f"https://github.com/apps/{self.slug}/installations/new"
- self.flows: dict[str, GitHubFlow] = {}
async def revoke(self, token: str) -> None:
try:
@@ -152,56 +134,10 @@ class GitHubConnections:
except httpx.HTTPError:
raise GitHubError("Could not reach GitHub to revoke the sign-in.") from None
- def start(
- self,
- tenant_id: str,
- user_id: str,
- port: int,
- completion_secret: str,
- flow_id: str,
- owner_label: str,
- ) -> str:
- self.flows = {k: v for k, v in self.flows.items() if v.expires_at > time.time()}
- if flow_id in self.flows:
- raise GitHubError(
- "This authorization has already been used. Connect again."
- )
- for key in list(self.flows):
- flow = self.flows[key]
- if (flow.tenant_id, flow.user_id) == (tenant_id, user_id):
- del self.flows[key]
- if len(self.flows) >= 256:
- raise GitHubError("GitHub connection is busy. Please try again shortly.")
- self.flows[flow_id] = GitHubFlow(
- tenant_id,
- user_id,
- time.time() + 600,
- secrets.token_urlsafe(48),
- secrets.token_urlsafe(32),
- port,
- completion_secret,
- owner_label,
- )
- return flow_id
-
- def flow(self, flow_id: str) -> GitHubFlow:
- flow = self.flows.get(flow_id)
- if flow is None or flow.expires_at <= time.time():
- self.flows.pop(flow_id, None)
- raise GitHubError(
- "Sign-in was interrupted. Start it again from Switch Console."
- )
- return flow
-
- def authorize_url(self, flow_id: str) -> str:
- flow = self.flow(flow_id)
- if flow.status != "pending":
- raise GitHubError(
- "This authorization has already been used. Connect again."
- )
- flow.status = "authorizing"
+ def authorize_url(self, state: str, verifier: str) -> str:
+ """Where the browser signs in at GitHub for the flow under `state`."""
challenge = (
- base64.urlsafe_b64encode(hashlib.sha256(flow.verifier.encode()).digest())
+ base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest())
.decode()
.rstrip("=")
)
@@ -209,7 +145,7 @@ class GitHubConnections:
{
"client_id": self.client_id,
"redirect_uri": self.callback,
- "state": flow_id,
+ "state": state,
"code_challenge": challenge,
"code_challenge_method": "S256",
"prompt": "select_account",
@@ -299,44 +235,27 @@ class GitHubConnections:
"refresh_expires_at": time.time() + result["refresh_token_expires_in"],
}
- async def complete(self, flow_id: str, completion_secret: str, code: str) -> None:
- flow = self.flow(flow_id)
- if (
- not secrets.compare_digest(flow.completion_secret, completion_secret)
- or flow.status != "delivered"
- ):
- raise GitHubError(
- "GitHub authorization could not be verified. Connect again."
- )
- flow.status = "checking"
- try:
- credentials = await self.exchange(
- {
- "code": code,
- "redirect_uri": self.callback,
- "code_verifier": flow.verifier,
- }
- )
- user = await self.request(
- "GET",
- "https://api.github.com/user",
- token=credentials["access_token"],
- data=None,
- )
- if not isinstance(user.get("login"), str) or not isinstance(
- user.get("id"), int
- ):
- raise GitHubError("GitHub did not return a valid account.")
- flow.login = user["login"]
- flow.credentials = {
- **credentials,
- "login": user["login"],
- "user_id": user["id"],
+ async def sign_in(self, code: str, verifier: str) -> dict:
+ """Exchange the code the browser brought back, and read whose account
+ it signed in to: the credentials a connection stores."""
+ credentials = await self.exchange(
+ {
+ "code": code,
+ "redirect_uri": self.callback,
+ "code_verifier": verifier,
}
- flow.status = "ready"
- except (GitHubError, asyncio.CancelledError):
- flow.status = "failed"
- raise
+ )
+ user = await self.request(
+ "GET",
+ "https://api.github.com/user",
+ token=credentials["access_token"],
+ data=None,
+ )
+ if not isinstance(user.get("login"), str) or not isinstance(
+ user.get("id"), int
+ ):
+ raise GitHubError("GitHub did not return a valid account.")
+ return {**credentials, "login": user["login"], "user_id": user["id"]}
async def pages(self, path: str, key: str, token: str) -> list[dict]:
values = []
diff --git a/core/switch_core/provisioning/postgres.py b/core/switch_core/provisioning/postgres.py
index 51e235188..505f87ba8 100644
--- a/core/switch_core/provisioning/postgres.py
+++ b/core/switch_core/provisioning/postgres.py
@@ -7,11 +7,17 @@
next synced; here a live client is woken through the invite bus and joins
itself, which is what keeps its transport watching the room. When nobody is
live the membership is written directly, because a client that is not
- running has nothing to wake and will find the room when it starts.
+ running has nothing to wake and will find the room when it starts. Agents
+ are never live in that sense: nothing reads a room for them, so their
+ membership is always written here, and the arrival goes through the router
+ (`delivery/router.py`), which greets the room on the agent's behalf and hands
+ the arrival to the agents listening for joins.
- **Removal** deletes the membership and wakes a live client so it drops the
- room, mirroring the invitation. No leave event is written: a departure is
- not something a reader of the room needs explained, and the timeline the log
- serves is what was said.
+ room, mirroring the invitation. An agent's own events in the room are
+ deleted in the same transaction, and its connections and controller in this
+ process let go of the room once it commits. No leave event is written: a
+ departure is not something a reader of the room needs explained, and the
+ timeline the log serves is what was said.
- **Discarding a room** is left to the caller's own delete, which cascades.
Nothing here has a second copy to clean up.
"""
@@ -28,8 +34,8 @@ if TYPE_CHECKING:
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.db.stores.client_store import ClientStore
- from switch_core.db.stores.message_store import MessageStore
from switch_core.db.stores.room_store import RoomStore
+ from switch_core.delivery.router import MessageRouter
from switch_core.transport.invites import InviteBus
logger = logging.getLogger(__name__)
@@ -60,13 +66,13 @@ class PostgresProvisioning:
session_factory: async_sessionmaker[AsyncSession],
room_store: RoomStore,
client_store: ClientStore,
- message_store: MessageStore,
+ router: MessageRouter,
invites: InviteBus,
) -> None:
self._session_factory = session_factory
self._room_store = room_store
self._client_store = client_store
- self._message_store = message_store
+ self._router = router
self._invites = invites
async def create_room(self, name: str, topic: str) -> str:
@@ -92,7 +98,7 @@ class PostgresProvisioning:
what has to be exactly-once; a wake-up is cheap and idempotent.
"""
async with self._session_factory() as session:
- switch_room_id, client_id, display_name = await self._resolve(
+ switch_room_id, tenant_id, client_id, display_name = await self._resolve(
session, room_id, user_id
)
existing = await session.get(
@@ -110,7 +116,12 @@ class PostgresProvisioning:
return
async with self._session_factory() as session:
- await self._room_store.add_client(session, client_id, switch_room_id)
+ if not await self._room_store.add_client(
+ session, client_id, switch_room_id
+ ):
+ # Recorded by another writer meanwhile, which wrote the
+ # arrival too.
+ return
arrival = Message(
room_id=switch_room_id,
transport_event_id=f"sw_{uuid.uuid4().hex}",
@@ -124,8 +135,16 @@ class PostgresProvisioning:
thread_root_event_id=None,
content={"membership": "join", "displayname": display_name},
)
- await self._message_store.create(session, arrival, [])
+ routed = await self._router.store(
+ session,
+ tenant_id=tenant_id,
+ room_id=switch_room_id,
+ transport_room_id=room_id,
+ message=arrival,
+ attachments=[],
+ )
await session.commit()
+ routed.committed()
async def kick_user(self, room_id: str, user_id: str) -> None:
"""Remove a membership, stopping a live client's delivery with it.
@@ -146,11 +165,18 @@ class PostgresProvisioning:
the right one carried on reading a room it is not.
"""
async with self._session_factory() as session:
- switch_room_id, client_id, _ = await self._resolve(
+ switch_room_id, tenant_id, client_id, _ = await self._resolve(
session, room_id, user_id
)
await self._room_store.remove_client(session, client_id, switch_room_id)
+ removed = await self._router.remove(
+ session,
+ tenant_id=tenant_id,
+ client_id=client_id,
+ room_id=switch_room_id,
+ )
await session.commit()
+ await removed.committed()
await self._invites.remove(client_id, room_id)
async def delete_room(self, room_id: str) -> None:
@@ -166,7 +192,8 @@ class PostgresProvisioning:
async def _resolve(
self, session: AsyncSession, transport_room_id: str, user_id: str
- ) -> tuple[str, str, str]:
+ ) -> tuple[str, str, str, str]:
+ """The room's id and tenant, and the client's id and name."""
room = await self._room_store.get_by_transport_room_id(
session, transport_room_id
)
@@ -175,4 +202,4 @@ class PostgresProvisioning:
client = await self._client_store.get_by_transport_user_id(session, user_id)
if client is None:
raise ProvisioningError(f"{user_id} is not a Switch client")
- return room.id, client.id, client.display_name
+ return room.id, room.tenant_id, client.id, client.display_name
diff --git a/core/switch_core/room_service.py b/core/switch_core/room_service.py
index 7fafe7f3a..b5549d8e6 100644
--- a/core/switch_core/room_service.py
+++ b/core/switch_core/room_service.py
@@ -664,6 +664,8 @@ class RoomService:
collaboration_core.add_room_mapping(
room.id, transport_room_id, external_channel_id, room.tenant_id
)
+ if bridge_id and external_channel_id:
+ self._collab_lifecycle.rooms_changed(bridge_id)
finally:
if collaboration_core and external_channel_id:
collaboration_core.end_provisioning(external_channel_id)
@@ -855,6 +857,7 @@ class RoomService:
collaboration_core = self._collab_lifecycle.get(bridge_id)
if collaboration_core:
collaboration_core.remove_room_mapping(room.id, room.transport_room_id)
+ self._collab_lifecycle.rooms_changed(bridge_id)
logger.info("Deleted room %s", room_id)
@@ -1252,6 +1255,9 @@ class RoomService:
collaboration_core.add_room_mapping(
room.id, room.transport_room_id, external_channel_id, room.tenant_id
)
+ if external_channel_id:
+ self._collab_lifecycle.rooms_changed(bridge_id)
+ if collaboration_core and external_channel_id:
await self._ensure_channel_capture(
collaboration_core, external_channel_id, channel_type
)
@@ -1274,6 +1280,7 @@ class RoomService:
collaboration_core = self._collab_lifecycle.get(bridge_id)
if collaboration_core:
collaboration_core.remove_room_mapping(room.id, room.transport_room_id)
+ self._collab_lifecycle.rooms_changed(bridge_id)
logger.info("Unlinked bridge from room %s", room_id)
@@ -1395,6 +1402,7 @@ class RoomService:
new_bridge.add_room_mapping(
room_id, transport_room_id, external_channel_id, room.tenant_id
)
+ self._collab_lifecycle.rooms_changed(bridge_id)
finally:
new_bridge.end_provisioning(external_channel_id)
@@ -1421,6 +1429,7 @@ class RoomService:
)
if old_bridge_id:
+ self._collab_lifecycle.rooms_changed(old_bridge_id)
old_bridge = self._collab_lifecycle.get(old_bridge_id)
if old_bridge is not None:
old_bridge.remove_room_mapping(room_id, transport_room_id)
diff --git a/core/switch_core/session_activity/publisher.py b/core/switch_core/session_activity/publisher.py
index fc69b9fdf..3d6b29ab8 100644
--- a/core/switch_core/session_activity/publisher.py
+++ b/core/switch_core/session_activity/publisher.py
@@ -34,7 +34,6 @@ a wait gets exactly that wait.
from __future__ import annotations
import asyncio
-import contextlib
import logging
import secrets
from collections import OrderedDict
@@ -257,12 +256,20 @@ class AgentSessionActivityPublisher:
if self._unsubscribe is not None:
self._unsubscribe()
self._unsubscribe = None
- for task in self._tasks:
+ tasks, self._tasks = self._tasks, []
+ for task in tasks:
task.cancel()
- for task in self._tasks:
- with contextlib.suppress(asyncio.CancelledError):
- await task
- self._tasks = []
+ # A task that had already died is logged rather than raised: the
+ # bridge stopping this publisher goes on to close its platform
+ # connection, and a second stop has nothing left to raise.
+ results = await asyncio.gather(*tasks, return_exceptions=True)
+ for task, result in zip(tasks, results, strict=True):
+ if isinstance(result, Exception):
+ logger.error(
+ "Session activity task %s had stopped on an error",
+ task.get_name(),
+ exc_info=result,
+ )
# ── Listener callbacks: queue only ────────────────────────────────────────
diff --git a/core/switch_core/signals.py b/core/switch_core/signals.py
new file mode 100644
index 000000000..c86be35f0
--- /dev/null
+++ b/core/switch_core/signals.py
@@ -0,0 +1,418 @@
+"""Signals between switch-core processes, over Postgres `NOTIFY`.
+
+A signal says that something changed and who should look: "controller C has a
+new assignment", "controller C was revoked". It never carries the change
+itself. Whoever holds the client the signal is about reads the database, the
+same rule `messages/notify.py` follows for rooms, so a signal that is lost
+costs a delay and never a wrong answer.
+
+Every process listens on one channel. The process that makes a change acts on
+it locally at once and publishes the signal for the others, which skip their
+own: the holder of a controller's socket may be any process, and the database
+is the one path that reaches it wherever it is.
+
+Publishing is batched. A transaction that sends a `NOTIFY` takes a lock that
+every other notifying transaction waits on as it commits, so signals raised
+together go out in one statement rather than one transaction each. Payloads
+are ids and small fields only: `NOTIFY` is not subject to row-level security,
+and any session listening on the channel can read what is sent.
+
+Reconnecting resyncs. Signals sent while the listening connection was down
+were never queued, so every registered resync runs once the connection is back,
+and each re-reads what its clients are owed.
+
+Nothing here is allowed to lose a change for good. A batch that fails to send
+is kept and sent again, as the controller connection ledger keeps its writes.
+And because a signal can still be missed (its sender died between the commit
+and the send, or a receiver was busy restarting), every family also registers
+a periodic re-check that compares what this process holds with the database.
+A missed signal then costs at most `RECHECK_SECONDS` of delay. That is only
+safe because every handler and every re-check re-reads state and is
+idempotent: signals from different processes arrive in no particular order.
+
+Work a signal asks for that needs the database goes through `PendingWork`:
+the handler adds a key and returns at once, one task re-reads each key, and a
+key whose re-read fails is kept and tried again.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import json
+import logging
+from collections.abc import Awaitable, Callable, Hashable
+from typing import TYPE_CHECKING, Any
+
+from sqlalchemy import text
+
+if TYPE_CHECKING:
+ from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker
+
+logger = logging.getLogger(__name__)
+
+SIGNAL_CHANNEL = "switch_signal"
+
+# How often the listening connection is proved alive, and how long the proof may
+# take, as `MessageListener` does.
+HEARTBEAT_SECONDS = 20.0
+HEARTBEAT_TIMEOUT_SECONDS = 60.0
+RECONNECT_BACKOFF_BASE = 1.0
+RECONNECT_BACKOFF_CAP = 15.0
+
+# Postgres refuses a payload of 8000 bytes or more. Ours are a few dozen.
+MAX_PAYLOAD_BYTES = 7999
+
+# How long a batch that failed to send, or a key whose re-read failed, waits
+# before it is tried again.
+RETRY_SECONDS = 2.0
+
+# How many unsent signals are kept while the database refuses them. Past this
+# the oldest are dropped and logged: by then the database has been unreachable
+# for a long while, and the periodic re-checks bring every process back in
+# line once it returns, so holding more would only cost memory.
+MAX_UNSENT = 10_000
+
+# How often every family's re-check runs, so that a signal missed for any
+# reason costs at most this long rather than lasting until a restart.
+RECHECK_SECONDS = 30.0
+
+SignalHandler = Callable[[dict[str, Any]], None]
+Resync = Callable[[], Awaitable[None]]
+Recheck = Callable[[], Awaitable[None]]
+EngineFactory = Callable[[], "AsyncEngine"]
+
+_NOTIFY_MANY = text(
+ "SELECT pg_notify(:channel, payload) FROM unnest(CAST(:payloads AS text[])) AS payload"
+)
+
+
+class PendingWork[K: Hashable]:
+ """Keys whose state has to be re-read, worked one at a time by one task.
+
+ A key added again before its turn is still one piece of work, which is
+ what lets a burst of signals about one thing cost one read. A key whose
+ work raises is kept and tried again after `RETRY_SECONDS`, so a database
+ error on the receiving side delays a change rather than dropping it.
+ """
+
+ def __init__(self, name: str, work: Callable[[K], Awaitable[None]]) -> None:
+ self.name = name
+ self._work = work
+ # A dict rather than a set, so keys are worked in the order they came.
+ self._pending: dict[K, None] = {}
+ self._wake = asyncio.Event()
+
+ def add(self, key: K) -> None:
+ self._pending[key] = None
+ self._wake.set()
+
+ @property
+ def pending(self) -> int:
+ return len(self._pending)
+
+ async def run(self) -> None:
+ """Work keys as they are added, until cancelled."""
+ while True:
+ await self._wake.wait()
+ self._wake.clear()
+ failed = await self.work_all()
+ if failed:
+ logger.error(
+ "%d %s re-read(s) failed; kept and retried in %.0fs",
+ len(failed),
+ self.name,
+ RETRY_SECONDS,
+ )
+ await asyncio.sleep(RETRY_SECONDS)
+ self._wake.set()
+
+ async def work_all(self) -> list[K]:
+ """Work every key now pending. Returns the keys that failed, which
+ are pending again."""
+ failed: list[K] = []
+ while self._pending:
+ key = next(iter(self._pending))
+ del self._pending[key]
+ try:
+ await self._work(key)
+ except asyncio.CancelledError:
+ self._pending.setdefault(key, None)
+ raise
+ except Exception:
+ logger.warning("Re-reading %s %r failed", self.name, key, exc_info=True)
+ failed.append(key)
+ for key in failed:
+ self._pending.setdefault(key, None)
+ return failed
+
+
+class SignalBus:
+ """Publishes signals to every process and hands each process its own."""
+
+ def __init__(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ engine_factory: EngineFactory,
+ process_id: str,
+ ) -> None:
+ self._session_factory = session_factory
+ # Signals carry the sender's process id, and a process skips its own:
+ # whoever publishes has already acted on the change locally.
+ self.process_id = process_id
+ # A factory, as for `MessageListener`: a reconnect builds a fresh engine.
+ self._engine_factory = engine_factory
+ self._handlers: dict[str, SignalHandler] = {}
+ self._resyncs: list[Resync] = []
+ self._rechecks: list[Recheck] = []
+ self._work: list[PendingWork[Any]] = []
+ self._outbox: list[str] = []
+ self._has_outbox = asyncio.Event()
+ self._inbox: asyncio.Queue[dict[str, Any]] = asyncio.Queue()
+ self._tasks: list[asyncio.Task[None]] = []
+ self._running = False
+ self.connected = asyncio.Event()
+
+ # ── Registration ──────────────────────────────────────────────────────────
+
+ def handle(self, kind: str, handler: SignalHandler) -> None:
+ """Run `handler` for every signal of `kind` another process sends.
+
+ Handlers run in the dispatch loop and must not block: they hand the
+ signal to whatever holds the client, which does its own I/O."""
+ if kind in self._handlers:
+ raise ValueError(f"A handler for signal {kind!r} is already registered")
+ self._handlers[kind] = handler
+
+ def on_resync(self, resync: Resync) -> None:
+ """Run `resync` each time the listening connection is (re)established."""
+ self._resyncs.append(resync)
+
+ def on_recheck(self, recheck: Recheck) -> None:
+ """Run `recheck` every `RECHECK_SECONDS` while the bus runs.
+
+ A family's re-check compares what this process holds with the
+ database and repairs any difference, so it must be idempotent and
+ cheap when nothing changed: a read, never a write."""
+ self._rechecks.append(recheck)
+
+ def pending_work[K: Hashable](
+ self, name: str, work: Callable[[K], Awaitable[None]]
+ ) -> PendingWork[K]:
+ """A `PendingWork` whose task runs while the bus does: where a
+ handler puts what it has to read from the database."""
+ pending = PendingWork(name, work)
+ self._work.append(pending)
+ if self._running:
+ self._tasks.append(
+ asyncio.create_task(pending.run(), name=f"signal-work-{name}")
+ )
+ return pending
+
+ # ── Publishing ────────────────────────────────────────────────────────────
+
+ def publish(self, kind: str, **fields: Any) -> None:
+ """Send a signal to every process, after the change it announces has
+ committed. Never blocks the caller: the send is batched, and a batch
+ that fails is kept and sent again rather than raised, since the
+ change itself is already durable."""
+ payload = json.dumps(
+ {"kind": kind, "from": self.process_id, **fields}, separators=(",", ":")
+ )
+ if len(payload.encode()) > MAX_PAYLOAD_BYTES:
+ raise ValueError(
+ f"Signal {kind!r} is too large to send ({len(payload)} bytes)"
+ )
+ self._outbox.append(payload)
+ self._has_outbox.set()
+
+ @property
+ def unsent(self) -> int:
+ return len(self._outbox)
+
+ async def _send_forever(self) -> None:
+ while self._running:
+ await self._has_outbox.wait()
+ self._has_outbox.clear()
+ batch, self._outbox = self._outbox, []
+ if not batch:
+ continue
+ # Set once the batch's transaction has committed, which is when
+ # Postgres sends it: what fails after that (closing the session)
+ # did not stop it going, and keeping it would send it twice.
+ committed = False
+ try:
+ async with self._session_factory() as session:
+ await session.execute(
+ _NOTIFY_MANY, {"channel": SIGNAL_CHANNEL, "payloads": batch}
+ )
+ await session.commit()
+ committed = True
+ except asyncio.CancelledError:
+ if not committed:
+ self._outbox = batch + self._outbox
+ raise
+ except Exception:
+ if committed:
+ logger.warning(
+ "Sent %d signal(s), then closing the session failed",
+ len(batch),
+ exc_info=True,
+ )
+ continue
+ # The receivers' listening connections did not drop, so none
+ # of them will resync: a batch dropped here would be lost
+ # until the next re-check. Kept ahead of anything published
+ # since, so a process still receives one sender's signals in
+ # the order they were raised.
+ self._outbox = batch + self._outbox
+ if len(self._outbox) > MAX_UNSENT:
+ dropped = len(self._outbox) - MAX_UNSENT
+ self._outbox = self._outbox[dropped:]
+ logger.error(
+ "Dropped the %d oldest unsent signal(s); the periodic "
+ "re-checks repair what they announced",
+ dropped,
+ )
+ logger.error(
+ "Could not send %d signal(s); kept and retried in %.0fs",
+ len(self._outbox),
+ RETRY_SECONDS,
+ exc_info=True,
+ )
+ await asyncio.sleep(RETRY_SECONDS)
+ self._has_outbox.set()
+
+ # ── Lifecycle ─────────────────────────────────────────────────────────────
+
+ async def start(self) -> None:
+ if self._running:
+ raise RuntimeError("SignalBus is already started")
+ self._running = True
+ self._tasks = [
+ asyncio.create_task(self._listen_forever(), name="signal-listener"),
+ asyncio.create_task(self._dispatch_forever(), name="signal-dispatch"),
+ asyncio.create_task(self._send_forever(), name="signal-sender"),
+ asyncio.create_task(self._recheck_forever(), name="signal-recheck"),
+ ]
+ self._tasks += [
+ asyncio.create_task(pending.run(), name=f"signal-work-{pending.name}")
+ for pending in self._work
+ ]
+
+ async def stop(self) -> None:
+ self._running = False
+ for task in self._tasks:
+ task.cancel()
+ for task in self._tasks:
+ with contextlib.suppress(asyncio.CancelledError):
+ await task
+ self._tasks = []
+ self.connected.clear()
+
+ # ── The listening connection ──────────────────────────────────────────────
+
+ async def _listen_forever(self) -> None:
+ attempt = 0
+ while self._running:
+ try:
+ await self._listen_once()
+ attempt = 0
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ attempt += 1
+ delay = min(
+ RECONNECT_BACKOFF_BASE * 2 ** (attempt - 1), RECONNECT_BACKOFF_CAP
+ )
+ logger.error(
+ "Signal listener lost its connection; retrying in %.1fs",
+ delay,
+ exc_info=True,
+ )
+ self.connected.clear()
+ await asyncio.sleep(delay)
+
+ async def _listen_once(self) -> None:
+ engine = self._engine_factory()
+ try:
+ async with engine.connect() as connection:
+ raw = await connection.get_raw_connection()
+ driver = raw.driver_connection
+ if driver is None:
+ raise RuntimeError(
+ "No driver connection behind the signal listener; "
+ "LISTEN needs asyncpg directly"
+ )
+ await driver.add_listener(SIGNAL_CHANNEL, self._on_notify)
+ try:
+ self.connected.set()
+ await self._resync_all()
+ await self._hold_open(driver)
+ finally:
+ self.connected.clear()
+ with contextlib.suppress(Exception):
+ await driver.remove_listener(SIGNAL_CHANNEL, self._on_notify)
+ finally:
+ await engine.dispose()
+
+ async def _hold_open(self, driver: Any) -> None:
+ while self._running:
+ await asyncio.sleep(HEARTBEAT_SECONDS)
+ await driver.execute("SELECT 1", timeout=HEARTBEAT_TIMEOUT_SECONDS)
+
+ def _on_notify(
+ self, _connection: Any, _pid: int, _channel: str, payload: str
+ ) -> None:
+ """asyncpg's callback, in the connection's read loop: queue and return."""
+ try:
+ signal = json.loads(payload)
+ if not isinstance(signal, dict) or not isinstance(signal.get("kind"), str):
+ raise ValueError("no kind")
+ except ValueError:
+ logger.warning("Ignoring an unreadable signal: %r", payload[:200])
+ return
+ self._inbox.put_nowait(signal)
+
+ async def _recheck_forever(self) -> None:
+ while self._running:
+ await asyncio.sleep(RECHECK_SECONDS)
+ await self.recheck_all()
+
+ async def recheck_all(self) -> None:
+ """Run every family's re-check once, one after another so they never
+ compete for the pool, each failure logged and left to the next run."""
+ for recheck in self._rechecks:
+ try:
+ await recheck()
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ logger.error("A signal re-check failed", exc_info=True)
+
+ async def _resync_all(self) -> None:
+ for resync in self._resyncs:
+ try:
+ await resync()
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ logger.error("A signal resync failed", exc_info=True)
+
+ # ── Dispatch ──────────────────────────────────────────────────────────────
+
+ async def _dispatch_forever(self) -> None:
+ while self._running:
+ signal = await self._inbox.get()
+ if signal.get("from") == self.process_id:
+ continue
+ handler = self._handlers.get(signal["kind"])
+ if handler is None:
+ # A newer process may send kinds an older one does not know,
+ # during a rolling deploy.
+ logger.debug("No handler for signal %r", signal["kind"])
+ continue
+ try:
+ handler(signal)
+ except Exception:
+ logger.error("Handling signal %r failed", signal["kind"], exc_info=True)
diff --git a/core/switch_core/telemetry/messages.py b/core/switch_core/telemetry/messages.py
index a68497903..82077aebd 100644
--- a/core/switch_core/telemetry/messages.py
+++ b/core/switch_core/telemetry/messages.py
@@ -4,9 +4,9 @@ Everything else in the catalogue fires a handful of times a day. These fire
once per message, so they are the only events where reporting could slow the
thing being reported on. Three rules keep it off that path:
-- **The sender never waits.** The transport and the agent consumer hand over a
- small record and return. One worker task looks up what the event needs — the
- room's platform, type and membership, who the sender is — and emits.
+- **The sender never waits.** The transport and the router hand over a small
+ record and return. One worker task looks up what the event needs (the
+ room's platform, type and membership, who the sender is) and emits.
- **Lookups are cached.** A busy room says a lot and changes membership
rarely, so a room is read at most once per `_CACHE_TTL_SECONDS` rather than
once per message. The member counts can therefore be that far behind. A
@@ -145,9 +145,9 @@ class MessageTelemetry:
"""Reports `room_message_sent`, `agent_message_sent` and
`agent_message_received`.
- The transport's `ParticipantMessageObserver`, and what the agent consumer
- tells when a message is addressed to its agent. Does nothing at all — no
- queue, no lookups — while telemetry is off.
+ The transport's `ParticipantMessageObserver`, and what the router tells
+ once a message addressed to an agent has committed. Does nothing at all
+ (no queue, no lookups) while telemetry is off.
"""
def __init__(
diff --git a/core/switch_core/telemetry/reporter.py b/core/switch_core/telemetry/reporter.py
index 701b40b36..8db62e051 100644
--- a/core/switch_core/telemetry/reporter.py
+++ b/core/switch_core/telemetry/reporter.py
@@ -3,6 +3,11 @@
Anchored to a watermark rather than to uptime, so restarts do not change the
cadence.
+Every switch-core process runs this loop, so the watermark is claimed before
+anything is sent: a conditional write moves it to now only while it is still
+due, and only the process whose write took sends. Two processes polling in the
+same minute would otherwise both read "due" and both send the day's snapshot.
+
The first pass sends the snapshot but not the room-activation events: with no
watermark, "first interaction since then" means every room that ever went
active, arriving at once and all dated today.
@@ -15,7 +20,7 @@ import logging
from collections.abc import Callable
from datetime import UTC, datetime, timedelta
-from sqlalchemy import select
+from sqlalchemy import delete, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -78,13 +83,22 @@ class SnapshotReporter:
await asyncio.sleep(_POLL_INTERVAL_SECONDS)
async def run_once_if_due(self) -> bool:
- """Send a snapshot if one is due. True if one was sent."""
+ """Send a snapshot if one is due and no other process has claimed
+ it. True if this process sent it."""
now = datetime.now(UTC)
last_sent = await self._read_watermark()
if last_sent is not None and now - last_sent < self._interval:
return False
- await self.run_once(now=now, since=last_sent)
- await self._write_watermark(now)
+ if not await self._claim(now, last_sent):
+ return False
+ try:
+ await self.run_once(now=now, since=last_sent)
+ except BaseException:
+ # Nothing was sent: `run_once` raises only before its emit. The
+ # claim is handed back so the next poll, here or in another
+ # process, tries again rather than skipping the day.
+ await self._release(now, last_sent)
+ raise
await self._prune(now)
return True
@@ -115,9 +129,9 @@ class SnapshotReporter:
),
)
- # And nothing after that emit may raise out of this method. The
- # watermark advances only when `run_once` returns, and the loop retries
- # every poll interval — so a raise here would re-send the same snapshot
+ # And nothing after that emit may raise out of this method. A raise
+ # hands the claimed watermark back, and the loop retries every poll
+ # interval, so a raise here would re-send the same snapshot
# every five minutes for as long as the condition lasted, turning one
# lost pass into hundreds of duplicates of a figure meant to be daily.
# The rooms are the part that is given up instead: the window has moved
@@ -191,13 +205,45 @@ class SnapshotReporter:
stamp = row.last_sent_at
return stamp.replace(tzinfo=UTC) if stamp.tzinfo is None else stamp
- async def _write_watermark(self, when: datetime) -> None:
+ async def _claim(self, now: datetime, last_sent: datetime | None) -> bool:
+ """Move the watermark from `last_sent` to `now`, if no other process
+ has moved it since this one read it. True if this process did."""
async with self._session_factory() as session:
- await session.execute(
- pg_insert(TelemetrySnapshotWatermark)
- .values(id=1, last_sent_at=when)
- .on_conflict_do_update(
- index_elements=["id"], set_={"last_sent_at": when}
+ if last_sent is None:
+ result = await session.execute(
+ pg_insert(TelemetrySnapshotWatermark)
+ .values(id=1, last_sent_at=now)
+ .on_conflict_do_nothing(index_elements=["id"])
+ .returning(TelemetrySnapshotWatermark.id)
+ )
+ else:
+ result = await session.execute(
+ update(TelemetrySnapshotWatermark)
+ .where(
+ TelemetrySnapshotWatermark.id == 1,
+ TelemetrySnapshotWatermark.last_sent_at == last_sent,
+ )
+ .values(last_sent_at=now)
+ .returning(TelemetrySnapshotWatermark.id)
+ )
+ claimed = result.first() is not None
+ await session.commit()
+ return claimed
+
+ async def _release(self, claimed_at: datetime, last_sent: datetime | None) -> None:
+ """Put the watermark back where it was, unless something has moved it
+ on since this process claimed it."""
+ async with self._session_factory() as session:
+ if last_sent is None:
+ await session.execute(
+ delete(TelemetrySnapshotWatermark).where(
+ TelemetrySnapshotWatermark.last_sent_at == claimed_at
+ )
+ )
+ else:
+ await session.execute(
+ update(TelemetrySnapshotWatermark)
+ .where(TelemetrySnapshotWatermark.last_sent_at == claimed_at)
+ .values(last_sent_at=last_sent)
)
- )
await session.commit()
diff --git a/core/switch_core/telemetry/sink.py b/core/switch_core/telemetry/sink.py
index 0e5f7e38d..77cd2af8a 100644
--- a/core/switch_core/telemetry/sink.py
+++ b/core/switch_core/telemetry/sink.py
@@ -329,6 +329,7 @@ def _resource_from(record: TelemetryRecord) -> OtlpResource:
deployment_id=record.resource["flint.client_id"],
commit_sha=None,
repository_url=None,
+ instance_id=None,
)
diff --git a/core/switch_core/template_runs.py b/core/switch_core/template_runs.py
index 2a1d61f0d..47de16a9e 100644
--- a/core/switch_core/template_runs.py
+++ b/core/switch_core/template_runs.py
@@ -491,9 +491,10 @@ class TemplateAgents:
async def _listening(self, created: list[CreatedAgent]) -> list[CreatedAgent]:
"""Wait, up to the listen timeout in all, for each new agent's client
- to be taking events. A client that starts after a room exists reads
- it only from then on, so without this the kickoff posted the moment
- the room is made never reaches an agent created a moment before."""
+ to be taking events, so the kickoff posted the moment the room is made
+ reaches an agent created a moment before. An agent's client is taking
+ events as soon as it exists (`AgentActor.wait_receiving`): the rows of
+ its rooms are decided for it as they are written."""
loop = asyncio.get_running_loop()
deadline = loop.time() + self._listen_timeout
out: list[CreatedAgent] = []
diff --git a/core/switch_core/transport/__init__.py b/core/switch_core/transport/__init__.py
index 0f846c78d..76e35e76f 100644
--- a/core/switch_core/transport/__init__.py
+++ b/core/switch_core/transport/__init__.py
@@ -5,6 +5,7 @@ Only modules inside this package may import a transport library — everything
else talks to the port and the neutral types in `types`.
"""
+from switch_core.transport.derived_ids import writes_derived_from
from switch_core.transport.port import (
Handler,
MessageTransport,
@@ -45,4 +46,5 @@ __all__ = [
"TransportError",
"TransportHandlers",
"UploadResult",
+ "writes_derived_from",
]
diff --git a/core/switch_core/transport/cursors.py b/core/switch_core/transport/cursors.py
new file mode 100644
index 000000000..e7398ab23
--- /dev/null
+++ b/core/switch_core/transport/cursors.py
@@ -0,0 +1,344 @@
+"""Where a consumer starts reading each room, and how far it has got.
+
+A consumer reads its rooms through a `DeliveryProgress`, which says where to
+start in each room and hears how far the consumer has got. There are two.
+
+**`ConsumerCursors`, for a consumer one process owns.** The consumer reads its
+rooms from `delivery_cursors`, one row per membership, so the process that
+owns it (`ownership.py`) starts where the last owner stopped rather than at
+each room's newest row. Rows written while no process was reading are handed
+over when one is, and a hand-over between processes neither repeats nor skips
+what the old owner had finished. Progress is written after the effects it
+covers, so a row whose effects were made but whose cursor was not written is
+handled again by the next owner, and whatever runs under ownership makes its
+effects safe to repeat.
+
+**`FromHead`, for a reader nothing is stored for.** Each room is read from its
+newest row, as it was before cursors were stored, and progress lives only in
+the transport's memory. Every consumer is owned now and reads through
+`ConsumerCursors`; this is for a transport read outside any ownership, as the
+transport's own tests read.
+
+**One writer per process.** Every consumer reads every new row in its rooms,
+so a stored cursor moves for every owned consumer with every message. Written
+one at a time, that is a pool checkout per consumer per message, which is what
+the shared room read (`room_cache.py`) exists to avoid on the read side. The
+writer takes everything pending, writes it in one statement per tenant, and
+takes what has piled up meanwhile next, so the writes cost at most one
+connection however many consumers there are.
+
+**Fenced.** Each advance names the lease its consumer was started under, and
+the statement writes only where this process still holds it under that token
+(`DeliveryCursorStore.advance`). An advance refused because the lease moved is
+reported to the ownership, which stops the consumer here.
+
+**A cap on what a new owner replays.** A consumer resuming a room skips
+through every row older than `BACKLOG_MAX_AGE`: an hour-old refusal or "I am
+offline" posted now would only mislead. The skip is logged and counted, and
+the rows it passed are still there to read.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from collections.abc import Callable, Collection
+from datetime import timedelta
+from typing import TYPE_CHECKING, Protocol
+
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.delivery_cursor_store import CursorAdvance
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.observability.catalogue import DELIVERY_BACKLOG_SKIPPED
+from switch_core.observability.metrics import metrics
+
+if TYPE_CHECKING:
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+ from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
+ from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+ from switch_core.ownership import Fence
+
+logger = logging.getLogger(__name__)
+
+# How old a row may be and still be handled by a consumer that resumes past
+# it. Long enough to cover a deploy or a hand-over after a process died, short
+# enough that what is answered is still being waited for.
+BACKLOG_MAX_AGE = timedelta(minutes=10)
+
+# How long a batch the database refused waits before it is written again.
+WRITE_RETRY_SECONDS = 2.0
+
+
+class DeliveryProgress(Protocol):
+ """Where one consumer run starts each room, and where it records how far
+ it got."""
+
+ async def resume(
+ self, session: AsyncSession, room_ids: Collection[str]
+ ) -> dict[str, int]:
+ """Where to read each of `room_ids` from, in the caller's session. A
+ room left out of the answer is one the client is no longer in."""
+ ...
+
+ async def resume_joined(
+ self, session: AsyncSession, room_id: str, arrival_seq: int
+ ) -> int:
+ """Where to read a room from that the client has just joined itself,
+ writing its arrival at `arrival_seq`: from before the arrival, so the
+ client reads its own join the way every other member does."""
+ ...
+
+ def advance(self, room_id: str, seq: int) -> None:
+ """Every row of the room through `seq` is handled."""
+ ...
+
+
+class FromHead:
+ """For a reader nothing is stored for: each room from its newest row.
+
+ Nothing is stored. A process that starts reads only what is written from
+ then on, and every process reads the same rows for itself.
+ """
+
+ def __init__(self, messages: MessageStore) -> None:
+ self._messages = messages
+
+ async def resume(
+ self, session: AsyncSession, room_ids: Collection[str]
+ ) -> dict[str, int]:
+ return await self._messages.head_seqs(session, list(room_ids))
+
+ async def resume_joined(
+ self, session: AsyncSession, room_id: str, arrival_seq: int
+ ) -> int:
+ return arrival_seq - 1
+
+ def advance(self, room_id: str, seq: int) -> None:
+ return None
+
+
+class ConsumerCursors:
+ """For a consumer one process owns: one client, under one hold of its
+ lease, reading from the client's stored cursors.
+
+ Built when the process starts the consumer and dropped when it stops it,
+ so a later run here, under a newer token, never shares one.
+ """
+
+ def __init__(
+ self,
+ *,
+ writer: CursorWriter,
+ store: DeliveryCursorStore,
+ tenant_id: str,
+ client_id: str,
+ fence: Fence,
+ actor_role: str,
+ on_lost: Callable[[Fence], None],
+ ) -> None:
+ self._writer = writer
+ self._store = store
+ self.tenant_id = tenant_id
+ self.client_id = client_id
+ self.fence = fence
+ self._actor_role = actor_role
+ self.on_lost = on_lost
+
+ async def resume(
+ self, session: AsyncSession, room_ids: Collection[str]
+ ) -> dict[str, int]:
+ """The stored cursor, or for a membership that has none (one written
+ before cursors were, by something that does not write them) the
+ room's newest row, stored now. Rows past the cursor older than
+ `BACKLOG_MAX_AGE` are skipped, and the skip is recorded as progress.
+ """
+ if not room_ids:
+ return {}
+ places = await self._store.positions(session, self.client_id, room_ids)
+ missing = [room_id for room_id in room_ids if room_id not in places]
+ if missing:
+ await self._store.seed_memberships(session, self.client_id, missing)
+ places.update(await self._store.positions(session, self.client_id, missing))
+ await session.commit()
+ ends = await self._store.backlog_ends(session, places, BACKLOG_MAX_AGE)
+ for room_id, end in sorted(ends.items()):
+ if end <= places[room_id]:
+ continue
+ skipped = end - places[room_id]
+ logger.warning(
+ "Client %s resumes room %s past %d row(s) older than %s "
+ "(through seq %d): too old to act on now. They stay in the "
+ "room's history.",
+ self.client_id,
+ room_id,
+ skipped,
+ BACKLOG_MAX_AGE,
+ end,
+ )
+ metrics().increment(
+ DELIVERY_BACKLOG_SKIPPED, {"actor": self._actor_role}, skipped
+ )
+ places[room_id] = end
+ self.advance(room_id, end)
+ return places
+
+ async def resume_joined(
+ self, session: AsyncSession, room_id: str, arrival_seq: int
+ ) -> int:
+ """The stored cursor, which the membership was written with, just
+ before the arrival (`RoomStore.add_client`)."""
+ places = await self.resume(session, [room_id])
+ return places.get(room_id, arrival_seq - 1)
+
+ def advance(self, room_id: str, seq: int) -> None:
+ """Written shortly, by the process's writer."""
+ self._writer.advance(self, room_id, seq)
+
+ async def flush(self) -> None:
+ """Write what is pending now, for a consumer about to stop."""
+ await self._writer.flush()
+
+
+class CursorWriter:
+ """Writes every owned consumer's progress in this process, in batches."""
+
+ def __init__(
+ self,
+ *,
+ session_factory: async_sessionmaker[AsyncSession],
+ cursors: DeliveryCursorStore,
+ leases: OwnershipLeaseStore,
+ process_id: str,
+ ) -> None:
+ self._session_factory = session_factory
+ self._cursors = cursors
+ self._leases = leases
+ self._process_id = process_id
+ # The newest place each (client, room) has reached and not yet been
+ # written, with the consumer run that reached it.
+ self._pending: dict[tuple[str, str], tuple[ConsumerCursors, int]] = {}
+ self._wake = asyncio.Event()
+ # One write at a time, whether the loop's or a flush's: two would
+ # each write half of what the other has taken.
+ self._lock = asyncio.Lock()
+
+ def advance(self, consumer: ConsumerCursors, room_id: str, seq: int) -> None:
+ key = (consumer.client_id, room_id)
+ current = self._pending.get(key)
+ if current is None or current[0] is not consumer or current[1] < seq:
+ self._pending[key] = (consumer, seq)
+ self._wake.set()
+
+ @property
+ def pending(self) -> int:
+ return len(self._pending)
+
+ async def run(self) -> None:
+ """Write whatever is pending, as soon as it is, until cancelled."""
+ while True:
+ await self._wake.wait()
+ self._wake.clear()
+ try:
+ await self.flush()
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ logger.error(
+ "Writing %d delivery cursor(s) failed; kept and retried in %.0fs",
+ len(self._pending),
+ WRITE_RETRY_SECONDS,
+ exc_info=True,
+ )
+ await asyncio.sleep(WRITE_RETRY_SECONDS)
+ self._wake.set()
+
+ async def flush(self) -> None:
+ """Write everything pending now. Raises if a tenant's batch could not
+ be written, after keeping it to be written again."""
+ async with self._lock:
+ if not self._pending:
+ return
+ batch, self._pending = self._pending, {}
+ by_tenant: dict[
+ str, dict[tuple[str, str], tuple[ConsumerCursors, int]]
+ ] = {}
+ for key, entry in batch.items():
+ by_tenant.setdefault(entry[0].tenant_id, {})[key] = entry
+ failed: BaseException | None = None
+ for tenant_id, entries in by_tenant.items():
+ try:
+ await self._write(tenant_id, entries)
+ except Exception as exc:
+ self._keep(entries)
+ failed = exc
+ if failed is not None:
+ raise failed
+
+ def _keep(
+ self, entries: dict[tuple[str, str], tuple[ConsumerCursors, int]]
+ ) -> None:
+ """Put entries that were not written back, unless something newer from
+ the same consumer run arrived meanwhile."""
+ for key, (consumer, seq) in entries.items():
+ current = self._pending.get(key)
+ if current is None or (current[0] is consumer and current[1] < seq):
+ self._pending[key] = (consumer, seq)
+
+ async def _write(
+ self,
+ tenant_id: str,
+ entries: dict[tuple[str, str], tuple[ConsumerCursors, int]],
+ ) -> None:
+ advances = [
+ CursorAdvance(
+ client_id=client_id,
+ room_id=room_id,
+ seq=seq,
+ kind=consumer.fence.kind,
+ key=consumer.fence.key,
+ token=consumer.fence.token,
+ )
+ for (client_id, room_id), (consumer, seq) in entries.items()
+ ]
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ written = await self._cursors.advance(
+ session, process_id=self._process_id, advances=advances
+ )
+ await session.commit()
+ refused = {
+ consumer.fence: consumer
+ for key, (consumer, _) in entries.items()
+ if key not in written
+ }
+ still_held = await self._still_held(session, refused)
+ for fence, consumer in refused.items():
+ if fence in still_held:
+ # The lease is still this process's, so the membership went:
+ # the membership signal stops the room's delivery.
+ continue
+ logger.warning(
+ "Client %s's cursor was refused: its %s lease is no longer "
+ "this process's (token %d)",
+ consumer.client_id,
+ fence.kind,
+ fence.token,
+ )
+ consumer.on_lost(fence)
+
+ async def _still_held(
+ self, session: AsyncSession, refused: dict[Fence, ConsumerCursors]
+ ) -> set[Fence]:
+ by_kind: dict[str, dict[str, Fence]] = {}
+ for fence in refused:
+ by_kind.setdefault(fence.kind, {})[fence.key] = fence
+ held: set[Fence] = set()
+ for kind, fences in by_kind.items():
+ keys = await self._leases.held(
+ session,
+ kind=kind,
+ tokens={key: fence.token for key, fence in fences.items()},
+ process_id=self._process_id,
+ )
+ held.update(fences[key] for key in keys)
+ return held
diff --git a/core/switch_core/transport/derived_ids.py b/core/switch_core/transport/derived_ids.py
new file mode 100644
index 000000000..8b13a8a23
--- /dev/null
+++ b/core/switch_core/transport/derived_ids.py
@@ -0,0 +1,70 @@
+"""Ids for rows a client writes on behalf of something that may arrive twice.
+
+A consumer may handle a row twice: an owner that stopped between acting on a
+row and recording its progress leaves the row to the next owner. A messaging
+platform may deliver one post twice, as Slack sends an event again when its
+acknowledgement did not arrive. Every row the client writes meanwhile is given
+an id derived from what it is acting on and how many it has written so far,
+and `messages.transport_event_id` is unique, so a second run finds the first
+one's writes and writes nothing new.
+
+Held in a context variable, so a task started inside the block carries it too
+and its writes count as the source's. Only the named client's writes are
+derived: another client writing inside the block writes as usual.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import hashlib
+from collections.abc import Iterator
+from contextvars import ContextVar
+from dataclasses import dataclass, field
+
+
+def derived_event_id(client_id: str, source_id: str, ordinal: int) -> str:
+ """The id of the `ordinal`-th row a client writes on behalf of
+ `source_id`: the same every time. Shaped like a fresh event id (`sw_` and
+ 32 hex digits), so nothing can tell the two apart."""
+ digest = hashlib.sha256(
+ f"{client_id}\x1f{source_id}\x1f{ordinal}".encode()
+ ).hexdigest()
+ return f"sw_{digest[:32]}"
+
+
+@dataclass
+class DerivedWrites:
+ """What one client is writing on behalf of, while it does."""
+
+ client_id: str
+ source_id: str
+ written: int = field(default=0)
+
+ def next_event_id(self) -> str:
+ event_id = derived_event_id(self.client_id, self.source_id, self.written)
+ self.written += 1
+ return event_id
+
+
+_DERIVED: ContextVar[DerivedWrites | None] = ContextVar("derived_writes", default=None)
+
+
+def writes_derived_from(client_id: str, source_id: str) -> Iterator[None]:
+ """Give the rows `client_id` writes inside the block ids derived from
+ `source_id`: the row its consumer is handling, or a post a messaging
+ platform may deliver again."""
+ token = _DERIVED.set(DerivedWrites(client_id=client_id, source_id=source_id))
+ try:
+ yield
+ finally:
+ _DERIVED.reset(token)
+
+
+def derived_writes(client_id: str) -> DerivedWrites | None:
+ """What `client_id` is writing on behalf of right now, or None when its
+ writes get fresh ids."""
+ current = _DERIVED.get()
+ if current is None or current.client_id != client_id:
+ return None
+ return current
diff --git a/core/switch_core/transport/ephemeral.py b/core/switch_core/transport/ephemeral.py
index 3c4199381..1c9823d79 100644
--- a/core/switch_core/transport/ephemeral.py
+++ b/core/switch_core/transport/ephemeral.py
@@ -18,10 +18,11 @@ reason. A position is right when the value is durable and the reader can go and
fetch it. Presence has nowhere to fetch from, so the announcement has to be the
value, and a missed one costs a briefly stale indicator rather than data.
-**It is in-process, the same limitation `InviteBus` documents**, and it should
-be lifted at the same time and in the same way. A second replica needs a
-cross-process signal for both, which is why switch-core runs as a single
-replica until then.
+**It is in-process.** Membership changes, which used to share this
+limitation, now reach every process through a trigger on `client_rooms` and
+the signal bus (`transport/membership_signals.py`). Presence has not followed
+yet: a second replica needs it carried the same way (`signals.py`), which is
+one reason switch-core runs as a single replica until then.
"""
from __future__ import annotations
diff --git a/core/switch_core/transport/invites.py b/core/switch_core/transport/invites.py
index 5f3ab05f1..cfe638bd3 100644
--- a/core/switch_core/transport/invites.py
+++ b/core/switch_core/transport/invites.py
@@ -18,12 +18,18 @@ room. Here the subscription is the client's own, so deleting the membership row
leaves it reading a room it is no longer in. A removal must therefore be rung
as well as written.
-**It is in-process, and that is a real limitation.** A client running in
-another replica of switch-core would not hear it. It is one of the reasons
-switch-core is single-replica today (the Helm chart refuses anything else);
-lifting it is its own step. When nobody is listening for a client the caller
-writes the membership itself, so the room is still joined the next time that client
-starts — the invitation is a wake-up, never the record.
+**This bus is in-process; the other processes hear through the table.** It
+wakes the copy of a client running in the process that served the request,
+which is the copy that joins first and hears its own arrival. Every
+switch-core process runs its own copy of each client, and those learn of the
+change from the `membership.changed` signal a trigger on `client_rooms` sends
+(`db/membership_notify_ddl.py`, `transport/membership_signals.py`): each
+re-reads its client's rows and starts or stops reading the rooms that differ,
+through the third handler a client registers here. The same re-read runs
+periodically, so a signal missed for any reason costs a delay rather than a
+room read by someone removed from it. When nobody is listening for a client
+the caller writes the membership itself, so the room is still joined the next
+time that client starts: the invitation is a wake-up, never the record.
**Keyed by client id, which is a primary key.** It used to be keyed by
`matrix_user_id`, and that was wrong the moment a second tenant existed:
@@ -46,6 +52,9 @@ from collections.abc import Awaitable, Callable
logger = logging.getLogger(__name__)
InviteHandler = Callable[[str], Awaitable[None]]
+# Re-read the client's memberships and start or stop reading the rooms that
+# differ from what it reads now.
+MembershipReconcile = Callable[[], Awaitable[None]]
class InviteBus:
@@ -59,27 +68,38 @@ class InviteBus:
def __init__(self) -> None:
self._handlers: dict[str, InviteHandler] = {}
self._removal_handlers: dict[str, InviteHandler] = {}
+ self._reconciles: dict[str, MembershipReconcile] = {}
def register(
- self, client_id: str, handler: InviteHandler, removal_handler: InviteHandler
+ self,
+ client_id: str,
+ handler: InviteHandler,
+ removal_handler: InviteHandler,
+ reconcile: MembershipReconcile,
) -> None:
- """Make this pair the live listeners for `client_id`, displacing any other.
+ """Make these the live listeners for `client_id`, displacing any other.
One slot per client, so a client that starts while an older instance of
itself is still unwinding takes the slot from it. That is the right
answer — the newer one is the client with a live receive loop — and it
is why `unregister` wants the handlers back.
- Both arms are taken together and never separately. A client registered
+ All arms are taken together and never separately. A client registered
for invitations but not removals is the shape of a leak — it would be
told about every room it gains and none it loses — so the state is made
- unrepresentable here rather than guarded against at every use.
+ unrepresentable here rather than guarded against at every use. The
+ reconcile arm is how a change made in another process reaches it.
"""
self._handlers[client_id] = handler
self._removal_handlers[client_id] = removal_handler
+ self._reconciles[client_id] = reconcile
def unregister(
- self, client_id: str, handler: InviteHandler, removal_handler: InviteHandler
+ self,
+ client_id: str,
+ handler: InviteHandler,
+ removal_handler: InviteHandler,
+ reconcile: MembershipReconcile,
) -> None:
"""Give the slot up, but only if these handlers still hold it.
@@ -97,6 +117,22 @@ class InviteBus:
del self._handlers[client_id]
if self._removal_handlers.get(client_id) == removal_handler:
del self._removal_handlers[client_id]
+ if self._reconciles.get(client_id) == reconcile:
+ del self._reconciles[client_id]
+
+ def client_ids(self) -> list[str]:
+ """The clients with a live listener in this process."""
+ return list(self._reconciles)
+
+ def is_listening(self, client_id: str) -> bool:
+ return client_id in self._reconciles
+
+ async def reconcile(self, client_id: str) -> None:
+ """Have `client_id`, if it is running here, re-read its memberships
+ and start or stop reading the rooms that changed."""
+ reconcile = self._reconciles.get(client_id)
+ if reconcile is not None:
+ await reconcile()
async def invite(self, client_id: str, transport_room_id: str) -> bool:
"""Tell `client_id` it is in `transport_room_id`.
diff --git a/core/switch_core/transport/membership_signals.py b/core/switch_core/transport/membership_signals.py
new file mode 100644
index 000000000..0e59b86a1
--- /dev/null
+++ b/core/switch_core/transport/membership_signals.py
@@ -0,0 +1,75 @@
+"""Room membership changes, reaching the copy of each client in every process.
+
+Every switch-core process runs its own copy of each client, and each copy
+reads the rooms its client is a member of. `InviteBus` wakes only the copy in
+the process that served the change. The others hear of it from the
+`membership.changed` signal that a trigger on `client_rooms` sends for every
+row inserted or deleted (`db/membership_notify_ddl.py`), and the copy here
+re-reads its client's rows and starts or stops reading the rooms that differ
+(`PostgresTransport._reconcile_membership`). The process that wrote the row
+hears its own signal too; its copy already reads the room, or is joining it,
+so the re-read changes nothing.
+
+A removed member has to stop receiving the room on every process, so a missed
+signal must not be the end of it. A re-read that fails is kept and tried
+again, every client running here is re-read when the signal connection comes
+back, and again on each periodic re-check.
+
+An agent reads no room, so it has no copy here to wake. What an agent's
+membership changes in this process is what is held in its memory: the room
+claims of the agent's connections held here, and the rooms of a controller
+streaming it from here. Every change is handed to the router for that
+(`MessageRouter.membership_changed`), and retried the same way when it fails.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING, Any
+
+from switch_core.db.membership_notify_ddl import MEMBERSHIP_CHANGED_SIGNAL
+
+if TYPE_CHECKING:
+ from switch_core.delivery.router import MessageRouter
+ from switch_core.signals import SignalBus
+ from switch_core.transport.invites import InviteBus
+
+logger = logging.getLogger(__name__)
+
+
+class MembershipSignals:
+ def __init__(
+ self, *, invites: InviteBus, bus: SignalBus, router: MessageRouter
+ ) -> None:
+ self._invites = invites
+ self._owed = bus.pending_work("memberships", invites.reconcile)
+ self._agents_owed = bus.pending_work(
+ "agent memberships", router.membership_changed
+ )
+ bus.handle(MEMBERSHIP_CHANGED_SIGNAL, self._on_changed)
+ bus.on_resync(self.reconcile_all)
+ bus.on_recheck(self.reconcile_all)
+
+ def _on_changed(self, signal: dict[str, Any]) -> None:
+ client_id = signal["client_id"]
+ if self._invites.is_listening(client_id):
+ self._owed.add(client_id)
+ return
+ # Not a client that reads rooms here: an agent, whose connections and
+ # controller this process may hold, or a person's actor, which the
+ # router finds is no agent and leaves alone.
+ self._agents_owed.add((signal["tenant_id"], client_id, signal["room_id"]))
+
+ async def reconcile_all(self) -> None:
+ """Re-read every client running here, one after another so a
+ re-check never asks the pool for more than one connection."""
+ for client_id in self._invites.client_ids():
+ try:
+ await self._invites.reconcile(client_id)
+ except Exception:
+ logger.warning(
+ "Could not re-read the memberships of client %s; retrying",
+ client_id,
+ exc_info=True,
+ )
+ self._owed.add(client_id)
diff --git a/core/switch_core/transport/port.py b/core/switch_core/transport/port.py
index 777b8a010..a7fc91e57 100644
--- a/core/switch_core/transport/port.py
+++ b/core/switch_core/transport/port.py
@@ -14,7 +14,7 @@ from __future__ import annotations
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
-from typing import Any, Protocol, runtime_checkable
+from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
from switch_core.transport.types import (
DownloadResult,
@@ -24,7 +24,13 @@ from switch_core.transport.types import (
UploadResult,
)
+if TYPE_CHECKING:
+ from switch_core.transport.cursors import DeliveryProgress
+
Handler = Callable[[Any, Any], Awaitable[None]]
+# Given a room's transport id, the lowest room position whose effect the
+# client has not made yet, or None when it holds nothing back.
+HeldFrom = Callable[[str], int | None]
@dataclass
@@ -45,6 +51,11 @@ class TransportHandlers:
# the room by the time this fires; it is for whatever downstream still
# holds the room's events, which the transport knows nothing about.
on_removed: Handler | None = None
+ # Rows a client has been handed but not acted on yet, such as the files of
+ # a multi-file message waiting for the rest. Progress is not recorded past
+ # them, so the next reader of the client handles them if this one stops
+ # first.
+ held_from: HeldFrom | None = None
@runtime_checkable
@@ -65,8 +76,15 @@ class MessageTransport(Protocol):
"""Bind inbound callbacks. Call before `receive_forever`."""
...
- async def receive_forever(self) -> None:
- """Deliver inbound events to the registered handlers until closed."""
+ async def receive_forever(self, progress: DeliveryProgress) -> None:
+ """Deliver inbound events to the registered handlers until closed,
+ starting each room where `progress` says and recording through it how
+ far the client got."""
+ ...
+
+ def held_back_released(self, room_id: str) -> None:
+ """The client acted on rows of `room_id` it had been holding back
+ (`TransportHandlers.held_from`): record the progress waiting on them."""
...
async def wait_receiving(self, timeout: float) -> bool:
diff --git a/core/switch_core/transport/postgres.py b/core/switch_core/transport/postgres.py
index 417e989ff..558e36e68 100644
--- a/core/switch_core/transport/postgres.py
+++ b/core/switch_core/transport/postgres.py
@@ -3,12 +3,16 @@
A transport carries an event from the client that sent it to the clients that
should see it, and remembers it in between. Here the `messages` table does all
of that. **The write is the send.** A send is an INSERT, numbered per room
-under an advisory lock (`MessageStore.create`); the `messages_notify` trigger
-announces it on commit (`db/notify_ddl.py`); one `MessageListener` wakes the
-transports watching that room (`messages/notify.py`); and each transport reads
-the rows after its own cursor, `DELIVERY_PAGE` at a time. There is no second
-store to agree with. Switch used to run on a Matrix homeserver; the
-`matrix_*` column names are what is left of it, kept as stable ids.
+under an advisory lock (`MessageStore.create`), made by the router
+(`delivery/router.py`), which decides in the same transaction which agents in
+the room the row is for and stores their own events with it. The
+`messages_notify` trigger announces the row on commit (`db/notify_ddl.py`);
+one `MessageListener` wakes the transports watching that room
+(`messages/notify.py`); and each transport reads the rows after its own
+cursor, `DELIVERY_PAGE` at a time. Agents have no transport watching for them:
+what is for them was decided as it was written. There is no second store to
+agree with. Switch used to run on a Matrix homeserver; the `matrix_*` column
+names are what is left of it, kept as stable ids.
Three consequences worth stating up front:
@@ -26,6 +30,12 @@ Three consequences worth stating up front:
History is the one method left unimplemented, and deliberately: the read path
queries these rows directly.
+
+**Where reading starts is the consumer's.** `receive_forever` is handed a
+`DeliveryProgress` (`transport/cursors.py`) that says where to start each room
+and hears how far the client has got: a consumer, which one process owns,
+reads from its stored cursors and resumes where the last owner stopped.
+Writing needs nothing of the kind: any process may write as any client.
"""
from __future__ import annotations
@@ -34,7 +44,7 @@ import asyncio
import logging
import uuid
from datetime import UTC, datetime
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Any, cast
from switch_core.attachments import parse_attachment_group
from switch_core.db.models import (
@@ -64,6 +74,7 @@ from switch_core.observability.metrics import metrics
from switch_core.observability.throttle import WarningThrottle
from switch_core.tenant_context import no_tenant, tenant_scope
from switch_core.transport.content import media_content, message_content
+from switch_core.transport.derived_ids import derived_writes, writes_derived_from
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
from switch_core.transport.observer import (
@@ -94,7 +105,9 @@ if TYPE_CHECKING:
from switch_core.db.stores.message_store import MessageStore
from switch_core.db.stores.room_store import RoomStore
from switch_core.db.stores.usage_store import UsageStore
+ from switch_core.delivery.router import MessageRouter
from switch_core.messages.notify import MessageListener
+ from switch_core.transport.cursors import DeliveryProgress
from switch_core.transport.room_cache import RoomDeliveryCache
from switch_core.transport.stored import DeliveryAttachments, DeliveryRow
@@ -156,6 +169,10 @@ def _age_ms(sent_at: object) -> float | None:
MEMBERSHIP_EVENT_TYPE = "m.room.member"
+# How long a consumer that is told to stop may take to finish the row in hand,
+# before what is left of it is cancelled.
+STOP_GRACE_SECONDS = 5.0
+
class PostgresTransport:
"""Carries a room's events in the `messages` table.
@@ -182,6 +199,7 @@ class PostgresTransport:
ephemeral: EphemeralBus,
room_cache: RoomDeliveryCache,
message_observer: ParticipantMessageObserver,
+ router: MessageRouter,
) -> None:
self.user_id = user_id
self.client_id = client_id
@@ -217,15 +235,51 @@ class PostgresTransport:
# A shared read only counts for this client if it started after this.
self._woken: dict[str, int] = {}
self._message_observer = message_observer
+ # Writes every row, deciding as it does which agents the row is for.
+ self._router = router
self._handlers = TransportHandlers()
# Per-room delivery position, and the transport-side id to hand back
# to a handler. Both are keyed by the Switch room id, which is what
# the listener announces.
self._cursors: dict[str, int] = {}
self._watching: dict[str, str] = {}
+ # When the client joined each room it reads, as the membership row the
+ # room was read under says. `client_rooms.joined_at` is the time of
+ # the transaction that wrote the row, so a client taken out of a room
+ # and put back has a new one, which is how a re-read tells it from a
+ # client that never left (`_compare_membership`).
+ self._joined_at: dict[str, datetime] = {}
+ # Rooms this client is joining itself right now, between writing its
+ # membership row and watching the room from its own arrival. A
+ # reconcile that reads the row in that window must leave the room to
+ # the join, or it would watch from the head and the client would miss
+ # its own arrival.
+ self._joining: set[str] = set()
+ # Rooms a re-check has handed to `on_invite` because it read this
+ # client's membership row, until the join that follows runs. That
+ # join only takes up a row the read saw. Finding the row gone means a
+ # removal committed after the read, and writing it again would undo
+ # the removal.
+ self._adopting: set[str] = set()
+ # Held while this client reads its rooms and compares them with the
+ # ones it watches: on its first read at start, and on every re-read
+ # after (`_reconcile_membership`). A re-read during the first read
+ # would find every room the client is already in unwatched, and hand
+ # each to `on_invite` as if someone had just added the client to it.
+ self._reading_rooms = asyncio.Lock()
self._receiving = False
self._receiving_event = asyncio.Event()
+ # Set to end the current `receive_forever`. Replaced at the start of
+ # each, so a consumer stopped here and started again later reads again.
self._closed = asyncio.Event()
+ # Set while the current receive is ending: the delivery loop finishes
+ # the row in hand, records how far it got, and stops.
+ self._stopping = False
+ # Where the current receive starts each room and records how far it
+ # got, None while not receiving, and how far each room's progress has
+ # been recorded through it.
+ self._progress: DeliveryProgress | None = None
+ self._recorded: dict[str, int] = {}
# Rooms this client has been told moved but has not read yet, and the
# signal its delivery loop waits on. A set because a room that moved
# twice while the loop was busy is still one read.
@@ -242,47 +296,91 @@ class PostgresTransport:
"""Nothing to authenticate: the caller already knows who it is."""
async def close(self) -> None:
- """Stop receiving. Sessions are borrowed per operation, so that is all
- there is to release."""
+ """Stop receiving, after the row in hand. Sessions are borrowed per
+ operation, so that is all there is to release, and writing goes on
+ working: a client that stops reading here may still be written as."""
self._closed.set()
def register_handlers(self, handlers: TransportHandlers) -> None:
self._handlers = handlers
- async def receive_forever(self) -> None:
- """Deliver every row written to this client's rooms from now on.
+ async def receive_forever(self, progress: DeliveryProgress) -> None:
+ """Deliver the rows written to this client's rooms, until closed.
- Each room starts at its current head, so what an agent missed while it
- was away is not delivered here. That is a delivery-cursor question,
- and delivery cursors are a layer above this one.
+ Each room is read from where `progress` says, and progress is recorded
+ through it as rows are handled. Ending finishes the row in hand first
+ and records it, so a reader that resumes from the record neither
+ repeats nor skips it.
"""
+ self._closed = asyncio.Event()
+ self._stopping = False
+ self._progress = progress
# Before the rooms are read, not after they are subscribed: reading
# them and watching each is several round trips, and a removal rung
# during those has no membership row for the caller to fall back to.
- self._invites.register(self.client_id, self._on_invited, self._on_removed)
+ self._invites.register(
+ self.client_id,
+ self._on_invited,
+ self._on_removed,
+ self._reconcile_membership,
+ )
self._receiving = True
self._receiving_event.set()
try:
- rooms = await self.joined_rooms()
- if not rooms:
- logger.error(
- "Client %s is receiving but is a member of no room: it will "
- "hear nothing until something adds it to one. Membership "
- "is the client_rooms table, so a client whose rows were "
- "never written is silent rather than broken",
- self.user_id,
- )
delivery = asyncio.create_task(self._deliver_forever())
try:
- await self._watch_joined(rooms)
+ async with self._reading_rooms:
+ rooms = await self.joined_rooms()
+ if not rooms:
+ logger.error(
+ "Client %s is receiving but is a member of no room: "
+ "it will hear nothing until something adds it to "
+ "one. Membership is the client_rooms table, so a "
+ "client whose rows were never written is silent "
+ "rather than broken",
+ self.user_id,
+ )
+ await self._watch_joined(rooms)
await self._closed.wait()
finally:
- delivery.cancel()
+ await self._finish(delivery)
finally:
self._receiving = False
self._receiving_event.clear()
- self._invites.unregister(self.client_id, self._on_invited, self._on_removed)
+ self._invites.unregister(
+ self.client_id,
+ self._on_invited,
+ self._on_removed,
+ self._reconcile_membership,
+ )
self._unwatch_all()
+ self._progress = None
+ self._recorded.clear()
+
+ async def _finish(self, delivery: asyncio.Task[None]) -> None:
+ """End the delivery loop after the row in hand, or cancel it once
+ `STOP_GRACE_SECONDS` have passed. A row cut short by the cancel is not
+ recorded as handled, so the next reader handles it again."""
+ self._stopping = True
+ self._wake.set()
+ try:
+ await asyncio.wait_for(asyncio.shield(delivery), STOP_GRACE_SECONDS)
+ except TimeoutError:
+ logger.warning(
+ "Client %s did not finish its row within %.0fs of being told to "
+ "stop; cancelling it",
+ self.user_id,
+ STOP_GRACE_SECONDS,
+ )
+ except Exception:
+ logger.error(
+ "Client %s's delivery loop ended on an error",
+ self.user_id,
+ exc_info=True,
+ )
+ finally:
+ delivery.cancel()
+ await asyncio.gather(delivery, return_exceptions=True)
async def wait_receiving(self, timeout: float) -> bool:
try:
@@ -315,14 +413,18 @@ class PostgresTransport:
await self._deliver_forever_unbound()
async def _deliver_forever_unbound(self) -> None:
- while True:
+ while not self._stopping:
await self._wake.wait()
self._wake.clear()
+ if self._stopping:
+ return
rooms = list(self._pending)
self._pending.clear()
self._delivering = True
try:
for room_id in rooms:
+ if self._stopping:
+ return
# Bound around the drain rather than named in the message
# below, so every line this room's delivery produces —
# including the handlers' own — can be filtered to it.
@@ -355,6 +457,10 @@ class PostgresTransport:
is the client's decision and it already has one — the same one it made
when the invitation arrived over sync.
"""
+ await self._invite_self(transport_room_id, _now_ms())
+
+ async def _invite_self(self, transport_room_id: str, joined_at_ms: int) -> None:
+ """Hand `on_invite` the room, stamped with when the client joined it."""
handler = self._handlers.on_invite
if handler is None:
return
@@ -364,13 +470,104 @@ class PostgresTransport:
room_id=transport_room_id,
event_id=new_event_id(),
sender=self.user_id,
- timestamp=_now_ms(),
+ timestamp=joined_at_ms,
state_key=self.user_id,
membership="invite",
display_name=self.display_name,
),
)
+ async def _reconcile_membership(self) -> None:
+ """Start or stop reading the rooms whose membership changed underneath.
+
+ A membership written by another switch-core process reaches the
+ process reading for the client only as a `membership.changed` signal,
+ and a signal can be missed, so this also runs periodically. It
+ re-reads the client's rows and compares them with what it reads now. A
+ room it is no longer in goes through `_on_removed`, the path a removal
+ rung in this process takes. A room it is in but does not read goes
+ through `on_invite`, as an invitation would, and is read from where
+ the receive's progress starts it: for a consumer one process owns,
+ the cursor the membership was written with, just before the arrival,
+ so that one process hears the arrival and greets; for one every
+ process runs, the room's head, since the process that wrote the row
+ has already heard the arrival and greeted. The join time handed on is
+ the row's, the database's clock.
+
+ Only rooms already read before the rows were fetched can be dropped. A
+ room joined since was joined after its row committed, so the read
+ cannot have missed it; one joined during the read would otherwise look
+ like a removal.
+
+ A client taken out of a room and put back before the re-read still has
+ a row for it, but a new one with its own `joined_at`. Such a room goes
+ through `_on_removed` and then `on_invite`, as the removal and the
+ addition would have on their own, instead of being read on from where
+ the old membership had got to, which would hand over what was said
+ while the client was out. The `joined_at` compared is the one recorded
+ before the rows were fetched, for the reason above: a room joined
+ again during the read would otherwise look put back.
+ """
+ async with self._reading_rooms:
+ if self._receiving:
+ await self._compare_membership()
+
+ async def _compare_membership(self) -> None:
+ """`_reconcile_membership`'s read and comparison, under its lock."""
+ read_before = dict(self._watching)
+ joined_before = dict(self._joined_at)
+ async with tenant_session(self._session_factory, self.tenant_id) as session:
+ memberships = await self._room_store.memberships_of(session, self.client_id)
+ current = {membership.room_id: membership for membership in memberships}
+ for room_id, transport_room_id in read_before.items():
+ joined_at = joined_before.get(room_id)
+ # Dropped, or watched again, while the rows were being read.
+ if (
+ self._watching.get(room_id) != transport_room_id
+ or self._joined_at.get(room_id) != joined_at
+ ):
+ continue
+ membership = current.get(room_id)
+ if membership is None:
+ logger.info(
+ "Client %s is no longer in %s; it stops reading the room",
+ self.user_id,
+ transport_room_id,
+ )
+ elif joined_at is not None and membership.joined_at != joined_at:
+ logger.info(
+ "Client %s was taken out of %s and put back; it reads the "
+ "room again as a new member",
+ self.user_id,
+ transport_room_id,
+ )
+ else:
+ # Still the membership the room was read under, or a room
+ # whose `_watch` has not read the row yet and will take the
+ # one it finds.
+ continue
+ await self._on_removed(transport_room_id)
+ for membership in memberships:
+ self._room_ids[membership.transport_room_id] = membership.room_id
+ if (
+ membership.room_id in self._watching
+ or membership.room_id in self._joining
+ ):
+ continue
+ logger.info(
+ "Client %s was added to %s elsewhere; it starts reading the room",
+ self.user_id,
+ membership.transport_room_id,
+ )
+ self._adopting.add(membership.room_id)
+ try:
+ await self._invite_self(
+ membership.transport_room_id,
+ int(membership.joined_at.timestamp() * 1000),
+ )
+ finally:
+ self._adopting.discard(membership.room_id)
+
async def _on_removed(self, transport_room_id: str) -> None:
"""This client was taken out of a room while it was running.
@@ -404,90 +601,113 @@ class PostgresTransport:
),
)
- async def _watch(
- self, transport_room_id: str, *, from_seq: int | None = None
- ) -> None:
- """Start delivering a room, from its head or from a given position.
+ async def _watch(self, transport_room_id: str, *, arrival_seq: int | None) -> None:
+ """Start delivering a room, from where the receive's progress says.
- `from_seq` exists so a client can hear its own arrival. Over Matrix a
- join came back down the joining client's own connection — the
- homeserver turned it into an event and delivered it to everyone,
- including the person it was about — and code downstream learned it had
- joined by being told. Writing the row and then watching from the head
- steps over the client's own footprint: the room hears the arrival and
- the arriving client does not.
+ `arrival_seq` is the client's own arrival, when it has just joined the
+ room itself: reading starts before it, so the client hears its own
+ arrival. Over Matrix a join came back down the joining client's own
+ connection: the homeserver turned it into an event and delivered it
+ to everyone, including the person it was about, and code downstream
+ learned it had joined by being told. Writing the row and then
+ watching from the head would step over the client's own footprint:
+ the room hears the arrival and the arriving client does not.
Restoring the loop-back rather than dispatching the one handler that
noticed keeps every guard downstream in play, and covers whatever else
depended on hearing its own join.
- The slot in `_watching` is claimed before the head is read, so a
+ The slot in `_watching` is claimed before the start is read, so a
removal landing on that await has something to take back; finding the
claim gone afterwards is the signal to subscribe to nothing.
"""
room_id, tenant_id = await self._resolve_room_and_tenant(transport_room_id)
if room_id in self._watching:
return
+ progress = self._progress
+ if progress is None:
+ return
self._watching[room_id] = transport_room_id
- if from_seq is not None:
- # The caller wrote the membership row and committed it a moment
- # ago, so there is nothing to confirm.
- seq = from_seq
- else:
- try:
- async with tenant_session(self._session_factory, tenant_id) as session:
- member = await session.get(
- ClientRoom,
- {"client_id": self.client_id, "room_id": room_id},
+ try:
+ async with tenant_session(self._session_factory, tenant_id) as session:
+ # Read even when the caller wrote the row a moment ago: its
+ # `joined_at` is what a later re-read compares the membership
+ # by (`_compare_membership`).
+ member = await session.get(
+ ClientRoom,
+ {"client_id": self.client_id, "room_id": room_id},
+ )
+ if member is None:
+ # Removed between the room list this came from and
+ # here. The wake-up for it found nothing to unwatch, so
+ # the row is the only thing that still says so.
+ self._release_claim(room_id, transport_room_id)
+ return
+ if arrival_seq is not None:
+ seq: int | None = await progress.resume_joined(
+ session, room_id, arrival_seq
)
- if member is None:
- # Removed between the room list this came from and
- # here. The wake-up for it found nothing to unwatch, so
- # the row is the only thing that still says so.
- self._release_claim(room_id, transport_room_id)
- return
- seq = await self._message_store.head_seq(session, room_id)
- except Exception:
- # A claim left behind would make every later `_watch` return
- # early on a room nothing is subscribed to.
- self._release_claim(room_id, transport_room_id)
- raise
+ else:
+ seq = (await progress.resume(session, [room_id])).get(room_id)
+ head = await self._message_store.head_seq(session, room_id)
+ except Exception:
+ # A claim left behind would make every later `_watch` return
+ # early on a room nothing is subscribed to.
+ self._release_claim(room_id, transport_room_id)
+ raise
if self._watching.get(room_id) != transport_room_id:
- # Removed while the row and head were being read. `_unwatch` has
+ # Removed while the row and start were being read. `_unwatch` has
# already undone the claim and there is nothing subscribed to undo.
return
- self._subscribe_room(room_id, transport_room_id, seq)
- if from_seq is not None:
- # The announcement for anything already written went out before
- # this subscription existed, so nothing will wake the loop for it.
+ if seq is None:
+ self._release_claim(room_id, transport_room_id)
+ return
+ self._subscribe_room(
+ room_id, transport_room_id, seq, cast(datetime, member.joined_at)
+ )
+ if seq < head:
+ # Anything already written past the start was announced before
+ # this subscription existed, so nothing else will wake the loop
+ # for it.
self._mark_pending(room_id)
- def _subscribe_room(self, room_id: str, transport_room_id: str, seq: int) -> None:
- """Deliver a claimed room from `seq` on."""
+ def _subscribe_room(
+ self, room_id: str, transport_room_id: str, seq: int, joined_at: datetime
+ ) -> None:
+ """Deliver a claimed room from `seq` on, for the membership whose row
+ says the client joined at `joined_at`."""
self._cursors[room_id] = seq
+ self._recorded[room_id] = seq
self._room_cache.attach(self.tenant_id, room_id, self)
self._listener.subscribe(room_id, self._on_room_advanced)
self._ephemeral.subscribe(transport_room_id, self._on_ephemeral)
+ self._joined_at[room_id] = joined_at
async def _watch_joined(self, transport_room_ids: list[str]) -> None:
- """`_watch` every room the client is in, from each room's head, reading
- membership and heads for all of them at once.
+ """`_watch` every room the client is in, from where the progress
+ starts each, reading membership, starts and heads for all of them at
+ once.
At boot every client does this at the same moment, and one session
per room per check was over a thousand sessions in a few seconds,
enough to drain the pool. The guard is `_watch`'s, applied to the whole
set: every room is claimed before anything is read, so a removal that
lands on the read finds a claim to take back, and a room whose
- membership row is gone by the time it is read is released.
+ membership row is gone by the time it is read is released. Only a room
+ whose start is behind its head is read at once: at boot that is the
+ few that moved while nobody read for the client, not all of them.
A room `joined_rooms` did not resolve (it always does, but the cache is
the only thing this relies on) goes through `_watch` on its own.
"""
+ progress = self._progress
+ if progress is None:
+ return
claimed: dict[str, str] = {}
for transport_room_id in transport_room_ids:
room_id = self._room_ids.get(transport_room_id)
if room_id is None:
- await self._watch(transport_room_id)
+ await self._watch(transport_room_id, arrival_seq=None)
continue
if room_id in self._watching:
continue
@@ -500,19 +720,24 @@ class PostgresTransport:
members = await self._room_store.member_room_ids(
session, self.client_id, list(claimed)
)
- heads = await self._message_store.head_seqs(session, list(claimed))
+ places = await progress.resume(session, members)
+ heads = await self._message_store.head_seqs(session, list(members))
except Exception:
for room_id, transport_room_id in claimed.items():
self._release_claim(room_id, transport_room_id)
raise
for room_id, transport_room_id in claimed.items():
- if room_id not in members:
+ if room_id not in members or room_id not in places:
# Removed between the room list and here; see `_watch`.
self._release_claim(room_id, transport_room_id)
continue
if self._watching.get(room_id) != transport_room_id:
continue
- self._subscribe_room(room_id, transport_room_id, heads[room_id])
+ self._subscribe_room(
+ room_id, transport_room_id, places[room_id], members[room_id]
+ )
+ if places[room_id] < heads[room_id]:
+ self._mark_pending(room_id)
def _unwatch(self, transport_room_id: str) -> None:
"""Stop delivering one room. Not watching it is success.
@@ -536,7 +761,9 @@ class PostgresTransport:
self._listener.unsubscribe(room_id, self._on_room_advanced)
self._ephemeral.unsubscribe(transport_room_id, self._on_ephemeral)
self._watching.pop(room_id, None)
+ self._joined_at.pop(room_id, None)
self._cursors.pop(room_id, None)
+ self._recorded.pop(room_id, None)
# A wake-up already queued for this room would otherwise be drained
# after the subscription was dropped.
self._pending.discard(room_id)
@@ -582,6 +809,31 @@ class PostgresTransport:
self._pending.add(room_id)
self._wake.set()
+ def held_back_released(self, transport_room_id: str) -> None:
+ """The client acted on rows it was holding back in a room: have the
+ delivery loop record the progress that was waiting on them."""
+ room_id = self._room_ids.get(transport_room_id)
+ if room_id is not None and room_id in self._watching:
+ self._mark_pending(room_id)
+
+ def _record_progress(self, room_id: str) -> None:
+ """Record the room's progress through the last row handled, or up to
+ the first row the client is still holding back, if that is further
+ than what was recorded last."""
+ progress = self._progress
+ transport_room_id = self._watching.get(room_id)
+ cursor = self._cursors.get(room_id)
+ if progress is None or transport_room_id is None or cursor is None:
+ return
+ held_from = self._handlers.held_from
+ held = held_from(transport_room_id) if held_from is not None else None
+ through = cursor if held is None else min(cursor, held - 1)
+ recorded = self._recorded.get(room_id)
+ if recorded is not None and through <= recorded:
+ return
+ self._recorded[room_id] = through
+ progress.advance(room_id, through)
+
async def _drain_room(self, room_id: str) -> None:
"""Read what this client has not seen in one room, and hand it over.
@@ -637,14 +889,16 @@ class PostgresTransport:
# shared read: these rows are from the old
# subscription's position, not the new one's.
continue
- if not rows:
- return
for row, attachments in rows:
- if room_id not in self._watching:
- return
+ if room_id not in self._watching or self._stopping:
+ break
self._cursors[room_id] = row.seq
await self._deliver(transport_room_id, row, attachments)
- if done:
+ # After the page's rows are handled, never before: a reader
+ # resuming from it would otherwise skip a row whose effects
+ # were cut short.
+ self._record_progress(room_id)
+ if done or not rows or self._stopping:
return
async def _read_page(
@@ -710,7 +964,10 @@ class PostgresTransport:
metrics().observe(
DELIVERY_LAG, {"kind": kind, "actor": self._actor_role}, lag_ms
)
- await handler(room, event)
+ # What the client writes while handling the row is numbered from it,
+ # so handling it a second time writes nothing new.
+ with writes_derived_from(self.client_id, row.transport_event_id):
+ await handler(room, event)
def _handler_for(self, event: InboundEvent) -> Handler | None:
if isinstance(event, InboundMedia):
@@ -800,11 +1057,16 @@ class PostgresTransport:
The commit is the acceptance: the trigger on `messages` announces the
row as part of it, so no subscriber can be woken for a row that a later
- rollback removes.
+ rollback removes. The agents the row is for have their own events in
+ the same commit, and anything the row leaves an agent to say is posted
+ only after it.
"""
- result = SendResult(
- event_id=new_event_id(), event_type=event_type, content=content
- )
+ # A row this client writes while handling one it was handed, or on
+ # behalf of a platform post, gets an id derived from what it acts on
+ # (`derived_ids.py`), so acting on it again writes nothing new.
+ derived = None if event_type in EPHEMERAL else derived_writes(self.client_id)
+ event_id = new_event_id() if derived is None else derived.next_event_id()
+ result = SendResult(event_id=event_id, event_type=event_type, content=content)
kind = _sent_kind(event_type, content)
if event_type in EPHEMERAL:
# Presence-like state, replaced by its own next value. Storing it
@@ -841,9 +1103,23 @@ class PostgresTransport:
content=content,
client_txn_id=None,
)
- await self._message_store.create(
- session, message, attachments_in(content)
+ routed = await self._router.store(
+ session,
+ tenant_id=tenant_id,
+ room_id=room_id,
+ transport_room_id=transport_room_id,
+ message=message,
+ attachments=attachments_in(content),
+ unless_stored=derived is not None,
)
+ if routed is None:
+ logger.info(
+ "Client %s already wrote %s for what it is acting on; "
+ "not writing it again",
+ self.user_id,
+ result.event_id,
+ )
+ return result
if metered and kind in _METERED_KINDS:
await self._usage_store.record(
session,
@@ -862,6 +1138,7 @@ class PostgresTransport:
SEND_FAILURES, {"kind": kind, "actor": self._actor_role}
)
raise
+ routed.committed()
metrics().increment(MESSAGES_SENT, {"kind": kind, "actor": self._actor_role})
if metered and kind in _METERED_KINDS:
self._observe(tenant_id, room_id, kind, content)
@@ -1002,15 +1279,41 @@ class PostgresTransport:
once and the subscription is taken whenever it is missing.
"""
switch_room_id, tenant_id = await self._resolve_room_and_tenant(room_id)
+ adopting = switch_room_id in self._adopting
+ self._adopting.discard(switch_room_id)
+ self._joining.add(switch_room_id)
+ try:
+ return await self._join_room(
+ room_id, switch_room_id, tenant_id, adopting=adopting
+ )
+ finally:
+ self._joining.discard(switch_room_id)
+
+ async def _join_room(
+ self, room_id: str, switch_room_id: str, tenant_id: str, *, adopting: bool
+ ) -> bool:
async with tenant_session(self._session_factory, tenant_id) as session:
existing = await session.get(
ClientRoom, {"client_id": self.client_id, "room_id": switch_room_id}
)
- if existing is not None:
+ if existing is None and adopting:
+ logger.info(
+ "Client %s was removed from %s while it re-read its rooms; "
+ "it does not join",
+ self.user_id,
+ room_id,
+ )
+ return False
+ joined = existing is None and await self._room_store.add_client(
+ session, self.client_id, switch_room_id
+ )
+ if not joined:
+ # A member already, or made one a moment ago by another
+ # writer, which wrote the arrival too: one arrival per
+ # membership, so the room is greeted once.
if self._receiving:
- await self._watch(room_id)
+ await self._watch(room_id, arrival_seq=None)
return True
- await self._room_store.add_client(session, self.client_id, switch_room_id)
arrival = Message(
room_id=switch_room_id,
transport_event_id=new_event_id(),
@@ -1027,13 +1330,21 @@ class PostgresTransport:
# something a later change can take back.
content={"membership": "join", "displayname": self.display_name},
)
- await self._message_store.create(session, arrival, [])
+ routed = await self._router.store(
+ session,
+ tenant_id=tenant_id,
+ room_id=switch_room_id,
+ transport_room_id=room_id,
+ message=arrival,
+ attachments=[],
+ )
await session.commit()
- arrival_seq = arrival.seq
+ routed.committed()
+ arrival_seq = routed.seq
if self._receiving:
# From just before the arrival, so this client reads its own join
# the way every other member does.
- await self._watch(room_id, from_seq=arrival_seq - 1)
+ await self._watch(room_id, arrival_seq=arrival_seq)
return True
async def joined_rooms(self) -> list[str]:
diff --git a/core/tests/benchmarks/server.py b/core/tests/benchmarks/server.py
index 74caf5276..1d4de57a0 100644
--- a/core/tests/benchmarks/server.py
+++ b/core/tests/benchmarks/server.py
@@ -43,7 +43,6 @@ from switch_core.bridges.agent.protocol.types import (
)
from switch_core.bridges.resource.service import ResourceService
from switch_core.clients.actor import Actor, AgentActor, HumanActor
-from switch_core.clients.agent_consumer import AgentConsumer
from switch_core.clients.client_factory import ClientFactory
from switch_core.clients.client_lifecycle_service import ClientLifecycleService
from switch_core.db.engine import create_unpooled_engine
@@ -56,8 +55,14 @@ from switch_core.db.models import (
agent_event_boot_sequence,
)
from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.budget_store import BudgetStore
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
from switch_core.db.stores.tenant_store import TenantStore
from switch_core.db.stores.usage_store import UsageStore
+from switch_core.delivery.recipient import RecipientServices
+from switch_core.delivery.router import MessageRouter
from switch_core.main import (
_agent_event_retention_loop,
_connection_sweep_loop,
@@ -66,6 +71,7 @@ from switch_core.main import (
)
from switch_core.messages.notify import MessageListener
from switch_core.observability.runtime import EventLoopLag
+from switch_core.ownership import Ownership
from switch_core.provisioning import Provisioning
from switch_core.provisioning.postgres import PostgresProvisioning
from switch_core.room_service import RoomCreateConfig, RoomService
@@ -73,10 +79,12 @@ from switch_core.session_activity.listener import AgentSessionActivityListener
from switch_core.session_activity.outcomes import ApprovalOutcomes
from switch_core.session_activity.service import AgentSessionActivityService, SwitchUser
from switch_core.sessions.contract import ApprovalResult
+from switch_core.signals import SignalBus
from switch_core.telemetry.messages import MessageTelemetry
from switch_core.telemetry.service import TelemetryService
from switch_core.telemetry.sink import NullSink
from switch_core.tenant_context import bind_tenant_id, tenant_scope
+from switch_core.transport.cursors import CursorWriter
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
@@ -95,6 +103,7 @@ from tests.integration.conftest import (
_NoBridges,
_truncate_all,
)
+from tests.ownership import process_lease_on
# Agents driven by a host process, which is what the benchmark spawns. The
# profile has to say so: `session_passive` would suppress the addressed
@@ -269,7 +278,7 @@ class BenchServer:
client = await self._await_client(agent.agent_id, deadline)
await client.wait_ready()
- async def _await_client(self, agent_id: str, deadline: float) -> AgentConsumer:
+ async def _await_client(self, agent_id: str, deadline: float) -> AgentActor:
while asyncio.get_event_loop().time() < deadline:
client = self.client_lifecycle.get_by_agent_id(agent_id)
if client is not None:
@@ -457,9 +466,12 @@ async def _serve(
# production. The events themselves are rows and outlive the restart.
async with session_env.engine.begin() as connection:
event_boot = await connection.scalar(agent_event_boot_sequence.next_value())
+ message_listener = MessageListener(lambda: create_unpooled_engine(config))
+ await message_listener.start()
event_buffer = EventBuffer(
session_factory,
sequence_base=event_boot << 32,
+ rooms=message_listener,
retention_seconds=config.agent_event_retention_seconds,
)
agent_event_listener = AgentEventListener(
@@ -468,8 +480,6 @@ async def _serve(
connections = AgentConnectionRegistry()
collab_lifecycle = _NoBridges()
- message_listener = MessageListener(lambda: create_unpooled_engine(config))
- await message_listener.start()
await agent_event_listener.start()
activity_listener = AgentSessionActivityListener(
lambda: create_unpooled_engine(config)
@@ -488,11 +498,45 @@ async def _serve(
session_factory=session_factory,
)
+ # Telemetry off: these flows are not about reporting, but the
+ # message hook must still be wired for the clients to build.
+ message_telemetry = MessageTelemetry(
+ telemetry=TelemetryService(
+ sink=NullSink(),
+ enabled=False,
+ client_id="",
+ version=None,
+ telemetry_environment="dev",
+ telemetry_internal=True,
+ ),
+ session_factory=session_factory,
+ )
+
+ router = MessageRouter(
+ services=RecipientServices(
+ session_factory=session_factory,
+ event_buffer=event_buffer,
+ agent_store=session_env.agent_store,
+ room_store=session_env.room_store,
+ document_store=session_env.document_store,
+ reference_store=session_env.reference_store,
+ agent_session_store=session_env.agent_session_store,
+ room_role_store=session_env.room_role_store,
+ external_user_store=session_env.external_user_store,
+ connections=connections,
+ frontend_base_url=config.frontend_base_url,
+ message_telemetry=message_telemetry,
+ ),
+ message_store=session_env.message_store,
+ client_store=session_env.client_store,
+ budget_store=BudgetStore(),
+ )
+
provisioning: Provisioning = PostgresProvisioning(
session_factory=session_factory,
room_store=session_env.room_store,
client_store=session_env.client_store,
- message_store=session_env.message_store,
+ router=router,
invites=invites,
)
@@ -507,20 +551,6 @@ async def _serve(
),
)
- # Telemetry off: these flows are not about reporting, but the
- # message hook must still be wired for the clients to build.
- message_telemetry = MessageTelemetry(
- telemetry=TelemetryService(
- sink=NullSink(),
- enabled=False,
- client_id="",
- version=None,
- telemetry_environment="dev",
- telemetry_internal=True,
- ),
- session_factory=session_factory,
- )
-
client_factory = ClientFactory(
client_store=session_env.client_store,
session_factory=session_factory,
@@ -534,27 +564,16 @@ async def _serve(
ephemeral=ephemeral,
room_cache=room_cache,
message_observer=message_telemetry,
+ router=router,
)
- client_factory.register(
- "agent",
- AgentActor,
- AgentConsumer,
- event_buffer=event_buffer,
- agent_store=session_env.agent_store,
- room_store=session_env.room_store,
- bridge_store=session_env.bridge_store,
- document_store=session_env.document_store,
- reference_store=session_env.reference_store,
- agent_session_store=session_env.agent_session_store,
- room_role_store=session_env.room_role_store,
- external_user_store=session_env.external_user_store,
- connections=connections,
- frontend_base_url=config.frontend_base_url,
- message_telemetry=message_telemetry,
- )
+ client_factory.register("agent", AgentActor)
client_factory.register("user", HumanActor)
client_factory.register("bridge", Actor)
+ # This process's lease and what it owns, as `main` wires them. Only agent
+ # clients run here, which have no consumer, so nothing is ever claimed;
+ # the signal bus is never started.
+ lease = process_lease_on(session_factory)
client_lifecycle = ClientLifecycleService(
provisioning=provisioning,
client_store=session_env.client_store,
@@ -563,6 +582,23 @@ async def _serve(
session_factory=session_factory,
config=config,
tenants_isolated=True,
+ ownership=Ownership(
+ lease=lease,
+ session_factory=session_factory,
+ bus=SignalBus(
+ session_factory,
+ lambda: create_unpooled_engine(config),
+ process_id=lease.process_id,
+ ),
+ store=OwnershipLeaseStore(),
+ processes=SwitchCoreProcessStore(),
+ ),
+ cursor_writer=CursorWriter(
+ session_factory=session_factory,
+ cursors=DeliveryCursorStore(),
+ leases=OwnershipLeaseStore(),
+ process_id=lease.process_id,
+ ),
)
room_service = RoomService(
diff --git a/core/tests/conftest.py b/core/tests/conftest.py
index 052c4c9d9..72aa32d6f 100644
--- a/core/tests/conftest.py
+++ b/core/tests/conftest.py
@@ -17,6 +17,7 @@ from sqlalchemy.ext.asyncio import (
async_sessionmaker,
create_async_engine,
)
+from sqlalchemy.pool import NullPool
from testcontainers.postgres import PostgresContainer
# Importing the models module registers every table on Base.metadata so
@@ -31,6 +32,7 @@ from switch_core.db.engine import create_session_factory
from switch_core.db.models import TENANT_ZERO_ID, Tenant
from switch_core.db.runtime_role import grant_runtime_role
from switch_core.keys import Keyring
+from switch_core.messages.notify import MessageListener
from switch_core.tenant_context import tenant_scope
from tests.db_buffer import db_buffer_on
from tests.switch_core.pool_checkouts import PoolCheckouts
@@ -381,3 +383,19 @@ def db_buffer(session_factory: async_sessionmaker[AsyncSession]) -> EventBuffer:
"""The db-buffer on this test's database. Seed the agents it queues for
with `tests.db_buffer.seed_agent`."""
return db_buffer_on(session_factory)
+
+
+@pytest_asyncio.fixture
+async def room_bells(postgres_schema: str) -> AsyncIterator[MessageListener]:
+ """The room announcements of this test's database, listened to: for a
+ test whose stream reads a room's conversation and has to be woken when a
+ row is written there (`db_buffer_on(..., rooms=room_bells)`)."""
+ listener = MessageListener(
+ lambda: create_async_engine(postgres_schema, poolclass=NullPool)
+ )
+ await listener.start()
+ try:
+ await asyncio.wait_for(listener.connected.wait(), 10.0)
+ yield listener
+ finally:
+ await listener.stop()
diff --git a/core/tests/db_buffer.py b/core/tests/db_buffer.py
index d016f3923..4cfc7e25e 100644
--- a/core/tests/db_buffer.py
+++ b/core/tests/db_buffer.py
@@ -13,7 +13,10 @@ from typing import TYPE_CHECKING, Any
from sqlalchemy import func, select, update
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
+from switch_core.bridges.agent.protocol.event_buffer import (
+ EventBuffer,
+ RoomAnnouncements,
+)
from switch_core.db.models import (
TENANT_ZERO_ID,
Agent,
@@ -29,23 +32,42 @@ from switch_core.db.session_scope import tenant_session
from switch_core.db.stores.message_store import MessageStore
if TYPE_CHECKING:
+ from collections.abc import Awaitable, Callable
+
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+class QuietRooms:
+ """Room announcements for a test that has none to make.
+
+ A stream that reads a room's conversation asks to be woken when the room
+ has a new row. A test that writes conversation while a stream waits, and
+ expects the stream to wake for it, gives the buffer a running
+ `MessageListener` instead (`tests/routing.py`, `room_bells`).
+ """
+
+ def subscribe(self, room_id: str, waker: Callable[[str], Awaitable[None]]) -> None:
+ return None
+
+ def unsubscribe(
+ self, room_id: str, waker: Callable[[str], Awaitable[None]]
+ ) -> None:
+ return None
+
+
def db_buffer_on(
session_factory: async_sessionmaker[AsyncSession],
*,
sequence_base: int = 0,
retention_seconds: float = 3600,
- settle_seconds: float = 0.2,
+ rooms: RoomAnnouncements | None = None,
) -> EventBuffer:
- """The db-buffer, settling conversation for a fraction of the real time:
- a test has no consumer to race, and `say` writes rows already settled."""
+ """The db-buffer on a test's database."""
return EventBuffer(
session_factory,
sequence_base=sequence_base,
+ rooms=rooms if rooms is not None else QuietRooms(),
retention_seconds=retention_seconds,
- settle_seconds=settle_seconds,
)
@@ -203,11 +225,14 @@ async def say(
event_type: str = "m.room.message",
content: dict[str, Any] | None = None,
) -> Message:
- """Write one row to a room's log, as the transport does, and return it
- (its `seq` is its place in the room). The room must exist.
+ """Write one row to a room's log and return it (its `seq` is its place
+ in the room). The room must exist.
- Stamped a second ago by the database's clock, the one conversation is
- settled by, so it reads as conversation straight away."""
+ Written straight to the table, as a row from before the router or from a
+ process on the release before it would be: decided for nobody. A test of
+ what the router decides writes through it instead (`tests/routing.py`).
+ Stamped a second ago by the database's clock, so it reads as written
+ after any membership the test made a moment before."""
async with tenant_session(session_factory, tenant_id) as session:
if sent_at is None:
sent_at = await session.scalar(select(func.now() - timedelta(seconds=1)))
@@ -230,3 +255,18 @@ async def say(
await session.commit()
await session.refresh(message)
return message
+
+
+async def forget_room(
+ session_factory: async_sessionmaker[AsyncSession],
+ buffer: EventBuffer,
+ agent_id: str,
+ room_id: str,
+ *,
+ tenant_id: str = TENANT_ZERO_ID,
+) -> None:
+ """What taking the agent out of the room does to its own events there,
+ in a transaction of its own (`EventBuffer.forget_room`)."""
+ async with tenant_session(session_factory, tenant_id) as session:
+ await buffer.forget_room(session, agent_id=agent_id, room_id=room_id)
+ await session.commit()
diff --git a/core/tests/integration/conftest.py b/core/tests/integration/conftest.py
index ec1937221..1dd9cf8e7 100644
--- a/core/tests/integration/conftest.py
+++ b/core/tests/integration/conftest.py
@@ -4,7 +4,7 @@ Unlike the unit suite (which fakes the transport), these fixtures boot a real
database and wire up the subset of `switch_core.main:run()` the feature under
test needs, in-process. Messaging, membership and provisioning all run against
that database, so a test drives the genuine path RoomService → provisioning →
-AgentConsumer receive loop → EventBuffer (the agent_events tables).
+MessageRouter → EventBuffer (the agent_events tables).
The container mirrors the `postgres` service in
`deploy/local/docker-compose.yml` (see constants below — keep in sync). It gets
@@ -51,7 +51,6 @@ from switch_core.bridges.collaboration.lifecycle_service import (
)
from switch_core.bridges.resource.service import ResourceService
from switch_core.clients.actor import Actor, AgentActor, HumanActor
-from switch_core.clients.agent_consumer import AgentConsumer
from switch_core.clients.client_factory import ClientFactory
from switch_core.clients.client_lifecycle_service import ClientLifecycleService
from switch_core.config import SwitchConfig
@@ -67,37 +66,47 @@ from switch_core.db.stores.agent_session_store import AgentSessionStore
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.api_key_store import ApiKeyStore
from switch_core.db.stores.bridge_message_map_store import BridgeMessageMapStore
+from switch_core.db.stores.budget_store import BudgetStore
from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
from switch_core.db.stores.document_store import DocumentStore
from switch_core.db.stores.external_user_store import ExternalUserStore
from switch_core.db.stores.media_store import MediaStore
from switch_core.db.stores.message_store import MessageStore
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
from switch_core.db.stores.package_store import PackageStore
from switch_core.db.stores.reference_store import ReferenceStore
from switch_core.db.stores.reference_type_store import ReferenceTypeStore
from switch_core.db.stores.room_link_store import RoomLinkStore
from switch_core.db.stores.room_role_store import RoomRoleStore
from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
from switch_core.db.stores.task_store import TaskStore
from switch_core.db.stores.tenant_store import TenantStore
from switch_core.db.stores.usage_store import UsageStore
from switch_core.db.stores.user_store import UserStore
+from switch_core.delivery.recipient import RecipientServices
+from switch_core.delivery.router import MessageRouter
from switch_core.main import _seed_agent_registration_bootstrap_key
from switch_core.messages.notify import MessageListener
+from switch_core.ownership import Ownership
from switch_core.provisioning import Provisioning
from switch_core.provisioning.postgres import PostgresProvisioning
from switch_core.room_service import RoomService
from switch_core.session_activity.listener import AgentSessionActivityListener
from switch_core.session_activity.service import AgentSessionActivityService
+from switch_core.signals import SignalBus
from switch_core.telemetry.messages import MessageTelemetry
from switch_core.telemetry.service import TelemetryService
from switch_core.telemetry.sink import NullSink
from switch_core.tenant_context import tenant_scope
+from switch_core.transport.cursors import CursorWriter
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
from switch_core.trust.client import NullTrustClient
+from tests.ownership import process_lease_on
# ── Mirrors deploy/local/docker-compose.yml — keep in sync ──────────────────────
POSTGRES_IMAGE = "postgres:16-alpine"
@@ -269,8 +278,12 @@ class Harness:
owner_id: str,
session_factory: object,
collab_lifecycle: object,
+ ownership: Ownership,
) -> None:
self.protocol = protocol
+ # Which work this one process owns: all of it, as with one process
+ # in production, once `start_clients` has balanced.
+ self.ownership = ownership
# The real lifecycle under the `collaboration_bridges` marker, the
# no-bridges stand-in otherwise.
self.collab_lifecycle = collab_lifecycle
@@ -325,16 +338,17 @@ class Harness:
async def start_clients(self, timeout: float = 20.0) -> None:
"""Start every registered agent's client and await readiness.
- `start_all` launches each client's receive loop as a background task; the
- client only loads its Agent (so `get_by_agent_id` resolves) once that
- task has run. Poll until each is present and ready before returning.
+ `start_all` connects each client's actor as a background task; the actor
+ only loads its Agent (so `get_by_agent_id` resolves) once that task has
+ run. Poll until each is present and ready before returning.
"""
await self.client_lifecycle.start_all()
+ await self.ownership.start()
for agent_id in self._registered:
client = await self._await_client(agent_id, timeout)
await client.wait_ready()
- async def _await_client(self, agent_id: str, timeout: float) -> AgentConsumer:
+ async def _await_client(self, agent_id: str, timeout: float) -> AgentActor:
deadline = asyncio.get_event_loop().time() + timeout
while asyncio.get_event_loop().time() < deadline:
client = self.client_lifecycle.get_by_agent_id(agent_id)
@@ -343,7 +357,7 @@ class Harness:
await asyncio.sleep(0.1)
raise AssertionError(f"No running client for agent {agent_id} after {timeout}s")
- def client_for(self, agent_id: str) -> AgentConsumer:
+ def client_for(self, agent_id: str) -> AgentActor:
client = self.client_lifecycle.get_by_agent_id(agent_id)
if client is None:
raise AssertionError(f"No running client for agent {agent_id}")
@@ -552,9 +566,14 @@ async def harness(
# queued events themselves are rows, cleared by the truncation above.
# No `AgentEventListener`: it wakes readers for rows another process
# wrote, and everything here runs in this one.
+ # The transport's wake-up path: rows are announced over LISTEN/NOTIFY, so
+ # nothing is delivered to a client until this connection is up.
+ message_listener = MessageListener(lambda: create_unpooled_engine(config))
+ await message_listener.start()
event_buffer = EventBuffer(
session_factory, # type: ignore[arg-type]
sequence_base=0,
+ rooms=message_listener,
retention_seconds=config.agent_event_retention_seconds,
)
connections = AgentConnectionRegistry()
@@ -566,10 +585,6 @@ async def harness(
lambda: create_unpooled_engine(config)
)
- # The transport's wake-up path: rows are announced over LISTEN/NOTIFY, so
- # nothing is delivered to a client until this connection is up.
- message_listener = MessageListener(lambda: create_unpooled_engine(config))
- await message_listener.start()
invites = InviteBus()
ephemeral = EphemeralBus()
@@ -582,11 +597,45 @@ async def harness(
session_factory=session_factory,
)
+ # Telemetry off: these flows are not about reporting, but the
+ # message hook must still be wired for the clients to build.
+ message_telemetry = MessageTelemetry(
+ telemetry=TelemetryService(
+ sink=NullSink(),
+ enabled=False,
+ client_id="",
+ version=None,
+ telemetry_environment="dev",
+ telemetry_internal=True,
+ ),
+ session_factory=session_factory,
+ )
+
+ router = MessageRouter(
+ services=RecipientServices(
+ session_factory=session_factory, # type: ignore[arg-type]
+ event_buffer=event_buffer,
+ agent_store=session_env.agent_store,
+ room_store=session_env.room_store,
+ document_store=session_env.document_store,
+ reference_store=session_env.reference_store,
+ agent_session_store=session_env.agent_session_store,
+ room_role_store=session_env.room_role_store,
+ external_user_store=session_env.external_user_store,
+ connections=connections,
+ frontend_base_url=config.frontend_base_url,
+ message_telemetry=message_telemetry,
+ ),
+ message_store=session_env.message_store,
+ client_store=session_env.client_store,
+ budget_store=BudgetStore(),
+ )
+
provisioning: Provisioning = PostgresProvisioning(
session_factory=session_factory,
room_store=session_env.room_store,
client_store=session_env.client_store,
- message_store=session_env.message_store,
+ router=router,
invites=invites,
)
@@ -601,20 +650,6 @@ async def harness(
),
)
- # Telemetry off: these flows are not about reporting, but the
- # message hook must still be wired for the clients to build.
- message_telemetry = MessageTelemetry(
- telemetry=TelemetryService(
- sink=NullSink(),
- enabled=False,
- client_id="",
- version=None,
- telemetry_environment="dev",
- telemetry_internal=True,
- ),
- session_factory=session_factory,
- )
-
client_factory = ClientFactory(
client_store=session_env.client_store,
session_factory=session_factory,
@@ -628,27 +663,34 @@ async def harness(
ephemeral=ephemeral,
room_cache=room_cache,
message_observer=message_telemetry,
+ router=router,
)
- client_factory.register(
- "agent",
- AgentActor,
- AgentConsumer,
- event_buffer=event_buffer,
- agent_store=session_env.agent_store,
- room_store=session_env.room_store,
- bridge_store=session_env.bridge_store,
- document_store=session_env.document_store,
- reference_store=session_env.reference_store,
- agent_session_store=session_env.agent_session_store,
- room_role_store=session_env.room_role_store,
- external_user_store=session_env.external_user_store,
- connections=connections,
- frontend_base_url=config.frontend_base_url,
- message_telemetry=message_telemetry,
- )
+ client_factory.register("agent", AgentActor)
client_factory.register("user", HumanActor)
client_factory.register("bridge", Actor)
+ # This process's lease and what it owns, as `main` wires them. The
+ # signal bus is never started: there is no other process to tell.
+ lease = process_lease_on(session_factory) # type: ignore[arg-type]
+ ownership = Ownership(
+ lease=lease,
+ session_factory=session_factory, # type: ignore[arg-type]
+ bus=SignalBus(
+ session_factory, # type: ignore[arg-type]
+ lambda: create_unpooled_engine(config),
+ process_id=lease.process_id,
+ ),
+ store=OwnershipLeaseStore(),
+ processes=SwitchCoreProcessStore(),
+ )
+ cursor_writer = CursorWriter(
+ session_factory=session_factory, # type: ignore[arg-type]
+ cursors=DeliveryCursorStore(),
+ leases=OwnershipLeaseStore(),
+ process_id=lease.process_id,
+ )
+ cursor_writer_task = asyncio.create_task(cursor_writer.run())
+
client_lifecycle = ClientLifecycleService(
provisioning=provisioning,
client_store=session_env.client_store,
@@ -657,6 +699,8 @@ async def harness(
session_factory=session_factory,
config=config,
tenants_isolated=True,
+ ownership=ownership,
+ cursor_writer=cursor_writer,
)
if real_bridges:
@@ -676,6 +720,8 @@ async def harness(
session_activity_listener=session_activity_listener,
session_activity_service=AgentSessionActivityService(session_factory),
connections=connections,
+ ownership=ownership,
+ cursor_writer=cursor_writer,
)
room_service = RoomService(
@@ -727,10 +773,13 @@ async def harness(
owner_id=owner_id,
session_factory=session_factory,
collab_lifecycle=collab_lifecycle,
+ ownership=ownership,
)
try:
yield h
finally:
+ await ownership.stop()
+ cursor_writer_task.cancel()
if isinstance(collab_lifecycle, CollaborationBridgeLifecycleService):
await collab_lifecycle.stop_all()
# Stop the agents' receive loops before the next test truncates, so none
diff --git a/core/tests/integration/test_agent_greeting_e2e.py b/core/tests/integration/test_agent_greeting_e2e.py
index cc03d99fa..9384bf5b9 100644
--- a/core/tests/integration/test_agent_greeting_e2e.py
+++ b/core/tests/integration/test_agent_greeting_e2e.py
@@ -1,8 +1,9 @@
"""End-to-end test for the agent self-join greeting over the real transport.
-Drives the genuine path: RoomService creates a room and invites the agent →
-the agent's AgentConsumer sees the invite → auto-joins → on_self_join posts the
-greeting. The assertion reads the room timeline back out of the messages table,
+Drives the genuine path: RoomService creates a room and invites the agent,
+provisioning writes the agent's membership and its arrival, and the router,
+deciding the arrival, posts the greeting on the agent's behalf once it has
+committed. The assertion reads the room timeline back out of the messages table,
so nothing is inferred from in-process state.
"""
@@ -22,12 +23,11 @@ pytestmark = [pytest.mark.integration, pytest.mark.asyncio(loop_scope="session")
async def _wait_joined(
client: object, transport_room_id: str, timeout: float = 30
) -> None:
- deadline = asyncio.get_event_loop().time() + timeout
- while asyncio.get_event_loop().time() < deadline:
- if transport_room_id in client.room_join_times: # type: ignore[attr-defined]
- return
- await asyncio.sleep(0.25)
- raise AssertionError(f"client never joined {transport_room_id} within {timeout}s")
+ """Block until the agent is a member of the room: its membership is a
+ row, written when it was invited (`Actor.wait_joined` reads it)."""
+ assert await client.wait_joined(transport_room_id, timeout), ( # type: ignore[attr-defined]
+ f"client never joined {transport_room_id} within {timeout}s"
+ )
async def _timeline_messages(
diff --git a/core/tests/integration/test_harness_smoke.py b/core/tests/integration/test_harness_smoke.py
index 3e6b17966..a008fdb8c 100644
--- a/core/tests/integration/test_harness_smoke.py
+++ b/core/tests/integration/test_harness_smoke.py
@@ -8,8 +8,6 @@ exercises the harness itself.
from __future__ import annotations
-import asyncio
-
import pytest
from switch_core.room_service import RoomCreateConfig
@@ -24,13 +22,11 @@ pytestmark = [pytest.mark.integration, pytest.mark.asyncio(loop_scope="session")
async def _wait_joined(
client: object, transport_room_id: str, timeout: float = 30
) -> None:
- """Block until the agent's own client has joined the room."""
- deadline = asyncio.get_event_loop().time() + timeout
- while asyncio.get_event_loop().time() < deadline:
- if transport_room_id in client.room_join_times: # type: ignore[attr-defined]
- return
- await asyncio.sleep(0.25)
- raise AssertionError(f"client never joined {transport_room_id} within {timeout}s")
+ """Block until the agent is a member of the room: its membership is a
+ row, written when it was invited (`Actor.wait_joined` reads it)."""
+ assert await client.wait_joined(transport_room_id, timeout), ( # type: ignore[attr-defined]
+ f"client never joined {transport_room_id} within {timeout}s"
+ )
async def test_harness_boots_and_agents_join_room(harness: Harness) -> None:
diff --git a/core/tests/integration/test_removal_stops_delivery_e2e.py b/core/tests/integration/test_removal_stops_delivery_e2e.py
index 33e9498ce..5a8ea1bab 100644
--- a/core/tests/integration/test_removal_stops_delivery_e2e.py
+++ b/core/tests/integration/test_removal_stops_delivery_e2e.py
@@ -1,15 +1,14 @@
"""End-to-end: removing an agent from a room stops delivering that room to it.
Drives the genuine path, no mocks and no stubs. `RoomService` removes the agent
-the way the gateway's `DELETE /rooms/{id}/agents/{id}` does → the kick is rung
-over the invite bus → the transport drops the subscription and tells its client
-→ the client empties the event buffer of that room → every reader stops serving
-it.
+the way the gateway's `DELETE /rooms/{id}/agents/{id}` does, the kick deletes
+the agent's own events in that room in the transaction that removes the
+membership (`MessageRouter.remove`), and every reader stops serving it.
The unit tests cover each of those links. What only this can show is that they
-are joined: severing the one line in `Actor.setup` that wires the removal
-handler to the transport left the whole unit suite green while a kick reached
-nothing.
+are joined: a removal that reached the membership but not the agent's events
+would leave the whole unit suite green while a resuming reader was handed the
+room.
Two readers rather than one, because they fail differently. `poll_events`
applies membership from the database on every poll, so it is right even if the
@@ -34,12 +33,11 @@ pytestmark = [pytest.mark.integration, pytest.mark.asyncio(loop_scope="session")
async def _wait_joined(
client: object, transport_room_id: str, timeout: float = 30
) -> None:
- deadline = asyncio.get_event_loop().time() + timeout
- while asyncio.get_event_loop().time() < deadline:
- if transport_room_id in client.room_join_times: # type: ignore[attr-defined]
- return
- await asyncio.sleep(0.25)
- raise AssertionError(f"client never joined {transport_room_id} within {timeout}s")
+ """Block until the agent is a member of the room: its membership is a
+ row, written when it was invited (`Actor.wait_joined` reads it)."""
+ assert await client.wait_joined(transport_room_id, timeout), ( # type: ignore[attr-defined]
+ f"client never joined {transport_room_id} within {timeout}s"
+ )
async def _wait_buffered(
diff --git a/core/tests/integration/test_room_join_e2e.py b/core/tests/integration/test_room_join_e2e.py
index 226f4567a..6e8bff4e9 100644
--- a/core/tests/integration/test_room_join_e2e.py
+++ b/core/tests/integration/test_room_join_e2e.py
@@ -1,9 +1,9 @@
"""End-to-end test for the room_join event over the real transport.
-Drives the genuine path: RoomService adds an agent to a room → the invite reaches
-the joiner → the watcher's AgentConsumer receive loop sees the arrival →
-on_member_event enqueues a `room_join` AgentEvent → it is read off the event
-buffer. No mocks, no HTTP layer.
+Drives the genuine path: RoomService adds an agent to a room, provisioning
+writes the joiner's membership and arrival, and the router stores a
+`room_join` event for the watcher in the same transaction, which is read off
+the event buffer. No mocks, no HTTP layer.
"""
from __future__ import annotations
@@ -25,13 +25,11 @@ pytestmark = [pytest.mark.integration, pytest.mark.asyncio(loop_scope="session")
async def _wait_joined(
client: object, transport_room_id: str, timeout: float = 30
) -> None:
- """Block until the agent's own client has joined the room."""
- deadline = asyncio.get_event_loop().time() + timeout
- while asyncio.get_event_loop().time() < deadline:
- if transport_room_id in client.room_join_times: # type: ignore[attr-defined]
- return
- await asyncio.sleep(0.25)
- raise AssertionError(f"client never joined {transport_room_id} within {timeout}s")
+ """Block until the agent is a member of the room: its membership is a
+ row, written when it was invited (`Actor.wait_joined` reads it)."""
+ assert await client.wait_joined(transport_room_id, timeout), ( # type: ignore[attr-defined]
+ f"client never joined {transport_room_id} within {timeout}s"
+ )
async def _drain_room_join(
@@ -60,7 +58,8 @@ async def test_join_emits_room_join_event(harness: Harness) -> None:
await harness.client_for(joiner.agent_id).wait_ready()
# Room with only the watcher, so the joiner's join is strictly later than the
- # watcher's own join-time (the AgentConsumer filters out events predating it).
+ # watcher's own membership (the router hands no member a row from before it
+ # joined).
result = await harness.room_service.create_room(
RoomCreateConfig(
name="e2e-room-join",
@@ -103,6 +102,6 @@ async def test_self_join_does_not_emit_room_join(harness: Harness) -> None:
)
await _wait_joined(watcher_client, result.room.transport_room_id)
- # Give the receive loop time to deliver any (erroneous) self-join event.
+ # Give anything erroneous time to arrive.
event = await _drain_room_join(harness, watcher.agent_id, result.room.id, timeout=8)
assert event is None, "self-join should not produce a room_join event"
diff --git a/core/tests/integration/test_switch_trust_agent_e2e.py b/core/tests/integration/test_switch_trust_agent_e2e.py
index 8dfc3cb93..a53427f44 100644
--- a/core/tests/integration/test_switch_trust_agent_e2e.py
+++ b/core/tests/integration/test_switch_trust_agent_e2e.py
@@ -10,8 +10,6 @@ nothing is inferred from in-process state.
from __future__ import annotations
-import asyncio
-
import pytest
from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
@@ -50,12 +48,12 @@ class _FailingTrustClient:
async def _wait_joined(
client: object, transport_room_id: str, timeout: float = 30
) -> None:
- deadline = asyncio.get_event_loop().time() + timeout
- while asyncio.get_event_loop().time() < deadline:
- if transport_room_id in client.room_join_times: # type: ignore[attr-defined]
- return
- await asyncio.sleep(0.25)
- raise AssertionError(f"client never joined {transport_room_id} within {timeout}s")
+ """Block until the agent is a member of the room: its membership is a
+ row, written when it was invited (`Actor.wait_joined` reads it). An
+ agent reads no room, so it does not hear its own join."""
+ assert await client.wait_joined(transport_room_id, timeout), ( # type: ignore[attr-defined]
+ f"client never joined {transport_room_id} within {timeout}s"
+ )
async def _timeline_messages(
diff --git a/core/tests/ownership.py b/core/tests/ownership.py
new file mode 100644
index 000000000..30344f20e
--- /dev/null
+++ b/core/tests/ownership.py
@@ -0,0 +1,253 @@
+"""A switch-core process's hold on work, for tests that read rooms.
+
+A client's rooms are read only by the process that owns the client, under its
+lease (`switch_core/ownership.py`), and the reading records progress through
+that hold's cursors (`switch_core/transport/cursors.py`). A test standing in
+for such a process claims the same rows production does: its own process
+lease, then the lease for each client it reads for, and runs the cursor writer
+the process would run.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from dataclasses import dataclass, field
+from typing import TYPE_CHECKING, Any
+
+from switch_core.db.models import TENANT_ZERO_ID
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.ownership import Fence, StartWork, StopWork
+from switch_core.process_identity import new_process_id
+from switch_core.process_lease import LEASE_TTL, ProcessLease
+from switch_core.transport.cursors import ConsumerCursors, CursorWriter
+from tests.conftest import TEST_KEYRING
+
+if TYPE_CHECKING:
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+ from switch_core.transport.postgres import PostgresTransport
+
+logger = logging.getLogger(__name__)
+
+CLIENT = "client"
+
+
+def process_lease_on(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> ProcessLease:
+ """A test process's lease, under a name of its own, as `main` claims one
+ for each switch-core process (`process_identity.py`)."""
+ return ProcessLease(
+ process_id=new_process_id("test-pod"),
+ session_factory=session_factory,
+ processes=SwitchCoreProcessStore(),
+ key_ids=[key.id for key in TEST_KEYRING.keys],
+ )
+
+
+@dataclass
+class Holder:
+ """One process: its lease, its cursor writer, and the fences it holds."""
+
+ session_factory: async_sessionmaker[AsyncSession]
+ lease: ProcessLease
+ writer: CursorWriter
+ task: asyncio.Task[None]
+ lost: list[Fence] = field(default_factory=list)
+
+ @property
+ def process_id(self) -> str:
+ return self.lease.process_id
+
+ async def claim(self, key: str, *, kind: str = CLIENT) -> Fence:
+ """Claim the lease for `key`, as the ownership would. Raises when
+ another live process holds it."""
+ async with self.session_factory() as session:
+ claimed = await OwnershipLeaseStore().claim(
+ session,
+ kind=kind,
+ keys=[key],
+ process_id=self.process_id,
+ lease_ttl=LEASE_TTL,
+ )
+ await session.commit()
+ if key not in claimed:
+ raise AssertionError(f"{kind} {key} is held by another process")
+ return Fence(kind=kind, key=key, process_id=self.process_id, token=claimed[key])
+
+ async def cursors(
+ self,
+ client_id: str,
+ *,
+ tenant_id: str = TENANT_ZERO_ID,
+ kind: str = CLIENT,
+ key: str | None = None,
+ actor_role: str = "agent",
+ ) -> ConsumerCursors:
+ """Cursors for reading `client_id`'s rooms under a fresh claim."""
+ fence = await self.claim(key or client_id, kind=kind)
+ return ConsumerCursors(
+ writer=self.writer,
+ store=DeliveryCursorStore(),
+ tenant_id=tenant_id,
+ client_id=client_id,
+ fence=fence,
+ actor_role=actor_role,
+ on_lost=self.lost.append,
+ )
+
+ async def receiving(self, transport: PostgresTransport) -> asyncio.Task[None]:
+ """Start `transport` reading its client's rooms under this process."""
+ cursors = await self.cursors(transport.client_id, tenant_id=transport.tenant_id)
+ return asyncio.create_task(transport.receive_forever(cursors))
+
+ async def stop(self) -> None:
+ self.task.cancel()
+ await asyncio.gather(self.task, return_exceptions=True)
+
+
+async def start_holder(session_factory: async_sessionmaker[AsyncSession]) -> Holder:
+ """A process whose lease is claimed and whose cursor writer runs."""
+ lease = process_lease_on(session_factory)
+ await lease.renew()
+ writer = CursorWriter(
+ session_factory=session_factory,
+ cursors=DeliveryCursorStore(),
+ leases=OwnershipLeaseStore(),
+ process_id=lease.process_id,
+ )
+ return Holder(
+ session_factory=session_factory,
+ lease=lease,
+ writer=writer,
+ task=asyncio.create_task(writer.run()),
+ )
+
+
+class NeverOwned:
+ """An ownership under which this process claims nothing, for tests of a
+ lifecycle that are not about which process owns what: work is wanted and
+ dropped as the code under test asks, and never started."""
+
+ def kind(self, name: str, *, start: object, stop: object) -> _NeverOwnedKind:
+ return _NeverOwnedKind()
+
+
+class _NeverOwnedKind:
+ def __init__(self) -> None:
+ self._wanted: set[str] = set()
+
+ def want(self, key: str) -> None:
+ self._wanted.add(key)
+
+ def wants(self, key: str) -> bool:
+ return key in self._wanted
+
+ async def drop(self, key: str) -> None:
+ self._wanted.discard(key)
+
+ def fence(self, key: str) -> Fence | None:
+ return None
+
+ def lost(self, fence: Fence) -> None:
+ return None
+
+
+class SoleOwner:
+ """An ownership under which this process is the only one, and holds all
+ the work it wants, as a deployment of one process does after its first
+ balance. For unit tests of a lifecycle that run without a database: work
+ wanted starts when the code under test settles it or the test calls
+ `start()`, as `main` does at boot, and every key reads as held here, so
+ work a test starts by hand runs as its holder would."""
+
+ process_id = "the-only-process"
+
+ def __init__(self) -> None:
+ self.stopping = False
+ self._kinds: list[_SoleKind] = []
+
+ def kind(self, name: str, *, start: StartWork, stop: StopWork) -> _SoleKind:
+ kind = _SoleKind(self, name, start, stop)
+ self._kinds.append(kind)
+ return kind
+
+ async def start(self) -> None:
+ for kind in self._kinds:
+ await kind.settle()
+
+ async def stop(self) -> None:
+ self.stopping = True
+ for kind in self._kinds:
+ for key in sorted(kind.held()):
+ await kind.drop(key)
+
+ @staticmethod
+ def cursor_writer(
+ session_factory: async_sessionmaker[AsyncSession] | Any,
+ ) -> CursorWriter:
+ """The process's cursor writer, never run: what a consumer under it
+ records is written only when flushed."""
+ return CursorWriter(
+ session_factory=session_factory,
+ cursors=DeliveryCursorStore(),
+ leases=OwnershipLeaseStore(),
+ process_id=SoleOwner.process_id,
+ )
+
+
+class _SoleKind:
+ def __init__(
+ self, owner: SoleOwner, name: str, start: StartWork, stop: StopWork
+ ) -> None:
+ self.name = name
+ self._owner = owner
+ self._start = start
+ self._stop = stop
+ self._wanted: set[str] = set()
+ self._started: set[str] = set()
+
+ def want(self, key: str) -> None:
+ self._wanted.add(key)
+
+ def wanted(self) -> set[str]:
+ return set(self._wanted)
+
+ def wants(self, key: str) -> bool:
+ return key in self._wanted
+
+ @property
+ def stopping(self) -> bool:
+ return self._owner.stopping
+
+ async def settle(self) -> None:
+ for key in sorted(self._wanted - self._started):
+ self._started.add(key)
+ fence = self.fence(key)
+ assert fence is not None
+ # As the ownership does: one key failing to start is logged, and
+ # the rest still start.
+ try:
+ await self._start(key, fence)
+ except Exception:
+ logger.exception("Starting %s %s failed", self.name, key)
+
+ def held(self) -> set[str]:
+ return set(self._started)
+
+ def fence(self, key: str) -> Fence | None:
+ return Fence(
+ kind=self.name, key=key, process_id=self._owner.process_id, token=1
+ )
+
+ async def drop(self, key: str) -> None:
+ self._wanted.discard(key)
+ if key in self._started:
+ self._started.discard(key)
+ await self._stop(key)
+
+ def lost(self, fence: Fence) -> None:
+ return None
diff --git a/core/tests/routing.py b/core/tests/routing.py
new file mode 100644
index 000000000..00b4113a6
--- /dev/null
+++ b/core/tests/routing.py
@@ -0,0 +1,449 @@
+"""The message router on the test database, and the rooms, agents and people
+it routes between.
+
+A row is decided as it is written, so a test of what an agent receives writes
+the row the way production does: through a client's transport, which routes
+it (`delivery/router.py`). What the agent was given is then read back from
+`agent_events`, and what was said in the room from `messages`, because both
+are the record: a fake standing in for either would test the fake.
+
+A `Process` is one switch-core process's worth of routing: its own router,
+client factory, event buffer and connection registry, on the shared
+database. Two of them are two processes, which is how the tests show a row is
+routed once, by the process that wrote it.
+"""
+
+from __future__ import annotations
+
+import uuid
+from dataclasses import dataclass, field
+from types import SimpleNamespace
+from typing import TYPE_CHECKING, Any
+
+from sqlalchemy import insert, select
+
+from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
+from switch_core.bridges.agent.protocol.event_buffer import (
+ EventBuffer,
+ RoomAnnouncements,
+)
+from switch_core.clients.client_factory import ClientFactory
+from switch_core.db.models import (
+ TENANT_ZERO_ID,
+ Agent,
+ AgentEventRow,
+ ApiKey,
+ Client,
+ ClientRoom,
+ Message,
+ Room,
+ User,
+ room_agents,
+)
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.agent_session_store import AgentSessionStore
+from switch_core.db.stores.agent_store import AgentStore
+from switch_core.db.stores.budget_store import BudgetStore
+from switch_core.db.stores.client_store import ClientStore
+from switch_core.db.stores.document_store import DocumentStore
+from switch_core.db.stores.external_user_store import ExternalUserStore
+from switch_core.db.stores.media_store import MediaStore
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.db.stores.reference_store import ReferenceStore
+from switch_core.db.stores.room_role_store import RoomRoleStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.usage_store import UsageStore
+from switch_core.delivery.recipient import RecipientServices
+from switch_core.delivery.router import MessageRouter
+from switch_core.messages.notify import MessageListener
+from switch_core.provisioning.postgres import PostgresProvisioning
+from switch_core.transport.ephemeral import EphemeralBus
+from switch_core.transport.invites import InviteBus
+from switch_core.transport.postgres import PostgresTransport
+from switch_core.transport.room_cache import (
+ DELIVERY_PAGE,
+ RoomCacheLimits,
+ RoomDeliveryCache,
+)
+from tests.db_buffer import db_buffer_on
+
+if TYPE_CHECKING:
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+ from switch_core.transport.observer import ParticipantMessage
+
+
+class RecordedTelemetry:
+ """Message telemetry that records what it was told: the agents asked for
+ something (`addressed`) and the messages people and agents sent
+ (`observed`)."""
+
+ def __init__(self) -> None:
+ self.enabled = True
+ self.addressed: list[dict[str, Any]] = []
+ self.observed: list[ParticipantMessage] = []
+
+ def agent_addressed(self, **report: Any) -> None:
+ self.addressed.append(report)
+
+ def observe(self, message: ParticipantMessage) -> None:
+ self.observed.append(message)
+
+
+@dataclass
+class Process:
+ """One switch-core process's routing on the test database."""
+
+ session_factory: async_sessionmaker[AsyncSession]
+ router: MessageRouter
+ factory: ClientFactory
+ buffer: EventBuffer
+ connections: AgentConnectionRegistry
+ telemetry: RecordedTelemetry
+ invites: InviteBus
+ provisioning: PostgresProvisioning
+ transports: dict[str, PostgresTransport] = field(default_factory=dict)
+
+ def transport(self, client: Client) -> PostgresTransport:
+ """The transport `client` writes through in this process."""
+ transport = self.transports.get(client.id)
+ if transport is None:
+ transport = self.factory.transport_for(
+ SimpleNamespace( # type: ignore[arg-type]
+ transport_user_id=client.transport_user_id,
+ client_id=client.id,
+ tenant_id=client.tenant_id,
+ display_name=client.display_name,
+ role="agent" if client.type == "agent" else "human",
+ )
+ )
+ assert isinstance(transport, PostgresTransport)
+ self.transports[client.id] = transport
+ return transport
+
+ async def say(
+ self,
+ room: Room,
+ body: str,
+ *,
+ sender: Client,
+ thread_root_id: str | None = None,
+ extra_content: dict[str, object] | None = None,
+ ) -> str:
+ """Write a message as `sender`, and return its event id."""
+ result = await self.transport(sender).send_message(
+ room.transport_room_id,
+ body,
+ sender_name=sender.display_name,
+ metered=False,
+ format="markdown",
+ thread_root_id=thread_root_id,
+ extra_content=extra_content,
+ )
+ return result.event_id
+
+ async def send_file(
+ self,
+ room: Room,
+ filename: str,
+ *,
+ sender: Client,
+ caption: str | None = None,
+ group: dict[str, object] | None = None,
+ ) -> str:
+ """Write one file as `sender`, a part of a multi-file post when
+ `group` says so, and return its event id."""
+ result = await self.transport(sender).send_media(
+ room.transport_room_id,
+ f"switch-media://{uuid.uuid4().hex}",
+ filename,
+ "image/png",
+ 5,
+ sender_name=sender.display_name,
+ metered=False,
+ msgtype="m.image",
+ caption=caption,
+ group=group,
+ )
+ return result.event_id
+
+ async def command(
+ self, room: Room, command: str, args: str, *, sender: Client
+ ) -> str:
+ """Type a room command as `sender`, and return its event id."""
+ result = await self.transport(sender).send_event(
+ room.transport_room_id,
+ "com.switch.command",
+ {
+ "command": command,
+ "args": args,
+ "user_id": sender.transport_user_id,
+ "user_name": sender.display_name,
+ },
+ )
+ return result.event_id
+
+ async def idle(self) -> None:
+ """Wait for every reply the routed rows owe to have been posted."""
+ await self.router.idle()
+
+
+def router_on(
+ session_factory: async_sessionmaker[AsyncSession],
+ *,
+ buffer: EventBuffer | None = None,
+ connections: AgentConnectionRegistry | None = None,
+ telemetry: RecordedTelemetry | None = None,
+) -> MessageRouter:
+ """A router on the test database with real stores. A test that has no
+ agent in its rooms can write through it with nothing else set up."""
+ return MessageRouter(
+ services=RecipientServices(
+ session_factory=session_factory,
+ event_buffer=buffer
+ if buffer is not None
+ else db_buffer_on(session_factory),
+ agent_store=AgentStore(),
+ room_store=RoomStore(),
+ document_store=DocumentStore(),
+ reference_store=ReferenceStore(),
+ agent_session_store=AgentSessionStore(),
+ room_role_store=RoomRoleStore(),
+ external_user_store=ExternalUserStore(),
+ connections=connections
+ if connections is not None
+ else AgentConnectionRegistry(),
+ frontend_base_url=None,
+ message_telemetry=telemetry # type: ignore[arg-type]
+ if telemetry is not None
+ else RecordedTelemetry(),
+ ),
+ message_store=MessageStore(),
+ client_store=ClientStore(),
+ budget_store=BudgetStore(),
+ )
+
+
+def process_on(
+ session_factory: async_sessionmaker[AsyncSession],
+ *,
+ rooms: RoomAnnouncements | None = None,
+ connections: AgentConnectionRegistry | None = None,
+) -> Process:
+ """A process's routing on the test database: real stores, its own
+ registry and buffer, telemetry that records. A test of presence across
+ processes hands it a registry that sees the other processes'
+ connections (`connection_rows.SharedConnections`)."""
+ if connections is None:
+ connections = AgentConnectionRegistry()
+ buffer = db_buffer_on(session_factory, rooms=rooms)
+ telemetry = RecordedTelemetry()
+ message_store = MessageStore()
+ room_store = RoomStore()
+ client_store = ClientStore()
+ router = router_on(
+ session_factory, buffer=buffer, connections=connections, telemetry=telemetry
+ )
+ invites = InviteBus()
+ factory = ClientFactory(
+ client_store=client_store,
+ session_factory=session_factory,
+ config=SimpleNamespace(), # type: ignore[arg-type]
+ room_store=room_store,
+ message_store=message_store,
+ usage_store=UsageStore(),
+ media_store=MediaStore(),
+ # Never started: these transports only write, and nothing they
+ # deliver to is listening.
+ listener=MessageListener(lambda: None), # type: ignore[arg-type, return-value]
+ invites=invites,
+ ephemeral=EphemeralBus(),
+ room_cache=RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=message_store,
+ limits=RoomCacheLimits(
+ max_bytes=64 * 1024 * 1024,
+ max_rooms=1000,
+ max_rows_per_room=1000,
+ max_age_seconds=300.0,
+ ),
+ page=DELIVERY_PAGE,
+ ),
+ message_observer=telemetry,
+ router=router,
+ )
+ provisioning = PostgresProvisioning(
+ session_factory=session_factory,
+ room_store=room_store,
+ client_store=client_store,
+ router=router,
+ invites=invites,
+ )
+ return Process(
+ session_factory=session_factory,
+ router=router,
+ factory=factory,
+ buffer=buffer,
+ connections=connections,
+ telemetry=telemetry,
+ invites=invites,
+ provisioning=provisioning,
+ )
+
+
+# ── Rooms, agents and people ──────────────────────────────────────────────────
+
+
+async def make_room(
+ session_factory: async_sessionmaker[AsyncSession],
+ name: str = "room",
+ *,
+ channel_type: str | None = "channel_public",
+ bridge_id: str | None = None,
+ tenant_id: str = TENANT_ZERO_ID,
+) -> Room:
+ async with tenant_session(session_factory, tenant_id) as session:
+ room = Room(
+ transport_room_id=f"!{name}-{uuid.uuid4().hex[:8]}:test",
+ name=name,
+ description="",
+ channel_type=channel_type,
+ bridge_id=bridge_id,
+ )
+ session.add(room)
+ await session.commit()
+ await session.refresh(room)
+ return room
+
+
+async def make_person(
+ session_factory: async_sessionmaker[AsyncSession],
+ name: str = "alice",
+ *,
+ tenant_id: str = TENANT_ZERO_ID,
+) -> Client:
+ """A person on another platform, as the client their messages come from."""
+ async with tenant_session(session_factory, tenant_id) as session:
+ client = Client(
+ transport_user_id=f"@{name}-{uuid.uuid4().hex[:6]}:test",
+ display_name=name,
+ type="user",
+ )
+ session.add(client)
+ await session.commit()
+ await session.refresh(client)
+ return client
+
+
+async def make_agent(
+ session_factory: async_sessionmaker[AsyncSession],
+ name: str,
+ *,
+ connection_model: str = "always_on",
+ addressing_policy: dict[str, Any] | None = None,
+ tenant_id: str = TENANT_ZERO_ID,
+) -> tuple[Agent, Client]:
+ """An agent, its client, and the owner and key it needs."""
+ async with tenant_session(session_factory, tenant_id) as session:
+ suffix = uuid.uuid4().hex[:8]
+ user = User(name=f"owner-{suffix}", email=f"{suffix}@test.local", role="user")
+ session.add(user)
+ await session.flush()
+ key = ApiKey(
+ user_id=user.id,
+ key_hash=f"hash-{suffix}",
+ encrypted_key="enc",
+ label=name,
+ type="agent",
+ )
+ client = Client(
+ transport_user_id=f"@{name}-{suffix}:test",
+ display_name=name,
+ type="agent",
+ )
+ session.add_all([key, client])
+ await session.flush()
+ agent = Agent(
+ name=name,
+ description="",
+ agent_type=connection_model,
+ connector_type="claude_code",
+ integration_profile={"connection_model": connection_model},
+ client_id=client.id,
+ api_key_id=key.id,
+ owner_id=user.id,
+ addressing_policy=addressing_policy,
+ )
+ session.add(agent)
+ await session.commit()
+ await session.refresh(agent)
+ await session.refresh(client)
+ return agent, client
+
+
+async def add_member(
+ session_factory: async_sessionmaker[AsyncSession],
+ room: Room,
+ client: Client,
+ *,
+ agent: Agent | None = None,
+ alias: str | None = None,
+ listening: bool = False,
+) -> None:
+ """Put a client in a room, written straight to the tables as a member
+ from before the test would be: no arrival is written, so nobody is
+ greeted. An agent's `room_agents` row says its alias and whether it
+ listens for joins."""
+ async with tenant_session(session_factory, room.tenant_id) as session:
+ session.add(ClientRoom(client_id=client.id, room_id=room.id))
+ if agent is not None:
+ await session.execute(
+ insert(room_agents).values(
+ room_id=room.id,
+ agent_id=agent.id,
+ alias=alias,
+ receives_join_events=listening,
+ )
+ )
+ await session.commit()
+
+
+async def own_events(
+ session_factory: async_sessionmaker[AsyncSession],
+ agent_id: str,
+ *,
+ tenant_id: str = TENANT_ZERO_ID,
+) -> list[AgentEventRow]:
+ """The agent's own events, oldest first."""
+ async with tenant_session(session_factory, tenant_id) as session:
+ return list(
+ (
+ await session.scalars(
+ select(AgentEventRow)
+ .where(AgentEventRow.agent_id == agent_id)
+ .order_by(AgentEventRow.seq)
+ )
+ ).all()
+ )
+
+
+async def said_by(
+ session_factory: async_sessionmaker[AsyncSession],
+ room: Room,
+ client: Client,
+) -> list[Message]:
+ """What `client` has said in `room`, oldest first."""
+ async with tenant_session(session_factory, room.tenant_id) as session:
+ return list(
+ (
+ await session.scalars(
+ select(Message)
+ .where(
+ Message.room_id == room.id,
+ Message.sender_client_id == client.id,
+ Message.event_type == "m.room.message",
+ )
+ .order_by(Message.seq)
+ )
+ ).all()
+ )
diff --git a/core/tests/switch_core/bridges/agent/api/test_connection_placements.py b/core/tests/switch_core/bridges/agent/api/test_connection_placements.py
index c9b8943ca..989c5cce8 100644
--- a/core/tests/switch_core/bridges/agent/api/test_connection_placements.py
+++ b/core/tests/switch_core/bridges/agent/api/test_connection_placements.py
@@ -147,6 +147,24 @@ async def test_a_room_the_agent_is_not_in_refuses_the_whole_request(
assert conn.rooms == {ROOM_A}
+async def test_an_over_long_session_id_refuses_the_whole_request(
+ db_buffer: EventBuffer,
+) -> None:
+ """A session id travels in the signals switch-core processes send each
+ other when its room is taken, which Postgres caps at 8000 bytes."""
+ protocol = _Protocol(db_buffer)
+ conn = _open(protocol, "watcher")
+ await _place(protocol, conn, {"session-1": ROOM_A})
+
+ with pytest.raises(HTTPException) as refused:
+ await _place(protocol, conn, {"session-1": ROOM_B, "s" * 7900: ROOM_A})
+
+ assert refused.value.status_code == 400
+ assert protocol.connections.connection_placements(conn) == {"session-1": ROOM_A}
+ assert conn.rooms == {ROOM_A}
+
+
@pytest.mark.asyncio
async def test_two_sessions_in_one_room_are_refused(db_buffer: EventBuffer) -> None:
protocol = _Protocol(db_buffer)
diff --git a/core/tests/switch_core/bridges/agent/api/test_connection_socket.py b/core/tests/switch_core/bridges/agent/api/test_connection_socket.py
index eb54471da..273bab5a4 100644
--- a/core/tests/switch_core/bridges/agent/api/test_connection_socket.py
+++ b/core/tests/switch_core/bridges/agent/api/test_connection_socket.py
@@ -210,6 +210,20 @@ def test_a_refusal_is_sent_then_the_socket_closes_with_its_status(
assert closed.value.code == 4400
+def test_an_over_long_connection_id_is_refused(
+ client: TestClient, protocol: _Protocol
+) -> None:
+ with client.websocket_connect(_url(connection_id="c" * 7900)) as ws:
+ refused = ws.receive_json()
+
+ assert refused["event"] == "refused"
+ assert refused["data"]["status"] == 400
+ with pytest.raises(WebSocketDisconnect) as closed:
+ ws.receive_json()
+ assert closed.value.code == 4400
+ assert protocol.connections.get("c" * 7900) is None
+
+
def test_closing_the_socket_detaches_the_stream(
client: TestClient, protocol: _Protocol
) -> None:
diff --git a/core/tests/switch_core/bridges/agent/api/test_events_endpoint_dispatch.py b/core/tests/switch_core/bridges/agent/api/test_events_endpoint_dispatch.py
index 2cbe07a53..bd4b43fa8 100644
--- a/core/tests/switch_core/bridges/agent/api/test_events_endpoint_dispatch.py
+++ b/core/tests/switch_core/bridges/agent/api/test_events_endpoint_dispatch.py
@@ -202,6 +202,27 @@ async def test_streaming_without_a_connection_id_is_refused(
assert "connection_id is required" in excinfo.value.detail
+async def test_streaming_with_an_over_long_connection_id_is_refused(
+ db_buffer: EventBuffer,
+) -> None:
+ """The id travels in the signals switch-core processes send each other,
+ which Postgres caps at 8000 bytes, so one this long cannot be opened."""
+ protocol = _Protocol(db_buffer)
+ with pytest.raises(HTTPException) as excinfo:
+ await poll_events(
+ agent_id=AGENT_ID,
+ agent=_agent(),
+ protocol=protocol, # type: ignore[arg-type]
+ config=None, # type: ignore[arg-type]
+ accept="text/event-stream",
+ connection_id="c" * 7900,
+ )
+
+ assert excinfo.value.status_code == 400
+ assert "connection_id" in excinfo.value.detail
+ assert protocol.connections.get("c" * 7900) is None
+
+
@pytest.mark.parametrize(
("field", "value"),
[("scope", "everything"), ("event_filter", "some")],
diff --git a/core/tests/switch_core/bridges/agent/api/test_session_selector.py b/core/tests/switch_core/bridges/agent/api/test_session_selector.py
index 771c818b9..17a3c54e1 100644
--- a/core/tests/switch_core/bridges/agent/api/test_session_selector.py
+++ b/core/tests/switch_core/bridges/agent/api/test_session_selector.py
@@ -100,6 +100,17 @@ async def test_a_session_selector_needs_its_connection() -> None:
assert alias.lower() in refused.value.detail.lower()
+async def test_an_over_long_session_id_is_refused() -> None:
+ """The session's id is recorded with its room, and travels in the
+ signals switch-core processes send each other when another session takes
+ that room, which Postgres caps at 8000 bytes."""
+ with pytest.raises(HTTPException) as refused:
+ await _resolve(_protocol(), connection_id=CONNECTION, session_id="s" * 7900)
+
+ assert refused.value.status_code == 400
+
+
@pytest.mark.asyncio
async def test_a_dead_or_foreign_connection_is_refused() -> None:
protocol = _protocol()
diff --git a/core/tests/switch_core/bridges/agent/mcp/test_sessions_across_processes.py b/core/tests/switch_core/bridges/agent/mcp/test_sessions_across_processes.py
new file mode 100644
index 000000000..d2ac85b9f
--- /dev/null
+++ b/core/tests/switch_core/bridges/agent/mcp/test_sessions_across_processes.py
@@ -0,0 +1,87 @@
+"""The MCP door holds no session in memory, so a client that initialized on
+one switch-core process carries on through another.
+
+Two transports stand in for two processes: each is its own stateless
+`http_app`, with nothing shared between them but the client's session id.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from contextlib import AsyncExitStack, asynccontextmanager
+from typing import Any
+
+import httpx
+from starlette.types import ASGIApp, Receive, Scope, Send
+
+from switch_core.bridges.agent.mcp import server as mcp_server
+
+AGENT = "agent-1"
+HEADERS = {
+ "accept": "application/json, text/event-stream",
+ "content-type": "application/json",
+}
+INITIALIZE = {
+ "jsonrpc": "2.0",
+ "id": 1,
+ "method": "initialize",
+ "params": {
+ "protocolVersion": "2025-06-18",
+ "capabilities": {},
+ "clientInfo": {"name": "test", "version": "0"},
+ },
+}
+
+
+def _as_agent(app: ASGIApp) -> ASGIApp:
+ """What the bearer middleware leaves on the scope of an agent's request."""
+
+ async def wrapped(scope: Scope, receive: Receive, send: Send) -> None:
+ if scope["type"] == "http":
+ scope = {**scope, "agent_id": AGENT}
+ await app(scope, receive, send)
+
+ return wrapped
+
+
+@asynccontextmanager
+async def _process() -> AsyncIterator[httpx.AsyncClient]:
+ starlette_app: Any = mcp_server.mcp_http_app()
+ async with (
+ starlette_app.router.lifespan_context(starlette_app),
+ httpx.AsyncClient(
+ transport=httpx.ASGITransport(
+ app=_as_agent(mcp_server.SessionIds(starlette_app))
+ ),
+ base_url="http://switch.test",
+ ) as client,
+ ):
+ yield client
+
+
+async def test_a_session_started_on_one_process_goes_on_on_another() -> None:
+ async with AsyncExitStack() as stack:
+ first = await stack.enter_async_context(_process())
+ second = await stack.enter_async_context(_process())
+
+ initialized = await first.post("/", json=INITIALIZE, headers=HEADERS)
+ assert initialized.status_code == 200, initialized.text
+ session_id = initialized.headers["mcp-session-id"]
+
+ listed = await second.post(
+ "/",
+ json={"jsonrpc": "2.0", "id": 2, "method": "tools/list"},
+ headers={**HEADERS, "mcp-session-id": session_id},
+ )
+ assert listed.status_code == 200, listed.text
+ # A session id the other process never issued is not refused, and is
+ # not replaced: the client keeps the one it has.
+ assert "mcp-session-id" not in listed.headers
+
+
+def test_the_binding_key_belongs_to_the_agent() -> None:
+ """A session id comes from the client; the agent's own id comes from its
+ credential. Together they keep one agent off another's room binding."""
+ assert mcp_server.session_key("agent-1", "s") != mcp_server.session_key(
+ "agent-2", "s"
+ )
diff --git a/core/tests/switch_core/bridges/agent/protocol/registration_harness.py b/core/tests/switch_core/bridges/agent/protocol/registration_harness.py
index 8d3bbfb76..08d822687 100644
--- a/core/tests/switch_core/bridges/agent/protocol/registration_harness.py
+++ b/core/tests/switch_core/bridges/agent/protocol/registration_harness.py
@@ -12,6 +12,9 @@ from types import SimpleNamespace
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.bridges.agent.api_key_cache import ApiKeyCache
+from switch_core.bridges.agent.protocol.agent_connections import (
+ AgentConnectionRegistry,
+)
from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.bridges.agent.protocol.types import (
IntegrationProfile,
@@ -58,6 +61,9 @@ class FakeClientLifecycle:
def start_client(self, client: Client) -> None:
self.started.append(client.id)
+ def announce(self, tenant_id: str) -> None:
+ return None
+
class NoBridges:
def bridges_for_tenant(self, tenant_id: str) -> list[object]:
@@ -75,6 +81,7 @@ def make_service(
svc.api_key_cache = ApiKeyCache(ttl_seconds=5.0, max_entries=8) # type: ignore[attr-defined]
svc.client_lifecycle = FakeClientLifecycle(session_factory) # type: ignore[attr-defined]
svc.collab_lifecycle = NoBridges() # type: ignore[attr-defined]
+ svc.connections = AgentConnectionRegistry() # type: ignore[attr-defined]
svc.config = SimpleNamespace(
keyring=Keyring.parse("test:" + "x" * 40, legacy_secret=None),
third_party_avatars_enabled=True,
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_agent_tenant_scoped_admin.py b/core/tests/switch_core/bridges/agent/protocol/test_agent_tenant_scoped_admin.py
index f36e8b556..8877738cc 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_agent_tenant_scoped_admin.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_agent_tenant_scoped_admin.py
@@ -77,6 +77,9 @@ class _FakeClientLifecycle:
def start_client(self, client: Client) -> None:
pass
+ def announce(self, tenant_id: str) -> None:
+ return None
+
class _NoBridges:
def bridges_for_tenant(self, tenant_id: str) -> list[object]:
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_bridge_identity_tenant_scope.py b/core/tests/switch_core/bridges/agent/protocol/test_bridge_identity_tenant_scope.py
index 4b8a69ae6..f3cf03999 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_bridge_identity_tenant_scope.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_bridge_identity_tenant_scope.py
@@ -143,6 +143,9 @@ class _StoppableClientLifecycle:
def start_client(self, client: Client) -> None:
pass
+ def announce(self, tenant_id: str) -> None:
+ return None
+
async def stop(self, client_id: str) -> None:
self.stopped.append(client_id)
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_connections_across_processes.py b/core/tests/switch_core/bridges/agent/protocol/test_connections_across_processes.py
new file mode 100644
index 000000000..37278e9bd
--- /dev/null
+++ b/core/tests/switch_core/bridges/agent/protocol/test_connections_across_processes.py
@@ -0,0 +1,1174 @@
+"""Agent connections seen, fenced and served by every switch-core process.
+
+Two processes on one database: each has its own connection registry, its own
+signal bus listening on the real `NOTIFY` channel, and its own lease. A
+connection's socket is held by one of them (A); its other requests, presence
+questions and old-client heartbeats may reach the other (B). Nothing here is a
+stand-in for the database or the bus, because what is under test is exactly
+what crosses between the two.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+from collections.abc import AsyncIterator, Callable
+from dataclasses import dataclass
+from datetime import timedelta
+from types import SimpleNamespace
+from typing import Any
+
+import pytest
+from fastapi import HTTPException
+from sqlalchemy import delete, select, update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.bridges.agent.api.handlers import (
+ connection_beat,
+ connection_placements,
+ connection_subscribe,
+)
+from switch_core.bridges.agent.api.operations import resolve_caller
+from switch_core.bridges.agent.api.schemas import (
+ ConnectionBeatRequest,
+ ConnectionPlacementsRequest,
+ ConnectionSubscribeRequest,
+)
+from switch_core.bridges.agent.protocol.agent_connections import (
+ HEARTBEAT_TTL_SECONDS,
+ PROTOCOL_VERSION,
+ AgentConnection,
+ AgentConnectionRegistry,
+ ClientDeclaration,
+ Closure,
+ SupersededReattachError,
+ UnknownConnectionError,
+)
+from switch_core.bridges.agent.protocol.connection_rows import SharedConnections
+from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
+from switch_core.bridges.agent.protocol.positions import OWN_EVENTS
+from switch_core.bridges.agent.protocol.presence import (
+ agents_present_in,
+ rooms_occupied,
+)
+from switch_core.bridges.agent.protocol.stream import KEEPALIVE, event_frames
+from switch_core.db.models import (
+ TENANT_ZERO_ID,
+ Agent,
+ AgentConnectionRecord,
+ AgentConnectionRoom,
+ AgentRoomRead,
+ AgentSessionPlacement,
+ SwitchCoreProcess,
+ Tenant,
+)
+from switch_core.db.stores.agent_connection_store import AgentConnectionStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.delivery.recipient import AgentRecipient
+from switch_core.process_identity import new_process_id
+from switch_core.process_lease import LEASE_TTL, ProcessLease
+from switch_core.signals import SignalBus
+from switch_core.tenant_context import tenant_scope
+from tests.conftest import TEST_KEYRING
+from tests.db_buffer import db_buffer_on, seed_agent, seed_room
+from tests.routing import (
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+ said_by,
+)
+
+AGENT = "agent-1"
+ROOM = "room-1"
+OTHER_ROOM = "room-2"
+CONN = "conn-1"
+
+SessionFactory = async_sessionmaker[AsyncSession]
+
+WAIT_SECONDS = 10.0
+
+
+@dataclass
+class Process:
+ """One switch-core process, as far as agent connections reach."""
+
+ registry: AgentConnectionRegistry
+ shared: SharedConnections
+ bus: SignalBus
+ lease: ProcessLease
+ buffer: EventBuffer
+ run: asyncio.Task[None] | None = None
+
+ @property
+ def protocol(self) -> Any:
+ """Enough of `AgentCore` for the connection handlers."""
+
+ async def require_room_member(agent_id: str, room_id: str) -> Any:
+ return SimpleNamespace(id=room_id)
+
+ return SimpleNamespace(
+ connections=self.registry,
+ event_buffer=self.buffer,
+ require_room_member=require_room_member,
+ )
+
+
+class _Agent:
+ id = AGENT
+ tenant_id = TENANT_ZERO_ID
+ metadata_ = None
+
+
+async def _start_process(postgres_url: str, session_factory: SessionFactory) -> Process:
+ process_id = new_process_id("test-pod")
+ bus = SignalBus(
+ session_factory,
+ lambda: create_async_engine(postgres_url, poolclass=NullPool),
+ process_id=process_id,
+ )
+ lease = ProcessLease(
+ process_id=process_id,
+ session_factory=session_factory,
+ processes=SwitchCoreProcessStore(),
+ key_ids=[key.id for key in TEST_KEYRING.keys],
+ )
+ registry = AgentConnectionRegistry()
+ shared = SharedConnections(
+ registry=registry,
+ session_factory=session_factory,
+ store=AgentConnectionStore(),
+ bus=bus,
+ lease=lease,
+ )
+ await lease.renew()
+ await shared.start()
+ await bus.start()
+ await asyncio.wait_for(bus.connected.wait(), WAIT_SECONDS)
+ process = Process(
+ registry=registry,
+ shared=shared,
+ bus=bus,
+ lease=lease,
+ buffer=db_buffer_on(session_factory),
+ )
+ process.run = asyncio.create_task(shared.run())
+ return process
+
+
+async def _stop_process(process: Process) -> None:
+ # The bus first, so no signal starts work after the process stopped
+ # running; then the run, which waits for the work it started.
+ await process.bus.stop()
+ if process.run is not None:
+ process.run.cancel()
+ await asyncio.gather(process.run, return_exceptions=True)
+
+
+async def processes(
+ postgres_url: str, session_factory: SessionFactory
+) -> AsyncIterator[tuple[Process, Process]]:
+ await seed_agent(session_factory, AGENT)
+ await seed_room(session_factory, ROOM)
+ await seed_room(session_factory, OTHER_ROOM)
+ first = await _start_process(postgres_url, session_factory)
+ second = await _start_process(postgres_url, session_factory)
+ try:
+ yield first, second
+ finally:
+ await _stop_process(first)
+ await _stop_process(second)
+
+
+class _NoHeartbeatRows:
+ """The heartbeat-row arm of presence, for clients that still poll: none
+ here, so only the connection can make the agent available."""
+
+ async def get_live_agent_ids(
+ self, _session: Any, _agent_ids: list[str], _room_id: str | None
+ ) -> set[str]:
+ return set()
+
+
+def _recipient_on(process: Process) -> Any:
+ """Enough of the agent's recipient, the router's side of the agent on this
+ process, to ask whether the agent is available."""
+ recipient = SimpleNamespace(
+ agent=SimpleNamespace(id=AGENT),
+ _connections=process.registry,
+ _agent_session_store=_NoHeartbeatRows(),
+ )
+ recipient._is_available = AgentRecipient._is_available.__get__(recipient)
+ return recipient
+
+
+async def _eventually(check: Callable[[], bool]) -> None:
+ deadline = time.monotonic() + WAIT_SECONDS
+ while not check():
+ if time.monotonic() > deadline:
+ raise AssertionError("the other process never caught up")
+ await asyncio.sleep(0.02)
+
+
+async def _open(
+ process: Process,
+ *,
+ agent_id: str = AGENT,
+ connection_id: str = CONN,
+ scope: str = "single",
+ expected_generation: int | None = None,
+ speaks: int = PROTOCOL_VERSION,
+) -> AgentConnection:
+ """Open a connection as the stream and socket handlers do."""
+ with tenant_scope(TENANT_ZERO_ID):
+ async with process.registry.slots(agent_id):
+ async with process.registry.recorded(agent_id):
+ return process.registry.open(
+ agent_id=agent_id,
+ connection_id=connection_id,
+ scope=scope, # type: ignore[arg-type]
+ delivery_filter="addressed",
+ spawn_capable=scope == "all",
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(speaks=speaks),
+ expected_generation=expected_generation,
+ )
+
+
+async def _subscribe(
+ process: Process, room_id: str, generation: int | None
+) -> dict[str, Any]:
+ return await connection_subscribe(
+ AGENT,
+ ConnectionSubscribeRequest(
+ connection_id=CONN, room_id=room_id, generation=generation
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=process.protocol,
+ )
+
+
+async def _place(
+ process: Process, generation: int, placements: dict[str, str]
+) -> dict[str, Any]:
+ return await connection_placements(
+ AGENT,
+ ConnectionPlacementsRequest(
+ connection_id=CONN, placements=placements, generation=generation
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=process.protocol,
+ )
+
+
+class TestPresence:
+ async def test_an_agent_connected_to_one_process_reads_online_from_the_other(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ a, b = processes
+ conn = await _open(a)
+ await _subscribe(a, ROOM, conn.stream_generation)
+
+ await _eventually(lambda: b.registry.live_in_room(AGENT, ROOM))
+
+ assert b.registry.is_live(AGENT)
+ assert agents_present_in([AGENT], ROOM, b.registry) == {AGENT}
+ assert CONN in b.registry.live_connection_ids()
+ # The copy is not counted in this process's own gauge: each process
+ # reports the connections it holds.
+ assert b.registry.live_agents_by_transport() == {}
+
+ async def test_a_router_on_the_other_process_owes_no_offline_reply(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ """The process that writes an addressed message decides whether the
+ agent is available, and need not be the one holding its connection.
+ It must still find the agent available, or it answers the message
+ with "not online" while the agent is answering it."""
+ a, b = processes
+ conn = await _open(a, scope="all")
+ await _subscribe(a, ROOM, conn.stream_generation)
+ await _eventually(lambda: b.registry.claimant_of(AGENT, ROOM) is not None)
+
+ for model in ("always_on", "auto_session", "session_addressable"):
+ recipient = _recipient_on(b)
+ agent = SimpleNamespace(
+ id=AGENT, integration_profile={"connection_model": model}
+ )
+ assert await recipient._is_available(object(), agent, ROOM), model
+
+ async def test_a_lapse_on_one_process_is_seen_by_the_other(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ a, b = processes
+ conn = await _open(a)
+ await _eventually(lambda: b.registry.is_live(AGENT))
+
+ conn.last_beat = time.monotonic() - HEARTBEAT_TTL_SECONDS - 1
+ assert [closed.id for closed in a.registry.sweep()] == [CONN]
+
+ await _eventually(lambda: not b.registry.is_live(AGENT))
+
+ async def test_a_process_whose_lease_lapses_stops_counting(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """A process that died wrote no closing: its lease going stale is what
+ takes its connections offline everywhere else."""
+ a, b = processes
+ await _open(a)
+ await _eventually(lambda: b.registry.is_live(AGENT))
+ a.run.cancel() # type: ignore[union-attr]
+ async with session_factory() as session:
+ await session.execute(
+ update(SwitchCoreProcess)
+ .where(SwitchCoreProcess.id == a.lease.process_id)
+ .values(
+ beat_at=SwitchCoreProcess.beat_at - LEASE_TTL - timedelta(seconds=1)
+ )
+ )
+ await session.commit()
+
+ await b.shared.reload_all()
+
+ assert not b.registry.is_live(AGENT)
+
+ async def test_the_runtime_state_sweep_sees_a_session_placed_elsewhere(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ """A "working" row owned by a session on another process must read as
+ occupied, or the sweep there clears it every pass."""
+ a, b = processes
+ conn = await _open(a, scope="all")
+ await connection_placements(
+ AGENT,
+ ConnectionPlacementsRequest(
+ connection_id=CONN,
+ placements={"session-1": ROOM},
+ generation=conn.stream_generation,
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=a.protocol,
+ )
+
+ await _eventually(lambda: ROOM in rooms_occupied(AGENT, b.registry))
+
+ assert b.registry.session_room(AGENT, "session-1") == ROOM
+
+
+class TestRoutedAnywhere:
+ """A row is decided by the process that writes it, which need not be the
+ one holding the connection of the agent the row is for."""
+
+ async def test_a_message_written_through_the_other_process_finds_the_agent_online(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ a, b = processes
+ writer = process_on(session_factory, connections=b.registry)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ helper, helper_client = await make_agent(session_factory, "helper")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, helper_client, agent=helper)
+ await _open(a, agent_id=helper.id, connection_id="conn-helper", scope="all")
+ await _eventually(lambda: b.registry.is_live(helper.id))
+
+ await writer.say(room, "@helper can you look at this", sender=alice)
+ await writer.idle()
+
+ assert len(await own_events(session_factory, helper.id)) == 1
+ # No "not online" notice from the writer, which holds no connection.
+ assert await said_by(session_factory, room, helper_client) == []
+
+ async def test_a_removal_written_through_the_other_process_releases_the_claim(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """The process that removes the agent from a room takes the room off
+ the connection claiming it, through the recorded claims, and the
+ process holding that connection is told."""
+ a, b = processes
+ writer = process_on(session_factory, connections=b.registry)
+ room = await make_room(session_factory)
+ helper, helper_client = await make_agent(session_factory, "helper")
+ await add_member(session_factory, room, helper_client, agent=helper)
+ conn = await _open(a, agent_id=helper.id, connection_id="conn-helper")
+ await connection_subscribe(
+ helper.id,
+ ConnectionSubscribeRequest(
+ connection_id="conn-helper",
+ room_id=room.id,
+ generation=conn.stream_generation,
+ ),
+ agent=SimpleNamespace( # type: ignore[arg-type]
+ id=helper.id, tenant_id=TENANT_ZERO_ID, metadata_=None
+ ),
+ protocol=a.protocol,
+ )
+ assert conn.rooms == {room.id}
+
+ await writer.provisioning.kick_user(
+ room.transport_room_id, helper_client.transport_user_id
+ )
+
+ await _eventually(lambda: room.id not in conn.rooms)
+ async with session_factory() as session:
+ claimant = await session.scalar(
+ select(AgentConnectionRoom.connection_id).where(
+ AgentConnectionRoom.agent_id == helper.id,
+ AgentConnectionRoom.room_id == room.id,
+ )
+ )
+ assert claimant is None
+
+
+class TestServedAnywhere:
+ async def test_a_tool_call_on_the_process_without_the_socket_succeeds(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """Asked before the other process has heard of the connection: the
+ row is read rather than the caller refused."""
+ a, b = processes
+ await _open(a)
+
+ with tenant_scope(TENANT_ZERO_ID):
+ bound, caller = await resolve_caller(
+ agent_id=AGENT,
+ protocol=b.protocol,
+ factory=session_factory,
+ connection_id=CONN,
+ session_id=None,
+ host_id=None,
+ epoch=None,
+ )
+
+ assert bound == CONN
+ assert caller is None
+
+ async def test_a_session_selector_resolves_its_room_on_the_other_process(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ a, b = processes
+ conn = await _open(a, scope="all")
+ await connection_placements(
+ AGENT,
+ ConnectionPlacementsRequest(
+ connection_id=CONN,
+ placements={"session-1": ROOM},
+ generation=conn.stream_generation,
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=a.protocol,
+ )
+
+ with tenant_scope(TENANT_ZERO_ID):
+ _, caller = await resolve_caller(
+ agent_id=AGENT,
+ protocol=b.protocol,
+ factory=session_factory,
+ connection_id=CONN,
+ session_id="session-1",
+ host_id="host",
+ epoch="1",
+ )
+
+ assert caller is not None
+ assert caller.room_id == ROOM
+
+ async def test_a_session_moved_through_one_process_calls_from_its_new_room(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """The move is answered before the other process has read it, and
+ the session's next call may land there: it must post where the
+ session is now, not where that process last saw it."""
+ a, b = processes
+ conn = await _open(a, scope="all")
+ await _place(a, conn.stream_generation, {"session-1": ROOM})
+ await _eventually(lambda: b.registry.session_room(AGENT, "session-1") == ROOM)
+ await b.bus.stop() # `b` hears nothing more from `a`
+
+ await _place(a, conn.stream_generation, {"session-1": OTHER_ROOM})
+ with tenant_scope(TENANT_ZERO_ID):
+ _, caller = await resolve_caller(
+ agent_id=AGENT,
+ protocol=b.protocol,
+ factory=session_factory,
+ connection_id=CONN,
+ session_id="session-1",
+ host_id="host",
+ epoch="1",
+ )
+
+ assert caller is not None
+ assert caller.room_id == OTHER_ROOM
+
+ async def test_a_claim_made_on_the_other_process_wakes_the_parked_stream(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ """The session's stream is parked until its first room is claimed;
+ the claim arriving at the other process must still reach it."""
+ a, b = processes
+ conn = await _open(a)
+ with tenant_scope(TENANT_ZERO_ID):
+ stream = event_frames(
+ conn=conn, registry=a.registry, buffer=a.buffer, approvals=None
+ )
+ first = await anext(stream)
+ assert first is not KEEPALIVE
+ assert first.event == "connection_state"
+ pending = asyncio.ensure_future(anext(stream))
+ await asyncio.sleep(0.2)
+ assert not pending.done(), "the stream should be parked with no room"
+
+ answer = await _subscribe(b, ROOM, conn.stream_generation)
+ frame = await asyncio.wait_for(pending, WAIT_SECONDS)
+ await stream.aclose()
+
+ assert answer["rooms"] == [ROOM]
+ assert frame is not KEEPALIVE
+ assert frame.event == "subscription_changed"
+ assert frame.data["rooms"] == [ROOM]
+ assert conn.rooms == {ROOM}
+
+ async def test_a_claim_on_the_other_process_releases_the_room_here(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """One connection of an agent acts in a room: a second connection
+ taking it through another process takes it off the first, which is
+ told with `room_released`."""
+ a, b = processes
+ first = await _open(a)
+ await _subscribe(a, ROOM, first.stream_generation)
+ second = await _open(b, connection_id="conn-2")
+
+ await connection_subscribe(
+ AGENT,
+ ConnectionSubscribeRequest(
+ connection_id="conn-2",
+ room_id=ROOM,
+ generation=second.stream_generation,
+ takeover=True,
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=b.protocol,
+ )
+
+ await _eventually(lambda: ROOM not in first.rooms)
+ await _eventually(lambda: ROOM in first.released_rooms)
+ async with session_factory() as session:
+ claimant = await session.scalar(
+ select(AgentConnectionRoom.connection_id).where(
+ AgentConnectionRoom.agent_id == AGENT,
+ AgentConnectionRoom.room_id == ROOM,
+ )
+ )
+ assert claimant == "conn-2"
+
+ async def test_an_old_clients_beat_on_the_other_process_reaches_the_holder(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ a, b = processes
+ conn = await _open(a)
+ conn.last_beat = time.monotonic() - 4
+ stale = conn.last_beat
+
+ with tenant_scope(TENANT_ZERO_ID):
+ answer = await connection_beat(
+ AGENT,
+ ConnectionBeatRequest(
+ connection_id=CONN, generation=conn.stream_generation
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=b.protocol,
+ )
+
+ assert answer["ok"] is True
+ await _eventually(lambda: conn.last_beat > stale)
+ assert conn.beats == 1
+
+ async def test_a_session_command_relayed_on_the_other_process_reaches_the_watcher(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ a, b = processes
+ watcher = await _open(a, scope="all")
+ await _eventually(lambda: b.registry.is_live(AGENT))
+ frame = {"command": "interrupt", "origin": {"roomId": ROOM}}
+
+ assert b.registry.relay_session_command(AGENT, frame)
+
+ await _eventually(lambda: watcher.session_commands == [frame])
+
+
+class TestTakeover:
+ async def test_a_takeover_through_the_other_process_fences_the_loser(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ a, b = processes
+ loser = await _open(a)
+ held = loser.stream_generation
+
+ winner = await _open(b, expected_generation=held)
+
+ assert winner.stream_generation != held
+ await _eventually(lambda: loser.closure is not None)
+ assert loser.closure is not None
+ assert loser.closure.code == "taken_over"
+ # The loser's late beat on the process it was on is refused as taken
+ # over, on the recorded row rather than on that process's memory.
+ with tenant_scope(TENANT_ZERO_ID), pytest.raises(HTTPException) as refused:
+ await connection_beat(
+ AGENT,
+ ConnectionBeatRequest(connection_id=CONN, generation=held),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=a.protocol,
+ )
+ assert refused.value.status_code == 409
+ assert refused.value.detail["code"] == "taken_over"
+ # And its reattach naming what it held is refused without moving
+ # anything, wherever it lands.
+ with pytest.raises(SupersededReattachError):
+ await _open(a, expected_generation=held)
+ assert b.registry.get(CONN) is winner
+ assert winner.closure is None
+
+ async def test_a_room_request_from_the_loser_is_refused_on_either_process(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ a, b = processes
+ loser = await _open(a)
+ held = loser.stream_generation
+ await _open(b, expected_generation=held)
+
+ for process in (a, b):
+ with pytest.raises(HTTPException) as refused:
+ await _subscribe(process, ROOM, held)
+ assert refused.value.status_code == 409
+ assert refused.value.detail["code"] == "taken_over"
+
+ async def test_incarnations_from_two_processes_never_collide(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ a, b = processes
+ first = await _open(a, connection_id="conn-a")
+ second = await _open(b, connection_id="conn-b")
+
+ assert first.stream_generation != second.stream_generation
+ assert min(first.stream_generation, second.stream_generation) >= 2**33
+
+ async def test_a_reattach_elsewhere_draws_a_later_incarnation(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ """Each process draws incarnations in batches, so one drawn from an
+ earlier batch is not later than the one recorded. A reattach must
+ still move the connection's incarnation forward: a process that sees
+ a later one presented knows its own copy is the stale one."""
+ a, b = processes
+ await _open(b, connection_id="conn-b") # b draws its batch first
+ held = (await _open(a)).stream_generation
+
+ winner = await _open(b, expected_generation=held)
+
+ assert winner.stream_generation > held
+
+ async def test_the_previous_holder_admits_the_current_incarnation(
+ self, processes: tuple[Process, Process]
+ ) -> None:
+ """The client reattached through the other process and its next
+ requests reach the process it left, before that one has read the
+ reattach: they are served, not refused as taken over."""
+ a, b = processes
+ first = await _open(a)
+ a.registry.detach_stream(first, first.stream_generation)
+ await a.bus.stop() # `a` hears nothing from `b`
+ second = await _open(b, expected_generation=first.stream_generation)
+
+ answer = await connection_placements(
+ AGENT,
+ ConnectionPlacementsRequest(
+ connection_id=CONN,
+ placements={"session-1": ROOM},
+ generation=second.stream_generation,
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=a.protocol,
+ )
+ with tenant_scope(TENANT_ZERO_ID):
+ beat = await connection_beat(
+ AGENT,
+ ConnectionBeatRequest(
+ connection_id=CONN, generation=second.stream_generation
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=a.protocol,
+ )
+
+ assert answer["placements"] == {"session-1": ROOM}
+ assert beat["ok"] is True
+ assert first.closure is not None
+ assert first.closure.code == "taken_over"
+ assert second.closure is None
+ # The client it took the connection from is still refused.
+ with pytest.raises(HTTPException) as refused:
+ await _subscribe(a, ROOM, first.stream_generation)
+ assert refused.value.detail["code"] == "taken_over"
+
+ async def test_another_agent_cannot_take_a_connection_id_on_the_other_process(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """The other process has no copy of the connection yet, so only the
+ row can refuse the second agent; it keeps the first agent's."""
+ a, b = processes
+ await b.bus.stop() # `b` never hears of the first agent's connection
+ held = await _open(a)
+ intruder = "agent-2"
+ await seed_agent(session_factory, intruder)
+
+ with tenant_scope(TENANT_ZERO_ID), pytest.raises(UnknownConnectionError):
+ async with b.registry.slots(intruder), b.registry.recorded(intruder):
+ b.registry.open(
+ agent_id=intruder,
+ connection_id=CONN,
+ scope="single",
+ delivery_filter="addressed",
+ spawn_capable=False,
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
+ expected_generation=None,
+ )
+
+ async with session_factory() as session:
+ row = await session.get(AgentConnectionRecord, CONN)
+ assert row is not None
+ assert (row.agent_id, row.process_id, row.incarnation) == (
+ AGENT,
+ a.lease.process_id,
+ held.stream_generation,
+ )
+ assert b.registry.get(CONN) is None
+
+
+class TestOneProcess:
+ async def test_an_open_and_a_close_are_recorded_once_each(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ a, _ = processes
+ conn = await _open(a)
+
+ async with session_factory() as session:
+ row = await session.get(AgentConnectionRecord, CONN)
+ assert row is not None
+ assert row.process_id == a.lease.process_id
+ assert row.incarnation == conn.stream_generation
+ assert row.closed_at is None
+
+ a.registry.close(
+ conn.id, Closure(code="closed", message="closed by the test", room_id=None)
+ )
+ await a.shared.flush()
+
+ async with session_factory() as session:
+ row = await session.get(AgentConnectionRecord, CONN)
+ assert row is not None
+ assert row.closed_at is not None
+ assert row.close_reason == "closed"
+
+ async def test_a_beat_writes_nothing(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ a, _ = processes
+ conn = await _open(a)
+ async with session_factory() as session:
+ before = await session.get(AgentConnectionRecord, CONN)
+ assert before is not None
+
+ for _ in range(3):
+ a.registry.beat(AGENT, CONN, conn.stream_generation)
+
+ async with session_factory() as session:
+ after = await session.get(
+ AgentConnectionRecord, CONN, populate_existing=True
+ )
+ assert after is not None
+ assert after.attached_at == before.attached_at
+
+ async def test_reopening_before_the_close_is_written_starts_without_rooms(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """A connection that lapsed and opens again on the same id is a fresh
+ open, as it was in memory alone, even while its closing is still
+ queued and its row still names the room it had."""
+ a, _ = processes
+ assert a.run is not None
+ a.run.cancel()
+ await asyncio.gather(a.run, return_exceptions=True)
+ conn = await _open(a)
+ await _subscribe(a, ROOM, conn.stream_generation)
+ conn.last_beat = time.monotonic() - HEARTBEAT_TTL_SECONDS - 1
+ a.registry.sweep()
+
+ reopened = await _open(a)
+ with tenant_scope(TENANT_ZERO_ID):
+ await a.registry.refresh(AGENT)
+
+ assert reopened is not conn
+ assert reopened.rooms == set()
+ async with session_factory() as session:
+ claimant = await session.scalar(
+ select(AgentConnectionRoom.connection_id).where(
+ AgentConnectionRoom.agent_id == AGENT
+ )
+ )
+ assert claimant is None
+
+ async def test_a_reopen_after_a_restart_is_a_fresh_open(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """A process that stopped takes its connections with it: a client
+ reattaching naming the old incarnation opens afresh, as it did when
+ connections were in memory alone."""
+ a, b = processes
+ old = await _open(a)
+ await _subscribe(a, ROOM, old.stream_generation)
+ await a.lease.stop()
+ await b.shared.reload_all()
+
+ reopened = await _open(b, expected_generation=old.stream_generation)
+
+ assert reopened.stream_generation != old.stream_generation
+ assert reopened.rooms == set()
+
+
+async def _closing_written(session_factory: SessionFactory, connection_id: str) -> None:
+ """Wait for the connection's closing to reach its row."""
+ deadline = time.monotonic() + WAIT_SECONDS
+ while True:
+ async with session_factory() as session:
+ closed_at = await session.scalar(
+ select(AgentConnectionRecord.closed_at).where(
+ AgentConnectionRecord.id == connection_id
+ )
+ )
+ if closed_at is not None:
+ return
+ if time.monotonic() > deadline:
+ raise AssertionError("the closing was never written")
+ await asyncio.sleep(0.02)
+
+
+class TestSignalsStayUnderThePayloadLimit:
+ """Postgres refuses a `NOTIFY` payload of 8000 bytes or more, and a
+ signal carries ids the client chose."""
+
+ async def test_a_beat_too_large_to_forward_is_dropped_and_the_rest_go_on(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """A connection recorded with an id too long for one signal, by a
+ process that does not bound ids, has its forwarded beat refused by
+ the bus. That beat is dropped; the next is still forwarded, and the
+ process still writes its closings."""
+ a, b = processes
+ oversized = await _open(a, connection_id="c" * 8000)
+ conn = await _open(a)
+ own = await _open(b, connection_id="conn-b")
+ await _eventually(
+ lambda: (
+ b.registry.get(oversized.id) is not None
+ and b.registry.get(CONN) is not None
+ )
+ )
+
+ b.registry.beat(AGENT, oversized.id, oversized.stream_generation)
+ b.registry.beat(AGENT, CONN, conn.stream_generation)
+
+ await _eventually(lambda: conn.beats == 1)
+ assert oversized.beats == 0
+ b.registry.close(
+ own.id, Closure(code="closed", message="closed by the test", room_id=None)
+ )
+ await _closing_written(session_factory, own.id)
+
+ async def test_a_move_releasing_many_rooms_tells_their_holder_of_each(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """A session id is bounded in characters, and JSON writes some
+ characters in twelve bytes: the rooms one move releases are split over
+ as many signals as they take."""
+ a, b = processes
+ rooms = [ROOM, OTHER_ROOM] + [
+ await seed_room(session_factory, f"room-{n}") for n in range(3, 13)
+ ]
+ holder = await _open(a, scope="all")
+ await _place(
+ a,
+ holder.stream_generation,
+ {
+ "\U0001f600" * 63 + chr(ord("a") + n): room
+ for n, room in enumerate(rooms)
+ },
+ )
+ taker = await _open(b, connection_id="conn-2", scope="all")
+
+ await connection_placements(
+ AGENT,
+ ConnectionPlacementsRequest(
+ connection_id="conn-2",
+ placements={f"session-{n}": room for n, room in enumerate(rooms)},
+ generation=taker.stream_generation,
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=b.protocol,
+ )
+
+ await _eventually(lambda: set(holder.released_rooms) == set(rooms))
+
+
+async def _occupant(session_factory: SessionFactory, room_id: str) -> tuple[str, bool]:
+ """Who the room's unread count belongs to: a session, or a connection."""
+ async with session_factory() as session:
+ row = (
+ await session.execute(
+ select(
+ AgentRoomRead.occupant_id, AgentRoomRead.occupant_is_session
+ ).where(
+ AgentRoomRead.agent_id == AGENT, AgentRoomRead.room_id == room_id
+ )
+ )
+ ).one()
+ return row[0], row[1]
+
+
+async def _unrecordable(*args: Any, **kwargs: Any) -> None:
+ raise RuntimeError("the database went away")
+
+
+class TestUnreadCountsFollowOnlyRecordedMoves:
+ """A room's unread count belongs to whoever is in the room, and moves
+ with it only once the move is recorded: a move that fails, and so is set
+ back, leaves the count with the reader still there. The failure reaches
+ the caller, which FastAPI answers with a 500."""
+
+ async def test_a_placement_that_cannot_be_recorded_moves_no_count(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ a, _ = processes
+ conn = await _open(a, scope="all")
+ await _place(a, conn.stream_generation, {"session-1": ROOM})
+ assert await _occupant(session_factory, ROOM) == ("session-1", True)
+ monkeypatch.setattr(AgentConnectionStore, "replace_slots", _unrecordable)
+
+ with pytest.raises(RuntimeError):
+ await _place(a, conn.stream_generation, {"session-2": ROOM})
+
+ assert a.registry.placements(AGENT) == {"session-1": ROOM}
+ async with session_factory() as session:
+ placed = (
+ await session.execute(
+ select(
+ AgentSessionPlacement.session_id, AgentSessionPlacement.room_id
+ ).where(AgentSessionPlacement.agent_id == AGENT)
+ )
+ ).all()
+ assert [tuple(row) for row in placed] == [("session-1", ROOM)]
+ assert await _occupant(session_factory, ROOM) == ("session-1", True)
+
+ async def test_a_claim_that_cannot_be_recorded_moves_no_count(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ a, _ = processes
+ first = await _open(a)
+ await _subscribe(a, ROOM, first.stream_generation)
+ second = await _open(a, connection_id="conn-2")
+ assert await _occupant(session_factory, ROOM) == (CONN, False)
+ monkeypatch.setattr(AgentConnectionStore, "replace_slots", _unrecordable)
+
+ with pytest.raises(RuntimeError):
+ await connection_subscribe(
+ AGENT,
+ ConnectionSubscribeRequest(
+ connection_id="conn-2",
+ room_id=ROOM,
+ generation=second.stream_generation,
+ takeover=True,
+ ),
+ agent=_Agent(), # type: ignore[arg-type]
+ protocol=a.protocol,
+ )
+
+ assert first.rooms == {ROOM}
+ assert second.rooms == set()
+ assert await _occupant(session_factory, ROOM) == (CONN, False)
+
+
+class TestStartingUp:
+ async def test_a_tenant_that_cannot_be_read_fails_the_start_not_a_reread(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ postgres_url: str,
+ monkeypatch: pytest.MonkeyPatch,
+ caplog: pytest.LogCaptureFixture,
+ ) -> None:
+ """A process that served before reading every tenant would answer for
+ the agents connected elsewhere in the one it missed as offline. Once
+ serving, a tenant that cannot be read again is logged and the others
+ are still read."""
+ a, _ = processes
+ await _open(a)
+ async with session_factory() as session:
+ session.add(Tenant(id="unreadable", slug="unreadable", name="Unreadable"))
+ await session.commit()
+ read_tenant = AgentConnectionStore.read_tenant
+
+ async def failing(
+ store: AgentConnectionStore,
+ session: AsyncSession,
+ tenant_id: str,
+ **kwargs: Any,
+ ) -> Any:
+ if tenant_id == "unreadable":
+ raise RuntimeError("the tenant's rows could not be read")
+ return await read_tenant(store, session, tenant_id, **kwargs)
+
+ monkeypatch.setattr(AgentConnectionStore, "read_tenant", failing)
+ process_id = new_process_id("test-pod")
+ registry = AgentConnectionRegistry()
+ starting = SharedConnections(
+ registry=registry,
+ session_factory=session_factory,
+ store=AgentConnectionStore(),
+ bus=SignalBus(
+ session_factory,
+ lambda: create_async_engine(postgres_url, poolclass=NullPool),
+ process_id=process_id,
+ ),
+ lease=ProcessLease(
+ process_id=process_id,
+ session_factory=session_factory,
+ processes=SwitchCoreProcessStore(),
+ key_ids=[key.id for key in TEST_KEYRING.keys],
+ ),
+ )
+
+ with pytest.raises(RuntimeError):
+ await starting.start()
+ await starting.reload_all()
+
+ assert registry.is_live(AGENT)
+ assert "unreadable" in caplog.text
+
+
+async def _delete_agent_row(session_factory: SessionFactory) -> None:
+ """Delete the agent's row, which takes its connections' rows with it."""
+ async with session_factory() as session:
+ await session.execute(delete(Agent).where(Agent.id == AGENT))
+ await session.commit()
+
+
+class TestADeletedAgent:
+ """Deleting an agent deletes its connections' rows. Its connections end
+ wherever their sockets are held, not only in the process deleting it."""
+
+ async def test_deleted_through_the_other_process_its_connection_closes(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """As `AgentCore.delete_agent` deletes it through the process that
+ does not hold its socket: the row goes, then the others are told."""
+ a, b = processes
+ conn = await _open(a)
+ await _eventually(lambda: b.registry.is_live(AGENT))
+
+ await _delete_agent_row(session_factory)
+ with tenant_scope(TENANT_ZERO_ID):
+ await b.registry.agent_deleted(AGENT)
+
+ await _eventually(lambda: conn.closure is not None)
+ assert conn.closure is not None
+ assert conn.closure.code == "closed"
+ assert not a.registry.is_live(AGENT)
+ assert not b.registry.is_live(AGENT)
+
+ async def test_a_holder_never_told_closes_it_on_its_next_pass(
+ self,
+ processes: tuple[Process, Process],
+ session_factory: SessionFactory,
+ ) -> None:
+ """Deleted by a process that told no one, or whose signal never
+ arrived: the holder finds the rows gone when it next reads them all."""
+ a, _ = processes
+ conn = await _open(a)
+
+ await _delete_agent_row(session_factory)
+ await a.shared.reload_all()
+
+ assert conn.closure is not None
+ assert conn.closure.code == "closed"
+ assert not a.registry.is_live(AGENT)
+
+ async def test_a_pass_that_read_before_an_open_was_recorded_leaves_it_open(
+ self,
+ processes: tuple[Process, Process],
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """The pass read the tenant's rows just before the connection's row
+ was written: finding no row for it then says nothing about its agent."""
+ a, b = processes
+ for process in (a, b):
+ await process.bus.stop() # no pass reads the rows but this one
+ read_tenant = AgentConnectionStore.read_tenant
+ opened: list[AgentConnection] = []
+
+ async def opened_meanwhile(
+ store: AgentConnectionStore,
+ session: AsyncSession,
+ tenant_id: str,
+ **kwargs: Any,
+ ) -> Any:
+ found = await read_tenant(store, session, tenant_id, **kwargs)
+ opened.append(await _open(a))
+ return found
+
+ monkeypatch.setattr(AgentConnectionStore, "read_tenant", opened_meanwhile)
+
+ await a.shared.reload_all()
+
+ (conn,) = opened
+ assert conn.closure is None
+ assert a.registry.get(CONN) is conn
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_controller_presence.py b/core/tests/switch_core/bridges/agent/protocol/test_controller_presence.py
index 467df576d..7d6ce59f5 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_controller_presence.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_controller_presence.py
@@ -36,12 +36,6 @@ from switch_core.bridges.agent.protocol.presence import (
)
from switch_core.bridges.agent.protocol.statuses import compute_agent_statuses
from switch_core.bridges.agent.protocol.types import AgentStatus
-from switch_core.clients.agent_consumer import (
- _STARTING_SESSION_MESSAGE,
- AgentConsumer,
- CloudMachineState,
- _GateOutcome,
-)
from switch_core.db.models import (
TENANT_ZERO_ID,
Agent,
@@ -55,6 +49,12 @@ from switch_core.db.models import (
from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.room_role_store import RoomRoleStore
from switch_core.db.stores.room_store import RoomStore
+from switch_core.delivery.recipient import (
+ _STARTING_SESSION_MESSAGE,
+ AgentRecipient,
+ CloudMachineState,
+ _GateOutcome,
+)
from switch_core.transport import InboundMessage, RoomRef
CONTROLLER = "controller-1"
@@ -373,12 +373,12 @@ def _client(
_wake_cloud_machine=_wake_cloud_machine,
_cloud_waking_notices=set(),
)
- ns._is_available = AgentConsumer._is_available.__get__(ns)
+ ns._is_available = AgentRecipient._is_available.__get__(ns)
ns._reply_when_unavailable_here = (
- AgentConsumer._reply_when_unavailable_here.__get__(ns)
+ AgentRecipient._reply_when_unavailable_here.__get__(ns)
)
ns._controller_unavailable_reply = (
- AgentConsumer._controller_unavailable_reply.__get__(ns)
+ AgentRecipient._controller_unavailable_reply.__get__(ns)
)
return ns
@@ -578,17 +578,14 @@ class _Sent:
def _addressing_client(
registry: AgentConnectionRegistry, agent_id: str, connection_model: str
) -> tuple[SimpleNamespace, _Sent, list[str]]:
- """An agent client for `on_message`, addressed in ROOM, its availability
- and replies the real ones over `registry`."""
+ """An agent recipient deciding a message addressed to it in ROOM, its
+ availability and replies the real ones over `registry`."""
meta = SimpleNamespace(
room_id=ROOM, name="Room", bridge_id=None, channel_type="channel_private"
)
sent = _Sent()
enqueued: list[str] = []
- async def _resolve_room_meta(_transport_room_id: str) -> SimpleNamespace:
- return meta
-
async def _addressed(*_args: Any) -> bool:
return True
@@ -605,34 +602,34 @@ def _addressing_client(
integration_profile={"connection_model": connection_model},
)
ns.session_factory = _no_session
- ns._resolve_room_meta = _resolve_room_meta
+ ns._meta = meta
+ ns._room = RoomRef(room_id="!room:test")
ns._addressed_without_lookup = lambda _event, _meta: True
ns._compute_addressed = _addressed
ns._fresh_agent = _fresh_agent
ns._gate_addressed = _gate_addressed
ns._report_addressed = lambda *_a, **_k: None
- ns._triggered_by_auto_reply = AgentConsumer._triggered_by_auto_reply
+ ns._triggered_by_auto_reply = AgentRecipient._triggered_by_auto_reply
ns.actor = SimpleNamespace(send_message=sent)
- async def _enqueue(
- _tenant_id: str,
- _agent_id: str,
- _room_id: str,
- event: Any,
- *,
- room_seq: int,
- message_key: str | None,
- ) -> int:
- enqueued.append(event.payload.message_id)
- return len(enqueued)
-
ns.tenant_id = TENANT_ZERO_ID
- ns._event_buffer = SimpleNamespace(enqueue=_enqueue)
- ns._sender_handle = AgentConsumer._sender_handle.__get__(ns)
- ns._post_auto_reply = AgentConsumer._post_auto_reply.__get__(ns)
+ ns.enqueued = enqueued
+ ns._sender_handle = AgentRecipient._sender_handle.__get__(ns)
+ ns._post_auto_reply = AgentRecipient._post_auto_reply.__get__(ns)
+ ns._after_message = AgentRecipient._after_message.__get__(ns)
return ns, sent, enqueued
+async def _deliver(client: SimpleNamespace, event: InboundMessage) -> None:
+ """Decide the message for the agent as the router does: what is stored
+ for it is recorded, and what it is left to say is said."""
+ delivery = await AgentRecipient.decide_message(client, None, event) # type: ignore[arg-type]
+ if delivery.own_event is not None:
+ client.enqueued.append(delivery.own_event(event).event.payload.message_id)
+ if delivery.after is not None:
+ await delivery.after(event)
+
+
def _addressing(message_id: str) -> InboundMessage:
return InboundMessage(
room_id="!room:test",
@@ -658,11 +655,7 @@ class TestAddressingAControllerBackedAgent:
_go_live(registry, ("agent", {ROOM}))
client, sent, enqueued = _addressing_client(registry, "agent", model)
- await AgentConsumer.on_message(
- client, # type: ignore[arg-type]
- RoomRef(room_id="!room:test"),
- _addressing("$asked"),
- )
+ await _deliver(client, _addressing("$asked"))
assert enqueued == ["$asked"], model
assert sent.bodies == [], model
@@ -684,11 +677,7 @@ class TestAddressingAControllerBackedAgent:
registry, "agent", "auto_session"
)
- await AgentConsumer.on_message(
- client, # type: ignore[arg-type]
- RoomRef(room_id="!room:test"),
- _addressing("$asked"),
- )
+ await _deliver(client, _addressing("$asked"))
assert len(sent.bodies) == 1, case
assert expected in sent.bodies[0], (case, sent.bodies[0])
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_conversation_reading.py b/core/tests/switch_core/bridges/agent/protocol/test_conversation_reading.py
index 5f201fe58..7072b6ccd 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_conversation_reading.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_conversation_reading.py
@@ -4,8 +4,10 @@ What a reader that wants everything (`filter=all`, the legacy polls) is sent
besides the agent's own events. Three properties are pinned here, each found
running the stream end to end:
-- a row is read once the agent's consumer has decided it is not the agent's
- own, and no sooner, so it is never sent twice;
+- a row is read as soon as it is written, because whether it is the agent's
+ own was decided in the transaction that wrote it, so it is never sent twice;
+ the one row read later is a file of a post the router is still waiting on,
+ which holds the read back until the post is decided;
- the agent's own event for a message stands for that message up to the row it
was delivered through, so a file that arrives after its group was delivered
is still read;
@@ -18,17 +20,23 @@ from __future__ import annotations
from datetime import UTC, datetime, timedelta
import pytest
-from sqlalchemy import delete, func, select, update
+from sqlalchemy import delete
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.attachments import ATTACHMENT_GROUP_KEY
from switch_core.bridges.agent.protocol.event_buffer import BufferedEvent, EventBuffer
from switch_core.bridges.agent.protocol.positions import OWN_EVENTS, room_source
from switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload
-from switch_core.db.models import TENANT_ZERO_ID, Agent, Message, MessageAttachment
+from switch_core.db.models import (
+ TENANT_ZERO_ID,
+ Agent,
+ Message,
+ MessageAttachment,
+ PendingAttachmentGroup,
+)
from switch_core.db.session_scope import tenant_session
from switch_core.db.stores.message_store import MessageStore
-from tests.db_buffer import db_buffer_on, say, seed_member, seed_room
+from tests.db_buffer import say, seed_member, seed_room
AGENT = "agent-1"
ROOM = "room-1"
@@ -43,12 +51,6 @@ async def _rows(session_factory: SessionFactory) -> None:
await seed_room(session_factory, EMPTY_ROOM)
-def _unsettling(session_factory: SessionFactory) -> EventBuffer:
- """A buffer that takes a minute to settle a row on its own, so a test row
- is read only when the agent's own events say it was decided."""
- return db_buffer_on(session_factory, settle_seconds=60.0)
-
-
async def _keep(buffer: EventBuffer, room_seq: int, message_key: str) -> None:
"""The agent's own event for a message addressed to it, as its consumer
keeps it."""
@@ -117,42 +119,67 @@ def _bodies(items: list[BufferedEvent]) -> list[str | None]:
return [getattr(item.event.payload, "body", None) for item in items]
-async def test_what_came_before_the_agents_own_event_is_read_at_once(
- session_factory: SessionFactory,
+async def test_what_came_before_and_after_the_agents_own_event_is_read_at_once(
+ db_buffer: EventBuffer, session_factory: SessionFactory
) -> None:
- """The consumer takes a room's rows one at a time, in order, so once it has
- kept a row as the agent's own, every row before it is decided. Waiting for
- them to settle would send the agent its mention before the conversation
- that led up to it."""
- buffer = _unsettling(session_factory)
+ """A row and the own events decided for it commit together, so a row that
+ can be read is decided: the conversation around a mention is read at once,
+ and the mention, already the agent's own, is not read again. Waiting for
+ rows to settle sent the agent its mention ahead of the conversation that
+ led up to it."""
+ buffer = db_buffer
await say(session_factory, ROOM, "before")
asked = await say(session_factory, ROOM, "@agent over to you")
await _keep(buffer, asked.seq, asked.transport_event_id)
- await say(session_factory, ROOM, "after")
+ after = await say(session_factory, ROOM, "after")
read = await buffer.read_conversation(TENANT_ZERO_ID, AGENT, {ROOM: 0}, limit=100)
- assert _bodies(read.events) == ["before"]
- # Looked through the agent's own row, and no further: "after" is not
- # decided yet.
- assert read.through == {ROOM: asked.seq}
+ assert _bodies(read.events) == ["before", "after"]
+ assert read.through == {ROOM: after.seq}
-async def test_a_row_not_yet_decided_holds_back_the_rows_after_it(
- session_factory: SessionFactory,
+async def test_a_post_not_yet_decided_holds_back_the_rows_after_it(
+ db_buffer: EventBuffer, session_factory: SessionFactory
) -> None:
- """Rows are read as an unbroken run from the reader's place. A settled row
- after an unsettled one waits for it, or the reader's place would move past
- the unsettled row and it would never be read."""
- buffer = db_buffer_on(session_factory, settle_seconds=30.0)
- now = datetime.now(UTC)
- await say(session_factory, ROOM, "settled", sent_at=now - timedelta(minutes=5))
- await say(session_factory, ROOM, "fresh", sent_at=now)
- await say(session_factory, ROOM, "stamped late", sent_at=now - timedelta(minutes=5))
+ """Rows are read as an unbroken run from the reader's place. A post whose
+ files are still arriving is decided as a whole, so the read stops at its
+ first file, and the rows after it wait with it: had the reader's place
+ moved past the file, the post would reach the agent in pieces."""
+ buffer = db_buffer
+ long_ago = datetime.now(UTC) - timedelta(minutes=5)
+ await say(session_factory, ROOM, "decided", sent_at=long_ago)
+ first = await say(
+ session_factory,
+ ROOM,
+ "first of two",
+ content={ATTACHMENT_GROUP_KEY: {"id": "grp", "index": 0, "total": 2}},
+ sent_at=long_ago,
+ )
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ session.add(
+ PendingAttachmentGroup(
+ room_id=ROOM,
+ group_id="grp",
+ total=2,
+ from_seq=first.seq,
+ deadline=datetime.now(UTC) + timedelta(minutes=1),
+ )
+ )
+ await session.commit()
+ await say(session_factory, ROOM, "after it", sent_at=long_ago)
+
+ read = await buffer.read_conversation(TENANT_ZERO_ID, AGENT, {ROOM: 0}, limit=100)
+ assert _bodies(read.events) == ["decided"]
+
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ await session.execute(delete(PendingAttachmentGroup))
+ await session.commit()
read = await buffer.read_conversation(TENANT_ZERO_ID, AGENT, {ROOM: 0}, limit=100)
- assert _bodies(read.events) == ["settled"]
+ # Decided: the read goes on from the post's first file.
+ assert _bodies(read.events) == ["decided", "first of two", "after it"]
async def test_a_backlog_longer_than_one_read_keeps_its_order(
@@ -240,17 +267,23 @@ async def test_a_group_held_back_by_another_keeps_its_first_file(
hold, so the read must not move past A's first file either, or the next
read would send A with only its second file."""
buffer = db_buffer
- async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
- now = await session.scalar(select(func.now()))
- assert now is not None
- # Past the settle time (0.2 s for `db_buffer`) and within the group grace
- # time (5 s), so B, missing its second file, is held.
- recent = now - timedelta(seconds=3)
+ long_ago = datetime.now(UTC) - timedelta(minutes=5)
a_first = await _file(
- session_factory, "A", 0, "a-one.pdf", recent, caption="see these"
+ session_factory, "A", 0, "a-one.pdf", long_ago, caption="see these"
)
- await _file(session_factory, "B", 0, "b-one.pdf", recent)
- await _file(session_factory, "A", 1, "a-two.pdf", recent)
+ b_first = await _file(session_factory, "B", 0, "b-one.pdf", long_ago)
+ await _file(session_factory, "A", 1, "a-two.pdf", long_ago)
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ session.add(
+ PendingAttachmentGroup(
+ room_id=ROOM,
+ group_id="B",
+ total=2,
+ from_seq=b_first.seq,
+ deadline=datetime.now(UTC) + timedelta(minutes=1),
+ )
+ )
+ await session.commit()
first = await buffer.read_conversation(TENANT_ZERO_ID, AGENT, {ROOM: 0}, limit=100)
@@ -258,13 +291,9 @@ async def test_a_group_held_back_by_another_keeps_its_first_file(
place = first.through.get(ROOM, 0)
assert place < a_first.seq
- # B's grace time runs out without its second file.
+ # B is decided without its second file.
async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
- await session.execute(
- update(Message)
- .where(Message.room_id == ROOM)
- .values(sent_at=Message.sent_at - timedelta(minutes=1))
- )
+ await session.execute(delete(PendingAttachmentGroup))
await session.commit()
second = await buffer.read_conversation(
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_delete_agent.py b/core/tests/switch_core/bridges/agent/protocol/test_delete_agent.py
index 8902214d9..4a2226b65 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_delete_agent.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_delete_agent.py
@@ -6,16 +6,21 @@ which `ClientStore.delete` clears — reached through
`ClientLifecycleService.delete_record` in the same transaction as the agent's
delete. Nothing else clears the second, so this pins the whole chain against a
real database.
+
+A delete that fails leaves the agent in place, and its client runs again.
"""
from __future__ import annotations
import uuid
+import pytest
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+from switch_core.bridges.agent.protocol.agent_connections import ClientDeclaration
from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.db.models import Room, require_tenant_id
+from switch_core.bridges.agent.protocol.positions import OWN_EVENTS
+from switch_core.db.models import TENANT_ZERO_ID, Room, require_tenant_id
from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.room_store import RoomStore
from tests.switch_core.bridges.agent.protocol.registration_harness import (
@@ -24,6 +29,7 @@ from tests.switch_core.bridges.agent.protocol.registration_harness import (
make_service,
register,
)
+from tests.switch_core.clients.test_client_signals import _lifecycle
class _DeletingClientLifecycle(FakeClientLifecycle):
@@ -91,3 +97,65 @@ class TestDeleteAgentInARoom:
assert await RoomStore().get_client_ids(session, room_id) == []
# Only the memberships go; the room outlives its member.
assert await RoomStore().get(session, room_id) is not None
+
+
+class TestDeleteAgentWithAConnection:
+ async def test_its_connection_is_closed(
+ self, session_factory: async_sessionmaker[AsyncSession], db_buffer: EventBuffer
+ ) -> None:
+ """Rather than left serving a deleted agent's stream, and reporting
+ the agent live, until its client goes."""
+ svc = make_service(session_factory)
+ svc.client_lifecycle = _DeletingClientLifecycle(session_factory) # type: ignore[assignment]
+ svc.event_buffer = db_buffer
+ owner = await make_owner(session_factory)
+ agent_id = await register(svc, "connected", owner)
+ conn = svc.connections.open(
+ agent_id=agent_id,
+ connection_id="its-connection",
+ scope="all",
+ delivery_filter="addressed",
+ spawn_capable=False,
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(),
+ expected_generation=None,
+ )
+
+ await svc.delete_agent(agent_id=agent_id)
+
+ assert conn.closure is not None
+ assert conn.closure.code == "closed"
+ assert not svc.connections.is_live(agent_id)
+
+
+class TestDeleteAgentThatFails:
+ async def test_its_client_is_started_again_by_the_next_recheck(
+ self, session_factory: async_sessionmaker[AsyncSession], db_buffer: EventBuffer
+ ) -> None:
+ """`delete_agent` stops the agent's client before its rows go, and a
+ re-check leaves a client stopped that way alone. A delete that then
+ fails leaves the agent and its client's row in place, so the next
+ re-check starts the client again, rather than leaving it stopped here
+ until the process restarts."""
+ svc = make_service(session_factory)
+ svc.event_buffer = db_buffer
+ owner = await make_owner(session_factory)
+ agent_id = await register(svc, "stays", owner)
+ async with session_factory() as session:
+ agent = await svc.agent_store.get(session, agent_id)
+ assert agent is not None
+ client_id = agent.client_id
+ lifecycle = _lifecycle(session_factory)
+ await lifecycle.start_all()
+ svc.client_lifecycle = lifecycle
+
+ async def _fail(*_args: object) -> None:
+ raise OSError("connection lost")
+
+ svc.agent_store.delete = _fail # type: ignore[method-assign]
+ with pytest.raises(OSError):
+ await svc.delete_agent(agent_id=agent_id)
+ await lifecycle.reconcile_tenant(TENANT_ZERO_ID)
+
+ assert lifecycle.get(client_id) is not None
+ await lifecycle.stop_all()
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_event_buffer.py b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer.py
index 63c147d73..f7719895e 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_event_buffer.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_event_buffer.py
@@ -41,7 +41,7 @@ from switch_core.db.models import (
AgentPollPosition,
)
from switch_core.db.session_scope import tenant_session
-from tests.db_buffer import db_buffer_on, say, seed_agent, seed_member
+from tests.db_buffer import db_buffer_on, forget_room, say, seed_agent, seed_member
AGENT = "agent-1"
ROOM = "room-1"
@@ -282,7 +282,7 @@ async def test_dropping_a_room_forgets_what_it_still_held(
await _asked(q, session_factory, ROOM)
await _asked(q, session_factory, "room-2")
- await q.drop_room(TENANT_ZERO_ID, AGENT, ROOM)
+ await forget_room(session_factory, q, AGENT, ROOM)
# Gone from the low-level read every reader is built on, filter or no
# filter — which is what closes it for the stream, the one reader with no
@@ -304,7 +304,7 @@ async def test_dropping_a_room_does_not_report_a_gap(
await _asked(q, session_factory, ROOM)
await _asked(q, session_factory, "room-2")
- await q.drop_room(TENANT_ZERO_ID, AGENT, ROOM)
+ await forget_room(session_factory, q, AGENT, ROOM)
assert await q.rooms_dropped_after(TENANT_ZERO_ID, AGENT, 0) == ()
assert [item.seq for item in (await q.read_from(TENANT_ZERO_ID, AGENT, 0))] == [2]
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_register_overwrite_owner_guard.py b/core/tests/switch_core/bridges/agent/protocol/test_register_overwrite_owner_guard.py
index 6be41a782..3a5362869 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_register_overwrite_owner_guard.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_register_overwrite_owner_guard.py
@@ -57,6 +57,9 @@ class _FakeClientLifecycle:
def start_client(self, client: Client) -> None:
pass
+ def announce(self, tenant_id: str) -> None:
+ return None
+
class _NoBridges:
def bridges_for_tenant(self, tenant_id: str) -> list[object]:
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_room_roster_authz.py b/core/tests/switch_core/bridges/agent/protocol/test_room_roster_authz.py
index dc85fa1a6..e26fdb388 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_room_roster_authz.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_room_roster_authz.py
@@ -55,6 +55,9 @@ class _FakeClientLifecycle:
def start_client(self, client: Client) -> None:
pass
+ def announce(self, tenant_id: str) -> None:
+ return None
+
class _NoBridges:
def bridges_for_tenant(self, tenant_id: str) -> list[object]:
diff --git a/core/tests/switch_core/bridges/agent/protocol/test_stream.py b/core/tests/switch_core/bridges/agent/protocol/test_stream.py
index 224719aaf..8f68dfbf9 100644
--- a/core/tests/switch_core/bridges/agent/protocol/test_stream.py
+++ b/core/tests/switch_core/bridges/agent/protocol/test_stream.py
@@ -51,8 +51,16 @@ from switch_core.bridges.agent.protocol.types import (
from switch_core.db.models import TENANT_ZERO_ID, Agent, Message
from switch_core.db.session_scope import tenant_session
from switch_core.db.stores.room_store import RoomStore
+from switch_core.messages.notify import MessageListener
from switch_core.observability.metrics import MetricsRegistry, install, uninstall
-from tests.db_buffer import age_events, db_buffer_on, say, seed_member, sender_of
+from tests.db_buffer import (
+ age_events,
+ db_buffer_on,
+ forget_room,
+ say,
+ seed_member,
+ sender_of,
+)
AGENT = "agent-1"
ROOM_A = "room-a"
@@ -96,6 +104,16 @@ def _command(name: str, *, room: str = ROOM_A) -> AgentEvent:
)
+def db_buffer(
+ session_factory: async_sessionmaker[AsyncSession], room_bells: MessageListener
+) -> EventBuffer:
+ """The db-buffer, with its rooms' announcements listened to: a stream that
+ reads a room's conversation is woken by a row written there, not by a
+ timer, so a test that writes one while a stream waits needs them."""
+ return db_buffer_on(session_factory, rooms=room_bells)
+
+
async def _keep(buffer: EventBuffer, event: AgentEvent, *, agent: str = AGENT) -> int:
"""Keep `event` as one of the agent's own events and return its number.
@@ -379,15 +397,13 @@ async def test_the_conversation_is_read_from_where_the_room_was_when_covered(
async def test_the_chatter_before_a_mention_arrives_with_it_and_ahead_of_it(
- session_factory: async_sessionmaker[AsyncSession],
+ db_buffer: EventBuffer, session_factory: async_sessionmaker[AsyncSession]
) -> None:
- """The agent's consumer takes a room's rows in order, so every row up to
- one it kept for the agent has been decided: those are read straight away
- rather than once settled, and what led up to a mention arrives ahead of
- it."""
+ """Every row is decided as it is written, so the rows around a mention are
+ read straight away rather than once settled, and what led up to a mention
+ arrives ahead of it."""
registry = AgentConnectionRegistry()
- # Settles long after this test is over.
- buffer = db_buffer_on(session_factory, settle_seconds=60)
+ buffer = db_buffer
conn = _open(registry, position={OWN_EVENTS: 0, room_source(ROOM_A): 0})
registry.claim_room(conn, ROOM_A)
await say(session_factory, ROOM_A, "chatter")
@@ -1189,10 +1205,10 @@ async def test_a_resumed_stream_does_not_replay_a_room_the_agent_was_removed_fro
await _keep(buffer, _message("said before the removal"))
await _keep(buffer, _message("still a member here", room=ROOM_B))
- # What a kick does: the membership goes, and `AgentConsumer.on_removed`
- # makes these two calls.
+ # What a kick does: the membership goes, its own events in the room go
+ # with it (`MessageRouter.remove`), and the room is let go of here.
await _leave(session_factory, ROOM_A)
- await buffer.drop_room(TENANT_ZERO_ID, AGENT, ROOM_A)
+ await forget_room(session_factory, buffer, AGENT, ROOM_A)
registry.release_room_everywhere(AGENT, ROOM_A)
# The supervisor comes back and resumes from before all of it.
@@ -1210,7 +1226,7 @@ async def test_a_resumed_stream_does_not_replay_a_room_the_agent_was_removed_fro
async def test_a_last_event_id_reconnect_does_not_replay_a_removed_room(
- db_buffer: EventBuffer,
+ db_buffer: EventBuffer, session_factory: async_sessionmaker[AsyncSession]
) -> None:
"""The resume a real client actually performs.
@@ -1228,7 +1244,7 @@ async def test_a_last_event_id_reconnect_does_not_replay_a_removed_room(
await _keep(buffer, _message("while it was still a member"))
await _keep(buffer, _message("in the room it kept", room=ROOM_B))
- await buffer.drop_room(TENANT_ZERO_ID, AGENT, ROOM_A)
+ await forget_room(session_factory, buffer, AGENT, ROOM_A)
registry.release_room_everywhere(AGENT, ROOM_A)
# `seen` is what the client would send back as Last-Event-ID: a cursor from
diff --git a/core/tests/switch_core/bridges/agent/server_connectors/test_connector_ownership.py b/core/tests/switch_core/bridges/agent/server_connectors/test_connector_ownership.py
new file mode 100644
index 000000000..6f5a80b51
--- /dev/null
+++ b/core/tests/switch_core/bridges/agent/server_connectors/test_connector_ownership.py
@@ -0,0 +1,375 @@
+"""A server-side connector runs in the one process that holds it.
+
+Against real PostgreSQL, two "processes" on one database, each with its own
+signal bus, process lease, ownership and connector lifecycle, wired as `main`
+wires them. The connector's server here (`_Server`) records which copies of
+the connector are connected to it, so a test can say the connector, and so
+each of its agents' poll loops, runs in one process at a time.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+from collections.abc import AsyncIterator, Callable
+from dataclasses import dataclass
+from types import SimpleNamespace
+from typing import Any, ClassVar
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.bridges.agent.protocol.types import (
+ CommandPayload,
+ IntegrationProfile,
+ MessagePayload,
+ TaskProtocolConfig,
+)
+from switch_core.bridges.agent.server_connectors.base import (
+ ConnectorReporter,
+ DiscoveredAgent,
+ ServerSideConnector,
+ ServerSideConnectorConfig,
+)
+from switch_core.bridges.agent.server_connectors.lifecycle import (
+ ServerSideConnectorLifecycleService,
+)
+from switch_core.bridges.agent.server_connectors.signals import ConnectorSignals
+from switch_core.db.models import ApiKey, ServerConnector, User
+from switch_core.db.stores.api_key_store import ApiKeyStore
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+from switch_core.db.stores.server_connector_store import ServerConnectorStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.keys import Keyring
+from switch_core.outbound import OutboundPolicy
+from switch_core.ownership import Ownership
+from switch_core.process_lease import ProcessLease
+from switch_core.signals import SignalBus
+from tests.ownership import process_lease_on
+from tests.switch_core.test_room_wide_mention_wakes_no_agent import _make_agent
+
+TIMEOUT = 10.0
+KIND = "fake-server"
+KEYRING = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
+
+
+class _Server:
+ """The agent server every copy of the connector talks to."""
+
+ def __init__(self) -> None:
+ self.connected: list[_Connector] = []
+
+
+class _Config(ServerSideConnectorConfig):
+ pass
+
+
+class _Connector(ServerSideConnector):
+ server: ClassVar[_Server]
+
+ def __init__(self, *, config: _Config, outbound_policy: OutboundPolicy) -> None:
+ self.config = config
+
+ async def start(self) -> None:
+ self.server.connected.append(self)
+
+ async def stop(self) -> None:
+ self.server.connected.remove(self)
+
+ async def discover_agents(self) -> list[DiscoveredAgent]:
+ return [
+ DiscoveredAgent(
+ name="helper",
+ description="",
+ integration_profile=IntegrationProfile(
+ connection_model="always_on",
+ message_exchange=True,
+ pre_invocation_mediation=[],
+ post_invocation_mediation=[],
+ event_reporting=[],
+ task_protocol=TaskProtocolConfig(
+ can_delegate=False, can_accept=False
+ ),
+ ),
+ )
+ ]
+
+ async def handle_message(
+ self,
+ agent_name: str,
+ room_id: str,
+ message: MessagePayload,
+ reporter: ConnectorReporter,
+ ) -> str | None:
+ return None
+
+ async def handle_command(
+ self, agent_name: str, room_id: str, command: CommandPayload
+ ) -> None:
+ return None
+
+
+class _Protocol:
+ """`AgentCore` as far as a connector reaches it: its agents register,
+ then long-poll for events that never come."""
+
+ def __init__(self) -> None:
+ self.deleted: list[str] = []
+
+ async def register_agent_with_token(self, **_: Any) -> Any:
+ return SimpleNamespace(agent_id=f"agent-{uuid.uuid4().hex[:8]}")
+
+ async def poll_events(self, agent_id: str, timeout: int) -> list[Any]:
+ await asyncio.sleep(3600)
+ return []
+
+ async def delete_agent(self, *, agent_id: str) -> None:
+ self.deleted.append(agent_id)
+
+
+@dataclass
+class _Process:
+ lease: ProcessLease
+ ownership: Ownership
+ bus: SignalBus
+ connectors: ServerSideConnectorLifecycleService
+ protocol: _Protocol
+ lease_task: asyncio.Task[None]
+ stopped: bool = False
+
+ async def stop(self) -> None:
+ """Shut down as `main` does: owned work first, then the rest."""
+ if self.stopped:
+ return
+ self.stopped = True
+ await self.ownership.stop()
+ await self.lease.stop()
+ await self.connectors.stop_all()
+ self.lease_task.cancel()
+ await asyncio.gather(self.lease_task, return_exceptions=True)
+ await self.bus.stop()
+
+
+async def _process(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession]
+) -> _Process:
+ lease = process_lease_on(session_factory)
+ await lease.renew()
+ bus = SignalBus(
+ session_factory,
+ lambda: create_async_engine(postgres_url, poolclass=NullPool),
+ process_id=lease.process_id,
+ )
+ ownership = Ownership(
+ lease=lease,
+ session_factory=session_factory,
+ bus=bus,
+ store=OwnershipLeaseStore(),
+ processes=SwitchCoreProcessStore(),
+ )
+ protocol = _Protocol()
+ connectors = ServerSideConnectorLifecycleService(
+ connector_store=ServerConnectorStore(),
+ api_key_store=ApiKeyStore(),
+ protocol=protocol, # type: ignore[arg-type]
+ session_factory=session_factory,
+ keyring=KEYRING,
+ outbound_policy=OutboundPolicy.parse(""),
+ ownership=ownership,
+ )
+ connectors.register_connector_type(KIND, _Connector, _Config)
+ ConnectorSignals(lifecycle=connectors, bus=bus)
+ await bus.start()
+ await asyncio.wait_for(bus.connected.wait(), TIMEOUT)
+ return _Process(
+ lease=lease,
+ ownership=ownership,
+ bus=bus,
+ connectors=connectors,
+ protocol=protocol,
+ lease_task=asyncio.create_task(lease.run()),
+ )
+
+
+def server() -> _Server:
+ server = _Server()
+ _Connector.server = server
+ return server
+
+
+@dataclass
+class _Stored:
+ connector_id: str
+ user_id: str
+
+
+async def stored(session_factory: async_sessionmaker[AsyncSession]) -> _Stored:
+ """A connector registered before the processes start, with its
+ registration key, and one agent it registered before."""
+ async with session_factory() as session:
+ user = User(name="owner", email="owner@test", role="user", password_hash="x")
+ session.add(user)
+ await session.flush()
+ key = ApiKey(
+ user_id=user.id,
+ key_hash=f"hash-{uuid.uuid4().hex}",
+ encrypted_key=KEYRING.encrypt("registration-token"),
+ label="server-connector:test",
+ type="registration",
+ )
+ session.add(key)
+ await session.flush()
+ connector = ServerConnector(
+ type=KIND,
+ display_name="Fake",
+ connection_config={},
+ api_key_id=key.id,
+ status="active",
+ )
+ session.add(connector)
+ await session.flush()
+ agent = await _make_agent(session, f"helper-{uuid.uuid4().hex[:6]}")
+ agent.metadata_ = {"server_connector_id": connector.id}
+ await session.commit()
+ return _Stored(connector_id=connector.id, user_id=user.id)
+
+
+async def two(
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ server: _Server,
+ stored: _Stored,
+) -> AsyncIterator[tuple[_Process, _Process]]:
+ """Two processes booted once the connector is stored, as after a deploy."""
+ processes: list[_Process] = []
+ try:
+ for _ in range(2):
+ process = await _process(postgres_url, session_factory)
+ processes.append(process)
+ # As `main` boots: the first balance, then the connectors wanted.
+ await process.ownership.start()
+ await process.connectors.start_all()
+ # The first balance of each cannot see the other; balance once more
+ # now that both leases are there, as the interval would.
+ for process in processes:
+ await process.ownership.balance()
+ yield processes[0], processes[1]
+ finally:
+ for process in processes:
+ await process.stop()
+
+
+def _holder(
+ two: tuple[_Process, _Process], connector_id: str
+) -> tuple[_Process, _Process]:
+ a, b = two
+ holders = [p for p in (a, b) if p.connectors.holds(connector_id)]
+ assert len(holders) == 1
+ return (a, b) if holders[0] is a else (b, a)
+
+
+async def _eventually(check: Callable[[], bool]) -> None:
+ async with asyncio.timeout(TIMEOUT):
+ while not check():
+ await asyncio.sleep(0.02)
+
+
+class TestOneProcessRunsIt:
+ async def test_the_connector_runs_in_the_process_that_holds_it(
+ self, two: tuple[_Process, _Process], server: _Server, stored: _Stored
+ ) -> None:
+ holder, other = _holder(two, stored.connector_id)
+
+ # A connector taken over starts in a task of its own.
+ await _eventually(lambda: holder.connectors.running_count() == 1)
+ assert len(server.connected) == 1
+ assert other.connectors.running_count() == 0
+ assert other.connectors.expected_count() == 0
+
+ async def test_every_process_names_its_agents(
+ self, two: tuple[_Process, _Process], stored: _Stored
+ ) -> None:
+ for process in two:
+ names = await process.connectors.get_agent_names(stored.connector_id)
+ assert len(names) == 1 and names[0].startswith("helper-")
+
+ async def test_when_its_process_stops_the_other_runs_it(
+ self, two: tuple[_Process, _Process], server: _Server, stored: _Stored
+ ) -> None:
+ holder, other = _holder(two, stored.connector_id)
+
+ await _eventually(lambda: holder.connectors.running_count() == 1)
+ await holder.stop()
+ await other.ownership.balance()
+
+ assert other.connectors.holds(stored.connector_id)
+ await _eventually(lambda: other.connectors.running_count() == 1)
+ assert len(server.connected) == 1
+
+
+class TestChangesMadeElsewhere:
+ async def test_one_registered_through_either_process_runs_once(
+ self, two: tuple[_Process, _Process], server: _Server, stored: _Stored
+ ) -> None:
+ a, _ = two
+
+ added = await a.connectors.register(
+ connector_type=KIND,
+ display_name="Second",
+ connection_config={},
+ user_id=stored.user_id,
+ )
+
+ await _eventually(lambda: len(server.connected) == 2)
+ holders = [p for p in two if p.connectors.holds(added.id)]
+ assert len(holders) == 1
+ await asyncio.sleep(0.3)
+ assert len(server.connected) == 2
+
+ async def test_one_removed_through_the_other_process_stops_and_takes_its_agents(
+ self, two: tuple[_Process, _Process], server: _Server, stored: _Stored
+ ) -> None:
+ holder, other = _holder(two, stored.connector_id)
+ await _eventually(lambda: len(server.connected) == 1)
+
+ await other.connectors.remove(stored.connector_id)
+
+ await _eventually(lambda: server.connected == [])
+ assert not holder.connectors.holds(stored.connector_id)
+ assert len(other.protocol.deleted) == 1
+
+
+class TestASlowServer:
+ async def test_it_holds_up_no_other_work_while_its_connector_starts(
+ self,
+ two: tuple[_Process, _Process],
+ server: _Server,
+ stored: _Stored,
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """A connector's start dials its server. One that hangs is taken over
+ and left starting in its own task: the next balance, and the
+ hand-over of anything else, go on meanwhile."""
+ holder, other = _holder(two, stored.connector_id)
+ await _eventually(lambda: len(server.connected) == 1)
+ release = asyncio.Event()
+
+ async def hanging_start(self: _Connector) -> None:
+ await release.wait()
+ self.server.connected.append(self)
+
+ monkeypatch.setattr(_Connector, "start", hanging_start)
+ await holder.stop()
+ async with asyncio.timeout(TIMEOUT):
+ await other.ownership.balance()
+ # Balancing again does not wait on the start either.
+ await other.ownership.balance()
+ assert other.connectors.holds(stored.connector_id)
+ assert other.connectors.running_count() == 0
+
+ release.set()
+ await _eventually(lambda: other.connectors.running_count() == 1)
diff --git a/core/tests/switch_core/bridges/agent/server_connectors/test_lifecycle_remove_tenant.py b/core/tests/switch_core/bridges/agent/server_connectors/test_lifecycle_remove_tenant.py
index 3955f03fd..244c3e154 100644
--- a/core/tests/switch_core/bridges/agent/server_connectors/test_lifecycle_remove_tenant.py
+++ b/core/tests/switch_core/bridges/agent/server_connectors/test_lifecycle_remove_tenant.py
@@ -41,6 +41,7 @@ from switch_core.db.stores.server_connector_store import ServerConnectorStore
from switch_core.keys import Keyring
from switch_core.outbound import OutboundPolicy
from switch_core.tenant_context import tenant_scope
+from tests.ownership import SoleOwner
pytestmark = pytest.mark.no_ambient_tenant
@@ -109,6 +110,7 @@ def _service(
session_factory=session_factory,
keyring=Keyring.parse("test:" + "x" * 40, legacy_secret=None),
outbound_policy=OutboundPolicy.parse(""),
+ ownership=SoleOwner(),
)
diff --git a/core/tests/switch_core/bridges/agent/test_api_key_cache.py b/core/tests/switch_core/bridges/agent/test_api_key_cache.py
index 8b950dc86..85e49d7bb 100644
--- a/core/tests/switch_core/bridges/agent/test_api_key_cache.py
+++ b/core/tests/switch_core/bridges/agent/test_api_key_cache.py
@@ -17,7 +17,10 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.bridges.agent.api_key_cache import ApiKeyCache
from switch_core.bridges.agent.auth import BearerAuthMiddleware
-from switch_core.bridges.agent.protocol.agent_connections import HEARTBEAT_TTL_SECONDS
+from switch_core.bridges.agent.protocol.agent_connections import (
+ HEARTBEAT_TTL_SECONDS,
+ AgentConnectionRegistry,
+)
from switch_core.bridges.agent.protocol.agent_core import AgentCore
from switch_core.bridges.agent.protocol.types import (
IntegrationProfile,
@@ -73,6 +76,9 @@ class _FakeClientLifecycle:
def start_client(self, client: Client) -> None:
self.started.append(client.id)
+ def announce(self, tenant_id: str) -> None:
+ return None
+
async def stop(self, client_id: str) -> None:
return None
@@ -426,6 +432,7 @@ def _agent_core(
svc.client_lifecycle = _FakeClientLifecycle(session_factory) # type: ignore[attr-defined]
svc.collab_lifecycle = _NoBridges() # type: ignore[attr-defined]
svc.event_buffer = SimpleNamespace(remove=lambda _agent_id: None) # type: ignore[attr-defined]
+ svc.connections = AgentConnectionRegistry() # type: ignore[attr-defined]
svc.config = SimpleNamespace(
keyring=Keyring.parse("test:" + "x" * 40, legacy_secret=None),
third_party_avatars_enabled=True,
diff --git a/core/tests/switch_core/bridges/agent/test_control_commands.py b/core/tests/switch_core/bridges/agent/test_control_commands.py
index 51506446b..3b4bce41e 100644
--- a/core/tests/switch_core/bridges/agent/test_control_commands.py
+++ b/core/tests/switch_core/bridges/agent/test_control_commands.py
@@ -46,7 +46,7 @@ def _client(
placed: str | None = None,
relayed: list[Any] | None = None,
) -> SimpleNamespace:
- """Minimal AgentConsumer stand-in for _dispatch_control_command."""
+ """Minimal AgentRecipient stand-in for _dispatch_control_command."""
agent = SimpleNamespace(
id="agent-1",
diff --git a/core/tests/switch_core/bridges/collaboration/outbound_fakes.py b/core/tests/switch_core/bridges/collaboration/outbound_fakes.py
new file mode 100644
index 000000000..2cfcbb40e
--- /dev/null
+++ b/core/tests/switch_core/bridges/collaboration/outbound_fakes.py
@@ -0,0 +1,56 @@
+"""The outbound path's database calls, faked for tests that assemble a
+`CollaborationCore` without a database.
+
+Relaying a room message out to the platform reads whether its sender is a
+person on the bridge (`_is_human_sender`) and which channel the room is
+bridged to (`_channel_for_room`), each from the database when it is not known
+in memory, and claims the message before posting it, so a hand-over between
+processes never posts it twice (`_claim_outbound`, then settled with the
+post's id or released). These stand in for those calls: a sender is a person
+and a room has a channel only if the test says so, every claim succeeds, and
+each post is recorded as the message map would hold it.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+def posting_without_a_database(core: Any) -> list[dict[str, str]]:
+ """Fake the outbound database calls on `core`, which may be a bare
+ `CollaborationCore` or a namespace standing in for one. Returns the list
+ every settled post is appended to."""
+ recorded: list[dict[str, str]] = []
+ claimed: dict[str, str] = {}
+
+ async def is_human_sender(sender: str) -> bool:
+ return sender in core._human_user_ids
+
+ async def channel_for_room(transport_room_id: str) -> str | None:
+ channel_id: str | None = core._find_channel(transport_room_id=transport_room_id)
+ return channel_id
+
+ async def claim(channel_id: str, transport_event_id: str) -> bool:
+ claimed[transport_event_id] = channel_id
+ return True
+
+ async def settle(transport_event_id: str, external_post_id: str | None) -> None:
+ channel_id = claimed.pop(transport_event_id)
+ if external_post_id is not None:
+ recorded.append(
+ {
+ "external_channel_id": channel_id,
+ "transport_event_id": transport_event_id,
+ "external_post_id": external_post_id,
+ }
+ )
+
+ async def release(transport_event_id: str) -> None:
+ claimed.pop(transport_event_id, None)
+
+ core._is_human_sender = is_human_sender
+ core._channel_for_room = channel_for_room
+ core._claim_outbound = claim
+ core._settle_outbound = settle
+ core._release_outbound = release
+ return recorded
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_inbound_media.py b/core/tests/switch_core/bridges/collaboration/test_bridge_inbound_media.py
index a8e68b59d..e498a1588 100644
--- a/core/tests/switch_core/bridges/collaboration/test_bridge_inbound_media.py
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_inbound_media.py
@@ -13,6 +13,7 @@ from switch_core.trust.client import NullTrustClient
class _FakeHumanActor:
+ client_id = "human-actor-client"
transport_user_id = "@human actor:s"
def __init__(self) -> None:
@@ -88,6 +89,7 @@ def _fake_bridge() -> SimpleNamespace:
return None
ns = SimpleNamespace(
+ _bridge_id="bridge-1",
_handle_text_answer=_no_text_answer,
_repair_placeholder_username=_repair_placeholder_username,
_adapter=_FakeAdapter(),
@@ -163,6 +165,27 @@ async def test_three_attachments_are_stamped_as_one_group() -> None:
assert bridge.human_actor.messages == []
+async def test_a_post_delivered_again_writes_into_the_same_group() -> None:
+ # A delivery that stopped after the first file is sent again by the
+ # platform; the files it still writes must complete the first group, not
+ # start one of their own.
+ bridge = _fake_bridge()
+ post = _msg(
+ content="two files",
+ attachments=[
+ _attachment("cat.png", "image/png"),
+ _attachment("notes.md", "text/markdown"),
+ ],
+ )
+
+ await CollaborationCore._handle_inbound_message(bridge, post)
+ await CollaborationCore._handle_inbound_message(bridge, post)
+
+ media = bridge.human_actor.media
+ assert len(media) == 4
+ assert len({m["group"]["id"] for m in media}) == 1
+
+
async def test_single_attachment_carries_no_group_marker() -> None:
# A group of one needs no marker — the receiver treats an unmarked event as
# a complete message and never buffers it.
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_inbound_trust.py b/core/tests/switch_core/bridges/collaboration/test_bridge_inbound_trust.py
index 5953d8b6d..103215973 100644
--- a/core/tests/switch_core/bridges/collaboration/test_bridge_inbound_trust.py
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_inbound_trust.py
@@ -34,6 +34,7 @@ class _FakeHumanActor:
the join race `test_bridge_unknown_member_first_message.py` covers."""
def __init__(self, *, fail_send: bool = False) -> None:
+ self.client_id = "alice-client"
self.transport_user_id = "@ext_alice:switch.local"
self.sent: list[tuple[str, str]] = []
self._fail_send = fail_send
@@ -97,6 +98,7 @@ def _bridge(
relayed.append((kwargs["transport_event_id"], kwargs["external_post_id"]))
bridge = SimpleNamespace(
+ _bridge_id="bridge-1",
_repair_placeholder_username=_noop_repair,
_is_registered_agent=_is_registered_agent,
_ensure_human_in_room=_ensure_human_in_room,
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_mention_resolution.py b/core/tests/switch_core/bridges/collaboration/test_bridge_mention_resolution.py
index 183853614..9891641b1 100644
--- a/core/tests/switch_core/bridges/collaboration/test_bridge_mention_resolution.py
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_mention_resolution.py
@@ -136,6 +136,10 @@ async def test_new_human_actor_is_mentionable_without_waiting_for_a_restart() ->
async def _create(_session: object, _user: object) -> None:
return None
+ async def _minted_elsewhere(_external_user_id: str) -> None:
+ # No other process has minted this person either.
+ return None
+
bridge = SimpleNamespace(
_adapter=adapter,
_bridge_id=BRIDGE_ID,
@@ -148,6 +152,7 @@ async def test_new_human_actor_is_mentionable_without_waiting_for_a_restart() ->
_agent_store=SimpleNamespace(get_by_name=_get_by_name),
_client_lifecycle=SimpleNamespace(create_and_start=_create_and_start),
_external_user_store=SimpleNamespace(create=_create),
+ _adopt_human_actor=_minted_elsewhere,
)
bridge._create_human_actor_locked = functools.partial(
CollaborationCore._create_human_actor_locked, bridge
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_admin_rendering.py b/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_admin_rendering.py
index 5a54e7c82..edfaca108 100644
--- a/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_admin_rendering.py
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_admin_rendering.py
@@ -19,6 +19,9 @@ from switch_core.bridges.collaboration.collaboration_core import (
CollaborationCore,
)
from switch_core.transport import InboundMessage, RoomRef
+from tests.switch_core.bridges.collaboration.outbound_fakes import (
+ posting_without_a_database,
+)
BODY = "**Agents in this room:**\n- **scout** — Claude Code"
@@ -78,15 +81,11 @@ def _bridge(adapter: _RecordingAdapter) -> CollaborationCore:
core._find_channel = lambda **_kwargs: "C1" # type: ignore[assignment]
core._channel_to_room = {"C1": ("room-uuid", "!r:switch.local")} # type: ignore[assignment]
core._room_tenant = _tenant # type: ignore[assignment]
- core._record_message_map = _noop # type: ignore[assignment]
+ posting_without_a_database(core)
core._outbound_thread_root_ref = _none # type: ignore[assignment]
return core
-async def _noop(*_args: Any, **_kwargs: Any) -> None:
- return None
-
-
async def _none(*_args: Any, **_kwargs: Any) -> None:
return None
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_media.py b/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_media.py
index 5abf75b86..8a83c80d6 100644
--- a/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_media.py
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_media.py
@@ -11,6 +11,9 @@ from switch_core.transport import (
RoomRef,
TransportError,
)
+from tests.switch_core.bridges.collaboration.outbound_fakes import (
+ posting_without_a_database,
+)
def _media_event(
@@ -122,15 +125,11 @@ def _fake_bridge(
max_bytes: int = 1024,
) -> SimpleNamespace:
adapter = _FakeAdapter()
- recorded: list[dict[str, str]] = []
lookup = room_to_external or {}
async def _external_post_for_event(transport_event_id: str) -> str | None:
return lookup.get(transport_event_id)
- async def _record_message_map(**kwargs: str) -> None:
- recorded.append(kwargs)
-
async def _room_tenant(room_id: str) -> str:
return "tenant-1"
@@ -141,13 +140,13 @@ def _fake_bridge(
_workspace_consumer_transport_user_id="@bridge:s",
_max_attachment_bytes=max_bytes,
_external_post_for_event=_external_post_for_event,
- _record_message_map=_record_message_map,
- recorded=recorded,
_outbound_groups={},
_outbound_group_timers={},
+ _flushing_outbound_groups=[],
_channel_to_room={"chan-1": ("room-uuid", "!room:s")},
_room_tenant=_room_tenant,
)
+ ns.recorded = posting_without_a_database(ns)
ns._find_channel = lambda room_id=None, transport_room_id=None: (
"chan-1" if transport_room_id == "!room:s" else None
)
@@ -167,14 +166,20 @@ def _fake_bridge(
ns._flush_incomplete_outbound_group = (
CollaborationCore._flush_incomplete_outbound_group.__get__(ns)
)
+ ns._flush_outbound_group = CollaborationCore._flush_outbound_group.__get__(ns)
async def _download_media(uri: str) -> DownloadResult:
if isinstance(download, TransportError):
raise download
return download
+ released: list[str] = []
+ ns.released = released
ns.client = SimpleNamespace(
- transport=SimpleNamespace(download_media=_download_media)
+ running=True,
+ transport=SimpleNamespace(
+ download_media=_download_media, held_back_released=released.append
+ ),
)
return ns
@@ -375,3 +380,121 @@ async def test_incomplete_group_is_flushed_with_a_disclosed_notice() -> None:
assert [f.filename for f in batch["files"]] == ["only.md"]
assert "1 of 3" in (batch["caption"] or "")
assert bridge._outbound_groups == {}
+ # Posted, so the bridge's progress in the room may move past it.
+ assert bridge.released == ["!room:s"]
+
+
+async def test_an_incomplete_group_is_left_to_the_next_holder_once_stopped() -> None:
+ """A bridge that stopped while a group waited for its other files posts
+ nothing: its progress in the room stops before the group's first file,
+ so whichever process holds the bridge next reads the group again."""
+ import switch_core.bridges.collaboration.collaboration_core as bc
+
+ original = bc.OUTBOUND_GROUP_TIMEOUT_SECONDS
+ bc.OUTBOUND_GROUP_TIMEOUT_SECONDS = 0.01
+ try:
+ bridge = _fake_bridge()
+ await CollaborationCore.handle_outbound_media(
+ bridge,
+ _room(),
+ _media_event(
+ msgtype="m.file",
+ body="only.md",
+ mimetype="text/markdown",
+ group={"id": "grp-3", "index": 0, "total": 2},
+ ),
+ bridge.client,
+ )
+ bridge.client.running = False
+
+ await asyncio.sleep(0.1)
+ finally:
+ bc.OUTBOUND_GROUP_TIMEOUT_SECONDS = original
+
+ assert bridge._adapter.batches == []
+ assert bridge.released == []
+
+
+async def test_a_file_arriving_while_its_group_times_out_is_still_posted() -> None:
+ """The timeout posts the files that arrived; one arriving while that post
+ is on its way is posted on its own, not added to a post already sent
+ without it."""
+ import switch_core.bridges.collaboration.collaboration_core as bc
+
+ bridge = _fake_bridge()
+ adapter = bridge._adapter
+ posting = asyncio.Event()
+ posted = asyncio.Event()
+ send_attachments = adapter.send_attachments
+
+ async def slow_send_attachments(*args: Any, **kwargs: Any) -> str:
+ if not posting.is_set():
+ posting.set()
+ await posted.wait()
+ result: str = await send_attachments(*args, **kwargs)
+ return result
+
+ adapter.send_attachments = slow_send_attachments
+ original = bc.OUTBOUND_GROUP_TIMEOUT_SECONDS
+ bc.OUTBOUND_GROUP_TIMEOUT_SECONDS = 0.01
+ try:
+ for index, name in [(0, "first.md"), (1, "late.md")]:
+ await CollaborationCore.handle_outbound_media(
+ bridge,
+ _room(),
+ _media_event(
+ msgtype="m.file",
+ body=name,
+ mimetype="text/markdown",
+ event_id=f"$part-{index}",
+ group={"id": "grp-4", "index": index, "total": 2},
+ ),
+ bridge.client,
+ )
+ if index == 0:
+ await asyncio.wait_for(posting.wait(), 1.0)
+ posted.set()
+ await asyncio.sleep(0.1)
+ finally:
+ bc.OUTBOUND_GROUP_TIMEOUT_SECONDS = original
+
+ assert [[f.filename for f in batch["files"]] for batch in adapter.batches] == [
+ ["first.md"],
+ ["late.md"],
+ ]
+ assert bridge._outbound_groups == {}
+ assert bridge._flushing_outbound_groups == []
+
+
+async def test_a_post_cancelled_by_a_stop_lets_its_claim_go() -> None:
+ """A bridge stopping past its grace cancels the post in hand. The claim
+ goes with it, so the next holder, reading the message again, posts it."""
+ bridge = _fake_bridge()
+ sending = asyncio.Event()
+
+ async def hanging_send_attachment(*args: Any, **kwargs: Any) -> str:
+ sending.set()
+ await asyncio.Event().wait()
+ raise AssertionError("never answered")
+
+ bridge._adapter.send_attachment = hanging_send_attachment
+ released: list[str] = []
+ release = bridge._release_outbound
+
+ async def recording_release(transport_event_id: str) -> None:
+ released.append(transport_event_id)
+ await release(transport_event_id)
+
+ bridge._release_outbound = recording_release
+ posting = asyncio.create_task(
+ CollaborationCore.handle_outbound_media(
+ bridge, _room(), _media_event(event_id="$cut-short"), bridge.client
+ )
+ )
+ await asyncio.wait_for(sending.wait(), 1.0)
+
+ posting.cancel()
+ await asyncio.gather(posting, return_exceptions=True)
+
+ assert posting.cancelled()
+ assert released == ["$cut-short"]
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_platform_rendering.py b/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_platform_rendering.py
index 8acc1d6dc..f6e83e3d5 100644
--- a/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_platform_rendering.py
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_outbound_platform_rendering.py
@@ -13,6 +13,9 @@ from typing import Any
from switch_core.bridges.collaboration.collaboration_core import CollaborationCore
from switch_core.clients.admin_messages import PLATFORM_MARKER
from switch_core.transport import InboundMessage, RoomRef
+from tests.switch_core.bridges.collaboration.outbound_fakes import (
+ posting_without_a_database,
+)
BODY = "@scout kicking off **task-1**."
@@ -64,7 +67,7 @@ def _bridge(adapter: _RecordingAdapter) -> CollaborationCore:
core._find_channel = lambda **_kwargs: "C1" # type: ignore[assignment]
core._channel_to_room = {"C1": ("room-uuid", "!r:switch.local")} # type: ignore[assignment]
core._room_tenant = _tenant # type: ignore[assignment]
- core._record_message_map = _noop # type: ignore[assignment]
+ posting_without_a_database(core)
core._move_indicator_for_sender = _noop # type: ignore[assignment]
core._outbound_thread_root_ref = _noop # type: ignore[assignment]
return core
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_ownership.py b/core/tests/switch_core/bridges/collaboration/test_bridge_ownership.py
new file mode 100644
index 000000000..ba2f16c85
--- /dev/null
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_ownership.py
@@ -0,0 +1,1060 @@
+"""One process listens for a collaboration bridge; every other runs it quietly.
+
+Against real PostgreSQL, two "processes" on one database, each wired the way
+`main` wires one for bridges: its own message listener and signal bus, process
+lease, ownership and cursor writer, client lifecycle, room service and bridge
+lifecycle. Both run the bridge. The platform here (`_Platform`) is the one
+messaging service both copies talk to: it knows which copies listen to it and
+records what is posted to it, so a test can say that a reply reached the
+channel once and that only one process holds the connection.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+from collections.abc import AsyncIterator, Awaitable, Callable
+from dataclasses import dataclass, field
+from datetime import timedelta
+from typing import Any, ClassVar
+from unittest.mock import MagicMock
+
+import pytest
+from sqlalchemy import func, select, update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
+from switch_core.bridges.collaboration.adapter import PlatformAdapter
+from switch_core.bridges.collaboration.bridge_signals import BridgeSignals
+from switch_core.bridges.collaboration.lifecycle_service import (
+ CollaborationBridgeLifecycleService,
+)
+from switch_core.bridges.collaboration.models import (
+ BridgeConnectionConfig,
+ ChannelType,
+ InboundAgentJoin,
+ InboundAppJoin,
+ InboundCommand,
+ InboundMessage,
+ InboundUserJoin,
+)
+from switch_core.bridges.resource.service import ResourceService
+from switch_core.clients.actor import Actor, HumanActor
+from switch_core.clients.client_factory import ClientFactory
+from switch_core.clients.client_lifecycle_service import ClientLifecycleService
+from switch_core.db.models import (
+ TENANT_ZERO_ID,
+ BridgeMessageMap,
+ Client,
+ CollaborationBridge,
+ ExternalUser,
+ Message,
+ Room,
+)
+from switch_core.db.stores.agent_store import AgentStore
+from switch_core.db.stores.bridge_message_map_store import BridgeMessageMapStore
+from switch_core.db.stores.client_store import ClientStore
+from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
+from switch_core.db.stores.document_store import DocumentStore
+from switch_core.db.stores.external_user_store import ExternalUserStore
+from switch_core.db.stores.media_store import MediaStore
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+from switch_core.db.stores.package_store import PackageStore
+from switch_core.db.stores.reference_store import ReferenceStore
+from switch_core.db.stores.reference_type_store import ReferenceTypeStore
+from switch_core.db.stores.room_link_store import RoomLinkStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.db.stores.tenant_store import TenantStore
+from switch_core.db.stores.usage_store import UsageStore
+from switch_core.keys import Keyring
+from switch_core.messages.notify import MessageListener
+from switch_core.ownership import Ownership
+from switch_core.process_lease import ProcessLease
+from switch_core.provisioning.postgres import PostgresProvisioning
+from switch_core.room_service import RoomCreateConfig, RoomService
+from switch_core.signals import SignalBus
+from switch_core.tenant_context import tenant_scope
+from switch_core.transport.cursors import CursorWriter
+from switch_core.transport.ephemeral import EphemeralBus
+from switch_core.transport.invites import InviteBus
+from switch_core.transport.membership_signals import MembershipSignals
+from switch_core.transport.observer import IgnoreParticipantMessages
+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
+from tests.ownership import process_lease_on
+from tests.routing import router_on
+
+TIMEOUT = 10.0
+CHAT = "fakechat"
+LOUD_CHAT = "loudchat"
+CHANNEL = "C1"
+
+
+class _Platform:
+ """The messaging service every copy of the bridge talks to."""
+
+ def __init__(self) -> None:
+ self.listening: list[_Chat] = []
+ self.posts: list[tuple[str, str, str]] = []
+
+ async def say(self, *, user_id: str, text: str, ref: str) -> None:
+ """Someone writes in the channel. Every open connection hears it, as
+ Mattermost hands every post to every socket. A bridge connects in its
+ own task once it is started, so this waits for a connection first."""
+ async with asyncio.timeout(TIMEOUT):
+ while not self.listening:
+ await asyncio.sleep(0.01)
+ for chat in list(self.listening):
+ await chat.hear(_post(user_id=user_id, text=text, ref=ref))
+
+ async def command(self, *, user_id: str, command: str, ref: str) -> None:
+ """Someone types a command in the channel, heard like a post."""
+ async with asyncio.timeout(TIMEOUT):
+ while not self.listening:
+ await asyncio.sleep(0.01)
+ for chat in list(self.listening):
+ await chat.hear_command(
+ InboundCommand(
+ channel_id=CHANNEL,
+ channel_type="channel_public",
+ sender_id=user_id,
+ sender_name=user_id.lower(),
+ command=command,
+ args="",
+ message_ref=ref,
+ )
+ )
+
+ def posted(self) -> list[str]:
+ return [text for _, _, text in self.posts]
+
+
+def _post(*, user_id: str, text: str, ref: str) -> InboundMessage:
+ return InboundMessage(
+ channel_id=CHANNEL,
+ channel_type="channel_public",
+ sender_id=user_id,
+ sender_name=user_id.lower(),
+ content=text,
+ message_ref=ref,
+ )
+
+
+class _ChatConfig(BridgeConnectionConfig):
+ token: str
+
+
+class _Chat(PlatformAdapter):
+ """A platform that listens over a connection of its own, like Slack's
+ Socket Mode or Mattermost's websocket, and keeps quiet when told to."""
+
+ display_name: ClassVar[str] = "Fake Chat"
+ supports_channel_creation: ClassVar[bool] = False
+ supports_directory_search: ClassVar[bool] = False
+ keeps_quiet_unless_listening: ClassVar[bool] = True
+ platform: ClassVar[_Platform]
+
+ def __init__(self, *, config: _ChatConfig) -> None:
+ super().__init__()
+ self.config = config
+
+ async def start(
+ self,
+ on_message: Callable[[InboundMessage], Awaitable[None]],
+ on_command: Callable[[InboundCommand], Awaitable[None]],
+ on_agent_joined: Callable[[InboundAgentJoin], Awaitable[None]],
+ on_user_joined: Callable[[InboundUserJoin], Awaitable[None]],
+ on_app_joined: Callable[[InboundAppJoin], Awaitable[None]],
+ ) -> None:
+ self._on_message = on_message
+ self._on_command = on_command
+ if self._listening or not self.keeps_quiet_unless_listening:
+ self.platform.listening.append(self)
+
+ async def stop(self) -> None:
+ if self in self.platform.listening:
+ self.platform.listening.remove(self)
+
+ async def hear(self, message: InboundMessage) -> None:
+ """A delivery from the platform: over the connection where this copy
+ listens, or a webhook reaching a quiet one."""
+ assert self._on_message is not None
+ await self._on_message(message)
+
+ async def hear_command(self, command: InboundCommand) -> None:
+ assert self._on_command is not None
+ await self._on_command(command)
+
+ async def send_message(
+ self,
+ channel_id: str,
+ sender_name: str,
+ content: str,
+ thread_root_id: str | None = None,
+ *,
+ room_wide_mention: bool = False,
+ ) -> str | None:
+ self.platform.posts.append((channel_id, sender_name, content))
+ return f"post-{len(self.platform.posts)}"
+
+ async def update_message(self, *a: Any, **k: Any) -> Any:
+ return None
+
+ async def delete_message(self, *a: Any, **k: Any) -> Any:
+ return None
+
+ async def send_typing(self, *a: Any, **k: Any) -> Any:
+ return None
+
+ async def create_channel(self, *a: Any, **k: Any) -> Any:
+ raise NotImplementedError
+
+ async def get_channel_type(self, channel_id: str) -> ChannelType:
+ return "channel_public"
+
+ async def add_agents_to_channel(self, *a: Any, **k: Any) -> Any:
+ return None
+
+ async def add_users_to_channel(self, *a: Any, **k: Any) -> Any:
+ return None
+
+ async def create_agent_identity(self, *a: Any, **k: Any) -> Any:
+ return None
+
+ async def remove_agent_identity(self, *a: Any, **k: Any) -> Any:
+ return None
+
+ async def get_channel_agent_names(self, *a: Any, **k: Any) -> Any:
+ return []
+
+ def _render_outbound(self, content: str) -> str:
+ return content
+
+ def translate_inbound(self, raw: Any) -> str:
+ return str(raw)
+
+
+class _LoudChat(_Chat):
+ """A platform whose adapter cannot keep quiet, as Discord's gateway and
+ Teams' listener are today: every process's copy listens to it."""
+
+ keeps_quiet_unless_listening: ClassVar[bool] = False
+
+
+class _PublisherThatFailsToStop:
+ """A bridge's session-activity publisher whose stop raises. Stands in for
+ anything that can go wrong as a bridge stops, before its adapter does."""
+
+ async def stop(self) -> None:
+ raise RuntimeError("the publisher's task had died")
+
+
+@dataclass
+class _Process:
+ lease: ProcessLease
+ ownership: Ownership
+ bus: SignalBus
+ writer: CursorWriter
+ listener: MessageListener
+ clients: ClientLifecycleService
+ bridges: CollaborationBridgeLifecycleService
+ tasks: list[asyncio.Task[None]] = field(default_factory=list)
+ stopped: bool = False
+
+ async def boot(self) -> None:
+ """Start as `main` does: every client, every bridge wanted, the first
+ balance starting what this process holds, then the quiet copies."""
+ await self.clients.start_all()
+ await self.bridges.start_all()
+ await self.ownership.start()
+ await self.bridges.start_quiet_copies()
+
+ async def settled(self) -> None:
+ """Wait out the quiet copies a hand-over starts in the background."""
+ async with asyncio.timeout(TIMEOUT):
+ while self.bridges._quiet_starts:
+ await asyncio.sleep(0.01)
+
+ async def stop(self) -> None:
+ """Shut down as `main` does: owned work first, then the rest."""
+ if self.stopped:
+ return
+ self.stopped = True
+ await self.ownership.stop()
+ await self.bridges.stop_all()
+ await self.lease.stop()
+ await self.clients.stop_all()
+ for task in self.tasks:
+ task.cancel()
+ await asyncio.gather(*self.tasks, return_exceptions=True)
+ await self.bus.stop()
+ await self.listener.stop()
+
+
+def _config() -> Any:
+ config = MagicMock()
+ config.gateway_public_url = "https://gw.example"
+ config.messaging_public_url = None
+ config.collaboration_callback_host = "127.0.0.1"
+ config.collaboration_callback_port = 0
+ config.keyring = Keyring.parse("test:" + "x" * 40, legacy_secret=None)
+ config.agent_media_max_bytes = 1 << 20
+ config.id_server_name = "test"
+ return config
+
+
+async def _process(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession]
+) -> _Process:
+ def engine() -> Any:
+ return create_async_engine(postgres_url, poolclass=NullPool)
+
+ listener = MessageListener(engine)
+ await listener.start()
+ await asyncio.wait_for(listener.connected.wait(), TIMEOUT)
+ lease = process_lease_on(session_factory)
+ await lease.renew()
+ bus = SignalBus(session_factory, engine, process_id=lease.process_id)
+ ownership = Ownership(
+ lease=lease,
+ session_factory=session_factory,
+ bus=bus,
+ store=OwnershipLeaseStore(),
+ processes=SwitchCoreProcessStore(),
+ )
+ writer = CursorWriter(
+ session_factory=session_factory,
+ cursors=DeliveryCursorStore(),
+ leases=OwnershipLeaseStore(),
+ process_id=lease.process_id,
+ )
+ invites = InviteBus()
+ # This process's own router: the rows it writes are routed here.
+ router = router_on(session_factory)
+ MembershipSignals(invites=invites, bus=bus, router=router)
+ room_cache = RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=MessageStore(),
+ limits=RoomCacheLimits(
+ max_bytes=1 << 24,
+ max_rooms=100,
+ max_rows_per_room=200,
+ max_age_seconds=60.0,
+ ),
+ )
+ factory = ClientFactory(
+ client_store=ClientStore(),
+ session_factory=session_factory,
+ config=MagicMock(),
+ room_store=RoomStore(),
+ message_store=MessageStore(),
+ usage_store=UsageStore(),
+ media_store=MediaStore(),
+ listener=listener,
+ invites=invites,
+ ephemeral=EphemeralBus(),
+ room_cache=room_cache,
+ message_observer=IgnoreParticipantMessages(),
+ router=router,
+ )
+ factory.register("user", HumanActor)
+ factory.register("bridge", Actor)
+ provisioning = PostgresProvisioning(
+ session_factory=session_factory,
+ room_store=RoomStore(),
+ client_store=ClientStore(),
+ router=router,
+ invites=invites,
+ )
+ clients = ClientLifecycleService(
+ provisioning=provisioning,
+ client_store=ClientStore(),
+ tenant_store=TenantStore(),
+ client_factory=factory,
+ session_factory=session_factory,
+ # The server name the bridges read back clients' ids with.
+ config=_config(),
+ tenants_isolated=True,
+ ownership=ownership,
+ cursor_writer=writer,
+ )
+ bridges = CollaborationBridgeLifecycleService(
+ bridge_store=CollaborationBridgeStore(),
+ external_user_store=ExternalUserStore(),
+ bridge_message_map_store=BridgeMessageMapStore(),
+ room_store=RoomStore(),
+ agent_store=AgentStore(),
+ client_store=ClientStore(),
+ client_lifecycle=clients,
+ room_service=None, # type: ignore[arg-type] # set below
+ provisioning=provisioning,
+ session_factory=session_factory,
+ config=_config(),
+ client_factory=factory,
+ session_activity_listener=MagicMock(),
+ session_activity_service=MagicMock(),
+ connections=AgentConnectionRegistry(),
+ ownership=ownership,
+ cursor_writer=writer,
+ )
+ bridges.register_adapter(CHAT, _Chat, _ChatConfig)
+ bridges.register_adapter(LOUD_CHAT, _LoudChat, _ChatConfig)
+ bridges._room_service = RoomService(
+ provisioning=provisioning,
+ room_store=RoomStore(),
+ agent_store=AgentStore(),
+ client_lifecycle=clients,
+ collab_lifecycle=bridges,
+ collab_bridge_store=CollaborationBridgeStore(),
+ resource_service=ResourceService(
+ reference_store=ReferenceStore(),
+ reference_type_store=ReferenceTypeStore(),
+ document_store=DocumentStore(),
+ package_store=PackageStore(),
+ room_link_store=RoomLinkStore(),
+ session_factory=session_factory,
+ ),
+ session_factory=session_factory,
+ room_cache=room_cache,
+ )
+ BridgeSignals(lifecycle=bridges, bus=bus)
+ clients.add_agents_changed_listener(bridges.agents_changed)
+ await bus.start()
+ await asyncio.wait_for(bus.connected.wait(), TIMEOUT)
+ return _Process(
+ lease=lease,
+ ownership=ownership,
+ bus=bus,
+ writer=writer,
+ listener=listener,
+ clients=clients,
+ bridges=bridges,
+ tasks=[
+ asyncio.create_task(lease.run()),
+ asyncio.create_task(writer.run()),
+ ],
+ )
+
+
+@dataclass
+class _Bridge:
+ bridge_id: str
+ client_id: str
+ room_id: str
+ transport_room_id: str
+ speaker_id: str
+ speaker: str
+
+
+def platform() -> AsyncIterator[_Platform] | Any:
+ platform = _Platform()
+ _Chat.platform = platform
+ return platform
+
+
+async def bridge(session_factory: async_sessionmaker[AsyncSession]) -> _Bridge:
+ """A bridge on the fake platform carrying one room, bridged to `CHANNEL`,
+ and someone in the room whose messages the bridge relays out."""
+ suffix = uuid.uuid4().hex[:8]
+ async with session_factory() as session:
+ bridge_client = Client(
+ transport_user_id=f"@switch-bridge-{suffix}:test",
+ display_name="bridge",
+ type="bridge",
+ )
+ speaker = Client(
+ transport_user_id=f"@speaker-{suffix}:test",
+ display_name="speaker",
+ type="user",
+ )
+ session.add_all([bridge_client, speaker])
+ await session.flush()
+ row = CollaborationBridge(
+ type=CHAT,
+ display_name="Fake",
+ connection_config={"token": "first"},
+ client_id=bridge_client.id,
+ status="active",
+ channel_creation_enabled=False,
+ preconfigured=False,
+ )
+ session.add(row)
+ await session.flush()
+ room = Room(
+ transport_room_id=f"!room-{suffix}:test",
+ name="bridged",
+ description="",
+ bridge_id=row.id,
+ external_channel_id=CHANNEL,
+ channel_type="channel_public",
+ )
+ session.add(room)
+ await session.flush()
+ await RoomStore().add_client(session, bridge_client.id, room.id)
+ await RoomStore().add_client(session, speaker.id, room.id)
+ await session.commit()
+ return _Bridge(
+ bridge_id=row.id,
+ client_id=bridge_client.id,
+ room_id=room.id,
+ transport_room_id=room.transport_room_id,
+ speaker_id=speaker.id,
+ speaker=speaker.transport_user_id,
+ )
+
+
+async def processes(
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> AsyncIterator[list[_Process]]:
+ started: list[_Process] = []
+ try:
+ yield started
+ finally:
+ for process in started:
+ await process.stop()
+
+
+async def _boot_two(
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+) -> tuple[_Process, _Process]:
+ for _ in range(2):
+ process = await _process(postgres_url, session_factory)
+ processes.append(process)
+ await process.boot()
+ # The first balance of each cannot see the other; balance once more now
+ # that both leases are there, as the interval would.
+ for process in processes:
+ await process.ownership.balance()
+ for process in processes:
+ await process.settled()
+ return processes[0], processes[1]
+
+
+def _holder(
+ two: tuple[_Process, _Process], bridge_id: str
+) -> tuple[_Process, _Process]:
+ a, b = two
+ holders = [p for p in (a, b) if p.bridges.holds(bridge_id)]
+ assert len(holders) == 1
+ return (a, b) if holders[0] is a else (b, a)
+
+
+async def _say(
+ session_factory: async_sessionmaker[AsyncSession],
+ bridge: _Bridge,
+ body: str,
+ *,
+ room_id: str | None = None,
+) -> str:
+ """Someone in Switch writes in the room, for the bridge to relay out."""
+ event_id = f"sw_{uuid.uuid4().hex}"
+ async with session_factory() as session:
+ await MessageStore().create(
+ session,
+ Message(
+ room_id=room_id or bridge.room_id,
+ transport_event_id=event_id,
+ sender_id=bridge.speaker,
+ sender_client_id=bridge.speaker_id,
+ sender_name="speaker",
+ event_type="m.room.message",
+ msgtype="m.text",
+ body=body,
+ formatted_body=None,
+ content={"body": body, "sender_name": "speaker"},
+ ),
+ [],
+ )
+ await session.commit()
+ return event_id
+
+
+async def _eventually(check: Callable[[], bool]) -> None:
+ async with asyncio.timeout(TIMEOUT):
+ while not check():
+ await asyncio.sleep(0.02)
+
+
+async def _posted(platform: _Platform, count: int) -> list[str]:
+ await _eventually(lambda: len(platform.posts) >= count)
+ # Long enough for a second relay, had there been one, to have posted.
+ await asyncio.sleep(0.5)
+ return platform.posted()
+
+
+async def _room_bodies(
+ session_factory: async_sessionmaker[AsyncSession], room_id: str, sender: str
+) -> list[str | None]:
+ async with session_factory() as session:
+ rows = (
+ await session.scalars(
+ select(Message.body)
+ .where(
+ Message.room_id == room_id,
+ Message.sender_id == sender,
+ Message.event_type == "m.room.message",
+ )
+ .order_by(Message.seq)
+ )
+ ).all()
+ return list(rows)
+
+
+class TestOneConnection:
+ async def test_one_process_listens_and_the_other_runs_quietly(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ holder, quiet = _holder(
+ await _boot_two(postgres_url, session_factory, processes),
+ bridge.bridge_id,
+ )
+
+ # A bridge connects in its own task, after it is started.
+ await _eventually(lambda: len(platform.listening) == 1)
+ await asyncio.sleep(0.5)
+ assert len(platform.listening) == 1
+ held = holder.bridges.get(bridge.bridge_id)
+ copy = quiet.bridges.get(bridge.bridge_id)
+ assert held is not None and held.active
+ assert copy is not None and not copy.active
+ assert platform.listening == [held.adapter]
+
+ async def test_a_reply_is_posted_once(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ await _boot_two(postgres_url, session_factory, processes)
+
+ await _say(session_factory, bridge, "hello channel")
+
+ assert await _posted(platform, 1) == ["hello channel"]
+
+
+class TestHandingOver:
+ async def test_the_next_holder_resumes_with_no_repeat_and_no_gap(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ holder, other = _holder(
+ await _boot_two(postgres_url, session_factory, processes),
+ bridge.bridge_id,
+ )
+ await _say(session_factory, bridge, "one")
+ await _posted(platform, 1)
+
+ await holder.stop()
+ # Said while no process listens for the bridge.
+ await _say(session_factory, bridge, "two")
+ await other.ownership.balance()
+ await _say(session_factory, bridge, "three")
+
+ assert await _posted(platform, 3) == ["one", "two", "three"]
+ assert other.bridges.holds(bridge.bridge_id)
+ assert len(platform.listening) == 1
+
+ async def test_a_bridge_that_fails_to_stop_still_closes_its_connection(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ """The hand-over's stop raised before it reached the adapter. The
+ lease is let go of all the same, so the connection is closed first:
+ the next holder's must not run beside it."""
+ holder, other = _holder(
+ await _boot_two(postgres_url, session_factory, processes),
+ bridge.bridge_id,
+ )
+ held = holder.bridges.get(bridge.bridge_id)
+ assert held is not None
+ await _eventually(lambda: platform.listening == [held.adapter])
+ held._activity_publisher = _PublisherThatFailsToStop() # type: ignore[assignment]
+
+ # As `main` shuts a process down: its owned work, then its lease.
+ await holder.ownership.stop()
+ await holder.lease.stop()
+ await other.ownership.balance()
+
+ assert holder.bridges.get(bridge.bridge_id) is None
+ taken = other.bridges.get(bridge.bridge_id)
+ assert taken is not None and taken.active
+ await _eventually(lambda: platform.listening == [taken.adapter])
+
+ async def test_what_the_last_holder_posted_is_not_posted_again(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ """A holder that died after posting a message and before storing how
+ far it had got leaves the message for the next holder to read again.
+ Its claim on the post is what stops the second one."""
+ posted = await _say(session_factory, bridge, "posted before it died")
+ async with session_factory() as session:
+ session.add(
+ BridgeMessageMap(
+ bridge_id=bridge.bridge_id,
+ external_channel_id=CHANNEL,
+ transport_event_id=posted,
+ external_post_id="post-from-the-last-holder",
+ )
+ )
+ await session.commit()
+
+ await _boot_two(postgres_url, session_factory, processes)
+ await _say(session_factory, bridge, "said after")
+
+ assert await _posted(platform, 1) == ["said after"]
+
+ async def test_a_post_the_last_holder_claimed_and_never_sent_is_posted(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ """A holder that died between claiming a message and posting it
+ leaves a claim with no post. The next holder reaches it once that
+ holder's lease has lapsed, by when the claim is long past any post in
+ flight, and posts it. A claim as fresh as a post in flight is left."""
+ claims = {
+ "claimed before it died": func.now() - timedelta(seconds=30),
+ "claimed just now": func.now(),
+ }
+ for body, claimed_at in claims.items():
+ event_id = await _say(session_factory, bridge, body)
+ async with session_factory() as session:
+ session.add(
+ BridgeMessageMap(
+ bridge_id=bridge.bridge_id,
+ external_channel_id=CHANNEL,
+ transport_event_id=event_id,
+ external_post_id=None,
+ created_at=claimed_at,
+ )
+ )
+ await session.commit()
+
+ await _boot_two(postgres_url, session_factory, processes)
+ await _say(session_factory, bridge, "said after")
+
+ assert await _posted(platform, 2) == ["claimed before it died", "said after"]
+
+ async def test_a_post_delivered_again_to_the_next_holder_is_stored_once(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ """As Slack does with an event whose acknowledgement it did not get:
+ the holder stored it and stopped before acknowledging it, and the
+ platform delivers it again to whoever listens next."""
+ holder, other = _holder(
+ await _boot_two(postgres_url, session_factory, processes),
+ bridge.bridge_id,
+ )
+ await platform.say(user_id="U1", text="only once", ref="m1")
+ await holder.stop()
+ await other.ownership.balance()
+
+ await platform.say(user_id="U1", text="only once", ref="m1")
+
+ async with session_factory() as session:
+ people = (
+ await session.scalars(
+ select(ExternalUser.client_id).where(
+ ExternalUser.bridge_id == bridge.bridge_id
+ )
+ )
+ ).all()
+ person = await session.get(Client, people[0])
+ assert len(people) == 1 and person is not None
+ assert await _room_bodies(
+ session_factory, bridge.room_id, person.transport_user_id
+ ) == ["only once"]
+
+
+class TestChangesMadeElsewhere:
+ async def test_an_edit_through_the_quiet_process_reaches_the_holder(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ holder, quiet = _holder(
+ await _boot_two(postgres_url, session_factory, processes),
+ bridge.bridge_id,
+ )
+
+ # What the gateway's edit route does: store the config, then restart.
+ async with session_factory() as session:
+ await session.execute(
+ update(CollaborationBridge)
+ .where(CollaborationBridge.id == bridge.bridge_id)
+ .values(connection_config={"token": "second"})
+ )
+ await session.commit()
+ await quiet.bridges.restart(bridge.bridge_id)
+
+ await _eventually(
+ lambda: [chat.config.token for chat in platform.listening] == ["second"]
+ )
+ held = holder.bridges.get(bridge.bridge_id)
+ assert held is not None and held.active
+ assert platform.listening == [held.adapter]
+
+ async def test_what_a_bridge_learned_and_stored_restarts_no_copy(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ """Teams stores the service URL and each channel's team as it learns
+ them from traffic, in the bridge's config. That is not an edit: the
+ re-check restarts no copy of the bridge for it."""
+ holder, quiet = _holder(
+ await _boot_two(postgres_url, session_factory, processes),
+ bridge.bridge_id,
+ )
+ await _eventually(lambda: quiet.bridges.is_connected(bridge.bridge_id))
+ running = [p.bridges.get(bridge.bridge_id) for p in (holder, quiet)]
+
+ await holder.bridges._persist_service_url(
+ bridge.bridge_id, TENANT_ZERO_ID, "https://smba.example/emea/"
+ )
+ await holder.bridges._persist_channel_team(
+ bridge.bridge_id, TENANT_ZERO_ID, "C9", "team-9"
+ )
+ for process in (holder, quiet):
+ await process.bridges.reconcile_all()
+
+ assert [p.bridges.get(bridge.bridge_id) for p in (holder, quiet)] == running
+
+ async def test_a_bridge_removed_through_the_quiet_process_stops_listening(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ holder, quiet = _holder(
+ await _boot_two(postgres_url, session_factory, processes),
+ bridge.bridge_id,
+ )
+
+ await quiet.bridges.remove(bridge.bridge_id)
+
+ await _eventually(lambda: platform.listening == [])
+ await _eventually(lambda: holder.bridges.get(bridge.bridge_id) is None)
+ assert not holder.bridges.holds(bridge.bridge_id)
+
+ async def test_a_room_bridged_through_the_quiet_process_is_relayed(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ holder, quiet = _holder(
+ await _boot_two(postgres_url, session_factory, processes),
+ bridge.bridge_id,
+ )
+
+ with tenant_scope(TENANT_ZERO_ID):
+ created = await quiet.bridges._room_service.create_room(
+ RoomCreateConfig(
+ name="second",
+ description="",
+ bridge_id=bridge.bridge_id,
+ external_channel_id="C2",
+ channel_type="channel_public",
+ created_by_kind="system",
+ )
+ )
+ async with session_factory() as session:
+ await RoomStore().add_client(session, bridge.speaker_id, created.room.id)
+ await session.commit()
+ await _say(session_factory, bridge, "in the new room", room_id=created.room.id)
+
+ await _eventually(lambda: len(platform.posts) >= 1)
+ assert platform.posts[0][0] == "C2"
+ assert platform.posted() == ["in the new room"]
+
+
+class TestPeopleOnThePlatform:
+ async def test_a_person_minted_by_the_quiet_process_is_known_to_the_holder(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ holder, quiet = _holder(
+ await _boot_two(postgres_url, session_factory, processes),
+ bridge.bridge_id,
+ )
+ copy = quiet.bridges.get(bridge.bridge_id)
+ assert copy is not None
+ # The copy comes up in its own task, after it is started.
+ await _eventually(lambda: quiet.bridges.is_connected(bridge.bridge_id))
+
+ # A webhook delivery reaching the quiet process: it mints the person.
+ await copy.adapter.hear(_post(user_id="U7", text="first words", ref="m1")) # type: ignore[attr-defined]
+ await _say(session_factory, bridge, "answer")
+ # The holder relays the room out: the person's own words are not
+ # echoed back to them, though the holder never minted them.
+ assert await _posted(platform, 1) == ["answer"]
+
+ # Then the platform tells the holder about them: adopted, not minted
+ # a second time.
+ await platform.say(user_id="U7", text="second words", ref="m2")
+
+ async with session_factory() as session:
+ people = (
+ await session.scalars(
+ select(ExternalUser.client_id).where(
+ ExternalUser.bridge_id == bridge.bridge_id
+ )
+ )
+ ).all()
+ person = await session.get(Client, people[0])
+ assert len(people) == 1 and person is not None
+ assert await _room_bodies(
+ session_factory, bridge.room_id, person.transport_user_id
+ ) == ["first words", "second words"]
+ assert platform.posted() == ["answer"]
+
+ async def test_two_people_whose_names_read_alike_stay_two_people(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ """`ann.marie` and `ann-marie` sanitize to the same client localpart.
+ The second to speak is a person of their own, not the first."""
+ await _boot_two(postgres_url, session_factory, processes)
+
+ await platform.say(user_id="Ann.Marie", text="from the first", ref="m1")
+ await platform.say(user_id="Ann-Marie", text="from the second", ref="m2")
+
+ async with session_factory() as session:
+ people = dict(
+ (
+ await session.execute(
+ select(ExternalUser.external_user_id, Client.transport_user_id)
+ .join(Client, Client.id == ExternalUser.client_id)
+ .where(ExternalUser.bridge_id == bridge.bridge_id)
+ )
+ )
+ .tuples()
+ .all()
+ )
+ assert set(people) == {"Ann.Marie", "Ann-Marie"}
+ assert people["Ann.Marie"] != people["Ann-Marie"]
+ assert await _room_bodies(
+ session_factory, bridge.room_id, people["Ann.Marie"]
+ ) == ["from the first"]
+ assert await _room_bodies(
+ session_factory, bridge.room_id, people["Ann-Marie"]
+ ) == ["from the second"]
+
+
+class TestAPlatformThatCannotKeepQuiet:
+ """Every process listens to it, so every process hears each post. The
+ room still gets each one once: what a post writes is numbered from it."""
+
+ async def test_a_post_every_process_hears_is_stored_once(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ await _loud(session_factory, bridge)
+ await _boot_two(postgres_url, session_factory, processes)
+ await _eventually(lambda: len(platform.listening) == 2)
+
+ await platform.say(user_id="U3", text="heard twice", ref="m9")
+ await platform.command(user_id="U3", command="help", ref="m10")
+
+ async with session_factory() as session:
+ rows = (
+ await session.execute(
+ select(Message.event_type, Message.body).where(
+ Message.room_id == bridge.room_id,
+ Message.event_type.in_(
+ ["m.room.message", "com.switch.command"]
+ ),
+ )
+ )
+ ).all()
+ assert sorted(event_type for event_type, _ in rows) == [
+ "com.switch.command",
+ "m.room.message",
+ ]
+
+ async def test_a_reply_is_still_posted_once(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ processes: list[_Process],
+ platform: _Platform,
+ bridge: _Bridge,
+ ) -> None:
+ await _loud(session_factory, bridge)
+ await _boot_two(postgres_url, session_factory, processes)
+
+ await _say(session_factory, bridge, "relayed by the holder alone")
+
+ assert await _posted(platform, 1) == ["relayed by the holder alone"]
+
+
+async def _loud(
+ session_factory: async_sessionmaker[AsyncSession], bridge: _Bridge
+) -> None:
+ async with session_factory() as session:
+ await session.execute(
+ update(CollaborationBridge)
+ .where(CollaborationBridge.id == bridge.bridge_id)
+ .values(type=LOUD_CHAT)
+ )
+ await session.commit()
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_type_registry.py b/core/tests/switch_core/bridges/collaboration/test_bridge_type_registry.py
index 47e7faf72..55e7c34df 100644
--- a/core/tests/switch_core/bridges/collaboration/test_bridge_type_registry.py
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_type_registry.py
@@ -29,6 +29,7 @@ from switch_core.bridges.collaboration.telegram.adapter import (
TelegramConnectionConfig,
)
from switch_core.keys import Keyring
+from tests.ownership import SoleOwner
def _service() -> CollaborationBridgeLifecycleService:
@@ -59,6 +60,8 @@ def _service() -> CollaborationBridgeLifecycleService:
session_activity_listener=None, # type: ignore[arg-type]
session_activity_service=None, # type: ignore[arg-type]
connections=None, # type: ignore[arg-type]
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(None),
)
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_unknown_member_first_message.py b/core/tests/switch_core/bridges/collaboration/test_bridge_unknown_member_first_message.py
index 05650173f..dda2aee4f 100644
--- a/core/tests/switch_core/bridges/collaboration/test_bridge_unknown_member_first_message.py
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_unknown_member_first_message.py
@@ -47,6 +47,7 @@ class _FakeHumanActor:
"""Stands in for a Actor human actor whose join lands after the invite."""
def __init__(self) -> None:
+ self.client_id = "alice-client"
self.transport_user_id = "@ext_alice:switch.local"
self._joined = asyncio.Event()
self.sent: list[tuple[str, str]] = []
@@ -194,6 +195,7 @@ async def test_first_message_from_app_sender_is_relayed() -> None:
return await CollaborationCore._ensure_human_in_room(inner, **kwargs) # type: ignore[arg-type]
bridge = SimpleNamespace(
+ _bridge_id="bridge-1",
_repair_placeholder_username=_noop_repair,
_is_registered_agent=_is_registered_agent,
_ensure_human_in_room=_ensure_human_in_room,
@@ -240,6 +242,7 @@ async def test_first_message_from_unknown_member_is_relayed() -> None:
relayed.append((kwargs["transport_event_id"], kwargs["external_post_id"]))
bridge = SimpleNamespace(
+ _bridge_id="bridge-1",
_repair_placeholder_username=_noop_repair,
_is_registered_agent=_is_registered_agent,
_ensure_human_in_room=_ensure_human_in_room,
diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_workspace_claims.py b/core/tests/switch_core/bridges/collaboration/test_bridge_workspace_claims.py
index 2ce78642e..be4bfe55d 100644
--- a/core/tests/switch_core/bridges/collaboration/test_bridge_workspace_claims.py
+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_workspace_claims.py
@@ -41,6 +41,7 @@ from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
from switch_core.db.stores.room_store import RoomStore
from switch_core.tenant_context import tenant_scope
+from tests.ownership import SoleOwner
_SLACK = {"bot_token": "xoxb-test", "app_token": "xapp-test", "workspace_id": "T1"}
_DISCORD = {"bot_token": "discord-test", "guild_id": "111"}
@@ -71,6 +72,8 @@ def _service(
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(session_factory),
)
service.register_adapter("slack", SlackAdapter, SlackConnectionConfig)
service.register_adapter("discord", DiscordAdapter, DiscordConnectionConfig)
diff --git a/core/tests/switch_core/bridges/collaboration/test_command_thread_mapping.py b/core/tests/switch_core/bridges/collaboration/test_command_thread_mapping.py
index b09f7aa18..4dde39256 100644
--- a/core/tests/switch_core/bridges/collaboration/test_command_thread_mapping.py
+++ b/core/tests/switch_core/bridges/collaboration/test_command_thread_mapping.py
@@ -66,6 +66,7 @@ def _fake_bridge(
recorded.append(kwargs)
bridge = SimpleNamespace(
+ _bridge_id="bridge-1",
_channel_to_room={"chan-1": ("room-uuid", "!matrix:switch.local")},
_adapter=SimpleNamespace(translate_inbound=translate),
_ensure_human_in_room=_ensure_human_in_room,
@@ -218,6 +219,7 @@ class TestCommandResultThreadingRace:
return SimpleNamespace(
transport_user_id="@human actor:switch.local",
+ client_id="human-actor-1",
send_event=_send_event,
)
diff --git a/core/tests/switch_core/bridges/collaboration/test_install_claim.py b/core/tests/switch_core/bridges/collaboration/test_install_claim.py
index 33e02a5ad..3daae17d4 100644
--- a/core/tests/switch_core/bridges/collaboration/test_install_claim.py
+++ b/core/tests/switch_core/bridges/collaboration/test_install_claim.py
@@ -851,6 +851,81 @@ class TestProposals:
)
+class TestAnsweredThroughAnotherProcess:
+ """A claim's proposal is posted by whichever switch-core process the claim
+ reached, and the admin's answer may reach any other: which chat it was put
+ to is read from the state's row, not held by the process that put it."""
+
+ def _another_process(
+ self, harness: RLSHarness, fixture: _Fixture
+ ) -> MessagingInstallService:
+ installers = MessagingInstallerRegistry()
+ installers.register(fixture.installer)
+ return MessagingInstallService(
+ session_factory=harness.restricted,
+ store=MessagingInstallStore(),
+ receipts=MessagingEventReceiptStore(),
+ installers=installers,
+ lifecycle=fixture.lifecycle, # type: ignore[arg-type]
+ users=UserStore(),
+ rooms=fixture.rooms,
+ public_origin=_ORIGIN,
+ keyring=_KEYRING,
+ )
+
+ async def test_connect_reaching_another_process_installs_the_chat(
+ self, rls_harness: RLSHarness
+ ) -> None:
+ fixture = await _fixture(rls_harness)
+ token = await _link(
+ rls_harness.restricted, fixture, fixture.tenant_a, fixture.admin_a
+ )
+ await _ask(fixture, token, "-1001")
+
+ installed = await self._another_process(rls_harness, fixture).connect(
+ platform=_PLATFORM,
+ claim=InstallClaim(token=token, grant=_grant("-1001"), claimant="42"),
+ )
+
+ assert installed.external_workspace_id == "-1001"
+ assert [
+ i.external_workspace_id for i in await _active_installs(rls_harness)
+ ] == ["-1001"]
+
+ async def test_an_answer_from_another_chat_is_refused_there_too(
+ self, rls_harness: RLSHarness
+ ) -> None:
+ fixture = await _fixture(rls_harness)
+ token = await _link(
+ rls_harness.restricted, fixture, fixture.tenant_a, fixture.admin_a
+ )
+ await _ask(fixture, token, "-1001")
+
+ with pytest.raises(MessagingInstallStateError):
+ await self._another_process(rls_harness, fixture).connect(
+ platform=_PLATFORM,
+ claim=InstallClaim(token=token, grant=_grant("-1002"), claimant="42"),
+ )
+ assert await _active_installs(rls_harness) == []
+
+ async def test_a_repeated_claim_is_recognised_by_another_process(
+ self, rls_harness: RLSHarness
+ ) -> None:
+ """Telegram sends the claim twice and the copies land on different
+ processes: the second is the same question, not an expired link."""
+ fixture = await _fixture(rls_harness)
+ token = await _link(
+ rls_harness.restricted, fixture, fixture.tenant_a, fixture.admin_a
+ )
+ await _ask(fixture, token, "-1001")
+
+ with pytest.raises(install_service_module.InstallClaimRepeated):
+ await self._another_process(rls_harness, fixture).propose(
+ platform=_PLATFORM,
+ claim=InstallClaim(token=token, grant=_grant("-1001"), claimant="42"),
+ )
+
+
class TestTheRoute:
"""A claim arrives as an ordinary event and only asks; an admin's answer,
another event, is what connects the chat, and is then delivered as one."""
diff --git a/core/tests/switch_core/bridges/collaboration/test_lifecycle_callback_endpoint.py b/core/tests/switch_core/bridges/collaboration/test_lifecycle_callback_endpoint.py
index 3e604b7db..49e2acf32 100644
--- a/core/tests/switch_core/bridges/collaboration/test_lifecycle_callback_endpoint.py
+++ b/core/tests/switch_core/bridges/collaboration/test_lifecycle_callback_endpoint.py
@@ -58,7 +58,7 @@ class _FailingClient:
client_id = "bridge-client"
- async def start(self) -> None:
+ async def start(self, progress: object) -> None:
raise RuntimeError("the room client is down")
@@ -144,9 +144,14 @@ async def test_a_bridge_that_crashes_takes_its_place_with_it(
try:
assert await _post(port, bridge_id) == 200
+ # What `start` leaves for the task of a bridge this process holds: its
+ # workspace consumer, and the cursors it reads its rooms under.
+ failing = _FailingClient()
+ service._workspace_consumers[bridge_id] = failing # type: ignore[assignment]
+ service._workspace_cursors[bridge_id] = object() # type: ignore[assignment]
await type(service)._run_bridge(
- service, bridge_id, tenant, _StubCore(), _FailingClient()
- )
+ service, bridge_id, tenant, _StubCore(), failing
+ ) # type: ignore[arg-type]
assert await _post(port, bridge_id) == 404
finally:
diff --git a/core/tests/switch_core/bridges/collaboration/test_lifecycle_config_edit.py b/core/tests/switch_core/bridges/collaboration/test_lifecycle_config_edit.py
index 388959439..95aec3cf2 100644
--- a/core/tests/switch_core/bridges/collaboration/test_lifecycle_config_edit.py
+++ b/core/tests/switch_core/bridges/collaboration/test_lifecycle_config_edit.py
@@ -21,6 +21,7 @@ from switch_core.bridges.collaboration.teams.adapter import (
TeamsAdapter,
TeamsConnectionConfig,
)
+from tests.ownership import SoleOwner
_SHARED = {"event_delivery": "shared", "tenant_id": "org-1"}
@@ -42,6 +43,8 @@ def _service() -> CollaborationBridgeLifecycleService:
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(MagicMock()),
)
service.register_adapter("teams", TeamsAdapter, TeamsConnectionConfig)
service.register_adapter(
diff --git a/core/tests/switch_core/bridges/collaboration/test_lifecycle_reconcile.py b/core/tests/switch_core/bridges/collaboration/test_lifecycle_reconcile.py
new file mode 100644
index 000000000..7dcd11be2
--- /dev/null
+++ b/core/tests/switch_core/bridges/collaboration/test_lifecycle_reconcile.py
@@ -0,0 +1,111 @@
+"""A re-check starts a bridge only from a version of its row not tried here.
+
+Every process re-reads the bridges' rows when a `bridge.changed` signal says
+one changed and on a periodic re-check (`bridge_signals.py`). What it starts or
+restarts is decided by whether it has started the bridge from that version of
+the row already. So a bridge that crashed or failed to start stays down until
+it is edited or restarted, as it does with one process, rather than being
+tried again every re-check; and an edit made elsewhere is applied here.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+
+import pytest
+from sqlalchemy import update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.bridges.collaboration import lifecycle_service as lifecycle_module
+from switch_core.db.models import CollaborationBridge
+from tests.switch_core.bridges.collaboration.test_lifecycle_tenant_binding import (
+ _make_bridge,
+ _make_tenant,
+ _service,
+ _StubAdapter,
+ _StubConfig,
+)
+
+
+async def _bridge(session_factory: async_sessionmaker[AsyncSession]) -> str:
+ tenant = f"tenant-{uuid.uuid4().hex[:8]}"
+ async with session_factory() as session:
+ await _make_tenant(session, tenant)
+ bridge_id, _ = await _make_bridge(session, tenant_id=tenant)
+ await session.commit()
+ return bridge_id
+
+
+async def _edit(
+ session_factory: async_sessionmaker[AsyncSession], bridge_id: str
+) -> None:
+ async with session_factory() as session:
+ await session.execute(
+ update(CollaborationBridge)
+ .where(CollaborationBridge.id == bridge_id)
+ .values(display_name="renamed")
+ )
+ await session.commit()
+
+
+async def test_a_running_bridge_is_left_alone_until_its_row_changes(
+ session_factory: async_sessionmaker[AsyncSession],
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ # The stand-in task below runs until cancelled rather than until its
+ # consumer stops, so a stop need not give it the grace a real one gets.
+ monkeypatch.setattr(lifecycle_module, "STOP_GRACE_SECONDS", 0.01)
+ bridge_id = await _bridge(session_factory)
+ service = _service(session_factory)
+ service.register_adapter("mattermost", _StubAdapter, _StubConfig)
+ starts: list[str] = []
+
+ async def run(bridge_id: str, *_: object) -> None:
+ # Running until stopped, as a bridge's task does.
+ starts.append(bridge_id)
+ await asyncio.Event().wait()
+
+ service._run_bridge = run # type: ignore[method-assign]
+ try:
+ await service.reconcile_all()
+ await asyncio.sleep(0)
+ assert starts == [bridge_id]
+
+ await service.reconcile_all()
+ await asyncio.sleep(0)
+ assert starts == [bridge_id]
+
+ await _edit(session_factory, bridge_id)
+ await service.reconcile_bridge(bridge_id)
+ await asyncio.sleep(0)
+ assert starts == [bridge_id, bridge_id]
+ finally:
+ await service.stop_all()
+
+
+async def test_a_bridge_that_failed_to_start_is_tried_again_only_once_edited(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ """No adapter is registered for the bridge's platform, so every start
+ fails before anything runs."""
+ bridge_id = await _bridge(session_factory)
+ service = _service(session_factory)
+ attempts: list[str] = []
+ start = service._start_bridge
+
+ async def counted(bridge_id: str) -> None:
+ attempts.append(bridge_id)
+ await start(bridge_id)
+
+ service._start_bridge = counted # type: ignore[method-assign]
+ try:
+ await service.reconcile_all()
+ await service.reconcile_all()
+ assert attempts == [bridge_id]
+
+ await _edit(session_factory, bridge_id)
+ await service.reconcile_all()
+ assert attempts == [bridge_id, bridge_id]
+ finally:
+ await service.stop_all()
diff --git a/core/tests/switch_core/bridges/collaboration/test_lifecycle_remove.py b/core/tests/switch_core/bridges/collaboration/test_lifecycle_remove.py
index cbad03d23..271c7a256 100644
--- a/core/tests/switch_core/bridges/collaboration/test_lifecycle_remove.py
+++ b/core/tests/switch_core/bridges/collaboration/test_lifecycle_remove.py
@@ -26,6 +26,7 @@ from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStore
from switch_core.db.stores.external_user_store import ExternalUserStore
from switch_core.db.stores.room_store import RoomStore
+from tests.ownership import SoleOwner
class _ClientLifecycle:
@@ -90,6 +91,8 @@ def _service(
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(session_factory),
)
diff --git a/core/tests/switch_core/bridges/collaboration/test_lifecycle_tenant_binding.py b/core/tests/switch_core/bridges/collaboration/test_lifecycle_tenant_binding.py
index f7120f173..751d77012 100644
--- a/core/tests/switch_core/bridges/collaboration/test_lifecycle_tenant_binding.py
+++ b/core/tests/switch_core/bridges/collaboration/test_lifecycle_tenant_binding.py
@@ -44,6 +44,16 @@ from switch_core.db.stores.room_store import RoomStore
from switch_core.keys import Keyring
from switch_core.tenant_context import current_tenant_id, tenant_scope
from tests.conftest import RLSHarness
+from tests.ownership import SoleOwner
+
+
+async def _boot(service: CollaborationBridgeLifecycleService) -> None:
+ """Start every bridge as `main` does at boot: each is wanted, the bridges
+ this process holds start with the ownership's first balance, and the rest
+ start quietly after it."""
+ await service.start_all()
+ await service._owned.settle()
+ await service.start_quiet_copies()
def _service(
@@ -72,6 +82,8 @@ def _service(
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(session_factory),
)
@@ -156,7 +168,7 @@ async def test_start_all_binds_nothing_around_starting_a_bridge(
service.start = _fake_start # type: ignore[method-assign]
assert current_tenant_id() is None
- await service.start_all()
+ await _boot(service)
assert current_tenant_id() is None
assert seen == {bridge_a: None, bridge_b: None}
@@ -187,7 +199,7 @@ async def test_a_failing_bridge_leaves_nothing_bound_for_the_next_one(
service.start = _fake_start # type: ignore[method-assign]
- await service.start_all()
+ await _boot(service)
assert seen == {bridge_a: None, bridge_b: None}
assert current_tenant_id() is None
@@ -267,12 +279,16 @@ async def test_the_bridge_task_unbinds_and_then_binds_per_unit_of_work(
class _Client:
client_id = ""
- async def start(self) -> None:
+ async def start(self, progress: object) -> None:
seen["client"] = current_tenant_id()
workspace_consumer = _Client()
workspace_consumer.client_id = client_id
+ # What `start` leaves for the task: the workspace consumer, and the
+ # cursors it reads its rooms under.
+ service._workspace_consumers[bridge_id] = workspace_consumer # type: ignore[assignment]
+ service._workspace_cursors[bridge_id] = MagicMock()
leaked: list[str | None] = []
with tenant_scope(caller_tenant):
await service._run_bridge(
diff --git a/core/tests/switch_core/bridges/collaboration/test_mattermost_bot_sockets.py b/core/tests/switch_core/bridges/collaboration/test_mattermost_bot_sockets.py
new file mode 100644
index 000000000..6f7ece58c
--- /dev/null
+++ b/core/tests/switch_core/bridges/collaboration/test_mattermost_bot_sockets.py
@@ -0,0 +1,103 @@
+"""A Mattermost bridge that stops closes its bots' sockets for good.
+
+Each agent bot's socket runs in a thread of its own that reconnects whenever
+the connection drops. A stopped bridge used to leave those threads running for
+the life of the process, so a bridge handed to another process kept sockets
+here that Mattermost went on handing every post to.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import threading
+from collections.abc import Callable
+from typing import Any
+
+from switch_core.bridges.collaboration.mattermost.adapter import MattermostAdapter
+from tests.switch_core.bridges.collaboration.test_bridge_agent_display_names import (
+ _agent,
+ _bridge,
+ _mattermost_identity_adapter,
+ _run,
+)
+
+
+class _Bot:
+ """A bot driver whose socket stays open until it is disconnected."""
+
+ def __init__(self) -> None:
+ self.websocket: object | None = None
+ self.connects = 0
+ self.disconnected = threading.Event()
+
+ def login(self) -> None:
+ return None
+
+ async def init_websocket(self, handler: Any) -> None:
+ self.websocket = object()
+ self.connects += 1
+ while not self.disconnected.is_set():
+ await asyncio.sleep(0.01)
+
+ def disconnect(self) -> None:
+ self.disconnected.set()
+
+
+def _adapter() -> tuple[MattermostAdapter, _Bot]:
+ bridge = _bridge(_agent("switchdev", "Switch Dev"))
+ adapter, _ = _mattermost_identity_adapter(bridge, [], "full_name")
+ bot = _Bot()
+ adapter._create_driver = lambda **_kwargs: bot # type: ignore[assignment,method-assign,return-value]
+ return adapter, bot
+
+
+def _socket_thread(agent_name: str, before: set[threading.Thread]) -> threading.Thread:
+ """The socket thread opened for `agent_name` since `before`: other tests'
+ fakes leave threads of the same name parked for good."""
+ return next(
+ thread
+ for thread in threading.enumerate()
+ if thread.name == f"mm-ws-{agent_name}" and thread not in before
+ )
+
+
+async def _until(check: Callable[[], bool]) -> None:
+ async with asyncio.timeout(5):
+ while not check():
+ await asyncio.sleep(0.01)
+
+
+def _closed_by(close: Callable[[MattermostAdapter], Any]) -> tuple[_Bot, bool]:
+ """Open the agent's socket, close it the way `close` does, and say
+ whether its thread has finished."""
+ adapter, bot = _adapter()
+
+ async def go() -> bool:
+ adapter._main_loop = asyncio.get_running_loop()
+ before = set(threading.enumerate())
+ await adapter.create_agent_identity("switchdev", "a Switch agent")
+ thread = _socket_thread("switchdev", before)
+ await _until(lambda: bot.connects == 1)
+
+ await close(adapter)
+ await asyncio.to_thread(thread.join, 5)
+ return thread.is_alive()
+
+ alive = _run(go())
+ return bot, alive
+
+
+def test_stopping_the_adapter_closes_each_socket_for_good() -> None:
+ bot, alive = _closed_by(lambda adapter: adapter.stop())
+
+ assert bot.disconnected.is_set()
+ assert not alive
+ assert bot.connects == 1
+
+
+def test_removing_an_agent_closes_its_socket_for_good() -> None:
+ bot, alive = _closed_by(lambda adapter: adapter.remove_agent_identity("switchdev"))
+
+ assert bot.disconnected.is_set()
+ assert not alive
+ assert bot.connects == 1
diff --git a/core/tests/switch_core/bridges/collaboration/test_room_wide_mention_rendering.py b/core/tests/switch_core/bridges/collaboration/test_room_wide_mention_rendering.py
index fd95b829b..4092b4637 100644
--- a/core/tests/switch_core/bridges/collaboration/test_room_wide_mention_rendering.py
+++ b/core/tests/switch_core/bridges/collaboration/test_room_wide_mention_rendering.py
@@ -36,6 +36,9 @@ from switch_core.bridges.collaboration.telegram.adapter import (
)
from switch_core.room_wide_mention import room_wide_mention_content
from switch_core.transport import InboundMessage, RoomRef
+from tests.switch_core.bridges.collaboration.outbound_fakes import (
+ posting_without_a_database,
+)
ZWSP = "\u200b"
@@ -293,7 +296,7 @@ def _bridge(adapter: _Recording) -> CollaborationCore:
core._find_channel = lambda **_kwargs: "C1" # type: ignore[assignment]
core._channel_to_room = {"C1": ("room-uuid", "!r:switch.local")} # type: ignore[assignment]
core._room_tenant = _tenant # type: ignore[assignment]
- core._record_message_map = _noop # type: ignore[assignment]
+ posting_without_a_database(core)
core._move_indicator_for_sender = _noop # type: ignore[assignment]
core._outbound_thread_root_ref = _noop # type: ignore[assignment]
return core
diff --git a/core/tests/switch_core/bridges/collaboration/test_slack_socket_ack.py b/core/tests/switch_core/bridges/collaboration/test_slack_socket_ack.py
new file mode 100644
index 000000000..f0a54fcd6
--- /dev/null
+++ b/core/tests/switch_core/bridges/collaboration/test_slack_socket_ack.py
@@ -0,0 +1,157 @@
+"""A Socket Mode event is stored before Slack is told it arrived.
+
+Slack sends an event again when it is not acknowledged within three seconds,
+so acknowledging only once the event is stored means a process that stops in
+between is sent the event again instead of losing it. Storing that runs long
+is acknowledged in time anyway, storing that fails is not acknowledged, so
+Slack sends the event again, and the copy is stored, and a slash command or
+button press, which Slack never sends again, is acknowledged first as before.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Any
+from unittest.mock import AsyncMock
+
+import pytest
+from slack_sdk.socket_mode.request import SocketModeRequest
+
+from switch_core.bridges.collaboration.models import InboundMessage
+from switch_core.bridges.collaboration.slack import adapter as slack_module
+from switch_core.bridges.collaboration.slack.adapter import (
+ SlackAdapter,
+ SlackConnectionConfig,
+)
+
+from .slack_fakes import FakeWebClient
+
+
+def _adapter() -> SlackAdapter:
+ slack = SlackAdapter(
+ config=SlackConnectionConfig(
+ bot_token="test", app_token="test", workspace_id="T1"
+ )
+ )
+ slack._web_client = FakeWebClient() # type: ignore[assignment]
+ return slack
+
+
+def _slack(order: list[str], stored: asyncio.Event | None = None) -> SlackAdapter:
+ slack = _adapter()
+
+ async def dispatch(*, envelope_type: str, payload: dict[str, Any]) -> None:
+ if stored is not None:
+ await stored.wait()
+ order.append("stored")
+
+ slack.dispatch_event = dispatch # type: ignore[method-assign]
+ return slack
+
+
+def _socket(order: list[str]) -> Any:
+ socket = AsyncMock()
+ socket.send_socket_mode_response.side_effect = lambda _response: order.append(
+ "acknowledged"
+ )
+ return socket
+
+
+def _envelope(kind: str) -> SocketModeRequest:
+ return SocketModeRequest(type=kind, envelope_id="envelope-1", payload={})
+
+
+def _posted(envelope_id: str) -> SocketModeRequest:
+ """Someone's post in a channel, as Slack delivers it, the same each time."""
+ return SocketModeRequest(
+ type="events_api",
+ envelope_id=envelope_id,
+ payload={
+ "event": {
+ "type": "message",
+ "channel": "C1",
+ "channel_type": "channel",
+ "ts": "1700000000.000100",
+ "user": "U1",
+ "text": "hello",
+ }
+ },
+ )
+
+
+async def test_an_event_is_acknowledged_once_it_is_stored() -> None:
+ order: list[str] = []
+ slack = _slack(order)
+
+ await slack._handle_socket_event(_socket(order), _envelope("events_api"))
+
+ assert order == ["stored", "acknowledged"]
+
+
+async def test_an_event_slow_to_store_is_acknowledged_in_time_and_still_stored(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(slack_module, "SOCKET_ACK_WITHIN_SECONDS", 0.05)
+ order: list[str] = []
+ stored = asyncio.Event()
+ slack = _slack(order, stored)
+
+ handling = asyncio.create_task(
+ slack._handle_socket_event(_socket(order), _envelope("events_api"))
+ )
+ await asyncio.sleep(0.2)
+ assert order == ["acknowledged"]
+
+ stored.set()
+ await handling
+ assert order == ["acknowledged", "stored"]
+
+
+async def test_an_event_that_fails_to_store_is_not_acknowledged() -> None:
+ order: list[str] = []
+ slack = _slack(order)
+
+ async def failing(*, envelope_type: str, payload: dict[str, Any]) -> None:
+ raise RuntimeError("the database refused the post")
+
+ slack.dispatch_event = failing # type: ignore[method-assign]
+
+ with pytest.raises(RuntimeError):
+ await slack._handle_socket_event(_socket(order), _envelope("events_api"))
+
+ assert order == []
+
+
+async def test_an_event_that_failed_to_store_is_stored_when_sent_again() -> None:
+ """Only the process that holds the bridge has its socket, so the copy
+ Slack sends of an event that failed to store reaches the same adapter,
+ which must not skip it as a message it has seen."""
+ order: list[str] = []
+ slack = _adapter()
+ refusals = [RuntimeError("the database refused the post")]
+
+ async def on_message(message: InboundMessage) -> None:
+ if refusals:
+ raise refusals.pop()
+ order.append("stored")
+
+ slack._on_message = on_message
+ socket = _socket(order)
+
+ with pytest.raises(RuntimeError):
+ await slack._handle_socket_event(socket, _posted("envelope-1"))
+ await slack._handle_socket_event(socket, _posted("envelope-2"))
+
+ assert order == ["stored", "acknowledged"]
+
+
[email protected]("kind", ["slash_commands", "interactive"])
+async def test_a_command_or_press_is_acknowledged_before_it_is_handled(
+ kind: str,
+) -> None:
+ order: list[str] = []
+ slack = _slack(order)
+
+ await slack._handle_socket_event(_socket(order), _envelope(kind))
+
+ assert order == ["acknowledged", "stored"]
diff --git a/core/tests/switch_core/bridges/collaboration/test_teams_zero_config.py b/core/tests/switch_core/bridges/collaboration/test_teams_zero_config.py
index f3fc88bf9..4849ee3d0 100644
--- a/core/tests/switch_core/bridges/collaboration/test_teams_zero_config.py
+++ b/core/tests/switch_core/bridges/collaboration/test_teams_zero_config.py
@@ -29,6 +29,7 @@ from switch_core.bridges.collaboration.teams.crypto import (
load_certificate_der_b64,
)
from switch_core.tenant_context import tenant_scope
+from tests.ownership import SoleOwner
def _raw_config(**overrides: Any) -> dict[str, Any]:
@@ -325,6 +326,8 @@ def _lifecycle() -> CollaborationBridgeLifecycleService:
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(MagicMock()),
)
service.register_adapter("recording", _RecordingAdapter, _RecordingConfig)
return service
@@ -440,6 +443,8 @@ def _service_with_existing(
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(MagicMock(return_value=session)),
)
service.register_adapter("teams", TeamsAdapter, TeamsConnectionConfig)
return service
@@ -626,6 +631,8 @@ async def test_concurrent_registration_cannot_take_the_same_port_twice(
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(MagicMock(return_value=session)),
)
service.register_adapter("teams", TeamsAdapter, TeamsConnectionConfig)
monkeypatch.setattr(
@@ -699,6 +706,8 @@ async def test_start_refuses_a_second_bridge_already_holding_the_port(
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(MagicMock(return_value=session)),
)
service.register_adapter("teams", TeamsAdapter, TeamsConnectionConfig)
# A bridge already running and holding the port.
diff --git a/core/tests/switch_core/bridges/collaboration/test_telegram_adapter.py b/core/tests/switch_core/bridges/collaboration/test_telegram_adapter.py
index adaa83b48..743d8b06e 100644
--- a/core/tests/switch_core/bridges/collaboration/test_telegram_adapter.py
+++ b/core/tests/switch_core/bridges/collaboration/test_telegram_adapter.py
@@ -1577,6 +1577,27 @@ def test_starting_begins_polling_and_learns_the_bot_id() -> None:
]
+def test_a_quiet_copy_calls_the_bot_api_without_polling() -> None:
+ """Another switch-core process holds the bridge and polls Telegram for it:
+ Telegram hands each update to one poller, so this copy must not take a
+ share. It still knows its bot, for the Bot API calls it serves."""
+ adapter = TelegramAdapter(
+ config=TelegramConnectionConfig(bot_token="token", bot_username=BOT_USERNAME)
+ )
+ adapter.set_listening(False)
+ app = _FakeApplication()
+
+ with patch.object(adapter_module, "ApplicationBuilder", lambda: _FakeBuilder(app)):
+ _run(adapter.start(_noop, _noop, _noop, _noop, _noop))
+ assert app.started is False
+ assert app.updater.polling_kwargs is None
+ assert adapter._bot_user_id == BOT_USER_ID
+ _run(adapter.stop())
+
+ assert app.updater.stopped is False
+ assert app.shut_down is True
+
+
def test_a_second_poller_on_the_same_token_is_named(
caplog: pytest.LogCaptureFixture,
) -> None:
diff --git a/core/tests/switch_core/bridges/collaboration/test_telegram_app_lifecycle.py b/core/tests/switch_core/bridges/collaboration/test_telegram_app_lifecycle.py
index b007c88e8..8fca6a1c2 100644
--- a/core/tests/switch_core/bridges/collaboration/test_telegram_app_lifecycle.py
+++ b/core/tests/switch_core/bridges/collaboration/test_telegram_app_lifecycle.py
@@ -49,6 +49,7 @@ from switch_core.bridges.collaboration.telegram.install import (
from switch_core.db.models import CollaborationBridge
from switch_core.gateway.collaborations import update_bridge
from switch_core.gateway.schemas import BridgeUpdateRequest
+from tests.ownership import SoleOwner
_SECRET = "placeholder-secret"
@@ -420,6 +421,8 @@ def _lifecycle() -> CollaborationBridgeLifecycleService:
session_activity_listener=None, # type: ignore[arg-type]
session_activity_service=None, # type: ignore[arg-type]
connections=None, # type: ignore[arg-type]
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(None),
)
lifecycle.register_adapter("telegram", TelegramAdapter, TelegramConnectionConfig)
return lifecycle
diff --git a/core/tests/switch_core/clients/test_agent_actor.py b/core/tests/switch_core/clients/test_agent_actor.py
new file mode 100644
index 000000000..a8b79bae3
--- /dev/null
+++ b/core/tests/switch_core/clients/test_agent_actor.py
@@ -0,0 +1,92 @@
+"""An agent's actor loads the agent row it stands for when it connects.
+
+An agent reads no room, so it has no consumer, and its actor is what the
+bridges post the agent's replies through, found by agent id. So the actor
+itself has to know its agent: read once when it connects, or handed to it
+from the bulk read at boot.
+"""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.clients.actor import AgentActor, ClientConfig
+from switch_core.db.models import TENANT_ZERO_ID, Agent
+from switch_core.db.stores.client_store import ClientStore
+from tests.routing import make_agent, make_person
+from tests.switch_core.statement_counts import StatementCounts
+
+
+def _actor(
+ session_factory: async_sessionmaker[AsyncSession], client_id: str
+) -> AgentActor:
+ return AgentActor(
+ client_id=client_id,
+ tenant_id=TENANT_ZERO_ID,
+ transport_user_id="@agent:test",
+ display_name="agent",
+ session_factory=session_factory,
+ client_store=ClientStore(),
+ config=ClientConfig(),
+ transport_factory=lambda _actor: MagicMock(connect=_no_op),
+ )
+
+
+async def _no_op() -> None:
+ return None
+
+
+async def test_connecting_loads_the_agent_it_stands_for(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ agent, client = await make_agent(session_factory, "stands-for")
+ actor = _actor(session_factory, client.id)
+
+ await actor.connect()
+
+ assert actor.agent_id == agent.id
+ assert actor.agent.name == "stands-for"
+
+
+async def test_a_client_with_no_agent_row_fails_to_connect(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ """Loud rather than an actor that answers to no agent: the lifecycle
+ counts it crashed and leaves it stopped."""
+ person = await make_person(session_factory, "not-an-agent")
+ actor = _actor(session_factory, person.id)
+
+ with pytest.raises(RuntimeError, match="No agent found for client"):
+ await actor.connect()
+
+
+async def test_a_preloaded_row_needs_no_read(
+ session_factory: async_sessionmaker[AsyncSession],
+ statement_counts: StatementCounts,
+) -> None:
+ agent, client = await make_agent(session_factory, "preloaded")
+ actor = _actor(session_factory, client.id)
+ actor.preload_agent(agent)
+
+ with statement_counts.scope("connect") as scope:
+ await actor.connect()
+
+ assert scope.statements == []
+ assert isinstance(actor.agent, Agent)
+ assert actor.agent_id == agent.id
+
+
+async def test_an_agent_hears_its_rooms_without_waiting_for_its_client(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ """A template run waits for each new agent's client to take events before
+ it posts the kickoff. An agent's rows are decided as they are written,
+ so its client has nothing to start first: the answer is yes at once,
+ even before it connects."""
+ _, client = await make_agent(session_factory, "new-hire")
+ actor = _actor(session_factory, client.id)
+
+ assert await actor.wait_receiving(0)
diff --git a/core/tests/switch_core/clients/test_agent_consumer_attachment_groups.py b/core/tests/switch_core/clients/test_agent_consumer_attachment_groups.py
deleted file mode 100644
index 681b90527..000000000
--- a/core/tests/switch_core/clients/test_agent_consumer_attachment_groups.py
+++ /dev/null
@@ -1,628 +0,0 @@
-from __future__ import annotations
-
-import asyncio
-from contextlib import asynccontextmanager
-from types import SimpleNamespace
-from typing import Any
-
-import pytest
-from sqlalchemy import select
-from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-
-import switch_core.clients.agent_consumer as ac
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.types import AgentEvent
-from switch_core.clients.agent_consumer import AgentConsumer, _GateOutcome
-from switch_core.clients.room_meta import RoomMeta
-from switch_core.db.models import TENANT_ZERO_ID, AgentEventRow
-from switch_core.db.session_scope import tenant_session
-from switch_core.transport import InboundMedia, RoomRef
-from tests.db_buffer import seed_agent
-
-AGENT = "agent-1"
-
-
[email protected](autouse=True)
-async def _agent_row(session_factory: async_sessionmaker[AsyncSession]) -> None:
- """The agent the consumer queues for, so its events have a row to belong to."""
- await seed_agent(session_factory, AGENT)
-
-
-def _media_event(
- *,
- body: str,
- filename: str | None = None,
- mimetype: str = "image/png",
- msgtype: str = "m.image",
- group: dict[str, Any] | None = None,
- event_id: str = "$evt",
- seq: int = 0,
-) -> InboundMedia:
- content: dict[str, Any] = {
- "msgtype": msgtype,
- "body": body,
- "url": "mxc://s/abc",
- "info": {"mimetype": mimetype, "size": 5},
- "sender_name": "alice",
- }
- if filename is not None:
- content["filename"] = filename
- if group is not None:
- content["com.switch.attachment_group"] = group
- return InboundMedia(
- room_id="!room:s",
- event_id=event_id,
- sender="@alice:s",
- timestamp=1700000000000,
- content=content,
- body=body,
- sender_name="alice",
- msgtype=msgtype,
- uri="mxc://s/abc",
- filename=filename,
- mimetype=mimetype,
- size=5,
- group=group,
- seq=seq,
- )
-
-
-@asynccontextmanager
-async def _session_factory(): # type: ignore[no-untyped-def]
- yield object()
-
-
-class _FakeMessageTelemetry:
- def __init__(self) -> None:
- self.enabled = True
- self.addressed: list[dict[str, Any]] = []
-
- def agent_addressed(self, **kwargs: Any) -> None:
- self.addressed.append(kwargs)
-
-
-async def _events(client: SimpleNamespace) -> list[AgentEvent]:
- """What the consumer queued for its agent, read back from the db-buffer.
-
- Only what is for the agent is queued; the rest of the room is read from
- the room's log by whoever wants it. So a test about what a post becomes
- addresses the post to the agent, in its caption (part 0's body).
- """
- buffer: EventBuffer = client._event_buffer
- return [item.event for item in await buffer.read_from(TENANT_ZERO_ID, AGENT, 0)]
-
-
-def _fake_client(buffer: EventBuffer) -> SimpleNamespace:
- meta = RoomMeta(room_id="room-1", name="Room", bridge_id="bridge-1")
-
- async def _resolve_room_meta(_transport_room_id: str) -> RoomMeta:
- return meta
-
- async def _addressed(event: Any, _meta: RoomMeta) -> bool:
- # Mirrors production, where addressing is read off the message text.
- # Standing in a constant `True` here is what let the coalescing bug
- # through: every part looked addressed, including the caption-less ones
- # that in reality address nobody.
- return f"@{ns.agent.name}" in getattr(event, "body", "")
-
- async def _fresh_agent(_session: Any) -> Any:
- return ns.agent
-
- async def _gate_addressed(
- _session: Any, _agent: Any, _event: Any, _meta: Any
- ) -> _GateOutcome:
- return _GateOutcome(addressed=True, refusal=None)
-
- async def _is_available(_session: Any, _agent: Any, _room_id: str) -> bool:
- ns.availability_checks += 1
- return True
-
- ns = SimpleNamespace(
- agent=SimpleNamespace(id=AGENT, name="agent-a", metadata_=None),
- tenant_id=TENANT_ZERO_ID,
- session_factory=_session_factory,
- _event_buffer=buffer,
- _message_telemetry=_FakeMessageTelemetry(),
- _triggered_by_auto_reply=AgentConsumer._triggered_by_auto_reply,
- _attachment_groups={},
- _attachment_group_timers={},
- _resolve_room_meta=_resolve_room_meta,
- _addressed=_addressed,
- _fresh_agent=_fresh_agent,
- _gate_addressed=_gate_addressed,
- _is_available=_is_available,
- availability_checks=0,
- )
- ns._emit_media = AgentConsumer._emit_media.__get__(ns)
- ns._report_addressed = AgentConsumer._report_addressed.__get__(ns)
- ns._schedule_attachment_group_flush = (
- AgentConsumer._schedule_attachment_group_flush.__get__(ns)
- )
- ns._cancel_attachment_group_flush = (
- AgentConsumer._cancel_attachment_group_flush.__get__(ns)
- )
- ns._flush_incomplete_attachment_group = (
- AgentConsumer._flush_incomplete_attachment_group.__get__(ns)
- )
- return ns
-
-
-def _room() -> RoomRef:
- return RoomRef(room_id="!room:s")
-
-
-async def test_grouped_media_coalesces_into_one_event(db_buffer: EventBuffer) -> None:
- client = _fake_client(db_buffer)
- parts = [
- ("cat.png", "image/png", "m.image"),
- ("notes.md", "text/markdown", "m.file"),
- ("data.csv", "text/csv", "m.file"),
- ]
-
- for index, (name, mimetype, msgtype) in enumerate(parts):
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a three files" if index == 0 else name,
- filename=name if index == 0 else None,
- mimetype=mimetype,
- msgtype=msgtype,
- event_id=f"$part-{index}",
- group={"id": "grp-1", "index": index, "total": 3},
- ),
- )
- if index < 2:
- assert await _events(client) == []
-
- assert len(await _events(client)) == 1
- payload = (await _events(client))[0].payload
- assert [a.filename for a in payload.attachments] == [
- "cat.png",
- "notes.md",
- "data.csv",
- ]
- assert payload.body == "@agent-a three files"
- assert client._attachment_groups == {}
- assert client._attachment_group_timers == {}
-
-
-async def test_grouped_media_coalesces_out_of_order(db_buffer: EventBuffer) -> None:
- client = _fake_client(db_buffer)
- order = [1, 0, 2]
- names = {0: "a.png", 1: "b.png", 2: "c.png"}
-
- for index in order:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a captioned" if index == 0 else names[index],
- filename=names[index] if index == 0 else None,
- event_id=f"$part-{index}",
- group={"id": "grp-2", "index": index, "total": 3},
- ),
- )
-
- assert len(await _events(client)) == 1
- payload = (await _events(client))[0].payload
- # Sorted by group index, not arrival order.
- assert [a.filename for a in payload.attachments] == ["a.png", "b.png", "c.png"]
- assert payload.body == "@agent-a captioned"
- assert client._attachment_groups == {}
- assert client._attachment_group_timers == {}
-
-
-async def test_a_group_is_kept_as_the_one_message_it_was_posted_as(
- session_factory: async_sessionmaker[AsyncSession], db_buffer: EventBuffer
-) -> None:
- """Each part is a row of its own in the room's log, and the conversation
- read from there leaves out a row whose message the agent already has, by
- the id the message is known by: for a part, its group's. So the group is
- kept under the group's id, at the part furthest on in the room. Kept under
- one part's id, the other parts would reach the agent again as
- conversation."""
- client = _fake_client(db_buffer)
- # The part that completes the group is not the furthest on.
- for index, seq in [(0, 11), (2, 13), (1, 12)]:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a three" if index == 0 else f"f{index}.png",
- filename="f0.png" if index == 0 else None,
- event_id=f"$part-{index}",
- group={"id": "grp-key", "index": index, "total": 3},
- seq=seq,
- ),
- )
-
- async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
- kept = (
- await session.execute(
- select(AgentEventRow.room_seq, AgentEventRow.message_key).where(
- AgentEventRow.agent_id == AGENT
- )
- )
- ).all()
- assert [tuple(row) for row in kept] == [(13, "grp-key")]
-
-
-async def test_ungrouped_media_emits_immediately(db_buffer: EventBuffer) -> None:
- client = _fake_client(db_buffer)
-
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(body="@agent-a a cat", filename="cat.png", mimetype="image/png"),
- )
-
- assert len(await _events(client)) == 1
- payload = (await _events(client))[0].payload
- assert [a.filename for a in payload.attachments] == ["cat.png"]
- assert payload.body == "@agent-a a cat"
- assert client._attachment_groups == {}
- assert client._attachment_group_timers == {}
-
-
-async def test_incomplete_group_flushes_with_disclosed_notice(
- db_buffer: EventBuffer,
-) -> None:
- original = ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS
- ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS = 0.01
- try:
- client = _fake_client(db_buffer)
- for index, name in [(0, "cat.png"), (1, "notes.md")]:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a two of three" if index == 0 else name,
- filename=name if index == 0 else None,
- event_id=f"$part-{index}",
- group={"id": "grp-3", "index": index, "total": 3},
- ),
- )
- assert await _events(client) == []
-
- await asyncio.sleep(0.15)
- finally:
- ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS = original
-
- assert len(await _events(client)) == 1
- payload = (await _events(client))[0].payload
- assert [a.filename for a in payload.attachments] == ["cat.png", "notes.md"]
- assert "two of three" in payload.body
- assert "incomplete attachment group: 2 of 3" in payload.body
- # No leak: the buffer and its timer are gone once flushed.
- assert client._attachment_groups == {}
- assert client._attachment_group_timers == {}
-
-
-async def test_group_is_anchored_on_part_zero_not_the_completing_part(
- db_buffer: EventBuffer,
-) -> None:
- """The coalesced message must carry part 0's event id, so a reply threads
- off the canonical first event rather than whichever part landed last."""
- client = _fake_client(db_buffer)
- # Part 0 arrives FIRST, so a later part completes the group — otherwise the
- # completing event happens to be part 0 and the assertion proves nothing.
- for index, name in [(0, "a.png"), (1, "b.md"), (2, "c.csv")]:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a three" if index == 0 else name,
- filename=name if index == 0 else None,
- event_id=f"$part-{index}",
- group={"id": "grp-anchor", "index": index, "total": 3},
- ),
- )
-
- assert len(await _events(client)) == 1
- # Part 2 completed the group, but part 0 anchors the payload.
- assert (await _events(client))[0].payload.message_id == "$part-0"
-
-
-async def test_incomplete_group_is_anchored_on_part_zero(
- db_buffer: EventBuffer,
-) -> None:
- original = ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS
- ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS = 0.01
- try:
- client = _fake_client(db_buffer)
- for index, name in [(0, "a.png"), (2, "c.csv")]:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a anchored" if index == 0 else name,
- filename=name if index == 0 else None,
- event_id=f"$part-{index}",
- group={"id": "grp-anchor-2", "index": index, "total": 3},
- ),
- )
- await asyncio.sleep(0.15)
- finally:
- ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS = original
-
- assert len(await _events(client)) == 1
- assert (await _events(client))[0].payload.message_id == "$part-0"
-
-
-async def test_group_timeout_bounds_the_group_not_the_gap_between_parts(
- monkeypatch: Any, db_buffer: EventBuffer
-) -> None:
- """The safety-net timer is armed once per group. A batch dribbling in just
- under the timeout must not be able to hold the buffer open indefinitely.
-
- The deadline is taken rather than waited on. Sleeping a fraction under a
- shortened timeout tests the rule only on a machine quiet enough to keep to
- the fraction, and on one that is not it reports a product defect — the
- group flushing early with fewer parts than the test arranged — for what is
- the suite's own load. Holding the arming and firing of the timer instead
- says the same thing about a group whose parts arrive over any interval at
- all, including the real five seconds nothing would wait for.
- """
- armed: list[tuple[float, Any]] = []
-
- def call_later(delay: float, callback: Any, *args: Any) -> Any:
- armed.append((delay, callback))
- return SimpleNamespace(cancel=lambda: None)
-
- monkeypatch.setattr(asyncio.get_running_loop(), "call_later", call_later)
- client = _fake_client(db_buffer)
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a slow batch",
- filename="a.png",
- event_id="$part-0",
- group={"id": "grp-slow", "index": 0, "total": 4},
- ),
- )
- first_timer = client._attachment_group_timers["grp-slow"]
- assert [delay for delay, _ in armed] == [ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS]
-
- # Parts keep trickling in below the deadline; the timer must NOT be
- # pushed back by each arrival.
- for index, name in [(1, "b.md"), (2, "c.csv")]:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body=name,
- event_id=f"$part-{index}",
- group={"id": "grp-slow", "index": index, "total": 4},
- ),
- )
- assert client._attachment_group_timers["grp-slow"] is first_timer
- assert len(armed) == 1
-
- running = asyncio.all_tasks()
- armed[0][1]()
- await asyncio.gather(*(asyncio.all_tasks() - running))
-
- # Fired on the group's own deadline rather than being extended forever.
- assert len(await _events(client)) == 1
- payload = (await _events(client))[0].payload
- assert "incomplete attachment group: 3 of 4" in payload.body
- assert client._attachment_groups == {}
- assert client._attachment_group_timers == {}
-
-
-class TestAddressingSurvivesCoalescing:
- """Two screenshots in one Slack message reached nobody (CHOO-2173).
-
- Matrix has no multi-attachment event, so one such message is split: the
- text and its `@mention` ride on part 0 and every later part is a bare
- file. Addressing was read from whichever part completed the group — a
- filename — so the reassembled message claimed to address nobody and each
- consumer, correctly, ignored it.
- """
-
- async def test_a_mention_on_part_zero_addresses_the_whole_group(
- self, db_buffer: EventBuffer
- ) -> None:
- client = _fake_client(db_buffer)
- for index, name in [(0, "one.png"), (1, "two.png")]:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a look at these" if index == 0 else name,
- filename=name if index == 0 else None,
- event_id=f"$part-{index}",
- group={"id": "grp-addr", "index": index, "total": 2},
- ),
- )
-
- assert len(await _events(client)) == 1
- payload = (await _events(client))[0].payload
- assert payload.addressed is True
- # The text travels with the images rather than being left on the part
- # it arrived on.
- assert payload.body == "@agent-a look at these"
- assert [a.filename for a in payload.attachments] == ["one.png", "two.png"]
-
- async def test_it_holds_when_the_captioned_part_arrives_last(
- self, db_buffer: EventBuffer
- ) -> None:
- client = _fake_client(db_buffer)
- for index, name in [(1, "two.png"), (0, "one.png")]:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a look at these" if index == 0 else name,
- filename=name if index == 0 else None,
- event_id=f"$part-{index}",
- group={"id": "grp-addr-rev", "index": index, "total": 2},
- ),
- )
-
- assert (await _events(client))[0].payload.addressed is True
-
- async def test_a_group_that_mentions_nobody_stays_unaddressed(
- self, db_buffer: EventBuffer
- ) -> None:
- # The fix must carry addressing across the group, not assert it. A
- # group for nobody is the room's conversation, so nothing is queued for
- # the agent: it is read from the room's log by a reader that wants it.
- client = _fake_client(db_buffer)
- for index, name in [(0, "one.png"), (1, "two.png")]:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="just some pictures" if index == 0 else name,
- filename=name if index == 0 else None,
- event_id=f"$part-{index}",
- group={"id": "grp-unaddr", "index": index, "total": 2},
- ),
- )
-
- assert await _events(client) == []
- assert client._attachment_groups == {}
-
- async def test_an_incomplete_group_is_still_addressed(
- self, db_buffer: EventBuffer
- ) -> None:
- # A batch that never completes is delivered by the safety net, and
- # losing a file is no reason to also lose who it was for.
- original = ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS
- ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS = 0.01
- try:
- client = _fake_client(db_buffer)
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a two of three",
- filename="one.png",
- event_id="$part-0",
- group={"id": "grp-addr-partial", "index": 0, "total": 3},
- ),
- )
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="two.png",
- event_id="$part-1",
- group={"id": "grp-addr-partial", "index": 1, "total": 3},
- ),
- )
- await asyncio.sleep(0.15)
- finally:
- ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS = original
-
- assert len(await _events(client)) == 1
- payload = (await _events(client))[0].payload
- assert payload.addressed is True
- assert "@agent-a two of three" in payload.body
-
- async def test_a_single_attachment_is_unaffected(
- self, db_buffer: EventBuffer
- ) -> None:
- # One screenshot always worked; it takes no group and no buffering.
- client = _fake_client(db_buffer)
- await AgentConsumer.on_media(
- client, _room(), _media_event(body="@agent-a one picture")
- )
-
- assert (await _events(client))[0].payload.addressed is True
-
-
-async def test_a_grouped_post_addressed_to_the_agent_is_reported_once(
- db_buffer: EventBuffer,
-) -> None:
- """Three files in one post are one message asked of the agent, not three:
- the report follows the coalesced payload, not the parts."""
- client = _fake_client(db_buffer)
- for index in range(3):
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body="@agent-a look at these" if index == 0 else f"f{index}.png",
- event_id=f"$part-{index}",
- group={"id": "grp-r", "index": index, "total": 3},
- ),
- )
-
- assert client._message_telemetry.addressed == [
- {
- "tenant_id": TENANT_ZERO_ID,
- "room_id": "room-1",
- "sender_transport_user_id": "@alice:s",
- "from_platform": False,
- "agent_metadata": None,
- "agent_live": True,
- "has_attachment": True,
- }
- ]
-
-
[email protected]("late_index", [0, 2])
-async def test_a_post_split_by_the_timeout_is_reported_once(
- late_index: int, db_buffer: EventBuffer
-) -> None:
- """A group that times out incomplete is delivered again when the rest
- arrives. In a direct room both pieces address the agent, but it is one
- post: counted with its first part, as the transport counts it."""
- original = ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS
- ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS = 0.01
- try:
- client = _fake_client(db_buffer)
-
- async def _direct_room(_event: Any, _meta: RoomMeta) -> bool:
- return True
-
- client._addressed = _direct_room
- on_time = [index for index in range(3) if index != late_index]
- for batch in (on_time, [late_index]):
- for index in batch:
- await AgentConsumer.on_media(
- client,
- _room(),
- _media_event(
- body=f"f{index}.png",
- event_id=f"$part-{index}",
- group={"id": "grp-late", "index": index, "total": 3},
- # Each part is a row of its own in the room's log.
- seq=index + 1,
- ),
- )
- await asyncio.sleep(0.15)
- finally:
- ac.ATTACHMENT_GROUP_TIMEOUT_SECONDS = original
-
- assert len(await _events(client)) == 2
- assert len(client._message_telemetry.addressed) == 1
-
-
-async def test_media_skips_the_availability_check_while_telemetry_is_off(
- db_buffer: EventBuffer,
-) -> None:
- """The media path reads it only to report it, so off means no query."""
- client = _fake_client(db_buffer)
- client._message_telemetry.enabled = False
- await AgentConsumer.on_media(
- client, _room(), _media_event(body="@agent-a look", event_id="$solo")
- )
-
- assert client.availability_checks == 0
- assert len(await _events(client)) == 1
-
-
-async def test_media_that_does_not_address_the_agent_is_not_reported(
- db_buffer: EventBuffer,
-) -> None:
- client = _fake_client(db_buffer)
- await AgentConsumer.on_media(
- client, _room(), _media_event(body="just a picture", event_id="$solo")
- )
-
- assert client._message_telemetry.addressed == []
diff --git a/core/tests/switch_core/clients/test_agent_consumer_pool_checkouts.py b/core/tests/switch_core/clients/test_agent_consumer_pool_checkouts.py
deleted file mode 100644
index b8de4aff9..000000000
--- a/core/tests/switch_core/clients/test_agent_consumer_pool_checkouts.py
+++ /dev/null
@@ -1,309 +0,0 @@
-"""How many pool checkouts one inbound room message costs an agent consumer.
-
-A room fans every Matrix message out to *all* of its agent clients at once, in
-a single event-loop tick. So this number is multiplied by the size of the room
-before it ever reaches the pool: at two checkouts per client, a ten-agent room
-turned one Slack message into twenty concurrent checkouts out of forty, and the
-addressed client alone took up to nine more. That is the burst that pushed the
-pool to its ceiling and started failing heartbeats.
-
-The contract is one lookup checkout per client per message, and none at all
-for a message that needs no lookup — plus, as with the room service, no session
-held open while the client posts to Matrix. Queuing a message for the agent it
-addresses is one more, its own short transaction on the db-buffer, taken after
-the lookup has returned its connection rather than alongside it. Nothing else
-is queued: the rest of the room is read from the room's log by whoever wants
-it, so a client that is not addressed writes nothing.
-"""
-
-from __future__ import annotations
-
-from types import SimpleNamespace
-from typing import Any
-
-from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-
-from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
-from switch_core.budgets import BudgetGuard
-from switch_core.clients.actor import AgentActor
-from switch_core.clients.agent_consumer import AgentConsumer
-from switch_core.clients.room_meta import RoomMeta
-from switch_core.db.models import Agent, ApiKey, Client, Room, User
-from switch_core.db.stores.agent_session_store import AgentSessionStore
-from switch_core.db.stores.agent_store import AgentStore
-from switch_core.db.stores.budget_store import BudgetStore
-from switch_core.db.stores.client_store import ClientStore
-from switch_core.db.stores.external_user_store import ExternalUserStore
-from switch_core.db.stores.room_role_store import RoomRoleStore
-from switch_core.db.stores.room_store import RoomStore
-from switch_core.delivery.addressing import AddressingResolver
-from switch_core.transport import InboundMessage, RoomRef
-from tests.db_buffer import db_buffer_on
-
-TRANSPORT_ROOM_ID = "!r:test"
-
-# The db-buffer's write, in a transaction of its own: made only by the client
-# whose agent the message addresses. Chatter, a message for somebody else and a
-# refused one stay in the room's log and cost the client no write.
-ENQUEUE = 1
-
-
-class _CountingSessionFactory:
- """The real factory, counting sessions opened and tracking how many are
- open right now. One session is one pool checkout."""
-
- def __init__(self, inner: async_sessionmaker[AsyncSession]) -> None:
- self._inner = inner
- self.opened = 0
- self.live = 0
- self.peak = 0
-
- def __call__(self) -> Any:
- self.opened += 1
- return _Tracked(self, self._inner())
-
-
-class _Tracked:
- def __init__(self, tracker: _CountingSessionFactory, inner: Any) -> None:
- self._tracker = tracker
- self._inner = inner
-
- async def __aenter__(self) -> AsyncSession:
- self._tracker.live += 1
- self._tracker.peak = max(self._tracker.peak, self._tracker.live)
- return await self._inner.__aenter__() # type: ignore[no-any-return]
-
- async def __aexit__(self, *exc: object) -> bool:
- self._tracker.live -= 1
- return await self._inner.__aexit__(*exc) # type: ignore[no-any-return]
-
-
-async def _seed(
- session_factory: async_sessionmaker[AsyncSession], *, names: list[str]
-) -> tuple[str, dict[str, Agent]]:
- async with session_factory() as session:
- user = User(name="o", email="o@test", role="user", password_hash="x")
- session.add(user)
- await session.flush()
- agents: dict[str, Agent] = {}
- for name in names:
- key = ApiKey(
- user_id=user.id,
- key_hash=f"hash-{name}",
- encrypted_key="enc",
- label=name,
- type="agent",
- )
- client = Client(
- transport_user_id=f"@{name}:test",
- display_name=name,
- type="agent",
- )
- session.add_all([key, client])
- await session.flush()
- agent = Agent(
- name=name,
- description="d",
- agent_type="session_addressable",
- connector_type="claude_code",
- integration_profile={"connection_model": "session_addressable"},
- client_id=client.id,
- api_key_id=key.id,
- owner_id=user.id,
- )
- session.add(agent)
- await session.flush()
- agents[name] = agent
- room = Room(transport_room_id=TRANSPORT_ROOM_ID, name="room", description="d")
- session.add(room)
- await session.flush()
- await RoomStore().add_agents(session, room.id, [a.id for a in agents.values()])
- await session.commit()
- return room.id, agents
-
-
-def _client(
- counting: _CountingSessionFactory, agent: Agent, room_id: str
-) -> AgentConsumer:
- """An AgentConsumer wired to the real stores, with its room-meta cache warm.
-
- The cache is warm because that is the steady state: a client resolves a
- room once and then answers every later message in it from memory.
- """
- actor = object.__new__(AgentActor)
- actor.session_factory = counting # type: ignore[assignment]
- actor.client_store = ClientStore()
- actor.transport_user_id = f"@{agent.name}:test"
- actor.client_id = agent.client_id
- actor.tenant_id = agent.tenant_id
- actor._agent = agent
- client = object.__new__(AgentConsumer)
- client.actor = actor
- client._agent_store = AgentStore()
- client._room_store = RoomStore()
- client._room_role_store = RoomRoleStore()
- client._agent_session_store = AgentSessionStore()
- client._external_user_store = ExternalUserStore()
- client._connections = AgentConnectionRegistry()
- client._addressing = AddressingResolver(
- room_store=client._room_store,
- room_role_store=client._room_role_store,
- client_store=client.client_store,
- agent_store=client._agent_store,
- external_user_store=client._external_user_store,
- live_connection_ids=client._connections.live_connection_ids,
- )
- client._budget_guard = BudgetGuard(BudgetStore())
- client._message_telemetry = SimpleNamespace( # type: ignore[assignment]
- agent_addressed=lambda **_k: None
- )
- client._frontend_base_url = None
- client._room_meta = {
- TRANSPORT_ROOM_ID: RoomMeta(
- room_id=room_id,
- name="room",
- bridge_id=None,
- channel_type="channel_public",
- )
- }
- client._attachment_groups = {}
- client._attachment_group_timers = {}
- client.sent = [] # type: ignore[attr-defined]
-
- async def _send_message(transport_room_id: str, body: str, **kwargs: Any) -> str:
- client.sent.append((body, counting.live)) # type: ignore[attr-defined]
- return "$sent"
-
- actor.send_message = _send_message # type: ignore[assignment, method-assign]
- client._event_buffer = db_buffer_on(counting) # type: ignore[arg-type]
- return client
-
-
-def _message(body: str, sender: str = "@switch-slack-louisa:test") -> InboundMessage:
- return InboundMessage(
- room_id=TRANSPORT_ROOM_ID,
- event_id="$trigger",
- sender=sender,
- timestamp=0,
- content={"body": body, "sender_name": "louisa"},
- body=body,
- sender_name="louisa",
- )
-
-
-def _room() -> RoomRef:
- return RoomRef(room_id=TRANSPORT_ROOM_ID)
-
-
-class TestInboundMessageCheckouts:
- async def test_chatter_with_no_at_costs_nothing(
- self, session_factory: async_sessionmaker[AsyncSession]
- ) -> None:
- # The common case in a busy room, and the one multiplied by every agent
- # in it: no `@`, so there is nothing an alias or a role lease could
- # match and no lookup to make. Nor is it queued: it addresses nobody,
- # so it is the room's conversation, read from the room's log.
- room_id, agents = await _seed(session_factory, names=["member"])
- counting = _CountingSessionFactory(session_factory)
- client = _client(counting, agents["member"], room_id)
-
- await client.on_message(_room(), _message("just talking amongst ourselves"))
-
- assert counting.opened == 0
-
- async def test_an_at_message_for_somebody_else_costs_one_lookup(
- self, session_factory: async_sessionmaker[AsyncSession]
- ) -> None:
- # Was two: the room alias and the role lease each took a session. The
- # lookup finds the message is not for this agent, so nothing is queued.
- room_id, agents = await _seed(session_factory, names=["member", "other"])
- counting = _CountingSessionFactory(session_factory)
- client = _client(counting, agents["member"], room_id)
-
- await client.on_message(_room(), _message("@other can you look at this"))
-
- assert counting.opened == 1
-
- async def test_being_addressed_and_offline_still_costs_one_lookup(
- self, session_factory: async_sessionmaker[AsyncSession]
- ) -> None:
- # The expensive path: addressing policy, liveness, and the whole
- # offline-reply derivation. Was up to nine checkouts.
- room_id, agents = await _seed(session_factory, names=["member"])
- counting = _CountingSessionFactory(session_factory)
- client = _client(counting, agents["member"], room_id)
-
- await client.on_message(_room(), _message("@member can you look at this"))
-
- assert counting.opened == 1 + ENQUEUE
- assert len(client.sent) == 1 # type: ignore[attr-defined]
-
- async def test_no_session_is_open_while_the_auto_reply_is_posted(
- self, session_factory: async_sessionmaker[AsyncSession]
- ) -> None:
- # Posting to Matrix is an unbounded wait as far as the pool is
- # concerned; doing it inside the transaction is how a slot ends up
- # idle-in-transaction for seconds.
- room_id, agents = await _seed(session_factory, names=["member"])
- counting = _CountingSessionFactory(session_factory)
- client = _client(counting, agents["member"], room_id)
-
- await client.on_message(_room(), _message("@member you there?"))
-
- assert [live for _body, live in client.sent] == [0] # type: ignore[attr-defined]
-
- async def test_a_restricted_agent_refusing_a_sender_costs_one_lookup(
- self, session_factory: async_sessionmaker[AsyncSession]
- ) -> None:
- room_id, agents = await _seed(session_factory, names=["member"])
- async with session_factory() as session:
- await AgentStore().update(
- session,
- agents["member"].id,
- addressing_policy={"rules": [{"agents": ["nobody"], "users": []}]},
- )
- await session.commit()
- counting = _CountingSessionFactory(session_factory)
- client = _client(counting, agents["member"], room_id)
-
- await client.on_message(_room(), _message("@member you there?"))
-
- # The refusal leaves the message as chatter, so nothing is queued.
- assert counting.opened == 1
- assert [live for _body, live in client.sent] == [0] # type: ignore[attr-defined]
-
- async def test_an_alias_tag_costs_one_lookup(
- self, session_factory: async_sessionmaker[AsyncSession]
- ) -> None:
- # Reached only after the alias lookup misses on name, so this is the
- # path where alias and role used to be two separate checkouts.
- room_id, agents = await _seed(session_factory, names=["member"])
- async with session_factory() as session:
- await RoomStore().set_alias(session, room_id, agents["member"].id, "fixer")
- await session.commit()
- counting = _CountingSessionFactory(session_factory)
- client = _client(counting, agents["member"], room_id)
-
- await client.on_message(_room(), _message("@fixer please look"))
-
- assert counting.opened == 1 + ENQUEUE
- assert len(client.sent) == 1 # type: ignore[attr-defined]
-
- async def test_a_ten_agent_room_costs_ten_lookups(
- self, session_factory: async_sessionmaker[AsyncSession]
- ) -> None:
- # The burst, end to end. One message, every client in the room handling
- # it at once: ten lookups, where it used to be twenty plus the addressed
- # client's own handful. Only the client the message addresses queues
- # it for its agent, after its lookup has let go, so it never holds two
- # slots; the other nine leave it in the room's log.
- names = [f"a{i}" for i in range(10)]
- room_id, agents = await _seed(session_factory, names=names)
- counting = _CountingSessionFactory(session_factory)
- clients = [_client(counting, agents[name], room_id) for name in names]
-
- for client in clients:
- await client.on_message(_room(), _message("@a3 can you take this"))
-
- assert counting.opened == 10 + ENQUEUE
- assert counting.peak == 1
diff --git a/core/tests/switch_core/clients/test_agent_consumer_reports_addressed.py b/core/tests/switch_core/clients/test_agent_consumer_reports_addressed.py
deleted file mode 100644
index 1de2c63b0..000000000
--- a/core/tests/switch_core/clients/test_agent_consumer_reports_addressed.py
+++ /dev/null
@@ -1,243 +0,0 @@
-"""An agent's consumer reports each message its agent was actually asked to act on.
-
-"Asked" means addressed *and* let through: a message the addressing policy or a
-budget turned away was not a request the agent received, and Switch's own
-auto-replies are not anyone asking for anything. Whether the agent was there
-to take it is reported alongside, not used to leave the message out.
-"""
-
-from __future__ import annotations
-
-from contextlib import asynccontextmanager
-from types import SimpleNamespace
-from typing import Any
-
-import pytest
-from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.clients import agent_consumer as agent_consumer_module
-from switch_core.clients.admin_messages import PLATFORM_MARKER
-from switch_core.clients.agent_consumer import (
- AUTO_REPLY_FLAG,
- AgentConsumer,
- _GateOutcome,
-)
-from switch_core.clients.room_meta import RoomMeta
-from switch_core.db.models import TENANT_ZERO_ID
-from switch_core.observability.metrics import MetricsRegistry, install, uninstall
-from switch_core.observability.throttle import WarningThrottle
-from switch_core.transport import InboundMessage, RoomRef
-from tests.db_buffer import seed_agent
-
-AGENT = "agent-1"
-
-
[email protected](autouse=True)
-async def _agent_row(session_factory: async_sessionmaker[AsyncSession]) -> None:
- """The agent the consumer queues for, so its events have a row to belong to."""
- await seed_agent(session_factory, AGENT)
-
-
-@asynccontextmanager
-async def _session_factory(): # type: ignore[no-untyped-def]
- yield object()
-
-
-class _FakeMessageTelemetry:
- def __init__(self) -> None:
- self.addressed: list[dict[str, Any]] = []
-
- def agent_addressed(self, **kwargs: Any) -> None:
- self.addressed.append(kwargs)
-
-
-class _BrokenMessageTelemetry:
- def agent_addressed(self, **kwargs: Any) -> None:
- raise RuntimeError("telemetry bug")
-
-
-def _consumer(
- buffer: EventBuffer,
- *,
- addressed: bool,
- allowed: bool = True,
- live: bool = True,
-) -> SimpleNamespace:
- """An AgentConsumer stand-in with addressing, the policy gate and whether
- the agent is live in the room fixed by the test."""
- meta = RoomMeta(
- room_id="room-1", name="Room", bridge_id=None, channel_type="channel_public"
- )
-
- async def _resolve_room_meta(_transport_room_id: str) -> RoomMeta:
- return meta
-
- async def _compute_addressed(_s: object, _e: object, _m: object) -> bool:
- return addressed
-
- async def _fresh_agent(_s: object) -> object:
- return ns.agent
-
- async def _gate_addressed(
- _s: object, _a: object, _e: object, _m: object
- ) -> _GateOutcome:
- return _GateOutcome(addressed=allowed, refusal=None)
-
- async def _is_available(_s: object, _a: object, _r: str) -> bool:
- return live
-
- async def _reply_when_unavailable_here(
- _s: object, _a: object, _m: object, _h: str
- ) -> str | None:
- return None
-
- ns = SimpleNamespace(
- agent=SimpleNamespace(
- id=AGENT, name="agent-a", metadata_={"known_agent_type": "codex"}
- ),
- tenant_id=TENANT_ZERO_ID,
- session_factory=_session_factory,
- _resolve_room_meta=_resolve_room_meta,
- _addressed_without_lookup=lambda _e, _m: None,
- _compute_addressed=_compute_addressed,
- _fresh_agent=_fresh_agent,
- _gate_addressed=_gate_addressed,
- _is_available=_is_available,
- _reply_when_unavailable_here=_reply_when_unavailable_here,
- _post_auto_reply=_ignore_post,
- _sender_handle=lambda _e: "@alice",
- _triggered_by_auto_reply=AgentConsumer._triggered_by_auto_reply,
- _event_buffer=buffer,
- _message_telemetry=_FakeMessageTelemetry(),
- )
- ns._report_addressed = AgentConsumer._report_addressed.__get__(ns)
- return ns
-
-
-async def _ignore_post(*_args: object, **_kwargs: object) -> None:
- return None
-
-
-def _message(event_id: str = "$evt", **content: object) -> InboundMessage:
- return InboundMessage(
- room_id="!room:s",
- event_id=event_id,
- sender="@alice:s",
- timestamp=0,
- content={"sender_name": "alice", **content},
- body="@agent-a can you help",
- sender_name="alice",
- )
-
-
-async def _deliver(consumer: SimpleNamespace, event: InboundMessage) -> None:
- await AgentConsumer.on_message(consumer, RoomRef(room_id="!room:s"), event) # type: ignore[arg-type]
-
-
-async def test_an_addressed_message_is_reported_once(db_buffer: EventBuffer) -> None:
- consumer = _consumer(db_buffer, addressed=True)
-
- await _deliver(consumer, _message())
-
- assert consumer._message_telemetry.addressed == [
- {
- "tenant_id": TENANT_ZERO_ID,
- "room_id": "room-1",
- "sender_transport_user_id": "@alice:s",
- "from_platform": False,
- "agent_metadata": {"known_agent_type": "codex"},
- "agent_live": True,
- "has_attachment": False,
- }
- ]
-
-
-async def test_room_chatter_is_not_reported(db_buffer: EventBuffer) -> None:
- consumer = _consumer(db_buffer, addressed=False)
-
- await _deliver(consumer, _message())
-
- assert consumer._message_telemetry.addressed == []
-
-
-async def test_a_message_the_policy_turned_away_is_not_reported(
- db_buffer: EventBuffer,
-) -> None:
- consumer = _consumer(db_buffer, addressed=True, allowed=False)
-
- await _deliver(consumer, _message())
-
- assert consumer._message_telemetry.addressed == []
-
-
-async def test_an_auto_reply_is_not_reported(db_buffer: EventBuffer) -> None:
- consumer = _consumer(db_buffer, addressed=True)
-
- await _deliver(consumer, _message(**{AUTO_REPLY_FLAG: True}))
-
- assert consumer._message_telemetry.addressed == []
-
-
-async def test_a_platform_message_says_it_came_from_the_platform(
- db_buffer: EventBuffer,
-) -> None:
- consumer = _consumer(db_buffer, addressed=True)
-
- await _deliver(consumer, _message(**{PLATFORM_MARKER: {}}))
-
- [report] = consumer._message_telemetry.addressed
- assert report["from_platform"] is True
-
-
-async def test_a_message_to_an_offline_agent_is_reported_as_not_live(
- db_buffer: EventBuffer,
-) -> None:
- consumer = _consumer(db_buffer, addressed=True, live=False)
-
- await _deliver(consumer, _message())
-
- [report] = consumer._message_telemetry.addressed
- assert report["agent_live"] is False
-
-
-async def test_a_telemetry_failure_does_not_cost_the_agent_the_message(
- monkeypatch: pytest.MonkeyPatch,
- caplog: pytest.LogCaptureFixture,
- db_buffer: EventBuffer,
-) -> None:
- """Caught, so it is counted, and logged once rather than once per
- message."""
- consumer = _consumer(db_buffer, addressed=True)
- consumer._message_telemetry = _BrokenMessageTelemetry()
- registry = MetricsRegistry()
- install(registry)
- monkeypatch.setattr(
- agent_consumer_module, "_report_failures", WarningThrottle(60.0)
- )
-
- try:
- with caplog.at_level("ERROR"):
- await _deliver(consumer, _message(event_id="$first"))
- await _deliver(consumer, _message(event_id="$second"))
- failures = next(
- p
- for p in registry.collect()
- if p.name == "switch.messages.observer_failures"
- )
- finally:
- uninstall()
-
- assert len(await db_buffer.read_from(TENANT_ZERO_ID, AGENT, 0)) == 2
- assert failures.numbers[0].value == 2.0
- assert dict(failures.numbers[0].attributes) == {"actor": "agent"}
- assert (
- len(
- [
- r
- for r in caplog.records
- if "Could not report a message addressed" in r.getMessage()
- ]
- )
- == 1
- )
diff --git a/core/tests/switch_core/clients/test_client_lifecycle_tenant_binding.py b/core/tests/switch_core/clients/test_client_lifecycle_tenant_binding.py
index 351bb72d2..154b3a155 100644
--- a/core/tests/switch_core/clients/test_client_lifecycle_tenant_binding.py
+++ b/core/tests/switch_core/clients/test_client_lifecycle_tenant_binding.py
@@ -28,6 +28,7 @@ from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.tenant_store import TenantStore
from switch_core.logging_context import current_log_context, log_context
from switch_core.tenant_context import current_tenant_id, tenant_scope
+from tests.ownership import NeverOwned
class _RecordingClient:
@@ -38,11 +39,11 @@ class _RecordingClient:
self.transport_user_id = "@recorded:test"
self._seen = seen
- async def start(self) -> None:
+ async def connect(self) -> None:
self._seen.append(current_tenant_id())
self.room_seen = current_log_context().room_id
- async def stop(self) -> None:
+ async def close(self) -> None:
return None
@@ -58,6 +59,8 @@ def _service(
session_factory=session_factory,
config=MagicMock(),
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
@@ -72,7 +75,7 @@ async def test_a_client_task_runs_with_no_tenant_bound(
with tenant_scope(room_tenant):
recording = _RecordingClient(seen)
- service._start_task("client-1", recording, recording) # type: ignore[arg-type]
+ service._start_task("client-1", recording) # type: ignore[arg-type]
await asyncio.sleep(0)
assert seen == [None], (
@@ -90,7 +93,7 @@ async def test_a_client_task_does_not_keep_the_room_it_was_created_in(
service = _service(session_factory, MagicMock())
with log_context(room_id="room-where-it-was-minted"):
- service._start_task("client-1", client, client) # type: ignore[arg-type]
+ service._start_task("client-1", client) # type: ignore[arg-type]
await asyncio.sleep(0)
assert client.room_seen is None
@@ -122,10 +125,10 @@ async def test_start_all_binds_nothing_around_starting_each_client(
seen: list[str | None] = []
client_factory = MagicMock()
- client_factory.create.side_effect = lambda record: (
- (recording := _RecordingClient(seen)),
- recording,
- )
+ # Each client's actor connects in every process.
+ client_factory.create_actor.side_effect = lambda record: _RecordingClient(seen)
+ client_factory.create_consumer.side_effect = lambda client_type, actor: actor
+ client_factory.has_consumer.return_value = False
service = _service(session_factory, client_factory)
await service.start_all()
diff --git a/core/tests/switch_core/clients/test_client_registry_tenants.py b/core/tests/switch_core/clients/test_client_registry_tenants.py
index 6807d3507..c9577b649 100644
--- a/core/tests/switch_core/clients/test_client_registry_tenants.py
+++ b/core/tests/switch_core/clients/test_client_registry_tenants.py
@@ -31,6 +31,7 @@ from switch_core.clients.client_lifecycle_service import ClientLifecycleService
from switch_core.db.models import TENANT_ZERO_ID, Client, Tenant
from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.tenant_store import TenantStore
+from tests.ownership import NeverOwned
TENANT_B = "11111111-1111-1111-1111-111111111111"
@@ -39,7 +40,7 @@ class _StubClient:
"""A client that stays running until stopped, and touches nothing else.
It has to actually stay running: `_start_task` puts every client in a task
- that removes it from all three registries if `start` returns or raises, so
+ that removes it from all three registries if `connect` returns or raises, so
a stub that finished immediately would empty the very dicts under test.
"""
@@ -49,18 +50,25 @@ class _StubClient:
self.display_name = record.display_name
self._stopped = asyncio.Event()
- async def start(self) -> None:
+ async def connect(self) -> None:
await self._stopped.wait()
- async def stop(self) -> None:
+ async def close(self) -> None:
self._stopped.set()
class _StubFactory:
- def create(self, record: Client) -> Any:
- # The stub reads for itself: it is the actor and its own consumer.
- client = _StubClient(record)
- return client, client
+ """The stub reads for itself: it is the actor and its own consumer, and
+ runs in every process."""
+
+ def create_actor(self, record: Client) -> Any:
+ return _StubClient(record)
+
+ def create_consumer(self, client_type: str, actor: Any) -> Any:
+ return actor
+
+ def has_consumer(self, client_type: str) -> bool:
+ return False
def _service(
@@ -74,6 +82,8 @@ def _service(
session_factory=session_factory,
config=SimpleNamespace(id_server_name="test"), # type: ignore[arg-type]
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
diff --git a/core/tests/switch_core/clients/test_client_signals.py b/core/tests/switch_core/clients/test_client_signals.py
new file mode 100644
index 000000000..ffb03989a
--- /dev/null
+++ b/core/tests/switch_core/clients/test_client_signals.py
@@ -0,0 +1,246 @@
+"""A client registered or deleted after boot reaches every process.
+
+Two "processes" on one database, each with its own lifecycle service, client
+factory and signal bus: a registration or deletion announced by one has to
+start or stop the other's copy, which only the signal and the re-read of the
+tenant's rows can do.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+from collections.abc import AsyncIterator, Callable
+from typing import Any
+from unittest.mock import MagicMock
+
+import pytest
+from sqlalchemy import delete
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.clients.client_lifecycle_service import ClientLifecycleService
+from switch_core.clients.client_signals import ClientSignals
+from switch_core.db.models import TENANT_ZERO_ID, Agent, Client
+from switch_core.db.stores.client_store import ClientStore
+from switch_core.db.stores.tenant_store import TenantStore
+from switch_core.signals import SignalBus
+from tests.ownership import NeverOwned
+from tests.switch_core.test_room_wide_mention_wakes_no_agent import _make_agent
+
+DELIVERY_TIMEOUT = 10.0
+
+
+class _Consumer:
+ def __init__(self) -> None:
+ self.display_name = "agent"
+ self.transport_user_id = "@agent:test"
+ self.stopped = False
+
+ def preload_agent(self, agent: Agent) -> None:
+ return None
+
+ async def connect(self) -> None:
+ return None
+
+ async def close(self) -> None:
+ self.stopped = True
+
+
+class _Factory:
+ """Each client reads for itself, in every process."""
+
+ def create_actor(self, record: Any) -> _Consumer:
+ return _Consumer()
+
+ def create_consumer(self, client_type: str, actor: _Consumer) -> _Consumer:
+ return actor
+
+ def has_consumer(self, client_type: str) -> bool:
+ return False
+
+
+def _lifecycle(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> ClientLifecycleService:
+ return ClientLifecycleService(
+ provisioning=MagicMock(),
+ client_store=ClientStore(),
+ tenant_store=TenantStore(),
+ client_factory=_Factory(), # type: ignore[arg-type]
+ session_factory=session_factory,
+ config=MagicMock(),
+ tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
+ )
+
+
+def _bus(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession]
+) -> SignalBus:
+ return SignalBus(
+ session_factory,
+ lambda: create_async_engine(postgres_url, poolclass=NullPool),
+ process_id=str(uuid.uuid4()),
+ )
+
+
+async def _eventually(check: Callable[[], bool]) -> None:
+ async with asyncio.timeout(DELIVERY_TIMEOUT):
+ while not check():
+ await asyncio.sleep(0.02)
+
+
+async def _register(session_factory: async_sessionmaker[AsyncSession]) -> Agent:
+ async with session_factory() as session:
+ agent = await _make_agent(session, f"late-{uuid.uuid4().hex[:8]}")
+ await session.commit()
+ return agent
+
+
+async def _delete(
+ session_factory: async_sessionmaker[AsyncSession], agent: Agent
+) -> None:
+ async with session_factory() as session:
+ await session.execute(delete(Agent).where(Agent.id == agent.id))
+ await session.execute(delete(Client).where(Client.id == agent.client_id))
+ await session.commit()
+
+
+async def two_processes(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession]
+) -> AsyncIterator[tuple[ClientLifecycleService, ClientLifecycleService]]:
+ lifecycles = [_lifecycle(session_factory) for _ in range(2)]
+ buses = [_bus(postgres_url, session_factory) for _ in range(2)]
+ for lifecycle, bus in zip(lifecycles, buses, strict=True):
+ ClientSignals(lifecycle=lifecycle, bus=bus)
+ await lifecycle.start_all()
+ await bus.start()
+ for bus in buses:
+ await asyncio.wait_for(bus.connected.wait(), DELIVERY_TIMEOUT)
+ yield lifecycles[0], lifecycles[1]
+ for lifecycle, bus in zip(lifecycles, buses, strict=True):
+ await bus.stop()
+ await lifecycle.stop_all()
+
+
+class TestAcrossProcesses:
+ async def test_a_client_registered_in_one_process_starts_in_the_other(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two_processes: tuple[ClientLifecycleService, ClientLifecycleService],
+ ) -> None:
+ here, there = two_processes
+ agent = await _register(session_factory)
+ assert agent.client_id is not None
+
+ here.announce(TENANT_ZERO_ID)
+
+ await _eventually(lambda: there.get(agent.client_id) is not None) # type: ignore[arg-type]
+ # The announcing process re-reads too, and starts what it lacks.
+ await _eventually(lambda: here.get(agent.client_id) is not None) # type: ignore[arg-type]
+
+ async def test_a_client_deleted_in_one_process_stops_in_the_other(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two_processes: tuple[ClientLifecycleService, ClientLifecycleService],
+ ) -> None:
+ here, there = two_processes
+ agent = await _register(session_factory)
+ client_id = agent.client_id
+ assert client_id is not None
+ here.announce(TENANT_ZERO_ID)
+ await _eventually(lambda: there.get(client_id) is not None)
+ copy = there.get(client_id)
+
+ # As `AgentCore.delete_agent` does: stop here, delete, announce.
+ await here.stop(client_id)
+ await _delete(session_factory, agent)
+ here.announce(TENANT_ZERO_ID)
+
+ await _eventually(lambda: there.get(client_id) is None)
+ assert isinstance(copy, _Consumer) and copy.stopped
+
+
+class TestReconcile:
+ async def test_the_recheck_starts_a_client_whose_signal_was_missed(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ lifecycle = _lifecycle(session_factory)
+ await lifecycle.start_all()
+ agent = await _register(session_factory)
+
+ await lifecycle.reconcile_all()
+
+ assert lifecycle.get(agent.client_id) is not None # type: ignore[arg-type]
+ await lifecycle.stop_all()
+
+ async def test_an_agent_client_waits_for_its_agent_row(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """Registration commits the client row before the agent row, and an
+ agent actor started between the two would find no agent."""
+ lifecycle = _lifecycle(session_factory)
+ async with session_factory() as session:
+ client = Client(
+ transport_user_id="@half-registered:test",
+ display_name="half",
+ type="agent",
+ )
+ session.add(client)
+ await session.commit()
+ client_id = client.id
+
+ await lifecycle.reconcile_tenant(TENANT_ZERO_ID)
+
+ assert lifecycle.get(client_id) is None
+
+ async def test_a_client_stopped_on_purpose_stays_stopped(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """An agent being deleted stops its client before its rows go. A
+ re-check in between must not start it again."""
+ agent = await _register(session_factory)
+ assert agent.client_id is not None
+ lifecycle = _lifecycle(session_factory)
+ await lifecycle.start_all()
+ await lifecycle.stop(agent.client_id)
+
+ await lifecycle.reconcile_tenant(TENANT_ZERO_ID)
+
+ assert lifecycle.get(agent.client_id) is None
+
+ async def test_a_client_registered_during_a_recheck_keeps_running(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """A registration in this process can commit and start its client
+ while a re-check awaits its read. The re-check's rows predate that
+ client, which must not make it look deleted."""
+ lifecycle = _lifecycle(session_factory)
+ await lifecycle.start_all()
+ store = lifecycle._client_store
+ real_get_all = store.get_all
+ registered: list[str] = []
+
+ async def read_then_register(session: Any) -> Any:
+ records = await real_get_all(session)
+ if not registered:
+ # As `AgentCore._create_agent` does: commit, then start.
+ agent = await _register(session_factory)
+ async with session_factory() as other:
+ record = await other.get(Client, agent.client_id)
+ assert record is not None
+ lifecycle.start_client(record)
+ registered.append(record.id)
+ return records
+
+ store.get_all = read_then_register # type: ignore[method-assign]
+ try:
+ await lifecycle.reconcile_tenant(TENANT_ZERO_ID)
+ finally:
+ store.get_all = real_get_all # type: ignore[method-assign]
+
+ assert lifecycle.get(registered[0]) is not None
+ await lifecycle.stop_all()
diff --git a/core/tests/switch_core/clients/test_client_start_retry.py b/core/tests/switch_core/clients/test_client_start_retry.py
index 8103d58c5..e6575f002 100644
--- a/core/tests/switch_core/clients/test_client_start_retry.py
+++ b/core/tests/switch_core/clients/test_client_start_retry.py
@@ -15,6 +15,7 @@ from sqlalchemy.exc import TimeoutError as PoolTimeoutError
from switch_core.clients import client_lifecycle_service as lifecycle
from switch_core.clients.client_lifecycle_service import ClientLifecycleService
+from tests.ownership import NeverOwned
class _FlakyClient:
@@ -28,14 +29,14 @@ class _FlakyClient:
self._error = error
self.running = asyncio.Event()
- async def start(self) -> None:
+ async def connect(self) -> None:
self.starts += 1
if self.starts <= self._failures:
raise self._error
self.running.set()
await asyncio.Event().wait()
- async def stop(self) -> None:
+ async def close(self) -> None:
return None
@@ -48,6 +49,8 @@ def _service() -> ClientLifecycleService:
session_factory=MagicMock(),
config=MagicMock(),
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
@@ -61,7 +64,7 @@ async def test_a_pool_timeout_at_start_is_retried_until_the_client_runs() -> Non
client = _FlakyClient(failures=2, error=PoolTimeoutError("QueuePool limit reached"))
service._clients["c1"] = client # type: ignore[assignment]
- service._start_task("c1", client, client) # type: ignore[arg-type]
+ service._start_task("c1", client) # type: ignore[arg-type]
await asyncio.wait_for(client.running.wait(), timeout=2)
assert client.starts == 3
@@ -74,7 +77,7 @@ async def test_any_other_error_still_ends_the_client() -> None:
client = _FlakyClient(failures=1, error=ValueError("a bug in the client"))
service._clients["c1"] = client # type: ignore[assignment]
- service._start_task("c1", client, client) # type: ignore[arg-type]
+ service._start_task("c1", client) # type: ignore[arg-type]
for _ in range(50):
if service.get("c1") is None:
break
diff --git a/core/tests/switch_core/clients/test_client_task_replacement.py b/core/tests/switch_core/clients/test_client_task_replacement.py
index fc371cb3f..a857ee99e 100644
--- a/core/tests/switch_core/clients/test_client_task_replacement.py
+++ b/core/tests/switch_core/clients/test_client_task_replacement.py
@@ -16,21 +16,21 @@ from unittest.mock import MagicMock
from switch_core.clients.client_lifecycle_service import ClientLifecycleService
from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.tenant_store import TenantStore
+from tests.ownership import NeverOwned
class _BlockingClient:
- """A consumer whose task never finishes on its own, like a receive loop.
- It stands in for its own actor too: the task only reads a name off it."""
+ """An actor whose task never finishes on its own, like a receive loop."""
def __init__(self) -> None:
self.display_name = "blocking"
self.transport_user_id = "@blocking:test"
self.stopped = False
- async def start(self) -> None:
+ async def connect(self) -> None:
await asyncio.Event().wait()
- async def stop(self) -> None:
+ async def close(self) -> None:
self.stopped = True
@@ -43,18 +43,20 @@ def _service() -> ClientLifecycleService:
session_factory=MagicMock(),
config=MagicMock(),
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
async def test_starting_a_client_twice_cancels_the_first_task() -> None:
service = _service()
client = _BlockingClient()
- service._start_task("client-1", client, client) # type: ignore[arg-type]
+ service._start_task("client-1", client) # type: ignore[arg-type]
await asyncio.sleep(0)
first = service._tasks["client-1"]
client = _BlockingClient()
- service._start_task("client-1", client, client) # type: ignore[arg-type]
+ service._start_task("client-1", client) # type: ignore[arg-type]
second = service._tasks["client-1"]
await asyncio.gather(first, return_exceptions=True)
@@ -72,10 +74,9 @@ async def test_stop_all_waits_for_the_tasks_it_cancelled() -> None:
service = _service()
client = _BlockingClient()
service._clients["client-1"] = client # type: ignore[assignment]
- service._consumers["client-1"] = client # type: ignore[assignment]
service._client_types["client-1"] = "agent"
service._client_tenants["client-1"] = "tenant-1"
- service._start_task("client-1", client, client) # type: ignore[arg-type]
+ service._start_task("client-1", client) # type: ignore[arg-type]
await asyncio.sleep(0)
task = service._tasks["client-1"]
diff --git a/core/tests/switch_core/clients/test_client_tenant_plumbing.py b/core/tests/switch_core/clients/test_client_tenant_plumbing.py
index 18e2eae70..8af66da4e 100644
--- a/core/tests/switch_core/clients/test_client_tenant_plumbing.py
+++ b/core/tests/switch_core/clients/test_client_tenant_plumbing.py
@@ -52,6 +52,8 @@ from switch_core.tenant_context import tenant_scope
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
from switch_core.transport.observer import IgnoreParticipantMessages
+from tests.ownership import NeverOwned
+from tests.routing import router_on
def _factory(session_factory: async_sessionmaker[AsyncSession]) -> ClientFactory:
@@ -68,6 +70,7 @@ def _factory(session_factory: async_sessionmaker[AsyncSession]) -> ClientFactory
ephemeral=EphemeralBus(),
room_cache=MagicMock(),
message_observer=IgnoreParticipantMessages(),
+ router=router_on(session_factory),
)
factory.register("user", Actor)
return factory
@@ -90,7 +93,7 @@ class TestTheFactoryReadsItOffTheRow:
) -> None:
tenant_id = f"tenant-{uuid.uuid4().hex[:8]}"
- client, _ = _factory(session_factory).create(_record(tenant_id))
+ client = _factory(session_factory).create_actor(_record(tenant_id))
assert client.tenant_id == tenant_id
@@ -104,7 +107,7 @@ class TestTheFactoryReadsItOffTheRow:
tenant_id = f"tenant-{uuid.uuid4().hex[:8]}"
factory = _factory(session_factory)
- client, _ = factory.create(_record(tenant_id))
+ client = factory.create_actor(_record(tenant_id))
transport = factory.transport_for(client)
assert transport.tenant_id == tenant_id # type: ignore[attr-defined]
@@ -155,6 +158,8 @@ class TestAFreshlyWrittenRowAlreadyCarriesIt:
session_factory=session_factory,
config=SimpleNamespace(id_server_name="test"), # type: ignore[arg-type]
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
with tenant_scope(tenant_id):
@@ -177,6 +182,8 @@ class TestAFreshlyWrittenRowAlreadyCarriesIt:
session_factory=session_factory,
config=SimpleNamespace(id_server_name="test"), # type: ignore[arg-type]
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
record = await service.create_client(
diff --git a/core/tests/switch_core/clients/test_command_consumer_ownership.py b/core/tests/switch_core/clients/test_command_consumer_ownership.py
new file mode 100644
index 000000000..4ec814e78
--- /dev/null
+++ b/core/tests/switch_core/clients/test_command_consumer_ownership.py
@@ -0,0 +1,326 @@
+"""The admin's consumer runs in one process, and answers once.
+
+Against real PostgreSQL, two "processes" on one database, each wired the way
+`main` wires one: its own message listener, client factory, lifecycle,
+process lease, ownership and cursor writer. Both run the tenant's admin actor,
+so either can write as Switch, but only the one that owns the admin client
+runs its `CommandConsumer`. A message tagging an agent that is not in the room
+is answered with one warning, not one per process, and one said while no
+process read for the admin is answered by the next owner, once.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+from collections.abc import AsyncIterator
+from dataclasses import dataclass
+from typing import Any
+from unittest.mock import MagicMock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
+from switch_core.clients.actor import AgentActor, SystemActor
+from switch_core.clients.admin_messages import ADMIN_MARKER, AdminMessageType
+from switch_core.clients.client_factory import ClientFactory
+from switch_core.clients.client_lifecycle_service import ClientLifecycleService
+from switch_core.clients.command_consumer import CommandConsumer
+from switch_core.db.models import TENANT_ZERO_ID, Client, Message, Room
+from switch_core.db.stores.agent_session_store import AgentSessionStore
+from switch_core.db.stores.agent_store import AgentStore
+from switch_core.db.stores.client_store import ClientStore
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
+from switch_core.db.stores.document_store import DocumentStore
+from switch_core.db.stores.media_store import MediaStore
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+from switch_core.db.stores.reference_store import ReferenceStore
+from switch_core.db.stores.room_role_store import RoomRoleStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.db.stores.tenant_store import TenantStore
+from switch_core.db.stores.usage_store import UsageStore
+from switch_core.messages.notify import MessageListener
+from switch_core.ownership import Ownership
+from switch_core.process_lease import ProcessLease
+from switch_core.signals import SignalBus
+from switch_core.transport.cursors import CursorWriter
+from switch_core.transport.ephemeral import EphemeralBus
+from switch_core.transport.invites import InviteBus
+from switch_core.transport.observer import IgnoreParticipantMessages
+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
+from tests.ownership import process_lease_on
+from tests.routing import router_on
+from tests.switch_core.test_room_wide_mention_wakes_no_agent import _make_agent
+
+TIMEOUT = 10.0
+
+
+@dataclass
+class _Process:
+ lifecycle: ClientLifecycleService
+ ownership: Ownership
+ lease: ProcessLease
+ listener: MessageListener
+ writer_task: asyncio.Task[None]
+
+ async def stop(self) -> None:
+ """Shut down as `main` does: owned work first, then the lease."""
+ await self.ownership.stop()
+ await self.lease.stop()
+ await self.lifecycle.stop_all()
+ await self.listener.stop()
+ self.writer_task.cancel()
+ await asyncio.gather(self.writer_task, return_exceptions=True)
+
+
+def _never_listening() -> Any:
+ raise AssertionError("these processes' signal buses are never started")
+
+
+async def _process(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession]
+) -> _Process:
+ listener = MessageListener(
+ lambda: create_async_engine(postgres_url, poolclass=NullPool)
+ )
+ await listener.start()
+ await asyncio.wait_for(listener.connected.wait(), TIMEOUT)
+ factory = ClientFactory(
+ client_store=ClientStore(),
+ session_factory=session_factory,
+ config=MagicMock(),
+ room_store=RoomStore(),
+ message_store=MessageStore(),
+ usage_store=UsageStore(),
+ media_store=MediaStore(),
+ listener=listener,
+ invites=InviteBus(),
+ ephemeral=EphemeralBus(),
+ room_cache=RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=MessageStore(),
+ limits=RoomCacheLimits(
+ max_bytes=1 << 24,
+ max_rooms=100,
+ max_rows_per_room=200,
+ max_age_seconds=60.0,
+ ),
+ ),
+ message_observer=IgnoreParticipantMessages(),
+ router=router_on(session_factory),
+ )
+ # The agent tagged in the room is a client too; it reads nothing here.
+ factory.register("agent", AgentActor)
+ factory.register(
+ "admin",
+ SystemActor,
+ CommandConsumer,
+ agent_store=AgentStore(),
+ room_store=RoomStore(),
+ room_role_store=RoomRoleStore(),
+ document_store=DocumentStore(),
+ reference_store=ReferenceStore(),
+ agent_session_store=AgentSessionStore(),
+ room_service=MagicMock(),
+ connections=AgentConnectionRegistry(),
+ frontend_base_url=None,
+ )
+ lease = process_lease_on(session_factory)
+ ownership = Ownership(
+ lease=lease,
+ session_factory=session_factory,
+ bus=SignalBus(session_factory, _never_listening, process_id=lease.process_id),
+ store=OwnershipLeaseStore(),
+ processes=SwitchCoreProcessStore(),
+ )
+ writer = CursorWriter(
+ session_factory=session_factory,
+ cursors=DeliveryCursorStore(),
+ leases=OwnershipLeaseStore(),
+ process_id=lease.process_id,
+ )
+ lifecycle = ClientLifecycleService(
+ provisioning=MagicMock(),
+ client_store=ClientStore(),
+ tenant_store=TenantStore(),
+ client_factory=factory,
+ session_factory=session_factory,
+ config=MagicMock(),
+ tenants_isolated=True,
+ ownership=ownership,
+ cursor_writer=writer,
+ )
+ return _Process(
+ lifecycle=lifecycle,
+ ownership=ownership,
+ lease=lease,
+ listener=listener,
+ writer_task=asyncio.create_task(writer.run()),
+ )
+
+
+@dataclass
+class _Room:
+ room_id: str
+ transport_room_id: str
+ admin_id: str
+ speaker_id: str
+ speaker: str
+
+
+async def room(session_factory: async_sessionmaker[AsyncSession]) -> _Room:
+ """A room with the admin client in it, a person who speaks there, and an
+ agent that is not in it, for the person to tag."""
+ suffix = uuid.uuid4().hex[:8]
+ async with session_factory() as session:
+ admin = Client(
+ transport_user_id=f"@switch-admin-{suffix}:test",
+ display_name="admin",
+ type="admin",
+ )
+ speaker = Client(
+ transport_user_id=f"@person-{suffix}:test",
+ display_name="person",
+ type="user",
+ )
+ room = Room(transport_room_id=f"!room-{suffix}:test", name="r", description="")
+ session.add_all([admin, speaker, room])
+ await session.flush()
+ await RoomStore().add_client(session, admin.id, room.id)
+ await RoomStore().add_client(session, speaker.id, room.id)
+ await _make_agent(session, "absentee")
+ await session.commit()
+ return _Room(
+ room_id=room.id,
+ transport_room_id=room.transport_room_id,
+ admin_id=admin.id,
+ speaker_id=speaker.id,
+ speaker=speaker.transport_user_id,
+ )
+
+
+async def two(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession], room: _Room
+) -> AsyncIterator[tuple[_Process, _Process]]:
+ processes = [await _process(postgres_url, session_factory) for _ in range(2)]
+ for process in processes:
+ await process.lifecycle.start_all()
+ await process.ownership.start()
+ # The first balance of each cannot see the other; balance once more now
+ # that both leases are there, as the interval would.
+ for process in processes:
+ await process.ownership.balance()
+ try:
+ yield processes[0], processes[1]
+ finally:
+ for process in processes:
+ await process.stop()
+
+
+async def _say(
+ session_factory: async_sessionmaker[AsyncSession], room: _Room, body: str
+) -> None:
+ async with session_factory() as session:
+ await MessageStore().create(
+ session,
+ Message(
+ room_id=room.room_id,
+ transport_event_id=f"sw_{uuid.uuid4().hex}",
+ sender_id=room.speaker,
+ sender_client_id=room.speaker_id,
+ sender_name="person",
+ event_type="m.room.message",
+ msgtype="m.text",
+ body=body,
+ formatted_body=None,
+ content={"body": body, "sender_name": "person"},
+ ),
+ [],
+ )
+ await session.commit()
+
+
+async def _warnings(
+ session_factory: async_sessionmaker[AsyncSession], room: _Room
+) -> list[Message]:
+ async with session_factory() as session:
+ rows = (
+ await session.scalars(
+ select(Message)
+ .where(Message.room_id == room.room_id)
+ .order_by(Message.seq)
+ )
+ ).all()
+ return [
+ row
+ for row in rows
+ if (row.content or {}).get(ADMIN_MARKER, {}).get("type")
+ == AdminMessageType.ABSENT_AGENT.value
+ ]
+
+
+async def _eventually_warned(
+ session_factory: async_sessionmaker[AsyncSession], room: _Room, count: int
+) -> list[Message]:
+ async with asyncio.timeout(TIMEOUT):
+ while len(await _warnings(session_factory, room)) < count:
+ await asyncio.sleep(0.05)
+ # Long enough for a second reader, had there been one, to have answered.
+ await asyncio.sleep(0.5)
+ return await _warnings(session_factory, room)
+
+
+class TestOneAdminConsumer:
+ async def test_two_processes_run_one_command_consumer_between_them(
+ self, two: tuple[_Process, _Process], room: _Room
+ ) -> None:
+ a, b = two
+
+ owners = [p for p in (a, b) if p.lifecycle.owns(room.admin_id)]
+
+ assert len(owners) == 1
+ assert owners[0].lifecycle.running_count() == 1
+ other = b if owners[0] is a else a
+ assert other.lifecycle.running_count() == 0
+ # Both can write as Switch.
+ for process in (a, b):
+ assert process.lifecycle.get_by_type("admin", TENANT_ZERO_ID)
+
+ async def test_a_message_is_answered_once(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ room: _Room,
+ ) -> None:
+ await _say(session_factory, room, "@absentee are you there?")
+
+ warnings = await _eventually_warned(session_factory, room, 1)
+
+ assert len(warnings) == 1
+
+ async def test_the_next_owner_answers_what_was_said_while_nobody_read(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ room: _Room,
+ ) -> None:
+ a, b = two
+ owner, other = (a, b) if a.lifecycle.owns(room.admin_id) else (b, a)
+ await _say(session_factory, room, "@absentee first")
+ await _eventually_warned(session_factory, room, 1)
+
+ await owner.stop()
+ await _say(session_factory, room, "@absentee said with nobody reading")
+ await other.ownership.balance()
+
+ warnings = await _eventually_warned(session_factory, room, 2)
+
+ assert other.lifecycle.owns(room.admin_id)
+ assert len(warnings) == 2
diff --git a/core/tests/switch_core/clients/test_enqueue_debug_logging.py b/core/tests/switch_core/clients/test_enqueue_debug_logging.py
deleted file mode 100644
index 7787236af..000000000
--- a/core/tests/switch_core/clients/test_enqueue_debug_logging.py
+++ /dev/null
@@ -1,182 +0,0 @@
-"""The debug line for an enqueued event costs nothing unless DEBUG is on.
-
-It pretty-prints the whole event, and an argument to `logger.debug` is built
-before the level is checked. That ran once per agent per message: on a busy
-server, thousands of serialisations a second for a line nobody was logging.
-"""
-
-from __future__ import annotations
-
-import logging
-from contextlib import asynccontextmanager
-from types import SimpleNamespace
-
-import pytest
-from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.types import AgentEvent
-from switch_core.clients.agent_consumer import AgentConsumer, _GateOutcome
-from switch_core.db.models import TENANT_ZERO_ID
-from switch_core.transport import InboundMedia, InboundMessage, RoomRef
-from tests.db_buffer import seed_agent
-
-_LOGGER = "switch_core.clients.agent_consumer"
-AGENT = "agent-1"
-
-
[email protected](autouse=True)
-async def _agent_row(session_factory: async_sessionmaker[AsyncSession]) -> None:
- """The agent the consumer queues for, so its events have a row to belong to."""
- await seed_agent(session_factory, AGENT)
-
-
-async def _enqueued(buffer: EventBuffer) -> list[AgentEvent]:
- return [item.event for item in await buffer.read_from(TENANT_ZERO_ID, AGENT, 0)]
-
-
-@asynccontextmanager
-async def _session_factory(): # type: ignore[no-untyped-def]
- yield object()
-
-
-def _consumer(buffer: EventBuffer) -> SimpleNamespace:
- """A delivery addressed to a live agent, every lookup answered in place.
-
- Addressed because only an event for the agent is enqueued: the rest of the
- room stays in the room's log and never reaches the line under test.
- """
-
- async def _resolve_room_meta(_room_id: str) -> SimpleNamespace:
- return SimpleNamespace(
- room_id="room-1", bridge_id=None, channel_type="channel_public"
- )
-
- async def _yes(*_args: object) -> bool:
- return True
-
- async def _fresh_agent(_session: object) -> object:
- return ns.agent
-
- async def _gate_addressed(*_args: object) -> _GateOutcome:
- return _GateOutcome(addressed=True, refusal=None)
-
- ns = SimpleNamespace(
- agent=SimpleNamespace(id=AGENT, name="helper"),
- tenant_id=TENANT_ZERO_ID,
- session_factory=_session_factory,
- _resolve_room_meta=_resolve_room_meta,
- _addressed_without_lookup=lambda _event, _meta: True,
- _compute_addressed=_yes,
- _fresh_agent=_fresh_agent,
- _gate_addressed=_gate_addressed,
- _triggered_by_auto_reply=AgentConsumer._triggered_by_auto_reply,
- _is_available=_yes,
- _message_telemetry=SimpleNamespace(enabled=False),
- _report_addressed=lambda *_a, **_k: None,
- _event_buffer=buffer,
- )
- return ns
-
-
-def _message() -> InboundMessage:
- return InboundMessage(
- room_id="!room:server",
- event_id="$evt",
- sender="@someone:switch.local",
- timestamp=1,
- content={"body": "@helper hello"},
- body="@helper hello",
- sender_name="someone",
- )
-
-
-def dumps(monkeypatch: pytest.MonkeyPatch) -> list[int]:
- calls: list[int] = []
- real = AgentEvent.model_dump_json
-
- def counting(self: AgentEvent, **kwargs: object) -> str:
- calls.append(1)
- return real(self, **kwargs) # type: ignore[arg-type]
-
- monkeypatch.setattr(AgentEvent, "model_dump_json", counting)
- return calls
-
-
-async def test_an_event_is_not_serialised_for_a_debug_line_nobody_logs(
- dumps, caplog, db_buffer
-):
- caplog.set_level(logging.INFO, logger=_LOGGER)
-
- await AgentConsumer.on_message(
- _consumer(db_buffer), # type: ignore[arg-type]
- RoomRef(room_id="!room:server"),
- _message(),
- )
-
- assert len(await _enqueued(db_buffer)) == 1
- assert dumps == []
- assert not [r for r in caplog.records if "Enqueuing event" in r.getMessage()]
-
-
-async def test_the_debug_line_is_still_logged_when_debug_is_on(
- dumps, caplog, db_buffer
-):
- caplog.set_level(logging.DEBUG, logger=_LOGGER)
-
- await AgentConsumer.on_message(
- _consumer(db_buffer), # type: ignore[arg-type]
- RoomRef(room_id="!room:server"),
- _message(),
- )
-
- assert len(await _enqueued(db_buffer)) == 1
- assert dumps == [1]
- lines = [r for r in caplog.records if "Enqueuing event" in r.getMessage()]
- assert len(lines) == 1
- assert '"body": "@helper hello"' in lines[0].getMessage()
-
-
[email protected]("level", [logging.INFO, logging.DEBUG])
-async def test_a_media_event_is_serialised_only_when_debug_is_on(
- dumps, caplog, level, db_buffer
-):
- caplog.set_level(level, logger=_LOGGER)
- media = InboundMedia(
- room_id="!room:server",
- event_id="$media",
- sender="@someone:switch.local",
- timestamp=1,
- content={"body": "a file"},
- body="a file",
- uri="mxc://server/file",
- )
-
- await AgentConsumer._emit_media(
- _consumer(db_buffer), # type: ignore[arg-type]
- RoomRef(room_id="!room:server"),
- media,
- SimpleNamespace( # type: ignore[arg-type]
- room_id="room-1", bridge_id=None, channel_type="channel_public"
- ),
- True,
- "someone",
- None,
- [],
- "a file",
- room_seq=1,
- message_key="$media",
- )
-
- assert len(await _enqueued(db_buffer)) == 1
- logged = [r for r in caplog.records if "Enqueuing media event" in r.getMessage()]
- if level == logging.DEBUG:
- assert dumps == [1]
- assert len(logged) == 1
- else:
- assert dumps == []
- assert logged == []
diff --git a/core/tests/switch_core/clients/test_ensure_system_client.py b/core/tests/switch_core/clients/test_ensure_system_client.py
index 86d65b39d..e39d5fad6 100644
--- a/core/tests/switch_core/clients/test_ensure_system_client.py
+++ b/core/tests/switch_core/clients/test_ensure_system_client.py
@@ -41,6 +41,7 @@ from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.tenant_store import TenantStore
from switch_core.tenant_context import current_tenant_id
from tests.conftest import RLSHarness
+from tests.ownership import NeverOwned
def _service(
@@ -54,6 +55,8 @@ def _service(
session_factory=session_factory,
config=SimpleNamespace(id_server_name="test"), # type: ignore[arg-type]
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
diff --git a/core/tests/switch_core/clients/test_removed_from_room.py b/core/tests/switch_core/clients/test_removed_from_room.py
index 6308928f2..c139c9a27 100644
--- a/core/tests/switch_core/clients/test_removed_from_room.py
+++ b/core/tests/switch_core/clients/test_removed_from_room.py
@@ -1,71 +1,16 @@
-"""What an agent client does when it is taken out of a room while running.
-
-Dropping the transport's subscription governs only what has not been read yet.
-Everything already read is in the event buffer, which is keyed by agent and
-knows nothing about who is in what: it holds each event for the retention
-window and serves it to a long poll, to the notification stream, and to an SSE
-reader resuming from an old cursor. So the removal has to reach the things
-holding the events, not only the reader that would have fetched more.
+"""What a reading client does when it is taken out of a room while running.
+
+Dropping the transport's subscription governs only what has not been read yet,
+so the removal is handed to the client as well (`Consumer.on_removed`). An
+agent reads no room: what its removal does is the router's
+(`tests/switch_core/delivery/test_removed_from_room.py`).
"""
from __future__ import annotations
-from types import SimpleNamespace
-
-import pytest
-from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-
-from switch_core.bridges.agent.protocol.agent_connections import (
- PROTOCOL_VERSION,
- AgentConnectionRegistry,
- ClientDeclaration,
-)
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.positions import OWN_EVENTS
-from switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload
from switch_core.clients.actor import Actor
-from switch_core.clients.agent_consumer import AgentConsumer, RoomMeta
from switch_core.clients.consumer import Consumer
-from switch_core.db.models import TENANT_ZERO_ID
from switch_core.transport import InboundMembership, RoomRef
-from tests.db_buffer import seed_agent
-
-AGENT = "agent-1"
-LEFT = "room-left"
-KEPT = "room-kept"
-
-
[email protected](autouse=True)
-async def _agent_row(session_factory: async_sessionmaker[AsyncSession]) -> None:
- """The agent whose events are retained, so they have a row to belong to."""
- await seed_agent(session_factory, AGENT)
-
-
-def _message(room_id: str, body: str) -> AgentEvent:
- return AgentEvent(
- type="message",
- room_id=room_id,
- payload=MessagePayload(
- addressed=True,
- sender="@someone:test",
- sender_name="someone",
- message_id=f"$m-{body}",
- body=body,
- timestamp=0,
- ),
- )
-
-
-async def _keep(buffer: EventBuffer, room_id: str, body: str) -> None:
- """A message addressed to the agent in `room_id`, kept as one of its own."""
- await buffer.enqueue(
- TENANT_ZERO_ID,
- AGENT,
- room_id,
- _message(room_id, body),
- room_seq=1,
- message_key=f"$m-{body}",
- )
def _leave(transport_room_id: str) -> InboundMembership:
@@ -80,87 +25,6 @@ def _leave(transport_room_id: str) -> InboundMembership:
)
-def _client(
- buffer: EventBuffer, connections: AgentConnectionRegistry
-) -> SimpleNamespace:
- """A minimal fake `self` for the unbound `AgentConsumer.on_removed`."""
- meta = {
- "!left:test": RoomMeta(
- room_id=LEFT,
- name="left",
- bridge_id=None,
- channel_type="channel_public",
- ),
- }
-
- async def _resolve_room_meta(transport_room_id: str) -> RoomMeta | None:
- return meta.get(transport_room_id)
-
- return SimpleNamespace(
- _event_buffer=buffer,
- _connections=connections,
- agent=SimpleNamespace(id=AGENT),
- tenant_id=TENANT_ZERO_ID,
- _resolve_room_meta=_resolve_room_meta,
- )
-
-
-async def _removed(stub: SimpleNamespace, transport_room_id: str) -> None:
- await AgentConsumer.on_removed(
- stub, # type: ignore[arg-type]
- RoomRef(room_id=transport_room_id),
- _leave(transport_room_id),
- )
-
-
-async def test_the_rooms_retained_events_are_forgotten(db_buffer: EventBuffer) -> None:
- buffer = db_buffer
- await _keep(buffer, LEFT, "said in the old room")
- await _keep(buffer, KEPT, "said in this one")
-
- await _removed(_client(buffer, AgentConnectionRegistry()), "!left:test")
-
- # Read at the level every reader is built on, filter or no filter: this is
- # what closes it for the stream, which has no membership of its own to
- # apply and would otherwise replay the room on every resume.
- assert [
- item.room_id for item in await buffer.read_from(TENANT_ZERO_ID, AGENT, 0)
- ] == [KEPT]
-
-
-async def test_the_rooms_claim_is_released(db_buffer: EventBuffer) -> None:
- connections = AgentConnectionRegistry()
- conn = connections.open(
- agent_id=AGENT,
- connection_id="c1",
- scope="single",
- delivery_filter="all",
- spawn_capable=False,
- position={OWN_EVENTS: 0},
- declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
- expected_generation=None,
- )
- connections.claim_room(conn, LEFT)
-
- await _removed(_client(db_buffer, connections), "!left:test")
-
- assert conn.rooms == set()
-
-
-async def test_a_room_that_cannot_be_resolved_drops_nothing(
- db_buffer: EventBuffer,
-) -> None:
- """Rather than guessing which room was meant and emptying the wrong one."""
- buffer = db_buffer
- await _keep(buffer, LEFT, "still here")
-
- await _removed(_client(buffer, AgentConnectionRegistry()), "!unknown:test")
-
- assert [
- item.room_id for item in await buffer.read_from(TENANT_ZERO_ID, AGENT, 0)
- ] == [LEFT]
-
-
class _CapturingTransport:
def __init__(self) -> None:
self.handlers: object | None = None
diff --git a/core/tests/switch_core/clients/test_room_join_events.py b/core/tests/switch_core/clients/test_room_join_events.py
deleted file mode 100644
index fbcc54708..000000000
--- a/core/tests/switch_core/clients/test_room_join_events.py
+++ /dev/null
@@ -1,229 +0,0 @@
-from __future__ import annotations
-
-from contextlib import asynccontextmanager
-from types import SimpleNamespace
-
-import pytest
-from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-
-from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
-from switch_core.bridges.agent.protocol.types import AgentEvent, RoomJoinPayload
-from switch_core.clients.agent_consumer import AgentConsumer, RoomMeta
-from switch_core.db.models import TENANT_ZERO_ID
-from switch_core.transport import InboundMembership, RoomRef
-from tests.db_buffer import seed_agent
-
-AGENT = "agent-1"
-
-
[email protected](autouse=True)
-async def _agent_row(session_factory: async_sessionmaker[AsyncSession]) -> None:
- """The agent the consumer queues for, so its events have a row to belong to."""
- await seed_agent(session_factory, AGENT)
-
-
-async def _enqueued(client: SimpleNamespace) -> list[tuple[str, AgentEvent]]:
- """What the consumer queued for its agent, read back from the db-buffer as
- (room, event). Reading under the agent's id is what checks which agent it
- was queued for."""
- buffer: EventBuffer = client._event_buffer
- return [
- (item.room_id, item.event)
- for item in await buffer.read_from(TENANT_ZERO_ID, AGENT, 0)
- ]
-
-
-class _FakeClientStore:
- """Resolves a matrix id to the Switch client that owns it, or nothing."""
-
- def __init__(self, names: dict[str, str]) -> None:
- self._names = names
-
- async def get_by_transport_user_id(
- self, _session: object, transport_user_id: str
- ) -> object | None:
- name = self._names.get(transport_user_id)
- return SimpleNamespace(display_name=name) if name else None
-
-
-class _FakeRoomStore:
- def __init__(self, receives: bool) -> None:
- self._receives = receives
-
- async def get_receives_join_events(
- self, _session: object, _room_id: str, _agent_id: str
- ) -> bool:
- return self._receives
-
-
-def _client(
- buffer: EventBuffer,
- meta: RoomMeta | None = None,
- *,
- receives: bool = True,
- known: dict[str, str] | None = None,
-) -> SimpleNamespace:
- """A minimal fake `self` for the unbound AgentConsumer.on_member_event.
-
- Queues on the db-buffer; stubs agent identity, room-meta resolution, and the room
- store / session factory so the handler can run without a DB or live Matrix
- client. `receives` is what the stubbed store reports for this agent's
- per-room join-event setting.
- """
- resolved = meta or RoomMeta(
- room_id="room-1",
- name="Room",
- bridge_id="bridge-1",
- channel_type="channel_public",
- )
-
- async def _resolve_room_meta(_transport_room_id: str) -> RoomMeta | None:
- return resolved
-
- @asynccontextmanager
- async def _session_factory():
- yield SimpleNamespace()
-
- stub = SimpleNamespace(
- _event_buffer=buffer,
- agent=SimpleNamespace(id=AGENT),
- tenant_id=TENANT_ZERO_ID,
- _resolve_room_meta=_resolve_room_meta,
- _room_store=_FakeRoomStore(receives),
- client_store=_FakeClientStore(
- {"@alice:switch.local": "Alice"} if known is None else known
- ),
- session_factory=_session_factory,
- )
- # Bound, so the handler resolves the name through the real implementation.
- stub._member_name = lambda session, event: AgentConsumer._member_name(
- stub, session, event
- )
- return stub
-
-
-def _member_event(
- *,
- membership: str = "join",
- prev_membership: str | None = "leave",
- state_key: str = "@alice:switch.local",
- displayname: str | None = "Alice",
- timestamp: int = 123,
-) -> InboundMembership:
- content: dict[str, object] = {}
- if displayname is not None:
- content["displayname"] = displayname
- return InboundMembership(
- room_id="!matrix:switch.local",
- event_id="$member",
- sender=state_key,
- timestamp=timestamp,
- content=content,
- state_key=state_key,
- membership=membership,
- prev_membership=prev_membership,
- display_name=displayname,
- )
-
-
-async def _run(client: SimpleNamespace, event: InboundMembership) -> None:
- await AgentConsumer.on_member_event(client, RoomRef("!matrix:switch.local"), event)
-
-
-class TestRoomJoinEnqueue:
- async def test_join_enqueues_room_join_event(self, db_buffer: EventBuffer) -> None:
- client = _client(db_buffer)
- await _run(client, _member_event())
-
- assert len(await _enqueued(client)) == 1
- room_id, event = (await _enqueued(client))[0]
- assert room_id == "room-1"
- assert event.type == "room_join"
- assert event.bridge_id == "bridge-1"
- assert event.channel_type == "channel_public"
- assert isinstance(event.payload, RoomJoinPayload)
- assert event.payload.member == "@alice:switch.local"
- assert event.payload.member_name == "Alice"
- assert event.payload.timestamp == 123
- # Stubbed store opts this agent in → listening is True.
- assert event.payload.listening is True
-
- async def test_listening_reflects_store_setting(
- self, db_buffer: EventBuffer
- ) -> None:
- # The per-room, per-agent receives_join_events setting (off here)
- # decides whether the arrival is one of the agent's own events. Off, it
- # is the room's conversation: nothing is queued, and a reader that
- # wants the conversation reads the arrival from the room's log, with
- # `listening` False.
- client = _client(db_buffer, receives=False)
- await _run(client, _member_event())
-
- assert await _enqueued(client) == []
-
- async def test_the_member_is_named_the_way_switch_knows_them(
- self, db_buffer: EventBuffer
- ) -> None:
- """A profile carries the source platform's spelling of a name; the log
- records the Switch one. Delivering the profile name would make live
- events and history disagree about the same person."""
- client = _client(db_buffer, known={"@alice:switch.local": "alice"})
- await _run(client, _member_event(displayname="Alice \U0001f495"))
-
- _, event = (await _enqueued(client))[0]
- assert event.payload.member_name == "alice"
-
- async def test_a_member_no_switch_client_owns_falls_back_to_the_event(
- self, db_buffer: EventBuffer
- ) -> None:
- client = _client(db_buffer, known={})
- await _run(
- client, _member_event(displayname=None, state_key="@bob:switch.local")
- )
-
- assert len(await _enqueued(client)) == 1
- _, event = (await _enqueued(client))[0]
- assert event.payload.member_name == "bob"
-
- async def test_leave_does_not_enqueue(self, db_buffer: EventBuffer) -> None:
- client = _client(db_buffer)
- await _run(client, _member_event(membership="leave", prev_membership="join"))
- assert await _enqueued(client) == []
-
- async def test_profile_update_does_not_enqueue(
- self, db_buffer: EventBuffer
- ) -> None:
- # membership-preserving update (e.g. display-name change): join -> join.
- client = _client(db_buffer)
- await _run(client, _member_event(membership="join", prev_membership="join"))
- assert await _enqueued(client) == []
-
- async def test_no_room_meta_does_not_enqueue(self, db_buffer: EventBuffer) -> None:
- async def _none(_transport_room_id: str) -> RoomMeta | None:
- return None
-
- client = _client(db_buffer)
- client._resolve_room_meta = _none
- await _run(client, _member_event())
- assert await _enqueued(client) == []
-
-
-class TestRoomJoinPayloadRegistry:
- def test_agent_event_parses_room_join_payload(self) -> None:
- event = AgentEvent.model_validate(
- {
- "type": "room_join",
- "room_id": "room-1",
- "bridge_id": None,
- "channel_type": "channel_public",
- "payload": {
- "member": "@alice:switch.local",
- "member_name": "Alice",
- "timestamp": 123,
- "listening": True,
- },
- }
- )
- assert isinstance(event.payload, RoomJoinPayload)
- assert event.payload.member_name == "Alice"
- assert event.payload.listening is True
diff --git a/core/tests/switch_core/clients/test_self_join_dispatch.py b/core/tests/switch_core/clients/test_self_join_dispatch.py
index 6bd624195..e18cad90b 100644
--- a/core/tests/switch_core/clients/test_self_join_dispatch.py
+++ b/core/tests/switch_core/clients/test_self_join_dispatch.py
@@ -1,8 +1,7 @@
"""The self-join hook must still fire when the client joined the room itself.
-`join_room` records membership as soon as the join returns so `wait_joined` and
-the `_should_ignore` cutoff are accurate (CHOO-1781). Auto-accepting an invite
-goes through it, so by the time the join arrives over sync the room is already
+`join_room` records membership as soon as the join returns so `wait_joined` is
+accurate (CHOO-1781). Auto-accepting an invite goes through it, so by the time the join arrives over sync the room is already
in `room_join_times` — CHOO-2170: gating the hook on that made the agent
greeting unreachable on every room an agent is invited to, which is all of them.
"""
@@ -29,7 +28,6 @@ class _Client(Consumer):
actor._room_joined_events = {}
self.actor = actor
self._self_join_dispatched = set()
- self._startup_ts = 1000
self.self_joins: list[str] = []
self.member_events: list[str] = []
@@ -97,12 +95,15 @@ async def test_profile_update_is_not_an_arrival() -> None:
@pytest.mark.asyncio
-async def test_join_predating_this_process_is_not_announced() -> None:
+async def test_a_join_handed_over_late_is_still_announced() -> None:
+ """Whether a join is news is the cursor's call, not the clock's. A join
+ the client had not handled when its last reader stopped is handed to the
+ next one, however long ago it was, and is announced there."""
client = _Client()
await client._handle_member_event(_room(), _member_event(timestamp=500))
- assert client.self_joins == []
+ assert client.self_joins == [TRANSPORT_ROOM_ID]
@pytest.mark.asyncio
@@ -130,17 +131,3 @@ async def test_another_member_joining_is_not_a_self_join() -> None:
assert client.self_joins == []
assert client.member_events == [other]
-
-
-async def test_an_arrival_is_logged_even_when_it_is_not_announced() -> None:
- """Announcing and logging answer different questions.
-
- A join older than this process is nobody's news, so it is not announced.
- It is still how that participant got into the room, so it is still history.
- """
- client = _Client()
-
- await client._handle_member_event(_room(), _member_event(timestamp=500))
-
- assert client.self_joins == []
diff --git a/core/tests/switch_core/clients/test_start_all_preloads_agents.py b/core/tests/switch_core/clients/test_start_all_preloads_agents.py
index 350060fdb..04cb2364f 100644
--- a/core/tests/switch_core/clients/test_start_all_preloads_agents.py
+++ b/core/tests/switch_core/clients/test_start_all_preloads_agents.py
@@ -1,5 +1,5 @@
-"""`start_all` reads every agent row once per tenant and hands each consumer
-its own, so no agent client queries for itself at boot."""
+"""`start_all` reads every agent row once per tenant and hands each agent's
+actor its own, so no agent client queries for itself at boot."""
from __future__ import annotations
@@ -14,10 +14,14 @@ from switch_core.clients.client_lifecycle_service import ClientLifecycleService
from switch_core.db.models import Agent
from switch_core.db.stores.client_store import ClientStore
from switch_core.db.stores.tenant_store import TenantStore
+from tests.ownership import NeverOwned
from tests.switch_core.test_room_wide_mention_wakes_no_agent import _make_agent
-class _Consumer:
+class _Actor:
+ """An agent's actor: it reads no room, so the lifecycle connects it and
+ builds no consumer for it."""
+
def __init__(self) -> None:
self.display_name = "agent"
self.transport_user_id = "@agent:test"
@@ -26,24 +30,30 @@ class _Consumer:
def preload_agent(self, agent: Agent) -> None:
self.preloaded = agent
- async def start(self) -> None:
+ async def connect(self) -> None:
return None
- async def stop(self) -> None:
+ async def close(self) -> None:
return None
class _Factory:
def __init__(self) -> None:
- self.consumers: dict[str, _Consumer] = {}
+ self.actors: dict[str, _Actor] = {}
+
+ def create_actor(self, record: Any) -> _Actor:
+ actor = _Actor()
+ self.actors[record.id] = actor
+ return actor
+
+ def create_consumer(self, client_type: str, actor: _Actor) -> None:
+ return None
- def create(self, record: Any) -> tuple[_Consumer, _Consumer]:
- consumer = _Consumer()
- self.consumers[record.id] = consumer
- return consumer, consumer
+ def has_consumer(self, client_type: str) -> bool:
+ return False
-async def test_each_agent_consumer_gets_its_own_row_before_it_starts(
+async def test_each_agent_actor_gets_its_own_row_before_it_connects(
session_factory: async_sessionmaker[AsyncSession],
) -> None:
async with session_factory() as session:
@@ -58,12 +68,14 @@ async def test_each_agent_consumer_gets_its_own_row_before_it_starts(
session_factory=session_factory,
config=MagicMock(),
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
await service.start_all()
for agent in agents:
- preloaded = factory.consumers[agent.client_id].preloaded
+ preloaded = factory.actors[agent.client_id].preloaded
assert preloaded is not None and preloaded.id == agent.id
@@ -85,6 +97,8 @@ async def test_a_failed_bulk_read_still_starts_every_client(
session_factory=session_factory,
config=MagicMock(),
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
async def _fail(_records: Any) -> dict[str, Agent]:
@@ -95,5 +109,5 @@ async def test_a_failed_bulk_read_still_starts_every_client(
await service.start_all()
for agent in agents:
- assert agent.client_id in factory.consumers
- assert factory.consumers[agent.client_id].preloaded is None
+ assert agent.client_id in factory.actors
+ assert factory.actors[agent.client_id].preloaded is None
diff --git a/core/tests/switch_core/connections/test_sign_in_flows.py b/core/tests/switch_core/connections/test_sign_in_flows.py
new file mode 100644
index 000000000..1a2947e13
--- /dev/null
+++ b/core/tests/switch_core/connections/test_sign_in_flows.py
@@ -0,0 +1,244 @@
+"""Sign-in flows in Postgres, as two switch-core processes see them.
+
+Through the restricted runtime role, so the tenant policies apply: the
+browser's steps find a flow's tenant through `tenant_of_sign_in_flow` and
+nothing else, and a superuser connection would pass whether or not that
+worked.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+
+import pytest
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.connections.sign_in_flows import (
+ MAX_FLOWS,
+ FlowGoneError,
+ FlowsBusyError,
+ FlowUsedError,
+ SignInFlows,
+ StoredFlow,
+ digest,
+)
+from switch_core.db.models import SignInFlow, Tenant, User
+from tests.conftest import TEST_KEYRING, RLSHarness
+
+pytestmark = pytest.mark.no_ambient_tenant
+
+
+async def _tenants_and_user(rls_harness: RLSHarness) -> tuple[str, str, str]:
+ suffix = uuid.uuid4().hex[:8]
+ tenant_a, tenant_b = f"a-{suffix}", f"b-{suffix}"
+ async with rls_harness.owner() as session:
+ for tenant_id in (tenant_a, tenant_b):
+ session.add(Tenant(id=tenant_id, slug=tenant_id, name=tenant_id))
+ await session.flush()
+ user = User(name="Ada", email=f"ada-{suffix}@example.test", role="user")
+ session.add(user)
+ await session.commit()
+ return tenant_a, tenant_b, user.id
+
+
+async def _person(rls_harness: RLSHarness, name: str) -> str:
+ async with rls_harness.owner() as session:
+ user = User(
+ name=name,
+ email=f"{name.lower()}-{uuid.uuid4().hex[:8]}@example.test",
+ role="user",
+ )
+ session.add(user)
+ await session.commit()
+ return user.id
+
+
+async def _start(
+ flows: SignInFlows, tenant_id: str, user_id: str, state: str
+) -> StoredFlow:
+ return await flows.start(
+ kind="service",
+ service="example",
+ tenant_id=tenant_id,
+ user_id=user_id,
+ state=state,
+ mode="core",
+ port=43123,
+ redirect_uri="https://switch.example.com/callback",
+ owner_label="Ada in A",
+ details={"name": "Example"},
+ completion_secret="completion-secret",
+ )
+
+
+class _HeldBeforeCommit:
+ """Sessions that stop just before they commit until let go, with their
+ transaction, and any lock taken in it, still open."""
+
+ def __init__(self, factory: async_sessionmaker[AsyncSession]) -> None:
+ self._factory = factory
+ self.reached = asyncio.Event()
+ self.release = asyncio.Event()
+
+ def __call__(self) -> AsyncSession:
+ session = self._factory()
+ commit = session.commit
+
+ async def held() -> None:
+ self.reached.set()
+ await self.release.wait()
+ await commit()
+
+ session.commit = held # type: ignore[method-assign]
+ return session
+
+
+async def _start_beside_a_held_one(
+ rls_harness: RLSHarness,
+ tenant_id: str,
+ held: tuple[str, str],
+ other: tuple[str, str],
+) -> list[StoredFlow | BaseException]:
+ """Start two flows in `tenant_id` from two processes, each a
+ `(user_id, state)`. The first is held open just before it commits, long
+ enough for the second to finish if nothing makes it wait for the first."""
+ gate = _HeldBeforeCommit(rls_harness.restricted)
+ first = asyncio.create_task(
+ _start(SignInFlows(gate, TEST_KEYRING), tenant_id, *held) # type: ignore[arg-type]
+ )
+ await asyncio.wait_for(gate.reached.wait(), 5)
+ second = asyncio.create_task(
+ _start(SignInFlows(rls_harness.restricted, TEST_KEYRING), tenant_id, *other)
+ )
+ await asyncio.sleep(0.3)
+ gate.release.set()
+ return await asyncio.gather(first, second, return_exceptions=True)
+
+
+async def test_a_flow_started_by_one_process_is_found_by_another(
+ rls_harness: RLSHarness,
+) -> None:
+ tenant_a, tenant_b, user_id = await _tenants_and_user(rls_harness)
+ first = SignInFlows(rls_harness.restricted, TEST_KEYRING)
+ second = SignInFlows(rls_harness.restricted, TEST_KEYRING)
+
+ started = await _start(first, tenant_a, user_id, "state-1")
+ # The browser's step carries only the state, and reaches the other process.
+ found = await second.find("state-1")
+
+ assert found.tenant_id == tenant_a
+ assert found.verifier == started.verifier
+ assert found.holds_secret("completion-secret")
+ assert not found.holds_secret("something-else")
+ with pytest.raises(FlowGoneError):
+ await second.get("state-1", tenant_b)
+ with pytest.raises(FlowGoneError):
+ await second.find("no-such-state")
+
+
+async def test_two_processes_racing_for_one_step_take_it_once(
+ rls_harness: RLSHarness,
+) -> None:
+ tenant_a, _, user_id = await _tenants_and_user(rls_harness)
+ first = SignInFlows(rls_harness.restricted, TEST_KEYRING)
+ second = SignInFlows(rls_harness.restricted, TEST_KEYRING)
+ flow = await _start(first, tenant_a, user_id, "state-2")
+
+ taken = await asyncio.gather(
+ first.advance(flow, expect=("pending",), to="authorizing", changes={}),
+ second.advance(flow, expect=("pending",), to="authorizing", changes={}),
+ )
+
+ assert sorted(result is None for result in taken) == [False, True]
+
+
+async def test_two_starts_by_one_person_side_by_side_leave_one_flow(
+ rls_harness: RLSHarness,
+) -> None:
+ """Neither start sees the other's flow while it is uncommitted, so unless
+ the second waits for the first, both delete nothing and both insert."""
+ tenant_a, _, user_id = await _tenants_and_user(rls_harness)
+
+ started = await _start_beside_a_held_one(
+ rls_harness, tenant_a, (user_id, "state-5"), (user_id, "state-6")
+ )
+
+ assert not [result for result in started if isinstance(result, BaseException)]
+ async with rls_harness.owner() as session:
+ states = await session.scalars(
+ select(SignInFlow.id).where(SignInFlow.tenant_id == tenant_a)
+ )
+ assert states.all() == ["state-6"]
+
+
+async def test_two_starts_side_by_side_at_the_cap_let_one_in(
+ rls_harness: RLSHarness,
+) -> None:
+ """Each counts the workspace's flows before the other's is committed, so
+ unless the second waits for the first, both take the one place left."""
+ tenant_a, _, user_id = await _tenants_and_user(rls_harness)
+ await _start(
+ SignInFlows(rls_harness.restricted, TEST_KEYRING), tenant_a, user_id, "state-7"
+ )
+ async with rls_harness.owner() as session:
+ template = await session.scalar(select(SignInFlow))
+ assert template is not None
+ columns = {
+ column.key: getattr(template, column.key)
+ for column in SignInFlow.__table__.columns
+ }
+ for i in range(MAX_FLOWS - 2):
+ session.add(SignInFlow(**{**columns, "id": f"filler-{i}"}))
+ await session.commit()
+ grace = await _person(rls_harness, "Grace")
+ alan = await _person(rls_harness, "Alan")
+
+ started = await _start_beside_a_held_one(
+ rls_harness, tenant_a, (grace, "state-8"), (alan, "state-9")
+ )
+
+ assert [type(result) for result in started].count(FlowsBusyError) == 1
+ async with rls_harness.owner() as session:
+ assert (
+ await session.scalar(
+ select(func.count())
+ .select_from(SignInFlow)
+ .where(SignInFlow.tenant_id == tenant_a)
+ )
+ == MAX_FLOWS
+ )
+
+
+async def test_a_state_is_used_once_even_in_another_workspace(
+ rls_harness: RLSHarness,
+) -> None:
+ tenant_a, tenant_b, user_id = await _tenants_and_user(rls_harness)
+ flows = SignInFlows(rls_harness.restricted, TEST_KEYRING)
+ await _start(flows, tenant_a, user_id, "state-3")
+
+ with pytest.raises(FlowUsedError):
+ await _start(flows, tenant_b, user_id, "state-3")
+
+
+async def test_secrets_are_stored_encrypted_or_hashed(
+ rls_harness: RLSHarness,
+) -> None:
+ tenant_a, _, user_id = await _tenants_and_user(rls_harness)
+ flows = SignInFlows(rls_harness.restricted, TEST_KEYRING)
+ flow = await _start(flows, tenant_a, user_id, "state-4")
+ await flows.advance(
+ flow,
+ expect=("pending",),
+ to="ready",
+ changes={"encrypted_result": flows.seal({"refresh_token": "SYNTHETIC"})},
+ )
+
+ async with rls_harness.owner() as session:
+ row = await session.scalar(select(SignInFlow).where(SignInFlow.id == "state-4"))
+ assert row is not None
+ assert flow.verifier not in row.encrypted_verifier
+ assert row.completion_secret_hash == digest("completion-secret")
+ assert "SYNTHETIC" not in (row.encrypted_result or "")
+ assert (await flows.find("state-4")).result == {"refresh_token": "SYNTHETIC"}
diff --git a/core/tests/switch_core/db/stores/test_bridge_message_map_store.py b/core/tests/switch_core/db/stores/test_bridge_message_map_store.py
index e6fdde730..af6e22fc9 100644
--- a/core/tests/switch_core/db/stores/test_bridge_message_map_store.py
+++ b/core/tests/switch_core/db/stores/test_bridge_message_map_store.py
@@ -1,7 +1,9 @@
from __future__ import annotations
import uuid
+from datetime import timedelta
+from sqlalchemy import func, update
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.db.models import BridgeMessageMap, Client, CollaborationBridge
@@ -122,3 +124,46 @@ class TestBridgeMessageMapStore:
assert (
await store.get_by_external_post_id(session, bridge_b, "post1") is None
)
+
+ async def test_a_stale_claim_claimed_again_records_the_new_channel(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """The holder that claimed the message died before posting it, and
+ the room moved to another channel of the bridge since. The next
+ holder posts it to the new channel, and the row says so."""
+ store = BridgeMessageMapStore()
+ unposted_after = timedelta(seconds=10)
+ async with session_factory() as session:
+ bridge_id = await _make_bridge(session)
+ assert await store.claim(
+ session,
+ bridge_id=bridge_id,
+ external_channel_id="chan-1",
+ transport_event_id="$evt1",
+ unposted_after=unposted_after,
+ )
+ await session.execute(
+ update(BridgeMessageMap)
+ .where(BridgeMessageMap.bridge_id == bridge_id)
+ .values(created_at=func.now() - timedelta(seconds=30))
+ )
+ await session.commit()
+
+ assert await store.claim(
+ session,
+ bridge_id=bridge_id,
+ external_channel_id="chan-2",
+ transport_event_id="$evt1",
+ unposted_after=unposted_after,
+ )
+ await store.fill(
+ session,
+ bridge_id=bridge_id,
+ transport_event_id="$evt1",
+ external_post_id="post1",
+ )
+ await session.commit()
+
+ async with session_factory() as session:
+ posted = await store.get_by_external_post_id(session, bridge_id, "post1")
+ assert posted is not None and posted.external_channel_id == "chan-2"
diff --git a/core/tests/switch_core/db/stores/test_delivery_cursor_store.py b/core/tests/switch_core/db/stores/test_delivery_cursor_store.py
index 7da3a46f1..d582d4d50 100644
--- a/core/tests/switch_core/db/stores/test_delivery_cursor_store.py
+++ b/core/tests/switch_core/db/stores/test_delivery_cursor_store.py
@@ -1,148 +1,275 @@
-"""A delivery cursor moves forward and only forward."""
+"""A client's delivery cursor in a room: written with the membership, moved
+only forward, only by the process holding the client's lease, and deleted
+with the membership."""
from __future__ import annotations
import uuid
+from datetime import timedelta
+from sqlalchemy import func, select, update
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
-from switch_core.db.models import Agent, ApiKey, Client, Room, User
-from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
+from switch_core.db.models import (
+ Client,
+ ClientRoom,
+ DeliveryCursor,
+ Message,
+ Room,
+ SwitchCoreProcess,
+)
+from switch_core.db.stores.delivery_cursor_store import (
+ CursorAdvance,
+ DeliveryCursorStore,
+)
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.process_lease import LEASE_TTL
+KIND = "client"
-async def _agent(session: AsyncSession) -> Agent:
- """Minimal User → ApiKey → Client → Agent chain."""
- name = f"agent-{uuid.uuid4().hex[:8]}"
- user = User(name=name, email=f"{name}@test", role="user", password_hash="x")
- session.add(user)
- await session.flush()
- api_key = ApiKey(
- user_id=user.id,
- key_hash=f"hash-{name}",
- encrypted_key="enc",
- label=name,
- type="agent",
- )
+
+async def _client_and_room(session: AsyncSession) -> tuple[str, str]:
client = Client(
- transport_user_id=f"@{name}:test",
- display_name=name,
+ transport_user_id=f"@c-{uuid.uuid4().hex[:8]}:test",
+ display_name="client",
type="agent",
)
- session.add_all([api_key, client])
- await session.flush()
- agent = Agent(
- name=name,
- description="",
- agent_type="always_on",
- connector_type="claude_code",
- integration_profile={"connection_model": "always_on"},
- client_id=client.id,
- api_key_id=api_key.id,
- )
- session.add(agent)
- await session.flush()
- return agent
-
-
-async def _make_agent_and_room(session: AsyncSession) -> tuple[str, str]:
room = Room(
transport_room_id=f"!room-{uuid.uuid4().hex[:8]}:test",
name="a room",
description="",
)
- session.add(room)
- agent = await _agent(session)
+ session.add_all([client, room])
await session.flush()
- return agent.id, room.id
+ return client.id, room.id
+
+
+async def _say(session: AsyncSession, room_id: str, body: str = "hi") -> int:
+ message = await MessageStore().create(
+ session,
+ Message(
+ room_id=room_id,
+ transport_event_id=f"$e-{uuid.uuid4().hex}",
+ sender_id="@someone:test",
+ sender_name="someone",
+ event_type="m.room.message",
+ msgtype="m.text",
+ body=body,
+ formatted_body=None,
+ content={"body": body},
+ ),
+ [],
+ )
+ return message.seq
+
+async def _hold(session: AsyncSession, key: str, process_id: str) -> int:
+ """Claim `key`'s lease for `process_id`, as its ownership would, and
+ return the token."""
+ await SwitchCoreProcessStore().renew(session, process_id, key_ids=[])
+ claimed = await OwnershipLeaseStore().claim(
+ session, kind=KIND, keys=[key], process_id=process_id, lease_ttl=LEASE_TTL
+ )
+ return claimed[key]
+
+
+def _advance(client_id: str, room_id: str, seq: int, token: int) -> CursorAdvance:
+ return CursorAdvance(
+ client_id=client_id,
+ room_id=room_id,
+ seq=seq,
+ kind=KIND,
+ key=client_id,
+ token=token,
+ )
-class TestPosition:
- async def test_an_agent_that_was_never_delivered_is_at_zero(
+
+class TestWrittenWithTheMembership:
+ async def test_a_membership_starts_its_cursor_at_the_room_head(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
- """Zero, not an error: live messages are numbered from 1, so zero
- already says "behind every live message"."""
+ """So a consumer that learns of the membership later, in another
+ process, still starts reading where the client joined."""
async with session_factory() as session:
- agent_id, room_id = await _make_agent_and_room(session)
+ client_id, room_id = await _client_and_room(session)
+ await _say(session, room_id)
+ head = await _say(session, room_id)
+ await RoomStore().add_client(session, client_id, room_id)
await session.commit()
- store = DeliveryCursorStore()
- assert await store.position(session, agent_id, room_id) == 0
+ places = await DeliveryCursorStore().positions(
+ session, client_id, [room_id]
+ )
- async def test_a_cursor_reads_back_where_it_was_left(
+ assert places == {room_id: head}
+
+ async def test_an_existing_membership_keeps_its_cursor(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
async with session_factory() as session:
- agent_id, room_id = await _make_agent_and_room(session)
- store = DeliveryCursorStore()
- await store.advance(session, agent_id, room_id, 7)
+ client_id, room_id = await _client_and_room(session)
+ await RoomStore().add_client(session, client_id, room_id)
+ await _say(session, room_id)
+ await RoomStore().add_client(session, client_id, room_id)
+ await session.commit()
+
+ places = await DeliveryCursorStore().positions(
+ session, client_id, [room_id]
+ )
+
+ assert places == {room_id: 0}
+
+ async def test_leaving_the_room_deletes_the_cursor(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """A client added back starts from its new arrival, never from where
+ it had got to before it left."""
+ async with session_factory() as session:
+ client_id, room_id = await _client_and_room(session)
+ await RoomStore().add_client(session, client_id, room_id)
+ await RoomStore().remove_client(session, client_id, room_id)
await session.commit()
- assert await store.position(session, agent_id, room_id) == 7
+ places = await DeliveryCursorStore().positions(
+ session, client_id, [room_id]
+ )
- async def test_positions_omit_agents_that_have_none(
+ assert places == {}
+
+ async def test_memberships_without_a_cursor_are_seeded_in_one_statement(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
- """Absent rather than zero, so a caller can tell an agent that has
- never been delivered from one sitting at the start."""
+ """Memberships written before cursors were get one at their room's
+ head; a room the client is not in gets none."""
async with session_factory() as session:
- first, room_id = await _make_agent_and_room(session)
- second = await _agent(session)
- session.add(second)
+ client_id, member_of = await _client_and_room(session)
+ _, not_in = await _client_and_room(session)
+ # Written the way memberships were before cursors were: no cursor.
+ session.add(ClientRoom(client_id=client_id, room_id=member_of))
+ head = await _say(session, member_of)
await session.flush()
store = DeliveryCursorStore()
- await store.advance(session, first, room_id, 3)
+ await store.seed_memberships(session, client_id, [member_of, not_in])
await session.commit()
- positions = await store.positions_in(session, room_id, [first, second.id])
+ places = await store.positions(session, client_id, [member_of, not_in])
- assert positions == {first: 3}
+ assert places == {member_of: head}
class TestAdvancing:
- async def test_advancing_twice_keeps_the_further_position(
+ async def test_the_holder_moves_the_cursor_forward_and_never_back(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
async with session_factory() as session:
- agent_id, room_id = await _make_agent_and_room(session)
+ client_id, room_id = await _client_and_room(session)
+ await RoomStore().add_client(session, client_id, room_id)
+ token = await _hold(session, client_id, "process-a")
store = DeliveryCursorStore()
- await store.advance(session, agent_id, room_id, 4)
- await store.advance(session, agent_id, room_id, 9)
+
+ written = await store.advance(
+ session,
+ process_id="process-a",
+ advances=[_advance(client_id, room_id, 9, token)],
+ )
+ await store.advance(
+ session,
+ process_id="process-a",
+ advances=[_advance(client_id, room_id, 4, token)],
+ )
await session.commit()
- assert await store.position(session, agent_id, room_id) == 9
+ places = await store.positions(session, client_id, [room_id])
+
+ assert written == {(client_id, room_id)}
+ assert places == {room_id: 9}
- async def test_a_late_write_cannot_rewind_the_cursor(
+ async def test_a_process_that_was_taken_over_is_refused(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
- """Two deliveries can finish out of order. The later write must not
- undo the further one, because rewinding redelivers and a redelivered
- message is indistinguishable from a new one."""
+ """Its hold lapsed and another process claimed the client: whatever
+ it still tries to record is refused, so the new holder's progress is
+ the only one."""
async with session_factory() as session:
- agent_id, room_id = await _make_agent_and_room(session)
+ client_id, room_id = await _client_and_room(session)
+ await RoomStore().add_client(session, client_id, room_id)
+ stale = await _hold(session, client_id, "process-a")
+ # Process a stops renewing; process b takes the client over.
+ await session.execute(
+ update(SwitchCoreProcess)
+ .where(SwitchCoreProcess.id == "process-a")
+ .values(beat_at=func.now() - timedelta(minutes=1))
+ )
+ fresh = await _hold(session, client_id, "process-b")
store = DeliveryCursorStore()
- await store.advance(session, agent_id, room_id, 9)
- await store.advance(session, agent_id, room_id, 4)
+
+ refused = await store.advance(
+ session,
+ process_id="process-a",
+ advances=[_advance(client_id, room_id, 7, stale)],
+ )
+ written = await store.advance(
+ session,
+ process_id="process-b",
+ advances=[_advance(client_id, room_id, 3, fresh)],
+ )
+ await session.commit()
+
+ places = await store.positions(session, client_id, [room_id])
+
+ assert fresh == stale + 1
+ assert refused == set()
+ assert written == {(client_id, room_id)}
+ assert places == {room_id: 3}
+
+ async def test_a_room_the_client_has_left_is_not_written(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ async with session_factory() as session:
+ client_id, room_id = await _client_and_room(session)
+ await RoomStore().add_client(session, client_id, room_id)
+ token = await _hold(session, client_id, "process-a")
+ await RoomStore().remove_client(session, client_id, room_id)
+
+ written = await DeliveryCursorStore().advance(
+ session,
+ process_id="process-a",
+ advances=[_advance(client_id, room_id, 5, token)],
+ )
await session.commit()
+ rows = (await session.scalars(select(DeliveryCursor))).all()
+
+ assert written == set()
+ assert rows == []
- assert await store.position(session, agent_id, room_id) == 9
- async def test_cursors_in_different_rooms_are_independent(
+class TestBacklog:
+ async def test_the_end_of_what_is_too_old_to_replay(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
async with session_factory() as session:
- agent_id, first_room = await _make_agent_and_room(session)
- second_room = Room(
- transport_room_id=f"!room-{uuid.uuid4().hex[:8]}:test",
- name="another room",
- description="",
+ client_id, room_id = await _client_and_room(session)
+ old_one = await _say(session, room_id, "old")
+ old_two = await _say(session, room_id, "old too")
+ await _say(session, room_id, "new")
+ await session.execute(
+ update(Message)
+ .where(Message.room_id == room_id, Message.seq <= old_two)
+ .values(sent_at=func.now() - timedelta(hours=1))
)
- session.add(second_room)
- await session.flush()
+ await session.commit()
store = DeliveryCursorStore()
- await store.advance(session, agent_id, first_room, 5)
- await session.commit()
+ ends = await store.backlog_ends(
+ session, {room_id: old_one - 1}, timedelta(minutes=10)
+ )
+ none_past = await store.backlog_ends(
+ session, {room_id: old_two}, timedelta(minutes=10)
+ )
- assert await store.position(session, agent_id, second_room.id) == 0
+ assert ends == {room_id: old_two}
+ assert none_past == {}
diff --git a/core/tests/switch_core/db/test_connection_budget.py b/core/tests/switch_core/db/test_connection_budget.py
new file mode 100644
index 000000000..6926fb230
--- /dev/null
+++ b/core/tests/switch_core/db/test_connection_budget.py
@@ -0,0 +1,83 @@
+"""The boot check of how many connections the deployment's processes can hold
+against what Postgres allows, on a real server so `SHOW` answers as it does in
+production."""
+
+from __future__ import annotations
+
+import logging
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.config import SwitchConfig
+from switch_core.db.connection_budget import (
+ LISTENING_CONNECTIONS,
+ ConnectionBudgetError,
+ check_connection_budget,
+ per_process,
+ processes_at_once,
+)
+from tests.switch_core.observability.test_bootstrap import _config
+
+
+def test_one_replica_is_replaced_and_several_are_rolled() -> None:
+ assert processes_at_once(1) == 1
+ assert processes_at_once(3) == 4
+
+
+def test_a_process_holds_its_pool_and_its_listeners(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ config = _config(monkeypatch, DB_POOL_SIZE="10", DB_MAX_OVERFLOW="5")
+ assert per_process(config) == 15 + LISTENING_CONNECTIONS
+
+
+def test_exporting_metrics_adds_the_sampler(monkeypatch: pytest.MonkeyPatch) -> None:
+ config = _config(
+ monkeypatch,
+ DB_POOL_SIZE="10",
+ DB_MAX_OVERFLOW="5",
+ OTLP_ENDPOINT="https://collector.example",
+ DEPLOYMENT_ID="0e5d1b3a-6c1f-4c22-9a4c-3a9f5a2b7d10",
+ )
+ assert per_process(config) == 15 + LISTENING_CONNECTIONS + 1
+
+
+def _sized(monkeypatch: pytest.MonkeyPatch, replicas: int, pool: int) -> SwitchConfig:
+ return _config(
+ monkeypatch,
+ CORE_REPLICAS=str(replicas),
+ DB_POOL_SIZE=str(pool),
+ DB_MAX_OVERFLOW="0",
+ )
+
+
+async def test_a_deployment_within_budget_starts(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ await check_connection_budget(
+ _sized(monkeypatch, replicas=2, pool=5), session_factory.kw["bind"]
+ )
+
+
+async def test_several_replicas_over_budget_refuse_to_start(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ with pytest.raises(ConnectionBudgetError, match="3 switch-core process"):
+ await check_connection_budget(
+ _sized(monkeypatch, replicas=2, pool=1000), session_factory.kw["bind"]
+ )
+
+
+async def test_one_replica_over_budget_warns_and_starts(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ with caplog.at_level(logging.WARNING, logger="switch_core.db.connection_budget"):
+ await check_connection_budget(
+ _sized(monkeypatch, replicas=1, pool=1000), session_factory.kw["bind"]
+ )
+ assert "more database connections than Postgres allows" in caplog.text
diff --git a/core/tests/switch_core/db/test_key_rotation_rollout.py b/core/tests/switch_core/db/test_key_rotation_rollout.py
new file mode 100644
index 000000000..f3a808824
--- /dev/null
+++ b/core/tests/switch_core/db/test_key_rotation_rollout.py
@@ -0,0 +1,241 @@
+"""Rolling a new front key out to several processes, in two phases.
+
+Each process records the ids of its keys on its lease; one starting with a
+front key that a live process lacks refuses to start, since what it writes
+under that key would be unreadable to the other.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Any
+
+import pytest
+from sqlalchemy import select, text, update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.bridges.agent.registration_bootstrap import BOOTSTRAP_KEY_TYPE
+from switch_core.config import SwitchConfig
+from switch_core.db.key_rotation import (
+ KeyRotationError,
+ reencrypt_stale_text,
+ require_front_key_readable,
+)
+from switch_core.db.models import ApiKey, TrustSettings
+from switch_core.db.stores.agent_store import AgentStore
+from switch_core.db.stores.api_key_store import ApiKeyStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.db.stores.user_store import UserStore
+from switch_core.keys import Keyring
+from switch_core.main import _seed_admin_and_bootstrap_key
+from switch_core.process_lease import ProcessLease
+
+OLD = "old:" + "o" * 40
+NEW = "new:" + "n" * 40
+
+
+async def _running(
+ session_factory: async_sessionmaker[AsyncSession],
+ process_id: str,
+ key_ids: list[str],
+) -> ProcessLease:
+ lease = ProcessLease(
+ process_id=process_id,
+ session_factory=session_factory,
+ processes=SwitchCoreProcessStore(),
+ key_ids=key_ids,
+ )
+ await lease.renew()
+ return lease
+
+
+def _keyring(value: str) -> Keyring:
+ return Keyring.parse(value, legacy_secret=None)
+
+
+async def _boot_seeding(
+ session_factory: async_sessionmaker[AsyncSession],
+ process_id: str,
+ secret_keys: str,
+ registration_token: str,
+) -> None:
+ """The seeding a process does at boot, with these keys and token."""
+ config = SwitchConfig(
+ db_host="unused",
+ db_port="5432",
+ db_user="unused",
+ db_password="unused",
+ db_name="unused",
+ id_server_name="test",
+ agent_registration_token=registration_token,
+ secret_keys=secret_keys,
+ gateway_admin_email="[email protected]",
+ gateway_admin_password="unused",
+ )
+ await _seed_admin_and_bootstrap_key(
+ session_factory, UserStore(), ApiKeyStore(), AgentStore(), config, process_id
+ )
+
+
+async def test_a_new_front_key_another_process_lacks_is_refused(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ await _running(session_factory, "old-pod", ["old"])
+ starting = _keyring(f"{NEW},{OLD}")
+ await _running(session_factory, "new-pod", ["new", "old"])
+
+ with pytest.raises(KeyRotationError, match="two phases"):
+ await require_front_key_readable(session_factory, starting, "new-pod")
+
+
+async def test_a_refused_process_has_encrypted_nothing_under_its_front_key(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ """The boot seeding rewrites the agent-registration bootstrap key under
+ the front key when AGENT_REGISTRATION_TOKEN changes. A process refused
+ only after that would already have stored a value the old one cannot
+ decrypt."""
+ await _running(session_factory, "old-pod", ["old"])
+ await _boot_seeding(session_factory, "old-pod", OLD, "first-token")
+
+ with pytest.raises(KeyRotationError, match="two phases"):
+ await _boot_seeding(session_factory, "new-pod", f"{NEW},{OLD}", "second-token")
+
+ async with session_factory() as session:
+ stored = await session.scalar(
+ select(ApiKey.encrypted_key).where(ApiKey.type == BOOTSTRAP_KEY_TYPE)
+ )
+ assert stored is not None
+ assert _keyring(OLD).decrypt(stored) == "first-token"
+
+
+async def test_a_key_every_process_already_holds_may_go_to_the_front(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ """The second phase: the old processes were given the key at the back."""
+ await _running(session_factory, "old-pod", ["old", "new"])
+ await _running(session_factory, "new-pod", ["new", "old"])
+
+ await require_front_key_readable(
+ session_factory, _keyring(f"{NEW},{OLD}"), "new-pod"
+ )
+
+
+async def test_the_first_phase_adds_a_key_at_the_back(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ await _running(session_factory, "old-pod", ["old"])
+ await _running(session_factory, "new-pod", ["old", "new"])
+
+ await require_front_key_readable(
+ session_factory, _keyring(f"{OLD},{NEW}"), "new-pod"
+ )
+
+
+async def test_a_process_that_stopped_is_not_counted(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ """One replica replaced rather than rolled rotates in one step, as before."""
+ old = await _running(session_factory, "old-pod", ["old"])
+ await old.stop()
+
+ await require_front_key_readable(session_factory, _keyring(NEW), "new-pod")
+
+
+async def test_a_process_from_before_the_check_is_named_not_refused(
+ session_factory: async_sessionmaker[AsyncSession],
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ async with session_factory() as session:
+ await session.execute(
+ text("INSERT INTO switch_core_processes (id) VALUES ('older-pod')")
+ )
+ await session.commit()
+
+ with caplog.at_level(logging.WARNING, logger="switch_core.db.key_rotation"):
+ await require_front_key_readable(session_factory, _keyring(NEW), "new-pod")
+
+ assert "older-pod" in caplog.text
+
+
+class _WritesFirst:
+ """A keyring that, as the sweep re-encrypts a value, lets another process
+ write a fresh one to the same row and commit it first."""
+
+ def __init__(self, keyring: Keyring, write: Any) -> None:
+ self._keyring = keyring
+ self._write = write
+
+ def current_prefix(self) -> str:
+ return self._keyring.current_prefix()
+
+ def decrypt(self, value: str) -> str:
+ return self._keyring.decrypt(value)
+
+ def encrypt(self, plaintext: str) -> str:
+ self._write()
+ return self._keyring.encrypt(plaintext)
+
+
+async def test_a_value_written_during_the_sweep_is_kept(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ old, new = _keyring(OLD), _keyring(f"{NEW},{OLD}")
+ async with session_factory() as session:
+ session.add(
+ TrustSettings(
+ id="rollout-test",
+ endpoint="https://trust.example.test",
+ api_key_encrypted=old.encrypt("read-by-the-sweep"),
+ )
+ )
+ await session.commit()
+
+ pending: list[Any] = []
+
+ def write_elsewhere() -> None:
+ async def write() -> None:
+ async with session_factory() as other:
+ await other.execute(
+ update(TrustSettings)
+ .where(TrustSettings.id == "rollout-test")
+ .values(api_key_encrypted=new.encrypt("written-meanwhile"))
+ )
+ await other.commit()
+
+ pending.append(write())
+
+ sweeping = _WritesFirst(new, write_elsewhere)
+ async with session_factory() as session:
+ rows = await session.execute(
+ select(TrustSettings.id, TrustSettings.api_key_encrypted).where(
+ TrustSettings.id == "rollout-test"
+ )
+ )
+ assert rows.all()
+ # Run the other process's write between the sweep's read and its update.
+ original_execute = session.execute
+
+ async def execute(statement: Any, *args: Any, **kwargs: Any) -> Any:
+ for write in pending:
+ await write
+ pending.clear()
+ return await original_execute(statement, *args, **kwargs)
+
+ session.execute = execute # type: ignore[method-assign]
+ rewritten = await reencrypt_stale_text(
+ session,
+ sweeping, # type: ignore[arg-type]
+ TrustSettings,
+ "api_key_encrypted",
+ )
+ await session.commit()
+
+ assert rewritten == 0
+ async with session_factory() as session:
+ stored = await session.scalar(
+ select(TrustSettings.api_key_encrypted).where(
+ TrustSettings.id == "rollout-test"
+ )
+ )
+ assert stored is not None and new.decrypt(stored) == "written-meanwhile"
diff --git a/core/tests/switch_core/db/test_schema_version.py b/core/tests/switch_core/db/test_schema_version.py
index db36bbd2b..e491f656b 100644
--- a/core/tests/switch_core/db/test_schema_version.py
+++ b/core/tests/switch_core/db/test_schema_version.py
@@ -8,14 +8,21 @@ from __future__ import annotations
from collections.abc import AsyncIterator
from pathlib import Path
+from typing import Any
import pytest
from alembic.config import Config as AlembicConfig
+from alembic.runtime.environment import EnvironmentContext
from alembic.script import ScriptDirectory
-from sqlalchemy import text
+from sqlalchemy import Connection, text
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
-from switch_core.db.schema_version import SchemaNotAtHeadError, require_schema_at_head
+from switch_core.db.schema_version import (
+ SchemaNotAtHeadError,
+ record_expand_only,
+ require_schema_at_head,
+ schema_ahead_through_expand_only,
+)
_ALEMBIC_INI = Path(__file__).resolve().parents[3] / "alembic.ini"
_SCRATCH_DB = "schema_version_check"
@@ -83,3 +90,144 @@ async def test_a_database_never_migrated_is_refused(
) -> None:
with pytest.raises(SchemaNotAtHeadError, match="none"):
await require_schema_at_head(scratch_engine, alembic_cfg)
+
+
+async def _record(engine: AsyncEngine, *chain: tuple[str, str]) -> None:
+ async with engine.begin() as connection:
+ await connection.execute(
+ text(
+ "CREATE TABLE IF NOT EXISTS schema_expand_only_revisions "
+ "(revision text PRIMARY KEY, down_revision text NOT NULL, "
+ "applied_at timestamptz NOT NULL DEFAULT now())"
+ )
+ )
+ for revision, down in chain:
+ await connection.execute(
+ text("INSERT INTO schema_expand_only_revisions VALUES (:r, :d)"),
+ {"r": revision, "d": down},
+ )
+
+
+def _head(alembic_cfg: AlembicConfig) -> str:
+ (head,) = ScriptDirectory.from_config(alembic_cfg).get_heads()
+ return head
+
+
+class TestAheadByExpandOnlyRevisions:
+ """An older release during a rolling deploy, or after a rollback: the
+ database's head is a revision this build has no file for."""
+
+ async def test_one_recorded_revision_ahead_is_served(
+ self, scratch_engine: AsyncEngine, alembic_cfg: AlembicConfig
+ ) -> None:
+ await _stamp(scratch_engine, "future1")
+ await _record(scratch_engine, ("future1", _head(alembic_cfg)))
+
+ await require_schema_at_head(scratch_engine, alembic_cfg)
+ assert await schema_ahead_through_expand_only(scratch_engine, alembic_cfg)
+
+ async def test_a_chain_of_recorded_revisions_is_served(
+ self, scratch_engine: AsyncEngine, alembic_cfg: AlembicConfig
+ ) -> None:
+ await _stamp(scratch_engine, "future2")
+ await _record(
+ scratch_engine,
+ ("future2", "future1"),
+ ("future1", _head(alembic_cfg)),
+ )
+
+ await require_schema_at_head(scratch_engine, alembic_cfg)
+
+ async def test_a_revision_not_recorded_as_expand_only_is_refused(
+ self, scratch_engine: AsyncEngine, alembic_cfg: AlembicConfig
+ ) -> None:
+ """It may drop or rename what this release reads."""
+ await _stamp(scratch_engine, "future2")
+ await _record(scratch_engine, ("future1", _head(alembic_cfg)))
+
+ with pytest.raises(SchemaNotAtHeadError):
+ await require_schema_at_head(scratch_engine, alembic_cfg)
+ assert not await schema_ahead_through_expand_only(scratch_engine, alembic_cfg)
+
+ async def test_a_chain_that_lands_below_this_builds_head_is_refused(
+ self, scratch_engine: AsyncEngine, alembic_cfg: AlembicConfig
+ ) -> None:
+ """A branch off an older revision is not ahead of this build."""
+ script = ScriptDirectory.from_config(alembic_cfg)
+ parents = script.get_revision(_head(alembic_cfg)).down_revision
+ older = parents if isinstance(parents, str) else parents[0]
+ await _stamp(scratch_engine, "future1")
+ await _record(scratch_engine, ("future1", older))
+
+ with pytest.raises(SchemaNotAtHeadError):
+ await require_schema_at_head(scratch_engine, alembic_cfg)
+
+ async def test_at_head_is_not_ahead(
+ self, scratch_engine: AsyncEngine, alembic_cfg: AlembicConfig
+ ) -> None:
+ await _stamp(scratch_engine, _head(alembic_cfg))
+ await _record(scratch_engine)
+
+ assert not await schema_ahead_through_expand_only(scratch_engine, alembic_cfg)
+
+
+def _run(connection: Connection, alembic_cfg: AlembicConfig, target: str) -> None:
+ """Upgrade or downgrade on an explicit connection, with the recorder the
+ repository's env.py installs, without loading env.py itself."""
+ script = ScriptDirectory.from_config(alembic_cfg)
+
+ def migrate(revision: Any, context: Any) -> Any:
+ if target == "head":
+ return script._upgrade_revs("head", revision)
+ return script._downgrade_revs(target, revision)
+
+ with EnvironmentContext(alembic_cfg, script, fn=migrate) as environment:
+ environment.configure(
+ connection=connection, on_version_apply=[record_expand_only]
+ )
+ with environment.begin_transaction():
+ environment.run_migrations()
+
+
+async def test_applying_the_chain_records_the_expand_only_revisions(
+ scratch_engine: AsyncEngine, alembic_cfg: AlembicConfig
+) -> None:
+ async with scratch_engine.begin() as connection:
+ await connection.run_sync(_run, alembic_cfg, "head")
+ async with scratch_engine.connect() as connection:
+ recorded = dict(
+ (
+ await connection.execute(
+ text(
+ "SELECT revision, down_revision FROM schema_expand_only_revisions"
+ )
+ )
+ ).all()
+ )
+ # Each is recorded with the revision it follows in the chain, whatever that
+ # is when the chain is extended or reordered.
+ script = ScriptDirectory.from_config(alembic_cfg)
+ for revision in ("e2b0df5be5ff", "e5956cbd3bdc"):
+ assert recorded[revision] == script.get_revision(revision).down_revision
+ assert recorded["e5956cbd3bdc"] == "e2b0df5be5ff"
+ for revision in recorded:
+ assert script.get_revision(revision).module.expand_only is True
+
+ async with scratch_engine.begin() as connection:
+ await connection.run_sync(_run, alembic_cfg, "e2b0df5be5ff")
+ async with scratch_engine.connect() as connection:
+ left = set(
+ (
+ await connection.execute(
+ text("SELECT revision FROM schema_expand_only_revisions")
+ )
+ ).scalars()
+ )
+ assert left == {"e2b0df5be5ff"}
+ async with scratch_engine.begin() as connection:
+ await connection.run_sync(_run, alembic_cfg, "c4d8e2f6a913")
+ async with scratch_engine.connect() as connection:
+ table = await connection.scalar(
+ text("SELECT to_regclass('schema_expand_only_revisions')")
+ )
+ assert table is None
diff --git a/core/tests/switch_core/db/test_tenant_exemption_allowlist.py b/core/tests/switch_core/db/test_tenant_exemption_allowlist.py
index 0cc144ea7..2e834c873 100644
--- a/core/tests/switch_core/db/test_tenant_exemption_allowlist.py
+++ b/core/tests/switch_core/db/test_tenant_exemption_allowlist.py
@@ -70,6 +70,11 @@ _ALLOWED_MODULES = {
# time; `register_agent_with_token` resolves a registration credential by
# its globally unique hash, which is the read that produces a tenant.
"switch_core.bridges.agent.protocol.agent_core",
+ # Each process keeps a copy of every other process's agent connections, so
+ # its periodic pass over the recorded rows reads every tenant's: enumerate
+ # the tenants, then read each one's rows under its own policy, the same
+ # shape as the runtime-state sweep.
+ "switch_core.bridges.agent.protocol.connection_rows",
# Bearer and OIDC authentication: the credential's tenant, before its row.
"switch_core.bridges.agent.auth",
# Agent-controller enrollment and token exchange carry their secret (a
@@ -83,6 +88,11 @@ _ALLOWED_MODULES = {
# tenants, then read each one's definitions under its own policy, the same
# boot-style fan-out `main` does.
"switch_core.management.bindings",
+ # Each process keeps its own copy of which process holds each
+ # controller's socket, and reads every tenant's controllers again at
+ # startup, on resync and on each re-check in case a signal was lost: the
+ # same fan-out as the bindings.
+ "switch_core.management.signals",
# The gateway's JWT subject resolves to its membership, and
# `tenant_members` is scoped, so nothing else can answer it. In the
# authentication module itself rather than on an injected store: a store
@@ -109,6 +119,12 @@ _ALLOWED_MODULES = {
# the tenant it produced, so a wrong answer here is a miss rather than a
# cross-tenant read.
"switch_core.bridges.collaboration.install_service",
+ # A sign-in's browser steps (leaving for the vendor, coming back,
+ # continuing to Switch Console) carry no Switch session, only the flow's
+ # state, and may reach any process. The state is the one thing that names
+ # the flow's tenant, and the flow is then read scoped to it, so a wrong
+ # answer is a missing flow rather than a cross-tenant read.
+ "switch_core.connections.sign_in_flows",
# `switch_core.transport.postgres` and `switch_core.clients.agent_consumer`
# came off this list with `tenant_of_client`: both were built from a
# `clients` row that already named the tenant, so they carry it instead of
@@ -152,7 +168,7 @@ _RAW_SESSION_FACTORY_MODULES = {
"switch_core.bridges.collaboration.collaboration_core",
"switch_core.bridges.collaboration.lifecycle_service",
"switch_core.bridges.agent.server_connectors.lifecycle",
- "switch_core.clients.agent_consumer",
+ "switch_core.delivery.recipient",
"switch_core.clients.command_consumer",
"switch_core.clients.client_lifecycle_service",
"switch_core.provisioning.postgres",
@@ -167,6 +183,10 @@ _RAW_SESSION_FACTORY_MODULES = {
# table here would be the one check that passes in development and raises
# under the restricted runtime role in production.
"switch_core.observability.health",
+ # ── The signal sender, which calls `pg_notify` and reads no table. A
+ # signal is for every process regardless of tenant, so there is no tenant
+ # to bind.
+ "switch_core.signals",
# `switch_core.transport.postgres` and `switch_core.bridges.agent.auth`
# came off this list with the runtime role: the transport learned its own
# tenant from its client row and the middleware learned a credential's
@@ -188,7 +208,12 @@ _RAW_SESSION_FACTORY_MODULES = {
# This process's lease in `switch_core_processes`, a table with no tenant
# and no policy (`rls_ddl.GLOBAL_TABLES`): a process serves every tenant,
# so its lease belongs to none. It touches only that table.
- "switch_core.management.process_lease",
+ "switch_core.process_lease",
+ # Which process runs each piece of work that must run once: it reads
+ # `switch_core_processes` and claims and releases `ownership_leases`, both
+ # tables with no tenant and no policy (`rls_ddl.GLOBAL_TABLES`). One
+ # process holds work for every tenant at once.
+ "switch_core.ownership",
# The boot-time re-encryption sweep's per-tenant pass binds a tenant
# (`tenant_session`); this raw call is its separate pass over
# the global tables' encrypted values (`trust_settings`'s key,
diff --git a/core/tests/switch_core/db/test_tenant_lookup.py b/core/tests/switch_core/db/test_tenant_lookup.py
index 8d3934d4b..3a0b6eef8 100644
--- a/core/tests/switch_core/db/test_tenant_lookup.py
+++ b/core/tests/switch_core/db/test_tenant_lookup.py
@@ -106,6 +106,7 @@ _ADDED_SINCE = {
"tenants_of_cloud_machine": "b6e1c9d4a7f2",
"tenants_inviting_email": "4b8e2d61c9f7",
"tenants_open_to_domain": "7d3f5a19e2c8",
+ "tenant_of_sign_in_flow": "e5956cbd3bdc",
}
# A third kind of change, and the quietest: a lookup whose body a later
diff --git a/core/tests/switch_core/db/test_tenant_schema_catalogue.py b/core/tests/switch_core/db/test_tenant_schema_catalogue.py
index 6d17e2651..1c97afe1f 100644
--- a/core/tests/switch_core/db/test_tenant_schema_catalogue.py
+++ b/core/tests/switch_core/db/test_tenant_schema_catalogue.py
@@ -112,6 +112,10 @@ class TestEveryTableIsScopedUnlessItIsNamedGlobal:
# sockets for every tenant at once, and the row says only that it
# is alive.
"switch_core_processes",
+ # Which of those processes runs each piece of work that must run
+ # once: a process holds work for every tenant at once, and the row
+ # names only which process runs which id.
+ "ownership_leases",
# Switch Trust's one guardrails policy for the whole deployment.
# A policy that varied per tenant would be a new scoped table.
"trust_settings",
@@ -119,6 +123,9 @@ class TestEveryTableIsScopedUnlessItIsNamedGlobal:
# vendor sees one Switch, so one client per deployment and
# service, holding no person's account.
"service_oauth_clients",
+ # Which schema revisions a release from before them can still
+ # serve on, as Alembic applied them: a fact about the schema.
+ "schema_expand_only_revisions",
}
async def test_scoped_is_everything_else(self) -> None:
diff --git a/core/tests/switch_core/delivery/__init__.py b/core/tests/switch_core/delivery/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/core/tests/switch_core/clients/test_addressing_enforcement.py b/core/tests/switch_core/delivery/test_addressing_enforcement.py
similarity index 95%
rename from core/tests/switch_core/clients/test_addressing_enforcement.py
rename to core/tests/switch_core/delivery/test_addressing_enforcement.py
index f845b809c..97fd2b485 100644
--- a/core/tests/switch_core/clients/test_addressing_enforcement.py
+++ b/core/tests/switch_core/delivery/test_addressing_enforcement.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+import functools
from contextlib import asynccontextmanager
from datetime import UTC, datetime
from types import SimpleNamespace
@@ -7,7 +8,6 @@ from types import SimpleNamespace
from switch_core.budgets import BudgetGuard
from switch_core.clients.admin_messages import AUTO_REPLY_FLAG
from switch_core.clients.admin_messages import PLATFORM_MARKER as _PLATFORM_MARKER
-from switch_core.clients.agent_consumer import AgentConsumer
from switch_core.db.stores.budget_store import BudgetStanding
from switch_core.delivery.addressing import (
ADDRESSING_DENIED_MESSAGE as _ADDRESSING_DENIED_MESSAGE,
@@ -24,6 +24,7 @@ from switch_core.delivery.addressing import (
from switch_core.delivery.addressing import (
SenderPrincipal as _SenderPrincipal,
)
+from switch_core.delivery.recipient import AgentRecipient
from switch_core.transport import InboundMessage, RoomRef
@@ -220,12 +221,16 @@ def _allowed_client(
async def _claimed(_session, _user_id): # type: ignore[no-untyped-def]
return [SimpleNamespace(id=ext_id) for ext_id in claimed_external_ids]
- return SimpleNamespace(
+ resolver = SimpleNamespace(
agent=agent,
resolve_sender=_resolve,
_room_store=SimpleNamespace(get=_get_room),
_external_user_store=SimpleNamespace(get_by_user=_claimed),
)
+ # The real reads `permitted` makes through the stores above.
+ resolver.group_of = AddressingResolver.group_of.__get__(resolver)
+ resolver.claimed_accounts = AddressingResolver.claimed_accounts.__get__(resolver)
+ return resolver
async def _decide( # type: ignore[no-untyped-def]
@@ -561,9 +566,9 @@ def _gate_client( # type: ignore[no-untyped-def]
_addressing_allowed=_addressing_allowed,
actor=SimpleNamespace(send_message=_send_message),
_sender_handle=_sender_handle,
- _triggered_by_auto_reply=AgentConsumer._triggered_by_auto_reply,
+ _triggered_by_auto_reply=AgentRecipient._triggered_by_auto_reply,
)
- client._post_auto_reply = AgentConsumer._post_auto_reply.__get__(client)
+ client._post_auto_reply = AgentRecipient._post_auto_reply.__get__(client)
client.sent = sent # type: ignore[attr-defined]
return client
@@ -576,7 +581,7 @@ async def _gate(client: SimpleNamespace, event: SimpleNamespace): # type: ignor
"""Run the gate and post whatever refusal it hands back, the way on_message
does — the gate itself never touches Matrix, so that its caller can close
the database session first."""
- outcome = await AgentConsumer._gate_addressed(
+ outcome = await AgentRecipient._gate_addressed(
client, None, client.agent, event, _meta()
)
if outcome.refusal is not None:
@@ -626,7 +631,7 @@ class TestGateAddressed:
# session while the gate runs and must close it before talking to
# Matrix.
client = _gate_client(allowed=False)
- outcome = await AgentConsumer._gate_addressed(
+ outcome = await AgentRecipient._gate_addressed(
client, None, client.agent, _event(), _meta()
)
assert outcome.refusal == _ADDRESSING_DENIED_MESSAGE
@@ -698,6 +703,9 @@ def _command_client(*, allowed: bool, targets_me: bool): # type: ignore[no-unty
reply_command=_reply_command,
)
client.replies = replies # type: ignore[attr-defined]
+ client._command_refusal = functools.partial( # type: ignore[attr-defined]
+ AgentRecipient._command_refusal, client
+ )
return client
@@ -707,7 +715,7 @@ class TestGateCommand:
async def test_allowed_passes_through(self) -> None:
client = _command_client(allowed=True, targets_me=True)
- result = await AgentConsumer._gate_command(
+ result = await AgentRecipient._gate_command(
client, _room(), _command(args="@fixer"), _meta()
)
assert result is True
@@ -715,7 +723,7 @@ class TestGateCommand:
async def test_denied_and_explicitly_targeted_replies(self) -> None:
client = _command_client(allowed=False, targets_me=True)
- result = await AgentConsumer._gate_command(
+ result = await AgentRecipient._gate_command(
client, _room(), _command(args="@fixer"), _meta()
)
assert result is False
@@ -729,7 +737,7 @@ class TestGateCommand:
# `!reset-all-agents` (or a bare `!reset`) makes no claim about this
# agent; answering every one would flood a room of restricted agents.
client = _command_client(allowed=False, targets_me=False)
- result = await AgentConsumer._gate_command(
+ result = await AgentRecipient._gate_command(
client, _room(), _command(command="reset-all-agents", args=""), _meta()
)
assert result is False
@@ -752,7 +760,7 @@ class TestCommandTargetsMeExplicitly:
_text_tags_my_alias=_tags_alias,
_text_tags_my_role=_tags_role,
)
- client._args_tag_my_name = lambda text: AgentConsumer._args_tag_my_name( # type: ignore[attr-defined]
+ client._args_tag_my_name = lambda text: AgentRecipient._args_tag_my_name( # type: ignore[attr-defined]
client, text
)
return client
@@ -760,7 +768,7 @@ class TestCommandTargetsMeExplicitly:
async def test_no_at_token_is_not_explicit(self) -> None:
client = self._client()
assert (
- await AgentConsumer._command_targets_me_explicitly(
+ await AgentRecipient._command_targets_me_explicitly(
client, None, "", "room-1"
)
is False
@@ -769,7 +777,7 @@ class TestCommandTargetsMeExplicitly:
async def test_own_name_is_explicit(self) -> None:
client = self._client()
assert (
- await AgentConsumer._command_targets_me_explicitly(
+ await AgentRecipient._command_targets_me_explicitly(
client, None, "@fixer", "room-1"
)
is True
@@ -778,7 +786,7 @@ class TestCommandTargetsMeExplicitly:
async def test_another_agents_name_is_not_explicit(self) -> None:
client = self._client()
assert (
- await AgentConsumer._command_targets_me_explicitly(
+ await AgentRecipient._command_targets_me_explicitly(
client, None, "@someone-else", "room-1"
)
is False
@@ -787,7 +795,7 @@ class TestCommandTargetsMeExplicitly:
async def test_alias_is_explicit(self) -> None:
client = self._client(alias=True)
assert (
- await AgentConsumer._command_targets_me_explicitly(
+ await AgentRecipient._command_targets_me_explicitly(
client, None, "@nickname", "room-1"
)
is True
@@ -796,7 +804,7 @@ class TestCommandTargetsMeExplicitly:
async def test_held_role_is_explicit(self) -> None:
client = self._client(role=True)
assert (
- await AgentConsumer._command_targets_me_explicitly(
+ await AgentRecipient._command_targets_me_explicitly(
client, None, "@manager", "room-1"
)
is True
diff --git a/core/tests/switch_core/delivery/test_attachment_groups.py b/core/tests/switch_core/delivery/test_attachment_groups.py
new file mode 100644
index 000000000..8003a2858
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_attachment_groups.py
@@ -0,0 +1,557 @@
+"""Several files posted as one message reach an agent as one message.
+
+Each file is a row of its own in the room, written one after another and
+sharing a group marker. The router decides the post once, as a whole: in the
+transaction that writes the file completing it, or, when the rest never
+comes, once the post has waited `ATTACHMENT_GROUP_TIMEOUT_SECONDS` since its
+first file, with the files that did arrive and a note saying how many. What
+is waited on is a row in `pending_attachment_groups`, not memory, so any
+process can decide a post another started.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections.abc import Callable
+from datetime import timedelta
+from typing import Any
+
+import pytest
+from sqlalchemy import select, update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.bridges.agent.protocol.agent_connections import (
+ PROTOCOL_VERSION,
+ ClientDeclaration,
+)
+from switch_core.bridges.agent.protocol.positions import OWN_EVENTS
+from switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload
+from switch_core.db.models import (
+ TENANT_ZERO_ID,
+ Agent,
+ AgentEventRow,
+ Client,
+ Message,
+ PendingAttachmentGroup,
+ Room,
+)
+from switch_core.db.session_scope import tenant_session
+from switch_core.delivery import router as router_module
+from switch_core.delivery.recipient import AgentRecipient
+from tests.routing import (
+ Process,
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+)
+
+
+class _Scene:
+ def __init__(
+ self, process: Process, room: Room, alice: Client, agent: Agent
+ ) -> None:
+ self.process = process
+ self.room = room
+ self.alice = alice
+ self.agent = agent
+
+ async def part(
+ self,
+ index: int,
+ total: int,
+ group: str,
+ filename: str,
+ *,
+ caption: str | None = None,
+ ) -> str:
+ return await self.process.send_file(
+ self.room,
+ filename,
+ sender=self.alice,
+ caption=caption,
+ group={"id": group, "index": index, "total": total},
+ )
+
+
+async def _scene(
+ session_factory: async_sessionmaker[AsyncSession], *, channel_type: str
+) -> _Scene:
+ process = process_on(session_factory)
+ room = await make_room(session_factory, channel_type=channel_type)
+ alice = await make_person(session_factory, "alice")
+ agent, client = await make_agent(session_factory, "agent-a")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, client, agent=agent)
+ # Live, so the report says the agent could take what it was asked.
+ process.connections.open(
+ agent_id=agent.id,
+ connection_id="c1",
+ scope="all",
+ delivery_filter="addressed",
+ spawn_capable=False,
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
+ expected_generation=None,
+ )
+ return _Scene(process, room, alice, agent)
+
+
+async def scene(session_factory: async_sessionmaker[AsyncSession]) -> _Scene:
+ return await _scene(session_factory, channel_type="channel_public")
+
+
+def short_wait(monkeypatch: pytest.MonkeyPatch) -> None:
+ """Posts wait a hundredth of a second for their missing files."""
+ monkeypatch.setattr(router_module, "ATTACHMENT_GROUP_TIMEOUT_SECONDS", 0.01)
+
+
+async def _events(
+ session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+) -> list[AgentEvent]:
+ """What the router stored for the agent. Only what is for the agent is
+ stored; the rest of the room is read from the room's log by whoever
+ wants it. So a test about what a post becomes addresses the post to the
+ agent, in its caption (part 0's body)."""
+ return [
+ AgentEvent.model_validate(row.payload)
+ for row in await own_events(session_factory, scene.agent.id)
+ ]
+
+
+def _payload(event: AgentEvent) -> MessagePayload:
+ assert isinstance(event.payload, MessagePayload)
+ return event.payload
+
+
+async def _pending(session_factory: async_sessionmaker[AsyncSession]) -> list[Any]:
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ return list((await session.scalars(select(PendingAttachmentGroup))).all())
+
+
+async def _after_the_wait(scene: _Scene) -> None:
+ await asyncio.sleep(0.15)
+ await scene.process.idle()
+
+
+async def test_grouped_media_coalesces_into_one_event(
+ session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+) -> None:
+ parts = ["cat.png", "notes.md", "data.csv"]
+
+ for index, name in enumerate(parts):
+ await scene.part(
+ index,
+ 3,
+ "grp-1",
+ name,
+ caption="@agent-a three files" if index == 0 else None,
+ )
+ if index < 2:
+ assert await _events(session_factory, scene) == []
+
+ events = await _events(session_factory, scene)
+ assert len(events) == 1
+ payload = _payload(events[0])
+ assert [a.filename for a in payload.attachments] == [
+ "cat.png",
+ "notes.md",
+ "data.csv",
+ ]
+ assert payload.body == "@agent-a three files"
+ assert await _pending(session_factory) == []
+
+
+async def test_grouped_media_coalesces_out_of_order(
+ session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+) -> None:
+ names = {0: "a.png", 1: "b.png", 2: "c.png"}
+
+ for index in [1, 0, 2]:
+ await scene.part(
+ index,
+ 3,
+ "grp-2",
+ names[index],
+ caption="@agent-a captioned" if index == 0 else None,
+ )
+
+ events = await _events(session_factory, scene)
+ assert len(events) == 1
+ payload = _payload(events[0])
+ # Sorted by group index, not arrival order.
+ assert [a.filename for a in payload.attachments] == ["a.png", "b.png", "c.png"]
+ assert payload.body == "@agent-a captioned"
+ assert await _pending(session_factory) == []
+
+
+async def test_a_group_is_kept_as_the_one_message_it_was_posted_as(
+ session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+) -> None:
+ """Each part is a row of its own in the room's log, and the conversation
+ read from there leaves out a row whose message the agent already has, by
+ the id the message is known by: for a part, its group's. So the group is
+ kept under the group's id, at the part furthest on in the room. Kept under
+ one part's id, the other parts would reach the agent again as
+ conversation."""
+ for index in [0, 2, 1]:
+ await scene.part(
+ index,
+ 3,
+ "grp-key",
+ f"f{index}.png",
+ caption="@agent-a three" if index == 0 else None,
+ )
+
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ furthest = await session.scalar(
+ select(Message.seq)
+ .where(Message.room_id == scene.room.id)
+ .order_by(Message.seq.desc())
+ .limit(1)
+ )
+ kept = (
+ await session.execute(
+ select(AgentEventRow.room_seq, AgentEventRow.message_key).where(
+ AgentEventRow.agent_id == scene.agent.id
+ )
+ )
+ ).all()
+ assert [tuple(row) for row in kept] == [(furthest, "grp-key")]
+
+
+async def test_ungrouped_media_emits_immediately(
+ session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+) -> None:
+ await scene.process.send_file(
+ scene.room, "cat.png", sender=scene.alice, caption="@agent-a a cat"
+ )
+
+ events = await _events(session_factory, scene)
+ assert len(events) == 1
+ payload = _payload(events[0])
+ assert [a.filename for a in payload.attachments] == ["cat.png"]
+ assert payload.body == "@agent-a a cat"
+ assert await _pending(session_factory) == []
+
+
+async def test_incomplete_group_flushes_with_disclosed_notice(
+ session_factory: async_sessionmaker[AsyncSession],
+ scene: _Scene,
+ short_wait: None,
+) -> None:
+ for index, name in [(0, "cat.png"), (1, "notes.md")]:
+ await scene.part(
+ index,
+ 3,
+ "grp-3",
+ name,
+ caption="@agent-a two of three" if index == 0 else None,
+ )
+ assert await _events(session_factory, scene) == []
+
+ await _after_the_wait(scene)
+
+ events = await _events(session_factory, scene)
+ assert len(events) == 1
+ payload = _payload(events[0])
+ assert [a.filename for a in payload.attachments] == ["cat.png", "notes.md"]
+ assert "two of three" in payload.body
+ assert "incomplete attachment group: 2 of 3" in payload.body
+ # No leak: the pending post is gone once decided.
+ assert await _pending(session_factory) == []
+
+
+async def test_an_agent_that_joins_while_a_post_waits_is_not_given_it(
+ session_factory: async_sessionmaker[AsyncSession],
+ scene: _Scene,
+ short_wait: None,
+) -> None:
+ """A member counts from when it joined: one that arrives after a post's
+ first file was written was never handed that file, so the post decided
+ without the rest does not reach it either."""
+ await scene.part(0, 2, "grp-late-member", "one.png", caption="@agent-a @late both")
+ late, late_client = await make_agent(session_factory, "late")
+ await add_member(session_factory, scene.room, late_client, agent=late)
+
+ await _after_the_wait(scene)
+
+ assert len(await _events(session_factory, scene)) == 1
+ assert await own_events(session_factory, late.id) == []
+
+
+async def test_group_is_anchored_on_part_zero_not_the_completing_part(
+ session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+) -> None:
+ """The coalesced message must carry part 0's event id, so a reply threads
+ off the canonical first event rather than whichever part landed last."""
+ # Part 0 arrives FIRST, so a later part completes the group; otherwise the
+ # completing event happens to be part 0 and the assertion proves nothing.
+ ids = [
+ await scene.part(
+ index,
+ 3,
+ "grp-anchor",
+ name,
+ caption="@agent-a three" if index == 0 else None,
+ )
+ for index, name in [(0, "a.png"), (1, "b.md"), (2, "c.csv")]
+ ]
+
+ events = await _events(session_factory, scene)
+ assert len(events) == 1
+ # Part 2 completed the group, but part 0 anchors the payload.
+ assert _payload(events[0]).message_id == ids[0]
+
+
+async def test_incomplete_group_is_anchored_on_part_zero(
+ session_factory: async_sessionmaker[AsyncSession],
+ scene: _Scene,
+ short_wait: None,
+) -> None:
+ first = await scene.part(0, 3, "grp-anchor-2", "a.png", caption="@agent-a anchored")
+ await scene.part(2, 3, "grp-anchor-2", "c.csv")
+
+ await _after_the_wait(scene)
+
+ events = await _events(session_factory, scene)
+ assert len(events) == 1
+ assert _payload(events[0]).message_id == first
+
+
+async def test_group_timeout_bounds_the_group_not_the_gap_between_parts(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+ scene: _Scene,
+) -> None:
+ """The wait is armed once per group. A batch dribbling in just under the
+ timeout must not be able to hold the post back indefinitely.
+
+ The deadline is read rather than waited on: the post's first file sets
+ it, from the database's clock, and the files after it leave it where it
+ is. Only one timer is set, for the deadline the first file set.
+ """
+ armed: list[float] = []
+ real = scene.process.router.flush_groups_after
+
+ def flush_groups_after(tenant_id: str, delay: float) -> None:
+ armed.append(delay)
+
+ monkeypatch.setattr(scene.process.router, "flush_groups_after", flush_groups_after)
+ await scene.part(0, 4, "grp-slow", "a.png", caption="@agent-a slow batch")
+ [pending] = await _pending(session_factory)
+ deadline = pending.deadline
+ assert len(armed) == 1
+ assert 0 < armed[0] <= router_module.ATTACHMENT_GROUP_TIMEOUT_SECONDS
+
+ # Parts keep trickling in below the deadline; it must NOT be pushed back
+ # by each arrival.
+ for index, name in [(1, "b.md"), (2, "c.csv")]:
+ await scene.part(index, 4, "grp-slow", name)
+ [pending] = await _pending(session_factory)
+ assert pending.deadline == deadline
+ assert len(armed) == 1
+
+ # The deadline passes.
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ await session.execute(
+ update(PendingAttachmentGroup).values(
+ deadline=PendingAttachmentGroup.deadline - timedelta(seconds=10)
+ )
+ )
+ await session.commit()
+ monkeypatch.setattr(scene.process.router, "flush_groups_after", real)
+ assert await scene.process.router.flush_due_groups(TENANT_ZERO_ID) == 1
+ await scene.process.idle()
+
+ # Decided on the group's own deadline rather than being extended forever.
+ events = await _events(session_factory, scene)
+ assert len(events) == 1
+ assert "incomplete attachment group: 3 of 4" in _payload(events[0]).body
+ assert await _pending(session_factory) == []
+
+
+class TestAddressingSurvivesCoalescing:
+ """Two screenshots in one Slack message reached nobody (CHOO-2173).
+
+ One such message is split: the text and its `@mention` ride on part 0 and
+ every later part is a bare file. Addressing was read from whichever part
+ completed the group (a filename), so the reassembled message claimed to
+ address nobody and each agent, correctly, ignored it.
+ """
+
+ async def test_a_mention_on_part_zero_addresses_the_whole_group(
+ self, session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+ ) -> None:
+ for index, name in [(0, "one.png"), (1, "two.png")]:
+ await scene.part(
+ index,
+ 2,
+ "grp-addr",
+ name,
+ caption="@agent-a look at these" if index == 0 else None,
+ )
+
+ events = await _events(session_factory, scene)
+ assert len(events) == 1
+ payload = _payload(events[0])
+ assert payload.addressed is True
+ # The text travels with the images rather than being left on the part
+ # it arrived on.
+ assert payload.body == "@agent-a look at these"
+ assert [a.filename for a in payload.attachments] == ["one.png", "two.png"]
+
+ async def test_it_holds_when_the_captioned_part_arrives_last(
+ self, session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+ ) -> None:
+ for index, name in [(1, "two.png"), (0, "one.png")]:
+ await scene.part(
+ index,
+ 2,
+ "grp-addr-rev",
+ name,
+ caption="@agent-a look at these" if index == 0 else None,
+ )
+
+ assert _payload((await _events(session_factory, scene))[0]).addressed is True
+
+ async def test_a_group_that_mentions_nobody_stays_unaddressed(
+ self, session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+ ) -> None:
+ # The fix must carry addressing across the group, not assert it. A
+ # group for nobody is the room's conversation, so nothing is stored for
+ # the agent: it is read from the room's log by a reader that wants it.
+ for index, name in [(0, "one.png"), (1, "two.png")]:
+ await scene.part(
+ index,
+ 2,
+ "grp-unaddr",
+ name,
+ caption="just some pictures" if index == 0 else None,
+ )
+
+ assert await _events(session_factory, scene) == []
+ assert await _pending(session_factory) == []
+
+ async def test_an_incomplete_group_is_still_addressed(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ scene: _Scene,
+ short_wait: None,
+ ) -> None:
+ # A batch that never completes is decided by the safety net, and
+ # losing a file is no reason to also lose who it was for.
+ await scene.part(
+ 0, 3, "grp-addr-partial", "one.png", caption="@agent-a two of three"
+ )
+ await scene.part(1, 3, "grp-addr-partial", "two.png")
+ await _after_the_wait(scene)
+
+ events = await _events(session_factory, scene)
+ assert len(events) == 1
+ payload = _payload(events[0])
+ assert payload.addressed is True
+ assert "@agent-a two of three" in payload.body
+
+ async def test_a_single_attachment_is_unaffected(
+ self, session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+ ) -> None:
+ # One screenshot always worked; it takes no group and no waiting.
+ await scene.process.send_file(
+ scene.room, "one.png", sender=scene.alice, caption="@agent-a one picture"
+ )
+
+ assert _payload((await _events(session_factory, scene))[0]).addressed is True
+
+
+async def test_a_grouped_post_addressed_to_the_agent_is_reported_once(
+ scene: _Scene,
+) -> None:
+ """Three files in one post are one message asked of the agent, not three:
+ the report follows the coalesced payload, not the parts."""
+ for index in range(3):
+ await scene.part(
+ index,
+ 3,
+ "grp-r",
+ f"f{index}.png",
+ caption="@agent-a look at these" if index == 0 else None,
+ )
+ await scene.process.idle()
+
+ assert scene.process.telemetry.addressed == [
+ {
+ "tenant_id": TENANT_ZERO_ID,
+ "room_id": scene.room.id,
+ "sender_transport_user_id": scene.alice.transport_user_id,
+ "from_platform": False,
+ "agent_metadata": None,
+ "agent_live": True,
+ "has_attachment": True,
+ }
+ ]
+
+
[email protected]("late_index", [0, 2])
+async def test_a_post_split_by_the_timeout_is_reported_once(
+ late_index: int,
+ session_factory: async_sessionmaker[AsyncSession],
+ short_wait: None,
+) -> None:
+ """A group that times out incomplete is delivered again when the rest
+ arrives. In a direct room both pieces address the agent, but it is one
+ post: counted with its first part, as the transport counts it."""
+ scene = await _scene(session_factory, channel_type="direct")
+ on_time = [index for index in range(3) if index != late_index]
+ for batch in (on_time, [late_index]):
+ for index in batch:
+ await scene.part(index, 3, "grp-late", f"f{index}.png")
+ await _after_the_wait(scene)
+
+ assert len(await _events(session_factory, scene)) == 2
+ assert len(scene.process.telemetry.addressed) == 1
+
+
+async def test_media_skips_the_availability_check_while_telemetry_is_off(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+ scene: _Scene,
+) -> None:
+ """The media path reads it only to report it, so off means no query."""
+ checks = _count_calls(monkeypatch, "_is_available")
+ scene.process.telemetry.enabled = False
+
+ await scene.process.send_file(
+ scene.room, "a.png", sender=scene.alice, caption="@agent-a look"
+ )
+ await scene.process.idle()
+
+ assert checks == []
+ assert len(await _events(session_factory, scene)) == 1
+
+
+async def test_media_that_does_not_address_the_agent_is_not_reported(
+ scene: _Scene,
+) -> None:
+ await scene.process.send_file(
+ scene.room, "a.png", sender=scene.alice, caption="just a picture"
+ )
+ await scene.process.idle()
+
+ assert scene.process.telemetry.addressed == []
+
+
+def _count_calls(monkeypatch: pytest.MonkeyPatch, name: str) -> list[object]:
+ calls: list[object] = []
+ real: Callable[..., Any] = getattr(AgentRecipient, name)
+
+ async def counting(self: AgentRecipient, *args: Any, **kwargs: Any) -> Any:
+ calls.append(args)
+ return await real(self, *args, **kwargs)
+
+ monkeypatch.setattr(AgentRecipient, name, counting)
+ return calls
diff --git a/core/tests/switch_core/delivery/test_command_order.py b/core/tests/switch_core/delivery/test_command_order.py
new file mode 100644
index 000000000..4100c9cdc
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_command_order.py
@@ -0,0 +1,89 @@
+"""A session command reaches the agent in the order it was written.
+
+`!reset`, `!compact` and `!interrupt` are queued for the agent's controller
+as own events. They are decided in the transaction that writes the command,
+like every other row, so a message written after the command is numbered
+after it for the agent: the agent is never handed a new task before the reset
+that was meant to come first. Only the answer in the room waits for the
+commit.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Any
+
+import pytest
+from sqlalchemy import update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import Agent
+from switch_core.db.session_scope import tenant_session
+from switch_core.delivery.recipient import AgentRecipient
+from tests.routing import (
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+ said_by,
+)
+
+ACKS = {
+ "reset": "Resetting my session",
+ "compact": "Compacting my session",
+ "interrupt": "Interrupted my current turn",
+}
+
+
[email protected]("command", ["reset", "compact", "interrupt"])
+async def test_a_message_written_after_a_command_reaches_the_agent_after_it(
+ command: str,
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ await add_member(session_factory, room, alice)
+ agent, client = await make_agent(session_factory, "worker")
+ async with tenant_session(session_factory, agent.tenant_id) as session:
+ await session.execute(
+ update(Agent)
+ .where(Agent.id == agent.id)
+ .values(
+ integration_profile={
+ "connection_model": "always_on",
+ "command_capabilities": {command: "always"},
+ }
+ )
+ )
+ await session.commit()
+ await add_member(session_factory, room, client, agent=agent)
+
+ # The command's answer in the room is held until the next message has
+ # been written, as a slow post of the acknowledgement would.
+ answering = asyncio.Event()
+ written = asyncio.Event()
+ reply = AgentRecipient.reply_command
+
+ async def held_reply(self: AgentRecipient, *args: Any, **kwargs: Any) -> None:
+ answering.set()
+ await written.wait()
+ await reply(self, *args, **kwargs)
+
+ monkeypatch.setattr(AgentRecipient, "reply_command", held_reply)
+
+ await process.command(room, command, "@worker", sender=alice)
+ await answering.wait()
+ await process.say(room, "@worker start this task", sender=alice)
+ written.set()
+ await process.idle()
+
+ events = await own_events(session_factory, agent.id)
+ assert [e.type for e in events] == ["command", "message"]
+ assert events[0].seq < events[1].seq
+ # The command was answered in the room once it had committed.
+ said = [m.body or "" for m in await said_by(session_factory, room, client)]
+ assert any(body.startswith(ACKS[command]) for body in said), said
diff --git a/core/tests/switch_core/delivery/test_decision_failures.py b/core/tests/switch_core/delivery/test_decision_failures.py
new file mode 100644
index 000000000..70a3ca419
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_decision_failures.py
@@ -0,0 +1,132 @@
+"""One agent the router cannot decide for costs only that agent.
+
+A row is decided for every agent of its room in the writer's transaction, and
+a multi-file post given up on is decided in the sweep's. A decision that
+raises for one agent is logged and counted, and the row is written and
+decided for the others, as a failure to store one agent's own event already
+was. A post the sweep cannot decide at all is put off and tried again later,
+and the sweep goes on to the posts after it.
+"""
+
+from __future__ import annotations
+
+from datetime import timedelta
+from typing import Any
+
+import pytest
+from sqlalchemy import func, select, update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import TENANT_ZERO_ID, Message, PendingAttachmentGroup
+from switch_core.db.session_scope import tenant_session
+from switch_core.delivery.recipient import AgentRecipient
+from switch_core.delivery.router import MessageRouter
+from tests.routing import (
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+)
+
+
+async def test_a_decision_that_raises_for_one_agent_does_not_stop_the_write(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ await add_member(session_factory, room, alice)
+ broken, broken_client = await make_agent(session_factory, "broken")
+ fine, fine_client = await make_agent(session_factory, "fine")
+ await add_member(session_factory, room, broken_client, agent=broken)
+ await add_member(session_factory, room, fine_client, agent=fine)
+
+ decide = AgentRecipient.decide_message
+
+ async def raise_for_broken(self: AgentRecipient, *args: Any, **kwargs: Any) -> Any:
+ if self.agent.name == "broken":
+ raise RuntimeError("a rule that cannot be applied")
+ return await decide(self, *args, **kwargs)
+
+ monkeypatch.setattr(AgentRecipient, "decide_message", raise_for_broken)
+
+ await process.say(room, "@broken @fine please look", sender=alice)
+ await process.idle()
+
+ async with tenant_session(session_factory, room.tenant_id) as session:
+ bodies = list(
+ (
+ await session.scalars(
+ select(Message.body).where(Message.room_id == room.id)
+ )
+ ).all()
+ )
+ assert "@broken @fine please look" in bodies
+ assert len(await own_events(session_factory, fine.id)) == 1
+ assert await own_events(session_factory, broken.id) == []
+
+
+async def test_a_post_the_sweep_cannot_decide_does_not_hold_up_the_rest(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ agent, client = await make_agent(session_factory, "agent-a")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, client, agent=agent)
+ for group in ("grp-stuck", "grp-next"):
+ await process.send_file(
+ room,
+ f"{group}.png",
+ sender=alice,
+ caption="@agent-a one of two",
+ group={"id": group, "index": 0, "total": 2},
+ )
+ # Both waits are over, the stuck post's first.
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ for group, overdue in (("grp-stuck", 20), ("grp-next", 10)):
+ await session.execute(
+ update(PendingAttachmentGroup)
+ .where(PendingAttachmentGroup.group_id == group)
+ .values(
+ deadline=PendingAttachmentGroup.deadline
+ - timedelta(seconds=overdue)
+ )
+ )
+ await session.commit()
+
+ files_of = MessageRouter._files_of
+
+ async def fail_for_stuck(
+ self: MessageRouter,
+ session: AsyncSession,
+ reads: Any,
+ group_id: str,
+ *rest: Any,
+ ) -> Any:
+ if group_id == "grp-stuck":
+ raise RuntimeError("the post's files cannot be read")
+ return await files_of(self, session, reads, group_id, *rest)
+
+ monkeypatch.setattr(MessageRouter, "_files_of", fail_for_stuck)
+
+ assert await process.router.flush_due_groups(TENANT_ZERO_ID) == 1
+ await process.idle()
+
+ events = await own_events(session_factory, agent.id)
+ assert [e.message_key for e in events] == ["grp-next"]
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ stuck = await session.scalar(
+ select(PendingAttachmentGroup).where(
+ PendingAttachmentGroup.group_id == "grp-stuck"
+ )
+ )
+ now = await session.scalar(select(func.now()))
+ # Kept, and put off rather than tried again at once.
+ assert stuck is not None
+ assert stuck.deadline > now
diff --git a/core/tests/switch_core/delivery/test_enqueue_debug_logging.py b/core/tests/switch_core/delivery/test_enqueue_debug_logging.py
new file mode 100644
index 000000000..f4e99f17c
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_enqueue_debug_logging.py
@@ -0,0 +1,113 @@
+"""The debug line for a stored own event costs nothing unless DEBUG is on.
+
+It pretty-prints the whole event, and an argument to `logger.debug` is built
+before the level is checked. That ran once per agent per message: on a busy
+server, thousands of serialisations a second for a line nobody was logging.
+"""
+
+from __future__ import annotations
+
+import logging
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.bridges.agent.protocol.types import AgentEvent
+from switch_core.db.models import Agent, Client, Room
+from tests.routing import (
+ Process,
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+)
+
+_LOGGER = "switch_core.delivery.recipient"
+
+
+async def scene(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> tuple[Process, Room, Client, Agent]:
+ """`helper` in a room with someone who addresses it: only an event for
+ the agent is stored, so the rest of the room never reaches the line
+ under test."""
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ someone = await make_person(session_factory, "someone")
+ helper, client = await make_agent(session_factory, "helper")
+ await add_member(session_factory, room, someone)
+ await add_member(session_factory, room, client, agent=helper)
+ return process, room, someone, helper
+
+
+def dumps(monkeypatch: pytest.MonkeyPatch) -> list[int]:
+ calls: list[int] = []
+ real = AgentEvent.model_dump_json
+
+ def counting(self: AgentEvent, **kwargs: object) -> str:
+ calls.append(1)
+ return real(self, **kwargs) # type: ignore[arg-type]
+
+ monkeypatch.setattr(AgentEvent, "model_dump_json", counting)
+ return calls
+
+
+async def test_an_event_is_not_serialised_for_a_debug_line_nobody_logs(
+ session_factory: async_sessionmaker[AsyncSession],
+ dumps: list[int],
+ caplog: pytest.LogCaptureFixture,
+ scene: tuple[Process, Room, Client, Agent],
+) -> None:
+ process, room, someone, helper = scene
+ caplog.set_level(logging.INFO, logger=_LOGGER)
+
+ await process.say(room, "@helper hello", sender=someone)
+
+ assert len(await own_events(session_factory, helper.id)) == 1
+ assert dumps == []
+ assert not [r for r in caplog.records if "Enqueuing event" in r.getMessage()]
+
+
+async def test_the_debug_line_is_still_logged_when_debug_is_on(
+ session_factory: async_sessionmaker[AsyncSession],
+ dumps: list[int],
+ caplog: pytest.LogCaptureFixture,
+ scene: tuple[Process, Room, Client, Agent],
+) -> None:
+ process, room, someone, helper = scene
+ caplog.set_level(logging.DEBUG, logger=_LOGGER)
+
+ await process.say(room, "@helper hello", sender=someone)
+
+ assert len(await own_events(session_factory, helper.id)) == 1
+ assert dumps == [1]
+ lines = [r for r in caplog.records if "Enqueuing event" in r.getMessage()]
+ assert len(lines) == 1
+ assert '"body": "@helper hello"' in lines[0].getMessage()
+
+
[email protected]("level", [logging.INFO, logging.DEBUG])
+async def test_a_media_event_is_serialised_only_when_debug_is_on(
+ session_factory: async_sessionmaker[AsyncSession],
+ dumps: list[int],
+ caplog: pytest.LogCaptureFixture,
+ level: int,
+ scene: tuple[Process, Room, Client, Agent],
+) -> None:
+ process, room, someone, helper = scene
+ caplog.set_level(level, logger=_LOGGER)
+
+ await process.send_file(room, "a.png", sender=someone, caption="@helper a file")
+
+ assert len(await own_events(session_factory, helper.id)) == 1
+ logged = [r for r in caplog.records if "Enqueuing media event" in r.getMessage()]
+ if level == logging.DEBUG:
+ assert dumps == [1]
+ assert len(logged) == 1
+ else:
+ assert dumps == []
+ assert logged == []
diff --git a/core/tests/switch_core/delivery/test_every_write_is_routed.py b/core/tests/switch_core/delivery/test_every_write_is_routed.py
new file mode 100644
index 000000000..0d4d740fb
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_every_write_is_routed.py
@@ -0,0 +1,96 @@
+"""Every row written to `messages` goes through the router.
+
+The router decides, in the transaction that writes a room row, which agents
+the row is for, and stores their own events with it. A row written any other
+way is decided for nobody: an addressed message that never reaches the agent
+it names, with nothing to say so. So the calls that number and insert a live
+row, `MessageStore.create` and `MessageStore.create_unless_stored`, are made
+by the router and nowhere else, and this test fails the moment another module
+makes one, or inserts into `messages` some other way.
+
+`MessageStore.create_historical` is exempt by design: it backfills history
+below a room's live rows, which nothing delivers.
+"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+import switch_core
+
+_PACKAGE_ROOT = Path(switch_core.__file__).resolve().parent
+_ROUTER = _PACKAGE_ROOT / "delivery" / "router.py"
+_STORE = _PACKAGE_ROOT / "db" / "stores" / "message_store.py"
+_STORE_NAMES = {"message_store", "_message_store", "messages", "_messages"}
+
+
+def _calls(tree: ast.AST) -> list[ast.Call]:
+ return [node for node in ast.walk(tree) if isinstance(node, ast.Call)]
+
+
+def _is_message_store_create(call: ast.Call) -> bool:
+ """`<the message store>.create(...)` or `MessageStore().create(...)`, or
+ the same with `create_unless_stored`.
+
+ Matched on the receiver's name, the way every caller has spelled it: a
+ store held as `message_store`, `_message_store` or `_messages`, or built
+ on the spot.
+ """
+ func = call.func
+ if not isinstance(func, ast.Attribute) or func.attr not in {
+ "create",
+ "create_unless_stored",
+ }:
+ return False
+ receiver = func.value
+ if isinstance(receiver, ast.Attribute):
+ return receiver.attr in _STORE_NAMES
+ if isinstance(receiver, ast.Name):
+ return receiver.id in _STORE_NAMES
+ if isinstance(receiver, ast.Call) and isinstance(receiver.func, ast.Name):
+ return receiver.func.id == "MessageStore"
+ return False
+
+
+def _inserts_into_messages(call: ast.Call) -> bool:
+ """`insert(Message)` / `pg_insert(Message)`: a row written past the store."""
+ func = call.func
+ name = (
+ func.id
+ if isinstance(func, ast.Name)
+ else func.attr
+ if isinstance(func, ast.Attribute)
+ else None
+ )
+ return (
+ name in {"insert", "pg_insert"}
+ and len(call.args) == 1
+ and isinstance(call.args[0], ast.Name)
+ and call.args[0].id == "Message"
+ )
+
+
+def test_only_the_router_writes_room_rows() -> None:
+ offenders: list[str] = []
+ for path in sorted(_PACKAGE_ROOT.rglob("*.py")):
+ if path in (_ROUTER, _STORE) or "migrations" in path.parts:
+ continue
+ tree = ast.parse(path.read_text(), filename=str(path))
+ for call in _calls(tree):
+ if _is_message_store_create(call) or _inserts_into_messages(call):
+ offenders.append(f"{path.relative_to(_PACKAGE_ROOT)}:{call.lineno}")
+
+ assert offenders == [], (
+ "These write a room row without the router, so no agent is ever told "
+ f"about it: {offenders}. Write it with `MessageRouter.store` in the "
+ "caller's transaction, and call `Routed.committed()` after the commit."
+ )
+
+
+def test_the_router_is_what_writes_them() -> None:
+ """The other half: the router does call the store, so the check above is
+ looking for the right thing rather than passing on a rename."""
+ tree = ast.parse(_ROUTER.read_text(), filename=str(_ROUTER))
+
+ assert any(_is_message_store_create(call) for call in _calls(tree))
diff --git a/core/tests/switch_core/delivery/test_group_read_race.py b/core/tests/switch_core/delivery/test_group_read_race.py
new file mode 100644
index 000000000..1619b0244
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_group_read_race.py
@@ -0,0 +1,77 @@
+"""A multi-file post completed while a room is being read is not split.
+
+`read_conversation` reads the room's rows, then the agent's own events, and
+holds back a post whose files are still arriving. Under READ COMMITTED each
+statement sees what has committed by then, so whether a post is pending is
+read with the rows, in one statement: the file that completes a post commits
+its row and deletes the pending one together, and a reader that took the
+pending state from a later statement would hand out the first file alone (and
+the agent would get it again inside the addressed event).
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
+from switch_core.bridges.agent.protocol.types import MessagePayload
+from switch_core.db.models import TENANT_ZERO_ID
+from tests.routing import (
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ process_on,
+)
+
+
+async def test_a_post_completed_during_a_read_is_not_split(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ agent, client = await make_agent(session_factory, "reader")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, client, agent=agent)
+
+ group = {"id": "grp-race", "index": 0, "total": 2}
+ await process.send_file(
+ room, "one.png", sender=alice, caption="two files", group=group
+ )
+
+ real = EventBuffer._conversation_in
+ completed: list[bool] = []
+
+ async def complete_the_post_first(
+ self: EventBuffer, *args: Any, **kwargs: Any
+ ) -> Any:
+ # The second file commits after the reader has read the room's rows
+ # but before it asks which posts are pending.
+ if not completed:
+ completed.append(True)
+ await process.send_file(
+ room,
+ "two.png",
+ sender=alice,
+ group={"id": "grp-race", "index": 1, "total": 2},
+ )
+ return await real(self, *args, **kwargs)
+
+ monkeypatch.setattr(EventBuffer, "_conversation_in", complete_the_post_first)
+
+ first = await process.buffer.read_conversation(
+ TENANT_ZERO_ID, agent.id, {room.id: 0}, limit=100
+ )
+ bodies = [
+ e.event.payload.body
+ for e in first.events
+ if isinstance(e.event.payload, MessagePayload)
+ ]
+ # Expected: the post is held (nothing yet) or delivered whole. Not the
+ # first file alone, flagged as an incomplete post.
+ assert not any("incomplete attachment group" in (b or "") for b in bodies), bodies
diff --git a/core/tests/switch_core/delivery/test_join_during_decision.py b/core/tests/switch_core/delivery/test_join_during_decision.py
new file mode 100644
index 000000000..56dbbd1ab
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_join_during_decision.py
@@ -0,0 +1,168 @@
+"""An agent that joins while a row is being decided is handed the row.
+
+The router reads the room's agent members before `MessageStore.create` takes
+the room's lock and numbers the row. An agent whose arrival commits in between
+is a member when the row is numbered (its arrival comes first in the room), so
+the row is decided for it too.
+
+The mirror image of `test_a_message_decided_before_the_removal_stores_nothing_after_it`.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import Message
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.message_store import MessageStore
+from tests.routing import (
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+)
+
+BODY = "@newcomer welcome, please take this"
+
+
+async def test_a_mention_numbered_after_the_agents_arrival_reaches_it(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ await add_member(session_factory, room, alice)
+ agent, client = await make_agent(session_factory, "newcomer")
+
+ decided = asyncio.Event()
+ joined = asyncio.Event()
+ create = MessageStore.create
+
+ async def create_after_the_join(self, session, message, attachments): # type: ignore[no-untyped-def]
+ if message.body == BODY:
+ decided.set()
+ await joined.wait()
+ return await create(self, session, message, attachments)
+
+ monkeypatch.setattr(MessageStore, "create", create_after_the_join)
+
+ sending = asyncio.create_task(process.say(room, BODY, sender=alice))
+ await decided.wait()
+ # The agent is added while the message is between its decision and its
+ # write; the arrival takes the room's lock first and commits.
+ await process.provisioning.invite_to_room(
+ room.transport_room_id, client.transport_user_id
+ )
+ joined.set()
+ await sending
+ await process.idle()
+
+ async with tenant_session(session_factory, room.tenant_id) as session:
+ rows = list(
+ (
+ await session.scalars(
+ select(Message)
+ .where(Message.room_id == room.id)
+ .order_by(Message.seq)
+ )
+ ).all()
+ )
+ arrival = next(
+ r
+ for r in rows
+ if r.event_type == "m.room.member" and r.sender_client_id == client.id
+ )
+ mention = next(r for r in rows if r.body == BODY)
+ # In room order the agent is a member before the mention was written.
+ assert arrival.seq < mention.seq
+
+ # So the mention is one of its own events.
+ events = await own_events(session_factory, agent.id)
+ assert [e.room_seq for e in events] == [mention.seq], (
+ f"arrival seq {arrival.seq}, mention seq {mention.seq}, "
+ f"own events {[(e.type, e.room_seq) for e in events]}"
+ )
+
+
+BODY2 = "@latecomer here is the brief"
+
+
+async def test_a_join_that_began_first_but_committed_during_the_decision(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ """The join's transaction starts before the message's, so the mention is
+ stamped after the arrival (`sent_at`) as well as numbered after it. The
+ members were read before the join committed."""
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ await add_member(session_factory, room, alice)
+ agent, client = await make_agent(session_factory, "latecomer")
+
+ join_waiting = asyncio.Event()
+ mention_decided = asyncio.Event()
+ joined = asyncio.Event()
+ create = MessageStore.create
+
+ async def interleave(self, session, message, attachments): # type: ignore[no-untyped-def]
+ if (
+ message.event_type == "m.room.member"
+ and message.sender_client_id == client.id
+ ):
+ # The join has written its membership row; hold it uncommitted
+ # until the mention has been decided.
+ join_waiting.set()
+ await mention_decided.wait()
+ elif message.body == BODY2:
+ mention_decided.set()
+ await joined.wait()
+ return await create(self, session, message, attachments)
+
+ monkeypatch.setattr(MessageStore, "create", interleave)
+
+ joining = asyncio.create_task(
+ process.provisioning.invite_to_room(
+ room.transport_room_id, client.transport_user_id
+ )
+ )
+ await join_waiting.wait()
+ sending = asyncio.create_task(process.say(room, BODY2, sender=alice))
+ await joining
+ joined.set()
+ await sending
+ await process.idle()
+
+ async with tenant_session(session_factory, room.tenant_id) as session:
+ rows = list(
+ (
+ await session.scalars(
+ select(Message)
+ .where(Message.room_id == room.id)
+ .order_by(Message.seq)
+ )
+ ).all()
+ )
+ arrival = next(
+ r
+ for r in rows
+ if r.event_type == "m.room.member" and r.sender_client_id == client.id
+ )
+ mention = next(r for r in rows if r.body == BODY2)
+ assert arrival.seq < mention.seq
+ # Stamped after the arrival too: the old consumer would not have ignored it.
+ assert arrival.sent_at < mention.sent_at
+
+ events = await own_events(session_factory, agent.id)
+ assert [e.room_seq for e in events] == [mention.seq], (
+ f"arrival seq {arrival.seq} at {arrival.sent_at}, mention seq "
+ f"{mention.seq} at {mention.sent_at}, own events "
+ f"{[(e.type, e.room_seq) for e in events]}"
+ )
diff --git a/core/tests/switch_core/clients/test_join_greeting_suppression.py b/core/tests/switch_core/delivery/test_join_greeting_suppression.py
similarity index 67%
rename from core/tests/switch_core/clients/test_join_greeting_suppression.py
rename to core/tests/switch_core/delivery/test_join_greeting_suppression.py
index e748f4862..4a88ab19c 100644
--- a/core/tests/switch_core/clients/test_join_greeting_suppression.py
+++ b/core/tests/switch_core/delivery/test_join_greeting_suppression.py
@@ -4,7 +4,7 @@ from types import SimpleNamespace
import pytest
-from switch_core.clients.agent_consumer import AgentConsumer
+from switch_core.delivery.recipient import AgentRecipient
class _Recorder:
@@ -24,31 +24,22 @@ def _fake_self(
is_direct: bool = False,
agent_greetings_enabled: bool = True,
) -> SimpleNamespace:
- async def _is_direct_room(_transport_room_id: str) -> bool:
- return is_direct
-
- async def _resolve_room_meta(_transport_room_id: str) -> SimpleNamespace:
- return SimpleNamespace(
+ """The agent arriving in a room, as the router hands it its arrival."""
+ return SimpleNamespace(
+ agent=SimpleNamespace(id="agent-1", name="cc-bug-fixing"),
+ _meta=SimpleNamespace(
room_id="room-uuid",
name="Some Room",
bridge_id="bridge-1",
agent_greetings_enabled=agent_greetings_enabled,
- channel_type="channel_private",
- )
-
- return SimpleNamespace(
- agent=SimpleNamespace(id="agent-1", name="cc-bug-fixing"),
- _connections=SimpleNamespace(
- controllers=SimpleNamespace(room_joined=lambda _agent, _room: None)
+ channel_type="direct" if is_direct else "channel_private",
),
- _is_direct_room=_is_direct_room,
- _resolve_room_meta=_resolve_room_meta,
+ _room=_ROOM,
actor=SimpleNamespace(send_message=send_message),
)
_ROOM = SimpleNamespace(room_id="!matrix:server")
-_EVENT = SimpleNamespace()
@pytest.mark.asyncio
@@ -56,18 +47,14 @@ async def test_self_join_greeting_suppressed_when_disabled() -> None:
# CHOO-617: when the room's bridge has agent greetings toggled off, the
# per-agent self-join greeting is suppressed.
send_message = _Recorder()
- await AgentConsumer.on_self_join(
- _fake_self(send_message, agent_greetings_enabled=False), _ROOM, _EVENT
- )
+ await AgentRecipient.greet(_fake_self(send_message, agent_greetings_enabled=False)) # type: ignore[arg-type]
assert send_message.calls == []
@pytest.mark.asyncio
async def test_self_join_greeting_sent_when_enabled() -> None:
send_message = _Recorder()
- await AgentConsumer.on_self_join(
- _fake_self(send_message, agent_greetings_enabled=True), _ROOM, _EVENT
- )
+ await AgentRecipient.greet(_fake_self(send_message, agent_greetings_enabled=True)) # type: ignore[arg-type]
assert len(send_message.calls) == 1
@@ -76,10 +63,8 @@ async def test_direct_greeting_suppressed_when_disabled() -> None:
# The toggle also covers the 1:1 "Hi! I'm …" greeting — it's an agent
# greeting too.
send_message = _Recorder()
- await AgentConsumer.on_self_join(
- _fake_self(send_message, is_direct=True, agent_greetings_enabled=False),
- _ROOM,
- _EVENT,
+ await AgentRecipient.greet(
+ _fake_self(send_message, is_direct=True, agent_greetings_enabled=False) # type: ignore[arg-type]
)
assert send_message.calls == []
@@ -87,10 +72,8 @@ async def test_direct_greeting_suppressed_when_disabled() -> None:
@pytest.mark.asyncio
async def test_direct_greeting_sent_when_enabled() -> None:
send_message = _Recorder()
- await AgentConsumer.on_self_join(
- _fake_self(send_message, is_direct=True, agent_greetings_enabled=True),
- _ROOM,
- _EVENT,
+ await AgentRecipient.greet(
+ _fake_self(send_message, is_direct=True, agent_greetings_enabled=True) # type: ignore[arg-type]
)
assert len(send_message.calls) == 1
assert send_message.calls[0]["body"].startswith("Hi! I'm cc-bug-fixing")
diff --git a/core/tests/switch_core/clients/test_mention_routing.py b/core/tests/switch_core/delivery/test_mention_routing.py
similarity index 88%
rename from core/tests/switch_core/clients/test_mention_routing.py
rename to core/tests/switch_core/delivery/test_mention_routing.py
index 1445b5973..510dc9b0b 100644
--- a/core/tests/switch_core/clients/test_mention_routing.py
+++ b/core/tests/switch_core/delivery/test_mention_routing.py
@@ -8,15 +8,15 @@ import pytest_asyncio
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.bridges.agent.commands import _addressed_by_name_or_role
-from switch_core.clients.agent_consumer import (
- _STARTING_SESSION_MESSAGE,
- AgentConsumer,
- _role_elsewhere_message,
-)
from switch_core.delivery.addressing import (
AddressingResolver,
IncomingMessage,
)
+from switch_core.delivery.recipient import (
+ _STARTING_SESSION_MESSAGE,
+ AgentRecipient,
+ _role_elsewhere_message,
+)
@asynccontextmanager
@@ -59,7 +59,13 @@ def _resolver(**attrs: object) -> SimpleNamespace:
is enough to run the real logic against.
"""
resolver = SimpleNamespace(**attrs)
- for name in ("mentions_name", "mentions_alias", "mentions_role"):
+ for name in (
+ "mentions_name",
+ "mentions_alias",
+ "mentions_role",
+ "alias_of",
+ "role_of",
+ ):
setattr(
resolver,
name,
@@ -75,12 +81,12 @@ def _resolver(**attrs: object) -> SimpleNamespace:
def _client(name: str) -> SimpleNamespace:
- """An AgentConsumer-shaped double: an agent, an mxid, and a resolver."""
+ """An AgentRecipient-shaped double: an agent, an mxid, and a resolver."""
return SimpleNamespace(
agent=SimpleNamespace(name=name),
transport_user_id=f"@switch-agent-{name}:switch.local",
_addressing=_resolver(),
- _as_incoming=AgentConsumer._as_incoming,
+ _as_incoming=AgentRecipient._as_incoming,
)
@@ -99,7 +105,7 @@ async def _command_addresses(name: str, args: str) -> bool:
client = SimpleNamespace(
agent=SimpleNamespace(name=name), session_factory=_session_factory
)
- client._args_tag_my_name = lambda text: AgentConsumer._args_tag_my_name(
+ client._args_tag_my_name = lambda text: AgentRecipient._args_tag_my_name(
client, text
)
@@ -160,7 +166,7 @@ class TestTruePositives:
body="hello",
formatted_body='<a href="...@switch-agent-cc-bug-fixing:switch.local">x</a>',
)
- assert AgentConsumer._is_mentioned(client, event) is True
+ assert AgentRecipient._is_mentioned(client, event) is True
class TestMediaEventWithoutFormattedBody:
@@ -176,27 +182,27 @@ class TestMediaEventWithoutFormattedBody:
media_event = SimpleNamespace(
body="@cc-bug-fixing look at this", content={}, sender="@u:test"
)
- assert AgentConsumer._is_mentioned(client, media_event) is True
+ assert AgentRecipient._is_mentioned(client, media_event) is True
def test_media_without_mention_not_addressed(self) -> None:
client = _client("cc-bug-fixing")
media_event = SimpleNamespace(body="cat.png", content={}, sender="@u:test")
- assert AgentConsumer._is_mentioned(client, media_event) is False
+ assert AgentRecipient._is_mentioned(client, media_event) is False
class TestStripMention:
def test_strips_exact_mention_only(self) -> None:
client = _client("cc-bug-fixing")
# Should strip its own mention...
- assert AgentConsumer._strip_mention(client, "@cc-bug-fixing hello").strip() == (
- "hello"
- )
+ assert AgentRecipient._strip_mention(
+ client, "@cc-bug-fixing hello"
+ ).strip() == ("hello")
def test_does_not_strip_longer_name(self) -> None:
client = _client("cc-bug-fixing")
# ...but must not partially strip a longer agent's mention.
assert (
- AgentConsumer._strip_mention(client, "@cc-bug-fixing-2 hello")
+ AgentRecipient._strip_mention(client, "@cc-bug-fixing-2 hello")
== "@cc-bug-fixing-2 hello"
)
@@ -227,7 +233,7 @@ class TestRoleMentionRouting:
async def test_role_holder_is_addressed(self) -> None:
client = _role_client("ephemeral-cc", held_role="manager")
assert (
- await AgentConsumer._text_tags_my_role(
+ await AgentRecipient._text_tags_my_role(
client, None, "@manager review", "room-1"
)
is True
@@ -237,7 +243,7 @@ class TestRoleMentionRouting:
# This agent holds no role, so an @manager tag does not reach it.
client = _role_client("ephemeral-cc", held_role=None)
assert (
- await AgentConsumer._text_tags_my_role(
+ await AgentRecipient._text_tags_my_role(
client, None, "@manager review", "room-1"
)
is False
@@ -247,7 +253,7 @@ class TestRoleMentionRouting:
# Holds "worker" but "@manager" is tagged → not addressed.
client = _role_client("ephemeral-cc", held_role="worker")
assert (
- await AgentConsumer._text_tags_my_role(
+ await AgentRecipient._text_tags_my_role(
client, None, "@manager review", "room-1"
)
is False
@@ -257,7 +263,7 @@ class TestRoleMentionRouting:
# Boundary safety: holding "lead" must not match "@lead-dev".
client = _role_client("ephemeral-cc", held_role="lead")
assert (
- await AgentConsumer._text_tags_my_role(
+ await AgentRecipient._text_tags_my_role(
client, None, "@lead-dev ping", "room-1"
)
is False
@@ -266,7 +272,9 @@ class TestRoleMentionRouting:
async def test_no_at_short_circuits(self) -> None:
client = _role_client("ephemeral-cc", held_role="manager")
assert (
- await AgentConsumer._text_tags_my_role(client, None, "no at here", "room-1")
+ await AgentRecipient._text_tags_my_role(
+ client, None, "no at here", "room-1"
+ )
is False
)
@@ -293,7 +301,7 @@ class TestAliasMentionRouting:
async def test_alias_addresses_agent(self) -> None:
client = _alias_client("claude-code.aq-switch-2", {"room-1": "fixer"})
assert (
- await AgentConsumer._text_tags_my_alias(
+ await AgentRecipient._text_tags_my_alias(
client, None, "@fixer please look", "room-1"
)
is True
@@ -302,7 +310,9 @@ class TestAliasMentionRouting:
async def test_alias_case_insensitive(self) -> None:
client = _alias_client("claude-code.aq-switch-2", {"room-1": "fixer"})
assert (
- await AgentConsumer._text_tags_my_alias(client, None, "@FIXER hi", "room-1")
+ await AgentRecipient._text_tags_my_alias(
+ client, None, "@FIXER hi", "room-1"
+ )
is True
)
@@ -310,14 +320,18 @@ class TestAliasMentionRouting:
# Alias set in room-1 only — tagging it in room-2 does not address us.
client = _alias_client("claude-code.aq-switch-2", {"room-1": "fixer"})
assert (
- await AgentConsumer._text_tags_my_alias(client, None, "@fixer hi", "room-2")
+ await AgentRecipient._text_tags_my_alias(
+ client, None, "@fixer hi", "room-2"
+ )
is False
)
async def test_no_alias_not_addressed(self) -> None:
client = _alias_client("claude-code.aq-switch-2", {})
assert (
- await AgentConsumer._text_tags_my_alias(client, None, "@fixer hi", "room-1")
+ await AgentRecipient._text_tags_my_alias(
+ client, None, "@fixer hi", "room-1"
+ )
is False
)
@@ -325,7 +339,7 @@ class TestAliasMentionRouting:
# Boundary safety: alias "fix" must not match "@fixer".
client = _alias_client("claude-code.aq-switch-2", {"room-1": "fix"})
assert (
- await AgentConsumer._text_tags_my_alias(
+ await AgentRecipient._text_tags_my_alias(
client, None, "@fixer ping", "room-1"
)
is False
@@ -334,7 +348,7 @@ class TestAliasMentionRouting:
async def test_no_at_short_circuits(self) -> None:
client = _alias_client("claude-code.aq-switch-2", {"room-1": "fixer"})
assert (
- await AgentConsumer._text_tags_my_alias(
+ await AgentRecipient._text_tags_my_alias(
client, None, "no at here", "room-1"
)
is False
@@ -351,21 +365,21 @@ class TestCommandRoleTargeting:
async def test_holder_targeted_by_role(self) -> None:
client = _role_client("ephemeral-cc", held_role="manager")
assert (
- await AgentConsumer._text_tags_my_role(client, None, "@manager", "room-1")
+ await AgentRecipient._text_tags_my_role(client, None, "@manager", "room-1")
is True
)
async def test_non_holder_not_targeted(self) -> None:
client = _role_client("ephemeral-cc", held_role=None)
assert (
- await AgentConsumer._text_tags_my_role(client, None, "@manager", "room-1")
+ await AgentRecipient._text_tags_my_role(client, None, "@manager", "room-1")
is False
)
async def test_role_prefix_not_targeted(self) -> None:
client = _role_client("ephemeral-cc", held_role="lead")
assert (
- await AgentConsumer._text_tags_my_role(client, None, "@lead-dev", "room-1")
+ await AgentRecipient._text_tags_my_role(client, None, "@lead-dev", "room-1")
is False
)
@@ -467,7 +481,7 @@ class TestUnavailableHereReply:
) -> None:
# Holds a role here, but its assuming session is attending room-B.
client = _unavailable_client(live_rooms=["room-B"], role_here=True)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == _role_elsewhere_message("Room room-B")
@@ -478,7 +492,7 @@ class TestUnavailableHereReply:
# Defers to the known-agent reply (paste-ready connect command) with
# the other rooms threaded in, rather than the role-flavoured wording.
client = _unavailable_client(live_rooms=["room-B", "room-C"], role_here=False)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == "ELSEWHERE: Room room-B, Room room-C"
@@ -489,7 +503,7 @@ class TestUnavailableHereReply:
# The only live session is the addressed room itself → nothing elsewhere.
client = _unavailable_client(live_rooms=["room-A"], role_here=False)
assert (
- await AgentConsumer._reply_when_unavailable_here(
+ await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
== "OFFLINE"
@@ -498,7 +512,7 @@ class TestUnavailableHereReply:
async def test_no_live_sessions_is_offline(self, db: AsyncSession) -> None:
client = _unavailable_client(live_rooms=[], role_here=False)
assert (
- await AgentConsumer._reply_when_unavailable_here(
+ await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
== "OFFLINE"
@@ -513,14 +527,14 @@ class TestConnectedNotLive:
self, db: AsyncSession
) -> None:
client = _unavailable_client(live_rooms=[], role_here=False, bound_here=True)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == "NOT_LIVE"
async def test_no_binding_is_offline(self, db: AsyncSession) -> None:
client = _unavailable_client(live_rooms=[], role_here=False, bound_here=False)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == "OFFLINE"
@@ -535,7 +549,7 @@ class TestConnectedNotLive:
connection_model="session_passive",
bound_here=True,
)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == "OFFLINE"
@@ -548,7 +562,7 @@ class TestConnectedNotLive:
# not live → connected-not-live, not a confusing same-name pointer.
# _here() is room-A / "Room A"; room id "A" → name "Room A" (collision).
client = _unavailable_client(live_rooms=["A"], role_here=False, bound_here=True)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == "NOT_LIVE"
@@ -560,7 +574,7 @@ class TestConnectedNotLive:
client = _unavailable_client(
live_rooms=["room-B"], role_here=False, bound_here=True
)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == "ELSEWHERE: Room room-B"
@@ -575,7 +589,7 @@ class TestStartingASessionIsAPromise:
client = _unavailable_client(
live_rooms=[], role_here=False, connection_model="auto_session", spawns=True
)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == _STARTING_SESSION_MESSAGE
@@ -590,7 +604,7 @@ class TestStartingASessionIsAPromise:
connection_model="auto_session",
watching=True,
)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == _STARTING_SESSION_MESSAGE
@@ -611,7 +625,7 @@ class TestStartingASessionIsAPromise:
connection_model="auto_session",
connected=True,
)
- msg = await AgentConsumer._reply_when_unavailable_here(
+ msg = await AgentRecipient._reply_when_unavailable_here(
client, db, client.agent, _here(), "asker"
)
assert msg == "OFFLINE"
diff --git a/core/tests/switch_core/clients/test_no_session_reply_threading.py b/core/tests/switch_core/delivery/test_no_session_reply_threading.py
similarity index 82%
rename from core/tests/switch_core/clients/test_no_session_reply_threading.py
rename to core/tests/switch_core/delivery/test_no_session_reply_threading.py
index e6464f497..77fa99252 100644
--- a/core/tests/switch_core/clients/test_no_session_reply_threading.py
+++ b/core/tests/switch_core/delivery/test_no_session_reply_threading.py
@@ -6,13 +6,13 @@ from types import SimpleNamespace
import pytest
from switch_core.clients.admin_messages import PLATFORM_MARKER
-from switch_core.clients.agent_consumer import (
+from switch_core.db.models import TENANT_ZERO_ID
+from switch_core.delivery.recipient import (
_STARTING_SESSION_MESSAGE,
AUTO_REPLY_FLAG,
- AgentConsumer,
+ AgentRecipient,
_GateOutcome,
)
-from switch_core.db.models import TENANT_ZERO_ID
from switch_core.transport import InboundMessage, RoomRef
@@ -21,11 +21,6 @@ async def _session_factory(): # type: ignore[no-untyped-def]
yield object()
-async def _ignore_enqueue(*_args: object, **_kwargs: object) -> int:
- # What the agent is sent is not under test here, only where the reply goes.
- return 0
-
-
class _Recorder:
"""Captures the send_message call so we can assert on thread_root_id."""
@@ -50,10 +45,7 @@ def _fake_self(
send_message: _Recorder,
unavailable_reply: str = "I don't have a session connected to this room.",
) -> SimpleNamespace:
- """A minimal AgentConsumer stand-in: addressed, offline, non-moderator."""
-
- async def _resolve_room_meta(_transport_room_id: str) -> SimpleNamespace:
- return _meta()
+ """A minimal AgentRecipient stand-in: addressed, offline, non-moderator."""
def _addressed_without_lookup(_event: object, _meta: object) -> bool:
return True
@@ -84,24 +76,34 @@ def _fake_self(
agent=SimpleNamespace(id="agent-1", name="cc-bug-fixing"),
tenant_id=TENANT_ZERO_ID,
session_factory=_session_factory,
- _resolve_room_meta=_resolve_room_meta,
+ _meta=_meta(),
+ _room=RoomRef(room_id="!matrix:server"),
_addressed_without_lookup=_addressed_without_lookup,
_compute_addressed=_compute_addressed,
_fresh_agent=_fresh_agent,
_gate_addressed=_gate_addressed,
_is_available=_is_available,
_reply_when_unavailable_here=_reply_when_unavailable_here,
- _triggered_by_auto_reply=AgentConsumer._triggered_by_auto_reply,
+ _triggered_by_auto_reply=AgentRecipient._triggered_by_auto_reply,
actor=SimpleNamespace(send_message=send_message),
- _event_buffer=SimpleNamespace(enqueue=_ignore_enqueue),
_report_addressed=lambda *a, **k: None,
)
# Exercise the real sender-tagging and auto-reply helpers.
- ns._sender_handle = AgentConsumer._sender_handle.__get__(ns)
- ns._post_auto_reply = AgentConsumer._post_auto_reply.__get__(ns)
+ ns._sender_handle = AgentRecipient._sender_handle.__get__(ns)
+ ns._post_auto_reply = AgentRecipient._post_auto_reply.__get__(ns)
+ ns._after_message = AgentRecipient._after_message.__get__(ns)
return ns
+async def _deliver(fake: SimpleNamespace, event: InboundMessage) -> None:
+ """Decide the message for the agent as the router does, then do what it
+ leaves to do once it has committed. What the agent is sent is not under
+ test here, only where the reply goes."""
+ delivery = await AgentRecipient.decide_message(fake, None, event) # type: ignore[arg-type]
+ if delivery.after is not None:
+ await delivery.after(event)
+
+
def _event(thread_id: str | None, *, is_auto_reply: bool = False) -> InboundMessage:
content: dict[str, object] = {"sender_name": "louisa"}
if is_auto_reply:
@@ -121,10 +123,7 @@ def _event(thread_id: str | None, *, is_auto_reply: bool = False) -> InboundMess
@pytest.mark.asyncio
async def test_no_session_reply_threads_under_triggering_mention() -> None:
send_message = _Recorder()
- room = RoomRef(room_id="!matrix:server")
- await AgentConsumer.on_message(
- _fake_self(send_message), room, _event(thread_id="$thread-root")
- )
+ await _deliver(_fake_self(send_message), _event(thread_id="$thread-root"))
assert len(send_message.calls) == 1
assert send_message.calls[0]["thread_root_id"] == "$thread-root"
@@ -142,7 +141,7 @@ async def test_no_session_reply_threads_under_triggering_mention() -> None:
@pytest.mark.asyncio
async def test_a_command_reply_is_not_metered() -> None:
send_message = _Recorder()
- await AgentConsumer.reply_command(
+ await AgentRecipient.reply_command(
SimpleNamespace(actor=SimpleNamespace(send_message=send_message)), # type: ignore[arg-type]
"!matrix:server",
"the result",
@@ -157,11 +156,8 @@ async def test_no_session_reply_not_triggered_by_another_auto_reply() -> None:
# must not ping-pong. A message already flagged as an auto-reply gets no
# reply, so the loop can never form.
send_message = _Recorder()
- room = RoomRef(room_id="!matrix:server")
- await AgentConsumer.on_message(
- _fake_self(send_message),
- room,
- _event(thread_id="$thread-root", is_auto_reply=True),
+ await _deliver(
+ _fake_self(send_message), _event(thread_id="$thread-root", is_auto_reply=True)
)
assert send_message.calls == []
@@ -173,12 +169,10 @@ async def test_no_session_reply_does_not_double_tag_the_asker() -> None:
# configured operator). When that's the same person as the asker, we must
# tag them once, not "@louisa @louisa".
send_message = _Recorder()
- room = RoomRef(room_id="!matrix:server")
- await AgentConsumer.on_message(
+ await _deliver(
_fake_self(
send_message, unavailable_reply="@louisa\n\nmy operator should run …"
),
- room,
_event(thread_id=None),
)
@@ -193,12 +187,10 @@ async def test_no_session_reply_tags_distinct_asker_and_operator() -> None:
# When the reply's embedded mention is a DIFFERENT person (the operator),
# both the asker and that operator are tagged.
send_message = _Recorder()
- room = RoomRef(room_id="!matrix:server")
- await AgentConsumer.on_message(
+ await _deliver(
_fake_self(
send_message, unavailable_reply="@operator\n\nmy operator should run …"
),
- room,
_event(thread_id=None),
)
@@ -225,11 +217,8 @@ class TestStartingSessionNoticeGoesWhereItWasAsked:
@pytest.mark.asyncio
async def test_asked_at_the_root_it_answers_at_the_root(self) -> None:
send_message = _Recorder()
- room = RoomRef(room_id="!matrix:server")
-
- await AgentConsumer.on_message(
+ await _deliver(
_fake_self(send_message, unavailable_reply=_STARTING_SESSION_MESSAGE),
- room,
_event(thread_id=None),
)
@@ -242,11 +231,8 @@ class TestStartingSessionNoticeGoesWhereItWasAsked:
# The other half of the rule — replying at the root here would strand
# the notice away from the conversation it belongs to.
send_message = _Recorder()
- room = RoomRef(room_id="!matrix:server")
-
- await AgentConsumer.on_message(
+ await _deliver(
_fake_self(send_message, unavailable_reply=_STARTING_SESSION_MESSAGE),
- room,
_event(thread_id="$thread-root"),
)
@@ -266,11 +252,7 @@ class TestTerminalReplyThreadsOffTheTrigger:
@pytest.mark.asyncio
async def test_asked_at_the_root_it_opens_a_thread_on_the_trigger(self) -> None:
send_message = _Recorder()
- room = RoomRef(room_id="!matrix:server")
-
- await AgentConsumer.on_message(
- _fake_self(send_message), room, _event(thread_id=None)
- )
+ await _deliver(_fake_self(send_message), _event(thread_id=None))
assert len(send_message.calls) == 1
assert send_message.calls[0]["thread_root_id"] == "$trigger"
@@ -280,11 +262,7 @@ class TestTerminalReplyThreadsOffTheTrigger:
# Already in a thread: the trigger's thread root IS that thread, so it
# lands there rather than starting a nested one.
send_message = _Recorder()
- room = RoomRef(room_id="!matrix:server")
-
- await AgentConsumer.on_message(
- _fake_self(send_message), room, _event(thread_id="$thread-root")
- )
+ await _deliver(_fake_self(send_message), _event(thread_id="$thread-root"))
assert send_message.calls[0]["thread_root_id"] == "$thread-root"
@@ -295,13 +273,12 @@ async def test_a_kickoff_that_wants_the_channel_gets_its_reply_at_top_level() ->
# but flags that the work belongs in the channel: the agent's reply goes
# to the top level, not under the kickoff.
send_message = _Recorder()
- room = RoomRef(room_id="!matrix:server")
event = _event(thread_id="$kickoff-headline")
event.content[PLATFORM_MARKER] = {
"on_behalf_of": {"user_id": "u9", "name": "dantas.abel"},
"reply_in_channel": True,
}
- await AgentConsumer.on_message(_fake_self(send_message), room, event)
+ await _deliver(_fake_self(send_message), event)
assert len(send_message.calls) == 1
assert send_message.calls[0]["thread_root_id"] is None
diff --git a/core/tests/switch_core/clients/test_offline_owner_reply.py b/core/tests/switch_core/delivery/test_offline_owner_reply.py
similarity index 97%
rename from core/tests/switch_core/clients/test_offline_owner_reply.py
rename to core/tests/switch_core/delivery/test_offline_owner_reply.py
index 04d41aaeb..4bce42283 100644
--- a/core/tests/switch_core/clients/test_offline_owner_reply.py
+++ b/core/tests/switch_core/delivery/test_offline_owner_reply.py
@@ -2,7 +2,7 @@ from __future__ import annotations
from types import SimpleNamespace
-from switch_core.clients.agent_consumer import AgentConsumer, _offline_owner_message
+from switch_core.delivery.recipient import AgentRecipient, _offline_owner_message
"""The reply an auto_session agent gives when it is addressed and nothing is
there to bring it online (CHOO-2344).
@@ -54,7 +54,7 @@ def _client(owner_handle: str | None) -> SimpleNamespace:
async def _reply(
client: SimpleNamespace, agent: SimpleNamespace, **kwargs: object
) -> str:
- return await AgentConsumer._unavailable_reply(
+ return await AgentRecipient._unavailable_reply(
client,
None,
_meta(),
diff --git a/core/tests/switch_core/clients/test_owner_notification_handle.py b/core/tests/switch_core/delivery/test_owner_notification_handle.py
similarity index 87%
rename from core/tests/switch_core/clients/test_owner_notification_handle.py
rename to core/tests/switch_core/delivery/test_owner_notification_handle.py
index 902884054..6e2f48a0c 100644
--- a/core/tests/switch_core/clients/test_owner_notification_handle.py
+++ b/core/tests/switch_core/delivery/test_owner_notification_handle.py
@@ -14,7 +14,7 @@ from __future__ import annotations
from types import SimpleNamespace
-from switch_core.clients.agent_consumer import AgentConsumer
+from switch_core.delivery.recipient import AgentRecipient
def _claimed(bridge_id: str, username: str) -> SimpleNamespace:
@@ -39,7 +39,7 @@ class TestResolvingTheOwnersHandle:
async def test_the_owners_account_on_this_bridge(self) -> None:
client = _client([_claimed("slack-1", "louis.amaudruz")])
- handle = await AgentConsumer.owner_handle_in(
+ handle = await AgentRecipient.owner_handle_in(
client, None, _agent("u1"), "slack-1"
)
@@ -53,7 +53,7 @@ class TestResolvingTheOwnersHandle:
)
assert (
- await AgentConsumer.owner_handle_in(
+ await AgentRecipient.owner_handle_in(
client, None, _agent("u1"), "telegram-1"
)
== "louisa"
@@ -65,7 +65,7 @@ class TestResolvingTheOwnersHandle:
client = _client([_claimed("slack-1", "louis.amaudruz")])
assert (
- await AgentConsumer.owner_handle_in(
+ await AgentRecipient.owner_handle_in(
client, None, _agent("u1"), "telegram-1"
)
is None
@@ -75,7 +75,7 @@ class TestResolvingTheOwnersHandle:
client = _client([_claimed("slack-1", "louis.amaudruz")])
assert (
- await AgentConsumer.owner_handle_in(client, None, _agent(None), "slack-1")
+ await AgentRecipient.owner_handle_in(client, None, _agent(None), "slack-1")
is None
)
@@ -84,7 +84,7 @@ class TestResolvingTheOwnersHandle:
client = _client([_claimed("slack-1", "louis.amaudruz")])
assert (
- await AgentConsumer.owner_handle_in(client, None, _agent("u1"), None)
+ await AgentRecipient.owner_handle_in(client, None, _agent("u1"), None)
is None
)
@@ -93,10 +93,10 @@ class TestResolvingTheOwnersHandle:
# between one message and the next.
client = _client([_claimed("slack-1", "zoe"), _claimed("slack-1", "adam")])
- first = await AgentConsumer.owner_handle_in(
+ first = await AgentRecipient.owner_handle_in(
client, None, _agent("u1"), "slack-1"
)
- second = await AgentConsumer.owner_handle_in(
+ second = await AgentRecipient.owner_handle_in(
client, None, _agent("u1"), "slack-1"
)
diff --git a/core/tests/switch_core/clients/test_platform_sender_replies.py b/core/tests/switch_core/delivery/test_platform_sender_replies.py
similarity index 91%
rename from core/tests/switch_core/clients/test_platform_sender_replies.py
rename to core/tests/switch_core/delivery/test_platform_sender_replies.py
index 1e9188aea..150661b6a 100644
--- a/core/tests/switch_core/clients/test_platform_sender_replies.py
+++ b/core/tests/switch_core/delivery/test_platform_sender_replies.py
@@ -7,7 +7,7 @@ from __future__ import annotations
from types import SimpleNamespace
from switch_core.clients.admin_messages import PLATFORM_MARKER
-from switch_core.clients.agent_consumer import AgentConsumer
+from switch_core.delivery.recipient import AgentRecipient
from switch_core.transport import InboundMessage
@@ -25,7 +25,7 @@ def _event(content: dict) -> InboundMessage:
def _handle(event: InboundMessage) -> str:
ns = SimpleNamespace()
- return AgentConsumer._sender_handle(ns, event) # type: ignore[arg-type]
+ return AgentRecipient._sender_handle(ns, event) # type: ignore[arg-type]
def test_reply_handle_is_the_person_behind_a_platform_message() -> None:
diff --git a/core/tests/switch_core/delivery/test_removed_from_room.py b/core/tests/switch_core/delivery/test_removed_from_room.py
new file mode 100644
index 000000000..6c566d0e3
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_removed_from_room.py
@@ -0,0 +1,310 @@
+"""What taking an agent out of a room does to what it holds.
+
+Its own events are kept for the retention window and served to a long poll,
+to the notification stream, and to a stream resuming from an old position,
+none of which has a membership of its own to apply. So the removal deletes
+the agent's own events in the room, in the transaction that removes the
+membership, and lets go of the room in this process's memory once it
+commits: the connections that claimed it, and a controller streaming the
+agent.
+
+A message written while the agent is being removed must not leave an event
+behind in a room it is no longer in: the removal and the router lock the same
+row of the agent's (`agent_event_heads`), and the router stores an event only
+for a member. Nor may it leave the agent anything to say there: what a row
+leaves an agent to say is said after the row commits, and only by a member.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections.abc import Callable, Coroutine
+from typing import Any
+
+import pytest
+from sqlalchemy import update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.bridges.agent.protocol.agent_connections import (
+ PROTOCOL_VERSION,
+ ClientDeclaration,
+)
+from switch_core.bridges.agent.protocol.positions import OWN_EVENTS
+from switch_core.db.models import TENANT_ZERO_ID, Agent, Client, Message, Room
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.provisioning.postgres import ProvisioningError
+from switch_core.transport.postgres import new_event_id
+from tests.routing import (
+ Process,
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+ said_by,
+)
+
+# Lets no person address the agent, so whoever names it is refused.
+_REFUSES_EVERYONE = {"rules": [{"agents": ["nobody"], "users": []}]}
+
+
+async def scene(session_factory: async_sessionmaker[AsyncSession]): # type: ignore[no-untyped-def]
+ """An agent in two rooms, addressed once in each by alice."""
+ process = process_on(session_factory)
+ left = await make_room(session_factory, "left")
+ kept = await make_room(session_factory, "kept")
+ alice = await make_person(session_factory, "alice")
+ agent, client = await make_agent(session_factory, "fixer")
+ for room in (left, kept):
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, client, agent=agent)
+ await process.say(room, f"@fixer said in {room.name}", sender=alice)
+ await process.idle()
+ return process, left, kept, alice, agent, client
+
+
+async def test_the_rooms_retained_events_are_forgotten(
+ session_factory: async_sessionmaker[AsyncSession], scene
+) -> None: # type: ignore[no-untyped-def]
+ process, left, kept, _, agent, client = scene
+
+ await process.provisioning.kick_user(
+ left.transport_room_id, client.transport_user_id
+ )
+
+ # Read at the level every reader is built on, filter or no filter: this is
+ # what closes it for the stream, which has no membership of its own to
+ # apply and would otherwise replay the room on every resume.
+ assert [
+ item.room_id
+ for item in await process.buffer.read_from(TENANT_ZERO_ID, agent.id, 0)
+ ] == [kept.id]
+
+
+async def test_the_rooms_claim_is_released(scene) -> None: # type: ignore[no-untyped-def]
+ process, left, _, _, agent, client = scene
+ conn = process.connections.open(
+ agent_id=agent.id,
+ connection_id="c1",
+ scope="single",
+ delivery_filter="all",
+ spawn_capable=False,
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
+ expected_generation=None,
+ )
+ process.connections.claim_room(conn, left.id)
+
+ await process.provisioning.kick_user(
+ left.transport_room_id, client.transport_user_id
+ )
+
+ assert conn.rooms == set()
+
+
+async def test_a_room_that_cannot_be_resolved_drops_nothing(scene) -> None: # type: ignore[no-untyped-def]
+ """Rather than guessing which room was meant and emptying the wrong one."""
+ process, left, kept, _, agent, client = scene
+
+ with pytest.raises(ProvisioningError):
+ await process.provisioning.kick_user("!unknown:test", client.transport_user_id)
+
+ assert sorted(
+ item.room_id
+ for item in await process.buffer.read_from(TENANT_ZERO_ID, agent.id, 0)
+ ) == sorted([left.id, kept.id])
+
+
+async def test_a_message_routed_while_the_agent_is_removed_leaves_nothing(
+ session_factory: async_sessionmaker[AsyncSession], scene
+) -> None: # type: ignore[no-untyped-def]
+ """The message's transaction holds the agent's row in `agent_event_heads`
+ when the removal starts, so the removal waits for it and deletes what it
+ stored."""
+ process, left, kept, alice, agent, client = scene
+
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ routed = await process.router.store(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ room_id=left.id,
+ transport_room_id=left.transport_room_id,
+ message=_message(left.id, alice, "@fixer one more"),
+ attachments=[],
+ )
+ removal = asyncio.create_task(
+ process.provisioning.kick_user(
+ left.transport_room_id, client.transport_user_id
+ )
+ )
+ await asyncio.sleep(0.3)
+ assert not removal.done(), "the removal did not wait for the message"
+ await session.commit()
+ routed.committed()
+ await removal
+
+ assert [row.room_id for row in await own_events(session_factory, agent.id)] == [
+ kept.id
+ ]
+
+
+async def test_a_message_decided_before_the_removal_stores_nothing_after_it(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+ scene, # type: ignore[no-untyped-def]
+) -> None:
+ """Decided while the agent was a member, written once it no longer is:
+ the event is stored only for a member, so nothing is."""
+ process, left, kept, alice, agent, client = scene
+ decided = asyncio.Event()
+ removed = asyncio.Event()
+ create = MessageStore.create
+
+ async def create_after_the_removal(self, session, message, attachments): # type: ignore[no-untyped-def]
+ if message.room_id == left.id and message.body == "@fixer late":
+ decided.set()
+ await removed.wait()
+ return await create(self, session, message, attachments)
+
+ monkeypatch.setattr(MessageStore, "create", create_after_the_removal)
+ sending = asyncio.create_task(process.say(left, "@fixer late", sender=alice))
+ await decided.wait()
+ await process.provisioning.kick_user(
+ left.transport_room_id, client.transport_user_id
+ )
+ removed.set()
+ await sending
+ await process.idle()
+
+ assert {row.room_id for row in await own_events(session_factory, agent.id)} == {
+ kept.id
+ }
+
+
+async def _names_it(process: Process, room: Room, sender: Client) -> None:
+ await process.say(room, "@fixer late", sender=sender)
+
+
+async def _sends_it_a_file(process: Process, room: Room, sender: Client) -> None:
+ await process.send_file(room, "late.png", sender=sender, caption="@fixer late")
+
+
+async def _resets_it(process: Process, room: Room, sender: Client) -> None:
+ await process.command(room, "reset", "@fixer", sender=sender)
+
+
+ ("policy", "write"),
+ [
+ # Named while it is offline: it owes an offline notice.
+ (None, _names_it),
+ # Named by a sender its policy refuses: it owes a refusal.
+ (_REFUSES_EVERYONE, _names_it),
+ # Sent a file by a sender its policy refuses: it owes a refusal.
+ (_REFUSES_EVERYONE, _sends_it_a_file),
+ # Told to reset, which it cannot do: it owes the command's answer.
+ (None, _resets_it),
+ ],
+ ids=["offline-notice", "refusal", "file-refusal", "command-answer"],
+)
+async def test_a_row_decided_before_the_removal_leaves_the_agent_nothing_to_say(
+ policy: dict[str, Any] | None,
+ write: Callable[[Process, Room, Client], Coroutine[Any, Any, None]],
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+ scene, # type: ignore[no-untyped-def]
+) -> None:
+ """Decided while the agent was a member, committed once it no longer is:
+ what the row left the agent to say is not said, as the agent, in a room
+ it has left."""
+ process, left, _, alice, agent, client = scene
+ if policy is not None:
+ await _set_policy(session_factory, agent, policy)
+ said_before = await _said(session_factory, left, client)
+ decided = asyncio.Event()
+ removed = asyncio.Event()
+ create = MessageStore.create
+
+ async def create_after_the_removal(self, session, message, attachments): # type: ignore[no-untyped-def]
+ if message.room_id == left.id and message.sender_client_id == alice.id:
+ decided.set()
+ await removed.wait()
+ return await create(self, session, message, attachments)
+
+ monkeypatch.setattr(MessageStore, "create", create_after_the_removal)
+ sending = asyncio.create_task(write(process, left, alice))
+ await decided.wait()
+ await process.provisioning.kick_user(
+ left.transport_room_id, client.transport_user_id
+ )
+ removed.set()
+ await sending
+ await process.idle()
+
+ assert await _said(session_factory, left, client) == said_before
+
+
+async def test_an_agent_taken_out_before_its_greeting_does_not_greet(
+ monkeypatch: pytest.MonkeyPatch,
+ session_factory: async_sessionmaker[AsyncSession],
+ scene, # type: ignore[no-untyped-def]
+) -> None:
+ """Its arrival committed and left it a greeting to say, and it was taken
+ out of the room before the greeting's turn came: the room's earlier work
+ runs first, and a removal does not wait for it."""
+ process, _, _, _, _, client = scene
+ visited = await make_room(session_factory, "visited")
+ # The work the arrival leaves is held back, as a room still busy with
+ # earlier work would hold it, and handed on once the agent is out.
+ owed: list[tuple[Any, ...]] = []
+ owe = process.router.after_commit
+ monkeypatch.setattr(process.router, "after_commit", lambda *work: owed.append(work))
+
+ await process.provisioning.invite_to_room(
+ visited.transport_room_id, client.transport_user_id
+ )
+ await process.provisioning.kick_user(
+ visited.transport_room_id, client.transport_user_id
+ )
+ assert owed, "the arrival left the agent no greeting to say"
+ for work in owed:
+ owe(*work)
+ await process.idle()
+
+ assert await _said(session_factory, visited, client) == []
+
+
+async def _set_policy(
+ session_factory: async_sessionmaker[AsyncSession],
+ agent: Agent,
+ policy: dict[str, Any],
+) -> None:
+ async with tenant_session(session_factory, agent.tenant_id) as session:
+ await session.execute(
+ update(Agent).where(Agent.id == agent.id).values(addressing_policy=policy)
+ )
+ await session.commit()
+
+
+async def _said(
+ session_factory: async_sessionmaker[AsyncSession], room: Room, client: Client
+) -> list[str | None]:
+ """What `client` has said in `room`, oldest first."""
+ return [said.body for said in await said_by(session_factory, room, client)]
+
+
+def _message(room_id: str, sender, body: str) -> Message: # type: ignore[no-untyped-def]
+ return Message(
+ room_id=room_id,
+ transport_event_id=new_event_id(),
+ sender_id=sender.transport_user_id,
+ sender_client_id=sender.id,
+ sender_name=sender.display_name,
+ event_type="m.room.message",
+ msgtype="m.text",
+ body=body,
+ content={"body": body, "msgtype": "m.text"},
+ )
diff --git a/core/tests/switch_core/delivery/test_reports_addressed.py b/core/tests/switch_core/delivery/test_reports_addressed.py
new file mode 100644
index 000000000..e9a4f888e
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_reports_addressed.py
@@ -0,0 +1,192 @@
+"""The router reports each message an agent was actually asked to act on.
+
+"Asked" means addressed *and* let through: a message the addressing policy or a
+budget turned away was not a request the agent received, and Switch's own
+auto-replies are not anyone asking for anything. Whether the agent was there
+to take it is reported alongside, not used to leave the message out.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+from typing import Any
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.bridges.agent.protocol.agent_connections import (
+ HEARTBEAT_LAPSED,
+ PROTOCOL_VERSION,
+ ClientDeclaration,
+)
+from switch_core.bridges.agent.protocol.positions import OWN_EVENTS
+from switch_core.clients.admin_messages import AUTO_REPLY_FLAG, PLATFORM_MARKER
+from switch_core.db.models import Agent, Client, Room
+from switch_core.delivery import recipient as recipient_module
+from switch_core.observability.metrics import MetricsRegistry, install, uninstall
+from switch_core.observability.throttle import WarningThrottle
+from tests.routing import (
+ Process,
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+)
+
+
+class _BrokenMessageTelemetry:
+ enabled = True
+
+ def agent_addressed(self, **kwargs: Any) -> None:
+ raise RuntimeError("telemetry bug")
+
+ def observe(self, message: object) -> None:
+ return None
+
+
+class _Scene:
+ process: Process
+ room: Room
+ alice: Client
+ agent: Agent
+
+
+async def scene(session_factory: async_sessionmaker[AsyncSession]) -> _Scene:
+ """`agent-a` in a room with alice, connected and so live there."""
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ agent, client = await make_agent(session_factory, "agent-a")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, client, agent=agent)
+ process.connections.open(
+ agent_id=agent.id,
+ connection_id="c1",
+ scope="all",
+ delivery_filter="addressed",
+ spawn_capable=False,
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
+ expected_generation=None,
+ )
+ return _Scene(process, room, alice, agent)
+
+
+async def _say(scene: _Scene, body: str, **content: object) -> None:
+ await scene.process.say(
+ scene.room, body, sender=scene.alice, extra_content=content or None
+ )
+ await scene.process.idle()
+
+
+async def test_an_addressed_message_is_reported_once(scene: _Scene) -> None:
+ await _say(scene, "@agent-a can you help")
+
+ assert scene.process.telemetry.addressed == [
+ {
+ "tenant_id": scene.room.tenant_id,
+ "room_id": scene.room.id,
+ "sender_transport_user_id": scene.alice.transport_user_id,
+ "from_platform": False,
+ "agent_metadata": None,
+ "agent_live": True,
+ "has_attachment": False,
+ }
+ ]
+
+
+async def test_room_chatter_is_not_reported(scene: _Scene) -> None:
+ await _say(scene, "just talking")
+
+ assert scene.process.telemetry.addressed == []
+
+
+async def test_a_message_the_policy_turned_away_is_not_reported(
+ session_factory: async_sessionmaker[AsyncSession], scene: _Scene
+) -> None:
+ restricted, client = await make_agent(
+ session_factory,
+ "restricted",
+ addressing_policy={"rules": [{"agents": ["nobody"], "users": []}]},
+ )
+ await add_member(session_factory, scene.room, client, agent=restricted)
+
+ await _say(scene, "@restricted can you help")
+
+ assert scene.process.telemetry.addressed == []
+
+
+async def test_an_auto_reply_is_not_reported(scene: _Scene) -> None:
+ await _say(scene, "@agent-a can you help", **{AUTO_REPLY_FLAG: True})
+
+ assert scene.process.telemetry.addressed == []
+
+
+async def test_a_platform_message_says_it_came_from_the_platform(
+ scene: _Scene,
+) -> None:
+ # Said by a person here rather than the admin client, which leaves the
+ # open policy admitting it; the marker is what the report reads.
+ await _say(scene, "@agent-a can you help", **{PLATFORM_MARKER: {}})
+
+ [report] = scene.process.telemetry.addressed
+ assert report["from_platform"] is True
+
+
+async def test_a_message_to_an_offline_agent_is_reported_as_not_live(
+ scene: _Scene,
+) -> None:
+ scene.process.connections.close("c1", HEARTBEAT_LAPSED)
+
+ await _say(scene, "@agent-a can you help")
+
+ [report] = scene.process.telemetry.addressed
+ assert report["agent_live"] is False
+
+
+async def test_a_telemetry_failure_does_not_cost_the_agent_the_message(
+ monkeypatch: pytest.MonkeyPatch,
+ caplog: pytest.LogCaptureFixture,
+ session_factory: async_sessionmaker[AsyncSession],
+ scene: _Scene,
+) -> None:
+ """Caught, so it is counted, and logged once rather than once per
+ message."""
+ router = scene.process.router
+ router.services = dataclasses.replace(
+ router.services,
+ message_telemetry=_BrokenMessageTelemetry(), # type: ignore[arg-type]
+ )
+ registry = MetricsRegistry()
+ install(registry)
+ monkeypatch.setattr(recipient_module, "_report_failures", WarningThrottle(60.0))
+
+ try:
+ with caplog.at_level("ERROR"):
+ await _say(scene, "@agent-a first")
+ await _say(scene, "@agent-a second")
+ failures = next(
+ p
+ for p in registry.collect()
+ if p.name == "switch.messages.observer_failures"
+ )
+ finally:
+ uninstall()
+
+ assert len(await own_events(session_factory, scene.agent.id)) == 2
+ assert failures.numbers[0].value == 2.0
+ assert dict(failures.numbers[0].attributes) == {"actor": "agent"}
+ assert (
+ len(
+ [
+ r
+ for r in caplog.records
+ if "Could not report a message addressed" in r.getMessage()
+ ]
+ )
+ == 1
+ )
diff --git a/core/tests/switch_core/delivery/test_role_held_elsewhere.py b/core/tests/switch_core/delivery/test_role_held_elsewhere.py
new file mode 100644
index 000000000..3b3ea460e
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_role_held_elsewhere.py
@@ -0,0 +1,98 @@
+"""A role held over a connection another process holds is found by any process.
+
+A message tagged `@reviewer` is for the agent that live-holds the role. A
+holder that stopped renewing the lease itself is live for as long as the
+connection it took the seat over is, and that connection may be held by
+another process, whose copy here can be missing or late. So the router asks
+the recorded connections (`agent_connections`) as well as this process's
+memory: the seat is held while its connection's row is open and the process
+named on it holds its lease.
+"""
+
+from __future__ import annotations
+
+import uuid
+from datetime import UTC, datetime, timedelta
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import (
+ CONNECTION_INCARNATION_FLOOR,
+ AgentConnectionRecord,
+ RoleLease,
+ RoomRole,
+ SwitchCoreProcess,
+)
+from switch_core.db.session_scope import tenant_session
+from tests.routing import (
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+)
+
+
+ ("holder_beat", "handed"),
+ [
+ # The process holding the connection is live: the seat is held.
+ (timedelta(seconds=0), 1),
+ # It stopped beating long ago: the connection, and the seat, are gone.
+ (timedelta(minutes=10), 0),
+ ],
+)
+async def test_a_role_held_over_another_process_s_connection(
+ holder_beat: timedelta,
+ handed: int,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ # This process holds no copy of the holder's connection.
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ agent, client = await make_agent(session_factory, "worker")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, client, agent=agent)
+
+ other = str(uuid.uuid4())
+ now = datetime.now(UTC)
+ async with tenant_session(session_factory, room.tenant_id) as session:
+ session.add(
+ SwitchCoreProcess(id=other, started_at=now, beat_at=now - holder_beat)
+ )
+ await session.flush()
+ session.add(
+ AgentConnectionRecord(
+ id="conn-elsewhere",
+ agent_id=agent.id,
+ process_id=other,
+ incarnation=CONNECTION_INCARNATION_FLOOR + 1,
+ scope="all",
+ delivery_filter="addressed",
+ spawn_capable=False,
+ transport="sse",
+ )
+ )
+ role = RoomRole(room_id=room.id, name="reviewer", instructions="Review.")
+ session.add(role)
+ await session.flush()
+ session.add(
+ RoleLease(
+ role_id=role.id,
+ room_id=room.id,
+ agent_id=agent.id,
+ transport_session_id="conn-elsewhere",
+ # The holder renews nothing itself; only its connection
+ # keeps the seat.
+ last_seen_at=now - timedelta(minutes=5),
+ )
+ )
+ await session.commit()
+
+ await process.say(room, "@reviewer please look at this", sender=alice)
+ await process.idle()
+
+ assert len(await own_events(session_factory, agent.id)) == handed
diff --git a/core/tests/switch_core/delivery/test_room_join_events.py b/core/tests/switch_core/delivery/test_room_join_events.py
new file mode 100644
index 000000000..e4b888e89
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_room_join_events.py
@@ -0,0 +1,260 @@
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import IntegrityError
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.bridges.agent.protocol.types import AgentEvent, RoomJoinPayload
+from switch_core.db.models import TENANT_ZERO_ID, AgentEventRow, Message
+from switch_core.db.session_scope import tenant_session
+from switch_core.delivery.recipient import AgentRecipient
+from switch_core.transport import InboundMembership
+from tests.routing import (
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+)
+
+AGENT = "agent-1"
+
+
+class _FakeClientNames:
+ """Resolves a matrix id to the Switch client that owns it, or nothing."""
+
+ def __init__(self, names: dict[str, str]) -> None:
+ self._names = names
+
+ async def get_by_transport_user_id(
+ self, _session: object, transport_user_id: str
+ ) -> object | None:
+ name = self._names.get(transport_user_id)
+ return SimpleNamespace(display_name=name) if name else None
+
+
+def _client(
+ *,
+ receives: bool = True,
+ known: dict[str, str] | None = None,
+) -> SimpleNamespace:
+ """A minimal fake `self` for the unbound `AgentRecipient.decide_arrival`.
+
+ `receives` is the agent's per-room join-event setting, as the router read
+ it with the agent's membership.
+ """
+ stub = SimpleNamespace(
+ agent=SimpleNamespace(id=AGENT),
+ tenant_id=TENANT_ZERO_ID,
+ _member=SimpleNamespace(listening=receives),
+ _meta=SimpleNamespace(
+ room_id="room-1",
+ name="Room",
+ bridge_id="bridge-1",
+ channel_type="channel_public",
+ ),
+ _clients=_FakeClientNames(
+ {"@alice:switch.local": "Alice"} if known is None else known
+ ),
+ )
+ # Bound, so the handler resolves the name through the real implementation.
+ stub._member_name = lambda session, event: AgentRecipient._member_name(
+ stub, # type: ignore[arg-type]
+ session,
+ event,
+ )
+ return stub
+
+
+def _member_event(
+ *,
+ membership: str = "join",
+ prev_membership: str | None = "leave",
+ state_key: str = "@alice:switch.local",
+ displayname: str | None = "Alice",
+ timestamp: int = 123,
+) -> InboundMembership:
+ content: dict[str, object] = {}
+ if displayname is not None:
+ content["displayname"] = displayname
+ return InboundMembership(
+ room_id="!matrix:switch.local",
+ event_id="$member",
+ sender=state_key,
+ timestamp=timestamp,
+ content=content,
+ state_key=state_key,
+ membership=membership,
+ prev_membership=prev_membership,
+ display_name=displayname,
+ )
+
+
+async def _enqueued(
+ client: SimpleNamespace, event: InboundMembership
+) -> list[tuple[str, AgentEvent]]:
+ """What the router would store for the agent as its own events, as
+ (room, event). The arrival is passed as the row it was written as."""
+ delivery = await AgentRecipient.decide_arrival(client, None, event) # type: ignore[arg-type]
+ if delivery.own_event is None:
+ return []
+ own = delivery.own_event(event)
+ return [(own.event.room_id, own.event)]
+
+
+class TestRoomJoinEnqueue:
+ async def test_join_enqueues_room_join_event(self) -> None:
+ enqueued = await _enqueued(_client(), _member_event())
+
+ assert len(enqueued) == 1
+ room_id, event = enqueued[0]
+ assert room_id == "room-1"
+ assert event.type == "room_join"
+ assert event.bridge_id == "bridge-1"
+ assert event.channel_type == "channel_public"
+ assert isinstance(event.payload, RoomJoinPayload)
+ assert event.payload.member == "@alice:switch.local"
+ assert event.payload.member_name == "Alice"
+ assert event.payload.timestamp == 123
+ # The agent is opted in → listening is True.
+ assert event.payload.listening is True
+
+ async def test_listening_reflects_store_setting(self) -> None:
+ # The per-room, per-agent receives_join_events setting (off here)
+ # decides whether the arrival is one of the agent's own events. Off, it
+ # is the room's conversation: nothing is stored, and a reader that
+ # wants the conversation reads the arrival from the room's log, with
+ # `listening` False.
+ assert await _enqueued(_client(receives=False), _member_event()) == []
+
+ async def test_the_member_is_named_the_way_switch_knows_them(self) -> None:
+ """A profile carries the source platform's spelling of a name; the log
+ records the Switch one. Delivering the profile name would make live
+ events and history disagree about the same person."""
+ enqueued = await _enqueued(
+ _client(known={"@alice:switch.local": "alice"}),
+ _member_event(displayname="Alice \U0001f495"),
+ )
+
+ _, event = enqueued[0]
+ assert event.payload.member_name == "alice"
+
+ async def test_a_member_no_switch_client_owns_falls_back_to_the_event(
+ self,
+ ) -> None:
+ enqueued = await _enqueued(
+ _client(known={}),
+ _member_event(displayname=None, state_key="@bob:switch.local"),
+ )
+
+ assert len(enqueued) == 1
+ _, event = enqueued[0]
+ assert event.payload.member_name == "bob"
+
+ async def test_leave_does_not_enqueue(self) -> None:
+ assert (
+ await _enqueued(
+ _client(), _member_event(membership="leave", prev_membership="join")
+ )
+ == []
+ )
+
+ async def test_profile_update_does_not_enqueue(self) -> None:
+ # membership-preserving update (e.g. display-name change): join -> join.
+ assert (
+ await _enqueued(
+ _client(), _member_event(membership="join", prev_membership="join")
+ )
+ == []
+ )
+
+ async def test_no_room_meta_does_not_enqueue(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """The router reads the room in the transaction that writes the row, so
+ an arrival in a room that cannot be read is never decided for anyone:
+ the row cannot be written either, and nothing is kept."""
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ agent, client = await make_agent(session_factory, "listener")
+ await add_member(session_factory, room, client, agent=agent, listening=True)
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ arrival = Message(
+ room_id="no-such-room",
+ transport_event_id="$arrival",
+ sender_id=alice.transport_user_id,
+ sender_client_id=alice.id,
+ sender_name=alice.display_name,
+ event_type="m.room.member",
+ content={"membership": "join"},
+ )
+ with pytest.raises(IntegrityError):
+ await process.router.store(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ room_id="no-such-room",
+ transport_room_id="!gone:test",
+ message=arrival,
+ attachments=[],
+ )
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ assert (await session.scalars(select(AgentEventRow))).all() == []
+
+
+class TestArrivalsThroughTheRouter:
+ async def test_an_arrival_reaches_the_agents_listening_for_it(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """End to end: a person joining is one own event for the agent that
+ listens for joins here, stamped with the row's time, and none for the
+ agent that does not."""
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ listener, listener_client = await make_agent(session_factory, "listener")
+ deaf, deaf_client = await make_agent(session_factory, "deaf")
+ await add_member(
+ session_factory, room, listener_client, agent=listener, listening=True
+ )
+ await add_member(session_factory, room, deaf_client, agent=deaf)
+ alice = await make_person(session_factory, "alice")
+
+ await process.provisioning.invite_to_room(
+ room.transport_room_id, alice.transport_user_id
+ )
+ await process.idle()
+
+ [row] = await own_events(session_factory, listener.id)
+ assert row.type == "room_join"
+ payload = AgentEvent.model_validate(row.payload).payload
+ assert isinstance(payload, RoomJoinPayload)
+ assert payload.member == alice.transport_user_id
+ assert payload.member_name == "alice"
+ assert payload.listening is True
+ assert payload.timestamp > 0
+ assert await own_events(session_factory, deaf.id) == []
+
+
+class TestRoomJoinPayloadRegistry:
+ def test_agent_event_parses_room_join_payload(self) -> None:
+ event = AgentEvent.model_validate(
+ {
+ "type": "room_join",
+ "room_id": "room-1",
+ "bridge_id": None,
+ "channel_type": "channel_public",
+ "payload": {
+ "member": "@alice:switch.local",
+ "member_name": "Alice",
+ "timestamp": 123,
+ "listening": True,
+ },
+ }
+ )
+ assert isinstance(event.payload, RoomJoinPayload)
+ assert event.payload.member_name == "Alice"
+ assert event.payload.listening is True
diff --git a/core/tests/switch_core/delivery/test_router.py b/core/tests/switch_core/delivery/test_router.py
new file mode 100644
index 000000000..057ae385c
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_router.py
@@ -0,0 +1,648 @@
+"""The router decides each room row once, as it is written.
+
+What these pin, each the reason the router exists:
+
+- **Once, on the process that wrote the row.** Two processes share one
+ database; a row written through one is decided by it alone: one own event
+ per agent, one offline notice, one refusal, one command answer, one
+ greeting. The other process does no work for it.
+- **Atomic.** A row and the own events decided for it commit together: a
+ transaction that rolls back leaves neither.
+- **One agent's failure is that agent's.** Storing one agent's event runs in a
+ savepoint of its own, so an error there costs that agent the event and
+ nothing more: the row is written and the other agents get theirs.
+- **No deadlock.** Agents' rows in `agent_event_heads` are locked in the order
+ of their ids, whatever order a message names them in.
+- **A cost that does not grow with the room.** Deciding a message reads the
+ same few things whether the room holds one agent or many; what grows is one
+ write per agent the message is for.
+- **A post given up on is decided once** by whichever process's sweep claims
+ it first.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+from typing import Any
+
+import pytest
+from sqlalchemy import select, text, update
+from sqlalchemy.exc import DBAPIError
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.bridges.agent.protocol.agent_connections import (
+ PROTOCOL_VERSION,
+ ClientDeclaration,
+)
+from switch_core.bridges.agent.protocol.controller_presence import Binding
+from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
+from switch_core.bridges.agent.protocol.positions import OWN_EVENTS
+from switch_core.db.models import (
+ TENANT_ZERO_ID,
+ AgentEventHead,
+ AgentEventRow,
+ Client,
+ Message,
+ PendingAttachmentGroup,
+ Room,
+)
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.delivery import router as router_module
+from switch_core.delivery.addressing import ADDRESSING_DENIED_MESSAGE
+from switch_core.delivery.router import MessageRouter
+from switch_core.observability.metrics import MetricsRegistry, install, uninstall
+from switch_core.transport.postgres import new_event_id
+from tests.routing import (
+ Process,
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+ said_by,
+)
+from tests.switch_core.statement_counts import StatementCounts
+
+SessionFactory = async_sessionmaker[AsyncSession]
+
+_RESTRICTED = {"rules": [{"agents": ["nobody"], "users": []}]}
+
+
+def _go_live(process: Process, agent_id: str) -> None:
+ """A connection for the agent in this process, so it reads as online."""
+ process.connections.open(
+ agent_id=agent_id,
+ connection_id=f"conn-{agent_id}",
+ scope="all",
+ delivery_filter="addressed",
+ spawn_capable=False,
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
+ expected_generation=None,
+ )
+
+
+def _never_routes(monkeypatch: pytest.MonkeyPatch, router: MessageRouter) -> list[str]:
+ """Record every row `router` is asked to route or flush, by method."""
+ asked: list[str] = []
+ for name in ("store", "remove", "flush_due_groups", "after_commit"):
+ real = getattr(router, name)
+
+ def recording(
+ *args: Any, _real: Any = real, _name: str = name, **kw: Any
+ ) -> Any:
+ asked.append(_name)
+ return _real(*args, **kw)
+
+ monkeypatch.setattr(router, name, recording)
+ return asked
+
+
+async def _messages(session_factory: SessionFactory, room: Room) -> list[Message]:
+ async with tenant_session(session_factory, room.tenant_id) as session:
+ return list(
+ (
+ await session.scalars(
+ select(Message)
+ .where(Message.room_id == room.id)
+ .order_by(Message.seq)
+ )
+ ).all()
+ )
+
+
+class TestTwoProcesses:
+ async def test_a_message_is_routed_once_by_the_process_that_wrote_it(
+ self, monkeypatch: pytest.MonkeyPatch, session_factory: SessionFactory
+ ) -> None:
+ here = process_on(session_factory)
+ there = process_on(session_factory)
+ asked_there = _never_routes(monkeypatch, there.router)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ offline, offline_client = await make_agent(
+ session_factory, "offline", connection_model="session_passive"
+ )
+ strict, strict_client = await make_agent(
+ session_factory, "strict", addressing_policy=_RESTRICTED
+ )
+ for agent, client in [(offline, offline_client), (strict, strict_client)]:
+ await add_member(session_factory, room, client, agent=agent)
+ await add_member(session_factory, room, alice)
+
+ await here.say(room, "@offline @strict can you both look", sender=alice)
+ await here.idle()
+ await there.idle()
+
+ # One own event for the agent the message reached, none for the one
+ # whose policy refused the sender.
+ assert len(await own_events(session_factory, offline.id)) == 1
+ assert await own_events(session_factory, strict.id) == []
+ # One offline notice, and one refusal.
+ notices = await said_by(session_factory, room, offline_client)
+ refusals = await said_by(session_factory, room, strict_client)
+ assert len(notices) == 1, [n.body for n in notices]
+ assert len(refusals) == 1, [r.body for r in refusals]
+ assert ADDRESSING_DENIED_MESSAGE in (refusals[0].body or "")
+ # The other process was asked nothing.
+ assert asked_there == []
+
+ async def test_a_command_is_answered_once(
+ self, monkeypatch: pytest.MonkeyPatch, session_factory: SessionFactory
+ ) -> None:
+ here = process_on(session_factory)
+ there = process_on(session_factory)
+ asked_there = _never_routes(monkeypatch, there.router)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ greeter, greeter_client = await make_agent(session_factory, "greeter")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, greeter_client, agent=greeter)
+
+ await here.command(room, "agents-greet", "", sender=alice)
+ await here.idle()
+ await there.idle()
+
+ answers = await said_by(session_factory, room, greeter_client)
+ assert len(answers) == 1, [a.body for a in answers]
+ assert "@greeter" in (answers[0].body or "")
+ assert asked_there == []
+
+ async def test_an_arrival_is_greeted_once(
+ self, monkeypatch: pytest.MonkeyPatch, session_factory: SessionFactory
+ ) -> None:
+ here = process_on(session_factory)
+ there = process_on(session_factory)
+ asked_there = _never_routes(monkeypatch, there.router)
+ room = await make_room(session_factory)
+ newcomer, newcomer_client = await make_agent(session_factory, "newcomer")
+
+ await here.provisioning.invite_to_room(
+ room.transport_room_id, newcomer_client.transport_user_id
+ )
+ # A second invitation finds the membership and writes nothing.
+ await here.provisioning.invite_to_room(
+ room.transport_room_id, newcomer_client.transport_user_id
+ )
+ await here.idle()
+ await there.idle()
+
+ greetings = await said_by(session_factory, room, newcomer_client)
+ assert len(greetings) == 1
+ assert "@newcomer" in (greetings[0].body or "")
+ assert asked_there == []
+
+ async def test_a_removal_written_by_one_process_is_let_go_of_in_the_other(
+ self, session_factory: SessionFactory
+ ) -> None:
+ """The removal's own events go once, with the membership; what the
+ other process holds in memory for the agent (its connections' claim on
+ the room) it lets go of when it hears the membership changed."""
+ here = process_on(session_factory)
+ there = process_on(session_factory)
+ room = await make_room(session_factory)
+ agent, client = await make_agent(session_factory, "leaver")
+ await add_member(session_factory, room, client, agent=agent)
+ conn = there.connections.open(
+ agent_id=agent.id,
+ connection_id="held-there",
+ scope="single",
+ delivery_filter="all",
+ spawn_capable=False,
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
+ expected_generation=None,
+ )
+ there.connections.claim_room(conn, room.id)
+
+ await here.provisioning.kick_user(
+ room.transport_room_id, client.transport_user_id
+ )
+ assert conn.rooms == {room.id}, "the other process is told by a signal"
+ await there.router.membership_changed((room.tenant_id, client.id, room.id))
+
+ assert conn.rooms == set()
+
+ async def test_a_membership_signal_for_a_room_still_held_changes_nothing(
+ self, session_factory: SessionFactory
+ ) -> None:
+ there = process_on(session_factory)
+ room = await make_room(session_factory)
+ agent, client = await make_agent(session_factory, "stayer")
+ await add_member(session_factory, room, client, agent=agent)
+ conn = there.connections.open(
+ agent_id=agent.id,
+ connection_id="held",
+ scope="single",
+ delivery_filter="all",
+ spawn_capable=False,
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
+ expected_generation=None,
+ )
+ there.connections.claim_room(conn, room.id)
+
+ await there.router.membership_changed((room.tenant_id, client.id, room.id))
+
+ assert conn.rooms == {room.id}
+
+ async def test_a_controller_streaming_from_the_other_process_hears_of_a_join(
+ self, session_factory: SessionFactory
+ ) -> None:
+ """A managed agent's controller streams from one process; the agent
+ is added to a room through another. The controller's room set, kept
+ in the memory of the process it streams from, grows when that process
+ hears the membership changed, and again on a re-check if it never
+ did."""
+ here = process_on(session_factory)
+ there = process_on(session_factory)
+ first = await make_room(session_factory, "first")
+ second = await make_room(session_factory, "second")
+ third = await make_room(session_factory, "third")
+ agent, client = await make_agent(session_factory, "managed")
+ await add_member(session_factory, first, client, agent=agent)
+ presence = there.connections.controllers
+ presence.bind(
+ Binding(
+ agent_id=agent.id,
+ controller_id="controller-1",
+ tenant_id=TENANT_ZERO_ID,
+ controller_name="machine",
+ running=True,
+ )
+ )
+ presence.set_rooms(agent.id, {first.id})
+
+ await here.provisioning.invite_to_room(
+ second.transport_room_id, client.transport_user_id
+ )
+ await there.router.membership_changed((TENANT_ZERO_ID, client.id, second.id))
+ assert presence.rooms(agent.id) == {first.id, second.id}
+
+ await add_member(session_factory, third, client, agent=agent)
+ await there.router.recheck_agent_rooms(TENANT_ZERO_ID)
+ assert presence.rooms(agent.id) == {first.id, second.id, third.id}
+
+ async def test_a_missed_signal_is_made_up_by_the_recheck(
+ self, session_factory: SessionFactory
+ ) -> None:
+ """A process that never heard a removal finds it on its next
+ re-check: the claim on the room left goes, the one on the room kept
+ stays."""
+ there = process_on(session_factory)
+ left = await make_room(session_factory, "left")
+ kept = await make_room(session_factory, "kept")
+ agent, client = await make_agent(session_factory, "agent")
+ for room in (left, kept):
+ await add_member(session_factory, room, client, agent=agent)
+ held = []
+ for room in (left, kept):
+ conn = there.connections.open(
+ agent_id=agent.id,
+ connection_id=f"held-{room.name}",
+ scope="single",
+ delivery_filter="all",
+ spawn_capable=False,
+ position={OWN_EVENTS: 0},
+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),
+ expected_generation=None,
+ )
+ there.connections.claim_room(conn, room.id)
+ held.append(conn)
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ await RoomStore().remove_client(session, client.id, left.id)
+ await session.commit()
+
+ await there.router.recheck_agent_rooms(TENANT_ZERO_ID)
+
+ assert [conn.rooms for conn in held] == [set(), {kept.id}]
+
+ async def test_a_post_given_up_on_is_decided_once_by_two_sweeps(
+ self, monkeypatch: pytest.MonkeyPatch, session_factory: SessionFactory
+ ) -> None:
+ """Both processes sweep at the same moment; the post is claimed by one
+ of them and decided once."""
+ # Long enough that the writer's own timer does not fire in the test.
+ monkeypatch.setattr(router_module, "ATTACHMENT_GROUP_TIMEOUT_SECONDS", 60.0)
+ here = process_on(session_factory)
+ there = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ agent, client = await make_agent(session_factory, "agent-a")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, client, agent=agent)
+ group = "grp-two-sweeps"
+ await here.send_file(
+ room,
+ "one.png",
+ sender=alice,
+ caption="@agent-a two of three",
+ group={"id": group, "index": 0, "total": 3},
+ )
+ await here.send_file(
+ room, "two.png", sender=alice, group={"id": group, "index": 1, "total": 3}
+ )
+ here.router.close()
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ await session.execute(
+ update(PendingAttachmentGroup).values(
+ deadline=PendingAttachmentGroup.deadline
+ - text("interval '2 minutes'")
+ )
+ )
+ await session.commit()
+
+ decided = await asyncio.gather(
+ here.router.flush_due_groups(TENANT_ZERO_ID),
+ there.router.flush_due_groups(TENANT_ZERO_ID),
+ )
+ await here.idle()
+ await there.idle()
+
+ assert sorted(decided) == [0, 1]
+ [row] = await own_events(session_factory, agent.id)
+ assert "incomplete attachment group: 2 of 3" in row.payload["payload"]["body"]
+
+
+class TestAtomicity:
+ async def test_a_rolled_back_row_leaves_no_own_event(
+ self, session_factory: SessionFactory
+ ) -> None:
+ """What a process that stops before it commits leaves behind: nothing.
+ The row is not written, so it was never decided either."""
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ agent, client = await make_agent(
+ session_factory, "fixer", connection_model="session_passive"
+ )
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, client, agent=agent)
+
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ routed = await process.router.store(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ room_id=room.id,
+ transport_room_id=room.transport_room_id,
+ message=_message(room, alice, "@fixer are you there"),
+ attachments=[],
+ )
+ # Decided and written, inside the transaction.
+ assert (await session.scalars(select(AgentEventRow))).all() != []
+ await session.rollback()
+ await process.idle()
+
+ assert await _messages(session_factory, room) == []
+ assert await own_events(session_factory, agent.id) == []
+ # Nor was anything owed posted: `committed()` is for a commit.
+ assert await said_by(session_factory, room, client) == []
+ assert routed.seq > 0
+
+
+class TestFailuresStayWithTheAgent:
+ async def test_one_agents_failure_costs_that_agent_alone(
+ self, monkeypatch: pytest.MonkeyPatch, session_factory: SessionFactory
+ ) -> None:
+ """A database error while storing one agent's event is rolled back to
+ that agent's savepoint: the message is written, the other agent gets
+ its event, and the failure is counted."""
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ broken, broken_client = await make_agent(session_factory, "broken")
+ fine, fine_client = await make_agent(session_factory, "fine")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, broken_client, agent=broken)
+ await add_member(session_factory, room, fine_client, agent=fine)
+ _go_live(process, broken.id)
+ _go_live(process, fine.id)
+ claim = EventBuffer._claim
+
+ def failing_claim(self: EventBuffer, tenant_id: str, agent_id: str) -> Any:
+ if agent_id == broken.id:
+ return text("SELECT 1 / 0")
+ return claim(self, tenant_id, agent_id)
+
+ monkeypatch.setattr(EventBuffer, "_claim", failing_claim)
+ registry = MetricsRegistry()
+ install(registry)
+ try:
+ await process.say(room, "@broken @fine one for each", sender=alice)
+ await process.idle()
+ failures = next(
+ p
+ for p in registry.collect()
+ if p.name == "switch.messages.routing_failures"
+ )
+ finally:
+ uninstall()
+
+ assert [m.body for m in await _messages(session_factory, room)] == [
+ "@broken @fine one for each"
+ ]
+ assert await own_events(session_factory, broken.id) == []
+ assert len(await own_events(session_factory, fine.id)) == 1
+ assert failures.numbers[0].value == 1.0
+ assert dict(failures.numbers[0].attributes) == {"stage": "store"}
+
+
+class TestLockOrder:
+ async def test_agents_are_locked_in_the_order_of_their_ids(
+ self, postgres_schema: str, session_factory: SessionFactory
+ ) -> None:
+ """A message naming the later agent first still locks the earlier one
+ first. While another transaction holds the later agent's row, the
+ message's transaction holds the earlier agent's and waits; were the
+ order the message's own, two messages naming the same two agents the
+ other way round could each hold one row and wait for the other."""
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ one, one_client = await make_agent(session_factory, "one")
+ two, two_client = await make_agent(session_factory, "two")
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, one_client, agent=one)
+ await add_member(session_factory, room, two_client, agent=two)
+ _go_live(process, one.id)
+ _go_live(process, two.id)
+ first, second = sorted([one, two], key=lambda agent: agent.id)
+ # Both have a numbering row, so locking it is visible to others.
+ await process.say(room, f"@{first.name} @{second.name} warm up", sender=alice)
+ await process.idle()
+
+ engine = create_async_engine(postgres_schema, poolclass=NullPool)
+ try:
+ async with engine.connect() as holder:
+ await holder.execute(text("BEGIN"))
+ await holder.execute(
+ select(AgentEventHead.agent_id)
+ .where(AgentEventHead.agent_id == second.id)
+ .with_for_update()
+ )
+ writing = asyncio.create_task(
+ process.say(
+ room, f"@{second.name} then @{first.name}", sender=alice
+ )
+ )
+ await asyncio.sleep(0.5)
+ assert not writing.done(), "the write did not wait for the lock"
+
+ async with engine.connect() as prober:
+ await prober.execute(text("BEGIN"))
+ with pytest.raises(DBAPIError, match="could not obtain lock"):
+ await prober.execute(
+ select(AgentEventHead.agent_id)
+ .where(AgentEventHead.agent_id == first.id)
+ .with_for_update(nowait=True)
+ )
+ await prober.execute(text("ROLLBACK"))
+ await holder.execute(text("ROLLBACK"))
+ await writing
+ finally:
+ await engine.dispose()
+
+ assert len(await own_events(session_factory, first.id)) == 2
+ assert len(await own_events(session_factory, second.id)) == 2
+
+ async def test_two_rooms_naming_the_same_two_agents_both_go_through(
+ self, session_factory: SessionFactory
+ ) -> None:
+ """Many at once, each pair naming the agents in opposite orders from
+ two rooms: none waits on another for good, and every own event is
+ stored."""
+ process = process_on(session_factory)
+ rooms = [await make_room(session_factory, name) for name in ("left", "right")]
+ alice = await make_person(session_factory, "alice")
+ one, one_client = await make_agent(session_factory, "one")
+ two, two_client = await make_agent(session_factory, "two")
+ for room in rooms:
+ await add_member(session_factory, room, alice)
+ await add_member(session_factory, room, one_client, agent=one)
+ await add_member(session_factory, room, two_client, agent=two)
+ _go_live(process, one.id)
+ _go_live(process, two.id)
+ registry = MetricsRegistry()
+ install(registry)
+ try:
+ await asyncio.wait_for(
+ asyncio.gather(
+ *(
+ process.say(
+ rooms[n % 2],
+ "@one @two go" if n % 2 else "@two @one go",
+ sender=alice,
+ )
+ for n in range(20)
+ )
+ ),
+ timeout=30,
+ )
+ await process.idle()
+ failures = [
+ p
+ for p in registry.collect()
+ if p.name == "switch.messages.routing_failures"
+ ]
+ finally:
+ uninstall()
+
+ assert len(await own_events(session_factory, one.id)) == 20
+ assert len(await own_events(session_factory, two.id)) == 20
+ assert failures == []
+
+
+class TestCost:
+ @staticmethod
+ async def _room_of(
+ session_factory: SessionFactory, size: int
+ ) -> tuple[Process, Room, Client, list[str]]:
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ alice = await make_person(session_factory, "alice")
+ await add_member(session_factory, room, alice)
+ names = []
+ for n in range(size):
+ name = f"a{n}-{uuid.uuid4().hex[:4]}"
+ agent, client = await make_agent(session_factory, name)
+ await add_member(session_factory, room, client, agent=agent)
+ _go_live(process, agent.id)
+ names.append(name)
+ # The steady state: alice's transport knows the room already.
+ await process.say(room, "hello", sender=alice)
+ await process.idle()
+ return process, room, alice, names
+
+ @staticmethod
+ def _kinds(statements: list[str]) -> dict[str, int]:
+ kinds: dict[str, int] = {}
+ for statement in statements:
+ kind = statement.lstrip().split(None, 1)[0].upper()
+ kinds[kind] = kinds.get(kind, 0) + 1
+ return kinds
+
+ @pytest.mark.parametrize("body", ["just chatting", "@{0} can you look"])
+ async def test_reads_do_not_grow_with_the_room(
+ self,
+ body: str,
+ session_factory: SessionFactory,
+ statement_counts: StatementCounts,
+ ) -> None:
+ """The room, its agent members, and what the message asks of them
+ (role leases, budgets) are each read once for the whole room. Two
+ agents and twelve cost the same: with two, one is named and one is
+ not, so every read the room can need is already made."""
+ counted = []
+ for size in (2, 12):
+ process, room, alice, names = await self._room_of(session_factory, size)
+ with statement_counts.scope(f"{size} agents") as scope:
+ await process.say(room, body.format(names[0]), sender=alice)
+ await process.idle()
+ counted.append(self._kinds(scope.statements))
+
+ small, large = counted
+ assert small.get("SELECT") == large.get("SELECT"), (small, large)
+ assert small == large
+
+ async def test_writes_grow_with_the_agents_a_message_is_for(
+ self, session_factory: SessionFactory, statement_counts: StatementCounts
+ ) -> None:
+ """Naming every agent in the room: the reads are what they are for a
+ room of one, and each agent named costs its own event's writes."""
+ counted = []
+ for size in (1, 6):
+ process, room, alice, names = await self._room_of(session_factory, size)
+ body = " ".join(f"@{name}" for name in names) + " all of you"
+ with statement_counts.scope(f"{size} agents") as scope:
+ await process.say(room, body, sender=alice)
+ await process.idle()
+ counted.append(self._kinds(scope.statements))
+
+ small, large = counted
+ assert small.get("SELECT") == large.get("SELECT"), (small, large)
+ # One message, then per agent named: its number and its event, each
+ # in a savepoint of its own.
+ per_agent = {
+ kind: (large.get(kind, 0) - small.get(kind, 0)) // 5
+ for kind in ("INSERT", "SAVEPOINT", "RELEASE")
+ }
+ assert per_agent == {"INSERT": 2, "SAVEPOINT": 1, "RELEASE": 1}, (small, large)
+ assert large.get("INSERT", 0) == 1 + 2 * 6
+
+
+def _message(room: Room, sender: Client, body: str) -> Message:
+ return Message(
+ room_id=room.id,
+ transport_event_id=new_event_id(),
+ sender_id=sender.transport_user_id,
+ sender_client_id=sender.id,
+ sender_name=sender.display_name,
+ event_type="m.room.message",
+ msgtype="m.text",
+ body=body,
+ content={"body": body, "msgtype": "m.text"},
+ )
diff --git a/core/tests/switch_core/delivery/test_router_sessions.py b/core/tests/switch_core/delivery/test_router_sessions.py
new file mode 100644
index 000000000..a277fed2d
--- /dev/null
+++ b/core/tests/switch_core/delivery/test_router_sessions.py
@@ -0,0 +1,209 @@
+"""How many pool connections routing one room message costs.
+
+A room used to fan every message out to all of its agent clients at once,
+each taking a pool slot to decide it and another to store what it decided,
+so the cost of one message was multiplied by the size of the room before it
+ever reached the pool: a ten-agent room turned one Slack message into eleven
+checkouts, the burst that pushed the pool to its ceiling.
+
+The router decides in the writer's own session, so the contract is one
+connection for writing a message, whatever the room holds, and never two at
+once. The replies a message owes are posted once its connection is back in
+the pool; the read an offline notice needs takes a connection of its own and
+gives it back before the notice is posted, because posting is a write that
+can wait, and a slot held across it sits idle in a transaction.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import pytest
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import Agent, Client, Room
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.delivery.recipient import AgentRecipient
+from tests.routing import (
+ Process,
+ add_member,
+ make_agent,
+ make_person,
+ make_room,
+ own_events,
+ process_on,
+)
+from tests.switch_core.pool_checkouts import PoolCheckouts
+
+
+async def _seed(
+ session_factory: async_sessionmaker[AsyncSession],
+ *,
+ names: list[str],
+ addressing_policy: dict[str, Any] | None = None,
+) -> tuple[Process, Room, Client, dict[str, tuple[Agent, Client]]]:
+ process = process_on(session_factory)
+ room = await make_room(session_factory)
+ louisa = await make_person(session_factory, "louisa")
+ await add_member(session_factory, room, louisa)
+ agents = {}
+ for name in names:
+ agent, client = await make_agent(
+ session_factory,
+ name,
+ connection_model="session_addressable",
+ addressing_policy=addressing_policy,
+ )
+ await add_member(session_factory, room, client, agent=agent)
+ agents[name] = (agent, client)
+ # The steady state: louisa's transport has written here before, so it
+ # knows the room and needs no read of its own to find it.
+ await process.say(room, "hello", sender=louisa)
+ return process, room, louisa, agents
+
+
+def posted(
+ monkeypatch: pytest.MonkeyPatch, session_factory: async_sessionmaker[AsyncSession]
+) -> list[int]:
+ """How many pool connections were checked out as each reply was posted,
+ across the whole pool: nothing else is running in the test, so anything
+ held then is held by the work posting the reply."""
+ pool = session_factory.kw["bind"].sync_engine.pool
+ held: list[int] = []
+ real = AgentRecipient._post_auto_reply
+
+ async def recording(self: AgentRecipient, *args: Any, **kwargs: Any) -> None:
+ held.append(pool.checkedout())
+ await real(self, *args, **kwargs)
+
+ monkeypatch.setattr(AgentRecipient, "_post_auto_reply", recording)
+ return held
+
+
+class TestWritingAMessage:
+ async def test_chatter_with_no_at_costs_the_write_alone(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ ) -> None:
+ # The common case in a busy room: no `@`, so there is nothing an alias
+ # or a role lease could match. It addresses nobody, so it is the room's
+ # conversation, read from the room's log, and nothing is stored.
+ process, room, louisa, agents = await _seed(session_factory, names=["member"])
+
+ with pool_checkouts.scope("say") as scope:
+ await process.say(room, "just talking amongst ourselves", sender=louisa)
+
+ assert (scope.taken, scope.peak) == (1, 1)
+ assert await own_events(session_factory, agents["member"][0].id) == []
+
+ async def test_an_at_message_for_somebody_else_costs_the_write_alone(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ ) -> None:
+ # Was a lookup of its own per agent: the room alias and the role lease
+ # are read once, in the writer's session, for every agent at once.
+ process, room, louisa, _ = await _seed(
+ session_factory, names=["member", "other"]
+ )
+
+ with pool_checkouts.scope("say") as scope:
+ await process.say(room, "@other can you look at this", sender=louisa)
+
+ assert (scope.taken, scope.peak) == (1, 1)
+
+ async def test_being_addressed_and_offline_costs_the_write_alone(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ posted: list[int],
+ ) -> None:
+ # The expensive path: addressing policy, liveness, and the whole
+ # offline-reply derivation. The decision and the stored event ride the
+ # write; the offline notice is composed and posted after it.
+ process, room, louisa, agents = await _seed(session_factory, names=["member"])
+
+ with pool_checkouts.scope("say") as scope:
+ await process.say(room, "@member can you look at this", sender=louisa)
+ await process.idle()
+
+ assert (scope.taken, scope.peak) == (1, 1)
+ assert len(await own_events(session_factory, agents["member"][0].id)) == 1
+ assert len(posted) == 1
+
+ async def test_no_session_is_open_while_the_auto_reply_is_posted(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ posted: list[int],
+ ) -> None:
+ # Posting is an unbounded wait as far as the pool is concerned; doing
+ # it inside a transaction is how a slot ends up idle in it for seconds.
+ process, room, louisa, _ = await _seed(session_factory, names=["member"])
+
+ await process.say(room, "@member you there?", sender=louisa)
+ await process.idle()
+
+ assert posted == [0]
+
+ async def test_a_restricted_agent_refusing_a_sender_costs_the_write_alone(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ posted: list[int],
+ ) -> None:
+ process, room, louisa, agents = await _seed(
+ session_factory,
+ names=["member"],
+ addressing_policy={"rules": [{"agents": ["nobody"], "users": []}]},
+ )
+
+ with pool_checkouts.scope("say") as scope:
+ await process.say(room, "@member you there?", sender=louisa)
+ await process.idle()
+
+ # The refusal leaves the message as chatter, so nothing is stored.
+ assert (scope.taken, scope.peak) == (1, 1)
+ assert await own_events(session_factory, agents["member"][0].id) == []
+ assert posted == [0]
+
+ async def test_an_alias_tag_costs_the_write_alone(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ posted: list[int],
+ ) -> None:
+ # Reached only after the name misses, so this is the path where alias
+ # and role used to be separate checkouts.
+ process, room, louisa, agents = await _seed(session_factory, names=["member"])
+ async with session_factory() as session:
+ await RoomStore().set_alias(
+ session, room.id, agents["member"][0].id, "fixer"
+ )
+ await session.commit()
+
+ with pool_checkouts.scope("say") as scope:
+ await process.say(room, "@fixer please look", sender=louisa)
+ await process.idle()
+
+ assert (scope.taken, scope.peak) == (1, 1)
+ assert len(await own_events(session_factory, agents["member"][0].id)) == 1
+ assert len(posted) == 1
+
+ async def test_a_ten_agent_room_costs_the_write_alone(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ pool_checkouts: PoolCheckouts,
+ ) -> None:
+ # The burst, end to end. One message to a room of ten agents: one
+ # connection, where it used to be one per agent plus the addressed
+ # agent's own write.
+ names = [f"a{i}" for i in range(10)]
+ process, room, louisa, agents = await _seed(session_factory, names=names)
+
+ with pool_checkouts.scope("say") as scope:
+ await process.say(room, "@a3 can you take this", sender=louisa)
+
+ assert (scope.taken, scope.peak) == (1, 1)
+ assert len(await own_events(session_factory, agents["a3"][0].id)) == 1
diff --git a/core/tests/switch_core/clients/test_system_message_never_addresses.py b/core/tests/switch_core/delivery/test_system_message_never_addresses.py
similarity index 97%
rename from core/tests/switch_core/clients/test_system_message_never_addresses.py
rename to core/tests/switch_core/delivery/test_system_message_never_addresses.py
index 2315249a3..ee57b941e 100644
--- a/core/tests/switch_core/clients/test_system_message_never_addresses.py
+++ b/core/tests/switch_core/delivery/test_system_message_never_addresses.py
@@ -21,9 +21,9 @@ from types import SimpleNamespace
import pytest
from switch_core.clients.admin_messages import ADMIN_MARKER, admin_extra_content
-from switch_core.clients.agent_consumer import AgentConsumer
from switch_core.clients.room_meta import RoomMeta
from switch_core.delivery.addressing import AddressingResolver
+from switch_core.delivery.recipient import AgentRecipient
from switch_core.transport import InboundMessage
@@ -82,7 +82,7 @@ async def _addressed(
agent_user_id=resolver.transport_user_id,
room_id=meta.room_id,
channel_type=meta.channel_type,
- message=AgentConsumer._as_incoming(event),
+ message=AgentRecipient._as_incoming(event),
)
diff --git a/core/tests/switch_core/gateway/conftest.py b/core/tests/switch_core/gateway/conftest.py
index c5672b7eb..bcc5f0342 100644
--- a/core/tests/switch_core/gateway/conftest.py
+++ b/core/tests/switch_core/gateway/conftest.py
@@ -38,6 +38,7 @@ from switch_core.provisioning.postgres import PostgresProvisioning
from switch_core.room_service import RoomService
from switch_core.session_activity.listener import Change
from switch_core.transport.invites import InviteBus
+from tests.routing import router_on
from tests.switch_core.gateway.agent_route_harness import add_agent
_MESSAGE_STORE = MessageStore()
@@ -223,11 +224,12 @@ async def chats(
room_store = RoomStore()
lifecycle = _FakeLifecycle()
+ message_router = router_on(session_factory)
provisioning = PostgresProvisioning(
session_factory=session_factory,
room_store=room_store,
client_store=ClientStore(),
- message_store=_MESSAGE_STORE,
+ router=message_router,
invites=InviteBus(),
)
room_service = RoomService(
@@ -254,6 +256,7 @@ async def chats(
agent_store=AgentStore(),
user_store=UserStore(),
message_store=_MESSAGE_STORE,
+ router=message_router,
media_store=MediaStore(),
usage_store=UsageStore(),
id_server_name="switch.test",
diff --git a/core/tests/switch_core/gateway/test_chats_bridge_integration.py b/core/tests/switch_core/gateway/test_chats_bridge_integration.py
index cf5fa5487..7f4b42952 100644
--- a/core/tests/switch_core/gateway/test_chats_bridge_integration.py
+++ b/core/tests/switch_core/gateway/test_chats_bridge_integration.py
@@ -24,6 +24,8 @@ from switch_core.db.models import (
Message,
Room,
)
+from switch_core.db.stores.bridge_message_map_store import BridgeMessageMapStore
+from switch_core.db.stores.external_user_store import ExternalUserStore
from switch_core.events import CommandEvent
from switch_core.transport import InboundMessage as TransportMessage
from switch_core.transport import RoomRef
@@ -289,6 +291,7 @@ async def _fake_collab_core_full(
actor = SimpleNamespace(
send_message=send_message,
+ client_id=f"fake-client-{external_user_id}",
transport_user_id=f"@fake-{external_user_id}:switch.test",
)
human_actors[external_user_id] = actor
@@ -310,11 +313,17 @@ async def _fake_collab_core_full(
assert room is not None
transport_room_id = room.transport_room_id
- ns = SimpleNamespace(
+ ns = CollaborationCore.__new__(CollaborationCore)
+ vars(ns).update(
_bridge_id=bridge.id,
_bridge_type="fake",
_adapter=adapter,
_human_user_ids=set(),
+ _not_human_user_ids=set(),
+ _session_factory=chats.factory,
+ _bridge_tenant_id=bridge.tenant_id,
+ _external_user_store=ExternalUserStore(),
+ _bridge_message_map_store=BridgeMessageMapStore(),
_workspace_consumer_transport_user_id="@bridge-bot:switch.test",
_channel_to_room={"fake-chan-1": (room_id, transport_room_id)},
_channel_locks={},
@@ -334,17 +343,8 @@ async def _fake_collab_core_full(
ns._find_channel = lambda room_id=None, transport_room_id=None: (
"fake-chan-1" if transport_room_id == transport_room_id else None
)
- ns._outbound_thread_root_ref = CollaborationCore._outbound_thread_root_ref.__get__(
- ns
- )
- ns._relay_outbound_message = CollaborationCore._relay_outbound_message.__get__(ns)
- ns.handle_outbound_message = CollaborationCore.handle_outbound_message.__get__(ns)
- ns._relay_outbound_media = CollaborationCore._relay_outbound_media.__get__(ns)
- ns._counted_outbound = CollaborationCore._counted_outbound.__get__(ns)
- ns._handle_inbound_message = CollaborationCore._handle_inbound_message.__get__(ns)
- ns._download_media = CollaborationCore._download_media.__get__(ns)
-
- return ns # type: ignore[return-value]
+
+ return ns
@pytest.mark.asyncio
diff --git a/core/tests/switch_core/gateway/test_chats_routes.py b/core/tests/switch_core/gateway/test_chats_routes.py
index 2facd7a6e..3c78dc03e 100644
--- a/core/tests/switch_core/gateway/test_chats_routes.py
+++ b/core/tests/switch_core/gateway/test_chats_routes.py
@@ -23,6 +23,7 @@ from sqlalchemy import delete, func, select, update
from switch_core.addressing import owner_only_policy
from switch_core.db.models import (
Agent,
+ AgentEventRow,
AgentSessionActivityItem,
ApprovalRequest,
ChatOperation,
@@ -413,6 +414,19 @@ async def test_a_sent_message_reads_back_as_the_console_member(chats: _Harness)
assert body["hasMore"] is False
+async def test_a_sent_message_is_an_event_of_the_agent_in_the_chat(
+ chats: _Harness,
+) -> None:
+ agent = await chats.agent("helper", owner=chats.alice)
+ room_id = await chats.new_chat(chats.alice, agent, "c1")
+
+ sent = await chats.send(chats.alice, room_id, "s1", "hello")
+ assert sent.status_code == 200, sent.text
+ [message] = sent.json()["messages"]
+
+ assert (room_id, message["seq"]) in await _events_of(chats, agent)
+
+
async def test_a_plain_sent_message_carries_no_html(chats: _Harness) -> None:
agent = await chats.agent("helper", owner=chats.alice)
room_id = await chats.new_chat(chats.alice, agent, "c1")
@@ -1682,3 +1696,14 @@ async def test_the_activity_stream_forgets_revisions_past_its_cap(
await chats.activity.change(_turn_change(agent, "t1"))
assert [(await events.next())[1]["itemId"] for _ in range(2)] == ["turn", "tool-1"]
await events.stream.aclose() # type: ignore[attr-defined]
+
+
+async def _events_of(chats: _Harness, agent: Agent) -> list[tuple[str, int]]:
+ """The rows stored as the agent's own events, by room and place."""
+ async with chats.factory() as session:
+ rows = await session.execute(
+ select(AgentEventRow.room_id, AgentEventRow.room_seq).where(
+ AgentEventRow.agent_id == agent.id
+ )
+ )
+ return [(room_id, room_seq) for room_id, room_seq in rows.all()]
diff --git a/core/tests/switch_core/gateway/test_cloud_controller_machines.py b/core/tests/switch_core/gateway/test_cloud_controller_machines.py
index 8648826d2..387d8fe8d 100644
--- a/core/tests/switch_core/gateway/test_cloud_controller_machines.py
+++ b/core/tests/switch_core/gateway/test_cloud_controller_machines.py
@@ -52,7 +52,7 @@ from switch_core.gateway.dependencies import (
from switch_core.gateway.hosted_controller import router as controller_router
from switch_core.gateway.hosted_machines import router as machines_router
from switch_core.management.placement import placement_refusal
-from switch_core.management.process_lease import ProcessLeases
+from switch_core.process_lease import ProcessLeases
from switch_core.providers.hosted import HostedControllerSettings
from switch_core.tenant_context import tenant_scope
from tests.switch_core.hosted_machine_helpers import add_tenant
diff --git a/core/tests/switch_core/gateway/test_github_connections.py b/core/tests/switch_core/gateway/test_github_connections.py
index fa18169ec..175683015 100644
--- a/core/tests/switch_core/gateway/test_github_connections.py
+++ b/core/tests/switch_core/gateway/test_github_connections.py
@@ -10,14 +10,16 @@ from urllib.parse import parse_qs, urlsplit
import httpx
import pytest
from fastapi import FastAPI
-from sqlalchemy import select
+from sqlalchemy import select, update
from switch_core.connections.adapters.github import GitHubAdapter
from switch_core.connections.broker import ServiceBroker
from switch_core.connections.loader import CATALOG
+from switch_core.connections.sign_in_flows import SignInFlows
from switch_core.db.models import (
ServiceConnection,
ServiceTokenIssuance,
+ SignInFlow,
User,
require_tenant_id,
)
@@ -87,12 +89,24 @@ async def github_app(tmp_path, session_factory):
await session.commit()
identity = {"user": user}
+ app = _process(github, identity, session_factory)
+ async with httpx.AsyncClient(
+ transport=httpx.ASGITransport(app=app), base_url="https://switch.example.com"
+ ) as client:
+ yield client, github, identity, session_factory
+
+
+def _process(github, identity, session_factory) -> FastAPI:
+ """One switch-core process's gateway: its own app, broker and flow store,
+ on the shared database."""
+
async def sessions():
async with session_factory() as session:
yield session
app = FastAPI()
app.state.github_connections = github
+ app.state.sign_in_flows = SignInFlows(session_factory, TEST_KEYRING)
app.state.service_broker = ServiceBroker(
session_factory=session_factory,
keyring=TEST_KEYRING,
@@ -106,10 +120,21 @@ async def github_app(tmp_path, session_factory):
app.dependency_overrides[get_current_user] = lambda: identity["user"]
app.dependency_overrides[get_session] = sessions
app.dependency_overrides[get_config] = lambda: SimpleNamespace(keyring=TEST_KEYRING)
- async with httpx.AsyncClient(
- transport=httpx.ASGITransport(app=app), base_url="https://switch.example.com"
- ) as client:
- yield client, github, identity, session_factory
+ return app
+
+
+class _EachRequestToTheNextProcess(httpx.AsyncBaseTransport):
+ """A load balancer with no stickiness: each request goes to the next of
+ the processes, in turn."""
+
+ def __init__(self, *apps: FastAPI) -> None:
+ self._transports = [httpx.ASGITransport(app=app) for app in apps]
+ self._next = 0
+
+ async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
+ transport = self._transports[self._next % len(self._transports)]
+ self._next += 1
+ return await transport.handle_async_request(request)
BASE = "/gateway/provider-connections/github"
@@ -240,22 +265,84 @@ async def test_flow_is_bound_to_initiating_user_and_tenant(github_app):
assert (await client.get(BASE + f"/flows/{flow_id}")).status_code == 404
assert (await complete(client, flow_id)).status_code == 404
identity["user"] = SimpleNamespace(id="github-user")
+ # Another workspace does not see the flow at all.
with tenant_scope("other-tenant"):
- assert (await complete(client, flow_id)).status_code == 404
+ assert (await complete(client, flow_id)).status_code == 410
github.exchange.assert_not_called()
-async def test_cancel_expiry_and_restart_are_visible(github_app):
- client, github, _, _ = github_app
+async def test_cancel_and_expiry_are_visible(github_app):
+ client, github, _, factory = github_app
flow_id = await authorize(client)
assert (await client.delete(BASE + f"/flows/{flow_id}")).status_code == 204
assert (await client.get(BASE + f"/flows/{flow_id}")).status_code == 410
flow_id = await authorize(client)
- github.flows[flow_id].expires_at = 0
+ async with factory() as session:
+ await session.execute(
+ update(SignInFlow)
+ .where(SignInFlow.id == flow_id)
+ .values(expires_at=datetime.now(UTC) - timedelta(seconds=1))
+ )
+ await session.commit()
assert (await client.get(BASE + f"/flows/{flow_id}")).status_code == 410
+ assert (
+ await client.get(BASE + "/authorize", params={"state": flow_id})
+ ).status_code == 400
+
+
+async def test_a_flow_keeps_nothing_secret_in_the_clear(github_app):
+ client, _, _, factory = github_app
flow_id = await authorize(client)
- github.flows.clear()
- assert (await client.get(BASE + f"/flows/{flow_id}")).status_code == 410
+ await relay(client, flow_id)
+ assert (await complete(client, flow_id)).status_code == 204
+ async with factory() as session:
+ row = await session.scalar(select(SignInFlow).where(SignInFlow.id == flow_id))
+ assert row is not None
+ assert SECRET not in row.completion_secret_hash
+ assert "SYNTHETIC-REFRESH" not in (row.encrypted_result or "")
+ assert row.browser_nonce_hash is not None
+ assert row.account_label == "example-user"
+
+
+async def test_a_sign_in_started_on_one_process_completes_on_another(
+ github_app,
+):
+ """Every request of the sign-in lands on the other process from the one
+ before: start, the browser's three steps, completion, status and confirm."""
+ _, github, identity, factory = github_app
+ first = _process(github, identity, factory)
+ second = _process(github, identity, factory)
+ async with httpx.AsyncClient(
+ transport=_EachRequestToTheNextProcess(first, second),
+ base_url="https://switch.example.com",
+ ) as client:
+ flow_id = await authorize(client)
+ await relay(client, flow_id)
+ assert (await complete(client, flow_id)).status_code == 204
+ assert (await client.get(BASE + f"/flows/{flow_id}")).json() == {
+ "status": "ready",
+ "login": "example-user",
+ }
+ assert (await confirm(client, flow_id)).status_code == 200
+ assert (await client.get(BASE)).json()["login"] == "example-user"
+ github.exchange.assert_awaited_once()
+
+
+async def test_two_processes_racing_to_complete_exchange_the_code_once(github_app):
+ _, github, identity, factory = github_app
+ first = _process(github, identity, factory)
+ second = _process(github, identity, factory)
+ async with httpx.AsyncClient(
+ transport=_EachRequestToTheNextProcess(first, second),
+ base_url="https://switch.example.com",
+ ) as client:
+ flow_id = await authorize(client)
+ await relay(client, flow_id)
+ results = await asyncio.gather(
+ complete(client, flow_id), complete(client, flow_id)
+ )
+ assert sorted(r.status_code for r in results) == [204, 400]
+ github.exchange.assert_awaited_once()
async def test_refresh_saved_before_repository_failure(github_app):
diff --git a/core/tests/switch_core/gateway/test_google_workspace.py b/core/tests/switch_core/gateway/test_google_workspace.py
index 6c9c06262..12ba7f62a 100644
--- a/core/tests/switch_core/gateway/test_google_workspace.py
+++ b/core/tests/switch_core/gateway/test_google_workspace.py
@@ -24,6 +24,7 @@ from switch_core.connections.flows import ServiceFlows
from switch_core.connections.loader import CATALOG
from switch_core.connections.oauth_clients import LOOPBACK_CALLBACK_PATH
from switch_core.connections.registry import ClientRegistration, build_adapters
+from switch_core.connections.sign_in_flows import SignInFlows
from switch_core.db.models import ServiceConnection, User
from switch_core.db.stores.service_connection_store import ServiceConnectionStore
from switch_core.gateway.service_connections import (
@@ -112,7 +113,9 @@ def _world(
)
harness = build_harness(session_factory)
harness.gateway_app.state.service_broker = broker
- harness.gateway_app.state.service_flows = ServiceFlows(PUBLIC)
+ harness.gateway_app.state.service_flows = ServiceFlows(
+ PUBLIC, SignInFlows(harness.session_factory, TEST_KEYRING)
+ )
harness.gateway_app.include_router(service_connections_router)
harness.gateway_app.include_router(service_flows_router)
return World(harness, google, broker)
diff --git a/core/tests/switch_core/gateway/test_service_flows.py b/core/tests/switch_core/gateway/test_service_flows.py
index 4da2db84b..f907906c9 100644
--- a/core/tests/switch_core/gateway/test_service_flows.py
+++ b/core/tests/switch_core/gateway/test_service_flows.py
@@ -10,25 +10,25 @@ from __future__ import annotations
import json
import secrets
import shutil
-import time
-from datetime import timedelta
+from datetime import UTC, datetime, timedelta
from pathlib import Path
from urllib.parse import parse_qs, urlsplit
import httpx
-from sqlalchemy import select
+from sqlalchemy import select, update
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.connections.adapters.oauth import OAuthAdapter
from switch_core.connections.broker import ServiceBroker
-from switch_core.connections.flows import MAX_FLOWS, ServiceFlows
+from switch_core.connections.flows import ServiceFlows
from switch_core.connections.loader import CATALOG_ROOT, load_catalog
from switch_core.connections.oauth_clients import (
LOOPBACK_CALLBACK_PATH,
RegisteredClient,
redirect_uris,
)
-from switch_core.db.models import ServiceConnection, User
+from switch_core.connections.sign_in_flows import MAX_FLOWS, SignInFlows
+from switch_core.db.models import ServiceConnection, SignInFlow, User
from switch_core.db.stores.service_connection_store import ServiceConnectionStore
from switch_core.gateway.service_connections import (
router as service_connections_router,
@@ -112,7 +112,7 @@ def _world(
)
harness = build_harness(session_factory)
harness.gateway_app.state.service_broker = broker
- flows = ServiceFlows(public_url)
+ flows = ServiceFlows(public_url, SignInFlows(session_factory, TEST_KEYRING))
harness.gateway_app.state.service_flows = flows
harness.gateway_app.include_router(service_connections_router)
harness.gateway_app.include_router(service_flows_router)
@@ -165,6 +165,11 @@ def _listener_code(url: str, state: str) -> str:
return query["code"][0]
+async def _flow_ids(session_factory: async_sessionmaker[AsyncSession]) -> set[str]:
+ async with session_factory() as session:
+ return set((await session.execute(select(SignInFlow.id))).scalars())
+
+
async def _connection(
session_factory: async_sessionmaker[AsyncSession], user_id: str
) -> ServiceConnection | None:
@@ -235,7 +240,7 @@ class TestLoopback:
assert connection.granted_scopes == ["read:items", "write:items"]
secret = json.loads(TEST_KEYRING.decrypt(connection.encrypted_secret))
assert secret["refresh_token"] in world.vendor.live_refresh
- assert world.flows.flows == {}
+ assert await _flow_ids(session_factory) == set()
async def test_a_code_is_completed_once(self, session_factory, tmp_path) -> None:
world = _world(session_factory, tmp_path / "catalog")
@@ -287,7 +292,13 @@ class TestLoopback:
code = _listener_code(
world.vendor.authorize(started.json()["url"]), console.state
)
- world.flows.flows[console.state].expires_at = time.time() - 1
+ async with session_factory() as session:
+ await session.execute(
+ update(SignInFlow)
+ .where(SignInFlow.id == console.state)
+ .values(expires_at=datetime.now(UTC) - timedelta(seconds=1))
+ )
+ await session.commit()
assert (await console.complete(code)).status_code == 410
assert (await console.status()).status_code == 410
@@ -315,11 +326,16 @@ class TestLoopback:
users = [await add_member(session_factory, f"u{i}") for i in range(2)]
async with world.harness.client() as client:
assert (await Console(client, users[0]).start()).status_code == 200
- template = next(iter(world.flows.flows.values()))
- for i in range(MAX_FLOWS - 1):
- world.flows.flows[f"filler-{i}"] = type(template)(
- **{**template.__dict__, "user_id": f"someone-{i}"}
- )
+ async with session_factory() as session:
+ template = await session.scalar(select(SignInFlow))
+ assert template is not None
+ columns = {
+ column.key: getattr(template, column.key)
+ for column in SignInFlow.__table__.columns
+ }
+ for i in range(MAX_FLOWS - 1):
+ session.add(SignInFlow(**{**columns, "id": f"filler-{i}"}))
+ await session.commit()
busy = await Console(client, users[1]).start()
assert busy.status_code == 409
assert "Too many sign-ins" in busy.json()["detail"]
@@ -445,7 +461,7 @@ class TestLoopbackPorts:
refused = await Console(client, ada).start(port=43999)
assert refused.status_code == 409
assert "port 43123, 43124" in refused.json()["detail"]
- assert world.flows.flows == {}
+ assert await _flow_ids(session_factory) == set()
class TestCoreCallback:
diff --git a/core/tests/switch_core/gateway/test_tenant_api_routes.py b/core/tests/switch_core/gateway/test_tenant_api_routes.py
index 56cd6e0ce..c3bb10ada 100644
--- a/core/tests/switch_core/gateway/test_tenant_api_routes.py
+++ b/core/tests/switch_core/gateway/test_tenant_api_routes.py
@@ -34,12 +34,14 @@ from switch_core.clients.client_lifecycle_service import ClientLifecycleService
from switch_core.connections.adapters.github import GitHubAdapter
from switch_core.connections.broker import ServiceBroker
from switch_core.connections.loader import CATALOG
+from switch_core.connections.sign_in_flows import SignInFlows
from switch_core.db.models import (
Agent,
ApiKey,
Client,
Invitation,
ServiceTokenIssuance,
+ SignInFlow,
Tenant,
TenantMember,
UsageMetric,
@@ -64,6 +66,7 @@ from switch_core.gateway.invite_mail import (
)
from switch_core.gateway.tenants import router as tenants_router
from switch_core.keys import Keyring
+from tests.ownership import NeverOwned
_SECRET = "unit-test-jwt-key-unit-test-jwt-key-unit-test" # gitleaks:allow
_KEYRING = Keyring.parse("test:" + _SECRET, legacy_secret=None)
@@ -149,6 +152,7 @@ def _app(
app.dependency_overrides[gw_deps.get_client_lifecycle] = lambda: (
client_lifecycle or _FakeClientLifecycle(session_factory)
)
+ app.state.sign_in_flows = SignInFlows(session_factory, _KEYRING)
app.state.service_broker = ServiceBroker(
session_factory=session_factory,
keyring=_KEYRING,
@@ -446,6 +450,8 @@ class TestCreateTenant:
session_factory=session_factory,
config=SimpleNamespace(id_server_name="test"), # type: ignore[arg-type]
tenants_isolated=False,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
app = _app(session_factory, client_lifecycle=lifecycle, signup_mode="open")
@@ -1506,13 +1512,25 @@ class TestMemberRoutes:
await session.commit()
app = _app(session_factory)
- github = SimpleNamespace(
- flows={
- "removed": SimpleNamespace(tenant_id=TENANT_A, user_id=target_id),
- "other": SimpleNamespace(tenant_id=TENANT_A, user_id=owner_id),
- },
- revoke=AsyncMock(),
- )
+ for state, user_id, kind in (
+ ("removed-github", target_id, "github"),
+ ("removed-service", target_id, "service"),
+ ("other", owner_id, "github"),
+ ):
+ await app.state.sign_in_flows.start(
+ kind=kind,
+ service="github" if kind == "github" else "example",
+ tenant_id=TENANT_A,
+ user_id=user_id,
+ state=state,
+ mode="core",
+ port=43123,
+ redirect_uri="https://switch.example.com/callback",
+ owner_label="someone",
+ details={},
+ completion_secret="c" * 43,
+ )
+ github = SimpleNamespace(revoke=AsyncMock())
app.state.github_connections = github
app.state.service_broker = ServiceBroker(
session_factory=session_factory,
@@ -1528,7 +1546,10 @@ class TestMemberRoutes:
assert response.status_code == 200, response.text
github.revoke.assert_awaited_once_with("SYNTHETIC-USER-TOKEN")
- assert set(github.flows) == {"other"}
+ async with session_factory() as session:
+ assert set((await session.execute(select(SignInFlow.id))).scalars()) == {
+ "other"
+ }
async with session_factory() as session:
assert await session.get(TenantMember, (TENANT_A, target_id)) is None
issued = await session.scalar(
diff --git a/core/tests/switch_core/management/harness.py b/core/tests/switch_core/management/harness.py
index de215534c..2e125201b 100644
--- a/core/tests/switch_core/management/harness.py
+++ b/core/tests/switch_core/management/harness.py
@@ -19,7 +19,7 @@ from __future__ import annotations
import asyncio
import json
import uuid
-from collections.abc import AsyncIterator
+from collections.abc import AsyncIterator, Callable
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from pathlib import Path
@@ -59,14 +59,18 @@ from switch_core.db.stores.agent_store import AgentStore
from switch_core.db.stores.api_key_store import ApiKeyStore
from switch_core.db.stores.room_role_store import RoomRoleStore
from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
from switch_core.db.stores.user_store import UserStore
from switch_core.gateway import dependencies as gw_deps
from switch_core.gateway.auth import create_jwt
from switch_core.gateway.providers import router as providers_router
from switch_core.keys import Keyring
from switch_core.management import controller_routes
+from switch_core.management.dependencies import get_authenticator, get_management
from switch_core.management.errors import ManagementError
from switch_core.management.wiring import Management, build_management
+from switch_core.process_lease import ProcessLease
+from switch_core.signals import SignalBus
from tests.db_buffer import db_buffer_on
from tests.switch_core.connections.fake_vendor import FakeVendor
from tests.switch_core.connections.github_seed import github_broker
@@ -125,6 +129,9 @@ class _FakeClientLifecycle:
def start_client(self, client: Client) -> None:
return None
+ def announce(self, tenant_id: str) -> None:
+ return None
+
def get_by_agent_id(self, agent_id: str) -> None:
"""No client is ever started here."""
return None
@@ -165,6 +172,9 @@ class Harness:
# Mounted at `/gateway` on `app`, for a test to add routes to.
gateway_app: FastAPI
management: Management
+ # This process's signal bus, started only by a test that runs several
+ # processes against one database.
+ bus: SignalBus
protocol: AgentCore
cache: ApiKeyCache
controller_auth_cache: ControllerAuthCache
@@ -192,6 +202,10 @@ class Harness:
)
+def _no_listening_engine() -> Any:
+ raise AssertionError("the harness's signal bus is never started")
+
+
def build_harness(
session_factory: async_sessionmaker[AsyncSession],
*,
@@ -199,6 +213,7 @@ def build_harness(
server_url: str | None = SERVER_URL,
third_party_avatars: bool = True,
controller_release: ControllerRelease | None = None,
+ signal_engine: Callable[[], Any] = _no_listening_engine,
) -> Harness:
"""The controller-token cache is on, as it is by default in production,
so every management test runs through it."""
@@ -208,6 +223,8 @@ def build_harness(
ttl_seconds=controller_auth_ttl_seconds, max_entries=64
)
protocol = protocol_service(session_factory, cache)
+ process_id = str(uuid.uuid4())
+ signal_bus = SignalBus(session_factory, signal_engine, process_id=process_id)
management = build_management(
token_secret=TOKEN_SECRET,
status_interval_seconds=STATUS_INTERVAL,
@@ -218,7 +235,15 @@ def build_harness(
presence=protocol.connections.controllers,
auth_cache=controller_auth_cache,
clock=clock,
- process_id=str(uuid.uuid4()),
+ # Never started unless a test runs several processes: with one, every
+ # nudge is delivered locally and what it would publish goes nowhere.
+ signal_bus=signal_bus,
+ lease=ProcessLease(
+ process_id=process_id,
+ session_factory=session_factory,
+ processes=SwitchCoreProcessStore(),
+ key_ids=[key.id for key in TEST_KEYRING.keys],
+ ),
)
async def _session() -> AsyncIterator[AsyncSession]:
@@ -235,6 +260,13 @@ def build_harness(
management.install(
agent_bridge_app=agent_app, gateway_app=gateway_app, protocol=protocol
)
+ # `install` sets management's request dependencies in module state, one
+ # per process, and two harnesses in one test are two processes: each app
+ # is pointed at its own, or the first's routes record what they do as the
+ # second process.
+ for app in (agent_app, gateway_app):
+ app.dependency_overrides[get_management] = lambda: management.service
+ app.dependency_overrides[get_authenticator] = lambda: management.authenticator
agent_app.include_router(activity_router)
agent_app.include_router(api_router, prefix="/agents")
@@ -269,6 +301,7 @@ def build_harness(
app=agent_app,
gateway_app=gateway_app,
management=management,
+ bus=signal_bus,
protocol=protocol,
cache=cache,
controller_auth_cache=controller_auth_cache,
@@ -588,7 +621,7 @@ async def connect(
conn = presence.current_connection(controller.controller_id)
assert conn is not None
presence.attach_stream(conn)
- await harness.management.start()
+ await harness.management.lease.renew()
await harness.management.ledger.flush_all()
return opened
@@ -604,7 +637,7 @@ async def ensure_connected(
if conn is None or not conn.stream_attached:
await connect(client, controller)
return
- await harness.management.start()
+ await harness.management.lease.renew()
await harness.management.ledger.flush_all()
diff --git a/core/tests/switch_core/management/test_connection_state.py b/core/tests/switch_core/management/test_connection_state.py
index 1e379a656..f4c3452aa 100644
--- a/core/tests/switch_core/management/test_connection_state.py
+++ b/core/tests/switch_core/management/test_connection_state.py
@@ -28,9 +28,12 @@ from switch_core.bridges.agent.protocol.controller_presence import (
from switch_core.bridges.agent.protocol.liveness import HEARTBEAT_TTL_SECONDS
from switch_core.db.models import AgentController, SwitchCoreProcess
from switch_core.db.stores.agent_controller_store import AgentControllerStore
-from switch_core.db.stores.switch_core_process_store import ProcessLeaseRow
+from switch_core.db.stores.switch_core_process_store import (
+ ProcessLeaseRow,
+ SwitchCoreProcessStore,
+)
from switch_core.management.connection_ledger import ControllerConnectionLedger
-from switch_core.management.process_lease import LEASE_TTL, ProcessLeases
+from switch_core.process_lease import LEASE_TTL, ProcessLeases
from tests.switch_core.management.harness import (
EnrolledController,
Harness,
@@ -135,19 +138,23 @@ class TestLeases:
return ProcessLeases(read_at=self.now, leases=rows)
def test_a_fresh_lease_holds(self) -> None:
- leases = self._leases(p=ProcessLeaseRow(beat_at=self.now, stopped_at=None))
+ leases = self._leases(
+ p=ProcessLeaseRow(beat_at=self.now, stopped_at=None, key_ids=None)
+ )
assert leases.holds("p")
def test_a_lease_older_than_the_ttl_does_not(self) -> None:
beat_at = self.now - LEASE_TTL
- leases = self._leases(p=ProcessLeaseRow(beat_at=beat_at, stopped_at=None))
+ leases = self._leases(
+ p=ProcessLeaseRow(beat_at=beat_at, stopped_at=None, key_ids=None)
+ )
assert not leases.holds("p")
assert leases.ended("p") == (beat_at, "server_lost")
def test_a_stopped_lease_does_not(self) -> None:
stopped_at = self.now - timedelta(seconds=1)
leases = self._leases(
- p=ProcessLeaseRow(beat_at=self.now, stopped_at=stopped_at)
+ p=ProcessLeaseRow(beat_at=self.now, stopped_at=stopped_at, key_ids=None)
)
assert not leases.holds("p")
assert leases.ended("p") == (stopped_at, "server_shutdown")
@@ -178,7 +185,7 @@ class TestTheMachineState:
owner = await add_member(harness.session_factory, "ada")
async with harness.client() as client:
controller = await enroll_console(harness, client, owner)
- await harness.management.start()
+ await harness.management.lease.renew()
await open_connection(client, controller)
await _flush(harness)
listed = await _listed(client, controller)
@@ -246,6 +253,7 @@ class TestTheMachineState:
session_factory=harness.session_factory,
controllers=store,
clock=harness.clock,
+ announce=lambda tenant_id, controller_id: None,
)
presence.use_ledger(ledger)
owner = await add_member(harness.session_factory, "ada")
@@ -407,7 +415,7 @@ class TestTheProcessLease:
async with holder.client() as client, other.client() as elsewhere:
controller = await enroll_console(holder, client, owner)
await connect(client, controller)
- await other.management.start()
+ await other.management.lease.renew()
online = await _listed(elsewhere, controller)
await _age_lease(holder, LEASE_TTL.total_seconds() + 1)
await other.management.lease.renew()
@@ -447,7 +455,8 @@ class TestTheProcessLease:
async with holder.client() as client, other.client() as elsewhere:
controller = await enroll_console(holder, client, owner)
await connect(client, controller)
- await holder.management.stop()
+ await holder.management.flush()
+ await holder.management.lease.stop()
listed = await _listed(elsewhere, controller)
assert listed["state"] == "offline"
assert listed["connection"]["disconnect_reason"] == "server_shutdown"
@@ -491,6 +500,7 @@ class TestTheLedger:
session_factory=harness.session_factory,
controllers=AgentControllerStore(),
clock=harness.clock,
+ announce=lambda tenant_id, controller_id: None,
)
elsewhere.disconnected(earlier, "heartbeat_lapsed")
await elsewhere.flush_all()
@@ -509,6 +519,7 @@ class TestTheLedger:
session_factory=harness.session_factory,
controllers=store,
clock=harness.clock,
+ announce=lambda tenant_id, controller_id: None,
)
owner = await add_member(harness.session_factory, "ada")
async with harness.client() as client:
@@ -531,6 +542,7 @@ class TestTheLedger:
session_factory=harness.session_factory,
controllers=store,
clock=harness.clock,
+ announce=lambda tenant_id, controller_id: None,
)
owner = await add_member(harness.session_factory, "ada")
async with harness.client() as client:
@@ -545,3 +557,18 @@ class TestTheLedger:
row = await _row(harness, controller.controller_id)
assert row.disconnect_reason == "heartbeat_lapsed"
assert store.failures == 0
+
+
+async def test_a_released_lease_is_not_claimed_back_by_its_renewal(
+ harness: Harness,
+) -> None:
+ """A draining process releases its lease while it still runs, and its
+ renewal loop may tick once more before it is cancelled."""
+ lease = harness.management.lease
+ await lease.renew()
+ await lease.stop()
+ await lease.renew()
+
+ async with harness.session_factory() as session:
+ _, leases = await SwitchCoreProcessStore().read(session)
+ assert leases[lease.process_id].stopped_at is not None
diff --git a/core/tests/switch_core/management/test_controller_signals.py b/core/tests/switch_core/management/test_controller_signals.py
new file mode 100644
index 000000000..6dbe84c5a
--- /dev/null
+++ b/core/tests/switch_core/management/test_controller_signals.py
@@ -0,0 +1,402 @@
+"""Management's nudges reach the controller wherever it streams from."""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+from datetime import UTC, datetime
+from typing import TYPE_CHECKING
+
+import pytest
+from sqlalchemy.ext.asyncio import create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistry
+from switch_core.bridges.agent.protocol.controller_presence import Binding
+from switch_core.db.models import TENANT_ZERO_ID
+from switch_core.db.session_scope import tenant_session
+from switch_core.db.stores.agent_controller_operation_store import (
+ AgentControllerOperationStore,
+)
+from switch_core.db.stores.agent_controller_store import AgentControllerStore
+from switch_core.db.stores.agent_definition_store import AgentDefinitionStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.management.notifier import (
+ ASSIGNMENT_CHANGED,
+ CREDENTIAL_REVOKED,
+ OPERATION_PENDING,
+ ControllerNotifier,
+)
+from switch_core.management.signals import ControllerSignals
+from switch_core.signals import SignalBus
+from tests.switch_core.management.harness import (
+ Harness,
+ add_member,
+ build_harness,
+ cookies_for,
+ enroll_console,
+ open_connection,
+ open_stream,
+ place_agent,
+ provider,
+ report_status,
+ take,
+)
+
+if TYPE_CHECKING:
+ from collections.abc import AsyncIterator, Callable
+
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+DELIVERY_TIMEOUT = 10.0
+
+
+def _process(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession]
+) -> tuple[ControllerSignals, AgentConnectionRegistry, SignalBus]:
+ """One switch-core process's share of it: its own presence, notifier and bus."""
+ registry = AgentConnectionRegistry()
+ bus = SignalBus(
+ session_factory,
+ lambda: create_async_engine(postgres_url, poolclass=NullPool),
+ process_id=str(uuid.uuid4()),
+ )
+ signals = ControllerSignals(
+ notifier=ControllerNotifier(),
+ presence=registry.controllers,
+ bus=bus,
+ session_factory=session_factory,
+ controllers=AgentControllerStore(),
+ definitions=AgentDefinitionStore(),
+ operations=AgentControllerOperationStore(),
+ rooms=RoomStore(),
+ processes=SwitchCoreProcessStore(),
+ clock=lambda: datetime.now(UTC),
+ )
+ return signals, registry, bus
+
+
+async def two_processes(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession]
+) -> AsyncIterator[tuple[ControllerSignals, ControllerSignals]]:
+ first, _, first_bus = _process(postgres_url, session_factory)
+ second, _, second_bus = _process(postgres_url, session_factory)
+ for bus in (first_bus, second_bus):
+ await bus.start()
+ for bus in (first_bus, second_bus):
+ await asyncio.wait_for(bus.connected.wait(), DELIVERY_TIMEOUT)
+ yield first, second
+ await first_bus.stop()
+ await second_bus.stop()
+
+
+async def _woken(subscription: object) -> None:
+ await asyncio.wait_for(subscription.wake.wait(), DELIVERY_TIMEOUT) # type: ignore[attr-defined]
+
+
+class TestAcrossProcesses:
+ async def test_a_change_in_one_process_nudges_a_stream_in_another(
+ self, two_processes: tuple[ControllerSignals, ControllerSignals]
+ ) -> None:
+ handled_here, streaming_there = two_processes
+ subscription = streaming_there.notifier.subscribe("c1")
+
+ handled_here.assignment_changed(TENANT_ZERO_ID, "c1", 7)
+ handled_here.operation_pending(
+ "c1", operation_id="o1", kind="provider.recheck", agent_id=None
+ )
+ await _woken(subscription)
+ async with asyncio.timeout(DELIVERY_TIMEOUT):
+ frames = subscription.drain()
+ while len(frames) < 2:
+ await _woken(subscription)
+ frames += subscription.drain()
+
+ assert frames == [
+ (ASSIGNMENT_CHANGED, {"revision": 7}),
+ (
+ OPERATION_PENDING,
+ {"operation_id": "o1", "kind": "provider.recheck", "agent_id": None},
+ ),
+ ]
+
+ async def test_the_process_that_made_the_change_nudges_at_once(
+ self, two_processes: tuple[ControllerSignals, ControllerSignals]
+ ) -> None:
+ handled_here, _ = two_processes
+ subscription = handled_here.notifier.subscribe("c1")
+
+ handled_here.assignment_changed(TENANT_ZERO_ID, "c1", 2)
+
+ assert subscription.drain() == [(ASSIGNMENT_CHANGED, {"revision": 2})]
+
+ async def test_a_revocation_reaches_the_other_process(
+ self, two_processes: tuple[ControllerSignals, ControllerSignals]
+ ) -> None:
+ handled_here, streaming_there = two_processes
+ subscription = streaming_there.notifier.subscribe("c1")
+
+ handled_here.revoked("c1")
+ await _woken(subscription)
+
+ assert subscription.drain() == [(CREDENTIAL_REVOKED, {})]
+ assert streaming_there._presence.is_revoked("c1")
+
+
+def harness(session_factory: async_sessionmaker[AsyncSession]) -> Harness:
+ return build_harness(session_factory)
+
+
+class TestResync:
+ async def test_a_held_controller_is_renudged_from_the_database(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ opened = await open_connection(client, controller)
+ stream = await open_stream(harness, controller, opened)
+ (opening,) = await take(stream, 1)
+
+ await harness.management.service.signals.resync()
+ nudged = await take(stream, 1)
+ await stream.aclose()
+
+ assert nudged == [
+ (ASSIGNMENT_CHANGED, {"revision": opening[1]["assignment_revision"]})
+ ]
+
+ async def test_a_controller_revoked_meanwhile_is_closed(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ opened = await open_connection(client, controller)
+ stream = await open_stream(harness, controller, opened)
+ await take(stream, 1)
+ async with tenant_session(
+ harness.session_factory, TENANT_ZERO_ID
+ ) as session:
+ await AgentControllerStore().mark_revoked(
+ session, TENANT_ZERO_ID, controller.controller_id, datetime.now(UTC)
+ )
+ await session.commit()
+
+ await harness.management.service.signals.resync()
+ final = await take(stream, 1)
+ with pytest.raises(StopAsyncIteration):
+ await anext(stream)
+
+ assert final == [(CREDENTIAL_REVOKED, {})]
+
+
+async def processes(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession]
+) -> AsyncIterator[
+ tuple[ControllerSignals, tuple[ControllerSignals, AgentConnectionRegistry]]
+]:
+ """A process that announces changes, and another with its own bindings."""
+ announcing, _, announcing_bus = _process(postgres_url, session_factory)
+ other, registry, other_bus = _process(postgres_url, session_factory)
+ for bus in (announcing_bus, other_bus):
+ await bus.start()
+ for bus in (announcing_bus, other_bus):
+ await asyncio.wait_for(bus.connected.wait(), DELIVERY_TIMEOUT)
+ yield announcing, (other, registry)
+ await announcing_bus.stop()
+ await other_bus.stop()
+
+
+async def _eventually(check: Callable[[], bool]) -> None:
+ async with asyncio.timeout(DELIVERY_TIMEOUT):
+ while not check():
+ await asyncio.sleep(0.02)
+
+
+async def _revision(
+ session_factory: async_sessionmaker[AsyncSession], controller_id: str
+) -> int:
+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
+ controller = await AgentControllerStore().get(
+ session, TENANT_ZERO_ID, controller_id
+ )
+ assert controller is not None
+ return controller.assignment_revision
+
+
+class TestBindingsAcrossProcesses:
+ """A placement changes in the process that served the request. The
+ harness is that process; `processes` is a process announcing it and
+ another that has to bind the agent from the tables."""
+
+ async def test_a_placement_binds_the_agent_in_the_other_process(
+ self,
+ harness: Harness,
+ processes: tuple[
+ ControllerSignals, tuple[ControllerSignals, AgentConnectionRegistry]
+ ],
+ ) -> None:
+ announcing, (_, there) = processes
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+
+ announcing.assignment_changed(
+ TENANT_ZERO_ID,
+ controller.controller_id,
+ await _revision(harness.session_factory, controller.controller_id),
+ )
+ await _eventually(lambda: there.controllers.is_bound(agent_id))
+
+ binding = there.controllers.binding(agent_id)
+ assert binding is not None
+ assert (binding.controller_id, binding.controller_name, binding.running) == (
+ controller.controller_id,
+ "laptop",
+ True,
+ )
+
+ async def test_a_move_and_a_removal_follow_in_the_other_process(
+ self,
+ harness: Harness,
+ processes: tuple[
+ ControllerSignals, tuple[ControllerSignals, AgentConnectionRegistry]
+ ],
+ ) -> None:
+ announcing, (_, there) = processes
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ first = await enroll_console(harness, client, owner, "one")
+ second = await enroll_console(harness, client, owner, "two")
+ await report_status(client, second, 1, providers=[provider("claude")])
+ agent_id = await place_agent(client, first, name="reviewer")
+ announcing.assignment_changed(TENANT_ZERO_ID, first.controller_id, 1)
+ await _eventually(lambda: there.controllers.is_bound(agent_id))
+
+ moved = await client.patch(
+ f"/gateway/management/agents/{agent_id}",
+ json={"controller_id": second.controller_id},
+ cookies=cookies_for(owner),
+ )
+ assert moved.status_code == 200, moved.text
+ # Only the controller it left is announced: the agent is bound where
+ # it went all the same, whichever controller's signal comes first.
+ announcing.assignment_changed(TENANT_ZERO_ID, first.controller_id, 2)
+ await _eventually(
+ lambda: (
+ getattr(there.controllers.binding(agent_id), "controller_id", None)
+ == second.controller_id
+ )
+ )
+
+ await harness.management.agent_removed(TENANT_ZERO_ID, agent_id)
+ announcing.assignment_changed(TENANT_ZERO_ID, second.controller_id, 3)
+ await _eventually(lambda: not there.controllers.is_bound(agent_id))
+
+ async def test_a_rename_reaches_the_other_process(
+ self,
+ harness: Harness,
+ processes: tuple[
+ ControllerSignals, tuple[ControllerSignals, AgentConnectionRegistry]
+ ],
+ ) -> None:
+ announcing, (_, there) = processes
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+ announcing.assignment_changed(TENANT_ZERO_ID, controller.controller_id, 1)
+ await _eventually(lambda: there.controllers.is_bound(agent_id))
+
+ async with tenant_session(harness.session_factory, TENANT_ZERO_ID) as session:
+ await AgentControllerStore().update_details(
+ session,
+ TENANT_ZERO_ID,
+ controller.controller_id,
+ {"name": "workstation"},
+ )
+ await session.commit()
+ announcing.renamed(TENANT_ZERO_ID, controller.controller_id)
+
+ await _eventually(
+ lambda: (
+ getattr(there.controllers.binding(agent_id), "controller_name", None)
+ == "workstation"
+ )
+ )
+
+
+class TestRecheck:
+ async def test_a_missed_signal_is_caught_by_the_recheck(
+ self, harness: Harness, postgres_url: str
+ ) -> None:
+ # Never started: no signal reaches this process.
+ other, registry, _ = _process(postgres_url, harness.session_factory)
+ owner = await add_member(harness.session_factory, "ada")
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+
+ await other.recheck()
+ assert registry.controllers.is_bound(agent_id)
+
+ await harness.management.agent_removed(TENANT_ZERO_ID, agent_id)
+ await other.recheck()
+ assert not registry.controllers.is_bound(agent_id)
+
+ async def test_a_held_controller_is_nudged_only_with_what_it_was_not_told(
+ self, harness: Harness
+ ) -> None:
+ owner = await add_member(harness.session_factory, "ada")
+ signals = harness.management.service.signals
+ async with harness.client() as client:
+ controller = await enroll_console(harness, client, owner)
+ await open_connection(client, controller)
+ subscription = signals.notifier.subscribe(controller.controller_id)
+ opening = await _revision(harness.session_factory, controller.controller_id)
+
+ # This process never nudged it, so the first re-check does, once.
+ await signals.recheck()
+ first = subscription.drain()
+ await signals.recheck()
+ second = subscription.drain()
+ # A change whose signal never came is a newer revision.
+ async with tenant_session(harness.session_factory, TENANT_ZERO_ID) as session:
+ revision = await AgentControllerStore().bump_assignment_revision(
+ session, TENANT_ZERO_ID, controller.controller_id
+ )
+ await session.commit()
+ await signals.recheck()
+ missed = subscription.drain()
+
+ assert first == [(ASSIGNMENT_CHANGED, {"revision": opening})]
+ assert second == []
+ assert missed == [(ASSIGNMENT_CHANGED, {"revision": revision})]
+
+
+class TestReconcileKeepsNewerLocalChanges:
+ def test_a_binding_made_after_the_read_is_left_alone(self) -> None:
+ presence = AgentConnectionRegistry().controllers
+ since = presence.change_mark()
+ presence.bind(
+ Binding(
+ agent_id="a1",
+ controller_id="c1",
+ tenant_id=TENANT_ZERO_ID,
+ controller_name="laptop",
+ running=True,
+ )
+ )
+
+ # The read started before the bind, so it did not see it.
+ presence.reconcile([], agent_ids={"a1"}, since=since)
+
+ assert presence.is_bound("a1")
+ presence.reconcile([], agent_ids={"a1"}, since=presence.change_mark())
+ assert not presence.is_bound("a1")
diff --git a/core/tests/switch_core/management/test_controllers_across_processes.py b/core/tests/switch_core/management/test_controllers_across_processes.py
new file mode 100644
index 000000000..3b0a9d375
--- /dev/null
+++ b/core/tests/switch_core/management/test_controllers_across_processes.py
@@ -0,0 +1,680 @@
+"""Controller-backed agents, seen and served by every switch-core process.
+
+Two processes on one database, each with its own presence, its own management
+and a signal bus listening on the real `NOTIFY` channel. A controller's socket
+is held by one of them; the other must read its agents as live, keep the same
+bindings, let the socket attach after an open it did not handle, and close its
+own copy when the controller is taken over elsewhere.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+from collections.abc import AsyncIterator, Callable
+from datetime import timedelta
+from typing import Any
+
+import pytest
+from sqlalchemy import select, update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.bridges.agent.protocol.connection_rows import SharedConnections
+from switch_core.bridges.agent.protocol.presence import agents_present_in
+from switch_core.bridges.agent.protocol.types import AgentEvent, MessagePayload
+from switch_core.db.models import TENANT_ZERO_ID, AgentController, SwitchCoreProcess
+from switch_core.db.stores.agent_connection_store import AgentConnectionStore
+from switch_core.process_lease import LEASE_TTL
+from tests.db_buffer import say
+from tests.switch_core.management.harness import (
+ Harness,
+ add_member,
+ add_room,
+ attach,
+ beat,
+ build_harness,
+ connect,
+ cookies_for,
+ enroll_console,
+ open_connection,
+ open_stream,
+ place_agent,
+ take,
+)
+
+SessionFactory = async_sessionmaker[AsyncSession]
+
+WAIT_SECONDS = 10.0
+
+
+async def _eventually(check: Callable[[], bool]) -> None:
+ deadline = time.monotonic() + WAIT_SECONDS
+ while not check():
+ if time.monotonic() > deadline:
+ raise AssertionError("the other process never caught up")
+ await asyncio.sleep(0.02)
+
+
+async def _ask(harness: Harness, agent_id: str, room_id: str, body: str) -> int:
+ """An addressed message to the agent, queued for it. Returns its seq."""
+ row = await say(
+ harness.session_factory, room_id, body, sender="@ada:test", sender_name="Ada"
+ )
+ return await harness.protocol.event_buffer.enqueue(
+ TENANT_ZERO_ID,
+ agent_id,
+ room_id,
+ AgentEvent(
+ type="message",
+ room_id=room_id,
+ payload=MessagePayload(
+ addressed=True,
+ sender="@ada:test",
+ sender_name="Ada",
+ message_id=row.transport_event_id,
+ body=body,
+ timestamp=0,
+ ),
+ ),
+ room_seq=row.seq,
+ message_key=row.transport_event_id,
+ )
+
+
+async def _until(stream: Any, event: str) -> dict[str, Any]:
+ """The data of the next frame of this kind, skipping the others."""
+ while True:
+ ((name, data),) = await take(stream, 1)
+ if name == event:
+ return dict(data)
+
+
+async def processes(
+ postgres_url: str, session_factory: SessionFactory
+) -> AsyncIterator[tuple[Harness, Harness]]:
+ harnesses = [
+ build_harness(
+ session_factory,
+ signal_engine=lambda: create_async_engine(postgres_url, poolclass=NullPool),
+ )
+ for _ in range(2)
+ ]
+ shared: list[SharedConnections] = []
+ for harness in harnesses:
+ shared.append(
+ SharedConnections(
+ registry=harness.protocol.connections,
+ session_factory=session_factory,
+ store=AgentConnectionStore(),
+ bus=harness.bus,
+ lease=harness.management.lease,
+ )
+ )
+ await harness.management.lease.renew()
+ for harness in harnesses:
+ await harness.bus.start()
+ for harness in harnesses:
+ await asyncio.wait_for(harness.bus.connected.wait(), WAIT_SECONDS)
+ ledgers = [
+ asyncio.create_task(harness.management.run()) for harness in harnesses
+ ] + [asyncio.create_task(process.run()) for process in shared]
+ try:
+ yield harnesses[0], harnesses[1]
+ finally:
+ # The buses first, so no signal starts work after the processes
+ # stopped running; then the runs, which wait for the work they started.
+ for harness in harnesses:
+ await harness.bus.stop()
+ for task in ledgers:
+ task.cancel()
+ await asyncio.gather(*ledgers, return_exceptions=True)
+
+
+class TestBindings:
+ async def test_an_agent_placed_through_one_process_is_bound_in_the_other(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+
+ await _eventually(
+ lambda: there.protocol.connections.controllers.is_bound(agent_id)
+ )
+ binding = there.protocol.connections.controllers.binding(agent_id)
+ assert binding is not None
+ assert binding.controller_id == controller.controller_id
+
+ removed = await client.delete(
+ f"/gateway/management/agents/{agent_id}",
+ cookies=cookies_for(owner),
+ )
+ assert removed.status_code == 200, removed.text
+
+ await _eventually(
+ lambda: not there.protocol.connections.controllers.is_bound(agent_id)
+ )
+
+
+class TestPresence:
+ async def test_a_managed_agent_connected_to_one_process_reads_live_from_the_other(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+ room_id = await add_room(here.session_factory, agent_id)
+ await connect(client, controller)
+
+ presence = there.protocol.connections.controllers
+ await _eventually(lambda: presence.is_live(agent_id))
+ await _eventually(lambda: presence.live_in_room(agent_id, room_id))
+ assert agents_present_in([agent_id], room_id, there.protocol.connections) == {
+ agent_id
+ }
+ assert agent_id in there.protocol.connections.live_agent_ids()
+ # The process holding the socket is the one that reports it.
+ assert presence.live_connection_count() == 0
+ assert there.protocol.connections.live_agents_by_transport() == {}
+
+ async def test_an_agent_bound_to_a_controller_held_elsewhere_is_in_its_rooms(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ """The binding reaches the other process after the controller's
+ socket did, and the agent is present in its rooms there at once."""
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ presence = there.protocol.connections.controllers
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+ await _eventually(lambda: presence.is_live(agent_id))
+ room_id = await add_room(here.session_factory, agent_id)
+ for state in ("stopped", "running"):
+ changed = await client.patch(
+ f"/gateway/management/agents/{agent_id}",
+ json={"desired_state": state},
+ cookies=cookies_for(owner),
+ )
+ assert changed.status_code == 200, changed.text
+ await _eventually(
+ lambda state=state: (
+ presence.is_live(agent_id) == (state == "running")
+ )
+ )
+
+ await _eventually(lambda: presence.live_in_room(agent_id, room_id))
+
+ async def test_the_socket_going_is_seen_by_the_other(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+ await connect(client, controller)
+ presence = there.protocol.connections.controllers
+ await _eventually(lambda: presence.is_live(agent_id))
+
+ held = here.protocol.connections.controllers
+ conn = held.current_connection(controller.controller_id)
+ assert conn is not None
+ held.detach_stream(conn, conn.stream_token)
+
+ await _eventually(lambda: not presence.is_live(agent_id))
+
+ async def test_a_dead_holder_stops_counting_on_the_next_pass(
+ self, processes: tuple[Harness, Harness], session_factory: SessionFactory
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+ await connect(client, controller)
+ presence = there.protocol.connections.controllers
+ await _eventually(lambda: presence.is_live(agent_id))
+ async with session_factory() as session:
+ await session.execute(
+ update(SwitchCoreProcess)
+ .where(SwitchCoreProcess.id == here.management.lease.process_id)
+ .values(
+ beat_at=SwitchCoreProcess.beat_at - LEASE_TTL - timedelta(seconds=1)
+ )
+ )
+ await session.commit()
+
+ await there.management.service.signals.read_connections()
+
+ assert not presence.is_live(agent_id)
+
+ async def test_a_session_command_reaches_the_controller_on_the_other_process(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+ await connect(client, controller)
+ await _eventually(
+ lambda: there.protocol.connections.controllers.is_live(agent_id)
+ )
+ frame = {"command": "interrupt", "origin": {"roomId": "room"}}
+
+ assert there.protocol.connections.relay_session_command(agent_id, frame)
+
+ conn = here.protocol.connections.controllers.current_connection(
+ controller.controller_id
+ )
+ assert conn is not None
+ await _eventually(lambda: conn.session_commands == [(agent_id, frame)])
+
+
+class TestTheConnectionOnAnyProcess:
+ async def test_the_socket_attaches_through_the_process_that_did_not_open_it(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ opened = await open_connection(client, controller)
+
+ stream = await open_stream(there, controller, opened)
+ ((name, state),) = await take(stream, 1)
+ await stream.aclose() # type: ignore[attr-defined]
+
+ assert name == "connection_state"
+ assert state["connection_id"] == opened["connection_id"]
+ assert state["generation"] == opened["generation"]
+ adopted = there.protocol.connections.controllers.current_connection(
+ controller.controller_id
+ )
+ assert adopted is not None
+ assert adopted.id == opened["connection_id"]
+ # The process that opened it lets its copy go once told.
+ await _eventually(
+ lambda: (
+ here.protocol.connections.controllers.current_connection(
+ controller.controller_id
+ )
+ is None
+ )
+ )
+
+ async def test_a_socket_reattaching_through_the_other_process_resumes_there(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ """The socket dropped and came back through the other process: the
+ connection is taken up there, each agent resuming where the beats had
+ confirmed it, which were written when the socket went."""
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+ held = here.protocol.connections.controllers
+ conn = held.current_connection(controller.controller_id)
+ assert conn is not None
+ held.resume_from(conn, agent_id, 42)
+ held.detach_stream(conn, conn.stream_token)
+ await here.management.ledger.flush_all()
+
+ stream = await open_stream(
+ there,
+ controller,
+ {"connection_id": conn.id, "generation": conn.generation},
+ )
+ await take(stream, 1)
+ await stream.aclose() # type: ignore[attr-defined]
+
+ adopted = there.protocol.connections.controllers.current_connection(
+ controller.controller_id
+ )
+ assert adopted is not None
+ assert adopted.id == conn.id
+ assert adopted.resume_cursors[agent_id] == 42
+
+ async def test_a_reattach_after_its_holder_died_opens_again_from_its_cursors(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ """The process holding the socket dies having written nothing since
+ the open, so the row's cursors are the open's: the agent was placed
+ after it and its first event confirmed only by a beat there. Taking
+ the connection up elsewhere would resume past what came next; the
+ controller is told to open again, and resumes from its own cursor."""
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ agent_id = await place_agent(client, controller, name="reviewer")
+ room_id = await add_room(here.session_factory, agent_id)
+ conn = here.protocol.connections.controllers.current_connection(
+ controller.controller_id
+ )
+ assert conn is not None
+ opened = {"connection_id": conn.id, "generation": conn.generation}
+ stream = await open_stream(here, controller, opened)
+ await _until(stream, "agent.attached")
+ first = await _ask(here, agent_id, room_id, "one")
+ assert (await _until(stream, "agent.event"))["seq"] == first
+ confirmed = await beat(
+ here,
+ controller,
+ connection_id=conn.id,
+ generation=conn.generation,
+ cursors={agent_id: first},
+ )
+ assert confirmed.status_code == 200, confirmed.text
+ await here.management.ledger.flush_all()
+ # `here` dies now, writing nothing more; then the next event.
+ second = await _ask(here, agent_id, room_id, "two")
+
+ refused = await attach(
+ there, controller, opened["connection_id"], opened["generation"]
+ )
+ assert refused.status_code == 404, refused.text
+ async with there.client() as elsewhere:
+ reopened = await open_connection(
+ elsewhere, controller, cursors={agent_id: first}
+ )
+ resumed = await open_stream(there, controller, reopened)
+ attached = await _until(resumed, "agent.attached")
+ event = await _until(resumed, "agent.event")
+ await resumed.aclose() # type: ignore[attr-defined]
+ await stream.aclose() # type: ignore[attr-defined]
+
+ assert attached["from_seq"] == first
+ assert event["seq"] == second
+
+ async def test_an_attach_naming_another_generation_is_still_refused(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ opened = await open_connection(client, controller)
+
+ with pytest.raises(Exception) as refused:
+ await open_stream(
+ there,
+ controller,
+ {
+ "connection_id": opened["connection_id"],
+ "generation": opened["generation"] + 1,
+ },
+ )
+
+ assert getattr(refused.value, "status_code", None) == 404
+ assert (
+ there.protocol.connections.controllers.current_connection(
+ controller.controller_id
+ )
+ is None
+ )
+
+ async def test_opening_through_the_other_process_takes_the_connection_over(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ await connect(client, controller)
+ loser = here.protocol.connections.controllers.current_connection(
+ controller.controller_id
+ )
+ assert loser is not None
+
+ async with there.client() as elsewhere:
+ await open_connection(elsewhere, controller)
+
+ await _eventually(lambda: loser.closure is not None)
+ assert loser.closure is not None
+ assert loser.closure.code == "taken_over"
+ assert (
+ here.protocol.connections.controllers.current_connection(
+ controller.controller_id
+ )
+ is None
+ )
+
+
+class TestTakenOverOnlyByANewerConnection:
+ """`taken_over` is terminal for a controller: it exits and nothing
+ restarts it. So it is the answer only for a connection older than the one
+ the controller's row names, never for the current one reaching a process
+ whose memory of it is a moment old."""
+
+ async def test_a_reattach_through_the_opener_after_the_other_held_it(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ opened = await open_connection(client, controller)
+ stream = await open_stream(there, controller, opened)
+ await take(stream, 1)
+ presence = here.protocol.connections.controllers
+ await _eventually(
+ lambda: presence.current_connection(controller.controller_id) is None
+ )
+ await stream.aclose() # type: ignore[attr-defined] # the socket drops
+ await there.management.ledger.flush_all()
+
+ answer = await attach(
+ here, controller, opened["connection_id"], opened["generation"]
+ )
+
+ assert answer.status_code == 200, answer.text
+ held = presence.current_connection(controller.controller_id)
+ assert held is not None
+ assert held.id == opened["connection_id"]
+ await _eventually(
+ lambda: (
+ there.protocol.connections.controllers.current_connection(
+ controller.controller_id
+ )
+ is None
+ )
+ )
+
+ async def test_a_reattach_through_the_opener_before_it_heard_of_the_other(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ """The opener has not yet read that the other process took the
+ connection up when the socket comes back through it; reading that
+ later must not close the connection the socket is on."""
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ opened = await open_connection(client, controller)
+ await here.bus.stop() # `here` hears nothing from `there`
+ stream = await open_stream(there, controller, opened)
+ await take(stream, 1)
+ await stream.aclose() # type: ignore[attr-defined] # the socket drops
+ await there.management.ledger.flush_all()
+
+ answer = await attach(
+ here, controller, opened["connection_id"], opened["generation"]
+ )
+ assert answer.status_code == 200, answer.text
+ presence = here.protocol.connections.controllers
+ held = presence.current_connection(controller.controller_id)
+ assert held is not None
+ await here.management.service.signals._read_connection(
+ TENANT_ZERO_ID, controller.controller_id
+ )
+
+ assert held.closure is None
+ assert presence.current_connection(controller.controller_id) is held
+
+ async def test_a_read_of_the_row_during_an_open_leaves_the_new_connection(
+ self,
+ processes: tuple[Harness, Harness],
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """One process is enough: a re-check of the controller's row runs
+ while the open is being recorded."""
+ here, _ = processes
+ owner = await add_member(here.session_factory, "ada")
+ service = here.management.service
+ store = service.controllers
+ record_opened = store.record_opened
+
+ async def with_a_read_alongside(
+ session: AsyncSession, tenant_id: str, controller_id: str, **kwargs: Any
+ ) -> None:
+ await service.signals._read_connection(tenant_id, controller_id)
+ await record_opened(session, tenant_id, controller_id, **kwargs)
+
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ await connect(client, controller)
+ monkeypatch.setattr(store, "record_opened", with_a_read_alongside)
+ second = await open_connection(client, controller)
+
+ answer = await attach(
+ here, controller, second["connection_id"], second["generation"]
+ )
+ assert answer.status_code == 200, answer.text
+
+ async def test_a_connection_the_controller_replaced_is_still_taken_over(
+ self, processes: tuple[Harness, Harness]
+ ) -> None:
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ first = await open_connection(client, controller)
+ async with there.client() as elsewhere:
+ await open_connection(elsewhere, controller)
+ presence = here.protocol.connections.controllers
+ await _eventually(
+ lambda: presence.current_connection(controller.controller_id) is None
+ )
+
+ answer = await attach(
+ here, controller, first["connection_id"], first["generation"]
+ )
+
+ assert answer.status_code == 409
+ assert answer.json()["error"]["code"] == "taken_over"
+
+
+async def _controller_row(harness: Harness, controller_id: str) -> AgentController:
+ async with harness.session_factory() as session:
+ row = await session.scalar(
+ select(AgentController).where(AgentController.id == controller_id)
+ )
+ assert row is not None
+ return row
+
+
+class TestTheRowNamesTheConnectionOpenedLast:
+ """A socket attaching is written as connected only on the connection the
+ controller opened last, held by the process writing it: a socket on an
+ earlier one must not be written over the later one, whichever lands
+ first."""
+
+ async def test_a_late_connected_write_leaves_the_newer_connection_standing(
+ self,
+ processes: tuple[Harness, Harness],
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """The first connection's socket attached here, and its write is
+ still on its way when the controller opens a second connection through
+ the other process and that socket attaches there."""
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ await open_connection(client, controller)
+ store = here.management.service.controllers
+ record_connected = store.record_connected
+ writing = asyncio.Event()
+ written = asyncio.Event()
+
+ async def held_back(
+ session: AsyncSession, tenant_id: str, controller_id: str, **kwargs: Any
+ ) -> bool:
+ writing.set()
+ await written.wait()
+ return await record_connected(session, tenant_id, controller_id, **kwargs)
+
+ monkeypatch.setattr(store, "record_connected", held_back)
+ presence = here.protocol.connections.controllers
+ first = presence.current_connection(controller.controller_id)
+ assert first is not None
+ presence.attach_stream(first)
+ await asyncio.wait_for(writing.wait(), WAIT_SECONDS)
+
+ async with there.client() as elsewhere:
+ second = await open_connection(elsewhere, controller)
+ stream = await open_stream(there, controller, second)
+ await take(stream, 1)
+ await there.management.ledger.flush_all()
+ written.set()
+ await here.management.ledger.flush_all()
+ await stream.aclose() # type: ignore[attr-defined]
+
+ row = await _controller_row(here, controller.controller_id)
+ assert row.connection_id == second["connection_id"]
+ assert row.connection_process_id == there.management.lease.process_id
+ assert row.disconnected_at is None
+
+ async def test_an_attach_overtaken_while_it_read_the_row_is_refused(
+ self,
+ processes: tuple[Harness, Harness],
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """The controller opens a second connection through the other process,
+ whose socket attaches there, while an attach of the first through this
+ process is between reading the row and attaching: it is refused as
+ taken over, and this process still knows the other holds the socket."""
+ here, there = processes
+ owner = await add_member(here.session_factory, "ada")
+ async with here.client() as client:
+ controller = await enroll_console(here, client, owner)
+ first = await open_connection(client, controller)
+ service = here.management.service
+ connection_state = service.connection_state
+
+ async def overtaken(session: AsyncSession, principal: Any) -> dict[str, Any]:
+ async with there.client() as elsewhere:
+ await open_connection(elsewhere, controller)
+ other = there.protocol.connections.controllers
+ second = other.current_connection(controller.controller_id)
+ assert second is not None
+ other.attach_stream(second)
+ await there.management.ledger.flush_all()
+ await service.signals._read_connection(
+ TENANT_ZERO_ID, controller.controller_id
+ )
+ return await connection_state(session, principal)
+
+ monkeypatch.setattr(service, "connection_state", overtaken)
+
+ answer = await attach(
+ here, controller, first["connection_id"], first["generation"]
+ )
+
+ assert answer.status_code == 409, answer.text
+ assert answer.json()["error"]["code"] == "taken_over"
+ assert (
+ here.protocol.connections.controllers.holder_elsewhere(
+ controller.controller_id
+ )
+ == there.management.lease.process_id
+ )
diff --git a/core/tests/switch_core/management/test_middleware.py b/core/tests/switch_core/management/test_middleware.py
index b86851811..adeab0150 100644
--- a/core/tests/switch_core/management/test_middleware.py
+++ b/core/tests/switch_core/management/test_middleware.py
@@ -32,6 +32,7 @@ from switch_core.management.auth import ManagementAuthenticator
from switch_core.management.wiring import create_management
from switch_core.tenant_context import current_tenant_id
from switch_core.trust.client import NullTrustClient
+from tests.db_buffer import QuietRooms
from tests.switch_core.management.harness import (
TOKEN_SECRET,
Harness,
@@ -279,7 +280,13 @@ class TestTheFlagOff:
agent_management_enabled = False
assert (
- create_management(_Off(), object(), AgentConnectionRegistry().controllers) # type: ignore[arg-type]
+ create_management(
+ _Off(), # type: ignore[arg-type]
+ object(), # type: ignore[arg-type]
+ AgentConnectionRegistry().controllers,
+ object(), # type: ignore[arg-type]
+ object(), # type: ignore[arg-type]
+ )
is None
)
@@ -301,7 +308,7 @@ class TestTheFlagOff:
room_service=object(), # type: ignore[arg-type]
client_lifecycle=object(), # type: ignore[arg-type]
collab_lifecycle=object(), # type: ignore[arg-type]
- event_buffer=EventBuffer(object(), sequence_base=0), # type: ignore[arg-type]
+ event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]
task_store=object(), # type: ignore[arg-type]
resource_service=object(), # type: ignore[arg-type]
api_key_store=object(), # type: ignore[arg-type]
diff --git a/core/tests/switch_core/observability/test_bootstrap.py b/core/tests/switch_core/observability/test_bootstrap.py
index 02edafaf9..4ad9a7e15 100644
--- a/core/tests/switch_core/observability/test_bootstrap.py
+++ b/core/tests/switch_core/observability/test_bootstrap.py
@@ -71,6 +71,8 @@ def _probes(**overrides) -> RuntimeProbes:
defaults = dict(
listener_connected=lambda: True,
session_activity_listener_connected=lambda: True,
+ agent_event_listener_connected=lambda: True,
+ signal_bus_connected=lambda: True,
bridges_running=lambda: 2,
bridges_running_by_platform=lambda: {"slack": 1, "mattermost": 1},
bridges_configured=lambda: 2,
@@ -100,6 +102,7 @@ async def test_health_runs_even_with_nothing_configured(monkeypatch):
config=_config(monkeypatch),
version="1.0.0",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(),
)
try:
@@ -117,6 +120,7 @@ async def test_a_dead_database_fails_readiness(monkeypatch):
config=_config(monkeypatch),
version="1.0.0",
session_factory=_session_factory(fail=True),
+ process_id="core-0-abcd1234",
probes=_probes(),
)
try:
@@ -135,6 +139,7 @@ async def test_a_crashed_bridge_is_reported_without_taking_the_server_down(
config=_config(monkeypatch),
version="1.0.0",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(bridges_running=lambda: 1, bridges_configured=lambda: 3),
)
try:
@@ -158,6 +163,7 @@ async def test_an_endpoint_installs_the_registry_and_reports_state(monkeypatch):
),
version="1.0.0",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(),
)
try:
@@ -207,6 +213,7 @@ async def test_the_room_cache_reports_what_it_holds(monkeypatch):
config=config,
version="1.0.0",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(
room_cache_stats=lambda: RoomCacheStats(bytes=4096, rooms=3, rows=12)
),
@@ -230,6 +237,7 @@ async def test_a_start_is_counted_once_so_restarts_line_up(monkeypatch):
),
version="1.0.0",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(),
)
try:
@@ -253,6 +261,7 @@ async def test_a_pool_that_reports_nothing_produces_no_reading(monkeypatch):
),
version="1.0.0",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(pool_stats=lambda: None),
)
try:
@@ -280,6 +289,7 @@ async def test_metrics_reach_the_collector(monkeypatch):
),
version="9.9.9",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(),
)
try:
@@ -312,6 +322,7 @@ async def test_closing_stops_the_loops_and_uninstalls(monkeypatch):
),
version="1.0.0",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(),
)
monkeypatch.setattr(
@@ -343,6 +354,7 @@ async def test_the_database_sampler_starts_with_the_metrics_exporter(monkeypatch
),
version="1.0.0",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(),
db_server_engine=engine,
)
@@ -369,6 +381,7 @@ async def test_a_connection_kind_that_empties_out_is_reported_as_zero(monkeypatc
),
version="1.0.0",
session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
probes=_probes(agents_connected=lambda: dict(connected)),
)
try:
@@ -391,3 +404,54 @@ async def test_a_connection_kind_that_empties_out_is_reported_as_zero(monkeypatc
}
finally:
await observability.aclose()
+
+
+async def test_one_replica_stays_in_service_with_a_listener_down(monkeypatch):
+ """With nowhere else to send the traffic, late is better than none."""
+ observability = start_observability(
+ config=_config(monkeypatch),
+ version="1.0.0",
+ session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
+ probes=_probes(
+ agent_event_listener_connected=lambda: False,
+ signal_bus_connected=lambda: False,
+ ),
+ )
+ try:
+ report = await observability.monitor.refresh()
+ assert report.ready is True
+ checks = report.as_response()["checks"]
+ assert checks["agent_event_listener"]["healthy"] is False
+ assert checks["signal_bus"]["healthy"] is False
+ finally:
+ await observability.aclose()
+
+
+ "dead",
+ [
+ "listener_connected",
+ "session_activity_listener_connected",
+ "agent_event_listener_connected",
+ "signal_bus_connected",
+ ],
+)
+async def test_with_several_replicas_a_dead_listener_takes_the_pod_out(
+ monkeypatch, dead
+):
+ """Another replica can take the traffic this one would answer late."""
+ observability = start_observability(
+ config=_config(monkeypatch, CORE_REPLICAS="2"),
+ version="1.0.0",
+ session_factory=_session_factory(),
+ process_id="core-0-abcd1234",
+ probes=_probes(**{dead: lambda: False}),
+ )
+ try:
+ report = await observability.monitor.refresh()
+ assert report.ready is False
+ finally:
+ await observability.aclose()
diff --git a/core/tests/switch_core/observability/test_connection_metrics.py b/core/tests/switch_core/observability/test_connection_metrics.py
index 07c5cb86f..8b79389a7 100644
--- a/core/tests/switch_core/observability/test_connection_metrics.py
+++ b/core/tests/switch_core/observability/test_connection_metrics.py
@@ -38,6 +38,7 @@ from switch_core.db.stores.api_key_store import ApiKeyStore
from switch_core.observability.bootstrap import RuntimeProbes, _state_readings
from switch_core.observability.metrics import MetricsRegistry, install, uninstall
from switch_core.transport.room_cache import RoomCacheStats
+from tests.db_buffer import QuietRooms
AGENT_ID = "agent-1"
@@ -156,6 +157,8 @@ def _probes(agents_connected: Any) -> RuntimeProbes:
return RuntimeProbes(
listener_connected=lambda: True,
session_activity_listener_connected=lambda: True,
+ agent_event_listener_connected=lambda: True,
+ signal_bus_connected=lambda: True,
bridges_running=lambda: 0,
bridges_running_by_platform=lambda: {},
bridges_configured=lambda: 0,
@@ -250,7 +253,7 @@ class _EmptyBuffer(EventBuffer):
"""
def __init__(self) -> None:
- super().__init__(object(), sequence_base=0) # type: ignore[arg-type]
+ super().__init__(object(), sequence_base=0, rooms=QuietRooms()) # type: ignore[arg-type]
async def head(self, tenant_id: str, agent_id: str) -> int:
return self.sequence_floor - 1
diff --git a/core/tests/switch_core/observability/test_exporter.py b/core/tests/switch_core/observability/test_exporter.py
index ce535b782..acb62fc25 100644
--- a/core/tests/switch_core/observability/test_exporter.py
+++ b/core/tests/switch_core/observability/test_exporter.py
@@ -20,6 +20,7 @@ RESOURCE = OtlpResource(
deployment_id="0e5d1b3a-6c1f-4c22-9a4c-3a9f5a2b7d10",
commit_sha=None,
repository_url=None,
+ instance_id=None,
)
ATTRS = {"route": "/health", "method": "GET", "status_class": "2xx"}
diff --git a/core/tests/switch_core/observability/test_health.py b/core/tests/switch_core/observability/test_health.py
index 00062d68b..bd5918ada 100644
--- a/core/tests/switch_core/observability/test_health.py
+++ b/core/tests/switch_core/observability/test_health.py
@@ -7,10 +7,12 @@ from switch_core.observability.health import (
CheckOutcome,
HealthCheck,
HealthMonitor,
+ agent_event_listener_check,
bridges_check,
connectors_check,
message_listener_check,
session_activity_listener_check,
+ signal_bus_check,
)
from switch_core.observability.metrics import MetricsRegistry
@@ -125,7 +127,7 @@ async def test_the_stale_report_still_shows_what_was_last_known(monkeypatch):
@pytest.mark.asyncio
async def test_message_listener_check_follows_the_connection():
connected = [True]
- check = message_listener_check(lambda: connected[0])
+ check = message_listener_check(lambda: connected[0], gates_readiness=False)
assert (await check.probe()).healthy is True
connected[0] = False
@@ -215,7 +217,7 @@ async def test_connectors_check_is_healthy_when_none_are_configured():
@pytest.mark.asyncio
async def test_session_activity_listener_check_follows_the_connection():
connected = [True]
- check = session_activity_listener_check(lambda: connected[0])
+ check = session_activity_listener_check(lambda: connected[0], gates_readiness=False)
assert (await check.probe()).healthy is True
connected[0] = False
@@ -223,3 +225,24 @@ async def test_session_activity_listener_check_follows_the_connection():
assert outcome.healthy is False
assert "LISTEN" in outcome.detail
assert check.gates_readiness is False
+
+
+ "build, name",
+ [
+ (agent_event_listener_check, "agent_event_listener"),
+ (signal_bus_check, "signal_bus"),
+ ],
+)
+async def test_the_cross_process_listeners_follow_their_connections(build, name):
+ connected = [True]
+ check = build(lambda: connected[0], gates_readiness=True)
+
+ assert (await check.probe()).healthy is True
+ connected[0] = False
+ outcome = await check.probe()
+ assert outcome.name == name
+ assert outcome.healthy is False
+ assert "LISTEN" in outcome.detail
+ assert check.gates_readiness is True
diff --git a/core/tests/switch_core/observability/test_log_context_bindings.py b/core/tests/switch_core/observability/test_log_context_bindings.py
index f17fafa46..68331604b 100644
--- a/core/tests/switch_core/observability/test_log_context_bindings.py
+++ b/core/tests/switch_core/observability/test_log_context_bindings.py
@@ -68,6 +68,7 @@ class TestTheDeliveryLoopBindsItsRoom:
transport._wake = asyncio.Event() # type: ignore[attr-defined]
transport._pending = {"room-7"} # type: ignore[attr-defined]
transport._delivering = False # type: ignore[attr-defined]
+ transport._stopping = False # type: ignore[attr-defined]
async def _boom(room_id: str) -> None:
logging.getLogger("switch_core.test").info("inside the drain")
@@ -97,6 +98,7 @@ class TestTheDeliveryLoopBindsItsRoom:
transport._wake = asyncio.Event() # type: ignore[attr-defined]
transport._pending = {"room-7"} # type: ignore[attr-defined]
transport._delivering = False # type: ignore[attr-defined]
+ transport._stopping = False # type: ignore[attr-defined]
async def _ok(room_id: str) -> None:
return None
diff --git a/core/tests/switch_core/observability/test_logs.py b/core/tests/switch_core/observability/test_logs.py
index 3ade5f1f0..0dcfa160c 100644
--- a/core/tests/switch_core/observability/test_logs.py
+++ b/core/tests/switch_core/observability/test_logs.py
@@ -19,6 +19,7 @@ RESOURCE = OtlpResource(
deployment_id="0e5d1b3a-6c1f-4c22-9a4c-3a9f5a2b7d10",
commit_sha=None,
repository_url=None,
+ instance_id=None,
)
diff --git a/core/tests/switch_core/observability/test_otlp.py b/core/tests/switch_core/observability/test_otlp.py
index 45f8676e4..f9e6d1ea9 100644
--- a/core/tests/switch_core/observability/test_otlp.py
+++ b/core/tests/switch_core/observability/test_otlp.py
@@ -33,6 +33,7 @@ RESOURCE = OtlpResource(
deployment_id="0e5d1b3a-6c1f-4c22-9a4c-3a9f5a2b7d10",
commit_sha=None,
repository_url=None,
+ instance_id=None,
)
START_NANOS = 1_700_000_000_000_000_000
@@ -81,6 +82,7 @@ def test_unknown_version_is_omitted_rather_than_placeheld():
deployment_id=RESOURCE.deployment_id,
commit_sha=None,
repository_url=None,
+ instance_id=None,
)
attributes = resource.attributes()
@@ -88,6 +90,21 @@ def test_unknown_version_is_omitted_rather_than_placeheld():
assert "deployment.environment" not in attributes
assert "git.commit.sha" not in attributes
assert "git.repository_url" not in attributes
+ assert "service.instance.id" not in attributes
+
+
+def test_resource_names_the_process_so_pods_report_apart():
+ resource = OtlpResource(
+ service_name="switch-core",
+ service_version="1.2.3",
+ environment="pilot",
+ deployment_id=RESOURCE.deployment_id,
+ commit_sha=None,
+ repository_url=None,
+ instance_id="switch-core-7d9f-abcd1234",
+ )
+
+ assert resource.attributes()["service.instance.id"] == "switch-core-7d9f-abcd1234"
def test_resource_names_the_commit_it_was_built_from():
@@ -98,6 +115,7 @@ def test_resource_names_the_commit_it_was_built_from():
deployment_id=RESOURCE.deployment_id,
commit_sha="0123456789abcdef0123456789abcdef01234567",
repository_url="https://github.com/example/switch",
+ instance_id=None,
)
attributes = resource.attributes()
diff --git a/core/tests/switch_core/pool_checkouts.py b/core/tests/switch_core/pool_checkouts.py
index c89e398c4..75129c4f0 100644
--- a/core/tests/switch_core/pool_checkouts.py
+++ b/core/tests/switch_core/pool_checkouts.py
@@ -32,11 +32,13 @@ from sqlalchemy.ext.asyncio import AsyncEngine
@dataclass
class CheckoutScope:
- """One request's connections: how many it holds now, and the most at once."""
+ """One request's connections: how many it holds now, the most at once,
+ and how many it borrowed in all."""
label: str
held: int = 0
peak: int = 0
+ taken: int = 0
_current: ContextVar[CheckoutScope | None] = ContextVar(
@@ -61,6 +63,7 @@ class PoolCheckouts:
if scope is None:
return
scope.held += 1
+ scope.taken += 1
scope.peak = max(scope.peak, scope.held)
self._owners[id(record)] = scope
diff --git a/core/tests/switch_core/provisioning/test_postgres_provisioning.py b/core/tests/switch_core/provisioning/test_postgres_provisioning.py
index ef92ac2ed..867a3dccd 100644
--- a/core/tests/switch_core/provisioning/test_postgres_provisioning.py
+++ b/core/tests/switch_core/provisioning/test_postgres_provisioning.py
@@ -26,6 +26,7 @@ from switch_core.provisioning.postgres import (
from switch_core.tenant_context import tenant_scope
from switch_core.transport.invites import InviteBus
from tests.conftest import RLSHarness
+from tests.routing import router_on
async def _client(session: AsyncSession, name: str = "someone") -> Client:
@@ -55,6 +56,11 @@ async def _never(_room_id: str) -> None:
raise AssertionError("the other half of the registration was rung")
+async def _no_reconcile() -> None:
+ """The re-read a change made in another process asks for. Nothing here
+ calls it: these tests make every change in this process."""
+
+
def _provisioning(
session_factory: async_sessionmaker[AsyncSession],
invites: InviteBus | None = None,
@@ -63,7 +69,7 @@ def _provisioning(
session_factory=session_factory,
room_store=RoomStore(),
client_store=ClientStore(),
- message_store=MessageStore(),
+ router=router_on(session_factory),
invites=invites or InviteBus(),
)
@@ -135,7 +141,7 @@ class TestMembership:
# because `transport_user_id` is unique per tenant and every tenant's
# admin client carries the same one.
invites = InviteBus()
- invites.register(client_id, _handler, _never)
+ invites.register(client_id, _handler, _never, _no_reconcile)
await _provisioning(session_factory, invites).invite_to_room(
transport_room_id, user_id
)
@@ -239,7 +245,7 @@ class TestMembership:
invites = InviteBus()
provisioning = _provisioning(session_factory, invites)
await provisioning.invite_to_room(transport_room_id, user_id)
- invites.register(client_id, _never, _handler)
+ invites.register(client_id, _never, _handler, _no_reconcile)
await provisioning.kick_user(transport_room_id, user_id)
assert told == [transport_room_id]
@@ -277,11 +283,11 @@ class TestMembership:
provisioning = _provisioning(session_factory, invites)
# Registered under the handle, which is what the bus is *not* keyed on.
- invites.register(user_id, _never, _handler)
+ invites.register(user_id, _never, _handler, _no_reconcile)
await provisioning.kick_user(transport_room_id, user_id)
assert told == []
- invites.register(client_id, _never, _handler)
+ invites.register(client_id, _never, _handler, _no_reconcile)
await provisioning.kick_user(transport_room_id, user_id)
assert told == [transport_room_id]
@@ -341,7 +347,7 @@ class TestMembership:
await provisioning.invite_to_room(transport_room_id, user_id)
invites = InviteBus()
- invites.register(client.id, _handler, _never)
+ invites.register(client.id, _handler, _never, _no_reconcile)
await _provisioning(session_factory, invites).invite_to_room(
transport_room_id, user_id
)
@@ -417,7 +423,9 @@ class TestTwoTenantsSharingAHandle:
invites = InviteBus()
for tenant_id in tenants:
- invites.register(clients[tenant_id], _handler_for(tenant_id), _never)
+ invites.register(
+ clients[tenant_id], _handler_for(tenant_id), _never, _no_reconcile
+ )
provisioning = _provisioning(rls_harness.restricted, invites)
for tenant_id in tenants:
diff --git a/core/tests/switch_core/session_activity/test_publisher.py b/core/tests/switch_core/session_activity/test_publisher.py
index abcc49d96..f5c2b5187 100644
--- a/core/tests/switch_core/session_activity/test_publisher.py
+++ b/core/tests/switch_core/session_activity/test_publisher.py
@@ -15,6 +15,7 @@ from switch_core.db.models import (
BridgeMessageMap,
TurnStatusPost,
)
+from switch_core.session_activity import publisher as publisher_module
from switch_core.session_activity.listener import AgentSessionActivityListener
from switch_core.session_activity.publisher import (
AgentSessionActivityPublisher,
@@ -598,3 +599,39 @@ async def test_a_restart_redraws_running_turns_but_not_ones_already_shown_ended(
]
finally:
await publisher.stop()
+
+
+# ── Stopping ─────────────────────────────────────────────────────────────────
+
+
+async def test_a_task_that_died_is_logged_on_stopping_and_not_raised(
+ service, session_factory, bridged, listener, platform, monkeypatch, caplog
+):
+ """The tick has no guard of its own, so an online check that raises ends
+ it. Stopping logs what ended it rather than raising it, so the bridge
+ stopping the publisher goes on to close its connection, and a second stop
+ has nothing left to raise."""
+ monkeypatch.setattr(publisher_module, "_TICK_SECONDS", 0.01)
+ answering = True
+ refused = asyncio.Event()
+
+ def online(_agent_id: str) -> bool:
+ if answering:
+ return True
+ refused.set()
+ raise RuntimeError("the connection registry is gone")
+
+ publisher = _publisher(session_factory, bridged, listener, platform, online)
+ publisher.start()
+ await _turn(service, bridged.room_id, "running", 1)
+ await platform.wait_until(lambda: len(_turns(platform)) == 1)
+ answering = False
+ await asyncio.wait_for(refused.wait(), 5)
+
+ with caplog.at_level("ERROR"):
+ await publisher.stop()
+ await publisher.stop()
+
+ died = [r for r in caplog.records if "stopped on an error" in r.getMessage()]
+ assert len(died) == 1
+ assert "tick" in died[0].getMessage()
diff --git a/core/tests/switch_core/telemetry/test_bridge_connect_reporting.py b/core/tests/switch_core/telemetry/test_bridge_connect_reporting.py
index 28a720b28..bfebd621e 100644
--- a/core/tests/switch_core/telemetry/test_bridge_connect_reporting.py
+++ b/core/tests/switch_core/telemetry/test_bridge_connect_reporting.py
@@ -42,6 +42,7 @@ from switch_core.db.stores.room_store import RoomStore
from switch_core.keys import Keyring
from switch_core.telemetry.service import TelemetryService
from switch_core.telemetry.sink import TelemetryRecord
+from tests.ownership import SoleOwner
from tests.switch_core.bridges.collaboration.test_lifecycle_callback_endpoint import (
_FailingClient,
)
@@ -103,6 +104,8 @@ def _service_with_telemetry(
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(session_factory),
telemetry=telemetry,
)
return service, sink
@@ -421,10 +424,11 @@ class TestStartAllsBootPathIsCoveredToo:
async def test_a_bridge_that_fails_at_boot_is_reported_and_the_rest_still_start(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
- """`start_all` (`:309-313`) only ever logged and moved on past a
- bridge that failed to start — it still does, but the failure now
- reaches telemetry too, because `start` (which it calls unmodified)
- reports before raising back out to it."""
+ """Boot (`start_all` wanting every bridge, then the ownership's first
+ balance starting each one this process holds) only ever logged and
+ moved on past a bridge that failed to start. It still does, but the
+ failure now reaches telemetry too, because `start` reports before
+ raising back out to it."""
tenant = f"tenant-{uuid.uuid4().hex[:8]}"
async with session_factory() as session:
await _make_tenant(session, tenant)
@@ -443,6 +447,7 @@ class TestStartAllsBootPathIsCoveredToo:
service._run_bridge = AsyncMock() # type: ignore[method-assign]
await service.start_all()
+ await service._owned.settle()
await service._telemetry.aclose() # type: ignore[union-attr]
events = await _bridge_connected_events(sink)
diff --git a/core/tests/switch_core/telemetry/test_deployment_and_reporter.py b/core/tests/switch_core/telemetry/test_deployment_and_reporter.py
index 85e295b2a..20c3be2c8 100644
--- a/core/tests/switch_core/telemetry/test_deployment_and_reporter.py
+++ b/core/tests/switch_core/telemetry/test_deployment_and_reporter.py
@@ -9,6 +9,7 @@ relies on a primary-key collision being the guard.
from __future__ import annotations
+import asyncio
import contextlib
import logging
import uuid
@@ -434,3 +435,105 @@ class TestTheSnapshotSchedule:
await session.commit()
assert await reporter.run_once_if_due() is True
+
+
+class TestOneProcessSendsTheSnapshot:
+ """Every switch-core process runs the snapshot loop. The watermark is
+ claimed before anything is sent, so two processes due at the same moment
+ send one snapshot between them."""
+
+ def _reporter(
+ self, sink: _RecordingSink, session_factory: async_sessionmaker[AsyncSession]
+ ) -> SnapshotReporter:
+ return TestTheSnapshotSchedule()._reporter(sink, session_factory)
+
+ async def _sent(self, sink: _RecordingSink) -> int:
+ return len([r for r in sink.sent if r.name == "switch_core.usage_snapshot"])
+
+ @pytest.mark.parametrize("watermark_days_ago", [None, 2])
+ async def test_two_processes_due_together_send_one_snapshot(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ watermark_days_ago: int | None,
+ ) -> None:
+ async with session_factory() as session:
+ await session.execute(delete(TelemetrySnapshotWatermark))
+ if watermark_days_ago is not None:
+ session.add(
+ TelemetrySnapshotWatermark(
+ id=1,
+ last_sent_at=datetime.now(UTC)
+ - timedelta(days=watermark_days_ago),
+ )
+ )
+ await session.commit()
+ first_sink, second_sink = _RecordingSink(), _RecordingSink()
+ first = self._reporter(first_sink, session_factory)
+ second = self._reporter(second_sink, session_factory)
+
+ # Both read the watermark as due before either claims it: the window
+ # the claim exists to close.
+ read_both = asyncio.Barrier(2)
+ for reporter in (first, second):
+ original = reporter._read_watermark
+
+ async def read_then_wait(original=original) -> datetime | None:
+ value = await original()
+ await read_both.wait()
+ return value
+
+ reporter._read_watermark = read_then_wait # type: ignore[method-assign]
+
+ results = await asyncio.gather(
+ first.run_once_if_due(), second.run_once_if_due()
+ )
+
+ assert sorted(results) == [False, True]
+ assert await self._sent(first_sink) + await self._sent(second_sink) == 1
+
+ async def test_a_pass_that_fails_before_sending_hands_the_claim_back(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """Otherwise one failed count would skip the day for every process."""
+ before = datetime.now(UTC) - timedelta(days=2)
+ async with session_factory() as session:
+ await session.execute(delete(TelemetrySnapshotWatermark))
+ session.add(TelemetrySnapshotWatermark(id=1, last_sent_at=before))
+ await session.commit()
+
+ async def broken(*args: object, **kwargs: object) -> object:
+ raise RuntimeError("the count fell over")
+
+ monkeypatch.setattr("switch_core.telemetry.reporter.collect_usage", broken)
+ sink = _RecordingSink()
+ reporter = self._reporter(sink, session_factory)
+
+ with pytest.raises(RuntimeError, match="the count fell over"):
+ await reporter.run_once_if_due()
+
+ assert await reporter._read_watermark() == before
+ monkeypatch.undo()
+ assert await reporter.run_once_if_due() is True
+ assert await self._sent(sink) == 1
+
+ async def test_a_first_pass_that_fails_leaves_no_watermark(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ async with session_factory() as session:
+ await session.execute(delete(TelemetrySnapshotWatermark))
+ await session.commit()
+
+ async def broken(*args: object, **kwargs: object) -> object:
+ raise RuntimeError("the count fell over")
+
+ monkeypatch.setattr("switch_core.telemetry.reporter.collect_usage", broken)
+ reporter = self._reporter(_RecordingSink(), session_factory)
+
+ with pytest.raises(RuntimeError):
+ await reporter.run_once_if_due()
+
+ assert await reporter._read_watermark() is None
diff --git a/core/tests/switch_core/telemetry/test_emitters_are_live.py b/core/tests/switch_core/telemetry/test_emitters_are_live.py
index 4ed0d087d..4f41407d5 100644
--- a/core/tests/switch_core/telemetry/test_emitters_are_live.py
+++ b/core/tests/switch_core/telemetry/test_emitters_are_live.py
@@ -30,6 +30,7 @@ from switch_core.keys import Keyring
from switch_core.telemetry.service import TelemetryService
from switch_core.telemetry.sink import NullSink
from switch_core.trust.client import NullTrustClient
+from tests.db_buffer import QuietRooms
def _telemetry() -> TelemetryService:
@@ -52,7 +53,7 @@ def _build(telemetry: TelemetryService) -> Any:
room_service=object(), # type: ignore[arg-type]
client_lifecycle=object(), # type: ignore[arg-type]
collab_lifecycle=object(), # type: ignore[arg-type]
- event_buffer=EventBuffer(object(), sequence_base=0), # type: ignore[arg-type]
+ event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]
task_store=object(), # type: ignore[arg-type]
resource_service=object(), # type: ignore[arg-type]
api_key_store=object(), # type: ignore[arg-type]
diff --git a/core/tests/switch_core/test_ingress_agent_api_paths.py b/core/tests/switch_core/test_ingress_agent_api_paths.py
index fdd063dbf..bb05c4f7a 100644
--- a/core/tests/switch_core/test_ingress_agent_api_paths.py
+++ b/core/tests/switch_core/test_ingress_agent_api_paths.py
@@ -19,6 +19,7 @@ from switch_core.bridges.agent.protocol.event_buffer import EventBuffer
from switch_core.keys import Keyring
from switch_core.management.controller_routes import router as controller_router
from switch_core.trust.client import NullTrustClient
+from tests.db_buffer import QuietRooms
_CHART = Path(__file__).resolve().parents[3] / "deploy/remote/helm/switch"
@@ -53,7 +54,7 @@ def _served_prefixes() -> set[str]:
room_service=object(), # type: ignore[arg-type]
client_lifecycle=object(), # type: ignore[arg-type]
collab_lifecycle=object(), # type: ignore[arg-type]
- event_buffer=EventBuffer(object(), sequence_base=0), # type: ignore[arg-type]
+ event_buffer=EventBuffer(object(), sequence_base=0, rooms=QuietRooms()), # type: ignore[arg-type]
task_store=object(), # type: ignore[arg-type]
resource_service=object(), # type: ignore[arg-type]
api_key_store=object(), # type: ignore[arg-type]
diff --git a/core/tests/switch_core/test_logging_config.py b/core/tests/switch_core/test_logging_config.py
index ab8535138..b83c2abb2 100644
--- a/core/tests/switch_core/test_logging_config.py
+++ b/core/tests/switch_core/test_logging_config.py
@@ -59,7 +59,7 @@ class _Capture(logging.Handler):
@pytest.fixture
def json_lines(monkeypatch: pytest.MonkeyPatch):
config = _config(monkeypatch, LOG_FORMAT="json", ENVIRONMENT="pilot")
- capture = _Capture(build_handler(config, "1.2.3"))
+ capture = _Capture(build_handler(config, "1.2.3", "core-0-abcd1234"))
logger = logging.getLogger("switch_core.test.json")
logger.propagate = False
logger.setLevel(logging.DEBUG)
@@ -85,13 +85,14 @@ def test_json_line_carries_tenant_without_anything_binding_it(json_lines) -> Non
assert entry["service"] == "switch-core"
assert entry["env"] == "pilot"
assert entry["version"] == "1.2.3"
+ assert entry["process_id"] == "core-0-abcd1234"
assert entry["logger"]["name"] == "switch_core.test.json"
assert "timestamp" in entry
def test_configured_tenant_is_used(monkeypatch: pytest.MonkeyPatch) -> None:
config = _config(monkeypatch, LOG_FORMAT="json", TENANT_ID="acme")
- capture = _Capture(build_handler(config, None))
+ capture = _Capture(build_handler(config, None, "core-0-abcd1234"))
logger = logging.getLogger("switch_core.test.tenant")
logger.propagate = False
logger.handlers = [capture]
@@ -169,7 +170,7 @@ def test_unserialisable_value_does_not_lose_the_line(json_lines) -> None:
def test_text_format_shows_the_context(monkeypatch: pytest.MonkeyPatch) -> None:
config = _config(monkeypatch, LOG_FORMAT="text")
- capture = _Capture(build_handler(config, "1.2.3"))
+ capture = _Capture(build_handler(config, "1.2.3", "core-0-abcd1234"))
logger = logging.getLogger("switch_core.test.text")
logger.propagate = False
logger.setLevel(logging.DEBUG)
@@ -180,14 +181,17 @@ def test_text_format_shows_the_context(monkeypatch: pytest.MonkeyPatch) -> None:
logger.info("plain")
with_context, without = capture.lines
- assert "[tenant_id=default request_id=req-1] hello" in with_context
- assert "[tenant_id=default] plain" in without
+ assert (
+ "[process_id=core-0-abcd1234 tenant_id=default request_id=req-1] hello"
+ in with_context
+ )
+ assert "[process_id=core-0-abcd1234 tenant_id=default] plain" in without
def test_a_third_party_logger_gets_the_same_fields(monkeypatch: pytest.MonkeyPatch):
"""The filter is on the handler, so library records carry the fields too."""
config = _config(monkeypatch, LOG_FORMAT="json")
- capture = _Capture(build_handler(config, None))
+ capture = _Capture(build_handler(config, None, "core-0-abcd1234"))
logger = logging.getLogger("uvicorn.access")
logger.propagate = False
logger.handlers = [capture]
@@ -235,7 +239,7 @@ def test_a_record_that_missed_the_filter_still_formats(
) -> None:
"""A formatter must not raise on a record from a handler without the filter."""
config = _config(monkeypatch, LOG_FORMAT="json")
- handler = build_handler(config, None)
+ handler = build_handler(config, None, "core-0-abcd1234")
formatter = handler.formatter
assert formatter is not None
record = logging.LogRecord(
@@ -357,7 +361,7 @@ class TestTheLogAgreesWithTheDatabase:
@pytest.mark.parametrize("format", ["text", "json"])
def test_access_logs_remove_authorization_query_parameters(monkeypatch, format):
config = _config(monkeypatch, LOG_FORMAT=format)
- capture = _Capture(build_handler(config, None))
+ capture = _Capture(build_handler(config, None, "core-0-abcd1234"))
record = logging.LogRecord(
"uvicorn.access",
logging.INFO,
diff --git a/core/tests/switch_core/test_main_release.py b/core/tests/switch_core/test_main_release.py
new file mode 100644
index 000000000..b5d32da77
--- /dev/null
+++ b/core/tests/switch_core/test_main_release.py
@@ -0,0 +1,103 @@
+"""A process shutting down hands over the work it owns before its lease goes.
+
+The other processes may take over a process's owned work (a client's
+consumer, a bridge's connection, a server-side connector) as soon as its
+lease reads as stopped. So the lease is marked stopped only after that work
+has stopped here; if it cannot be stopped, the lease is left to lapse on its
+own, which gives the work time to end before anyone else starts it.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Any
+
+import pytest
+
+from switch_core import main as main_module
+from switch_core.main import _release_process
+
+
+class _Ownership:
+ def __init__(self, calls: list[str], *, stop: Any = None) -> None:
+ self._calls = calls
+ self._stop = stop
+
+ async def stop(self) -> None:
+ if self._stop is not None:
+ await self._stop()
+ self._calls.append("owned work handed over")
+
+ def gone(self) -> None:
+ self._calls.append("others told")
+
+
+class _Connections:
+ def __init__(self, calls: list[str]) -> None:
+ self._calls = calls
+
+ async def flush(self) -> None:
+ self._calls.append("connections recorded")
+
+
+class _Lease:
+ process_id = "test-pod-1"
+
+ def __init__(self, calls: list[str]) -> None:
+ self._calls = calls
+
+ async def stop(self) -> None:
+ self._calls.append("lease released")
+
+
+async def test_owned_work_is_handed_over_before_the_lease_is_released() -> None:
+ calls: list[str] = []
+
+ await _release_process(
+ _Ownership(calls), # type: ignore[arg-type]
+ _Connections(calls), # type: ignore[arg-type]
+ None,
+ _Lease(calls), # type: ignore[arg-type]
+ )
+
+ assert calls == [
+ "owned work handed over",
+ "connections recorded",
+ "lease released",
+ "others told",
+ ]
+
+
+async def test_a_hand_over_that_fails_leaves_the_lease_to_lapse() -> None:
+ calls: list[str] = []
+
+ async def fail() -> None:
+ raise RuntimeError("a bridge would not stop")
+
+ await _release_process(
+ _Ownership(calls, stop=fail), # type: ignore[arg-type]
+ _Connections(calls), # type: ignore[arg-type]
+ None,
+ _Lease(calls), # type: ignore[arg-type]
+ )
+
+ assert "lease released" not in calls
+
+
+async def test_a_hand_over_that_hangs_leaves_the_lease_to_lapse(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(main_module, "_HAND_OVER_SECONDS", 0.05)
+ calls: list[str] = []
+
+ async def hang() -> None:
+ await asyncio.Event().wait()
+
+ await _release_process(
+ _Ownership(calls, stop=hang), # type: ignore[arg-type]
+ _Connections(calls), # type: ignore[arg-type]
+ None,
+ _Lease(calls), # type: ignore[arg-type]
+ )
+
+ assert calls == []
diff --git a/core/tests/switch_core/test_main_teams_app.py b/core/tests/switch_core/test_main_teams_app.py
index 26493ce78..bc622806d 100644
--- a/core/tests/switch_core/test_main_teams_app.py
+++ b/core/tests/switch_core/test_main_teams_app.py
@@ -84,6 +84,11 @@ async def test_the_apps_client_is_closed_only_after_the_bridges_stop(
def begin_shutdown(self) -> None:
calls.append("controllers told")
+ async def released() -> None:
+ # What `main` releases first: the work this process owns, handed
+ # over, then its connections and its lease.
+ calls.append("owned work handed over")
+
class _Exited(Exception):
pass
@@ -100,6 +105,9 @@ async def test_the_apps_client_is_closed_only_after_the_bridges_stop(
with pytest.raises(_Exited):
await _shutdown(
server=server,
+ begin_draining=lambda: None,
+ release=released,
+ drain_seconds=0.0,
controllers=_Controllers(), # type: ignore[arg-type]
client_lifecycle=_Stoppable("clients"), # type: ignore[arg-type]
collab_lifecycle=_Lifecycle(calls), # type: ignore[arg-type]
@@ -111,6 +119,9 @@ async def test_the_apps_client_is_closed_only_after_the_bridges_stop(
)
assert calls.index("bridges stopped") < calls.index("teams app closed")
+ # The work this process owns is handed over before anything it runs is
+ # stopped, so the other processes take it over at once.
+ assert calls.index("owned work handed over") < calls.index("bridges stopped")
assert server.should_exit is True
@@ -131,3 +142,71 @@ async def test_the_package_carries_the_environments_app_name() -> None:
assert manifest["name"]["short"] == "Agent Switch (dev)"
assert app is not None
await app.aclose()
+
+
+async def test_shutdown_drains_before_it_closes_anything(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ """Readiness fails and the lease is released before uvicorn stops
+ accepting, and with several replicas the process keeps serving for the
+ drain window while the Service moves its traffic elsewhere."""
+ calls: list[str] = []
+
+ class _Stoppable:
+ def __init__(self, name: str) -> None:
+ self._name = name
+
+ async def stop_all(self) -> None:
+ calls.append(f"{self._name} stopped")
+
+ async def close(self) -> None:
+ calls.append(f"{self._name} closed")
+
+ class _Controllers:
+ def begin_shutdown(self) -> None:
+ calls.append("controllers told")
+
+ class _Server:
+ def __setattr__(self, name: str, value: object) -> None:
+ if name == "should_exit" and value:
+ calls.append("sockets closing")
+ object.__setattr__(self, name, value)
+
+ class _Exited(Exception):
+ pass
+
+ def fake_exit(code: int) -> None:
+ raise _Exited
+
+ async def sleep(seconds: float) -> None:
+ calls.append(f"slept {seconds:g}")
+
+ async def release() -> None:
+ calls.append("lease released")
+
+ monkeypatch.setattr(main_module.os, "_exit", fake_exit)
+ monkeypatch.setattr(main_module.asyncio, "sleep", sleep)
+
+ with pytest.raises(_Exited):
+ await _shutdown(
+ server=_Server(), # type: ignore[arg-type]
+ begin_draining=lambda: calls.append("not ready"),
+ release=release,
+ drain_seconds=5.0,
+ controllers=_Controllers(), # type: ignore[arg-type]
+ client_lifecycle=_Stoppable("clients"), # type: ignore[arg-type]
+ collab_lifecycle=_Lifecycle(calls), # type: ignore[arg-type]
+ connector_lifecycle=_Stoppable("connectors"), # type: ignore[arg-type]
+ provisioning=_Stoppable("provisioning"), # type: ignore[arg-type]
+ discord_gateway=None,
+ discord_gateway_task=None,
+ teams_app=None,
+ )
+
+ assert calls[:5] == [
+ "not ready",
+ "controllers told",
+ "lease released",
+ "slept 5",
+ "sockets closing",
+ ]
diff --git a/core/tests/switch_core/test_outbound_fetch_sites.py b/core/tests/switch_core/test_outbound_fetch_sites.py
index 559aa1cbe..a2c3f9b72 100644
--- a/core/tests/switch_core/test_outbound_fetch_sites.py
+++ b/core/tests/switch_core/test_outbound_fetch_sites.py
@@ -47,6 +47,7 @@ from switch_core.db.stores.collaboration_bridge_store import CollaborationBridge
from switch_core.db.stores.room_store import RoomStore
from switch_core.keys import Keyring
from switch_core.outbound import OutboundPolicy, OutboundURLRefused
+from tests.ownership import SoleOwner
def _mattermost_config(url: str) -> dict[str, object]:
@@ -77,6 +78,8 @@ def _bridge_lifecycle(policy: OutboundPolicy) -> CollaborationBridgeLifecycleSer
session_activity_listener=MagicMock(),
session_activity_service=MagicMock(),
connections=MagicMock(),
+ ownership=SoleOwner(),
+ cursor_writer=SoleOwner.cursor_writer(MagicMock()),
)
service.register_adapter(
"mattermost", MattermostAdapter, MattermostConnectionConfig
@@ -146,6 +149,7 @@ class TestConnectorConfigs:
session_factory=MagicMock(),
keyring=Keyring.parse("test:" + "x" * 40, legacy_secret=None),
outbound_policy=OutboundPolicy.parse(""),
+ ownership=SoleOwner(),
)
service.register_connector_type(
"opencode", OpenCodeConnector, OpenCodeConnectionConfig
diff --git a/core/tests/switch_core/test_ownership.py b/core/tests/switch_core/test_ownership.py
new file mode 100644
index 000000000..925ff3ea0
--- /dev/null
+++ b/core/tests/switch_core/test_ownership.py
@@ -0,0 +1,539 @@
+"""One owner for each piece of work, among several processes on one database.
+
+Each "process" here is what `main` builds for one: its own lease in
+`switch_core_processes`, its own `Ownership` and its own work, all on the same
+database, so who owns what is decided only by the rows, as it is between real
+processes. Balances are run by hand instead of on the interval.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+from collections.abc import AsyncIterator, Callable
+from dataclasses import dataclass, field
+from datetime import timedelta
+from typing import Any
+
+import pytest
+from sqlalchemy import func, select, update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core import main as main_module
+from switch_core.db.models import OwnershipLease, SwitchCoreProcess
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+from switch_core.db.stores.switch_core_process_store import SwitchCoreProcessStore
+from switch_core.main import _release_process
+from switch_core.ownership import Fence, OwnedKind, Ownership, owner_of
+from switch_core.process_lease import LEASE_TTL, ProcessLease
+from switch_core.signals import SignalBus
+from tests.ownership import process_lease_on
+
+KIND = "work"
+KEYS = [f"key-{n}" for n in range(24)]
+
+
+def _never_listening() -> Any:
+ raise AssertionError("these processes' signal buses are never started")
+
+
+@dataclass
+class _Everywhere:
+ """Which processes run each key right now, across all of them, so a test
+ can say a key never ran in two at once."""
+
+ running: dict[str, set[str]] = field(default_factory=dict)
+ overlaps: list[str] = field(default_factory=list)
+
+
+@dataclass
+class _Process:
+ lease: ProcessLease
+ ownership: Ownership
+ kind: OwnedKind
+ started: list[Fence] = field(default_factory=list)
+ stopped: list[str] = field(default_factory=list)
+
+ @property
+ def process_id(self) -> str:
+ return self.lease.process_id
+
+ def runs(self) -> set[str]:
+ return self.kind.held()
+
+
+def _process(
+ session_factory: async_sessionmaker[AsyncSession], everywhere: _Everywhere
+) -> _Process:
+ lease = process_lease_on(session_factory)
+ process_id = lease.process_id
+ ownership = Ownership(
+ lease=lease,
+ session_factory=session_factory,
+ bus=SignalBus(session_factory, _never_listening, process_id=process_id),
+ store=OwnershipLeaseStore(),
+ processes=SwitchCoreProcessStore(),
+ )
+ holder: dict[str, _Process] = {}
+
+ async def start(key: str, fence: Fence) -> None:
+ holder["p"].started.append(fence)
+ running = everywhere.running.setdefault(key, set())
+ if running:
+ everywhere.overlaps.append(key)
+ running.add(process_id)
+
+ async def stop(key: str) -> None:
+ holder["p"].stopped.append(key)
+ everywhere.running.get(key, set()).discard(process_id)
+
+ kind = ownership.kind(KIND, start=start, stop=stop)
+ holder["p"] = _Process(lease=lease, ownership=ownership, kind=kind)
+ return holder["p"]
+
+
+def everywhere() -> _Everywhere:
+ return _Everywhere()
+
+
+async def two(
+ session_factory: async_sessionmaker[AsyncSession], everywhere: _Everywhere
+) -> AsyncIterator[tuple[_Process, _Process]]:
+ a = _process(session_factory, everywhere)
+ b = _process(session_factory, everywhere)
+ for process in (a, b):
+ await process.lease.renew()
+ for key in KEYS:
+ process.kind.want(key)
+ yield a, b
+
+
+async def _holders(session_factory: async_sessionmaker[AsyncSession]) -> dict[str, str]:
+ async with session_factory() as session:
+ rows = await session.execute(
+ select(OwnershipLease.key, OwnershipLease.process_id).where(
+ OwnershipLease.kind == KIND
+ )
+ )
+ return {key: process_id for key, process_id in rows.tuples().all() if process_id}
+
+
+async def _lapse(
+ session_factory: async_sessionmaker[AsyncSession], process: _Process
+) -> None:
+ """Make `process`'s lease read as lapsed to everyone: as if it had not
+ renewed for longer than the lease lives."""
+ async with session_factory() as session:
+ await session.execute(
+ update(SwitchCoreProcess)
+ .where(SwitchCoreProcess.id == process.process_id)
+ .values(beat_at=func.now() - LEASE_TTL - timedelta(seconds=1))
+ )
+ await session.commit()
+
+
+class TestOneOwner:
+ async def test_two_processes_split_the_keys_and_run_each_once(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ everywhere: _Everywhere,
+ ) -> None:
+ a, b = two
+
+ await a.ownership.balance()
+ await b.ownership.balance()
+
+ assert a.runs() | b.runs() == set(KEYS)
+ assert a.runs() & b.runs() == set()
+ assert a.runs() and b.runs()
+ assert everywhere.overlaps == []
+ holders = await _holders(session_factory)
+ assert {key for key, holder in holders.items() if holder == a.process_id} == (
+ a.runs()
+ )
+
+ async def test_each_runs_what_the_hash_gives_it(
+ self, two: tuple[_Process, _Process]
+ ) -> None:
+ a, b = two
+ await a.ownership.balance()
+ await b.ownership.balance()
+
+ live = sorted([a.process_id, b.process_id])
+ for key in KEYS:
+ owner = a if owner_of(KIND, key, live) == a.process_id else b
+ assert key in owner.runs()
+
+ async def test_one_process_owns_everything(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ everywhere: _Everywhere,
+ ) -> None:
+ only = _process(session_factory, everywhere)
+ await only.lease.renew()
+ for key in KEYS:
+ only.kind.want(key)
+
+ await only.ownership.balance()
+
+ assert only.runs() == set(KEYS)
+ assert [fence.token for fence in only.started] == [1] * len(KEYS)
+
+
+class TestHandingOver:
+ async def test_a_process_that_joins_takes_its_share_without_overlap(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ everywhere: _Everywhere,
+ ) -> None:
+ a = _process(session_factory, everywhere)
+ await a.lease.renew()
+ for key in KEYS:
+ a.kind.want(key)
+ await a.ownership.balance()
+ assert a.runs() == set(KEYS)
+
+ b = _process(session_factory, everywhere)
+ await b.lease.renew()
+ for key in KEYS:
+ b.kind.want(key)
+ # b's share is still a's: nothing is claimed from a live holder.
+ await b.ownership.balance()
+ assert b.runs() == set()
+ # a sees b and lets go of b's share; b then claims it.
+ await a.ownership.balance()
+ await b.ownership.balance()
+
+ live = sorted([a.process_id, b.process_id])
+ b_share = {key for key in KEYS if owner_of(KIND, key, live) == b.process_id}
+ assert b.runs() == b_share
+ assert a.runs() == set(KEYS) - b_share
+ assert set(a.stopped) == b_share
+ assert everywhere.overlaps == []
+ # Every claim moves the token on, so a's old holds are fenced off.
+ assert {fence.token for fence in b.started} == {2}
+
+ async def test_a_process_that_stops_hands_everything_over(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ everywhere: _Everywhere,
+ ) -> None:
+ a, b = two
+ await a.ownership.balance()
+ await b.ownership.balance()
+ a_had = a.runs()
+
+ # As `main` shuts a process down: its work, then its lease.
+ await a.ownership.stop()
+ await a.lease.stop()
+ await b.ownership.balance()
+
+ assert b.runs() == set(KEYS)
+ assert set(a.stopped) == a_had
+ assert everywhere.overlaps == []
+ assert await _holders(session_factory) == dict.fromkeys(KEYS, b.process_id)
+
+ async def test_a_process_that_stops_is_taken_over_on_its_last_signal(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ everywhere: _Everywhere,
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """Shut down as `main` does (`_release_process`). The other process
+ balances when it hears a signal, so the last one a sends must come
+ once a's lease reads as stopped, or b claims nothing until its next
+ interval."""
+ a, b = two
+ await a.ownership.balance()
+ await b.ownership.balance()
+ a_had = a.runs()
+ assert a_had
+
+ sent_with_lease_stopped: list[bool] = []
+ publish = a.ownership._bus.publish
+
+ def recording(kind: str, **fields: Any) -> None:
+ sent_with_lease_stopped.append(a.lease._stopped)
+ publish(kind, **fields)
+
+ monkeypatch.setattr(a.ownership._bus, "publish", recording)
+
+ class _NoConnections:
+ async def flush(self) -> None:
+ pass
+
+ await _release_process(
+ a.ownership,
+ _NoConnections(), # type: ignore[arg-type]
+ None,
+ a.lease,
+ )
+
+ assert sent_with_lease_stopped[-1:] == [True]
+ await b.ownership.balance() # on hearing it
+ assert b.runs() == set(KEYS)
+ assert everywhere.overlaps == []
+
+ async def test_work_that_would_not_stop_leaves_the_lease_to_lapse(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ everywhere: _Everywhere,
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """`main` releases twice as it shuts down, from the signal handler
+ and from the lifespan's teardown, in either order. The first gave up
+ on a stop that hung, so the work may still run here: the second must
+ not mark the lease stopped either, and the work passes on once the
+ lease lapses."""
+ a, b = two
+ await a.ownership.balance()
+ await b.ownership.balance()
+ a_had = a.runs()
+ assert a_had
+
+ async def hang(key: str) -> None:
+ await asyncio.Event().wait()
+
+ monkeypatch.setattr(a.kind, "_stop", hang)
+ monkeypatch.setattr(main_module, "_HAND_OVER_SECONDS", 0.05)
+
+ class _NoConnections:
+ async def flush(self) -> None:
+ pass
+
+ for _ in range(2):
+ await _release_process(
+ a.ownership,
+ _NoConnections(), # type: ignore[arg-type]
+ None,
+ a.lease,
+ )
+
+ assert not a.lease._stopped
+ await b.ownership.balance()
+ assert b.runs() == set(KEYS) - a_had
+ assert everywhere.overlaps == []
+
+ await _lapse(session_factory, a)
+ await b.ownership.balance()
+ assert b.runs() == set(KEYS)
+
+ async def test_a_process_that_died_is_taken_over_once_its_lease_lapses(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ ) -> None:
+ a, b = two
+ await a.ownership.balance()
+ await b.ownership.balance()
+ a_had = a.runs()
+
+ # a dies without a word. While its lease still holds, b waits.
+ await b.ownership.balance()
+ assert b.runs() == set(KEYS) - a_had
+
+ await _lapse(session_factory, a)
+ await b.ownership.balance()
+
+ assert b.runs() == set(KEYS)
+ assert a.stopped == []
+
+ async def test_a_process_unsure_of_its_lease_stops_its_work_first(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ everywhere: _Everywhere,
+ ) -> None:
+ """Its renewals have been failing: it stops at half the lease's
+ lifetime by its own clock, before anyone may read the lease as
+ lapsed and take its work."""
+ a, b = two
+ await a.ownership.balance()
+ await b.ownership.balance()
+ a_had = a.runs()
+
+ a.lease._renewed_at = time.monotonic() - LEASE_TTL.total_seconds()
+ await a.ownership.balance()
+ assert a.runs() == set()
+ assert set(a.stopped) == a_had
+
+ await _lapse(session_factory, a)
+ await b.ownership.balance()
+ assert b.runs() == set(KEYS)
+ assert everywhere.overlaps == []
+
+ # Renewing again, a gets its share back from b.
+ await a.lease.renew()
+ await a.ownership.balance()
+ await b.ownership.balance()
+ await a.ownership.balance()
+ assert a.runs() == a_had
+ assert everywhere.overlaps == []
+
+ async def test_work_dropped_here_is_stopped_and_let_go_of(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ ) -> None:
+ a, b = two
+ await a.ownership.balance()
+ key = sorted(a.runs())[0]
+
+ await a.kind.drop(key)
+
+ assert key in a.stopped
+ assert key not in a.runs()
+ assert key not in await _holders(session_factory)
+ # Not wanted here any more, so not claimed again either.
+ await a.ownership.balance()
+ assert key not in a.runs()
+
+ async def test_work_found_taken_over_is_stopped_here(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two: tuple[_Process, _Process],
+ ) -> None:
+ a, b = two
+ await a.ownership.balance()
+ key = sorted(a.runs())[0]
+ fence = a.kind.fence(key)
+ assert fence is not None
+
+ a.kind.lost(fence)
+ await a.ownership.balance()
+
+ assert key in a.stopped
+ # Still a's by the hash and free in the table, so a claims it again
+ # under a new token.
+ assert key in a.runs()
+ renewed = a.kind.fence(key)
+ assert renewed is not None and renewed.token == fence.token + 1
+
+
+class TestRendezvous:
+ def test_a_process_leaving_moves_only_its_own_keys(self) -> None:
+ keys = [f"k{n}" for n in range(500)]
+ three = ["p1", "p2", "p3"]
+ before = {key: owner_of(KIND, key, three) for key in keys}
+ after = {key: owner_of(KIND, key, ["p1", "p3"]) for key in keys}
+
+ moved = {key for key in keys if before[key] != after[key]}
+
+ assert moved == {key for key in keys if before[key] == "p2"}
+
+ def test_the_keys_spread_over_the_processes(self) -> None:
+ keys = [f"k{n}" for n in range(600)]
+ counts: dict[str, int] = {}
+ for key in keys:
+ owner = owner_of(KIND, key, ["p1", "p2", "p3"])
+ assert owner is not None
+ counts[owner] = counts.get(owner, 0) + 1
+
+ assert set(counts) == {"p1", "p2", "p3"}
+ assert min(counts.values()) > 120
+
+ def test_no_process_owns_nothing(self) -> None:
+ empty: Callable[[], list[str]] = list
+ assert owner_of(KIND, "k", empty()) is None
+
+
+class TestThisProcessLease:
+ async def test_a_renewed_lease_is_sure_until_half_its_lifetime(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ lease = process_lease_on(session_factory)
+ assert not lease.surely_held()
+
+ await lease.renew()
+ held_until = lease.held_until()
+ assert lease.surely_held()
+ assert held_until is not None
+ assert held_until - time.monotonic() <= LEASE_TTL.total_seconds() / 2
+
+ lease._renewed_at = time.monotonic() - LEASE_TTL.total_seconds() / 2
+ assert not lease.surely_held()
+
+ async def test_a_stopped_lease_is_not_held(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ lease = process_lease_on(session_factory)
+ await lease.renew()
+
+ await lease.stop()
+
+ assert not lease.surely_held()
+ assert lease.held_until() is None
+
+ async def test_work_claimed_at_boot_survives_a_slow_boot(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ everywhere: _Everywhere,
+ ) -> None:
+ """As `main` boots: `ownership.start()` renews and claims, then the
+ rest of the boot (the quiet copies, the room reconcile under the boot
+ lock) runs before the renewal loop is first scheduled."""
+ p = _process(session_factory, everywhere)
+ for key in KEYS:
+ p.kind.want(key)
+ await p.ownership.start()
+ assert p.runs() == set(KEYS)
+
+ await asyncio.sleep(4.0)
+ tasks = [
+ asyncio.create_task(p.lease.run()),
+ asyncio.create_task(p.ownership.run()),
+ ]
+ try:
+ # Past the half lease counted from the renewal at boot.
+ await asyncio.sleep(4.5)
+ assert p.stopped == []
+ assert p.runs() == set(KEYS)
+ finally:
+ for task in tasks:
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+
+ async def test_one_slow_renewal_does_not_stop_the_work(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ everywhere: _Everywhere,
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """One renewal held up for 2.6 s, as a pool checkout under load or a
+ slow commit would, and nothing else wrong."""
+ p = _process(session_factory, everywhere)
+ for key in KEYS:
+ p.kind.want(key)
+ await p.ownership.start()
+
+ store = p.lease._processes
+ real_renew = store.renew
+ renewals = 0
+
+ async def renew(*args: Any, **kwargs: Any) -> None:
+ nonlocal renewals
+ renewals += 1
+ if renewals == 1:
+ await asyncio.sleep(2.6)
+ await real_renew(*args, **kwargs)
+
+ monkeypatch.setattr(store, "renew", renew)
+ tasks = [
+ asyncio.create_task(p.lease.run()),
+ asyncio.create_task(p.ownership.run()),
+ ]
+ try:
+ await asyncio.sleep(9.0)
+ assert renewals >= 2
+ assert p.stopped == []
+ assert p.runs() == set(KEYS)
+ finally:
+ for task in tasks:
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
diff --git a/core/tests/switch_core/test_process_identity.py b/core/tests/switch_core/test_process_identity.py
new file mode 100644
index 000000000..4dec9ea71
--- /dev/null
+++ b/core/tests/switch_core/test_process_identity.py
@@ -0,0 +1,21 @@
+"""The name a switch-core process goes by."""
+
+from __future__ import annotations
+
+import socket
+
+from switch_core.process_identity import new_process_id
+
+
+def test_the_pod_name_leads_the_id() -> None:
+ assert new_process_id("switch-core-7d9f-x2k4").startswith("switch-core-7d9f-x2k4-")
+
+
+def test_outside_kubernetes_the_host_name_stands_in() -> None:
+ assert new_process_id(None).startswith(socket.gethostname() + "-")
+
+
+def test_two_lives_of_one_pod_are_told_apart() -> None:
+ """A pod restarted in place keeps its name; the lease and connections of
+ the process before must not read as the process after's."""
+ assert new_process_id("switch-core-0") != new_process_id("switch-core-0")
diff --git a/core/tests/switch_core/test_room_service_change_bridge.py b/core/tests/switch_core/test_room_service_change_bridge.py
index d4f941ab8..727099d1b 100644
--- a/core/tests/switch_core/test_room_service_change_bridge.py
+++ b/core/tests/switch_core/test_room_service_change_bridge.py
@@ -128,10 +128,15 @@ class _FakeCollaborationCore:
class _FakeLifecycle:
def __init__(self, bridges: dict[str, Any]) -> None:
self._bridges = bridges
+ # Bridges whose rooms changed, announced for the other processes.
+ self.rooms_changed_for: list[str] = []
def get(self, bridge_id: str) -> Any:
return self._bridges.get(bridge_id)
+ def rooms_changed(self, bridge_id: str) -> None:
+ self.rooms_changed_for.append(bridge_id)
+
class _FakeProvisioning:
def __init__(self, events: list[Any]) -> None:
@@ -235,6 +240,11 @@ class TestChangeBridge:
assert ("kick", "@bot-old:switch.local") in events
assert ("remove_room_mapping", "room-1") in events
assert ("add_room_mapping", "room-1", "chan-new") in events
+ # Both bridges' rooms changed, for every other process to map again.
+ assert svc._collab_lifecycle.rooms_changed_for == [ # type: ignore[attr-defined]
+ "bridge-new",
+ "bridge-old",
+ ]
# Ordering invariants: the new channel exists and is mapped before the
# old bridge is torn down.
diff --git a/core/tests/switch_core/test_room_service_default_bridge_multi_tenant.py b/core/tests/switch_core/test_room_service_default_bridge_multi_tenant.py
index dd1fb393f..bb28624fc 100644
--- a/core/tests/switch_core/test_room_service_default_bridge_multi_tenant.py
+++ b/core/tests/switch_core/test_room_service_default_bridge_multi_tenant.py
@@ -103,10 +103,15 @@ class _FakeCollaborationCore:
class _FakeLifecycle:
def __init__(self, bridges: dict[str, _FakeCollaborationCore]) -> None:
self._bridges = bridges
+ # Bridges whose rooms changed, announced for the other processes.
+ self.rooms_changed_for: list[str] = []
def get(self, bridge_id: str) -> Any:
return self._bridges.get(bridge_id)
+ def rooms_changed(self, bridge_id: str) -> None:
+ self.rooms_changed_for.append(bridge_id)
+
class _NoRunningClients:
"""No agents, no already-running system clients — keeps the test to
diff --git a/core/tests/switch_core/test_room_service_delete_room.py b/core/tests/switch_core/test_room_service_delete_room.py
index 40e557079..abb0542bd 100644
--- a/core/tests/switch_core/test_room_service_delete_room.py
+++ b/core/tests/switch_core/test_room_service_delete_room.py
@@ -46,10 +46,15 @@ class _FakeCollaborationCore:
class _FakeLifecycle:
def __init__(self, bridges: dict[str, Any]) -> None:
self._bridges = bridges
+ # Bridges whose rooms changed, announced for the other processes.
+ self.rooms_changed_for: list[str] = []
def get(self, bridge_id: str) -> Any:
return self._bridges.get(bridge_id)
+ def rooms_changed(self, bridge_id: str) -> None:
+ self.rooms_changed_for.append(bridge_id)
+
class _FakeClientLifecycle:
def __init__(self, clients: dict[str, Any]) -> None:
@@ -121,6 +126,8 @@ class TestDeleteRoom:
assert room_store.deleted == ["room-1"]
assert ("delete_room", "!mx:switch.local") in events
assert ("remove_room_mapping", "room-1", "!mx:switch.local") in events
+ # And every other process unmaps it too.
+ assert svc._collab_lifecycle.rooms_changed_for == ["bridge-x"] # type: ignore[attr-defined]
async def test_non_bridged_room_does_not_touch_collaboration_core(self) -> None:
events: list[Any] = []
diff --git a/core/tests/switch_core/test_room_service_tenant_bindings.py b/core/tests/switch_core/test_room_service_tenant_bindings.py
index c3e86142f..dc81072b0 100644
--- a/core/tests/switch_core/test_room_service_tenant_bindings.py
+++ b/core/tests/switch_core/test_room_service_tenant_bindings.py
@@ -66,6 +66,7 @@ from switch_core.db.stores.room_store import RoomStore
from switch_core.db.stores.tenant_store import TenantStore
from switch_core.room_service import RoomService
from switch_core.tenant_context import current_tenant_id, tenant_scope
+from tests.ownership import NeverOwned
pytestmark = pytest.mark.no_ambient_tenant
@@ -162,9 +163,9 @@ class _AgentRegistry:
The *system*-client half is the real service (`_registry` below), because
that lookup is where the cross-tenant bug lived and a stand-in would have
reimplemented the fix rather than exercised it. Resolving an agent to its
- running client is not tenant-sensitive — an agent id is unique across the
- deployment — and building real `AgentConsumer`s to say so would add a
- transport and a sync loop to a test about bindings.
+ running client is not tenant-sensitive (an agent id is unique across the
+ deployment), and building real `AgentActor`s to say so would add a
+ transport to a test about bindings.
"""
def __init__(self, agents: dict[str, _RunningClient]) -> None:
@@ -183,7 +184,7 @@ class _AgentRegistry:
class _StubClient:
"""A client that stays running until stopped, and touches nothing else.
- `_start_task` removes a client from the registry the moment `start`
+ `_start_task` removes a client from the registry the moment `connect`
returns, so a stub that finished immediately would empty the registry
these tests are about.
"""
@@ -194,18 +195,25 @@ class _StubClient:
self.display_name = record.display_name
self._stopped = asyncio.Event()
- async def start(self) -> None:
+ async def connect(self) -> None:
await self._stopped.wait()
- async def stop(self) -> None:
+ async def close(self) -> None:
self._stopped.set()
class _StubFactory:
- def create(self, record: Client) -> Any:
- # The stub reads for itself: it is the actor and its own consumer.
- client = _StubClient(record)
- return client, client
+ """The stub reads for itself: it is the actor and its own consumer, and
+ runs in every process."""
+
+ def create_actor(self, record: Client) -> Any:
+ return _StubClient(record)
+
+ def create_consumer(self, client_type: str, actor: Any) -> Any:
+ return actor
+
+ def has_consumer(self, client_type: str) -> bool:
+ return False
def _registry(
@@ -226,6 +234,8 @@ def _registry(
session_factory=session_factory,
config=SimpleNamespace(id_server_name="switch.local"), # type: ignore[arg-type]
tenants_isolated=True,
+ ownership=NeverOwned(), # type: ignore[arg-type]
+ cursor_writer=MagicMock(),
)
for tenant_id, client_id, transport_user_id in running:
service.start_client(
diff --git a/core/tests/switch_core/test_signals.py b/core/tests/switch_core/test_signals.py
new file mode 100644
index 000000000..665986565
--- /dev/null
+++ b/core/tests/switch_core/test_signals.py
@@ -0,0 +1,308 @@
+"""Signals between switch-core processes.
+
+Against real PostgreSQL, two buses on one database standing in for two
+processes: the point is that a `NOTIFY` sent by one reaches the other's
+`LISTEN`, which no stand-in can show.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+from typing import TYPE_CHECKING, Any
+
+import pytest
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core import signals as signals_module
+from switch_core.signals import SIGNAL_CHANNEL, SignalBus
+
+if TYPE_CHECKING:
+ from collections.abc import AsyncIterator, Callable
+
+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+DELIVERY_TIMEOUT = 10.0
+
+
+class Received:
+ def __init__(self) -> None:
+ self.signals: list[dict[str, Any]] = []
+ self._anything = asyncio.Event()
+
+ def __call__(self, signal: dict[str, Any]) -> None:
+ self.signals.append(signal)
+ self._anything.set()
+
+ async def wait(self) -> None:
+ await asyncio.wait_for(self._anything.wait(), DELIVERY_TIMEOUT)
+ self._anything.clear()
+
+
+def make_bus(
+ postgres_url: str, session_factory: async_sessionmaker[AsyncSession]
+) -> Callable[[], SignalBus]:
+ def _make() -> SignalBus:
+ return SignalBus(
+ session_factory,
+ lambda: create_async_engine(postgres_url, poolclass=NullPool),
+ process_id=str(uuid.uuid4()),
+ )
+
+ return _make
+
+
+async def two_buses(
+ make_bus: Callable[[], SignalBus],
+) -> AsyncIterator[tuple[SignalBus, SignalBus]]:
+ first, second = make_bus(), make_bus()
+ yield first, second
+ await first.stop()
+ await second.stop()
+
+
+async def _start(*buses: SignalBus) -> None:
+ for bus in buses:
+ await bus.start()
+ for bus in buses:
+ await asyncio.wait_for(bus.connected.wait(), DELIVERY_TIMEOUT)
+
+
+class TestBetweenProcesses:
+ async def test_a_signal_reaches_the_other_process(
+ self, two_buses: tuple[SignalBus, SignalBus]
+ ) -> None:
+ sender, receiver = two_buses
+ received = Received()
+ receiver.handle("thing.changed", received)
+ await _start(sender, receiver)
+
+ sender.publish("thing.changed", thing_id="t1", revision=3)
+ await received.wait()
+
+ assert received.signals == [
+ {
+ "kind": "thing.changed",
+ "from": sender.process_id,
+ "thing_id": "t1",
+ "revision": 3,
+ }
+ ]
+
+ async def test_the_sender_skips_its_own(
+ self, two_buses: tuple[SignalBus, SignalBus]
+ ) -> None:
+ sender, receiver = two_buses
+ own, other = Received(), Received()
+ sender.handle("thing.changed", own)
+ receiver.handle("thing.changed", other)
+ await _start(sender, receiver)
+
+ sender.publish("thing.changed", thing_id="t1")
+ await other.wait()
+
+ assert own.signals == []
+
+ async def test_signals_raised_together_all_arrive(
+ self, two_buses: tuple[SignalBus, SignalBus]
+ ) -> None:
+ sender, receiver = two_buses
+ received = Received()
+ receiver.handle("thing.changed", received)
+ await _start(sender, receiver)
+
+ for n in range(20):
+ sender.publish("thing.changed", n=n)
+ async with asyncio.timeout(DELIVERY_TIMEOUT):
+ while len(received.signals) < 20:
+ await received.wait()
+
+ assert [signal["n"] for signal in received.signals] == list(range(20))
+
+ async def test_a_kind_nobody_handles_is_ignored(
+ self, two_buses: tuple[SignalBus, SignalBus]
+ ) -> None:
+ sender, receiver = two_buses
+ received = Received()
+ receiver.handle("known", received)
+ await _start(sender, receiver)
+
+ sender.publish("unknown", x=1)
+ sender.publish("known", x=2)
+ await received.wait()
+
+ assert [signal["x"] for signal in received.signals] == [2]
+
+
+class TestResync:
+ async def test_connecting_runs_every_resync(
+ self, make_bus: Callable[[], SignalBus]
+ ) -> None:
+ bus = make_bus()
+ ran = asyncio.Event()
+
+ async def resync() -> None:
+ ran.set()
+
+ bus.on_resync(resync)
+ try:
+ await bus.start()
+ await asyncio.wait_for(ran.wait(), DELIVERY_TIMEOUT)
+ finally:
+ await bus.stop()
+
+
+class TestPublishing:
+ def test_an_oversized_signal_is_refused(
+ self, make_bus: Callable[[], SignalBus]
+ ) -> None:
+ with pytest.raises(ValueError, match="too large"):
+ make_bus().publish("thing.changed", blob="x" * 8000)
+
+ def test_a_kind_has_one_handler(self, make_bus: Callable[[], SignalBus]) -> None:
+ bus = make_bus()
+ bus.handle("thing.changed", lambda _signal: None)
+ with pytest.raises(ValueError, match="already registered"):
+ bus.handle("thing.changed", lambda _signal: None)
+
+
+class _FailingOnce:
+ """A session factory whose first session fails on the statement, as a
+ database that refuses for a moment does."""
+
+ def __init__(self, real: async_sessionmaker[AsyncSession]) -> None:
+ self._real = real
+ self.failures = 1
+
+ def __call__(self) -> Any:
+ if self.failures:
+ self.failures -= 1
+ return _RefusingSession()
+ return self._real()
+
+
+class _RefusingSession:
+ async def __aenter__(self) -> _RefusingSession:
+ return self
+
+ async def __aexit__(self, *_exc: object) -> None:
+ return None
+
+ async def execute(self, *_args: object, **_kwargs: object) -> None:
+ raise ConnectionError("the database refused")
+
+
+class TestNothingIsLost:
+ async def test_a_batch_that_fails_to_send_is_sent_again(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ make_bus: Callable[[], SignalBus],
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ monkeypatch.setattr(signals_module, "RETRY_SECONDS", 0.05)
+ sender = SignalBus(
+ _FailingOnce(session_factory), # type: ignore[arg-type]
+ lambda: create_async_engine(postgres_url, poolclass=NullPool),
+ process_id=str(uuid.uuid4()),
+ )
+ receiver = make_bus()
+ received = Received()
+ receiver.handle("thing.changed", received)
+ try:
+ await _start(sender, receiver)
+ sender.publish("thing.changed", n=1)
+ sender.publish("thing.changed", n=2)
+ async with asyncio.timeout(DELIVERY_TIMEOUT):
+ while len(received.signals) < 2:
+ await received.wait()
+ finally:
+ await sender.stop()
+ await receiver.stop()
+
+ assert [signal["n"] for signal in received.signals] == [1, 2]
+ assert sender.unsent == 0
+
+ async def test_every_recheck_runs_periodically(
+ self, make_bus: Callable[[], SignalBus], monkeypatch: pytest.MonkeyPatch
+ ) -> None:
+ monkeypatch.setattr(signals_module, "RECHECK_SECONDS", 0.02)
+ bus = make_bus()
+ runs: list[str] = []
+
+ async def failing() -> None:
+ runs.append("failing")
+ raise RuntimeError("one family's re-check failed")
+
+ async def working() -> None:
+ runs.append("working")
+
+ bus.on_recheck(failing)
+ bus.on_recheck(working)
+ try:
+ await bus.start()
+ async with asyncio.timeout(DELIVERY_TIMEOUT):
+ while runs.count("working") < 2:
+ await asyncio.sleep(0.01)
+ finally:
+ await bus.stop()
+
+ # A failing re-check does not stop the others, now or next time.
+ assert runs.count("failing") >= 2
+
+ async def test_work_that_fails_is_kept_and_tried_again(
+ self, make_bus: Callable[[], SignalBus], monkeypatch: pytest.MonkeyPatch
+ ) -> None:
+ monkeypatch.setattr(signals_module, "RETRY_SECONDS", 0.02)
+ bus = make_bus()
+ attempts: list[str] = []
+ done = asyncio.Event()
+
+ async def reread(key: str) -> None:
+ attempts.append(key)
+ if len(attempts) == 1:
+ raise ConnectionError("the database refused")
+ done.set()
+
+ work = bus.pending_work("things", reread)
+ try:
+ await bus.start()
+ work.add("t1")
+ work.add("t1")
+ await asyncio.wait_for(done.wait(), DELIVERY_TIMEOUT)
+ finally:
+ await bus.stop()
+
+ # Added twice before its turn, so one piece of work, retried once.
+ assert attempts == ["t1", "t1"]
+ assert work.pending == 0
+
+ async def test_a_signal_with_no_sender_reaches_every_process(
+ self,
+ two_buses: tuple[SignalBus, SignalBus],
+ session_factory: async_sessionmaker[AsyncSession],
+ ) -> None:
+ """What a database trigger sends: no process skips it as its own."""
+ first, second = two_buses
+ heard_first, heard_second = Received(), Received()
+ first.handle("thing.changed", heard_first)
+ second.handle("thing.changed", heard_second)
+ await _start(first, second)
+
+ async with session_factory() as session:
+ await session.execute(
+ text("SELECT pg_notify(:channel, :payload)"),
+ {
+ "channel": SIGNAL_CHANNEL,
+ "payload": '{"kind": "thing.changed", "thing_id": "t1"}',
+ },
+ )
+ await session.commit()
+ await heard_first.wait()
+ await heard_second.wait()
+
+ assert heard_first.signals == heard_second.signals
diff --git a/core/tests/switch_core/transport/test_handled_twice.py b/core/tests/switch_core/transport/test_handled_twice.py
new file mode 100644
index 000000000..e20ab18f5
--- /dev/null
+++ b/core/tests/switch_core/transport/test_handled_twice.py
@@ -0,0 +1,171 @@
+"""A row handled twice gets its replies written once.
+
+A consumer may be handed the same row twice: its owner stopped between acting
+on the row and recording that it had, or every process runs a copy of it.
+What the client writes while handling a row is numbered from the row, so the
+second handling finds the first one's writes and writes nothing new.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import Message
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.transport import InboundMessage, RoomRef, TransportHandlers
+from switch_core.transport.cursors import FromHead
+from switch_core.transport.derived_ids import derived_event_id, writes_derived_from
+from switch_core.transport.postgres import PostgresTransport
+from tests.switch_core.transport.test_postgres_transport import (
+ _FakeListener,
+ _make_client,
+ _make_room,
+ _settled,
+ _transport,
+ _watched_room,
+)
+
+
+def _replying(transport: PostgresTransport, replies: int) -> TransportHandlers:
+ """Handlers that answer every message from someone else with `replies`
+ messages of the client's own."""
+
+ async def on_message(room: RoomRef, event: InboundMessage) -> None:
+ if event.sender == transport.user_id:
+ return
+ for n in range(replies):
+ await transport.send_message(
+ room.room_id, f"reply {n}", sender_name="agent one", metered=False
+ )
+
+ return TransportHandlers(on_message=on_message)
+
+
+async def _bodies(
+ session_factory: async_sessionmaker[AsyncSession], room_id: str
+) -> list[str | None]:
+ async with session_factory() as session:
+ rows = (
+ await session.scalars(
+ select(Message).where(Message.room_id == room_id).order_by(Message.seq)
+ )
+ ).all()
+ return [row.body for row in rows if row.event_type == "m.room.message"]
+
+
+class TestTwoCopiesOfOneClient:
+ async def test_both_handle_the_row_and_each_reply_is_written_once(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """As when every process runs a copy of the client's consumer, each
+ reading the room for itself."""
+ async with session_factory() as session:
+ room_id, room, client_id, user_id = await _make_room(session)
+ speaker_id, speaker = await _make_client(session, "speaker")
+ await session.commit()
+
+ listener = _FakeListener()
+ copies = [
+ _transport(
+ session_factory, client_id=client_id, user_id=user_id, listener=listener
+ )
+ for _ in range(2)
+ ]
+ await copies[0].join_room(room)
+ tasks = []
+ for copy in copies:
+ copy.register_handlers(_replying(copy, replies=2))
+ tasks.append(
+ asyncio.create_task(copy.receive_forever(FromHead(MessageStore())))
+ )
+ await _watched_room(copy)
+ speaking = _transport(session_factory, client_id=speaker_id, user_id=speaker)
+
+ await speaking.send_message(room, "hello", sender_name="speaker", metered=False)
+ await listener.announce(room_id)
+ for copy in copies:
+ await _settled(copy)
+
+ assert await _bodies(session_factory, room_id) == [
+ "hello",
+ "reply 0",
+ "reply 1",
+ ]
+ for task in tasks:
+ task.cancel()
+
+ async def test_a_write_outside_any_row_is_a_fresh_one_each_time(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ async with session_factory() as session:
+ room_id, room, client_id, user_id = await _make_room(session)
+ await session.commit()
+ transport = _transport(session_factory, client_id=client_id, user_id=user_id)
+
+ first = await transport.send_message(
+ room, "same words", sender_name="a", metered=False
+ )
+ second = await transport.send_message(
+ room, "same words", sender_name="a", metered=False
+ )
+
+ assert first.event_id != second.event_id
+ assert await _bodies(session_factory, room_id) == ["same words"] * 2
+
+
+class TestWritesOnBehalfOfSomethingOutside:
+ async def test_a_post_delivered_twice_is_written_once(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """A messaging platform delivering one post again: what the post's
+ author writes for it is numbered from the post."""
+ async with session_factory() as session:
+ room_id, room, client_id, user_id = await _make_room(session)
+ await session.commit()
+ author = _transport(session_factory, client_id=client_id, user_id=user_id)
+
+ for _ in range(2):
+ with writes_derived_from(client_id, "bridge\x1fchannel\x1fpost-1"):
+ await author.send_message(
+ room, "from the platform", sender_name="a", metered=False
+ )
+
+ assert await _bodies(session_factory, room_id) == ["from the platform"]
+
+ async def test_another_client_writing_meanwhile_is_not_numbered(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ async with session_factory() as session:
+ room_id, room, client_id, user_id = await _make_room(session)
+ other_id, other = await _make_client(session, "other")
+ await session.commit()
+ writer = _transport(session_factory, client_id=other_id, user_id=other)
+
+ for _ in range(2):
+ with writes_derived_from(client_id, "bridge\x1fchannel\x1fpost-1"):
+ await writer.send_message(
+ room, "a notice", sender_name="b", metered=False
+ )
+
+ assert await _bodies(session_factory, room_id) == ["a notice"] * 2
+
+
+def test_the_id_depends_on_client_row_and_position() -> None:
+ one = derived_event_id("client", "$row", 0)
+
+ assert one == derived_event_id("client", "$row", 0)
+ assert one.startswith("sw_") and len(one) == len("sw_") + 32
+ assert (
+ len(
+ {
+ one,
+ derived_event_id("client", "$row", 1),
+ derived_event_id("client", "$other", 0),
+ derived_event_id("another client", "$row", 0),
+ }
+ )
+ == 4
+ )
diff --git a/core/tests/switch_core/transport/test_invites.py b/core/tests/switch_core/transport/test_invites.py
index dc13bbe6e..cc12da4ba 100644
--- a/core/tests/switch_core/transport/test_invites.py
+++ b/core/tests/switch_core/transport/test_invites.py
@@ -29,6 +29,7 @@ class _Listener:
def __init__(self) -> None:
self.rooms: list[str] = []
self.removed: list[str] = []
+ self.reconciled = 0
async def on_invited(self, transport_room_id: str) -> None:
self.rooms.append(transport_room_id)
@@ -36,13 +37,16 @@ class _Listener:
async def on_removed(self, transport_room_id: str) -> None:
self.removed.append(transport_room_id)
+ async def reconcile(self) -> None:
+ self.reconciled += 1
+
async def test_the_newest_registration_is_the_one_that_hears() -> None:
bus = InviteBus()
old, new = _Listener(), _Listener()
- bus.register("client-1", old.on_invited, old.on_removed)
- bus.register("client-1", new.on_invited, new.on_removed)
+ bus.register("client-1", old.on_invited, old.on_removed, old.reconcile)
+ bus.register("client-1", new.on_invited, new.on_removed, new.reconcile)
assert await bus.invite("client-1", "!room:test") is True
assert (old.rooms, new.rooms) == ([], ["!room:test"])
@@ -51,10 +55,10 @@ async def test_the_newest_registration_is_the_one_that_hears() -> None:
async def test_a_superseded_listener_cannot_unregister_the_live_one() -> None:
bus = InviteBus()
old, new = _Listener(), _Listener()
- bus.register("client-1", old.on_invited, old.on_removed)
- bus.register("client-1", new.on_invited, new.on_removed)
+ bus.register("client-1", old.on_invited, old.on_removed, old.reconcile)
+ bus.register("client-1", new.on_invited, new.on_removed, new.reconcile)
- bus.unregister("client-1", old.on_invited, old.on_removed)
+ bus.unregister("client-1", old.on_invited, old.on_removed, old.reconcile)
assert await bus.invite("client-1", "!room:test") is True, (
"the superseded transport's teardown deafened the live client"
@@ -73,10 +77,10 @@ async def test_a_superseded_listener_cannot_unregister_the_live_removal() -> Non
"""
bus = InviteBus()
old, new = _Listener(), _Listener()
- bus.register("client-1", old.on_invited, old.on_removed)
- bus.register("client-1", new.on_invited, new.on_removed)
+ bus.register("client-1", old.on_invited, old.on_removed, old.reconcile)
+ bus.register("client-1", new.on_invited, new.on_removed, new.reconcile)
- bus.unregister("client-1", old.on_invited, old.on_removed)
+ bus.unregister("client-1", old.on_invited, old.on_removed, old.reconcile)
await bus.remove("client-1", "!room:test")
assert (old.removed, new.removed) == ([], ["!room:test"])
@@ -85,9 +89,13 @@ async def test_a_superseded_listener_cannot_unregister_the_live_removal() -> Non
async def test_the_live_listener_can_still_unregister_itself() -> None:
bus = InviteBus()
listener = _Listener()
- bus.register("client-1", listener.on_invited, listener.on_removed)
+ bus.register(
+ "client-1", listener.on_invited, listener.on_removed, listener.reconcile
+ )
- bus.unregister("client-1", listener.on_invited, listener.on_removed)
+ bus.unregister(
+ "client-1", listener.on_invited, listener.on_removed, listener.reconcile
+ )
assert await bus.invite("client-1", "!room:test") is False
await bus.remove("client-1", "!room:test")
@@ -97,7 +105,9 @@ async def test_the_live_listener_can_still_unregister_itself() -> None:
async def test_unregistering_a_client_nobody_registered_is_a_no_op() -> None:
bus = InviteBus()
listener = _Listener()
- bus.unregister("client-1", listener.on_invited, listener.on_removed)
+ bus.unregister(
+ "client-1", listener.on_invited, listener.on_removed, listener.reconcile
+ )
assert await bus.invite("client-1", "!room:test") is False
@@ -127,3 +137,21 @@ async def test_a_client_listening_only_for_invitations_is_reported(
await bus.remove("client-1", "!room:test")
assert "not for removals" in caplog.text
+
+
+async def test_a_reconcile_reaches_only_the_live_listener() -> None:
+ """How a membership change made in another process reaches this one's
+ copy of the client: through the same slot, given up the same way."""
+ bus = InviteBus()
+ old, new = _Listener(), _Listener()
+ bus.register("client-1", old.on_invited, old.on_removed, old.reconcile)
+ bus.register("client-1", new.on_invited, new.on_removed, new.reconcile)
+ bus.unregister("client-1", old.on_invited, old.on_removed, old.reconcile)
+
+ await bus.reconcile("client-1")
+ await bus.reconcile("client-2")
+
+ assert (old.reconciled, new.reconciled) == (0, 1)
+ assert bus.client_ids() == ["client-1"]
+ assert bus.is_listening("client-1")
+ assert not bus.is_listening("client-2")
diff --git a/core/tests/switch_core/transport/test_membership_signals.py b/core/tests/switch_core/transport/test_membership_signals.py
new file mode 100644
index 000000000..29e8b1275
--- /dev/null
+++ b/core/tests/switch_core/transport/test_membership_signals.py
@@ -0,0 +1,640 @@
+"""Room membership changes reach the copy of a client in every process.
+
+Against real PostgreSQL, two "processes" on one database: each with its own
+signal bus, invitation bus and transport for the same client, as every
+switch-core process runs its own copy of each client. A membership written
+through one must start or stop the room's delivery in the other, by way of the
+trigger on `client_rooms` and the signal it sends, which no stand-in can show.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import uuid
+from collections.abc import AsyncIterator, Callable
+from dataclasses import dataclass, field
+from typing import Any
+
+import pytest
+from sqlalchemy import delete, func, select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
+from sqlalchemy.pool import NullPool
+
+from switch_core.db.models import TENANT_ZERO_ID, Client, ClientRoom, Message, Room
+from switch_core.db.stores.client_store import ClientStore
+from switch_core.db.stores.media_store import MediaStore
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.usage_store import UsageStore
+from switch_core.provisioning.postgres import PostgresProvisioning
+from switch_core.signals import SignalBus
+from switch_core.transport import InboundMessage, TransportHandlers
+from switch_core.transport.cursors import FromHead
+from switch_core.transport.ephemeral import EphemeralBus
+from switch_core.transport.invites import InviteBus
+from switch_core.transport.membership_signals import MembershipSignals
+from switch_core.transport.observer import IgnoreParticipantMessages
+from switch_core.transport.postgres import DELIVERY_PAGE, PostgresTransport
+from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
+from tests.routing import router_on
+
+DELIVERY_TIMEOUT = 10.0
+
+
+class _Listener:
+ """Stands in for the message listener, so a test can say "the room
+ moved" to one process."""
+
+ def __init__(self) -> None:
+ self.wakers: dict[str, set[Any]] = {}
+
+ def subscribe(self, room_id: str, waker: Any) -> None:
+ self.wakers.setdefault(room_id, set()).add(waker)
+
+ def unsubscribe(self, room_id: str, waker: Any) -> None:
+ self.wakers.get(room_id, set()).discard(waker)
+
+ async def announce(self, room_id: str) -> None:
+ for waker in list(self.wakers.get(room_id, ())):
+ await waker(room_id)
+
+
+@dataclass
+class _Process:
+ """One switch-core process's copy of the client, and what it was handed."""
+
+ transport: PostgresTransport
+ listener: _Listener
+ invites: InviteBus
+ membership: MembershipSignals
+ bus: SignalBus
+ bodies: list[str] = field(default_factory=list)
+ invited_at: list[int] = field(default_factory=list)
+ removed_from: list[str] = field(default_factory=list)
+
+
+async def _eventually(check: Callable[[], bool]) -> None:
+ async with asyncio.timeout(DELIVERY_TIMEOUT):
+ while not check():
+ await asyncio.sleep(0.02)
+
+
+async def _settled(transport: PostgresTransport) -> None:
+ await _eventually(lambda: not transport._pending and not transport._delivering)
+
+
+async def room_and_client(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> tuple[str, str, str, str, str]:
+ """A room, an agent client that is a member of it, and a second client
+ that is not, which speaks in it. Returns (room id, transport room id,
+ client id, client user id, speaker id)."""
+ suffix = uuid.uuid4().hex[:8]
+ async with session_factory() as session:
+ client = Client(
+ transport_user_id=f"@agent-{suffix}:test",
+ display_name="agent one",
+ type="agent",
+ )
+ speaker = Client(
+ transport_user_id=f"@speaker-{suffix}:test",
+ display_name="speaker",
+ type="agent",
+ )
+ room = Room(
+ transport_room_id=f"!room-{suffix}:test", name="a room", description=""
+ )
+ session.add_all([client, speaker, room])
+ await session.flush()
+ await RoomStore().add_client(session, client.id, room.id)
+ await RoomStore().add_client(session, speaker.id, room.id)
+ await session.commit()
+ return (
+ room.id,
+ room.transport_room_id,
+ client.id,
+ client.transport_user_id,
+ speaker.id,
+ )
+
+
+async def two_processes(
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ room_and_client: tuple[str, str, str, str, str],
+) -> AsyncIterator[tuple[_Process, _Process]]:
+ _, _, client_id, user_id, _ = room_and_client
+ processes = [
+ _process(postgres_url, session_factory, client_id, user_id) for _ in range(2)
+ ]
+ tasks = [
+ asyncio.create_task(process.transport.receive_forever(FromHead(MessageStore())))
+ for process in processes
+ ]
+ for process in processes:
+ await process.bus.start()
+ for process in processes:
+ await asyncio.wait_for(process.bus.connected.wait(), DELIVERY_TIMEOUT)
+ await _eventually(lambda p=process: bool(p.transport._watching)) # type: ignore[misc]
+ yield processes[0], processes[1]
+ for task in tasks:
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+ for process in processes:
+ await process.bus.stop()
+
+
+def _process(
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ client_id: str,
+ user_id: str,
+) -> _Process:
+ listener = _Listener()
+ invites = InviteBus()
+ bus = SignalBus(
+ session_factory,
+ lambda: create_async_engine(postgres_url, poolclass=NullPool),
+ process_id=str(uuid.uuid4()),
+ )
+ transport = PostgresTransport(
+ user_id=user_id,
+ client_id=client_id,
+ tenant_id=TENANT_ZERO_ID,
+ display_name="agent one",
+ actor_role="agent",
+ session_factory=session_factory,
+ room_store=RoomStore(),
+ message_store=MessageStore(),
+ usage_store=UsageStore(),
+ media_store=MediaStore(),
+ listener=listener, # type: ignore[arg-type]
+ invites=invites,
+ ephemeral=EphemeralBus(),
+ room_cache=RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=MessageStore(),
+ limits=RoomCacheLimits(
+ max_bytes=1024 * 1024,
+ max_rooms=10,
+ max_rows_per_room=DELIVERY_PAGE,
+ max_age_seconds=60.0,
+ ),
+ page=DELIVERY_PAGE,
+ ),
+ message_observer=IgnoreParticipantMessages(),
+ router=router_on(session_factory),
+ )
+ process = _Process(
+ transport=transport,
+ listener=listener,
+ invites=invites,
+ membership=MembershipSignals(
+ invites=invites, bus=bus, router=router_on(session_factory)
+ ),
+ bus=bus,
+ )
+
+ async def _on_message(_room: Any, event: InboundMessage) -> None:
+ process.bodies.append(event.body)
+
+ async def _on_invite(room: Any, event: Any) -> None:
+ # What a consumer does with an invitation: join, which watches.
+ process.invited_at.append(event.timestamp)
+ await transport.join_room(room.room_id)
+
+ async def _on_removed(room: Any, _event: Any) -> None:
+ process.removed_from.append(room.room_id)
+
+ async def _ignore(*_args: Any) -> None:
+ return None
+
+ transport.register_handlers(
+ TransportHandlers(
+ on_message=_on_message,
+ on_media=_ignore,
+ on_member_event=_ignore,
+ on_custom_event=_ignore,
+ on_invite=_on_invite,
+ on_removed=_on_removed,
+ )
+ )
+ return process
+
+
+def _provisioning(
+ session_factory: async_sessionmaker[AsyncSession], invites: InviteBus
+) -> PostgresProvisioning:
+ return PostgresProvisioning(
+ session_factory=session_factory,
+ room_store=RoomStore(),
+ client_store=ClientStore(),
+ router=router_on(session_factory),
+ invites=invites,
+ )
+
+
+async def _is_member(
+ session_factory: async_sessionmaker[AsyncSession], client_id: str, room_id: str
+) -> bool:
+ async with session_factory() as session:
+ row = await session.get(
+ ClientRoom, {"client_id": client_id, "room_id": room_id}
+ )
+ return row is not None
+
+
+async def _arrivals(
+ session_factory: async_sessionmaker[AsyncSession], room_id: str, user_id: str
+) -> int:
+ async with session_factory() as session:
+ return int(
+ await session.scalar(
+ select(func.count())
+ .select_from(Message)
+ .where(
+ Message.room_id == room_id,
+ Message.sender_id == user_id,
+ Message.event_type == "m.room.member",
+ )
+ )
+ or 0
+ )
+
+
+def _kick_after_read(
+ transport: PostgresTransport,
+ kick: Callable[[], Any],
+ room_id: str,
+ kicked: list[bool],
+) -> None:
+ """Make the transport's next membership read that includes `room_id`
+ return what it read only after `kick` has committed."""
+ store = transport._room_store
+ real = store.memberships_of
+
+ async def read_then_kick(session: Any, client_id: str) -> Any:
+ memberships = await real(session, client_id)
+ if not kicked and any(m.room_id == room_id for m in memberships):
+ await kick()
+ kicked.append(True)
+ return memberships
+
+ store.memberships_of = read_then_kick # type: ignore[method-assign]
+
+
+async def _say(
+ session_factory: async_sessionmaker[AsyncSession],
+ speaker: PostgresTransport,
+ transport_room_id: str,
+ body: str,
+ *processes: _Process,
+ room_id: str,
+) -> None:
+ await speaker.send_message(transport_room_id, body, sender_name="s", metered=False)
+ for process in processes:
+ await process.listener.announce(room_id)
+ await _settled(process.transport)
+
+
+def _speaker(
+ session_factory: async_sessionmaker[AsyncSession], speaker_id: str
+) -> PostgresTransport:
+ return PostgresTransport(
+ user_id="@speaker:test",
+ client_id=speaker_id,
+ tenant_id=TENANT_ZERO_ID,
+ display_name="speaker",
+ actor_role="agent",
+ session_factory=session_factory,
+ room_store=RoomStore(),
+ message_store=MessageStore(),
+ usage_store=UsageStore(),
+ media_store=MediaStore(),
+ listener=_Listener(), # type: ignore[arg-type]
+ invites=InviteBus(),
+ ephemeral=EphemeralBus(),
+ room_cache=RoomDeliveryCache(
+ session_factory=session_factory,
+ message_store=MessageStore(),
+ limits=RoomCacheLimits(
+ max_bytes=1024 * 1024,
+ max_rooms=10,
+ max_rows_per_room=DELIVERY_PAGE,
+ max_age_seconds=60.0,
+ ),
+ page=DELIVERY_PAGE,
+ ),
+ message_observer=IgnoreParticipantMessages(),
+ router=router_on(session_factory),
+ )
+
+
+class TestAcrossProcesses:
+ async def test_a_member_removed_through_one_process_stops_hearing_the_room_in_the_other(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ room_and_client: tuple[str, str, str, str, str],
+ two_processes: tuple[_Process, _Process],
+ ) -> None:
+ room_id, transport_room_id, _, user_id, speaker_id = room_and_client
+ here, there = two_processes
+ speaker = _speaker(session_factory, speaker_id)
+ await _say(
+ session_factory,
+ speaker,
+ transport_room_id,
+ "while a member",
+ there,
+ room_id=room_id,
+ )
+ assert there.bodies == ["while a member"]
+
+ await _provisioning(session_factory, here.invites).kick_user(
+ transport_room_id, user_id
+ )
+ await _eventually(lambda: room_id not in there.transport._watching)
+ await _say(
+ session_factory,
+ speaker,
+ transport_room_id,
+ "after removal",
+ there,
+ room_id=room_id,
+ )
+
+ assert room_id not in here.transport._watching
+ assert there.bodies == ["while a member"]
+
+ async def test_a_member_added_through_one_process_hears_the_room_in_the_other(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ room_and_client: tuple[str, str, str, str, str],
+ two_processes: tuple[_Process, _Process],
+ ) -> None:
+ _, _, client_id, user_id, speaker_id = room_and_client
+ here, there = two_processes
+ async with session_factory() as session:
+ room = Room(
+ transport_room_id=f"!second-{uuid.uuid4().hex[:8]}:test",
+ name="another room",
+ description="",
+ )
+ session.add(room)
+ await session.flush()
+ await RoomStore().add_client(session, speaker_id, room.id)
+ await session.commit()
+ second_id, second_transport_id = room.id, room.transport_room_id
+
+ await _provisioning(session_factory, here.invites).invite_to_room(
+ second_transport_id, user_id
+ )
+ # The copy in the process that served the change joined itself.
+ assert second_id in here.transport._watching
+ await _eventually(lambda: second_id in there.transport._watching)
+ async with session_factory() as session:
+ membership = await session.get(
+ ClientRoom, {"client_id": client_id, "room_id": second_id}
+ )
+ assert membership is not None
+ joined_at_ms = int(membership.joined_at.timestamp() * 1000) # type: ignore[attr-defined]
+
+ await _say(
+ session_factory,
+ _speaker(session_factory, speaker_id),
+ second_transport_id,
+ "said after the join",
+ there,
+ room_id=second_id,
+ )
+
+ assert there.bodies == ["said after the join"]
+ # The other process filters the room from the row's own time, not
+ # from whenever its own clock heard of it.
+ assert there.invited_at == [joined_at_ms]
+
+ async def test_a_kick_during_the_other_process_recheck_sticks(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ room_and_client: tuple[str, str, str, str, str],
+ two_processes: tuple[_Process, _Process],
+ ) -> None:
+ """Added through one process and kicked through it while the other
+ is re-reading for the add. The other's join must not write back the
+ membership its read saw."""
+ _, _, client_id, user_id, speaker_id = room_and_client
+ here, there = two_processes
+ async with session_factory() as session:
+ room = Room(
+ transport_room_id=f"!second-{uuid.uuid4().hex[:8]}:test",
+ name="another room",
+ description="",
+ )
+ session.add(room)
+ await session.flush()
+ await RoomStore().add_client(session, speaker_id, room.id)
+ await session.commit()
+ second_id, second_transport_id = room.id, room.transport_room_id
+ kicked: list[bool] = []
+
+ async def kick() -> None:
+ await _provisioning(session_factory, here.invites).kick_user(
+ second_transport_id, user_id
+ )
+
+ _kick_after_read(there.transport, kick, second_id, kicked)
+
+ await _provisioning(session_factory, here.invites).invite_to_room(
+ second_transport_id, user_id
+ )
+ await _eventually(lambda: bool(kicked))
+ # Long enough for both processes to act on the signals that follow.
+ await asyncio.sleep(1.5)
+
+ assert not await _is_member(session_factory, client_id, second_id)
+ assert await _arrivals(session_factory, second_id, user_id) == 1
+ assert second_id not in there.transport._watching
+
+ async def test_a_member_put_back_before_the_other_process_re_reads_is_not_handed_what_was_said_while_out(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ room_and_client: tuple[str, str, str, str, str],
+ two_processes: tuple[_Process, _Process],
+ ) -> None:
+ """Removed and added back, both committed before the other process
+ re-reads, so its re-read finds a row for the room either way. The row
+ is a new membership: that process is told the client was removed, and
+ reads the room again as a new member rather than from where it had
+ got to, which would hand it what was said while the client was out."""
+ room_id, transport_room_id, client_id, _, speaker_id = room_and_client
+ _, there = two_processes
+ speaker = _speaker(session_factory, speaker_id)
+
+ # The other process re-reads under this lock, so holding it keeps
+ # both changes' signals waiting until both have committed.
+ async with there.transport._reading_rooms:
+ async with session_factory() as session:
+ await session.execute(
+ delete(ClientRoom).where(
+ ClientRoom.client_id == client_id,
+ ClientRoom.room_id == room_id,
+ )
+ )
+ await session.commit()
+ await _say(
+ session_factory,
+ speaker,
+ transport_room_id,
+ "said while out",
+ room_id=room_id,
+ )
+ async with session_factory() as session:
+ await RoomStore().add_client(session, client_id, room_id)
+ await session.commit()
+ await there.membership.reconcile_all()
+ await _say(
+ session_factory,
+ speaker,
+ transport_room_id,
+ "said after being put back",
+ there,
+ room_id=room_id,
+ )
+
+ assert there.removed_from == [transport_room_id]
+ assert there.bodies == ["said after being put back"]
+
+
+class TestRecheck:
+ async def test_a_removal_whose_signal_was_missed_is_caught_by_the_recheck(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ room_and_client: tuple[str, str, str, str, str],
+ ) -> None:
+ room_id, _, client_id, user_id, _ = room_and_client
+ # The bus is never started, so no signal reaches this process.
+ process = _process(postgres_url, session_factory, client_id, user_id)
+ task = asyncio.create_task(
+ process.transport.receive_forever(FromHead(MessageStore()))
+ )
+ try:
+ await _eventually(lambda: room_id in process.transport._watching)
+ async with session_factory() as session:
+ await session.execute(
+ delete(ClientRoom).where(ClientRoom.client_id == client_id)
+ )
+ await session.commit()
+
+ await process.membership.reconcile_all()
+
+ assert room_id not in process.transport._watching
+ finally:
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
+
+ async def test_a_room_being_joined_is_left_to_the_join(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ room_and_client: tuple[str, str, str, str, str],
+ ) -> None:
+ """The join watches from just before the client's own arrival. A
+ re-check that read the new row first and watched from the head would
+ make the client miss hearing itself join."""
+ room_id, _, client_id, user_id, _ = room_and_client
+ process = _process(postgres_url, session_factory, client_id, user_id)
+ task = asyncio.create_task(
+ process.transport.receive_forever(FromHead(MessageStore()))
+ )
+ try:
+ await _eventually(lambda: room_id in process.transport._watching)
+ process.transport._unwatch(next(iter(process.transport._watching.values())))
+ process.transport._joining.add(room_id)
+
+ await process.membership.reconcile_all()
+
+ assert process.invited_at == []
+ assert room_id not in process.transport._watching
+ finally:
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
+
+ async def test_a_recheck_during_the_first_read_hands_on_no_room_already_joined(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ room_and_client: tuple[str, str, str, str, str],
+ monkeypatch: pytest.MonkeyPatch,
+ ) -> None:
+ """The signal connection comes up, or a re-check runs, while the
+ client is still reading the rooms it is in. Until that read is done it
+ watches none of them, and a re-read in between would hand each one to
+ `on_invite` as if someone had just added the client to it."""
+ room_id, _, client_id, user_id, _ = room_and_client
+ process = _process(postgres_url, session_factory, client_id, user_id)
+ read = asyncio.Event()
+ finish = asyncio.Event()
+ joined_rooms = PostgresTransport.joined_rooms
+
+ async def held_open(transport: PostgresTransport) -> list[str]:
+ rooms = await joined_rooms(transport)
+ read.set()
+ await finish.wait()
+ return rooms
+
+ monkeypatch.setattr(PostgresTransport, "joined_rooms", held_open)
+ task = asyncio.create_task(
+ process.transport.receive_forever(FromHead(MessageStore()))
+ )
+ try:
+ await asyncio.wait_for(read.wait(), DELIVERY_TIMEOUT)
+ recheck = asyncio.create_task(process.membership.reconcile_all())
+ # Long enough for a re-read that does not wait for the first one
+ # to read the rows and hand the room on.
+ await asyncio.sleep(0.3)
+ finish.set()
+ await asyncio.wait_for(recheck, DELIVERY_TIMEOUT)
+ await _eventually(lambda: room_id in process.transport._watching)
+
+ assert process.invited_at == []
+ finally:
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
+
+ async def test_a_kick_during_a_recheck_sticks(
+ self,
+ postgres_url: str,
+ session_factory: async_sessionmaker[AsyncSession],
+ room_and_client: tuple[str, str, str, str, str],
+ ) -> None:
+ """The periodic re-check reads the rows, then a kick in the same
+ process commits and drops the room. The re-check finds the room it
+ read unwatched, and must not join it back."""
+ room_id, transport_room_id, client_id, user_id, _ = room_and_client
+ process = _process(postgres_url, session_factory, client_id, user_id)
+ task = asyncio.create_task(
+ process.transport.receive_forever(FromHead(MessageStore()))
+ )
+ try:
+ await _eventually(lambda: room_id in process.transport._watching)
+ kicked: list[bool] = []
+
+ async def kick() -> None:
+ await _provisioning(session_factory, process.invites).kick_user(
+ transport_room_id, user_id
+ )
+
+ _kick_after_read(process.transport, kick, room_id, kicked)
+
+ await process.membership.reconcile_all()
+
+ assert kicked
+ assert not await _is_member(session_factory, client_id, room_id)
+ assert await _arrivals(session_factory, room_id, user_id) == 0
+ assert room_id not in process.transport._watching
+ finally:
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
diff --git a/core/tests/switch_core/transport/test_postgres_transport.py b/core/tests/switch_core/transport/test_postgres_transport.py
index 9d0b81b0f..017698495 100644
--- a/core/tests/switch_core/transport/test_postgres_transport.py
+++ b/core/tests/switch_core/transport/test_postgres_transport.py
@@ -53,6 +53,7 @@ from switch_core.transport import (
TransportHandlers,
)
from switch_core.transport import postgres as postgres_module
+from switch_core.transport.cursors import FromHead
from switch_core.transport.ephemeral import EphemeralBus
from switch_core.transport.invites import InviteBus
from switch_core.transport.observer import (
@@ -63,6 +64,7 @@ from switch_core.transport.observer import (
from switch_core.transport.postgres import DELIVERY_PAGE, PostgresTransport
from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
from tests.conftest import RLSHarness
+from tests.routing import router_on
# The room cache `_transport` hands out. One per session factory, because that
# is what the process has: one, shared by every transport built on the same
@@ -248,6 +250,7 @@ def _transport(
ephemeral=ephemeral or EphemeralBus(),
room_cache=_room_cache(session_factory),
message_observer=message_observer or IgnoreParticipantMessages(),
+ router=router_on(session_factory),
)
@@ -827,6 +830,7 @@ class TestMedia:
ephemeral=EphemeralBus(),
room_cache=_room_cache(session_factory),
message_observer=IgnoreParticipantMessages(),
+ router=router_on(session_factory),
)
@@ -864,7 +868,7 @@ class TestReceiving:
)
transport.register_handlers(received.handlers())
await transport.join_room(transport_room_id)
- task = asyncio.create_task(transport.receive_forever())
+ task = asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
self._tasks.append((task, transport))
await _watched_room(transport)
return transport, listener, received, transport_room_id, user_id
@@ -920,7 +924,7 @@ class TestReceiving:
)
reader.register_handlers(received.handlers())
await reader.join_room(room)
- task = asyncio.create_task(reader.receive_forever())
+ task = asyncio.create_task(reader.receive_forever(FromHead(MessageStore())))
self._tasks.append((task, reader))
switch_room_id = await _watched_room(reader)
await listener.announce(switch_room_id)
@@ -1031,7 +1035,9 @@ class TestPresence:
)
watcher.register_handlers(received.handlers())
await watcher.join_room(room)
- self._tasks.append(asyncio.create_task(watcher.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(watcher.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(watcher)
sender = _transport(
@@ -1064,7 +1070,9 @@ class TestPresence:
)
watcher.register_handlers(received.handlers())
await watcher.join_room(room)
- self._tasks.append(asyncio.create_task(watcher.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(watcher.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(watcher)
sender = _transport(
@@ -1095,7 +1103,7 @@ class TestPresence:
)
watcher.register_handlers(received.handlers())
await watcher.join_room(room)
- task = asyncio.create_task(watcher.receive_forever())
+ task = asyncio.create_task(watcher.receive_forever(FromHead(MessageStore())))
self._tasks.append(task)
await _watched_room(watcher)
await watcher.close()
@@ -1138,7 +1146,9 @@ class TestOneClientCannotStallAnother:
)
transport.register_handlers(received.handlers())
await transport.join_room(room)
- self._tasks.append(asyncio.create_task(transport.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
+ )
switch_room_id = await _watched_room(transport)
await transport.send_message(
@@ -1171,7 +1181,9 @@ class TestOneClientCannotStallAnother:
)
transport.register_handlers(received.handlers())
await transport.join_room(room)
- self._tasks.append(asyncio.create_task(transport.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
+ )
switch_room_id = await _watched_room(transport)
await transport.send_message(
@@ -1216,7 +1228,9 @@ class TestJoiningARoomAlreadyRecorded:
)
transport.register_handlers(received.handlers())
await transport.join_room(room)
- self._tasks.append(asyncio.create_task(transport.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(transport)
# Somebody else records the membership — what a moderation invite does
@@ -1269,7 +1283,9 @@ class TestHearingYourOwnArrival:
)
transport.register_handlers(received.handlers())
await transport.join_room(room)
- self._tasks.append(asyncio.create_task(transport.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(transport)
await transport.join_room(later)
@@ -1305,7 +1321,9 @@ class TestHearingYourOwnArrival:
)
transport.register_handlers(received.handlers())
await transport.join_room(room)
- self._tasks.append(asyncio.create_task(transport.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(transport)
await transport.join_room(later)
@@ -1378,7 +1396,9 @@ class TestDeliveryBindsTheRoomsTenant:
)
transport.register_handlers(received.handlers())
await transport.join_room(transport_room_id)
- self._tasks.append(asyncio.create_task(transport.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
+ )
switch_room_id = await _watched_room(transport)
await transport.send_message(
@@ -1623,7 +1643,9 @@ class TestATransportActsInItsClientsTenant:
transport.register_handlers(received.handlers())
await transport.join_room(mxid_a)
await transport.join_room(mxid_b)
- self._tasks.append(asyncio.create_task(transport.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(transport)
await transport.send_message(
@@ -1772,7 +1794,9 @@ class TestWhatIsMeasured:
)
transport.register_handlers(handlers or _Received().handlers())
await transport.join_room(transport_room_id)
- self._tasks.append(asyncio.create_task(transport.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(transport)
return transport, listener, transport_room_id
@@ -1956,7 +1980,7 @@ class TestBeingRemovedFromARoom:
session_factory=session_factory,
room_store=RoomStore(),
client_store=ClientStore(),
- message_store=MessageStore(),
+ router=router_on(session_factory),
invites=invites,
)
@@ -2000,8 +2024,12 @@ class TestBeingRemovedFromARoom:
invites=invites,
)
member.register_handlers(_Received().handlers())
- self._tasks.append(asyncio.create_task(removed.receive_forever()))
- self._tasks.append(asyncio.create_task(member.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(removed.receive_forever(FromHead(MessageStore())))
+ )
+ self._tasks.append(
+ asyncio.create_task(member.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(removed)
await _watched_room(member)
@@ -2062,8 +2090,12 @@ class TestBeingRemovedFromARoom:
invites=invites,
)
member.register_handlers(_Received().handlers())
- self._tasks.append(asyncio.create_task(removed.receive_forever()))
- self._tasks.append(asyncio.create_task(member.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(removed.receive_forever(FromHead(MessageStore())))
+ )
+ self._tasks.append(
+ asyncio.create_task(member.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(removed)
await _watched_room(member)
@@ -2114,7 +2146,9 @@ class TestBeingRemovedFromARoom:
invites=invites,
)
removed.register_handlers(received.handlers(on_removed=_on_removed))
- self._tasks.append(asyncio.create_task(removed.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(removed.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(removed)
await self._provisioning(session_factory, invites).kick_user(room, leaving[1])
@@ -2161,8 +2195,12 @@ class TestBeingRemovedFromARoom:
invites=invites,
)
removed.register_handlers(_Received().handlers(on_message=_kick_on_the_first))
- self._tasks.append(asyncio.create_task(removed.receive_forever()))
- self._tasks.append(asyncio.create_task(member.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(removed.receive_forever(FromHead(MessageStore())))
+ )
+ self._tasks.append(
+ asyncio.create_task(member.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(removed)
await _watched_room(member)
@@ -2219,7 +2257,9 @@ class TestBeingRemovedFromARoom:
invites=invites,
)
removed.register_handlers(_Received().handlers())
- self._tasks.append(asyncio.create_task(removed.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(removed.receive_forever(FromHead(MessageStore())))
+ )
await asyncio.wait_for(kicked.wait(), timeout=5)
# Let the rest of the startup path run and decide what to subscribe
@@ -2260,14 +2300,16 @@ class TestBeingRemovedFromARoom:
invites=InviteBus(),
)
removed.register_handlers(_Received().handlers())
- self._tasks.append(asyncio.create_task(removed.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(removed.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(removed)
# A subscription dropped without its cursor: the shape the two maps
# take when anything writes one and not the other.
del removed._watching[room_id]
- await removed._watch(room)
+ await removed._watch(room, arrival_seq=None)
assert room_id in removed._watching
@@ -2319,8 +2361,12 @@ class TestBeingRemovedFromARoom:
removed.register_handlers(
_Received().handlers(on_message=_kick_on_the_page_boundary)
)
- self._tasks.append(asyncio.create_task(removed.receive_forever()))
- self._tasks.append(asyncio.create_task(member.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(removed.receive_forever(FromHead(MessageStore())))
+ )
+ self._tasks.append(
+ asyncio.create_task(member.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(removed)
await _watched_room(member)
@@ -2367,9 +2413,13 @@ class TestBeingRemovedFromARoom:
return t
old, new = _make(), _make()
- self._tasks.append(asyncio.create_task(old.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(old.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(old)
- self._tasks.append(asyncio.create_task(new.receive_forever()))
+ self._tasks.append(
+ asyncio.create_task(new.receive_forever(FromHead(MessageStore())))
+ )
await _watched_room(new)
# The older loop finishes unwinding after the newer one took the slot.
diff --git a/core/tests/switch_core/transport/test_room_delivery_cache.py b/core/tests/switch_core/transport/test_room_delivery_cache.py
index c2f10b304..adbc92bd4 100644
--- a/core/tests/switch_core/transport/test_room_delivery_cache.py
+++ b/core/tests/switch_core/transport/test_room_delivery_cache.py
@@ -28,10 +28,12 @@ from switch_core.db.stores.room_store import RoomStore
from switch_core.messages.notify import MessageListener
from switch_core.provisioning.postgres import PostgresProvisioning
from switch_core.transport import InboundMedia, InboundMessage
+from switch_core.transport.cursors import FromHead
from switch_core.transport.invites import InviteBus
from switch_core.transport.postgres import DELIVERY_PAGE, PostgresTransport
from switch_core.transport.room_cache import RoomCacheLimits, RoomDeliveryCache
from tests.conftest import RLSHarness
+from tests.routing import router_on
from tests.switch_core.transport.test_postgres_transport import (
_FakeListener,
_make_client,
@@ -119,7 +121,9 @@ class _Room:
transport.register_handlers(received.handlers())
room.members.append(transport)
room.received.append(received)
- room.tasks.append(asyncio.create_task(transport.receive_forever()))
+ room.tasks.append(
+ asyncio.create_task(transport.receive_forever(FromHead(MessageStore())))
+ )
for transport in room.members:
await _watched_room(transport)
return room
@@ -427,7 +431,7 @@ class TestMembershipIsStillEachClients:
session_factory=session_factory,
room_store=RoomStore(),
client_store=ClientStore(),
- message_store=MessageStore(),
+ router=router_on(session_factory),
invites=invites,
)
await provisioning.kick_user(room.transport_room_id, leaving.user_id)
@@ -469,7 +473,7 @@ class TestMembershipIsStillEachClients:
session_factory=session_factory,
room_store=RoomStore(),
client_store=ClientStore(),
- message_store=MessageStore(),
+ router=router_on(session_factory),
invites=invites,
)
for member in room.members:
@@ -552,8 +556,8 @@ class TestMembershipIsStillEachClients:
a.register_handlers(a_got.handlers())
b.register_handlers(b_got.handlers())
tasks = [
- asyncio.create_task(a.receive_forever()),
- asyncio.create_task(b.receive_forever()),
+ asyncio.create_task(a.receive_forever(FromHead(MessageStore()))),
+ asyncio.create_task(b.receive_forever(FromHead(MessageStore()))),
]
try:
await _watched_room(a)
@@ -624,7 +628,11 @@ class TestTheListenerIsNeverHeldUp:
]
for member in members:
member.register_handlers(_Received().handlers())
- tasks.append(asyncio.create_task(member.receive_forever()))
+ tasks.append(
+ asyncio.create_task(
+ member.receive_forever(FromHead(MessageStore()))
+ )
+ )
for member in members:
await _watched_room(member)
for member in members:
diff --git a/core/tests/switch_core/transport/test_start_statements.py b/core/tests/switch_core/transport/test_start_statements.py
index 26a4ed6c1..5df2416a0 100644
--- a/core/tests/switch_core/transport/test_start_statements.py
+++ b/core/tests/switch_core/transport/test_start_statements.py
@@ -3,22 +3,51 @@
At boot every client starts in all its rooms at once. Reading each room's id,
membership and head one at a time was over a thousand short sessions in a few
seconds on pilot, enough to drain the pool. The count must not grow with the
-number of rooms.
+number of rooms, whether the client reads from each room's head or resumes
+from cursors it stores, writing the ones it has none for.
"""
from __future__ import annotations
import asyncio
+from collections.abc import AsyncIterator, Awaitable, Callable
+import pytest
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.db.models import ClientRoom, Room
+from switch_core.db.stores.message_store import MessageStore
+from switch_core.transport.cursors import DeliveryProgress, FromHead
+from tests.ownership import start_holder
from tests.switch_core.statement_counts import StatementCounts
from tests.switch_core.transport.test_postgres_transport import (
_make_client,
_transport,
)
+Progress = Callable[[str], Awaitable[DeliveryProgress]]
+
+
[email protected](params=["from head", "stored cursors"])
+async def progress_for(
+ request: pytest.FixtureRequest,
+ session_factory: async_sessionmaker[AsyncSession],
+) -> AsyncIterator[Progress]:
+ """How a client is given its progress: read from each room's head, or
+ owned by this test's process and resumed from stored cursors."""
+ if request.param == "from head":
+
+ async def from_head(client_id: str) -> DeliveryProgress:
+ return FromHead(MessageStore())
+
+ yield from_head
+ return
+ holder = await start_holder(session_factory)
+ try:
+ yield holder.cursors
+ finally:
+ await holder.stop()
+
async def _client_in_rooms(
session_factory: async_sessionmaker[AsyncSession], rooms: int
@@ -43,13 +72,15 @@ async def _client_in_rooms(
async def _start(
session_factory: async_sessionmaker[AsyncSession],
counts: StatementCounts,
+ progress_for: Progress,
client_id: str,
user_id: str,
rooms: int,
) -> tuple[int, set[str]]:
transport = _transport(session_factory, client_id=client_id, user_id=user_id)
+ progress = await progress_for(client_id)
with counts.scope(f"start in {rooms}") as scope:
- task = asyncio.create_task(transport.receive_forever())
+ task = asyncio.create_task(transport.receive_forever(progress))
for _ in range(300):
if len(transport._cursors) == rooms:
break
@@ -63,15 +94,16 @@ async def _start(
async def test_starting_in_six_rooms_runs_as_many_statements_as_in_one(
session_factory: async_sessionmaker[AsyncSession],
statement_counts: StatementCounts,
+ progress_for: Progress,
) -> None:
one = await _client_in_rooms(session_factory, 1)
six = await _client_in_rooms(session_factory, 6)
count_one, watched_one = await _start(
- session_factory, statement_counts, *one[:2], 1
+ session_factory, statement_counts, progress_for, *one[:2], 1
)
count_six, watched_six = await _start(
- session_factory, statement_counts, *six[:2], 6
+ session_factory, statement_counts, progress_for, *six[:2], 6
)
assert watched_one == set(one[2])
@@ -82,6 +114,7 @@ async def test_starting_in_six_rooms_runs_as_many_statements_as_in_one(
async def test_a_room_left_before_its_membership_is_read_is_not_watched(
session_factory: async_sessionmaker[AsyncSession],
statement_counts: StatementCounts,
+ progress_for: Progress,
) -> None:
"""`_watch`'s guard, for the batch: the room list said member, the read
after the claim says not, so the claim is released."""
@@ -91,10 +124,11 @@ async def test_a_room_left_before_its_membership_is_read_is_not_watched(
real = transport._room_store.member_room_ids
async def _left_one(session, client, ids): # type: ignore[no-untyped-def]
- return (await real(session, client, ids)) - {gone}
+ members = await real(session, client, ids)
+ return {room: joined for room, joined in members.items() if room != gone}
transport._room_store.member_room_ids = _left_one # type: ignore[method-assign]
- task = asyncio.create_task(transport.receive_forever())
+ task = asyncio.create_task(transport.receive_forever(await progress_for(client_id)))
for _ in range(300):
if len(transport._cursors) == 2:
break
@@ -108,13 +142,15 @@ async def test_a_room_left_before_its_membership_is_read_is_not_watched(
async def test_a_failed_batch_read_releases_every_claim(
session_factory: async_sessionmaker[AsyncSession],
+ progress_for: Progress,
) -> None:
"""If the membership and head read fails, no room is left claimed with
nothing subscribed, so a retry starts from a clean slate."""
client_id, user_id, _ = await _client_in_rooms(session_factory, 3)
transport = _transport(session_factory, client_id=client_id, user_id=user_id)
+ transport._progress = await progress_for(client_id)
- async def _fail(*_args: object) -> set[str]:
+ async def _fail(*_args: object) -> dict[str, object]:
raise OSError("connection lost")
transport._room_store.member_room_ids = _fail # type: ignore[method-assign]
diff --git a/core/tests/switch_core/transport/test_stored_progress.py b/core/tests/switch_core/transport/test_stored_progress.py
new file mode 100644
index 000000000..da8712168
--- /dev/null
+++ b/core/tests/switch_core/transport/test_stored_progress.py
@@ -0,0 +1,322 @@
+"""A consumer one process owns reads from its stored cursors.
+
+Against real PostgreSQL, two "processes" on one database, each with its own
+process lease and cursor writer (`tests/ownership.py`), reading for the same
+client in turn: what the first finished, the second does not repeat, and what
+was written while neither read, the second delivers. A process that was taken
+over without knowing it is refused when it records progress.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections.abc import AsyncIterator
+from datetime import timedelta
+
+import pytest
+from sqlalchemy import func, select, update
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.db.models import Message, SwitchCoreProcess
+from switch_core.db.stores.delivery_cursor_store import DeliveryCursorStore
+from switch_core.db.stores.ownership_lease_store import OwnershipLeaseStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.observability.metrics import MetricsRegistry, install, uninstall
+from switch_core.process_lease import LEASE_TTL
+from switch_core.transport import InboundMembership, InboundMessage
+from switch_core.transport.cursors import BACKLOG_MAX_AGE
+from switch_core.transport.postgres import PostgresTransport
+from tests.ownership import Holder, start_holder
+from tests.switch_core.statement_counts import StatementCounts
+from tests.switch_core.transport.test_postgres_transport import (
+ _FakeListener,
+ _make_client,
+ _make_room,
+ _Received,
+ _transport,
+ _watched_room,
+)
+
+
+async def two_processes(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> AsyncIterator[tuple[Holder, Holder]]:
+ first = await start_holder(session_factory)
+ second = await start_holder(session_factory)
+ try:
+ yield first, second
+ finally:
+ await first.stop()
+ await second.stop()
+
+
+async def _member_room(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> tuple[str, str, str, str]:
+ """A room and a client that is a member of it, its cursor written with
+ the membership. Returns (room id, transport room id, client id, client
+ user id)."""
+ async with session_factory() as session:
+ room_id, transport_room_id, client_id, user_id = await _make_room(session)
+ await RoomStore().add_client(session, client_id, room_id)
+ await session.commit()
+ return room_id, transport_room_id, client_id, user_id
+
+
+async def _say(
+ session_factory: async_sessionmaker[AsyncSession],
+ transport_room_id: str,
+ body: str,
+) -> None:
+ """Write a message into the room, as someone other than the client."""
+ async with session_factory() as session:
+ client_id, user_id = await _make_client(session, "speaker")
+ await session.commit()
+ writer = _transport(session_factory, client_id=client_id, user_id=user_id)
+ await writer.send_message(transport_room_id, body, sender_name=body, metered=False)
+
+
+def _bodies(received: _Received) -> list[str]:
+ return [
+ event.body for event in received.events if isinstance(event, InboundMessage)
+ ]
+
+
+async def _read(
+ holder: Holder,
+ session_factory: async_sessionmaker[AsyncSession],
+ client_id: str,
+ user_id: str,
+) -> tuple[PostgresTransport, _FakeListener, _Received, asyncio.Task[None]]:
+ listener = _FakeListener()
+ received = _Received()
+ transport = _transport(
+ session_factory, client_id=client_id, user_id=user_id, listener=listener
+ )
+ transport.register_handlers(received.handlers())
+ task = await holder.receiving(transport)
+ await _watched_room(transport)
+ # Watching starts with a claim; the room is read from once its start is
+ # known and it is subscribed.
+ for _ in range(500):
+ if transport._cursors:
+ break
+ await asyncio.sleep(0.005)
+ return transport, listener, received, task
+
+
+async def _drained(transport: PostgresTransport) -> None:
+ for _ in range(500):
+ if not transport._pending and not transport._delivering:
+ return
+ await asyncio.sleep(0.005)
+ raise AssertionError("the transport never finished delivering")
+
+
+async def _stop(
+ transport: PostgresTransport, task: asyncio.Task[None], holder: Holder
+) -> None:
+ """Stop reading as an owner hands a client over: after the row in hand,
+ with its progress stored."""
+ await transport.close()
+ await task
+ await holder.writer.flush()
+
+
+class TestAHandOver:
+ async def test_the_next_owner_resumes_with_no_repeat_and_no_gap(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two_processes: tuple[Holder, Holder],
+ ) -> None:
+ room_id, room, client_id, user_id = await _member_room(session_factory)
+ first, second = two_processes
+
+ reader, listener, before, task = await _read(
+ first, session_factory, client_id, user_id
+ )
+ await _say(session_factory, room, "one")
+ await _say(session_factory, room, "two")
+ await listener.announce(room_id)
+ await _stop(reader, task, first)
+ async with session_factory() as session:
+ await OwnershipLeaseStore().release_all(session, first.process_id)
+ await session.commit()
+
+ # Said while nobody read for the client.
+ await _say(session_factory, room, "three")
+ await _say(session_factory, room, "four")
+
+ reader, listener, after, task = await _read(
+ second, session_factory, client_id, user_id
+ )
+ await _drained(reader)
+ await _say(session_factory, room, "five")
+ await listener.announce(room_id)
+
+ assert _bodies(before) == ["one", "two"]
+ assert _bodies(after) == ["three", "four", "five"]
+ await _stop(reader, task, second)
+
+ async def test_a_reader_taken_over_unawares_is_refused_and_told(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two_processes: tuple[Holder, Holder],
+ ) -> None:
+ room_id, room, client_id, user_id = await _member_room(session_factory)
+ first, second = two_processes
+ stale, stale_listener, _, stale_task = await _read(
+ first, session_factory, client_id, user_id
+ )
+ # The first process stops renewing, unawares, and the second takes the
+ # client over.
+ async with session_factory() as session:
+ await session.execute(
+ update(SwitchCoreProcess)
+ .where(SwitchCoreProcess.id == first.process_id)
+ .values(beat_at=func.now() - LEASE_TTL - timedelta(seconds=1))
+ )
+ await session.commit()
+ fresh, listener, received, task = await _read(
+ second, session_factory, client_id, user_id
+ )
+
+ await _say(session_factory, room, "hello")
+ await stale_listener.announce(room_id)
+ await listener.announce(room_id)
+ await first.writer.flush()
+ await second.writer.flush()
+
+ assert [fence.process_id for fence in first.lost] == [first.process_id]
+ assert second.lost == []
+ async with session_factory() as session:
+ places = await DeliveryCursorStore().positions(
+ session, client_id, [room_id]
+ )
+ head = await session.scalar(
+ select(func.max(Message.seq)).where(Message.room_id == room_id)
+ )
+ assert places == {room_id: head}
+ assert _bodies(received) == ["hello"]
+ stale_task.cancel()
+ await _stop(fresh, task, second)
+
+
+class TestResuming:
+ async def test_rows_too_old_to_act_on_are_stepped_over_and_counted(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two_processes: tuple[Holder, Holder],
+ ) -> None:
+ room_id, room, client_id, user_id = await _member_room(session_factory)
+ await _say(session_factory, room, "an hour ago")
+ await _say(session_factory, room, "just now")
+ async with session_factory() as session:
+ await session.execute(
+ update(Message)
+ .where(Message.room_id == room_id, Message.body == "an hour ago")
+ .values(sent_at=func.now() - BACKLOG_MAX_AGE - timedelta(hours=1))
+ )
+ await session.commit()
+ registry = MetricsRegistry()
+ install(registry)
+ try:
+ reader, _, received, task = await _read(
+ two_processes[0], session_factory, client_id, user_id
+ )
+ await _drained(reader)
+ counted = sum(
+ number.value
+ for payload in registry.collect()
+ if payload.name == "switch.messages.backlog_skipped"
+ for number in payload.numbers
+ )
+ finally:
+ uninstall()
+
+ assert _bodies(received) == ["just now"]
+ assert counted == 1
+ await _stop(reader, task, two_processes[0])
+
+ async def test_a_client_that_joins_hears_its_own_arrival(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two_processes: tuple[Holder, Holder],
+ ) -> None:
+ async with session_factory() as session:
+ room_id, room, client_id, user_id = await _make_room(session)
+ other_room, other, _, _ = await _make_room(session)
+ # Reading somewhere already, so the join is to a room it is
+ # watching for the first time.
+ await RoomStore().add_client(session, client_id, other_room)
+ await session.commit()
+ reader, listener, received, task = await _read(
+ two_processes[0], session_factory, client_id, user_id
+ )
+
+ await reader.join_room(room)
+ await _drained(reader)
+
+ arrivals = [
+ event
+ for event in received.events
+ if isinstance(event, InboundMembership) and event.state_key == user_id
+ ]
+ assert [event.room_id for event in arrivals] == [room]
+ await _stop(reader, task, two_processes[0])
+
+ async def test_a_room_left_is_not_taken_for_a_lost_lease(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ two_processes: tuple[Holder, Holder],
+ ) -> None:
+ room_id, room, client_id, user_id = await _member_room(session_factory)
+ holder = two_processes[0]
+ cursors = await holder.cursors(client_id)
+ async with session_factory() as session:
+ await RoomStore().remove_client(session, client_id, room_id)
+ await session.commit()
+
+ cursors.advance(room_id, 5)
+ await holder.writer.flush()
+
+ assert holder.lost == []
+
+
+class TestTheWriter:
+ async def test_many_rooms_are_written_in_as_few_statements_as_one(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ statement_counts: StatementCounts,
+ two_processes: tuple[Holder, Holder],
+ ) -> None:
+ holder = two_processes[0]
+
+ async def rooms_of_a_client(count: int) -> tuple[str, list[str]]:
+ async with session_factory() as session:
+ client_id, _ = await _make_client(session, f"many-{count}")
+ rooms = []
+ for _ in range(count):
+ room_id, _, _, _ = await _make_room(session)
+ await RoomStore().add_client(session, client_id, room_id)
+ rooms.append(room_id)
+ await session.commit()
+ return client_id, rooms
+
+ async def write(count: int) -> int:
+ client_id, rooms = await rooms_of_a_client(count)
+ cursors = await holder.cursors(client_id)
+ # The loop would race this test for the batch; it is stopped and
+ # the batch flushed by hand.
+ holder.task.cancel()
+ for room_id in rooms:
+ cursors.advance(room_id, 1)
+ with statement_counts.scope(f"write {count}") as scope:
+ await holder.writer.flush()
+ return scope.count
+
+ one = await write(1)
+ five = await write(5)
+
+ assert five == one
diff --git a/core/tests/test_migrate_entrypoint.py b/core/tests/test_migrate_entrypoint.py
index e448cd121..4449669d4 100644
--- a/core/tests/test_migrate_entrypoint.py
+++ b/core/tests/test_migrate_entrypoint.py
@@ -14,10 +14,13 @@ unreadable by the pods the release then rolls.
from __future__ import annotations
+import contextlib
import re
import tomllib
from pathlib import Path
+from switch_core import main
+
REPO_ROOT = Path(__file__).resolve().parents[2]
MIGRATE_JOB = (
REPO_ROOT / "deploy/remote/helm/switch/templates/switch-core/migrate-job.yaml"
@@ -69,3 +72,34 @@ def test_the_entry_point_resolves_to_a_callable() -> None:
module = __import__(module_path, fromlist=[attribute])
assert callable(getattr(module, attribute))
+
+
+async def test_an_older_release_finds_nothing_to_migrate_on_a_newer_schema(
+ monkeypatch,
+) -> None:
+ """During a rolling deploy or a rollback, the older release's migrate step
+ meets a head it has no file for. When only expand-only revisions lie
+ between, it exits cleanly instead of failing the pod's init container."""
+ calls: list[str] = []
+
+ @contextlib.asynccontextmanager
+ async def no_lock(config):
+ yield
+
+ async def ahead(config, alembic_cfg) -> bool:
+ return True
+
+ def upgrade(*args) -> None:
+ calls.append("upgrade")
+
+ async def grant(config) -> None:
+ calls.append("grant")
+
+ monkeypatch.setattr(main, "boot_lock", no_lock)
+ monkeypatch.setattr(main, "_schema_ahead", ahead)
+ monkeypatch.setattr(main.alembic_command, "upgrade", upgrade)
+ monkeypatch.setattr(main, "_prepare_database", grant)
+
+ await main._migrate_and_grant(object()) # type: ignore[arg-type]
+
+ assert calls == []
diff --git a/deploy/observability/dashboard.json b/deploy/observability/dashboard.json
index 1316b538f..41eeb3dfb 100644
--- a/deploy/observability/dashboard.json
+++ b/deploy/observability/dashboard.json
@@ -288,7 +288,7 @@
{
"definition": {
"type": "timeseries",
- "title": "Delivery and message-observer failures per second",
+ "title": "Delivery, routing and message-observer failures, and rows too old to replay, per second",
"show_legend": true,
"requests": [
{
@@ -298,12 +298,26 @@
"palette": "warm"
}
},
+ {
+ "q": "sum:switch.messages.routing_failures{$env,$service} by {stage}.as_rate()",
+ "display_type": "bars",
+ "style": {
+ "palette": "purple"
+ }
+ },
{
"q": "sum:switch.messages.observer_failures{$env,$service} by {actor}.as_rate()",
"display_type": "bars",
"style": {
"palette": "orange"
}
+ },
+ {
+ "q": "sum:switch.messages.backlog_skipped{$env,$service} by {actor}.as_rate()",
+ "display_type": "bars",
+ "style": {
+ "palette": "cool"
+ }
}
]
},
diff --git a/deploy/remote/helm/switch/README.md b/deploy/remote/helm/switch/README.md
index 0a1500f94..93c88b4d1 100644
--- a/deploy/remote/helm/switch/README.md
+++ b/deploy/remote/helm/switch/README.md
@@ -156,13 +156,21 @@ example), leave `tls.enabled` false and annotate.
## Constraints you cannot configure away
-**switch-core runs exactly one replica.** Four things live in the process
-rather than the database: the per-agent event buffer, the invite bus, the
-presence bus, and the message listener's subscriber registry. A second replica
-would hold its own copy of each, so a signal raised in one would never reach a
-client attached to the other. Bridge connections live in memory too. `switchCore.replicaCount` other than 1 fails the render,
-and the deployment strategy is `Recreate`, so upgrades have a brief outage
-rather than two pods fighting.
+**switch-core can run more than one replica.** `switchCore.replicaCount` sets
+how many. Shared state lives in the database: replicas signal each other through
+Postgres notifications, and each bridge's listener and outbound relay, each
+server connector and each command reader is owned by one replica at a time
+through a lease. Any replica can hold any client's connection. Two things are
+not single-owner and run on every replica: the shared Discord gateway
+connection of the distributed Discord app, and each Teams bridge's Graph
+subscriptions, which every replica adopts and renews.
+
+One replica upgrades with `Recreate`, so an upgrade has a brief outage. More
+than one rolls (`switchCore.rollingUpdate`, one extra pod at a time by
+default) and gets a PodDisruptionBudget (`switchCore.podDisruptionBudget`) and
+a soft spread across nodes (`switchCore.topologySpread`). On SIGTERM a pod
+hands its owned work to the others before it stops, inside
+`switchCore.terminationGracePeriodSeconds`.
**Bridges are created at runtime, not in values.** You register a collaboration
bridge in the gateway dashboard, with its credentials, after the chart is
diff --git a/deploy/remote/helm/switch/templates/_helpers.tpl b/deploy/remote/helm/switch/templates/_helpers.tpl
index 2f80cc2a3..884ed4a1f 100644
--- a/deploy/remote/helm/switch/templates/_helpers.tpl
+++ b/deploy/remote/helm/switch/templates/_helpers.tpl
@@ -442,9 +442,12 @@ preStop drain for a pod that rolls with a surge (a new pod is Ready before the
old one stops): hold the old pod for a few seconds after it is marked
terminating, so endpoint and load-balancer deregistration land before the
process gets SIGTERM. Not for a Recreate Deployment, where nothing else is
-serving and the wait only lengthens the outage. The native `sleep` action needs
-no shell (the images are distroless) but needs Kubernetes 1.30+, so nothing
-renders on an older cluster.
+serving and the wait only lengthens the outage. Not for switch-core either:
+with more than one replica it drains itself after SIGTERM, first handing the
+work it owns to the other pods and then serving on for a few seconds out of
+readiness, and a sleep here would only hold that hand-over back. The native
+`sleep` action needs no shell (the images are distroless) but needs Kubernetes
+1.30+, so nothing renders on an older cluster.
Usage: {{- include "switch.preStopDrain" . | nindent 10 }}
*/}}
{{- define "switch.preStopDrain" -}}
@@ -709,6 +712,17 @@ the owner's password.
{{- end }}
- name: DB_MIGRATION_LOCK_TIMEOUT
value: {{ .Values.postgresql.migrationLockTimeout | quote }}
+{{- /* The pod's own name names the process on its lease, its signals, its
+logs and its metrics, so one pod can be told from another. */}}
+- name: POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
+{{- /* How many replicas share the database: sizes the connection budget
+switch-core checks at boot, and decides whether a dead LISTEN connection takes
+a pod out of service. */}}
+- name: CORE_REPLICAS
+ value: {{ .Values.switchCore.replicaCount | quote }}
- name: AGENT_AUTH_CACHE_TTL_SECONDS
value: {{ .Values.switchCore.authCache.ttlSeconds | quote }}
- name: ID_SERVER_NAME
diff --git a/deploy/remote/helm/switch/templates/switch-core/deployment.yaml b/deploy/remote/helm/switch/templates/switch-core/deployment.yaml
index 02d4675f6..37767b6e1 100644
--- a/deploy/remote/helm/switch/templates/switch-core/deployment.yaml
+++ b/deploy/remote/helm/switch/templates/switch-core/deployment.yaml
@@ -1,10 +1,14 @@
-{{- if ne (int .Values.switchCore.replicaCount) 1 }}
-{{- fail "switchCore.replicaCount must be 1: switch-core keeps the per-agent event buffer, the invite bus, the presence bus and the message listener's subscriber registry in memory, so a second replica would never hear signals raised in the first." }}
-{{- end }}
{{- include "switch.validateDbTls" . }}
{{- include "switch.validateLogging" . }}
{{- include "switch.validateObservability" . }}
{{- include "switch.validateControllerRelease" . }}
+{{- $roll := .Values.switchCore.rollingUpdate }}
+{{- if not (has (toString $roll.maxSurge) (list "0" "1")) }}
+{{- fail (printf "switchCore.rollingUpdate.maxSurge must be 0 or 1, got %v: switch-core checks its database connection budget for at most one pod beyond replicaCount during a roll, so a larger surge could open more connections than Postgres allows." $roll.maxSurge) }}
+{{- end }}
+{{- if and (eq (toString $roll.maxSurge) "0") (eq (toString $roll.maxUnavailable) "0") }}
+{{- fail "switchCore.rollingUpdate.maxSurge and maxUnavailable are both 0, so a roll could never replace a pod. Set maxSurge to 1 (the default) or maxUnavailable above 0." }}
+{{- end }}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -14,12 +18,21 @@ metadata:
{{- include "switch.selectorLabels" (dict "Release" .Release "component" "switch-core") | nindent 4 }}
spec:
replicas: {{ .Values.switchCore.replicaCount }}
- # switch-core is a singleton (in-process delivery state + collaboration-bridge
- # sessions in memory). A rolling update would briefly run two pods whose bridge/client
- # sessions collide and leave bridges stuck; Recreate tears the old pod down
- # before starting the new one.
+ # Each bridge, connector and reader is held by one pod at a time through a
+ # lease in Postgres, so two pods side by side during a roll do not both run
+ # it: the old pod hands its work over on SIGTERM, and the new one takes it
+ # over. One replica still uses Recreate, because switch-core sizes its
+ # connection budget and its shutdown drain for a single replica being
+ # replaced rather than rolled (CORE_REPLICAS=1).
strategy:
+ {{- if gt (int .Values.switchCore.replicaCount) 1 }}
+ type: RollingUpdate
+ rollingUpdate:
+ maxSurge: {{ .Values.switchCore.rollingUpdate.maxSurge }}
+ maxUnavailable: {{ .Values.switchCore.rollingUpdate.maxUnavailable }}
+ {{- else }}
type: Recreate
+ {{- end }}
selector:
matchLabels:
{{- include "switch.selectorLabels" (dict "Release" .Release "component" "switch-core") | nindent 6 }}
@@ -32,6 +45,17 @@ spec:
{{- if .Values.global.imagePullSecrets }}
{{- include "switch.imagePullSecrets" . | nindent 6 }}
{{- end }}
+ terminationGracePeriodSeconds: {{ .Values.switchCore.terminationGracePeriodSeconds }}
+ {{- $spread := .Values.switchCore.topologySpread }}
+ {{- if and (gt (int .Values.switchCore.replicaCount) 1) $spread.enabled }}
+ topologySpreadConstraints:
+ - maxSkew: {{ $spread.maxSkew }}
+ topologyKey: {{ $spread.topologyKey }}
+ whenUnsatisfiable: {{ $spread.whenUnsatisfiable }}
+ labelSelector:
+ matchLabels:
+ {{- include "switch.selectorLabels" (dict "Release" .Release "component" "switch-core") | nindent 14 }}
+ {{- end }}
initContainers:
{{- include "switch.waitFor" (dict "name" "postgres" "host" (include "switch.postgresHost" .) "port" (include "switch.postgresPort" .)) | nindent 8 }}
# Migrates and re-issues grants as the schema owner, so the serving
@@ -93,11 +117,13 @@ spec:
{{- end }}
{{- include "switch.coreEnv" (merge (dict "omitOwnerCredentials" true) .) | nindent 12 }}
# Readiness checks the database; liveness deliberately does not.
- # There is one replica, so a failing readiness probe empties the
- # Service rather than shifting traffic — which is right while the
- # database is unreachable, and would be catastrophic as a restart
- # trigger. `/health` stays the cheap always-ok reply that liveness
- # and the other workloads' boot waits use.
+ # With one replica a failing readiness probe empties the Service
+ # rather than shifting traffic, which is right while the database is
+ # unreachable and would be catastrophic as a restart trigger. With
+ # several, traffic shifts to the pods still ready, and a pod also
+ # leaves readiness as it starts shutting down. `/health` stays the
+ # cheap always-ok reply that liveness and the other workloads' boot
+ # waits use.
readinessProbe:
httpGet:
path: /health/ready
diff --git a/deploy/remote/helm/switch/templates/switch-core/pdb.yaml b/deploy/remote/helm/switch/templates/switch-core/pdb.yaml
new file mode 100644
index 000000000..1a22aa7b2
--- /dev/null
+++ b/deploy/remote/helm/switch/templates/switch-core/pdb.yaml
@@ -0,0 +1,22 @@
+{{- $pdb := .Values.switchCore.podDisruptionBudget }}
+{{- if and (gt (int .Values.switchCore.replicaCount) 1) $pdb.enabled }}
+# Keeps a node drain or cluster upgrade from evicting every switch-core pod at
+# once. A pod evicted here hands its bridges, connectors and readers to the
+# pods still running, the same as on a roll.
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: {{ include "switch.fullname" . }}-switch-core
+ labels:
+ {{- include "switch.labels" . | nindent 4 }}
+ {{- include "switch.selectorLabels" (dict "Release" .Release "component" "switch-core") | nindent 4 }}
+spec:
+ {{- if $pdb.minAvailable }}
+ minAvailable: {{ $pdb.minAvailable }}
+ {{- else }}
+ maxUnavailable: {{ $pdb.maxUnavailable }}
+ {{- end }}
+ selector:
+ matchLabels:
+ {{- include "switch.selectorLabels" (dict "Release" .Release "component" "switch-core") | nindent 6 }}
+{{- end }}
diff --git a/deploy/remote/helm/switch/values.yaml b/deploy/remote/helm/switch/values.yaml
index a744625e9..4f8f8b6d8 100644
--- a/deploy/remote/helm/switch/values.yaml
+++ b/deploy/remote/helm/switch/values.yaml
@@ -14,8 +14,10 @@ global:
# the endpoint and any load balancer stop routing to it while its replacement
# is already serving, and the roll answers no requests with 502/503s. Only
# rendered on Kubernetes 1.30+ (the native `sleep` lifecycle action); 0 turns
- # it off. switch-core and Mattermost roll with Recreate, where there is no
- # replacement to drain to, so they get no hook.
+ # it off. Mattermost rolls with Recreate, where there is no replacement to
+ # drain to, so it gets no hook. switch-core gets none either: with more than
+ # one replica it drains itself after SIGTERM, leaving readiness and serving
+ # on for a few seconds once it has handed its work over.
preStopSleepSeconds: 5
# ── Ingress ─────────────────────────────────────────────────────────────────
@@ -120,6 +122,10 @@ secrets:
# each secret at least 32 characters. Every signing and encryption key is
# derived from these. To rotate, put a new entry first and keep the old one
# until a boot has re-encrypted stored values (docs/old/key-rotation.md).
+ # With more than one replica, rotate in two upgrades: first add the new entry
+ # last, then once every pod runs with it, move it first. A pod whose first
+ # key a running pod lacks refuses to start, since that pod could not read
+ # what it writes.
# Generate one with: echo "$(date +%Y%m):$(openssl rand -hex 32)"
secretKeys: ""
# Legacy: the single secret used before secretKeys. Keep it set through the
@@ -614,13 +620,52 @@ switchCore:
# for it, so this must match switch-core's own COLLABORATION_CALLBACK_PORT
# default.
port: 8081
- # switch-core MUST run as a single replica. Live state stays in the process
- # rather than the database: GitHub sign-in flows, the per-agent event buffer,
- # the invite and presence buses, and the message listener's subscriber registry. A
- # second replica would hold its own copy of each, so a signal raised in one
- # would never reach a client attached to the other. This is enforced at
- # render time; do not raise it.
+ # How many switch-core pods to run. Every pod serves the API and holds client
+ # connections. Each bridge's listener and outbound relay, each server
+ # connector and each command reader runs on one pod at a time, through
+ # leases in the database, and each message is routed to its agents as it is
+ # written, so any pod can take any request. Two things still run on every
+ # pod: the deployment's shared Discord gateway connection (the distributed
+ # Discord app), and each Teams bridge's Graph subscriptions, which every pod
+ # adopts and renews.
+ #
+ # One replica is replaced (Recreate): the old pod stops before the new one
+ # starts, so an upgrade has a short outage. More than one rolls (see
+ # `rollingUpdate`), and also gets a PodDisruptionBudget and a spread across
+ # nodes, below.
replicaCount: 1
+ # How a deployment of more than one replica rolls. switch-core sizes its
+ # database connection budget for at most one pod beyond `replicaCount` during
+ # a roll, so `maxSurge` must be 0 or 1. The default starts a new pod, waits
+ # for it to be ready, then stops an old one, so capacity never drops.
+ rollingUpdate:
+ maxSurge: 1
+ maxUnavailable: 0
+ # With more than one replica, keep voluntary disruptions (a node drain, a
+ # cluster upgrade) to one switch-core pod at a time. Set `minAvailable`
+ # instead to bound it from below; it wins when set. Never rendered at one
+ # replica, where any budget would either block every drain or allow nothing.
+ podDisruptionBudget:
+ enabled: true
+ maxUnavailable: 1
+ minAvailable: ""
+ # With more than one replica, prefer to put the pods on different nodes, so
+ # losing a node does not take every pod with it. Soft by default
+ # (ScheduleAnyway): a cluster with one node still schedules them all.
+ topologySpread:
+ enabled: true
+ maxSkew: 1
+ topologyKey: kubernetes.io/hostname
+ whenUnsatisfiable: ScheduleAnyway
+ # How long the kubelet waits after SIGTERM before killing the pod. On SIGTERM
+ # switch-core hands the bridges, connectors and readers it owns to the other
+ # pods (up to 10s), releases its lease (up to 1s), keeps serving for 5s while
+ # the Service stops routing to it (only with more than one replica), stops
+ # the bridges it still runs (giving one still starting up to 5s) and gives
+ # its teardown 3s. That is about 24s plus the time the bridges take to
+ # close; Kubernetes' default of 30s leaves too little margin, and a pod
+ # killed mid-hand-over leaves its work unowned until its 15s lease lapses.
+ terminationGracePeriodSeconds: 45
# Run DB migrations (alembic upgrade head) as a pre-upgrade hook Job, before
# the new pods roll, so a failed migration blocks the upgrade instead of
# crash-looping the new pods. The switch-core pod also migrates in an init
@@ -756,7 +801,12 @@ postgresql:
# past the server's TTL loses its connection — so under-sizing this shows up
# as fleet-wide agent disconnects, not as database errors.
# size + maxOverflow must stay under Postgres max_connections, remembering
- # Mattermost and Keycloak share the same instance.
+ # Mattermost and Keycloak share the same instance. The pool is per pod: every
+ # replica holds its own, plus four LISTEN connections (and one for the
+ # database sampler when metrics are exported), and a rolling deploy of
+ # several replicas runs one extra pod. switch-core adds this up at boot
+ # against max_connections; over budget, several replicas refuse to start and
+ # one replica logs a warning.
pool:
size: 30
maxOverflow: 10
diff --git a/docs/contributing/add-a-messaging-platform.md b/docs/contributing/add-a-messaging-platform.md
index 074fd3649..971be2a23 100644
--- a/docs/contributing/add-a-messaging-platform.md
+++ b/docs/contributing/add-a-messaging-platform.md
@@ -115,6 +115,29 @@ them.
https URLs. Console opens only http(s) links for platforms it has no
special handling for.
+### Works on several pods
+
+switch-core can run as several pods on one database. One pod holds each
+bridge (`core/switch_core/ownership.py`); the others run a quiet copy of it, to
+serve the requests and webhooks that reach them. So:
+
+- **Long-lived connections run only on the pod holding the lease.** Set
+ `keeps_quiet_unless_listening = True`, and when `self._listening` is False
+ (`set_listening`, called before `start`) open nothing the platform delivers
+ events over: no socket, no polling, no subscription kept alive. Only call the
+ platform's API. Webhooks are fine on any pod.
+- **Drop duplicates using the platform's event id.** Set `message_ref` to the
+ platform's own id for the post, the same on every delivery, and
+ `external_event_id` on webhooks (section 3). A second delivery then writes
+ nothing, whichever pod it reaches.
+- **Keep no state in one process that another pod needs.** Memory is lost when
+ the bridge moves to another pod, and the other pods never see it. Keep such
+ state in the database, or make it safe to lose.
+- **Write rows through the transport.** Hand what arrives to the callbacks
+ given to `start`, and never write to the database yourself: the router
+ decides each row as it is written (`core/switch_core/delivery/router.py`), and
+ a row written any other way reaches no agent.
+
## 3. Receiving events over webhooks (optional)
Some platforms can't keep a connection open to Switch: email, SMS, WhatsApp,
diff --git a/docs/contributing/new-messaging-platform.md b/docs/contributing/new-messaging-platform.md
index b3aea4782..ee3aa665f 100644
--- a/docs/contributing/new-messaging-platform.md
+++ b/docs/contributing/new-messaging-platform.md
@@ -13,6 +13,9 @@ request has to show. The technical details are in
for testing a tunnel such as ngrok is enough.
- **Have a test account on the platform** where you can create a bot or app
and add it to a chat. You'll need it to test.
+- **Make it work on several pods.** switch-core can run as several pods; read
+ [Works on several pods](add-a-messaging-platform.md#works-on-several-pods)
+ before you write the adapter.
## 1. Write the code
diff --git a/docs/old/api/AGENT_PROTOCOL.md b/docs/old/api/AGENT_PROTOCOL.md
index e07fab082..2977eb8d3 100644
--- a/docs/old/api/AGENT_PROTOCOL.md
+++ b/docs/old/api/AGENT_PROTOCOL.md
@@ -82,10 +82,17 @@ stream and it owns:
A connection belongs to exactly one agent. An agent may have several.
-The connection is held **in memory only**. It cannot outlive the server
-process (its socket and buffer cannot), so persisting it would recreate the
-staleness bug. Every question of the form "is this agent reachable in room R?"
-is answered from the live connection set, not from a table.
+The connection is held **in the memory of the process holding its socket**:
+its socket, heartbeat and the frames it is owed cannot be anywhere else. What
+other `switch-core` processes need to know about it (which process holds it,
+its current incarnation, its room claims and session placements) is written
+to `agent_connections` and the tables beside it when it opens, moves or
+closes, never per heartbeat, and each process keeps a copy of the others'
+live connections. A row counts only while the holding process's lease in
+`switch_core_processes` is held, so a connection still cannot outlive its
+process. Every question of the form "is this agent reachable in room R?" is
+answered from the live connection set, which includes those copies, not from a
+query per question.
### 2.2 Scope
diff --git a/docs/old/key-rotation.md b/docs/old/key-rotation.md
index 8d57f8526..9c9f7b12a 100644
--- a/docs/old/key-rotation.md
+++ b/docs/old/key-rotation.md
@@ -40,11 +40,12 @@ before the upgrade, exactly as they were made.
2. Deploy. On boot, switch-core re-encrypts every stored credential under the
new key and logs how many it rewrote. Sessions signed before the upgrade
stay valid until they expire, and buttons on existing Mattermost cards keep
- working. The Helm chart replaces switch-core rather than rolling it, so
- two versions never serve at once. Where they do (several replicas updated
- one by one), a replica still on the previous version cannot verify a
- session the new version signed, so someone who signs in mid-rollout may be
- asked to sign in once more.
+ working. At one replica the Helm chart replaces switch-core rather than
+ rolling it, so two versions never serve at once. Where they do (with
+ `switchCore.replicaCount` above 1 the chart updates replicas one by one), a
+ replica still on the previous version cannot verify a session the new
+ version signed, so someone who signs in mid-rollout may be asked to sign in
+ once more.
**This step is one-way.** Once the new version has booted, stored
credentials are encrypted in a format the previous version cannot read, so
@@ -63,27 +64,41 @@ continuing. That is a key removed too early: put it back.
## Rotating
Rotation takes two deploys, so that where replicas are updated one by one, no
-replica is handed a session signed with a key it does not have yet. With the
-Helm chart, which replaces switch-core in one step, step 1 is still what keeps
-the old key available for decrypting and verifying.
+replica is handed a session signed with a key it does not have yet. The Helm
+chart updates replicas one by one when `switchCore.replicaCount` is above 1.
+At one replica it replaces switch-core in one step, and step 1 is still what
+keeps the old key available for decrypting and verifying.
1. Generate a new key and add it **after** the current one:
`SECRET_KEYS=202610:<old>,202611:<new>`. Deploy. Every replica can now
verify and decrypt with the new key; nothing uses it yet.
2. Move the new key **first**: `SECRET_KEYS=202611:<new>,202610:<old>`.
Deploy. New sessions, signatures and encrypted values use the new key, and
- the boot re-encrypts stored credentials under it.
+ the boot re-encrypts stored credentials under it. Where replicas are
+ updated one by one, a replica not yet replaced keeps encrypting under the
+ old key after the new replicas' boots have run, so restart switch-core
+ once the roll has finished: that boot re-encrypts what they wrote.
3. Wait at least 24 hours, so sessions signed with the old key have expired.
4. Remove the old key and deploy. Buttons on Mattermost cards posted before
step 2 stop working; nothing else is affected.
+Step 1 is enforced. Each switch-core process records the ids of its keys on
+its lease, and a process whose first key a running process lacks refuses to
+start, naming that process. A single replica replaced in one step has no
+running process left to check, so it is not held back.
+
Rotate on a schedule, and at once if a key may have leaked. For a suspected
-leak, skip step 1 and the wait: deploy `<new>,<old>`, check the boot log
-reports the re-encryption, then deploy `<new>` alone straight away. Stored
-credentials need the old key once to be re-encrypted; until the second deploy,
-sessions signed with the old key still verify, so keep that window short.
-Removing the old key ends every session and stops every Mattermost button
-posted before, which is the point.
+leak, skip the wait: deploy `<old>,<new>`, deploy `<new>,<old>` once that has
+reached every replica, restart switch-core once that has too (where replicas
+are updated one by one), check the boot log reports the re-encryption, then
+deploy `<new>` alone straight away. Skip step 1 as well only where no
+switch-core process is still running when the first new one starts, as with a
+single replica replaced in one step. While an old process is still running,
+the new one refuses to start, and the roll stalls until you deploy step 1.
+Stored credentials need the old key once to be re-encrypted; until the last
+deploy, sessions signed with the old key still verify, so keep that window
+short. Removing the old key ends every session and stops every Mattermost
+button posted before, which is the point.
## Ids
diff --git a/docs/old/observability.md b/docs/old/observability.md
index bbcfc0b24..23eea617d 100644
--- a/docs/old/observability.md
+++ b/docs/old/observability.md
@@ -236,14 +236,23 @@ handled in a way that leaves no trace.
- **`switch.messages.delivery_failures`** — the delivery loop swallows one
room's exception so it cannot stop the others. Correct, and it means a room
that stopped delivering leaves only a log line.
+- **`switch.messages.backlog_skipped`**: a consumer resuming from its
+ stored cursor steps over rows older than its replay cap rather than answer
+ them late. The rows stay in the room; what they would have caused (a
+ command's reply, a platform post) never happens, and only this says so.
- **`switch.messages.send_failures`** — a send that raises before the commit
writes nothing, so the symptom is an absence, and an absence is what a quiet
room looks like too.
- **`switch.messages.observer_failures`** — reporting a message for product
- telemetry raised, after the transport committed it or while an agent's
- consumer was delivering it. Caught, because the message itself is fine: a
- send that raised after its commit would be retried and posted twice. So the
- only trace is a rate-limited log line.
+ telemetry raised, after the transport committed it or after the router
+ decided an agent was asked something in it. Caught, because the message
+ itself is fine: a send that raised after its commit would be retried and
+ posted twice. So the only trace is a rate-limited log line.
+- **`switch.messages.routing_failures`**: the router could not store one
+ agent's own event for a message it was writing, could not post a reply that
+ message owed (an offline notice, a refusal, a command's answer), or could not
+ decide a multi-file post it had given up waiting for. Each is caught so that
+ one agent cannot cost the room its message, so `stage` says which it was.
- **`switch.agent.events_dropped`** — an agent's buffer discards events when it
overflows or when they age out. The agent is *told* it missed them, so
nothing is hidden from the agent; nothing told anybody else.
diff --git a/docs/old/telemetry-events.md b/docs/old/telemetry-events.md
index fde594368..e90e0e679 100644
--- a/docs/old/telemetry-events.md
+++ b/docs/old/telemetry-events.md
@@ -260,8 +260,8 @@ reasons. Each is now answered rather than ignored:
sink now batches: events wait up to five seconds, or until two hundred are
queued, and go as one request. Its buffer is bounded, so a relay that is down
costs events, never memory.
-2. **The sender must not pay for it.** The transport and the agent consumer
- hand a small record to a queue and return. One worker looks up the room and
+2. **The sender must not pay for it.** The transport and the router hand a
+ small record to a queue and return. One worker looks up the room and
the sender — cached, so a busy room is read once every five minutes rather
than once per message — and emits. That queue is bounded too, and drops with
a warning when it falls behind.
@@ -1004,7 +1004,7 @@ Two things to set up first, or a third of the list cannot fire at all:
same agent and session emits nothing, across restarts, for 7 days.
**Messages** — `telemetry/messages.py`, fed by `transport/postgres.py` and
-`clients/agent_consumer.py`
+`delivery/recipient.py`
- `room_message_sent` — any person or agent posting in any room. A post with
three files is one event. A command reply or an offline notice is none.