Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Leases on the Discord gateway and Teams subscriptions (#880)
Abel Dantas2fab6c8cc397
diff --git a/core/switch_core/bridges/collaboration/adapter.py b/core/switch_core/bridges/collaboration/adapter.pyindex ab6a2bb46..79d34b171 100644--- a/core/switch_core/bridges/collaboration/adapter.py+++ b/core/switch_core/bridges/collaboration/adapter.py@@ -595,6 +595,9 @@ class PlatformAdapter(ABC):# default icons, which someone notices, rather than quietly sending# names to avatar services an operator has turned off.self._third_party_avatars = False+ # Set by the lifecycle (`set_holder_relay`): hands a request to the+ # process that holds the bridge, for work only the holder does.+ self._relay_to_holder: Callable[[dict[str, str]], None] | None = Nonedef set_max_attachment_bytes(self, max_bytes: int) -> None:self._max_attachment_bytes = max_bytes@@ -605,6 +608,31 @@ class PlatformAdapter(ABC):keep quiet ignores it (`keeps_quiet_unless_listening`)."""self._listening = listening+ def set_holder_relay(self, relay: Callable[[dict[str, str]], None]) -> None:+ """How a quiet copy hands the process holding the bridge a piece of+ work only the holder does, which reached this process instead: a+ webhook arrives at whichever process the load balancer picks. The+ holder receives it in `receive_relayed`."""+ self._relay_to_holder = relay++ def _hand_to_holder(self, request: dict[str, str]) -> None:+ """Hand `request` to the process holding the bridge. Nobody may hold+ it right now; the next holder catches up as it takes the bridge over,+ as it does for anything that happened while nobody held it."""+ if self._relay_to_holder is None:+ logger.warning(+ "No way to reach the process holding this bridge; dropping %r",+ request,+ )+ return+ self._relay_to_holder(request)++ async def receive_relayed(self, request: Mapping[str, str]) -> None:+ """Do a piece of work a quiet copy in another process handed here+ (`_hand_to_holder`), as this process holds the bridge. Nothing hands+ work to an adapter that does not override this."""+ logger.warning("Ignoring a relayed request this bridge has no use for")+def set_third_party_avatars(self, enabled: bool) -> None:"""Whether this adapter may hand platforms an icon drawn by an outsideavatar service (`agent_icon.is_third_party_avatar`)."""@@ -686,6 +714,17 @@ class PlatformAdapter(ABC):"""return None+ @classmethod+ def held_with(cls, connection_config: dict[str, object]) -> tuple[str, str] | None:+ """Other owned work this bridge must be held beside, as (kind, key) in+ `ownership.py`, or None for a bridge placed on its own.++ For a bridge whose platform events reach it in-process from something+ else that runs once per deployment, such as Discord's one shared+ Gateway connection: held anywhere else, it would never hear them.+ """+ return None+@classmethoddef outbound_urls(cls, connection_config: dict[str, object]) -> list[str]:"""The URLs in this config that Switch itself connects to.diff --git a/core/switch_core/bridges/collaboration/bridge_signals.py b/core/switch_core/bridges/collaboration/bridge_signals.pyindex 7f10aaba9..e5002ee33 100644--- a/core/switch_core/bridges/collaboration/bridge_signals.py+++ b/core/switch_core/bridges/collaboration/bridge_signals.py@@ -22,6 +22,11 @@ again (`reload_rooms`), so the one listening to the platform stops carryingthe channel into a room that no longer has it. A room bridged elsewhere isalso found on the spot when a message reaches a process that has not mappedit yet (`CollaborationCore._channel_for_room`).++Work only the holder does that reached a quiet copy instead, such as a Teams+webhook saying a Graph subscription needs renewing, is relayed to the holder+as `bridge.for_holder` (`PlatformAdapter.set_holder_relay`). Every process+hears it and only the one holding the bridge acts on it."""from __future__ import annotations@@ -36,6 +41,11 @@ if TYPE_CHECKING:BRIDGE_CHANGED_SIGNAL = "bridge.changed"BRIDGE_ROOMS_CHANGED_SIGNAL = "bridge.rooms_changed"+BRIDGE_FOR_HOLDER_SIGNAL = "bridge.for_holder"++# A relayed request as `PendingWork` keeps it: the bridge, then the request's+# fields in order, so the same request relayed twice is one piece of work.+_Relayed = tuple[str, tuple[tuple[str, str], ...]]class BridgeSignals:@@ -45,8 +55,11 @@ class BridgeSignals:self._bus = busself._owed = bus.pending_work("bridges", lifecycle.reconcile_bridge)self._rooms_owed = bus.pending_work("bridge-rooms", lifecycle.reload_rooms)+ self._lifecycle = lifecycle+ self._relayed = bus.pending_work("bridge-for-holder", self._receive_relayed)bus.handle(BRIDGE_CHANGED_SIGNAL, self._on_changed)bus.handle(BRIDGE_ROOMS_CHANGED_SIGNAL, self._on_rooms_changed)+ bus.handle(BRIDGE_FOR_HOLDER_SIGNAL, self._on_for_holder)bus.on_resync(lifecycle.reconcile_all)bus.on_recheck(lifecycle.reconcile_all)bus.on_resync(lifecycle.reload_all_rooms)@@ -63,6 +76,20 @@ class BridgeSignals:committed: every other process maps the bridge's rooms again."""self._bus.publish(BRIDGE_ROOMS_CHANGED_SIGNAL, bridge_id=bridge_id)+ def for_holder(self, bridge_id: str, request: dict[str, str]) -> None:+ """A quiet copy here has work for the process holding the bridge."""+ self._bus.publish(+ BRIDGE_FOR_HOLDER_SIGNAL, bridge_id=bridge_id, request=request+ )++ def _on_for_holder(self, signal: dict[str, Any]) -> None:+ request = signal["request"]+ self._relayed.add((signal["bridge_id"], tuple(sorted(request.items()))))++ async def _receive_relayed(self, relayed: _Relayed) -> None:+ bridge_id, request = relayed+ await self._lifecycle.receive_relayed(bridge_id, dict(request))+def _on_changed(self, signal: dict[str, Any]) -> None:self._owed.add(signal["bridge_id"])diff --git a/core/switch_core/bridges/collaboration/discord/adapter.py b/core/switch_core/bridges/collaboration/discord/adapter.pyindex a740f880c..fb22dab01 100644--- a/core/switch_core/bridges/collaboration/discord/adapter.py+++ b/core/switch_core/bridges/collaboration/discord/adapter.py@@ -12,6 +12,7 @@ from datetime import UTC, datetime, timedeltafrom typing import Any, ClassVar, Literalimport discord+from discord.http import Routefrom pydantic import Field, model_validatorfrom pydantic.json_schema import SkipJsonSchema@@ -102,6 +103,13 @@ ALLOWED_MESSAGE_TYPES = frozenset({discord.MessageType.default, discord.MessageType.reply})+# The one shared Gateway connection, as owned work (`ownership.py`): it runs+# in one switch-core process, and every shared-delivery bridge is held beside+# it (`DiscordAdapter.held_with`), since the events it receives reach a bridge+# in-process. One key, for the deployment's one Discord app.+DISCORD_GATEWAY = "discord-gateway"+DISCORD_GATEWAY_KEY = "app"+# Put on the message an agent is working on for as long as its turn lasts._REACTION: dict[ActivityMark, str] = {"working": "π", "queued": "β³"}@@ -506,6 +514,16 @@ class DiscordAdapter(PlatformAdapter):display_name: ClassVar[str] = "Discord"docs_slug: ClassVar[str | None] = "discord"+ # Quiet, a bridge with its own token logs in for the API alone and opens+ # no Gateway socket. A shared-delivery bridge opens none either way: it+ # listens where the shared connection runs (`held_with`).+ keeps_quiet_unless_listening: ClassVar[bool] = True++ @classmethod+ def held_with(cls, connection_config: dict[str, object]) -> tuple[str, str] | None:+ if connection_config.get("event_delivery") == "shared":+ return (DISCORD_GATEWAY, DISCORD_GATEWAY_KEY)+ return None@classmethoddef classify_failure(cls, exc: BaseException) -> FailureReason | None:@@ -657,10 +675,22 @@ class DiscordAdapter(PlatformAdapter):)return+ conn = self._require_connection()+ if not self._listening:+ # Another process holds the bridge and listens for it. This copy+ # serves the requests that reach it, over the API alone.+ await conn.login()+ logger.info(+ "Discord adapter logged in as %s (guild %s), quiet: another "+ "process listens",+ conn.client.user,+ self._config.guild_id,+ )+ return+# One guild, one handler; the DM handler is the same adapter so direct# messages still reach it. The connection routes each message by guild# id, which for a single-guild bridge is exactly the old filter.- conn = self._require_connection()conn.register_message_handler(self._guild_id, self._handle_message)conn.set_dm_handler(self._handle_message)conn.set_interaction_handler(self._make_on_interaction())@@ -2287,6 +2317,10 @@ class DiscordAdapter(PlatformAdapter):term = query.strip()if not term:return []+ if not self._listening:+ results = await self._search_members_by_api(term)+ results.sort(key=lambda u: u.display_name.lower())+ return resultsself._require_connection().take_member_search()guild = await self._get_guild()@@ -2314,6 +2348,41 @@ class DiscordAdapter(PlatformAdapter):results.sort(key=lambda u: u.display_name.lower())return results+ async def _search_members_by_api(self, term: str) -> list[DirectoryUser]:+ """The member search of a quiet copy, which has no Gateway socket to+ query members over: Discord's search endpoint, matching the same+ prefix of a username or nickname. discord.py has no call for it."""+ client = self._require_client()+ try:+ found = await client.http.request(+ Route(+ "GET",+ "/guilds/{guild_id}/members/search",+ guild_id=self._guild_id,+ ),+ params={"query": term, "limit": 100},+ )+ except discord.HTTPException as e:+ raise RuntimeError(f"Discord member search failed: {e}") from e+ results: list[DirectoryUser] = []+ for member in found:+ user = member.get("user") or {}+ if user.get("bot"):+ continue+ username = str(user.get("username", ""))+ results.append(+ DirectoryUser(+ external_user_id=str(user.get("id", "")),+ username=username,+ # Discord's own order for a member's display name.+ display_name=str(+ member.get("nick") or user.get("global_name") or username+ ),+ email=None,+ )+ )+ return results+async def create_dm_channel(self,*,@@ -2465,6 +2534,14 @@ class DiscordAdapter(PlatformAdapter):returnguild = await self._get_guild()+ if not self._listening:+ # The process listening for the bridge makes the role. Two making+ # it at once would make two, so this copy only learns it, for+ # rendering the agent's mentions.+ existing = discord.utils.get(guild.roles, name=agent_name)+ if existing is not None:+ self._agent_role_ids[agent_name.casefold()] = existing.id+ return# A role whose name is exactly the agent's is taken to be that agent's,# so a role made by hand β the only way to use this where the bot may# not manage roles β is adopted rather than duplicated. The match has to@@ -2515,6 +2592,11 @@ class DiscordAdapter(PlatformAdapter):"""if not self._agent_roles_available():return+ if not self._listening:+ # Deleted by the process listening for the bridge, which alone can+ # see who holds the role: this copy has no member list to check.+ self._agent_role_ids.pop(agent_name.casefold(), None)+ returnguild = await self._get_guild()role_id = self._agent_role_ids.pop(agent_name.casefold(), None)diff --git a/core/switch_core/bridges/collaboration/discord/connection.py b/core/switch_core/bridges/collaboration/discord/connection.pyindex 19e1bf430..384ae4b77 100644--- a/core/switch_core/bridges/collaboration/discord/connection.py+++ b/core/switch_core/bridges/collaboration/discord/connection.py@@ -178,6 +178,23 @@ class DiscordConnection:)self._member_searches.append(now)+ async def login(self) -> None:+ """Log in for Discord's API alone, opening no Gateway socket.++ For a switch-core process that serves requests touching Discord (post,+ make a channel, search members by API) while another process holds the+ socket: Discord delivers events to each socket opened, so only one+ process opens it. No handler is called and no command is synced here.+ `close` ends it as it ends a connection.+ """+ client = discord.Client(+ intents=self._intents, allowed_mentions=NO_MASS_MENTIONS+ )+ self._client = client+ await client.login(self._bot_token)+ assert client.user is not None+ self._bot_user_id = client.user.id+async def connect(self,*,diff --git a/core/switch_core/bridges/collaboration/discord/gateway.py b/core/switch_core/bridges/collaboration/discord/gateway.pyindex 972f15e48..4eb57c2d0 100644--- a/core/switch_core/bridges/collaboration/discord/gateway.py+++ b/core/switch_core/bridges/collaboration/discord/gateway.py@@ -7,9 +7,15 @@ bot token, opens the socket, and **binds no tenant** β each event is routed toguild's inert bridge, and every tenant binding happens below, per room.It is not a bridge and does not go through the collaboration lifecycle. It is a-single deployment-level object, started at boot alongside the bridges and living-in the one switch-core pod (a forced singleton), so there is never a second-owner of the socket and no leader election to arrange.+single deployment-level object in every switch-core process, and the socket is+owned work (`ownership.py`, kind `DISCORD_GATEWAY`): Discord delivers every+event to every socket the bot opens, so one process opens it. Every+shared-delivery bridge is held beside it (`DiscordAdapter.held_with`), since+the events it receives reach a bridge in-process. Every other process logs the+same connection in for Discord's API alone, so the quiet copies of the bridges+there can still serve the requests that reach them. The connection object is+the same one throughout, whichever it is doing, so a bridge attached to it+stays attached when this process takes the socket over or hands it on.It opens the shared socket with the right intents, routes each guild message,each global slash invocation and each press on a card's button to the bridge its@@ -22,7 +28,7 @@ from __future__ import annotationsimport asyncioimport loggingfrom collections.abc import Awaitable, Callable-from typing import Any+from typing import TYPE_CHECKING, Anyimport discord@@ -30,6 +36,8 @@ from switch_core.bridges.agent.commands import Command as InRoomCommandfrom switch_core.bridges.collaboration.adapter import PlatformAdapterfrom switch_core.bridges.collaboration.discord.adapter import (ALLOWED_MESSAGE_TYPES,+ DISCORD_GATEWAY,+ DISCORD_GATEWAY_KEY,DiscordAdapter,)from switch_core.bridges.collaboration.discord.connection import DiscordConnection@@ -41,6 +49,9 @@ from switch_core.bridges.collaboration.install_service import ()from switch_core.tenant_context import no_tenant+if TYPE_CHECKING:+ from switch_core.ownership import Fence, OwnedKind, Ownership+logger = logging.getLogger(__name__)_PLATFORM = "discord"@@ -68,9 +79,15 @@ class DiscordGatewayClient:# Fired once, after the first successful connect: boot walks the running# bridges and attaches the ones on a shared connection (see main.py).self._on_connected = on_connected- # Set once the first connect succeeds, and never cleared: discord.py- # reconnects a dropped socket on the same connection object.+ # Whether the connection is up, as the socket or for the API alone.+ # Cleared while this process switches between the two; a socket that+ # drops is discord.py's to reconnect, on the same connection object.self._live = False+ # The connect, or the log-in for the API alone, under way in the+ # background, retried until it succeeds.+ self._task: asyncio.Task[None] | None = None+ # The socket as owned work, once `own` has registered it.+ self._owned: OwnedKind | None = None# command_guild_id=None β commands register globally, once for the# application across every guild (decision #7); guild-scoped registration# is the self-registered adapter's, which serves one guild.@@ -140,26 +157,35 @@ class DiscordGatewayClient:)async def start_with_retry(self) -> None:- """Connect, retrying the initial connect with backoff, then attach.-- Run as a supervised background task so a configured-but-unreachable- Discord app never blocks or fails boot. Only the *initial* connect is- retried here β once it is up, discord.py reconnects transient drops on- its own (the same connection object, so attached bridges keep working).- On the first success `on_connected` fires, which walks the running+ """Connect, retrying the initial connect with backoff, then attach."""+ await self._bring_up(listen=True)++ async def _bring_up(self, *, listen: bool) -> None:+ """Open the socket, or log in for the API alone, retrying with backoff+ until it succeeds, then attach the bridges.++ Run in the background so a configured-but-unreachable Discord app+ never blocks or fails boot, or a take-over. Only the *initial* connect+ is retried here β once it is up, discord.py reconnects transient drops+ on its own (the same connection object, so attached bridges keep+ working). On success `on_connected` fires, which walks the runningbridges and attaches the ones on a shared connection."""delay = _INITIAL_RETRY_DELAYwhile True:try:- await self.start()+ if listen:+ await self.start()+ else:+ await self._connection.login()breakexcept asyncio.CancelledError:raiseexcept Exception:logger.exception(- "Discord shared Gateway connection failed to start; Discord "- "installs are inert until it connects. Retrying in %.0fs",+ "Discord shared %s failed to start; Discord installs are "+ "inert until it does. Retrying in %.0fs",+ "Gateway connection" if listen else "API log-in",delay,)await self._connection.close()@@ -171,8 +197,66 @@ class DiscordGatewayClient:self._live = Trueawait self._on_connected(self._connection)+ def own(self, ownership: Ownership) -> None:+ """Make the socket owned work: opened in the one process that holds+ it, and by the next once that one stops or dies. Before the+ ownership's first balance, so a process that holds it at boot opens+ it there and then."""+ self._owned = ownership.kind(+ DISCORD_GATEWAY, start=self._take_over, stop=self._hand_over+ )+ self._owned.want(DISCORD_GATEWAY_KEY)++ def start_quietly(self) -> None:+ """After the ownership's first balance: unless this process holds the+ socket, log the connection in for the API alone."""+ if self._task is None and (self._owned is None or not self._owned.held()):+ self._task = asyncio.create_task(+ self._bring_up(listen=False), name="discord-api-login"+ )++ async def _take_over(self, key: str, fence: Fence) -> None:+ """This process now holds the socket: open it in place of the log-in+ for the API alone. In the background, as at boot."""+ logger.info("Taking over the Discord shared Gateway connection")+ await self._close()+ self._task = asyncio.create_task(+ self._bring_up(listen=True), name="discord-gateway-start"+ )++ async def _hand_over(self, key: str) -> None:+ """This process no longer holds the socket: close it, so it delivers+ nothing more here before the next holder opens its own, then log in+ again for the API alone, unless the process is shutting down."""+ logger.info("Handing the Discord shared Gateway connection over")+ await self._close()+ if self._owned is not None and not self._owned.stopping:+ self._task = asyncio.create_task(+ self._bring_up(listen=False), name="discord-api-login"+ )++ async def _close(self) -> None:+ """Stop whatever is being brought up, and close the connection.++ Closed even when the bring-up failed after connecting, in+ `on_connected`, and its error is raised from here: the ownership logs+ a hand-over that raises and releases the socket's lease all the same,+ and the next holder opens a socket of its own.+ """+ task, self._task = self._task, None+ try:+ if task is not None:+ task.cancel()+ try:+ await task+ except asyncio.CancelledError:+ pass+ finally:+ self._live = False+ await self._connection.close()+async def stop(self) -> None:- await self._connection.close()+ await self._close()async def _on_guild_message(self, message: discord.Message) -> None:"""Route one guild message to the bridge its guild resolves to.diff --git a/core/switch_core/bridges/collaboration/lifecycle_service.py b/core/switch_core/bridges/collaboration/lifecycle_service.pyindex e67665dd6..72c180b93 100644--- a/core/switch_core/bridges/collaboration/lifecycle_service.py+++ b/core/switch_core/bridges/collaboration/lifecycle_service.py@@ -720,7 +720,7 @@ class CollaborationBridgeLifecycleService:logger.info("Wanting %d collaboration bridges", len(bridges))for bridge in bridges:- self._owned.want(bridge.id)+ self._want(bridge)async def start_quiet_copies(self) -> None:"""Start, quietly, every wanted bridge this process does not hold.@@ -983,7 +983,7 @@ class CollaborationBridgeLifecycleService:try:# Which process holds it is settled first, so it starts once, in# the form it will run here.- self._owned.want(bridge.id)+ self._want(bridge)await self._owned.settle()failed = self._failed_take_overs.pop(bridge.id, None)if failed is not None:@@ -1162,6 +1162,9 @@ class CollaborationBridgeLifecycleService:bridge_id, tenant_id, channel_id, team_id))+ adapter.set_holder_relay(+ lambda request: self._relay_to_holder(bridge_id, request)+ )adapter.set_max_attachment_bytes(self._config.agent_media_max_bytes)adapter.set_third_party_avatars(self._config.third_party_avatars_enabled)@@ -1729,10 +1732,45 @@ class CollaborationBridgeLifecycleService:self._quiet_starts[bridge_id] = asyncio.create_task(run())+ def _want(self, bridge: CollaborationBridge) -> None:+ """This process could hold `bridge`, beside whatever its platform+ says it must be held with (`PlatformAdapter.held_with`)."""+ adapter_cls = self._adapter_registry.get(bridge.type)+ held_with = (+ adapter_cls.held_with(bridge.connection_config or {})+ if adapter_cls is not None+ else None+ )+ self._owned.want(bridge.id, placed_with=held_with)+def holds(self, bridge_id: str) -> bool:"""Whether this process holds the bridge and listens for it."""return self._owned.fence(bridge_id) is not None+ def _relay_to_holder(self, bridge_id: str, request: dict[str, str]) -> None:+ if self._signals is None:+ logger.warning(+ "Collaboration bridge %s has work for its holder and no signal "+ "bus to send it on; dropping %r",+ bridge_id,+ request,+ )+ return+ self._signals.for_holder(bridge_id, request)++ async def receive_relayed(self, bridge_id: str, request: dict[str, str]) -> None:+ """Work a quiet copy of the bridge in another process relayed to its+ holder. Done here only if this process holds the bridge and its copy+ is running; otherwise it is another process's, or nobody's until the+ next holder catches up as it takes the bridge over."""+ if not self.holds(bridge_id):+ return+ running = self._running(bridge_id)+ if running is None or not running.active:+ return+ with no_tenant():+ await running.adapter.receive_relayed(request)+# ββ Changes made in other processes ββββββββββββββββββββββββββββββββββββββasync def reconcile_bridge(self, bridge_id: str) -> None:@@ -1797,7 +1835,7 @@ class CollaborationBridgeLifecycleService:if not self._owned.wants(bridge_id):# New here: which process holds it is settled first, so it starts# once, in the form it runs in.- self._owned.want(bridge_id)+ self._want(bridge)await self._owned.settle()if bridge_id in self._quiet_starts:# Starting in the background, from the row as it reads then.diff --git a/core/switch_core/bridges/collaboration/teams/adapter.py b/core/switch_core/bridges/collaboration/teams/adapter.pyindex 572b69ad0..dac5530c7 100644--- a/core/switch_core/bridges/collaboration/teams/adapter.py+++ b/core/switch_core/bridges/collaboration/teams/adapter.py@@ -767,6 +767,12 @@ class TeamsAdapter(PlatformAdapter):display_name: ClassVar[str] = "Microsoft Teams"docs_slug: ClassVar[str | None] = "microsoft-teams"+ # Graph delivers a channel's messages to every subscription made for it,+ # so only the process holding the bridge makes, adopts and renews them.+ # A quiet copy still receives activities and notifications, which reach+ # whichever process the load balancer picks, and relays to the holder+ # what would change its subscriptions (`receive_relayed`).+ keeps_quiet_unless_listening: ClassVar[bool] = True# Teams keeps only http(s) anchors: a link on any other scheme is stripped# whole, label and all, so a `switchdash://` deeplink left the literal@@ -1162,9 +1168,12 @@ class TeamsAdapter(PlatformAdapter):# Only once there is something to answer with: an activity taken# before then is acknowledged and lost, and Teams does not resend.await self._open_listener()+ # The approval is checked everywhere, so any copy can say it is gone.+ self._renewal_task = asyncio.create_task(self._renewal_loop())+ if not self._listening:+ returnasync with self._sub_lock:await self._adopt_existing_subscriptions()- self._renewal_task = asyncio.create_task(self._renewal_loop())self._repair_task = asyncio.create_task(self._repair_loop())async def _open_listener(self) -> None:@@ -1200,6 +1209,10 @@ class TeamsAdapter(PlatformAdapter):This is also why one Entra app must never serve two environments: eachwould read the other's subscriptions as stale and delete them.+ A quiet copy only learns which subscriptions are this bridge's, to+ check the notifications that reach it, and deletes none: they are the+ holder's to manage.+Until this has succeeded once, no subscription is made: one made for achannel whose live subscription could not be seen would be a second,delivering every message twice until the first runs out. A failure here@@ -1232,7 +1245,7 @@ class TeamsAdapter(PlatformAdapter):self._subscription_expiry[channel_id] = expirycontinuestale_id = str(sub.get("id", ""))- if not stale_id:+ if not stale_id or not self._listening:continuetry:await self._graph.delete_subscription(subscription_id=stale_id)@@ -1286,7 +1299,8 @@ class TeamsAdapter(PlatformAdapter):# with it renewal, silently and for good.try:await self._check_approval()- await self._renew_due_subscriptions()+ if self._listening:+ await self._renew_due_subscriptions()except Exception:logger.exception("Teams subscription renewal failed this round; trying again in %ss",@@ -3206,6 +3220,9 @@ class TeamsAdapter(PlatformAdapter):async def _switch_joined_team(self, team_id: str) -> None:"""Switch is in a team now: capture the channels waiting on that."""+ if not self._listening:+ self._hand_to_holder({"action": "team_joined", "team_id": team_id})+ returnself._team_has_switch[team_id] = (True, time.monotonic())waiting = [c for c, t in self._capture_parked.items() if t == team_id]for channel_id in waiting:@@ -3214,6 +3231,9 @@ class TeamsAdapter(PlatformAdapter):async def _switch_left_team(self, team_id: str) -> None:"""Switch is out of a team: stop capturing it, ready to resume on return."""+ if not self._listening:+ self._hand_to_holder({"action": "team_left", "team_id": team_id})+ returnself._team_has_switch[team_id] = (False, time.monotonic())known = (self._capture_wanted | set(self._subscriptions) | set(self._team_of_channel)@@ -4032,7 +4052,19 @@ class TeamsAdapter(PlatformAdapter):Wanted before it is checked for, so a channel whose subscription wasadopted at start, and later lost, is still one the repair loop owes.++ A quiet copy hands the channel, and the team it knows it is in, to the+ process holding the bridge, which makes the subscription."""+ if not self._listening:+ self._hand_to_holder(+ {+ "action": "capture",+ "channel_id": channel_id,+ "team_id": self._capture_team(channel_id) or "",+ }+ )+ returnself._capture_wanted.add(channel_id)if self._graph is None or self._withdrawing:return@@ -4254,7 +4286,11 @@ class TeamsAdapter(PlatformAdapter):answered, so they would be lost until the repair loop next reads them.An unknown one is a reason to read them now."""- if self._owns_subscription(subscription_id) or self._adopted:+ # A quiet copy reads them again for one it does not know, cooldown+ # permitting: the holder may have made it since this copy last read.+ if self._owns_subscription(subscription_id) or (+ self._adopted and self._listening+ ):return self._owns_subscription(subscription_id)async with self._sub_lock:now = time.monotonic()@@ -4262,7 +4298,7 @@ class TeamsAdapter(PlatformAdapter):self._adopt_on_demand_at is Noneor now - self._adopt_on_demand_at >= _ADOPT_ON_DEMAND_COOLDOWN_SECONDS)- if not self._adopted and due:+ if (not self._adopted or not self._listening) and due:# Spaced out, so a run of unknown ones β Graph failing, or# someone holding the organisation's clientState β is not one# listing each.@@ -4273,6 +4309,15 @@ class TeamsAdapter(PlatformAdapter):async def _handle_lifecycle_event(self, item: dict[str, Any]) -> None:event = item.get("lifecycleEvent")subscription_id = str(item.get("subscriptionId", ""))+ if not self._listening:+ self._hand_to_holder(+ {+ "action": "lifecycle",+ "event": str(event),+ "subscription_id": subscription_id,+ }+ )+ returnif event == "reauthorizationRequired":await self._renew_subscription(self._channel_of_subscription(subscription_id), subscription_id@@ -4289,6 +4334,32 @@ class TeamsAdapter(PlatformAdapter):event,)+ async def receive_relayed(self, request: Mapping[str, str]) -> None:+ """What a quiet copy in another process relayed here, the holder:+ each the same call the quiet copy would have made had it held the+ bridge. Checked there as it would have been here: the activity's+ token, or the notification's `clientState` and subscription."""+ action = request.get("action")+ if action == "capture":+ channel_id = request["channel_id"]+ team_id = request.get("team_id")+ if team_id:+ await self._learn_channel_team(channel_id, team_id)+ await self._ensure_channel_subscription(channel_id)+ elif action == "team_joined":+ await self._switch_joined_team(request["team_id"])+ elif action == "team_left":+ await self._switch_left_team(request["team_id"])+ elif action == "lifecycle":+ await self._handle_lifecycle_event(+ {+ "lifecycleEvent": request["event"],+ "subscriptionId": request["subscription_id"],+ }+ )+ else:+ logger.warning("Ignoring a relayed Teams request %r", request)+def _channel_of_subscription(self, subscription_id: str) -> str | None:return next((c for c, s in self._subscriptions.items() if s == subscription_id),diff --git a/core/switch_core/main.py b/core/switch_core/main.pyindex 8e1272393..9ab2280fa 100644--- a/core/switch_core/main.py+++ b/core/switch_core/main.py@@ -1375,15 +1375,16 @@ async def run(config: SwitchConfig, process_id: str) -> None:)await collab_lifecycle.start_all()- # The one shared Discord Gateway connection. Started after the bridges so it- # can attach to the inert ones the moment it connects β a shared-delivery- # bridge opens no socket of its own, and attaching re-runs the agent-identity- # provisioning that could not run at start. Supervised in the background: a- # configured-but-unreachable Discord app must never block or fail a boot that- # serves every other platform, so its initial connect retries with backoff- # and Discord installs stay inert until it succeeds.+ # The one shared Discord Gateway connection, opened in the one process that+ # holds it (`ownership.py`) and logged in for the API alone in every other.+ # Wanted after the bridges so it can attach to the inert ones the moment it+ # connects β a shared-delivery bridge opens no socket of its own, and+ # attaching re-runs the agent-identity provisioning that could not run at+ # start. Brought up in the background: a configured-but-unreachable Discord+ # app must never block or fail a boot that serves every other platform, so+ # its initial connect retries with backoff and Discord installs stay inert+ # until it succeeds.discord_gateway: DiscordGatewayClient | None = None- discord_gateway_task: asyncio.Task[None] | None = Noneif config.discord_app_bot_token:# install_service is present whenever an installer is registered, and the# Discord bot token being set means the Discord installer is β so this is@@ -1393,8 +1394,8 @@ async def run(config: SwitchConfig, process_id: str) -> None:async def _attach_shared_discord_bridges(connection: DiscordConnection,) -> None:- # Runs once the socket is up: hand it to every already-running- # Discord bridge. Narrowed to Discord's adapter and not only to the+ # Runs once the connection is up, as the socket or for the API+ # alone: hand it to every already-running Discord bridge. Narrowed to Discord's adapter and not only to the# capability, because Telegram's shared bridges have it too and# must be handed Telegram's bot, not this socket.for adapter in collab_lifecycle.iter_adapters():@@ -1409,9 +1410,7 @@ async def run(config: SwitchConfig, process_id: str) -> None:on_connected=_attach_shared_discord_bridges,)collab_lifecycle.add_bridge_starting_listener(discord_gateway.attach_if_live)- discord_gateway_task = asyncio.create_task(- discord_gateway.start_with_retry(), name="discord-gateway-start"- )+ discord_gateway.own(ownership)# The shared Telegram bot: `getMe`, then `setWebhook`. In the background# for the reason the Discord connection is β an unreachable Telegram must@@ -1434,6 +1433,8 @@ async def run(config: SwitchConfig, process_id: str) -> None:# stops being sure stops the work it holds there and then.ownership_task = asyncio.create_task(ownership.run())await collab_lifecycle.start_quiet_copies()+ if discord_gateway is not None:+ discord_gateway.start_quietly()# Backfill room membership: system clients (e.g. the admin client) added# after a room was created, and any agent whose invite did not land. The@@ -1478,7 +1479,6 @@ async def run(config: SwitchConfig, process_id: str) -> None:connector_lifecycle,provisioning,discord_gateway,- discord_gateway_task,teams_app,telegram_app,telegram_app_task,@@ -1989,7 +1989,6 @@ async def _shutdown(connector_lifecycle: ServerSideConnectorLifecycleService,provisioning: Provisioning,discord_gateway: DiscordGatewayClient | None,- discord_gateway_task: asyncio.Task[None] | None,teams_app: TeamsSharedApp | None,# Defaulted so a caller with no Telegram app, as the shutdown-order tests# are, need not name it; `run` always passes both.@@ -2012,14 +2011,10 @@ async def _shutdown(server.should_exit = Trueawait connector_lifecycle.stop_all()await collab_lifecycle.stop_all()- # Cancel the supervised connect/retry loop before closing the socket, so a- # retry in flight cannot re-open what stop() just closed.- if discord_gateway_task is not None:- discord_gateway_task.cancel()- try:- await discord_gateway_task- except asyncio.CancelledError:- pass+ # The API log-in of a process that did not hold the socket; the socket+ # itself was closed with the rest of the owned work in `release`. `stop`+ # cancels a connect or log-in still retrying before it closes, so a retry+ # in flight cannot re-open what it just closed.if discord_gateway is not None:await discord_gateway.stop()# After the bridges, which borrow its HTTP client until they stop.diff --git a/core/switch_core/ownership.py b/core/switch_core/ownership.pyindex 172cc82b9..85a252beb 100644--- a/core/switch_core/ownership.py+++ b/core/switch_core/ownership.py@@ -14,7 +14,8 @@ is decided twice over:`switch_core_processes` holds: every process scores every key against everylive process and the highest score wins. All processes reading the sameleases agree, and a process coming or going moves only the keys it wins or- held, not everyone's.+ held, not everyone's. A key that can only run beside another kind's key is+ hashed as that key (`OwnedKind.want`), so both land on the same process.- **Who does.** A row in `ownership_leases`, claimed in one statement and onlywhile nobody holds it or its holder's process lease has lapsed. The hash sayswho tries; the row makes sure only one succeeds, whatever two processes@@ -129,16 +130,28 @@ class OwnedKind:self._start = startself._stop = stopself._wanted: set[str] = set()+ # Keys placed with another kind's key (`want`), by that kind and key.+ self._placed_with: dict[str, tuple[str, str]] = {}self._held: dict[str, Fence] = {}# Fences the work reported taken over, kept until the next balance# stops it.self._lost: set[Fence] = set()- def want(self, key: str) -> None:- """This process could run `key`. It runs it once it owns it."""- if key in self._wanted:+ def want(self, key: str, *, placed_with: tuple[str, str] | None = None) -> None:+ """This process could run `key`. It runs it once it owns it.++ `placed_with` names another kind's key, as (kind, key), whose owner+ should own this one too: work that can only run beside it, such as a+ Discord bridge whose events reach it in-process from the one shared+ Gateway connection. Both are hashed the same, so every process puts+ them on the same one. Each still has a lease of its own."""+ if key in self._wanted and self._placed_with.get(key) == placed_with:returnself._wanted.add(key)+ if placed_with is None:+ self._placed_with.pop(key, None)+ else:+ self._placed_with[key] = placed_withself._ownership.nudge()def wanted(self) -> set[str]:@@ -170,6 +183,7 @@ class OwnedKind:runs, and let go of its lease. For work being deleted, which must bestopped before its rows go."""self._wanted.discard(key)+ self._placed_with.pop(key, None)async with self._ownership.balancing():if key in self._held:await self._stop_held([key])@@ -194,7 +208,7 @@ class OwnedKind:", ".join(sorted(lost)),)await self._stop_held(lost)- mine = {key for key in self._wanted if owner_of(self.name, key, live) == me}+ mine = {key for key in self._wanted if self._owner(key, live) == me}leaving = [key for key in self._held if key not in mine]if leaving:logger.info(@@ -220,6 +234,10 @@ class OwnedKind:except Exception:logger.exception("Starting %s %s failed", self.name, key)+ def _owner(self, key: str, live: Sequence[str]) -> str | None:+ kind, placed_key = self._placed_with.get(key, (self.name, key))+ return owner_of(kind, placed_key, live)+async def _stop_held(self, keys: Sequence[str]) -> None:"""Stop the work of `keys` here, all at once, each failure logged."""stopping = [key for key in keys if self._held.pop(key, None) is not None]@@ -256,6 +274,10 @@ class Ownership:self._bus = busself._store = storeself._processes = processes+ # A kind can be registered while a balance is under way (Discord's+ # gateway registers its kind once the signal bus has started, and a+ # resync balances at any time), so every loop over the kinds that+ # awaits goes over a copy taken before it starts.self._kinds: dict[str, OwnedKind] = {}# One balance, drop or stop at a time: each reads what is held, then# awaits stops and claims, and two interleaved would claim or release@@ -352,7 +374,7 @@ class Ownership:)returnself._stopped = True- for kind in self._kinds.values():+ for kind in list(self._kinds.values()):await kind._stop_all()self._work_stopped = Truetry:@@ -406,7 +428,7 @@ class Ownership:self.process_id,)self._suspended = False- for kind in self._kinds.values():+ for kind in list(self._kinds.values()):await kind._balance(live)async def _prune_now_and_then(self) -> None:@@ -433,7 +455,7 @@ class Ownership:self.process_id,)self._suspended = True- for kind in self._kinds.values():+ for kind in list(self._kinds.values()):await kind._stop_all()async def claim(self, kind: str, keys: Sequence[str]) -> dict[str, int]:diff --git a/core/tests/ownership.py b/core/tests/ownership.pyindex 30344f20e..e9a680346 100644--- a/core/tests/ownership.py+++ b/core/tests/ownership.py@@ -140,7 +140,7 @@ class _NeverOwnedKind:def __init__(self) -> None:self._wanted: set[str] = set()- def want(self, key: str) -> None:+ def want(self, key: str, *, placed_with: tuple[str, str] | None = None) -> None:self._wanted.add(key)def wants(self, key: str) -> bool:@@ -210,7 +210,7 @@ class _SoleKind:self._wanted: set[str] = set()self._started: set[str] = set()- def want(self, key: str) -> None:+ def want(self, key: str, *, placed_with: tuple[str, str] | None = None) -> None:self._wanted.add(key)def wanted(self) -> set[str]:diff --git a/core/tests/switch_core/bridges/collaboration/test_bridge_ownership.py b/core/tests/switch_core/bridges/collaboration/test_bridge_ownership.pyindex ba2f16c85..ed1e83da6 100644--- a/core/tests/switch_core/bridges/collaboration/test_bridge_ownership.py+++ b/core/tests/switch_core/bridges/collaboration/test_bridge_ownership.py@@ -238,8 +238,8 @@ class _Chat(PlatformAdapter):class _LoudChat(_Chat):- """A platform whose adapter cannot keep quiet, as Discord's gateway and- Teams' listener are today: every process's copy listens to it."""+ """A platform whose adapter cannot keep quiet: every process's copy+ listens to it."""keeps_quiet_unless_listening: ClassVar[bool] = Falsediff --git a/core/tests/switch_core/bridges/collaboration/test_discord_adapter.py b/core/tests/switch_core/bridges/collaboration/test_discord_adapter.pyindex f5a3eda84..f92254e6f 100644--- a/core/tests/switch_core/bridges/collaboration/test_discord_adapter.py+++ b/core/tests/switch_core/bridges/collaboration/test_discord_adapter.py@@ -1229,6 +1229,31 @@ def test_start_raises_when_gateway_connection_fails() -> None:_run(scenario())+def test_a_quiet_copy_logs_in_for_the_api_and_opens_no_socket() -> None:+ """Another process holds the bridge and listens for it. Discord delivers+ every event to every socket, so this copy opens none, and leaves the+ commands to the one that does."""+ adapter = DiscordAdapter(+ config=DiscordConnectionConfig(bot_token="token", guild_id=str(GUILD_ID))+ )+ adapter.set_listening(False)+ fake = _FakeGatewayClient()++ async def scenario() -> None:+ with patch.object(adapter_module.discord, "Client", lambda **kw: fake):+ await adapter.start(*_noop_callbacks())+ assert fake.logged_in+ assert adapter._connection._connect_task is None+ assert fake.registered_events == []+ assert fake.http.bulk_upserts == []+ assert adapter._require_client() is fake++ await adapter.stop()+ assert fake.closed++ _run(scenario())++def test_start_times_out_when_never_ready_and_stops() -> None:adapter = DiscordAdapter(config=DiscordConnectionConfig(bot_token="token", guild_id=str(GUILD_ID))diff --git a/core/tests/switch_core/bridges/collaboration/test_discord_directory_search.py b/core/tests/switch_core/bridges/collaboration/test_discord_directory_search.pyindex 7febe7f4c..2d5f668d7 100644--- a/core/tests/switch_core/bridges/collaboration/test_discord_directory_search.py+++ b/core/tests/switch_core/bridges/collaboration/test_discord_directory_search.py@@ -8,6 +8,7 @@ linked after they had spoken, even though Discord can answer the question.from __future__ import annotationsimport asyncio+from types import SimpleNamespacefrom typing import Anyimport discord@@ -137,3 +138,41 @@ def test_failures_surface_as_runtime_error(index: int) -> None:guild = _FakeGuild(_errors()[index])with pytest.raises(RuntimeError):_run(_adapter(guild).search_directory_users("lou"))+++class _SearchHTTP:+ """Discord's member search endpoint, as a quiet copy reaches it."""++ def __init__(self, found: list[dict[str, Any]]) -> None:+ self.found = found+ self.requests: list[tuple[str, dict[str, Any]]] = []++ async def request(self, route: Any, **kwargs: Any) -> list[dict[str, Any]]:+ self.requests.append((route.path, kwargs.get("params") or {}))+ return self.found+++def test_a_quiet_copy_searches_over_the_api() -> None:+ """It has no Gateway socket to query members over."""+ http = _SearchHTTP(+ [+ {"user": {"id": "1", "username": "ann", "global_name": "Ann"}},+ {"user": {"id": "2", "username": "bob"}, "nick": "Bobby"},+ {"user": {"id": "3", "username": "helper", "bot": True}},+ ]+ )+ adapter = DiscordAdapter(+ config=DiscordConnectionConfig(bot_token="token", guild_id=str(GUILD_ID))+ )+ adapter.set_listening(False)+ adapter._require_client = lambda: SimpleNamespace(http=http) # type: ignore[method-assign]++ found = _run(adapter.search_directory_users("a"))++ assert [(u.external_user_id, u.username, u.display_name) for u in found] == [+ ("1", "ann", "Ann"),+ ("2", "bob", "Bobby"),+ ]+ assert http.requests == [+ ("/guilds/{guild_id}/members/search", {"query": "a", "limit": 100})+ ]diff --git a/core/tests/switch_core/bridges/collaboration/test_discord_gateway_ownership.py b/core/tests/switch_core/bridges/collaboration/test_discord_gateway_ownership.pynew file mode 100644index 000000000..75811d33f--- /dev/null+++ b/core/tests/switch_core/bridges/collaboration/test_discord_gateway_ownership.py@@ -0,0 +1,344 @@+"""Discord's one shared Gateway connection opens in one process, and every+shared-delivery bridge is held there.++Against real PostgreSQL, two "processes" on one database, each wired as+`test_bridge_ownership.py` wires one, with the shared Gateway client `main`+builds when a Discord app is configured. Discord here (`_Discord`) knows+which connections hold a socket, which Discord delivers every event to, and+which are logged in for the API alone, which it delivers nothing to. The+gateway hands each event to the bridge in its own process, so a bridge held+anywhere but beside the socket would never hear one.+"""++from __future__ import annotations++import asyncio+import uuid+from collections.abc import AsyncIterator, Awaitable, Callable+from dataclasses import dataclass+from datetime import timedelta+from typing import Any+from unittest.mock import AsyncMock, MagicMock++import pytest+from sqlalchemy import func, update+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++from switch_core.bridges.collaboration.discord.adapter import (+ DiscordAdapter,+ DiscordConnectionConfig,+)+from switch_core.bridges.collaboration.discord.connection import DiscordConnection+from switch_core.bridges.collaboration.discord.gateway import DiscordGatewayClient+from switch_core.db.models import Client, CollaborationBridge, SwitchCoreProcess+from switch_core.process_lease import LEASE_TTL+from tests.switch_core.bridges.collaboration.test_bridge_ownership import (+ _Process,+)+from tests.switch_core.bridges.collaboration.test_bridge_ownership import (+ _process as _bridge_process,+)++TIMEOUT = 10.0+# Enough bridges that, placed each on its own, some would land on the process+# without the socket: all six landing beside it is a chance of 1 in 64.+BRIDGES = 6+++class _Discord:+ """Discord, as far as the bot's connections reach it."""++ def __init__(self) -> None:+ self.sockets: list[_Connection] = []+ self.logged_in: list[_Connection] = []+ # The most sockets the bot ever held open at once.+ self.most_sockets = 0+++class _Connection:+ """A `DiscordConnection` that reaches `_Discord` instead of Discord."""++ def __init__(self, discord: _Discord) -> None:+ self.discord = discord+ self.bot_user_id = 1+ self.client = MagicMock()++ def set_guild_message_handler(self, handler: Any) -> None:+ pass++ def set_interaction_handler(self, handler: Any) -> None:+ pass++ def set_guild_lifecycle_handlers(self, **handlers: Any) -> None:+ pass++ async def connect(self, *, commands: Any) -> None:+ self.discord.sockets.append(self)+ self.discord.most_sockets = max(+ self.discord.most_sockets, len(self.discord.sockets)+ )++ async def login(self) -> None:+ self.discord.logged_in.append(self)++ async def close(self) -> None:+ if self in self.discord.sockets:+ self.discord.sockets.remove(self)+ if self in self.discord.logged_in:+ self.discord.logged_in.remove(self)+++@dataclass+class _WithGateway:+ process: _Process+ gateway: DiscordGatewayClient+ connection: _Connection++ async def boot(self) -> None:+ """Start as `main` does, the gateway wanted before the first balance+ and logged in for the API alone after it where not held."""+ await self.process.clients.start_all()+ await self.process.bridges.start_all()+ self.gateway.own(self.process.ownership)+ await self.process.ownership.start()+ await self.process.bridges.start_quiet_copies()+ self.gateway.start_quietly()++ async def stop(self) -> None:+ """Shut down as `main` does: the owned work, socket included, first,+ then the quiet copies, then the API log-in."""+ await self.process.stop()+ await self.gateway.stop()++ def has_socket(self) -> bool:+ return self.connection in self.connection.discord.sockets++ def holds_every(self, bridge_ids: list[str]) -> bool:+ return all(self.process.bridges.holds(b) for b in bridge_ids)+++async def _with_gateway(+ postgres_url: str,+ session_factory: async_sessionmaker[AsyncSession],+ discord: _Discord,+ *,+ attach_fails: bool = False,+) -> _WithGateway:+ process = await _bridge_process(postgres_url, session_factory)+ process.bridges.register_adapter("discord", DiscordAdapter, DiscordConnectionConfig)+ bridges = process.bridges++ async def attach(connection: DiscordConnection) -> None:+ if attach_fails:+ raise RuntimeError("attaching the bridges failed")+ for adapter in bridges.iter_adapters():+ if isinstance(adapter, DiscordAdapter):+ adapter.attach_shared_connection(connection)++ gateway = DiscordGatewayClient(+ bot_token="bot-token",+ message_content=False,+ members=False,+ install_service=AsyncMock(),+ on_connected=attach,+ )+ connection = _Connection(discord)+ gateway._connection = connection # type: ignore[assignment]+ bridges.add_bridge_starting_listener(gateway.attach_if_live)+ return _WithGateway(process=process, gateway=gateway, connection=connection)+++async def shared_bridges(+ session_factory: async_sessionmaker[AsyncSession],+) -> list[str]:+ """Discord servers installed through the deployment's shared app."""+ ids: list[str] = []+ async with session_factory() as session:+ for n in range(BRIDGES):+ client = Client(+ transport_user_id=f"@switch-bridge-{uuid.uuid4().hex[:8]}:test",+ display_name="bridge",+ type="bridge",+ )+ session.add(client)+ await session.flush()+ row = CollaborationBridge(+ type="discord",+ display_name=f"Server {n}",+ connection_config={+ "guild_id": str(1000 + n),+ "event_delivery": "shared",+ },+ client_id=client.id,+ status="active",+ channel_creation_enabled=False,+ preconfigured=False,+ )+ session.add(row)+ await session.flush()+ ids.append(row.id)+ await session.commit()+ return ids+++async def two(+ postgres_url: str,+ session_factory: async_sessionmaker[AsyncSession],+ shared_bridges: list[str],+) -> AsyncIterator[tuple[_WithGateway, _WithGateway]]:+ discord = _Discord()+ started: list[_WithGateway] = []+ try:+ for _ in range(2):+ process = await _with_gateway(postgres_url, session_factory, discord)+ started.append(process)+ await process.boot()+ # The first balance of each cannot see the other; balance once more+ # now that both leases are there, as the interval would.+ for process in started:+ await process.process.ownership.balance()+ for process in started:+ await process.process.settled()+ yield started[0], started[1]+ finally:+ for process in started:+ await process.stop()+++async def _eventually(check: Callable[[], bool | Awaitable[bool]]) -> None:+ async with asyncio.timeout(TIMEOUT):+ while True:+ answer = check()+ if answer if isinstance(answer, bool) else await answer:+ return+ await asyncio.sleep(0.02)+++def _with_socket(+ two: tuple[_WithGateway, _WithGateway],+) -> tuple[_WithGateway, _WithGateway]:+ a, b = two+ owners = [p for p in (a, b) if p.has_socket()]+ assert len(owners) == 1+ return (a, b) if owners[0] is a else (b, a)+++class TestOneSocket:+ async def test_one_process_opens_the_socket_and_holds_every_shared_bridge(+ self,+ two: tuple[_WithGateway, _WithGateway],+ shared_bridges: list[str],+ ) -> None:+ a, b = two+ await _eventually(lambda: a.has_socket() or b.has_socket())+ owner, other = _with_socket(two)++ await _eventually(+ lambda: other.connection in other.connection.discord.logged_in+ )+ assert owner.holds_every(shared_bridges)+ assert not any(other.process.bridges.holds(b) for b in shared_bridges)+ assert owner.connection.discord.most_sockets == 1++ async def test_every_copy_of_a_bridge_is_attached_to_its_own_processs_connection(+ self,+ two: tuple[_WithGateway, _WithGateway],+ shared_bridges: list[str],+ ) -> None:+ """The quiet copies serve requests over the API log-in."""+ for process in two:+ for bridge_id in shared_bridges:+ await _eventually(+ lambda p=process, b=bridge_id: (+ getattr(p.process.bridges.get_adapter(b), "_connection", None)+ is p.connection+ )+ )+++class TestHandingOver:+ async def test_a_process_stopping_hands_the_socket_and_its_bridges_over(+ self,+ two: tuple[_WithGateway, _WithGateway],+ shared_bridges: list[str],+ ) -> None:+ a, b = two+ await _eventually(lambda: a.has_socket() or b.has_socket())+ owner, other = _with_socket(two)++ await owner.stop()+ await other.process.ownership.balance()++ await _eventually(other.has_socket)+ assert other.holds_every(shared_bridges)+ assert other.connection.discord.sockets == [other.connection]+ assert other.connection.discord.most_sockets == 1++ async def test_a_socket_whose_bring_up_failed_is_closed_as_it_is_handed_over(+ self,+ postgres_url: str,+ session_factory: async_sessionmaker[AsyncSession],+ shared_bridges: list[str],+ ) -> None:+ """Attaching the bridges failed once the socket was up, and handing+ the socket over raises that failure. The ownership releases the+ socket's lease all the same, and the next holder opens a socket of its+ own, so this one has to be closed by then."""+ discord = _Discord()+ started: list[_WithGateway] = []+ try:+ failing = await _with_gateway(+ postgres_url, session_factory, discord, attach_fails=True+ )+ started.append(failing)+ # Alone, so it takes the socket.+ await failing.boot()+ await _eventually(failing.has_socket)+ other = await _with_gateway(postgres_url, session_factory, discord)+ started.append(other)+ await other.boot()++ # Not the gateway's own `stop`, which `main` calls only after the+ # drain: only the hand-over can have closed the socket by now.+ await failing.process.stop()+ await other.process.ownership.balance()++ await _eventually(other.has_socket)+ assert other.holds_every(shared_bridges)+ assert discord.sockets == [other.connection]+ assert discord.most_sockets == 1+ finally:+ for process in started:+ await process.stop()++ async def test_a_process_that_died_is_taken_over_once_its_lease_lapses(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ two: tuple[_WithGateway, _WithGateway],+ shared_bridges: list[str],+ ) -> None:+ a, b = two+ await _eventually(lambda: a.has_socket() or b.has_socket())+ owner, other = _with_socket(two)++ # It dies without a word: no more renewals, and Discord drops the+ # socket of a bot that stopped answering.+ for task in owner.process.tasks:+ task.cancel()+ await owner.connection.close()+ await other.process.ownership.balance()+ assert not other.has_socket()++ async with session_factory() as session:+ await session.execute(+ update(SwitchCoreProcess)+ .where(SwitchCoreProcess.id == owner.process.lease.process_id)+ .values(beat_at=func.now() - LEASE_TTL - timedelta(seconds=1))+ )+ await session.commit()+ await other.process.ownership.balance()++ await _eventually(other.has_socket)+ assert other.holds_every(shared_bridges)diff --git a/core/tests/switch_core/bridges/collaboration/test_teams_subscription_ownership.py b/core/tests/switch_core/bridges/collaboration/test_teams_subscription_ownership.pynew file mode 100644index 000000000..c8f4882bd--- /dev/null+++ b/core/tests/switch_core/bridges/collaboration/test_teams_subscription_ownership.py@@ -0,0 +1,291 @@+"""A Teams bridge's Graph subscriptions are made, adopted and renewed by the+one process that holds the bridge.++Against real PostgreSQL, two "processes" on one database, each wired as+`test_bridge_ownership.py` wires one, both running a bring-your-own Teams+bridge. Graph here (`_Graph`) is the one both copies' clients reach: it keeps+the subscriptions, and each client records what it asked of it. Graph+delivers a channel's messages to every subscription made for it, and its+webhooks reach whichever process the load balancer picks, so a quiet copy+hands what would change the subscriptions to the holder.+"""++from __future__ import annotations++import asyncio+import uuid+from collections.abc import AsyncIterator, Callable+from dataclasses import dataclass+from datetime import UTC, datetime, timedelta+from typing import Any+from unittest.mock import MagicMock++import pytest+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++from switch_core.bridges.collaboration.teams import adapter as adapter_module+from switch_core.bridges.collaboration.teams.adapter import (+ TeamsAdapter,+ TeamsConnectionConfig,+)+from switch_core.db.models import Client, CollaborationBridge, Room+from switch_core.db.stores.room_store import RoomStore+from tests.switch_core.bridges.collaboration.test_bridge_ownership import (+ _Process,+)+from tests.switch_core.bridges.collaboration.test_bridge_ownership import (+ _process as _bridge_process,+)+from tests.switch_core.bridges.collaboration.test_teams_capture_repair import (+ _config,+)++TIMEOUT = 10.0+TEAM = "team-1"+CHANNEL = "19:[email protected]"+NEW_CHANNEL = "19:[email protected]"+NOTIFICATION_URL = "https://switch.example/api/teams/notifications"+++def _resource(channel_id: str) -> str:+ return f"teams/{TEAM}/channels/{channel_id}/messages"+++class _Graph:+ """Microsoft Graph, as far as these bridges reach it."""++ def __init__(self) -> None:+ self.subscriptions: dict[str, dict[str, Any]] = {}++ def add(self, channel_id: str, *, expires_in: timedelta) -> str:+ subscription_id = f"sub-{uuid.uuid4().hex[:8]}"+ self.subscriptions[subscription_id] = {+ "id": subscription_id,+ "resource": _resource(channel_id),+ "notificationUrl": NOTIFICATION_URL,+ "expirationDateTime": (datetime.now(UTC) + expires_in).isoformat(),+ }+ return subscription_id+++class _GraphClient:+ """One copy's Graph client, recording what that copy asked."""++ def __init__(self, graph: _Graph) -> None:+ self.graph = graph+ self.listed = 0+ self.created: list[str] = []+ self.renewed: list[str] = []+ self.deleted: list[str] = []++ async def list_subscriptions(self) -> list[dict[str, Any]]:+ self.listed += 1+ return list(self.graph.subscriptions.values())++ async def create_subscription(+ self, *, resource: str, expiration_iso: str, **_: Any+ ) -> dict[str, Any]:+ self.created.append(resource)+ channel_id = resource.split("/channels/")[1].split("/")[0]+ subscription_id = self.graph.add(channel_id, expires_in=timedelta(minutes=55))+ return {"id": subscription_id, "expirationDateTime": expiration_iso}++ async def renew_subscription(+ self, *, subscription_id: str, expiration_iso: str+ ) -> None:+ self.renewed.append(subscription_id)++ async def delete_subscription(self, *, subscription_id: str) -> None:+ self.deleted.append(subscription_id)+ self.graph.subscriptions.pop(subscription_id, None)++ async def create_channel(self, *, display_name: str, **_: Any) -> dict[str, Any]:+ return {"id": NEW_CHANNEL, "displayName": display_name}++ def __getattr__(self, name: str) -> Any:+ # The rest of Graph is not what these tests are about: reading a+ # channel's type or name fails, as an unreachable Graph would.+ async def unreachable(*args: Any, **kwargs: Any) -> Any:+ raise RuntimeError(f"Graph {name} is not part of this test")++ return unreachable+++def graph(monkeypatch: pytest.MonkeyPatch) -> _Graph:+ graph = _Graph()+ monkeypatch.setattr(adapter_module, "GraphClient", lambda **_: _GraphClient(graph))+ for name in (+ "TeamsTokenProvider",+ "BotFrameworkAuthenticator",+ "SigningKeys",+ "BotConnectorClient",+ ):+ monkeypatch.setattr(adapter_module, name, lambda **_: MagicMock())++ async def no_listener(self: TeamsAdapter) -> None:+ return None++ # Both copies run in this one test process: neither binds its port.+ monkeypatch.setattr(TeamsAdapter, "_open_listener", no_listener)+ return graph+++@dataclass+class _Teams:+ bridge_id: str+++async def teams(session_factory: async_sessionmaker[AsyncSession]) -> _Teams:+ """A bring-your-own Teams bridge carrying one channel."""+ async with session_factory() as session:+ client = Client(+ transport_user_id=f"@switch-bridge-{uuid.uuid4().hex[:8]}:test",+ display_name="bridge",+ type="bridge",+ )+ session.add(client)+ await session.flush()+ row = CollaborationBridge(+ type="teams",+ display_name="Teams",+ connection_config=_config().model_dump(),+ client_id=client.id,+ status="active",+ channel_creation_enabled=True,+ preconfigured=False,+ )+ session.add(row)+ await session.flush()+ room = Room(+ transport_room_id=f"!room-{uuid.uuid4().hex[:8]}:test",+ name="captured",+ description="",+ bridge_id=row.id,+ external_channel_id=CHANNEL,+ channel_type="channel_public",+ )+ session.add(room)+ await session.flush()+ await RoomStore().add_client(session, client.id, room.id)+ await session.commit()+ return _Teams(bridge_id=row.id)+++async def two(+ postgres_url: str,+ session_factory: async_sessionmaker[AsyncSession],+ graph: _Graph,+ teams: _Teams,+) -> AsyncIterator[tuple[_Process, _Process]]:+ started: list[_Process] = []+ try:+ for _ in range(2):+ process = await _bridge_process(postgres_url, session_factory)+ process.bridges.register_adapter(+ "teams", TeamsAdapter, TeamsConnectionConfig+ )+ started.append(process)+ await process.boot()+ # The first balance of each cannot see the other; balance once more+ # now that both leases are there, as the interval would.+ for process in started:+ await process.ownership.balance()+ for process in started:+ await process.settled()+ for process in started:+ await _eventually(lambda p=process: p.bridges.is_connected(teams.bridge_id))+ yield started[0], started[1]+ finally:+ for process in started:+ await process.stop()+++async def _eventually(check: Callable[[], bool]) -> None:+ async with asyncio.timeout(TIMEOUT):+ while not check():+ await asyncio.sleep(0.02)+++def _holder(+ two: tuple[_Process, _Process], bridge_id: str+) -> tuple[_Process, _Process]:+ a, b = two+ holders = [p for p in (a, b) if p.bridges.holds(bridge_id)]+ assert len(holders) == 1+ return (a, b) if holders[0] is a else (b, a)+++def _adapter(process: _Process, bridge_id: str) -> TeamsAdapter:+ adapter = process.bridges.get_adapter(bridge_id)+ assert isinstance(adapter, TeamsAdapter)+ return adapter+++def _graph_of(process: _Process, bridge_id: str) -> _GraphClient:+ client = _adapter(process, bridge_id)._graph+ assert isinstance(client, _GraphClient)+ return client+++class TestOnlyTheHolder:+ @pytest.fixture+ def graph(self, graph: _Graph) -> _Graph:+ # Made before either process started, and close to running out.+ graph.add(CHANNEL, expires_in=timedelta(minutes=5))+ return graph++ async def test_adopts_and_renews_the_subscriptions(+ self, two: tuple[_Process, _Process], teams: _Teams, graph: _Graph+ ) -> None:+ holder, quiet = _holder(two, teams.bridge_id)+ held = _graph_of(holder, teams.bridge_id)++ await _eventually(lambda: len(held.renewed) == 1)+ # Long enough for the quiet copy's renewal, had there been one.+ await asyncio.sleep(0.5)++ assert held.renewed == list(graph.subscriptions)+ copy = _graph_of(quiet, teams.bridge_id)+ assert copy.listed == 0+ assert copy.renewed == []+ assert copy.created == []+++class TestWebhooksReachingTheQuietProcess:+ async def test_a_channel_made_through_it_is_subscribed_once_by_the_holder(+ self, two: tuple[_Process, _Process], teams: _Teams, graph: _Graph+ ) -> None:+ holder, quiet = _holder(two, teams.bridge_id)++ await _adapter(quiet, teams.bridge_id).create_channel("new", "")++ held = _graph_of(holder, teams.bridge_id)+ await _eventually(lambda: _resource(NEW_CHANNEL) in held.created)+ await asyncio.sleep(0.5)+ assert held.created.count(_resource(NEW_CHANNEL)) == 1+ assert _graph_of(quiet, teams.bridge_id).created == []++ async def test_a_subscription_needing_renewal_is_renewed_by_the_holder(+ self, two: tuple[_Process, _Process], teams: _Teams, graph: _Graph+ ) -> None:+ holder, quiet = _holder(two, teams.bridge_id)+ held = _graph_of(holder, teams.bridge_id)+ # The holder subscribes to the bridge's channel as it starts.+ await _eventually(lambda: len(graph.subscriptions) == 1)+ (subscription_id,) = graph.subscriptions+ renewed_before = list(held.renewed)++ await _adapter(quiet, teams.bridge_id).receive_notification(+ {+ "subscriptionId": subscription_id,+ "clientState": "s3cr3t",+ "lifecycleEvent": "reauthorizationRequired",+ }+ )++ await _eventually(lambda: held.renewed != renewed_before)+ assert held.renewed[len(renewed_before) :] == [subscription_id]+ assert _graph_of(quiet, teams.bridge_id).renewed == []diff --git a/core/tests/switch_core/test_main_teams_app.py b/core/tests/switch_core/test_main_teams_app.pyindex bc622806d..6de0f25af 100644--- a/core/tests/switch_core/test_main_teams_app.py+++ b/core/tests/switch_core/test_main_teams_app.py@@ -114,7 +114,6 @@ async def test_the_apps_client_is_closed_only_after_the_bridges_stop(connector_lifecycle=_Stoppable("connectors"), # type: ignore[arg-type]provisioning=_Stoppable("provisioning"), # type: ignore[arg-type]discord_gateway=None,- discord_gateway_task=None,teams_app=_TeamsApp(), # type: ignore[arg-type])@@ -199,7 +198,6 @@ async def test_shutdown_drains_before_it_closes_anything(connector_lifecycle=_Stoppable("connectors"), # type: ignore[arg-type]provisioning=_Stoppable("provisioning"), # type: ignore[arg-type]discord_gateway=None,- discord_gateway_task=None,teams_app=None,)diff --git a/core/tests/switch_core/test_ownership.py b/core/tests/switch_core/test_ownership.pyindex 925ff3ea0..647a61ceb 100644--- a/core/tests/switch_core/test_ownership.py+++ b/core/tests/switch_core/test_ownership.py@@ -167,6 +167,21 @@ class TestOneOwner:owner = a if owner_of(KIND, key, live) == a.process_id else bassert key in owner.runs()+ async def test_keys_placed_with_another_kinds_key_run_beside_it(+ self, two: tuple[_Process, _Process]+ ) -> None:+ a, b = two+ beside = ("socket", "the-one")+ for process in two:+ for key in KEYS:+ process.kind.want(key, placed_with=beside)+ await a.ownership.balance()+ await b.ownership.balance()++ live = sorted([a.process_id, b.process_id])+ owner = a if owner_of(*beside, live) == a.process_id else b+ assert owner.runs() == set(KEYS)+async def test_one_process_owns_everything(self,session_factory: async_sessionmaker[AsyncSession],@@ -182,6 +197,42 @@ class TestOneOwner:assert only.runs() == set(KEYS)assert [fence.token for fence in only.started] == [1] * len(KEYS)+ async def test_a_kind_registered_while_balancing_runs_from_the_next_balance(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """Discord's gateway registers its kind once the signal bus has+ started, so a resync's balance can be under way as it does. That+ balance goes on with the kinds it began with; the next one takes the+ new kind in."""+ lease = process_lease_on(session_factory)+ ownership = Ownership(+ lease=lease,+ session_factory=session_factory,+ bus=SignalBus(+ session_factory, _never_listening, process_id=lease.process_id+ ),+ store=OwnershipLeaseStore(),+ processes=SwitchCoreProcessStore(),+ )+ late_started: list[str] = []++ async def late_start(key: str, _fence: Fence) -> None:+ late_started.append(key)++ async def nothing(_key: str) -> None:+ return None++ async def start_and_register(_key: str, _fence: Fence) -> None:+ ownership.kind("socket", start=late_start, stop=nothing).want("the-one")++ ownership.kind(KIND, start=start_and_register, stop=nothing).want("key-0")+ await lease.renew()++ await ownership.balance()+ assert late_started == []+ await ownership.balance()+ assert late_started == ["the-one"]+class TestHandingOver:async def test_a_process_that_joins_takes_its_share_without_overlap(diff --git a/deploy/remote/helm/switch/README.md b/deploy/remote/helm/switch/README.mdindex 93c88b4d1..98e4252de 100644--- a/deploy/remote/helm/switch/README.md+++ b/deploy/remote/helm/switch/README.md@@ -160,10 +160,11 @@ example), leave `tls.enabled` false and annotate.how many. Shared state lives in the database: replicas signal each other throughPostgres notifications, and each bridge's listener and outbound relay, eachserver connector and each command reader is owned by one replica at a time-through a lease. Any replica can hold any client's connection. Two things are-not single-owner and run on every replica: the shared Discord gateway-connection of the distributed Discord app, and each Teams bridge's Graph-subscriptions, which every replica adopts and renews.+through a lease. That includes the shared Discord gateway connection of the+distributed Discord app, whose Discord bridges run beside it on the same+replica, and each Teams bridge's Graph subscriptions, which only the replica+holding the bridge makes and renews. Any replica can hold any client's+connection, and a platform webhook can reach any replica.One replica upgrades with `Recreate`, so an upgrade has a brief outage. Morethan one rolls (`switchCore.rollingUpdate`, one extra pod at a time bydiff --git a/deploy/remote/helm/switch/values.yaml b/deploy/remote/helm/switch/values.yamlindex 4f8f8b6d8..50e0317be 100644--- a/deploy/remote/helm/switch/values.yaml+++ b/deploy/remote/helm/switch/values.yaml@@ -624,10 +624,10 @@ switchCore:# connections. Each bridge's listener and outbound relay, each server# connector and each command reader runs on one pod at a time, through# leases in the database, and each message is routed to its agents as it is- # written, so any pod can take any request. Two things still run on every- # pod: the deployment's shared Discord gateway connection (the distributed- # Discord app), and each Teams bridge's Graph subscriptions, which every pod- # adopts and renews.+ # written, so any pod can take any request. So does the deployment's shared+ # Discord gateway connection (the distributed Discord app), with its Discord+ # bridges on the same pod, and each Teams bridge's Graph subscriptions, made+ # and renewed by the pod that holds the bridge.## One replica is replaced (Recreate): the old pod stops before the new one# starts, so an upgrade has a short outage. More than one rolls (see