Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Announce messages after their commit, in batches, so they stop queueing on the NOTIFY lock (CHOO-3667)
Abel Dantasf28fc5c8071b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 882218229..b93d180be 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -182,6 +182,19 @@ version of their own to them without also giving them a release of their own.them, and to roll back, first downgrade the database to the revision before`238c68b7306e` (`alembic downgrade 238c68b7306e-1`), then start the olderrelease.+- **Messages no longer queue for the database's NOTIFY lock.** Postgres+ lets one transaction that sends a `NOTIFY` commit at a time, and the+ triggers on `messages` and `agent_events` sent one from every message's+ transaction, so the whole deployment stopped near 300 messages a second+ however many pods or vCPUs it had. The router now writes without them and+ announces its rows once they have committed, through one announcer per+ process that sends everything queued since its last send in a single+ transaction (`announcer.py`). Same channels and payloads, so pods of the+ previous release and this one hear each other during a rolling update; the+ migration is expand only. A reader re-checks its rooms, and the agent+ event listener forgets the heads it was told, every 20 to 30 seconds, so an+ announcement lost with a process that died right after its commit costs a+ delay, not a message. Signals go through the same announcer.- **The statements every message write costs run on asyncpg directly.**Taking the room's lock, reading its head, storing the agents' own eventsand reading the room's agent members again under the lock go straight todiff --git a/core/switch_core/announcer.py b/core/switch_core/announcer.pynew file mode 100644index 000000000..2e2ef05d5--- /dev/null+++ b/core/switch_core/announcer.py@@ -0,0 +1,185 @@+"""This process's `NOTIFY`s, sent after the commits they announce, in batches.++A transaction that sends a `NOTIFY` takes a lock that every other notifying+transaction waits on, held until its commit is durable (Postgres takes it in+`PreCommit_Notify`; `pg_locks` shows it as an `object` lock on database 0). One+such commit at a time, for the whole database: on Aurora about 700 a second,+however many connections or vCPUs are added. A trigger that notifies from+inside every message's transaction puts every message through that lock.++So the writers of the busiest tables commit without notifying+(`WRITER_ANNOUNCES` in `db/notify_ddl.py`) and hand their announcements here,+and this sends everything handed to it since its last send in one+transaction, with one transaction in flight. The lock is taken once per+transaction however many notifications it carries, so a process takes it at+most once per round trip whatever its message rate, and the messages+themselves never wait on it.++What this costs and promises:++- **Announced after the commit, never before.** A caller hands over an+ announcement only once its transaction has committed, so a listener never+ hears of a row it cannot read, and a write that rolled back announces+ nothing.+- **Nothing a commit announces is dropped while the database answers.** A+ batch that fails is kept, ahead of anything handed over since, and sent+ again. Past `MAX_UNSENT` the oldest are dropped and logged: the database+ has refused for a long while by then, and the listeners' re-checks repair+ what they announced.+- **An announcement can still be lost**, which the trigger could not: a+ process that dies between a commit and the send takes it with it. Every+ listener therefore treats an announcement as a hint and re-checks on a+ timer (`AgentEventListener` forgets the heads it was told,+ `PostgresTransport` asks which of its rooms moved), so a lost one costs a+ delay, never a row.+- **Same channels and payloads as the triggers**, so a listener cannot tell+ which sent one, and processes of an older release, which still announce+ through the triggers, deliver to this release's listeners and back.+"""++from __future__ import annotations++import asyncio+import contextlib+import logging+from typing import TYPE_CHECKING++from sqlalchemy import text++if TYPE_CHECKING:+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++logger = logging.getLogger(__name__)++# Postgres refuses a payload of 8000 bytes or more.+MAX_PAYLOAD_BYTES = 7999++# How long a batch that failed to send waits before it is tried again.+RETRY_SECONDS = 2.0++# How many unsent announcements are kept while the database refuses them.+MAX_UNSENT = 100_000++# How long stopping waits for what is left to go out.+FLUSH_SECONDS = 5.0++_NOTIFY_MANY = text(+ "SELECT pg_notify(channel, payload) "+ "FROM unnest(CAST(:channels AS text[]), CAST(:payloads AS text[])) "+ "AS announcement(channel, payload)"+)+++class Announcer:+ """Sends the `NOTIFY`s handed to it, after their commits, in batches."""++ def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:+ self._session_factory = session_factory+ self._outbox: list[tuple[str, str]] = []+ self._has_outbox = asyncio.Event()+ # Set while nothing is queued or being sent, so `stop` can wait for+ # the last batch rather than guess.+ self._idle = asyncio.Event()+ self._idle.set()+ self._task: asyncio.Task[None] | None = None++ def announce(self, channel: str, payload: str) -> None:+ """Send `payload` on `channel` with the next batch. Never blocks and+ never raises for the database: the row it announces has committed."""+ if len(payload.encode()) > MAX_PAYLOAD_BYTES:+ raise ValueError(+ f"An announcement on {channel!r} is too large to send "+ f"({len(payload)} bytes)"+ )+ self._outbox.append((channel, payload))+ self._idle.clear()+ self._has_outbox.set()++ @property+ def unsent(self) -> int:+ return len(self._outbox)++ async def start(self) -> None:+ if self._task is not None:+ raise RuntimeError("Announcer is already started")+ self._task = asyncio.create_task(self._send_forever(), name="announcer")++ async def stop(self) -> None:+ """Send what is left, for up to `FLUSH_SECONDS`, then stop."""+ if self._task is None:+ return+ with contextlib.suppress(TimeoutError):+ await asyncio.wait_for(self._idle.wait(), FLUSH_SECONDS)+ if self._outbox:+ logger.error(+ "Stopping with %d announcement(s) unsent; the listeners' "+ "re-checks find what they announced",+ len(self._outbox),+ )+ self._task.cancel()+ with contextlib.suppress(asyncio.CancelledError):+ await self._task+ self._task = None++ async def _send_forever(self) -> None:+ while True:+ await self._has_outbox.wait()+ self._has_outbox.clear()+ batch, self._outbox = self._outbox, []+ if not batch:+ self._idle.set()+ continue+ # Set once the batch's transaction has committed, which is when+ # Postgres sends it: what fails after that (closing the session)+ # did not stop it going, and keeping it would send it twice.+ committed = False+ try:+ async with self._session_factory() as session:+ await session.execute(+ _NOTIFY_MANY,+ {+ "channels": [channel for channel, _ in batch],+ "payloads": [payload for _, payload in batch],+ },+ )+ await session.commit()+ committed = True+ except asyncio.CancelledError:+ if not committed:+ self._outbox = batch + self._outbox+ raise+ except Exception:+ if committed:+ logger.warning(+ "Sent %d announcement(s), then closing the session failed",+ len(batch),+ exc_info=True,+ )+ else:+ self._keep(batch)+ await asyncio.sleep(RETRY_SECONDS)+ self._has_outbox.set()+ continue+ if not self._outbox:+ self._idle.set()+ else:+ self._has_outbox.set()++ def _keep(self, batch: list[tuple[str, str]]) -> None:+ """Put a batch that failed back ahead of anything handed over since,+ so a process still receives one sender's announcements in order."""+ self._outbox = batch + self._outbox+ if len(self._outbox) > MAX_UNSENT:+ dropped = len(self._outbox) - MAX_UNSENT+ self._outbox = self._outbox[dropped:]+ logger.error(+ "Dropped the %d oldest unsent announcement(s); the listeners' "+ "re-checks find what they announced",+ dropped,+ )+ logger.error(+ "Could not send %d announcement(s); kept and retried in %.0fs",+ len(self._outbox),+ RETRY_SECONDS,+ exc_info=True,+ )diff --git a/core/switch_core/bridges/agent/protocol/agent_event_listener.py b/core/switch_core/bridges/agent/protocol/agent_event_listener.pyindex 4af301c59..1adf89e7d 100644--- a/core/switch_core/bridges/agent/protocol/agent_event_listener.py+++ b/core/switch_core/bridges/agent/protocol/agent_event_listener.py@@ -17,6 +17,13 @@ connection is down are lost, so each time it starts or stops listening everyhead this process was told is forgotten, and when it starts every stream iswoken to look for events it may have missed (`EventBuffer.listening`).+The router announces the events it stores after its commit rather than from+inside it (`announcer.py`), so an announcement can also be lost to a process+that died between the two, with this connection never dropping. Every+heartbeat therefore forgets the heads this process was told+(`EventBuffer.forget_told_heads`), and a lost announcement costs at most one+heartbeat and one pass.+Connection handling follows `MessageListener`: a dedicated connection outsidethe pool, a periodic heartbeat query, and reconnects with a capped backoff."""@@ -113,6 +120,7 @@ class AgentEventListener:await driver.execute("SELECT 1", timeout=HEARTBEAT_TIMEOUT_SECONDS)+ self._buffer.forget_told_heads()finally:self.connected.clear()self._buffer.listening(False)diff --git a/core/switch_core/bridges/agent/protocol/event_buffer.py b/core/switch_core/bridges/agent/protocol/event_buffer.pyindex ef95cee7b..f1491d109 100644--- a/core/switch_core/bridges/agent/protocol/event_buffer.py+++ b/core/switch_core/bridges/agent/protocol/event_buffer.py@@ -1106,6 +1106,18 @@ class EventBuffer:for bell in bells:bell.set()+ def forget_told_heads(self) -> None:+ """Forget every head this process was told, without waking anyone.++ A head kept as told stands until a ring raises it, which holds only+ while every new event is announced. The router announces its events+ after its commit (`announcer.py`), and a process that died between+ the two announced nothing; so the listener calls this on a timer, and+ the next pass of any stream reads those heads from the table.+ """+ self._told_heads.clear()+ self._told_epoch += 1+def listening(self, now_listening: bool) -> None:"""Told by the notification listener each time it starts or stopslistening on a Postgres connection. Notifications sent while it wasdiff --git a/core/switch_core/db/agent_event_notify_ddl.py b/core/switch_core/db/agent_event_notify_ddl.pyindex 1e70c65d0..d34c0560a 100644--- a/core/switch_core/db/agent_event_notify_ddl.py+++ b/core/switch_core/db/agent_event_notify_ddl.py@@ -7,8 +7,18 @@ connection, and that process reads the new rows from the table, so a lostnotification delays delivery but loses nothing.`bridges/agent/protocol/agent_event_listener.py` is the process side thatlistens on the channel.++The router, which stores nearly every own event, announces its own after the+commit instead, as it does its messages (`WRITER_ANNOUNCES_SETTING` in+`notify_ddl.py`): the trigger stands down for a transaction that sets it."""+from __future__ import annotations++import json++from switch_core.db.notify_ddl import WRITER_ANNOUNCES_SETTING+AGENT_EVENT_CHANNEL = "switch_agent_event"AGENT_EVENT_NOTIFY_FUNCTION_NAME = "switch_notify_agent_event"@@ -17,6 +27,10 @@ AGENT_EVENT_NOTIFY_TRIGGER_NAME = "agent_events_notify"CREATE_AGENT_EVENT_NOTIFY_FUNCTION = f"""CREATE OR REPLACE FUNCTION {AGENT_EVENT_NOTIFY_FUNCTION_NAME}() RETURNS trigger AS $$BEGIN+ -- The writer announces this row itself, after its commit.+ IF current_setting('{WRITER_ANNOUNCES_SETTING}', true) = 'on' THEN+ RETURN NULL;+ END IF;PERFORM pg_notify('{AGENT_EVENT_CHANNEL}',json_build_object('agent_id', NEW.agent_id, 'seq', NEW.seq)::text@@ -38,3 +52,8 @@ DROP_AGENT_EVENT_NOTIFY_TRIGGER = (DROP_AGENT_EVENT_NOTIFY_FUNCTION = (f"DROP FUNCTION IF EXISTS {AGENT_EVENT_NOTIFY_FUNCTION_NAME}()")+++def agent_event_announcement(agent_id: str, seq: int) -> str:+ """What the trigger sends for a row, for a writer that announces it itself."""+ return json.dumps({"agent_id": agent_id, "seq": seq}, separators=(",", ":"))diff --git a/core/switch_core/db/notify_ddl.py b/core/switch_core/db/notify_ddl.pyindex 773fc38c8..1347f5e39 100644--- a/core/switch_core/db/notify_ddl.py+++ b/core/switch_core/db/notify_ddl.py@@ -19,6 +19,15 @@ psql) announces it, and the announcement commits withthe row or not at all. `NOTIFY` is only delivered on commit, so a listenernever learns about a row it could not then read.+**The router announces its own rows instead.** A transaction that notifies+takes a database-wide lock until its commit is durable, so every message+announced from inside its own transaction waits on every other one: on Aurora+about 700 a second for the whole database. The router writes nearly every row,+so it sets `WRITER_ANNOUNCES` in its transaction, the trigger stands down for+that transaction, and the router hands the same announcement to the process's+`Announcer` once it has committed, which sends a batch in one transaction.+Every other writer still announces through the trigger.+One channel carries every room, and listeners filter. A channel per room wouldsave wakeups, but each one needs a `LISTEN` issued before the first row itshould catch, which is a race the consumer has to close by reading the table@@ -34,9 +43,18 @@ tests exercise the same trigger the server runs.from __future__ import annotations+import json+# Deliberately not per-room. See the module docstring.MESSAGE_CHANNEL = "switch_message"+# Set for one transaction by a writer that announces its rows itself, after+# its commit (`announcer.py`): the triggers on `messages` and `agent_events`+# then send nothing for that transaction. A writer that sets it must announce+# every row of either table it writes in that transaction.+WRITER_ANNOUNCES_SETTING = "switch.writer_announces"+SET_WRITER_ANNOUNCES = f"set_config('{WRITER_ANNOUNCES_SETTING}', 'on', true)"+NOTIFY_FUNCTION_NAME = "switch_notify_message"NOTIFY_TRIGGER_NAME = "messages_notify"@@ -50,6 +68,10 @@ BEGINIF NEW.seq <= 0 THENRETURN NULL;END IF;+ -- The writer announces this row itself, after its commit.+ IF current_setting('{WRITER_ANNOUNCES_SETTING}', true) = 'on' THEN+ RETURN NULL;+ END IF;PERFORM pg_notify('{MESSAGE_CHANNEL}',json_build_object(@@ -72,3 +94,11 @@ CREATE TRIGGER {NOTIFY_TRIGGER_NAME}DROP_NOTIFY_TRIGGER = f"DROP TRIGGER IF EXISTS {NOTIFY_TRIGGER_NAME} ON messages"DROP_NOTIFY_FUNCTION = f"DROP FUNCTION IF EXISTS {NOTIFY_FUNCTION_NAME}()"+++def message_announcement(room_id: str, seq: int, message_id: str, row_type: str) -> str:+ """What the trigger sends for a row, for a writer that announces it itself."""+ return json.dumps(+ {"room_id": room_id, "seq": seq, "id": message_id, "type": row_type},+ separators=(",", ":"),+ )diff --git a/core/switch_core/db/stores/message_store.py b/core/switch_core/db/stores/message_store.pyindex f70e3e62b..2312f4de4 100644--- a/core/switch_core/db/stores/message_store.py+++ b/core/switch_core/db/stores/message_store.py@@ -17,6 +17,7 @@ from sqlalchemy.ext.asyncio import AsyncSessionfrom switch_core.db.id_lists import any_offrom switch_core.db.models import Message, MessageAttachment+from switch_core.db.notify_ddl import SET_WRITER_ANNOUNCESfrom switch_core.db.raw import RawStatement@@ -26,6 +27,8 @@ class MessageStore:session: AsyncSession,message: Message,attachments: list[MessageAttachment],+ *,+ announced_by_writer: bool = False,) -> Message:"""Persist a message and its files as one unit.@@ -33,8 +36,15 @@ class MessageStore:they were sent in.The caller must not have set `seq`; this assigns it.++ With `announced_by_writer`, the triggers on `messages` and+ `agent_events` send nothing for the rest of the transaction, and the+ caller announces every row of either table it writes in it once it+ has committed (`WRITER_ANNOUNCES_SETTING` in `db/notify_ddl.py`)."""- message.seq = await self._next_seq(session, message.room_id)+ message.seq = await self._next_seq(+ session, message.room_id, announced_by_writer=announced_by_writer+ )session.add(message)await session.flush()for position, attachment in enumerate(attachments):@@ -49,6 +59,8 @@ class MessageStore:session: AsyncSession,message: Message,attachments: list[MessageAttachment],+ *,+ announced_by_writer: bool = False,) -> bool:"""`create`, unless a message with the same `transport_event_id` isalready stored, in which case nothing is written. Returns whether the@@ -58,8 +70,12 @@ class MessageStore:twice writes once. The check runs under the room's lock, which everywriter to the room holds to commit, so an earlier write of the samemessage has either committed and is found, or has not started.++ `announced_by_writer` as for `create`."""- message.seq = await self._next_seq(session, message.room_id)+ message.seq = await self._next_seq(+ session, message.room_id, announced_by_writer=announced_by_writer+ )if await self.get_by_transport_event_id(session, message.transport_event_id):return Falsesession.add(message)@@ -129,7 +145,9 @@ class MessageStore:)return min(int(result.scalar_one()), 0) - 1- async def _next_seq(self, session: AsyncSession, room_id: str) -> int:+ async def _next_seq(+ self, session: AsyncSession, room_id: str, *, announced_by_writer: bool = False+ ) -> int:"""The next position in this room, allocated in commit order.One above the room's highest, so live positions run 1, 2, 3 with no@@ -158,8 +176,12 @@ class MessageStore:The cost is per-room serialisation of a single INSERT. Rooms do notcontend with each other.++ A writer that announces its own rows sets that up in the same+ statement as the lock, so it costs no round trip of its own."""- await _LOCK_ROOM.all(session, {"room_id": room_id})+ lock = _LOCK_ROOM_ANNOUNCED_BY_WRITER if announced_by_writer else _LOCK_ROOM+ await lock.all(session, {"room_id": room_id})head = await _ROOM_HEAD.all(session, {"room_id": room_id})return int(head[0].head) + 1@@ -210,26 +232,30 @@ class MessageStore:return {room_id: heads.get(room_id, 0) for room_id in room_ids}async def rooms_past(- self, session: AsyncSession, cursors: dict[str, int]+ self,+ session: AsyncSession,+ cursors: dict[str, int],+ *,+ types: Collection[str] | None = None,) -> set[str]:"""Of the rooms in `cursors`, the ones with a row after their cursor,- in one query whose result is only those rooms."""+ in one query whose result is only those rooms. With `types`, a row+ of one of those types."""if not cursors:return set()position = values(column("room_id", Text), column("seq", BigInteger), name="position").data(list(cursors.items()))- result = await session.execute(- select(Message.room_id)- .join(- position,- and_(- Message.room_id == position.c.room_id,- Message.seq > position.c.seq,- ),- )- .distinct()+ query = select(Message.room_id).join(+ position,+ and_(+ Message.room_id == position.c.room_id,+ Message.seq > position.c.seq,+ ),)+ if types is not None:+ query = query.where(Message.event_type.in_(list(types)))+ result = await session.execute(query.distinct())return set(result.scalars().all())async def get_by_transport_event_id(@@ -389,6 +415,9 @@ class MessageStore:# (`db/raw.py`): every message an agent or a person sends takes the room's# lock and reads its head._LOCK_ROOM = RawStatement(text("SELECT pg_advisory_xact_lock(hashtext(:room_id))"))+_LOCK_ROOM_ANNOUNCED_BY_WRITER = RawStatement(+ text(f"SELECT pg_advisory_xact_lock(hashtext(:room_id)), {SET_WRITER_ANNOUNCES}")+)_ROOM_HEAD = RawStatement(select(func.coalesce(func.max(Message.seq), 0).label("head")).where(Message.room_id == bindparam("room_id")diff --git a/core/switch_core/delivery/recipient.py b/core/switch_core/delivery/recipient.pyindex 120d759d5..e05c9c000 100644--- a/core/switch_core/delivery/recipient.py+++ b/core/switch_core/delivery/recipient.py@@ -98,6 +98,7 @@ from switch_core.transport import (if TYPE_CHECKING:from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker+ from switch_core.announcer import Announcerfrom switch_core.bridges.agent.protocol.agent_connections import (AgentConnectionRegistry,)@@ -198,6 +199,10 @@ class RecipientServices:connections: AgentConnectionRegistryfrontend_base_url: str | Nonemessage_telemetry: MessageTelemetry+ # The process's announcer, through which the router announces the rows it+ # writes once they have committed (`announcer.py`). Without one, the+ # triggers announce them from inside the transaction, as they did.+ announcer: Announcer | None = Noneclass SwitchNotices(Protocol):diff --git a/core/switch_core/delivery/router.py b/core/switch_core/delivery/router.pyindex 8f74dc4fe..c6d397e39 100644--- a/core/switch_core/delivery/router.py+++ b/core/switch_core/delivery/router.py@@ -77,6 +77,10 @@ from switch_core.clients.actor import AgentActor, ClientConfig, SystemActorfrom switch_core.clients.admin_messages import ADMIN_MARKER, PLATFORM_MARKERfrom switch_core.clients.mention_warnings import MentionWarnings, unmatched_mentionsfrom switch_core.clients.room_meta import RoomMeta+from switch_core.db.agent_event_notify_ddl import (+ AGENT_EVENT_CHANNEL,+ agent_event_announcement,+)from switch_core.db.models import (Agent,AgentConnectionRecord,@@ -90,6 +94,7 @@ from switch_core.db.models import (SwitchCoreProcess,room_agents,)+from switch_core.db.notify_ddl import MESSAGE_CHANNEL, message_announcementfrom switch_core.db.raw import RawStatementfrom switch_core.db.session_scope import tenant_sessionfrom switch_core.db.stores.room_store import follow_ups_enabled_in, main_agent_in@@ -544,6 +549,7 @@ class Routed:seq: int,stored: list[tuple[str, int]],deadline_in: float | None,+ announcement: str | None = None,) -> None:self._router = routerself._reads = reads@@ -553,6 +559,9 @@ class Routed:self.seq = seqself._stored = storedself._deadline_in = deadline_in+ # What the trigger would have announced for the row, when the writer+ # announces it and its own events itself once it has committed.+ self._announcement = announcementdef committed(self) -> None:"""Wake the agents the row is for, and leave the replies it owes to@@ -575,6 +584,13 @@ class Routed:def _committed(self) -> None:router = self._router+ announcer = router.services.announcer+ if self._announcement is not None and announcer is not None:+ announcer.announce(MESSAGE_CHANNEL, self._announcement)+ for agent_id, seq in self._stored:+ announcer.announce(+ AGENT_EVENT_CHANNEL, agent_event_announcement(agent_id, seq)+ )for agent_id, seq in self._stored:router.services.event_buffer.ring(agent_id, seq)reads = self._reads@@ -880,10 +896,22 @@ class MessageRouter:plan = await self._decide(session, reads, unwritten, sender_client_id=message.sender_client_id)+ # With an announcer, the row and its own events are announced once the+ # caller has committed (`Routed.committed`), in a batch, rather than by+ # the triggers from inside this transaction (`announcer.py`).+ announced_by_writer = self.services.announcer is not Noneif not unless_stored:- await self._messages.create(session, message, attachments)+ await self._messages.create(+ session,+ message,+ attachments,+ announced_by_writer=announced_by_writer,+ )elif not await self._messages.create_unless_stored(- session, message, attachments+ session,+ message,+ attachments,+ announced_by_writer=announced_by_writer,):return Noneif reads is not None and await self._joined_since(session, reads):@@ -922,6 +950,13 @@ class MessageRouter:seq=message.seq,stored=stored,deadline_in=deadline_in,+ announcement=(+ message_announcement(+ room_id, message.seq, message.id, message.event_type+ )+ if announced_by_writer+ else None+ ),)@staticmethoddiff --git a/core/switch_core/main.py b/core/switch_core/main.pyindex c3bbdf146..23548d9d5 100644--- a/core/switch_core/main.py+++ b/core/switch_core/main.py@@ -27,6 +27,7 @@ from sqlalchemy.ext.asyncio import ()from sqlalchemy.pool import NullPool+from switch_core.announcer import Announcerfrom switch_core.bridges.agent.app import create_agent_bridge_appfrom switch_core.bridges.agent.protocol.agent_connections import (HEARTBEAT_TTL_SECONDS,@@ -672,12 +673,17 @@ async def run(config: SwitchConfig, process_id: str) -> None:session_activity_listener = AgentSessionActivityListener(lambda: create_unpooled_engine(config))+ # Every `NOTIFY` this process sends from its own code, after the commits+ # they announce, one transaction per batch (`announcer.py`): the router's+ # messages and own events, and the signals below.+ announcer = Announcer(session_factory)# Signals between switch-core processes (see `signals.py`): the same# arrangement again, carrying "who should look" rather than a room's rows.signal_bus = SignalBus(session_factory,lambda: create_unpooled_engine(config),process_id=process_id,+ announcer=announcer,)# This process's row in `switch_core_processes`, claimed before anything# can record a connection against it and held until the process drains.@@ -865,6 +871,7 @@ async def run(config: SwitchConfig, process_id: str) -> None:connections=connections,frontend_base_url=config.dashboard_base_url,message_telemetry=message_telemetry,+ announcer=announcer,),message_store=message_store,client_store=client_store,@@ -1451,6 +1458,7 @@ async def run(config: SwitchConfig, process_id: str) -> None:if telemetry.enabledelse None)+ await announcer.start()await message_listener.start()await session_activity_listener.start()await agent_event_listener.start()@@ -1485,6 +1493,9 @@ async def run(config: SwitchConfig, process_id: str) -> None:await session_activity_listener.stop()await agent_event_listener.stop()await signal_bus.stop()+ # Last of them: what the router and the bus handed it is sent,+ # for a few seconds at most, before the pool closes.+ await announcer.stop()# Before the operational flush, and bounded: the whole# teardown runs inside `_FORCED_EXIT_GRACE_SECONDS` and a# product event is the least valuable thing in it.diff --git a/core/switch_core/migrations/versions/ad15e53358fe_writers_announce_after_commit.py b/core/switch_core/migrations/versions/ad15e53358fe_writers_announce_after_commit.pynew file mode 100644index 000000000..bbaf84378--- /dev/null+++ b/core/switch_core/migrations/versions/ad15e53358fe_writers_announce_after_commit.py@@ -0,0 +1,114 @@+"""Let a writer announce its messages and own events after its commit++A transaction that sends a `NOTIFY` takes a lock that every other notifying+transaction waits on until its commit is durable: on Aurora about 700 such+commits a second for the whole database, whatever its size. The triggers on+`messages` and `agent_events` notified from inside every message's own+transaction, so every message in the system waited on that lock.++The triggers now stand down for a transaction that sets+`switch.writer_announces` to `on`. The router sets it, and once its+transaction has committed it hands the same announcements to the process's+`Announcer`, which sends them in batches, one transaction each. Every other+writer still announces through the triggers.++Expand only: a process from before this revision never sets the setting, so+for it the triggers announce as they did, and its listeners hear this+release's announcements on the same channels with the same payloads.++Revision ID: ad15e53358fe+Revises: 4103210581d6+Create Date: 2026-11-02 00:00:00.000000+"""++from alembic import op++revision: str = "ad15e53358fe"+down_revision: str | None = "4103210581d6"+branch_labels = None+depends_on = None+expand_only: bool = True+++def upgrade() -> None:+ op.execute(_MESSAGE_WRITER_ANNOUNCES)+ op.execute(_AGENT_EVENT_WRITER_ANNOUNCES)+++def downgrade() -> None:+ op.execute(_AGENT_EVENT_ALWAYS)+ op.execute(_MESSAGE_ALWAYS)+++# Frozen copies of `switch_core.db.notify_ddl` and+# `switch_core.db.agent_event_notify_ddl` as of this revision and before it;+# the live modules may change later, this migration must not.+_MESSAGE_WRITER_ANNOUNCES = """+CREATE OR REPLACE FUNCTION switch_notify_message() RETURNS trigger AS $$+BEGIN+ IF NEW.seq <= 0 THEN+ RETURN NULL;+ END IF;+ IF current_setting('switch.writer_announces', true) = 'on' THEN+ RETURN NULL;+ END IF;+ PERFORM pg_notify(+ 'switch_message',+ json_build_object(+ 'room_id', NEW.room_id,+ 'seq', NEW.seq,+ 'id', NEW.id,+ 'type', NEW.event_type+ )::text+ );+ RETURN NULL;+END;+$$ LANGUAGE plpgsql+"""++_AGENT_EVENT_WRITER_ANNOUNCES = """+CREATE OR REPLACE FUNCTION switch_notify_agent_event() RETURNS trigger AS $$+BEGIN+ IF current_setting('switch.writer_announces', true) = 'on' THEN+ RETURN NULL;+ END IF;+ PERFORM pg_notify(+ 'switch_agent_event',+ json_build_object('agent_id', NEW.agent_id, 'seq', NEW.seq)::text+ );+ RETURN NULL;+END;+$$ LANGUAGE plpgsql+"""++_MESSAGE_ALWAYS = """+CREATE OR REPLACE FUNCTION switch_notify_message() RETURNS trigger AS $$+BEGIN+ IF NEW.seq <= 0 THEN+ RETURN NULL;+ END IF;+ PERFORM pg_notify(+ 'switch_message',+ json_build_object(+ 'room_id', NEW.room_id,+ 'seq', NEW.seq,+ 'id', NEW.id,+ 'type', NEW.event_type+ )::text+ );+ RETURN NULL;+END;+$$ LANGUAGE plpgsql+"""++_AGENT_EVENT_ALWAYS = """+CREATE OR REPLACE FUNCTION switch_notify_agent_event() RETURNS trigger AS $$+BEGIN+ PERFORM pg_notify(+ 'switch_agent_event',+ json_build_object('agent_id', NEW.agent_id, 'seq', NEW.seq)::text+ );+ RETURN NULL;+END;+$$ LANGUAGE plpgsql+"""diff --git a/core/switch_core/signals.py b/core/switch_core/signals.pyindex ea827c7d0..55088d7fe 100644--- a/core/switch_core/signals.py+++ b/core/switch_core/signals.py@@ -12,17 +12,19 @@ own: the holder of a controller's socket may be any process, and the databaseis the one path that reaches it wherever it is.Publishing is batched. A transaction that sends a `NOTIFY` takes a lock that-every other notifying transaction waits on as it commits, so signals raised-together go out in one statement rather than one transaction each. Payloads-are ids and small fields only: `NOTIFY` is not subject to row-level security,-and any session listening on the channel can read what is sent.+every other notifying transaction waits on as it commits, so signals go out+through the process's `Announcer`, with everything else it announces, in one+transaction per batch rather than one each. Payloads are ids and small fields+only: `NOTIFY` is not subject to row-level security, and any session listening+on the channel can read what is sent.Reconnecting resyncs. Signals sent while the listening connection was downwere never queued, so every registered resync runs once the connection is back,and each re-reads what its clients are owed.Nothing here is allowed to lose a change for good. A batch that fails to send-is kept and sent again, as the controller connection ledger keeps its writes.+is kept and sent again (`Announcer`), as the controller connection ledger keeps+its writes.And because a signal can still be missed (its sender died between the commitand the send, or a receiver was busy restarting), every family also registersa periodic re-check that compares what this process holds with the database.@@ -44,7 +46,7 @@ import loggingfrom collections.abc import Awaitable, Callable, Hashablefrom typing import TYPE_CHECKING, Any-from sqlalchemy import text+from switch_core.announcer import MAX_PAYLOAD_BYTES, Announcerif TYPE_CHECKING:from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker@@ -60,19 +62,9 @@ HEARTBEAT_TIMEOUT_SECONDS = 60.0RECONNECT_BACKOFF_BASE = 1.0RECONNECT_BACKOFF_CAP = 15.0-# Postgres refuses a payload of 8000 bytes or more. Ours are a few dozen.-MAX_PAYLOAD_BYTES = 7999--# How long a batch that failed to send, or a key whose re-read failed, waits-# before it is tried again.+# How long a key whose re-read failed waits before it is tried again.RETRY_SECONDS = 2.0-# How many unsent signals are kept while the database refuses them. Past this-# the oldest are dropped and logged: by then the database has been unreachable-# for a long while, and the periodic re-checks bring every process back in-# line once it returns, so holding more would only cost memory.-MAX_UNSENT = 10_000-# How often every family's re-check runs, so that a signal missed for any# reason costs at most this long rather than lasting until a restart.RECHECK_SECONDS = 30.0@@ -82,10 +74,6 @@ Resync = Callable[[], Awaitable[None]]Recheck = Callable[[], Awaitable[None]]EngineFactory = Callable[[], "AsyncEngine"]-_NOTIFY_MANY = text(- "SELECT pg_notify(:channel, payload) FROM unnest(CAST(:payloads AS text[])) AS payload"-)-class PendingWork[K: Hashable]:"""Keys whose state has to be re-read, worked one at a time by one task.@@ -155,8 +143,14 @@ class SignalBus:session_factory: async_sessionmaker[AsyncSession],engine_factory: EngineFactory,process_id: str,+ announcer: Announcer | None = None,) -> None:self._session_factory = session_factory+ # The process's announcer, so signals go out in the same transactions+ # as everything else it announces; one of its own when none is given,+ # started and stopped with the bus.+ self._announcer = announcer or Announcer(session_factory)+ self._owns_announcer = announcer is None# Signals carry the sender's process id, and a process skips its own:# whoever publishes has already acted on the change locally.self.process_id = process_id@@ -166,8 +160,6 @@ class SignalBus:self._resyncs: list[Resync] = []self._rechecks: list[Recheck] = []self._work: list[PendingWork[Any]] = []- self._outbox: list[str] = []- self._has_outbox = asyncio.Event()self._inbox: asyncio.Queue[dict[str, Any]] = asyncio.Queue()self._tasks: list[asyncio.Task[None]] = []self._running = False@@ -228,65 +220,11 @@ class SignalBus:raise ValueError(f"Signal {kind!r} is too large to send ({len(payload)} bytes)")- self._outbox.append(payload)- self._has_outbox.set()+ self._announcer.announce(SIGNAL_CHANNEL, payload)@propertydef unsent(self) -> int:- return len(self._outbox)-- async def _send_forever(self) -> None:- while self._running:- await self._has_outbox.wait()- self._has_outbox.clear()- batch, self._outbox = self._outbox, []- if not batch:- continue- # Set once the batch's transaction has committed, which is when- # Postgres sends it: what fails after that (closing the session)- # did not stop it going, and keeping it would send it twice.- committed = False- try:- async with self._session_factory() as session:- await session.execute(- _NOTIFY_MANY, {"channel": SIGNAL_CHANNEL, "payloads": batch}- )- await session.commit()- committed = True- except asyncio.CancelledError:- if not committed:- self._outbox = batch + self._outbox- raise- except Exception:- if committed:- logger.warning(- "Sent %d signal(s), then closing the session failed",- len(batch),- exc_info=True,- )- continue- # The receivers' listening connections did not drop, so none- # of them will resync: a batch dropped here would be lost- # until the next re-check. Kept ahead of anything published- # since, so a process still receives one sender's signals in- # the order they were raised.- self._outbox = batch + self._outbox- if len(self._outbox) > MAX_UNSENT:- dropped = len(self._outbox) - MAX_UNSENT- self._outbox = self._outbox[dropped:]- logger.error(- "Dropped the %d oldest unsent signal(s); the periodic "- "re-checks repair what they announced",- dropped,- )- logger.error(- "Could not send %d signal(s); kept and retried in %.0fs",- len(self._outbox),- RETRY_SECONDS,- exc_info=True,- )- await asyncio.sleep(RETRY_SECONDS)- self._has_outbox.set()+ return self._announcer.unsent# ── Lifecycle ─────────────────────────────────────────────────────────────@@ -294,10 +232,11 @@ class SignalBus:if self._running:raise RuntimeError("SignalBus is already started")self._running = True+ if self._owns_announcer:+ await self._announcer.start()self._tasks = [asyncio.create_task(self._listen_forever(), name="signal-listener"),asyncio.create_task(self._dispatch_forever(), name="signal-dispatch"),- asyncio.create_task(self._send_forever(), name="signal-sender"),asyncio.create_task(self._recheck_forever(), name="signal-recheck"),]self._tasks += [@@ -313,6 +252,8 @@ class SignalBus:with contextlib.suppress(asyncio.CancelledError):await taskself._tasks = []+ if self._owns_announcer:+ await self._announcer.stop()self.connected.clear()# ── The listening connection ──────────────────────────────────────────────diff --git a/core/switch_core/transport/postgres.py b/core/switch_core/transport/postgres.pyindex 69b47dd1c..1f8b8ba0e 100644--- a/core/switch_core/transport/postgres.py+++ b/core/switch_core/transport/postgres.py@@ -41,6 +41,7 @@ Writing needs nothing of the kind: any process may write as any client.from __future__ import annotationsimport asyncio+import contextlibimport loggingimport uuidfrom collections.abc import Awaitable, Callable@@ -174,6 +175,10 @@ MEMBERSHIP_EVENT_TYPE = "m.room.member"# before what is left of it is cancelled.STOP_GRACE_SECONDS = 5.0+# How often a reader asks which of its rooms have a row past where it read+# (`PostgresTransport._recheck_forever`).+RECHECK_SECONDS = 30.0+class RoomIds:"""The room id behind each transport room id a process's transports@@ -417,6 +422,7 @@ class PostgresTransport:self._receiving_event.set()try:delivery = asyncio.create_task(self._deliver_forever())+ recheck = asyncio.create_task(self._recheck_forever())try:async with self._reading_rooms:rooms = await self.joined_rooms()@@ -433,6 +439,9 @@ class PostgresTransport:await self._watch_joined(rooms)await self._closed.wait()finally:+ recheck.cancel()+ with contextlib.suppress(asyncio.CancelledError):+ await recheckawait self._finish(delivery)finally:self._receiving = False@@ -447,6 +456,46 @@ class PostgresTransport:self._progress = Noneself._recorded.clear()+ async def _recheck_forever(self) -> None:+ """Every `RECHECK_SECONDS`, wake the rooms with a row past where this+ client read, found in one query.++ The process that writes a room's row announces it after its commit+ (`announcer.py`), and one that died between the two announced+ nothing: without this, the row would wait for the room's next one.+ A wake costs a read from the cursor, so a room woken for a row its+ announcement was about to bring reads it a moment early.+ """+ with no_tenant():+ while not self._stopping:+ await asyncio.sleep(RECHECK_SECONDS)+ cursors = {+ room_id: cursor+ for room_id, cursor in self._cursors.items()+ if room_id in self._watching+ }+ if not cursors:+ continue+ try:+ async with tenant_session(+ self._session_factory, self.tenant_id+ ) as session:+ moved = await self._message_store.rooms_past(+ session, cursors, types=self._row_types+ )+ except Exception:+ logger.warning(+ "Client %s could not check its rooms for rows it was "+ "not told of; trying again in %.0fs",+ self.user_id,+ RECHECK_SECONDS,+ exc_info=True,+ )+ continue+ for room_id in moved:+ if room_id in self._watching:+ self._mark_pending(room_id)+async def _finish(self, delivery: asyncio.Task[None]) -> None:"""End the delivery loop after the row in hand, or cancel it once`STOP_GRACE_SECONDS` have passed. A row cut short by the cancel is notdiff --git a/core/tests/routing.py b/core/tests/routing.pyindex ad1725f98..169b52076 100644--- a/core/tests/routing.py+++ b/core/tests/routing.py@@ -22,6 +22,7 @@ from typing import TYPE_CHECKING, Anyfrom sqlalchemy import insert, select+from switch_core.announcer import Announcerfrom switch_core.bridges.agent.protocol.agent_connections import AgentConnectionRegistryfrom switch_core.bridges.agent.protocol.event_buffer import (EventBuffer,@@ -197,10 +198,13 @@ def router_on(connections: AgentConnectionRegistry | None = None,telemetry: RecordedTelemetry | None = None,conversation: ConversationSettings | None = None,+ announcer: Announcer | None = None,) -> MessageRouter:"""A router on the test database with real stores. A test that has noagent in its rooms can write through it with nothing else set up.- Following the conversation is off unless `conversation` turns it on."""+ Following the conversation is off unless `conversation` turns it on.+ With `announcer`, it announces its rows after their commits, as the+ server's does; without, the triggers announce them."""return MessageRouter(services=RecipientServices(session_factory=session_factory,@@ -222,6 +226,7 @@ def router_on(message_telemetry=telemetry # type: ignore[arg-type]if telemetry is not Noneelse RecordedTelemetry(),+ announcer=announcer,),message_store=MessageStore(),client_store=ClientStore(),@@ -244,6 +249,7 @@ def process_on(rooms: RoomAnnouncements | None = None,connections: AgentConnectionRegistry | None = None,conversation: ConversationSettings | None = None,+ announcer: Announcer | None = None,) -> Process:"""A process's routing on the test database: real stores, its ownregistry and buffer, telemetry that records. A test of presence across@@ -262,6 +268,7 @@ def process_on(connections=connections,telemetry=telemetry,conversation=conversation,+ announcer=announcer,)invites = InviteBus()factory = ClientFactory(diff --git a/core/tests/switch_core/delivery/test_router_announces.py b/core/tests/switch_core/delivery/test_router_announces.pynew file mode 100644index 000000000..459f6954c--- /dev/null+++ b/core/tests/switch_core/delivery/test_router_announces.py@@ -0,0 +1,296 @@+"""The router announces the rows it writes once they have committed.++A transaction that sends a `NOTIFY` holds every other notifying commit in the+database back until it is durable, so the triggers on `messages` and+`agent_events` stand down for the router's transactions, and the router hands+the same announcements to the process's `Announcer` after its commit.++Against real PostgreSQL with a real `LISTEN`: what a listener hears is the+point. Every other writer still announces through the triggers, and listeners+re-check on a timer for an announcement a dying process never sent.+"""++from __future__ import annotations++import asyncio+import json+from typing import TYPE_CHECKING++import pytest+from sqlalchemy import select+from sqlalchemy.ext.asyncio import create_async_engine+from sqlalchemy.pool import NullPool++from switch_core.announcer import Announcer+from switch_core.bridges.agent.protocol import (+ agent_event_listener as agent_event_listener_module,+)+from switch_core.bridges.agent.protocol.agent_connections import (+ PROTOCOL_VERSION,+ ClientDeclaration,+)+from switch_core.bridges.agent.protocol.agent_event_listener import AgentEventListener+from switch_core.bridges.agent.protocol.positions import OWN_EVENTS+from switch_core.db.agent_event_notify_ddl import (+ AGENT_EVENT_CHANNEL,+ agent_event_announcement,+)+from switch_core.db.models import TENANT_ZERO_ID, Message+from switch_core.db.notify_ddl import MESSAGE_CHANNEL, message_announcement+from switch_core.db.session_scope import tenant_session+from switch_core.db.stores.message_store import MessageStore+from switch_core.messages.row import message_row+from tests.routing import (+ Process,+ add_member,+ make_agent,+ make_person,+ make_room,+ own_events,+ process_on,+)+from tests.switch_core.test_announcer import heard++if TYPE_CHECKING:+ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++ from switch_core.db.models import Agent, Client, Room+++async def _room_with_an_agent(+ session_factory: async_sessionmaker[AsyncSession], process: Process+) -> tuple[Room, Client, Agent]:+ room = await make_room(session_factory)+ louisa = await make_person(session_factory, "louisa")+ await add_member(session_factory, room, louisa)+ agent, client = await make_agent(+ session_factory, "member", connection_model="session_addressable"+ )+ await add_member(session_factory, room, client, agent=agent)+ # Connected, so a message addressed to it is its own event and nothing+ # more: an agent away would also be sent a reply saying so.+ process.connections.open(+ agent_id=agent.id,+ connection_id="c1",+ scope="single",+ delivery_filter="all",+ spawn_capable=False,+ position={OWN_EVENTS: 0},+ declaration=ClientDeclaration(speaks=PROTOCOL_VERSION),+ expected_generation=None,+ )+ # Louisa's transport knows the room from here on.+ await process.say(room, "hello", sender=louisa)+ return room, louisa, agent+++async def _written_since(+ session_factory: async_sessionmaker[AsyncSession],+ room: Room,+ agent: Agent,+ *,+ after_seq: int,+ after_event: int,+) -> list[tuple[str, dict[str, object]]]:+ """What the triggers would have announced for the rows written past+ those positions: each message of the room, and each own event of the+ agent."""+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:+ messages = (+ await session.scalars(+ select(Message)+ .where(Message.room_id == room.id, Message.seq > after_seq)+ .order_by(Message.seq)+ )+ ).all()+ events = [+ event+ for event in await own_events(session_factory, agent.id)+ if event.seq > after_event+ ]+ assert messages and events+ return [+ (+ MESSAGE_CHANNEL,+ json.loads(+ message_announcement(+ room.id, message.seq, message.id, message.event_type+ )+ ),+ )+ for message in messages+ ] + [+ (AGENT_EVENT_CHANNEL, json.loads(agent_event_announcement(agent.id, event.seq)))+ for event in events+ ]+++async def _positions(+ session_factory: async_sessionmaker[AsyncSession], room: Room, agent: Agent+) -> tuple[int, int]:+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:+ head = await MessageStore().head_seq(session, room.id)+ events = await own_events(session_factory, agent.id)+ return head, max((event.seq for event in events), default=0)+++def _sorted(+ announcements: list[tuple[str, dict[str, object]]],+) -> list[tuple[str, dict[str, object]]]:+ return sorted(announcements, key=lambda each: (each[0], json.dumps(each[1])))+++class TestAfterTheCommit:+ async def test_every_row_is_announced_once_as_the_trigger_would(+ self, postgres_url: str, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """Once each, with the payloads the triggers send: the triggers sent+ nothing for the router's transactions."""+ announcer = Announcer(session_factory)+ process = process_on(session_factory, announcer=announcer)+ room, louisa, agent = await _room_with_an_agent(session_factory, process)+ after_seq, after_event = await _positions(session_factory, room, agent)+ await announcer.start()++ async with heard(postgres_url) as announcements:+ await process.say(room, "@member can you look at this", sender=louisa)+ await process.idle()+ await announcer.stop()++ written = await _written_since(+ session_factory, room, agent, after_seq=after_seq, after_event=after_event+ )+ heard_now = [+ (channel, json.loads(payload)) for channel, payload in announcements+ ]+ assert _sorted(heard_now) == _sorted(written)+ # A message is announced before the own events it caused.+ assert heard_now[0][0] == MESSAGE_CHANNEL++ async def test_without_an_announcer_the_triggers_announce_the_same(+ self, postgres_url: str, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """A process without an announcer leaves it to the triggers, as a+ process of an older release does; a listener hears the same."""+ process = process_on(session_factory)+ room, louisa, agent = await _room_with_an_agent(session_factory, process)+ after_seq, after_event = await _positions(session_factory, room, agent)++ async with heard(postgres_url) as announcements:+ await process.say(room, "@member can you look at this", sender=louisa)+ await process.idle()++ written = await _written_since(+ session_factory, room, agent, after_seq=after_seq, after_event=after_event+ )+ heard_now = [+ (channel, json.loads(payload)) for channel, payload in announcements+ ]+ assert _sorted(heard_now) == _sorted(written)++ async def test_a_write_that_rolls_back_announces_nothing(+ self, postgres_url: str, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ announcer = Announcer(session_factory)+ process = process_on(session_factory, announcer=announcer)+ room, louisa, _ = await _room_with_an_agent(session_factory, process)+ await announcer.start()++ async with heard(postgres_url) as announcements:+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:+ routed = await process.router.store(+ session,+ tenant_id=TENANT_ZERO_ID,+ room_id=room.id,+ transport_room_id=room.transport_room_id,+ message=message_row(+ room_id=room.id,+ event_id="$never-committed",+ sender_id=louisa.transport_user_id,+ sender_client_id=louisa.id,+ sender_name=louisa.display_name,+ event_type="m.room.message",+ content={"msgtype": "m.text", "body": "@member never mind"},+ client_txn_id=None,+ ),+ attachments=[],+ )+ assert routed is not None+ await session.rollback()+ await announcer.stop()++ assert announcements == []++ async def test_a_writer_that_does_not_announce_is_still_heard(+ self, postgres_url: str, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """Only a transaction that says it announces its rows itself turns+ the triggers off: a row written any other way announces itself."""+ process = process_on(session_factory)+ room, louisa, _ = await _room_with_an_agent(session_factory, process)++ async with heard(postgres_url, (MESSAGE_CHANNEL,)) as announcements:+ async with tenant_session(session_factory, TENANT_ZERO_ID) as session:+ message = message_row(+ room_id=room.id,+ event_id="$by-hand",+ sender_id=louisa.transport_user_id,+ sender_client_id=louisa.id,+ sender_name=louisa.display_name,+ event_type="m.room.message",+ content={"msgtype": "m.text", "body": "written by hand"},+ client_txn_id=None,+ )+ await MessageStore().create(session, message, [])+ await session.commit()++ assert [json.loads(payload)["id"] for _, payload in announcements] == [+ message.id+ ]+++class TestALostAnnouncement:+ async def test_the_heads_told_are_forgotten_and_read_again(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ process = process_on(session_factory)+ room, louisa, agent = await _room_with_an_agent(session_factory, process)+ process.buffer.listening(True)+ process.buffer.doorbell(agent.id)+ await process.say(room, "@member can you look at this", sender=louisa)+ await process.idle()+ [*_, event] = await own_events(session_factory, agent.id)+ assert process.buffer._told_heads == {agent.id: event.seq}++ process.buffer.forget_told_heads()++ assert process.buffer._told_heads == {}+ heads = await process.buffer.heads(TENANT_ZERO_ID, {agent.id})+ assert heads == {agent.id: event.seq}++ async def test_the_listener_forgets_them_every_heartbeat(+ self, postgres_url: str, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setattr(agent_event_listener_module, "HEARTBEAT_SECONDS", 0.02)++ class _Buffer:+ forgotten = 0++ def listening(self, _now: bool) -> None:+ pass++ def forget_told_heads(self) -> None:+ self.forgotten += 1++ buffer = _Buffer()+ listener = AgentEventListener(+ lambda: create_async_engine(postgres_url, poolclass=NullPool),+ buffer, # type: ignore[arg-type]+ )+ await listener.start()+ try:+ async with asyncio.timeout(10):+ while buffer.forgotten < 2:+ await asyncio.sleep(0.01)+ finally:+ await listener.stop()diff --git a/core/tests/switch_core/test_announcer.py b/core/tests/switch_core/test_announcer.pynew file mode 100644index 000000000..cc4a10f09--- /dev/null+++ b/core/tests/switch_core/test_announcer.py@@ -0,0 +1,141 @@+"""The process's announcements, sent after their commits in batches.++Against real PostgreSQL, with a `LISTEN` standing in for the other processes:+what matters is what they hear, in what order, and how many transactions it+took to send it.+"""++from __future__ import annotations++import asyncio+import contextlib+import json+from typing import TYPE_CHECKING, Any++import pytest+from sqlalchemy.ext.asyncio import create_async_engine+from sqlalchemy.pool import NullPool++from switch_core import announcer as announcer_module+from switch_core.announcer import Announcer+from tests.session_counting import CountingSessionFactory+from tests.switch_core.test_signals import _FailingOnce, _RefusingSession++if TYPE_CHECKING:+ from collections.abc import AsyncIterator++ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker++CHANNELS = ("switch_message", "switch_agent_event")+_SENTINEL = "heard-everything"+++async def heard(+ postgres_url: str, channels: tuple[str, ...] = CHANNELS+) -> AsyncIterator[list[tuple[str, str]]]:+ """What was announced on `channels` while the block ran, in the order it+ arrived. A sentinel goes out on each channel after the block, and+ announcements arrive in commit order, so once every sentinel is heard,+ everything committed before them has been heard too."""+ engine = create_async_engine(postgres_url, poolclass=NullPool)+ announcements: list[tuple[str, str]] = []+ waiting = set(channels)+ done = asyncio.Event()++ def on_notify(_conn: Any, _pid: int, channel: str, payload: str) -> None:+ if payload == _SENTINEL:+ waiting.discard(channel)+ if not waiting:+ done.set()+ else:+ announcements.append((channel, payload))++ try:+ async with engine.connect() as connection:+ raw = await connection.get_raw_connection()+ driver = raw.driver_connection+ assert driver is not None+ for channel in channels:+ await driver.add_listener(channel, on_notify)+ yield announcements+ for channel in channels:+ await driver.execute(f"NOTIFY {channel}, '{_SENTINEL}'")+ await asyncio.wait_for(done.wait(), 10)+ finally:+ await engine.dispose()+++class TestSending:+ async def test_announcements_arrive_on_their_channels_in_order(+ self, postgres_url: str, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ announcer = Announcer(session_factory)+ sent = [+ (channel, json.dumps({"n": n})) for n in range(3) for channel in CHANNELS+ ]++ async with heard(postgres_url) as announcements:+ await announcer.start()+ for channel, payload in sent:+ announcer.announce(channel, payload)+ await announcer.stop()++ assert announcements == sent++ async def test_what_is_handed_over_together_goes_in_one_transaction(+ self, postgres_url: str, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """The lock a notifying commit takes is taken once per transaction,+ so a hundred announcements cost the database one."""+ counting = CountingSessionFactory(session_factory)+ announcer = Announcer(counting) # type: ignore[arg-type]+ for n in range(100):+ announcer.announce("switch_message", json.dumps({"n": n}))++ async with heard(postgres_url) as announcements:+ await announcer.start()+ await announcer.stop()++ assert len(announcements) == 100+ assert counting.opened == 1++ def test_an_oversized_announcement_is_refused(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ with pytest.raises(ValueError, match="too large"):+ Announcer(session_factory).announce("switch_message", "x" * 8000)+++class TestNothingIsLost:+ async def test_a_batch_that_fails_is_sent_again_ahead_of_what_came_after(+ self,+ postgres_url: str,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ ) -> None:+ monkeypatch.setattr(announcer_module, "RETRY_SECONDS", 0.05)+ announcer = Announcer(_FailingOnce(session_factory)) # type: ignore[arg-type]++ async with heard(postgres_url) as announcements:+ await announcer.start()+ announcer.announce("switch_message", "first")+ await asyncio.sleep(0.01)+ announcer.announce("switch_message", "second")+ await announcer.stop()++ assert [payload for _, payload in announcements] == ["first", "second"]+ assert announcer.unsent == 0++ async def test_stopping_waits_a_while_then_gives_up_on_a_refusing_database(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setattr(announcer_module, "RETRY_SECONDS", 0.02)+ monkeypatch.setattr(announcer_module, "FLUSH_SECONDS", 0.1)+ announcer = Announcer(lambda: _RefusingSession()) # type: ignore[arg-type,return-value]++ await announcer.start()+ announcer.announce("switch_message", "never sent")+ await announcer.stop()++ assert announcer.unsent == 1diff --git a/core/tests/switch_core/test_signals.py b/core/tests/switch_core/test_signals.pyindex 665986565..5eef8a1a4 100644--- a/core/tests/switch_core/test_signals.py+++ b/core/tests/switch_core/test_signals.py@@ -16,6 +16,7 @@ from sqlalchemy import textfrom sqlalchemy.ext.asyncio import create_async_enginefrom sqlalchemy.pool import NullPool+from switch_core import announcer as announcer_modulefrom switch_core import signals as signals_modulefrom switch_core.signals import SIGNAL_CHANNEL, SignalBus@@ -204,7 +205,7 @@ class TestNothingIsLost:make_bus: Callable[[], SignalBus],monkeypatch: pytest.MonkeyPatch,) -> None:- monkeypatch.setattr(signals_module, "RETRY_SECONDS", 0.05)+ monkeypatch.setattr(announcer_module, "RETRY_SECONDS", 0.05)sender = SignalBus(_FailingOnce(session_factory), # type: ignore[arg-type]lambda: create_async_engine(postgres_url, poolclass=NullPool),diff --git a/core/tests/switch_core/transport/test_postgres_transport.py b/core/tests/switch_core/transport/test_postgres_transport.pyindex daaeb9597..ab61a627e 100644--- a/core/tests/switch_core/transport/test_postgres_transport.py+++ b/core/tests/switch_core/transport/test_postgres_transport.py@@ -906,6 +906,78 @@ class TestReceiving:# the internal one the listener announces.assert event.room_id == room+ async def test_a_row_nobody_announced_is_found_by_the_recheck(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ ) -> None:+ """The process that writes a row announces it after its commit, and+ one that died between the two announced nothing: the reader asks+ which of its rooms moved past where it read."""+ monkeypatch.setattr(postgres_module, "RECHECK_SECONDS", 0.05)+ transport, _, received, room, _ = await self._receiving(session_factory)++ sent = await transport.send_message(+ room, "never announced", sender_name="agent one", metered=True+ )+ async with asyncio.timeout(10):+ while not received.events:+ await asyncio.sleep(0.01)++ assert [event.event_id for event in received.events] == [sent.event_id] # type: ignore[attr-defined]++ async def test_a_reader_of_some_types_is_rechecked_for_those_only(+ self,+ session_factory: async_sessionmaker[AsyncSession],+ monkeypatch: pytest.MonkeyPatch,+ ) -> None:+ """Switch's own reader's cursor moves only with command rows, so the+ rows of every other type past it must not wake it at every check."""+ monkeypatch.setattr(postgres_module, "RECHECK_SECONDS", 0.02)+ async with session_factory() as session:+ _, room, client_id, user_id = await _make_room(session)+ await session.commit()+ received = _Received()+ reader = _transport(+ session_factory,+ client_id=client_id,+ user_id=user_id,+ listener=_FakeListener(),+ )+ reader.register_handlers(received.handlers())+ await reader.join_room(room)+ task = asyncio.create_task(+ reader.receive_forever(+ FromHead(MessageStore()),+ row_types=frozenset({"com.switch.command"}),+ )+ )+ self._tasks.append((task, reader))+ await _watched_room(reader)+ woken: list[str] = []+ mark = reader._mark_pending++ def counting(room_id: str) -> None:+ woken.append(room_id)+ mark(room_id)++ reader._mark_pending = counting # type: ignore[method-assign]++ await reader.send_message(room, "chat", sender_name="a", metered=True)+ await asyncio.sleep(0.2)+ assert woken == []++ sent = await reader.send_event(+ room,+ "com.switch.command",+ {"command": "help", "args": "", "user_id": user_id, "user_name": "a"},+ )+ async with asyncio.timeout(10):+ while not received.events:+ await asyncio.sleep(0.01)++ assert [event.event_id for event in received.events] == [sent.event_id] # type: ignore[attr-defined]+async def test_a_reader_of_some_types_is_handed_only_those(self, session_factory: async_sessionmaker[AsyncSession]) -> None: