Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Merge remote-tracking branch 'origin/admin-reads-commands' into scale-core

Abel Dantas15d446cdb925
diff --cc CHANGELOG.md
index b7474239c,c87954044..4250e777d
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -171,43 -171,21 +171,58 @@@ version of their own to them without al
them, and to roll back, first downgrade the database to the revision before
`238c68b7306e` (`alembic downgrade 238c68b7306e-1`), then start the older
release.
+- **The statements every message write costs run on asyncpg directly.**
+ Taking the room's lock, reading its head, storing the agents' own events
+ and reading the room's agent members again under the lock go straight to
+ asyncpg on the transaction's connection (`db/raw.py`), as the delivery
+ reads do, rather than through SQLAlchemy's execution.
+- **The two reads behind every delivery run on asyncpg directly.** A
+ stream reads an agent's events, and counts its unread messages, for every
+ event it delivers, and under load SQLAlchemy's work around each statement
+ cost more than Postgres took to run it. `db/raw.py` runs a statement
+ SQLAlchemy compiled once on the session's own asyncpg connection, in its
+ transaction and under its tenant, with SQLAlchemy's type conversions and
+ timed into `db.query.duration` as every other statement is.
+- **Pods start in seconds again with many rooms.** Every process checks at
+ startup that each room has its clients, under a lock that lets one
+ process at a time through, and read each room on its own to do it. With
+ tens of thousands of rooms that took about 100 seconds, so processes
+ starting together were restarted while they waited. Rooms are now read a
+ thousand at a time.
+- **Idle agents behind a controller cost the database far less.** A
+ controller stream's pass, including the one after every idle interval,
+ read each of its agents' events in a session of its own; it now reads all
+ their heads in one query and reads only the agents with something new. And
+ each process's 30-second re-check of controllers held by another process
+ read every agent's rooms with a query of its own, which grew with
+ processes times agents; it is now one query per controller.
+- **A delivery to an agent behind a controller takes two database
+ transactions instead of four.** The controller's stream read the heads of
+ all its agents to find the work, again after delivering, and again before
+ waiting. It now reads them once a pass, and an event that arrives after
+ that read wakes the stream for the next pass.
+- **A controller keeps its socket when a delivery pass cannot get a
+ database connection.** When a pod was too busy to hand its streams a
+ connection within the pool timeout, the controller's stream ended and its
+ socket closed, and every agent on it reconnected, which needs more of
+ what was short. The pass is now tried again after a short wait, doubling
+ to 5 seconds, and the socket closes only if it is still failing after a
+ minute.
+ - **Switch's own reader is woken only by commands.** The process that holds
+ a workspace's admin client read every row of every room to find the
+ `!command` rows it answers. A new row's announcement now names its type,
+ and that reader is woken for command rows only and reads only those, from
+ a partial index on them. Its warnings for tags nobody can answer are
+ already posted by the process that writes the message.
+ - **The heads-up for tagging an agent that isn't in the room comes from the
+ process that wrote the message, and costs nothing when every name tagged
+ is one of the room's agents.** The admin's consumer, which reads every
+ message in a workspace on one process, checked each message with an `@`:
+ two database sessions and a lookup for every name tagged. Under load that
+ one process fell behind the rest. The check now runs once a message has
+ committed, on whichever process wrote it, and only when the message tags
+ a name that is none of the room's agents (a role, a person, an agent not
+ in the room). The warnings read as before.
- **An agent's stream reads in one database session per pass, and pooled
connections are no longer pinged before use.** A pass that delivered a
message opened a session for each of its reads, six pool checkouts and six
diff --cc core/switch_core/db/stores/message_store.py
index 743f68c0c,f6e292d2a..fc6504a11
--- a/core/switch_core/db/stores/message_store.py
+++ b/core/switch_core/db/stores/message_store.py
@@@ -158,11 -146,24 +158,17 @@@ class MessageStore
The cost is per-room serialisation of a single INSERT. Rooms do not
contend with each other.
"""
- await session.execute(
- text("SELECT pg_advisory_xact_lock(hashtext(:room_id))"),
- {"room_id": room_id},
- )
- result = await session.execute(
- select(func.coalesce(func.max(Message.seq), 0)).where(
- Message.room_id == room_id
- )
- )
- return int(result.scalar_one()) + 1
+ await _LOCK_ROOM.all(session, {"room_id": room_id})
+ head = await _ROOM_HEAD.all(session, {"room_id": room_id})
+ return int(head[0].head) + 1
- async def head_seq(self, session: AsyncSession, room_id: str) -> int:
+ async def head_seq(
+ self,
+ session: AsyncSession,
+ room_id: str,
+ *,
+ types: Collection[str] | None = None,
+ ) -> int:
"""The room's current position, or 0 when nothing has been sent.
What a subscriber starts from when it wants everything after now
diff --cc core/switch_core/delivery/router.py
index e32e6dca2,ad0c74434..33d9aa198
--- a/core/switch_core/delivery/router.py
+++ b/core/switch_core/delivery/router.py
@@@ -71,10 -71,10 +71,11 @@@ from sqlalchemy.dialects.postgresql imp
from sqlalchemy.exc import DBAPIError
from switch_core.attachments import ATTACHMENT_GROUP_KEY, parse_attachment_group
+from switch_core.bridges.agent.protocol.event_buffer import OwnEventToStore
from switch_core.budgets import BudgetGuard
- from switch_core.clients.actor import AgentActor, ClientConfig
- from switch_core.clients.admin_messages import PLATFORM_MARKER
+ from switch_core.clients.actor import AgentActor, ClientConfig, SystemActor
+ from switch_core.clients.admin_messages import ADMIN_MARKER, PLATFORM_MARKER
+ from switch_core.clients.mention_warnings import MentionWarnings, unmatched_mentions
from switch_core.clients.room_meta import RoomMeta
from switch_core.db.models import (
Agent,