Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Merge remote-tracking branch 'origin/hold-room-access' into scale-core

Abel Dantasbf58d5734a0c
diff --cc CHANGELOG.md
index fe9200308,5e43e349f..5344c41f7
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -171,85 -171,15 +171,94 @@@ version of their own to them without al
them, and to roll back, first downgrade the database to the revision before
`238c68b7306e` (`alembic downgrade 238c68b7306e-1`), then start the older
release.
+- **The statements every message write costs run on asyncpg directly.**
+ Taking the room's lock, reading its head, storing the agents' own events
+ and reading the room's agent members again under the lock go straight to
+ asyncpg on the transaction's connection (`db/raw.py`), as the delivery
+ reads do, rather than through SQLAlchemy's execution.
+- **The two reads behind every delivery run on asyncpg directly.** A
+ stream reads an agent's events, and counts its unread messages, for every
+ event it delivers, and under load SQLAlchemy's work around each statement
+ cost more than Postgres took to run it. `db/raw.py` runs a statement
+ SQLAlchemy compiled once on the session's own asyncpg connection, in its
+ transaction and under its tenant, with SQLAlchemy's type conversions and
+ timed into `db.query.duration` as every other statement is.
+- **Pods start in seconds again with many rooms.** Every process checks at
+ startup that each room has its clients, under a lock that lets one
+ process at a time through, and read each room on its own to do it. With
+ tens of thousands of rooms that took about 100 seconds, so processes
+ starting together were restarted while they waited. Rooms are now read a
+ thousand at a time.
+- **Idle agents behind a controller cost the database far less.** A
+ controller stream's pass, including the one after every idle interval,
+ read each of its agents' events in a session of its own; it now reads all
+ their heads in one query and reads only the agents with something new. And
+ each process's 30-second re-check of controllers held by another process
+ read every agent's rooms with a query of its own, which grew with
+ processes times agents; it is now one query per controller.
+- **A delivery to an agent behind a controller takes two database
+ transactions instead of four.** The controller's stream read the heads of
+ all its agents to find the work, again after delivering, and again before
+ waiting. It now reads them once a pass, and an event that arrives after
+ that read wakes the stream for the next pass.
+- **A controller keeps its socket when a delivery pass cannot get a
+ database connection.** When a pod was too busy to hand its streams a
+ connection within the pool timeout, the controller's stream ended and its
+ socket closed, and every agent on it reconnected, which needs more of
+ what was short. The pass is now tried again after a short wait, doubling
+ to 5 seconds, and the socket closes only if it is still failing after a
+ minute.
+- **Switch's own reader is woken only by commands.** The process that holds
+ a workspace's admin client read every row of every room to find the
+ `!command` rows it answers. A new row's announcement now names its type,
+ and that reader is woken for command rows only and reads only those, from
+ a partial index on them. Its warnings for tags nobody can answer are
+ already posted by the process that writes the message.
+- **The heads-up for tagging an agent that isn't in the room comes from the
+ process that wrote the message, and costs nothing when every name tagged
+ is one of the room's agents.** The admin's consumer, which reads every
+ message in a workspace on one process, checked each message with an `@`:
+ two database sessions and a lookup for every name tagged. Under load that
+ one process fell behind the rest. The check now runs once a message has
+ committed, on whichever process wrote it, and only when the message tags
+ a name that is none of the room's agents (a role, a person, an agent not
+ in the room). The warnings read as before.
+- **The server spends less CPU on its event loop and on finding the agent
+ sending a message.** It now runs on uvloop wherever uvloop is installed,
+ which is every platform but Windows. Finding the actor of the agent
+ sending a message no longer looks through every client's actor, and
+ compiled mention patterns are kept per name. In a CPU profile under
+ message load the event loop's own work was about 15% of a pod's CPU and
+ the actor lookup about 3%.
+- **Creating a room or an agent no longer looks through every running
+ client.** Every process keeps an actor for every client, and finding a
+ workspace's admin, or the actor of an agent registered in another process
+ a moment before, read all of them. Each is now found from an index of the
+ running clients by type and by agent. While 10,000 agents were being
+ created, these lookups took about 11% of a process's CPU in a profile.
+- **Each database transaction costs the server a little less.** Binding the
+ workspace to a transaction built and compiled the same statement every
+ time; it is now sent to the driver as it is. And timing who holds a pooled
+ connection worked out the borrower's name and a short stack on every
+ checkout; the name is now kept per function, and the stack is written out
+ only for a hold long enough to log.
+- **Usage analytics read the database far less per message in a large
+ workspace.** With usage reporting on, each message looked up its room's
+ platform and member counts, its sender and the agent's runtime, each kept
+ for five minutes in caches of at most 10,000 entries. Quiet rooms missed
+ every time, and with thousands of agents the caches evicted what they were
+ about to be asked again. A room's facts are now kept for fifteen minutes,
+ a sender and an agent's runtime, which do not change, for an hour, and up
+ to 50,000 of each.
+ - **A post's first checks are kept until a signal says they changed.**
+ Every post checks that its agent is in the room and that no budget
+ covering it has run out, and a controller relaying it checks the
+ membership again: two transactions per post. Each process now keeps those
+ answers as it keeps a controller token's rows. Triggers on `room_agents`,
+ `rooms` and `usage_budgets` signal every change, so an answer is kept while
+ the signal connection listens and for `AGENT_AUTH_CACHE_TTL_SECONDS`
+ otherwise. A refusal is never kept. `ROOM_ACCESS_CACHE_MAX_ENTRIES`
+ (20,000) bounds what is kept.
- **An agent's stream reads in one database session per pass, and pooled
connections are no longer pinged before use.** A pass that delivered a
message opened a session for each of its reads, six pool checkouts and six
@@@ -315,76 -245,15 +324,83 @@@
group whose subgroup would move up next to a namesake is refused too, and
the dashboard now shows why. Names can no longer contain `/` or be blank.
Existing duplicates are left as they are and can be reached by id.
+- **Delivering an event to an agent behind a controller no longer reads the
+ agent's head first.** The notification of a new agent event already carries
+ its number, and the controller's stream now keeps it as the agent's head, so
+ a delivery costs the agent's own read alone, where it read the heads of all
+ the controller's agents first. The heads are still read after each idle
+ interval, when an agent is attached, and after the listener reconnects.
+- **Delivering an event to an agent costs three statements, where it cost
+ seven.** The rooms dropped after the agent's cursor are read with its
+ events, and the unread count reads the agent's baseline, the room's window
+ and the agent's user id in the statement that counts.
+- **Writing a message no longer costs four statements for each agent it is
+ for.** The agents' own events are stored together, in two statements and
+ one savepoint, where each agent's event took a savepoint, a numbering, an
+ insert and a release of its own.
+- **A controller's stream waits on three events, where it waited on one per
+ agent and two more.** Every attached agent's doorbell now rings the stream's
+ one bell.
+- **Reading an agent's own events no longer builds its query each time.**
+ Each shape of the read is built once and run with bind parameters.
+- **A message addressed to a connected agent no longer reads the agent's
+ membership after it commits.** That read stops an agent removed from the
+ room in the meantime from saying anything there, and a connected agent has
+ nothing to say about a message addressed to it, so the read is now made only
+ before a refusal or an offline notice is posted, or a machine woken.
+- **Message telemetry reads a workspace's rooms, clients and agents at once.**
+ The first lookup that misses reads all of them, where each room, sender and
+ agent was read on its own, so in a large workspace most messages no longer
+ cost a lookup.
+- **Registering, renaming or deleting an agent no longer makes every process
+ read every client and agent.** The change now names the clients it touched,
+ and each process reads only those, where it read every client of the
+ workspace and, with a messaging app connected, every agent as well. The
+ 30-second re-check reads every client only when a fingerprint of the
+ workspace's clients and agents has changed since its last full read, where
+ it read them all every time.
+- **The 30-second re-check of which controller runs each agent reads one
+ query, and nothing when no placement changed.** Every process read every
+ placed agent's definition, document included, then each controller with a
+ query of its own. It now reads the columns a binding is made of in one
+ query per workspace, and only when a fingerprint of the placements has
+ changed since its last full read.
+- **The 30-second re-check of who holds each controller's socket no longer
+ reads every controller each time.** It compares a fingerprint of their
+ connections and of the live processes first, and when it reads, reads only
+ the connection columns, where it read every row with its status and
+ platform, and the rooms of every agent on a controller another process held.
+- **Writing to a room for the first time through a process no longer reads the
+ room.** A process's clients now share the room ids they look up, where each
+ kept its own, and the first one a process misses reads every open room of
+ the workspace at once, so with many agents behind a load balancer most sends
+ no longer start with a read of the room.
+- **Checking a message against Switch Trust no longer reads its settings each
+ time.** Each process keeps the settings it read until they change: saving or
+ clearing them in the gateway tells every process, so a change still takes
+ effect on the next message. Settings are read again after 30 seconds at most,
+ for a change whose signal was lost.
+- **Reading an agent's budget standings no longer builds their query each
+ time.** The standings are read twice for every message an agent posts, and
+ building the query cost more than running it; it is now built once.
+- **Reading a room's context costs one transaction.** The room's newest
+ message number, the membership check and the history read share one
+ session, where each opened its own.
+ - **A controller's requests no longer read its row and its agent's row each
+ time.** They were kept for 5 seconds, and a controller's requests, spread
+ over several processes, rarely came back to one within that. They are now
+ kept until a signal says they changed, which triggers on `agents` and
+ `agent_controllers` send, and a miss reads all of the controller's agents
+ at once. `CONTROLLER_AUTH_CACHE_MAX_AGENTS` (20,000) bounds the agent rows
+ kept.
#### Fixed
+- **A client in more than 32,767 rooms starts reading them.** The reads a
+ client makes as it starts matched its rooms with one parameter each, and
+ Postgres takes at most 32,767 per statement. Switch's own client is in
+ every room of its workspace, archived ones included, so past that many it
+ failed to start, retried, and failed again while the process ran, taking
+ up rooms one at a time in between. The rooms now go as one array.
- **On Slack, the 👀 and the activity card no longer arrive after the agent's
reply.** Every running turn's card was redrawn every 5 seconds to tick its
clock. Slack closes a streamed card after 10 minutes, so on a longer turn
diff --cc core/switch_core/bridges/agent/protocol/agent_core.py
index cfb642161,b69535cbb..a1d0ae800
--- a/core/switch_core/bridges/agent/protocol/agent_core.py
+++ b/core/switch_core/bridges/agent/protocol/agent_core.py
@@@ -1453,18 -1458,14 +1460,23 @@@ class AgentCore
async def require_room_member(self, agent_id: str, room_id: str) -> RoomDescriptor:
"""Get room and verify agent is a member. Raises PermissionError if not."""
+ tenant_id = current_tenant_id()
+ if tenant_id is not None:
+ held = self.room_access.member(tenant_id, agent_id, room_id)
+ if held is not None:
+ return held
- async with self.session_factory() as session:
+ async with self._bound_session() as session:
return await self._room_member_in(session, agent_id, room_id)
+ def _bound_session(self) -> AbstractAsyncContextManager[AsyncSession]:
+ """A session for the bound tenant: the one a `shared_tenant_session`
+ block lends this task, if it is inside one, so a caller can put
+ several reads in one transaction."""
+ tenant_id = current_tenant_id()
+ if tenant_id is None:
+ return self.session_factory()
+ return tenant_session(self.session_factory, tenant_id)
+
async def require_room_poster(self, agent_id: str, room_id: str) -> RoomDescriptor:
"""`require_room_member`, and raise `BudgetExceeded` if the agent has
reached a budget covering it. One session for both, so a post still