Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Merge remote-tracking branch 'origin/asyncpg-message-write' into scale-core

Abel Dantas6f2e5cc4edc6
diff --cc CHANGELOG.md
index bc67fcf48,430b32ccd..b7474239c
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -159,18 -142,43 +159,55 @@@ version of their own to them without al
compares it only when someone signs in.
#### Changed
+- **Workspaces belong to organizations. Upgrade by replacing every pod, and
+ downgrade the database before rolling back.** Every existing workspace gets
+ an organization of its own, and its owners become the organization's admins.
+ An organization admin administers every workspace in it without being added
+ to each. A release from before organizations doesn't know about organization
+ admins: if old and new pods serve together during a rolling update, or after
+ a rollback that keeps the new database, an owner removed from a workspace by
+ an old pod is still the organization's admin and can come back into it. So
+ for this release, replace every switch-core pod at once rather than rolling
+ them, and to roll back, first downgrade the database to the revision before
+ `238c68b7306e` (`alembic downgrade 238c68b7306e-1`), then start the older
+ release.
+ - **The statements every message write costs run on asyncpg directly.**
+ Taking the room's lock, reading its head, storing the agents' own events
+ and reading the room's agent members again under the lock go straight to
+ asyncpg on the transaction's connection (`db/raw.py`), as the delivery
+ reads do, rather than through SQLAlchemy's execution.
+ - **The two reads behind every delivery run on asyncpg directly.** A
+ stream reads an agent's events, and counts its unread messages, for every
+ event it delivers, and under load SQLAlchemy's work around each statement
+ cost more than Postgres took to run it. `db/raw.py` runs a statement
+ SQLAlchemy compiled once on the session's own asyncpg connection, in its
+ transaction and under its tenant, with SQLAlchemy's type conversions and
+ timed into `db.query.duration` as every other statement is.
+ - **Pods start in seconds again with many rooms.** Every process checks at
+ startup that each room has its clients, under a lock that lets one
+ process at a time through, and read each room on its own to do it. With
+ tens of thousands of rooms that took about 100 seconds, so processes
+ starting together were restarted while they waited. Rooms are now read a
+ thousand at a time.
+ - **Idle agents behind a controller cost the database far less.** A
+ controller stream's pass, including the one after every idle interval,
+ read each of its agents' events in a session of its own; it now reads all
+ their heads in one query and reads only the agents with something new. And
+ each process's 30-second re-check of controllers held by another process
+ read every agent's rooms with a query of its own, which grew with
+ processes times agents; it is now one query per controller.
+ - **A delivery to an agent behind a controller takes two database
+ transactions instead of four.** The controller's stream read the heads of
+ all its agents to find the work, again after delivering, and again before
+ waiting. It now reads them once a pass, and an event that arrives after
+ that read wakes the stream for the next pass.
+ - **A controller keeps its socket when a delivery pass cannot get a
+ database connection.** When a pod was too busy to hand its streams a
+ connection within the pool timeout, the controller's stream ended and its
+ socket closed, and every agent on it reconnected, which needs more of
+ what was short. The pass is now tried again after a short wait, doubling
+ to 5 seconds, and the socket closes only if it is still failing after a
+ minute.
- **An agent's stream reads in one database session per pass, and pooled
connections are no longer pinged before use.** A pass that delivered a
message opened a session for each of its reads, six pool checkouts and six
@@@ -236,35 -235,27 +273,54 @@@
group whose subgroup would move up next to a namesake is refused too, and
the dashboard now shows why. Names can no longer contain `/` or be blank.
Existing duplicates are left as they are and can be reached by id.
+ - **Delivering an event to an agent behind a controller no longer reads the
+ agent's head first.** The notification of a new agent event already carries
+ its number, and the controller's stream now keeps it as the agent's head, so
+ a delivery costs the agent's own read alone, where it read the heads of all
+ the controller's agents first. The heads are still read after each idle
+ interval, when an agent is attached, and after the listener reconnects.
+ - **Delivering an event to an agent costs three statements, where it cost
+ seven.** The rooms dropped after the agent's cursor are read with its
+ events, and the unread count reads the agent's baseline, the room's window
+ and the agent's user id in the statement that counts.
+ - **Writing a message no longer costs four statements for each agent it is
+ for.** The agents' own events are stored together, in two statements and
+ one savepoint, where each agent's event took a savepoint, a numbering, an
+ insert and a release of its own.
+ - **A controller's stream waits on three events, where it waited on one per
+ agent and two more.** Every attached agent's doorbell now rings the stream's
+ one bell.
+ - **Reading an agent's own events no longer builds its query each time.**
+ Each shape of the read is built once and run with bind parameters.
#### Fixed
+- **On Slack, the 👀 and the activity card no longer arrive after the agent's
+ reply.** Every running turn's card was redrawn every 5 seconds to tick its
+ clock. Slack closes a streamed card after 10 minutes, so on a longer turn
+ each tick was a full message edit, and Slack allows about 50 of those a
+ minute for the whole workspace. A few long turns used that up, Slack made the
+ bridge wait, and new turns' cards and 👀 waited behind the clock redraws. The
+ clock now moves once a minute, and a running card shows whole minutes
+ ("Working… 3m"). Clock redraws stop while Slack is making the bridge wait,
+ and new turns go ahead of them. The 👀 goes on as soon as a turn starts
+ rather than once its card has posted. A card is redrawn as soon as Slack
+ closes its stream, instead of showing still-running steps as failed until
+ the next change. A turn whose first draw into its stream fails keeps that
+ card rather than posting a second one under it.
+- **An agent is told why a machine's CLI cannot be used.** When
+ `create_agent` or `update_agent_detail` was refused because a provider was
+ not logged in, it said the owner needed to log in, even when the machine had
+ reported a login its agents could not use, which logging in does not fix
+ (a Codex login in the OS keyring, say). The refusal now gives what the
+ machine said instead, and `list_machines` shows it as each provider's
+ `problem`.
+- **A Teams app on a federated credential can deploy from the Helm chart.**
+ switch-core's migrate init container and the pre-upgrade migration Job were
+ given the Teams app's settings but not its projected token, so
+ `TEAMS_APP_FEDERATED_TOKEN_FILE` named a missing file and both refused to
+ load their configuration: with `switchCore.teamsApp.credential: federated`,
+ no release could migrate or start. The two that migrate are no longer given
+ the Teams app at all.
- **The agent HTTP API answers a refused or malformed request with a 4xx, not
a 500.** Inviting an agent to a room the caller may not change answered 500
instead of 403, and so did looking up a room the agent is not in. A room
diff --cc core/switch_core/db/tenant_session.py
index 683d7c756,cdc7c65e9..401647dac
--- a/core/switch_core/db/tenant_session.py
+++ b/core/switch_core/db/tenant_session.py
@@@ -207,21 -197,12 +207,21 @@@ def _refuse_a_flush_that_drifted
session: Session, flush_context: object, instances: object
) -> None:
"""The same check, on the path `do_orm_execute` does not see."""
- _refuse_drift(session)
+ refuse_drift(session)
- def _refuse_drift(session: Session) -> None:
+ def refuse_drift(session: Session) -> None:
if _STAMPED not in session.info:
return
+ stamped_organization = session.info.get(_STAMPED_ORGANIZATION)
+ bound_organization = current_organization_id()
+ if bound_organization != stamped_organization:
+ raise TenantBindingDriftError(
+ f"this session's transaction was opened with organization "
+ f"{stamped_organization!r} and the organization bound now is "
+ f"{bound_organization!r}. Open the session inside the "
+ "organization_scope, or commit this transaction before rebinding."
+ )
stamped = session.info[_STAMPED]
bound = current_tenant_id()
if bound == stamped: