Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Sweep decisions only in workspaces with one waiting, four at a time
agent-runner353bef5d2da2c33
diff --git a/core/switch_core/config.py b/core/switch_core/config.pyindex e74f68459..2f68dea48 100644--- a/core/switch_core/config.py+++ b/core/switch_core/config.py@@ -263,6 +263,9 @@ class SwitchConfig(BaseSettings):# A message still waiting for a decision after this long (a restart, a# long outage) is dropped and logged rather than answered late.decision_pending_max_age_seconds: int = Field(default=120, ge=1, le=3600)+ # How many tenants' waiting decisions each process's sweep settles at+ # once, so one tenant with a slow backend does not hold up the others.+ decision_sweep_concurrency: int = Field(default=4, ge=1, le=64)@field_validator("decision_endpoint_url", "decision_api_key", mode="before")@classmethoddiff --git a/core/switch_core/db/tenant_lookup.py b/core/switch_core/db/tenant_lookup.pyindex cdc5d07e1..bbcefda88 100644--- a/core/switch_core/db/tenant_lookup.py+++ b/core/switch_core/db/tenant_lookup.py@@ -1,4 +1,4 @@-"""The whole exemption from row-level security, written out as eighteen functions.+"""The whole exemption from row-level security, written out as nineteen functions.Row-level security is enforced by `require_tenant_id()` (`db/rls_ddl.py`),which raises when no tenant is bound. That is the property everything else@@ -203,6 +203,15 @@ tenant on the server in turn to find the few that belong to it. It discloseswhich workspaces an organization has, what `all_tenant_ids` alreadydiscloses, narrowed; the gateway asks it only for the platform operator.+**`tenants_with_pending_decisions` is the nineteenth**: the workspaces with+a message waiting for the decision service (`pending_decisions`). Every+process sweeps those messages every few seconds so that one whose process+stopped is still decided; binding every tenant in turn to find the few with+any cost each tenant several sessions per sweep whether or not auto-route was+on anywhere in it. It discloses which workspaces have a decision waiting at+that moment, which says that auto-route is on and in use there; the sweep is+its only caller.+Why not the obvious alternatives is argued in`docs/old/multi-tenancy-phase1-db.md`, "The bootstrap problem"; the shortversion is that returning rows instead of tenant ids would put a second copy@@ -296,7 +305,7 @@ class TenantLookup:return f"{self.name}({', '.join('text' for _ in self.arguments)})"-# The eighteen of them. Ordered as the three shapes above: enumeration, then+# The nineteen of them. Ordered as the three shapes above: enumeration, then# credential resolution, then deriving a tenant from an identifier in hand.TENANT_LOOKUPS: tuple[TenantLookup, ...] = (TenantLookup(@@ -516,6 +525,16 @@ TENANT_LOOKUPS: tuple[TenantLookup, ...] = ("who may bind any workspace and holds no membership to narrow by."),),+ TenantLookup(+ name="tenants_with_pending_decisions",+ arguments=(),+ query="SELECT DISTINCT tenant_id FROM pending_decisions ORDER BY tenant_id",+ purpose=(+ "The workspaces with a message waiting for the decision service, "+ "so the decision sweep binds only those rather than every tenant "+ "on the server."+ ),+ ),TenantLookup(name="tenant_of_messaging_install",arguments=("platform", "external_workspace_id"),@@ -589,12 +608,12 @@ def attach_tenant_lookups(metadata: MetaData) -> None:# ── Calling them ──────────────────────────────────────────────────────────────# One `text()` per lookup, written out rather than assembled from `lookup.name`-# at call time: the eighteen names are fixed and known here, so there is nothing+# at call time: the nineteen names are fixed and known here, so there is nothing# for a call site to build. Each bind is named after the lookup's own argument,# which is what lets `_call` zip them positionally against the dataclass and# fail loudly on a mismatch rather than binding the workspace to the platform.# The assertion below is what keeps this dict from quietly falling behind-# `TENANT_LOOKUPS`: a nineteenth lookup with no entry here fails at import, not+# `TENANT_LOOKUPS`: a twentieth lookup with no entry here fails at import, not# with a `KeyError` on whatever request reaches it first._LOOKUP_STATEMENTS: dict[str, TextClause] = {"all_tenant_ids": text("SELECT tenant_id FROM all_tenant_ids() AS tenant_id"),@@ -652,6 +671,9 @@ _LOOKUP_STATEMENTS: dict[str, TextClause] = {"tenants_of_organization": text("SELECT tenant_id FROM tenants_of_organization(:organization_id) AS tenant_id"),+ "tenants_with_pending_decisions": text(+ "SELECT tenant_id FROM tenants_with_pending_decisions() AS tenant_id"+ ),}assert _LOOKUP_STATEMENTS.keys() == TENANT_LOOKUPS_BY_NAME.keys(), (@@ -703,6 +725,14 @@ async def all_tenant_ids(return await _call(session_factory, TENANT_LOOKUPS_BY_NAME["all_tenant_ids"])+async def tenants_with_pending_decisions(+ session_factory: async_sessionmaker[AsyncSession],+) -> list[str]:+ return await _call(+ session_factory, TENANT_LOOKUPS_BY_NAME["tenants_with_pending_decisions"]+ )++async def all_organization_ids(session_factory: async_sessionmaker[AsyncSession],) -> list[str]:diff --git a/core/switch_core/delivery/router.py b/core/switch_core/delivery/router.pyindex 8b8705e0b..546959ae1 100644--- a/core/switch_core/delivery/router.py+++ b/core/switch_core/delivery/router.py@@ -180,6 +180,28 @@ DECISION_SWEEP_SECONDS = 5.0# The most rooms one pass decides at once, one message each._DECISION_BATCH = 20++async def sweep_pending_decisions(+ tenant_ids: list[str],+ decide: Callable[[str], Awaitable[int]],+ concurrency: int,+) -> None:+ """Decide the waiting messages of each of `tenant_ids`, at most+ `concurrency` tenants at a time, so a tenant with a slow or failing+ backend holds up one slot rather than every tenant after it. A tenant+ that fails is logged and leaves the others to finish."""+ gate = asyncio.Semaphore(concurrency)++ async def one(tenant_id: str) -> None:+ async with gate:+ try:+ await decide(tenant_id)+ except Exception:+ logger.exception("Decision sweep failed for tenant %s", tenant_id)++ await asyncio.gather(*(one(tenant_id) for tenant_id in tenant_ids))++# The one custom event an agent can be part of. The rest (reports,# observations) are the platform's own bookkeeping and are nobody's own event._COMMAND_EVENT = "com.switch.command"diff --git a/core/switch_core/main.py b/core/switch_core/main.pyindex 09a9bd0e3..37f0f4903 100644--- a/core/switch_core/main.py+++ b/core/switch_core/main.py@@ -177,7 +177,10 @@ from switch_core.db.stores.tenant_store import TenantStorefrom switch_core.db.stores.usage_store import UsageStorefrom switch_core.db.stores.user_store import UserStorefrom switch_core.db.stores.workspace_service_store import WorkspaceServiceStore-from switch_core.db.tenant_lookup import all_tenant_ids+from switch_core.db.tenant_lookup import (+ all_tenant_ids,+ tenants_with_pending_decisions,+)from switch_core.decision.setup import build_decision_clientfrom switch_core.delivery.conversation import ConversationRules, ConversationSettingsfrom switch_core.delivery.recipient import RecipientServices@@ -185,6 +188,7 @@ from switch_core.delivery.router import (DECISION_SWEEP_SECONDS,GROUP_SWEEP_SECONDS,MessageRouter,+ sweep_pending_decisions,)from switch_core.feature_flags import AGENT_MANAGEMENT, SWITCH_BUDDYfrom switch_core.gateway.app import create_gateway_app@@ -359,25 +363,24 @@ async def _attachment_group_sweep_loop(async def _decision_sweep_loop(- session_factory: async_sessionmaker[AsyncSession], router: MessageRouter+ session_factory: async_sessionmaker[AsyncSession],+ router: MessageRouter,+ concurrency: int,) -> None:- """Decide the messages waiting for the decision service in every tenant:- the process that wrote one starts at once, and this is what decides one- whose process stopped first, or that a restart left behind. Every process- runs it; each message is claimed by one."""+ """Decide the messages waiting for the decision service: the process that+ wrote one starts at once, and this is what decides one whose process+ stopped first, or that a restart left behind. Every process runs it; each+ message is claimed by one. Only tenants with a message waiting are+ visited, `concurrency` at a time."""with no_tenant():while True:await asyncio.sleep(DECISION_SWEEP_SECONDS)try:- tenants = await all_tenant_ids(session_factory)+ tenants = await tenants_with_pending_decisions(session_factory)except Exception:logger.exception("Decision sweep failed")continue- for tenant_id in tenants:- try:- await router.decide_pending(tenant_id)- except Exception:- logger.exception("Decision sweep failed for tenant %s", tenant_id)+ await sweep_pending_decisions(tenants, router.decide_pending, concurrency)async def _recheck_agent_rooms(@@ -1597,7 +1600,11 @@ async def run(config: SwitchConfig, process_id: str) -> None:_attachment_group_sweep_loop(session_factory, router))decision_task = (- asyncio.create_task(_decision_sweep_loop(session_factory, router))+ asyncio.create_task(+ _decision_sweep_loop(+ session_factory, router, config.decision_sweep_concurrency+ )+ )if config.decision_routing_enabledelse None)diff --git a/core/switch_core/migrations/versions/689d7d8ff913_decision_settings_and_pending_decisions.py b/core/switch_core/migrations/versions/689d7d8ff913_decision_settings_and_pending_decisions.pyindex ca183ae3e..cb6f3c84a 100644--- a/core/switch_core/migrations/versions/689d7d8ff913_decision_settings_and_pending_decisions.py+++ b/core/switch_core/migrations/versions/689d7d8ff913_decision_settings_and_pending_decisions.py@@ -9,9 +9,15 @@ Two tenant-scoped tables for decision routing (`delivery/decision.py`):written with the message and deleted once it is decided, so a restart doesnot lose it.-The row-level-security DDL is a verbatim copy of `switch_core/db/rls_ddl.py`-as it stood when this migration was written, copied rather than imported for-the reason `265ed188ad6f` gives.+It also adds the nineteenth lookup to the exemption from row-level security,+`tenants_with_pending_decisions`, so the decision sweep binds only the+workspaces with a message waiting. See `db/tenant_lookup.py` for what it+discloses.++The row-level-security DDL is a verbatim copy of `switch_core/db/rls_ddl.py`,+and the lookup's DDL of `switch_core/db/tenant_lookup.py`, as they stood when+this migration was written, copied rather than imported for the reason+`265ed188ad6f` gives.Expand only: a process from before this revision never reads either table,and decides untagged messages in memory, as it always did.@@ -40,6 +46,18 @@ PENDING = "pending_decisions"_PREDICATE = f'"tenant_id" = (SELECT {REQUIRE_TENANT_FUNCTION_NAME}())'+SECURE_SEARCH_PATH = "pg_catalog, public, pg_temp"++CREATE_TENANTS_WITH_PENDING_DECISIONS = f"""CREATE OR REPLACE FUNCTION tenants_with_pending_decisions()+ RETURNS SETOF text+ LANGUAGE sql STABLE SECURITY DEFINER+ SET search_path = {SECURE_SEARCH_PATH}+AS $$SELECT DISTINCT tenant_id FROM pending_decisions ORDER BY tenant_id$$"""++DROP_TENANTS_WITH_PENDING_DECISIONS = (+ "DROP FUNCTION IF EXISTS tenants_with_pending_decisions()"+)+def _create_policy(table: str) -> str:return (@@ -111,9 +129,11 @@ def upgrade() -> None:)op.execute(f'ALTER TABLE "{PENDING}" ENABLE ROW LEVEL SECURITY')op.execute(_create_policy(PENDING))+ op.execute(CREATE_TENANTS_WITH_PENDING_DECISIONS)def downgrade() -> None:+ op.execute(DROP_TENANTS_WITH_PENDING_DECISIONS)op.execute(f'DROP POLICY IF EXISTS {POLICY_NAME} ON "{PENDING}"')op.drop_index("ix_pending_decisions_created_at", table_name=PENDING)op.drop_table(PENDING)diff --git a/core/tests/switch_core/db/test_tenant_lookup.py b/core/tests/switch_core/db/test_tenant_lookup.pyindex e63e3b8a4..c8fecd5e9 100644--- a/core/tests/switch_core/db/test_tenant_lookup.py+++ b/core/tests/switch_core/db/test_tenant_lookup.py@@ -42,7 +42,7 @@ from types import ModuleTypeimport pytestfrom alembic.config import Configfrom alembic.script import ScriptDirectory-from sqlalchemy import text+from sqlalchemy import select, textfrom sqlalchemy.exc import DBAPIError, IntegrityErrorimport switch_core@@ -54,6 +54,7 @@ from switch_core.db.models import (Invitation,MessagingInstall,Organization,+ PendingDecision,Room,ServerConnector,Tenant,@@ -79,6 +80,7 @@ from switch_core.db.tenant_lookup import (tenants_of_organization,tenants_of_user,tenants_open_to_domain,+ tenants_with_pending_decisions,)from switch_core.tenant_context import tenant_scopefrom tests.conftest import RLSHarness@@ -115,6 +117,7 @@ _ADDED_SINCE = {"directory_organizations_of_user": "f2b8d6c1a4e9","directory_organizations": "a7c3e9f15b20","tenants_of_organization": "c3d9a1f7b246",+ "tenants_with_pending_decisions": "689d7d8ff913",}# A third kind of change, and the quietest: a lookup whose body a later@@ -414,6 +417,30 @@ class TestWhatTheyAnswer:rls_harness.restricted, organization_id) == [fixture.tenant_a]+ async def test_tenants_with_pending_decisions_sees_only_those_waiting(+ self, rls_harness: RLSHarness+ ) -> None:+ fixture = await _two_populated_tenants(rls_harness)+ async with rls_harness.owner() as session:+ room_b = await session.scalar(+ select(Room.id).where(Room.tenant_id == fixture.tenant_b)+ )+ assert room_b is not None+ session.add(+ PendingDecision(+ tenant_id=fixture.tenant_b,+ room_id=room_b,+ room_seq=1,+ message_key="m-1",+ )+ )+ await session.commit()++ found = await tenants_with_pending_decisions(rls_harness.restricted)++ assert fixture.tenant_b in found+ assert fixture.tenant_a not in found+async def test_all_organization_ids_sees_every_organization(self, rls_harness: RLSHarness) -> None:diff --git a/core/tests/switch_core/delivery/test_decision_routing.py b/core/tests/switch_core/delivery/test_decision_routing.pyindex 1b3e82feb..2beaa1abc 100644--- a/core/tests/switch_core/delivery/test_decision_routing.py+++ b/core/tests/switch_core/delivery/test_decision_routing.py@@ -41,10 +41,12 @@ from switch_core.db.models import (from switch_core.db.stores.decision_settings_store import DecisionSettingsStorefrom switch_core.db.stores.pending_decision_store import PendingDecisionStorefrom switch_core.db.stores.room_store import DECISION_NOTICE_AT_KEY, RoomStore+from switch_core.db.tenant_lookup import tenants_with_pending_decisionsfrom switch_core.decision.client import DynamicDecisionClientfrom switch_core.decision.settings import DecisionAvailability, ServerDefaultfrom switch_core.delivery.decision import DECISION_NOTICE, THREAD_CONTEXTfrom switch_core.delivery.recipient import AgentRecipient+from switch_core.delivery.router import sweep_pending_decisionsfrom switch_core.outbound import OutboundPolicyfrom switch_core.tenant_context import current_tenant_id, no_tenantfrom switch_core.transport import InboundMessage@@ -741,6 +743,28 @@ class TestDecisionsSurviveARestart:).all()assert len(events) == 1+ async def test_the_sweep_finds_only_the_workspace_with_a_message_waiting(+ self, world: World, room: Journey, service: DecisionService+ ) -> None:+ _stop_waking(room)+ service.pick = "helper"+ assert TENANT_ZERO_ID not in await tenants_with_pending_decisions(+ world.session_factory+ )++ told = await room.say("bob", "can someone check the deploy?", at=10)++ assert await tenants_with_pending_decisions(world.session_factory) == [+ TENANT_ZERO_ID+ ]+ await sweep_pending_decisions(+ await tenants_with_pending_decisions(world.session_factory),+ room.process.router.decide_pending,+ 4,+ )+ assert await room.reached(told) == {"helper"}+ assert await tenants_with_pending_decisions(world.session_factory) == []+async def test_a_message_waiting_too_long_is_dropped_and_logged(self,world: World,diff --git a/core/tests/switch_core/delivery/test_decision_sweep.py b/core/tests/switch_core/delivery/test_decision_sweep.pynew file mode 100644index 000000000..e5c28f6b1--- /dev/null+++ b/core/tests/switch_core/delivery/test_decision_sweep.py@@ -0,0 +1,84 @@+"""The decision sweep across tenants: at most `concurrency` tenants at once,+and one tenant's failure leaves the others to finish."""++from __future__ import annotations++import asyncio+import logging++import pytest++from switch_core.delivery.router import sweep_pending_decisions+++class _Decider:+ """Records how many tenants are being decided at once."""++ def __init__(self, *, failing: set[str] | None = None) -> None:+ self.running = 0+ self.most = 0+ self.done: list[str] = []+ self.failing = failing or set()++ async def __call__(self, tenant_id: str) -> int:+ self.running += 1+ self.most = max(self.most, self.running)+ try:+ await asyncio.sleep(0.01)+ if tenant_id in self.failing:+ raise RuntimeError(f"backend down for {tenant_id}")+ self.done.append(tenant_id)+ return 1+ finally:+ self.running -= 1++[email protected]("concurrency", [1, 4])+async def test_at_most_concurrency_tenants_at_once(concurrency: int) -> None:+ decide = _Decider()+ tenants = [f"t{i}" for i in range(10)]++ await sweep_pending_decisions(tenants, decide, concurrency)++ assert sorted(decide.done) == sorted(tenants)+ assert decide.most == concurrency+++async def test_a_slow_tenant_does_not_hold_up_the_others() -> None:+ finished: list[str] = []+ release = asyncio.Event()++ async def decide(tenant_id: str) -> int:+ if tenant_id == "slow":+ await release.wait()+ finished.append(tenant_id)+ if len(finished) == 3:+ release.set()+ return 1++ await asyncio.wait_for(+ sweep_pending_decisions(["slow", "a", "b", "c"], decide, 2), timeout=5+ )++ assert finished[:3] == ["a", "b", "c"]+ assert finished[-1] == "slow"+++async def test_a_failing_tenant_is_logged_and_the_rest_finish(+ caplog: pytest.LogCaptureFixture,+) -> None:+ decide = _Decider(failing={"t1"})++ with caplog.at_level(logging.ERROR):+ await sweep_pending_decisions(["t0", "t1", "t2"], decide, 4)++ assert sorted(decide.done) == ["t0", "t2"]+ assert "Decision sweep failed for tenant t1" in caplog.text+++async def test_no_tenants_waiting_decides_nothing() -> None:+ decide = _Decider()++ await sweep_pending_decisions([], decide, 4)++ assert decide.done == []diff --git a/core/tests/switch_core/test_config_decision_routing.py b/core/tests/switch_core/test_config_decision_routing.pyindex 220040b53..9832c8da4 100644--- a/core/tests/switch_core/test_config_decision_routing.py+++ b/core/tests/switch_core/test_config_decision_routing.py@@ -58,6 +58,7 @@ def test_off_by_default_with_nothing_configured() -> None:assert config.decision_timeout_seconds == 5.0assert config.decision_bedrock_enabled is Falseassert config.decision_pending_max_age_seconds == 120+ assert config.decision_sweep_concurrency == 4client = build_decision_client(config, _NO_DB, None)assert client.availability.enabled is Falseassert client.server.configured is False@@ -106,6 +107,12 @@ def test_the_wait_is_bounded(seconds: int) -> None:_config(decision_pending_max_age_seconds=seconds)[email protected]("tenants", [0, 65])+def test_the_sweep_concurrency_is_bounded(tenants: int) -> None:+ with pytest.raises(ValidationError, match="decision_sweep_concurrency"):+ _config(decision_sweep_concurrency=tenants)++@pytest.mark.parametrize("timeout", [0, -1])def test_a_non_positive_timeout_is_refused(timeout: float) -> None:with pytest.raises(ValidationError, match="DECISION_TIMEOUT_SECONDS"):diff --git a/docs/old/decision-routing.md b/docs/old/decision-routing.mdindex 074af707f..4c1160271 100644--- a/docs/old/decision-routing.md+++ b/docs/old/decision-routing.md@@ -27,6 +27,7 @@ has not chosen one:| `DECISION_TIMEOUT_SECONDS` | How long one decision may take (default 5), for every workspace's backend. || `DECISION_BEDROCK_ENABLED` | Whether a workspace may choose Haiku on Amazon Bedrock (default off). Bedrock runs in the server's decision service on the server's own AWS credentials, which is why the operator has to allow it. || `DECISION_PENDING_MAX_AGE_SECONDS` | How long a message may wait for a decision, across a restart or an outage, before it is dropped rather than answered late (default 120). |+| `DECISION_SWEEP_CONCURRENCY` | How many workspaces' waiting decisions each process's sweep settles at once (default 4, 1 to 64). |With no `DECISION_ENDPOINT_URL`, there is no server default: auto-route isunavailable in every workspace until its admins choose a backend, and Haiku is@@ -162,7 +163,10 @@ A message waiting for a decision is recorded with the message(`pending_decisions`) and decided by whichever process gets to it first: theone that wrote it at once, any other on its sweep every few seconds, or thesame one after a restart. Each is delivered at most once. One still waiting-after `DECISION_PENDING_MAX_AGE_SECONDS` is dropped and logged.+after `DECISION_PENDING_MAX_AGE_SECONDS` is dropped and logged. The sweep+visits only the workspaces with a message waiting (one query across+workspaces), up to `DECISION_SWEEP_CONCURRENCY` of them at a time, so a+workspace whose backend is slow or failing holds up only its own messages.## What is reported