Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

feat(db): let an install end, and a workspace be installed again (CHOO-2626)

Petr Bauchd054494ef414
diff --git a/core/switch_core/db/models.py b/core/switch_core/db/models.py
index ddc33ca2d..2099d6ed6 100644
--- a/core/switch_core/db/models.py
+++ b/core/switch_core/db/models.py
@@ -1145,19 +1145,33 @@ class MessagingInstall(TenantScoped, Base):
token, and what it may do.
**`(platform, external_workspace_id)` is unique across the whole
- deployment, not per tenant**, and that is the single most important line
- here. Inbound events arrive over one public endpoint carrying a workspace
- id and no tenant, so a workspace claimed by two tenants is a message with
- two possible destinations and no way to choose — which is the failure this
- whole phase exists to make unrepresentable. The database decides it rather
- than a read-then-insert in application code, because the check and the
- write cannot be made atomic from outside.
-
- That constraint is also the one place a tenant learns something about
- another: claiming a workspace somebody else already claimed fails, and the
- failure says so. It is the right answer — the alternative is a silent
- second claim — and what it discloses is that *some* tenant holds a
- workspace the caller was already able to name.
+ deployment among rows that are still `active`**, and that is the single
+ most important line here. Inbound events arrive over one public endpoint
+ carrying a workspace id and no tenant, so a workspace claimed by two
+ tenants is a message with two possible destinations and no way to choose —
+ which is the failure this whole phase exists to make unrepresentable. The
+ database decides it rather than a read-then-insert in application code,
+ because the check and the write cannot be made atomic from outside.
+
+ It is a *partial* index rather than a plain constraint because an install
+ has to be able to end. A customer who removes the app in Slack, or an
+ operator who disconnects it here, leaves a row behind — and a row that
+ still occupied the workspace would mean nobody could ever install that
+ workspace again, including the customer who just removed it. Ending an
+ install therefore frees the workspace, and keeps the record of the one
+ that ended.
+
+ That index is also the one place a tenant learns something about another:
+ claiming a workspace somebody else already holds fails, and the failure
+ says so. It is the right answer — the alternative is a silent second claim
+ — and what it discloses is that *some* tenant holds a workspace the caller
+ was already able to name.
+
+ `status` is `active`, `disconnected` (an operator here ended it) or
+ `revoked` (the platform told us it was over). The two endings are recorded
+ apart because they call for different things: one is somebody's decision
+ and the other is news, and an operator looking at a bridge that stopped
+ working needs to know which.
`bridge_id` is nullable because the install row is written before anything
is built on it, and because removing a bridge should not force the
@@ -1167,7 +1181,10 @@ class MessagingInstall(TenantScoped, Base):
`encrypted_bot_token` uses the same key as every other credential this
schema stores (`crypto.encrypt_token` over the configured secret), so it
is protected against a stolen dump and not against a compromised process.
- A per-tenant key is a stronger boundary and a later decision.
+ A per-tenant key is a stronger boundary and a later decision. It is
+ nullable so that an install which has ended can keep its record without
+ keeping its secret: the token is worthless by then, and a worthless
+ credential still reads like a credential to whoever finds the dump.
`scopes` is the platform's own spelling of what was granted, stored
verbatim rather than parsed into a list — a scope string that means
@@ -1177,10 +1194,12 @@ class MessagingInstall(TenantScoped, Base):
__tablename__ = "messaging_installs"
__table_args__ = (
- UniqueConstraint(
+ Index(
+ "uq_messaging_installs_workspace",
"platform",
"external_workspace_id",
- name="uq_messaging_installs_workspace",
+ unique=True,
+ postgresql_where=text("status = 'active'"),
),
UniqueConstraint("id", "tenant_id", name="uq_messaging_installs_id_tenant"),
ForeignKeyConstraint(
@@ -1193,7 +1212,7 @@ class MessagingInstall(TenantScoped, Base):
id: Mapped[str] = mapped_column(Text, primary_key=True, default=_uuid)
platform: Mapped[str] = mapped_column(Text, nullable=False)
external_workspace_id: Mapped[str] = mapped_column(Text, nullable=False)
- encrypted_bot_token: Mapped[str] = mapped_column(Text, nullable=False)
+ encrypted_bot_token: Mapped[str | None] = mapped_column(Text, nullable=True)
scopes: Mapped[str] = mapped_column(Text, nullable=False)
status: Mapped[str] = mapped_column(Text, nullable=False)
installed_by_user_id: Mapped[str] = mapped_column(
@@ -1203,6 +1222,9 @@ class MessagingInstall(TenantScoped, Base):
installed_at: Mapped[str] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
+ ended_at: Mapped[datetime | None] = mapped_column(
+ DateTime(timezone=True), nullable=True
+ )
class MessagingInstallState(TenantScoped, Base):
diff --git a/core/switch_core/db/stores/messaging_install_store.py b/core/switch_core/db/stores/messaging_install_store.py
index 269560850..b91a6d2a5 100644
--- a/core/switch_core/db/stores/messaging_install_store.py
+++ b/core/switch_core/db/stores/messaging_install_store.py
@@ -18,6 +18,18 @@ from switch_core.db.models import (
#: is worthless by the time anyone reads it.
STATE_TTL = timedelta(minutes=10)
+#: An install that is serving. The only status the workspace uniqueness index
+#: covers, so it is also the answer to "who holds this workspace".
+INSTALL_ACTIVE = "active"
+
+#: Ended here, by someone who decided to. The bridge is gone with it.
+INSTALL_DISCONNECTED = "disconnected"
+
+#: Ended there: the platform told us the app was removed or its token killed.
+#: Distinct from `disconnected` because an operator whose bridge stopped
+#: working needs to know whether it was news or a decision.
+INSTALL_REVOKED = "revoked"
+
class MessagingInstallClaimedError(RuntimeError):
"""Another tenant has already installed the app into this workspace.
@@ -103,17 +115,21 @@ class MessagingInstallStore:
"""Claim a workspace for the bound tenant.
The claim is the insert: `(platform, external_workspace_id)` is unique
- across the deployment, so the database decides who holds a workspace
- rather than a read followed by a write that cannot be made atomic with
- it. A second tenant's install therefore fails here, loudly, instead of
- producing an event with two possible destinations.
+ across the deployment among active rows, so the database decides who
+ holds a workspace rather than a read followed by a write that cannot be
+ made atomic with it. A second tenant's install therefore fails here,
+ loudly, instead of producing an event with two possible destinations.
+
+ Installs that have ended are not in the index, so re-installing a
+ workspace somebody released is an ordinary insert and needs no check of
+ its own.
"""
install = MessagingInstall(
platform=platform,
external_workspace_id=external_workspace_id,
encrypted_bot_token=encrypted_bot_token,
scopes=scopes,
- status="active",
+ status=INSTALL_ACTIVE,
installed_by_user_id=user_id,
)
session.add(install)
@@ -124,26 +140,33 @@ class MessagingInstallStore:
raise
raise MessagingInstallClaimedError(
f"the {platform} workspace {external_workspace_id} is already "
- "connected to Switch. Remove the existing install before "
- "connecting it again."
+ "connected to Switch. Disconnect the existing install — from "
+ "the organisation that holds it, which may not be yours — "
+ "before connecting it again."
) from exc
return install
async def get_for_workspace(
self, session: AsyncSession, *, platform: str, external_workspace_id: str
) -> MessagingInstall | None:
- """The bound tenant's install of one workspace, if it is theirs.
-
- Deliberately still scoped, even though the unique constraint means at
- most one row exists deployment-wide: the caller is an inbound webhook
- that resolved a tenant from the workspace a moment ago, and this
- re-reading it under RLS is what makes a mistake there a miss rather
- than a cross-tenant read.
+ """The bound tenant's live install of one workspace, if it is theirs.
+
+ Deliberately still scoped, even though the unique index means at most
+ one active row exists deployment-wide: the caller is an inbound webhook
+ that resolved a tenant from the workspace a moment ago, and re-reading
+ it under RLS is what makes a mistake there a miss rather than a
+ cross-tenant read.
+
+ The status predicate matches the index's and `tenant_of_messaging_
+ install`'s. Without it a workspace installed, removed and installed
+ again returns two rows and this raises — an outage for the live
+ install, caused by the ended one.
"""
result = await session.execute(
select(MessagingInstall).where(
MessagingInstall.platform == platform,
MessagingInstall.external_workspace_id == external_workspace_id,
+ MessagingInstall.status == INSTALL_ACTIVE,
)
)
return result.scalars().one_or_none()
diff --git a/core/switch_core/db/tenant_lookup.py b/core/switch_core/db/tenant_lookup.py
index adcd78dfd..371a457da 100644
--- a/core/switch_core/db/tenant_lookup.py
+++ b/core/switch_core/db/tenant_lookup.py
@@ -122,6 +122,13 @@ lookup on the id alone would answer twice the first time two platforms
happened to mint the same string, and refuse a customer's traffic for a
reason in someone else's account.
+It also filters on `status = 'active'`, which is not a refinement but the
+other half of the same uniqueness. Only active rows are unique by index, so a
+workspace that was installed, removed and installed again has two rows and
+this would otherwise answer twice — refusing a customer's live traffic on the
+strength of an install they themselves ended. The predicate here and the one
+on the index are the same predicate, and have to stay that way.
+
Why not the obvious alternatives is argued in
`docs/old/multi-tenancy-phase1-db.md`, "The bootstrap problem"; the short
version is that returning rows instead of tenant ids would put a second copy
@@ -306,14 +313,17 @@ TENANT_LOOKUPS: tuple[TenantLookup, ...] = (
query=(
"SELECT tenant_id FROM messaging_installs "
"WHERE platform = p_platform "
- "AND external_workspace_id = p_external_workspace_id"
+ "AND external_workspace_id = p_external_workspace_id "
+ "AND status = 'active'"
),
purpose=(
"Which tenant an inbound event from an installed workspace belongs "
"to. The public webhook is unauthenticated by nature and knows only "
"the platform it was posted to and the workspace the payload names, "
- "so this runs before anything else the request does. Unique by "
- "constraint on exactly this pair, so it answers at most once."
+ "so this runs before anything else the request does. The status "
+ "predicate is the index's own: only active rows are unique, and an "
+ "install that ended must stop answering for its workspace rather "
+ "than compete with the one that replaced it."
),
),
)
diff --git a/core/switch_core/migrations/versions/a7f2c3e9b481_messaging_installs_can_end.py b/core/switch_core/migrations/versions/a7f2c3e9b481_messaging_installs_can_end.py
new file mode 100644
index 000000000..98a1c950d
--- /dev/null
+++ b/core/switch_core/migrations/versions/a7f2c3e9b481_messaging_installs_can_end.py
@@ -0,0 +1,108 @@
+"""an install that can end, and a workspace that can be installed again
+
+`c8a4e21f6d30` gave `messaging_installs` a plain unique constraint on
+`(platform, external_workspace_id)`, on the grounds that a workspace claimed
+by two tenants is an event with two destinations. That grounds is still right;
+the constraint was too strong for it. It made a claim permanent — the row it
+protects is never deleted, so a customer who removed the Switch app from their
+own Slack could not put it back, and the error they got told them to "remove
+the existing install" through a path that did not exist.
+
+What actually has to be unique is the set of installs that are *serving*. So
+the constraint becomes a unique index over `status = 'active'` under the same
+name, and ending an install frees the workspace while keeping the record of
+what happened to it.
+
+`tenant_of_messaging_install` is redefined against the same predicate, and
+that is not a tidy-up. The function resolves a workspace to a tenant for
+traffic nobody has authenticated, and it was written to rely on the old
+constraint answering at most once. Left alone, the first workspace to be
+installed twice would make it answer twice and the caller refuses an ambiguous
+answer — so the customer's live install would stop receiving events because of
+one they had themselves ended.
+
+Two column changes come with it. `ended_at` records when, alongside the
+`status` that says which of the two ways. And `encrypted_bot_token` becomes
+nullable so an ended install can keep its record without keeping its secret:
+the token is dead by then, and a dead credential still reads like a credential
+to whoever finds the dump.
+
+The lookup DDL below is a verbatim copy of `switch_core/db/tenant_lookup.py`
+as it stood when this migration was written, copied rather than imported for
+the reason every revision in this chain copies: a migration records a change
+that already happened, and importing the live module would let a later edit
+silently change what this one means.
+`tests/switch_core/db/test_frozen_ddl_matches_create_all.py` is what keeps the
+copy from drifting.
+
+Revision ID: a7f2c3e9b481
+Revises: d3f6b0c95a17
+Create Date: 2026-09-15 00:00:00.000000
+
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "a7f2c3e9b481"
+down_revision: str | None = "d3f6b0c95a17"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+TABLE = "messaging_installs"
+INDEX_NAME = "uq_messaging_installs_workspace"
+
+SECURE_SEARCH_PATH = "pg_catalog, public, pg_temp"
+
+CREATE_TENANT_OF_MESSAGING_INSTALL = f"""CREATE OR REPLACE FUNCTION tenant_of_messaging_install(p_platform text, p_external_workspace_id text)
+ RETURNS SETOF text
+ LANGUAGE sql STABLE SECURITY DEFINER
+ SET search_path = {SECURE_SEARCH_PATH}
+AS $$SELECT tenant_id FROM messaging_installs WHERE platform = p_platform AND external_workspace_id = p_external_workspace_id AND status = 'active'$$"""
+
+RESTORE_TENANT_OF_MESSAGING_INSTALL = f"""CREATE OR REPLACE FUNCTION tenant_of_messaging_install(p_platform text, p_external_workspace_id text)
+ RETURNS SETOF text
+ LANGUAGE sql STABLE SECURITY DEFINER
+ SET search_path = {SECURE_SEARCH_PATH}
+AS $$SELECT tenant_id FROM messaging_installs WHERE platform = p_platform AND external_workspace_id = p_external_workspace_id$$"""
+
+
+def upgrade() -> None:
+ # Same name as the constraint it replaces, deliberately: the store reads
+ # the name out of the integrity error to tell a claimed workspace from any
+ # other write that failed, and Postgres reports a unique index by the same
+ # name a unique constraint would have carried.
+ op.drop_constraint(INDEX_NAME, TABLE, type_="unique")
+ op.create_index(
+ INDEX_NAME,
+ TABLE,
+ ["platform", "external_workspace_id"],
+ unique=True,
+ postgresql_where=sa.text("status = 'active'"),
+ )
+ op.add_column(
+ TABLE, sa.Column("ended_at", sa.DateTime(timezone=True), nullable=True)
+ )
+ op.alter_column(
+ TABLE, "encrypted_bot_token", existing_type=sa.Text(), nullable=True
+ )
+ op.execute(CREATE_TENANT_OF_MESSAGING_INSTALL)
+
+
+def downgrade() -> None:
+ op.execute(RESTORE_TENANT_OF_MESSAGING_INSTALL)
+ op.alter_column(
+ TABLE, "encrypted_bot_token", existing_type=sa.Text(), nullable=False
+ )
+ op.drop_column(TABLE, "ended_at")
+ op.drop_index(INDEX_NAME, table_name=TABLE)
+ # Both of the statements above and this one fail on data the old schema
+ # cannot hold: an ended install whose token was discarded, and a workspace
+ # holding more than one row. That is the honest outcome — going back means
+ # choosing which of those installs survives, and a migration is not the
+ # thing that gets to choose.
+ op.create_unique_constraint(
+ INDEX_NAME, TABLE, ["platform", "external_workspace_id"]
+ )
diff --git a/core/tests/switch_core/db/test_messaging_install_claim.py b/core/tests/switch_core/db/test_messaging_install_claim.py
index 1319744ae..c1af5cd94 100644
--- a/core/tests/switch_core/db/test_messaging_install_claim.py
+++ b/core/tests/switch_core/db/test_messaging_install_claim.py
@@ -19,16 +19,26 @@ That refusal discloses one bit — some tenant holds this workspace — to a cal
who could already name it. That is deliberate, and the alternative is worse: a
silent second claim, discovered when a customer's messages start arriving in
somebody else's rooms.
+
+The claim covers **active** installs only, and the second half of this file is
+about that. A claim that outlived the install would mean a workspace could be
+connected once ever — including by the customer who had just disconnected it —
+so ending an install releases the workspace. The lookup has to agree with the
+index about which rows count, and the tests below measure that agreement from
+both directions: a released workspace can be claimed again, and a workspace
+with history resolves to whoever holds it now rather than refusing because
+two rows name it.
"""
from __future__ import annotations
import uuid
+from datetime import UTC, datetime
import pytest
-from sqlalchemy import select
+from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
-from sqlalchemy.ext.asyncio import async_sessionmaker
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from switch_core.db.models import MessagingInstall, Tenant, User
from switch_core.db.session_scope import tenant_session
@@ -76,6 +86,20 @@ def _install(fixture: _Fixture, tenant_id: str) -> MessagingInstall:
)
+async def _end(session: AsyncSession, install_id: str, status: str) -> None:
+ """End an install the way the store will, without depending on it yet.
+
+ A direct write, because what is under test here is the index and the
+ lookup: the properties have to hold for any row in that state, not only
+ for rows a particular method happened to produce.
+ """
+ await session.execute(
+ update(MessagingInstall)
+ .where(MessagingInstall.id == install_id)
+ .values(status=status, ended_at=datetime.now(UTC), encrypted_bot_token=None)
+ )
+
+
async def test_a_second_tenant_cannot_claim_a_claimed_workspace(
rls_harness: RLSHarness,
) -> None:
@@ -149,3 +173,96 @@ async def test_the_same_workspace_on_another_platform_is_a_separate_claim(
)
== fixture.tenant_b
)
+
+
+async def test_an_install_that_ended_releases_the_workspace(
+ rls_harness: RLSHarness,
+) -> None:
+ """The claim lasts as long as the install and not a moment longer.
+
+ A permanent claim is not a stricter version of this guarantee, it is a
+ different and wrong one: a customer who tries Switch, disconnects, and
+ comes back finds their own workspace held by a row nobody can see and
+ nobody can release.
+ """
+ fixture = await _two_tenants_and_a_workspace(rls_harness.owner)
+
+ async with tenant_session(rls_harness.restricted, fixture.tenant_a) as session:
+ install = _install(fixture, fixture.tenant_a)
+ session.add(install)
+ await session.flush()
+ await _end(session, install.id, "disconnected")
+ await session.commit()
+
+ async with tenant_session(rls_harness.restricted, fixture.tenant_b) as session:
+ session.add(_install(fixture, fixture.tenant_b))
+ await session.commit()
+
+ assert (
+ await tenant_of_messaging_install(
+ rls_harness.restricted, "slack", fixture.workspace
+ )
+ == fixture.tenant_b
+ )
+
+
+async def test_the_lookup_does_not_answer_twice_for_a_workspace_with_history(
+ rls_harness: RLSHarness,
+) -> None:
+ """The failure the status predicate exists to prevent.
+
+ The lookup refuses an ambiguous answer rather than picking one, which is
+ right — but it makes a second row for the same workspace an outage for the
+ live install rather than a stale record. So the predicate on the function
+ has to be the index's predicate, and this measures it with enough history
+ to catch a lookup that merely takes the first row: two ended installs and
+ one live one, inserted in that order.
+ """
+ fixture = await _two_tenants_and_a_workspace(rls_harness.owner)
+
+ for _ in range(2):
+ async with tenant_session(rls_harness.restricted, fixture.tenant_a) as session:
+ install = _install(fixture, fixture.tenant_a)
+ session.add(install)
+ await session.flush()
+ await _end(session, install.id, "revoked")
+ await session.commit()
+
+ async with tenant_session(rls_harness.restricted, fixture.tenant_a) as session:
+ session.add(_install(fixture, fixture.tenant_a))
+ await session.commit()
+
+ assert (
+ await tenant_of_messaging_install(
+ rls_harness.restricted, "slack", fixture.workspace
+ )
+ == fixture.tenant_a
+ )
+
+
+async def test_a_workspace_nobody_holds_any_longer_resolves_to_nobody(
+ rls_harness: RLSHarness,
+) -> None:
+ """An ended install must not go on routing the workspace's traffic.
+
+ The app can still be sitting in the customer's Slack after a disconnect
+ here, posting events at us for as long as someone leaves it there. Those
+ events belong to no tenant now, and the lookup saying so is what turns
+ them into a refusal instead of a delivery into rooms the customer has
+ stopped paying for.
+ """
+ fixture = await _two_tenants_and_a_workspace(rls_harness.owner)
+
+ async with tenant_session(rls_harness.restricted, fixture.tenant_a) as session:
+ install = _install(fixture, fixture.tenant_a)
+ session.add(install)
+ await session.flush()
+ await _end(session, install.id, "disconnected")
+ await session.commit()
+
+ assert (
+ await tenant_of_messaging_install(
+ rls_harness.restricted, "slack", fixture.workspace
+ )
+ is None
+ )
diff --git a/core/tests/switch_core/db/test_tenant_lookup.py b/core/tests/switch_core/db/test_tenant_lookup.py
index 5e39f0bf9..a68e939ef 100644
--- a/core/tests/switch_core/db/test_tenant_lookup.py
+++ b/core/tests/switch_core/db/test_tenant_lookup.py
@@ -103,6 +103,15 @@ _ADDED_SINCE = {
"tenant_of_messaging_install": "c8a4e21f6d30",
}
+# A third kind of change, and the quietest: a lookup whose body a later
+# revision replaced. Creation and removal both show up as a function that is
+# there or is not; a redefinition leaves a function of the right name and
+# signature answering a different question, which nothing about the shape of
+# the schema reveals. So the revision that last wrote the body is named here
+# and compared against the module, while the creating revision above goes on
+# owning the drop.
+_REDEFINED_SINCE = {"tenant_of_messaging_install": "a7f2c3e9b481"}
+
def _revision_module(revision: str) -> ModuleType:
"""One revision module, loaded through Alembic.
@@ -722,17 +731,24 @@ class TestTheMigrationInstallsTheSameThing:
Driven by `_ADDED_SINCE` rather than naming one lookup, so the next
entry is covered by adding it there and nowhere else.
+
+ The body is compared against `_REDEFINED_SINCE` where there is an
+ entry, because the revision that creates a function is not always the
+ one that last says what it does. The drop is still the creating
+ revision's: a redefinition replaces a body and leaves the function it
+ replaced nothing to undo.
"""
for name, revision in _ADDED_SINCE.items():
lookup = TENANT_LOOKUPS_BY_NAME[name]
- module = _revision_module(revision)
+ defining = _REDEFINED_SINCE.get(name, revision)
+ module = _revision_module(defining)
assert getattr(module, f"CREATE_{name.upper()}") == create_lookup_ddl(
lookup
), (
- f"revision {revision} would install {name} with different DDL "
+ f"revision {defining} would install {name} with different DDL "
"from the one db/tenant_lookup.py builds."
)
- assert getattr(module, f"DROP_{name.upper()}") == (
+ assert getattr(_revision_module(revision), f"DROP_{name.upper()}") == (
f"DROP FUNCTION IF EXISTS {lookup.signature}"
)