Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
fix(db): make the tenant catalogue fail closed, not open (CHOO-2623)
Wojciech Wojtyniak655255996b6a
diff --git a/core/switch_core/db/rls_ddl.py b/core/switch_core/db/rls_ddl.pyindex 62bfd87ea..fb433b245 100644--- a/core/switch_core/db/rls_ddl.py+++ b/core/switch_core/db/rls_ddl.py@@ -65,15 +65,31 @@ The policy text is one shape, applied identically to every scoped table:- **`tenants` compares on `id`**, because a tenant *is* the boundary ratherthan belonging to one. Every other scoped table compares on `tenant_id`.-**The table list is derived, not copied.** A table is scoped if it carries a+**Scoped is the default; global is the list.** The rule is stated the way+round that fails closed: every table in the metadata is scoped *unless* it is+named in `GLOBAL_TABLES`, and a table that is neither named there nor+actually carrying a tenant raises `UnscopedTableError` — at import, because+`db/models.py` calls `attach_row_level_security` at the bottom, which is to+say the process does not start.++This was originally written the other way round, deriving "scoped" from *has+a `tenant_id` foreign key to `tenants.id`*, and that version failed open: a+new table holding customer data and no tenant column simply was not in the+set, so nothing enabled a policy on it, nothing complained, and the+catalogue tests — which take their table list from here — passed. That is+precisely the regression these checks exist to catch, so the invariant is+inverted. Adding a table is now a choice between inheriting `TenantScoped`+and arguing in a diff for an entry in `GLOBAL_TABLES`; forgetting to choose+is not one of the options.++The membership test for "actually carries a tenant" is unchanged: a`tenant_id` column whose foreign key targets `tenants.id` — exactly`TenantScoped`'s column, and exactly what a plain `Table()` junction tablegets by adding the equivalent column inline, since the mixin only works on a-declarative class. Deriving it here means a table added later either follows-the pattern and is picked up automatically, or does not and is exempt from-these policies while still being visible as a gap to the catalogue test in-`tests/switch_core/db/test_row_level_security.py` — never silently skipped by-a list someone forgot to extend.+declarative class. The foreign key, not just the column name: a table that+merely happens to have a column called `tenant_id` without the constraint is+not tied to a tenant, and a policy comparing it would reject every row the+table holds.**Where the SQL lives.** Same house pattern as `notify_ddl.py`: this module'sDDL is attached to `Base.metadata` (`db/models.py`) so `create_all` builds it@@ -86,11 +102,35 @@ this module later does.from __future__ import annotations-from sqlalchemy import DDL, MetaData, event+from sqlalchemy import DDL, MetaData, Table, eventREQUIRE_TENANT_FUNCTION_NAME = "require_tenant_id"POLICY_NAME = "tenant_isolation"+# The whole exemption from tenancy, written out. Everything else in+# `Base.metadata` is scoped, so this list is the only place a table can be+# excused from carrying a tenant, and extending it is an edit a reviewer sees+# — `tests/switch_core/db/test_tenant_schema_catalogue.py` pins the contents,+# so widening it fails the suite until someone changes the test too.+#+# A person is not a tenant member (`users`), nor is the record of how they+# prove who they are (`oidc_identities`); `feature_flags` is a deployment+# switch, and a flag that needs to vary per customer is a new scoped table+# rather than a nullable column here. Alembic's `alembic_version` needs no+# entry: Alembic owns that table and never registers it on this metadata.+GLOBAL_TABLES = frozenset({"users", "oidc_identities", "feature_flags"})+++class UnscopedTableError(RuntimeError):+ """A table is neither tenant-scoped nor named in `GLOBAL_TABLES`.++ Raised while attaching the policies, which happens at the bottom of+ `db/models.py` — so a table added without a tenant and without a+ deliberate exemption stops the process at import rather than quietly+ shipping a table with customer data in it and no isolation policy.+ """++# SQLSTATE 42501, insufficient_privilege — the closest standard code for "you# may not see or touch this row", and the one an operator's tooling is likely# to already recognise as an authorization failure rather than a generic@@ -135,29 +175,71 @@ def drop_policy_ddl(table_name: str) -> str:return f'DROP POLICY IF EXISTS {POLICY_NAME} ON "{table_name}"'+def _tenant_column(table: Table) -> str | None:+ """The column this table's policy would compare, or None if it has none.++ `tenants` is the one table compared on `id`, because a tenant *is* the+ boundary rather than belonging to one. Every other table qualifies by+ carrying a `tenant_id` column whose foreign key targets `tenants.id`.+ """+ if table.name == "tenants":+ return "id"+ tenant_id_col = table.columns.get("tenant_id")+ if tenant_id_col is None:+ return None+ if any(+ fk.column.table.name == "tenants" and fk.column.name == "id"+ for fk in tenant_id_col.foreign_keys+ ):+ return "tenant_id"+ return None+++def unscoped_tables(metadata: MetaData) -> list[str]:+ """Every table that breaks the rule: not global, and carrying no tenant.++ Empty in a healthy schema. `scoped_tables` refuses to answer at all while+ this is non-empty; it is exposed separately so a test can state the+ invariant in its own words, and so a caller that wants the diagnosis+ rather than the exception can ask for it.+ """+ return sorted(+ table.name+ for table in metadata.tables.values()+ if table.name not in GLOBAL_TABLES and _tenant_column(table) is None+ )++def scoped_tables(metadata: MetaData) -> dict[str, str]:- """Table name -> the column its policy compares, derived from the schema.-- `tenants` is the one table scoped by `id`: every other table is scoped by- carrying a `tenant_id` column whose foreign key targets `tenants.id`. That- second test — the foreign key, not just the column name — is deliberate:- a table that merely happens to have a column called `tenant_id` without- the constraint is not actually tied to a tenant and must not silently pick- up a policy that would then reject every row it holds.+ """Table name -> the column its policy compares: everything not global.++ Raises `UnscopedTableError` naming any table that is neither in+ `GLOBAL_TABLES` nor actually carrying a tenant. The failure is the point:+ deriving the set from "has a tenant column" instead would quietly answer+ with a shorter list, and a table holding customer data with no isolation+ on it would be indistinguishable from one that was never added."""tables: dict[str, str] = {}+ violations: list[str] = []for table in metadata.tables.values():- if table.name == "tenants":- tables[table.name] = "id"+ if table.name in GLOBAL_TABLES:continue- tenant_id_col = table.columns.get("tenant_id")- if tenant_id_col is None:- continue- if any(- fk.column.table.name == "tenants" and fk.column.name == "id"- for fk in tenant_id_col.foreign_keys- ):- tables[table.name] = "tenant_id"+ column = _tenant_column(table)+ if column is None:+ violations.append(table.name)+ else:+ tables[table.name] = column+ if violations:+ raise UnscopedTableError(+ f"table(s) carry no tenant and are not global: {sorted(violations)}. "+ "Every table in the model metadata is tenant-scoped unless it is "+ "named in db/rls_ddl.py's GLOBAL_TABLES. Inherit TenantScoped (or, "+ "for a plain Table(), add the equivalent tenant_id column with a "+ "foreign key to tenants.id), or — if the table genuinely holds no "+ "customer data — add it to GLOBAL_TABLES, which is a decision "+ "tests/switch_core/db/test_tenant_schema_catalogue.py makes you "+ "state twice."+ )return tablesdiff --git a/core/tests/switch_core/db/stores/test_reference_type_store.py b/core/tests/switch_core/db/stores/test_reference_type_store.pyindex f26bc7f9d..748d3be08 100644--- a/core/tests/switch_core/db/stores/test_reference_type_store.py+++ b/core/tests/switch_core/db/stores/test_reference_type_store.py@@ -332,6 +332,49 @@ class TestReferenceTypesAreScopedToTheBoundTenant:theirs = await store.get(session, "notion")assert theirs is not None and theirs.display_name == "Theirs"+ async def test_a_duplicate_slug_clashes_cleanly_when_another_tenant_has_it_too(+ self, session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """A clash in the second tenant is a `ValueError`, not a 500.++ `create` reports "already exists" by looking the clashing row up+ *inside* its `IntegrityError` handler, and that lookup is a+ `scalar_one_or_none()`. The slug is only unique per tenant, so the+ moment two tenants hold the same one an unfiltered lookup matches two+ rows and raises `MultipleResultsFound` from inside the handler —+ which the gateway has no `except` for, so a 400 the caller can act on+ (`gateway/references.py` maps `ValueError` to one) becomes a 500. The+ filter on the row's own tenant is what keeps it single-valued, and+ this is the arrangement that tells the two apart: the same slug in+ tenant zero *and* in the tenant doing the create.+ """+ other = f"tenant-{uuid.uuid4().hex[:8]}"+ store = ReferenceTypeStore()+ async with session_factory() as session:+ session.add(Tenant(id=other, slug=other, name=other))+ ada = await _make_user(session, "ada")+ await session.flush()+ await store.create(session, _type("notion", ada.id))+ with tenant_scope(other):+ await store.create(session, _type("notion", ada.id))+ await session.commit()++ with tenant_scope(other):+ with pytest.raises(+ ValueError, match="Reference type 'notion' already exists"+ ):+ await store.create(session, _type("notion", ada.id))++ # The savepoint means the caller's transaction survives it,+ # and neither tenant's row was disturbed.+ await store.create(session, _type("linear", ada.id))+ await session.commit()+ assert {rt.type for rt in await store.list_all(session)} == {+ "notion",+ "linear",+ }+ assert {rt.type for rt in await store.list_all(session)} == {"notion"}+async def test_counting_references_ignores_another_tenants(self, session_factory: async_sessionmaker[AsyncSession]) -> None:diff --git a/core/tests/switch_core/db/test_row_level_security.py b/core/tests/switch_core/db/test_row_level_security.pyindex d581a8ccd..7e22f6410 100644--- a/core/tests/switch_core/db/test_row_level_security.py+++ b/core/tests/switch_core/db/test_row_level_security.py@@ -43,9 +43,11 @@ import switch_corefrom switch_core.db.base import Basefrom switch_core.db.models import RoomGroup, Tenantfrom switch_core.db.rls_ddl import (+ GLOBAL_TABLES,POLICY_NAME,REQUIRE_TENANT_FUNCTION_NAME,scoped_tables,+ unscoped_tables,)from switch_core.db.session_scope import tenant_sessionfrom switch_core.db.stores.room_group_store import RoomGroupStore@@ -347,19 +349,29 @@ class TestCatalogueCoverage:hand-kept list. The table list itself comes from `rls_ddl.scoped_tables`,the same derivation `create_all` used to attach the policies in the firstplace, not a copy of it.++ That derivation is *everything but `GLOBAL_TABLES`*, and the direction+ matters. Derived the other way round — "the tables that carry a+ `tenant_id`" — a new table holding customer data with no tenant column+ would not be in the list, so nothing below would look for a policy on it+ and nothing would fail: the regression these tests exist to catch,+ passing. Inverted, that table has no policy to find and this class says+ so; and `scoped_tables` itself refuses to answer at all, so in practice+ the failure arrives even earlier, at import."""- async def test_scoped_table_list_is_not_trivially_small(+ async def test_every_table_is_scoped_unless_it_is_named_global(self, session_factory: async_sessionmaker) -> None:+ """The list the rest of this class rests on, checked for the failure+ mode a derived list has: silently being short."""+ assert unscoped_tables(Base.metadata) == []scoped = scoped_tables(Base.metadata)+ assert set(scoped) == set(Base.metadata.tables) - set(GLOBAL_TABLES)assert len(scoped) >= 30, scoped- assert "tenants" in scopedassert scoped["tenants"] == "id"- assert "messages" in scoped- assert "agents" in scoped- for global_table in ("users", "oidc_identities", "feature_flags"):- assert global_table not in scoped+ assert scoped["messages"] == "tenant_id"+ assert scoped["tenant_members"] == "tenant_id"async def test_every_scoped_table_has_rls_enabled_and_the_right_policy(self, session_factory: async_sessionmaker@@ -418,9 +430,14 @@ class TestCatalogueCoverage:async def test_global_tables_carry_no_policy(self, session_factory: async_sessionmaker) -> None:+ """The other side of the exemption: a table named in `GLOBAL_TABLES`+ really is left unpoliced, so the list is not a place to hide a table+ that then quietly gets a policy anyway. Driven by the list itself, so+ an addition to it is checked here rather than skipped."""scoped = scoped_tables(Base.metadata)async with session_factory() as session:policies = await _tenant_isolation_policies(session)- for global_table in ("users", "oidc_identities", "feature_flags"):+ for global_table in sorted(GLOBAL_TABLES):+ assert global_table in Base.metadata.tablesassert global_table not in scopedassert global_table not in policiesdiff --git a/core/tests/switch_core/db/test_tenant_schema_catalogue.py b/core/tests/switch_core/db/test_tenant_schema_catalogue.pyindex a9f10ed56..c2b32c10d 100644--- a/core/tests/switch_core/db/test_tenant_schema_catalogue.py+++ b/core/tests/switch_core/db/test_tenant_schema_catalogue.py@@ -8,6 +8,18 @@ fails here even if nobody remembers to update these tests by hand. SeeEvery table holding customer data carries a non-null `tenant_id`, andevery foreign key between two such tables carries `tenant_id` as well.++**The invariant is stated the way round that fails closed**, which it was+not when this file was written. It used to derive "scoped" as *carries a+`tenant_id` foreign key*, and then check things about the tables in that+set — so a new table full of customer data and no tenant column was not in+the set, was checked by nothing, and passed. That is the exact regression+these tests exist to prevent, so the derivation is inverted: the small set+of tables with no tenant is hardcoded in `rls_ddl.GLOBAL_TABLES`, and every+other table in the metadata must be scoped. Adding a table with customer+data and no tenant column now fails at import — `db/models.py` attaches the+policies at the bottom and `scoped_tables` refuses — and the two tests below+pin both halves of the escape hatch so it cannot be widened quietly."""from __future__ import annotations@@ -19,31 +31,17 @@ from sqlalchemy.ext.asyncio import AsyncConnection, AsyncSession, async_sessionm# junction tables) on `Base.metadata`.import switch_core.db.models # noqa: F401from switch_core.db.base import Base+from switch_core.db.rls_ddl import GLOBAL_TABLES, scoped_tables, unscoped_tablesdef _scoped_tables() -> set[str]:- """Every table carrying a `tenant_id` column whose foreign key targets- `tenants.id` — the schema's own definition of "scoped".-- Deliberately not the list from the design doc: a table that gains a- `tenant_id` later by copying the mixin (or the equivalent inline column,- for the plain `Table()` junction tables) is picked up automatically, and- one that forgets the foreign key to `tenants` is correctly left out and- so exempt from these checks — the accompanying test that every scoped- table's FKs carry `tenant_id` only means anything once a table is on this- list at all.+ """Every table the schema says is tenant-scoped: all but the globals.++ The same derivation `create_all` used to attach the policies+ (`rls_ddl.scoped_tables`), not a second copy of it — a private copy here+ is what let this file and the policies disagree in the first place."""- scoped = set()- for table in Base.metadata.tables.values():- tenant_id_col = table.columns.get("tenant_id")- if tenant_id_col is None:- continue- if any(- fk.column.table.name == "tenants" and fk.column.name == "id"- for fk in tenant_id_col.foreign_keys- ):- scoped.add(table.name)- return scoped+ return set(scoped_tables(Base.metadata))def _get_foreign_keys(sync_conn, table_name: str) -> list[dict]:@@ -71,28 +69,53 @@ async def _unique_column_sets(return await conn.run_sync(_get_unique_column_sets, table_name)-class TestScopedTableCatalogue:- async def test_scoped_tables_exist_and_are_not_trivial(- self, session_factory: async_sessionmaker[AsyncSession]- ) -> None:- """Sanity check on the derivation itself: if this list ever collapses- to empty or to one table, the two tests below would pass vacuously."""+class TestEveryTableIsScopedUnlessItIsNamedGlobal:+ """The inverted invariant itself, in two halves.++ Half one: nothing in the metadata is unaccounted for. Half two: the list+ of things excused is exactly the list someone signed off on. Without the+ second, the first is trivially satisfiable by adding the new table to+ `GLOBAL_TABLES` — which is a legitimate thing to do and must simply be+ visible in a diff rather than available as a quick way to get a red suite+ green.+ """++ async def test_no_table_escapes_the_rule(self) -> None:+ assert unscoped_tables(Base.metadata) == [], (+ "table(s) hold customer data with no tenant and no exemption; "+ "inherit TenantScoped, or name them in rls_ddl.GLOBAL_TABLES and "+ "update the test below"+ )++ async def test_the_global_table_list_is_exactly_these(self) -> None:+ """Hardcoded on purpose, and hardcoded twice on purpose.++ `users` is a person rather than a tenant member, `oidc_identities`+ records how that person proves who they are, and `feature_flags` is a+ deployment switch — a flag that has to vary per customer is a new+ scoped table, not a nullable column there. `alembic_version` is+ global too but is not in this metadata: Alembic owns it.+ """+ assert set(GLOBAL_TABLES) == {"users", "oidc_identities", "feature_flags"}++ async def test_scoped_is_everything_else(self) -> None:scoped = _scoped_tables()+ assert scoped == set(Base.metadata.tables) - set(GLOBAL_TABLES)+ # Not vacuous, and the shape is what the design doc describes.assert len(scoped) >= 30, scoped- assert "messages" in scoped- assert "agents" in scoped- # Global tables must not appear.- assert "users" not in scoped- assert "oidc_identities" not in scoped- assert "feature_flags" not in scoped+ assert {"messages", "agents", "tenants", "tenant_members"} <= scoped++class TestScopedTableCatalogue:async def test_every_cross_scoped_foreign_key_carries_tenant_id(self, session_factory: async_sessionmaker[AsyncSession]) -> None:"""A foreign key from one scoped table to another must be compositeon `tenant_id`, so a row can never reference a parent in anothertenant. Foreign keys to a *global* table (e.g. `rooms.owner_id ->- users.id`) are single-column by design and are not checked here."""+ users.id`) are single-column by design and are not checked here, and+ neither is a table's own key to `tenants` — that key is what makes it+ scoped in the first place and is single-column by construction."""scoped = _scoped_tables()violations: list[tuple[str, str, str]] = []async with session_factory() as session:@@ -100,7 +123,7 @@ class TestScopedTableCatalogue:for table_name in scoped:for fk in await _foreign_keys(conn, table_name):ref_table = fk["referred_table"]- if ref_table not in scoped:+ if ref_table not in scoped or ref_table == "tenants":continuelocal_cols = set(fk["constrained_columns"])ref_cols = set(fk["referred_columns"])diff --git a/docs/old/multi-tenancy-phase1-db.md b/docs/old/multi-tenancy-phase1-db.mdindex 3411bbc9f..03cb62727 100644--- a/docs/old/multi-tenancy-phase1-db.md+++ b/docs/old/multi-tenancy-phase1-db.md@@ -127,6 +127,26 @@ object. `oidc_identities` records how that person proves who they are, equallytenant-independent. `feature_flags` is a deployment switch; a flag that needsto vary per tenant is a new table, not a nullable column.+**This is the list that is written down, and the scoped list is the one that+is derived** — that way round, deliberately. The three in the model metadata+are hardcoded in `db/rls_ddl.py`'s `GLOBAL_TABLES` (`alembic_version` needs no+entry; Alembic owns it and never registers it there), and **every other table+in the metadata is scoped by definition**: it must carry the tenant, must have+a policy, and must carry `tenant_id` on its foreign keys to other scoped+tables. A table that is neither in that list nor actually carrying a tenant+raises `UnscopedTableError` from `scoped_tables()`, which runs at the bottom+of `db/models.py` — so the process does not start.++Stated the other way round it fails open, and it was written that way round+first: deriving "scoped" as *has a `tenant_id` foreign key* means a new table+full of customer data and no tenant column is simply not in the set, so+nothing attaches a policy to it, nothing checks its foreign keys, and every+catalogue test passes. That is the regression these tests exist to prevent,+passing. Widening `GLOBAL_TABLES` is the intended escape hatch and stays+available — it just has to be argued for in a diff, because+`tests/switch_core/db/test_tenant_schema_catalogue.py` pins the contents of+the list as well as the rule.+Leaving `users` and `oidc_identities` unpoliced has a consequence, and it isrecorded as an open item rather than buried: see "What Phase 1 does not close".@@ -629,10 +649,13 @@ metadata rather than by running migrations, so a policy that existed only in amigration would be invisible to every test — the isolation test would passagainst a database with no isolation in it.-A `TenantScoped` declarative mixin carries the column, its default and its-registration in the policy list, so adding a table means inheriting from it and-writing the foreign keys. Without the mixin a new table needs seven separate-things remembered; with it, two, and both are checked by tests.+A `TenantScoped` declarative mixin carries the column and its default, so+adding a table means inheriting from it and writing the foreign keys. Without+the mixin a new table needs seven separate things remembered; with it, two,+and both are checked by tests. There is no third thing to remember —+registering the table in the policy list — because there is no list to+register in: a table is in it unless `GLOBAL_TABLES` says otherwise, and a+table that forgets the mixin does not get skipped, it stops the import.## The migration@@ -723,13 +746,23 @@ real environment would pass for the wrong reason.Alongside it, the cheap tests that catch the regressions this design existsto prevent — in that file unless another is named:+- **Every table in the metadata is scoped unless it is named in+ `rls_ddl.GLOBAL_TABLES`**, and that list is pinned by its own test so+ extending it is a two-place, reviewable change rather than the quick way to+ quieten a red suite (`tests/switch_core/db/test_tenant_schema_catalogue.py`,+ and the same list drives every check below). This is the inverted form+ described under "Which tables are scoped"; the original derivation —+ "scoped" means *carries a `tenant_id` foreign key* — was checked against a+ table added with customer data and no tenant column, and the whole+ catalogue passed.- Every scoped table has row-level security enabled and a policy whose`using` and `with check` both isolate on that table's tenant column,asserted by querying `pg_policies` and `pg_tables` rather than by reading- the code, with the table list itself derived from `rls_ddl.scoped_tables`- rather than copied. The predicates are compared by content, not merely for- being present: `using (true)` is a policy that is enabled, catalogued and- isolates nothing.+ the code, with the table list itself coming from `rls_ddl.scoped_tables` —+ the same derivation that attached the policies, not a second copy of it,+ which is what let the catalogue and the policies disagree before. The+ predicates are compared by content, not merely for being present: `using+ (true)` is a policy that is enabled, catalogued and isolates nothing.- The migration's frozen `SCOPED_TABLES` matches that same derivation. Afrozen copy is the right shape (a migration must not change meaningbecause a model did) and its failure mode is falling behind in silence —@@ -756,6 +789,17 @@ to prevent — in that file unless another is named:database and on a deployment that gained a tenant between two boots(`tests/switch_core/clients/test_ensure_system_client.py`). It had no testat all until it stopped booting.+- A reference type whose slug clashes *while another tenant holds the same+ slug* still reports "already exists" rather than a 500+ (`tests/switch_core/db/stores/test_reference_type_store.py`).+ `ReferenceTypeStore.create` names the clashing row by looking it up inside+ its own `IntegrityError` handler, with a `scalar_one_or_none()`; the slug is+ only unique per tenant, so an unfiltered lookup there matches two rows and+ raises `MultipleResultsFound` from inside the handler — turning the 400+ `gateway/references.py` maps `ValueError` to into an unhandled 500. The+ lookup names the row's own tenant, and the arrangement that tells the two+ apart needs the slug present in two tenants at once, which no+ single-tenant test could produce.The foreign-key-carries-`tenant_id` catalogue check this section originallyasked for shipped earlier, with the schema migration — see