Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
fix(db): close what a review of the restricted role found (CHOO-2623)
Wojciech Wojtyniak09cf3ea0e45b
diff --git a/core/switch_core/clients/client_lifecycle_service.py b/core/switch_core/clients/client_lifecycle_service.pyindex 2a19927c2..d9c3c58e4 100644--- a/core/switch_core/clients/client_lifecycle_service.py+++ b/core/switch_core/clients/client_lifecycle_service.py@@ -327,9 +327,13 @@ class ClientLifecycleService:A puppet in particular is reused for every room the person it standsfor speaks in, so the first room's tenant is exactly the value thatmust not survive into the second. Everything the client does binds- the tenant of the room it is acting on, or — for the two lookups that- answer *which* tenant, its own row and its room list — is unscoped on- purpose.+ the tenant of the room it is acting on. The one lookup that still runs+ with nothing bound is `PostgresTransport._tenant`, the exemption call+ (`tenant_of_client`) that answers *which* tenant this client's own row+ belongs to — asking that question on a session already bound to a+ tenant would beg it. Its room list is not exempt any more: once the+ transport knows its own tenant, `joined_rooms` reads under it, through+ an ordinary `tenant_session`, rather than unscoped."""with no_tenant():try:diff --git a/core/switch_core/db/runtime_role.py b/core/switch_core/db/runtime_role.pyindex 5efc674f8..59e7f3aab 100644--- a/core/switch_core/db/runtime_role.py+++ b/core/switch_core/db/runtime_role.py@@ -10,29 +10,39 @@ that installs 38 policies and then connects as the owner has none of them.Two halves live here.**Grants.** The runtime role owns nothing and can create nothing, so it needs-to be given access to every table the owner created. Re-issued on every boot,-immediately after the migration that may have added one, rather than once by-hand: `ALTER DEFAULT PRIVILEGES` covers a table created *later by the owner*-and is easy to get subtly wrong (it is per-granting-role, and silently does-nothing for objects an unnamed role creates), whereas a fresh `GRANT ... ON-ALL TABLES` after `alembic upgrade head` cannot fall behind the schema it was-just run against. It is idempotent and costs one round trip at boot.--**The self-check.** Four questions asked of the runtime connection before the-server listens, because this failure mode is silent and CI cannot ask it: CI-has no production connection, and a Switch that believes it is isolating-tenants and is not looks exactly like one that is. In order of how much they-prove:+to be given access to every table the owner created. The actual mechanism is+the four explicit `GRANT`s, re-issued fresh on every boot, immediately after+the migration that may have added a table: naming everything the schema+contains at that moment cannot fall behind it, the way a rule that has to fire+once per object, at the moment of its creation, can. The three `ALTER DEFAULT+PRIVILEGES` statements alongside them are belt to that grant's braces, not a+second mechanism for the same job: they cover whatever the owner creates+*between* one boot and the next, at the cost of being easy to get subtly+wrong as a strategy on their own — the setting is per-granting-role, and+silently grants nothing for an object some other role creates — a gap the+next boot's fresh `GRANT` closes regardless of whether the default privilege+fired. Both are idempotent and cost one round trip at boot.++**The self-check.** Questions asked of the runtime connection before the+server listens, in increasing order of how much they prove, because this+failure mode is silent and CI cannot ask it: CI has no production connection,+and a Switch that believes it is isolating tenants and is not looks exactly+like one that is.1. The role is not a superuser and carries no `BYPASSRLS`, nor is it a memberof a role that has either — membership is a `SET ROLE` away from both.-2. It owns none of the tables carrying a policy.+2. It owns none of the tables carrying a policy, nor inherits the privileges+ of a role that does.3. No scoped table has `force row level security`, which would take theownership exemption away from the lookup functions and leave the processunable to resolve a credential at all.-4. Every tenant lookup exists and this role may execute it, since a role that+4. Every scoped table still has row-level security switched on and still+ carries its policy — the schema half of the guarantee, which a check about+ the connection alone cannot prove; a table someone quietly disarmed would+ pass every check above it.+5. Every tenant lookup exists and this role may execute it, since a role thatcannot would authenticate nobody.-5. **A read of `tenants` with nothing bound raises.** This is the one that+6. **A read of `tenants` with nothing bound raises.** This is the one thatactually proves it, because the others are inferences from the catalogueand this is the behaviour. `tenants` is the table to ask it of: the checkis only meaningful against a populated table — Postgres does not evaluate a@@ -55,6 +65,12 @@ from sqlalchemy import textfrom sqlalchemy.exc import DBAPIErrorfrom sqlalchemy.ext.asyncio import AsyncConnection, AsyncEngine+# Imported for its side effect: registering every table on `Base.metadata`, so+# `scoped_tables` below answers with the schema this process expects rather+# than with an empty set — which would make the policy check pass vacuously.+import switch_core.db.models # noqa: F401+from switch_core.db.base import Base+from switch_core.db.rls_ddl import POLICY_NAME, scoped_tablesfrom switch_core.db.tenant_lookup import TENANT_LOOKUPSlogger = logging.getLogger(__name__)@@ -81,6 +97,18 @@ async def grant_runtime_role(owner: AsyncConnection, role: str) -> None:covers `require_tenant_id()` and the tenant lookups without naming them,so a ninth lookup needs no change here.+ The four plain `GRANT`s are what actually does the job, and doing it fresh+ on every boot is what keeps it from falling behind: it names every table,+ sequence and function the schema holds at that moment, so it cannot miss+ one the way a rule that has to fire once per object, at creation, can. The+ three `ALTER DEFAULT PRIVILEGES` statements after them are belt to that+ grant's braces, covering an object the owner creates between this boot and+ the next — at the cost of being unfit to carry the job alone, since the+ setting is per-granting-role and silently grants nothing for an object+ some other role creates. The next boot's fresh `GRANT` would close that+ gap either way, which is why the default privileges are additional rather+ than load-bearing on their own.+No `CREATE` on the schema, and no ownership: the role is meant to beunable to alter the tables it reads, which is what keeps it subject totheir policies.@@ -116,6 +144,7 @@ async def verify_restricted_role(engine: AsyncEngine) -> None:await _refuse_privileged_role(connection, role)await _refuse_table_owner(connection, role)await _refuse_forced_row_level_security(connection)+ await _require_every_policy(connection)await _require_the_lookups(connection, role)# The probe is the one check that provokes an error on purpose, and a# statement that raises aborts the transaction it ran in — every query@@ -161,6 +190,16 @@ async def _refuse_privileged_role(connection: AsyncConnection, role: str) -> Nonasync def _refuse_table_owner(connection: AsyncConnection, role: str) -> None:+ """Refuse a role that has the owner's privileges, not merely one that is it.++ `pg_has_role(current_user, c.relowner, 'USAGE')` rather than comparing+ names, because that is the test Postgres itself applies: `check_enable_rls`+ asks `has_privs_of_role`, so a role granted membership in the owner with+ `INHERIT` bypasses every policy while being a different role entirely.+ Comparing identities would call that connection clean. Measured on 14 and+ 16, not inferred: a plain restricted role is refused the unbound read, an+ inheriting member of the owner reads every row.+ """result = await connection.execute(text("""@@ -169,7 +208,7 @@ async def _refuse_table_owner(connection: AsyncConnection, role: str) -> None:JOIN pg_namespace n ON n.oid = c.relnamespaceWHERE n.nspname = 'public'AND c.relrowsecurity- AND pg_get_userbyid(c.relowner) = current_user+ AND pg_has_role(current_user, c.relowner, 'USAGE')ORDER BY c.relname""")@@ -177,12 +216,14 @@ async def _refuse_table_owner(connection: AsyncConnection, role: str) -> None:owned = [row[0] for row in result]if owned:raise RuntimeRoleError(- f"The database role {role!r} owns {len(owned)} of the tables "- f"carrying a tenant-isolation policy (for example {owned[:3]}). "- "Postgres exempts a table's owner from its own policies unless "- "FORCE ROW LEVEL SECURITY is set, which this design deliberately "- "does not set, so those policies do nothing on this connection. "- "Run the service as a role that owns no table."+ f"The database role {role!r} owns, or has the privileges of the "+ f"owner of, {len(owned)} of the tables carrying a tenant-isolation "+ f"policy (for example {owned[:3]}). Postgres exempts a table's "+ "owner — and anything that inherits its privileges — from its own "+ "policies unless FORCE ROW LEVEL SECURITY is set, which this "+ "design deliberately does not set, so those policies do nothing on "+ "this connection. Run the service as a role that owns no table and "+ "is a member of no role that does.")@@ -244,6 +285,52 @@ async def _refuse_a_connection_no_policy_stops()+async def _require_every_policy(connection: AsyncConnection) -> None:+ """Every scoped table still has row-level security on, and still has its policy.++ The checks above are all about the *connection*, and would each pass on a+ schema somebody had quietly disarmed. `ALTER TABLE messages DISABLE ROW+ LEVEL SECURITY` or `DROP POLICY tenant_isolation ON messages` leaves the+ role restricted, owning nothing, forcing nothing — and leaves `messages`+ readable by every tenant. The behavioural probe would not catch it either,+ since it asks only `tenants`.++ The expected set comes from `rls_ddl.scoped_tables`, the same derivation+ that attached the policies in the first place, so this cannot drift from+ what the schema is supposed to have. `test_tenant_schema_catalogue.py`+ asks the same question of a database built by `create_all`; this asks it+ of the one about to serve traffic, which is the only place the answer can+ have changed.+ """+ expected = set(scoped_tables(Base.metadata))+ rows = await connection.execute(+ text(+ """+ SELECT c.relname, c.relrowsecurity, p.polname IS NOT NULL AS policied+ FROM pg_class c+ JOIN pg_namespace n ON n.oid = c.relnamespace+ LEFT JOIN pg_policy p ON p.polrelid = c.oid AND p.polname = :policy+ WHERE n.nspname = 'public' AND c.relkind = 'r'+ AND c.relname = ANY(:names)+ """+ ),+ {"policy": POLICY_NAME, "names": sorted(expected)},+ )+ state = {name: (bool(enabled), bool(policied)) for name, enabled, policied in rows}+ disarmed = sorted(+ name for name in expected if state.get(name, (False, False)) != (True, True)+ )+ if disarmed:+ raise RuntimeRoleError(+ f"{len(disarmed)} scoped table(s) have no tenant-isolation policy, "+ f"or have row-level security switched off (for example "+ f"{disarmed[:3]}). The connection is restricted, but those tables "+ "are readable and writable across every tenant regardless. Either "+ "the schema is behind the migration that installs the policies, or "+ "something has disabled them since."+ )++async def _require_the_lookups(connection: AsyncConnection, role: str) -> None:"""The exempt lookups must be callable, or nothing can resolve a tenant.@@ -287,15 +374,29 @@ async def _require_the_lookups(connection: AsyncConnection, role: str) -> None:"EXECUTE, or let boot do it by configuring DB_OWNER_USER / ""DB_OWNER_PASSWORD.")- found = (- await connection.execute(text("SELECT count(*) FROM all_tenant_ids()"))- ).scalar_one()+ # Calling one for real, not just asking the catalogue about it. A lookup+ # that exists and is executable can still be unable to *read* — installed+ # SECURITY INVOKER, or owned by a role that has since lost its grants —+ # and the failure that produces is a bare driver error the escape hatch+ # does not catch, so it is turned into a RuntimeRoleError here rather than+ # left to kill a boot that asked to continue.+ try:+ found = (+ await connection.execute(text("SELECT count(*) FROM all_tenant_ids()"))+ ).scalar_one()+ except DBAPIError as exc:+ raise RuntimeRoleError(+ f"all_tenant_ids() is installed and executable by {role!r} but "+ f"could not read: {exc}. It is a SECURITY DEFINER function, so it "+ "reads as its owner — most likely it was not installed as one, or "+ "its owner no longer has access to `tenants`. Nothing in this "+ "process can resolve a tenant while that is true."+ ) from excif not found:raise RuntimeRoleError(- "all_tenant_ids() answered with no tenants at all. Either the "- "schema predates the migration that seeds tenant zero, or the "- "function is not the SECURITY DEFINER one this process expects — "- "either way nothing here can be scoped to a tenant."+ "all_tenant_ids() answered with no tenants at all, so the schema "+ "predates the migration that seeds tenant zero and nothing here "+ "can be scoped to a tenant.")diff --git a/core/switch_core/db/session_scope.py b/core/switch_core/db/session_scope.pyindex 5e698311f..eb93d289f 100644--- a/core/switch_core/db/session_scope.py+++ b/core/switch_core/db/session_scope.py@@ -22,11 +22,13 @@ documented — but only because every environment connected to Postgres as thetables' owner, which Postgres exempts from their policies. Under therestricted runtime role an unscoped session is not a hatch at all: unscoped isprecisely the state `require_tenant_id()` raises on, so every one of its-seventeen call sites either died at boot or silently read nothing. The whole-model was inverted, and `db/tenant_lookup.py` is what replaced it: eight-`SECURITY DEFINER` functions that answer *which tenant* and never return a-row, so a cross-tenant question is asked in one place with a fixed shape, and-the work it fans out into is scoped like everything else.+seventeen call sites — the ones that went through this helper by name, which+is not the same count as the exemption's full inventory; see+`db/tenant_lookup.py` for the other two — either died at boot or silently+read nothing. The whole model was inverted, and `db/tenant_lookup.py` is what+replaced it: eight `SECURITY DEFINER` functions that answer *which tenant*+and never return a row, so a cross-tenant question is asked in one place with+a fixed shape, and the work it fans out into is scoped like everything else.What is left is the one helper. Two things still open a session with nothingbound, and neither is cross-tenant:diff --git a/core/switch_core/db/tenant_lookup.py b/core/switch_core/db/tenant_lookup.pyindex 21ed7fa0b..090994a23 100644--- a/core/switch_core/db/tenant_lookup.py+++ b/core/switch_core/db/tenant_lookup.py@@ -13,7 +13,10 @@ that. Under the owner connection every environment used, an unscoped sessiondid read across tenants, so the model looked complete. Under a restrictedrole it is not a hatch at all — unscoped is precisely the state`require_tenant_id()` raises on — and a real deployment under a restricted-role died at the first of nineteen such sites before it finished booting.+role died at the first of nineteen such sites before it finished booting:+seventeen of them the `unscoped_session` helper's own call sites+(`db/session_scope.py`), and two more raw-session credential reads that+leaned on the same ownership exemption without ever calling it.**What is exempt is this module, and nothing else.** Each function below is`SECURITY DEFINER`, owned by the schema owner, and so runs outside the@@ -106,10 +109,21 @@ class TenantLookup:`argument` is the parameter name *without* the `p_` prefix the SQL carries.The prefix is not decoration: a `LANGUAGE sql` function whose parameter isspelled like a column of a table in its own query resolves the name to the- column, silently, so `WHERE id = client_id` against a table that has a- `client_id` column compares the column with itself and matches every row.- Prefixing every parameter makes that unrepresentable rather than a thing to- check per function.+ column, silently, rather than to the parameter. Verified on Postgres 16,+ because it is easy to get backwards: against a table that has a+ `client_id` column, `WHERE id = client_id` becomes a comparison between+ two columns of the same row, `id` and the table's own `client_id`, which+ is false for every row whose `client_id` does not happen to equal its+ `id` — a confident, silent *zero* rows, not every row. The sharper+ failure needs the parameter named after the very column being compared:+ `tenant_of_api_key(key_hash text) ... WHERE key_hash = key_hash` shadows+ to a column compared with itself, true unconditionally, and *that* matches+ every row regardless of the argument. Either shape is a caller left+ unable to resolve a tenant it should have found, or handed every tenant+ the row could belong to, in place of the one it actually named —+ so prefixing every parameter makes the shadowing unrepresentable rather+ than a thing to check per function, whichever direction it would have+ failed."""name: strdiff --git a/core/switch_core/db/tenant_session.py b/core/switch_core/db/tenant_session.pyindex 4a48e8726..3c4213c75 100644--- a/core/switch_core/db/tenant_session.py+++ b/core/switch_core/db/tenant_session.py@@ -34,10 +34,16 @@ one" — two paths never become a `Session` at all, both deliberately:When no tenant is bound (see `switch_core.tenant_context`), the hook doesnothing rather than substituting one — a system session (auth resolution,-Alembic, startup seeding) has no tenant, and the database does not yet reject-that (that lands with `require_tenant_id()` and the row-level-security-policies, in a later change). Until then, a query issued with no tenant set-simply runs unscoped, same as before this hook existed.+Alembic, startup seeding) has no tenant, and there is none to stamp. What+happens next to a query issued on it is no longer this module's to decide:+`require_tenant_id()` and the row-level-security policy on every scoped table+(`db/rls_ddl.py`) now raise the moment such a session touches one, so a+session with nothing bound may only read what carries no policy at all —+`users`, `oidc_identities`, and the tenant lookups' own exempt functions in+`db/tenant_lookup.py`. This hook's job is unchanged either way: it stamps+whatever is bound, including nothing. Whether nothing bound then reads+quietly or is refused loudly is the policies' call, not this hook's, and it+used to be the former."""from __future__ import annotations@@ -91,12 +97,31 @@ def register_tenant_session_hook() -> None:event.listens_for(Session, "after_begin")(_set_tenant_on_begin)event.listens_for(Session, "after_transaction_end")(_forget_the_stamp)event.listens_for(Session, "do_orm_execute")(_refuse_a_binding_that_came_too_late)+ # `do_orm_execute` does not cover a flush: the INSERTs and UPDATEs the unit+ # of work emits go to the connection directly rather than through+ # `Session.execute`, so a late binding followed only by `session.add(...)`+ # and `commit()` would slip past the check above. That is the shape a+ # *write* under a drifted binding actually takes, which makes it the case+ # that matters most: the column default reads the late tenant while the+ # transaction still carries the early one, so the row claims one tenant+ # and the policy compares another.+ event.listens_for(Session, "before_flush")(_refuse_a_flush_that_drifted)_registered = Truedef _set_tenant_on_begin(session: Session, transaction: SessionTransaction, connection: Connection) -> None:+ if transaction.nested:+ # A savepoint, not a transaction. `after_begin` fires for one, but the+ # setting it would issue belongs to the enclosing transaction: an+ # `is_local` `set_config` inside a savepoint survives `RELEASE` and+ # reverts on `ROLLBACK TO`, so re-stamping here would leave the record+ # holding the savepoint's tenant while the connection had gone back to+ # the outer one. Measured on 16, not assumed. The enclosing+ # transaction already carries the right value; a savepoint has nothing+ # to add and no business overwriting it.+ returntenant_id = current_tenant_id()# Recorded even when it is None, because "this transaction was stamped# with nothing" is exactly what the drift check needs to be able to tell@@ -128,12 +153,23 @@ def _refuse_a_binding_that_came_too_late(state: ORMExecuteState) -> None:"""Raise if what is bound now is not what this transaction was stamped with.Fires before every `Session.execute`, which is every statement this- codebase issues through a session. For the first statement of a- transaction there is no stamp yet — `after_begin` has not run, since the- transaction begins as part of executing it — so there is nothing to- compare and nothing to refuse. From the second onwards the two must agree.+ codebase issues through a session other than a flush. For the first+ statement of a transaction there is no stamp yet — `after_begin` has not+ run, since the transaction begins as part of executing it — so there is+ nothing to compare and nothing to refuse. From the second onwards the two+ must agree."""- session = state.session+ _refuse_drift(state.session)+++def _refuse_a_flush_that_drifted(+ session: Session, flush_context: object, instances: object+) -> None:+ """The same check, on the path `do_orm_execute` does not see."""+ _refuse_drift(session)+++def _refuse_drift(session: Session) -> None:if _STAMPED not in session.info:returnstamped = session.info[_STAMPED]diff --git a/core/switch_core/gateway/dependencies.py b/core/switch_core/gateway/dependencies.pyindex 2123d0611..4b1a61e05 100644--- a/core/switch_core/gateway/dependencies.py+++ b/core/switch_core/gateway/dependencies.py@@ -110,8 +110,9 @@ async def get_system_session() -> AsyncIterator[AsyncSession]:reads as a deliberate, reviewable exception rather than anaccidentally-unscoped session, and so the guard test above can hold for`get_session` without exemptions. Nothing about opening it differs from- `get_session` today, since row-level security is not enforced yet; the two- must stay interchangeable in behaviour only, never in name.+ `get_session` today — it is the same session and the same hook, simply+ with nothing bound around it — so the two must stay interchangeable in+ behaviour only, never in name."No tenant bound" is not the same as "writes land nowhere in particular":the OIDC callback provisions a user and picks its tenant explicitly (seediff --git a/core/switch_core/main.py b/core/switch_core/main.pyindex eaaa0d6d7..c595e1a87 100644--- a/core/switch_core/main.py+++ b/core/switch_core/main.py@@ -312,6 +312,16 @@ async def run(config: SwitchConfig) -> None:# one tenant at a time. Which tenants there are is the one read that# cannot be scoped to any of them, so it goes through the exemption# (`db/tenant_lookup.py`) rather than through a session.+ #+ # No `row.tenant_id == tenant_id` filter here, unlike the fan-outs in+ # `ClientLifecycleService`: `ReferenceTypeStore.list_all` names `tenant_id+ # = require_tenant_id()` in its own query rather than leaning on the+ # policy for it (it is the one store that has to, per `db/tenant_lookup.py`+ # and `reference_type_store.py` — `reference_types` has no `id` column of+ # its own to filter by otherwise). That WHERE clause narrows correctly on+ # an owner connection too, so each pass through this loop already sees+ # only the tenant it just bound; a row-by-row filter on top of it would+ # compare a value against one it can never disagree with.tenant_ids = await all_tenant_ids(session_factory)for tenant_id in tenant_ids:async with tenant_session(session_factory, tenant_id) as session:@@ -690,6 +700,17 @@ async def _seed_agent_registration_bootstrap_key(key_tenant_id = await _bootstrap_key_tenant(session_factory, api_key_store, tenant_ids)+ # Retiring a stale legacy key is deployment-wide even though adopting one+ # is not. The block below works in a single tenant, which is right for the+ # key — it is one per deployment — and wrong for the sweep beside it: a+ # legacy admin-owned registration row in some *other* tenant still+ # authenticates and still registers agents with admin authority, which is+ # the case that sweep exists to stop. It reads every tenant, one at a+ # time, and hands back what it revoked so the owner's record below covers+ # those hashes too.+ retired_elsewhere = await _retire_legacy_keys_outside(+ session_factory, api_key_store, tenant_ids, key_tenant_id+ )async with tenant_session(session_factory, key_tenant_id) as session:token_hash = hashlib.sha256(config.agent_registration_token.encode()@@ -725,6 +746,10 @@ async def _seed_agent_registration_bootstrap_key()revoked_hashes: list[str] = list(raw_revoked_hashes or [])meta_dirty = False+ for retired in retired_elsewhere:+ if retired not in revoked_hashes:+ revoked_hashes.append(retired)+ meta_dirty = Trueif (bootstrap_key is None@@ -867,6 +892,60 @@ async def _seed_agent_registration_bootstrap_key(await session.commit()+async def _retire_legacy_keys_outside(+ session_factory: async_sessionmaker[AsyncSession],+ api_key_store: ApiKeyStore,+ tenant_ids: list[str],+ key_tenant_id: str,+) -> list[str]:+ """Retire every legacy admin-owned registration key outside `key_tenant_id`.++ The seeding proper works in one tenant, because the bootstrap key is one+ per deployment. This sweep cannot: a `type: "registration"` row carrying+ the legacy label is indistinguishable from a live credential that+ authenticates as the admin, and one sitting in another tenant goes on+ doing so. Adoption stays where the key is; retirement goes everywhere,+ because "no longer authenticates" is not a per-tenant claim.++ Retired rather than deleted, for the reason the seeding gives at length:+ every consumer already refuses `RETIRED_KEY_TYPE`, so this stops it+ authenticating exactly as deletion would, while the row stays on the API+ Keys page with a label saying why. The hashes come back so the bootstrap+ owner's `revoked_hashes` covers them and restoring an old .env cannot+ bring one back through the rotation path.+ """+ retired: list[str] = []+ for tenant_id in tenant_ids:+ if tenant_id == key_tenant_id:+ continue+ async with tenant_session(session_factory, tenant_id) as session:+ stale_rows = [+ row+ for row in await api_key_store.get_by_label(+ session, LEGACY_BOOTSTRAP_KEY_LABEL+ )+ if row.type == "registration" and row.tenant_id == tenant_id+ ]+ for stale in stale_rows:+ stale.type = RETIRED_KEY_TYPE+ stale.label = f"{stale.label} (retired: stale, no longer authenticates)"+ retired.append(stale.key_hash)+ logger.warning(+ "Retired a stale admin-owned registration key (id %s) in "+ "tenant %s: the deployment's agent-registration bootstrap "+ "key lives in tenant %s, so this one authenticates nothing "+ "the operator intends and would otherwise keep registering "+ "agents with admin authority indefinitely. It is now "+ "visible on the API Keys page for a human to review.",+ stale.id,+ tenant_id,+ key_tenant_id,+ )+ if stale_rows:+ await session.commit()+ return retired++async def _bootstrap_key_tenant(session_factory: async_sessionmaker[AsyncSession],api_key_store: ApiKeyStore,diff --git a/core/switch_core/migrations/versions/9c41a7b0e5d8_tenant_lookup_functions.py b/core/switch_core/migrations/versions/9c41a7b0e5d8_tenant_lookup_functions.pyindex b2c000c0f..eefd3b59f 100644--- a/core/switch_core/migrations/versions/9c41a7b0e5d8_tenant_lookup_functions.py+++ b/core/switch_core/migrations/versions/9c41a7b0e5d8_tenant_lookup_functions.py@@ -24,8 +24,16 @@ cardinality and refuses rather than picking.Each parameter is prefixed `p_`, which is load-bearing rather than a stylechoice: a `LANGUAGE sql` parameter spelled like a column of a table in its own-query resolves to the column, so `WHERE id = client_id` against a table with a-`client_id` column compares that column with itself and matches every row.+query resolves to the column, not to the parameter — and which way that fails+depends on which column it shadows. Against a table with a `client_id`+column, `WHERE id = client_id` becomes a comparison between two columns of+the same row, `id` and the table's own `client_id`; that is false for every+row and so a confident, silent *zero* rows, not every row. The sharper+failure needs the parameter named after the very column being compared —+`WHERE key_hash = key_hash`, say — which shadows to a column compared with+itself, true unconditionally, and matches every row regardless of the+argument. Either way the prefix makes the shadowing unrepresentable rather+than a thing to check per function.`SET search_path = pg_catalog, public, pg_temp`, with `pg_temp` last, per thePostgres note on writing `SECURITY DEFINER` functions safely: a role able todiff --git a/core/switch_core/provisioning/postgres.py b/core/switch_core/provisioning/postgres.pyindex f5ac7c271..dbbb34f45 100644--- a/core/switch_core/provisioning/postgres.py+++ b/core/switch_core/provisioning/postgres.py@@ -98,7 +98,12 @@ class PostgresProvisioning:ClientRoom, {"client_id": client_id, "room_id": switch_room_id})- if await self._invites.invite(user_id, room_id):+ # By client id, not `user_id`: `clients.matrix_user_id` is unique per+ # tenant, and every tenant's admin client carries the same one, so+ # waking "the client for @switch-admin" would wake whichever tenant's+ # transport last claimed the slot. `_resolve` above has already turned+ # the handle into this tenant's client row.+ if await self._invites.invite(client_id, room_id):returnif existing is not None:returndiff --git a/core/switch_core/transport/invites.py b/core/switch_core/transport/invites.pyindex 60969f0cb..22a9dee99 100644--- a/core/switch_core/transport/invites.py+++ b/core/switch_core/transport/invites.py@@ -16,9 +16,21 @@ code path as before.another replica of switch-core would not hear it. That is the same constraintMatrix sync sessions imposed and the reason switch-core is single-replicatoday; lifting it is its own step, and until then this is no worse than what it-replaces. When nobody is listening for a user the caller writes the membership-itself, so the room is still joined the next time that client starts — the-invitation is a wake-up, never the record.+replaces. When nobody is listening for a client the caller writes the+membership itself, so the room is still joined the next time that client+starts — the invitation is a wake-up, never the record.++**Keyed by client id, which is a primary key.** It used to be keyed by+`matrix_user_id`, and that was wrong the moment a second tenant existed:+`clients.matrix_user_id` is unique *per tenant*, not globally, and+`ensure_system_client` deliberately gives every tenant's admin client the same+`@switch-admin:<server>` handle. Two tenants running an admin client meant one+slot and one winner, so an invitation for tenant A's admin woke tenant B's,+which could not resolve the room and logged a swallowed error — while `invite`+returned True, telling the caller a live client had joined itself, so the+membership row was never written either. A tenant's rooms silently had no admin+participant. `clients.id` is a uuid primary key, so there is nothing for a+tenant to disambiguate and the collision is unrepresentable."""from __future__ import annotations@@ -32,42 +44,42 @@ InviteHandler = Callable[[str], Awaitable[None]]class InviteBus:- """Routes an invitation to the live transport for a user, if there is one."""+ """Routes an invitation to the live transport for a client, if there is one."""def __init__(self) -> None:self._handlers: dict[str, InviteHandler] = {}- def register(self, user_id: str, handler: InviteHandler) -> None:- """Make `handler` the live listener for `user_id`, displacing any other.+ def register(self, client_id: str, handler: InviteHandler) -> None:+ """Make `handler` the live listener for `client_id`, displacing any other.- One slot per user, so a client that starts while an older instance of+ One slot per client, so a client that starts while an older instance ofitself is still unwinding takes the slot from it. That is the rightanswer — the newer one is the client with a live receive loop — and itis why `unregister` wants the handler back."""- self._handlers[user_id] = handler+ self._handlers[client_id] = handler- def unregister(self, user_id: str, handler: InviteHandler) -> None:+ def unregister(self, client_id: str, handler: InviteHandler) -> None:"""Give the slot up, but only if `handler` still holds it.A transport unregisters from the `finally` of its receive loop, which- runs whenever that loop ends — including long after a newer client for- the same user took the slot. Clearing by user id alone would deafen- that newer client: `invite` would find nobody listening, the caller- would write the membership row itself, and the client would sit in a- room it was never told it was in until something restarted it.+ runs whenever that loop ends — including long after a newer instance of+ the same client took the slot. Clearing by id alone would deafen that+ newer one: `invite` would find nobody listening, the caller would write+ the membership row itself, and the client would sit in a room it was+ never told it was in until something restarted it."""- if self._handlers.get(user_id) == handler:- del self._handlers[user_id]+ if self._handlers.get(client_id) == handler:+ del self._handlers[client_id]- async def invite(self, user_id: str, transport_room_id: str) -> bool:- """Tell `user_id` it is in `transport_room_id`.+ async def invite(self, client_id: str, transport_room_id: str) -> bool:+ """Tell `client_id` it is in `transport_room_id`.Returns whether anyone was listening. False is not an error — a clientthat is not running has nothing to wake — but the caller then ownswriting the membership, so the answer must not be ignored."""- handler = self._handlers.get(user_id)+ handler = self._handlers.get(client_id)if handler is None:return Falseawait handler(transport_room_id)diff --git a/core/switch_core/transport/postgres.py b/core/switch_core/transport/postgres.pyindex fdb0567c6..9e83b0a2f 100644--- a/core/switch_core/transport/postgres.py+++ b/core/switch_core/transport/postgres.py@@ -142,12 +142,6 @@ class PostgresTransport:# A room's transport id never changes, so this only grows and never# goes stale.self._room_ids: dict[str, str] = {}- # Switch room id -> the tenant that room belongs to. Filled by the same- # lookup that fills `_room_ids`, from the same row; a room's tenant is- # as immutable as its transport id. Each unit of work (one delivery,- # one send) binds the room it is actually for rather than reading a- # tenant off this transport.- self._room_tenants: dict[str, str] = {}# This client's own tenant, resolved once from its row and then held.# Not a guess and not ambient: `client_rooms` carries composite foreign# keys to both `clients` and `rooms`, so every room this client can be@@ -190,12 +184,12 @@ class PostgresTransport:for transport_room_id in rooms:await self._watch(transport_room_id)self._receiving = True- self._invites.register(self.user_id, self._on_invited)+ self._invites.register(self.client_id, self._on_invited)try:await self._closed.wait()finally:self._receiving = False- self._invites.unregister(self.user_id, self._on_invited)+ self._invites.unregister(self.client_id, self._on_invited)self._unwatch_all()delivery.cancel()@@ -354,9 +348,13 @@ class PostgresTransport:transport_room_id = self._watching.get(room_id)if transport_room_id is None:return- # `_watch` cannot have set up delivery for this room without already- # resolving it, which is what fills this in — see `_resolve_room`.- tenant_id = self._room_tenants[room_id]+ # Every room this client can be in is in this client's own tenant --+ # `client_rooms` and `messages` both key to `rooms` and to `clients`+ # through `tenant_id` -- so there is one answer here rather than one+ # per room. This used to be a per-room map filled from each room's own+ # row; the map only ever held this value, and two sources for one fact+ # is how they come to disagree.+ tenant_id = await self._tenant()# Bound for the read *and* the delivery below: a handler (posting to a# bridge, gating a command) opens its own sessions rather than reusing# this one, and those still need the room's tenant. The contextvar@@ -704,7 +702,6 @@ class PostgresTransport:if room is None:raise TransportError(f"{transport_room_id} is not a Switch room")self._room_ids[transport_room_id] = room.id- self._room_tenants[room.id] = room.tenant_idreturn room.idasync def _resolve_room_and_tenant(self, transport_room_id: str) -> tuple[str, str]:@@ -717,14 +714,13 @@ class PostgresTransport:both narrower and exact. Once cached, resolving costs nothing: nosession, no query, no round trip."""+ tenant_id = await self._tenant()cached = self._room_ids.get(transport_room_id)if cached is not None:- return cached, self._room_tenants[cached]- async with tenant_session(- self._session_factory, await self._tenant()- ) as session:+ return cached, tenant_id+ async with tenant_session(self._session_factory, tenant_id) as session:room_id = await self._resolve_room(session, transport_room_id)- return room_id, self._room_tenants[room_id]+ return room_id, tenant_idasync def _tenant(self) -> str:"""This client's tenant, resolved once from its own row.diff --git a/core/tests/integration/conftest.py b/core/tests/integration/conftest.pyindex 189e3a806..3655ff8e3 100644--- a/core/tests/integration/conftest.py+++ b/core/tests/integration/conftest.py@@ -89,10 +89,15 @@ from switch_core.transport.invites import InviteBus# ── Mirrors deploy/local/docker-compose.yml — keep in sync ──────────────────────POSTGRES_IMAGE = "postgres:16-alpine"-# The role the application runs as here. Named the same as the one-# `deploy/local/docker-compose.yml` creates, so what this suite exercises and-# what a developer's stack runs are the same shape.-RUNTIME_ROLE = "switch_app"+# Prefix for the role the application runs as here. `deploy/local/docker-compose.yml`+# creates a fixed-name `switch_app`, and this suite's role is the same shape —+# a plain restricted LOGIN role, granted the same way — but not the same name:+# roles are cluster-wide, and a crashed session that never reached its own+# teardown would otherwise leave `switch_app` behind for the next run's+# `CREATE ROLE` to collide with. Each session gets its own `switch_app_<hex>`+# instead, the same shape `rls_harness` (`core/tests/conftest.py`) already uses+# for the same reason.+RUNTIME_ROLE_PREFIX = "switch_app"# ── Throwaway test constants (not secrets — local ephemeral infra) ──────────────SERVER_NAME = "localhost"@@ -375,8 +380,8 @@ async def session_env(switch_stack: StackInfo) -> AsyncIterator[SessionEnv]:either exercises the real code against a database with no isolation in it— which is exactly how nineteen call sites came to depend on reads adeployed system refuses, and why several of them were invisible until a- row existed. Wiring the app to `switch_app` makes this suite the thing- that catches the twentieth.+ row existed. Wiring the app to a restricted role of its own makes this+ suite the thing that catches the twentieth.The owner connection stays, for the two jobs that genuinely need DDL andthat the application never does: building the schema, and truncating it@@ -401,6 +406,11 @@ async def session_env(switch_stack: StackInfo) -> AsyncIterator[SessionEnv]:await conn.run_sync(Base.metadata.create_all)await _seed_tenant_zero(owner_engine)+ # Cluster-wide and unique per session, not the fixed name Compose creates+ # (see `RUNTIME_ROLE_PREFIX` above): a session that crashes before its own+ # teardown runs must not leave a role behind for the next run's `CREATE+ # ROLE` to collide with.+ runtime_role = f"{RUNTIME_ROLE_PREFIX}_{uuid.uuid4().hex[:12]}"runtime_password = uuid.uuid4().hexasync with owner_engine.begin() as conn:# CREATE ROLE takes no bind parameter for PASSWORD, so a fresh random@@ -409,15 +419,15 @@ async def session_env(switch_stack: StackInfo) -> AsyncIterator[SessionEnv]:# a privilege production has not got.await conn.execute(text(- f'CREATE ROLE "{RUNTIME_ROLE}" LOGIN '+ f'CREATE ROLE "{runtime_role}" LOGIN 'f"PASSWORD '{runtime_password}' ""NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS"))- await grant_runtime_role(conn, RUNTIME_ROLE)+ await grant_runtime_role(conn, runtime_role)config = _build_config(- switch_stack, db_name, user=RUNTIME_ROLE, password=runtime_password+ switch_stack, db_name, user=runtime_role, password=runtime_password)engine = create_engine_from_config(config)session_factory = create_session_factory(engine)@@ -460,14 +470,14 @@ async def session_env(switch_stack: StackInfo) -> AsyncIterator[SessionEnv]:# OWNED BY` revokes what survives the database drop; it owns nothing,# so there is nothing for it to drop.async with owner_engine.begin() as conn:- await conn.execute(text(f'DROP OWNED BY "{RUNTIME_ROLE}"'))+ await conn.execute(text(f'DROP OWNED BY "{runtime_role}"'))await owner_engine.dispose()admin_conn = await asyncpg.connect(admin_dsn)try:await admin_conn.execute(f'DROP DATABASE IF EXISTS "{db_name}" WITH (FORCE)')- await admin_conn.execute(f'DROP ROLE IF EXISTS "{RUNTIME_ROLE}"')+ await admin_conn.execute(f'DROP ROLE IF EXISTS "{runtime_role}"')finally:await admin_conn.close()diff --git a/core/tests/switch_core/db/stores/test_reference_type_store.py b/core/tests/switch_core/db/stores/test_reference_type_store.pyindex f12cc98a6..9ce531090 100644--- a/core/tests/switch_core/db/stores/test_reference_type_store.py+++ b/core/tests/switch_core/db/stores/test_reference_type_store.py@@ -10,7 +10,6 @@ from switch_core.db.models import Reference, ReferenceType, Tenant, Userfrom switch_core.db.session_scope import tenant_sessionfrom switch_core.db.stores.reference_store import ReferenceStorefrom switch_core.db.stores.reference_type_store import ReferenceTypeStore-from switch_core.tenant_context import tenant_scopeasync def _make_user(session: AsyncSession, name: str) -> User:@@ -270,32 +269,58 @@ class TestReferenceTypesAreScopedToTheBoundTenant:async def test_a_write_lands_in_the_bound_tenant(self, session_factory: async_sessionmaker[AsyncSession]) -> None:+ """The write under test happens on a session opened with+ `tenant_session(other)`, not on the tenant-zero session `ada` and the+ `other` tenant row are arranged on with `tenant_scope(other)` layered+ over it afterwards. `set_config` runs once, at `after_begin`, so by+ the time this session's first flush stamped its transaction with+ tenant zero, entering `tenant_scope(other)` around the create would+ only rebind the contextvar — the transaction, and so the row's own+ `with check`, would still see tenant zero. Opening a fresh session+ inside the binding stamps its transaction with `other` from the+ start.+ """other = f"tenant-{uuid.uuid4().hex[:8]}"store = ReferenceTypeStore()async with session_factory() as session:session.add(Tenant(id=other, slug=other, name=other))ada = await _make_user(session, "ada")- await session.flush()- with tenant_scope(other):- created = await store.create(session, _type("notion", ada.id))await session.commit()- assert created.tenant_id == other++ async with tenant_session(session_factory, other) as other_session:+ created = await store.create(other_session, _type("notion", ada.id))+ await other_session.commit()++ assert created.tenant_id == otherasync def test_a_read_does_not_see_another_tenants_type(self, session_factory: async_sessionmaker[AsyncSession]) -> None:+ """`other`'s type is written on a session opened with+ `tenant_session(other)` rather than on the tenant-zero session used+ to arrange `other` itself, with `tenant_scope(other)` entered around+ just the create. The stamp that fixes what a transaction's rows are+ checked against is set once, when it begins, so rebinding the+ contextvar over an already-open transaction changes nothing the+ database can see — the row would land under whatever tenant the+ session's first statement stamped it with, not `other`. The read+ assertions below run on a fresh tenant-zero session, matching what a+ request bound to tenant zero would actually see.+ """other = f"tenant-{uuid.uuid4().hex[:8]}"store = ReferenceTypeStore()async with session_factory() as session:session.add(Tenant(id=other, slug=other, name=other))ada = await _make_user(session, "ada")- await session.flush()- with tenant_scope(other):- await store.create(- session, _type("notion", ada.id, read_visibility="public")- )await session.commit()+ async with tenant_session(session_factory, other) as other_session:+ await store.create(+ other_session, _type("notion", ada.id, read_visibility="public")+ )+ await other_session.commit()++ async with session_factory() as session:# Bound to tenant zero, which `session_factory` binds by default.assert await store.get(session, "notion") is Noneassert await store.get_many(session, ["notion"]) == []@@ -308,30 +333,50 @@ class TestReferenceTypesAreScopedToTheBoundTenant:) -> None:"""The collision the per-tenant primary key exists for, through thestore: two customers both call something `notion`, and editing or- deleting one leaves the other alone."""+ deleting one leaves the other alone.++ Tenant zero's row is created and deleted through ordinary sessions+ from `session_factory`, which binds tenant zero for the whole test.+ `other`'s row is created, updated and read back each through its own+ session opened with `tenant_session(other)` — never by entering+ `tenant_scope(other)` on a session already carrying tenant zero's+ stamp. `set_config` is issued once, when a transaction begins, so a+ session whose transaction already began under tenant zero keeps+ telling the database it is tenant zero no matter what the contextvar+ is rebound to afterwards; only opening the session inside the new+ binding moves it.+ """other = f"tenant-{uuid.uuid4().hex[:8]}"store = ReferenceTypeStore()+async with session_factory() as session:session.add(Tenant(id=other, slug=other, name=other))ada = await _make_user(session, "ada")- await session.flush()- await store.create(session, _type("notion", ada.id))- with tenant_scope(other):- await store.create(session, _type("notion", ada.id))await session.commit()- with tenant_scope(other):- await store.update_fields(session, "notion", display_name="Theirs")+ async with session_factory() as session:+ await store.create(session, _type("notion", ada.id))await session.commit()+ async with tenant_session(session_factory, other) as other_session:+ await store.create(other_session, _type("notion", ada.id))+ await other_session.commit()++ async with tenant_session(session_factory, other) as other_session:+ await store.update_fields(other_session, "notion", display_name="Theirs")+ await other_session.commit()++ async with session_factory() as session:mine = await store.get(session, "notion")- assert mine is not None and mine.display_name == "Notion"+ assert mine is not None and mine.display_name == "Notion"+ async with session_factory() as session:await store.delete(session, "notion")await session.commit()- with tenant_scope(other):- theirs = await store.get(session, "notion")- assert theirs is not None and theirs.display_name == "Theirs"++ async with tenant_session(session_factory, other) as other_session:+ theirs = await store.get(other_session, "notion")+ assert theirs is not None and theirs.display_name == "Theirs"async def test_a_duplicate_slug_clashes_cleanly_when_another_tenant_has_it_too(self, session_factory: async_sessionmaker[AsyncSession]diff --git a/core/tests/switch_core/db/test_runtime_role.py b/core/tests/switch_core/db/test_runtime_role.pyindex 1ee0abad4..c854c0a7e 100644--- a/core/tests/switch_core/db/test_runtime_role.py+++ b/core/tests/switch_core/db/test_runtime_role.py@@ -54,32 +54,50 @@ class TestItPassesForARoleThePoliciesApplyTo:since boot would then succeed and the first query would fail. Thegrants come from `grant_runtime_role`, the same function a deploymentruns, so this is a statement about production and not about a fixture.++ Two tenants, and a row in each, so the count at the end says something.+ Against one tenant on a fresh schema `count(*) == 1` is satisfied by a+ database with no policies in it at all — it would prove the grants and+ nothing else, which is not what the name claims. With a second+ tenant's row present, the same 1 is the policy filtering."""- tenant_id = f"tenant-{uuid.uuid4().hex[:8]}"+ mine = f"tenant-{uuid.uuid4().hex[:8]}"+ theirs = f"tenant-{uuid.uuid4().hex[:8]}"async with rls_harness.owner() as session:+ for tenant_id in (mine, theirs):+ await session.execute(+ text("INSERT INTO tenants (id, slug, name) VALUES (:id, :id, :id)"),+ {"id": tenant_id},+ )await session.execute(- text("INSERT INTO tenants (id, slug, name) VALUES (:id, :id, :id)"),- {"id": tenant_id},+ text(+ "INSERT INTO room_groups (id, tenant_id, name) "+ "VALUES (:id, :t, 'somebody else')"+ ),+ {"id": str(uuid.uuid4()), "t": theirs},)await session.commit()async with rls_harness.restricted() as session:await session.execute(text("SELECT set_config('app.tenant_id', :t, true)"),- {"t": tenant_id},+ {"t": mine},)await session.execute(text("INSERT INTO room_groups (id, tenant_id, name) ""VALUES (:id, :t, 'a group')"),- {"id": str(uuid.uuid4()), "t": tenant_id},+ {"id": str(uuid.uuid4()), "t": mine},)visible = (await session.execute(text("SELECT count(*) FROM room_groups"))).scalar_one()await session.commit()- assert visible == 1+ assert visible == 1, (+ "the restricted role saw a row it does not own, so it can read and "+ "write but is not actually scoped"+ )class TestItRefusesEveryShapeThatWouldMakeThePoliciesInert:diff --git a/core/tests/switch_core/db/test_tenant_session_hook.py b/core/tests/switch_core/db/test_tenant_session_hook.pyindex 191a766ba..9d4115f47 100644--- a/core/tests/switch_core/db/test_tenant_session_hook.py+++ b/core/tests/switch_core/db/test_tenant_session_hook.py@@ -25,6 +25,7 @@ from sqlalchemy.orm import Sessionfrom switch_core.db import tenant_sessionfrom switch_core.db.base import Basefrom switch_core.db.engine import create_session_factory+from switch_core.db.models import Tenantfrom switch_core.tenant_context import tenant_scopeTENANT_A = "tenant-hook-a"@@ -233,3 +234,73 @@ class TestNoLeakAcrossAPooledConnection:pid_c, setting_c = await _pid_and_setting(session)assert pid_c == pid_aassert setting_c is None+++class TestABindingThatArrivesTooLateIsRefused:+ """`set_config` rides `after_begin`, so it is issued once — when the+ transaction opens. Binding a tenant after that rebinds a contextvar and+ nothing the database can see, and the call site looks correct while every+ statement runs under whatever the transaction was actually told.++ On a connection Postgres exempts from the policies (every environment+ before the runtime role, and the whole unit suite still) both halves of+ that are invisible: no policy narrows the reads, none rejects the writes.+ So the process refuses rather than leaving it to a reviewer's eye —+ `TenantBindingDriftError`, from two hooks, because a read and a write take+ different paths out of a `Session`.+ """++ async def test_a_read_after_a_late_binding_raises(+ self, single_connection_session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ async with single_connection_session_factory() as session:+ # Opens the transaction with nothing bound; the hook stamps None.+ assert (await _pid_and_setting(session))[1] is None+ with tenant_scope(TENANT_A):+ with pytest.raises(tenant_session.TenantBindingDriftError) as raised:+ await session.execute(text("SELECT 1"))+ assert "opened with tenant None" in str(raised.value)+ assert TENANT_A in str(raised.value)++ async def test_a_flush_after_a_late_binding_raises(+ self, single_connection_session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """The path `do_orm_execute` does not see, and the one that matters+ most: the unit of work emits its INSERTs to the connection directly+ rather than through `Session.execute`. A late binding followed only by+ `add` and `commit` is exactly how a row comes to claim one tenant on a+ transaction that carries another."""+ async with single_connection_session_factory() as session:+ assert (await _pid_and_setting(session))[1] is None+ with tenant_scope(TENANT_A):+ session.add(Tenant(id=TENANT_A, slug=TENANT_A, name=TENANT_A))+ with pytest.raises(tenant_session.TenantBindingDriftError):+ await session.flush()++ async def test_rebinding_the_same_tenant_is_not_drift(+ self, single_connection_session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """A redundant binding is not a wrong one. `tenant_session` nested+ inside a `tenant_scope` for the same tenant is an ordinary shape on the+ delivery path, and refusing it would be a false positive."""+ async with single_connection_session_factory() as session:+ with tenant_scope(TENANT_A):+ assert (await _pid_and_setting(session))[1] == TENANT_A+ with tenant_scope(TENANT_A):+ await session.execute(text("SELECT 1"))+ await session.commit()++ async def test_a_commit_clears_the_stamp_so_the_next_binding_is_free(+ self, single_connection_session_factory: async_sessionmaker[AsyncSession]+ ) -> None:+ """The escape the error message offers has to actually work. A session+ reused after a commit opens a fresh transaction, which gets its own+ `set_config` and its own stamp, so binding a different tenant between+ the two is correct rather than drift."""+ async with single_connection_session_factory() as session:+ with tenant_scope(TENANT_A):+ assert (await _pid_and_setting(session))[1] == TENANT_A+ await session.commit()+ with tenant_scope(TENANT_B):+ assert (await _pid_and_setting(session))[1] == TENANT_B+ await session.commit()diff --git a/core/tests/switch_core/provisioning/test_postgres_provisioning.py b/core/tests/switch_core/provisioning/test_postgres_provisioning.pyindex 6c943f146..fa992e330 100644--- a/core/tests/switch_core/provisioning/test_postgres_provisioning.py+++ b/core/tests/switch_core/provisioning/test_postgres_provisioning.py@@ -7,11 +7,13 @@ up holding — a membership, an arrival, and the absence of either.from __future__ import annotationsimport uuid+from typing import Anyimport pytestfrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker-from switch_core.db.models import Client, ClientRoom, Room+from switch_core.db.models import Client, ClientRoom, Room, Tenant+from switch_core.db.session_scope import tenant_sessionfrom switch_core.db.stores.client_store import ClientStorefrom switch_core.db.stores.message_store import MessageStorefrom switch_core.db.stores.room_store import RoomStore@@ -21,7 +23,9 @@ from switch_core.provisioning.postgres import (ProvisioningError,new_room_id,)+from switch_core.tenant_context import tenant_scopefrom switch_core.transport.invites import InviteBus+from tests.conftest import RLSHarnessasync def _client(session: AsyncSession, name: str = "someone") -> Client:@@ -118,8 +122,11 @@ class TestMembership:async def _handler(invited_to: str) -> None:woken.append(invited_to)+ # By client id: the bus is keyed on `clients.id`, not on the handle,+ # because `matrix_user_id` is unique per tenant and every tenant's+ # admin client carries the same one.invites = InviteBus()- invites.register(user_id, _handler)+ invites.register(client_id, _handler)await _provisioning(session_factory, invites).invite_to_room(transport_room_id, user_id)@@ -227,9 +234,91 @@ class TestMembership:await provisioning.invite_to_room(transport_room_id, user_id)invites = InviteBus()- invites.register(user_id, _handler)+ invites.register(client.id, _handler)await _provisioning(session_factory, invites).invite_to_room(transport_room_id, user_id)assert woken == [transport_room_id]+++class TestTwoTenantsSharingAHandle:+ """Two admin clients, one handle, and the invitation that used to go to+ the wrong one.++ `clients.matrix_user_id` is unique *per tenant*, and+ `ClientLifecycleService.ensure_system_client` deliberately gives every+ tenant's admin client the same `@switch-admin:<server>` — one row per+ tenant, one handle. The invite bus used to key its handler slot on that+ handle, process-wide, so the second admin client to start displaced the+ first and every invitation for either woke whichever had won.++ What made it silent is the return value. `invite` answered True because+ *a* handler existed, so `invite_to_room` took that as "a live client has+ joined itself" and returned without writing the membership row — while the+ woken transport, scoped to its own tenant, could not resolve a room in the+ other one and logged an error nobody was looking for. A tenant's rooms+ simply had no admin participant. The bus is keyed on `clients.id` now,+ which is a uuid primary key, so there is nothing left to disambiguate.++ Asked of `rls_harness.restricted` rather than the plain fixture, and it+ has to be: `ClientStore.get_by_matrix_user_id` calls `scalar_one_or_none`+ on a column that is unique per tenant, so on the owner connection — where+ no policy narrows the read — resolving the shared handle raises+ `MultipleResultsFound` before the bus is ever consulted. That is a known+ open item of this phase, and it is the policies that close it.+ """++ async def test_each_tenants_client_is_woken_for_its_own_room(+ self, rls_harness: RLSHarness+ ) -> None:+ handle = f"@switch-admin-{uuid.uuid4().hex[:8]}:test"+ tenants = [f"tenant-{uuid.uuid4().hex[:8]}" for _ in range(2)]+ rooms: dict[str, str] = {}+ clients: dict[str, str] = {}+ for tenant_id in tenants:+ # Each tenant's rows on a session opened *inside* that tenant's+ # binding: the `set_config` rides `after_begin`, so a scope entered+ # around an already-open transaction stamps nothing.+ async with tenant_session(rls_harness.owner, tenant_id) as session:+ session.add(Tenant(id=tenant_id, slug=tenant_id, name=tenant_id))+ await session.flush()+ client = Client(+ tenant_id=tenant_id,+ matrix_user_id=handle,+ display_name="admin",+ type="admin",+ )+ room = Room(+ tenant_id=tenant_id,+ matrix_room_id=new_room_id(),+ name="a room",+ description="",+ )+ session.add_all([client, room])+ await session.commit()+ clients[tenant_id] = client.id+ rooms[tenant_id] = room.matrix_room_id++ woken: dict[str, list[str]] = {tenant_id: [] for tenant_id in tenants}++ def _handler_for(tenant_id: str) -> Any:+ async def _handler(invited_to: str) -> None:+ woken[tenant_id].append(invited_to)++ return _handler++ invites = InviteBus()+ for tenant_id in tenants:+ invites.register(clients[tenant_id], _handler_for(tenant_id))++ provisioning = _provisioning(rls_harness.restricted, invites)+ for tenant_id in tenants:+ with tenant_scope(tenant_id):+ await provisioning.invite_to_room(rooms[tenant_id], handle)++ for tenant_id in tenants:+ assert woken[tenant_id] == [rooms[tenant_id]], (+ f"tenant {tenant_id}'s admin client was not the one woken for "+ "its own room; the invite bus routed by the shared handle"+ )diff --git a/core/tests/switch_core/test_room_service_default_bridge_multi_tenant.py b/core/tests/switch_core/test_room_service_default_bridge_multi_tenant.pyindex ea3498f60..b08381252 100644--- a/core/tests/switch_core/test_room_service_default_bridge_multi_tenant.py+++ b/core/tests/switch_core/test_room_service_default_bridge_multi_tenant.py@@ -31,6 +31,7 @@ import pytestfrom sqlalchemy.ext.asyncio import AsyncSession, async_sessionmakerfrom switch_core.db.models import Client, CollaborationBridge, Tenant+from switch_core.db.session_scope import tenant_sessionfrom switch_core.db.stores.agent_store import AgentStorefrom switch_core.db.stores.collaboration_bridge_store import CollaborationBridgeStorefrom switch_core.db.stores.room_store import RoomStore@@ -116,9 +117,22 @@ class _NoRunningClients:return []-async def _make_default_bridge(session: AsyncSession, *, tenant_id: str) -> str:- """A bridge client and a bridge nominated default, filed under `tenant_id`."""- with tenant_scope(tenant_id):+async def _make_default_bridge(+ session_factory: async_sessionmaker[AsyncSession], *, tenant_id: str+) -> str:+ """A bridge client and a bridge nominated default, filed under `tenant_id`.++ Opened on its own session, inside `tenant_session(tenant_id)`, rather+ than on a session shared with the other tenant's fixture rows with+ `tenant_scope(tenant_id)` layered around just this call. `set_config`+ rides `after_begin`, so it is issued once, the first time a session's+ transaction opens — a session whose transaction had already begun under+ a different tenant (or none, here, since this suite runs under+ `no_ambient_tenant`) would keep telling the database that regardless of+ what the contextvar is rebound to afterwards. Opening a fresh session+ inside the binding is what actually moves it.+ """+ async with tenant_session(session_factory, tenant_id) as session:client = Client(matrix_user_id=f"@bridge-{uuid.uuid4().hex[:8]}:switch.local",display_name="bridge client",@@ -135,6 +149,7 @@ async def _make_default_bridge(session: AsyncSession, *, tenant_id: str) -> str:)session.add(bridge)await session.flush()+ await session.commit()return bridge.id@@ -168,11 +183,11 @@ async def test_create_room_succeeds_for_each_tenants_own_default_bridge(Tenant(id=tenant_b, slug=tenant_b, name="B"),])- await session.flush()- bridge_a = await _make_default_bridge(session, tenant_id=tenant_a)- bridge_b = await _make_default_bridge(session, tenant_id=tenant_b)await session.commit()+ bridge_a = await _make_default_bridge(session_factory, tenant_id=tenant_a)+ bridge_b = await _make_default_bridge(session_factory, tenant_id=tenant_b)+matrix = _FakeMatrix()svc = _service(session_factory,diff --git a/justfile b/justfileindex cf3318d40..e80ef38d9 100644--- a/justfile+++ b/justfile@@ -38,7 +38,26 @@ init-env:echo " The stack binds to 127.0.0.1 only (set SWITCH_BIND_ADDR to expose it)."# ── Dev infrastructure ─────────────────────────────────────────────────────────+# Refuses an `.env` written before the two-role split, rather than starting a+# stack that cannot work. Without DB_OWNER_USER the postgres container comes up+# with an empty POSTGRES_USER, and the failure surfaces several steps later as+# something that looks unrelated. `just init-env` fills both for a new `.env`+# and deliberately never touches an existing one, so an `.env` from before this+# change is the ordinary case rather than an exotic one — worth one check here+# instead of a puzzle for whoever hits it.up:+ #!/usr/bin/env bash+ set -euo pipefail+ if [ -z "${DB_OWNER_USER:-}" ] || [ -z "${DB_OWNER_PASSWORD:-}" ]; then+ echo "✋ No DB_OWNER_USER / DB_OWNER_PASSWORD in .env." >&2+ echo " Switch runs as a restricted database role now, and the schema" >&2+ echo " owner is configured separately. Add both (see .env.example)," >&2+ echo " set DB_USER=switch_app, and run 'just reset' so the Postgres" >&2+ echo " volume is rebuilt with the runtime role created — 'init-db'" >&2+ echo " only creates it on a fresh volume." >&2+ echo " docs/old/LOCAL_DEVELOPMENT.md, 'Two database roles, not one'." >&2+ exit 1+ fidocker compose -f deploy/local/docker-compose.yml --project-directory . up -d --builddown: