Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

fix(db): close the cross-tenant gaps a review of the stack found (CHOO-2623)

Wojciech Wojtyniak440e70e038ad
diff --git a/.env.example b/.env.example
index 7a3bd0219..d1815527a 100644
--- a/.env.example
+++ b/.env.example
@@ -57,9 +57,11 @@ DB_NAME=switch
# Emitted on every JSON line as `service` / `env`.
# SERVICE_NAME=switch-core
# ENVIRONMENT=development
-# The tenant every log line is attributed to. Switch serves one organisation
-# per deployment, so this is a deployment-wide constant — set it per deployment
-# and the logs of two of them can be told apart in one pipeline.
+# What a log line says when no tenant is bound to it. A request logs the
+# caller's tenant and background work logs the tenant of the row it is acting
+# on, so this only covers lines belonging to neither — which are also the lines
+# that can write nothing. Not a tenant id, and setting it to one would make
+# every unattributed line look like that tenant's own.
# TENANT_ID=default
# ── Client identity ──────────────────────────────────────────────────────────
diff --git a/core/switch_core/bridges/agent/protocol/service.py b/core/switch_core/bridges/agent/protocol/service.py
index 3695760eb..de16930c8 100644
--- a/core/switch_core/bridges/agent/protocol/service.py
+++ b/core/switch_core/bridges/agent/protocol/service.py
@@ -467,12 +467,17 @@ class ProtocolService:
The token also decides the *tenant* the new agent and its API key land
in, and that is bound here rather than left to the caller. The HTTP
registration endpoint has ``BearerAuthMiddleware`` in front of it doing
- the same thing; this path has nothing in front of it at all — a
- server-side connector registers its agents from a startup task
- (``server_connectors/core.py``), where no request and so no tenant
- exists. Without this the rows would fall back to tenant zero, and
- because bearer authentication then reads ``api_keys.tenant_id`` back as
- the source of truth, that guess would confirm itself forever.
+ the same thing. This path does not: a server-side connector calls it
+ in-process from ``server_connectors/core.py``, which binds its own
+ row's tenant around the call (``_register_agent``) — from boot, and
+ from ``POST /connectors``, where the tenant already bound is the
+ *requesting operator's*, which is not necessarily the connector's.
+
+ So the binding here is not filling a vacuum; it is overriding whatever
+ the caller had, with the one thing that is authoritative for these
+ rows. Bearer authentication later reads ``api_keys.tenant_id`` back as
+ the source of truth, so a key filed under the wrong tenant would keep
+ confirming itself on every subsequent call.
"""
token_hash = hashlib.sha256(registration_token.encode()).hexdigest()
# Resolving a credential is unscoped by definition — `key_hash` is one
diff --git a/core/switch_core/bridges/agent/server_connectors/lifecycle.py b/core/switch_core/bridges/agent/server_connectors/lifecycle.py
index 8b1d804f3..b36364885 100644
--- a/core/switch_core/bridges/agent/server_connectors/lifecycle.py
+++ b/core/switch_core/bridges/agent/server_connectors/lifecycle.py
@@ -91,6 +91,11 @@ class ServerSideConnectorLifecycleService:
type="registration",
)
+ # Reached only from `POST /connectors`, so the session inherits the
+ # tenant the auth dependency bound and both rows below land in the
+ # operator's own — which is the answer, since it is their connector.
+ # `start` then re-reads the row unscoped to learn that tenant back,
+ # rather than assuming this one is still bound by then.
async with self._session_factory() as session:
await self._api_key_store.create(session, reg_key)
@@ -162,6 +167,25 @@ class ServerSideConnectorLifecycleService:
await self.stop(connector_id)
async def remove(self, connector_id: str) -> None:
+ """Delete a connector: its agents, its running core, and its row.
+
+ Scoped to the caller, and checked before anything is torn down. The
+ session inherits the tenant the request bound (`gateway/auth.py`), so
+ a connector belonging to another tenant is simply not there — but
+ `_cores` is a process-wide registry with every tenant's connectors in
+ it, so tearing down first and deleting second let a caller from the
+ wrong tenant stop another tenant's connector and delete its agents,
+ then match zero rows and report success. The read below is what makes
+ the tenant check happen before the damage rather than after it.
+
+ `ServerConnectorStore.delete` raises when it matches nothing, so the
+ window between the two — the row deleted underneath us — surfaces as
+ an error rather than as a second false success.
+ """
+ async with self._session_factory() as session:
+ if await self._connector_store.get(session, connector_id) is None:
+ raise ValueError(f"Connector not found: {connector_id}")
+
core = self._cores.get(connector_id)
if core is not None:
await core.delete_agents()
diff --git a/core/switch_core/clients/client_lifecycle_service.py b/core/switch_core/clients/client_lifecycle_service.py
index 222b5a34a..42e6d3bd3 100644
--- a/core/switch_core/clients/client_lifecycle_service.py
+++ b/core/switch_core/clients/client_lifecycle_service.py
@@ -39,6 +39,12 @@ class ClientLifecycleService:
self._config = config
self._clients: dict[str, ClientBase[ClientConfig]] = {}
self._client_types: dict[str, str] = {}
+ # Which tenant each running client's row belongs to. A `ClientBase`
+ # does not carry it — a client's own work resolves the tenant from the
+ # room it is acting on — but this registry holds every tenant's
+ # clients at once, so anything picking clients *out* of it has to be
+ # able to say which tenant it wants. See `get_by_type`.
+ self._client_tenants: dict[str, str] = {}
self._tasks: dict[str, asyncio.Task[None]] = {}
def _make_user_id(self, localpart: str) -> str:
@@ -99,6 +105,7 @@ class ClientLifecycleService:
client = self._client_factory.create(record)
self._clients[record.id] = client
self._client_types[record.id] = record.type
+ self._client_tenants[record.id] = record.tenant_id
self._start_task(record.id, client)
async def create_client(
@@ -132,6 +139,7 @@ class ClientLifecycleService:
client = self._client_factory.create(record)
self._clients[record.id] = client
self._client_types[record.id] = record.type
+ self._client_tenants[record.id] = record.tenant_id
self._start_task(record.id, client)
logger.info(
"Started client %s (%s)", record.display_name, record.matrix_user_id
@@ -163,6 +171,7 @@ class ClientLifecycleService:
self._cancel_task(client_id)
del self._clients[client_id]
self._client_types.pop(client_id, None)
+ self._client_tenants.pop(client_id, None)
logger.info("Stopped client %s", client_id)
async def stop_all(self) -> None:
@@ -173,6 +182,7 @@ class ClientLifecycleService:
task.cancel()
self._clients.clear()
self._client_types.clear()
+ self._client_tenants.clear()
self._tasks.clear()
async def remove(self, client_id: str) -> None:
@@ -192,11 +202,27 @@ class ClientLifecycleService:
return client # type: ignore[return-value]
return None
- def get_by_type(self, client_type: str) -> list[ClientBase[ClientConfig]]:
+ def get_by_type(
+ self, client_type: str, tenant_id: str
+ ) -> list[ClientBase[ClientConfig]]:
+ """This tenant's running clients of `client_type`.
+
+ The tenant is required rather than optional, and that is the fix for a
+ real crash. `clients` is scoped, so `ensure_system_client` creates one
+ admin client *per tenant* and this registry holds all of them at once.
+ A caller that asked by type alone got every tenant's, and putting one
+ of those into another tenant's room is not a bookkeeping slip:
+ `client_rooms` carries a composite foreign key on
+ `(tenant_id, client_id)`, so the insert raises
+ `ForeignKeyViolationError`. In `reconcile_room_clients` that runs at
+ startup and outside any per-room try/except, so the second tenant to
+ own a room takes the whole process down with it.
+ """
return [
client
for client_id, client in self._clients.items()
if self._client_types.get(client_id) == client_type
+ and self._client_tenants.get(client_id) == tenant_id
]
def _start_task(self, client_id: str, client: ClientBase[ClientConfig]) -> None:
@@ -231,4 +257,6 @@ class ClientLifecycleService:
"Client %s (%s) crashed", client.display_name, client.matrix_user_id
)
self._clients.pop(client_id, None)
+ self._client_types.pop(client_id, None)
+ self._client_tenants.pop(client_id, None)
self._tasks.pop(client_id, None)
diff --git a/core/switch_core/config.py b/core/switch_core/config.py
index 299b524f3..d3e85b167 100644
--- a/core/switch_core/config.py
+++ b/core/switch_core/config.py
@@ -91,11 +91,17 @@ class SwitchConfig(BaseSettings):
service_name: str = "switch-core"
environment: str | None = None
- # The tenant a log line is attributed to when nothing bound a real one.
- # An authenticated request binds the caller's actual tenant
- # (`gateway/auth.py`, `bridges/agent/auth.py`); this is only what a
- # background task, a startup script, or anything else with no request
- # in flight falls back to.
+ # The placeholder a log line carries when no tenant is bound at all —
+ # deliberately not a tenant id, and it must not be set to one. An
+ # authenticated request binds the caller's tenant (`gateway/auth.py`,
+ # `bridges/agent/auth.py`); background work binds the tenant of the row it
+ # is acting on (`tenant_context.py`); `LogContextFilter` prefers either
+ # over this. What is left is code that has bound neither, which cannot
+ # write a scoped row at all — `db/models.require_tenant_id` raises — so
+ # this value never names where anything landed. Set it to a real tenant's
+ # id and it starts to: every unattributed line in the deployment would be
+ # indistinguishable from that tenant's own when an operator filters on
+ # `tenant_id`.
tenant_id: str = "default"
server_host: str = "0.0.0.0"
diff --git a/core/switch_core/db/stores/server_connector_store.py b/core/switch_core/db/stores/server_connector_store.py
index c0ac0c3f8..86ac7321a 100644
--- a/core/switch_core/db/stores/server_connector_store.py
+++ b/core/switch_core/db/stores/server_connector_store.py
@@ -37,7 +37,16 @@ class ServerConnectorStore:
await session.flush()
async def delete(self, session: AsyncSession, connector_id: str) -> None:
+ """Delete the connector, raising when this session cannot see it.
+
+ Matching `update_status` above rather than the silent no-op this used
+ to be. Row-level security makes "no such connector" and "not this
+ tenant's connector" the same empty lookup, so returning quietly is not
+ a harmless idempotent delete — it is the shape a cross-tenant delete
+ takes, and the caller went on to log a removal that had not happened.
+ """
connector = await session.get(ServerConnector, connector_id)
- if connector:
- await session.delete(connector)
- await session.flush()
+ if connector is None:
+ raise ValueError(f"Connector not found: {connector_id}")
+ await session.delete(connector)
+ await session.flush()
diff --git a/core/switch_core/db/stores/tenant_member_store.py b/core/switch_core/db/stores/tenant_member_store.py
index 0945ab862..c439fdd41 100644
--- a/core/switch_core/db/stores/tenant_member_store.py
+++ b/core/switch_core/db/stores/tenant_member_store.py
@@ -22,11 +22,18 @@ class TenantMembershipError(Exception):
class TenantMemberStore:
- async def create(
- self, session: AsyncSession, *, tenant_id: str, user_id: str, role: str
- ) -> None:
- session.add(TenantMember(tenant_id=tenant_id, user_id=user_id, role=role))
- await session.flush()
+ """Reads memberships. Deliberately does not write them.
+
+ There was a `create` here and nothing ever called it. Membership is not a
+ thing code decides to write on its own: it is written by, and only by, the
+ paths that produce or repair an account, so that "every account has exactly
+ one" holds by construction rather than by everyone remembering. That write
+ is `UserStore.ensure_membership`, which is idempotent and derives the role
+ from the user. A second, unguarded way in — taking `tenant_id`, `user_id`
+ and `role` from whatever the caller felt like — is how an account ends up
+ with two, and `get_sole_tenant_id` below refuses to pick between two just
+ as firmly as it refuses to invent one out of zero.
+ """
async def get_sole_tenant_id(self, session: AsyncSession, user_id: str) -> str:
"""The one tenant this user belongs to, raising if that isn't true.
diff --git a/core/switch_core/db/stores/user_store.py b/core/switch_core/db/stores/user_store.py
index 53e4c5b83..a0feea59f 100644
--- a/core/switch_core/db/stores/user_store.py
+++ b/core/switch_core/db/stores/user_store.py
@@ -59,17 +59,27 @@ class UserStore:
"""
session.add(user)
await session.flush()
- await self._ensure_membership(session, user)
+ await self.ensure_membership(session, user)
- async def _ensure_membership(self, session: AsyncSession, user: User) -> None:
+ async def ensure_membership(self, session: AsyncSession, user: User) -> bool:
"""Give `user` a membership if they have none; leave any they have.
- Called from every path that can produce a user who would otherwise
- have zero: creation, and linking an identity to an account that
- predates memberships existing. Idempotent because the link path runs
- against accounts that usually already have one, and adding a second
- would be worse than adding none — resolution refuses to pick between
- two.
+ Returns whether one was written, so a repair path can say it repaired
+ something instead of logging on every boot.
+
+ Called from every path that can produce, or inherit, a user who would
+ otherwise have zero: creation, linking an identity to an account that
+ predates memberships existing, and the startup admin seeding
+ (`main.py`), which reaches accounts none of the others do. Idempotent
+ because most of those run against accounts that already have one, and
+ adding a second would be worse than adding none — resolution refuses
+ to pick between two.
+
+ Public for the sake of that last caller. An account with no membership
+ cannot sign in at all (`gateway/auth.py` answers 403), and until this
+ was reachable from seeding, the only thing that ever repaired one was
+ an OIDC login — so a password-only account in that state had no remedy
+ inside the product and needed direct SQL.
Joins the tenant bound to the caller's context, so an admin creating a
user joins them to their own tenant. There is no fallback: `tenant_id`
@@ -89,7 +99,7 @@ class UserStore:
select(TenantMember.tenant_id).where(TenantMember.user_id == user.id)
)
if existing.first() is not None:
- return
+ return False
session.add(
TenantMember(
tenant_id=require_tenant_id(),
@@ -98,6 +108,7 @@ class UserStore:
)
)
await session.flush()
+ return True
async def get(self, session: AsyncSession, user_id: str) -> User | None:
return await session.get(User, user_id)
@@ -278,9 +289,10 @@ class UserStore:
"Linking OIDC identity (iss=%r, sub=%r) to user %s", iss, sub, user.id
)
# Linking reaches accounts this store did not create, including any
- # that predate memberships. Nothing else repairs an account with none,
- # and the symptom would be that the person can never sign in again.
- await self._ensure_membership(session, user)
+ # that predate memberships — and an account with none can never sign
+ # in again. The startup admin seeding repairs the deployment's own
+ # admin; this repairs anyone else who signs in through an IdP.
+ await self.ensure_membership(session, user)
async def get_all(self, session: AsyncSession) -> list[User]:
result = await session.execute(select(User))
diff --git a/core/switch_core/gateway/connectors.py b/core/switch_core/gateway/connectors.py
index ba83ab098..94b186edf 100644
--- a/core/switch_core/gateway/connectors.py
+++ b/core/switch_core/gateway/connectors.py
@@ -142,5 +142,11 @@ async def delete_connector(
status_code=403, detail="Not authorized to delete this connector"
)
- await connector_lifecycle.remove(connector_id)
+ try:
+ await connector_lifecycle.remove(connector_id)
+ except ValueError as exc:
+ # Deleted between the read above and the removal. `remove` refuses to
+ # report a removal it did not perform, so the race surfaces as the
+ # same 404 the read would have given a moment earlier.
+ raise HTTPException(status_code=404, detail="Connector not found") from exc
return {"ok": True}
diff --git a/core/switch_core/logging_context.py b/core/switch_core/logging_context.py
index 5c5597024..958f30429 100644
--- a/core/switch_core/logging_context.py
+++ b/core/switch_core/logging_context.py
@@ -9,9 +9,22 @@ record and the formatters in :mod:`switch_core.logging_config` render it.
customer or everyone?" to be answerable from the logs alone, and that is far
cheaper to build before tenants exist than after. It was built before tenants
existed, and now that they do, ``gateway/auth.py`` and ``bridges/agent/auth.py``
-bind the real one for every authenticated request. Outside a request — a
-background task, a startup script, anything that never binds one — the filter
-falls back to the deployment's configured tenant; see ``SwitchConfig.tenant_id``.
+bind the real one for every authenticated request.
+
+Background work binds a tenant too, but through
+:mod:`switch_core.tenant_context` rather than through this module — and that
+binding is the one the database actually writes under, since
+``db/tenant_session.py`` turns it into ``set_config('app.tenant_id')`` on every
+transaction. Nothing taught the log about it, so a startup seed running inside
+``tenant_scope(TENANT_ZERO_ID)`` wrote its rows into ``00000000-…`` and logged
+``tenant_id=default``: a line naming a tenant that was not the one written.
+:class:`LogContextFilter` reads that binding as its second choice for exactly
+that reason.
+
+Only when neither is bound does the deployment's configured placeholder apply;
+see ``SwitchConfig.tenant_id``. Nothing scoped can be written in that state —
+``db/models.require_tenant_id`` raises — so the placeholder cannot contradict a
+row the way the two bindings could.
"""
from __future__ import annotations
@@ -22,6 +35,8 @@ from contextlib import contextmanager
from contextvars import ContextVar, Token
from dataclasses import dataclass, replace
+from switch_core.tenant_context import current_tenant_id
+
# The context fields, in the order they are rendered. Anything logged as
# context must be listed here, so that a typo in a bind call is an error rather
# than a field that silently never appears.
@@ -81,6 +96,18 @@ class LogContextFilter(logging.Filter):
``tenant_id`` always has a value; the rest are ``None`` outside a request
and are omitted by the formatters rather than rendered as empty.
+
+ Three sources, in this order, and the order is the whole point:
+
+ 1. what a request bound here, which is the caller's tenant;
+ 2. what :mod:`switch_core.tenant_context` has bound, which is the tenant
+ every statement on this task's sessions is being written under;
+ 3. the configured placeholder, which stands for "no tenant at all".
+
+ Without (2) a background line reports (3) while its transaction writes a
+ real tenant, and the two do not even use the same vocabulary — the
+ placeholder is a slug, ``app.tenant_id`` is a uuid — so the line does not
+ read as merely imprecise. It reads as a different tenant.
"""
def __init__(self, default_tenant_id: str) -> None:
@@ -89,7 +116,9 @@ class LogContextFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
context = _current.get()
- record.tenant_id = context.tenant_id or self._default_tenant_id
+ record.tenant_id = (
+ context.tenant_id or current_tenant_id() or self._default_tenant_id
+ )
record.request_id = context.request_id
record.agent_id = context.agent_id
record.user_id = context.user_id
diff --git a/core/switch_core/main.py b/core/switch_core/main.py
index 3bbd2459f..2e0a729d2 100644
--- a/core/switch_core/main.py
+++ b/core/switch_core/main.py
@@ -508,7 +508,25 @@ async def _seed_admin_user(
async with tenant_session(session_factory, TENANT_ZERO_ID) as session:
existing = await user_store.get_by_email(session, config.gateway_admin_email)
if existing is not None:
- logger.info("Admin user already exists: %s", config.gateway_admin_email)
+ # Existing, but not necessarily whole. An account with no
+ # `tenant_members` row cannot sign in at all — `gateway/auth.py`
+ # refuses to guess a tenant and answers 403 — and until this ran,
+ # the only thing in the product that ever wrote a missing
+ # membership was an OIDC login, so a password account in that
+ # state was locked out with no way back in short of SQL. The
+ # migration backfilled every account that existed when it ran;
+ # this covers the ones that did not, and any whose membership is
+ # lost later.
+ if await user_store.ensure_membership(session, existing):
+ await session.commit()
+ logger.warning(
+ "Admin user %s had no tenant membership and could not have "
+ "signed in; joined it to tenant %s.",
+ config.gateway_admin_email,
+ TENANT_ZERO_ID,
+ )
+ else:
+ logger.info("Admin user already exists: %s", config.gateway_admin_email)
return
admin = User(
diff --git a/core/switch_core/room_service.py b/core/switch_core/room_service.py
index 95d9b43b9..4ad055ceb 100644
--- a/core/switch_core/room_service.py
+++ b/core/switch_core/room_service.py
@@ -584,12 +584,12 @@ class RoomService:
)
agent_clients = self._resolve_agent_clients(agent_ids)
- system_clients = self._resolve_system_clients()
+ system_clients = self._resolve_system_clients(room.tenant_id)
all_clients = {**agent_clients, **system_clients}
await self._invite_clients(matrix_room_id, all_clients)
- async with self._session_factory() as session:
+ async with tenant_session(self._session_factory, room.tenant_id) as session:
for client_id in all_clients:
await self._room_store.add_client(session, client_id, room.id)
await session.commit()
@@ -1131,11 +1131,19 @@ class RoomService:
result[client.client_id] = client.matrix_user_id
return result
- def _resolve_system_clients(self) -> dict[str, str]:
- """Returns {client_id: matrix_user_id} for all system clients."""
+ def _resolve_system_clients(self, tenant_id: str) -> dict[str, str]:
+ """`{client_id: matrix_user_id}` for `tenant_id`'s system clients.
+
+ Per tenant, not per deployment. `clients` is scoped, so there is an
+ admin client per tenant rather than one for everyone, and the running
+ registry this reads holds all of them at once. Handing another
+ tenant's to a room is caught by `client_rooms`' composite foreign key
+ on `(tenant_id, client_id)` — as an exception, not as a wrong row —
+ so it is the room's own tenant that has to be asked for here.
+ """
result: dict[str, str] = {}
for client_type in SYSTEM_CLIENT_TYPES:
- for client in self._client_lifecycle.get_by_type(client_type):
+ for client in self._client_lifecycle.get_by_type(client_type, tenant_id):
result[client.client_id] = client.matrix_user_id
return result
@@ -1161,36 +1169,57 @@ class RoomService:
registry; a pending invite is accepted on the client's first sync.
Idempotent: `invite_to_room` is a no-op for an already-joined user, and
DB membership is only recorded where it is missing.
+
+ Every tenant's rooms in one pass, and one room's failure is contained
+ to that room. This runs inline in `main.run()` before the server is
+ listening, so an exception escaping here is not a failed
+ reconciliation — it is a deployment that does not start, for every
+ tenant, because of one room belonging to one of them. The failures are
+ counted and reported at `error`, so a contained one is still an
+ operator's problem rather than a silent one.
"""
- system_clients = self._resolve_system_clients()
- # Every tenant's rooms in one pass at startup, so this read is
- # unscoped by nature; each room's own tenant is bound only around
- # reconciling that one room, not the whole fan-out.
+ # Which rooms exist is the read that spans tenants; each room's own
+ # tenant is bound only around reconciling that one room.
async with unscoped_session(self._session_factory) as session:
rooms = await self._room_store.get_all(session, include_archived=True)
+
+ failures: list[str] = []
for room in rooms:
- with tenant_scope(room.tenant_id):
- async with self._session_factory() as session:
- existing = set(
- await self._room_store.get_client_ids(session, room.id)
- )
- expected = await self._room_store.get_member_agent_clients(
- session, room.id
- )
- expected.update(system_clients)
- missing = {
- cid: uid for cid, uid in expected.items() if cid not in existing
- }
- if not missing:
- continue
- await self._invite_clients(room.matrix_room_id, missing)
- async with self._session_factory() as session:
- for client_id in missing:
- await self._room_store.add_client(session, client_id, room.id)
- await session.commit()
- logger.info(
- "Reconciled %d client(s) into room %s", len(missing), room.id
+ try:
+ await self._reconcile_one_room(room)
+ except Exception:
+ logger.exception("Failed to reconcile clients into room %s", room.id)
+ failures.append(room.id)
+ if failures:
+ logger.error(
+ "Could not reconcile clients into %d of %d room(s): %s. Those "
+ "rooms are missing system or agent clients until this is fixed "
+ "and the service restarted.",
+ len(failures),
+ len(rooms),
+ ", ".join(failures),
+ )
+
+ async def _reconcile_one_room(self, room: Room) -> None:
+ with tenant_scope(room.tenant_id):
+ async with self._session_factory() as session:
+ existing = set(await self._room_store.get_client_ids(session, room.id))
+ expected = await self._room_store.get_member_agent_clients(
+ session, room.id
)
+ # Resolved per room rather than once for the fan-out: the system
+ # clients belong to a tenant, and which ones are this room's
+ # depends on the room.
+ expected.update(self._resolve_system_clients(room.tenant_id))
+ missing = {cid: uid for cid, uid in expected.items() if cid not in existing}
+ if not missing:
+ return
+ await self._invite_clients(room.matrix_room_id, missing)
+ async with self._session_factory() as session:
+ for client_id in missing:
+ await self._room_store.add_client(session, client_id, room.id)
+ await session.commit()
+ logger.info("Reconciled %d client(s) into room %s", len(missing), room.id)
async def ensure_client_in_room(self, room_id: str, client_id: str) -> None:
"""Invite a single running client to the room (it auto-joins) and record
diff --git a/core/tests/switch_core/bridges/agent/server_connectors/test_lifecycle_remove_tenant.py b/core/tests/switch_core/bridges/agent/server_connectors/test_lifecycle_remove_tenant.py
new file mode 100644
index 000000000..8d1fe1720
--- /dev/null
+++ b/core/tests/switch_core/bridges/agent/server_connectors/test_lifecycle_remove_tenant.py
@@ -0,0 +1,199 @@
+"""Removing a server-side connector is scoped, and says so (CHOO-2623).
+
+`remove()` was the one lifecycle verb the tenant rework did not reach. It
+opened a raw session — which inherits the requesting operator's tenant — and
+handed it to a store `delete` that returned quietly when it matched nothing.
+Under the policies those two make a specific, silent failure: a tenant-A
+operator deleting a tenant-B connector matched zero rows and the service
+logged "Removed server-side connector <id>".
+
+Two halves, both tested here against the role the policies actually apply to
+(`rls_harness.restricted` — the plain fixture connects as the tables' owner
+and would pass whether or not a single policy existed):
+
+- the delete refuses to report a removal it did not perform;
+- and it refuses *before* the teardown, because `_cores` is a process-wide
+ registry holding every tenant's connectors. Tearing down first meant a
+ caller from the wrong tenant stopped another tenant's connector and deleted
+ its agents, and only then failed to delete the row.
+"""
+
+from __future__ import annotations
+
+import logging
+
+import pytest
+from sqlalchemy import insert, select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.bridges.agent.server_connectors.lifecycle import (
+ ServerSideConnectorLifecycleService,
+)
+from switch_core.db.models import (
+ TENANT_ZERO_ID,
+ ApiKey,
+ ServerConnector,
+ Tenant,
+ User,
+)
+from switch_core.db.stores.api_key_store import ApiKeyStore
+from switch_core.db.stores.server_connector_store import ServerConnectorStore
+from switch_core.tenant_context import tenant_scope
+
+pytestmark = pytest.mark.no_ambient_tenant
+
+TENANT_B = "11111111-1111-1111-1111-111111111111"
+
+
+class _SpyCore:
+ """Stands in for a running `ConnectorCore` and records being torn down."""
+
+ def __init__(self) -> None:
+ self.deleted_agents = False
+ self.stopped = False
+
+ async def delete_agents(self) -> None:
+ self.deleted_agents = True
+
+ async def stop(self) -> None:
+ self.stopped = True
+
+
+async def _seed_connector(
+ owner_factory: async_sessionmaker[AsyncSession], tenant_id: str
+) -> str:
+ """A connector in `tenant_id`, written as the owner so the fixture itself
+ needs no tenant of its own."""
+ async with owner_factory() as session:
+ if tenant_id != TENANT_ZERO_ID:
+ session.add(Tenant(id=tenant_id, slug=tenant_id, name=tenant_id))
+ await session.flush()
+ user = User(name=tenant_id, email=f"{tenant_id}@x.test", role="user")
+ session.add(user)
+ await session.flush()
+ await session.execute(
+ insert(ApiKey.__table__).values(
+ id=f"key-{tenant_id}",
+ tenant_id=tenant_id,
+ user_id=user.id,
+ key_hash=f"hash-{tenant_id}",
+ encrypted_key="enc",
+ label="server-connector:test",
+ type="registration",
+ )
+ )
+ await session.execute(
+ insert(ServerConnector.__table__).values(
+ id=f"connector-{tenant_id}",
+ tenant_id=tenant_id,
+ type="test-type",
+ display_name="Test Connector",
+ connection_config={},
+ api_key_id=f"key-{tenant_id}",
+ status="active",
+ )
+ )
+ await session.commit()
+ return f"connector-{tenant_id}"
+
+
+def _service(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> ServerSideConnectorLifecycleService:
+ return ServerSideConnectorLifecycleService(
+ connector_store=ServerConnectorStore(),
+ api_key_store=ApiKeyStore(),
+ protocol=None, # type: ignore[arg-type]
+ session_factory=session_factory,
+ encryption_secret="s" * 32,
+ )
+
+
+async def _still_there(
+ owner_factory: async_sessionmaker[AsyncSession], connector_id: str
+) -> bool:
+ async with owner_factory() as session:
+ row = (
+ await session.execute(
+ select(ServerConnector.id).where(ServerConnector.id == connector_id)
+ )
+ ).first()
+ return row is not None
+
+
+class TestRemovingAnotherTenantsConnector:
+ async def test_it_raises_rather_than_reporting_a_removal(
+ self, rls_harness, caplog: pytest.LogCaptureFixture
+ ) -> None:
+ connector_id = await _seed_connector(rls_harness.owner, TENANT_B)
+ service = _service(rls_harness.restricted)
+
+ with caplog.at_level(logging.INFO, logger=service.__module__):
+ with tenant_scope(TENANT_ZERO_ID):
+ with pytest.raises(ValueError, match="Connector not found"):
+ await service.remove(connector_id)
+
+ assert await _still_there(rls_harness.owner, connector_id)
+ assert not any(
+ "Removed server-side connector" in r.getMessage() for r in caplog.records
+ ), "reported a removal that did not happen"
+
+ async def test_it_does_not_tear_the_connector_down_first(self, rls_harness) -> None:
+ """`_cores` spans tenants, so an authorization failure discovered
+ after the teardown is a cross-tenant outage with a 500 attached."""
+ connector_id = await _seed_connector(rls_harness.owner, TENANT_B)
+ service = _service(rls_harness.restricted)
+ core = _SpyCore()
+ service._cores[connector_id] = core # type: ignore[assignment]
+
+ with tenant_scope(TENANT_ZERO_ID):
+ with pytest.raises(ValueError):
+ await service.remove(connector_id)
+
+ assert not core.deleted_agents
+ assert not core.stopped
+ assert service._cores[connector_id] is core
+
+
+class TestRemovingYourOwn:
+ async def test_it_deletes_the_row_and_tears_the_core_down(
+ self, rls_harness
+ ) -> None:
+ connector_id = await _seed_connector(rls_harness.owner, TENANT_B)
+ service = _service(rls_harness.restricted)
+ core = _SpyCore()
+ service._cores[connector_id] = core # type: ignore[assignment]
+
+ with tenant_scope(TENANT_B):
+ await service.remove(connector_id)
+
+ assert not await _still_there(rls_harness.owner, connector_id)
+ assert core.deleted_agents
+ assert core.stopped
+ assert connector_id not in service._cores
+
+ async def test_removing_it_twice_is_an_error_not_a_shrug(self, rls_harness) -> None:
+ """The second call has nothing to delete, and saying so is the whole
+ point: "matched nothing" is also what a cross-tenant delete looks
+ like, so it cannot be treated as success anywhere."""
+ connector_id = await _seed_connector(rls_harness.owner, TENANT_B)
+ service = _service(rls_harness.restricted)
+
+ with tenant_scope(TENANT_B):
+ await service.remove(connector_id)
+ with pytest.raises(ValueError, match="Connector not found"):
+ await service.remove(connector_id)
+
+
+class TestTheStore:
+ async def test_delete_raises_on_a_row_this_session_cannot_see(
+ self, rls_harness
+ ) -> None:
+ connector_id = await _seed_connector(rls_harness.owner, TENANT_B)
+
+ with tenant_scope(TENANT_ZERO_ID):
+ async with rls_harness.restricted() as session:
+ with pytest.raises(ValueError, match="Connector not found"):
+ await ServerConnectorStore().delete(session, connector_id)
+
+ assert await _still_there(rls_harness.owner, connector_id)
diff --git a/core/tests/switch_core/clients/test_client_registry_tenants.py b/core/tests/switch_core/clients/test_client_registry_tenants.py
new file mode 100644
index 000000000..835e7a8aa
--- /dev/null
+++ b/core/tests/switch_core/clients/test_client_registry_tenants.py
@@ -0,0 +1,165 @@
+"""The running-client registry knows which tenant each client belongs to.
+
+`ClientLifecycleService` holds every tenant's clients in one process-wide
+dict, and `ensure_system_client` puts an admin client in it *per tenant*. So
+"the running admin client" is not a thing: there are as many as there are
+tenants, and picking one by type alone picks a stranger's as readily as your
+own.
+
+`room_service` did exactly that, and the result was not a subtly wrong row.
+`client_rooms` has a composite foreign key on `(tenant_id, client_id)`, so
+offering tenant A's admin client to tenant B's room is a
+`ForeignKeyViolationError` — raised from `reconcile_room_clients`, which runs
+inline at startup, so the second tenant to own a room stopped the deployment
+booting for everyone. The end-to-end proof is in
+`tests/switch_core/test_room_service_tenant_bindings.py`; this pins the
+registry side of it.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from types import SimpleNamespace
+from typing import Any
+from unittest.mock import MagicMock
+
+import pytest
+from sqlalchemy import insert
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.clients.client_lifecycle_service import ClientLifecycleService
+from switch_core.db.models import TENANT_ZERO_ID, Client, Tenant
+from switch_core.db.stores.client_store import ClientStore
+from switch_core.db.stores.tenant_store import TenantStore
+
+TENANT_B = "11111111-1111-1111-1111-111111111111"
+
+
+class _StubClient:
+ """A client that stays running until stopped, and touches nothing else.
+
+ It has to actually stay running: `_start_task` puts every client in a task
+ that removes it from all three registries if `start` returns or raises, so
+ a stub that finished immediately would empty the very dicts under test.
+ """
+
+ def __init__(self, record: Client) -> None:
+ self.client_id = record.id
+ self.matrix_user_id = record.matrix_user_id
+ self.display_name = record.display_name
+ self._stopped = asyncio.Event()
+
+ async def start(self) -> None:
+ await self._stopped.wait()
+
+ async def stop(self) -> None:
+ self._stopped.set()
+
+
+class _StubFactory:
+ def create(self, record: Client) -> Any:
+ return _StubClient(record)
+
+
+def _service(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> ClientLifecycleService:
+ return ClientLifecycleService(
+ matrix_admin=MagicMock(),
+ client_store=ClientStore(),
+ tenant_store=TenantStore(),
+ client_factory=_StubFactory(), # type: ignore[arg-type]
+ session_factory=session_factory,
+ config=SimpleNamespace(matrix_server_name="test"), # type: ignore[arg-type]
+ )
+
+
+def _record(tenant_id: str, client_id: str) -> Client:
+ return Client(
+ id=client_id,
+ tenant_id=tenant_id,
+ matrix_user_id="@switch-admin:test",
+ display_name="admin",
+ type="admin",
+ )
+
+
+class TestGetByType:
+ async def test_it_returns_only_the_named_tenants_clients(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ service = _service(session_factory)
+ service.start_client(_record(TENANT_ZERO_ID, "admin-zero"))
+ service.start_client(_record(TENANT_B, "admin-b"))
+ try:
+ assert [
+ c.client_id for c in service.get_by_type("admin", TENANT_ZERO_ID)
+ ] == ["admin-zero"]
+ assert [c.client_id for c in service.get_by_type("admin", TENANT_B)] == [
+ "admin-b"
+ ]
+ finally:
+ await service.stop_all()
+
+ async def test_a_tenant_with_no_client_of_that_type_gets_nothing(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """Not "somebody else's, since there is one running". A tenant with no
+ admin client has no admin client, and the caller has to cope with that
+ rather than be handed a client it cannot legally reference."""
+ service = _service(session_factory)
+ service.start_client(_record(TENANT_ZERO_ID, "admin-zero"))
+ try:
+ assert service.get_by_type("admin", TENANT_B) == []
+ finally:
+ await service.stop_all()
+
+ async def test_stopping_a_client_forgets_its_tenant_too(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """The three registries are keyed alike and must empty alike, or a
+ client id reused later inherits a stale tenant."""
+ service = _service(session_factory)
+ service.start_client(_record(TENANT_ZERO_ID, "admin-zero"))
+
+ await service.stop("admin-zero")
+
+ assert service.get_by_type("admin", TENANT_ZERO_ID) == []
+ assert "admin-zero" not in service._client_tenants
+
+
[email protected]_ambient_tenant
+async def test_start_all_records_every_tenants_clients_with_their_own_tenant(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> None:
+ """Boot reads every tenant's clients in one unscoped pass, so the tenant
+ has to come off each row rather than off the context — there isn't one."""
+ async with session_factory() as session:
+ session.add(Tenant(id=TENANT_B, slug="tenant-b", name="Tenant B"))
+ await session.flush()
+ for tenant_id, client_id in (
+ (TENANT_ZERO_ID, "admin-zero"),
+ (TENANT_B, "admin-b"),
+ ):
+ await session.execute(
+ insert(Client.__table__).values(
+ id=client_id,
+ tenant_id=tenant_id,
+ matrix_user_id="@switch-admin:test",
+ display_name="admin",
+ type="admin",
+ )
+ )
+ await session.commit()
+
+ service = _service(session_factory)
+ await service.start_all()
+ try:
+ assert [c.client_id for c in service.get_by_type("admin", TENANT_ZERO_ID)] == [
+ "admin-zero"
+ ]
+ assert [c.client_id for c in service.get_by_type("admin", TENANT_B)] == [
+ "admin-b"
+ ]
+ finally:
+ await service.stop_all()
diff --git a/core/tests/switch_core/test_logging_config.py b/core/tests/switch_core/test_logging_config.py
index ed0c676f8..0e2f89753 100644
--- a/core/tests/switch_core/test_logging_config.py
+++ b/core/tests/switch_core/test_logging_config.py
@@ -5,6 +5,7 @@ import logging
import pytest
from switch_core.config import SwitchConfig
+from switch_core.db.models import TENANT_ZERO_ID, TenantNotBoundError, require_tenant_id
from switch_core.logging_config import build_handler
from switch_core.logging_context import (
CONTEXT_FIELDS,
@@ -14,6 +15,7 @@ from switch_core.logging_context import (
log_context,
unbind_log_context,
)
+from switch_core.tenant_context import current_tenant_id, tenant_scope
BASE_ENV = {
"DB_HOST": "localhost",
@@ -253,3 +255,61 @@ def test_bound_tenant_beats_the_deployment_default() -> None:
LogContextFilter("default").filter(record)
assert record.tenant_id == "tenant-b"
+
+
+class TestTheLogAgreesWithTheDatabase:
+ """A line must not name a tenant other than the one being written.
+
+ Background work binds its tenant through `tenant_context`, not through
+ `log_context` — and that is the binding `db/tenant_session.py` turns into
+ `set_config('app.tenant_id')`, so it is where the rows go. The filter
+ ignored it, so the startup seeding wrote into tenant zero's uuid and
+ logged `tenant_id=default`. That is not a shorter spelling of the same
+ thing: it names a different tenant, and one that appears in no table.
+ """
+
+ def test_the_tenant_the_transaction_writes_is_the_one_logged(self) -> None:
+ record = logging.LogRecord(
+ "x", logging.INFO, __file__, 1, "m", args=(), exc_info=None
+ )
+
+ with tenant_scope(TENANT_ZERO_ID):
+ # The value a scoped column's default would stamp on a row written
+ # right here — read from the same place, so the two cannot drift.
+ written = require_tenant_id()
+ LogContextFilter("default").filter(record)
+
+ assert record.tenant_id == written
+
+ def test_a_request_binding_still_wins_over_it(self) -> None:
+ """`gateway/auth.py` binds both and they agree. If they ever did not,
+ the one the request resolved is the caller's own and is the more
+ specific answer."""
+ record = logging.LogRecord(
+ "x", logging.INFO, __file__, 1, "m", args=(), exc_info=None
+ )
+
+ with tenant_scope(TENANT_ZERO_ID), log_context(tenant_id="tenant-b"):
+ LogContextFilter("default").filter(record)
+
+ assert record.tenant_id == "tenant-b"
+
+ def test_with_nothing_bound_the_placeholder_stands(self) -> None:
+ """And it is safe for it to, because nothing scoped can be written
+ from here: `require_tenant_id` raises rather than choosing a tenant.
+ So the placeholder contradicts no row."""
+ record = logging.LogRecord(
+ "x", logging.INFO, __file__, 1, "m", args=(), exc_info=None
+ )
+
+ assert current_tenant_id() is None
+ LogContextFilter("default").filter(record)
+
+ assert record.tenant_id == "default"
+ with pytest.raises(TenantNotBoundError):
+ require_tenant_id()
+
+ def test_the_placeholder_is_not_a_real_tenant_id(self) -> None:
+ """Naming tenant zero here would file every unattributed line under a
+ real customer's id, which is worse than saying nothing at all."""
+ assert SwitchConfig.model_fields["tenant_id"].default != TENANT_ZERO_ID
diff --git a/core/tests/switch_core/test_room_service_pool_holds.py b/core/tests/switch_core/test_room_service_pool_holds.py
index d7715c7b0..bd6f60377 100644
--- a/core/tests/switch_core/test_room_service_pool_holds.py
+++ b/core/tests/switch_core/test_room_service_pool_holds.py
@@ -79,7 +79,7 @@ class _RunningClients:
def get(self, client_id: str) -> Any:
return self._by_id.get(client_id)
- def get_by_type(self, client_type: str) -> list[Any]:
+ def get_by_type(self, client_type: str, tenant_id: str) -> list[Any]:
return []
diff --git a/core/tests/switch_core/test_room_service_reconcile.py b/core/tests/switch_core/test_room_service_reconcile.py
index 3d3a75123..3e8b60fe1 100644
--- a/core/tests/switch_core/test_room_service_reconcile.py
+++ b/core/tests/switch_core/test_room_service_reconcile.py
@@ -47,13 +47,24 @@ class _FakeRoomStore:
class _FakeClientLifecycle:
- """Resolves system clients by type, mirroring the real lookup."""
+ """Resolves system clients by type *and tenant*, mirroring the real lookup.
+
+ The tenant is not decoration. The real registry is process-wide and holds
+ one system client per tenant, so a lookup by type alone hands another
+ tenant's client to this room — which `client_rooms`' composite foreign key
+ rejects outright. `test_room_service_tenant_bindings.py` is where that is
+ proven against Postgres; here the signature is simply kept honest.
+ """
def __init__(self, by_type: dict[str, list[Any]]) -> None:
self._by_type = by_type
- def get_by_type(self, client_type: str) -> list[Any]:
- return list(self._by_type.get(client_type, []))
+ def get_by_type(self, client_type: str, tenant_id: str) -> list[Any]:
+ return [
+ client
+ for client in self._by_type.get(client_type, [])
+ if client.tenant_id == tenant_id
+ ]
class _FakeMatrix:
@@ -64,9 +75,11 @@ class _FakeMatrix:
self.invited.append((matrix_room_id, matrix_user_id))
-def _admin() -> SimpleNamespace:
+def _admin(tenant_id: str = "tenant-1") -> SimpleNamespace:
return SimpleNamespace(
- client_id="admin-client", matrix_user_id="@switch-admin:switch.local"
+ client_id="admin-client",
+ matrix_user_id="@switch-admin:switch.local",
+ tenant_id=tenant_id,
)
@@ -105,6 +118,21 @@ class TestReconcileRoomClients:
assert matrix.invited == [("!mx:switch.local", "@switch-admin:switch.local")]
assert room_store.added == [("admin-client", "room-1")]
+ async def test_another_tenants_admin_client_is_not_offered(self) -> None:
+ room = SimpleNamespace(
+ id="room-1", tenant_id="tenant-1", matrix_room_id="!mx:switch.local"
+ )
+ svc, room_store, matrix = _build_service(
+ rooms=[room],
+ client_ids_by_room={"room-1": []},
+ by_type={"admin": [_admin(tenant_id="tenant-2")]},
+ )
+
+ await svc.reconcile_room_clients()
+
+ assert matrix.invited == []
+ assert room_store.added == []
+
async def test_skips_room_that_already_has_the_admin(self) -> None:
room = SimpleNamespace(
id="room-1", tenant_id="tenant-1", matrix_room_id="!mx:switch.local"
diff --git a/core/tests/switch_core/test_room_service_tenant_bindings.py b/core/tests/switch_core/test_room_service_tenant_bindings.py
new file mode 100644
index 000000000..3714c34c8
--- /dev/null
+++ b/core/tests/switch_core/test_room_service_tenant_bindings.py
@@ -0,0 +1,720 @@
+"""What `room_service.py`'s tenant bindings do, asked of Postgres (CHOO-2623).
+
+`room_service.py` is the largest single piece of the tenant rework and it had
+no test that could tell the rework from its absence. The proof was mechanical:
+shadow `tenant_scope`, `tenant_session` and `unscoped_session` with no-ops
+inside that one module and the whole suite still passed, 2833 to 2833. Not one
+assertion anywhere depended on a single binding this file makes.
+
+This module is built so that experiment fails. Two things make that possible
+where the rest of the suite could not:
+
+**It asks the database, not the service.** Every binding assertion here reads
+``current_setting('app.tenant_id')`` on the very session the service handed the
+store — the value ``db/tenant_session.py``'s ``after_begin`` hook wrote, which
+is what the policies compare and what every scoped write is filed under. An
+assertion about rows cannot see a missing binding while Switch runs as the
+tables' owner, which is what Phase 1 deploys: Postgres exempts an owner from
+its own policies, so a session that forgot its tenant reads and writes exactly
+as much as one that remembered (the restricted runtime role is CHOO-2685).
+The setting is the one thing that still differs.
+
+**It uses two tenants.** Half of what these bindings are for only has an
+observable effect once a second tenant owns something, which no other test in
+the suite arranges.
+
+`_TenantRecordingRoomStore` is a real `RoomStore` — the queries, the composite
+foreign keys and the policies are all the production ones. It notes what
+Postgres thought the tenant was, and delegates.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import inspect
+import uuid
+from types import SimpleNamespace
+from typing import Any
+from unittest.mock import MagicMock
+
+import pytest
+from sqlalchemy import insert, select, text
+from sqlalchemy.exc import StatementError
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.clients.client_lifecycle_service import ClientLifecycleService
+from switch_core.db.models import (
+ TENANT_ZERO_ID,
+ Agent,
+ ApiKey,
+ Client,
+ ClientRoom,
+ Room,
+ Tenant,
+ TenantNotBoundError,
+ User,
+ room_agents,
+)
+from switch_core.db.stores.client_store import ClientStore
+from switch_core.db.stores.room_store import RoomStore
+from switch_core.db.stores.tenant_store import TenantStore
+from switch_core.room_service import RoomService
+from switch_core.tenant_context import current_tenant_id, tenant_scope
+
+pytestmark = pytest.mark.no_ambient_tenant
+
+TENANT_B = "11111111-1111-1111-1111-111111111111"
+
+_APP_TENANT = text("SELECT current_setting('app.tenant_id', true)")
+
+
+class _TenantRecordingRoomStore(RoomStore):
+ """A real `RoomStore` that records the tenant Postgres saw for each call.
+
+ The setting is read on the session the service passed in, as its first
+ statement, so it reports what `after_begin` stamped on that transaction.
+ An unset `app.tenant_id` reads back as NULL on a fresh connection and as
+ the empty string on a pooled one whose previous transaction released an
+ `is_local` setting (see `db/rls_ddl.py`); both mean "nothing bound", so
+ both are normalised to `None`.
+ """
+
+ def __init__(self) -> None:
+ self.seen: list[tuple[str, str | None]] = []
+
+ async def _note(self, method: str, session: AsyncSession) -> None:
+ value = (await session.execute(_APP_TENANT)).scalar()
+ self.seen.append((method, value or None))
+
+ def tenants_for(self, method: str) -> list[str | None]:
+ return [tenant for name, tenant in self.seen if name == method]
+
+ async def get_all(
+ self, session: AsyncSession, *, include_archived: bool = False
+ ) -> list[Room]:
+ await self._note("get_all", session)
+ return await super().get_all(session, include_archived=include_archived)
+
+ async def get(self, session: AsyncSession, room_id: str) -> Room | None:
+ await self._note("get", session)
+ return await super().get(session, room_id)
+
+ async def get_client_ids(self, session: AsyncSession, room_id: str) -> list[str]:
+ await self._note("get_client_ids", session)
+ return await super().get_client_ids(session, room_id)
+
+ async def add_client(
+ self, session: AsyncSession, client_id: str, room_id: str
+ ) -> None:
+ await self._note("add_client", session)
+ await super().add_client(session, client_id, room_id)
+
+ async def add_agents(
+ self,
+ session: AsyncSession,
+ room_id: str,
+ agent_ids: list[str],
+ *,
+ join_event_listeners: set[str] | None = None,
+ ) -> None:
+ await self._note("add_agents", session)
+ await super().add_agents(
+ session, room_id, agent_ids, join_event_listeners=join_event_listeners
+ )
+
+ async def set_archived(
+ self, session: AsyncSession, room_id: str, archived: bool
+ ) -> None:
+ await self._note("set_archived", session)
+ await super().set_archived(session, room_id, archived)
+
+
+class _FakeMatrix:
+ """Provisioning, which room_service calls with no session open."""
+
+ def __init__(self, *, fail_for: set[str] | None = None) -> None:
+ self.invited: list[tuple[str, str]] = []
+ self._fail_for = fail_for or set()
+
+ async def invite_to_room(self, matrix_room_id: str, matrix_user_id: str) -> None:
+ if matrix_room_id in self._fail_for:
+ raise RuntimeError(f"platform refused the invite to {matrix_room_id}")
+ self.invited.append((matrix_room_id, matrix_user_id))
+
+
+class _RunningClient:
+ def __init__(self, client_id: str, matrix_user_id: str) -> None:
+ self.client_id = client_id
+ self.matrix_user_id = matrix_user_id
+
+
+class _AgentRegistry:
+ """Only the agent-client half of `ClientLifecycleService`.
+
+ The *system*-client half is the real service (`_registry` below), because
+ that lookup is where the cross-tenant bug lived and a stand-in would have
+ reimplemented the fix rather than exercised it. Resolving an agent to its
+ running client is not tenant-sensitive — an agent id is unique across the
+ deployment — and building real `AgentClient`s to say so would add a
+ transport and a sync loop to a test about bindings.
+ """
+
+ def __init__(self, agents: dict[str, _RunningClient]) -> None:
+ self._agents = agents
+
+ def get_by_type(self, client_type: str, tenant_id: str) -> list[_RunningClient]:
+ return []
+
+ def get_by_agent_id(self, agent_id: str) -> _RunningClient | None:
+ return self._agents.get(agent_id)
+
+ def get(self, client_id: str) -> _RunningClient | None:
+ return None
+
+
+class _StubClient:
+ """A client that stays running until stopped, and touches nothing else.
+
+ `_start_task` removes a client from the registry the moment `start`
+ returns, so a stub that finished immediately would empty the registry
+ these tests are about.
+ """
+
+ def __init__(self, record: Client) -> None:
+ self.client_id = record.id
+ self.matrix_user_id = record.matrix_user_id
+ self.display_name = record.display_name
+ self._stopped = asyncio.Event()
+
+ async def start(self) -> None:
+ await self._stopped.wait()
+
+ async def stop(self) -> None:
+ self._stopped.set()
+
+
+class _StubFactory:
+ def create(self, record: Client) -> Any:
+ return _StubClient(record)
+
+
+def _registry(
+ session_factory: async_sessionmaker[AsyncSession],
+ running: list[tuple[str, str, str]],
+) -> ClientLifecycleService:
+ """The real registry, holding `(tenant_id, client_id, matrix_user_id)`.
+
+ The real one on purpose: `reconcile_room_clients` asks it which system
+ clients belong to a room's tenant, and getting that answer from the whole
+ deployment instead of from one tenant is the defect these tests exist for.
+ """
+ service = ClientLifecycleService(
+ matrix_admin=MagicMock(),
+ client_store=ClientStore(),
+ tenant_store=TenantStore(),
+ client_factory=_StubFactory(), # type: ignore[arg-type]
+ session_factory=session_factory,
+ config=SimpleNamespace(matrix_server_name="switch.local"), # type: ignore[arg-type]
+ )
+ for tenant_id, client_id, matrix_user_id in running:
+ service.start_client(
+ Client(
+ id=client_id,
+ tenant_id=tenant_id,
+ matrix_user_id=matrix_user_id,
+ display_name="admin",
+ type="admin",
+ )
+ )
+ return service
+
+
+def _service(
+ session_factory: async_sessionmaker[AsyncSession],
+ *,
+ store: _TenantRecordingRoomStore,
+ clients: Any,
+ matrix: _FakeMatrix,
+) -> RoomService:
+ svc = object.__new__(RoomService)
+ svc._session_factory = session_factory # type: ignore[assignment]
+ svc._room_store = store # type: ignore[assignment]
+ svc._client_lifecycle = clients # type: ignore[assignment]
+ svc._matrix_admin = matrix # type: ignore[assignment]
+ svc._collab_lifecycle = None # type: ignore[assignment]
+ return svc
+
+
+async def _seed_tenant_b(session: AsyncSession) -> None:
+ session.add(Tenant(id=TENANT_B, slug="tenant-b", name="Tenant B"))
+ await session.flush()
+
+
+async def _seed_room(
+ session: AsyncSession, *, tenant_id: str, room_id: str, matrix_room_id: str
+) -> None:
+ """Insert a room directly, naming its tenant.
+
+ Core-level so no tenant needs binding to build the fixture: which tenants
+ a test's fixtures live in is the test's own decision, and going through
+ the ORM default here would make it the ambient context's.
+ """
+ await session.execute(
+ insert(Room.__table__).values(
+ id=room_id,
+ tenant_id=tenant_id,
+ name=room_id,
+ description="d",
+ matrix_room_id=matrix_room_id,
+ )
+ )
+
+
+async def _seed_client(
+ session: AsyncSession,
+ *,
+ tenant_id: str,
+ client_id: str,
+ matrix_user_id: str,
+ client_type: str = "admin",
+) -> None:
+ await session.execute(
+ insert(Client.__table__).values(
+ id=client_id,
+ tenant_id=tenant_id,
+ matrix_user_id=matrix_user_id,
+ display_name=client_type,
+ type=client_type,
+ )
+ )
+
+
+async def _seed_agent(
+ session: AsyncSession, *, tenant_id: str, agent_id: str, client_id: str
+) -> None:
+ """An agent and the user/key/client rows its foreign keys need."""
+ user_id = str(uuid.uuid4())
+ session.add(
+ User(id=user_id, name=agent_id, email=f"{agent_id}@x.test", role="user")
+ )
+ await session.flush()
+ key_id = str(uuid.uuid4())
+ await session.execute(
+ insert(ApiKey.__table__).values(
+ id=key_id,
+ tenant_id=tenant_id,
+ user_id=user_id,
+ key_hash=f"hash-{agent_id}",
+ encrypted_key="enc",
+ label=agent_id,
+ type="agent",
+ )
+ )
+ await _seed_client(
+ session,
+ tenant_id=tenant_id,
+ client_id=client_id,
+ matrix_user_id=f"@{agent_id}:switch.local",
+ client_type="agent",
+ )
+ await session.execute(
+ insert(Agent.__table__).values(
+ id=agent_id,
+ tenant_id=tenant_id,
+ name=agent_id,
+ description="d",
+ agent_type="external",
+ connector_type="test",
+ integration_profile={},
+ client_id=client_id,
+ api_key_id=key_id,
+ owner_id=user_id,
+ )
+ )
+
+
+async def _client_room_rows(
+ session_factory: async_sessionmaker[AsyncSession],
+) -> set[tuple[str, str, str]]:
+ async with session_factory() as session:
+ rows = (
+ await session.execute(
+ select(ClientRoom.tenant_id, ClientRoom.client_id, ClientRoom.room_id)
+ )
+ ).all()
+ return {(t, c, r) for t, c, r in rows}
+
+
+class TestReconcileRoomClients:
+ """The one room_service entry point genuinely reached with nothing bound.
+
+ `main.run()` calls it inline at startup, before uvicorn is listening, so
+ everything it does has to name its own tenant and anything it raises is a
+ deployment that does not start.
+ """
+
+ async def test_each_room_gets_its_own_tenants_admin_client(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """Two tenants, an admin client each, a room each.
+
+ Before this, the admin clients were resolved once for the whole
+ fan-out from a registry holding every tenant's, so tenant B's room was
+ offered tenant zero's admin client. `client_rooms` carries a composite
+ foreign key on `(tenant_id, client_id)`, so that is not a wrong row —
+ it is a `ForeignKeyViolationError` out of startup.
+ """
+ async with session_factory() as session:
+ await _seed_tenant_b(session)
+ await _seed_room(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ room_id="room-zero",
+ matrix_room_id="!zero:switch.local",
+ )
+ await _seed_room(
+ session,
+ tenant_id=TENANT_B,
+ room_id="room-b",
+ matrix_room_id="!b:switch.local",
+ )
+ await _seed_client(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ client_id="admin-zero",
+ matrix_user_id="@switch-admin:switch.local",
+ )
+ await _seed_client(
+ session,
+ tenant_id=TENANT_B,
+ client_id="admin-b",
+ matrix_user_id="@switch-admin:switch.local",
+ )
+ await session.commit()
+
+ store = _TenantRecordingRoomStore()
+ registry = _registry(
+ session_factory,
+ [
+ (TENANT_ZERO_ID, "admin-zero", "@switch-admin:switch.local"),
+ (TENANT_B, "admin-b", "@switch-admin:switch.local"),
+ ],
+ )
+ svc = _service(
+ session_factory,
+ store=store,
+ clients=registry,
+ matrix=_FakeMatrix(),
+ )
+
+ try:
+ await svc.reconcile_room_clients()
+ finally:
+ await registry.stop_all()
+
+ assert await _client_room_rows(session_factory) == {
+ (TENANT_ZERO_ID, "admin-zero", "room-zero"),
+ (TENANT_B, "admin-b", "room-b"),
+ }
+ # Each room's work ran under that room's tenant, not under whichever
+ # room happened to come first.
+ assert sorted(store.tenants_for("add_client")) == sorted(
+ [TENANT_ZERO_ID, TENANT_B]
+ )
+ assert sorted(store.tenants_for("get_client_ids")) == sorted(
+ [TENANT_ZERO_ID, TENANT_B]
+ )
+
+ async def test_the_room_enumeration_refuses_the_callers_tenant(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """ "Which rooms exist" must span tenants whether or not one is bound.
+
+ That is `unscoped_session`'s whole contract (`db/session_scope.py`): a
+ helper that named the intent while inheriting the ambient tenant would
+ have the hook stamp it on the transaction and the cross-tenant read
+ would silently be a single-tenant one — here, a boot that reconciles
+ tenant zero's rooms and quietly skips everyone else's.
+ """
+ async with session_factory() as session:
+ await _seed_tenant_b(session)
+ await _seed_room(
+ session,
+ tenant_id=TENANT_B,
+ room_id="room-b",
+ matrix_room_id="!b:switch.local",
+ )
+ await _seed_client(
+ session,
+ tenant_id=TENANT_B,
+ client_id="admin-b",
+ matrix_user_id="@switch-admin:switch.local",
+ )
+ await session.commit()
+
+ store = _TenantRecordingRoomStore()
+ registry = _registry(
+ session_factory, [(TENANT_B, "admin-b", "@switch-admin:switch.local")]
+ )
+ svc = _service(
+ session_factory,
+ store=store,
+ clients=registry,
+ matrix=_FakeMatrix(),
+ )
+
+ try:
+ with tenant_scope(TENANT_ZERO_ID):
+ await svc.reconcile_room_clients()
+ # And it hands the caller's context back untouched.
+ assert current_tenant_id() == TENANT_ZERO_ID
+ finally:
+ await registry.stop_all()
+
+ assert store.tenants_for("get_all") == [None], (
+ "the room enumeration inherited the caller's tenant; under the "
+ "policies it would have returned none of tenant B's rooms"
+ )
+ assert await _client_room_rows(session_factory) == {
+ (TENANT_B, "admin-b", "room-b")
+ }
+
+ async def test_one_rooms_failure_does_not_take_the_others_down(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """This runs before the server is listening. An exception escaping it
+ is not a skipped room, it is a deployment that does not start — for
+ every tenant, because of one room belonging to one of them."""
+ async with session_factory() as session:
+ await _seed_tenant_b(session)
+ await _seed_room(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ room_id="room-broken",
+ matrix_room_id="!broken:switch.local",
+ )
+ await _seed_room(
+ session,
+ tenant_id=TENANT_B,
+ room_id="room-b",
+ matrix_room_id="!b:switch.local",
+ )
+ await _seed_client(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ client_id="admin-zero",
+ matrix_user_id="@switch-admin:switch.local",
+ )
+ await _seed_client(
+ session,
+ tenant_id=TENANT_B,
+ client_id="admin-b",
+ matrix_user_id="@switch-admin:switch.local",
+ )
+ await session.commit()
+
+ registry = _registry(
+ session_factory,
+ [
+ (TENANT_ZERO_ID, "admin-zero", "@switch-admin:switch.local"),
+ (TENANT_B, "admin-b", "@switch-admin:switch.local"),
+ ],
+ )
+ svc = _service(
+ session_factory,
+ store=_TenantRecordingRoomStore(),
+ clients=registry,
+ matrix=_FakeMatrix(fail_for={"!broken:switch.local"}),
+ )
+
+ try:
+ await svc.reconcile_room_clients()
+ finally:
+ await registry.stop_all()
+
+ assert await _client_room_rows(session_factory) == {
+ (TENANT_B, "admin-b", "room-b")
+ }
+
+ async def test_the_failure_is_reported_not_swallowed(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ caplog: pytest.LogCaptureFixture,
+ ) -> None:
+ """Contained, but not quiet: a room left unreconciled is an operator's
+ problem, so it is logged at `error` naming the room."""
+ async with session_factory() as session:
+ await _seed_room(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ room_id="room-broken",
+ matrix_room_id="!broken:switch.local",
+ )
+ await _seed_client(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ client_id="admin-zero",
+ matrix_user_id="@switch-admin:switch.local",
+ )
+ await session.commit()
+
+ registry = _registry(
+ session_factory,
+ [(TENANT_ZERO_ID, "admin-zero", "@switch-admin:switch.local")],
+ )
+ svc = _service(
+ session_factory,
+ store=_TenantRecordingRoomStore(),
+ clients=registry,
+ matrix=_FakeMatrix(fail_for={"!broken:switch.local"}),
+ )
+
+ try:
+ with caplog.at_level("ERROR", logger="switch_core.room_service"):
+ await svc.reconcile_room_clients()
+ finally:
+ await registry.stop_all()
+
+ errors = [r.getMessage() for r in caplog.records if r.levelname == "ERROR"]
+ assert any("room-broken" in message for message in errors), errors
+
+
+class TestBindingsOnTheRoomsOwnTenant:
+ """The rest of the file's shape: read the room, then act as *its* tenant.
+
+ Exercised with nothing bound, which is what makes the binding observable
+ at all — with the caller's tenant already equal to the room's, a helper
+ that bound nothing and one that bound the right thing are the same run.
+ """
+
+ async def test_adding_an_agent_files_the_membership_under_the_rooms_tenant(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """`room_agents.tenant_id` has no server default — its Python-side
+ default is `require_tenant_id`, which raises rather than guessing. So
+ this write does not merely land in the wrong tenant without the
+ binding; it does not land at all."""
+ async with session_factory() as session:
+ await _seed_tenant_b(session)
+ await _seed_room(
+ session,
+ tenant_id=TENANT_B,
+ room_id="room-b",
+ matrix_room_id="!b:switch.local",
+ )
+ with tenant_scope(TENANT_B):
+ await _seed_agent(
+ session,
+ tenant_id=TENANT_B,
+ agent_id="agent-b",
+ client_id="client-b",
+ )
+ await session.commit()
+
+ store = _TenantRecordingRoomStore()
+ svc = _service(
+ session_factory,
+ store=store,
+ clients=_AgentRegistry(
+ {"agent-b": _RunningClient("client-b", "@agent-b:switch.local")}
+ ),
+ matrix=_FakeMatrix(),
+ )
+
+ assert current_tenant_id() is None
+ await svc.add_agents_to_room("room-b", agent_ids=["agent-b"])
+
+ assert store.tenants_for("add_agents") == [TENANT_B]
+ assert store.tenants_for("add_client") == [TENANT_B]
+ async with session_factory() as session:
+ rows = (
+ await session.execute(
+ select(room_agents.c.tenant_id, room_agents.c.agent_id)
+ )
+ ).all()
+ assert [(t, a) for t, a in rows] == [(TENANT_B, "agent-b")]
+ assert await _client_room_rows(session_factory) == {
+ (TENANT_B, "client-b", "room-b")
+ }
+
+ async def test_an_update_runs_under_the_rooms_tenant_and_lets_go_after(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """`tenant_session` binds for the block and unbinds in a `finally`.
+
+ The unbinding half matters as much as the binding: room_service is
+ reached from long-lived callers, and a tenant that outlived its block
+ would decide what the next unit of work reads.
+ """
+ async with session_factory() as session:
+ await _seed_tenant_b(session)
+ await _seed_room(
+ session,
+ tenant_id=TENANT_B,
+ room_id="room-b",
+ matrix_room_id="!b:switch.local",
+ )
+ await session.commit()
+
+ store = _TenantRecordingRoomStore()
+ svc = _service(
+ session_factory,
+ store=store,
+ clients=_AgentRegistry({}),
+ matrix=_FakeMatrix(),
+ )
+
+ await svc.set_room_archived("room-b", True)
+
+ assert store.tenants_for("set_archived") == [TENANT_B]
+ assert current_tenant_id() is None, (
+ "the room's tenant outlived the block that bound it"
+ )
+ async with session_factory() as session:
+ archived_at = (
+ await session.execute(
+ select(Room.archived_at).where(Room.id == "room-b")
+ )
+ ).scalar()
+ assert archived_at is not None
+
+ async def test_a_scoped_write_with_nothing_bound_raises_rather_than_guessing(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """The floor everything above rests on, stated once.
+
+ `require_tenant_id` used to answer tenant zero when nothing was bound.
+ It raises now, which is what turns "this call site forgot to bind"
+ from a row quietly filed under a real customer into a failure — and
+ what makes the tests above able to tell the difference at all.
+ """
+ async with session_factory() as session:
+ await _seed_room(
+ session,
+ tenant_id=TENANT_ZERO_ID,
+ room_id="room-zero",
+ matrix_room_id="!zero:switch.local",
+ )
+ await session.commit()
+
+ async with session_factory() as session:
+ # SQLAlchemy wraps a column default that raises, so the cause is
+ # where the rule is stated rather than the type pytest sees.
+ with pytest.raises(StatementError) as raised:
+ await RoomStore().add_agents(session, "room-zero", ["agent-x"])
+ assert isinstance(raised.value.orig, TenantNotBoundError)
+
+
+class TestTheStandInMatchesTheRealSignature:
+ """`_AgentRegistry` covers the half of `ClientLifecycleService` these
+ tests do not use for real.
+
+ A stand-in that drifts from the thing it stands for is how a suite keeps
+ passing against an interface nobody implements any more, so the shapes are
+ compared rather than assumed.
+ """
+
+ def test_get_by_type_takes_a_tenant(self) -> None:
+ real = inspect.signature(ClientLifecycleService.get_by_type)
+ stand_in = inspect.signature(_AgentRegistry.get_by_type)
+ assert list(real.parameters) == list(stand_in.parameters)
diff --git a/core/tests/switch_core/test_seed_admin_membership.py b/core/tests/switch_core/test_seed_admin_membership.py
new file mode 100644
index 000000000..f62c9c378
--- /dev/null
+++ b/core/tests/switch_core/test_seed_admin_membership.py
@@ -0,0 +1,213 @@
+"""An account with no tenant membership can get back in (CHOO-2623).
+
+`gateway/auth.py` refuses to guess a tenant for a user who has none and
+answers 403. That is the right call — picking one would place a person in
+somebody else's data — but it means a membership-less account is not degraded,
+it is locked out, on every route, permanently.
+
+Nothing in the product repaired one except an OIDC login (`UserStore.
+_link_identity`). A password account had no way back in at all: no endpoint
+writes a membership, and every endpoint that could is behind the 403. The
+remedy was `INSERT INTO tenant_members` by hand on the box.
+
+The state is reachable. The migration backfilled every account that existed
+when it ran, and `UserStore.create` has written one ever since — but only
+since the commit that added it, and a deployment tracking this stack ran the
+migration several commits earlier. Every account created in that window has
+none. So does one whose membership is later removed.
+
+Startup seeding is the repair: it already looks the configured admin up by
+email on every boot, it is the one path that runs before anyone has to be
+able to sign in, and a restart is something an operator can do without a
+database client.
+"""
+
+from __future__ import annotations
+
+import logging
+
+import pytest
+from sqlalchemy import delete, select
+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
+
+from switch_core.config import SwitchConfig
+from switch_core.db.models import TENANT_ZERO_ID, TenantMember, User
+from switch_core.db.stores.tenant_member_store import (
+ TenantMembershipError,
+ TenantMemberStore,
+)
+from switch_core.db.stores.user_store import UserStore
+from switch_core.gateway.auth import hash_password
+from switch_core.main import _seed_admin_user
+from switch_core.tenant_context import tenant_scope
+
+pytestmark = pytest.mark.no_ambient_tenant
+
+ADMIN_EMAIL = "[email protected]"
+
+
+def _config() -> SwitchConfig:
+ return SwitchConfig(
+ db_host="unused",
+ db_port="5432",
+ db_user="unused",
+ db_password="unused",
+ db_name="unused",
+ matrix_server_name="test",
+ agent_registration_token="unused",
+ jwt_secret_key="test-jwt-secret",
+ gateway_admin_email=ADMIN_EMAIL,
+ gateway_admin_password="hunter2",
+ )
+
+
+async def _memberships(
+ session_factory: async_sessionmaker[AsyncSession], user_id: str
+) -> list[tuple[str, str]]:
+ async with session_factory() as session:
+ rows = (
+ await session.execute(
+ select(TenantMember.tenant_id, TenantMember.role).where(
+ TenantMember.user_id == user_id
+ )
+ )
+ ).all()
+ return [(t, r) for t, r in rows]
+
+
+async def _admin_id(session_factory: async_sessionmaker[AsyncSession]) -> str:
+ async with session_factory() as session:
+ return (
+ await session.execute(select(User.id).where(User.email == ADMIN_EMAIL))
+ ).scalar_one()
+
+
+async def _strand_the_admin(
+ session_factory: async_sessionmaker[AsyncSession], user_id: str
+) -> None:
+ """Reproduce the state a mid-stack deployment leaves behind: the account
+ exists, with a password, and has no membership."""
+ async with session_factory() as session:
+ await session.execute(
+ delete(TenantMember).where(TenantMember.user_id == user_id)
+ )
+ await session.commit()
+
+
+class TestSeedingRepairsTheAdmin:
+ async def test_a_fresh_database_gets_an_admin_with_a_membership(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ await _seed_admin_user(session_factory, UserStore(), _config())
+
+ user_id = await _admin_id(session_factory)
+ assert await _memberships(session_factory, user_id) == [
+ (TENANT_ZERO_ID, "owner")
+ ]
+
+ async def test_a_stranded_admin_is_rejoined_on_the_next_boot(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """The whole point. Before this, the existing-admin branch returned
+ without looking, so no number of restarts made any difference."""
+ await _seed_admin_user(session_factory, UserStore(), _config())
+ user_id = await _admin_id(session_factory)
+ await _strand_the_admin(session_factory, user_id)
+ assert await _memberships(session_factory, user_id) == []
+
+ await _seed_admin_user(session_factory, UserStore(), _config())
+
+ assert await _memberships(session_factory, user_id) == [
+ (TENANT_ZERO_ID, "owner")
+ ]
+
+ async def test_the_repair_is_reported_loudly(
+ self,
+ session_factory: async_sessionmaker[AsyncSession],
+ caplog: pytest.LogCaptureFixture,
+ ) -> None:
+ """An account that could not sign in until this boot is not routine
+ news, so it is a warning naming the address — and the ordinary case
+ stays at `info` so the warning means something."""
+ await _seed_admin_user(session_factory, UserStore(), _config())
+ user_id = await _admin_id(session_factory)
+ await _strand_the_admin(session_factory, user_id)
+
+ with caplog.at_level(logging.INFO, logger="switch_core.main"):
+ await _seed_admin_user(session_factory, UserStore(), _config())
+ repaired = [r for r in caplog.records if r.levelname == "WARNING"]
+ caplog.clear()
+ await _seed_admin_user(session_factory, UserStore(), _config())
+ settled = [r for r in caplog.records if r.levelname == "WARNING"]
+
+ assert [ADMIN_EMAIL in r.getMessage() for r in repaired] == [True]
+ assert settled == []
+
+ async def test_an_admin_that_already_belongs_somewhere_is_left_alone(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """Two memberships fail resolution exactly as hard as none do, so a
+ repair that added one unconditionally would break the accounts it was
+ meant to fix."""
+ await _seed_admin_user(session_factory, UserStore(), _config())
+ user_id = await _admin_id(session_factory)
+
+ await _seed_admin_user(session_factory, UserStore(), _config())
+
+ assert await _memberships(session_factory, user_id) == [
+ (TENANT_ZERO_ID, "owner")
+ ]
+
+
+class TestWhatBeingStrandedCosts:
+ """Why the repair is worth having, stated as the failure it prevents."""
+
+ async def test_no_membership_means_no_tenant_and_so_no_access(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ async with session_factory() as session:
+ with tenant_scope(TENANT_ZERO_ID):
+ user = User(
+ name="Nobody",
+ role="user",
+ password_hash=hash_password("pw"),
+ )
+ session.add(user)
+ await session.flush()
+ user_id = user.id
+ await session.commit()
+ await _strand_the_admin(session_factory, user_id)
+
+ async with session_factory() as session:
+ with pytest.raises(TenantMembershipError):
+ await TenantMemberStore().get_sole_tenant_id(session, user_id)
+
+ async def test_creating_a_user_still_writes_one(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """The forward fix, kept honest alongside the repair: every path that
+ makes a user leaves exactly one membership, so the repair is for
+ history rather than for today's writes."""
+ async with session_factory() as session:
+ with tenant_scope(TENANT_ZERO_ID):
+ user = User(name="New", email="[email protected]", role="user")
+ await UserStore().create(session, user)
+ user_id = user.id
+ await session.commit()
+
+ assert await _memberships(session_factory, user_id) == [
+ (TENANT_ZERO_ID, "member")
+ ]
+
+
+def test_the_membership_store_offers_no_second_way_to_write_one() -> None:
+ """`TenantMemberStore.create` existed and nothing called it.
+
+ Removed rather than wired up: "exactly one membership per account" holds
+ because a single idempotent function writes them all, and an unguarded
+ `create(tenant_id=…, user_id=…, role=…)` beside it is an invitation to
+ write a second — which `get_sole_tenant_id` rejects just as firmly as it
+ rejects none.
+ """
+ assert not hasattr(TenantMemberStore, "create")
diff --git a/docs/old/multi-tenancy-phase1-db.md b/docs/old/multi-tenancy-phase1-db.md
index 03cb62727..b409eddc9 100644
--- a/docs/old/multi-tenancy-phase1-db.md
+++ b/docs/old/multi-tenancy-phase1-db.md
@@ -119,6 +119,15 @@ Three entries are worth justifying:
anywhere. The distinction has teeth: "one exists" is true for a deployment
whose second tenant has no admin client in any of its rooms.
+ One row per tenant means the *running* registry holds one per tenant too,
+ so `ClientLifecycleService.get_by_type` takes a tenant and there is no way
+ left to ask it for "the admin client". Every caller has a room in hand and
+ so has a tenant to ask with. Not tidiness: `client_rooms` carries a
+ composite foreign key on `(tenant_id, client_id)`, so offering one tenant's
+ admin client to another's room raises `ForeignKeyViolationError` — and the
+ caller that did it, `reconcile_room_clients`, runs inline at startup, where
+ an exception is a deployment that does not boot, for anyone.
+
**Global — no `tenant_id`, no policy:** `users`, `oidc_identities`,
`feature_flags`, `alembic_version`.
@@ -558,9 +567,54 @@ it:
Applying that marker suite-wide turns 354 tests red, which is the expected
shape rather than a finding.
-The same context value feeds the logging filter, which has had a `tenant_id`
-field since before tenants existed and which nothing has ever bound
-per-request. That closes half of Phase 0's logging item as a side effect.
+### An account with no membership, and how it gets one back
+
+Resolution refuses to guess (above), so an account with no `tenant_members`
+row cannot sign in on any route — 403, permanently, not degraded. The
+migration gave one to every account that existed when it ran, and
+`UserStore.create` has written one on every path that makes a user since. But
+a deployment tracking this work applied the migration several commits before
+`create` learned to, so anything created in that window has none; so does an
+account whose membership is later removed.
+
+Nothing repaired that except an OIDC login (`UserStore._link_identity`, now
+`ensure_membership`). A password account had no remedy inside the product at
+all — no endpoint writes a membership, and every endpoint that could is behind
+the 403 — which left `INSERT INTO tenant_members` on the box.
+
+Startup seeding is the repair: `_seed_admin_user` already looks the configured
+admin up by email on every boot, so it calls `ensure_membership` on the
+existing-admin branch instead of returning. Idempotent, and it logs at
+`warning` only when it actually wrote one, because an account that could not
+sign in until this boot is not routine news. That fixes the deployment's own
+admin, which is enough to get an operator back in; anyone else they can then
+repair from the UI.
+
+`TenantMemberStore` grew no `create` to go with it — it had one, unused, and
+it is gone. "Exactly one membership per account" holds because a single
+idempotent function writes them all, and a second unguarded way in, taking
+`tenant_id`, `user_id` and `role` from whatever the caller felt like, is how
+an account ends up with two. Resolution rejects two as firmly as none.
+
+### The log has to name the tenant the transaction writes
+
+The logging filter has carried a `tenant_id` field since before tenants
+existed. It read the *log* context, which `gateway/auth.py` and
+`bridges/agent/auth.py` bind per request — and nothing else. Background work
+binds its tenant through `tenant_context`, which is the binding
+`db/tenant_session.py` turns into `set_config('app.tenant_id')`, so the
+startup seeding wrote rows into `00000000-…` while logging
+`tenant_id=default`. Not a longer spelling of the same thing: a different
+tenant, and one that appears in no table.
+
+`LogContextFilter` now reads three sources in order — the request's binding,
+then `current_tenant_id()`, then the configured placeholder. `TENANT_ID` stays
+what it was and is documented as a placeholder rather than a tenant id: it
+only applies where neither binding exists, which is also where nothing scoped
+can be written, so it contradicts no row. Setting it to a real tenant's id
+would be worse than leaving it — every unattributed line in the deployment
+would then be indistinguishable from that tenant's own under a `tenant_id`
+filter.
## The bootstrap problem, and the two exceptions
@@ -789,6 +843,42 @@ to prevent — in that file unless another is named:
database and on a deployment that gained a tenant between two boots
(`tests/switch_core/clients/test_ensure_system_client.py`). It had no test
at all until it stopped booting.
+- **`room_service.py`'s bindings, asked of Postgres**
+ (`tests/switch_core/test_room_service_tenant_bindings.py`). Worth its own
+ entry because of how it came about: `tenant_scope`, `tenant_session` and
+ `unscoped_session` could be shadowed with no-ops inside that one module and
+ the suite still passed, 2833 to 2833 — the largest file in the rework, and
+ nothing depended on a single binding it made. Two things had to change for
+ a test to be able to tell. It reads
+ `current_setting('app.tenant_id')` on the session the service handed the
+ store, rather than asserting on rows: while Switch connects as the tables'
+ owner no policy bites, so a session that forgot its tenant writes exactly
+ as much as one that remembered, and the setting is the only remaining
+ difference. And it arranges two tenants, which nothing else in the suite
+ does, because half of what these bindings are for is only observable once a
+ second tenant owns something. Shadowing the three helpers now fails five of
+ its eight tests; that experiment is the file's acceptance criterion and is
+ written down in its module docstring.
+- **`get_by_type` is per tenant, and boot records the tenant off each row**
+ (`tests/switch_core/clients/test_client_registry_tenants.py`). Reverting the
+ filter reproduces `ForeignKeyViolationError` on `fk_client_rooms_client`
+ through the room-service tests above, which is the shape the bug took at
+ startup.
+- **The startup admin seeding rejoins a stranded admin**
+ (`tests/switch_core/test_seed_admin_membership.py`), and the same file pins
+ what being stranded costs — `get_sole_tenant_id` raising — so the repair is
+ measured against the failure it prevents rather than against itself.
+- **The log line names the tenant the transaction writes**
+ (`tests/switch_core/test_logging_config.py`), reading the value from
+ `require_tenant_id()` rather than restating a constant, so the two cannot
+ drift apart again.
+- **Removing a server-side connector is scoped and fails loudly**
+ (`tests/switch_core/bridges/agent/server_connectors/test_lifecycle_remove_tenant.py`),
+ against `rls_harness.restricted` — the plain fixture is the owner and would
+ pass with no policies at all. It covers both halves: the delete refuses to
+ report a removal it did not make, and it refuses *before* tearing the
+ connector down, since `_cores` spans tenants and the teardown would
+ otherwise be a cross-tenant outage discovered one statement too late.
- A reference type whose slug clashes *while another tenant holds the same
slug* still reports "already exists" rather than a 500
(`tests/switch_core/db/stores/test_reference_type_store.py`).
@@ -829,18 +919,6 @@ Named so they are decisions rather than omissions.
exists in another tenant fails on the index rather than being handled — an
existence oracle and a bad error. Phase 2.
- **System sessions can read across tenants** by design, as above.
-- **`RoomService._resolve_system_clients` is tenant-blind**, and now that
- `ensure_system_client` makes one admin client per tenant there is more than
- one for it to be blind about. It returns every running system client, and
- both callers — creating a room, and `reconcile_room_clients` at startup —
- put all of them in the room. With two tenants that means offering tenant
- B's admin client to tenant A's room, which `client_rooms`' composite
- foreign key refuses, so the failure is loud rather than a leak. It was
- equally broken before, in the other direction: one admin client in tenant
- zero, offered to every tenant's rooms, refused the same way. The fix is a
- filter on the room's tenant at both call sites, and it belongs with
- whatever onboards the second tenant rather than in the change that made
- the row exist.
- **No indexes on `tenant_id`.** With one tenant the column has no selectivity
and an index is pure write cost. Add them when the second tenant lands, in
one migration, `concurrently`, with data to measure against.