Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
docs(multi-tenancy): design the Phase 1 database schema (CHOO-2623)
Wojciech Wojtyniakb59a7f08577a
diff --git a/docs/old/multi-tenancy-phase1-db.md b/docs/old/multi-tenancy-phase1-db.mdnew file mode 100644index 000000000..3757c592e--- /dev/null+++ b/docs/old/multi-tenancy-phase1-db.md@@ -0,0 +1,457 @@+# Multi-tenancy Phase 1: the database design++Status: design, reviewed once. Implements §2 of `multi-tenancy.md` (spike+CHOO-2603) for CHOO-2623. Not yet built.++This document is the schema half of Phase 1: what tables change, what the+policies look like, how a session learns which tenant it is acting for, and+what the one migration does.++Postgres 16 everywhere — local Compose, the chart, and the test containers —+and two things below need at least 15, so that is a floor, not an incidental+detail.++## The rule++Everything follows from one sentence:++> **Every table holding customer data carries a non-null `tenant_id`, carries+> the same isolation policy, and every foreign key between two such tables+> carries `tenant_id` as well.**++Uniformity is the point. A reviewer should be able to open any table and know+without thinking whether it is scoped and what its policy says, because there+is only one kind of scoped table and one policy text.++The rule buys two separate guarantees, and they are worth separating:++- **A query that forgets its filter returns nothing** rather than another+ tenant's rows. That is row-level security.+- **A row cannot reference a parent in another tenant.** That is the composite+ foreign key, and row-level security does not give it — referential integrity+ checks in Postgres deliberately bypass policies, so without it a bug can+ write a message into tenant A pointing at tenant B's room. The row is+ unreadable from either side, but it exists, and cleaning that up later is+ worse than preventing it now.++## New tables++```+tenants+ id text primary key+ slug text not null unique+ name text not null+ created_at timestamptz not null default now()++tenant_members+ tenant_id text not null references tenants(id)+ user_id text not null references users(id)+ role text not null check (role in ('owner','admin','member'))+ created_at timestamptz not null default now()+ primary key (tenant_id, user_id)+```++Membership is a row, not a column on the user: a person has one login and may+belong to several tenants. That decision is argued in the spike.++`tenants` deliberately has no `plan`, `status` or `deleted_at`. Phase 3 adds+plans, Phase 5 adds deletion, and a `deleted_at` that nothing honours is worse+than no column — it reads as a guarantee the code does not make.++`role` is a checked string rather than an enum type, matching how `users.role`+is already stored. Phase 2 owns making these roles mean something; Phase 1 only+records them so the migration loses no information.++## Which tables are scoped++**Scoped — 36 tables, each gains `tenant_id text not null references+tenants(id)`:**++`api_keys`, `clients`, `client_rooms`, `agents`, `tools`, `models`, `skills`,+`agent_skills`, `rooms`, `room_agents`, `room_skills`, `room_groups`,+`room_links`, `room_roles`, `role_leases`, `tasks`, `references`,+`reference_types`, `documents`, `room_references`, `room_documents`,+`packages`, `room_packages`, `package_references`, `package_documents`,+`collaboration_bridges`, `server_connectors`, `external_users`,+`external_user_claims`, `agent_sessions`, `agent_runtime_states`,+`bridge_message_map`, `messages`, `message_attachments`, `delivery_cursors`,+`media_blobs`.++Longer than the spike's thirteen, because the spike named top-level entities+and the schema has children hanging off them. The test is mechanical: if a row+would be meaningless to another customer, it is scoped.++Three entries are worth justifying:++- **Junction tables are scoped like everything else.** `room_agents` joins a+ room to an agent; both parents are scoped, and the pair is exactly where a+ cross-tenant reference gets introduced. Its own `tenant_id` plus composite+ keys to both parents makes "an agent from tenant B in a room from tenant A"+ unrepresentable. Inheriting tenancy through a subquery policy instead is+ slower and needs per-table reasoning.+- **`media_blobs` is scoped** although nothing references it by foreign key. It+ holds attachment bytes reached by an opaque URI, and unguessable identifiers+ are not an isolation boundary.+- **`clients` is scoped**, so the admin/system client becomes one row per+ tenant rather than one per deployment. Today that is a no-op — the single+ existing row backfills to tenant zero — and it is the right shape later.++**Global — no `tenant_id`, no policy:** `users`, `oidc_identities`,+`feature_flags`, `alembic_version`.++A user is a person, not a tenant member; the membership row is the per-tenant+object. `oidc_identities` records how that person proves who they are, equally+tenant-independent. `feature_flags` is a deployment switch; a flag that needs+to vary per tenant is a new table, not a nullable column.++Leaving `users` and `oidc_identities` unpoliced has a consequence, and it is+recorded as an open item rather than buried: see "What Phase 1 does not close".++## Uniqueness changes++Adding a column is the easy half. Constraints are what break silently when a+second tenant appears, so the whole list was walked.++**Becomes per-tenant:**++- `agents.name` → `unique (tenant_id, name)`. Two customers can both have a+ `reviewer`. The registration path already treats owner equality as a de facto+ tenant boundary and calls a mismatch a cross-tenant takeover; this makes it+ real.+- `clients.matrix_user_id` → `unique (tenant_id, matrix_user_id)`, required by+ per-tenant system clients.+- `rooms.matrix_room_id` → `unique (tenant_id, matrix_room_id)`.+- `reference_types` primary key → `(tenant_id, type)`. Customer-defined type+ slugs collide. This table has no `id` column at all, so it is the one scoped+ table that cannot carry a `unique (id, tenant_id)` — harmless, because+ nothing references it by foreign key.+- `collaboration_bridges` default-bridge index → `unique (tenant_id) where+ is_default`. "One default bridge for the deployment" is exactly the global+ singleton that breaks on the second tenant.++**Stays global, deliberately:**++- `users.email`. A person is one account across tenants.+- `api_keys.key_hash`. Bearer authentication resolves the hash *before* it+ knows a tenant. Load-bearing — see the bootstrap section.+- `oidc_identities (iss, sub)`. The issuer already namespaces it.+- `media_blobs.uri`, `messages.transport_event_id`. Random globally-unique+ identifiers; scoping them buys nothing.++**Already scoped transitively, left alone:** `messages (room_id, seq)`,+`documents (room_id, name)`, `room_roles (room_id, name)`,+`delivery_cursors (agent_id, room_id)`, `agent_sessions`,+`agent_runtime_states`, `external_users (bridge_id, external_user_id)`,+`bridge_message_map`, `rooms (bridge_id, external_channel_id)`,+`agents.client_id`. Each is unique within a parent that is itself scoped, and+the composite foreign key keeps the parent honest.++`role_leases` is unique on `agent_id` alone across all rooms by design — an+agent has one session and one lease — and an agent implies a tenant, so no+change.++## Composite foreign keys++Every foreign key from a scoped table to a scoped table gains `tenant_id`:++```sql+alter table rooms add constraint uq_rooms_id_tenant unique (id, tenant_id);++alter table messages+ add constraint fk_messages_room+ foreign key (tenant_id, room_id) references rooms (tenant_id, id);+```++Only tables that are actually referenced need the extra `unique (id,+tenant_id)` — about thirteen, not one per scoped table. Foreign keys to+*global* tables (`rooms.owner_id → users.id` and friends) stay single-column.++Two mechanical details decide whether this works at all:++- **Nullable parents keep Postgres's default `MATCH SIMPLE`**, under which a+ NULL in any column skips the check entirely. That is what we want: no parent,+ nothing to verify. `MATCH FULL` would reject a row with a non-null tenant and+ a null parent, which is a legitimate row.+- **`ON DELETE SET NULL` must name its column.** A plain multi-column `SET+ NULL` nulls *every* referencing column, `tenant_id` included, so the delete+ fails on the not-null constraint. Five scoped-to-scoped keys are affected —+ `agents.parent_agent_id`, `rooms.group_id`, `room_groups.parent_group_id`,+ `documents.created_by_agent_id`, `messages.sender_client_id` — and each needs+ the Postgres 15+ form `on delete set null (parent_agent_id)`. Without it,+ deleting a room group or a parent agent is broken from day one. This is the+ single easiest thing to get wrong here.++`ON DELETE CASCADE` needs no such care: it removes the child row entirely.++The cost is every foreign key rewritten and thirteen indexes added. The benefit+is invisible until something goes wrong. The cheaper variant is composite keys+on the junction tables and `messages` only, accepting that everything else+relies on the application writing the right parent id; I recommend against it,+because an inconsistent rule is harder to hold in your head than a uniform one+and the tables left out would be the ones nobody thinks about.++## Row-level security++One function, one policy text, applied identically to all 38 scoped tables (the+36 above plus `tenants` and `tenant_members`).++```sql+create or replace function require_tenant_id() returns text+language plpgsql stable as $$+declare v text := current_setting('app.tenant_id', true);+begin+ if v is null or v = '' then+ raise exception 'app.tenant_id is not set on this session'+ using errcode = '42501';+ end if;+ return v;+end $$;+```++Three cases, one behaviour. Never set in this session returns NULL because of+the `true`; set and then released at commit returns the empty string, not NULL+— the trap that produced a P1 elsewhere in the company; set returns the value.+Both empty cases raise. **The function fails closed, and every other safety+property here rests on that.**++```sql+alter table <t> enable row level security;++create policy tenant_isolation on <t>+ for all+ using (tenant_id = (select require_tenant_id()))+ with check (tenant_id = (select require_tenant_id()));+```++- **`with check` is not optional.** `using` filters what you can read; without+ `with check` a write is unconstrained and you can insert a row into another+ tenant that you cannot then read back. Two of the four prior-art bugs are+ this.+- **`(select require_tenant_id())`** rather than a bare call, so the planner+ evaluates it once per query as an InitPlan rather than once per row. Marking+ the function `stable` is not sufficient on its own.+- **No `TO <role>` clause.** An earlier draft named the runtime role; that+ makes the DDL fail wherever the role does not exist yet, including+ `create_all` in tests. The owner-bypass model below already defines exactly+ who is and is not subject to policies, so the clause bought nothing.+- `tenants` uses `id = (select require_tenant_id())`; every other table uses+ `tenant_id`.++## One new database role++Today the application connects to Postgres as the superuser in local Compose,+in the chart and in the test containers. **A superuser ignores row-level+security unconditionally** — not "unless forced", unconditionally. Shipping+policies without changing this produces a schema full of decoration and an+isolation test that passes for the wrong reason. This is the largest piece of+work in Phase 1 and the only part that touches deployment.++The change is deliberately small: **add one role, transfer nothing.**++- **The existing role keeps owning the schema** and keeps running Alembic.+ Because it owns the tables it bypasses their policies, and we deliberately do+ **not** set `force row level security`, so that stays true. No `ALTER TABLE+ … OWNER TO`, no `REASSIGN OWNED`, nothing to go wrong during a cutover.+- **`switch_app` is new**: the runtime role, not the owner, not a superuser, no+ `BYPASSRLS`. Policies apply to it and it is the only role the request path+ ever uses.++Roles are cluster-level, so the migration does not create them — it grants to a+role that must already exist and fails loudly if it does not. Creating the role+belongs to Compose, to the chart, and to the RDS runbook, which already+recreates roles by hand. `alter default privileges` covers tables added later,+so a future migration cannot ship a table the runtime cannot read.++**The check that matters runs at startup, not in CI.** CI asserts nothing about+production, and the failure mode here is silent. On boot the service verifies+its own connection: not a superuser, no `BYPASSRLS`, not the owner of+`messages`, and a probe query against a scoped table with no tenant set must+raise. If any of those is wrong the service refuses to start, because a Switch+that thinks it is isolating tenants and is not is worse than one that is down.++**Sequencing.** The role is cheapest to add during the RDS cutover+(CHOO-2622), which already creates an application role by hand, and that+restore uses `--no-owner --no-privileges` — so who ends up owning the restored+tables is a decision that has to be made deliberately there, not discovered+afterwards. Coordinate, do not race.++## Setting the tenant++```sql+select set_config('app.tenant_id', $1, true) -- is_local = true+```++Transaction-scoped, released at commit, so it cannot leak to the next request+that borrows the pooled connection. Both remaining prior-art bugs are+session-scoped settings on a pooled connection; this is the fix, and the+fail-closed function is the backstop when it is missed. Issuing it outside an+explicit transaction only warns and does nothing — survivable here only because+a missing tenant raises rather than matching nothing.++**Where the tenant comes from, per principal:**++| Principal | Source |+| --- | --- |+| Gateway user (JWT cookie) | the user's membership row |+| Agent bearer token | `api_keys.tenant_id` |+| Bridge inbound event | the bridge row's tenant |+| Per-room background work | the room's tenant |++Never from a request parameter; no endpoint accepts a tenant id as input.++Phase 1 has one tenant, so membership resolution returns the single row and+raises if there is more than one. Phase 2 adds tenant switching and an+active-tenant claim; that is a change to one function.++**The request seam has to move.** `get_session` is a FastAPI dependency that+resolves *before* the one that authenticates the user, so there is no moment+today where the principal is known and the session is not yet open. Endpoints+therefore stop depending on `get_session` and depend on a `tenant_session` that+composes it with the principal and issues the `set_config`. That is a+mechanical edit at 107 call sites, and it puts the seam in every signature+where a reader can see it. A test asserts no endpoint uses the raw session.++**Background work is not a short list.** There are roughly 206 places that open+a session from the factory directly. They fall into three shapes — acting for a+room, acting for a bridge, acting for the deployment — so they get two named+helpers, `tenant_session(tenant_id)` and `system_session()`, and a test that+the raw factory is not called anywhere else.++**Writes fill the column automatically.** `tenant_id` gets a Python-side+default reading the request's tenant, so ordinary ORM inserts need no change+across roughly 250 store methods, and `with check` catches anything that+disagrees. Two caveats, both real: the default yields nothing in a system+session, so the startup seeding paths must pass tenant zero explicitly; and one+Core `executemany` in the room store needs checking rather than assuming.++The same context value feeds the logging filter, which has had a `tenant_id`+field since before tenants existed and which nothing has ever bound+per-request. That closes half of Phase 0's logging item as a side effect.++## The bootstrap problem, and the two exceptions++Authentication happens before a tenant is known — resolving an API key by hash,+or a JWT subject to its memberships, is unscoped by definition. So:++> **Authentication and tenant resolution run in a system session. Everything+> downstream runs in a tenant session as `switch_app`.**++The same escape hatch covers work that is legitimately cross-tenant: Alembic,+admin and bootstrap seeding at startup, the bridge and connector lifecycle+loops that start every row at boot, and the runtime-state and connection+sweeps. A test pins the set of modules allowed to open one, so a new one is a+deliberate act.++The delivery listener needs no exception: it holds one unpooled connection that+relays NOTIFY payloads and never reads a scoped table. Its consumers do read+per room — so **the notify payload gains `tenant_id`** alongside the room, seq+and id it already carries. Without it a consumer has to make an unscoped read+just to learn which tenant to scope by, which is a circle.++Two exceptions to the uniform rule, in full:++1. **`api_keys.key_hash` stays globally unique**, because authentication+ resolves it before a tenant exists.+2. **System sessions bypass policies by ownership.** This is a fail-open hatch+ inside a fail-closed design and is named as such: nothing stops a pinned+ module reading across tenants once a second one exists. Phase 1 accepts+ that; the pinned list is what keeps it reviewable.++An earlier draft had a third — a membership-based policy on `users` — and it+was wrong. Its `with check` made creating a user impossible: the membership row+cannot exist before the user, and the user cannot be inserted before the+membership. It also would not have held, because `tenant_members` has no+constraint on *which* user id a tenant may add to itself. Both problems are+Phase 2's to solve properly, alongside invitations.++## Where the SQL lives++The repository already has the pattern in the delivery trigger: the DDL lives+in one module, is attached to the tables so `create_all` builds it, and each+migration carries its own frozen verbatim copy so that editing the module never+changes what a past migration means. The policies follow it exactly, in+`db/rls_ddl.py`.++This matters more than it sounds. Tests build their schema from the model+metadata rather than by running migrations, so a policy that existed only in a+migration would be invisible to every test — the isolation test would pass+against a database with no isolation in it.++A `TenantScoped` declarative mixin carries the column, its default and its+registration in the policy list, so adding a table means inheriting from it and+writing the foreign keys. Without the mixin a new table needs seven separate+things remembered; with it, two, and both are checked by tests.++## The migration++One revision:++1. Create `tenants` and `tenant_members`; create `require_tenant_id()`.+2. Insert tenant zero, slug `default`, with a fixed id constant written into+ the migration. Not read from the environment: `TENANT_ID` is config today+ and a later edit to it must not silently desync from the row.+3. Insert a membership for every existing user — `owner` for accounts with the+ global admin role, `member` otherwise.+4. Per scoped table: `add column tenant_id text not null default '<zero>'`,+ then drop the default. Since Postgres 11 that is a metadata-only operation;+ the add-nullable-then-backfill-then-set-not-null sequence the spike+ describes rewrites the whole table under an exclusive lock, which on+ `messages` and on `media_blobs` with its 20MB rows is the difference between+ a blink and an outage.+5. Add the foreign key to `tenants` and, where the table is referenced, the+ `unique (id, tenant_id)`.+6. Swap the uniqueness constraints listed above.+7. Rewrite each scoped-to-scoped foreign key as composite, `not valid` first+ and `validate constraint` after, so the validation scan does not hold an+ exclusive lock.+8. Enable row-level security and create the policy on each table.+9. Grant to `switch_app` and set default privileges.++Only tenant zero exists when this runs, so the usual expand-then-contract+caution does not apply yet. From the next customer onward it does, and that+note belongs in the migration's docstring.++## Done when++An integration test proves tenant A cannot read tenant B's rows through the+ordinary application paths — connected as the runtime role, through the store+layer, not by hand-written SQL. That needs test infrastructure that does not+exist yet: the fixtures create `switch_app`, grant to it, connect as it, and+seed tenant zero, because the truncation between tests now empties `tenants`+too.++Alongside it, three cheap tests that catch the regressions this design exists+to prevent:++- Every scoped table has row-level security enabled and a policy, asserted by+ querying `pg_policies` and `pg_tables` rather than by reading the code.+- Every foreign key between two scoped tables includes `tenant_id`, asserted+ the same way from the catalogue. Without this a new table passes every other+ check while quietly falling back to a single-column key.+- A query issued with no tenant set raises rather than returning nothing.++## What Phase 1 does not close++Named so they are decisions rather than omissions.++- **Cross-tenant user enumeration.** `users` and `oidc_identities` have no+ policy, so any tenant session can read every account. There is no exposure+ while one tenant exists, and the correct fix needs invitations and a+ per-tenant user list, which is Phase 2's scope. It must not ship without it.+- **`tenant_members` self-service.** A tenant session can insert a membership+ row naming any user id. Same owner, same phase.+- **`users.email` is globally unique**, so signing up with an address that+ exists in another tenant fails on the index rather than being handled — an+ existence oracle and a bad error. Phase 2.+- **System sessions can read across tenants** by design, as above.+- **No indexes on `tenant_id`.** With one tenant the column has no selectivity+ and an index is pure write cost. Add them when the second tenant lands, in+ one migration, `concurrently`, with data to measure against.+- No plan, status or soft-delete on `tenants`; no tenant deletion; no+ per-tenant feature flags; no tenant switching.+- No per-tenant agent registration credential — Phase 2 owns it as a security+ item, not a refactor.+- No renaming of the residual `matrix_*` columns. Worth doing while these+ tables are open, but not in the migration that changes isolation.+- The room advisory lock hashes only the room id, so it is a cluster-global+ namespace shared across tenants. Harmless at today's scale, worth a note.