Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

fix(db): make the tenant catalogue fail closed, not open (CHOO-2623)

Wojciech Wojtyniak655255996b6a
diff --git a/core/switch_core/db/rls_ddl.py b/core/switch_core/db/rls_ddl.py
index 62bfd87ea..fb433b245 100644
--- a/core/switch_core/db/rls_ddl.py
+++ b/core/switch_core/db/rls_ddl.py
@@ -65,15 +65,31 @@ The policy text is one shape, applied identically to every scoped table:
- **`tenants` compares on `id`**, because a tenant *is* the boundary rather
than belonging to one. Every other scoped table compares on `tenant_id`.
-**The table list is derived, not copied.** A table is scoped if it carries a
+**Scoped is the default; global is the list.** The rule is stated the way
+round that fails closed: every table in the metadata is scoped *unless* it is
+named in `GLOBAL_TABLES`, and a table that is neither named there nor
+actually carrying a tenant raises `UnscopedTableError` — at import, because
+`db/models.py` calls `attach_row_level_security` at the bottom, which is to
+say the process does not start.
+
+This was originally written the other way round, deriving "scoped" from *has
+a `tenant_id` foreign key to `tenants.id`*, and that version failed open: a
+new table holding customer data and no tenant column simply was not in the
+set, so nothing enabled a policy on it, nothing complained, and the
+catalogue tests — which take their table list from here — passed. That is
+precisely the regression these checks exist to catch, so the invariant is
+inverted. Adding a table is now a choice between inheriting `TenantScoped`
+and arguing in a diff for an entry in `GLOBAL_TABLES`; forgetting to choose
+is not one of the options.
+
+The membership test for "actually carries a tenant" is unchanged: a
`tenant_id` column whose foreign key targets `tenants.id` — exactly
`TenantScoped`'s column, and exactly what a plain `Table()` junction table
gets by adding the equivalent column inline, since the mixin only works on a
-declarative class. Deriving it here means a table added later either follows
-the pattern and is picked up automatically, or does not and is exempt from
-these policies while still being visible as a gap to the catalogue test in
-`tests/switch_core/db/test_row_level_security.py` — never silently skipped by
-a list someone forgot to extend.
+declarative class. The foreign key, not just the column name: a table that
+merely happens to have a column called `tenant_id` without the constraint is
+not tied to a tenant, and a policy comparing it would reject every row the
+table holds.
**Where the SQL lives.** Same house pattern as `notify_ddl.py`: this module's
DDL is attached to `Base.metadata` (`db/models.py`) so `create_all` builds it
@@ -86,11 +102,35 @@ this module later does.
from __future__ import annotations
-from sqlalchemy import DDL, MetaData, event
+from sqlalchemy import DDL, MetaData, Table, event
REQUIRE_TENANT_FUNCTION_NAME = "require_tenant_id"
POLICY_NAME = "tenant_isolation"
+# The whole exemption from tenancy, written out. Everything else in
+# `Base.metadata` is scoped, so this list is the only place a table can be
+# excused from carrying a tenant, and extending it is an edit a reviewer sees
+# — `tests/switch_core/db/test_tenant_schema_catalogue.py` pins the contents,
+# so widening it fails the suite until someone changes the test too.
+#
+# A person is not a tenant member (`users`), nor is the record of how they
+# prove who they are (`oidc_identities`); `feature_flags` is a deployment
+# switch, and a flag that needs to vary per customer is a new scoped table
+# rather than a nullable column here. Alembic's `alembic_version` needs no
+# entry: Alembic owns that table and never registers it on this metadata.
+GLOBAL_TABLES = frozenset({"users", "oidc_identities", "feature_flags"})
+
+
+class UnscopedTableError(RuntimeError):
+ """A table is neither tenant-scoped nor named in `GLOBAL_TABLES`.
+
+ Raised while attaching the policies, which happens at the bottom of
+ `db/models.py` — so a table added without a tenant and without a
+ deliberate exemption stops the process at import rather than quietly
+ shipping a table with customer data in it and no isolation policy.
+ """
+
+
# SQLSTATE 42501, insufficient_privilege — the closest standard code for "you
# may not see or touch this row", and the one an operator's tooling is likely
# to already recognise as an authorization failure rather than a generic
@@ -135,29 +175,71 @@ def drop_policy_ddl(table_name: str) -> str:
return f'DROP POLICY IF EXISTS {POLICY_NAME} ON "{table_name}"'
+def _tenant_column(table: Table) -> str | None:
+ """The column this table's policy would compare, or None if it has none.
+
+ `tenants` is the one table compared on `id`, because a tenant *is* the
+ boundary rather than belonging to one. Every other table qualifies by
+ carrying a `tenant_id` column whose foreign key targets `tenants.id`.
+ """
+ if table.name == "tenants":
+ return "id"
+ tenant_id_col = table.columns.get("tenant_id")
+ if tenant_id_col is None:
+ return None
+ if any(
+ fk.column.table.name == "tenants" and fk.column.name == "id"
+ for fk in tenant_id_col.foreign_keys
+ ):
+ return "tenant_id"
+ return None
+
+
+def unscoped_tables(metadata: MetaData) -> list[str]:
+ """Every table that breaks the rule: not global, and carrying no tenant.
+
+ Empty in a healthy schema. `scoped_tables` refuses to answer at all while
+ this is non-empty; it is exposed separately so a test can state the
+ invariant in its own words, and so a caller that wants the diagnosis
+ rather than the exception can ask for it.
+ """
+ return sorted(
+ table.name
+ for table in metadata.tables.values()
+ if table.name not in GLOBAL_TABLES and _tenant_column(table) is None
+ )
+
+
def scoped_tables(metadata: MetaData) -> dict[str, str]:
- """Table name -> the column its policy compares, derived from the schema.
-
- `tenants` is the one table scoped by `id`: every other table is scoped by
- carrying a `tenant_id` column whose foreign key targets `tenants.id`. That
- second test — the foreign key, not just the column name — is deliberate:
- a table that merely happens to have a column called `tenant_id` without
- the constraint is not actually tied to a tenant and must not silently pick
- up a policy that would then reject every row it holds.
+ """Table name -> the column its policy compares: everything not global.
+
+ Raises `UnscopedTableError` naming any table that is neither in
+ `GLOBAL_TABLES` nor actually carrying a tenant. The failure is the point:
+ deriving the set from "has a tenant column" instead would quietly answer
+ with a shorter list, and a table holding customer data with no isolation
+ on it would be indistinguishable from one that was never added.
"""
tables: dict[str, str] = {}
+ violations: list[str] = []
for table in metadata.tables.values():
- if table.name == "tenants":
- tables[table.name] = "id"
+ if table.name in GLOBAL_TABLES:
continue
- tenant_id_col = table.columns.get("tenant_id")
- if tenant_id_col is None:
- continue
- if any(
- fk.column.table.name == "tenants" and fk.column.name == "id"
- for fk in tenant_id_col.foreign_keys
- ):
- tables[table.name] = "tenant_id"
+ column = _tenant_column(table)
+ if column is None:
+ violations.append(table.name)
+ else:
+ tables[table.name] = column
+ if violations:
+ raise UnscopedTableError(
+ f"table(s) carry no tenant and are not global: {sorted(violations)}. "
+ "Every table in the model metadata is tenant-scoped unless it is "
+ "named in db/rls_ddl.py's GLOBAL_TABLES. Inherit TenantScoped (or, "
+ "for a plain Table(), add the equivalent tenant_id column with a "
+ "foreign key to tenants.id), or — if the table genuinely holds no "
+ "customer data — add it to GLOBAL_TABLES, which is a decision "
+ "tests/switch_core/db/test_tenant_schema_catalogue.py makes you "
+ "state twice."
+ )
return tables
diff --git a/core/tests/switch_core/db/stores/test_reference_type_store.py b/core/tests/switch_core/db/stores/test_reference_type_store.py
index f26bc7f9d..748d3be08 100644
--- a/core/tests/switch_core/db/stores/test_reference_type_store.py
+++ b/core/tests/switch_core/db/stores/test_reference_type_store.py
@@ -332,6 +332,49 @@ class TestReferenceTypesAreScopedToTheBoundTenant:
theirs = await store.get(session, "notion")
assert theirs is not None and theirs.display_name == "Theirs"
+ async def test_a_duplicate_slug_clashes_cleanly_when_another_tenant_has_it_too(
+ self, session_factory: async_sessionmaker[AsyncSession]
+ ) -> None:
+ """A clash in the second tenant is a `ValueError`, not a 500.
+
+ `create` reports "already exists" by looking the clashing row up
+ *inside* its `IntegrityError` handler, and that lookup is a
+ `scalar_one_or_none()`. The slug is only unique per tenant, so the
+ moment two tenants hold the same one an unfiltered lookup matches two
+ rows and raises `MultipleResultsFound` from inside the handler —
+ which the gateway has no `except` for, so a 400 the caller can act on
+ (`gateway/references.py` maps `ValueError` to one) becomes a 500. The
+ filter on the row's own tenant is what keeps it single-valued, and
+ this is the arrangement that tells the two apart: the same slug in
+ tenant zero *and* in the tenant doing the create.
+ """
+ other = f"tenant-{uuid.uuid4().hex[:8]}"
+ store = ReferenceTypeStore()
+ async with session_factory() as session:
+ session.add(Tenant(id=other, slug=other, name=other))
+ ada = await _make_user(session, "ada")
+ await session.flush()
+ await store.create(session, _type("notion", ada.id))
+ with tenant_scope(other):
+ await store.create(session, _type("notion", ada.id))
+ await session.commit()
+
+ with tenant_scope(other):
+ with pytest.raises(
+ ValueError, match="Reference type 'notion' already exists"
+ ):
+ await store.create(session, _type("notion", ada.id))
+
+ # The savepoint means the caller's transaction survives it,
+ # and neither tenant's row was disturbed.
+ await store.create(session, _type("linear", ada.id))
+ await session.commit()
+ assert {rt.type for rt in await store.list_all(session)} == {
+ "notion",
+ "linear",
+ }
+ assert {rt.type for rt in await store.list_all(session)} == {"notion"}
+
async def test_counting_references_ignores_another_tenants(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
diff --git a/core/tests/switch_core/db/test_row_level_security.py b/core/tests/switch_core/db/test_row_level_security.py
index d581a8ccd..7e22f6410 100644
--- a/core/tests/switch_core/db/test_row_level_security.py
+++ b/core/tests/switch_core/db/test_row_level_security.py
@@ -43,9 +43,11 @@ import switch_core
from switch_core.db.base import Base
from switch_core.db.models import RoomGroup, Tenant
from switch_core.db.rls_ddl import (
+ GLOBAL_TABLES,
POLICY_NAME,
REQUIRE_TENANT_FUNCTION_NAME,
scoped_tables,
+ unscoped_tables,
)
from switch_core.db.session_scope import tenant_session
from switch_core.db.stores.room_group_store import RoomGroupStore
@@ -347,19 +349,29 @@ class TestCatalogueCoverage:
hand-kept list. The table list itself comes from `rls_ddl.scoped_tables`,
the same derivation `create_all` used to attach the policies in the first
place, not a copy of it.
+
+ That derivation is *everything but `GLOBAL_TABLES`*, and the direction
+ matters. Derived the other way round — "the tables that carry a
+ `tenant_id`" — a new table holding customer data with no tenant column
+ would not be in the list, so nothing below would look for a policy on it
+ and nothing would fail: the regression these tests exist to catch,
+ passing. Inverted, that table has no policy to find and this class says
+ so; and `scoped_tables` itself refuses to answer at all, so in practice
+ the failure arrives even earlier, at import.
"""
- async def test_scoped_table_list_is_not_trivially_small(
+ async def test_every_table_is_scoped_unless_it_is_named_global(
self, session_factory: async_sessionmaker
) -> None:
+ """The list the rest of this class rests on, checked for the failure
+ mode a derived list has: silently being short."""
+ assert unscoped_tables(Base.metadata) == []
scoped = scoped_tables(Base.metadata)
+ assert set(scoped) == set(Base.metadata.tables) - set(GLOBAL_TABLES)
assert len(scoped) >= 30, scoped
- assert "tenants" in scoped
assert scoped["tenants"] == "id"
- assert "messages" in scoped
- assert "agents" in scoped
- for global_table in ("users", "oidc_identities", "feature_flags"):
- assert global_table not in scoped
+ assert scoped["messages"] == "tenant_id"
+ assert scoped["tenant_members"] == "tenant_id"
async def test_every_scoped_table_has_rls_enabled_and_the_right_policy(
self, session_factory: async_sessionmaker
@@ -418,9 +430,14 @@ class TestCatalogueCoverage:
async def test_global_tables_carry_no_policy(
self, session_factory: async_sessionmaker
) -> None:
+ """The other side of the exemption: a table named in `GLOBAL_TABLES`
+ really is left unpoliced, so the list is not a place to hide a table
+ that then quietly gets a policy anyway. Driven by the list itself, so
+ an addition to it is checked here rather than skipped."""
scoped = scoped_tables(Base.metadata)
async with session_factory() as session:
policies = await _tenant_isolation_policies(session)
- for global_table in ("users", "oidc_identities", "feature_flags"):
+ for global_table in sorted(GLOBAL_TABLES):
+ assert global_table in Base.metadata.tables
assert global_table not in scoped
assert global_table not in policies
diff --git a/core/tests/switch_core/db/test_tenant_schema_catalogue.py b/core/tests/switch_core/db/test_tenant_schema_catalogue.py
index a9f10ed56..c2b32c10d 100644
--- a/core/tests/switch_core/db/test_tenant_schema_catalogue.py
+++ b/core/tests/switch_core/db/test_tenant_schema_catalogue.py
@@ -8,6 +8,18 @@ fails here even if nobody remembers to update these tests by hand. See
Every table holding customer data carries a non-null `tenant_id`, and
every foreign key between two such tables carries `tenant_id` as well.
+
+**The invariant is stated the way round that fails closed**, which it was
+not when this file was written. It used to derive "scoped" as *carries a
+`tenant_id` foreign key*, and then check things about the tables in that
+set — so a new table full of customer data and no tenant column was not in
+the set, was checked by nothing, and passed. That is the exact regression
+these tests exist to prevent, so the derivation is inverted: the small set
+of tables with no tenant is hardcoded in `rls_ddl.GLOBAL_TABLES`, and every
+other table in the metadata must be scoped. Adding a table with customer
+data and no tenant column now fails at import — `db/models.py` attaches the
+policies at the bottom and `scoped_tables` refuses — and the two tests below
+pin both halves of the escape hatch so it cannot be widened quietly.
"""
from __future__ import annotations
@@ -19,31 +31,17 @@ from sqlalchemy.ext.asyncio import AsyncConnection, AsyncSession, async_sessionm
# junction tables) on `Base.metadata`.
import switch_core.db.models # noqa: F401
from switch_core.db.base import Base
+from switch_core.db.rls_ddl import GLOBAL_TABLES, scoped_tables, unscoped_tables
def _scoped_tables() -> set[str]:
- """Every table carrying a `tenant_id` column whose foreign key targets
- `tenants.id` — the schema's own definition of "scoped".
-
- Deliberately not the list from the design doc: a table that gains a
- `tenant_id` later by copying the mixin (or the equivalent inline column,
- for the plain `Table()` junction tables) is picked up automatically, and
- one that forgets the foreign key to `tenants` is correctly left out and
- so exempt from these checks — the accompanying test that every scoped
- table's FKs carry `tenant_id` only means anything once a table is on this
- list at all.
+ """Every table the schema says is tenant-scoped: all but the globals.
+
+ The same derivation `create_all` used to attach the policies
+ (`rls_ddl.scoped_tables`), not a second copy of it — a private copy here
+ is what let this file and the policies disagree in the first place.
"""
- scoped = set()
- for table in Base.metadata.tables.values():
- tenant_id_col = table.columns.get("tenant_id")
- if tenant_id_col is None:
- continue
- if any(
- fk.column.table.name == "tenants" and fk.column.name == "id"
- for fk in tenant_id_col.foreign_keys
- ):
- scoped.add(table.name)
- return scoped
+ return set(scoped_tables(Base.metadata))
def _get_foreign_keys(sync_conn, table_name: str) -> list[dict]:
@@ -71,28 +69,53 @@ async def _unique_column_sets(
return await conn.run_sync(_get_unique_column_sets, table_name)
-class TestScopedTableCatalogue:
- async def test_scoped_tables_exist_and_are_not_trivial(
- self, session_factory: async_sessionmaker[AsyncSession]
- ) -> None:
- """Sanity check on the derivation itself: if this list ever collapses
- to empty or to one table, the two tests below would pass vacuously."""
+class TestEveryTableIsScopedUnlessItIsNamedGlobal:
+ """The inverted invariant itself, in two halves.
+
+ Half one: nothing in the metadata is unaccounted for. Half two: the list
+ of things excused is exactly the list someone signed off on. Without the
+ second, the first is trivially satisfiable by adding the new table to
+ `GLOBAL_TABLES` — which is a legitimate thing to do and must simply be
+ visible in a diff rather than available as a quick way to get a red suite
+ green.
+ """
+
+ async def test_no_table_escapes_the_rule(self) -> None:
+ assert unscoped_tables(Base.metadata) == [], (
+ "table(s) hold customer data with no tenant and no exemption; "
+ "inherit TenantScoped, or name them in rls_ddl.GLOBAL_TABLES and "
+ "update the test below"
+ )
+
+ async def test_the_global_table_list_is_exactly_these(self) -> None:
+ """Hardcoded on purpose, and hardcoded twice on purpose.
+
+ `users` is a person rather than a tenant member, `oidc_identities`
+ records how that person proves who they are, and `feature_flags` is a
+ deployment switch — a flag that has to vary per customer is a new
+ scoped table, not a nullable column there. `alembic_version` is
+ global too but is not in this metadata: Alembic owns it.
+ """
+ assert set(GLOBAL_TABLES) == {"users", "oidc_identities", "feature_flags"}
+
+ async def test_scoped_is_everything_else(self) -> None:
scoped = _scoped_tables()
+ assert scoped == set(Base.metadata.tables) - set(GLOBAL_TABLES)
+ # Not vacuous, and the shape is what the design doc describes.
assert len(scoped) >= 30, scoped
- assert "messages" in scoped
- assert "agents" in scoped
- # Global tables must not appear.
- assert "users" not in scoped
- assert "oidc_identities" not in scoped
- assert "feature_flags" not in scoped
+ assert {"messages", "agents", "tenants", "tenant_members"} <= scoped
+
+class TestScopedTableCatalogue:
async def test_every_cross_scoped_foreign_key_carries_tenant_id(
self, session_factory: async_sessionmaker[AsyncSession]
) -> None:
"""A foreign key from one scoped table to another must be composite
on `tenant_id`, so a row can never reference a parent in another
tenant. Foreign keys to a *global* table (e.g. `rooms.owner_id ->
- users.id`) are single-column by design and are not checked here."""
+ users.id`) are single-column by design and are not checked here, and
+ neither is a table's own key to `tenants` — that key is what makes it
+ scoped in the first place and is single-column by construction."""
scoped = _scoped_tables()
violations: list[tuple[str, str, str]] = []
async with session_factory() as session:
@@ -100,7 +123,7 @@ class TestScopedTableCatalogue:
for table_name in scoped:
for fk in await _foreign_keys(conn, table_name):
ref_table = fk["referred_table"]
- if ref_table not in scoped:
+ if ref_table not in scoped or ref_table == "tenants":
continue
local_cols = set(fk["constrained_columns"])
ref_cols = set(fk["referred_columns"])
diff --git a/docs/old/multi-tenancy-phase1-db.md b/docs/old/multi-tenancy-phase1-db.md
index 3411bbc9f..03cb62727 100644
--- a/docs/old/multi-tenancy-phase1-db.md
+++ b/docs/old/multi-tenancy-phase1-db.md
@@ -127,6 +127,26 @@ object. `oidc_identities` records how that person proves who they are, equally
tenant-independent. `feature_flags` is a deployment switch; a flag that needs
to vary per tenant is a new table, not a nullable column.
+**This is the list that is written down, and the scoped list is the one that
+is derived** — that way round, deliberately. The three in the model metadata
+are hardcoded in `db/rls_ddl.py`'s `GLOBAL_TABLES` (`alembic_version` needs no
+entry; Alembic owns it and never registers it there), and **every other table
+in the metadata is scoped by definition**: it must carry the tenant, must have
+a policy, and must carry `tenant_id` on its foreign keys to other scoped
+tables. A table that is neither in that list nor actually carrying a tenant
+raises `UnscopedTableError` from `scoped_tables()`, which runs at the bottom
+of `db/models.py` — so the process does not start.
+
+Stated the other way round it fails open, and it was written that way round
+first: deriving "scoped" as *has a `tenant_id` foreign key* means a new table
+full of customer data and no tenant column is simply not in the set, so
+nothing attaches a policy to it, nothing checks its foreign keys, and every
+catalogue test passes. That is the regression these tests exist to prevent,
+passing. Widening `GLOBAL_TABLES` is the intended escape hatch and stays
+available — it just has to be argued for in a diff, because
+`tests/switch_core/db/test_tenant_schema_catalogue.py` pins the contents of
+the list as well as the rule.
+
Leaving `users` and `oidc_identities` unpoliced has a consequence, and it is
recorded as an open item rather than buried: see "What Phase 1 does not close".
@@ -629,10 +649,13 @@ metadata rather than by running migrations, so a policy that existed only in a
migration would be invisible to every test — the isolation test would pass
against a database with no isolation in it.
-A `TenantScoped` declarative mixin carries the column, its default and its
-registration in the policy list, so adding a table means inheriting from it and
-writing the foreign keys. Without the mixin a new table needs seven separate
-things remembered; with it, two, and both are checked by tests.
+A `TenantScoped` declarative mixin carries the column and its default, so
+adding a table means inheriting from it and writing the foreign keys. Without
+the mixin a new table needs seven separate things remembered; with it, two,
+and both are checked by tests. There is no third thing to remember —
+registering the table in the policy list — because there is no list to
+register in: a table is in it unless `GLOBAL_TABLES` says otherwise, and a
+table that forgets the mixin does not get skipped, it stops the import.
## The migration
@@ -723,13 +746,23 @@ real environment would pass for the wrong reason.
Alongside it, the cheap tests that catch the regressions this design exists
to prevent — in that file unless another is named:
+- **Every table in the metadata is scoped unless it is named in
+ `rls_ddl.GLOBAL_TABLES`**, and that list is pinned by its own test so
+ extending it is a two-place, reviewable change rather than the quick way to
+ quieten a red suite (`tests/switch_core/db/test_tenant_schema_catalogue.py`,
+ and the same list drives every check below). This is the inverted form
+ described under "Which tables are scoped"; the original derivation —
+ "scoped" means *carries a `tenant_id` foreign key* — was checked against a
+ table added with customer data and no tenant column, and the whole
+ catalogue passed.
- Every scoped table has row-level security enabled and a policy whose
`using` and `with check` both isolate on that table's tenant column,
asserted by querying `pg_policies` and `pg_tables` rather than by reading
- the code, with the table list itself derived from `rls_ddl.scoped_tables`
- rather than copied. The predicates are compared by content, not merely for
- being present: `using (true)` is a policy that is enabled, catalogued and
- isolates nothing.
+ the code, with the table list itself coming from `rls_ddl.scoped_tables` —
+ the same derivation that attached the policies, not a second copy of it,
+ which is what let the catalogue and the policies disagree before. The
+ predicates are compared by content, not merely for being present: `using
+ (true)` is a policy that is enabled, catalogued and isolates nothing.
- The migration's frozen `SCOPED_TABLES` matches that same derivation. A
frozen copy is the right shape (a migration must not change meaning
because a model did) and its failure mode is falling behind in silence —
@@ -756,6 +789,17 @@ to prevent — in that file unless another is named:
database and on a deployment that gained a tenant between two boots
(`tests/switch_core/clients/test_ensure_system_client.py`). It had no test
at all until it stopped booting.
+- A reference type whose slug clashes *while another tenant holds the same
+ slug* still reports "already exists" rather than a 500
+ (`tests/switch_core/db/stores/test_reference_type_store.py`).
+ `ReferenceTypeStore.create` names the clashing row by looking it up inside
+ its own `IntegrityError` handler, with a `scalar_one_or_none()`; the slug is
+ only unique per tenant, so an unfiltered lookup there matches two rows and
+ raises `MultipleResultsFound` from inside the handler — turning the 400
+ `gateway/references.py` maps `ValueError` to into an unhandled 500. The
+ lookup names the row's own tenant, and the arrangement that tells the two
+ apart needs the slug present in two tenants at once, which no
+ single-tenant test could produce.
The foreign-key-carries-`tenant_id` catalogue check this section originally
asked for shipped earlier, with the schema migration — see