The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Replace timeout-prone permissions-collapse migration with aggregate-first query (#77727)
Alexander Polyankinfcd561b654d7
diff --git a/resources/migrations/062/20260707_collapse_uniform_table_permissions.yaml b/resources/migrations/062/20260707_collapse_uniform_table_permissions.yamlindex 25532bec019..0f2ddef2d46 100644--- a/resources/migrations/062/20260707_collapse_uniform_table_permissions.yaml+++ b/resources/migrations/062/20260707_collapse_uniform_table_permissions.yaml@@ -1,81 +1,141 @@-## Collapse redundant table-level data_permissions rows into db-level rows.+## Collapse redundant table-level data_permissions rows into db-level rows (#76077).##-## A bug in the new-table default-permissions path ("going granular") deleted a group's db-level row and-## wrote one row per table even when the db-level row was already `blocked` and every table's new value was-## `blocked`. On instances with many databases and groups this ballooned data_permissions into tens of-## millions of rows encoding no information beyond their db-level equivalents, which in turn caused-## OOMs in code paths that read a user's or the whole instance's permission rows (#76077).-##-## Two steps, both restricted to perms/view-data and semantically lossless:-## 1. For each (group_id, db_id) with no db-level view-data row whose table-level view-data rows ALL-## carry the value 'blocked' — including rows referencing inactive tables, so collapsing never-## discards a value a reactivated table would otherwise pick back up — and cover every active table-## of the database: insert the equivalent db-level row.-## 2. Drop every table-level view-data row whose (group_id, db_id, perm_value) matches a db-level-## row — such rows are redundant no matter how they came to be. Because that can be tens of millions-## of rows, this is done by staging the surviving rows, truncating the table, and reloading them,-## rather than a row-by-row DELETE (which scales super-linearly); see the second changeSet.-##-## Restricted to view-data / blocked because:-## - only the least-permissive value is safe to collapse at all: table-level rows are ignored for-## inactive tables (they read as least-permissive), while a db-level row applies to inactive tables-## too, so collapsing any other value would silently upgrade inactive tables;-## - the permissions graph API intentionally reports other perm types' table-level rows as granular-## even when uniform (#73520), so collapsing them would change what the admin UI displays;-## - the going-granular bug only ever produced view-data rows, so this still covers the explosion.-##-## Combos with partial table coverage, mixed values, or a uniform non-blocked value are intentionally-## left untouched.-##-## Lives in 062 so the changeSet runs on upgrades to v62 and later (the fix is not backported earlier).+## A bug wrote one 'blocked' view-data row per table even when the db-level row was already 'blocked',+## ballooning data_permissions into millions of redundant rows. Two steps:+## 1. For each (group, db) with no db-level view-data row, whose table-level view-data rows are all+## 'blocked' and cover every active table: insert the equivalent db-level row.+## 2. Drop every table-level view-data row made redundant by an equal-value db-level row.+## Only view-data/blocked is safe to collapse; all other combos are left untouched.databaseChangeLog:- logicalFilePath: migrations/062_update_migrations.yaml+ # Step 1 originally shipped as v62.2026-07-07T00:00:00, which timed out on large instances (its plan+ # was O(rows x databases) on postgres). The changeSets below replace it under new ids; this DELETE just+ # removes the retired changeSet's stale changelog entry from instances that already ran it. For them the+ # replacements re-collapse already-collapsed data, which is a cheap no-op.- changeSet:- id: v62.2026-07-07T00:00:00+ id: v62.2026-07-14T00:00:00author: ranquild+ comment: Retire the timeout-prone original of the collapse changeSets below (#76077)+ changes:+ - sql:+ sql: DELETE FROM DATABASECHANGELOG WHERE id = 'v62.2026-07-07T00:00:00';+ rollback: # not necessary++ # Step 1. Aggregate to (group_id, db_id) first, then join. Coverage is counted in rows, not DISTINCT+ # tables — equivalent unless duplicate rows exist, which the app never writes. Postgres runs the+ # staged, parallel version in the changeSets below instead.+ - changeSet:+ id: v62.2026-07-14T00:00:01+ author: ranquild+ dbms: mysql,mariadb,h2comment: Insert db-level data_permissions rows for uniform full-coverage blocked table-level view-data permissions (#76077)changes:- sql:sql: >-INSERT INTO data_permissions (group_id, perm_type, db_id, perm_value)- SELECT dp.group_id, dp.perm_type, dp.db_id, MIN(dp.perm_value)- FROM data_permissions dp- LEFT JOIN metabase_table mt ON mt.id = dp.table_id+ SELECT agg.group_id, 'perms/view-data', agg.db_id, 'blocked'+ FROM (+ SELECT dp.group_id, dp.db_id,+ MIN(dp.perm_value) AS min_value,+ MAX(dp.perm_value) AS max_value,+ COUNT(*) AS covered_rows,+ COUNT(it.id) AS inactive_rows+ FROM data_permissions dp+ LEFT JOIN metabase_table it ON it.id = dp.table_id AND it.active = FALSE+ WHERE dp.perm_type = 'perms/view-data'+ AND dp.table_id IS NOT NULL+ GROUP BY dp.group_id, dp.db_id+ ) aggJOIN (SELECT db_id, COUNT(*) AS n FROM metabase_table WHERE active = TRUE GROUP BY db_id) tc- ON tc.db_id = dp.db_id- WHERE dp.perm_type = 'perms/view-data'- AND dp.table_id IS NOT NULL+ ON tc.db_id = agg.db_id+ WHERE agg.min_value = 'blocked'+ AND agg.max_value = 'blocked'+ AND agg.covered_rows - agg.inactive_rows = tc.nAND NOT EXISTS (SELECT 1 FROM data_permissions dbp- WHERE dbp.group_id = dp.group_id- AND dbp.db_id = dp.db_id- AND dbp.perm_type = dp.perm_type- AND dbp.table_id IS NULL)- GROUP BY dp.group_id, dp.db_id, dp.perm_type, tc.n- HAVING COUNT(DISTINCT dp.perm_value) = 1- AND MIN(dp.perm_value) = 'blocked'- AND COUNT(DISTINCT CASE WHEN mt.active = TRUE THEN dp.table_id END) = tc.n;+ WHERE dbp.group_id = agg.group_id+ AND dbp.db_id = agg.db_id+ AND dbp.perm_type = 'perms/view-data'+ AND dbp.table_id IS NULL);rollback: # not necessary; the collapsed representation is understood by all versions >= 49- # The rows to keep are every data_permissions row EXCEPT the table-level view-data rows that a db-level- # row of the same (group, db, value) has made redundant — the anti-join below. Deleting the redundant rows- # directly scales super-linearly on a table ballooned to tens of millions of rows (each delete maintains- # six index trees), so instead of a DELETE we stage the few thousand survivors, empty the table, and- # reload them. Migrations run before the instance serves traffic, so the brief window in which- # data_permissions is empty is never observed; the survivors persist in the staging table until the reload- # finishes (each step is its own changeSet, so a failure resumes from the incomplete step with the staging- # table still present), and because the table is only truncated — never dropped — its indexes and foreign- # keys stay in place. The steps are split into one changeSet each because the migration linter allows only- # a single change per changeSet.+ # Step 1 for postgres. CREATE TABLE AS gets a parallel plan (INSERT ... SELECT and DISTINCT+ # aggregates do not), so the one heavy pass over data_permissions runs on all available workers.+ - changeSet:+ id: v62.2026-07-14T00:00:02+ author: ranquild+ dbms: postgresql+ comment: Stage per group-and-database aggregates of table-level view-data permissions (#76077)+ changes:+ - sql:+ sql: >-+ SET LOCAL max_parallel_workers_per_gather = 8;+ CREATE TABLE data_permissions_collapse_agg AS+ SELECT dp.group_id, dp.db_id,+ COUNT(*) AS covered_rows,+ COUNT(it.id) AS inactive_rows,+ MIN(dp.perm_value) AS min_value,+ MAX(dp.perm_value) AS max_value+ FROM data_permissions dp+ LEFT JOIN metabase_table it ON it.id = dp.table_id AND it.active = FALSE+ WHERE dp.perm_type = 'perms/view-data'+ AND dp.table_id IS NOT NULL+ GROUP BY dp.group_id, dp.db_id;+ rollback: # not necessary; the staging table is dropped at the end of this migration++ - changeSet:+ id: v62.2026-07-14T00:00:03+ author: ranquild+ dbms: postgresql+ comment: Analyze the staged collapse aggregates (#76077)+ changes:+ - sql:+ sql: ANALYZE data_permissions_collapse_agg;+ rollback: # not necessary++ # NOT IN evaluates as one hashed subplan instead of an index probe per staged row.+ - changeSet:+ id: v62.2026-07-14T00:00:04+ author: ranquild+ dbms: postgresql+ comment: Insert db-level data_permissions rows for uniform full-coverage blocked table-level view-data permissions (#76077)+ changes:+ - sql:+ sql: >-+ INSERT INTO data_permissions (group_id, perm_type, db_id, perm_value)+ SELECT agg.group_id, 'perms/view-data', agg.db_id, 'blocked'+ FROM data_permissions_collapse_agg agg+ JOIN (SELECT db_id, COUNT(*) AS n FROM metabase_table WHERE active = TRUE GROUP BY db_id) tc+ ON tc.db_id = agg.db_id+ WHERE agg.min_value = 'blocked'+ AND agg.max_value = 'blocked'+ AND agg.covered_rows - agg.inactive_rows = tc.n+ AND (agg.group_id, agg.db_id) NOT IN (+ SELECT dbp.group_id, dbp.db_id FROM data_permissions dbp+ WHERE dbp.perm_type = 'perms/view-data'+ AND dbp.table_id IS NULL);+ rollback: # not necessary; the collapsed representation is understood by all versions >= 49++ - changeSet:+ id: v62.2026-07-14T00:00:05+ author: ranquild+ dbms: postgresql+ comment: Drop the staged collapse aggregates (#76077)+ changes:+ - sql:+ sql: DROP TABLE data_permissions_collapse_agg;+ rollback: # not necessary++ # Step 2. A row-by-row DELETE of millions of rows is too slow, so: stage the surviving rows,+ # truncate data_permissions, reload. One changeSet per step, so a failed run resumes where it stopped.- changeSet:id: v62.2026-07-07T00:00:01author: ranquildcomment: Create the staging table that will hold the surviving data_permissions rows (#76077)changes:- # A createTable change (rather than a raw CREATE TABLE ... AS SELECT) so Liquibase renders it per- # engine and, on MySQL/MariaDB, appends the ENGINE/CHARACTER SET/COLLATE clause the app-db expects.+ # createTable (not raw SQL) so Liquibase adds MySQL/MariaDB's ENGINE/charset clauses.- createTable:tableName: data_permissions_collapse_keepremarks: Transient staging table holding the data_permissions rows to keep while the table is collapsed; dropped at the end of this migrationdiff --git a/test/metabase/app_db/schema_migrations_test.clj b/test/metabase/app_db/schema_migrations_test.cljindex 3f0d427c763..0ab9a2eab3b 100644--- a/test/metabase/app_db/schema_migrations_test.clj+++ b/test/metabase/app_db/schema_migrations_test.clj@@ -2889,9 +2889,9 @@attrs)))(deftest collapse-uniform-table-permissions-migration-test- (testing "Migrations v62.2026-07-07T00:00:00 through v62.2026-07-07T00:00:05:+ (testing "Migrations v62.2026-07-14T00:00:00 through v62.2026-07-07T00:00:05:uniform full-coverage :blocked view-data table rows collapse to a db-level row (#76077)"- (impl/test-migrations ["v62.2026-07-07T00:00:00" "v62.2026-07-07T00:00:05"] [migrate!]+ (impl/test-migrations ["v62.2026-07-14T00:00:00" "v62.2026-07-07T00:00:05"] [migrate!](let [group-id (t2/insert-returning-pk! :permissions_group {:name "collapse-test-group"})db! (fn [db-name](t2/insert-returning-pk! :metabase_database@@ -2947,7 +2947,19 @@t6-off (table! db-6 false);; db-7: table row duplicating an equal-value db-level row → duplicate deleteddb-7 (db! "collapse-dup-of-db-level")- t7a (table! db-7 true)]+ t7a (table! db-7 true)+ ;; db-8: duplicate row alongside full coverage → conservatively NOT collapsed: coverage is+ ;; row-counted, so the duplicate makes the count exceed the active-table count (the app never+ ;; writes duplicate rows; leaving them uncollapsed preserves the current state)+ db-8 (db! "collapse-dup-full-coverage")+ t8a (table! db-8 true)+ t8b (table! db-8 true)+ ;; db-9: duplicate row exactly offsetting a MISSING table row → collapsed; lossless because a+ ;; table with no row in a group with no db-level row already reads as the least permissive+ ;; value, blocked, which is what the inserted db-level row says+ db-9 (db! "collapse-dup-masks-missing")+ t9a (table! db-9 true)+ _t9b (table! db-9 true)](perm! db-1 t1a "perms/view-data" "blocked")(perm! db-1 t1b "perms/view-data" "blocked")(perm! db-1 t1-off "perms/view-data" "blocked")@@ -2960,6 +2972,11 @@(perm! db-6 t6-off "perms/view-data" "unrestricted")(perm! db-7 nil "perms/view-data" "unrestricted")(perm! db-7 t7a "perms/view-data" "unrestricted")+ (perm! db-8 t8a "perms/view-data" "blocked")+ (perm! db-8 t8a "perms/view-data" "blocked")+ (perm! db-8 t8b "perms/view-data" "blocked")+ (perm! db-9 t9a "perms/view-data" "blocked")+ (perm! db-9 t9a "perms/view-data" "blocked")(migrate!)(testing "uniform :blocked collapses to a single db-level row, dropping the inactive table's row too"(is (= #{[nil "blocked"]} (rows db-1 "perms/view-data"))))@@ -2974,7 +2991,11 @@(testing "a differing value on an inactive table's row blocks the collapse"(is (= #{[t6a "blocked"] [t6-off "unrestricted"]} (rows db-6 "perms/view-data"))))(testing "table rows duplicating an equal-value db-level row are deleted"- (is (= #{[nil "unrestricted"]} (rows db-7 "perms/view-data"))))))))+ (is (= #{[nil "unrestricted"]} (rows db-7 "perms/view-data"))))+ (testing "a duplicate row alongside full coverage conservatively blocks the collapse"+ (is (= #{[t8a "blocked"] [t8b "blocked"]} (rows db-8 "perms/view-data"))))+ (testing "a duplicate row offsetting a missing table row collapses losslessly"+ (is (= #{[nil "blocked"]} (rows db-9 "perms/view-data"))))))))(defn- collection-entity-id[collection-id]