The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Collapse uniform blocked view-data table permissions to db-level rows (#77130)
Alexander Polyankin21f2c65b15d3
diff --git a/resources/migrations/062/20260707_collapse_uniform_table_permissions.yaml b/resources/migrations/062/20260707_collapse_uniform_table_permissions.yamlnew file mode 100644index 00000000000..25532bec019--- /dev/null+++ b/resources/migrations/062/20260707_collapse_uniform_table_permissions.yaml@@ -0,0 +1,138 @@+## Collapse redundant table-level data_permissions rows into db-level rows.+##+## A bug in the new-table default-permissions path ("going granular") deleted a group's db-level row and+## wrote one row per table even when the db-level row was already `blocked` and every table's new value was+## `blocked`. On instances with many databases and groups this ballooned data_permissions into tens of+## millions of rows encoding no information beyond their db-level equivalents, which in turn caused+## OOMs in code paths that read a user's or the whole instance's permission rows (#76077).+##+## Two steps, both restricted to perms/view-data and semantically lossless:+## 1. For each (group_id, db_id) with no db-level view-data row whose table-level view-data rows ALL+## carry the value 'blocked' — including rows referencing inactive tables, so collapsing never+## discards a value a reactivated table would otherwise pick back up — and cover every active table+## of the database: insert the equivalent db-level row.+## 2. Drop every table-level view-data row whose (group_id, db_id, perm_value) matches a db-level+## row — such rows are redundant no matter how they came to be. Because that can be tens of millions+## of rows, this is done by staging the surviving rows, truncating the table, and reloading them,+## rather than a row-by-row DELETE (which scales super-linearly); see the second changeSet.+##+## Restricted to view-data / blocked because:+## - only the least-permissive value is safe to collapse at all: table-level rows are ignored for+## inactive tables (they read as least-permissive), while a db-level row applies to inactive tables+## too, so collapsing any other value would silently upgrade inactive tables;+## - the permissions graph API intentionally reports other perm types' table-level rows as granular+## even when uniform (#73520), so collapsing them would change what the admin UI displays;+## - the going-granular bug only ever produced view-data rows, so this still covers the explosion.+##+## Combos with partial table coverage, mixed values, or a uniform non-blocked value are intentionally+## left untouched.+##+## Lives in 062 so the changeSet runs on upgrades to v62 and later (the fix is not backported earlier).++databaseChangeLog:+ - logicalFilePath: migrations/062_update_migrations.yaml++ - changeSet:+ id: v62.2026-07-07T00:00:00+ author: ranquild+ comment: Insert db-level data_permissions rows for uniform full-coverage blocked table-level view-data permissions (#76077)+ changes:+ - sql:+ sql: >-+ INSERT INTO data_permissions (group_id, perm_type, db_id, perm_value)+ SELECT dp.group_id, dp.perm_type, dp.db_id, MIN(dp.perm_value)+ FROM data_permissions dp+ LEFT JOIN metabase_table mt ON mt.id = dp.table_id+ JOIN (SELECT db_id, COUNT(*) AS n FROM metabase_table WHERE active = TRUE GROUP BY db_id) tc+ ON tc.db_id = dp.db_id+ WHERE dp.perm_type = 'perms/view-data'+ AND dp.table_id IS NOT NULL+ AND NOT EXISTS (+ SELECT 1 FROM data_permissions dbp+ WHERE dbp.group_id = dp.group_id+ AND dbp.db_id = dp.db_id+ AND dbp.perm_type = dp.perm_type+ AND dbp.table_id IS NULL)+ GROUP BY dp.group_id, dp.db_id, dp.perm_type, tc.n+ HAVING COUNT(DISTINCT dp.perm_value) = 1+ AND MIN(dp.perm_value) = 'blocked'+ AND COUNT(DISTINCT CASE WHEN mt.active = TRUE THEN dp.table_id END) = tc.n;+ rollback: # not necessary; the collapsed representation is understood by all versions >= 49++ # The rows to keep are every data_permissions row EXCEPT the table-level view-data rows that a db-level+ # row of the same (group, db, value) has made redundant — the anti-join below. Deleting the redundant rows+ # directly scales super-linearly on a table ballooned to tens of millions of rows (each delete maintains+ # six index trees), so instead of a DELETE we stage the few thousand survivors, empty the table, and+ # reload them. Migrations run before the instance serves traffic, so the brief window in which+ # data_permissions is empty is never observed; the survivors persist in the staging table until the reload+ # finishes (each step is its own changeSet, so a failure resumes from the incomplete step with the staging+ # table still present), and because the table is only truncated — never dropped — its indexes and foreign+ # keys stay in place. The steps are split into one changeSet each because the migration linter allows only+ # a single change per changeSet.+ - changeSet:+ id: v62.2026-07-07T00:00:01+ author: ranquild+ comment: Create the staging table that will hold the surviving data_permissions rows (#76077)+ changes:+ # A createTable change (rather than a raw CREATE TABLE ... AS SELECT) so Liquibase renders it per+ # engine and, on MySQL/MariaDB, appends the ENGINE/CHARACTER SET/COLLATE clause the app-db expects.+ - createTable:+ tableName: data_permissions_collapse_keep+ remarks: Transient staging table holding the data_permissions rows to keep while the table is collapsed; dropped at the end of this migration+ columns:+ - column: {name: id, type: int, remarks: Copy of data_permissions.id}+ - column: {name: group_id, type: int, remarks: Copy of data_permissions.group_id}+ - column: {name: perm_type, type: varchar(64), remarks: Copy of data_permissions.perm_type}+ - column: {name: db_id, type: int, remarks: Copy of data_permissions.db_id}+ - column: {name: schema_name, type: varchar(254), remarks: Copy of data_permissions.schema_name}+ - column: {name: table_id, type: int, remarks: Copy of data_permissions.table_id}+ - column: {name: perm_value, type: varchar(64), remarks: Copy of data_permissions.perm_value}+ rollback: # not necessary; the collapsed representation is understood by all versions >= 49++ - changeSet:+ id: v62.2026-07-07T00:00:02+ author: ranquild+ comment: Stage the surviving data_permissions rows before collapsing the redundant ones (#76077)+ changes:+ - sql:+ sql: >-+ INSERT INTO data_permissions_collapse_keep (id, group_id, perm_type, db_id, schema_name, table_id, perm_value)+ SELECT dp.id, dp.group_id, dp.perm_type, dp.db_id, dp.schema_name, dp.table_id, dp.perm_value+ FROM data_permissions dp+ LEFT JOIN data_permissions dbp+ ON dp.perm_type = 'perms/view-data' AND dp.table_id IS NOT NULL+ AND dbp.group_id = dp.group_id AND dbp.db_id = dp.db_id+ AND dbp.perm_type = dp.perm_type AND dbp.perm_value = dp.perm_value+ AND dbp.table_id IS NULL+ WHERE dbp.id IS NULL;+ rollback: # not necessary; the collapsed representation is understood by all versions >= 49++ - changeSet:+ id: v62.2026-07-07T00:00:03+ author: ranquild+ comment: Empty data_permissions before reloading only the surviving rows (#76077)+ changes:+ - sql:+ sql: TRUNCATE TABLE data_permissions;+ rollback: # not necessary; the collapsed representation is understood by all versions >= 49++ - changeSet:+ id: v62.2026-07-07T00:00:04+ author: ranquild+ comment: Reload the surviving data_permissions rows (#76077)+ changes:+ - sql:+ sql: >-+ INSERT INTO data_permissions (id, group_id, perm_type, db_id, schema_name, table_id, perm_value)+ SELECT id, group_id, perm_type, db_id, schema_name, table_id, perm_value+ FROM data_permissions_collapse_keep;+ rollback: # not necessary; the collapsed representation is understood by all versions >= 49++ - changeSet:+ id: v62.2026-07-07T00:00:05+ author: ranquild+ comment: Drop the staging table used to collapse data_permissions (#76077)+ changes:+ - sql:+ sql: DROP TABLE data_permissions_collapse_keep;+ rollback: # not necessary; the collapsed representation is understood by all versions >= 49diff --git a/src/metabase/permissions/models/data_permissions.clj b/src/metabase/permissions/models/data_permissions.cljindex 89f1fb177f4..f12fff8a150 100644--- a/src/metabase/permissions/models/data_permissions.clj+++ b/src/metabase/permissions/models/data_permissions.clj@@ -1176,14 +1176,19 @@[{:keys [db-id db-level-idx all-db-tables batch-table-ids]}[[group-id perm-type] entries]](let [db-perm (get db-level-idx [group-id perm-type])- any-blocked? (and db-perm (some #(= :blocked (:actual-value %)) entries))+ ;; Only go granular when a batch table needs `:blocked` and the DB-level row has some *other*+ ;; value: a `:blocked` DB-level row already covers the new tables, and expanding it would write+ ;; one redundant row per table (see #76077, where this ballooned data_permissions to 46M rows).+ go-granular? (and db-perm+ (not= :blocked (:perm_value db-perm))+ (some #(= :blocked (:actual-value %)) entries))mk (fn [v table-id schema](mk-perm-row group-id perm-type v db-id table-id schema))](cond;; Going-granular fires once for the whole `(group, perm-type)`:;; delete the DB-level row, expand non-batch tables to the old value,;; then write each batch table at its actual-value.- any-blocked?+ go-granular?(let [expansion-value (case (:perm_value db-perm):query-builder-and-native :query-builder(:perm_value db-perm))]@@ -1229,8 +1234,8 @@(defn set-default-table-permissions!"Set default permissions for a newly-created table across all relevantgroups. Handles three cases per `(group, perm-type)`:- - Group has DB-level perm matching default → no-op- - Group has DB-level perm but new table needs `:blocked` → going-granular+ - Group has DB-level perm covering the default (including `:blocked` covering `:blocked`) → no-op+ - Group has non-`:blocked` DB-level perm but new table needs `:blocked` → going-granular- Group has no DB-level perm → simple insert`group-perm-defaults` is a seq of `{:group-id :perm-type :default-value}`diff --git a/test/metabase/app_db/schema_migrations_test.clj b/test/metabase/app_db/schema_migrations_test.cljindex 9dd4f82003d..3f0d427c763 100644--- a/test/metabase/app_db/schema_migrations_test.clj+++ b/test/metabase/app_db/schema_migrations_test.clj@@ -2888,6 +2888,94 @@:created_at :%now}attrs)))+(deftest collapse-uniform-table-permissions-migration-test+ (testing "Migrations v62.2026-07-07T00:00:00 through v62.2026-07-07T00:00:05:+ uniform full-coverage :blocked view-data table rows collapse to a db-level row (#76077)"+ (impl/test-migrations ["v62.2026-07-07T00:00:00" "v62.2026-07-07T00:00:05"] [migrate!]+ (let [group-id (t2/insert-returning-pk! :permissions_group {:name "collapse-test-group"})+ db! (fn [db-name]+ (t2/insert-returning-pk! :metabase_database+ {:details "{}"+ :created_at :%now+ :updated_at :%now+ :engine "h2"+ :is_sample false+ :name db-name}))+ table! (fn [db-id active?]+ (t2/insert-returning-pk! :metabase_table+ {:db_id db-id+ :name (mt/random-name)+ :schema "PUBLIC"+ :created_at :%now+ :updated_at :%now+ :active active?}))+ perm! (fn [db-id table-id perm-type perm-value]+ (t2/insert! :data_permissions+ {:group_id group-id+ :db_id db-id+ :table_id table-id+ :schema_name (when table-id "PUBLIC")+ :perm_type perm-type+ :perm_value perm-value}))+ rows (fn [db-id perm-type]+ (t2/select-fn-set (juxt :table_id :perm_value) :data_permissions+ :group_id group-id :db_id db-id :perm_type perm-type))+ ;; db-1: uniform :blocked covering all active tables (+ a :blocked row for an inactive table)+ ;; → collapses+ db-1 (db! "collapse-uniform")+ t1a (table! db-1 true)+ t1b (table! db-1 true)+ t1-off (table! db-1 false)+ ;; db-2: partial coverage → untouched+ db-2 (db! "collapse-partial")+ t2a (table! db-2 true)+ _t2b (table! db-2 true)+ ;; db-3: mixed values → untouched+ db-3 (db! "collapse-mixed")+ t3a (table! db-3 true)+ t3b (table! db-3 true)+ ;; db-4: uniform but :unrestricted → untouched+ db-4 (db! "collapse-unrestricted")+ t4a (table! db-4 true)+ ;; db-5: uniform :no create-queries (not view-data) → untouched+ db-5 (db! "collapse-other-type")+ t5a (table! db-5 true)+ ;; db-6: active tables uniformly :blocked, but an INACTIVE table's row differs → untouched;+ ;; the differing row must survive so a reactivated table picks its value back up+ db-6 (db! "collapse-inactive-differs")+ t6a (table! db-6 true)+ t6-off (table! db-6 false)+ ;; db-7: table row duplicating an equal-value db-level row → duplicate deleted+ db-7 (db! "collapse-dup-of-db-level")+ t7a (table! db-7 true)]+ (perm! db-1 t1a "perms/view-data" "blocked")+ (perm! db-1 t1b "perms/view-data" "blocked")+ (perm! db-1 t1-off "perms/view-data" "blocked")+ (perm! db-2 t2a "perms/view-data" "blocked")+ (perm! db-3 t3a "perms/view-data" "blocked")+ (perm! db-3 t3b "perms/view-data" "unrestricted")+ (perm! db-4 t4a "perms/view-data" "unrestricted")+ (perm! db-5 t5a "perms/create-queries" "no")+ (perm! db-6 t6a "perms/view-data" "blocked")+ (perm! db-6 t6-off "perms/view-data" "unrestricted")+ (perm! db-7 nil "perms/view-data" "unrestricted")+ (perm! db-7 t7a "perms/view-data" "unrestricted")+ (migrate!)+ (testing "uniform :blocked collapses to a single db-level row, dropping the inactive table's row too"+ (is (= #{[nil "blocked"]} (rows db-1 "perms/view-data"))))+ (testing "partial coverage untouched"+ (is (= #{[t2a "blocked"]} (rows db-2 "perms/view-data"))))+ (testing "mixed values untouched"+ (is (= #{[t3a "blocked"] [t3b "unrestricted"]} (rows db-3 "perms/view-data"))))+ (testing "uniform non-blocked value untouched"+ (is (= #{[t4a "unrestricted"]} (rows db-4 "perms/view-data"))))+ (testing "non-view-data perm types untouched"+ (is (= #{[t5a "no"]} (rows db-5 "perms/create-queries"))))+ (testing "a differing value on an inactive table's row blocks the collapse"+ (is (= #{[t6a "blocked"] [t6-off "unrestricted"]} (rows db-6 "perms/view-data"))))+ (testing "table rows duplicating an equal-value db-level row are deleted"+ (is (= #{[nil "unrestricted"]} (rows db-7 "perms/view-data"))))))))+(defn- collection-entity-id[collection-id](str/trim (t2/select-one-fn :entity_id :collection :id collection-id)))diff --git a/test/metabase/permissions/models/data_permissions_test.clj b/test/metabase/permissions/models/data_permissions_test.cljindex 86faaa7047a..4e9bb4b83ac 100644--- a/test/metabase/permissions/models/data_permissions_test.clj+++ b/test/metabase/permissions/models/data_permissions_test.clj@@ -896,6 +896,30 @@:group_id group-id :db_id db-id:table_id table-id-3 :perm_type :perms/view-data))))))))+(deftest set-default-table-permissions!-blocked-db-stays-collapsed-test+ (testing "A :blocked DB-level row already covers a new :blocked table, so we don't go granular (#76077)"+ (mt/with-temp [:model/Database {db-id :id} {}+ :model/PermissionsGroup {group-id :id} {}+ :model/Table {table-id-1 :id} {:db_id db-id :schema "PUBLIC" :active true}]+ (data-perms/set-database-permission! group-id db-id :perms/view-data :blocked)+ (mt/with-temp [:model/Table {table-id-2 :id} {:db_id db-id :schema "PUBLIC" :active true}]+ (t2/delete! :model/DataPermissions :group_id group-id :table_id table-id-2 :perm_type :perms/view-data)+ (mt/with-dynamic-fn-redefs [data-perms/new-table-view-data-permission-levels+ (fn [_db-id group-ids] (zipmap group-ids (repeat :blocked)))]+ (data-perms/set-default-table-permissions!+ {:id table-id-2 :db_id db-id :schema "PUBLIC"}+ [{:group-id group-id :perm-type :perms/view-data :default-value :unrestricted}]))+ (testing "the DB-level :blocked row remains and no per-table rows were written"+ (is (= :blocked (t2/select-one-fn :perm_value :model/DataPermissions+ :group_id group-id :db_id db-id :table_id nil+ :perm_type :perms/view-data)))+ (is (zero? (t2/count :model/DataPermissions+ :group_id group-id :db_id db-id :perm_type :perms/view-data+ :table_id [:not= nil]))))+ (testing "effective permission for the new table is still :blocked"+ (is (= :blocked (data-perms/table-permission-for-groups #{group-id} :perms/view-data db-id table-id-2)))+ (is (= :blocked (data-perms/table-permission-for-groups #{group-id} :perms/view-data db-id table-id-1))))))))+(deftest set-default-table-permissions!-simple-insert-test(testing "When group is already table-granular, new table gets a simple insert"(mt/with-temp [:model/Database {db-id :id} {}