Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Fixes from the RDE getting-started guide trials (#84117)

Alexander Polyankin0e5f03fc9e43
diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.edn
index 2dd5b1fc8cd..ae0589c33c2 100644
--- a/.clj-kondo/config/modules/config.edn
+++ b/.clj-kondo/config/modules/config.edn
@@ -3043,7 +3043,8 @@
:model/TableIndex
:model/Transform
:model/TransformRun}
- :uses #{database-routing
+ :uses #{app-db
+ database-routing
driver
events
lib
diff --git a/docs/data-modeling/models/actions/introduction.md b/docs/data-modeling/models/actions/introduction.md
index 793698f816b..3e59aafaa18 100644
--- a/docs/data-modeling/models/actions/introduction.md
+++ b/docs/data-modeling/models/actions/introduction.md
@@ -30,7 +30,7 @@ If you're an admin who just wants to edit data in your tables, check out [Editab
For actions to work, you'll first need to do the following two things:
-1. **Enable model actions for the database connection**. To enable actions for a database connection, admins should click on the **grid** icon in the upper right and navigate to **Admin** > **Databases**, then click on the database you want to create actions for. On the right side of the connection settings form, toggle the **Model actions** option. For actions to work, the database user account (the account you're using to connect to the database) must have [write permissions](../../../databases/users-roles-privileges.md#privileges-to-enable-actions-and-editable-table-data). And for now, actions are only supported on PostgreSQL and MySQL databases.
+1. **Enable data actions for the database connection**. To enable actions for a database connection, admins should click on the **grid** icon in the upper right and navigate to **Admin** > **Databases**, then click on the database you want to create actions for. On the right side of the connection settings form, toggle the **Data actions** option. For actions to work, the database user account (the account you're using to connect to the database) must have [write permissions](../../../databases/users-roles-privileges.md#privileges-to-enable-actions-and-editable-table-data). And for now, actions are only supported on PostgreSQL and MySQL databases.
2. **Create at least one model from that database.** Actions are associated with models, so you'll need to have created (or have access to) at least one model before you can start creating actions.
## Who can use actions
diff --git a/docs/data-modeling/semantic-layer/library.md b/docs/data-modeling/semantic-layer/library.md
index d2a9e399958..c42326b04fc 100644
--- a/docs/data-modeling/semantic-layer/library.md
+++ b/docs/data-modeling/semantic-layer/library.md
@@ -53,9 +53,9 @@ Before you can add items to the semantic layer, you'll need to create the Librar
1. Click the **grid** icon in the upper right and select **Data Studio**.
2. In the left sidebar, click **Semantic layer**.
-3. Click **Create my Library**.
+3. Click **Create my semantic layer**.
-Metabase will create the Library with its **Data** and **Metrics** collections. By default, everyone can view the Library, and people in the Data Analysts group can curate it. See [Library permissions](#library-permissions).
+Metabase will create the Library with its **Data**, **Metrics**, and **Dashboards** collections. By default, everyone can view the Library, and people in the Data Analysts group can curate it. See [Library permissions](#library-permissions).
If you try to [publish a table](published-tables.md) from **Connected data** before you've created the Library, Metabase will ask you to create the Library first.
@@ -71,6 +71,7 @@ The **+ New** menu has options to:
- **Published table**: [Publish a table](published-tables.md#publish-a-table-from-the-semantic-layer) to the Library.
- **Metric**: [Create a metric](#metrics) in the Library.
+- **Dashboard**: Create a dashboard in the Library's **Dashboards** collection.
- **Snippet**: [Create a SQL snippet](#sql-snippets).
- **Collection**: [Create a subcollection or snippet folder](#library-organization).
@@ -80,10 +81,11 @@ To manage the glossary, see [Glossary](glossary.md).
![Library organization](./images/library-org.png)
-The semantic layer lives in a special collection called Library, which has three root sections:
+The semantic layer lives in a special collection called Library, which has four root sections:
- **Data**: For [published tables](published-tables.md).
- **Metrics**: For [official metrics](#metrics).
+- **Dashboards**: For official dashboards. This collection holds only dashboards, so questions on these dashboards are saved to the dashboard itself rather than to a collection.
- **SQL snippets**: For all the [SQL snippets](#sql-snippets) on your instance.
These root sections are predefined. You can't rename or archive them, but you can use [permissions](#library-permissions) to control who sees them.
diff --git a/docs/databases/connections/mysql.md b/docs/databases/connections/mysql.md
index a8fa37131be..954bb24e3a9 100644
--- a/docs/databases/connections/mysql.md
+++ b/docs/databases/connections/mysql.md
@@ -195,9 +195,9 @@ Set up an additional connection used for write operations. See [Writable connect
Choose whether to enable features related to [Metabase models](../../data-modeling/models/models.md). These features will often require that the database user account, the one you use to connect to your database, has both read and write privileges.
-### Model actions
+### Data actions
-Turn this setting on to allow [actions](../../data-modeling/models/actions/introduction.md) from models created from this data to be run. Actions can read, write, and delete data. Your database user will need write permissions.
+Turn this setting on to allow [actions](../../data-modeling/models/actions/introduction.md) that use this database to be run. Actions can read, write, and delete data. Your database user will need write permissions.
### Model persistence
diff --git a/docs/databases/connections/postgresql.md b/docs/databases/connections/postgresql.md
index 77666e0cc78..2bf31dcc587 100644
--- a/docs/databases/connections/postgresql.md
+++ b/docs/databases/connections/postgresql.md
@@ -194,9 +194,9 @@ Set up an additional connection used for write operations. See [Writable connect
Choose whether to enable features related to [Metabase models](../../data-modeling/models/models.md). These features will often require that the database user account, the one you use to connect to your database, has both read and write privileges.
-### Model actions
+### Data actions
-Turn this setting on to allow [actions](../../data-modeling/models/actions/introduction.md) from models created from this data to be run. Actions can read, write, and delete data. Your database user will need write permissions.
+Turn this setting on to allow [actions](../../data-modeling/models/actions/introduction.md) that use this database to be run. Actions can read, write, and delete data. Your database user will need write permissions.
### Model persistence
diff --git a/docs/installation-and-operation/remote-sync.md b/docs/installation-and-operation/remote-sync.md
index cf8c6514aa2..bb99847f5fb 100644
--- a/docs/installation-and-operation/remote-sync.md
+++ b/docs/installation-and-operation/remote-sync.md
@@ -139,9 +139,10 @@ In the Metabase instance that you use for development:
- Paste the personal access token (PAT) you created earlier. Make sure the token has [read and write permissions](#2-create-a-token-for-your-read-and-write-metabase). Metabase encrypts your token before storing it.
-5. Save and test the connection:
+5. Test the connection:
- - Click "Save changes". Metabase will check whether it can reach your repository. If the connection fails, make sure your token has the appropriate permissions and hasn't expired. If you copied the token incorrectly, generate a new one.
+ - Click **Test connection** to check whether Metabase can reach your repository. If the connection fails, make sure your token has the appropriate permissions and hasn't expired. If you copied the token incorrectly, generate a new one.
+ - Metabase also checks the connection when you save your settings.
6. (Optional) If you have [multi-tenant user strategy enabled](../embedding/tenants.md#enable-multi-tenant-strategy), you can also choose which [shared collection](../embedding/tenants.md#changing-tenant-strategy) to sync.
@@ -201,9 +202,10 @@ In your production Metabase instance:
- Paste the read-only personal access token you created for this production Metabase.
-5. Save and test the connection:
+5. Test the connection:
- - Click "Save changes". Metabase will verify it can reach your repository. If the connection fails, verify your token has the appropriate permissions and hasn't expired.
+ - Click **Test connection** to check whether Metabase can reach your repository. If the connection fails, verify your token has the appropriate permissions and hasn't expired.
+ - Metabase also checks the connection when you save your settings.
6. Sync your content:
- Click "Pull changes now" to immediately sync content from your repository.
@@ -217,7 +219,9 @@ At this point, you should be all set up. Exit Admin, then reload your browser. Y
### 8. Configure transforms syncing (optional)
-To version your data transformation logic, you can sync your [Transforms](../data-modeling/transforms/transforms-overview.md), including all your tags and jobs. Some things to keep in mind:
+To version your data transformation logic, you can sync your [Transforms](../data-modeling/transforms/transforms-overview.md), including their tags and tests. Some things to keep in mind:
+
+- **Jobs don't sync.** Each Metabase keeps its own jobs. Tag your transforms with the built-in hourly, daily, weekly, or monthly tags so the matching built-in jobs run them on every Metabase, or create the jobs you need on each Metabase.
- **Transform syncing is all or nothing**: Metabase will sync your entire transforms namespace. You can't selectively sync specific transform folders.
- **This setting only determines whether Metabase pushes transforms from Read-write mode.** When you _pull_ from a repository, all content present in the repo is loaded—including any transforms—regardless of this setting. Think of it like pulling a repo that has a new collection you hadn't previously synced: the setting doesn't filter what comes in, only what goes out.
@@ -363,17 +367,22 @@ Remote Sync uses the same serialization format as the [Metabase CLI serializatio
- Documents
- Timelines and events
- Collection structure and metadata
-- Library content (published tables, metrics, snippets, segments, measures, glossary)
-- Transforms (including jobs and folders)
+- Library content (published tables, metrics, dashboards, snippets, segments, measures, glossary)
+- Transforms (including tags, tests, and folders)
**What doesn't sync:**
- Users, groups, and permissions
- Alerts and subscriptions
-- Database connections
+- Database connections (see [Synced content that uses a database your Metabase doesn't have](#synced-content-that-uses-a-database-your-metabase-doesnt-have))
+- Transform jobs
- Personal collections
- Table metadata (table names, column types, descriptions, visibility settings, etc.)
+### Synced content that uses a database your Metabase doesn't have
+
+Synced content points to databases by their display name, so give the database the same name on your development and production Metabases. If a pull brings in content that uses a database your Metabase doesn't have, Metabase creates a placeholder for that database, without connection details. In **Admin** > **Databases**, the placeholder's status is **Stubbed**. Questions and transforms that use the database can't run until an admin opens the stubbed database and enters its connection details. Once you save the connection details, the status changes to **Active**.
+
## Branch management
Branching is only available in Read-write mode.
diff --git a/enterprise/backend/src/metabase_enterprise/data_apps/resource_validation.clj b/enterprise/backend/src/metabase_enterprise/data_apps/resource_validation.clj
index 0ff0dcae478..e1a1ba113f8 100644
--- a/enterprise/backend/src/metabase_enterprise/data_apps/resource_validation.clj
+++ b/enterprise/backend/src/metabase_enterprise/data_apps/resource_validation.clj
@@ -4,7 +4,7 @@
reads: it updates whatever row carries an entity ID and resolves references to any local entity. These checks run
on the ingested files first, so an app's resources can only be its own collection, in the `data-apps` namespace,
and, in it, questions, metrics, and query actions that belong to no model, referencing nothing else of Metabase's
- but what already exists."
+ but what already exists, apart from the databases, tables and fields that a load stubs as it does for any card."
(:require
[clojure.string :as str]
[clojure.walk :as walk]
@@ -16,7 +16,8 @@
(set! *warn-on-reflection* true)
(def ^:private external-dependency-models
- "Models a resource may reference outside the app. They must already exist: a load never creates them."
+ "Models a resource may reference outside the app. A snippet, segment or measure must already exist or come with the
+ same pull; a database, table or field is never a dependency, as a load stubs one that is missing."
#{"Database" "Table" "Field" "NativeQuerySnippet" "Segment" "Measure"})
(def ^:private card-types #{"question" "metric"})
@@ -196,33 +197,26 @@
(select-keys entity [:dataset_query :query :parameter_mappings :result_metadata]))
@refs))
-(defn- missing-table-and-field-problems
- "Serialization creates an inactive placeholder for a table or field a query names that this instance doesn't
- have. A data app's query would then run against nothing, so the file is refused instead."
- [{:keys [path] :as file}]
- (let [table-id (fn [[db-name schema table-name]]
- (some-> (models.db/database-id-by-name db-name)
- (as-> database-id (models.db/table-id-by-name table-name schema database-id))
- ;; an inactive table is one that sync no longer finds, or a placeholder
- (as-> id (when (data-apps.db/active-table? id) id))))]
- (for [[kind [db-name schema table-name & field-names :as ref]] (sort-by second (portable-refs (:entity file)))
- :when (or (nil? (table-id [db-name schema table-name]))
- (and (= kind :field)
- (nil? (models.db/field-pk-in-path (table-id [db-name schema table-name]) (reverse field-names)))))]
- (problem path (tru "{0} references {1} {2}, which does not exist on this instance."
- path (if (= kind :table) "table" "field") (pr-str ref))))))
-
-(defn- inactive-field-warnings
- "A warning for each inactive field `file` references: one sync no longer finds, which the app's query would run
- against. One column isn't the app, so the file loads with the field logged, unlike an inactive table."
+(defn- table-and-field-warnings
+ "A warning for each table or field `file` references that this instance lacks or that is inactive, which the load
+ resolves as it does for any card, through a stub database and inactive placeholders that a sync later fills in."
[{:keys [path] :as file}]
(for [[kind [db-name schema table-name & field-names :as ref]] (sort-by second (portable-refs (:entity file)))
- :when (= kind :field)
- :let [field-id (some-> (models.db/database-id-by-name db-name)
- (as-> database-id (models.db/table-id-by-name table-name schema database-id))
- (models.db/field-pk-in-path (reverse field-names)))]
- :when (and field-id (not (data-apps.db/active-field? field-id)))]
- (problem path (tru "{0} references field {1}, which is inactive on this instance." path (pr-str ref)))))
+ :let [table-id (some->> (models.db/database-id-by-name db-name)
+ (models.db/table-id-by-name table-name schema))
+ id (if (= kind :table)
+ table-id
+ (some-> table-id (models.db/field-pk-in-path (reverse field-names))))
+ label (if (= kind :table) "table" "field")
+ message (cond
+ (nil? id)
+ (tru "{0} references {1} {2}, which does not exist on this instance yet, so the app''s queries fail until it is synced."
+ path label (pr-str ref))
+
+ (not (if (= kind :table) (data-apps.db/active-table? id) (data-apps.db/active-field? id)))
+ (tru "{0} references {1} {2}, which is inactive on this instance." path label (pr-str ref)))]
+ :when message]
+ (problem path message)))
(defn- ownership-problems
"Serdes would update any row carrying an entity ID a file names, so a file may only name what this app owns:
@@ -322,8 +316,7 @@
(mapcat numeric-reference-problems resources)
(mapcat (partial dependency-problems collection-entity-id card-entity-ids) resources)
(ownership-problems app-entity-id (:path manifest) collection-entity-id resources)
- (external-dependency-problems defined resources)
- (mapcat missing-table-and-field-problems resources)))))))
+ (external-dependency-problems defined resources)))))))
(defn- shared-collection-problems
"Two apps can't name one collection: only one of them would own it."
@@ -412,7 +405,7 @@
[files]
(for [{:keys [entity]} (manifests-among files)
resource (app-resources (:collection entity) files)
- warning (inactive-field-warnings resource)]
+ warning (table-and-field-warnings resource)]
warning))
(defn problems
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj
index 3d56fb3daa0..796f6fca94d 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj
@@ -1470,6 +1470,12 @@
;;; ------------------------------------------- Task Result Handling -------------------------------------------
+(defn- outcome-on-branch
+ "Names `branch`, the branch the task ran on, in an `outcome` that names one, since the setting the outcome was built from switches to it only after the task."
+ [outcome branch]
+ (cond-> outcome
+ (and branch (contains? outcome :branch)) (assoc :branch branch)))
+
(defn handle-task-result!
"Handles the outcome of running import! or export! by updating the RemoteSyncTask record.
@@ -1506,7 +1512,7 @@
:success (do
(when branch
(settings/remote-sync-branch! branch))
- (remote-sync.task/complete-sync-task! task-id (:outcome result)))
+ (remote-sync.task/complete-sync-task! task-id (outcome-on-branch (:outcome result) branch)))
:conflict (do
(remote-sync.task/set-version! task-id (:version result))
(remote-sync.task/conflict-sync-task! task-id (:conflicts result) (:outcome result)))
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj
index a50e9d8fe56..193cdf9aa59 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj
@@ -103,40 +103,33 @@
Used to represent the entire transforms feature being enabled/disabled."
-1)
+(defn- replace-transforms-root-rso!
+ "Replaces the Transforms root RSO with one in `status`."
+ [status]
+ (remote-sync.db/delete-rso-of! "Collection" transforms-root-id)
+ (remote-sync.db/insert-rso! {:model_type "Collection"
+ :model_id transforms-root-id
+ :model_name "Transforms"
+ :status status
+ :status_changed_at (t/offset-date-time)}))
+
(defn sync-transform-tracking!
- "Called when remote-sync-transforms setting changes.
- Creates a single 'Transforms' RSO entry with model_id=-1 as a sentinel value.
- When enabled: status is 'create' to indicate transforms should be synced.
- When disabled: status is 'delete' to indicate transforms should be removed.
- When disabling and there's no existing Transforms RSO, does nothing (avoids creating
- spurious 'delete' entries when going from default false to explicitly false)."
+ "Records transforms sync state as the Transforms root RSO (model_id -1): 'create' when newly enabled, 'delete'
+ when disabled after being tracked, left alone when already in that state (the settings cache also calls this on
+ every load), and absent on a read-only instance, which has nothing to publish."
[enabled?]
- (let [timestamp (t/offset-date-time)
- existing-rso (remote-sync.db/rso "Collection" transforms-root-id)]
+ (let [status (:status (remote-sync.db/rso "Collection" transforms-root-id))]
(cond
- ;; When enabling, always create/update to 'create' status
+ (= :read-only (remote-sync-type))
+ (when status
+ (remote-sync.db/delete-rso-of! "Collection" transforms-root-id))
+
enabled?
- (do
- (when existing-rso
- (remote-sync.db/delete-rso-of! "Collection" transforms-root-id))
- (remote-sync.db/insert-rso! {:model_type "Collection"
- :model_id transforms-root-id
- :model_name "Transforms"
- :status "create"
- :status_changed_at timestamp}))
- ;; When disabling and there's an existing RSO, update to 'delete' status
- existing-rso
- (do
- (remote-sync.db/delete-rso-of! "Collection" transforms-root-id)
- (remote-sync.db/insert-rso! {:model_type "Collection"
- :model_id transforms-root-id
- :model_name "Transforms"
- :status "delete"
- :status_changed_at timestamp}))
- ;; When disabling and there's no existing RSO, do nothing
- ;; (this avoids creating spurious 'delete' entries when going from default false to explicitly false)
- :else
- nil)))
+ (when-not (#{"create" "synced"} status)
+ (replace-transforms-root-rso! "create"))
+
+ (and status (not= "delete" status))
+ (replace-transforms-root-rso! "delete"))))
(defn- sync-transform-tracking-on-change
"Called when remote-sync-transforms setting changes."
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj
index 4298d178e00..4a931ce058b 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj
@@ -583,14 +583,15 @@
(defn- has-unsynced-entities-for-feature?
"Returns true if any model in the feature group has local entities not tracked in RemoteSyncObject.
- Excludes entities filtered by export-conditions (e.g., built-in TransformTags) from the count since
- they are system-created and not user data. Namespace collections are not checked here because they are
- organizational containers, not user data that would be lost on import."
+ Excludes entities filtered by removal-conditions (e.g., built-in TransformTags and the built-in common.py
+ PythonLibrary every instance gets from a migration) from the count since they are system-created and not
+ user data. Namespace collections are not checked here because they are organizational containers, not
+ user data that would be lost on import."
[specs-for-feature]
(some (fn [[_ spec]]
(let [model-key (:model-key spec)
model-type (:model-type spec)
- conditions (export-conditions spec)
+ conditions (removal-conditions spec)
local-count (remote-sync.db/count-where model-key conditions)
synced-count (remote-sync.db/rso-count-of-type model-type)]
(and (pos? local-count)
diff --git a/enterprise/backend/test/metabase_enterprise/data_apps/resource_validation_test.clj b/enterprise/backend/test/metabase_enterprise/data_apps/resource_validation_test.clj
index f96b3afb2e6..b006859d092 100644
--- a/enterprise/backend/test/metabase_enterprise/data_apps/resource_validation_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/data_apps/resource_validation_test.clj
@@ -58,6 +58,9 @@
(defn- messages [tree]
(mapv :message (data-apps/problems (files tree))))
+(defn- warnings [tree]
+ (mapv :message (data-apps/warnings (files tree))))
+
(defn- first-problem [tree]
(first (data-apps/problems (files tree))))
@@ -108,10 +111,6 @@
(testing "a file whose serdes/meta doesn't identify what it holds"
(refused (edit-file (shop (question-resources)) question-path #(assoc % :serdes/meta [{:model "Card" :id "someOtherEntityId0001"}]))
"must hold a single Card whose serdes/meta ID is its entity_id"))
- (testing "a table that doesn't exist"
- (refused (edit-file (shop (question-resources)) question-path
- #(assoc-in % [:dataset_query :stages 0 :source-table] [(:name (mt/db)) "PUBLIC" "NO_SUCH_TABLE"]))
- "NO_SUCH_TABLE"))
(testing "a manifest that doesn't identify the app, which a load would refuse"
(refused (edit-file (shop nil) "data_apps/shop/data_app.yaml" #(dissoc % :entity_id))
"must hold a single DataApp whose serdes/meta ID is its entity_id")
@@ -170,8 +169,8 @@
:model/Card _ {:name "Mine" :entity_id question-eid :collection_id collection-id}]
(is (= [] (messages (shop (question-resources)))))))))
-(deftest refuses-a-field-that-doesnt-exist-test
- (testing "a field a query names has to exist, on a database without schemas too"
+(deftest warns-of-a-field-that-doesnt-exist-test
+ (testing "a field a query names that doesn't exist loads as a placeholder and is logged, on a database without schemas too"
(mt/with-temp [:model/Database {db-id :id} {:engine :h2 :name "no-schemas"}
:model/Table {table-id :id} {:db_id db-id :name "T" :schema nil :active true}
:model/Field _ {:table_id table-id :name "ID" :base_type :type/Integer}]
@@ -183,17 +182,19 @@
:stages [{:lib/type "mbql.stage/mbql"
:source-table [db schema table]
:filters [["=" {} ["field" {} [db schema table field]] 1]]}]})))]
- (is (some #(str/includes? % "NOPE") (messages (filtering-on "no-schemas" nil "T" "NOPE"))))
- (is (= [] (messages (filtering-on "no-schemas" nil "T" "ID"))))
- (is (some #(str/includes? % "NOPE") (messages (filtering-on (:name (mt/db)) "PUBLIC" "VENUES" "NOPE"))))))))
-
-(deftest refuses-a-table-or-field-named-in-any-reference-test
- (testing "serialization makes a placeholder for a missing table or field wherever a file references one"
+ (is (= [] (messages (filtering-on "no-schemas" nil "T" "NOPE"))))
+ (is (some #(str/includes? % "NOPE") (warnings (filtering-on "no-schemas" nil "T" "NOPE"))))
+ (is (= [] (warnings (filtering-on "no-schemas" nil "T" "ID"))))
+ (is (some #(str/includes? % "NOPE") (warnings (filtering-on (:name (mt/db)) "PUBLIC" "VENUES" "NOPE"))))))))
+
+(deftest warns-of-a-table-or-field-named-in-any-reference-test
+ (testing "serialization makes a placeholder for a missing table or field wherever a file references one, and the pull
+ logs it"
(let [db (:name (mt/db))
refused (fn [missing f]
- (is (some #(str/includes? % missing)
- (messages (edit-file (shop (question-resources)) question-path f)))
- missing))]
+ (let [tree (edit-file (shop (question-resources)) question-path f)]
+ (is (= [] (messages tree)) missing)
+ (is (some #(str/includes? % missing) (warnings tree)) missing)))]
(testing "result metadata"
(refused "NOPE_META" #(assoc % :result_metadata [{:name "NOPE"
:base_type "type/Text"
@@ -217,10 +218,22 @@
:id "7f2c2a0e-6c1e-4b53-9d0a-0d5a3a1d1c12"
:table-id [db "PUBLIC" "NOPE_TAG_TABLE"]}}}]})))
(testing "a list of strings that isn't a reference is left alone, even when it starts with a database's name"
- (is (= [] (messages (edit-file (shop (question-resources)) question-path
+ (is (= [] (warnings (edit-file (shop (question-resources)) question-path
#(assoc % :visualization_settings {:some.custom/labels [db "Alpha" "Beta"]}))))))
- (testing "and a file that names only what exists has no problems"
- (is (= [] (messages (shop (question-resources)))))))))
+ (testing "and a file that names only what exists has nothing to log"
+ (is (= [] (warnings (shop (question-resources)))))))))
+
+(deftest accepts-a-database-the-instance-does-not-have-yet-test
+ (testing "a query on a database that isn't connected yet loads as a card's does, against a stub database that a
+ later connection and sync fill in"
+ (let [tree (edit-file (shop (question-resources)) question-path
+ #(-> %
+ (assoc :database_id "Not connected yet")
+ (assoc-in [:dataset_query :database] "Not connected yet")
+ (assoc-in [:dataset_query :stages 0 :source-table] ["Not connected yet" "analytics" "accounts"])))]
+ (is (= [] (messages tree)))
+ (is (some #(str/includes? % "references table [\"Not connected yet\" \"analytics\" \"accounts\"], which does not exist")
+ (warnings tree))))))
(deftest refuses-more-of-what-a-resource-may-not-hold-test
(let [refused (fn [tree message]
@@ -383,11 +396,12 @@
(is (some #(str/includes? % "already exists outside this data app's collection")
(messages (shop (question-resources))))))))))
-(deftest refuses-a-table-that-is-inactive-test
- (testing "an inactive table is one that sync no longer finds, so the app's query would run against nothing"
+(deftest warns-of-a-table-that-is-inactive-test
+ (testing "an inactive table is one that sync no longer finds, or a placeholder a load made, so the pull logs it"
(let [resources (question-resources)]
(mt/with-temp-vals-in-db :model/Table (mt/id :venues) {:active false}
- (is (some #(str/includes? % "references table [") (messages (shop resources))))))))
+ (is (= [] (messages (shop resources))))
+ (is (some #(str/includes? % "references table [") (warnings (shop resources))))))))
(deftest warns-of-a-field-that-is-inactive-test
(testing "an inactive field is one that sync no longer finds: one column isn't the app, so the file loads and the
@@ -397,8 +411,7 @@
[{:entity_id question-eid :name "VenuePrices"
:query {:stages [{:source {:type "table" :id (mt/id :venues)}
:fields [{:type "column" :name "PRICE"}]}]}}]
- [])
- warnings (fn [tree] (mapv :message (data-apps/warnings (files tree))))]
+ [])]
(mt/with-temp-vals-in-db :model/Field (mt/id :venues :price) {:active false}
(is (= [] (messages (shop resources))))
(is (some #(str/includes? % "references field") (warnings (shop resources)))))
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/data_apps_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/data_apps_test.clj
index ce60183368e..e44a94e268e 100644
--- a/enterprise/backend/test/metabase_enterprise/remote_sync/data_apps_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/data_apps_test.clj
@@ -673,6 +673,31 @@
:current "v0")]
(is (= :error (:status (import-at! src "v0" :force? true)))))))))))
+(deftest pull-loads-a-resource-on-a-database-not-connected-yet-test
+ (testing "a resource whose database isn't connected yet loads as a card's does, against a stub database and an
+ inactive table that a later connection and sync fill in"
+ (with-data-apps-sync
+ (mt/with-model-cleanup [:model/Database]
+ (let [db-name "Data app warehouse not connected yet"
+ resources (question-resources)
+ path (some #(when (str/includes? % question-eid) %) (keys resources))
+ elsewhere (-> (yaml/parse-string (get resources path))
+ (assoc :database_id db-name)
+ (dissoc :result_metadata)
+ (assoc-in [:dataset_query :database] db-name)
+ (assoc-in [:dataset_query :stages 0 :source-table] [db-name "analytics" "accounts"]))
+ src (test-helpers/versioned-source
+ :trees {"v0" (shop-tree (assoc resources path (yaml/generate-string elsewhere)))}
+ :current "v0")
+ result (import-at! src "v0" :force? true)
+ db-id (t2/select-one-pk :model/Database :name db-name)
+ table-id (t2/select-one-pk :model/Table :db_id db-id :schema "analytics" :name "accounts")]
+ (is (= :success (:status result)) (:message result))
+ (is (=? {:is_stub true} (t2/select-one :model/Database :id db-id)))
+ (is (=? {:active false} (t2/select-one :model/Table :id table-id)))
+ (is (=? {:database_id db-id :collection_id (shop-collection-id)}
+ (t2/select-one :model/Card :entity_id question-eid))))))))
+
(deftest pull-records-the-table-a-native-query-names-test
(testing "a native query's table is matched by name whatever its case"
(with-data-apps-sync
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj
index 3782885db4e..9d717a1eb09 100644
--- a/enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj
@@ -237,6 +237,20 @@
(is (= {:kind "pulled" :count 3 :branch "main"} (:outcome task)))
(is (= :successful (:status (t2/hydrate task :status)))))))))
+(deftest handle-task-result!-names-the-task-branch-in-outcome-test
+ (testing "an import that switches branches records the branch it imported, not the one the setting held while it ran"
+ (mt/with-temporary-setting-values [remote-sync-branch "main"]
+ (mt/with-model-cleanup [:model/RemoteSyncTask]
+ (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask
+ {:sync_task_type "import"
+ :initiated_by (mt/user->id :rasta)})]
+ (impl/handle-task-result! {:status :success
+ :outcome {:kind "pulled" :count 3 :branch "main"}}
+ task-id
+ "feature")
+ (is (= {:kind "pulled" :count 3 :branch "feature"}
+ (:outcome (t2/select-one :model/RemoteSyncTask :id task-id)))))))))
+
(deftest handle-task-result!-stores-conflict-categories-test
(testing "handle-task-result! records a conflict result's category set on the task as a vector"
(mt/with-model-cleanup [:model/RemoteSyncTask]
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/transforms_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/transforms_test.clj
index 7f1ab446289..153e000c75a 100644
--- a/enterprise/backend/test/metabase_enterprise/remote_sync/transforms_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/transforms_test.clj
@@ -158,7 +158,8 @@
(deftest enable-transform-sync-marks-all-transforms-test
(testing "Enabling transform sync creates a single Transforms RSO with model_id=-1 and status 'create'"
(mt/with-premium-features #{:transforms-basic}
- (mt/with-temporary-setting-values [remote-sync-enabled true]
+ (mt/with-temporary-setting-values [remote-sync-enabled true
+ remote-sync-type :read-write]
;; Clean up any existing Transforms root RSO first
(t2/delete! :model/RemoteSyncObject
:model_type "Collection"
@@ -186,7 +187,8 @@
(testing "Disabling transform sync creates a Transforms RSO with status 'delete'"
(mt/with-premium-features #{:transforms-basic}
(mt/with-temporary-setting-values [remote-sync-transforms false
- remote-sync-enabled true]
+ remote-sync-enabled true
+ remote-sync-type :read-write]
(mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection" :namespace collection/transforms-ns}
:model/Transform _transform {:name "Test Transform" :collection_id coll-id}
:model/TransformTag _tag {:name "Test Tag"}]
@@ -203,6 +205,51 @@
:status "delete")
"Transforms root should now have 'delete' status")))))))
+(defn- transforms-root-status
+ "The status of the Transforms root RSO, or nil when there is none."
+ []
+ (t2/select-one-fn :status :model/RemoteSyncObject :model_type "Collection" :model_id settings/transforms-root-id))
+
+(deftest transform-tracking-keeps-tracked-root-test
+ (testing "Re-applying an enabled transforms setting, as every settings cache load does, keeps a synced root synced"
+ (mt/with-temporary-setting-values [remote-sync-type :read-write]
+ (settings/sync-transform-tracking! true)
+ (t2/update! :model/RemoteSyncObject {:model_type "Collection" :model_id settings/transforms-root-id}
+ {:status "synced"})
+ (settings/sync-transform-tracking! true)
+ (is (= "synced" (transforms-root-status)))
+ (testing "and disabling then re-enabling marks it again"
+ (settings/sync-transform-tracking! false)
+ (is (= "delete" (transforms-root-status)))
+ (settings/sync-transform-tracking! true)
+ (is (= "create" (transforms-root-status)))))))
+
+(deftest transform-tracking-read-only-test
+ (testing "A read-only instance never tracks the Transforms root, so it never reports it as unsaved"
+ (mt/with-temporary-setting-values [remote-sync-type :read-only]
+ (settings/sync-transform-tracking! true)
+ (is (nil? (transforms-root-status)))
+ (is (not (sync-object/dirty?)))
+ (testing "and drops one left over from before the fix"
+ (t2/insert! :model/RemoteSyncObject {:model_type "Collection" :model_id settings/transforms-root-id
+ :model_name "Transforms" :status "create"
+ :status_changed_at (t/offset-date-time)})
+ (settings/sync-transform-tracking! true)
+ (is (nil? (transforms-root-status)))))))
+
+(deftest switching-to-read-only-with-transforms-test
+ (testing "Saving read-only settings with transforms on, as the setup guide does, leaves nothing unsaved"
+ (mt/with-premium-features #{:remote-sync}
+ (mt/with-temporary-setting-values [remote-sync-type :read-write
+ remote-sync-transforms false]
+ (mt/with-dynamic-fn-redefs [settings/check-git-settings! (constantly nil)
+ impl/finish-remote-config! (constantly nil)]
+ (mt/user-http-request :crowberto :put 200 "ee/remote-sync/settings"
+ {:remote-sync-type :read-only :remote-sync-transforms true})
+ (is (true? (settings/remote-sync-transforms)))
+ (is (nil? (transforms-root-status)))
+ (is (not (sync-object/dirty?))))))))
+
(deftest transforms-included-in-dirty-check-when-enabled-test
(testing "Transforms are included in dirty check when setting is enabled"
(mt/with-premium-features #{:transforms-basic}
@@ -733,6 +780,28 @@ serdes/meta:
(is (t2/exists? :model/PythonLibrary :entity_id transforms-python/builtin-entity-id)
"Built-in PythonLibrary should NOT be deleted after import"))))))))
+(deftest first-import-does-not-conflict-on-builtin-python-library-test
+ (testing "A fresh instance's built-in common.py does not block a first import that contains common.py"
+ (mt/with-premium-features #{:transforms-basic}
+ (doseq [transforms-synced? [false true]]
+ (testing (str "remote-sync-transforms " transforms-synced?)
+ (mt/with-temporary-setting-values [remote-sync-transforms transforms-synced?
+ remote-sync-enabled true]
+ (mt/with-model-cleanup [:model/RemoteSyncTask :model/PythonLibrary]
+ (t2/insert! :model/PythonLibrary {:path "common.py"
+ :source ""
+ :entity_id transforms-python/builtin-entity-id})
+ (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "import"
+ :initiated_by (mt/user->id :rasta)})
+ test-files {"main" {"python_libraries/common.py.yaml"
+ (generate-python-library-yaml transforms-python/builtin-entity-id
+ "common.py"
+ "def shared_func():\n return 42")}}
+ result (impl/import! (source.p/snapshot (test-helpers/create-mock-source :initial-files test-files))
+ task-id)]
+ (is (= :success (:status result))
+ (str "First import should not conflict. Result: " result))))))))))
+
(deftest import-replaces-python-library-with-remote-version-test
(testing "Import updates local PythonLibrary when remote has same entity_id"
(mt/with-premium-features #{:transforms-basic}
diff --git a/enterprise/backend/test/metabase_enterprise/serialization/v2/load_test.clj b/enterprise/backend/test/metabase_enterprise/serialization/v2/load_test.clj
index 2980feeb8cf..20171203b05 100644
--- a/enterprise/backend/test/metabase_enterprise/serialization/v2/load_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/serialization/v2/load_test.clj
@@ -17,6 +17,7 @@
[metabase.models.serialization :as serdes]
[metabase.permissions.core :as perms]
[metabase.search.core :as search]
+ [metabase.sync.core :as sync]
[metabase.test :as mt]
[metabase.util :as u]
[metabase.util.humanization :as u.humanization]
@@ -248,6 +249,46 @@
(is (= ["Email"] (t2/select-fn-vec :name :model/Dimension
:field_id (t2/select-one-pk :model/Field :table_id (:id table) :name "email"))))))))))
+(deftest field-user-settings-survive-connecting-a-stub-database-test
+ (testing "Field settings imported onto a stub database survive connecting it and syncing, and export again"
+ (let [db-name (mt/random-name)
+ extract #(into [] (serdes/extract-all "FieldUserSettings" {:filter-column :field_id :filter-ids %}))
+ serialized (mt/with-temp [:model/Database {db-id :id} {:name db-name}
+ :model/Table {table-id :id} {:db_id db-id :schema "PUBLIC" :name "VENUES"}
+ :model/Field {price-id :id} {:table_id table-id :name "PRICE"}
+ :model/Field {name-id :id} {:table_id table-id :name "NAME"}]
+ (t2/insert! :model/FieldUserSettings {:field_id price-id
+ :semantic_type :type/Currency
+ :effective_type :type/Integer
+ :description "In dollars"})
+ (t2/insert! :model/FieldUserSettings {:field_id name-id :semantic_type :type/City})
+ (extract [price-id name-id]))]
+ (try
+ (serdes.load/load-metabase! (ingestion-in-memory serialized))
+ (let [stub (t2/select-one :model/Database :name db-name)
+ table-id (t2/select-one-pk :model/Table :db_id (:id stub) :schema "PUBLIC" :name "VENUES")
+ field-ids (t2/select-pks-vec :model/Field :table_id table-id {:order-by [[:name :asc]]})
+ settings #(t2/select-fn->fn :field_id (juxt :semantic_type :semantic_type_set :effective_type)
+ :model/FieldUserSettings :field_id [:in field-ids])]
+ (is (true? (:is_stub stub)))
+ (is (= {:type/Currency true :type/City true}
+ (into {} (map (juxt first second)) (vals (settings)))))
+ (t2/update! :model/Database (:id stub) {:engine :h2 :details (:details (mt/db)) :is_stub false})
+ (sync/sync-db-metadata! (t2/select-one :model/Database :id (:id stub)))
+ (testing "sync reactivates the stub Fields and gives them their real base types"
+ (is (=? [{:name "NAME" :active true :base_type :type/Text}
+ {:name "PRICE" :active true :base_type :type/Integer}]
+ (t2/select [:model/Field :name :active :base_type] :id [:in field-ids] {:order-by [[:name :asc]]}))))
+ (testing "the imported user settings are kept"
+ (is (= #{[:type/Currency true :type/Integer] [:type/City true nil]}
+ (set (vals (settings))))))
+ (testing "and a later export still carries them"
+ (is (=? [{:semantic_type :type/City :semantic_type_set true}
+ {:semantic_type :type/Currency :semantic_type_set true :description "In dollars"}]
+ (sort-by (comp str :semantic_type) (extract field-ids))))))
+ (finally
+ (t2/delete! :model/Database :name db-name))))))
+
(deftest stub-database-survives-a-rolled-back-referrer-test
(testing "A stub created inside a failed entity's transaction is not reused from the cache by the next entity"
(let [serialized (atom nil)]
diff --git a/src/metabase/sync/sync_metadata/fields/sync_metadata.clj b/src/metabase/sync/sync_metadata/fields/sync_metadata.clj
index 43d3de9a1c5..3d4180d6b37 100644
--- a/src/metabase/sync/sync_metadata/fields/sync_metadata.clj
+++ b/src/metabase/sync/sync_metadata/fields/sync_metadata.clj
@@ -127,8 +127,9 @@
(common/field-metadata-name-for-logging table metabase-field)
old-base-type
new-base-type)
- (field-user-settings/unset-user-settings!
- (select-keys metabase-field [:id]) [:effective_type :coercion_strategy :semantic_type])
+ (when-not (= old-base-type :type/*)
+ (field-user-settings/unset-user-settings!
+ (select-keys metabase-field [:id]) [:effective_type :coercion_strategy :semantic_type]))
{:base_type new-base-type
:effective_type new-base-type
:coercion_strategy nil
diff --git a/src/metabase/transforms/util.clj b/src/metabase/transforms/util.clj
index d767cdb9c11..e6436e95be3 100644
--- a/src/metabase/transforms/util.clj
+++ b/src/metabase/transforms/util.clj
@@ -8,7 +8,6 @@
[clojure.core.async :as a]
[clojure.string :as str]
[metabase.api.common :as api]
- [metabase.driver :as driver]
[metabase.driver.connection :as driver.conn]
[metabase.driver.settings :as driver.settings]
[metabase.driver.sql-jdbc :as sql-jdbc]
@@ -190,9 +189,7 @@
reliable-row-count? (or (driver.u/supports? driver :transforms/accurate-rows-affected
{:lib/type :metadata/database :id db-id})
(not full-incremental?))]
- (when (and (not (str/blank? output-schema))
- (not (driver/schema-exists? driver db-id output-schema)))
- (driver/create-schema-if-needed! driver conn-spec output-schema))
+ (transforms-base.u/create-target-schema! driver db-id conn-spec output-schema)
(transforms-base.u/save-run-checkpoint-range! run-id source-range-params)
(when-let [{:keys [rows-available] :as srp} source-range-params]
(tracing/add-span-attrs! :tasks
diff --git a/src/metabase/transforms_base/query.clj b/src/metabase/transforms_base/query.clj
index c00700c8ff0..261c003dff1 100644
--- a/src/metabase/transforms_base/query.clj
+++ b/src/metabase/transforms_base/query.clj
@@ -4,7 +4,6 @@
This namespace handles MBQL/native query transform execution and returns
results in memory rather than writing to transform_run rows."
(:require
- [clojure.string :as str]
[metabase.driver :as driver]
[metabase.driver.util :as driver.u]
[metabase.indexes.schema :as indexes.schema]
@@ -129,10 +128,7 @@
(throw (ex-info "The database does not support the requested transform target type."
{:driver driver, :database database, :features features})))
(log/info "Executing transform" id "with target" (pr-str target))
- ;; Create schema if needed
- (when (and (not (str/blank? (:schema target)))
- (not (driver/schema-exists? driver db (:schema target))))
- (driver/create-schema-if-needed! driver (:conn-spec transform-details) (:schema target)))
+ (transforms-base.u/create-target-schema! driver db (:conn-spec transform-details) (:schema target))
;; Check cancellation before running query
(when (and cancelled? (cancelled?))
(throw (ex-info "Transform cancelled before query execution" {:status :cancelled})))
diff --git a/src/metabase/transforms_base/util.clj b/src/metabase/transforms_base/util.clj
index 709fa4089e6..a96803cf2cf 100644
--- a/src/metabase/transforms_base/util.clj
+++ b/src/metabase/transforms_base/util.clj
@@ -6,6 +6,7 @@
(:require
[clojure.string :as str]
[java-time.api :as t]
+ [metabase.app-db.cluster-lock :as cluster-lock]
[metabase.database-routing.core :as database-routing]
[metabase.driver :as driver]
[metabase.events.core :as events]
@@ -54,6 +55,15 @@
[m t]
(= t (keyword (:type m))))
+(defn create-target-schema!
+ "Creates a transform's target `schema` on database `db-id` when it doesn't exist, under a cluster lock per database so concurrent transforms targeting a new schema don't race to create it."
+ [driver db-id conn-spec schema]
+ (when (and (not (str/blank? schema))
+ (not (driver/schema-exists? driver db-id schema)))
+ (cluster-lock/with-cluster-lock (keyword "metabase.transforms-base.util" (str "create-target-schema-" db-id))
+ (when-not (driver/schema-exists? driver db-id schema)
+ (driver/create-schema-if-needed! driver conn-spec schema)))))
+
(defn query-transform?
"Check if this is a query transform: native query / mbql query."
[transform]
diff --git a/test/metabase/sync/sync_metadata/fields/sync_metadata_test.clj b/test/metabase/sync/sync_metadata/fields/sync_metadata_test.clj
index 5134afef79d..70f447acb2e 100644
--- a/test/metabase/sync/sync_metadata/fields/sync_metadata_test.clj
+++ b/test/metabase/sync/sync_metadata/fields/sync_metadata_test.clj
@@ -361,6 +361,27 @@
(finally
(t2/delete! :model/Database (mt/id))))))))
+(deftest untyped-field-getting-its-base-type-keeps-user-settings-test
+ (testing "A Field that never had a real base type, like one serdes synthesized for a stub database, keeps its user settings"
+ (mt/with-temp [:model/Field {field-id :id} {:base_type :type/* :active false}]
+ (field-user-settings/upsert-user-settings {:id field-id} {:semantic_type :type/Category})
+ (is (= [["Field"
+ field-id
+ {:base_type :type/Text
+ :effective_type :type/Text
+ :coercion_strategy nil
+ :fingerprint_version 0
+ :fingerprint nil
+ :semantic_type nil}]]
+ (updates-that-will-be-performed!
+ (merge default-metadata
+ {:base-type :type/Text
+ :effective-type :type/Text})
+ (merge default-metadata
+ {:id field-id
+ :base-type :type/*
+ :effective-type :type/*})))))))
+
(deftest create-or-replace-table-updates-effective-type-test
(testing "GHY-3388: when a column's database type changes in place (e.g. TEXT -> numeric via
Snowflake's CREATE OR REPLACE TABLE AS SELECT TRY_TO_NUMBER(...)), sync should update
diff --git a/test/metabase/transforms_base/util_test.clj b/test/metabase/transforms_base/util_test.clj
index 0d6fb87efbe..a10c465e4cb 100644
--- a/test/metabase/transforms_base/util_test.clj
+++ b/test/metabase/transforms_base/util_test.clj
@@ -3,6 +3,7 @@
[clojure.test :refer :all]
[java-time.api :as t]
[metabase.config.core :as config]
+ [metabase.driver :as driver]
[metabase.sync.sync :as sync]
[metabase.test :as mt]
[metabase.transforms-base.util :as transforms-base.u]
@@ -10,6 +11,23 @@
(set! *warn-on-reflection* true)
+(deftest create-target-schema!-serializes-concurrent-creation-test
+ (testing "concurrent transforms targeting a missing schema create it once, one at a time"
+ (let [created (atom #{})
+ inside (atom 0)
+ peak (atom 0)
+ calls (atom 0)]
+ (with-redefs [driver/schema-exists? (fn [_driver _db-id schema] (contains? @created schema))
+ driver/create-schema-if-needed! (fn [_driver _conn-spec schema]
+ (swap! calls inc)
+ (swap! peak max (swap! inside inc))
+ (Thread/sleep 50)
+ (swap! created conj schema)
+ (swap! inside dec))]
+ (run! deref (doall (repeatedly 8 #(future (transforms-base.u/create-target-schema! :postgres 1 {} "analytics")))))
+ (is (= 1 @peak))
+ (is (= 1 @calls))))))
+
;;; ------------------------------------------------- Merge target helpers -------------------------------------------------
(deftest merge-target?-test