Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Remove unused code (#85487)

Alexander Polyankin24c3fb2cd930
diff --git a/.claude/agents/drivers-and-sync-backend-expert.md b/.claude/agents/drivers-and-sync-backend-expert.md
index 02556efd155..259ade17209 100644
--- a/.claude/agents/drivers-and-sync-backend-expert.md
+++ b/.claude/agents/drivers-and-sync-backend-expert.md
@@ -14,7 +14,7 @@ You work on Metabase's warehouse side: the driver system, metadata sync, and the
Driver core (OSS, `src/metabase/driver/`):
- `metabase.driver` - the driver multimethods, `register!`, the hierarchy, and the `database-supports?` feature list.
- `metabase.driver.impl` - hierarchy and initialization. `metabase.driver.util` - `supports?`, connection testing.
-- `metabase.driver.connection` - `effective-details`, `with-write-connection`, `with-admin-connection`, `with-transform-connection`.
+- `metabase.driver.connection` - `effective-details`, `with-write-connection`, `with-transform-connection`.
- `metabase.driver.sql`, `metabase.driver.sql-jdbc` - abstract parents.
- `metabase.driver.sql-jdbc.connection` - c3p0 pool cache, `invalidate-pool-for-db!`. `metabase.driver.sql-jdbc.connection.ssh-tunnel` - tunnels for JDBC drivers.
- `metabase.driver.sql-jdbc.execute` - JDBC execution, `read-column-thunk`, `set-parameter`.
diff --git a/.claude/agents/enterprise-backend-expert.md b/.claude/agents/enterprise-backend-expert.md
index a2ad4585b3e..a14cfe2dded 100644
--- a/.claude/agents/enterprise-backend-expert.md
+++ b/.claude/agents/enterprise-backend-expert.md
@@ -24,7 +24,7 @@ EE code lives in `enterprise/backend/src/metabase_enterprise/<module>/`. OSS shi
- **SCIM**: `metabase-enterprise.scim.{routes,auth,api,settings,db,core}` and `scim.v2.api`.
- **Tenants**: OSS shim `metabase.tenants.core`. EE `metabase-enterprise.tenants.{core,models,api,auth-provider,permissions,schema,db}`. The `use-tenants` setting lives in `metabase.permissions.settings`.
- **Database routing**
- - OSS: `metabase.database-routing.core` (`with-database-routing-on`/`-off`).
+ - OSS: `metabase.database-routing.core`; connections are routed where they are obtained (`metabase.driver.connection/connection-database`), and `driver.conn/check-not-routed` refuses shared work (sync, FieldValues, model indexes, uploads) for routed users. There is no way to turn routing off.
- EE: `metabase-enterprise.database-routing.{common,middleware,models,api,schema,db}`.
- **Dependencies**: `metabase-enterprise.dependencies.{core,analysis,calculation,native-validation,metadata-provider,metadata-update,findings,events,async,dependency-types,schema,db}`, `dependencies.models.*`, `dependencies.task.{backfill,entity-check}`.
- **Remote sync**
@@ -46,8 +46,8 @@ EE code lives in `enterprise/backend/src/metabase_enterprise/<module>/`. OSS shi
- Every Toucan model must appear in `serialization.v2.models` as exported, inlined, or excluded. Tests enforce this. `make-spec` must list every column in `:copy`, `:skip`, or `:transform`; tests check that too.
- "Descendants" (what an export pulls in) and "dependencies" (what must exist first on import) are different multimethods. Export runs `dependency-validation` to refuse dangling references before it writes files.
- Remote sync reuses serdes v2 for extract and load. `remote-sync.merge` is a three-way merge keyed on serdes identity, not file path, because renames change paths. `remote-sync.guards` refuses work while another task runs. A model joins remote sync through an entry in `remote-sync.spec/remote-sync-specs`.
-- Database routing is keyed by a user attribute, not by tenant. The router DB maps the attribute value to a destination Database row. Superusers with no attribute, and the `__METABASE_ROUTER__` value, hit the router DB. Non-admins with no attribute get a 400. Anonymous users always get an error.
-- `swap-destination-db` must be the last middleware before execution. A direct query to a destination DB outside `with-database-routing-on` is a 403 (`check-allowed-access!`). Sync runs with routing off.
+- Database routing is keyed by a user attribute, not by tenant. The router DB maps the attribute value to a destination Database row. Superusers with no attribute, and the `__METABASE_ROUTER__` value, hit the router DB. Non-admins with no attribute get a 400. No user bound (guest embeds, public links, scheduled jobs) means the router.
+- Connections to a router resolve to the current user's destination in `driver.conn/effective-details` and the JDBC pool; the QP's `bind-destination-database` only records the destination. Any direct access to a destination DB is a 403 (`check-allowed-access!`). Sync runs with routing off.
- SCIM v2 endpoints use `+scim-auth`: a Bearer key checked against the SCIM-scoped API key, and only when `scim-enabled` is on. The SCIM config endpoints in `scim.api` use normal session auth.
- Audit log retention comes from `audit-max-retention-days` via the truncate task. `audit-app.task.partitions` handles `query_execution` partitioning.
- Support access grants extend the emailed-secret auth provider. The user's password expires when the grant ends.
@@ -58,7 +58,7 @@ EE code lives in `enterprise/backend/src/metabase_enterprise/<module>/`. OSS shi
2. Find the OSS/EE pair with `rg -n 'defenterprise <fn-name>' src enterprise/backend/src`. Confirm the `:feature` keyword matches what the token grants.
3. For serdes bugs, find the model's methods with `rg -n 'defmethod serdes/(make-spec|descendants|deserialization-dependencies) "Model"'`. Reproduce with `metabase-enterprise.serialization.test-util` (`with-world`, `with-random-dump-dir`, `with-dbs`).
4. For routing, call `database-routing.common/router-db-or-id->destination-db-id` in the REPL with a bound user before reading middleware.
-5. For dependency questions, start at `metabase-enterprise.dependencies.core/errors-from-proposed-edits`.
+5. For dependency questions, start at `metabase-enterprise.dependencies.models.dependency/transitive-dependents` and the analysis job in `metabase-enterprise.dependencies.analysis-finding`.
Tests by area (EE tests under `enterprise/backend/test/metabase_enterprise/`):
diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.edn
index 9e92cc7aad4..bfe852a7117 100644
--- a/.clj-kondo/config/modules/config.edn
+++ b/.clj-kondo/config/modules/config.edn
@@ -890,7 +890,7 @@
database-routing
{:team "UX West"
:api #{metabase.database-routing.core}
- :uses #{premium-features}}
+ :uses #{lib.schema premium-features}}
dependencies
{:team "Graphy"
@@ -962,6 +962,7 @@
appearance
audit-app
auth-provider
+ database-routing
events
lib
lib.be
@@ -2853,7 +2854,6 @@
driver
events
interestingness
- lib.be
lib.metadata
lib.schema
models
@@ -3116,6 +3116,7 @@
:ns-prefix "metabase.indexes-rest"
:api #{metabase.indexes-rest.api}
:uses #{api
+ driver
lib.schema
transforms.base
transforms.indexes
@@ -3402,6 +3403,7 @@
app-db
audit-app
collections
+ database-routing
driver
lib
lib.be
@@ -3880,7 +3882,6 @@
{:team "UX West"
:api #{metabase-enterprise.database-routing.api}
:uses #{api
- database-routing
driver
events
lib.metadata
@@ -4229,7 +4230,7 @@
warehouses.schema
enterprise/permissions.sandbox}
:model-exports #{:model/ConnectionImpersonation}
- :model-imports #{:model/DataPermissions :model/Database :model/PermissionsGroupMembership}}
+ :model-imports #{:model/DataPermissions :model/PermissionsGroupMembership}}
enterprise/permissions.sandbox
{:team "UX West"
@@ -4583,8 +4584,7 @@
sql-tools.parsing
task
transforms.base
- util
- warehouses}
+ util}
:model-exports #{:model/TransformTest}
:model-imports #{:model/Collection
:model/Database
diff --git a/docs/developers-guide/driver-changelog.md b/docs/developers-guide/driver-changelog.md
index c0fada3f42b..611773f6473 100644
--- a/docs/developers-guide/driver-changelog.md
+++ b/docs/developers-guide/driver-changelog.md
@@ -13,6 +13,23 @@ title: Driver interface changelog
## Metabase 0.65.0
+- `metabase.driver.sql-jdbc.execute/set-role-if-supported!` moved to
+ `metabase.driver.sql-jdbc.connection/set-role-if-supported!`. The JDBC connection pool now sets the connection
+ impersonation role on every checkout, so a driver that overrides `do-with-connection-with-options` no longer calls
+ it, and `do-with-connection-with-options` no longer sets it either.
+
+- `metabase.driver.connection/effective-details` and `metabase.driver.sql-jdbc.connection/db->pooled-connection-spec`
+ resolve a database routing router to the current user's destination. Read connection details through
+ `effective-details` and get JDBC connections through the pool, so routing applies; building a connection from raw
+ details (`sql-jdbc.conn/connection-details->spec`, `driver.u/can-connect-with-details?`) is a discouraged var
+ outside driver namespaces.
+
+- The admin connection is removed: `metabase.driver.connection/with-admin-connection`,
+ `metabase.driver.connection/database-admin-details`, the `:admin` connection type,
+ `metabase.driver.common/admin-connection-option` (and its `"admin-connection"` `visible-if` marker) and
+ `metabase.driver.util/fields-hidden-for-admin-connection` are gone, and a Database no longer has `:admin_details`.
+ A driver's `normalize-db-details` normalizes only `:details` and `:write_data_details`.
+
- `metabase.driver/disallowed-connection-parameters` `[driver]` -- new multimethod returning the names of client
connection parameters a user may never set, whether in `:additional-options` or as a detail key. Metabase refuses
details that set one when a connection is tested, when a connection pool is created, and when a Database is saved.
diff --git a/enterprise/backend/src/metabase_enterprise/advanced_config/file/databases.clj b/enterprise/backend/src/metabase_enterprise/advanced_config/file/databases.clj
index a5ff8b360b3..7bdfafab2d5 100644
--- a/enterprise/backend/src/metabase_enterprise/advanced_config/file/databases.clj
+++ b/enterprise/backend/src/metabase_enterprise/advanced_config/file/databases.clj
@@ -97,7 +97,7 @@
"Kick off a sync of `database` when syncing databases from the config file is enabled."
[database]
(if (advanced-config.settings/config-from-file-sync-databases)
- (quick-task/submit-task! (fn [] (sync/sync-database! database)))
+ (quick-task/submit-task! (fn [] (sync/sync-database! (u/the-id database))))
(log/info "Sync on database creation when initializing from file is disabled. Skipping sync.")))
(defn- init-from-config-file!
diff --git a/enterprise/backend/src/metabase_enterprise/audit_app/analytics_dev.clj b/enterprise/backend/src/metabase_enterprise/audit_app/analytics_dev.clj
index ca6fc00a114..4cdbff7d443 100644
--- a/enterprise/backend/src/metabase_enterprise/audit_app/analytics_dev.clj
+++ b/enterprise/backend/src/metabase_enterprise/audit_app/analytics_dev.clj
@@ -23,6 +23,7 @@
[metabase.setup.core :as setup]
[metabase.startup.core :as startup]
[metabase.sync.core :as sync]
+ [metabase.util :as u]
[metabase.util.i18n :refer [tru]]
[metabase.util.log :as log])
(:import
@@ -85,7 +86,7 @@
:creator_id user-id
:auto_run_queries true})]
(log/info "Created analytics dev database:" (:id db))
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (u/the-id db) {:scan :schema})
db))))
(defn delete-analytics-dev-database!
diff --git a/enterprise/backend/src/metabase_enterprise/audit_app/audit.clj b/enterprise/backend/src/metabase_enterprise/audit_app/audit.clj
index 37fb0d35b0d..23bee04d92d 100644
--- a/enterprise/backend/src/metabase_enterprise/audit_app/audit.clj
+++ b/enterprise/backend/src/metabase_enterprise/audit_app/audit.clj
@@ -294,7 +294,7 @@
(when (contains? #{::installed ::updated} result)
(when-let [db (audit-app.db/audit-database)]
(log/info "Syncing Audit DB")
- (log/with-no-logs (sync/sync-database! db {:scan :schema}))
+ (log/with-no-logs (sync/sync-database! (u/the-id db) {:scan :schema}))
;; this sync saw every current view, so `maybe-sync-audit-db!` need not sync again for stale views
(audit-app.settings/last-analytics-views-checksum! (views-checksum))))
result))
@@ -321,7 +321,7 @@
(log/infof "Syncing Audit DB schema (engine-changed? %s, views-stale? %s, sync-pending? %s)"
engine-changed? views-stale? sync-pending?)
(try
- (log/with-no-logs (sync/sync-database! audit-db {:scan :schema}))
+ (log/with-no-logs (sync/sync-database! (u/the-id audit-db) {:scan :schema}))
(when current
(audit-app.settings/last-analytics-views-checksum! current))
(audit-app.settings/audit-db-dialect-sync-pending! false)
diff --git a/enterprise/backend/src/metabase_enterprise/database_routing/api.clj b/enterprise/backend/src/metabase_enterprise/database_routing/api.clj
index a1bb32e9356..1ed6d05b2a3 100644
--- a/enterprise/backend/src/metabase_enterprise/database_routing/api.clj
+++ b/enterprise/backend/src/metabase_enterprise/database_routing/api.clj
@@ -113,7 +113,6 @@
(api/check-400 (not (:router_database_id db)) "Cannot make a destination database a router database")
(api/check-400 (not (:uploads_enabled db)) "Cannot enable database routing for a database with uploads enabled")
(api/check-400 (not (:write_data_details db)) "Cannot enable database routing for a database with a write connection configured")
- (api/check-400 (not (:admin_details db)) "Cannot enable database routing for a database with an admin connection configured")
(api/check-400 (not (database-routing.db/transform-exists-for-source-database? id)) "Cannot enable database routing for a database with transforms")
(setting/with-database db
(api/check-400 (not (setting/get :persist-models-enabled)) "Cannot enable database routing for a database with model persistence enabled")
diff --git a/enterprise/backend/src/metabase_enterprise/database_routing/common.clj b/enterprise/backend/src/metabase_enterprise/database_routing/common.clj
index 4b503712e3e..a974f264075 100644
--- a/enterprise/backend/src/metabase_enterprise/database_routing/common.clj
+++ b/enterprise/backend/src/metabase_enterprise/database_routing/common.clj
@@ -2,33 +2,44 @@
(:require
[metabase-enterprise.database-routing.db :as database-routing.db]
[metabase.api.common :as api]
+ [metabase.driver.util :as driver.u]
[metabase.permissions.core :as perms]
- [metabase.premium-features.core :refer [defenterprise]]
- [metabase.util :as u]
+ [metabase.premium-features.core :as premium-features :refer [defenterprise]]
[metabase.util.i18n :refer [tru]]))
-(defn- user-attribute
- "Which user attribute should we use for this RouterDB?"
- [db-or-id]
- (database-routing.db/router-user-attribute (u/the-id db-or-id)))
-
#_{:clj-kondo/ignore [:metabase/discourage-dynamic-vars]}
-(def ^:dynamic ^:private *database-routing-on* :unset)
+(def ^:dynamic ^:private *anonymous-router-access?*
+ "Whether work with no current user may reach a router itself; see [[do-with-anonymous-router-access]]."
+ false)
+
+(defenterprise do-with-anonymous-router-access
+ "Enterprise version. Calls `thunk` with no current user, letting it reach a router itself."
+ :feature :none
+ [thunk]
+ (binding [api/*current-user-id* nil
+ *anonymous-router-access?* true]
+ (thunk)))
-(defn- router-db-or-id->destination-db-id*
- [is-anonymous-user? user-attributes is-superuser? db-or-id]
- (when-let [attr-name (user-attribute db-or-id)]
+(defn- user-attribute
+ "The user attribute the router `db`, the Lib metadata of a Database, routes on; read from the app DB only when `db` was
+ converted in memory and so lacks it."
+ [db]
+ (if (contains? db :router-user-attribute)
+ (:router-user-attribute db)
+ (database-routing.db/router-user-attribute (:id db))))
+
+(defn- router-db->destination-db-id*
+ [is-anonymous-user? user-attributes is-superuser? db]
+ (when-let [attr-name (user-attribute db)]
(let [database-name (get user-attributes attr-name)]
(cond
- ;; if database routing is EXPLICITLY off, e.g. in `POST /api/database/:id/sync_schema`, don't do any routing.
- (= :off *database-routing-on*)
- nil
-
is-anonymous-user?
- (throw (ex-info (tru "Anonymous users cannot access a database with routing enabled.") {:status-code 400
- :database-routing-enabled true
- :database-or-id db-or-id
- :database-name (database-routing.db/database-name (u/the-id db-or-id))}))
+ (when-not *anonymous-router-access?*
+ (throw (ex-info (tru "Anonymous users cannot access a database with routing enabled.")
+ {:status-code 400
+ :database-routing-enabled true
+ :database-id (:id db)
+ :database-name (:name db)})))
(= database-name "__METABASE_ROUTER__")
nil
@@ -42,31 +53,42 @@
(nil? database-name)
(throw (ex-info (tru "Required user attribute is missing. Cannot route to a Destination Database.")
{:database-name database-name
- :router-database-id (u/the-id db-or-id)
+ :router-database-id (:id db)
:status-code 400}))
:else
- (or (database-routing.db/destination-database-id (u/the-id db-or-id) database-name)
+ (or (database-routing.db/destination-database-id (:id db) database-name)
(throw (ex-info (tru "Database Routing error: No Destination Database with slug `{0}` found."
database-name)
{:database-name database-name
- :router-database-id (u/the-id db-or-id)
+ :router-database-id (:id db)
:status-code 400})))))))
-(defn router-db-or-id->destination-db-id
- "Given a user and a database (or id), returns the ID of the destination database that the user's query should ultimately be
- routed to. If the database is not a Router Database, returns `nil`. If the database is a Router Database but no
- current user exists, an exception will be thrown."
- [db-or-id]
- (router-db-or-id->destination-db-id*
- (nil? @api/*current-user*)
- (api/current-user-attributes)
+(defn router-db->destination-db-id
+ "The ID of the destination database that the current user's queries against `db`, the Lib metadata of a Database,
+ are routed to, or nil when `db` is not a Router Database or no user is bound in [[do-with-anonymous-router-access]];
+ throws when no user is bound outside it."
+ [db]
+ (router-db->destination-db-id*
+ (nil? api/*current-user-id*)
+ (when api/*current-user-id* (api/current-user-attributes))
api/*is-superuser?*
- db-or-id))
+ db))
+
+(defenterprise routed-database
+ "Enterprise version. The Lib metadata of the destination Database that the current user's connections to `database`
+ go to, or nil when
+ they go to `database` itself; with no user bound, a router is refused outside [[do-with-anonymous-router-access]]."
+ :feature :none
+ [database]
+ (when-not (and (nil? api/*current-user-id*) *anonymous-router-access?*)
+ (when-let [destination-id (router-db->destination-db-id database)]
+ (premium-features/assert-has-feature :database-routing (tru "Database Routing"))
+ (driver.u/lib-database destination-id))))
(defenterprise routed-user?
"Whether the current user's queries against ANY router database they can view data from resolve to a destination
- database rather than the router itself, mirroring [[router-db-or-id->destination-db-id*]]. Routers whose
+ database rather than the router itself, mirroring [[router-db->destination-db-id*]]. Routers whose
`:perms/view-data` is `:blocked` for the user are ignored: they cannot run anything against them, so routing there
doesn't change what they see. A user whose attribute is the `__METABASE_ROUTER__` sentinel is not routed. A superuser
missing the attribute defaults to the router, so is not routed, but a superuser whose attribute names a destination
@@ -106,66 +128,21 @@
throw when a routed non-admin is missing the required routing attribute."
:feature :none
[db-id]
- (when-let [dest (router-db-or-id->destination-db-id db-id)]
+ (when-let [dest (router-db->destination-db-id (driver.u/lib-database db-id))]
{:destination-db-id dest}))
-;; We want, at all times, a guarantee that we are not hitting a router *or* destination database without being
-;; intentional about it. It would be bad to EITHER:
-;;
-;; - (a) accidentally hit a router database because we didn't include the middleware necessary to turn on Database
-;; Routing, OR
-;;
-;; (b) accidentally hit a destination database when doing so is nonsensical, e.g. for database sync processes that should
-;; only use router databases. In these cases we don't want database routing, we just want to ensure we're not hitting
-;; a Destination Database
-;;
-;; `*database-routing-on*` records our intent: `:on` inside a routed query (destinations are expected), `:off` when we
-;; explicitly want the router (e.g. sync), `:unset` otherwise. Concretely:
-;;
-;; (a) looks like:
-;; - I am looking at a Router Database,
-;; - the current user's attribute would route me to a *different* destination, and
-;; - `*database-routing-on*` is `:on` or `:unset`.
-;; A correctness concern, owned by the routing middleware (which makes the routing decision); not enforced here.
-;;
-;; (b) looks like:
-;; - I am looking at a destination Database, and
-;; - `*database-routing-on*` is not `:on` (i.e. `:off` or `:unset`).
-;; A tenancy boundary, enforced by `check-allowed-access!` below.
-
-(defenterprise with-database-routing-on-fn
- "Enterprise version. Calls the function with Database Routing allowed."
- :feature :database-routing
- [f]
- (binding [*database-routing-on* :on]
- (f)))
-
-(defenterprise with-database-routing-off-fn
- "Enterprise version. Calls the function with Database Routing prohibited."
- :feature :database-routing
- [f]
- (binding [*database-routing-on* :off]
- (f)))
-
-(defn- is-disallowed-destination-db-access?
- [db-or-id]
- (and (database-routing.db/destination-database? db-or-id)
- (not= *database-routing-on* :on)))
-
(defn assert-not-direct-destination-access!
- "Throws a 403 when `db-or-id` is a destination database queried directly, outside a routing-on
- context. A destination is reachable only through its router; a direct query bypasses the router's
- attribute check, and with it tenant isolation."
- [db-or-id]
- (when (is-disallowed-destination-db-access? (u/the-id db-or-id))
+ "Throws a 403 when `db`, the Lib metadata of a Database, is a destination database: a destination is reachable only
+ through its router, and a direct query bypasses the router's attribute check, and with it tenant isolation."
+ [db]
+ (when (:router-database-id db)
(throw (ex-info (tru "You cannot query a destination database directly.")
{:status-code 403}))))
(defenterprise check-allowed-access!
- "Throws a 403 if `db-or-id-or-spec` is a destination database accessed while database routing is
- not `:on`, i.e. a direct hit that bypasses its router. Legitimate access to a destination goes
- through its router, which turns routing `:on`."
- :feature :database-routing
- [db-or-id-or-spec]
- (when-let [db-id (u/id db-or-id-or-spec)]
- (assert-not-direct-destination-access! db-id)))
+ "Throws a 403 if `db`, the Lib metadata of a Database, is a destination database: connections to a destination are only
+ made through its router, by [[metabase.driver.connection/connection-database]], even while the `:database-routing`
+ feature is unavailable."
+ :feature :none
+ [db]
+ (assert-not-direct-destination-access! db))
diff --git a/enterprise/backend/src/metabase_enterprise/database_routing/db.clj b/enterprise/backend/src/metabase_enterprise/database_routing/db.clj
index 074035fa6c9..c984737f513 100644
--- a/enterprise/backend/src/metabase_enterprise/database_routing/db.clj
+++ b/enterprise/backend/src/metabase_enterprise/database_routing/db.clj
@@ -12,16 +12,6 @@
[database-id :- ::lib.schema.id/database]
(t2/select-one :model/Database :id database-id))
-(mu/defn database-name
- "The name of the Database with `database-id`."
- [database-id :- ::lib.schema.id/database]
- (t2/select-one-fn :name :model/Database database-id))
-
-(mu/defn destination-database?
- "Whether the Database with `database-id` is a routing destination."
- [database-id :- ::lib.schema.id/database]
- (t2/exists? :model/Database :id database-id :router_database_id [:not= nil]))
-
(mu/defn destination-database-id
"The ID of the destination Database of the router with `router-database-id` named `database-name`, or nil."
[router-database-id :- ::lib.schema.id/database
diff --git a/enterprise/backend/src/metabase_enterprise/database_routing/middleware.clj b/enterprise/backend/src/metabase_enterprise/database_routing/middleware.clj
index 66e9fda3ee1..5f02f332681 100644
--- a/enterprise/backend/src/metabase_enterprise/database_routing/middleware.clj
+++ b/enterprise/backend/src/metabase_enterprise/database_routing/middleware.clj
@@ -3,8 +3,7 @@
`:destination-database/id`, on the query if a destination database should be used. The second middleware runs around query
execution, and should be THE LAST middleware before we hit the database and query execution actually occurs."
(:require
- [metabase-enterprise.database-routing.common :as common :refer [router-db-or-id->destination-db-id]]
- [metabase.database-routing.core :refer [with-database-routing-on]]
+ [metabase-enterprise.database-routing.common :as common :refer [router-db->destination-db-id]]
[metabase.driver.util :as driver.u]
[metabase.lib.metadata :as lib.metadata]
[metabase.premium-features.core :as premium-features :refer [defenterprise]]
@@ -12,7 +11,6 @@
;; legacy usage -- don't do things like this going forward
^{:clj-kondo/ignore [:deprecated-namespace :discouraged-namespace]}
[metabase.query-processor.store :as qp.store]
- [metabase.util :as u]
[metabase.util.i18n :refer [tru]]))
#_{:clj-kondo/ignore [:metabase/discourage-dynamic-vars]}
@@ -24,22 +22,15 @@
nil)
(defenterprise swap-destination-db
- "Must be the last middleware before we actually hit the database. If a Router Database is specified, swaps out the
- Metadata Provider for one that has the appropriate destination database."
+ "Must be the last middleware before we actually hit the database. If a Router Database is specified, binds
+ [[*destination-database-id*]] for query logging; the connection itself is routed by
+ [[metabase.driver.connection/connection-database]]."
:feature :database-routing
[qp]
(fn [query rff]
(if-let [destination-db-id (:destination-database/id query)]
- (let [current-database (lib.metadata/database (qp.store/metadata-provider))
- rff* (fn [metadata]
- (binding [qp.store/*DANGER-allow-replacing-metadata-provider* true]
- (qp.store/with-metadata-provider (u/id current-database)
- (rff metadata))))]
- (binding [qp.store/*DANGER-allow-replacing-metadata-provider* true
- *destination-database-id* destination-db-id]
- (qp.store/with-metadata-provider destination-db-id
- (with-database-routing-on
- (qp query rff*)))))
+ (binding [*destination-database-id* destination-db-id]
+ (qp query rff))
(qp query rff))))
(defenterprise currently-db-routed?
@@ -67,7 +58,7 @@
;; nothing for the assert to protect there.
destination-db-id (when-not qp.i/*skip-middleware-because-app-db-access*
(common/assert-not-direct-destination-access! database)
- (router-db-or-id->destination-db-id database))]
+ (router-db->destination-db-id database))]
(when destination-db-id
(premium-features/assert-has-feature :database-routing (tru "Database Routing"))
(when-not (driver.u/supports? (:engine (lib.metadata/database (qp.store/metadata-provider)))
diff --git a/enterprise/backend/src/metabase_enterprise/database_routing/models.clj b/enterprise/backend/src/metabase_enterprise/database_routing/models.clj
index d0eff929fdb..519565ed467 100644
--- a/enterprise/backend/src/metabase_enterprise/database_routing/models.clj
+++ b/enterprise/backend/src/metabase_enterprise/database_routing/models.clj
@@ -1,9 +1,11 @@
(ns metabase-enterprise.database-routing.models
(:require
- [metabase-enterprise.database-routing.common :refer [router-db-or-id->destination-db-id]]
+ [metabase-enterprise.database-routing.common :refer [router-db->destination-db-id]]
[metabase-enterprise.database-routing.db :as database-routing.db]
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.models.interface :as mi]
- [metabase.premium-features.core :as premium-features :refer [defenterprise]]
+ [metabase.premium-features.core :as premium-features :refer [defenterprise defenterprise-schema]]
[metabase.util :as u]
[metabase.warehouse-schema.models.field :as field]
[methodical.core :as methodical]
@@ -29,7 +31,7 @@
The destination is nil while the `:database-routing` feature is unavailable."
:feature :none
[field]
- (when-let [destination-db-id (some->> field u/the-id field/field-id->database-id router-db-or-id->destination-db-id)]
+ (when-let [destination-db-id (some->> field u/the-id field/field-id->database-id driver.u/lib-database router-db->destination-db-id)]
{:destination-db-id (when (premium-features/has-feature? :database-routing) destination-db-id)}))
(defenterprise delete-associated-database-router!
@@ -38,11 +40,11 @@
[db-id]
(database-routing.db/delete-router! db-id))
-(defenterprise db-routing-enabled?
- "Returns whether or not the given database is either a router or destination database."
- :feature :database-routing
- :fallback :oss
- [db-or-id]
- (or (database-routing.db/router-exists? (u/the-id db-or-id))
- (some->> (:router-database-id db-or-id)
- database-routing.db/router-exists?)))
+(defenterprise-schema db-routing-enabled? :- :boolean
+ "Returns whether or not the Lib metadata `db` of a Database is either a router or destination database, even while
+ the `:database-routing` feature is unavailable."
+ :feature :none
+ [db :- ::lib.schema.metadata/database]
+ (boolean (or (database-routing.db/router-exists? (:id db))
+ (some->> (:router-database-id db)
+ database-routing.db/router-exists?))))
diff --git a/enterprise/backend/src/metabase_enterprise/dependencies/core.clj b/enterprise/backend/src/metabase_enterprise/dependencies/core.clj
deleted file mode 100644
index 75f904983e5..00000000000
--- a/enterprise/backend/src/metabase_enterprise/dependencies/core.clj
+++ /dev/null
@@ -1,193 +0,0 @@
-(ns metabase-enterprise.dependencies.core
- "API namespace for the `metabase-enterprise.dependencies` module.
-
- Call [[errors-from-proposed-edits]] to find out what things will break downstream of a set of new/updated entities."
- (:require
- [metabase-enterprise.dependencies.analysis :as deps.analysis]
- [metabase-enterprise.dependencies.db :as dependencies.db]
- [metabase-enterprise.dependencies.metadata-provider :as deps.provider]
- [metabase-enterprise.dependencies.models.dependency :as deps.graph]
- [metabase-enterprise.dependencies.schema :as deps.schema]
- [metabase.graph.core :as graph]
- [metabase.lib-be.core :as lib-be]
- [metabase.lib.core :as lib]
- [metabase.lib.metadata :as lib.metadata]
- [metabase.lib.metadata.protocols :as lib.metadata.protocols]
- [metabase.lib.schema.id :as lib.schema.id]
- [metabase.lib.schema.metadata :as lib.schema.metadata]
- [metabase.lib.schema.validate :as lib.schema.validate]
- [metabase.util :as u]
- [metabase.util.log :as log]
- [metabase.util.malli :as mu]
- [metabase.util.malli.registry :as mr]
- [metabase.util.malli.schema :as ms]))
-
-(mr/def ::entity-type
- [:enum :card :transform :snippet :table])
-
-(mr/def ::updates-map
- [:map {:closed true}
- [:card {:optional true} [:sequential ::lib.schema.metadata/card]]
- [:transform {:optional true} [:sequential ::lib.schema.metadata/transform]]
- [:snippet {:optional true} [:sequential ::lib.schema.metadata/native-query-snippet]]
- [:table {:optional true} [:sequential ::lib.schema.metadata/table]]])
-
-(mu/defn- transitive-dependents :- ::deps.schema/type->ids
- "The transitive dependents of `updated-entities` in `graph` (the app DB by default), leaving out native entities and
- their children unless `include-native?`."
- [& {:keys [graph updated-entities include-native?]}
- :- [:map {:closed true}
- [:graph {:optional true} [:maybe ::graph/graph]]
- [:updated-entities ::updates-map]
- [:include-native? {:optional true} [:maybe :boolean]]]]
- (if include-native?
- (deps.graph/transitive-dependents graph updated-entities)
- (deps.graph/transitive-mbql-dependents graph updated-entities)))
-
-(mu/defn- metadata-provider :- ::lib.schema.metadata/metadata-provider
- "Constructs a `MetadataProvider` with some pending edits applied.
-
- The edits are *transitive*! Since a card's output columns can depend on its (possibly updated) inputs, all dependents
- of editing things are considered edited too.
-
- Note that if `:result-metadata` is present on an updated Card, it **will be used!** The only case where that is
- actually useful is with a native query which has been executed, so the caller has driver metadata to give us. Any
- old, pre-update `:result-metadata` should be dropped from any other cards in `updated-entities`.
-
- Also note that `include-native?` is only relevant if you don't pass in `dependents`. If you pass in `dependents`,
- those will be included in the metadata provider whether or not they are native. If you don't pass in `dependents`,
- `include-native` will determine whether native dependents are included in the metadata provider."
- ([base-provider :- ::lib.schema.metadata/metadata-provider
- updated-entities :- ::updates-map
- & {:keys [graph dependents include-native?]}
- :- [:maybe
- [:map {:closed true}
- [:graph {:optional true} [:maybe ::graph/graph]]
- [:dependents {:optional true} [:maybe [:map-of ::entity-type [:set :int]]]]
- [:include-native? {:optional true} [:maybe :boolean]]]]]
- ;; Reusing the cache with different overrides breaks the caching of [[lib.metadata/card]] calls.
- (lib.metadata.protocols/clear-cache! base-provider)
- (let [dependents (or dependents (transitive-dependents :graph graph
- :updated-entities updated-entities
- :include-native? include-native?))]
- (deps.provider/override-metadata-provider
- {:base-provider base-provider
- :updated-entities updated-entities
- :dependent-ids dependents}))))
-
-(mu/defn- check-query-soundness :- [:map-of ::entity-type [:map-of :int [:set [:ref ::lib.schema.validate/error-with-source]]]]
- "Given a `MetadataProvider` as returned by [[metadata-provider]], scan all its updated entities and their dependents
- to check that everything is still sound.
-
- In particular, we're looking for bad clauses with refs to columns which no longer exist, since those queries fail
- to compile.
-
- May return false positives: if a card already has a bad ref in it, then it will be returned even if that bad ref has
- nothing to do with the updates in the `provider`.
-
- Returns a map `{:card {card-id [bad-ref ...]}, :transform {...}}`. It will be empty, if there are no bad refs
- detected."
- [provider :- ::lib.schema.metadata/metadata-provider]
- (let [overrides (deps.provider/all-overrides provider)
- errors (volatile! {})]
- (doseq [[entity-type ids] overrides
- id ids
- :let [bad-refs (try
- (deps.analysis/check-entity provider entity-type id)
- (catch Exception e
- (log/warnf "Error checking %s %s: %s" entity-type id (ex-message e))
- #{(lib/validation-exception-error (ex-message e))}))]]
- (when (seq bad-refs)
- (vswap! errors assoc-in [entity-type id] bad-refs)))
- @errors))
-
-(mu/defn- group-by-db :- [:map-of ::lib.schema.id/database [:map-of ::entity-type [:set ms/PositiveInt]]]
- "The card, table, transform, and snippet ids of `deps` grouped by the id of the Database each belongs to, with the
- snippets copied to every Database."
- [deps :- ::deps.schema/type->ids]
- (let [by-db (volatile! {})]
- (when (seq (:card deps))
- (doseq [[db-id card-ids] (->> (dependencies.db/card-database-ids (:card deps))
- (u/group-by :database_id :id conj #{}))]
- (vswap! by-db assoc-in [db-id :card] card-ids)))
- (when (seq (:table deps))
- (doseq [[db-id table-ids] (->> (dependencies.db/table-database-ids (:table deps))
- (u/group-by :db_id :id conj #{}))]
- (vswap! by-db assoc-in [db-id :table] table-ids)))
- (when (seq (:transform deps))
- (doseq [[db-id transform-ids] (->> (dependencies.db/transform-sources (:transform deps))
- (u/group-by #(get-in % [:source :query :database]) :id conj #{}))]
- (if db-id
- (vswap! by-db assoc-in [db-id :transform] transform-ids)
- (log/warnf "Unable to infer database from transforms %s" transform-ids))))
- (when (seq (:snippet deps))
- ;; Copy any snippet deps to each database, since they span them all.
- (vswap! by-db update-vals #(assoc % :snippet (:snippet deps))))
- @by-db))
-
-(mu/defn errors-from-proposed-edits :- [:map-of ::entity-type [:map-of :int [:set [:ref ::lib.schema.validate/error-with-source]]]]
- "Given a regular `MetadataProvider`, and a map of entity types (`:card`, `:transform`, `:snippet`) to lists of
- updated entities, this returns a map of `{entity-type {entity-id [bad-ref ...]}}`.
-
- The 1-arity groups all the dependents by which Database they are part of, and runs the analysis for each of them.
-
- The output is a map: `{entity-type {id [errors...]}}`; an empty map is returned when there are no errors
- detected.
-
- When `include-native?` is false, this function will ignore any entities using native sql and their children."
- ([edits :- ::updates-map
- & {:keys [base-provider graph include-native?]}
- :- [:maybe
- [:map {:closed true}
- [:base-provider {:optional true} [:maybe ::lib.schema.metadata/metadata-provider]]
- [:graph {:optional true} [:maybe ::graph/graph]]
- [:include-native? {:optional true} [:maybe :boolean]]]]]
- (let [valid-edits (if include-native?
- edits
- (into {}
- (map (fn [[entity-type entities]]
- [entity-type (remove #(deps.graph/is-native-entity? entity-type %)
- entities)]))
- edits))]
- (cond
- (not (some seq (vals valid-edits)))
- {}
-
- base-provider
- (-> base-provider
- (metadata-provider valid-edits :graph graph :include-native? include-native?)
- check-query-soundness)
-
- :else
- (let [all-deps (transitive-dependents :updated-entities valid-edits :include-native? include-native?)
- by-db (group-by-db all-deps)]
- (reduce (fn [errors [db-id deps]]
- (-> (lib-be/application-database-metadata-provider db-id)
- (metadata-provider valid-edits :dependents deps)
- check-query-soundness
- (merge errors)))
- {} by-db))))))
-
-(comment
- (require '[metabase.premium-features.core])
- ;; This should work on any fresh-ish Metabase instance; these are the built-in example questions.
- (let [base-mp (lib-be/application-database-metadata-provider 39)
- transform (lib.metadata/transform base-mp 1)
- transform (-> transform
- ;; NOTE: NO RAW MANIPULATION OUTSIDE OF LIB! DO NOT EVER DO THIS PLEASE <3
- (update-in [:source :query :query :expressions]
- ;; Replacing the expression Age with Duration - this breaks downstream uses!
- update-keys (constantly "Fair Die"))
- #_(dissoc :result-metadata))
- mp (metadata-provider base-mp {:transform [transform]})]
- #_(deps.provider/all-overrides mp)
- (check-query-soundness mp))
-
- (dependencies.db/card 124)
- (deps.graph/transitive-dependents {:transform [{:id 1}]})
- (deps.graph/transitive-dependents {:table [{:id 155}]})
- (metabase.premium-features.core/token-features)
-
- (let [card (dependencies.db/card 121)
- mp (lib-be/application-database-metadata-provider (:database_id card))]
- (upstream-deps:card mp card)))
diff --git a/enterprise/backend/src/metabase_enterprise/dependencies/db.clj b/enterprise/backend/src/metabase_enterprise/dependencies/db.clj
index 385f9fa1d32..8185dfc5bdf 100644
--- a/enterprise/backend/src/metabase_enterprise/dependencies/db.clj
+++ b/enterprise/backend/src/metabase_enterprise/dependencies/db.clj
@@ -689,21 +689,11 @@
[card-ids :- [:sequential ::lib.schema.id/card]]
(t2/select-fn->fn :id :type [:model/Card :id :type] :id [:in card-ids]))
-(mu/defn card-database-ids
- "The `:id` and `:database_id` of the Cards with `card-ids`."
- [card-ids :- [:sequential ::lib.schema.id/card]]
- (t2/select [:model/Card :id :database_id] :id [:in card-ids]))
-
(mu/defn tables
"The Tables with `table-ids`."
[table-ids :- [:set ::lib.schema.id/table]]
(t2/select :model/Table :id [:in table-ids] {:from [(warehouse-schema-overlay/table-query)]}))
-(mu/defn table-database-ids
- "The `:id` and `:db_id` of the Tables with `table-ids`."
- [table-ids :- [:sequential ::lib.schema.id/table]]
- (t2/select [:model/Table :id :db_id] :id [:in table-ids] {:from [(warehouse-schema-overlay/table-query {:user-settings? false})]}))
-
(mu/defn dashcard-dashboard-id :- [:maybe ::lib.schema.id/dashboard]
"The `:dashboard_id` of the DashboardCard with `dashcard-id`, or nil."
[dashcard-id :- ::lib.schema.id/dashcard]
@@ -714,11 +704,6 @@
[field-id :- ::lib.schema.id/field]
(t2/select-one-fn :table_id :model/Field :id field-id {:from [(warehouse-schema-overlay/field-query {:user-settings? false})]}))
-(mu/defn transform-sources
- "The `:id` and `:source` of the Transforms with `transform-ids`."
- [transform-ids :- [:sequential ::lib.schema.id/transform]]
- (t2/select [:model/Transform :id :source] :id [:in transform-ids]))
-
(mu/defn source-database-transform-ids :- [:set ::lib.schema.id/transform]
"The IDs of the Transforms reading from the Database with `db-id`."
[db-id :- ::lib.schema.id/database]
@@ -1097,11 +1082,6 @@
[field-id :- ::lib.schema.id/field]
(t2/exists? :model/Dimension :field_id field-id :type "internal"))
-(mu/defn native-card-ids :- [:set ::lib.schema.id/card]
- "The IDs of the native Cards among `card-ids`."
- [card-ids :- [:set ::lib.schema.id/card]]
- (or (t2/select-pks-set :model/Card :id [:in card-ids] :query_type :native) #{}))
-
(mu/defn card-state :- [:maybe ::dependencies.schema/card-state]
"The `updated_at` and `result_metadata` of the Card with `card-id`, locking its row until the transaction ends when
`lock?`, or nil."
diff --git a/enterprise/backend/src/metabase_enterprise/dependencies/metadata_provider.clj b/enterprise/backend/src/metabase_enterprise/dependencies/metadata_provider.clj
deleted file mode 100644
index cf4bccfa9ef..00000000000
--- a/enterprise/backend/src/metabase_enterprise/dependencies/metadata_provider.clj
+++ /dev/null
@@ -1,294 +0,0 @@
-(ns metabase-enterprise.dependencies.metadata-provider
- "A `MetadataProvider` wrapper that understands how to update entities, and to lazily compute the new metadata for
- updated entities on demand. This is useful when analyzing a tree of dependents, which might depend on each other in
- complex ways.
-
- Just to illustrate, imagine I change a card A which is used by a transform B whose output table C is used by a card
- D. If I remove a column from card A, and transform B selects all columns, that field needs to disappear from C
- before we can correctly analyze card D.
-
- By representing the updated entities as lazy `delay`s, this `MetadataProvider` can support analysis of the
- downstream entities in any order."
- (:require
- [medley.core :as m]
- [metabase-enterprise.dependencies.analysis :as deps.analysis]
- [metabase.lib.core :as lib]
- [metabase.lib.metadata :as lib.metadata]
- [metabase.lib.metadata.protocols :as lib.metadata.protocols]
- [metabase.util :as u]
- [metabase.util.log :as log]
- [pretty.core :as pretty]))
-
-(set! *warn-on-reflection* true)
-
-(defn- metadatas [delegate overrides {metadata-type :lib/type
- search-name :name
- :keys [id card-ids table-ids]
- :as metadata-spec}]
- (let [type-overrides (get overrides metadata-type)
- metadata-keys (set (or id search-name))]
- (log/tracef "OverridingMetadataProvider request for %s" metadata-spec)
- (cond
- ;; For tables, cards, snippets, transforms, and columns, return overrides if present and looking for specific
- ;; keys and delegate if not.
- (and (#{:metadata/table :metadata/card :metadata/transform :metadata/native-query-snippet :metadata/column} metadata-type)
- (seq metadata-keys))
- (let [overrides (if (seq id)
- (select-keys type-overrides id)
- (into {} (keep #(when-let [x (deref %)]
- (when (search-name (:name x))
- [(:name x) %])))
- (vals type-overrides)))
- override-key (if (seq id) :id :name)
- overridden-keys (set (keys overrides))
- missing-keys (remove overridden-keys metadata-keys)]
- (log/tracef "Overridden keys %s; missing keys %s" (sort overridden-keys) (sort missing-keys))
- (cond-> (map deref (vals overrides))
- (seq missing-keys)
- (concat (lib.metadata.protocols/metadatas delegate {:lib/type metadata-type
- override-key (set missing-keys)}))))
-
- ;; if fetching all tables or transforms, get every override and everything from the delegate and then dedupe
- (and (empty? metadata-keys)
- (#{:metadata/table :metadata/transform} metadata-type))
- (as-> (vals type-overrides) entities
- (into [] (map force) entities)
- (into entities (lib.metadata.protocols/metadatas delegate metadata-spec))
- (m/distinct-by :id entities))
-
- ;; For columns, return overrides if they're present. `:card-ids` and `:table-ids` are sets of ids; entities
- ;; with overrides use the overridden columns, the rest are delegated.
- (= metadata-type :metadata/column)
- (let [[override-k spec-k ids] (cond
- card-ids [::card-columns :card-ids card-ids]
- table-ids [::table-columns :table-ids table-ids])]
- (if-not override-k
- (lib.metadata.protocols/metadatas delegate metadata-spec)
- (let [id->overridden (into {}
- (keep (fn [id]
- (when-let [columns (get-in overrides [override-k id])]
- [id columns])))
- ids)
- missing-ids (into #{} (remove id->overridden) ids)]
- (concat (mapcat (comp deref val) id->overridden)
- (when (seq missing-ids)
- (lib.metadata.protocols/metadatas delegate (assoc metadata-spec spec-k missing-ids)))))))
-
- :else (throw (ex-info "Unknown :lib/metadata type for OverridingMetadataProvider" metadata-spec)))))
-
-(declare all-overrides setup-transform! setup-transforms!)
-
-(deftype OverridingMetadataProvider [delegate *overrides]
- lib.metadata.protocols/MetadataProvider
- (database [_this] (lib.metadata.protocols/database delegate))
- (metadatas [this metadata-spec]
- (setup-transforms! this)
- (metadatas delegate @*overrides metadata-spec))
- (setting [_this setting-key]
- (lib.metadata.protocols/setting delegate setting-key))
-
- ;; Cached things are direct delegation.
- lib.metadata.protocols/CachedMetadataProvider
- (cached-metadatas [_this metadata-type metadata-ids]
- (lib.metadata.protocols/cached-metadatas delegate metadata-type metadata-ids))
- (store-metadata! [_this a-metadata]
- (lib.metadata.protocols/store-metadata! delegate a-metadata))
- (cached-value [_this k not-found]
- (lib.metadata.protocols/cached-value delegate k not-found))
- (cache-value! [_this k v]
- (lib.metadata.protocols/cache-value! delegate k v))
- (has-cache? [_this]
- (lib.metadata.protocols/has-cache? delegate))
- (clear-cache! [_this]
- (lib.metadata.protocols/clear-cache! delegate))
-
- pretty/PrettyPrintable
- (pretty [this]
- (list `overriding-metadata-provider (all-overrides this) delegate)))
-
-(defmulti add-override
- "Given an `OverridingMetadataProvider`, an `entity-type` and that entity, add this entity as an override."
- {:arglists '([metadata-provider entity-type id updates-or-nil])}
- (fn [_mp entity-type _id _updates]
- entity-type))
-
-(defn- inner-mp [^OverridingMetadataProvider mp]
- (.delegate mp))
-
-(defn- with-overrides [^OverridingMetadataProvider mp kvs]
- (swap! (.*overrides mp)
- #(reduce (fn [m [ks v]]
- (assoc-in m ks v))
- % kvs)))
-
-(defn- returned-columns [mp queryable]
- (deps.analysis/returned-columns (:engine (lib.metadata/database mp))
- (lib/query mp queryable)))
-
-(defmethod add-override :card [^OverridingMetadataProvider mp _entity-type id updates]
- (with-overrides mp
- ;; Only `:result-metadata` from the `updates` is used; any from the inner-mp is ignored.
- {[:metadata/card id] (delay (-> (merge (when id
- (-> (inner-mp mp)
- (lib.metadata.protocols/card id)))
- updates)
- (assoc :result-metadata (:result-metadata updates))))
- ;; This uses the outer OMP and so the overrides are visible!
- [::card-columns id] (delay (returned-columns mp (lib.metadata/card mp id)))}))
-
-(defonce ^:private last-fake-id (atom 2000000000))
-
-(defn- fake-id []
- (swap! last-fake-id inc))
-
-(defn- setup-transform!
- [^OverridingMetadataProvider mp s+n->table {:keys [source target] :as _transform}]
- ;; See if the output table exists in the underlying MetadataProvider.
- ;; - If the table doesn't exist, invent an ID for it.
- ;; - Always add an override for the table, even if it already existed.
- ;; - Compare the transform's output columns to the fields that exist in the metadata.
- ;; - For any newly added fields, invent an ID and add an override for the field.
- ;; - For any removed fields that used to exist, add an override for their ID that is nil!
- ;; - Add an internal [::table-columns table-id] with the list of output columns.
- ;; Note that this assumes that all other overrides are already loaded!
- (let [existing-table (get s+n->table [(:schema target) (:name target)])
- output-table (or existing-table
- (merge (select-keys target [:schema :name])
- {:lib/type :metadata/table
- :id (fake-id)
- :db-id (:id (lib.metadata/database (inner-mp mp)))
- :display-name (:name target)
- :visibility-type nil}))
- existing-cols (when existing-table
- (lib/returned-columns (lib/query (inner-mp mp) existing-table)))
- output-cols (delay
- ;; Note that this will analyze the query with any upstream changes included!
- (let [new-cols (returned-columns mp (:query source))
- by-name (m/index-by :lib/desired-column-alias existing-cols)]
- (into [] (for [col new-cols
- :let [old-col (by-name (:lib/desired-column-alias col))]]
- (merge (select-keys col [:display-name
- :base-type :effective-type :semantic-type])
- {:name (:lib/desired-column-alias col)
- :lib/type :metadata/column
- :id (or (:id old-col) (fake-id))
- :table-id (:id output-table)
- :active true})))))
- outputs-by-id (delay (m/index-by :id @output-cols))]
- ;; NOTE: Any newly added output columns can't be looked up directly by ID, with this version.
- ;; The `output-cols` code above can be adapted to add them to the `*overrides` atom, but YAGNI for now.
- (with-overrides mp
- (into {[:metadata/table (:id output-table)] (delay output-table)
- [::table-columns (:id output-table)] output-cols}
- ;; For each pre-existing output column, we need to add an override for a direct read by ID.
- ;; If the column does not exist in the output, return the original with :active false
- (map (fn [existing-col]
- [[:metadata/column (:id existing-col)]
- (delay (or (get @outputs-by-id (:id existing-col))
- (assoc existing-col :active false)))]))
- existing-cols))))
-
-(defn- setup-transforms! [^OverridingMetadataProvider mp]
- (locking mp
- (when-not (-> (.*overrides mp) deref ::setup-complete)
- (swap! (.*overrides mp) assoc ::setup-complete true)
- (let [;; Fetching all the pre-existing tables.
- ;; TODO: Once Cam's new by-name logic in the `MetadataProvider` lands, we can use that here.
- tables (lib.metadata/tables (inner-mp mp))
- s+n->table (m/index-by (juxt :schema :name) tables)]
- (doseq [transform (-> (.*overrides mp) deref :metadata/transform vals)]
- (setup-transform! mp s+n->table (deref transform)))))))
-
-(defmethod add-override :transform [^OverridingMetadataProvider mp _entity-type id updates]
- ;; Transforms are complicated:
- ;; 1. Treated as a card on the input side.
- ;; 2. The returned-columns of that card become the columns of the output table.
- ;; 3. The output table is constructed from the transform.
- ;; 4. Its fields are adapted from the transform's input card.
- ;; All of these can be wrapped in delays and therefore computed lazily on demand.
- ;;
- ;; The trouble is that a transform does not know the output table's ID, nor its field IDs.
- ;; A modified transform might produce a new table, or have new fields or removed fields.
- ;; The OverridingMetadataProvider has to account for all of this when it returns tables or fields by ID, or gets
- ;; the column list for a table.
- ;;
- ;; This is achieved by putting `[:metadata/transform id]` into overrides for all transforms, and then one-time
- ;; logic that runs before the first use of the OverridingMetadataProvider as a MetadataProvider will walk all the
- ;; transforms and make sure their tables etc. are overridden properly.
- (let [transform (merge (when id
- (lib.metadata/transform (inner-mp mp) id))
- updates)]
- (with-overrides mp
- {;; The transform itself, merged with the incoming updates.
- [:metadata/transform id] (delay transform)
- #_#_[::transform-columns id] (delay (->> (lib.metadata/transform mp id)
- :source
- :query
- (lib/query mp)
- lib/returned-columns))})))
-
-(defmethod add-override :table [^OverridingMetadataProvider mp _entity-type id updates]
- (when (seq updates)
- (throw (ex-info "Updating a table directly is not supported" {:table-id id, :edited updates})))
- (with-overrides mp
- {[:metadata/table id] (delay (log/warnf "Can't happen: Table %d dep should be replaced by its transform" id)
- (lib.metadata/table (inner-mp mp) id))
- [::table-columns id] (delay
- (log/warnf "Can't happen: Table %d ::table-columns should be replaced by its transform" id)
- (lib.metadata/fields (inner-mp mp) id))}))
-
-(defmethod add-override :snippet [^OverridingMetadataProvider mp _entity-type id updates]
- (with-overrides mp
- {[:metadata/native-query-snippet id] (delay (merge (when id
- (lib.metadata/native-query-snippet (inner-mp mp) id))
- (u/normalize-map updates)))}))
-
-(defmethod add-override :dashboard [^OverridingMetadataProvider mp _entity-type _id _updates]
- mp)
-
-(defmethod add-override :document [^OverridingMetadataProvider mp _entity-type _id _updates]
- mp)
-
-(defmethod add-override :sandbox [^OverridingMetadataProvider mp _entity-type _id _updates]
- mp)
-
-(defmethod add-override :action [^OverridingMetadataProvider mp _entity-type _id _updates]
- mp)
-
-(defmethod add-override :data-app [^OverridingMetadataProvider mp _entity-type _id _updates]
- mp)
-
-(defmethod add-override :segment [^OverridingMetadataProvider mp _entity-type id updates]
- {[:metadata/segment id] (delay (merge (when id
- (lib.metadata/segment (inner-mp mp) id))
- updates))})
-
-(defn all-overrides
- "Returns all the overrides by ID, in the same form as the map input to [[with-deps]]:
- `{:card [1 2 3], :transform [45 99]}."
- [^OverridingMetadataProvider override-metadata-provider]
- (-> (.*overrides override-metadata-provider)
- deref
- (dissoc ::setup-complete)
- (update-vals keys)
- (update-keys {:metadata/card :card
- :metadata/transform :transform
- :metadata/native-query-snippet :snippet})
- (select-keys [:card :transform :snippet])))
-
-(defn override-metadata-provider
- "Given an underlying `MetadataProvider`, wraps it to support in-memory overrides of cards, fields, etc.
-
- Important note: all overrides must be added first, before this is used as a `MetadataProvider`. The easiest way to
- do that is to pass in updated entities and/or dependent ids, which will be immediately added as overrides."
- [{:keys [base-provider updated-entities dependent-ids]}]
- (let [^OverridingMetadataProvider omp (->OverridingMetadataProvider base-provider (atom {}))]
- (doseq [[entity-type updates] updated-entities
- updated-entity updates]
- (add-override omp entity-type (:id updated-entity) updated-entity))
- (doseq [[entity-type dependents] dependent-ids
- :let [updated (into #{} (map :id) (get updated-entities entity-type))]
- id dependents
- :when (not (updated id))]
- (add-override omp entity-type id nil))
- omp))
diff --git a/enterprise/backend/src/metabase_enterprise/dependencies/models/dependency.clj b/enterprise/backend/src/metabase_enterprise/dependencies/models/dependency.clj
index def1b780a68..3738699bacc 100644
--- a/enterprise/backend/src/metabase_enterprise/dependencies/models/dependency.clj
+++ b/enterprise/backend/src/metabase_enterprise/dependencies/models/dependency.clj
@@ -4,18 +4,8 @@
[metabase-enterprise.dependencies.db :as dependencies.db]
[metabase-enterprise.dependencies.dependency-types :as deps.dependency-types]
[metabase-enterprise.dependencies.schema :as dependencies.schema]
- [metabase-enterprise.sandbox.schema]
- [metabase.actions.schema :as actions.schema]
- [metabase.documents.schema :as documents.schema]
[metabase.graph.core :as graph]
- [metabase.lib.core :as lib]
- [metabase.lib.schema.metadata :as lib.schema.metadata]
- [metabase.measures.schema]
[metabase.models.interface :as mi]
- [metabase.native-query-snippets.schema]
- [metabase.queries.schema :as queries.schema]
- [metabase.segments.schema]
- [metabase.transforms.schema :as transforms.schema]
[metabase.util :as u]
[metabase.util.malli :as mu]
[metabase.util.malli.registry :as mr]
@@ -243,78 +233,6 @@
(->> (graph/transitive graph starters) ; This returns a flat list.
group-nodes))))
-(mu/defn is-native-entity? :- :boolean
- "Checks whether an entity involves native sql. `entity` can either be a toucan object or a metadata object."
- [entity-type :- ::deps.dependency-types/dependency-types
- entity :- [:multi {:dispatch (fn [x] (if (:lib/type x) :lib-metadata :row))}
- [:lib-metadata [:multi {:dispatch :lib/type}
- [:metadata/card ::lib.schema.metadata/card]
- [:metadata/transform ::lib.schema.metadata/transform]
- [:metadata/native-query-snippet ::lib.schema.metadata/native-query-snippet]
- [:metadata/table ::lib.schema.metadata/table]]]
- [:row [:or
- ::queries.schema/card
- :metabase.warehouse-schema.schema/table
- :metabase.native-query-snippets.schema/native-query-snippet
- ::transforms.schema/transform
- :metabase.dashboards.schema/dashboard
- ::documents.schema/document
- :metabase-enterprise.sandbox.schema/sandbox
- :metabase.segments.schema/segment
- :metabase.measures.schema/measure
- ::actions.schema/action
- :metabase-enterprise.data-apps.schema/data-app]]]]
- (boolean
- (case entity-type
- :card (some-> entity
- ((some-fn :dataset-query :dataset_query))
- lib/any-native-stage?)
- :transform (some-> entity
- :source
- :query
- lib/any-native-stage?)
- :snippet true
- :action (= :query (:type entity))
- false)))
-
-(mu/defn- native-lookup-map :- [:map-of ::dependencies.schema/node :boolean]
- "Whether each node in `children` is native."
- [children :- ::dependencies.schema/child-map]
- (let [grouped (-> (graph/all-map-nodes children)
- group-nodes)]
- (into {}
- (mapcat (fn [[node-type ids]]
- (if (= node-type :card)
- (let [native (dependencies.db/native-card-ids ids)]
- (map (fn [id] [[:card id] (contains? native id)]) ids))
- (mapv (fn [entity]
- [[node-type (:id entity)]
- (is-native-entity? node-type entity)])
- (dependencies.db/entities node-type ids)))))
- grouped)))
-
-(mu/defn transitive-mbql-dependents :- ::dependencies.schema/type->ids
- "Equivalent to `transitive-dependents`, except it excludes any native cards/transforms/segments and their children.
-
- Also, the order is more flexible (though consistent between runs).
-
- Note that this does not check the passed in entities for native-ness -- the filter is only applied to their
- transitive children."
- ([entities-map :- EntitiesMap]
- (transitive-mbql-dependents nil entities-map))
- ([graph :- [:maybe ::graph/graph]
- entities-map :- EntitiesMap]
- (let [start-nodes (set (entities->nodes entities-map))
- children (graph/transitive-children-of (or graph (graph-dependents)) (seq start-nodes))
- native-lookup (native-lookup-map children)]
- (group-nodes
- (graph/keep-children (fn [node]
- (cond
- (start-nodes node) nil
- (native-lookup node) ::graph/stop
- :else node))
- children)))))
-
(mu/defn- ensure-entity-id :- ::deps.dependency-types/entity-id
"`id`, throwing a 400 when it is not a positive integer."
[id :- UncheckedEntityId]
diff --git a/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj b/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj
index c3bbd67cc04..8ebd939f7b3 100644
--- a/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj
+++ b/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj
@@ -23,7 +23,7 @@
parameter substitution must happen INSIDE the *compile-with-inline-parameters* binding
to produce inline literals instead of ? placeholders."
[query :- ::lib.schema/query]
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(let [with-params (lib/add-parameters-for-template-tags query)
compiled (qp.compile/compile-with-inline-parameters with-params)]
(lib/native-query with-params (:query compiled)))))
diff --git a/enterprise/backend/src/metabase_enterprise/gsheets/api.clj b/enterprise/backend/src/metabase_enterprise/gsheets/api.clj
index 576a7b71525..96a905700a9 100644
--- a/enterprise/backend/src/metabase_enterprise/gsheets/api.clj
+++ b/enterprise/backend/src/metabase_enterprise/gsheets/api.clj
@@ -403,4 +403,4 @@
#_{:clj-kondo/ignore [:metabase/modules]}
(require '[metabase.sync.sync-metadata :as sync-metadata])
(sync-metadata/sync-db-metadata!
- (gsheets.db/attached-dwh-database))))
+ (:id (gsheets.db/attached-dwh-database)))))
diff --git a/enterprise/backend/src/metabase_enterprise/impersonation/db.clj b/enterprise/backend/src/metabase_enterprise/impersonation/db.clj
index 45eeec52ea0..4dbd0765ddb 100644
--- a/enterprise/backend/src/metabase_enterprise/impersonation/db.clj
+++ b/enterprise/backend/src/metabase_enterprise/impersonation/db.clj
@@ -87,8 +87,3 @@
current user, such as an internal or unauthenticated context)."
[user-id :- [:maybe ::lib.schema.id/user]]
(t2/select-fn-set :group_id :model/PermissionsGroupMembership :user_id user-id))
-
-(mu/defn database
- "The Database with `database-id`, or nil."
- [database-id :- ::lib.schema.id/database]
- (t2/select-one :model/Database :id database-id))
diff --git a/enterprise/backend/src/metabase_enterprise/impersonation/driver.clj b/enterprise/backend/src/metabase_enterprise/impersonation/driver.clj
index b862be4c36a..a59a6877693 100644
--- a/enterprise/backend/src/metabase_enterprise/impersonation/driver.clj
+++ b/enterprise/backend/src/metabase_enterprise/impersonation/driver.clj
@@ -83,7 +83,7 @@
role (get user-attributes role-attribute)
database (if (map? database-or-id)
database-or-id
- (impersonation.db/database (u/the-id database-or-id)))
+ (driver.u/lib-database (u/the-id database-or-id)))
default-role (driver.sql/default-database-role (driver.u/database->driver database) database)]
(cond
(nil? role)
@@ -136,17 +136,24 @@
(defenterprise set-role-if-supported!
"Executes a `USE ROLE` or similar statement on the given connection, if connection impersonation is enabled for the
given driver. For these drivers, the role is set to either the default role, or to a specific role configured for
- the current user, depending on the connection impersonation settings. This is a no-op for databases that do not
- support connection impersonation, or for non-EE instances."
- :feature :advanced-permissions
+ the current user, depending on the connection impersonation settings; a routing destination uses its router's
+ impersonation policies, and a connection that should get a specific role is refused while the `:advanced-permissions`
+ feature is unavailable. This is a no-op for databases that do not support connection impersonation, or for non-EE
+ instances."
+ :feature :none
[driver ^Connection conn database]
(when (driver.u/supports? driver :connection-impersonation database)
(try
- (let [enabled? (impersonation-enabled-for-db? database)
+ (let [policy-database (if-let [router-id (:router-database-id database)]
+ (driver.u/lib-database router-id)
+ database)
+ enabled? (impersonation-enabled-for-db? policy-database)
default-role (driver.sql/default-database-role driver database)
;; *impersonation-role* is bound by middleware in the context of a query - otherwise, we can calculate it ourselves.
impersonation-role (or *impersonation-role*
- (connection-impersonation-role database))]
+ (connection-impersonation-role policy-database))]
+ (when impersonation-role
+ (premium-features/assert-has-feature :advanced-permissions (tru "Advanced Permissions")))
(when (and enabled? (not default-role))
(throw (ex-info (tru "Connection impersonation is enabled for this database, but no default role is found")
{:user-id api/*current-user-id*
diff --git a/enterprise/backend/src/metabase_enterprise/impersonation/util.clj b/enterprise/backend/src/metabase_enterprise/impersonation/util.clj
index 694edbc9109..33ee45c28ba 100644
--- a/enterprise/backend/src/metabase_enterprise/impersonation/util.clj
+++ b/enterprise/backend/src/metabase_enterprise/impersonation/util.clj
@@ -8,8 +8,9 @@
(defenterprise impersonation-enforced-for-db?
"Returns a boolean if the current user has a connection impersonation policy which should be enforced for the provided
- database. Will throw an error if [[api/*current-user-id*]] is not bound."
- :feature :advanced-permissions
+ database, even while the `:advanced-permissions` feature is unavailable, so callers fail closed. Will throw an error
+ if [[api/*current-user-id*]] is not bound."
+ :feature :none
[db-or-id]
(boolean
(when-not *is-superuser?*
diff --git a/enterprise/backend/src/metabase_enterprise/transform_testing/db.clj b/enterprise/backend/src/metabase_enterprise/transform_testing/db.clj
index 0a5e47a4c27..d99a8ef622f 100644
--- a/enterprise/backend/src/metabase_enterprise/transform_testing/db.clj
+++ b/enterprise/backend/src/metabase_enterprise/transform_testing/db.clj
@@ -117,10 +117,10 @@
[transform-id :- ::lib.schema.id/transform]
(t2/select-one :model/Transform :id transform-id))
-(mu/defn database
- "The Database with `database-id`, or nil."
+(mu/defn database-exists?
+ "Whether a Database with `database-id` exists."
[database-id :- ::lib.schema.id/database]
- (t2/select-one :model/Database :id database-id))
+ (t2/exists? :model/Database :id database-id))
(mu/defn transform-test-by-entity-id :- [:maybe ::transform-testing.schema/transform-test]
"The TransformTest with `entity-id`, or nil."
diff --git a/enterprise/backend/src/metabase_enterprise/transform_testing/runner.clj b/enterprise/backend/src/metabase_enterprise/transform_testing/runner.clj
index 28c3ee84b27..64da975f4d9 100644
--- a/enterprise/backend/src/metabase_enterprise/transform_testing/runner.clj
+++ b/enterprise/backend/src/metabase_enterprise/transform_testing/runner.clj
@@ -34,12 +34,12 @@
[metabase.driver :as driver]
[metabase.driver.util :as driver.u]
[metabase.lib.schema.id :as lib.schema.id]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.sql-parsing.core :as sql-parsing]
[metabase.transforms-base.util :as transforms-base.u]
[metabase.util.i18n :refer [tru]]
[metabase.util.malli :as mu]
- [metabase.util.malli.registry :as mr]
- [metabase.warehouses.schema :as warehouses.schema]))
+ [metabase.util.malli.registry :as mr]))
(set! *warn-on-reflection* true)
@@ -52,7 +52,7 @@
"What [[validated-plan]] hands the run that follows it."
[:map {:closed true}
[:driver :keyword]
- [:database ::warehouses.schema/database]
+ [:database ::lib.schema.metadata/database]
[:input->table [:map-of ::transform-testing.schema/input :string]]
[:output-table :string]
[:labels [:map-of :string :string]]
@@ -130,8 +130,9 @@
(defn- testable-database
"The database `transform` reads, if its driver can run a transform test."
[transform]
- (let [database (api/check-404 (transform-testing.db/database
- (transforms-base.u/transform-source-database transform)))
+ (let [database-id (transforms-base.u/transform-source-database transform)
+ _ (api/check-404 (some-> database-id transform-testing.db/database-exists?))
+ database (driver.u/lib-database database-id)
driver (keyword (:engine database))]
(when-not (driver.u/supports? driver :transforms/testing database)
(throw (transform-testing.errors/ex
@@ -205,7 +206,7 @@
source (parsed-source driver transform transform-id)
input->table (into {} (map (fn [input] [input (driver/temp-table-name driver)])) inputs)
output-table (driver/temp-table-name driver)
- default-schema (:default_schema database)
+ default-schema (:default-schema database)
replacements (transform-testing.compile/table-replacements transform input->table output-table default-schema)
compiled (transform-testing.compile/compile-transform driver source replacements)]
(transform-testing.validator/validate
diff --git a/enterprise/backend/src/metabase_enterprise/transforms_python/base.clj b/enterprise/backend/src/metabase_enterprise/transforms_python/base.clj
index f347aa22f47..2ad2774b50d 100644
--- a/enterprise/backend/src/metabase_enterprise/transforms_python/base.clj
+++ b/enterprise/backend/src/metabase_enterprise/transforms_python/base.clj
@@ -8,7 +8,6 @@
[clojure.core.async :as a]
[clojure.java.io :as io]
[clojure.string :as str]
- [metabase-enterprise.transforms-python.db :as transforms-python.db]
[metabase-enterprise.transforms-python.python-runner :as python-runner]
[metabase-enterprise.transforms-python.s3 :as s3]
[metabase-enterprise.transforms-python.settings :as transforms-python.settings]
@@ -16,6 +15,7 @@
[metabase.driver.sql :as driver.sql]
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.driver.util :as driver.u]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.transforms-base.interface :as transforms-base.i]
[metabase.transforms-base.schema :as transforms-base.schema]
[metabase.transforms-base.util :as transforms-base.u]
@@ -262,7 +262,7 @@
Options:
- `with-stage-timing-fn` - optional, (fn [run-id stage thunk] result) for instrumentation"
[{:keys [source] :as transform} :- ::transforms-base.schema/transform
- db :- :metabase.warehouses.schema/database
+ db :- ::lib.schema.metadata/database
run-id :- pos-int?
cancel-chan :- ::transforms-base.schema/chan
message-log :- ::transforms-base.schema/atom
@@ -366,7 +366,7 @@
(when (and cancelled? (cancelled?))
(throw (ex-info "Transform cancelled before start" {:status :cancelled})))
(let [{:keys [target] transform-id :id} transform
- db (transforms-python.db/database (:database target))
+ db (driver.u/lib-database (:database target))
;; Use run-id if provided, otherwise generate a temp one for python runner
effective-run-id (or run-id (rand-int Integer/MAX_VALUE))
cancel-chan (or cancel-chan
diff --git a/enterprise/backend/src/metabase_enterprise/transforms_python/db.clj b/enterprise/backend/src/metabase_enterprise/transforms_python/db.clj
index a3a6e7b211c..7b3220bf549 100644
--- a/enterprise/backend/src/metabase_enterprise/transforms_python/db.clj
+++ b/enterprise/backend/src/metabase_enterprise/transforms_python/db.clj
@@ -19,11 +19,6 @@
[table-id :- ::lib.schema.id/table]
(t2/select-one-fn :db_id (t2/table-name :model/Table) :id table-id {:from [(warehouse-schema-overlay/table-query {:user-settings? false})]}))
-(mu/defn database
- "The Database with `database-id`, or nil."
- [database-id :- ::lib.schema.id/database]
- (t2/select-one :model/Database database-id))
-
(mu/defn database-engine
"The engine of the Database with `database-id`."
[database-id :- ::lib.schema.id/database]
diff --git a/enterprise/backend/src/metabase_enterprise/transforms_python/execute.clj b/enterprise/backend/src/metabase_enterprise/transforms_python/execute.clj
index e2ad7a94c1b..52f29aa8ac0 100644
--- a/enterprise/backend/src/metabase_enterprise/transforms_python/execute.clj
+++ b/enterprise/backend/src/metabase_enterprise/transforms_python/execute.clj
@@ -8,6 +8,7 @@
[metabase.app-db.core :as app-db]
[metabase.driver :as driver]
[metabase.driver.connection :as driver.conn]
+ [metabase.driver.util :as driver.u]
[metabase.tracing.core :as tracing]
[metabase.transforms-base.interface :as transforms-base.i]
[metabase.transforms-base.util :as transforms-base.u]
@@ -105,7 +106,7 @@
(try
(let [message-log (base/empty-message-log)
{:keys [target owner_user_id creator_id] transform-id :id} transform
- {driver :engine :as db} (transforms-python.db/database (transforms-base.i/target-db-id transform))
+ {driver :engine :as db} (driver.u/lib-database (transforms-base.i/target-db-id transform))
run-user-id (if (and (= run-method :manual) user-id)
user-id
(or owner_user_id creator_id))
diff --git a/enterprise/backend/test/metabase_enterprise/action_v2/api_test.clj b/enterprise/backend/test/metabase_enterprise/action_v2/api_test.clj
index bf4947e0922..a825ac17e4a 100644
--- a/enterprise/backend/test/metabase_enterprise/action_v2/api_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/action_v2/api_test.clj
@@ -14,6 +14,7 @@
[metabase.driver.mysql :as mysql]
[metabase.driver.sql-jdbc :as sql-jdbc]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.query-processor.test :as qp]
[metabase.sync.core :as sync]
[metabase.test :as mt]
@@ -334,7 +335,7 @@
(mt/test-drivers (mt/normal-drivers-with-feature :actions/data-editing)
(mt/with-premium-features #{actions-feature-flag}
(action-v2.tu/with-actions-temp-db mutual-recursion-users-teams
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(sql.tx/add-fk-sql driver/*driver*
mutual-recursion-users-teams
{:table-name "user"}
@@ -447,7 +448,7 @@
field-id (t2/select-one-fn :id [:model/Field :id] :table_id table-id :name "o")
driver driver/*driver*
get-qp-state (fn [] (map #(zipmap [:id :o] %) (table-rows table-id)))
- get-db-state (fn [] (sql-jdbc/query driver (mt/id) {:select [:*] :from [table-name-kw]}))]
+ get-db-state (fn [] (sql-jdbc/query driver (lib.metadata/database (mt/metadata-provider)) {:select [:*] :from [table-name-kw]}))]
(t2/update! :model/Field field-id {:coercion_strategy coercion-strategy})
(testing "create"
(let [row {:o input}
@@ -734,7 +735,7 @@
:dbms_version {:flavor "MySQL"}
:settings {:database-enable-table-editing true
:database-enable-actions true}}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [writable-id (t2/select-one-fn :id :model/Table :db_id (:id database) :name "writable_table")]
(testing "the writable table syncs as editable despite the REVOKE on the other table"
(is (true? (t2/select-one-fn :is_writable :model/Table :id writable-id))))
@@ -780,7 +781,7 @@
:dbms_version {:flavor "MySQL"}
:settings {:database-enable-table-editing true
:database-enable-actions true}}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [table-id (t2/select-one-fn :id :model/Table :db_id (:id database) :name "optimistic_table")]
(testing "the table optimistically syncs as writable (the schema-level REVOKE line is ignored)"
(is (true? (t2/select-one-fn :is_writable :model/Table :id table-id))))
diff --git a/enterprise/backend/test/metabase_enterprise/action_v2/test_util.clj b/enterprise/backend/test/metabase_enterprise/action_v2/test_util.clj
index f0739efa992..a906fb06020 100644
--- a/enterprise/backend/test/metabase_enterprise/action_v2/test_util.clj
+++ b/enterprise/backend/test/metabase_enterprise/action_v2/test_util.clj
@@ -5,6 +5,7 @@
[clojure.walk :as walk]
[metabase.actions.test-util :as actions.tu]
[metabase.driver :as driver]
+ [metabase.driver.util :as driver.u]
[metabase.sync.core :as sync]
[metabase.test :as mt]
[toucan2.core :as t2]))
@@ -49,13 +50,13 @@
(defn sync-new-table!
"Syncs a new table by name, returns the table id."
[db table-name]
- (let [table (sync/create-table! db {:name table-name
- ;; todo figure out how to determine default schema from driver
- :schema (case (:engine db) :postgres "public" nil)
- :display_name table-name
- :field_order :database
- :is_writable true})]
- (sync/sync-fields-for-table! db table)
+ (let [table (sync/create-table! (driver.u/lib-database (:id db)) {:name table-name
+ ;; todo figure out how to determine default schema from driver
+ :schema (case (:engine db) :postgres "public" nil)
+ :display_name table-name
+ :field_order :database
+ :is_writable true})]
+ (sync/sync-fields-for-table! table)
(:id table)))
(defn- create-test-table! [db table-name column-map create-table-opts]
diff --git a/enterprise/backend/test/metabase_enterprise/advanced_config/file/databases_test.clj b/enterprise/backend/test/metabase_enterprise/advanced_config/file/databases_test.clj
index 97941ae8fb5..cd6befeee38 100644
--- a/enterprise/backend/test/metabase_enterprise/advanced_config/file/databases_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/advanced_config/file/databases_test.clj
@@ -314,7 +314,7 @@
;; Now, all the tables are hidden, so do another sync with empty auto-cruft-tables setting
;; and make sure they are still hidden:
(is (= {:hidden 8} (frequencies vis-types)))
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(testing "Hidden tables stay hidden"
(let [vis-types (t2/select-fn-vec :visibility_type :model/Table :db_id (u/the-id db))]
(is (= {:hidden 8} (frequencies vis-types)))))))
@@ -332,7 +332,7 @@
(is (future? sync-future))
;; wait for the sync to finish or crash out after 30 seconds
(u/deref-with-timeout sync-future (u/seconds->ms 30))
- (sync-metadata/sync-db-metadata! (t2/select-one :model/Database :name test-db-name))
+ (sync-metadata/sync-db-metadata! (:id (t2/select-one :model/Database :name test-db-name)))
(is (= 1 (t2/count :model/Database :name test-db-name)))
(let [db (t2/select-one :model/Database :name test-db-name)
tables (t2/select :model/Table :db_id (u/the-id db))
diff --git a/enterprise/backend/test/metabase_enterprise/audit_app/analytics_dev_test.clj b/enterprise/backend/test/metabase_enterprise/audit_app/analytics_dev_test.clj
index 569a6c28924..3f4bf7509ba 100644
--- a/enterprise/backend/test/metabase_enterprise/audit_app/analytics_dev_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/audit_app/analytics_dev_test.clj
@@ -6,6 +6,7 @@
[metabase-enterprise.audit-app.audit :as ee-audit]
[metabase.app-db.core :as mdb]
[metabase.driver :as driver]
+ [metabase.driver.util :as driver.u]
[metabase.test :as mt]
[metabase.test.fixtures :as fixtures]
[toucan2.core :as t2]))
@@ -197,7 +198,7 @@
"Get the set of field names that actually exist in the database for a given table."
[database table]
(let [table-metadata (driver/describe-table (driver/the-initialized-driver (:engine database))
- database
+ (driver.u/ensure-lib-database database)
table)]
(into #{} (map :name) (:fields table-metadata))))
diff --git a/enterprise/backend/test/metabase_enterprise/audit_app/audit_test.clj b/enterprise/backend/test/metabase_enterprise/audit_app/audit_test.clj
index 1f228e70313..453475dfb1d 100644
--- a/enterprise/backend/test/metabase_enterprise/audit_app/audit_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/audit_app/audit_test.clj
@@ -346,7 +346,7 @@
(testing "precondition: an interrupted host-adjust + schema sync duplicates the view row"
;; flip the schema to a value the host driver will not report, so the sync diff mismatches
(t2/update! :model/Table orig-id {:schema (if (nil? (:schema v-content)) "public" nil)})
- (sync/sync-database! (t2/select-one :model/Database :is_audit true) {:scan :schema})
+ (sync/sync-database! (:id (t2/select-one :model/Database :is_audit true)) {:scan :schema})
(is (= 2 (t2/count :model/Table :db_id audit/audit-db-id :name (:name v-content)))
"stale-schema sync should produce a duplicate pair")
(is (false? (t2/select-one-fn :active :model/Table :id orig-id))
@@ -384,7 +384,7 @@
:dataset_query {:database audit/audit-db-id :type :query :query {:source-table orig-id}}}]
(testing "precondition: a real Mode A duplicate pair, customer card on the new/active row"
(t2/update! :model/Table orig-id {:schema (if (nil? (:schema v-content)) "public" nil)})
- (sync/sync-database! (t2/select-one :model/Database :is_audit true) {:scan :schema})
+ (sync/sync-database! (:id (t2/select-one :model/Database :is_audit true)) {:scan :schema})
(let [rows (t2/select [:model/Table :id :active] :db_id audit/audit-db-id
{:where [:= [:lower :name] "v_content"]})
new-id (:id (first (filter :active rows)))]
@@ -470,10 +470,10 @@
;; "Boot 2": the full, un-stubbed pipeline, spying on audit-DB syncs.
(let [audit-db-syncs (atom 0)
real-sync (mt/original-fn #'sync.core/sync-database!)]
- (mt/with-dynamic-fn-redefs [sync.core/sync-database! (fn [db & [opts]]
- (when (:is_audit db)
+ (mt/with-dynamic-fn-redefs [sync.core/sync-database! (fn [db-id & [opts]]
+ (when (= db-id (:id audit-db))
(swap! audit-db-syncs inc))
- (real-sync db opts))]
+ (real-sync db-id opts))]
(mbc/ensure-audit-db-installed!))
(testing "the boot after an interrupted engine-changed sync re-syncs the audit DB"
(is (pos? @audit-db-syncs)))))))
diff --git a/enterprise/backend/test/metabase_enterprise/audit_app/permissions_test.clj b/enterprise/backend/test/metabase_enterprise/audit_app/permissions_test.clj
index 4df92716def..f9a79b9c6bb 100644
--- a/enterprise/backend/test/metabase_enterprise/audit_app/permissions_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/audit_app/permissions_test.clj
@@ -52,7 +52,7 @@
:db_id audit/audit-db-id
{:where [:in [:lower :name] audit-app.permissions/audit-db-view-names]})]
(when-not (some-> audit-view :name u/lower-case-en (str/starts-with? "v_"))
- (sync/sync-database! (t2/select-one :model/Database audit/audit-db-id)))
+ (sync/sync-database! (:id (t2/select-one :model/Database audit/audit-db-id))))
(is (partial=
{:status :completed}
(qp/process-query
diff --git a/enterprise/backend/test/metabase_enterprise/database_routing/connections_test.clj b/enterprise/backend/test/metabase_enterprise/database_routing/connections_test.clj
new file mode 100644
index 00000000000..05ae1f8a7c4
--- /dev/null
+++ b/enterprise/backend/test/metabase_enterprise/database_routing/connections_test.clj
@@ -0,0 +1,176 @@
+(ns metabase-enterprise.database-routing.connections-test
+ (:require
+ [clojure.java.jdbc :as jdbc]
+ [clojure.test :refer [deftest is testing]]
+ [metabase-enterprise.database-routing.common :as database-routing.common]
+ [metabase-enterprise.database-routing.e2e-test :as e2e]
+ [metabase-enterprise.test :as met]
+ [metabase.api.common :as api]
+ [metabase.app-db.core :as mdb]
+ [metabase.database-routing.core :as database-routing]
+ [metabase.driver.connection :as driver.conn]
+ [metabase.driver.settings :as driver.settings]
+ [metabase.driver.sql-jdbc.actions :as sql-jdbc.actions]
+ [metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
+ [metabase.driver.util :as driver.u]
+ [metabase.sync.core :as sync]
+ [metabase.sync.field-values :as sync.field-values]
+ [metabase.test :as mt]
+ [metabase.util :as u]
+ [metabase.warehouse-schema.models.field-values :as field-values]
+ [toucan2.core :as t2]))
+
+(set! *warn-on-reflection* true)
+
+(defn- query-h2
+ "Rows of `sql` run straight against the H2 file of `db`."
+ [db sql]
+ (sql-jdbc.execute/do-with-connection-with-options
+ :h2 (mdb/spec :h2 (:details db)) {}
+ (fn [^java.sql.Connection conn]
+ (jdbc/query {:connection conn} [sql]))))
+
+(defn- owner-of-row-1
+ "The `owner` of row 1 of `acct` in the H2 file of `db`."
+ [db]
+ (:owner (first (query-h2 db "SELECT \"owner\" FROM \"acct\" WHERE \"id\" = 1"))))
+
+(defn- set-owner-of-row-1!
+ "Sets the `owner` of row 1 of `acct` through the connection that implicit actions open on `router-id`."
+ [router-id owner]
+ (sql-jdbc.actions/do-with-jdbc-transaction
+ router-id
+ (fn [^java.sql.Connection conn]
+ (jdbc/execute! {:connection conn} ["UPDATE \"acct\" SET \"owner\" = ? WHERE \"id\" = 1" owner] {:transaction? false}))))
+
+(deftest implicit-actions-write-to-destination-test
+ (mt/with-premium-features #{:database-routing}
+ (binding [driver.settings/*allow-testing-h2-connections* true]
+ (met/with-user-attributes! :rasta {"db_name" "destination-db-1"}
+ (e2e/with-routing-setup! [router-db [[dest-1 "destination-db-1"]]]
+ (doseq [db [router-db dest-1]]
+ (e2e/execute-statement! db "CREATE TABLE \"acct\" (\"id\" INT PRIMARY KEY, \"owner\" TEXT NOT NULL)")
+ (e2e/execute-statement! db "GRANT ALL ON \"acct\" TO GUEST"))
+ (e2e/execute-statement! router-db "INSERT INTO \"acct\" VALUES (1, 'tenant-b')")
+ (e2e/execute-statement! dest-1 "INSERT INTO \"acct\" VALUES (1, 'tenant-a')")
+ (let [router-id (u/the-id router-db)]
+ (mt/with-temp [:model/DatabaseRouter _ {:database_id router-id :user_attribute "db_name"}]
+ (mt/with-current-user (mt/user->id :rasta)
+ (set-owner-of-row-1! router-id "by-rasta"))
+ (testing "a routed user's implicit action writes with the destination's credentials"
+ (is (= {:router "tenant-b" :destination "by-rasta"}
+ {:router (owner-of-row-1 router-db) :destination (owner-of-row-1 dest-1)})))
+ (mt/with-current-user (mt/user->id :crowberto)
+ (set-owner-of-row-1! router-id "by-crowberto"))
+ (testing "an admin without the attribute writes with the router's credentials"
+ (is (= {:router "by-crowberto" :destination "by-rasta"}
+ {:router (owner-of-row-1 router-db) :destination (owner-of-row-1 dest-1)}))))))))))
+
+(defn- owner-via-connection
+ "The `owner` of row 1 of `acct`, read through a connection for the Database with `database-id`."
+ [database-id]
+ (sql-jdbc.execute/do-with-connection-with-options
+ :h2 database-id {}
+ (fn [^java.sql.Connection conn]
+ (:owner (first (jdbc/query {:connection conn} ["SELECT \"owner\" FROM \"acct\" WHERE \"id\" = 1"]))))))
+
+(deftest connections-go-to-users-destination-test
+ (mt/with-premium-features #{:database-routing}
+ (binding [driver.settings/*allow-testing-h2-connections* true]
+ (met/with-user-attributes! :rasta {"db_name" "destination-db-1"}
+ (e2e/with-routing-setup! [router-db [[dest-1 "destination-db-1"]]]
+ (doseq [[db owner] [[router-db "router"] [dest-1 "destination"]]]
+ (e2e/execute-statement! db "CREATE TABLE \"acct\" (\"id\" INT PRIMARY KEY, \"owner\" TEXT NOT NULL)")
+ (e2e/execute-statement! db "GRANT ALL ON \"acct\" TO GUEST")
+ (e2e/execute-statement! db (format "INSERT INTO \"acct\" VALUES (1, '%s')" owner)))
+ (let [router-id (u/the-id router-db)]
+ (mt/with-temp [:model/DatabaseRouter _ {:database_id router-id :user_attribute "db_name"}]
+ (testing "a routed user's connection to the router goes to their destination"
+ (mt/with-current-user (mt/user->id :rasta)
+ (is (= "destination" (owner-via-connection router-id)))
+ (is (= (u/the-id dest-1)
+ (::driver.conn/database-id (driver.conn/effective-details (driver.u/lib-database router-id)))))))
+ (testing "an admin without the attribute connects to the router"
+ (mt/with-current-user (mt/user->id :crowberto)
+ (is (= "router" (owner-via-connection router-id)))))
+ (testing "with no user bound, the connection is refused"
+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"Anonymous users cannot access a database with routing enabled"
+ (owner-via-connection router-id))))
+ (testing "with no user bound, allowed anonymous access goes to the router"
+ (database-routing/with-anonymous-router-access
+ (is (= "router" (owner-via-connection router-id)))))
+ (testing "with only the user id cleared, the query and its connection agree that no user is bound"
+ (mt/with-current-user (mt/user->id :rasta)
+ (binding [api/*current-user-id* nil]
+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"Anonymous users cannot access a database with routing enabled"
+ (database-routing.common/router-db->destination-db-id (driver.u/lib-database router-id))))
+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"Anonymous users cannot access a database with routing enabled"
+ (owner-via-connection router-id)))
+ (database-routing/with-anonymous-router-access
+ (is (nil? (database-routing.common/router-db->destination-db-id (driver.u/lib-database router-id))))
+ (is (= "router" (owner-via-connection router-id)))))))
+ (testing "allowed anonymous access goes to the router even inside a routed user's request"
+ (mt/with-current-user (mt/user->id :rasta)
+ (database-routing/with-anonymous-router-access
+ (is (= "router" (owner-via-connection router-id))))))
+ (testing "a routed user can't connect to a lapsed router"
+ (mt/with-premium-features #{}
+ (mt/with-current-user (mt/user->id :rasta)
+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"Database Routing"
+ (owner-via-connection router-id))))
+ (testing "and destinations stay reachable only through their router"
+ (mt/with-current-user (mt/user->id :crowberto)
+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"(?i)cannot query a destination database directly"
+ (owner-via-connection (u/the-id dest-1))))))
+ (is (database-routing/db-routing-enabled? (driver.u/lib-database router-id))))))))))))
+
+(deftest routed-users-sync-router-itself-test
+ (testing "a routed user's sync, analysis and field value scans read the router itself, so its shared metadata stays the router's"
+ (mt/with-premium-features #{:database-routing}
+ (binding [driver.settings/*allow-testing-h2-connections* true]
+ (met/with-user-attributes! :rasta {"db_name" "destination-db-1"}
+ (e2e/with-routing-setup! [router-db [[dest-1 "destination-db-1"]]]
+ (e2e/execute-statement! router-db "CREATE TABLE \"acct\" (\"id\" INT PRIMARY KEY, \"owner\" TEXT)")
+ (e2e/execute-statement! router-db "INSERT INTO \"acct\" VALUES (1, 'router-owner')")
+ (e2e/execute-statement! dest-1 "CREATE TABLE \"acct\" (\"id\" INT PRIMARY KEY, \"owner\" TEXT, \"tenant_secret\" TEXT)")
+ (e2e/execute-statement! dest-1 "INSERT INTO \"acct\" VALUES (1, 'tenant-owner', 'secret')")
+ (sync/sync-database! (:id router-db) {:scan :schema})
+ (let [router-id (u/the-id router-db)
+ table (t2/select-one :model/Table :db_id router-id :name "acct")
+ owner (t2/select-one :model/Field :table_id (:id table) :name "owner")]
+ (field-values/create-or-update-full-field-values! owner)
+ (mt/with-temp [:model/DatabaseRouter _ {:database_id router-id :user_attribute "db_name"}]
+ (mt/with-current-user (mt/user->id :rasta)
+ (doseq [[step f] {"table sync" #(sync/sync-table! table)
+ "metadata sync" #(sync/sync-db-metadata! router-id)
+ "analysis" #(sync/analyze-db! router-id)
+ "refingerprint" #(sync/refingerprint-field! owner)
+ "table values" #(sync/update-field-values-for-table! table)
+ "field values" #(field-values/create-or-update-full-field-values! owner)}]
+ (testing step
+ (f)))))
+ (is (= #{"id" "owner"}
+ (t2/select-fn-set :name :model/Field :table_id (:id table) :active true)))
+ (is (= ["router-owner"] (:values (field-values/get-latest-full-field-values (:id owner))))))))))))
+
+(deftest routed-users-keep-shared-values-test
+ (testing "a routed user's on-demand values refresh reads the router itself"
+ (mt/with-premium-features #{:database-routing}
+ (binding [driver.settings/*allow-testing-h2-connections* true]
+ (met/with-user-attributes! :rasta {"db_name" "destination-db-1"}
+ (e2e/with-routing-setup! [router-db [[dest-1 "destination-db-1"]]]
+ (doseq [[db owner] [[router-db "router-owner"] [dest-1 "tenant-owner"]]]
+ (e2e/execute-statement! db "CREATE TABLE \"acct\" (\"id\" INT PRIMARY KEY, \"owner\" TEXT)")
+ (e2e/execute-statement! db "GRANT ALL ON \"acct\" TO GUEST")
+ (e2e/execute-statement! db (format "INSERT INTO \"acct\" VALUES (1, '%s')" owner)))
+ (sync/sync-database! (:id router-db) {:scan :schema})
+ (let [router-id (u/the-id router-db)
+ table (t2/select-one :model/Table :db_id router-id :name "acct")
+ owner (t2/select-one :model/Field :table_id (:id table) :name "owner")]
+ (t2/update! :model/Field (:id owner) {:has_field_values "list"})
+ (field-values/create-or-update-full-field-values! (t2/select-one :model/Field (:id owner)))
+ (t2/update! :model/Database router-id {:is_on_demand true})
+ (mt/with-temp [:model/DatabaseRouter _ {:database_id router-id :user_attribute "db_name"}]
+ (mt/with-current-user (mt/user->id :rasta)
+ (sync.field-values/update-field-values-for-on-demand-dbs! [(:id owner)]))
+ (is (= ["router-owner"] (:values (field-values/get-latest-full-field-values (:id owner)))))))))))))
diff --git a/enterprise/backend/test/metabase_enterprise/database_routing/e2e_test.clj b/enterprise/backend/test/metabase_enterprise/database_routing/e2e_test.clj
index 9a5c449f78a..6d4689c78a8 100644
--- a/enterprise/backend/test/metabase_enterprise/database_routing/e2e_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/database_routing/e2e_test.clj
@@ -12,10 +12,10 @@
[metabase-enterprise.test :as met]
[metabase.app-db.core :as mdb]
[metabase.config.core :as config]
- [metabase.database-routing.core :as database-routing]
[metabase.driver :as driver]
[metabase.driver.settings :as driver.settings]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
+ [metabase.driver.util :as driver.u]
[metabase.permissions.models.permissions-group :as perms-group]
[metabase.query-processor.test :as qp]
[metabase.sync.core :as sync]
@@ -90,7 +90,7 @@
~(wrap-destinations (rest dests) body)))))))]
;; Router: a normal synced DB, exactly as `with-temp-dbs!` builds one.
`(with-temp-dbs! [~router-binding]
- (sync/sync-database! ~router-binding)
+ (sync/sync-database! (:id ~router-binding))
~(wrap-destinations destinations body))))
(deftest destination-databases-get-used
@@ -211,12 +211,9 @@
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"(?i)cannot query a destination database directly"
- (common/check-allowed-access! dest-id))))
- (testing "destination access is allowed inside a routing-on context (legit routed query)"
- (is (nil? (database-routing/with-database-routing-on
- (common/check-allowed-access! dest-id)))))
+ (common/check-allowed-access! (driver.u/lib-database dest-id)))))
(testing "the router database itself is not treated as a forbidden destination"
- (is (nil? (common/check-allowed-access! router-id))))))))))
+ (is (nil? (common/check-allowed-access! (driver.u/lib-database router-id)))))))))))
(deftest an-error-is-thrown-if-user-attribute-is-missing-or-no-match
(mt/with-premium-features #{:database-routing}
@@ -428,7 +425,7 @@
{:details (merge (:details router)
(router-dataset-details driver/*driver*))})
(let [router (t2/select-one :model/Database :id (u/the-id router))]
- (sync/sync-database! router {:scan :schema})
+ (sync/sync-database! (:id router) {:scan :schema})
;; Load the physical routed warehouse only for its connection-details + name; discard the normal
;; `:model/Database` row `mt/dataset` registers. The destination is inserted prod-shaped below.
(let [[routed-name routed-details]
@@ -486,7 +483,7 @@
(t2/update! :model/Database (u/the-id router)
{:details (assoc (:details router) :dbname nil)})
(let [router (t2/select-one :model/Database :id (u/the-id router))]
- (sync/sync-database! router {:scan :schema})
+ (sync/sync-database! (:id router) {:scan :schema})
;; Load the physical routed warehouse only for its connection-details + name; discard the normal
;; `:model/Database` row `mt/dataset` registers. The destination is inserted prod-shaped below.
(let [[routed-name routed-details]
diff --git a/enterprise/backend/test/metabase_enterprise/database_routing/embedding_test.clj b/enterprise/backend/test/metabase_enterprise/database_routing/embedding_test.clj
index 5d613efb472..db4cd758c55 100644
--- a/enterprise/backend/test/metabase_enterprise/database_routing/embedding_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/database_routing/embedding_test.clj
@@ -150,3 +150,29 @@
response (mt/user-http-request :crowberto :get 202 (str "preview_embed/card/" token "/query"))]
(is (= [["router-data"]] (mt/rows response))
"Preview should show router-database data even though the admin's attribute routes to a destination")))))))))))
+
+(deftest guest-and-public-access-ignore-session-test
+ (testing "guest embeds and public links behave the same with or without a routed user's session"
+ (mt/with-premium-features #{:database-routing}
+ (binding [driver.settings/*allow-testing-h2-connections* true]
+ (met/with-user-attributes! :crowberto {"db_name" "destination-db"}
+ (with-routing-setup! [router-db [[destination-db "destination-db"]]]
+ (let [uuid (str (random-uuid))]
+ (mt/with-temporary-setting-values [enable-public-sharing true]
+ (mt/with-temp [:model/DatabaseRouter _ {:database_id (u/the-id router-db)
+ :user_attribute "db_name"}
+ :model/Card card {:enable_embedding true
+ :public_uuid uuid
+ :dataset_query {:database (u/the-id router-db)
+ :type :query
+ :query {:source-table (t2/select-one-pk :model/Table :db_id (u/the-id router-db))}}}]
+ (execute-statement! router-db "INSERT INTO \"my_database_name\" (str) VALUES ('router-data')")
+ (execute-statement! destination-db "INSERT INTO \"my_database_name\" (str) VALUES ('destination-data')")
+ (with-embedding-enabled-and-new-secret-key!
+ (testing "a guest embed shows the router even with a routed user's session"
+ (is (= [["router-data"]]
+ (mt/rows (mt/user-http-request :crowberto :get 202 (str "embed/card/" (card-token card) "/query")))))))
+ (testing "a public link on a router is refused"
+ (doseq [response [(client/client :get (str "public/card/" uuid "/query"))
+ (mt/user-http-request :crowberto :get (str "public/card/" uuid "/query"))]]
+ (is (=? {:status "failed"} response)))))))))))))
diff --git a/enterprise/backend/test/metabase_enterprise/dependencies/core_test.clj b/enterprise/backend/test/metabase_enterprise/dependencies/core_test.clj
deleted file mode 100644
index f57c94c5f12..00000000000
--- a/enterprise/backend/test/metabase_enterprise/dependencies/core_test.clj
+++ /dev/null
@@ -1,292 +0,0 @@
-(ns metabase-enterprise.dependencies.core-test
- (:require
- [clojure.test :refer [deftest is testing]]
- [medley.core :as m]
- [metabase-enterprise.dependencies.core :as dependencies]
- [metabase-enterprise.dependencies.test-util :as deps.tu]
- [metabase.graph.core :as graph]
- [metabase.lib.core :as lib]
- [metabase.lib.metadata :as lib.metadata]
- [metabase.lib.test-util :as lib.tu]
- [metabase.test :as mt]
- [metabase.util.malli :as mu]))
-
-(defn- only-missing-column-errors
- "Filters an error map to only include :missing-column type errors.
- SQLGlot and Macaw produce different :missing-table-alias errors,
- but only :missing-column errors are used in production."
- [errors]
- (-> errors
- (update :card (fn [m]
- (into {}
- (keep (fn [[k v]]
- (let [filtered (into #{} (filter #(= (:type %) :missing-column) v))]
- (when (seq filtered)
- [k filtered]))))
- m)))
- (update :transform (fn [m]
- (into {}
- (keep (fn [[k v]]
- (let [filtered (into #{} (filter #(= (:type %) :missing-column) v))]
- (when (seq filtered)
- [k filtered]))))
- m)))))
-
-(defn- testbed
- "A `MetadataProvider` with a chain of MBQL cards and transforms for testing."
- []
- (let [mp (deps.tu/default-metadata-provider)
- query1 (-> (lib/query mp (lib.metadata/table (mt/metadata-provider) (mt/id :orders)))
- (lib/expression "Tax Rate" (lib// (lib.metadata/field (mt/metadata-provider) (mt/id :orders :tax))
- (lib.metadata/field (mt/metadata-provider) (mt/id :orders :subtotal)))))
- mp (lib.tu/metadata-provider-with-card-from-query mp 101 query1)
- card1 (lib.metadata/card mp 101)
- card1-query (lib/query mp card1)
- card1-cols (lib/returned-columns card1-query)
- tax-rate (m/find-first #(= (:lib/desired-column-alias %) "Tax Rate") card1-cols)
- query2 (-> card1-query
- (lib/filter (lib/> tax-rate 0.06)))
- mp (lib.tu/metadata-provider-with-card-from-query mp 102 query2)
-
- ;; A transform that depends on card1
- tf1-query (lib/query mp card1)
- transform1 {:id 30
- :name "MBQL Transform"
- :source {:query tf1-query}
- :target {:schema "Transformed"
- :name "output_tf30"}}
- tf1-output (-> (lib.metadata/table (mt/metadata-provider) (mt/id :orders))
- (assoc :id 1234567
- :schema "Transformed"
- :name "output_tf30"
- :display-name "Transform 30 Output"))
- tf1-cols (map-indexed (fn [i col]
- (assoc col
- :id (+ 123456700 i)
- :table-id 1234567
- :lib/source :source/table-defaults))
- card1-cols)
- mp (lib.tu/mock-metadata-provider mp {:tables [tf1-output]
- :fields tf1-cols
- :transforms [transform1]})
-
- ;; An MBQL card that consumes the output table of transform1.
- ;; References some fields but does not change the output columns.
- tf1-consumer (-> (lib/query mp (lib.metadata/table mp 1234567))
- (lib/filter (lib/= (m/find-first #(= (:name %) "Tax Rate") tf1-cols)
- 101)))
- mp (lib.tu/metadata-provider-with-card-from-query mp 301 tf1-consumer)
-
- ;; SQL cards and snippets
- ;; Card 7 from [[deps.tu/default-metadata-provider]] uses the "outer" snippet (2).
- ;; This adds a SQL transform that consumes card 7, and a card that selects from that transform.
- tf2-query (lib/native-query mp "SELECT * FROM {{#7}}")
- transform2 {:id 31
- :name "SQL transform"
- :source {:query (lib/->legacy-MBQL tf2-query)}
- :target {:schema "Transformed"
- :name "output_tf31"}}
- tf2-output (-> (lib.metadata/table (mt/metadata-provider) (mt/id :orders))
- (assoc :id 1234568
- :schema "Transformed"
- :name "output_tf31"
- :display-name "Transform 31 Output"))
- ;; Output columns for transform2 are derived from card 7. That's "SELECT * FROM {{snippet: outer}}";
- ;; the outer snippet references {{snippet: inner}}; the inner snippet is {{#card-ref}} to card 1;
- ;; card 1 is a basic MBQL query of products.
- card7 (-> (lib.metadata/card mp 7)
- (assoc :result-metadata (:result-metadata (:products/native (lib.tu/mock-cards)))))
- mp (lib.tu/mock-metadata-provider mp {:cards [card7]})
- tf2-cols (->> card7
- (lib/query mp)
- lib/returned-columns
- (map-indexed (fn [i col]
- (assoc col
- :id (+ 123456800 i)
- :table-id 1234568
- :lib/source :source/table-defaults))))
- mp (lib.tu/mock-metadata-provider mp {:tables [tf2-output]
- :fields tf2-cols
- :transforms [transform2]})
-
- ;; MBQL card consuming tf2-output.
- query3 (-> (lib/query mp tf2-output)
- (lib/filter (lib/< (m/find-first #(= (:name %) "RATING") tf2-cols)
- 3)))
- ;; SQL card consuming tf2-output.
- query4 (lib/native-query mp "SELECT * FROM Transformed.output_tf31")
- mp (-> mp
- (lib.tu/metadata-provider-with-card-from-query 303 query3)
- (lib.tu/metadata-provider-with-card-from-query 304 query4))]
- {:provider mp
- ;; NOTE: This is a *downstream* graph! So it's a map of keys to those entities which depend on them.
- :graph (graph/in-memory {[:card 101] #{[:card 102]
- [:transform 30]}
- [:transform 30] #{[:table 1234567]}
- [:table 1234567] #{[:card 301]}
-
- [:snippet 1] #{[:snippet 2]}
- [:snippet 2] #{[:card 7]}
- [:card 7] #{[:transform 31]}
- [:transform 31] #{[:table 1234568]}
- [:table 1234568] #{[:card 303]
- [:card 304]}})
- :mbql-base card1
- :mbql-dependent (lib.metadata/card mp 102)
- :mbql-transform transform1
- :mbql-transform-output tf1-output
- :mbql-transform-cols (m/index-by :name tf1-cols)
- :mbql-transform-consumer (lib.metadata/card mp 301)
-
- :snippet-inner (lib.metadata/native-query-snippet mp 1)
- :snippet-outer (lib.metadata/native-query-snippet mp 2)
- :sql-base (lib.metadata/card mp 7)
- :sql-transform transform2
- :sql-transform-output tf2-output
- :sql-transform-mbql-consumer (lib.metadata/card mp 303)
- :sql-transform-sql-consumer (lib.metadata/card mp 304)}))
-
-(comment
- *e
- (:mbql-dependent (testbed))
- (:mbql-transform-consumer (testbed)))
-
-(deftest ^:parallel basic-mbql-card-test
- (testing "when changing an MBQL card with dependents"
- (let [{:keys [provider graph mbql-base]
- {downstream-card-id :id} :mbql-dependent
- {transformed-card-id :id} :mbql-transform-consumer} (testbed)]
- (testing "a column that no longer exists will cause errors when referenced"
- (let [card' (-> mbql-base
- (assoc :dataset-query
- (-> (lib/query provider (lib.metadata/table (mt/metadata-provider) (mt/id :orders)))
- ;; ridiculous
- (lib/expression "Sales Taxes" (lib// (lib.metadata/field (mt/metadata-provider) (mt/id :orders :tax))
- (lib.metadata/field (mt/metadata-provider) (mt/id :orders :subtotal))))))
- (dissoc :result-metadata))
- errors (dependencies/errors-from-proposed-edits {:card [card']}
- :base-provider provider
- :graph graph)]
- (is (=? {:card {downstream-card-id #{{:type :missing-column
- :name "Tax Rate"
- :source-entity-type :card
- :source-entity-id 101}}
- transformed-card-id #{{:type :missing-column
- :name "Tax Rate"
- :source-entity-type :table
- :source-entity-id 1234567}}}}
- errors))
- (is (= [:card] (keys errors)))
- (is (= #{downstream-card-id transformed-card-id} (set (keys (:card errors)))))))
- (testing "changing something unrelated will cause no errors"
- (let [card' (-> mbql-base
- (update :dataset-query lib/filter (lib/> (lib.metadata/field (mt/metadata-provider) (mt/id :orders :quantity))
- 100))
- (dissoc :result-metadata))]
- (is (= {} (dependencies/errors-from-proposed-edits {:card [card']}
- :base-provider provider
- :graph graph))))))))
-(deftest ^:parallel model-edit-with-action-dependent-test
- (testing "editing a card that an action depends on doesn't throw"
- (let [{:keys [provider mbql-base]} (testbed)
- graph (graph/in-memory {[:card 101] #{[:action 900]}})
- card' (-> mbql-base
- (update :dataset-query lib/filter (lib/> (lib.metadata/field (mt/metadata-provider) (mt/id :orders :quantity))
- 100))
- (dissoc :result-metadata))]
- (is (= {} (dependencies/errors-from-proposed-edits {:card [card']}
- :base-provider provider
- :graph graph))))))
-
-(deftest ^:parallel card-edit-with-data-app-dependent-test
- (testing "editing a card that a data app depends on reports no errors"
- (let [{:keys [provider mbql-base]} (testbed)
- graph (graph/in-memory {[:card 101] #{[:data-app 900]}})
- card' (-> mbql-base
- (update :dataset-query lib/filter (lib/> (lib.metadata/field (mt/metadata-provider) (mt/id :orders :quantity))
- 100))
- (dissoc :result-metadata))]
- (is (= {} (dependencies/errors-from-proposed-edits {:card [card']}
- :base-provider provider
- :graph graph))))))
-
-(deftest ^:parallel sql-snippet->card->transform->cards-test
- (testing "changing a snippet correctly finds downstream errors when asked"
- (let [{:keys [provider graph snippet-inner]
- {transformed-mbql-card-id :id} :sql-transform-mbql-consumer} (testbed)]
- (testing "when breaking the inner snippet with a nonexistent table"
- (let [snippet' (assoc snippet-inner
- :content "nonexistent_table"
- :template-tags {})
- errors (dependencies/errors-from-proposed-edits {:snippet [snippet']}
- :base-provider provider
- :graph graph
- :include-native? true)]
- (is (= #{:card} (set (keys errors))))
- ;; That breaks (1) the SQL card which uses the snippets, (2) the transforms, (3) both the MBQL and (4) SQL
- ;; queries that consume the transform's table.
- ;; We only check :missing-column errors since SQLGlot and Macaw produce different
- ;; :missing-table-alias errors, and only :missing-column errors are used in production.
- (is (= {:card {transformed-mbql-card-id #{{:type :missing-column
- :name "RATING"
- :source-entity-type :table
- :source-entity-id 1234568}}}
- :transform {}}
- (only-missing-column-errors errors))))))))
-
-(deftest ^:parallel sql-snippet-without-including-native-test
- (testing "changing a snippet correctly ignores downstream errors"
- (let [{:keys [provider graph snippet-inner]} (testbed)]
- (testing "when breaking the inner snippet with a nonexistent table"
- (let [snippet' (assoc snippet-inner
- :content "nonexistent_table"
- :template-tags {})
- errors (dependencies/errors-from-proposed-edits {:snippet [snippet']}
- :base-provider provider
- :graph graph)]
- ;; Because we are changing a snippet, don't check anything downstream
- (is (= {} errors)))))))
-
-(deftest ^:parallel check-entity-exception-test
- (testing "when a dependent card throws during compilation, errors-from-proposed-edits catches it
- and returns a validation-exception-error instead of crashing (#GHY-3151)"
- (mu/disable-enforcement
- (let [{:keys [provider mbql-base]} (testbed)
- ;; Add a card with a broken dataset-query that will throw during check-entity
- broken-card {:lib/type :metadata/card
- :id 999
- :database-id (:id (lib.metadata/database provider))
- :name "broken-card"
- :dataset-query nil}
- provider (lib.tu/mock-metadata-provider provider {:cards [broken-card]})
- ;; Graph where card 101 -> card 999 (broken card depends on the base card)
- graph (graph/in-memory {[:card 101] #{[:card 999]}})
- card' (-> mbql-base
- (update :dataset-query lib/filter (lib/> (lib.metadata/field (mt/metadata-provider) (mt/id :orders :quantity)) 100))
- (dissoc :result-metadata))
- errors (dependencies/errors-from-proposed-edits {:card [card']}
- :base-provider provider
- :graph graph)]
- (is (contains? errors :card))
- (is (contains? (:card errors) 999))
- (is (= #{:validation-exception-error}
- (into #{} (map :type) (get-in errors [:card 999]))))))))
-
-(deftest ^:parallel self-referential-dependency-test
- (testing "errors-from-proposed-edits terminates when the dependency graph has a self-loop (#70452)"
- (let [{:keys [provider mbql-base]} (testbed)
- ;; Graph where dashboard 999 depends on card 101, and dashboard 999 depends on itself
- ;; (as happens with click behavior linking to another tab on the same dashboard)
- graph (graph/in-memory {[:card 101] #{[:dashboard 999]}
- [:dashboard 999] #{[:dashboard 999]}})
- card' (-> mbql-base
- (update :dataset-query lib/filter (lib/> (lib.metadata/field (mt/metadata-provider) (mt/id :orders :quantity)) 100))
- (dissoc :result-metadata))
- fut (future (dependencies/errors-from-proposed-edits {:card [card']}
- :base-provider provider
- :graph graph))
- result (deref fut 1000 ::timeout)]
- (when (= result ::timeout)
- (future-cancel fut))
- (is (not= ::timeout result) "errors-from-proposed-edits hung (possible infinite loop)")
- (is (= {} result)))))
diff --git a/enterprise/backend/test/metabase_enterprise/dependencies/metadata_provider_test.clj b/enterprise/backend/test/metabase_enterprise/dependencies/metadata_provider_test.clj
deleted file mode 100644
index d811600e2c6..00000000000
--- a/enterprise/backend/test/metabase_enterprise/dependencies/metadata_provider_test.clj
+++ /dev/null
@@ -1,129 +0,0 @@
-(ns metabase-enterprise.dependencies.metadata-provider-test
- (:require
- [clojure.test :refer [deftest is testing]]
- [medley.core :as m]
- [metabase-enterprise.dependencies.metadata-provider :as deps.mp]
- [metabase.lib.core :as lib]
- [metabase.lib.metadata :as lib.metadata]
- [metabase.lib.test-metadata :as meta]
- [metabase.lib.test-util :as lib.tu]
- [metabase.lib.test-util.macros :as lib.tu.macros]))
-
-(deftest basic-cards-test
- (testing "overriding a single card, it is returned correctly"
- (let [card (:orders (lib.tu/mock-cards))
- mp (deps.mp/override-metadata-provider
- {:base-provider (lib.tu/metadata-provider-with-mock-card card)})]
- (deps.mp/add-override mp :card (:id card)
- (-> card
- (assoc-in [:dataset-query :query :fields]
- (lib.tu.macros/$ids orders
- [$subtotal $created-at]))
- (dissoc :result-metadata)))
- (is (=? [(meta/field-metadata :orders :subtotal)
- (meta/field-metadata :orders :created-at)]
- (->> (lib.metadata/card mp (:id card))
- (lib/query mp)
- lib/returned-columns))))))
-
-(defn- replace-query [new-query card]
- (-> card
- (assoc :dataset-query (lib/->legacy-MBQL new-query))
- (dissoc :result-metadata)))
-
-(deftest upstream-card-test
- (testing "overriding an upstream card that's consumed by a downstream card"
- (let [upstream (-> (lib/query meta/metadata-provider (meta/table-metadata :orders))
- (lib/breakout (meta/field-metadata :products :category))
- (lib/aggregate (lib/count)))
- mp1 (lib.tu/metadata-provider-with-card-from-query 1 upstream)
- card1 (lib.metadata/card mp1 1)
- by-name (m/index-by :name (lib/returned-columns (lib/query mp1 card1)))
- downstream (-> (lib/query mp1 card1)
- (lib/filter (lib/= (get by-name "count") 1)))
- mp (deps.mp/override-metadata-provider
- {:base-provider (lib.tu/metadata-provider-with-card-from-query mp1 2 downstream)})]
- (deps.mp/add-override mp :card 1
- (-> upstream
- (lib/breakout (lib/with-temporal-bucket (meta/field-metadata :orders :created-at) :day))
- (replace-query card1)))
- ;; NOTE: This line is important: this is what would happen to all the dependents returned by the graph.
- ;; There's no update (`nil`) but it still includes the downstream stuff in the scope of the
- ;; OverrideMetadataProvider.
- (deps.mp/add-override mp :card 2 nil)
- (testing "upstream card's columns are correct"
- (is (=? [{:name "CATEGORY"}
- {:name "CREATED_AT"}
- {:name "count"}]
- (->> (lib.metadata/card mp 1)
- (lib/query mp)
- lib/returned-columns))))
- (testing "downstream card (which selects all) is also correct"
- (is (=? [{:name "CATEGORY"}
- {:name "CREATED_AT"}
- {:name "count"}]
- (->> (lib.metadata/card mp 2)
- (lib/query mp)
- lib/returned-columns)))))))
-
-(deftest card->transform->card-test
- (testing "changing the columns of an upstream card that is transformed and then used by a card"
- (let [upstream (-> (lib/query meta/metadata-provider (meta/table-metadata :orders))
- (lib/breakout (meta/field-metadata :products :category))
- (lib/aggregate (lib/count)))
- mp1 (lib.tu/metadata-provider-with-card-from-query 1 upstream)
- card1 (lib.metadata/card mp1 1)
- card1-cols (-> (lib/query mp1 card1)
- lib/returned-columns)
-
- transform {:lib/type :metadata/transform
- :id 123
- :name "The Transform"
- :source {:type :query
- :query (-> (lib/query mp1 (lib.metadata/card mp1 1))
- lib/->legacy-MBQL)}
- :target {:type "table"
- :schema "transform_schema_123"
- :name "autobots"}}
- mp2 (lib.tu/mock-metadata-provider
- mp1 {:tables [{:id 123000000
- :lib/type :metadata/table
- :name "autobots"
- :schema "transform_schema_123"
- :db-id (meta/id)
- :visibility-type nil
- :active true}]
- :fields (map-indexed (fn [i col]
- (assoc col
- :id (+ 123000001 i)
- :table-id 123000000))
- card1-cols)})
-
- xform-cols (m/index-by :name (lib.metadata/fields mp2 123000000))
- downstream (-> (lib/query mp2 (lib.metadata/table mp2 123000000))
- (lib/filter (lib/= (get xform-cols "count") 1)))
- mp (deps.mp/override-metadata-provider
- {:base-provider (lib.tu/metadata-provider-with-card-from-query mp2 2 downstream)})]
- (deps.mp/add-override mp :card 1
- (-> upstream
- (lib/remove-clause (first (lib/breakouts upstream)))
- (lib/breakout (lib/with-temporal-bucket (meta/field-metadata :orders :created-at) :day))
- (replace-query card1)))
- ;; NOTE: These lines are important: this is what would happen to all the dependents returned by the graph.
- ;; There's no update (`nil`) but it still includes the downstream stuff in the scope of the
- ;; OverrideMetadataProvider.
- (deps.mp/add-override mp :card 2 nil)
- (deps.mp/add-override mp :transform 123 transform)
- (testing "upstream card's columns are correct"
- (is (=? [{:name "CREATED_AT"}
- {:name "count"}]
- (->> (lib.metadata/card mp 1)
- (lib/query mp)
- lib/returned-columns))))
- (testing "downstream card (which selects all) is also correct"
- (is (=? [{:name "CREATED_AT"}
- {:name "count"}]
- (->> (lib.metadata/card mp 2)
- (lib/query mp)
- lib/returned-columns
- (sort-by :name))))))))
diff --git a/enterprise/backend/test/metabase_enterprise/dependencies/native_validation_test.clj b/enterprise/backend/test/metabase_enterprise/dependencies/native_validation_test.clj
index 171abeb27d5..c36579e1d46 100644
--- a/enterprise/backend/test/metabase_enterprise/dependencies/native_validation_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/dependencies/native_validation_test.clj
@@ -626,8 +626,6 @@
(mt/with-temp [:model/DatabaseRouter _ {:database_id (mt/id)
:user_attribute "db_name"}]
(testing "validate-native-query doesn't throw"
- ;; Without the fix, this would throw:
- ;; "Anonymous users cannot access a database with routing enabled."
(is (set? (deps.native-validation/validate-native-query
driver
(lib/native-query mp "SELECT * FROM ORDERS")))))
diff --git a/enterprise/backend/test/metabase_enterprise/dependencies/sync_test.clj b/enterprise/backend/test/metabase_enterprise/dependencies/sync_test.clj
index deb1e3bbfb3..4b4d2736529 100644
--- a/enterprise/backend/test/metabase_enterprise/dependencies/sync_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/dependencies/sync_test.clj
@@ -32,7 +32,7 @@
(doseq [statement ["CREATE TABLE \"test_table\" (\"id\" INTEGER PRIMARY KEY, \"filter_col\" VARCHAR);"
"INSERT INTO \"test_table\" (\"id\", \"filter_col\") VALUES (1, 'value1'), (2, 'value2');"]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [table-id (t2/select-one-fn :id :model/Table :db_id (mt/id) :name "test_table")
filter-field-id (t2/select-one-fn :id :model/Field :table_id table-id :name "filter_col")
mp (lib-be/application-database-metadata-provider (mt/id))
@@ -62,7 +62,7 @@
(with-syncable-db!
(fn [{:keys [card-id filter-field-id]}]
(jdbc/execute! one-off-dbs/*conn* ["ALTER TABLE \"test_table\" DROP COLUMN \"filter_col\";"])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (false? (t2/select-one-fn :active :model/Field :id filter-field-id)))
(is (pending? card-id))
(deps.test/synchronously-run-dependency-job!)
@@ -78,7 +78,7 @@
(with-syncable-db!
(fn [{:keys [card-id]}]
(jdbc/execute! one-off-dbs/*conn* ["ALTER TABLE \"test_table\" DROP COLUMN \"filter_col\";"])
- (sync/sync-db-metadata! (mt/db))
+ (sync/sync-db-metadata! (mt/id))
(is (pending? card-id))))))
(deftest ^:synchronized sync-without-changes-does-not-trigger-reanalysis-test
@@ -87,7 +87,7 @@
(fn [{:keys [card-id filter-field-id table-id]}]
(let [table-before (into {} (t2/select-one :model/Table :id table-id))
filter-field-before (into {} (t2/select-one :model/Field :id filter-field-id))]
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= table-before (into {} (t2/select-one :model/Table :id table-id))))
(is (= filter-field-before (into {} (t2/select-one :model/Field :id filter-field-id))))
(is (not (pending? card-id)))
@@ -109,13 +109,13 @@
(with-syncable-db!
(fn [{:keys [card-id]}]
(jdbc/execute! one-off-dbs/*conn* ["CREATE TABLE \"ref_table\" (\"id\" INTEGER PRIMARY KEY, \"test_id\" INTEGER);"])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [ref-table-id (t2/select-one-fn :id :model/Table :db_id (mt/id) :name "ref_table")
ref-field-id (t2/select-one-fn :id :model/Field :table_id ref-table-id :name "test_id")]
(t2/insert! :model/Dependency {:from_entity_type :card :from_entity_id card-id
:to_entity_type :table :to_entity_id ref-table-id})
(record-processed! card-id)
(jdbc/execute! one-off-dbs/*conn* ["ALTER TABLE \"ref_table\" ADD FOREIGN KEY (\"test_id\") REFERENCES \"test_table\"(\"id\");"])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (some? (t2/select-one-fn :fk_target_field_id :model/Field :id ref-field-id)))
(is (pending? card-id)))))))
diff --git a/enterprise/backend/test/metabase_enterprise/dependencies/task/dependency_test.clj b/enterprise/backend/test/metabase_enterprise/dependencies/task/dependency_test.clj
index f0f686b6d65..ea16a26efc5 100644
--- a/enterprise/backend/test/metabase_enterprise/dependencies/task/dependency_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/dependencies/task/dependency_test.clj
@@ -1237,9 +1237,6 @@
(assert-pending :data-app app-id)
(deps.test/synchronously-run-dependency-job!)
(is (= #{[:card card-id]} (data-app-deps app-id))))
- (testing "the app is an MBQL dependent of its card"
- (is (= {:data-app #{app-id}}
- (dependencies.model/transitive-mbql-dependents {:card [{:id card-id}]}))))
(testing "deleting the app removes its dependencies and status"
(t2/delete! :model/DataApp app-id)
(deps.test/synchronously-run-dependency-job!)
diff --git a/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj b/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj
index 6e524e9dffa..ac9bb6db366 100644
--- a/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj
@@ -15,8 +15,10 @@
[metabase.driver.connection :as driver.conn]
[metabase.driver.settings :as driver.settings]
[metabase.driver.sql :as driver.sql]
+ [metabase.driver.sql-jdbc.actions :as sql-jdbc.actions]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
+ [metabase.driver.util :as driver.u]
[metabase.lib.core :as lib]
[metabase.lib.metadata :as lib.metadata]
[metabase.lib.test-metadata :as meta]
@@ -33,6 +35,7 @@
[toucan2.core :as t2]
[toucan2.tools.with-temp :as t2.with-temp])
(:import
+ (com.mchange.v2.c3p0 PooledDataSource)
(java.sql Connection)
(java.util.concurrent CountDownLatch)))
@@ -41,14 +44,14 @@
(deftest ^:parallel connection-impersonation-role-test
(testing "Returns nil when no impersonations are in effect"
(mt/with-test-user :rasta
- (is (nil? (impersonation.driver/connection-impersonation-role (mt/db)))))))
+ (is (nil? (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider))))))))
(deftest connection-impersonation-role-test-2
(testing "Correctly fetches the impersonation when one is in effect"
(impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" "impersonation_role"}}
(is (= "impersonation_role"
- (impersonation.driver/connection-impersonation-role (mt/db)))))))
+ (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider))))))))
(deftest connection-impersonation-role-test-3
(testing "Throws exception if multiple conflicting impersonations are in effect"
@@ -60,7 +63,7 @@
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Multiple conflicting connection impersonation policies found for current user"
- (impersonation.driver/connection-impersonation-role (mt/db))))))))
+ (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider)))))))))
(deftest connection-impersonation-role-test-4
(testing "Returns nil if the permissions in another group supersede the impersonation"
@@ -69,14 +72,14 @@
;; `with-impersonations!` creates a new group and revokes data perms in `all users`, so if we re-grant data perms
;; for all users, it should supersede the impersonation policy in the new group
(mt/with-full-data-perms-for-all-users!
- (is (nil? (impersonation.driver/connection-impersonation-role (mt/db))))))))
+ (is (nil? (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider)))))))))
(deftest connection-impersonation-role-test-5
(testing "Returns nil for superuser, even if they are in a group with an impersonation policy defined"
(impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" "impersonation_role"}}
(request/as-admin
- (is (nil? (impersonation.driver/connection-impersonation-role (mt/db))))))))
+ (is (nil? (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider)))))))))
(deftest connection-impersonation-role-test-6
(testing "Does not throw an exception if passed a nil `database-or-id`"
@@ -91,7 +94,7 @@
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"User does not have attribute required for connection impersonation."
- (impersonation.driver/connection-impersonation-role (mt/db)))))))
+ (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider))))))))
(deftest connection-impersonation-role-test-8
(testing "Throws an exception if impersonation should be enforced, but the user's attribute is not a single string"
@@ -100,7 +103,7 @@
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Connection impersonation attribute is invalid: role must be a single non-empty string."
- (impersonation.driver/connection-impersonation-role (mt/db)))))))
+ (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider))))))))
(deftest connection-impersonation-role-test-9
(testing "Throws an exception if sandboxing policies are also defined for the current user on the DB"
@@ -112,7 +115,7 @@
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Conflicting sandboxing and impersonation policies found."
- (impersonation.driver/connection-impersonation-role (mt/db))))))))))
+ (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider)))))))))))
(deftest connection-impersonation-role-test-10
(testing "Rejects a role attribute equal to the driver's default-role sentinel (case-insensitively)"
@@ -122,12 +125,12 @@
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Connection impersonation attribute is invalid: role must not be the database default role."
- (impersonation.driver/connection-impersonation-role (mt/db)))))
+ (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider))))))
(testing "but a normal role value is still returned unchanged"
(impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" "impersonation_role"}}
(is (= "impersonation_role"
- (impersonation.driver/connection-impersonation-role (mt/db)))))))))
+ (impersonation.driver/connection-impersonation-role (lib.metadata/database (mt/metadata-provider))))))))))
(deftest conn-impersonation-test-postgres
(mt/test-driver :postgres
@@ -146,7 +149,7 @@
"GRANT SELECT ON TABLE \"conn_impersonation_test\".PUBLIC.table_with_access TO \"impersonation.role\";"]]
(jdbc/execute! spec [statement]))
(mt/with-temp [:model/Database database {:engine :postgres, :details details}]
- (mt/with-db database (sync/sync-database! database)
+ (mt/with-db database (sync/sync-database! (:id database))
(impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" "impersonation.role"}}
(is (= []
@@ -305,13 +308,13 @@
{:engine driver
:name (format "impersonation-%s-%d" (name driver) (u/the-id source-db))
:details details})))]
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(t2/update! :model/Database :id (u/the-id database)
{:details (cond-> (impersonation-details driver source-db)
(driver/database-supports? driver :connection-impersonation-requires-role nil)
(assoc :role (impersonation-default-role driver)))})
;; the pools opened during sync authenticate as the granting user; tests must not inherit them
- (driver/notify-database-updated driver database)
+ (driver/notify-database-updated driver (driver.u/lib-database (:id database)))
(let [database (t2/select-one :model/Database :id (u/the-id database))]
(swap! impersonation-databases assoc cache-key database)
database)))))
@@ -377,6 +380,103 @@
(mt/run-mbql-query venues
{:aggregation [[:count]]}))))))))))
+(deftest pooled-connections-get-impersonation-role-test
+ (testing "a connection taken straight from the pool, without do-with-connection-with-options, has the current user's role"
+ (mt/test-driver :postgres
+ (mt/with-premium-features #{:advanced-permissions}
+ (let [venues-table (sql.tx/qualify-and-quote driver/*driver* "test-data" "venues")
+ role-a (u/lower-case-en (mt/random-name))
+ current-role #(-> (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database (mt/id))) ["SELECT current_user AS role"])
+ first
+ :role)]
+ (tx/with-temp-roles! driver/*driver*
+ (impersonation-granting-details driver/*driver* (mt/db))
+ {role-a {venues-table {}}}
+ (impersonation-default-user driver/*driver*)
+ (impersonation-default-role driver/*driver*)
+ (with-impersonation-db!
+ (impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
+ :attributes {"impersonation_attr" role-a}}
+ (is (= role-a (current-role))))
+ (testing "the next borrower doesn't inherit the role"
+ (is (not= role-a (current-role)))))))))))
+
+(deftest pool-made-before-database-change-still-sets-role-test
+ (testing "a support check that reads the Database row, as ClickHouse's version check does, sees it as the running query does, not as it was when the pool was made"
+ (mt/test-driver :postgres
+ (mt/with-premium-features #{:advanced-permissions}
+ (let [venues-table (sql.tx/qualify-and-quote driver/*driver* "test-data" "venues")
+ role-a (u/lower-case-en (mt/random-name))
+ supports? (mt/original-fn #'driver.u/supports?)
+ current-role #(-> (mt/native-query {:query "SELECT current_user AS role"})
+ mt/process-query
+ mt/rows
+ ffirst)]
+ (tx/with-temp-roles! driver/*driver*
+ (impersonation-granting-details driver/*driver* (mt/db))
+ {role-a {venues-table {}}}
+ (impersonation-default-user driver/*driver*)
+ (impersonation-default-role driver/*driver*)
+ (with-impersonation-db!
+ (mt/with-dynamic-fn-redefs [driver.u/supports? (fn [driver feature database]
+ (and (or (not= feature :connection-impersonation)
+ (= "checked" (some #(get-in database [% :version]) [:dbms-version :dbms_version])))
+ (supports? driver feature database)))]
+ (current-role)
+ (mt/with-temp-vals-in-db :model/Database (mt/id) {:dbms_version {:version "checked"}}
+ (impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
+ :attributes {"impersonation_attr" role-a}}
+ (is (= role-a (current-role)))))))))))))
+
+(defn- current-role
+ "The role of the session of `conn`."
+ [^Connection conn]
+ (:role (first (jdbc/query {:connection conn} ["SELECT current_user AS role"]))))
+
+(deftest routed-impersonated-users-get-their-role-on-every-connection-test
+ (testing "a routed user's connections to the destination have the role the router's impersonation policy gives them"
+ (mt/test-driver :postgres
+ (mt/with-premium-features #{:advanced-permissions :database-routing}
+ (let [venues-table (sql.tx/qualify-and-quote driver/*driver* "test-data" "venues")
+ role-a (u/lower-case-en (mt/random-name))
+ details (:details (mt/db))]
+ (tx/with-temp-roles! driver/*driver* details {role-a {venues-table {}}}
+ (impersonation-default-user driver/*driver*)
+ nil
+ (mt/with-temp [:model/Database router {:engine :postgres :details details}
+ :model/Database dest {:engine :postgres :details details :name "routed-impersonated-dest"
+ :router_database_id (u/the-id router)}
+ :model/DatabaseRouter _ {:database_id (u/the-id router) :user_attribute "db"}]
+ (try
+ (mt/with-db router
+ (impersonation.util-test/with-impersonations! {:impersonations [{:db-id (u/the-id router) :attribute "impersonation_attr"}]
+ :attributes {"impersonation_attr" role-a
+ "db" "routed-impersonated-dest"}}
+ (is (= role-a (ffirst (mt/rows (mt/process-query (mt/native-query {:query "SELECT current_user"})))))
+ "query processor")
+ (is (= role-a (sql-jdbc.execute/do-with-connection-with-options :postgres (u/the-id router) {} current-role))
+ "do-with-connection-with-options")
+ (is (= role-a (sql-jdbc.actions/do-with-jdbc-transaction (u/the-id router) current-role))
+ "implicit action")))
+ (finally
+ (driver/notify-database-updated :postgres router)
+ (driver/notify-database-updated :postgres dest))))))))))
+
+(deftest role-that-cannot-be-set-fails-checkout-once-test
+ (testing "when the role can't be set, the caller gets that error, and the pool doesn't retry the checkout"
+ (mt/test-driver :postgres
+ (mt/with-premium-features #{:advanced-permissions}
+ (try
+ (impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
+ :attributes {"impersonation_attr" "no_such_role"}}
+ (let [data-source (:datasource (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database (mt/id))))
+ failed (.getNumFailedCheckoutsDefaultUser ^PooledDataSource data-source)]
+ (is (thrown-with-msg? Exception #"no_such_role"
+ (sql-jdbc.execute/do-with-connection-with-options :postgres (mt/id) {} current-role)))
+ (is (= failed (.getNumFailedCheckoutsDefaultUser ^PooledDataSource data-source)))))
+ (finally
+ (driver/notify-database-updated :postgres (mt/db))))))))
+
(deftest conn-impersonation-with-write-connection-test
(testing "Impersonation works correctly when queries go through the write connection pool"
(mt/test-drivers (mt/normal-drivers-with-feature :connection-impersonation)
@@ -396,7 +496,7 @@
(when (driver/database-supports? driver/*driver* :connection-impersonation-requires-role nil)
(t2/update! :model/Database :id (mt/id)
(assoc-in (mt/db) [:details :role] (impersonation-default-role driver/*driver*))))
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(impersonation.util-test/with-impersonations!
{:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" role-name}}
@@ -427,7 +527,7 @@
(let [details (:details (mt/db))
password (:password details)
readonly-user (u/lower-case-en (mt/random-name))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
venues-table (sql.tx/qualify-and-quote :postgres "test-data" "venues")
role-name (u/lower-case-en (mt/random-name))]
(try
@@ -448,7 +548,7 @@
:details readonly-details
:write_data_details write-details}]
(mt/with-db database
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(impersonation.util-test/with-impersonations!
{:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" role-name}}
@@ -601,7 +701,7 @@
(mt/with-temp [:model/Database database {:engine driver/*driver*,
:details details}]
(mt/with-db database
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" role-a}}
(is (thrown-with-msg?
@@ -613,7 +713,7 @@
(mt/with-temp [:model/Database database {:engine driver/*driver*,
:details (merge details {:role impersonation-user})}]
(mt/with-db database
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" role-a}}
(is (= [[100]]
@@ -655,7 +755,7 @@
:engine :postgres
:details destination-details
:name destination-db-name}]
- (mt/with-db router-database (sync/sync-database! router-database)
+ (mt/with-db router-database (sync/sync-database! (:id router-database))
(testing "Database Routing OFF"
(testing "impersonation OFF"
(testing "both crowberto and rasta see all rows in the ROUTER database"
@@ -753,7 +853,7 @@
(format "GRANT SELECT ON TABLE \"%s\".table_with_access TO %s;" schema user)]]
(jdbc/execute! spec statement))
(mt/with-db database
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" user}}
(is (= []
@@ -896,35 +996,6 @@
:database_id (mt/id)
:card_id (:id model))))))))))))
-(deftest resilient-connection-options-test
- (testing "resilient connections have the correct role set"
- (mt/test-drivers (mt/normal-driver-select {:+parent :sql-jdbc
- :+features [:connection-impersonation]})
- (mt/with-premium-features #{:advanced-permissions}
- (let [venues-table (sql.tx/qualify-and-quote driver/*driver* "test-data" "venues")
- role-a (u/lower-case-en (mt/random-name))]
- (tx/with-temp-roles! driver/*driver*
- (impersonation-granting-details driver/*driver* (mt/db))
- {role-a {venues-table {}}}
- (impersonation-default-user driver/*driver*)
- (impersonation-default-role driver/*driver*)
- (with-impersonation-db!
- (let [tables-set #(->> (driver/describe-database
- driver/*driver*
- (t2/select-one :model/Database (mt/id)))
- :tables
- (into #{}))
- default-table-set (tables-set)
- do-with-resolved-connection (mt/original-fn #'sql-jdbc.execute/do-with-resolved-connection)]
- (mt/with-dynamic-fn-redefs [sql-jdbc.execute/do-with-resolved-connection
- (fn [driver db options f]
- (do-with-resolved-connection driver db options
- (fn [conn]
- (when-not (:connection db)
- (driver/set-role! driver/*driver* conn role-a))
- (f conn))))]
- (is (= default-table-set (tables-set))))))))))))
-
(defn do-on-all-connection-in-pool [driver db-id options f]
(let [max-pool-size (driver.settings/jdbc-data-warehouse-max-connection-pool-size)
^CountDownLatch start-latch (java.util.concurrent.CountDownLatch. max-pool-size)
@@ -944,6 +1015,32 @@
(when-not (.await ^CountDownLatch finish-latch 60 java.util.concurrent.TimeUnit/SECONDS)
(throw (ex-info "Timeout waiting for all futures to complete" {})))))
+(deftest resilient-connection-options-test
+ (testing "a connection checked out of the pool doesn't keep the role its previous borrower set"
+ (mt/test-drivers (mt/normal-driver-select {:+parent :sql-jdbc
+ :+features [:connection-impersonation]})
+ (mt/with-premium-features #{:advanced-permissions}
+ (let [venues-table (sql.tx/qualify-and-quote driver/*driver* "test-data" "venues")
+ role-a (u/lower-case-en (mt/random-name))]
+ (tx/with-temp-roles! driver/*driver*
+ (impersonation-granting-details driver/*driver* (mt/db))
+ {role-a {venues-table {}}}
+ (impersonation-default-user driver/*driver*)
+ (impersonation-default-role driver/*driver*)
+ (with-impersonation-db!
+ (let [tables-set #(->> (driver/describe-database
+ driver/*driver*
+ (driver.u/ensure-lib-database (t2/select-one :model/Database (mt/id))))
+ :tables
+ (into #{}))
+ default-table-set (tables-set)]
+ ;; Give the connection pool time to release any connections held by sync
+ (Thread/sleep 1000)
+ (do-on-all-connection-in-pool driver/*driver* (mt/id) {}
+ (fn [^Connection conn]
+ (driver/set-role! driver/*driver* conn role-a)))
+ (is (= default-table-set (tables-set)))))))))))
+
(deftest nested-do-with-connection-with-options-test
(testing "nested calls to `do-with-connection-with-options` have the correct connection options set"
(mt/test-drivers (mt/normal-driver-select {:+parent :sql-jdbc
@@ -1006,7 +1103,12 @@
#"Advanced Permissions is a paid feature not currently available"
(mt/formatted-rows [int]
(mt/run-mbql-query venues
- {:aggregation [[:count]]})))))
+ {:aggregation [[:count]]}))))
+ (is (thrown-with-msg? Exception
+ #"Advanced Permissions is a paid feature not currently available"
+ (sql-jdbc.execute/do-with-connection-with-options driver/*driver* (mt/id) nil
+ (constantly nil)))
+ "outside query processor"))
(testing "admin should still be able to query"
(request/as-admin
(is (= [100]
diff --git a/enterprise/backend/test/metabase_enterprise/replacement/source_check_test.clj b/enterprise/backend/test/metabase_enterprise/replacement/source_check_test.clj
index f501ad485a9..26b5d657040 100644
--- a/enterprise/backend/test/metabase_enterprise/replacement/source_check_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/replacement/source_check_test.clj
@@ -380,13 +380,13 @@
"CREATE TABLE table_d (fk_id integer NOT NULL REFERENCES table_a (id));"]]
(jdbc/execute! spec [statement]))
(mt/with-temp [:model/Database database {:engine :postgres, :details details}]
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(let [table-id (fn [table-name] (t2/select-one-pk :model/Table :db_id (:id database) :name table-name))
fk-field (fn [table-name] (t2/select-one :model/Field :table_id (table-id table-name) :name "fk_id"))
table-a-id (table-id "table_a")
table-b-id (table-id "table_b")]
(jdbc/execute! spec ["DROP TABLE table_d;"])
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(testing "precondition: the dropped table is inactive but its FK field is still active"
(is (false? (t2/select-one-fn :active :model/Table :id (table-id "table_d"))))
(is (=? {:active true, :fk_target_field_id some?} (fk-field "table_d"))))
diff --git a/enterprise/backend/test/metabase_enterprise/sandbox/query_processor/middleware/sandboxing_test.clj b/enterprise/backend/test/metabase_enterprise/sandbox/query_processor/middleware/sandboxing_test.clj
index 3b5f0448a90..97fdbab418f 100644
--- a/enterprise/backend/test/metabase_enterprise/sandbox/query_processor/middleware/sandboxing_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/sandbox/query_processor/middleware/sandboxing_test.clj
@@ -2340,7 +2340,7 @@
(is (seq rows))
(is (every? #(= "Gizmo" (first %)) rows))))
(testing "after the sandbox filter column is dropped, the query fails closed rather than returning unfiltered rows"
- (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))]
+ (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))]
(jdbc/execute! db-spec ["ALTER TABLE \"PUBLIC\".\"PRODUCTS\" DROP COLUMN \"CATEGORY\";"]))
;; The consumer query selects only ID (a column that still exists). This is what makes the
;; assertion distinguish fail-CLOSED from fail-OPEN: if the sandbox were silently dropped
diff --git a/enterprise/backend/test/metabase_enterprise/sandbox/sample_database_test.clj b/enterprise/backend/test/metabase_enterprise/sandbox/sample_database_test.clj
index 883527e6d1c..2789ce682ed 100644
--- a/enterprise/backend/test/metabase_enterprise/sandbox/sample_database_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/sandbox/sample_database_test.clj
@@ -27,7 +27,7 @@
(mt/with-temp [:model/Database db {:details (#'sample-data/try-to-extract-sample-database! :sqlite)
:engine :sqlite
:name "Sample Database"}]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(thunk db))))
(defn- table-id [db table-name]
diff --git a/enterprise/backend/test/metabase_enterprise/serialization/api_test.clj b/enterprise/backend/test/metabase_enterprise/serialization/api_test.clj
index 6a557e46cba..8de67a242c4 100644
--- a/enterprise/backend/test/metabase_enterprise/serialization/api_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/serialization/api_test.clj
@@ -601,7 +601,8 @@
(let [db (t2/select-one :model/Database :name db-name)]
(is (some? db))
(is (=? {:host "localhost" :port 5432 :dbname "mydb"} (:details db)))
- (doseq [k [:details :write_data_details :admin_details]]
+ (is (not (contains? db :admin_details)) "an older archive's admin_details is ignored")
+ (doseq [k [:details :write_data_details]]
(is (= #{} (set (keys (select-keys (get db k) [:subname :classname :connection-uri :INIT]))))
(str k " should keep none of the low-level keys"))))
(finally
diff --git a/enterprise/backend/test/metabase_enterprise/serialization/v2/load_test.clj b/enterprise/backend/test/metabase_enterprise/serialization/v2/load_test.clj
index 48d09d7bd72..a08d3351387 100644
--- a/enterprise/backend/test/metabase_enterprise/serialization/v2/load_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/serialization/v2/load_test.clj
@@ -260,7 +260,7 @@
(is (= {:type/Currency true :type/City true}
(into {} (map (juxt first second)) (vals (settings)))))
(t2/update! :model/Database (:id stub) {:engine :h2 :details (:details (mt/db)) :is_stub false})
- (sync/sync-db-metadata! (t2/select-one :model/Database :id (:id stub)))
+ (sync/sync-db-metadata! (:id (t2/select-one :model/Database :id (:id stub))))
(testing "sync reactivates the stub Fields and gives them their real base types"
(is (=? [{:name "NAME" :active true :base_type :type/Text}
{:name "PRICE" :active true :base_type :type/Integer}]
diff --git a/enterprise/backend/test/metabase_enterprise/transform_testing/remapping_test.clj b/enterprise/backend/test/metabase_enterprise/transform_testing/remapping_test.clj
index 8024737ade3..09f33a18a1a 100644
--- a/enterprise/backend/test/metabase_enterprise/transform_testing/remapping_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transform_testing/remapping_test.clj
@@ -329,4 +329,4 @@
:name "the output holds only the fixture row"
:sql (str "SELECT * FROM " schema "." target " WHERE id <> -1")}]}]
(is (= :passed (:status (transform-testing.runner/run-transform-test! transform-test))))
- (is (not (driver/table-exists? driver/*driver* (mt/db) {:schema schema :name target}))))))))
+ (is (not (driver/table-exists? driver/*driver* (lib.metadata/database (mt/metadata-provider)) {:schema schema :name target}))))))))
diff --git a/enterprise/backend/test/metabase_enterprise/transform_testing/reported_type_test.clj b/enterprise/backend/test/metabase_enterprise/transform_testing/reported_type_test.clj
index c544e67430b..a7d6be67d6b 100644
--- a/enterprise/backend/test/metabase_enterprise/transform_testing/reported_type_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transform_testing/reported_type_test.clj
@@ -27,7 +27,7 @@
[driver sql]
(driver/do-with-test-connection
driver
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
(fn [conn]
(let [table (driver/temp-table-name driver)]
(try
diff --git a/enterprise/backend/test/metabase_enterprise/transforms/incremental_test.clj b/enterprise/backend/test/metabase_enterprise/transforms/incremental_test.clj
index e239e6a9338..5d29130f841 100644
--- a/enterprise/backend/test/metabase_enterprise/transforms/incremental_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transforms/incremental_test.clj
@@ -11,6 +11,7 @@
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.driver.sql.util :as sql.u]
+ [metabase.driver.util :as driver.u]
[metabase.lib-be.core :as lib-be]
[metabase.lib.core :as lib]
[metabase.lib.metadata :as lib.metadata]
@@ -186,7 +187,7 @@
"Insert new products into the transforms_products table."
[products]
(let [[schema source-table-name] (t2/select-one-fn (juxt :schema :name) :model/Table (mt/id :transforms_products))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
table-sql (sql.u/quote-name driver/*driver* :table schema source-table-name)
field-sql #(sql.u/quote-name driver/*driver* :field %)]
(if (#{:clickhouse :snowflake} driver/*driver*)
@@ -216,7 +217,7 @@
(defn- delete-test-products!
[products]
(let [[schema source-table-name] (t2/select-one-fn (juxt :schema :name) :model/Table (mt/id :transforms_products))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
delete-sql (format "DELETE FROM %s WHERE %s IN (%s)"
(sql.u/quote-name driver/*driver* :table schema source-table-name)
(sql.u/quote-name driver/*driver* :field "name")
@@ -511,7 +512,7 @@
(mt/dataset transforms-dataset/transforms-test
(with-transform-cleanup! [target-table "empty_table_target"]
(let [db-id (mt/id)
- db-spec (sql-jdbc.conn/db->pooled-connection-spec db-id)
+ db-spec (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))
table-id (mt/id :transforms_products)
checkpoint-field-id (t2/select-one-pk :model/Field :name "id" :table_id table-id)
;; Query that returns no rows but has the checkpoint column
@@ -621,7 +622,7 @@
(mt/dataset transforms-dataset/transforms-test
(with-transform-cleanup! [target-table "change_table_target"]
(let [db-id (mt/id)
- db-spec (sql-jdbc.conn/db->pooled-connection-spec db-id)
+ db-spec (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))
table-id (mt/id :transforms_products)
checkpoint-field-id (t2/select-one-pk :model/Field :name "id" :table_id table-id)
make-source (fn [query-sql]
diff --git a/enterprise/backend/test/metabase_enterprise/transforms/merge_test.clj b/enterprise/backend/test/metabase_enterprise/transforms/merge_test.clj
index 76cf0f27dcc..82278be66a3 100644
--- a/enterprise/backend/test/metabase_enterprise/transforms/merge_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transforms/merge_test.clj
@@ -7,6 +7,7 @@
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql.util :as sql.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.query-processor :as qp]
[metabase.test :as mt]
[metabase.transforms.execute :as transforms.execute]
@@ -115,7 +116,7 @@
"Append `rows` (vectors of `[order-id status change-seq]`) to the source table."
[rows]
(let [[schema table-name] (t2/select-one-fn (juxt :schema :name) :model/Table (mt/id :order_status))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
values (str/join ", " (map (fn [[oid st sq]] (format "(%d, '%s', %d)" oid st sq)) rows))
sql (format "INSERT INTO %s (%s) VALUES %s"
(sql.u/quote-name driver/*driver* :table schema table-name)
@@ -191,7 +192,7 @@
insert! (fn [rows]
(let [[schema tbl] (t2/select-one-fn (juxt :schema :name) :model/Table
(mt/id :order_status_ts))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
values (str/join ", " (map (fn [[oid st ts]]
(format "(%d, '%s', '%s')" oid st ts))
rows))
@@ -279,7 +280,7 @@
insert! (fn [rows]
(let [[schema tbl] (t2/select-one-fn (juxt :schema :name) :model/Table
(mt/id :order_status_lb))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
values (str/join ", " (map (fn [[oid st ts]]
(format "(%d, '%s', '%s')" oid st ts))
rows))
@@ -300,7 +301,7 @@
delete-order! (fn [order-id]
(let [[schema tbl] (t2/select-one-fn (juxt :schema :name) :model/Table
(mt/id :order_status_lb))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
sql (format "DELETE FROM %s WHERE %s = %d"
(sql.u/quote-name driver/*driver* :table schema tbl)
(q "order_id") order-id)]
@@ -383,7 +384,7 @@
insert! (fn [rows]
(let [[schema tbl] (t2/select-one-fn (juxt :schema :name) :model/Table
(mt/id :order_status_unix))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
values (str/join ", " (map (fn [[oid st ts]]
(format "(%d, '%s', %d)" oid st ts))
rows))
@@ -458,7 +459,7 @@
insert! (fn [rows]
(let [[schema tbl] (t2/select-one-fn (juxt :schema :name) :model/Table
(mt/id :order_region_status))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
values (str/join ", " (map (fn [[oid rg st sq]]
(format "(%d, '%s', '%s', %d)" oid rg st sq))
rows))
diff --git a/enterprise/backend/test/metabase_enterprise/transforms_python/base_test.clj b/enterprise/backend/test/metabase_enterprise/transforms_python/base_test.clj
index 303696053b2..827197d2c50 100644
--- a/enterprise/backend/test/metabase_enterprise/transforms_python/base_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transforms_python/base_test.clj
@@ -72,7 +72,7 @@
{:source {:type :python :source-tables [] :body "stub"}
:target {:type "table" :name "out"}
:id 7}
- {:engine :h2 :id 1}
+ {:lib/type :metadata/database :engine :h2 :id 1}
42
cancel-chan
stub-message-log
diff --git a/enterprise/backend/test/metabase_enterprise/transforms_python/drivers_test.clj b/enterprise/backend/test/metabase_enterprise/transforms_python/drivers_test.clj
index 2505c19560e..9cc88a7d2d9 100644
--- a/enterprise/backend/test/metabase_enterprise/transforms_python/drivers_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transforms_python/drivers_test.clj
@@ -142,7 +142,7 @@
:else
v)) row))
data)}))
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(t2/select-one-pk :model/Table :name (name qualified-table-name) :db_id db-id)))
(defn- cleanup-table!
diff --git a/enterprise/backend/test/metabase_enterprise/transforms_python/e2e_test.clj b/enterprise/backend/test/metabase_enterprise/transforms_python/e2e_test.clj
index 6d6dcb8ff62..e7472106544 100644
--- a/enterprise/backend/test/metabase_enterprise/transforms_python/e2e_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transforms_python/e2e_test.clj
@@ -4,6 +4,7 @@
[clojure.test :refer :all]
[metabase-enterprise.transforms-python.models.python-library :as python-library]
[metabase.driver :as driver]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.test :as mt]
[metabase.test.util :as mt.util]
[metabase.transforms.test-dataset :as transforms-dataset]
@@ -51,7 +52,7 @@
transform-payload)]
(transforms.tu/test-run transform-id)
(transforms.tu/wait-for-table table-name 5000)
- (is (true? (driver/table-exists? driver/*driver* (mt/db) target)))
+ (is (true? (driver/table-exists? driver/*driver* (lib.metadata/database (mt/metadata-provider)) target)))
(let [rows (transforms.tu/table-rows table-name)]
(is (= 5 (count rows)))
(is (every? #(= (* 2 (first %)) (second %)) rows))
@@ -80,6 +81,6 @@
original)]
(transforms.tu/test-run transform-id)
(transforms.tu/wait-for-table table-name 5000)
- (is (true? (driver/table-exists? driver/*driver* (mt/db) target)))
+ (is (true? (driver/table-exists? driver/*driver* (lib.metadata/database (mt/metadata-provider)) target)))
(is (= [["Alice" 25] ["Bob" 30]]
(transforms.tu/table-rows table-name)))))))))))
diff --git a/enterprise/backend/test/metabase_enterprise/transforms_python/job_test.clj b/enterprise/backend/test/metabase_enterprise/transforms_python/job_test.clj
index b9d36465781..0d05934b8df 100644
--- a/enterprise/backend/test/metabase_enterprise/transforms_python/job_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transforms_python/job_test.clj
@@ -2,6 +2,7 @@
(:require
[clojure.test :refer :all]
[metabase.driver :as driver]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.task.core :as task]
[metabase.test :as mt]
[metabase.transforms.jobs :as jobs]
@@ -38,7 +39,7 @@
:model/TransformJobTransformTag _ {:job_id job-id :tag_id tag-id :position 0}]
(transforms.schedule/initialize-job! job)
(transforms.schedule/update-job! job-id "* * * * * ? *")
- (is (true? (u/poll {:thunk (fn [] (driver/table-exists? driver/*driver* (mt/db) target))
+ (is (true? (u/poll {:thunk (fn [] (driver/table-exists? driver/*driver* (lib.metadata/database (mt/metadata-provider)) target))
:done? true?
:timeout-ms 20000
:interval-ms 1000})))
diff --git a/enterprise/backend/test/metabase_enterprise/transforms_python/python_runner_test.clj b/enterprise/backend/test/metabase_enterprise/transforms_python/python_runner_test.clj
index 86f1f3d8132..2264ff85861 100644
--- a/enterprise/backend/test/metabase_enterprise/transforms_python/python_runner_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transforms_python/python_runner_test.clj
@@ -11,6 +11,7 @@
[metabase-enterprise.transforms-python.settings :as transforms-python.settings]
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.sync.core :as sync]
[metabase.test :as mt]
[metabase.test.data.interface :as tx]
@@ -175,12 +176,12 @@
(testing "transform function successfully connects to PostgreSQL database and reads data"
(mt/test-drivers #{:postgres}
(mt/with-empty-db
- (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
_ (jdbc/execute! db-spec ["DROP TABLE IF EXISTS students"])
_ (jdbc/execute! db-spec ["CREATE TABLE students (id INTEGER PRIMARY KEY, name VARCHAR(100), score INTEGER)"])
_ (jdbc/execute! db-spec ["INSERT INTO students (id, name, score) VALUES (1, 'Alice', 85), (2, 'Bob', 92), (3, 'Charlie', 88), (4, 'Dana', 90)"])
- _ (sync/sync-database! (mt/db))
+ _ (sync/sync-database! (mt/id))
transform-code (str "import pandas as pd\n"
"\n"
@@ -242,11 +243,11 @@
(testing "transform function successfully connects to PostgreSQL database and reads data"
(mt/test-drivers #{:postgres}
(mt/with-empty-db
- (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
_ (jdbc/execute! db-spec ["DROP TABLE IF EXISTS students"])
_ (jdbc/execute! db-spec ["CREATE TABLE students (id INTEGER PRIMARY KEY, name VARCHAR(100), score INTEGER)"])
- _ (sync/sync-database! (mt/db))
+ _ (sync/sync-database! (mt/id))
transform-code (str "import pandas as pd\n"
"\n"
@@ -273,12 +274,12 @@
(testing "transform function successfully connects to PostgreSQL database and reads data"
(mt/test-drivers #{:postgres}
(mt/with-empty-db
- (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
_ (jdbc/execute! db-spec ["DROP TABLE IF EXISTS students"])
_ (jdbc/execute! db-spec ["CREATE TABLE students (id INTEGER PRIMARY KEY, name VARCHAR(100), score INTEGER)"])
_ (jdbc/execute! db-spec ["INSERT INTO students (id, name, score) VALUES (1, 'Alice', 85), (2, 'Bob', 92), (3, 'Charlie', 88), (4, 'Dana', 90)"])
- _ (sync/sync-database! (mt/db))
+ _ (sync/sync-database! (mt/id))
transform-code (str "import pandas as pd\n"
"\n"
@@ -311,7 +312,7 @@
(deftest transform-type-test
(mt/test-drivers #{:postgres}
(mt/with-empty-db
- (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
_ (jdbc/execute! db-spec ["DROP TABLE IF EXISTS sample_table"])
_ (jdbc/execute! db-spec ["CREATE TABLE sample_table (
id BIGSERIAL PRIMARY KEY,
@@ -331,7 +332,7 @@
('Product B', 'Another product with longer description text', 50, 15.50, false, '2024-02-01', '2024-02-01 09:15:30', '2024-02-02 16:30:00-05'),
('Product C', NULL, 0, 0.0, true, '2024-03-10', '2024-03-10 18:45:15', '2024-03-11 08:00:00+02')"])
- _ (sync/sync-database! (mt/db))
+ _ (sync/sync-database! (mt/id))
transform-code (str "import pandas as pd\n"
"\n"
@@ -466,7 +467,7 @@
_ (driver/insert-from-source! driver db-id table-schema
{:type :rows :data row-values})
- _ (sync/sync-database! (mt/db) {:scan :schema})
+ _ (sync/sync-database! (mt/id) {:scan :schema})
transform-code (str "import pandas as pd\n"
"\n"
diff --git a/enterprise/backend/test/metabase_enterprise/transforms_python/transforms_api_test.clj b/enterprise/backend/test/metabase_enterprise/transforms_python/transforms_api_test.clj
index a203d751602..a2909fd4c75 100644
--- a/enterprise/backend/test/metabase_enterprise/transforms_python/transforms_api_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/transforms_python/transforms_api_test.clj
@@ -7,6 +7,7 @@
[metabase-enterprise.transforms-python.init]
[metabase-enterprise.transforms-python.python-runner :as transforms-python.python-runner]
[metabase.driver :as driver]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.permissions.models.permissions-group :as perms-group]
[metabase.test :as mt]
[metabase.transforms.test-dataset :as transforms-dataset]
@@ -155,7 +156,7 @@
{transform-id :id} (mt/user-http-request :crowberto :post 200 "transform" original)]
(transforms.tu/test-run transform-id)
(transforms.tu/wait-for-table table-name 10000)
- (is (true? (driver/table-exists? driver/*driver* (mt/db) target)))
+ (is (true? (driver/table-exists? driver/*driver* (lib.metadata/database (mt/metadata-provider)) target)))
(is (= [["Alice" 25] ["Bob" 30]] (sort-by first (transforms.tu/table-rows table-name))))))))))))
(defn- subsequence?
@@ -483,7 +484,7 @@
(is (not (str/includes? (str (:message last-run)) "script started"))))))
(when (some? expect-write)
(testing "table existence"
- (is (= expect-write (driver/table-exists? driver/*driver* (mt/db) target)))))))))
+ (is (= expect-write (driver/table-exists? driver/*driver* (lib.metadata/database (mt/metadata-provider)) target)))))))))
; todo We have not yet covered the case where we rerun the same transform while there might be some hangover.
; Cancellation addresses the transform and not the run, there is shared mutable state and races on it are possible
(testing "the runner is not blocked for a new run"
diff --git a/frontend/src/metabase-types/api/database.ts b/frontend/src/metabase-types/api/database.ts
index dc10a8362d0..6a30eb16504 100644
--- a/frontend/src/metabase-types/api/database.ts
+++ b/frontend/src/metabase-types/api/database.ts
@@ -104,7 +104,6 @@ export interface DatabaseData {
// [[metabase.models.interface/to-json]] for `:model/Database`:
details?: Record<string, unknown>;
write_data_details?: Record<string, unknown> | null;
- admin_details?: Record<string, unknown> | null;
schedules: DatabaseSchedules;
auto_run_queries: boolean | null;
refingerprint: boolean | null;
@@ -230,7 +229,6 @@ export interface UpdateDatabaseRequest {
refingerprint?: boolean | null;
details?: Record<string, unknown>;
write_data_details?: Record<string, unknown> | null;
- admin_details?: Record<string, unknown> | null;
schedules?: DatabaseSchedules;
description?: string;
caveats?: string;
diff --git a/frontend/src/metabase/common/utils/database.ts b/frontend/src/metabase/common/utils/database.ts
index 683db4bd84f..995616b1cf3 100644
--- a/frontend/src/metabase/common/utils/database.ts
+++ b/frontend/src/metabase/common/utils/database.ts
@@ -46,12 +46,6 @@ export const hasWritableConnectionDetails = (
return database.write_data_details != null;
};
-export const hasAdminConnectionDetails = (
- database: Pick<Database, "admin_details">,
-) => {
- return database.admin_details != null;
-};
-
export const hasDbRoutingEnabled = (
database: Pick<Database, "router_user_attribute">,
) => {
diff --git a/frontend/src/metabase/urls/admin.ts b/frontend/src/metabase/urls/admin.ts
index 43132e63ecc..f5a0dad8b5f 100644
--- a/frontend/src/metabase/urls/admin.ts
+++ b/frontend/src/metabase/urls/admin.ts
@@ -83,10 +83,6 @@ export function editDatabaseWritableConnection(databaseId: DatabaseId) {
return `/admin/databases/${databaseId}/write-data`;
}
-export function editDatabaseAdminConnection(databaseId: DatabaseId) {
- return `/admin/databases/${databaseId}/admin`;
-}
-
type DataModelParams = {
databaseId?: DatabaseId;
schemaName?: SchemaName | null;
diff --git a/modules/drivers/athena/src/metabase/driver/athena.clj b/modules/drivers/athena/src/metabase/driver/athena.clj
index dac899dc86d..fb26de355a2 100644
--- a/modules/drivers/athena/src/metabase/driver/athena.clj
+++ b/modules/drivers/athena/src/metabase/driver/athena.clj
@@ -62,7 +62,7 @@
(defmethod driver/database-supports? [:athena :schemas]
[_driver _feature db]
- (not (seq (:dbname (driver.conn/effective-details db)))))
+ (not (seq (:dbname (:details db)))))
;;; +----------------------------------------------------------------------------------------------------------------+
;;; | metabase.driver.sql-jdbc method impls |
diff --git a/modules/drivers/athena/test/metabase/driver/athena_test.clj b/modules/drivers/athena/test/metabase/driver/athena_test.clj
index 7853c569159..74884051a73 100644
--- a/modules/drivers/athena/test/metabase/driver/athena_test.clj
+++ b/modules/drivers/athena/test/metabase/driver/athena_test.clj
@@ -9,7 +9,9 @@
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
[metabase.driver.sql.query-processor :as sql.qp]
+ [metabase.driver.util :as driver.u]
[metabase.lib.core :as lib]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.lib.test-metadata :as meta]
[metabase.lib.test-util :as lib.tu]
[metabase.premium-features.core :as premium-features]
@@ -197,7 +199,7 @@
{:name "latitude", :base-type :type/Float, :database-type "double", :database-position 3}
{:name "longitude", :base-type :type/Float, :database-type "double", :database-position 4}
{:name "price", :base-type :type/Integer, :database-type "int", :database-position 5}}
- (#'athena/describe-table-fields-with-nested-fields (mt/db) "v3_test_data" "venues")))))
+ (#'athena/describe-table-fields-with-nested-fields (lib.metadata/database (mt/metadata-provider)) "v3_test_data" "venues")))))
(deftest ^:parallel endpoint-test
(testing "AWS Endpoint URL"
@@ -425,20 +427,20 @@
:secret_key (tx/db-test-env-var-or-throw :athena :without-get-table-metadata-secret-key)
:catalog catalog)]
(mt/with-temp [:model/Database db {:engine :athena, :details details}]
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(let [table (t2/select-one :model/Table :db_id (:id db) :name "airport")]
(testing "Check that .getColumns returns no results, meaning the athena JDBC driver still has a bug"
;; If this test fails and .getColumns returns results, the athena JDBC driver has been fixed and we can
;; undo the changes in https://github.com/metabase/metabase/pull/44032
(is (empty? (sql-jdbc.execute/do-with-connection-with-options
:athena
- db
+ (driver.u/ensure-lib-database db)
nil
(fn [^java.sql.Connection conn]
(let [metadata (.getMetaData conn)]
(#'athena/get-columns metadata catalog (:schema table) (:name table))))))))
(testing "`describe-table` returns the fields anyway"
- (is (not-empty (:fields (driver/describe-table :athena db table))))))))))))
+ (is (not-empty (:fields (driver/describe-table :athena (driver.u/ensure-lib-database db) table))))))))))))
(deftest describe-table-falls-back-to-describe-on-duplicate-jdbc-columns-test
(testing "`describe-table-fields` uses DESCRIBE if the JDBC driver returns duplicate column names (#58441, GHY-3273)"
@@ -457,7 +459,7 @@
{:database-position 3, :name "latitude", :database-type "double", :base-type :type/Float}
{:database-position 4, :name "longitude", :database-type "double", :base-type :type/Float}
{:database-position 5, :name "municipality_id", :database-type "int", :base-type :type/Integer}}
- (:fields (driver/describe-table :athena db table))))
+ (:fields (driver/describe-table :athena (driver.u/ensure-lib-database db) table))))
(is (true? @get-columns-called))))))))
(deftest column-name-with-question-mark-test
@@ -629,11 +631,11 @@
{:name "people", :schema nil, :description nil}
{:name "products", :schema nil, :description nil}
{:name "reviews", :schema nil, :description nil}}}
- (driver/describe-database driver/*driver* (mt/db)))))
+ (driver/describe-database driver/*driver* (lib.metadata/database (mt/metadata-provider))))))
(testing "when the dbname is not specified describe-database returns tables from all databases and does include the schema"
(mt/with-temp [:model/Database db {:engine :athena,
:details (dissoc (:details (mt/db)) :dbname)}]
- (let [tables (driver/describe-database driver/*driver* db)
+ (let [tables (driver/describe-database driver/*driver* (driver.u/ensure-lib-database db))
;; athena CI has many (possibly changing) databases so we'll just filter for a few
filter-dbs #{"v3_test_data" "airports" "db_router_data" "db_routed_data" "diff_time_zones_athena_cases"}
filtered-tables {:tables (set (filter (comp filter-dbs :schema) (:tables tables)))}]
diff --git a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj
index c5e549a94d3..8eccc3d5165 100644
--- a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj
+++ b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj
@@ -784,7 +784,7 @@
(defmethod driver/table-rows-sample :bigquery-cloud-sdk
[driver {table-name :name, dataset-id :schema :as table} fields rff opts]
- (let [database (driver-api/table->database table)
+ (let [database (driver.u/lib-database (:db_id table))
bq-table (get-table database dataset-id table-name)]
(if (or (#{TableDefinition$Type/MATERIALIZED_VIEW TableDefinition$Type/VIEW
;; We couldn't easily test if the following two can show up as
@@ -1212,8 +1212,8 @@
;; TODO: THIS METHOD SHOULD NOT BE UPDATING THE APP-DB (which it does in [convert-dataset-id-to-filters!])
;; Issue: https://github.com/metabase/metabase/issues/39392
;; This normalize is a legacy migration (OAuth -> service-account, dataset-id -> filters); it
-;; runs only against `:details` because the overlay maps (`:write-data-details`, `:admin-details`)
-;; never carry these legacy fields, and `convert-dataset-id-to-filters!` writes side effects
+;; runs only against `:details` because the overlay map (`:write-data-details`)
+;; never carries these legacy fields, and `convert-dataset-id-to-filters!` writes side effects
;; specific to the primary details.
(defmethod driver/normalize-db-details :bigquery-cloud-sdk
[_driver database]
@@ -1393,14 +1393,14 @@
(let [base (#'driver.sql-jdbc/create-table!-sql driver table-name column-definitions :primary-key primary-key)
cluster (clustering-clause indexes)
sql (if cluster (str base " " cluster) base)
- database (bigquery.db/database database-id)
+ database (driver.u/lib-database database-id)
conn-spec (driver/connection-spec driver database)]
(driver/execute-raw-queries! driver conn-spec [sql])))
(defmethod driver/drop-table! :bigquery-cloud-sdk
[driver database-id table-name]
(let [sql (driver/compile-drop-table driver table-name)
- database (bigquery.db/database database-id)
+ database (driver.u/lib-database database-id)
conn-spec (driver/connection-spec driver database)]
(driver/execute-raw-queries! driver conn-spec [sql])))
@@ -1455,7 +1455,7 @@
;; update their metadata caches frequently enough for timely transform runs (or interactive previews).
;; rather than waiting many minutes, we trade torward consistency by using SQL DML, whose table metadata
;; is consistent, and we do not see cached non-existence and things like that causing trouble.
- (let [database (bigquery.db/database db-id)
+ (let [database (driver.u/lib-database db-id)
col-kws (perf/mapv (comp keyword name :name) columns)
num-cols (count col-kws)
;; bigquery allows 10k query parameters per request
diff --git a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/db.clj b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/db.clj
index a82f03d1757..4d983cb1b1f 100644
--- a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/db.clj
+++ b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/db.clj
@@ -10,11 +10,6 @@
^{:clj-kondo/ignore [:discouraged-namespace]}
[toucan2.core :as t2]))
-(mu/defn database
- "The Database with `database-id`, or nil."
- [database-id :- ::lib.schema.id/database]
- (t2/select-one :model/Database database-id))
-
(mu/defn update-database-details!
"Set the details of the Database with `database-id`, returning the number updated."
[database-id :- ::lib.schema.id/database
diff --git a/modules/drivers/bigquery-cloud-sdk/test/metabase/driver/bigquery_cloud_sdk/query_processor_test.clj b/modules/drivers/bigquery-cloud-sdk/test/metabase/driver/bigquery_cloud_sdk/query_processor_test.clj
index ecdf999ace2..8322012b2da 100644
--- a/modules/drivers/bigquery-cloud-sdk/test/metabase/driver/bigquery_cloud_sdk/query_processor_test.clj
+++ b/modules/drivers/bigquery-cloud-sdk/test/metabase/driver/bigquery_cloud_sdk/query_processor_test.clj
@@ -607,7 +607,7 @@
(with-test-db-name
(format "INSERT INTO `v4_test_data.%s` (ts, dt) VALUES (TIMESTAMP \"2020-01-01 00:00:00 UTC\", DATETIME \"2020-01-01 00:00:00\")"
table-name)))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(f table-name)
(finally
(bigquery.tx/execute! (with-test-db-name "DROP TABLE IF EXISTS `v4_test_data.%s`") table-name)))))))
diff --git a/modules/drivers/bigquery-cloud-sdk/test/metabase/driver/bigquery_cloud_sdk_test.clj b/modules/drivers/bigquery-cloud-sdk/test/metabase/driver/bigquery_cloud_sdk_test.clj
index ca4cb53faea..3c260953981 100644
--- a/modules/drivers/bigquery-cloud-sdk/test/metabase/driver/bigquery_cloud_sdk_test.clj
+++ b/modules/drivers/bigquery-cloud-sdk/test/metabase/driver/bigquery_cloud_sdk_test.clj
@@ -43,7 +43,7 @@
(deftest default-schema-test
(mt/test-driver :bigquery-cloud-sdk
- (is (nil? (driver.sql/default-schema :bigquery-cloud-sdk (mt/db))))))
+ (is (nil? (driver.sql/default-schema :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))))))
(defn ^:private get-test-data-name
[]
@@ -442,7 +442,7 @@
(deftest ^:parallel db-default-timezone-test
(mt/test-driver :bigquery-cloud-sdk
(is (= "UTC"
- (driver/db-default-timezone :bigquery-cloud-sdk (mt/db))))))
+ (driver/db-default-timezone :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))))))
(def ^:private numeric-val "-1.2E20")
(def ^:private decimal-val "2.3E16")
@@ -471,7 +471,7 @@
(fn [& args]
(swap! invocation-count inc)
(apply orig-describe-dataset-fields-reducible args))]
- (is (not= [] (into [] (take 10) (driver/describe-fields :bigquery-cloud-sdk (mt/db)))))
+ (is (not= [] (into [] (take 10) (driver/describe-fields :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider))))))
(is (= 1 @invocation-count)))))
(testing "describe-fields correctly partitions tables for a reducible result"
(let [invocation-count (atom 0)]
@@ -480,7 +480,7 @@
(fn [& args]
(swap! invocation-count inc)
(apply orig-describe-dataset-fields-reducible args))]
- (is (<= 22000 (count (into [] (driver/describe-fields :bigquery-cloud-sdk (mt/db))))))
+ (is (<= 22000 (count (into [] (driver/describe-fields :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))))))
(is (<= 20 @invocation-count))))))))))
(deftest ^:parallel describe-fields-truncates-data-type-test
@@ -651,13 +651,13 @@
(mt/dataset
native-dataset
(let [view-name "category_view"]
- (is (contains? (into #{} (:tables (driver/describe-database :bigquery-cloud-sdk (mt/db))))
+ (is (contains? (into #{} (:tables (driver/describe-database :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))))
{:schema (get-test-data-name) :name view-name :database_require_filter false})
"`describe-database` should see the view")
(is (= [{:name "id", :database-type "INTEGER" :base-type :type/Integer :database-position 0 :database-partitioned false :table-name view-name :table-schema (get-test-data-name)}
{:name "venue_name", :database-type "STRING" :base-type :type/Text :database-position 1 :database-partitioned false :table-name view-name :table-schema (get-test-data-name)}
{:name "category_name", :database-type "STRING" :base-type :type/Text :database-position 2 :database-partitioned false :table-name view-name :table-schema (get-test-data-name)}]
- (into [] (driver/describe-fields :bigquery-cloud-sdk (mt/db) {:table-names [view-name], :schema-names [(get-test-data-name)]})))
+ (into [] (driver/describe-fields :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)) {:table-names [view-name], :schema-names [(get-test-data-name)]})))
"`describe-fields` should see the fields in the view")
(testing "describe-database"
(qp.store/with-metadata-provider (mt/id)
@@ -666,7 +666,7 @@
:database_require_filter false}}
(into #{}
(filter (comp #{view-name} :name))
- (:tables (driver/describe-database :bigquery-cloud-sdk (mt/db))))))))
+ (:tables (driver/describe-database :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))))))))
(testing "We should be able to run queries against the view (#3414)"
(is (= [[1 "Red Medicine" "Asian"]
[2 "Stout Burgers & Beers" "Burger"]
@@ -732,13 +732,13 @@
(testing "describe-database carries table-level descriptions"
(let [descriptions (into {}
(map (juxt :name :description))
- (:tables (driver/describe-database :bigquery-cloud-sdk (mt/db))))]
+ (:tables (driver/describe-database :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))))]
(is (= "Top-level table description for sync test (#18872)."
(descriptions "described_table")))
(testing "including the escape sequences BigQuery writes into TABLE_OPTIONS"
(is (= escaped-description (descriptions "escaped_table"))))))
(testing "describe-fields carries column descriptions"
- (let [fields (into [] (driver/describe-fields :bigquery-cloud-sdk (mt/db)
+ (let [fields (into [] (driver/describe-fields :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider))
{:table-names ["described_table" "escaped_table"]
:schema-names [(get-test-data-name)]}))
by-name (into {} (map (juxt :name identity)) fields)]
@@ -775,7 +775,7 @@
:database_require_filter false}}
(into #{}
(filter (comp #{view-name} :name))
- (:tables (driver/describe-database :bigquery-cloud-sdk (mt/db))))))))
+ (:tables (driver/describe-database :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))))))))
(testing "We should be able to run queries against the view (#3414)"
(is (= [[42]]
(mt/rows
@@ -813,7 +813,7 @@
:bigquery-cloud-sdk
(mt/dataset
nested-records
- (let [database (driver/describe-database :bigquery-cloud-sdk (mt/db))
+ (let [database (driver/describe-database :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))
tables (sort-by :name (into [] (:tables database)))]
(is (=? [{:name "records"} {:name "records_o"}] tables))
(is (=? [{:name "id"}
@@ -848,7 +848,7 @@
(if (set? n)
(sort-by :name n)
n))
- (into [] (driver/describe-fields :bigquery-cloud-sdk (mt/db) {:table-names ["records"]})))))
+ (into [] (driver/describe-fields :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)) {:table-names ["records"]})))))
(is (=? [{:name "id"}
{:name "r"
:database-type "RECORD",
@@ -863,7 +863,7 @@
(sort-by :name n)
n))
(into [] (filter (fn [x] (= (:table-name x) "records_o"))
- (driver/describe-fields :bigquery-cloud-sdk (mt/db) {:table-names ["records" "records_o"]}))))))))))
+ (driver/describe-fields :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)) {:table-names ["records" "records_o"]}))))))))))
(deftest ^:parallel query-nested-fields-test
(mt/test-driver :bigquery-cloud-sdk
@@ -908,7 +908,7 @@
"partition_by_ingestion_time"
"partition_by_range_not_required"
"partition_by_ingestion_time_not_required"} :name))
- (:tables (driver/describe-database :bigquery-cloud-sdk (mt/db))))))))
+ (:tables (driver/describe-database :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))))))))
(testing "tables that require a filter are correctly identified"
(is (= table-name->is-filter-required?
(t2/select-fn->fn :name :database_require_filter :model/Table
@@ -996,13 +996,13 @@
PARTITION BY RANGE_BUCKET(customer_id, GENERATE_ARRAY(0, 100, 10));"
(fmt-table-name "partitioned_table")))
(testing "sanity check that it's not required at first"
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(is (false? (t2/select-one-fn :database_require_filter :model/Table :name table-name))))
(testing "sync should update require filter and set it to true"
(bigquery.tx/execute! (format "ALTER TABLE IF EXISTS %s
SET OPTIONS(require_partition_filter = true);"
(fmt-table-name table-name)))
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(is (true? (t2/select-one-fn :database_require_filter :model/Table :name table-name :db_id (mt/id)))))
(finally
(drop-table-if-exists! table-name)))))))))
@@ -1018,12 +1018,12 @@
(let [category-field-id (u/auto-retry 1
(try (mt/id :fv_partitioned_table :category)
(catch Exception e
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(throw e))))
category-field (t2/select-one :model/Field :id category-field-id)
table (t2/select-one :model/Table :id (:table_id category-field))]
(testing "Dispatcher identifies this as a non-batch-able table"
- (is (false? (#'sync.field-values/can-batch-distinct? table))))
+ (is (false? (#'sync.field-values/can-batch-distinct? (lib.metadata/database (mt/metadata-provider)) table))))
(testing "Distinct values come back correctly via the fallback per-field path"
(t2/update! :model/Field category-field-id {:has_field_values :list})
(t2/delete! :model/FieldValues :field_id category-field-id)
@@ -1044,7 +1044,7 @@
(try
(mt/id :fv_partitioned_table :category)
(catch Exception e
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(throw e))))]
(t2/update! :model/Field category-field-id {:has_field_values :search})
(t2/delete! :model/FieldValues :field_id category-field-id)
@@ -1174,7 +1174,7 @@
:dataset-filters-patterns "chicago_taxi_trips"))}]
(mt/with-db temp-db
(testing " for sync"
- (sync/sync-database! temp-db {:scan :schema})
+ (sync/sync-database! (:id temp-db) {:scan :schema})
(let [[tbl & more-tbl] (t2/select :model/Table :db_id db-id)]
(is (some? tbl))
(is (nil? more-tbl))
@@ -1263,7 +1263,7 @@
decimal-val
bignumeric-val
bigdecimal-val)
- (is (contains? (into #{} (:tables (driver/describe-database :bigquery-cloud-sdk (mt/db))))
+ (is (contains? (into #{} (:tables (driver/describe-database :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)))))
{:schema (get-test-data-name) :name tbl-nm :database_require_filter false})
"`describe-database` should see the table")
(is (= [{:base-type :type/Decimal
@@ -1336,9 +1336,9 @@
:base-type :type/Decimal,
:database-partitioned false,
:database-position 9}]
- (into [] (driver/describe-fields :bigquery-cloud-sdk (mt/db) {:table-names [tbl-nm] :schema-names [(get-test-data-name)]})))
+ (into [] (driver/describe-fields :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)) {:table-names [tbl-nm] :schema-names [(get-test-data-name)]})))
"`describe-fields` should see the fields in the table")
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(testing "We should be able to run queries against the table"
(doseq [[col-nm param-v] {"numeric_col" (bigdec numeric-val)
"decimal_col" (bigdec decimal-val)
@@ -1394,7 +1394,7 @@
:base-type :type/Array,
:database-partitioned false,
:database-position 3}]
- (into [] (driver/describe-fields :bigquery-cloud-sdk (mt/db) {:table-names [tbl-nm] :schema-names [(get-test-data-name)]})))
+ (into [] (driver/describe-fields :bigquery-cloud-sdk (lib.metadata/database (mt/metadata-provider)) {:table-names [tbl-nm] :schema-names [(get-test-data-name)]})))
"`describe-fields` should detect the correct base-type for array type columns"))))))
(deftest sync-inactivates-old-duplicate-tables
@@ -1405,7 +1405,7 @@
(let [synced-tables (t2/select :model/Table :db_id (mt/id))]
(is (= 2 (count synced-tables)))
(t2/insert! :model/Table (map #(dissoc % :id :entity_id :schema) synced-tables))
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(let [synced-tables (t2/select :model/Table :db_id (mt/id))]
(is (partial= {true [{:name "messages"} {:name "users"}]
false [{:name "messages"} {:name "users"}]}
@@ -1527,7 +1527,7 @@
(defn- synced-tables [db-attributes]
(mt/with-temp [:model/Database db db-attributes]
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(t2/select :model/Table :db_id (u/the-id db))))
(deftest dataset-filtering-test
@@ -1685,7 +1685,7 @@
[22345678901234567890.1234567890M]
[32345678901234567890.1234567890M]]]])
;; Must sync field values
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= "BIGNUMERIC"
(t2/select-one-fn :database_type :model/Field :id (mt/id :bigthings :bd))))
(is (= [[12000000000000000000M 1]
@@ -1701,7 +1701,7 @@
(deftest bigquery-process-stop-test
(mt/test-driver
:bigquery-cloud-sdk
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(let [before-names (future-thread-names)]
(doseq [:let [callbacks (atom 0)]
[stop-tag stopper] [[:exception #(throw (Exception. "My Exception"))]
diff --git a/modules/drivers/clickhouse/src/metabase/driver/clickhouse.clj b/modules/drivers/clickhouse/src/metabase/driver/clickhouse.clj
index b38ead37423..7e75ca94e99 100644
--- a/modules/drivers/clickhouse/src/metabase/driver/clickhouse.clj
+++ b/modules/drivers/clickhouse/src/metabase/driver/clickhouse.clj
@@ -21,7 +21,7 @@
[metabase.driver.sql-jdbc.sync :as sql-jdbc.sync]
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.driver.sql.util :as sql.u]
- [metabase.util :as u]
+ [metabase.driver.util :as driver.u]
[metabase.util.i18n :refer [deferred-tru]]
[metabase.util.log :as log]
[metabase.util.performance :as perf])
@@ -108,16 +108,6 @@
db-or-id-or-spec
options
(fn [^java.sql.Connection conn]
- (when-let [db (cond
- ;; id?
- (integer? db-or-id-or-spec)
- (driver-api/with-metadata-provider db-or-id-or-spec
- (driver-api/database (driver-api/metadata-provider)))
- ;; db?
- (u/id db-or-id-or-spec) db-or-id-or-spec
- ;; otherwise it's a spec and we can't get the db
- :else nil)]
- (sql-jdbc.execute/set-role-if-supported! driver conn db))
(when-not (sql-jdbc.execute/recursive-connection?)
(let [^com.clickhouse.jdbc.ConnectionImpl clickhouse-conn (.unwrap conn com.clickhouse.jdbc.ConnectionImpl)
query-settings (new QuerySettings)]
@@ -523,7 +513,7 @@
(defmethod driver/rename-table! :clickhouse
[_driver db-id old-table-name new-table-name]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(with-open [stmt (.createStatement ^java.sql.Connection (:connection conn))]
(let [sql (format "RENAME TABLE %s TO %s" (quote-name old-table-name) (quote-name new-table-name))]
(.execute stmt sql)))))
diff --git a/modules/drivers/clickhouse/test/metabase/driver/clickhouse/index_test.clj b/modules/drivers/clickhouse/test/metabase/driver/clickhouse/index_test.clj
index 63798fb29c3..e46dcb04d0c 100644
--- a/modules/drivers/clickhouse/test/metabase/driver/clickhouse/index_test.clj
+++ b/modules/drivers/clickhouse/test/metabase/driver/clickhouse/index_test.clj
@@ -13,6 +13,7 @@
[metabase.driver :as driver]
[metabase.driver.clickhouse :as clickhouse]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.driver.util :as driver.u]
[metabase.test :as mt]
[metabase.util.malli.registry :as mr]))
@@ -164,7 +165,7 @@
(jdbc/execute! conn-spec (into [sql] params))
(is (= expected (sorting-key conn-spec "default" ctas-table)))
(testing "and fetch-table-indexes reports it, with the schema named"
- (is (=? fetched (driver/fetch-table-indexes :clickhouse db "default" ctas-table)))))
+ (is (=? fetched (driver/fetch-table-indexes :clickhouse (driver.u/lib-database (:id db)) "default" ctas-table)))))
(finally (drop! ctas-table))))
(testing "CREATE TABLE seam (create-table!): create the table, read the sorting key back"
(drop! crt-table)
@@ -173,7 +174,7 @@
order-by-columns {:indexes indexes})
(is (= expected (sorting-key conn-spec "default" crt-table)))
(testing "and fetch-table-indexes reports it, with a blank schema (the session database)"
- (is (=? fetched (driver/fetch-table-indexes :clickhouse db nil crt-table))))
+ (is (=? fetched (driver/fetch-table-indexes :clickhouse (driver.u/lib-database (:id db)) nil crt-table))))
(finally (drop! crt-table))))))))))))
(deftest ^:synchronized skip-index-live-test
@@ -208,7 +209,7 @@
:key-columns ["a"]
:include-columns []
:partial-predicate nil}]
- (driver/fetch-table-indexes :clickhouse db "default" table))))
+ (driver/fetch-table-indexes :clickhouse (driver.u/lib-database (:id db)) "default" table))))
(finally (drop!))))))))
(deftest ^:synchronized escaped-names-live-test
@@ -229,7 +230,7 @@
(driver/compile-create-index :clickhouse nil table index))
(is (=? [{:name "ix`tick" :kind :skip-index :key-columns [column]}
{:name nil :kind :order-by :key-columns [column]}]
- (driver/fetch-table-indexes :clickhouse db "default" table)))
+ (driver/fetch-table-indexes :clickhouse (driver.u/lib-database (:id db)) "default" table)))
(finally (drop!))))))))
(deftest ^:synchronized restricted-user-live-test
@@ -258,7 +259,7 @@
(testing "fetch-table-indexes still reads both index kinds"
(is (=? [{:name "evt_minmax" :kind :skip-index :access-method "minmax" :key-columns ["a"]}
{:name nil :kind :order-by :key-columns ["a" "b"]}]
- (driver/fetch-table-indexes :clickhouse db "default" table))))
+ (driver/fetch-table-indexes :clickhouse (driver.u/lib-database (:id db)) "default" table))))
(finally
(exec! (format "DROP TABLE IF EXISTS `%s`" table))
(exec! (format "DROP USER IF EXISTS `%s`" user)))))))))
diff --git a/modules/drivers/clickhouse/test/metabase/driver/clickhouse_impersonation_test.clj b/modules/drivers/clickhouse/test/metabase/driver/clickhouse_impersonation_test.clj
index 673e37e5f0f..bc950db57d5 100644
--- a/modules/drivers/clickhouse/test/metabase/driver/clickhouse_impersonation_test.clj
+++ b/modules/drivers/clickhouse/test/metabase/driver/clickhouse_impersonation_test.clj
@@ -167,7 +167,7 @@
"insert_quorum" "2"})
(ctd/exec-statements grant-statements cluster-port {"wait_end_of_query" "1"})
(t2.with-temp/with-temp [:model/Database db cluster-details]
- (mt/with-db db (sync.core/sync-database! db)
+ (mt/with-db db (sync.core/sync-database! (:id db))
(letfn [(check-impersonation! [roles expected]
(impersonation.tu/with-impersonations!
{:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
diff --git a/modules/drivers/clickhouse/test/metabase/driver/clickhouse_introspection_test.clj b/modules/drivers/clickhouse/test/metabase/driver/clickhouse_introspection_test.clj
index 6efb058fe1b..78b20f282bd 100644
--- a/modules/drivers/clickhouse/test/metabase/driver/clickhouse_introspection_test.clj
+++ b/modules/drivers/clickhouse/test/metabase/driver/clickhouse_introspection_test.clj
@@ -5,6 +5,8 @@
[clojure.test :refer :all]
[metabase.driver :as driver]
[metabase.driver.common :as driver.common]
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.query-processor.test :as qp]
[metabase.sync.core :as sync]
[metabase.test :as mt]
@@ -16,7 +18,7 @@
(defn- desc-table!
[db table-name]
(into #{} (map #(select-keys % [:name :database-type :base-type :database-required])
- (:fields (driver/describe-table :clickhouse db {:name table-name})))))
+ (:fields (driver/describe-table :clickhouse (driver.u/ensure-lib-database db) {:name table-name})))))
(mt/defdataset introspection_db
[["enums_base_types"
@@ -466,7 +468,7 @@
:database-type "SimpleAggregateFunction(sum, Int64)"
:base-type :type/BigInteger
:database-position 4})]}
- (-> (driver/describe-table :clickhouse (mt/db) {:name "aggregate_functions_filter_test"})
+ (-> (driver/describe-table :clickhouse (lib.metadata/database (mt/metadata-provider)) {:name "aggregate_functions_filter_test"})
(update :fields (partial sort-by :database-position))))))))
(deftest clickhouse-filtered-aggregate-functions-test-result-set
@@ -501,7 +503,7 @@
(mt/test-driver :clickhouse
(mt/dataset metabase_db_scan_test
(is (= {:tables test-tables}
- (driver/describe-database :clickhouse (mt/db)))))))
+ (driver/describe-database :clickhouse (lib.metadata/database (mt/metadata-provider))))))))
(deftest ^:parallel clickhouse-describe-database-all
(mt/test-driver :clickhouse
@@ -513,7 +515,7 @@
:clickhouse :db
{:database-name "default"})
{:db-filters-type "all"})}]
- (let [describe-result (driver/describe-database :clickhouse db)
+ (let [describe-result (driver/describe-database :clickhouse (driver.u/ensure-lib-database db))
tables (into #{} (:tables describe-result))]
;; check the existence of at least some test tables here
(doseq [table test-tables]
@@ -536,7 +538,7 @@
{:db-filters-type "inclusion"
:db-filters-patterns "metabase_db_scan_test, information_schema"})}]
(let [{:keys [tables] :as _describe-result}
- (driver/describe-database :clickhouse db)
+ (driver/describe-database :clickhouse (driver.u/ensure-lib-database db))
tables-table {:name "tables"
:description nil
:schema "information_schema"}
@@ -567,7 +569,7 @@
;; Sync the database - this should not crash despite the parameterized view
(is (not= ::thrown
(try
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(catch Throwable _e
::thrown)))
"Sync should not throw an exception when encountering a parameterized view")
diff --git a/modules/drivers/clickhouse/test/metabase/driver/clickhouse_test.clj b/modules/drivers/clickhouse/test/metabase/driver/clickhouse_test.clj
index ef5acb863b8..d457661b0f0 100644
--- a/modules/drivers/clickhouse/test/metabase/driver/clickhouse_test.clj
+++ b/modules/drivers/clickhouse/test/metabase/driver/clickhouse_test.clj
@@ -14,6 +14,7 @@
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
[metabase.driver.sql.query-processor :as sql.qp]
+ [metabase.driver.util :as driver.u]
[metabase.lib-be.core :as lib-be]
[metabase.lib.card :as lib.card]
[metabase.lib.core :as lib]
@@ -40,17 +41,17 @@
db-name (or (:dbname base-details) (:db base-details))]
(testing "database named by the older `db` spelling, which never reaches the JDBC URL"
(is (= db-name
- (driver.sql/default-schema :clickhouse (mt/db)))))
+ (driver.sql/default-schema :clickhouse (lib.metadata/database (mt/metadata-provider))))))
(testing "database configured in the connection details"
(let [details (assoc base-details :dbname db-name)]
(mt/with-temp [:model/Database database {:engine :clickhouse, :details details}]
(is (= db-name
- (driver.sql/default-schema :clickhouse database))))))
+ (driver.sql/default-schema :clickhouse (driver.u/ensure-lib-database database)))))))
(testing "details naming no database: the server reports the one the connection opened"
(mt/with-temp [:model/Database database {:engine :clickhouse
:details (dissoc base-details :db :dbname)}]
(is (= "default"
- (driver.sql/default-schema :clickhouse database))))))))
+ (driver.sql/default-schema :clickhouse (driver.u/ensure-lib-database database)))))))))
;; the mt/with-dynamic-redefs macro was renamed to mt/with-dynamic-fn-redefs for 0.53+
;; as 0.52 is still tested by CI we will check which macro is defined and use that
@@ -338,7 +339,7 @@
[:model/Database db
{:engine :clickhouse
:details (mt/dbdef->connection-details :clickhouse :db {:database-name "default"})}]
- (let [version (driver/dbms-version :clickhouse db)]
+ (let [version (driver/dbms-version :clickhouse (driver.u/ensure-lib-database db))]
(is (number? (get-in version [:semantic-version :major])))
(is (number? (get-in version [:semantic-version :minor])))
(is (string? (get version :version)))))))
@@ -362,7 +363,7 @@
(deftest ^:synchronized clickhouse-session-timezone-does-not-leak-across-borrows-test
(mt/test-driver :clickhouse
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id))
(let [underlying-conn-ids (atom [])
observe (fn [opts]
(sql-jdbc.execute/do-with-connection-with-options
@@ -441,34 +442,29 @@
:clickhouse
{})))))
-(deftest ^:parallel clickhouse-tls
+(deftest clickhouse-tls
(mt/test-driver :clickhouse
- (let [working-dir (System/getProperty "user.dir")
- cert-path (str working-dir "/modules/drivers/clickhouse/.docker/clickhouse/single_node_tls/certificates/ca.crt")
- additional-options (str "sslrootcert=" cert-path)]
+ (let [working-dir (System/getProperty "user.dir")
+ cert-path (str working-dir "/modules/drivers/clickhouse/.docker/clickhouse/single_node_tls/certificates/ca.crt")
+ additional-options (str "sslrootcert=" cert-path)
+ timezone (fn [details]
+ (mt/with-temp [:model/Database {db-id :id} {:engine :clickhouse :details details}]
+ (driver/db-default-timezone :clickhouse (driver.u/lib-database db-id))))]
(testing "simple connection with a single database"
(is (= "UTC"
- (driver/db-default-timezone
- :clickhouse
- (sql-jdbc.conn/connection-details->spec
- :clickhouse
- {:ssl true
- :host "server.clickhouseconnect.test"
- :port 8443
- :additional-options additional-options})))))
+ (timezone {:ssl true
+ :host "server.clickhouseconnect.test"
+ :port 8443
+ :additional-options additional-options}))))
(testing "connection with multiple databases"
(is (= "UTC"
- (driver/db-default-timezone
- :clickhouse
- (sql-jdbc.conn/connection-details->spec
- :clickhouse
- {:ssl true
- :host "server.clickhouseconnect.test"
- :port 8443
- :enable-multiple-db true
- :db-filters-patterns "default, system"
- :db-filters-type "inclusion"
- :additional-options additional-options}))))))))
+ (timezone {:ssl true
+ :host "server.clickhouseconnect.test"
+ :port 8443
+ :enable-multiple-db true
+ :db-filters-patterns "default, system"
+ :db-filters-type "inclusion"
+ :additional-options additional-options})))))))
(deftest ^:parallel clickhouse-nippy
(mt/test-driver :clickhouse
@@ -497,7 +493,7 @@
(deftest pool-does-not-test-connections-on-checkout-test
(with-redefs [driver-api/is-test? false]
- (is (false? (get (sql-jdbc.conn/data-warehouse-connection-pool-properties :clickhouse (mt/db)) "testConnectionOnCheckout")))))
+ (is (false? (get (sql-jdbc.conn/data-warehouse-connection-pool-properties :clickhouse (lib.metadata/database (mt/metadata-provider))) "testConnectionOnCheckout")))))
(deftest ^:parallel clickhouse-can-connect
(mt/test-driver :clickhouse
@@ -554,7 +550,7 @@
(driver/insert-into! :clickhouse (:id db) table [:id :name] [[42 "Bob"] [43 "Alice"]])
(is (= #{{:id 42, :name "Bob"}
{:id 43, :name "Alice"}}
- (set (sql-jdbc/query :clickhouse db {:select [:*] :from [table]}))))
+ (set (sql-jdbc/query :clickhouse (driver.u/ensure-lib-database db) {:select [:*] :from [table]}))))
(finally
(driver/drop-table! :clickhouse (:id db) table)))))))
@@ -741,7 +737,7 @@
(mt/test-driver :clickhouse
(let [details (-> (mt/dbdef->connection-details :clickhouse :db {:database-name "uploads_schema"})
(assoc :enable-multiple-db false))
- dbms-version (driver/dbms-version :clickhouse (mt/db))
+ dbms-version (driver/dbms-version :clickhouse (lib.metadata/database (mt/metadata-provider)))
conn-spec (sql-jdbc.conn/connection-details->spec :clickhouse details)]
(driver/create-schema-if-needed! :clickhouse conn-spec "uploads_schema")
(try
@@ -773,7 +769,7 @@
(is (= [[1 " Luke Skywalker"]
[2 " Darth Vader"]]
(query-model)))
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(is (= [[1 " Luke Skywalker"]
[2 " Darth Vader"]]
(query-model))))))))
diff --git a/modules/drivers/databricks/test/metabase/driver/databricks_test.clj b/modules/drivers/databricks/test/metabase/driver/databricks_test.clj
index 3b8d4561a0b..fc45acd7e6e 100644
--- a/modules/drivers/databricks/test/metabase/driver/databricks_test.clj
+++ b/modules/drivers/databricks/test/metabase/driver/databricks_test.clj
@@ -59,11 +59,11 @@
{:name "reviews", :schema (maybe-qualify-schema "test-data"), :description nil}
{:name "orders", :schema (maybe-qualify-schema "test-data"), :description nil}
{:name "products", :schema (maybe-qualify-schema "test-data"), :description nil}}
- (into #{} (:tables (driver/describe-database :databricks (db)))))))
+ (into #{} (:tables (driver/describe-database :databricks (driver.u/ensure-lib-database (db))))))))
(testing "`driver/describe-database` returns expected results for `all` schema filters."
- (let [actual-tables (update (driver/describe-database :databricks (-> (db)
- (update :details dissoc :schema-filters-patterns)
- (update :details assoc :schema-filters-type "all")))
+ (let [actual-tables (update (driver/describe-database :databricks (driver.u/ensure-lib-database (-> (db)
+ (update :details dissoc :schema-filters-patterns)
+ (update :details assoc :schema-filters-type "all"))))
;; `:tables` is a reducible (streamed); realize it for the set ops below
:tables #(into #{} %))]
(testing "tables from multiple schemas were found"
@@ -76,9 +76,9 @@
(testing "information_schema is excluded"
(is (empty? (filter #(str/includes? "information_schema" (:schema %)) (:tables actual-tables)))))))
(testing "`driver/describe-database` returns expected results for `exclusion` schema filters."
- (let [actual-tables (update (driver/describe-database :databricks (update (db) :details assoc
- :schema-filters-patterns (maybe-qualify-schema "test-data")
- :schema-filters-type "exclusion"))
+ (let [actual-tables (update (driver/describe-database :databricks (driver.u/ensure-lib-database (update (db) :details assoc
+ :schema-filters-patterns (maybe-qualify-schema "test-data")
+ :schema-filters-type "exclusion")))
;; `:tables` is a reducible (streamed); realize it for the set ops below
:tables #(into #{} %))]
(testing "tables from multiple schemas were found"
@@ -89,8 +89,8 @@
(deftest ^:parallel describe-fields-test
(with-and-without-multi-level
(mt/test-driver :databricks
- (let [fields (vec (driver/describe-fields :databricks (db) {:schema-names [(maybe-qualify-schema "test-data")]
- :table-names ["orders"]}))]
+ (let [fields (vec (driver/describe-fields :databricks (driver.u/ensure-lib-database (db)) {:schema-names [(maybe-qualify-schema "test-data")]
+ :table-names ["orders"]}))]
(testing "Underlying query returns only fields from selected catalog"
(is (= 9 (count fields))))
(testing "Expected fields are returned"
@@ -174,8 +174,8 @@
(with-and-without-multi-level
(mt/test-driver :databricks
(let [fks (driver/describe-fks :databricks
- (assoc (lib.metadata/database (mt/metadata-provider))
- :details (:details (db)))
+ (driver.u/ensure-lib-database (assoc (lib.metadata/database (mt/metadata-provider))
+ :details (:details (db))))
{:schema-names [(maybe-qualify-schema "test-data")]
:table-names ["orders"]})]
(testing "Expected fks are returned"
@@ -386,7 +386,7 @@
(mt/with-temp [:model/Database {db-id :id :as db} {:engine :databricks :details details}]
(mt/with-db db
(testing "With multi-level-schema default (off)"
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= #{"test-data"} (t2/select-fn-set :schema :model/Table :db_id (mt/id) :active true)))
(is (= 52 (count (t2/select
:model/Field
@@ -399,7 +399,7 @@
:schema-filters-patterns multi-pattern)})
(mt/with-db
(t2/select-one :model/Database db-id)
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= #{(format "%s.%s" (:catalog details) "test-data")
(format "%s.%s" multicatalog multicatalog-schema)}
(t2/select-fn-set :schema :model/Table :db_id (mt/id) :active true)))
@@ -413,7 +413,7 @@
(t2/update! :model/Database db-id {:details (assoc details :multi-level-schema false)})
(mt/with-db
(t2/select-one :model/Database db-id)
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= #{"test-data"} (t2/select-fn-set :schema :model/Table :db_id (mt/id) :active true)))
(is (= 52 (count (t2/select
:model/Field
@@ -436,7 +436,7 @@
;; First sync with multi-level-schema off
(mt/with-db db
(testing "With multi-level-schema default (off)"
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
;; originally we have unqualified schemas, only in one catalog
(is (= #{"test-data"}
(t2/select-fn-set :schema :model/Table :db_id (mt/id))))))
@@ -448,7 +448,7 @@
(t2/update! :model/Table {:db_id db-id} {:active false})
(mt/with-db
(t2/select-one :model/Database db-id)
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= #{(format "%s.%s" (:catalog details) "test-data")
(format "%s.%s" multicatalog multicatalog-schema)}
;; active` *and* inactive tables both have their schemas changed.
@@ -470,7 +470,7 @@
:schema-filters-patterns (str/join ", " schema-filters)))}]
(mt/with-db
(t2/select-one :model/Database db-id)
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(let [table-schemas (t2/select-fn-set :schema :model/Table :db_id (mt/id) :active true)]
(is (= schema-filters table-schemas))
(is (nil? (some #(str/starts-with? % "__databricks") table-schemas)))))))))
@@ -491,7 +491,7 @@
:schema-filters-patterns multi-pattern)]
(mt/with-temp [:model/Database {db-id :id :as db} {:engine :databricks :details details}]
(mt/with-db db
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(let [mp (lib-be/application-database-metadata-provider db-id)
[t1-id t2-id] (t2/select-fn-vec
:id
diff --git a/modules/drivers/druid-jdbc/test/metabase/driver/druid_jdbc/nested_field_columns_test.clj b/modules/drivers/druid-jdbc/test/metabase/driver/druid_jdbc/nested_field_columns_test.clj
index ba3dc95dcaf..c3831de3568 100644
--- a/modules/drivers/druid-jdbc/test/metabase/driver/druid_jdbc/nested_field_columns_test.clj
+++ b/modules/drivers/druid-jdbc/test/metabase/driver/druid_jdbc/nested_field_columns_test.clj
@@ -8,6 +8,7 @@
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
[metabase.driver.sql-jdbc.sync :as sql-jdbc.sync]
[metabase.driver.sql-jdbc.sync.describe-table :as sql-jdbc.describe-table]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.sync.core :as sync]
[metabase.test :as mt]
[toucan2.core :as t2]))
@@ -23,7 +24,7 @@
(let [table (t2/select-one :model/Table :id (mt/id :json))]
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (mt/id)
nil
(fn [^java.sql.Connection conn]
(let [fields (sql-jdbc.describe-table/describe-table-fields driver/*driver* conn table nil)
@@ -40,7 +41,7 @@
json
(testing "Nested field column listing"
(is (= [:type/JSON :type/SerializedJSON]
- (->> (sql-jdbc.sync/describe-table driver/*driver* (mt/db) {:name "json"})
+ (->> (sql-jdbc.sync/describe-table driver/*driver* (lib.metadata/database (mt/metadata-provider)) {:name "json"})
:fields
(filter #(= (:name %) "json_bit"))
first
@@ -110,7 +111,7 @@
:nfc-path [:json_bit "title"]}}
(sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{:name "json" :id (mt/id "json")})))))))
(deftest ^:parallel big-nested-field-column-test
@@ -122,7 +123,7 @@
(is (= sql-jdbc.describe-table/max-nested-field-columns
(count (sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{:name "big_json" :id (mt/id "big_json")}))))))))
(deftest json-unfolding-initially-false-test
@@ -137,7 +138,7 @@
:details (assoc (:details database) :json-unfolding false)}]
(mt/with-db database
;; Sync the new database
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [get-field (fn [] (t2/select-one :model/Field :id (mt/id :json :json_bit)))
get-database (fn [] (t2/select-one :model/Database :id (mt/id)))
set-json-unfolding-for-field! (fn [v]
@@ -166,11 +167,11 @@
(set-json-unfolding-for-db! true)
(is (true? (:json-unfolding (:details (get-database)))))
(is (true? (:json_unfolding (get-field))))
- (sync/sync-database! (get-database))
+ (sync/sync-database! (:id (get-database)))
(is (seq (nested-fields))))
(testing "nested fields are removed when json unfolding is disabled again"
(set-json-unfolding-for-db! false)
- (sync/sync-database! (get-database))
+ (sync/sync-database! (:id (get-database)))
(is (empty? (nested-fields))))))))))))
(deftest json-unfolding-initially-true-test
@@ -184,7 +185,7 @@
:details (assoc (:details database) :json-unfolding true)}]
(mt/with-db database
;; Sync the new database
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [field (t2/select-one :model/Field :id (mt/id :json :json_bit))
get-database (fn [] (t2/select-one :model/Database :id (mt/id)))
set-json-unfolding-for-field! (fn [v]
@@ -203,11 +204,11 @@
(is (seq (nested-fields))))
(testing "nested fields are removed when json unfolding is disabled for the DB"
(set-json-unfolding-for-db! false)
- (sync/sync-database! (get-database))
+ (sync/sync-database! (:id (get-database)))
(is (empty? (nested-fields))))
(testing "nested fields are added when json unfolding is enabled again for the DB"
(set-json-unfolding-for-db! true)
- (sync/sync-database! (get-database))
+ (sync/sync-database! (:id (get-database)))
(is (seq (nested-fields))))
(testing "nested fields are removed when json unfolding is disabled for the field"
(set-json-unfolding-for-field! false)
diff --git a/modules/drivers/druid-jdbc/test/metabase/driver/druid_jdbc_test.clj b/modules/drivers/druid-jdbc/test/metabase/driver/druid_jdbc_test.clj
index 8a0e8e73629..48e7668697e 100644
--- a/modules/drivers/druid-jdbc/test/metabase/driver/druid_jdbc_test.clj
+++ b/modules/drivers/druid-jdbc/test/metabase/driver/druid_jdbc_test.clj
@@ -35,7 +35,7 @@
(is (= #{{:schema "druid", :name "checkins" :description nil, :is_writable nil}
{:schema "druid", :name "json" :description nil, :is_writable nil}
{:schema "druid", :name "big_json" :description nil, :is_writable nil}}
- (into #{} (:tables (driver/describe-database :druid-jdbc (mt/db)))))))
+ (into #{} (:tables (driver/describe-database :druid-jdbc (lib.metadata/database (mt/metadata-provider))))))))
(testing "describe-table"
(is (=? {:schema "druid"
:name "checkins"
@@ -135,11 +135,11 @@
:base-type :type/BigInteger,
:database-position 9,
:json-unfolding false}]}
- (-> (driver/describe-table :druid-jdbc (mt/db) {:schema "druid" :name "checkins"})
+ (-> (driver/describe-table :druid-jdbc (lib.metadata/database (mt/metadata-provider)) {:schema "druid" :name "checkins"})
(update :fields (partial sort-by :name))))))
(testing "Full sync does not throw an exception (field values are calculated only for eligible fields)"
(is (=? [::success some?]
- (try (let [result (sync/sync-database! (mt/db))]
+ (try (let [result (sync/sync-database! (mt/id))]
[::success result])
(catch Throwable t
[::failure t]))))))))
@@ -151,7 +151,7 @@
(mt/with-temporary-setting-values [driver.settings/nested-field-columns-value-length-limit 10]
(testing "nested fields when length limit is exceeded"
(is (= #{}
- (sql-jdbc.sync.interface/describe-nested-field-columns driver/*driver* (mt/db) (t2/select-one :model/Table (mt/id :json)))))))
+ (sql-jdbc.sync.interface/describe-nested-field-columns driver/*driver* (lib.metadata/database (mt/metadata-provider)) (t2/select-one :model/Table (mt/id :json)))))))
(testing "nested fields when length limit is exceeded"
(is (= #{{:name "json_bit → noop",
:database-type "timestamp",
@@ -216,7 +216,7 @@
:json-unfolding false,
:visibility-type :normal,
:nfc-path [:json_bit "title"]}}
- (sql-jdbc.sync.interface/describe-nested-field-columns driver/*driver* (mt/db) (t2/select-one :model/Table (mt/id :json)))))))))
+ (sql-jdbc.sync.interface/describe-nested-field-columns driver/*driver* (lib.metadata/database (mt/metadata-provider)) (t2/select-one :model/Table (mt/id :json)))))))))
(defn- db-dbms-version [db-or-id]
(t2/select-one-fn :dbms_version :model/Database :id (u/the-id db-or-id)))
@@ -236,7 +236,7 @@
_ (t2/update! :model/Database (u/the-id db) {:dbms_version nil})
db (t2/select-one :model/Database :id (u/the-id db))
version-after-update (db-dbms-version db)
- _ (sync-dbms-ver/sync-dbms-version! db)]
+ _ (sync-dbms-ver/sync-dbms-version! (driver.u/lib-database (:id db)))]
(testing "On startup is the dbms-version specified?"
(is (nil? (check-dbms-version version-on-load))))
(testing "Check to make sure the test removed the timezone"
diff --git a/modules/drivers/mongo/src/metabase/driver/mongo.clj b/modules/drivers/mongo/src/metabase/driver/mongo.clj
index 9af083a126e..53f0825d707 100644
--- a/modules/drivers/mongo/src/metabase/driver/mongo.clj
+++ b/modules/drivers/mongo/src/metabase/driver/mongo.clj
@@ -93,8 +93,8 @@
;; either may name several hosts (a replica set). Parsing the connection string the driver itself will use is the
;; only way to be sure we see the same hosts it will connect to.
(if (and (not use-conn-uri) (str/blank? host))
- ;; nothing to parse and nothing to connect to. `:write_data_details` and `:admin_details` are overlays merged on
- ;; top of `:details`, so they arrive here as partial maps; one that carries no host of its own names no hosts,
+ ;; nothing to parse and nothing to connect to. `:write_data_details` is an overlay merged on top of `:details`,
+ ;; so it arrives here as a partial map; one that carries no host of its own names no hosts,
;; which is what the default implementation reports for the same input.
[]
;; Parsing failures are left to propagate. It is tempting to fall back to the `:host` field, since a connection
@@ -487,7 +487,7 @@
Each row represents leaf in sampled documents, its type and indices of keys present in the path of mongo of nested
object."
- [database :- :metabase.warehouses.schema/database
+ [database :- driver-api/schema.metadata.database
collection-name :- :string]
(let [pipeline (describe-table-pipeline {:collection-name collection-name
:sample-size (* table-rows-sample/nested-field-sample-limit 2)
diff --git a/modules/drivers/mongo/test/metabase/driver/mongo/database_test.clj b/modules/drivers/mongo/test/metabase/driver/mongo/database_test.clj
index 06f38d040ea..3db6c174ab1 100644
--- a/modules/drivers/mongo/test/metabase/driver/mongo/database_test.clj
+++ b/modules/drivers/mongo/test/metabase/driver/mongo/database_test.clj
@@ -20,11 +20,13 @@
(mongo.db/db-name {:conn-uri "mongodb://localhost/some_db"})))))
(deftest ^:parallel details-normalized-respects-connection-type-test
- (testing "Mongo's details-normalized uses effective-details for full database objects"
+ (testing "Mongo's details-normalized uses effective-details for the Lib metadata of a database"
(let [read-details {:dbname "test-db" :host "read-host"}
write-details {:dbname "test-db" :host "write-host"}
- database {:details read-details
- :write_data_details write-details}]
+ database {:lib/type :metadata/database
+ :id 1
+ :details read-details
+ :write-data-details write-details}]
(testing "default connection type uses :details"
(let [result (mongo.db/details-normalized database)]
(is (= "read-host" (:host result)))))
@@ -34,7 +36,9 @@
(is (= "write-host" (:host result))))))))
(testing "Mongo's details-normalized falls back to :details when no write details"
(let [read-details {:dbname "test-db" :host "read-host"}
- database {:details read-details}]
+ database {:lib/type :metadata/database
+ :id 1
+ :details read-details}]
(driver.conn/with-write-connection
(let [result (mongo.db/details-normalized database)]
(is (= "read-host" (:host result))))))))
@@ -43,8 +47,10 @@
(testing "Mongo's details-normalized uses effective-details for conn-uri databases"
(let [read-details {:conn-uri "mongodb://read-host:27017/test-db"}
write-details {:conn-uri "mongodb://write-host:27017/test-db"}
- database {:details read-details
- :write_data_details write-details}]
+ database {:lib/type :metadata/database
+ :id 1
+ :details read-details
+ :write-data-details write-details}]
(testing "default connection type uses :details"
(let [result (mongo.db/details-normalized database)]
(is (= "mongodb://read-host:27017/test-db" (:conn-uri result)))))
diff --git a/modules/drivers/mongo/test/metabase/driver/mongo/query_processor_test.clj b/modules/drivers/mongo/test/metabase/driver/mongo/query_processor_test.clj
index 174aa0437f5..de573213e13 100644
--- a/modules/drivers/mongo/test/metabase/driver/mongo/query_processor_test.clj
+++ b/modules/drivers/mongo/test/metabase/driver/mongo/query_processor_test.clj
@@ -796,7 +796,7 @@
(let [mp (mt/metadata-provider)
dogs (lib.metadata/table mp (mt/id :dogs))
person-id (lib.metadata/field mp (mt/id :dogs :person_id))]
- (if (-> (driver/dbms-version :mongo (mt/db)) :semantic-version (driver.u/semantic-version-gte [8]))
+ (if (-> (driver/dbms-version :mongo (lib.metadata/database (mt/metadata-provider))) :semantic-version (driver.u/semantic-version-gte [8]))
(do
(is (= [[1 #uuid "27e164bc-54f8-47a0-a85a-9f0e90dd7667" "Ivan" #uuid "d6b02fa2-bf7b-4b32-80d5-060b649c9859"]
[2 #uuid "3a0c0508-6b00-40ff-97f6-549666b2d16b" "Zach" #uuid "d6b02fa2-bf7b-4b32-80d5-060b649c9859"]]
diff --git a/modules/drivers/mongo/test/metabase/driver/mongo/sharded_cluster_test.clj b/modules/drivers/mongo/test/metabase/driver/mongo/sharded_cluster_test.clj
index 31f8e5c6e64..3a97643fa1f 100644
--- a/modules/drivers/mongo/test/metabase/driver/mongo/sharded_cluster_test.clj
+++ b/modules/drivers/mongo/test/metabase/driver/mongo/sharded_cluster_test.clj
@@ -2,6 +2,7 @@
(:require
[clojure.test :refer :all]
[metabase.driver :as driver]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.sync.core :as sync]
[metabase.test :as mt]))
@@ -11,6 +12,6 @@
(mt/test-driver
:mongo
(testing "Mongo driver can connect to a sharded cluster"
- (is (true? (driver/can-connect? :mongo (mt/db)))))
+ (is (true? (driver/can-connect? :mongo (lib.metadata/database (mt/metadata-provider))))))
(testing "Mongo can sync"
- (sync/sync-database! (mt/db)))))
+ (sync/sync-database! (mt/id)))))
diff --git a/modules/drivers/mongo/test/metabase/driver/mongo/util_test.clj b/modules/drivers/mongo/test/metabase/driver/mongo/util_test.clj
index 34b2985476f..d60cf175150 100644
--- a/modules/drivers/mongo/test/metabase/driver/mongo/util_test.clj
+++ b/modules/drivers/mongo/test/metabase/driver/mongo/util_test.clj
@@ -8,6 +8,6 @@
(deftest do-find-returns-lazy-seq-test
(mt/test-driver :mongo
(testing "do-find returns a lazy seq #42133"
- (mongo.connection/with-mongo-database [db (mt/db)]
+ (mongo.connection/with-mongo-database [db (mt/id)]
(is (= clojure.lang.LazySeq
(type (mongo.util/do-find (mongo.util/collection db "venues")))))))))
diff --git a/modules/drivers/mongo/test/metabase/driver/mongo_test.clj b/modules/drivers/mongo/test/metabase/driver/mongo_test.clj
index b868abdd57b..898bb4b5b12 100644
--- a/modules/drivers/mongo/test/metabase/driver/mongo_test.clj
+++ b/modules/drivers/mongo/test/metabase/driver/mongo_test.clj
@@ -92,7 +92,7 @@
(if (tdm/ssl-required?)
(mt/test-driver :mongo
(mt/dataset test-data (mt/db)
- (mongo.connection/with-mongo-client [c (mt/db)]
+ (mongo.connection/with-mongo-client [c (mt/id)]
(try
;; allow connecting with the client cert
;; https://www.mongodb.com/docs/manual/tutorial/configure-x509-client-authentication/#add-x-509-certificate-subject-as-a-user
@@ -270,7 +270,7 @@
{:schema nil, :name "people"}
{:schema nil, :name "products"}
{:schema nil, :name "reviews"}}
- (:tables (driver/describe-database :mongo (mt/db)))))))
+ (:tables (driver/describe-database :mongo (lib.metadata/database (mt/metadata-provider))))))))
(tx/defdataset nested-bindata-coll
(let [not-uuid (Binary. (byte 0) (byte-array 1))
@@ -321,7 +321,7 @@
:base-type :type/Integer
:pk? true
:database-position 0}}}
- (driver/describe-table :mongo (mt/db) (t2/select-one :model/Table :id (mt/id :venues)))))
+ (driver/describe-table :mongo (lib.metadata/database (mt/metadata-provider)) (t2/select-one :model/Table :id (mt/id :venues)))))
(mt/dataset uuid-dogs
(testing "binData uuid fields are identified as type/MongoBinData"
(is (= {:schema nil,
@@ -331,7 +331,7 @@
{:name "name", :database-type "string", :base-type :type/Text, :database-position 2}
{:name "person_id", :database-type "binData", :base-type :type/MongoBinData, :database-position 3}
{:name "id", :database-type "binData", :base-type :type/MongoBinData, :database-position 1}}}
- (driver/describe-table :mongo (mt/db) (t2/select-one :model/Table :id (mt/id :dogs)))))))
+ (driver/describe-table :mongo (lib.metadata/database (mt/metadata-provider)) (t2/select-one :model/Table :id (mt/id :dogs)))))))
(mt/dataset nested-bindata-coll
(testing "nested fields with mixed binData subtypes are identified as type/*"
(is (= {:schema nil, :name "nested-bindata",
@@ -350,7 +350,7 @@
:nested-fields #{{:name "nested_data_2", :database-type "binData", :base-type :type/MongoBinData, :database-position 5,
:nfc-path ["nested_mixed_not_uuid" "nested_data_2"]}}
:visibility-type :details-only}}}
- (driver/describe-table :mongo (mt/db) (t2/select-one :model/Table :id (mt/id :nested-bindata)))))))))
+ (driver/describe-table :mongo (lib.metadata/database (mt/metadata-provider)) (t2/select-one :model/Table :id (mt/id :nested-bindata)))))))))
(deftest describe-table-respects-sync-max-fields-per-table-test
(mt/test-driver :mongo
@@ -359,7 +359,7 @@
;; or deeply-nested collection can't OOM sync).
(mt/with-temporary-setting-values [sync-max-fields-per-table 2]
(is (= 2 (count (:fields (driver/describe-table
- :mongo (mt/db)
+ :mongo (lib.metadata/database (mt/metadata-provider))
(t2/select-one :model/Table :id (mt/id :venues)))))))))))
;; Index sync is turned off across the application as it is not used ATM.
@@ -382,15 +382,15 @@
(is (true? (t2/select-one-fn :database_indexed :model/Field (mt/id :singly-index :indexed))))
(is (false? (t2/select-one-fn :database_indexed :model/Field (mt/id :singly-index :not-indexed)))))
(testing "compound index"
- (mongo.connection/with-mongo-database [db (mt/db)]
+ (mongo.connection/with-mongo-database [db (mt/id)]
(mongo.util/create-index (mongo.util/collection db "compound-index") (array-map "first" 1 "second" 1)))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (true? (t2/select-one-fn :database_indexed :model/Field (mt/id :compound-index :first))))
(is (false? (t2/select-one-fn :database_indexed :model/Field (mt/id :compound-index :second)))))
(testing "multi key index"
- (mongo.connection/with-mongo-database [db (mt/db)]
+ (mongo.connection/with-mongo-database [db (mt/id)]
(mongo.util/create-index (mongo.util/collection db "multi-key-index") (array-map "url.small" 1)))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (false? (t2/select-one-fn :database_indexed :model/Field :name "url")))
(is (true? (t2/select-one-fn :database_indexed :model/Field :name "small"))))
(finally
@@ -410,9 +410,9 @@
[{:field-name "name" :indexed? false :base-type :type/Text}
{:field-name "class" :indexed? false :base-type :type/Text}]
[["Metabase" {"name" "Physics"}]]])
- (mongo.connection/with-mongo-database [db (mt/db)]
+ (mongo.connection/with-mongo-database [db (mt/id)]
(mongo.util/create-index (mongo.util/collection db "nested-indexed") (array-map "class.name" 1)))
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(testing "top level indexed, nested not"
(let [name-fields (t2/select [:model/Field :name :parent_id :database_indexed]
:table_id (mt/id :top-level-indexed) :name "name")]
@@ -461,11 +461,11 @@
{:field-name "text-field" :indexed? false :base-type :type/Text}
{:field-name "geospatial-field" :indexed? false :base-type :type/Text}]
[["Ngoc" "Khuat" [10 20]]]])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(try
(let [describe-indexes (fn [table-name]
- (driver/describe-table-indexes :mongo (mt/db) (t2/select-one :model/Table (mt/id table-name))))]
- (mongo.connection/with-mongo-database [db (mt/db)]
+ (driver/describe-table-indexes :mongo (lib.metadata/database (mt/metadata-provider)) (t2/select-one :model/Table (mt/id table-name))))]
+ (mongo.connection/with-mongo-database [db (mt/id)]
(testing "single column index"
(mongo.util/create-index (mongo.util/collection db "singly-index") {"a" 1})
(is (= #{{:type :normal-column-index :value "_id"}
@@ -551,7 +551,7 @@
(mt/test-driver :mongo
(mt/dataset all-null-columns
;; do a full sync on the DB to get the correct semantic type info
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= [{:name "_id", :database_type "long", :base_type :type/Integer, :semantic_type :type/PK}
{:name "favorite_snack", :database_type "null", :base_type :type/*, :semantic_type nil}
{:name "name", :database_type "string", :base_type :type/Text, :semantic_type :type/Name}]
@@ -577,7 +577,7 @@
["Common Blackbird" "earthworms" nil]
["Silvereye" "cherries" nil]]]]
;; do a full sync on the DB to get the correct semantic type info
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= #{{:name "_id", :database_type "long", :base_type :type/Integer, :semantic_type :type/PK}
{:name "favorite_snack", :database_type "string", :base_type :type/Text, :semantic_type :type/Category}
{:name "name", :database_type "string", :base_type :type/Text, :semantic_type :type/Name}
@@ -823,7 +823,7 @@
(count row-maps) (pr-str database-name) (pr-str collection-name))))
;; now sync the Database.
(let [db (first (t2/insert-returning-instances! :model/Database {:name database-name, :engine "mongo", :details details}))]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
db)))))
(defn- json-from-file [^String filename]
@@ -839,7 +839,7 @@
(deftest sync-missing-fields-test
(mt/test-driver :mongo
(mt/with-db (missing-fields-db)
- (sync/sync-database! (missing-fields-db))
+ (sync/sync-database! (:id (missing-fields-db)))
(testing "Test that fields with missing or null values get synced correctly"
(let [results (map #(into {} %)
(t2/select [:model/Field :id :name :database_type :base_type :semantic_type :parent_id]
@@ -912,18 +912,18 @@
"versionArray" [4 0 29 -100]})]
(is (= {:version "4.0.28-23"
:semantic-version [4 0 29]}
- (driver/dbms-version :mongo (mt/db))))))
+ (driver/dbms-version :mongo (lib.metadata/database (mt/metadata-provider)))))))
(testing "Any values after rubbish in versionArray are ignored"
(mt/with-dynamic-fn-redefs [mongo.util/run-command (constantly {"version" "4.0.28-23"
"versionArray" [4 0 "NaN" 29]})]
(is (= {:version "4.0.28-23"
:semantic-version [4 0]}
- (driver/dbms-version :mongo (mt/db))))))))
+ (driver/dbms-version :mongo (lib.metadata/database (mt/metadata-provider)))))))))
(deftest object-columns-export-as-json
(mt/test-driver :mongo
(mt/with-db (missing-fields-db)
- (sync/sync-database! (missing-fields-db))
+ (sync/sync-database! (:id (missing-fields-db)))
(testing "Objects are formatted correctly as JSON in downloads"
(mt/with-temp [:model/Card card {:display :table
:dataset_query {:database (mt/id)
@@ -1084,7 +1084,7 @@
(binding [mongo.execute/*aggregate* (fn [db _coll session stages timeout-ms]
(mongo.execute/aggregate-database db session stages timeout-ms))
mongo/*sample-stages* (fn [& _#] [{"$documents" documents}])]
- (let [dbfields (delay (@#'mongo/fetch-dbfields (mt/db) (t2/select-one-fn :name :model/Table :id (mt/id :venues))))
+ (let [dbfields (delay (@#'mongo/fetch-dbfields (lib.metadata/database (mt/metadata-provider)) (t2/select-one-fn :name :model/Table :id (mt/id :venues))))
ftree (delay (@#'mongo/dbfields->ftree @dbfields))
nested-fields (delay (@#'mongo/ftree->nested-fields @ftree))]
(thunk dbfields ftree nested-fields)))))
@@ -1406,8 +1406,8 @@
{:use-conn-uri true :conn-uri "mongodb+srv://user:[email protected]/db"}
["cluster.example.com"]))
(testing "a details map naming no host of its own contributes no hosts, as it does for every other driver"
- ;; `write_data_details` and `admin_details` are overlays merged on top of `:details`, so they reach us as partial
- ;; maps; one that only carries credentials repoints nothing, and the map it overlays is validated on its own
+ ;; `write_data_details` is an overlay merged on top of `:details`, so it reaches us as a partial
+ ;; map; one that only carries credentials repoints nothing, and the map it overlays is validated on its own
(are [details] (= [] (driver/connection-hosts :mongo details))
{}
{:user "admin" :pass "x"}
diff --git a/modules/drivers/mongo/test/metabase/test/data/mongo.clj b/modules/drivers/mongo/test/metabase/test/data/mongo.clj
index 87f72a87c33..0f64100ce74 100644
--- a/modules/drivers/mongo/test/metabase/test/data/mongo.clj
+++ b/modules/drivers/mongo/test/metabase/test/data/mongo.clj
@@ -9,6 +9,7 @@
[metabase.driver.mongo.connection :as mongo.connection]
[metabase.driver.mongo.util :as mongo.util]
[metabase.test.data.interface :as tx]
+ [metabase.util :as u]
[metabase.util.json :as json])
(:import
(com.mongodb.client MongoDatabase)))
@@ -140,12 +141,12 @@
[_driver database view-name table-name {:keys [materialized?]}]
(when materialized?
(throw (Exception. "Material Views not supported.")))
- (mongo.connection/with-mongo-database [^MongoDatabase db database]
+ (mongo.connection/with-mongo-database [^MongoDatabase db (u/the-id database)]
(.createView db view-name table-name [])))
(defmethod tx/drop-view! :mongo
[_driver database view-name {:keys [materialized?]}]
(when materialized?
(throw (Exception. "Material Views not supported.")))
- (mongo.connection/with-mongo-database [^MongoDatabase db database]
+ (mongo.connection/with-mongo-database [^MongoDatabase db (u/the-id database)]
(some-> db (mongo.util/collection view-name) .drop)))
diff --git a/modules/drivers/oracle/test/metabase/driver/oracle_test.clj b/modules/drivers/oracle/test/metabase/driver/oracle_test.clj
index 03f01ab8665..0a2538c9116 100644
--- a/modules/drivers/oracle/test/metabase/driver/oracle_test.clj
+++ b/modules/drivers/oracle/test/metabase/driver/oracle_test.clj
@@ -338,7 +338,6 @@
{:name "advanced-options"}
{:name "destination-database"}
{:name "write-data-connection"}
- {:name "admin-connection"}
{:name "auto_run_queries"}
{:name "let-user-control-scheduling"}
{:name "schedules.metadata_sync"}
@@ -381,7 +380,7 @@
(deftest timezone-id-test
(mt/test-driver :oracle
(is (= (t/zone-id "Z")
- (-> (driver/db-default-timezone :oracle (mt/db))
+ (-> (driver/db-default-timezone :oracle (lib.metadata/database (mt/metadata-provider)))
t/zone-id
.normalized)))))
@@ -483,7 +482,7 @@
(execute! "INSERT INTO \"%s\".\"messages\" (\"id\", \"message\") VALUES (1, 'Hello')" username)
(execute! "INSERT INTO \"%s\".\"messages\" (\"id\", \"message\") VALUES (2, NULL)" username)
(binding [oracle.tx/*override-describe-database-to-filter-by-db-name?* false]
- (sync/sync-database! (mt/db) {:scan :schema}))
+ (sync/sync-database! (mt/id) {:scan :schema}))
(let [table (t2/select-one :model/Table :schema username, :name "messages", :db_id (mt/id))
id-field (t2/select-one :model/Field :table_id (u/the-id table), :name "id")]
(testing "The CLOB is synced as a text field"
@@ -610,7 +609,7 @@
:details details}]
(mt/with-db database
(testing " can sync correctly"
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
;; should be four tables from test-data
(is (= 8 (t2/count :model/Table :db_id (u/the-id database) :name [:like "test_data%"])))
(binding [api/*current-user-id* orig-user-id ; restore original user-id to avoid perm errors
@@ -684,7 +683,7 @@
sql (format "SELECT CAST(TIMESTAMP '%s' AS TIMESTAMP WITH LOCAL TIME ZONE) AS timestamp_tz FROM dual" test-date)
query (-> (mt/native-query {:query sql})
(assoc-in [:middleware :format-rows?] false))
- spec #(sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ spec #(sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
old-format (-> (jdbc/query (spec) "SELECT value FROM NLS_SESSION_PARAMETERS WHERE PARAMETER = 'NLS_TIMESTAMP_TZ_FORMAT'")
ffirst
val)
@@ -854,7 +853,7 @@
(deftest table-privileges-test
(mt/test-driver :oracle
(testing "`current-user-table-privileges` returns correct structure and privileges"
- (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
privileges (sql-jdbc.sync/current-user-table-privileges :oracle conn-spec)]
(is (seq privileges) "Should return at least one table")
(doseq [priv privileges]
diff --git a/modules/drivers/oracle/test/metabase/test/data/oracle.clj b/modules/drivers/oracle/test/metabase/test/data/oracle.clj
index 6b8fda61e5b..7e4745f4a21 100644
--- a/modules/drivers/oracle/test/metabase/test/data/oracle.clj
+++ b/modules/drivers/oracle/test/metabase/test/data/oracle.clj
@@ -10,6 +10,8 @@
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
[metabase.driver.sql.query-processor :as sql.qp]
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.test :as mt]
[metabase.test.data.impl :as data.impl]
[metabase.test.data.interface :as tx]
@@ -354,7 +356,7 @@
"airports_region"}
(into #{}
(map :name)
- (:tables (driver/describe-database :oracle (mt/db))))))))))
+ (:tables (driver/describe-database :oracle (lib.metadata/database (mt/metadata-provider)))))))))))
(defmethod tx/drop-view! :oracle
[driver database view-name {:keys [materialized?]}]
@@ -362,7 +364,7 @@
qualified-view (sql.tx/qualify-and-quote driver database-name (name view-name))]
(u/ignore-exceptions
;; If exists does not exist in oracle
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec database)
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
(sql/format
{(if materialized? :drop-materialized-view :drop-view) [[[:raw qualified-view]]]}
:dialect (sql.qp/quote-style driver))
diff --git a/modules/drivers/presto-jdbc/test/metabase/driver/presto_jdbc_test.clj b/modules/drivers/presto-jdbc/test/metabase/driver/presto_jdbc_test.clj
index 5fa0346fa7f..ba9fef0d041 100644
--- a/modules/drivers/presto-jdbc/test/metabase/driver/presto_jdbc_test.clj
+++ b/modules/drivers/presto-jdbc/test/metabase/driver/presto_jdbc_test.clj
@@ -12,6 +12,7 @@
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.lib.core :as lib]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.query-processor.compile :as qp.compile]
[metabase.query-processor.test :as qp]
[metabase.sync.core :as sync]
@@ -38,7 +39,7 @@
{:name "test_data_checkins" :schema "default"}
{:name "test_data_users" :schema "default"}
{:name "test_data_venues" :schema "default"}}}
- (-> (driver/describe-database :presto-jdbc (mt/db))
+ (-> (driver/describe-database :presto-jdbc (lib.metadata/database (mt/metadata-provider)))
(update :tables #(into #{} (filter (comp #{"test_data_categories"
"test_data_venues"
"test_data_checkins"
@@ -75,7 +76,7 @@
:database-type "integer"
:base-type :type/Integer
:database-position 0}}}
- (driver/describe-table :presto-jdbc (mt/db) (t2/select-one 'Table :id (mt/id :venues)))))))
+ (driver/describe-table :presto-jdbc (lib.metadata/database (mt/metadata-provider)) (t2/select-one 'Table :id (mt/id :venues)))))))
(deftest ^:parallel table-rows-sample-test
(mt/test-driver :presto-jdbc
@@ -126,7 +127,7 @@
(deftest ^:parallel db-default-timezone-test
(mt/test-driver :presto-jdbc
(is (= "UTC"
- (driver/db-default-timezone :presto-jdbc (mt/db))))))
+ (driver/db-default-timezone :presto-jdbc (lib.metadata/database (mt/metadata-provider)))))))
(deftest ^:synchronized template-tag-timezone-test
(mt/test-driver :presto-jdbc
@@ -224,7 +225,7 @@
(mt/with-temp [:model/Database db {:engine :presto-jdbc, :name "Temp Presto JDBC Schema DB", :details with-schema}]
(mt/with-db db
;; same as test_data, but with schema, so should NOT pick up venues, users, etc.
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (= [{:name t, :schema s, :db_id (mt/id)}]
(map #(select-keys % [:name :schema :db_id]) (t2/select :model/Table :db_id (mt/id)))))))
(execute-ddl! [(format "DROP TABLE %s.%s" s t)
diff --git a/modules/drivers/redshift/src/metabase/driver/redshift.clj b/modules/drivers/redshift/src/metabase/driver/redshift.clj
index c6cdf779d83..1152406cf31 100644
--- a/modules/drivers/redshift/src/metabase/driver/redshift.clj
+++ b/modules/drivers/redshift/src/metabase/driver/redshift.clj
@@ -20,6 +20,7 @@
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.driver.sql.util :as sql.u]
[metabase.driver.sync :as driver.s]
+ [metabase.driver.util :as driver.u]
[metabase.util :as u]
[metabase.util.date-2 :as u.date]
[metabase.util.honey-sql-2 :as h2x]
@@ -447,7 +448,7 @@
;; Same as the inherited `:sql-jdbc` impl, but inlines a sortkey when one is present. Non-transform callers
;; (e.g. uploads) pass no `:indexes`, so the rendered SQL is unchanged for them.
(let [sql (create-table-sql driver table-name column-definitions opts)]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(jdbc/execute! conn sql))))
;;; +----------------------------------------------------------------------------------------------------------------+
@@ -561,10 +562,6 @@
db-or-id-or-spec
options
(fn [^Connection conn]
- (let [db (cond (integer? db-or-id-or-spec) (driver-api/with-metadata-provider db-or-id-or-spec
- (driver-api/database (driver-api/metadata-provider)))
- (u/id db-or-id-or-spec) db-or-id-or-spec)]
- (sql-jdbc.execute/set-role-if-supported! driver conn db))
(when-not (sql-jdbc.execute/recursive-connection?)
(sql-jdbc.execute/set-best-transaction-level! driver conn)
(sql-jdbc.execute/set-time-zone-if-supported! driver conn session-timezone)
diff --git a/modules/drivers/redshift/test/metabase/driver/redshift/index_test.clj b/modules/drivers/redshift/test/metabase/driver/redshift/index_test.clj
index 40fc8ee538f..085359ec1b5 100644
--- a/modules/drivers/redshift/test/metabase/driver/redshift/index_test.clj
+++ b/modules/drivers/redshift/test/metabase/driver/redshift/index_test.clj
@@ -12,6 +12,7 @@
[metabase.driver :as driver]
[metabase.driver.redshift :as redshift]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.driver.util :as driver.u]
[metabase.test :as mt]
[metabase.test.data.interface :as tx]
[metabase.test.data.redshift :as redshift.tx]
@@ -141,7 +142,7 @@
(let [db-details (tx/dbdef->connection-details :redshift nil nil)]
(mt/with-temp [:model/Database database {:engine :redshift, :details db-details}]
(let [schema (redshift.tx/unique-session-schema)
- spec (sql-jdbc.conn/db->pooled-connection-spec database)]
+ spec (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))]
(doseq [{:keys [label indexes expected]} live-cases]
(testing label
;; unique per-case table names so the two seams (and parallel runs) don't collide in the shared schema
diff --git a/modules/drivers/redshift/test/metabase/driver/redshift_test.clj b/modules/drivers/redshift/test/metabase/driver/redshift_test.clj
index 2e4b2fdd5a2..28d479290f1 100644
--- a/modules/drivers/redshift/test/metabase/driver/redshift_test.clj
+++ b/modules/drivers/redshift/test/metabase/driver/redshift_test.clj
@@ -14,6 +14,7 @@
[metabase.driver.sql-jdbc.sync.interface :as sql-jdbc.sync]
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.driver.sync :as driver.s]
+ [metabase.driver.util :as driver.u]
[metabase.lib.core :as lib]
[metabase.lib.metadata :as lib.metadata]
[metabase.lib.test-util :as lib.tu]
@@ -43,13 +44,13 @@
(mt/test-driver :redshift
(testing "default"
(is (= "public"
- (driver.sql/default-schema :redshift (mt/db)))))
+ (driver.sql/default-schema :redshift (lib.metadata/database (mt/metadata-provider))))))
(testing "schema configured in the JDBC additional options"
(let [schema (redshift.tx/unique-session-schema)
details (assoc (:details (mt/db)) :additional-options (str "currentSchema=" schema))]
(mt/with-temp [:model/Database database {:engine :redshift, :details details}]
(is (= schema
- (driver.sql/default-schema :redshift database))))))))
+ (driver.sql/default-schema :redshift (driver.u/ensure-lib-database database)))))))))
(use-fixtures :once (fixtures/initialize :plugins))
(use-fixtures :once (fixtures/initialize :db))
@@ -248,7 +249,7 @@
;; sync the schema again to pick up the new view (and table, though we aren't checking that)
(binding [sync-util/*log-exceptions-and-continue?* false
redshift.tx/*override-describe-database-to-filter-by-db-name?* false]
- (sync/sync-database! database {:scan :schema}))
+ (sync/sync-database! (:id database) {:scan :schema}))
(testing "the new view should have been synced"
(is (contains?
(t2/select-fn-set :name :model/Table :db_id (u/the-id database))
@@ -279,7 +280,7 @@
qual-view-nm)
(binding [sync-util/*log-exceptions-and-continue?* false
redshift.tx/*override-describe-database-to-filter-by-db-name?* false]
- (sync/sync-database! database {:scan :schema}))
+ (sync/sync-database! (:id database) {:scan :schema}))
(testing "the new view should have been synced without errors"
(is (contains?
(t2/select-fn-set :name :model/Table :db_id (u/the-id database))
@@ -306,7 +307,7 @@
(mt/with-temp [:model/Database db {:engine :redshift, :details details}]
(let [table-is-in-results? (fn []
(binding [redshift.tx/*override-describe-database-to-filter-by-db-name?* false]
- (->> (:tables (driver/describe-database :redshift db))
+ (->> (:tables (driver/describe-database :redshift (driver.u/ensure-lib-database db)))
(map :name)
(some #{table-name})
boolean)))
@@ -346,13 +347,13 @@
:model/Database db {:engine :redshift, :details details}]
(binding [redshift.tx/*override-describe-database-to-filter-by-db-name?* false]
(persist-models!)
- (let [synced-schemas (into #{} (map :schema) (:tables (driver/describe-database :redshift db)))]
+ (let [synced-schemas (into #{} (map :schema) (:tables (driver/describe-database :redshift (driver.u/ensure-lib-database db))))]
(testing "sense check: there are results matching some schemas in the schema-filters-patterns"
;; the schema this run just loaded `avian-singles` into, rather than anything shaped like a
;; session schema: it ties the assertion to the name [[unique-prefix]] actually produces, so
;; renaming those schemas fails here instead of silently emptying the result
(is (contains? synced-schemas (redshift.tx/unique-session-schema))))
- (let [all-schemas (map :table_schema (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (let [all-schemas (map :table_schema (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
"select distinct table_schema from information_schema.tables;"))]
(testing "metabase_cache_ tables are excluded from results"
(let [match? #(re-matches #"metabase_cache(.*)" %)]
@@ -379,7 +380,7 @@
qual-mview-nm)
(binding [redshift.tx/*override-describe-database-to-filter-by-db-name?* false]
(u/auto-retry 3
- (let [table-names (into #{} (map :name) (:tables (driver/describe-database :redshift database)))]
+ (let [table-names (into #{} (map :name) (:tables (driver/describe-database :redshift (driver.u/ensure-lib-database database))))]
(when-not (contains? table-names mview-nm)
(Thread/sleep 1000)
(throw (ex-info "Materialized view not yet visible in describe-database results"
@@ -442,7 +443,7 @@
qual-view-name (format "\"%s\".\"%s\"" schema-name view-nm)
mview-name (tx/db-qualified-table-name db-name "mview")
qual-mview-name (format "\"%s\".\"%s\"" schema-name mview-name)
- conn-spec (sql-jdbc.conn/db->pooled-connection-spec database)
+ conn-spec (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
get-privileges (fn []
(sql-jdbc.conn/with-connection-spec-for-testing-connection
[spec [:redshift (assoc (:details database) :user username)]
@@ -849,4 +850,4 @@
["stringly" "false"]]]
{:name nm, :schema "s", :type "table", :description nil, :selectable selectable}))]
(is (= [{:name "readable", :schema "s", :description nil}]
- (into [] (#'redshift/describe-database-tables db))))))))
+ (into [] (#'redshift/describe-database-tables (driver.u/lib-database (:id db))))))))))
diff --git a/modules/drivers/redshift/test/metabase/test/data/redshift.clj b/modules/drivers/redshift/test/metabase/test/data/redshift.clj
index 61bbad874af..799041bffe6 100644
--- a/modules/drivers/redshift/test/metabase/test/data/redshift.clj
+++ b/modules/drivers/redshift/test/metabase/test/data/redshift.clj
@@ -21,6 +21,7 @@
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
[metabase.driver.sql.test-util.unique-prefix :as sql.tu.unique-prefix]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.test :as mt]
[metabase.test.data.impl :as data.impl]
[metabase.test.data.interface :as tx]
@@ -426,7 +427,7 @@
"airports_region"}
(into #{}
(map :name)
- (:tables (driver/describe-database :redshift (mt/db))))))))))
+ (:tables (driver/describe-database :redshift (lib.metadata/database (mt/metadata-provider)))))))))))
(defmethod ddl.i/format-name :redshift
[_driver s]
diff --git a/modules/drivers/snowflake/src/metabase/driver/snowflake.clj b/modules/drivers/snowflake/src/metabase/driver/snowflake.clj
index ccf6f3fe8ad..15b929f446c 100644
--- a/modules/drivers/snowflake/src/metabase/driver/snowflake.clj
+++ b/modules/drivers/snowflake/src/metabase/driver/snowflake.clj
@@ -27,6 +27,7 @@
[metabase.driver.sql.query-processor.util :as sql.qp.u]
[metabase.driver.sql.util :as sql.u]
[metabase.driver.sync :as driver.s]
+ [metabase.driver.util :as driver.u]
[metabase.lib.schema.common :as lib.schema.common]
[metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.util :as u]
@@ -1192,7 +1193,7 @@
(nil? (:private-key-value details)))
(assoc :use-password true)
- ;; Skip when nothing implies a mode — overlay maps (:write_data_details / :admin_details)
+ ;; Skip when nothing implies a mode — an overlay map (:write_data_details)
;; that only tweak account/warehouse must not gain an :auth-mode, which would override
;; :details' :auth-mode during the merge in effective-details.
(and (not (contains? details :auth-mode))
@@ -1207,8 +1208,7 @@
[_ database]
(-> database
(m/update-existing :details normalize-details)
- (m/update-existing :write_data_details normalize-details)
- (m/update-existing :admin_details normalize-details)))
+ (m/update-existing :write_data_details normalize-details)))
;;; If you try to read a Snowflake `timestamptz` as a String with `.getString` it always comes back in
;;; `America/Los_Angeles` for some reason I cannot figure out. Let's just read them out as UTC, which is what they're
@@ -1298,7 +1298,7 @@
:rename-table (keyword to-table)}
:quoted true
:dialect (sql.qp/quote-style driver)))]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(jdbc/execute! conn sql))))
;;; +----------------------------------------------------------------------------------------------------------------+
diff --git a/modules/drivers/snowflake/test/metabase/driver/snowflake_test.clj b/modules/drivers/snowflake/test/metabase/driver/snowflake_test.clj
index 0d8c4e63449..a37496c71cb 100644
--- a/modules/drivers/snowflake/test/metabase/driver/snowflake_test.clj
+++ b/modules/drivers/snowflake/test/metabase/driver/snowflake_test.clj
@@ -71,12 +71,12 @@
(mt/test-driver :snowflake
(testing "details naming no schema fall back to PUBLIC, which is how Metabase has always read them"
(is (= "PUBLIC"
- (driver.sql/default-schema :snowflake (mt/db)))))
+ (driver.sql/default-schema :snowflake (lib.metadata/database (mt/metadata-provider))))))
(testing "schema configured in the JDBC additional options"
(let [details (assoc (:details (mt/db)) :additional-options "schema=INFORMATION_SCHEMA")]
(mt/with-temp [:model/Database database {:engine :snowflake, :details details}]
(is (= "INFORMATION_SCHEMA"
- (driver.sql/default-schema :snowflake database))))))))
+ (driver.sql/default-schema :snowflake (driver.u/ensure-lib-database database)))))))))
(deftest ^:parallel connection-hosts-test
(are [details expected] (= expected (driver/connection-hosts :snowflake details))
@@ -166,7 +166,7 @@
:json-unfolding false
:database-position 5}]
(sort-by :database-position
- (into [] (fetch-metadata/fields-metadata (mt/db) {:table-names ["venues"]})))))))
+ (into [] (fetch-metadata/fields-metadata (lib.metadata/database (mt/metadata-provider)) {:table-names ["venues"]})))))))
(deftest ^:parallel quote-name-test
(is (nil? (#'driver.snowflake/quote-name nil)))
@@ -522,7 +522,7 @@
(let [pool-cache @#'sql-jdbc.conn/pool-cache-key->connection-pool
cache-key [(:id db) :default]
query! (fn []
- (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec db)
+ (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database db))
["SELECT CURRENT_USER() AS \"user\""]))]
(testing "initial query works with token-a"
(is (= [{:user wif-user}] (query!))))
@@ -645,7 +645,7 @@
(qp.store/with-metadata-provider (mt/id)
(sql-jdbc.execute/do-with-connection-with-options
:snowflake
- (mt/db)
+ (mt/id)
nil
(fn [^java.sql.Connection conn]
(is (sql-jdbc.sync/have-select-privilege? :snowflake conn "PUBLIC" "venues")))))))
@@ -707,16 +707,16 @@
{:name "airport", :schema "PUBLIC", :description nil}}]
(testing "should work with normal details"
(is (= expected-tables
- (into #{} (:tables (driver/describe-database :snowflake (mt/db)))))))
+ (into #{} (:tables (driver/describe-database :snowflake (lib.metadata/database (mt/metadata-provider))))))))
(testing "should accept either `:db` or `:dbname` in the details, working around a bug with the original impl"
(is (= expected-tables
- (into #{} (:tables (driver/describe-database :snowflake (update (mt/db) :details set/rename-keys {:db :dbname})))))))
+ (into #{} (:tables (driver/describe-database :snowflake (driver.u/ensure-lib-database (update (mt/db) :details set/rename-keys {:db :dbname}))))))))
(testing "should throw an Exception if details have neither `:db` nor `:dbname`"
(is (thrown? Exception
- (driver/describe-database :snowflake (update (mt/db) :details set/rename-keys {:db :xyz})))))
+ (driver/describe-database :snowflake (driver.u/ensure-lib-database (update (mt/db) :details set/rename-keys {:db :xyz}))))))
(testing "should use the NAME FROM DETAILS instead of the DB DISPLAY NAME to fetch metadata (#8864)"
(is (= expected-tables
- (:tables (driver/describe-database :snowflake (assoc (mt/db) :name "ABC"))))))))))))
+ (:tables (driver/describe-database :snowflake (driver.u/ensure-lib-database (assoc (mt/db) :name "ABC")))))))))))))
(deftest describe-database-default-schema-test
(testing "describe-database should include Tables from all schemas even if the DB has a default schema (#38135)"
@@ -748,7 +748,7 @@
;; fetch metadata
(mt/with-temp [:model/Database database {:engine :snowflake, :details details}]
(is (=? {:tables #{{:name table-name, :schema schema-name, :description nil}}}
- (driver/describe-database :snowflake database))))))))
+ (driver/describe-database :snowflake (driver.u/ensure-lib-database database)))))))))
(deftest describe-database-views-test
(mt/test-driver :snowflake
@@ -762,7 +762,7 @@
(jdbc/execute! spec [stmt] {:transaction? false}))
;; create the DB object
(mt/with-temp [:model/Database database {:engine :snowflake, :details details}]
- (let [sync! #(sync/sync-database! database)]
+ (let [sync! #(sync/sync-database! (:id database))]
;; create a view
(doseq [statement [(format "CREATE VIEW \"%s\".\"PUBLIC\".\"example_view\" AS SELECT 'hello world' AS \"name\";" db-name)
(format "GRANT SELECT ON \"%s\".\"PUBLIC\".\"example_view\" TO PUBLIC;" db-name)]]
@@ -789,7 +789,7 @@
[(format "CREATE OR REPLACE DYNAMIC TABLE \"%s\".\"PUBLIC\".\"metabase_fan\" target_lag = DOWNSTREAM warehouse = 'COMPUTE_WH' AS
SELECT * FROM \"%s\".\"PUBLIC\".\"metabase_users\" WHERE \"%s\".\"PUBLIC\".\"metabase_users\".\"name\" LIKE 'MB_%%';"
db-name db-name db-name)])
- (sync/sync-database! (t2/select-one :model/Database (mt/id)) {:scan :schema})
+ (sync/sync-database! (:id (t2/select-one :model/Database (mt/id))) {:scan :schema})
(thunk))))
(defmacro with-dynamic-table!
@@ -816,7 +816,7 @@
(with-dynamic-table!
(sql-jdbc.execute/do-with-connection-with-options
:snowflake
- (mt/db)
+ (mt/id)
nil
(fn [conn]
(let [dynamic-table (or (t2/select-one :model/Table :name "metabase_fan" :db_id (mt/id))
@@ -870,11 +870,11 @@
(deftest ^:synchronized describe-table-fields-uuid-column-test
(mt/test-driver :snowflake
(testing "Snowflake tables with UUID columns should sync successfully (#71595)"
- (let [db-name (#'driver.snowflake/db-name (mt/db))
+ (let [db-name (#'driver.snowflake/db-name (lib.metadata/database (mt/metadata-provider)))
table-name (str "uuid_test_" (u.random/random-name))]
(sql-jdbc.execute/do-with-connection-with-options
:snowflake
- (mt/db)
+ (mt/id)
nil
(fn [^java.sql.Connection conn]
(try
@@ -914,12 +914,12 @@
the new base_type and clear stale coercion_strategy/semantic_type. Reproduces the
in-place update path (repro v1 from the Linear issue) where Metabase's Field row is
preserved across the schema change."
- (let [db-name (#'driver.snowflake/db-name (mt/db))
+ (let [db-name (#'driver.snowflake/db-name (lib.metadata/database (mt/metadata-provider)))
table-name (str "ghy_3388_" (u.random/random-name))
run-sql! (fn [stmts]
(sql-jdbc.execute/do-with-connection-with-options
:snowflake
- (mt/db)
+ (mt/id)
nil
(fn [^java.sql.Connection conn]
(doseq [stmt stmts]
@@ -931,7 +931,7 @@
(run-sql! [(format "CREATE OR REPLACE TRANSIENT TABLE %s (\"id\" INTEGER, \"text_column\" TEXT);" qualified)
(format "INSERT INTO %s (\"id\", \"text_column\") VALUES (1, '100'), (2, '200'), (3, '300');" qualified)])
;; step 2: sync — Metabase creates Table + Field rows for our new table
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [original-table (t2/select-one :model/Table :db_id (mt/id) :name table-name)
_ (is (some? original-table) "table should be synced")
original-field (t2/select-one :model/Field :table_id (:id original-table) :name "text_column")]
@@ -944,7 +944,7 @@
"SELECT \"id\", TRY_TO_NUMBER(\"text_column\") AS \"text_column\" FROM %s;")
qualified qualified)])
;; step 4: sync again
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [new-field (t2/select-one :model/Field :id (:id original-field))]
(testing "after sync, base_type and effective_type both reflect the new numeric column,
and stale coercion/semantic type are cleared"
@@ -964,11 +964,11 @@
(t2/delete! :model/Table :id (:id original-table)))
(run-sql! [(format "CREATE OR REPLACE TRANSIENT TABLE %s (\"id\" INTEGER, \"text_column\" TEXT);" qualified)
(format "INSERT INTO %s (\"id\", \"text_column\") VALUES (1, '100'), (2, '200'), (3, '300');" qualified)])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(run-sql! [(format (str "CREATE OR REPLACE TABLE %s AS "
"SELECT \"id\", TRY_TO_NUMBER(\"text_column\", 38, 2) AS \"text_column\" FROM %s;")
qualified qualified)])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [fresh-table (t2/select-one :model/Table :db_id (mt/id) :name table-name)
fresh-field (t2/select-one :model/Field :table_id (:id fresh-table) :name "text_column")]
(testing "after fresh-state CREATE OR REPLACE to numeric, base_type and effective_type
@@ -989,7 +989,7 @@
(t2/delete! :model/Table :id (:id fresh-table)))
(run-sql! [(format "CREATE OR REPLACE TRANSIENT TABLE %s (\"id\" INTEGER, \"text_column\" TEXT);" qualified)
(format "INSERT INTO %s (\"id\", \"text_column\") VALUES (1, '100'), (2, '200'), (3, '300');" qualified)])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [pre-coerce-table (t2/select-one :model/Table :db_id (mt/id) :name table-name)
pre-coerce-field (t2/select-one :model/Field :table_id (:id pre-coerce-table) :name "text_column")]
;; user enables coercion: this TEXT column is really an integer
@@ -999,7 +999,7 @@
(run-sql! [(format (str "CREATE OR REPLACE TABLE %s AS "
"SELECT \"id\", TRY_TO_NUMBER(\"text_column\") AS \"text_column\" FROM %s;")
qualified qualified)])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [after-field (t2/select-one :model/Field :id (:id pre-coerce-field))]
(testing "after the underlying TEXT column becomes a native number, the previously
user-set coercion strategy should be cleared and effective_type should
@@ -1021,7 +1021,7 @@
(t2/delete! :model/Table :id (:id stale-table)))
(run-sql! [(format "CREATE OR REPLACE TRANSIENT TABLE %s (\"id\" INTEGER, \"text_column\" TEXT);" qualified)
(format "INSERT INTO %s (\"id\", \"text_column\") VALUES (1, '100'), (2, '200'), (3, '300');" qualified)])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [v4-table (t2/select-one :model/Table :db_id (mt/id) :name table-name)
v4-field (t2/select-one :model/Field :table_id (:id v4-table) :name "text_column")]
(field-user-settings/upsert-user-settings v4-field
@@ -1033,7 +1033,7 @@
(run-sql! [(format (str "CREATE OR REPLACE TABLE %s AS "
"SELECT \"id\", TRY_TO_NUMBER(\"text_column\") AS \"text_column\" FROM %s;")
qualified qualified)])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [after-field (t2/select-one :model/Field :id (:id v4-field))
after-user-settings (t2/select-one :model/FieldUserSettings :field_id (:id v4-field))]
(testing "after CREATE OR REPLACE to numeric, the user-set coercion stored in
@@ -1060,7 +1060,7 @@
(t2/delete! :model/Table :id (:id stale-table)))
(run-sql! [(format "CREATE OR REPLACE TRANSIENT TABLE %s (\"id\" INTEGER, \"text_column\" TEXT);" qualified)
(format "INSERT INTO %s (\"id\", \"text_column\") VALUES (1, '100'), (2, '200'), (3, '300');" qualified)])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [v5-table (t2/select-one :model/Table :db_id (mt/id) :name table-name)
v5-field (t2/select-one :model/Field :table_id (:id v5-table) :name "text_column")]
(run-sql! [(format (str "CREATE OR REPLACE TABLE %s AS "
@@ -1105,7 +1105,7 @@
:database-is-nullable false
:database-required true
:json-unfolding false}]}
- (-> (driver/describe-table :snowflake (assoc (mt/db) :name "ABC") (t2/select-one :model/Table :id (mt/id :categories)))
+ (-> (driver/describe-table :snowflake (driver.u/ensure-lib-database (assoc (mt/db) :name "ABC")) (t2/select-one :model/Table :id (mt/id :categories)))
(update :fields (partial sort-by :name))))))))
(deftest ^:synchronized describe-fks-test
@@ -1121,8 +1121,8 @@
(into []
(driver/describe-fks
:snowflake
- (-> (lib.metadata/database (mt/metadata-provider))
- (assoc :name "ABC"))
+ (driver.u/ensure-lib-database (-> (lib.metadata/database (mt/metadata-provider))
+ (assoc :name "ABC")))
{:schema-names [(:schema table)]
:table-names [(:name table)]}))))))))
@@ -1795,7 +1795,7 @@
(sort-by last
(into []
(map (juxt :name :database-type :base-type :database-position))
- (fetch-metadata/fields-metadata (mt/db)))))))))
+ (fetch-metadata/fields-metadata (lib.metadata/database (mt/metadata-provider))))))))))
;; The test needs user with no report timezone set and database timezone other than UTC. That's the reason for redefs
;; prior to dataset generation.
@@ -1919,7 +1919,7 @@
(binding [sql-jdbc.describe-database/*select-probe-query-timeout-seconds* 1]
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (mt/id)
nil
(fn [^java.sql.Connection conn]
(is (true? (sql-jdbc.sync.interface/have-select-privilege?
@@ -2173,7 +2173,7 @@
{:name "region", :schema "PUBLIC", :description nil}
{:name "country", :schema "PUBLIC", :description nil}
{:name "airport", :schema "PUBLIC", :description nil}}
- (:tables (driver/describe-database :snowflake db)))))))))))
+ (:tables (driver/describe-database :snowflake (driver.u/ensure-lib-database db))))))))))))
;;; ------------------------------------------------ Fake Sync Tests ------------------------------------------------
;; Tests to validate that fake sync produces correct metadata for Snowflake.
diff --git a/modules/drivers/sqlserver/src/metabase/driver/sqlserver.clj b/modules/drivers/sqlserver/src/metabase/driver/sqlserver.clj
index 77aa7b349d3..64f41f9dae8 100644
--- a/modules/drivers/sqlserver/src/metabase/driver/sqlserver.clj
+++ b/modules/drivers/sqlserver/src/metabase/driver/sqlserver.clj
@@ -1329,7 +1329,7 @@
(defmethod driver/rename-table! :sqlserver
[_driver db-id old-table-name new-table-name]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(with-open [stmt (.createStatement ^java.sql.Connection (:connection conn))]
(let [sql (format "EXEC sp_rename %s, %s;"
(sql.u/quote-literal (name old-table-name) :ansi)
diff --git a/modules/drivers/sqlserver/test/metabase/driver/sqlserver_test.clj b/modules/drivers/sqlserver/test/metabase/driver/sqlserver_test.clj
index 8406b0f8d01..434556130a5 100644
--- a/modules/drivers/sqlserver/test/metabase/driver/sqlserver_test.clj
+++ b/modules/drivers/sqlserver/test/metabase/driver/sqlserver_test.clj
@@ -44,7 +44,7 @@
(deftest default-schema-test
(mt/test-driver :sqlserver
(is (= "dbo"
- (driver.sql/default-schema :sqlserver (mt/db))))))
+ (driver.sql/default-schema :sqlserver (lib.metadata/database (mt/metadata-provider)))))))
(deftest ^:parallel hour-bucketing-time-without-database-type-test
(testing (str "Hour bucketing on a TIME-typed expression without `:database-type` (as happens for "
@@ -491,7 +491,7 @@
;; are working we need to add to in from of the query we're trying to check
(sql-jdbc.execute/do-with-connection-with-options
:sqlserver
- (mt/db)
+ (mt/id)
{:session-timezone (qp.timezone/report-timezone-id-if-supported :sqlserver (mt/db))}
(fn [^java.sql.Connection conn]
(.setAutoCommit conn false)
@@ -521,7 +521,7 @@
;; persist otherwise
(sql-jdbc.execute/do-with-connection-with-options
:sqlserver
- (mt/db)
+ (mt/id)
{:session-timezone (qp.timezone/report-timezone-id-if-supported :sqlserver (mt/db))}
(fn [^java.sql.Connection conn]
(doseq [datefirst (range 1 8)]
@@ -546,7 +546,7 @@
;; persist otherwise
(sql-jdbc.execute/do-with-connection-with-options
:sqlserver
- (mt/db)
+ (mt/id)
{:session-timezone (qp.timezone/report-timezone-id-if-supported :sqlserver (mt/db))}
(fn [^java.sql.Connection conn]
(doseq [datefirst (range 1 8)]
@@ -587,7 +587,7 @@
(let [sql (format "SELECT %s AS t;" (sql.qp/inline-value :sqlserver t))]
(sql-jdbc.execute/do-with-connection-with-options
:sqlserver
- (mt/db)
+ (mt/id)
nil
(fn [^java.sql.Connection conn]
(with-open [stmt (sql-jdbc.execute/prepared-statement :sqlserver conn sql nil)
@@ -1075,7 +1075,7 @@
(deftest ^:parallel db-default-timezone-test
(mt/test-driver :sqlserver
- (is (= "Z" (str (driver/db-default-timezone :sqlserver (mt/db)))))))
+ (is (= "Z" (str (driver/db-default-timezone :sqlserver (lib.metadata/database (mt/metadata-provider))))))))
(deftest ^:parallel default-database-role-test
(testing "SQL Server default database role handling"
@@ -1172,7 +1172,7 @@
(mt/test-driver :sqlserver
(create-udt-alias-db!)
(mt/with-temp [:model/Database database {:engine :sqlserver, :details @udt-alias-db-details}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [table-id (t2/select-one-pk :model/Table :db_id (:id database) :name "udt_alias_test")
fields (into {} (map (juxt :name identity))
(t2/select :model/Field :table_id table-id))]
@@ -1310,7 +1310,7 @@
(mt/test-driver :sqlserver
(testing "`current-user-table-privileges` returns correct structure and privileges"
(sql-jdbc.execute/do-with-connection-with-options
- :sqlserver (mt/db) nil
+ :sqlserver (mt/id) nil
(fn [conn]
(let [privileges (sql-jdbc.sync/current-user-table-privileges :sqlserver {:connection conn})]
(is (seq privileges) "Should return at least one table")
@@ -1432,12 +1432,12 @@
(testing "a transform test's temp table is visible to its own session only"
(let [table (driver/temp-table-name :sqlserver)]
(driver/do-with-test-connection
- :sqlserver (mt/db)
+ :sqlserver (lib.metadata/database (mt/metadata-provider))
(fn [conn]
(create-temp-table! conn table "SELECT 1 AS id" [])
(is (= [[1]] (:rows (temp-table-rows conn table))))
(driver/do-with-test-connection
- :sqlserver (mt/db)
+ :sqlserver (lib.metadata/database (mt/metadata-provider))
(fn [other-conn]
(is (thrown-with-msg? Exception #"Invalid object name"
(temp-table-rows other-conn table)))))))))))
@@ -1447,7 +1447,7 @@
(testing "a temp table created by a query with bound parameters outlives the statement that created it"
(let [table (driver/temp-table-name :sqlserver)]
(driver/do-with-test-connection
- :sqlserver (mt/db)
+ :sqlserver (lib.metadata/database (mt/metadata-provider))
(fn [conn]
(create-temp-table! conn table "SELECT CAST(? AS nvarchar(50)) AS name, CAST(? AS decimal(10,2)) AS price"
["abc" 1.5M])
@@ -1455,7 +1455,7 @@
(testing "the test connection runs temp table statements as batches, and gets its prepare method back afterwards"
(let [physical (atom nil)]
(driver/do-with-test-connection
- :sqlserver (mt/db)
+ :sqlserver (lib.metadata/database (mt/metadata-provider))
(fn [^java.sql.Connection conn]
(let [sqlserver-conn (.unwrap conn com.microsoft.sqlserver.jdbc.ISQLServerConnection)]
(reset! physical sqlserver-conn)
diff --git a/modules/drivers/starburst/src/metabase/driver/starburst.clj b/modules/drivers/starburst/src/metabase/driver/starburst.clj
index 39161172f16..5005d544944 100644
--- a/modules/drivers/starburst/src/metabase/driver/starburst.clj
+++ b/modules/drivers/starburst/src/metabase/driver/starburst.clj
@@ -594,16 +594,6 @@
db-or-id-or-spec
options
(fn [^java.sql.Connection conn]
- (when-let [db (cond
- ;; id?
- (integer? db-or-id-or-spec)
- (driver-api/with-metadata-provider db-or-id-or-spec
- (driver-api/database (driver-api/metadata-provider)))
- ;; db?
- (u/id db-or-id-or-spec) db-or-id-or-spec
- ;; otherwise it's a spec and we can't get the db
- :else nil)]
- (sql-jdbc.execute/set-role-if-supported! driver conn db))
(try
(sql-jdbc.execute/set-best-transaction-level! driver conn)
(let [underlying-conn (pooled-conn->starburst-conn conn)]
diff --git a/modules/drivers/starburst/test/metabase/driver/starburst_test.clj b/modules/drivers/starburst/test/metabase/driver/starburst_test.clj
index 1e8f33cdd7c..586a9377fb4 100644
--- a/modules/drivers/starburst/test/metabase/driver/starburst_test.clj
+++ b/modules/drivers/starburst/test/metabase/driver/starburst_test.clj
@@ -12,6 +12,7 @@
[metabase.driver.sql-jdbc.sync.interface :as sql-jdbc.sync.interface]
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.driver.starburst :as starburst]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.query-processor.compile :as qp.compile]
[metabase.query-processor.test :as qp]
[metabase.query-processor.timezones-test :as timezones-test]
@@ -81,7 +82,7 @@
{:name "venues" :schema "default"}
{:name "checkins" :schema "default"}
{:name "users" :schema "default"}}}
- (-> (driver/describe-database :starburst (mt/db))
+ (-> (driver/describe-database :starburst (lib.metadata/database (mt/metadata-provider)))
(update :tables #(into #{} (filter (comp #{"categories"
"venues"
"checkins"
@@ -118,7 +119,7 @@
:database-type "integer"
:base-type :type/Integer
:database-position 0}}}
- (driver/describe-table :starburst (mt/db) (t2/select-one :model/Table :id (mt/id :venues)))))))
+ (driver/describe-table :starburst (lib.metadata/database (mt/metadata-provider)) (t2/select-one :model/Table :id (mt/id :venues)))))))
(deftest table-rows-sample-test
(mt/test-driver :starburst
@@ -156,7 +157,7 @@
(mt/test-driver :starburst
(testing "If global timezone is 'SYSTEM', should use system timezone"
(is (= "UTC"
- (driver/db-default-timezone driver/*driver* (mt/db)))))))
+ (driver/db-default-timezone driver/*driver* (lib.metadata/database (mt/metadata-provider))))))))
(deftest template-tag-timezone-test
(mt/test-driver :starburst
@@ -233,7 +234,7 @@
(t2.with-temp/with-temp [:model/Database db {:engine :starburst, :name "Temp starburst JDBC Schema DB", :details with-schema}]
(mt/with-db db
;; same as test_data, but with schema, so should NOT pick up venues, users, etc.
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (= [{:name t, :schema s, :db_id (mt/id)}]
(map #(select-keys % [:name :schema :db_id]) (t2/select :model/Table :db_id (mt/id)))))))
(execute-ddl! [(format "DROP TABLE %s.%s" s t)
@@ -397,7 +398,7 @@
(testing "have-select-privilege? returns true if table exists"
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (mt/id)
nil
(fn [^Connection conn]
(is (true?
@@ -405,7 +406,7 @@
(testing "have-select-privilege? returns false if table does not exist"
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (mt/id)
nil
(fn [^Connection conn]
(is (false?
diff --git a/modules/drivers/vertica/test/metabase/driver/vertica_test.clj b/modules/drivers/vertica/test/metabase/driver/vertica_test.clj
index a835ef2c44e..cec4f3f0ac9 100644
--- a/modules/drivers/vertica/test/metabase/driver/vertica_test.clj
+++ b/modules/drivers/vertica/test/metabase/driver/vertica_test.clj
@@ -5,6 +5,7 @@
[clojure.test :refer :all]
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.query-processor.compile :as qp.compile]
[metabase.test :as mt]
[metabase.test.data.interface :as tx]))
@@ -14,7 +15,7 @@
(deftest db-timezone-test
(mt/test-driver :vertica
(is (= "UTC"
- (driver/db-default-timezone :vertica (mt/db))))))
+ (driver/db-default-timezone :vertica (lib.metadata/database (mt/metadata-provider)))))))
(deftest ^:parallel additional-connection-string-options-test
(testing "Make sure you can add additional connection string options (#6651)"
diff --git a/resources/migrations/065/20261026_drop_admin_details.yaml b/resources/migrations/065/20261026_drop_admin_details.yaml
new file mode 100644
index 00000000000..5c19694f528
--- /dev/null
+++ b/resources/migrations/065/20261026_drop_admin_details.yaml
@@ -0,0 +1,21 @@
+## The admin connection is gone, so its connection details column goes too.
+
+databaseChangeLog:
+ - changeSet:
+ id: v65.2026-10-26T00:00:00
+ author: ranquild
+ comment: Drop metabase_database.admin_details
+ changes:
+ - dropColumn:
+ tableName: metabase_database
+ columnName: admin_details
+ rollback:
+ - addColumn:
+ tableName: metabase_database
+ columns:
+ - column:
+ name: admin_details
+ type: ${text.type}
+ remarks: Encrypted JSON containing connection details for admin (highest-privilege) operations such as DDL and ownership changes
+ constraints:
+ nullable: true
diff --git a/src/metabase/api_routes/routes.clj b/src/metabase/api_routes/routes.clj
index 9cc03a4f82f..70c31366185 100644
--- a/src/metabase/api_routes/routes.clj
+++ b/src/metabase/api_routes/routes.clj
@@ -57,6 +57,7 @@
[metabase.pulse.api]
[metabase.queries-rest.api]
[metabase.query-processor.api]
+ [metabase.request.core :as request]
[metabase.revisions.api]
[metabase.search.api]
[metabase.segments.rest.api]
@@ -176,6 +177,17 @@
(def ^:private ^{:arglists '([handler])} +sanitize-streaming-errors
(routes.common/wrap-middleware-for-open-api-spec-generation sanitize-streaming-errors))
+(defn- without-current-user
+ "Middleware that runs `handler` with no current user, keeping only the request's locale: guest embeds and public links
+ are authorized by their signed token or public UUID, never by the visitor's session."
+ [handler]
+ (fn [request respond raise]
+ (request/do-with-current-user (select-keys request [:user-locale])
+ #(handler request respond raise))))
+
+(def ^:private ^{:arglists '([handler])} +without-current-user
+ (routes.common/wrap-middleware-for-open-api-spec-generation without-current-user))
+
(defn- +auth [handler] (routes.common/+auth (->handler handler)))
(defn- +message-only-exceptions [handler] (routes.common/+message-only-exceptions (+sanitize-streaming-errors (->handler handler))))
(defn- +public-exceptions [handler] (routes.common/+public-exceptions (+sanitize-streaming-errors (->handler handler))))
@@ -226,7 +238,7 @@
"/document" (+auth metabase.documents.api/routes)
"/eid-translation" (+auth 'metabase.eid-translation.api)
"/email" (+auth metabase.channel.rest.api/email-routes)
- "/embed" (+message-only-exceptions metabase.embedding-rest.api/embedding-routes)
+ "/embed" (+message-only-exceptions (+without-current-user metabase.embedding-rest.api/embedding-routes))
"/embed-mcp" (+auth metabase.mcp.callback-api/routes)
"/embed-theme" (+auth metabase.embedding-rest.api/theme-routes)
"/embedding-hub" metabase.embedding-hub.api/routes
@@ -262,7 +274,7 @@
"/persist" (+auth 'metabase.model-persistence.api)
"/premium-features" (+auth metabase.premium-features.rest.api/routes)
"/preview_embed" (+auth metabase.embedding-rest.api/preview-embedding-routes)
- "/public" (+public-exceptions metabase.public-sharing-rest.api/routes)
+ "/public" (+public-exceptions (+without-current-user metabase.public-sharing-rest.api/routes))
"/pulse" metabase.pulse.api/pulse-routes
"/revision" (+auth 'metabase.revisions.api)
"/search" (+auth metabase.search.api/routes)
diff --git a/src/metabase/app_db/connection_pool_setup.clj b/src/metabase/app_db/connection_pool_setup.clj
index bc0da5c246d..85f5f84a137 100644
--- a/src/metabase/app_db/connection_pool_setup.clj
+++ b/src/metabase/app_db/connection_pool_setup.clj
@@ -4,6 +4,7 @@
[java-time.api :as t]
[metabase.config.core :as config]
[metabase.connection-pool :as connection-pool]
+ [metabase.util.connection :as u.connection]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
[metabase.util.malli.schema :as ms]
@@ -55,7 +56,7 @@
;; no-op
)
-(p/deftype+ MetabaseConnectionCustomizer []
+(p/deftype+ AppDbConnectionCustomizer []
ConnectionCustomizer
(onAcquire [_this connection _identity-token]
(on-acquire connection))
@@ -66,19 +67,7 @@
(onDestroy [_this connection _identity-token]
(on-destroy connection)))
-(defn- register-customizer!
- "c3p0 allows for hooking into lifecycles with its interface
- ConnectionCustomizer. https://www.mchange.com/projects/c3p0/apidocs/com/mchange/v2/c3p0/ConnectionCustomizer.html. But
- Clojure defined code is in memory in a dynamic class loader not available to c3p0's use of Class/forName. Luckily it
- looks up the instances in a cache which I pre-seed with out impl here. Issue for better access here:
- https://github.com/swaldman/c3p0/issues/166"
- [^Class klass]
- (let [field (doto (.getDeclaredField com.mchange.v2.c3p0.C3P0Registry "classNamesToConnectionCustomizers")
- (.setAccessible true))]
- (.put ^java.util.HashMap (.get field com.mchange.v2.c3p0.C3P0Registry)
- (.getName klass) (.newInstance klass))))
-
-(register-customizer! MetabaseConnectionCustomizer)
+(u.connection/register-c3p0-connection-customizer! AppDbConnectionCustomizer)
(defn- application-db-connection-pool-props
"Options for c3p0 connection pool for the application DB. These are set in code instead of the `c3p0.properties` file
@@ -105,7 +94,7 @@
;; https://www.mchange.com/projects/c3p0/#connectionCustomizerClassName
;;
"connectionCustomizerClassName"
- (.getName MetabaseConnectionCustomizer)
+ (.getName AppDbConnectionCustomizer)
;;
;; Number of seconds that Connections in excess of minPoolSize should be permitted to remain idle in the pool before
;; being culled. Intended for applications that wish to aggressively minimize the number of open Connections,
diff --git a/src/metabase/app_db/encryption.clj b/src/metabase/app_db/encryption.clj
index a5ea6e567c2..c70f426fec8 100644
--- a/src/metabase/app_db/encryption.clj
+++ b/src/metabase/app_db/encryption.clj
@@ -37,7 +37,6 @@
[[:metabase_database :details]
[:metabase_database :settings]
[:metabase_database :write_data_details]
- [:metabase_database :admin_details]
[:core_user :settings]
[:channel :details]
[:api_key :key]
diff --git a/src/metabase/classloader/impl.clj b/src/metabase/classloader/impl.clj
index 3bccd3b03c6..be6af4f21bb 100644
--- a/src/metabase/classloader/impl.clj
+++ b/src/metabase/classloader/impl.clj
@@ -138,6 +138,8 @@
:system-classpath (sort (str/split (System/getProperty "java.class.path") #"[:;]"))}
e))))))
+(defonce ^:private fully-loaded (atom #{}))
+
(defn require
"Just like vanilla `require`, but ensures we're using our shared classloader to do it. Always use this over vanilla
`require` -- otherwise namespaces might get loaded by the wrong ClassLoader, resulting in weird, hard-to-debug
@@ -145,13 +147,15 @@
Added benefit -- this is also thread-safe, unlike vanilla require."
([x]
- ;; Check whether the lib is already loaded (we only do this in simple cases where with just one arg -- this is
- ;; most of the calls anyway). If the lib is already loaded we can skip acquiring the lock and expensive stuff like
- ;; bindings and the try-catch
- (let [already-loaded? (and (symbol? x)
- ((loaded-libs) x))]
- (when-not already-loaded?
- (require* x))))
+ ;; Check whether this function already finished loading the lib (we only do this in simple cases where with just
+ ;; one arg -- this is most of the calls anyway). If so we can skip acquiring the lock and expensive stuff like
+ ;; bindings and the try-catch. `loaded-libs` can't answer this: a namespace joins it as soon as its `ns` form runs,
+ ;; while another thread may still be loading the rest of the file.
+ (when-not (and (symbol? x) (@fully-loaded x))
+ (require* x)
+ (when (and (symbol? x) (not *compile-files*))
+ (swap! fully-loaded conj x))
+ nil))
([x & more]
(apply require* x more)))
diff --git a/src/metabase/data_studio/api/table.clj b/src/metabase/data_studio/api/table.clj
index e14f0e7c2c0..396ee4a3c8d 100644
--- a/src/metabase/data_studio/api/table.clj
+++ b/src/metabase/data_studio/api/table.clj
@@ -210,7 +210,7 @@
(defn- sync-schema-async!
[table user-id]
(events/publish-event! :event/table-manual-sync {:object table :user-id user-id})
- (quick-task/submit-task! #(database-routing/with-database-routing-off (sync/sync-table! table))))
+ (quick-task/submit-task! #(database-routing/with-anonymous-router-access (sync/sync-table! table))))
(api.macros/defendpoint :post "/sync-schema" :- :nil
"Batch version of /table/:id/sync_schema. Takes an abstract table selection as /table/edit does.
diff --git a/src/metabase/database_routing/core.clj b/src/metabase/database_routing/core.clj
index 813e3c8cebd..8ea15f3a73b 100644
--- a/src/metabase/database_routing/core.clj
+++ b/src/metabase/database_routing/core.clj
@@ -1,39 +1,34 @@
(ns metabase.database-routing.core
"The OSS namespace for database routing."
(:require
- [metabase.premium-features.core :refer [defenterprise]]))
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
+ [metabase.premium-features.core :refer [defenterprise defenterprise-schema]]))
-(defenterprise with-database-routing-on-fn
- "OSS Version, does nothing"
+(defenterprise-schema routed-database :- [:maybe ::lib.schema.metadata/database]
+ "The Lib metadata of the destination Database that the current user's connections to the Lib metadata `database` go
+ to, or nil when they go to `database` itself; OSS has no database routing, so it is always nil."
metabase-enterprise.database-routing.common
- [f]
- (f))
-
-(defmacro with-database-routing-on
- "Turns database routing on. Access to a Router Database in this block will be an error (unless things are configured
- that way, of course.)"
- [& body]
- `(with-database-routing-on-fn
- (fn []
- ~@body)))
+ [_database :- ::lib.schema.metadata/database]
+ nil)
-(defenterprise with-database-routing-off-fn
- "OSS version, does nothing"
+(defenterprise do-with-anonymous-router-access
+ "Calls `thunk` with no current user, letting it reach a router itself; outside it, work with no user is refused on a
+ router. OSS has no routers, so there is nothing to allow."
metabase-enterprise.database-routing.common
- [f]
- (f))
+ [thunk]
+ (thunk))
-(defmacro with-database-routing-off
- "Turns database routing off. Access to a destination database within this block will result in an error."
+(defmacro with-anonymous-router-access
+ "Runs body with no current user, letting work whose results every user shares, such as sync or a guest embed, reach a
+ router itself."
[& body]
- `(with-database-routing-off-fn
- (fn []
- ~@body)))
+ `(do-with-anonymous-router-access (fn [] ~@body)))
-(defenterprise check-allowed-access!
- "OSS version, does nothing"
+(defenterprise-schema check-allowed-access! :- :nil
+ "Throws if the Lib metadata `db` of a Database may not be connected to directly; OSS has no destinations, so it does
+ nothing."
metabase-enterprise.database-routing.common
- [_db-or-id-or-spec]
+ [_db :- ::lib.schema.metadata/database]
nil)
(defenterprise delete-associated-database-router!
@@ -42,8 +37,8 @@
[_db-id]
nil)
-(defenterprise db-routing-enabled?
- "Returns whether or not the given database is either a router or destination database."
+(defenterprise-schema db-routing-enabled? :- :boolean
+ "Returns whether or not the Lib metadata `db` of a Database is either a router or destination database."
metabase-enterprise.database-routing.models
- [_db-or-id]
+ [_db :- ::lib.schema.metadata/database]
false)
diff --git a/src/metabase/driver.clj b/src/metabase/driver.clj
index 05a4020056d..80f6fc1fec4 100644
--- a/src/metabase/driver.clj
+++ b/src/metabase/driver.clj
@@ -1249,9 +1249,9 @@
;; TODO -- shouldn't this be called `notify-database-updated!`, since the expectation is that it is done for side
;; effects? issue: https://github.com/metabase/metabase/issues/39367
(defmulti notify-database-updated
- "Notify the driver that the attributes of a `database` have changed, or that `database was deleted. This is
- specifically relevant in the event that the driver was doing some caching or connection pooling; the driver should
- release ALL related resources when this is called."
+ "Notify the driver that the attributes of `database`, the Lib metadata of a Database, have changed, or that it was
+ deleted. This is specifically relevant in the event that the driver was doing some caching or connection pooling;
+ the driver should release ALL related resources when this is called."
{:added "0.32.0" :arglists '([driver database])}
dispatch-on-initialized-driver
:hierarchy #'hierarchy)
@@ -1413,10 +1413,9 @@
update the value for `:details`. The default implementation is essentially `identity` (i.e returns `database`
unchanged). This multimethod will only be called if `:details` is actually present in the `database` map.
- Implementations should normalize `:details`, `:write-data-details`, and `:admin-details` (each if present),
- since `:write-data-details` and `:admin-details` are merged on top of `:details` by
- [[metabase.driver.connection/effective-details]] under their respective contexts. Un-normalized fields in
- the overlay maps can leak through to the merged result. See [[metabase.driver.connection]] for more
+ Implementations should normalize `:details` and `:write-data-details` (each if present), since
+ `:write-data-details` is merged on top of `:details` by [[metabase.driver.connection/effective-details]] in a
+ write context. Un-normalized fields in the overlay map can leak through to the merged result. See [[metabase.driver.connection]] for more
information."
{:added "0.41.0" :arglists '([driver database])}
dispatch-on-initialized-driver
diff --git a/src/metabase/driver/common.clj b/src/metabase/driver/common.clj
index 44fef6eaa2a..873e335983e 100644
--- a/src/metabase/driver/common.clj
+++ b/src/metabase/driver/common.clj
@@ -3,7 +3,6 @@
(:require
[clojure.string :as str]
[metabase.driver :as driver]
- [metabase.driver.connection :as driver.conn]
[metabase.premium-features.core :as premium-features]
[metabase.settings.core :as setting]
[metabase.util.i18n :refer [deferred-tru]]
@@ -140,12 +139,6 @@
:type :hidden
:default false})
-(def admin-connection-option
- "Map representing the 'is this an admin connection' option"
- {:name "admin-connection"
- :type :hidden
- :default false})
-
;;; +----------------------------------------------------------------------------------------------------------------+
;;; | Index field descriptors (Index Manager) |
;;; +----------------------------------------------------------------------------------------------------------------+
@@ -226,8 +219,7 @@
(str "We execute the underlying query when you explore data using Summarize or Filter. "
"This is on by default but you can turn it off if performance is slow."))
:visible-if {"advanced-options" true
- "write-data-connection" false
- "admin-connection" false}})
+ "write-data-connection" false}})
(def let-user-control-scheduling
"Map representing the `let-user-control-scheduling` option in a DB connection form."
@@ -236,8 +228,7 @@
:display-name (deferred-tru "Choose when syncs and scans happen")
:description (deferred-tru "By default, Metabase does a lightweight hourly sync and an intensive daily scan of field values. If you have a large database, turn this on to make changes.")
:visible-if {"advanced-options" true
- "write-data-connection" false
- "admin-connection" false}})
+ "write-data-connection" false}})
(def metadata-sync-schedule
"Map representing the `schedules.metadata_sync` option in a DB connection form, which should be only visible if
@@ -267,8 +258,7 @@
:display-name (deferred-tru "Allow unfolding of JSON columns")
:type :boolean
:visible-if {"advanced-options" true
- "write-data-connection" false
- "admin-connection" false}
+ "write-data-connection" false}
:description (deferred-tru
(str "This enables unfolding JSON columns into their component fields. "
"Disable unfolding if performance is slow. If enabled, you can still disable unfolding for "
@@ -284,8 +274,7 @@
(str "This enables Metabase to scan for additional field values during syncs allowing smarter "
"behavior, like improved auto-binning on your bar charts."))
:visible-if {"advanced-options" true
- "write-data-connection" false
- "admin-connection" false}})
+ "write-data-connection" false}})
(def multi-level-schema
"Map representing the `multi-level-schema` option for databases. Stores schemas with multiple levels of hierarchy."
@@ -297,7 +286,6 @@
"Vector containing the most common options present in the advanced option section of the DB connection form."
[destination-database-option
write-data-connection-option
- admin-connection-option
auto-run-queries
let-user-control-scheduling
metadata-sync-schedule
@@ -500,7 +488,7 @@
;; This allows adding support for nested-field-columns for drivers in the future and
;; have json-unfolding enabled by default, without
;; needing a migration to add the `json-unfolding=true` key to the database details.
- (let [json-unfolding (:json-unfolding (driver.conn/effective-details database))]
+ (let [json-unfolding (:json-unfolding (:details database))]
(if (nil? json-unfolding)
true
json-unfolding)))
diff --git a/src/metabase/driver/connection.clj b/src/metabase/driver/connection.clj
index 6ca728e6eb0..3dfb9eb4666 100644
--- a/src/metabase/driver/connection.clj
+++ b/src/metabase/driver/connection.clj
@@ -10,10 +10,13 @@
Primary API: [[effective-details]], [[with-write-connection]], [[default-details]]."
(:require
[metabase.analytics-interface.core :as analytics]
+ [metabase.database-routing.core :as database-routing]
[metabase.driver.util :as driver.u]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.premium-features.core :refer [defenterprise]]
[metabase.util :as u]
[metabase.util.log :as log]
+ [metabase.util.malli :as mu]
[metabase.util.malli.registry :as mr]
[metabase.util.performance :as perf]))
@@ -61,7 +64,7 @@
(def connection-types
"All valid values for [[*connection-type*]]."
- [:default :write-data :admin :transform])
+ [:default :write-data :transform])
(mr/def ::connection-type
(into [:enum] connection-types))
@@ -72,13 +75,12 @@
- `:default` — primary `:details`
- `:write-data` — `:write-data-details` merged over `:details` (if configured)
- - `:admin` — `:admin-details` merged over `:details` (if configured)
- `:transform` — same details as `:write-data` (transforms write, so `:write-data-details`
are taken into account when configured), but resolved through a separate connection pool
whose c3p0 leak-detector tolerates transform-length runtimes. Set by
[[with-transform-connection]] from the transform runner.
- Bind via [[with-write-connection]], [[with-admin-connection]], or [[with-transform-connection]],
+ Bind via [[with-write-connection]] or [[with-transform-connection]],
not directly."
:default)
@@ -97,26 +99,6 @@
[& body]
`(do-with-write-connection (fn [] ~@body)))
-(defn do-with-admin-connection
- "Functional core of [[with-admin-connection]]."
- [thunk]
- (let [prior *connection-type*]
- (when (not= prior :admin)
- (log/infof "Entering :admin connection scope (from %s)" prior))
- (binding [*connection-type* :admin]
- (thunk))))
-
-(defmacro with-admin-connection
- "Establishes an admin-connection context for body.
-
- [[effective-details]] calls within this scope resolve to take `:admin-details`
- into account (if configured) instead of only primary `:details`. The admin
- connection carries the highest-privilege credentials for a database (DDL,
- ownership, schema management). Code that needs admin access must opt in
- explicitly."
- [& body]
- `(do-with-admin-connection (fn [] ~@body)))
-
(defn do-with-transform-connection
"Functional core of [[with-transform-connection]]."
[thunk]
@@ -152,12 +134,12 @@
[_database]
nil)
-(defenterprise database-admin-details
- "Returns the `:admin-details` for a database, or `nil` if the connection-overlays feature is not available.
- OSS implementation always returns `nil`."
- metabase-enterprise.connection-overlays.core
- [_database]
- nil)
+(defn connection-database
+ "The Database that a connection for `database` goes to in the current context: the current user's routing
+ destination when `database` is a router, otherwise `database` itself."
+ [database]
+ (or (database-routing/routed-database database)
+ database))
(defn- overlay-details-for-type
"Returns the connection-type-specific details overlay for `database`, or nil if the
@@ -170,8 +152,7 @@
(case connection-type
:default nil
:write-data (database-write-data-details database)
- :transform (database-write-data-details database)
- :admin (database-admin-details database)))
+ :transform (database-write-data-details database)))
(defn- resolve-effective-type
"Maps a requested `connection-type` plus whether an `overlay` was resolved to the effective
@@ -184,48 +165,49 @@
overlay connection-type
:else :default))
-(defn effective-details
+(mu/defn effective-details
"Returns the connection details map appropriate for the current context.
- Accepts a database (Toucan2 instance or lib/metadata). Returns nil for nil input.
+ Takes the Lib metadata of a Database.
+
+ Resolves database routing first: for a router, the details are those of the current user's destination (see
+ [[connection-database]]).
- By default, returns the primary `:details`. Within a [[with-write-connection]] or
- [[with-admin-connection]] scope, takes the corresponding `:write-data-details` /
- `:admin-details` into account (if configured). Within a [[with-transform-connection]]
+ By default, returns the primary `:details`. Within a [[with-write-connection]] scope, takes
+ `:write-data-details` into account (if configured). Within a [[with-transform-connection]]
scope, takes `:write-data-details` into account (if configured) — transforms write, so
they get write-data credentials when set — but resolves to a *different* pool key
(`:transform` vs `:write-data`) so the pool properties (e.g. `unreturnedConnectionTimeout`)
can differ."
- [database]
- (when-let [database (some-> database driver.u/ensure-lib-database)]
- (let [overlay (perf/not-empty (overlay-details-for-type database *connection-type*))
- base (merge (:details database) overlay)
- eff-type (resolve-effective-type *connection-type* overlay)]
- ;; Track when an overlay is genuinely used (not fallback).
- ;; Default resolutions are not tracked here — see :metabase-db-connection/write-op for
- ;; pool-level connection acquisition metrics.
- (when (and overlay
- (not *suppress-resolution-telemetry*))
- (try (analytics/inc! :metabase-db-connection/type-resolved
- {:connection-type (name *connection-type*)})
- (catch Exception _ nil)))
- (-> base
- (assoc ::connection-pool-type eff-type)
- (assoc ::database-id (u/id database))))))
+ [database :- ::lib.schema.metadata/database]
+ (let [database (connection-database database)
+ overlay (perf/not-empty (overlay-details-for-type database *connection-type*))
+ base (merge (:details database) overlay)
+ eff-type (resolve-effective-type *connection-type* overlay)]
+ ;; Track when an overlay is genuinely used (not fallback).
+ ;; Default resolutions are not tracked here — see :metabase-db-connection/write-op for
+ ;; pool-level connection acquisition metrics.
+ (when (and overlay
+ (not *suppress-resolution-telemetry*))
+ (try (analytics/inc! :metabase-db-connection/type-resolved
+ {:connection-type (name *connection-type*)})
+ (catch Exception _ nil)))
+ (-> base
+ (assoc ::connection-pool-type eff-type)
+ (assoc ::database-id (u/id database)))))
(defn details-for-exact-type
"Returns the details map for exactly the given connection-type, with no fallback or merging.
- Unlike [[effective-details]], `:write-data` and `:admin` return only their respective
- overlay maps (possibly nil), not a merge with `:details`. Use when you need to inspect
+ Unlike [[effective-details]], `:write-data` returns only its overlay map (possibly nil), not
+ a merge with `:details`. Use when you need to inspect
or update a specific details map without resorting to raw key access."
[database connection-type]
(let [database (driver.u/ensure-lib-database database)]
(case connection-type
:default (:details database)
:write-data (database-write-data-details database)
- :transform (:details database)
- :admin (database-admin-details database))))
+ :transform (:details database))))
(defn connection-telemetry-info
"A human-readable description of the current connection context, for logging only. Prose, not a
diff --git a/src/metabase/driver/db.clj b/src/metabase/driver/db.clj
index 88d8056e606..7d090be0959 100644
--- a/src/metabase/driver/db.clj
+++ b/src/metabase/driver/db.clj
@@ -13,16 +13,16 @@
[]
(t2/select :model/Database))
+(mu/defn database-exists?
+ "Whether the Database with `database-id` exists."
+ [database-id :- ::lib.schema.id/database]
+ (t2/exists? :model/Database :id database-id))
+
(mu/defn table-field-names
"The names of the Fields of the Table with `table-id`."
[table-id :- ::lib.schema.id/table]
(t2/select-fn-vec :name [:model/Field :name] :table_id table-id {:from [(warehouse-schema-overlay/field-query {:user-settings? false})]}))
-(mu/defn database-connection-details
- "The engine and connection details of the Database with `database-id`, or nil."
- [database-id :- ::lib.schema.id/database]
- (t2/select-one [:model/Database :id :engine :details :write_data_details :admin_details] :id database-id))
-
(mu/defn json-field-names-with-unfolding-disabled
"The names of the JSON Fields of the Table with `table-id` that have JSON unfolding disabled."
[table-id :- ::lib.schema.id/table]
diff --git a/src/metabase/driver/events/driver_notifications.clj b/src/metabase/driver/events/driver_notifications.clj
index 567eda2b8f4..bb726015fef 100644
--- a/src/metabase/driver/events/driver_notifications.clj
+++ b/src/metabase/driver/events/driver_notifications.clj
@@ -8,6 +8,7 @@
[medley.core :as m]
[metabase.driver :as driver]
[metabase.driver-api.core :as driver-api]
+ [metabase.driver.util :as driver.u]
[metabase.events.core :as events]
[metabase.util.log :as log]
[methodical.core :as methodical]))
@@ -30,6 +31,6 @@
(when (or details-changed?
(not= (remove-irrelevant-data database)
(remove-irrelevant-data previous-database)))
- (driver/notify-database-updated (:engine database) database)))
+ (driver/notify-database-updated (:engine database) (driver.u/ensure-lib-database database))))
(catch Throwable e
(log/warnf "Failed to process driver notifications event %s: %s" topic (ex-message e)))))
diff --git a/src/metabase/driver/events/report_timezone_updated.clj b/src/metabase/driver/events/report_timezone_updated.clj
index 413fad4da63..245f87740ba 100644
--- a/src/metabase/driver/events/report_timezone_updated.clj
+++ b/src/metabase/driver/events/report_timezone_updated.clj
@@ -3,6 +3,7 @@
[metabase.driver :as driver]
[metabase.driver-api.core :as driver-api]
[metabase.driver.db :as driver.db]
+ [metabase.driver.util :as driver.u]
[metabase.events.core :as events]
[metabase.util.log :as log]
[methodical.core :as methodical]))
@@ -20,7 +21,7 @@
[]
(doseq [{driver :engine, id :id, :as database} (driver.db/databases)]
(try
- (driver/notify-database-updated driver database)
+ (driver/notify-database-updated driver (driver.u/ensure-lib-database database))
(catch Throwable e
(log/errorf "Failed to notify %s Database %s updated: %s" driver id (ex-message e))))))
diff --git a/src/metabase/driver/h2.clj b/src/metabase/driver/h2.clj
index 4fc81705e4a..1cc0bc99b88 100644
--- a/src/metabase/driver/h2.clj
+++ b/src/metabase/driver/h2.clj
@@ -19,7 +19,6 @@
[metabase.driver.sql.normalize :as sql.normalize]
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.driver.sql.query-processor.like-escape-char-built-in :as like-escape-char-built-in]
- [metabase.lib.schema.id :as lib.schema.id]
[metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.util :as u]
[metabase.util.honey-sql-2 :as h2x]
@@ -212,10 +211,10 @@
(ex-info (tru "Running SQL queries against H2 databases using the default (admin) database user is forbidden.")
{:type driver-api/qp.error-type.db})))))))
-(defn- make-h2-parser
- "Returns an H2 Parser object for the given (H2) database ID"
- ^Parser [h2-database-or-id]
- (with-open [conn (.getConnection (sql-jdbc.execute/datasource-with-diagnostic-info! :h2 h2-database-or-id))]
+(mu/defn- make-h2-parser :- [:maybe (driver-api/instance-of-class Parser)]
+ "Returns an H2 Parser object for the Lib metadata `h2-database` of an H2 Database."
+ [h2-database :- ::lib.schema.metadata/database]
+ (with-open [conn (.getConnection (sql-jdbc.execute/datasource-with-diagnostic-info! :h2 h2-database))]
;; The H2 Parser class is created from the H2 JDBC session, but these fields are not public
(let [^org.h2.jdbc.JdbcConnection inner (try
;; May be a wrapper, get the innermost object that has session field
@@ -223,7 +222,7 @@
(assert (instance? org.h2.jdbc.JdbcConnection <>)))
(catch java.sql.SQLException e
(throw (ex-info "Not an H2 connection. Are we sure this is an H2 database?"
- {:database h2-database-or-id
+ {:database (:id h2-database)
:conn conn}
e))))
session (get-field inner "session")]
@@ -236,7 +235,7 @@
[:map
[:command-types [:vector pos-int?]]
[:remaining-sql [:maybe :string]]]]
- "Takes an h2 db id, and a query, returns the command-types from `query` and any remaining sql.
+ "Takes the Lib metadata of an h2 Database, and a query, returns the command-types from `query` and any remaining sql.
More info on command types here:
https://github.com/h2database/h2database/blob/master/h2/src/main/org/h2/command/CommandInterface.java
@@ -245,11 +244,9 @@
- Each `command-type` corresponds to a value in org.h2.command.CommandInterface, and match the commands from `query` in order.
- `remaining-sql` is a nillable sql string that is unable to be classified without running preceding queries first.
Usually if `remaining-sql` exists we will deny the query."
- [database-or-id :- [:or
- ::lib.schema.id/database
- ::lib.schema.metadata/database]
- ^String query :- :string]
- (when-let [h2-parser (make-h2-parser database-or-id)]
+ [database :- ::lib.schema.metadata/database
+ ^String query :- :string]
+ (when-let [^Parser h2-parser (make-h2-parser database)]
(try
(let [command (.prepareCommand h2-parser query)
first-command-type (.getCommandType command)
@@ -334,7 +331,9 @@
(defn- check-action-commands-allowed [{:keys [database] {:keys [query]} :native}]
(when query
(check-no-unsupported-functions query)
- (when-let [query-classification (classify-query database query)]
+ (when-let [query-classification (classify-query (driver-api/with-metadata-provider database
+ (driver-api/database (driver-api/metadata-provider)))
+ query)]
(when-not (every-command-allowed-for-actions? query-classification)
(throw (ex-info "DDL commands are not allowed to be used with H2."
{:classification query-classification}))))))
diff --git a/src/metabase/driver/h2/actions.clj b/src/metabase/driver/h2/actions.clj
index c5031e95ea5..3c2cff12427 100644
--- a/src/metabase/driver/h2/actions.clj
+++ b/src/metabase/driver/h2/actions.clj
@@ -45,7 +45,7 @@
(defn- constraint->column-names
"Given a constraint with `constraint-name` fetch the column names associated with that constraint."
[database table-name constraint-name]
- (let [jdbc-spec (sql-jdbc.conn/db->pooled-connection-spec (u/the-id database))
+ (let [jdbc-spec (sql-jdbc.conn/db->pooled-connection-spec database)
sql-args ["SELECT C.TABLE_CATALOG, C.TABLE_SCHEMA, K.COLUMN_NAME
FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS C
JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE K ON C.CONSTRAINT_NAME = K.CONSTRAINT_NAME
diff --git a/src/metabase/driver/mysql.clj b/src/metabase/driver/mysql.clj
index 1772fd1b1a4..625c1d9d628 100644
--- a/src/metabase/driver/mysql.clj
+++ b/src/metabase/driver/mysql.clj
@@ -27,6 +27,7 @@
[metabase.driver.sql.query-processor.like-escape-char-built-in :as like-escape-char-built-in]
[metabase.driver.sql.query-processor.util :as sql.qp.u]
[metabase.driver.sql.util :as sql.u]
+ [metabase.driver.util :as driver.u]
[metabase.lib.schema.common :as lib.schema.common]
[metabase.util :as u]
[metabase.util.date-2 :as u.date]
@@ -1034,26 +1035,26 @@
(t/local-date-time offset-time zone-id)))
(defmulti ^:private value->string
- "Convert a value into a string that's safe for insertion"
- {:arglists '([driver val])}
- (fn [_ val] (type val)))
+ "Convert a value into a string that's safe for insertion into `database`."
+ {:arglists '([driver database val])}
+ (fn [_ _ val] (type val)))
(defmethod value->string :default
- [_driver val]
+ [_driver _database val]
(str val))
(defmethod value->string nil
- [_driver _val]
+ [_driver _database _val]
nil)
(defmethod value->string Boolean
- [_driver val]
+ [_driver _database val]
(if val
"1"
"0"))
(defmethod value->string LocalDateTime
- [_driver val]
+ [_driver _database val]
(t/format :iso-local-date-time val))
(let [zulu-fmt "yyyy-MM-dd'T'HH:mm:ss"
@@ -1061,14 +1062,13 @@
zulu-formatter (DateTimeFormatter/ofPattern zulu-fmt)
offset-formatter (DateTimeFormatter/ofPattern (str zulu-fmt offset-fmt))]
(defmethod value->string OffsetDateTime
- [driver ^OffsetDateTime val]
- (let [uploads-db (driver-api/current-database)]
- (if (mariadb? uploads-db)
- (offset-datetime->unoffset-datetime driver uploads-db val)
- (t/format (if (.equals (.getOffset val) ZoneOffset/UTC)
- zulu-formatter
- offset-formatter)
- val)))))
+ [driver database ^OffsetDateTime val]
+ (if (mariadb? database)
+ (offset-datetime->unoffset-datetime driver database val)
+ (t/format (if (.equals (.getOffset val) ZoneOffset/UTC)
+ zulu-formatter
+ offset-formatter)
+ val))))
(defn- sanitize-value
;; Per https://dev.mysql.com/doc/refman/8.0/en/load-data.html#load-data-field-line-handling
@@ -1084,11 +1084,11 @@
"\t" "\\t"})))
(defn- row->tsv
- [driver column-count row]
+ [driver database column-count row]
(when (not= column-count (count row))
(throw (Exception. (format "ERROR: missing data in row \"%s\"" (str/join "," row)))))
(->> row
- (map (comp sanitize-value (partial value->string driver)))
+ (map (comp sanitize-value (partial value->string driver database)))
(str/join "\t")))
(defn- get-global-variable
@@ -1097,7 +1097,7 @@
[db-id var-name]
(:value
(first
- (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec db-id)
+ (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))
["show global variables like ?" var-name]))))
(defmethod driver/insert-into! :mysql
@@ -1110,7 +1110,8 @@
(let [temp-file (File/createTempFile (name table-name) ".tsv")
file-path (.getAbsolutePath temp-file)]
(try
- (let [tsvs (map (partial row->tsv driver (count column-names)) values)
+ (let [database (driver.u/lib-database db-id)
+ tsvs (map (partial row->tsv driver database (count column-names)) values)
dialect (sql.qp/quote-style driver)
sql (sql/format {::load [file-path (keyword table-name)]
:columns (quote-columns driver column-names)}
diff --git a/src/metabase/driver/mysql/actions.clj b/src/metabase/driver/mysql/actions.clj
index 33987d62820..88777fdb8e6 100644
--- a/src/metabase/driver/mysql/actions.clj
+++ b/src/metabase/driver/mysql/actions.clj
@@ -38,7 +38,7 @@
(defn- constraint->column-names
"Given a constraint with `constraint-name` fetch the column names associated with that constraint."
[database constraint-name]
- (let [jdbc-spec (sql-jdbc.conn/db->pooled-connection-spec (u/the-id database))
+ (let [jdbc-spec (sql-jdbc.conn/db->pooled-connection-spec database)
sql-args ["select table_catalog, table_schema, column_name from information_schema.key_column_usage where constraint_name = ?" constraint-name]]
(first
(reduce
diff --git a/src/metabase/driver/postgres.clj b/src/metabase/driver/postgres.clj
index 722128fd899..0886bc1ce18 100644
--- a/src/metabase/driver/postgres.clj
+++ b/src/metabase/driver/postgres.clj
@@ -28,6 +28,7 @@
[metabase.driver.sql.query-processor.like-escape-char-built-in :as like-escape-char-built-in]
[metabase.driver.sql.query-processor.util :as sql.qp.u]
[metabase.driver.sql.util :as sql.u]
+ [metabase.driver.util :as driver.u]
[metabase.util :as u]
[metabase.util.date-2 :as u.date]
[metabase.util.honey-sql-2 :as h2x]
@@ -1211,7 +1212,7 @@
:quoted true
:dialect (sql.qp/quote-style driver))))
sorted-rename-map)]
- (jdbc/with-db-transaction [t-conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [t-conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(with-open [stmt (.createStatement ^java.sql.Connection (:connection t-conn))]
(doseq [sql sqls]
(.addBatch ^java.sql.Statement stmt ^String sql))
@@ -1300,7 +1301,7 @@
(defmethod driver/insert-into! :postgres
[driver db-id table-name column-names values]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(let [copy-manager (CopyManager. (.unwrap ^Connection (:connection conn) PgConnection))
dialect (sql.qp/quote-style driver)
[sql & _] (sql/format {::copy (keyword table-name)
diff --git a/src/metabase/driver/postgres/actions.clj b/src/metabase/driver/postgres/actions.clj
index 97e5fcc0165..c8701ab63d1 100644
--- a/src/metabase/driver/postgres/actions.clj
+++ b/src/metabase/driver/postgres/actions.clj
@@ -17,7 +17,7 @@
(defn- constraint->column-names
"Given a constraint with `constraint-name` fetch the column names associated with that constraint."
[database constraint-name]
- (let [jdbc-spec (sql-jdbc.conn/db->pooled-connection-spec (u/the-id database))
+ (let [jdbc-spec (sql-jdbc.conn/db->pooled-connection-spec database)
sql-args ["select column_name from information_schema.constraint_column_usage where constraint_name = ?" constraint-name]]
(into []
(map :column_name)
diff --git a/src/metabase/driver/sql_jdbc.clj b/src/metabase/driver/sql_jdbc.clj
index f2c544d4603..2e252fd42f0 100644
--- a/src/metabase/driver/sql_jdbc.clj
+++ b/src/metabase/driver/sql_jdbc.clj
@@ -21,6 +21,7 @@
[metabase.driver.sync :as driver.s]
[metabase.driver.util :as driver.u]
[metabase.lib.schema.metadata :as lib.schema.metadata]
+ [metabase.util :as u]
[metabase.util.honey-sql-2 :as h2x]
[metabase.util.malli :as mu]
[metabase.util.performance :refer [mapv not-empty select-keys]]
@@ -113,7 +114,7 @@
(defmethod driver/notify-database-updated :sql-jdbc
[_ database]
- (sql-jdbc.conn/invalidate-pool-for-db! database)
+ (sql-jdbc.conn/invalidate-pool-for-db! (u/the-id database))
(memoize/memo-clear! driver-api/secret-value-as-file!))
(defmethod driver/dbms-version :sql-jdbc
@@ -186,7 +187,7 @@
(defmethod driver/create-table! :sql-jdbc
[driver database-id table-name column-definitions & {:keys [primary-key]}]
(let [sql (create-table!-sql driver table-name column-definitions :primary-key primary-key)]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec database-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database database-id))]
(jdbc/execute! conn sql))))
(defn- drop-table-sql [driver table-name]
@@ -197,7 +198,7 @@
(defmethod driver/drop-table! :sql-jdbc
[driver db-id table-name]
(let [sql (drop-table-sql driver table-name)]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(jdbc/execute! conn sql))))
(defmulti create-index-sql
@@ -225,7 +226,7 @@
(defmethod driver/create-index! :sql-jdbc
[driver database-id schema table-name index-name column-names & {:as opts}]
(let [sql (create-index-sql driver schema table-name index-name column-names opts)]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec database-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database database-id))]
(jdbc/execute! conn sql))
nil))
@@ -244,7 +245,7 @@
(defmethod driver/drop-index! :sql-jdbc
[driver database-id schema table-name index-name & _]
(let [sql (drop-index-sql driver schema table-name index-name)]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec database-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database database-id))]
(jdbc/execute! conn sql))
nil))
@@ -254,7 +255,7 @@
sql (sql/format {:truncate table-name}
:quoted true
:dialect (sql.qp/quote-style driver))]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(jdbc/execute! conn sql))))
(defn- lift-boolean [v] (if (boolean? v) [:lift v] v))
@@ -290,7 +291,7 @@
(defmethod driver/insert-into! :sql-jdbc
[driver db-id table-name column-names values]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(doseq [sql (insert-into!-sqls driver table-name column-names values false)]
(jdbc/execute! conn sql))))
@@ -314,7 +315,7 @@
column-definitions)}
:quoted true
:dialect (sql.qp/quote-style driver)))]
- (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec db-id)]
+ (jdbc/with-db-transaction [conn (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database db-id))]
(jdbc/execute! conn sql)))))
;; kept for get-method driver compatibility
diff --git a/src/metabase/driver/sql_jdbc/connection.clj b/src/metabase/driver/sql_jdbc/connection.clj
index f06ac35fac0..78d742502ce 100644
--- a/src/metabase/driver/sql_jdbc/connection.clj
+++ b/src/metabase/driver/sql_jdbc/connection.clj
@@ -14,15 +14,18 @@
[metabase.driver.sql-jdbc.connection.pool-lock :as pool-lock]
[metabase.driver.sql-jdbc.connection.ssh-tunnel :as ssh]
[metabase.driver.util :as driver.u]
+ [metabase.lib.schema.id :as lib.schema.id]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
+ [metabase.premium-features.core :refer [defenterprise]]
[metabase.util :as u]
+ [metabase.util.connection :as u.connection]
[metabase.util.i18n :refer [tru]]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
[metabase.util.performance :refer [get-in mapv select-keys]]
- [metabase.warehouses.schema :as warehouses.schema]
[potemkin :as p])
(:import
- (com.mchange.v2.c3p0 C3P0ProxyConnection DataSources PooledDataSource)
+ (com.mchange.v2.c3p0 C3P0ProxyConnection ConnectionCustomizer DataSources PooledDataSource)
(java.sql Connection SQLException)
(javax.sql DataSource)
(org.apache.logging.log4j Level)))
@@ -60,7 +63,7 @@
"The connection spec `details` would produce, or nil when they do not make one. Used to read where a connection would
go, never to open it.
- A `:write_data_details` or `:admin_details` overlay reaches us as a partial map that may not make a whole spec.
+ A `:write_data_details` overlay reaches us as a partial map that may not make a whole spec.
Details that cannot produce a spec cannot open a connection either, so nil is a safe answer rather than a blind
spot."
[driver details]
@@ -135,13 +138,13 @@
(def ^:private pool-type->name-suffix
"Suffix appended to the pool name for each non-default [[driver.conn/connection-pool-type]]."
{:write-data "-write"
- :admin "-admin"
:transform "-transform"})
-(defn- pool-data-source-name
+(mu/defn- pool-data-source-name :- :string
"The c3p0 `dataSourceName` for `database`'s pool of the current connection type: `db-<id>-<driver>-<name>`, with
`-write` (or the other suffixes in [[pool-type->name-suffix]]) appended for a non-default pool type."
- [driver database]
+ [driver :- :keyword
+ database :- ::lib.schema.metadata/database]
(str (format "db-%d-%s-%s"
(u/the-id database)
(name driver)
@@ -262,12 +265,58 @@
[spec-or-details]
(select-keys spec-or-details [:tunnel-enabled :tunnel-session :tunnel-tracker :tunnel-entrance-port :tunnel-entrance-host]))
-(defn- create-pool!
+(defenterprise set-role-if-supported!
+ "Sets the connection impersonation role of the current user, or the default role, on the connection `conn` to
+ `database`; OSS has no connection impersonation, so the connection keeps its own user."
+ metabase-enterprise.impersonation.driver
+ [_driver _conn _database])
+
+(defonce ^:private pool-databases
+ (atom {}))
+
+(defonce ^:private ^ThreadLocal checkout-error
+ (ThreadLocal.))
+
+(defn- checkout-database
+ "The Database to set the role for at checkout: the running query's own Database when it is the pool's, which is read
+ fresh for each query, or else the `database` the pool was made for."
+ [database]
+ (or (when (driver-api/initialized?)
+ (let [current (driver-api/database (driver-api/metadata-provider))]
+ (when (= (:id current) (:id database))
+ current)))
+ database))
+
+(p/deftype+ WarehouseConnectionCustomizer []
+ ConnectionCustomizer
+ (onAcquire [_this _connection _identity-token])
+ (onCheckIn [_this _connection _identity-token])
+ (onCheckOut [_this connection identity-token]
+ (try
+ (let [[driver database] (or (get @pool-databases identity-token)
+ (throw (ex-info (tru "This connection pool is closed.") {})))]
+ (set-role-if-supported! driver connection (checkout-database database)))
+ (catch Throwable e
+ (.set checkout-error e)
+ (.close ^Connection connection))))
+ (onDestroy [_this _connection _identity-token]))
+
+(u.connection/register-c3p0-connection-customizer! WarehouseConnectionCustomizer)
+
+(defn take-checkout-error!
+ "The error raised setting the connection impersonation role on the connection last checked out on this thread, whose
+ physical connection was closed instead, or nil; clears it."
+ []
+ (let [e (.get checkout-error)]
+ (.remove checkout-error)
+ e))
+
+(mu/defn- create-pool!
"Create a new C3P0 `ComboPooledDataSource` for connecting to the given `database`.
Uses [[driver.conn/effective-details]] to select the appropriate connection details
- for the current connection context."
- [{:keys [id], driver :engine, :as database}]
- {:pre [(map? database)]}
+ for the current connection context, and [[WarehouseConnectionCustomizer]] to set the current user's connection
+ impersonation role on every checkout."
+ [{:keys [id], driver :engine, :as database} :- ::lib.schema.metadata/database]
(log/debug (u/format-color :cyan "Creating new connection pool for %s database %s (%s) ..."
driver id (driver.conn/connection-telemetry-info)))
(let [details (driver.conn/effective-details database)
@@ -288,9 +337,12 @@
;; the auth provider's response is merged into the details, so it can set parameters too
_ (driver.u/validate-connection-parameters! driver details-with-auth)
spec (connection-details->spec driver details-with-auth)
- properties (data-warehouse-connection-pool-properties driver database)]
+ properties (assoc (data-warehouse-connection-pool-properties driver database)
+ "connectionCustomizerClassName" (.getName WarehouseConnectionCustomizer))
+ pool-spec (connection-pool-spec spec properties)]
+ (swap! pool-databases assoc (.getIdentityToken ^PooledDataSource (:datasource pool-spec)) [driver database])
(merge
- (connection-pool-spec spec properties)
+ pool-spec
;; also capture entries related to ssh tunneling for later use
(select-internal-keys-for-spec details-with-auth)
(select-internal-keys-for-spec spec)
@@ -306,6 +358,9 @@
;; closed" (see DEV-2161). Pool destruction is rare and significant enough to warrant INFO in production too.
(log/info (u/format-color :red "Closing old connection pool for database %s ..." database-id))
(driver-api/destroy-connection-pool! pool-spec)
+ (when-let [data-source (:datasource pool-spec)]
+ (when (instance? PooledDataSource data-source)
+ (swap! pool-databases dissoc (.getIdentityToken ^PooledDataSource data-source))))
(ssh/close-tunnel! pool-spec))
(defonce ^:private ^{:doc "A map of our currently open connection pools,
@@ -319,21 +374,20 @@
pool-cache-key->jdbc-spec-hash
(atom {}))
-(defn- pool-cache-key
+(mu/defn- pool-cache-key
"Returns the cache key for connection pools: `[database-id, connection-type]`.
Uses [[driver.conn/connection-pool-type]] so that a requested write connection
without configured `:write-data-details` resolves to `:default`, reusing the
existing pool instead of creating a duplicate."
- [database]
+ [database :- ::lib.schema.metadata/database]
[(u/the-id database) (driver.conn/connection-pool-type database)])
(mu/defn- jdbc-spec-hash
"Computes a hash value for the JDBC connection spec based on the effective connection details, for the purpose of
determining if details changed and therefore the existing connection pool needs to be invalidated.
Uses [[driver.conn/effective-details]] to select the appropriate details for the current connection context."
- [{driver :engine, :as database} :- [:maybe ::warehouses.schema/database-or-metadata]]
- (when (some? database)
- (hash (connection-details->spec driver (driver.conn/effective-details database)))))
+ [{driver :engine, :as database} :- ::lib.schema.metadata/database]
+ (hash (connection-details->spec driver (driver.conn/effective-details database))))
(defn- set-canonical-pool!
"Atomically update the canonical connection pool for a database and connection type.
@@ -354,12 +408,11 @@
(swap! pool-cache-key->jdbc-spec-hash assoc cache-key details-hash)
nil)
-(defn invalidate-pool-for-db!
- "Invalidates all connection pools for the given database (all connection types) by closing
+(mu/defn invalidate-pool-for-db!
+ "Invalidates all connection pools for the Database with `db-id` (all connection types) by closing
them and removing from cache."
- [database]
- (let [db-id (u/the-id database)
- canonical-keys (mapv (fn [conn-type] [db-id conn-type])
+ [db-id :- ::lib.schema.id/database]
+ (let [canonical-keys (mapv (fn [conn-type] [db-id conn-type])
driver.conn/connection-types)
pool-map @pool-cache-key->connection-pool
canonical-count (count (filter pool-map canonical-keys))]
@@ -461,8 +514,9 @@
;; the hash didn't match, but it's possible that a stale instance of `DatabaseInstance`
;; was passed in (ex: from a long-running sync operation); fetch the latest one from
;; our app DB, and see if it STILL doesn't match
- (not= curr-hash (-> (driver.db/database-connection-details database-id)
- jdbc-spec-hash)))))
+ (or (not (driver.db/database-exists? database-id))
+ (not= curr-hash (-> (driver.u/lib-database database-id)
+ jdbc-spec-hash))))))
(defn- get-canonical-pool
"Get a canonical pool if it exists and is valid, otherwise return nil.
@@ -485,62 +539,42 @@
:else
(get-pool-if-valid pool-spec database-id log-invalidation?))))
-(defn db->pooled-connection-spec
- "Return a JDBC connection spec that includes a c3p0 `ComboPooledDataSource`. These connection pools are cached so we
- don't create multiple ones for the same DB and connection type. The connection type follows the current
- connection context — use [[driver.conn/with-write-connection]] to get a write connection pool."
- [db-or-id-or-spec]
- (when-let [db-id (u/id db-or-id-or-spec)]
- (driver-api/check-allowed-access! db-id))
- (cond
- ;; db-or-id-or-spec is a Database instance or an integer ID
- (u/id db-or-id-or-spec)
- (let [database-id (u/the-id db-or-id-or-spec)
- ;; we need the Database instance no matter what (in order to calculate details hash)
- db (or (when (driver-api/instance-of? :model/Database db-or-id-or-spec)
- (driver-api/instance->metadata db-or-id-or-spec :metadata/database))
- (when (= (:lib/type db-or-id-or-spec) :metadata/database)
- db-or-id-or-spec)
- (driver-api/with-metadata-provider database-id
- (driver-api/database (driver-api/metadata-provider))))
- cache-key (pool-cache-key db)
- ;; Calculate hash from effective details (includes write-connection merge)
- details-hash (jdbc-spec-hash db)]
- (driver.conn/track-connection-acquisition! (driver.conn/effective-details db))
- (cond
- (or (:is-audit db)
- (and (audit-app/analytics-dev-mode)
- (get-in db [:details :is-audit-dev])))
- {:datasource (driver-api/data-source)}
-
- (get-in db [:details :is-audit-dev])
- (throw (ex-info (tru "Cannot open a connection for an analytics-dev database unless analytics dev mode is enabled.")
- {:database-id (:id db)}))
-
- :else
- (or
- ;; we have an existing valid pool for this database, so use it
- (get-canonical-pool cache-key details-hash true)
- ;; We don't want to end up with a bunch of simultaneous threads creating pools only to have them destroyed
- ;; the very next instant. This will cause their queries to fail. Thus we should do the usual locking here
- ;; and make sure only one thread will be creating a pool at a given instant.
- (locking pool-lock/monitor
- (or
- ;; check if another thread created the pool while we were waiting to acquire the lock
- (get-canonical-pool cache-key details-hash false)
- ;; create a new pool and add it to our cache, then return it
- (u/prog1 (create-pool! db)
- (set-canonical-pool! cache-key details-hash <>)))))))
-
- ;; already a `clojure.java.jdbc` spec map
- (map? db-or-id-or-spec)
- db-or-id-or-spec
-
- ;; invalid. Throw Exception
- :else
- (throw (ex-info (tru "Not a valid Database/Database ID/JDBC spec")
- ;; don't log the actual spec lest we accidentally expose credentials
- {:input (class db-or-id-or-spec)}))))
+(mu/defn db->pooled-connection-spec
+ "Return a JDBC connection spec that includes a c3p0 `ComboPooledDataSource` for the Lib metadata `db` of a Database.
+ These connection pools are cached so we don't create multiple ones for the same DB and connection type. The
+ connection type follows the current connection context — use [[driver.conn/with-write-connection]] to get a write
+ connection pool."
+ [db :- ::lib.schema.metadata/database]
+ (driver-api/check-allowed-access! db)
+ (let [db (driver.conn/connection-database db)
+ cache-key (pool-cache-key db)
+ ;; Calculate hash from effective details (includes write-connection merge)
+ details-hash (jdbc-spec-hash db)]
+ (driver.conn/track-connection-acquisition! (driver.conn/effective-details db))
+ (cond
+ (or (:is-audit db)
+ (and (audit-app/analytics-dev-mode)
+ (get-in db [:details :is-audit-dev])))
+ {:datasource (driver-api/data-source)}
+
+ (get-in db [:details :is-audit-dev])
+ (throw (ex-info (tru "Cannot open a connection for an analytics-dev database unless analytics dev mode is enabled.")
+ {:database-id (:id db)}))
+
+ :else
+ (or
+ ;; we have an existing valid pool for this database, so use it
+ (get-canonical-pool cache-key details-hash true)
+ ;; We don't want to end up with a bunch of simultaneous threads creating pools only to have them destroyed
+ ;; the very next instant. This will cause their queries to fail. Thus we should do the usual locking here
+ ;; and make sure only one thread will be creating a pool at a given instant.
+ (locking pool-lock/monitor
+ (or
+ ;; check if another thread created the pool while we were waiting to acquire the lock
+ (get-canonical-pool cache-key details-hash false)
+ ;; create a new pool and add it to our cache, then return it
+ (u/prog1 (create-pool! db)
+ (set-canonical-pool! cache-key details-hash <>))))))))
;;; +----------------------------------------------------------------------------------------------------------------+
;;; | metabase.driver impls |
diff --git a/src/metabase/driver/sql_jdbc/execute.clj b/src/metabase/driver/sql_jdbc/execute.clj
index f8c3ede345b..fda67c298d9 100644
--- a/src/metabase/driver/sql_jdbc/execute.clj
+++ b/src/metabase/driver/sql_jdbc/execute.clj
@@ -19,9 +19,9 @@
[metabase.driver.sql-jdbc.execute.diagnostic :as sql-jdbc.execute.diagnostic]
[metabase.driver.sql-jdbc.execute.old-impl :as sql-jdbc.execute.old]
[metabase.driver.sql-jdbc.sync.interface :as sql-jdbc.sync.interface]
+ [metabase.driver.util :as driver.u]
[metabase.lib.schema.id :as lib.schema.id]
[metabase.lib.schema.info :as lib.schema.info]
- [metabase.premium-features.core :refer [defenterprise]]
[metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.connection :as u.connection]
@@ -198,18 +198,18 @@
;;; +----------------------------------------------------------------------------------------------------------------+
(defn datasource
- "Fetch the connection pool `DataSource` associated with `db-or-id-or-spec`."
+ "Fetch the connection pool `DataSource` associated with the Lib metadata `db` of a Database."
{:added "0.35.0"}
- ^DataSource [db-or-id-or-spec]
- (:datasource (sql-jdbc.conn/db->pooled-connection-spec db-or-id-or-spec)))
+ ^DataSource [db]
+ (:datasource (sql-jdbc.conn/db->pooled-connection-spec db)))
(defn datasource-with-diagnostic-info!
- "Fetch the connection pool `DataSource` associated with `database`, while also recording diagnostic info for the
- pool. To be used in conjunction with `sql-jdbc.execute.diagnostic/capturing-diagnostic-info`."
+ "Fetch the connection pool `DataSource` associated with the Lib metadata `db` of a Database, while also recording
+ diagnostic info for the pool. To be used in conjunction with `sql-jdbc.execute.diagnostic/capturing-diagnostic-info`."
{:added "0.40.0"}
- ^DataSource [driver db-or-id]
- (let [ds (datasource db-or-id)]
- (sql-jdbc.execute.diagnostic/record-diagnostic-info-for-pool! driver (u/the-id db-or-id) ds)
+ ^DataSource [driver db]
+ (let [ds (datasource db)]
+ (sql-jdbc.execute.diagnostic/record-diagnostic-info-for-pool! driver (u/the-id db) ds)
ds))
(defn set-time-zone-if-supported!
@@ -232,11 +232,6 @@
(catch Throwable e
(log/errorf "Failed to set timezone '%s' for %s database: %s" timezone-id driver (ex-message e)))))))
-(defenterprise set-role-if-supported!
- "OSS no-op implementation of `set-role-if-supported!`."
- metabase-enterprise.impersonation.driver
- [_driver _conn _database])
-
;; TODO - since we're not running the queries in a transaction, does this make any difference at all? (metabase#40012)
(defn set-best-transaction-level!
"Set the connection transaction isolation level to the least-locking level supported by the DB. See
@@ -286,25 +281,25 @@
;; inside do-with-connection-with-options' own default impl; a raw jdbc spec has no DataSource
#_{:clj-kondo/ignore [:discouraged-var]}
(jdbc/get-connection db-or-id-or-spec)))
- ;; otherwise this is either a Database or Database ID.
- (if-let [old-method-impl (get-method
- ;; honor drivers still implementing the old connection-with-timezone
- #_{:clj-kondo/ignore [:deprecated-var]} sql-jdbc.execute.old/connection-with-timezone
- driver)]
- ;; use the deprecated impl for `connection-with-timezone` if one exists.
- (do
- (log/warnf "%s is deprecated in Metabase 0.47.0. Implement %s instead."
- 'connection-with-timezone
- 'do-with-connection-with-options)
- ;; for compatibility, make sure we pass it an actual Database instance.
- (let [database (if (integer? db-or-id-or-spec)
- (driver-api/with-metadata-provider db-or-id-or-spec
- (driver-api/database (driver-api/metadata-provider)))
- db-or-id-or-spec)]
+ ;; otherwise this is a Database ID, or the Lib metadata of a Database.
+ (let [db (cond
+ (integer? db-or-id-or-spec) (driver.u/lib-database db-or-id-or-spec)
+ (= :metadata/database (:lib/type db-or-id-or-spec)) db-or-id-or-spec
+ :else (throw (ex-info (tru "Pass a Database ID or the Lib metadata of a Database, not a Database instance.")
+ {:database-id (u/id db-or-id-or-spec)})))]
+ (if-let [old-method-impl (get-method
+ ;; honor drivers still implementing the old connection-with-timezone
+ #_{:clj-kondo/ignore [:deprecated-var]} sql-jdbc.execute.old/connection-with-timezone
+ driver)]
+ ;; use the deprecated impl for `connection-with-timezone` if one exists.
+ (do
+ (log/warnf "%s is deprecated in Metabase 0.47.0. Implement %s instead."
+ 'connection-with-timezone
+ 'do-with-connection-with-options)
(reify DataSource
(getConnection [_this]
- (old-method-impl driver database session-timezone)))))
- (datasource-with-diagnostic-info! driver db-or-id-or-spec))))
+ (old-method-impl driver db session-timezone))))
+ (datasource-with-diagnostic-info! driver db)))))
#_{:clj-kondo/ignore [:metabase/discourage-dynamic-vars]}
(def ^:private ^:dynamic ^{:added "0.47.0"} *connection-recursion-depth*
@@ -353,6 +348,15 @@
;; if we can't read the stat for some reason, fail open rather than blocking queries
false)))))
+(defn- check-checkout-error
+ "Returns `conn`, unless setting the connection impersonation role failed when it was checked out, in which case the
+ connection is closed and that error is thrown."
+ ^Connection [^Connection conn]
+ (if-let [e (sql-jdbc.conn/take-checkout-error!)]
+ (do (.close conn)
+ (throw e))
+ conn))
+
(mu/defn do-with-resolved-connection
"Execute
@@ -375,23 +379,25 @@
(throw (ex-info (tru "Too many queries are running. Please try again in a moment.")
{:type driver-api/qp.error-type.connection-pool-checkout-queue-full
:driver driver})))
- (try
- (.getConnection conn-data-source)
- (catch Throwable e
- (let [timed-out? (connection-checkout-timeout? e)]
- (when-not timed-out?
- (try
- (sql-jdbc.conn/invalidate-empty-pool! conn-data-source)
- (catch Exception e2
- (log/warnf e2 "Error closing the connection pool for database %s" (u/id db-or-id-or-spec)))))
- (throw (ex-info (if timed-out?
- (tru "Too many queries are running. Please try again in a moment.")
- (tru "Unable to connect to the database: {0}" (ex-message e)))
- {:type (if timed-out?
- driver-api/qp.error-type.connection-pool-checkout-timeout
- driver-api/qp.error-type.unable-to-acquire-connection)
- :driver driver}
- e)))))))]
+ (sql-jdbc.conn/take-checkout-error!)
+ (check-checkout-error
+ (try
+ (.getConnection conn-data-source)
+ (catch Throwable e
+ (let [timed-out? (connection-checkout-timeout? e)]
+ (when-not timed-out?
+ (try
+ (sql-jdbc.conn/invalidate-empty-pool! conn-data-source)
+ (catch Exception e2
+ (log/warnf e2 "Error closing the connection pool for database %s" (u/id db-or-id-or-spec)))))
+ (throw (ex-info (if timed-out?
+ (tru "Too many queries are running. Please try again in a moment.")
+ (tru "Unable to connect to the database: {0}" (ex-message e)))
+ {:type (if timed-out?
+ driver-api/qp.error-type.connection-pool-checkout-timeout
+ driver-api/qp.error-type.unable-to-acquire-connection)
+ :driver driver}
+ e))))))))]
(if (:keep-open? options)
(f (get-conn))
(with-open [conn ^Connection (get-conn)]
@@ -405,22 +411,12 @@
Connection."
{:added "0.47.0"}
[driver :- :keyword
- db-or-id-or-spec :- [:or
+ _db-or-id-or-spec :- [:or
::lib.schema.id/database
driver-api/schema.metadata.database
:metabase.lib.schema.common/database-details]
^Connection conn :- (driver-api/instance-of-class Connection)
{:keys [^String session-timezone write?], :as options} :- ConnectionOptions]
- (when-let [db (cond
- ;; id?
- (integer? db-or-id-or-spec)
- (driver-api/with-metadata-provider db-or-id-or-spec
- (driver-api/database (driver-api/metadata-provider)))
- ;; db?
- (u/id db-or-id-or-spec) db-or-id-or-spec
- ;; otherwise it's a spec and we can't get the db
- :else nil)]
- (set-role-if-supported! driver conn db))
(when-not (recursive-connection?)
(log/tracef "Setting default connection options with options %s" (pr-str options))
(set-best-transaction-level! driver conn)
diff --git a/src/metabase/driver/sql_jdbc/sync/describe_database.clj b/src/metabase/driver/sql_jdbc/sync/describe_database.clj
index 1fb827ce7dc..8fce5624e15 100644
--- a/src/metabase/driver/sql_jdbc/sync/describe_database.clj
+++ b/src/metabase/driver/sql_jdbc/sync/describe_database.clj
@@ -11,12 +11,12 @@
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.driver.sync :as driver.s]
[metabase.driver.util :as driver.u]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.util.connection :as u.connection]
[metabase.util.honey-sql-2 :as h2x]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
- [metabase.util.performance :refer [get-in]]
- [metabase.warehouses.schema :as warehouses.schema])
+ [metabase.util.performance :refer [get-in]])
(:import
(java.sql Connection DatabaseMetaData ResultSet)))
@@ -291,19 +291,6 @@
(assoc :is_writable (privilege-fn table :write))))))
(sql-jdbc.sync.interface/db-tables driver (.getMetaData conn) nil db-name-or-nil))))
-(defn db-or-id-or-spec->database
- "Get database instance from `db-or-id-or-spec`."
- [db-or-id-or-spec]
- (cond (driver-api/instance-of? :model/Database db-or-id-or-spec)
- db-or-id-or-spec
-
- (int? db-or-id-or-spec)
- (driver-api/with-metadata-provider db-or-id-or-spec
- (driver-api/database (driver-api/metadata-provider)))
-
- :else
- nil))
-
(mu/defn describe-database
"Default implementation of [[metabase.driver/describe-database]] for SQL JDBC drivers. Uses JDBC DatabaseMetaData.
@@ -312,18 +299,17 @@
list streams a batch at a time instead of being realized into a set up front. Building the reducible is
eager -- satisfying [[metabase.driver/do-with-resilient-connection]], which requires an eager `f` -- while
the connection is opened lazily at reduction time (with a fresh connection, outside the resilient scope)."
- [driver :- :keyword
- db-or-id-or-spec :- [:or :int ::warehouses.schema/database]]
+ [driver :- :keyword
+ database :- ::lib.schema.metadata/database]
{:tables
(reify clojure.lang.IReduceInit
(reduce [_this rf init]
(sql-jdbc.execute/do-with-connection-with-options
driver
- db-or-id-or-spec
+ database
nil
(fn [^Connection conn]
(let [schema-filter-prop (driver.u/find-schema-filters-prop driver)
- database (db-or-id-or-spec->database db-or-id-or-spec)
[inclusion-patterns
exclusion-patterns] (when (some? schema-filter-prop)
(driver.s/db-details->schema-filter-patterns (:name schema-filter-prop) database))]
diff --git a/src/metabase/driver/util.clj b/src/metabase/driver/util.clj
index 22bd6743f7d..d1c24dd17f1 100644
--- a/src/metabase/driver/util.clj
+++ b/src/metabase/driver/util.clj
@@ -8,12 +8,14 @@
[metabase.app-db.core :as mdb]
[metabase.auth-provider.core :as auth-provider]
[metabase.config.core :as config]
+ [metabase.database-routing.core :as database-routing]
[metabase.driver :as driver]
[metabase.driver.settings :as driver.settings]
[metabase.lib-be.core :as lib-be]
[metabase.lib.metadata :as lib.metadata]
[metabase.lib.metadata.protocols :as lib.metadata.protocols]
[metabase.lib.schema.id :as lib.schema.id]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.premium-features.core :as premium-features]
[metabase.query-processor.error-type :as qp.error-type]
;; the legacy QP pipeline still conveys the metadata provider via the ambient store; no MBQL 5 path yet
@@ -350,7 +352,8 @@
how long the driver may take -- callers bound it at whatever granularity suits them."
[driver feature database]
(try
- (driver/database-supports? driver feature database)
+ (database-routing/with-anonymous-router-access
+ (driver/database-supports? driver feature database))
(catch Throwable e
(log/error (u/format-color 'red "Failed to check feature '%s' for database %s: %s" (u/qualified-name feature) (:id database) (ex-message e)))
false)))
@@ -379,6 +382,11 @@
(lib-be/instance->metadata database :metadata/database)
database))
+(mu/defn lib-database :- ::lib.schema.metadata/database
+ "The Database with `database-id` from the application database, as the Lib metadata that connections take."
+ [database-id :- ::lib.schema.id/database]
+ (lib.metadata/database (lib-be/application-database-metadata-provider database-id)))
+
(mu/defn supports?
"A defensive wrapper around [[metabase.driver/database-supports?]]. It adds logging, caching, and error handling to
avoid crashing the app if this method takes a long time to execute or throws an exception. This is useful because
@@ -919,13 +927,6 @@
[driver]
(fields-hidden-by-form-marker driver "write-data-connection"))
-(defn fields-hidden-for-admin-connection
- "Returns the set of field names (strings) that should NOT appear in `admin_details` for the given `driver`.
- These are fields whose resolved `visible-if` includes `\"admin-connection\" false`, meaning they are hidden
- when the admin-connection form marker is true."
- [driver]
- (fields-hidden-by-form-marker driver "admin-connection"))
-
(defn fetch-and-incorporate-auth-provider-details
"Incorporates auth-provider responses with db-details.
diff --git a/src/metabase/embedding_rest/api/common.clj b/src/metabase/embedding_rest/api/common.clj
index a6336df84b0..1df2eada4ce 100644
--- a/src/metabase/embedding_rest/api/common.clj
+++ b/src/metabase/embedding_rest/api/common.clj
@@ -374,7 +374,7 @@
:embedded-question))
;;; Embedded viewers have no Metabase account, so there are no user attributes to route by: embedded query execution
-;;; always uses the router (primary) database. The `with-database-routing-off` wraps live here, in the shared
+;;; always uses the router (primary) database. The `with-anonymous-router-access` wraps live here, in the shared
;;; execution helpers that every /api/embed and /api/preview_embed endpoint funnels through, so that individual
;;; endpoints cannot forget them. (For preview_embed this also means the preview shows what the published embed will
;;; show, rather than routing via the previewing admin's own user attribute.)
@@ -389,7 +389,7 @@
{:pre [(map? card) (pos-int? (:id card)) (u/maybe? map? embedding-params) (map? token-params) (map? query-params)]}
(let [merged-slug->value (validate-and-merge-params embedding-params token-params (normalize-query-params query-params))
parameters (apply-slug->value (resolve-card-parameters card) merged-slug->value)]
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(m/mapply api.public/process-query-for-card-with-id
card export-format parameters
:context (get-embed-card-context export-format)
@@ -430,7 +430,7 @@
"Like [[metabase.tiles.api/process-tiles-query-for-card]], but takes a pre-loaded Card entity and runs with database
routing off (see above). Used by the embed tiles endpoints. Returns a Ring response."
[card parameters zoom x y lat-field lon-field]
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(api.tiles/process-tiles-query-for-card card parameters zoom x y lat-field lon-field)))
;;; -------------------------- Dashboard Fns used by both /api/embed and /api/preview_embed --------------------------
@@ -499,7 +499,7 @@
(map? token-params) (map? query-params)]}
(let [slug->value (validate-and-merge-params embedding-params token-params (normalize-query-params query-params))
parameters (resolve-dashboard-parameters dashboard slug->value)]
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(api.public/process-query-for-dashcard
:dashboard dashboard
:card card
@@ -516,7 +516,7 @@
entities and runs with database routing off (see the comment above [[process-query-for-card-with-params]]). Used by
the embed tiles endpoints. Callers select each entity exactly once and thread it here. Returns a Ring response."
[dashboard dashcard card parameters zoom x y lat-field lon-field]
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(api.public/process-tiles-query-for-dashcard dashboard dashcard card
parameters zoom x y lat-field lon-field)))
@@ -541,7 +541,7 @@
{:status-code 400})))
(try
;; guest embeds always use the router (primary) database, never a routed destination
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(request/as-admin
(queries/card-param-values card param-key search-prefix
(queries/card-param-constraints
@@ -588,7 +588,7 @@
(pr-str searched-param-slug))
{:status-code 400})))
(try
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(request/as-admin
(queries/card-param-remapped-value card param-key value
(queries/card-param-constraints
@@ -650,7 +650,7 @@
;; ok, at this point we can run the query
(let [merged-id-params (param-values-merged-params id->slug slug->id embedding-params slug-token-params id-query-params)]
(try
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(request/as-admin
(parameters.dashboard/param-values dashboard searched-param-id merged-id-params prefix)))
(catch Throwable e
@@ -707,6 +707,6 @@
{:status-code 400})))
(let [constraints (-> (param-values-merged-params id->slug slug->id embedding-params slug-token-params {})
(select-keys locked-param-ids))]
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(request/as-admin
(parameters.dashboard/dashboard-param-remapped-value dashboard param-key value constraints)))))))
diff --git a/src/metabase/indexes/reconcile.clj b/src/metabase/indexes/reconcile.clj
index 50c26d97855..8f3dbfe3e6f 100644
--- a/src/metabase/indexes/reconcile.clj
+++ b/src/metabase/indexes/reconcile.clj
@@ -5,6 +5,7 @@
(:require
[metabase.driver :as driver]
[metabase.indexes.schema :as indexes.schema]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.util :as u]
[metabase.util.malli :as mu]
[metabase.util.malli.registry :as mr]
@@ -138,7 +139,7 @@
(mu/defn fetch-warehouse-indexes :- [:sequential :map]
"Physical indexes on `table-name` (`schema`) in `database` via `driver/fetch-table-indexes`. Throws when the
warehouse can't be read, so an empty result always means no indexes."
- [database :- :metabase.warehouses.schema/database
+ [database :- ::lib.schema.metadata/database
schema :- [:maybe :string]
table-name :- :string]
(vec (driver/fetch-table-indexes (:engine database) database schema table-name)))
diff --git a/src/metabase/indexes_rest/api.clj b/src/metabase/indexes_rest/api.clj
index 0f5481976de..c93810a0881 100644
--- a/src/metabase/indexes_rest/api.clj
+++ b/src/metabase/indexes_rest/api.clj
@@ -7,6 +7,7 @@
(:require
[metabase.api.common :as api]
[metabase.api.macros :as api.macros]
+ [metabase.driver.util :as driver.u]
[metabase.indexes-rest.db :as indexes-rest.db]
[metabase.indexes.models.table-index :as table-index]
[metabase.indexes.reconcile :as reconcile]
@@ -72,7 +73,9 @@
[_route-params
{:keys [transform-id]} :- [:map {:closed true} [:transform-id ms/PositiveInt]]]
(let [transform (api/read-check :model/Transform transform-id)
- database (api/check-404 (indexes-rest.db/database (transforms-base.i/target-db-id transform)))
+ database-id (transforms-base.i/target-db-id transform)
+ _ (api/check-404 (some-> database-id indexes-rest.db/database-exists?))
+ database (driver.u/lib-database database-id)
{:keys [schema] table-name :name} (:target transform)
managed (table-index/select-for-transform transform-id)
[warehouse warehouse-error]
diff --git a/src/metabase/indexes_rest/db.clj b/src/metabase/indexes_rest/db.clj
index 0f772b75ed2..83484c14f21 100644
--- a/src/metabase/indexes_rest/db.clj
+++ b/src/metabase/indexes_rest/db.clj
@@ -8,10 +8,10 @@
[metabase.util.malli.schema :as ms]
[toucan2.core :as t2]))
-(mu/defn database
- "The Database with `database-id`, or nil."
+(mu/defn database-exists?
+ "Whether a Database with `database-id` exists."
[database-id :- ::lib.schema.id/database]
- (t2/select-one :model/Database database-id))
+ (t2/exists? :model/Database :id database-id))
(mu/defn table-index
"The TableIndex with `id`, or nil."
diff --git a/src/metabase/lib/schema/metadata.cljc b/src/metabase/lib/schema/metadata.cljc
index 0b17dc7bbd9..ca0e3e76ce9 100644
--- a/src/metabase/lib/schema/metadata.cljc
+++ b/src/metabase/lib/schema/metadata.cljc
@@ -1108,7 +1108,6 @@
[:timezone {:optional true} [:maybe :string]]
[:default-schema {:optional true} [:maybe :string]]
[:write-data-details {:optional true} [:maybe ::lib.schema.common/database-details]]
- [:admin-details {:optional true} [:maybe ::lib.schema.common/database-details]]
[:router-database-id {:optional true} [:maybe ::lib.schema.id/database]]
[:auto-run-queries {:optional true} [:maybe :boolean]]
[:cache-field-values-schedule {:optional true} [:maybe :string]]
diff --git a/src/metabase/lib_be/metadata/jvm.clj b/src/metabase/lib_be/metadata/jvm.clj
index 4d8745a284b..693414bcec5 100644
--- a/src/metabase/lib_be/metadata/jvm.clj
+++ b/src/metabase/lib_be/metadata/jvm.clj
@@ -165,13 +165,38 @@
(t2/resolve-model :model/Database) ; for side-effects
model)
+(methodical/defmethod t2.query/apply-kv-arg [#_model :metadata/database
+ #_resolved-query clojure.lang.IPersistentMap
+ #_k :default]
+ "Qualify unqualified kv-args when fetching a `:metadata/database`."
+ [model honeysql k v]
+ (let [k (if (not (qualified-key? k))
+ (keyword "database" (name k))
+ k)]
+ (next-method model honeysql k v)))
+
(methodical/defmethod t2.pipeline/build [#_query-type :toucan.query-type/select.*
#_model :metadata/database
#_resolved-query clojure.lang.IPersistentMap]
[query-type model parsed-args honeysql]
- (merge (next-method query-type model parsed-args honeysql)
- {:select [:id :engine :name :dbms_version :settings :is_audit :is_attached_dwh :details :write_data_details
- :admin_details :timezone :default_schema :router_database_id]}))
+ (merge
+ (next-method query-type model parsed-args honeysql)
+ {:select [:database/id
+ :database/engine
+ :database/name
+ :database/dbms_version
+ :database/settings
+ :database/is_audit
+ :database/is_attached_dwh
+ :database/details
+ :database/write_data_details
+ :database/timezone
+ :database/default_schema
+ :database/router_database_id
+ [:router/user_attribute :router_user_attribute]]
+ :from [[(t2/table-name :model/Database) :database]]
+ :left-join [[:db_router :router]
+ [:= :router/database_id :database/id]]}))
(t2/define-after-select :metadata/database
[database]
diff --git a/src/metabase/model_persistence/api.clj b/src/metabase/model_persistence/api.clj
index ca6b8ded625..79cd5139515 100644
--- a/src/metabase/model_persistence/api.clj
+++ b/src/metabase/model_persistence/api.clj
@@ -252,7 +252,7 @@
;; todo: some other response if already persisted?
api/generic-204-no-content
(driver.conn/with-write-connection
- (let [[success? error] (ddl.i/check-can-persist database)
+ (let [[success? error] (ddl.i/check-can-persist (driver.u/lib-database id))
schema (ddl.i/schema-name database (system/site-uuid))]
(if success?
;; do secrets require special handling to not clobber them or mess up encryption?
diff --git a/src/metabase/model_persistence/task/persist_refresh.clj b/src/metabase/model_persistence/task/persist_refresh.clj
index e211e8b9949..b6b9756ee84 100644
--- a/src/metabase/model_persistence/task/persist_refresh.clj
+++ b/src/metabase/model_persistence/task/persist_refresh.clj
@@ -8,6 +8,7 @@
[medley.core :as m]
[metabase.driver :as driver]
[metabase.driver.ddl.interface :as ddl.i]
+ [metabase.driver.util :as driver.u]
[metabase.events.core :as events]
[metabase.lib.schema.id :as lib.schema.id]
[metabase.model-persistence.db :as model-persistence.db]
@@ -131,7 +132,9 @@
"Seam for tests to pass in specific deletables to drop."
[refresher deletables]
(when (seq deletables)
- (let [db-id->db (m/index-by :id (model-persistence.db/databases (map :database_id deletables)))
+ (let [db-id->db (into {}
+ (map (fn [{:keys [id]}] [id (driver.u/lib-database id)]))
+ (model-persistence.db/databases (distinct (map :database_id deletables))))
unpersist-fn (fn []
(reduce (fn [stats persisted-info]
;; Since this could be long running, double check state just before deleting
@@ -183,7 +186,7 @@
[database-id refresher trigger]
(log/infof "Starting persisted model refresh task for Database %s." database-id)
(persisted-info/ready-unpersisted-models! database-id)
- (let [database (model-persistence.db/database database-id)
+ (let [database (driver.u/lib-database database-id)
persisted (refreshable-models database-id)
thunk (fn []
(reduce (partial refresh-with-stats! refresher database)
@@ -197,8 +200,8 @@
"Refresh an individual model based on [[PersistedInfo]]."
[persisted-info-id refresher trigger]
(let [persisted-info (model-persistence.db/persisted-info persisted-info-id)
- database (when persisted-info
- (model-persistence.db/database (:database_id persisted-info)))]
+ database (when (and persisted-info (model-persistence.db/database (:database_id persisted-info)))
+ (driver.u/lib-database (:database_id persisted-info)))]
(if (and persisted-info database)
(do
(save-task-history! "persist-refresh" (u/the-id database)
diff --git a/src/metabase/query_processor/setup.clj b/src/metabase/query_processor/setup.clj
index ebf6247a935..f38630e8ce3 100644
--- a/src/metabase/query_processor/setup.clj
+++ b/src/metabase/query_processor/setup.clj
@@ -151,13 +151,12 @@
:else
;; `:database` here is whatever the user submitted, so a destination at this point is a direct
- ;; request to query one; reject it before building a provider. Legitimate routing never passes
+ ;; request to query one; reject it before running anything. Legitimate routing never passes
;; through this branch: when a router query is routed to a destination, that swap happens later,
;; in the execution middleware, binding its own metadata provider.
- (do
- (database-routing/check-allowed-access! (:database query))
- (qp.store/with-metadata-provider (:database query)
- (f (maybe-attach-metadata-provider-to-query query)))))))
+ (qp.store/with-metadata-provider (:database query)
+ (database-routing/check-allowed-access! (lib.metadata/database (qp.store/metadata-provider)))
+ (f (maybe-attach-metadata-provider-to-query query))))))
(mu/defn- do-with-driver :- fn?
[f :- fn?]
diff --git a/src/metabase/sample_data/impl.clj b/src/metabase/sample_data/impl.clj
index e87f0bd5d7e..05355115997 100644
--- a/src/metabase/sample_data/impl.clj
+++ b/src/metabase/sample_data/impl.clj
@@ -8,6 +8,7 @@
[metabase.plugins.core :as plugins]
[metabase.sample-data.db :as sample-data.db]
[metabase.sync.core :as sync]
+ [metabase.util :as u]
[metabase.util.files :as u.files]
[metabase.util.i18n :refer [trs]]
[metabase.util.log :as log]
@@ -107,7 +108,7 @@
(sample-data.db/database (first (sample-data.db/set-sample-database-details! details)))
(sample-data.db/insert-sample-database! sample-database-name details engine))]
(log/debug "Syncing Sample Database...")
- (sync/sync-database! db))
+ (sync/sync-database! (u/the-id db)))
(log/debug "Finished adding Sample Database.")
(catch Throwable e
(log/errorf "Failed to load sample database: %s" (ex-message e)))))
@@ -162,7 +163,7 @@
;; permission checks (e.g. schema visibility in the data picker) stop counting them. Delete+reinsert would churn
;; ids for a rename that doesn't change any permission value.
(sample-data.db/set-database-table-permissions-schema-name! (:id old-sample-db) (table-schema-for-engine engine)))
- (sync/sync-database! (sample-data.db/database (:id old-sample-db)))))
+ (sync/sync-database! (:id old-sample-db))))
(defn update-sample-database-if-needed!
"Reconcile the existing sample database with the bundled one. When the bundled engine changed
diff --git a/src/metabase/secrets/models/secret.clj b/src/metabase/secrets/models/secret.clj
index b44cb453013..8aaeac96bd7 100644
--- a/src/metabase/secrets/models/secret.clj
+++ b/src/metabase/secrets/models/secret.clj
@@ -278,19 +278,18 @@
(defn delete-orphaned-secrets!
"Delete Secret instances from the app DB, that will become orphaned when `database` is deleted. For now, this will
- simply delete any Secret whose ID appears in the details blobs (`:details`, `:write_data_details`, and
- `:admin_details`), since every Secret instance that is currently created is exclusively associated with a single
- Database.
+ simply delete any Secret whose ID appears in the details blobs (`:details` and `:write_data_details`), since every
+ Secret instance that is currently created is exclusively associated with a single Database.
In the future, if/when we allow arbitrary association of secret instances to database instances, this will need to
change and become more complicated (likely by consulting a many-to-many join table)."
- [{:keys [id details write_data_details admin_details] :as database}]
+ [{:keys [id details write_data_details] :as database}]
(when-let [possible-secret-prop-names (seq (keys (secret-conn-props-by-name (driver.u/database->driver database))))]
(doseq [secret-id (reduce (fn [acc prop-name]
(let [id-kw (->id-kw prop-name)]
(into acc
(keep #(get % id-kw))
- [details write_data_details admin_details])))
+ [details write_data_details])))
#{}
possible-secret-prop-names)]
(log/infof "Deleting secret ID %s from app DB because the owning database (%s) is being deleted" secret-id id)
@@ -329,7 +328,7 @@
This is a transformation on `:model/Database` `to-json`
Fetches the stored secret and fills in `-path` `-options` `-value` for each secret property.
- Operates on `:details`, `:write_data_details`, and `:admin_details`."
+ Operates on `:details` and `:write_data_details`."
[database]
(let [driver (if-let [engine (:engine database)]
(keyword engine)
@@ -339,8 +338,7 @@
;; Very low-level operation here, so not using driver.conn/* utils:
(-> database
(m/update-existing :details hydrate)
- (m/update-existing :write_data_details hydrate)
- (m/update-existing :admin_details hydrate))))
+ (m/update-existing :write_data_details hydrate))))
(defn clean-secret-properties-from-details
"Ensures that all possible secret property values are removed from `:details`.
@@ -355,15 +353,14 @@
(apply dissoc db-details (vals (->possible-secret-property-names conn-prop-nm))))))
(defn clean-secret-properties-from-database
- "Ensures that all possible secret property values are removed from `:details`, `:write_data_details`, and
- `:admin_details`. This is a transformation on `:model/Database` `results-transform`."
+ "Ensures that all possible secret property values are removed from `:details` and `:write_data_details`. This is a
+ transformation on `:model/Database` `results-transform`."
[database]
(let [clean #(clean-secret-properties-from-details % (keyword (:engine database)))]
;; Very low-level operation here, so not using driver.conn/* utils:
(-> database
(m/update-existing :details clean)
- (m/update-existing :write_data_details clean)
- (m/update-existing :admin_details clean))))
+ (m/update-existing :write_data_details clean))))
(defn- handle-secrets-for-details-key
"Process secret-type connection properties in `details-key` of `database`, converting raw secret
@@ -399,7 +396,7 @@
database))
(defn handle-incoming-client-secrets!
- "Converts incoming secret values in `:details`, `:write_data_details`, and `:admin_details` into Secrets.
+ "Converts incoming secret values in `:details` and `:write_data_details` into Secrets.
This is a transformation on `:model/Database` `before-insert` and `before-update`.
Only the Secret id should be stored in the details maps. All other secret props should be cleared.
@@ -419,5 +416,4 @@
[database]
(-> database
(handle-secrets-for-details-key :details)
- (handle-secrets-for-details-key :write_data_details)
- (handle-secrets-for-details-key :admin_details)))
+ (handle-secrets-for-details-key :write_data_details)))
diff --git a/src/metabase/sync/analyze.clj b/src/metabase/sync/analyze.clj
index 7934697c34f..e87f1f70dff 100644
--- a/src/metabase/sync/analyze.clj
+++ b/src/metabase/sync/analyze.clj
@@ -4,6 +4,8 @@
like running MBQL queries and fetching values to do things like determine Table row counts
and infer field semantic types."
(:require
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.schema.id :as lib.schema.id]
[metabase.sync.analyze.classify :as classify]
[metabase.sync.analyze.data-sensitivity :as sync.data-sensitivity]
[metabase.sync.analyze.fingerprint :as sync.fingerprint]
@@ -131,26 +133,29 @@
driver supports, but includes things like cardinality testing and table row counting. This also updates the
`:last_analyzed` value for each affected Field. Subject to the `disable-auto-sync` setting; for an
explicit user-requested analysis use [[analyze-db-explicit!]]."
- [database :- i/DatabaseInstance]
- (sync-util/sync-operation :analyze database (format "Analyze data for %s" (sync-util/name-for-logging database))
- (analyze-db!* database)))
+ [database-id :- ::lib.schema.id/database]
+ (let [database (driver.u/lib-database database-id)]
+ (sync-util/sync-operation :analyze database (format "Analyze data for %s" (sync-util/name-for-logging database))
+ (analyze-db!* database))))
(mu/defn analyze-db-explicit!
"Like [[analyze-db!]], but for an explicit, user-requested sync (e.g. the Sync-now button): runs even
when the `disable-auto-sync` setting is enabled."
- [database :- i/DatabaseInstance]
- (sync-util/explicit-sync-operation :analyze database (format "Analyze data for %s" (sync-util/name-for-logging database))
- (analyze-db!* database)))
+ [database-id :- ::lib.schema.id/database]
+ (let [database (driver.u/lib-database database-id)]
+ (sync-util/explicit-sync-operation :analyze database (format "Analyze data for %s" (sync-util/name-for-logging database))
+ (analyze-db!* database))))
(mu/defn refingerprint-db!
"Refingerprint a subset of tables in a given `database`. This will re-fingerprint tables up to a threshold amount of
[[fingerprint/max-refingerprint-field-count]]."
- [database :- i/DatabaseInstance]
- (sync-util/sync-operation :refingerprint database (format "Refingerprinting tables for %s" (sync-util/name-for-logging database))
- (let [log-fn (fn [step table]
- (log/info (u/format-color 'blue "%s Analyzed %s" step (sync-util/name-for-logging table))))]
- (sync-util/run-sync-operation "refingerprint database"
- database
- [(sync-util/create-sync-step "refingerprinting fields"
- #(sync.fingerprint/refingerprint-fields-for-db! % log-fn)
- fingerprint-fields-summary)]))))
+ [database-id :- ::lib.schema.id/database]
+ (let [database (driver.u/lib-database database-id)]
+ (sync-util/sync-operation :refingerprint database (format "Refingerprinting tables for %s" (sync-util/name-for-logging database))
+ (let [log-fn (fn [step table]
+ (log/info (u/format-color 'blue "%s Analyzed %s" step (sync-util/name-for-logging table))))]
+ (sync-util/run-sync-operation "refingerprint database"
+ database
+ [(sync-util/create-sync-step "refingerprinting fields"
+ #(sync.fingerprint/refingerprint-fields-for-db! % log-fn)
+ fingerprint-fields-summary)])))))
diff --git a/src/metabase/sync/analyze/data_sensitivity.clj b/src/metabase/sync/analyze/data_sensitivity.clj
index 05d192874b8..67151105c52 100644
--- a/src/metabase/sync/analyze/data_sensitivity.clj
+++ b/src/metabase/sync/analyze/data_sensitivity.clj
@@ -8,6 +8,8 @@
change can reach fields that already carry a category."
(:require
[metabase.analyze.core :as analyze]
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.schema.id :as lib.schema.id]
[metabase.sync.db :as sync.db]
[metabase.sync.interface :as i]
[metabase.sync.settings :as sync.settings]
@@ -15,7 +17,6 @@
[metabase.util :as u]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
- [toucan2.core :as t2]
[toucan2.realize :as t2.realize]))
(set! *warn-on-reflection* true)
@@ -101,26 +102,43 @@
(sync.db/tables-by-schema-and-name-reducible table-ids))))))
(mu/defn reset-data-sensitivity! :- :int
- "REPL entry point: clear every classifier-written `data_sensitivity` in a Database or a single Table so the next
+ "REPL entry point: clear every classifier-written `data_sensitivity` in the Database with `database-id` so the next
scan recomputes them, including fields that carry a category and would otherwise never be reselected. Labels with
a value in the `FieldUserSettings` mirror are human-set and untouched. Returns the number of fields cleared."
- [database-or-table :- [:or i/DatabaseInstance i/TableInstance]]
- (let [n (case (t2/model database-or-table)
- :model/Table (sync.db/reset-classifier-data-sensitivity-for-table! (u/the-id database-or-table))
- :model/Database (sync.db/reset-classifier-data-sensitivity-for-database! (u/the-id database-or-table)))]
- (log/infof "Data sensitivity reset %d classifier-written labels in %s" n (sync-util/name-for-logging database-or-table))
+ [database-id :- ::lib.schema.id/database]
+ (let [n (sync.db/reset-classifier-data-sensitivity-for-database! database-id)]
+ (log/infof "Data sensitivity reset %d classifier-written labels in %s"
+ n (sync-util/name-for-logging (driver.u/lib-database database-id)))
n))
+(mu/defn reset-table-data-sensitivity! :- :int
+ "Like [[reset-data-sensitivity!]], for a single `table`."
+ [table :- i/TableInstance]
+ (let [n (sync.db/reset-classifier-data-sensitivity-for-table! (u/the-id table))]
+ (log/infof "Data sensitivity reset %d classifier-written labels in %s" n (sync-util/name-for-logging table))
+ n))
+
+(def ^:private ScanDataSensitivityOptions
+ [:maybe [:map {:closed true}
+ [:force? {:optional true} [:maybe :boolean]]
+ [:reset? {:optional true} [:maybe :boolean]]]])
+
(mu/defn scan-data-sensitivity! :- Stats
- "Scan `database-or-table` for data sensitivity regardless of the setting; `:force?` rescans `:PUBLIC` Fields and
- `:reset?` clears the classifier's labels first."
- [database-or-table :- [:or i/DatabaseInstance i/TableInstance]
- & {:keys [force? reset?]} :- [:maybe [:map {:closed true}
- [:force? {:optional true} [:maybe :boolean]]
- [:reset? {:optional true} [:maybe :boolean]]]]]
- (let [reset (when reset? (reset-data-sensitivity! database-or-table))
- stats (case (t2/model database-or-table)
- :model/Table (scan-table! database-or-table :force? force? :ignore-setting? true)
- :model/Database (scan-fields-for-db! database-or-table (constantly nil) :force? force? :ignore-setting? true))]
+ "Scan the Database with `database-id` for data sensitivity regardless of the setting; `:force?` rescans `:PUBLIC`
+ Fields and `:reset?` clears the classifier's labels first."
+ [database-id :- ::lib.schema.id/database
+ & {:keys [force? reset?]} :- ScanDataSensitivityOptions]
+ (let [reset (when reset? (reset-data-sensitivity! database-id))
+ stats (scan-fields-for-db! (driver.u/lib-database database-id) (constantly nil)
+ :force? force? :ignore-setting? true)]
+ (cond-> stats
+ reset (assoc :fields-reset reset))))
+
+(mu/defn scan-table-data-sensitivity! :- Stats
+ "Like [[scan-data-sensitivity!]], for a single `table`."
+ [table :- i/TableInstance
+ & {:keys [force? reset?]} :- ScanDataSensitivityOptions]
+ (let [reset (when reset? (reset-table-data-sensitivity! table))
+ stats (scan-table! table :force? force? :ignore-setting? true)]
(cond-> stats
reset (assoc :fields-reset reset))))
diff --git a/src/metabase/sync/analyze/fingerprint.clj b/src/metabase/sync/analyze/fingerprint.clj
index 58be557674c..446731cb7bf 100644
--- a/src/metabase/sync/analyze/fingerprint.clj
+++ b/src/metabase/sync/analyze/fingerprint.clj
@@ -17,7 +17,6 @@
[metabase.util.malli.registry :as mr]
[metabase.util.malli.schema :as ms]
[metabase.warehouse-schema.models.field :as field]
- [metabase.warehouse-schema.models.table :as table]
[redux.core :as redux]))
(defn incomplete-analysis-kvs
@@ -90,7 +89,7 @@
(update count-info :updated-fingerprints inc))))
(empty-stats-map (count fingerprints))
(map vector fields fingerprints)))))
- driver (driver.u/database->driver (table/database table))
+ driver (driver.u/database->driver (:db_id table))
opts {:truncation-size *truncation-size*}]
(tracing/with-span :sync "sync.fingerprint.sample" {:sync/table (:name table) :sync/field-count (count fields)}
(driver/table-rows-sample driver table fields rff opts))))
diff --git a/src/metabase/sync/api/notify.clj b/src/metabase/sync/api/notify.clj
index 828de8252d7..33f671cbccc 100644
--- a/src/metabase/sync/api/notify.clj
+++ b/src/metabase/sync/api/notify.clj
@@ -38,14 +38,14 @@
(let [schema? (when scan (#{"schema" :schema} scan))
table-sync-fn (if schema? sync-metadata/sync-table-metadata! sync/sync-table!)
db-sync-fn (if schema? sync-metadata/sync-db-metadata! sync/sync-database!)]
- (api/let-404 [database (sync.db/database id)]
- (let [table (cond
- table_id (api/check-404 (sync.db/table-in-database id (int table_id)))
- table_name (api/check-404 (sync.db/table-by-name id table_name)))]
- (cond-> (future (if table
- (table-sync-fn table)
- (db-sync-fn database)))
- synchronous? deref))))
+ (api/check-404 (sync.db/database-exists? id))
+ (let [table (cond
+ table_id (api/check-404 (sync.db/table-in-database id (int table_id)))
+ table_name (api/check-404 (sync.db/table-by-name id table_name)))]
+ (cond-> (future (if table
+ (table-sync-fn table)
+ (db-sync-fn id)))
+ synchronous? deref)))
{:success true})
(defn- without-stacktrace [^Throwable throwable]
@@ -53,10 +53,11 @@
(.setStackTrace (make-array StackTraceElement 0))))
(defn- find-and-sync-new-table
- [database table-name schema-name]
- (let [driver (driver.u/database->driver database)
- table {:name table-name
- :schema schema-name}]
+ [database-id table-name schema-name]
+ (let [database (driver.u/lib-database database-id)
+ driver (driver.u/database->driver database)
+ table {:name table-name
+ :schema schema-name}]
(if (driver/table-exists? driver database table)
(let [created (sync-tables/create-or-reactivate-table! database table)]
(doto created
@@ -88,16 +89,16 @@
[:table_name {:optional true} [:maybe ms/NonBlankString]]
[:schema_name {:optional true} [:maybe string?]]
[:synchronous? {:default false} [:maybe ms/BooleanValue]]]]
- (api/let-404 [database (sync.db/attached-dwh-database)]
+ (api/let-404 [database-id (sync.db/attached-dwh-database-id)]
(if (str/blank? table_name)
- (cond-> (future (sync-metadata/sync-db-metadata! database))
+ (cond-> (future (sync-metadata/sync-db-metadata! database-id))
synchronous? deref)
- (if-let [table (sync.db/table-by-schema-and-name (:id database) schema_name table_name)]
+ (if-let [table (sync.db/table-by-schema-and-name database-id schema_name table_name)]
(cond-> (future (sync-metadata/sync-table-metadata! table))
synchronous? deref)
;; find and sync is always synchronous. And we want it to be so since the "can't find this table" error is
;; rather informative. If it's on a future we won't see it.
- (find-and-sync-new-table database table_name schema_name))))
+ (find-and-sync-new-table database-id table_name schema_name))))
{:success true})
;; TODO (Cam 2025-11-25) please add a response schema to this API endpoint, it makes it easier for our customers to
@@ -113,12 +114,12 @@
{:keys [schema_name table_name]} :- [:map {:closed true}
[:schema_name ms/NonBlankString]
[:table_name ms/NonBlankString]]]
- (api/let-404 [database (sync.db/database id)]
- (if-not (sync.db/table-by-schema-and-name id schema_name table_name)
- (find-and-sync-new-table database table_name schema_name)
- (throw (without-stacktrace
- (ex-info (trs "Table ''{0}.{1}'' already exists"
- schema_name table_name)
- {:status-code 400
- :schema_name schema_name
- :table_name table_name}))))))
+ (api/check-404 (sync.db/database-exists? id))
+ (if-not (sync.db/table-by-schema-and-name id schema_name table_name)
+ (find-and-sync-new-table id table_name schema_name)
+ (throw (without-stacktrace
+ (ex-info (trs "Table ''{0}.{1}'' already exists"
+ schema_name table_name)
+ {:status-code 400
+ :schema_name schema_name
+ :table_name table_name})))))
diff --git a/src/metabase/sync/db.clj b/src/metabase/sync/db.clj
index f1aa269e146..9225daf11a1 100644
--- a/src/metabase/sync/db.clj
+++ b/src/metabase/sync/db.clj
@@ -35,10 +35,25 @@
[database-id :- ::lib.schema.id/database]
(t2/select-one :model/Database :id database-id))
-(mu/defn attached-dwh-database
- "The attached data warehouse Database, or nil."
+(mu/defn database-exists?
+ "Whether a Database with `database-id` exists."
+ [database-id :- ::lib.schema.id/database]
+ (t2/exists? :model/Database :id database-id))
+
+(mu/defn database-initial-sync-status
+ "The `initial_sync_status` of the Database with `database-id`."
+ [database-id :- ::lib.schema.id/database]
+ (t2/select-one-fn :initial_sync_status :model/Database :id database-id))
+
+(mu/defn database-sample?
+ "Whether the Database with `database-id` is the Sample Database."
+ [database-id :- ::lib.schema.id/database]
+ (boolean (t2/select-one-fn :is_sample :model/Database :id database-id)))
+
+(mu/defn attached-dwh-database-id
+ "The ID of the attached data warehouse Database, or nil."
[]
- (t2/select-one :model/Database :is_attached_dwh true))
+ (t2/select-one-pk :model/Database :is_attached_dwh true))
(mu/defn database-stub?
"Whether the Database with `database-id` is a stub."
diff --git a/src/metabase/sync/events/sync_database.clj b/src/metabase/sync/events/sync_database.clj
index 541c6586c3e..21c35a4deed 100644
--- a/src/metabase/sync/events/sync_database.clj
+++ b/src/metabase/sync/events/sync_database.clj
@@ -30,8 +30,8 @@
(try
;; only do the 'full' sync if this is a "full sync" database. Otherwise just do metadata sync only
(if (:is_full_sync database)
- (sync/sync-database! database)
- (sync-metadata/sync-db-metadata! database))
+ (sync/sync-database! (u/the-id database))
+ (sync-metadata/sync-db-metadata! (u/the-id database)))
(catch Throwable e
(log/errorf "Error syncing Database %s: %s" (u/the-id database) (ex-message e)))))))
(catch Throwable e
diff --git a/src/metabase/sync/fetch_metadata.clj b/src/metabase/sync/fetch_metadata.clj
index 16c23c20d44..b069a45cc46 100644
--- a/src/metabase/sync/fetch_metadata.clj
+++ b/src/metabase/sync/fetch_metadata.clj
@@ -7,7 +7,6 @@
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.sync :as sql-jdbc.sync]
[metabase.driver.util :as driver.u]
- [metabase.lib-be.core :as lib-be]
[metabase.sync.db :as sync.db]
[metabase.sync.interface :as i]
[metabase.sync.util :as sync-util]
@@ -108,9 +107,7 @@
(log-if-error "fk-metadata"
(let [driver (driver.u/database->driver database)]
(when (driver.u/supports? driver :metadata/key-constraints database)
- (cond->> (driver/describe-fks driver
- (lib-be/instance->metadata database :metadata/database)
- args)
+ (cond->> (driver/describe-fks driver database args)
;; This is a workaround for the fact that [[mu/defn]] can't check reducible collections yet
(mu.fn/instrument-ns? *ns*)
(eduction (map #(mu.fn/validate-output {} i/FKMetadataEntry %))))))))
diff --git a/src/metabase/sync/field_values.clj b/src/metabase/sync/field_values.clj
index d9f9c02e746..bba86d03a01 100644
--- a/src/metabase/sync/field_values.clj
+++ b/src/metabase/sync/field_values.clj
@@ -2,8 +2,10 @@
"Logic for updating FieldValues for fields in a database."
(:require
[java-time.api :as t]
+ [metabase.database-routing.core :as database-routing]
[metabase.driver :as driver]
[metabase.driver.util :as driver.u]
+ [metabase.lib.schema.id :as lib.schema.id]
[metabase.sync.db :as sync.db]
[metabase.sync.interface :as i]
[metabase.sync.settings :as sync.settings]
@@ -58,9 +60,8 @@
but `run-distinct-batch` runs as a native query that bypasses that middleware.
Tables that fail any check fall back to the per-field path."
- [table]
- (let [database (sync.db/database (:db_id table))
- engine (:engine database)]
+ [database table]
+ (let [engine (:engine database)]
(and (isa? driver/hierarchy engine :sql)
(driver.u/supports? engine :nested-queries database)
(not (:database_require_filter table)))))
@@ -128,10 +129,10 @@
hand-rolled HoneySQL CAST does not).
Returns a counts map of outcomes."
- [table fields fvs-map]
+ [database table fields fvs-map]
(when (seq fields)
(let [{nested true plain false} (group-by #(boolean (:parent_id %)) fields)
- batch? (can-batch-distinct? table)
+ batch? (can-batch-distinct? database table)
batch-fields (when batch? plain)
per-field-fields (concat nested (when-not batch? plain))
{:keys [results queries failed-fields]}
@@ -163,10 +164,11 @@
(let [eligible (filter field-values/field-should-have-field-values? fields)]
(when (seq eligible)
(let [fvs-map (field-values/batched-get-latest-full-field-values (map u/the-id eligible))
- by-table (group-by :table_id eligible)]
+ by-table (group-by :table_id eligible)
+ database (memoize driver.u/lib-database)]
(transduce (map (fn [[table-id table-fields]]
(let [table (sync.db/table table-id)]
- (sync-fields-for-table! table table-fields fvs-map))))
+ (sync-fields-for-table! (database (:db_id table)) table table-fields fvs-map))))
(completing (partial merge-with +))
empty-counts
by-table)))))
@@ -175,50 +177,55 @@
"Update the FieldValues for all Fields (as needed) for `table`.
SQL drivers use a single UNION ALL query covering every list-eligible field with an active
FieldValues row. Non-SQL drivers fall back to sequential per-field DISTINCT queries."
- [table :- i/TableInstance]
- (let [limit (sync.settings/scan-max-fields-per-table)
- scanned (table->fields-to-scan table (inc limit))
- _ (when (> (count scanned) limit)
- (warn-too-many-fields! table limit))
- all-fields (take limit scanned)
- {to-sync true
- to-clear false} (group-by #(boolean (field-values/field-should-have-field-values? %))
- all-fields)
- clear-counts (reduce (fn [counts field]
- (let [result (sync-util/with-error-handling
- (format "Error clearing field values for %s"
+ ([table :- i/TableInstance]
+ (update-field-values-for-table! (driver.u/lib-database (:db_id table)) table))
+
+ ([database :- i/DatabaseInstance
+ table :- i/TableInstance]
+ (let [limit (sync.settings/scan-max-fields-per-table)
+ scanned (table->fields-to-scan table (inc limit))
+ _ (when (> (count scanned) limit)
+ (warn-too-many-fields! table limit))
+ all-fields (take limit scanned)
+ {to-sync true
+ to-clear false} (group-by #(boolean (field-values/field-should-have-field-values? %))
+ all-fields)
+ clear-counts (reduce (fn [counts field]
+ (let [result (sync-util/with-error-handling
+ (format "Error clearing field values for %s"
+ (sync-util/name-for-logging field))
+ (clear-field-values-for-field! field))]
+ (merge-with + counts (result->delta result))))
+ empty-counts
+ to-clear)
+ fvs-map (field-values/batched-get-latest-full-field-values (map u/the-id to-sync))
+ fields-to-sync (filterv (fn [field]
+ (let [fv (get fvs-map (u/the-id field))]
+ (cond
+ (not fv)
+ (do (log/tracef "%s does not have FieldValues. Skipping..."
(sync-util/name-for-logging field))
- (clear-field-values-for-field! field))]
- (merge-with + counts (result->delta result))))
- empty-counts
- to-clear)
- fvs-map (field-values/batched-get-latest-full-field-values (map u/the-id to-sync))
- fields-to-sync (filterv (fn [field]
- (let [fv (get fvs-map (u/the-id field))]
- (cond
- (not fv)
- (do (log/tracef "%s does not have FieldValues. Skipping..."
- (sync-util/name-for-logging field))
- false)
-
- (field-values/inactive? fv)
- (do (log/infof "%s has not been used since %s. Skipping..."
- (sync-util/name-for-logging field)
- (t/format "yyyy-MM-dd"
- (t/local-date-time (:last_used_at fv))))
- false)
-
- :else
- true)))
- to-sync)
- sync-counts (sync-fields-for-table! table fields-to-sync fvs-map)]
- (merge-with + clear-counts (or sync-counts empty-counts))))
+ false)
+
+ (field-values/inactive? fv)
+ (do (log/infof "%s has not been used since %s. Skipping..."
+ (sync-util/name-for-logging field)
+ (t/format "yyyy-MM-dd"
+ (t/local-date-time (:last_used_at fv))))
+ false)
+
+ :else
+ true)))
+ to-sync)
+ sync-counts (database-routing/with-anonymous-router-access
+ (sync-fields-for-table! database table fields-to-sync fvs-map))]
+ (merge-with + clear-counts (or sync-counts empty-counts)))))
(mu/defn- update-field-values-for-database!
[database :- i/DatabaseInstance]
(tracing/with-span :sync "field-values.update" {:db/id (:id database)}
(let [tables (sync-util/reducible-sync-tables database)]
- (transduce (map update-field-values-for-table!) (partial merge-with +) tables))))
+ (transduce (map (partial update-field-values-for-table! database)) (partial merge-with +) tables))))
(defn- update-field-values-summary [{:keys [created updated deleted errors probed queries]}]
(format "Updated %d field value sets, created %d, deleted %d with %d errors (probed %d fields in %d queries)"
@@ -263,7 +270,8 @@
(when (seq on-demand-fields)
(log/debugf "Updating FieldValues for %d on-demand fields across %d tables"
(count on-demand-fields) (count (set (map :table_id on-demand-fields))))
- (sync-fields-grouped-by-table! on-demand-fields))))
+ (database-routing/with-anonymous-router-access
+ (sync-fields-grouped-by-table! on-demand-fields)))))
(defn- delete-expired-advanced-field-values-for-field!
[field]
@@ -306,8 +314,9 @@
(mu/defn update-field-values!
"Update the advanced FieldValues (distinct values for categories and certain other fields that are shown
- in widgets like filters) for the Tables in `database` (as needed)."
- [database :- i/DatabaseInstance]
- (sync-util/sync-operation :cache-field-values database (format "Cache field values in %s"
- (sync-util/name-for-logging database))
- (sync-util/run-sync-operation "field values scanning" database sync-field-values-steps)))
+ in widgets like filters) for the Tables in the Database with `database-id` (as needed)."
+ [database-id :- ::lib.schema.id/database]
+ (let [database (driver.u/lib-database database-id)]
+ (sync-util/sync-operation :cache-field-values database (format "Cache field values in %s"
+ (sync-util/name-for-logging database))
+ (sync-util/run-sync-operation "field values scanning" database sync-field-values-steps))))
diff --git a/src/metabase/sync/interface.clj b/src/metabase/sync/interface.clj
index 702be1000ab..3bdb8b0d197 100644
--- a/src/metabase/sync/interface.clj
+++ b/src/metabase/sync/interface.clj
@@ -3,6 +3,7 @@
(:require
[malli.util :as mut]
[metabase.lib.schema.common :as lib.schema.common]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.util.malli.registry :as mr]
[metabase.util.malli.schema :as ms]))
@@ -141,12 +142,10 @@
(mr/def ::no-kebab-case-keys (ms/MapWithNoKebabKeys))
(mr/def ::DatabaseInstance
- [:and
- (ms/InstanceOf :model/Database)
- ::no-kebab-case-keys])
+ [:ref ::lib.schema.metadata/database])
(def DatabaseInstance
- "Schema for a valid instance of a Metabase Database."
+ "Schema for the Lib metadata of a Database, the only form of a Database inside sync."
[:ref ::DatabaseInstance])
(mr/def ::TableInstance
diff --git a/src/metabase/sync/sync.clj b/src/metabase/sync/sync.clj
index e5519f256ae..89368e1b4c5 100644
--- a/src/metabase/sync/sync.clj
+++ b/src/metabase/sync/sync.clj
@@ -9,10 +9,13 @@
In the near future these steps will be scheduled individually, meaning those functions will
be called directly instead of calling the [[sync-database!]] function to do all three at once."
(:require
+ [metabase.database-routing.core :as database-routing]
[metabase.driver.settings :as driver.settings]
[metabase.driver.util :as driver.u]
+ [metabase.lib.schema.id :as lib.schema.id]
[metabase.sync.analyze :as analyze]
[metabase.sync.analyze.fingerprint :as sync.fingerprint]
+ [metabase.sync.db :as sync.db]
[metabase.sync.field-values :as sync.field-values]
[metabase.sync.interface :as i]
[metabase.sync.sync-metadata :as sync-metadata]
@@ -20,8 +23,7 @@
[metabase.tracing.core :as tracing]
[metabase.util.malli :as mu]
[metabase.util.malli.schema :as ms]
- [metabase.warehouse-schema.models.field :as field]
- [metabase.warehouse-schema.models.table :as table])
+ [metabase.warehouse-schema.models.field :as field])
(:import
(java.time.temporal Temporal)))
@@ -52,11 +54,11 @@
:else
[:metadata :analyze :field-values]))
-(defn- do-phase! [database phase]
- (tracing/with-span :sync (str "sync." (name phase)) {:db/id (:id database)
+(defn- do-phase! [database-id phase]
+ (tracing/with-span :sync (str "sync." (name phase)) {:db/id database-id
:sync/phase (name phase)}
(let [f (phase->fn phase)
- result (f database)]
+ result (f database-id)]
(if (instance? Throwable result)
;; do nothing if we're configured to just move on.
(when-not sync-util/*log-exceptions-and-continue?*
@@ -65,42 +67,44 @@
(assoc result :name (name phase))))))
(mu/defn sync-database! :- SyncDatabaseResults
- "Perform all the different sync operations synchronously for `database`.
+ "Perform all the different sync operations synchronously for the Database with `database-id`.
By default, does a `:full` sync that performs all the different sync operations consecutively. You may instead
specify only a `:schema` sync that will sync just the schema but skip analysis. A `:full` sync skips the FieldValues
- scan when the `:initial_sync_status` of `database` is not `\"complete\"`.
+ scan when the `:initial_sync_status` of the Database is not `\"complete\"`.
Please note that this function is *not* what is called by the scheduled tasks; those call different steps
independently. This function is called when a Database is first added."
- ([database :- i/DatabaseInstance]
- (sync-database! database nil))
+ ([database-id :- ::lib.schema.id/database]
+ (sync-database! database-id nil))
- ([database :- i/DatabaseInstance
+ ([database-id :- ::lib.schema.id/database
{:keys [scan], :or {scan :full}} :- [:maybe [:map {:closed true}
[:scan {:optional true} [:maybe [:enum :schema :full]]]]]]
- (tracing/with-span :sync "sync.database" {:db/id (:id database)}
- (sync-util/sync-operation :sync database (format "Sync %s" (sync-util/name-for-logging database))
- (->> (scan-phases scan (= "complete" (:initial_sync_status database)))
- (keep (partial do-phase! database))
- (doall))))))
+ (let [database (driver.u/lib-database database-id)]
+ (tracing/with-span :sync "sync.database" {:db/id database-id}
+ (sync-util/sync-operation :sync database (format "Sync %s" (sync-util/name-for-logging database))
+ (->> (scan-phases scan (= "complete" (sync.db/database-initial-sync-status database-id)))
+ (keep (partial do-phase! database-id))
+ (doall)))))))
(mu/defn sync-table!
"Perform all the different sync operations synchronously for a given `table`. Since often called on a sequence of
tables, caller should check if can connect."
[table :- i/TableInstance]
- (doto table
- sync-metadata/sync-table-metadata!
- analyze/analyze-table!
- sync.field-values/update-field-values-for-table!
- sync-util/set-initial-table-sync-complete!))
+ (database-routing/with-anonymous-router-access
+ (let [database (driver.u/lib-database (:db_id table))]
+ (sync-metadata/sync-table-metadata! database table)
+ (analyze/analyze-table! table)
+ (sync.field-values/update-field-values-for-table! database table)
+ (sync-util/set-initial-table-sync-complete! table)
+ table)))
(mu/defn refingerprint-field!
"Refingerprint a field, usually after its type changes. Checks if can connect to database, returning
`:sync/no-connection` if not."
[field :- i/FieldInstance]
- (let [table (field/table field)
- database (table/database table)]
+ (let [database (driver.u/lib-database (:db_id (field/table field)))]
;; it's okay to allow testing H2 connections during sync. We only want to disallow you from testing them for the
;; purposes of creating a new H2 database.
(if (binding [driver.settings/*allow-testing-h2-connections* true
@@ -108,5 +112,6 @@
(driver.u/can-connect-with-details? (:engine database) (:details database)))
(sync-util/with-error-handling (format "Error refingerprinting field %s"
(sync-util/name-for-logging field))
- (sync.fingerprint/refingerprint-field! field))
+ (database-routing/with-anonymous-router-access
+ (sync.fingerprint/refingerprint-field! field)))
:sync/no-connection)))
diff --git a/src/metabase/sync/sync_metadata.clj b/src/metabase/sync/sync_metadata.clj
index 42c4b6adefe..3689064c7b9 100644
--- a/src/metabase/sync/sync_metadata.clj
+++ b/src/metabase/sync/sync_metadata.clj
@@ -8,6 +8,9 @@
4. Sync FKs and indexes
5. Sync Metabase Metadata table (`metabase.sync.sync-metadata.metabase-metadata`)"
(:require
+ [metabase.database-routing.core :as database-routing]
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.schema.id :as lib.schema.id]
[metabase.sync.fetch-metadata :as fetch-metadata]
[metabase.sync.interface :as i]
[metabase.sync.sync-metadata.dbms-version :as sync-dbms-ver]
@@ -21,8 +24,7 @@
[metabase.sync.util :as sync-util]
[metabase.tracing.core :as tracing]
[metabase.util.log :as log]
- [metabase.util.malli :as mu]
- [metabase.warehouse-schema.models.table :as table]))
+ [metabase.util.malli :as mu]))
(defn- sync-dbms-version-summary [{:keys [version] :as _step-info}]
(if version
@@ -106,21 +108,27 @@
"Sync the metadata for a Metabase `database`. This makes sure child Table & Field objects are synchronized.
Subject to the `disable-auto-sync` setting; for an explicit user-requested sync use
[[sync-db-metadata-explicit!]]."
- [database :- i/DatabaseInstance]
- (sync-util/sync-operation :sync-metadata database (format "Sync metadata for %s" (sync-util/name-for-logging database))
- (sync-db-metadata!* database)))
+ [database-id :- ::lib.schema.id/database]
+ (let [database (driver.u/lib-database database-id)]
+ (sync-util/sync-operation :sync-metadata database (format "Sync metadata for %s" (sync-util/name-for-logging database))
+ (sync-db-metadata!* database))))
(mu/defn sync-db-metadata-explicit!
"Like [[sync-db-metadata!]], but for an explicit, user-requested sync (e.g. the Sync-now button):
runs even when the `disable-auto-sync` setting is enabled."
- [database :- i/DatabaseInstance]
- (sync-util/explicit-sync-operation :sync-metadata database (format "Sync metadata for %s" (sync-util/name-for-logging database))
- (sync-db-metadata!* database)))
+ [database-id :- ::lib.schema.id/database]
+ (let [database (driver.u/lib-database database-id)]
+ (sync-util/explicit-sync-operation :sync-metadata database (format "Sync metadata for %s" (sync-util/name-for-logging database))
+ (sync-db-metadata!* database))))
(mu/defn sync-table-metadata!
"Sync the metadata for an individual `table` -- make sure Fields and FKs are up-to-date."
- [table :- i/TableInstance]
- (let [database (table/database table)]
- (sync-fields/sync-fields-for-table! database table)
- (sync-fks/sync-fks-for-table! database table)
- (sync-indexes/maybe-sync-indexes-for-table! database table)))
+ ([table :- i/TableInstance]
+ (sync-table-metadata! (driver.u/lib-database (:db_id table)) table))
+
+ ([database :- i/DatabaseInstance
+ table :- i/TableInstance]
+ (database-routing/with-anonymous-router-access
+ (sync-fields/sync-fields-for-table! database table)
+ (sync-fks/sync-fks-for-table! database table)
+ (sync-indexes/maybe-sync-indexes-for-table! database table))))
diff --git a/src/metabase/sync/sync_metadata/dbms_version.clj b/src/metabase/sync/sync_metadata/dbms_version.clj
index 7dd220ba6c9..324c8cda80d 100644
--- a/src/metabase/sync/sync_metadata/dbms_version.clj
+++ b/src/metabase/sync/sync_metadata/dbms_version.clj
@@ -17,6 +17,6 @@
[database :- i/DatabaseInstance]
(let [driver (driver.u/database->driver database)
version (driver/dbms-version driver database)]
- (when (not= version (:dbms_version database))
+ (when (not= version (:dbms-version database))
(sync.db/update-database! (:id database) {:dbms_version version}))
version))
diff --git a/src/metabase/sync/sync_metadata/default_schema.clj b/src/metabase/sync/sync_metadata/default_schema.clj
index 00bb34abdf1..911a252e0dd 100644
--- a/src/metabase/sync/sync_metadata/default_schema.clj
+++ b/src/metabase/sync/sync_metadata/default_schema.clj
@@ -11,6 +11,6 @@
[database :- i/DatabaseInstance]
(let [driver (driver.u/database->driver database)
default-schema (driver.sql/default-schema driver database)]
- (when-not (= default-schema (:default_schema database))
+ (when-not (= default-schema (:default-schema database))
(sync.db/update-database! (:id database) {:default_schema default-schema}))
{:default-schema default-schema}))
diff --git a/src/metabase/sync/sync_metadata/fields.clj b/src/metabase/sync/sync_metadata/fields.clj
index 74e211d5c8d..d42543d1678 100644
--- a/src/metabase/sync/sync_metadata/fields.clj
+++ b/src/metabase/sync/sync_metadata/fields.clj
@@ -52,7 +52,6 @@
[metabase.util.log :as log]
[metabase.util.malli :as mu]
[metabase.util.malli.schema :as ms]
- [metabase.warehouse-schema.models.table :as table]
[toucan2.util :as t2.util]))
;;; +----------------------------------------------------------------------------------------------------------------+
@@ -186,7 +185,7 @@
(mu/defn sync-fields-for-table!
"Sync the Fields in the Metabase application database for a specific `table`."
([table :- i/TableInstance]
- (sync-fields-for-table! (table/database table) table))
+ (sync-fields-for-table! (driver.u/lib-database (:db_id table)) table))
([database :- i/DatabaseInstance
table :- i/TableInstance]
diff --git a/src/metabase/sync/sync_metadata/fks.clj b/src/metabase/sync/sync_metadata/fks.clj
index 8556bbf3665..65861bd9f99 100644
--- a/src/metabase/sync/sync_metadata/fks.clj
+++ b/src/metabase/sync/sync_metadata/fks.clj
@@ -8,8 +8,7 @@
[metabase.sync.util :as sync-util]
[metabase.util :as u]
[metabase.util.log :as log]
- [metabase.util.malli :as mu]
- [metabase.warehouse-schema.models.table :as table]))
+ [metabase.util.malli :as mu]))
(mu/defn- mark-fk!
"Updates the `fk_target_field_id` of a Field. Returns 1 if the Field was successfully updated, 0 otherwise."
@@ -30,7 +29,7 @@
(mu/defn sync-fks-for-table!
"Sync the foreign keys for a specific `table`."
([table :- i/TableInstance]
- (sync-fks-for-table! (table/database table) table))
+ (sync-fks-for-table! (driver.u/lib-database (:db_id table)) table))
([database :- i/DatabaseInstance
table :- i/TableInstance]
diff --git a/src/metabase/sync/sync_metadata/tables.clj b/src/metabase/sync/sync_metadata/tables.clj
index 3049c84d5db..4506a353809 100644
--- a/src/metabase/sync/sync_metadata/tables.clj
+++ b/src/metabase/sync/sync_metadata/tables.clj
@@ -129,7 +129,7 @@
database :- i/DatabaseInstance
sync-stage :- [:enum ::reactivate ::create ::update]]
(let [is-crufty? (if (and (= sync-stage ::update)
- (not (:is_attached_dwh database)))
+ (not (:is-attached-dwh database)))
;; TODO: we should add an updated_by column to metabase_table in
;; [[metabase.warehouse-schema-rest.api.table/update-table!*]] to track occasions where the table
;; was updated by an admin, and respect their choices during an update.
@@ -152,11 +152,11 @@
:else "incomplete")
:visibility_type (when is-crufty? :cruft)}))
-(mu/defn create-table! :- (ms/InstanceOf :model/Table)
- "Creates a new table in the database, ready to be synced.
- Throws an exception if there is already a table with the same name, schema and database ID."
- [database :- i/DatabaseInstance
- table :- TableMetadata]
+(mu/defn- insert-table! :- (ms/InstanceOf :model/Table)
+ "Inserts a new table in `database`, ready to be synced; `:sample?` is whether `database` is the Sample Database."
+ [database :- i/DatabaseInstance
+ table :- TableMetadata
+ {:keys [sample?]} :- [:map {:closed true} [:sample? :boolean]]]
(sync.db/insert-table!
(merge (cruft-dependent-cols table database ::create)
{:active true
@@ -174,21 +174,30 @@
{:data_source (:data_source table)})
(when (:data_authority table)
{:data_authority (:data_authority table)})
- (when (:is_sample database)
+ (when sample?
{:data_authority :ingested
:data_source :ingested}))))
+(mu/defn create-table! :- (ms/InstanceOf :model/Table)
+ "Creates a new table in the database, ready to be synced.
+ Throws an exception if there is already a table with the same name, schema and database ID."
+ [database :- i/DatabaseInstance
+ table :- TableMetadata]
+ (insert-table! database table {:sample? (sync.db/database-sample? (:id database))}))
+
(mu/defn- reactivate-table!
"Mark an existing inactive Table active again, refreshing its cruft-dependent columns. `existing`
- is the app-DB Table instance we already hold, so we don't re-select it."
- [database :- i/DatabaseInstance
- existing :- (ms/InstanceOf :model/Table)]
+ is the app-DB Table instance we already hold, so we don't re-select it; `:sample?` is whether `database` is the Sample
+ Database."
+ [database :- i/DatabaseInstance
+ existing :- (ms/InstanceOf :model/Table)
+ {:keys [sample?]} :- [:map {:closed true} [:sample? :boolean]]]
(sync.db/update-table! (:id existing)
(cond-> (cruft-dependent-cols existing database ::reactivate)
;; do not unhide tables w/ cruft settings
(some? (:visibility_type existing)) (dissoc :visibility_type)
true (assoc :active true)
- (:is_sample database) (assoc :data_authority :ingested
+ sample? (assoc :data_authority :ingested
:data_source :ingested))))
(mu/defn create-or-reactivate-table!
@@ -196,7 +205,7 @@
[database :- i/DatabaseInstance
{schema :schema table-name :name :as table} :- TableMetadata]
(if-let [existing (sync.db/inactive-table-by-schema-and-name (u/the-id database) schema table-name)]
- (reactivate-table! database existing)
+ (reactivate-table! database existing {:sample? (sync.db/database-sample? (u/the-id database))})
(create-table! database table)))
;; TODO - should we make this logic case-insensitive like it is for fields?
@@ -242,15 +251,16 @@
(mu/defn- create-tables! :- [:sequential (ms/InstanceOf :model/Table)]
"Insert brand-new tables for `database`, returning the created `:model/Table` instances. Sorted by
`[schema name]` so auto-increment ids are assigned deterministically."
- [database :- i/DatabaseInstance
- new-table-metadatas :- [:set i/DatabaseMetadataTable]]
+ [database :- i/DatabaseInstance
+ new-table-metadatas :- [:set i/DatabaseMetadataTable]
+ options :- [:map {:closed true} [:sample? :boolean]]]
(doseq [table-metadata new-table-metadatas]
(log/info "Found new table:"
(sync-util/name-for-logging (mi/instance :model/Table table-metadata))))
(let [field-order (some-> (:engine database) driver/default-field-order)]
(dependencies.hooks/with-batched-marks
(mapv (fn [table-metadata]
- (create-table! database (m/assoc-some table-metadata :field_order field-order)))
+ (insert-table! database (m/assoc-some table-metadata :field_order field-order) options))
(sort-by (juxt :schema :name) new-table-metadatas)))))
(mu/defn- retire-tables!
@@ -267,7 +277,7 @@
"Update the table metadata if it has changed. Returns true if a row was updated."
[table-metadata :- i/DatabaseMetadataTable
metabase-table :- (ms/InstanceOf :model/Table)
- metabase-database :- (ms/InstanceOf :model/Database)]
+ metabase-database :- i/DatabaseInstance]
(let [old-table (select-keys metabase-table keys-to-update)
new-table (-> (zipmap keys-to-update (repeat nil))
(merge table-metadata
@@ -308,7 +318,7 @@
the number of tables actually updated."
[table-metadatas :- [:set i/DatabaseMetadataTable]
metabase-tables :- [:set (ms/InstanceOf :model/Table)]
- metabase-database :- (ms/InstanceOf :model/Database)]
+ metabase-database :- i/DatabaseInstance]
(let [name+schema->table-metadata (m/index-by (juxt :name :schema) table-metadatas)
name+schema->metabase-table (m/index-by (juxt :name :schema) metabase-tables)]
(reduce (fn [updated name+schema]
@@ -425,6 +435,7 @@
[:created :int] ; running count of created/reactivated tables
[:updated :int] ; running count of metadata-updated tables
[:complete? :boolean] ; false once any batch fails -- then we don't retire
+ [:sample? :boolean] ; whether the database is the Sample Database
[:seen (ms/InstanceOfClass ITransientSet)] ; transient set of reconciled `:model/Table` ids
[:metabase-metadata [:vector i/DatabaseMetadataTable]]]) ; captured `_metabase_metadata` table
@@ -450,9 +461,9 @@
to-create (into #{} (remove existing-row) tables)
to-reactivate (into #{} (filter #(and (existing-row %) (not (:active (existing-row %))))) tables)
created (when (seq to-create)
- (create-tables! database to-create))]
+ (create-tables! database to-create (select-keys context [:sample?])))]
(doseq [table-metadata to-reactivate]
- (reactivate-table! database (existing-row table-metadata)))
+ (reactivate-table! database (existing-row table-metadata) (select-keys context [:sample?])))
(when (seq to-reactivate)
(let [rows (existing-tables-by-name+schema database to-reactivate)]
(update-tables-metadata-if-needed! to-reactivate
@@ -524,6 +535,7 @@
context (transduce (partition-all table-sync-batch-size)
(completing reconcile-batch)
{:created 0, :updated 0, :complete? true
+ :sample? (sync.db/database-sample? (:id database))
:seen (transient #{}), :metabase-metadata []}
(:tables db-metadata))
{:keys [created updated complete?]} context]
diff --git a/src/metabase/sync/task/sync_databases.clj b/src/metabase/sync/task/sync_databases.clj
index a1e10b7345a..1336964db62 100644
--- a/src/metabase/sync/task/sync_databases.clj
+++ b/src/metabase/sync/task/sync_databases.clj
@@ -85,17 +85,17 @@
(catch Throwable e
e))]
(log/warnf "Cannot sync Database %d: %s" database-id (ex-message ex))
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(let [db-id (:id database)
metadata-results (tracing/with-span :sync "sync.metadata" {:db/id db-id}
- (sync-metadata/sync-db-metadata! database))
+ (sync-metadata/sync-db-metadata! db-id))
analyze-results (when (:is_full_sync database)
(tracing/with-span :sync "sync.analyze" {:db/id db-id}
- (analyze/analyze-db! database)))
+ (analyze/analyze-db! db-id)))
refingerprint-results (when (and (:refingerprint database)
(should-refingerprint-fields? analyze-results))
(tracing/with-span :sync "sync.refingerprint" {:db/id db-id}
- (analyze/refingerprint-db! database)))]
+ (analyze/refingerprint-db! db-id)))]
(cond-> {:metadata-results metadata-results}
analyze-results (assoc :analyze-results analyze-results)
refingerprint-results (assoc :refingerprint-results refingerprint-results)))))))
@@ -143,7 +143,7 @@
(sync-databases-trigger/unschedule-tasks-for-db! (mi/instance :model/Database {:id database-id}))
(log/warnf "Cannot update Field values for Database %d: Database does not exist." database-id)))]
(if (:is_full_sync database)
- (sync.field-values/update-field-values! database)
+ (sync.field-values/update-field-values! database-id)
(log/infof "Skipping update, automatic Field value updates are disabled for Database %d." database-id)))))))
(task/defjob UpdateFieldValues
diff --git a/src/metabase/sync/util.clj b/src/metabase/sync/util.clj
index 24b11147ff8..a61144aeab1 100644
--- a/src/metabase/sync/util.clj
+++ b/src/metabase/sync/util.clj
@@ -7,10 +7,10 @@
[java-time.api :as t]
[medley.core :as m]
[metabase.analytics-interface.core :as analytics]
+ [metabase.database-routing.core :as database-routing]
[metabase.driver :as driver]
[metabase.driver.util :as driver.u]
[metabase.events.core :as events]
- [metabase.lib.schema.id :as lib.schema.id]
[metabase.models.interface :as mi]
[metabase.premium-features.core :as premium-features]
[metabase.query-processor.interface :as qp.i]
@@ -67,9 +67,6 @@
{:error/message "Sync event deriving from :metabase.sync.util/event"}
#(events/isa? % ::event)]])
-(def ^:private DatabaseOrId
- [:or ::lib.schema.id/database :metabase.warehouses.schema/database-or-metadata])
-
;;; +----------------------------------------------------------------------------------------------------------------+
;;; | SYNC OPERATION "MIDDLEWARE" |
;;; +----------------------------------------------------------------------------------------------------------------+
@@ -112,18 +109,18 @@
:sync database-id
#(sync-db! database-id))"
{:style/indent [:form]}
- [operation database-or-id f]
+ [operation database f]
(fn []
- (when-not (contains? (@operation->db-ids operation) (u/the-id database-or-id))
+ (when-not (contains? (@operation->db-ids operation) (u/the-id database))
(try
;; mark this database as currently syncing so we can prevent duplicate sync attempts (#2337)
- (swap! operation->db-ids update operation #(conj (or % #{}) (u/the-id database-or-id)))
+ (swap! operation->db-ids update operation #(conj (or % #{}) (u/the-id database)))
(log/debug "Sync operations in flight:" (m/filter-vals seq @operation->db-ids))
;; do our work
(f)
;; always take the ID out of the set when we are through
(finally
- (swap! operation->db-ids update operation #(disj % (u/the-id database-or-id))))))))
+ (swap! operation->db-ids update operation #(disj % (u/the-id database))))))))
(mu/defn- with-sync-events
"Publish events related to beginning and ending a sync-like process, e.g. `:sync-database` or `:cache-values`, for a
@@ -131,7 +128,7 @@
{:style/indent [:form]}
;; we can do everyone a favor and infer the name of the individual begin and sync events
([event-name-prefix :- :keyword
- database-or-id :- DatabaseOrId
+ database :- i/DatabaseInstance
f :- fn?]
(letfn [(event-keyword [prefix suffix]
(keyword (or (namespace event-name-prefix) "event")
@@ -139,21 +136,21 @@
(with-sync-events
(event-keyword event-name-prefix "-begin")
(event-keyword event-name-prefix "-end")
- database-or-id
+ database
f)))
([begin-event-name :- Topic
end-event-name :- Topic
- database-or-id :- DatabaseOrId
+ database :- i/DatabaseInstance
f :- fn?]
(fn []
(let [start-time (System/nanoTime)
tracking-hash (str (random-uuid))]
- (events/publish-event! begin-event-name {:database_id (u/the-id database-or-id), :custom_id tracking-hash})
+ (events/publish-event! begin-event-name {:database_id (u/the-id database), :custom_id tracking-hash})
(let [return (f)
total-time-ms (int (/ (- (System/nanoTime) start-time)
1000000.0))]
- (events/publish-event! end-event-name {:database_id (u/the-id database-or-id)
+ (events/publish-event! end-event-name {:database_id (u/the-id database)
:custom_id tracking-hash
:running_time total-time-ms})
return)))))
@@ -314,7 +311,7 @@
wrapped in the surrounding machinery (task-history, events, logging, duplicate-op prevention).
Performs no eligibility gating itself — the public wrappers decide whether to call it."
[operation :- :keyword ; something like `:sync-metadata` or `:refingerprint`
- database :- (ms/InstanceOf :model/Database)
+ database :- i/DatabaseInstance
message :- ms/NonBlankString
f :- fn?]
(if-let [busy (external-busy)]
@@ -334,7 +331,7 @@
(with-db-logging-disabled
(sync-in-context database
(partial do-with-error-handling (format "Error in sync step %s" message) f))))))
- result (sync-fn)]
+ result (database-routing/with-anonymous-router-access (sync-fn))]
(when (instance? Throwable result)
(analytics/inc! :metabase-sync/failures {:driver (name (:engine database))}))
result)))))
@@ -345,10 +342,10 @@
[[metabase.warehouses.models.database/should-auto-sync?]]) — i.e. it is suppressed when the
`disable-auto-sync` setting is on."
[operation :- :keyword
- database :- (ms/InstanceOf :model/Database)
+ database :- i/DatabaseInstance
message :- ms/NonBlankString
f :- fn?]
- (when (database/should-auto-sync? database)
+ (when (database/should-auto-sync? (u/the-id database))
(do-sync-operation* operation database message f)))
(mu/defn do-explicit-sync-operation
@@ -357,10 +354,10 @@
syncable at all (see [[metabase.warehouses.models.database/should-sync?]]), ignoring the
`disable-auto-sync` setting, which suppresses only automatically-triggered syncs."
[operation :- :keyword
- database :- (ms/InstanceOf :model/Database)
+ database :- i/DatabaseInstance
message :- ms/NonBlankString
f :- fn?]
- (when (database/should-sync? database)
+ (when (database/should-sync? (u/the-id database))
(do-sync-operation* operation database message f)))
(defmacro sync-operation
@@ -464,8 +461,8 @@
(defn set-initial-table-sync-complete-for-db!
"Marks initial sync for all tables in `db` as complete so that it becomes usable in the UI, if not already
set."
- [database-or-id]
- (let [ids (sync.db/sync-table-ids (u/the-id database-or-id))]
+ [database]
+ (let [ids (sync.db/sync-table-ids (u/the-id database))]
(reduce (fn [acc ids']
(+ acc (sync.db/update-tables! ids' {:initial_sync_status "complete"})))
0
@@ -474,7 +471,7 @@
(defn set-initial-database-sync-complete!
"Marks initial sync as complete for this database so that this is reflected in the UI, if not already set"
[database]
- (when (not= (:initial_sync_status database) "complete")
+ (when (not= (sync.db/database-initial-sync-status (u/the-id database)) "complete")
(sync.db/update-database! (u/the-id database) {:initial_sync_status "complete"
:initial_sync_error nil})))
@@ -482,7 +479,7 @@
"Marks initial sync as aborted for this database, recording the message of `error` (the Throwable that aborted it)
so that the cause can be displayed on the UI"
[database error]
- (when (not= (:initial_sync_status database) "complete")
+ (when (not= (sync.db/database-initial-sync-status (u/the-id database)) "complete")
(sync.db/update-database! (u/the-id database) {:initial_sync_status "aborted"
:initial_sync_error (ex-message error)})))
@@ -491,31 +488,31 @@
;;; +----------------------------------------------------------------------------------------------------------------+
(mu/defn reducible-sync-tables
- "Returns a reducible of all the Tables that should go through the sync processes for `database-or-id`.
+ "Returns a reducible of all the Tables that should go through the sync processes for `database`.
Returns tables ordered by `[schema name]` so results match the order expected by [[metabase.driver/describe-fks]]."
- [database-or-id :- DatabaseOrId
+ [database :- i/DatabaseInstance
& {:keys [schema-names table-names]} :- [:maybe ::driver/describe-fks.options]]
(eduction (map t2.realize/realize)
- (sync.db/sync-tables-reducible (u/the-id database-or-id) schema-names table-names)))
+ (sync.db/sync-tables-reducible (u/the-id database) schema-names table-names)))
(defn sync-tables-count
- "The count of all tables that should be synced for `database-or-id`."
- [database-or-id]
- (sync.db/sync-tables-count (u/the-id database-or-id)))
+ "The count of all tables that should be synced for `database`."
+ [database]
+ (sync.db/sync-tables-count (u/the-id database)))
(defn refingerprint-reducible-sync-tables
- "A reducible collection of all the Tables that should go through the sync processes for `database-or-id`, in the
+ "A reducible collection of all the Tables that should go through the sync processes for `database`, in the
order they should be refingerprinted (by earliest last_analyzed timestamp)."
- [database-or-id]
+ [database]
(eduction (map t2.realize/realize)
- (sync.db/sync-tables-by-earliest-analyzed-reducible (u/the-id database-or-id))))
+ (sync.db/sync-tables-by-earliest-analyzed-reducible (u/the-id database))))
(defn sync-schemas
- "Returns all the Schemas that have their metadata sync'd for `database-or-id`.
+ "Returns all the Schemas that have their metadata sync'd for `database`.
Assumes the database supports schemas."
- [database-or-id]
- (vec (map :schema (sync.db/sync-table-schemas (u/the-id database-or-id)))))
+ [database]
+ (vec (map :schema (sync.db/sync-table-schemas (u/the-id database)))))
(defmulti name-for-logging
"Return an appropriate string for logging an object in sync logging messages. Should be something like
@@ -525,9 +522,10 @@
This function is used all over the sync code to make sure we have easy access to consistently formatted descriptions
of various objects."
{:arglists '([instance])}
- mi/model)
+ (fn [instance]
+ (or (:lib/type instance) (mi/model instance))))
-(defmethod name-for-logging :model/Database
+(defmethod name-for-logging :metadata/database
[{database-name :name, id :id, engine :engine}]
(format "%s Database %s ''%s''" (name engine) (str (or id "")) database-name))
diff --git a/src/metabase/transforms/crud.clj b/src/metabase/transforms/crud.clj
index a08d1794c44..b0602578aab 100644
--- a/src/metabase/transforms/crud.clj
+++ b/src/metabase/transforms/crud.clj
@@ -37,7 +37,7 @@
(deferred-tru "Cannot run transforms on audit databases."))
(api/check-400 (every? (fn [feature] (driver.u/supports? (:engine database) feature database)) features)
(deferred-tru "The database does not support the requested transform features."))
- (api/check-400 (not (database-routing/db-routing-enabled? database))
+ (api/check-400 (not (database-routing/db-routing-enabled? (driver.u/lib-database (:id database))))
(deferred-tru "Transforms are not supported on databases with DB routing enabled."))))
(defn check-feature-enabled!
@@ -132,7 +132,8 @@
"The index methods the target db's driver can create on `transform`'s target table, or nil when none are available."
[transform]
(when-let [db-id (transforms-base.i/target-db-id transform)]
- (when-let [database (transforms.db/database db-id)]
+ (when-let [database (when (transforms.db/database-exists? db-id)
+ (driver.u/lib-database db-id))]
(let [methods (try
(driver/supported-index-methods (:engine database) database)
(catch Throwable e
diff --git a/src/metabase/transforms/query_impl.clj b/src/metabase/transforms/query_impl.clj
index 02b1abc305f..0943d7f802e 100644
--- a/src/metabase/transforms/query_impl.clj
+++ b/src/metabase/transforms/query_impl.clj
@@ -3,11 +3,11 @@
[clojure.core.async :as a]
[metabase.driver :as driver]
[metabase.driver.connection :as driver.conn]
+ [metabase.driver.util :as driver.u]
[metabase.request.core :as request]
[metabase.tracing.core :as tracing]
[metabase.transforms-base.interface :as transforms-base.i]
[metabase.transforms-base.util :as transforms-base.u]
- [metabase.transforms.db :as transforms.db]
[metabase.transforms.instrumentation :as transforms.instrumentation]
[metabase.transforms.interface :as transforms.i]
[metabase.transforms.util :as transforms.u]
@@ -20,7 +20,7 @@
([{:keys [id source target owner_user_id creator_id] :as transform}
{:keys [run-method on-start user-id parent-run]}]
(try
- (let [db (transforms.db/database (get-in source [:query :database]))
+ (let [db (driver.u/lib-database (get-in source [:query :database]))
driver (:engine db)
_ (transforms-base.u/throw-if-db-routing-enabled! transform db)
run-user-id (if (and (= run-method :manual) user-id)
diff --git a/src/metabase/transforms_base/db.clj b/src/metabase/transforms_base/db.clj
index d01bf5f99f7..3497b1dc174 100644
--- a/src/metabase/transforms_base/db.clj
+++ b/src/metabase/transforms_base/db.clj
@@ -42,10 +42,10 @@
[:checkpoint_hi_value {:optional true} [:maybe :string]]]]
(t2/update! :model/TransformRun run-id changes))
-(mu/defn database
- "The Database with `database-id`, or nil."
+(mu/defn database-exists?
+ "Whether a Database with `database-id` exists."
[database-id :- ::lib.schema.id/database]
- (t2/select-one :model/Database database-id))
+ (t2/exists? :model/Database :id database-id))
(mu/defn field-table-id
"The Table id of the Field with `field-id`, or nil."
diff --git a/src/metabase/transforms_base/query.clj b/src/metabase/transforms_base/query.clj
index 4e801e96622..7bb8b5abe70 100644
--- a/src/metabase/transforms_base/query.clj
+++ b/src/metabase/transforms_base/query.clj
@@ -94,7 +94,8 @@
(when (and cancelled? (cancelled?))
(throw (ex-info "Transform cancelled before start" {:status :cancelled})))
(let [db (get-in source [:query :database])
- {driver :engine :as database} (when db (transforms-base.db/database db))
+ {driver :engine :as database} (when (and db (transforms-base.db/database-exists? db))
+ (driver.u/lib-database db))
_ (when-not database
(throw (ex-info "Source database for this transform has been deleted."
{:transform-id (:id transform)
diff --git a/src/metabase/transforms_base/schema.clj b/src/metabase/transforms_base/schema.clj
index 5c07895a6d7..2ebb8814e47 100644
--- a/src/metabase/transforms_base/schema.clj
+++ b/src/metabase/transforms_base/schema.clj
@@ -54,6 +54,22 @@
[:target-incremental-strategy {:optional true} ::target-incremental-strategy]
[:index-request-ids {:optional true} [:sequential pos-int?]]])
+(mr/def ::synced-target
+ "A transform target with the columns of the Table that syncing it creates."
+ [:merge
+ ::transform-target
+ [:map {:closed true}
+ [:data_authority {:optional true} :keyword]
+ [:data_source {:optional true} :keyword]
+ [:is_writable {:optional true} :boolean]]])
+
+(mr/def ::indexed-target
+ "A transform target with the ID of the transform whose index requests it carries."
+ [:merge
+ ::transform-target
+ [:map {:closed true}
+ [:transform-id {:optional true} pos-int?]]])
+
(mr/def ::transform
"A transform map as expected by execute-base! implementations. The full transforms module
(metabase.transforms.schema) hydrates and stores more columns on this same map; they're declared here,
diff --git a/src/metabase/transforms_base/util.clj b/src/metabase/transforms_base/util.clj
index 093888530da..f1d447563fc 100644
--- a/src/metabase/transforms_base/util.clj
+++ b/src/metabase/transforms_base/util.clj
@@ -10,6 +10,7 @@
[metabase.database-routing.core :as database-routing]
[metabase.dependencies.hooks :as dependencies.hooks]
[metabase.driver :as driver]
+ [metabase.driver.util :as driver.u]
[metabase.events.core :as events]
[metabase.indexes.models.table-index :as table-index]
[metabase.indexes.reconcile :as reconcile]
@@ -18,6 +19,7 @@
[metabase.lib.core :as lib]
[metabase.lib.metadata :as lib.metadata]
[metabase.lib.schema.common :as lib.schema.common]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.lib.types.isa :as lib.types.isa]
[metabase.lib.util :as lib.util]
[metabase.query-processor.compile :as qp.compile]
@@ -187,13 +189,15 @@
;;; ------------------------------------------------- Table Names -------------------------------------------------
-(defn- resolve-nil-schema
+(mu/defn- resolve-nil-schema
"When a table has nil schema, check if the physical table exists under the driver's
default schema. If so, return that schema. Otherwise return nil.
This handles the case where transforms create tables without explicit schema
but the driver needs a schema to find the table during sync."
- [driver database table]
- (when-let [default-schema (:default_schema database)]
+ [driver :- :keyword
+ database :- ::lib.schema.metadata/database
+ table :- :metabase.warehouse-schema.schema/table]
+ (when-let [default-schema (:default-schema database)]
(when (driver/table-exists? driver database {:schema default-schema :name (:name table)})
default-schema)))
@@ -515,12 +519,16 @@
"Test if the target table of a transform already exists."
[{:keys [target] :as transform}]
(let [db-id (transforms-base.i/target-db-id transform)
- {driver :engine :as database} (transforms-base.db/database db-id)]
+ {driver :engine :as database} (driver.u/lib-database db-id)]
(driver/table-exists? driver database target)))
-(defn- sync-table!
- ([database target] (sync-table! database target nil))
- ([database target {:keys [create?]}]
+(mu/defn- sync-table!
+ ([database :- ::lib.schema.metadata/database
+ target :- ::transforms-base.schema/synced-target]
+ (sync-table! database target nil))
+ ([database :- ::lib.schema.metadata/database
+ target :- ::transforms-base.schema/synced-target
+ {:keys [create?]} :- [:maybe [:map {:closed true} [:create? {:optional true} :boolean]]]]
(when-let [table (or (target-table (:id database) target)
(when create?
(sync/create-table! database (select-keys target [:schema :name :data_source :data_authority :is_writable]))))]
@@ -536,9 +544,10 @@
(sync/sync-table! table)
table))))
-(defn activate-table-and-mark-computed!
+(mu/defn activate-table-and-mark-computed!
"Activate table for `target` in `database` in the app db."
- [database target]
+ [database :- ::lib.schema.metadata/database
+ target :- ::transforms-base.schema/transform-target]
(when-let [table (sync-table! database (assoc target
:data_authority :computed
:data_source :metabase-transform
@@ -548,16 +557,18 @@
(transforms-base.db/update-table! (:id table) {:active true}))
table))
-(defn sync-target!
+(mu/defn sync-target!
"Sync target of a transform"
- [target database]
+ [target :- ::transforms-base.schema/transform-target
+ database :- ::lib.schema.metadata/database]
;; sync the new table (note that even a failed sync status means that the execution succeeded)
(log/info "Syncing target" (pr-str target) "for transform")
(activate-table-and-mark-computed! database target))
-(defn deactivate-table!
+(mu/defn deactivate-table!
"Deactivate table for `target` in `database` in the app db."
- [database target]
+ [database :- ::lib.schema.metadata/database
+ target :- ::transforms-base.schema/transform-target]
(when-let [table (sync-table! database target)]
;; TODO this should probably be a function in the sync module
(transforms-base.db/update-table! (:id table) {:active false})))
@@ -568,7 +579,8 @@
(when target
(let [database-id (transforms-base.i/target-db-id transform)]
(when database-id
- (if-let [{driver :engine :as database} (transforms-base.db/database database-id)]
+ (if-let [{driver :engine :as database} (when (transforms-base.db/database-exists? database-id)
+ (driver.u/lib-database database-id))]
(let [drop-target (update target :type keyword)]
(driver/drop-transform-target! driver database drop-target)
(log/info "Deactivating target " (pr-str target) "for transform" id)
@@ -640,11 +652,12 @@
(when transform-id
(transforms-base.db/mark-table-index-failed! transform-id (reconcile/index-name index) (ex-message t))))
-(defn- apply-standalone-indexes!
+(mu/defn- apply-standalone-indexes!
"Create the target's `:standalone` indexes as separate DDL, now that the table exists. `:inline` kinds render at
table creation, so they're filtered out here. Each create uses `:if-not-exists`, so re-applying is a no-op. On a
per-index DDL throw, marks the request failed (when `:transform-id` is set) and re-throws."
- [database {:keys [indexes schema transform-id] table-name :name}]
+ [database :- ::lib.schema.metadata/database
+ {:keys [indexes schema transform-id] table-name :name} :- ::transforms-base.schema/indexed-target]
(let [driver (:engine database)
methods (driver/supported-index-methods driver database)
standalone (filter #(= :standalone (get-in methods [(:kind %) :lifecycle])) indexes)]
@@ -669,7 +682,7 @@
[transform]
(when (and (seq (:indexes (:target transform)))
(full-create-run? transform))
- (let [database (transforms-base.db/database (transforms-base.i/target-db-id transform))]
+ (let [database (driver.u/lib-database (transforms-base.i/target-db-id transform))]
(apply-standalone-indexes! database (assoc (:target transform) :transform-id (:id transform))))))
(defn- apply-index-outcomes!
@@ -705,7 +718,7 @@
(table-index/select-for-verification (:id transform) (:index-request-ids target))
(table-index/select-for-transform (:id transform)))]
(when-let [managed (seq managed)]
- (let [database (transforms-base.db/database (transforms-base.i/target-db-id transform))
+ (let [database (driver.u/lib-database (transforms-base.i/target-db-id transform))
{:keys [schema] table-name :name} (:target transform)]
(when-some [warehouse-indexes (try
(reconcile/fetch-warehouse-indexes database schema table-name)
@@ -732,7 +745,7 @@
{:keys [publish-events?]
:or {publish-events? true}} opts
db-id (transforms-base.i/target-db-id transform)
- database (transforms-base.db/database db-id)
+ database (driver.u/lib-database db-id)
table (sync-target! target database)]
;; Sync target table, set target_table_id on transform, and mark table as owned by this transform
(when table
diff --git a/src/metabase/upload/impl.clj b/src/metabase/upload/impl.clj
index 73ec10f9bba..445985a4344 100644
--- a/src/metabase/upload/impl.clj
+++ b/src/metabase/upload/impl.clj
@@ -467,8 +467,8 @@
:asynchronous)
(defn- scan-and-sync-table!
- [database table]
- (sync/sync-fields-for-table! database table)
+ [table]
+ (sync/sync-fields-for-table! table)
(case *auxiliary-sync-steps*
:asynchronous (future (sync/sync-table! table))
:synchronous (sync/sync-table! table)
@@ -525,7 +525,7 @@
(ex-info (tru "You don''t have permissions to do that.")
{:status-code 403})
(and (some? schema-name)
- (not (driver.s/include-schema? db schema-name)))
+ (not (driver.s/include-schema? (driver.u/lib-database (:id db)) schema-name)))
(ex-info (tru "The schema {0} is not syncable." schema-name)
{:status-code 422}))
(can-use-uploads-error db)))
@@ -572,12 +572,13 @@
;; have unrestricted access to this schema (see `can-create-upload-error`) -- gets a table that inherits
;; the schema's permissions rather than failing safe to `:blocked` the way a sync-discovered table would
;; (UXW-3217). `mark-table-upload!` below re-asserts it along with the other upload columns.
- table (sync/create-table! db {:name table-name
- :schema (not-empty schema)
- :display_name display-name
- :data_source :upload})
+ table (sync/create-table! (driver.u/lib-database (:id db))
+ {:name table-name
+ :schema (not-empty schema)
+ :display_name display-name
+ :data_source :upload})
_set_is_upload (upload.db/mark-table-upload! (:id table))
- _sync (scan-and-sync-table! db table)
+ _sync (scan-and-sync-table! table)
_set_names (set-display-names! (:id table) columns)
;; Set the display_name of the auto-generated primary key column to the same as its name, so that if users
;; download results from the table as a CSV and reupload, we'll recognize it as the same column
@@ -867,7 +868,7 @@
(add-columns! driver database table
{auto-pk-column-keyword ::upload-types/auto-incrementing-int-pk}
:primary-key [auto-pk-column-keyword]))
- (scan-and-sync-table! database table)
+ (scan-and-sync-table! table)
(set-display-names! (:id table) (zipmap column-names display-names))
(when create-auto-pk?
(let [auto-pk-field (table-id->auto-pk-column driver (:id table))]
diff --git a/src/metabase/util/connection.clj b/src/metabase/util/connection.clj
index 31a2952f4d0..60ce72111c9 100644
--- a/src/metabase/util/connection.clj
+++ b/src/metabase/util/connection.clj
@@ -10,6 +10,15 @@
(set! *warn-on-reflection* true)
+(defn register-c3p0-connection-customizer!
+ "Seeds c3p0's customizer cache with an instance of `klass`, since c3p0's `Class/forName` can't see classes defined in
+ Clojure's dynamic class loader (https://github.com/swaldman/c3p0/issues/166)."
+ [^Class klass]
+ (let [field (doto (.getDeclaredField com.mchange.v2.c3p0.C3P0Registry "classNamesToConnectionCustomizers")
+ (.setAccessible true))]
+ (.put ^java.util.HashMap (.get field com.mchange.v2.c3p0.C3P0Registry)
+ (.getName klass) (.newInstance klass))))
+
(defn- consume-rset [^ResultSet rset cb]
(into {} (iteration
(fn [_]
diff --git a/src/metabase/warehouse_schema/models/field_values.clj b/src/metabase/warehouse_schema/models/field_values.clj
index ada9dc21d90..34fd175bcfb 100644
--- a/src/metabase/warehouse_schema/models/field_values.clj
+++ b/src/metabase/warehouse_schema/models/field_values.clj
@@ -27,6 +27,7 @@
[java-time.api :as t]
[medley.core :as m]
[metabase.analyze.core :as analyze]
+ [metabase.database-routing.core :as database-routing]
[metabase.lib-be.core :as lib-be]
[metabase.lib.core :as lib]
[metabase.lib.schema.metadata :as lib.schema.metadata]
@@ -519,7 +520,7 @@
;; `distinct-values` returns nil when the warehouse scan failed, which is not the same as a
;; field that genuinely has no values: persisting it would read as "empty" and delete the
;; values we already have. Leave whatever is cached in place instead.
- (if-let [{rows :values} (distinct-values field)]
+ (if-let [{rows :values} (database-routing/with-anonymous-router-access (distinct-values field))]
;; rows are 1-tuples — unwrap for storage.
(persist-field-values! field existing-fv (seq (map first rows)))
::fv-fetch-failed))
diff --git a/src/metabase/warehouse_schema_rest/api/table.clj b/src/metabase/warehouse_schema_rest/api/table.clj
index 3a3f293cc46..c051812bc15 100644
--- a/src/metabase/warehouse_schema_rest/api/table.clj
+++ b/src/metabase/warehouse_schema_rest/api/table.clj
@@ -534,7 +534,7 @@
(defn- sync-schema-async!
[table user-id]
(events/publish-event! :event/table-manual-sync {:object table :user-id user-id})
- (quick-task/submit-task! #(database-routing/with-database-routing-off (sync/sync-table! table))))
+ (quick-task/submit-task! #(database-routing/with-anonymous-router-access (sync/sync-table! table))))
;; TODO (Cam 10/28/25) -- fix this endpoint route to use kebab-case for consistency with the rest of our REST API
;;
diff --git a/src/metabase/warehouses/models/database.clj b/src/metabase/warehouses/models/database.clj
index 51139492c71..f301374dc6a 100644
--- a/src/metabase/warehouses/models/database.clj
+++ b/src/metabase/warehouses/models/database.clj
@@ -55,7 +55,6 @@
(t2/deftransforms :model/Database
{:details (mi/transform-encrypted-json "metabase_database.details")
:write_data_details (mi/transform-encrypted-json "metabase_database.write_data_details")
- :admin_details (mi/transform-encrypted-json "metabase_database.admin_details")
:engine mi/transform-keyword
:metadata_sync_schedule mi/transform-cron-string
:cache_field_values_schedule mi/transform-cron-string
@@ -212,26 +211,26 @@
(select-keys database [:let-user-control-scheduling :is_full_sync :is_on_demand]))))))
(defn should-sync?
- "Should this database be synced at all? Destination (router-child) databases are never synced.
+ "Should the Database with `database-id` be synced at all? Destination (router-child) databases are never synced.
This is the gate for *explicit*, user-requested syncs (e.g. the Sync-now button), which run
regardless of the `disable-auto-sync` setting. For automatically-triggered syncs use
[[should-auto-sync?]]."
- [db]
- (not (is-destination? db)))
+ [database-id]
+ (nil? (warehouses.db/router-database-id database-id)))
(defn should-auto-sync?
"Should this database be synced *automatically* — scheduled sync/analyze, the post-add initial
sync, and Quartz trigger registration? False when the database isn't syncable at all, or when
auto-sync is globally suppressed via the `disable-auto-sync` setting."
- [db]
- (and (should-sync? db)
+ [database-id]
+ (and (should-sync? database-id)
(not (warehouses.settings/disable-auto-sync))))
(defn- check-and-schedule-tasks-for-db!
"(Re)schedule sync operation tasks for `database`. (Existing scheduled tasks will be deleted first.)"
[database]
(try
- (when (should-auto-sync? database)
+ (when (should-auto-sync? (u/the-id database))
(sync-databases-trigger/check-and-schedule-tasks-for-db! database))
(catch Throwable e
(log/errorf "Error scheduling tasks for DB: %s" (ex-message e)))))
@@ -288,7 +287,6 @@
"Checks database health off-thread.
- checks connectivity for the default connection
- checks connectivity for the write connection (if configured)
- - checks connectivity for the admin connection (if configured)
- cleans-up ambiguous legacy db-details"
[{:keys [engine] :as database}]
(when-not (or (:is_audit database) (:is_sample database))
@@ -312,15 +310,8 @@
(catch Throwable provider-e
(log/warnf "Error during provider detection for database {:id %d}: %s" (:id database) (ex-message provider-e)))))))
(when (driver.conn/database-write-data-details lib-db)
- (let [write-details (driver.conn/without-resolution-telemetry
- (driver.conn/with-write-connection
- (driver.conn/effective-details database)))]
- (check-connection! database driver engine-str (assoc write-details :engine engine-str) "write-data")))
- (when (driver.conn/database-admin-details lib-db)
- (let [admin-details (driver.conn/without-resolution-telemetry
- (driver.conn/with-admin-connection
- (driver.conn/effective-details database)))]
- (check-connection! database driver engine-str (assoc admin-details :engine engine-str) "admin"))))))))
+ (let [write-details (merge details (driver.conn/details-for-exact-type lib-db :write-data))]
+ (check-connection! database driver engine-str (assoc write-details :engine engine-str) "write-data"))))))))
(defn- health-check-candidates
"The databases to health check at startup: one representative per unique engine — the lowest-id regular database of
@@ -393,8 +384,7 @@
driver
(-> db
(m/update-existing-in [:details :auth-provider] keyword)
- (m/update-existing-in [:write_data_details :auth-provider] keyword)
- (m/update-existing-in [:admin_details :auth-provider] keyword)))))]
+ (m/update-existing-in [:write_data_details :auth-provider] keyword)))))]
(cond-> database
;; TODO - this is only really needed for API responses. This should be a `hydrate` thing instead!
(and driver
@@ -441,7 +431,7 @@
(when (not= :postgres (mdb/db-type))
(warehouses.db/delete-cards-for-database! id))
(try
- (driver/notify-database-updated driver database)
+ (driver/notify-database-updated driver (driver.u/ensure-lib-database database))
(catch Throwable e
(log/errorf "Error sending database deletion notification: %s" (ex-message e)))))
@@ -462,7 +452,7 @@
(def ^:private details-keys
"Every place a Database stores a set of connection details."
- [:details :write_data_details :admin_details])
+ [:details :write_data_details])
(defn- validate-connection-details!
"Refuse to store details pointing at a private/internal network address, or setting a connection parameter the driver
@@ -526,8 +516,7 @@
infer-db-schedules
(or (some? (:details changes))
- (some? (:write_data_details changes))
- (some? (:admin_details changes)))
+ (some? (:write_data_details changes)))
secret/handle-incoming-client-secrets!
(:uploads_enabled changes)
@@ -618,7 +607,7 @@
driver.u/default-sensitive-fields))
(methodical/defmethod mi/to-json :model/Database
- "When encoding a Database as JSON remove the `details`, `write_data_details`, `admin_details`, and
+ "When encoding a Database as JSON remove the `details`, `write_data_details`, and
`initial_sync_error` for any User without write perms for the DB. Users with write perms can see the details but remove anything resembling a
password. No one gets to see this in an API response!
@@ -632,13 +621,12 @@
(next-method
(let [db (if (not (mi/can-write? db))
(do (log/debug "Fully redacting database details during json encoding.")
- (dissoc db :details :write_data_details :admin_details :initial_sync_error))
+ (dissoc db :details :write_data_details :initial_sync_error))
(do (log/debug "Redacting sensitive fields within database details during json encoding.")
(-> db
(secret/to-json-hydrate-redacted-secrets)
(update :details redact-sensitive-fields)
- (m/update-existing :write_data_details redact-sensitive-fields)
- (m/update-existing :admin_details redact-sensitive-fields))))]
+ (m/update-existing :write_data_details redact-sensitive-fields))))]
(update db :settings
(fn [settings]
(when (map? settings)
@@ -689,7 +677,6 @@
:metadata_sync_schedule schedule-transform
:details details-transform
:write_data_details details-transform
- :admin_details details-transform
:creator_id (serdes/fk :model/User)
:router_database_id (serdes/fk :model/Database)
:initial_sync_status {:export-with-context (fn [_current _ _status] ::serdes/skip)
@@ -761,8 +748,7 @@
(assert-sqlite-not-hosted! ingested)
(serdes/default-load-one! (cond-> ingested
(:details ingested) (update :details driver/sanitize-db-details)
- (:write_data_details ingested) (update :write_data_details driver/sanitize-db-details)
- (:admin_details ingested) (update :admin_details driver/sanitize-db-details))
+ (:write_data_details ingested) (update :write_data_details driver/sanitize-db-details))
maybe-local))
(def ^{:arglists '([table-id])} table-id->database-id
diff --git a/src/metabase/warehouses/schema.clj b/src/metabase/warehouses/schema.clj
index 865665abbcc..1ba468416c6 100644
--- a/src/metabase/warehouses/schema.clj
+++ b/src/metabase/warehouses/schema.clj
@@ -34,10 +34,6 @@
"The `:write_data_details` column of a Database, decoded."
[:or ms/DatabaseDetails ::database.undecryptable-column])
-(mr/def ::database.admin-details
- "The `:admin_details` column of a Database, decoded."
- [:or ms/DatabaseDetails ::database.undecryptable-column])
-
(mr/def ::database
"A Database as selected from the app DB: every column of `:metabase_database`, plus `:features` added by the model's after-select hook."
[:merge
@@ -89,7 +85,6 @@
[:router_database_id {:optional true} [:maybe ::lib.schema.id/database]]
[:provider_name {:optional true} [:maybe :string]]
[:write_data_details {:optional true} [:maybe ::database.write-data-details]]
- [:admin_details {:optional true} [:maybe ::database.admin-details]]
[:is_stub {:optional true} [:maybe :boolean]]])
(mr/def ::database-or-metadata
diff --git a/src/metabase/warehouses_rest/api.clj b/src/metabase/warehouses_rest/api.clj
index dfeeb194aea..d1edbab3e39 100644
--- a/src/metabase/warehouses_rest/api.clj
+++ b/src/metabase/warehouses_rest/api.clj
@@ -813,7 +813,7 @@
:database engine
:database-id (u/the-id <>)
:source connection_source
- :dbms-version (:version (driver/dbms-version (keyword engine) <>))}))
+ :dbms-version (:version (driver/dbms-version (keyword engine) (driver.u/lib-database (u/the-id <>))))}))
;; failed to connect, return error
(do
(analytics/track-event! :snowplow/database
@@ -1014,9 +1014,9 @@
(let [would-be-database (cond-> existing-database
details-with-secrets (assoc :details details-with-secrets)
write-data-details-with-secrets (assoc :write_data_details write-data-details-with-secrets))]
- (driver.conn/with-write-connection
- (warehouses/test-database-connection (or engine (:engine would-be-database))
- (driver.conn/effective-details would-be-database)))))
+ (warehouses/test-database-connection (or engine (:engine would-be-database))
+ (merge (driver.conn/details-for-exact-type would-be-database :default)
+ (driver.conn/details-for-exact-type would-be-database :write-data)))))
full-sync? (some-> is_full_sync boolean)
on-demand? (boolean is_on_demand)]
(if-let [conn-error (or main-conn-error write-conn-error)]
@@ -1149,10 +1149,10 @@
(analytics/track-event! :snowplow/simple_event {:event "database_manual_sync" :target_id id})
(quick-task/submit-task!
(fn []
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
;; explicit, user-requested sync — runs even when `disable-auto-sync` is enabled
- (sync/sync-db-metadata-explicit! db)
- (sync/analyze-db-explicit! db))))
+ (sync/sync-db-metadata-explicit! (u/the-id db))
+ (sync/analyze-db-explicit! (u/the-id db)))))
{:status :ok}))))
;; TODO (Cam 10/28/25) -- fix this endpoint route to use kebab-case for consistency with the rest of our REST API
@@ -1207,12 +1207,12 @@
;; but no data perms, they should stll be able to trigger a sync of field values. This is fine because we don't
;; return any actual field values from this API. (#21764)
(request/as-admin
- (database-routing/with-database-routing-off
+ (database-routing/with-anonymous-router-access
(if *rescan-values-async*
(quick-task/submit-task!
(fn []
- (sync/update-field-values! db)))
- (sync/update-field-values! db)))))
+ (sync/update-field-values! (u/the-id db))))
+ (sync/update-field-values! (u/the-id db))))))
{:status :ok})
(defn- delete-all-field-values-for-database! [database-or-id]
@@ -1255,7 +1255,7 @@
(perms/has-db-transforms-permission? api/*current-user-id* (:id db))
(and (mi/can-write? db)
(mi/can-read? db))))
- (->> db
+ (->> (driver.u/lib-database (:id db))
(driver/syncable-schemas (:engine db))
(vec)
(sort))))
diff --git a/test/metabase/actions/actions_test.clj b/test/metabase/actions/actions_test.clj
index f0e7e30ecf5..d5c00ebc9e2 100644
--- a/test/metabase/actions/actions_test.clj
+++ b/test/metabase/actions/actions_test.clj
@@ -782,7 +782,7 @@
(mt/with-temp [:model/Database test-database {:engine driver/*driver*, :details test-user-details :settings {:database-enable-actions true
:database-enable-table-editing true}}]
;; Sync the database to get table metadata
- (sync/sync-database! test-database)
+ (sync/sync-database! (:id test-database))
(let [table-id (t2/select-one-pk :model/Table :db_id (:id test-database) :name "categories")]
(testing (str "INSERT permission denied for " driver/*driver*)
(let [result (try
@@ -841,7 +841,7 @@
:write-data
:default))]
(try
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id))
(testing "write pool does not exist before action execution"
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))))
(binding [*current-user-permissions-set* (delay #{"/"})]
@@ -851,7 +851,7 @@
(testing "write pool is created during action execution"
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key)))
(finally
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db)))))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id)))))))))
(deftest implicit-action-audit-trail-test
(testing "an implicit action records one QueryExecution row holding an action descriptor, not SQL"
diff --git a/test/metabase/actions/execution_write_pool_test.clj b/test/metabase/actions/execution_write_pool_test.clj
index 605617d6410..c00755a9077 100644
--- a/test/metabase/actions/execution_write_pool_test.clj
+++ b/test/metabase/actions/execution_write_pool_test.clj
@@ -19,7 +19,7 @@
:write-data
:default))]
(try
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id))
(testing "write pool does not exist before query action execution"
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))))
(mt/with-test-user :crowberto
@@ -31,4 +31,4 @@
(testing "write pool is created during query action execution"
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key)))
(finally
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db)))))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id)))))))))
diff --git a/test/metabase/actions/models_test.clj b/test/metabase/actions/models_test.clj
index 991e22c54db..0ebd867594b 100644
--- a/test/metabase/actions/models_test.clj
+++ b/test/metabase/actions/models_test.clj
@@ -491,7 +491,7 @@
"GRANT ALL ON \"DEFAULT_UUID\" TO GUEST;"
"GRANT ALL ON \"REQUIRED_UUID\" TO GUEST;"]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(testing "make sure we synced the fields correctly"
(is (= [{:database_is_auto_increment false
:database_required false
@@ -544,7 +544,7 @@
"CREATE TABLE \"FOO\" (\"id\" IDENTITY PRIMARY KEY, \"toggle\" BOOLEAN);"
"GRANT ALL ON \"FOO\" TO GUEST;"]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(mt/with-actions-enabled
(mt/with-actions [{model-id :id} {:type :model, :dataset_query (mt/mbql-query foo)}]
(let [action-data {:type :implicit
@@ -565,7 +565,7 @@
"CREATE TABLE \"FOO\" (\"id\" IDENTITY PRIMARY KEY, \"name\" TEXT);"
"GRANT ALL ON \"FOO\" TO GUEST;"]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(mt/with-actions-enabled
(mt/with-actions [{model-id :id} {:type :model, :dataset_query (mt/mbql-query foo)}]
(let [action-data {:type :implicit
@@ -574,7 +574,7 @@
:model_id model-id}
action-id (action/insert! action-data)
exec! #(do (jdbc/execute! one-off-dbs/*conn* [%])
- (sync/sync-database! (mt/db)))]
+ (sync/sync-database! (mt/id)))]
(testing "if a column type is varied the parameter types are updated on select-action"
(exec! "ALTER TABLE \"FOO\" ALTER COLUMN \"name\" BIGINT;")
(is (= [["name" :number]] (map (juxt :id :type) (:parameters (action/select-action action-id)))))
@@ -582,7 +582,7 @@
(let [action (action/select-action action-id)]
(action/update! (assoc action :name "create foo2") action)
(exec! "ALTER TABLE \"FOO\" ALTER COLUMN \"name\" TEXT;")
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= [["name" :text]] (map (juxt :id :type) (:parameters (action/select-action action-id))))))))
(testing "if a column added, the model column set is used, not the table"
(exec! "ALTER TABLE \"FOO\" ADD COLUMN \"name2\" BIGINT;")
diff --git a/test/metabase/actions/test_util.clj b/test/metabase/actions/test_util.clj
index 8317777d0e2..8815d17c639 100644
--- a/test/metabase/actions/test_util.clj
+++ b/test/metabase/actions/test_util.clj
@@ -170,7 +170,7 @@
(row-count))))
(testing "delete row"
(is (= [1]
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (data/id))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database (data/id)))
(delete-categories-1-query)))))
(testing "after"
(is (= [[74]]
diff --git a/test/metabase/analytics/prometheus_test.clj b/test/metabase/analytics/prometheus_test.clj
index 5e118befb46..d42cbff053b 100644
--- a/test/metabase/analytics/prometheus_test.clj
+++ b/test/metabase/analytics/prometheus_test.clj
@@ -9,6 +9,7 @@
[metabase.app-db.connection-pool-setup :as mdb.connection-pool-setup]
[metabase.app-db.data-source :as mdb.data-source]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.driver.util :as driver.u]
[metabase.search.engine :as search.engine]
[metabase.test :as mt]
[metabase.test.fixtures :as fixtures]
@@ -200,7 +201,7 @@
;; start once scraping is under way, so a wedged guard is what this thread would run into
(when (true? (deref first-scrape timeout-ms false))
(try
- (sql-jdbc.conn/db->pooled-connection-spec db)
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database db))
(deliver warehouse-built true)
(catch Exception e
(swap! errors conj e))))))
@@ -214,7 +215,7 @@
(is (empty? @errors))
(finally
(reset! stop? true)
- (sql-jdbc.conn/invalidate-pool-for-db! db)))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id db))))))))
(deftest email-collector-test
(testing "Registry has email metrics registered"
diff --git a/test/metabase/app_db/connection_pool_setup_test.clj b/test/metabase/app_db/connection_pool_setup_test.clj
index 91f151bdf0a..2dfa15e0425 100644
--- a/test/metabase/app_db/connection_pool_setup_test.clj
+++ b/test/metabase/app_db/connection_pool_setup_test.clj
@@ -9,6 +9,7 @@
[metabase.config.core :as config]
[metabase.connection-pool :as connection-pool]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.driver.util :as driver.u]
[metabase.test :as mt]
[metabase.test.fixtures :as fixtures]
[metabase.util :as u]
@@ -17,7 +18,7 @@
(com.mchange.v2.c3p0 C3P0Registry ConnectionCustomizer DataSources PoolBackedDataSource WrapperConnectionPoolDataSource)
(java.lang.management ManagementFactory)
(javax.management ObjectName)
- (metabase.app_db.connection_pool_setup MetabaseConnectionCustomizer)))
+ (metabase.app_db.connection_pool_setup AppDbConnectionCustomizer)))
(set! *warn-on-reflection* true)
@@ -67,26 +68,26 @@
;; a fresh database, so this builds a pool rather than finding a cached one
(mt/with-temp [:model/Database db {:engine :h2, :details {:db (str "mem:" (mt/random-name))}}]
(try
- (sql-jdbc.conn/db->pooled-connection-spec db)
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database db))
(is (empty? (c3p0-beans)))
(finally
- (sql-jdbc.conn/invalidate-pool-for-db! db)))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id db))))))
(finally
(connection-pool/destroy-connection-pool! app-pool))))))
-(deftest MetabaseConnectionCustomizer-test
+(deftest AppDbConnectionCustomizer-test
(testing "connection customizer is registered"
- (let [customizer (C3P0Registry/getConnectionCustomizer (.getName MetabaseConnectionCustomizer))]
+ (let [customizer (C3P0Registry/getConnectionCustomizer (.getName AppDbConnectionCustomizer))]
(is (some? customizer) "ConnectionCustomizer is not registered with c3p0")
(is (instance? ConnectionCustomizer customizer)
"checkin tracker must satisfy the c3p0 ConnectionCustomizer interface")
- (is (instance? MetabaseConnectionCustomizer customizer)
- "ConnectionCustomizer is not an instance of our MetabaseConnectionCustomizer")))
+ (is (instance? AppDbConnectionCustomizer customizer)
+ "ConnectionCustomizer is not an instance of our AppDbConnectionCustomizer")))
(testing "db activity resets counter"
(try
(let [updated? (promise)]
(add-watch (var-get #'mdb.connection-pool-setup/latest-activity)
- ::MetabaseConnectionCustomizer-test
+ ::AppDbConnectionCustomizer-test
(fn [_key _ref _old-state _new-state]
(deliver updated? ::completed)))
(reset! (var-get #'mdb.connection-pool-setup/latest-activity) nil)
@@ -98,7 +99,7 @@
(is (instance? java.time.temporal.Temporal recent-checkin)
"recent-checkin should be a temporal type (OffsetDateTime)")))
(finally (remove-watch (var-get #'mdb.connection-pool-setup/latest-activity)
- ::MetabaseConnectionCustomizer-test)))))
+ ::AppDbConnectionCustomizer-test)))))
(deftest recent-activity-test
;; these tests are difficult to make non-flaky. Other threads can hit the db of course, and the lifecycle of the
;; connection pool is worked from other threads. This means we can't isolate the `latest-checkin` atom. Many will take
diff --git a/test/metabase/auth_provider/impl_test.clj b/test/metabase/auth_provider/impl_test.clj
index f20df9eb647..c98379ed1b1 100644
--- a/test/metabase/auth_provider/impl_test.clj
+++ b/test/metabase/auth_provider/impl_test.clj
@@ -32,7 +32,7 @@
(testing "Connection tests"
(is (nil? (warehouses/test-database-connection (:engine db) (:details db)))))
(testing "Syncing does not blow up"
- (sync/sync-database! (mt/db)))
+ (sync/sync-database! (mt/id)))
(testing "Querying"
(is (= [["Polo Lounge"]]
(mt/rows (mt/run-mbql-query venues {:filter [:= $id 60] :fields [$name]}))))))))))))
diff --git a/test/metabase/core/core_test.clj b/test/metabase/core/core_test.clj
index 6abfd9420f5..d7c6cee7351 100644
--- a/test/metabase/core/core_test.clj
+++ b/test/metabase/core/core_test.clj
@@ -17,7 +17,7 @@
(let [h2-db (t2/insert-returning-instance! :model/Database
{:name "Sample Database" :engine :h2 :is_sample true
:details (#'sample-data.impl/try-to-extract-sample-database! :h2)})]
- (sync/sync-database! h2-db)
+ (sync/sync-database! (:id h2-db))
(let [before-tables (t2/select-fn-set :id :model/Table :db_id (:id h2-db))]
(#'core/reconcile-sample-database!)
(let [after (t2/select-one :model/Database :id (:id h2-db))]
@@ -37,7 +37,7 @@
(let [db (t2/insert-returning-instance! :model/Database
{:name "Sample Database" :engine :sqlite :is_sample true
:details (#'sample-data.impl/try-to-extract-sample-database! :sqlite)})
- _ (sync/sync-database! db)
+ _ (sync/sync-database! (:id db))
messages (mt/with-log-messages-for-level [messages [metabase.sample-data.impl :error]]
(#'core/reconcile-sample-database!)
(messages))]
diff --git a/test/metabase/driver/common_test.clj b/test/metabase/driver/common_test.clj
index fe65d0c40d3..c73f611ef3f 100644
--- a/test/metabase/driver/common_test.clj
+++ b/test/metabase/driver/common_test.clj
@@ -7,14 +7,18 @@
[metabase.driver.common :as driver.common]
[metabase.driver.mysql :as mysql]
[metabase.driver.util :as driver.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.premium-features.core :as premium-features]
[metabase.settings.core :as setting]
[metabase.test :as mt]
[metabase.test.data.sql :as sql.tx]
+ [metabase.test.fixtures :as fixtures]
[metabase.util.json :as json]))
(set! *warn-on-reflection* true)
+(use-fixtures :once (fixtures/initialize :db))
+
(deftest ^:parallel base-type-inference-test
(is (= :type/Text
(transduce identity (driver.common/values->base-type) ["A" "B" "C"])))
@@ -134,8 +138,8 @@
:columns columns}
data-source)]
(when-let [inserted-rows (and (not (#{:bigquery-cloud-sdk :snowflake :mongo} driver)) ;table-rows-seq not implemented
- (driver/table-rows-seq driver/*driver* (mt/db) {:name table-name
- :schema schema-name}))]
+ (driver/table-rows-seq driver/*driver* (lib.metadata/database (mt/metadata-provider)) {:name table-name
+ :schema schema-name}))]
(is (= (count data) (count inserted-rows)) "Should insert all test rows including nulls"))))
(finally
(driver/drop-table! driver db-id qualified-table-name))))))))
@@ -175,8 +179,8 @@
:columns columns}
data-source)]
(when-let [inserted-rows (and (not (#{:bigquery-cloud-sdk :snowflake :mongo} driver)) ;table-rows-seq not implemented
- (driver/table-rows-seq driver/*driver* (mt/db) {:name table-name
- :schema schema-name}))]
+ (driver/table-rows-seq driver/*driver* (lib.metadata/database (mt/metadata-provider)) {:name table-name
+ :schema schema-name}))]
(is (= (count data) (count inserted-rows))
"Should insert all test rows from JSONL file including nulls")))
(finally
diff --git a/test/metabase/driver/connection_test.clj b/test/metabase/driver/connection_test.clj
index 5cb38e936e8..d9d17d1f207 100644
--- a/test/metabase/driver/connection_test.clj
+++ b/test/metabase/driver/connection_test.clj
@@ -31,14 +31,6 @@
:id 1
:details details
:write-data-details write-data-details}]
- (driver.conn/with-write-connection
- (is (=? write-data-details
- (driver.conn/effective-details database))))))
- (testing "with snake_case key (Toucan2 instance style)"
- (let [database (t2/instance :model/Database
- {:id 1
- :details details
- :write_data_details write-data-details})]
(driver.conn/with-write-connection
(is (=? write-data-details
(driver.conn/effective-details database)))))))))))
@@ -75,47 +67,19 @@
(is (=? details
(driver.conn/effective-details database))))))))
-(deftest effective-details-nil-test
- (testing "effective-details returns nil when database is nil"
- (is (nil? (driver.conn/effective-details nil)))
- (mt/with-premium-features #{:writable-connection}
- (driver.conn/with-write-connection
- (is (nil? (driver.conn/effective-details nil)))))))
+(deftest effective-details-refuses-database-instance-test
+ (testing "effective-details takes only the Lib metadata of a Database, so a Database instance fails fast"
+ (is (thrown? clojure.lang.ExceptionInfo
+ (driver.conn/effective-details (t2/select-one :model/Database (mt/id)))))))
(deftest connection-telemetry-info-test
(testing "connection-telemetry-info describes the current connection context as prose, for logging only"
(is (= "the default connection" (driver.conn/connection-telemetry-info)))
(driver.conn/with-write-connection
(is (= "the write-data connection" (driver.conn/connection-telemetry-info))))
- (driver.conn/with-admin-connection
- (is (= "the admin connection" (driver.conn/connection-telemetry-info))))
(driver.conn/with-transform-connection
(is (= "the transform connection" (driver.conn/connection-telemetry-info))))))
-(deftest effective-details-default-with-admin-details-test
- (testing "effective-details returns :details when *connection-type* is :default, even when :admin-details present"
- (let [details {:host "read-host" :port 5432}
- admin-details {:host "admin-host" :port 5432 :user "root"}
- database {:lib/type :metadata/database
- :id 1
- :details details
- :admin-details admin-details}]
- (is (=? details
- (driver.conn/effective-details database))))))
-
-(deftest effective-details-admin-without-feature-test
- (testing "without an admin-overlay feature, with-admin-connection falls back to main connection details"
- (mt/with-premium-features #{}
- (let [details {:host "read-host" :port 5432}
- admin-details {:host "admin-host" :port 5432 :user "root"}
- database {:lib/type :metadata/database
- :id 1
- :details details
- :admin-details admin-details}]
- (driver.conn/with-admin-connection
- (is (=? details
- (driver.conn/effective-details database))))))))
-
(deftest type-resolved-metric-test
(mt/when-ee-evailable
(mt/with-premium-features #{:writable-connection}
diff --git a/test/metabase/driver/h2_test.clj b/test/metabase/driver/h2_test.clj
index 9de358a5fe4..e57182f750e 100644
--- a/test/metabase/driver/h2_test.clj
+++ b/test/metabase/driver/h2_test.clj
@@ -17,8 +17,10 @@
[metabase.driver.sql-jdbc.actions :as sql-jdbc.actions]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql.query-processor :as sql.qp]
+ [metabase.driver.util :as driver.u]
[metabase.lib-be.core :as lib-be]
[metabase.lib.core :as lib]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.query-processor.compile :as qp.compile]
[metabase.query-processor.preprocess :as qp.preprocess]
[metabase.query-processor.test :as qp]
@@ -32,12 +34,12 @@
(mt/test-driver :h2
(testing "default"
(is (= "PUBLIC"
- (driver.sql/default-schema :h2 (mt/db)))))
+ (driver.sql/default-schema :h2 (lib.metadata/database (mt/metadata-provider))))))
(testing "schema configured in the JDBC URL"
(let [details (update (:details (mt/db)) :db str ";SCHEMA=INFORMATION_SCHEMA")]
(mt/with-temp [:model/Database database {:engine :h2, :details details}]
(is (= "INFORMATION_SCHEMA"
- (driver.sql/default-schema :h2 database))))))))
+ (driver.sql/default-schema :h2 (driver.u/lib-database (:id database))))))))))
(deftest database-error-while-reading-rows-is-invalid-query-test
(mt/test-driver :h2
@@ -142,7 +144,7 @@
(deftest ^:parallel db-default-timezone-test
(mt/test-driver :h2
(is (= (t/zone-id "Z")
- (-> (driver/db-default-timezone :h2 (mt/db))
+ (-> (driver/db-default-timezone :h2 (lib.metadata/database (mt/metadata-provider)))
t/zone-id
.normalized)))))
@@ -263,7 +265,7 @@
(deftest ^:parallel check-action-commands-test
(mt/test-driver :h2
- (are [query] (= true (#'h2/every-command-allowed-for-actions? (#'h2/classify-query (mt/id) query)))
+ (are [query] (= true (#'h2/every-command-allowed-for-actions? (#'h2/classify-query (lib.metadata/database (mt/metadata-provider)) query)))
"select 1"
"update venues set name = 'bill'"
"delete venues"
@@ -283,7 +285,7 @@
"insert into venues (name) values ('bill')"
"create table venues"
"alter table venues add column address varchar(255)")
- (are [query] (= false (#'h2/every-command-allowed-for-actions? (#'h2/classify-query (mt/id) query)))
+ (are [query] (= false (#'h2/every-command-allowed-for-actions? (#'h2/classify-query (lib.metadata/database (mt/metadata-provider)) query)))
"select * from venues; update venues set name = 'stomp';
CREATE ALIAS EXEC AS 'String shellexec(String cmd) throws java.io.IOException {Runtime.getRuntime().exec(cmd);return \"y4tacker\";}';
EXEC ('open -a Calculator.app')"
@@ -422,7 +424,7 @@
(testing "`syncable-schemas` should return schemas that should be synced"
(mt/with-empty-db
(is (= #{"PUBLIC"}
- (driver/syncable-schemas driver/*driver* (mt/db))))))))
+ (driver/syncable-schemas driver/*driver* (lib.metadata/database (mt/metadata-provider)))))))))
(deftest syncable-audit-db-test
(mt/test-driver :h2
@@ -435,7 +437,7 @@
(testing "spec obtained from audit db has no connection string, and that works OK."
(let [audit-db-id (t2/select-one-fn :id 'Database :is_audit true)]
(is (= audit-db-expected-id audit-db-id))
- (let [audit-db-pooled-spec (sql-jdbc.conn/db->pooled-connection-spec audit-db-id)]
+ (let [audit-db-pooled-spec (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database audit-db-id))]
(is (= "com.mchange.v2.c3p0.PoolBackedDataSource" (pr-str (type (:datasource audit-db-pooled-spec)))))
(let [spec (sql-jdbc.conn/connection-details->spec :h2 audit-db-pooled-spec)]
(is (= #{:classname :subprotocol :subname :datasource}
diff --git a/test/metabase/driver/mysql_test.clj b/test/metabase/driver/mysql_test.clj
index cbc79955f18..99335b5ad6e 100644
--- a/test/metabase/driver/mysql_test.clj
+++ b/test/metabase/driver/mysql_test.clj
@@ -27,6 +27,7 @@
[metabase.driver.sql-jdbc.sync.describe-database :as sql-jdbc.describe-database]
[metabase.driver.sql-jdbc.sync.interface :as sql-jdbc.sync.interface]
[metabase.driver.sql.query-processor :as sql.qp]
+ [metabase.driver.util :as driver.u]
[metabase.lib-be.core :as lib-be]
[metabase.lib.card :as lib.card]
[metabase.lib.core :as lib]
@@ -117,7 +118,7 @@
(deftest default-schema-test
(mt/test-driver :mysql
- (is (nil? (driver.sql/default-schema :mysql (mt/db))))))
+ (is (nil? (driver.sql/default-schema :mysql (lib.metadata/database (mt/metadata-provider)))))))
(use-fixtures :each (fn [thunk]
;; 1. If sync fails when loading a test dataset, don't swallow the error; throw an Exception so we
@@ -172,7 +173,7 @@
(jdbc/execute! spec [sql]))
;; create & sync MB DB
(mt/with-temp [:model/Database database {:engine "mysql", :details details}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(mt/with-db database
;; run the query
(is (= [[1 nil]]
@@ -199,7 +200,7 @@
:let [sql (str/join "\n" lines)]]
(jdbc/execute! spec [sql]))
(mt/with-temp [:model/Database database {:engine "mysql", :details details}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(is (= [{:pk-table-schema nil
:pk-table-name "foo"
:pk-column-name "id"
@@ -208,7 +209,7 @@
:fk-column-name "foo_id"}]
(into []
(driver/describe-fks driver/*driver*
- (lib-be/instance->metadata database :metadata/database)
+ (driver.u/ensure-lib-database (lib-be/instance->metadata database :metadata/database))
{:table-names #{"foo" "bar"}})))))))))
(deftest multiple-schema-test
@@ -223,9 +224,9 @@
(doseq [details [(tx/dbdef->connection-details :mysql :db {:database-name "dbone"})
(set/rename-keys (tx/dbdef->connection-details :mysql :db {:database-name "dbone"}) {:db :dbname})]]
(mt/with-temp [:model/Database database {:engine "mysql", :details details}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(is (= #{"dbone_a" "dbone_b" "dbone_c"}
- (into #{} (map :name) (driver/describe-fields :mysql database)))))))))
+ (into #{} (map :name) (driver/describe-fields :mysql (driver.u/ensure-lib-database database))))))))))
(deftest ^:parallel hour-bucketing-time-without-database-type-test
(testing (str "Hour bucketing on a TIME-typed expression without `:database-type` (as happens for "
@@ -292,7 +293,7 @@
(mt/test-driver :mysql
(mt/dataset tiny-int-ones
;; trigger a full sync on this database so fields are categorized correctly
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(testing "By default TINYINT(1) should be a boolean"
(is (= #{{:name "number-of-cans", :base_type :type/Boolean, :semantic_type nil}
{:name "id", :base_type :type/Integer, :semantic_type :type/PK}
@@ -302,7 +303,7 @@
(mt/with-temp [:model/Database db {:engine "mysql"
:details (assoc (:details (mt/db))
:additional-options "tinyInt1isBit=false")}]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (= #{{:name "number-of-cans", :base_type :type/Integer, :semantic_type :type/Quantity}
{:name "id", :base_type :type/Integer, :semantic_type :type/PK}
{:name "thing", :base_type :type/Text, :semantic_type :type/Category}}
@@ -312,7 +313,7 @@
(mt/test-driver :mysql
(mt/dataset tiny-int-ones
;; trigger a full sync on this database so fields are categorized correctly
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(testing "Can filter on TINYINT(1) boolean columns in models (metabase#65826)"
(let [mp (as-> (mt/metadata-provider) $mp
(lib.tu/mock-metadata-provider $mp {:cards
@@ -363,7 +364,7 @@
(str/includes? sql-args "GLOBAL.time_zone"))
[result-row]
(apply orig spec sql-args options))))]
- (driver/db-default-timezone driver/*driver* db))))]
+ (driver/db-default-timezone driver/*driver* (driver.u/ensure-lib-database db)))))]
(testing "Should use global timezone by default"
(is (= "US/Pacific"
(timezone {:global_tz "US/Pacific", :system_tz "UTC"}))))
@@ -381,7 +382,7 @@
(timezone {:global_tz "PDT", :system_tz "UTC", :offset "00:00"})))))
(testing "real timezone query doesn't fail"
(is (nil? (try
- (driver/db-default-timezone driver/*driver* (mt/db))
+ (driver/db-default-timezone driver/*driver* (lib.metadata/database (mt/metadata-provider)))
nil
(catch Throwable e
e)))))))
@@ -594,7 +595,7 @@
(throw se))))]
(when compat
(mt/with-temp [:model/Database database {:engine "mysql", :details details}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(is (= [{:name "src1"
:fields [{:name "id"
:base_type :type/Integer}
@@ -618,7 +619,7 @@
(mt/test-driver :mysql
(mt/with-empty-db
(create-enums-table! (mt/db))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(testing "ENUM columns are synced with the correct base and semantic types"
(is (=? {:base_type :type/MySQLEnum
:semantic_type :type/Category}
@@ -635,7 +636,7 @@
(testing "actions with enums"
(mt/with-empty-db
(create-enums-table! (mt/db))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(mt/with-actions-enabled
(mt/with-actions [model {:type :model
:dataset_query (mt/mbql-query birds)}
@@ -783,7 +784,7 @@
(mt/with-temp
[:model/Database database {:name "composite_pks_test" :engine driver/*driver* :details details}]
(mt/with-db database
- (sync-tables/sync-tables-and-database! database)
+ (sync-tables/sync-tables-and-database! (driver.u/lib-database (:id database)))
(is (= #{{:name "json_val → int_turn_string",
:database-type "text"
:base-type :type/Text
@@ -793,7 +794,7 @@
:nfc-path [:json_val "int_turn_string"]}}
(sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- database
+ (driver.u/lib-database (:id database))
(t2/select-one :model/Table :db_id (mt/id) :name "json_table"))))))))))))
(deftest json-unfolding-ignores-table-differing-only-in-case-test
@@ -808,7 +809,7 @@
"INSERT INTO `orders` VALUES (1, '{\"a\": 1}');"]]
(jdbc/execute! spec [statement]))
(mt/with-temp [:model/Database database {:engine driver/*driver* :details details}]
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(is (= #{"id" "data" "data → a"}
(t2/select-fn-set :name :model/Field
:table_id (t2/select-one-pk :model/Table :db_id (:id database) :name "orders")
@@ -871,7 +872,7 @@
(deftest ddl-execute-with-timeout-test
(mt/test-driver :mysql
(mt/dataset json
- (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))]
+ (let [db-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))]
(testing "When the query takes longer that the timeout, it is killed."
(is (thrown-with-msg?
Exception
@@ -885,7 +886,7 @@
(testing "`syncable-schemas` should return an empty set because mysql doesn't support schemas"
(mt/with-empty-db
(is (= #{}
- (driver/syncable-schemas driver/*driver* (mt/db))))))))
+ (driver/syncable-schemas driver/*driver* (lib.metadata/database (mt/metadata-provider)))))))))
;;; ------------------------------------------------ Actions related ------------------------------------------------
@@ -959,7 +960,7 @@
(jdbc/execute! (sql-jdbc.conn/connection-details->spec driver/*driver* details) [stmt]))
(mt/with-temp [:model/Database database {:engine driver/*driver* :details details}]
(mt/with-db database
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(mt/with-actions-enabled
(testing "when creating with invalid email"
(is (=? {:errors {}
@@ -991,7 +992,7 @@
(jdbc/execute! (sql-jdbc.conn/connection-details->spec driver/*driver* details) [stmt]))
(mt/with-temp [:model/Database database {:engine driver/*driver* :details details}]
(mt/with-db database
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(mt/with-actions-enabled
(testing "when creating"
(is (=? {:type :metabase.actions.error/violate-unique-constraint
@@ -1156,7 +1157,7 @@
(binding [sql-jdbc.describe-database/*select-probe-query-timeout-seconds* 1]
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (mt/id)
nil
(fn [^java.sql.Connection conn]
(is (true? (sql-jdbc.sync.interface/have-select-privilege?
@@ -1186,7 +1187,7 @@
:ssl true
:additional-options "trustServerCertificate=true")]
(mt/with-temp [:model/Database database {:engine "mysql", :details user-connection-details}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(testing "only fullaccess table has writeable permissions"
(is (= {"readonly_table" false
"readwrite_table" false
@@ -1195,7 +1196,7 @@
(testing "After granting full access to all tables and re-syncing"
(doseq [table-name ["readonly_table" "readwrite_table"]]
(jdbc/execute! spec (format "GRANT INSERT, UPDATE, DELETE ON sync_writable_test.`%s` TO 'sync_writable_test_user'" table-name)))
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(is (= {"readonly_table" true
"readwrite_table" true
"fullaccess_table" true}
@@ -1229,7 +1230,7 @@
(jdbc/execute! spec "SET GLOBAL partial_revokes = OFF;")
(is (true? (driver/database-supports? driver/*driver* :metadata/table-writable-check database))
"Should support metadata/table-writable-check when partial_revokes is OFF")
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(is (= {"writable_table" true, "revoked_table" true}
(t2/select-fn->fn :name :is_writable :model/Table :db_id (:id database)))))
(testing "With partial_revokes ON and INSERT revoked on one table, the check stays enabled"
@@ -1237,7 +1238,7 @@
(jdbc/execute! spec "REVOKE INSERT ON partial_revokes_test.revoked_table FROM 'partial_revokes_test_user';")
(is (true? (driver/database-supports? driver/*driver* :metadata/table-writable-check database))
"Should still support metadata/table-writable-check when partial_revokes is ON")
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
;; The bug (metabase#73276): turning partial_revokes ON disabled the writable check for the whole
;; database, so *every* table became uneditable. The fix removes that blanket gate, so writability is
;; computed per-table from the GRANT lines again: the untouched table stays writable, and the table
@@ -1419,7 +1420,7 @@
delivered (atom 0)
failure (atom nil)]
(sql-jdbc.execute/do-with-connection-with-options
- driver/*driver* db {:write? true}
+ driver/*driver* (driver.u/ensure-lib-database db) {:write? true}
(fn [^Connection conn]
(.setAutoCommit conn false)
(with-open [stmt (.createStatement conn)]
diff --git a/test/metabase/driver/postgres/index_test.clj b/test/metabase/driver/postgres/index_test.clj
index c9cca41a1ce..a437ff2f88a 100644
--- a/test/metabase/driver/postgres/index_test.clj
+++ b/test/metabase/driver/postgres/index_test.clj
@@ -10,6 +10,7 @@
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql.util :as sql.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.test :as mt]))
(deftest ^:parallel supports-standalone-create-test
@@ -145,8 +146,8 @@
(testing "the standalone-create path runs the rendered DDL and then ANALYZE against a real table"
(mt/test-driver :postgres
(mt/with-empty-db
- (let [admin-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
- conn-spec (driver/connection-spec :postgres (mt/db))
+ (let [admin-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
+ conn-spec (driver/connection-spec :postgres (lib.metadata/database (mt/metadata-provider)))
schema "public"]
(doseq [{:keys [label table columns structured expected]} execute-cases]
(testing label
@@ -167,7 +168,7 @@
(driver/compile-create-index :postgres schema table structured))
(is (= expected (index-info admin-spec index-name))))
(testing "refresh-table-stats! runs ANALYZE without error"
- (is (some? (driver/refresh-table-stats! :postgres (mt/db) schema table :table))))
+ (is (some? (driver/refresh-table-stats! :postgres (lib.metadata/database (mt/metadata-provider)) schema table :table))))
(finally
(jdbc/execute! admin-spec [(format "DROP TABLE IF EXISTS %s" qtable)])))))))))))
@@ -175,7 +176,7 @@
(testing "driver/create-index! executes through the same per-driver rendering as compile-create-index"
(mt/test-driver :postgres
(mt/with-empty-db
- (let [admin-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (let [admin-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
db-id (:id (mt/db))]
(jdbc/execute! admin-spec ["DROP TABLE IF EXISTS \"public\".\"perf_idx_legacy\""])
(jdbc/execute! admin-spec ["CREATE TABLE \"public\".\"perf_idx_legacy\" (id INT, user_id INT)"])
diff --git a/test/metabase/driver/postgres_test.clj b/test/metabase/driver/postgres_test.clj
index 97acbd9f068..a95ef286cd6 100644
--- a/test/metabase/driver/postgres_test.clj
+++ b/test/metabase/driver/postgres_test.clj
@@ -31,6 +31,7 @@
[metabase.driver.sql-jdbc.sync.interface :as sql-jdbc.sync.interface]
[metabase.driver.sql.query-processor :as sql.qp]
[metabase.driver.sql.query-processor-test-util :as sql.qp-test-util]
+ [metabase.driver.util :as driver.u]
[metabase.lib-be.metadata.jvm :as lib.metadata.jvm]
[metabase.lib.convert :as lib.convert]
[metabase.lib.core :as lib]
@@ -78,12 +79,12 @@
(mt/test-driver :postgres
(testing "default"
(is (= "public"
- (driver.sql/default-schema :postgres (mt/db)))))
+ (driver.sql/default-schema :postgres (lib.metadata/database (mt/metadata-provider))))))
(testing "schema configured in the JDBC additional options"
(let [details (assoc (:details (mt/db)) :additional-options "currentSchema=information_schema")]
(mt/with-temp [:model/Database database {:engine :postgres, :details details}]
(is (= "information_schema"
- (driver.sql/default-schema :postgres database))))))))
+ (driver.sql/default-schema :postgres (driver.u/ensure-lib-database database)))))))))
(use-fixtures :each (fn [thunk]
;; 1. If sync fails when loading a test dataset, don't swallow the error; throw an Exception so we
@@ -227,7 +228,7 @@
(tx/drop-if-exists-and-create-db! driver/*driver* "hyphen-names-test")
;; create the DB object
(mt/with-temp [:model/Database database {:engine :postgres, :details (assoc details :dbname "hyphen-names-test")}]
- (let [sync! #(sync/sync-database! database)]
+ (let [sync! #(sync/sync-database! (:id database))]
;; populate the DB and create a view
(exec! spec ["CREATE SCHEMA \"x-mas\";"
"CREATE TABLE \"x-mas\".\"presents-and-gifts\" (\"gift-description\" TEXT NOT NULL);"
@@ -240,12 +241,12 @@
:query {:source-table (t2/select-one-pk :model/Table :name "presents-and-gifts")}}))))))))
(testing "Make sure that Schemas / Tables / Fields with backslashes in their names get escaped properly"
(mt/with-empty-db
- (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))]
+ (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))]
(doseq [stmt ["CREATE SCHEMA \"my\\schema\";"
"CREATE TABLE \"my\\schema\".\"my\\table\" (\"my\\field\" INTEGER);"
"INSERT INTO \"my\\schema\".\"my\\table\" (\"my\\field\") VALUES (42);"]]
(jdbc/execute! conn-spec stmt))
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(is (= [[42]]
(mt/rows (qp/process-query
{:database (mt/id)
@@ -305,12 +306,12 @@
ANALYZE test_mview;"])
(mt/with-temp [:model/Database database {:engine :postgres, :details (assoc details :dbname "materialized_views_test")}]
(is (=? [(default-table-result "test_mview")]
- (describe-database->tables :postgres database)))
+ (describe-database->tables :postgres (driver.u/lib-database (:id database)))))
(is (=? [{:table-schema "public"
:table-name "test_mview"
:name "true_facts"
:field-comment "Facts that are true"}]
- (into [] (driver/describe-fields :postgres database {:table-names ["test_mview"]})))))))))
+ (into [] (driver/describe-fields :postgres (driver.u/ensure-lib-database database) {:table-names ["test_mview"]})))))))))
(deftest foreign-tables-test
(mt/test-driver :postgres
@@ -344,7 +345,7 @@
(mt/with-temp [:model/Database database {:engine :postgres, :details (assoc details :dbname "fdw_test")}]
(is (=? [(default-table-result "foreign_table")
(default-table-result "local_table" {:estimated_row_count nil})]
- (describe-database->tables :postgres database))))))))
+ (describe-database->tables :postgres (driver.u/lib-database (:id database))))))))))
(deftest estimated-row-count-hides-zero-test
(mt/test-driver :postgres
@@ -356,7 +357,7 @@
ANALYZE empty_table;"])
(mt/with-temp [:model/Database database {:engine :postgres, :details (assoc details :dbname "estimated_count_test")}]
(is (=? [(default-table-result "empty_table" {:estimated_row_count nil})]
- (describe-database->tables :postgres database))))))))
+ (describe-database->tables :postgres (driver.u/lib-database (:id database))))))))))
(deftest recreated-views-test
(mt/test-driver :postgres
@@ -368,7 +369,7 @@
(tx/drop-if-exists-and-create-db! driver/*driver* "dropped_views_test")
;; create the DB object
(mt/with-temp [:model/Database database {:engine :postgres, :details (assoc details :dbname "dropped_views_test")}]
- (let [sync! #(sync/sync-database! database)]
+ (let [sync! #(sync/sync-database! (:id database))]
;; populate the DB and create a view
(exec! spec ["CREATE table birds (name VARCHAR UNIQUE NOT NULL);"
"INSERT INTO birds (name) VALUES ('Rasta'), ('Lucky'), ('Parroty');"
@@ -408,7 +409,7 @@
(if (>= major-v 10)
;; create the DB object
(mt/with-temp [:model/Database database {:engine :postgres, :details (assoc details :dbname db-name)}]
- (let [sync! #(sync/sync-database! database)]
+ (let [sync! #(sync/sync-database! (:id database))]
;; create a main partitioned table and two partitions for it
(exec! spec ["CREATE TABLE part_vals (val bigint NOT NULL) PARTITION BY RANGE (\"val\");"
"CREATE TABLE part_vals_0 (val bigint NOT NULL);"
@@ -422,7 +423,7 @@
(sync!)
;; all three of these tables should appear in the metadata (including, importantly, the "main" table)
(is (=? (map default-table-result ["part_vals" "part_vals_0" "part_vals_1"])
- (describe-database->tables :postgres database)))))
+ (describe-database->tables :postgres (driver.u/lib-database (:id database)))))))
(log/warn
(u/format-color
'yellow
@@ -786,7 +787,7 @@
"INSERT INTO bobdobbs.describe_json_table (trivial_json) VALUES ('{\"a\": 1}');")])
(mt/with-temp [:model/Database database {:engine :postgres, :details details}]
(mt/with-db database
- (sync-tables/sync-tables-and-database! database)
+ (sync-tables/sync-tables-and-database! (driver.u/lib-database (:id database)))
(is (= #{{:name "trivial_json → a",
:database-type "decimal",
:base-type :type/Integer,
@@ -796,7 +797,7 @@
:nfc-path [:trivial_json "a"]}}
(sql-jdbc.sync/describe-nested-field-columns
:postgres
- database
+ (driver.u/lib-database (:id database))
{:schema "bobdobbs" :name "describe_json_table" :id (mt/id "describe_json_table")})))))))))
(deftest describe-funky-name-table-nested-field-columns-test
@@ -811,7 +812,7 @@
"INSERT INTO \"AAAH_#\".\"dESCribe_json_table_%\" (trivial_json) VALUES ('{\"a\": 1}');")])
(mt/with-temp [:model/Database database {:engine :postgres, :details details}]
(mt/with-db database
- (sync-tables/sync-tables-and-database! database)
+ (sync-tables/sync-tables-and-database! (driver.u/lib-database (:id database)))
(is (= #{{:name "trivial_json → a",
:database-type "decimal",
:base-type :type/Integer,
@@ -821,7 +822,7 @@
:nfc-path [:trivial_json "a"]}}
(sql-jdbc.sync/describe-nested-field-columns
:postgres
- database
+ (driver.u/lib-database (:id database))
{:schema "AAAH_#" :name "dESCribe_json_table_%" :id (mt/id "dESCribe_json_table_%")})))))))))
(mt/defdataset with-uuid
@@ -964,7 +965,7 @@
"Katie Parakeet" 23.99]]]
(next.jdbc/execute! conn sql+args))))
(mt/with-temp [:model/Database db {:engine :postgres, :details (assoc details :dbname "money_columns_test")}]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(mt/with-db db
(thunk)))))
@@ -973,7 +974,7 @@
(testing "It should be possible to return money column results (#3754)"
(sql-jdbc.execute/do-with-connection-with-options
:postgres
- (mt/db)
+ (mt/id)
nil
(fn [conn]
(with-open [stmt (sql-jdbc.execute/prepared-statement :postgres conn "SELECT 1000::money AS \"money\";" nil)
@@ -1059,9 +1060,9 @@
(defn- do-with-enums-db! [f]
(create-enums-db!)
(mt/with-temp [:model/Database database {:engine :postgres, :details (enums-test-db-details)}]
- (sync-metadata/sync-db-metadata! database)
+ (sync-metadata/sync-db-metadata! (:id database))
(f database)
- (driver/notify-database-updated :postgres database)))
+ (driver/notify-database-updated :postgres (driver.u/lib-database (:id database)))))
(deftest ^:parallel enums-test
(mt/test-driver :postgres
@@ -1076,7 +1077,7 @@
(fn [db]
(testing "check that we can actually fetch the enum types from a DB"
(is (= #{"\"bird_schema\".\"bird_status\"" "bird type" "bird_status"}
- (#'postgres/enum-types db))))
+ (#'postgres/enum-types (driver.u/lib-database (:id db))))))
(testing "check that describe-table properly describes the database & base types of the enum fields"
(is (=? [{:table-schema "public"
@@ -1116,7 +1117,7 @@
:database-required true
:database-is-auto-increment false
:json-unfolding false}]
- (->> (driver/describe-fields :postgres db {:table-names ["birds"]})
+ (->> (driver/describe-fields :postgres (driver.u/ensure-lib-database db) {:table-names ["birds"]})
(into #{})
(sort-by :database-position)))))
@@ -1186,7 +1187,7 @@
(do-with-enums-db!
(fn [db]
(tx/create-view-of-table! driver/*driver* db "birds_m" "birds" true)
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(testing "check that describe-table properly describes the database & base types of the enum fields"
(is (=? [{:table-schema "public"
@@ -1226,7 +1227,7 @@
:database-required false
:database-is-auto-increment false
:json-unfolding false}]
- (->> (driver/describe-fields :postgres db {:table-names ["birds_m"]})
+ (->> (driver/describe-fields :postgres (driver.u/ensure-lib-database db) {:table-names ["birds_m"]})
(into #{})
(sort-by :database-position)))))
@@ -1471,7 +1472,7 @@
(jdbc/execute! (sql-jdbc.conn/connection-details->spec :postgres details) [stmt]))
(mt/with-temp [:model/Database database {:engine driver/*driver* :details details}]
(mt/with-db database
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(mt/with-actions-enabled
(testing "when creating with invalid email"
(is (=? {:errors {}
@@ -1502,7 +1503,7 @@
(jdbc/execute! (sql-jdbc.conn/connection-details->spec :postgres details) [stmt]))
(mt/with-temp [:model/Database database {:engine driver/*driver* :details details}]
(mt/with-db database
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(mt/with-actions-enabled
(testing "when creating"
(is (=? {:errors {"column1" "This Column1 value already exists."
@@ -1567,7 +1568,7 @@
"INSERT INTO toucan_sleep_schedule (start_time, end_time, reason) "
" VALUES ('22:00'::time, '9:00'::time, 'Beauty Sleep');")])
(mt/with-temp [:model/Database database {:engine :postgres, :details (assoc details :dbname "time_field_test")}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [fingerprints (t2/select-fn->fn :name :fingerprint :model/Field
:table_id (t2/select-one-pk :model/Table :db_id (u/the-id database)))
;; Strip extended interestingness stats — this test covers the core TIME fingerprint
@@ -1669,7 +1670,7 @@
(mt/with-temp [:model/Database database {:engine :postgres, :details test-user-details}]
;; make sure that sync still succeeds even tho some tables are not SELECTable.
(binding [sync-util/*log-exceptions-and-continue?* false]
- (is (some? (sync/sync-database! database {:scan :schema}))))
+ (is (some? (sync/sync-database! (:id database) {:scan :schema}))))
(is (= #{"table_with_perms"}
(t2/select-fn-set :name :model/Table :db_id (:id database)))))))))
@@ -1693,13 +1694,13 @@
(try
(mt/with-temp [:model/Database database {:engine :postgres, :details details}]
(testing "sanity check: a user that can read every column syncs all of them"
- (sync/sync-database! database {:scan :schema})
+ (sync/sync-database! (:id database) {:scan :schema})
(is (= #{"id" "col_visible" "col_json" "col_json → key_a" "col_json_hidden" "col_json_hidden → key_b" "col_hidden"}
(active-fields database))))
(t2/update! :model/Database (:id database) {:details (assoc details :user role :password "password")})
(let [database (t2/select-one :model/Database (:id database))]
(binding [sync-util/*log-exceptions-and-continue?* false]
- (sync/sync-database! database {:scan :schema}))
+ (sync/sync-database! (:id database) {:scan :schema}))
(testing "after switching to a user with column-level grants, unreadable columns are retired and readable JSON is still unfolded"
(is (= #{"id" "col_visible" "col_json" "col_json → key_a"}
(active-fields database))))
@@ -1726,7 +1727,7 @@
"json_val::jsonb ?& array['a', 'b']"
"FROM \"json_table\";")]
(mt/with-temp [:model/Database database {:engine :postgres, :details json-db-details}]
- (mt/with-db database (sync/sync-database! database)
+ (mt/with-db database (sync/sync-database! (:id database))
(is (= [[true false true]]
(-> {:query query}
(mt/native-query)
@@ -1767,7 +1768,7 @@
(jdbc/execute! spec [statement]))
(mt/with-temp [:model/Database database {:engine driver/*driver* :details details}]
(mt/with-db database
- (sync-tables/sync-tables-and-database! database)
+ (sync-tables/sync-tables-and-database! (driver.u/lib-database (:id database)))
(is (= #{{:name "json_val → int_turn_string",
:database-type "text"
:base-type :type/Text
@@ -1777,7 +1778,7 @@
:nfc-path [:json_val "int_turn_string"]}}
(sql-jdbc.sync/describe-nested-field-columns
:postgres
- database
+ (driver.u/lib-database (:id database))
(t2/select-one :model/Table :db_id (mt/id) :name "json_table")))))))))))
(defn- pretty-sql [s]
@@ -2059,11 +2060,11 @@
(testing "`syncable-schemas` should return schemas that should be synced"
(mt/with-empty-db
(is (= #{"public"}
- (driver/syncable-schemas driver/*driver* (mt/db))))))
+ (driver/syncable-schemas driver/*driver* (lib.metadata/database (mt/metadata-provider)))))))
(testing "metabase_cache schemas should be excluded"
(mt/dataset test-data
(mt/with-persistence-enabled! [persist-models!]
- (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))]
+ (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))]
(mt/with-temp [:model/Card _ {:name "model"
:type :model
:dataset_query (mt/mbql-query categories)
@@ -2072,13 +2073,13 @@
(is (some (partial re-matches #"metabase_cache(.*)")
(map :schema_name (jdbc/query conn-spec "SELECT schema_name from INFORMATION_SCHEMA.SCHEMATA;"))))
(is (nil? (some (partial re-matches #"metabase_cache(.*)")
- (driver/syncable-schemas driver/*driver* (mt/db))))))))))))
+ (driver/syncable-schemas driver/*driver* (lib.metadata/database (mt/metadata-provider)))))))))))))
(deftest table-privileges-test
(mt/test-driver :postgres
(testing "`table-privileges` should return the correct data for current_user and role privileges"
(mt/with-empty-db
- (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
get-privileges (fn []
(sql-jdbc.conn/with-connection-spec-for-testing-connection
[spec [:postgres (assoc (:details (mt/db)) :user "privilege_rows_test_example_role")]]
@@ -2155,7 +2156,7 @@
(mt/dataset test-data
(let [long-sleep-sql "select pg_sleep(5)"]
(sql-jdbc.execute/do-with-connection-with-options
- :postgres (mt/db) nil
+ :postgres (mt/id) nil
(fn [conn]
(with-open [stmt (sql-jdbc.sync.common/prepare-statement :postgres conn long-sleep-sql [])]
(try (doto stmt
@@ -2189,7 +2190,7 @@
(mt/with-empty-db
(sql-jdbc.execute/do-with-connection-with-options
:postgres
- (mt/db)
+ (mt/id)
nil
(fn [^Connection conn]
(let [do-test (fn [& {:keys [schema-pattern table-pattern schemas tables]
@@ -2202,7 +2203,7 @@
:postgres (.getMetaData conn) nil schema-pattern table-pattern
["TABLE" "PARTITIONED TABLE" "VIEW" "FOREIGN TABLE" "MATERIALIZED VIEW"]))
(into #{} (map #(dissoc % :estimated_row_count))
- (#'postgres/get-tables (mt/db) schemas tables)))))]
+ (#'postgres/get-tables (lib.metadata/database (mt/metadata-provider)) schemas tables)))))]
(doseq [stmt ["CREATE TABLE public.table (id INTEGER, type TEXT);"
"CREATE UNIQUE INDEX idx_table_type ON public.table(type);"
"CREATE TABLE public.partition_table (id INTEGER) PARTITION BY RANGE (id);"
@@ -2288,7 +2289,7 @@
(binding [sql-jdbc.describe-database/*select-probe-query-timeout-seconds* 1]
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (mt/id)
nil
(fn [^java.sql.Connection conn]
(is (true? (sql-jdbc.sync.interface/have-select-privilege?
@@ -2319,7 +2320,7 @@
:user "sync_writable_test_user"
:password "password")]
(mt/with-temp [:model/Database database {:engine "postgres", :details user-connection-details}]
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(testing "only fullaccess table has writeable permissions"
(is (= {"readonly_table" false
"readwrite_table" false
@@ -2328,7 +2329,7 @@
(testing "After granting full access to all tables and re-syncing"
(doseq [table-name ["readonly_table" "readwrite_table"]]
(jdbc/execute! conn (format "GRANT INSERT, UPDATE, DELETE ON sync_test_schema.%s TO sync_writable_test_user" table-name)))
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(is (= {"readonly_table" true
"readwrite_table" true
"fullaccess_table" true}
@@ -2705,6 +2706,6 @@
;; streaming the JDBC driver buffers the whole ResultSet (~2.1B rows) and OOMs at any heap size.
(let [n Integer/MAX_VALUE
result (sql-jdbc.execute/reducible-query
- (mt/db) [(format "SELECT generate_series(1, %d) AS i" n)])]
+ (lib.metadata/database (mt/metadata-provider)) [(format "SELECT generate_series(1, %d) AS i" n)])]
(testing "only the first rows are pulled, not all ~2 billion"
(is (= [1 2 3] (into [] (comp (take 3) (map :i)) result))))))))
diff --git a/test/metabase/driver/sql/query_processor_test.clj b/test/metabase/driver/sql/query_processor_test.clj
index c54078190d0..72971d839e4 100644
--- a/test/metabase/driver/sql/query_processor_test.clj
+++ b/test/metabase/driver/sql/query_processor_test.clj
@@ -31,6 +31,7 @@
[metabase.settings.core :as setting]
[metabase.test :as mt]
[metabase.test.data.env :as tx.env]
+ [metabase.test.fixtures :as fixtures]
[metabase.util.date-2 :as u.date]
[metabase.util.honey-sql-2 :as h2x]
[metabase.util.malli.registry :as mr]))
@@ -39,6 +40,8 @@
(set! *warn-on-reflection* true)
+(use-fixtures :once (fixtures/initialize :db))
+
(deftest ^:parallel compiled-test
(is (= [:raw "x"]
(sql.qp/->honeysql :sql (sql.qp/compiled [:raw "x"])))))
diff --git a/test/metabase/driver/sql/transforms_test.clj b/test/metabase/driver/sql/transforms_test.clj
index ab18fb14db3..b657a59b563 100644
--- a/test/metabase/driver/sql/transforms_test.clj
+++ b/test/metabase/driver/sql/transforms_test.clj
@@ -188,8 +188,8 @@
;; H2-only: the inline `VALUES` source keeps the transform self-contained (no source table needed).
(testing "run-transform! returns a single {:rows-affected <int>} map, never a nested/double-wrapped one"
(mt/test-drivers #{:h2}
- (let [base {:conn-spec (driver/connection-spec :h2 (mt/db))
- :database (mt/db)
+ (let [base {:conn-spec (driver/connection-spec :h2 (lib.metadata/database (mt/metadata-provider)))
+ :database (lib.metadata/database (mt/metadata-provider))
:output-schema "PUBLIC"
:query {:query "SELECT * FROM (VALUES (1),(2),(3)) AS t(a)"}}]
(transforms.tu/with-transform-cleanup! [full-target {:type :table :schema "PUBLIC" :name "rows_affected_full"}
@@ -263,9 +263,9 @@
:database (mt/id)}]
(let [expected (venues-row-count)
details {:db-id (mt/id)
- :database (mt/db)
+ :database (lib.metadata/database (mt/metadata-provider))
:transform-type :table
- :conn-spec (driver/connection-spec driver/*driver* (mt/db))
+ :conn-spec (driver/connection-spec driver/*driver* (lib.metadata/database (mt/metadata-provider)))
:query (transforms-base.u/compile-source
{:source {:type "query" :query (venues-source-query)} :target target} nil)
:output-schema (:schema target)
@@ -290,9 +290,9 @@
(let [transform {:source {:type "query" :query (venues-source-query)}
:target target}
details {:db-id (mt/id)
- :database (mt/db)
+ :database (lib.metadata/database (mt/metadata-provider))
:transform-type :table
- :conn-spec (driver/connection-spec driver/*driver* (mt/db))
+ :conn-spec (driver/connection-spec driver/*driver* (lib.metadata/database (mt/metadata-provider)))
:query (transforms-base.u/compile-source transform nil)
:output-schema (:schema target)
:output-table (transforms-base.u/qualified-table-name driver/*driver* target)}
@@ -312,9 +312,9 @@
(let [transform {:source {:type "query" :query (venues-source-query)}
:target target}
details {:db-id (mt/id)
- :database (mt/db)
+ :database (lib.metadata/database (mt/metadata-provider))
:transform-type :table-incremental
- :conn-spec (driver/connection-spec driver/*driver* (mt/db))
+ :conn-spec (driver/connection-spec driver/*driver* (lib.metadata/database (mt/metadata-provider)))
:query (transforms-base.u/compile-source transform nil)
:output-schema (:schema target)
:output-table (transforms-base.u/qualified-table-name driver/*driver* target)}]
@@ -329,7 +329,7 @@
(mt/test-driver :h2
(mt/with-premium-features #{:transforms-basic}
(let [dashed-schema "dashed-schema-probe"
- conn-spec (driver/connection-spec driver/*driver* (mt/db))
+ conn-spec (driver/connection-spec driver/*driver* (lib.metadata/database (mt/metadata-provider)))
quoted-schema (sql.u/quote-name driver/*driver* :schema dashed-schema)
execute! #(driver/execute-raw-queries! driver/*driver* conn-spec [[%]])]
(execute! (str "CREATE SCHEMA IF NOT EXISTS " quoted-schema))
@@ -339,7 +339,7 @@
:name "dashed_table_probe"
:database (mt/id)}]
(let [target-table (:name target)
- exists? #(driver/table-exists? driver/*driver* (mt/db)
+ exists? #(driver/table-exists? driver/*driver* (lib.metadata/database (mt/metadata-provider))
{:schema dashed-schema, :name target-table})]
(driver/create-table! driver/*driver* (mt/id) (keyword dashed-schema target-table)
{"id" (driver/type->database-type driver/*driver* :type/Integer)} {})
diff --git a/test/metabase/driver/sql_jdbc/connection_test.clj b/test/metabase/driver/sql_jdbc/connection_test.clj
index e159a151320..4d3d2420ee9 100644
--- a/test/metabase/driver/sql_jdbc/connection_test.clj
+++ b/test/metabase/driver/sql_jdbc/connection_test.clj
@@ -22,6 +22,7 @@
[metabase.driver.sql-jdbc.connection.ssh-tunnel-test :as ssh-test]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
[metabase.driver.util :as driver.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.premium-features.core :as premium-features]
[metabase.query-processor.test :as qp]
[metabase.query-processor.test-util :as qp.test-util]
@@ -75,9 +76,9 @@
(testing "when a connection pool is created"
(is (thrown-with-msg?
clojure.lang.ExceptionInfo #"dangerous"
- (#'sql-jdbc.conn/create-pool! (t2/instance :model/Database {:id Integer/MAX_VALUE
- :engine driver/*driver*
- :details details})))))
+ (#'sql-jdbc.conn/create-pool! (driver.u/ensure-lib-database (t2/instance :model/Database {:id Integer/MAX_VALUE
+ :engine driver/*driver*
+ :details details}))))))
(testing "when a connection is tested"
(is (thrown-with-msg?
clojure.lang.ExceptionInfo #"dangerous"
@@ -98,9 +99,9 @@
(testing "when a connection pool is created"
(is (thrown-with-msg?
clojure.lang.ExceptionInfo #"dangerous"
- (#'sql-jdbc.conn/create-pool! (t2/instance :model/Database {:id Integer/MAX_VALUE
- :engine driver/*driver*
- :details details})))))
+ (#'sql-jdbc.conn/create-pool! (driver.u/ensure-lib-database (t2/instance :model/Database {:id Integer/MAX_VALUE
+ :engine driver/*driver*
+ :details details}))))))
(testing "when a connection is tested"
(is (thrown-with-msg?
clojure.lang.ExceptionInfo #"dangerous"
@@ -127,17 +128,17 @@
(next.jdbc/execute! conn ["CREATE TABLE birds (name varchar)"])
(next.jdbc/execute! conn ["INSERT INTO birds values ('rasta'),('lucky')"])
(mt/with-temp [:model/Database database {:engine :h2, :details connection-details}]
- (let [cache-key (pool-cache-key database)]
+ (let [cache-key (pool-cache-key (driver.u/ensure-lib-database database))]
(testing "database id is not in our connection map initially"
;; deref'ing a var to get the atom. looks weird
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool cache-key))))
(testing "when getting a pooled connection it is now in our connection map"
- (let [stored-spec (sql-jdbc.conn/db->pooled-connection-spec database)
+ (let [stored-spec (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
birds (jdbc/query stored-spec ["SELECT * FROM birds"])]
(is (seq birds))
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool cache-key))))
(testing "and is no longer in our connection map after cleanup"
- (driver/notify-database-updated :h2 database)
+ (driver/notify-database-updated :h2 (driver.u/lib-database (:id database)))
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool cache-key))))
(testing "the pool has been destroyed"
(is @destroyed?)))))))))))
@@ -165,17 +166,17 @@
default-cache-key [db-id :default]
write-cache-key [db-id :write-data]]
;; Ensure pools are cleared
- (sql-jdbc.conn/invalidate-pool-for-db! database)
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id database))
(testing "initially no pools exist"
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool default-cache-key)))
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))))
(testing "getting a default connection creates only the default pool"
- (sql-jdbc.conn/db->pooled-connection-spec database)
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool default-cache-key))
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))))
(testing "getting a write connection creates a separate write pool"
(driver.conn/with-write-connection
- (sql-jdbc.conn/db->pooled-connection-spec database))
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database)))
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool default-cache-key))
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key)))
(testing "the two pools are different objects"
@@ -185,7 +186,7 @@
(is (some? write-pool))
(is (not (identical? default-pool write-pool)))))
;; Cleanup
- (sql-jdbc.conn/invalidate-pool-for-db! database)))))))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id database))))))))))))
(deftest write-connection-reuses-default-pool-when-unconfigured-test
(mt/test-driver :h2
@@ -195,19 +196,19 @@
(let [db-id (u/the-id database)
default-cache-key [db-id :default]
write-cache-key [db-id :write-data]]
- (sql-jdbc.conn/invalidate-pool-for-db! database)
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id database))
(testing "get the default pool"
- (sql-jdbc.conn/db->pooled-connection-spec database)
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool default-cache-key)))
(testing "with-write-connection reuses the default pool, no duplicate created"
(let [default-pool (get @@#'sql-jdbc.conn/pool-cache-key->connection-pool default-cache-key)
write-pool (driver.conn/with-write-connection
- (sql-jdbc.conn/db->pooled-connection-spec database))]
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database)))]
(is (identical? default-pool write-pool)
"Should return the exact same pool object")
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))
"No separate write pool entry should exist")))
- (sql-jdbc.conn/invalidate-pool-for-db! database))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id database)))))))
(deftest write-pool-uses-write-details-test
(mt/test-driver :h2
@@ -222,27 +223,27 @@
:write_data_details write-details}]
(let [db-id (u/the-id database)]
;; Ensure pools are cleared
- (sql-jdbc.conn/invalidate-pool-for-db! database)
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id database))
(testing "jdbc-spec-hash differs between default and write connection types"
- (let [default-hash (#'sql-jdbc.conn/jdbc-spec-hash database)
+ (let [default-hash (#'sql-jdbc.conn/jdbc-spec-hash (driver.u/ensure-lib-database database))
write-hash (driver.conn/with-write-connection
- (#'sql-jdbc.conn/jdbc-spec-hash database))]
+ (#'sql-jdbc.conn/jdbc-spec-hash (driver.u/ensure-lib-database database)))]
(is (integer? default-hash))
(is (integer? write-hash))
(is (not= default-hash write-hash)
"Hash should differ because effective-details returns different details")))
(testing "hash cache uses composite keys"
;; Get both pools
- (sql-jdbc.conn/db->pooled-connection-spec database)
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
(driver.conn/with-write-connection
- (sql-jdbc.conn/db->pooled-connection-spec database))
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database)))
(let [default-cached-hash (get @@#'sql-jdbc.conn/pool-cache-key->jdbc-spec-hash [db-id :default])
write-cached-hash (get @@#'sql-jdbc.conn/pool-cache-key->jdbc-spec-hash [db-id :write-data])]
(is (some? default-cached-hash))
(is (some? write-cached-hash))
(is (not= default-cached-hash write-cached-hash))))
;; Cleanup
- (sql-jdbc.conn/invalidate-pool-for-db! database)))))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id database))))))))))
(deftest invalidate-pool-clears-all-connection-types-test
(mt/test-driver :h2
@@ -258,14 +259,14 @@
(let [db-id (u/the-id database)
default-cache-key [db-id :default]
write-cache-key [db-id :write-data]]
- (sql-jdbc.conn/db->pooled-connection-spec database)
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
(driver.conn/with-write-connection
- (sql-jdbc.conn/db->pooled-connection-spec database))
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database)))
(testing "all pools exist before invalidation"
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool default-cache-key))
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key)))
(testing "invalidate-pool-for-db! removes all pools"
- (sql-jdbc.conn/invalidate-pool-for-db! database)
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id database))
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool default-cache-key)))
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))))))))))))
@@ -273,7 +274,7 @@
(mt/test-drivers (mt/driver-select {:+parent :sql-jdbc})
(testing "The dataSourceName c3p0 property is set properly for a database"
(let [db (mt/db)
- props (sql-jdbc.conn/data-warehouse-connection-pool-properties driver/*driver* db)
+ props (sql-jdbc.conn/data-warehouse-connection-pool-properties driver/*driver* (driver.u/ensure-lib-database db))
[_ db-nm] (re-matches (re-pattern (format "^db-%d-%s-(.*)$" (u/the-id db) (name driver/*driver*)))
(get props "dataSourceName"))]
(is (some? db-nm))
@@ -293,7 +294,7 @@
(mt/with-temp [:model/Database database {:engine :h2
:details {:db "mem:pool_name_default_db"}
:write_data_details {:db "mem:pool_name_write_db"}}]
- (let [pool-name #(get (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 database) "dataSourceName")
+ (let [pool-name #(get (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (driver.u/ensure-lib-database database)) "dataSourceName")
db-id (u/the-id database)]
(is (= (format "db-%d-h2-mem:pool_name_default_db" db-id)
(pool-name)))
@@ -301,7 +302,7 @@
(driver.conn/with-write-connection (pool-name)))))))
(testing "a write connection without write details resolves to the default pool, so its name has no suffix"
(mt/with-temp [:model/Database database {:engine :h2, :details {:db "mem:pool_name_default_only_db"}}]
- (let [pool-name #(get (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 database) "dataSourceName")
+ (let [pool-name #(get (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (driver.u/ensure-lib-database database)) "dataSourceName")
db-id (u/the-id database)]
(is (= (format "db-%d-h2-mem:pool_name_default_only_db" db-id)
(driver.conn/with-write-connection (pool-name))))
@@ -364,11 +365,11 @@
true
(original-supports? driver feature database)))]
(try
- (sql-jdbc.conn/invalidate-pool-for-db! db)
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id db))
(mt/with-dynamic-fn-redefs [sql-jdbc.conn/log-jdbc-spec-hash-change-msg! hash-change-fn
driver.u/supports? supports?-fn]
- (let [pool-spec-1 (sql-jdbc.conn/db->pooled-connection-spec db)
- db-hash-1 (get @@#'sql-jdbc.conn/pool-cache-key->jdbc-spec-hash (#'sql-jdbc.conn/pool-cache-key db))]
+ (let [pool-spec-1 (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database db))
+ db-hash-1 (get @@#'sql-jdbc.conn/pool-cache-key->jdbc-spec-hash (#'sql-jdbc.conn/pool-cache-key (driver.u/ensure-lib-database db)))]
(testing "hash value calculated correctly for new pooled conn"
(is (some? pool-spec-1))
(is (integer? db-hash-1))
@@ -376,13 +377,13 @@
(testing "changing DB details results in hash value changing and connection being invalidated"
(let [db-perturbed (perturb-db-details db)]
(testing "The calculated hash should be different"
- (is (not= (#'sql-jdbc.conn/jdbc-spec-hash db)
- (#'sql-jdbc.conn/jdbc-spec-hash db-perturbed))))
+ (is (not= (#'sql-jdbc.conn/jdbc-spec-hash (driver.u/ensure-lib-database db))
+ (#'sql-jdbc.conn/jdbc-spec-hash (driver.u/ensure-lib-database db-perturbed)))))
(t2/update! :model/Database (mt/id) {:details (:details db-perturbed)})
(let [;; this call should result in the connection pool becoming invalidated, and the new hash value
;; being stored based upon these updated details
- pool-spec-2 (sql-jdbc.conn/db->pooled-connection-spec db-perturbed)
- db-hash-2 (get @@#'sql-jdbc.conn/pool-cache-key->jdbc-spec-hash (#'sql-jdbc.conn/pool-cache-key db))]
+ pool-spec-2 (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database db-perturbed))
+ db-hash-2 (get @@#'sql-jdbc.conn/pool-cache-key->jdbc-spec-hash (#'sql-jdbc.conn/pool-cache-key (driver.u/ensure-lib-database db)))]
;; to throw a wrench into things, kick off a sync of the original db (unperturbed); this
;; simulates a long running sync that began before the perturbed details were saved to the app DB
;; the sync steps SHOULD NOT invalidate the connection pool, because doing so could cause a seesaw
@@ -395,7 +396,7 @@
;; this should still see a hash mismatch in the case that the DB details were updated external to
;; this process (i.e. by a different instance), since our in-memory hash value still wouldn't match
;; even after getting the latest `DatabaseInstance`
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(is (some? pool-spec-2))
(is (= 1 @hash-change-called-times) "One hash change should have been logged")
(is (integer? db-hash-2))
@@ -429,8 +430,8 @@
(:sslrootcert (#'sql-jdbc.conn/connection-details->spec :postgres
(:details db))))
"Secrets not loaded for db connections")
- (is (= (#'sql-jdbc.conn/jdbc-spec-hash db)
- (#'sql-jdbc.conn/jdbc-spec-hash db))
+ (is (= (#'sql-jdbc.conn/jdbc-spec-hash (driver.u/ensure-lib-database db))
+ (#'sql-jdbc.conn/jdbc-spec-hash (driver.u/ensure-lib-database db)))
"Same db produced different hashes due to secrets")))))
;;; TODO -- this set is hardcoded in dozens of places in the codebase, we should unify them all into one single
@@ -448,8 +449,8 @@
audit-db-id (t2/select-one-fn :id :model/Database {:where [:= :is_audit true]})
_ (is (= :metabase-enterprise.audit-app.audit/installed status))
_ (is (= 13371337 audit-db-id))
- first-pool (sql-jdbc.conn/db->pooled-connection-spec audit-db-id)
- second-pool (sql-jdbc.conn/db->pooled-connection-spec audit-db-id)]
+ first-pool (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database audit-db-id))
+ second-pool (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database audit-db-id))]
(is (= first-pool second-pool))
(is (= ::audit-db-not-in-cache!
(get @#'sql-jdbc.conn/pool-cache-key->connection-pool audit-db-id ::audit-db-not-in-cache!)))))))
@@ -460,7 +461,7 @@
:is_audit false}]
(let [routes-to-app-db? (fn []
(= {:datasource (mdb/data-source)}
- (sql-jdbc.conn/db->pooled-connection-spec (:id db))))]
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database (:id db)))))]
(testing "dev mode off: reaching the connection layer with an :is-audit-dev db is an invariant violation, so
it throws rather than silently building a connection against the wrong host"
(mt/with-temporary-setting-values [analytics-dev-mode false]
@@ -475,7 +476,7 @@
(deftest ^:parallel include-unreturned-connection-timeout-test
(testing "We should be setting unreturnedConnectionTimeout; it should be the same as the query timeout (#33646)"
(is (=? {"unreturnedConnectionTimeout" integer?}
- (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (mt/db))))))
+ (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (lib.metadata/database (mt/metadata-provider)))))))
(deftest unreturned-connection-timeout-test
(testing "We should be able to set jdbc-data-warehouse-unreturned-connection-timeout-seconds via env var (#33646)"
@@ -486,7 +487,7 @@
(deftest ^:parallel include-checkout-timeout-test
(testing "We should be setting checkoutTimeout so a saturated pool fails fast instead of queueing forever"
(is (=? {"checkoutTimeout" integer?}
- (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (mt/db))))))
+ (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (lib.metadata/database (mt/metadata-provider)))))))
(deftest checkout-timeout-env-var-test
(testing "We should be able to set jdbc-data-warehouse-connection-pool-checkout-timeout-ms via env var"
@@ -494,7 +495,7 @@
(is (= 5000
(driver.settings/jdbc-data-warehouse-connection-pool-checkout-timeout-ms)))
(is (= 5000
- (get (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (mt/db))
+ (get (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (lib.metadata/database (mt/metadata-provider)))
"checkoutTimeout"))))))
(deftest max-pending-checkouts-env-var-test
@@ -506,7 +507,7 @@
(deftest ^:parallel include-debug-unreturned-connection-stack-traces-test
(testing "We should be setting debugUnreturnedConnectionStackTraces (#47981)"
(is (=? {"debugUnreturnedConnectionStackTraces" boolean?}
- (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (mt/db))))))
+ (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (lib.metadata/database (mt/metadata-provider)))))))
(deftest debug-unreturned-connection-stack-traces-test
(testing "We should be able to set jdbc-data-warehouse-debug-unreturned-connection-stack-traces via env var (#47981)"
@@ -530,7 +531,7 @@
(mt/with-log-level [com.mchange c3p0-log-level]
(mt/with-log-messages-for-level [warnings :warn]
(and (= setting
- (get (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (mt/db))
+ (get (sql-jdbc.conn/data-warehouse-connection-pool-properties :h2 (lib.metadata/database (mt/metadata-provider)))
"debugUnreturnedConnectionStackTraces"))
(= warning-expected? (warning-found? (warnings))))))))]
(are [c3p0-log-level setting warning-expected?] (warnings-logged? c3p0-log-level setting warning-expected?)
@@ -559,9 +560,9 @@
(when (and (:password original-details) (:user original-details))
(mt/with-temp [:model/Database db {:engine (tx/driver), :details original-details}]
(mt/with-db db
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= 1 (count (mt/rows (mt/run-mbql-query venues {:limit 1})))))
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id))
(let [new-details (assoc original-details :user "baduser")
start (t/instant)]
(t2/update! :model/Database :id (mt/id) {:details new-details})
@@ -598,11 +599,11 @@
(try
;; since Metabase is running and using the pool of this DB, the sync might fail
;; if the connection pool is shut down during the sync
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(catch Exception _))
;; after "fixing" the expiry, we should get a connection from a pool that doesn't get shut down
(reset! expires-in "10000")
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= [["Polo Lounge"]]
(mt/rows (mt/run-mbql-query venues {:filter [:= $id 60] :fields [$name]}))))
;; we must have created more than one connection
@@ -689,13 +690,13 @@
:tunnel-user ssh-test/ssh-username
:tunnel-pass ssh-test/ssh-password)]
(try
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id))
(t2/update! :model/Database (mt/id) {:details tunnel-db-details#})
(mt/with-db (t2/select-one :model/Database (mt/id))
(try
~@body
(finally
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db)))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id)))))
(finally
(t2/update! :model/Database (mt/id) {:details original-details#})))))
@@ -729,7 +730,7 @@
;; check that some data can be queried
(check-row)
;; kill the ssh tunnel; fortunately, we have an existing function that can do that
- (ssh/close-tunnel! (sql-jdbc.conn/db->pooled-connection-spec (mt/db)))
+ (ssh/close-tunnel! (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database (mt/db))))
;; check the query again; the tunnel should have been reestablished
(check-row)))))
@@ -756,7 +757,7 @@
(try
(mt/with-temp [:model/Database db {:engine :h2, :details h2-db}]
(mt/with-db db
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (=? {:cols [{:base_type :type/Text
:effective_type :type/Text
:display_name "COL1"
@@ -801,7 +802,7 @@
(try
(mt/with-temp [:model/Database db {:engine :h2, :details h2-db}]
(mt/with-db db
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(letfn [(check-data [] (is (=? {:cols [{:base_type :type/Text
:effective_type :type/Text
:display_name "COL1"
@@ -824,7 +825,7 @@
;; check that some data can be queried
(check-data)
;; kill the ssh tunnel; fortunately, we have an existing function that can do that
- (ssh/close-tunnel! (sql-jdbc.conn/db->pooled-connection-spec db))
+ (ssh/close-tunnel! (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database db)))
;; check the query again; the tunnel should have been reestablished
(check-data))))
(finally (.stop ^Server server))))))))
diff --git a/test/metabase/driver/sql_jdbc/sync/describe_database_test.clj b/test/metabase/driver/sql_jdbc/sync/describe_database_test.clj
index 0be423e5549..42d067ca797 100644
--- a/test/metabase/driver/sql_jdbc/sync/describe_database_test.clj
+++ b/test/metabase/driver/sql_jdbc/sync/describe_database_test.clj
@@ -13,6 +13,7 @@
[metabase.driver.sql-jdbc.sync.describe-database :as sql-jdbc.describe-database]
[metabase.driver.sql-jdbc.sync.interface :as sql-jdbc.sync.interface]
[metabase.driver.util :as driver.u]
+ [metabase.lib.metadata :as lib.metadata]
;; binds mock metadata providers via the ambient store, which the code under test reads
^{:clj-kondo/ignore [:deprecated-namespace]} [metabase.query-processor.store :as qp.store]
[metabase.query-processor.test :as qp]
@@ -70,7 +71,7 @@
"REVIEWS" "USERS" "VENUES"}
(sql-jdbc.execute/do-with-connection-with-options
(or driver/*driver* :h2)
- (mt/db)
+ (mt/id)
nil
(fn [^Connection conn]
;; We have to mock this to make it work with all DBs
@@ -86,7 +87,7 @@
"REVIEWS" "USERS" "VENUES"}
(sql-jdbc.execute/do-with-connection-with-options
:h2
- (mt/db)
+ (mt/id)
nil
(fn [^Connection conn]
(->> (into [] (sql-jdbc.describe-database/post-filtered-active-tables
@@ -103,7 +104,7 @@
{:name "PEOPLE", :schema "PUBLIC", :description nil, :is_writable true}
{:name "PRODUCTS", :schema "PUBLIC", :description nil, :is_writable true}
{:name "REVIEWS", :schema "PUBLIC", :description nil, :is_writable true}}
- (into #{} (:tables (sql-jdbc.describe-database/describe-database :h2 (mt/id)))))))
+ (into #{} (:tables (sql-jdbc.describe-database/describe-database :h2 (lib.metadata/database (mt/metadata-provider))))))))
(defn- describe-database-with-open-resultset-count!
"Just like `describe-database`, but instead of returning the database description returns the number of ResultSet
@@ -122,11 +123,11 @@
;; open before they would normally get closed
(sql-jdbc.execute/do-with-connection-with-options
driver
- db
+ (driver.u/ensure-lib-database db)
nil
(fn [_conn]
;; `:tables` is a reducible -- realize it so `active-tables` actually opens (and closes) its ResultSets
- (let [_ (into [] (:tables (sql-jdbc.describe-database/describe-database driver db)))]
+ (let [_ (into [] (:tables (sql-jdbc.describe-database/describe-database driver (driver.u/ensure-lib-database db))))]
(reduce + (for [^ResultSet rs @resultsets]
(if (.isClosed rs) 0 1)))))))))
@@ -142,11 +143,11 @@
"drop table if exists \"birds\";"
"create table \"birds\" ();"]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= 1 (count-active-tables-in-db (mt/id))))
;; We have to mock this as H2 doesn't have the notion of a user connecting to it
(with-redefs [sql-jdbc.sync.interface/have-select-privilege? (constantly false)]
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= 0 (count-active-tables-in-db (mt/id)))
"We shouldn't sync tables for which we don't have select privilege"))))
@@ -159,7 +160,7 @@
(defn- sync-and-assert-filtered-tables [database assert-table-fn]
(mt/with-temp [:model/Database db-filtered database]
- (sync/sync-database! db-filtered {:scan :schema})
+ (sync/sync-database! (:id db-filtered) {:scan :schema})
(let [tables (t2/select :model/Table :db_id (u/the-id db-filtered))]
(doseq [table tables]
(assert-table-fn table)))))
@@ -249,7 +250,7 @@
(testing (pr-str {:auto-commit auto-commit :schema schema :name table-name})
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (mt/id)
nil
(fn [^Connection conn]
;; Databricks does not support setting auto commit to false. Catching the setAutoCommit
@@ -294,7 +295,7 @@
:-features [:table-privileges]})
(let [closed-first (volatile! false)
execute-select-probe-query (mt/original-fn #'sql-jdbc.describe-database/execute-select-probe-query)
- all-tables (driver/describe-database driver/*driver* (mt/id))]
+ all-tables (driver/describe-database driver/*driver* (lib.metadata/database (mt/metadata-provider)))]
(mt/with-dynamic-fn-redefs [sql-jdbc.describe-database/execute-select-probe-query
(fn [driver ^Connection conn query]
(when-not @closed-first
@@ -302,7 +303,7 @@
(.close conn))
(execute-select-probe-query driver conn query))]
(let [table-names #(into #{} (map :name) (:tables %))
- all-tables-sans-one (table-names (driver/describe-database driver/*driver* (mt/id)))]
+ all-tables-sans-one (table-names (driver/describe-database driver/*driver* (lib.metadata/database (mt/metadata-provider))))]
;; there is at maximum one missing table
(is (>= 1 (count (set/difference all-tables all-tables-sans-one)))))))))))
@@ -310,7 +311,7 @@
[probe-errors query-canceled probe-error-fn]
(let [{schema :schema, table-name :name} (t2/select-one :model/Table (mt/id :checkins))]
(sql-jdbc.execute/do-with-connection-with-options
- driver/*driver* (mt/db) nil
+ driver/*driver* (mt/id) nil
(fn [^Connection conn]
(let [select-probes (atom 0)]
;; query-canceled? is a multimethod, so we can't use with-dynamic-fn-redefs for the pair.
diff --git a/test/metabase/driver/sql_jdbc/sync/describe_table_test.clj b/test/metabase/driver/sql_jdbc/sync/describe_table_test.clj
index 7d391056827..74d113c5d43 100644
--- a/test/metabase/driver/sql_jdbc/sync/describe_table_test.clj
+++ b/test/metabase/driver/sql_jdbc/sync/describe_table_test.clj
@@ -113,7 +113,7 @@
:database-is-nullable true
:jdbc-type java.sql.Types/INTEGER
:json-unfolding false}}}
- (driver/describe-table :h2 (mt/db) {:name "VENUES"})))))
+ (driver/describe-table :h2 (lib.metadata/database (mt/metadata-provider)) {:name "VENUES"})))))
(deftest describe-auto-increment-on-non-pk-field-test
(testing "a non-pk field with auto-increment should be have metabase_field.database_is_auto_increment=true"
@@ -127,7 +127,7 @@
"CREATE TABLE \"employee_counter\" (\"id\" INTEGER AUTO_INCREMENT PRIMARY KEY, \"count\" INTEGER AUTO_INCREMENT, \"rank\" INTEGER NOT NULL)"
"GRANT ALL ON \"employee_counter\" TO GUEST;"]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= {:fields #{{:base-type :type/Integer
:database-is-auto-increment true
:database-is-generated false
@@ -160,10 +160,11 @@
:name "rank"
:json-unfolding false}}
:name "employee_counter"}
- (sql-jdbc.describe-table/describe-table :h2 (mt/id) {:name "employee_counter"}))))))
+ (sql-jdbc.describe-table/describe-table :h2 (lib.metadata/database (mt/metadata-provider)) {:name "employee_counter"}))))))
(defn- describe-fields-for-table [db table]
- (let [driver (driver.u/database->driver db)]
+ (let [driver (driver.u/database->driver db)
+ db (driver.u/ensure-lib-database db)]
(sort-by :database-position
(if (driver.u/supports? driver :describe-fields db)
(vec (m/mapply driver/describe-fields
@@ -288,7 +289,7 @@
(mt/test-drivers (mt/normal-drivers-with-feature :nested-field-columns)
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (mt/id)
nil
(fn [conn]
(is (= ["id"]
@@ -304,7 +305,7 @@
(let [table (t2/select-one :model/Table :id (mt/id :json))]
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (mt/id)
nil
(fn [^java.sql.Connection conn]
(let [fields (sql-jdbc.describe-table/describe-table-fields driver/*driver* conn table nil)
@@ -339,7 +340,7 @@
(when-not (mysql/mariadb? (mt/db))
(testing "Nested field column listing"
(is (= [:type/JSON :type/SerializedJSON]
- (->> (sql-jdbc.sync/describe-table driver/*driver* (mt/db) {:name "json"})
+ (->> (sql-jdbc.sync/describe-table driver/*driver* (lib.metadata/database (mt/metadata-provider)) {:name "json"})
:fields
(filter #(= (:name %) "json_bit"))
first
@@ -409,7 +410,7 @@
:nfc-path [:json_bit "title"]}}
(sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{:name "json" :id (mt/id "json")}))))))))
(deftest json-columns-with-values-are-not-object-test
@@ -424,9 +425,9 @@
(testing "there should be no nested fields"
(is (= #{} (sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{:name "json_table" :id (mt/id "json_table")}))))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (=? (if (mysql/mariadb? (mt/db))
#{{:name "id"
:base_type :type/Integer}
@@ -457,12 +458,12 @@
(count
(sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{:name "big_json_table" :id (mt/id "big_json_table")}))))
(mt/with-log-messages-for-level [messages :warn]
(sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{:name "big_json_table" :id (mt/id "big_json_table")})
(is (str/includes?
(-> (messages) first :message)
@@ -476,7 +477,7 @@
(is (= sql-jdbc.describe-table/max-nested-field-columns
(count (sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{:name "big_json" :id (mt/id "big_json")})))))))))
(mt/defdataset long-json
@@ -496,8 +497,8 @@
(mt/with-temporary-setting-values [sql-jdbc.describe-table/nested-field-columns-value-length-limit
(dec (count (json/encode {:b (apply str (repeat 10 "y"))})))]
(mt/dataset long-json
- (sync/sync-database! (mt/db) {:scan :schema})
- (let [jdbc-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (sync/sync-database! (mt/id) {:scan :schema})
+ (let [jdbc-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
table (t2/select-one :model/Table :db_id (mt/id) :name "long_json_table")
json-fields (t2/select :model/Field :table_id (:id table) :name [:in ["short_json" "long_json"]])
pks ["id"]
@@ -542,7 +543,7 @@
(mt/test-drivers (mt/normal-drivers-with-feature :nested-field-columns)
(when-not (mysql/mariadb? (mt/db))
(mt/dataset json-unwrap-bigint-and-boolean
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(testing "Fields marked as :type/SerializedJSON are fingerprinted that way"
(is (= #{{:name "id", :base_type :type/Integer, :semantic_type :type/PK}
{:name "jsoncol", :base_type :type/JSON, :semantic_type :type/SerializedJSON}
@@ -566,7 +567,7 @@
:nfc-path [:jsoncol "myint"]}}
(sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
(t2/select-one :model/Table :db_id (mt/id) :name "bigint-and-bool-table")))))))))
(mt/defdataset json-int-turn-string
@@ -608,7 +609,7 @@
table-rows-sample/nested-field-sample-limit 4]
(mt/dataset json-int-turn-string
(when-not (mysql/mariadb? (mt/db))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(testing "if table has an pk, we fetch both first and last rows thus detect the change in type"
(is (= [{:name "json_col → int_turn_string"
:database-type "text"
@@ -619,7 +620,7 @@
:nfc-path [:json_col "int_turn_string"]}]
(into [] (sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
(t2/select-one :model/Table :db_id (mt/id) :name "json_with_pk")))))
(testing "if table doesn't have pk, we fail to detect the change in type but it still syncable"
(is (= [{:name "json_col → int_turn_string"
@@ -631,7 +632,7 @@
:nfc-path [:json_col "int_turn_string"]}]
(into [] (sql-jdbc.sync/describe-nested-field-columns
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
(t2/select-one :model/Table :db_id (mt/id) :name "json_without_pk")))))))))))))
(defn- describe-table-indexes
@@ -647,10 +648,10 @@
(comp
(map (fn [{:keys [field-name]}] {:type :normal-column-index :value field-name}))
(map lowercase-value))
- (driver/describe-indexes driver database {:table-names [(:name table)]}))
+ (driver/describe-indexes driver (driver.u/ensure-lib-database database) {:table-names [(:name table)]}))
(into #{}
(map lowercase-value)
- (driver/describe-table-indexes driver database table)))))
+ (driver/describe-table-indexes driver (driver.u/ensure-lib-database database) table)))))
(deftest describe-table-indexes-test
(mt/test-drivers (mt/normal-driver-select {:+parent :sql-jdbc
@@ -669,9 +670,9 @@
{:type :normal-column-index :value "indexed"}}
(describe-table-indexes (t2/select-one :model/Table (mt/id :single_index))))))
(testing "for composite indexes, we only care about the 1st column"
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(sql.tx/create-index-sql driver/*driver* "composite_index" ["first" "second"]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= #{{:type :normal-column-index :value "id"}
{:type :normal-column-index :value "first"}}
(describe-table-indexes (t2/select-one :model/Table (mt/id :composite_index)))))))))
@@ -693,7 +694,7 @@
[{:field-name "column" :indexed? false :base-type :type/Integer}]
[[1 2]]]]
(testing "unique index"
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(sql.tx/create-index-sql driver/*driver* "unique_index" ["column"] {:method "hash"}))
(is (= #{{:type :normal-column-index :value "id"}
{:type :normal-column-index :value "column"}}
@@ -707,7 +708,7 @@
[{:field-name "column" :indexed? false :base-type :type/Integer}]
[[1 2]]]]
(testing "hashed index"
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(sql.tx/create-index-sql driver/*driver* "hashed_index" ["column"] {:unique? true}))
(is (= #{{:type :normal-column-index :value "id"}
{:type :normal-column-index :value "column"}}
@@ -729,9 +730,9 @@
[{:field-name "column" :indexed? false :base-type :type/Integer}]
[[1 2]]]]
(testing "clustered index"
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(sql.tx/create-index-sql driver/*driver* "clustered_index" ["column"]))
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
"CLUSTER clustered_index USING idx_clustered_index_column;")
(is (= #{{:type :normal-column-index :value "id"}
{:type :normal-column-index :value "column"}}
@@ -755,14 +756,14 @@
[{:field-name "column" :indexed? false :base-type :type/Integer}]
[[1 2]]]]
(testing "conditional index are ignored"
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(sql.tx/create-index-sql driver/*driver* "conditional_index" ["column"] {:condition "id > 2"}))
(is (= #{{:type :normal-column-index :value "id"}}
(describe-table-indexes (t2/select-one :model/Table (mt/id :conditional_index)))))))))
(deftest describe-fields-are-sorted-test
(mt/test-drivers (mt/normal-drivers-with-feature :describe-fields)
- (let [returned-fields (into [] (driver/describe-fields driver/*driver* (mt/db)))
+ (let [returned-fields (into [] (driver/describe-fields driver/*driver* (lib.metadata/database (mt/metadata-provider))))
sorted-fields (sort-by (juxt :table-schema :table-name :database-position) returned-fields)]
(is (= sorted-fields
returned-fields)))))
@@ -823,7 +824,7 @@
(testing (if materialized? "Materialized View" "View")
(tx/drop-view! driver/*driver* (mt/db) view-name {:materialized? materialized?})
(tx/create-view-of-table! driver/*driver* (mt/db) view-name table-name {:materialized? materialized?})
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(let [orders-id (:id (tx/metabase-instance (tx/map->TableDefinition {:table-name table-name}) (mt/db)))
view-instance (tx/metabase-instance (tx/map->TableDefinition {:table-name view-name}) (mt/db))
orders-m-id (:id view-instance)
@@ -837,7 +838,7 @@
:model/Field
:table_id orders-m-id
{:order-by [:database_position]})]
- (is (contains? (into #{} (map :name) (:tables (driver/describe-database driver/*driver* (mt/db))))
+ (is (contains? (into #{} (map :name) (:tables (driver/describe-database driver/*driver* (lib.metadata/database (mt/metadata-provider)))))
(:name view-instance)))
(is (some? orders-m-id))
(is (some? orders-id))
diff --git a/test/metabase/driver/sql_jdbc_test.clj b/test/metabase/driver/sql_jdbc_test.clj
index ac42a7ce6d3..2c7f2154dc9 100644
--- a/test/metabase/driver/sql_jdbc_test.clj
+++ b/test/metabase/driver/sql_jdbc_test.clj
@@ -31,14 +31,14 @@
"ORDERS" "PEOPLE" "PRODUCTS" "REVIEWS"]]
{:name table, :schema "PUBLIC",
:description nil, :is_writable true}))
- (into #{} (:tables (driver/describe-database :h2 (mt/db)))))))
+ (into #{} (:tables (driver/describe-database :h2 (lib.metadata/database (mt/metadata-provider))))))))
(deftest describe-fields-sync-with-composite-pks-test
(testing "Make sure syncing a table that has a composite pks works"
(mt/test-driver (mt/normal-drivers-with-feature :describe-fields)
(mt/dataset dataset-definition-test/composite-pk
(let [songs (t2/select-one :model/Table (mt/id :songs))
- fk-metadata (driver/describe-fields driver/*driver* (mt/db)
+ fk-metadata (driver/describe-fields driver/*driver* (lib.metadata/database (mt/metadata-provider))
:table-names [(:name songs)]
:schema-names [(:schema songs)])]
(is (= #{{:name "song_id", :pk? true} {:name "artist_id", :pk? true}}
@@ -68,7 +68,7 @@
{:name "Wurstküche", :price 2, :category_id 29, :id 4}
{:name "Brite Spot Family Restaurant", :price 2, :category_id 20, :id 5}]
(for [row (take 5 (sort-by :id (driver/table-rows-seq driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
(t2/select-one :model/Table :id (mt/id :venues)))))]
;; different DBs use different precisions for these
(-> (dissoc row :latitude :longitude)
@@ -238,7 +238,7 @@
(eduction
cat
[(orig metadata) [fake-schema-name]])))]
- (let [syncable (driver/syncable-schemas driver/*driver* (mt/db))]
+ (let [syncable (driver/syncable-schemas driver/*driver* (lib.metadata/database (mt/metadata-provider)))]
(is (contains? syncable "public"))
(is (contains? syncable fake-schema-name))))
(let [driver (driver.u/database->driver (mt/db))
@@ -251,14 +251,14 @@
(update :details assoc filter-type-prop "inclusion", patterns-type-prop "public")
;; so we don't stomp on the connection pool for the normal test DB.
(assoc :id Integer/MAX_VALUE))
- syncable (driver/syncable-schemas driver/*driver* db-filtered)]
+ syncable (driver/syncable-schemas driver/*driver* (driver.u/ensure-lib-database db-filtered))]
(is (contains? syncable "public"))
(is (not (contains? syncable fake-schema-name)))))
(testing "with an exclusion filter"
(let [db-filtered (-> (mt/db)
(update :details assoc filter-type-prop "exclusion", patterns-type-prop "public")
(assoc :id Integer/MAX_VALUE))
- syncable (driver/syncable-schemas driver/*driver* db-filtered)]
+ syncable (driver/syncable-schemas driver/*driver* (driver.u/ensure-lib-database db-filtered))]
(is (not (contains? syncable "public")))
(is (not (contains? syncable fake-schema-name))))))))))
@@ -348,8 +348,8 @@
(defn- table-rows
"Helper function to get table rows for testing"
[qualified-table-name]
- (->> (driver/table-rows-seq driver/*driver* (mt/db) {:name (name qualified-table-name)
- :schema (namespace qualified-table-name)})
+ (->> (driver/table-rows-seq driver/*driver* (lib.metadata/database (mt/metadata-provider)) {:name (name qualified-table-name)
+ :schema (namespace qualified-table-name)})
(map #(vector (:id %) (:name %)))
sort))
@@ -381,10 +381,10 @@
(driver/rename-tables! driver db-id
{qualified-table-1 qualified-temp-1
qualified-table-2 qualified-temp-2})
- (is (driver/table-exists? driver (mt/db) {:name temp-table-1 :schema schema}))
- (is (driver/table-exists? driver (mt/db) {:name temp-table-2 :schema schema}))
- (is (not (driver/table-exists? driver (mt/db) {:name test-table-1 :schema schema})))
- (is (not (driver/table-exists? driver (mt/db) {:name test-table-2 :schema schema})))
+ (is (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name temp-table-1 :schema schema}))
+ (is (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name temp-table-2 :schema schema}))
+ (is (not (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name test-table-1 :schema schema})))
+ (is (not (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name test-table-2 :schema schema})))
(is (= test-data-1 (table-rows qualified-temp-1)))
(is (= test-data-2 (table-rows qualified-temp-2)))
(driver/rename-tables! driver db-id
@@ -400,11 +400,11 @@
{qualified-table-1 qualified-temp-1
qualified-table-2 qualified-conflict})))
(testing "original tables should still exist after failed atomic rename"
- (is (driver/table-exists? driver (mt/db) {:name test-table-1 :schema schema}))
- (is (driver/table-exists? driver (mt/db) {:name test-table-2 :schema schema})))
+ (is (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name test-table-1 :schema schema}))
+ (is (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name test-table-2 :schema schema})))
(testing "temp tables should not exist after failed atomic rename"
- (is (not (driver/table-exists? driver (mt/db) {:name temp-table-1 :schema schema})))
- (is (not (driver/table-exists? driver (mt/db) {:name temp-table-2 :schema schema}))))
+ (is (not (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name temp-table-1 :schema schema})))
+ (is (not (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name temp-table-2 :schema schema}))))
(testing "original data should be intact after failed atomic rename"
(is (= test-data-1 (table-rows qualified-table-1)))
(is (= test-data-2 (table-rows qualified-table-2))))
@@ -431,14 +431,14 @@
(try
(testing "single table rename works correctly"
(driver/rename-table! driver db-id qualified-table qualified-renamed)
- (is (driver/table-exists? driver (mt/db) {:name renamed-table :schema schema})
+ (is (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name renamed-table :schema schema})
"Renamed table should exist")
- (is (not (driver/table-exists? driver (mt/db) {:name test-table :schema schema}))
+ (is (not (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name test-table :schema schema}))
"Original table should not exist"))
(finally
- (when (driver/table-exists? driver (mt/db) {:name renamed-table :schema schema})
+ (when (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name renamed-table :schema schema})
(driver/drop-table! driver db-id qualified-renamed))
- (when (driver/table-exists? driver (mt/db) {:name test-table :schema schema})
+ (when (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:name test-table :schema schema})
(driver/drop-table! driver db-id qualified-table)))))))))
(defn- sql-jdbc-drivers
diff --git a/test/metabase/driver/sqlite_test.clj b/test/metabase/driver/sqlite_test.clj
index 2a3df51a46c..f082010342a 100644
--- a/test/metabase/driver/sqlite_test.clj
+++ b/test/metabase/driver/sqlite_test.clj
@@ -312,7 +312,7 @@
(deftest default-schema-test
(mt/test-driver :sqlite
- (is (nil? (driver.sql/default-schema :sqlite (mt/db))))))
+ (is (nil? (driver.sql/default-schema :sqlite (lib.metadata/database (mt/metadata-provider)))))))
(deftest ^:parallel database-error-while-reading-rows-is-invalid-query-test
(mt/test-driver :sqlite
@@ -356,7 +356,7 @@
(deftest ^:parallel timezone-id-test
(mt/test-driver :sqlite
(is (= "UTC"
- (driver/db-default-timezone :sqlite (mt/db))))))
+ (driver/db-default-timezone :sqlite (lib.metadata/database (mt/metadata-provider)))))))
(deftest ^:parallel hour-bucketing-time-without-database-type-test
(testing (str "Hour bucketing on a TIME-typed expression without `:database-type` (as happens for "
@@ -417,7 +417,7 @@
(map (fn [m] [(:table m) m])
(sql-jdbc.sync/current-user-table-privileges
:sqlite
- (sql-jdbc.conn/db->pooled-connection-spec (mt/db))))))))))
+ (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))))))))))
(deftest ^:parallel filter-by-date-test
(testing "Make sure filtering against a LocalDate works correctly in SQLite"
@@ -460,7 +460,7 @@
"create view if not exists v_src as select id, strftime('%s', time) as time from src;"
"insert into src values(1, '2020-03-01 12:20:35');"]]
(jdbc/execute! spec [statement]))
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(is (= [{:name "src"
:fields [{:name "id"
:base_type :type/Integer}
@@ -492,7 +492,7 @@
GROUP BY symbol
ORDER by dt;"]]
(jdbc/execute! spec [statement]))
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(is (= [{:name "groupby_test"
:fields [{:name "id"
:base_type :type/Integer}
@@ -529,12 +529,12 @@
(jdbc/execute! (sql-jdbc.conn/connection-details->spec driver details)
[stmt]))
(mt/with-temp [:model/Database db {:engine driver :details (assoc details :dbname db-name)}]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(mt/with-db db
(testing "timestamp columns"
(testing "database should be synced"
(is (= (set (map default-table-result ["timestamp_table"]))
- (into #{} (:tables (driver/describe-database driver db))))))
+ (into #{} (:tables (driver/describe-database driver (driver.u/ensure-lib-database db)))))))
(testing "timestamp column should exist"
(is (=? {:name "timestamp_table"
:schema nil
@@ -547,7 +547,7 @@
:database-is-auto-increment false
:database-is-nullable true
:database-is-generated false}]}
- (-> (driver/describe-table driver db (t2/select-one :model/Table :id (mt/id :timestamp_table)))
+ (-> (driver/describe-table driver (driver.u/ensure-lib-database db) (t2/select-one :model/Table :id (mt/id :timestamp_table)))
(update :fields (partial sort-by :name)))))))))))))
(deftest select-query-datetime
@@ -570,7 +570,7 @@
(jdbc/execute! (sql-jdbc.conn/connection-details->spec :sqlite details)
[stmt]))
(mt/with-temp [:model/Database db {:engine :sqlite :details (assoc details :dbname db-name)}]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
;; In SQLite, you can actually store any value in any date/timestamp column,
;; let's test only values we'd reasonably run into.
;; Caveat: TIMESTAMP stored as string doesn't get parsed and is returned as-is by the driver,
diff --git a/test/metabase/driver/util_test.clj b/test/metabase/driver/util_test.clj
index 092c8eecda2..13298f3286e 100644
--- a/test/metabase/driver/util_test.clj
+++ b/test/metabase/driver/util_test.clj
@@ -885,9 +885,9 @@
(try
(testing "an attached DWH on a private address still gets a connection pool"
;; fetch by id so the database takes the metadata-provider path, which must carry `is-attached-dwh`
- (is (some? (sql-jdbc.conn/db->pooled-connection-spec (u/the-id database)))))
+ (is (some? (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database (u/the-id database))))))
(finally
- (sql-jdbc.conn/invalidate-pool-for-db! database))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (:id database)))))))
;; write the ordinary row under allow-all, the way a formerly-lax instance would have, then flip the policy
(mt/with-temp-env-var-value! [mb-warehouse-allowed-networks "allow-all"]
(mt/with-temp [:model/Database database {:engine :postgres
@@ -897,7 +897,7 @@
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"private or internal network address"
- (sql-jdbc.conn/db->pooled-connection-spec (u/the-id database))))))))))
+ (sql-jdbc.conn/db->pooled-connection-spec (driver.u/lib-database (u/the-id database)))))))))))
(deftest warehouse-allowed-networks-default-test
(testing "self-hosted, with nothing configured, all networks are allowed"
diff --git a/test/metabase/driver_test.clj b/test/metabase/driver_test.clj
index 716f99bfc9e..02eff38e300 100644
--- a/test/metabase/driver_test.clj
+++ b/test/metabase/driver_test.clj
@@ -88,12 +88,12 @@
(mt/test-drivers (mt/normal-drivers)
(if (driver/database-supports? driver/*driver* :schemas (mt/db))
(testing "`describe-database` should return schemas with tables if the database supports schemas"
- (is (some? (->> (driver/describe-database driver/*driver* (mt/db))
+ (is (some? (->> (driver/describe-database driver/*driver* (lib.metadata/database (mt/metadata-provider)))
:tables
(into [])
(some :schema)))))
(testing "`describe-database` should not return schemas with tables if the database doesn't support schemas"
- (is (nil? (->> (driver/describe-database driver/*driver* (mt/db))
+ (is (nil? (->> (driver/describe-database driver/*driver* (lib.metadata/database (mt/metadata-provider)))
:tables
(into [])
(some :schema))))))))
@@ -123,7 +123,7 @@
(is (true? (binding [driver.settings/*allow-testing-h2-connections* true]
(driver/can-connect? driver/*driver* details)))))
;; release db resources like connection pools so we don't have to wait to finish syncing before destroying the db
- (driver/notify-database-updated driver/*driver* db)
+ (driver/notify-database-updated driver/*driver* (driver.u/lib-database (:id db)))
(testing "after deleting a database, can-connect? should return false or throw an exception"
(let [;; in the case of some cloud databases, the test database is never created, and can't or shouldn't be destroyed.
;; so fake it by changing the database details
@@ -168,7 +168,7 @@
(is (= {:status "ok"}
(mt/user-http-request :crowberto :post 200 (str "/database/" (u/the-id db) "/sync_schema")))))))
;; release db resources like connection pools so we don't have to wait to finish syncing before destroying the db
- (driver/notify-database-updated driver/*driver* db)
+ (driver/notify-database-updated driver/*driver* (driver.u/lib-database (:id db)))
;; destroy the db
(if (driver/database-supports? driver/*driver* :test/cannot-destroy-db (mt/db))
;; in the case of some cloud databases, the test database is never created, and can't or shouldn't be destroyed.
@@ -188,7 +188,7 @@
(deftest ^:parallel supports-table-privileges-matches-implementations-test
(mt/test-drivers (mt/normal-drivers-with-feature :table-privileges)
- (is (some? (driver/current-user-table-privileges driver/*driver* (mt/db))))))
+ (is (some? (driver/current-user-table-privileges driver/*driver* (lib.metadata/database (mt/metadata-provider)))))))
(deftest ^:parallel mongo-prettify-native-form-test
(mt/test-driver :mongo
@@ -278,9 +278,9 @@
(mt/test-drivers (mt/normal-drivers-with-feature :metadata/table-existence-check)
(let [venues-table (t2/select-one :model/Table :id (mt/id :venues))
fake-table {:name "fake_table_xyz123" :schema (:schema venues-table)}]
- (is (driver/table-exists? driver/*driver* (mt/db) venues-table)
+ (is (driver/table-exists? driver/*driver* (lib.metadata/database (mt/metadata-provider)) venues-table)
(str "Driver " driver/*driver* " should detect that venues table exists"))
- (is (not (driver/table-exists? driver/*driver* (mt/db) fake-table))
+ (is (not (driver/table-exists? driver/*driver* (lib.metadata/database (mt/metadata-provider)) fake-table))
(str "Driver " driver/*driver* " should detect that fake table doesn't exist"))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -291,10 +291,10 @@
(sort-by :database-position
(if (driver/database-supports? driver/*driver* :describe-fields db)
(mapv #(dissoc % :table-name :table-schema)
- (driver/describe-fields driver/*driver* db
+ (driver/describe-fields driver/*driver* (driver.u/ensure-lib-database db)
:schema-names [(:schema table)]
:table-names [(:name table)]))
- (:fields (driver/describe-table driver/*driver* db table)))))
+ (:fields (driver/describe-table driver/*driver* (driver.u/ensure-lib-database db) table)))))
(deftest ^:parallel describe-fields-or-table-test
(testing "test `describe-fields` or `describe-table` returns some basic metadata"
@@ -398,10 +398,10 @@
orders (t2/select-one :model/Table (mt/id :orders))
products (t2/select-one :model/Table (mt/id :products))
people (t2/select-one :model/Table (mt/id :people))
- entire-db-fks (driver/describe-fks driver/*driver* db)
- schema-db-fks (driver/describe-fks driver/*driver* db
+ entire-db-fks (driver/describe-fks driver/*driver* (driver.u/ensure-lib-database db))
+ schema-db-fks (driver/describe-fks driver/*driver* (driver.u/ensure-lib-database db)
:schema-names (when (:schema orders) [(:schema orders)]))
- table-db-fks (driver/describe-fks driver/*driver* db
+ table-db-fks (driver/describe-fks driver/*driver* (driver.u/ensure-lib-database db)
:schema-names (when (:schema orders) [(:schema orders)])
:table-names [(:name orders)])]
(doseq [[description orders-fks]
@@ -491,7 +491,7 @@
(let [table (driver/temp-table-name driver/*driver*)]
(driver/do-with-test-connection
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
(fn [conn]
(driver/execute-on-connection! driver/*driver* conn
(driver/compile-create-temp-table
diff --git a/test/metabase/indexes/reconcile_test.clj b/test/metabase/indexes/reconcile_test.clj
index e18215f49a7..7af9187bf86 100644
--- a/test/metabase/indexes/reconcile_test.clj
+++ b/test/metabase/indexes/reconcile_test.clj
@@ -3,6 +3,7 @@
[clojure.test :refer :all]
[metabase.driver]
[metabase.indexes.reconcile :as reconcile]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.test :as mt]))
(def ^:private managed-btree
@@ -131,14 +132,14 @@
(testing "delegates to the driver method"
(with-redefs [metabase.driver/fetch-table-indexes (fn [_driver _db _schema _table] [wh-btree])]
(is (= [wh-btree]
- (reconcile/fetch-warehouse-indexes (mt/db) "public" "t")))))
+ (reconcile/fetch-warehouse-indexes (lib.metadata/database (mt/metadata-provider)) "public" "t")))))
(testing "preserves a successful empty warehouse response"
(with-redefs [metabase.driver/fetch-table-indexes (fn [& _] [])]
- (is (= [] (reconcile/fetch-warehouse-indexes (mt/db) "public" "t")))))
+ (is (= [] (reconcile/fetch-warehouse-indexes (lib.metadata/database (mt/metadata-provider)) "public" "t")))))
(testing "lets driver/connection errors through, so callers can report the reason"
(with-redefs [metabase.driver/fetch-table-indexes (fn [& _] (throw (ex-info "boom" {})))]
(is (thrown-with-msg? clojure.lang.ExceptionInfo #"boom"
- (reconcile/fetch-warehouse-indexes (mt/db) "public" "t"))))))
+ (reconcile/fetch-warehouse-indexes (lib.metadata/database (mt/metadata-provider)) "public" "t"))))))
(deftest ^:parallel driver-error-message-test
(testing "a driver with no trimming of its own passes the message through"
diff --git a/test/metabase/model_persistence/test_util.clj b/test/metabase/model_persistence/test_util.clj
index 8f6a5e476e8..1d6ef01d30f 100644
--- a/test/metabase/model_persistence/test_util.clj
+++ b/test/metabase/model_persistence/test_util.clj
@@ -1,6 +1,7 @@
(ns metabase.model-persistence.test-util
(:require
[metabase.driver.ddl.interface :as ddl.i]
+ [metabase.driver.util :as driver.u]
[metabase.model-persistence.models.persisted-info :as persisted-info]
[metabase.model-persistence.task.persist-refresh :as task.persist-refresh]
[metabase.test.data :as data]
@@ -11,7 +12,7 @@
[f]
(tu/with-temporary-setting-values [:persisted-models-enabled true]
(dynamic-redefs/with-dynamic-fn-redefs [persisted-info/default-persistent-info-state (fn [] "creating")]
- (ddl.i/check-can-persist (data/db))
+ (ddl.i/check-can-persist (driver.u/lib-database (data/id)))
(persisted-info/ready-database! (data/id))
(let [persist-fn (fn persist-fn []
(#'task.persist-refresh/refresh-tables!
diff --git a/test/metabase/query_processor/parameters/values_test.clj b/test/metabase/query_processor/parameters/values_test.clj
index 876d4a5e12b..f54de83af72 100644
--- a/test/metabase/query_processor/parameters/values_test.clj
+++ b/test/metabase/query_processor/parameters/values_test.clj
@@ -472,7 +472,7 @@
order by c_orig.id desc limit 3"
(u/the-id model))
tag-name (format "#%d" (u/the-id model))]
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
[update-query])
(is (= [["Winery" "Winery from cached table"]
["Wine Bar" "Wine Bar from cached table"]
diff --git a/test/metabase/query_processor/persistence_test.clj b/test/metabase/query_processor/persistence_test.clj
index 200b8984a88..ebdfaf58453 100644
--- a/test/metabase/query_processor/persistence_test.clj
+++ b/test/metabase/query_processor/persistence_test.clj
@@ -46,7 +46,7 @@
(mt/test-drivers (mt/normal-drivers-with-feature :persist-models)
(testing (str driver/*driver* " can persist")
(mt/dataset test-data
- (let [[success? error] (ddl.i/check-can-persist (mt/db))]
+ (let [[success? error] (ddl.i/check-can-persist (lib.metadata/database (mt/metadata-provider)))]
(is success? (str "Not able to persist on " driver/*driver*))
(is (= :persist.check/valid error)))
(testing "Populates the `cache_info` table with v1 information"
diff --git a/test/metabase/sample_data/impl_test.clj b/test/metabase/sample_data/impl_test.clj
index 17737c05e49..d86377210c1 100644
--- a/test/metabase/sample_data/impl_test.clj
+++ b/test/metabase/sample_data/impl_test.clj
@@ -6,6 +6,7 @@
[clojure.test :refer :all]
[metabase.config.core :as config]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.permissions.core :as perms]
[metabase.plugins.impl :as plugins]
[metabase.query-processor :as qp]
@@ -36,7 +37,7 @@
{:style/indent 1}
[[db-binding] & body]
`(mt/with-temp [:model/Database db# (sample-database-db)]
- (sync/sync-database! db#)
+ (sync/sync-database! (:id db#))
(let [~db-binding db#]
~@body)))
@@ -60,7 +61,7 @@
(let [h2-db (t2/insert-returning-instance! :model/Database
{:name "Sample Database" :engine :h2 :is_sample true
:details (#'sample-data/try-to-extract-sample-database! :h2)})]
- (sync/sync-database! h2-db)
+ (sync/sync-database! (:id h2-db))
(let [orders-id (t2/select-one-pk :model/Table :db_id (:id h2-db) :name "ORDERS")
total-id (t2/select-one-pk :model/Field :table_id orders-id :name "TOTAL")
user-card (t2/insert-returning-instance! :model/Card
@@ -97,7 +98,7 @@
(let [sqlite-db (t2/insert-returning-instance! :model/Database
{:name "Sample Database" :engine :sqlite :is_sample true
:details (#'sample-data/try-to-extract-sample-database! :sqlite)})]
- (sync/sync-database! sqlite-db)
+ (sync/sync-database! (:id sqlite-db))
(let [orders-id (t2/select-one-pk :model/Table :db_id (:id sqlite-db) :name "ORDERS")
total-id (t2/select-one-pk :model/Field :table_id orders-id :name "TOTAL")
user-card (t2/insert-returning-instance! :model/Card
@@ -154,7 +155,7 @@
(let [h2-db (t2/insert-returning-instance! :model/Database
{:name "Sample Database" :engine :h2 :is_sample true
:details (#'sample-data/try-to-extract-sample-database! :h2)})]
- (sync/sync-database! h2-db)
+ (sync/sync-database! (:id h2-db))
(let [db-id (:id h2-db)
orders-id (t2/select-one-pk :model/Table :db_id db-id :name "ORDERS")
people-id (t2/select-one-pk :model/Table :db_id db-id :name "PEOPLE")]
@@ -233,9 +234,9 @@
(deftest sample-database-is-read-only-test
(testing "The Sample Database connection is read-only: reads succeed but INSERT/UPDATE/DELETE are rejected"
(mt/with-temp [:model/Database db (sample-database-db)]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(mt/with-db db
- (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))]
+ (let [conn-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))]
(testing "reads succeed"
(is (= [2]
(->> (jdbc/query conn-spec "SELECT QUANTITY FROM ORDERS WHERE ID = 1;")
diff --git a/test/metabase/sync/analyze/classify_test.clj b/test/metabase/sync/analyze/classify_test.clj
index 7c784f864a9..374ca30ebf7 100644
--- a/test/metabase/sync/analyze/classify_test.clj
+++ b/test/metabase/sync/analyze/classify_test.clj
@@ -2,6 +2,7 @@
(:require
[clojure.test :refer :all]
[metabase.analyze.classifiers.no-preview-display :as classifiers.no-preview-display]
+ [metabase.driver.util :as driver.u]
[metabase.sync.analyze.classify :as classify]
[metabase.sync.interface :as i]
[metabase.test :as mt]
@@ -77,7 +78,7 @@
:global {:distinct-count 3}}
:last_analyzed nil}]
(is (nil? (:semantic_type (t2/select-one :model/Field :id (u/the-id field)))))
- (classify/classify-fields-for-db! db (constantly nil))
+ (classify/classify-fields-for-db! (driver.u/lib-database (:id db)) (constantly nil))
(is (= :type/Income (:semantic_type (t2/select-one :model/Field :id (u/the-id field))))))))
(deftest classify-decimal-fields-test
@@ -95,7 +96,7 @@
:global {:distinct-count 3}}
:last_analyzed nil}]
(is (nil? (:semantic_type (t2/select-one :model/Field :id (u/the-id field)))))
- (classify/classify-fields-for-db! db (constantly nil))
+ (classify/classify-fields-for-db! (driver.u/lib-database (:id db)) (constantly nil))
(is (= :type/Income (:semantic_type (t2/select-one :model/Field :id (u/the-id field))))))))
(deftest single-name-field-per-table-test
diff --git a/test/metabase/sync/analyze/data_sensitivity_test.clj b/test/metabase/sync/analyze/data_sensitivity_test.clj
index a60baf3b14e..5ebb9be6305 100644
--- a/test/metabase/sync/analyze/data_sensitivity_test.clj
+++ b/test/metabase/sync/analyze/data_sensitivity_test.clj
@@ -2,6 +2,7 @@
(:require
[clojure.test :refer :all]
[metabase.analyze.core :as analyze]
+ [metabase.driver.util :as driver.u]
[metabase.sync.analyze.data-sensitivity :as sync.data-sensitivity]
[metabase.sync.core :as sync]
[metabase.test :as mt]
@@ -61,10 +62,10 @@
:model/Field _ {:table_id (:id foreign) :name "f" :base_type :type/Text}]
(testing "hidden and cruft tables with unscanned fields are selected; converged, inactive, and foreign tables are not"
(is (= #{(:id hidden) (:id cruft)}
- (#'sync.data-sensitivity/table-ids-with-unscanned-fields db false))))
+ (#'sync.data-sensitivity/table-ids-with-unscanned-fields (driver.u/lib-database (:id db)) false))))
(testing "force reselects the table whose only unlabeled field is PUBLIC"
(is (= #{(:id hidden) (:id cruft) (:id done)}
- (#'sync.data-sensitivity/table-ids-with-unscanned-fields db true))))))
+ (#'sync.data-sensitivity/table-ids-with-unscanned-fields (driver.u/lib-database (:id db)) true))))))
(deftest scan-fields-for-db-converges-test
(mt/with-temporary-setting-values [data-sensitivity-scan-enabled true]
@@ -74,14 +75,14 @@
:model/Field foo {:table_id (:id table) :name "foo" :base_type :type/Text}]
(testing "the first scan labels every unscanned field, PUBLIC when nothing matches, without creating mirror rows"
(is (= {:fields-scanned 2 :fields-labeled 1 :fields-failed 0}
- (sync.data-sensitivity/scan-fields-for-db! db nil)))
+ (sync.data-sensitivity/scan-fields-for-db! (driver.u/lib-database (:id db)) nil)))
(is (= :PII (label ssn)))
(is (= :PUBLIC (label foo)))
(is (nil? (mirror-label ssn)))
(is (nil? (mirror-label foo))))
(testing "a second scan selects nothing"
(is (= {:fields-scanned 0 :fields-labeled 0 :fields-failed 0}
- (sync.data-sensitivity/scan-fields-for-db! db nil)))))))
+ (sync.data-sensitivity/scan-fields-for-db! (driver.u/lib-database (:id db)) nil)))))))
(deftest setting-gates-scans-test
(mt/with-temporary-setting-values [data-sensitivity-scan-enabled false]
@@ -90,13 +91,13 @@
:model/Field ssn {:table_id (:id table) :name "ssn" :base_type :type/Text}]
(testing "with the setting off both entry points return zero stats and write nothing"
(is (= {:fields-scanned 0 :fields-labeled 0 :fields-failed 0}
- (sync.data-sensitivity/scan-fields-for-db! db nil)))
+ (sync.data-sensitivity/scan-fields-for-db! (driver.u/lib-database (:id db)) nil)))
(is (= {:fields-scanned 0 :fields-labeled 0 :fields-failed 0}
(sync.data-sensitivity/scan-table! table)))
(is (nil? (label ssn))))
(testing "the REPL entry point ignores the setting"
(is (= {:fields-scanned 1 :fields-labeled 1 :fields-failed 0}
- (sync.data-sensitivity/scan-data-sensitivity! db)))
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db))))
(is (= :PII (label ssn)))))))
(deftest force-rescans-public-only-test
@@ -105,16 +106,16 @@
:model/Table table {:db_id (:id db) :name "app_users"}
:model/Field ssn {:table_id (:id table) :name "ssn" :base_type :type/Text}
:model/Field foo {:table_id (:id table) :name "foo" :base_type :type/Text}]
- (sync.data-sensitivity/scan-fields-for-db! db nil)
+ (sync.data-sensitivity/scan-fields-for-db! (driver.u/lib-database (:id db)) nil)
(is (= :PUBLIC (label foo)))
(mt/with-dynamic-fn-redefs [analyze/infer-data-sensitivity (constantly :PHI)]
(testing "a scheduled scan does not revisit PUBLIC fields even when the rules would now match"
(is (= {:fields-scanned 0 :fields-labeled 0 :fields-failed 0}
- (sync.data-sensitivity/scan-fields-for-db! db nil)))
+ (sync.data-sensitivity/scan-fields-for-db! (driver.u/lib-database (:id db)) nil)))
(is (= :PUBLIC (label foo))))
(testing "a forced scan revisits PUBLIC fields and leaves categorized fields alone"
(is (= {:fields-scanned 1 :fields-labeled 1 :fields-failed 0}
- (sync.data-sensitivity/scan-data-sensitivity! db :force? true)))
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db) :force? true)))
(is (= :PHI (label foo)))
(is (= :PII (label ssn))))))))
@@ -133,7 +134,7 @@
(is (= :PHI (label notes))))
(testing "the scan still classifies both fields on the raw Field"
(is (= {:fields-scanned 2 :fields-labeled 1 :fields-failed 0}
- (sync.data-sensitivity/scan-data-sensitivity! db))))
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db)))))
(testing "the classifier wrote its own label, and the user's still wins on read"
(is (= :PII (sync-label ssn)))
(is (= :PUBLIC (mirror-label ssn)))
@@ -142,7 +143,7 @@
(is (= :PHI (mirror-label notes)))
(is (= :PHI (label notes))))
(testing "a forced rescan does not revisit a categorized Field, mirror or not"
- (sync.data-sensitivity/scan-data-sensitivity! db :force? true)
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db) :force? true)
(is (= :PII (sync-label ssn)))
(is (= :PUBLIC (mirror-label ssn)))
(is (= :PUBLIC (label ssn)))))))
@@ -161,16 +162,16 @@
:model/Field far {:table_id (:id foreign) :name "ssn" :base_type :type/Text}]
(field-user-settings/upsert-user-settings notes {:data_sensitivity :PHI})
(t2/insert! :model/FieldUserSettings {:field_id (:id email)})
- (sync.data-sensitivity/scan-data-sensitivity! db)
- (sync.data-sensitivity/scan-data-sensitivity! other-db)
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db))
+ (sync.data-sensitivity/scan-data-sensitivity! (:id other-db))
(is (= [:PII :PUBLIC :PUBLIC :PII :PUBLIC :PII]
(map sync-label [ssn foo notes email total far])))
(testing "a table scope clears only that table's classifier labels"
- (is (= 1 (sync.data-sensitivity/reset-data-sensitivity! orders)))
+ (is (= 1 (sync.data-sensitivity/reset-table-data-sensitivity! orders)))
(is (nil? (sync-label total)))
(is (= :PII (sync-label ssn))))
(testing "a database scope clears categories and PUBLIC alike, including fields with a label-less mirror row"
- (is (= 3 (sync.data-sensitivity/reset-data-sensitivity! db)))
+ (is (= 3 (sync.data-sensitivity/reset-data-sensitivity! (:id db))))
(is (= [nil nil nil] (map sync-label [ssn foo email]))))
(testing "a label backed by the mirror is human-set; the reset clears only the classifier's own label"
(is (= :PUBLIC (sync-label notes)))
@@ -179,7 +180,7 @@
(testing "other databases are untouched"
(is (= :PII (sync-label far))))
(testing "a second reset finds nothing"
- (is (zero? (sync.data-sensitivity/reset-data-sensitivity! db))))))
+ (is (zero? (sync.data-sensitivity/reset-data-sensitivity! (:id db)))))))
(deftest scan-with-reset-relabels-categorized-fields-test
(mt/with-temp [:model/Database db {}
@@ -187,15 +188,15 @@
:model/Field ssn {:table_id (:id table) :name "ssn" :base_type :type/Text}
:model/Field notes {:table_id (:id table) :name "notes" :base_type :type/Text}]
(field-user-settings/upsert-user-settings notes {:data_sensitivity :PHI})
- (sync.data-sensitivity/scan-data-sensitivity! db)
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db))
(is (= :PII (sync-label ssn)))
(mt/with-dynamic-fn-redefs [analyze/infer-data-sensitivity (constantly :SEC_KEY)]
(testing "a forced scan leaves a categorized field on its old label"
- (sync.data-sensitivity/scan-data-sensitivity! db :force? true)
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db) :force? true)
(is (= :PII (sync-label ssn))))
(testing "a reset scan recomputes it under the current rules and reports the reset count"
(is (= {:fields-scanned 1 :fields-labeled 1 :fields-failed 0 :fields-reset 1}
- (sync.data-sensitivity/scan-data-sensitivity! db :reset? true)))
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db) :reset? true)))
(is (= :SEC_KEY (sync-label ssn))))
(testing "the user's label in the mirror is untouched, and is still what readers see"
(is (= :SEC_KEY (sync-label notes)) "reclassified by the earlier forced scan, which also revisits PUBLIC fields")
@@ -210,10 +211,10 @@
(mt/with-dynamic-fn-redefs [analyze/infer-data-sensitivity (fn [& _] (swap! calls inc) (throw (ex-info "boom" {})))]
(testing "a throwing rule counts as scanned and failed and leaves the field unscanned"
(is (= {:fields-scanned 1 :fields-labeled 0 :fields-failed 1}
- (sync.data-sensitivity/scan-data-sensitivity! db)))
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db))))
(is (nil? (label ssn))))
(testing "the next scan retries the field"
- (sync.data-sensitivity/scan-data-sensitivity! db)
+ (sync.data-sensitivity/scan-data-sensitivity! (:id db))
(is (= 2 @calls)))))))
(deftest sync-database-end-to-end-test
@@ -232,7 +233,7 @@
{:field-name "price", :base-type :type/Float}]
[["Widget" "Berlin" 9.99]]]]
(mt/with-temporary-setting-values [data-sensitivity-scan-enabled true]
- (sync/sync-database! (mt/db)))
+ (sync/sync-database! (mt/id)))
(let [labels (fn [table]
(into {} (map (juxt (comp u/lower-case-en :name) :data_sensitivity))
(t2/select [:model/Field :name :data_sensitivity] :table_id (mt/id table))))]
@@ -272,7 +273,7 @@
(t2/update! :model/Field :id [:in field-ids] {:data_sensitivity nil})
(is (= #{[:type/IPAddress :type/IPAddress]}
(t2/select-fn-set (juxt :base_type :semantic_type) :model/Field :id [:in field-ids])))
- (sync/scan-data-sensitivity! (mt/db))
+ (sync/scan-data-sensitivity! (mt/id))
(is (= {"client_ip" :SYS_TELEMETRY "addr" :SYS_TELEMETRY}
(t2/select-fn->fn :name :data_sensitivity :model/Field :id [:in field-ids]))))))))
@@ -285,7 +286,7 @@
:model/Field creds {:table_id (:id table) :name "credentials" :base_type :type/JSON}
:model/Field meta {:table_id (:id table) :name "metadata" :base_type :type/JSON}]
(is (= {:fields-scanned 4 :fields-labeled 3 :fields-failed 0}
- (sync.data-sensitivity/scan-data-sensitivity! table)))
+ (sync.data-sensitivity/scan-table-data-sensitivity! table)))
(is (= :SYS_TELEMETRY (label ip)))
(is (= :SYS_TELEMETRY (label addr)))
(is (= :SEC_KEY (label creds)))
@@ -299,6 +300,6 @@
:model/Field email {:table_id (:id other) :name "email" :base_type :type/Text}]
(testing "scanning a Table instance labels only that table's fields"
(is (= {:fields-scanned 1 :fields-labeled 1 :fields-failed 0}
- (sync.data-sensitivity/scan-data-sensitivity! users)))
+ (sync.data-sensitivity/scan-table-data-sensitivity! users)))
(is (= :PII (label ssn)))
(is (nil? (label email))))))
diff --git a/test/metabase/sync/analyze/fingerprint_test.clj b/test/metabase/sync/analyze/fingerprint_test.clj
index 3722616e4b1..809c1a236dd 100644
--- a/test/metabase/sync/analyze/fingerprint_test.clj
+++ b/test/metabase/sync/analyze/fingerprint_test.clj
@@ -4,6 +4,7 @@
[clojure.test :refer :all]
[metabase.analyze.fingerprint.fingerprinters :as fingerprinters]
[metabase.app-db.core :as app-db]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.query-processor :as qp]
[metabase.sync.analyze.fingerprint :as sync.fingerprint]
[metabase.sync.db :as sync.db]
@@ -339,7 +340,7 @@
(with-redefs [sync.fingerprint/save-fingerprint! (constantly nil)
sync.fingerprint/max-refingerprint-field-count 31] ;; prime number so we don't have exact matches
(let [results (sync.fingerprint/refingerprint-fields-for-db!
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
(constantly nil))
attempted (:fingerprints-attempted results)]
;; it can exceed the max field count as our resolution is after each table check it.
@@ -407,5 +408,5 @@
sync.fingerprint/*refingerprint?* true]
(is (= (assoc (sync.fingerprint/empty-stats-map 0)
:failed-fingerprints 1)
- (sync.fingerprint/fingerprint-fields-for-db! (mt/db) (constantly nil))))
+ (sync.fingerprint/fingerprint-fields-for-db! (lib.metadata/database (mt/metadata-provider)) (constantly nil))))
(is (= 1 @tables-fingerprinted)))))))
diff --git a/test/metabase/sync/analyze/interestingness_test.clj b/test/metabase/sync/analyze/interestingness_test.clj
index ee1bda7a893..a8431ff4134 100644
--- a/test/metabase/sync/analyze/interestingness_test.clj
+++ b/test/metabase/sync/analyze/interestingness_test.clj
@@ -1,6 +1,7 @@
(ns metabase.sync.analyze.interestingness-test
(:require
[clojure.test :refer [deftest is testing]]
+ [metabase.driver.util :as driver.u]
[metabase.interestingness.core :as interestingness]
[metabase.interestingness.dimension :as dim]
[metabase.sync.analyze.interestingness :as sync.interestingness]
@@ -40,11 +41,11 @@
:model/Field field {:table_id (:id table)}]
(is (nil? (t2/select-one-fn :dimension_interestingness :model/Field :id (:id field))))
(is (= {:fields-scored 1 :fields-failed 0}
- (#'sync.interestingness/score-missing-leftovers! database)))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database)))))
(is (some? (t2/select-one-fn :dimension_interestingness :model/Field :id (:id field))))
(testing "once scored, the field is no longer selected"
(is (= {:fields-scored 0 :fields-failed 0}
- (#'sync.interestingness/score-missing-leftovers! database)))))))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database)))))))))
(deftest score-missing-leftovers-does-not-retry-failed-fields-test
(testing "a field whose scoring attempt failed is not re-attempted by later leftovers passes in this process"
@@ -56,10 +57,10 @@
(swap! calls inc)
(throw (ex-info "boom" {})))]
(is (= {:fields-scored 0 :fields-failed 1}
- (#'sync.interestingness/score-missing-leftovers! database)))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database)))))
(is (= 1 @calls))
(is (= {:fields-scored 0 :fields-failed 0}
- (#'sync.interestingness/score-missing-leftovers! database)))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database)))))
(is (= 1 @calls) "the failed field should be skipped, not re-scored on every sync"))))))
;;; ------------------------------------------- leftovers pass: paging and limits -------------------------------------
@@ -82,7 +83,7 @@
(with-group-size! 2
(fn []
(is (= {:fields-scored 5 :fields-failed 0}
- (#'sync.interestingness/score-missing-leftovers! database))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database))))
"a group size smaller than the field count must not stop the pass early")))
(is (every? some? (map score-of [f1 f2 f3 f4 f5]))))))
@@ -96,7 +97,7 @@
:model/Field _f4 {:table_id (:id table)}
:model/Field _f5 {:table_id (:id table)}]
(mt/with-temporary-setting-values [interestingness-max-fields-per-sync 2]
- (let [stats (#'sync.interestingness/score-missing-leftovers! database)]
+ (let [stats (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database)))]
(is (= 2 (:fields-scored stats)))
(is (= 0 (:fields-failed stats)))
(is (= 3 (:fields-remaining stats))
@@ -106,10 +107,10 @@
(testing "the next run continues on the remainder rather than starting over"
(mt/with-temporary-setting-values [interestingness-max-fields-per-sync 100]
(is (= {:fields-scored 3 :fields-failed 0}
- (#'sync.interestingness/score-missing-leftovers! database)))))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database)))))))
(testing "and a run with nothing left to do reports no remainder"
(is (= {:fields-scored 0 :fields-failed 0}
- (#'sync.interestingness/score-missing-leftovers! database)))))))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database)))))))))
(deftest score-missing-leftovers-keeps-going-past-a-failing-field-test
(testing "a field whose scoring throws does not stop the rest of the run"
@@ -126,7 +127,7 @@
(with-group-size! 1
(fn []
(is (= {:fields-scored 2 :fields-failed 1}
- (#'sync.interestingness/score-missing-leftovers! database))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database))))
"the failing field must not halt the pass, nor stall the cursor on itself"))))
(is (= [0.5 nil 0.5] (map score-of [f1 f2 f3]))))))
@@ -141,7 +142,7 @@
:semantic_type :type/CreationTimestamp
:fingerprint fingerprint}]
(is (= {:fields-scored 1 :fields-failed 0}
- (#'sync.interestingness/score-missing-leftovers! database)))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database)))))
(is (= (interestingness/dimension-interestingness
(t2/select-one :model/Field :id (:id field)))
(score-of field)))))))
@@ -155,7 +156,7 @@
(t2/insert! :model/FieldUserSettings {:field_id (:id hidden)
:visibility_type :sensitive})
(is (= {:fields-scored 1 :fields-failed 0}
- (#'sync.interestingness/score-missing-leftovers! database)))
+ (#'sync.interestingness/score-missing-leftovers! (driver.u/lib-database (:id database)))))
(is (some? (score-of visible)))
(is (nil? (score-of hidden))
"the user-set visibility_type must still exclude the field after the rewrite"))))
diff --git a/test/metabase/sync/analyze_test.clj b/test/metabase/sync/analyze_test.clj
index 3aaa766451b..edb6424cbae 100644
--- a/test/metabase/sync/analyze_test.clj
+++ b/test/metabase/sync/analyze_test.clj
@@ -192,7 +192,7 @@
(defn- analyze-table! [table]
;; we're calling `analyze-db!` instead of `analyze-table!` because the latter doesn't care if you try to sync a
;; hidden table and will allow that. TODO - Does that behavior make sense?
- (analyze/analyze-db! (t2/select-one :model/Database :id (:db_id table))))
+ (analyze/analyze-db! (:id (t2/select-one :model/Database :id (:db_id table)))))
(deftest dont-analyze-hidden-tables-test
(testing "expect all the kinds of hidden tables to stay un-analyzed through transitions and repeated syncing"
diff --git a/test/metabase/sync/api/notify_test.clj b/test/metabase/sync/api/notify_test.clj
index 2e30f56ce9a..8bdd62f3d19 100644
--- a/test/metabase/sync/api/notify_test.clj
+++ b/test/metabase/sync/api/notify_test.clj
@@ -148,7 +148,7 @@
{:request-options api-headers}
(merge {:synchronous? true}
payload)))))
- sync! #(sync/sync-database! database)]
+ sync! #(sync/sync-database! (:id database))]
;; Create the initial table and sync it.
(exec! spec ["CREATE TABLE public.FOO (val bigint NOT NULL);"])
(sync!)
@@ -208,7 +208,7 @@
payload)))))]
;; Create the initial table and sync it.
(exec! spec ["CREATE TABLE public.FOO (val bigint NOT NULL);"])
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [tables (tableset database)]
(is (= #{"public.foo"} tables)))
(testing "We can sync an existing database"
diff --git a/test/metabase/sync/fetch_metadata_test.clj b/test/metabase/sync/fetch_metadata_test.clj
index 000e90b1747..64fa88c88fb 100644
--- a/test/metabase/sync/fetch_metadata_test.clj
+++ b/test/metabase/sync/fetch_metadata_test.clj
@@ -1,6 +1,7 @@
(ns metabase.sync.fetch-metadata-test
(:require
[clojure.test :refer :all]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.sync.fetch-metadata :as fetch-metadata]
[metabase.test :as mt])
(:import
@@ -13,7 +14,7 @@
[e]
(let [attempted (atom 0)
;; outside the redef: the first `mt/db` creates and syncs the test DB, which must not see the failures
- db (mt/db)]
+ db (lib.metadata/database (mt/metadata-provider))]
(mt/with-dynamic-fn-redefs [fetch-metadata/table-fields-metadata (fn [_database _table]
(swap! attempted inc)
(throw e))]
diff --git a/test/metabase/sync/field_values_test.clj b/test/metabase/sync/field_values_test.clj
index fbcdf607320..2b6c8cad9ef 100644
--- a/test/metabase/sync/field_values_test.clj
+++ b/test/metabase/sync/field_values_test.clj
@@ -5,6 +5,7 @@
[clojure.test :refer :all]
[java-time.api :as t]
[metabase.analyze.core :as analyze]
+ [metabase.driver.util :as driver.u]
[metabase.sync.core :as sync]
[metabase.sync.db :as sync.db]
[metabase.sync.field-values :as sync.field-values]
@@ -86,7 +87,7 @@
(letfn [(field-values-after-sync! []
(field-values/get-or-create-full-field-values! (t2/select-one :model/Field (mt/id :venues :price)))
(t2/update! :model/FieldValues :field_id (mt/id :venues :price) :type :full {:values [10 20 30 40]})
- (sync/update-field-values! (data/db))
+ (sync/update-field-values! (:id (data/db)))
(venues-price-field-values))]
(is (= [1 2 3 4] (field-values-after-sync!)))
(mt/user-http-request :crowberto :put 200 (format "table/%d" (mt/id :venues)) {:visibility_type "hidden"})
@@ -318,7 +319,7 @@
(is (=?
{:steps [["delete-expired-advanced-field-values" {}]
["update-field-values" {:throwable #(instance? Exception %)}]]}
- (sync/update-field-values! (data/db))))))))
+ (sync/update-field-values! (:id (data/db)))))))))
;;; ---------------------------------- table->fields-to-scan ----------------------------------
@@ -349,11 +350,11 @@
(testing "SQL driver with :nested-queries and no required filter → batch path"
(mt/with-temp [:model/Database {db-id :id} {:engine :h2}
:model/Table table {:db_id db-id, :name "t", :database_require_filter false}]
- (is (true? (#'sync.field-values/can-batch-distinct? table)))))
+ (is (true? (#'sync.field-values/can-batch-distinct? (driver.u/lib-database db-id) table)))))
(testing "Tables with :database_require_filter true (e.g. partitioned BigQuery) fall back to per-field"
(mt/with-temp [:model/Database {db-id :id} {:engine :h2}
:model/Table table {:db_id db-id, :name "t", :database_require_filter true}]
- (is (false? (#'sync.field-values/can-batch-distinct? table))
+ (is (false? (#'sync.field-values/can-batch-distinct? (driver.u/lib-database db-id) table))
"Partitioned tables that require a WHERE on the partition column can't go through the
native UNION ALL path because it bypasses add-required-filters-if-needed middleware"))))
@@ -392,7 +393,7 @@
{:results (into {} (map (fn [f] [(:id f) {:values []}])) fields)
:queries (count fields)
:failed-fields #{}})]
- (#'sync.field-values/sync-fields-for-table! table [nested-a nested-b plain-c] {})
+ (#'sync.field-values/sync-fields-for-table! (driver.u/lib-database db-id) table [nested-a nested-b plain-c] {})
(testing "Plain field goes to batch path"
(is (= [#{(:id plain-c)}] @calls-to-fetch-distinct-for-table)))
(testing "Both nested-JSON children with the same name go to per-field, preserving each field's identity"
diff --git a/test/metabase/sync/sync_dynamic_test.clj b/test/metabase/sync/sync_dynamic_test.clj
index 8de904b0ad3..89325860df7 100644
--- a/test/metabase/sync/sync_dynamic_test.clj
+++ b/test/metabase/sync/sync_dynamic_test.clj
@@ -36,6 +36,6 @@
(deftest sync-nested-fields-test
(testing "basic test to make sure syncing nested fields works. This is sort of a higher-level test."
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery}]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (= (remove-nonsense toucanery/toucanery-tables-and-fields)
(remove-nonsense (get-tables db)))))))
diff --git a/test/metabase/sync/sync_metadata/comments_test.clj b/test/metabase/sync/sync_metadata/comments_test.clj
index 787ac2575dd..eb53dac7c16 100644
--- a/test/metabase/sync/sync_metadata/comments_test.clj
+++ b/test/metabase/sync/sync_metadata/comments_test.clj
@@ -5,6 +5,7 @@
[clojure.test :refer :all]
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.sync.core :as sync]
[metabase.sync.sync-metadata.tables :as sync-tables]
[metabase.test :as mt]
@@ -78,7 +79,7 @@
;; and re-create a modified version. As such, let the SQL JDBC driver know the DB is being "modified" so it can
;; destroy its current connection pool
(mt/dataset comment-after-sync
- (driver/notify-database-updated driver/*driver* (mt/db))
+ (driver/notify-database-updated driver/*driver* (lib.metadata/database (mt/metadata-provider)))
(let [modified-dbdef (update
comment-after-sync
:table-definitions
@@ -136,7 +137,7 @@
;; change the description in metabase while the source table comment remains the same
(t2/update! :model/Table {:id (mt/id "table_with_updated_desc")} {:description "updated table description"})
;; now sync the DB again, this should NOT overwrite the manually updated description
- (sync-tables/sync-tables-and-database! (mt/db))
+ (sync-tables/sync-tables-and-database! (lib.metadata/database (mt/metadata-provider)))
(is (= #{{:name (get-table-name driver/*driver* "table_with_updated_desc")
:description "updated table description"}}
(db->tables (mt/db)))))))))
@@ -149,12 +150,12 @@
dbdef (basic-table table-name nil)]
(mt/dataset dbdef
;; create the comment
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
[(sql.tx/standalone-table-comment-sql
driver/*driver*
dbdef
(tx/map->TableDefinition {:table-name table-name
:table-comment added-comment}))]
{:transaction? false}) ;; trino needs transactions off
- (sync-tables/sync-tables-and-database! (mt/db))
+ (sync-tables/sync-tables-and-database! (lib.metadata/database (mt/metadata-provider)))
(is (true? (t2/exists? :model/Table :db_id (mt/id) :description added-comment))))))))
diff --git a/test/metabase/sync/sync_metadata/dbms_version_test.clj b/test/metabase/sync/sync_metadata/dbms_version_test.clj
index 89e24ef3798..1b0de85b413 100644
--- a/test/metabase/sync/sync_metadata/dbms_version_test.clj
+++ b/test/metabase/sync/sync_metadata/dbms_version_test.clj
@@ -2,6 +2,7 @@
(:require
[clojure.test :refer :all]
[malli.error :as me]
+ [metabase.driver.util :as driver.u]
[metabase.sync.sync-metadata.dbms-version :as sync-dbms-ver]
[metabase.test :as mt]
[metabase.util :as u]
@@ -25,7 +26,7 @@
_ (t2/update! :model/Database (u/the-id db) {:dbms_version nil})
db (t2/select-one :model/Database :id (u/the-id db))
version-after-update (db-dbms-version db)
- _ (sync-dbms-ver/sync-dbms-version! db)]
+ _ (sync-dbms-ver/sync-dbms-version! (driver.u/lib-database (:id db)))]
(testing "On startup is the dbms-version specified?"
(is (nil? (check-dbms-version version-on-load))))
(testing "Check to make sure the test removed the timezone"
diff --git a/test/metabase/sync/sync_metadata/fields/our_metadata_test.clj b/test/metabase/sync/sync_metadata/fields/our_metadata_test.clj
index c0d875e3c44..144d5badde4 100644
--- a/test/metabase/sync/sync_metadata/fields/our_metadata_test.clj
+++ b/test/metabase/sync/sync_metadata/fields/our_metadata_test.clj
@@ -13,7 +13,7 @@
;; `our-metadata` should match up with what we have in the DB
(deftest does-metadata-match-test
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery}]
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(is (= #{{:name "id"
:database-type "SERIAL"
:base-type :type/Integer
diff --git a/test/metabase/sync/sync_metadata/fields/sync_instances_test.clj b/test/metabase/sync/sync_metadata/fields/sync_instances_test.clj
index a8c92154e28..744040bac8f 100644
--- a/test/metabase/sync/sync_metadata/fields/sync_instances_test.clj
+++ b/test/metabase/sync/sync_metadata/fields/sync_instances_test.clj
@@ -60,7 +60,7 @@
(testing "Make sure nested fields get resynced correctly if their parent field didnt' change"
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery}]
;; do the initial sync
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; delete our entry for the `transactions.toucan.details.age` field
(let [transactions-table-id (u/the-id (t2/select-one-pk :model/Table :db_id (u/the-id db), :name "transactions"))
toucan-field-id (u/the-id (t2/select-one-pk :model/Field :table_id transactions-table-id, :name "toucan"))
@@ -68,7 +68,7 @@
age-field-id (u/the-id (t2/select-one-pk :model/Field :table_id transactions-table-id, :name "age", :parent_id details-field-id))]
(t2/delete! :model/Field :id age-field-id)
;; now sync again.
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; field should be added back
(is (= #{"weight" "age"}
(t2/select-fn-set :name :model/Field :table_id transactions-table-id, :parent_id details-field-id, :active true)))))))
@@ -77,7 +77,7 @@
(testing "Syncing can reactivate a field"
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery}]
;; do the initial sync
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; delete our entry for the `transactions.toucan.details.age` field
(let [transactions-table-id (u/the-id (t2/select-one-pk :model/Table :db_id (u/the-id db), :name "transactions"))
toucan-field-id (u/the-id (t2/select-one-pk :model/Field :table_id transactions-table-id, :name "toucan"))
@@ -85,7 +85,7 @@
age-field-id (u/the-id (t2/select-one-pk :model/Field :table_id transactions-table-id, :name "age", :parent_id details-field-id))]
(t2/update! :model/Field age-field-id {:active false})
;; now sync again.
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; field should be reactivated
(is (t2/select-one-fn :active :model/Field :id age-field-id))))))
@@ -94,7 +94,7 @@
(mt/test-helpers-set-global-values!
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery}]
;; do the initial sync
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; delete our entry for the `transactions.toucan.details.age` field
(let [transactions-table-id (u/the-id (t2/select-one-pk :model/Table :db_id (u/the-id db), :name "transactions"))
toucan-field-id (u/the-id (t2/select-one-pk :model/Field :table_id transactions-table-id, :name "toucan"))
@@ -102,7 +102,7 @@
age-field-id (u/the-id (t2/select-one-pk :model/Field :table_id transactions-table-id, :name "age", :parent_id details-field-id))]
(t2/update! :model/Field details-field-id {:active false})
;; now sync again.
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; field should be reactivated
(is (t2/select-one-fn :active :model/Field :id age-field-id)))))))
@@ -110,7 +110,7 @@
(testing "Nested fields can be marked inactive"
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery}]
;; do the initial sync
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; Add an entry for a `transactions.toucan.details.gender` field
(let [transactions-table-id (u/the-id (t2/select-one-pk :model/Table :db_id (u/the-id db), :name "transactions"))
toucan-field-id (u/the-id (t2/select-one-pk :model/Field :table_id transactions-table-id, :name "toucan"))
@@ -123,7 +123,7 @@
:parent_id details-field-id
:active true)))]
;; now sync again.
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; field should become inactive
(is (false? (t2/select-one-fn :active :model/Field :id gender-field-id)))))))
@@ -131,7 +131,7 @@
(testing "When a nested field is marked inactive so are its children"
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery}]
;; do the initial sync
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; Add an entry for a `transactions.toucan.details.gender` field
(let [transactions-table-id (u/the-id (t2/select-one-pk :model/Table :db_id (u/the-id db), :name "transactions"))
toucan-field-id (u/the-id (t2/select-one-pk :model/Field :table_id transactions-table-id, :name "toucan"))
@@ -151,14 +151,14 @@
:parent_id food-likes-field-id
:active true))]
;; now sync again.
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
;; field should become inactive
(is (false? (t2/select-one-fn :active :model/Field :id blueberries-field-id)))))))
(defn run-cruft-test [patterns freqs]
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery
:settings {:auto-cruft-columns patterns}}]
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(let [tables (t2/select :model/Table :db_id (u/the-id db))
fields (mapcat (fn [{:keys [id]}] (t2/select :model/Field :table_id id)) tables)]
(is (= freqs
diff --git a/test/metabase/sync/sync_metadata/fields/sync_metadata_test.clj b/test/metabase/sync/sync_metadata/fields/sync_metadata_test.clj
index 70f447acb2e..1f6707a37c6 100644
--- a/test/metabase/sync/sync_metadata/fields/sync_metadata_test.clj
+++ b/test/metabase/sync/sync_metadata/fields/sync_metadata_test.clj
@@ -2,6 +2,8 @@
(:require
[clojure.test :refer :all]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.sync.core :as sync]
[metabase.sync.sync-metadata.fields.sync-metadata :as sync-metadata]
[metabase.test :as mt]
@@ -25,7 +27,7 @@
(swap! update-operations conj [(name model) id updates])
(count updates))]
(#'sync-metadata/update-field-metadata-if-needed!
- db
+ (driver.u/lib-database (:id db))
table
new-metadata-from-sync
metadata-in-application-db)
@@ -342,7 +344,7 @@
;; drop the column and create a new one with the same name
(sql-jdbc.execute/do-with-connection-with-options
:h2
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{}
(fn [conn]
(doseq [sql ["ALTER TABLE \"TABLE\" DROP COLUMN \"FIELD\";"
@@ -393,7 +395,7 @@
:base-type :type/Text}]
[["100"] ["200"] ["300"]]]]
(try
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [original-field (t2/select-one :model/Field (mt/id :my_table :text_column))]
(testing "sanity check: original column is text"
(is (=? {:base_type :type/Text
@@ -405,14 +407,14 @@
;; not translate to H2, so we use ALTER COLUMN to achieve the same "in-place update" effect.
(sql-jdbc.execute/do-with-connection-with-options
:h2
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{}
(fn [conn]
(doseq [sql ["DROP TABLE \"MY_TABLE\";"
"CREATE TABLE \"MY_TABLE\" (\"TEXT_COLUMN\" DECIMAL(38,2));"
"INSERT INTO \"MY_TABLE\"(text_column) VALUES(100.00),(200.00),(300.00);"]]
(next.jdbc/execute! conn [sql]))))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [new-field (t2/select-one :model/Field (mt/id :my_table :text_column))]
(testing "after sync, base_type and effective_type both reflect the new numeric column
and stale coercion/semantic type are cleared"
@@ -424,7 +426,7 @@
new-field))))
(sql-jdbc.execute/do-with-connection-with-options
:h2
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{}
(fn [conn]
(doseq [sql ["DROP TABLE \"MY_TABLE\";"]]
@@ -433,23 +435,23 @@
(t2/delete! :model/Field :table_id (:table_id original-field))
(sql-jdbc.execute/do-with-connection-with-options
:h2
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{}
(fn [conn]
(doseq [sql ["CREATE TABLE \"MY_TABLE\" (\"TEXT_COLUMN\" TEXT);"
"INSERT INTO \"MY_TABLE\"(text_column) VALUES('100'),('200'),('300');"]]
(next.jdbc/execute! conn [sql]))))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(sql-jdbc.execute/do-with-connection-with-options
:h2
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{}
(fn [conn]
(doseq [sql ["DROP TABLE \"MY_TABLE\";"
"CREATE TABLE \"MY_TABLE\" (\"TEXT_COLUMN\" DECIMAL(38,2));"
"INSERT INTO \"MY_TABLE\"(text_column) VALUES(100.00),(200.00),(300.00);"]]
(next.jdbc/execute! conn [sql]))))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [new-field (t2/select-one :model/Field :name "TEXT_COLUMN")]
(testing "after sync, base_type and effective_type both reflect the new numeric column
and stale coercion/semantic type are cleared"
@@ -538,7 +540,7 @@
(mt/user-http-request :crowberto :put 200 (format "field/%d" field-id) {:data_sensitivity "PII"})
(sql-jdbc.execute/do-with-connection-with-options
:h2
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{}
(fn [conn]
(doseq [sql ["ALTER TABLE \"TABLE\" DROP COLUMN \"FIELD\";"
diff --git a/test/metabase/sync/sync_metadata/fields_test.clj b/test/metabase/sync/sync_metadata/fields_test.clj
index de172fb7152..e6f214f1453 100644
--- a/test/metabase/sync/sync_metadata/fields_test.clj
+++ b/test/metabase/sync/sync_metadata/fields_test.clj
@@ -7,6 +7,7 @@
[medley.core :as m]
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.query-processor.test :as qp]
[metabase.sync.core :as sync]
[metabase.sync.fetch-metadata :as fetch-metadata]
@@ -65,13 +66,13 @@
;; first, create a new in-memory test DB and add some data to it
(with-test-db
;; now sync
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
;; ok, let's see what (f) gives us
(let [f-before (f (mt/db))]
;; ok cool! now delete one of those columns...
(jdbc/execute! one-off-dbs/*conn* [alter-sql])
;; ...and re-sync...
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
;; ...now let's see how (f) may have changed! Compare to original.
{:before-sync f-before
:after-sync (f (mt/db))})))
@@ -131,7 +132,7 @@
"(2, 'Steven Seagull'), "
"(3, 'Colin Fowl');")]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [tables (t2/select-pks-set :model/Table :db_id (mt/id))
get-field-to-update (fn []
(t2/select-one
@@ -147,7 +148,7 @@
;; ok cool! now delete one of those columns...
(jdbc/execute! one-off-dbs/*conn* ["ALTER TABLE \"birds\" DROP COLUMN \"example_name\";"])
;; ...and re-sync...
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
;; ...now let's see how the field may have changed! Compare to original.
(is (=? {:before-sync {:semantic_type :type/FK
:fk_target_field_id int?}
@@ -160,19 +161,19 @@
(testing "when a column changes base type, coercion_strategy is reset"
(mt/with-temp-copy-of-db
(let [db (mt/db)
- db-spec (sql-jdbc.conn/db->pooled-connection-spec db)]
+ db-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))]
(doseq [statement ["DROP TABLE IF EXISTS \"base_type_change_test\";"
"CREATE TABLE \"base_type_change_test\" (\"string_tbc_int_col\" VARCHAR);"
"INSERT INTO \"base_type_change_test\" (\"string_tbc_int_col\") VALUES ('1'), ('2'), ('3');"]]
(jdbc/execute! db-spec [statement]))
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(let [field (t2/select-one [:model/Field :id] :name "string_tbc_int_col")]
(mt/user-http-request :crowberto :put 200 (format "field/%d" (:id field)) {:coercion_strategy :Coercion/String->Integer})
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (=? {:effective_type :type/Integer :coercion_strategy :Coercion/String->Integer}
(t2/select-one :model/Field :id (:id field) {:from [(warehouse-schema-overlay/field-query)]})))
(jdbc/execute! db-spec ["ALTER TABLE \"base_type_change_test\" ALTER COLUMN \"string_tbc_int_col\" TYPE int USING \"string_tbc_int_col\"::integer;"])
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(testing "the base type change unsets the user's coercion, on the Field and for the user"
(is (=? {:coercion_strategy nil}
(t2/select-one :model/Field :id (:id field))))
@@ -275,18 +276,18 @@
[name-field-def {:field-name "continent_id", :base-type :type/Integer}]
[]]]
(let [db (mt/db)
- db-spec (sql-jdbc.conn/db->pooled-connection-spec db)
+ db-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
get-fk #(t2/select-one :model/Field (mt/id :country :continent_id))]
;; 1. add FK relationship in the database targeting continent_1
(jdbc/execute! db-spec "ALTER TABLE country ADD CONSTRAINT country_continent_id_fkey FOREIGN KEY (continent_id) REFERENCES continent_1(id);")
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(testing "initially country's continent_id is targeting continent_1"
(is (=? {:fk_target_field_id (mt/id :continent_1 :id)
:semantic_type :type/FK}
(get-fk))))
;; 2. drop the FK relationship in the database with SQL
(jdbc/execute! db-spec "ALTER TABLE country DROP CONSTRAINT country_continent_id_fkey;")
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
;; FIXME: The following test fails. The FK relationship is still there in the Metabase database (metabase#39687)
#_(testing "after dropping the FK relationship, country's continent_id is targeting nothing"
(is (=? {:fk_target_field_id nil
@@ -294,7 +295,7 @@
(get-fk))))
;; 3. add back the FK relationship but targeting continent_2
(jdbc/execute! db-spec "ALTER TABLE country ADD CONSTRAINT country_continent_id_fkey FOREIGN KEY (continent_id) REFERENCES continent_2(id);")
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(testing "initially country's continent_id is targeting continent_2"
(is (=? {:fk_target_field_id (mt/id :continent_2 :id)
:semantic_type :type/FK}
@@ -370,7 +371,7 @@
"('a', 'b'), "
"('c', 'd');")]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (let [sync-info (sync/sync-database! (mt/db))
+ (let [sync-info (sync/sync-database! (mt/id))
field-sync-info (->> sync-info
(m/find-first (comp #{"metadata"} :name))
:steps
@@ -392,7 +393,7 @@
"INSERT INTO \"birds\" (\"event\") VALUES ('a'), ('b')"
"INSERT INTO \"BIRDS\" (\"event\") VALUES ('c'), ('d')"]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (let [sync-info (sync/sync-database! (mt/db))
+ (let [sync-info (sync/sync-database! (mt/id))
field-sync-info (->> sync-info
(m/find-first (comp #{"metadata"} :name))
:steps
@@ -418,8 +419,8 @@
(run! sync-fields/sync-fields-for-table! tables)
(run! sync-fks/sync-fks-for-table! tables))
"for entire DB" (fn []
- (sync-fields/sync-fields! (mt/db))
- (sync-fks/sync-fks! (mt/db)))}]
+ (sync-fields/sync-fields! (lib.metadata/database (mt/metadata-provider)))
+ (sync-fks/sync-fks! (lib.metadata/database (mt/metadata-provider))))}]
(testing message
;; do this in a transaction so deleting all the Fields isn't permanent
(t2/with-transaction [_ t2.connection/*current-connectable* {:rollback-only true}]
@@ -462,7 +463,7 @@
(testing "Make sure a db's settings.auto-cruft-tables actually mark tables as crufty"
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery
:settings {:auto-cruft-columns ["l"]}}]
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(is (= [["details" :details-only]]
(->> (db->fields db)
(filter #(= :details-only (:visibility_type %)))
@@ -472,14 +473,14 @@
(testing "Make sure a db's settings.auto-cruft-tables actually mark tables as crufty"
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery
:settings {:auto-cruft-columns [".*"]}}]
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(is (= {:details-only 12}
(->> (db->fields db)
(mapv :visibility_type)
frequencies)))
;; remove cruft column directive:
(t2/update! :model/Database (u/the-id db) {:settings {:auto-cruft-columns []}})
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(is (= {:details-only 12}
(->> (db->fields db)
(mapv :visibility_type)
@@ -488,7 +489,7 @@
(deftest auto-cruft-sets-preview-display-false-test
(testing "When a field becomes crufty via auto-cruft settings, preview_display should be set to false (#10851)"
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery}]
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(let [details-field (t2/select-one :model/Field :name "details"
:table_id [:in (map :id (t2/select :model/Table :db_id (u/the-id db)))])]
(is (=? {:visibility_type :normal
@@ -496,7 +497,7 @@
details-field)
"before auto-cruft: field has default visibility and preview_display")
(t2/update! :model/Database (u/the-id db) {:settings {:auto-cruft-columns ["details"]}})
- (sync-metadata/sync-db-metadata! (t2/select-one :model/Database (u/the-id db)))
+ (sync-metadata/sync-db-metadata! (:id (t2/select-one :model/Database (u/the-id db))))
(is (=? {:visibility_type :details-only
:preview_display false}
(t2/select-one :model/Field :id (:id details-field)))
@@ -507,7 +508,7 @@
(mt/test-drivers (mt/normal-drivers)
(let [table (t2/instance :model/Table {:id 321 :name "tbl"})]
(is (not= ::thrown
- (try (sync-fields/sync-fields-for-table! (mt/db) table)
+ (try (sync-fields/sync-fields-for-table! (lib.metadata/database (mt/metadata-provider)) table)
(catch Throwable _ ::thrown))))))))
(deftest visibility-type-stays-normal-after-manual-change-test
@@ -529,7 +530,7 @@
(jdbc/execute! spec [statement]))
(mt/with-temp [:model/Database database {:engine :postgres, :details details}]
(mt/with-db database
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [table-id (t2/select-one-pk :model/Table :db_id (u/the-id database) :name "test_table")
field-after-first-sync (t2/select-one :model/Field :table_id table-id :name "something")]
(is (= :details-only (:visibility_type field-after-first-sync))
@@ -540,7 +541,7 @@
(is (= :normal (:visibility_type (t2/select-one :model/Field :id field-id
{:from [(warehouse-schema-overlay/field-query)]})))
"Manual change should set visibility_type to :normal"))
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [table-id (t2/select-one-pk :model/Table :db_id (u/the-id database) :name "test_table")
field-id (t2/select-one-pk :model/Field :table_id table-id :name "something")]
(is (= :normal (:visibility_type (t2/select-one :model/Field :id field-id
@@ -560,12 +561,12 @@
["Foo" "foo" {"x" true, "y" true}]]]
(testing (format "hiding %s" hidden)
(mt/with-temp [:model/Database database {:engine :postgres, :details details}]
- (sync-metadata/sync-db-metadata! database)
+ (sync-metadata/sync-db-metadata! (:id database))
(let [table-id #(t2/select-one-pk :model/Table :db_id (u/the-id database) :name %)
active-by-name #(t2/select-fn->fn :name :active :model/Field :table_id (table-id %))]
(is (= expected (active-by-name visible)))
(mt/user-http-request :crowberto :put 200 (format "table/%d" (table-id hidden)) {:visibility_type "hidden"})
- (sync-metadata/sync-db-metadata! database)
+ (sync-metadata/sync-db-metadata! (:id database))
(is (= expected (active-by-name visible)))))))))))
(deftest user-set-fks-are-preserved-by-sync-test
@@ -577,7 +578,7 @@
"(2, 'Steven Seagull'), "
"(3, 'Colin Fowl');")]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [tables (t2/select-pks-set :model/Table :db_id (mt/id))
birds-example-name-field (t2/select-one :model/Field :name "example_name" :table_id [:in tables])
flocks-example-bird-name-field (t2/select-one :model/Field :name "example_bird_name" :table_id [:in tables])]
@@ -589,7 +590,7 @@
{:semantic_type :type/FK
:fk_target_field_id (u/the-id birds-example-name-field)})
(testing "after sync, user-set FK is preserved"
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [field-after-sync (t2/select-one :model/Field :id (u/the-id flocks-example-bird-name-field) {:from [(warehouse-schema-overlay/field-query)]})]
(is (= :type/FK (:semantic_type field-after-sync)))
(is (= (u/the-id birds-example-name-field) (:fk_target_field_id field-after-sync)))))))))
@@ -598,7 +599,7 @@
(with-test-db
(let [ddl "CREATE TABLE t (a INTEGER DEFAULT 42, b INTEGER NULL)"]
(jdbc/execute! one-off-dbs/*conn* [ddl])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [table (t2/select-one :model/Table :db_id (mt/id) :name "T")
{a "A", b "B"} (u/index-by :name (t2/select :model/Field :table_id (:id table)))]
(is (= "42" (:database_default a)))
@@ -608,7 +609,7 @@
(with-test-db
(let [ddl "CREATE TABLE t (a INTEGER GENERATED ALWAYS AS 42, b INTEGER NULL)"]
(jdbc/execute! one-off-dbs/*conn* [ddl])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [table (t2/select-one :model/Table :db_id (mt/id) :name "T")
{a "A", b "B"} (u/index-by :name (t2/select :model/Field :table_id (:id table)))]
(is (true? (:database_is_generated a)))
@@ -618,7 +619,7 @@
(with-test-db
(let [ddl "CREATE TABLE t (a INTEGER NULL, b INTEGER NOT NULL)"]
(jdbc/execute! one-off-dbs/*conn* [ddl])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(let [table (t2/select-one :model/Table :db_id (mt/id) :name "T")
{a "A", b "B"} (u/index-by :name (t2/select :model/Field :table_id (:id table)))]
(is (true? (:database_is_nullable a)))
@@ -638,7 +639,7 @@
(is (not (t2/exists? :model/FieldUserSettings :field_id field-id))))
(mt/user-http-request :crowberto :put 200 (format "field/%d" field-id) {:data_sensitivity "PII"})
(testing "after a full sync, the classifier skips the user-overridden Field"
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (nil? (field)))
(is (= :PII (mirror))))
(testing "a bare update to the Field only changes the classifier's own value"
@@ -658,7 +659,7 @@
(mt/user-http-request :crowberto :put 200 (format "field/%d" (mt/id :app_users :email)) {:data_sensitivity "PUBLIC"})
(is (nil? (field :ssn)))
(mt/with-temporary-setting-values [data-sensitivity-scan-enabled true]
- (sync/sync-database! (mt/db)))
+ (sync/sync-database! (mt/id)))
(testing "the classifier still infers and writes its own category on the raw Field"
(is (= :PII (field :email))))
(testing "the user's PUBLIC override on email survives in FieldUserSettings"
@@ -676,18 +677,18 @@
(try
(let [db (mt/db)
- db-spec (sql-jdbc.conn/db->pooled-connection-spec db)
+ db-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
field-id (mt/id :readd_table :email)
field #(t2/select-one [:model/Field :active :data_sensitivity] :id field-id)
mirror #(t2/select-one-fn :data_sensitivity :model/FieldUserSettings :field_id field-id)]
(mt/user-http-request :crowberto :put 200 (format "field/%d" field-id) {:data_sensitivity "PHI"})
(jdbc/execute! db-spec ["ALTER TABLE \"READD_TABLE\" DROP COLUMN \"EMAIL\";"])
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(testing "the field is inactive while the column is gone; the mirror keeps the label"
(is (=? {:active false, :data_sensitivity nil} (field)))
(is (= :PHI (mirror))))
(jdbc/execute! db-spec ["ALTER TABLE \"READD_TABLE\" ADD COLUMN \"EMAIL\" VARCHAR;"])
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(testing "the same Field row is reactivated and the mirror still keeps its label"
(is (=? {:active true, :data_sensitivity nil} (field)))
(is (= :PHI (mirror)))))
@@ -701,7 +702,7 @@
(mt/with-dynamic-fn-redefs [fetch-metadata/include-nested-fields-for-table
(fn [& _] (swap! attempted inc) (throw e))]
(mt/with-log-messages-for-level [messages [metabase.sync.sync-metadata.fields :warn]]
- (let [threw? (try (sync-fields/sync-fields! (mt/db)) false
+ (let [threw? (try (sync-fields/sync-fields! (lib.metadata/database (mt/metadata-provider))) false
(catch Throwable _ true))]
{:attempted @attempted
:threw? threw?
@@ -723,7 +724,7 @@
(mt/with-dynamic-fn-redefs [fetch-metadata/include-nested-fields-for-table
(fn [& _] (throw (ex-info "boom" {})))]
(mt/with-log-messages-for-level [messages [metabase.sync.sync-metadata.fields :warn]]
- (sync-fields/sync-fields! (mt/db))
+ (sync-fields/sync-fields! (lib.metadata/database (mt/metadata-provider)))
(let [msgs (map :message (messages))]
(is (seq msgs))
(is (not-any? #(re-find #"Table Table" %) msgs)))))))
diff --git a/test/metabase/sync/sync_metadata/indexes_test.clj b/test/metabase/sync/sync_metadata/indexes_test.clj
index 86c5c9f918e..2a4f3b32554 100644
--- a/test/metabase/sync/sync_metadata/indexes_test.clj
+++ b/test/metabase/sync/sync_metadata/indexes_test.clj
@@ -5,6 +5,8 @@
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql.util :as sql.u]
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.sync.core :as sync]
[metabase.sync.sync-metadata.indexes :as sync.indexes]
[metabase.test :as mt]
@@ -29,9 +31,9 @@
{:field-name "second" :indexed? false :base-type :type/Integer}]
[[1 2]]]]]
(mt/with-temp-test-data ds
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(sql.tx/create-index-sql driver/*driver* "table" ["first" "second"]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (true? (t2/select-one-fn :database_indexed :model/Field (mt/id :table :first))))
(is (not= true (t2/select-one-fn :database_indexed :model/Field (mt/id :table :second))))))))
@@ -41,13 +43,13 @@
(testing "do nothing if the driver doesn't support indexing"
(mt/with-temp [:model/Database db {:engine ::not-support-index-test}
:model/Table table {:db_id (:id db)}]
- (is (= @#'sync.indexes/empty-stats (sync.indexes/maybe-sync-indexes! db)))
- (is (= @#'sync.indexes/empty-stats (sync.indexes/maybe-sync-indexes-for-table! db table))))))
+ (is (= @#'sync.indexes/empty-stats (sync.indexes/maybe-sync-indexes! (driver.u/lib-database (:id db)))))
+ (is (= @#'sync.indexes/empty-stats (sync.indexes/maybe-sync-indexes-for-table! (driver.u/lib-database (:id db)) table))))))
(deftest all-indexes->fields-ids-many-indexes-test
(testing "no exception is thrown when there are very many indexes"
(mt/test-drivers (mt/normal-drivers-with-feature :describe-indexes :index-info)
- (let [indexes (into [] (driver/describe-indexes driver/*driver* (mt/db)))
+ (let [indexes (into [] (driver/describe-indexes driver/*driver* (lib.metadata/database (mt/metadata-provider))))
many-indexes (into indexes (repeat 100000 {:table-schema "public",
:table-name "fake_table",
:field-name "id"}))
@@ -82,9 +84,9 @@
(testing "All indexed fields picked up by sync (first, second, third)"
(doseq [field ["first" "second" "third"]
:let [sql (sql.tx/create-index-sql driver/*driver* "first_table" [field])]]
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db)) sql))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider))) sql))
(binding [sync.indexes/*update-partition-size* 2]
- (#'sync.indexes/sync-all-indexes! (mt/db)))
+ (#'sync.indexes/sync-all-indexes! (lib.metadata/database (mt/metadata-provider))))
(is (every? :database_indexed
(t2/select :model/Field
{:where [:in :table_id (t2/select-fn-vec :id :model/Table :db_id (mt/id))]}))))
@@ -92,9 +94,9 @@
(doseq [field ["first" "second" "third"]
:let [sql (format "DROP INDEX %s;" (sql.u/quote-name
driver/*driver* :index (str "idx_first_table_" field)))]]
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db)) sql))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider))) sql))
(binding [sync.indexes/*update-partition-size* 2]
- (#'sync.indexes/sync-all-indexes! (mt/db)))
+ (#'sync.indexes/sync-all-indexes! (lib.metadata/database (mt/metadata-provider))))
(is (every? (complement :database_indexed)
(t2/select :model/Field
{:where [:and
diff --git a/test/metabase/sync/sync_metadata/metabase_metadata_test.clj b/test/metabase/sync/sync_metadata/metabase_metadata_test.clj
index 7770fdead56..ab9b234c6ec 100644
--- a/test/metabase/sync/sync_metadata/metabase_metadata_test.clj
+++ b/test/metabase/sync/sync_metadata/metabase_metadata_test.clj
@@ -2,6 +2,7 @@
"Tests for the logic that syncs the `_metabase_metadata` Table."
(:require
[clojure.test :refer :all]
+ [metabase.driver.util :as driver.u]
[metabase.sync.sync-metadata.metabase-metadata :as metabase-metadata]
[metabase.test :as mt]
[metabase.test.mock.moviedb :as moviedb]
@@ -34,7 +35,7 @@
:fields [{:name "filming", :description nil}]}
(get-table-and-fields-descriptions table)))
(is (nil? (:description (t2/select-one :model/Database :id (u/the-id db))))))
- (metabase-metadata/sync-metabase-metadata! db)
+ (metabase-metadata/sync-metabase-metadata! (driver.u/lib-database (:id db)))
(testing "after"
(is (= {:name "movies"
:description "A cinematic adventure."
@@ -49,7 +50,7 @@
:model/Table table {:db_id (u/the-id db), :name "movies", :active true}
:model/Field field {:table_id (u/the-id table), :name "filming", :base_type :type/Boolean}]
(let [set-property! (fn [keypath value]
- (#'metabase-metadata/set-property! db (#'metabase-metadata/parse-keypath keypath) value))]
+ (#'metabase-metadata/set-property! (driver.u/lib-database (:id db)) (#'metabase-metadata/parse-keypath keypath) value))]
(testing "descriptive properties are set"
(is (true? (set-property! "caveats" "db caveat")))
(is (true? (set-property! "movies.caveats" "table caveat")))
@@ -82,5 +83,5 @@
(is (=? [{:level :warn
:message #"Ignoring 2 _metabase_metadata row\(s\).*: movies.filming.description, movies.description"}]
(mt/with-log-messages-for-level [messages [metabase.sync.sync-metadata.metabase-metadata :warn]]
- (metabase-metadata/sync-metabase-metadata! db)
+ (metabase-metadata/sync-metabase-metadata! (driver.u/lib-database (:id db)))
(messages)))))))
diff --git a/test/metabase/sync/sync_metadata/sync_database_type_test.clj b/test/metabase/sync/sync_metadata/sync_database_type_test.clj
index 0b6f4c17569..03ab2490597 100644
--- a/test/metabase/sync/sync_metadata/sync_database_type_test.clj
+++ b/test/metabase/sync/sync_metadata/sync_database_type_test.clj
@@ -11,7 +11,7 @@
(deftest update-database-type-test
(testing "make sure that if a driver reports back a different database-type the Field gets updated accordingly"
(mt/with-temp [:model/Database db (select-keys (mt/db) [:details :engine])]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(let [venues-table (t2/select-one :model/Table :db_id (u/the-id db), :display_name "Venues")]
;; ok, now give all the Fields `?` as their `database_type`. (This is what the DB migration does for existing
;; Fields)
diff --git a/test/metabase/sync/sync_metadata/sync_timezone_test.clj b/test/metabase/sync/sync_metadata/sync_timezone_test.clj
index f8b920140f5..78a64c78183 100644
--- a/test/metabase/sync/sync_metadata/sync_timezone_test.clj
+++ b/test/metabase/sync/sync_metadata/sync_timezone_test.clj
@@ -5,6 +5,7 @@
[java-time.api :as t]
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.driver.util :as driver.u]
[metabase.sync.sync-metadata.sync-timezone :as sync-tz]
[metabase.sync.util-test :as sync.util-test]
[metabase.test :as mt]
@@ -53,7 +54,7 @@
tz-after-update (db-timezone db)
;; It looks like we can get some stale timezone information depending on which thread is used for querying the
;; database in sync. Clearing the connection pool to ensure we get the most updated TZ data
- _ (driver/notify-database-updated driver/*driver* db)
+ _ (driver/notify-database-updated driver/*driver* (driver.u/lib-database (:id db)))
{:keys [step-info task-history]} (sync.util-test/sync-database! "sync-timezone" (assoc db :timezone nil))]
(testing "only step keys"
(is (= {:timezone-id "UTC"}
diff --git a/test/metabase/sync/sync_metadata/tables_test.clj b/test/metabase/sync/sync_metadata/tables_test.clj
index 3fdb3548521..25850d456f0 100644
--- a/test/metabase/sync/sync_metadata/tables_test.clj
+++ b/test/metabase/sync/sync_metadata/tables_test.clj
@@ -8,6 +8,8 @@
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.sync.core :as sync]
[metabase.sync.sync-metadata :as sync-metadata]
[metabase.sync.sync-metadata.tables :as sync-tables]
@@ -73,12 +75,12 @@
(mt/dataset dbdef
(t2/update! :model/Table (mt/id :user) {:active false})
;; table description is changed
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
[(sql.tx/standalone-table-comment-sql
(:engine (mt/db))
dbdef
(tx/map->TableDefinition {:table-name "user" :table-comment "added comment"}))])
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(is (=? {:active true
:description "added comment"}
(t2/select-one :model/Table (mt/id :user))))))))
@@ -90,11 +92,11 @@
;; row count is estimated so we VACUUM so the statistic table is updated before syncing
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
nil
(fn [conn]
(next.jdbc/execute! conn ["VACUUM;"])))
- (sync/sync-database! (mt/db) {:scan :schema})
+ (sync/sync-database! (mt/id) {:scan :schema})
(is (= 100
(t2/select-one-fn :estimated_row_count :model/Table (mt/id :venues))))))))
@@ -109,7 +111,7 @@
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery
:settings {:auto-cruft-tables [".*"]}}]
(run-twice
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(is (= #{:cruft}
(t2/select-fn-set :visibility_type
:model/Table
@@ -121,14 +123,14 @@
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery
:settings {:auto-cruft-tables [".*"]}}]
(let [->tables-info #(t2/select-fn-set (juxt :name :visibility_type) :model/Table :db_id (u/the-id db))]
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(testing "Observe: the tables are marked as crufty"
(is (= #{["employees" :cruft] ["transactions" :cruft]} (->tables-info))))
;; make employees visible:
(t2/update! :model/Table :db_id (u/the-id db) :name "employees" {:visibility_type nil})
(testing "Observe: employees is visible, but transactions is still crufty"
(is (= #{["employees" nil] ["transactions" :cruft]} (->tables-info))))
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(testing "tables are unchanged after sync"
(is (= #{["employees" nil] ["transactions" :cruft]} (->tables-info))))))))
@@ -136,9 +138,9 @@
(testing (str "Make sure a db's settings.auto-cruft-tables do not unhide " original-vis-type " tables")
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery
:settings {:auto-cruft-tables []}}]
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(t2/update! :model/Table :db_id (u/the-id db) {:visibility_type original-vis-type})
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(is (= #{["employees" original-vis-type]
["transactions" original-vis-type]}
(t2/select-fn-set (juxt :name :visibility_type) :model/Table :db_id (u/the-id db)))))))
@@ -152,7 +154,7 @@
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery
:settings {:auto-cruft-tables ["employees"]}}]
(run-twice
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(is (= #{["employees" :cruft]
["transactions" nil]}
(t2/select-fn-set (juxt :name :visibility_type) :model/Table :db_id (u/the-id db))))))))
@@ -162,7 +164,7 @@
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery
:settings {:auto-cruft-tables ["l"]}}]
(run-twice
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(is (= #{["employees" :cruft]
["transactions" nil]}
(t2/select-fn-set (juxt :name :visibility_type) :model/Table :db_id (u/the-id db))))))))
@@ -172,7 +174,7 @@
(mt/with-temp [:model/Database db {:engine ::toucanery/toucanery
:settings {:auto-cruft-tables ["l" "y"]}}]
(run-twice
- (sync-metadata/sync-db-metadata! db)
+ (sync-metadata/sync-db-metadata! (:id db))
(is (= #{["employees" :cruft]
["transactions" nil]}
(t2/select-fn-set (juxt :name :visibility_type) :model/Table :db_id (u/the-id db))))))))
@@ -191,7 +193,7 @@
:model/Table table-3 {:name "active_table"
:db_id (u/the-id db)
:active true}]
- (#'sync-tables/archive-tables! db)
+ (#'sync-tables/archive-tables! (driver.u/lib-database (:id db)))
(testing "Old deactivated table is archived with suffix"
(let [archived-table (t2/select-one :model/Table (:id table-1))]
(is (some? (:archived_at archived-table)))
@@ -219,7 +221,7 @@
:active false
:transform_target false
:deactivated_at (t/minus (t/offset-date-time) (t/days 30))}]
- (#'sync-tables/archive-tables! db)
+ (#'sync-tables/archive-tables! (driver.u/lib-database (:id db)))
(testing "Transform target table is not archived or renamed"
(let [table (t2/select-one :model/Table (:id provisional))]
(is (nil? (:archived_at table)))
@@ -239,7 +241,7 @@
:archived_at (t/minus (t/offset-date-time) (t/days 5))}]
(let [original-name (:name table)
original-archived-at (:archived_at table)]
- (#'sync-tables/archive-tables! db)
+ (#'sync-tables/archive-tables! (driver.u/lib-database (:id db)))
(let [updated-table (t2/select-one :model/Table (:id table))]
(is (= original-name (:name updated-table)))
(is (= original-archived-at (:archived_at updated-table))))))))
@@ -272,7 +274,7 @@
:db_id (u/the-id db)
:active false
:deactivated_at (t/minus (t/offset-date-time) (t/days 20))}]
- (#'sync-tables/archive-tables! db)
+ (#'sync-tables/archive-tables! (driver.u/lib-database (:id db)))
(testing "the original table was archived and renamed"
(let [archived-table (t2/select-one :model/Table (:id original-table))]
(is (some? (:archived_at archived-table)))
@@ -301,11 +303,11 @@
(#'sync-tables/update-table-metadata-if-needed!
{:name "computed_table" :schema nil :is_writable true}
(t2/select-one select-cols (:id computed-table))
- db)
+ (driver.u/lib-database (:id db)))
(#'sync-tables/update-table-metadata-if-needed!
{:name "normal_table" :schema nil :is_writable true}
(t2/select-one select-cols (:id normal-table))
- db))
+ (driver.u/lib-database (:id db))))
(testing "computed table should remain non-writable"
(is (false? (t2/select-one-fn :is_writable :model/Table (:id computed-table)))))
(testing "normal table should be updated to writable"
@@ -338,14 +340,14 @@
(#'sync-tables/update-table-metadata-if-needed!
matching-metadata
(t2/select-one select-cols (:id tbl))
- db)
+ (driver.u/lib-database (:id db)))
(is (= initial-updated-at
(t2/select-one-fn :updated_at :model/Table (:id tbl)))))
(testing "a real change still bumps updated_at"
(#'sync-tables/update-table-metadata-if-needed!
(assoc matching-metadata :estimated_row_count 999)
(t2/select-one select-cols (:id tbl))
- db)
+ (driver.u/lib-database (:id db)))
(is (not= initial-updated-at
(t2/select-one-fn :updated_at :model/Table (:id tbl))))
(is (= 999
@@ -358,7 +360,7 @@
{:schema "public" :name "apple"}
{:schema "public" :name "mango"}}]
(mt/with-temp [:model/Database db {}]
- (#'sync-tables/create-tables! db metadatas)
+ (#'sync-tables/create-tables! (driver.u/lib-database (:id db)) metadatas {:sample? false})
(is (= ["beta" "apple" "mango" "zebra"]
(map :name (t2/select [:model/Table :name]
:db_id (u/the-id db)
@@ -371,17 +373,17 @@
(let [sample-table-metadata {:name "sample_table"}
normal-table-metadata {:name "normal_table"}]
(testing "creating a table in a sample database"
- (let [created-table (sync-tables/create-table! sample-db sample-table-metadata)]
+ (let [created-table (sync-tables/create-table! (driver.u/lib-database (:id sample-db)) sample-table-metadata)]
(is (= :ingested (:data_authority created-table)))))
(testing "creating a table in a normal database"
- (let [created-table (sync-tables/create-table! normal-db normal-table-metadata)]
+ (let [created-table (sync-tables/create-table! (driver.u/lib-database (:id normal-db)) normal-table-metadata)]
(is (= :unconfigured (:data_authority created-table)))))
(testing "reactivating a table in a sample database"
(mt/with-temp [:model/Table existing-table {:db_id (:id sample-db)
:name "existing_sample_table"
:active false
:data_authority :computed}]
- (sync-tables/create-or-reactivate-table! sample-db {:name "existing_sample_table"})
+ (sync-tables/create-or-reactivate-table! (driver.u/lib-database (:id sample-db)) {:name "existing_sample_table"})
(let [updated-table (t2/select-one :model/Table (:id existing-table))]
(is (= :ingested (:data_authority updated-table)))
(is (:active updated-table)))))))))
@@ -389,7 +391,7 @@
(deftest remove-tables-with-too-long-names-test
(testing "Tables whose name is too long to store in the app DB are dropped, so they don't abort the creation pass"
(let [remove-too-long @#'sync-tables/remove-tables-with-too-long-names
- database (t2/instance :model/Database {:id 1, :name "db", :engine :h2})
+ database {:lib/type :metadata/database, :id 1, :name "db", :engine :h2}
tbl (fn [nm] {:name nm, :schema "public"})
ok (tbl "short_name")
too-long (tbl (apply str (repeat 300 "a")))]
diff --git a/test/metabase/sync/sync_metadata_test.clj b/test/metabase/sync/sync_metadata_test.clj
index aa6984ac363..11cac76a1dd 100644
--- a/test/metabase/sync/sync_metadata_test.clj
+++ b/test/metabase/sync/sync_metadata_test.clj
@@ -1,6 +1,7 @@
(ns metabase.sync.sync-metadata-test
(:require
[clojure.test :refer :all]
+ [metabase.driver.util :as driver.u]
[metabase.sync.fetch-metadata :as fetch-metadata]
[metabase.sync.sync-metadata :as sync-metadata]
[metabase.sync.sync-metadata.fields :as sync-fields]
@@ -44,7 +45,7 @@
(mt/with-temp [:model/Database db {:initial_sync_status "incomplete"}]
(mt/with-dynamic-fn-redefs [fetch-metadata/db-metadata (fn [_] (throw (ex-info "boom" {})))]
(is (thrown? Throwable
- (#'sync-metadata/sync-db-metadata!* db))))
+ (#'sync-metadata/sync-db-metadata!* (driver.u/lib-database (:id db))))))
(is (= "aborted"
(t2/select-one-fn :initial_sync_status :model/Database (:id db))))
(testing "and the message of the error is recorded as the cause (GHY-3856)"
@@ -62,7 +63,7 @@
(throw (doto (java.sql.SQLSyntaxErrorException.
"Unknown column 'generation_expression' in 'field list'")
(.setStackTrace (into-array StackTraceElement [])))))]
- (#'sync-metadata/sync-db-metadata!* (t2/select-one :model/Database :id (mt/id))))
+ (#'sync-metadata/sync-db-metadata!* (driver.u/lib-database (mt/id))))
(is (= "aborted"
(t2/select-one-fn :initial_sync_status :model/Database (mt/id))))
(testing "and the step's own error message is recorded as the cause, not a schema-validation error (GHY-3856)"
@@ -74,7 +75,7 @@
(mt/with-temp-copy-of-db
(t2/update! :model/Database (mt/id) {:initial_sync_status "aborted"
:initial_sync_error "boom"})
- (#'sync-metadata/sync-db-metadata!* (t2/select-one :model/Database :id (mt/id)))
+ (#'sync-metadata/sync-db-metadata!* (driver.u/lib-database (mt/id)))
(is (= {:initial_sync_status "complete", :initial_sync_error nil}
(t2/select-one [:model/Database :initial_sync_status :initial_sync_error] :id (mt/id)))))))
@@ -86,6 +87,6 @@
"Connections could not be acquired from the underlying database!"
(com.mchange.v2.resourcepool.CannotAcquireResourceException. "x"))))]
(mt/with-log-messages-for-level [messages [metabase.sync.sync-metadata :warn]]
- (is (thrown? Throwable (sync-metadata/sync-db-metadata! (mt/db))))
+ (is (thrown? Throwable (sync-metadata/sync-db-metadata! (mt/id))))
(is (=? [{:level :warn, :message #"Could not fetch metadata for .*Connections could not be acquired.*"}]
(messages)))))))
diff --git a/test/metabase/sync/sync_test.clj b/test/metabase/sync/sync_test.clj
index c1f109f1d9f..507752f3957 100644
--- a/test/metabase/sync/sync_test.clj
+++ b/test/metabase/sync/sync_test.clj
@@ -251,7 +251,7 @@
(binding [*supports-schemas?* supports-schemas?
sync-util/*log-exceptions-and-continue?* false]
(mt/with-temp [:model/Database db {:engine ::sync-test}]
- (let [results (sync/sync-database! db)]
+ (let [results (sync/sync-database! (:id db))]
(testing "Returns results from sync-database step"
(is (= ["metadata" "analyze"]
(map :name results)))))
@@ -290,7 +290,7 @@
(deftest scanning-test
(mt/with-temp [:model/Database db {:engine ::sync-test}]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(let [query-field-and-values (fn []
(let [table-id (t2/select-one-pk :model/Table :db_id (u/the-id db) :name "movie")
title-field (t2/select-one :model/Field :table_id table-id :name "title")
@@ -325,13 +325,13 @@
(testing "New values come in after sync"
(binding [*execute-response* (fn [_query respond] (respond {:cols [{:name "field"}]}
(partition-all 1 ["d" "e" "f"])))]
- (sync/sync-database! (t2/select-one :model/Database (u/the-id db))))
+ (sync/sync-database! (:id (t2/select-one :model/Database (u/the-id db)))))
(is (=? {:f {:name "title" :has_field_values :auto-list}
:fv {:values ["d" "e" "f"] :has_more_values false}}
(query-field-and-values))))
(testing "After setting to search it should stay search and sync removes field-values"
(t2/update! :model/Field (:id field) {:has_field_values "search"})
- (sync/sync-database! (t2/select-one :model/Database (u/the-id db)))
+ (sync/sync-database! (:id (t2/select-one :model/Database (u/the-id db))))
(get-or-create-vals ["x" "y" "z"])
(is (=? {:f {:name "title" :has_field_values :search}
:fv nil}
@@ -348,7 +348,7 @@
(testing "Errors in sync-database! should be caught and handled correctly (#45848)"
(mt/with-temp [:model/Database db {:engine ::sync-database-error-test}]
(binding [sync-util/*log-exceptions-and-continue?* true]
- (let [results (sync/sync-database! db)]
+ (let [results (sync/sync-database! (:id db))]
(testing "Skips the metadata step"
(is (= ["analyze"]
(map :name results)))))))))
@@ -360,12 +360,12 @@
(mt/with-temp [:model/Database db {:engine ::sync-test, :initial_sync_status "incomplete"}]
(testing "the initial sync skips the field-values phase"
(is (= ["metadata" "analyze"]
- (map :name (sync/sync-database! db)))))
+ (map :name (sync/sync-database! (:id db))))))
(testing "a later sync of the Database, with its initial sync complete, runs it"
(let [synced-db (t2/select-one :model/Database (u/the-id db))]
(is (= "complete" (:initial_sync_status synced-db)))
(is (= ["metadata" "analyze" "field-values"]
- (map :name (sync/sync-database! synced-db))))))))))
+ (map :name (sync/sync-database! (:id synced-db)))))))))))
;; !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
;; !! !!
diff --git a/test/metabase/sync/util_test.clj b/test/metabase/sync/util_test.clj
index 492fcaadbe3..f0961e3e8cb 100644
--- a/test/metabase/sync/util_test.clj
+++ b/test/metabase/sync/util_test.clj
@@ -6,8 +6,8 @@
[clojure.test :refer :all]
[java-time.api :as t]
[metabase.driver :as driver]
+ [metabase.driver.util :as driver.u]
[metabase.events.core :as events]
- [metabase.models.interface :as mi]
[metabase.sync.core :as sync]
[metabase.sync.sync-metadata :as sync-metadata]
[metabase.sync.sync-metadata.fields :as sync-fields]
@@ -49,15 +49,15 @@
(mt/with-temp [:model/Database db {:engine ::concurrent-sync-test}]
(reset! calls-to-describe-database 0)
;; start a sync processes in the background. It should take 1000 ms to finish
- (let [f1 (future (sync/sync-database! db))
+ (let [f1 (future (sync/sync-database! (:id db)))
f2 (do
;; wait 200 ms to make sure everything is going
(Thread/sleep 200)
;; Start another in the background. Nothing should happen here because the first is already running
- (future (sync/sync-database! db)))]
+ (future (sync/sync-database! (:id db))))]
;; Start another in the foreground. Again, nothing should happen here because the original should still be
;; running
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
;; make sure both of the futures have finished
(deref f1)
(deref f2)
@@ -88,7 +88,7 @@
`step`. This function is useful for validating that each step's metadata correctly reflects the changes that were
made via a test scenario."
[step db]
- (let [{:keys [operation-results task-history-ids]} (call-with-operation-info! #(sync/sync-database! db))]
+ (let [{:keys [operation-results task-history-ids]} (call-with-operation-info! #(sync/sync-database! (:id db)))]
{:step-info (-> (into {} (mapcat :steps operation-results))
(get step))
:task-history (when (seq task-history-ids)
@@ -118,7 +118,7 @@
step-2-name (mt/random-name)
sync-steps [(sync-util/create-sync-step step-1-name (fn [_] (Thread/sleep 10) {:total-tables 1}))
(sync-util/create-sync-step step-2-name (fn [_] (Thread/sleep 10)))]
- mock-db (mi/instance :model/Database {:name "test", :id 1, :engine :h2})
+ mock-db {:lib/type :metadata/database, :name "test", :id 1, :engine :h2}
[results] (:operation-results
(call-with-operation-info! #(sync-util/run-sync-operation process-name mock-db sync-steps)))]
(testing "valid operation metadata?"
@@ -144,7 +144,7 @@
(let [process-name (mt/random-name)
step-name-1 (mt/random-name)
step-name-2 (mt/random-name)
- mock-db (mi/instance :model/Database {:name "test", :id 1, :engine :h2})
+ mock-db {:lib/type :metadata/database, :name "test", :id 1, :engine :h2}
sync-steps [(sync-util/create-sync-step step-name-1
(fn [_]
(throw (ex-info "Sorry" {}))))
@@ -186,7 +186,7 @@
" contains the important parts and it doesn't throw an exception")
(let [step-log-text (mt/random-name)
results (#'sync-util/make-log-sync-summary-str operation
- (mi/instance :model/Database {:name db-name})
+ {:lib/type :metadata/database, :id 1, :name db-name}
(create-test-sync-summary step-name
(constantly step-log-text)))]
(testing "has-operation?"
@@ -210,7 +210,7 @@
(testing (str "The `log-summary-fn` part of step info is optional as not all steps have it. Validate that we"
" properly handle that case")
(let [results (#'sync-util/make-log-sync-summary-str operation
- (mi/instance :model/Database {:name db-name})
+ {:lib/type :metadata/database, :id 1, :name db-name}
(create-test-sync-summary step-name nil))]
(testing "has-operation?"
(is (true?
@@ -238,10 +238,10 @@
"outer"
(java.net.ConnectException.
"inner, this one triggers the failure"))
- actual (sync-util/sync-operation ::sync-error-handling (mt/db) "sync error handling test"
+ actual (sync-util/sync-operation ::sync-error-handling (driver.u/lib-database (mt/id)) "sync error handling test"
(sync-util/run-sync-operation
"sync"
- (mt/db)
+ (driver.u/lib-database (mt/id))
[(sync-util/create-sync-step "failure-step"
(fn [_]
(throw expected)))
@@ -266,10 +266,10 @@
(java.lang.IllegalArgumentException.
"third level, does not trigger")))]]
(testing "Other errors will not cause sync to stop"
- (let [actual (sync-util/sync-operation ::sync-error-handling (mt/db) "sync error handling test"
+ (let [actual (sync-util/sync-operation ::sync-error-handling (driver.u/lib-database (mt/id)) "sync error handling test"
(sync-util/run-sync-operation
"sync"
- (mt/db)
+ (driver.u/lib-database (mt/id))
[(sync-util/create-sync-step "failure-step"
(fn [_]
(throw ex)))
@@ -292,19 +292,19 @@
(testing "If `initial-sync-status` on a DB is `incomplete`, it is marked as `complete` when sync-metadata has finished"
(let [_ (t2/update! :model/Database (mt/id) {:initial_sync_status "incomplete"})
db (t2/select-one :model/Database :id (mt/id))]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (= "complete" (t2/select-one-fn :initial_sync_status :model/Database :id (:id db))))))
(testing "If `initial-sync-status` on a DB is `complete`, it remains `complete` when sync is run again"
(let [_ (t2/update! :model/Database (mt/id) {:initial_sync_status "complete"})
db (t2/select-one :model/Database :id (mt/id))]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (= "complete" (t2/select-one-fn :initial_sync_status :model/Database :id (:id db))))))
(testing "If `initial-sync-status` on a table is `incomplete`, it is marked as `complete` after the sync-fks step
has finished"
(let [table-id (t2/select-one-fn :id :model/Table :db_id (mt/id) :active true)
_ (t2/update! :model/Table table-id {:initial_sync_status "incomplete"})
_table (t2/select-one :model/Table :id table-id)]
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (= "complete" (t2/select-one-fn :initial_sync_status :model/Table :id table-id)))))
(testing "Database and table syncs are marked as complete even if the initial scan is :schema only"
(let [_ (t2/update! :model/Database (mt/id) {:initial_sync_status "incomplete"})
@@ -312,7 +312,7 @@
table-id (t2/select-one-fn :id :model/Table :db_id (mt/id) :active true)
_ (t2/update! :model/Table table-id {:initial_sync_status "incomplete"})
_table (t2/select-one :model/Table :id table-id)]
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(is (= "complete" (t2/select-one-fn :initial_sync_status :model/Database :id (:id db))))
(is (= "complete" (t2/select-one-fn :initial_sync_status :model/Table :id table-id)))))
(testing "If a non-recoverable error occurs during sync, `initial-sync-status` on the database is set to `aborted`"
@@ -322,7 +322,7 @@
[(sync-util/create-sync-step
"fake-step"
(fn [_] (throw (java.net.ConnectException. "Connection refused"))))])]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (= "aborted" (t2/select-one-fn :initial_sync_status :model/Database :id (:id db))))
(testing "and the step's error message is recorded as the cause (GHY-3856)"
(is (= "Connection refused" (t2/select-one-fn :initial_sync_error :model/Database :id (:id db))))))))
@@ -331,7 +331,7 @@
(let [_ (t2/update! :model/Database (mt/id) {:initial_sync_status "complete"
:initial_sync_error nil})
db (t2/select-one :model/Database :id (mt/id))]
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(is (= "complete" (t2/select-one-fn :initial_sync_status :model/Database :id (:id db))))))))
(deftest initial-sync-status-table-only-test
@@ -351,7 +351,7 @@
(a/>!! syncing-chan ::syncing)
(sync-fields! database))]
(future
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(a/>!! completed-chan ::sync-completed))
(a/<!! syncing-chan)
(testing "for existing tables initial_sync_status is complete while sync is running"
@@ -371,7 +371,7 @@
(mt/with-temp [:model/Database db {:engine :h2}]
(let [initial (mt/metric-value system :metabase-sync/failures {:driver "h2"})]
(sync-util/do-sync-operation
- :sync db "test sync failure"
+ :sync (driver.u/lib-database (:id db)) "test sync failure"
(fn [] (throw (Exception. "sync boom"))))
(is (< initial (mt/metric-value system :metabase-sync/failures {:driver "h2"}))))))))
@@ -380,7 +380,7 @@
;;; +----------------------------------------------------------------------------------------------------------------+
(defn- in-sync-tables? [table-id]
- (contains? (into #{} (map :id) (sync-util/reducible-sync-tables (mt/id))) table-id))
+ (contains? (into #{} (map :id) (sync-util/reducible-sync-tables (driver.u/lib-database (mt/id)))) table-id))
(defn- set-visibility-type-via-api! [table-id visibility-type]
(mt/user-http-request :crowberto :put 200 (format "table/%d" table-id) {:visibility_type visibility-type}))
diff --git a/test/metabase/task_history/models/task_run_test.clj b/test/metabase/task_history/models/task_run_test.clj
index 328466dad5b..ccea363ee1f 100644
--- a/test/metabase/task_history/models/task_run_test.clj
+++ b/test/metabase/task_history/models/task_run_test.clj
@@ -3,7 +3,6 @@
(:require
[clojure.test :refer :all]
[metabase.channel.core :as channel]
- [metabase.models.interface :as mi]
[metabase.notification.send :as notification.send]
[metabase.notification.test-util :as notification.tu]
[metabase.pulse.send :as pulse.send]
@@ -268,7 +267,7 @@
(deftest sync-creates-task-run-test
(mt/with-model-cleanup [:model/TaskRun :model/TaskHistory]
(testing "sync-operation creates a task run with type :sync"
- (let [mock-db (mi/instance :model/Database {:name "test" :id 999 :engine :h2})
+ (let [mock-db {:lib/type :metadata/database :name "test" :id 999 :engine :h2}
step-name (mt/random-name)
sync-steps [(sync-util/create-sync-step step-name (fn [_] {:total-tables 1}))]]
(sync-util/sync-operation :sync-metadata mock-db "Test sync"
@@ -283,7 +282,7 @@
(deftest analyze-creates-fingerprint-run-test
(mt/with-model-cleanup [:model/TaskRun :model/TaskHistory]
(testing "analyze operation creates a task run with type :fingerprint"
- (let [mock-db (mi/instance :model/Database {:name "test" :id 998 :engine :h2})
+ (let [mock-db {:lib/type :metadata/database :name "test" :id 998 :engine :h2}
step-name (mt/random-name)
sync-steps [(sync-util/create-sync-step step-name (fn [_] {:total-tables 1}))]]
(sync-util/sync-operation :analyze mock-db "Test analyze"
@@ -296,7 +295,7 @@
(deftest nested-sync-operations-share-run-test
(mt/with-model-cleanup [:model/TaskRun :model/TaskHistory]
(testing "nested sync operations don't create multiple task runs"
- (let [mock-db (mi/instance :model/Database {:name "test" :id 997 :engine :h2})
+ (let [mock-db {:lib/type :metadata/database :name "test" :id 997 :engine :h2}
outer-step (mt/random-name)
inner-step (mt/random-name)
outer-steps [(sync-util/create-sync-step outer-step
diff --git a/test/metabase/test/data/impl.clj b/test/metabase/test/data/impl.clj
index b9c14f2cc87..2800a0877ce 100644
--- a/test/metabase/test/data/impl.clj
+++ b/test/metabase/test/data/impl.clj
@@ -6,6 +6,7 @@
[metabase.driver :as driver]
[metabase.driver.util :as driver.u]
[metabase.lib.schema.id :as lib.schema.id]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.secrets.core :as secret]
[metabase.test.data.impl.get-or-create :as test.data.impl.get-or-create]
[metabase.test.data.impl.verify :as verify]
@@ -217,7 +218,7 @@
(mu/defn database-source-dataset-name :- :string
"Get the name of the test dataset this Database was created from, e.g. `test-data`."
- [database :- [:or ::warehouses.schema/database ::warehouses.schema/database.update]]
+ [database :- [:or ::lib.schema.metadata/database ::warehouses.schema/database ::warehouses.schema/database.update]]
(get-in database [:settings :database-source-dataset-name]))
(mu/defn the-table-id :- ::lib.schema.id/table
diff --git a/test/metabase/test/data/impl/get_or_create.clj b/test/metabase/test/data/impl/get_or_create.clj
index f431e79fcec..9bb470ee239 100644
--- a/test/metabase/test/data/impl/get_or_create.clj
+++ b/test/metabase/test/data/impl/get_or_create.clj
@@ -276,7 +276,7 @@
;; Only run fingerprinting for :full scan, matching the real sync behavior
(when (= scan :full)
(log/info "Running real fingerprinting for :full scan after fake sync")
- (sync.analyze/analyze-db! db))))
+ (sync.analyze/analyze-db! (:id db)))))
;;; ----------------------------------------------- End Fake Sync -----------------------------------------------
@@ -303,7 +303,7 @@
;; Redshift and BigQuery list tables from metadata that lags the tables themselves, so a table dropped by
;; a concurrent CI job can still appear in the listing and then 404 when sync reads it.
(binding [sync-util/*log-exceptions-and-continue?* (contains? #{:redshift :bigquery-cloud-sdk} driver)]
- (sync/sync-database! db {:scan scan}))
+ (sync/sync-database! (:id db) {:scan scan}))
;; add extra metadata for fields
(try
(add-extra-metadata! database-definition db)
diff --git a/test/metabase/test/data/impl/verify.clj b/test/metabase/test/data/impl/verify.clj
index ce35cff00d5..690a001a72f 100644
--- a/test/metabase/test/data/impl/verify.clj
+++ b/test/metabase/test/data/impl/verify.clj
@@ -2,6 +2,7 @@
"Logic for verifying that test data was loaded correctly."
(:require
[metabase.driver :as driver]
+ [metabase.driver.util :as driver.u]
[metabase.query-processor.core :as qp]
[metabase.test.data.interface :as tx]
[metabase.util :as u]
@@ -18,12 +19,12 @@
(defn- loaded-tables
"Actual Tables loaded into `database`. Returns a set of `[schema-name table-name]` pairs."
[{:keys [driver database]}]
- (into #{} (map (juxt :schema :name)) (:tables (driver/describe-database driver database))))
+ (into #{} (map (juxt :schema :name)) (:tables (driver/describe-database driver (driver.u/ensure-lib-database database)))))
(defn- loaded-fields
"Actual Fields loaded into a Table. Returns set of field names."
[{:keys [driver database actual-schema actual-table-name]}]
- (->> (driver/describe-table driver database {:schema actual-schema, :name actual-table-name, :db_id (:id database)})
+ (->> (driver/describe-table driver (driver.u/ensure-lib-database database) {:schema actual-schema, :name actual-table-name, :db_id (:id database)})
:fields
(map :name)
set))
diff --git a/test/metabase/test/data/one_off_dbs.clj b/test/metabase/test/data/one_off_dbs.clj
index f6a00202ef6..c5d67e3ea1a 100644
--- a/test/metabase/test/data/one_off_dbs.clj
+++ b/test/metabase/test/data/one_off_dbs.clj
@@ -8,8 +8,7 @@
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
[metabase.sync.core :as sync]
[metabase.test :as mt]
- [metabase.test.data :as data]
- [toucan2.core :as t2]))
+ [metabase.test.data :as data]))
(def ^:dynamic *conn*
"Bound to a JDBC connection spec when using one of the `with-db` macros below."
@@ -80,4 +79,4 @@
[values]
(jdbc/execute! *conn* [(insert-sql values)])
;; `(data/db)` is the map from before the first sync; its stale `initial_sync_status` would skip the FieldValues scan
- (sync/sync-database! (t2/select-one :model/Database (data/id))))
+ (sync/sync-database! (data/id)))
diff --git a/test/metabase/test/data/sql_jdbc.clj b/test/metabase/test/data/sql_jdbc.clj
index 11d1c3bbbc4..24af8a81e3c 100644
--- a/test/metabase/test/data/sql_jdbc.clj
+++ b/test/metabase/test/data/sql_jdbc.clj
@@ -5,6 +5,7 @@
[metabase.driver :as driver]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
[metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]
+ [metabase.driver.util :as driver.u]
[metabase.test.data.interface :as tx]
[metabase.test.data.sql :as sql.tx]
[metabase.test.data.sql-jdbc.load-data :as load-data]
@@ -106,12 +107,12 @@
(defmethod tx/create-view-of-table! :sql-jdbc/test-extensions
[driver database view-name table-name options]
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec database)
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
(sql.tx/create-view-of-table-sql driver database view-name table-name options)
{:transaction? false}))
(defmethod tx/drop-view! :sql-jdbc/test-extensions
[driver database view-name options]
- (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec database)
+ (jdbc/execute! (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
(sql.tx/drop-view-sql driver database view-name options)
{:transaction? false}))
diff --git a/test/metabase/test/sync.clj b/test/metabase/test/sync.clj
index eca74e7a391..05520169715 100644
--- a/test/metabase/test/sync.clj
+++ b/test/metabase/test/sync.clj
@@ -14,7 +14,7 @@
[]
(data/with-temp-copy-of-db
;; `sync-database!` does both sync an analysis steps
- (sync/sync-database! (t2/select-one :model/Database :id (data/id)))
+ (sync/sync-database! (data/id))
(t2/count :model/TaskHistory :db_id (data/id))))
(defn crash-fn
diff --git a/test/metabase/transforms/execute_test.clj b/test/metabase/transforms/execute_test.clj
index e96b2623a89..6bf29fa1e41 100644
--- a/test/metabase/transforms/execute_test.clj
+++ b/test/metabase/transforms/execute_test.clj
@@ -144,7 +144,7 @@
(mt/as-admin
(testing "create-table-from-schema! should create the table successfully"
(transforms-base.u/create-table-from-schema! driver db-id table-schema)
- (let [table-exists? (driver/table-exists? driver (mt/db) {:schema schema-name :name table-name})]
+ (let [table-exists? (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:schema schema-name :name table-name})]
(is (true? table-exists?) "Table should exist in the database schema")
(driver/drop-table! driver db-id (:name table-schema)))))))))
@@ -191,7 +191,7 @@
(let [details (:details (mt/db))
password (:password details)
no-schema-user (u/lower-case-en (mt/random-name))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))]
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))]
(try
(driver/execute-raw-queries! driver/*driver* spec
[[(format "CREATE ROLE %s WITH LOGIN PASSWORD '%s';" no-schema-user password)]
@@ -203,7 +203,7 @@
:details (assoc details
:user no-schema-user)}]
(mt/with-db db
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(let [mp (mt/metadata-provider)
transforms-products (lib.metadata/table mp (mt/id :transforms_products))
products-category (lib.metadata/field mp (mt/id :transforms_products :category))
@@ -444,7 +444,7 @@
(let [details (:details (mt/db))
password (:password details)
readonly-user (u/lower-case-en (mt/random-name))
- spec (sql-jdbc.conn/db->pooled-connection-spec (mt/id))
+ spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
schema (t2/select-one-fn :schema :model/Table (mt/id :transforms_products))
make-widget-query
(fn [mp]
@@ -473,7 +473,7 @@
(fn [db-opts target-name]
(mt/with-temp [:model/Database db (merge {:engine :postgres} db-opts)]
(mt/with-db db
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(let [query (make-widget-query (mt/metadata-provider))]
(with-transform-cleanup! [target-table {:type "table"
:schema schema
@@ -522,7 +522,7 @@
(when-not old-write-details
(t2/update! :model/Database db-id {:write_data_details (:details (mt/db))}))
(try
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id))
(testing "transform pool does not exist before transform execution"
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool transform-cache-key))))
(with-transform-cleanup! [target-table {:type "table"
@@ -543,7 +543,7 @@
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool transform-cache-key))))))
(finally
(t2/update! :model/Database db-id {:write_data_details old-write-details})
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db))))))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id))))))))))
(deftest transform-run-refreshes-target-stats-test
(testing "completing a transform run leaves its target table analyzed, so a dependent transform plans
@@ -553,7 +553,7 @@
(mt/dataset transforms-dataset/transforms-test
(let [schema (t2/select-one-fn :schema :model/Table (mt/id :transforms_products))
source-name (t2/select-one-fn :name :model/Table (mt/id :transforms_products))
- admin-spec (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ admin-spec (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
;; A table has one pg_statistic row per analyzed column; a never-analyzed CTAS table has none.
analyzed-column-count
(fn [table-name]
diff --git a/test/metabase/transforms/index_test.clj b/test/metabase/transforms/index_test.clj
index 54524e51cd7..e6b2af4c880 100644
--- a/test/metabase/transforms/index_test.clj
+++ b/test/metabase/transforms/index_test.clj
@@ -8,6 +8,7 @@
[clojure.test :refer :all]
[metabase.driver :as driver]
[metabase.indexes.reconcile :as reconcile]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.test :as mt]
[metabase.test.data.sql :as sql.tx]
[metabase.transforms-base.util :as transforms-base.u]
@@ -195,7 +196,7 @@
;; GET /indexes API consumes reads them back correctly, including catalog shapes the apply path can't produce
;; (e.g. a Postgres gin or partial index). Tables are created directly from raw DDL.
(mt/test-drivers (index-util/index-test-drivers)
- (let [db (mt/db)
+ (let [db (lib.metadata/database (mt/metadata-provider))
schema (fetch-schema driver/*driver*)
cases (index-util/fetch-cases driver/*driver*)]
(is (driver/database-supports? driver/*driver* :index/fetch db)
@@ -286,7 +287,7 @@
:target {:database (mt/id) :type "table" :schema "public" :name "t"}}
:model/TableIndex {idx-id :id} {:transform_id tid :index_name "boom_idx"
:structured {:kind :btree :name "boom_idx" :columns [{:name "x"}]}}]
- (let [transform {:id tid :target {:database (mt/id) :schema "public" :name "t"
+ (let [transform {:id tid :target {:database (mt/id) :type "table" :schema "public" :name "t"
:indexes [{:kind :btree :name "boom_idx" :columns [{:name "x"}]}]}}]
(with-redefs [driver/supported-index-methods (fn [& _] {:btree {:lifecycle :standalone}})
driver/connection-spec (fn [& _] {})
@@ -294,7 +295,7 @@
driver/execute-raw-queries! (fn [& _] (throw (ex-info "ddl boom" {})))]
(is (thrown? Throwable
(#'transforms-base.u/apply-standalone-indexes!
- {:engine :postgres :id (mt/id)} (assoc (:target transform) :transform-id tid))))
+ {:lib/type :metadata/database :engine :postgres :id (mt/id)} (assoc (:target transform) :transform-id tid))))
(is (= :failed (t2/select-one-fn :status :model/TableIndex idx-id)))
(is (re-find #"ddl boom" (t2/select-one-fn :error_message :model/TableIndex idx-id)))))))
diff --git a/test/metabase/transforms/index_test_util.clj b/test/metabase/transforms/index_test_util.clj
index cfc7c9b9c1e..537a1ed9309 100644
--- a/test/metabase/transforms/index_test_util.clj
+++ b/test/metabase/transforms/index_test_util.clj
@@ -6,13 +6,14 @@
[clojure.set :as set]
[clojure.string :as str]
[metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]
+ [metabase.driver.util :as driver.u]
[metabase.test :as mt]))
(set! *warn-on-reflection* true)
(defn- postgres-indexes
[database schema table]
- (->> (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec database)
+ (->> (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
["SELECT indexname FROM pg_indexes WHERE schemaname = ? AND tablename = ?" schema table])
(map :indexname)
set))
@@ -22,7 +23,7 @@
`sortkey` is the column's 1-based position in the key; an interleaved key alternates the sign, so a negative value
anywhere means interleaved while `abs` still gives the position. `distkey` is true on the single KEY-dist column."
[database schema table]
- (let [rows (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec database)
+ (let [rows (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
[(str "SELECT column_name, sortkey, distkey FROM svv_redshift_columns "
"WHERE schema_name = ? AND table_name = ? ORDER BY abs(sortkey)")
schema table])
@@ -35,7 +36,7 @@
(defn- mysql-indexes
[database schema table]
;; MySQL has no schemas, so `schema` is nil; fall back to the connection's database.
- (->> (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec database)
+ (->> (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
[(str "SELECT DISTINCT index_name FROM information_schema.statistics "
"WHERE table_schema = COALESCE(?, DATABASE()) AND table_name = ?")
schema table])
@@ -45,7 +46,7 @@
(defn- clickhouse-indexes
[database schema table]
- (let [spec (sql-jdbc.conn/db->pooled-connection-spec database)]
+ (let [spec (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))]
{:sorting-key (-> (jdbc/query spec ["SELECT sorting_key FROM system.tables WHERE database = ? AND name = ?"
schema table])
first :sorting_key)
@@ -73,7 +74,7 @@
"Read a Snowflake table's clustering key back as an ordered vector of column names, or [] when unclustered.
`CLUSTERING_KEY` comes back as a string like `LINEAR(category, price)`."
[database schema table]
- (let [clustering-key (-> (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec database)
+ (let [clustering-key (-> (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
[(str "SELECT clustering_key FROM information_schema.tables "
"WHERE table_schema = ? AND table_name = ?")
schema table])
@@ -89,7 +90,7 @@
(defn- sqlserver-indexes
[database schema table]
- (->> (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec database)
+ (->> (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database database))
[(str "SELECT i.name AS index_name FROM sys.indexes i "
"WHERE i.object_id = OBJECT_ID(?) AND i.type IN (1, 2) AND i.name IS NOT NULL")
(format "[%s].[%s]" schema table)])
diff --git a/test/metabase/transforms/test_util.clj b/test/metabase/transforms/test_util.clj
index 2a1eda8b134..ce4cf54069c 100644
--- a/test/metabase/transforms/test_util.clj
+++ b/test/metabase/transforms/test_util.clj
@@ -6,6 +6,7 @@
[metabase.config.core :as config]
[metabase.driver :as driver]
[metabase.driver.util :as driver.u]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.test :as mt]
[metabase.test.data.interface :as tx]
[metabase.util :as u]
@@ -92,10 +93,10 @@
api/*current-user-id* (mt/user->id :crowberto)]
;; Drop the actual table/view from the database
(try
- (driver/drop-transform-target! driver (mt/db) (update target :type (fn [target-type]
- (if (string? target-type)
- (keyword target-type)
- target-type))))
+ (driver/drop-transform-target! driver (lib.metadata/database (mt/metadata-provider)) (update target :type (fn [target-type]
+ (if (string? target-type)
+ (keyword target-type)
+ target-type))))
(catch Exception e
(log/warnf e "Failed to drop transform target table %s.%s for driver %s"
(:schema target) (:name target) driver)))
@@ -285,16 +286,16 @@
(doseq [driver [:postgres :snowflake]]
(defmethod delete-schema! driver [driver db schema]
- (let [conn-spec (driver/connection-spec driver db)
+ (let [conn-spec (driver/connection-spec driver (driver.u/ensure-lib-database db))
sql [[(format "DROP SCHEMA IF EXISTS \"%s\" CASCADE;" schema)]]]
(driver/execute-raw-queries! driver/*driver* conn-spec sql))))
(defmethod delete-schema! :sqlserver [driver db schema]
- (let [conn-spec (driver/connection-spec driver db)
+ (let [conn-spec (driver/connection-spec driver (driver.u/ensure-lib-database db))
sql [[(format "IF EXISTS (SELECT * FROM sys.schemas WHERE name = '%s') DROP SCHEMA [%s];" schema schema)]]]
(driver/execute-raw-queries! driver/*driver* conn-spec sql)))
(defmethod delete-schema! :clickhouse [driver db schema]
- (let [conn-spec (driver/connection-spec driver db)
+ (let [conn-spec (driver/connection-spec driver (driver.u/ensure-lib-database db))
sql [[(format "DROP DATABASE IF EXISTS `%s`;" schema)]]]
(driver/execute-raw-queries! driver/*driver* conn-spec sql)))
diff --git a/test/metabase/transforms/util_test.clj b/test/metabase/transforms/util_test.clj
index bef57ed0788..bcd9c985738 100644
--- a/test/metabase/transforms/util_test.clj
+++ b/test/metabase/transforms/util_test.clj
@@ -13,6 +13,7 @@
[metabase.driver.util :as driver.u]
[metabase.lib-be.core :as lib-be]
[metabase.lib.core :as lib]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.permissions.core :as perms]
[metabase.permissions.models.data-permissions :as data-perms]
[metabase.permissions.models.permissions-group :as perms-group]
@@ -66,7 +67,7 @@
(testing "Can create table with generated temp name"
(driver/create-table! driver db-id qualified-table-name column-definitions {})
(when-not (= driver :mongo) ;; mongo doesn't actually create tables
- (is (driver/table-exists? driver (mt/db) {:schema schema-name :name (name table-name)}))))
+ (is (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:schema schema-name :name (name table-name)}))))
(finally
(try
(driver/drop-table! driver db-id qualified-table-name)
@@ -102,11 +103,11 @@
(try
(testing "Creating table with ordered columns"
(transforms-base.u/create-table-from-schema! driver db-id table-schema)
- (is (driver/table-exists? driver (mt/db) {:schema schema-name :name (name table-name)})))
+ (is (driver/table-exists? driver (lib.metadata/database (mt/metadata-provider)) {:schema schema-name :name (name table-name)})))
(when (get-method driver/describe-table driver)
(testing "Column order matches schema definition order (not alphabetical)"
(let [table-metadata {:schema schema-name :name (name table-name)}
- described-fields (:fields (driver/describe-table driver (mt/db) table-metadata))
+ described-fields (:fields (driver/describe-table driver (lib.metadata/database (mt/metadata-provider)) table-metadata))
sorted-fields (sort-by :database-position described-fields)
column-names (mapv :name sorted-fields)
expected-names ["zebra_col" "apple_col" "mango_col"]]
@@ -444,7 +445,7 @@
created))
sync/sync-table! (constantly nil)
driver/table-exists? (constantly false)]
- (transforms-base.u/activate-table-and-mark-computed! db target)
+ (transforms-base.u/activate-table-and-mark-computed! (driver.u/lib-database (:id db)) target)
(is (some? @synced-table))
(let [table (t2/select-one :model/Table (:id @synced-table))]
(is (= :computed (:data_authority table)))
@@ -583,7 +584,7 @@
excluded; we just verify the Statement was created."
(mt/test-drivers (into #{} (filter #(isa? driver/hierarchy % :sql-jdbc)) (mt/normal-drivers))
(sql-jdbc.execute/do-with-connection-with-options
- driver/*driver* (mt/db) {:write? false}
+ driver/*driver* (mt/id) {:write? false}
(fn [^java.sql.Connection conn]
(doseq [minutes [3 90]]
(binding [driver.settings/*query-timeout-ms* (u/minutes->ms minutes)]
@@ -796,7 +797,7 @@
;; Simulate: physical table exists in "PUBLIC" (H2's default schema)
driver/table-exists? (fn [_driver _db table]
(= "PUBLIC" (:schema table)))]
- (transforms-base.u/activate-table-and-mark-computed! db target)
+ (transforms-base.u/activate-table-and-mark-computed! (driver.u/lib-database (:id db)) target)
(is (some? @synced-table))
(let [table (t2/select-one :model/Table (:id @synced-table))]
(is (= "PUBLIC" (:schema table))
@@ -820,7 +821,7 @@
sync/sync-table! (constantly nil)
;; Simulate: physical table does NOT exist under default schema
driver/table-exists? (constantly false)]
- (transforms-base.u/activate-table-and-mark-computed! db target)
+ (transforms-base.u/activate-table-and-mark-computed! (driver.u/lib-database (:id db)) target)
(is (some? @synced-table))
(let [table (t2/select-one :model/Table (:id @synced-table))]
(is (nil? (:schema table))
diff --git a/test/metabase/transforms_base/util_test.clj b/test/metabase/transforms_base/util_test.clj
index a10c465e4cb..d619b26e3ab 100644
--- a/test/metabase/transforms_base/util_test.clj
+++ b/test/metabase/transforms_base/util_test.clj
@@ -4,6 +4,7 @@
[java-time.api :as t]
[metabase.config.core :as config]
[metabase.driver :as driver]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.sync.sync :as sync]
[metabase.test :as mt]
[metabase.transforms-base.util :as transforms-base.u]
@@ -75,7 +76,7 @@
(mt/with-premium-features #{}
(is (nil? (transforms-base.u/throw-if-db-routing-enabled!
{:name "OSS transform"}
- (mt/db))))))
+ (lib.metadata/database (mt/metadata-provider)))))))
(when config/ee-available?
(testing "with :database-routing premium feature enabled, the check throws on a routing-enabled database"
(mt/with-premium-features #{:database-routing}
@@ -86,7 +87,7 @@
#".*database routing turned on"
(transforms-base.u/throw-if-db-routing-enabled!
{:name "Routing transform"}
- (mt/db)))))))))
+ (lib.metadata/database (mt/metadata-provider))))))))))
(deftest activate-table-syncs-despite-disable-auto-sync-test
(testing "disable-auto-sync gates *automatic* syncs only; a transform finalizing its"
@@ -98,7 +99,7 @@
(mt/with-dynamic-fn-redefs [sync/sync-table! (fn [_] (swap! calls inc))]
(mt/with-temporary-setting-values [disable-auto-sync true]
(transforms-base.u/activate-table-and-mark-computed!
- (mt/db)
+ (lib.metadata/database (mt/metadata-provider))
{:type "table" :schema nil :name "disable_auto_sync_target"}))
(is (= 1 @calls)
"Expected the transform path to run sync/sync-table! exactly once with disable-auto-sync on.")))))))
diff --git a/test/metabase/transforms_rest/api/transform_test.clj b/test/metabase/transforms_rest/api/transform_test.clj
index 4d4a2ae0c26..e6155e452ec 100644
--- a/test/metabase/transforms_rest/api/transform_test.clj
+++ b/test/metabase/transforms_rest/api/transform_test.clj
@@ -680,7 +680,7 @@
:target {:type "table" :schema (get-test-schema) :name table-name}}
created (mt/user-http-request user :post 200 "transform" body)
get-resp (mt/user-http-request user :get 200 (format "transform/%s" (:id created)))
- kinds (set (keys (driver/supported-index-methods driver/*driver* (mt/db))))]
+ kinds (set (keys (driver/supported-index-methods driver/*driver* (lib.metadata/database (mt/metadata-provider)))))]
(testing "the key is always present"
(is (contains? get-resp :requestable_indexes)))
(if (seq kinds)
diff --git a/test/metabase/upload/impl_test.clj b/test/metabase/upload/impl_test.clj
index 229ba84062d..5818557d310 100644
--- a/test/metabase/upload/impl_test.clj
+++ b/test/metabase/upload/impl_test.clj
@@ -60,13 +60,13 @@
(def ^:private text-type ::upload-types/text)
(defn- local-infile-on? []
- (= "ON" (-> (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (= "ON" (-> (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(jdbc/query "show global variables like 'local_infile'")
first
:value)))
(defn- set-local-infile! [on?]
- (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (mt/db)) (str "set global local_infile = " (if on? 1 0))))
+ (jdbc/query (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider))) (str "set global local_infile = " (if on? 1 0))))
(defn- do-with-mysql-local-infile-on
[thunk]
@@ -124,10 +124,10 @@
(let [table-name (ddl.i/format-name driver/*driver* table-name)
schema-name (or (some->> schema-name (ddl.i/format-name driver/*driver*))
(sql.tx/session-schema driver/*driver*))
- table (sync-tables/create-or-reactivate-table! database {:name table-name :schema schema-name})]
+ table (sync-tables/create-or-reactivate-table! (driver.u/lib-database (:id database)) {:name table-name :schema schema-name})]
(t2/update! :model/Table (:id table) {:is_upload true})
(binding [upload/*auxiliary-sync-steps* :synchronous]
- (#'upload/scan-and-sync-table! database table))
+ (#'upload/scan-and-sync-table! table))
(t2/select-one :model/Table (:id table))))
(defn- tmp-file [prefix extension]
@@ -448,7 +448,7 @@
:table-prefix "uploaded_magic_"}
(constantly nil)))
(is (not (some #(re-find #"uploaded_magic_" (:name % ""))
- (driver/describe-database driver/*driver* (mt/db))))))))
+ (driver/describe-database driver/*driver* (lib.metadata/database (mt/metadata-provider)))))))))
(deftest create-from-csv-display-name-test
(mt/test-drivers (mt/normal-drivers-with-feature :uploads)
@@ -1132,7 +1132,7 @@
(mt/test-drivers [:mysql]
(do-with-mysql-local-infile-on
(fn []
- (is (= "ON" (-> (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (is (= "ON" (-> (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(jdbc/query "show global variables like 'local_infile'")
first
:value)))))))
@@ -1140,7 +1140,7 @@
(mt/test-drivers [:mysql]
(do-with-mysql-local-infile-off
(fn []
- (is (= "OFF" (-> (sql-jdbc.conn/db->pooled-connection-spec (mt/db))
+ (is (= "OFF" (-> (sql-jdbc.conn/db->pooled-connection-spec (lib.metadata/database (mt/metadata-provider)))
(jdbc/query "show global variables like 'local_infile'")
first
:value))))))))
@@ -2678,14 +2678,14 @@
:write-data
:default))]
(try
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id))
(testing "write pool does not exist before upload create"
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))))
(with-upload-table! [_table (create-from-csv-and-sync-with-defaults!)]
(testing "write pool is created during upload create"
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))))
(finally
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db)))))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id)))))))))
(deftest upload-delete-creates-write-pool-test
(mt/test-drivers (mt/normal-driver-select {:+parent :sql-jdbc, :+features [:uploads]})
@@ -2699,11 +2699,11 @@
:default))]
(try
(let [table (create-upload-table!)]
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id))
(testing "write pool does not exist before upload delete"
(is (not (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))))
(upload/delete-upload! table)
(testing "write pool is created during upload delete"
(is (contains? @@#'sql-jdbc.conn/pool-cache-key->connection-pool write-cache-key))))
(finally
- (sql-jdbc.conn/invalidate-pool-for-db! (mt/db)))))))))
+ (sql-jdbc.conn/invalidate-pool-for-db! (mt/id)))))))))
diff --git a/test/metabase/warehouse_schema/models/field_values_test.clj b/test/metabase/warehouse_schema/models/field_values_test.clj
index 1d57e07b00d..8f9031d1e75 100644
--- a/test/metabase/warehouse_schema/models/field_values_test.clj
+++ b/test/metabase/warehouse_schema/models/field_values_test.clj
@@ -200,7 +200,7 @@
(select-keys [:values :human_readable_values])))
(defn- sync-and-find-values! [db field-values-id]
- (sync/sync-database! (t2/select-one :model/Database (u/the-id db)))
+ (sync/sync-database! (:id (t2/select-one :model/Database (u/the-id db))))
(find-values field-values-id))
(deftest implicit-deduplication-test
@@ -456,7 +456,7 @@
:is_full_sync true
:details "{\"db\": \"mem:temp\"}"}]
;; Sync the database so we have the new table and it's fields
- (sync/sync-database! db)
+ (sync/sync-database! (:id db))
(let [table-id (t2/select-one-fn :id :model/Table :db_id (u/the-id db) :name "FOO")
field-id (t2/select-one-fn :id :model/Field :table_id table-id :name "CATEGORY_ID")
;; Manually activate Field values since they are not created during sync (#53387)
diff --git a/test/metabase/warehouse_schema/models/table_test.clj b/test/metabase/warehouse_schema/models/table_test.clj
index 5cbda33a6f6..d323f850812 100644
--- a/test/metabase/warehouse_schema/models/table_test.clj
+++ b/test/metabase/warehouse_schema/models/table_test.clj
@@ -58,12 +58,12 @@
"('California Gull', 'Steven Seagull'), "
"('Chicken', 'Colin Fowl');")]]
(jdbc/execute! one-off-dbs/*conn* [statement]))
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (#'table-user-settings/valid-field-order? (mt/id :birds)
[(mt/id :birds :species)
(mt/id :birds :example_name)]))
(jdbc/execute! one-off-dbs/*conn* ["ALTER TABLE \"BIRDS\" DROP COLUMN \"EXAMPLE_NAME\";"])
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(is (#'table-user-settings/valid-field-order? (mt/id :birds)
[(mt/id :birds :species)])))))
diff --git a/test/metabase/warehouse_schema_rest/api/field_test.clj b/test/metabase/warehouse_schema_rest/api/field_test.clj
index 0a3b3d759c1..886ad8c6cea 100644
--- a/test/metabase/warehouse_schema_rest/api/field_test.clj
+++ b/test/metabase/warehouse_schema_rest/api/field_test.clj
@@ -288,7 +288,7 @@
(testing "Refingerprints field when updated"
(mt/with-dynamic-fn-redefs [quick-task/submit-task! (fn [task] (task))]
(mt/dataset integer-coerceable
- (sync/sync-database! (t2/select-one :model/Database :id (mt/id)))
+ (sync/sync-database! (:id (t2/select-one :model/Database :id (mt/id))))
(let [field-id (mt/id :t :f)
set-strategy! (fn [strategy]
(mt/user-http-request :crowberto :put 200 (format "field/%d" field-id)
@@ -929,7 +929,7 @@
(mt/with-temp [:model/Database database {:engine driver/*driver* :details (assoc (:details database) :json-unfolding true)}]
(mt/with-db database
;; Sync the new database
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [field (t2/select-one :model/Field :id (mt/id :json :json_bit))
get-database (fn [] (t2/select-one :model/Database :id (mt/id)))
set-json-unfolding-for-field! (fn [v]
@@ -948,11 +948,11 @@
(is (seq (nested-fields))))
(testing "nested fields are removed when json unfolding is disabled for the DB"
(set-json-unfolding-for-db! false)
- (sync/sync-database! (get-database))
+ (sync/sync-database! (:id (get-database)))
(is (empty? (nested-fields))))
(testing "nested fields are added when json unfolding is enabled again for the DB"
(set-json-unfolding-for-db! true)
- (sync/sync-database! (get-database))
+ (sync/sync-database! (:id (get-database)))
(is (seq (nested-fields))))
(testing "nested fields are removed when json unfolding is disabled for the field"
(set-json-unfolding-for-field! false)
@@ -971,7 +971,7 @@
(mt/with-temp [:model/Database database {:engine driver/*driver* :details (assoc (:details database) :json-unfolding false)}]
(mt/with-db database
;; Sync the new database
- (sync/sync-database! database)
+ (sync/sync-database! (:id database))
(let [get-field (fn [] (t2/select-one :model/Field :id (mt/id :json :json_bit)))
get-database (fn [] (t2/select-one :model/Database :id (mt/id)))
set-json-unfolding-for-field! (fn [v]
@@ -1000,11 +1000,11 @@
(set-json-unfolding-for-db! true)
(is (true? (:json-unfolding (:details (get-database)))))
(is (true? (:json_unfolding (get-field))))
- (sync/sync-database! (get-database))
+ (sync/sync-database! (:id (get-database)))
(is (seq (nested-fields))))
(testing "nested fields are removed when json unfolding is disabled again"
(set-json-unfolding-for-db! false)
- (sync/sync-database! (get-database))
+ (sync/sync-database! (:id (get-database)))
(is (empty? (nested-fields)))))))))))))
(deftest coercion-strategy-is-respected-after-follow-up-request-test
diff --git a/test/metabase/warehouse_schema_rest/api/table_test.clj b/test/metabase/warehouse_schema_rest/api/table_test.clj
index 725622d195c..b852c0d00a4 100644
--- a/test/metabase/warehouse_schema_rest/api/table_test.clj
+++ b/test/metabase/warehouse_schema_rest/api/table_test.clj
@@ -260,7 +260,7 @@
(testing "GET api/table/:id/query_metadata?include_sensitive_fields"
(testing "Sensitive fields are included"
(is (= (merge
- (update (table-defaults) :db merge {:details nil, :write_data_details nil, :admin_details nil, :initial_sync_error nil})
+ (update (table-defaults) :db merge {:details nil, :write_data_details nil, :initial_sync_error nil})
(t2/hydrate (t2/select-one [:model/Table :created_at :updated_at :initial_sync_status :view_count]
:id (mt/id :users))
:collection)
@@ -481,7 +481,7 @@
(is (= (merge
(-> (table-defaults)
(dissoc :segments :field_values :metrics :measures :updated_at)
- (update :db merge (select-keys (mt/db) [:details :write_data_details :admin_details :initial_sync_error])))
+ (update :db merge (select-keys (mt/db) [:details :write_data_details :initial_sync_error])))
(t2/hydrate (t2/select-one [:model/Table :id :schema :name :created_at :initial_sync_status] :id (u/the-id table))
:pk_field :collection)
{:description "What a nice table!"
@@ -1525,18 +1525,18 @@
["country" [{:field-name "continent_id", :base-type :type/Integer}]
[]]]
(let [db (mt/db)
- db-spec (sql-jdbc.conn/db->pooled-connection-spec db)
+ db-spec (sql-jdbc.conn/db->pooled-connection-spec (driver.u/ensure-lib-database db))
get-fk-target #(t2/select-one-fn :fk_target_field_id :model/Field (mt/id :country :continent_id))]
;; 1. add FK relationship in the database targeting continent_1
(jdbc/execute! db-spec "ALTER TABLE country ADD CONSTRAINT country_continent_id_fkey FOREIGN KEY (continent_id) REFERENCES continent(id);")
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(testing "initially country's continent_id is targeting continent_1"
(is (= (mt/id :continent :id)
(get-fk-target))))
(is (= 1 (count (mt/user-http-request :rasta :get 200 (format "table/%d/fks" (mt/id :continent))))))
;; 2. drop the country table
(jdbc/execute! db-spec "DROP TABLE country;")
- (sync/sync-database! db {:scan :schema})
+ (sync/sync-database! (:id db) {:scan :schema})
(is (= () (mt/user-http-request :rasta :get 200 (format "table/%d/fks" (mt/id :continent)))))))))
;;; -------------------------------------------- can-query filter tests --------------------------------------------
diff --git a/test/metabase/warehouses/models/database_test.clj b/test/metabase/warehouses/models/database_test.clj
index 5eb7c569be3..73142de3c0b 100644
--- a/test/metabase/warehouses/models/database_test.clj
+++ b/test/metabase/warehouses/models/database_test.clj
@@ -40,24 +40,24 @@
(testing "so that newly inserted or edited databases skip Quartz trigger registration."
(mt/with-temp [:model/Database db {}]
(testing "default (disable-auto-sync=false): a regular database is auto-sync-eligible"
- (is (true? (database/should-auto-sync? db))))
+ (is (true? (database/should-auto-sync? (:id db)))))
(testing "with disable-auto-sync=true: a regular database is not auto-sync-eligible"
(mt/with-temporary-setting-values [disable-auto-sync true]
- (is (false? (database/should-auto-sync? db)))))))
+ (is (false? (database/should-auto-sync? (:id db))))))))
(testing "but should-sync? (the gate for explicit, user-requested syncs) ignores the setting"
(mt/with-temp [:model/Database db {}]
- (is (true? (database/should-sync? db)))
+ (is (true? (database/should-sync? (:id db))))
(mt/with-temporary-setting-values [disable-auto-sync true]
- (is (true? (database/should-sync? db))
+ (is (true? (database/should-sync? (:id db)))
"an explicit Sync-now must remain available even when auto-sync is disabled")))))
(testing "destination databases (router children) are excluded from both, regardless of the flag"
(mt/with-temp [:model/Database router {}
:model/Database dest {:router_database_id (:id router)}]
- (is (false? (database/should-sync? dest)))
- (is (false? (database/should-auto-sync? dest)))
+ (is (false? (database/should-sync? (:id dest))))
+ (is (false? (database/should-auto-sync? (:id dest))))
(mt/with-temporary-setting-values [disable-auto-sync true]
- (is (false? (database/should-sync? dest)))
- (is (false? (database/should-auto-sync? dest)))))))
+ (is (false? (database/should-sync? (:id dest))))
+ (is (false? (database/should-auto-sync? (:id dest))))))))
(defn- trigger-for-db [db-id]
(some (fn [{trigger-key :key, :as trigger}]
@@ -816,7 +816,7 @@
(t2/update! :model/Database (:id db) {:engine (u/qualified-name ::alternate-details-driver)}))))))
(deftest overlay-details-are-validated-the-way-they-are-resolved-test
- ;; `:write_data_details` and `:admin_details` are merged onto `:details` by
+ ;; `:write_data_details` is merged onto `:details` by
;; [[metabase.driver.connection/effective-details]] rather than used on their own, so that merge is what has to
;; satisfy the policy -- an overlay holding nothing but credentials repoints nothing.
(mt/with-temp-env-var-value! [mb-warehouse-allowed-networks "external-only"]
@@ -828,7 +828,7 @@
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"private or internal network address"
- (t2/update! :model/Database (:id db) {:admin_details {:host "127.0.0.1" :user "hummingbird"}})))))))
+ (t2/update! :model/Database (:id db) {:write_data_details {:host "127.0.0.1" :user "hummingbird"}})))))))
(deftest disallowed-connection-parameters-are-refused-on-save-test
;; like the network policy, enforced on the model so serialization import and config-file provisioning -- which
diff --git a/test/metabase/warehouses_rest/api_test.clj b/test/metabase/warehouses_rest/api_test.clj
index 592dfab764b..73228862a3a 100644
--- a/test/metabase/warehouses_rest/api_test.clj
+++ b/test/metabase/warehouses_rest/api_test.clj
@@ -165,7 +165,7 @@
(testing "DB details visibility"
(testing "Regular users should not see DB details"
(is (= (-> (db-details)
- (dissoc :details :write_data_details :admin_details :initial_sync_error :schedules))
+ (dissoc :details :write_data_details :initial_sync_error :schedules))
(-> (mt/user-http-request :rasta :get 200 (format "database/%d" (mt/id)))
(dissoc :schedules :can_upload)))))
(testing "Superusers should see DB details"
@@ -879,12 +879,12 @@
db (first (t2/insert-returning-instances! :model/Database {:name database-name
:engine (u/qualified-name driver/*driver*)
:details connection-details}))
- _ (sync/sync-database! db)
+ _ (sync/sync-database! (:id db))
;; 2. start a long running process on another thread that uses a connection
connections-stay-open? (future
(sql-jdbc.execute/do-with-connection-with-options
driver/*driver*
- db
+ (driver.u/ensure-lib-database db)
nil
(fn [^Connection conn]
;; sleep long enough to make sure the PUT request below finishes processing,
@@ -906,7 +906,7 @@
(deftest ^:parallel fetch-database-metadata-test
(testing "GET /api/database/:id/metadata"
- (is (= (merge (dissoc (db-details) :details :write_data_details :admin_details :initial_sync_error :router_user_attribute)
+ (is (= (merge (dissoc (db-details) :details :write_data_details :initial_sync_error :router_user_attribute)
{:engine "h2"
:name "test-data (h2)"
:features (map u/qualified-name (driver.u/features :h2 (mt/db)))
@@ -1237,7 +1237,7 @@
(testing "Database details/settings *should not* come back for Rasta since she's not a superuser"
(let [expected-keys (-> #{:features :native_permissions :can_upload :router_user_attribute :transforms_permissions}
(into (keys (t2/select-one :model/Database :id (mt/id))))
- (disj :details :write_data_details :admin_details :initial_sync_error))]
+ (disj :details :write_data_details :initial_sync_error))]
(doseq [db (:data (mt/user-http-request :rasta :get 200 "database"))]
(testing (format "Database %s %d %s" (:engine db) (u/the-id db) (pr-str (:name db)))
(is (= expected-keys
@@ -3391,54 +3391,6 @@
{:write_data_details {:host "write-host"
:write-data-connection true}}))))))))
-;;; ----------------------------------------- admin_details tests -----------------------------------------
-
-(deftest ^:parallel upsert-sensitive-fields-admin-details-test
- (testing "upsert-sensitive-fields works with :admin_details key"
- (is (= {:host "localhost"
- :port 5432
- :password "new-password"}
- (#'api.database/upsert-sensitive-fields
- {:engine :h2
- :id (mt/id)
- :details {:host "localhost" :port 5432 :password "main-pass"}
- :admin_details {:host "localhost" :port 5432 :password "admin-pass"}}
- {:host "localhost"
- :port 5432
- :password "new-password"}
- :admin_details)))
- (testing "protected passwords are replaced from original"
- (is (= {:host "localhost"
- :port 5432
- :password "admin-pass"}
- (#'api.database/upsert-sensitive-fields
- {:engine :h2
- :id (mt/id)
- :details {:host "localhost" :port 5432 :password "main-pass"}
- :admin_details {:host "localhost" :port 5432 :password "admin-pass"}}
- {:host "localhost"
- :port 5432
- :password secret/protected-password}
- :admin_details))))))
-
-(deftest get-database-admin-details-test
- (testing "GET /api/database/:id"
- (testing "Superusers see admin_details with sensitive fields redacted"
- (mt/with-temp [:model/Database {db-id :id} {:engine :h2
- :details {:host "localhost"}
- :admin_details {:host "admin-host"
- :password "secret-admin-pass"}}]
- (let [response (mt/user-http-request :crowberto :get 200 (format "database/%d" db-id))]
- (is (= "admin-host" (get-in response [:admin_details :host])))
- (is (= secret/protected-password (get-in response [:admin_details :password]))))))
- (testing "Regular users do not see admin_details"
- (mt/with-temp [:model/Database {db-id :id} {:engine :h2
- :details {:host "localhost"}
- :admin_details {:host "admin-host"}}]
- (let [response (mt/user-http-request :rasta :get 200 (format "database/%d" db-id))]
- (is (not (contains? response :admin_details)))
- (is (not (contains? response :details))))))))
-
(deftest update-database-preserves-overlay-details-test
(testing "PUT /api/database/:id without write_data_details/provider_name keys"
(testing "preserves nullable fields instead of nil-ing them out"
diff --git a/test/metabase/xrays/related_test.clj b/test/metabase/xrays/related_test.clj
index 868da09349b..f9580a410ce 100644
--- a/test/metabase/xrays/related_test.clj
+++ b/test/metabase/xrays/related_test.clj
@@ -173,7 +173,7 @@
(is (= 1
(count-related-fields))))
(exec! "ALTER TABLE blueberries_consumed DROP COLUMN weight")
- (sync/sync-database! (mt/db))
+ (sync/sync-database! (mt/id))
(testing "after"
(is (= 0
(count-related-fields)))))))