The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Give Quartz a dedicated app-db connection pool to prevent deadlocks (#72261)
Edward Pagetb244f3c7ca73
diff --git a/docs/configuring-metabase/environment-variables.md b/docs/configuring-metabase/environment-variables.mdindex 031937eeae1..37d9aab824d 100644--- a/docs/configuring-metabase/environment-variables.md+++ b/docs/configuring-metabase/environment-variables.md@@ -3479,6 +3479,16 @@ Default: `"db"`Current cache backend. Dynamically rebindable primarily for test purposes.+### `MB_QUARTZ_MAX_CONNECTION_POOL_SIZE`++Type: integer<br>+Default: `5`<br>+Since: v64.0++Maximum number of connections in the dedicated pool that Metabase's internal task scheduler (Quartz) uses to talk to the application database. This pool is separate from the main application database pool (see [MB_APPLICATION_DB_MAX_CONNECTION_POOL_SIZE](#mb_application_db_max_connection_pool_size)), so scheduled tasks can always reach the application database even when the main pool is fully in use.++Scheduler operations are short, so the default is enough for most deployments. Consider raising it only if you run a very large number of scheduled items (subscriptions, alerts, syncs) and see tasks firing late.+### `MB_SESSION_SECRET_KEY`Type: string<br>diff --git a/src/metabase/app_db/connection.clj b/src/metabase/app_db/connection.cljindex 9c64a7970a9..8052ff4d4b0 100644--- a/src/metabase/app_db/connection.clj+++ b/src/metabase/app_db/connection.clj@@ -24,6 +24,12 @@(p/defrecord+ ApplicationDB [^clojure.lang.Keyword db-type^javax.sql.DataSource data-source+ ;; Dedicated (smaller) connection pool for the Quartz JDBC job store, so job-store+ ;; operations can't be starved by application code saturating the main pool. When the+ ;; ApplicationDB is created without pooling (`:create-pool?` false, e.g. for test DBs)+ ;; this is just `data-source` itself. Access it via [[quartz-data-source]], which+ ;; respects `lock` below.+ ^javax.sql.DataSource quartz-data-source;; used by [[metabase.app-db.setup-db!]] and [[metabase.app-db.db-is-set-up?]] to record whether;; the usual setup steps have been performed (i.e., running Liquibase and Clojure-land data;; migrations).@@ -72,9 +78,11 @@Options:- * `:create-pool?` -- whether to create a c3p0 connection pool data source for this application database if- `data-source` is not already a pooled data source. Default: `false`. You should only do this for application DBs- that are expected to be long-lived; for test DBs that will be destroyed at the end of the test it's hardly worth it."+ * `:create-pool?` -- whether to create a c3p0 connection pool data source for this application database (plus a+ dedicated Quartz pool -- see [[metabase.app-db.connection-pool-setup/quartz-connection-pool-data-source]]).+ Default: `false`. Requires an unpooled `data-source`: passing an already-pooled one throws. You should only do+ this for application DBs that are expected to be long-lived; for test DBs that will be destroyed at the end of+ the test it's hardly worth it."^ApplicationDB [db-type data-source & {:keys [create-pool?], :or {create-pool? false}}];; this doesn't use [[schema.core/defn]] because [[schema.core/defn]] doesn't like optional keyword args{:pre [(#{:h2 :mysql :postgres} db-type)@@ -84,6 +92,9 @@:data-source (if create-pool?(connection-pool-setup/connection-pool-data-source db-type data-source)data-source)+ :quartz-data-source (if create-pool?+ (connection-pool-setup/quartz-connection-pool-data-source db-type data-source)+ data-source):status (atom initial-db-status);; for memoization purposes. See [[unique-identifier]] for more information.:id (swap! application-db-counter inc)@@ -116,6 +127,30 @@^javax.sql.DataSource [](.data-source *application-db*))+(defn quartz-data-source+ "Get a [[javax.sql.DataSource]] for the Quartz JDBC job store, backed by the current [[*application-db*]]'s+ dedicated Quartz connection pool (or its regular data source if it was created without pooling).++ Like connections acquired through the [[ApplicationDB]] itself, acquiring a connection through this takes the+ application DB's read lock, so the testing API can block new connections while restoring the app DB."+ ^javax.sql.DataSource []+ (let [^ApplicationDB app-db *application-db*+ ^ReentrantReadWriteLock lock (.lock app-db)+ ^javax.sql.DataSource data-source (.quartz-data-source app-db)]+ (reify javax.sql.DataSource+ (getConnection [_]+ (try+ (.. lock readLock lock)+ (.getConnection data-source)+ (finally+ (.. lock readLock unlock))))+ (getConnection [_ user password]+ (try+ (.. lock readLock lock)+ (.getConnection data-source user password)+ (finally+ (.. lock readLock unlock)))))))+;; I didn't call this `id` so there's no confusing this with a data warehouse [[metabase.warehouses.models.database]] instance --;; it's a number that I don't want getting mistaken for an `Database` `id`. Also the fact that it's an Integer is not;; something callers of this function really need to be concerned aboutdiff --git a/src/metabase/app_db/connection_pool_setup.clj b/src/metabase/app_db/connection_pool_setup.cljindex c30fe3edf74..4a08cc5821e 100644--- a/src/metabase/app_db/connection_pool_setup.clj+++ b/src/metabase/app_db/connection_pool_setup.clj@@ -184,13 +184,53 @@(mu/defn connection-pool-data-source :- (ms/InstanceOfClass PoolBackedDataSource)"Create a connection pool [[javax.sql.DataSource]] from an unpooled [[javax.sql.DataSource]] `data-source`. If- `data-source` is already pooled, this will return `data-source` as-is."+ `data-source` is already pooled, this will return `data-source` as-is, ignoring `props-overrides`.+ `props-overrides` (if provided) are merged over [[application-db-connection-pool-props]]."+ (^PoolBackedDataSource [db-type data-source]+ (connection-pool-data-source db-type data-source nil))+ (^PoolBackedDataSource [db-type :- :keyword+ ^javax.sql.DataSource data-source :- (ms/InstanceOfClass javax.sql.DataSource)+ props-overrides :- [:maybe [:map-of :string :any]]]+ (if (instance? PoolBackedDataSource data-source)+ data-source+ (let [ds-name (format "metabase-%s-app-db" (name db-type))+ pool-props (merge (application-db-connection-pool-props)+ {"dataSourceName" ds-name}+ props-overrides)]+ (com.mchange.v2.c3p0.DataSources/pooledDataSource+ data-source+ (connection-pool/map->properties pool-props))))))++(def ^:private default-quartz-max-pool-size 5)++(mu/defn quartz-connection-pool-data-source :- (ms/InstanceOfClass PoolBackedDataSource)+ "Create a small connection pool for the Quartz JDBC job store, separate from the main application DB pool.++ Quartz gets a dedicated pool so job-store operations can never be starved by application code saturating the main+ pool: a thread inside a `with-transaction` that triggers a Quartz operation would otherwise deadlock waiting for a+ connection that can never be freed (the outer transaction won't release its connection until the block completes).++ Inherits all the main pool's properties (connection customizer, credential-rotation settings, etc.) except the+ pool sizing.++ `data-source` must be unpooled: an already-pooled `data-source` is rejected."^PoolBackedDataSource [db-type :- :keyword- ^PoolBackedDataSource data-source :- (ms/InstanceOfClass javax.sql.DataSource)]- (if (instance? PoolBackedDataSource data-source)- data-source- (let [ds-name (format "metabase-%s-app-db" (name db-type))- pool-props (assoc (application-db-connection-pool-props) "dataSourceName" ds-name)]- (com.mchange.v2.c3p0.DataSources/pooledDataSource- data-source- (connection-pool/map->properties pool-props)))))+ data-source :- (ms/InstanceOfClass javax.sql.DataSource)]+ ;; [[connection-pool-data-source]] returns an already-pooled data-source as-is, which here would silently make the+ ;; "dedicated" Quartz pool the same pool as the main one -- reintroducing the exact starvation deadlock this pool+ ;; exists to prevent -- so reject pooled input instead.+ (when (instance? PoolBackedDataSource data-source)+ (throw (ex-info (str "quartz-connection-pool-data-source requires an unpooled data-source: an already-pooled one"+ " would be shared with the main pool, defeating the deadlock protection.")+ {:data-source-name (.getDataSourceName ^PoolBackedDataSource data-source)})))+ (connection-pool-data-source+ db-type+ data-source+ ;; a small pool is safe: job-store operations are short (acquire the QRTZ_LOCKS row lock, run a few statements,+ ;; commit), hold exactly one connection at a time, and never wait on the main pool while holding one, so+ ;; undersizing can only queue operations briefly -- it cannot deadlock.+ {"dataSourceName" (format "metabase-%s-quartz" (name db-type))+ "maxPoolSize" (or (config/config-int :mb-quartz-max-connection-pool-size)+ default-quartz-max-pool-size)+ "minPoolSize" 1+ "initialPoolSize" 1}))diff --git a/src/metabase/app_db/core.clj b/src/metabase/app_db/core.cljindex a31a2d713d8..6ed291214ce 100644--- a/src/metabase/app_db/core.clj+++ b/src/metabase/app_db/core.clj@@ -42,6 +42,7 @@do-before-commitdo-after-commitin-transaction?+ quartz-data-sourcequoting-styleunique-identifiertransaction-statediff --git a/src/metabase/cmd/resources/other-env-vars.md b/src/metabase/cmd/resources/other-env-vars.mdindex 30d6e1ee49e..ebbebd6f777 100644--- a/src/metabase/cmd/resources/other-env-vars.md+++ b/src/metabase/cmd/resources/other-env-vars.md@@ -494,6 +494,16 @@ Default: `"db"`Current cache backend. Dynamically rebindable primarily for test purposes.+### `MB_QUARTZ_MAX_CONNECTION_POOL_SIZE`++Type: integer<br>+Default: `5`<br>+Since: v64.0++Maximum number of connections in the dedicated pool that Metabase's internal task scheduler (Quartz) uses to talk to the application database. This pool is separate from the main application database pool (see [MB_APPLICATION_DB_MAX_CONNECTION_POOL_SIZE](#mb_application_db_max_connection_pool_size)), so scheduled tasks can always reach the application database even when the main pool is fully in use.++Scheduler operations are short, so the default is enough for most deployments. Consider raising it only if you run a very large number of scheduled items (subscriptions, alerts, syncs) and see tasks firing late.+### `MB_SESSION_SECRET_KEY`Type: string<br>diff --git a/src/metabase/task/bootstrap.clj b/src/metabase/task/bootstrap.cljindex 8a9f1e972a7..545e4d06ed3 100644--- a/src/metabase/task/bootstrap.clj+++ b/src/metabase/task/bootstrap.clj@@ -5,10 +5,11 @@(set! *warn-on-reflection* true)-;; Custom `ConnectionProvider` implementation that uses our application DB connection pool to provide connections.+;; Custom `ConnectionProvider` implementation that uses a dedicated connection pool for the application DB to provide+;; connections.-(defn- app-db ^javax.sql.DataSource []- ((requiring-resolve 'metabase.app-db.core/app-db)))+(defn- quartz-data-source ^javax.sql.DataSource []+ ((requiring-resolve 'metabase.app-db.core/quartz-data-source)));; Optional interceptor for wrapping JDBC connections before Quartz uses them.;; Set by task.tracing to add SQL-level tracing. nil means no interception.@@ -24,13 +25,17 @@org.quartz.utils.ConnectionProvider(initialize [_])(getConnection [_]- ;; get a connection from our application DB connection pool. Quartz will close it (i.e., return it to the pool)- ;; when it's done+ ;; get a connection from the dedicated Quartz connection pool. Quartz will close it (i.e., return it to the pool)+ ;; when it's done.;;- ;; very important! Fetch a new connection from the connection pool rather than using currently bound Connection if- ;; one already exists -- because Quartz will close this connection when done, we don't want to screw up the- ;; calling block- (let [conn (.getConnection (app-db))]+ ;; very important! Fetch a new connection from the connection pool rather than reusing a Connection already bound+ ;; to the calling thread (e.g. toucan2's *current-connectable*) -- Quartz manages the connection's whole+ ;; lifecycle (setAutoCommit/commit/rollback/close), and its cluster locking relies on commit/rollback to release+ ;; row locks on the QRTZ_LOCKS table, so it must never share a connection with an outer transaction.+ ;;+ ;; the pool is separate from the main application DB pool so that a Quartz operation triggered by a thread inside+ ;; a `with-transaction` block can't deadlock when application code has saturated the main pool.+ (let [conn (.getConnection (quartz-data-source))](if-let [interceptor @connection-interceptor](interceptor conn)conn)))diff --git a/test/metabase/app_db/connection_pool_setup_test.clj b/test/metabase/app_db/connection_pool_setup_test.cljindex 3121c8e8aa3..52036814921 100644--- a/test/metabase/app_db/connection_pool_setup_test.clj+++ b/test/metabase/app_db/connection_pool_setup_test.clj@@ -13,7 +13,7 @@[metabase.util :as u][toucan2.core :as t2])(:import- (com.mchange.v2.c3p0 C3P0Registry ConnectionCustomizer PoolBackedDataSource)+ (com.mchange.v2.c3p0 C3P0Registry ConnectionCustomizer DataSources PoolBackedDataSource WrapperConnectionPoolDataSource)(metabase.app_db.connection_pool_setup MetabaseConnectionCustomizer)))(set! *warn-on-reflection* true)@@ -195,3 +195,30 @@(testing "on-check-in should be called; Connection read-only should be reset after checking in"(let [^java.sql.Connection conn (u/deref-with-timeout connection* (u/seconds->ms 5))](is (not (.isReadOnly conn))))))))))++(deftest quartz-connection-pool-data-source-test+ (testing "the Quartz pool defaults to maxPoolSize 5"+ (let [data-source (mdb.data-source/raw-connection-string->DataSource "jdbc:h2:mem:quartz-pool-default-size-test")+ pool (mdb.connection-pool-setup/quartz-connection-pool-data-source :h2 data-source)]+ (try+ (is (= 5 (.getMaxPoolSize ^WrapperConnectionPoolDataSource (.getConnectionPoolDataSource pool))))+ (finally+ (DataSources/destroy pool)))))+ (testing "MB_QUARTZ_MAX_CONNECTION_POOL_SIZE overrides the Quartz pool's max size"+ (mt/with-temp-env-var-value! [mb-quartz-max-connection-pool-size 7]+ (let [data-source (mdb.data-source/raw-connection-string->DataSource "jdbc:h2:mem:quartz-pool-env-size-test")+ pool (mdb.connection-pool-setup/quartz-connection-pool-data-source :h2 data-source)]+ (try+ (is (= 7 (.getMaxPoolSize ^WrapperConnectionPoolDataSource (.getConnectionPoolDataSource pool))))+ (finally+ (DataSources/destroy pool)))))))++(deftest quartz-connection-pool-rejects-pre-pooled-test+ (testing "an already-pooled data-source is rejected -- the Quartz pool must be distinct from the main pool"+ (let [data-source (mdb.data-source/raw-connection-string->DataSource "jdbc:h2:mem:quartz-pool-reject-pooled-test")+ pre-pooled (DataSources/pooledDataSource ^javax.sql.DataSource data-source)]+ (try+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"already-pooled"+ (mdb.connection-pool-setup/quartz-connection-pool-data-source :h2 pre-pooled)))+ (finally+ (DataSources/destroy pre-pooled))))))diff --git a/test/metabase/app_db/connection_test.clj b/test/metabase/app_db/connection_test.cljindex a7786a1aa6c..22934575314 100644--- a/test/metabase/app_db/connection_test.clj+++ b/test/metabase/app_db/connection_test.clj@@ -2,6 +2,7 @@(:require[clojure.test :refer :all][metabase.app-db.connection :as mdb.connection]+ [metabase.app-db.data-source :as mdb.data-source][metabase.config.core :as config][metabase.test :as mt][metabase.test.fixtures :as fixtures]@@ -9,8 +10,10 @@[toucan2.core :as t2][toucan2.jdbc.options :as t2.jdbc.options])(:import+ (com.mchange.v2.c3p0 DataSources PoolBackedDataSource WrapperConnectionPoolDataSource)(java.sql Connection SQLException)- (java.util.concurrent Semaphore)))+ (java.util.concurrent Semaphore)+ (java.util.concurrent.locks ReentrantReadWriteLock)))(set! *warn-on-reflection* true)@@ -615,3 +618,59 @@(is (not (t2/exists? :metabase_cluster_lock :lock_name lock-name))))(finally(t2/delete! :metabase_cluster_lock :lock_name lock-name)))))++(deftest quartz-data-source-pool-construction-test+ (testing "with :create-pool? true, the Quartz job store gets its own (smaller) c3p0 pool"+ (let [data-source (mdb.data-source/raw-connection-string->DataSource "jdbc:h2:mem:quartz-pool-construction-test")+ app-db (mdb.connection/application-db :h2 data-source :create-pool? true)]+ (try+ (let [^PoolBackedDataSource main (:data-source app-db)+ ^PoolBackedDataSource quartz (:quartz-data-source app-db)]+ (is (instance? PoolBackedDataSource main))+ (is (instance? PoolBackedDataSource quartz))+ (is (not (identical? main quartz)))+ (is (= "metabase-h2-quartz" (.getDataSourceName quartz)))+ (let [^WrapperConnectionPoolDataSource pool-config (.getConnectionPoolDataSource quartz)]+ (is (= 5 (.getMaxPoolSize pool-config)))+ (is (= 1 (.getMinPoolSize pool-config)))+ (is (= 1 (.getInitialPoolSize pool-config)))))+ (finally+ (DataSources/destroy ^javax.sql.DataSource (:data-source app-db))+ (DataSources/destroy ^javax.sql.DataSource (:quartz-data-source app-db)))))))++(deftest quartz-data-source-rejects-pre-pooled-test+ (testing ":create-pool? true with an already-pooled data-source throws instead of silently sharing one pool"+ (let [data-source (mdb.data-source/raw-connection-string->DataSource "jdbc:h2:mem:quartz-pre-pooled-test")+ pre-pooled (DataSources/pooledDataSource ^javax.sql.DataSource data-source)]+ (try+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"already-pooled"+ (mdb.connection/application-db :h2 pre-pooled :create-pool? true)))+ (finally+ (DataSources/destroy pre-pooled))))))++(deftest quartz-data-source-no-pool-test+ (testing "with :create-pool? false, the Quartz data source is just the raw data source itself"+ (let [data-source (mdb.data-source/raw-connection-string->DataSource "jdbc:h2:mem:quartz-no-pool-test")+ app-db (mdb.connection/application-db :h2 data-source)]+ (is (identical? data-source (:data-source app-db)))+ (is (identical? data-source (:quartz-data-source app-db))))))++(deftest quartz-data-source-respects-lock-test+ (testing "acquiring a Quartz connection blocks while the application DB write lock is held"+ (let [data-source (mdb.data-source/raw-connection-string->DataSource+ "jdbc:h2:mem:quartz-lock-test;DB_CLOSE_DELAY=-1")+ app-db (mdb.connection/application-db :h2 data-source)]+ (binding [mdb.connection/*application-db* app-db]+ (let [quartz-ds (mdb.connection/quartz-data-source)+ ^ReentrantReadWriteLock lock (:lock app-db)]+ (.. lock writeLock lock)+ (let [acquire (future+ (with-open [^Connection conn (.getConnection quartz-ds)]+ (instance? Connection conn)))]+ (try+ (is (= ::blocked (deref acquire 300 ::blocked))+ "getConnection should block while the write lock is held")+ (finally+ (.. lock writeLock unlock)))+ (is (true? (deref acquire 5000 ::timed-out))+ "getConnection should proceed once the write lock is released")))))))diff --git a/test/metabase/task/bootstrap_test.clj b/test/metabase/task/bootstrap_test.cljnew file mode 100644index 00000000000..3441118412b--- /dev/null+++ b/test/metabase/task/bootstrap_test.clj@@ -0,0 +1,70 @@+(ns metabase.task.bootstrap-test+ (:require+ [clojure.test :refer :all]+ [metabase.app-db.connection :as mdb.connection]+ [metabase.app-db.connection-pool-setup :as mdb.connection-pool-setup]+ [metabase.app-db.data-source :as mdb.data-source]+ [metabase.task.bootstrap :as task.bootstrap]+ [toucan2.connection :as t2.conn])+ (:import+ (com.mchange.v2.c3p0 DataSources)+ (java.sql Connection)+ (org.quartz.utils ConnectionProvider)))++(set! *warn-on-reflection* true)++(defn- h2-data-source ^javax.sql.DataSource [db-name]+ (mdb.data-source/raw-connection-string->DataSource (str "jdbc:h2:mem:" db-name ";DB_CLOSE_DELAY=-1")))++(deftest getConnection-ignores-bound-connection-test+ (testing "the provider fetches a fresh connection even when toucan2's *current-connectable* is bound to a Connection"+ (let [data-source (h2-data-source "quartz-provider-test")+ app-db (mdb.connection/application-db :h2 data-source)]+ (binding [mdb.connection/*application-db* app-db]+ (with-open [^Connection outer (.getConnection data-source)]+ (binding [t2.conn/*current-connectable* outer]+ (let [^ConnectionProvider provider (task.bootstrap/->ConnectionProvider)]+ (with-open [^Connection conn (.getConnection provider)]+ (is (instance? Connection conn))+ (is (not (identical? outer conn))+ "Quartz must never reuse the Connection bound to the calling thread")+ ;; also guard against a *wrapper* around the bound connection (as the old non-closeable-proxy+ ;; approach used): unwrapping must not lead back to the bound connection+ (is (not (identical? outer (.unwrap conn Connection)))+ "the Quartz connection must be a different physical connection, not a wrapper around the bound one"))+ (testing "closing the Quartz connection must not close the bound connection"+ (is (not (.isClosed outer)))))))))))++(deftest no-deadlock-when-main-pool-saturated-test+ (testing "a Quartz operation from a thread holding the main pool's last connection completes (UXW-307)"+ (let [orig-props @#'mdb.connection-pool-setup/application-db-connection-pool-props]+ (with-redefs-fn {#'mdb.connection-pool-setup/application-db-connection-pool-props+ (fn []+ (assoc (orig-props)+ "maxPoolSize" 1+ "minPoolSize" 1+ "initialPoolSize" 1+ ;; if the fix ever regresses, fail fast with a checkout exception instead of+ ;; hanging the test until the deref timeout below+ "checkoutTimeout" 3000))}+ (fn []+ (let [data-source (h2-data-source "quartz-deadlock-test")+ app-db (mdb.connection/application-db :h2 data-source :create-pool? true)]+ (try+ (binding [mdb.connection/*application-db* app-db]+ ;; saturate the main pool by checking out its only connection...+ (with-open [^Connection main-conn (.getConnection ^javax.sql.DataSource app-db)]+ ;; ...and simulate being inside a `with-transaction` block on it+ (binding [t2.conn/*current-connectable* main-conn]+ (let [^ConnectionProvider provider (task.bootstrap/->ConnectionProvider)+ result (future+ (with-open [^Connection conn (.getConnection provider)]+ (with-open [stmt (.createStatement conn)]+ (with-open [rs (.executeQuery stmt "SELECT 1")]+ (.next rs)+ (.getInt rs 1)))))]+ (is (= 1 (deref result 10000 ::deadlocked))+ "the Quartz connection must come from the dedicated pool, not the saturated main pool")))))+ (finally+ (DataSources/destroy ^javax.sql.DataSource (:data-source app-db))+ (DataSources/destroy ^javax.sql.DataSource (:quartz-data-source app-db))))))))))