Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Fix duplicate alerts/subscriptions firing in multi-pod clusters (#70474)

Ngoc Khuat5b6e3071ac51
diff --git a/resources/quartz.properties b/resources/quartz.properties
index 6d3e14a9d0c..830815a650f 100644
--- a/resources/quartz.properties
+++ b/resources/quartz.properties
@@ -18,6 +18,12 @@ org.quartz.jobStore.dataSource = db
org.quartz.jobStore.isClustered = true
# Added to see if we can prevent multiple firings (which should never happen). Source https://www.quartz-scheduler.org/documentation/quartz-1.8.6/configuration/ConfigJDBCJobStoreClustering.html
org.quartz.jobStore.clusterCheckinInterval = 20000
+# Acquire the TRIGGER_ACCESS lock before selecting triggers to fire.
+# Without this, trigger acquisition runs without a lock, relying on an optimistic
+# UPDATE to prevent duplicate firing across cluster nodes. This has been observed
+# to fail in practice (GDGT-1790), likely due to interactions between autocommit
+# and Quartz's commit/retry logic.
+org.quartz.jobStore.acquireTriggersWithinLock = true
# By default, Quartz will fire triggers up to a minute late without considering them to be misfired; if it cannot fire
# anything within that period for one reason or another (such as all threads in the thread pool being tied up), the
diff --git a/src/metabase/app_db/custom_migrations/util.clj b/src/metabase/app_db/custom_migrations/util.clj
index 7bd28c5f6f6..392e8f86ebd 100644
--- a/src/metabase/app_db/custom_migrations/util.clj
+++ b/src/metabase/app_db/custom_migrations/util.clj
@@ -5,6 +5,8 @@
[metabase.classloader.core :as classloader]
[metabase.task.bootstrap]))
+(set! *warn-on-reflection* true)
+
(defn- set-jdbc-backend-properties! []
(metabase.task.bootstrap/set-jdbc-backend-properties! (mdb.connection/db-type)))
@@ -27,7 +29,19 @@
(when *allow-temp-scheduling*
(classloader/the-classloader)
(set-jdbc-backend-properties!)
+ ;; quartz.properties sets acquireTriggersWithinLock=true to prevent duplicate trigger firing in
+ ;; multi-pod clusters (GDGT-1790). However, the temp migration scheduler is single-instance and
+ ;; doesn't need that lock. We must disable it here because on a fresh DB, Quartz's
+ ;; StdRowLockSemaphore lazily INSERTs into QRTZ_LOCKS, and if the row already exists (e.g. from a
+ ;; prior migration step), the INSERT fails. On Postgres, a failed INSERT poisons the entire
+ ;; transaction, causing subsequent operations to fail with
+ ;; "current transaction is aborted, commands ignored until end of transaction block".
+ ;; We use a System property override because StdSchedulerFactory merges system props on top of
+ ;; quartz.properties during initialize(), and we clear it immediately after so the main scheduler
+ ;; later picks up the correct `true` value from quartz.properties.
+ (System/setProperty "org.quartz.jobStore.acquireTriggersWithinLock" "false")
(let [scheduler (qs/initialize)]
+ (System/clearProperty "org.quartz.jobStore.acquireTriggersWithinLock")
(when (qs/started? scheduler)
(throw (ex-info "Scheduler is already started, cannot start temporary one" {})))