The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Fix duplicate alerts/subscriptions firing in multi-pod clusters (#70474)
Ngoc Khuat5b6e3071ac51
diff --git a/resources/quartz.properties b/resources/quartz.propertiesindex 6d3e14a9d0c..830815a650f 100644--- a/resources/quartz.properties+++ b/resources/quartz.properties@@ -18,6 +18,12 @@ org.quartz.jobStore.dataSource = dborg.quartz.jobStore.isClustered = true# Added to see if we can prevent multiple firings (which should never happen). Source https://www.quartz-scheduler.org/documentation/quartz-1.8.6/configuration/ConfigJDBCJobStoreClustering.htmlorg.quartz.jobStore.clusterCheckinInterval = 20000+# Acquire the TRIGGER_ACCESS lock before selecting triggers to fire.+# Without this, trigger acquisition runs without a lock, relying on an optimistic+# UPDATE to prevent duplicate firing across cluster nodes. This has been observed+# to fail in practice (GDGT-1790), likely due to interactions between autocommit+# and Quartz's commit/retry logic.+org.quartz.jobStore.acquireTriggersWithinLock = true# By default, Quartz will fire triggers up to a minute late without considering them to be misfired; if it cannot fire# anything within that period for one reason or another (such as all threads in the thread pool being tied up), thediff --git a/src/metabase/app_db/custom_migrations/util.clj b/src/metabase/app_db/custom_migrations/util.cljindex 7bd28c5f6f6..392e8f86ebd 100644--- a/src/metabase/app_db/custom_migrations/util.clj+++ b/src/metabase/app_db/custom_migrations/util.clj@@ -5,6 +5,8 @@[metabase.classloader.core :as classloader][metabase.task.bootstrap]))+(set! *warn-on-reflection* true)+(defn- set-jdbc-backend-properties! [](metabase.task.bootstrap/set-jdbc-backend-properties! (mdb.connection/db-type)))@@ -27,7 +29,19 @@(when *allow-temp-scheduling*(classloader/the-classloader)(set-jdbc-backend-properties!)+ ;; quartz.properties sets acquireTriggersWithinLock=true to prevent duplicate trigger firing in+ ;; multi-pod clusters (GDGT-1790). However, the temp migration scheduler is single-instance and+ ;; doesn't need that lock. We must disable it here because on a fresh DB, Quartz's+ ;; StdRowLockSemaphore lazily INSERTs into QRTZ_LOCKS, and if the row already exists (e.g. from a+ ;; prior migration step), the INSERT fails. On Postgres, a failed INSERT poisons the entire+ ;; transaction, causing subsequent operations to fail with+ ;; "current transaction is aborted, commands ignored until end of transaction block".+ ;; We use a System property override because StdSchedulerFactory merges system props on top of+ ;; quartz.properties during initialize(), and we clear it immediately after so the main scheduler+ ;; later picks up the correct `true` value from quartz.properties.+ (System/setProperty "org.quartz.jobStore.acquireTriggersWithinLock" "false")(let [scheduler (qs/initialize)]+ (System/clearProperty "org.quartz.jobStore.acquireTriggersWithinLock")(when (qs/started? scheduler)(throw (ex-info "Scheduler is already started, cannot start temporary one" {})))