The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Persistent Queue (#76352)
Nathan Voxlandb3cfb8d510a0
diff --git a/.clj-kondo/config.edn b/.clj-kondo/config.ednindex 6afe479774b..ba79b8591b2 100644--- a/.clj-kondo/config.edn+++ b/.clj-kondo/config.edn@@ -624,6 +624,9 @@metabase.model-persistence.api-test/with-persistence-setup! hooks.common/with-one-top-level-bindingmetabase.model-persistence.api-test/with-setup! hooks.common/with-one-top-level-bindingmetabase.model-persistence.test-util/with-persistence-enabled! hooks.metabase.model-persistence.test-util/with-persistence-enabled!+ metabase.mq.core/with-queue hooks.common/let-second+ metabase.mq.test-util/with-test-mq hooks.common/with-one-binding+ metabase.mq.core/def-listener! hooks.metabase.mq.impl/def-listenermetabase.models.interface/define-batched-hydration-method hooks.metabase.models.interface/define-hydration-methodmetabase.models.interface/define-simple-hydration-method hooks.metabase.models.interface/define-hydration-methodmetabase.permissions.core/allow-changing-all-users-group-members hooks.common/do*diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.ednindex 50a7908fece..23153120b3f 100644--- a/.clj-kondo/config/modules/config.edn+++ b/.clj-kondo/config/modules/config.edn@@ -1435,6 +1435,17 @@:model/Table:model/User}}+ mq+ {:team "UX West"+ :api #{metabase.mq.init}+ :uses #{analytics-interface+ app-db+ classloader+ settings+ startup+ task+ util}}+native-query-snippets{:team "Graphy":api #{metabase.native-query-snippets.apidiff --git a/.clj-kondo/src/hooks/metabase/mq/impl.clj b/.clj-kondo/src/hooks/metabase/mq/impl.cljnew file mode 100644index 00000000000..16fac00fb1a--- /dev/null+++ b/.clj-kondo/src/hooks/metabase/mq/impl.clj@@ -0,0 +1,18 @@+(ns hooks.metabase.mq.impl+ (:require+ [clj-kondo.hooks-api :as hooks]))++(defn def-listener+ "Kondo hook for `mq/def-listener`. Rewrites to a `fn` form so kondo can analyze bindings.++ Form:+ (mq/def-listener! :queue/bar [msgs] body...)"+ [{:keys [node]}]+ (let [[_ _channel-name bindings & body] (:children node)+ ;; rewrite as: (fn [bindings...] body...)+ new-node (hooks/list-node+ (list*+ (hooks/token-node 'fn)+ bindings+ body))]+ {:node new-node}))diff --git a/deps.edn b/deps.ednindex 87b7c679ec7..42288416d43 100644--- a/deps.edn+++ b/deps.edn@@ -651,7 +651,15 @@;; https://clojurians.slack.com/archives/C03S1KBA2/p1667925853699669;; An issue has been filed here https://github.com/jonase/eastwood/issues/444;; and a simple reproduction repo can be found here https://github.com/metabase/snowplow-eastwood-issue- :exclude-namespaces [metabase.analytics.snowplow]}}+ ;;+ ;; The two Quartz queue-affinity DriverDelegate shims are `gen-class` namespaces whose+ ;; only body is a `(.superRtp this ...)` call to their own exposed super method. That+ ;; method exists only on the class the namespace itself generates, so `this` can't be+ ;; type-hinted with it at compile time (the class isn't defined yet) — the reflective+ ;; call is unavoidable. Eastwood also can't reliably analyze these gen-class shims.+ :exclude-namespaces [metabase.analytics.snowplow+ metabase.mq.quartz-affinity-delegate-std+ metabase.mq.quartz-affinity-delegate-postgres]}};; Test counterpart of :eastwood. Starts with :reflection only; add more linters as we fix other eastwood;; issues in tests.diff --git a/dev/src/dev/mq_perf.clj b/dev/src/dev/mq_perf.cljnew file mode 100644index 00000000000..d6e79c99f6e--- /dev/null+++ b/dev/src/dev/mq_perf.clj@@ -0,0 +1,273 @@+(ns dev.mq-perf+ "Load/perf harness for the **Quartz** MQ backend (`metabase.mq.queue.quartz`).++ Unlike the old appdb harness this drives the real Quartz path: each published batch becomes a+ trigger on a per-queue durable job, fired by Quartz's worker pool. It therefore needs a running+ Quartz scheduler with the JDBC JobStore — i.e. run it inside a normal dev system (the dev REPL+ already has `db` + scheduler + mq up) rather than the old `start-mq!` standalone mode.++ Key thing to watch: the publish buffer coalesces messages published close together into one+ batch (one trigger carrying up to `:max-batch-messages` messages). So Quartz fires far fewer+ triggers than there are messages — the benchmarks report both message throughput and the trigger+ count / coalescing factor, since the trigger rate is what actually stresses the QRTZ_* tables.++ Transactional modes (`:mode`): every benchmark takes a `:mode` selecting which publish path in+ `metabase.mq.publish` it drives — this is what makes the QRTZ pressure differ dramatically:++ :none-no-txn — `:never` queue, published *outside* a transaction. Immediate publish through the+ sliding-window buffer, which coalesces across publishers (few, fat triggers).+ :none-in-txn — `:never` queue, published *inside* a transaction. Deferred to after-commit (in+ memory, no outbox table), then goes through the same buffer — still coalesced.+ :require — `:require` queue, published *inside* a transaction (mandatory — throws otherwise).+ Messages are written to the `queue_message_outbox` table in the business txn and+ published *directly to the backend at commit*, BYPASSING the buffer. So there is+ no cross-transaction coalescing: one committed publish ≈ one trigger. This is the+ durable path, and its trigger rate tracks the message rate — the QRTZ tables feel+ every message. (See the perf note on `def-queue!`.)++ So the same message rate produces wildly different trigger counts across modes; the benchmarks+ report `:mode`, triggers fired, and the coalescing factor so the tradeoff is visible.++ Usage (in the dev REPL, with the system started):++ (require 'dev.mq-perf)+ (dev.mq-perf/run-all!) ; sweeps every mode+ (dev.mq-perf/bench-sustained! {:duration-sec 10 :target-msgs-sec 2000 :mode :require})+ (dev.mq-perf/bench-e2e-latency! {:n 500 :msgs-sec 500 :mode :none-in-txn})+ (dev.mq-perf/bench-publish-throughput! {:n 5000 :mode :none-no-txn})"+ (:require+ [metabase.mq.core :as mq]+ [metabase.mq.listener :as listener]+ [metabase.mq.publish-buffer :as publish-buffer]+ [metabase.mq.queue.quartz :as q.quartz]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.task.core :as task]+ [metabase.util.log :as log]+ [toucan2.core :as t2])+ (:import+ (org.quartz JobKey JobListener Scheduler)+ (org.quartz.impl.matchers GroupMatcher)))++(set! *warn-on-reflection* true)++;;; --------------------------------------------- helpers ---------------------------------------------++(defn- ensure-queue!+ ([ch] (ensure-queue! ch {:transactional :try}))+ ([ch config] (when-not (q.registry/get-queue ch) (q.registry/register-queue! ch config))))++(def scenarios+ "Publish paths the harness exercises, each selecting a distinct route through+ `metabase.mq.publish/publish-collected!`. `:transactional` is the queue's declared mode; `:in-txn?`+ is whether the benchmark wraps each publish in a DB transaction. See the ns docstring for details."+ {:none-no-txn {:transactional :never :in-txn? false}+ :none-in-txn {:transactional :never :in-txn? true}+ :require {:transactional :require :in-txn? true}})++(defn- scenario-queue+ "Per-mode queue name, so the three modes never collide on config (a queue is registered once)."+ [base mode]+ (keyword "queue" (str base "-" (name mode))))++(defn- mode-config+ "Queue config for `mode`, merged with any `extra` (e.g. `:max-batch-messages`)."+ [mode extra]+ (merge {:transactional (:transactional (scenarios mode))} extra))++(defn- putter+ "Returns `(fn [msg])` publishing one message to `ch` on `mode`'s path — wrapping the publish in a DB+ transaction for the transactional modes so `:require` hits the outbox and `:none-in-txn` hits the+ after-commit defer."+ [ch mode]+ (if (:in-txn? (scenarios mode))+ (fn [msg] (t2/with-transaction [_conn] (mq/with-queue ch [q] (mq/put q msg))))+ (fn [msg] (mq/with-queue ch [q] (mq/put q msg)))))++(defn- outbox-depth+ "Rows still sitting in the transactional outbox for `ch` — should be 0 after a clean run (the+ after-commit path deletes each row once it's published). A non-zero value means publishes are being+ left for the recovery sweep, i.e. the backend couldn't keep up at commit time."+ [ch]+ (t2/count :queue_message_outbox :queue_name (name ch)))++(defn- pct [sorted p]+ (when (seq sorted)+ (nth sorted (min (dec (count sorted)) (long (* p (count sorted)))))))++(defn- stats [xs]+ (let [s (vec (sort xs))]+ {:n (count s) :min (first s) :max (peek s)+ :p50 (pct s 0.50) :p95 (pct s 0.95) :p99 (pct s 0.99)+ :avg (when (seq s) (/ (reduce + 0.0 s) (count s)))}))++(defn- fmt [x] (when x (format "%.2f" (double x))))++(defn- publish-at-rate!+ "Calls `(f i)` `n` times, pacing to ~`msgs-sec` (nil = as fast as possible)."+ [n msgs-sec f]+ (let [interval-ns (when (and msgs-sec (pos? msgs-sec)) (long (/ 1e9 msgs-sec)))+ start (System/nanoTime)]+ (dotimes [i n]+ (f i)+ (when interval-ns+ (let [target (+ start (* (long (inc i)) interval-ns))+ now (System/nanoTime)]+ (when (> target now)+ (let [sleep-ms (long (/ (- target now) 1e6))]+ (when (pos? sleep-ms) (Thread/sleep sleep-ms)))))))))++(defn- do-with-trigger-count!+ "Counts actual Quartz job executions (= triggers fired) in the mq group while `thunk` runs, via a+ temporary JobListener. Returns `[thunk-result trigger-count]`. This is the true trigger rate — not+ the listener-invocation count, which is inflated by max-batch-messages slicing."+ [thunk]+ (let [^Scheduler scheduler (task/scheduler)+ counter (atom 0)+ lname "mq-perf-trigger-counter"+ jl (reify JobListener+ (getName [_] lname)+ (jobToBeExecuted [_ _ctx])+ (jobExecutionVetoed [_ _ctx])+ (jobWasExecuted [_ _ctx _ex] (swap! counter inc)))]+ (.addJobListener (.getListenerManager scheduler) jl (GroupMatcher/jobGroupEquals "metabase.mq.queue"))+ (try+ [(thunk) @counter]+ (finally (.removeJobListener (.getListenerManager scheduler) lname)))))++(defn clean!+ "Unlisten + drop all `:queue/bench-*` queues and delete their durable Quartz jobs (and any+ leftover triggers) so a benchmark run leaves the scheduler tidy.++ Also evicts the bench queues from the quartz backend's private `ensured-jobs` process cache. That+ cache is deliberately trusted over the scheduler on the publish hot path, so without this eviction+ a *re-run* of a same-named benchmark would skip recreating the just-deleted durable job and every+ publish would throw `JobPersistenceException` (silently dropped by the publish buffer → 0 delivered)."+ []+ (doseq [ch (keys @q.registry/*queues*)+ :when (.startsWith (name ch) "bench-")]+ (try (mq/unlisten! ch) (catch Throwable _))+ (swap! q.registry/*queues* dissoc ch))+ ;; `ensured-jobs` is private to the quartz backend; reach it via its var (dev harness only).+ (swap! @#'q.quartz/ensured-jobs (fn [s] (into #{} (remove (fn [[ch _]] (.startsWith (name ch) "bench-"))) s)))+ (when-let [^Scheduler scheduler (task/scheduler)]+ (doseq [^JobKey jk (.getJobKeys scheduler (GroupMatcher/jobGroupEquals "metabase.mq.queue"))+ :when (.startsWith (.getName jk) "bench-")]+ (.deleteJob scheduler jk))))++;;; --------------------------------------------- benchmarks ---------------------------------------------++(defn bench-publish-throughput!+ "How fast can the publish side enqueue + schedule triggers? Publishes `n` single-message batches+ as fast as possible (buffer flushed at the end), measuring publish-side msgs/sec. Does not wait+ for delivery. `:mode` selects the publish path (see [[scenarios]]); the transactional modes pay a+ DB round-trip per message, so this is where their per-publish cost shows up."+ [{:keys [n mode] :or {n 5000 mode :none-no-txn}}]+ (let [ch (scenario-queue "bench-pub" mode)+ put (putter ch mode)]+ (ensure-queue! ch (mode-config mode nil))+ (let [start (System/nanoTime)]+ (publish-at-rate! n nil (fn [i] (put {:i i})))+ (publish-buffer/flush-publish-buffer!)+ (let [secs (/ (- (System/nanoTime) start) 1e9)]+ {:bench :publish-throughput :mode mode :n n+ :elapsed-sec (fmt secs) :msgs-sec (fmt (/ n secs))}))))++(defn bench-e2e-latency!+ "Publishes `n` messages at ~`msgs-sec` and measures publish->receive latency (ms) percentiles+ through the full pipeline (buffer -> quartz trigger -> worker thread -> listener)."+ [{:keys [n msgs-sec mode] :or {n 500 msgs-sec 500 mode :none-no-txn}}]+ (let [ch (scenario-queue "bench-e2e" mode)+ put (putter ch mode)+ latencies (atom [])+ received (atom 0)]+ (ensure-queue! ch (mode-config mode nil))+ (listener/batch-listen! ch (fn [msgs]+ (let [now (System/nanoTime)]+ (doseq [m msgs]+ (when-let [p (:pub-ns m)]+ (swap! latencies conj (/ (- now (double p)) 1e6)))+ (swap! received inc)))))+ (try+ (publish-at-rate! n msgs-sec (fn [_] (put {:pub-ns (System/nanoTime)})))+ (publish-buffer/flush-publish-buffer!)+ (let [deadline (+ (System/currentTimeMillis) 30000)]+ (while (and (< @received n) (< (System/currentTimeMillis) deadline))+ (Thread/sleep 50)))+ (merge {:bench :e2e-latency :mode mode :published n :received @received :target-msgs-sec msgs-sec}+ (update-vals (select-keys (stats @latencies) [:p50 :p95 :p99 :min :max :avg]) fmt))+ (finally (mq/unlisten! ch)))))++(defn bench-sustained!+ "Publishes at ~`target-msgs-sec` for `duration-sec`, then drains. Reports delivered messages/sec,+ the number of Quartz triggers actually fired, and the resulting coalescing factor (messages per+ trigger). `:mode` selects the publish path (see [[scenarios]]): the buffered modes coalesce so the+ trigger rate stays flat, while `:require` publishes each committed batch straight to the backend so+ the trigger rate tracks the message rate — compare `:triggers-sec` / `:msgs-per-trigger` across+ modes to see the QRTZ pressure tradeoff. `:outbox-remaining` should be 0 (rows deleted on publish)."+ [{:keys [duration-sec target-msgs-sec max-batch mode]+ :or {duration-sec 10 target-msgs-sec 2000 max-batch 100 mode :none-no-txn}}]+ (let [ch (scenario-queue (str "bench-sustained-mb" max-batch) mode)+ put (putter ch mode)+ received (atom 0)]+ (ensure-queue! ch (mode-config mode {:max-batch-messages max-batch}))+ (listener/batch-listen! ch (fn [msgs] (swap! received + (count msgs))))+ (try+ (let [n (* duration-sec target-msgs-sec)+ [result triggers]+ (do-with-trigger-count!+ (fn []+ (let [start (System/nanoTime)]+ (publish-at-rate! n target-msgs-sec (fn [i] (put {:i i})))+ (publish-buffer/flush-publish-buffer!)+ (let [pub-secs (/ (- (System/nanoTime) start) 1e9)+ deadline (+ (System/currentTimeMillis) 60000)]+ (while (and (< @received n) (< (System/currentTimeMillis) deadline))+ (Thread/sleep 100))+ {:pub-secs pub-secs :total-secs (/ (- (System/nanoTime) start) 1e9)}))))]+ {:bench :sustained :mode mode :target-msgs-sec target-msgs-sec :duration-sec duration-sec :max-batch max-batch+ :published n :received @received :outbox-remaining (outbox-depth ch)+ :publish-sec (fmt (:pub-secs result)) :total-sec (fmt (:total-secs result))+ :delivered-msgs-sec (fmt (/ @received (:total-secs result)))+ :triggers-fired triggers+ :triggers-sec (fmt (/ triggers (:total-secs result)))+ :msgs-per-trigger (fmt (/ @received (max 1 triggers)))})+ (finally (mq/unlisten! ch)))))++(defn start-system!+ "Headless bootstrap for a benchmark node: app DB + Quartz scheduler + MQ subsystem (no Jetty).+ Idempotent enough for a fresh process. Point `MB_DB_*` at a DEDICATED database — running several+ nodes against the same DB exercises Quartz clustering, but aiming at a live instance's DB would+ make this node also run that instance's scheduled jobs."+ []+ ((requiring-resolve 'metabase.app-db.core/setup-db!) :create-sample-content? false)+ ((requiring-resolve 'metabase.task.core/start-scheduler!))+ (require 'metabase.mq.init)+ ;; invoking the startup-logic method for ::MqStart runs mq.init/start!+ ((requiring-resolve 'metabase.startup.core/def-startup-logic!) :metabase.mq.init/MqStart)+ (log/info "MQ benchmark node started (DB + scheduler + MQ)."))++(defn run-modes!+ "Runs the sustained throughput benchmark across all three [[scenarios]] at the same message rate so+ the QRTZ-pressure tradeoff is directly comparable. `clean!`s between modes so each starts fresh.+ `:require` pays a DB round-trip per publish, so it defaults to a gentler rate."+ [{:keys [duration-sec target-msgs-sec require-msgs-sec]+ :or {duration-sec 10 target-msgs-sec 2000 require-msgs-sec 500}}]+ (mapv (fn [mode]+ (log/infof "MQ quartz perf: sustained (%s)" (name mode))+ (clean!)+ (bench-sustained! {:duration-sec duration-sec+ :target-msgs-sec (if (= mode :require) require-msgs-sec target-msgs-sec)+ :mode mode}))+ [:none-no-txn :none-in-txn :require]))++(defn run-all!+ "Runs the standard suite (publish throughput, e2e latency, then a sustained sweep across every+ transactional mode) and returns a vector of result maps. Assumes the system is already up (use+ [[start-system!]] first in a headless process)."+ []+ (log/info "MQ quartz perf: publish throughput")+ (let [r1 (bench-publish-throughput! {:n 5000})]+ (log/info "MQ quartz perf: e2e latency")+ (let [r2 (bench-e2e-latency! {:n 500 :msgs-sec 500})]+ (log/info "MQ quartz perf: sustained throughput (all modes)")+ (into [r1 r2] (run-modes! {:duration-sec 10 :target-msgs-sec 2000})))))diff --git a/dev/src/dev/mq_perf_multi.sh b/dev/src/dev/mq_perf_multi.shnew file mode 100755index 00000000000..d2a8e9e7d72--- /dev/null+++ b/dev/src/dev/mq_perf_multi.sh@@ -0,0 +1,98 @@+#!/usr/bin/env bash+# Runs multiple Quartz-MQ benchmark nodes against the same database to exercise clustering.+#+# Usage:+# ./dev/src/dev/mq_perf_multi.sh start # Start 2 nodes (default)+# ./dev/src/dev/mq_perf_multi.sh start 4 # Start 4 nodes+# ./dev/src/dev/mq_perf_multi.sh stop # Stop all running nodes+# ./dev/src/dev/mq_perf_multi.sh logs # Tail all node logs+#+# Each node starts a headless JVM (DB + Quartz scheduler + MQ, no Jetty) and runs the benchmark+# suite immediately. Because the Quartz backend is clustered via the JDBC JobStore, all nodes+# pointed at the same DB form a cluster and share trigger execution.+#+# IMPORTANT: point MB_DB_* at a DEDICATED throwaway database, e.g.:+# MB_DB_TYPE=postgres MB_DB_DBNAME=mq_bench MB_DB_USER=... ./dev/src/dev/mq_perf_multi.sh start 3+# Aiming at a live instance's DB would make these nodes also run that instance's scheduled jobs.+#+# (A synchronized "go" start across nodes is not implemented — nodes begin as soon as they boot.)++set -euo pipefail+cd "$(git rev-parse --show-toplevel)"++LOG_DIR="target/mq-perf-logs"+PID_DIR="target/mq-perf-pids"++cmd_start() {+ local num_nodes="${1:-2}"+ mkdir -p "$LOG_DIR" "$PID_DIR"++ # Clean up stale pids+ rm -f "$PID_DIR"/*.pid++ echo "Starting $num_nodes MQ benchmark nodes..."+ echo "Logs in $LOG_DIR/"+ echo++ for i in $(seq 1 "$num_nodes"); do+ local log_file="$LOG_DIR/node-$i.log"+ clojure -M:dev:ee:ee-dev -e "+ (require 'dev.mq-perf)+ (dev.mq-perf/start-system!)+ (clojure.pprint/pprint (dev.mq-perf/run-all!))+ (dev.mq-perf/clean!)+ (shutdown-agents)+ " > "$log_file" 2>&1 &+ local pid=$!+ echo "$pid" > "$PID_DIR/node-$i.pid"+ echo " Node $i started (PID $pid, log: $log_file)"+ done++ echo+ echo "Nodes are running the suite. Watch progress with:"+ echo " $0 logs"+}++cmd_stop() {+ if [ ! -d "$PID_DIR" ]; then+ echo "No nodes running."+ return+ fi++ for pid_file in "$PID_DIR"/*.pid; do+ [ -f "$pid_file" ] || continue+ local pid+ pid=$(cat "$pid_file")+ local name+ name=$(basename "$pid_file" .pid)+ if kill -0 "$pid" 2>/dev/null; then+ kill "$pid"+ echo "Stopped $name (PID $pid)"+ else+ echo "$name (PID $pid) already exited"+ fi+ rm -f "$pid_file"+ done+}++cmd_logs() {+ if [ ! -d "$LOG_DIR" ]; then+ echo "No logs found."+ return+ fi+ tail -f "$LOG_DIR"/node-*.log+}++case "${1:-}" in+ start) cmd_start "${2:-2}" ;;+ stop) cmd_stop ;;+ logs) cmd_logs ;;+ *)+ echo "Usage: $0 {start [N]|stop|logs}"+ echo+ echo " start [N] Start N benchmark nodes (default 2); each runs the suite immediately"+ echo " stop Kill all running nodes"+ echo " logs Tail all node logs"+ exit 1+ ;;+esacdiff --git a/frontend/src/metabase/nav/components/search/SearchBar/SearchBar.styled.tsx b/frontend/src/metabase/nav/components/search/SearchBar/SearchBar.styled.tsxindex 686182239a6..1433c072446 100644--- a/frontend/src/metabase/nav/components/search/SearchBar/SearchBar.styled.tsx+++ b/frontend/src/metabase/nav/components/search/SearchBar/SearchBar.styled.tsx@@ -40,7 +40,6 @@ export const SearchInputContainer = styled.div<{}return css`background-color: var(--mb-color-background_page-primary);-&:hover {background-color: var(--mb-color-background_page-secondary);}diff --git a/mage/src/mage/modules.clj b/mage/src/mage/modules.cljindex 25bb9db39a4..ce76fab3ab6 100644--- a/mage/src/mage/modules.clj+++ b/mage/src/mage/modules.clj@@ -127,6 +127,7 @@login-historymcpmetabot+ mqnotificationoauth-serverpermissionsdiff --git a/resources/migrations/064/20260624_persistent_queue.yaml b/resources/migrations/064/20260624_persistent_queue.yamlnew file mode 100644index 00000000000..13880e0f7b4--- /dev/null+++ b/resources/migrations/064/20260624_persistent_queue.yaml@@ -0,0 +1,108 @@+## Adds the transactional outbox table backing persistent queue publishes++databaseChangeLog:+ - changeSet:+ id: v64.qmoutbox1+ author: nvoxland+ comment: Create the transactional outbox table backing persistent queue publishes+ changes:+ - createTable:+ tableName: queue_message_outbox+ remarks: >-+ Transactional outbox for queue publishes. Rows are inserted inside the business+ transaction (before-commit) and handed to the queue backend after it commits, so a+ message is published iff the transaction that produced it commits. A recovery job+ republishes any rows a crash left behind.+ columns:+ - column:+ name: id+ type: BIGINT+ autoIncrement: true+ constraints:+ primaryKey: true+ nullable: false+ - column:+ name: queue_name+ type: varchar(254)+ remarks: The name of the queue this batch will be published to+ constraints:+ nullable: false+ - column:+ remarks: The timestamp of when the row was written to the outbox+ name: created_at+ type: ${timestamp_type}+ defaultValueComputed: current_timestamp+ constraints:+ nullable: false+ - column:+ remarks: JSON-encoded payload (one stored batch)+ name: payload+ type: ${text.type}+ constraints:+ nullable: false+ - column:+ name: publish_attempts+ type: int+ defaultValueNumeric: 0+ remarks: >-+ Number of consecutive times the recovery sweep has failed to publish this row.+ Drives the exponential backoff between recovery attempts; the row is never+ dropped (it is retried until it publishes).+ constraints:+ nullable: false+ - column:+ name: next_attempt_at+ type: ${timestamp_type}+ remarks: >-+ Earliest time the recovery sweep may retry this row after a failed publish+ (now + exponential backoff). Null until the first failure; until then the sweep+ gates on created_at instead.++ - changeSet:+ id: v64.qmoutbox2+ author: nvoxland+ comment: Index queue_message_outbox.created_at for the recovery sweep+ changes:+ - createIndex:+ tableName: queue_message_outbox+ indexName: idx_queue_message_outbox_created_at+ columns:+ - column:+ name: created_at++ - changeSet:+ id: v64.qmoutbox3+ author: nvoxland+ dbms: postgresql+ comment: >-+ Tune autovacuum for the high-churn outbox table.+ changes:+ - sql:+ sql: >-+ ALTER TABLE queue_message_outbox SET (+ autovacuum_vacuum_scale_factor = 0.0,+ autovacuum_vacuum_threshold = 50,+ autovacuum_analyze_scale_factor = 0.0,+ autovacuum_analyze_threshold = 50+ )+ rollback:+ - sql:+ sql: >-+ ALTER TABLE queue_message_outbox RESET (+ autovacuum_vacuum_scale_factor,+ autovacuum_vacuum_threshold,+ autovacuum_analyze_scale_factor,+ autovacuum_analyze_threshold+ )++ - changeSet:+ id: v64.qmoutbox4+ author: nvoxland+ comment: Index queue_message_outbox.next_attempt_at for the recovery sweep's due-row lookup+ changes:+ - createIndex:+ tableName: queue_message_outbox+ indexName: idx_queue_message_outbox_next_attempt_at+ columns:+ - column:+ name: next_attempt_atdiff --git a/src/metabase/analytics/prometheus.clj b/src/metabase/analytics/prometheus.cljindex 584844aa87a..fe5ab0d8383 100644--- a/src/metabase/analytics/prometheus.clj+++ b/src/metabase/analytics/prometheus.clj@@ -760,6 +760,41 @@{:description "Duration in milliseconds of used-tables extraction.";; 1ms -> 30s:buckets [1 5 10 25 50 100 250 500 1000 2500 5000 10000 30000]})+ ;; messaging metrics+ (prometheus/gauge :metabase-mq/queue-depth+ {:description "Batch count per queue by status, across all queue backends."+ :labels [:backend :channel :status]})+ (prometheus/counter :metabase-mq/queue-poll-results+ {:description "Queue poll results by outcome, across all queue backends."+ :labels [:backend :result]})+ (prometheus/counter :metabase-mq/batch-stale-recoveries+ {:description "Batches recovered from stale processing state."+ :labels [:backend :transport :channel]})+ (prometheus/counter :metabase-mq/batches-handled+ {:description "Batches handled by status."+ :labels [:transport :channel :status]})+ (prometheus/counter :metabase-mq/dedup-messages-dropped+ {:description "Messages dropped by dedup before publishing."+ :labels [:channel]})+ (prometheus/histogram :metabase-mq/handle-duration-ms+ {:description "Duration in milliseconds to process a batch."+ :labels [:transport :channel]+ :buckets [1 5 10 50 100 500 1000 5000 10000 30000]})+ (prometheus/counter :metabase-mq/messages-published+ {:description "Total messages published."+ :labels [:transport :channel]})+ (prometheus/gauge :metabase-mq/publish-buffer-depth+ {:description "Messages sitting in the publish buffer awaiting flush."+ :labels [:channel]})+ (prometheus/counter :metabase-mq/messages-received+ {:description "Individual messages delivered to handlers."+ :labels [:transport :channel]})+ (prometheus/counter :metabase-mq/batches-retried+ {:description "Queue batches that will be re-attempted, by `reason`"+ :labels [:channel :reason]})+ (prometheus/counter :metabase-mq/batches-dropped+ {:description "Queue batches permanently dropped by `reason`"+ :labels [:channel :reason]});; release dashboard metrics(prometheus/counter :metabase-sync/failures{:description "Number of sync operation failures."diff --git a/src/metabase/app_db/connection.clj b/src/metabase/app_db/connection.cljindex ce2c4edead0..575696f8f88 100644--- a/src/metabase/app_db/connection.clj+++ b/src/metabase/app_db/connection.clj@@ -139,10 +139,35 @@[](pos? *transaction-depth*))-;; Accumulates thunks to run after the outermost transaction commits. Bound to a fresh atom when the-;; outermost transaction starts (see [[do-with-transaction]]); nil outside any transaction.+;; Accumulate 0-arity thunks to run just before / just after the outermost transaction commits. Each is+;; bound to a fresh atom when the outermost transaction starts (see [[do-with-transaction]]) and shared by+;; the whole nested-transaction tree; nil outside any transaction.+(def ^:private ^:dynamic *before-commit-callbacks* nil)(def ^:private ^:dynamic *after-commit-callbacks* nil)+(def ^:dynamic *transaction-state*+ "When non-nil, an atom holding a map of arbitrary per-transaction data, shared by the whole+ nested-transaction tree and thrown away when the outermost transaction ends. Any subsystem can stash+ namespaced keys here to pass data between the transaction body and its before-/after-commit callbacks+ (e.g. the mq outbox stashes messages to insert before commit and the rows to publish after commit).+ Bound to a fresh atom at the outermost transaction boundary; nil outside any transaction."+ nil)++(defn transaction-state+ "Returns the current per-transaction [[*transaction-state*]] atom, or nil if not in a transaction."+ []+ *transaction-state*)++(defn do-before-commit+ "Run `thunk` just before the current outermost transaction commits — while the transaction is still+ open, so any DB writes it makes commit atomically with it, and a throw from it rolls the whole+ transaction back. Outside a transaction, runs `thunk` immediately. Mirror of [[do-after-commit]] for+ work that must land *inside* the committing transaction."+ [thunk]+ (if-let [callbacks *before-commit-callbacks*]+ (do (swap! callbacks conj thunk) nil)+ (thunk)))+(defn do-after-commit"Run `thunk` after the current outermost transaction commits successfully — never on rollback.Outside a transaction (autocommit), runs `thunk` immediately — the surrounding write already committed.@@ -169,32 +194,50 @@(try (thunk) (catch Throwable t (log/error t "after-commit callback failed"))))(reset! callbacks [])))-(defn- discard-after-commit-callbacks-after! [callback-count]- (when-let [callbacks *after-commit-callbacks*]+(defn- discard-callbacks-after!+ "Truncate the `callbacks` atom back to its first `n` entries, dropping any that a now-rolling-back+ nested transaction registered after its savepoint so they never fire at outer-commit time."+ [callbacks n]+ (when callbacks(swap! callbacks- (fn [callbacks]+ (fn [cbs];; copy rather than return the subvec view, which would retain the discarded callbacks (and their;; captured closures) through the backing array until the outer transaction finishes- (into [] (subvec callbacks 0 (min callback-count (count callbacks))))))))+ (into [] (subvec cbs 0 (min n (count cbs))))))))(defn- do-transaction [^java.sql.Connection connection f](letfn [(thunk [](let [savepoint (.setSavepoint connection)- callback-count (some-> *after-commit-callbacks* deref count)]+ before-count (some-> *before-commit-callbacks* deref count)+ after-count (some-> *after-commit-callbacks* deref count)+ state-snapshot (when (and *transaction-state* (> *transaction-depth* 1))+ @*transaction-state*)](try(let [result (f connection)](when (= *transaction-depth* 1)- ;; top-level transaction; post-commit side effects run after the transaction bindings unwind+ ;; top-level transaction. Run before-commit callbacks first, while the transaction is+ ;; still open, so their writes commit atomically with it. They are NOT wrapped in+ ;; try/catch — a throwing callback must propagate to the catch below and roll back.+ (loop []+ (when-let [callbacks (seq (first (reset-vals! *before-commit-callbacks* [])))]+ (doseq [cb callbacks] (cb))+ ;; a before-commit callback may register more (before- or after-commit); run those too+ (recur)))+ ;; commit; after-commit side effects run after the transaction bindings unwind(.commit connection))result)(catch Throwable txn-e- ;; the nested body failed, so its callbacks must never fire — discard them before attempting- ;; rollback, otherwise a throwing .rollback would leave them in the shared accumulator to run- ;; at outer-commit time for data that was rolled back- (when callback-count- (discard-after-commit-callbacks-after! callback-count))+ ;; the nested body failed, so its before-/after-commit callbacks must never fire — discard+ ;; those it registered before rolling back, otherwise a throwing .rollback would leave them+ ;; in the shared accumulators to run at outer-commit time for data that was rolled back+ (when after-count (discard-callbacks-after! *after-commit-callbacks* after-count))+ (when before-count (discard-callbacks-after! *before-commit-callbacks* before-count))(try(.rollback connection savepoint)+ ;; restore transaction-state to its pre-savepoint value so data the rolled-back+ ;; sub-transaction stashed is discarded and not seen by the outer commit+ (when state-snapshot+ (reset! *transaction-state* state-snapshot))(catch Exception rollback-e(throw (ex-info(str "Error rolling back after previous error: " (ex-message txn-e))@@ -247,9 +290,11 @@:else(let [outermost? (zero? *transaction-depth*)callbacks (if outermost? (atom []) *after-commit-callbacks*)- result (binding [*transaction-depth* (inc *transaction-depth*)- ;; one accumulator for the whole tree, created when the outermost transaction starts- *after-commit-callbacks* callbacks]+ result (binding [*transaction-depth* (inc *transaction-depth*)+ ;; one set of accumulators + state for the whole tree, created at the outermost txn+ *before-commit-callbacks* (if outermost? (atom []) *before-commit-callbacks*)+ *transaction-state* (if outermost? (atom {}) *transaction-state*)+ *after-commit-callbacks* callbacks](do-transaction connection f))](when outermost?(run-after-commit-callbacks! callbacks))diff --git a/src/metabase/app_db/core.clj b/src/metabase/app_db/core.cljindex 59a351f1dd8..57d32570b3e 100644--- a/src/metabase/app_db/core.clj+++ b/src/metabase/app_db/core.clj@@ -37,10 +37,12 @@application-dbdata-sourcedb-type+ do-before-commitdo-after-commitin-transaction?quoting-style- unique-identifier]+ unique-identifier+ transaction-state][mdb.connection-pool-setuprecent-activity?][mdb.data-sourcediff --git a/src/metabase/core/core.clj b/src/metabase/core/core.cljindex 82118ad7547..19b5de88d22 100644--- a/src/metabase/core/core.clj+++ b/src/metabase/core/core.clj@@ -104,6 +104,7 @@[](log/info "Metabase Shutting Down ...")(queue/stop-listeners!)+ (startup/run-shutdown-logic!)(task/stop-scheduler!)(server/stop-web-server!)(tracing/shutdown!)diff --git a/src/metabase/core/init.clj b/src/metabase/core/init.cljindex 5d7a3752330..ada7f75686e 100644--- a/src/metabase/core/init.clj+++ b/src/metabase/core/init.clj@@ -44,6 +44,7 @@[metabase.metabot.init][metabase.model-persistence.init][metabase.models.init]+ [metabase.mq.init][metabase.notification.init][metabase.oauth-server.init][metabase.parameters.init]diff --git a/src/metabase/mq/README.md b/src/metabase/mq/README.mdnew file mode 100644index 00000000000..3985286a797--- /dev/null+++ b/src/metabase/mq/README.md@@ -0,0 +1,175 @@+# metabase.mq++A persistent message queue for Metabase. Supports both durable (database-backed) and in-memory transports with a+unified API.++## Quick Start++```clojure+;; Declare the queue (its broker-side identity and properties).+(mq/def-queue! :queue/my-task {:transactional :try})++;; Register a listener (the consumer-side handler). The body receives a *vector* of messages.+(mq/def-listener! :queue/my-task+ [messages]+ (doseq [msg messages] (do-work! msg)))++;; Publish a message (inside a transaction — delivers after commit)+(t2/with-transaction [_]+ (mq/with-queue :queue/my-task [q]+ (mq/put q {:key "value"})))+```++All queue config — `:exclusive`, `:max-batch-messages`, `:dedup-fn` — belongs on `def-queue!`,+because it takes effect at publish time on every node, regardless of where listeners are+registered. `def-listener!` only wires up the consumer-side handler. A listener registration+throws if its queue hasn't been declared yet — typos are caught at startup, not at first+publish.++## Publishing Semantics++`with-queue` is designed to read like a direct publish — you write a message and move on. Under the hood, delivery is intentionally deferred through several layers:++1. **Macro body must succeed.** Messages are only enqueued if the body of `with-queue` returns normally. An exception discards them.++2. **Surrounding transaction must commit.** If the call is inside a `t2/with-transaction` block (directly or transitively), messages are held until the transaction commits. A rollback discards them. This means a message will never be delivered for a database change that didn't happen.++3. **Batching window.** After the transaction commits, messages enter a time-windowed buffer. Rapid-fire publishes to the same channel within a sliding window are coalesced into a single batch before hitting the transport layer. This amortizes per-batch overhead during bursts. If a batch can't reach the backend when the window flushes, it is handed to the durable outbox (see below) instead of being retried in memory and eventually dropped — so a non-transactional publish survives a backend outage as long as the app DB is reachable.++4. **Deduplication.** Before dispatch, duplicate messages in the same batch are removed.++## Transactional publishing (the outbox)++Every queue declares a `:transactional` mode that controls how a publish relates to the surrounding+DB transaction. The routing (see `metabase.mq.publish/publish-collected!`):++| Mode | In a transaction | Outside a transaction |+|------------|-------------------------------------------------------------------|-------------------------------------------|+| `:require` | Routed through the outbox | **Throws** — a transaction is mandatory |+| `:try` | Routed through the outbox | Published immediately\* |+| `:never` | Deferred to after-commit, but held only in memory (no outbox row) | Published immediately\* |++\* "Immediately" means handed to the publish pipeline right away — which still passes through the+~100ms coalescing buffer (see step 3 above), so it is buffered in memory briefly rather than written to+the backend synchronously. A crash within that window loses a non-transactional publish; use `:require`+or `:try`-in-a-transaction if that matters.++### Why an outbox?++For `:require`/`:try` inside a transaction, handing the message straight to the backend after the+business write is unsafe: if the node crashes after the business transaction commits but before the+publish, the message is lost. The transactional outbox (`metabase.mq.queue.outbox`, table+`queue_message_outbox` in the app DB) closes that gap:++1. **before-commit** — the messages collected for each channel are dedup'd, chunked by+ `:max-batch-messages`, encoded, and inserted as `queue_message_outbox` rows *inside the still-open+ business transaction*, so they commit atomically with the business writes. A throwing+ before-commit callback rolls the whole transaction back.+2. **after-commit** — each inserted row is published to the backend and then deleted — straight+ through, no buffering. Each row has its own try/catch: if the publish fails (backend down,+ scheduler unavailable, …) the row is **left in place** for the recovery sweep. A row is only+ deleted once its publish has succeeded.+3. **recovery** — a periodic sweep (`recover-outbox!`, driven by the Quartz job in+ `metabase.mq.task.outbox`) republishes any rows older than ~1 minute that a crash — or a failed+ after-commit publish — left behind, deleting each row once its publish succeeds. It claims rows+ with `FOR UPDATE SKIP LOCKED`, so concurrent sweeps on different nodes pick up disjoint rows. A+ publish failure is almost always transient (backend/DB connectivity), so a failing row is **never+ dropped**: its `publish_attempts` is bumped and its next retry scheduled with exponential backoff+ (`next_attempt_at`, 1m doubling up to 10m), and it is retried until it publishes.++A row for a queue that was later *removed* from the code isn't a special case: publishing to the backend+doesn't check queue existence, so the row publishes normally and is deleted; the resulting trigger then+ages out through the queue reaper (`metabase.mq.task.queue-reaper`) like any other unlistened message.++The net guarantee: **a message is published if and only if the business transaction that produced it+commits**, regardless of which backend the queue uses — the outbox table always lives in the app DB.+Delivery stays at-least-once (a crash in the after-commit window makes the sweep republish), so+listeners must be idempotent.++`:never` skips the outbox *on the happy path*: inside a transaction it still waits for commit (a+rollback discards it) but is held only in memory — no outbox row — so it stays cheap under load. It is+best-effort in the sense that a *crash* before the buffer flushes loses it. It is **not** best-effort+about backend availability: if the flush can't reach the backend, a `:never` batch falls back to the+outbox just like `:try`, so a backend outage doesn't lose messages. Use `:never` when you want to avoid+the per-publish outbox write, not when you're willing to drop messages.++### Publish-time outbox fallback (non-transactional publishes)++A publish that isn't routed through the outbox up front — `:try` outside a transaction, or any `:never`+publish — goes through the batching buffer and then straight to the backend. If that backend write+fails (scheduler down, momentary DB blip), the buffer does **not** manage its own retry/drop: it writes+the batch to `queue_message_outbox` via `outbox/insert-batch!` (leaving `next_attempt_at` null, like a+crash-orphaned row), and the same recovery sweep takes over — it picks the row up once it has aged past+`recovery-age-ms` (~1 minute), then retries with backoff, never dropping. So a handed-off batch is+delayed by up to that recovery window plus the sweep interval, not published instantly — but it is not+lost. The only remaining loss is if the outbox insert *also* fails, i.e. the app DB is down too (a total+outage), which is logged and metered as `batches-dropped{reason=outbox-handoff-failed}`.++## Message serialization++Messages must be JSON-serializable. A batch is encoded to a JSON string once, at the publish+boundary (`mq.transport/publish!` → `mq.payload/encode`), and decoded once, at the delivery+boundary (`mq.impl/deliver!` → `mq.payload/decode`). In between, backends move the opaque string+around and never look inside it — so encoding is not a backend concern, and every backend+delivers an identical shape. Decoding keywordizes map keys, but values pass through JSON:+keyword values become strings, sets become vectors, dates become strings.++## Queue semantics++| | Queue |+|---------------|----------------------------------------------------------------------|+| Delivery | At-least-once per listener |+| Retry | Yes — up to `queue-max-retries` (default 5) |+| Backing store | Quartz `QRTZ_*` tables; `queue_message_outbox` for transactional publishes |+| Backlog | Persists until processed or exhausted |+| Use when | Work must complete reliably |++## Architecture++```+User Code+ with-queue+ │+ ▼+Publish Pipeline (mq.publish)+ collect in with-queue → deduplicate → route by :transactional mode:+ immediate → 100ms coalescing buffer (mq.publish-buffer)+ outbox → queue_message_outbox row,+ published after commit (mq.queue.outbox)+ │+ ▼+Transport Dispatch (mq.transport) ← multimethod on channel namespace+ :queue → queue backends+ │+ ▼+Backends+ quartz — one-shot Quartz job per batch (push; clustered JDBC JobStore)+ memory — LinkedBlockingQueue per channel+ │+ ▼+Delivery Engine (mq.impl)+ worker pool → handle! → listener fn+```+### Idempotency++Queue listeners must be idempotent. A batch can return to `pending` after a partial failure, and the same messages will+be redelivered.++## Testing++Use `with-test-mq` from `metabase.mq.test-util` to run tests against an isolated in-memory backend.+Declare the queue at namespace top level (so it registers into every run's fresh registry), then+pass a listener map right after the binding vector. Pass `{:duplicate-delivery? true}` in the opts+map to deliver every message twice and prove your listener is idempotent.++```clojure+(mq/def-queue! :queue/my-task {:transactional :try})++(deftest my-test+ (let [processed (atom [])]+ (mq.tu/with-test-mq [ctx]+ {:queue/my-task (fn [messages] (swap! processed into messages))}+ (mq/with-queue :queue/my-task [q] (mq/put q {:id 1}))+ (mq.tu/eventually! ctx #(= 1 (count @processed)))+ (is (= [{:id 1}] @processed)))))+```diff --git a/src/metabase/mq/core.clj b/src/metabase/mq/core.cljnew file mode 100644index 00000000000..6dca4a67640--- /dev/null+++ b/src/metabase/mq/core.clj@@ -0,0 +1,65 @@+(ns metabase.mq.core+ "Unified public API for the Metabase messaging subsystem.++ **Queue** — single-consumer, at-least-once delivery. Each message is processed by exactly+ one listener and removed after successful processing. Failed messages are retried up to a+ configurable limit.++ A queue is declared once with `def-queue!` — this is the source of truth for what queues+ exist and where all queue config lives (exclusivity, batch size, dedup). The consumer-side+ handler is added with `def-listener!`. Splitting the two means publishers can route to+ queues declared anywhere and listeners can live on any node, with queue config taking+ effect at publish time on every node regardless of where listeners are registered.++ (mq/def-queue! :queue/simple-task {:transactional :try})+ (mq/def-listener! :queue/simple-task [messages]+ (doseq [msg messages] (process msg)))++ Queue-level config:+ (mq/def-queue! :queue/search-reindex {:transactional :require :exclusive true :max-batch-messages 50})+ (mq/def-listener! :queue/search-reindex [messages]+ (process-batch messages))++ See [[metabase.mq.queue.registry/def-queue!]] for the full set of queue options.++ Publishing is done using `with-queue` which binds a queue you can put messages on.+ When the body finishes successfully, the message(s) in the bound queue actually publishes the messages.++ (with-queue :queue/simple-task [q]+ (put q message))"+ (:require+ [metabase.mq.impl :as mq.impl]+ [metabase.mq.init :as mq.init]+ [metabase.mq.listener :as mq.listener]+ [metabase.mq.publish :as mq.publish]+ [metabase.mq.queue.impl :as q.impl]+ [metabase.mq.queue.memory :as q.memory]+ [metabase.mq.queue.registry :as q.registry]+ [potemkin :as p]))++(set! *warn-on-reflection* true)++(comment+ mq.impl/keep-me+ mq.init/keep-me+ mq.listener/keep-me+ mq.publish/keep-me+ q.impl/keep-me+ q.memory/keep-me+ q.registry/keep-me)++(p/import-vars+ [mq.listener+ def-listener!+ register-listeners!+ unlisten!]+ [mq.publish+ put+ with-queue]+ [mq.impl+ last-activity]+ [mq.init+ start!+ stop!]+ [q.registry+ def-queue!])diff --git a/src/metabase/mq/impl.clj b/src/metabase/mq/impl.cljnew file mode 100644index 00000000000..da5d3c6abbb--- /dev/null+++ b/src/metabase/mq/impl.clj@@ -0,0 +1,101 @@+(ns metabase.mq.impl+ "Transport-agnostic message delivery, shared by every queue backend today — and intended to be the+ code that **topics and queues share** once topics exist."+ (:require+ [metabase.analytics-interface.core :as analytics]+ [metabase.mq.listener :as listener]+ [metabase.mq.payload :as payload]+ [metabase.util.log :as log]))++(set! *warn-on-reflection* true)++;;; ------------------------------------------ Activity tracking ------------------------------------------++(def ^:private last-activity*+ "channel keyword → nanoTime of last publish or handler completion."+ (atom {}))++(defn last-activity+ "Returns the nanoTime of the last publish or handler completion for the given channel, or nil."+ [channel]+ (get @last-activity* channel))++(defn record-publish-activity!+ "Records a publish event for the given channel. Called by the publishing pipeline."+ [channel]+ (swap! last-activity* assoc channel (System/nanoTime)))++;;; --------------------------------------- Normalized deliver! results ---------------------------------------++(def no-listener+ "Marker [[deliver!]] returns when `channel` has no listener running on this node. Push backends+ (e.g. Quartz) treat it as 'defer to a node that has a listener' rather than a delivered/dropped+ message; poll backends leave the batch for the stale-recovery sweep. The message must not be+ consumed here."+ ::no-listener)++(def undecodable+ "Marker [[deliver!]] returns when a payload can't be decoded — corrupt or incompatible, so no node+ can ever deliver it. Already metered as `batches-dropped{reason=undecodable}`, so callers just drop+ the message (a poll backend acks the stored batch; a push backend completes the trigger)."+ ::undecodable)++;;; ------------------------------------------- Delivery core -------------------------------------------++(defn- run-listener!+ "Invokes `listener` with the decoded `messages` batch for `channel`.+ Returns nil on success, or the `Throwable` the listener threw."+ [channel listener messages]+ (let [labels {:transport (namespace channel) :channel (name channel)}+ start (System/nanoTime)]+ (try+ (listener messages)+ (analytics/inc! :metabase-mq/batches-handled (assoc labels :status "success"))+ nil+ (catch Exception e+ (log/error e (str "Error handling " (namespace channel) " message") labels)+ (analytics/inc! :metabase-mq/batches-handled (assoc labels :status "error"))+ e)+ (finally+ (analytics/observe! :metabase-mq/handle-duration-ms labels+ (/ (double (- (System/nanoTime) start)) 1e6))))))++(defn- decode-or-drop!+ "Decodes the opaque `payload` for `channel`. Returns the decoded messages on success. If the payload+ can't be decoded (corrupt/incompatible — something retrying can never fix) logs an error,+ increments `batches-dropped` `reason=undecodable`, and returns [[undecodable]] so callers drop the+ message instead of letting the exception escape past the retry/drop machinery."+ [channel payload]+ (try+ (payload/decode payload)+ (catch Exception e+ (log/error e "Dropping queue message with an undecodable payload"+ {:channel channel :payload-bytes (count (str payload))})+ (analytics/inc! :metabase-mq/batches-dropped {:channel (name channel) :reason "undecodable"})+ undecodable)))++(defn deliver!+ "Transport-agnostic delivery core. Decodes the opaque `payload` for `channel`, resolves the channel's listener,+ and invokes it with the decoded batch and the same metrics on every backend.++ Returns a *normalized* result the calling backend interprets; it never acks, nacks, reschedules, or+ otherwise touches backend state, so poll and push backends can share it:++ [[no-listener]] — no listener for `channel` runs on this node (the message wasn't consumed)+ [[undecodable]] — the payload is corrupt/incompatible and can never be delivered on any node+ nil — delivered successfully+ a Throwable — the listener failed; the caller retries or drops per its own policy++ Poll backends turn these into ack/nack on a stored batch ([[metabase.mq.queue.polling]]); push+ backends turn them into trigger completion/requeue/refire ([[metabase.mq.queue.quartz]])."+ [channel payload]+ (swap! last-activity* assoc channel (System/nanoTime))+ (let [messages (decode-or-drop! channel payload)]+ (if (= undecodable messages)+ undecodable+ (if-let [listener (:listener (listener/get-listener channel))]+ (do+ (analytics/inc! :metabase-mq/messages-received+ {:transport (namespace channel) :channel (name channel)} (count messages))+ (run-listener! channel listener messages))+ no-listener))))diff --git a/src/metabase/mq/init.clj b/src/metabase/mq/init.cljnew file mode 100644index 00000000000..a7f0f4ab3d5--- /dev/null+++ b/src/metabase/mq/init.clj@@ -0,0 +1,120 @@+(ns metabase.mq.init+ "Initializes the mq subsystem at startup."+ (:require+ [metabase.mq.listener :as listener]+ [metabase.mq.publish-buffer :as publish-buffer]+ [metabase.mq.quartz-affinity :as quartz-affinity]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.memory :as q.memory]+ [metabase.mq.queue.polling :as q.polling]+ [metabase.mq.queue.quartz :as q.quartz]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.settings :as mq.settings]+ [metabase.mq.task.outbox]+ [metabase.mq.task.queue-reaper]+ [metabase.startup.core :as startup]+ [metabase.task.bootstrap :as task.bootstrap]+ [metabase.util.log :as log]))++(def ^:private queue-backends+ {q.quartz/backend-id q.quartz/backend+ q.memory/backend-id q.memory/backend})++(def ^:private valid-queue-backends (set (keys queue-backends)))++;; Install the queue node-affinity Quartz DriverDelegate when Quartz's JDBC properties are set.+;; Registered at load time — `mq` depends on `task`, not the reverse, so `task.bootstrap` calls this+;; rather than referencing `mq`. install-delegate! falls back to the plain per-DB delegate if the affinity subclass can't be loaded.+(task.bootstrap/register-jdbc-property-setter! quartz-affinity/install-delegate!)++(defn- resolve-backend [label table kw-or-instance]+ (if (keyword? kw-or-instance)+ (or (get table kw-or-instance)+ (throw (ex-info (str "Unknown " label " backend: " kw-or-instance)+ {:backend kw-or-instance :valid (set (keys table))})))+ kw-or-instance))++(defn start!+ "Initializes the MQ subsystem with the given backend. Returns a handle that must be+ passed to [[stop!]] to shut down and restore the previous backend state.++ `queue-be` may be either a keyword identifier (e.g. `:queue.backend/memory`)+ or a concrete backend instance.++ Called by [[startup/def-startup-logic!]] with production backends (handle discarded;+ production shutdown uses [[startup/def-shutdown-logic!]] instead) and by+ `metabase.mq.test-util/with-test-mq` with isolated memory backends.++ Backends are set globally via `alter-var-root`. For test isolation, bind+ `listener/*listeners*` and `publish-buffer/*publish-buffer*` with fresh atoms.++ The returned handle records whether THIS call started the global worker pool and+ publish-buffer flush executor (`:owns-worker-pool?`, `:owns-buffer-flush?`). The+ matching [[stop!]] only shuts those down if this call started them — so a test+ fixture running against an already-live server can no-op the global teardown."+ [queue-be]+ (let [queue-instance (resolve-backend "queue" queue-backends queue-be)+ prev-queue-be q.backend/*backend*]+ (alter-var-root #'q.backend/*backend* (constantly queue-instance))+ (log/infof "Queue backend: %s" queue-be)+ (q.registry/register-queues!)+ (listener/register-listeners!)+ ;; With the Quartz backend, tell the affinity delegate which queues this node can handle, so it+ ;; only acquires triggers for queues we have a listener for. Queried live, so dynamic+ ;; register/unlisten is reflected. Harmless (never set) for non-Quartz backends, which have no+ ;; queue jobs in the Quartz store.+ (when (= q.quartz/backend-id (q.backend/backend-id queue-instance))+ (quartz-affinity/set-capability-fn!+ (fn [] (into #{} (map name) (listener/queue-names)))))+ (let [owns-buffer-flush? (publish-buffer/start-publish-buffer-flush!)+ owns-worker-pool? (q.polling/start-worker-pool!)]+ (q.backend/start! queue-instance)+ {:prev-queue-be prev-queue-be+ :queue-be queue-instance+ :owns-buffer-flush? owns-buffer-flush?+ :owns-worker-pool? owns-worker-pool?})))++(defn stop!+ "Shuts down the MQ subsystem.++ Called with no arguments (production): shuts down all services using the current backend vars.++ Called with a handle (the map returned by [[start!]]): additionally restores the previous+ backend state and only shuts down globals (worker pool, publish-buffer flush) if this call+ owned them. Used by `metabase.mq.test-util/with-test-mq` for test isolation — a test+ that borrows an already-running server's globals leaves them alone on teardown."+ ([]+ (publish-buffer/stop-publish-buffer-flush!)+ (when-let [qb q.backend/*backend*] (q.backend/shutdown! qb))+ (q.polling/shutdown-worker-pool!)+ (reset! listener/*listeners* {}))+ ([{:keys [prev-queue-be queue-be+ owns-buffer-flush? owns-worker-pool?]}]+ ;; Always force-drain the (currently bound) publish buffer so test fixtures that bound a+ ;; fresh `*publish-buffer*` don't lose buffered messages when their teardown runs without+ ;; owning the global flush executor. The executor itself is only shut down if THIS call+ ;; started it.+ (publish-buffer/flush-publish-buffer! true)+ (when owns-buffer-flush? (publish-buffer/stop-publish-buffer-flush!))+ (when queue-be (q.backend/shutdown! queue-be))+ (when owns-worker-pool? (q.polling/shutdown-worker-pool!))+ (reset! listener/*listeners* {})+ (alter-var-root #'q.backend/*backend* (constantly prev-queue-be))))++(defn- resolve-queue-be []+ (let [queue-be (keyword "queue.backend" (mq.settings/queue-backend))]+ (when-not (contains? valid-queue-backends queue-be)+ (throw (ex-info (str "Invalid queue backend: " queue-be+ ". Valid backends: " valid-queue-backends)+ {:backend queue-be :valid valid-queue-backends})))+ queue-be))++(defmethod startup/def-startup-logic! ::MqStart [_]+ (start! (resolve-queue-be)))++;; Run after all other startup logic so listener registration, transports, and polling threads+;; only come up once the rest of the system is ready.+(defmethod startup/startup-priority ::MqStart [_] Long/MAX_VALUE)++(defmethod startup/def-shutdown-logic! ::MqShutdown [_]+ (stop!))diff --git a/src/metabase/mq/listener.clj b/src/metabase/mq/listener.cljnew file mode 100644index 00000000000..37a34507c13--- /dev/null+++ b/src/metabase/mq/listener.clj@@ -0,0 +1,111 @@+(ns metabase.mq.listener+ "Listener registry: registration, lookup, and the `def-listener!` macro.++ A listener is the consumer-side wiring for a queue — just the handler fn. Everything else+ about a queue (its broker-side properties, batch size, dedup) is declared on the queue via+ [[metabase.mq.queue.registry/def-queue!]], which takes effect on every node regardless of+ whether a listener is registered locally. A listener for a queue requires that queue to be+ declared first."+ (:require+ [metabase.mq.queue.registry :as q.registry]+ [metabase.util.log :as log]+ [metabase.util.malli.registry :as mr]))++(set! *warn-on-reflection* true)++(def ^:dynamic *listeners*+ "channel → {:listener fn} for all channels."+ (atom {}))++(defn queue-names+ "Returns the seq of queue channel names currently registered in `*listeners*`."+ []+ (filter #(= "queue" (namespace %)) (keys @*listeners*)))++(defn get-listener+ "Returns the listener config map for `channel`, or nil if not registered."+ [channel]+ (get @*listeners* channel))++(defn- register-listener!+ "Atomically registers a listener for the given channel.++ Throws if the queue has not been declared via [[q.registry/def-queue!]] — catching missing-queue+ typos at startup rather than at first publish. Also throws if a listener is already+ registered for the channel."+ [channel listener-map]+ (when (and (= "queue" (namespace channel))+ (nil? (q.registry/get-queue channel)))+ (throw (ex-info (str "No queue declared for " channel+ " — declare it with `def-queue!` before registering a listener.")+ {:channel channel+ :known-queues (set (keys @q.registry/*queues*))})))+ (let [[old _] (swap-vals! *listeners*+ (fn [m]+ (if (contains? m channel)+ m+ (assoc m channel listener-map))))]+ (when (contains? old channel)+ (throw (ex-info (str "Listener already registered for " (namespace channel) " " (name channel))+ {:channel channel})))))++(defn batch-listen!+ "Low-level listener registration. Prefer [[def-listener!]] for production code — the macro+ is the supported way to wire a handler so it gets activated through `register-listeners!`+ at the correct point in startup.++ Use this directly only when you need a runtime-dynamic registration (e.g. plugins, ad-hoc+ tests). The listener is invoked with a vec of messages, sized up to the queue's+ `:max-batch-messages`."+ [channel listener]+ (register-listener! channel {:listener listener}))++(defn unlisten!+ "Removes the listener for a channel."+ [channel]+ (swap! *listeners* dissoc channel))++(mr/def ::channel+ [:and :keyword [:fn {:error/message "Channel must be namespaced to 'queue'"}+ #(= "queue" (namespace %))]])++(defmulti def-listener*+ "Multimethod backing [[def-listener!]]."+ {:arglists '([channel])}+ identity)++(defmacro def-listener!+ "Declares a listener for a queue.++ The queue itself must already be declared via `def-queue!` — that's where batch size,+ exclusivity, and dedup live. The listener body receives a vec of messages; for per-message+ handling write `(doseq [m messages] ...)` inside the body. Queue channels are namespaced+ `:queue/*`.++ Examples:++ (mq/def-queue! :queue/simple-task {:transactional :try})+ (mq/def-listener! :queue/simple-task [messages]+ (doseq [msg messages] (process msg)))++ (mq/def-queue! :queue/search-reindex {:transactional :require :exclusive true :max-batch-messages 50})+ (mq/def-listener! :queue/search-reindex [messages]+ (process-batch messages))"+ {:arglists '([channel bindings & body])}+ [channel bindings & body]+ `(defmethod def-listener* ~channel [~'_]+ (batch-listen! ~channel (fn [~@bindings] ~@body))))++(defn register-listeners!+ "Call all [[def-listener!]] implementations to register their listeners.+ Called at startup and in test setup (from `with-test-mq`).+ Throws on the first registration failure so broken listeners are caught early."+ []+ (doseq [[k f] (methods def-listener*)]+ (try+ (f k)+ (catch Throwable e+ (log/errorf e "Failed to register listener %s" k)+ (throw (ex-info (str "Failed to register listener " k)+ {:channel k}+ e))))))diff --git a/src/metabase/mq/memory.clj b/src/metabase/mq/memory.cljnew file mode 100644index 00000000000..82fc07c5ab9--- /dev/null+++ b/src/metabase/mq/memory.clj@@ -0,0 +1,80 @@+(ns metabase.mq.memory+ "In-memory message store for the memory queue backend. Rather than a FIFO queue of opaque+ payloads, each `layer` tracks messages individually by a monotonically-increasing id (insertion+ order) — mirroring the appdb row / redis stream-entry model — so a per-message failure count+ lives on the message itself and the oldest message is simply the lowest id. Purely in-memory and+ non-durable.++ Tests construct isolated layers with [[make-layer]]; the poll context / poll thread lives on the+ backend record, driven by [[metabase.mq.queue.polling]]. This layer only owns the storage.")++(set! *warn-on-reflection* true)++(defn make-layer+ "Creates a fresh memory layer: a `sorted-map` of message id → message (ordered by id, so the+ oldest message is first) plus the id counter."+ []+ {:messages (atom (sorted-map))+ :next-id (atom 0)})++(defonce ^{:doc "Process-wide singleton layer shared by the production memory backend."}+ default-layer+ (make-layer))++(defn publish!+ "Stores a new pending message for `channel-name` with the given opaque payload string."+ [{:keys [messages next-id]} channel-name payload]+ (let [id (swap! next-id inc)]+ (swap! messages assoc id {:id id :queue channel-name :payload payload :failures 0})))++(defn take-oldest+ "Returns the oldest stored message for `channel-name` as a `{:queue :payload :batch-id}`+ descriptor, or nil if there are none. The message stays in the store (tracked by id) until it's+ acked or dropped — the poll loop's channel-busy gate keeps it from being re-fetched while it's+ in-flight."+ [{:keys [messages]} channel-name]+ (when-let [{:keys [id payload]} (->> (vals @messages)+ (filter #(= channel-name (:queue %)))+ first)]+ {:queue channel-name :payload payload :batch-id id}))++(defn message-failures+ "Returns how many times message `id` has already failed, or nil if it's no longer stored."+ [{:keys [messages]} id]+ (:failures (get @messages id)))++(defn ack!+ "Removes message `id` from the store (on success or permanent failure)."+ [{:keys [messages]} id]+ (swap! messages dissoc id))++(defn retry!+ "Increments the failure count of message `id`, leaving it stored so it gets re-delivered."+ [{:keys [messages]} id]+ (swap! messages (fn [m] (if (contains? m id) (update-in m [id :failures] inc) m))))++(defn depths+ "Returns a `{:channel :status :count}` row per channel describing how many messages are stored."+ [{:keys [messages]}]+ (->> (vals @messages)+ (group-by :queue)+ (map (fn [[queue msgs]] {:channel (name queue) :status "pending" :count (count msgs)}))))++(defn drop-orphaned!+ "Removes stored messages whose `:queue` is not in `live-queues` (a set of channel keywords that+ currently have a registered listener). Returns the number of messages dropped. The memory backend+ is single-process and non-durable, so a message for a queue with no listener can never be+ consumed; dropping it keeps the store (and the queue-depth gauge) from growing without bound."+ [{:keys [messages]} live-queues]+ (let [[old new] (swap-vals! messages+ (fn [m]+ (into (sorted-map)+ (filter (fn [[_id msg]] (contains? live-queues (:queue msg))))+ m)))]+ (- (count old) (count new))))++(defn clear!+ "Clears all stored messages. Called by the backend's `shutdown!`."+ [{:keys [messages next-id]}]+ (reset! messages (sorted-map))+ (reset! next-id 0))diff --git a/src/metabase/mq/payload.clj b/src/metabase/mq/payload.cljnew file mode 100644index 00000000000..ea9168d689c--- /dev/null+++ b/src/metabase/mq/payload.clj@@ -0,0 +1,74 @@+(ns metabase.mq.payload+ "Canonical wire encoding for queue messages.++ A batch of messages is [[encode]]d to a JSON string once, at the publish boundary+ ([[metabase.mq.transport/publish!]]), and [[decode]]d once, at the delivery boundary+ ([[metabase.mq.impl/deliver!]]). In between, backends move the opaque string around and+ never look inside it — so storage is purely a backend concern and message shape is+ identical regardless of which backend delivered it.++ Messages must be JSON-serializable. Decoding keywordizes map keys, but values pass through+ JSON's type system: keyword/symbol values become strings, sets become vectors, dates become+ strings, etc."+ (:require+ [metabase.util.json :as json])+ (:import+ (java.time.temporal Temporal)+ (java.util Date UUID)))++(set! *warn-on-reflection* true)++(defn- json-safe-scalar?+ "True for leaf values that JSON round-trips to something meaningful."+ [v]+ (or (nil? v)+ (string? v)+ (boolean? v)+ (number? v)+ (keyword? v)+ (symbol? v)+ (char? v)+ (instance? UUID v)+ (instance? Date v)+ (instance? Temporal v)))++(defn- check-value!+ "Walks `v`, throwing if it (or anything nested in it) is not JSON-serializable to a meaningful value.+ `path` is the location within the message, carried for a helpful error."+ [path v]+ (cond+ (map? v)+ (doseq [[k mv] v]+ (when-not (or (keyword? k) (string? k) (symbol? k))+ (throw (ex-info (str "Queue message has a map key that is not JSON-serializable at " (pr-str path)+ ": keys must be a keyword, string, or symbol, got " (some-> k class .getName))+ {:path path :key-class (some-> k class)})))+ (check-value! (conj path k) mv))++ (or (sequential? v) (set? v))+ (dorun (map-indexed (fn [i e] (check-value! (conj path i) e)) v))++ (json-safe-scalar? v) nil++ :else+ (throw (ex-info (str "Queue message contains a value that is not JSON-serializable at " (pr-str path)+ ": " (some-> v class .getName))+ {:path path :value-class (some-> v class)}))))++(defn check-serializable!+ "Throws if `message` contains a value that would not JSON-round-trip to something+ meaningful. Called at publish time so such mistakes fail at the+ call site, rather than being silently corrupted on the wire and becoming undeliverable."+ [message]+ (check-value! [] message))++(defn encode+ "Serializes a vector of messages to a JSON string for storage/transport."+ ^String [messages]+ (json/encode messages))++(defn decode+ "Deserializes a JSON string produced by [[encode]] back into a vector of messages,+ keywordizing map keys."+ [payload]+ (json/decode+kw payload))diff --git a/src/metabase/mq/polling.clj b/src/metabase/mq/polling.cljnew file mode 100644index 00000000000..372081e747f--- /dev/null+++ b/src/metabase/mq/polling.clj@@ -0,0 +1,80 @@+(ns metabase.mq.polling+ "Shared polling infrastructure for appdb and memory mq backends:+ time-gated execution and polling thread lifecycle."+ (:require+ [metabase.util.log :as log])+ (:import+ (java.util.concurrent Future)))++(set! *warn-on-reflection* true)++(defn periodically!+ "Runs `f` at most once per `interval-ms`. Uses `state-atom` (an atom holding a long timestamp)+ to track the last run time. Catches and logs exceptions using `label` for context."+ [state-atom interval-ms label f]+ (let [now (System/currentTimeMillis)]+ (when (> (- now @state-atom) interval-ms)+ (reset! state-atom now)+ (try+ (f)+ (catch Exception e+ (log/error e (str "Error in " label)))))))++(def ^:private active-poll-states+ "Set of poll states that currently have a running polling thread."+ (atom #{}))++(defn make-poll-state+ "Creates a polling state map with an atom for the background process and a notify object."+ []+ {:process (atom nil)+ :running? (atom false)+ :notify (Object.)})++(defn start-polling!+ "Starts a polling thread that calls `poll-fn` in a loop, waiting on the notify object+ for `wait-ms` between iterations. If `poll-fn` returns a truthy value (indicating work+ was found), the next iteration runs immediately without waiting. Idempotent — second+ call is a no-op."+ [{:keys [process running? notify] :as poll-state} label wait-ms poll-fn]+ (when (compare-and-set! running? false true)+ (log/infof "Starting %s polling thread" label)+ (swap! active-poll-states conj poll-state)+ (reset! process+ (future+ (try+ (loop []+ (when @running?+ (let [found-work? (try+ (poll-fn)+ (catch InterruptedException e (throw e))+ (catch Exception e+ (log/errorf e "Error in %s polling loop" label)+ false))]+ (when-not found-work?+ (locking notify (.wait ^Object notify (long wait-ms)))))+ (recur)))+ (catch InterruptedException _+ (log/infof "%s polling thread interrupted" label)))))))++(defn stop-polling!+ "Stops a polling thread."+ [{:keys [process running?] :as poll-state} label]+ (swap! active-poll-states disj poll-state)+ (reset! running? false)+ (when-let [f @process]+ (reset! process nil)+ (when (instance? Future f)+ (.cancel ^Future f true))+ (log/infof "%s polling thread stopped" label)))++(defn notify!+ "Wakes the polling thread so it runs immediately."+ [{:keys [notify]}]+ (locking notify (.notifyAll ^Object notify)))++(defn notify-all!+ "Wakes all active polling threads so they run immediately."+ []+ (doseq [ps @active-poll-states]+ (notify! ps)))diff --git a/src/metabase/mq/publish.clj b/src/metabase/mq/publish.cljnew file mode 100644index 00000000000..75209c04069--- /dev/null+++ b/src/metabase/mq/publish.clj@@ -0,0 +1,125 @@+(ns metabase.mq.publish+ "Publishing pipeline: buffering, transaction support, and the `with-buffer` macro."+ (:require+ [metabase.analytics-interface.core :as analytics]+ [metabase.app-db.core :as mdb]+ [metabase.mq.impl :as mq.impl]+ [metabase.mq.payload :as payload]+ [metabase.mq.publish-buffer :as publish-buffer]+ [metabase.mq.queue.outbox :as q.outbox]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.transaction :as mq.tx]+ [metabase.util.log :as log]))++(set! *warn-on-reflection* true)++(defprotocol MessageBuffer+ "Protocol for buffering messages before publishing."+ (put [this msg]+ "Put a message on the buffer."))++(defn publish!+ "Publishes messages to a channel. Applies dedup-fn if registered, buffers,+ publishes to the appropriate backend, and records analytics."+ [channel messages]+ (let [dedup-fn (q.registry/dedup-fn channel)+ before-count (count messages)+ messages (if dedup-fn (dedup-fn messages) messages)]+ (when (and dedup-fn (< (count messages) before-count))+ (analytics/inc! :metabase-mq/dedup-messages-dropped+ {:channel (name channel)}+ (- before-count (count messages))))+ (when (seq messages)+ (mq.impl/record-publish-activity! channel)+ (publish-buffer/buffered-publish! channel messages)+ (analytics/inc! :metabase-mq/messages-published+ {:transport (namespace channel) :channel (name channel)}+ (count messages)))))++(defn flush-deferred-messages!+ "Flushes all deferred messages accumulated during a transaction.+ Called as an after-commit callback. Per-channel try/catch so one failure doesn't block others.+ Takes the transaction's `state` atom explicitly (captured when the callback was registered) because+ after-commit callbacks run after the transaction bindings unwind, when the dynamic+ [[mdb/transaction-state]] is no longer bound."+ [state]+ (let [deferred (::deferred-messages @state)]+ (doseq [[channel msgs] deferred]+ (try+ (publish! channel msgs)+ (catch Exception e+ (log/error e "Error flushing deferred messages" {:channel channel}))))))++(defn defer-in-transaction!+ "Accumulates msgs in *transaction-state* under [::deferred-messages channel].+ Registers flush-deferred-messages! as an after-commit callback once per transaction."+ [channel msgs]+ (mq.tx/accumulate-and-register!+ ::deferred-messages ::flush-registered? channel msgs+ ;; capture `state` — the after-commit callback runs after the transaction bindings unwind, when+ ;; the dynamic transaction-state is no longer bound.+ (fn [state] (mdb/do-after-commit #(flush-deferred-messages! state)))))++(defn- publish-collected!+ "Routes the messages collected by [[run-with-buffer]] for `channel` according to the queue's+ `:transactional` mode and whether a transaction is currently active:++ | mode | in a txn? | behavior |+ |--------------------|-----------|------------------------------------------------------------|+ | `:require`/`:try` | yes | transactional outbox (before-commit insert, after-commit publish) |+ | `:require` | no | throw — a transaction is mandatory |+ | `:try`/`:never` | no | publish immediately |+ | `:never` | yes | defer to after-commit (in-memory; no outbox table) |"+ [channel msgs]+ (let [mode (q.registry/transactional channel)+ in-txn? (some? (mdb/transaction-state))]+ (cond+ (and in-txn? (#{:require :try} mode)) (q.outbox/defer-transactional! channel msgs)+ (and (= mode :require) (not in-txn?)) (throw (ex-info (str "Queue " channel " is :transactional :require "+ "and must be published inside a transaction.")+ {:channel channel}))+ in-txn? (defer-in-transaction! channel msgs)+ :else (publish! channel msgs))))++(defn run-with-buffer+ "Runs `body-fn` with a MessageBuffer. On success, routes the collected messages via+ [[publish-collected!]] (transactional outbox, deferred, or immediate depending on the queue's+ `:transactional` mode). On exception, discards buffered messages and rethrows."+ [channel error-label body-fn]+ (let [buffer (atom [])+ msg-buffer (reify MessageBuffer+ (put [_ msg] (swap! buffer conj msg)))]+ (try+ (let [result (body-fn msg-buffer)+ msgs @buffer]+ (when (seq msgs)+ ;; validate serializability here before any routing. A message that JSON+ ;; can't round-trip fails here instead of being silently corrupted on the wire+ ;; and being lost.+ (run! payload/check-serializable! msgs)+ (publish-collected! channel msgs))+ result)+ (catch Exception e+ (log/error e error-label)+ (throw e)))))++(defmacro with-buffer+ "Runs the body with a MessageBuffer for the given channel.+ Messages are buffered and only published if the body completes successfully.+ Inside a transaction, messages are deferred until after commit."+ [channel [binding] & body]+ `(run-with-buffer+ ~channel+ (str "Error in " (namespace ~channel) " processing")+ (fn [~binding] ~@body)))++(defmacro with-queue+ "Runs the body with the ability to add messages to the given queue.+ Messages are buffered and only published if the body completes successfully.+ If an exception occurs, no messages are published and the exception is rethrown.++ When called inside a database transaction, messages are accumulated and published+ as a single batch after the transaction commits successfully. This prevents consumers+ from reading uncommitted data."+ [queue-name [queue-binding] & body]+ `(with-buffer ~queue-name [~queue-binding] ~@body))diff --git a/src/metabase/mq/publish_buffer.clj b/src/metabase/mq/publish_buffer.cljnew file mode 100644index 00000000000..a6c12c351b5--- /dev/null+++ b/src/metabase/mq/publish_buffer.clj@@ -0,0 +1,145 @@+(ns metabase.mq.publish-buffer+ "Publish buffering: batches rapid-fire publishes into time-windowed groups+ and flushes them on a background scheduled thread."+ (:require+ [metabase.analytics-interface.core :as analytics]+ [metabase.mq.payload :as payload]+ [metabase.mq.queue.outbox :as q.outbox]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.transport :as transport]+ [metabase.util.log :as log])+ (:import+ (java.util.concurrent Executors ScheduledExecutorService ThreadFactory TimeUnit)))++(set! *warn-on-reflection* true)++(def ^:dynamic *publish-buffer-ms*+ "Time window in ms to buffer rapid-fire publishes. 0 = immediate publish.+ Resets on every new message (sliding window)."+ 100)++(def ^:dynamic *publish-buffer-max-ms*+ "Maximum time in ms since the first message before a flush is forced. 0 = no max."+ 5000)++(def ^:dynamic *publish-buffer*+ "Accumulation buffer for rapid-fire publishes.+ channel → {:messages [...] :deadline-ms long :created-ms long}"+ (atom {}))++(defn- max-batch-size [channel]+ (q.registry/max-batch-messages channel))++(defn buffered-publish!+ "Routes messages through *publish-buffer* or publishes immediately based on `*publish-buffer-ms*`.++ The sync path (when `*publish-buffer-ms*` is 0) splits `messages` into chunks of+ `:max-batch-messages` before handing them to the transport.++ The buffered path appends to the buffer and, if buffering pushes the channel's queued+ count to or past `:max-batch-messages`, trips the channel's deadline so the next tick+ of the background flush executor drains it. The actual flush stays on the executor+ thread — publisher threads never block on backend writes.++ `:max-batch-messages` is a soft target here, not a hard cap, a fuller over-the-wire batch is cheaper than several small ones."+ [channel messages]+ (let [max-size (max-batch-size channel)]+ (if (zero? *publish-buffer-ms*)+ (doseq [chunk (partition-all max-size messages)]+ (transport/publish! channel (vec chunk)))+ (let [[_ new] (swap-vals! *publish-buffer*+ (fn [buf]+ (let [now (System/currentTimeMillis)]+ (update buf channel+ (fn [entry]+ (-> (or entry {:messages [] :deadline-ms 0 :created-ms now})+ (update :messages into messages)+ (assoc :deadline-ms (+ now *publish-buffer-ms*))))))))]+ (when (>= (count (get-in new [channel :messages])) max-size)+ ;; Buffer reached the per-channel max-batch limit. Trip its deadline so the next+ ;; background flush tick drains it. Other channels' deadlines are left untouched.+ ;; The publisher does not flush synchronously — that would block on a backend write.+ (swap! *publish-buffer*+ (fn [buf] (cond-> buf (contains? buf channel) (assoc-in [channel :deadline-ms] 1)))))))))++(defn- handoff-to-outbox!+ "A flush of `messages` on `channel` couldn't reach the backend. Hand the batch to the durable outbox for further attempts."+ [channel messages ^Exception e]+ (try+ (q.outbox/insert-batch! channel (payload/encode messages))+ (analytics/inc! :metabase-mq/batches-retried {:channel (name channel) :reason "publish-outbox-handoff"})+ (log/warnf e "Backend publish failed for %s; handed %d message(s) to the durable outbox for recovery"+ channel (count messages))+ (catch Exception oe+ (analytics/inc! :metabase-mq/batches-dropped {:channel (name channel) :reason "outbox-handoff-failed"})+ (log/errorf oe "Backend publish AND outbox handoff both failed for %s; dropping %d message(s). Publish error: %s"+ channel (count messages) (ex-message e)))))++(defn- flush-accumulation!+ "Drains accumulation entries past their deadline (or all of them when `force?`) and publishes each.+ An entry is due once its sliding `:deadline-ms` arrives or it has sat for `*publish-buffer-max-ms*`+ since its first message."+ [force?]+ (doseq [channel (keys @*publish-buffer*)]+ (let [drained (atom nil)]+ (swap! *publish-buffer*+ (fn [buf]+ (if-let [{:keys [deadline-ms created-ms] :as entry} (get buf channel)]+ (let [now (System/currentTimeMillis)]+ (if (or force?+ (and (pos? deadline-ms) (>= now deadline-ms))+ (and (pos? *publish-buffer-max-ms*) (>= now (+ created-ms *publish-buffer-max-ms*))))+ (do (reset! drained entry) (dissoc buf channel))+ buf))+ buf)))+ (when-let [{:keys [messages]} @drained]+ (try (transport/publish! channel messages)+ (catch Exception e+ (handoff-to-outbox! channel messages e)))))))++(defn flush-publish-buffer!+ "Drains accumulation entries past their deadline, handing any that can't reach the backend to the+ durable outbox. When `force?` is true (the graceful-shutdown path) every entry is drained+ regardless of deadline, so nothing buffered is lost on shutdown."+ ([] (flush-publish-buffer! false))+ ([force?]+ (doseq [[channel entry] @*publish-buffer*]+ (analytics/set-gauge! :metabase-mq/publish-buffer-depth+ {:channel (name channel)}+ (count (:messages entry))))+ (flush-accumulation! force?)))++(defonce ^:private publish-buffer-executor (atom nil))++(defn- safe-flush!+ "Runs `flush-publish-buffer!` swallowing any throwable. `scheduleAtFixedRate` silently stops+ rescheduling a task that throws an uncaught exception, which would freeze the flusher and stall+ every channel; catching here keeps the periodic task alive across transient flush errors."+ []+ (try+ (flush-publish-buffer!)+ (catch Throwable t+ (log/error t "Unexpected error in publish-buffer flush; flusher continues"))))++(defn start-publish-buffer-flush!+ "Starts a daemon thread that flushes the publish buffer every 100ms. Idempotent.+ Returns `true` if THIS call created the executor, `false` if one was already running."+ []+ (let [exec (Executors/newSingleThreadScheduledExecutor+ (reify ThreadFactory+ (newThread [_ r]+ (doto (Thread. r "mq-publish-buffer-flush")+ (.setDaemon true)))))]+ (if (compare-and-set! publish-buffer-executor nil exec)+ (do (.scheduleAtFixedRate exec ^Runnable safe-flush! 100 100 TimeUnit/MILLISECONDS)+ true)+ (do (.shutdown exec) false))))++(defn stop-publish-buffer-flush!+ "Stops the publish buffer flush thread and force-drains all remaining entries+ (graceful shutdown — buffered messages are flushed regardless of deadline)."+ []+ (when-let [^ScheduledExecutorService exec+ (first (reset-vals! publish-buffer-executor nil))]+ (.shutdownNow exec))+ (flush-publish-buffer! true))diff --git a/src/metabase/mq/quartz_affinity.clj b/src/metabase/mq/quartz_affinity.cljnew file mode 100644index 00000000000..803d63e291d--- /dev/null+++ b/src/metabase/mq/quartz_affinity.clj@@ -0,0 +1,164 @@+(ns metabase.mq.quartz-affinity+ "Node-affinity for the Quartz-backed queue backend.++ Quartz clustering has no native node-affinity: every node sharing the JDBC job store is an+ interchangeable worker, so a fired queue trigger can land on a node that has no listener for that+ queue. Rather than execute-then-bounce, we make a node simply *not acquire* triggers for queues it+ can't currently handle — they stay `WAITING` in the shared store until a capable node's acquire+ loop takes them.++ The hook is a `DriverDelegate` subclass that overrides just `selectTriggerToAcquire`+ ([[select-trigger-to-acquire]]): it re-issues Quartz's own acquire query with this node's capability+ predicate spliced into the `WHERE` clause, so uncapable queue triggers are never selected. Only the+ queue job-group ([[queue-job-group]]) is gated; every other Quartz job (sync, pulses, …) is acquired+ exactly as before."+ (:require+ [clojure.java.io :as io]+ [clojure.string :as str]+ [metabase.classloader.core :as classloader]+ [metabase.util.log :as log])+ (:import+ (java.math BigDecimal)+ (java.net URL)+ (java.sql Connection PreparedStatement ResultSet)+ (org.quartz TriggerKey)+ (org.quartz.impl.jdbcjobstore Constants StdJDBCConstants)))++(set! *warn-on-reflection* true)++(def queue-job-group+ "Quartz job group every queue job lives in. MUST match `metabase.mq.queue.quartz`'s `job-group`+ (a test asserts they agree)."+ "metabase.mq.queue")++;; A fn returning the set of queue job-names (a Quartz queue job's name is `(name channel)`) this node+;; currently has a listener for — or the sentinel :all to disable filtering. Default :all so that+;; until the mq subsystem wires this up (and as a safe fallback) acquisition behaves exactly like+;; stock Quartz.+(defonce ^:private capability-fn* (atom (constantly :all)))++(defn set-capability-fn!+ "Install the fn the acquisition filter consults for this node's currently-listenable queue names.+ Must return a set of job-name strings, or :all to disable filtering."+ [f]+ (reset! capability-fn* f))++(defn- capable [] (@capability-fn*))++(defn- sql-quote ^String [s]+ (str \' (str/replace (str s) "'" "''") \'))++(defn rewrite-acquisition-sql+ "Splice a capability predicate into acquisition `sql` so this node only acquires queue triggers+ whose `job_name` is in `capable-names` — or none of them when the set is empty. `:all` returns+ `sql` unchanged. Non-queue jobs are never filtered (the `job_group <> queue-job-group` branch is+ always true for them). Capability names are inlined (escaped), so the query's existing `?`+ parameters are untouched."+ ^String [^String sql capable-names]+ (let [clause (cond+ (= capable-names :all) nil+ (empty? capable-names) (str "(job_group <> " (sql-quote queue-job-group) ")")+ :else (str "(job_group <> " (sql-quote queue-job-group)+ " OR job_name IN ("+ (str/join ", " (map sql-quote capable-names)) "))"))]+ (if clause+ (str/replace-first sql #"(?i)\s+order\s+by" (str " AND " clause " ORDER BY"))+ sql)))++(defn select-trigger-to-acquire+ "Re-implementation of Quartz's `StdJDBCDelegate.selectTriggerToAcquire` with this node's queue-affinity+ predicate baked straight into the acquire query's `WHERE`, so a node simply never *acquires* a queue+ trigger it has no listener for. Both affinity delegates call this — `StdJDBCDelegate` (H2/MySQL) and+ `PostgreSQLDelegate` (Postgres) inherit an identical `selectTriggerToAcquire`, so nothing here is+ db-specific.++ `rtp` is the delegate's own table-prefix / sched-name substitution (its protected `rtp`). We run it+ on the stock query *first* and splice the predicate in *after*, because `rtp` is `MessageFormat`-based+ and would otherwise eat the single quotes around our inlined queue names.++ Returns a `java.util.List` of `TriggerKey` (a Clojure vector, which Quartz only iterates) — matching+ the method it overrides."+ ^java.util.List [^Connection conn no-later no-earlier max-count rtp]+ (let [max-count (int (max 1 (long max-count)))+ sql ^String (rewrite-acquisition-sql (rtp StdJDBCConstants/SELECT_NEXT_TRIGGER_TO_ACQUIRE)+ (capable))]+ (with-open [ps (doto (.prepareStatement conn sql)+ (.setMaxRows max-count)+ (.setFetchSize max-count)+ (.setString 1 Constants/STATE_WAITING)+ (.setBigDecimal 2 (BigDecimal. (str no-later)))+ (.setBigDecimal 3 (BigDecimal. (str no-earlier))))+ rs (.executeQuery ^PreparedStatement ps)]+ (loop [acc []]+ (if (and (.next ^ResultSet rs) (< (count acc) max-count))+ (recur (conj acc (TriggerKey. (.getString ^ResultSet rs Constants/COL_TRIGGER_NAME)+ (.getString ^ResultSet rs Constants/COL_TRIGGER_GROUP))))+ acc)))))++;;; ---------------------------------------------------------------------------------------------+;;; Delegate installation+;;;+;;; The affinity delegate is a `gen-class` subclass. In the uberjar it is AOT-compiled by the build;+;;; in dev there is no AOT, so we compile it at runtime into a temp dir and add that to the classpath+;;; (Metabase's classloader is dynamic). Either way the class is then loadable by name, which is how+;;; Quartz instantiates a `driverDelegateClass`.++(def ^:private delegate-class-name+ "Affinity `DriverDelegate` subclass to use per app-db type. Postgres must extend `PostgreSQLDelegate`+ (BLOB handling); H2/MySQL extend `StdJDBCDelegate`."+ {:postgres "metabase.mq.QueueAffinityPostgresDelegate"})++(def ^:private default-delegate-class-name "metabase.mq.QueueAffinityStdDelegate")++(def ^:private base-delegate-class-name+ "The plain Quartz delegate to fall back to (no affinity) if the affinity subclass can't be+ loaded/compiled — so the scheduler always starts. Mirrors Metabase's normal per-DB choice."+ {:postgres "org.quartz.impl.jdbcjobstore.PostgreSQLDelegate"})++(def ^:private default-base-delegate-class-name "org.quartz.impl.jdbcjobstore.StdJDBCDelegate")++(def ^:private delegate-ns+ {"metabase.mq.QueueAffinityPostgresDelegate" 'metabase.mq.quartz-affinity-delegate-postgres+ "metabase.mq.QueueAffinityStdDelegate" 'metabase.mq.quartz-affinity-delegate-std})++(def ^:private driver-delegate-property "org.quartz.jobStore.driverDelegateClass")++(defn- class-loadable? [^String class-name]+ (try+ (boolean (Class/forName class-name false (classloader/the-classloader)))+ (catch ClassNotFoundException _ false)))++(defn- compile-delegate! [ns-sym]+ ;; compile into a temp dir added to the (dynamic) classpath, so the gen-class is loadable by name+ (let [dir (io/file (System/getProperty "java.io.tmpdir") "mb-quartz-delegate")]+ (.mkdirs dir)+ (classloader/add-url-to-classpath! ^URL (-> dir .toURI .toURL))+ (binding [*compile-path* (str dir)]+ (compile ns-sym))))++(defn ensure-delegate-loadable!+ "Ensure the affinity `DriverDelegate` subclass for `db-type` is loadable — already AOT-compiled in+ the uberjar, runtime-compiled into the classpath in dev — and return its class name. No global side+ effects beyond adding the compiled class to the classpath (idempotent)."+ [db-type]+ (let [class-name (get delegate-class-name db-type default-delegate-class-name)]+ (when-not (class-loadable? class-name)+ (compile-delegate! (delegate-ns class-name))+ (log/infof "Compiled Quartz queue-affinity delegate %s" class-name))+ class-name))++(defn install-delegate!+ "Ensure the affinity `DriverDelegate` for `db-type` is loadable and set it as Quartz's+ `driverDelegateClass`. Must run before the scheduler initializes. If the affinity delegate can't be+ loaded/compiled for any reason, falls back to the plain per-DB delegate (node affinity disabled) so+ the scheduler still starts. Returns the installed class name."+ [db-type]+ (let [class-name (try+ (ensure-delegate-loadable! db-type)+ (catch Throwable t+ (let [fallback (get base-delegate-class-name db-type default-base-delegate-class-name)]+ (log/warn t (str "Could not install Quartz queue-affinity delegate; falling back to "+ fallback " (queue node-affinity disabled)"))+ fallback)))]+ (System/setProperty driver-delegate-property class-name)+ class-name))diff --git a/src/metabase/mq/quartz_affinity_delegate_postgres.clj b/src/metabase/mq/quartz_affinity_delegate_postgres.cljnew file mode 100644index 00000000000..14147b4eb28--- /dev/null+++ b/src/metabase/mq/quartz_affinity_delegate_postgres.clj@@ -0,0 +1,26 @@+(ns metabase.mq.quartz-affinity-delegate-postgres+ "Affinity `DriverDelegate` for Postgres: a `PostgreSQLDelegate` subclass (Metabase uses+ `PostgreSQLDelegate` on Postgres for BLOB handling) that overrides only `selectTriggerToAcquire`,+ re-issuing the acquire query with this node's queue capability spliced into its `WHERE` clause.++ Identical to `metabase.mq.quartz-affinity-delegate-std` except for the base class — keep the two in+ sync. See [[metabase.mq.quartz-affinity]]."+ (:gen-class :extends org.quartz.impl.jdbcjobstore.PostgreSQLDelegate+ :name metabase.mq.QueueAffinityPostgresDelegate+ :exposes-methods {rtp superRtp})+ (:require+ [metabase.mq.quartz-affinity :as quartz-affinity]))++(set! *warn-on-reflection* true)++(defn -selectTriggerToAcquire+ "gen-class override: acquire triggers with this node's queue-affinity predicate baked into the query+ (see [[metabase.mq.quartz-affinity/select-trigger-to-acquire]]), rather than Quartz's unfiltered scan."+ ;; `(.superRtp this ...)` is a reflective call: `superRtp` is the gen-class's own exposed super+ ;; method, so it only exists on the class this very namespace generates — `this` can't be type-hinted+ ;; with that class at its own compile time (it isn't defined yet). The reflection is therefore+ ;; unavoidable here and harmless (one call per trigger-acquire query); this shim ns is excluded from+ ;; Eastwood's reflection linter in deps.edn for the same reason.+ [this conn no-later no-earlier max-count]+ (quartz-affinity/select-trigger-to-acquire conn no-later no-earlier max-count+ (fn [sql] (.superRtp this sql))))diff --git a/src/metabase/mq/quartz_affinity_delegate_std.clj b/src/metabase/mq/quartz_affinity_delegate_std.cljnew file mode 100644index 00000000000..161f4967545--- /dev/null+++ b/src/metabase/mq/quartz_affinity_delegate_std.clj@@ -0,0 +1,28 @@+(ns metabase.mq.quartz-affinity-delegate-std+ "Affinity `DriverDelegate` for H2/MySQL: a `StdJDBCDelegate` subclass that overrides only+ `selectTriggerToAcquire`, re-issuing the acquire query with this node's queue capability spliced into+ its `WHERE` clause. See [[metabase.mq.quartz-affinity]] for the why and how.++ This is `gen-class`, so it is AOT-compiled in the uberjar and runtime-compiled in dev by+ [[metabase.mq.quartz-affinity/install-delegate!]]. Keep this namespace tiny — all logic lives in+ `metabase.mq.quartz-affinity` — and keep it in sync with `metabase.mq.quartz-affinity-delegate-postgres`+ (the two differ only in their base class)."+ (:gen-class :extends org.quartz.impl.jdbcjobstore.StdJDBCDelegate+ :name metabase.mq.QueueAffinityStdDelegate+ :exposes-methods {rtp superRtp})+ (:require+ [metabase.mq.quartz-affinity :as quartz-affinity]))++(set! *warn-on-reflection* true)++(defn -selectTriggerToAcquire+ "gen-class override: acquire triggers with this node's queue-affinity predicate baked into the query+ (see [[metabase.mq.quartz-affinity/select-trigger-to-acquire]]), rather than Quartz's unfiltered scan."+ ;; `(.superRtp this ...)` is a reflective call: `superRtp` is the gen-class's own exposed super+ ;; method, so it only exists on the class this very namespace generates — `this` can't be type-hinted+ ;; with that class at its own compile time (it isn't defined yet). The reflection is therefore+ ;; unavoidable here and harmless (one call per trigger-acquire query); this shim ns is excluded from+ ;; Eastwood's reflection linter in deps.edn for the same reason.+ [this conn no-later no-earlier max-count]+ (quartz-affinity/select-trigger-to-acquire conn no-later no-earlier max-count+ (fn [sql] (.superRtp this sql))))diff --git a/src/metabase/mq/queue/backend.clj b/src/metabase/mq/queue/backend.cljnew file mode 100644index 00000000000..3a2cd3c6857--- /dev/null+++ b/src/metabase/mq/queue/backend.clj@@ -0,0 +1,61 @@+(ns metabase.mq.queue.backend+ "Backend abstraction for the message queue system.+ Each concrete backend (appdb, memory, sync) is a record implementing `QueueBackend`.")++(set! *warn-on-reflection* true)++(defprotocol QueueBackend+ "A queue backend handles durable storage, delivery, and retry semantics for queue messages.+ Every backend is poll-loop-driven: the shared driver (`metabase.mq.queue.polling`) runs the+ four maintenance tasks on a fixed cadence and the `fetch!`/submit cycle each iteration, so every+ backend gets identical loop and respin behavior. Maintenance tasks that don't apply to a backend+ (e.g. heartbeats for the single-process memory backend) are implemented as no-ops."+ (backend-id [this]+ "Returns this backend's identifying keyword (e.g. `:queue.backend/redis`)")+ (publish! [this queue-name payload]+ "Publishes a payload to the given queue. `payload` is an opaque, already-encoded string.+ The backend stores/transports it without looking inside.")+ (fetch! [this available-queue-names]+ "Fetches up to one batch per given queue. Returns a seq of+ `{:queue :payload :batch-id}` maps (or nil/empty when there's nothing to do).")+ (queue-depths [this]+ "Returns a seq of `{:channel :status :count}` maps describing current queue depth.")+ (batch-successful! [this queue-name batch-id]+ "Marks a batch as successfully processed.")+ (failure-count [this queue-name batch-id]+ "Returns how many times the batch has already failed (0 on its first failure), or nil if the+ batch isn't owned/known by this backend — in which case the caller no-ops.")+ (retry-batch! [this queue-name batch-id]+ "Re-enqueues the batch for another delivery attempt, incrementing its stored failure count.")+ (fail-batch! [this queue-name batch-id]+ "Permanently drops / marks-failed the batch after it has exhausted its retries.")+ (recover-stale! [this stale-timeout-ms max-retries]+ "Recovers batches in-flight longer than `stale-timeout-ms` (abandoned by a crashed/stale+ consumer) — re-queuing for retry, or dropping once `max-retries` is reached. Both thresholds+ are passed by the shared driver so the policy is uniform. Returns a seq of+ `{:channel :recovered :failed}` summarizing what happened per channel. Returns nil/empty when+ nothing was recovered.")+ (run-heartbeats! [this]+ "Renews the in-flight lease on batches this node is currently processing.")+ (start! [this]+ "Starts the backend polling loop. Called once at init time.")+ (shutdown! [this]+ "Shuts down all queue resources for this backend."))++(def ^:dynamic *backend*+ "The active `QueueBackend` instance. Set by `metabase.mq.init/start!`."+ nil)++(def ^:private backend-unavailable-key ::backend-unavailable)++(defn backend-unavailable-ex+ "Build an exception marking that the *backend itself* is unreachable. `publish!` implementations+ throw this so callers can tell the two apart."+ ([msg data] (backend-unavailable-ex msg data nil))+ ([msg data cause] (ex-info msg (assoc data backend-unavailable-key true) cause)))++(defn backend-unavailable?+ "True if `e` (or anything in its cause chain) was built by [[backend-unavailable-ex]]."+ [^Throwable e]+ (boolean (some #(get (ex-data %) backend-unavailable-key)+ (take-while some? (iterate ex-cause e)))))diff --git a/src/metabase/mq/queue/impl.clj b/src/metabase/mq/queue/impl.cljnew file mode 100644index 00000000000..2eb159fdcd6--- /dev/null+++ b/src/metabase/mq/queue/impl.clj@@ -0,0 +1,29 @@+(ns metabase.mq.queue.impl+ "Queue-specific delivery policy — the queue counterpart to the transport-agnostic delivery core in [[metabase.mq.impl]]."+ (:require+ [metabase.analytics-interface.core :as analytics]+ [metabase.mq.settings :as mq.settings]+ [metabase.util.log :as log]))++(set! *warn-on-reflection* true)++(defn handle-batch-failure-policy!+ "Retry-vs-drop policy for a just-failed queue batch. `failures` is the number of attempts that have+ already failed, so the just-failed attempt makes `(inc failures)` total. `error` is the exception+ that failed the batch, carried through so the give-up is diagnosable. When `(inc failures)` reaches+ `queue-max-retries` the batch is dropped (error-logged *with the cause* + `batches-dropped`+ `reason=delivery-exhausted`) by calling `on-drop`; otherwise `batches-retried` `reason=delivery` is+ emitted, a concise retry line is logged, and `on-retry` re-enqueues it."+ [channel failures error on-retry on-drop]+ (let [max-retries (mq.settings/queue-max-retries)]+ (if (>= (inc failures) max-retries)+ (do+ (log/error error "Dropping queue batch after exhausting retries"+ {:channel channel :max-retries max-retries})+ (analytics/inc! :metabase-mq/batches-dropped {:channel (name channel) :reason "delivery-exhausted"})+ (on-drop))+ (do+ (log/warnf "Queue batch for %s failed attempt %d of %d; will retry with backoff."+ channel (inc failures) max-retries)+ (analytics/inc! :metabase-mq/batches-retried {:channel (name channel) :reason "delivery"})+ (on-retry)))))diff --git a/src/metabase/mq/queue/memory.clj b/src/metabase/mq/queue/memory.cljnew file mode 100644index 00000000000..09a106ecb74--- /dev/null+++ b/src/metabase/mq/queue/memory.clj@@ -0,0 +1,76 @@+(ns metabase.mq.queue.memory+ "In-memory queue backend. Delegates message storage to a shared memory layer (an id-keyed map,+ like appdb's table) and the poll loop to the shared [[metabase.mq.queue.polling]] driver. Each+ `MemoryQueueBackend` carries a reference to its layer so tests can construct isolated instances.++ Being single-process and non-durable, the in-memory backend has no crashed-consumer recovery,+ no cross-node lease to heartbeat, and nothing to clean up, so those maintenance hooks are+ no-ops; only the depth gauge does real work."+ (:require+ [metabase.mq.listener :as listener]+ [metabase.mq.memory :as memory]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.polling :as q.polling]+ [metabase.util.log :as log]))++(set! *warn-on-reflection* true)++(def backend-id+ "This backend's identifier, registered with `metabase.mq.init`; its `name` labels metrics."+ :queue.backend/memory)++(defrecord MemoryQueueBackend [layer poll-context]+ q.backend/QueueBackend++ (backend-id [_this] backend-id)++ (publish! [_this queue-name payload]+ (memory/publish! layer queue-name payload))++ (fetch! [_this available-queues]+ (into [] (keep #(memory/take-oldest layer %)) available-queues))++ (queue-depths [_this]+ (memory/depths layer))++ ;; The batch-id is the layer-unique message id, so the queue isn't needed to resolve it.+ (batch-successful! [_this _queue-name batch-id]+ (memory/ack! layer batch-id))++ (failure-count [_this _queue-name batch-id]+ (memory/message-failures layer batch-id))++ (retry-batch! [_this _queue-name batch-id]+ (memory/retry! layer batch-id))++ (fail-batch! [_this _queue-name batch-id]+ (memory/ack! layer batch-id))++ ;; No crashed-consumer recovery (single process), but this periodic maintenance hook is reused to+ ;; reclaim messages for queues that have no listener: on a single-process, non-durable backend such+ ;; messages can never be delivered, so leaving them would grow the store and the depth gauge without+ ;; bound (unlike a persistent, cross-node backend, where another node may own the listener).+ (recover-stale! [_this _stale-timeout-ms _max-retries]+ (let [dropped (memory/drop-orphaned! layer (set (listener/queue-names)))]+ (when (pos? dropped)+ (log/debugf "Dropped %d orphaned in-memory message(s) for queues with no listener" dropped)))+ nil)++ (run-heartbeats! [_this] nil)++ (start! [this]+ (q.polling/start! this poll-context "Memory" 50))++ (shutdown! [_this]+ (q.polling/stop! poll-context "Memory")+ (memory/clear! layer)))++(defn make-backend+ "Constructs a `MemoryQueueBackend` with its own poll context. With no args, wraps the+ process-wide `memory/default-layer`. Tests can pass their own layer for isolation."+ ([] (make-backend memory/default-layer))+ ([layer] (->MemoryQueueBackend layer (q.polling/make-poll-context))))++(def backend+ "Singleton `MemoryQueueBackend` backed by `memory/default-layer`."+ (make-backend))diff --git a/src/metabase/mq/queue/outbox.clj b/src/metabase/mq/queue/outbox.cljnew file mode 100644index 00000000000..954e5672479--- /dev/null+++ b/src/metabase/mq/queue/outbox.clj@@ -0,0 +1,209 @@+(ns metabase.mq.queue.outbox+ "Transactional outbox for queue publishes.++ When a queue declared `:transactional :require`/`:try` is published to inside a DB transaction,+ the messages are not handed straight to the backend (which would be lost if the node crashed+ after the business transaction committed but before the publish). Instead:++ 1. before-commit — the accumulated messages are dedup'd, chunked by the queue's+ `:max-batch-messages`, encoded, and inserted into the `queue_message_outbox` table *inside+ the business transaction*. They commit atomically with the business writes.+ 2. after-commit — each inserted row is published to the backend and then deleted from the+ outbox. No waiting/buffering — straight through.+ 3. recovery — a periodic sweep ([[recover-outbox!]]) republishes any rows a crash left behind+ (rows older than [[recovery-age-ms]]), covering the window between commit and the+ after-commit publish.++ This makes a message published iff the business transaction that produced it commits, regardless+ of which backend the queue ultimately uses — the outbox table always lives in the app DB."+ (:require+ [metabase.analytics-interface.core :as analytics]+ [metabase.app-db.core :as mdb]+ [metabase.mq.payload :as payload]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.transaction :as mq.tx]+ [metabase.mq.transport :as transport]+ [metabase.util.log :as log]+ [toucan2.core :as t2])+ (:import+ (java.sql Timestamp)+ (java.time Instant)))++(set! *warn-on-reflection* true)++(def ^:private recovery-age-ms+ "Outbox rows older than this are assumed orphaned by a crash and republished by [[recover-outbox!]].+ The normal after-commit path publishes and deletes its rows immediately, well within this window,+ so the sweep only ever picks up rows a crash left behind."+ (* 60 1000))++(def ^:private recovery-page-size+ "How many outbox rows [[recover-outbox!]] claims per transaction (one keyset-paginated page)."+ 50)++(def ^:private recovery-retry-base-ms+ "Base delay before the recovery sweep retries a row after its first failed publish. Each further+ consecutive failure doubles the delay ([[retry-delay-ms]]), capped at [[recovery-retry-max-ms]]."+ (* 60 1000))++(def ^:private recovery-retry-max-ms+ "Cap on the exponential backoff between failed recovery publishes, so a row that keeps failing is+ still retried at least this often. Rows are never dropped — a committed message is retried until+ it publishes — so this bounds how long a transient backend outage delays delivery."+ (* 10 60 1000))++(defn retry-delay-ms+ "Backoff (ms) before the recovery sweep retries a row on its 1-based `attempts`-th consecutive+ failed publish: 1m, 2m, 4m, 8m … doubling, capped at [[recovery-retry-max-ms]]. Public for testing."+ [attempts]+ (long (min recovery-retry-max-ms+ (* recovery-retry-base-ms (Math/pow 2 (dec attempts))))))++(defn- for-update-clause+ "FOR UPDATE clause for the recovery sweep: SKIP LOCKED on Postgres/MySQL so concurrent sweeps on+ different nodes claim disjoint rows; plain FOR UPDATE on H2."+ []+ (if (#{:postgres :mysql} (mdb/db-type))+ [:update :skip-locked]+ [:update]))++(defn insert-batch!+ "Insert one already-encoded batch `payload` for `channel` as a `queue_message_outbox` row and return+ its primary key."+ [channel payload]+ (t2/insert-returning-pk! :queue_message_outbox+ {:queue_name (name channel)+ :payload payload}))++(defn insert-outbox-rows!+ "before-commit callback: for every channel buffered in `*transaction-state*`, apply the queue's+ dedup-fn, chunk by `:max-batch-messages`, encode, and insert one `queue_message_outbox` row per+ chunk — inside the still-open business transaction. Records the inserted rows+ back into the transaction state under `::rows` so the after-commit callback can publish and delete them."+ []+ (when-let [state (mdb/transaction-state)]+ (let [by-channel (::messages @state)+ rows (vec+ (for [[channel msgs] by-channel+ :let [dedup (q.registry/dedup-fn channel)+ deduped (if dedup (dedup msgs) msgs)]+ :when (seq deduped)+ chunk (partition-all (q.registry/max-batch-messages channel) deduped)+ :let [payload (payload/encode (vec chunk))+ id (insert-batch! channel payload)]]+ {:id id :channel channel :payload payload}))]+ (swap! state assoc ::rows rows))))++(defn publish-outbox-rows!+ "after-commit callback: publish each row inserted by [[insert-outbox-rows!]] to the backend, then+ delete the successfully-published rows in a single batched DELETE. Per-row try/catch — on failure+ the row is left for [[recover-outbox!]] to republish (at-least-once). Takes the transaction's `state`+ atom explicitly (captured when the callback was registered) because after-commit callbacks run after+ the transaction bindings unwind, when the dynamic [[mdb/transaction-state]] is no longer bound."+ [state]+ (let [published-ids (reduce+ (fn [ids {:keys [id channel payload]}]+ (try+ (transport/publish-encoded! channel payload)+ (conj ids id)+ (catch Exception e+ (log/error e "Error publishing outbox row; recovery sweep will retry"+ {:channel channel :outbox-id id})+ ids)))+ []+ (::rows @state))]+ (when (seq published-ids)+ (t2/delete! :queue_message_outbox :id [:in published-ids]))))++(defn defer-transactional!+ "Routes `msgs` for `channel` through the transactional outbox. Accumulates them per channel in+ `*transaction-state*` and, once per transaction, registers [[insert-outbox-rows!]] as a+ before-commit callback and [[publish-outbox-rows!]] as an after-commit callback. Must be called+ inside a transaction."+ [channel msgs]+ (mq.tx/accumulate-and-register!+ ::messages ::registered? channel msgs+ (fn [state]+ ;; capture `state` in the after-commit closure — it runs after the transaction bindings unwind,+ ;; when the dynamic transaction-state is no longer bound (the before-commit insert still runs while+ ;; the transaction is open, so it reads the dynamic var directly).+ (mdb/do-before-commit insert-outbox-rows!)+ (mdb/do-after-commit #(publish-outbox-rows! state)))))++(defn- bump-failed-row+ "Records one failed recovery publish into `acc`: bump the row's `publish_attempts` and schedule its+ next retry at `now` + exponential backoff ([[retry-delay-ms]]). The row is never dropped — a+ publish failure is almost always transient (backend/DB connectivity), and the outbox's guarantee is+ that a committed message is eventually delivered — so it is retried until it publishes."+ [acc ^Instant now {:keys [id queue_name publish_attempts]} ^Exception e]+ (let [next-attempts (inc publish_attempts)+ delay-ms (retry-delay-ms next-attempts)]+ (log/warn e "Failed to publish queue outbox row during recovery; will retry with backoff"+ {:queue queue_name :outbox-id id :publish-attempts next-attempts :retry-delay-ms delay-ms})+ (analytics/inc! :metabase-mq/batches-retried {:channel queue_name :reason "outbox-recovery"})+ (update acc :bumps conj {:id id :next-attempt-at (Timestamp/from (.plusMillis now delay-ms))})))++(defn- recover-page!+ "Runs one transaction of the recovery sweep over up to [[recovery-page-size]] *due* rows, in id order+ starting after `after-id` (keyset pagination). Publishes each row independently: published rows are+ deleted; rows that hit a *message-specific* failure are bumped and backed off ([[bump-failed-row]]).++ Returns `[recovered next-after-id]`, where `next-after-id` is the id to resume the next page from, or+ nil when the page was empty (no more due rows) OR the backend was found unavailable (stop the sweep)."+ [after-id]+ (t2/with-transaction [_conn]+ (let [now (Instant/now)+ now-ts (Timestamp/from now)+ rows (t2/query {:select [:id :queue_name :payload :publish_attempts]+ :from [:queue_message_outbox]+ :where [:and+ [:> :id after-id]+ [:or+ [:and [:= :next_attempt_at nil]+ [:< :created_at (Timestamp/from (.minusMillis now recovery-age-ms))]]+ [:<= :next_attempt_at now-ts]]]+ :order-by [[:id :asc]]+ :limit recovery-page-size+ :for (for-update-clause)})+ {:keys [recover-ids bumps backend-down?]}+ (reduce (fn [acc {:keys [id queue_name payload] :as row}]+ (try+ (transport/publish-encoded! (keyword "queue" queue_name) payload)+ (update acc :recover-ids conj id)+ (catch Exception e+ (if (q.backend/backend-unavailable? e)+ ;; backend is down — stop now (leave this row untouched) rather than bumping+ ;; every remaining row and hammering a backend we already know is unavailable.+ (reduced (assoc acc :backend-down? true))+ (bump-failed-row acc now row e)))))+ {:recover-ids [] :bumps [] :backend-down? false}+ rows)]+ ;; published rows are removed; message-specific failures have their attempt count bumped and next retry scheduled.+ (when (seq recover-ids) (t2/delete! :queue_message_outbox :id [:in recover-ids]))+ (doseq [{:keys [id next-attempt-at]} bumps]+ (t2/update! :queue_message_outbox :id id+ {:publish_attempts [:+ :publish_attempts [:inline 1]]+ :next_attempt_at next-attempt-at}))+ (when backend-down?+ (log/info "Outbox recovery: backend unavailable, remaining rows retry next run"))+ ;; nil next-after-id stops the sweep: no more due rows, or the backend is down.+ [(count recover-ids) (when (and (not backend-down?) (seq rows)) (:id (last rows)))])))++(defn recover-outbox!+ "Republishes outbox rows a crash left behind — rows older than [[recovery-age-ms]] (the normal+ after-commit path deletes its rows immediately).++ Rows that hit a message-specific failure are never dropped — the outbox's guarantee is that a+ committed message is eventually delivered — so they are retried forever (with backoff). If the backend+ itself is unavailable the sweep stops after the first such failure and lets the next scheduled run+ retry, rather than thrashing the whole backlog. The sweep never revisits rows within a sweep.++ Returns the number of rows successfully republished."+ []+ (loop [total 0+ after-id 0]+ (let [[recovered next-after-id] (recover-page! after-id)]+ (if next-after-id+ (recur (long (+ total recovered)) (long next-after-id))+ total))))diff --git a/src/metabase/mq/queue/polling.clj b/src/metabase/mq/queue/polling.cljnew file mode 100644index 00000000000..3653013e0a1--- /dev/null+++ b/src/metabase/mq/queue/polling.clj@@ -0,0 +1,227 @@+(ns metabase.mq.queue.polling+ "Shared poll driver for queue backends. Every poll-based backend (appdb, memory, redis) runs+ the same loop: four maintenance tasks on a fixed cadence, then a `fetch!`/submit pass that+ respins immediately when work was found. The per-backend differences (how a batch is fetched,+ what each maintenance task does) live behind the `QueueBackend` protocol, so the loop+ and its rate-limiting / respin behavior exist in exactly one place."+ (:require+ [metabase.analytics-interface.core :as analytics]+ [metabase.mq.impl :as mq.impl]+ [metabase.mq.listener :as listener]+ [metabase.mq.polling :as mq.polling]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.impl :as q.impl]+ [metabase.mq.settings :as mq.settings]+ [metabase.util.log :as log])+ (:import+ (java.util.concurrent Callable ExecutorService Executors Future)))++(set! *warn-on-reflection* true)++;;; ------------------------------- Worker pool & in-flight (busy) tracking -------------------------------++(def ^:private active-handlers+ "channel keyword → {:future Future, :gen Object}. Tracks which channels have an in-flight delivery+ on the worker pool, so a channel is delivered by at most one worker at a time (and so publishers+ know not to wake the loop for a channel that's already being processed)."+ (atom {}))++(def ^:private worker-pool+ "Shared thread pool that runs message delivery off the poll thread, so a slow listener doesn't+ stall polling."+ (atom nil))++(defn channel-busy?+ "Returns true if `channel` currently has an in-flight delivery on the worker pool."+ [channel]+ (contains? @active-handlers channel))++(defn busy-channels+ "Returns the set of channels that currently have an in-flight delivery."+ []+ (set (keys @active-handlers)))++(defn start-worker-pool!+ "Starts the shared worker pool for non-blocking message delivery. Idempotent.+ Returns `true` if THIS call created the pool, `false` if one was already running."+ []+ (let [pool (Executors/newCachedThreadPool)]+ (if (compare-and-set! worker-pool nil pool)+ true+ (do (.shutdown ^ExecutorService pool) false))))++(defn shutdown-worker-pool!+ "Cancels all active handler futures and shuts down the worker pool."+ []+ (doseq [[_ {:keys [^Future future]}] @active-handlers]+ (when future+ (.cancel future true)))+ (reset! active-handlers {})+ (when-let [^ExecutorService pool @worker-pool]+ (.shutdownNow pool)+ (reset! worker-pool nil)))++(defn- handle-batch-failure!+ "Decides retry vs. permanent failure for a just-failed batch based on its prior failure count, then+ tells the backend to re-enqueue or drop it. No-ops if the backend no longer owns/knows the batch+ (`failure-count` returns nil). `error` is the exception that failed the batch, passed through for+ diagnostic logging."+ [backend channel batch-id error]+ (when-let [failures (q.backend/failure-count backend channel batch-id)]+ (q.impl/handle-batch-failure-policy! channel failures error+ #(q.backend/retry-batch! backend channel batch-id)+ #(q.backend/fail-batch! backend channel batch-id))))++(defn- deliver-and-ack!+ "Worker-thread body for one fetched batch: run the shared delivery core, then ack or nack the stored+ batch based on the outcome. A delivered or undecodable batch is acked (removed from the store); a+ failed batch runs the retry/drop policy; a batch whose listener has vanished is left in place for+ the stale-recovery sweep to re-drive later. `batch-id`/`backend` are nil in tests that only exercise+ the busy-slot bookkeeping."+ [channel payload batch-id backend]+ (let [result (mq.impl/deliver! channel payload)]+ (cond+ (= mq.impl/no-listener result) nil+ (instance? Throwable result) (when batch-id (handle-batch-failure! backend channel batch-id result))+ :else (when batch-id (q.backend/batch-successful! backend channel batch-id)))))++(defn submit-delivery!+ "Submits one fetched batch to the worker pool for non-blocking delivery. Returns true if submitted,+ false if `channel` already has an in-flight delivery (enforcing at most one worker per channel).+ Uses `bound-fn` to convey dynamic bindings to the worker thread.++ The busy-check and slot claim are atomic via a single `swap-vals!`. A per-submission generation+ object prevents a completed future's cleanup from clobbering a resubmission that won the slot+ between the finally and the dissoc."+ [channel payload batch-id backend]+ (let [gen (Object.)+ [old _] (swap-vals! active-handlers+ (fn [handlers]+ (if (contains? handlers channel)+ handlers+ (assoc handlers channel {:future nil :gen gen}))))]+ (if (contains? old channel)+ false+ (let [^Callable task (bound-fn []+ (try+ (deliver-and-ack! channel payload batch-id backend)+ (finally+ ;; Only remove if this generation still owns the slot+ (swap! active-handlers+ (fn [handlers]+ (if (identical? gen (:gen (get handlers channel)))+ (dissoc handlers channel)+ handlers)))+ (mq.polling/notify-all!))))+ f (try+ (.submit ^ExecutorService @worker-pool task)+ (catch Throwable t+ ;; The slot was already claimed above. If the submit itself fails (e.g. the+ ;; worker pool isn't running / was shut down, so the deref or .submit throws),+ ;; release the slot we claimed before propagating — otherwise the channel stays+ ;; marked busy forever and the queue silently stops delivering.+ (swap! active-handlers+ (fn [handlers]+ (if (identical? gen (:gen (get handlers channel)))+ (dissoc handlers channel)+ handlers)))+ (throw t)))]+ ;; Only set the future if this generation still owns the slot+ (swap! active-handlers+ (fn [handlers]+ (if (identical? gen (:gen (get handlers channel)))+ (assoc-in handlers [channel :future] f)+ handlers)))+ true))))++(def ^:private stale-check-interval-ms (* 60 1000))+(def ^:private heartbeat-interval-ms (* 2 60 1000))+(def ^:private depth-gauge-interval-ms (* 30 1000))++(def ^:private stale-processing-timeout-ms+ "How long a batch can sit in-flight (delivered but unacked / 'processing') before it's assumed+ to belong to a crashed consumer and is recovered."+ (* 10 60 1000))++(defn- report-depths!+ "Emits a `:metabase-mq/queue-depth` gauge for each `{:channel :status :count}` the backend+ reports. Centralized here so backends only produce depth data, not metrics."+ [backend backend-name]+ (doseq [{:keys [channel status count]} (q.backend/queue-depths backend)]+ (analytics/set-gauge! :metabase-mq/queue-depth {:backend backend-name :channel channel :status status} count)))++(defn- run-recover-stale!+ "Runs the backend's stale-batch recovery and emits the resulting metrics + logs. The backend+ returns a seq of `{:channel :recovered :failed}` (batches re-queued for retry vs. dropped after+ exhausting retries); doing the analytics/logging here keeps them consistent across backends."+ [backend backend-name]+ (when-let [results (seq (q.backend/recover-stale! backend stale-processing-timeout-ms+ (mq.settings/queue-max-retries)))]+ (let [total-recovered (reduce + 0 (keep :recovered results))+ total-failed (reduce + 0 (keep :failed results))]+ (when (pos? total-recovered)+ (log/warnf "Recovered %d stale batch(es)" total-recovered))+ (when (pos? total-failed)+ (log/warnf "Dropped %d stale batch(es) that exhausted retries" total-failed))+ (doseq [{:keys [channel recovered failed]} results]+ (when (and recovered (pos? recovered))+ (analytics/inc! :metabase-mq/batch-stale-recoveries+ {:backend backend-name :transport "queue" :channel channel} recovered))+ (when (and failed (pos? failed))+ (analytics/inc! :metabase-mq/batches-dropped+ {:channel (name channel) :reason "stale-recovery-exhausted"} failed))))))++(defn notify-on-publish!+ "Wakes the poll loop for `poll-context` so it picks up a freshly published message — unless+ `channel` is already being processed, in which case the loop re-checks when that finishes.+ Called by the transport right after `publish!` (the caller reads the backend's `:poll-context`).++ No-ops when `poll-context` is nil: push backends (e.g. Quartz) have no poll loop to wake — their+ `publish!` already arranges delivery — so they carry no poll context."+ [poll-context channel]+ (when (and poll-context (not (channel-busy? channel)))+ (mq.polling/notify! (:poll-state poll-context))))++(defn make-poll-context+ "Creates the per-backend poll state: a unique instance `:id`, the polling thread handle, and the+ rate-limit atoms for each maintenance task. The `:id` identifies this instance when claiming+ work (the appdb row owner / the redis consumer name). Backends hold one of these and pass it to+ [[start!]]; `publish!` wakes the loop via `(mq.polling/notify! (:poll-state ctx))`."+ []+ {:id (str (random-uuid))+ :poll-state (mq.polling/make-poll-state)+ :last-stale-check-ms (atom 0)+ :last-heartbeat-ms (atom 0)+ :last-depth-gauge-ms (atom 0)})++(defn- poll-iteration!+ "One iteration: run each maintenance task if its interval has elapsed, then fetch and submit+ one batch per available queue. Returns true if any work was found so the polling+ thread respins immediately."+ [backend {:keys [last-stale-check-ms last-heartbeat-ms last-depth-gauge-ms]}]+ (let [backend-name (name (q.backend/backend-id backend))]+ (mq.polling/periodically! last-stale-check-ms stale-check-interval-ms "stale recovery"+ #(run-recover-stale! backend backend-name))+ (mq.polling/periodically! last-heartbeat-ms heartbeat-interval-ms "heartbeat"+ #(q.backend/run-heartbeats! backend))+ (mq.polling/periodically! last-depth-gauge-ms depth-gauge-interval-ms "depth gauge"+ #(report-depths! backend backend-name))+ (let [found? (boolean+ (when-let [available (seq (remove channel-busy? (listener/queue-names)))]+ (when-let [batches (seq (q.backend/fetch! backend available))]+ (doseq [{:keys [queue payload batch-id]} batches]+ (submit-delivery! queue payload batch-id backend))+ true)))]+ (analytics/inc! :metabase-mq/queue-poll-results {:backend backend-name :result (if found? "work" "empty")})+ found?)))++(defn start!+ "Starts the polling thread for `backend`, driving [[poll-iteration!]] every `wait-ms` (and+ immediately whenever it finds work or `publish!` calls `notify!`). Idempotent."+ [backend poll-context label wait-ms]+ (mq.polling/start-polling! (:poll-state poll-context) label wait-ms #(poll-iteration! backend poll-context)))++(defn stop!+ "Stops the polling thread for `poll-context`."+ [poll-context label]+ (mq.polling/stop-polling! (:poll-state poll-context) label))diff --git a/src/metabase/mq/queue/quartz.clj b/src/metabase/mq/queue/quartz.cljnew file mode 100644index 00000000000..1c0d8242238--- /dev/null+++ b/src/metabase/mq/queue/quartz.clj@@ -0,0 +1,254 @@+(ns metabase.mq.queue.quartz+ "Push-based queue backend built on Quartz.++ Every declared queue gets one durable Quartz job (keyed by the queue name); each published batch+ is a one-shot *trigger* on that job, carrying the encoded payload in the trigger's data map.+ Quartz's clustered JDBC JobStore (the `QRTZ_*` tables) is the durable, cross-node queue and+ Quartz's worker threads *push* delivery — there is no poll loop, no storage table of our own, and+ no lease/heartbeat. The poll-shaped `QueueBackend` hooks (`fetch!`, `recover-stale!`,+ `run-heartbeats!`, ack/nack, depth) are therefore no-ops; their responsibilities are owned by+ Quartz instead:++ - delivery — the scheduler fires the queue's job on a worker thread, once per trigger+ - retry — a failed batch reschedules itself (a fresh trigger) with backoff up to+ `queue-max-retries`+ - durability — the JDBC JobStore persists the trigger until it fires successfully+ - crash recovery — `requestRecovery` re-fires a job interrupted by a node crash on another node+ (at-least-once)++ Like the other backends this delivers at-least-once, so listeners must be idempotent.++ Exclusive queues: a queue declared `{:exclusive true}` uses a job class annotated+ `@DisallowConcurrentExecution`, which Quartz honors cluster-wide — at most one batch for that+ queue executes at a time across the whole cluster (one thread on one node). Because all of a+ queue's messages are triggers on a *single* per-queue job, the annotation (which keys on job+ identity) serializes them. Non-exclusive queues use a plain job class and run concurrently up to+ the Quartz thread pool. This is mutual exclusion, not ordering — Quartz does not fire blocked+ triggers in submission order."+ (:require+ [clojurewerkz.quartzite.jobs :as jobs]+ [clojurewerkz.quartzite.triggers :as triggers]+ [metabase.analytics-interface.core :as analytics]+ [metabase.mq.impl :as mq.impl]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.impl :as q.impl]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.task.core :as task]+ [metabase.util.log :as log])+ (:import+ (java.util Date)+ (org.quartz JobDetail JobExecutionContext JobExecutionException Scheduler SimpleScheduleBuilder)))++(set! *warn-on-reflection* true)++(def backend-id+ "This backend's identifier, registered with `metabase.mq.init`; its `name` labels metrics."+ :queue.backend/quartz)++(def ^:private job-group+ "Quartz group every queue job/trigger lives in, so they're easy to isolate from the rest of the+ scheduler's jobs."+ "metabase.mq.queue")++;; Job-data keys. All values are stored as strings so the data map round-trips through the JDBC+;; JobStore regardless of the `org.quartz.jobStore.useProperties` setting.+(def ^:private data-queue "queue")+(def ^:private data-payload "payload")+(def ^:private data-attempt "attempt")++(defn retry-delay-ms+ "Backoff (ms) before redelivering a failed batch on its 1-based `attempt`-th retry: 1s, 2s, 4s …+ capped at 60s so a permanently-flaky listener doesn't push redelivery arbitrarily far out.+ Redefable in tests that want immediate retries."+ [attempt]+ (min 60000 (* 1000 (long (Math/pow 2 (dec attempt))))))++(def ^:private ^Long refire-backoff-ms+ "Brief pause before asking Quartz to refire this job in place after a failed hand-off.+ Refiring with no delay would hot-loop the worker thread while the DB is down"+ 10000)++(defn- backoff-before-refire!+ []+ (try+ (Thread/sleep refire-backoff-ms)+ (catch InterruptedException _+ (.interrupt (Thread/currentThread)))))++(defn- queue-job-key [queue]+ (jobs/key (name queue) job-group))++(defn- ignore-misfire-schedule+ "One-shot simple schedule with `MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICY`. Queue triggers use this+ so a trigger no node with a listener has picked up yet stays `WAITING` and always-acquirable (rather than+ being re-dated or dropped by Quartz's misfire recovery) until a capable node acquires it — or the+ age reaper ([[metabase.mq.task.queue-reaper]]) removes it. It also keeps such triggers out of the+ misfire-scan queries entirely."+ ^SimpleScheduleBuilder []+ (.withMisfireHandlingInstructionIgnoreMisfires (SimpleScheduleBuilder/simpleSchedule)))++(defn- schedule-message-trigger!+ "Adds a one-shot trigger to `queue`'s durable job (which must already exist) that fires at `start-at`,+ carrying `payload` and the retry `attempt` counter in the trigger's data map."+ [^Scheduler scheduler queue payload attempt ^Date start-at]+ (.scheduleJob scheduler+ (triggers/build+ (triggers/with-identity (triggers/key (str (random-uuid)) job-group))+ (triggers/for-job (queue-job-key queue))+ (triggers/start-at start-at)+ (triggers/with-schedule (ignore-misfire-schedule))+ (triggers/using-job-data {data-payload payload+ data-attempt (str attempt)}))))++(defn- requeue-no-listener!+ "Handles a fired batch for a queue with no listener running on *this* node — with node-affinity, the+ rare race where the listener was torn down between acquisition and execution.++ We must *re-schedule* not refire: a refire re-runs on THIS node and bypasses acquisition, so it would+ loop here until node shutdown and never reach a capable node."+ [^Scheduler scheduler queue payload attempt]+ (analytics/inc! :metabase-mq/batches-retried {:channel (name queue) :reason "no-listener"})+ (log/debugf "Queue %s has no listener on this node; re-queuing for a node that does." queue)+ (schedule-message-trigger! scheduler queue payload attempt (Date.)))++(defn- deliver-batch!+ "Shared job body. Reads the merged job+trigger data and dispatches on the delivery result:++ - no listener for the queue is running on *this* node - have to reschedule.+ - the listener threw - reschedule a fresh trigger (attempt+1, backoff), or drop once+ `queue-max-retries` is reached.+ - delivered (or an undecodable payload that was dropped) — nothing more to do.++ The body only returns normally (letting Quartz complete the one-shot trigger) once the message has+ been delivered or its retry/requeue is *durably committed*. If instead our own hand-off fails —+ the reschedule can't be written because the scheduler's DB is momentarily unavailable — we throw a+ `JobExecutionException` with `refireImmediately`, so Quartz re-runs this exact job now rather than treating+ the one-shot as done and dropping the batch. The goal is to leave no window where Quartz considers the job finished+ but the message hasn't been handed off. A refire re-runs with the same job data, so `attempt` is+ unchanged — a refire doesn't burn the retry budget.++ Rescheduling uses `(.getScheduler ctx)` rather than the dynamic `(task/scheduler)`, which isn't bound+ on Quartz's worker threads; in a cluster the retry lands in the shared JDBC store so any node can+ pick it up."+ [^JobExecutionContext ctx]+ (let [data (.getMergedJobDataMap ctx)+ queue (keyword "queue" (.getString data data-queue))+ payload (.getString data data-payload)+ attempt (Integer/parseInt (or (.getString data data-attempt) "0"))]+ (try+ (let [result (mq.impl/deliver! queue payload)]+ (cond+ (= mq.impl/no-listener result)+ (requeue-no-listener! (.getScheduler ctx) queue payload attempt)++ (instance? Throwable result)+ (let [next-attempt (inc attempt)]+ (q.impl/handle-batch-failure-policy!+ queue attempt result+ #(schedule-message-trigger! (.getScheduler ctx) queue payload next-attempt+ (Date. (long (+ (System/currentTimeMillis) (retry-delay-ms next-attempt)))))+ (constantly nil)))))+ ;; Delivered, or the retry/requeue is durably committed. Return nil so Quartz completes the+ ;; one-shot trigger.+ nil+ (catch Throwable t+ ;; We couldn't hand the message off (couldn't schedule the retry, etc.). Don't let Quartz drop+ ;; the fired one-shot — have it re-run this job instead.+ (backoff-before-refire!)+ (throw (doto (JobExecutionException. t) (.setRefireImmediately true)))))))++(task/defjob ^{:doc "Delivers a queue batch (non-exclusive queues; runs concurrently)."}+ QueueMessageJob+ [ctx]+ (deliver-batch! ctx))++(task/defjob ^{org.quartz.DisallowConcurrentExecution true+ :doc "Delivers a batch for an :exclusive queue — at most one execution per queue cluster-wide."}+ ExclusiveQueueMessageJob+ [ctx]+ (deliver-batch! ctx))++(def ^:private ensured-jobs+ "Per-process cache of queues whose durable Quartz job has been created, keyed by+ `[queue exclusive?]`, so we don't re-issue `addJob` on every publish. Safe as a process-lifetime+ cache: production runs one persistent (JDBC) scheduler whose durable jobs survive restarts, and a+ change to a queue's `:exclusive` flag re-ensures under the new key ([[ensure-queue-job!]] then+ upgrades a concurrent job to exclusive, but never downgrades)."+ (atom #{}))++(defn- exclusive-job?+ "True if `job` is the `@DisallowConcurrentExecution` (exclusive) job class."+ [^JobDetail job]+ (= ExclusiveQueueMessageJob (.getJobClass job)))++(defn- ensure-queue-job!+ "Idempotently creates `queue`'s durable job — a `@DisallowConcurrentExecution` job when+ `exclusive?`, otherwise a plain concurrent one. The job carries no payload (messages are triggers+ on it), so it is durable (persists across the gaps between messages) and requests recovery+ (re-fires interrupted work after a crash).++ When the job already exists, exclusivity is treated as *sticky*: a concurrent job is upgraded to+ exclusive, but an exclusive job is never downgraded to concurrent. This keeps a node that+ disagrees about a queue's `:exclusive` flag (e.g. mid rolling deploy, where nodes run different+ code) from weakening another node's cluster-wide mutual-exclusion guarantee by flipping the+ shared job class back and forth.+ If we ever want to actually change a queue to be non-exclusive, we will need a way to handle that."+ [^Scheduler scheduler queue exclusive?]+ (when-not (contains? @ensured-jobs [queue exclusive?])+ (let [existing (.getJobDetail scheduler (queue-job-key queue))]+ (when (or (nil? existing)+ (and exclusive? (not (exclusive-job? existing))))+ (.addJob scheduler+ (jobs/build+ (jobs/of-type (if exclusive? ExclusiveQueueMessageJob QueueMessageJob))+ (jobs/with-identity (queue-job-key queue))+ (jobs/store-durably)+ (jobs/request-recovery)+ (jobs/using-job-data {data-queue (name queue)}))+ true)))+ (swap! ensured-jobs conj [queue exclusive?])))++(defrecord QuartzQueueBackend []+ q.backend/QueueBackend+ (backend-id [_this] backend-id)++ (publish! [_this queue payload]+ ;; A nil scheduler (e.g. MB_DISABLE_SCHEDULER, or during shutdown) must be a hard failure, not a+ ;; silent no-op: the transactional outbox relies on a thrown exception to keep its row for the+ ;; recovery sweep, and the publish buffer relies on it to retry / loudly drop. Returning nil+ ;; here would silently lose the message.+ (if-let [scheduler (task/scheduler)]+ (try+ (ensure-queue-job! scheduler queue (q.registry/exclusive? queue))+ (schedule-message-trigger! scheduler queue payload 0 (Date.))+ (catch Throwable t+ (throw (q.backend/backend-unavailable-ex+ (format "Failed to publish to queue %s: the Quartz scheduler/store is unavailable." queue)+ {:queue queue :backend backend-id} t))))+ (throw (q.backend/backend-unavailable-ex+ (format "Cannot publish to queue %s: the Quartz task scheduler is not running (is MB_DISABLE_SCHEDULER set?)."+ queue)+ {:queue queue :backend backend-id}))))++ ;; Push backend — Quartz drives everything below, so these poll-driver hooks are never invoked+ ;; (the poll loop is never started) and exist only to satisfy the protocol.+ (start! [_this])+ (shutdown! [_this])+ (fetch! [_this _available-queues] nil)+ (queue-depths [_this] nil)+ (batch-successful! [_this _queue-name _batch-id] nil)+ (failure-count [_this _queue-name _batch-id] nil)+ (retry-batch! [_this _queue-name _batch-id] nil)+ (fail-batch! [_this _queue-name _batch-id] nil)+ (recover-stale! [_this _stale-timeout-ms _max-retries] nil)+ (run-heartbeats! [_this] nil))++(defn make-backend+ "Constructs a `QuartzQueueBackend`. Stateless — all queue state lives in Quartz — so the+ production [[backend]] singleton and any test-constructed instances are interchangeable."+ []+ (->QuartzQueueBackend))++(def backend+ "Singleton instance of the Quartz queue backend."+ (make-backend))diff --git a/src/metabase/mq/queue/registry.clj b/src/metabase/mq/queue/registry.cljnew file mode 100644index 00000000000..bdf0b00a6ce--- /dev/null+++ b/src/metabase/mq/queue/registry.clj@@ -0,0 +1,175 @@+(ns metabase.mq.queue.registry+ "Queue registry. Holds [[*queues*]] (declared queues and their broker-side properties) and+ the [[def-queue!]] macro for populating it.++ Queues exist independently of any listener so publishers can route to them from anywhere+ in the cluster. Backends use this registry at startup to pre-arrange broker-side resources+ (e.g. RabbitMQ queue declarations) for the full set of known queues."+ (:require+ [metabase.util.malli.humanize :as mu.humanize]+ [metabase.util.malli.registry :as mr]))++(set! *warn-on-reflection* true)++(def default-max-batch-messages+ "Default `:max-batch-messages` for a queue that doesn't declare one. This is a *soft* target for how+ many messages are coalesced into one stored batch (and handed to the handler per invocation), not a+ hard cap: the async coalescing buffer will happily exceed it under load, since we'd rather send one+ fuller over-the-wire batch than several small ones. Listeners must therefore not assume a batch is+ no larger than this."+ 100)++(def transactional-modes+ "Valid values for a queue's required `:transactional` setting.++ `:require` — publishes must happen inside a DB transaction (throws otherwise). Messages are+ written to the `queue_message_outbox` table as part of that transaction and only+ handed to the backend after it commits — so a message is published iff the+ business write that produced it commits.+ `:try` — same outbox behavior when a transaction is active; publishes immediately when not.+ `:never` — never use the outbox table; publish immediately (deferred to after-commit when in+ a transaction, but kept only in memory)."+ #{:require :try :never})++(mr/def :metabase.mq.queue/queue-name+ [:and :qualified-keyword+ [:fn {:error/message "queue name must be namespaced to 'queue' (e.g. :queue/my-task)"}+ #(= "queue" (namespace %))]])++(mr/def :metabase.mq.queue/queue-config+ [:map {:closed true}+ [:transactional (into [:enum {:error/message (str "must be one of " transactional-modes)}]+ transactional-modes)]+ [:exclusive {:optional true} :boolean]+ [:max-batch-messages {:optional true} pos-int?]+ [:dedup-fn {:optional true} fn?]])++(def ^:dynamic *queues*+ "queue-name (keyword) → config map for every declared queue.++ Test fixtures rebind this to a fresh atom so test-defined queues don't leak across+ scenarios. Production code uses the root binding."+ (atom {}))++(defn get-queue+ "Returns the config map for `queue-name`, or nil if no such queue has been declared."+ [queue-name]+ (get @*queues* queue-name))++(defn queue-names+ "Returns the seq of currently-declared queue names."+ []+ (keys @*queues*))++(defn- validate-config!+ [queue-name config]+ (when-let [error (mr/explain :metabase.mq.queue/queue-config config)]+ (throw (ex-info (str "Invalid config for queue " queue-name ": " (mu.humanize/humanize error))+ {:queue queue-name :config config :errors (mu.humanize/humanize error)}))))++(defn register-queue!+ "Atomically registers `config` for `queue-name`. Re-registering with identical config is+ a no-op (handy for repeated `register-queues!` calls in tests); mismatched config throws.++ `config` must satisfy [[:metabase.mq.queue/queue-config]]. Invalid config throws."+ [queue-name config]+ (let [config (or config {})]+ (validate-config! queue-name config)+ (let [[old _] (swap-vals! *queues*+ (fn [m] (if (contains? m queue-name)+ m+ (assoc m queue-name config))))]+ (when-let [existing (get old queue-name)]+ (when (not= existing config)+ (throw (ex-info (str "Queue " queue-name " is already registered with different config.")+ {:queue queue-name+ :existing existing+ :new config})))))))++(defn exclusive?+ "Returns true if `queue-name` is declared with `:exclusive true`."+ [queue-name]+ (boolean (:exclusive (get @*queues* queue-name))))++(defn max-batch-messages+ "Returns the `:max-batch-messages` declared for `queue-name`, or [[default-max-batch-messages]]."+ [queue-name]+ (or (:max-batch-messages (get @*queues* queue-name)) default-max-batch-messages))++(defn dedup-fn+ "Returns the `:dedup-fn` declared for `queue-name`, or nil if none."+ [queue-name]+ (:dedup-fn (get @*queues* queue-name)))++(defn transactional+ "Returns the `:transactional` mode (`:require`/`:try`/`:never`) declared for `queue-name`, or nil+ if no such queue has been declared."+ [queue-name]+ (:transactional (get @*queues* queue-name)))++(defn exclusive-queue-names+ "Returns the set of queue names (as strings) declared with `:exclusive true`."+ []+ (into #{}+ (comp (filter (fn [[_k v]] (:exclusive v)))+ (map (fn [[k _v]] (name k))))+ @*queues*))++(defmulti def-queue*+ "Multimethod backing [[def-queue!]]. Each `(def-queue! :queue/foo …)` adds a method here;+ [[register-queues!]] iterates them at startup to populate `*queues*`."+ {:arglists '([queue-name])}+ identity)++(defmacro def-queue!+ "Declares a queue with its broker-side configuration. Queues exist independently of any+ listener — a publisher can route to a queue declared anywhere in the codebase, and a+ listener for that queue can live on a different node.++ Config is required and must include `:transactional`.++ Required config key:+ `:transactional` — `:require` / `:try` / `:never`; see [[transactional-modes]]. Controls+ whether publishes are routed through the transactional+ `queue_message_outbox` so a message is published iff the business+ transaction that produced it commits.+ SEE PERFORMANCE NOTE BELOW++ Optional config keys:+ `:exclusive` — when true, at most one batch for this queue is in-flight cluster-wide.+ `:max-batch-messages` — soft target batch size for this queue (defaults to+ [[default-max-batch-messages]]). Used at publish time (coalescing) and+ consume time (slicing).+ `:dedup-fn` — `messages -> messages` applied at publish time to drop duplicates+ from a batch before it is buffered.++ These are all properties of the queue itself — they take effect at publish time, on every+ node, regardless of whether a listener is registered locally.++ PERFORMANCE NOTE ON `:transactional`:+ When messages are sent transactionally (required or try with an active transaction), the batched messages are+ IMMEDIATELY sent as part of the transaction boundary. It does not use the sliding-time-window publish buffer+ that non-transactional delivery can use to batch messages across transactions under high load.+ If you are sending messages which have a risk of generating a storm of messages, consider using `:never` to+ improve performance but at the expense of potentially losing messages.++ Examples:++ (mq/def-queue! :queue/simple-task {:transactional :try})++ (mq/def-queue! :queue/search-reindex {:transactional :require :exclusive true :max-batch-messages 50})"+ {:arglists '([queue-name config])}+ [queue-name & [config]]+ `(defmethod def-queue* ~queue-name [~'_]+ (register-queue! ~queue-name ~config)))++(defn register-queues!+ "Realizes every [[def-queue!]] declaration into [[*queues*]]. Called at startup before+ the backend is started and before listeners are registered, so backends can pre-arrange+ broker-side resources for the full set of known queues."+ []+ (doseq [[k f] (methods def-queue*)]+ (try+ (f k)+ (catch Throwable e+ (throw (ex-info (str "Failed to register queue " k) {:queue k} e))))))diff --git a/src/metabase/mq/settings.clj b/src/metabase/mq/settings.cljnew file mode 100644index 00000000000..9eebda7ff40--- /dev/null+++ b/src/metabase/mq/settings.clj@@ -0,0 +1,33 @@+(ns metabase.mq.settings+ (:require+ [metabase.settings.core :refer [defsetting]]+ [metabase.util.i18n :refer [deferred-tru]]))++(defsetting queue-max-retries+ (deferred-tru "Maximum number of times a failed queue message will be retried before being dropped.")+ :type :integer+ :default 5+ :visibility :internal+ :encryption :no+ :export? false)++(defsetting queue-backend+ (deferred-tru "Which queue backend to use. Valid values: `quartz`, `memory`.")+ :type :string+ :default "quartz"+ :visibility :internal+ :encryption :no+ :export? false)++(defsetting queue-no-listener-max-age-ms+ (deferred-tru+ (str "How long (in milliseconds) a Quartz queue message with no listener anywhere in the cluster "+ "is kept before the reaper drops it. With node-affinity, such a message is never acquired and "+ "just waits in the store; this bounds that wait so a message for a queue no node handles (e.g. "+ "a new queue whose node was rolled back) does not linger forever. Generous by default (1 day) "+ "so a node can restart or be re-upgraded and still deliver it."))+ :type :integer+ :default (* 24 60 60 1000)+ :visibility :internal+ :encryption :no+ :export? false)diff --git a/src/metabase/mq/task/outbox.clj b/src/metabase/mq/task/outbox.cljnew file mode 100644index 00000000000..4b2c7b65124--- /dev/null+++ b/src/metabase/mq/task/outbox.clj@@ -0,0 +1,37 @@+(ns metabase.mq.task.outbox+ "Quartz job that sweeps the transactional outbox ([[metabase.mq.queue.outbox]]). The normal+ after-commit path publishes and deletes outbox rows immediately, so this sweep only ever picks up+ rows a crash left behind between the business commit and the after-commit publish — republishing+ them so the persistence guarantee holds across node failures."+ (:require+ [clojurewerkz.quartzite.jobs :as jobs]+ [clojurewerkz.quartzite.schedule.cron :as cron]+ [clojurewerkz.quartzite.triggers :as triggers]+ [metabase.mq.queue.outbox :as outbox]+ [metabase.task.core :as task]+ [metabase.util.log :as log]))++(set! *warn-on-reflection* true)++(def ^:private job-key (jobs/key "metabase.mq.task.outbox.job"))+(def ^:private trigger-key (triggers/key "metabase.mq.task.outbox.trigger"))++(task/defjob ^{org.quartz.DisallowConcurrentExecution true+ :doc "Republishes transactional-outbox rows orphaned by a crash."}+ FlushMessageOutbox+ [_]+ (let [n (outbox/recover-outbox!)]+ (when (pos? n)+ (log/infof "Recovered %d orphaned queue outbox row(s)" n))))++(defmethod task/init! ::FlushMessageOutbox [_]+ (let [job (jobs/build+ (jobs/of-type FlushMessageOutbox)+ (jobs/with-identity job-key))+ trigger (triggers/build+ (triggers/with-identity trigger-key)+ (triggers/start-now)+ (triggers/with-schedule+ ;; every 5 minutes+ (cron/cron-schedule "0 0/5 * * * ? *")))]+ (task/schedule-task! job trigger)))diff --git a/src/metabase/mq/task/queue_reaper.clj b/src/metabase/mq/task/queue_reaper.cljnew file mode 100644index 00000000000..699851b1479--- /dev/null+++ b/src/metabase/mq/task/queue_reaper.clj@@ -0,0 +1,109 @@+(ns metabase.mq.task.queue-reaper+ "Quartz job that periodically drops queue messages no node in the cluster has a listener for.++ With node-affinity ([[metabase.mq.quartz-affinity]]) a message for a queue this node can't handle is+ never acquired — it just waits in the shared store for a capable node. This reaper bounds that wait:+ a message no node has picked up for longer than `queue-no-listener-max-age-ms` (default 1 day) is+ dropped, so a message for a queue nobody handles (e.g. a new queue whose node was rolled back before+ anyone listened) doesn't linger forever.++ Age is the only signal we have: Quartz's cluster state records live *nodes* but not which queues each+ node has a listener for (node-affinity is a runtime acquisition filter, never persisted), and the+ durable per-queue job is created by the publisher, not the listener — so there is no cluster-wide way+ to tell 'a listener exists for this queue' apart from 'this trigger has sat unacquired for a long+ time'. A queue with a live but heavily backlogged listener (e.g. `:exclusive`) can therefore be+ reaped if it stays unacquired past the max age; the generous default (1 day) plus [[startup-grace-ms]]+ keep that from happening in normal operation.++ The reaper stays quiet for [[startup-grace-ms]] after the node comes up: after a cluster-wide outage+ a backlog of perfectly-deliverable messages will have piled up with old start times, and we must not+ drop them before the just-recovered cluster has had a chance to deliver them."+ (:require+ [clojurewerkz.quartzite.jobs :as jobs]+ [clojurewerkz.quartzite.schedule.cron :as cron]+ [clojurewerkz.quartzite.triggers :as triggers]+ [metabase.analytics-interface.core :as analytics]+ [metabase.mq.quartz-affinity :as quartz-affinity]+ [metabase.mq.settings :as mq.settings]+ [metabase.task.core :as task]+ [metabase.util.log :as log]+ [toucan2.core :as t2])+ (:import+ (java.lang.management ManagementFactory)+ (java.time Instant)+ (org.quartz JobExecutionContext Scheduler TriggerKey)))++(set! *warn-on-reflection* true)++(def ^:private startup-grace-ms+ "Don't drop any messages until the node has been up this long. After a cluster-wide outage, messages+ pile up with old start times; give the just-recovered cluster time to actually deliver them before+ reaping, so a long downtime doesn't cost us messages that a live cluster would happily deliver."+ (* 60 60 1000))++(defn- node-uptime-ms ^long []+ (.getUptime (ManagementFactory/getRuntimeMXBean)))++(defn- orphaned-trigger-rows+ "Queue message triggers in `sched-name`'s store that have sat `WAITING` — never acquired by any node —+ since before `threshold` (epoch ms). Returns rows of `{:trigger_name :job_name}`.++ The table is referenced as the unquoted, upper-case `QRTZ_TRIGGERS` on purpose: Metabase creates the+ Quartz tables upper-case on MySQL/MariaDB (where table names are case-sensitive on Linux) but+ lower-case on Postgres. An unquoted upper-case identifier resolves on every backend — Postgres folds+ it to lower-case, MySQL/H2 match the stored upper-case name — whereas Toucan's default *quoted+ lower-case* reference (`:qrtz_triggers`) fails on case-sensitive MySQL/MariaDB. (Quartz's own SQL+ references these tables the same unquoted upper-case way.)"+ [sched-name threshold]+ (t2/query [(str "SELECT trigger_name, job_name FROM QRTZ_TRIGGERS"+ " WHERE sched_name = ? AND trigger_group = ? AND trigger_state = 'WAITING'"+ " AND start_time < ?")+ sched-name quartz-affinity/queue-job-group threshold]))++(defn- drop-orphaned-triggers!+ "Drops queue message triggers that have sat `WAITING` — never acquired by any node — for longer than+ [[mq.settings/queue-no-listener-max-age-ms]]. With node-affinity, a message for a queue no node+ handles is never acquired (it isn't bounced), so this is the mechanism that eventually gives up on it+ rather than letting it linger forever. A retried/requeued message gets a fresh trigger with a recent+ start time, so only genuinely-stuck messages age out.++ Returns the number dropped."+ [^Scheduler scheduler]+ (let [sched-name (.getSchedulerName scheduler)+ max-age (mq.settings/queue-no-listener-max-age-ms)+ threshold (- (.toEpochMilli (Instant/now)) max-age)+ rows (orphaned-trigger-rows sched-name threshold)]+ (doseq [{:keys [trigger_name job_name]} rows]+ ;; unscheduleJob removes the trigger (and its simple-trigger row); the durable job stays.+ (.unscheduleJob scheduler (TriggerKey. trigger_name quartz-affinity/queue-job-group))+ (log/warnf "Dropping queue message for %s: no node in the cluster has a listener after %d ms"+ job_name max-age)+ (analytics/inc! :metabase-mq/batches-dropped {:channel job_name :reason "no-listener-expired"}))+ (count rows)))++(def ^:private job-key (jobs/key "metabase.mq.task.queue-reaper.job"))+(def ^:private trigger-key (triggers/key "metabase.mq.task.queue-reaper.trigger"))++(task/defjob ^{org.quartz.DisallowConcurrentExecution true+ :doc "Drops queue messages no node in the cluster has a listener for (older than the max age)."}+ QueueReaper+ [ctx]+ (if (< (node-uptime-ms) startup-grace-ms)+ ;; too soon after startup — a recovering cluster may still be draining a backlog of old-but-valid+ ;; messages; don't drop anything yet.+ (log/debug "Queue reaper skipped: node has been up less than the startup grace period")+ (let [n (drop-orphaned-triggers! (.getScheduler ^JobExecutionContext ctx))]+ (when (pos? n)+ (log/infof "Dropped %d orphaned queue message(s) that no node has a listener for" n)))))++(defmethod task/init! ::QueueReaper [_]+ (let [job (jobs/build+ (jobs/of-type QueueReaper)+ (jobs/with-identity job-key))+ trigger (triggers/build+ (triggers/with-identity trigger-key)+ (triggers/start-now)+ (triggers/with-schedule+ ;; every 15 minutes+ (cron/cron-schedule "0 0/15 * * * ? *")))]+ (task/schedule-task! job trigger)))diff --git a/src/metabase/mq/transaction.clj b/src/metabase/mq/transaction.cljnew file mode 100644index 00000000000..fee22f1d879--- /dev/null+++ b/src/metabase/mq/transaction.clj@@ -0,0 +1,24 @@+(ns metabase.mq.transaction+ "Shared helper for deferring queue publishes to the surrounding DB transaction's boundaries."+ (:require+ [metabase.app-db.core :as mdb]))++(set! *warn-on-reflection* true)++(defn accumulate-and-register!+ "Accumulate `msgs` for `channel` under `[msgs-key channel]` in the current transaction's state atom,+ and run `register-once!` exactly the first time this is called for `registered-key` within that+ transaction.++ `register-once!` is passed the transaction `state` atom — captured here and handed over explicitly+ because after-commit callbacks run after the dynamic [[mdb/transaction-state]] has unwound, so they+ can't read it live. Must be called inside a transaction."+ [msgs-key registered-key channel msgs register-once!]+ (let [state (mdb/transaction-state)+ old (first (swap-vals! state+ (fn [s]+ (-> s+ (update-in [msgs-key channel] (fnil into []) msgs)+ (assoc registered-key true)))))]+ (when-not (get old registered-key)+ (register-once! state))))diff --git a/src/metabase/mq/transport.clj b/src/metabase/mq/transport.cljnew file mode 100644index 00000000000..594600bd72f--- /dev/null+++ b/src/metabase/mq/transport.clj@@ -0,0 +1,34 @@+(ns metabase.mq.transport+ "Thin dispatcher that routes channel operations to the queue backend.+ Selection is based on the channel keyword's namespace (`:queue/*`)."+ (:require+ [metabase.mq.payload :as payload]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.polling :as q.polling]))++(set! *warn-on-reflection* true)++(defn transport-type+ "Returns :queue for a namespaced channel keyword."+ [channel]+ (keyword (namespace channel)))++(defn publish-encoded!+ "Publishes an already-encoded `payload` string to the backend for `channel`'s transport type,+ then wakes the backend's poll loop so it picks the work up promptly."+ [channel payload]+ (let [backend q.backend/*backend*]+ (case (transport-type channel)+ :queue (do+ (q.backend/publish! backend channel payload)+ (q.polling/notify-on-publish! (:poll-context backend) channel))+ (throw (ex-info (format "Unknown transport for channel %s: expected a :queue/* channel" channel)+ {:channel channel :transport-type (transport-type channel)})))))++(defn publish!+ "Publishes messages to the appropriate backend for the channel's transport type, then wakes the+ backend's poll loop so it picks the work up promptly. Messages are encoded to an opaque string+ payload here — once, upstream of every backend — so backends only move bytes around and every+ backend delivers an identical message shape."+ [channel messages]+ (publish-encoded! channel (payload/encode messages)))diff --git a/src/metabase/startup/core.clj b/src/metabase/startup/core.cljindex 6a4a7335a4a..0d729a94572 100644--- a/src/metabase/startup/core.clj+++ b/src/metabase/startup/core.clj@@ -1,5 +1,5 @@(ns metabase.startup.core- "Defines the `def-startup-logic!` multimethod, which is used to run initialization logic when the server starts up."+ "Defines the `def-startup-logic!` and `def-shutdown-logic!` multimethods for server lifecycle hooks."(:require[metabase.util :as u][metabase.util.log :as log]))@@ -15,17 +15,49 @@For logic that should run on only one node at a time, use `cluster-lock/with-cluster-lock` in your function.+ To control ordering, add a [[startup-priority]] method for the same dispatch value (lower+ priorities run first; default 0).+(defmethod startup/init-logic! ::ExampleLogic [_](future (println \"Running example logic...\")))"{:arglists '([job-name-string])}keyword)+(defmulti startup-priority+ "Ordering priority for [[def-startup-logic!]] implementations. Lower runs first; higher runs+ later. Default is 0. Use `Long/MAX_VALUE` for logic that must run after everything else."+ {:arglists '([job-name-string])}+ keyword)++(defmethod startup-priority :default [_] 500)+(defn run-startup-logic!- "Call all implementations of `def-startup-logic!`. Called by metabase.core/init!"+ "Call all implementations of `def-startup-logic!`. Called by metabase.core/init!+ Methods run in ascending [[startup-priority]] order, with ties broken by dispatch-value name+ for determinism."[]- (doseq [[k f] (methods def-startup-logic!)]+ (doseq [[k f] (sort-by (fn [[k _]] [(startup-priority k) (name k)])+ (methods def-startup-logic!))](try(log/infof "Running setup logic %s %s" (u/format-color 'green (name k)) (u/emoji "☑\uFE0F"))(f k)(catch Throwable e(log/errorf e "Error initializing startup logic %s" k)))))++(defmulti def-shutdown-logic!+ "Runs shutdown logic with a given name. All implementations are called during graceful server+ shutdown. Counterpart to [[def-startup-logic!]].++ The dispatch value can be any unique keyword and is used for logging."+ {:arglists '([job-name-string])}+ keyword)++(defn run-shutdown-logic!+ "Call all implementations of `def-shutdown-logic!`. Called during graceful server shutdown."+ []+ (doseq [[k f] (methods def-shutdown-logic!)]+ (try+ (log/infof "Running shutdown logic %s" (name k))+ (f k)+ (catch Throwable e+ (log/errorf e "Error running shutdown logic %s" k)))))diff --git a/src/metabase/task/bootstrap.clj b/src/metabase/task/bootstrap.cljindex 6e8ebd426f3..b0a4cdef6da 100644--- a/src/metabase/task/bootstrap.clj+++ b/src/metabase/task/bootstrap.clj@@ -1,6 +1,7 @@(ns metabase.task.bootstrap(:require- [metabase.classloader.core :as classloader]))+ [metabase.classloader.core :as classloader]+ [metabase.util.log :as log]))(set! *warn-on-reflection* true)@@ -54,10 +55,34 @@(when-not *compile-files*(System/setProperty "org.quartz.scheduler.classLoadHelper.class" (.getName ClassLoadHelper)))+(defonce ^:private jdbc-property-setters+ ;; Extra fns (of db-type) run by set-jdbc-backend-properties! right before the scheduler initializes.+ ;; Lets a higher-level module (e.g. `mq`, which installs its node-affinity Quartz DriverDelegate)+ ;; hook in without `task` depending on it — `task` cannot depend on the modules that depend on it.+ (atom []))++(defn register-jdbc-property-setter!+ "Register `f` (a fn of the app-db `db-type`) to run when Quartz's JDBC backend properties are set,+ just before the scheduler initializes. Used by the `mq` module to install its node-affinity+ `DriverDelegate` — registering here inverts the dependency so `task` never references `mq`."+ [f]+ (swap! jdbc-property-setters conj f))+(defn set-jdbc-backend-properties!"Set the appropriate system properties needed so Quartz can connect to the JDBC backend. (Since we don't know our DBconnection properties ahead of time, we'll need to set these at runtime rather than Setting them in the- `quartz.properties` file.)"+ `quartz.properties` file.)++ Sets the default per-DB `DriverDelegate` (`PostgreSQLDelegate` on Postgres for BLOB handling,+ otherwise the `StdJDBCDelegate` from `quartz.properties`), then runs any setters registered via+ [[register-jdbc-property-setter!]] — e.g. the `mq` module's queue node-affinity delegate, which+ overrides the delegate with a filtering subclass. A registered setter that throws is logged and+ skipped so the scheduler still gets a working delegate."[db-type](when (= db-type :postgres)- (System/setProperty "org.quartz.jobStore.driverDelegateClass" "org.quartz.impl.jdbcjobstore.PostgreSQLDelegate")))+ (System/setProperty "org.quartz.jobStore.driverDelegateClass" "org.quartz.impl.jdbcjobstore.PostgreSQLDelegate"))+ (doseq [setter @jdbc-property-setters]+ (try+ (setter db-type)+ (catch Throwable t+ (log/warn t "A registered Quartz JDBC property setter failed; continuing")))))diff --git a/test/metabase/analytics_interface/core_test.cljc b/test/metabase/analytics_interface/core_test.cljcindex 57cd04e1017..d47e054909d 100644--- a/test/metabase/analytics_interface/core_test.cljc+++ b/test/metabase/analytics_interface/core_test.cljc@@ -3,18 +3,25 @@[clojure.test :refer [deftest is testing]][metabase.analytics-interface.core :as analytics.interface]))-(defn- make-test-reporter [calls]- (reify analytics.interface/Reporter- (-inc! [_ metric labels amount]- (swap! calls conj {:op :inc! :metric metric :labels labels :amount amount}))- (-dec-gauge! [_ metric labels amount]- (swap! calls conj {:op :dec-gauge! :metric metric :labels labels :amount amount}))- (-set-gauge! [_ metric labels amount]- (swap! calls conj {:op :set-gauge! :metric metric :labels labels :amount amount}))- (-observe! [_ metric labels amount]- (swap! calls conj {:op :observe! :metric metric :labels labels :amount amount}))- (-clear! [_ metric]- (swap! calls conj {:op :clear! :metric metric}))))+(defn- make-test-reporter+ "Builds a Reporter that records calls into `calls`. The global reporter is process-wide, so a+ background thread emitting analytics (e.g. a queue backend's poll loop) would otherwise pollute+ the capture; record only calls made on the thread that installed the reporter — the test thread —+ so these assertions stay deterministic."+ [calls]+ (let [#?@(:clj [capture-thread (Thread/currentThread)])+ ours? (fn [] #?(:clj (identical? (Thread/currentThread) capture-thread) :cljs true))]+ (reify analytics.interface/Reporter+ (-inc! [_ metric labels amount]+ (when (ours?) (swap! calls conj {:op :inc! :metric metric :labels labels :amount amount})))+ (-dec-gauge! [_ metric labels amount]+ (when (ours?) (swap! calls conj {:op :dec-gauge! :metric metric :labels labels :amount amount})))+ (-set-gauge! [_ metric labels amount]+ (when (ours?) (swap! calls conj {:op :set-gauge! :metric metric :labels labels :amount amount})))+ (-observe! [_ metric labels amount]+ (when (ours?) (swap! calls conj {:op :observe! :metric metric :labels labels :amount amount})))+ (-clear! [_ metric]+ (when (ours?) (swap! calls conj {:op :clear! :metric metric}))))))(defn- do-with-test-reporter! [thunk](let [calls (atom [])diff --git a/test/metabase/app_db/connection_test.clj b/test/metabase/app_db/connection_test.cljindex a4d59faccce..e6d5eee3e1c 100644--- a/test/metabase/app_db/connection_test.clj+++ b/test/metabase/app_db/connection_test.clj@@ -244,3 +244,98 @@;; The original exception should be thrown, not the setAutoCommit exception(is (= msg (ex-message e))))(is (true? @autocommit-reset-called))))))++;;; ------------------------------ before-commit + transaction-state ------------------------------+;;; The mq transactional outbox relies on this machinery: it inserts its rows from a before-commit+;;; callback (so they commit atomically with the business transaction) and stashes the ids in+;;; transaction-state to publish them from an after-commit callback once the commit lands.++(deftest do-before-commit-runs-immediately-outside-transaction-test+ (let [calls (atom [])]+ (mdb.connection/do-before-commit (fn [] (swap! calls conj :ran)))+ (is (= [:ran] @calls) "outside a transaction do-before-commit runs the thunk immediately")))++(deftest before-commit-callbacks-run-before-commit-and-commit-their-writes-test+ (let [email (mt/random-email)+ order (atom [])]+ (try+ (t2/with-transaction [_conn]+ (mdb.connection/do-before-commit+ (fn []+ (swap! order conj :before-commit)+ (t2/insert! :model/User (assoc (mt/with-temp-defaults :model/User) :email email))))+ (swap! order conj :body)+ (is (= [:body] @order) "before-commit callback has not run during the body"))+ (is (= [:body :before-commit] @order)+ "before-commit runs after the body returns, as part of the commit sequence")+ (is (t2/exists? :model/User :email email)+ "a row inserted from a before-commit callback commits with the transaction")+ (finally+ (t2/delete! :model/User :email email)))))++(deftest before-commit-callbacks-not-run-on-rollback-test+ (let [calls (atom [])]+ (is (thrown?+ Exception+ (t2/with-transaction [_conn]+ (mdb.connection/do-before-commit (fn [] (swap! calls conj :should-not-run)))+ (throw (ex-info "force rollback" {})))))+ (is (= [] @calls) "before-commit callbacks do not run when the transaction rolls back")))++(deftest throwing-before-commit-rolls-back-the-transaction-test+ (let [email (mt/random-email)]+ (try+ (is (thrown-with-msg?+ Exception #"boom"+ (t2/with-transaction [_conn]+ (t2/insert! :model/User (assoc (mt/with-temp-defaults :model/User) :email email))+ (is (t2/exists? :model/User :email email) "row is visible inside the transaction")+ (mdb.connection/do-before-commit (fn [] (throw (ex-info "boom" {})))))))+ (is (not (t2/exists? :model/User :email email))+ "the business write rolls back when a before-commit callback throws")+ (finally+ (t2/delete! :model/User :email email)))))++(deftest before-commit-callbacks-from-rolled-back-nested-transaction-are-discarded-test+ (let [calls (atom [])]+ (t2/with-transaction [conn]+ (mdb.connection/do-before-commit (fn [] (swap! calls conj :outer)))+ (is (thrown?+ Exception+ (t2/with-transaction [_ conn]+ (mdb.connection/do-before-commit (fn [] (swap! calls conj :nested-should-not-run)))+ (throw (ex-info "force savepoint rollback" {}))))))+ (is (= [:outer] @calls)+ "only the outer before-commit callback runs; the rolled-back nested one is discarded")))++(deftest before-commit-can-schedule-after-commit-test+ ;; the mq transactional outbox works this way: a before-commit callback does its DB write, then+ ;; schedules the post-commit publish over what it just wrote.+ (let [order (atom [])]+ (t2/with-transaction [_conn]+ (mdb.connection/do-before-commit+ (fn []+ (swap! order conj :before)+ (mdb.connection/do-after-commit (fn [] (swap! order conj :after))))))+ (is (= [:before :after] @order)+ "an after-commit scheduled from a before-commit runs after the transaction commits")))++(deftest transaction-state-shared-across-nested-transactions-test+ (t2/with-transaction [conn]+ (let [outer-state mdb.connection/*transaction-state*]+ (is (some? outer-state) "transaction-state is bound inside a transaction")+ (t2/with-transaction [_ conn]+ (is (identical? outer-state mdb.connection/*transaction-state*)+ "nested transactions share the same transaction-state atom"))))+ (is (nil? mdb.connection/*transaction-state*) "transaction-state is nil outside a transaction"))++(deftest transaction-state-from-rolled-back-nested-transaction-is-discarded-test+ (t2/with-transaction [conn]+ (swap! mdb.connection/*transaction-state* assoc :outer-key "outer-val")+ (is (thrown?+ Exception+ (t2/with-transaction [_ conn]+ (swap! mdb.connection/*transaction-state* assoc :inner-key "inner-val")+ (throw (ex-info "force savepoint rollback" {})))))+ (is (= {:outer-key "outer-val"} @mdb.connection/*transaction-state*)+ "data stashed by a rolled-back nested transaction is discarded from transaction-state")))diff --git a/test/metabase/metabot/api_test.clj b/test/metabase/metabot/api_test.cljindex 171a82bd7e1..daf324a270f 100644--- a/test/metabase/metabot/api_test.clj+++ b/test/metabase/metabot/api_test.clj@@ -1148,20 +1148,21 @@(reset! captured-args args);; Return a minimal streaming responsenil)]- (api/streaming-request {:metabot_id test-metabot-id- :profile_id nil- :message "test message"- :context {}- :history []- :conversation_id (str (random-uuid))- :state {}- :debug false}- {:origin nil :referer nil :user-agent nil :ip-address nil})- (testing "metabot-id is included in the arguments"- (is (some? (:metabot-id @captured-args))- "metabot-id should not be nil")- (is (= test-metabot-id (:metabot-id @captured-args))- "metabot-id should match the input metabot_id"))))))+ (mt/with-test-user :rasta+ (api/streaming-request {:metabot_id test-metabot-id+ :profile_id nil+ :message "test message"+ :context {}+ :history []+ :conversation_id (str (random-uuid))+ :state {}+ :debug false}+ {:origin nil :referer nil :user-agent nil :ip-address nil})+ (testing "metabot-id is included in the arguments"+ (is (some? (:metabot-id @captured-args))+ "metabot-id should not be nil")+ (is (= test-metabot-id (:metabot-id @captured-args))+ "metabot-id should match the input metabot_id")))))))(deftest streaming-request-ip-address-test(mt/with-model-cleanup [:model/MetabotMessagediff --git a/test/metabase/mq/backend_parity_test.clj b/test/metabase/mq/backend_parity_test.cljnew file mode 100644index 00000000000..ac327749fe3--- /dev/null+++ b/test/metabase/mq/backend_parity_test.clj@@ -0,0 +1,115 @@+(ns metabase.mq.backend-parity-test+ "Contract-level parity between the queue backends.++ Each scenario is written once and run against every backend kind so that+ semantic drift between backend implementations is caught at the source+ rather than discovered later in production.++ Every scenario is ALSO run with `:duplicate-delivery? true` so that+ listeners in the parity suite are forced to tolerate the MQ's at-least-+ once contract. Assertions use multiset / containment checks rather than+ exact-sequence equality so that a single logical message can legitimately+ be delivered more than once.++ Scenarios use `mq.tu/eventually!` rather than `mq.tu/flush!` for their+ assertions, because `wait-for-idle!` cannot see pending rows inside a+ durable backend's storage — it only checks the publish buffer, the+ worker-pool busy set, and memory channels."+ (:require+ [clojure.test :refer :all]+ [metabase.mq.core :as mq]+ [metabase.mq.test-util :as mq.tu]))++(set! *warn-on-reflection* true)++(def ^:private backend-kinds+ "Backend kinds exercised by the parity scenarios. `:memory` covers the+ fast async in-process backend."+ [:memory])++(def ^:private delivery-modes+ "Delivery modes exercised by the parity scenarios. `:normal` is the+ production path; `:duplicate` wraps both backends with a decorator that+ publishes every message twice, forcing listeners to tolerate at-least-+ once delivery."+ [:normal :duplicate])++(def ^:private scenario-timeout-ms+ "Budget for waiting on eventual delivery in parity scenarios. A poll-based+ backend runs through the publish buffer → storage → 5s-default poll loop, but+ the fixture's `wait-for-idle!` calls `notify-all!` each tick so the effective+ poll latency is much lower."+ 10000)++(defn- unique-channel [transport-ns suffix]+ (keyword transport-ns (str suffix "-" (random-uuid))))++(defn- run-parity!+ "Runs `scenario-fn` once per (kind × delivery-mode) combination. Each+ invocation gets a freshly-minted unique channel keyword so scenarios do+ not interfere with each other."+ ([channel-prefix scenario-fn]+ (run-parity! backend-kinds delivery-modes channel-prefix scenario-fn))+ ([kinds modes channel-prefix scenario-fn]+ (doseq [kind kinds+ mode modes]+ (testing (str kind " backend / " mode " delivery")+ (let [channel (unique-channel (namespace channel-prefix) (name channel-prefix))]+ (mq.tu/with-test-mq [ctx {:backend kind+ :duplicate-delivery? (= mode :duplicate)}]+ (scenario-fn ctx channel)))))))++(deftest queue-delivers-published-messages-test+ (run-parity!+ :queue/parity-delivery+ (fn [ctx queue-name]+ (let [received (atom [])]+ (mq.tu/listen! queue-name #(swap! received conj %))+ (mq/with-queue queue-name [q]+ (mq/put q "a")+ (mq/put q "b")+ (mq/put q "c"))+ (mq.tu/eventually! ctx+ #(= #{"a" "b" "c"} (set @received))+ scenario-timeout-ms)+ (is (= #{"a" "b" "c"} (set @received))+ "Every unique message was delivered at least once")+ (is (every? pos? (vals (frequencies @received)))+ "No spurious messages appeared in @received")+ (mq/unlisten! queue-name)))))++(deftest queue-retries-failed-batch-test+ (run-parity!+ :queue/parity-retry+ (fn [ctx queue-name]+ (let [calls-by-msg (atom {})]+ (mq.tu/listen! queue-name+ (fn [msg]+ (let [n (get (swap! calls-by-msg update msg (fnil inc 0)) msg)]+ (when (= 1 n)+ (throw (ex-info "first attempt" {:msg msg}))))))+ (mq/with-queue queue-name [q]+ (mq/put q "retry-me"))+ (mq.tu/eventually! ctx+ #(>= (get @calls-by-msg "retry-me" 0) 2)+ scenario-timeout-ms)+ (is (>= (get @calls-by-msg "retry-me" 0) 2)+ "Message delivered at least twice (retry-after-failure or chaos duplication)")+ (mq/unlisten! queue-name)))))++(deftest queue-normalizes-message-shape-test+ (testing "Every backend delivers messages in the same canonical JSON-normalized shape:+ keyword keys preserved, keyword/symbol values and other non-JSON-native+ values collapsed to their JSON form."+ (run-parity!+ :queue/parity-shape+ (fn [ctx queue-name]+ (let [received (atom [])]+ (mq.tu/listen! queue-name #(swap! received conj %))+ (mq/with-queue queue-name [q]+ (mq/put q {:id 1 :status :active :nested {:flag true}}))+ (mq.tu/eventually! ctx #(seq @received) scenario-timeout-ms)+ (is (= #{{:id 1 :status "active" :nested {:flag true}}}+ (set @received))+ "Message delivered in canonical shape regardless of backend")+ (mq/unlisten! queue-name))))))diff --git a/test/metabase/mq/listener_test.clj b/test/metabase/mq/listener_test.cljnew file mode 100644index 00000000000..522aae9f2d0--- /dev/null+++ b/test/metabase/mq/listener_test.clj@@ -0,0 +1,92 @@+(ns metabase.mq.listener-test+ (:require+ [clojure.test :refer :all]+ [metabase.mq.core :as mq]+ [metabase.mq.listener :as listener]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.test-util :as mq.tu])+ (:import+ (clojure.lang ExceptionInfo)))++(set! *warn-on-reflection* true)++;;; Queues + listeners declared the production way — macros at namespace-load time. This lets+;;; us exercise the full def-queue! → register-queues! and def-listener! → register-listeners!+;;; activation path that mq.init/start! (and therefore with-test-mq) drives. Because the queues+;;; are declared, they coexist harmlessly with every other with-test-mq run in the suite (the+;;; handlers are registered but never invoked unless something publishes to these queues).++(def ^:private macro-received (atom []))++(mq/def-queue! :queue/listener-macro-test {:transactional :try})+(mq/def-listener! :queue/listener-macro-test [messages]+ (swap! macro-received into messages))++(def ^:private multi-count (atom 0))+(def ^:private multi-received (atom []))++(mq/def-queue! :queue/listener-multi-test {:transactional :try})+(mq/def-listener! :queue/listener-multi-test [messages]+ (swap! multi-count inc)+ (swap! multi-received into messages))++(deftest def-listener!-activation-test+ (reset! macro-received [])+ (mq.tu/with-test-mq [ctx]+ (testing "register-listeners! wires the macro-declared handler into *listeners*"+ (is (fn? (:listener (listener/get-listener :queue/listener-macro-test)))))+ (testing "the handler receives published messages as a vec"+ (mq/with-queue :queue/listener-macro-test [q]+ (mq/put q "a")+ (mq/put q "b"))+ (mq.tu/eventually! ctx #(= ["a" "b"] @macro-received) 5000)+ (is (= ["a" "b"] @macro-received)))))++(deftest def-listener!-multi-statement-body-test+ (reset! multi-count 0)+ (reset! multi-received [])+ (mq.tu/with-test-mq [ctx]+ (testing "every expression in a multi-statement body runs, and `messages` is bound"+ (mq/with-queue :queue/listener-multi-test [q]+ (mq/put q "m1"))+ (mq.tu/eventually! ctx #(seq @multi-received) 5000)+ (is (= ["m1"] @multi-received))+ (is (pos? @multi-count) "the side-effecting first statement also ran"))))++(deftest listener-requires-declared-queue-test+ (binding [listener/*listeners* (atom {})+ q.registry/*queues* (atom {})]+ (testing "registering a listener for an undeclared queue throws"+ (is (thrown-with-msg? ExceptionInfo #"No queue declared"+ (listener/batch-listen! :queue/never-declared (fn [_])))))+ (testing "succeeds once the queue is declared"+ (q.registry/register-queue! :queue/now-declared {:transactional :try})+ (listener/batch-listen! :queue/now-declared (fn [_]))+ (is (fn? (:listener (listener/get-listener :queue/now-declared)))))))++(deftest duplicate-listener-throws-test+ (binding [listener/*listeners* (atom {})+ q.registry/*queues* (atom {})]+ (q.registry/register-queue! :queue/dup {:transactional :try})+ (listener/batch-listen! :queue/dup (fn [_]))+ (testing "a second listener on the same queue throws"+ (is (thrown-with-msg? ExceptionInfo #"Listener already registered"+ (listener/batch-listen! :queue/dup (fn [_])))))))++(deftest register-listeners!-wraps-failures-test+ (let [channel :queue/register-failure-test]+ (try+ ;; Simulate a def-listener! whose queue was never declared.+ (defmethod listener/def-listener* channel [_]+ (listener/batch-listen! channel (fn [_])))+ (binding [listener/*listeners* (atom {})+ q.registry/*queues* (atom {})]+ ;; Declare every OTHER macro-registered queue so the only failure is ours.+ (doseq [qn (keys (methods listener/def-listener*))+ :when (not= qn channel)]+ (q.registry/register-queue! qn {:transactional :try}))+ (testing "register-listeners! rethrows a registration failure with context"+ (is (thrown-with-msg? ExceptionInfo #"Failed to register listener"+ (listener/register-listeners!)))))+ (finally+ (remove-method listener/def-listener* channel)))))diff --git a/test/metabase/mq/payload_test.clj b/test/metabase/mq/payload_test.cljnew file mode 100644index 00000000000..bcdae458852--- /dev/null+++ b/test/metabase/mq/payload_test.clj@@ -0,0 +1,42 @@+(ns metabase.mq.payload-test+ (:require+ [clojure.test :refer :all]+ [metabase.mq.payload :as payload])+ (:import+ (java.time Instant)+ (java.util UUID)))++(set! *warn-on-reflection* true)++(deftest check-serializable!-accepts-json-round-trippable-values-test+ (testing "values that JSON round-trips (some lossily but meaningfully) are accepted"+ (doseq [msg [{}+ {:a 1 :b "two" :c true :d nil}+ {:nested {:x [1 2 3] :y #{:a :b}}}+ {:kw :some/keyword :sym 'a-sym} ; keywords/symbols -> strings+ {:when (Instant/now) :id (UUID/randomUUID)} ; dates/UUIDs -> strings+ {:ratio 1/3 :big 9999999999999999999N :dec 1.5M}+ [{:a 1} {:b 2}]+ {"string-key" 1}]]+ (is (nil? (payload/check-serializable! msg))+ (str "should accept " (pr-str msg))))))++(deftest check-serializable!-rejects-opaque-values-test+ (testing "values JSON encoding would silently turn into .toString() junk are rejected loudly"+ (are [msg] (thrown-with-msg? clojure.lang.ExceptionInfo #"not JSON-serializable"+ (payload/check-serializable! msg))+ {:f (fn [])} ; a function+ {:a (atom 1)} ; an atom (IDeref)+ {:o (Object.)} ; an arbitrary object+ {:nested {:deep [1 {:x (fn [])}]}} ; nested inside a collection+ {[1 2] "vector-key"}))) ; a non-string/keyword/symbol map key++(deftest check-serializable!-error-points-at-the-offending-path-test+ (testing "the thrown error carries the path and class of the offending value for diagnosis"+ (let [e (try (payload/check-serializable! {:outer {:inner [:ok (fn [])]}}) nil+ (catch clojure.lang.ExceptionInfo e e))]+ (is (some? e))+ (is (= [:outer :inner 1] (:path (ex-data e)))+ "the path locates the bad value within the message")+ (is (isa? (:value-class (ex-data e)) clojure.lang.IFn)+ "the class of the offending value is reported"))))diff --git a/test/metabase/mq/publish_buffer_test.clj b/test/metabase/mq/publish_buffer_test.cljnew file mode 100644index 00000000000..6b3aaf3dc45--- /dev/null+++ b/test/metabase/mq/publish_buffer_test.clj@@ -0,0 +1,78 @@+(ns metabase.mq.publish-buffer-test+ (:require+ [clojure.test :refer :all]+ [metabase.mq.payload :as payload]+ [metabase.mq.publish-buffer :as publish-buffer]+ [metabase.mq.queue.outbox :as q.outbox]+ [metabase.mq.transport :as transport]+ [metabase.test :as mt]+ [metabase.test.util.dynamic-redefs :refer [with-dynamic-fn-redefs]]))++(set! *warn-on-reflection* true)++(defn- due-entry [messages]+ {:queue/test {:messages messages :deadline-ms 1 :created-ms 1}})++(deftest successful-flush-publishes-to-backend-test+ (testing "a due accumulation entry is published to the backend and cleared"+ (let [published (atom [])]+ (binding [publish-buffer/*publish-buffer* (atom (due-entry ["m1" "m2"]))+ publish-buffer/*publish-buffer-ms* 0+ publish-buffer/*publish-buffer-max-ms* 0]+ (with-dynamic-fn-redefs [transport/publish! (fn [ch msgs] (swap! published conj {:channel ch :messages msgs}))]+ (publish-buffer/flush-publish-buffer!)+ (is (= [{:channel :queue/test :messages ["m1" "m2"]}] @published))+ (is (empty? @publish-buffer/*publish-buffer*)))))))++(deftest flush-failure-hands-batch-to-outbox-test+ (testing "when the backend is unreachable, the flushed batch is handed to the durable outbox — not retried or dropped in memory"+ (let [outboxed (atom [])]+ (binding [publish-buffer/*publish-buffer* (atom (due-entry ["m1" "m2"]))+ publish-buffer/*publish-buffer-ms* 0+ publish-buffer/*publish-buffer-max-ms* 0]+ (with-dynamic-fn-redefs [transport/publish! (fn [_ _] (throw (ex-info "backend down" {})))+ q.outbox/insert-batch! (fn [ch payload] (swap! outboxed conj {:channel ch :messages (payload/decode payload)}))]+ (publish-buffer/flush-publish-buffer!)+ (is (= [{:channel :queue/test :messages ["m1" "m2"]}] @outboxed)+ "the batch (encoded) is written to the outbox for durable recovery")+ (is (empty? @publish-buffer/*publish-buffer*)+ "and cleared from the in-memory buffer — the buffer keeps no retry state"))))))++(deftest flush-handoff-emits-metric-test+ (testing "a successful outbox handoff surfaces as batches-retried{reason=publish-outbox-handoff}"+ (mt/with-prometheus-system! [_ system]+ (binding [publish-buffer/*publish-buffer* (atom (due-entry ["m1"]))+ publish-buffer/*publish-buffer-ms* 0+ publish-buffer/*publish-buffer-max-ms* 0]+ (with-dynamic-fn-redefs [transport/publish! (fn [_ _] (throw (ex-info "backend down" {})))+ q.outbox/insert-batch! (fn [_ _] nil)]+ (publish-buffer/flush-publish-buffer!)+ (is (pos? (mt/metric-value system :metabase-mq/batches-retried+ {:channel "test" :reason "publish-outbox-handoff"}))))))))++(deftest flush-failure-and-outbox-failure-drops-and-meters-test+ (testing "if the backend AND the outbox are both unreachable (total outage), the batch is dropped and metered"+ (mt/with-prometheus-system! [_ system]+ (binding [publish-buffer/*publish-buffer* (atom (due-entry ["m1"]))+ publish-buffer/*publish-buffer-ms* 0+ publish-buffer/*publish-buffer-max-ms* 0]+ (with-dynamic-fn-redefs [transport/publish! (fn [_ _] (throw (ex-info "backend down" {})))+ q.outbox/insert-batch! (fn [_ _] (throw (ex-info "db down" {})))]+ (publish-buffer/flush-publish-buffer!)+ (is (empty? @publish-buffer/*publish-buffer*))+ (is (pos? (mt/metric-value system :metabase-mq/batches-dropped+ {:channel "test" :reason "outbox-handoff-failed"}))+ "the drop is surfaced as batches-dropped{reason=outbox-handoff-failed}"))))))++(deftest max-ms-cap-forces-flush-test+ (testing "an accumulation entry force-flushes once it has sat for *publish-buffer-max-ms*, even if its sliding window hasn't elapsed"+ (let [published (atom [])]+ (binding [publish-buffer/*publish-buffer* (atom {:queue/test {:messages ["fresh"]+ :deadline-ms (+ (System/currentTimeMillis) 100000)+ :created-ms 0}})+ publish-buffer/*publish-buffer-ms* 100000 ; sliding window never fires on its own+ publish-buffer/*publish-buffer-max-ms* 50]+ (with-dynamic-fn-redefs [transport/publish! (fn [_ msgs] (swap! published into msgs))]+ (publish-buffer/flush-publish-buffer!)+ (is (= ["fresh"] @published) "the entry hit its max-ms cap and flushed")+ (is (empty? @publish-buffer/*publish-buffer*)))))))diff --git a/test/metabase/mq/publish_test.clj b/test/metabase/mq/publish_test.cljnew file mode 100644index 00000000000..4288f32d7ee--- /dev/null+++ b/test/metabase/mq/publish_test.clj@@ -0,0 +1,25 @@+(ns metabase.mq.publish-test+ (:require+ [clojure.test :refer :all]+ [metabase.mq.publish :as publish]))++(set! *warn-on-reflection* true)++(deftest run-with-buffer-validates-serializability-before-routing-test+ (let [routed (atom [])]+ (with-redefs-fn {#'publish/publish-collected! (fn [ch msgs] (swap! routed conj [ch msgs]))}+ (fn []+ (testing "serializable messages collected in the body are routed on success"+ (reset! routed [])+ (publish/run-with-buffer :queue/x "err"+ (fn [b] (publish/put b {:a 1}) (publish/put b {:b 2})))+ (is (= [[:queue/x [{:a 1} {:b 2}]]] @routed)))++ (testing "an unserializable message throws at the call site and nothing is routed"+ (reset! routed [])+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"not JSON-serializable"+ (publish/run-with-buffer :queue/x "err"+ (fn [b] (publish/put b {:ok 1})+ (publish/put b {:bad (fn [])})))))+ (is (= [] @routed)+ "validation precedes routing, so a bad batch never reaches the outbox/transport"))))))diff --git a/test/metabase/mq/quartz_affinity_test.clj b/test/metabase/mq/quartz_affinity_test.cljnew file mode 100644index 00000000000..22df00f5192--- /dev/null+++ b/test/metabase/mq/quartz_affinity_test.clj@@ -0,0 +1,119 @@+(ns metabase.mq.quartz-affinity-test+ (:require+ [clojure.string :as str]+ [clojure.test :refer :all]+ [metabase.classloader.core :as classloader]+ [metabase.mq.quartz-affinity :as affinity]+ [metabase.mq.queue.quartz :as q.quartz]+ [metabase.test.fixtures :as fixtures]+ [toucan2.core :as t2])+ (:import+ (java.lang.reflect Method)+ (java.time Instant)+ (org.quartz TriggerKey)+ (org.quartz.impl.jdbcjobstore Util)))++(set! *warn-on-reflection* true)++;; The integration test below reads/writes the QRTZ_* tables directly, so the app-db must be migrated.+(use-fixtures :once (fixtures/initialize :db))++;; The exact Quartz 2.3.2 selectTriggerToAcquire SQL (SCHED_NAME + table prefix already substituted).+(def ^:private acquire-sql+ (str "SELECT TRIGGER_NAME, TRIGGER_GROUP, NEXT_FIRE_TIME, PRIORITY FROM QRTZ_TRIGGERS"+ " WHERE SCHED_NAME = 'MetabaseScheduler' AND TRIGGER_STATE = ? AND NEXT_FIRE_TIME <= ?"+ " AND (MISFIRE_INSTR = -1 OR (MISFIRE_INSTR != -1 AND NEXT_FIRE_TIME >= ?))"+ " ORDER BY NEXT_FIRE_TIME ASC, PRIORITY DESC"))++(deftest rewrite-acquisition-sql-test+ (testing ":all disables filtering — sql unchanged"+ (is (= acquire-sql (affinity/rewrite-acquisition-sql acquire-sql :all))))+ (testing "empty capability set excludes ALL queue jobs, leaves other jobs"+ (let [out (affinity/rewrite-acquisition-sql acquire-sql #{})]+ (is (re-find #"AND \(job_group <> 'metabase\.mq\.queue'\) ORDER BY" out))+ (is (not (re-find #"job_name IN" out)))))+ (testing "a capability set gates the queue group by job_name IN (...)"+ (let [out (affinity/rewrite-acquisition-sql acquire-sql #{"alpha" "beta"})]+ (is (re-find #"AND \(job_group <> 'metabase\.mq\.queue' OR job_name IN \(" out))+ (is (re-find #"'alpha'" out))+ (is (re-find #"'beta'" out))+ (is (str/ends-with? out "ORDER BY NEXT_FIRE_TIME ASC, PRIORITY DESC"))))+ (testing "single-quotes in a queue name are escaped (no SQL injection / breakage)"+ (is (re-find #"'o''brien'" (affinity/rewrite-acquisition-sql acquire-sql #{"o'brien"})))))++(deftest queue-job-group-matches-quartz-backend-test+ (testing "the group we gate on is exactly the group the Quartz backend puts queue jobs in"+ (is (= @#'q.quartz/job-group affinity/queue-job-group))))++(deftest ensure-delegate-loadable!-produces-a-driver-delegate-subclass-test+ (testing "each app-db's affinity delegate loads (AOT in uberjar / runtime-compiled in dev), extends the right base, and overrides selectTriggerToAcquire"+ (doseq [[db-type base] {:h2 org.quartz.impl.jdbcjobstore.StdJDBCDelegate+ :mysql org.quartz.impl.jdbcjobstore.StdJDBCDelegate+ :postgres org.quartz.impl.jdbcjobstore.PostgreSQLDelegate}]+ (let [class-name (affinity/ensure-delegate-loadable! db-type)+ k (Class/forName class-name true (classloader/the-classloader))]+ (is (.isAssignableFrom ^Class base k)+ (str db-type " delegate extends " (.getSimpleName ^Class base)))+ (is (.isAssignableFrom org.quartz.impl.jdbcjobstore.DriverDelegate k)+ "is a DriverDelegate")+ (is (some (fn [^Method m] (and (= "selectTriggerToAcquire" (.getName m))+ (= 4 (count (.getParameterTypes m)))))+ (.getDeclaredMethods k))+ "overrides the 4-arg selectTriggerToAcquire")))))++;;; Integration: run the real `selectTriggerToAcquire` re-implementation against the app DB (whatever+;;; type CI uses), proving the affinity predicate actually filters rows — and never touches non-queue+;;; jobs. Rows are isolated under a random sched_name so concurrent tests / the real scheduler are+;;; unaffected. Each row's trigger_name == its job_name, so the acquired TriggerKey names read back as+;;; the job names we assert on.++(def ^:private mq-group affinity/queue-job-group)++(defn- epoch-ms ^long [] (.toEpochMilli (Instant/now)))++(defn- seed! [sched rows]+ (let [now (epoch-ms)]+ (doseq [[jn jg] rows]+ (t2/query (str "INSERT INTO QRTZ_JOB_DETAILS (sched_name,job_name,job_group,job_class_name,is_durable,is_nonconcurrent,is_update_data,requests_recovery)"+ " VALUES ('" sched "','" jn "','" jg "','x.Job',true,false,false,false)"))+ (t2/query (str "INSERT INTO QRTZ_TRIGGERS (sched_name,trigger_name,trigger_group,job_name,job_group,next_fire_time,priority,trigger_state,trigger_type,start_time,misfire_instr)"+ " VALUES ('" sched "','" jn "','" jg "','" jn "','" jg "'," (- now 60000)+ ",5,'WAITING','SIMPLE'," (- now 120000) ",-1)")))))++(defn- cleanup! [sched]+ (t2/query (str "DELETE FROM QRTZ_TRIGGERS WHERE sched_name='" sched "'"))+ (t2/query (str "DELETE FROM QRTZ_JOB_DETAILS WHERE sched_name='" sched "'")))++;; Stand in for the delegate's protected `rtp`: substitute {0}=table prefix, {1}=sched-name literal for+;; this test's isolated sched, using Quartz's own (MessageFormat-based) helper — so the query is exactly+;; what a real delegate would run, scoped to our rows.+(defn- test-rtp [sched]+ (fn [sql] (Util/rtp sql "QRTZ_" (str "'" sched "'"))))++(defn- acquire! [sched capability]+ (let [the-atom @#'affinity/capability-fn* ; deref the var -> the atom+ prev @the-atom ; deref the atom -> the current fn+ now (epoch-ms)]+ (affinity/set-capability-fn! (constantly capability))+ (try+ (t2/with-connection [conn]+ (->> (affinity/select-trigger-to-acquire conn (+ now 3600000) (- now 3600000) 100 (test-rtp sched))+ (map (fn [^TriggerKey k] (.getName k)))+ set))+ (finally (reset! the-atom prev)))))++(deftest select-trigger-to-acquire-filters-by-capability-against-appdb-test+ (testing "select-trigger-to-acquire filters the acquisition query by capability, on the real app DB"+ (let [sched (str "AFFINITY_TEST_" (random-uuid))]+ (cleanup! sched)+ (seed! sched [["aff-queue-a" mq-group] ["aff-queue-b" mq-group] ["aff-sync" "DEFAULT"]])+ (try+ (is (= #{"aff-queue-a" "aff-queue-b" "aff-sync"} (acquire! sched :all))+ ":all acquires everything")+ (is (= #{"aff-queue-a" "aff-sync"} (acquire! sched #{"aff-queue-a"}))+ "capable of only queue-a acquires it + the non-queue job, not queue-b")+ (is (= #{"aff-sync"} (acquire! sched #{}))+ "no queue listeners → acquires no queue jobs, but the non-queue job still runs")+ (is (= #{"aff-sync"} (acquire! sched #{"nonexistent"}))+ "capable of an unrelated queue → only the non-queue job")+ (finally (cleanup! sched))))))diff --git a/test/metabase/mq/queue/core_test.clj b/test/metabase/mq/queue/core_test.cljnew file mode 100644index 00000000000..650fc34da98--- /dev/null+++ b/test/metabase/mq/queue/core_test.clj@@ -0,0 +1,300 @@+(ns metabase.mq.queue.core-test+ (:require+ [clojure.test :refer :all]+ [metabase.app-db.connection :as app-db.conn]+ [metabase.mq.core :as mq]+ [metabase.mq.listener :as listener]+ [metabase.mq.payload :as payload]+ [metabase.mq.publish :as mq.publish]+ [metabase.mq.publish-buffer :as publish-buffer]+ [metabase.mq.queue.polling :as q.polling]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.test-util :as mq.tu]+ [toucan2.core :as t2])+ (:import (clojure.lang ExceptionInfo)+ (java.util.concurrent CountDownLatch CyclicBarrier TimeUnit)))++(set! *warn-on-reflection* true)++(deftest e2e-test+ (let [heard-messages (atom [])]+ (mq.tu/with-test-mq [test-mq]+ {:queue/test (fn [message] (swap! heard-messages conj message))}+ (testing "The messages are heard and processed"+ (mq/with-queue :queue/test [q]+ (mq/put q "test message 1")+ (mq/put q "test message 2"))+ (mq.tu/flush! test-mq)+ (is (= ["test message 1" "test message 2"] @heard-messages))))))++(deftest publish-to-unlistened-queue-test+ (mq.tu/with-test-mq [_test-mq]+ (testing "with-queue on a queue with no listener still succeeds"+ (mq/with-queue :queue/nonexistent [q]+ (mq/put q "msg")))))++(deftest with-queue-success-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [test-mq]+ {:queue/test (fn [msg] (swap! heard conj msg))}+ (testing "with-queue publishes buffered messages on success"+ (let [result (mq/with-queue :queue/test [q]+ (mq/put q "a")+ (mq/put q "b")+ :done)]+ (is (= :done result))+ (mq.tu/flush! test-mq)+ (is (= ["a" "b"] @heard)))))))++(deftest with-queue-exception-discards-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [test-mq]+ {:queue/test (fn [msg] (swap! heard conj msg))}+ (testing "with-queue discards buffered messages on exception"+ (is (thrown? Exception+ (mq/with-queue :queue/test [q]+ (mq/put q "should-be-discarded")+ (throw (ex-info "boom" {})))))+ (mq.tu/flush! test-mq)+ (is (empty? @heard))))))++(deftest with-queue-no-listener-test+ (mq.tu/with-test-mq [_test-mq]+ (testing "with-queue on queue with no listener still succeeds"+ (mq/with-queue :queue/nonexistent [q]+ (mq/put q "msg")))))++(deftest double-listen-throws-test+ (mq.tu/with-test-mq [_test-mq]+ {:queue/test (fn [_] nil)}+ (testing "Registering a second listener on the same queue throws"+ (is (thrown-with-msg? ExceptionInfo #"Listener already registered"+ (mq.tu/listen! :queue/test (fn [_] nil)))))))++(deftest concurrent-listen-throws-test+ (mq.tu/with-test-mq [_test-mq]+ (let [queue-name :queue/test+ n 10+ barrier (CyclicBarrier. n)+ results (atom [])]+ (testing "Concurrent listen! calls: exactly one succeeds, rest throw"+ (let [threads (mapv (fn [_]+ (let [f (bound-fn []+ (.await barrier)+ (try+ (mq.tu/listen! queue-name (fn [_] nil))+ (swap! results conj :ok)+ (catch ExceptionInfo _+ (swap! results conj :error))))]+ (Thread. ^Runnable f)))+ (range n))]+ (run! (fn [^Thread t] (.start t)) threads)+ (run! (fn [^Thread t] (.join t 5000)) threads)+ (is (= 1 (count (filter #{:ok} @results))))+ (is (= (dec n) (count (filter #{:error} @results))))))+ (mq/unlisten! queue-name))))++(deftest exclusive-listen-test+ (let [heard-messages (atom [])]+ (mq.tu/with-test-mq [test-mq]+ (q.registry/register-queue! :queue/exclusive-test {:transactional :try :exclusive true :max-batch-messages 1})+ (mq.tu/listen! :queue/exclusive-test #(swap! heard-messages conj %))+ (testing "Exclusive queue processes messages via the async memory backend"+ (mq/with-queue :queue/exclusive-test [q]+ (mq/put q "msg1")+ (mq/put q "msg2"))+ (mq.tu/flush! test-mq)+ (is (= ["msg1" "msg2"] @heard-messages))))))++(deftest batch-listen-exclusive-test+ (let [heard-batches (atom [])]+ (mq.tu/with-test-mq [test-mq]+ (q.registry/register-queue! :queue/batch-exclusive {:transactional :try :exclusive true :max-batch-messages 10})+ (listener/batch-listen! :queue/batch-exclusive+ (fn [batch] (swap! heard-batches conj batch)))+ (testing "Exclusive batch-listen! registers and processes"+ (mq/with-queue :queue/batch-exclusive [q]+ (mq/put q "a")+ (mq/put q "b"))+ (mq.tu/flush! test-mq)+ (is (= [["a" "b"]] @heard-batches))))))++(deftest transaction-defers-publish-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [test-mq]+ ;; `:never` keeps the in-memory after-commit defer path under test (a `:transactional` queue+ ;; would route through the outbox table instead — see metabase.mq.queue.outbox-test).+ (q.registry/register-queue! :queue/test {:transactional :never})+ (mq.tu/listen! :queue/test (fn [msg] (swap! heard conj msg)))+ (testing "Inside a transaction, messages are accumulated, not published immediately"+ (t2/with-transaction [_conn]+ (mq/with-queue :queue/test [q]+ (mq/put q "msg1")+ (mq/put q "msg2"))+ (mq/with-queue :queue/test [q]+ (mq/put q "msg3"))+ (testing "Messages are deferred in transaction state"+ (is (= ["msg1" "msg2" "msg3"]+ (get-in @app-db.conn/*transaction-state*+ [::mq.publish/deferred-messages :queue/test]))))+ (testing "Listener has not been called yet"+ (is (= [] @heard)))))+ (testing "After the transaction commits, all messages are delivered"+ (mq.tu/flush! test-mq)+ (is (= ["msg1" "msg2" "msg3"] @heard))))))++(deftest transaction-rollback-discards-messages-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [test-mq]+ (q.registry/register-queue! :queue/test {:transactional :never})+ (mq.tu/listen! :queue/test (fn [msg] (swap! heard conj msg)))+ (testing "Messages accumulated during a failed transaction are discarded"+ (is (thrown? Exception+ (t2/with-transaction [_conn]+ (mq/with-queue :queue/test [q]+ (mq/put q "should-be-discarded"))+ (throw (ex-info "boom" {})))))+ (mq.tu/flush! test-mq)+ (is (= [] @heard) "no messages are delivered from a rolled-back transaction")))))++(deftest outside-transaction-publishes-immediately-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [test-mq]+ {:queue/test (fn [msg] (swap! heard conj msg))}+ (testing "Outside a transaction, messages are published"+ (is (nil? app-db.conn/*transaction-state*))+ (mq/with-queue :queue/test [q]+ (mq/put q "msg1"))+ (mq.tu/flush! test-mq)+ (is (= ["msg1"] @heard))))))++(deftest multiple-queues-in-transaction-test+ (let [heard-a (atom [])+ heard-b (atom [])]+ (mq.tu/with-test-mq [test-mq]+ (q.registry/register-queue! :queue/test-a {:transactional :never})+ (q.registry/register-queue! :queue/test-b {:transactional :never})+ (mq.tu/listen! :queue/test-a (fn [msg] (swap! heard-a conj msg)))+ (mq.tu/listen! :queue/test-b (fn [msg] (swap! heard-b conj msg)))+ (t2/with-transaction [_conn]+ (mq/with-queue :queue/test-a [q]+ (mq/put q "a1")+ (mq/put q "a2"))+ (mq/with-queue :queue/test-b [q]+ (mq/put q "b1"))+ (testing "Both queues have deferred messages"+ (is (= ["a1" "a2"]+ (get-in @app-db.conn/*transaction-state*+ [::mq.publish/deferred-messages :queue/test-a])))+ (is (= ["b1"]+ (get-in @app-db.conn/*transaction-state*+ [::mq.publish/deferred-messages :queue/test-b])))))+ (testing "After the transaction commits, both queues receive their messages"+ (mq.tu/flush! test-mq)+ (is (= ["a1" "a2"] @heard-a))+ (is (= ["b1"] @heard-b))))))++(deftest buffering-combines-messages-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [test-mq]+ (q.registry/register-queue! :queue/test {:transactional :try :max-batch-messages 50})+ (listener/batch-listen! :queue/test+ (fn [batch] (swap! heard into batch)))+ (binding [publish-buffer/*publish-buffer-ms* 100+ publish-buffer/*publish-buffer* (atom {})]+ (testing "buffered-publish! buffers when *publish-buffer-ms* > 0"+ (mq.publish/publish! :queue/test ["msg1"])+ (mq.publish/publish! :queue/test ["msg2"])+ (is (empty? @heard)+ "Nothing delivered yet because buffer window hasn't elapsed")+ (is (= ["msg1" "msg2"]+ (:messages (get @publish-buffer/*publish-buffer* :queue/test)))+ "Messages are buffered"))+ (testing "After flush, all messages are delivered"+ (swap! publish-buffer/*publish-buffer*+ update :queue/test assoc :deadline-ms 1)+ (publish-buffer/flush-publish-buffer!)+ (mq.tu/flush! test-mq)+ (is (= ["msg1" "msg2"] @heard)+ "Both messages delivered after flush"))))))++(deftest buffering-immediate-when-zero-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [test-mq]+ {:queue/test (fn [msg] (swap! heard conj msg))}+ (binding [publish-buffer/*publish-buffer-ms* 0+ publish-buffer/*publish-buffer* (atom {})]+ (testing "When *publish-buffer-ms* is 0, messages publish without buffering"+ (mq/with-queue :queue/test [q]+ (mq/put q "msg1"))+ (mq.tu/flush! test-mq)+ (is (= ["msg1"] @heard))+ (mq/with-queue :queue/test [q]+ (mq/put q "msg2"))+ (mq.tu/flush! test-mq)+ (is (= ["msg1" "msg2"] @heard)))))))++(deftest buffering-flush-delivers-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [test-mq]+ (q.registry/register-queue! :queue/test {:transactional :try :max-batch-messages 3})+ (listener/batch-listen! :queue/test+ (fn [batch] (swap! heard into batch)))+ (binding [publish-buffer/*publish-buffer-ms* 1+ publish-buffer/*publish-buffer-max-ms* 0+ publish-buffer/*publish-buffer* (atom {})]+ (testing "Messages are buffered until flushed"+ (mq.publish/publish! :queue/test ["a" "b"])+ (is (empty? @heard) "not yet flushed")+ (mq.publish/publish! :queue/test ["c"])+ (is (empty? @heard) "still buffered, no proactive flush")+ (Thread/sleep 5)+ (publish-buffer/flush-publish-buffer!)+ (mq.tu/flush! test-mq)+ (is (= ["a" "b" "c"] @heard) "all delivered after flush"))))))++(deftest fifo-ordering-test+ (let [received (atom [])]+ (mq.tu/with-test-mq [test-mq]+ {:queue/test (fn [message] (swap! received conj message))}+ (doseq [i (range 10)]+ (mq/with-queue :queue/test [q]+ (mq/put q i)))+ (mq.tu/flush! test-mq)+ (testing "All messages are delivered in order"+ (is (= (range 10) @received))))))++(deftest exclusive-queue-single-active-handler-test+ "Verifies that submit-delivery! returns false for a channel that already has an+ active handler, enforcing at-most-one concurrent delivery per channel."+ (q.polling/start-worker-pool!)+ (mq.tu/with-test-mq [_test-mq]+ (let [queue-name :queue/exclusive-concurrency-test+ started (CountDownLatch. 1) ; counted down once the worker thread enters the listener+ release (CountDownLatch. 1)] ; blocks the listener until the test releases it+ (mq.tu/listen! queue-name+ (fn [_] (.countDown started) (.await release)))+ (testing "First submission succeeds and marks the channel as busy"+ (is (true? (q.polling/submit-delivery! queue-name (payload/encode ["msg1"]) nil nil)))+ (is (.await started 5 TimeUnit/SECONDS) "worker thread started and entered the listener")+ (is (true? (q.polling/channel-busy? queue-name))))+ (testing "Second submission returns false while channel is busy"+ (is (false? (q.polling/submit-delivery! queue-name (payload/encode ["msg2"]) nil nil))))+ (.countDown release) ; Release the listener+ (testing "Channel is no longer busy after delivery completes"+ (is (true? (mq.tu/wait-for! #(not (q.polling/channel-busy? queue-name)) 5000))+ "active-handlers clears once delivery completes"))+ (mq/unlisten! queue-name))))++(deftest submit-delivery-frees-slot-when-pool-unavailable-test+ (testing "if submitting to the worker pool fails (e.g. pool not running), the channel slot is+ released rather than left permanently busy — otherwise the queue would silently stop+ delivering forever"+ (with-redefs [q.polling/worker-pool (atom nil) ; deref/.submit NPEs+ q.polling/active-handlers (atom {})]+ (let [queue-name :queue/submit-failure-test]+ (is (thrown? Exception+ (q.polling/submit-delivery! queue-name (payload/encode ["x"]) nil nil))+ "the submit failure propagates to the caller")+ (is (false? (q.polling/channel-busy? queue-name))+ "the channel is not left marked busy after the failed submit")))))diff --git a/test/metabase/mq/queue/memory_test.clj b/test/metabase/mq/queue/memory_test.cljnew file mode 100644index 00000000000..fce1772fcd0--- /dev/null+++ b/test/metabase/mq/queue/memory_test.clj@@ -0,0 +1,66 @@+(ns metabase.mq.queue.memory-test+ (:require+ [clojure.test :refer :all]+ [metabase.mq.core :as mq]+ [metabase.mq.listener :as listener]+ [metabase.mq.memory :as memory]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.memory :as q.memory]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.test-util :as mq.tu]))++(set! *warn-on-reflection* true)++(defn- depth-for [layer channel-name]+ (some #(when (= channel-name (:channel %)) (:count %)) (memory/depths layer)))++(deftest drop-orphaned-removes-listener-less-messages-test+ (testing "drop-orphaned! removes only messages whose queue is not in the live set"+ (let [layer (memory/make-layer)]+ (memory/publish! layer :queue/live "a")+ (memory/publish! layer :queue/orphan "b")+ (memory/publish! layer :queue/live "c")+ (is (= 1 (memory/drop-orphaned! layer #{:queue/live})) "one orphaned message dropped")+ (is (= 2 (depth-for layer "live")) "live-queue messages retained")+ (is (nil? (depth-for layer "orphan")) "orphaned-queue messages gone"))))++(deftest recover-stale-drops-orphaned-messages-test+ (testing "the memory backend's periodic recovery sweep drops messages for queues with no listener"+ (binding [listener/*listeners* (atom {})+ q.registry/*queues* (atom {})]+ (let [layer (memory/make-layer)+ backend (q.memory/make-backend layer)]+ (mq.tu/listen! :queue/has-listener (fn [_]))+ (memory/publish! layer :queue/has-listener "keep")+ (memory/publish! layer :queue/no-listener "drop")+ (q.backend/recover-stale! backend 1000 5)+ (is (= 1 (depth-for layer "has-listener")) "messages with a listener are kept")+ (is (nil? (depth-for layer "no-listener")) "orphaned messages are dropped")))))++(deftest publish-test+ (mq.tu/with-test-mq [ctx]+ (let [queue-name (keyword "queue" (str "publish-test-" (gensym)))+ received (atom [])]+ (mq.tu/listen! queue-name (fn [msg] (swap! received conj msg)))+ (testing "Publish lazily creates channel and delivers messages"+ (mq/with-queue queue-name [q]+ (mq/put q "test-message")+ (mq/put q "test-message2"))+ (mq.tu/flush! ctx)+ (is (= ["test-message" "test-message2"] @received))))))++(deftest exclusive-memory-test+ (mq.tu/with-test-mq [ctx]+ (let [queue-name :queue/exclusive-test+ received (atom [])]+ (q.registry/register-queue! queue-name {:transactional :try :exclusive true})+ (mq.tu/listen! queue-name (fn [msg] (swap! received conj msg)))+ (testing "Exclusive queue processes all messages"+ (mq/with-queue queue-name [q]+ (mq/put q "msg1")+ (mq/put q "msg2")+ (mq/put q "msg3")+ (mq/put q "msg4")+ (mq/put q "msg5"))+ (mq.tu/flush! ctx)+ (is (= ["msg1" "msg2" "msg3" "msg4" "msg5"] @received))))))diff --git a/test/metabase/mq/queue/outbox_test.clj b/test/metabase/mq/queue/outbox_test.cljnew file mode 100644index 00000000000..ac06a030020--- /dev/null+++ b/test/metabase/mq/queue/outbox_test.clj@@ -0,0 +1,415 @@+(ns ^:synchronous metabase.mq.queue.outbox-test+ (:require+ [clojure.test :refer :all]+ [metabase.app-db.connection :as app-db.conn]+ [metabase.mq.core :as mq]+ [metabase.mq.payload :as payload]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.outbox :as outbox]+ [metabase.mq.queue.quartz :as q.quartz]+ [metabase.mq.test-util :as mq.tu]+ [metabase.mq.transport :as transport]+ [metabase.task.impl :as task.impl]+ [metabase.test :as mt]+ [metabase.test.fixtures :as fixtures]+ [metabase.test.util.dynamic-redefs :as dynamic-redefs :refer [with-dynamic-fn-redefs]]+ [toucan2.core :as t2])+ (:import+ (clojure.lang ExceptionInfo)+ (java.sql Timestamp)+ (java.time Instant)))++(set! *warn-on-reflection* true)++;;; Queues declared the production way (namespace-load-time macros) so `register-queues!` realizes+;;; them into every `with-test-mq` run's fresh `*queues*` atom with the `:transactional` mode under test.+(mq/def-queue! :queue/outbox-require {:transactional :require})+(mq/def-queue! :queue/outbox-try {:transactional :try})+(mq/def-queue! :queue/outbox-never {:transactional :never})+(mq/def-queue! :queue/outbox-chunked {:transactional :require :max-batch-messages 2})+(mq/def-queue! :queue/outbox-dedup {:transactional :require :dedup-fn distinct})+(mq/def-queue! :queue/outbox-recover {:transactional :require})++(def ^:private test-queue-names+ ["outbox-require" "outbox-try" "outbox-never" "outbox-chunked" "outbox-dedup" "outbox-recover"])++(defn- clear-outbox! [t]+ (t2/delete! :queue_message_outbox :queue_name [:in test-queue-names])+ (try (t) (finally (t2/delete! :queue_message_outbox :queue_name [:in test-queue-names]))))++;; These tests read/write the `queue_message_outbox` table directly, so the app-db must be migrated.+(use-fixtures :once (fixtures/initialize :db))+(use-fixtures :each clear-outbox!)++(defn- outbox-count [qname]+ (t2/count :queue_message_outbox :queue_name qname))++(deftest require-mode-no-transaction-throws-test+ (mq.tu/with-test-mq [_ctx]+ (testing ":require published outside a transaction throws"+ (is (thrown-with-msg?+ ExceptionInfo #":transactional :require"+ (mq/with-queue :queue/outbox-require [q]+ (mq/put q "nope"))))+ (is (zero? (outbox-count "outbox-require")) "nothing written to the outbox"))))++(deftest require-mode-atomic-with-business-write-test+ (let [heard (atom [])+ email (mt/random-email)]+ (mq.tu/with-test-mq [ctx]+ {:queue/outbox-require (fn [m] (swap! heard conj m))}+ (testing "rollback: neither the business write nor the message survive"+ (is (thrown? Exception+ (t2/with-transaction [_conn]+ (t2/insert! :model/User (assoc (mt/with-temp-defaults :model/User) :email email))+ (mq/with-queue :queue/outbox-require [q]+ (mq/put q "nope"))+ (throw (ex-info "rollback" {})))))+ (mq.tu/flush! ctx)+ (is (not (t2/exists? :model/User :email email)) "business write rolled back")+ (is (= [] @heard) "no message delivered")+ (is (zero? (outbox-count "outbox-require")) "no outbox row persisted"))+ (testing "commit: the business write commits and the message is delivered, outbox cleared"+ (try+ (t2/with-transaction [_conn]+ (t2/insert! :model/User (assoc (mt/with-temp-defaults :model/User) :email email))+ (mq/with-queue :queue/outbox-require [q]+ (mq/put q "hello")))+ (mq.tu/eventually! ctx #(= ["hello"] @heard) 5000)+ (is (= ["hello"] @heard))+ (is (t2/exists? :model/User :email email))+ (is (zero? (outbox-count "outbox-require")) "outbox row deleted after publish")+ (finally+ (t2/delete! :model/User :email email)))))))++(deftest try-mode-no-transaction-publishes-immediately-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [ctx]+ {:queue/outbox-try (fn [m] (swap! heard conj m))}+ (mq/with-queue :queue/outbox-try [q]+ (mq/put q "x"))+ (mq.tu/eventually! ctx #(= ["x"] @heard) 5000)+ (is (= ["x"] @heard))+ (is (zero? (outbox-count "outbox-try")) "no outbox row used outside a transaction"))))++(deftest try-mode-in-transaction-uses-outbox-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [ctx]+ {:queue/outbox-try (fn [m] (swap! heard conj m))}+ (t2/with-transaction [_conn]+ (mq/with-queue :queue/outbox-try [q]+ (mq/put q "y")))+ (mq.tu/eventually! ctx #(= ["y"] @heard) 5000)+ (is (= ["y"] @heard) ":try inside a transaction is delivered after commit"))))++(deftest never-mode-in-transaction-defers-in-memory-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [ctx]+ {:queue/outbox-never (fn [m] (swap! heard conj m))}+ (t2/with-transaction [_conn]+ (mq/with-queue :queue/outbox-never [q]+ (mq/put q "z")))+ (mq.tu/eventually! ctx #(= ["z"] @heard) 5000)+ (is (= ["z"] @heard))+ (is (zero? (outbox-count "outbox-never")) ":never never touches the outbox table"))))++(deftest chunking-respects-max-batch-messages-test+ (mq.tu/with-test-mq [_ctx]+ (testing "messages are chunked by :max-batch-messages into one outbox row per chunk"+ (binding [app-db.conn/*transaction-state*+ (atom {:metabase.mq.queue.outbox/messages+ {:queue/outbox-chunked ["a" "b" "c" "d" "e"]}})]+ (outbox/insert-outbox-rows!))+ (is (= 3 (outbox-count "outbox-chunked")) "5 messages, max-batch 2 -> 3 rows (2,2,1)")+ (let [payloads (->> (t2/query {:select [:payload]+ :from [:queue_message_outbox]+ :where [:= :queue_name "outbox-chunked"]+ :order-by [[:id :asc]]})+ (map (comp payload/decode :payload)))]+ (is (= [["a" "b"] ["c" "d"] ["e"]] payloads))))))++(deftest dedup-applied-in-outbox-test+ (mq.tu/with-test-mq [_ctx]+ (testing "the queue's dedup-fn is applied before messages are written to the outbox"+ (binding [app-db.conn/*transaction-state*+ (atom {:metabase.mq.queue.outbox/messages+ {:queue/outbox-dedup ["a" "a" "b" "a"]}})]+ (outbox/insert-outbox-rows!))+ (is (= 1 (outbox-count "outbox-dedup")))+ (let [msgs (->> (t2/query {:select [:payload]+ :from [:queue_message_outbox]+ :where [:= :queue_name "outbox-dedup"]})+ (mapcat (comp payload/decode :payload)))]+ (is (= ["a" "b"] (vec msgs)))))))++(deftest multiple-publishes-one-transaction-register-callbacks-once-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [ctx]+ {:queue/outbox-require (fn [m] (swap! heard conj m))}+ (testing "several with-queue publishes in one txn register the before/after callbacks once — no duplicate delivery"+ (t2/with-transaction [_conn]+ (mq/with-queue :queue/outbox-require [q] (mq/put q "a"))+ (mq/with-queue :queue/outbox-require [q] (mq/put q "b"))+ (mq/with-queue :queue/outbox-require [q] (mq/put q "c")))+ (mq.tu/eventually! ctx #(<= 3 (count @heard)) 5000)+ (mq.tu/flush! ctx)+ (is (= ["a" "b" "c"] (sort @heard))+ "each message delivered exactly once (a second after-commit run would duplicate them)")+ (is (zero? (outbox-count "outbox-require")) "outbox cleared after publish")))))++(deftest nested-transaction-commit-delivers-all-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [ctx]+ {:queue/outbox-require (fn [m] (swap! heard conj m))}+ (testing "publishes from outer and committed nested savepoints are all delivered once"+ (t2/with-transaction [_outer]+ (mq/with-queue :queue/outbox-require [q] (mq/put q "outer"))+ (t2/with-transaction [_inner]+ (mq/with-queue :queue/outbox-require [q] (mq/put q "inner"))))+ (mq.tu/eventually! ctx #(= #{"outer" "inner"} (set @heard)) 5000)+ (is (= #{"outer" "inner"} (set @heard)))+ (is (zero? (outbox-count "outbox-require")) "outbox cleared after publish")))))++(deftest nested-transaction-inner-rollback-discards-inner-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [ctx]+ {:queue/outbox-require (fn [m] (swap! heard conj m))}+ (testing "a message published in a rolled-back nested savepoint is discarded; the outer one survives"+ (t2/with-transaction [_outer]+ (mq/with-queue :queue/outbox-require [q] (mq/put q "outer"))+ (try+ (t2/with-transaction [_inner]+ (mq/with-queue :queue/outbox-require [q] (mq/put q "inner"))+ (throw (ex-info "rollback inner" {})))+ (catch Exception _)))+ (mq.tu/eventually! ctx #(= ["outer"] @heard) 5000)+ (is (= ["outer"] @heard) "only the outer message is delivered")+ (is (zero? (outbox-count "outbox-require")) "outbox cleared after publish")))))++(deftest recover-outbox-republishes-stale-rows-test+ (let [heard (atom [])]+ (mq.tu/with-test-mq [ctx]+ {:queue/outbox-recover (fn [m] (swap! heard conj m))}+ (testing "a row a crash left behind is republished and deleted by the recovery sweep"+ (t2/insert! :queue_message_outbox+ {:queue_name "outbox-recover"+ :payload (payload/encode ["recovered"])+ :created_at (Timestamp/from (.minusMillis (Instant/now) (* 5 60 1000)))})+ (is (= 1 (outbox-count "outbox-recover")))+ (is (= 1 (outbox/recover-outbox!)) "one row republished")+ (mq.tu/eventually! ctx #(= ["recovered"] @heard) 5000)+ (is (= ["recovered"] @heard))+ (is (zero? (outbox-count "outbox-recover")) "row deleted after republish")))))++(deftest publish-outbox-rows-retains-row-when-publish-fails-test+ (testing "if the backend publish fails (e.g. quartz with no scheduler) the row is left for the recovery sweep, not deleted"+ (mq.tu/with-test-mq [_ctx]+ (let [pl (payload/encode ["keep-me"])+ id (t2/insert-returning-pk! :queue_message_outbox+ {:queue_name "outbox-require" :payload pl})]+ ;; Point the active backend at quartz with no scheduler so publish! throws; the per-row+ ;; try/catch in publish-outbox-rows! must then leave the row in place for recovery.+ (binding [q.backend/*backend* q.quartz/backend+ task.impl/*quartz-scheduler* (atom nil)]+ (outbox/publish-outbox-rows!+ (atom {:metabase.mq.queue.outbox/rows+ [{:id id :channel :queue/outbox-require :payload pl}]})))+ (is (t2/exists? :queue_message_outbox :id id)+ "publish failed, so the row is retained (not silently deleted)")))))++(deftest recover-outbox-partial-failure-keeps-delivered-rows-deleted-test+ (testing "a publish failure partway through the sweep does not roll back rows already published+deleted"+ (mq.tu/with-test-mq [_ctx]+ (let [good (payload/encode ["good"])+ bad (payload/encode ["bad"])+ old (Timestamp/from (.minusMillis (Instant/now) (* 5 60 1000)))+ good-id (t2/insert-returning-pk! :queue_message_outbox+ {:queue_name "outbox-recover" :payload good :created_at old})+ ;; higher id -> published after the good row (order-by id asc)+ bad-id (t2/insert-returning-pk! :queue_message_outbox+ {:queue_name "outbox-recover" :payload bad :created_at old})+ published (atom [])+ ;; capture the unpatched fn via original-fn — a bare var ref would resolve to the+ ;; with-dynamic-fn-redefs proxy and recurse (see that macro's docstring).+ real-publish (dynamic-redefs/original-fn #'transport/publish-encoded!)]+ (with-dynamic-fn-redefs [transport/publish-encoded!+ (fn [channel payload]+ (if (= payload bad)+ (throw (ex-info "boom" {}))+ (do (swap! published conj payload)+ (real-publish channel payload))))]+ (is (= 1 (outbox/recover-outbox!)) "only the good row counts as recovered"))+ (is (= [good] @published) "the good row was published exactly once")+ (is (not (t2/exists? :queue_message_outbox :id good-id))+ "the published row is deleted and its delete is NOT rolled back by the later failure")+ (is (t2/exists? :queue_message_outbox :id bad-id)+ "the failed row is retained for a later sweep")))))++(deftest publish-outbox-rows-batches-deletes-test+ (testing "successfully published rows are deleted as a batch; a row whose publish fails is retained"+ (mq.tu/with-test-mq [_ctx]+ (let [p1 (payload/encode ["one"])+ p2 (payload/encode ["two"])+ p3 (payload/encode ["three"])+ id1 (t2/insert-returning-pk! :queue_message_outbox {:queue_name "outbox-require" :payload p1})+ id2 (t2/insert-returning-pk! :queue_message_outbox {:queue_name "outbox-require" :payload p2})+ id3 (t2/insert-returning-pk! :queue_message_outbox {:queue_name "outbox-require" :payload p3})+ published (atom [])+ real-publish (dynamic-redefs/original-fn #'transport/publish-encoded!)]+ (with-dynamic-fn-redefs [transport/publish-encoded!+ (fn [channel payload]+ ;; the middle row fails to publish+ (if (= payload p2)+ (throw (ex-info "boom" {}))+ (do (swap! published conj payload)+ (real-publish channel payload))))]+ (outbox/publish-outbox-rows!+ (atom {:metabase.mq.queue.outbox/rows+ [{:id id1 :channel :queue/outbox-require :payload p1}+ {:id id2 :channel :queue/outbox-require :payload p2}+ {:id id3 :channel :queue/outbox-require :payload p3}]})))+ (is (not (t2/exists? :queue_message_outbox :id id1)) "published row deleted")+ (is (t2/exists? :queue_message_outbox :id id2) "failed row retained for recovery")+ (is (not (t2/exists? :queue_message_outbox :id id3)) "published row deleted")))))++(deftest recover-outbox-skips-fresh-rows-test+ (mq.tu/with-test-mq [_ctx]+ (testing "rows younger than the recovery age are left for the normal after-commit path"+ (t2/insert! :queue_message_outbox+ {:queue_name "outbox-recover"+ :payload (payload/encode ["fresh"])})+ (is (= 0 (outbox/recover-outbox!)) "no rows old enough to recover")+ (is (= 1 (outbox-count "outbox-recover")) "fresh row left in place"))))++(defn- ->instant+ "Coerce a timestamp column value to an Instant — the app DB may hand it back as an OffsetDateTime+ (Postgres timestamptz) or a java.sql.Timestamp (H2), depending on the backend."+ ^java.time.Instant [t]+ (condp instance? t+ java.time.OffsetDateTime (.toInstant ^java.time.OffsetDateTime t)+ java.sql.Timestamp (.toInstant ^java.sql.Timestamp t)+ java.time.Instant t))++(def ^:private stale-ts (Timestamp/from (.minusMillis (Instant/now) (* 5 60 1000))))++(defn- insert-stale-row! [payload]+ (t2/insert-returning-pk! :queue_message_outbox+ {:queue_name "outbox-recover" :payload payload :created_at stale-ts}))++(defn- always-fail-publish []+ (fn [_channel _payload] (throw (ex-info "always boom" {}))))++(deftest recover-outbox-bumps-attempts-on-publish-failure-test+ (testing "a row whose publish keeps failing is retained with its attempts bumped and a future retry scheduled — not dropped, and not re-attempted within the same sweep"+ (mq.tu/with-test-mq [_ctx]+ (let [id (insert-stale-row! (payload/encode ["poison"]))]+ (with-dynamic-fn-redefs [transport/publish-encoded! (always-fail-publish)]+ (is (= 0 (outbox/recover-outbox!)) "nothing recovered"))+ (is (t2/exists? :queue_message_outbox :id id) "poison row retained for a later sweep")+ (let [row (t2/select-one [:queue_message_outbox :publish_attempts :next_attempt_at] :id id)]+ (is (= 1 (:publish_attempts row))+ "attempts bumped exactly once for the one sweep (the failed row is skipped, not re-claimed)")+ (is (some? (:next_attempt_at row)) "a backed-off retry time is scheduled"))))))++(deftest recover-outbox-never-drops-and-backs-off-failing-row-test+ (testing "a row whose publish always fails is retried with exponential backoff forever — never dropped"+ (mq.tu/with-test-mq [_ctx]+ (let [id (insert-stale-row! (payload/encode ["always-fails"]))]+ (with-dynamic-fn-redefs [transport/publish-encoded! (always-fail-publish)]+ ;; sweep 1: attempt bumped and a future next_attempt_at scheduled (backing off)+ (is (= 0 (outbox/recover-outbox!)) "nothing recovered")+ (let [row (t2/select-one [:queue_message_outbox :publish_attempts :next_attempt_at] :id id)]+ (is (= 1 (:publish_attempts row)) "attempts bumped")+ (is (.isAfter (->instant (:next_attempt_at row)) (Instant/now))+ "next attempt scheduled in the future"))+ ;; immediate re-sweep: the row is not due yet, so it is skipped (attempts unchanged)+ (is (= 0 (outbox/recover-outbox!)) "nothing recovered")+ (is (= 1 (:publish_attempts (t2/select-one [:queue_message_outbox :publish_attempts] :id id)))+ "a row that is not yet due is not re-attempted before its backoff elapses")+ ;; force the row due and sweep again: re-attempted, attempt count grows, still retained+ (t2/update! :queue_message_outbox :id id+ {:next_attempt_at (Timestamp/from (.minusSeconds (Instant/now) 1))})+ (is (= 0 (outbox/recover-outbox!)) "nothing recovered")+ (is (= 2 (:publish_attempts (t2/select-one [:queue_message_outbox :publish_attempts] :id id)))+ "a due row is re-attempted and its attempt count keeps growing")+ (is (t2/exists? :queue_message_outbox :id id)+ "the row is never dropped no matter how many times it fails"))))))++(deftest retry-delay-ms-is-exponential-and-capped-test+ (testing "recovery retry backoff doubles per consecutive failure and is capped"+ (is (= [60000 120000 240000 480000 (* 10 60 1000)] (map outbox/retry-delay-ms [1 2 3 4 5]))+ "exponential: 1m base, doubling — attempt 5 would be 16m but caps at the 10m max")+ (is (= (* 10 60 1000) (outbox/retry-delay-ms 50)) "capped at the max backoff")+ (is (= (* 10 60 1000) (outbox/retry-delay-ms 100000)) "no overflow at extreme attempt counts")))++(deftest recover-outbox-poison-row-does-not-block-newer-rows-test+ (testing "a full page of poison rows at the head does not starve the recoverable rows behind them in the same sweep"+ (mq.tu/with-test-mq [_ctx]+ ;; shrink the per-transaction page so two poison rows fill it; the good row (higher id) is only+ ;; reachable on the next loop iteration, which the old code never got to (it stopped on deleted=0).+ (with-redefs-fn {#'outbox/recovery-page-size 2}+ (fn []+ (let [poison (payload/encode ["poison"])+ good (payload/encode ["good"])+ _p1 (insert-stale-row! poison)+ _p2 (insert-stale-row! poison)+ good-id (insert-stale-row! good)+ published (atom [])+ real-publish (dynamic-redefs/original-fn #'transport/publish-encoded!)]+ (with-dynamic-fn-redefs [transport/publish-encoded!+ (fn [channel payload]+ (if (= payload poison)+ (throw (ex-info "boom" {}))+ (do (swap! published conj payload)+ (real-publish channel payload))))]+ (is (= 1 (outbox/recover-outbox!))+ "the good row is recovered despite the full batch of poison rows ahead of it"))+ (is (= [good] @published) "the good row was published exactly once")+ (is (not (t2/exists? :queue_message_outbox :id good-id)) "good row published and deleted")+ (is (= 2 (outbox-count "outbox-recover")) "only the two poison rows remain")))))))++(deftest recover-outbox-emits-retry-metric-on-publish-failure-test+ (testing "a recovery failure that will be retried surfaces as batches-retried{recovery} (rows are never dropped)"+ (mt/with-prometheus-system! [_ system]+ (mq.tu/with-test-mq [_ctx]+ (insert-stale-row! (payload/encode ["fails"]))+ (with-dynamic-fn-redefs [transport/publish-encoded! (always-fail-publish)]+ (outbox/recover-outbox!)+ (is (pos? (mt/metric-value system :metabase-mq/batches-retried {:channel "outbox-recover" :reason "outbox-recovery"}))+ "a failed recovery publish that will be retried increments batches-retried{reason=recovery}"))))))++(deftest recover-outbox-stops-early-when-backend-unavailable-test+ (testing "when a publish reports the backend is unavailable, the sweep stops after the first attempt instead of thrashing (and bumping) every remaining row"+ (mq.tu/with-test-mq [_ctx]+ (let [ids (vec (repeatedly 3 #(insert-stale-row! (payload/encode ["x"]))))+ calls (atom 0)]+ (with-dynamic-fn-redefs [transport/publish-encoded!+ (fn [_ _]+ (swap! calls inc)+ (throw (q.backend/backend-unavailable-ex "backend down" {})))]+ (is (= 0 (outbox/recover-outbox!)) "nothing recovered while the backend is down"))+ (is (= 1 @calls)+ "only ONE publish is attempted — the sweep stops on the first backend-unavailable error")+ (doseq [id ids]+ (is (t2/exists? :queue_message_outbox :id id) "every row is retained for the next sweep")+ (is (= 0 (:publish_attempts (t2/select-one [:queue_message_outbox :publish_attempts] :id id)))+ "no row is bumped — a backend being down doesn't burn anyone's retry budget"))))))++(deftest recover-outbox-message-specific-failure-still-continues-past-it-test+ (testing "a message-specific (non-backend) failure is still skipped past — only backend-unavailable stops the sweep"+ (mq.tu/with-test-mq [_ctx]+ (with-redefs-fn {#'outbox/recovery-page-size 2}+ (fn []+ (let [poison (payload/encode ["poison"])+ good (payload/encode ["good"])+ _p (insert-stale-row! poison)+ good-id (insert-stale-row! good)+ real-publish (dynamic-redefs/original-fn #'transport/publish-encoded!)]+ (with-dynamic-fn-redefs [transport/publish-encoded!+ (fn [channel payload]+ (if (= payload poison)+ (throw (ex-info "message-specific boom" {})) ; NOT backend-unavailable+ (real-publish channel payload)))]+ (is (= 1 (outbox/recover-outbox!))+ "the good row behind a message-specific failure is still recovered"))+ (is (not (t2/exists? :queue_message_outbox :id good-id)) "good row published and deleted")))))))diff --git a/test/metabase/mq/queue/polling_test.clj b/test/metabase/mq/queue/polling_test.cljnew file mode 100644index 00000000000..8ac0912215c--- /dev/null+++ b/test/metabase/mq/queue/polling_test.clj@@ -0,0 +1,37 @@+(ns metabase.mq.queue.polling-test+ (:require+ [clojure.test :refer :all]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.polling :as q.polling]+ [metabase.test :as mt]))++(set! *warn-on-reflection* true)++(deftest run-recover-stale-drops-use-distinct-reason-test+ (testing "batches dropped by stale recovery are metered with their own reason (stale-recovery-exhausted) and a stringified channel — not conflated with normal delivery-retry exhaustion"+ (mt/with-prometheus-system! [_ system]+ (let [backend (reify q.backend/QueueBackend+ (backend-id [_] :queue.backend/test)+ (publish! [_ _queue _payload] nil)+ (fetch! [_ _available] nil)+ (queue-depths [_] nil)+ (batch-successful! [_ _queue _batch-id] nil)+ (failure-count [_ _queue _batch-id] nil)+ (retry-batch! [_ _queue _batch-id] nil)+ (fail-batch! [_ _queue _batch-id] nil)+ (recover-stale! [_ _stale-timeout-ms _max-retries]+ ;; keyword channel on purpose: the emitter must stringify it to match every+ ;; other batches-dropped emitter (which use (name channel))+ [{:channel :orphaned-queue :recovered 0 :failed 3}])+ (run-heartbeats! [_] nil)+ (start! [_] nil)+ (shutdown! [_] nil))]+ (#'q.polling/run-recover-stale! backend "test-backend")+ (is (pos? (or (mt/metric-value system :metabase-mq/batches-dropped+ {:channel "orphaned-queue" :reason "stale-recovery-exhausted"})+ 0))+ "stale-recovery drops use reason=stale-recovery-exhausted with a stringified channel label")+ (is (not (pos? (or (mt/metric-value system :metabase-mq/batches-dropped+ {:channel "orphaned-queue" :reason "delivery-exhausted"})+ 0)))+ "stale-recovery drops are NOT conflated with the delivery-retry exhaustion reason")))))diff --git a/test/metabase/mq/queue/quartz_test.clj b/test/metabase/mq/queue/quartz_test.cljnew file mode 100644index 00000000000..9d268bf07f6--- /dev/null+++ b/test/metabase/mq/queue/quartz_test.clj@@ -0,0 +1,330 @@+(ns metabase.mq.queue.quartz-test+ "Tests for the push-based Quartz queue backend. Quartz fires jobs on its own worker threads, so —+ unlike the poll backends exercised via `with-test-mq` — these tests register the listener on the+ root `*listeners*` (the production registration path; Quartz threads don't inherit the test+ thread's dynamic bindings) and drive the backend directly under an in-memory scheduler."+ (:require+ [clojure.test :refer :all]+ [metabase.mq.impl :as mq.impl]+ [metabase.mq.listener :as listener]+ [metabase.mq.payload :as payload]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.quartz :as q.quartz]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.settings :as mq.settings]+ [metabase.task.impl :as task.impl]+ [metabase.test :as mt]+ [metabase.test.fixtures :as fixtures])+ (:import+ (java.util Properties)+ (java.util.concurrent CountDownLatch CyclicBarrier TimeUnit)+ (org.quartz JobDataMap JobDetail JobExecutionContext JobExecutionException Scheduler Trigger)+ (org.quartz.impl StdSchedulerFactory)))++(set! *warn-on-reflection* true)++(use-fixtures :once (fixtures/initialize :db))++(defn- do-with-multithread-scheduler!+ "Runs `thunk` with a fresh in-memory (RAMJobStore) Quartz scheduler bound as the current scheduler,+ using `thread-count` worker threads. `mt/with-temp-scheduler!` only gives one thread, which can't+ demonstrate cross-thread mutual exclusion; this gives enough threads for batches to actually run+ concurrently when allowed."+ [thread-count thunk]+ (let [props (doto (Properties.)+ (.setProperty "org.quartz.scheduler.instanceName" (str "mq-quartz-test-" (random-uuid)))+ (.setProperty "org.quartz.scheduler.skipUpdateCheck" "true")+ (.setProperty "org.quartz.threadPool.class" "org.quartz.simpl.SimpleThreadPool")+ (.setProperty "org.quartz.threadPool.threadCount" (str thread-count))+ (.setProperty "org.quartz.jobStore.class" "org.quartz.simpl.RAMJobStore"))+ scheduler (.getScheduler (StdSchedulerFactory. props))]+ (binding [task.impl/*quartz-scheduler* (atom scheduler)]+ (.start scheduler)+ (try (thunk) (finally (.shutdown scheduler true))))))++(defn- wait-for!+ "Polls `pred` until it returns truthy or `timeout-ms` elapses; returns the final value."+ ([pred] (wait-for! pred 5000))+ ([pred timeout-ms]+ (let [deadline (+ (System/currentTimeMillis) timeout-ms)]+ (loop []+ (or (pred)+ (when (< (System/currentTimeMillis) deadline)+ (Thread/sleep 10)+ (recur)))))))++(defn- do-with-queue!+ "Registers `queue` (with a listener) on the root registry/listeners for the duration of `thunk`,+ cleaning both up afterwards. Mirrors how production wires a queue + listener globally."+ [queue listener thunk]+ (q.registry/register-queue! queue {:transactional :try})+ (listener/batch-listen! queue listener)+ (try+ (thunk)+ (finally+ (listener/unlisten! queue)+ (swap! q.registry/*queues* dissoc queue))))++(defn- publish! [queue & msgs]+ (q.backend/publish! q.quartz/backend queue (payload/encode (vec msgs))))++(defn- job-triggers [^Scheduler scheduler queue]+ (.getTriggersOfJob scheduler (#'q.quartz/queue-job-key queue)))++(deftest quartz-delivers-published-message-test+ (testing "a published batch is delivered to the registered listener"+ (mt/with-temp-scheduler!+ (let [received (atom [])+ queue (keyword "queue" (str "quartz-deliver-" (random-uuid)))]+ (do-with-queue!+ queue+ (fn [msgs] (swap! received into msgs))+ (fn []+ (publish! queue "a" "b" "c")+ (wait-for! #(= 3 (count @received)))+ (is (= #{"a" "b" "c"} (set @received))+ "every published message reaches the listener")))))))++(deftest quartz-retries-failed-batch-test+ (testing "a batch whose listener throws is redelivered until it succeeds"+ ;; uses the real (1s) first-retry backoff — fast enough for a single retry — rather than+ ;; redefining retry-delay-ms, which the Quartz worker thread wouldn't see.+ (mt/with-temp-scheduler!+ (let [attempts (atom 0)+ queue (keyword "queue" (str "quartz-retry-" (random-uuid)))]+ (do-with-queue!+ queue+ (fn [_msgs]+ ;; fail the first attempt, succeed on the retry+ (when (= 1 (swap! attempts inc))+ (throw (ex-info "boom" {}))))+ (fn []+ (publish! queue "retry-me")+ (is (wait-for! #(>= @attempts 2) 8000)+ "listener is invoked again after the first failure")+ (is (= 2 @attempts)+ "delivery stops once the retry succeeds")))))))++(deftest quartz-drops-after-max-retries-test+ (testing "a batch that keeps failing is dropped after queue-max-retries attempts"+ ;; cap retries at 2 so the batch drops after a single real-backoff retry instead of climbing the+ ;; exponential backoff to the default 5.+ (mt/with-temporary-setting-values [queue-max-retries 2]+ (mt/with-temp-scheduler!+ (let [attempts (atom 0)+ extra-attempt (promise) ; delivered only if an (erroneous) attempt past the max fires+ queue (keyword "queue" (str "quartz-drop-" (random-uuid)))]+ (do-with-queue!+ queue+ (fn [_msgs]+ (when (> (swap! attempts inc) (mq.settings/queue-max-retries))+ (deliver extra-attempt :extra))+ (throw (ex-info "always boom" {})))+ (fn []+ (publish! queue "doomed")+ (is (wait-for! #(= (mq.settings/queue-max-retries) @attempts) 8000)+ "the batch is attempted exactly queue-max-retries times")+ ;; An erroneous extra retry would fire after the ~2s second-level backoff. Wait on the+ ;; promise (returns early if a bad attempt fires) and assert it never resolves.+ (is (= ::none (deref extra-attempt 2500 ::none))+ "no further attempts after the batch is dropped"))))))))++(deftest quartz-no-listener-returns-marker-test+ (testing "deliver! returns the no-listener marker (not nil/success) when no listener is running, so the caller defers rather than dropping"+ (mt/with-temp-scheduler!+ (let [queue (keyword "queue" (str "quartz-nolistener-" (random-uuid)))]+ (is (= mq.impl/no-listener (mq.impl/deliver! queue (payload/encode ["orphan"]))))))))++(deftest quartz-undecodable-payload-is-dropped-test+ (testing "an undecodable payload is dropped (no throw, no retry) and metered, never reaching the listener"+ (mt/with-prometheus-system! [_ system]+ (let [called (atom false)+ queue (keyword "queue" (str "quartz-baddecode-" (random-uuid)))]+ (do-with-queue!+ queue+ (fn [_msgs] (reset! called true))+ (fn []+ (is (= mq.impl/undecodable (mq.impl/deliver! queue "not-json{{{"))+ "deliver! drops an undecodable payload (returns the undecodable marker) rather than throwing past the retry/drop logic")+ (is (false? @called)+ "the listener is never invoked for an undecodable payload")+ (is (pos? (mt/metric-value system :metabase-mq/batches-dropped+ {:channel (name queue) :reason "undecodable"}))+ "the drop is surfaced as batches-dropped{reason=undecodable}")))))))++(deftest quartz-undecodable-payload-dropped-without-listener-test+ (testing "an undecodable payload is dropped as `undecodable` even when no listener is running on this node — it is NOT treated as no-listener (which would needlessly refire a poison payload that can never decode)"+ (mt/with-prometheus-system! [_ system]+ (let [queue (keyword "queue" (str "quartz-baddecode-nolistener-" (random-uuid)))]+ ;; no listener registered here — decoding happens first, so a poison payload is dropped as+ ;; undecodable rather than deferred as no-listener+ (is (= mq.impl/undecodable (mq.impl/deliver! queue "not-json{{{"))+ "an undecodable payload returns the undecodable marker (dropped), not the no-listener marker")+ (is (pos? (mt/metric-value system :metabase-mq/batches-dropped+ {:channel (name queue) :reason "undecodable"}))+ "the drop is metered as undecodable, not deferred/mislabeled as no-listener")))))++(deftest quartz-handler-failure-returns-cause-test+ (testing "deliver! returns the handler's own exception so the caller can retry/drop with the cause"+ (mt/with-temp-scheduler!+ (let [queue (keyword "queue" (str "quartz-cause-" (random-uuid)))]+ (do-with-queue!+ queue+ (fn [_msgs] (throw (ex-info "kaboom" {:detail 42})))+ (fn []+ (let [result (mq.impl/deliver! queue (payload/encode ["x"]))]+ (is (instance? Throwable result) "a failed delivery returns the throwable, not a boolean")+ ;; the listener's exception is returned directly (no wrapping) for diagnostics+ (is (= "kaboom" (ex-message result))+ "the handler's exception is returned as-is"))))))))++(deftest deliver-batch-refires-immediately-when-handoff-fails-test+ (testing "if deliver-batch! can't durably hand off (its reschedule throws), it throws JobExecutionException with refireImmediately so Quartz re-runs the job rather than dropping the message"+ (let [queue (keyword "queue" (str "quartz-refire-" (random-uuid)))+ payload (payload/encode ["x"])+ ctx (reify JobExecutionContext+ (getMergedJobDataMap [_]+ (doto (JobDataMap.)+ (.put "queue" (name queue))+ (.put "payload" payload)+ (.put "attempt" "0")))+ (getScheduler [_] nil))]+ (q.registry/register-queue! queue {:transactional :try})+ (listener/batch-listen! queue (fn [_msgs] (throw (ex-info "handler boom" {}))))+ (try+ ;; simulate the scheduler being momentarily unavailable so the retry can't be written+ (with-redefs-fn {#'q.quartz/schedule-message-trigger! (fn [& _] (throw (ex-info "scheduler down" {})))}+ (fn []+ (let [ex (try (#'q.quartz/deliver-batch! ctx) nil+ (catch JobExecutionException e e))]+ (is (instance? JobExecutionException ex)+ "a failed hand-off throws a JobExecutionException")+ (is (true? (.refireImmediately ^JobExecutionException ex))+ "with refireImmediately set, so Quartz re-runs the job immediately"))))+ (finally+ (listener/unlisten! queue)+ (swap! q.registry/*queues* dissoc queue))))))++(deftest deliver-batch-backs-off-before-refire-test+ (testing "before refiring a failed hand-off, deliver-batch! pauses briefly so it retries at ~1/interval rather than hot-looping the worker thread"+ (let [queue (keyword "queue" (str "quartz-refire-backoff-" (random-uuid)))+ payload (payload/encode ["x"])+ backed-off (atom 0)+ ctx (reify JobExecutionContext+ (getMergedJobDataMap [_]+ (doto (JobDataMap.)+ (.put "queue" (name queue))+ (.put "payload" payload)+ (.put "attempt" "0")))+ (getScheduler [_] nil))]+ (q.registry/register-queue! queue {:transactional :try})+ (listener/batch-listen! queue (fn [_msgs] (throw (ex-info "handler boom" {}))))+ (try+ (with-redefs-fn {#'q.quartz/schedule-message-trigger! (fn [& _] (throw (ex-info "scheduler down" {})))+ #'q.quartz/backoff-before-refire! (fn [] (swap! backed-off inc))}+ (fn []+ (let [ex (try (#'q.quartz/deliver-batch! ctx) nil+ (catch JobExecutionException e e))]+ (is (instance? JobExecutionException ex)+ "a failed hand-off still throws a JobExecutionException")+ (is (true? (.refireImmediately ^JobExecutionException ex))+ "with refireImmediately set")+ (is (= 1 @backed-off)+ "the backoff pause runs exactly once before the refire is thrown"))))+ (finally+ (listener/unlisten! queue)+ (swap! q.registry/*queues* dissoc queue))))))++(defn- run-concurrency-probe!+ "Publishes two batches to `queue` whose listener parks on a `CyclicBarrier` of 2. If both batches+ run at once they trip the barrier (`met` -> true); if they're serialized the awaits time out and+ the barrier is never met. Returns whether concurrency was observed. Uses 5 scheduler threads so+ serialization, when seen, is enforced by the backend rather than a thread shortage."+ [queue exclusive?]+ (do-with-multithread-scheduler!+ 5+ (fn []+ (let [met (atom false)+ barrier (CyclicBarrier. 2)+ done (CountDownLatch. 2)]+ (q.registry/register-queue! queue (cond-> {:transactional :try}+ exclusive? (assoc :exclusive true)))+ (listener/batch-listen! queue+ (fn [_msgs]+ (try+ (.await barrier 2 TimeUnit/SECONDS)+ (reset! met true)+ (catch Exception _)+ (finally (.countDown done)))))+ (try+ (q.backend/publish! q.quartz/backend queue (payload/encode ["m1"]))+ (q.backend/publish! q.quartz/backend queue (payload/encode ["m2"]))+ (.await done 8 TimeUnit/SECONDS)+ @met+ (finally+ (listener/unlisten! queue)+ (swap! q.registry/*queues* dissoc queue)))))))++(deftest quartz-exclusive-queue-serializes-cluster-wide-test+ (testing ":exclusive queue never runs two batches at once (DisallowConcurrentExecution)"+ (is (false? (run-concurrency-probe! (keyword "queue" (str "quartz-excl-" (random-uuid))) true))+ "the two batches were serialized — concurrency was never observed")))++(deftest quartz-non-exclusive-queue-allows-concurrency-test+ (testing "a non-exclusive queue can run batches concurrently (proves the probe can observe overlap)"+ (is (true? (run-concurrency-probe! (keyword "queue" (str "quartz-conc-" (random-uuid))) false))+ "the two batches ran at the same time")))++(deftest quartz-publish-without-scheduler-throws-test+ (testing "publishing with no running scheduler throws instead of silently dropping the message"+ ;; A nil scheduler (e.g. MB_DISABLE_SCHEDULER) must NOT be a silent no-op: the transactional+ ;; outbox relies on a thrown failure to keep the row for the recovery sweep, and the publish+ ;; buffer relies on it to retry/loudly drop rather than silently lose the message.+ (binding [task.impl/*quartz-scheduler* (atom nil)]+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo #"scheduler is not running"+ (publish! (keyword "queue" (str "quartz-nosched-" (random-uuid))) "x"))))))++(deftest quartz-requeue-no-listener-reschedules-not-refires-test+ (testing "no listener on this node re-schedules the message back into the store (so it goes back through acquisition and node-affinity routes it to a node that has the listener), rather than refiring it here — a refire would loop on this node forever"+ (mt/with-temp-scheduler!+ (let [^Scheduler scheduler (task.impl/scheduler)+ queue (keyword "queue" (str "quartz-nolistener-requeue-" (random-uuid)))]+ (#'q.quartz/ensure-queue-job! scheduler queue false)+ (try+ (#'q.quartz/requeue-no-listener! scheduler queue (payload/encode ["x"]) 0)+ (let [triggers (job-triggers scheduler queue)]+ (is (= 1 (count triggers))+ "a single re-queue trigger is placed back into the store (re-acquired via affinity, not fired here)")+ (is (instance? Trigger (first triggers))))+ (finally (.deleteJob scheduler (#'q.quartz/queue-job-key queue))))))))++(deftest quartz-exclusive-job-not-downgraded-test+ (testing "a node that disagrees about :exclusive can't downgrade an existing exclusive job"+ ;; Simulates three nodes (the per-process `ensured-jobs` cache is reset between them) publishing+ ;; to the same queue with different views of `:exclusive`. Exclusivity must be sticky: once any+ ;; node makes the queue's job exclusive, a node that thinks it's concurrent must not weaken it.+ (do-with-multithread-scheduler!+ 1+ (fn []+ (let [^Scheduler scheduler (task.impl/scheduler)+ queue (keyword "queue" (str "quartz-nodowngrade-" (random-uuid)))+ jk (#'q.quartz/queue-job-key queue)+ reset-cache! #(reset! @#'q.quartz/ensured-jobs #{})+ job-class (fn [] (when-let [^JobDetail jd (.getJobDetail scheduler jk)]+ (.getSimpleName (.getJobClass jd))))]+ (try+ ;; node A: concurrent — creates a plain job+ (reset-cache!)+ (#'q.quartz/ensure-queue-job! scheduler queue false)+ (is (= "QueueMessageJob" (job-class)))+ ;; node B: exclusive — upgrades to the DisallowConcurrentExecution class+ (reset-cache!)+ (#'q.quartz/ensure-queue-job! scheduler queue true)+ (is (= "ExclusiveQueueMessageJob" (job-class)))+ ;; node C: concurrent again — must NOT downgrade+ (reset-cache!)+ (#'q.quartz/ensure-queue-job! scheduler queue false)+ (is (= "ExclusiveQueueMessageJob" (job-class))+ "exclusive is sticky; a disagreeing node cannot downgrade it back to concurrent")+ (finally (.deleteJob scheduler jk))))))))diff --git a/test/metabase/mq/queue/registry_test.clj b/test/metabase/mq/queue/registry_test.cljnew file mode 100644index 00000000000..bcb54fce0c9--- /dev/null+++ b/test/metabase/mq/queue/registry_test.clj@@ -0,0 +1,67 @@+(ns metabase.mq.queue.registry-test+ (:require+ [clojure.test :refer :all]+ [metabase.mq.queue.registry :as q.registry])+ (:import+ (clojure.lang ExceptionInfo)))++(set! *warn-on-reflection* true)++(defn- with-fresh-queues [f]+ (binding [q.registry/*queues* (atom {})]+ (f)))++(use-fixtures :each with-fresh-queues)++(deftest transactional-is-required-test+ (testing "register-queue! throws when :transactional is missing"+ (is (thrown-with-msg?+ ExceptionInfo #"(?i)transactional"+ (q.registry/register-queue! :queue/no-transactional {}))))+ (testing "register-queue! throws when :transactional is an invalid value"+ (is (thrown-with-msg?+ ExceptionInfo #"(?i)transactional"+ (q.registry/register-queue! :queue/bad-transactional {:transactional :sometimes})))))++(deftest transactional-accepts-each-mode-test+ (doseq [mode [:require :try :never]]+ (testing (str ":transactional " mode " is accepted")+ (q.registry/register-queue! (keyword "queue" (str "txn-" (name mode))) {:transactional mode})+ (is (= mode (q.registry/transactional (keyword "queue" (str "txn-" (name mode)))))))))++(deftest transactional-accessor-returns-nil-for-unknown-queue-test+ (is (nil? (q.registry/transactional :queue/never-declared))))++(deftest other-config-still-works-with-transactional-test+ (q.registry/register-queue! :queue/configured+ {:transactional :try :exclusive true :max-batch-messages 7})+ (is (true? (q.registry/exclusive? :queue/configured)))+ (is (= 7 (q.registry/max-batch-messages :queue/configured)))+ (is (= :try (q.registry/transactional :queue/configured))))++(deftest rejects-invalid-config-values-test+ (testing "register-queue! rejects a non-positive :max-batch-messages"+ (is (thrown-with-msg?+ ExceptionInfo #"max-batch-messages"+ (q.registry/register-queue! :queue/neg-batch {:transactional :try :max-batch-messages -5})))+ (is (thrown-with-msg?+ ExceptionInfo #"max-batch-messages"+ (q.registry/register-queue! :queue/zero-batch {:transactional :try :max-batch-messages 0}))))+ (testing "register-queue! rejects a non-boolean :exclusive"+ (is (thrown-with-msg?+ ExceptionInfo #"exclusive"+ (q.registry/register-queue! :queue/bad-exclusive {:transactional :try :exclusive "yes"}))))+ (testing "register-queue! rejects a non-fn :dedup-fn"+ (is (thrown-with-msg?+ ExceptionInfo #"dedup-fn"+ (q.registry/register-queue! :queue/bad-dedup {:transactional :try :dedup-fn 42}))))+ (testing "register-queue! rejects unknown/typo'd keys (closed map)"+ (is (thrown-with-msg?+ ExceptionInfo #"disallowed"+ (q.registry/register-queue! :queue/typo {:transactional :try :max-batch-mesages 3})))))++(deftest valid-config-still-registers-test+ (testing "a fully-valid config registers without error"+ (q.registry/register-queue! :queue/ok {:transactional :require :exclusive true+ :max-batch-messages 50 :dedup-fn identity})+ (is (= 50 (q.registry/max-batch-messages :queue/ok)))))diff --git a/test/metabase/mq/task/queue_reaper_test.clj b/test/metabase/mq/task/queue_reaper_test.cljnew file mode 100644index 00000000000..ba2ca923446--- /dev/null+++ b/test/metabase/mq/task/queue_reaper_test.clj@@ -0,0 +1,68 @@+(ns metabase.mq.task.queue-reaper-test+ (:require+ [clojure.test :refer :all]+ [metabase.mq.quartz-affinity :as quartz-affinity]+ [metabase.mq.task.queue-reaper :as queue-reaper]+ [metabase.test.fixtures :as fixtures]+ [toucan2.core :as t2])+ (:import+ (java.time Instant)+ (org.quartz Scheduler TriggerKey)))++(set! *warn-on-reflection* true)++;; This test reads/writes the QRTZ_* tables directly, so the app-db must be migrated.+(use-fixtures :once (fixtures/initialize :db))++(defn- insert-trigger!+ "Inserts a queue job + trigger row directly into the app-db QRTZ_* tables under `sched`."+ [sched jn state start]+ (let [jg quartz-affinity/queue-job-group]+ (t2/query (str "INSERT INTO QRTZ_JOB_DETAILS (sched_name,job_name,job_group,job_class_name,is_durable,is_nonconcurrent,is_update_data,requests_recovery)"+ " VALUES ('" sched "','" jn "','" jg "','x.Job',true,false,false,false)"))+ (t2/query (str "INSERT INTO QRTZ_TRIGGERS (sched_name,trigger_name,trigger_group,job_name,job_group,next_fire_time,priority,trigger_state,trigger_type,start_time,misfire_instr)"+ " VALUES ('" sched "','t-" jn "','" jg "','" jn "','" jg "'," start ",5,'" state "','SIMPLE'," start ",-1)"))))++(defn- cleanup-sched! [sched]+ (t2/query (str "DELETE FROM QRTZ_TRIGGERS WHERE sched_name='" sched "'"))+ (t2/query (str "DELETE FROM QRTZ_JOB_DETAILS WHERE sched_name='" sched "'")))++(deftest orphaned-trigger-rows-selects-only-stuck-queue-triggers-test+ (testing "orphaned-trigger-rows selects only queue-group WAITING triggers older than the threshold"+ (let [sched (str "REAPER_TEST_" (random-uuid))+ now (.toEpochMilli (Instant/now))+ old (- now (* 2 24 60 60 1000)) ; 2 days ago+ recent (- now 60000)] ; 1 min ago+ (try+ (insert-trigger! sched "stuck-old" "WAITING" old) ; the only one that should be reaped+ (insert-trigger! sched "fresh" "WAITING" recent) ; too recent+ (insert-trigger! sched "old-acquired" "ACQUIRED" old) ; not WAITING (being delivered)+ ;; a non-queue trigger (e.g. sync) in a different group+ (t2/query (str "INSERT INTO QRTZ_JOB_DETAILS (sched_name,job_name,job_group,job_class_name,is_durable,is_nonconcurrent,is_update_data,requests_recovery)"+ " VALUES ('" sched "','other-group','DEFAULT','x.Job',true,false,false,false)"))+ (t2/query (str "INSERT INTO QRTZ_TRIGGERS (sched_name,trigger_name,trigger_group,job_name,job_group,next_fire_time,priority,trigger_state,trigger_type,start_time,misfire_instr)"+ " VALUES ('" sched "','t-other-group','DEFAULT','other-group','DEFAULT'," old ",5,'WAITING','SIMPLE'," old ",-1)"))+ (let [threshold (- now (* 24 60 60 1000)) ; 1 day+ rows (#'queue-reaper/orphaned-trigger-rows sched threshold)]+ (is (= #{"stuck-old"} (set (map :job_name rows)))+ "only the old queue-group WAITING trigger is selected; fresh / acquired / non-queue are left alone"))+ (finally+ (cleanup-sched! sched))))))++(deftest drop-orphaned-triggers-unschedules-stuck-triggers-test+ (testing "each selected stuck trigger is unscheduled and counted"+ (let [sched (str "REAPER_DROP_" (random-uuid))+ old (- (.toEpochMilli (Instant/now)) (* 2 24 60 60 1000))+ unscheduled (atom [])+ scheduler (reify Scheduler+ (getSchedulerName [_] sched)+ (unscheduleJob [_ tk] (swap! unscheduled conj (.getName ^TriggerKey tk)) true))]+ (try+ (insert-trigger! sched "stuck-a" "WAITING" old)+ (insert-trigger! sched "stuck-b" "WAITING" old)+ (insert-trigger! sched "fresh" "WAITING" (- (.toEpochMilli (Instant/now)) 60000)) ; too recent to reap+ (is (= 2 (#'queue-reaper/drop-orphaned-triggers! scheduler))+ "both aged triggers are dropped; the fresh one is not")+ (is (= #{"t-stuck-a" "t-stuck-b"} (set @unscheduled)))+ (finally+ (cleanup-sched! sched))))))diff --git a/test/metabase/mq/test_util.clj b/test/metabase/mq/test_util.cljnew file mode 100644index 00000000000..cfa5cd07d11--- /dev/null+++ b/test/metabase/mq/test_util.clj@@ -0,0 +1,307 @@+(ns metabase.mq.test-util+ "Test fixture for exercising the MQ subsystem in tests. `with-test-mq` starts a+ fresh async memory backend and exposes a fixture context that carries the layer,+ the backend instances, and convenience fns (`flush!`, `wait-for-idle!`,+ `eventually!`) for driving delivery and waiting for quiescence.++ `do-with-test-mq!` also accepts `:duplicate-delivery? true` to wrap the+ backends with a decorator that publishes every message twice. The duplicate-+ delivery mode exists to force listeners under test to prove they handle the+ MQ's at-least-once contract — any scenario that silently relies on exactly-+ once delivery will fail loudly under this mode."+ (:require+ [clojure.test :refer :all]+ [metabase.mq.core :as mq]+ [metabase.mq.init :as mq.init]+ [metabase.mq.listener :as listener]+ [metabase.mq.memory :as memory]+ [metabase.mq.polling :as mq.polling]+ [metabase.mq.publish-buffer :as publish-buffer]+ [metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.memory :as q.memory]+ [metabase.mq.queue.polling :as q.polling]+ [metabase.mq.queue.registry :as q.registry]))++(set! *warn-on-reflection* true)++;;; ------------------------------------------- Quiescence -------------------------------------------++(defn- layer-drained?+ "Returns true if no stored memory message belongs to a channel that currently has a listener.+ Messages on channels with no listener are orphaned by design (published after `unlisten!`) and+ would otherwise block teardown. Returns true for non-memory fixtures (appdb/redis) — no layer."+ [{:keys [layer]}]+ (if (nil? layer)+ true+ (let [listeners @listener/*listeners*]+ (not-any? #(contains? listeners (:queue %)) (vals @(:messages layer))))))++(defn- unmet-idle-conditions+ "Returns a set of the idle conditions that currently do NOT hold. Empty set == idle."+ [ctx]+ (cond-> #{}+ (seq @publish-buffer/*publish-buffer*) (conj :publish-buffer-nonempty)+ (seq (q.polling/busy-channels)) (conj :busy-channels)+ (not (layer-drained? ctx)) (conj :memory-messages-pending)))++(defn wait-for-idle!+ "Blocks until the MQ fixture reaches quiescence or `timeout-ms` elapses. Throws+ an ex-info whose `:unmet` key names the conditions still holding on timeout.++ Because `poll-once!` drains memory channels *before* setting `active-handlers`+ on the next submit-delivery, a single idle observation can occur in the gap+ between the two. We require two consecutive idle observations 20 ms apart."+ ([ctx] (wait-for-idle! ctx 5000))+ ([ctx timeout-ms]+ (let [deadline (+ (System/currentTimeMillis) timeout-ms)]+ (loop [consecutive-idle 0]+ ;; Wake every active polling thread on each tick so appdb backends drain+ ;; immediately instead of waiting on their 5s default poll interval.+ (mq.polling/notify-all!)+ (let [unmet (unmet-idle-conditions ctx)]+ (cond+ (and (empty? unmet) (>= consecutive-idle 1)) :idle+ (empty? unmet) (do (Thread/sleep 20) (recur (inc consecutive-idle)))+ (< (System/currentTimeMillis) deadline) (do (Thread/sleep 10) (recur 0))+ :else (throw (ex-info "MQ did not reach idle state before timeout"+ {:timeout-ms timeout-ms :unmet unmet}))))))))++(defn- force-expire-publish-buffer!+ "Marks every accumulation entry as past its deadline so the next `flush-publish-buffer!` call drains+ them immediately."+ []+ (swap! publish-buffer/*publish-buffer*+ (fn [buf]+ (into {} (map (fn [[k v]] [k (assoc v :deadline-ms 1)])) buf))))++(defn flush!+ "Force-drains the publish buffer and waits until the MQ is idle. This is the+ standard 'move time forward' primitive for tests: publish things, call `flush!`,+ then assert on the resulting state."+ ([ctx] (flush! ctx 5000))+ ([ctx timeout-ms]+ (force-expire-publish-buffer!)+ (publish-buffer/flush-publish-buffer!)+ (wait-for-idle! ctx timeout-ms)))++(defn wait-for!+ "Polls `pred` until it returns truthy or `timeout-ms` elapses; returns the final value (truthy, or+ nil/false on timeout). Use for async conditions that aren't expressible as 'the MQ is idle' and+ that don't need the polling threads woken (unlike [[eventually!]])."+ ([pred] (wait-for! pred 5000))+ ([pred timeout-ms]+ (let [deadline (+ (System/currentTimeMillis) timeout-ms)]+ (loop []+ (or (pred)+ (when (< (System/currentTimeMillis) deadline)+ (Thread/sleep 10)+ (recur)))))))++(defn eventually!+ "Polls `pred` until truthy or `timeout-ms` elapses. Returns the final value of+ `pred`. Use when the thing you're waiting on can't be expressed as 'MQ is idle'+ — for example, when waiting for a specific count to be reached or for+ appdb-backed delivery to round-trip through the database."+ ([_ctx pred] (eventually! _ctx pred 2000))+ ([_ctx pred timeout-ms]+ (let [deadline (+ (System/currentTimeMillis) timeout-ms)]+ (loop []+ ;; Wake any polling threads so appdb backends drain promptly instead of+ ;; blocking on their production poll interval.+ (mq.polling/notify-all!)+ (let [v (pred)]+ (if v+ v+ (if (< (System/currentTimeMillis) deadline)+ (do (Thread/sleep 10) (recur))+ v)))))))++;;; ------------------------------------------- Fixture -------------------------------------------++(defn listen!+ "Test helper — registers a per-message listener by wrapping `listener-fn` so it's called+ once per message in each batch, with error isolation. Production code should use+ [[metabase.mq.core/def-listener!]] instead; this is for ad-hoc test registration where+ declaring at namespace-load time isn't convenient.++ As a convenience, auto-declares the queue (with `{:transactional :try}`) if it hasn't been+ declared yet. Tests that need other queue-level config (e.g. `:exclusive`) or a different+ `:transactional` mode should call [[q.registry/register-queue!]] explicitly before calling this."+ [channel listener-fn]+ (when (and (= "queue" (namespace channel))+ (nil? (q.registry/get-queue channel)))+ (q.registry/register-queue! channel {:transactional :try}))+ (listener/batch-listen! channel+ (fn [messages]+ (let [error (volatile! nil)]+ (doseq [m messages]+ (try (listener-fn m)+ (catch Throwable e (vreset! error e))))+ (when-let [e @error] (throw e))))))++(defn- merge-listeners!+ "Merges a user-supplied listener map on top of whatever listeners+ `register-listeners!` already registered. Values can be plain single-message+ fns (wrapped to match the vec-of-messages contract that `handle!` expects)+ or full config maps with `:listener` etc. Map-form values are passed through+ as-is — their `:listener` must already accept a vec of messages.++ Auto-declares the queue for any `:queue/*` channel that isn't already declared, matching+ the convenience semantics of [[listen!]]."+ [listeners]+ (when (seq listeners)+ (doseq [k (keys listeners)+ :when (and (= "queue" (namespace k))+ (nil? (q.registry/get-queue k)))]+ (q.registry/register-queue! k {:transactional :try}))+ (swap! listener/*listeners*+ (fn [current]+ (reduce-kv (fn [m k v]+ (assoc m k (if (fn? v)+ (if-let [existing (get current k)]+ (assoc existing :listener (partial run! v))+ {:listener (partial run! v)})+ v)))+ current+ listeners)))))++(defn- make-fixture-backends+ "Constructs backend instances for the given fixture kind. Memory gets a fresh+ isolated layer."+ [kind]+ (case kind+ :memory (let [layer (memory/make-layer)]+ {:backend :memory+ :layer layer+ :queue-be (q.memory/make-backend layer)})))++;; A defrecord (not reify) so it exposes a `:poll-context` field — the transport reads+;; `(:poll-context backend)` to wake the poll loop, and that must resolve to the inner backend's+;; poll context (the one the inner poll loop actually waits on).+(defrecord DoubleDeliveryBackend [inner poll-context]+ q.backend/QueueBackend+ (publish! [_ queue-name messages]+ (q.backend/publish! inner queue-name messages)+ (q.backend/publish! inner queue-name messages))+ (batch-successful! [_ queue-name batch-id]+ (q.backend/batch-successful! inner queue-name batch-id))+ (failure-count [_ queue-name batch-id]+ (q.backend/failure-count inner queue-name batch-id))+ (retry-batch! [_ queue-name batch-id]+ (q.backend/retry-batch! inner queue-name batch-id))+ (fail-batch! [_ queue-name batch-id]+ (q.backend/fail-batch! inner queue-name batch-id))+ (start! [_] (q.backend/start! inner))+ (shutdown! [_] (q.backend/shutdown! inner))+ (backend-id [_] (q.backend/backend-id inner))+ (fetch! [_ available-queues] (q.backend/fetch! inner available-queues))+ (recover-stale! [_ stale-timeout-ms max-retries]+ (q.backend/recover-stale! inner stale-timeout-ms max-retries))+ (run-heartbeats! [_] (q.backend/run-heartbeats! inner))+ (queue-depths [_] (q.backend/queue-depths inner)))++(defn- double-delivery-queue-backend!+ "Wraps an inner `QueueBackend` so that `publish!` calls the inner backend's+ `publish!` twice with the same payload. Used by `do-with-test-mq!` when+ `:duplicate-delivery?` is set, to force listeners under test to handle the+ MQ's at-least-once contract."+ [inner]+ (->DoubleDeliveryBackend inner (:poll-context inner)))++(defn do-with-test-mq!+ "Function form of [[with-test-mq]]. `opts` may include:++ - `:backend` `:memory` (default)+ - `:listeners` listener map, same shape as `with-sync-mq` had+ - `:duplicate-delivery?` when true, wraps both backends with a decorator+ that publishes every message twice — used to+ verify listeners handle at-least-once delivery++ The fixture forces `*publish-buffer-ms*` to 0 so publishes go straight to the+ backend without waiting for the background flush scheduler. Tests that want to+ exercise buffering behaviour must rebind it explicitly in the body."+ ([f] (do-with-test-mq! {} f))+ ([opts f]+ (let [{:keys [backend listeners duplicate-delivery?] :or {backend :memory}} opts]+ (binding [listener/*listeners* (atom {})+ q.registry/*queues* (atom {})+ publish-buffer/*publish-buffer* (atom {})+ publish-buffer/*publish-buffer-ms* 0]+ (let [backends (make-fixture-backends backend)+ queue-be (cond-> (:queue-be backends)+ duplicate-delivery? double-delivery-queue-backend!)+ handle (mq.init/start! queue-be)+ ctx (assoc backends :queue-be queue-be :handle handle)]+ (try+ (merge-listeners! listeners)+ (let [result (f ctx)]+ (flush! ctx)+ result)+ (finally+ (mq.init/stop! handle))))))))++(defmacro with-test-mq+ "Starts an MQ subsystem, runs body with `ctx-sym` bound to a fixture context+ map, then drains and shuts down.++ The binding vector is `[ctx-sym]` or `[ctx-sym opts-map]` where `opts-map`+ may contain `:backend`, `:duplicate-delivery?`, etc. — anything+ [[do-with-test-mq!]] accepts.++ An optional listener map may follow the binding vector (same shape as the+ old `with-sync-mq`). It is merged into the opts under `:listeners`.++ Use `(mq.tu/flush! ctx)` between publishes and assertions, and+ `(mq.tu/wait-for-idle! ctx)` / `(mq.tu/eventually! ctx pred)` where finer+ control is needed.++ Examples:++ (with-test-mq [ctx]+ body)++ (with-test-mq [ctx]+ {:queue/foo (fn [msg] ...)}+ body)++ (with-test-mq [ctx {:duplicate-delivery? true}]+ body)"+ {:style/indent 1}+ [[ctx-sym & [opts-expr]] & body]+ (let [[listeners body] (if (map? (first body))+ [(first body) (rest body)]+ [nil body])]+ `(do-with-test-mq! (merge ~opts-expr {:listeners ~listeners})+ (fn [~ctx-sym] ~@body))))++;;; ------------------------------------------- Fixture self-tests -------------------------------------------++(deftest duplicate-delivery-doubles-queue-messages-test+ (testing "With :duplicate-delivery? true, the queue backend delivers each message twice"+ (with-test-mq [ctx {:duplicate-delivery? true}]+ (let [received (atom [])]+ (listen! :queue/chaos-smoke #(swap! received conj %))+ (mq/with-queue :queue/chaos-smoke [q]+ (mq/put q "hello"))+ (eventually! ctx #(= 2 (count @received)) 5000)+ (is (= ["hello" "hello"] @received)+ "Single published message is delivered to the listener twice")+ (mq/unlisten! :queue/chaos-smoke)))))++(deftest duplicate-delivery-off-by-default-test+ (testing "The default fixture delivers each message exactly once"+ (with-test-mq [ctx]+ (let [received (atom [])+ dup (promise)] ; delivered only if an (unexpected) second delivery arrives+ (listen! :queue/chaos-default (fn [m]+ (when (> (count (swap! received conj m)) 1)+ (deliver dup :dup))))+ (mq/with-queue :queue/chaos-default [q]+ (mq/put q "once"))+ (eventually! ctx #(>= (count @received) 1) 5000)+ ;; A second (unexpected) delivery would resolve `dup`; wait on it (returning early if it+ ;; fires) and confirm it never does.+ (is (= ::none (deref dup 200 ::none)) "no second (duplicate) delivery")+ (is (= ["once"] @received)+ "Default fixture delivers exactly once")+ (mq/unlisten! :queue/chaos-default)))))diff --git a/test/metabase/test/initialize.clj b/test/metabase/test/initialize.cljindex 905013cd84e..782f357a2ef 100644--- a/test/metabase/test/initialize.clj+++ b/test/metabase/test/initialize.clj@@ -93,9 +93,17 @@((resolve 'metabase.test.initialize.plugins/init-test-drivers!)[:driver-deprecation-test-legacy :driver-deprecation-test-new :secret-test-driver]))+;; Initialize the MQ subsystem: sync backends, no buffering, register listeners.+;; Must happen before :db so that listeners (e.g. connection-pool-invalidated) are available+;; when database initialization triggers events.+(define-initialization :mq+ (classloader/require 'metabase.test.initialize.mq)+ ((resolve 'metabase.test.initialize.mq/init!)))+;; initializing the DB also does setup needed so the scheduler will work correctly. (Remember that the scheduler uses;; a JDBC backend!)(define-initialization :db+ (initialize-if-needed! :mq)(classloader/require 'metabase.test.initialize.db)((resolve 'metabase.test.initialize.db/init!)))diff --git a/test/metabase/test/initialize/mq.clj b/test/metabase/test/initialize/mq.cljnew file mode 100644index 00000000000..29c94bfbe64--- /dev/null+++ b/test/metabase/test/initialize/mq.clj@@ -0,0 +1,17 @@+(ns metabase.test.initialize.mq+ "Initializes the MQ subsystem for test mode: memory backends, zero publish+ buffering, and registering all `def-listener!` implementations."+ (:require+ [metabase.mq.init :as mq.init]+ [metabase.mq.publish-buffer :as publish-buffer]))++(defn init! []+ (alter-var-root #'publish-buffer/*publish-buffer-ms* (constantly 0))+ ;; Load driver.init so namespaces containing def-listener! calls are loaded+ ;; before register-listeners! (called inside mq.init/start!) iterates them.+ ;; In production, core/init.clj loads these.+ (require 'metabase.driver.init)+ ;; Drive through mq.init/start! so the worker pool, publish-buffer flush thread,+ ;; and the memory backend's polling loop all come up — otherwise MQ-driven behaviour+ ;; outside `with-test-mq` would silently drop messages onto a non-polling queue.+ (mq.init/start! :queue.backend/memory))diff --git a/test/metabase/test/util/dynamic_redefs.clj b/test/metabase/test/util/dynamic_redefs.cljindex d49283f7610..88126b1087d 100644--- a/test/metabase/test/util/dynamic_redefs.clj+++ b/test/metabase/test/util/dynamic_redefs.clj@@ -50,12 +50,14 @@(fn [& args](let [depth (get *proxy-depths* a-var 0)](when (> depth max-proxy-depth)- (throw (ex-info (str "with-dynamic-fn-redefs: runaway recursion through proxy for " a-var ". "- "This usually means the replacement fn calls the redefined var directly "- "(closing over the var resolves to the proxy, not the original). "- "Use (metabase.test.util.dynamic-redefs/original-fn " (pr-str a-var) ") "- "to capture the unpatched function.")- {:var a-var, :depth depth})))+ ;; Throw an Error, not an Exception: and a plain Exception gets swallowed by a `(catch Exception ...)` in the code under test,+ ;; turning this diagnostic into silent, confusing behavior.+ (throw (AssertionError.+ (str "with-dynamic-fn-redefs: runaway recursion through proxy for " a-var " (depth " depth "). "+ "This usually means the replacement fn calls the redefined var directly "+ "(closing over the var resolves to the proxy, not the original). "+ "Use (metabase.test.util.dynamic-redefs/original-fn " (pr-str a-var) ") "+ "to capture the unpatched function."))))(binding [*proxy-depths* (assoc *proxy-depths* a-var (inc depth))](let [current-f (dynamic-value a-var)](apply current-f args))))))diff --git a/test/metabase/test/util_test.clj b/test/metabase/test/util_test.cljindex f22080d33f3..fcfe563abe1 100644--- a/test/metabase/test/util_test.clj+++ b/test/metabase/test/util_test.clj@@ -154,7 +154,7 @@(deftest ^:parallel with-dynamic-fn-redefs-capture-bug-test(testing "A replacement that delegates through the var itself is caught as runaway recursion"(is (thrown-with-msg?- clojure.lang.ExceptionInfo+ AssertionError#"runaway recursion through proxy"(mt/with-dynamic-fn-redefs [capture-bug-target (fn [x] (capture-bug-target (inc x)))](capture-bug-target 0)))))