The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Persistent queue improvements (#77779)
Nathan Voxland8771f8eb7392
diff --git a/src/metabase/mq/README.md b/src/metabase/mq/README.mdindex 3985286a797..1c2c2d8fc25 100644--- a/src/metabase/mq/README.md+++ b/src/metabase/mq/README.md@@ -20,7 +20,7 @@ unified API.(mq/put q {:key "value"})))```-All queue config — `:exclusive`, `:max-batch-messages`, `:dedup-fn` — belongs on `def-queue!`,+All queue config — `:exclusive`, `:max-concurrent-batches`, `:max-batch-messages`, `:dedup-fn` — belongs on `def-queue!`,because it takes effect at publish time on every node, regardless of where listeners areregistered. `def-listener!` only wires up the consumer-side handler. A listener registrationthrows if its queue hasn't been declared yet — typos are caught at startup, not at first@@ -119,11 +119,116 @@ keyword values become strings, sets become vectors, dates become strings.| | Queue ||---------------|----------------------------------------------------------------------|| Delivery | At-least-once per listener |+| Ordering | **None.** See below || Retry | Yes — up to `queue-max-retries` (default 5) || Backing store | Quartz `QRTZ_*` tables; `queue_message_outbox` for transactional publishes || Backlog | Persists until processed or exhausted || Use when | Work must complete reliably |+**Guaranteed delivery is the guarantee. Ordering is not.** Batches are delivered concurrently, retried+with backoff (so a failed batch lands after ones published later), and Quartz does not fire triggers in+submission order even when it serializes them. Don't write a listener that assumes it sees messages in+publish order — and don't infer the guarantee from a test that passed, since the memory backend can+look FIFO by accident. If a queue needs serialized delivery, say so with `:max-concurrent-batches 1`+(per node) or `:exclusive true` (cluster-wide) — and note that even those give you mutual exclusion,+not order.++### Limiting concurrency++By default a queue's batches run concurrently, bounded only by the Quartz thread pool — which is+shared with sync, pulses, and every other scheduled job. A queue that can fan out (one message per+row, per chart, per whatever) will happily eat that pool. Two knobs bound it:++| Config | Bound | Hard? | Enforced by |+|------------------------------|------------------------------------------|-------|-------------|+| `:exclusive true` | 1 batch **cluster-wide** | Yes | The backend (see below) |+| `:max-concurrent-batches n` | ~n batches **per node** | No — a throttle | Trigger acquisition / `fetch!` free-slot count |+| *(neither)* | Unbounded, up to the Quartz thread pool | — | — |++`:exclusive` is a **backend** guarantee, and each backend implements it its own way — Quartz with+`@DisallowConcurrentExecution` on the job class, the memory backend by never fetching a second batch+for a queue that already has one in flight. The shared poll driver knows nothing about the flag, so a+new backend has to implement it or it silently won't hold.++**Pick one — they don't layer.** Declaring both throws at registration: `:exclusive` is already the+strictest limit there is, so a per-node cap on top of it could never bind, and the two are enforced by+completely different machinery. A queue that wants "roughly one at a time, per node" wants+`:max-concurrent-batches 1`; a queue that needs *exactly* one at a time wants `:exclusive`.++`:max-concurrent-batches` may be an int or a 0-arg fn, so a cap can be backed by a setting and tuned+live:++```clojure+(mq/def-queue! :queue/exploration-query+ {:transactional :require+ :max-batch-messages 100+ :max-concurrent-batches #(explorations.settings/explorations-worker-count)})+```++A node at its cap **stops taking the work** rather than queueing it internally: it reports the queue as+one it currently can't handle, and the affinity delegate leaves those triggers `WAITING` in the shared+store for a node with room. That's the same mechanism that keeps a node from acquiring messages for a+queue it has no listener for. It's also the only shape of limit a real broker can honor — "fetch at+most N right now" is SQS's `MaxNumberOfMessages` and a Redis multi-pop count — so poll backends express+it by asking `fetch!` for a per-queue free-slot count.++**`:max-concurrent-batches` is a throttle, not a guarantee.** Both enforcement points race the work+they're gating — Quartz decides what to acquire on its scheduler thread *before* the job body runs, and+a poll backend can hand back a batch it fetched a moment before a slot filled — so a node can end up a+batch or two over its cap. When that happens the batch is simply **delivered**. It is not bounced: a+re-queue path would need its own backoff, its own metric, and a hot-loop to avoid if the upstream+throttle ever stopped working, all to shave an overshoot that is small, bounded, and drains itself.+Reach for `:exclusive` when a limit has to actually hold.++A queue's config means the same thing on every backend — that's a contract, not a coincidence. The+backends differ in *mechanism* (Quartz pushes and filters at trigger acquisition; poll backends ask+`fetch!` for a free-slot count), never in what the config means. If you add a backend, that is the bar.++### Terminal failures: `:on-error`++There is no dead-letter queue. Once a batch exhausts `queue-max-retries` it is dropped, and without a+handler the only trace is a log line and a `batches-dropped{reason=delivery-exhausted}` counter. That+is fine for fire-and-forget work, but it silently strands anything the *producer* left behind — a row+parked in a `pending` status waiting on the handler will sit there forever.++Declare an `:on-error` handler to record the terminal failure durably:++```clojure+(mq/def-queue! :queue/run-report+ {:transactional :require+ :on-error (fn [{:keys [channel messages error attempts]}]+ (doseq [{:keys [report-id]} messages]+ ;; Only fail a report that is still pending. See "at-least-once" below: this+ ;; handler can fire for a batch a peer already completed.+ (t2/update! :model/Report {:id report-id :status "pending"}+ {:status "error" :error_message (ex-message error)})))})+```++It is called on the terminal drop only — not on each failed attempt, and never for a batch that+recovers on a retry. Exceptions it throws are logged and swallowed: the batch is+dropped either way, so a broken handler cannot wedge the queue into redelivering an exhausted batch.++**`:on-error` is at-least-once, like delivery itself. Write it to be idempotent, and to tolerate+firing for work that actually succeeded** — hence the `:status "pending"` guard above.++A message carries no identity. A retry is a fresh trigger with a new id, and the attempt counter+rides inside it, so two deliveries of one payload are two *independent* batches with two independent+retry budgets and nothing joining them. If a duplicate delivery fails while the original succeeded,+the failing copy runs out its own budget and fires `:on-error` for a batch that is already done. That+is not a corner case reserved for crashes: Quartz's cluster failover re-fires a batch onto a second+node whenever the first misses a heartbeat, which a *live* node does under a long GC pause or app-DB+latency — it is never told to stop, and its copy carries on.++`metabase.mq.queue.on-error-test` pins this+(`duplicate-delivery-fires-on-error-for-an-already-succeeded-payload-test`). Closing it needs message+identity plus a record of what has already been delivered.++The hook lives in the shared retry-vs-drop policy (`metabase.mq.queue.impl/handle-batch-failure-policy!`)+rather than in any one backend, so it is inherited by every backend — a poll backend gets it through the+shared poll driver, a push backend by calling the policy directly (as Quartz does). A backend cannot+implement a retry budget without going through that policy, so `:on-error` comes along for free with any+backend added later.+## Architecture```@@ -152,8 +257,24 @@ Delivery Engine (mq.impl)```### Idempotency-Queue listeners must be idempotent. A batch can return to `pending` after a partial failure, and the same messages will-be redelivered.+Queue listeners must be idempotent, and must be safe to run **concurrently with another copy of+themselves on another node**. Two things to hold onto:++- **A batch can be redelivered.** It returns to `pending` after a partial failure, and the same+ messages come back.+- **A batch can be delivered *twice at once*.** Quartz's clustered failover re-fires a batch onto a+ second node when the first misses a heartbeat — and a live node misses heartbeats under a long GC+ pause, app-DB latency, or pool exhaustion. It is never interrupted, so both copies run. Separately,+ a queue that isn't declared `:exclusive` has no cross-node mutual exclusion at all: N nodes × the+ Quartz thread pool can be in the same listener at once.++A "have I already done this?" check is therefore not enough on its own — two copies can both check,+both see no, and both proceed. Where that matters, make the *write* safe rather than the check:+condition the update on the state that justified the message (`{:id x :status "pending"}`), reserve+via a unique constraint, or take a row lock and re-check inside the transaction.++`with-test-mq`'s `{:duplicate-delivery? true}` (below) delivers every message twice, which will catch+a listener that assumed exactly-once.## Testingdiff --git a/src/metabase/mq/core.clj b/src/metabase/mq/core.cljindex 6dca4a67640..ebcb62da7ca 100644--- a/src/metabase/mq/core.clj+++ b/src/metabase/mq/core.clj@@ -1,9 +1,13 @@(ns metabase.mq.core"Unified public API for the Metabase messaging subsystem.- **Queue** — single-consumer, at-least-once delivery. Each message is processed by exactly- one listener and removed after successful processing. Failed messages are retried up to a- configurable limit.+ **Queue** — single-consumer, at-least-once delivery. A queue has exactly one registered listener,+ which is handed each message and removed after successful processing. Failed messages are retried+ up to a configurable limit.++ \"Single-consumer\" is about *registration*, not delivery: it means one handler per queue, not that+ a message is handled exactly once. Delivery is at-least-once, so a listener must be idempotent —+ see the `Idempotency` section of `metabase/mq/README.md`.A queue is declared once with `def-queue!` — this is the source of truth for what queuesexist and where all queue config lives (exclusivity, batch size, dedup). The consumer-sidediff --git a/src/metabase/mq/impl.clj b/src/metabase/mq/impl.cljindex da5d3c6abbb..ad62267f902 100644--- a/src/metabase/mq/impl.clj+++ b/src/metabase/mq/impl.clj@@ -52,7 +52,7 @@(listener messages)(analytics/inc! :metabase-mq/batches-handled (assoc labels :status "success"))nil- (catch Exception e+ (catch Throwable e(log/error e (str "Error handling " (namespace channel) " message") labels)(analytics/inc! :metabase-mq/batches-handled (assoc labels :status "error"))e)diff --git a/src/metabase/mq/init.clj b/src/metabase/mq/init.cljindex a7f0f4ab3d5..b0bee9763ee 100644--- a/src/metabase/mq/init.clj+++ b/src/metabase/mq/init.clj@@ -3,7 +3,6 @@(:require[metabase.mq.listener :as listener][metabase.mq.publish-buffer :as publish-buffer]- [metabase.mq.quartz-affinity :as quartz-affinity][metabase.mq.queue.backend :as q.backend][metabase.mq.queue.memory :as q.memory][metabase.mq.queue.polling :as q.polling]@@ -13,7 +12,6 @@[metabase.mq.task.outbox][metabase.mq.task.queue-reaper][metabase.startup.core :as startup]- [metabase.task.bootstrap :as task.bootstrap][metabase.util.log :as log]))(def ^:private queue-backends@@ -22,11 +20,6 @@(def ^:private valid-queue-backends (set (keys queue-backends)))-;; Install the queue node-affinity Quartz DriverDelegate when Quartz's JDBC properties are set.-;; Registered at load time — `mq` depends on `task`, not the reverse, so `task.bootstrap` calls this-;; rather than referencing `mq`. install-delegate! falls back to the plain per-DB delegate if the affinity subclass can't be loaded.-(task.bootstrap/register-jdbc-property-setter! quartz-affinity/install-delegate!)-(defn- resolve-backend [label table kw-or-instance](if (keyword? kw-or-instance)(or (get table kw-or-instance)@@ -59,13 +52,6 @@(log/infof "Queue backend: %s" queue-be)(q.registry/register-queues!)(listener/register-listeners!)- ;; With the Quartz backend, tell the affinity delegate which queues this node can handle, so it- ;; only acquires triggers for queues we have a listener for. Queried live, so dynamic- ;; register/unlisten is reflected. Harmless (never set) for non-Quartz backends, which have no- ;; queue jobs in the Quartz store.- (when (= q.quartz/backend-id (q.backend/backend-id queue-instance))- (quartz-affinity/set-capability-fn!- (fn [] (into #{} (map name) (listener/queue-names)))))(let [owns-buffer-flush? (publish-buffer/start-publish-buffer-flush!)owns-worker-pool? (q.polling/start-worker-pool!)](q.backend/start! queue-instance)diff --git a/src/metabase/mq/memory.clj b/src/metabase/mq/memory.cljindex 82fc07c5ab9..2b5ff466203 100644--- a/src/metabase/mq/memory.clj+++ b/src/metabase/mq/memory.clj@@ -6,7 +6,9 @@non-durable.Tests construct isolated layers with [[make-layer]]; the poll context / poll thread lives on the- backend record, driven by [[metabase.mq.queue.polling]]. This layer only owns the storage.")+ backend record, driven by [[metabase.mq.queue.polling]]. This layer only owns the storage."+ (:require+ [metabase.util :as u]))(set! *warn-on-reflection* true)@@ -27,16 +29,65 @@(let [id (swap! next-id inc)](swap! messages assoc id {:id id :queue channel-name :payload payload :failures 0})))-(defn take-oldest- "Returns the oldest stored message for `channel-name` as a `{:queue :payload :batch-id}`- descriptor, or nil if there are none. The message stays in the store (tracked by id) until it's- acked or dropped — the poll loop's channel-busy gate keeps it from being re-fetched while it's- in-flight."+(defn take-pending!+ "Claims up to `n` not-yet-claimed messages for `channel-name`, marking each in-flight and returning them as+ `{:queue :payload :batch-id}` descriptors (possibly empty). A nil `n` means unbounded.++ Claiming is what makes it safe to fetch more than one batch per channel at a time: a message stays+ in the store until it's acked or dropped, so without an in-flight mark a second fetch — issued while+ the first is still being delivered — would hand out the same message again. `:claimed-at` starts a+ timer, so [[clear-stale-claims!]] can recover a claim whose delivery never completed.++ The claim is derived from `swap-vals!`'s before/after states rather than captured inside the swap+ fn, so a retried CAS can't report a message as claimed twice."+ [{:keys [messages]} channel-name n]+ (if (and n (not (pos? n)))+ []+ (let [claimable? (fn [msg] (and (= channel-name (:queue msg)) (not (:claimed-at msg))))+ timer (u/start-timer)+ [old new] (swap-vals! messages+ (fn [m]+ (let [available (filter claimable? (vals m))+ claiming (if n (take n available) available)]+ (reduce #(assoc-in %1 [%2 :claimed-at] timer)+ m+ (map :id claiming)))))]+ (into []+ (comp (filter (fn [[id msg]] (and (:claimed-at msg)+ (not (:claimed-at (get old id))))))+ (map (fn [[id msg]] {:queue channel-name :payload (:payload msg) :batch-id id})))+ new))))++(defn has-claim?+ "True if any message for `channel-name` is currently claimed — i.e. this backend has handed one out+ and has not yet seen it acked or retried.++ Lets a backend serialize a channel at the store, which is how [[metabase.mq.queue.memory]] implements+ `:exclusive`."[{:keys [messages]} channel-name]- (when-let [{:keys [id payload]} (->> (vals @messages)- (filter #(= channel-name (:queue %)))- first)]- {:queue channel-name :payload payload :batch-id id}))+ (boolean (some (fn [msg] (and (= channel-name (:queue msg)) (:claimed-at msg)))+ (vals @messages))))++(defn clear-stale-claims!+ "Un-claims messages that have been in-flight longer than `stale-timeout-ms`, making them fetchable+ again. Returns the number recovered.++ Stands in for the visibility timeout a real broker gives you: without it, a delivery that never+ reaches its ack (its worker thread was killed, or the driver claimed a slot it then couldn't fill)+ would leave the message marked in-flight and unfetchable forever."+ [{:keys [messages]} stale-timeout-ms]+ (let [stale? (fn [msg] (when-let [timer (:claimed-at msg)]+ (> (u/since-ms timer) stale-timeout-ms)))+ [old new] (swap-vals! messages+ (fn [m]+ (reduce-kv (fn [acc id msg]+ (cond-> acc+ (stale? msg) (-> (update-in [id :failures] inc)+ (update id dissoc :claimed-at))))+ m+ m)))]+ (count (filter (fn [[id msg]] (and (:claimed-at msg) (not (:claimed-at (get new id)))))+ old))))(defn message-failures"Returns how many times message `id` has already failed, or nil if it's no longer stored."@@ -49,9 +100,17 @@(swap! messages dissoc id))(defn retry!- "Increments the failure count of message `id`, leaving it stored so it gets re-delivered."+ "Increments the failure count of message `id`, leaving it stored so it gets re-delivered.++ Clearing `:claimed-at` is what makes it re-deliverable: the failed delivery claimed the message,+ and a claim that is never released is never fetched again."[{:keys [messages]} id]- (swap! messages (fn [m] (if (contains? m id) (update-in m [id :failures] inc) m))))+ (swap! messages (fn [m]+ (if (contains? m id)+ (-> m+ (update-in [id :failures] inc)+ (update id dissoc :claimed-at))+ m))))(defn depths"Returns a `{:channel :status :count}` row per channel describing how many messages are stored."diff --git a/src/metabase/mq/queue/backend.clj b/src/metabase/mq/queue/backend.cljindex 3a2cd3c6857..de37cd82066 100644--- a/src/metabase/mq/queue/backend.clj+++ b/src/metabase/mq/queue/backend.clj@@ -15,9 +15,17 @@(publish! [this queue-name payload]"Publishes a payload to the given queue. `payload` is an opaque, already-encoded string.The backend stores/transports it without looking inside.")- (fetch! [this available-queue-names]- "Fetches up to one batch per given queue. Returns a seq of- `{:queue :payload :batch-id}` maps (or nil/empty when there's nothing to do).")+ (fetch! [this queue->free-slots]+ "Fetches work for the queues in `queue->free-slots`, a map of queue name → how many batches this+ node may take for it right now. Fetch no more than that many per queue: the count is the node's+ remaining capacity, and anything beyond it must be left in the store for a node that can actually+ run it.++ A count of `nil` means the queue declared no `:max-concurrent-batches` and is therefore unbounded:+ take as many as you have. Queues already at capacity aren't in the map at all, so a count is+ never zero.++ Returns a seq of `{:queue :payload :batch-id}` maps (or nil/empty when there's nothing to do).")(queue-depths [this]"Returns a seq of `{:channel :status :count}` maps describing current queue depth.")(batch-successful! [this queue-name batch-id]diff --git a/src/metabase/mq/queue/concurrency.clj b/src/metabase/mq/queue/concurrency.cljnew file mode 100644index 00000000000..d402033a796--- /dev/null+++ b/src/metabase/mq/queue/concurrency.clj@@ -0,0 +1,145 @@+(ns metabase.mq.queue.concurrency+ "Per-node in-flight accounting for queues that declare `:max-concurrent-batches`.++ **The cap is a soft limit — a throttle, not a guarantee.** It is enforced by not *taking* work.++ What it deliberately does *not* do is refuse a batch that reaches us anyway. If a backend is racy a node+ can end up a batch or two over its cap. That is fine.++ A queue that declares no cap is unbounded — on every backend. It is never at capacity and reports no+ fetch limit.++ This is a per-node cap, not a cluster-wide one. For cluster-wide mutual exclusion see the queue's+ `:exclusive` flag — a queue picks one or the other, never both (the registry rejects a config that+ declares both). `:exclusive` *is* a hard guarantee; this is not."+ (:require+ [metabase.mq.listener :as listener]+ [metabase.mq.queue.registry :as q.registry])+ (:import+ (java.util.concurrent Callable ExecutorService)))++(set! *warn-on-reflection* true)++(def ^:dynamic *in-flight*+ "channel → how many batches this node is currently delivering for it. A channel with nothing in+ flight is absent rather than zero, so the map stays small.++ Test fixtures rebind this to a fresh atom so in-flight counts don't leak across scenarios (matching+ `registry/*queues*` and `listener/*listeners*`). Production code uses the root binding."+ (atom {}))++(defn in-flight+ "How many batches of `channel` this node is delivering right now."+ [channel]+ (get @*in-flight* channel 0))++(defn- at-limit?+ "True if `counts` already has `limit` batches of `channel` in flight. A nil `limit` means the queue+ declared no cap, which is never at the limit."+ [counts channel limit]+ (boolean (and limit (>= (get counts channel 0) limit))))++(defn- take-slot!+ "Records that this node has started delivering a batch of `channel`."+ [channel]+ (swap! *in-flight* update channel (fnil inc 0))+ nil)++(defn- release-slot!+ "Records that a batch of `channel` has finished delivering.++ Floors at zero rather than going negative: a double-release must not manufacture throughput the+ queue never declared."+ [channel]+ (swap! *in-flight*+ (fn [counts]+ (let [remaining (dec (get counts channel 0))]+ (if (pos? remaining)+ (assoc counts channel remaining)+ (dissoc counts channel)))))+ nil)++(defn do-with-slot+ "Function form of [[with-slot]]."+ [channel f]+ (take-slot! channel)+ (try+ (f)+ (finally+ (release-slot! channel))))++(defmacro with-slot+ "Runs `body` while counting one in-flight delivery of `channel` against the node, and stops counting+ it however `body` exits. Returns `body`'s value."+ {:style/indent 1}+ [channel & body]+ `(do-with-slot ~channel (fn [] ~@body)))++(defn submit-with-slot!+ "The asynchronous counterpart to [[with-slot]], for a caller that hands delivery off to a thread+ pool: the slot is taken *here*, on the calling thread, but released on the worker thread once+ `body-fn` finishes.++ Taking it here rather than inside the task matters — it means a batch counts against the node from+ the moment it is handed off, so the very next [[free-slots]]/[[at-capacity?]] check sees it, instead+ of the driver fetching more work in the window before the worker thread gets going.++ Submits `body-fn` (0-arg) to `executor` to run holding the slot. The slot is released however+ `body-fn` exits, and `after-release` (0-arg) then runs on that same thread — that is where 'a slot+ just freed up, go look for more work' belongs, since it must not run while the slot is still held.++ If the *submit itself* throws — the pool is shut down, so `body-fn` will never run and never release+ — the slot is released before the exception propagates. A failed hand-off must not leak capacity.++ Dynamic bindings are conveyed to the worker thread, so the slot is released against the very+ [[*in-flight*]] that took it (rather than the root binding, which would corrupt test isolation and+ leak a slot in the fixture's map)."+ [channel ^ExecutorService executor body-fn after-release]+ (take-slot! channel)+ (let [^Callable task (bound-fn []+ (try+ (body-fn)+ (finally+ (release-slot! channel)+ (after-release))))]+ (try+ (.submit executor task)+ (catch Throwable t+ (release-slot! channel)+ (throw t)))))++(defn at-capacity?+ "True if this node is already delivering at least `channel`'s `:max-concurrent-batches`. A queue with+ no declared cap is never at capacity."+ [channel]+ (at-limit? @*in-flight* channel (q.registry/max-concurrent-batches channel)))++(defn working?+ "True if `channel` has at least one delivery in flight on this node. Ask this to find out whether anything is+ still running (shutdown, test quiescence); ask [[at-capacity?]] to decide whether to take more work."+ [channel]+ (pos? (in-flight channel)))++(defn working-channels+ "The set of channels with at least one delivery in flight on this node."+ []+ (set (keys @*in-flight*)))++(defn free-slots+ "How many more batches of `channel` this node may take right now: its `:max-concurrent-batches` minus+ what's in flight, floored at zero — or **nil when the queue declares no cap**, meaning take as many+ as you have.++ A backend may fetch *fewer* than this — that's how a poll backend implements `:exclusive`, which is a+ backend concern and deliberately not modelled here."+ [channel]+ (when-let [limit (q.registry/max-concurrent-batches channel)]+ (max 0 (- limit (in-flight channel)))))++(defn takeable-queues+ "The queues this node will take work for *right now*: the ones it has a listener for, minus the ones+ it is [[at-capacity?]] on.++ Recomputed per call, so a listener registering and a slot freeing up both take effect immediately."+ []+ (remove at-capacity? (listener/queue-names)))diff --git a/src/metabase/mq/queue/impl.clj b/src/metabase/mq/queue/impl.cljindex 2eb159fdcd6..4dc1f344fd4 100644--- a/src/metabase/mq/queue/impl.clj+++ b/src/metabase/mq/queue/impl.clj@@ -2,28 +2,58 @@"Queue-specific delivery policy — the queue counterpart to the transport-agnostic delivery core in [[metabase.mq.impl]]."(:require[metabase.analytics-interface.core :as analytics]+ [metabase.mq.payload :as payload]+ [metabase.mq.queue.registry :as q.registry][metabase.mq.settings :as mq.settings]+ [metabase.util :as u][metabase.util.log :as log]))(set! *warn-on-reflection* true)+(defn- run-on-error!+ "Invokes the queue's `:on-error` terminal-failure hook, if it declares one, for a batch that is+ about to be dropped. `payload` is the still-encoded batch.++ The hook is a place to *record* a terminal failure, not a veto over it, so a handler that throws is+ logged and the batch is dropped anyway. That log is worth keeping: a throwing handler means the row+ the producer parked in `pending` is now stranded, and this line is the only trace of why."+ [channel payload error attempts]+ (when-let [handler (q.registry/on-error channel)]+ (try+ (handler {:channel channel+ :messages (payload/decode payload)+ :error error+ :attempts attempts})+ (catch Throwable t+ (log/error t "Queue :on-error handler threw; still dropping the batch" {:channel channel})))))+(defn handle-batch-failure-policy!"Retry-vs-drop policy for a just-failed queue batch. `failures` is the number of attempts that havealready failed, so the just-failed attempt makes `(inc failures)` total. `error` is the exception- that failed the batch, carried through so the give-up is diagnosable. When `(inc failures)` reaches- `queue-max-retries` the batch is dropped (error-logged *with the cause* + `batches-dropped`- `reason=delivery-exhausted`) by calling `on-drop`; otherwise `batches-retried` `reason=delivery` is- emitted, a concise retry line is logged, and `on-retry` re-enqueues it."- [channel failures error on-retry on-drop]- (let [max-retries (mq.settings/queue-max-retries)]- (if (>= (inc failures) max-retries)+ that failed the batch, carried through so the give-up is diagnosable. `payload` is the encoded+ batch, needed only to hand the messages to the queue's `:on-error` hook on the drop path.++ When `(inc failures)` reaches `queue-max-retries` the batch is dropped: the queue's `:on-error`+ hook (if any) runs first so the owner can record the terminal failure durably, then the drop is+ error-logged *with the cause* + `batches-dropped` `reason=delivery-exhausted` and `on-drop` is+ called. Otherwise `batches-retried` `reason=delivery` is emitted, a concise retry line is logged,+ and `on-retry` re-enqueues it."+ [channel payload failures error on-retry on-drop]+ (let [max-retries (mq.settings/queue-max-retries)+ attempts (inc failures)]+ (if (>= attempts max-retries)(do- (log/error error "Dropping queue batch after exhausting retries"- {:channel channel :max-retries max-retries})- (analytics/inc! :metabase-mq/batches-dropped {:channel (name channel) :reason "delivery-exhausted"})+ (run-on-error! channel payload error attempts)+ (u/ignore-exceptions+ (log/error error "Dropping queue batch after exhausting retries"+ {:channel channel :max-retries max-retries})+ (analytics/inc! :metabase-mq/batches-dropped+ {:channel (name channel) :reason "delivery-exhausted"}))(on-drop))(do- (log/warnf "Queue batch for %s failed attempt %d of %d; will retry with backoff."- channel (inc failures) max-retries)- (analytics/inc! :metabase-mq/batches-retried {:channel (name channel) :reason "delivery"})+ (u/ignore-exceptions+ (log/warnf "Queue batch for %s failed attempt %d of %d; will retry with backoff."+ channel attempts max-retries)+ (analytics/inc! :metabase-mq/batches-retried+ {:channel (name channel) :reason "delivery"}))(on-retry)))))diff --git a/src/metabase/mq/queue/memory.clj b/src/metabase/mq/queue/memory.cljindex 09a106ecb74..abac3a12f4a 100644--- a/src/metabase/mq/queue/memory.clj+++ b/src/metabase/mq/queue/memory.clj@@ -11,6 +11,7 @@[metabase.mq.memory :as memory][metabase.mq.queue.backend :as q.backend][metabase.mq.queue.polling :as q.polling]+ [metabase.mq.queue.registry :as q.registry][metabase.util.log :as log]))(set! *warn-on-reflection* true)@@ -19,6 +20,22 @@"This backend's identifier, registered with `metabase.mq.init`; its `name` labels metrics.":queue.backend/memory)+(defn- fetch-count+ "How many batches to take for `queue` this pass: normally the driver's `free-slots` (nil = unbounded).++ An `:exclusive` queue is capped at one in flight — and only if none already is. That's this+ backend's implementation of the flag; the poll driver has no notion of exclusivity, and shouldn't:+ it's the backend that owns delivery, so it's the backend that has to enforce mutual exclusion. Being+ single-process, one-per-node here *is* the cluster-wide guarantee the flag promises.++ Safe to check-then-take without a lock: only the poll thread claims messages, and everything else+ (ack, retry, stale recovery) only ever *releases* claims — so the count can fall between the check+ and the take, never rise."+ [layer queue free-slots]+ (if (q.registry/exclusive? queue)+ (if (memory/has-claim? layer queue) 0 1)+ free-slots))+(defrecord MemoryQueueBackend [layer poll-context]q.backend/QueueBackend@@ -27,8 +44,11 @@(publish! [_this queue-name payload](memory/publish! layer queue-name payload))- (fetch! [_this available-queues]- (into [] (keep #(memory/take-oldest layer %)) available-queues))+ (fetch! [_this queue->free-slots]+ (into []+ (mapcat (fn [[queue free-slots]]+ (memory/take-pending! layer queue (fetch-count layer queue free-slots))))+ queue->free-slots))(queue-depths [_this](memory/depths layer))@@ -46,15 +66,23 @@(fail-batch! [_this _queue-name batch-id](memory/ack! layer batch-id))- ;; No crashed-consumer recovery (single process), but this periodic maintenance hook is reused to- ;; reclaim messages for queues that have no listener: on a single-process, non-durable backend such- ;; messages can never be delivered, so leaving them would grow the store and the depth gauge without- ;; bound (unlike a persistent, cross-node backend, where another node may own the listener).- (recover-stale! [_this _stale-timeout-ms _max-retries]+ ;; There is no crashed *node* to recover from (single process), but there is still a crashed+ ;; *delivery*: a message claimed by `take-pending!` whose worker never acked it would stay marked+ ;; in-flight and never be fetched again. Clearing those stale claims is the in-memory equivalent of+ ;; a broker's visibility timeout.+ ;;+ ;; The hook is also reused to reclaim messages for queues that have no listener: on a+ ;; single-process, non-durable backend such messages can never be delivered, so leaving them would+ ;; grow the store and the depth gauge without bound (unlike a persistent, cross-node backend, where+ ;; another node may own the listener).+ (recover-stale! [_this stale-timeout-ms _max-retries](let [dropped (memory/drop-orphaned! layer (set (listener/queue-names)))](when (pos? dropped)(log/debugf "Dropped %d orphaned in-memory message(s) for queues with no listener" dropped)))- nil)+ (let [recovered (memory/clear-stale-claims! layer stale-timeout-ms)]+ (when (pos? recovered)+ (log/warnf "Recovered %d in-memory message(s) whose delivery never completed" recovered)+ [{:channel :memory :recovered recovered :failed 0}])))(run-heartbeats! [_this] nil)diff --git a/src/metabase/mq/queue/polling.clj b/src/metabase/mq/queue/polling.cljindex 3653013e0a1..e0d4a0acecb 100644--- a/src/metabase/mq/queue/polling.clj+++ b/src/metabase/mq/queue/polling.clj@@ -7,40 +7,24 @@(:require[metabase.analytics-interface.core :as analytics][metabase.mq.impl :as mq.impl]- [metabase.mq.listener :as listener][metabase.mq.polling :as mq.polling][metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.concurrency :as q.concurrency][metabase.mq.queue.impl :as q.impl][metabase.mq.settings :as mq.settings][metabase.util.log :as log])(:import- (java.util.concurrent Callable ExecutorService Executors Future)))+ (java.util.concurrent ExecutorService Executors)))(set! *warn-on-reflection* true)-;;; ------------------------------- Worker pool & in-flight (busy) tracking ---------------------------------(def ^:private active-handlers- "channel keyword → {:future Future, :gen Object}. Tracks which channels have an in-flight delivery- on the worker pool, so a channel is delivered by at most one worker at a time (and so publishers- know not to wake the loop for a channel that's already being processed)."- (atom {}))+;;; ------------------------------- Worker pool -------------------------------(def ^:private worker-pool"Shared thread pool that runs message delivery off the poll thread, so a slow listener doesn'tstall polling."(atom nil))-(defn channel-busy?- "Returns true if `channel` currently has an in-flight delivery on the worker pool."- [channel]- (contains? @active-handlers channel))--(defn busy-channels- "Returns the set of channels that currently have an in-flight delivery."- []- (set (keys @active-handlers)))-(defn start-worker-pool!"Starts the shared worker pool for non-blocking message delivery. Idempotent.Returns `true` if THIS call created the pool, `false` if one was already running."@@ -51,24 +35,26 @@(do (.shutdown ^ExecutorService pool) false))))(defn shutdown-worker-pool!- "Cancels all active handler futures and shuts down the worker pool."+ "Shuts down the worker pool, interrupting any deliveries still running, and clears the in-flight+ counts.++ `shutdownNow` interrupts every actively-executing task, so there is nothing to cancel by hand. The+ interrupted deliveries still run their `finally` and release their slots."[]- (doseq [[_ {:keys [^Future future]}] @active-handlers]- (when future- (.cancel future true)))- (reset! active-handlers {})(when-let [^ExecutorService pool @worker-pool](.shutdownNow pool)- (reset! worker-pool nil)))+ (reset! worker-pool nil))+ (reset! q.concurrency/*in-flight* {}))(defn- handle-batch-failure!"Decides retry vs. permanent failure for a just-failed batch based on its prior failure count, thentells the backend to re-enqueue or drop it. No-ops if the backend no longer owns/knows the batch(`failure-count` returns nil). `error` is the exception that failed the batch, passed through for- diagnostic logging."- [backend channel batch-id error]+ diagnostic logging; `payload` is the encoded batch, passed through so the shared policy can hand+ the messages to the queue's `:on-error` hook if it drops them."+ [backend channel payload batch-id error](when-let [failures (q.backend/failure-count backend channel batch-id)]- (q.impl/handle-batch-failure-policy! channel failures error+ (q.impl/handle-batch-failure-policy! channel payload failures error#(q.backend/retry-batch! backend channel batch-id)#(q.backend/fail-batch! backend channel batch-id))))@@ -82,60 +68,22 @@(let [result (mq.impl/deliver! channel payload)](cond(= mq.impl/no-listener result) nil- (instance? Throwable result) (when batch-id (handle-batch-failure! backend channel batch-id result))- :else (when batch-id (q.backend/batch-successful! backend channel batch-id)))))+ (instance? Throwable result) (when batch-id (handle-batch-failure! backend channel payload batch-id result))+ :else (when batch-id (q.backend/batch-successful! backend channel batch-id)))))(defn submit-delivery!- "Submits one fetched batch to the worker pool for non-blocking delivery. Returns true if submitted,- false if `channel` already has an in-flight delivery (enforcing at most one worker per channel).- Uses `bound-fn` to convey dynamic bindings to the worker thread.-- The busy-check and slot claim are atomic via a single `swap-vals!`. A per-submission generation- object prevents a completed future's cleanup from clobbering a resubmission that won the slot- between the finally and the dissoc."+ "Submits one fetched batch to the worker pool for non-blocking delivery."[channel payload batch-id backend]- (let [gen (Object.)- [old _] (swap-vals! active-handlers- (fn [handlers]- (if (contains? handlers channel)- handlers- (assoc handlers channel {:future nil :gen gen}))))]- (if (contains? old channel)- false- (let [^Callable task (bound-fn []- (try- (deliver-and-ack! channel payload batch-id backend)- (finally- ;; Only remove if this generation still owns the slot- (swap! active-handlers- (fn [handlers]- (if (identical? gen (:gen (get handlers channel)))- (dissoc handlers channel)- handlers)))- (mq.polling/notify-all!))))- f (try- (.submit ^ExecutorService @worker-pool task)- (catch Throwable t- ;; The slot was already claimed above. If the submit itself fails (e.g. the- ;; worker pool isn't running / was shut down, so the deref or .submit throws),- ;; release the slot we claimed before propagating — otherwise the channel stays- ;; marked busy forever and the queue silently stops delivering.- (swap! active-handlers- (fn [handlers]- (if (identical? gen (:gen (get handlers channel)))- (dissoc handlers channel)- handlers)))- (throw t)))]- ;; Only set the future if this generation still owns the slot- (swap! active-handlers- (fn [handlers]- (if (identical? gen (:gen (get handlers channel)))- (assoc-in handlers [channel :future] f)- handlers)))- true))))+ (q.concurrency/submit-with-slot!+ channel+ @worker-pool+ #(deliver-and-ack! channel payload batch-id backend)+ ;; Runs once the slot is released: a freed slot may mean there's room to fetch more, so re-drive the+ ;; poll loop rather than waiting out its idle interval.+ mq.polling/notify-all!))(def ^:private stale-check-interval-ms (* 60 1000))-(def ^:private heartbeat-interval-ms (* 2 60 1000))+(def ^:private heartbeat-interval-ms (* 2 60 1000))(def ^:private depth-gauge-interval-ms (* 30 1000))(def ^:private stale-processing-timeout-ms@@ -172,14 +120,15 @@{:channel (name channel) :reason "stale-recovery-exhausted"} failed))))))(defn notify-on-publish!- "Wakes the poll loop for `poll-context` so it picks up a freshly published message — unless- `channel` is already being processed, in which case the loop re-checks when that finishes.- Called by the transport right after `publish!` (the caller reads the backend's `:poll-context`).+ "Wakes the poll loop for `poll-context` so it picks up a freshly published message — unless `channel`+ is already at its concurrency cap, in which case the loop re-checks when one of those deliveries+ finishes and frees a slot. Called by the transport right after `publish!` (the caller reads the+ backend's `:poll-context`).No-ops when `poll-context` is nil: push backends (e.g. Quartz) have no poll loop to wake — their`publish!` already arranges delivery — so they carry no poll context."[poll-context channel]- (when (and poll-context (not (channel-busy? channel)))+ (when (and poll-context (not (q.concurrency/at-capacity? channel)))(mq.polling/notify! (:poll-state poll-context))))(defn make-poll-context@@ -194,10 +143,22 @@:last-heartbeat-ms (atom 0):last-depth-gauge-ms (atom 0)})+(defn- free-slots-by-queue+ "The poll driver's shape of [[q.concurrency/takeable-queues]]: queue → how many batches this node may+ still take for it. A queue with no declared cap maps to `nil` — unbounded, take what you have.++ Queues at capacity are absent entirely rather than mapped to 0, so [[q.backend/fetch!]] is never even+ asked about them: a node at its limit on a queue simply doesn't take more of its work, leaving it in+ the store for a node that can run it."+ []+ (into {}+ (map (juxt identity q.concurrency/free-slots))+ (q.concurrency/takeable-queues)))+(defn- poll-iteration!- "One iteration: run each maintenance task if its interval has elapsed, then fetch and submit- one batch per available queue. Returns true if any work was found so the polling- thread respins immediately."+ "One iteration: run each maintenance task if its interval has elapsed, then fetch and submit up to+ each queue's free-slot count. Returns true if any work was found so the polling thread respins+ immediately."[backend {:keys [last-stale-check-ms last-heartbeat-ms last-depth-gauge-ms]}](let [backend-name (name (q.backend/backend-id backend))](mq.polling/periodically! last-stale-check-ms stale-check-interval-ms "stale recovery"@@ -207,8 +168,8 @@(mq.polling/periodically! last-depth-gauge-ms depth-gauge-interval-ms "depth gauge"#(report-depths! backend backend-name))(let [found? (boolean- (when-let [available (seq (remove channel-busy? (listener/queue-names)))]- (when-let [batches (seq (q.backend/fetch! backend available))]+ (when-let [wanted (not-empty (free-slots-by-queue))]+ (when-let [batches (seq (q.backend/fetch! backend wanted))](doseq [{:keys [queue payload batch-id]} batches](submit-delivery! queue payload batch-id backend))true)))]diff --git a/src/metabase/mq/queue/quartz.clj b/src/metabase/mq/queue/quartz.cljindex 1c0d8242238..00e560ede62 100644--- a/src/metabase/mq/queue/quartz.clj+++ b/src/metabase/mq/queue/quartz.clj@@ -30,9 +30,12 @@[clojurewerkz.quartzite.triggers :as triggers][metabase.analytics-interface.core :as analytics][metabase.mq.impl :as mq.impl]+ [metabase.mq.quartz-affinity :as quartz-affinity][metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.concurrency :as q.concurrency][metabase.mq.queue.impl :as q.impl][metabase.mq.queue.registry :as q.registry]+ [metabase.task.bootstrap :as task.bootstrap][metabase.task.core :as task][metabase.util.log :as log])(:import@@ -45,6 +48,13 @@"This backend's identifier, registered with `metabase.mq.init`; its `name` labels metrics.":queue.backend/quartz)+;; Install the queue node-affinity DriverDelegate when Quartz's JDBC properties are set. Registered at+;; load time, and here rather than in `metabase.mq.init`, because it is a property of *this* backend —+;; init has no business knowing that one backend swaps out a Quartz internal. (`mq` depends on `task`,+;; not the reverse, so `task.bootstrap` calls back into this rather than referencing `mq`.)+;; install-delegate! falls back to the plain per-DB delegate if the affinity subclass can't be loaded.+(task.bootstrap/register-jdbc-property-setter! quartz-affinity/install-delegate!)+(def ^:private job-group"Quartz group every queue job/trigger lives in, so they're easy to isolate from the rest of thescheduler's jobs."@@ -105,11 +115,64 @@rare race where the listener was torn down between acquisition and execution.We must *re-schedule* not refire: a refire re-runs on THIS node and bypasses acquisition, so it would- loop here until node shutdown and never reach a capable node."- [^Scheduler scheduler queue payload attempt]+ loop here until node shutdown and never reach a capable node.++ The re-queued trigger keeps the batch's original `start-at` instead of taking `now`. The age reaper+ ([[metabase.mq.task.queue-reaper]]) gives up on a never-acquired queue trigger by its `start_time`,+ and a bounce writes a *fresh* trigger — so stamping `now` would reset the message's reaper clock on+ every bounce, and a message nobody can handle could never age out. That is exactly the message the+ reaper exists to drop, and nothing else bounds this path: a bounce carries no backoff and spends no+ retry budget. An already-due `start-at` is also what we want on the way out: the batch should reach+ a capable node as soon as one asks for work."+ [^Scheduler scheduler queue payload attempt ^Date start-at](analytics/inc! :metabase-mq/batches-retried {:channel (name queue) :reason "no-listener"})(log/debugf "Queue %s has no listener on this node; re-queuing for a node that does." queue)- (schedule-message-trigger! scheduler queue payload attempt (Date.)))+ (schedule-message-trigger! scheduler queue payload attempt start-at))++;;; ------------------------------------ Waking the acquire loop ------------------------------------++(task/defjob ^{:doc "Never fires. Exists only to give [[wake-scheduler!]] a durable job key to call+ `resumeJob` on."}+ QueueSlotNudgeJob+ [_ctx])++(def ^:private nudge-job-key (jobs/key "queue-slot-nudge" job-group))++(defonce ^:private nudge-job-ensured? (atom false))++(defn- ensure-nudge-job!+ "Idempotently creates the durable, trigger-less job that [[wake-scheduler!]] pokes."+ [^Scheduler scheduler]+ (when-not @nudge-job-ensured?+ (.addJob scheduler+ (jobs/build+ (jobs/of-type QueueSlotNudgeJob)+ (jobs/with-identity nudge-job-key)+ (jobs/store-durably))+ true)+ (reset! nudge-job-ensured? true)))++(defn- wake-scheduler!+ "Tells Quartz's trigger-acquisition loop to look again, right now.++ This is *required* for `:max-concurrent-batches` to work, not an optimization. Once the acquisition+ filter excludes a queue we're at capacity on, the scheduler thread finds nothing to acquire and+ parks on its `sigLock` for up to `idleWaitTime` (30s by default) — and nothing wakes it when one of+ our batches finishes and frees a slot, because Quartz only signals on completion for+ `@DisallowConcurrentExecution` jobs. Without this poke a capped queue would drain N batches, sleep+ 30s, drain N more.++ `resumeJob` is the cheapest *public* API that signals that loop: `QuartzScheduler.resumeJob` ends in+ `notifySchedulerThread(0L)`, which is exactly the signal the acquire loop is waiting on. We aim it at+ a job with no triggers, so the resume itself has nothing to resume and the whole call is a zero-row+ no-op — we are calling it purely for that side effect on the scheduler thread."+ [^Scheduler scheduler queue]+ (when (q.registry/max-concurrent-batches queue)+ (try+ (ensure-nudge-job! scheduler)+ (.resumeJob scheduler nudge-job-key)+ (catch Throwable t+ (log/debugf t "Could not wake the Quartz acquire loop after freeing a slot on %s" queue)))))(defn- deliver-batch!"Shared job body. Reads the merged job+trigger data and dispatches on the delivery result:@@ -131,29 +194,39 @@on Quartz's worker threads; in a cluster the retry lands in the shared JDBC store so any node canpick it up."[^JobExecutionContext ctx]- (let [data (.getMergedJobDataMap ctx)- queue (keyword "queue" (.getString data data-queue))- payload (.getString data data-payload)- attempt (Integer/parseInt (or (.getString data data-attempt) "0"))]+ (let [data (.getMergedJobDataMap ctx)+ queue (keyword "queue" (.getString data data-queue))+ payload (.getString data data-payload)+ attempt (Integer/parseInt (or (.getString data data-attempt) "0"))+ scheduler (.getScheduler ctx)](try- (let [result (mq.impl/deliver! queue payload)]- (cond- (= mq.impl/no-listener result)- (requeue-no-listener! (.getScheduler ctx) queue payload attempt)-- (instance? Throwable result)- (let [next-attempt (inc attempt)]- (q.impl/handle-batch-failure-policy!- queue attempt result- #(schedule-message-trigger! (.getScheduler ctx) queue payload next-attempt- (Date. (long (+ (System/currentTimeMillis) (retry-delay-ms next-attempt)))))- (constantly nil)))))+ (q.concurrency/with-slot queue+ (let [result (mq.impl/deliver! queue payload)]+ (cond+ (= mq.impl/no-listener result)+ ;; read the trigger lazily — only this path needs it+ (requeue-no-listener! scheduler queue payload attempt+ (.getStartTime (.getTrigger ctx)))++ (instance? Throwable result)+ (let [next-attempt (inc attempt)]+ (q.impl/handle-batch-failure-policy!+ queue payload attempt result+ #(schedule-message-trigger! scheduler queue payload next-attempt+ (Date. (long (+ (System/currentTimeMillis) (retry-delay-ms next-attempt)))))+ (constantly nil))))))+ ;; `with-slot` has returned, so the slot is already released — and a freed slot is invisible to+ ;; Quartz's acquire loop, which may be parked for up to `idleWaitTime` with this queue filtered+ ;; out. Tell it to look again. (Waking *before* the release would be pointless: the loop would+ ;; re-check, still see us at capacity, and park again.)+ (wake-scheduler! scheduler queue);; Delivered, or the retry/requeue is durably committed. Return nil so Quartz completes the;; one-shot trigger.nil(catch Throwable t;; We couldn't hand the message off (couldn't schedule the retry, etc.). Don't let Quartz drop- ;; the fired one-shot — have it re-run this job instead.+ ;; the fired one-shot — have it re-run this job instead. `with-slot` has already released the+ ;; slot on its way out, so the refire won't find the queue at capacity because of us.(backoff-before-refire!)(throw (doto (JobExecutionException. t) (.setRefireImmediately true)))))))@@ -208,6 +281,21 @@true)))(swap! ensured-jobs conj [queue exclusive?])))+;;; ------------------------------------ Node affinity (acquisition) ------------------------------------++(defn capable-queue-names+ "The queue names this node is willing to acquire triggers for *right now* — the set the affinity+ delegate splices into Quartz's acquire query.++ Quartz's shape of the backend-agnostic [[q.concurrency/takeable-queues]]: strings, because that's what+ a Quartz job is named. A queue drops out for having no listener here or for being at its+ `:max-concurrent-batches`, which mean the same thing to the acquisition filter — don't take this work,+ leave it in the shared store for a node that can run it.++ Called on every acquire, so a listener registering and a slot freeing up both take effect at once."+ []+ (into #{} (map name) (q.concurrency/takeable-queues)))+(defrecord QuartzQueueBackend []q.backend/QueueBackend(backend-id [_this] backend-id)@@ -230,11 +318,18 @@queue){:queue queue :backend backend-id}))))+ ;; There is no poll loop to start. What this backend *does* need at startup is its node-affinity+ ;; filter: tell the delegate which queues we'll acquire triggers for. That is a Quartz-specific+ ;; acquisition strategy, so it's wired here rather than in `metabase.mq.init` — the init ns shouldn't+ ;; have to know that one particular backend gates its own intake.+ (start! [_this]+ (quartz-affinity/set-capability-fn! capable-queue-names))++ (shutdown! [_this])+;; Push backend — Quartz drives everything below, so these poll-driver hooks are never invoked;; (the poll loop is never started) and exist only to satisfy the protocol.- (start! [_this])- (shutdown! [_this])- (fetch! [_this _available-queues] nil)+ (fetch! [_this _queue->free-slots] nil)(queue-depths [_this] nil)(batch-successful! [_this _queue-name _batch-id] nil)(failure-count [_this _queue-name _batch-id] nil)diff --git a/src/metabase/mq/queue/registry.clj b/src/metabase/mq/queue/registry.cljindex bdf0b00a6ce..b9622bf0c98 100644--- a/src/metabase/mq/queue/registry.clj+++ b/src/metabase/mq/queue/registry.clj@@ -6,6 +6,7 @@in the cluster. Backends use this registry at startup to pre-arrange broker-side resources(e.g. RabbitMQ queue declarations) for the full set of known queues."(:require+ [metabase.util.log :as log][metabase.util.malli.humanize :as mu.humanize][metabase.util.malli.registry :as mr]))@@ -37,12 +38,20 @@#(= "queue" (namespace %))]])(mr/def :metabase.mq.queue/queue-config- [:map {:closed true}- [:transactional (into [:enum {:error/message (str "must be one of " transactional-modes)}]- transactional-modes)]- [:exclusive {:optional true} :boolean]- [:max-batch-messages {:optional true} pos-int?]- [:dedup-fn {:optional true} fn?]])+ [:and+ [:map {:closed true}+ [:transactional (into [:enum {:error/message (str "must be one of " transactional-modes)}]+ transactional-modes)]+ [:exclusive {:optional true} :boolean]+ [:max-batch-messages {:optional true} pos-int?]+ [:max-concurrent-batches {:optional true} [:or pos-int? fn?]]+ [:dedup-fn {:optional true} fn?]+ [:on-error {:optional true} fn?]]+ [:fn {:error/message (str "a queue cannot declare both :exclusive and :max-concurrent-batches — "+ ":exclusive already limits it to one batch cluster-wide, which is "+ "stricter than any per-node cap. Drop one.")}+ (fn [{:keys [exclusive max-concurrent-batches]}]+ (not (and exclusive max-concurrent-batches)))]])(def ^:dynamic *queues*"queue-name (keyword) → config map for every declared queue.@@ -96,11 +105,40 @@[queue-name](or (:max-batch-messages (get @*queues* queue-name)) default-max-batch-messages))+(defn max-concurrent-batches+ "Returns how many batches of `queue-name` a single node may deliver at once, or nil if the queue+ declares no cap.++ A queue may declare the cap as a literal int or as a 0-arg fn; a fn is resolved on every call, so a+ cap backed by a `defsetting` tracks that setting live rather than freezing its value at+ registration. A nil cap means unbounded.++ A fn cap that throws is caught and treated as unbounded (nil): the cap is a best-effort throttle,+ so degrading to unbounded keeps work flowing rather than wedging the scheduler in quartz or other places."+ [queue-name]+ (let [cap (:max-concurrent-batches (get @*queues* queue-name))]+ (if (fn? cap)+ (try+ (cap)+ (catch Throwable t+ (log/warnf t "Error resolving :max-concurrent-batches for %s; treating as unbounded" queue-name)+ nil))+ cap)))+(defn dedup-fn"Returns the `:dedup-fn` declared for `queue-name`, or nil if none."[queue-name](:dedup-fn (get @*queues* queue-name)))+(defn on-error+ "Returns the `:on-error` handler declared for `queue-name`, or nil if none.++ The handler is the queue's terminal-failure hook: it runs when a batch has exhausted+ `queue-max-retries` and is about to be dropped, giving the owning module a chance to record the+ failure durably (mark rows failed, alert, dead-letter)."+ [queue-name]+ (:on-error (get @*queues* queue-name)))+(defn transactional"Returns the `:transactional` mode (`:require`/`:try`/`:never`) declared for `queue-name`, or nilif no such queue has been declared."@@ -137,11 +175,41 @@Optional config keys:`:exclusive` — when true, at most one batch for this queue is in-flight cluster-wide.+ Mutually exclusive with `:max-concurrent-batches` (declaring both throws).+ `:max-concurrent-batches`+ — how many batches of this queue a single node will deliver at once. A node+ already running that many stops *taking* new ones: the Quartz backend+ won't acquire the queue's triggers and the poll driver won't fetch its+ messages, so the work stays in the shared store for a node with capacity.+ May be an int, or a 0-arg fn (resolved per check, so a cap backed by a+ `defsetting` tracks it live).++ This is the per-node middle ground between `:exclusive` (exactly 1+ cluster-wide) and the default (unbounded) — pick one of the three, they+ are not layered. Declaring it alongside `:exclusive` throws: cluster-wide+ mutual exclusion is already stricter than any per-node cap, so the cap+ could never bind, and the two are enforced by different machinery.++ Omitting it means unbounded, on every backend.++ It is a *throttle, not a guarantee*: it bounds what a node **takes**, and+ that check races the work it gates, so a node can end up a batch or two+ over the cap — in which case the batch is simply delivered, not handed+ back. Use `:exclusive` when a limit must actually hold.`:max-batch-messages` — soft target batch size for this queue (defaults to[[default-max-batch-messages]]). Used at publish time (coalescing) andconsume time (slicing).`:dedup-fn` — `messages -> messages` applied at publish time to drop duplicatesfrom a batch before it is buffered.+ `:on-error` — terminal-failure hook, called with a single map+ `{:channel :messages :error :attempts}` when a batch has exhausted+ `queue-max-retries` and is about to be dropped. Use it to record the failure+ durably. Exceptions thrown by the handler are logged and swallowed: the+ batch is dropped either way, so a broken handler can't lock the queue.++ It fires on the terminal drop only — never for a batch that recovers on a+ retry — but it is *at-least-once*, so make it idempotent and make it+ tolerate firing for work that already succeeded.These are all properties of the queue itself — they take effect at publish time, on everynode, regardless of whether a listener is registered locally.diff --git a/test/metabase/mq/backend_parity_test.clj b/test/metabase/mq/backend_parity_test.cljindex ac327749fe3..be65d741235 100644--- a/test/metabase/mq/backend_parity_test.clj+++ b/test/metabase/mq/backend_parity_test.clj@@ -18,13 +18,21 @@(:require[clojure.test :refer :all][metabase.mq.core :as mq]+ [metabase.mq.queue.registry :as q.registry][metabase.mq.test-util :as mq.tu]))(set! *warn-on-reflection* true)(def ^:private backend-kinds"Backend kinds exercised by the parity scenarios. `:memory` covers the- fast async in-process backend."+ fast async in-process backend.++ Quartz is not here, and can't easily be: it fires jobs on its own worker+ threads, which don't inherit the dynamic bindings `with-test-mq` relies on+ for isolation (see `metabase.mq.queue.quartz-test`, which drives that+ backend directly instead). So a scenario passing here proves parity across+ the *poll* backends, and pins the contract for the next one added — it does+ not by itself prove Quartz agrees."[:memory])(def ^:private delivery-modes@@ -97,6 +105,48 @@"Message delivered at least twice (retry-after-failure or chaos duplication)")(mq/unlisten! queue-name)))))+(deftest exclusive-queue-runs-one-batch-at-a-time-test+ (testing "an :exclusive queue never has two batches in its listener at once, on any backend.++ This is the parity scenario most worth having, because `:exclusive` is the one guarantee+ each backend must implement *itself* — Quartz with @DisallowConcurrentExecution on the job+ class, the memory backend by not fetching a second batch while one is in flight. The shared+ poll driver implements it not at all, by design. So a backend that simply forgets the flag+ keeps working, delivers everything, and quietly offers no mutual exclusion whatsoever —+ which is exactly what the memory backend did until this was noticed. Nothing but a+ behavioral test catches that."+ (run-parity!+ :queue/parity-exclusive+ (fn [ctx queue-name]+ (let [in-flight (atom 0)+ peak (atom 0)+ received (atom [])]+ ;; one message per batch, so "batches at once" is what the listener actually observes+ (q.registry/register-queue! queue-name {:transactional :try+ :exclusive true+ :max-batch-messages 1})+ (mq.tu/listen! queue-name+ (fn [msg]+ (swap! peak max (swap! in-flight inc))+ ;; hold the batch long enough that an overlapping delivery would be seen+ (Thread/sleep 50)+ (swap! received conj msg)+ (swap! in-flight dec)))+ (mq/with-queue queue-name [q]+ (mq/put q "a")+ (mq/put q "b")+ (mq/put q "c"))+ (mq.tu/eventually! ctx+ #(= #{"a" "b" "c"} (set @received))+ scenario-timeout-ms)+ (is (= 1 @peak)+ "never two batches of an exclusive queue in the listener at the same time")+ ;; set, not sequence: exclusivity buys mutual exclusion, not ordering — and under+ ;; :duplicate delivery each message legitimately arrives more than once.+ (is (= #{"a" "b" "c"} (set @received))+ "and every message is still delivered — exclusivity serializes, it doesn't drop")+ (mq/unlisten! queue-name))))))+(deftest queue-normalizes-message-shape-test(testing "Every backend delivers messages in the same canonical JSON-normalized shape:keyword keys preserved, keyword/symbol values and other non-JSON-nativediff --git a/test/metabase/mq/quartz_affinity_test.clj b/test/metabase/mq/quartz_affinity_test.cljindex 22df00f5192..5b636f887cd 100644--- a/test/metabase/mq/quartz_affinity_test.clj+++ b/test/metabase/mq/quartz_affinity_test.clj@@ -3,8 +3,11 @@[clojure.string :as str][clojure.test :refer :all][metabase.classloader.core :as classloader]+ [metabase.mq.listener :as listener][metabase.mq.quartz-affinity :as affinity]+ [metabase.mq.queue.concurrency :as q.concurrency][metabase.mq.queue.quartz :as q.quartz]+ [metabase.mq.queue.registry :as q.registry][metabase.test.fixtures :as fixtures][toucan2.core :as t2])(:import@@ -117,3 +120,53 @@(is (= #{"aff-sync"} (acquire! sched #{"nonexistent"}))"capable of an unrelated queue → only the non-queue job")(finally (cleanup! sched))))))++;;; The test above pins the filter against a hand-supplied capability set. This one wires up the *real*+;;; capability fn (`q.quartz/capable-queue-names`) and proves the thing the concurrency cap actually+;;; depends on: a node at its `:max-concurrent-batches` stops acquiring that queue's triggers from the+;;; shared store — the same way it stops acquiring a queue it has no listener for.++(defn- acquire-with-real-capability-fn! [sched]+ (let [the-atom @#'affinity/capability-fn*+ prev @the-atom+ now (epoch-ms)]+ (affinity/set-capability-fn! q.quartz/capable-queue-names)+ (try+ (t2/with-connection [conn]+ (->> (affinity/select-trigger-to-acquire conn (+ now 3600000) (- now 3600000) 100 (test-rtp sched))+ (map (fn [^TriggerKey k] (.getName k)))+ set))+ (finally (reset! the-atom prev)))))++(deftest at-capacity-queue-is-not-acquired-against-appdb-test+ (testing "a node at a queue's :max-concurrent-batches stops acquiring its triggers, so they stay+ WAITING in the shared store for a node with a free slot"+ (let [sched (str "AFFINITY_CAP_TEST_" (random-uuid))+ capped (keyword "queue" (str "affcap-capped-" (random-uuid)))+ uncapped (keyword "queue" (str "affcap-uncapped-" (random-uuid)))]+ (q.registry/register-queue! capped {:transactional :try :max-concurrent-batches 1})+ (q.registry/register-queue! uncapped {:transactional :try})+ (listener/batch-listen! capped (fn [_] nil))+ (listener/batch-listen! uncapped (fn [_] nil))+ (cleanup! sched)+ (seed! sched [[(name capped) mq-group] [(name uncapped) mq-group] ["affcap-sync" "DEFAULT"]])+ (try+ (testing "with a free slot, the capped queue's trigger is acquirable"+ (is (= #{(name capped) (name uncapped) "affcap-sync"}+ (acquire-with-real-capability-fn! sched))))+ (q.concurrency/with-slot capped+ (testing "at capacity, its trigger is not selected by the acquire query at all"+ (is (= #{(name uncapped) "affcap-sync"}+ (acquire-with-real-capability-fn! sched))+ "the node at its cap leaves the capped queue's work in the store — and an uncapped+ queue and non-queue jobs (sync, pulses) are untouched, so a saturated queue can't+ starve them")))+ (testing "once the slot is released, its trigger is acquirable again"+ (is (= #{(name capped) (name uncapped) "affcap-sync"}+ (acquire-with-real-capability-fn! sched))))+ (finally+ (cleanup! sched)+ (listener/unlisten! capped)+ (listener/unlisten! uncapped)+ (swap! q.registry/*queues* dissoc capped)+ (swap! q.registry/*queues* dissoc uncapped))))))diff --git a/test/metabase/mq/queue/concurrency_test.clj b/test/metabase/mq/queue/concurrency_test.cljnew file mode 100644index 00000000000..dae5f053e83--- /dev/null+++ b/test/metabase/mq/queue/concurrency_test.clj@@ -0,0 +1,255 @@+(ns metabase.mq.queue.concurrency-test+ (:require+ [clojure.test :refer :all]+ [metabase.mq.listener :as listener]+ [metabase.mq.queue.concurrency :as q.concurrency]+ [metabase.mq.queue.registry :as q.registry])+ (:import+ (java.util.concurrent CountDownLatch ExecutorService Executors TimeUnit)))++(set! *warn-on-reflection* true)++(defn- with-fresh-state [f]+ (binding [q.registry/*queues* (atom {})+ q.concurrency/*in-flight* (atom {})+ listener/*listeners* (atom {})]+ (f)))++(use-fixtures :each with-fresh-state)++(defn- declare-queue! [queue-name cap]+ (q.registry/register-queue! queue-name (cond-> {:transactional :try}+ cap (assoc :max-concurrent-batches cap))))++(defn- wait-for! [pred]+ (let [deadline (+ (System/currentTimeMillis) 5000)]+ (loop []+ (or (pred)+ (when (< (System/currentTimeMillis) deadline)+ (Thread/sleep 5)+ (recur))))))++;;; ------------------------------------------- with-slot -------------------------------------------++(deftest slots-are-counted-and-released-test+ (declare-queue! :queue/capped 2)+ (q.concurrency/with-slot :queue/capped+ (is (= 1 (q.concurrency/in-flight :queue/capped)))+ (q.concurrency/with-slot :queue/capped+ (is (= 2 (q.concurrency/in-flight :queue/capped)))))+ (is (zero? (q.concurrency/in-flight :queue/capped))+ "every slot is released as its with-slot exits"))++(deftest over-the-cap-the-body-still-runs-test+ (testing "the cap is a throttle on what we take, not a veto on work already in hand: a delivery that+ arrives while we're at the cap is still delivered, and simply counts us over it"+ ;; Bouncing it instead would mean a re-queue path with its own backoff, its own metric, and a+ ;; hot-loop to avoid — all to shave an overshoot that is small, bounded and self-correcting.+ (declare-queue! :queue/capped 1)+ (q.concurrency/with-slot :queue/capped+ (is (true? (q.concurrency/at-capacity? :queue/capped)))+ (is (= ::ran (q.concurrency/with-slot :queue/capped ::ran))+ "the body ran even though the node was already at its cap")+ (is (= 1 (q.concurrency/in-flight :queue/capped))+ "and the over-cap slot was released again on the way out"))+ (is (zero? (q.concurrency/in-flight :queue/capped)))))++(deftest at-capacity?-keeps-saying-yes-while-over-the-cap-test+ (testing "a node that raced its way over the cap must keep refusing new work until it drains back+ under — hence >= rather than ="+ (declare-queue! :queue/capped 1)+ (q.concurrency/with-slot :queue/capped+ (q.concurrency/with-slot :queue/capped+ (is (= 2 (q.concurrency/in-flight :queue/capped)) "over the cap of 1")+ (is (true? (q.concurrency/at-capacity? :queue/capped)))+ (is (= 0 (q.concurrency/free-slots :queue/capped))+ "free-slots floors at zero — it never reports negative headroom")))))++(deftest with-slot-returns-the-body-value-test+ (declare-queue! :queue/capped 1)+ (is (= ::delivered (q.concurrency/with-slot :queue/capped ::delivered)))+ (is (nil? (q.concurrency/with-slot :queue/capped nil))))++(deftest with-slot-releases-when-the-body-throws-test+ (testing "a body that throws still releases its slot — a leak here would silently shrink the queue's+ capacity for the life of the process, with no error anywhere to explain why it stopped+ delivering"+ (declare-queue! :queue/capped 1)+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo #"boom"+ (q.concurrency/with-slot :queue/capped+ (throw (ex-info "boom" {})))))+ (is (zero? (q.concurrency/in-flight :queue/capped)))+ (is (false? (q.concurrency/at-capacity? :queue/capped))+ "and the queue can still deliver")))++(deftest at-capacity?-and-working?-track-slots-test+ (declare-queue! :queue/capped 1)+ (is (false? (q.concurrency/at-capacity? :queue/capped)))+ (is (false? (q.concurrency/working? :queue/capped)))+ (q.concurrency/with-slot :queue/capped+ (is (true? (q.concurrency/at-capacity? :queue/capped)))+ (is (true? (q.concurrency/working? :queue/capped))))+ (is (false? (q.concurrency/at-capacity? :queue/capped)))+ (is (false? (q.concurrency/working? :queue/capped))))++(deftest free-slots-counts-down-test+ (declare-queue! :queue/capped 3)+ (is (= 3 (q.concurrency/free-slots :queue/capped)))+ (q.concurrency/with-slot :queue/capped+ (is (= 2 (q.concurrency/free-slots :queue/capped)))+ (q.concurrency/with-slot :queue/capped+ (q.concurrency/with-slot :queue/capped+ (is (= 0 (q.concurrency/free-slots :queue/capped)))))))++(deftest uncapped-queue-is-unbounded-test+ ;; There is exactly one meaning of "declared no cap", and it is the same on every backend:+ ;; unbounded. No caller supplies its own default — that is how the backends used to disagree+ ;; (Quartz unbounded, the poll driver one-at-a-time) about what the same config meant.+ (declare-queue! :queue/uncapped nil)+ (testing "an uncapped queue always grants a slot and is never at capacity"+ (letfn [(nest [depth]+ (if (zero? depth)+ (do (is (= 25 (q.concurrency/in-flight :queue/uncapped)))+ (is (false? (q.concurrency/at-capacity? :queue/uncapped)))+ (is (true? (q.concurrency/working? :queue/uncapped))))+ (q.concurrency/with-slot :queue/uncapped+ (nest (dec depth)))))]+ (nest 25)))+ (testing "and reports no fetch limit — nil means 'take as many as you have', not 'take one'"+ (is (nil? (q.concurrency/free-slots :queue/uncapped)))))++(deftest exclusive-is-not-this-layers-concern-test+ (testing "an :exclusive queue is unbounded as far as this ns is concerned — mutual exclusion is the+ *backend's* job (Quartz's @DisallowConcurrentExecution; the memory backend's fetch),+ because the backend is what owns delivery. Modelling it here as a cap of one would look+ tidy and quietly downgrade a cluster-wide guarantee to a per-node one."+ (q.registry/register-queue! :queue/excl {:transactional :try :exclusive true})+ (is (nil? (q.concurrency/free-slots :queue/excl)))+ (q.concurrency/with-slot :queue/excl+ (is (false? (q.concurrency/at-capacity? :queue/excl))))))++(deftest slots-are-per-channel-test+ (declare-queue! :queue/a 1)+ (declare-queue! :queue/b 1)+ (q.concurrency/with-slot :queue/a+ (is (true? (q.concurrency/at-capacity? :queue/a)))+ (is (false? (q.concurrency/at-capacity? :queue/b))+ "queue/b has its own budget")+ (is (zero? (q.concurrency/in-flight :queue/b)))))++(deftest cap-fn-is-resolved-on-every-check-test+ (let [cap (atom 1)]+ (declare-queue! :queue/dynamic (fn [] @cap))+ (q.concurrency/with-slot :queue/dynamic+ (is (true? (q.concurrency/at-capacity? :queue/dynamic))+ "at capacity while the setting says 1")+ (reset! cap 2)+ (is (false? (q.concurrency/at-capacity? :queue/dynamic))+ "raising the setting immediately opens headroom — the cap is read per check, not frozen at+ registration")+ (is (= 1 (q.concurrency/free-slots :queue/dynamic))))))++(deftest concurrent-slots-are-accounted-exactly-test+ (testing "the in-flight count is exact under contention — it is what the throttle reads, so a lost+ update would let a queue quietly run over its cap forever"+ (declare-queue! :queue/capped 5)+ ;; `future` conveys the dynamic bindings from the fixture, so every thread races on the same+ ;; in-flight atom. All 20 start together on the latch to make the swap actually contend.+ (let [start (CountDownLatch. 1)+ release (CountDownLatch. 1)+ entered (atom 0)+ futures (doall+ (for [_ (range 20)]+ (future+ (.await start)+ (q.concurrency/with-slot :queue/capped+ (swap! entered inc)+ (.await release) ; hold the slot so all 20 are counted at once+ ::ran))))]+ (.countDown start)+ (is (true? (wait-for! #(= 20 @entered)))+ "every batch runs — the cap throttles fetching, it does not refuse work in hand")+ (is (= 20 (q.concurrency/in-flight :queue/capped))+ "and all 20 are counted, with no lost updates")+ (is (true? (q.concurrency/at-capacity? :queue/capped))+ "so the node correctly reports it should take no more")+ (.countDown release)+ (run! deref futures)+ (is (zero? (q.concurrency/in-flight :queue/capped))+ "every slot is released once its body finishes — no leak, no negative count"))))++;;; ---------------------------------------- submit-with-slot! ----------------------------------------++(deftest submit-with-slot-counts-from-hand-off-and-releases-on-the-worker-thread-test+ (declare-queue! :queue/capped 1)+ (let [pool (Executors/newCachedThreadPool)+ release (CountDownLatch. 1)+ entered (CountDownLatch. 1)+ woken (atom 0)]+ (try+ (q.concurrency/submit-with-slot!+ :queue/capped pool+ (fn [] (.countDown entered) (.await release))+ #(swap! woken inc))+ (is (= 1 (q.concurrency/in-flight :queue/capped))+ "the batch counts against the node the moment it is handed off — not when the worker thread+ gets around to it, which would let the driver fetch more work in the gap")+ (is (.await entered 5000 TimeUnit/MILLISECONDS))+ (is (true? (q.concurrency/at-capacity? :queue/capped))+ "the slot is held for as long as the task runs, over on the other thread")+ (.countDown release)+ (is (true? (wait-for! #(zero? (q.concurrency/in-flight :queue/capped))))+ "the slot is released when the task finishes")+ (is (true? (wait-for! #(= 1 @woken)))+ "and after-release runs — that's the 'a slot just freed up, go look for work' signal")+ (finally+ (.shutdownNow ^ExecutorService pool)))))++(deftest submit-with-slot-releases-when-the-task-throws-test+ (declare-queue! :queue/capped 1)+ (let [pool (Executors/newCachedThreadPool)]+ (try+ (q.concurrency/submit-with-slot!+ :queue/capped pool+ (fn [] (throw (ex-info "boom" {})))+ (fn []))+ (is (true? (wait-for! #(zero? (q.concurrency/in-flight :queue/capped))))+ "a task that throws still releases its slot")+ (finally+ (.shutdownNow ^ExecutorService pool)))))++(deftest submit-with-slot-releases-when-the-submit-itself-fails-test+ (testing "if the pool is gone the task will never run, and so will never release — the slot must be+ given back here or the queue sits at capacity forever and silently stops delivering"+ (declare-queue! :queue/capped 1)+ (let [pool (doto (Executors/newCachedThreadPool) (.shutdownNow))]+ (is (thrown? Exception+ (q.concurrency/submit-with-slot! :queue/capped pool (fn []) (fn []))))+ (is (zero? (q.concurrency/in-flight :queue/capped)))+ (is (false? (q.concurrency/at-capacity? :queue/capped))))))++;;; ---------------------------------------- takeable-queues ----------------------------------------++(deftest takeable-queues-is-the-shared-definition-test+ (testing "the queues this node will take work for: has a listener, not at capacity.++ Both backends derive their own shape from this one fn — Quartz the capability set it+ splices into its acquire query, the poll driver the free-slot map it hands fetch! — so that+ they cannot drift apart on the answer."+ (declare-queue! :queue/capped 1)+ (declare-queue! :queue/uncapped nil)+ (declare-queue! :queue/no-listener 1)+ (listener/batch-listen! :queue/capped (fn [_]))+ (listener/batch-listen! :queue/uncapped (fn [_]))+ (testing "a declared queue with no listener on this node is never takeable"+ (is (= #{:queue/capped :queue/uncapped} (set (q.concurrency/takeable-queues)))+ ":queue/no-listener is declared, but nothing here handles it"))+ (testing "a queue at its cap drops out, and comes back when the slot frees"+ (q.concurrency/with-slot :queue/capped+ (is (= #{:queue/uncapped} (set (q.concurrency/takeable-queues)))))+ (is (= #{:queue/capped :queue/uncapped} (set (q.concurrency/takeable-queues)))))+ (testing "an uncapped queue stays takeable however many batches are in flight"+ (q.concurrency/with-slot :queue/uncapped+ (q.concurrency/with-slot :queue/uncapped+ (is (contains? (set (q.concurrency/takeable-queues)) :queue/uncapped)))))))diff --git a/test/metabase/mq/queue/core_test.clj b/test/metabase/mq/queue/core_test.cljindex 650fc34da98..8aa45177678 100644--- a/test/metabase/mq/queue/core_test.clj+++ b/test/metabase/mq/queue/core_test.clj@@ -7,6 +7,7 @@[metabase.mq.payload :as payload][metabase.mq.publish :as mq.publish][metabase.mq.publish-buffer :as publish-buffer]+ [metabase.mq.queue.concurrency :as q.concurrency][metabase.mq.queue.polling :as q.polling][metabase.mq.queue.registry :as q.registry][metabase.mq.test-util :as mq.tu]@@ -104,7 +105,43 @@(mq/put q "msg1")(mq/put q "msg2"))(mq.tu/flush! test-mq)- (is (= ["msg1" "msg2"] @heard-messages))))))+ ;; delivery, not order — see `every-message-is-delivered-test`. Exclusivity buys mutual+ ;; exclusion, not FIFO.+ (is (= #{"msg1" "msg2"} (set @heard-messages)))))))++(deftest exclusive-queue-is-serialized-by-the-memory-backend-test+ (testing "an :exclusive queue runs one batch at a time on the memory backend too.++ Exclusivity is a *backend* guarantee, and each backend implements its own: Quartz uses+ @DisallowConcurrentExecution on the job class; the memory backend never fetches a second+ batch for a queue that already has one in flight. The shared poll driver knows nothing+ about the flag. Before the memory backend implemented it, an :exclusive queue serialized on+ Quartz and meant nothing at all here — it only *looked* right because the driver happened+ to deliver one batch per channel at a time."+ (q.polling/start-worker-pool!)+ (mq.tu/with-test-mq [_test-mq]+ (let [queue-name :queue/exclusive-serialization-test+ in-flight (atom 0)+ peak (atom 0)+ arrived (atom 0)+ release (CountDownLatch. 1)]+ (q.registry/register-queue! queue-name {:transactional :try :exclusive true :max-batch-messages 1})+ (mq.tu/listen! queue-name (fn [_]+ (swap! peak max (swap! in-flight inc))+ (swap! arrived inc)+ (.await release)+ (swap! in-flight dec)))+ (mq/with-queue queue-name [q]+ (mq/put q "a")+ (mq/put q "b")+ (mq/put q "c"))+ (is (true? (mq.tu/wait-for! #(= 1 @arrived) 5000)))+ (is (nil? (mq.tu/wait-for! #(> @arrived 1) 500))+ "no second batch is fetched while the first is in flight")+ (.countDown release)+ (is (true? (mq.tu/wait-for! #(= 3 @arrived) 5000)) "all three still get delivered")+ (is (= 1 @peak) "never more than one batch in the listener at a time")+ (mq/unlisten! queue-name)))))(deftest batch-listen-exclusive-test(let [heard-batches (atom [])]@@ -253,7 +290,17 @@(mq.tu/flush! test-mq)(is (= ["a" "b" "c"] @heard) "all delivered after flush"))))))-(deftest fifo-ordering-test+(deftest every-message-is-delivered-test+ ;; Deliberately NOT an ordering test. The MQ promises at-least-once delivery, not FIFO: Quartz fires+ ;; a queue's batches on its worker threads and explicitly does not fire them in submission order, and+ ;; an uncapped queue runs them concurrently. The poll driver used to deliver one batch at a time,+ ;; which made ordering *look* guaranteed if you only ever tested against the memory backend — a+ ;; guarantee no other backend has ever offered. Asserting it here would pin a memory-only behavior+ ;; and re-introduce exactly the cross-backend divergence `:max-concurrent-batches` exists to remove.+ ;;+ ;; A queue that genuinely needs serialized delivery must say so: `:max-concurrent-batches 1` (per+ ;; node) or `:exclusive true` (cluster-wide). Even then, order is not promised — only mutual+ ;; exclusion.(let [received (atom [])](mq.tu/with-test-mq [test-mq]{:queue/test (fn [message] (swap! received conj message))}@@ -261,40 +308,66 @@(mq/with-queue :queue/test [q](mq/put q i)))(mq.tu/flush! test-mq)- (testing "All messages are delivered in order"- (is (= (range 10) @received))))))+ (testing "every published message is delivered exactly once (in some order)"+ (is (= (set (range 10)) (set @received)))+ (is (= 10 (count @received)))))))-(deftest exclusive-queue-single-active-handler-test- "Verifies that submit-delivery! returns false for a channel that already has an- active handler, enforcing at-most-one concurrent delivery per channel."- (q.polling/start-worker-pool!)- (mq.tu/with-test-mq [_test-mq]- (let [queue-name :queue/exclusive-concurrency-test- started (CountDownLatch. 1) ; counted down once the worker thread enters the listener- release (CountDownLatch. 1)] ; blocks the listener until the test releases it- (mq.tu/listen! queue-name- (fn [_] (.countDown started) (.await release)))- (testing "First submission succeeds and marks the channel as busy"- (is (true? (q.polling/submit-delivery! queue-name (payload/encode ["msg1"]) nil nil)))+(deftest submit-delivery-counts-the-batch-against-the-node-test+ (testing "a submitted batch counts against the node's :max-concurrent-batches from hand-off until it+ finishes, which is what makes the queue report itself at capacity and stop fetching more"+ (q.polling/start-worker-pool!)+ (mq.tu/with-test-mq [_test-mq]+ (let [queue-name :queue/capped-concurrency-test+ started (CountDownLatch. 1) ; counted down once the worker thread enters the listener+ release (CountDownLatch. 1)] ; blocks the listener until the test releases it+ (q.registry/register-queue! queue-name {:transactional :try :max-concurrent-batches 1})+ (mq.tu/listen! queue-name+ (fn [_] (.countDown started) (.await release)))+ (q.polling/submit-delivery! queue-name (payload/encode ["msg1"]) nil nil)(is (.await started 5 TimeUnit/SECONDS) "worker thread started and entered the listener")- (is (true? (q.polling/channel-busy? queue-name))))- (testing "Second submission returns false while channel is busy"- (is (false? (q.polling/submit-delivery! queue-name (payload/encode ["msg2"]) nil nil))))- (.countDown release) ; Release the listener- (testing "Channel is no longer busy after delivery completes"- (is (true? (mq.tu/wait-for! #(not (q.polling/channel-busy? queue-name)) 5000))- "active-handlers clears once delivery completes"))- (mq/unlisten! queue-name))))+ (is (true? (q.concurrency/working? queue-name)))+ (is (true? (q.concurrency/at-capacity? queue-name))+ "at its cap of 1 — so the poll loop will fetch nothing more for this queue")+ (.countDown release) ; Release the listener+ (is (true? (mq.tu/wait-for! #(not (q.concurrency/working? queue-name)) 5000))+ "the in-flight count returns to zero once delivery completes")+ (is (false? (q.concurrency/at-capacity? queue-name))+ "and the queue can be fetched for again")+ (mq/unlisten! queue-name)))))++(deftest uncapped-queue-submits-without-limit-test+ (testing "a queue that declares no :max-concurrent-batches is unbounded — the same as on Quartz.+ The poll driver used to serialize it one-batch-at-a-time, which made the *same* queue+ config mean different things depending on which backend was running."+ (q.polling/start-worker-pool!)+ (mq.tu/with-test-mq [_test-mq]+ (let [queue-name :queue/uncapped-concurrency-test+ in-listener (CountDownLatch. 3)+ release (CountDownLatch. 1)]+ (mq.tu/listen! queue-name (fn [_] (.countDown in-listener) (.await release)))+ (dotimes [i 3]+ (q.polling/submit-delivery! queue-name (payload/encode [(str "msg" i)]) nil nil))+ (is (.await in-listener 5 TimeUnit/SECONDS)+ "all three deliveries are in the listener at the same time")+ (is (false? (q.concurrency/at-capacity? queue-name))+ "and it is never at capacity, however many are working — that is what uncapped means")+ (.countDown release)+ (is (true? (mq.tu/wait-for! #(not (q.concurrency/working? queue-name)) 5000)))+ (mq/unlisten! queue-name)))))(deftest submit-delivery-frees-slot-when-pool-unavailable-test- (testing "if submitting to the worker pool fails (e.g. pool not running), the channel slot is- released rather than left permanently busy — otherwise the queue would silently stop- delivering forever"- (with-redefs [q.polling/worker-pool (atom nil) ; deref/.submit NPEs- q.polling/active-handlers (atom {})]- (let [queue-name :queue/submit-failure-test]- (is (thrown? Exception- (q.polling/submit-delivery! queue-name (payload/encode ["x"]) nil nil))- "the submit failure propagates to the caller")- (is (false? (q.polling/channel-busy? queue-name))- "the channel is not left marked busy after the failed submit")))))+ (testing "if submitting to the worker pool fails (e.g. pool not running), the claimed slot is+ released rather than held forever — otherwise a capped queue would sit permanently at+ capacity and silently stop delivering"+ (binding [q.concurrency/*in-flight* (atom {})+ q.registry/*queues* (atom {})]+ (with-redefs [q.polling/worker-pool (atom nil)] ; deref/.submit NPEs+ (let [queue-name :queue/submit-failure-test]+ (q.registry/register-queue! queue-name {:transactional :try :max-concurrent-batches 1})+ (is (thrown? Exception+ (q.polling/submit-delivery! queue-name (payload/encode ["x"]) nil nil))+ "the submit failure propagates to the caller")+ (is (false? (q.concurrency/working? queue-name))+ "the slot claimed for the failed submit was given back")+ (is (false? (q.concurrency/at-capacity? queue-name))+ "so the queue can still deliver"))))))diff --git a/test/metabase/mq/queue/memory_test.clj b/test/metabase/mq/queue/memory_test.cljindex fce1772fcd0..4ada4892334 100644--- a/test/metabase/mq/queue/memory_test.clj+++ b/test/metabase/mq/queue/memory_test.clj@@ -37,6 +37,19 @@(is (= 1 (depth-for layer "has-listener")) "messages with a listener are kept")(is (nil? (depth-for layer "no-listener")) "orphaned messages are dropped")))))+(deftest clear-stale-claims-counts-as-a-failure-test+ (testing "recovering a stale claim increments the message's failure count, so a delivery that keeps+ timing out (never acks) eventually exhausts its retry budget instead of retrying forever"+ (let [layer (memory/make-layer)]+ (memory/publish! layer :queue/slow "p")+ (memory/take-pending! layer :queue/slow 1)+ (is (= 0 (memory/message-failures layer 1)) "claiming a message doesn't fail it")+ ;; a negative timeout makes every in-flight claim stale+ (is (= 1 (memory/clear-stale-claims! layer -1)) "the stale claim is recovered")+ (is (= 1 (memory/message-failures layer 1)) "and counts as one failure")+ (testing "the message is re-claimable after recovery"+ (is (= 1 (count (memory/take-pending! layer :queue/slow 1))))))))+(deftest publish-test(mq.tu/with-test-mq [ctx](let [queue-name (keyword "queue" (str "publish-test-" (gensym)))diff --git a/test/metabase/mq/queue/on_error_test.clj b/test/metabase/mq/queue/on_error_test.cljnew file mode 100644index 00000000000..1f435a4e95c--- /dev/null+++ b/test/metabase/mq/queue/on_error_test.clj@@ -0,0 +1,281 @@+(ns metabase.mq.queue.on-error-test+ "Tests for a queue's `:on-error` handler — the hook that fires when a batch has exhausted its+ retries and would otherwise be silently dropped.++ The handler is invoked from the shared retry-vs-drop policy+ ([[metabase.mq.queue.impl/handle-batch-failure-policy!]]), which every backend routes through, so a+ queue gets the same terminal-failure semantics no matter which backend is running. Backend parity+ is asserted here for the poll backend and in `metabase.mq.queue.quartz-test` for the push backend.++ Listeners here are registered with [[metabase.mq.listener/batch-listen!]] rather than `def-listener!`+ (which only registers a `defmethod` that `register-listeners!` realizes at startup — too late inside+ `with-test-mq`) or `mq.tu/listen!` (which isolates per-message errors, swallowing the very failures+ under test)."+ (:require+ [clojure.test :refer :all]+ [metabase.analytics-interface.core :as analytics]+ [metabase.mq.core :as mq]+ [metabase.mq.listener :as listener]+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.test-util :as mq.tu]+ [metabase.test :as mt]))++(set! *warn-on-reflection* true)++(deftest on-error-fires-once-with-messages-and-cause-test+ (testing "a batch that exhausts its retries invokes :on-error with the messages and the cause"+ (mt/with-temporary-setting-values [queue-max-retries 2]+ (let [dropped (atom [])+ attempts (atom 0)+ boom (ex-info "always boom" {:reason :boom})]+ (mq.tu/with-test-mq [ctx]+ (q.registry/register-queue!+ :queue/on-error-basic+ {:transactional :try+ :on-error (fn [info] (swap! dropped conj info))})+ (listener/batch-listen!+ :queue/on-error-basic+ (fn [_messages]+ (swap! attempts inc)+ (throw boom)))+ (mq/with-queue :queue/on-error-basic [q]+ (mq/put q {:id 1})+ (mq/put q {:id 2}))+ (is (mq.tu/eventually! ctx #(seq @dropped) 5000)+ ":on-error fires once the batch exhausts its retries")+ (is (= 1 (count @dropped))+ ":on-error fires once *per batch*, on the terminal drop — not on every failed attempt.+ (Once per batch is not once per payload: see+ duplicate-delivery-fires-on-error-for-an-already-succeeded-payload-test.)")+ (let [info (first @dropped)]+ (is (= #{:channel :messages :error :attempts} (set (keys info))))+ (is (= :queue/on-error-basic (:channel info)))+ (is (= [{:id 1} {:id 2}] (:messages info))+ "the handler receives the decoded messages of the dropped batch")+ (is (= boom (:error info))+ "the handler receives the exception that failed the last attempt")+ (is (= 2 (:attempts info))+ "the handler is told how many attempts were made")))))))++(deftest on-error-not-called-while-retries-remain-test+ (testing ":on-error does not fire for a failure that will be retried and then succeeds"+ (mt/with-temporary-setting-values [queue-max-retries 3]+ (let [dropped (atom [])+ attempts (atom 0)+ handled (atom nil)]+ (mq.tu/with-test-mq [ctx]+ (q.registry/register-queue!+ :queue/on-error-recovers+ {:transactional :try+ :on-error (fn [info] (swap! dropped conj info))})+ (listener/batch-listen!+ :queue/on-error-recovers+ (fn [messages]+ ;; fail the first attempt, succeed on the retry+ (when (= 1 (swap! attempts inc))+ (throw (ex-info "transient" {})))+ (reset! handled messages)))+ (mq/with-queue :queue/on-error-recovers [q]+ (mq/put q "eventually-ok"))+ (is (mq.tu/eventually! ctx #(some? @handled) 5000)+ "the batch is retried and succeeds")+ (is (= ["eventually-ok"] @handled))+ (is (= [] @dropped)+ ":on-error must not fire for a transient failure that later succeeds"))))))++(deftest on-error-not-called-on-success-test+ (testing ":on-error is never invoked for a batch that is delivered cleanly"+ (let [dropped (atom [])+ handled (atom nil)]+ (mq.tu/with-test-mq [ctx]+ (q.registry/register-queue!+ :queue/on-error-happy+ {:transactional :try+ :on-error (fn [info] (swap! dropped conj info))})+ (listener/batch-listen!+ :queue/on-error-happy+ (fn [messages] (reset! handled messages)))+ (mq/with-queue :queue/on-error-happy [q]+ (mq/put q "fine"))+ (mq.tu/flush! ctx)+ (is (= ["fine"] @handled))+ (is (= [] @dropped))))))++(deftest on-error-handler-that-throws-still-drops-the-batch-test+ (testing "a throwing :on-error handler is contained — the batch is still dropped, not retried forever"+ (mt/with-temporary-setting-values [queue-max-retries 2]+ (let [called (atom 0)+ attempts (atom 0)]+ (mq.tu/with-test-mq [ctx]+ (q.registry/register-queue!+ :queue/on-error-throws+ {:transactional :try+ :on-error (fn [_info]+ (swap! called inc)+ (throw (ex-info "handler itself blew up" {})))})+ (listener/batch-listen!+ :queue/on-error-throws+ (fn [_messages]+ (swap! attempts inc)+ (throw (ex-info "always boom" {}))))+ (mq/with-queue :queue/on-error-throws [q]+ (mq/put q "doomed"))+ (is (mq.tu/eventually! ctx #(pos? @called) 5000)+ ":on-error is invoked")+ ;; If the throwing handler escaped the drop path, the batch would be re-delivered and+ ;; attempts would keep climbing past the retry budget.+ (Thread/sleep 500)+ (is (= 2 @attempts)+ "the batch is still dropped after exactly queue-max-retries attempts")+ (is (= 1 @called)+ ":on-error is not re-invoked"))))))++(deftest listener-error-still-goes-through-the-retry-policy-test+ (testing "a listener that throws an `Error` rather than an `Exception` must still go through the+ retry/drop policy.++ `Error` is not exotic here: a failed `assert` or `{:pre ...}` throws `AssertionError`, and+ a namespace that fails to load lazily on a worker thread throws+ `ExceptionInInitializerError`. If the delivery core only catches `Exception`, an `Error`+ escapes past the retry budget entirely and into the backend's hand-off failure path —+ which is built to *refire the same job with the same attempt number*, on the assumption+ that the message was never delivered. The batch would then re-run forever on one node,+ burning no retry budget and never reaching `:on-error`."+ (mt/with-temporary-setting-values [queue-max-retries 2]+ (let [dropped (atom [])+ attempts (atom 0)]+ (mq.tu/with-test-mq [ctx]+ (q.registry/register-queue!+ :queue/on-error-throwable+ {:transactional :try+ :on-error (fn [info] (swap! dropped conj info))})+ (listener/batch-listen!+ :queue/on-error-throwable+ (fn [_messages]+ (swap! attempts inc)+ (throw (AssertionError. "an assert blew up inside the listener"))))+ (mq/with-queue :queue/on-error-throwable [q]+ (mq/put q "doomed"))+ (is (mq.tu/eventually! ctx #(seq @dropped) 5000)+ "an Error exhausts the retry budget and reaches :on-error, like any other failure")+ (Thread/sleep 500)+ (is (= 2 @attempts)+ "and is retried exactly queue-max-retries times — not refired without a budget")+ (is (= 1 (count @dropped))+ ":on-error fires once, on the terminal drop")+ (is (instance? AssertionError (:error (first @dropped)))+ "the handler receives the Error that failed the batch"))))))++(deftest drop-path-bookkeeping-failure-does-not-redeliver-test+ (testing "a failure in the drop path's *bookkeeping* — the log line and the `batches-dropped`+ counter — must not redeliver the batch or re-run its `:on-error`.++ `run-on-error!` is carefully built so it can never throw, precisely so a broken handler+ cannot escape into the backend's retry machinery. But the logging and metrics calls+ sitting next to it were unguarded, and a backend reads any exception out of this policy+ as \"the message was not handed off\" — so it refires the batch with its attempt counter+ untouched. The listener runs again, fails again, is over budget again, and `:on-error`+ fires a second time for a batch that was already dropped."+ (mt/with-temporary-setting-values [queue-max-retries 2]+ (let [dropped (atom [])+ attempts (atom 0)+ orig analytics/inc!]+ ;; a metrics backend that is down exactly when we try to record the drop+ (with-redefs [analytics/inc! (fn [metric & args]+ (if (= metric :metabase-mq/batches-dropped)+ (throw (ex-info "metrics backend is down" {}))+ (apply orig metric args)))]+ (mq.tu/with-test-mq [ctx]+ (q.registry/register-queue!+ :queue/on-error-drop-bookkeeping+ {:transactional :try+ :on-error (fn [info] (swap! dropped conj info))})+ (listener/batch-listen!+ :queue/on-error-drop-bookkeeping+ (fn [_messages]+ (swap! attempts inc)+ (throw (ex-info "always boom" {}))))+ (mq/with-queue :queue/on-error-drop-bookkeeping [q]+ (mq/put q "doomed"))+ (is (mq.tu/eventually! ctx #(seq @dropped) 5000)+ ":on-error still fires when the drop metric blows up")+ (Thread/sleep 500)+ (is (= 1 (count @dropped))+ ":on-error fires once — a failed drop metric must not resurrect the batch")+ (is (= 2 @attempts)+ "and the batch is not redelivered past its retry budget")))))))++(deftest queue-without-on-error-still-drops-test+ (testing "a queue that declares no :on-error keeps the old behavior: exhaust retries, then drop"+ (mt/with-temporary-setting-values [queue-max-retries 2]+ (let [attempts (atom 0)]+ (mq.tu/with-test-mq [ctx]+ (q.registry/register-queue! :queue/on-error-absent {:transactional :try})+ (listener/batch-listen!+ :queue/on-error-absent+ (fn [_messages]+ (swap! attempts inc)+ (throw (ex-info "always boom" {}))))+ (mq/with-queue :queue/on-error-absent [q]+ (mq/put q "doomed"))+ (is (mq.tu/eventually! ctx #(= 2 @attempts) 5000))+ (Thread/sleep 500)+ (is (= 2 @attempts)+ "still dropped after queue-max-retries, with no handler configured"))))))++(deftest duplicate-delivery-fires-on-error-for-an-already-succeeded-payload-test+ (testing "PINS A KNOWN LIMITATION — this is the behavior we have, not the behavior we want.++ A duplicate delivery whose copy *fails* will run out its own retry budget and fire+ `:on-error` for a payload that another copy already handled successfully.++ The reason is that a message has no identity. A retry is a brand-new trigger with a fresh+ id, and the attempt counter rides along inside it, so two deliveries of one payload are+ two independent batches with two independent retry budgets and nothing joining them. A+ success on one node cannot cancel, or even be observed by, a copy failing on another —+ which is precisely what happens when Quartz's cluster failover re-fires a batch onto a+ second node because the first missed a heartbeat while still running it.++ So an `:on-error` handler must be idempotent *and* tolerate firing for work that in fact+ succeeded. This is why the explorations handlers guard their writes on the row still+ being `pending` rather than blindly stamping it `error`.++ Closing this needs message identity plus a record of what has been delivered. When that+ lands, this test should flip to asserting `(= [] @dropped)`."+ (mt/with-temporary-setting-values [queue-max-retries 2]+ (let [dropped (atom [])+ succeeded (atom 0)+ calls (atom 0)]+ (mq.tu/with-test-mq [ctx {:duplicate-delivery? true}]+ (q.registry/register-queue!+ :queue/on-error-duplicate+ {:transactional :try+ :on-error (fn [info] (swap! dropped conj info))})+ (listener/batch-listen!+ :queue/on-error-duplicate+ (fn [_messages]+ ;; the first copy of the payload lands cleanly; the duplicate is the one that fails+ (if (= 1 (swap! calls inc))+ (swap! succeeded inc)+ (throw (ex-info "this delivery failed, but the work is already done" {})))))+ (mq/with-queue :queue/on-error-duplicate [q]+ (mq/put q "already-done"))+ (is (mq.tu/eventually! ctx #(seq @dropped) 5000)+ "the failing duplicate exhausts its retries")+ (is (= 1 @succeeded)+ "the payload was handled successfully — the work is done")+ (is (= 1 (count @dropped))+ ":on-error still fires for it, because the failing copy has its own retry budget and no+ way to learn that the work already succeeded"))))))++(deftest on-error-must-be-a-fn-test+ (testing "the queue config schema accepts an :on-error fn and rejects a non-fn"+ (binding [q.registry/*queues* (atom {})]+ (is (nil? (q.registry/register-queue! :queue/on-error-schema-ok+ {:transactional :try :on-error (fn [_] nil)}))+ "a fn is accepted")+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo #"Invalid config for queue"+ (q.registry/register-queue! :queue/on-error-schema-bad+ {:transactional :try :on-error "not-a-fn"}))+ "a non-fn :on-error is rejected at declaration time"))))diff --git a/test/metabase/mq/queue/polling_test.clj b/test/metabase/mq/queue/polling_test.cljindex 8ac0912215c..e717b36dbf7 100644--- a/test/metabase/mq/queue/polling_test.clj+++ b/test/metabase/mq/queue/polling_test.clj@@ -1,9 +1,15 @@(ns metabase.mq.queue.polling-test(:require[clojure.test :refer :all]+ [metabase.mq.core :as mq][metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.concurrency :as q.concurrency][metabase.mq.queue.polling :as q.polling]- [metabase.test :as mt]))+ [metabase.mq.queue.registry :as q.registry]+ [metabase.mq.test-util :as mq.tu]+ [metabase.test :as mt])+ (:import+ (java.util.concurrent CountDownLatch)))(set! *warn-on-reflection* true)@@ -13,7 +19,7 @@(let [backend (reify q.backend/QueueBackend(backend-id [_] :queue.backend/test)(publish! [_ _queue _payload] nil)- (fetch! [_ _available] nil)+ (fetch! [_ _queue->free-slots] nil)(queue-depths [_] nil)(batch-successful! [_ _queue _batch-id] nil)(failure-count [_ _queue _batch-id] nil)@@ -35,3 +41,83 @@{:channel "orphaned-queue" :reason "delivery-exhausted"})0)))"stale-recovery drops are NOT conflated with the delivery-retry exhaustion reason")))))++;;; ------------------------------- :max-concurrent-batches -------------------------------++(defn- gated-listener+ "A listener that records its own peak concurrency. Each delivery counts itself in, parks on+ `release`, then counts itself out — so `peak` ends up holding the most that were ever inside the+ listener at the same time, which is exactly what the cap is supposed to bound."+ [in-flight peak arrived ^CountDownLatch release]+ (fn [_msg]+ (swap! peak max (swap! in-flight inc))+ (swap! arrived inc)+ (.await release)+ (swap! in-flight dec)))++(defn- publish-n! [queue n]+ (mq/with-queue queue [q]+ (doseq [i (range n)] (mq/put q {:i i}))))++(deftest max-concurrent-batches-caps-in-flight-deliveries-test+ (testing "a node delivers at most :max-concurrent-batches batches of a queue at once, and takes the+ rest only as slots free up"+ (mq.tu/with-test-mq [_ctx]+ (let [queue :queue/capped-concurrency+ in-flight (atom 0)+ peak (atom 0)+ arrived (atom 0)+ release (CountDownLatch. 1)]+ ;; one message per batch, so "batches in flight" == "messages in flight"+ (q.registry/register-queue! queue {:transactional :try+ :max-batch-messages 1+ :max-concurrent-batches 2})+ (mq.tu/listen! queue (gated-listener in-flight peak arrived release))+ (publish-n! queue 5)+ (testing "exactly the cap is picked up while the listener is blocked"+ (is (true? (mq.tu/wait-for! #(= 2 @arrived) 5000))+ "two batches reached the listener")+ ;; Give the poll loop many chances to (wrongly) fetch a third: it spins every 50ms.+ (is (nil? (mq.tu/wait-for! #(> @arrived 2) 500))+ "no third batch is taken while the node is at capacity")+ (is (= 2 @peak) "never more than the cap in flight at once"))+ (testing "the remaining batches are delivered once the slots free"+ (.countDown release)+ (is (true? (mq.tu/wait-for! #(= 5 @arrived) 5000))+ "all 5 batches eventually delivered — capacity throttles, it doesn't drop")+ (is (= 2 @peak) "the cap held for the whole drain"))+ (mq/unlisten! queue)))))++(deftest uncapped-queue-is-unbounded-test+ (testing "a queue that declares no cap is unbounded on the poll driver too — the same as on Quartz.+ The driver used to take one batch at a time regardless, which made the *same* queue config+ mean different things depending on which backend was running."+ (mq.tu/with-test-mq [_ctx]+ (let [queue :queue/uncapped-concurrency+ in-flight (atom 0)+ peak (atom 0)+ arrived (atom 0)+ release (CountDownLatch. 1)]+ (q.registry/register-queue! queue {:transactional :try :max-batch-messages 1})+ (mq.tu/listen! queue (gated-listener in-flight peak arrived release))+ (publish-n! queue 3)+ (is (true? (mq.tu/wait-for! #(= 3 @arrived) 5000))+ "all three batches are picked up while the listener is still blocked — nothing throttles+ a queue that declared no cap")+ (is (= 3 @peak) "all three were in the listener at once")+ (.countDown release)+ (mq/unlisten! queue)))))++(deftest failed-delivery-releases-its-slot-test+ (testing "a slot is released even when the listener throws, so a failing queue can't leak capacity+ and quietly throttle itself to a standstill"+ (mq.tu/with-test-mq [_ctx]+ (let [queue :queue/slot-release+ attempts (atom 0)]+ (q.registry/register-queue! queue {:transactional :try :max-concurrent-batches 1})+ (mq.tu/listen! queue (fn [_] (swap! attempts inc) (throw (ex-info "boom" {}))))+ (publish-n! queue 1)+ (is (true? (mq.tu/wait-for! #(pos? @attempts) 5000)) "the listener ran and threw")+ (is (true? (mq.tu/wait-for! #(zero? (q.concurrency/in-flight queue)) 5000))+ "the slot is returned after the failed delivery, not leaked")+ (mq/unlisten! queue)))))diff --git a/test/metabase/mq/queue/quartz_test.clj b/test/metabase/mq/queue/quartz_test.cljindex 9d268bf07f6..1f0ca7be367 100644--- a/test/metabase/mq/queue/quartz_test.clj+++ b/test/metabase/mq/queue/quartz_test.clj@@ -4,21 +4,26 @@root `*listeners*` (the production registration path; Quartz threads don't inherit the testthread's dynamic bindings) and drive the backend directly under an in-memory scheduler."(:require+ [clojure.string :as str][clojure.test :refer :all][metabase.mq.impl :as mq.impl][metabase.mq.listener :as listener][metabase.mq.payload :as payload]+ [metabase.mq.quartz-affinity :as quartz-affinity][metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.concurrency :as q.concurrency][metabase.mq.queue.quartz :as q.quartz][metabase.mq.queue.registry :as q.registry][metabase.mq.settings :as mq.settings]+ [metabase.task.core :as task][metabase.task.impl :as task.impl][metabase.test :as mt][metabase.test.fixtures :as fixtures])(:import- (java.util Properties)+ (java.time Instant)+ (java.util Date Properties)(java.util.concurrent CountDownLatch CyclicBarrier TimeUnit)- (org.quartz JobDataMap JobDetail JobExecutionContext JobExecutionException Scheduler Trigger)+ (org.quartz JobDataMap JobDetail JobExecutionContext JobExecutionException JobKey Scheduler Trigger TriggerBuilder)(org.quartz.impl StdSchedulerFactory)))(set! *warn-on-reflection* true)@@ -292,13 +297,52 @@queue (keyword "queue" (str "quartz-nolistener-requeue-" (random-uuid)))](#'q.quartz/ensure-queue-job! scheduler queue false)(try- (#'q.quartz/requeue-no-listener! scheduler queue (payload/encode ["x"]) 0)+ (#'q.quartz/requeue-no-listener! scheduler queue (payload/encode ["x"]) 0 (Date.))(let [triggers (job-triggers scheduler queue)](is (= 1 (count triggers))"a single re-queue trigger is placed back into the store (re-acquired via affinity, not fired here)")(is (instance? Trigger (first triggers))))(finally (.deleteJob scheduler (#'q.quartz/queue-job-key queue))))))))+(deftest quartz-requeue-no-listener-preserves-the-messages-age-test+ (testing "a no-listener requeue keeps the message's original start time rather than stamping it `now`.++ The age reaper ([[metabase.mq.task.queue-reaper]]) gives up on a never-acquired queue+ trigger by its `start_time`. But a bounced message is re-queued as a *fresh* trigger, so+ stamping `now` resets its own reaper clock on every bounce — a message that keeps landing+ on nodes with no listener could never age out, which is exactly the message the reaper+ exists to drop. Nothing else bounds this path: `requeue-no-listener!` carries no backoff+ and spends no retry budget. It is most reachable during a rolling deploy, when nodes are+ tearing their listeners down, and in the startup window before node-affinity is installed+ (its capability fn defaults to `:all`, i.e. no filtering)."+ (mt/with-temp-scheduler!+ (let [^Scheduler scheduler (task.impl/scheduler)+ queue (keyword "queue" (str "quartz-nolistener-age-" (random-uuid)))+ payload (payload/encode ["x"])+ ;; a message that has already been waiting hours for a node that can handle it+ start-at (Date/from (.minusSeconds (Instant/now) (* 6 60 60)))+ ctx (reify JobExecutionContext+ (getMergedJobDataMap [_]+ (doto (JobDataMap.)+ (.put "queue" (name queue))+ (.put "payload" payload)+ (.put "attempt" "0")))+ (getScheduler [_] scheduler)+ (getTrigger [_] (-> (TriggerBuilder/newTrigger) (.startAt start-at) (.build))))]+ ;; deliberately no listener for `queue` on this node — that is the path under test+ (q.registry/register-queue! queue {:transactional :try})+ (#'q.quartz/ensure-queue-job! scheduler queue false)+ (try+ (#'q.quartz/deliver-batch! ctx)+ (let [triggers (job-triggers scheduler queue)]+ (is (= 1 (count triggers))+ "the message is re-queued for a node that does have the listener")+ (is (= start-at (.getStartTime ^Trigger (first triggers)))+ "and keeps its original start time, so the reaper's clock keeps running"))+ (finally+ (.deleteJob scheduler (#'q.quartz/queue-job-key queue))+ (swap! q.registry/*queues* dissoc queue)))))))+(deftest quartz-exclusive-job-not-downgraded-test(testing "a node that disagrees about :exclusive can't downgrade an existing exclusive job";; Simulates three nodes (the per-process `ensured-jobs` cache is reset between them) publishing@@ -328,3 +372,203 @@(is (= "ExclusiveQueueMessageJob" (job-class))"exclusive is sticky; a disagreeing node cannot downgrade it back to concurrent")(finally (.deleteJob scheduler jk))))))))++(deftest quartz-on-error-fires-on-exhaustion-test+ (testing "the :on-error terminal-failure hook fires on the push backend too, with the dropped batch"+ ;; Backend parity with `metabase.mq.queue.on-error-test` (poll backend): both route their+ ;; delivery failures through the shared `q.impl/handle-batch-failure-policy!`, so the hook must+ ;; behave identically here.+ (mt/with-temporary-setting-values [queue-max-retries 2]+ (mt/with-temp-scheduler!+ (let [dropped (atom [])+ queue (keyword "queue" (str "quartz-on-error-" (random-uuid)))+ boom (ex-info "always boom" {})]+ (q.registry/register-queue! queue {:transactional :try+ :on-error (fn [info] (swap! dropped conj info))})+ (listener/batch-listen! queue (fn [_msgs] (throw boom)))+ (try+ (publish! queue {:id 7})+ (is (wait-for! #(seq @dropped) 8000)+ ":on-error fires once the batch exhausts its retries")+ (is (= 1 (count @dropped)))+ (let [info (first @dropped)]+ (is (= queue (:channel info)))+ (is (= [{:id 7}] (:messages info))+ "the handler receives the decoded messages of the dropped batch")+ (is (= 2 (:attempts info))))+ (finally+ (listener/unlisten! queue)+ (swap! q.registry/*queues* dissoc queue))))))))++;;; ------------------------------- :max-concurrent-batches -------------------------------++;;; Where the cap is actually enforced on Quartz is *trigger acquisition* — a node at capacity stops+;;; acquiring the queue's triggers, so they stay WAITING for a node with room. That is exercised+;;; against the real app-DB acquire query in `metabase.mq.quartz-affinity-test` (the delegate is a JDBC+;;; DriverDelegate, so the tests here — which run on RAMJobStore — cannot see it at all).+;;;+;;; What is left to pin here is what `deliver-batch!` does with a batch that reaches it anyway.++(defn- job-ctx+ "A minimal `JobExecutionContext` for driving `deliver-batch!` directly, carrying `queue`'s payload+ and a real scheduler (so a re-queue actually writes a trigger we can count)."+ ^JobExecutionContext [^Scheduler scheduler queue payload]+ (reify JobExecutionContext+ (getMergedJobDataMap [_]+ (doto (JobDataMap.)+ (.put "queue" (name queue))+ (.put "payload" payload)+ (.put "attempt" "0")))+ (getScheduler [_] scheduler)))++(deftest over-capacity-batch-is-still-delivered-test+ (testing "a batch that reaches deliver-batch! while the node is already at its cap is delivered+ anyway, not handed back"+ ;; The cap is a throttle on what a node *takes* (trigger acquisition), and that check necessarily+ ;; races the job body: Quartz decides what to acquire on its scheduler thread, before this runs on+ ;; a worker thread. Bouncing the odd batch that slips through would cost a whole re-queue path —+ ;; backoff, metric, and a hot-loop to avoid if the acquisition filter ever stops working — to shave+ ;; an overshoot that is small, bounded and drains itself. So we run it and let the count go over.+ (mt/with-temp-scheduler!+ (let [queue (keyword "queue" (str "quartz-overcap-" (random-uuid)))+ scheduler (task/scheduler)+ ran (atom 0)]+ (q.registry/register-queue! queue {:transactional :try :max-concurrent-batches 1})+ (listener/batch-listen! queue (fn [_msgs] (swap! ran inc)))+ (try+ ;; Create the durable job without publishing — a live trigger would be fired by this+ ;; (running) scheduler on its own thread, adding a delivery we didn't ask for.+ (#'q.quartz/ensure-queue-job! scheduler queue false)+ (let [before (count (job-triggers scheduler queue))]+ ;; Hold the queue's only slot, so the fired job runs while we are already at the cap —+ ;; exactly the acquire/execute race the acquisition filter cannot close.+ (q.concurrency/with-slot queue+ (is (true? (q.concurrency/at-capacity? queue)) "we are at the cap before it fires")+ (is (nil? (#'q.quartz/deliver-batch!+ (job-ctx scheduler queue (payload/encode [{:id 1}]))))+ "returns normally, so Quartz completes the one-shot")+ (is (= 1 @ran)+ "the listener ran: the batch was delivered over the cap rather than bounced")+ (is (= before (count (job-triggers scheduler queue)))+ "and no trigger was written back to the store — there is no bounce path any more"))+ (is (zero? (q.concurrency/in-flight queue))+ "the over-cap slot was released again, so the node drains back under its limit"))+ (finally+ (listener/unlisten! queue)+ (swap! q.registry/*queues* dissoc queue)))))))++(deftest delivery-releases-its-slot-test+ (testing "the slot is released after delivery, on both the success and the failure path — otherwise+ a queue would throttle itself to a standstill after a few batches"+ (mt/with-temp-scheduler!+ (let [scheduler (task/scheduler)]+ (doseq [[label listener] [["success" (fn [_msgs] nil)]+ ["failure" (fn [_msgs] (throw (ex-info "boom" {})))]]]+ (testing label+ (let [queue (keyword "queue" (str "quartz-release-" label "-" (random-uuid)))]+ (q.registry/register-queue! queue {:transactional :try :max-concurrent-batches 1})+ (listener/batch-listen! queue listener)+ (try+ ;; Create the durable job *without* publishing: the failure path reschedules a retry+ ;; onto it, and it must exist for that. Publishing would also enqueue a live trigger,+ ;; which this (running) scheduler would fire on its own thread — a second, concurrent+ ;; delivery holding a slot of its own, which is exactly what we're trying to measure.+ (#'q.quartz/ensure-queue-job! scheduler queue false)+ (#'q.quartz/deliver-batch! (job-ctx scheduler queue (payload/encode [{:id 1}])))+ (is (zero? (q.concurrency/in-flight queue))+ "the slot is returned, not leaked")+ (finally+ (listener/unlisten! queue)+ (swap! q.registry/*queues* dissoc queue))))))))))++(deftest capability-fn-excludes-at-capacity-queues-test+ (testing "a queue this node is at capacity on is reported as one it cannot currently handle, so the+ affinity delegate leaves its triggers WAITING for a node that can"+ (let [capped (keyword "queue" (str "quartz-capable-capped-" (random-uuid)))+ uncapped (keyword "queue" (str "quartz-capable-uncapped-" (random-uuid)))]+ (q.registry/register-queue! capped {:transactional :try :max-concurrent-batches 1})+ (q.registry/register-queue! uncapped {:transactional :try})+ (listener/batch-listen! capped (fn [_] nil))+ (listener/batch-listen! uncapped (fn [_] nil))+ (try+ (testing "with a free slot, both queues are acquirable"+ (let [capable (q.quartz/capable-queue-names)]+ (is (contains? capable (name capped)))+ (is (contains? capable (name uncapped)))))+ (q.concurrency/with-slot capped+ (testing "once the cap is reached, the capped queue drops out of the acquire query"+ (let [capable (q.quartz/capable-queue-names)]+ (is (not (contains? capable (name capped)))+ "at capacity — this node must not acquire more of its triggers")+ (is (contains? capable (name uncapped))+ "an uncapped queue is unaffected: it stays unbounded, as it was before caps existed")))+ (testing "and the predicate spliced into Quartz's acquire SQL really does exclude it"+ (let [sql (quartz-affinity/rewrite-acquisition-sql+ "SELECT * FROM QRTZ_TRIGGERS WHERE x = 1 ORDER BY next_fire_time"+ (q.quartz/capable-queue-names))]+ (is (not (str/includes? sql (name capped))))+ (is (str/includes? sql (name uncapped))))))+ (testing "releasing the slot makes it acquirable again"+ (is (contains? (q.quartz/capable-queue-names) (name capped))))+ (finally+ (listener/unlisten! capped)+ (listener/unlisten! uncapped)+ (swap! q.registry/*queues* dissoc capped)+ (swap! q.registry/*queues* dissoc uncapped))))))++(deftest wake-scheduler-pokes-the-acquire-loop-for-capped-queues-only-test+ ;; The nudge is load-bearing, not an optimization: once the acquisition filter excludes a queue this+ ;; node is at capacity on, Quartz's acquire loop finds nothing and parks on its sigLock for up to+ ;; idleWaitTime (30s) — and a plain (non-@DisallowConcurrentExecution) job completing does NOT signal+ ;; it. Without this poke a capped queue would drain N batches, sleep 30s, drain N more.+ ;;+ ;; `resumeJob` is the public API we ride to get that signal: QuartzScheduler.resumeJob ends in+ ;; notifySchedulerThread(0L). We aim it at a durable job with no triggers, so the resume itself is a+ ;; no-op and only the signal survives. This pins our half of that contract.+ (mt/with-temp-scheduler!+ (let [scheduler (task/scheduler)+ capped (keyword "queue" (str "quartz-wake-capped-" (random-uuid)))+ uncapped (keyword "queue" (str "quartz-wake-uncapped-" (random-uuid)))+ ^JobKey nudge-key @#'q.quartz/nudge-job-key]+ (reset! @#'q.quartz/nudge-job-ensured? false)+ (q.registry/register-queue! capped {:transactional :try :max-concurrent-batches 1})+ (q.registry/register-queue! uncapped {:transactional :try})+ (try+ (testing "an uncapped queue is never filtered out, so there is no stall to break and no poke"+ (#'q.quartz/wake-scheduler! scheduler uncapped)+ (is (false? (.checkExists scheduler nudge-key))+ "no nudge job created — an uncapped queue doesn't pay the cost of the wake-up"))+ (testing "a capped queue pokes the acquire loop, via a durable job that carries no triggers"+ (#'q.quartz/wake-scheduler! scheduler capped)+ (is (true? (.checkExists scheduler nudge-key))+ "the nudge job exists")+ (is (empty? (.getTriggersOfJob scheduler nudge-key))+ "and has no triggers — it never fires; we call resumeJob purely for the signal"))+ (testing "poking is idempotent and safe to call on every completed batch"+ (dotimes [_ 3] (#'q.quartz/wake-scheduler! scheduler capped))+ (is (true? (.checkExists scheduler nudge-key))))+ (finally+ (swap! q.registry/*queues* dissoc capped)+ (swap! q.registry/*queues* dissoc uncapped))))))++(deftest delivery-wakes-the-acquire-loop-after-freeing-a-slot-test+ (testing "finishing a batch on a capped queue tells the acquire loop to look again — that is what+ turns a freed slot into the next batch actually being picked up"+ (mt/with-temp-scheduler!+ (let [scheduler (task/scheduler)+ queue (keyword "queue" (str "quartz-wake-on-release-" (random-uuid)))+ woke (atom [])]+ (q.registry/register-queue! queue {:transactional :try :max-concurrent-batches 1})+ (listener/batch-listen! queue (fn [_msgs] nil))+ (try+ (publish! queue {:seed 1})+ (with-redefs-fn {#'q.quartz/wake-scheduler! (fn [_sched q] (swap! woke conj q))}+ (fn []+ (#'q.quartz/deliver-batch! (job-ctx scheduler queue (payload/encode [{:id 1}])))))+ (is (= [queue] @woke)+ "the acquire loop is woken exactly once, after the slot is released")+ (is (zero? (q.concurrency/in-flight queue))+ "and the slot really was free by then")+ (finally+ (listener/unlisten! queue)+ (swap! q.registry/*queues* dissoc queue)))))))diff --git a/test/metabase/mq/queue/registry_test.clj b/test/metabase/mq/queue/registry_test.cljindex bcb54fce0c9..3c7a6afa89b 100644--- a/test/metabase/mq/queue/registry_test.clj+++ b/test/metabase/mq/queue/registry_test.clj@@ -65,3 +65,72 @@(q.registry/register-queue! :queue/ok {:transactional :require :exclusive true:max-batch-messages 50 :dedup-fn identity})(is (= 50 (q.registry/max-batch-messages :queue/ok)))))++(deftest max-concurrent-batches-test+ (testing "a queue that doesn't declare a cap has none"+ (q.registry/register-queue! :queue/uncapped {:transactional :try})+ (is (nil? (q.registry/max-concurrent-batches :queue/uncapped))))+ (testing "an undeclared queue has no cap"+ (is (nil? (q.registry/max-concurrent-batches :queue/never-declared))))+ (testing "a literal int cap is returned as-is"+ (q.registry/register-queue! :queue/capped {:transactional :try :max-concurrent-batches 3})+ (is (= 3 (q.registry/max-concurrent-batches :queue/capped))))+ (testing "a fn cap is resolved at read time, so it tracks a setting that changes"+ (let [cap (atom 2)]+ (q.registry/register-queue! :queue/dynamic {:transactional :try+ :max-concurrent-batches (fn [] @cap)})+ (is (= 2 (q.registry/max-concurrent-batches :queue/dynamic)))+ (reset! cap 5)+ (is (= 5 (q.registry/max-concurrent-batches :queue/dynamic))))))++(deftest max-concurrent-batches-fn-that-throws-is-treated-as-unbounded-test+ (testing "a cap fn that throws degrades to unbounded (nil) instead of propagating"+ ;; This resolver runs inside the Quartz trigger-acquisition filter, so a throw here would abort+ ;; acquisition for *every* queue on the node, not just this one. The cap is a best-effort throttle,+ ;; so a broken resolver must degrade to unbounded rather than wedge the scheduler.+ (q.registry/register-queue! :queue/boom-cap+ {:transactional :try+ :max-concurrent-batches (fn [] (throw (ex-info "boom" {})))})+ (is (nil? (q.registry/max-concurrent-batches :queue/boom-cap)))))++(deftest rejects-exclusive-with-max-concurrent-batches-test+ (testing "a queue cannot declare both :exclusive and :max-concurrent-batches"+ ;; :exclusive already pins the queue to one batch cluster-wide, which is strictly stronger than any+ ;; per-node cap. Declaring both reads like it means something and doesn't.+ (is (thrown-with-msg?+ ExceptionInfo #"(?i)exclusive"+ (q.registry/register-queue! :queue/both {:transactional :try+ :exclusive true+ :max-concurrent-batches 2}))))+ (testing "and not even with a cap of 1, which looks equivalent but isn't (per-node vs cluster-wide)"+ (is (thrown-with-msg?+ ExceptionInfo #"(?i)exclusive"+ (q.registry/register-queue! :queue/both-one {:transactional :try+ :exclusive true+ :max-concurrent-batches 1}))))+ (testing ":exclusive false alongside a cap is fine — that's just the per-node cap on its own"+ (q.registry/register-queue! :queue/not-exclusive {:transactional :try+ :exclusive false+ :max-concurrent-batches 3})+ (is (= 3 (q.registry/max-concurrent-batches :queue/not-exclusive)))+ (is (false? (q.registry/exclusive? :queue/not-exclusive))))+ (testing "either one on its own still registers"+ (q.registry/register-queue! :queue/just-exclusive {:transactional :try :exclusive true})+ (is (true? (q.registry/exclusive? :queue/just-exclusive)))+ (is (nil? (q.registry/max-concurrent-batches :queue/just-exclusive)))+ (q.registry/register-queue! :queue/just-capped {:transactional :try :max-concurrent-batches 2})+ (is (= 2 (q.registry/max-concurrent-batches :queue/just-capped)))+ (is (false? (q.registry/exclusive? :queue/just-capped)))))++(deftest rejects-invalid-max-concurrent-batches-test+ (testing "register-queue! rejects a non-positive :max-concurrent-batches"+ (is (thrown-with-msg?+ ExceptionInfo #"max-concurrent-batches"+ (q.registry/register-queue! :queue/zero-cap {:transactional :try :max-concurrent-batches 0})))+ (is (thrown-with-msg?+ ExceptionInfo #"max-concurrent-batches"+ (q.registry/register-queue! :queue/neg-cap {:transactional :try :max-concurrent-batches -1}))))+ (testing "register-queue! rejects a :max-concurrent-batches that is neither an int nor a fn"+ (is (thrown-with-msg?+ ExceptionInfo #"max-concurrent-batches"+ (q.registry/register-queue! :queue/str-cap {:transactional :try :max-concurrent-batches "2"})))))diff --git a/test/metabase/mq/test_util.clj b/test/metabase/mq/test_util.cljindex cfa5cd07d11..78b76b9f97f 100644--- a/test/metabase/mq/test_util.clj+++ b/test/metabase/mq/test_util.clj@@ -18,8 +18,8 @@[metabase.mq.polling :as mq.polling][metabase.mq.publish-buffer :as publish-buffer][metabase.mq.queue.backend :as q.backend]+ [metabase.mq.queue.concurrency :as q.concurrency][metabase.mq.queue.memory :as q.memory]- [metabase.mq.queue.polling :as q.polling][metabase.mq.queue.registry :as q.registry]))(set! *warn-on-reflection* true)@@ -40,17 +40,17 @@"Returns a set of the idle conditions that currently do NOT hold. Empty set == idle."[ctx](cond-> #{}- (seq @publish-buffer/*publish-buffer*) (conj :publish-buffer-nonempty)- (seq (q.polling/busy-channels)) (conj :busy-channels)- (not (layer-drained? ctx)) (conj :memory-messages-pending)))+ (seq @publish-buffer/*publish-buffer*) (conj :publish-buffer-nonempty)+ (seq (q.concurrency/working-channels)) (conj :channels-still-working)+ (not (layer-drained? ctx)) (conj :memory-messages-pending)))(defn wait-for-idle!"Blocks until the MQ fixture reaches quiescence or `timeout-ms` elapses. Throwsan ex-info whose `:unmet` key names the conditions still holding on timeout.- Because `poll-once!` drains memory channels *before* setting `active-handlers`- on the next submit-delivery, a single idle observation can occur in the gap- between the two. We require two consecutive idle observations 20 ms apart."+ Because the poll loop drains memory channels *before* the next submit-delivery+ claims its slot, a single idle observation can occur in the gap between the two.+ We require two consecutive idle observations 20 ms apart."([ctx] (wait-for-idle! ctx 5000))([ctx timeout-ms](let [deadline (+ (System/currentTimeMillis) timeout-ms)]@@ -194,7 +194,7 @@(start! [_] (q.backend/start! inner))(shutdown! [_] (q.backend/shutdown! inner))(backend-id [_] (q.backend/backend-id inner))- (fetch! [_ available-queues] (q.backend/fetch! inner available-queues))+ (fetch! [_ queue->free-slots] (q.backend/fetch! inner queue->free-slots))(recover-stale! [_ stale-timeout-ms max-retries](q.backend/recover-stale! inner stale-timeout-ms max-retries))(run-heartbeats! [_] (q.backend/run-heartbeats! inner))@@ -225,6 +225,7 @@(let [{:keys [backend listeners duplicate-delivery?] :or {backend :memory}} opts](binding [listener/*listeners* (atom {})q.registry/*queues* (atom {})+ q.concurrency/*in-flight* (atom {})publish-buffer/*publish-buffer* (atom {})publish-buffer/*publish-buffer-ms* 0](let [backends (make-fixture-backends backend)