Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Persistent queue improvements (#77779)

Nathan Voxland8771f8eb7392
diff --git a/src/metabase/mq/README.md b/src/metabase/mq/README.md
index 3985286a797..1c2c2d8fc25 100644
--- a/src/metabase/mq/README.md
+++ b/src/metabase/mq/README.md
@@ -20,7 +20,7 @@ unified API.
(mq/put q {:key "value"})))
```
-All queue config — `:exclusive`, `:max-batch-messages`, `:dedup-fn` — belongs on `def-queue!`,
+All queue config — `:exclusive`, `:max-concurrent-batches`, `:max-batch-messages`, `:dedup-fn` — belongs on `def-queue!`,
because it takes effect at publish time on every node, regardless of where listeners are
registered. `def-listener!` only wires up the consumer-side handler. A listener registration
throws if its queue hasn't been declared yet — typos are caught at startup, not at first
@@ -119,11 +119,116 @@ keyword values become strings, sets become vectors, dates become strings.
| | Queue |
|---------------|----------------------------------------------------------------------|
| Delivery | At-least-once per listener |
+| Ordering | **None.** See below |
| Retry | Yes — up to `queue-max-retries` (default 5) |
| Backing store | Quartz `QRTZ_*` tables; `queue_message_outbox` for transactional publishes |
| Backlog | Persists until processed or exhausted |
| Use when | Work must complete reliably |
+**Guaranteed delivery is the guarantee. Ordering is not.** Batches are delivered concurrently, retried
+with backoff (so a failed batch lands after ones published later), and Quartz does not fire triggers in
+submission order even when it serializes them. Don't write a listener that assumes it sees messages in
+publish order — and don't infer the guarantee from a test that passed, since the memory backend can
+look FIFO by accident. If a queue needs serialized delivery, say so with `:max-concurrent-batches 1`
+(per node) or `:exclusive true` (cluster-wide) — and note that even those give you mutual exclusion,
+not order.
+
+### Limiting concurrency
+
+By default a queue's batches run concurrently, bounded only by the Quartz thread pool — which is
+shared with sync, pulses, and every other scheduled job. A queue that can fan out (one message per
+row, per chart, per whatever) will happily eat that pool. Two knobs bound it:
+
+| Config | Bound | Hard? | Enforced by |
+|------------------------------|------------------------------------------|-------|-------------|
+| `:exclusive true` | 1 batch **cluster-wide** | Yes | The backend (see below) |
+| `:max-concurrent-batches n` | ~n batches **per node** | No — a throttle | Trigger acquisition / `fetch!` free-slot count |
+| *(neither)* | Unbounded, up to the Quartz thread pool | — | — |
+
+`:exclusive` is a **backend** guarantee, and each backend implements it its own way — Quartz with
+`@DisallowConcurrentExecution` on the job class, the memory backend by never fetching a second batch
+for a queue that already has one in flight. The shared poll driver knows nothing about the flag, so a
+new backend has to implement it or it silently won't hold.
+
+**Pick one — they don't layer.** Declaring both throws at registration: `:exclusive` is already the
+strictest limit there is, so a per-node cap on top of it could never bind, and the two are enforced by
+completely different machinery. A queue that wants "roughly one at a time, per node" wants
+`:max-concurrent-batches 1`; a queue that needs *exactly* one at a time wants `:exclusive`.
+
+`:max-concurrent-batches` may be an int or a 0-arg fn, so a cap can be backed by a setting and tuned
+live:
+
+```clojure
+(mq/def-queue! :queue/exploration-query
+ {:transactional :require
+ :max-batch-messages 100
+ :max-concurrent-batches #(explorations.settings/explorations-worker-count)})
+```
+
+A node at its cap **stops taking the work** rather than queueing it internally: it reports the queue as
+one it currently can't handle, and the affinity delegate leaves those triggers `WAITING` in the shared
+store for a node with room. That's the same mechanism that keeps a node from acquiring messages for a
+queue it has no listener for. It's also the only shape of limit a real broker can honor — "fetch at
+most N right now" is SQS's `MaxNumberOfMessages` and a Redis multi-pop count — so poll backends express
+it by asking `fetch!` for a per-queue free-slot count.
+
+**`:max-concurrent-batches` is a throttle, not a guarantee.** Both enforcement points race the work
+they're gating — Quartz decides what to acquire on its scheduler thread *before* the job body runs, and
+a poll backend can hand back a batch it fetched a moment before a slot filled — so a node can end up a
+batch or two over its cap. When that happens the batch is simply **delivered**. It is not bounced: a
+re-queue path would need its own backoff, its own metric, and a hot-loop to avoid if the upstream
+throttle ever stopped working, all to shave an overshoot that is small, bounded, and drains itself.
+Reach for `:exclusive` when a limit has to actually hold.
+
+A queue's config means the same thing on every backend — that's a contract, not a coincidence. The
+backends differ in *mechanism* (Quartz pushes and filters at trigger acquisition; poll backends ask
+`fetch!` for a free-slot count), never in what the config means. If you add a backend, that is the bar.
+
+### Terminal failures: `:on-error`
+
+There is no dead-letter queue. Once a batch exhausts `queue-max-retries` it is dropped, and without a
+handler the only trace is a log line and a `batches-dropped{reason=delivery-exhausted}` counter. That
+is fine for fire-and-forget work, but it silently strands anything the *producer* left behind — a row
+parked in a `pending` status waiting on the handler will sit there forever.
+
+Declare an `:on-error` handler to record the terminal failure durably:
+
+```clojure
+(mq/def-queue! :queue/run-report
+ {:transactional :require
+ :on-error (fn [{:keys [channel messages error attempts]}]
+ (doseq [{:keys [report-id]} messages]
+ ;; Only fail a report that is still pending. See "at-least-once" below: this
+ ;; handler can fire for a batch a peer already completed.
+ (t2/update! :model/Report {:id report-id :status "pending"}
+ {:status "error" :error_message (ex-message error)})))})
+```
+
+It is called on the terminal drop only — not on each failed attempt, and never for a batch that
+recovers on a retry. Exceptions it throws are logged and swallowed: the batch is
+dropped either way, so a broken handler cannot wedge the queue into redelivering an exhausted batch.
+
+**`:on-error` is at-least-once, like delivery itself. Write it to be idempotent, and to tolerate
+firing for work that actually succeeded** — hence the `:status "pending"` guard above.
+
+A message carries no identity. A retry is a fresh trigger with a new id, and the attempt counter
+rides inside it, so two deliveries of one payload are two *independent* batches with two independent
+retry budgets and nothing joining them. If a duplicate delivery fails while the original succeeded,
+the failing copy runs out its own budget and fires `:on-error` for a batch that is already done. That
+is not a corner case reserved for crashes: Quartz's cluster failover re-fires a batch onto a second
+node whenever the first misses a heartbeat, which a *live* node does under a long GC pause or app-DB
+latency — it is never told to stop, and its copy carries on.
+
+`metabase.mq.queue.on-error-test` pins this
+(`duplicate-delivery-fires-on-error-for-an-already-succeeded-payload-test`). Closing it needs message
+identity plus a record of what has already been delivered.
+
+The hook lives in the shared retry-vs-drop policy (`metabase.mq.queue.impl/handle-batch-failure-policy!`)
+rather than in any one backend, so it is inherited by every backend — a poll backend gets it through the
+shared poll driver, a push backend by calling the policy directly (as Quartz does). A backend cannot
+implement a retry budget without going through that policy, so `:on-error` comes along for free with any
+backend added later.
+
## Architecture
```
@@ -152,8 +257,24 @@ Delivery Engine (mq.impl)
```
### Idempotency
-Queue listeners must be idempotent. A batch can return to `pending` after a partial failure, and the same messages will
-be redelivered.
+Queue listeners must be idempotent, and must be safe to run **concurrently with another copy of
+themselves on another node**. Two things to hold onto:
+
+- **A batch can be redelivered.** It returns to `pending` after a partial failure, and the same
+ messages come back.
+- **A batch can be delivered *twice at once*.** Quartz's clustered failover re-fires a batch onto a
+ second node when the first misses a heartbeat — and a live node misses heartbeats under a long GC
+ pause, app-DB latency, or pool exhaustion. It is never interrupted, so both copies run. Separately,
+ a queue that isn't declared `:exclusive` has no cross-node mutual exclusion at all: N nodes × the
+ Quartz thread pool can be in the same listener at once.
+
+A "have I already done this?" check is therefore not enough on its own — two copies can both check,
+both see no, and both proceed. Where that matters, make the *write* safe rather than the check:
+condition the update on the state that justified the message (`{:id x :status "pending"}`), reserve
+via a unique constraint, or take a row lock and re-check inside the transaction.
+
+`with-test-mq`'s `{:duplicate-delivery? true}` (below) delivers every message twice, which will catch
+a listener that assumed exactly-once.
## Testing
diff --git a/src/metabase/mq/core.clj b/src/metabase/mq/core.clj
index 6dca4a67640..ebcb62da7ca 100644
--- a/src/metabase/mq/core.clj
+++ b/src/metabase/mq/core.clj
@@ -1,9 +1,13 @@
(ns metabase.mq.core
"Unified public API for the Metabase messaging subsystem.
- **Queue** — single-consumer, at-least-once delivery. Each message is processed by exactly
- one listener and removed after successful processing. Failed messages are retried up to a
- configurable limit.
+ **Queue** — single-consumer, at-least-once delivery. A queue has exactly one registered listener,
+ which is handed each message and removed after successful processing. Failed messages are retried
+ up to a configurable limit.
+
+ \"Single-consumer\" is about *registration*, not delivery: it means one handler per queue, not that
+ a message is handled exactly once. Delivery is at-least-once, so a listener must be idempotent —
+ see the `Idempotency` section of `metabase/mq/README.md`.
A queue is declared once with `def-queue!` — this is the source of truth for what queues
exist and where all queue config lives (exclusivity, batch size, dedup). The consumer-side
diff --git a/src/metabase/mq/impl.clj b/src/metabase/mq/impl.clj
index da5d3c6abbb..ad62267f902 100644
--- a/src/metabase/mq/impl.clj
+++ b/src/metabase/mq/impl.clj
@@ -52,7 +52,7 @@
(listener messages)
(analytics/inc! :metabase-mq/batches-handled (assoc labels :status "success"))
nil
- (catch Exception e
+ (catch Throwable e
(log/error e (str "Error handling " (namespace channel) " message") labels)
(analytics/inc! :metabase-mq/batches-handled (assoc labels :status "error"))
e)
diff --git a/src/metabase/mq/init.clj b/src/metabase/mq/init.clj
index a7f0f4ab3d5..b0bee9763ee 100644
--- a/src/metabase/mq/init.clj
+++ b/src/metabase/mq/init.clj
@@ -3,7 +3,6 @@
(:require
[metabase.mq.listener :as listener]
[metabase.mq.publish-buffer :as publish-buffer]
- [metabase.mq.quartz-affinity :as quartz-affinity]
[metabase.mq.queue.backend :as q.backend]
[metabase.mq.queue.memory :as q.memory]
[metabase.mq.queue.polling :as q.polling]
@@ -13,7 +12,6 @@
[metabase.mq.task.outbox]
[metabase.mq.task.queue-reaper]
[metabase.startup.core :as startup]
- [metabase.task.bootstrap :as task.bootstrap]
[metabase.util.log :as log]))
(def ^:private queue-backends
@@ -22,11 +20,6 @@
(def ^:private valid-queue-backends (set (keys queue-backends)))
-;; Install the queue node-affinity Quartz DriverDelegate when Quartz's JDBC properties are set.
-;; Registered at load time — `mq` depends on `task`, not the reverse, so `task.bootstrap` calls this
-;; rather than referencing `mq`. install-delegate! falls back to the plain per-DB delegate if the affinity subclass can't be loaded.
-(task.bootstrap/register-jdbc-property-setter! quartz-affinity/install-delegate!)
-
(defn- resolve-backend [label table kw-or-instance]
(if (keyword? kw-or-instance)
(or (get table kw-or-instance)
@@ -59,13 +52,6 @@
(log/infof "Queue backend: %s" queue-be)
(q.registry/register-queues!)
(listener/register-listeners!)
- ;; With the Quartz backend, tell the affinity delegate which queues this node can handle, so it
- ;; only acquires triggers for queues we have a listener for. Queried live, so dynamic
- ;; register/unlisten is reflected. Harmless (never set) for non-Quartz backends, which have no
- ;; queue jobs in the Quartz store.
- (when (= q.quartz/backend-id (q.backend/backend-id queue-instance))
- (quartz-affinity/set-capability-fn!
- (fn [] (into #{} (map name) (listener/queue-names)))))
(let [owns-buffer-flush? (publish-buffer/start-publish-buffer-flush!)
owns-worker-pool? (q.polling/start-worker-pool!)]
(q.backend/start! queue-instance)
diff --git a/src/metabase/mq/memory.clj b/src/metabase/mq/memory.clj
index 82fc07c5ab9..2b5ff466203 100644
--- a/src/metabase/mq/memory.clj
+++ b/src/metabase/mq/memory.clj
@@ -6,7 +6,9 @@
non-durable.
Tests construct isolated layers with [[make-layer]]; the poll context / poll thread lives on the
- backend record, driven by [[metabase.mq.queue.polling]]. This layer only owns the storage.")
+ backend record, driven by [[metabase.mq.queue.polling]]. This layer only owns the storage."
+ (:require
+ [metabase.util :as u]))
(set! *warn-on-reflection* true)
@@ -27,16 +29,65 @@
(let [id (swap! next-id inc)]
(swap! messages assoc id {:id id :queue channel-name :payload payload :failures 0})))
-(defn take-oldest
- "Returns the oldest stored message for `channel-name` as a `{:queue :payload :batch-id}`
- descriptor, or nil if there are none. The message stays in the store (tracked by id) until it's
- acked or dropped — the poll loop's channel-busy gate keeps it from being re-fetched while it's
- in-flight."
+(defn take-pending!
+ "Claims up to `n` not-yet-claimed messages for `channel-name`, marking each in-flight and returning them as
+ `{:queue :payload :batch-id}` descriptors (possibly empty). A nil `n` means unbounded.
+
+ Claiming is what makes it safe to fetch more than one batch per channel at a time: a message stays
+ in the store until it's acked or dropped, so without an in-flight mark a second fetch — issued while
+ the first is still being delivered — would hand out the same message again. `:claimed-at` starts a
+ timer, so [[clear-stale-claims!]] can recover a claim whose delivery never completed.
+
+ The claim is derived from `swap-vals!`'s before/after states rather than captured inside the swap
+ fn, so a retried CAS can't report a message as claimed twice."
+ [{:keys [messages]} channel-name n]
+ (if (and n (not (pos? n)))
+ []
+ (let [claimable? (fn [msg] (and (= channel-name (:queue msg)) (not (:claimed-at msg))))
+ timer (u/start-timer)
+ [old new] (swap-vals! messages
+ (fn [m]
+ (let [available (filter claimable? (vals m))
+ claiming (if n (take n available) available)]
+ (reduce #(assoc-in %1 [%2 :claimed-at] timer)
+ m
+ (map :id claiming)))))]
+ (into []
+ (comp (filter (fn [[id msg]] (and (:claimed-at msg)
+ (not (:claimed-at (get old id))))))
+ (map (fn [[id msg]] {:queue channel-name :payload (:payload msg) :batch-id id})))
+ new))))
+
+(defn has-claim?
+ "True if any message for `channel-name` is currently claimed — i.e. this backend has handed one out
+ and has not yet seen it acked or retried.
+
+ Lets a backend serialize a channel at the store, which is how [[metabase.mq.queue.memory]] implements
+ `:exclusive`."
[{:keys [messages]} channel-name]
- (when-let [{:keys [id payload]} (->> (vals @messages)
- (filter #(= channel-name (:queue %)))
- first)]
- {:queue channel-name :payload payload :batch-id id}))
+ (boolean (some (fn [msg] (and (= channel-name (:queue msg)) (:claimed-at msg)))
+ (vals @messages))))
+
+(defn clear-stale-claims!
+ "Un-claims messages that have been in-flight longer than `stale-timeout-ms`, making them fetchable
+ again. Returns the number recovered.
+
+ Stands in for the visibility timeout a real broker gives you: without it, a delivery that never
+ reaches its ack (its worker thread was killed, or the driver claimed a slot it then couldn't fill)
+ would leave the message marked in-flight and unfetchable forever."
+ [{:keys [messages]} stale-timeout-ms]
+ (let [stale? (fn [msg] (when-let [timer (:claimed-at msg)]
+ (> (u/since-ms timer) stale-timeout-ms)))
+ [old new] (swap-vals! messages
+ (fn [m]
+ (reduce-kv (fn [acc id msg]
+ (cond-> acc
+ (stale? msg) (-> (update-in [id :failures] inc)
+ (update id dissoc :claimed-at))))
+ m
+ m)))]
+ (count (filter (fn [[id msg]] (and (:claimed-at msg) (not (:claimed-at (get new id)))))
+ old))))
(defn message-failures
"Returns how many times message `id` has already failed, or nil if it's no longer stored."
@@ -49,9 +100,17 @@
(swap! messages dissoc id))
(defn retry!
- "Increments the failure count of message `id`, leaving it stored so it gets re-delivered."
+ "Increments the failure count of message `id`, leaving it stored so it gets re-delivered.
+
+ Clearing `:claimed-at` is what makes it re-deliverable: the failed delivery claimed the message,
+ and a claim that is never released is never fetched again."
[{:keys [messages]} id]
- (swap! messages (fn [m] (if (contains? m id) (update-in m [id :failures] inc) m))))
+ (swap! messages (fn [m]
+ (if (contains? m id)
+ (-> m
+ (update-in [id :failures] inc)
+ (update id dissoc :claimed-at))
+ m))))
(defn depths
"Returns a `{:channel :status :count}` row per channel describing how many messages are stored."
diff --git a/src/metabase/mq/queue/backend.clj b/src/metabase/mq/queue/backend.clj
index 3a2cd3c6857..de37cd82066 100644
--- a/src/metabase/mq/queue/backend.clj
+++ b/src/metabase/mq/queue/backend.clj
@@ -15,9 +15,17 @@
(publish! [this queue-name payload]
"Publishes a payload to the given queue. `payload` is an opaque, already-encoded string.
The backend stores/transports it without looking inside.")
- (fetch! [this available-queue-names]
- "Fetches up to one batch per given queue. Returns a seq of
- `{:queue :payload :batch-id}` maps (or nil/empty when there's nothing to do).")
+ (fetch! [this queue->free-slots]
+ "Fetches work for the queues in `queue->free-slots`, a map of queue name → how many batches this
+ node may take for it right now. Fetch no more than that many per queue: the count is the node's
+ remaining capacity, and anything beyond it must be left in the store for a node that can actually
+ run it.
+
+ A count of `nil` means the queue declared no `:max-concurrent-batches` and is therefore unbounded:
+ take as many as you have. Queues already at capacity aren't in the map at all, so a count is
+ never zero.
+
+ Returns a seq of `{:queue :payload :batch-id}` maps (or nil/empty when there's nothing to do).")
(queue-depths [this]
"Returns a seq of `{:channel :status :count}` maps describing current queue depth.")
(batch-successful! [this queue-name batch-id]
diff --git a/src/metabase/mq/queue/concurrency.clj b/src/metabase/mq/queue/concurrency.clj
new file mode 100644
index 00000000000..d402033a796
--- /dev/null
+++ b/src/metabase/mq/queue/concurrency.clj
@@ -0,0 +1,145 @@
+(ns metabase.mq.queue.concurrency
+ "Per-node in-flight accounting for queues that declare `:max-concurrent-batches`.
+
+ **The cap is a soft limit — a throttle, not a guarantee.** It is enforced by not *taking* work.
+
+ What it deliberately does *not* do is refuse a batch that reaches us anyway. If a backend is racy a node
+ can end up a batch or two over its cap. That is fine.
+
+ A queue that declares no cap is unbounded — on every backend. It is never at capacity and reports no
+ fetch limit.
+
+ This is a per-node cap, not a cluster-wide one. For cluster-wide mutual exclusion see the queue's
+ `:exclusive` flag — a queue picks one or the other, never both (the registry rejects a config that
+ declares both). `:exclusive` *is* a hard guarantee; this is not."
+ (:require
+ [metabase.mq.listener :as listener]
+ [metabase.mq.queue.registry :as q.registry])
+ (:import
+ (java.util.concurrent Callable ExecutorService)))
+
+(set! *warn-on-reflection* true)
+
+(def ^:dynamic *in-flight*
+ "channel → how many batches this node is currently delivering for it. A channel with nothing in
+ flight is absent rather than zero, so the map stays small.
+
+ Test fixtures rebind this to a fresh atom so in-flight counts don't leak across scenarios (matching
+ `registry/*queues*` and `listener/*listeners*`). Production code uses the root binding."
+ (atom {}))
+
+(defn in-flight
+ "How many batches of `channel` this node is delivering right now."
+ [channel]
+ (get @*in-flight* channel 0))
+
+(defn- at-limit?
+ "True if `counts` already has `limit` batches of `channel` in flight. A nil `limit` means the queue
+ declared no cap, which is never at the limit."
+ [counts channel limit]
+ (boolean (and limit (>= (get counts channel 0) limit))))
+
+(defn- take-slot!
+ "Records that this node has started delivering a batch of `channel`."
+ [channel]
+ (swap! *in-flight* update channel (fnil inc 0))
+ nil)
+
+(defn- release-slot!
+ "Records that a batch of `channel` has finished delivering.
+
+ Floors at zero rather than going negative: a double-release must not manufacture throughput the
+ queue never declared."
+ [channel]
+ (swap! *in-flight*
+ (fn [counts]
+ (let [remaining (dec (get counts channel 0))]
+ (if (pos? remaining)
+ (assoc counts channel remaining)
+ (dissoc counts channel)))))
+ nil)
+
+(defn do-with-slot
+ "Function form of [[with-slot]]."
+ [channel f]
+ (take-slot! channel)
+ (try
+ (f)
+ (finally
+ (release-slot! channel))))
+
+(defmacro with-slot
+ "Runs `body` while counting one in-flight delivery of `channel` against the node, and stops counting
+ it however `body` exits. Returns `body`'s value."
+ {:style/indent 1}
+ [channel & body]
+ `(do-with-slot ~channel (fn [] ~@body)))
+
+(defn submit-with-slot!
+ "The asynchronous counterpart to [[with-slot]], for a caller that hands delivery off to a thread
+ pool: the slot is taken *here*, on the calling thread, but released on the worker thread once
+ `body-fn` finishes.
+
+ Taking it here rather than inside the task matters — it means a batch counts against the node from
+ the moment it is handed off, so the very next [[free-slots]]/[[at-capacity?]] check sees it, instead
+ of the driver fetching more work in the window before the worker thread gets going.
+
+ Submits `body-fn` (0-arg) to `executor` to run holding the slot. The slot is released however
+ `body-fn` exits, and `after-release` (0-arg) then runs on that same thread — that is where 'a slot
+ just freed up, go look for more work' belongs, since it must not run while the slot is still held.
+
+ If the *submit itself* throws — the pool is shut down, so `body-fn` will never run and never release
+ — the slot is released before the exception propagates. A failed hand-off must not leak capacity.
+
+ Dynamic bindings are conveyed to the worker thread, so the slot is released against the very
+ [[*in-flight*]] that took it (rather than the root binding, which would corrupt test isolation and
+ leak a slot in the fixture's map)."
+ [channel ^ExecutorService executor body-fn after-release]
+ (take-slot! channel)
+ (let [^Callable task (bound-fn []
+ (try
+ (body-fn)
+ (finally
+ (release-slot! channel)
+ (after-release))))]
+ (try
+ (.submit executor task)
+ (catch Throwable t
+ (release-slot! channel)
+ (throw t)))))
+
+(defn at-capacity?
+ "True if this node is already delivering at least `channel`'s `:max-concurrent-batches`. A queue with
+ no declared cap is never at capacity."
+ [channel]
+ (at-limit? @*in-flight* channel (q.registry/max-concurrent-batches channel)))
+
+(defn working?
+ "True if `channel` has at least one delivery in flight on this node. Ask this to find out whether anything is
+ still running (shutdown, test quiescence); ask [[at-capacity?]] to decide whether to take more work."
+ [channel]
+ (pos? (in-flight channel)))
+
+(defn working-channels
+ "The set of channels with at least one delivery in flight on this node."
+ []
+ (set (keys @*in-flight*)))
+
+(defn free-slots
+ "How many more batches of `channel` this node may take right now: its `:max-concurrent-batches` minus
+ what's in flight, floored at zero — or **nil when the queue declares no cap**, meaning take as many
+ as you have.
+
+ A backend may fetch *fewer* than this — that's how a poll backend implements `:exclusive`, which is a
+ backend concern and deliberately not modelled here."
+ [channel]
+ (when-let [limit (q.registry/max-concurrent-batches channel)]
+ (max 0 (- limit (in-flight channel)))))
+
+(defn takeable-queues
+ "The queues this node will take work for *right now*: the ones it has a listener for, minus the ones
+ it is [[at-capacity?]] on.
+
+ Recomputed per call, so a listener registering and a slot freeing up both take effect immediately."
+ []
+ (remove at-capacity? (listener/queue-names)))
diff --git a/src/metabase/mq/queue/impl.clj b/src/metabase/mq/queue/impl.clj
index 2eb159fdcd6..4dc1f344fd4 100644
--- a/src/metabase/mq/queue/impl.clj
+++ b/src/metabase/mq/queue/impl.clj
@@ -2,28 +2,58 @@
"Queue-specific delivery policy — the queue counterpart to the transport-agnostic delivery core in [[metabase.mq.impl]]."
(:require
[metabase.analytics-interface.core :as analytics]
+ [metabase.mq.payload :as payload]
+ [metabase.mq.queue.registry :as q.registry]
[metabase.mq.settings :as mq.settings]
+ [metabase.util :as u]
[metabase.util.log :as log]))
(set! *warn-on-reflection* true)
+(defn- run-on-error!
+ "Invokes the queue's `:on-error` terminal-failure hook, if it declares one, for a batch that is
+ about to be dropped. `payload` is the still-encoded batch.
+
+ The hook is a place to *record* a terminal failure, not a veto over it, so a handler that throws is
+ logged and the batch is dropped anyway. That log is worth keeping: a throwing handler means the row
+ the producer parked in `pending` is now stranded, and this line is the only trace of why."
+ [channel payload error attempts]
+ (when-let [handler (q.registry/on-error channel)]
+ (try
+ (handler {:channel channel
+ :messages (payload/decode payload)
+ :error error
+ :attempts attempts})
+ (catch Throwable t
+ (log/error t "Queue :on-error handler threw; still dropping the batch" {:channel channel})))))
+
(defn handle-batch-failure-policy!
"Retry-vs-drop policy for a just-failed queue batch. `failures` is the number of attempts that have
already failed, so the just-failed attempt makes `(inc failures)` total. `error` is the exception
- that failed the batch, carried through so the give-up is diagnosable. When `(inc failures)` reaches
- `queue-max-retries` the batch is dropped (error-logged *with the cause* + `batches-dropped`
- `reason=delivery-exhausted`) by calling `on-drop`; otherwise `batches-retried` `reason=delivery` is
- emitted, a concise retry line is logged, and `on-retry` re-enqueues it."
- [channel failures error on-retry on-drop]
- (let [max-retries (mq.settings/queue-max-retries)]
- (if (>= (inc failures) max-retries)
+ that failed the batch, carried through so the give-up is diagnosable. `payload` is the encoded
+ batch, needed only to hand the messages to the queue's `:on-error` hook on the drop path.
+
+ When `(inc failures)` reaches `queue-max-retries` the batch is dropped: the queue's `:on-error`
+ hook (if any) runs first so the owner can record the terminal failure durably, then the drop is
+ error-logged *with the cause* + `batches-dropped` `reason=delivery-exhausted` and `on-drop` is
+ called. Otherwise `batches-retried` `reason=delivery` is emitted, a concise retry line is logged,
+ and `on-retry` re-enqueues it."
+ [channel payload failures error on-retry on-drop]
+ (let [max-retries (mq.settings/queue-max-retries)
+ attempts (inc failures)]
+ (if (>= attempts max-retries)
(do
- (log/error error "Dropping queue batch after exhausting retries"
- {:channel channel :max-retries max-retries})
- (analytics/inc! :metabase-mq/batches-dropped {:channel (name channel) :reason "delivery-exhausted"})
+ (run-on-error! channel payload error attempts)
+ (u/ignore-exceptions
+ (log/error error "Dropping queue batch after exhausting retries"
+ {:channel channel :max-retries max-retries})
+ (analytics/inc! :metabase-mq/batches-dropped
+ {:channel (name channel) :reason "delivery-exhausted"}))
(on-drop))
(do
- (log/warnf "Queue batch for %s failed attempt %d of %d; will retry with backoff."
- channel (inc failures) max-retries)
- (analytics/inc! :metabase-mq/batches-retried {:channel (name channel) :reason "delivery"})
+ (u/ignore-exceptions
+ (log/warnf "Queue batch for %s failed attempt %d of %d; will retry with backoff."
+ channel attempts max-retries)
+ (analytics/inc! :metabase-mq/batches-retried
+ {:channel (name channel) :reason "delivery"}))
(on-retry)))))
diff --git a/src/metabase/mq/queue/memory.clj b/src/metabase/mq/queue/memory.clj
index 09a106ecb74..abac3a12f4a 100644
--- a/src/metabase/mq/queue/memory.clj
+++ b/src/metabase/mq/queue/memory.clj
@@ -11,6 +11,7 @@
[metabase.mq.memory :as memory]
[metabase.mq.queue.backend :as q.backend]
[metabase.mq.queue.polling :as q.polling]
+ [metabase.mq.queue.registry :as q.registry]
[metabase.util.log :as log]))
(set! *warn-on-reflection* true)
@@ -19,6 +20,22 @@
"This backend's identifier, registered with `metabase.mq.init`; its `name` labels metrics."
:queue.backend/memory)
+(defn- fetch-count
+ "How many batches to take for `queue` this pass: normally the driver's `free-slots` (nil = unbounded).
+
+ An `:exclusive` queue is capped at one in flight — and only if none already is. That's this
+ backend's implementation of the flag; the poll driver has no notion of exclusivity, and shouldn't:
+ it's the backend that owns delivery, so it's the backend that has to enforce mutual exclusion. Being
+ single-process, one-per-node here *is* the cluster-wide guarantee the flag promises.
+
+ Safe to check-then-take without a lock: only the poll thread claims messages, and everything else
+ (ack, retry, stale recovery) only ever *releases* claims — so the count can fall between the check
+ and the take, never rise."
+ [layer queue free-slots]
+ (if (q.registry/exclusive? queue)
+ (if (memory/has-claim? layer queue) 0 1)
+ free-slots))
+
(defrecord MemoryQueueBackend [layer poll-context]
q.backend/QueueBackend
@@ -27,8 +44,11 @@
(publish! [_this queue-name payload]
(memory/publish! layer queue-name payload))
- (fetch! [_this available-queues]
- (into [] (keep #(memory/take-oldest layer %)) available-queues))
+ (fetch! [_this queue->free-slots]
+ (into []
+ (mapcat (fn [[queue free-slots]]
+ (memory/take-pending! layer queue (fetch-count layer queue free-slots))))
+ queue->free-slots))
(queue-depths [_this]
(memory/depths layer))
@@ -46,15 +66,23 @@
(fail-batch! [_this _queue-name batch-id]
(memory/ack! layer batch-id))
- ;; No crashed-consumer recovery (single process), but this periodic maintenance hook is reused to
- ;; reclaim messages for queues that have no listener: on a single-process, non-durable backend such
- ;; messages can never be delivered, so leaving them would grow the store and the depth gauge without
- ;; bound (unlike a persistent, cross-node backend, where another node may own the listener).
- (recover-stale! [_this _stale-timeout-ms _max-retries]
+ ;; There is no crashed *node* to recover from (single process), but there is still a crashed
+ ;; *delivery*: a message claimed by `take-pending!` whose worker never acked it would stay marked
+ ;; in-flight and never be fetched again. Clearing those stale claims is the in-memory equivalent of
+ ;; a broker's visibility timeout.
+ ;;
+ ;; The hook is also reused to reclaim messages for queues that have no listener: on a
+ ;; single-process, non-durable backend such messages can never be delivered, so leaving them would
+ ;; grow the store and the depth gauge without bound (unlike a persistent, cross-node backend, where
+ ;; another node may own the listener).
+ (recover-stale! [_this stale-timeout-ms _max-retries]
(let [dropped (memory/drop-orphaned! layer (set (listener/queue-names)))]
(when (pos? dropped)
(log/debugf "Dropped %d orphaned in-memory message(s) for queues with no listener" dropped)))
- nil)
+ (let [recovered (memory/clear-stale-claims! layer stale-timeout-ms)]
+ (when (pos? recovered)
+ (log/warnf "Recovered %d in-memory message(s) whose delivery never completed" recovered)
+ [{:channel :memory :recovered recovered :failed 0}])))
(run-heartbeats! [_this] nil)
diff --git a/src/metabase/mq/queue/polling.clj b/src/metabase/mq/queue/polling.clj
index 3653013e0a1..e0d4a0acecb 100644
--- a/src/metabase/mq/queue/polling.clj
+++ b/src/metabase/mq/queue/polling.clj
@@ -7,40 +7,24 @@
(:require
[metabase.analytics-interface.core :as analytics]
[metabase.mq.impl :as mq.impl]
- [metabase.mq.listener :as listener]
[metabase.mq.polling :as mq.polling]
[metabase.mq.queue.backend :as q.backend]
+ [metabase.mq.queue.concurrency :as q.concurrency]
[metabase.mq.queue.impl :as q.impl]
[metabase.mq.settings :as mq.settings]
[metabase.util.log :as log])
(:import
- (java.util.concurrent Callable ExecutorService Executors Future)))
+ (java.util.concurrent ExecutorService Executors)))
(set! *warn-on-reflection* true)
-;;; ------------------------------- Worker pool & in-flight (busy) tracking -------------------------------
-
-(def ^:private active-handlers
- "channel keyword → {:future Future, :gen Object}. Tracks which channels have an in-flight delivery
- on the worker pool, so a channel is delivered by at most one worker at a time (and so publishers
- know not to wake the loop for a channel that's already being processed)."
- (atom {}))
+;;; ------------------------------- Worker pool -------------------------------
(def ^:private worker-pool
"Shared thread pool that runs message delivery off the poll thread, so a slow listener doesn't
stall polling."
(atom nil))
-(defn channel-busy?
- "Returns true if `channel` currently has an in-flight delivery on the worker pool."
- [channel]
- (contains? @active-handlers channel))
-
-(defn busy-channels
- "Returns the set of channels that currently have an in-flight delivery."
- []
- (set (keys @active-handlers)))
-
(defn start-worker-pool!
"Starts the shared worker pool for non-blocking message delivery. Idempotent.
Returns `true` if THIS call created the pool, `false` if one was already running."
@@ -51,24 +35,26 @@
(do (.shutdown ^ExecutorService pool) false))))
(defn shutdown-worker-pool!
- "Cancels all active handler futures and shuts down the worker pool."
+ "Shuts down the worker pool, interrupting any deliveries still running, and clears the in-flight
+ counts.
+
+ `shutdownNow` interrupts every actively-executing task, so there is nothing to cancel by hand. The
+ interrupted deliveries still run their `finally` and release their slots."
[]
- (doseq [[_ {:keys [^Future future]}] @active-handlers]
- (when future
- (.cancel future true)))
- (reset! active-handlers {})
(when-let [^ExecutorService pool @worker-pool]
(.shutdownNow pool)
- (reset! worker-pool nil)))
+ (reset! worker-pool nil))
+ (reset! q.concurrency/*in-flight* {}))
(defn- handle-batch-failure!
"Decides retry vs. permanent failure for a just-failed batch based on its prior failure count, then
tells the backend to re-enqueue or drop it. No-ops if the backend no longer owns/knows the batch
(`failure-count` returns nil). `error` is the exception that failed the batch, passed through for
- diagnostic logging."
- [backend channel batch-id error]
+ diagnostic logging; `payload` is the encoded batch, passed through so the shared policy can hand
+ the messages to the queue's `:on-error` hook if it drops them."
+ [backend channel payload batch-id error]
(when-let [failures (q.backend/failure-count backend channel batch-id)]
- (q.impl/handle-batch-failure-policy! channel failures error
+ (q.impl/handle-batch-failure-policy! channel payload failures error
#(q.backend/retry-batch! backend channel batch-id)
#(q.backend/fail-batch! backend channel batch-id))))
@@ -82,60 +68,22 @@
(let [result (mq.impl/deliver! channel payload)]
(cond
(= mq.impl/no-listener result) nil
- (instance? Throwable result) (when batch-id (handle-batch-failure! backend channel batch-id result))
- :else (when batch-id (q.backend/batch-successful! backend channel batch-id)))))
+ (instance? Throwable result) (when batch-id (handle-batch-failure! backend channel payload batch-id result))
+ :else (when batch-id (q.backend/batch-successful! backend channel batch-id)))))
(defn submit-delivery!
- "Submits one fetched batch to the worker pool for non-blocking delivery. Returns true if submitted,
- false if `channel` already has an in-flight delivery (enforcing at most one worker per channel).
- Uses `bound-fn` to convey dynamic bindings to the worker thread.
-
- The busy-check and slot claim are atomic via a single `swap-vals!`. A per-submission generation
- object prevents a completed future's cleanup from clobbering a resubmission that won the slot
- between the finally and the dissoc."
+ "Submits one fetched batch to the worker pool for non-blocking delivery."
[channel payload batch-id backend]
- (let [gen (Object.)
- [old _] (swap-vals! active-handlers
- (fn [handlers]
- (if (contains? handlers channel)
- handlers
- (assoc handlers channel {:future nil :gen gen}))))]
- (if (contains? old channel)
- false
- (let [^Callable task (bound-fn []
- (try
- (deliver-and-ack! channel payload batch-id backend)
- (finally
- ;; Only remove if this generation still owns the slot
- (swap! active-handlers
- (fn [handlers]
- (if (identical? gen (:gen (get handlers channel)))
- (dissoc handlers channel)
- handlers)))
- (mq.polling/notify-all!))))
- f (try
- (.submit ^ExecutorService @worker-pool task)
- (catch Throwable t
- ;; The slot was already claimed above. If the submit itself fails (e.g. the
- ;; worker pool isn't running / was shut down, so the deref or .submit throws),
- ;; release the slot we claimed before propagating — otherwise the channel stays
- ;; marked busy forever and the queue silently stops delivering.
- (swap! active-handlers
- (fn [handlers]
- (if (identical? gen (:gen (get handlers channel)))
- (dissoc handlers channel)
- handlers)))
- (throw t)))]
- ;; Only set the future if this generation still owns the slot
- (swap! active-handlers
- (fn [handlers]
- (if (identical? gen (:gen (get handlers channel)))
- (assoc-in handlers [channel :future] f)
- handlers)))
- true))))
+ (q.concurrency/submit-with-slot!
+ channel
+ @worker-pool
+ #(deliver-and-ack! channel payload batch-id backend)
+ ;; Runs once the slot is released: a freed slot may mean there's room to fetch more, so re-drive the
+ ;; poll loop rather than waiting out its idle interval.
+ mq.polling/notify-all!))
(def ^:private stale-check-interval-ms (* 60 1000))
-(def ^:private heartbeat-interval-ms (* 2 60 1000))
+(def ^:private heartbeat-interval-ms (* 2 60 1000))
(def ^:private depth-gauge-interval-ms (* 30 1000))
(def ^:private stale-processing-timeout-ms
@@ -172,14 +120,15 @@
{:channel (name channel) :reason "stale-recovery-exhausted"} failed))))))
(defn notify-on-publish!
- "Wakes the poll loop for `poll-context` so it picks up a freshly published message — unless
- `channel` is already being processed, in which case the loop re-checks when that finishes.
- Called by the transport right after `publish!` (the caller reads the backend's `:poll-context`).
+ "Wakes the poll loop for `poll-context` so it picks up a freshly published message — unless `channel`
+ is already at its concurrency cap, in which case the loop re-checks when one of those deliveries
+ finishes and frees a slot. Called by the transport right after `publish!` (the caller reads the
+ backend's `:poll-context`).
No-ops when `poll-context` is nil: push backends (e.g. Quartz) have no poll loop to wake — their
`publish!` already arranges delivery — so they carry no poll context."
[poll-context channel]
- (when (and poll-context (not (channel-busy? channel)))
+ (when (and poll-context (not (q.concurrency/at-capacity? channel)))
(mq.polling/notify! (:poll-state poll-context))))
(defn make-poll-context
@@ -194,10 +143,22 @@
:last-heartbeat-ms (atom 0)
:last-depth-gauge-ms (atom 0)})
+(defn- free-slots-by-queue
+ "The poll driver's shape of [[q.concurrency/takeable-queues]]: queue → how many batches this node may
+ still take for it. A queue with no declared cap maps to `nil` — unbounded, take what you have.
+
+ Queues at capacity are absent entirely rather than mapped to 0, so [[q.backend/fetch!]] is never even
+ asked about them: a node at its limit on a queue simply doesn't take more of its work, leaving it in
+ the store for a node that can run it."
+ []
+ (into {}
+ (map (juxt identity q.concurrency/free-slots))
+ (q.concurrency/takeable-queues)))
+
(defn- poll-iteration!
- "One iteration: run each maintenance task if its interval has elapsed, then fetch and submit
- one batch per available queue. Returns true if any work was found so the polling
- thread respins immediately."
+ "One iteration: run each maintenance task if its interval has elapsed, then fetch and submit up to
+ each queue's free-slot count. Returns true if any work was found so the polling thread respins
+ immediately."
[backend {:keys [last-stale-check-ms last-heartbeat-ms last-depth-gauge-ms]}]
(let [backend-name (name (q.backend/backend-id backend))]
(mq.polling/periodically! last-stale-check-ms stale-check-interval-ms "stale recovery"
@@ -207,8 +168,8 @@
(mq.polling/periodically! last-depth-gauge-ms depth-gauge-interval-ms "depth gauge"
#(report-depths! backend backend-name))
(let [found? (boolean
- (when-let [available (seq (remove channel-busy? (listener/queue-names)))]
- (when-let [batches (seq (q.backend/fetch! backend available))]
+ (when-let [wanted (not-empty (free-slots-by-queue))]
+ (when-let [batches (seq (q.backend/fetch! backend wanted))]
(doseq [{:keys [queue payload batch-id]} batches]
(submit-delivery! queue payload batch-id backend))
true)))]
diff --git a/src/metabase/mq/queue/quartz.clj b/src/metabase/mq/queue/quartz.clj
index 1c0d8242238..00e560ede62 100644
--- a/src/metabase/mq/queue/quartz.clj
+++ b/src/metabase/mq/queue/quartz.clj
@@ -30,9 +30,12 @@
[clojurewerkz.quartzite.triggers :as triggers]
[metabase.analytics-interface.core :as analytics]
[metabase.mq.impl :as mq.impl]
+ [metabase.mq.quartz-affinity :as quartz-affinity]
[metabase.mq.queue.backend :as q.backend]
+ [metabase.mq.queue.concurrency :as q.concurrency]
[metabase.mq.queue.impl :as q.impl]
[metabase.mq.queue.registry :as q.registry]
+ [metabase.task.bootstrap :as task.bootstrap]
[metabase.task.core :as task]
[metabase.util.log :as log])
(:import
@@ -45,6 +48,13 @@
"This backend's identifier, registered with `metabase.mq.init`; its `name` labels metrics."
:queue.backend/quartz)
+;; Install the queue node-affinity DriverDelegate when Quartz's JDBC properties are set. Registered at
+;; load time, and here rather than in `metabase.mq.init`, because it is a property of *this* backend —
+;; init has no business knowing that one backend swaps out a Quartz internal. (`mq` depends on `task`,
+;; not the reverse, so `task.bootstrap` calls back into this rather than referencing `mq`.)
+;; install-delegate! falls back to the plain per-DB delegate if the affinity subclass can't be loaded.
+(task.bootstrap/register-jdbc-property-setter! quartz-affinity/install-delegate!)
+
(def ^:private job-group
"Quartz group every queue job/trigger lives in, so they're easy to isolate from the rest of the
scheduler's jobs."
@@ -105,11 +115,64 @@
rare race where the listener was torn down between acquisition and execution.
We must *re-schedule* not refire: a refire re-runs on THIS node and bypasses acquisition, so it would
- loop here until node shutdown and never reach a capable node."
- [^Scheduler scheduler queue payload attempt]
+ loop here until node shutdown and never reach a capable node.
+
+ The re-queued trigger keeps the batch's original `start-at` instead of taking `now`. The age reaper
+ ([[metabase.mq.task.queue-reaper]]) gives up on a never-acquired queue trigger by its `start_time`,
+ and a bounce writes a *fresh* trigger — so stamping `now` would reset the message's reaper clock on
+ every bounce, and a message nobody can handle could never age out. That is exactly the message the
+ reaper exists to drop, and nothing else bounds this path: a bounce carries no backoff and spends no
+ retry budget. An already-due `start-at` is also what we want on the way out: the batch should reach
+ a capable node as soon as one asks for work."
+ [^Scheduler scheduler queue payload attempt ^Date start-at]
(analytics/inc! :metabase-mq/batches-retried {:channel (name queue) :reason "no-listener"})
(log/debugf "Queue %s has no listener on this node; re-queuing for a node that does." queue)
- (schedule-message-trigger! scheduler queue payload attempt (Date.)))
+ (schedule-message-trigger! scheduler queue payload attempt start-at))
+
+;;; ------------------------------------ Waking the acquire loop ------------------------------------
+
+(task/defjob ^{:doc "Never fires. Exists only to give [[wake-scheduler!]] a durable job key to call
+ `resumeJob` on."}
+ QueueSlotNudgeJob
+ [_ctx])
+
+(def ^:private nudge-job-key (jobs/key "queue-slot-nudge" job-group))
+
+(defonce ^:private nudge-job-ensured? (atom false))
+
+(defn- ensure-nudge-job!
+ "Idempotently creates the durable, trigger-less job that [[wake-scheduler!]] pokes."
+ [^Scheduler scheduler]
+ (when-not @nudge-job-ensured?
+ (.addJob scheduler
+ (jobs/build
+ (jobs/of-type QueueSlotNudgeJob)
+ (jobs/with-identity nudge-job-key)
+ (jobs/store-durably))
+ true)
+ (reset! nudge-job-ensured? true)))
+
+(defn- wake-scheduler!
+ "Tells Quartz's trigger-acquisition loop to look again, right now.
+
+ This is *required* for `:max-concurrent-batches` to work, not an optimization. Once the acquisition
+ filter excludes a queue we're at capacity on, the scheduler thread finds nothing to acquire and
+ parks on its `sigLock` for up to `idleWaitTime` (30s by default) — and nothing wakes it when one of
+ our batches finishes and frees a slot, because Quartz only signals on completion for
+ `@DisallowConcurrentExecution` jobs. Without this poke a capped queue would drain N batches, sleep
+ 30s, drain N more.
+
+ `resumeJob` is the cheapest *public* API that signals that loop: `QuartzScheduler.resumeJob` ends in
+ `notifySchedulerThread(0L)`, which is exactly the signal the acquire loop is waiting on. We aim it at
+ a job with no triggers, so the resume itself has nothing to resume and the whole call is a zero-row
+ no-op — we are calling it purely for that side effect on the scheduler thread."
+ [^Scheduler scheduler queue]
+ (when (q.registry/max-concurrent-batches queue)
+ (try
+ (ensure-nudge-job! scheduler)
+ (.resumeJob scheduler nudge-job-key)
+ (catch Throwable t
+ (log/debugf t "Could not wake the Quartz acquire loop after freeing a slot on %s" queue)))))
(defn- deliver-batch!
"Shared job body. Reads the merged job+trigger data and dispatches on the delivery result:
@@ -131,29 +194,39 @@
on Quartz's worker threads; in a cluster the retry lands in the shared JDBC store so any node can
pick it up."
[^JobExecutionContext ctx]
- (let [data (.getMergedJobDataMap ctx)
- queue (keyword "queue" (.getString data data-queue))
- payload (.getString data data-payload)
- attempt (Integer/parseInt (or (.getString data data-attempt) "0"))]
+ (let [data (.getMergedJobDataMap ctx)
+ queue (keyword "queue" (.getString data data-queue))
+ payload (.getString data data-payload)
+ attempt (Integer/parseInt (or (.getString data data-attempt) "0"))
+ scheduler (.getScheduler ctx)]
(try
- (let [result (mq.impl/deliver! queue payload)]
- (cond
- (= mq.impl/no-listener result)
- (requeue-no-listener! (.getScheduler ctx) queue payload attempt)
-
- (instance? Throwable result)
- (let [next-attempt (inc attempt)]
- (q.impl/handle-batch-failure-policy!
- queue attempt result
- #(schedule-message-trigger! (.getScheduler ctx) queue payload next-attempt
- (Date. (long (+ (System/currentTimeMillis) (retry-delay-ms next-attempt)))))
- (constantly nil)))))
+ (q.concurrency/with-slot queue
+ (let [result (mq.impl/deliver! queue payload)]
+ (cond
+ (= mq.impl/no-listener result)
+ ;; read the trigger lazily — only this path needs it
+ (requeue-no-listener! scheduler queue payload attempt
+ (.getStartTime (.getTrigger ctx)))
+
+ (instance? Throwable result)
+ (let [next-attempt (inc attempt)]
+ (q.impl/handle-batch-failure-policy!
+ queue payload attempt result
+ #(schedule-message-trigger! scheduler queue payload next-attempt
+ (Date. (long (+ (System/currentTimeMillis) (retry-delay-ms next-attempt)))))
+ (constantly nil))))))
+ ;; `with-slot` has returned, so the slot is already released — and a freed slot is invisible to
+ ;; Quartz's acquire loop, which may be parked for up to `idleWaitTime` with this queue filtered
+ ;; out. Tell it to look again. (Waking *before* the release would be pointless: the loop would
+ ;; re-check, still see us at capacity, and park again.)
+ (wake-scheduler! scheduler queue)
;; Delivered, or the retry/requeue is durably committed. Return nil so Quartz completes the
;; one-shot trigger.
nil
(catch Throwable t
;; We couldn't hand the message off (couldn't schedule the retry, etc.). Don't let Quartz drop
- ;; the fired one-shot — have it re-run this job instead.
+ ;; the fired one-shot — have it re-run this job instead. `with-slot` has already released the
+ ;; slot on its way out, so the refire won't find the queue at capacity because of us.
(backoff-before-refire!)
(throw (doto (JobExecutionException. t) (.setRefireImmediately true)))))))
@@ -208,6 +281,21 @@
true)))
(swap! ensured-jobs conj [queue exclusive?])))
+;;; ------------------------------------ Node affinity (acquisition) ------------------------------------
+
+(defn capable-queue-names
+ "The queue names this node is willing to acquire triggers for *right now* — the set the affinity
+ delegate splices into Quartz's acquire query.
+
+ Quartz's shape of the backend-agnostic [[q.concurrency/takeable-queues]]: strings, because that's what
+ a Quartz job is named. A queue drops out for having no listener here or for being at its
+ `:max-concurrent-batches`, which mean the same thing to the acquisition filter — don't take this work,
+ leave it in the shared store for a node that can run it.
+
+ Called on every acquire, so a listener registering and a slot freeing up both take effect at once."
+ []
+ (into #{} (map name) (q.concurrency/takeable-queues)))
+
(defrecord QuartzQueueBackend []
q.backend/QueueBackend
(backend-id [_this] backend-id)
@@ -230,11 +318,18 @@
queue)
{:queue queue :backend backend-id}))))
+ ;; There is no poll loop to start. What this backend *does* need at startup is its node-affinity
+ ;; filter: tell the delegate which queues we'll acquire triggers for. That is a Quartz-specific
+ ;; acquisition strategy, so it's wired here rather than in `metabase.mq.init` — the init ns shouldn't
+ ;; have to know that one particular backend gates its own intake.
+ (start! [_this]
+ (quartz-affinity/set-capability-fn! capable-queue-names))
+
+ (shutdown! [_this])
+
;; Push backend — Quartz drives everything below, so these poll-driver hooks are never invoked
;; (the poll loop is never started) and exist only to satisfy the protocol.
- (start! [_this])
- (shutdown! [_this])
- (fetch! [_this _available-queues] nil)
+ (fetch! [_this _queue->free-slots] nil)
(queue-depths [_this] nil)
(batch-successful! [_this _queue-name _batch-id] nil)
(failure-count [_this _queue-name _batch-id] nil)
diff --git a/src/metabase/mq/queue/registry.clj b/src/metabase/mq/queue/registry.clj
index bdf0b00a6ce..b9622bf0c98 100644
--- a/src/metabase/mq/queue/registry.clj
+++ b/src/metabase/mq/queue/registry.clj
@@ -6,6 +6,7 @@
in the cluster. Backends use this registry at startup to pre-arrange broker-side resources
(e.g. RabbitMQ queue declarations) for the full set of known queues."
(:require
+ [metabase.util.log :as log]
[metabase.util.malli.humanize :as mu.humanize]
[metabase.util.malli.registry :as mr]))
@@ -37,12 +38,20 @@
#(= "queue" (namespace %))]])
(mr/def :metabase.mq.queue/queue-config
- [:map {:closed true}
- [:transactional (into [:enum {:error/message (str "must be one of " transactional-modes)}]
- transactional-modes)]
- [:exclusive {:optional true} :boolean]
- [:max-batch-messages {:optional true} pos-int?]
- [:dedup-fn {:optional true} fn?]])
+ [:and
+ [:map {:closed true}
+ [:transactional (into [:enum {:error/message (str "must be one of " transactional-modes)}]
+ transactional-modes)]
+ [:exclusive {:optional true} :boolean]
+ [:max-batch-messages {:optional true} pos-int?]
+ [:max-concurrent-batches {:optional true} [:or pos-int? fn?]]
+ [:dedup-fn {:optional true} fn?]
+ [:on-error {:optional true} fn?]]
+ [:fn {:error/message (str "a queue cannot declare both :exclusive and :max-concurrent-batches — "
+ ":exclusive already limits it to one batch cluster-wide, which is "
+ "stricter than any per-node cap. Drop one.")}
+ (fn [{:keys [exclusive max-concurrent-batches]}]
+ (not (and exclusive max-concurrent-batches)))]])
(def ^:dynamic *queues*
"queue-name (keyword) → config map for every declared queue.
@@ -96,11 +105,40 @@
[queue-name]
(or (:max-batch-messages (get @*queues* queue-name)) default-max-batch-messages))
+(defn max-concurrent-batches
+ "Returns how many batches of `queue-name` a single node may deliver at once, or nil if the queue
+ declares no cap.
+
+ A queue may declare the cap as a literal int or as a 0-arg fn; a fn is resolved on every call, so a
+ cap backed by a `defsetting` tracks that setting live rather than freezing its value at
+ registration. A nil cap means unbounded.
+
+ A fn cap that throws is caught and treated as unbounded (nil): the cap is a best-effort throttle,
+ so degrading to unbounded keeps work flowing rather than wedging the scheduler in quartz or other places."
+ [queue-name]
+ (let [cap (:max-concurrent-batches (get @*queues* queue-name))]
+ (if (fn? cap)
+ (try
+ (cap)
+ (catch Throwable t
+ (log/warnf t "Error resolving :max-concurrent-batches for %s; treating as unbounded" queue-name)
+ nil))
+ cap)))
+
(defn dedup-fn
"Returns the `:dedup-fn` declared for `queue-name`, or nil if none."
[queue-name]
(:dedup-fn (get @*queues* queue-name)))
+(defn on-error
+ "Returns the `:on-error` handler declared for `queue-name`, or nil if none.
+
+ The handler is the queue's terminal-failure hook: it runs when a batch has exhausted
+ `queue-max-retries` and is about to be dropped, giving the owning module a chance to record the
+ failure durably (mark rows failed, alert, dead-letter)."
+ [queue-name]
+ (:on-error (get @*queues* queue-name)))
+
(defn transactional
"Returns the `:transactional` mode (`:require`/`:try`/`:never`) declared for `queue-name`, or nil
if no such queue has been declared."
@@ -137,11 +175,41 @@
Optional config keys:
`:exclusive` — when true, at most one batch for this queue is in-flight cluster-wide.
+ Mutually exclusive with `:max-concurrent-batches` (declaring both throws).
+ `:max-concurrent-batches`
+ — how many batches of this queue a single node will deliver at once. A node
+ already running that many stops *taking* new ones: the Quartz backend
+ won't acquire the queue's triggers and the poll driver won't fetch its
+ messages, so the work stays in the shared store for a node with capacity.
+ May be an int, or a 0-arg fn (resolved per check, so a cap backed by a
+ `defsetting` tracks it live).
+
+ This is the per-node middle ground between `:exclusive` (exactly 1
+ cluster-wide) and the default (unbounded) — pick one of the three, they
+ are not layered. Declaring it alongside `:exclusive` throws: cluster-wide
+ mutual exclusion is already stricter than any per-node cap, so the cap
+ could never bind, and the two are enforced by different machinery.
+
+ Omitting it means unbounded, on every backend.
+
+ It is a *throttle, not a guarantee*: it bounds what a node **takes**, and
+ that check races the work it gates, so a node can end up a batch or two
+ over the cap — in which case the batch is simply delivered, not handed
+ back. Use `:exclusive` when a limit must actually hold.
`:max-batch-messages` — soft target batch size for this queue (defaults to
[[default-max-batch-messages]]). Used at publish time (coalescing) and
consume time (slicing).
`:dedup-fn` — `messages -> messages` applied at publish time to drop duplicates
from a batch before it is buffered.
+ `:on-error` — terminal-failure hook, called with a single map
+ `{:channel :messages :error :attempts}` when a batch has exhausted
+ `queue-max-retries` and is about to be dropped. Use it to record the failure
+ durably. Exceptions thrown by the handler are logged and swallowed: the
+ batch is dropped either way, so a broken handler can't lock the queue.
+
+ It fires on the terminal drop only — never for a batch that recovers on a
+ retry — but it is *at-least-once*, so make it idempotent and make it
+ tolerate firing for work that already succeeded.
These are all properties of the queue itself — they take effect at publish time, on every
node, regardless of whether a listener is registered locally.
diff --git a/test/metabase/mq/backend_parity_test.clj b/test/metabase/mq/backend_parity_test.clj
index ac327749fe3..be65d741235 100644
--- a/test/metabase/mq/backend_parity_test.clj
+++ b/test/metabase/mq/backend_parity_test.clj
@@ -18,13 +18,21 @@
(:require
[clojure.test :refer :all]
[metabase.mq.core :as mq]
+ [metabase.mq.queue.registry :as q.registry]
[metabase.mq.test-util :as mq.tu]))
(set! *warn-on-reflection* true)
(def ^:private backend-kinds
"Backend kinds exercised by the parity scenarios. `:memory` covers the
- fast async in-process backend."
+ fast async in-process backend.
+
+ Quartz is not here, and can't easily be: it fires jobs on its own worker
+ threads, which don't inherit the dynamic bindings `with-test-mq` relies on
+ for isolation (see `metabase.mq.queue.quartz-test`, which drives that
+ backend directly instead). So a scenario passing here proves parity across
+ the *poll* backends, and pins the contract for the next one added — it does
+ not by itself prove Quartz agrees."
[:memory])
(def ^:private delivery-modes
@@ -97,6 +105,48 @@
"Message delivered at least twice (retry-after-failure or chaos duplication)")
(mq/unlisten! queue-name)))))
+(deftest exclusive-queue-runs-one-batch-at-a-time-test
+ (testing "an :exclusive queue never has two batches in its listener at once, on any backend.
+
+ This is the parity scenario most worth having, because `:exclusive` is the one guarantee
+ each backend must implement *itself* — Quartz with @DisallowConcurrentExecution on the job
+ class, the memory backend by not fetching a second batch while one is in flight. The shared
+ poll driver implements it not at all, by design. So a backend that simply forgets the flag
+ keeps working, delivers everything, and quietly offers no mutual exclusion whatsoever —
+ which is exactly what the memory backend did until this was noticed. Nothing but a
+ behavioral test catches that."
+ (run-parity!
+ :queue/parity-exclusive
+ (fn [ctx queue-name]
+ (let [in-flight (atom 0)
+ peak (atom 0)
+ received (atom [])]
+ ;; one message per batch, so "batches at once" is what the listener actually observes
+ (q.registry/register-queue! queue-name {:transactional :try
+ :exclusive true
+ :max-batch-messages 1})
+ (mq.tu/listen! queue-name
+ (fn [msg]
+ (swap! peak max (swap! in-flight inc))
+ ;; hold the batch long enough that an overlapping delivery would be seen
+ (Thread/sleep 50)
+ (swap! received conj msg)
+ (swap! in-flight dec)))
+ (mq/with-queue queue-name [q]
+ (mq/put q "a")
+ (mq/put q "b")
+ (mq/put q "c"))
+ (mq.tu/eventually! ctx
+ #(= #{"a" "b" "c"} (set @received))
+ scenario-timeout-ms)
+ (is (= 1 @peak)
+ "never two batches of an exclusive queue in the listener at the same time")
+ ;; set, not sequence: exclusivity buys mutual exclusion, not ordering — and under
+ ;; :duplicate delivery each message legitimately arrives more than once.
+ (is (= #{"a" "b" "c"} (set @received))
+ "and every message is still delivered — exclusivity serializes, it doesn't drop")
+ (mq/unlisten! queue-name))))))
+
(deftest queue-normalizes-message-shape-test
(testing "Every backend delivers messages in the same canonical JSON-normalized shape:
keyword keys preserved, keyword/symbol values and other non-JSON-native
diff --git a/test/metabase/mq/quartz_affinity_test.clj b/test/metabase/mq/quartz_affinity_test.clj
index 22df00f5192..5b636f887cd 100644
--- a/test/metabase/mq/quartz_affinity_test.clj
+++ b/test/metabase/mq/quartz_affinity_test.clj
@@ -3,8 +3,11 @@
[clojure.string :as str]
[clojure.test :refer :all]
[metabase.classloader.core :as classloader]
+ [metabase.mq.listener :as listener]
[metabase.mq.quartz-affinity :as affinity]
+ [metabase.mq.queue.concurrency :as q.concurrency]
[metabase.mq.queue.quartz :as q.quartz]
+ [metabase.mq.queue.registry :as q.registry]
[metabase.test.fixtures :as fixtures]
[toucan2.core :as t2])
(:import
@@ -117,3 +120,53 @@
(is (= #{"aff-sync"} (acquire! sched #{"nonexistent"}))
"capable of an unrelated queue → only the non-queue job")
(finally (cleanup! sched))))))
+
+;;; The test above pins the filter against a hand-supplied capability set. This one wires up the *real*
+;;; capability fn (`q.quartz/capable-queue-names`) and proves the thing the concurrency cap actually
+;;; depends on: a node at its `:max-concurrent-batches` stops acquiring that queue's triggers from the
+;;; shared store — the same way it stops acquiring a queue it has no listener for.
+
+(defn- acquire-with-real-capability-fn! [sched]
+ (let [the-atom @#'affinity/capability-fn*
+ prev @the-atom
+ now (epoch-ms)]
+ (affinity/set-capability-fn! q.quartz/capable-queue-names)
+ (try
+ (t2/with-connection [conn]
+ (->> (affinity/select-trigger-to-acquire conn (+ now 3600000) (- now 3600000) 100 (test-rtp sched))
+ (map (fn [^TriggerKey k] (.getName k)))
+ set))
+ (finally (reset! the-atom prev)))))
+
+(deftest at-capacity-queue-is-not-acquired-against-appdb-test
+ (testing "a node at a queue's :max-concurrent-batches stops acquiring its triggers, so they stay
+ WAITING in the shared store for a node with a free slot"
+ (let [sched (str "AFFINITY_CAP_TEST_" (random-uuid))
+ capped (keyword "queue" (str "affcap-capped-" (random-uuid)))
+ uncapped (keyword "queue" (str "affcap-uncapped-" (random-uuid)))]
+ (q.registry/register-queue! capped {:transactional :try :max-concurrent-batches 1})
+ (q.registry/register-queue! uncapped {:transactional :try})
+ (listener/batch-listen! capped (fn [_] nil))
+ (listener/batch-listen! uncapped (fn [_] nil))
+ (cleanup! sched)
+ (seed! sched [[(name capped) mq-group] [(name uncapped) mq-group] ["affcap-sync" "DEFAULT"]])
+ (try
+ (testing "with a free slot, the capped queue's trigger is acquirable"
+ (is (= #{(name capped) (name uncapped) "affcap-sync"}
+ (acquire-with-real-capability-fn! sched))))
+ (q.concurrency/with-slot capped
+ (testing "at capacity, its trigger is not selected by the acquire query at all"
+ (is (= #{(name uncapped) "affcap-sync"}
+ (acquire-with-real-capability-fn! sched))
+ "the node at its cap leaves the capped queue's work in the store — and an uncapped
+ queue and non-queue jobs (sync, pulses) are untouched, so a saturated queue can't
+ starve them")))
+ (testing "once the slot is released, its trigger is acquirable again"
+ (is (= #{(name capped) (name uncapped) "affcap-sync"}
+ (acquire-with-real-capability-fn! sched))))
+ (finally
+ (cleanup! sched)
+ (listener/unlisten! capped)
+ (listener/unlisten! uncapped)
+ (swap! q.registry/*queues* dissoc capped)
+ (swap! q.registry/*queues* dissoc uncapped))))))
diff --git a/test/metabase/mq/queue/concurrency_test.clj b/test/metabase/mq/queue/concurrency_test.clj
new file mode 100644
index 00000000000..dae5f053e83
--- /dev/null
+++ b/test/metabase/mq/queue/concurrency_test.clj
@@ -0,0 +1,255 @@
+(ns metabase.mq.queue.concurrency-test
+ (:require
+ [clojure.test :refer :all]
+ [metabase.mq.listener :as listener]
+ [metabase.mq.queue.concurrency :as q.concurrency]
+ [metabase.mq.queue.registry :as q.registry])
+ (:import
+ (java.util.concurrent CountDownLatch ExecutorService Executors TimeUnit)))
+
+(set! *warn-on-reflection* true)
+
+(defn- with-fresh-state [f]
+ (binding [q.registry/*queues* (atom {})
+ q.concurrency/*in-flight* (atom {})
+ listener/*listeners* (atom {})]
+ (f)))
+
+(use-fixtures :each with-fresh-state)
+
+(defn- declare-queue! [queue-name cap]
+ (q.registry/register-queue! queue-name (cond-> {:transactional :try}
+ cap (assoc :max-concurrent-batches cap))))
+
+(defn- wait-for! [pred]
+ (let [deadline (+ (System/currentTimeMillis) 5000)]
+ (loop []
+ (or (pred)
+ (when (< (System/currentTimeMillis) deadline)
+ (Thread/sleep 5)
+ (recur))))))
+
+;;; ------------------------------------------- with-slot -------------------------------------------
+
+(deftest slots-are-counted-and-released-test
+ (declare-queue! :queue/capped 2)
+ (q.concurrency/with-slot :queue/capped
+ (is (= 1 (q.concurrency/in-flight :queue/capped)))
+ (q.concurrency/with-slot :queue/capped
+ (is (= 2 (q.concurrency/in-flight :queue/capped)))))
+ (is (zero? (q.concurrency/in-flight :queue/capped))
+ "every slot is released as its with-slot exits"))
+
+(deftest over-the-cap-the-body-still-runs-test
+ (testing "the cap is a throttle on what we take, not a veto on work already in hand: a delivery that
+ arrives while we're at the cap is still delivered, and simply counts us over it"
+ ;; Bouncing it instead would mean a re-queue path with its own backoff, its own metric, and a
+ ;; hot-loop to avoid — all to shave an overshoot that is small, bounded and self-correcting.
+ (declare-queue! :queue/capped 1)
+ (q.concurrency/with-slot :queue/capped
+ (is (true? (q.concurrency/at-capacity? :queue/capped)))
+ (is (= ::ran (q.concurrency/with-slot :queue/capped ::ran))
+ "the body ran even though the node was already at its cap")
+ (is (= 1 (q.concurrency/in-flight :queue/capped))
+ "and the over-cap slot was released again on the way out"))
+ (is (zero? (q.concurrency/in-flight :queue/capped)))))
+
+(deftest at-capacity?-keeps-saying-yes-while-over-the-cap-test
+ (testing "a node that raced its way over the cap must keep refusing new work until it drains back
+ under — hence >= rather than ="
+ (declare-queue! :queue/capped 1)
+ (q.concurrency/with-slot :queue/capped
+ (q.concurrency/with-slot :queue/capped
+ (is (= 2 (q.concurrency/in-flight :queue/capped)) "over the cap of 1")
+ (is (true? (q.concurrency/at-capacity? :queue/capped)))
+ (is (= 0 (q.concurrency/free-slots :queue/capped))
+ "free-slots floors at zero — it never reports negative headroom")))))
+
+(deftest with-slot-returns-the-body-value-test
+ (declare-queue! :queue/capped 1)
+ (is (= ::delivered (q.concurrency/with-slot :queue/capped ::delivered)))
+ (is (nil? (q.concurrency/with-slot :queue/capped nil))))
+
+(deftest with-slot-releases-when-the-body-throws-test
+ (testing "a body that throws still releases its slot — a leak here would silently shrink the queue's
+ capacity for the life of the process, with no error anywhere to explain why it stopped
+ delivering"
+ (declare-queue! :queue/capped 1)
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo #"boom"
+ (q.concurrency/with-slot :queue/capped
+ (throw (ex-info "boom" {})))))
+ (is (zero? (q.concurrency/in-flight :queue/capped)))
+ (is (false? (q.concurrency/at-capacity? :queue/capped))
+ "and the queue can still deliver")))
+
+(deftest at-capacity?-and-working?-track-slots-test
+ (declare-queue! :queue/capped 1)
+ (is (false? (q.concurrency/at-capacity? :queue/capped)))
+ (is (false? (q.concurrency/working? :queue/capped)))
+ (q.concurrency/with-slot :queue/capped
+ (is (true? (q.concurrency/at-capacity? :queue/capped)))
+ (is (true? (q.concurrency/working? :queue/capped))))
+ (is (false? (q.concurrency/at-capacity? :queue/capped)))
+ (is (false? (q.concurrency/working? :queue/capped))))
+
+(deftest free-slots-counts-down-test
+ (declare-queue! :queue/capped 3)
+ (is (= 3 (q.concurrency/free-slots :queue/capped)))
+ (q.concurrency/with-slot :queue/capped
+ (is (= 2 (q.concurrency/free-slots :queue/capped)))
+ (q.concurrency/with-slot :queue/capped
+ (q.concurrency/with-slot :queue/capped
+ (is (= 0 (q.concurrency/free-slots :queue/capped)))))))
+
+(deftest uncapped-queue-is-unbounded-test
+ ;; There is exactly one meaning of "declared no cap", and it is the same on every backend:
+ ;; unbounded. No caller supplies its own default — that is how the backends used to disagree
+ ;; (Quartz unbounded, the poll driver one-at-a-time) about what the same config meant.
+ (declare-queue! :queue/uncapped nil)
+ (testing "an uncapped queue always grants a slot and is never at capacity"
+ (letfn [(nest [depth]
+ (if (zero? depth)
+ (do (is (= 25 (q.concurrency/in-flight :queue/uncapped)))
+ (is (false? (q.concurrency/at-capacity? :queue/uncapped)))
+ (is (true? (q.concurrency/working? :queue/uncapped))))
+ (q.concurrency/with-slot :queue/uncapped
+ (nest (dec depth)))))]
+ (nest 25)))
+ (testing "and reports no fetch limit — nil means 'take as many as you have', not 'take one'"
+ (is (nil? (q.concurrency/free-slots :queue/uncapped)))))
+
+(deftest exclusive-is-not-this-layers-concern-test
+ (testing "an :exclusive queue is unbounded as far as this ns is concerned — mutual exclusion is the
+ *backend's* job (Quartz's @DisallowConcurrentExecution; the memory backend's fetch),
+ because the backend is what owns delivery. Modelling it here as a cap of one would look
+ tidy and quietly downgrade a cluster-wide guarantee to a per-node one."
+ (q.registry/register-queue! :queue/excl {:transactional :try :exclusive true})
+ (is (nil? (q.concurrency/free-slots :queue/excl)))
+ (q.concurrency/with-slot :queue/excl
+ (is (false? (q.concurrency/at-capacity? :queue/excl))))))
+
+(deftest slots-are-per-channel-test
+ (declare-queue! :queue/a 1)
+ (declare-queue! :queue/b 1)
+ (q.concurrency/with-slot :queue/a
+ (is (true? (q.concurrency/at-capacity? :queue/a)))
+ (is (false? (q.concurrency/at-capacity? :queue/b))
+ "queue/b has its own budget")
+ (is (zero? (q.concurrency/in-flight :queue/b)))))
+
+(deftest cap-fn-is-resolved-on-every-check-test
+ (let [cap (atom 1)]
+ (declare-queue! :queue/dynamic (fn [] @cap))
+ (q.concurrency/with-slot :queue/dynamic
+ (is (true? (q.concurrency/at-capacity? :queue/dynamic))
+ "at capacity while the setting says 1")
+ (reset! cap 2)
+ (is (false? (q.concurrency/at-capacity? :queue/dynamic))
+ "raising the setting immediately opens headroom — the cap is read per check, not frozen at
+ registration")
+ (is (= 1 (q.concurrency/free-slots :queue/dynamic))))))
+
+(deftest concurrent-slots-are-accounted-exactly-test
+ (testing "the in-flight count is exact under contention — it is what the throttle reads, so a lost
+ update would let a queue quietly run over its cap forever"
+ (declare-queue! :queue/capped 5)
+ ;; `future` conveys the dynamic bindings from the fixture, so every thread races on the same
+ ;; in-flight atom. All 20 start together on the latch to make the swap actually contend.
+ (let [start (CountDownLatch. 1)
+ release (CountDownLatch. 1)
+ entered (atom 0)
+ futures (doall
+ (for [_ (range 20)]
+ (future
+ (.await start)
+ (q.concurrency/with-slot :queue/capped
+ (swap! entered inc)
+ (.await release) ; hold the slot so all 20 are counted at once
+ ::ran))))]
+ (.countDown start)
+ (is (true? (wait-for! #(= 20 @entered)))
+ "every batch runs — the cap throttles fetching, it does not refuse work in hand")
+ (is (= 20 (q.concurrency/in-flight :queue/capped))
+ "and all 20 are counted, with no lost updates")
+ (is (true? (q.concurrency/at-capacity? :queue/capped))
+ "so the node correctly reports it should take no more")
+ (.countDown release)
+ (run! deref futures)
+ (is (zero? (q.concurrency/in-flight :queue/capped))
+ "every slot is released once its body finishes — no leak, no negative count"))))
+
+;;; ---------------------------------------- submit-with-slot! ----------------------------------------
+
+(deftest submit-with-slot-counts-from-hand-off-and-releases-on-the-worker-thread-test
+ (declare-queue! :queue/capped 1)
+ (let [pool (Executors/newCachedThreadPool)
+ release (CountDownLatch. 1)
+ entered (CountDownLatch. 1)
+ woken (atom 0)]
+ (try
+ (q.concurrency/submit-with-slot!
+ :queue/capped pool
+ (fn [] (.countDown entered) (.await release))
+ #(swap! woken inc))
+ (is (= 1 (q.concurrency/in-flight :queue/capped))
+ "the batch counts against the node the moment it is handed off — not when the worker thread
+ gets around to it, which would let the driver fetch more work in the gap")
+ (is (.await entered 5000 TimeUnit/MILLISECONDS))
+ (is (true? (q.concurrency/at-capacity? :queue/capped))
+ "the slot is held for as long as the task runs, over on the other thread")
+ (.countDown release)
+ (is (true? (wait-for! #(zero? (q.concurrency/in-flight :queue/capped))))
+ "the slot is released when the task finishes")
+ (is (true? (wait-for! #(= 1 @woken)))
+ "and after-release runs — that's the 'a slot just freed up, go look for work' signal")
+ (finally
+ (.shutdownNow ^ExecutorService pool)))))
+
+(deftest submit-with-slot-releases-when-the-task-throws-test
+ (declare-queue! :queue/capped 1)
+ (let [pool (Executors/newCachedThreadPool)]
+ (try
+ (q.concurrency/submit-with-slot!
+ :queue/capped pool
+ (fn [] (throw (ex-info "boom" {})))
+ (fn []))
+ (is (true? (wait-for! #(zero? (q.concurrency/in-flight :queue/capped))))
+ "a task that throws still releases its slot")
+ (finally
+ (.shutdownNow ^ExecutorService pool)))))
+
+(deftest submit-with-slot-releases-when-the-submit-itself-fails-test
+ (testing "if the pool is gone the task will never run, and so will never release — the slot must be
+ given back here or the queue sits at capacity forever and silently stops delivering"
+ (declare-queue! :queue/capped 1)
+ (let [pool (doto (Executors/newCachedThreadPool) (.shutdownNow))]
+ (is (thrown? Exception
+ (q.concurrency/submit-with-slot! :queue/capped pool (fn []) (fn []))))
+ (is (zero? (q.concurrency/in-flight :queue/capped)))
+ (is (false? (q.concurrency/at-capacity? :queue/capped))))))
+
+;;; ---------------------------------------- takeable-queues ----------------------------------------
+
+(deftest takeable-queues-is-the-shared-definition-test
+ (testing "the queues this node will take work for: has a listener, not at capacity.
+
+ Both backends derive their own shape from this one fn — Quartz the capability set it
+ splices into its acquire query, the poll driver the free-slot map it hands fetch! — so that
+ they cannot drift apart on the answer."
+ (declare-queue! :queue/capped 1)
+ (declare-queue! :queue/uncapped nil)
+ (declare-queue! :queue/no-listener 1)
+ (listener/batch-listen! :queue/capped (fn [_]))
+ (listener/batch-listen! :queue/uncapped (fn [_]))
+ (testing "a declared queue with no listener on this node is never takeable"
+ (is (= #{:queue/capped :queue/uncapped} (set (q.concurrency/takeable-queues)))
+ ":queue/no-listener is declared, but nothing here handles it"))
+ (testing "a queue at its cap drops out, and comes back when the slot frees"
+ (q.concurrency/with-slot :queue/capped
+ (is (= #{:queue/uncapped} (set (q.concurrency/takeable-queues)))))
+ (is (= #{:queue/capped :queue/uncapped} (set (q.concurrency/takeable-queues)))))
+ (testing "an uncapped queue stays takeable however many batches are in flight"
+ (q.concurrency/with-slot :queue/uncapped
+ (q.concurrency/with-slot :queue/uncapped
+ (is (contains? (set (q.concurrency/takeable-queues)) :queue/uncapped)))))))
diff --git a/test/metabase/mq/queue/core_test.clj b/test/metabase/mq/queue/core_test.clj
index 650fc34da98..8aa45177678 100644
--- a/test/metabase/mq/queue/core_test.clj
+++ b/test/metabase/mq/queue/core_test.clj
@@ -7,6 +7,7 @@
[metabase.mq.payload :as payload]
[metabase.mq.publish :as mq.publish]
[metabase.mq.publish-buffer :as publish-buffer]
+ [metabase.mq.queue.concurrency :as q.concurrency]
[metabase.mq.queue.polling :as q.polling]
[metabase.mq.queue.registry :as q.registry]
[metabase.mq.test-util :as mq.tu]
@@ -104,7 +105,43 @@
(mq/put q "msg1")
(mq/put q "msg2"))
(mq.tu/flush! test-mq)
- (is (= ["msg1" "msg2"] @heard-messages))))))
+ ;; delivery, not order — see `every-message-is-delivered-test`. Exclusivity buys mutual
+ ;; exclusion, not FIFO.
+ (is (= #{"msg1" "msg2"} (set @heard-messages)))))))
+
+(deftest exclusive-queue-is-serialized-by-the-memory-backend-test
+ (testing "an :exclusive queue runs one batch at a time on the memory backend too.
+
+ Exclusivity is a *backend* guarantee, and each backend implements its own: Quartz uses
+ @DisallowConcurrentExecution on the job class; the memory backend never fetches a second
+ batch for a queue that already has one in flight. The shared poll driver knows nothing
+ about the flag. Before the memory backend implemented it, an :exclusive queue serialized on
+ Quartz and meant nothing at all here — it only *looked* right because the driver happened
+ to deliver one batch per channel at a time."
+ (q.polling/start-worker-pool!)
+ (mq.tu/with-test-mq [_test-mq]
+ (let [queue-name :queue/exclusive-serialization-test
+ in-flight (atom 0)
+ peak (atom 0)
+ arrived (atom 0)
+ release (CountDownLatch. 1)]
+ (q.registry/register-queue! queue-name {:transactional :try :exclusive true :max-batch-messages 1})
+ (mq.tu/listen! queue-name (fn [_]
+ (swap! peak max (swap! in-flight inc))
+ (swap! arrived inc)
+ (.await release)
+ (swap! in-flight dec)))
+ (mq/with-queue queue-name [q]
+ (mq/put q "a")
+ (mq/put q "b")
+ (mq/put q "c"))
+ (is (true? (mq.tu/wait-for! #(= 1 @arrived) 5000)))
+ (is (nil? (mq.tu/wait-for! #(> @arrived 1) 500))
+ "no second batch is fetched while the first is in flight")
+ (.countDown release)
+ (is (true? (mq.tu/wait-for! #(= 3 @arrived) 5000)) "all three still get delivered")
+ (is (= 1 @peak) "never more than one batch in the listener at a time")
+ (mq/unlisten! queue-name)))))
(deftest batch-listen-exclusive-test
(let [heard-batches (atom [])]
@@ -253,7 +290,17 @@
(mq.tu/flush! test-mq)
(is (= ["a" "b" "c"] @heard) "all delivered after flush"))))))
-(deftest fifo-ordering-test
+(deftest every-message-is-delivered-test
+ ;; Deliberately NOT an ordering test. The MQ promises at-least-once delivery, not FIFO: Quartz fires
+ ;; a queue's batches on its worker threads and explicitly does not fire them in submission order, and
+ ;; an uncapped queue runs them concurrently. The poll driver used to deliver one batch at a time,
+ ;; which made ordering *look* guaranteed if you only ever tested against the memory backend — a
+ ;; guarantee no other backend has ever offered. Asserting it here would pin a memory-only behavior
+ ;; and re-introduce exactly the cross-backend divergence `:max-concurrent-batches` exists to remove.
+ ;;
+ ;; A queue that genuinely needs serialized delivery must say so: `:max-concurrent-batches 1` (per
+ ;; node) or `:exclusive true` (cluster-wide). Even then, order is not promised — only mutual
+ ;; exclusion.
(let [received (atom [])]
(mq.tu/with-test-mq [test-mq]
{:queue/test (fn [message] (swap! received conj message))}
@@ -261,40 +308,66 @@
(mq/with-queue :queue/test [q]
(mq/put q i)))
(mq.tu/flush! test-mq)
- (testing "All messages are delivered in order"
- (is (= (range 10) @received))))))
+ (testing "every published message is delivered exactly once (in some order)"
+ (is (= (set (range 10)) (set @received)))
+ (is (= 10 (count @received)))))))
-(deftest exclusive-queue-single-active-handler-test
- "Verifies that submit-delivery! returns false for a channel that already has an
- active handler, enforcing at-most-one concurrent delivery per channel."
- (q.polling/start-worker-pool!)
- (mq.tu/with-test-mq [_test-mq]
- (let [queue-name :queue/exclusive-concurrency-test
- started (CountDownLatch. 1) ; counted down once the worker thread enters the listener
- release (CountDownLatch. 1)] ; blocks the listener until the test releases it
- (mq.tu/listen! queue-name
- (fn [_] (.countDown started) (.await release)))
- (testing "First submission succeeds and marks the channel as busy"
- (is (true? (q.polling/submit-delivery! queue-name (payload/encode ["msg1"]) nil nil)))
+(deftest submit-delivery-counts-the-batch-against-the-node-test
+ (testing "a submitted batch counts against the node's :max-concurrent-batches from hand-off until it
+ finishes, which is what makes the queue report itself at capacity and stop fetching more"
+ (q.polling/start-worker-pool!)
+ (mq.tu/with-test-mq [_test-mq]
+ (let [queue-name :queue/capped-concurrency-test
+ started (CountDownLatch. 1) ; counted down once the worker thread enters the listener
+ release (CountDownLatch. 1)] ; blocks the listener until the test releases it
+ (q.registry/register-queue! queue-name {:transactional :try :max-concurrent-batches 1})
+ (mq.tu/listen! queue-name
+ (fn [_] (.countDown started) (.await release)))
+ (q.polling/submit-delivery! queue-name (payload/encode ["msg1"]) nil nil)
(is (.await started 5 TimeUnit/SECONDS) "worker thread started and entered the listener")
- (is (true? (q.polling/channel-busy? queue-name))))
- (testing "Second submission returns false while channel is busy"
- (is (false? (q.polling/submit-delivery! queue-name (payload/encode ["msg2"]) nil nil))))
- (.countDown release) ; Release the listener
- (testing "Channel is no longer busy after delivery completes"
- (is (true? (mq.tu/wait-for! #(not (q.polling/channel-busy? queue-name)) 5000))
- "active-handlers clears once delivery completes"))
- (mq/unlisten! queue-name))))
+ (is (true? (q.concurrency/working? queue-name)))
+ (is (true? (q.concurrency/at-capacity? queue-name))
+ "at its cap of 1 — so the poll loop will fetch nothing more for this queue")
+ (.countDown release) ; Release the listener
+ (is (true? (mq.tu/wait-for! #(not (q.concurrency/working? queue-name)) 5000))
+ "the in-flight count returns to zero once delivery completes")
+ (is (false? (q.concurrency/at-capacity? queue-name))
+ "and the queue can be fetched for again")
+ (mq/unlisten! queue-name)))))
+
+(deftest uncapped-queue-submits-without-limit-test
+ (testing "a queue that declares no :max-concurrent-batches is unbounded — the same as on Quartz.
+ The poll driver used to serialize it one-batch-at-a-time, which made the *same* queue
+ config mean different things depending on which backend was running."
+ (q.polling/start-worker-pool!)
+ (mq.tu/with-test-mq [_test-mq]
+ (let [queue-name :queue/uncapped-concurrency-test
+ in-listener (CountDownLatch. 3)
+ release (CountDownLatch. 1)]
+ (mq.tu/listen! queue-name (fn [_] (.countDown in-listener) (.await release)))
+ (dotimes [i 3]
+ (q.polling/submit-delivery! queue-name (payload/encode [(str "msg" i)]) nil nil))
+ (is (.await in-listener 5 TimeUnit/SECONDS)
+ "all three deliveries are in the listener at the same time")
+ (is (false? (q.concurrency/at-capacity? queue-name))
+ "and it is never at capacity, however many are working — that is what uncapped means")
+ (.countDown release)
+ (is (true? (mq.tu/wait-for! #(not (q.concurrency/working? queue-name)) 5000)))
+ (mq/unlisten! queue-name)))))
(deftest submit-delivery-frees-slot-when-pool-unavailable-test
- (testing "if submitting to the worker pool fails (e.g. pool not running), the channel slot is
- released rather than left permanently busy — otherwise the queue would silently stop
- delivering forever"
- (with-redefs [q.polling/worker-pool (atom nil) ; deref/.submit NPEs
- q.polling/active-handlers (atom {})]
- (let [queue-name :queue/submit-failure-test]
- (is (thrown? Exception
- (q.polling/submit-delivery! queue-name (payload/encode ["x"]) nil nil))
- "the submit failure propagates to the caller")
- (is (false? (q.polling/channel-busy? queue-name))
- "the channel is not left marked busy after the failed submit")))))
+ (testing "if submitting to the worker pool fails (e.g. pool not running), the claimed slot is
+ released rather than held forever — otherwise a capped queue would sit permanently at
+ capacity and silently stop delivering"
+ (binding [q.concurrency/*in-flight* (atom {})
+ q.registry/*queues* (atom {})]
+ (with-redefs [q.polling/worker-pool (atom nil)] ; deref/.submit NPEs
+ (let [queue-name :queue/submit-failure-test]
+ (q.registry/register-queue! queue-name {:transactional :try :max-concurrent-batches 1})
+ (is (thrown? Exception
+ (q.polling/submit-delivery! queue-name (payload/encode ["x"]) nil nil))
+ "the submit failure propagates to the caller")
+ (is (false? (q.concurrency/working? queue-name))
+ "the slot claimed for the failed submit was given back")
+ (is (false? (q.concurrency/at-capacity? queue-name))
+ "so the queue can still deliver"))))))
diff --git a/test/metabase/mq/queue/memory_test.clj b/test/metabase/mq/queue/memory_test.clj
index fce1772fcd0..4ada4892334 100644
--- a/test/metabase/mq/queue/memory_test.clj
+++ b/test/metabase/mq/queue/memory_test.clj
@@ -37,6 +37,19 @@
(is (= 1 (depth-for layer "has-listener")) "messages with a listener are kept")
(is (nil? (depth-for layer "no-listener")) "orphaned messages are dropped")))))
+(deftest clear-stale-claims-counts-as-a-failure-test
+ (testing "recovering a stale claim increments the message's failure count, so a delivery that keeps
+ timing out (never acks) eventually exhausts its retry budget instead of retrying forever"
+ (let [layer (memory/make-layer)]
+ (memory/publish! layer :queue/slow "p")
+ (memory/take-pending! layer :queue/slow 1)
+ (is (= 0 (memory/message-failures layer 1)) "claiming a message doesn't fail it")
+ ;; a negative timeout makes every in-flight claim stale
+ (is (= 1 (memory/clear-stale-claims! layer -1)) "the stale claim is recovered")
+ (is (= 1 (memory/message-failures layer 1)) "and counts as one failure")
+ (testing "the message is re-claimable after recovery"
+ (is (= 1 (count (memory/take-pending! layer :queue/slow 1))))))))
+
(deftest publish-test
(mq.tu/with-test-mq [ctx]
(let [queue-name (keyword "queue" (str "publish-test-" (gensym)))
diff --git a/test/metabase/mq/queue/on_error_test.clj b/test/metabase/mq/queue/on_error_test.clj
new file mode 100644
index 00000000000..1f435a4e95c
--- /dev/null
+++ b/test/metabase/mq/queue/on_error_test.clj
@@ -0,0 +1,281 @@
+(ns metabase.mq.queue.on-error-test
+ "Tests for a queue's `:on-error` handler — the hook that fires when a batch has exhausted its
+ retries and would otherwise be silently dropped.
+
+ The handler is invoked from the shared retry-vs-drop policy
+ ([[metabase.mq.queue.impl/handle-batch-failure-policy!]]), which every backend routes through, so a
+ queue gets the same terminal-failure semantics no matter which backend is running. Backend parity
+ is asserted here for the poll backend and in `metabase.mq.queue.quartz-test` for the push backend.
+
+ Listeners here are registered with [[metabase.mq.listener/batch-listen!]] rather than `def-listener!`
+ (which only registers a `defmethod` that `register-listeners!` realizes at startup — too late inside
+ `with-test-mq`) or `mq.tu/listen!` (which isolates per-message errors, swallowing the very failures
+ under test)."
+ (:require
+ [clojure.test :refer :all]
+ [metabase.analytics-interface.core :as analytics]
+ [metabase.mq.core :as mq]
+ [metabase.mq.listener :as listener]
+ [metabase.mq.queue.registry :as q.registry]
+ [metabase.mq.test-util :as mq.tu]
+ [metabase.test :as mt]))
+
+(set! *warn-on-reflection* true)
+
+(deftest on-error-fires-once-with-messages-and-cause-test
+ (testing "a batch that exhausts its retries invokes :on-error with the messages and the cause"
+ (mt/with-temporary-setting-values [queue-max-retries 2]
+ (let [dropped (atom [])
+ attempts (atom 0)
+ boom (ex-info "always boom" {:reason :boom})]
+ (mq.tu/with-test-mq [ctx]
+ (q.registry/register-queue!
+ :queue/on-error-basic
+ {:transactional :try
+ :on-error (fn [info] (swap! dropped conj info))})
+ (listener/batch-listen!
+ :queue/on-error-basic
+ (fn [_messages]
+ (swap! attempts inc)
+ (throw boom)))
+ (mq/with-queue :queue/on-error-basic [q]
+ (mq/put q {:id 1})
+ (mq/put q {:id 2}))
+ (is (mq.tu/eventually! ctx #(seq @dropped) 5000)
+ ":on-error fires once the batch exhausts its retries")
+ (is (= 1 (count @dropped))
+ ":on-error fires once *per batch*, on the terminal drop — not on every failed attempt.
+ (Once per batch is not once per payload: see
+ duplicate-delivery-fires-on-error-for-an-already-succeeded-payload-test.)")
+ (let [info (first @dropped)]
+ (is (= #{:channel :messages :error :attempts} (set (keys info))))
+ (is (= :queue/on-error-basic (:channel info)))
+ (is (= [{:id 1} {:id 2}] (:messages info))
+ "the handler receives the decoded messages of the dropped batch")
+ (is (= boom (:error info))
+ "the handler receives the exception that failed the last attempt")
+ (is (= 2 (:attempts info))
+ "the handler is told how many attempts were made")))))))
+
+(deftest on-error-not-called-while-retries-remain-test
+ (testing ":on-error does not fire for a failure that will be retried and then succeeds"
+ (mt/with-temporary-setting-values [queue-max-retries 3]
+ (let [dropped (atom [])
+ attempts (atom 0)
+ handled (atom nil)]
+ (mq.tu/with-test-mq [ctx]
+ (q.registry/register-queue!
+ :queue/on-error-recovers
+ {:transactional :try
+ :on-error (fn [info] (swap! dropped conj info))})
+ (listener/batch-listen!
+ :queue/on-error-recovers
+ (fn [messages]
+ ;; fail the first attempt, succeed on the retry
+ (when (= 1 (swap! attempts inc))
+ (throw (ex-info "transient" {})))
+ (reset! handled messages)))
+ (mq/with-queue :queue/on-error-recovers [q]
+ (mq/put q "eventually-ok"))
+ (is (mq.tu/eventually! ctx #(some? @handled) 5000)
+ "the batch is retried and succeeds")
+ (is (= ["eventually-ok"] @handled))
+ (is (= [] @dropped)
+ ":on-error must not fire for a transient failure that later succeeds"))))))
+
+(deftest on-error-not-called-on-success-test
+ (testing ":on-error is never invoked for a batch that is delivered cleanly"
+ (let [dropped (atom [])
+ handled (atom nil)]
+ (mq.tu/with-test-mq [ctx]
+ (q.registry/register-queue!
+ :queue/on-error-happy
+ {:transactional :try
+ :on-error (fn [info] (swap! dropped conj info))})
+ (listener/batch-listen!
+ :queue/on-error-happy
+ (fn [messages] (reset! handled messages)))
+ (mq/with-queue :queue/on-error-happy [q]
+ (mq/put q "fine"))
+ (mq.tu/flush! ctx)
+ (is (= ["fine"] @handled))
+ (is (= [] @dropped))))))
+
+(deftest on-error-handler-that-throws-still-drops-the-batch-test
+ (testing "a throwing :on-error handler is contained — the batch is still dropped, not retried forever"
+ (mt/with-temporary-setting-values [queue-max-retries 2]
+ (let [called (atom 0)
+ attempts (atom 0)]
+ (mq.tu/with-test-mq [ctx]
+ (q.registry/register-queue!
+ :queue/on-error-throws
+ {:transactional :try
+ :on-error (fn [_info]
+ (swap! called inc)
+ (throw (ex-info "handler itself blew up" {})))})
+ (listener/batch-listen!
+ :queue/on-error-throws
+ (fn [_messages]
+ (swap! attempts inc)
+ (throw (ex-info "always boom" {}))))
+ (mq/with-queue :queue/on-error-throws [q]
+ (mq/put q "doomed"))
+ (is (mq.tu/eventually! ctx #(pos? @called) 5000)
+ ":on-error is invoked")
+ ;; If the throwing handler escaped the drop path, the batch would be re-delivered and
+ ;; attempts would keep climbing past the retry budget.
+ (Thread/sleep 500)
+ (is (= 2 @attempts)
+ "the batch is still dropped after exactly queue-max-retries attempts")
+ (is (= 1 @called)
+ ":on-error is not re-invoked"))))))
+
+(deftest listener-error-still-goes-through-the-retry-policy-test
+ (testing "a listener that throws an `Error` rather than an `Exception` must still go through the
+ retry/drop policy.
+
+ `Error` is not exotic here: a failed `assert` or `{:pre ...}` throws `AssertionError`, and
+ a namespace that fails to load lazily on a worker thread throws
+ `ExceptionInInitializerError`. If the delivery core only catches `Exception`, an `Error`
+ escapes past the retry budget entirely and into the backend's hand-off failure path —
+ which is built to *refire the same job with the same attempt number*, on the assumption
+ that the message was never delivered. The batch would then re-run forever on one node,
+ burning no retry budget and never reaching `:on-error`."
+ (mt/with-temporary-setting-values [queue-max-retries 2]
+ (let [dropped (atom [])
+ attempts (atom 0)]
+ (mq.tu/with-test-mq [ctx]
+ (q.registry/register-queue!
+ :queue/on-error-throwable
+ {:transactional :try
+ :on-error (fn [info] (swap! dropped conj info))})
+ (listener/batch-listen!
+ :queue/on-error-throwable
+ (fn [_messages]
+ (swap! attempts inc)
+ (throw (AssertionError. "an assert blew up inside the listener"))))
+ (mq/with-queue :queue/on-error-throwable [q]
+ (mq/put q "doomed"))
+ (is (mq.tu/eventually! ctx #(seq @dropped) 5000)
+ "an Error exhausts the retry budget and reaches :on-error, like any other failure")
+ (Thread/sleep 500)
+ (is (= 2 @attempts)
+ "and is retried exactly queue-max-retries times — not refired without a budget")
+ (is (= 1 (count @dropped))
+ ":on-error fires once, on the terminal drop")
+ (is (instance? AssertionError (:error (first @dropped)))
+ "the handler receives the Error that failed the batch"))))))
+
+(deftest drop-path-bookkeeping-failure-does-not-redeliver-test
+ (testing "a failure in the drop path's *bookkeeping* — the log line and the `batches-dropped`
+ counter — must not redeliver the batch or re-run its `:on-error`.
+
+ `run-on-error!` is carefully built so it can never throw, precisely so a broken handler
+ cannot escape into the backend's retry machinery. But the logging and metrics calls
+ sitting next to it were unguarded, and a backend reads any exception out of this policy
+ as \"the message was not handed off\" — so it refires the batch with its attempt counter
+ untouched. The listener runs again, fails again, is over budget again, and `:on-error`
+ fires a second time for a batch that was already dropped."
+ (mt/with-temporary-setting-values [queue-max-retries 2]
+ (let [dropped (atom [])
+ attempts (atom 0)
+ orig analytics/inc!]
+ ;; a metrics backend that is down exactly when we try to record the drop
+ (with-redefs [analytics/inc! (fn [metric & args]
+ (if (= metric :metabase-mq/batches-dropped)
+ (throw (ex-info "metrics backend is down" {}))
+ (apply orig metric args)))]
+ (mq.tu/with-test-mq [ctx]
+ (q.registry/register-queue!
+ :queue/on-error-drop-bookkeeping
+ {:transactional :try
+ :on-error (fn [info] (swap! dropped conj info))})
+ (listener/batch-listen!
+ :queue/on-error-drop-bookkeeping
+ (fn [_messages]
+ (swap! attempts inc)
+ (throw (ex-info "always boom" {}))))
+ (mq/with-queue :queue/on-error-drop-bookkeeping [q]
+ (mq/put q "doomed"))
+ (is (mq.tu/eventually! ctx #(seq @dropped) 5000)
+ ":on-error still fires when the drop metric blows up")
+ (Thread/sleep 500)
+ (is (= 1 (count @dropped))
+ ":on-error fires once — a failed drop metric must not resurrect the batch")
+ (is (= 2 @attempts)
+ "and the batch is not redelivered past its retry budget")))))))
+
+(deftest queue-without-on-error-still-drops-test
+ (testing "a queue that declares no :on-error keeps the old behavior: exhaust retries, then drop"
+ (mt/with-temporary-setting-values [queue-max-retries 2]
+ (let [attempts (atom 0)]
+ (mq.tu/with-test-mq [ctx]
+ (q.registry/register-queue! :queue/on-error-absent {:transactional :try})
+ (listener/batch-listen!
+ :queue/on-error-absent
+ (fn [_messages]
+ (swap! attempts inc)
+ (throw (ex-info "always boom" {}))))
+ (mq/with-queue :queue/on-error-absent [q]
+ (mq/put q "doomed"))
+ (is (mq.tu/eventually! ctx #(= 2 @attempts) 5000))
+ (Thread/sleep 500)
+ (is (= 2 @attempts)
+ "still dropped after queue-max-retries, with no handler configured"))))))
+
+(deftest duplicate-delivery-fires-on-error-for-an-already-succeeded-payload-test
+ (testing "PINS A KNOWN LIMITATION — this is the behavior we have, not the behavior we want.
+
+ A duplicate delivery whose copy *fails* will run out its own retry budget and fire
+ `:on-error` for a payload that another copy already handled successfully.
+
+ The reason is that a message has no identity. A retry is a brand-new trigger with a fresh
+ id, and the attempt counter rides along inside it, so two deliveries of one payload are
+ two independent batches with two independent retry budgets and nothing joining them. A
+ success on one node cannot cancel, or even be observed by, a copy failing on another —
+ which is precisely what happens when Quartz's cluster failover re-fires a batch onto a
+ second node because the first missed a heartbeat while still running it.
+
+ So an `:on-error` handler must be idempotent *and* tolerate firing for work that in fact
+ succeeded. This is why the explorations handlers guard their writes on the row still
+ being `pending` rather than blindly stamping it `error`.
+
+ Closing this needs message identity plus a record of what has been delivered. When that
+ lands, this test should flip to asserting `(= [] @dropped)`."
+ (mt/with-temporary-setting-values [queue-max-retries 2]
+ (let [dropped (atom [])
+ succeeded (atom 0)
+ calls (atom 0)]
+ (mq.tu/with-test-mq [ctx {:duplicate-delivery? true}]
+ (q.registry/register-queue!
+ :queue/on-error-duplicate
+ {:transactional :try
+ :on-error (fn [info] (swap! dropped conj info))})
+ (listener/batch-listen!
+ :queue/on-error-duplicate
+ (fn [_messages]
+ ;; the first copy of the payload lands cleanly; the duplicate is the one that fails
+ (if (= 1 (swap! calls inc))
+ (swap! succeeded inc)
+ (throw (ex-info "this delivery failed, but the work is already done" {})))))
+ (mq/with-queue :queue/on-error-duplicate [q]
+ (mq/put q "already-done"))
+ (is (mq.tu/eventually! ctx #(seq @dropped) 5000)
+ "the failing duplicate exhausts its retries")
+ (is (= 1 @succeeded)
+ "the payload was handled successfully — the work is done")
+ (is (= 1 (count @dropped))
+ ":on-error still fires for it, because the failing copy has its own retry budget and no
+ way to learn that the work already succeeded"))))))
+
+(deftest on-error-must-be-a-fn-test
+ (testing "the queue config schema accepts an :on-error fn and rejects a non-fn"
+ (binding [q.registry/*queues* (atom {})]
+ (is (nil? (q.registry/register-queue! :queue/on-error-schema-ok
+ {:transactional :try :on-error (fn [_] nil)}))
+ "a fn is accepted")
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo #"Invalid config for queue"
+ (q.registry/register-queue! :queue/on-error-schema-bad
+ {:transactional :try :on-error "not-a-fn"}))
+ "a non-fn :on-error is rejected at declaration time"))))
diff --git a/test/metabase/mq/queue/polling_test.clj b/test/metabase/mq/queue/polling_test.clj
index 8ac0912215c..e717b36dbf7 100644
--- a/test/metabase/mq/queue/polling_test.clj
+++ b/test/metabase/mq/queue/polling_test.clj
@@ -1,9 +1,15 @@
(ns metabase.mq.queue.polling-test
(:require
[clojure.test :refer :all]
+ [metabase.mq.core :as mq]
[metabase.mq.queue.backend :as q.backend]
+ [metabase.mq.queue.concurrency :as q.concurrency]
[metabase.mq.queue.polling :as q.polling]
- [metabase.test :as mt]))
+ [metabase.mq.queue.registry :as q.registry]
+ [metabase.mq.test-util :as mq.tu]
+ [metabase.test :as mt])
+ (:import
+ (java.util.concurrent CountDownLatch)))
(set! *warn-on-reflection* true)
@@ -13,7 +19,7 @@
(let [backend (reify q.backend/QueueBackend
(backend-id [_] :queue.backend/test)
(publish! [_ _queue _payload] nil)
- (fetch! [_ _available] nil)
+ (fetch! [_ _queue->free-slots] nil)
(queue-depths [_] nil)
(batch-successful! [_ _queue _batch-id] nil)
(failure-count [_ _queue _batch-id] nil)
@@ -35,3 +41,83 @@
{:channel "orphaned-queue" :reason "delivery-exhausted"})
0)))
"stale-recovery drops are NOT conflated with the delivery-retry exhaustion reason")))))
+
+;;; ------------------------------- :max-concurrent-batches -------------------------------
+
+(defn- gated-listener
+ "A listener that records its own peak concurrency. Each delivery counts itself in, parks on
+ `release`, then counts itself out — so `peak` ends up holding the most that were ever inside the
+ listener at the same time, which is exactly what the cap is supposed to bound."
+ [in-flight peak arrived ^CountDownLatch release]
+ (fn [_msg]
+ (swap! peak max (swap! in-flight inc))
+ (swap! arrived inc)
+ (.await release)
+ (swap! in-flight dec)))
+
+(defn- publish-n! [queue n]
+ (mq/with-queue queue [q]
+ (doseq [i (range n)] (mq/put q {:i i}))))
+
+(deftest max-concurrent-batches-caps-in-flight-deliveries-test
+ (testing "a node delivers at most :max-concurrent-batches batches of a queue at once, and takes the
+ rest only as slots free up"
+ (mq.tu/with-test-mq [_ctx]
+ (let [queue :queue/capped-concurrency
+ in-flight (atom 0)
+ peak (atom 0)
+ arrived (atom 0)
+ release (CountDownLatch. 1)]
+ ;; one message per batch, so "batches in flight" == "messages in flight"
+ (q.registry/register-queue! queue {:transactional :try
+ :max-batch-messages 1
+ :max-concurrent-batches 2})
+ (mq.tu/listen! queue (gated-listener in-flight peak arrived release))
+ (publish-n! queue 5)
+ (testing "exactly the cap is picked up while the listener is blocked"
+ (is (true? (mq.tu/wait-for! #(= 2 @arrived) 5000))
+ "two batches reached the listener")
+ ;; Give the poll loop many chances to (wrongly) fetch a third: it spins every 50ms.
+ (is (nil? (mq.tu/wait-for! #(> @arrived 2) 500))
+ "no third batch is taken while the node is at capacity")
+ (is (= 2 @peak) "never more than the cap in flight at once"))
+ (testing "the remaining batches are delivered once the slots free"
+ (.countDown release)
+ (is (true? (mq.tu/wait-for! #(= 5 @arrived) 5000))
+ "all 5 batches eventually delivered — capacity throttles, it doesn't drop")
+ (is (= 2 @peak) "the cap held for the whole drain"))
+ (mq/unlisten! queue)))))
+
+(deftest uncapped-queue-is-unbounded-test
+ (testing "a queue that declares no cap is unbounded on the poll driver too — the same as on Quartz.
+ The driver used to take one batch at a time regardless, which made the *same* queue config
+ mean different things depending on which backend was running."
+ (mq.tu/with-test-mq [_ctx]
+ (let [queue :queue/uncapped-concurrency
+ in-flight (atom 0)
+ peak (atom 0)
+ arrived (atom 0)
+ release (CountDownLatch. 1)]
+ (q.registry/register-queue! queue {:transactional :try :max-batch-messages 1})
+ (mq.tu/listen! queue (gated-listener in-flight peak arrived release))
+ (publish-n! queue 3)
+ (is (true? (mq.tu/wait-for! #(= 3 @arrived) 5000))
+ "all three batches are picked up while the listener is still blocked — nothing throttles
+ a queue that declared no cap")
+ (is (= 3 @peak) "all three were in the listener at once")
+ (.countDown release)
+ (mq/unlisten! queue)))))
+
+(deftest failed-delivery-releases-its-slot-test
+ (testing "a slot is released even when the listener throws, so a failing queue can't leak capacity
+ and quietly throttle itself to a standstill"
+ (mq.tu/with-test-mq [_ctx]
+ (let [queue :queue/slot-release
+ attempts (atom 0)]
+ (q.registry/register-queue! queue {:transactional :try :max-concurrent-batches 1})
+ (mq.tu/listen! queue (fn [_] (swap! attempts inc) (throw (ex-info "boom" {}))))
+ (publish-n! queue 1)
+ (is (true? (mq.tu/wait-for! #(pos? @attempts) 5000)) "the listener ran and threw")
+ (is (true? (mq.tu/wait-for! #(zero? (q.concurrency/in-flight queue)) 5000))
+ "the slot is returned after the failed delivery, not leaked")
+ (mq/unlisten! queue)))))
diff --git a/test/metabase/mq/queue/quartz_test.clj b/test/metabase/mq/queue/quartz_test.clj
index 9d268bf07f6..1f0ca7be367 100644
--- a/test/metabase/mq/queue/quartz_test.clj
+++ b/test/metabase/mq/queue/quartz_test.clj
@@ -4,21 +4,26 @@
root `*listeners*` (the production registration path; Quartz threads don't inherit the test
thread's dynamic bindings) and drive the backend directly under an in-memory scheduler."
(:require
+ [clojure.string :as str]
[clojure.test :refer :all]
[metabase.mq.impl :as mq.impl]
[metabase.mq.listener :as listener]
[metabase.mq.payload :as payload]
+ [metabase.mq.quartz-affinity :as quartz-affinity]
[metabase.mq.queue.backend :as q.backend]
+ [metabase.mq.queue.concurrency :as q.concurrency]
[metabase.mq.queue.quartz :as q.quartz]
[metabase.mq.queue.registry :as q.registry]
[metabase.mq.settings :as mq.settings]
+ [metabase.task.core :as task]
[metabase.task.impl :as task.impl]
[metabase.test :as mt]
[metabase.test.fixtures :as fixtures])
(:import
- (java.util Properties)
+ (java.time Instant)
+ (java.util Date Properties)
(java.util.concurrent CountDownLatch CyclicBarrier TimeUnit)
- (org.quartz JobDataMap JobDetail JobExecutionContext JobExecutionException Scheduler Trigger)
+ (org.quartz JobDataMap JobDetail JobExecutionContext JobExecutionException JobKey Scheduler Trigger TriggerBuilder)
(org.quartz.impl StdSchedulerFactory)))
(set! *warn-on-reflection* true)
@@ -292,13 +297,52 @@
queue (keyword "queue" (str "quartz-nolistener-requeue-" (random-uuid)))]
(#'q.quartz/ensure-queue-job! scheduler queue false)
(try
- (#'q.quartz/requeue-no-listener! scheduler queue (payload/encode ["x"]) 0)
+ (#'q.quartz/requeue-no-listener! scheduler queue (payload/encode ["x"]) 0 (Date.))
(let [triggers (job-triggers scheduler queue)]
(is (= 1 (count triggers))
"a single re-queue trigger is placed back into the store (re-acquired via affinity, not fired here)")
(is (instance? Trigger (first triggers))))
(finally (.deleteJob scheduler (#'q.quartz/queue-job-key queue))))))))
+(deftest quartz-requeue-no-listener-preserves-the-messages-age-test
+ (testing "a no-listener requeue keeps the message's original start time rather than stamping it `now`.
+
+ The age reaper ([[metabase.mq.task.queue-reaper]]) gives up on a never-acquired queue
+ trigger by its `start_time`. But a bounced message is re-queued as a *fresh* trigger, so
+ stamping `now` resets its own reaper clock on every bounce — a message that keeps landing
+ on nodes with no listener could never age out, which is exactly the message the reaper
+ exists to drop. Nothing else bounds this path: `requeue-no-listener!` carries no backoff
+ and spends no retry budget. It is most reachable during a rolling deploy, when nodes are
+ tearing their listeners down, and in the startup window before node-affinity is installed
+ (its capability fn defaults to `:all`, i.e. no filtering)."
+ (mt/with-temp-scheduler!
+ (let [^Scheduler scheduler (task.impl/scheduler)
+ queue (keyword "queue" (str "quartz-nolistener-age-" (random-uuid)))
+ payload (payload/encode ["x"])
+ ;; a message that has already been waiting hours for a node that can handle it
+ start-at (Date/from (.minusSeconds (Instant/now) (* 6 60 60)))
+ ctx (reify JobExecutionContext
+ (getMergedJobDataMap [_]
+ (doto (JobDataMap.)
+ (.put "queue" (name queue))
+ (.put "payload" payload)
+ (.put "attempt" "0")))
+ (getScheduler [_] scheduler)
+ (getTrigger [_] (-> (TriggerBuilder/newTrigger) (.startAt start-at) (.build))))]
+ ;; deliberately no listener for `queue` on this node — that is the path under test
+ (q.registry/register-queue! queue {:transactional :try})
+ (#'q.quartz/ensure-queue-job! scheduler queue false)
+ (try
+ (#'q.quartz/deliver-batch! ctx)
+ (let [triggers (job-triggers scheduler queue)]
+ (is (= 1 (count triggers))
+ "the message is re-queued for a node that does have the listener")
+ (is (= start-at (.getStartTime ^Trigger (first triggers)))
+ "and keeps its original start time, so the reaper's clock keeps running"))
+ (finally
+ (.deleteJob scheduler (#'q.quartz/queue-job-key queue))
+ (swap! q.registry/*queues* dissoc queue)))))))
+
(deftest quartz-exclusive-job-not-downgraded-test
(testing "a node that disagrees about :exclusive can't downgrade an existing exclusive job"
;; Simulates three nodes (the per-process `ensured-jobs` cache is reset between them) publishing
@@ -328,3 +372,203 @@
(is (= "ExclusiveQueueMessageJob" (job-class))
"exclusive is sticky; a disagreeing node cannot downgrade it back to concurrent")
(finally (.deleteJob scheduler jk))))))))
+
+(deftest quartz-on-error-fires-on-exhaustion-test
+ (testing "the :on-error terminal-failure hook fires on the push backend too, with the dropped batch"
+ ;; Backend parity with `metabase.mq.queue.on-error-test` (poll backend): both route their
+ ;; delivery failures through the shared `q.impl/handle-batch-failure-policy!`, so the hook must
+ ;; behave identically here.
+ (mt/with-temporary-setting-values [queue-max-retries 2]
+ (mt/with-temp-scheduler!
+ (let [dropped (atom [])
+ queue (keyword "queue" (str "quartz-on-error-" (random-uuid)))
+ boom (ex-info "always boom" {})]
+ (q.registry/register-queue! queue {:transactional :try
+ :on-error (fn [info] (swap! dropped conj info))})
+ (listener/batch-listen! queue (fn [_msgs] (throw boom)))
+ (try
+ (publish! queue {:id 7})
+ (is (wait-for! #(seq @dropped) 8000)
+ ":on-error fires once the batch exhausts its retries")
+ (is (= 1 (count @dropped)))
+ (let [info (first @dropped)]
+ (is (= queue (:channel info)))
+ (is (= [{:id 7}] (:messages info))
+ "the handler receives the decoded messages of the dropped batch")
+ (is (= 2 (:attempts info))))
+ (finally
+ (listener/unlisten! queue)
+ (swap! q.registry/*queues* dissoc queue))))))))
+
+;;; ------------------------------- :max-concurrent-batches -------------------------------
+
+;;; Where the cap is actually enforced on Quartz is *trigger acquisition* — a node at capacity stops
+;;; acquiring the queue's triggers, so they stay WAITING for a node with room. That is exercised
+;;; against the real app-DB acquire query in `metabase.mq.quartz-affinity-test` (the delegate is a JDBC
+;;; DriverDelegate, so the tests here — which run on RAMJobStore — cannot see it at all).
+;;;
+;;; What is left to pin here is what `deliver-batch!` does with a batch that reaches it anyway.
+
+(defn- job-ctx
+ "A minimal `JobExecutionContext` for driving `deliver-batch!` directly, carrying `queue`'s payload
+ and a real scheduler (so a re-queue actually writes a trigger we can count)."
+ ^JobExecutionContext [^Scheduler scheduler queue payload]
+ (reify JobExecutionContext
+ (getMergedJobDataMap [_]
+ (doto (JobDataMap.)
+ (.put "queue" (name queue))
+ (.put "payload" payload)
+ (.put "attempt" "0")))
+ (getScheduler [_] scheduler)))
+
+(deftest over-capacity-batch-is-still-delivered-test
+ (testing "a batch that reaches deliver-batch! while the node is already at its cap is delivered
+ anyway, not handed back"
+ ;; The cap is a throttle on what a node *takes* (trigger acquisition), and that check necessarily
+ ;; races the job body: Quartz decides what to acquire on its scheduler thread, before this runs on
+ ;; a worker thread. Bouncing the odd batch that slips through would cost a whole re-queue path —
+ ;; backoff, metric, and a hot-loop to avoid if the acquisition filter ever stops working — to shave
+ ;; an overshoot that is small, bounded and drains itself. So we run it and let the count go over.
+ (mt/with-temp-scheduler!
+ (let [queue (keyword "queue" (str "quartz-overcap-" (random-uuid)))
+ scheduler (task/scheduler)
+ ran (atom 0)]
+ (q.registry/register-queue! queue {:transactional :try :max-concurrent-batches 1})
+ (listener/batch-listen! queue (fn [_msgs] (swap! ran inc)))
+ (try
+ ;; Create the durable job without publishing — a live trigger would be fired by this
+ ;; (running) scheduler on its own thread, adding a delivery we didn't ask for.
+ (#'q.quartz/ensure-queue-job! scheduler queue false)
+ (let [before (count (job-triggers scheduler queue))]
+ ;; Hold the queue's only slot, so the fired job runs while we are already at the cap —
+ ;; exactly the acquire/execute race the acquisition filter cannot close.
+ (q.concurrency/with-slot queue
+ (is (true? (q.concurrency/at-capacity? queue)) "we are at the cap before it fires")
+ (is (nil? (#'q.quartz/deliver-batch!
+ (job-ctx scheduler queue (payload/encode [{:id 1}]))))
+ "returns normally, so Quartz completes the one-shot")
+ (is (= 1 @ran)
+ "the listener ran: the batch was delivered over the cap rather than bounced")
+ (is (= before (count (job-triggers scheduler queue)))
+ "and no trigger was written back to the store — there is no bounce path any more"))
+ (is (zero? (q.concurrency/in-flight queue))
+ "the over-cap slot was released again, so the node drains back under its limit"))
+ (finally
+ (listener/unlisten! queue)
+ (swap! q.registry/*queues* dissoc queue)))))))
+
+(deftest delivery-releases-its-slot-test
+ (testing "the slot is released after delivery, on both the success and the failure path — otherwise
+ a queue would throttle itself to a standstill after a few batches"
+ (mt/with-temp-scheduler!
+ (let [scheduler (task/scheduler)]
+ (doseq [[label listener] [["success" (fn [_msgs] nil)]
+ ["failure" (fn [_msgs] (throw (ex-info "boom" {})))]]]
+ (testing label
+ (let [queue (keyword "queue" (str "quartz-release-" label "-" (random-uuid)))]
+ (q.registry/register-queue! queue {:transactional :try :max-concurrent-batches 1})
+ (listener/batch-listen! queue listener)
+ (try
+ ;; Create the durable job *without* publishing: the failure path reschedules a retry
+ ;; onto it, and it must exist for that. Publishing would also enqueue a live trigger,
+ ;; which this (running) scheduler would fire on its own thread — a second, concurrent
+ ;; delivery holding a slot of its own, which is exactly what we're trying to measure.
+ (#'q.quartz/ensure-queue-job! scheduler queue false)
+ (#'q.quartz/deliver-batch! (job-ctx scheduler queue (payload/encode [{:id 1}])))
+ (is (zero? (q.concurrency/in-flight queue))
+ "the slot is returned, not leaked")
+ (finally
+ (listener/unlisten! queue)
+ (swap! q.registry/*queues* dissoc queue))))))))))
+
+(deftest capability-fn-excludes-at-capacity-queues-test
+ (testing "a queue this node is at capacity on is reported as one it cannot currently handle, so the
+ affinity delegate leaves its triggers WAITING for a node that can"
+ (let [capped (keyword "queue" (str "quartz-capable-capped-" (random-uuid)))
+ uncapped (keyword "queue" (str "quartz-capable-uncapped-" (random-uuid)))]
+ (q.registry/register-queue! capped {:transactional :try :max-concurrent-batches 1})
+ (q.registry/register-queue! uncapped {:transactional :try})
+ (listener/batch-listen! capped (fn [_] nil))
+ (listener/batch-listen! uncapped (fn [_] nil))
+ (try
+ (testing "with a free slot, both queues are acquirable"
+ (let [capable (q.quartz/capable-queue-names)]
+ (is (contains? capable (name capped)))
+ (is (contains? capable (name uncapped)))))
+ (q.concurrency/with-slot capped
+ (testing "once the cap is reached, the capped queue drops out of the acquire query"
+ (let [capable (q.quartz/capable-queue-names)]
+ (is (not (contains? capable (name capped)))
+ "at capacity — this node must not acquire more of its triggers")
+ (is (contains? capable (name uncapped))
+ "an uncapped queue is unaffected: it stays unbounded, as it was before caps existed")))
+ (testing "and the predicate spliced into Quartz's acquire SQL really does exclude it"
+ (let [sql (quartz-affinity/rewrite-acquisition-sql
+ "SELECT * FROM QRTZ_TRIGGERS WHERE x = 1 ORDER BY next_fire_time"
+ (q.quartz/capable-queue-names))]
+ (is (not (str/includes? sql (name capped))))
+ (is (str/includes? sql (name uncapped))))))
+ (testing "releasing the slot makes it acquirable again"
+ (is (contains? (q.quartz/capable-queue-names) (name capped))))
+ (finally
+ (listener/unlisten! capped)
+ (listener/unlisten! uncapped)
+ (swap! q.registry/*queues* dissoc capped)
+ (swap! q.registry/*queues* dissoc uncapped))))))
+
+(deftest wake-scheduler-pokes-the-acquire-loop-for-capped-queues-only-test
+ ;; The nudge is load-bearing, not an optimization: once the acquisition filter excludes a queue this
+ ;; node is at capacity on, Quartz's acquire loop finds nothing and parks on its sigLock for up to
+ ;; idleWaitTime (30s) — and a plain (non-@DisallowConcurrentExecution) job completing does NOT signal
+ ;; it. Without this poke a capped queue would drain N batches, sleep 30s, drain N more.
+ ;;
+ ;; `resumeJob` is the public API we ride to get that signal: QuartzScheduler.resumeJob ends in
+ ;; notifySchedulerThread(0L). We aim it at a durable job with no triggers, so the resume itself is a
+ ;; no-op and only the signal survives. This pins our half of that contract.
+ (mt/with-temp-scheduler!
+ (let [scheduler (task/scheduler)
+ capped (keyword "queue" (str "quartz-wake-capped-" (random-uuid)))
+ uncapped (keyword "queue" (str "quartz-wake-uncapped-" (random-uuid)))
+ ^JobKey nudge-key @#'q.quartz/nudge-job-key]
+ (reset! @#'q.quartz/nudge-job-ensured? false)
+ (q.registry/register-queue! capped {:transactional :try :max-concurrent-batches 1})
+ (q.registry/register-queue! uncapped {:transactional :try})
+ (try
+ (testing "an uncapped queue is never filtered out, so there is no stall to break and no poke"
+ (#'q.quartz/wake-scheduler! scheduler uncapped)
+ (is (false? (.checkExists scheduler nudge-key))
+ "no nudge job created — an uncapped queue doesn't pay the cost of the wake-up"))
+ (testing "a capped queue pokes the acquire loop, via a durable job that carries no triggers"
+ (#'q.quartz/wake-scheduler! scheduler capped)
+ (is (true? (.checkExists scheduler nudge-key))
+ "the nudge job exists")
+ (is (empty? (.getTriggersOfJob scheduler nudge-key))
+ "and has no triggers — it never fires; we call resumeJob purely for the signal"))
+ (testing "poking is idempotent and safe to call on every completed batch"
+ (dotimes [_ 3] (#'q.quartz/wake-scheduler! scheduler capped))
+ (is (true? (.checkExists scheduler nudge-key))))
+ (finally
+ (swap! q.registry/*queues* dissoc capped)
+ (swap! q.registry/*queues* dissoc uncapped))))))
+
+(deftest delivery-wakes-the-acquire-loop-after-freeing-a-slot-test
+ (testing "finishing a batch on a capped queue tells the acquire loop to look again — that is what
+ turns a freed slot into the next batch actually being picked up"
+ (mt/with-temp-scheduler!
+ (let [scheduler (task/scheduler)
+ queue (keyword "queue" (str "quartz-wake-on-release-" (random-uuid)))
+ woke (atom [])]
+ (q.registry/register-queue! queue {:transactional :try :max-concurrent-batches 1})
+ (listener/batch-listen! queue (fn [_msgs] nil))
+ (try
+ (publish! queue {:seed 1})
+ (with-redefs-fn {#'q.quartz/wake-scheduler! (fn [_sched q] (swap! woke conj q))}
+ (fn []
+ (#'q.quartz/deliver-batch! (job-ctx scheduler queue (payload/encode [{:id 1}])))))
+ (is (= [queue] @woke)
+ "the acquire loop is woken exactly once, after the slot is released")
+ (is (zero? (q.concurrency/in-flight queue))
+ "and the slot really was free by then")
+ (finally
+ (listener/unlisten! queue)
+ (swap! q.registry/*queues* dissoc queue)))))))
diff --git a/test/metabase/mq/queue/registry_test.clj b/test/metabase/mq/queue/registry_test.clj
index bcb54fce0c9..3c7a6afa89b 100644
--- a/test/metabase/mq/queue/registry_test.clj
+++ b/test/metabase/mq/queue/registry_test.clj
@@ -65,3 +65,72 @@
(q.registry/register-queue! :queue/ok {:transactional :require :exclusive true
:max-batch-messages 50 :dedup-fn identity})
(is (= 50 (q.registry/max-batch-messages :queue/ok)))))
+
+(deftest max-concurrent-batches-test
+ (testing "a queue that doesn't declare a cap has none"
+ (q.registry/register-queue! :queue/uncapped {:transactional :try})
+ (is (nil? (q.registry/max-concurrent-batches :queue/uncapped))))
+ (testing "an undeclared queue has no cap"
+ (is (nil? (q.registry/max-concurrent-batches :queue/never-declared))))
+ (testing "a literal int cap is returned as-is"
+ (q.registry/register-queue! :queue/capped {:transactional :try :max-concurrent-batches 3})
+ (is (= 3 (q.registry/max-concurrent-batches :queue/capped))))
+ (testing "a fn cap is resolved at read time, so it tracks a setting that changes"
+ (let [cap (atom 2)]
+ (q.registry/register-queue! :queue/dynamic {:transactional :try
+ :max-concurrent-batches (fn [] @cap)})
+ (is (= 2 (q.registry/max-concurrent-batches :queue/dynamic)))
+ (reset! cap 5)
+ (is (= 5 (q.registry/max-concurrent-batches :queue/dynamic))))))
+
+(deftest max-concurrent-batches-fn-that-throws-is-treated-as-unbounded-test
+ (testing "a cap fn that throws degrades to unbounded (nil) instead of propagating"
+ ;; This resolver runs inside the Quartz trigger-acquisition filter, so a throw here would abort
+ ;; acquisition for *every* queue on the node, not just this one. The cap is a best-effort throttle,
+ ;; so a broken resolver must degrade to unbounded rather than wedge the scheduler.
+ (q.registry/register-queue! :queue/boom-cap
+ {:transactional :try
+ :max-concurrent-batches (fn [] (throw (ex-info "boom" {})))})
+ (is (nil? (q.registry/max-concurrent-batches :queue/boom-cap)))))
+
+(deftest rejects-exclusive-with-max-concurrent-batches-test
+ (testing "a queue cannot declare both :exclusive and :max-concurrent-batches"
+ ;; :exclusive already pins the queue to one batch cluster-wide, which is strictly stronger than any
+ ;; per-node cap. Declaring both reads like it means something and doesn't.
+ (is (thrown-with-msg?
+ ExceptionInfo #"(?i)exclusive"
+ (q.registry/register-queue! :queue/both {:transactional :try
+ :exclusive true
+ :max-concurrent-batches 2}))))
+ (testing "and not even with a cap of 1, which looks equivalent but isn't (per-node vs cluster-wide)"
+ (is (thrown-with-msg?
+ ExceptionInfo #"(?i)exclusive"
+ (q.registry/register-queue! :queue/both-one {:transactional :try
+ :exclusive true
+ :max-concurrent-batches 1}))))
+ (testing ":exclusive false alongside a cap is fine — that's just the per-node cap on its own"
+ (q.registry/register-queue! :queue/not-exclusive {:transactional :try
+ :exclusive false
+ :max-concurrent-batches 3})
+ (is (= 3 (q.registry/max-concurrent-batches :queue/not-exclusive)))
+ (is (false? (q.registry/exclusive? :queue/not-exclusive))))
+ (testing "either one on its own still registers"
+ (q.registry/register-queue! :queue/just-exclusive {:transactional :try :exclusive true})
+ (is (true? (q.registry/exclusive? :queue/just-exclusive)))
+ (is (nil? (q.registry/max-concurrent-batches :queue/just-exclusive)))
+ (q.registry/register-queue! :queue/just-capped {:transactional :try :max-concurrent-batches 2})
+ (is (= 2 (q.registry/max-concurrent-batches :queue/just-capped)))
+ (is (false? (q.registry/exclusive? :queue/just-capped)))))
+
+(deftest rejects-invalid-max-concurrent-batches-test
+ (testing "register-queue! rejects a non-positive :max-concurrent-batches"
+ (is (thrown-with-msg?
+ ExceptionInfo #"max-concurrent-batches"
+ (q.registry/register-queue! :queue/zero-cap {:transactional :try :max-concurrent-batches 0})))
+ (is (thrown-with-msg?
+ ExceptionInfo #"max-concurrent-batches"
+ (q.registry/register-queue! :queue/neg-cap {:transactional :try :max-concurrent-batches -1}))))
+ (testing "register-queue! rejects a :max-concurrent-batches that is neither an int nor a fn"
+ (is (thrown-with-msg?
+ ExceptionInfo #"max-concurrent-batches"
+ (q.registry/register-queue! :queue/str-cap {:transactional :try :max-concurrent-batches "2"})))))
diff --git a/test/metabase/mq/test_util.clj b/test/metabase/mq/test_util.clj
index cfa5cd07d11..78b76b9f97f 100644
--- a/test/metabase/mq/test_util.clj
+++ b/test/metabase/mq/test_util.clj
@@ -18,8 +18,8 @@
[metabase.mq.polling :as mq.polling]
[metabase.mq.publish-buffer :as publish-buffer]
[metabase.mq.queue.backend :as q.backend]
+ [metabase.mq.queue.concurrency :as q.concurrency]
[metabase.mq.queue.memory :as q.memory]
- [metabase.mq.queue.polling :as q.polling]
[metabase.mq.queue.registry :as q.registry]))
(set! *warn-on-reflection* true)
@@ -40,17 +40,17 @@
"Returns a set of the idle conditions that currently do NOT hold. Empty set == idle."
[ctx]
(cond-> #{}
- (seq @publish-buffer/*publish-buffer*) (conj :publish-buffer-nonempty)
- (seq (q.polling/busy-channels)) (conj :busy-channels)
- (not (layer-drained? ctx)) (conj :memory-messages-pending)))
+ (seq @publish-buffer/*publish-buffer*) (conj :publish-buffer-nonempty)
+ (seq (q.concurrency/working-channels)) (conj :channels-still-working)
+ (not (layer-drained? ctx)) (conj :memory-messages-pending)))
(defn wait-for-idle!
"Blocks until the MQ fixture reaches quiescence or `timeout-ms` elapses. Throws
an ex-info whose `:unmet` key names the conditions still holding on timeout.
- Because `poll-once!` drains memory channels *before* setting `active-handlers`
- on the next submit-delivery, a single idle observation can occur in the gap
- between the two. We require two consecutive idle observations 20 ms apart."
+ Because the poll loop drains memory channels *before* the next submit-delivery
+ claims its slot, a single idle observation can occur in the gap between the two.
+ We require two consecutive idle observations 20 ms apart."
([ctx] (wait-for-idle! ctx 5000))
([ctx timeout-ms]
(let [deadline (+ (System/currentTimeMillis) timeout-ms)]
@@ -194,7 +194,7 @@
(start! [_] (q.backend/start! inner))
(shutdown! [_] (q.backend/shutdown! inner))
(backend-id [_] (q.backend/backend-id inner))
- (fetch! [_ available-queues] (q.backend/fetch! inner available-queues))
+ (fetch! [_ queue->free-slots] (q.backend/fetch! inner queue->free-slots))
(recover-stale! [_ stale-timeout-ms max-retries]
(q.backend/recover-stale! inner stale-timeout-ms max-retries))
(run-heartbeats! [_] (q.backend/run-heartbeats! inner))
@@ -225,6 +225,7 @@
(let [{:keys [backend listeners duplicate-delivery?] :or {backend :memory}} opts]
(binding [listener/*listeners* (atom {})
q.registry/*queues* (atom {})
+ q.concurrency/*in-flight* (atom {})
publish-buffer/*publish-buffer* (atom {})
publish-buffer/*publish-buffer-ms* 0]
(let [backends (make-fixture-backends backend)