The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Unify image-buffer pool (#77836)
Alexander Polyankin2819934fdcae
diff --git a/src/metabase/analytics/prometheus.clj b/src/metabase/analytics/prometheus.cljindex fe5ab0d8383..75d2fcd102a 100644--- a/src/metabase/analytics/prometheus.clj+++ b/src/metabase/analytics/prometheus.clj@@ -531,6 +531,10 @@(prometheus/counter :metabase-notification/template-update{:description "Number of notification templates updated.":labels [:channel-type]})+ (prometheus/counter :metabase-notification/image-buffer-pooled+ {:description "Number of chart rasterization images backed by a pooled buffer."})+ (prometheus/counter :metabase-notification/image-buffer-unpooled+ {:description "Number of chart rasterization images too large for the pool, allocated fresh."})(prometheus/counter :metabase-gsheets/connection-creation-began{:description "How many times the instance has initiated a Google Sheets connection creation."})(prometheus/counter :metabase-gsheets/connection-creation-errordiff --git a/src/metabase/channel/render/image_buffer.clj b/src/metabase/channel/render/image_buffer.cljindex eba66fae006..879daddcee5 100644--- a/src/metabase/channel/render/image_buffer.clj+++ b/src/metabase/channel/render/image_buffer.clj@@ -1,195 +1,66 @@(ns metabase.channel.render.image-buffer- "A small pool of reusable, fixed-size `int[]` pixel arrays that back the ARGB [[java.awt.image.BufferedImage]]s chart- rasterization renders into, shared across all rendering threads.-- Why: rasterizing a chart (SVG via Batik) paints into a `width * height * 4` byte buffer -- 3-6 MB at dashboard- sizes. Under the default G1 GC any object over half a region (~1 MB at a 2 MB region) is \"humongous\": allocated- into dedicated regions a young GC can't reclaim. Subscriptions render in bursts, so without reuse these pile up- faster than mixed/full GCs reclaim the regions and the JVM OOMs on region exhaustion while most of the heap is free.-- What we reuse and why this shape: the only expensive, humongous part of a `BufferedImage` is its backing `int[]`.- Constructing the wrapper objects (`DataBufferInt` + raster + the *cached* `ColorModel/getRGBdefault`) around an- existing array is ~1 us (verified) -- and critically, one oversized `int[]` can back an image of ANY `w` x `h` whose- `w*h` fits it, because `createPackedRaster` re-interprets the flat array at the new scanline stride. So we don't- need a per-size cache: we pool a few `int[]`s of one fixed size big enough to cover the vast majority of renders,- and on each [[acquire]] wrap a throwaway correctly-sized image around one. A render that needs *more* pixels than- the fixed size just gets a fresh standard `BufferedImage` (the original behavior) -- a rare one-off, not pooled. We- pool the arrays, not the images, to keep the minimum in memory.-- Reuse is safe because Batik PNG-encodes the buffer eagerly and returns only a `byte[]`; the image never escapes. The- one load-bearing rule is clearing the used region on acquire (Batik composites with `SrcOver` and skips its- background fill when the background is nil, so stale pixels would bleed a previous render through).-- Concurrency: the idle arrays live in a `ConcurrentLinkedDeque`, so every operation is lock-free and thread-safe on- its own; there is no cross-operation invariant to protect. [[acquire]] never blocks and never refuses. Retention is- bounded to `:max-arrays` idle arrays (a best-effort check on [[release]] -- occasionally keeping one extra under a- race is harmless, it is only a retention cap). A daemon thread periodically clears idle arrays so memory drains to- zero when rendering stops.-- Pools are plain maps built by [[->pool]]; the core fns take one. The no-arg arities target the process-wide- [[default-pool]], which is built lazily on first use (so AOT compilation, which loads every namespace, doesn't- allocate it or spawn its sweeper thread). Only the default pool has a sweeper; tests build isolated, sweeper-free- pools with [[->pool]]."+ "Pools the oversized `int[]` pixel arrays that back the ARGB [[java.awt.image.BufferedImage]]s chart rasterization+ renders into. The arrays are humongous allocations under G1 (several MB each) and subscriptions render in bursts,+ so reusing a few fixed-size arrays keeps them from piling up faster than GC can reclaim them.++ One oversized `int[]` can back an image of any `w` x `h` whose `w*h` fits it, because `createPackedRaster`+ re-interprets the flat array at the new scanline stride, and wrapping it in a [[BufferedImage]] is ~1 us. The pool+ is the same dirigiste `Pool` the GraalVM static-viz contexts use, with the same semantics: 0-3 arrays, each held+ exclusively from [[acquire]] to [[release]], idle arrays dropped after ~10 minutes."(:require- [metabase.util.log :as log])+ [metabase.analytics-interface.core :as analytics]+ [metabase.util.pool :as u.pool])(:import+ (io.aleph.dirigiste Pool)(java.awt Point)- (java.awt.image BufferedImage ColorModel DataBufferInt Raster WritableRaster)- (java.util.concurrent ConcurrentLinkedDeque Executors ScheduledExecutorService ThreadFactory TimeUnit)- (java.util.concurrent.atomic LongAdder)))+ (java.awt.image BufferedImage ColorModel DataBufferInt Raster WritableRaster)))(set! *warn-on-reflection* true)-(def ^:private default-array-pixels- "Fixed size (in pixels = ints) of every pooled array. ~2x the size that covers the vast majority of dashboard renders- (1200 wide x up to ~1350 tall ~= 1.6M px), so it also fits 2x-supersampled variants. A render needing more pixels- falls back to a fresh standard image. At 4 bytes/pixel a pooled array is ~13 MB."+(def ^:private array-pixels+ "Size in pixels (= ints) of every pooled array: ~2x the largest common dashboard render (1200 wide x up to ~1350+ tall), so it also fits 2x-supersampled variants. ~13 MB at 4 bytes/pixel."(* 1200 2700))-(def ^:private default-max-arrays- "Max idle pooled arrays retained. Bounded by render concurrency (a few Jetty/notification threads), not request rate."- 6)--(def ^:private default-idle-ttl-ms- "How often the daemon drops idle arrays; if nothing was rendered in the interval, the pool drains to zero."- 60000)--;; Standard non-premultiplied sRGB ARGB color model -- a cached singleton. Using this (rather than constructing a-;; DirectColorModel) keeps wrapping cheap: a premultiplied/mismatched color model makes the BufferedImage ctor run a-;; full-raster per-pixel `coerceData` conversion (~7 ms), whereas getRGBdefault matches the raster (no-op).+;; Standard non-premultiplied sRGB ARGB color model (a cached singleton); anything else makes the BufferedImage ctor+;; run a per-pixel coerceData pass over the whole raster.(def ^:private ^ColorModel rgb-default (ColorModel/getRGBdefault))-(def ^:private ^"[I" argb-masks (int-array [0x00ff0000 0x0000ff00 0x000000ff (unchecked-int 0xff000000)]))-(declare default-pool)+(def ^:private ^"[I" argb-masks (int-array [0x00ff0000 0x0000ff00 0x000000ff (unchecked-int 0xff000000)]))-(defn ->pool- "Construct a fresh, independent pool: a plain map of its idle-array store, counters, and config. With no args uses- production sizing. Tests pass their own to isolate from the [[default-pool]] (and get no sweeper).+(def ^:private pool-key ::arrays)- `:idle` is a `ConcurrentLinkedDeque` of idle `int[]`, each of length `:array-pixels`. `:hits`/`:misses`/`:one-offs`- are counters (a \"one-off\" is a render too large for the fixed array, handed a fresh standard image)."- ([] (->pool {}))- ([{:keys [array-pixels max-arrays idle-ttl-ms]- :or {array-pixels default-array-pixels max-arrays default-max-arrays idle-ttl-ms default-idle-ttl-ms}}]- {:idle (ConcurrentLinkedDeque.)- :hits (LongAdder.)- :misses (LongAdder.)- :one-offs (LongAdder.)- :last-logged (atom {:hits 0 :misses 0})- :array-pixels array-pixels- :max-arrays max-arrays- :idle-ttl-ms idle-ttl-ms}))+(def ^:private ^Pool array-pool+ (u.pool/create-pool #(int-array array-pixels) (fn [_array]) {:max-size 3, :idle-minutes 10}))(defn- wrap-array- "Wrap an existing `int[]` (length >= w*h) as a `w` x `h` ARGB [[BufferedImage]]. Throwaway wrapper (~1 us); only- `backing` carries the humongous cost."+ "A `w` x `h` ARGB [[BufferedImage]] over the first `w*h` ints of `backing`."^BufferedImage [^"[I" backing ^long w ^long h]- (let [db (DataBufferInt. backing (int (* w h)))- ^WritableRaster raster (Raster/createPackedRaster db (int w) (int h) (int w) argb-masks (Point. 0 0))]+ (let [buffer (DataBufferInt. backing (int (* w h)))+ ^WritableRaster raster (Raster/createPackedRaster buffer (int w) (int h) (int w) argb-masks (Point. 0 0))](BufferedImage. rgb-default raster false nil)))-(defn- cleared-array- "An `int[]` of `array-pixels` length with its first `need` ints zeroed (the region this render will use). Reuses one- from `idle` if available (a hit), else allocates (a miss). Clearing is load-bearing -- see ns docstring."- ^"[I" [pool ^long need]- (let [^"[I" backing (if-let [^"[I" reused (.pollFirst ^ConcurrentLinkedDeque (:idle pool))]- (do (.increment ^LongAdder (:hits pool)) reused)- (do (.increment ^LongAdder (:misses pool)) (int-array (long (:array-pixels pool)))))]- (java.util.Arrays/fill backing 0 (int need) (int 0))- backing))-(defn acquire- "Return a cleared `w` x `h` ARGB [[BufferedImage]]. If `w*h` fits the pool's fixed array size, reuses a pooled `int[]`- (or allocates a fixed-size one if none is idle); otherwise returns a fresh standard `BufferedImage` (a one-off, not- pooled). NEVER blocks or refuses. With no pool arg, uses the [[default-pool]].-- Pass the result to [[release]] when done. The image is shared mutable state -- it must not escape the rasterization- (Batik PNG-encodes it before returning, which is what makes reuse safe)."- (^BufferedImage [w h] (acquire (default-pool) w h))- (^BufferedImage [pool ^long w ^long h]- (if (> (* w h) (long (:array-pixels pool)))- (do (.increment ^LongAdder (:one-offs pool)) ; too large: fresh standard image, not pooled- (BufferedImage. (int w) (int h) BufferedImage/TYPE_INT_ARGB))- (wrap-array (cleared-array pool (* w h)) w h))))--(defn- backing-of- "The backing `int[]` of an image produced by [[acquire]]."- ^"[I" [^BufferedImage img]- (.getData ^DataBufferInt (.getDataBuffer (.getRaster img))))+ "Return a cleared `w` x `h` ARGB [[BufferedImage]]. When `w*h` fits the pooled array size, the image wraps a pooled+ `int[]` -- blocking until one is available -- and must be handed back with [[release]]. Larger images are freshly+ allocated and not pooled. The image is shared mutable state and must not escape the rasterization; clearing the+ used region on acquire is load-bearing (Batik composites with `SrcOver`, so stale pixels would bleed a previous+ render through)."+ ^BufferedImage [^long w ^long h]+ (if (> (* w h) (long array-pixels))+ (do+ (analytics/inc! :metabase-notification/image-buffer-unpooled)+ (BufferedImage. (int w) (int h) BufferedImage/TYPE_INT_ARGB))+ (let [^"[I" backing (.acquire array-pool pool-key)]+ (analytics/inc! :metabase-notification/image-buffer-pooled)+ (java.util.Arrays/fill backing 0 (int (* w h)) (int 0))+ (wrap-array backing w h))))(defn release- "Return `img`'s backing array to `pool` for reuse, unless the pool already holds `:max-arrays` idle arrays, or the- array is a one-off (a different size than the pooled arrays) -- in either case it is simply dropped (GC reclaims it).- Safe with `nil`; never throws. With no pool arg, uses the [[default-pool]]."- ([img] (release (default-pool) img))- ([pool img]- (when img- (let [^"[I" backing (backing-of img)- ^ConcurrentLinkedDeque idle (:idle pool)]- (when (and (= (alength backing) (long (:array-pixels pool)))- (< (.size idle) (long (:max-arrays pool))))- (.addFirst idle backing))))))--(defn stats- "Snapshot of a pool's counters `{:hits n :misses n :one-offs n :idle n}`. With no arg, the [[default-pool]]."- ([] (stats (default-pool)))- ([pool]- {:hits (.sum ^LongAdder (:hits pool))- :misses (.sum ^LongAdder (:misses pool))- :one-offs (.sum ^LongAdder (:one-offs pool))- :idle (.size ^ConcurrentLinkedDeque (:idle pool))}))--(defn sweep!- "Drop the pool's idle arrays so memory is reclaimed when rendering stops. The daemon calls this periodically: if- rendering is active, acquire/release repopulate within microseconds; if idle, the pool stays empty. With no arg, the- [[default-pool]]."- ([] (sweep! (default-pool)))- ([pool] (.clear ^ConcurrentLinkedDeque (:idle pool))))--;; --- stats logging (cumulative + interval; periodic, NOT per-send -- a per-send rate is dominated by cold-start) -----(defn- rate-str [hits misses one-offs]- (let [total (+ hits misses)]- (format "%d/%d hits (%.1f%%), %d allocs, %d one-offs"- hits total (if (pos? total) (* 100.0 (/ hits (double total))) 0.0) misses one-offs)))--(defn log-stats!- "Log `pool` reuse: the hit rate since this pool's previous call and the cumulative rate, plus idle/one-off counts.- No-op for the interval line when nothing was rendered since the last call. With no arg, the [[default-pool]]."- ([] (log-stats! (default-pool)))- ([pool]- (let [now (stats pool)- prev @(:last-logged pool)- dh (- (:hits now) (:hits prev))- dm (- (:misses now) (:misses prev))]- (reset! (:last-logged pool) {:hits (:hits now) :misses (:misses now)})- (when (pos? (+ dh dm))- (log/infof "Image buffer reuse (interval): %s; (lifetime): %s; idle arrays: %d"- (rate-str dh dm 0)- (rate-str (:hits now) (:misses now) (:one-offs now))- (:idle now))))))--;; --- the default pool, built lazily on first use, with its one sweeper daemon --------------------------------------(defn- start-sweeper!- "Start the daemon that every `:idle-ttl-ms` sweeps `pool` and logs reuse, so the pool drains to zero when rendering- stops. No Quartz needed -- a process-local timer on a daemon thread (so it never blocks shutdown)."- [pool]- (let [exec (Executors/newScheduledThreadPool- 1 (reify ThreadFactory- (newThread [_ r]- (doto (Thread. ^Runnable r "image-buffer-maintenance") (.setDaemon true)))))]- (.scheduleWithFixedDelay ^ScheduledExecutorService exec- (fn [] (try (sweep! pool) (log-stats! pool)- (catch Throwable e (log/warn e "Error in image-buffer maintenance"))))- (long (:idle-ttl-ms pool)) (long (:idle-ttl-ms pool)) TimeUnit/MILLISECONDS)))--;; A delay so nothing runs at namespace load (AOT-safe). Realized exactly once on first use, which is also when -- and-;; the only place where -- the single sweeper daemon is started.-(defonce ^:private the-pool- (delay (doto (->pool) start-sweeper!)))--(defn- default-pool- "THE process-wide pool, built (and its sweeper started) on first call."- []- @the-pool)+ "Return the pooled array backing `img` (an image from [[acquire]]) to the pool; unpooled oversized images are just+ dropped for GC to reclaim. Safe with `nil`."+ [^BufferedImage img]+ (when img+ (let [^"[I" backing (.getData ^DataBufferInt (.getDataBuffer (.getRaster img)))]+ (when (= (alength backing) (long array-pixels))+ (.release array-pool pool-key backing)))))diff --git a/src/metabase/channel/render/js/common.clj b/src/metabase/channel/render/js/common.cljindex 41b318f6fce..a12449ea4c8 100644--- a/src/metabase/channel/render/js/common.clj+++ b/src/metabase/channel/render/js/common.clj@@ -1,11 +1,8 @@(ns metabase.channel.render.js.common"Shared helpers for pooled static-viz renderers (see [[metabase.channel.render.js.graal]]): the graal- bundle's classpath path, the test-init guard, and the dirigiste worker-pool factory."+ bundle's classpath path and the test-init guard."(:require- [metabase.config.core :as config])- (:import- (io.aleph.dirigiste IPool$Generator Pool Pools)- (java.util.concurrent TimeUnit)))+ [metabase.config.core :as config]))(set! *warn-on-reflection* true)@@ -20,24 +17,3 @@[](when config/tests-available?((requiring-resolve 'mb.hawk.init/assert-tests-are-not-initializing) "(mt/id ...) or (data/id ...)")))--(defn create-pool- "Build a dirigiste `Pool` of static-viz workers, each held exclusively per render (so at most `:max-size`- renders run at once). `:max-size` is the maximum number of concurrent workers. The utilization- controller targets 100% utilization with a min of 0, so when nothing is rendering the pool shrinks to 0- and `destroy` is called on each idle worker; it rechecks every `:idle-minutes`, so a worker- lingers up to that long before being reaped (keeping it warm through gaps between renders). `(generate)`- mints a worker; `(destroy worker)` tears one down. The other constructor args (queue size, sampling- interval) don't matter much."- ^Pool [generate destroy {:keys [max-size idle-minutes]}]- (let [max-queued-acquires 65000- sample-period-ms (.toMillis TimeUnit/MILLISECONDS 25)- control-period-ms (.toMillis TimeUnit/MINUTES idle-minutes)]- (Pool. (reify IPool$Generator- (generate [_ _] (generate))- (destroy [_ _ worker] (destroy worker)))- (Pools/utilizationController 1.0 max-size max-size)- max-queued-acquires- sample-period-ms- control-period-ms- TimeUnit/MILLISECONDS)))diff --git a/src/metabase/channel/render/js/graal.clj b/src/metabase/channel/render/js/graal.cljindex fcd0b324d72..d309522ad7e 100644--- a/src/metabase/channel/render/js/graal.clj+++ b/src/metabase/channel/render/js/graal.clj@@ -20,7 +20,8 @@[metabase.config.core :as config][metabase.util.i18n :refer [trs]][metabase.util.json :as json]- [metabase.util.malli :as mu])+ [metabase.util.malli :as mu]+ [metabase.util.pool :as u.pool])(:import(io.aleph.dirigiste Pool)(org.graalvm.polyglot Context Engine HostAccess Source Value)))@@ -154,8 +155,8 @@three renders run at once — one per context, on the shared engine. When idle for up to 10 minutes thepool shrinks to 0 and the generator's `destroy` closes the context (and, on the last one, the sharedengine); the first render after an idle gap rebuilds them. See- [[metabase.channel.render.js.common/create-pool]]."- (common/create-pool generate-context! destroy-context! {:max-size 3, :idle-minutes 10}))+ [[metabase.util.pool/create-pool]]."+ (u.pool/create-pool generate-context! destroy-context! {:max-size 3, :idle-minutes 10}))(defn- do-with-static-viz-context"Borrow a pooled static-viz context and call `f` with it, held exclusively for the call (never let it —diff --git a/src/metabase/sql_parsing/common.clj b/src/metabase/sql_parsing/common.cljindex af213aa2725..a9266b26a82 100644--- a/src/metabase/sql_parsing/common.clj+++ b/src/metabase/sql_parsing/common.clj@@ -1,7 +1,6 @@(ns metabase.sql-parsing.common"Shared helpers for pooled sqlglot parsers (see [[metabase.sql-parsing.graal]]): the python-sources- locations, the lazy dev sqlglot install, the protocol reify over a transport function, and the- dirigiste worker-pool factory (copied from [[metabase.channel.render.js.common]])."+ locations, the lazy dev sqlglot install, and the protocol reify over a transport function."(:require[clojure.java.io :as io][clojure.java.shell :as shell]@@ -9,9 +8,7 @@[metabase.util.json :as json][metabase.util.log :as log])(:import- (io.aleph.dirigiste IPool$Generator Pool Pools)- (java.io File)- (java.util.concurrent TimeUnit)))+ (java.io File)))(set! *warn-on-reflection* true)@@ -154,26 +151,3 @@(json/decode+kw (call "is_single_stmt_of_type" sql stmt-type dialect)))(transpile-sql [_ sql from-dialect to-dialect](json/decode+kw (call "transpile_sql" sql from-dialect to-dialect)))))--;;; -------------------------------------------------- pool -------------------------------------------------(defn create-pool- "Build a dirigiste `Pool` of workers, each held exclusively per call (so at most `:max-size`- calls run at once). `:max-size` is the maximum number of concurrent workers. The utilization- controller targets 100% utilization with a min of 0, so when nothing is running the pool shrinks to 0- and `destroy` is called on each idle worker; it rechecks every `:idle-minutes`, so a worker- lingers up to that long before being reaped (keeping it warm through gaps between calls). `(generate)`- mints a worker; `(destroy worker)` tears one down. The other constructor args (queue size, sampling- interval) don't matter much."- ^Pool [generate destroy {:keys [max-size idle-minutes]}]- (let [max-queued-acquires 65000- sample-period-ms (.toMillis TimeUnit/MILLISECONDS 25)- control-period-ms (.toMillis TimeUnit/MINUTES idle-minutes)]- (Pool. (reify IPool$Generator- (generate [_ _] (generate))- (destroy [_ _ worker] (destroy worker)))- (Pools/utilizationController 1.0 max-size max-size)- max-queued-acquires- sample-period-ms- control-period-ms- TimeUnit/MILLISECONDS)))diff --git a/src/metabase/sql_parsing/graal.clj b/src/metabase/sql_parsing/graal.cljindex 189fdb2d1a2..1b25e4db894 100644--- a/src/metabase/sql_parsing/graal.clj+++ b/src/metabase/sql_parsing/graal.clj@@ -16,7 +16,8 @@[metabase.analytics-interface.core :as analytics][metabase.sql-parsing.common :as common][metabase.util.files :as u.files]- [metabase.util.log :as log])+ [metabase.util.log :as log]+ [metabase.util.pool :as u.pool])(:import(io.aleph.dirigiste Pool)(java.net URI)@@ -189,8 +190,8 @@three sqlglot calls run at once — one per context, on the shared engine. When idle for up to 10minutes the pool shrinks to 0 and the generator's `destroy` closes the context (and, on the last one,the shared engine); the first call after an idle gap rebuilds them. See- [[metabase.sql-parsing.common/create-pool]]."- (common/create-pool generate-context! destroy-context! {:max-size 3, :idle-minutes 10}))+ [[metabase.util.pool/create-pool]]."+ (u.pool/create-pool generate-context! destroy-context! {:max-size 3, :idle-minutes 10}));;; ---------------------------------------------- timeout handling ---------------------------------------diff --git a/src/metabase/util/pool.clj b/src/metabase/util/pool.cljnew file mode 100644index 00000000000..84e9d7c470b--- /dev/null+++ b/src/metabase/util/pool.clj@@ -0,0 +1,27 @@+(ns metabase.util.pool+ "Dirigiste worker-pool factory, shared by the GraalVM static-viz contexts, the sqlglot contexts, and the+ image-render pixel buffers."+ (:import+ (io.aleph.dirigiste IPool$Generator Pool Pools)+ (java.util.concurrent TimeUnit)))++(set! *warn-on-reflection* true)++(defn create-pool+ "Build a dirigiste `Pool` of workers, each held exclusively from acquire to release (so at most `:max-size` are in+ use at once; further acquires block). The utilization controller targets 100% utilization with a min of 0, so an+ idle pool shrinks to 0 and `destroy` is called on each idle worker; it rechecks every `:idle-minutes`, so a worker+ lingers up to that long between uses (keeping it warm through short gaps). `(generate)` mints a worker;+ `(destroy worker)` tears one down. The other constructor args (queue size, sampling interval) don't matter much."+ ^Pool [generate destroy {:keys [max-size idle-minutes]}]+ (let [max-queued-acquires 65000+ sample-period-ms (.toMillis TimeUnit/MILLISECONDS 25)+ control-period-ms (.toMillis TimeUnit/MINUTES idle-minutes)]+ (Pool. (reify IPool$Generator+ (generate [_ _] (generate))+ (destroy [_ _ worker] (destroy worker)))+ (Pools/utilizationController 1.0 max-size max-size)+ max-queued-acquires+ sample-period-ms+ control-period-ms+ TimeUnit/MILLISECONDS)))diff --git a/test/metabase/channel/render/image_buffer_test.clj b/test/metabase/channel/render/image_buffer_test.cljindex 22f7d21d020..236ed9f66ba 100644--- a/test/metabase/channel/render/image_buffer_test.clj+++ b/test/metabase/channel/render/image_buffer_test.clj@@ -7,129 +7,56 @@(set! *warn-on-reflection* true)-;; Every test runs against its OWN pool built with [[image-buffer/->pool]], not the process-wide default pool. That-;; gives deterministic counters (no cross-test residue) and no background sweeper racing the assertions, so we can-;; assert exact hit/miss/idle counts rather than fuzzy "some reuse" thresholds.+;; All tests share the process-wide pool, which holds at most three arrays at a time and blocks further acquires, so+;; every test must release everything it acquires (and these tests must not run in parallel with each other).(deftest acquire-returns-argb-of-requested-size-test(testing "acquire yields an ARGB BufferedImage of exactly the requested size, for any size that fits the pool"- (let [p (image-buffer/->pool)- sizes [[320 240] [1200 800] [1 1] [1200 1322] [64 480]]]- (doseq [[w h] sizes]- (let [^BufferedImage img (image-buffer/acquire p w h)]+ (doseq [[w h] [[320 240] [1200 800] [1 1] [1200 1322] [64 480]]]+ (let [^BufferedImage img (image-buffer/acquire w h)]+ (try(is (instance? BufferedImage img) (format "%dx%d is a BufferedImage" w h))(is (= w (.getWidth img)) (format "%dx%d width" w h))- (is (= h (.getHeight img)) (format "%dx%d height" w h)))))))--(deftest sequential-cycle-reuses-after-one-allocation-test- (testing "cycling one size allocates once, then reuses every time"- (let [p (image-buffer/->pool)- n 100]- (dotimes [_ n]- (image-buffer/release p (image-buffer/acquire p 1207 503)))- (let [{:keys [hits misses idle]} (image-buffer/stats p)]- (is (= 1 misses) "only the very first acquire allocates; the rest must reuse")- (is (= (dec n) hits) "every acquire after the first reused a pooled array")- (is (= 1 idle) "one array cycles through the whole run")))))--(deftest different-sizes-reuse-the-same-array-test- (testing "the headline property: one fixed-size array backs ANY size that fits, so different sizes reuse it (NOT a- per-size cache). After the first allocation every differently-sized acquire is a hit."- (let [p (image-buffer/->pool)]- (image-buffer/release p (image-buffer/acquire p 1200 800)) ;; allocate once- (doseq [[w h] [[1000 600] [200 100] [1199 1000] [50 50]]] ;; all different sizes, all fit- (image-buffer/release p (image-buffer/acquire p w h)))- (let [{:keys [hits misses idle]} (image-buffer/stats p)]- (is (= 1 misses) "only the first acquire allocated; differently-sized acquires reused the same array")- (is (= 4 hits) "every differently-sized acquire after the first was a reuse")- (is (= 1 idle) "still just one array")))))+ (is (= h (.getHeight img)) (format "%dx%d height" w h))+ (finally+ (image-buffer/release img)))))))(deftest reused-array-is-cleared-test- (testing "a reused array is wiped in the region the next render uses (no data leak between renders)"- (let [p (image-buffer/->pool)]- (let [^BufferedImage img (image-buffer/acquire p 64 64)]- (.setRGB img 10 10 (unchecked-int 0xFF00FF00))- (is (not (zero? (.getRGB img 10 10))) "sanity: pixel was dirtied")- (image-buffer/release p img))- (let [^BufferedImage img (image-buffer/acquire p 64 64)]- (is (zero? (.getRGB img 10 10)) "the reused array no longer shows the previous render's pixel")))))--(deftest oversized-request-is-a-one-off-test- (testing "a request larger than the fixed array size falls back to a fresh standard image and is NOT pooled"- (let [p (image-buffer/->pool {:array-pixels (* 100 100)}) ;; tiny fixed size- ^BufferedImage big (image-buffer/acquire p 500 500)] ;; 250k px > 10k cap- (is (= 500 (.getWidth big)))- (is (= 500 (.getHeight big)))- (is (= BufferedImage/TYPE_INT_ARGB (.getType big)) "one-off uses a standard TYPE_INT_ARGB image")- (image-buffer/release p big)- (let [{:keys [one-offs idle]} (image-buffer/stats p)]- (is (= 1 one-offs) "counted as a one-off")- (is (= 0 idle) "the oversized array was not pooled on release")))))--(deftest oversized-and-normal-coexist-test- (testing "one-offs don't pollute the pool: a normal-size array still pools while oversized requests pass through"- (let [p (image-buffer/->pool {:array-pixels (* 100 100)})]- (image-buffer/release p (image-buffer/acquire p 80 80)) ;; fits -> pooled- (image-buffer/release p (image-buffer/acquire p 500 500)) ;; oversized -> one-off, dropped- (let [{:keys [idle one-offs misses]} (image-buffer/stats p)]- (is (= 1 idle) "only the fitting array is pooled")- (is (= 1 one-offs))- (is (= 1 misses) "the fitting acquire allocated once; the one-off is counted separately")))))--(deftest max-arrays-bounds-retention-test- (testing "the pool retains at most :max-arrays idle arrays; surplus releases are dropped"- (let [cap 3- p (image-buffer/->pool {:max-arrays cap})- ;; acquire more arrays than the cap (force misses by holding them all before releasing any)- held (vec (repeatedly (* 2 cap) #(image-buffer/acquire p 700 350)))]- (run! #(image-buffer/release p %) held)- (is (= cap (:idle (image-buffer/stats p))) "retained exactly :max-arrays; surplus discarded"))))--(deftest concurrent-allocates-at-most-thread-count-test- (testing "concurrent threads cycling renders allocate at most thread-count arrays"- (let [thread-count 8- cycles-per-thread 50- timeout-ms 5000- pool (image-buffer/->pool {:max-arrays thread-count})- ;; A barrier so all threads start at the same moment (genuine overlap, so the "<= thread-count allocations"- ;; assertion exercises real concurrency). `await` is bounded and the worker reaches it directly, so a thread- ;; that throws can't strand the others on the barrier.- started (java.util.concurrent.CyclicBarrier. thread-count)- worker (fn []- (.await started (long timeout-ms) java.util.concurrent.TimeUnit/MILLISECONDS)- (dotimes [_ cycles-per-thread]- (image-buffer/release pool (image-buffer/acquire pool 909 405)))- ::done)- futures (doall (repeatedly thread-count #(future (worker))))- results (mapv #(deref % timeout-ms ::timeout) futures)]- (is (every? #(= ::done %) results)- "every worker finished within the timeout (no deadlock / no thrown worker)")- (let [{:keys [hits misses]} (image-buffer/stats pool)]- (is (= (* thread-count cycles-per-thread) (+ hits misses)) "every acquire counted as a hit or a miss")- (is (<= misses thread-count) "never allocated more than the number of concurrent threads")- (is (pos? hits) "reuse happened")))))--(deftest sweep-drains-to-zero-test- (testing "sweep! drops idle arrays so memory is reclaimed when rendering stops"- (let [p (image-buffer/->pool)]- (image-buffer/release p (image-buffer/acquire p 480 480))- (image-buffer/release p (image-buffer/acquire p 481 481))- (is (pos? (:idle (image-buffer/stats p))) "arrays are pooled before the sweep")- (image-buffer/sweep! p)- (is (= 0 (:idle (image-buffer/stats p))) "sweep drains the pool to zero")- ;; and rendering after a sweep just repopulates- (image-buffer/release p (image-buffer/acquire p 480 480))- (is (= 1 (:idle (image-buffer/stats p))) "post-sweep acquire/release repopulates"))))--(deftest stats-idle-reflects-reality-test- (testing "reported idle count matches reality and never goes negative"- (let [p (image-buffer/->pool)]- (dotimes [i 20]- (let [img (image-buffer/acquire p (+ 600 (mod i 3)) 300)]- (when (even? i) (image-buffer/release p img))))- (let [idle (:idle (image-buffer/stats p))]- (is (>= idle 0) "idle count is never negative")- (is (<= idle (:max-arrays (image-buffer/->pool))) "idle never exceeds the retention cap")))))+ (testing "a reused array is wiped in the region the next render uses (no pixel leak between renders)"+ (let [^BufferedImage img (image-buffer/acquire 64 64)]+ (.setRGB img 10 10 (unchecked-int 0xFF00FF00))+ (is (not (zero? (.getRGB img 10 10))) "sanity: pixel was dirtied")+ (image-buffer/release img))+ (let [^BufferedImage img (image-buffer/acquire 64 64)]+ (try+ (is (zero? (.getRGB img 10 10)) "the reused array no longer shows the previous render's pixel")+ (finally+ (image-buffer/release img))))))++(deftest oversized-request-is-not-pooled-test+ (testing "a request larger than the pooled array size gets a fresh standard image; releasing it is a no-op"+ (let [side 2200 ; 2200^2 px > the 1200x2700 pooled array size+ ^BufferedImage big (image-buffer/acquire side side)]+ (is (= side (.getWidth big)))+ (is (= side (.getHeight big)))+ (is (= BufferedImage/TYPE_INT_ARGB (.getType big)) "oversized images use a standard TYPE_INT_ARGB image")+ (is (nil? (image-buffer/release big))))))++(deftest acquire-blocks-at-capacity-test+ (testing "a fourth concurrent acquire blocks until one of the three held images is released"+ (let [held (vec (repeatedly 3 #(image-buffer/acquire 100 100)))+ fourth (future+ (let [img (image-buffer/acquire 100 100)]+ (image-buffer/release img)+ ::acquired))]+ (try+ (is (= ::still-blocked (deref fourth 300 ::still-blocked))+ "no pooled array is free while three are held")+ (image-buffer/release (first held))+ (is (= ::acquired (deref fourth 5000 ::timed-out))+ "unblocks as soon as an array is released")+ (finally+ (run! image-buffer/release (rest held)))))))(deftest release-nil-is-safe-test- (is (nil? (image-buffer/release (image-buffer/->pool) nil))))+ (is (nil? (image-buffer/release nil))))diff --git a/test/metabase/sql_parsing/common_test.clj b/test/metabase/sql_parsing/common_test.cljindex 32e45fa1800..4dbb4f73847 100644--- a/test/metabase/sql_parsing/common_test.clj+++ b/test/metabase/sql_parsing/common_test.clj@@ -1,151 +1,12 @@(ns metabase.sql-parsing.common-test(:require[clojure.test :refer :all]- [metabase.sql-parsing.common :as common])- (:import- (io.aleph.dirigiste Pool)- (java.util.concurrent CountDownLatch)))+ [metabase.sql-parsing.common :as common]))(set! *warn-on-reflection* true)-(defn- mock-worker- "Create a mock 'worker' object. The pool doesn't care what type the objects are - it just stores and- returns them."- [id]- {:worker-id id- :created-at (System/currentTimeMillis)})--(defn- counting-generator- "Returns a generator function that counts how many workers it has created."- [counter-atom]- (fn []- (mock-worker (swap! counter-atom inc))))--(defn- failing-generator- "Returns a generator that fails after creating n workers."- [counter-atom fail-after]- (fn []- (let [id (swap! counter-atom inc)]- (when (> id fail-after)- (throw (ex-info "Generator limit reached" {:count id :limit fail-after})))- (mock-worker id))))--;;; ------------------------------------------------- Test Helpers -----------------------------------------------------(defn- test-pool- "Create a pool for testing with sensible defaults.- Returns [pool created-count-atom destroyed-atom]."- ([]- (test-pool {}))- ([config]- (let [created (atom 0)- destroyed (atom [])- pool (common/create-pool (counting-generator created)- (fn [worker] (swap! destroyed conj worker))- (merge {:max-size 2, :idle-minutes 1} config))]- [pool created destroyed])))--(defn- with-worker- "Execute f with a worker from pool, releasing it afterwards."- [^Pool pool f]- (let [worker (.acquire pool :test)]- (try- (f worker)- (finally- (.release pool :test worker)))))--(defn- run-concurrent- "Run n threads concurrently, each executing f with a worker from pool.- Returns vector of results."- [pool n f]- (let [latch (CountDownLatch. n)- results (atom [])- futures (doall- (for [i (range n)]- (future- (.countDown latch)- (.await latch)- (let [result (with-worker pool (fn [worker] (f i worker)))]- (swap! results conj result)))))]- (doseq [fut futures] @fut)- @results))--;;; ------------------------------------------------ Pool Size Tests ---------------------------------------------------(deftest pool-respects-maximum-size-test- (testing "Pool never creates more than max workers even under concurrent load"- (let [[pool created] (test-pool {:max-size 2})- num-threads 20- results (run-concurrent pool num-threads- (fn [_i worker]- (Thread/sleep (long (+ 10 (rand-int 50))))- worker))]- (is (<= @created 2)- (str "Pool created " @created " workers but max is 2"))- (is (= num-threads (count results))- "All threads should have received a worker"))))--(deftest pool-reuses-idle-workers-test- (testing "Sequential uses reuse the same worker instead of creating new ones"- (let [[pool created] (test-pool)]- (with-worker pool (fn [_worker] (is (= 1 @created) "First use creates one worker")))- (with-worker pool (fn [worker]- (is (= 1 @created) "Second use reuses the same worker")- (is (= 1 (:worker-id worker))))))))--;;; ------------------------------------------------ Disposal Tests ----------------------------------------------------(deftest pool-dispose-destroys-and-replaces-workers-test- (testing "Disposing a worker calls destroy and the next acquire generates a fresh one"- (let [[pool created destroyed] (test-pool)- worker (.acquire ^Pool pool :test)]- (.dispose ^Pool pool :test worker)- (is (= [worker] @destroyed) "Disposed worker should be destroyed")- (with-worker pool (fn [_worker] (is (= 2 @created) "Disposed worker should be replaced"))))))--;;; --------------------------------------------- Generator Failure Tests ----------------------------------------------(deftest pool-handles-generator-failures-test- (testing "Pool gracefully handles when generator fails"- (let [counter (atom 0)- pool (common/create-pool (failing-generator counter 1) (fn [_worker]) {:max-size 2, :idle-minutes 1})]- (is (some? (with-worker pool identity)) "First worker creation succeeds")- ;; Dispose to force new creation- (let [worker (.acquire ^Pool pool :test)]- (.dispose ^Pool pool :test worker))- (is (thrown? Exception (with-worker pool identity)) "Generator failure should propagate"))))--;;; ------------------------------------------- Concurrent Access Tests ------------------------------------------------(deftest pool-handles-concurrent-access-safely-test- (testing "Pool handles concurrent access without race conditions"- (let [[pool created] (test-pool {:max-size 2})- num-threads 50- results (run-concurrent pool num-threads- (fn [i _worker]- (Thread/sleep (long (rand-int 20)))- i))]- (is (= num-threads (count results)) "All threads should complete successfully")- (is (<= @created 2) (str "Created " @created " workers, expected <= 2")))))--;;; ----------------------------------------- Version Parsing ---------------------------------------------------------(deftest expected-sqlglot-version-test(testing "Sqlglot version is properly parsed from pyproject.toml"(let [version (common/expected-sqlglot-version)](is (string? version))(is (re-matches #"\d+\.\d+\.\d+" version)))))--;;; -------------------------------------------- Pool Behavior Tests ---------------------------------------------------(deftest pool-stores-and-returns-objects-test- (testing "Pool correctly stores and returns arbitrary objects"- (let [[pool created] (test-pool)- result1 (with-worker pool identity)- result2 (with-worker pool identity)]- (is (map? result1))- (is (= 1 (:worker-id result1)))- (is (= 1 @created))- (is (map? result2))- (is (= 1 (:worker-id result2))) ;; Same worker ID- (is (= 1 @created)))))diff --git a/test/metabase/util/pool_test.clj b/test/metabase/util/pool_test.cljnew file mode 100644index 00000000000..f57cfde241a--- /dev/null+++ b/test/metabase/util/pool_test.clj@@ -0,0 +1,143 @@+(ns metabase.util.pool-test+ (:require+ [clojure.test :refer :all]+ [metabase.util.pool :as u.pool])+ (:import+ (io.aleph.dirigiste Pool)+ (java.util.concurrent CountDownLatch)))++(set! *warn-on-reflection* true)++(defn- mock-worker+ "Create a mock 'worker' object. The pool doesn't care what type the objects are - it just stores and+ returns them."+ [id]+ {:worker-id id+ :created-at (System/currentTimeMillis)})++(defn- counting-generator+ "Returns a generator function that counts how many workers it has created."+ [counter-atom]+ (fn []+ (mock-worker (swap! counter-atom inc))))++(defn- failing-generator+ "Returns a generator that fails after creating n workers."+ [counter-atom fail-after]+ (fn []+ (let [id (swap! counter-atom inc)]+ (when (> id fail-after)+ (throw (ex-info "Generator limit reached" {:count id :limit fail-after})))+ (mock-worker id))))++;;; ------------------------------------------------- Test Helpers ---------------------------------------------------++(defn- test-pool+ "Create a pool for testing with sensible defaults.+ Returns [pool created-count-atom destroyed-atom]."+ ([]+ (test-pool {}))+ ([config]+ (let [created (atom 0)+ destroyed (atom [])+ pool (u.pool/create-pool (counting-generator created)+ (fn [worker] (swap! destroyed conj worker))+ (merge {:max-size 2, :idle-minutes 1} config))]+ [pool created destroyed])))++(defn- with-worker+ "Execute f with a worker from pool, releasing it afterwards."+ [^Pool pool f]+ (let [worker (.acquire pool :test)]+ (try+ (f worker)+ (finally+ (.release pool :test worker)))))++(defn- run-concurrent+ "Run n threads concurrently, each executing f with a worker from pool.+ Returns vector of results."+ [pool n f]+ (let [latch (CountDownLatch. n)+ results (atom [])+ futures (doall+ (for [i (range n)]+ (future+ (.countDown latch)+ (.await latch)+ (let [result (with-worker pool (fn [worker] (f i worker)))]+ (swap! results conj result)))))]+ (doseq [fut futures] @fut)+ @results))++;;; ------------------------------------------------ Pool Size Tests -------------------------------------------------++(deftest pool-respects-maximum-size-test+ (testing "Pool never creates more than max workers even under concurrent load"+ (let [[pool created] (test-pool {:max-size 2})+ num-threads 20+ results (run-concurrent pool num-threads+ (fn [_i worker]+ (Thread/sleep (long (+ 10 (rand-int 50))))+ worker))]+ (is (<= @created 2)+ (str "Pool created " @created " workers but max is 2"))+ (is (= num-threads (count results))+ "All threads should have received a worker"))))++(deftest pool-reuses-idle-workers-test+ (testing "Sequential uses reuse the same worker instead of creating new ones"+ (let [[pool created] (test-pool)]+ (with-worker pool (fn [_worker] (is (= 1 @created) "First use creates one worker")))+ (with-worker pool (fn [worker]+ (is (= 1 @created) "Second use reuses the same worker")+ (is (= 1 (:worker-id worker))))))))++;;; ------------------------------------------------ Disposal Tests --------------------------------------------------++(deftest pool-dispose-destroys-and-replaces-workers-test+ (testing "Disposing a worker calls destroy and the next acquire generates a fresh one"+ (let [[pool created destroyed] (test-pool)+ worker (.acquire ^Pool pool :test)]+ (.dispose ^Pool pool :test worker)+ (is (= [worker] @destroyed) "Disposed worker should be destroyed")+ (with-worker pool (fn [_worker] (is (= 2 @created) "Disposed worker should be replaced"))))))++;;; --------------------------------------------- Generator Failure Tests --------------------------------------------++(deftest pool-handles-generator-failures-test+ (testing "Pool gracefully handles when generator fails"+ (let [counter (atom 0)+ pool (u.pool/create-pool (failing-generator counter 1) (fn [_worker]) {:max-size 2, :idle-minutes 1})]+ (is (some? (with-worker pool identity)) "First worker creation succeeds")+ ;; Dispose to force new creation+ (let [worker (.acquire ^Pool pool :test)]+ (.dispose ^Pool pool :test worker))+ (is (thrown? Exception (with-worker pool identity)) "Generator failure should propagate"))))++;;; ------------------------------------------- Concurrent Access Tests ----------------------------------------------++(deftest pool-handles-concurrent-access-safely-test+ (testing "Pool handles concurrent access without race conditions"+ (let [[pool created] (test-pool {:max-size 2})+ num-threads 50+ results (run-concurrent pool num-threads+ (fn [i _worker]+ (Thread/sleep (long (rand-int 20)))+ i))]+ (is (= num-threads (count results)) "All threads should complete successfully")+ (is (<= @created 2) (str "Created " @created " workers, expected <= 2")))))++;;; -------------------------------------------- Pool Behavior Tests -------------------------------------------------++(deftest pool-stores-and-returns-objects-test+ (testing "Pool correctly stores and returns arbitrary objects"+ (let [[pool created] (test-pool)+ result1 (with-worker pool identity)+ result2 (with-worker pool identity)]+ (is (map? result1))+ (is (= 1 (:worker-id result1)))+ (is (= 1 @created))+ (is (map? result2))+ (is (= 1 (:worker-id result2))) ;; Same worker ID+ (is (= 1 @created)))))