Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Serialize static-viz GraalJS context creation (GHY-4077) (#77225)

dpsuttonc09cdbf49cea
diff --git a/src/metabase/channel/render/js/pool.clj b/src/metabase/channel/render/js/pool.clj
index aaa35056b93..5a2d074afff 100644
--- a/src/metabase/channel/render/js/pool.clj
+++ b/src/metabase/channel/render/js/pool.clj
@@ -3,11 +3,29 @@
(`metabase.channel.render.js.svg`) and table-cell color selection (`metabase.channel.render.js.color`). Every
pooled context has the static-viz bundle, the static-viz interface, and the color-selector sources loaded.
- All contexts share the process-lifetime engine and evaluate the same `Source` instances (one `defonce` per
- resource), so the engine's code cache — which is keyed on the `Source` instance — holds ONE parsed copy of the
- static-viz bundle (~130 MB). Building a fresh `Source` per context (the previous behavior) made the engine retain a
- separate parsed copy for every pool generation, and made each generation reparse the 27 MB bundle: a
- multi-hundred-MB allocation spike every context recycle.
+ ### Memory model (GraalJS 25, interpreter mode — measured, see GHY-4077)
+
+ - The engine plus the parsed sources cost ~220 MB retained, steady, paid once per process. All contexts share the
+ process-lifetime engine and evaluate the same `Source` instances (one `defonce` per resource), so the engine's
+ code cache — which is keyed on the `Source` instance — holds ONE parsed copy of the static-viz bundle. Building a
+ fresh `Source` per context (the previous behavior) made the engine retain a separate parsed copy for every pool
+ generation, and made each generation reparse the 27 MB bundle: a multi-hundred-MB allocation spike every context
+ recycle.
+ - Evaluating the sources in a fresh context (realm setup against the warm code cache) churns through ~500 MB of
+ allocation in ~1 s. That is collectable garbage, not footprint: each extra live context retains only ~16 MB. The
+ very first context in the JVM additionally pays the cold parse — ~1.2 GB / ~5 s.
+ - A render itself allocates on the order of ~1 MB per row of chart data, also transient.
+
+ ### Why context creation is serialized
+
+ Creation being ~500 MB of churn a pop, letting the pool grow 1→3 concurrently during a render burst stacks
+ ~1–1.5 GB of transient allocation on top of the in-flight renders — on a 2 GB heap (the Cloud default) that
+ outruns the collector and drives GC thrash / OOM. So creation takes a private monitor
+ (see [[context-creation-lock]]). A time-based throttle (\"no two creations within X seconds\") is the wrong shape:
+ a burst of concurrent renders genuinely needs multiple contexts, so we let the pool grow to meet demand — it just
+ warms one context per ~1 s instead of all at once, bounding the worst case to a single creation's churn. Waiting
+ acquires simply queue on the pool. Pooled contexts expire after ~10 minutes with jitter so members created
+ together don't all regenerate in the same window.
Consumers borrow a context within [[with-static-viz-context]] and return plain data — never the context or a
context-bound `Value`. Never acquire a context while already holding one: with a small pool, nested acquires can
@@ -63,18 +81,41 @@
"Dirigiste pools are keyed. The key itself is arbitrary — it just has to be the same for every pool operation."
:engines)
+(def ^:private context-creation-lock
+ "Monitor serializing pooled context creation. Even with the shared-`Source` code cache, evaluating the bundle in a
+ fresh context allocates ~500 MB, so letting the pool grow 1→3 concurrently during a render burst stacks ~1–1.5 GB
+ of transient allocation on top of the in-flight renders — enough to blow a 2 GB heap. Holding creations to one at a
+ time bounds the worst case to a single creation (~1 s each); waiting acquires just queue on the pool meanwhile.
+ Nothing else may lock this monitor."
+ (Object.))
+
+(defn- expiry-timestamp
+ "Expiry timestamp for a pooled context: now + 10 minutes ± up to 3 minutes of jitter. The jitter keeps pool members
+ created in the same burst from all expiring — and serially regenerating — in the same window."
+ ^long []
+ (+ (System/nanoTime)
+ (.toNanos TimeUnit/MINUTES 7)
+ (long (rand (.toNanos TimeUnit/MINUTES 6)))))
+
+(defn- generate-pool-entry
+ "Create a `[context expiry-timestamp]` tuple for the pool. At most one context creation runs at a time (see
+ [[context-creation-lock]]); acquisition/use is already exclusive per context, so this only bounds pool *growth*."
+ []
+ (locking context-creation-lock
+ [(create-static-viz-context) (expiry-timestamp)]))
+
(def ^:private ^Pool static-viz-context-pool
"Pool of Truffle JS contexts. They are not thread-safe, so access is exclusive from acquire to release. Generating a
context is cheap — realm setup plus top-level eval against the engine's code cache, no reparse — and per-context
memory is a realm, not a full parsed copy of the bundle. The pool targets 100% utilization with a maximum of 3
contexts (to bound memory; renders hold a context exclusively, so that is also the render concurrency), but at
- least 1 context is always kept in the pool to pick up. Each pooled tuple carries an expiry timestamp so a context
- is recycled after 10 minutes regardless, bounding per-context leak accumulation."
+ least 1 context is always kept in the pool to pick up. Each pooled tuple carries a (jittered, ~10-minute) expiry
+ timestamp so a context is recycled regardless, bounding per-context leak accumulation. Context creation is
+ serialized (see [[context-creation-lock]]) so pool growth never runs multiple ~500 MB creations concurrently."
(let [base-controller (Pools/utilizationController 1.0 3 3)]
(Pool. (reify IPool$Generator
(generate [_ _]
- ;; Generate a tuple of the context and the expiry timestamp.
- [(create-static-viz-context) (+ (System/nanoTime) (.toNanos TimeUnit/MINUTES 10))])
+ (generate-pool-entry))
(destroy [_ _ [^Context ctx _expiry]]
;; Close the context when it's disposed from the pool (expiry/idle shrink/shutdown). Without this, each
;; disposed static-viz context leaks its memory: GraalVM only releases it on `close`, not on GC.
diff --git a/test/metabase/channel/render/js/pool_test.clj b/test/metabase/channel/render/js/pool_test.clj
new file mode 100644
index 00000000000..95f7f3a6500
--- /dev/null
+++ b/test/metabase/channel/render/js/pool_test.clj
@@ -0,0 +1,37 @@
+(ns metabase.channel.render.js.pool-test
+ (:require
+ [clojure.test :refer :all]
+ [metabase.channel.render.js.pool :as js.pool])
+ (:import
+ (java.util.concurrent TimeUnit)))
+
+(set! *warn-on-reflection* true)
+
+(deftest generate-pool-entry-serializes-context-creation-test
+ (testing "no two static-viz context creations run concurrently (#GHY-4077)"
+ (let [in-flight (atom 0)
+ max-in-flight (atom 0)
+ fake-context (Object.)]
+ (with-redefs [js.pool/create-static-viz-context
+ (fn []
+ (let [n (swap! in-flight inc)]
+ (swap! max-in-flight max n)
+ (Thread/sleep 50)
+ (swap! in-flight dec)
+ fake-context))]
+ (let [entries (->> (repeatedly 4 #(future (#'js.pool/generate-pool-entry)))
+ doall
+ (mapv deref))]
+ (is (= 1 @max-in-flight))
+ (testing "every entry is a [context expiry-timestamp] tuple"
+ (doseq [[ctx expiry-ts] entries]
+ (is (identical? fake-context ctx))
+ (is (int? expiry-ts)))))))))
+
+(deftest expiry-timestamp-jitter-test
+ (testing "expiry is ~10 minutes out with ±3 minutes of jitter"
+ (let [now (System/nanoTime)
+ expiries (repeatedly 100 #(#'js.pool/expiry-timestamp))]
+ (doseq [expiry expiries
+ :let [delta (- expiry now)]]
+ (is (<= (.toNanos TimeUnit/MINUTES 7) delta (.toNanos TimeUnit/MINUTES 14)))))))