Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Don't GZIP sync-flush per object when serializing QP cache entries (#77786)

Alexander Polyankin1a8a17229a00
diff --git a/src/metabase/query_processor/middleware/cache/impl.clj b/src/metabase/query_processor/middleware/cache/impl.clj
index 170ee940286..60f2d4ab7ce 100644
--- a/src/metabase/query_processor/middleware/cache/impl.clj
+++ b/src/metabase/query_processor/middleware/cache/impl.clj
@@ -46,8 +46,7 @@
(defn- freeze!
[^OutputStream os obj]
(log/tracef "Freezing %s" (pr-str obj))
- (nippy/freeze-to-out! os obj)
- (.flush os))
+ (nippy/freeze-to-out! os obj))
(defn do-with-serialization
"Create output streams for serializing QP results and invoke `f`, a function of the form
@@ -58,9 +57,12 @@
will catch any exceptions thrown during serialization; these will be thrown later when invoking `result-fn`. After
the first exception `in-fn` will no-op for all subsequent calls.
- When you have serialized *all* objects, call `result-fn` to get the serialized byte array. If an error was
- encountered during serialization (such as the serialized bytes being longer than `max-bytes`), `result-fn` will
- throw an Exception rather than returning a byte array; be sure to handle this case.
+ When you have serialized *all* objects, call `result-fn` to finish the GZIP stream and get the serialized byte
+ array. If an error was encountered during serialization (such as the serialized bytes being longer than
+ `max-bytes`), `result-fn` will throw an Exception rather than returning a byte array; be sure to handle this case.
+
+ Compression is deliberately not sync-flushed per object: every sync flush is a native `Deflater` call in a JNI
+ critical region, which on JDK <= 21 (no G1 region pinning, see JEP 423) stalls GC via the GCLocker.
(do-with-serialization
(fn [in result]
@@ -74,7 +76,7 @@
(with-open [bos (ByteArrayOutputStream.)]
(let [os (-> (max-bytes-output-stream max-bytes bos)
BufferedOutputStream.
- (GZIPOutputStream. true)
+ GZIPOutputStream.
DataOutputStream.)
error (atom nil)]
(try
@@ -90,9 +92,11 @@
(when @error
(throw @error))
(log/trace "Getting result byte array")
+ ;; finish the stream (drain the deflater, write the GZIP trailer); may throw past the byte limit
+ (.close os)
(.toByteArray bos)))
;; this is done manually instead of `with-open` because it might throw an Exception when we close it if it's
- ;; past the byte limit; that's fine and we can ignore it
+ ;; past the byte limit; that's fine and we can ignore it. Re-closing after `result-fn` is a no-op.
(finally
(u/ignore-exceptions (.close os))))))))
diff --git a/src/metabase/query_processor/streaming/json.clj b/src/metabase/query_processor/streaming/json.clj
index 7d5feca1c43..251a0b349ab 100644
--- a/src/metabase/query_processor/streaming/json.clj
+++ b/src/metabase/query_processor/streaming/json.clj
@@ -80,8 +80,9 @@
(formatter/TextWrapper? res) (:text-str res)
:else res)))
@ordered-formatters cleaned-row)))
- writer {})
- (.flush writer))))
+ writer {}))))
+ ;; no flush per row: it would sync-flush the GZIP response per row (one native Deflater call each, TCP packet
+ ;; overhead per #34795); the BufferedWriter pushes rows through as it fills, like the CSV writer
(finish! [_ _]
(.write writer "\n]")