The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
[o11y] Log + metrics for `lib.convert/clean` making material changes (#67094)
Braden Shepherdson7d84882a3acb
diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.ednindex 232a7ea7bc3..ab2ecd65629 100644--- a/.clj-kondo/config/modules/config.edn+++ b/.clj-kondo/config/modules/config.edn@@ -702,6 +702,7 @@metabase.lib.binning.utilmetabase.lib.cardmetabase.lib.computed ; temporarily so that `qp.compile` can use it as lib memoization evolves+ metabase.lib.convert ; Only so [[metabase.queries.models.card]] can set dynamic vars.metabase.lib.coremetabase.lib.equalitymetabase.lib.fielddiff --git a/src/metabase/analytics/prometheus.clj b/src/metabase/analytics/prometheus.cljindex f33be5de57b..6ad8ae8798e 100644--- a/src/metabase/analytics/prometheus.clj+++ b/src/metabase/analytics/prometheus.clj@@ -253,6 +253,8 @@{:description "Number of queries cached in lib.computed/weak-map."})(prometheus/gauge :metabase-card/unique-cards-failed-conversion{:description "Number of distinct cards which have :dataset_query {}, meaning MBQL 4 to 5 conversion failed."})+ (prometheus/counter :metabase-card/conversions-requiring-cleaning+ {:description "Number of times this instance converted a card's MBQL 4 to 5 and `clean` made a real change"})(prometheus/gauge :metabase-database/status{:description "Does a given database using driver pass a health check.":labels [:driver :healthy :reason]})diff --git a/src/metabase/lib/convert.cljc b/src/metabase/lib/convert.cljcindex 0be65074243..e86ec788872 100644--- a/src/metabase/lib/convert.cljc+++ b/src/metabase/lib/convert.cljc@@ -100,6 +100,15 @@When converting queries at later stages of the preprocessing pipeline, this cleaning might not be desirable."true)+#?(:clj+ (def ^:dynamic *card-clean-hook*+ "Set by [[metabase.lib-be.models.transforms]] to a function which expects to be called like+ `(f pre-cleaning-query post-cleaning-query)`, when [[clean]] makes material changes.++ [[clean]] is expected to be a no-op in general and should not be removing clauses when a query is converted from+ MBQL 4 to 5 on being read from AppDB."+ nil))+(defn without-cleaning"Runs the provided function with cleaning of queries disabled.@@ -111,15 +120,19 @@(defn- clean [almost-query](if-not *clean-query*almost-query- (loop [almost-query almost-query- stage-index 0]- (let [current-stage (nth (:stages almost-query) stage-index)- new-stage (clean-stage current-stage)]- (if (= current-stage new-stage)- (if (= stage-index (dec (count (:stages almost-query))))- almost-query- (recur almost-query (inc stage-index)))- (recur (update almost-query :stages assoc stage-index new-stage) stage-index))))))+ (let [cleaned (loop [almost-query almost-query+ stage-index 0]+ (let [current-stage (nth (:stages almost-query) stage-index)+ new-stage (clean-stage current-stage)]+ (if (= current-stage new-stage)+ (if (= stage-index (dec (count (:stages almost-query))))+ almost-query+ (recur almost-query (inc stage-index)))+ (recur (update almost-query :stages assoc stage-index new-stage) stage-index))))]+ #?(:clj+ (when (and *card-clean-hook* (not= almost-query cleaned))+ (*card-clean-hook* almost-query cleaned)))+ cleaned)))(defmulti ->pMBQL"Coerce something to pMBQL (the version of MBQL manipulated by Metabase Lib v2) if it's not already pMBQL."diff --git a/src/metabase/queries/models/card.clj b/src/metabase/queries/models/card.cljindex 7b02b723907..713ecd6e35b 100644--- a/src/metabase/queries/models/card.clj+++ b/src/metabase/queries/models/card.clj@@ -17,6 +17,7 @@[metabase.dashboards.autoplace :as autoplace][metabase.events.core :as events][metabase.lib-be.core :as lib-be]+ [metabase.lib.convert :as lib.convert][metabase.lib.core :as lib][metabase.lib.metadata.protocols :as lib.metadata.protocols][metabase.lib.normalize :as lib.normalize]@@ -51,6 +52,7 @@[metabase.warehouse-schema.models.field-values :as field-values][methodical.core :as methodical][toucan2.core :as t2]+ [toucan2.pipeline :as t2.pipeline][toucan2.tools.hydrate :as t2.hydrate]))(set! *warn-on-reflection* true)@@ -718,12 +720,37 @@Always returns `card`."[card](when (= (:dataset_query card) {})- (log/infof "Card %d has a blank :dataset_query - this indicates a Metabase issue" (:id card))+ (log/infof "Card %d has a blank :dataset_query - this indicates a Metabase issue. Legacy MBQL: %s"+ (:id card) (:legacy_query card))(let [uniques (swap! unique-cards-with-blank-dataset-query conj (:id card))](analytics/set! :metabase-card/unique-cards-failed-conversion (count uniques))));; Always returns the original card.card)+(defn- mbql5-conversion-clean-callback [untransformed-card pre-cleaning-query post-cleaning-query]+ (analytics/inc! :metabase-card/conversions-requiring-cleaning)+ (log/infof "MBQL 4->5 conversion for Card %d had real 'clean' changes from %s into %s; :legacy_query is %s"+ (:id untransformed-card)+ (pr-str (dissoc pre-cleaning-query :lib/metadata))+ (pr-str (dissoc post-cleaning-query :lib/metadata))+ (:legacy_query untransformed-card)))++;; Dynamically binds [[lib.convert/*card-clean-hook*]] to a function that logs the impact of cleaning,+;; during `transform-out`, so that we can log whenever a card gets converted and [[lib.convert/clean]] makes material+;; changes, which likely indicates a bug.+(methodical/defmethod t2.pipeline/results-transform [:toucan.result-type/instances :model/Card]+ [query-type model]+ (let [xform (next-method query-type model)]+ (fn xform' [rf]+ (let [rf' (xform rf)]+ (fn rf''+ ([] (rf'))+ ([acc]+ (rf' acc))+ ([acc card]+ (binding [lib.convert/*card-clean-hook* (partial mbql5-conversion-clean-callback card)]+ (rf' acc card))))))))+(t2/define-after-select :model/Card[card];; +===============================================================================================+