The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
[QP] Properly escape SQL `LIKE` metachars for `:contains` and friends (#67665)
Braden Shepherdson3fcac725b21d
diff --git a/docs/developers-guide/driver-changelog.md b/docs/developers-guide/driver-changelog.mdindex e21324c7c7f..bd6856199cb 100644--- a/docs/developers-guide/driver-changelog.md+++ b/docs/developers-guide/driver-changelog.md@@ -18,6 +18,17 @@ title: Driver interface changelog- `grant-workspace-read-access!` - Grant read access on specified tables to a workspace's isolated user.- `check-isolation-permissions` - Test whether the database connection has sufficient permissions.+- Added support for escaping `LIKE` metacharacters to `:sql` driver's handling of `LIKE` clauses, which are used to+ implement the `:starts-with`, `:ends-with` and `:contains` filters. The default implementation uses backslashes to+ escape backslashes, `%` and `_`, the `LIKE` metacharacters, when the RHS of one of these filters is a literal string.+ Drivers which can handle `x LIKE y ESCAPE '\'` should just work. If a different way of escaping is needed, you can+ override the new multimethod `metabase.driver.sql.query-processor/escape-like-pattern`; see `:sqlserver` which uses+ `[%]` regex character classes. For drivers which already have backslash as the `ESCAPE` default, or don't support+ that `ESCAPE` syntax, override the new multimethod `metabase.driver.sql.query-processor/transform-literal-like-pattern-honeysql`,+ or as a shortcut if the override is an identity function, add the abstract driver+ `:metabase.driver.sql.query-processor.like-escape-char-built-in/like-escape-char-built-in` as a parent of your driver.+ See `metabase.driver.mysql` for an example of using the abstract driver.+## Metabase 0.58.0- Added a `:collate` feature for drivers that support collation settings on text fieldsdiff --git a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.cljindex 6472c192838..47fe881028e 100644--- a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj+++ b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj@@ -17,6 +17,7 @@[metabase.driver.sql-jdbc.sync.describe-database :as sql-jdbc.describe-database][metabase.driver.sql.normalize :as driver.sql.normalize][metabase.driver.sql.query-processor :as sql.qp]+ [metabase.driver.sql.query-processor.like-escape-char-built-in :as-alias like-escape-char-built-in][metabase.driver.sql.util :as sql.u][metabase.driver.sync :as driver.s][metabase.driver.util :as driver.u]@@ -72,7 +73,8 @@(set! *warn-on-reflection* true)-(driver/register! :bigquery-cloud-sdk, :parent :sql)+(driver/register! :bigquery-cloud-sdk, :parent #{:sql+ ::like-escape-char-built-in/like-escape-char-built-in});;; +----------------------------------------------------------------------------------------------------------------+;;; | Client |diff --git a/modules/drivers/redshift/src/metabase/driver/redshift.clj b/modules/drivers/redshift/src/metabase/driver/redshift.cljindex ee62f01d805..d50103a0f84 100644--- a/modules/drivers/redshift/src/metabase/driver/redshift.clj+++ b/modules/drivers/redshift/src/metabase/driver/redshift.clj@@ -16,6 +16,7 @@[metabase.driver.sql-jdbc.sync :as sql-jdbc.sync][metabase.driver.sql-jdbc.sync.describe-database :as sql-jdbc.describe-database][metabase.driver.sql.query-processor :as sql.qp]+ [metabase.driver.sql.query-processor.like-escape-char-built-in :as-alias like-escape-char-built-in][metabase.driver.sync :as driver.s][metabase.util :as u][metabase.util.date-2 :as u.date]@@ -35,7 +36,8 @@(set! *warn-on-reflection* true)-(driver/register! :redshift, :parent #{:postgres})+(driver/register! :redshift, :parent #{:postgres+ ::like-escape-char-built-in/like-escape-char-built-in})(doseq [[feature supported?] {:atomic-renames true:connection-impersonation truediff --git a/modules/drivers/sparksql/src/metabase/driver/sparksql.clj b/modules/drivers/sparksql/src/metabase/driver/sparksql.cljindex 47365478118..848ec408398 100644--- a/modules/drivers/sparksql/src/metabase/driver/sparksql.clj+++ b/modules/drivers/sparksql/src/metabase/driver/sparksql.clj@@ -16,6 +16,7 @@[metabase.driver.sql-jdbc.sync :as sql-jdbc.sync][metabase.driver.sql.parameters.substitution :as sql.params.substitution][metabase.driver.sql.query-processor :as sql.qp]+ [metabase.driver.sql.query-processor.like-escape-char-built-in :as-alias sql.qp.like-built-in][metabase.driver.sql.util :as sql.u][metabase.util.honey-sql-2 :as h2x][metabase.util.performance :refer [select-keys every? empty? not-empty get-in]])@@ -24,7 +25,7 @@(set! *warn-on-reflection* true)-(driver/register! :sparksql, :parent :hive-like)+(driver/register! :sparksql, :parent #{:hive-like ::sql.qp.like-built-in/like-escape-char-built-in});;; ------------------------------------------ Custom HoneySQL Clause Impls ------------------------------------------diff --git a/modules/drivers/sqlserver/src/metabase/driver/sqlserver.clj b/modules/drivers/sqlserver/src/metabase/driver/sqlserver.cljindex 99210c54bd3..c730dbe9165 100644--- a/modules/drivers/sqlserver/src/metabase/driver/sqlserver.clj+++ b/modules/drivers/sqlserver/src/metabase/driver/sqlserver.clj@@ -20,6 +20,7 @@[metabase.driver.sql.parameters.substitution :as sql.params.substitution][metabase.driver.sql.query-processor :as sql.qp][metabase.driver.sql.query-processor.boolean-to-comparison :as sql.qp.boolean-to-comparison]+ [metabase.driver.sql.query-processor.like-escape-char-built-in :as like-escape-char-built-in][metabase.driver.sql.query-processor.util :as sql.qp.u][metabase.driver.sql.util :as sql.u][metabase.driver.util :as driver.u]@@ -43,7 +44,7 @@(set! *warn-on-reflection* true)-(driver/register! :sqlserver, :parent #{:sql-jdbc})+(driver/register! :sqlserver, :parent #{:sql-jdbc ::like-escape-char-built-in/like-escape-char-built-in})(doseq [[feature supported?] {:case-sensitivity-string-filter-options false:connection-impersonation true@@ -633,6 +634,15 @@(->> (update query :filter sql.qp.boolean-to-comparison/boolean->comparison)(parent-method driver :filter honeysql-form))))+;; SQL Server doesn't like backslashes as the escape character for `LIKE` clauses. Use character classes instead to+;; escape the `LIKE` metacharacters `%` and `_`.+(defmethod sql.qp/escape-like-pattern :sqlserver+ [_driver like-pattern]+ (-> like-pattern+ (str/replace "\\" "[\\]")+ (str/replace "%" "[%]")+ (str/replace "_" "[_]")))+;; SQLServer doesn't support `TRUE`/`FALSE`; it uses `1`/`0`, respectively; convert these booleans to numbers.(defmethod sql.qp/->honeysql [:sqlserver Boolean][_ bool]diff --git a/src/metabase/driver/h2.clj b/src/metabase/driver/h2.cljindex 41b8c194a25..08ea620ca1f 100644--- a/src/metabase/driver/h2.clj+++ b/src/metabase/driver/h2.clj@@ -19,6 +19,7 @@[metabase.driver.sql-jdbc.sync :as sql-jdbc.sync][metabase.driver.sql.normalize :as sql.normalize][metabase.driver.sql.query-processor :as sql.qp]+ [metabase.driver.sql.query-processor.like-escape-char-built-in :as like-escape-char-built-in][metabase.driver.util :as driver.u][metabase.lib.schema.id :as lib.schema.id][metabase.lib.schema.metadata :as lib.schema.metadata]@@ -39,7 +40,7 @@;; method impls live in this namespace(comment h2.actions/keep-me)-(driver/register! :h2, :parent :sql-jdbc)+(driver/register! :h2, :parent #{:sql-jdbc ::like-escape-char-built-in/like-escape-char-built-in});;; this will prevent the H2 driver from showing up in the list of options when adding a new Database.(defmethod driver/superseded-by :h2 [_driver] :deprecated)diff --git a/src/metabase/driver/mysql.clj b/src/metabase/driver/mysql.cljindex ba9e27f05d1..b0a69e73c15 100644--- a/src/metabase/driver/mysql.clj+++ b/src/metabase/driver/mysql.clj@@ -22,6 +22,7 @@[metabase.driver.sql-jdbc.quoting :refer [quote-columns]][metabase.driver.sql-jdbc.sync :as sql-jdbc.sync][metabase.driver.sql.query-processor :as sql.qp]+ [metabase.driver.sql.query-processor.like-escape-char-built-in :as like-escape-char-built-in][metabase.driver.sql.query-processor.util :as sql.qp.u][metabase.driver.sql.util :as sql.u][metabase.driver.util :as driver.u]@@ -56,7 +57,7 @@mysql.actions/keep-memysql.ddl/keep-me)-(driver/register! :mysql, :parent :sql-jdbc)+(driver/register! :mysql, :parent #{:sql-jdbc ::like-escape-char-built-in/like-escape-char-built-in})(def ^:private ^:const min-supported-mysql-version 5.7)(def ^:private ^:const min-supported-mariadb-version 10.2)diff --git a/src/metabase/driver/sql/query_processor.clj b/src/metabase/driver/sql/query_processor.cljindex 233b4d09a84..28eb1dbb9c2 100644--- a/src/metabase/driver/sql/query_processor.clj+++ b/src/metabase/driver/sql/query_processor.clj@@ -1552,6 +1552,42 @@[:fn {:error/message "string value"} #(string? (second %))]]driver-api/mbql.schema.FieldOrExpressionDef])+(defmulti escape-like-pattern+ "Handle escaping a literal string into a `LIKE` clause pattern which will match literally.++ In particular, any `%` or `_` (the standard metacharacters) should be escaped. In some engines, that's done by a+ preceding backslash `\\_`; in others by wrapping it in a character class `[%]`.++ The default implementation for `:sql` uses backslashes, which appears to be the most common approach. Most SQL-based+ drivers should not need to implement this."+ {:added "0.59.0" :arglists '([driver like-pattern])}+ driver/dispatch-on-initialized-driver+ :hierarchy #'driver/hierarchy)++(defmethod escape-like-pattern :sql [_driver like-pattern]+ (-> like-pattern+ (str/replace "\\" "\\\\")+ (str/replace "_" "\\_")+ (str/replace "%" "\\%")))++(defmulti transform-literal-like-pattern-honeysql+ "When the RHS of a `LIKE` clause is a literal pattern, some drivers require extra HoneySQL clauses, such as+ `:escape`, added to the expression.++ This driver multimethod allows SQL drivers to adjust these clauses for any MBQL expression that uses `LIKE` under+ the hood (eg. `:starts-with`, `:contains`, `:ends-with`), with a single, simpler override.++ The default is to add `ESCAPE '\\'`, adopting a common default. This is compatible with the default+ [[escape-like-pattern]]. Drivers which have an `ESCAPE` character built in can skip editing this clause by inheriting+ from the `:metabase.driver.sql.query-processor.like-escape-char-built-in/like-escape-char-built-in` abstract driver."+ {:added "0.59.0" :arglists '([driver like-rhs-honeysql])}+ driver/dispatch-on-initialized-driver+ :hierarchy #'driver/hierarchy)++(defmethod transform-literal-like-pattern-honeysql :sql+ [_driver like-rhs-honeysql]+ [:escape like-rhs-honeysql [:inline "\\"]])+(mu/defn- generate-pattern"Generate pattern to match against in like clause. Lowercasing for case insensitive matching also happens here."[driver@@ -1560,8 +1596,10 @@post{:keys [case-sensitive] :or {case-sensitive true} :as _options}](if (= :value type)- (->honeysql driver (update arg 1 #(cond-> (str pre % post)- (not case-sensitive) u/lower-case-en)))+ (->> (update arg 1 #(cond-> (str pre (escape-like-pattern driver %) post)+ (not case-sensitive) u/lower-case-en))+ (->honeysql driver)+ (transform-literal-like-pattern-honeysql driver))(let [expr (->honeysql driver (into [:concat] (remove nil?) [pre arg post]))](if case-sensitiveexprdiff --git a/src/metabase/driver/sql/query_processor/like_escape_char_built_in.clj b/src/metabase/driver/sql/query_processor/like_escape_char_built_in.cljnew file mode 100644index 00000000000..b1f7c558f99--- /dev/null+++ b/src/metabase/driver/sql/query_processor/like_escape_char_built_in.clj@@ -0,0 +1,19 @@+(ns metabase.driver.sql.query-processor.like-escape-char-built-in+ "In MySQL and some other databases, `LIKE` clause RHS patterns have an escape character defined by default and have+ awkward double-escaping issues with setting a `LIKE pattern ESCAPE '\\'` to the same character.++ This abstract driver overrides [[sql.qp/transform-literal-like-pattern-honeysql]] to an identity function, overriding+ the `:default` behaviour of specifying the `ESCAPE '\\'` clause as the SQL standard suggests.++ See #67667 for more context."+ (:require+ [metabase.driver :as driver]+ [metabase.driver.sql.query-processor :as sql.qp]))++(driver/register! ::like-escape-char-built-in, :abstract? true)++(defmethod sql.qp/transform-literal-like-pattern-honeysql ::like-escape-char-built-in+ [_driver like-rhs-honeysql]+ like-rhs-honeysql)++(prefer-method sql.qp/transform-literal-like-pattern-honeysql ::like-escape-char-built-in :sql)diff --git a/test/metabase/query_processor/filter_test.clj b/test/metabase/query_processor/filter_test.cljindex f58a48b6f24..51c5886f777 100644--- a/test/metabase/query_processor/filter_test.clj+++ b/test/metabase/query_processor/filter_test.clj@@ -395,6 +395,52 @@:order-by [[:asc $id]]:limit 3})))))))+(mt/defdataset strings-that-need-escaping+ [["escaping_table"+ [{:field-name "string1", :base-type :type/Text}]+ [["\\Backslash"]+ ["Back\\slash"]+ ["Backslash\\"]+ ["_Underscore"]+ ["Under_score"]+ ["Underscore_"]+ ["%ile"]+ ["per%cent"]+ ["100%"]+ ["_many%\\escapes"]+ ["more%___%esca\\pes%"]+ ["%backslash\\%before_percent_"]+ ["\\backslash\\_before%underscore\\"]]]])++;; TODO: (Braden 2026-01-08) This set can be expanded further!+;; - :athena, :vertica, :clickhouse, :presto all support these clauses but have issues loading the test dataset.+;; - :mongo supports `:starts-with` etc. but is not SQL; its regex-based fuzzy matching could be upgraded.+(doseq [driver #{:h2 :sqlserver :mysql :maria :postgres :oracle :sqlite}]+ (defmethod driver/database-supports? [driver :test/escaping-like-metacharacters]+ [_driver _feature _database]+ true))++(defmethod driver/database-supports? [:default :test/escaping-like-metacharacters]+ [_driver _feature _database]+ false)++(deftest ^:parallel starts-with-escaped-char-in-literal-test+ (mt/test-drivers (mt/normal-drivers-with-feature :test/escaping-like-metacharacters)+ (mt/dataset strings-that-need-escaping+ (doseq [[lit exp] {"\\" [[1 "\\Backslash"]+ [13 "\\backslash\\_before%underscore\\"]]+ "_" [[4 "_Underscore"]+ [10 "_many%\\escapes"]]+ "%" [[7 "%ile"]+ [12 "%backslash\\%before_percent_"]]}]+ (testing (str "literal argument starts with " lit)+ (is (=? exp+ (mt/formatted-rows+ [int identity]+ (mt/run-mbql-query escaping_table+ {:filter [:starts-with $string1 lit]+ :limit 3})))))))))+;;; --------------------------------------------------- ends-with ----------------------------------------------------(deftest ^:parallel ends-with-test@@ -473,6 +519,23 @@:order-by [[:asc $id]]:limit 3})))))))+(deftest ^:parallel ends-with-escaped-char-in-literal-test+ (mt/test-drivers (mt/normal-drivers-with-feature :test/escaping-like-metacharacters)+ (mt/dataset strings-that-need-escaping+ (doseq [[lit exp] {"\\" [[3 "Backslash\\"]+ [13 "\\backslash\\_before%underscore\\"]]+ "_" [[6 "Underscore_"]+ [12 "%backslash\\%before_percent_"]]+ "%" [[9 "100%"]+ [11 "more%___%esca\\pes%"]]}]+ (testing (str "literal argument ends with " lit)+ (is (=? exp+ (mt/formatted-rows+ [int identity]+ (mt/run-mbql-query escaping_table+ {:filter [:ends-with $string1 lit]+ :limit 3})))))))))+;;; ---------------------------------------------------- contains ----------------------------------------------------(deftest ^:parallel contains-test@@ -547,6 +610,39 @@:order-by [[:asc $id]]:limit 3})))))))+(deftest ^:parallel contains-with-escaped-char-in-literal-test+ (mt/test-drivers (mt/normal-drivers-with-feature :test/escaping-like-metacharacters)+ (mt/dataset strings-that-need-escaping+ (doseq [[lit exp] {"\\" [[1 "\\Backslash"]+ [2 "Back\\slash"]+ [3 "Backslash\\"]+ [10 "_many%\\escapes"]+ [11 "more%___%esca\\pes%"]+ [12 "%backslash\\%before_percent_"]+ [13 "\\backslash\\_before%underscore\\"]]+ "_" [[4 "_Underscore"]+ [5 "Under_score"]+ [6 "Underscore_"]+ [10 "_many%\\escapes"]+ [11 "more%___%esca\\pes%"]+ [12 "%backslash\\%before_percent_"]+ [13 "\\backslash\\_before%underscore\\"]]+ "%" [[7 "%ile"]+ [8 "per%cent"]+ [9 "100%"]+ [10 "_many%\\escapes"]+ [11 "more%___%esca\\pes%"]+ [12 "%backslash\\%before_percent_"]+ [13 "\\backslash\\_before%underscore\\"]]}]+ (testing (str "literal argument contains " lit)+ (is (=? exp+ (mt/formatted-rows+ [int identity]+ (mt/run-mbql-query escaping_table+ {:filter [:contains $string1 lit]+ :order-by [[:asc $id]]+ :limit 8})))))))))+;;; +----------------------------------------------------------------------------------------------------------------+;;; | NESTED AND/OR CLAUSES |;;; +----------------------------------------------------------------------------------------------------------------+