Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Tighten impersonated native-query validation (#84202)

John Swansonb91c7cad94e4
diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.edn
index 86c6cfe76c0..dd568acdcec 100644
--- a/.clj-kondo/config/modules/config.edn
+++ b/.clj-kondo/config/modules/config.edn
@@ -4247,6 +4247,7 @@
:uses #{api
audit-app
driver
+ lib
lib.metadata
lib.schema
models
diff --git a/docs/developers-guide/driver-changelog.md b/docs/developers-guide/driver-changelog.md
index 42e84131d43..454eb931fa6 100644
--- a/docs/developers-guide/driver-changelog.md
+++ b/docs/developers-guide/driver-changelog.md
@@ -11,6 +11,13 @@ title: Driver interface changelog
error codes or exclude SQLStates that also cover operational failures. Unknown failures remain QP errors;
cancellation takes precedence over query-error evidence.
+- `metabase.driver/validate-impersonated-query` `[driver query]` has moved from `:sql-jdbc` to `:sql`, so every
+ SQL driver -- not only the JDBC ones -- inherits the check that a connection-impersonated native query is a
+ single statement of the allowed type. It now walks native stages nested inside joins as well as the top-level
+ ones, and throws rather than returning a verdict when it cannot parse the exact string that would execute.
+ A driver that supports `:connection-impersonation` without descending from `:sql` must implement the method
+ itself; impersonated native queries on such a driver are refused outright.
+
## Metabase 0.65.0
- `metabase.driver.sql-jdbc.execute/set-role-if-supported!` moved to
diff --git a/docs/permissions/impersonation.md b/docs/permissions/impersonation.md
index c52b6c710aa..814bece705e 100644
--- a/docs/permissions/impersonation.md
+++ b/docs/permissions/impersonation.md
@@ -179,6 +179,12 @@ Examples of queries Metabase will reject:
- Statements that create or change tables, including temporary tables (`CREATE`, `ALTER`, `DROP`)
- Role-switching statements like `SET ROLE` or `USE ROLE`
+### Very large hand-written value lists are rejected
+
+To validate a native query safely, Metabase must parse the whole statement first. A query with a very large literal list written directly in the SQL—a `VALUES` clause, `IN` list, or `ARRAY` literal with more than 2,000 rows or items—can't be validated, so Metabase rejects it rather than run something it can't check. You'll see a message asking you to reduce the size of the list.
+
+This limit only applies to values you type into the SQL itself. [Filter widgets and field filters](../questions/native-editor/sql-parameters.md) pass their values to the database separately, so dashboard and question filters with long lists of values keep working. If you need to match against a large set of hand-written values, load them into a table and join to it instead of listing them inline.
+
## Metabase gives people the most permissive access to data across all of their groups
So if a person is in two groups with different permissions for the same database:
diff --git a/enterprise/backend/src/metabase_enterprise/impersonation/middleware.clj b/enterprise/backend/src/metabase_enterprise/impersonation/middleware.clj
index fbce2930e8b..90315e0191e 100644
--- a/enterprise/backend/src/metabase_enterprise/impersonation/middleware.clj
+++ b/enterprise/backend/src/metabase_enterprise/impersonation/middleware.clj
@@ -3,13 +3,41 @@
[metabase-enterprise.impersonation.driver :as impersonation.driver]
[metabase.api.common :as api]
[metabase.driver :as driver]
+ [metabase.driver.util :as driver.u]
+ [metabase.lib.core :as lib]
[metabase.lib.metadata :as lib.metadata]
[metabase.premium-features.core :as premium-features :refer [defenterprise]]
+ [metabase.query-processor.error-type :as qp.error-type]
[metabase.query-processor.interface :as qp.i]
;; legacy usage -- don't do things like this going forward
^{:clj-kondo/ignore [:deprecated-namespace :discouraged-namespace]} [metabase.query-processor.store :as qp.store]
[metabase.util.i18n :refer [tru]]))
+(defn- validates-native-queries?
+ "Does `driver` supply its own `driver/validate-impersonated-query`?"
+ [driver]
+ (not= (get-method driver/validate-impersonated-query driver)
+ (get-method driver/validate-impersonated-query :default)))
+
+(defn- assert-native-validation-supported
+ "Refuse a native query on a driver whose validation resolves to the `:default` no-op, rather than running
+ it unchecked. MBQL stages are compiler-generated, so they pass through — only native SQL is checked."
+ [driver query]
+ (when (and (lib/any-native-stage? query)
+ (not (validates-native-queries? driver)))
+ (throw (ex-info (tru "Connection impersonation does not support native queries on this database.")
+ {:type qp.error-type/invalid-query}))))
+
+(defn- validate-impersonated-query
+ "Enforce the single-statement rule on `query` for `driver`."
+ [driver database query]
+ (if (driver.u/supports? driver :connection-impersonation database)
+ (do (assert-native-validation-supported driver query)
+ (driver/validate-impersonated-query
+ driver
+ (cond-> query api/*is-superuser?* (assoc :impersonation/admin? true))))
+ query))
+
(defenterprise apply-impersonation
"Pre-processing middleware. Validates that native queries on impersonated databases are single SELECT statements,
and adds an impersonation role key to the query for non-admin users. Currently used solely for caching."
@@ -27,9 +55,7 @@
impersonation-enabled?
(as-> q
(do (premium-features/assert-has-feature :advanced-permissions (tru "Advanced Permissions"))
- (driver/validate-impersonated-query
- driver/*driver*
- (cond-> q api/*is-superuser?* (assoc :impersonation/admin? true)))))
+ (validate-impersonated-query driver/*driver* database q)))
;; Only assign the role for non-admin impersonated users
role
(assoc :impersonation/role role)))))
diff --git a/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj b/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj
index ac9bb6db366..e594f9da75d 100644
--- a/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj
@@ -6,6 +6,7 @@
[clojure.string :as str]
[clojure.test :refer :all]
[metabase-enterprise.impersonation.driver :as impersonation.driver]
+ [metabase-enterprise.impersonation.middleware :as impersonation.middleware]
[metabase-enterprise.impersonation.util-test :as impersonation.util-test]
[metabase-enterprise.test :as met]
[metabase.actions.execution :as actions.execution]
@@ -1155,7 +1156,7 @@
(with-impersonation-db!
(impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" role-a}}
- (testing "A single SELECT statement works with normal impersonation"
+ (testing "A single select statement works with normal impersonation"
(let [mp (mt/metadata-provider)
run-native-query (fn [table] (->> (-> (lib/query mp (lib.metadata/table mp (mt/id table)))
(lib/aggregate (lib/count)))
@@ -1195,7 +1196,7 @@
(impersonation.util-test/with-impersonations-for-user! :crowberto
{:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
:attributes {"impersonation_attr" role-a}}
- (testing "A single SELECT statement works for admin"
+ (testing "A single select statement works for admin"
(let [mp (mt/metadata-provider)
run-native-query (fn [table] (->> (-> (lib/query mp (lib.metadata/table mp (mt/id table)))
(lib/aggregate (lib/count)))
@@ -1335,8 +1336,7 @@
(let [validate (fn [sql]
(try
(driver.sql/validate-impersonated-query*
- :postgres
- {:stages [{:lib/type :mbql.stage/native :native sql}]})
+ :postgres (lib/native-query meta/metadata-provider sql))
nil
(catch clojure.lang.ExceptionInfo e e)))]
(testing "syntax error in a CTE produces a parse-error message, not the single-select message"
@@ -1362,7 +1362,7 @@
(deftest ^:parallel validate-impersonated-query-keys-on-allow-write-flag-test
(testing "validate-impersonated-query* derives read-vs-write from *impersonation-allow-write?*"
- (let [query (fn [sql] {:stages [{:lib/type :mbql.stage/native :native sql}]})
+ (let [query (fn [sql] (lib/native-query meta/metadata-provider sql))
outcome (fn [q]
(try
(driver.sql/validate-impersonated-query* :postgres q)
@@ -1381,7 +1381,7 @@
(deftest validate-impersonated-query-is-enforced-for-all-impersonation-drivers-test
(testing "every driver that supports connection-impersonation enforces the single-statement guard (SEC-1189)"
(mt/test-drivers (mt/normal-drivers-with-feature :connection-impersonation)
- (let [query (fn [sql] {:stages [{:lib/type :mbql.stage/native :native sql}]})]
+ (let [query (fn [sql] {:lib/type :mbql/query :stages [{:lib/type :mbql.stage/native :native sql}]})]
(testing "a multi-statement native query is rejected"
(is (thrown-with-msg?
clojure.lang.ExceptionInfo #"single select statement"
@@ -1390,3 +1390,210 @@
(let [result (driver/validate-impersonated-query :mysql (query "SELECT 1"))]
(is (map? result))
(is (string? (get-in result [:stages 0 :native])))))))))
+
+(deftest validate-impersonated-query-inherited-by-sql-drivers-test
+ (testing "every SQL driver inherits the single-statement validator via the :sql method"
+ ;; The validator used to be registered on `:sql-jdbc`, so a SQL driver that is not JDBC-backed resolved
+ ;; to the `:default` no-op instead. It is now a `:sql` defmethod, and every driver advertising
+ ;; connection impersonation is a `:sql-jdbc` (hence `:sql`) descendant, so they all inherit it.
+ (let [default-method (get-method driver/validate-impersonated-query :default)
+ sql-method (get-method driver/validate-impersonated-query :sql)]
+ (is (some? sql-method) "a :sql validator must be registered")
+ (is (not= default-method sql-method) "the :sql validator must be distinct from the :default no-op")
+ (testing "sql-jdbc drivers resolve to the :sql validator, not the no-op default"
+ ;; A driver only takes its place under :sql in the hierarchy once its namespace has loaded, and
+ ;; module drivers are only on the classpath when their alias is active. Initialize each driver we
+ ;; can, then assert; skip the ones this run can't load. :h2 is always present.
+ (doseq [d [:h2 :postgres :redshift :sqlserver :snowflake :mysql :clickhouse :starburst]
+ :when (try (driver/the-initialized-driver d) true (catch Throwable _ false))]
+ (testing d
+ (is (= sql-method (get-method driver/validate-impersonated-query d))
+ "must inherit the shared validator, not fall through to the :default no-op")))))))
+
+(deftest ^:parallel impersonation-refuses-native-without-a-validator-test
+ (testing "native is refused when the driver has no validate-impersonated-query of its own"
+ ;; The `:sql` method covers every sql-jdbc driver, but a future impersonation driver that does not
+ ;; descend from `:sql` would resolve to the `:default` no-op, which says nothing about the native SQL.
+ ;; Refusing surfaces that as an error rather than letting the check be skipped silently.
+ (let [guard #'impersonation.middleware/assert-native-validation-supported
+ native (lib/native-query meta/metadata-provider "SELECT 1")
+ mbql (lib/query meta/metadata-provider (meta/table-metadata :orders))]
+ (testing "a driver with no validator (resolves to the :default no-op) refuses a native query"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"does not support native queries"
+ (guard ::driver-without-validator native))))
+ (testing "the same driver lets an MBQL query through — only native SQL is checked"
+ (is (nil? (guard ::driver-without-validator mbql))))
+ (testing "a driver that supplies the :sql validator is not refused"
+ (driver/the-initialized-driver :h2)
+ (is (nil? (guard :h2 native)))))))
+
+(deftest impersonation-skips-native-validation-on-non-impersonating-driver-test
+ (testing "a driver without :connection-impersonation never switches roles, so a policy on it does not validate (or refuse) native SQL"
+ ;; `set-role-if-supported!` is gated on the same feature check, so on such a driver everyone runs as the full
+ ;; connection user regardless of the policy. H2 also has no SQLGlot dialect, so if the validator ran here it
+ ;; would refuse every non-admin native query outright.
+ (mt/test-driver :h2
+ (mt/with-premium-features #{:advanced-permissions}
+ (impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
+ :attributes {"impersonation_attr" "impersonation_role"}}
+ (is (= [[1]]
+ (mt/rows (qp/process-query (lib/native-query (mt/metadata-provider) "SELECT 1"))))))))))
+
+(deftest ^:parallel impersonated-native-large-literal-list-test
+ (testing "impersonated native queries with large literal lists"
+ ;; `validate-impersonated-query*` validates the ORIGINAL SQL, so a large but ordinary literal list —
+ ;; the values a Number variable inlines for `IN ({{ids}})` — validates normally, a second statement
+ ;; nested in such a list is counted because the parser sees the real input, and only a list beyond the
+ ;; memory ceiling is refused. Returns the executed :native string when accepted, or [::refused type]
+ ;; when not.
+ (let [validate (fn [sql]
+ (try
+ (lib/raw-native-query
+ (driver.sql/validate-impersonated-query*
+ :postgres (lib/native-query meta/metadata-provider sql)))
+ (catch clojure.lang.ExceptionInfo e [::refused (:type (ex-data e))])))
+ big (str "VALUES " (str/join "," (repeat 101 "(1)")))]
+ (testing "a second statement nested in an oversized VALUES list is refused however it is written"
+ ;; parsing the original, sqlglot counts the statements — so it does not matter whether the extra `;`
+ ;; sits inside a line comment, a block comment, a dollar-quoted body, or an E'' escape, each of
+ ;; which can unbalance the JVM-side paren scan that decides whether to collapse the list.
+ (doseq [[label tail] [[:line-comment ",( -- ((\n 1 )) t ; SELECT 2 --)) t"]
+ [:block-comment ",( /* (( */ 1 )) t ; SELECT 2 /* )) t */"]
+ [:dollar-quote ",( 1 )) t ; DO $tag$ BEGIN PERFORM 1; END $tag$ --)) t"]
+ [:e-escape ",( 1 )) t ; SELECT E'\\')' ; SELECT 2 --)) t"]]]
+ (testing label
+ (is (= [::refused qp.error-type/invalid-query] (validate (str "SELECT * FROM (" big tail)))))))
+ (testing "an ordinary large-but-under-ceiling list validates and returns the reconstructed statement"
+ (is (string? (validate (str "SELECT x FROM (" big ") t(x)"))))
+ (binding [driver.settings/*impersonation-allow-write?* true]
+ (is (string? (validate (str "INSERT INTO t (" big ")"))))))
+ (testing "a list beyond the memory ceiling is refused"
+ (is (= [::refused qp.error-type/invalid-query]
+ (validate (str "SELECT x FROM t WHERE x IN (" (str/join "," (range 2500)) ")")))))
+ (testing "a query with no large literal list validates and executes its reconstruction"
+ (is (string? (validate "SELECT x FROM (VALUES (1),(2),(3)) t(x)")))))))
+
+(deftest ^:parallel impersonated-native-join-source-is-validated-test
+ (testing "a native query used as a join source is validated like a top-level native stage"
+ ;; sql.qp inlines a native join source verbatim as a sub-select, so it needs the same single-statement
+ ;; check as a top-level native stage. The validator must walk join stages, not just the top-level
+ ;; `:stages`.
+ (let [mp meta/metadata-provider
+ orders (lib/query mp (meta/table-metadata :orders))
+ [a b] (lib/visible-columns orders)
+ join-query (fn [native-sql]
+ ;; an MBQL query whose join source is a native query — the join carries its own
+ ;; `:stages`, which the top-level walk would miss.
+ (lib/join orders (-> (lib/join-clause (lib/native-query mp native-sql) [(lib/= a b)])
+ (lib/with-join-alias "j"))))]
+ (testing "a second statement in a native join source is rejected"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"single select statement"
+ (driver.sql/validate-impersonated-query* :postgres (join-query "SELECT 1; DROP TABLE t")))))
+ (testing "a bare role-switch in a native join source is rejected"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"single select statement"
+ (driver.sql/validate-impersonated-query* :postgres (join-query "SET ROLE NONE")))))
+ (testing "a single SELECT native join source is accepted (the join stage is walked and passes)"
+ (is (some? (driver.sql/validate-impersonated-query* :postgres (join-query "SELECT 1"))))))))
+
+(deftest ^:parallel impersonated-admin-not-refused-for-uncollapsible-query-test
+ (testing "an admin's native query is left untouched when it can't be collapsed and parsed"
+ ;; A superuser is never actually impersonated (`connection-impersonation-role` returns nil for them), so
+ ;; validating their native SQL changes nothing. A query we merely cannot collapse-then-parse must not be
+ ;; refused for an admin where it used to run — the same exemption they already have for
+ ;; non-single-statement queries.
+ (let [big (str "SELECT x FROM (VALUES " (str/join "," (repeat 2500 "(1)")) ") t(x)")
+ run (fn [admin?]
+ (driver.sql/validate-impersonated-query*
+ :postgres (cond-> (lib/native-query meta/metadata-provider big)
+ admin? (assoc :impersonation/admin? true))))]
+ (testing "a non-admin is still refused"
+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"Reduce the size" (run false))))
+ (testing "an admin runs the original SQL unchanged"
+ (is (= big (lib/raw-native-query (run true))))))))
+
+(deftest ^:parallel impersonated-refusal-preserves-reason-and-names-array-test
+ (testing "the unchecked-SQL refusal carries its :reason and names ARRAY as well as VALUES/IN"
+ (let [stripped (str "SELECT x FROM t WHERE x = ANY(ARRAY[" (str/join "," (range 2500)) "])")
+ e (try (driver.sql/validate-impersonated-query*
+ :postgres (lib/native-query meta/metadata-provider stripped))
+ nil
+ (catch clojure.lang.ExceptionInfo ex ex))]
+ (is (some? e) "an oversized ARRAY literal must be refused")
+ (is (= "large-literal-list" (:reason (ex-data e))) "the underlying :reason is preserved for operators")
+ (is (re-find #"(?i)ARRAY" (ex-message e)) "the message names ARRAY, which also triggers stripping"))))
+
+(deftest impersonated-native-single-statement-enforced-for-all-drivers-test
+ (testing "a multi-statement native query is rejected for every driver that advertises connection impersonation"
+ (mt/test-drivers (mt/normal-drivers-with-feature :connection-impersonation)
+ (mt/with-premium-features #{:advanced-permissions}
+ ;; Validation is pure parsing, so this exercises each driver's dialect and the shared `:sql`
+ ;; validator without needing a live warehouse round-trip.
+ (let [validate (fn [sql]
+ (try
+ (driver/validate-impersonated-query
+ driver/*driver* (lib/native-query meta/metadata-provider sql))
+ :accepted
+ (catch clojure.lang.ExceptionInfo e (:type (ex-data e)))))]
+ (testing "a plain single SELECT is accepted"
+ (is (= :accepted (validate "SELECT 1"))))
+ (testing "a second statement is rejected"
+ (is (= qp.error-type/invalid-query (validate "SELECT 1; DROP TABLE t"))))
+ (testing "a bare role-switch is rejected"
+ (is (= qp.error-type/invalid-query (validate "SET ROLE NONE")))))))))
+
+(deftest impersonated-native-multi-statement-rejected-test
+ (testing "a non-admin's multi-statement native query is rejected, comment-nested separator and all"
+ ;; End-to-end check that the rejection holds through the real QP preprocess chain, and that it happens
+ ;; *before* execution: we ask the warehouse directly rather than trusting the QP status, so that a
+ ;; refactor moving validation after execution begins would fail here.
+ (mt/test-drivers (mt/normal-driver-select {:+parent :postgres})
+ (mt/with-premium-features #{:advanced-permissions}
+ (let [venues-table (sql.tx/qualify-and-quote driver/*driver* "test-data" "venues")
+ role-a (u/lower-case-en (mt/random-name))]
+ (tx/with-temp-roles! driver/*driver*
+ (impersonation-granting-details driver/*driver* (mt/db))
+ {role-a {venues-table {}}}
+ (impersonation-default-user driver/*driver*)
+ (impersonation-default-role driver/*driver*)
+ (let [granting-details (impersonation-granting-details driver/*driver* (mt/db))
+ spec (sql-jdbc.conn/connection-details->spec driver/*driver* granting-details)
+ ;; The second statement creates a table instead of changing data: whether that table
+ ;; exists afterwards is the ground truth for "a second statement ran", and observing it
+ ;; costs no existing row. The impersonated role is granted nothing but `venues`, so the
+ ;; table can only appear if the statement ran as the connection user.
+ check-table (str "mb_single_stmt_check_" (u/lower-case-en (mt/random-name)))
+ check-table? (fn []
+ (boolean (seq (jdbc/query spec ["SELECT 1 FROM information_schema.tables WHERE table_name = ?"
+ check-table]))))
+ ;; 101 tuples push the VALUES list past the collapse threshold, and the 102nd opens a
+ ;; line comment that unbalances the JVM-side paren scan — so the collapsed form reads as
+ ;; one SELECT while the real input is two statements. Parsing the real input catches it.
+ sql (str "SELECT * FROM (VALUES " (str/join "," (repeat 101 "(1)"))
+ ",( -- ((\n 1 )) t ; SET ROLE NONE ; CREATE TABLE " check-table
+ " (id integer) --)) t")]
+ (mt/with-temp [:model/Database database {:engine driver/*driver*
+ :details (impersonation-details driver/*driver* (mt/db))}]
+ (mt/with-db database
+ (when (driver/database-supports? driver/*driver* :connection-impersonation-requires-role nil)
+ (t2/update! :model/Database :id (mt/id) (assoc-in (mt/db) [:details :role] (impersonation-default-role driver/*driver*))))
+ (sync/sync-database! (:id database) {:scan :schema})
+ (impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]
+ :attributes {"impersonation_attr" role-a}}
+ (try
+ (testing "the query is rejected"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"(?i)single select statement|reduce the size"
+ (-> (lib/native-query (mt/metadata-provider) sql)
+ (qp/process-query)
+ (mt/rows)))))
+ (testing "the warehouse never saw the second statement"
+ (is (not (check-table?))))
+ (finally
+ (jdbc/execute! spec [(format "DROP TABLE IF EXISTS %s" check-table)])))))))))))))
diff --git a/src/metabase/driver/sql.clj b/src/metabase/driver/sql.clj
index b6acb3d4aed..a8e637741fb 100644
--- a/src/metabase/driver/sql.clj
+++ b/src/metabase/driver/sql.clj
@@ -337,6 +337,51 @@
native-query :- :metabase.lib.schema/native-only-query]
(sql-tools/validate-query driver native-query))
+(defn- validate-impersonated-native-stage
+ "Validate one native `stage` for `driver`, returning it with `:native` replaced by the reconstructed,
+ validated SQL — or throwing a [[metabase.query-processor.error-type/invalid-query]] `ex-info` when it must
+ be refused.
+
+ `admin?` users run as the full connection user and are never actually impersonated."
+ [driver stage admin?]
+ (let [write? driver.settings/*impersonation-allow-write?*
+ stmt-type (if write? "write" "read")
+ {:keys [verdict single-stmt? sql reason error]}
+ (sql-tools/validate-executable-stmt driver (:native stage) stmt-type)]
+ (cond
+ error
+ (do
+ (log/warnf "Failed to parse native query: %s" error)
+ (throw (ex-info (tru "Unable to parse native query. There might be something wrong with your query.")
+ {:type qp.error-type/invalid-query
+ :sql (:native stage)})))
+
+ ;; The SQL was not checked at all, as opposed to checked and found to be two statements. Admins run as
+ ;; the full connection user and are never actually impersonated, so validating their native SQL
+ ;; changes nothing: leave such a query untouched for them rather than refuse it. Everyone else is
+ ;; refused, with the `:reason` logged so an operator has a thread to pull.
+ (= verdict :unknown)
+ (if admin?
+ stage
+ (do
+ (log/warnf "Refusing impersonated native query on %s: cannot validate (%s)" driver reason)
+ (throw (ex-info (tru "Unable to validate this native query on an impersonated database. Reduce the size of any large VALUES, IN, or ARRAY lists.")
+ {:type qp.error-type/invalid-query
+ :reason reason
+ :sql (:native stage)}))))
+
+ ;; Admins may also run a single statement that is not of the required type (e.g. `SHOW TIMEZONE`),
+ ;; but never more than one statement.
+ (or (= verdict :allowed) (and single-stmt? admin?))
+ (assoc stage :native sql)
+
+ :else
+ (throw (ex-info (if write?
+ (tru "Invalid impersonated native query. Must be a single insert, update, or delete statement.")
+ (tru "Invalid impersonated native query. Must be a single select statement."))
+ {:type qp.error-type/invalid-query
+ :sql (:native stage)})))))
+
(defn validate-impersonated-query*
"Validates a native query by parsing it and ensuring that it is a single statement.
Reads `:impersonation/allow-write?` on the query to decide what to require: when truthy (set by
@@ -345,34 +390,30 @@
Queries that cannot be parsed are always rejected. Admins are allowed to run parseable queries that
aren't a single select/write statement (e.g. `SHOW TIMEZONE`); non-admins are restricted to a single
- statement of the required type."
+ statement of the required type.
+
+ Throws a [[metabase.query-processor.error-type/invalid-query]] `ex-info` when the query is rejected —
+ including when the validator refused without checking the SQL at all (a large VALUES/IN/ARRAY list it
+ would have to collapse before parsing, or an unknown dialect), which it refuses for non-admins rather
+ than running a string it did not validate. Admins are exempt from that refusal: they run as the full
+ connection user, so refusing a query we merely cannot collapse-then-parse changes nothing.
+ Registered as the `:sql` method of [[driver/validate-impersonated-query]], so every SQL driver inherits
+ it."
+ [driver query]
+ (let [admin? (boolean (:impersonation/admin? query))]
+ ;; Walk *every* stage, including those nested inside joins — a native query used as a join source is
+ ;; inlined verbatim as a sub-select, so it needs the same single-statement guard as a top-level native
+ ;; stage. `walk-stages` returns nil for a stage to leave it unchanged, so non-native stages
+ ;; pass straight through.
+ (lib/walk-stages
+ query
+ (fn [_query _path stage]
+ (when (lib.util/native-stage? stage)
+ (validate-impersonated-native-stage driver stage admin?))))))
+
+(defmethod driver/validate-impersonated-query :sql
[driver query]
- (update query :stages
- (fn [stages]
- (mapv (fn [stage]
- (if (lib.util/native-stage? stage)
- (let [[stmt-type allowed-stmts] (if driver.settings/*impersonation-allow-write?*
- ["write" (tru "insert, update, or delete")]
- ["read" (tru "select")])
- {:keys [is-single-stmt? allowed-stmt-type? sql error]}
- (sql-tools/is-single-stmt-of-type? driver (:native stage) stmt-type)]
- (cond error
- (do
- (log/warnf "Failed to parse native query: %s" error)
- (throw (ex-info (tru "Unable to parse native query. There might be something wrong with your query.")
- {:type qp.error-type/invalid-query
- :sql (:native stage)})))
-
- (and is-single-stmt?
- (or allowed-stmt-type? (:impersonation/admin? query)))
- (assoc stage :native sql)
-
- :else
- (throw (ex-info (tru "Invalid impersonated native query. Must be a single {0} statement." allowed-stmts)
- {:type qp.error-type/invalid-query
- :sql (:native stage)}))))
- stage))
- stages))))
+ (validate-impersonated-query* driver query))
;;; +----------------------------------------------------------------------------------------------------------------+
;;; | Convenience Imports |
diff --git a/src/metabase/driver/sql_jdbc.clj b/src/metabase/driver/sql_jdbc.clj
index 2e252fd42f0..869914dec55 100644
--- a/src/metabase/driver/sql_jdbc.clj
+++ b/src/metabase/driver/sql_jdbc.clj
@@ -76,10 +76,6 @@
(driver/validate-db-details! driver details)
(sql-jdbc.conn/can-connect? driver details))
-(defmethod driver/validate-impersonated-query :sql-jdbc
- [driver query]
- (driver.sql/validate-impersonated-query* driver query))
-
(defmethod driver/table-rows-seq :sql-jdbc
[driver database table]
(query driver database table {:select [:*]}))
diff --git a/src/metabase/lib/core.cljc b/src/metabase/lib/core.cljc
index bb2cd882431..8d11d579e56 100644
--- a/src/metabase/lib/core.cljc
+++ b/src/metabase/lib/core.cljc
@@ -124,6 +124,7 @@
[metabase.lib.util :as lib.util]
[metabase.lib.util.unique-name-generator]
[metabase.lib.validate :as lib.validate]
+ [metabase.lib.walk]
[metabase.lib.walk.util]
[metabase.util.malli :as mu]
[metabase.util.namespaces :as shared.ns]))
@@ -1738,6 +1739,8 @@
missing-table-error
syntax-error
validation-exception-error]
+ [metabase.lib.walk
+ walk-stages]
[metabase.lib.walk.util
all-field-ids
all-referenced-entity-ids
diff --git a/src/metabase/sql_parsing/core.clj b/src/metabase/sql_parsing/core.clj
index a5bdb501bf3..732be2c6e62 100644
--- a/src/metabase/sql_parsing/core.clj
+++ b/src/metabase/sql_parsing/core.clj
@@ -107,11 +107,17 @@
;; CPython). We strip them on the JVM side before passing SQL to Python, because GraalPy is too
;; slow at character-by-character string scanning over multi-MB inputs.
-(def ^:private ^:const strip-threshold
+(def ^:private ^:const literal-list-strip-threshold
"Strip VALUES clauses with more than this many tuples, and IN lists / ARRAY literals with at
- least this many commas."
+ least this many commas. Tuned for the fail-soft *analysis* callers, which run in bulk background
+ passes over stored queries: over-stripping is free there (they never need the literal values), so a
+ low cutoff — far below the thousands of items where GraalPy actually OOMs — is the safe default."
100)
+(def ^:private ^:const executable-literal-list-ceiling
+ "The literal-list ceiling for a caller that will execute what it validates."
+ 2000)
+
(def ^:private literal-list-keyword-pattern
"Pattern to find a VALUES, IN, or ARRAY keyword followed by its opening delimiter."
(re-pattern "(?i)\\b(?:VALUES\\s*\\(|IN\\s*\\(|ARRAY\\s*\\[)"))
@@ -200,12 +206,17 @@
(defn- strip-values-at
"Decide whether to strip the VALUES clause whose `VALUES (` match spans [match-start, match-end).
Returns [replacement resume-pos]: a single-row NULL placeholder (column count taken from the
- first tuple) covering the SQL up to resume-pos, or [nil match-end] to leave the clause untouched."
- [^String sql ^long match-start ^long match-end ^long n]
- (let [paren-start (dec match-end)
+ first tuple) covering the SQL up to resume-pos, or [nil match-end] to leave the clause untouched.
+
+ Takes `threshold` rather than reading a var so the impersonation validator can ask for its own, much
+ higher ceiling; `n` is derived rather than passed because a fn carrying primitive hints is capped at
+ four args."
+ [^String sql ^long match-start ^long match-end ^long threshold]
+ (let [n (.length sql)
+ paren-start (dec match-end)
first-end (skip-balanced-parens sql paren-start n)
[tuple-count end-pos] (count-and-skip-tuples sql first-end n)]
- (if (and (> (long tuple-count) strip-threshold)
+ (if (and (> (long tuple-count) threshold)
(> first-end (inc paren-start)))
[(make-null-placeholder (extract-keyword sql match-start match-end)
(count-top-level-commas (.substring sql (inc paren-start) (dec first-end))))
@@ -241,20 +252,23 @@
[match-start, match-end). Returns [replacement resume-pos]: `IN (NULL)` / `ARRAY[NULL]`
covering the whole list, or [nil match-end] to leave it untouched (resuming just inside the
delimiter, so a large list nested in a subquery — `IN (SELECT ... WHERE x IN (...))` — is
- still found)."
- [^String sql ^long match-start ^long match-end ^long n]
+ still found).
+
+ Takes `threshold` rather than reading a var, and derives `n`, for the reasons
+ on [[strip-values-at]]."
+ [^String sql ^long match-start ^long match-end ^long threshold]
(let [open-pos (dec match-end)
- [commas end-pos] (simple-list-end sql open-pos n)]
- (if (and commas (>= (long commas) strip-threshold))
+ [commas end-pos] (simple-list-end sql open-pos (.length sql))]
+ (if (and commas (>= (long commas) threshold))
[(str (extract-keyword sql match-start match-end)
(if (= (.charAt sql open-pos) \[) "[NULL]" " (NULL)"))
end-pos]
[nil match-end])))
(defn- strip-large-literal-lists*
- "Single pass over `sql`, replacing every oversized literal list — VALUES clause or literal-only
- IN list — with a NULL placeholder. Returns `sql` itself when nothing was stripped."
- ^String [^String sql]
+ "Single pass over `sql`, replacing every literal list larger than `threshold` — VALUES clause
+ or literal-only IN list — with a NULL placeholder. Returns `sql` itself when nothing was stripped."
+ ^String [^String sql ^long threshold]
(let [matcher (re-matcher literal-list-keyword-pattern sql)
n (.length sql)]
(if-not (.find matcher)
@@ -269,8 +283,8 @@
match-end (.end matcher)
[replacement resume] (let [ch (.charAt sql match-start)]
(if (or (= ch \V) (= ch \v))
- (strip-values-at sql match-start match-end n)
- (strip-list-at sql match-start match-end n)))]
+ (strip-values-at sql match-start match-end threshold)
+ (strip-list-at sql match-start match-end threshold)))]
(if replacement
(-> sb (.append sql (int i) (int match-start)) (.append ^String replacement))
(.append sb sql (int i) (int resume)))
@@ -279,24 +293,29 @@
(.find matcher (int resume))))))))))
(defn strip-large-literal-lists
- "Replace large literal lists with NULL placeholders: VALUES clauses with more than
- [[strip-threshold]] tuples become a single-row NULL tuple (preserving the column count from the
- first tuple), and simple IN lists / ARRAY literals — numbers, single-quoted strings, and
- VALUES-style tuples of them — with at least [[strip-threshold]] commas become `IN (NULL)` /
- `ARRAY[NULL]`. All surrounding SQL structure is preserved.
+ "Replace large literal lists with NULL placeholders: VALUES clauses with more than `threshold`
+ tuples become a single-row NULL tuple (preserving the column count from the first tuple), and simple
+ IN lists / ARRAY literals — numbers, single-quoted strings, and VALUES-style tuples of them — with at
+ least `threshold` commas become `IN (NULL)` / `ARRAY[NULL]`. All surrounding SQL structure is preserved.
+ `threshold` defaults to [[literal-list-strip-threshold]] (the analysis default).
This runs on the JVM side (fast) before passing SQL to GraalPy (slow at char scanning).
On any error, returns the original SQL unchanged so parsing can proceed normally.
- Best-effort by design: stripping only feeds fail-soft analysis, and execution paths restore the
- original SQL when anything was stripped ([[is-single-stmt-of-type?]]), so edge cases like
- keyword-shaped text inside string literals or comments are deliberately not guarded against."
- ^String [^String sql]
- (try
- (strip-large-literal-lists* sql)
- (catch Exception e
- (log/warnf "Error stripping large literal lists, passing SQL through unchanged: %s" (ex-message e))
- sql)))
+ Best-effort by design: stripping feeds fail-soft *analysis* (lineage, referenced tables/fields, returned
+ columns), so edge cases like keyword-shaped text inside string literals or comments are deliberately not
+ guarded against. A caller that needs its answer to hold for the string it goes on to run must NOT use
+ the rewrite: [[validate-executable-stmt]] and [[classify-stmt]] parse the original SQL and call this
+ only to find out whether a list is over their ceiling, refusing outright when it is rather than
+ answering about a rewrite — those same unguarded edge cases are what make a rewrite unsound to reason
+ about."
+ (^String [^String sql] (strip-large-literal-lists sql literal-list-strip-threshold))
+ (^String [^String sql ^long threshold]
+ (try
+ (strip-large-literal-lists* sql threshold)
+ (catch Exception e
+ (log/warnf "Error stripping large literal lists, passing SQL through unchanged: %s" (ex-message e))
+ sql))))
;;; -------------------------------------------------- Public API --------------------------------------------------
@@ -574,38 +593,96 @@
[dialect sql replacements]
(protocol/replace-names (parser) dialect sql replacements))
-;; Parsing time and memory grow with the length of the SQL, whatever it holds, and the "read-only" check runs on SQL
-;; a user or a model wrote.
+;;; --------------------------------------- Single-statement classification ----------------------------------------
+
+;; Parsing time and memory grow with the length of the SQL, whatever it holds.
;; As of 2026-10-10, the densest SQL that fits, a list of about 8,000 one-character items, takes about a second to
;; check, and six such checks at once complete on a 1 GB heap. At 60,000 characters one check took 1 to 3 seconds.
-(def ^:private max-read-only-sql-chars
- "The longest SQL, in characters, that the \"read-only\" check parses."
+(def ^:private ^:const max-gate-sql-chars
+ "The longest SQL, in characters, that [[classify-stmt]] parses. A cheap ceiling on work spent for a
+ caller that is only gating, and whose SQL a user or a model wrote."
(* 16 1024))
-(defn is-single-stmt-of-type?
- "Validates that a query is a single read statement (SELECT) or a single write statement (INSERT, UPDATE, DELETE)
- and returns the query reconstructed from the parsed AST. `stmt-type` is \"read\", \"write\", or \"read-only\": a
- read statement that also holds nothing that writes or locks anywhere in its tree.
- For \"read-only\", SQL that is not allowed carries a `:reason`, and SQL that is too long or holds a literal list
- too large to parse whole is refused without being parsed."
+(def ^:private executable-limits
+ "Limits for a caller that will execute the `:sql` the verdict carries — [[validate-executable-stmt]].
+
+ Such a caller needs the verdict to hold for the exact string it goes on to run, so a known `dialect` is
+ required: `nil` makes SQLGlot parse in its generic dialect and hand back a confident verdict plus a
+ generically-regenerated `:sql`, which is not the string that was checked."
+ {:max-sql-chars nil
+ :literal-list-ceiling executable-literal-list-ceiling
+ :require-dialect? true})
+
+(def ^:private gate-limits
+ "Limits for a caller that only reads the verdict and never executes its `:sql` — [[classify-stmt]]."
+ {:max-sql-chars max-gate-sql-chars
+ :literal-list-ceiling literal-list-strip-threshold
+ :require-dialect? false})
+
+(defn- unknown
+ "A verdict that `sql` could not be checked, for `reason`."
+ [sql reason]
+ {:verdict :unknown, :reason reason, :sql sql})
+
+(defn- verdict
+ "Turn the parser's raw result for `sql` into a verdict map."
+ [sql raw]
+ (let [{:keys [is-single-stmt? allowed-stmt-type? error reason reason-detail] parsed-sql :sql}
+ (perf/update-keys raw (comp keyword u/->kebab-case-en))]
+ (cond
+ ;; SQL that does not parse is SQL we know nothing about. `:error` carries the parser's message, which
+ ;; a caller may want to word differently from the refusals it *can* explain.
+ error (assoc (unknown sql "unparseable") :error error)
+ allowed-stmt-type? {:verdict :allowed, :single-stmt? true, :sql parsed-sql}
+ :else (cond-> {:verdict :disallowed, :single-stmt? (boolean is-single-stmt?)}
+ reason (assoc :reason reason)
+ reason-detail (assoc :detail reason-detail)
+ parsed-sql (assoc :sql parsed-sql)))))
+
+(defn- classify-single-stmt
+ "Classify `sql` as a single statement of `stmt-type` for `dialect`, spending no more than `limits` allow.
+ Applies `limits` as given; it does not infer them from `stmt-type`."
+ [dialect sql stmt-type {:keys [max-sql-chars literal-list-ceiling require-dialect?]}]
+ (cond
+ ;; The length is checked before anything scans the SQL: stripping literal lists is itself work that
+ ;; grows with it.
+ (and max-sql-chars (< (long max-sql-chars) (count sql)))
+ (unknown sql "too-long")
+
+ (and require-dialect? (nil? dialect))
+ (unknown sql "unknown-dialect")
+
+ ;; Stripping is only ever a size guard: if it would collapse a list, parsing that list could OOM the
+ ;; shared JVM, so refuse rather than parse the collapsed form and answer about that instead. Stripping
+ ;; also does not skip comments, so a collapsed list can swallow a second statement into a comment.
+ ;; Under the ceiling nothing is stripped, so we parse exactly the string the caller holds.
+ (not= sql (strip-large-literal-lists sql literal-list-ceiling))
+ (unknown sql "large-literal-list")
+
+ :else
+ (verdict sql (protocol/single-stmt-of-type (parser) dialect sql stmt-type))))
+
+(defn validate-executable-stmt
+ "Whether `sql` is a single statement of `stmt-type` for `dialect`, for a caller that will **execute** the
+ `:sql` the verdict carries: `\"read\"` = SELECT, `\"write\"` = INSERT, UPDATE, or DELETE,
+ `\"read-only\"` = a read statement that also holds nothing that writes or locks anywhere in its tree.
+
+ The `:sql` is reconstructed from the parsed AST, and only ever from `sql` itself — SQL that cannot be
+ parsed as it stands comes back `:verdict :unknown` rather than parsed in a collapsed form and vouched
+ for in its original one. See [[executable-limits]] for what that costs, and [[classify-stmt]] for the
+ same question asked by a caller that only needs a verdict."
+ [dialect sql stmt-type]
+ (classify-single-stmt dialect sql stmt-type executable-limits))
+
+(defn classify-stmt
+ "Whether `sql` is a single statement of `stmt-type` for `dialect` (see [[validate-executable-stmt]] for
+ the `stmt-type` values), for a caller that only reads the verdict and never executes its `:sql`.
+
+ Cheaper and more permissive about `dialect` than [[validate-executable-stmt]] — see [[gate-limits]].
+ Do not execute the `:sql` this returns: with no `dialect` it is a generic-dialect regeneration, not the
+ string that was checked."
[dialect sql stmt-type]
- ;; The length is checked before anything scans the SQL: stripping literal lists is itself work that grows with it.
- (if (and (= stmt-type "read-only") (< max-read-only-sql-chars (count sql)))
- {:is-single-stmt? false, :allowed-stmt-type? false, :sql sql, :reason "too-long"}
- (let [stripped-sql (strip-large-literal-lists sql)]
- ;; Stripping does not skip comments, so it can rewrite a second statement into a commented-out list.
- ;; "read-only" must judge the SQL that will run, and parsing a large list unstripped risks a GraalPy OOM, so it
- ;; refuses.
- (if (and (= stmt-type "read-only") (not= sql stripped-sql))
- {:is-single-stmt? false, :allowed-stmt-type? false, :sql sql, :reason "large-literal-list"}
- (let [result (-> (protocol/single-stmt-of-type (parser) dialect stripped-sql stmt-type)
- (perf/update-keys (comp keyword u/->kebab-case-en)))]
- ;; The `:sql` in the `result` is the reconstructed SQL from the SQLGlot parser.
- ;; We generally want to use the reconstructed SQL, but if the original SQL had its VALUES/IN
- ;; literal lists stripped (to avoid GraalPy OOM) then we need to return the original SQL to
- ;; preserve the values. (#74284)
- (cond-> result
- (not= sql stripped-sql) (assoc :sql sql)))))))
+ (classify-single-stmt dialect sql stmt-type gate-limits))
(comment
(referenced-tables "postgres" "select * from transactions")
diff --git a/src/metabase/sql_tools/core.clj b/src/metabase/sql_tools/core.clj
index 59cb4ba9494..4164b526bb9 100644
--- a/src/metabase/sql_tools/core.clj
+++ b/src/metabase/sql_tools/core.clj
@@ -233,10 +233,28 @@
(interface/transpile-sql-impl (sql-tools.settings/sql-tools-parser-backend)
sql from-dialect to-dialect))
-(defn is-single-stmt-of-type?
- "Wrapper around `sql-parsing/is-single-stmt-of-type?`."
+(defn validate-executable-stmt
+ "Whether `sql` is a single statement of `stmt-type` for `driver`'s dialect, for a caller that will
+ **execute** the `:sql` the verdict carries: `\"read\"` = SELECT, `\"write\"` = INSERT, UPDATE, or
+ DELETE, `\"read-only\"` = a read statement that also holds nothing that writes or locks anywhere in its
+ tree (see [[read-only-select?]]).
+
+ The `:sql` is reconstructed from the parsed AST, and only ever from `sql` itself — SQL that cannot be
+ parsed as it stands (`driver` has no known dialect, or a literal list is large enough to risk a GraalPy
+ OOM) comes back `:verdict :unknown` rather than parsed in a collapsed form and vouched for in its
+ original one. Only `:verdict :allowed` is permission to proceed.
+
+ See [[metabase.sql-parsing.core/validate-executable-stmt]] for the verdict shape, and [[classify-stmt]]
+ for the same question asked by a caller that only needs a verdict."
[driver sql stmt-type]
- (sql-parsing/is-single-stmt-of-type? (sqlglot/driver->dialect driver) sql stmt-type))
+ (sql-parsing/validate-executable-stmt (sqlglot/driver->dialect driver) sql stmt-type))
+
+(defn classify-stmt
+ "Whether `sql` is a single statement of `stmt-type` for `driver`'s dialect, for a caller that only reads
+ the verdict and never executes its `:sql`. Cheaper and more permissive about the dialect than
+ [[validate-executable-stmt]]; do not execute the `:sql` it returns."
+ [driver sql stmt-type]
+ (sql-parsing/classify-stmt (sqlglot/driver->dialect driver) sql stmt-type))
(mu/defn read-only-select? :- :boolean
"Whether `sql` is exactly one plain query, a SELECT or a set operation of SELECTs, holding nothing known to write,
@@ -249,7 +267,7 @@
A function this does not name, a user-defined one included, passes: only the database's own permissions stop it."
[driver :- :keyword
sql :- :string]
- (true? (:allowed-stmt-type? (is-single-stmt-of-type? driver sql "read-only"))))
+ (= :allowed (:verdict (classify-stmt driver sql "read-only"))))
(mu/defn read-only-select-problem :- [:maybe [:map [:reason :keyword] [:detail [:maybe :string]]]]
"Why `sql` is not a read-only select ([[read-only-select?]]), or nil when it is one.
@@ -260,7 +278,7 @@
function's name."
[driver :- :keyword
sql :- :string]
- (let [{:keys [allowed-stmt-type? reason reason-detail]} (is-single-stmt-of-type? driver sql "read-only")]
- (when-not (true? allowed-stmt-type?)
+ (let [{:keys [verdict reason detail]} (classify-stmt driver sql "read-only")]
+ (when-not (= :allowed verdict)
{:reason (keyword (or reason "unparseable"))
- :detail reason-detail})))
+ :detail detail})))
diff --git a/test/metabase/driver/postgres_test.clj b/test/metabase/driver/postgres_test.clj
index a95ef286cd6..7ecbc6501aa 100644
--- a/test/metabase/driver/postgres_test.clj
+++ b/test/metabase/driver/postgres_test.clj
@@ -466,9 +466,8 @@
(let [parent (h2x/identifier :field "public" "test_table" "data")
nfc-field {:nfc-path ["data" "key"] :database-type "text"}
[json-sql] (sql/format-expr (#'sql.qp/json-query :postgres parent nfc-field))
- stage {:lib/type :mbql.stage/native :native (str "SELECT " json-sql " AS k FROM t")}
- out-sql (-> (driver.sql/validate-impersonated-query* :postgres {:stages [stage]})
- :stages first :native)]
+ query (lib/native-query meta/metadata-provider (str "SELECT " json-sql " AS k FROM t"))
+ out-sql (lib/raw-native-query (driver.sql/validate-impersonated-query* :postgres query))]
;; The broken reconstruction looks like `CAST(parent #>> array[?] AS TEXT[])` — i.e.
;; the cast wraps the entire `#>>` expression instead of just the path argument.
(is (not (re-find #"(?i)#>>\s+array\s*\[\?\]\s+AS\s+TEXT\s*\[\s*\]" out-sql))
diff --git a/test/metabase/sql_parsing/core_test.clj b/test/metabase/sql_parsing/core_test.clj
index 048dffc2f64..09ab5c19e17 100644
--- a/test/metabase/sql_parsing/core_test.clj
+++ b/test/metabase/sql_parsing/core_test.clj
@@ -96,13 +96,16 @@
(sort (keep :column (:returned-fields result))))
(pr-str sql)))))))
-;;; --------------------------------------------- is-single-stmt-of-type? ------------------------------------------
+;;; ------------------------------------------ single-stmt classification ------------------------------------------
-(deftest ^:parallel is-single-stmt-of-type?-test
+(deftest ^:parallel single-stmt-classification-test
(testing "whether a query is one statement of the kind asked for, which is what tells a caller the rest is unread"
(doseq [{:keys [name sql dialect stmt-type expected]} (cases "single_stmt")]
(testing name
- (is (=? expected (sql-parsing/is-single-stmt-of-type? dialect sql stmt-type))
+ (is (=? expected (sql-parsing/validate-executable-stmt dialect sql stmt-type))
+ (pr-str sql))
+ (is (= (sql-parsing/validate-executable-stmt dialect sql stmt-type)
+ (sql-parsing/classify-stmt dialect sql stmt-type))
(pr-str sql))))))
;;; ------------------------------------------- returned-columns-lineage -------------------------------------------
@@ -175,6 +178,14 @@
(is (= [[nil nil "accounts"]] (sql-parsing/referenced-tables "postgres" sql)))
(is (= [[nil nil "accounts" "id"] [nil nil "accounts" "name"]]
(sort (sql-parsing/referenced-fields "postgres" sql))))))
- (testing "and is one statement still"
- (let [sql (str "SELECT * FROM t WHERE id IN (" (str/join ", " (range 20000)) ")")]
- (is (=? {:is-single-stmt? true} (sql-parsing/is-single-stmt-of-type? "postgres" sql "read"))))))
+ (testing "but a classification never answers about a stripped rewrite"
+ ;; Analysis can read a stripped query because it only needs the shape. A classification's answer has to
+ ;; hold for the string the caller holds, so it parses the original and reports SQL over its ceiling as
+ ;; unchecked rather than answering about a rewrite.
+ (let [under (str "SELECT * FROM t WHERE id IN (" (str/join ", " (range 150)) ")")
+ over (str "SELECT * FROM t WHERE id IN (" (str/join ", " (range 20000)) ")")]
+ (testing "a list the analysis threshold would strip still validates as one statement"
+ (is (=? {:verdict :allowed} (sql-parsing/validate-executable-stmt "postgres" under "read"))))
+ (testing "a list past the ceiling comes back unchecked, not stripped and vouched for"
+ (is (= {:verdict :unknown, :reason "large-literal-list", :sql over}
+ (sql-parsing/validate-executable-stmt "postgres" over "read")))))))
diff --git a/test/metabase/sql_parsing/test_cases/single_stmt.edn b/test/metabase/sql_parsing/test_cases/single_stmt.edn
index 3d2166e5c08..ae90beb3b69 100644
--- a/test/metabase/sql_parsing/test_cases/single_stmt.edn
+++ b/test/metabase/sql_parsing/test_cases/single_stmt.edn
@@ -3,40 +3,40 @@
:dialect "postgres"
:sql "SELECT * FROM orders"
:stmt-type "read"
- :expected {:is-single-stmt? true :allowed-stmt-type? true}}
+ :expected {:verdict :allowed :single-stmt? true}}
{:name "two statements are not a single statement"
:dialect "postgres"
:sql "SELECT * FROM orders; DELETE FROM people"
:stmt-type "read"
- :expected {:is-single-stmt? false :allowed-stmt-type? false}}
+ :expected {:verdict :disallowed :single-stmt? false}}
{:name "a trailing semicolon does not make a second statement"
:dialect "postgres"
:sql "SELECT * FROM orders;"
:stmt-type "read"
- :expected {:is-single-stmt? true :allowed-stmt-type? true}}
+ :expected {:verdict :allowed :single-stmt? true}}
{:name "a semicolon inside a literal is not a statement boundary"
:dialect "postgres"
:sql "SELECT * FROM orders WHERE note = 'a; b'"
:stmt-type "read"
- :expected {:is-single-stmt? true :allowed-stmt-type? true}}
+ :expected {:verdict :allowed :single-stmt? true}}
{:name "a write is a single statement, but not a read"
:dialect "postgres"
:sql "DELETE FROM people WHERE id = 1"
:stmt-type "read"
- :expected {:is-single-stmt? true :allowed-stmt-type? false}}
+ :expected {:verdict :disallowed :single-stmt? true}}
{:name "the same write, asked for as a write"
:dialect "postgres"
:sql "DELETE FROM people WHERE id = 1"
:stmt-type "write"
- :expected {:is-single-stmt? true :allowed-stmt-type? true}}
+ :expected {:verdict :allowed :single-stmt? true}}
{:name "a CTE ahead of a select is still one read"
:dialect "postgres"
:sql "WITH c AS (SELECT id FROM people) SELECT * FROM c"
:stmt-type "read"
- :expected {:is-single-stmt? true :allowed-stmt-type? true}}]}
+ :expected {:verdict :allowed :single-stmt? true}}]}
diff --git a/test/metabase/sql_tools/core_test.clj b/test/metabase/sql_tools/core_test.clj
index 7c64709b725..d0ec054093b 100644
--- a/test/metabase/sql_tools/core_test.clj
+++ b/test/metabase/sql_tools/core_test.clj
@@ -212,7 +212,7 @@
(is (= :skipped status))
(is (= :missing-dialect reason))))))
-(deftest ^:parallel is-single-stmt-of-type?-test
+(deftest ^:parallel validate-executable-stmt-test
(mt/test-drivers (mt/normal-drivers-with-feature :connection-impersonation)
(let [mp (mt/metadata-provider)
products (lib.metadata/table mp (mt/id :products))
@@ -221,32 +221,39 @@
(lib/filter (lib/= product-category "Widget")))
native-query (:query (qp.compile/compile-with-inline-parameters query))]
(testing "A single SELECT statement returns true and the reconstructed SQL"
- (are [sql] (=? {:is-single-stmt? true :allowed-stmt-type? true :sql string?}
- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "read"))
+ (are [sql] (=? {:verdict :allowed :single-stmt? true :sql string?}
+ (sql-tools/validate-executable-stmt driver/*driver* sql "read"))
native-query
"SELECT 1"
"SELECT * FROM table"
"WITH x AS (SELECT * FROM foo) SELECT * from x"
"WITH x AS (SELECT a FROM foo), y AS (SELECT b FROM bar), z AS (SELECT c FROM baz) SELECT x.a, y.b, z.c FROM x, y, z")))
(testing "All other read queries are rejected"
- (are [sql is-single-stmt?] (=? {:is-single-stmt? is-single-stmt? :allowed-stmt-type? false}
- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "read"))
- "SELECT (" false
+ (are [sql single-stmt?] (=? {:verdict :disallowed :single-stmt? single-stmt?}
+ (sql-tools/validate-executable-stmt driver/*driver* sql "read"))
"SELECT 1; SELECT 2" false
"SET ROLE NONE" true
"DROP TABLE table" true
"SET ROLE NONE; DROP TABLE table" false
- "SELECT set_config('role', 'none', false); DROP TABLE table" false
- "DO $$ BEGIN EXECUTE 'SET ROLE NONE; DROP TABLE table'; END $$;" (isa? driver/hierarchy driver/*driver* :postgres)))
+ "SELECT set_config('role', 'none', false); DROP TABLE table" false))
+ (testing "SQL the dialect cannot parse is :unknown, not :disallowed — nothing was read, so nothing is claimed"
+ (is (=? {:verdict :unknown :reason "unparseable"}
+ (sql-tools/validate-executable-stmt driver/*driver* "SELECT (" "read"))))
+ (testing "a DO block smuggling SET ROLE is refused: one disallowed statement to a dialect that reads PL/pgSQL, unparseable to the rest"
+ (is (=? (if (isa? driver/hierarchy driver/*driver* :postgres)
+ {:verdict :disallowed :single-stmt? true}
+ {:verdict :unknown :reason "unparseable"})
+ (sql-tools/validate-executable-stmt
+ driver/*driver* "DO $$ BEGIN EXECUTE 'SET ROLE NONE; DROP TABLE table'; END $$;" "read"))))
(testing "A single insert, update or delete statement returns true and the reconstructed SQL"
- (are [sql] (=? {:is-single-stmt? true :allowed-stmt-type? true :sql string?}
- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "write"))
+ (are [sql] (=? {:verdict :allowed :single-stmt? true :sql string?}
+ (sql-tools/validate-executable-stmt driver/*driver* sql "write"))
"INSERT INTO table VALUES (1)"
"UPDATE table SET column = 1"
"DELETE FROM table WHERE id = 1"))
(testing "All other write queries are rejected"
- (are [sql is-single-stmt?] (=? {:is-single-stmt? is-single-stmt? :allowed-stmt-type? false}
- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "write"))
+ (are [sql single-stmt?] (=? {:verdict :disallowed :single-stmt? single-stmt?}
+ (sql-tools/validate-executable-stmt driver/*driver* sql "write"))
"SELECT 1" true
"INSERT INTO table VALUES (1); SELECT 1" false
"UPDATE table SET column = 1; SELECT 1" false
@@ -257,10 +264,10 @@
(doseq [op ["UNION ALL" "INTERSECT ALL" "EXCEPT ALL"]
ts [["foo" "bar"] ["foo" "bar" "baz"]]
:let [sql (str/join (str " " op " ") (map #(str "SELECT * FROM " %) ts))]]
- (is (=? {:is-single-stmt? true, :sql string?}
- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "read"))))
- (are [sql] (=? {:is-single-stmt? true, :allowed-stmt-type? true :sql string?}
- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "read"))
+ (is (=? {:verdict :allowed, :sql string?}
+ (sql-tools/validate-executable-stmt driver/*driver* sql "read"))))
+ (are [sql] (=? {:verdict :allowed, :single-stmt? true :sql string?}
+ (sql-tools/validate-executable-stmt driver/*driver* sql "read"))
"SELECT * FROM foo UNION ALL SELECT * FROM bar INTERSECT ALL SELECT * FROM baz"
"SELECT * FROM foo UNION ALL SELECT * FROM bar EXCEPT ALL SELECT * FROM baz"
"SELECT * FROM foo INTERSECT ALL SELECT * FROM bar UNION ALL SELECT * FROM baz"
@@ -272,50 +279,83 @@
[sql]
(count (re-seq #"\?" sql)))
-(deftest ^:parallel is-single-stmt-of-type-placeholder-cast-test
+(deftest ^:parallel validate-executable-stmt-placeholder-cast-test
(testing "queries with `?::` are parsed correctly"
(doseq [sql ["SELECT ?::date"
"SELECT (?::date - x::date)"
"SELECT ?::text, ?::integer, ?::boolean FROM t WHERE x = ?"
"SELECT (?::date - CURRENT_DATE) AS diff"]]
- (let [{out-sql :sql :as result} (sql-tools/is-single-stmt-of-type? :postgres sql "read")]
- (is (=? {:is-single-stmt? true :allowed-stmt-type? true :sql string?} result))
+ (let [{out-sql :sql :as result} (sql-tools/validate-executable-stmt :postgres sql "read")]
+ (is (=? {:verdict :allowed :single-stmt? true :sql string?} result))
(is (= (placeholder-count sql) (placeholder-count out-sql))))))
(testing "a query with `?::` inside string literals are left untouched"
- (is (= {:is-single-stmt? true :allowed-stmt-type? true :sql "SELECT '?::date'"}
- (sql-tools/is-single-stmt-of-type? :postgres "select '?::date'" "read"))))
+ (is (= {:verdict :allowed :single-stmt? true :sql "SELECT '?::date'"}
+ (sql-tools/validate-executable-stmt :postgres "select '?::date'" "read"))))
(testing "multi-statement queries with placeholder casts are still rejected"
- (are [sql] (=? {:is-single-stmt? false :allowed-stmt-type? false}
- (sql-tools/is-single-stmt-of-type? :postgres sql "read"))
+ (are [sql] (=? {:verdict :disallowed :single-stmt? false}
+ (sql-tools/validate-executable-stmt :postgres sql "read"))
"SELECT ?::date; DROP TABLE t"
"SET ROLE NONE; SELECT ?::date")))
-(deftest ^:parallel is-single-stmt-of-type-qdcolon-dialects-test
+(deftest ^:parallel validate-executable-stmt-qdcolon-dialects-test
(testing "databricks' native `expr?::type` try-cast operator is not split apart"
- (is (=? {:is-single-stmt? true :allowed-stmt-type? true :sql #"(?i).*TRY_CAST\(x AS DATE\).*"}
- (sql-tools/is-single-stmt-of-type? :databricks "SELECT x?::date FROM t" "read")))))
-
-(deftest ^:parallel is-single-stmt-of-type-not-stripped-test
- (testing "we don't remove value clauses when validating impersonated queries (#74284)"
- (let [values-query (str "SELECT x FROM (VALUES " (str/join ", " (repeat 105 "(1)")) ") AS t(x)")]
- (are [sql is-single-stmt? allowed-stmt-type?]
- (= {:is-single-stmt? is-single-stmt? :allowed-stmt-type? allowed-stmt-type? :sql sql}
- (sql-tools/is-single-stmt-of-type? :postgres sql "read"))
- values-query true true
- (str "SELECT 1; " values-query) false false
- (str "SET ROLE none; " values-query) false false
- (str values-query "; SELECT 1") false false
- (str values-query "; SET ROLE none") false false)))
- (testing "we don't remove large IN lists, tuple lists, or arrays when validating impersonated queries"
- (doseq [query [(str "SELECT x FROM t WHERE x IN (" (str/join ", " (range 105)) ")")
- (str "SELECT x FROM t WHERE (x, y) IN (" (str/join ", " (map #(format "(%d, %d)" % %) (range 105))) ")")
- (str "SELECT x FROM t WHERE x = ANY(ARRAY[" (str/join ", " (range 105)) "])")]]
- (are [sql is-single-stmt? allowed-stmt-type?]
- (= {:is-single-stmt? is-single-stmt? :allowed-stmt-type? allowed-stmt-type? :sql sql}
- (sql-tools/is-single-stmt-of-type? :postgres sql "read"))
- query true true
- (str "SELECT 1; " query) false false
- (str query "; SELECT 1") false false))))
+ (is (=? {:verdict :allowed :single-stmt? true :sql #"(?i).*TRY_CAST\(x AS DATE\).*"}
+ (sql-tools/validate-executable-stmt :databricks "SELECT x?::date FROM t" "read")))))
+
+(deftest ^:parallel validate-executable-stmt-large-literal-lists-test
+ ;; A classification reads the ORIGINAL SQL, never a collapsed rewrite of it. A large but
+ ;; ordinary literal list — the values a Number variable inlines for `IN ({{ids}})` — validates normally,
+ ;; and a second statement nested in such a list is counted as a second statement however it is written,
+ ;; because the parser sees the real input. Collapsing only guards, at a much higher threshold, against a
+ ;; list so huge it would OOM GraalPy — those are refused outright rather than substituted.
+ (testing "a large-but-under-ceiling literal list validates normally (the IN ({{ids}}) idiom)"
+ (are [sql] (=? {:verdict :allowed :single-stmt? true :sql string?}
+ (sql-tools/validate-executable-stmt :postgres sql "read"))
+ (str "SELECT x FROM (VALUES " (str/join ", " (repeat 105 "(1)")) ") AS t(x)")
+ (str "SELECT x FROM t WHERE x IN (" (str/join ", " (range 105)) ")")
+ (str "SELECT x FROM t WHERE (x, y) IN (" (str/join ", " (map #(format "(%d, %d)" % %) (range 105))) ")")
+ (str "SELECT x FROM t WHERE x = ANY(ARRAY[" (str/join ", " (range 105)) "])")))
+ (testing "a second statement alongside such a list is counted as a second statement"
+ (let [base (str "SELECT x FROM t WHERE x IN (" (str/join ", " (range 105)) ")")]
+ (are [sql] (=? {:verdict :disallowed}
+ (sql-tools/validate-executable-stmt :postgres sql "read"))
+ (str "SELECT 1; " base)
+ (str "SET ROLE none; " base)
+ (str base "; SELECT 1")
+ (str base "; SET ROLE none"))))
+ (testing "a statement separator nested in a comment inside an oversized VALUES list is still counted"
+ ;; a `--` comment can unbalance the JVM-side paren scan that decides whether to collapse a list; parsing
+ ;; the original rather than the collapsed form, sqlglot counts two statements either way
+ (let [sql (str "SELECT * FROM (VALUES " (str/join "," (repeat 101 "(1)"))
+ ",( -- ((\n 1 )) t ; SELECT 2 --)) t")]
+ (is (=? {:verdict :disallowed}
+ (sql-tools/validate-executable-stmt :postgres sql "read")))))
+ (testing "a literal list beyond the ceiling comes back unchecked (memory guard), not parsed"
+ (let [huge (str "SELECT x FROM t WHERE x IN (" (str/join "," (range 2500)) ")")]
+ (is (= {:verdict :unknown, :reason "large-literal-list", :sql huge}
+ (sql-tools/validate-executable-stmt :postgres huge "read")))
+ (testing "and the gate, whose ceiling is far lower, says the same of a list over its own"
+ (is (= {:verdict :unknown, :reason "large-literal-list", :sql huge}
+ (sql-tools/classify-stmt :postgres huge "read"))))))
+ (testing "a driver with no SQLGlot dialect cannot be validated for execution"
+ ;; sqlglot would otherwise parse in its generic dialect and hand back a confident verdict plus a
+ ;; generically-regenerated `:sql` string, which a caller that executes the `:sql` would go on to run.
+ (is (= {:verdict :unknown, :reason "unknown-dialect", :sql "SELECT 1"}
+ (sql-tools/validate-executable-stmt :h2 "SELECT 1" "read")))
+ (testing "while the gate, which executes nothing, accepts the generic dialect"
+ (is (=? {:verdict :allowed} (sql-tools/classify-stmt :h2 "SELECT 1" "read")))
+ (is (true? (sql-tools/read-only-select? :h2 "SELECT 1"))))))
+
+(deftest ^:parallel classify-stmt-length-cap-test
+ (testing "the gate stops reading SQL past its length cap, however harmless"
+ (let [padded (fn [n] (str "SELECT 1" (str/join (repeat (- n 8) " "))))
+ limit (* 16 1024)]
+ (is (=? {:verdict :allowed} (sql-tools/classify-stmt :postgres (padded limit) "read")))
+ (is (= {:verdict :unknown, :reason "too-long", :sql (padded (inc limit))}
+ (sql-tools/classify-stmt :postgres (padded (inc limit)) "read")))))
+ (testing "a caller that will execute the SQL has no length cap: long native queries are legitimate"
+ (let [long-sql (str "SELECT 1 FROM t WHERE x = '" (str/join (repeat (* 32 1024) "a")) "'")]
+ (is (=? {:verdict :allowed} (sql-tools/validate-executable-stmt :postgres long-sql "read"))))))
(deftest ^:parallel read-only-select?-test
(doseq [driver [:postgres :mysql :h2]]