The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Tighten impersonated native-query validation (#84202)
John Swansonb91c7cad94e4
diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.ednindex 86c6cfe76c0..dd568acdcec 100644--- a/.clj-kondo/config/modules/config.edn+++ b/.clj-kondo/config/modules/config.edn@@ -4247,6 +4247,7 @@:uses #{apiaudit-appdriver+ liblib.metadatalib.schemamodelsdiff --git a/docs/developers-guide/driver-changelog.md b/docs/developers-guide/driver-changelog.mdindex 42e84131d43..454eb931fa6 100644--- a/docs/developers-guide/driver-changelog.md+++ b/docs/developers-guide/driver-changelog.md@@ -11,6 +11,13 @@ title: Driver interface changelogerror codes or exclude SQLStates that also cover operational failures. Unknown failures remain QP errors;cancellation takes precedence over query-error evidence.+- `metabase.driver/validate-impersonated-query` `[driver query]` has moved from `:sql-jdbc` to `:sql`, so every+ SQL driver -- not only the JDBC ones -- inherits the check that a connection-impersonated native query is a+ single statement of the allowed type. It now walks native stages nested inside joins as well as the top-level+ ones, and throws rather than returning a verdict when it cannot parse the exact string that would execute.+ A driver that supports `:connection-impersonation` without descending from `:sql` must implement the method+ itself; impersonated native queries on such a driver are refused outright.+## Metabase 0.65.0- `metabase.driver.sql-jdbc.execute/set-role-if-supported!` moved todiff --git a/docs/permissions/impersonation.md b/docs/permissions/impersonation.mdindex c52b6c710aa..814bece705e 100644--- a/docs/permissions/impersonation.md+++ b/docs/permissions/impersonation.md@@ -179,6 +179,12 @@ Examples of queries Metabase will reject:- Statements that create or change tables, including temporary tables (`CREATE`, `ALTER`, `DROP`)- Role-switching statements like `SET ROLE` or `USE ROLE`+### Very large hand-written value lists are rejected++To validate a native query safely, Metabase must parse the whole statement first. A query with a very large literal list written directly in the SQL—a `VALUES` clause, `IN` list, or `ARRAY` literal with more than 2,000 rows or items—can't be validated, so Metabase rejects it rather than run something it can't check. You'll see a message asking you to reduce the size of the list.++This limit only applies to values you type into the SQL itself. [Filter widgets and field filters](../questions/native-editor/sql-parameters.md) pass their values to the database separately, so dashboard and question filters with long lists of values keep working. If you need to match against a large set of hand-written values, load them into a table and join to it instead of listing them inline.+## Metabase gives people the most permissive access to data across all of their groupsSo if a person is in two groups with different permissions for the same database:diff --git a/enterprise/backend/src/metabase_enterprise/impersonation/middleware.clj b/enterprise/backend/src/metabase_enterprise/impersonation/middleware.cljindex fbce2930e8b..90315e0191e 100644--- a/enterprise/backend/src/metabase_enterprise/impersonation/middleware.clj+++ b/enterprise/backend/src/metabase_enterprise/impersonation/middleware.clj@@ -3,13 +3,41 @@[metabase-enterprise.impersonation.driver :as impersonation.driver][metabase.api.common :as api][metabase.driver :as driver]+ [metabase.driver.util :as driver.u]+ [metabase.lib.core :as lib][metabase.lib.metadata :as lib.metadata][metabase.premium-features.core :as premium-features :refer [defenterprise]]+ [metabase.query-processor.error-type :as qp.error-type][metabase.query-processor.interface :as qp.i];; legacy usage -- don't do things like this going forward^{:clj-kondo/ignore [:deprecated-namespace :discouraged-namespace]} [metabase.query-processor.store :as qp.store][metabase.util.i18n :refer [tru]]))+(defn- validates-native-queries?+ "Does `driver` supply its own `driver/validate-impersonated-query`?"+ [driver]+ (not= (get-method driver/validate-impersonated-query driver)+ (get-method driver/validate-impersonated-query :default)))++(defn- assert-native-validation-supported+ "Refuse a native query on a driver whose validation resolves to the `:default` no-op, rather than running+ it unchecked. MBQL stages are compiler-generated, so they pass through — only native SQL is checked."+ [driver query]+ (when (and (lib/any-native-stage? query)+ (not (validates-native-queries? driver)))+ (throw (ex-info (tru "Connection impersonation does not support native queries on this database.")+ {:type qp.error-type/invalid-query}))))++(defn- validate-impersonated-query+ "Enforce the single-statement rule on `query` for `driver`."+ [driver database query]+ (if (driver.u/supports? driver :connection-impersonation database)+ (do (assert-native-validation-supported driver query)+ (driver/validate-impersonated-query+ driver+ (cond-> query api/*is-superuser?* (assoc :impersonation/admin? true))))+ query))+(defenterprise apply-impersonation"Pre-processing middleware. Validates that native queries on impersonated databases are single SELECT statements,and adds an impersonation role key to the query for non-admin users. Currently used solely for caching."@@ -27,9 +55,7 @@impersonation-enabled?(as-> q(do (premium-features/assert-has-feature :advanced-permissions (tru "Advanced Permissions"))- (driver/validate-impersonated-query- driver/*driver*- (cond-> q api/*is-superuser?* (assoc :impersonation/admin? true)))))+ (validate-impersonated-query driver/*driver* database q)));; Only assign the role for non-admin impersonated usersrole(assoc :impersonation/role role)))))diff --git a/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj b/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.cljindex ac9bb6db366..e594f9da75d 100644--- a/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj+++ b/enterprise/backend/test/metabase_enterprise/impersonation/driver_test.clj@@ -6,6 +6,7 @@[clojure.string :as str][clojure.test :refer :all][metabase-enterprise.impersonation.driver :as impersonation.driver]+ [metabase-enterprise.impersonation.middleware :as impersonation.middleware][metabase-enterprise.impersonation.util-test :as impersonation.util-test][metabase-enterprise.test :as met][metabase.actions.execution :as actions.execution]@@ -1155,7 +1156,7 @@(with-impersonation-db!(impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]:attributes {"impersonation_attr" role-a}}- (testing "A single SELECT statement works with normal impersonation"+ (testing "A single select statement works with normal impersonation"(let [mp (mt/metadata-provider)run-native-query (fn [table] (->> (-> (lib/query mp (lib.metadata/table mp (mt/id table)))(lib/aggregate (lib/count)))@@ -1195,7 +1196,7 @@(impersonation.util-test/with-impersonations-for-user! :crowberto{:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]:attributes {"impersonation_attr" role-a}}- (testing "A single SELECT statement works for admin"+ (testing "A single select statement works for admin"(let [mp (mt/metadata-provider)run-native-query (fn [table] (->> (-> (lib/query mp (lib.metadata/table mp (mt/id table)))(lib/aggregate (lib/count)))@@ -1335,8 +1336,7 @@(let [validate (fn [sql](try(driver.sql/validate-impersonated-query*- :postgres- {:stages [{:lib/type :mbql.stage/native :native sql}]})+ :postgres (lib/native-query meta/metadata-provider sql))nil(catch clojure.lang.ExceptionInfo e e)))](testing "syntax error in a CTE produces a parse-error message, not the single-select message"@@ -1362,7 +1362,7 @@(deftest ^:parallel validate-impersonated-query-keys-on-allow-write-flag-test(testing "validate-impersonated-query* derives read-vs-write from *impersonation-allow-write?*"- (let [query (fn [sql] {:stages [{:lib/type :mbql.stage/native :native sql}]})+ (let [query (fn [sql] (lib/native-query meta/metadata-provider sql))outcome (fn [q](try(driver.sql/validate-impersonated-query* :postgres q)@@ -1381,7 +1381,7 @@(deftest validate-impersonated-query-is-enforced-for-all-impersonation-drivers-test(testing "every driver that supports connection-impersonation enforces the single-statement guard (SEC-1189)"(mt/test-drivers (mt/normal-drivers-with-feature :connection-impersonation)- (let [query (fn [sql] {:stages [{:lib/type :mbql.stage/native :native sql}]})]+ (let [query (fn [sql] {:lib/type :mbql/query :stages [{:lib/type :mbql.stage/native :native sql}]})](testing "a multi-statement native query is rejected"(is (thrown-with-msg?clojure.lang.ExceptionInfo #"single select statement"@@ -1390,3 +1390,210 @@(let [result (driver/validate-impersonated-query :mysql (query "SELECT 1"))](is (map? result))(is (string? (get-in result [:stages 0 :native])))))))))++(deftest validate-impersonated-query-inherited-by-sql-drivers-test+ (testing "every SQL driver inherits the single-statement validator via the :sql method"+ ;; The validator used to be registered on `:sql-jdbc`, so a SQL driver that is not JDBC-backed resolved+ ;; to the `:default` no-op instead. It is now a `:sql` defmethod, and every driver advertising+ ;; connection impersonation is a `:sql-jdbc` (hence `:sql`) descendant, so they all inherit it.+ (let [default-method (get-method driver/validate-impersonated-query :default)+ sql-method (get-method driver/validate-impersonated-query :sql)]+ (is (some? sql-method) "a :sql validator must be registered")+ (is (not= default-method sql-method) "the :sql validator must be distinct from the :default no-op")+ (testing "sql-jdbc drivers resolve to the :sql validator, not the no-op default"+ ;; A driver only takes its place under :sql in the hierarchy once its namespace has loaded, and+ ;; module drivers are only on the classpath when their alias is active. Initialize each driver we+ ;; can, then assert; skip the ones this run can't load. :h2 is always present.+ (doseq [d [:h2 :postgres :redshift :sqlserver :snowflake :mysql :clickhouse :starburst]+ :when (try (driver/the-initialized-driver d) true (catch Throwable _ false))]+ (testing d+ (is (= sql-method (get-method driver/validate-impersonated-query d))+ "must inherit the shared validator, not fall through to the :default no-op")))))))++(deftest ^:parallel impersonation-refuses-native-without-a-validator-test+ (testing "native is refused when the driver has no validate-impersonated-query of its own"+ ;; The `:sql` method covers every sql-jdbc driver, but a future impersonation driver that does not+ ;; descend from `:sql` would resolve to the `:default` no-op, which says nothing about the native SQL.+ ;; Refusing surfaces that as an error rather than letting the check be skipped silently.+ (let [guard #'impersonation.middleware/assert-native-validation-supported+ native (lib/native-query meta/metadata-provider "SELECT 1")+ mbql (lib/query meta/metadata-provider (meta/table-metadata :orders))]+ (testing "a driver with no validator (resolves to the :default no-op) refuses a native query"+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"does not support native queries"+ (guard ::driver-without-validator native))))+ (testing "the same driver lets an MBQL query through — only native SQL is checked"+ (is (nil? (guard ::driver-without-validator mbql))))+ (testing "a driver that supplies the :sql validator is not refused"+ (driver/the-initialized-driver :h2)+ (is (nil? (guard :h2 native)))))))++(deftest impersonation-skips-native-validation-on-non-impersonating-driver-test+ (testing "a driver without :connection-impersonation never switches roles, so a policy on it does not validate (or refuse) native SQL"+ ;; `set-role-if-supported!` is gated on the same feature check, so on such a driver everyone runs as the full+ ;; connection user regardless of the policy. H2 also has no SQLGlot dialect, so if the validator ran here it+ ;; would refuse every non-admin native query outright.+ (mt/test-driver :h2+ (mt/with-premium-features #{:advanced-permissions}+ (impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]+ :attributes {"impersonation_attr" "impersonation_role"}}+ (is (= [[1]]+ (mt/rows (qp/process-query (lib/native-query (mt/metadata-provider) "SELECT 1"))))))))))++(deftest ^:parallel impersonated-native-large-literal-list-test+ (testing "impersonated native queries with large literal lists"+ ;; `validate-impersonated-query*` validates the ORIGINAL SQL, so a large but ordinary literal list —+ ;; the values a Number variable inlines for `IN ({{ids}})` — validates normally, a second statement+ ;; nested in such a list is counted because the parser sees the real input, and only a list beyond the+ ;; memory ceiling is refused. Returns the executed :native string when accepted, or [::refused type]+ ;; when not.+ (let [validate (fn [sql]+ (try+ (lib/raw-native-query+ (driver.sql/validate-impersonated-query*+ :postgres (lib/native-query meta/metadata-provider sql)))+ (catch clojure.lang.ExceptionInfo e [::refused (:type (ex-data e))])))+ big (str "VALUES " (str/join "," (repeat 101 "(1)")))]+ (testing "a second statement nested in an oversized VALUES list is refused however it is written"+ ;; parsing the original, sqlglot counts the statements — so it does not matter whether the extra `;`+ ;; sits inside a line comment, a block comment, a dollar-quoted body, or an E'' escape, each of+ ;; which can unbalance the JVM-side paren scan that decides whether to collapse the list.+ (doseq [[label tail] [[:line-comment ",( -- ((\n 1 )) t ; SELECT 2 --)) t"]+ [:block-comment ",( /* (( */ 1 )) t ; SELECT 2 /* )) t */"]+ [:dollar-quote ",( 1 )) t ; DO $tag$ BEGIN PERFORM 1; END $tag$ --)) t"]+ [:e-escape ",( 1 )) t ; SELECT E'\\')' ; SELECT 2 --)) t"]]]+ (testing label+ (is (= [::refused qp.error-type/invalid-query] (validate (str "SELECT * FROM (" big tail)))))))+ (testing "an ordinary large-but-under-ceiling list validates and returns the reconstructed statement"+ (is (string? (validate (str "SELECT x FROM (" big ") t(x)"))))+ (binding [driver.settings/*impersonation-allow-write?* true]+ (is (string? (validate (str "INSERT INTO t (" big ")"))))))+ (testing "a list beyond the memory ceiling is refused"+ (is (= [::refused qp.error-type/invalid-query]+ (validate (str "SELECT x FROM t WHERE x IN (" (str/join "," (range 2500)) ")")))))+ (testing "a query with no large literal list validates and executes its reconstruction"+ (is (string? (validate "SELECT x FROM (VALUES (1),(2),(3)) t(x)")))))))++(deftest ^:parallel impersonated-native-join-source-is-validated-test+ (testing "a native query used as a join source is validated like a top-level native stage"+ ;; sql.qp inlines a native join source verbatim as a sub-select, so it needs the same single-statement+ ;; check as a top-level native stage. The validator must walk join stages, not just the top-level+ ;; `:stages`.+ (let [mp meta/metadata-provider+ orders (lib/query mp (meta/table-metadata :orders))+ [a b] (lib/visible-columns orders)+ join-query (fn [native-sql]+ ;; an MBQL query whose join source is a native query — the join carries its own+ ;; `:stages`, which the top-level walk would miss.+ (lib/join orders (-> (lib/join-clause (lib/native-query mp native-sql) [(lib/= a b)])+ (lib/with-join-alias "j"))))]+ (testing "a second statement in a native join source is rejected"+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"single select statement"+ (driver.sql/validate-impersonated-query* :postgres (join-query "SELECT 1; DROP TABLE t")))))+ (testing "a bare role-switch in a native join source is rejected"+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"single select statement"+ (driver.sql/validate-impersonated-query* :postgres (join-query "SET ROLE NONE")))))+ (testing "a single SELECT native join source is accepted (the join stage is walked and passes)"+ (is (some? (driver.sql/validate-impersonated-query* :postgres (join-query "SELECT 1"))))))))++(deftest ^:parallel impersonated-admin-not-refused-for-uncollapsible-query-test+ (testing "an admin's native query is left untouched when it can't be collapsed and parsed"+ ;; A superuser is never actually impersonated (`connection-impersonation-role` returns nil for them), so+ ;; validating their native SQL changes nothing. A query we merely cannot collapse-then-parse must not be+ ;; refused for an admin where it used to run — the same exemption they already have for+ ;; non-single-statement queries.+ (let [big (str "SELECT x FROM (VALUES " (str/join "," (repeat 2500 "(1)")) ") t(x)")+ run (fn [admin?]+ (driver.sql/validate-impersonated-query*+ :postgres (cond-> (lib/native-query meta/metadata-provider big)+ admin? (assoc :impersonation/admin? true))))]+ (testing "a non-admin is still refused"+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"Reduce the size" (run false))))+ (testing "an admin runs the original SQL unchanged"+ (is (= big (lib/raw-native-query (run true))))))))++(deftest ^:parallel impersonated-refusal-preserves-reason-and-names-array-test+ (testing "the unchecked-SQL refusal carries its :reason and names ARRAY as well as VALUES/IN"+ (let [stripped (str "SELECT x FROM t WHERE x = ANY(ARRAY[" (str/join "," (range 2500)) "])")+ e (try (driver.sql/validate-impersonated-query*+ :postgres (lib/native-query meta/metadata-provider stripped))+ nil+ (catch clojure.lang.ExceptionInfo ex ex))]+ (is (some? e) "an oversized ARRAY literal must be refused")+ (is (= "large-literal-list" (:reason (ex-data e))) "the underlying :reason is preserved for operators")+ (is (re-find #"(?i)ARRAY" (ex-message e)) "the message names ARRAY, which also triggers stripping"))))++(deftest impersonated-native-single-statement-enforced-for-all-drivers-test+ (testing "a multi-statement native query is rejected for every driver that advertises connection impersonation"+ (mt/test-drivers (mt/normal-drivers-with-feature :connection-impersonation)+ (mt/with-premium-features #{:advanced-permissions}+ ;; Validation is pure parsing, so this exercises each driver's dialect and the shared `:sql`+ ;; validator without needing a live warehouse round-trip.+ (let [validate (fn [sql]+ (try+ (driver/validate-impersonated-query+ driver/*driver* (lib/native-query meta/metadata-provider sql))+ :accepted+ (catch clojure.lang.ExceptionInfo e (:type (ex-data e)))))]+ (testing "a plain single SELECT is accepted"+ (is (= :accepted (validate "SELECT 1"))))+ (testing "a second statement is rejected"+ (is (= qp.error-type/invalid-query (validate "SELECT 1; DROP TABLE t"))))+ (testing "a bare role-switch is rejected"+ (is (= qp.error-type/invalid-query (validate "SET ROLE NONE")))))))))++(deftest impersonated-native-multi-statement-rejected-test+ (testing "a non-admin's multi-statement native query is rejected, comment-nested separator and all"+ ;; End-to-end check that the rejection holds through the real QP preprocess chain, and that it happens+ ;; *before* execution: we ask the warehouse directly rather than trusting the QP status, so that a+ ;; refactor moving validation after execution begins would fail here.+ (mt/test-drivers (mt/normal-driver-select {:+parent :postgres})+ (mt/with-premium-features #{:advanced-permissions}+ (let [venues-table (sql.tx/qualify-and-quote driver/*driver* "test-data" "venues")+ role-a (u/lower-case-en (mt/random-name))]+ (tx/with-temp-roles! driver/*driver*+ (impersonation-granting-details driver/*driver* (mt/db))+ {role-a {venues-table {}}}+ (impersonation-default-user driver/*driver*)+ (impersonation-default-role driver/*driver*)+ (let [granting-details (impersonation-granting-details driver/*driver* (mt/db))+ spec (sql-jdbc.conn/connection-details->spec driver/*driver* granting-details)+ ;; The second statement creates a table instead of changing data: whether that table+ ;; exists afterwards is the ground truth for "a second statement ran", and observing it+ ;; costs no existing row. The impersonated role is granted nothing but `venues`, so the+ ;; table can only appear if the statement ran as the connection user.+ check-table (str "mb_single_stmt_check_" (u/lower-case-en (mt/random-name)))+ check-table? (fn []+ (boolean (seq (jdbc/query spec ["SELECT 1 FROM information_schema.tables WHERE table_name = ?"+ check-table]))))+ ;; 101 tuples push the VALUES list past the collapse threshold, and the 102nd opens a+ ;; line comment that unbalances the JVM-side paren scan — so the collapsed form reads as+ ;; one SELECT while the real input is two statements. Parsing the real input catches it.+ sql (str "SELECT * FROM (VALUES " (str/join "," (repeat 101 "(1)"))+ ",( -- ((\n 1 )) t ; SET ROLE NONE ; CREATE TABLE " check-table+ " (id integer) --)) t")]+ (mt/with-temp [:model/Database database {:engine driver/*driver*+ :details (impersonation-details driver/*driver* (mt/db))}]+ (mt/with-db database+ (when (driver/database-supports? driver/*driver* :connection-impersonation-requires-role nil)+ (t2/update! :model/Database :id (mt/id) (assoc-in (mt/db) [:details :role] (impersonation-default-role driver/*driver*))))+ (sync/sync-database! (:id database) {:scan :schema})+ (impersonation.util-test/with-impersonations! {:impersonations [{:db-id (mt/id) :attribute "impersonation_attr"}]+ :attributes {"impersonation_attr" role-a}}+ (try+ (testing "the query is rejected"+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"(?i)single select statement|reduce the size"+ (-> (lib/native-query (mt/metadata-provider) sql)+ (qp/process-query)+ (mt/rows)))))+ (testing "the warehouse never saw the second statement"+ (is (not (check-table?))))+ (finally+ (jdbc/execute! spec [(format "DROP TABLE IF EXISTS %s" check-table)])))))))))))))diff --git a/src/metabase/driver/sql.clj b/src/metabase/driver/sql.cljindex b6acb3d4aed..a8e637741fb 100644--- a/src/metabase/driver/sql.clj+++ b/src/metabase/driver/sql.clj@@ -337,6 +337,51 @@native-query :- :metabase.lib.schema/native-only-query](sql-tools/validate-query driver native-query))+(defn- validate-impersonated-native-stage+ "Validate one native `stage` for `driver`, returning it with `:native` replaced by the reconstructed,+ validated SQL — or throwing a [[metabase.query-processor.error-type/invalid-query]] `ex-info` when it must+ be refused.++ `admin?` users run as the full connection user and are never actually impersonated."+ [driver stage admin?]+ (let [write? driver.settings/*impersonation-allow-write?*+ stmt-type (if write? "write" "read")+ {:keys [verdict single-stmt? sql reason error]}+ (sql-tools/validate-executable-stmt driver (:native stage) stmt-type)]+ (cond+ error+ (do+ (log/warnf "Failed to parse native query: %s" error)+ (throw (ex-info (tru "Unable to parse native query. There might be something wrong with your query.")+ {:type qp.error-type/invalid-query+ :sql (:native stage)})))++ ;; The SQL was not checked at all, as opposed to checked and found to be two statements. Admins run as+ ;; the full connection user and are never actually impersonated, so validating their native SQL+ ;; changes nothing: leave such a query untouched for them rather than refuse it. Everyone else is+ ;; refused, with the `:reason` logged so an operator has a thread to pull.+ (= verdict :unknown)+ (if admin?+ stage+ (do+ (log/warnf "Refusing impersonated native query on %s: cannot validate (%s)" driver reason)+ (throw (ex-info (tru "Unable to validate this native query on an impersonated database. Reduce the size of any large VALUES, IN, or ARRAY lists.")+ {:type qp.error-type/invalid-query+ :reason reason+ :sql (:native stage)}))))++ ;; Admins may also run a single statement that is not of the required type (e.g. `SHOW TIMEZONE`),+ ;; but never more than one statement.+ (or (= verdict :allowed) (and single-stmt? admin?))+ (assoc stage :native sql)++ :else+ (throw (ex-info (if write?+ (tru "Invalid impersonated native query. Must be a single insert, update, or delete statement.")+ (tru "Invalid impersonated native query. Must be a single select statement."))+ {:type qp.error-type/invalid-query+ :sql (:native stage)})))))+(defn validate-impersonated-query*"Validates a native query by parsing it and ensuring that it is a single statement.Reads `:impersonation/allow-write?` on the query to decide what to require: when truthy (set by@@ -345,34 +390,30 @@Queries that cannot be parsed are always rejected. Admins are allowed to run parseable queries thataren't a single select/write statement (e.g. `SHOW TIMEZONE`); non-admins are restricted to a single- statement of the required type."+ statement of the required type.++ Throws a [[metabase.query-processor.error-type/invalid-query]] `ex-info` when the query is rejected —+ including when the validator refused without checking the SQL at all (a large VALUES/IN/ARRAY list it+ would have to collapse before parsing, or an unknown dialect), which it refuses for non-admins rather+ than running a string it did not validate. Admins are exempt from that refusal: they run as the full+ connection user, so refusing a query we merely cannot collapse-then-parse changes nothing.+ Registered as the `:sql` method of [[driver/validate-impersonated-query]], so every SQL driver inherits+ it."+ [driver query]+ (let [admin? (boolean (:impersonation/admin? query))]+ ;; Walk *every* stage, including those nested inside joins — a native query used as a join source is+ ;; inlined verbatim as a sub-select, so it needs the same single-statement guard as a top-level native+ ;; stage. `walk-stages` returns nil for a stage to leave it unchanged, so non-native stages+ ;; pass straight through.+ (lib/walk-stages+ query+ (fn [_query _path stage]+ (when (lib.util/native-stage? stage)+ (validate-impersonated-native-stage driver stage admin?))))))++(defmethod driver/validate-impersonated-query :sql[driver query]- (update query :stages- (fn [stages]- (mapv (fn [stage]- (if (lib.util/native-stage? stage)- (let [[stmt-type allowed-stmts] (if driver.settings/*impersonation-allow-write?*- ["write" (tru "insert, update, or delete")]- ["read" (tru "select")])- {:keys [is-single-stmt? allowed-stmt-type? sql error]}- (sql-tools/is-single-stmt-of-type? driver (:native stage) stmt-type)]- (cond error- (do- (log/warnf "Failed to parse native query: %s" error)- (throw (ex-info (tru "Unable to parse native query. There might be something wrong with your query.")- {:type qp.error-type/invalid-query- :sql (:native stage)})))-- (and is-single-stmt?- (or allowed-stmt-type? (:impersonation/admin? query)))- (assoc stage :native sql)-- :else- (throw (ex-info (tru "Invalid impersonated native query. Must be a single {0} statement." allowed-stmts)- {:type qp.error-type/invalid-query- :sql (:native stage)}))))- stage))- stages))))+ (validate-impersonated-query* driver query));;; +----------------------------------------------------------------------------------------------------------------+;;; | Convenience Imports |diff --git a/src/metabase/driver/sql_jdbc.clj b/src/metabase/driver/sql_jdbc.cljindex 2e252fd42f0..869914dec55 100644--- a/src/metabase/driver/sql_jdbc.clj+++ b/src/metabase/driver/sql_jdbc.clj@@ -76,10 +76,6 @@(driver/validate-db-details! driver details)(sql-jdbc.conn/can-connect? driver details))-(defmethod driver/validate-impersonated-query :sql-jdbc- [driver query]- (driver.sql/validate-impersonated-query* driver query))-(defmethod driver/table-rows-seq :sql-jdbc[driver database table](query driver database table {:select [:*]}))diff --git a/src/metabase/lib/core.cljc b/src/metabase/lib/core.cljcindex bb2cd882431..8d11d579e56 100644--- a/src/metabase/lib/core.cljc+++ b/src/metabase/lib/core.cljc@@ -124,6 +124,7 @@[metabase.lib.util :as lib.util][metabase.lib.util.unique-name-generator][metabase.lib.validate :as lib.validate]+ [metabase.lib.walk][metabase.lib.walk.util][metabase.util.malli :as mu][metabase.util.namespaces :as shared.ns]))@@ -1738,6 +1739,8 @@missing-table-errorsyntax-errorvalidation-exception-error]+ [metabase.lib.walk+ walk-stages][metabase.lib.walk.utilall-field-idsall-referenced-entity-idsdiff --git a/src/metabase/sql_parsing/core.clj b/src/metabase/sql_parsing/core.cljindex a5bdb501bf3..732be2c6e62 100644--- a/src/metabase/sql_parsing/core.clj+++ b/src/metabase/sql_parsing/core.clj@@ -107,11 +107,17 @@;; CPython). We strip them on the JVM side before passing SQL to Python, because GraalPy is too;; slow at character-by-character string scanning over multi-MB inputs.-(def ^:private ^:const strip-threshold+(def ^:private ^:const literal-list-strip-threshold"Strip VALUES clauses with more than this many tuples, and IN lists / ARRAY literals with at- least this many commas."+ least this many commas. Tuned for the fail-soft *analysis* callers, which run in bulk background+ passes over stored queries: over-stripping is free there (they never need the literal values), so a+ low cutoff — far below the thousands of items where GraalPy actually OOMs — is the safe default."100)+(def ^:private ^:const executable-literal-list-ceiling+ "The literal-list ceiling for a caller that will execute what it validates."+ 2000)+(def ^:private literal-list-keyword-pattern"Pattern to find a VALUES, IN, or ARRAY keyword followed by its opening delimiter."(re-pattern "(?i)\\b(?:VALUES\\s*\\(|IN\\s*\\(|ARRAY\\s*\\[)"))@@ -200,12 +206,17 @@(defn- strip-values-at"Decide whether to strip the VALUES clause whose `VALUES (` match spans [match-start, match-end).Returns [replacement resume-pos]: a single-row NULL placeholder (column count taken from the- first tuple) covering the SQL up to resume-pos, or [nil match-end] to leave the clause untouched."- [^String sql ^long match-start ^long match-end ^long n]- (let [paren-start (dec match-end)+ first tuple) covering the SQL up to resume-pos, or [nil match-end] to leave the clause untouched.++ Takes `threshold` rather than reading a var so the impersonation validator can ask for its own, much+ higher ceiling; `n` is derived rather than passed because a fn carrying primitive hints is capped at+ four args."+ [^String sql ^long match-start ^long match-end ^long threshold]+ (let [n (.length sql)+ paren-start (dec match-end)first-end (skip-balanced-parens sql paren-start n)[tuple-count end-pos] (count-and-skip-tuples sql first-end n)]- (if (and (> (long tuple-count) strip-threshold)+ (if (and (> (long tuple-count) threshold)(> first-end (inc paren-start)))[(make-null-placeholder (extract-keyword sql match-start match-end)(count-top-level-commas (.substring sql (inc paren-start) (dec first-end))))@@ -241,20 +252,23 @@[match-start, match-end). Returns [replacement resume-pos]: `IN (NULL)` / `ARRAY[NULL]`covering the whole list, or [nil match-end] to leave it untouched (resuming just inside thedelimiter, so a large list nested in a subquery — `IN (SELECT ... WHERE x IN (...))` — is- still found)."- [^String sql ^long match-start ^long match-end ^long n]+ still found).++ Takes `threshold` rather than reading a var, and derives `n`, for the reasons+ on [[strip-values-at]]."+ [^String sql ^long match-start ^long match-end ^long threshold](let [open-pos (dec match-end)- [commas end-pos] (simple-list-end sql open-pos n)]- (if (and commas (>= (long commas) strip-threshold))+ [commas end-pos] (simple-list-end sql open-pos (.length sql))]+ (if (and commas (>= (long commas) threshold))[(str (extract-keyword sql match-start match-end)(if (= (.charAt sql open-pos) \[) "[NULL]" " (NULL)"))end-pos][nil match-end])))(defn- strip-large-literal-lists*- "Single pass over `sql`, replacing every oversized literal list — VALUES clause or literal-only- IN list — with a NULL placeholder. Returns `sql` itself when nothing was stripped."- ^String [^String sql]+ "Single pass over `sql`, replacing every literal list larger than `threshold` — VALUES clause+ or literal-only IN list — with a NULL placeholder. Returns `sql` itself when nothing was stripped."+ ^String [^String sql ^long threshold](let [matcher (re-matcher literal-list-keyword-pattern sql)n (.length sql)](if-not (.find matcher)@@ -269,8 +283,8 @@match-end (.end matcher)[replacement resume] (let [ch (.charAt sql match-start)](if (or (= ch \V) (= ch \v))- (strip-values-at sql match-start match-end n)- (strip-list-at sql match-start match-end n)))]+ (strip-values-at sql match-start match-end threshold)+ (strip-list-at sql match-start match-end threshold)))](if replacement(-> sb (.append sql (int i) (int match-start)) (.append ^String replacement))(.append sb sql (int i) (int resume)))@@ -279,24 +293,29 @@(.find matcher (int resume))))))))))(defn strip-large-literal-lists- "Replace large literal lists with NULL placeholders: VALUES clauses with more than- [[strip-threshold]] tuples become a single-row NULL tuple (preserving the column count from the- first tuple), and simple IN lists / ARRAY literals — numbers, single-quoted strings, and- VALUES-style tuples of them — with at least [[strip-threshold]] commas become `IN (NULL)` /- `ARRAY[NULL]`. All surrounding SQL structure is preserved.+ "Replace large literal lists with NULL placeholders: VALUES clauses with more than `threshold`+ tuples become a single-row NULL tuple (preserving the column count from the first tuple), and simple+ IN lists / ARRAY literals — numbers, single-quoted strings, and VALUES-style tuples of them — with at+ least `threshold` commas become `IN (NULL)` / `ARRAY[NULL]`. All surrounding SQL structure is preserved.+ `threshold` defaults to [[literal-list-strip-threshold]] (the analysis default).This runs on the JVM side (fast) before passing SQL to GraalPy (slow at char scanning).On any error, returns the original SQL unchanged so parsing can proceed normally.- Best-effort by design: stripping only feeds fail-soft analysis, and execution paths restore the- original SQL when anything was stripped ([[is-single-stmt-of-type?]]), so edge cases like- keyword-shaped text inside string literals or comments are deliberately not guarded against."- ^String [^String sql]- (try- (strip-large-literal-lists* sql)- (catch Exception e- (log/warnf "Error stripping large literal lists, passing SQL through unchanged: %s" (ex-message e))- sql)))+ Best-effort by design: stripping feeds fail-soft *analysis* (lineage, referenced tables/fields, returned+ columns), so edge cases like keyword-shaped text inside string literals or comments are deliberately not+ guarded against. A caller that needs its answer to hold for the string it goes on to run must NOT use+ the rewrite: [[validate-executable-stmt]] and [[classify-stmt]] parse the original SQL and call this+ only to find out whether a list is over their ceiling, refusing outright when it is rather than+ answering about a rewrite — those same unguarded edge cases are what make a rewrite unsound to reason+ about."+ (^String [^String sql] (strip-large-literal-lists sql literal-list-strip-threshold))+ (^String [^String sql ^long threshold]+ (try+ (strip-large-literal-lists* sql threshold)+ (catch Exception e+ (log/warnf "Error stripping large literal lists, passing SQL through unchanged: %s" (ex-message e))+ sql))));;; -------------------------------------------------- Public API --------------------------------------------------@@ -574,38 +593,96 @@[dialect sql replacements](protocol/replace-names (parser) dialect sql replacements))-;; Parsing time and memory grow with the length of the SQL, whatever it holds, and the "read-only" check runs on SQL-;; a user or a model wrote.+;;; --------------------------------------- Single-statement classification ----------------------------------------++;; Parsing time and memory grow with the length of the SQL, whatever it holds.;; As of 2026-10-10, the densest SQL that fits, a list of about 8,000 one-character items, takes about a second to;; check, and six such checks at once complete on a 1 GB heap. At 60,000 characters one check took 1 to 3 seconds.-(def ^:private max-read-only-sql-chars- "The longest SQL, in characters, that the \"read-only\" check parses."+(def ^:private ^:const max-gate-sql-chars+ "The longest SQL, in characters, that [[classify-stmt]] parses. A cheap ceiling on work spent for a+ caller that is only gating, and whose SQL a user or a model wrote."(* 16 1024))-(defn is-single-stmt-of-type?- "Validates that a query is a single read statement (SELECT) or a single write statement (INSERT, UPDATE, DELETE)- and returns the query reconstructed from the parsed AST. `stmt-type` is \"read\", \"write\", or \"read-only\": a- read statement that also holds nothing that writes or locks anywhere in its tree.- For \"read-only\", SQL that is not allowed carries a `:reason`, and SQL that is too long or holds a literal list- too large to parse whole is refused without being parsed."+(def ^:private executable-limits+ "Limits for a caller that will execute the `:sql` the verdict carries — [[validate-executable-stmt]].++ Such a caller needs the verdict to hold for the exact string it goes on to run, so a known `dialect` is+ required: `nil` makes SQLGlot parse in its generic dialect and hand back a confident verdict plus a+ generically-regenerated `:sql`, which is not the string that was checked."+ {:max-sql-chars nil+ :literal-list-ceiling executable-literal-list-ceiling+ :require-dialect? true})++(def ^:private gate-limits+ "Limits for a caller that only reads the verdict and never executes its `:sql` — [[classify-stmt]]."+ {:max-sql-chars max-gate-sql-chars+ :literal-list-ceiling literal-list-strip-threshold+ :require-dialect? false})++(defn- unknown+ "A verdict that `sql` could not be checked, for `reason`."+ [sql reason]+ {:verdict :unknown, :reason reason, :sql sql})++(defn- verdict+ "Turn the parser's raw result for `sql` into a verdict map."+ [sql raw]+ (let [{:keys [is-single-stmt? allowed-stmt-type? error reason reason-detail] parsed-sql :sql}+ (perf/update-keys raw (comp keyword u/->kebab-case-en))]+ (cond+ ;; SQL that does not parse is SQL we know nothing about. `:error` carries the parser's message, which+ ;; a caller may want to word differently from the refusals it *can* explain.+ error (assoc (unknown sql "unparseable") :error error)+ allowed-stmt-type? {:verdict :allowed, :single-stmt? true, :sql parsed-sql}+ :else (cond-> {:verdict :disallowed, :single-stmt? (boolean is-single-stmt?)}+ reason (assoc :reason reason)+ reason-detail (assoc :detail reason-detail)+ parsed-sql (assoc :sql parsed-sql)))))++(defn- classify-single-stmt+ "Classify `sql` as a single statement of `stmt-type` for `dialect`, spending no more than `limits` allow.+ Applies `limits` as given; it does not infer them from `stmt-type`."+ [dialect sql stmt-type {:keys [max-sql-chars literal-list-ceiling require-dialect?]}]+ (cond+ ;; The length is checked before anything scans the SQL: stripping literal lists is itself work that+ ;; grows with it.+ (and max-sql-chars (< (long max-sql-chars) (count sql)))+ (unknown sql "too-long")++ (and require-dialect? (nil? dialect))+ (unknown sql "unknown-dialect")++ ;; Stripping is only ever a size guard: if it would collapse a list, parsing that list could OOM the+ ;; shared JVM, so refuse rather than parse the collapsed form and answer about that instead. Stripping+ ;; also does not skip comments, so a collapsed list can swallow a second statement into a comment.+ ;; Under the ceiling nothing is stripped, so we parse exactly the string the caller holds.+ (not= sql (strip-large-literal-lists sql literal-list-ceiling))+ (unknown sql "large-literal-list")++ :else+ (verdict sql (protocol/single-stmt-of-type (parser) dialect sql stmt-type))))++(defn validate-executable-stmt+ "Whether `sql` is a single statement of `stmt-type` for `dialect`, for a caller that will **execute** the+ `:sql` the verdict carries: `\"read\"` = SELECT, `\"write\"` = INSERT, UPDATE, or DELETE,+ `\"read-only\"` = a read statement that also holds nothing that writes or locks anywhere in its tree.++ The `:sql` is reconstructed from the parsed AST, and only ever from `sql` itself — SQL that cannot be+ parsed as it stands comes back `:verdict :unknown` rather than parsed in a collapsed form and vouched+ for in its original one. See [[executable-limits]] for what that costs, and [[classify-stmt]] for the+ same question asked by a caller that only needs a verdict."+ [dialect sql stmt-type]+ (classify-single-stmt dialect sql stmt-type executable-limits))++(defn classify-stmt+ "Whether `sql` is a single statement of `stmt-type` for `dialect` (see [[validate-executable-stmt]] for+ the `stmt-type` values), for a caller that only reads the verdict and never executes its `:sql`.++ Cheaper and more permissive about `dialect` than [[validate-executable-stmt]] — see [[gate-limits]].+ Do not execute the `:sql` this returns: with no `dialect` it is a generic-dialect regeneration, not the+ string that was checked."[dialect sql stmt-type]- ;; The length is checked before anything scans the SQL: stripping literal lists is itself work that grows with it.- (if (and (= stmt-type "read-only") (< max-read-only-sql-chars (count sql)))- {:is-single-stmt? false, :allowed-stmt-type? false, :sql sql, :reason "too-long"}- (let [stripped-sql (strip-large-literal-lists sql)]- ;; Stripping does not skip comments, so it can rewrite a second statement into a commented-out list.- ;; "read-only" must judge the SQL that will run, and parsing a large list unstripped risks a GraalPy OOM, so it- ;; refuses.- (if (and (= stmt-type "read-only") (not= sql stripped-sql))- {:is-single-stmt? false, :allowed-stmt-type? false, :sql sql, :reason "large-literal-list"}- (let [result (-> (protocol/single-stmt-of-type (parser) dialect stripped-sql stmt-type)- (perf/update-keys (comp keyword u/->kebab-case-en)))]- ;; The `:sql` in the `result` is the reconstructed SQL from the SQLGlot parser.- ;; We generally want to use the reconstructed SQL, but if the original SQL had its VALUES/IN- ;; literal lists stripped (to avoid GraalPy OOM) then we need to return the original SQL to- ;; preserve the values. (#74284)- (cond-> result- (not= sql stripped-sql) (assoc :sql sql)))))))+ (classify-single-stmt dialect sql stmt-type gate-limits))(comment(referenced-tables "postgres" "select * from transactions")diff --git a/src/metabase/sql_tools/core.clj b/src/metabase/sql_tools/core.cljindex 59cb4ba9494..4164b526bb9 100644--- a/src/metabase/sql_tools/core.clj+++ b/src/metabase/sql_tools/core.clj@@ -233,10 +233,28 @@(interface/transpile-sql-impl (sql-tools.settings/sql-tools-parser-backend)sql from-dialect to-dialect))-(defn is-single-stmt-of-type?- "Wrapper around `sql-parsing/is-single-stmt-of-type?`."+(defn validate-executable-stmt+ "Whether `sql` is a single statement of `stmt-type` for `driver`'s dialect, for a caller that will+ **execute** the `:sql` the verdict carries: `\"read\"` = SELECT, `\"write\"` = INSERT, UPDATE, or+ DELETE, `\"read-only\"` = a read statement that also holds nothing that writes or locks anywhere in its+ tree (see [[read-only-select?]]).++ The `:sql` is reconstructed from the parsed AST, and only ever from `sql` itself — SQL that cannot be+ parsed as it stands (`driver` has no known dialect, or a literal list is large enough to risk a GraalPy+ OOM) comes back `:verdict :unknown` rather than parsed in a collapsed form and vouched for in its+ original one. Only `:verdict :allowed` is permission to proceed.++ See [[metabase.sql-parsing.core/validate-executable-stmt]] for the verdict shape, and [[classify-stmt]]+ for the same question asked by a caller that only needs a verdict."[driver sql stmt-type]- (sql-parsing/is-single-stmt-of-type? (sqlglot/driver->dialect driver) sql stmt-type))+ (sql-parsing/validate-executable-stmt (sqlglot/driver->dialect driver) sql stmt-type))++(defn classify-stmt+ "Whether `sql` is a single statement of `stmt-type` for `driver`'s dialect, for a caller that only reads+ the verdict and never executes its `:sql`. Cheaper and more permissive about the dialect than+ [[validate-executable-stmt]]; do not execute the `:sql` it returns."+ [driver sql stmt-type]+ (sql-parsing/classify-stmt (sqlglot/driver->dialect driver) sql stmt-type))(mu/defn read-only-select? :- :boolean"Whether `sql` is exactly one plain query, a SELECT or a set operation of SELECTs, holding nothing known to write,@@ -249,7 +267,7 @@A function this does not name, a user-defined one included, passes: only the database's own permissions stop it."[driver :- :keywordsql :- :string]- (true? (:allowed-stmt-type? (is-single-stmt-of-type? driver sql "read-only"))))+ (= :allowed (:verdict (classify-stmt driver sql "read-only"))))(mu/defn read-only-select-problem :- [:maybe [:map [:reason :keyword] [:detail [:maybe :string]]]]"Why `sql` is not a read-only select ([[read-only-select?]]), or nil when it is one.@@ -260,7 +278,7 @@function's name."[driver :- :keywordsql :- :string]- (let [{:keys [allowed-stmt-type? reason reason-detail]} (is-single-stmt-of-type? driver sql "read-only")]- (when-not (true? allowed-stmt-type?)+ (let [{:keys [verdict reason detail]} (classify-stmt driver sql "read-only")]+ (when-not (= :allowed verdict){:reason (keyword (or reason "unparseable"))- :detail reason-detail})))+ :detail detail})))diff --git a/test/metabase/driver/postgres_test.clj b/test/metabase/driver/postgres_test.cljindex a95ef286cd6..7ecbc6501aa 100644--- a/test/metabase/driver/postgres_test.clj+++ b/test/metabase/driver/postgres_test.clj@@ -466,9 +466,8 @@(let [parent (h2x/identifier :field "public" "test_table" "data")nfc-field {:nfc-path ["data" "key"] :database-type "text"}[json-sql] (sql/format-expr (#'sql.qp/json-query :postgres parent nfc-field))- stage {:lib/type :mbql.stage/native :native (str "SELECT " json-sql " AS k FROM t")}- out-sql (-> (driver.sql/validate-impersonated-query* :postgres {:stages [stage]})- :stages first :native)]+ query (lib/native-query meta/metadata-provider (str "SELECT " json-sql " AS k FROM t"))+ out-sql (lib/raw-native-query (driver.sql/validate-impersonated-query* :postgres query))];; The broken reconstruction looks like `CAST(parent #>> array[?] AS TEXT[])` — i.e.;; the cast wraps the entire `#>>` expression instead of just the path argument.(is (not (re-find #"(?i)#>>\s+array\s*\[\?\]\s+AS\s+TEXT\s*\[\s*\]" out-sql))diff --git a/test/metabase/sql_parsing/core_test.clj b/test/metabase/sql_parsing/core_test.cljindex 048dffc2f64..09ab5c19e17 100644--- a/test/metabase/sql_parsing/core_test.clj+++ b/test/metabase/sql_parsing/core_test.clj@@ -96,13 +96,16 @@(sort (keep :column (:returned-fields result))))(pr-str sql)))))))-;;; --------------------------------------------- is-single-stmt-of-type? ------------------------------------------+;;; ------------------------------------------ single-stmt classification -------------------------------------------(deftest ^:parallel is-single-stmt-of-type?-test+(deftest ^:parallel single-stmt-classification-test(testing "whether a query is one statement of the kind asked for, which is what tells a caller the rest is unread"(doseq [{:keys [name sql dialect stmt-type expected]} (cases "single_stmt")](testing name- (is (=? expected (sql-parsing/is-single-stmt-of-type? dialect sql stmt-type))+ (is (=? expected (sql-parsing/validate-executable-stmt dialect sql stmt-type))+ (pr-str sql))+ (is (= (sql-parsing/validate-executable-stmt dialect sql stmt-type)+ (sql-parsing/classify-stmt dialect sql stmt-type))(pr-str sql))))));;; ------------------------------------------- returned-columns-lineage -------------------------------------------@@ -175,6 +178,14 @@(is (= [[nil nil "accounts"]] (sql-parsing/referenced-tables "postgres" sql)))(is (= [[nil nil "accounts" "id"] [nil nil "accounts" "name"]](sort (sql-parsing/referenced-fields "postgres" sql))))))- (testing "and is one statement still"- (let [sql (str "SELECT * FROM t WHERE id IN (" (str/join ", " (range 20000)) ")")]- (is (=? {:is-single-stmt? true} (sql-parsing/is-single-stmt-of-type? "postgres" sql "read"))))))+ (testing "but a classification never answers about a stripped rewrite"+ ;; Analysis can read a stripped query because it only needs the shape. A classification's answer has to+ ;; hold for the string the caller holds, so it parses the original and reports SQL over its ceiling as+ ;; unchecked rather than answering about a rewrite.+ (let [under (str "SELECT * FROM t WHERE id IN (" (str/join ", " (range 150)) ")")+ over (str "SELECT * FROM t WHERE id IN (" (str/join ", " (range 20000)) ")")]+ (testing "a list the analysis threshold would strip still validates as one statement"+ (is (=? {:verdict :allowed} (sql-parsing/validate-executable-stmt "postgres" under "read"))))+ (testing "a list past the ceiling comes back unchecked, not stripped and vouched for"+ (is (= {:verdict :unknown, :reason "large-literal-list", :sql over}+ (sql-parsing/validate-executable-stmt "postgres" over "read")))))))diff --git a/test/metabase/sql_parsing/test_cases/single_stmt.edn b/test/metabase/sql_parsing/test_cases/single_stmt.ednindex 3d2166e5c08..ae90beb3b69 100644--- a/test/metabase/sql_parsing/test_cases/single_stmt.edn+++ b/test/metabase/sql_parsing/test_cases/single_stmt.edn@@ -3,40 +3,40 @@:dialect "postgres":sql "SELECT * FROM orders":stmt-type "read"- :expected {:is-single-stmt? true :allowed-stmt-type? true}}+ :expected {:verdict :allowed :single-stmt? true}}{:name "two statements are not a single statement":dialect "postgres":sql "SELECT * FROM orders; DELETE FROM people":stmt-type "read"- :expected {:is-single-stmt? false :allowed-stmt-type? false}}+ :expected {:verdict :disallowed :single-stmt? false}}{:name "a trailing semicolon does not make a second statement":dialect "postgres":sql "SELECT * FROM orders;":stmt-type "read"- :expected {:is-single-stmt? true :allowed-stmt-type? true}}+ :expected {:verdict :allowed :single-stmt? true}}{:name "a semicolon inside a literal is not a statement boundary":dialect "postgres":sql "SELECT * FROM orders WHERE note = 'a; b'":stmt-type "read"- :expected {:is-single-stmt? true :allowed-stmt-type? true}}+ :expected {:verdict :allowed :single-stmt? true}}{:name "a write is a single statement, but not a read":dialect "postgres":sql "DELETE FROM people WHERE id = 1":stmt-type "read"- :expected {:is-single-stmt? true :allowed-stmt-type? false}}+ :expected {:verdict :disallowed :single-stmt? true}}{:name "the same write, asked for as a write":dialect "postgres":sql "DELETE FROM people WHERE id = 1":stmt-type "write"- :expected {:is-single-stmt? true :allowed-stmt-type? true}}+ :expected {:verdict :allowed :single-stmt? true}}{:name "a CTE ahead of a select is still one read":dialect "postgres":sql "WITH c AS (SELECT id FROM people) SELECT * FROM c":stmt-type "read"- :expected {:is-single-stmt? true :allowed-stmt-type? true}}]}+ :expected {:verdict :allowed :single-stmt? true}}]}diff --git a/test/metabase/sql_tools/core_test.clj b/test/metabase/sql_tools/core_test.cljindex 7c64709b725..d0ec054093b 100644--- a/test/metabase/sql_tools/core_test.clj+++ b/test/metabase/sql_tools/core_test.clj@@ -212,7 +212,7 @@(is (= :skipped status))(is (= :missing-dialect reason))))))-(deftest ^:parallel is-single-stmt-of-type?-test+(deftest ^:parallel validate-executable-stmt-test(mt/test-drivers (mt/normal-drivers-with-feature :connection-impersonation)(let [mp (mt/metadata-provider)products (lib.metadata/table mp (mt/id :products))@@ -221,32 +221,39 @@(lib/filter (lib/= product-category "Widget")))native-query (:query (qp.compile/compile-with-inline-parameters query))](testing "A single SELECT statement returns true and the reconstructed SQL"- (are [sql] (=? {:is-single-stmt? true :allowed-stmt-type? true :sql string?}- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "read"))+ (are [sql] (=? {:verdict :allowed :single-stmt? true :sql string?}+ (sql-tools/validate-executable-stmt driver/*driver* sql "read"))native-query"SELECT 1""SELECT * FROM table""WITH x AS (SELECT * FROM foo) SELECT * from x""WITH x AS (SELECT a FROM foo), y AS (SELECT b FROM bar), z AS (SELECT c FROM baz) SELECT x.a, y.b, z.c FROM x, y, z")))(testing "All other read queries are rejected"- (are [sql is-single-stmt?] (=? {:is-single-stmt? is-single-stmt? :allowed-stmt-type? false}- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "read"))- "SELECT (" false+ (are [sql single-stmt?] (=? {:verdict :disallowed :single-stmt? single-stmt?}+ (sql-tools/validate-executable-stmt driver/*driver* sql "read"))"SELECT 1; SELECT 2" false"SET ROLE NONE" true"DROP TABLE table" true"SET ROLE NONE; DROP TABLE table" false- "SELECT set_config('role', 'none', false); DROP TABLE table" false- "DO $$ BEGIN EXECUTE 'SET ROLE NONE; DROP TABLE table'; END $$;" (isa? driver/hierarchy driver/*driver* :postgres)))+ "SELECT set_config('role', 'none', false); DROP TABLE table" false))+ (testing "SQL the dialect cannot parse is :unknown, not :disallowed — nothing was read, so nothing is claimed"+ (is (=? {:verdict :unknown :reason "unparseable"}+ (sql-tools/validate-executable-stmt driver/*driver* "SELECT (" "read"))))+ (testing "a DO block smuggling SET ROLE is refused: one disallowed statement to a dialect that reads PL/pgSQL, unparseable to the rest"+ (is (=? (if (isa? driver/hierarchy driver/*driver* :postgres)+ {:verdict :disallowed :single-stmt? true}+ {:verdict :unknown :reason "unparseable"})+ (sql-tools/validate-executable-stmt+ driver/*driver* "DO $$ BEGIN EXECUTE 'SET ROLE NONE; DROP TABLE table'; END $$;" "read"))))(testing "A single insert, update or delete statement returns true and the reconstructed SQL"- (are [sql] (=? {:is-single-stmt? true :allowed-stmt-type? true :sql string?}- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "write"))+ (are [sql] (=? {:verdict :allowed :single-stmt? true :sql string?}+ (sql-tools/validate-executable-stmt driver/*driver* sql "write"))"INSERT INTO table VALUES (1)""UPDATE table SET column = 1""DELETE FROM table WHERE id = 1"))(testing "All other write queries are rejected"- (are [sql is-single-stmt?] (=? {:is-single-stmt? is-single-stmt? :allowed-stmt-type? false}- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "write"))+ (are [sql single-stmt?] (=? {:verdict :disallowed :single-stmt? single-stmt?}+ (sql-tools/validate-executable-stmt driver/*driver* sql "write"))"SELECT 1" true"INSERT INTO table VALUES (1); SELECT 1" false"UPDATE table SET column = 1; SELECT 1" false@@ -257,10 +264,10 @@(doseq [op ["UNION ALL" "INTERSECT ALL" "EXCEPT ALL"]ts [["foo" "bar"] ["foo" "bar" "baz"]]:let [sql (str/join (str " " op " ") (map #(str "SELECT * FROM " %) ts))]]- (is (=? {:is-single-stmt? true, :sql string?}- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "read"))))- (are [sql] (=? {:is-single-stmt? true, :allowed-stmt-type? true :sql string?}- (sql-tools/is-single-stmt-of-type? driver/*driver* sql "read"))+ (is (=? {:verdict :allowed, :sql string?}+ (sql-tools/validate-executable-stmt driver/*driver* sql "read"))))+ (are [sql] (=? {:verdict :allowed, :single-stmt? true :sql string?}+ (sql-tools/validate-executable-stmt driver/*driver* sql "read"))"SELECT * FROM foo UNION ALL SELECT * FROM bar INTERSECT ALL SELECT * FROM baz""SELECT * FROM foo UNION ALL SELECT * FROM bar EXCEPT ALL SELECT * FROM baz""SELECT * FROM foo INTERSECT ALL SELECT * FROM bar UNION ALL SELECT * FROM baz"@@ -272,50 +279,83 @@[sql](count (re-seq #"\?" sql)))-(deftest ^:parallel is-single-stmt-of-type-placeholder-cast-test+(deftest ^:parallel validate-executable-stmt-placeholder-cast-test(testing "queries with `?::` are parsed correctly"(doseq [sql ["SELECT ?::date""SELECT (?::date - x::date)""SELECT ?::text, ?::integer, ?::boolean FROM t WHERE x = ?""SELECT (?::date - CURRENT_DATE) AS diff"]]- (let [{out-sql :sql :as result} (sql-tools/is-single-stmt-of-type? :postgres sql "read")]- (is (=? {:is-single-stmt? true :allowed-stmt-type? true :sql string?} result))+ (let [{out-sql :sql :as result} (sql-tools/validate-executable-stmt :postgres sql "read")]+ (is (=? {:verdict :allowed :single-stmt? true :sql string?} result))(is (= (placeholder-count sql) (placeholder-count out-sql))))))(testing "a query with `?::` inside string literals are left untouched"- (is (= {:is-single-stmt? true :allowed-stmt-type? true :sql "SELECT '?::date'"}- (sql-tools/is-single-stmt-of-type? :postgres "select '?::date'" "read"))))+ (is (= {:verdict :allowed :single-stmt? true :sql "SELECT '?::date'"}+ (sql-tools/validate-executable-stmt :postgres "select '?::date'" "read"))))(testing "multi-statement queries with placeholder casts are still rejected"- (are [sql] (=? {:is-single-stmt? false :allowed-stmt-type? false}- (sql-tools/is-single-stmt-of-type? :postgres sql "read"))+ (are [sql] (=? {:verdict :disallowed :single-stmt? false}+ (sql-tools/validate-executable-stmt :postgres sql "read"))"SELECT ?::date; DROP TABLE t""SET ROLE NONE; SELECT ?::date")))-(deftest ^:parallel is-single-stmt-of-type-qdcolon-dialects-test+(deftest ^:parallel validate-executable-stmt-qdcolon-dialects-test(testing "databricks' native `expr?::type` try-cast operator is not split apart"- (is (=? {:is-single-stmt? true :allowed-stmt-type? true :sql #"(?i).*TRY_CAST\(x AS DATE\).*"}- (sql-tools/is-single-stmt-of-type? :databricks "SELECT x?::date FROM t" "read")))))--(deftest ^:parallel is-single-stmt-of-type-not-stripped-test- (testing "we don't remove value clauses when validating impersonated queries (#74284)"- (let [values-query (str "SELECT x FROM (VALUES " (str/join ", " (repeat 105 "(1)")) ") AS t(x)")]- (are [sql is-single-stmt? allowed-stmt-type?]- (= {:is-single-stmt? is-single-stmt? :allowed-stmt-type? allowed-stmt-type? :sql sql}- (sql-tools/is-single-stmt-of-type? :postgres sql "read"))- values-query true true- (str "SELECT 1; " values-query) false false- (str "SET ROLE none; " values-query) false false- (str values-query "; SELECT 1") false false- (str values-query "; SET ROLE none") false false)))- (testing "we don't remove large IN lists, tuple lists, or arrays when validating impersonated queries"- (doseq [query [(str "SELECT x FROM t WHERE x IN (" (str/join ", " (range 105)) ")")- (str "SELECT x FROM t WHERE (x, y) IN (" (str/join ", " (map #(format "(%d, %d)" % %) (range 105))) ")")- (str "SELECT x FROM t WHERE x = ANY(ARRAY[" (str/join ", " (range 105)) "])")]]- (are [sql is-single-stmt? allowed-stmt-type?]- (= {:is-single-stmt? is-single-stmt? :allowed-stmt-type? allowed-stmt-type? :sql sql}- (sql-tools/is-single-stmt-of-type? :postgres sql "read"))- query true true- (str "SELECT 1; " query) false false- (str query "; SELECT 1") false false))))+ (is (=? {:verdict :allowed :single-stmt? true :sql #"(?i).*TRY_CAST\(x AS DATE\).*"}+ (sql-tools/validate-executable-stmt :databricks "SELECT x?::date FROM t" "read")))))++(deftest ^:parallel validate-executable-stmt-large-literal-lists-test+ ;; A classification reads the ORIGINAL SQL, never a collapsed rewrite of it. A large but+ ;; ordinary literal list — the values a Number variable inlines for `IN ({{ids}})` — validates normally,+ ;; and a second statement nested in such a list is counted as a second statement however it is written,+ ;; because the parser sees the real input. Collapsing only guards, at a much higher threshold, against a+ ;; list so huge it would OOM GraalPy — those are refused outright rather than substituted.+ (testing "a large-but-under-ceiling literal list validates normally (the IN ({{ids}}) idiom)"+ (are [sql] (=? {:verdict :allowed :single-stmt? true :sql string?}+ (sql-tools/validate-executable-stmt :postgres sql "read"))+ (str "SELECT x FROM (VALUES " (str/join ", " (repeat 105 "(1)")) ") AS t(x)")+ (str "SELECT x FROM t WHERE x IN (" (str/join ", " (range 105)) ")")+ (str "SELECT x FROM t WHERE (x, y) IN (" (str/join ", " (map #(format "(%d, %d)" % %) (range 105))) ")")+ (str "SELECT x FROM t WHERE x = ANY(ARRAY[" (str/join ", " (range 105)) "])")))+ (testing "a second statement alongside such a list is counted as a second statement"+ (let [base (str "SELECT x FROM t WHERE x IN (" (str/join ", " (range 105)) ")")]+ (are [sql] (=? {:verdict :disallowed}+ (sql-tools/validate-executable-stmt :postgres sql "read"))+ (str "SELECT 1; " base)+ (str "SET ROLE none; " base)+ (str base "; SELECT 1")+ (str base "; SET ROLE none"))))+ (testing "a statement separator nested in a comment inside an oversized VALUES list is still counted"+ ;; a `--` comment can unbalance the JVM-side paren scan that decides whether to collapse a list; parsing+ ;; the original rather than the collapsed form, sqlglot counts two statements either way+ (let [sql (str "SELECT * FROM (VALUES " (str/join "," (repeat 101 "(1)"))+ ",( -- ((\n 1 )) t ; SELECT 2 --)) t")]+ (is (=? {:verdict :disallowed}+ (sql-tools/validate-executable-stmt :postgres sql "read")))))+ (testing "a literal list beyond the ceiling comes back unchecked (memory guard), not parsed"+ (let [huge (str "SELECT x FROM t WHERE x IN (" (str/join "," (range 2500)) ")")]+ (is (= {:verdict :unknown, :reason "large-literal-list", :sql huge}+ (sql-tools/validate-executable-stmt :postgres huge "read")))+ (testing "and the gate, whose ceiling is far lower, says the same of a list over its own"+ (is (= {:verdict :unknown, :reason "large-literal-list", :sql huge}+ (sql-tools/classify-stmt :postgres huge "read"))))))+ (testing "a driver with no SQLGlot dialect cannot be validated for execution"+ ;; sqlglot would otherwise parse in its generic dialect and hand back a confident verdict plus a+ ;; generically-regenerated `:sql` string, which a caller that executes the `:sql` would go on to run.+ (is (= {:verdict :unknown, :reason "unknown-dialect", :sql "SELECT 1"}+ (sql-tools/validate-executable-stmt :h2 "SELECT 1" "read")))+ (testing "while the gate, which executes nothing, accepts the generic dialect"+ (is (=? {:verdict :allowed} (sql-tools/classify-stmt :h2 "SELECT 1" "read")))+ (is (true? (sql-tools/read-only-select? :h2 "SELECT 1"))))))++(deftest ^:parallel classify-stmt-length-cap-test+ (testing "the gate stops reading SQL past its length cap, however harmless"+ (let [padded (fn [n] (str "SELECT 1" (str/join (repeat (- n 8) " "))))+ limit (* 16 1024)]+ (is (=? {:verdict :allowed} (sql-tools/classify-stmt :postgres (padded limit) "read")))+ (is (= {:verdict :unknown, :reason "too-long", :sql (padded (inc limit))}+ (sql-tools/classify-stmt :postgres (padded (inc limit)) "read")))))+ (testing "a caller that will execute the SQL has no length cap: long native queries are legitimate"+ (let [long-sql (str "SELECT 1 FROM t WHERE x = '" (str/join (repeat (* 32 1024) "a")) "'")]+ (is (=? {:verdict :allowed} (sql-tools/validate-executable-stmt :postgres long-sql "read"))))))(deftest ^:parallel read-only-select?-test(doseq [driver [:postgres :mysql :h2]]