The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Keep filters separate when Metabot edits a query (#82977)
Mihael Konjević17716b8a3ff3
diff --git a/resources/metabot/prompts/tools/construct_notebook_query.md b/resources/metabot/prompts/tools/construct_notebook_query.mdindex 680286e57c5..c779ce8847b 100644--- a/resources/metabot/prompts/tools/construct_notebook_query.md+++ b/resources/metabot/prompts/tools/construct_notebook_query.md@@ -64,10 +64,17 @@ Field options (all optional):## Per-clause examples-Filter (comparison + boolean combination):+Use one entry per filter condition. Metabase combines the entries with `and`.```json-"filters": [["and", {},+"filters": [[">", {}, ["field", {}, ["Sample Database", null, "ORDERS", "TOTAL"]], 100],+ ["<=", {}, ["field", {}, ["Sample Database", null, "ORDERS", "QUANTITY"]], 5]]+```++Use one entry for conditions combined with `or` or negated together by `not`; nest `and` inside them when needed. For example, "total over 100, or quantity at most 5":++```json+"filters": [["or", {},[">", {}, ["field", {}, ["Sample Database", null, "ORDERS", "TOTAL"]], 100],["<=", {}, ["field", {}, ["Sample Database", null, "ORDERS", "QUANTITY"]], 5]]]```@@ -277,6 +284,10 @@ Shape rules:- **Never invent a `source-card` entity_id.** It must be a 21-char string copied verbatim from search / `read_resource` — no patterns, no numeric ids, no `card__<id>`.- **`source-card` columns are referenced by output name** (string in slot 3), not portable FK.+Editing an existing query (the user asks to change, add, or remove part of the question they're viewing):+- Start from the query shown in context and change only what the user asked for. Copy every other clause unchanged, including filters, breakouts, aggregations, and their order.+- Keep each existing filter as its own `filters:` entry. To change one filter, replace just that entry. Never merge it with another filter into an `and`. The notebook shows a combined filter as one custom expression, so the user sees their untouched filters rewritten.+Anti-hallucination:- **Don't subtract dates** with `-`. Use `["datetime-diff", {}, <left>, <right>, "<unit>"]` for the integer count of units between two temporal values.- **For multi-value categorical filters, use `in` / `not-in`**, not `=` with a list literal. The tool rewrites the list form, but write canonical: `["in", {}, <field>, "a", "b"]`.@@ -320,7 +331,7 @@ Anti-hallucination:### FiltersBoolean:-- `["and", {}, <pred>, <pred>, ...]` — min 2 args.+- `["and", {}, <pred>, <pred>, ...]` – requires at least 2 arguments. Write plain "A and B" as two `filters:` entries; nest `and` under `or` or `not`.- `["or", {}, <pred>, <pred>, ...]` — min 2 args.- `["not", {}, <pred>]`diff --git a/resources/metabot/skills/construct-notebook-query-core.md b/resources/metabot/skills/construct-notebook-query-core.mdindex f10c0d38a5b..4ea16f9f308 100644--- a/resources/metabot/skills/construct-notebook-query-core.md+++ b/resources/metabot/skills/construct-notebook-query-core.md@@ -77,10 +77,17 @@ Field options (all optional):## Per-clause examples-Filter (comparison + boolean combination):+Use one entry per filter condition. Metabase combines the entries with `and`.```json-"filters": [["and", {},+"filters": [[">", {}, ["field", {}, ["Sample Database", null, "ORDERS", "TOTAL"]], 100],+ ["<=", {}, ["field", {}, ["Sample Database", null, "ORDERS", "QUANTITY"]], 5]]+```++Use one entry for conditions combined with `or` or negated together by `not`; nest `and` inside them when needed. For example, "total over 100, or quantity at most 5":++```json+"filters": [["or", {},[">", {}, ["field", {}, ["Sample Database", null, "ORDERS", "TOTAL"]], 100],["<=", {}, ["field", {}, ["Sample Database", null, "ORDERS", "QUANTITY"]], 5]]]```@@ -119,6 +126,10 @@ Shape rules:- **Never invent a `source-card` entity_id.** It must be a 21-char string copied verbatim from search / `read_resource` — no patterns, no numeric ids, no `card__<id>`.- **`source-card` columns are referenced by output name** (string in slot 3), not portable FK.+Editing an existing query (the user asks to change, add, or remove part of the question they're viewing):+- Start from the query shown in context and change only what the user asked for. Copy every other clause unchanged, including filters, breakouts, aggregations, and their order.+- Keep each existing filter as its own `filters:` entry. To change one filter, replace just that entry. Never merge it with another filter into an `and`. The notebook shows a combined filter as one custom expression, so the user sees their untouched filters rewritten.+Anti-hallucination:- **Don't subtract dates** with `-`. Use `["datetime-diff", {}, <left>, <right>, "<unit>"]` for the integer count of units between two temporal values.- **For multi-value categorical filters, use `in` / `not-in`**, not `=` with a list literal. The tool rewrites the list form, but write canonical: `["in", {}, <field>, "a", "b"]`.diff --git a/resources/metabot/skills/construct-notebook-query-operators.md b/resources/metabot/skills/construct-notebook-query-operators.mdindex 92657ef32e0..0974ff481b9 100644--- a/resources/metabot/skills/construct-notebook-query-operators.md+++ b/resources/metabot/skills/construct-notebook-query-operators.md@@ -39,7 +39,7 @@ These catalogs accompany `construct_notebook_query`. The clause shape and field## FiltersBoolean:-- `["and", {}, <pred>, <pred>, ...]` — min 2 args.+- `["and", {}, <pred>, <pred>, ...]` – requires at least 2 arguments. Write plain "A and B" as two `filters:` entries; nest `and` under `or` or `not`.- `["or", {}, <pred>, <pred>, ...]` — min 2 args.- `["not", {}, <pred>]`diff --git a/src/metabase/agent_lib/representations/repair.clj b/src/metabase/agent_lib/representations/repair.cljindex 589783fefcd..a106b654f22 100644--- a/src/metabase/agent_lib/representations/repair.clj+++ b/src/metabase/agent_lib/representations/repair.clj@@ -1396,6 +1396,49 @@(assoc "aggregation" [aggregation]))))stages)))))+;;; ============================================================+;;; Pass 1.56 -- split top-level `and` filters into separate `filters:` entries.+;;;+;;; A stage's `filters:` entries are implicitly combined with `and`, so+;;; `filters: [["and", {}, A, B]]` means the same as `filters: [A, B]`. The notebook renders the+;;; compound `and` as one custom expression (BOT-1446). Splitting it gives each condition its own+;;; editable filter without changing the rows, whether the model or the user wrote the `and`.+;;;+;;; Top-level `and` clauses and directly nested `and` clauses are split case-insensitively (`AND`).+;;; An `and` under `or` or `not` must stay nested to preserve the boolean logic. An `and` with a+;;; non-clause operand, such as a trailing map, stays intact so validation can report the original+;;; clause. The `and`'s options map is dropped because its operands need nothing from it in the+;;; portable form. Repeating the pass makes no further changes.+;;;+;;; Pass 1.56 runs in `normalize-shape*`, after Pass 1 adds options maps and before Pass 1.5.+;;; It must precede Pass 2.9 (`split-post-agg-filters*`), which moves whole `filters:` entries,+;;; so a row condition stays in stage 0 while only the condition on an aggregation moves.+;;;+;;; `metabase.lib.filter/flatten-and-clause`, used by `lib/atomic-filters`, does the same flattening+;;; but requires keyword-keyed MBQL. This pass receives the string-keyed portable form before resolution.+;;; ============================================================++(defn- and-filter-clause?+ [clause]+ (and (vector? clause)+ (>= (count clause) 3)+ (string? (nth clause 0))+ (= "and" (u/lower-case-en (nth clause 0)))+ (map? (nth clause 1))+ (every? clause-like? (drop 2 clause))))++(defn- and-filter-operands+ [clause]+ (if (and-filter-clause? clause)+ (mapcat and-filter-operands (drop 2 clause))+ [clause]))++(defn- split-top-level-and-filters*+ [form]+ (match/replace-all form+ {"filters" (filters :guard (and (sequential? filters) (some and-filter-clause? filters)))}+ (assoc &match "filters" (into [] (mapcat and-filter-operands) filters))))+;;; ============================================================;;; Pass 1.5 -- normalize `expressions:` shape (map -> sequential; stamp `lib/expression-name`);;;@@ -3114,6 +3157,7 @@normalise-case-clauses*normalise-fields-shape*normalise-aggregation-clauses*+ split-top-level-and-filters*normalize-expressions-shape*ensure-lib-types*))@@ -3131,6 +3175,8 @@1.5. normalise `expressions:` shape - accept map form `{Name: clause, …}` or thecanonical sequential form, always output sequential with `lib/expression-name`stamped into each clause's options from the map key when missing;+ 1.56. split a top-level `and` in `filters:` into separate entries that the notebook shows as+ individual filters (runs in `normalize-shape*`, before 1.5; must precede 2.9);1.75. strip stray surrounding double-quotes from the string segments of `field` clauses'portable-FK vector targets, e.g. `\"col\"` → `col` (cross-stage string targets are leftto the resolution-aware cross-stage matching in pass 5);diff --git a/test/metabase/agent_lib/representations/repair_test.clj b/test/metabase/agent_lib/representations/repair_test.cljindex c3bb9339984..ce1bcb2b0ad 100644--- a/test/metabase/agent_lib/representations/repair_test.clj+++ b/test/metabase/agent_lib/representations/repair_test.clj@@ -9,11 +9,13 @@[metabase.agent-lib.representations.repair :as repair][metabase.agent-lib.representations.resolve :as repr.resolve][metabase.lib-be.core :as lib-be]+ [metabase.lib.schema :as lib.schema][metabase.lib.test-util :as lib.tu][metabase.models.serialization.resolve :as serdes.resolve][metabase.models.serialization.resolve.mp :as resolve.mp][metabase.test :as mt]- [metabase.util.date-2 :as u.date]))+ [metabase.util.date-2 :as u.date]+ [metabase.util.malli.registry :as mr]))(set! *warn-on-reflection* true)@@ -747,8 +749,15 @@(get "stages")first))-(defn- repair-filter [filter-clause]- (first (get (repair-in-stage {"filters" [filter-clause]}) "filters")))+(defn- repair-filter+ "Repair and return one filter clause.++ Throw if repair splits it into several entries, as with a top-level `and`, so a test can't+ silently check only the first condition."+ [filter-clause]+ (let [filters (get (repair-in-stage {"filters" [filter-clause]}) "filters")]+ (assert (= 1 (count filters)) (str "repair split the filter into " (count filters) " entries"))+ (first filters)))(defn- hoist-filter"Run only Pass 2.95 over a stage carrying `filter-clause`, returning the rewritten clause.@@ -996,9 +1005,9 @@(deftest ^:parallel hoist-temporal-bucket-nested-in-filter-test(testing "hoisting reaches comparisons nested under boolean combinators"- (is (= ["and" {} ["=" {} (created-at-bucketed "month") "2025-01-01"]+ (is (= ["or" {} ["=" {} (created-at-bucketed "month") "2025-01-01"]["not" {} ["=" {} (created-at-bucketed "year") "2024-01-01"]]]- (repair-filter ["and" {}+ (repair-filter ["or" {}["=" {} created-at (abs-dt "2025-01-01" "month")]["not" {} ["=" {} created-at (abs-dt "2024-01-01" "year")]]])))))@@ -1374,6 +1383,118 @@twice (repair/repair trivial-mp once)](is (= once twice)))))+;;; ============================================================+;;; Pass 1.56 - split top-level `and` filters+;;; ============================================================++(def ^:private status-filter+ ["=" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "paid"])++(def ^:private total-filter+ [">" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] 100])++(def ^:private id-filter+ ["not-null" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "ID"]]])++(defn- filters-query [filters]+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "filters" filters}]})++(defn- repaired-filters [filters]+ (get-in (repair/repair trivial-mp (filters-query filters)) ["stages" 0 "filters"]))++(deftest ^:parallel split-top-level-and-filters-test+ (testing "a top-level `and` becomes separate `filters:` entries (BOT-1446)"+ (is (= [status-filter total-filter]+ (repaired-filters [["and" {} status-filter total-filter]]))))+ (testing "`and` nested directly in a top-level `and` is flattened too, keeping order"+ (is (= [id-filter status-filter total-filter]+ (repaired-filters [id-filter ["and" {} status-filter ["and" {} total-filter]]]))))+ (testing "a bare `and` with no options map is split once Pass 1 has added `{}`"+ (is (= [status-filter total-filter]+ (repaired-filters [["and" status-filter total-filter]]))))+ (testing "the `and` head is matched case-insensitively"+ (is (= [status-filter total-filter]+ (repaired-filters [["AND" {} status-filter total-filter]]))))+ (testing "an `and` with a non-clause operand stays intact so validation can report it"+ (let [filters [["and" {} status-filter total-filter {"trailing" "x"}]]]+ (is (= filters (repaired-filters filters))))))++(deftest ^:parallel split-top-level-and-filters-with-bare-field-path-test+ (let [clause ["and" {} id-filter ["Sample" "PUBLIC" "ORDERS" "IS_ACTIVE"]]+ repaired (repair/repair mp-temporal (filters-query [clause]))+ resolved (repr.resolve/resolve-query mp-temporal repaired)+ errors (:errors (mr/explain ::lib.schema/query resolved))]+ (is (= [clause] (get-in repaired ["stages" 0 "filters"])))+ (testing "validation reports the original `and` at filter index 0"+ (is (= [[:stages 0 :filters 0]]+ (mapv #(vec (take 4 (:in %))) errors))))))++(deftest ^:parallel split-top-level-and-filters-still-repairs-each-conjunct-test+ (testing "temporal buckets and trailing options are repaired in each condition from a split `and`"+ (is (= [["=" {} (created-at-bucketed "month") "2025-01-01"]+ ["contains" {"case-sensitive" false}+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "@gmail.com"]]+ (get (repair-in-stage+ {"filters" [["and" {}+ ["=" {} created-at (abs-dt "2025-01-01" "month")]+ ["contains" {}+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]]+ "@gmail.com" {"case-sensitive" false}]]]})+ "filters")))))++(deftest ^:parallel split-top-level-and-filters-keeps-boolean-logic-test+ (testing "an `and` under `or` or `not` stays nested"+ (let [filters [["or" {} ["and" {} status-filter total-filter] id-filter]+ ["not" {} ["and" {} status-filter total-filter]]]]+ (is (= filters (repaired-filters filters))))))++(deftest ^:parallel split-top-level-and-filters-nested-stages-test+ (testing "a top-level `and` is split in a later stage and in a join's stage, not just stage 0"+ (let [q {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "joins" [{"lib/type" "mbql/join"+ "alias" "Products"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "PRODUCTS"]+ "filters" [["and" {} status-filter total-filter]]}]+ "conditions" [["=" {}+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]+ ["field" {"join-alias" "Products"}+ ["Sample" "PUBLIC" "PRODUCTS" "ID"]]]]}]}+ {"lib/type" "mbql.stage/mbql"+ "filters" [["and" {} total-filter id-filter]]}]}+ out (repair/repair trivial-mp q)]+ (is (= [status-filter total-filter]+ (get-in out ["stages" 0 "joins" 0 "stages" 0 "filters"])))+ (is (= [total-filter id-filter]+ (get-in out ["stages" 1 "filters"]))))))++(deftest ^:parallel split-top-level-and-filters-before-post-agg-split-test+ (testing (str "an `and` with conditions on rows and aggregations is split before Pass 2.9,\n"+ "so only the condition on an aggregation moves to the new stage")+ (let [q {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "aggregation" [["count" {}]]+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]+ "filters" [["and" {} status-filter [">" {} ["aggregation" {} 0] 10]]]}]}+ stages (get (repair/repair trivial-mp q) "stages")]+ (is (=? [{"filters" [status-filter]}+ {"filters" [[">" {} ["field" {} "count"] 10]]}]+ stages)))))++(deftest ^:parallel split-top-level-and-filters-idempotent-test+ (let [once (repair/repair trivial-mp (filters-query [["and" {} status-filter ["and" {} total-filter id-filter]]]))+ twice (repair/repair trivial-mp once)]+ (is (= once twice))))+;;; ============================================================;;; Pass 1.87 - rewrite misspelled `lib/type` aliases;;; ============================================================@@ -3378,13 +3499,12 @@"x.*" {"case-sensitive" false}]))))))(deftest ^:parallel merge-trailing-options-nested-in-filters-test- (testing (str "the misplaced trailing options can be nested arbitrarily deep - repair's\n"- "postwalk reaches a time-interval clause nested inside `and` inside the\n"- "stage's filters: block.")+ (testing (str "postwalk finds trailing options at any nesting depth, including a time-interval\n"+ "clause inside `or` in the stage's `filters:` block")(let [q {"lib/type" "mbql/query""stages" [{"lib/type" "mbql.stage/mbql""source-table" ["Sample" "PUBLIC" "ORDERS"]- "filters" [["and" {}+ "filters" [["or" {}["=" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "x"]["time-interval" {}["field" {} ["Sample" "PUBLIC" "ORDERS" "CREATED_AT"]]@@ -3507,19 +3627,19 @@["field" {} ["S" "P" "T" "C"]] "x"{"case-sensitive" false}])))))-(deftest ^:parallel merge-string-filter-trailing-options-nested-in-and-test- (testing (str "a contains clause with trailing case-sensitivity options, nested inside `and`\n"- "inside the stage filters (the gmail-customers repro), is repaired via postwalk")+(deftest ^:parallel merge-string-filter-trailing-options-nested-in-or-test+ (testing (str "postwalk repairs a contains clause with trailing case-sensitivity options\n"+ "inside `or` in the stage filters (the gmail-customers repro)")(let [q {"lib/type" "mbql/query""stages" [{"lib/type" "mbql.stage/mbql""source-table" ["Sample" "PUBLIC" "ORDERS"]- "filters" [["and" {}+ "filters" [["or" {}["=" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "active"]["contains" {}["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]]"@gmail.com" {"case-sensitive" false}]]]}]}out (repair/repair trivial-mp q)- ;; and-clause is ["and" {} <=-cond> <contains-cond>]; the contains is at index 3+ ;; An `or` keeps both conditions in one entry; Pass 1.56 would split an `and`.c (get-in out ["stages" 0 "filters" 0 3])](is (= ["contains" {"case-sensitive" false}["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "@gmail.com"]diff --git a/test/metabase/metabot/skills_test.clj b/test/metabase/metabot/skills_test.cljindex 3e2f5687988..654bad709d3 100644--- a/test/metabase/metabot/skills_test.clj+++ b/test/metabase/metabot/skills_test.clj@@ -1,5 +1,6 @@(ns metabase.metabot.skills-test(:require+ [clojure.java.io :as io][clojure.string :as str][clojure.test :refer [deftest is testing]][metabase.metabot.agent.prompts :as prompts]@@ -213,3 +214,28 @@(is (= [] (skills/dialect-preload-parts nil))))(testing "returns empty when the dialect has no registered skill"(is (= [] (skills/dialect-preload-parts "nonexistent-engine")))))++(defn- section-between+ "Return the text of `s` from the first occurrence of `start` up to but excluding `end`."+ [s start end]+ (let [from (str/index-of s start)+ to (some->> from (str/index-of s end))]+ (when (and from to)+ (subs s from to))))++(deftest ^:parallel construct-notebook-query-filter-guidance-in-sync-test+ (testing (str "the `construct_notebook_query` tool prompt and construct-notebook-query skills\n"+ "give identical filter guidance")+ (let [tool-prompt (slurp (io/resource "metabot/prompts/tools/construct_notebook_query.md"))+ core (:body (skills/get-skill :construct-notebook-query-core))+ operators (:body (skills/get-skill :construct-notebook-query-operators))]+ (doseq [[start end] [["Use one entry per filter condition." "Aggregation (on a field, plus `count`):"]+ ["Editing an existing query" "Anti-hallucination:"]]+ :let [section (section-between tool-prompt start end)]]+ (testing start+ (is (some? section))+ (is (= section (section-between core start end)))))+ (testing "`and` operator entry"+ (let [line (first (filter #(str/starts-with? % "- `[\"and\"") (str/split-lines tool-prompt)))]+ (is (some? line))+ (is (str/includes? operators line)))))))diff --git a/test/metabase/metabot/tools/construct_representations_test.clj b/test/metabase/metabot/tools/construct_representations_test.cljindex 9e600629c95..e316f575160 100644--- a/test/metabase/metabot/tools/construct_representations_test.clj+++ b/test/metabase/metabot/tools/construct_representations_test.clj@@ -197,6 +197,23 @@(is (= exported(get-in redo [:structured-output :query-json])))))))))))+(deftest top-level-and-filter-split-into-separate-filters-end-to-end-test+ (testing "A top-level `and` becomes separate filters in the constructed MBQL 5 query."+ (with-mp-and-stubs!+ (fn []+ (let [q (get-in (construct/execute-representations-query+ (query-data+ {"lib/type" "mbql/query"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "filters" [["and" {}+ [">" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] 100]+ ["between" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "ID"]] 1 10]]]}]}))+ [:structured-output :query])]+ (is (=? [[:> {} [:field {} pos-int?] 100]+ [:between {} [:field {} pos-int?] 1 10]]+ (lib/filters q 0))))))))+(deftest unknown-table-surfaces-agent-error-test(with-mp-and-stubs!(fn []