Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Keep filters separate when Metabot edits a query (#82977)

Mihael Konjević17716b8a3ff3
diff --git a/resources/metabot/prompts/tools/construct_notebook_query.md b/resources/metabot/prompts/tools/construct_notebook_query.md
index 680286e57c5..c779ce8847b 100644
--- a/resources/metabot/prompts/tools/construct_notebook_query.md
+++ b/resources/metabot/prompts/tools/construct_notebook_query.md
@@ -64,10 +64,17 @@ Field options (all optional):
## Per-clause examples
-Filter (comparison + boolean combination):
+Use one entry per filter condition. Metabase combines the entries with `and`.
```json
-"filters": [["and", {},
+"filters": [[">", {}, ["field", {}, ["Sample Database", null, "ORDERS", "TOTAL"]], 100],
+ ["<=", {}, ["field", {}, ["Sample Database", null, "ORDERS", "QUANTITY"]], 5]]
+```
+
+Use one entry for conditions combined with `or` or negated together by `not`; nest `and` inside them when needed. For example, "total over 100, or quantity at most 5":
+
+```json
+"filters": [["or", {},
[">", {}, ["field", {}, ["Sample Database", null, "ORDERS", "TOTAL"]], 100],
["<=", {}, ["field", {}, ["Sample Database", null, "ORDERS", "QUANTITY"]], 5]]]
```
@@ -277,6 +284,10 @@ Shape rules:
- **Never invent a `source-card` entity_id.** It must be a 21-char string copied verbatim from search / `read_resource` — no patterns, no numeric ids, no `card__<id>`.
- **`source-card` columns are referenced by output name** (string in slot 3), not portable FK.
+Editing an existing query (the user asks to change, add, or remove part of the question they're viewing):
+- Start from the query shown in context and change only what the user asked for. Copy every other clause unchanged, including filters, breakouts, aggregations, and their order.
+- Keep each existing filter as its own `filters:` entry. To change one filter, replace just that entry. Never merge it with another filter into an `and`. The notebook shows a combined filter as one custom expression, so the user sees their untouched filters rewritten.
+
Anti-hallucination:
- **Don't subtract dates** with `-`. Use `["datetime-diff", {}, <left>, <right>, "<unit>"]` for the integer count of units between two temporal values.
- **For multi-value categorical filters, use `in` / `not-in`**, not `=` with a list literal. The tool rewrites the list form, but write canonical: `["in", {}, <field>, "a", "b"]`.
@@ -320,7 +331,7 @@ Anti-hallucination:
### Filters
Boolean:
-- `["and", {}, <pred>, <pred>, ...]` — min 2 args.
+- `["and", {}, <pred>, <pred>, ...]` – requires at least 2 arguments. Write plain "A and B" as two `filters:` entries; nest `and` under `or` or `not`.
- `["or", {}, <pred>, <pred>, ...]` — min 2 args.
- `["not", {}, <pred>]`
diff --git a/resources/metabot/skills/construct-notebook-query-core.md b/resources/metabot/skills/construct-notebook-query-core.md
index f10c0d38a5b..4ea16f9f308 100644
--- a/resources/metabot/skills/construct-notebook-query-core.md
+++ b/resources/metabot/skills/construct-notebook-query-core.md
@@ -77,10 +77,17 @@ Field options (all optional):
## Per-clause examples
-Filter (comparison + boolean combination):
+Use one entry per filter condition. Metabase combines the entries with `and`.
```json
-"filters": [["and", {},
+"filters": [[">", {}, ["field", {}, ["Sample Database", null, "ORDERS", "TOTAL"]], 100],
+ ["<=", {}, ["field", {}, ["Sample Database", null, "ORDERS", "QUANTITY"]], 5]]
+```
+
+Use one entry for conditions combined with `or` or negated together by `not`; nest `and` inside them when needed. For example, "total over 100, or quantity at most 5":
+
+```json
+"filters": [["or", {},
[">", {}, ["field", {}, ["Sample Database", null, "ORDERS", "TOTAL"]], 100],
["<=", {}, ["field", {}, ["Sample Database", null, "ORDERS", "QUANTITY"]], 5]]]
```
@@ -119,6 +126,10 @@ Shape rules:
- **Never invent a `source-card` entity_id.** It must be a 21-char string copied verbatim from search / `read_resource` — no patterns, no numeric ids, no `card__<id>`.
- **`source-card` columns are referenced by output name** (string in slot 3), not portable FK.
+Editing an existing query (the user asks to change, add, or remove part of the question they're viewing):
+- Start from the query shown in context and change only what the user asked for. Copy every other clause unchanged, including filters, breakouts, aggregations, and their order.
+- Keep each existing filter as its own `filters:` entry. To change one filter, replace just that entry. Never merge it with another filter into an `and`. The notebook shows a combined filter as one custom expression, so the user sees their untouched filters rewritten.
+
Anti-hallucination:
- **Don't subtract dates** with `-`. Use `["datetime-diff", {}, <left>, <right>, "<unit>"]` for the integer count of units between two temporal values.
- **For multi-value categorical filters, use `in` / `not-in`**, not `=` with a list literal. The tool rewrites the list form, but write canonical: `["in", {}, <field>, "a", "b"]`.
diff --git a/resources/metabot/skills/construct-notebook-query-operators.md b/resources/metabot/skills/construct-notebook-query-operators.md
index 92657ef32e0..0974ff481b9 100644
--- a/resources/metabot/skills/construct-notebook-query-operators.md
+++ b/resources/metabot/skills/construct-notebook-query-operators.md
@@ -39,7 +39,7 @@ These catalogs accompany `construct_notebook_query`. The clause shape and field
## Filters
Boolean:
-- `["and", {}, <pred>, <pred>, ...]` — min 2 args.
+- `["and", {}, <pred>, <pred>, ...]` – requires at least 2 arguments. Write plain "A and B" as two `filters:` entries; nest `and` under `or` or `not`.
- `["or", {}, <pred>, <pred>, ...]` — min 2 args.
- `["not", {}, <pred>]`
diff --git a/src/metabase/agent_lib/representations/repair.clj b/src/metabase/agent_lib/representations/repair.clj
index 589783fefcd..a106b654f22 100644
--- a/src/metabase/agent_lib/representations/repair.clj
+++ b/src/metabase/agent_lib/representations/repair.clj
@@ -1396,6 +1396,49 @@
(assoc "aggregation" [aggregation]))))
stages)))))
+;;; ============================================================
+;;; Pass 1.56 -- split top-level `and` filters into separate `filters:` entries.
+;;;
+;;; A stage's `filters:` entries are implicitly combined with `and`, so
+;;; `filters: [["and", {}, A, B]]` means the same as `filters: [A, B]`. The notebook renders the
+;;; compound `and` as one custom expression (BOT-1446). Splitting it gives each condition its own
+;;; editable filter without changing the rows, whether the model or the user wrote the `and`.
+;;;
+;;; Top-level `and` clauses and directly nested `and` clauses are split case-insensitively (`AND`).
+;;; An `and` under `or` or `not` must stay nested to preserve the boolean logic. An `and` with a
+;;; non-clause operand, such as a trailing map, stays intact so validation can report the original
+;;; clause. The `and`'s options map is dropped because its operands need nothing from it in the
+;;; portable form. Repeating the pass makes no further changes.
+;;;
+;;; Pass 1.56 runs in `normalize-shape*`, after Pass 1 adds options maps and before Pass 1.5.
+;;; It must precede Pass 2.9 (`split-post-agg-filters*`), which moves whole `filters:` entries,
+;;; so a row condition stays in stage 0 while only the condition on an aggregation moves.
+;;;
+;;; `metabase.lib.filter/flatten-and-clause`, used by `lib/atomic-filters`, does the same flattening
+;;; but requires keyword-keyed MBQL. This pass receives the string-keyed portable form before resolution.
+;;; ============================================================
+
+(defn- and-filter-clause?
+ [clause]
+ (and (vector? clause)
+ (>= (count clause) 3)
+ (string? (nth clause 0))
+ (= "and" (u/lower-case-en (nth clause 0)))
+ (map? (nth clause 1))
+ (every? clause-like? (drop 2 clause))))
+
+(defn- and-filter-operands
+ [clause]
+ (if (and-filter-clause? clause)
+ (mapcat and-filter-operands (drop 2 clause))
+ [clause]))
+
+(defn- split-top-level-and-filters*
+ [form]
+ (match/replace-all form
+ {"filters" (filters :guard (and (sequential? filters) (some and-filter-clause? filters)))}
+ (assoc &match "filters" (into [] (mapcat and-filter-operands) filters))))
+
;;; ============================================================
;;; Pass 1.5 -- normalize `expressions:` shape (map -> sequential; stamp `lib/expression-name`)
;;;
@@ -3114,6 +3157,7 @@
normalise-case-clauses*
normalise-fields-shape*
normalise-aggregation-clauses*
+ split-top-level-and-filters*
normalize-expressions-shape*
ensure-lib-types*))
@@ -3131,6 +3175,8 @@
1.5. normalise `expressions:` shape - accept map form `{Name: clause, …}` or the
canonical sequential form, always output sequential with `lib/expression-name`
stamped into each clause's options from the map key when missing;
+ 1.56. split a top-level `and` in `filters:` into separate entries that the notebook shows as
+ individual filters (runs in `normalize-shape*`, before 1.5; must precede 2.9);
1.75. strip stray surrounding double-quotes from the string segments of `field` clauses'
portable-FK vector targets, e.g. `\"col\"` → `col` (cross-stage string targets are left
to the resolution-aware cross-stage matching in pass 5);
diff --git a/test/metabase/agent_lib/representations/repair_test.clj b/test/metabase/agent_lib/representations/repair_test.clj
index c3bb9339984..ce1bcb2b0ad 100644
--- a/test/metabase/agent_lib/representations/repair_test.clj
+++ b/test/metabase/agent_lib/representations/repair_test.clj
@@ -9,11 +9,13 @@
[metabase.agent-lib.representations.repair :as repair]
[metabase.agent-lib.representations.resolve :as repr.resolve]
[metabase.lib-be.core :as lib-be]
+ [metabase.lib.schema :as lib.schema]
[metabase.lib.test-util :as lib.tu]
[metabase.models.serialization.resolve :as serdes.resolve]
[metabase.models.serialization.resolve.mp :as resolve.mp]
[metabase.test :as mt]
- [metabase.util.date-2 :as u.date]))
+ [metabase.util.date-2 :as u.date]
+ [metabase.util.malli.registry :as mr]))
(set! *warn-on-reflection* true)
@@ -747,8 +749,15 @@
(get "stages")
first))
-(defn- repair-filter [filter-clause]
- (first (get (repair-in-stage {"filters" [filter-clause]}) "filters")))
+(defn- repair-filter
+ "Repair and return one filter clause.
+
+ Throw if repair splits it into several entries, as with a top-level `and`, so a test can't
+ silently check only the first condition."
+ [filter-clause]
+ (let [filters (get (repair-in-stage {"filters" [filter-clause]}) "filters")]
+ (assert (= 1 (count filters)) (str "repair split the filter into " (count filters) " entries"))
+ (first filters)))
(defn- hoist-filter
"Run only Pass 2.95 over a stage carrying `filter-clause`, returning the rewritten clause.
@@ -996,9 +1005,9 @@
(deftest ^:parallel hoist-temporal-bucket-nested-in-filter-test
(testing "hoisting reaches comparisons nested under boolean combinators"
- (is (= ["and" {} ["=" {} (created-at-bucketed "month") "2025-01-01"]
+ (is (= ["or" {} ["=" {} (created-at-bucketed "month") "2025-01-01"]
["not" {} ["=" {} (created-at-bucketed "year") "2024-01-01"]]]
- (repair-filter ["and" {}
+ (repair-filter ["or" {}
["=" {} created-at (abs-dt "2025-01-01" "month")]
["not" {} ["=" {} created-at (abs-dt "2024-01-01" "year")]]])))))
@@ -1374,6 +1383,118 @@
twice (repair/repair trivial-mp once)]
(is (= once twice)))))
+;;; ============================================================
+;;; Pass 1.56 - split top-level `and` filters
+;;; ============================================================
+
+(def ^:private status-filter
+ ["=" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "paid"])
+
+(def ^:private total-filter
+ [">" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] 100])
+
+(def ^:private id-filter
+ ["not-null" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "ID"]]])
+
+(defn- filters-query [filters]
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "filters" filters}]})
+
+(defn- repaired-filters [filters]
+ (get-in (repair/repair trivial-mp (filters-query filters)) ["stages" 0 "filters"]))
+
+(deftest ^:parallel split-top-level-and-filters-test
+ (testing "a top-level `and` becomes separate `filters:` entries (BOT-1446)"
+ (is (= [status-filter total-filter]
+ (repaired-filters [["and" {} status-filter total-filter]]))))
+ (testing "`and` nested directly in a top-level `and` is flattened too, keeping order"
+ (is (= [id-filter status-filter total-filter]
+ (repaired-filters [id-filter ["and" {} status-filter ["and" {} total-filter]]]))))
+ (testing "a bare `and` with no options map is split once Pass 1 has added `{}`"
+ (is (= [status-filter total-filter]
+ (repaired-filters [["and" status-filter total-filter]]))))
+ (testing "the `and` head is matched case-insensitively"
+ (is (= [status-filter total-filter]
+ (repaired-filters [["AND" {} status-filter total-filter]]))))
+ (testing "an `and` with a non-clause operand stays intact so validation can report it"
+ (let [filters [["and" {} status-filter total-filter {"trailing" "x"}]]]
+ (is (= filters (repaired-filters filters))))))
+
+(deftest ^:parallel split-top-level-and-filters-with-bare-field-path-test
+ (let [clause ["and" {} id-filter ["Sample" "PUBLIC" "ORDERS" "IS_ACTIVE"]]
+ repaired (repair/repair mp-temporal (filters-query [clause]))
+ resolved (repr.resolve/resolve-query mp-temporal repaired)
+ errors (:errors (mr/explain ::lib.schema/query resolved))]
+ (is (= [clause] (get-in repaired ["stages" 0 "filters"])))
+ (testing "validation reports the original `and` at filter index 0"
+ (is (= [[:stages 0 :filters 0]]
+ (mapv #(vec (take 4 (:in %))) errors))))))
+
+(deftest ^:parallel split-top-level-and-filters-still-repairs-each-conjunct-test
+ (testing "temporal buckets and trailing options are repaired in each condition from a split `and`"
+ (is (= [["=" {} (created-at-bucketed "month") "2025-01-01"]
+ ["contains" {"case-sensitive" false}
+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "@gmail.com"]]
+ (get (repair-in-stage
+ {"filters" [["and" {}
+ ["=" {} created-at (abs-dt "2025-01-01" "month")]
+ ["contains" {}
+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]]
+ "@gmail.com" {"case-sensitive" false}]]]})
+ "filters")))))
+
+(deftest ^:parallel split-top-level-and-filters-keeps-boolean-logic-test
+ (testing "an `and` under `or` or `not` stays nested"
+ (let [filters [["or" {} ["and" {} status-filter total-filter] id-filter]
+ ["not" {} ["and" {} status-filter total-filter]]]]
+ (is (= filters (repaired-filters filters))))))
+
+(deftest ^:parallel split-top-level-and-filters-nested-stages-test
+ (testing "a top-level `and` is split in a later stage and in a join's stage, not just stage 0"
+ (let [q {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "joins" [{"lib/type" "mbql/join"
+ "alias" "Products"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "PRODUCTS"]
+ "filters" [["and" {} status-filter total-filter]]}]
+ "conditions" [["=" {}
+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]
+ ["field" {"join-alias" "Products"}
+ ["Sample" "PUBLIC" "PRODUCTS" "ID"]]]]}]}
+ {"lib/type" "mbql.stage/mbql"
+ "filters" [["and" {} total-filter id-filter]]}]}
+ out (repair/repair trivial-mp q)]
+ (is (= [status-filter total-filter]
+ (get-in out ["stages" 0 "joins" 0 "stages" 0 "filters"])))
+ (is (= [total-filter id-filter]
+ (get-in out ["stages" 1 "filters"]))))))
+
+(deftest ^:parallel split-top-level-and-filters-before-post-agg-split-test
+ (testing (str "an `and` with conditions on rows and aggregations is split before Pass 2.9,\n"
+ "so only the condition on an aggregation moves to the new stage")
+ (let [q {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "aggregation" [["count" {}]]
+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]
+ "filters" [["and" {} status-filter [">" {} ["aggregation" {} 0] 10]]]}]}
+ stages (get (repair/repair trivial-mp q) "stages")]
+ (is (=? [{"filters" [status-filter]}
+ {"filters" [[">" {} ["field" {} "count"] 10]]}]
+ stages)))))
+
+(deftest ^:parallel split-top-level-and-filters-idempotent-test
+ (let [once (repair/repair trivial-mp (filters-query [["and" {} status-filter ["and" {} total-filter id-filter]]]))
+ twice (repair/repair trivial-mp once)]
+ (is (= once twice))))
+
;;; ============================================================
;;; Pass 1.87 - rewrite misspelled `lib/type` aliases
;;; ============================================================
@@ -3378,13 +3499,12 @@
"x.*" {"case-sensitive" false}]))))))
(deftest ^:parallel merge-trailing-options-nested-in-filters-test
- (testing (str "the misplaced trailing options can be nested arbitrarily deep - repair's\n"
- "postwalk reaches a time-interval clause nested inside `and` inside the\n"
- "stage's filters: block.")
+ (testing (str "postwalk finds trailing options at any nesting depth, including a time-interval\n"
+ "clause inside `or` in the stage's `filters:` block")
(let [q {"lib/type" "mbql/query"
"stages" [{"lib/type" "mbql.stage/mbql"
"source-table" ["Sample" "PUBLIC" "ORDERS"]
- "filters" [["and" {}
+ "filters" [["or" {}
["=" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "x"]
["time-interval" {}
["field" {} ["Sample" "PUBLIC" "ORDERS" "CREATED_AT"]]
@@ -3507,19 +3627,19 @@
["field" {} ["S" "P" "T" "C"]] "x"
{"case-sensitive" false}])))))
-(deftest ^:parallel merge-string-filter-trailing-options-nested-in-and-test
- (testing (str "a contains clause with trailing case-sensitivity options, nested inside `and`\n"
- "inside the stage filters (the gmail-customers repro), is repaired via postwalk")
+(deftest ^:parallel merge-string-filter-trailing-options-nested-in-or-test
+ (testing (str "postwalk repairs a contains clause with trailing case-sensitivity options\n"
+ "inside `or` in the stage filters (the gmail-customers repro)")
(let [q {"lib/type" "mbql/query"
"stages" [{"lib/type" "mbql.stage/mbql"
"source-table" ["Sample" "PUBLIC" "ORDERS"]
- "filters" [["and" {}
+ "filters" [["or" {}
["=" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "active"]
["contains" {}
["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]]
"@gmail.com" {"case-sensitive" false}]]]}]}
out (repair/repair trivial-mp q)
- ;; and-clause is ["and" {} <=-cond> <contains-cond>]; the contains is at index 3
+ ;; An `or` keeps both conditions in one entry; Pass 1.56 would split an `and`.
c (get-in out ["stages" 0 "filters" 0 3])]
(is (= ["contains" {"case-sensitive" false}
["field" {} ["Sample" "PUBLIC" "ORDERS" "STATUS"]] "@gmail.com"]
diff --git a/test/metabase/metabot/skills_test.clj b/test/metabase/metabot/skills_test.clj
index 3e2f5687988..654bad709d3 100644
--- a/test/metabase/metabot/skills_test.clj
+++ b/test/metabase/metabot/skills_test.clj
@@ -1,5 +1,6 @@
(ns metabase.metabot.skills-test
(:require
+ [clojure.java.io :as io]
[clojure.string :as str]
[clojure.test :refer [deftest is testing]]
[metabase.metabot.agent.prompts :as prompts]
@@ -213,3 +214,28 @@
(is (= [] (skills/dialect-preload-parts nil))))
(testing "returns empty when the dialect has no registered skill"
(is (= [] (skills/dialect-preload-parts "nonexistent-engine")))))
+
+(defn- section-between
+ "Return the text of `s` from the first occurrence of `start` up to but excluding `end`."
+ [s start end]
+ (let [from (str/index-of s start)
+ to (some->> from (str/index-of s end))]
+ (when (and from to)
+ (subs s from to))))
+
+(deftest ^:parallel construct-notebook-query-filter-guidance-in-sync-test
+ (testing (str "the `construct_notebook_query` tool prompt and construct-notebook-query skills\n"
+ "give identical filter guidance")
+ (let [tool-prompt (slurp (io/resource "metabot/prompts/tools/construct_notebook_query.md"))
+ core (:body (skills/get-skill :construct-notebook-query-core))
+ operators (:body (skills/get-skill :construct-notebook-query-operators))]
+ (doseq [[start end] [["Use one entry per filter condition." "Aggregation (on a field, plus `count`):"]
+ ["Editing an existing query" "Anti-hallucination:"]]
+ :let [section (section-between tool-prompt start end)]]
+ (testing start
+ (is (some? section))
+ (is (= section (section-between core start end)))))
+ (testing "`and` operator entry"
+ (let [line (first (filter #(str/starts-with? % "- `[\"and\"") (str/split-lines tool-prompt)))]
+ (is (some? line))
+ (is (str/includes? operators line)))))))
diff --git a/test/metabase/metabot/tools/construct_representations_test.clj b/test/metabase/metabot/tools/construct_representations_test.clj
index 9e600629c95..e316f575160 100644
--- a/test/metabase/metabot/tools/construct_representations_test.clj
+++ b/test/metabase/metabot/tools/construct_representations_test.clj
@@ -197,6 +197,23 @@
(is (= exported
(get-in redo [:structured-output :query-json])))))))))))
+(deftest top-level-and-filter-split-into-separate-filters-end-to-end-test
+ (testing "A top-level `and` becomes separate filters in the constructed MBQL 5 query."
+ (with-mp-and-stubs!
+ (fn []
+ (let [q (get-in (construct/execute-representations-query
+ (query-data
+ {"lib/type" "mbql/query"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "filters" [["and" {}
+ [">" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] 100]
+ ["between" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "ID"]] 1 10]]]}]}))
+ [:structured-output :query])]
+ (is (=? [[:> {} [:field {} pos-int?] 100]
+ [:between {} [:field {} pos-int?] 1 10]]
+ (lib/filters q 0))))))))
+
(deftest unknown-table-surfaces-agent-error-test
(with-mp-and-stubs!
(fn []