Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

[BOT-1442] Fix metabot silently producing non-runnable MBQL (#76133)

Tyler Plude79002d46661e
diff --git a/resources/metabot/prompts/tools/construct_notebook_query.md b/resources/metabot/prompts/tools/construct_notebook_query.md
index 54203cf4752..8058fea2649 100644
--- a/resources/metabot/prompts/tools/construct_notebook_query.md
+++ b/resources/metabot/prompts/tools/construct_notebook_query.md
@@ -189,7 +189,11 @@ Re-aggregate — average daily total by month:
]
```
-- Cross-stage refs use a **string name** in slot 3 (e.g. `["field", {}, "count"]`). The name is whatever the previous stage's aggregation/breakout/field produced (`count`, `sum`, or the source field's name).
+- Cross-stage refs use a **string name** in slot 3 (e.g. `["field", {}, "count"]`). The name is the previous stage column's **machine name**, never its UI display label:
+ - An aggregation's output name is the **bare function**: `max`, `min`, `avg`, `sum`, `count`, `median` — the operand is dropped. `["max", {}, ["expression", {}, "Instance Has SC"]]` produces a column named **`max`** (display label "Max of Instance Has SC"). Reference it as `["field", {}, "max"]`, **not** `["field", {}, "Max of Instance Has SC"]`.
+ - A breakout keeps the source field's machine name even when its label is bucketed: a month breakout of `CREATED_AT` is still named `CREATED_AT` (display "Created At: Month").
+ - If a stage has several aggregations of the same function, the outputs are suffixed in order: `max`, `max_2`, `max_3`. So the second `max` is referenced as `["field", {}, "max_2"]`.
+- A `["field", {}, "<name>"]` cross-stage ref whose name matches no column the previous stage produced is an error — name it exactly.
- Within the **same stage**, refer to your own aggregation with `["aggregation", {}, <idx>]` (see §Aggregation references). In a **later** stage, use the cross-stage string-name form against the previous stage's output.
- Joins, expressions, filters, aggregation, breakout, order-by, limit are all valid in later stages.
@@ -199,7 +203,7 @@ Instead of `source-table`, use `source-card` to query an existing question or mo
1. Get the `portable_entity_id` from a tool response. `search` and `read_resource` (`metabase://question/<id>`, `metabase://model/<id>`) include it on the result tag — reuse what's already in context, no extra call needed.
2. Copy it **verbatim** into `source-card`. The id is opaque — never guess, construct, or abbreviate.
-3. Reference the card's columns by output **name** (string in slot 3), not portable FK. If you don't know the names, call `read_resource metabase://question/<numeric-id>/fields` (or `.../model/...`).
+3. Reference the card's columns by output **machine name** (string in slot 3), not portable FK and not the UI display label. If you don't know the names, call `read_resource metabase://question/<numeric-id>/fields` (or `.../model/...`) — a ref whose name matches no column the card produces is an error.
```json
{"lib/type": "mbql.stage/mbql",
@@ -389,6 +393,9 @@ Temporal:
- `["date", {}, <expr>]` / `["datetime", {}, <expr>]` / `["time", {}, <expr>]`
- `["now", {}]` / `["today", {}]`
+Window:
+- `["offset", {}, <expr>, <n>]` — value of `<expr>` from `<n>` rows back (negative `<n>`) or ahead (positive). Valid **only** inside an `aggregation:` or `order-by:` clause — **never** in a custom column (`expressions:`) or a filter.
+
### Temporal units (for `{"temporal-unit": ...}` on field refs and as `<unit>` args)
- Date truncation: `day`, `week`, `month`, `quarter`, `year`
diff --git a/src/metabase/agent_lib/representations/repair.clj b/src/metabase/agent_lib/representations/repair.clj
index df79beb63e5..299edb73072 100644
--- a/src/metabase/agent_lib/representations/repair.clj
+++ b/src/metabase/agent_lib/representations/repair.clj
@@ -31,6 +31,7 @@
[clojure.walk :as walk]
[metabase.agent-lib.representations.resolve :as repr.resolve]
[metabase.lib.core :as lib]
+ [metabase.lib.expression :as lib.expression]
[metabase.lib.metadata.protocols :as lib.metadata.protocols]
[metabase.lib.schema.mbql-clause :as mbql-clause]
[metabase.models.serialization.resolve :as resolve]
@@ -2232,10 +2233,22 @@
(assoc "effective-type" (let [et (:effective-type col)]
(if (keyword? et) (subs (str et) 1) (str et))))))))
-(defn- mini-resolved-columns-by-name
+(defn- column-descriptors
+ "Build the cross-stage match oracle from a `lib/returned-columns` sequence: a vector of
+ `{:name <machine-name>, :display-name <display-name>, :types {\"base-type\" …}|nil}`
+ descriptors. `:types` is nil when the column exposes no usable `base-type`. Carrying the
+ `:display-name` lets [[match-cross-stage-column]] recover a ref the LLM wrote with the UI
+ label (e.g. `\"Max of Instance Has SC\"`) instead of the machine name (`max`)."
+ [cols]
+ (mapv (fn [col]
+ {:name (:name col)
+ :display-name (:display-name col)
+ :types (types-from-column col)})
+ cols))
+
+(defn- mini-resolved-columns
"Resolve `stages[0..idx-1]` as a self-contained query, run it through `lib/query`, and return
- a `{column-name → {\"base-type\" ..., \"effective-type\" ...}}` map for the columns the
- prefix returns.
+ the prefix's returned columns as a vector of [[column-descriptors]].
Returns `nil` (logged at debug) if anything in the resolve / lib/query path throws. The
enclosing pass treats `nil` as 'skip this stage's repairs and let downstream surface the real
@@ -2247,11 +2260,7 @@
resolved (repr.resolve/resolve-query mp prefix-query content-store)
lib-q (lib/query mp resolved)
cols (lib/returned-columns lib-q)]
- (into {}
- (keep (fn [col]
- (when-let [types (types-from-column col)]
- [(:name col) types])))
- cols))
+ (column-descriptors cols))
(catch Exception e
(log/debugf e "[repr-repair] mini-resolve of stages[0..%d] failed; skipping cross-stage type inference for stage %d"
(dec stage-idx) stage-idx)
@@ -2283,34 +2292,34 @@
(-> s u/lower-case-en (str/replace #"[-\s]+" "_"))))
(defn- match-cross-stage-column
- "Find the canonical column entry in `name->types` for `col-name`. Returns `[canonical-name
- types]` or `nil` when no column matches.
+ "Find the column descriptor in `cols` (a vector of [[column-descriptors]]) that `col-name`
+ refers to. Returns `[canonical-machine-name types]` or `nil` when no column matches.
Tries, in order:
- 1. an exact match;
- 2. the double-quote-stripped name (only when stripping actually resolves);
- 3. a loose match that folds case and hyphens/whitespace to underscores, but **only when
- exactly one** real column matches - so `count-where` resolves to `count_where` while
- genuine ambiguity is left for the resolver to report.
-
- A column whose name legitimately needs quotes (vanishingly rare) is never clobbered, and
- unmatched names are passed through for the resolver to surface with a better message."
- [name->types col-name]
- (let [stripped (strip-surrounding-double-quotes col-name)]
- (cond
- (contains? name->types col-name)
- [col-name (get name->types col-name)]
-
- (and (not= stripped col-name) (contains? name->types stripped))
- [stripped (get name->types stripped)]
-
- :else
- (let [target (normalize-col-key stripped)
- hits (when target
- (filter #(= target (normalize-col-key %)) (keys name->types)))]
- (when (= 1 (count hits))
- (let [canon (first hits)]
- [canon (get name->types canon)]))))))
+ 1. an exact `:name` match;
+ 2. the double-quote-stripped `:name` (only when stripping actually resolves);
+ 3. a loose `:name` match that folds case and hyphens/whitespace to underscores, but **only
+ when exactly one** real column matches - so `count-where` resolves to `count_where`;
+ 4. a loose `:display-name` match (same folding, same one-hit guard) - so a ref the LLM
+ wrote with the UI label (`\"Max of Instance Has SC\"`) recovers the machine name (`max`).
+
+ Genuine ambiguity (more than one loose hit) is left unmatched at each layer. A column whose
+ name legitimately needs quotes (vanishingly rare) is never clobbered."
+ [cols col-name]
+ (let [stripped (strip-surrounding-double-quotes col-name)
+ exact (fn [n] (some #(when (= n (:name %)) %) cols))
+ loose-on (fn [key-fn raw]
+ (let [target (normalize-col-key raw)
+ hits (when target
+ (filter #(= target (normalize-col-key (key-fn %))) cols))]
+ (when (= 1 (count hits))
+ (first hits))))
+ match (or (exact col-name)
+ (when (not= stripped col-name) (exact stripped))
+ (loose-on :name stripped)
+ (loose-on :display-name stripped))]
+ (when match
+ [(:name match) (:types match)])))
(defn- maybe-fill-cross-stage-types
"Given a cross-stage field clause and a name→types map, return the clause with its column name
@@ -2319,17 +2328,19 @@
Idempotent: once the name is canonical it matches exactly on the next pass (no rename), and
once `base-type` is present it is left alone."
- [clause name->types]
+ [clause cols]
(let [opts (nth clause 1)
col-name (nth clause 2)]
- (if-let [[canonical-name types] (match-cross-stage-column name->types col-name)]
+ (if-let [[canonical-name types] (match-cross-stage-column cols col-name)]
(cond-> clause
- ;; Canonicalise the name when quote-stripping was needed to match. A bare `base-type`
- ;; stamp without this would leave the ref pointing at a non-existent column.
+ ;; Canonicalise the name when quote-stripping / display-name recovery was needed to
+ ;; match. A bare `base-type` stamp without this would leave the ref pointing at a
+ ;; non-existent column.
(not= canonical-name col-name)
(assoc 2 canonical-name)
- ;; Stamp inferred types only when the LLM didn't author a `base-type` already.
- (not (contains? opts "base-type"))
+ ;; Stamp inferred types only when the LLM didn't author a `base-type` already and the
+ ;; matched column actually exposes one.
+ (and types (not (contains? opts "base-type")))
(assoc 1 (merge opts types)))
clause)))
@@ -2338,13 +2349,13 @@
`\"base-type\"`. Skips descent into `\"joins\"` subtrees - join stages have their own
resolution context (the join's own `stages`) and shouldn't reach into their parent stage's
previous-stage columns."
- [stage name->types]
+ [stage cols]
(let [joins (get stage "joins")
stage' (cond-> stage (contains? stage "joins") (dissoc "joins"))
walked (walk/postwalk
(fn [node]
(if (string-cross-stage-field-clause? node)
- (maybe-fill-cross-stage-types node name->types)
+ (maybe-fill-cross-stage-types node cols)
node))
stage')]
(cond-> walked
@@ -2353,8 +2364,8 @@
(defn- mini-resolved-columns-for-source-card
"Resolve a single-stage query consisting only of `{:source-card <entity-id>}` (plus the
outer `mbql/query` wrapper and the warehouse database name), then return the
- `lib/returned-columns` output as a `name → types` map in the same shape as
- [[mini-resolved-columns-by-name]]. Used by [[infer-source-card-field-types*]] to stamp
+ `lib/returned-columns` output as a vector of [[column-descriptors]] in the same shape as
+ [[mini-resolved-columns]]. Used by [[infer-source-card-field-types*]] to stamp
`base-type` onto `[field, {}, \"<col>\"]` clauses in a stage whose source is a saved
question / model.
@@ -2372,11 +2383,7 @@
resolved (repr.resolve/resolve-query mp bare-query content-store)
lib-q (lib/query mp resolved)
cols (lib/returned-columns lib-q)]
- (into {}
- (keep (fn [col]
- (when-let [types (types-from-column col)]
- [(:name col) types])))
- cols))
+ (column-descriptors cols))
(catch Exception e
(log/debugf e "[repr-repair] source-card resolve of stage %d failed; skipping field-type inference"
stage-idx)
@@ -2400,8 +2407,8 @@
q
(let [stage (get-in q ["stages" i])
q' (if (and (map? stage) (get stage "source-card"))
- (if-let [name->types (mini-resolved-columns-for-source-card mp q i content-store)]
- (update-in q ["stages" i] infer-cross-stage-field-types-in-stage name->types)
+ (if-let [cols (mini-resolved-columns-for-source-card mp q i content-store)]
+ (update-in q ["stages" i] infer-cross-stage-field-types-in-stage cols)
q)
q)]
(recur (inc i) q')))))))
@@ -2420,12 +2427,75 @@
q query]
(if (>= i n)
q
- (let [name->types (mini-resolved-columns-by-name mp q i content-store)
- q' (if name->types
- (update-in q ["stages" i] infer-cross-stage-field-types-in-stage name->types)
- q)]
+ (let [cols (mini-resolved-columns mp q i content-store)
+ q' (if cols
+ (update-in q ["stages" i] infer-cross-stage-field-types-in-stage cols)
+ q)]
(recur (inc i) q')))))))
+;;; ============================================================
+;;; Pass 5.7 -- assert cross-stage / source-card string refs resolved
+;;;
+;;; A `["field" {} "<name>"]` ref into a previous stage or a source-card must name a real
+;;; output column. The inference passes above (5 / 5.5) stamp `base-type` onto every such ref
+;;; that matches; anything still missing `base-type` afterward matched nothing - the LLM
+;;; named a column that doesn't exist (typically a UI display label like "Max of X" instead of
+;;; the machine name "max"). lib's resolver passes a typeless string-named field ref through
+;;; unvalidated, producing a schema-invalid, non-runnable query that silently breaks the
+;;; notebook editor (BOT-1442). This pass turns that silent failure into the same loud,
+;;; retryable `:agent-error?` the LLM already recovers from for dangling `[:expression …]`
+;;; refs, naming the offending column and listing the valid ones.
+;;; ============================================================
+
+(defn- unstamped-cross-stage-ref?
+ "True if `node` is a `[\"field\" {} \"<name>\"]` cross-stage/source-card ref still lacking
+ `base-type` (an LLM-authored `base-type` is left for the resolver / schema to judge)."
+ [node]
+ (and (string-cross-stage-field-clause? node)
+ (not (contains? (nth node 1) "base-type"))))
+
+(defn- stage-has-unstamped-cross-stage-ref?
+ "Cheap structural pre-scan: does `stage` contain any [[unstamped-cross-stage-ref?]] outside
+ its `joins` subtrees? Lets the assert pass skip the (resolving) confirmation step entirely
+ on the happy path, where every ref already carries a stamped `base-type`."
+ [stage]
+ (let [stage' (cond-> stage (contains? stage "joins") (dissoc "joins"))]
+ (boolean (some unstamped-cross-stage-ref? (tree-seq coll? seq stage')))))
+
+(defn- first-unresolved-cross-stage-ref
+ "Return the first [[unstamped-cross-stage-ref?]] clause in `stage` that matches no column in
+ `cols`, or nil. Skips `joins` subtrees (their own resolution context)."
+ [stage cols]
+ (let [stage' (cond-> stage (contains? stage "joins") (dissoc "joins"))]
+ (some (fn [node]
+ (when (and (unstamped-cross-stage-ref? node)
+ (nil? (match-cross-stage-column cols (nth node 2))))
+ node))
+ (tree-seq coll? seq stage'))))
+
+(defn- assert-cross-stage-refs-resolved*
+ "Pass 5.7: raise an `:agent-error?` for any string-named cross-stage / source-card field ref
+ that resolves to no real column. No-op when `mp` is nil. Only stages that still carry an
+ unstamped ref pay the cost of re-resolving their column universe to build the message."
+ [query mp content-store]
+ (when (and mp (map? query) (vector? (get query "stages")))
+ (doseq [[idx stage] (map-indexed vector (get query "stages"))
+ :when (and (map? stage) (stage-has-unstamped-cross-stage-ref? stage))]
+ (when-let [cols (cond
+ (get stage "source-card") (mini-resolved-columns-for-source-card mp query idx content-store)
+ (pos? idx) (mini-resolved-columns mp query idx content-store))]
+ (when-let [bad (first-unresolved-cross-stage-ref stage cols)]
+ (throw (ex-info
+ (tru "No column named {0} is available to reference in this stage. Reference a column by its machine name (e.g. an aggregation is `max`, `count`, `sum` - never its display label like `Max of X`), not a UI label. Available column names: {1}."
+ (pr-str (nth bad 2))
+ (str/join ", " (map :name cols)))
+ {:agent-error? true
+ :error :unresolved-cross-stage-field
+ :stage idx
+ :clause bad
+ :available (mapv :name cols)}))))))
+ query)
+
;;; ============================================================
;;; Pass 6 -- friendly error messages for silently-accepted-but-wrong shapes.
;;;
@@ -2698,14 +2768,19 @@
prefix of stages and reading the returned columns' metadata.
5.5. infer `base-type` / `effective-type` on field references in a stage whose source is
a saved question / model (`source-card:`), using the card's resolved returned columns.
-
- Pass 4, Pass 5, and Pass 5.5 require `mp` (a `MetadataProvider`); they are best-effort
- no-ops when `mp` can't resolve the relevant pieces (so the subsequent validate/resolve
- stages can surface the real error with their own, better messages). Hard FK errors from
- Pass 4 (`:no-fk-path`, `:ambiguous-fk`) are raised as `:agent-error?` ex-info so the tool
- wrapper can relay them to the LLM. Pass 5 and Pass 5.5 never throw on their own - if a
- prefix / source-card can't be resolved, they just leave the affected clauses alone and let
- the schema validator complain.
+ 5.7. assert that every string-named cross-stage / source-card field ref resolved to a real
+ column (i.e. Pass 5 / 5.5 stamped its `base-type`). A ref that matched nothing is the
+ LLM naming a column that doesn't exist (often a display label); raise an `:agent-error?`
+ naming it and listing the valid columns, instead of letting it pass through into a
+ schema-invalid, non-runnable query (BOT-1442).
+
+ Pass 4, Pass 5, Pass 5.5, and Pass 5.7 require `mp` (a `MetadataProvider`); they are
+ best-effort no-ops when `mp` can't resolve the relevant pieces (so the subsequent
+ validate/resolve stages can surface the real error with their own, better messages). Hard FK
+ errors from Pass 4 (`:no-fk-path`, `:ambiguous-fk`) are raised as `:agent-error?` ex-info so
+ the tool wrapper can relay them to the LLM. Pass 5 and Pass 5.5 never throw on their own - if
+ a prefix / source-card can't be resolved, they just leave the affected clauses alone; Pass
+ 5.7 then raises the `:agent-error?` only for refs that survived to the end still unresolved.
Note: the database-name normalisation pass (\"Pass 2.5\") that previously lived here was
removed in `repr-plan.md` step 13, and the top-level `database:` field was removed from
@@ -2737,4 +2812,46 @@
(resolve-implicit-joins* mp content-store)
(infer-cross-stage-field-types* mp content-store)
(infer-source-card-field-types* mp content-store)
+ (assert-cross-stage-refs-resolved* mp content-store)
friendly-errors*)))
+
+;;; ============================================================
+;;; Post-resolve gate -- expressions the FE expression editor rejects
+;;; ============================================================
+
+(defn assert-editor-accepts-expressions!
+ "Run the FE expression editor's own validation, [[metabase.lib.expression/diagnose-expression]],
+ over every custom column, aggregation, and filter of the resolved `pmbql-query`, throwing a
+ retryable `:agent-error?` on the first clause it rejects. Returns `pmbql-query`.
+
+ Catches shapes that are schema-legal yet broken in the editor: `offset` in a filter, window
+ functions nested inside aggregation functions, cyclic expression references, type-incompatible
+ arguments, standalone constants without `:expression-literals` support.
+
+ Unlike the rest of this namespace this runs after resolve. Call it only on a query that already
+ passed the suppressed-type-check `::lib.schema/query` validation - `diagnose-expression`
+ validates its query argument against that same schema, and its raw validation error would
+ otherwise preempt the friendly message."
+ [pmbql-query]
+ (doseq [stage-idx (range (lib/stage-count pmbql-query))
+ [mode clauses] [[:expression (lib/expressions pmbql-query stage-idx)]
+ [:aggregation (lib/aggregations pmbql-query stage-idx)]
+ [:filter (lib/filters pmbql-query stage-idx)]]
+ [pos clause] (map-indexed vector clauses)]
+ (when-let [{:keys [message]} (lib.expression/diagnose-expression
+ pmbql-query stage-idx mode clause
+ (when (#{:expression :aggregation} mode) pos))]
+ (let [where (case mode
+ :expression "custom column (`expressions:`)"
+ :aggregation "`aggregation:`"
+ :filter "`filters:`")]
+ (throw (ex-info
+ (tru "The query builder''s expression editor rejects a {0} clause in stage {1}: {2}. The query would build but could not be visualized or saved - rewrite the offending clause."
+ where stage-idx message)
+ {:agent-error? true
+ :error :expression-editor-rejection
+ :status-code 400
+ :stage stage-idx
+ :mode mode
+ :clause clause})))))
+ pmbql-query)
diff --git a/src/metabase/metabot/tools/construct.clj b/src/metabase/metabot/tools/construct.clj
index 28de905256d..ecafcafca42 100644
--- a/src/metabase/metabot/tools/construct.clj
+++ b/src/metabase/metabot/tools/construct.clj
@@ -1,12 +1,15 @@
(ns metabase.metabot.tools.construct
"Notebook query construction tool wrappers."
(:require
+ [malli.error :as me]
[metabase.agent-lib.representations :as repr]
[metabase.agent-lib.representations.repair :as repr.repair]
[metabase.agent-lib.representations.resolve :as repr.resolve]
[metabase.api.common :as api]
[metabase.lib-be.core :as lib-be]
[metabase.lib.core :as lib]
+ [metabase.lib.schema]
+ [metabase.lib.schema.expression :as lib.schema.expression]
[metabase.metabot.agent.links :as links]
[metabase.metabot.agent.streaming :as streaming]
[metabase.metabot.scope :as scope]
@@ -24,6 +27,7 @@
[metabase.util.json :as json]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
+ [metabase.util.malli.registry :as mr]
[toucan2.core :as t2]))
(set! *warn-on-reflection* true)
@@ -278,6 +282,14 @@
(nil? (:status-code base)) (assoc :status-code 400))]
(ex-info (ex-message e) data e)))
+(defn- query-not-runnable-explanation
+ "When `pmbql-query` would make the FE's `canRun` gate return false, return a humanized Malli
+ explanation of why; nil when the query is runnable."
+ [pmbql-query]
+ (binding [lib.schema.expression/*suppress-expression-type-check?* true]
+ (when-let [explanation (mr/explain :metabase.lib.schema/query pmbql-query)]
+ (me/humanize explanation))))
+
(defn execute-representations-query
"Execute a notebook query in the canonical portable MBQL 5 representations format.
@@ -302,6 +314,11 @@
5. Sanity-check the post-repair shape against the portable repair schema.
6. Resolve portable FKs to numeric IDs and normalize through `lib.schema/query` against the
metadata-provider.
+ 6.5. Backstop-gate the resolved query: mirror the FE `canRun` schema validation
+ ([[query-not-runnable-explanation]]), then run the FE expression editor's own
+ diagnostics over every custom column / aggregation / filter
+ ([[repr.repair/assert-editor-accepts-expressions!]]). Either failure is a retryable
+ `:agent-error?` - success on a query the editor rejects is BOT-1442.
7. Export that final numeric pMBQL back to the portable form for the LLM-facing
`:query-json` / `query-content` output.
@@ -340,6 +357,18 @@
(let [repaired (repr.repair/repair mp parsed permission-aware-content-store)
_validated (repr/validate-query repaired)
pmbql-query (repr.resolve/resolve-query mp repaired permission-aware-content-store)
+ _runnable (when-let [why (query-not-runnable-explanation pmbql-query)]
+ (throw (ex-info (tru "The constructed query is not runnable - it would fail the query builder''s validation, so it cannot be visualized or saved. This usually means a field reference is missing its type or names a column that does not exist, or an aggregation/window function (e.g. `offset`) was placed in `expressions:` (custom columns) where it is not allowed - move it to `aggregation:` or `order-by:`. Schema validation details: {0}"
+ (pr-str why))
+ {:agent-error? true
+ :error :query-not-runnable
+ :status-code 400})))
+ ;; The FE expression editor rejects some shapes the (type-check-suppressed) canRun
+ ;; gate above accepts - `offset` in a filter, window functions nested in
+ ;; aggregations, cyclic expression refs, type-incompatible arguments. Must run
+ ;; after the `_runnable` gate: `diagnose-expression` itself validates its query
+ ;; argument against the same schema.
+ _editor-ok (repr.repair/assert-editor-accepts-expressions! pmbql-query)
exported-repr (repr.resolve/export-query mp pmbql-query permission-aware-content-store)
_validated' (repr/validate-query exported-repr)
query-id (u/generate-nano-id)]
diff --git a/test/metabase/agent_lib/representations/repair_test.clj b/test/metabase/agent_lib/representations/repair_test.clj
index d70be7977ef..5cdf571b5d2 100644
--- a/test/metabase/agent_lib/representations/repair_test.clj
+++ b/test/metabase/agent_lib/representations/repair_test.clj
@@ -1862,13 +1862,17 @@
(is (not (contains? opts "base-type"))))))
(deftest ^:parallel cross-stage-field-type-unknown-column-name-test
- (testing "If the referenced name isn't produced by the previous stage, leave the clause
- alone (the resolver will surface :unknown-column or similar with a better message)."
+ (testing "A cross-stage ref to a name no previous stage produces raises an :agent-error?
+ that names the bad column and lists the valid ones, rather than passing a typeless
+ ref through into a non-runnable query."
(let [q (assoc-in multi-stage-base-query
["stages" 1 "filters" 0 2 2] "no_such_column")
- out (repair/repair mp-fks q)
- opts (get-in out ["stages" 1 "filters" 0 2 1])]
- (is (not (contains? opts "base-type"))))))
+ e (try (repair/repair mp-fks q) nil (catch clojure.lang.ExceptionInfo ex ex))]
+ (is (some? e))
+ (is (true? (:agent-error? (ex-data e))))
+ (is (= :unresolved-cross-stage-field (:error (ex-data e))))
+ (is (= ["ID" "count"] (:available (ex-data e))))
+ (is (re-find #"no_such_column" (ex-message e))))))
(deftest ^:parallel cross-stage-field-type-strips-surrounding-double-quotes-test
(testing (str "BOT-1587: when the LLM quotes a cross-stage column name like a SQL identifier\n"
@@ -1887,17 +1891,16 @@
(is (= "type/Integer" (get opts "base-type")))
(is (= "type/Integer" (get opts "effective-type")))))))
-(deftest ^:parallel cross-stage-field-type-unmatched-quoted-name-left-alone-test
+(deftest ^:parallel cross-stage-field-type-unmatched-quoted-name-raises-test
(testing (str "Quote-stripping only rewrites when the stripped name matches a real column.\n"
- "A quoted name whose stripped form still isn't produced by the previous stage is\n"
- "left verbatim (the resolver surfaces the real error with a better message).")
- (let [q (assoc-in multi-stage-base-query
- ["stages" 1 "filters" 0 2 2] "\"no_such_column\"")
- out (repair/repair mp-fks q)
- field-clause (get-in out ["stages" 1 "filters" 0 2])
- opts (nth field-clause 1)]
- (is (= "\"no_such_column\"" (nth field-clause 2)) "name left untouched")
- (is (not (contains? opts "base-type"))))))
+ "A quoted name whose stripped form still isn't produced by the previous stage\n"
+ "raises an :agent-error? listing the valid columns.")
+ (let [q (assoc-in multi-stage-base-query
+ ["stages" 1 "filters" 0 2 2] "\"no_such_column\"")
+ e (try (repair/repair mp-fks q) nil (catch clojure.lang.ExceptionInfo ex ex))]
+ (is (some? e))
+ (is (true? (:agent-error? (ex-data e))))
+ (is (= ["ID" "count"] (:available (ex-data e)))))))
(deftest ^:parallel cross-stage-field-type-idempotent-test
(testing "cross-stage field-type inference is idempotent"
@@ -1933,25 +1936,57 @@
(is (contains? (nth field-clause 1) "base-type")))
(testing "idempotent"
(is (= out (repair/repair mp-fks out))))))
- (testing "a loose key matching no real column is left untouched for the resolver"
- (let [out (repair/repair mp-fks
- (assoc-in multi-stage-base-query
- ["stages" 1 "filters" 0 2 2] "totally-unknown"))]
- (is (= "totally-unknown" (get-in out ["stages" 1 "filters" 0 2 2])))))
- (testing "a loose key colliding with two columns is left for the resolver (the hits>1 guard)"
- ;; `normalize-col-key` folds case and hyphen/space to underscore, so `Count Where` and
- ;; `count-where` both normalize to `count_where`. Real lib output names don't collide like this,
- ;; so exercise the guard directly on the private matcher.
+ (testing "a loose key matching no real column raises an :agent-error? listing valid names"
+ (let [e (try (repair/repair mp-fks
+ (assoc-in multi-stage-base-query
+ ["stages" 1 "filters" 0 2 2] "totally-unknown"))
+ nil (catch clojure.lang.ExceptionInfo ex ex))]
+ (is (some? e))
+ (is (true? (:agent-error? (ex-data e))))
+ (is (= ["ID" "count"] (:available (ex-data e))))))
+ (testing "a loose key colliding with two columns is left unmatched (the hits>1 guard)"
+ ;; `normalize-col-key` folds case and hyphen/space to underscore, so both descriptors'
+ ;; names AND display-names collide on `count_where`; neither the name nor the display-name
+ ;; layer can pick a single column, so the matcher returns nil. Real lib output names don't
+ ;; collide like this, so exercise the guard directly on the private matcher.
(is (nil? (#'repair/match-cross-stage-column
- {"Count Where" {"base-type" "type/Integer"}
- "count-where" {"base-type" "type/Integer"}}
+ [{:name "Count Where" :display-name "Count Where" :types {"base-type" "type/Integer"}}
+ {:name "count-where" :display-name "count-where" :types {"base-type" "type/Integer"}}]
"count_where")))
(testing "but a single loose hit still resolves"
(is (= ["count_where" {"base-type" "type/Integer"}]
(#'repair/match-cross-stage-column
- {"count_where" {"base-type" "type/Integer"}}
+ [{:name "count_where" :display-name "Count Where" :types {"base-type" "type/Integer"}}]
"Count-Where"))))))
+(deftest ^:parallel cross-stage-field-type-recovers-display-name-test
+ (testing (str "BOT-1442: a cross-stage ref written with an aggregation column's UI display\n"
+ "label (`Max of ID`) instead of its machine name (`max`) is recovered - the ref\n"
+ "is rewritten to the machine name and `base-type` stamped - so the query stays\n"
+ "runnable instead of silently breaking the notebook editor.")
+ (let [stage0 {"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "aggregation" [["max" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "ID"]]]]
+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]
+ (doseq [label ["Max of ID" ; exact display label
+ "max of id"]] ; folded variant, mirroring the Cynthia ref's casing
+ (testing (str "ref written as " (pr-str label))
+ (let [q {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [stage0
+ {"lib/type" "mbql.stage/mbql"
+ "breakout" [["field" {} label]]
+ "aggregation" [["count" {}]]}]}
+ out (repair/repair mp-fks q)
+ field-clause (get-in out ["stages" 1 "breakout" 0])
+ opts (nth field-clause 1)]
+ (testing "the display label is rewritten to the machine name"
+ (is (= "max" (nth field-clause 2))))
+ (testing "base-type is stamped so the ref is schema-valid"
+ (is (= "type/Integer" (get opts "base-type"))))
+ (testing "idempotent"
+ (is (= out (repair/repair mp-fks out))))))))))
+
(deftest ^:parallel cross-stage-field-type-end-to-end-resolve-test
(testing (str "End-to-end: a multi-stage YAML with a stage-1 cross-stage ref lacking\n"
"base-type is repaired and then `resolve-query` + `lib/query` accept the\n"
diff --git a/test/metabase/metabot/tools/construct_representations_test.clj b/test/metabase/metabot/tools/construct_representations_test.clj
index 6138c96fefc..806889a5a72 100644
--- a/test/metabase/metabot/tools/construct_representations_test.clj
+++ b/test/metabase/metabot/tools/construct_representations_test.clj
@@ -83,6 +83,25 @@
api/query-check allow-read-check]
(f)))
+(def ^:private mp-coerced
+ "ORDERS with a UNIX-seconds column coerced to a timestamp: `base-type :type/Integer`,
+ `effective-type :type/DateTime`. Exercises the editor gate's unsuppressed expression
+ type check against coerced columns (the #41122 false-positive class that
+ `*suppress-expression-type-check?*` exists for)."
+ (lib.tu/mock-metadata-provider
+ {:database {:id 1 :name "Sample"}
+ :tables [{:id 10 :name "ORDERS" :schema "PUBLIC" :db-id 1}]
+ :fields [{:id 100 :name "ID" :table-id 10 :base-type :type/Integer}
+ {:id 104 :name "CREATED_AT_UNIX" :table-id 10 :base-type :type/Integer
+ :effective-type :type/DateTime :coercion-strategy :Coercion/UNIXSeconds->DateTime}]}))
+
+(defn- with-coerced-mp-and-stubs! [f]
+ (with-redefs [lib-be/application-database-metadata-provider (fn [_db-id] mp-coerced)
+ construct/resolve-database-id-from-first-stage (fn [_] 1)
+ api/read-check allow-read-check
+ api/query-check allow-read-check]
+ (f)))
+
(defn- query-data
"Build the keyword-keyed external-query map the tool accepts. Tests author the query as a
string-keyed map (the form documented in earlier YAML-era tests); we walk-keywordize so the
@@ -486,26 +505,211 @@
(deftest multi-stage-unknown-cross-stage-column-surfaces-error-test
(testing (str "If the LLM references a column name that the previous stage doesn't produce,\n"
- "repair leaves the clause alone (no `base-type` to infer) and the lib-schema\n"
- "validator surfaces the missing-key error - reflagged as :agent-error?\n"
- "by `execute-representations-query`'s catch.")
+ "the cross-stage assert pass raises an :agent-error? naming the column and\n"
+ "listing the valid ones, rather than passing a typeless ref through into a\n"
+ "non-runnable query (BOT-1442).")
(with-mp-and-stubs!
(fn []
- (try
- (construct/execute-representations-query
- (query-data
- {"lib/type" "mbql/query"
- "database" "Sample"
- "stages" [{"lib/type" "mbql.stage/mbql"
- "source-table" ["Sample" "PUBLIC" "ORDERS"]
- "aggregation" [["count" {}]]
- "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}
- {"lib/type" "mbql.stage/mbql"
- "filters" [[">" {} ["field" {} "no_such_column"] 10]]}]}))
- (is false "expected throw")
- (catch clojure.lang.ExceptionInfo e
- (let [d (ex-data e)]
- (is (true? (:agent-error? d))))))))))
+ (let [e (try
+ (construct/execute-representations-query
+ (query-data
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "aggregation" [["count" {}]]
+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}
+ {"lib/type" "mbql.stage/mbql"
+ "filters" [[">" {} ["field" {} "no_such_column"] 10]]}]}))
+ nil (catch clojure.lang.ExceptionInfo ex ex))]
+ (is (some? e))
+ (let [d (ex-data e)]
+ (is (true? (:agent-error? d)))
+ (is (= :unresolved-cross-stage-field (:error d)))
+ (is (contains? (set (:available d)) "count"))
+ (is (re-find #"no_such_column" (ex-message e)))))))))
+
+(deftest multi-stage-aggregation-display-name-recovered-end-to-end-test
+ (testing (str "BOT-1442 (Cynthia): a stage-1 breakout that references a previous stage's\n"
+ "aggregation column by its UI display label (`Max of Doubled`) instead of the\n"
+ "machine name (`max`) is recovered end-to-end: the tool succeeds, the ref is\n"
+ "rewritten to `max` with a stamped base-type, and the resolved query is runnable.")
+ (with-mp-and-stubs!
+ (fn []
+ (let [result (construct/execute-representations-query
+ (query-data
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "expressions" {"Doubled" ["*" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] 2]}
+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]
+ "aggregation" [["max" {} ["expression" {} "Doubled"]]]}
+ {"lib/type" "mbql.stage/mbql"
+ "breakout" [["field" {} "Max of Doubled"]]
+ "aggregation" [["count" {}]]}]}))
+ q (get-in result [:structured-output :query])
+ stage1-bo (get-in q [:stages 1 :breakout 0])]
+ (testing "the display label was rewritten to the aggregation's machine name"
+ (is (= "max" (nth stage1-bo 2))))
+ (testing "the recovered ref carries a base-type (schema-valid, runnable)"
+ (is (some? (:base-type (nth stage1-bo 1))))
+ (is (nil? (#'construct/query-not-runnable-explanation q)))))))))
+
+;;; ============================================================
+;;; Expression-editor diagnostics gate (post-resolve, mirrors FE `diagnose-expression`)
+;;; ============================================================
+
+(defn- editor-rejection
+ "Run `query` through the tool and return the thrown ExceptionInfo, or nil on success."
+ [string-keyed-query]
+ (try
+ (construct/execute-representations-query (query-data string-keyed-query))
+ nil
+ (catch clojure.lang.ExceptionInfo ex ex)))
+
+(deftest offset-in-custom-column-surfaces-error-end-to-end-test
+ (testing (str "BOT-1442 sibling: an `offset` in a custom column surfaces a loud, retryable\n"
+ "agent-error through the tool, instead of building a query the editor silently\n"
+ "can't run. The resolved query schema rejects any aggregation/window clause in\n"
+ "`:expressions`, so this is caught by the canRun-mirror gate, whose message\n"
+ "points the LLM at the misplaced-`offset` cause.")
+ (with-mp-and-stubs!
+ (fn []
+ (doseq [exprs [{"Prev" ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]}
+ ;; offset nested inside a larger expression is caught too
+ {"Delta" ["-" {}
+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]]
+ ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]]}]]
+ (let [e (editor-rejection
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "expressions" exprs
+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]})]
+ (is (some? e))
+ (is (true? (:agent-error? (ex-data e))))
+ (is (= :query-not-runnable (:error (ex-data e))))
+ (is (re-find #"`offset`" (ex-message e)))
+ (is (re-find #"non-aggregation expression" (ex-message e)))))))))
+
+(deftest editor-gate-offset-in-filter-surfaces-error-end-to-end-test
+ (testing (str "An `offset` anywhere inside a `filters:` clause surfaces an\n"
+ ":expression-editor-rejection agent-error - the editor rejects OFFSET in\n"
+ "custom filters even though the query schema accepts it.")
+ (with-mp-and-stubs!
+ (fn []
+ (let [e (editor-rejection
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "filters" [[">" {}
+ ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]
+ 10]]}]})]
+ (is (some? e))
+ (is (true? (:agent-error? (ex-data e))))
+ (is (= :expression-editor-rejection (:error (ex-data e))))
+ (is (= :filter (:mode (ex-data e))))
+ (is (re-find #"OFFSET is not supported in custom filters" (ex-message e))))))))
+
+(deftest editor-gate-offset-nested-in-aggregation-function-surfaces-error-end-to-end-test
+ (testing (str "A window function embedded *inside* an aggregation function\n"
+ "(`sum(offset(...))`) surfaces an :expression-editor-rejection agent-error -\n"
+ "only the outer position (`offset(sum(...))`) is supported.")
+ (with-mp-and-stubs!
+ (fn []
+ (let [e (editor-rejection
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "aggregation" [["sum" {}
+ ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]]]
+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]})]
+ (is (some? e))
+ (is (true? (:agent-error? (ex-data e))))
+ (is (= :expression-editor-rejection (:error (ex-data e))))
+ (is (= :aggregation (:mode (ex-data e)))))))))
+
+(deftest editor-gate-offset-in-supported-positions-passes-end-to-end-test
+ (testing "`offset` in its supported positions does NOT trigger the editor gate"
+ (with-mp-and-stubs!
+ (fn []
+ (testing "at the top of an aggregation (`offset(sum(...), -1)`)"
+ (is (some? (construct/execute-representations-query
+ (query-data
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "aggregation" [["offset" {}
+ ["sum" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]]]
+ -1]]
+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]})))))
+ (testing "inside an `order-by`"
+ (is (some? (construct/execute-representations-query
+ (query-data
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "order-by" [["asc" {}
+ ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]]]}]})))))))))
+
+(deftest editor-gate-coerced-column-expression-passes-end-to-end-test
+ (testing (str "The editor gate's unsuppressed type check does not false-positive on a\n"
+ "coerced column: an expression that uses the column per its effective-type\n"
+ "(`get-month` on a UNIX-seconds -> DateTime coercion) builds successfully.")
+ (with-coerced-mp-and-stubs!
+ (fn []
+ (is (some? (construct/execute-representations-query
+ (query-data
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "expressions" {"Created Month" ["get-month" {}
+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "CREATED_AT_UNIX"]]]}}]}))))))))
+
+(deftest editor-gate-type-incompatible-expression-surfaces-error-end-to-end-test
+ (testing (str "The editor gate type-checks expressions (which the canRun-mirror schema gate\n"
+ "suppresses): a non-boolean clause in `filters:` surfaces an\n"
+ ":expression-editor-rejection agent-error.")
+ (with-mp-and-stubs!
+ (fn []
+ (let [e (editor-rejection
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "filters" [["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]]]}]})]
+ (is (some? e))
+ (is (true? (:agent-error? (ex-data e))))
+ (is (= :expression-editor-rejection (:error (ex-data e))))
+ (is (= :filter (:mode (ex-data e)))))))))
+
+(deftest query-not-runnable-explanation-gate-test
+ (testing (str "The runnability backstop returns nil for a runnable resolved query and a\n"
+ "non-nil Malli explanation for one whose breakout field ref is missing its\n"
+ "base-type (the BOT-1442 schema invalidity that disables the editor's run/save).")
+ (with-mp-and-stubs!
+ (fn []
+ (let [result (construct/execute-representations-query
+ (query-data
+ {"lib/type" "mbql/query"
+ "database" "Sample"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-table" ["Sample" "PUBLIC" "ORDERS"]
+ "aggregation" [["count" {}]]
+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]}))
+ good (get-in result [:structured-output :query])
+ ;; Corrupt a breakout field ref into a string-named ref with no base-type - the
+ ;; exact shape that resolves structurally yet makes Lib.canRun false.
+ bad (assoc-in good [:stages 0 :breakout 0] [:field {} "PRODUCT_ID"])]
+ (is (nil? (#'construct/query-not-runnable-explanation good)))
+ (is (some? (#'construct/query-not-runnable-explanation bad))))))))
;;; ============================================================
;;; Step-9 contract: expressions (custom columns)