The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
[BOT-1442] Fix metabot silently producing non-runnable MBQL (#76133)
Tyler Plude79002d46661e
diff --git a/resources/metabot/prompts/tools/construct_notebook_query.md b/resources/metabot/prompts/tools/construct_notebook_query.mdindex 54203cf4752..8058fea2649 100644--- a/resources/metabot/prompts/tools/construct_notebook_query.md+++ b/resources/metabot/prompts/tools/construct_notebook_query.md@@ -189,7 +189,11 @@ Re-aggregate — average daily total by month:]```-- Cross-stage refs use a **string name** in slot 3 (e.g. `["field", {}, "count"]`). The name is whatever the previous stage's aggregation/breakout/field produced (`count`, `sum`, or the source field's name).+- Cross-stage refs use a **string name** in slot 3 (e.g. `["field", {}, "count"]`). The name is the previous stage column's **machine name**, never its UI display label:+ - An aggregation's output name is the **bare function**: `max`, `min`, `avg`, `sum`, `count`, `median` — the operand is dropped. `["max", {}, ["expression", {}, "Instance Has SC"]]` produces a column named **`max`** (display label "Max of Instance Has SC"). Reference it as `["field", {}, "max"]`, **not** `["field", {}, "Max of Instance Has SC"]`.+ - A breakout keeps the source field's machine name even when its label is bucketed: a month breakout of `CREATED_AT` is still named `CREATED_AT` (display "Created At: Month").+ - If a stage has several aggregations of the same function, the outputs are suffixed in order: `max`, `max_2`, `max_3`. So the second `max` is referenced as `["field", {}, "max_2"]`.+- A `["field", {}, "<name>"]` cross-stage ref whose name matches no column the previous stage produced is an error — name it exactly.- Within the **same stage**, refer to your own aggregation with `["aggregation", {}, <idx>]` (see §Aggregation references). In a **later** stage, use the cross-stage string-name form against the previous stage's output.- Joins, expressions, filters, aggregation, breakout, order-by, limit are all valid in later stages.@@ -199,7 +203,7 @@ Instead of `source-table`, use `source-card` to query an existing question or mo1. Get the `portable_entity_id` from a tool response. `search` and `read_resource` (`metabase://question/<id>`, `metabase://model/<id>`) include it on the result tag — reuse what's already in context, no extra call needed.2. Copy it **verbatim** into `source-card`. The id is opaque — never guess, construct, or abbreviate.-3. Reference the card's columns by output **name** (string in slot 3), not portable FK. If you don't know the names, call `read_resource metabase://question/<numeric-id>/fields` (or `.../model/...`).+3. Reference the card's columns by output **machine name** (string in slot 3), not portable FK and not the UI display label. If you don't know the names, call `read_resource metabase://question/<numeric-id>/fields` (or `.../model/...`) — a ref whose name matches no column the card produces is an error.```json{"lib/type": "mbql.stage/mbql",@@ -389,6 +393,9 @@ Temporal:- `["date", {}, <expr>]` / `["datetime", {}, <expr>]` / `["time", {}, <expr>]`- `["now", {}]` / `["today", {}]`+Window:+- `["offset", {}, <expr>, <n>]` — value of `<expr>` from `<n>` rows back (negative `<n>`) or ahead (positive). Valid **only** inside an `aggregation:` or `order-by:` clause — **never** in a custom column (`expressions:`) or a filter.+### Temporal units (for `{"temporal-unit": ...}` on field refs and as `<unit>` args)- Date truncation: `day`, `week`, `month`, `quarter`, `year`diff --git a/src/metabase/agent_lib/representations/repair.clj b/src/metabase/agent_lib/representations/repair.cljindex df79beb63e5..299edb73072 100644--- a/src/metabase/agent_lib/representations/repair.clj+++ b/src/metabase/agent_lib/representations/repair.clj@@ -31,6 +31,7 @@[clojure.walk :as walk][metabase.agent-lib.representations.resolve :as repr.resolve][metabase.lib.core :as lib]+ [metabase.lib.expression :as lib.expression][metabase.lib.metadata.protocols :as lib.metadata.protocols][metabase.lib.schema.mbql-clause :as mbql-clause][metabase.models.serialization.resolve :as resolve]@@ -2232,10 +2233,22 @@(assoc "effective-type" (let [et (:effective-type col)](if (keyword? et) (subs (str et) 1) (str et))))))))-(defn- mini-resolved-columns-by-name+(defn- column-descriptors+ "Build the cross-stage match oracle from a `lib/returned-columns` sequence: a vector of+ `{:name <machine-name>, :display-name <display-name>, :types {\"base-type\" …}|nil}`+ descriptors. `:types` is nil when the column exposes no usable `base-type`. Carrying the+ `:display-name` lets [[match-cross-stage-column]] recover a ref the LLM wrote with the UI+ label (e.g. `\"Max of Instance Has SC\"`) instead of the machine name (`max`)."+ [cols]+ (mapv (fn [col]+ {:name (:name col)+ :display-name (:display-name col)+ :types (types-from-column col)})+ cols))++(defn- mini-resolved-columns"Resolve `stages[0..idx-1]` as a self-contained query, run it through `lib/query`, and return- a `{column-name → {\"base-type\" ..., \"effective-type\" ...}}` map for the columns the- prefix returns.+ the prefix's returned columns as a vector of [[column-descriptors]].Returns `nil` (logged at debug) if anything in the resolve / lib/query path throws. Theenclosing pass treats `nil` as 'skip this stage's repairs and let downstream surface the real@@ -2247,11 +2260,7 @@resolved (repr.resolve/resolve-query mp prefix-query content-store)lib-q (lib/query mp resolved)cols (lib/returned-columns lib-q)]- (into {}- (keep (fn [col]- (when-let [types (types-from-column col)]- [(:name col) types])))- cols))+ (column-descriptors cols))(catch Exception e(log/debugf e "[repr-repair] mini-resolve of stages[0..%d] failed; skipping cross-stage type inference for stage %d"(dec stage-idx) stage-idx)@@ -2283,34 +2292,34 @@(-> s u/lower-case-en (str/replace #"[-\s]+" "_"))))(defn- match-cross-stage-column- "Find the canonical column entry in `name->types` for `col-name`. Returns `[canonical-name- types]` or `nil` when no column matches.+ "Find the column descriptor in `cols` (a vector of [[column-descriptors]]) that `col-name`+ refers to. Returns `[canonical-machine-name types]` or `nil` when no column matches.Tries, in order:- 1. an exact match;- 2. the double-quote-stripped name (only when stripping actually resolves);- 3. a loose match that folds case and hyphens/whitespace to underscores, but **only when- exactly one** real column matches - so `count-where` resolves to `count_where` while- genuine ambiguity is left for the resolver to report.-- A column whose name legitimately needs quotes (vanishingly rare) is never clobbered, and- unmatched names are passed through for the resolver to surface with a better message."- [name->types col-name]- (let [stripped (strip-surrounding-double-quotes col-name)]- (cond- (contains? name->types col-name)- [col-name (get name->types col-name)]-- (and (not= stripped col-name) (contains? name->types stripped))- [stripped (get name->types stripped)]-- :else- (let [target (normalize-col-key stripped)- hits (when target- (filter #(= target (normalize-col-key %)) (keys name->types)))]- (when (= 1 (count hits))- (let [canon (first hits)]- [canon (get name->types canon)]))))))+ 1. an exact `:name` match;+ 2. the double-quote-stripped `:name` (only when stripping actually resolves);+ 3. a loose `:name` match that folds case and hyphens/whitespace to underscores, but **only+ when exactly one** real column matches - so `count-where` resolves to `count_where`;+ 4. a loose `:display-name` match (same folding, same one-hit guard) - so a ref the LLM+ wrote with the UI label (`\"Max of Instance Has SC\"`) recovers the machine name (`max`).++ Genuine ambiguity (more than one loose hit) is left unmatched at each layer. A column whose+ name legitimately needs quotes (vanishingly rare) is never clobbered."+ [cols col-name]+ (let [stripped (strip-surrounding-double-quotes col-name)+ exact (fn [n] (some #(when (= n (:name %)) %) cols))+ loose-on (fn [key-fn raw]+ (let [target (normalize-col-key raw)+ hits (when target+ (filter #(= target (normalize-col-key (key-fn %))) cols))]+ (when (= 1 (count hits))+ (first hits))))+ match (or (exact col-name)+ (when (not= stripped col-name) (exact stripped))+ (loose-on :name stripped)+ (loose-on :display-name stripped))]+ (when match+ [(:name match) (:types match)])))(defn- maybe-fill-cross-stage-types"Given a cross-stage field clause and a name→types map, return the clause with its column name@@ -2319,17 +2328,19 @@Idempotent: once the name is canonical it matches exactly on the next pass (no rename), andonce `base-type` is present it is left alone."- [clause name->types]+ [clause cols](let [opts (nth clause 1)col-name (nth clause 2)]- (if-let [[canonical-name types] (match-cross-stage-column name->types col-name)]+ (if-let [[canonical-name types] (match-cross-stage-column cols col-name)](cond-> clause- ;; Canonicalise the name when quote-stripping was needed to match. A bare `base-type`- ;; stamp without this would leave the ref pointing at a non-existent column.+ ;; Canonicalise the name when quote-stripping / display-name recovery was needed to+ ;; match. A bare `base-type` stamp without this would leave the ref pointing at a+ ;; non-existent column.(not= canonical-name col-name)(assoc 2 canonical-name)- ;; Stamp inferred types only when the LLM didn't author a `base-type` already.- (not (contains? opts "base-type"))+ ;; Stamp inferred types only when the LLM didn't author a `base-type` already and the+ ;; matched column actually exposes one.+ (and types (not (contains? opts "base-type")))(assoc 1 (merge opts types)))clause)))@@ -2338,13 +2349,13 @@`\"base-type\"`. Skips descent into `\"joins\"` subtrees - join stages have their ownresolution context (the join's own `stages`) and shouldn't reach into their parent stage'sprevious-stage columns."- [stage name->types]+ [stage cols](let [joins (get stage "joins")stage' (cond-> stage (contains? stage "joins") (dissoc "joins"))walked (walk/postwalk(fn [node](if (string-cross-stage-field-clause? node)- (maybe-fill-cross-stage-types node name->types)+ (maybe-fill-cross-stage-types node cols)node))stage')](cond-> walked@@ -2353,8 +2364,8 @@(defn- mini-resolved-columns-for-source-card"Resolve a single-stage query consisting only of `{:source-card <entity-id>}` (plus theouter `mbql/query` wrapper and the warehouse database name), then return the- `lib/returned-columns` output as a `name → types` map in the same shape as- [[mini-resolved-columns-by-name]]. Used by [[infer-source-card-field-types*]] to stamp+ `lib/returned-columns` output as a vector of [[column-descriptors]] in the same shape as+ [[mini-resolved-columns]]. Used by [[infer-source-card-field-types*]] to stamp`base-type` onto `[field, {}, \"<col>\"]` clauses in a stage whose source is a savedquestion / model.@@ -2372,11 +2383,7 @@resolved (repr.resolve/resolve-query mp bare-query content-store)lib-q (lib/query mp resolved)cols (lib/returned-columns lib-q)]- (into {}- (keep (fn [col]- (when-let [types (types-from-column col)]- [(:name col) types])))- cols))+ (column-descriptors cols))(catch Exception e(log/debugf e "[repr-repair] source-card resolve of stage %d failed; skipping field-type inference"stage-idx)@@ -2400,8 +2407,8 @@q(let [stage (get-in q ["stages" i])q' (if (and (map? stage) (get stage "source-card"))- (if-let [name->types (mini-resolved-columns-for-source-card mp q i content-store)]- (update-in q ["stages" i] infer-cross-stage-field-types-in-stage name->types)+ (if-let [cols (mini-resolved-columns-for-source-card mp q i content-store)]+ (update-in q ["stages" i] infer-cross-stage-field-types-in-stage cols)q)q)](recur (inc i) q')))))))@@ -2420,12 +2427,75 @@q query](if (>= i n)q- (let [name->types (mini-resolved-columns-by-name mp q i content-store)- q' (if name->types- (update-in q ["stages" i] infer-cross-stage-field-types-in-stage name->types)- q)]+ (let [cols (mini-resolved-columns mp q i content-store)+ q' (if cols+ (update-in q ["stages" i] infer-cross-stage-field-types-in-stage cols)+ q)](recur (inc i) q')))))))+;;; ============================================================+;;; Pass 5.7 -- assert cross-stage / source-card string refs resolved+;;;+;;; A `["field" {} "<name>"]` ref into a previous stage or a source-card must name a real+;;; output column. The inference passes above (5 / 5.5) stamp `base-type` onto every such ref+;;; that matches; anything still missing `base-type` afterward matched nothing - the LLM+;;; named a column that doesn't exist (typically a UI display label like "Max of X" instead of+;;; the machine name "max"). lib's resolver passes a typeless string-named field ref through+;;; unvalidated, producing a schema-invalid, non-runnable query that silently breaks the+;;; notebook editor (BOT-1442). This pass turns that silent failure into the same loud,+;;; retryable `:agent-error?` the LLM already recovers from for dangling `[:expression …]`+;;; refs, naming the offending column and listing the valid ones.+;;; ============================================================++(defn- unstamped-cross-stage-ref?+ "True if `node` is a `[\"field\" {} \"<name>\"]` cross-stage/source-card ref still lacking+ `base-type` (an LLM-authored `base-type` is left for the resolver / schema to judge)."+ [node]+ (and (string-cross-stage-field-clause? node)+ (not (contains? (nth node 1) "base-type"))))++(defn- stage-has-unstamped-cross-stage-ref?+ "Cheap structural pre-scan: does `stage` contain any [[unstamped-cross-stage-ref?]] outside+ its `joins` subtrees? Lets the assert pass skip the (resolving) confirmation step entirely+ on the happy path, where every ref already carries a stamped `base-type`."+ [stage]+ (let [stage' (cond-> stage (contains? stage "joins") (dissoc "joins"))]+ (boolean (some unstamped-cross-stage-ref? (tree-seq coll? seq stage')))))++(defn- first-unresolved-cross-stage-ref+ "Return the first [[unstamped-cross-stage-ref?]] clause in `stage` that matches no column in+ `cols`, or nil. Skips `joins` subtrees (their own resolution context)."+ [stage cols]+ (let [stage' (cond-> stage (contains? stage "joins") (dissoc "joins"))]+ (some (fn [node]+ (when (and (unstamped-cross-stage-ref? node)+ (nil? (match-cross-stage-column cols (nth node 2))))+ node))+ (tree-seq coll? seq stage'))))++(defn- assert-cross-stage-refs-resolved*+ "Pass 5.7: raise an `:agent-error?` for any string-named cross-stage / source-card field ref+ that resolves to no real column. No-op when `mp` is nil. Only stages that still carry an+ unstamped ref pay the cost of re-resolving their column universe to build the message."+ [query mp content-store]+ (when (and mp (map? query) (vector? (get query "stages")))+ (doseq [[idx stage] (map-indexed vector (get query "stages"))+ :when (and (map? stage) (stage-has-unstamped-cross-stage-ref? stage))]+ (when-let [cols (cond+ (get stage "source-card") (mini-resolved-columns-for-source-card mp query idx content-store)+ (pos? idx) (mini-resolved-columns mp query idx content-store))]+ (when-let [bad (first-unresolved-cross-stage-ref stage cols)]+ (throw (ex-info+ (tru "No column named {0} is available to reference in this stage. Reference a column by its machine name (e.g. an aggregation is `max`, `count`, `sum` - never its display label like `Max of X`), not a UI label. Available column names: {1}."+ (pr-str (nth bad 2))+ (str/join ", " (map :name cols)))+ {:agent-error? true+ :error :unresolved-cross-stage-field+ :stage idx+ :clause bad+ :available (mapv :name cols)}))))))+ query)+;;; ============================================================;;; Pass 6 -- friendly error messages for silently-accepted-but-wrong shapes.;;;@@ -2698,14 +2768,19 @@prefix of stages and reading the returned columns' metadata.5.5. infer `base-type` / `effective-type` on field references in a stage whose source isa saved question / model (`source-card:`), using the card's resolved returned columns.-- Pass 4, Pass 5, and Pass 5.5 require `mp` (a `MetadataProvider`); they are best-effort- no-ops when `mp` can't resolve the relevant pieces (so the subsequent validate/resolve- stages can surface the real error with their own, better messages). Hard FK errors from- Pass 4 (`:no-fk-path`, `:ambiguous-fk`) are raised as `:agent-error?` ex-info so the tool- wrapper can relay them to the LLM. Pass 5 and Pass 5.5 never throw on their own - if a- prefix / source-card can't be resolved, they just leave the affected clauses alone and let- the schema validator complain.+ 5.7. assert that every string-named cross-stage / source-card field ref resolved to a real+ column (i.e. Pass 5 / 5.5 stamped its `base-type`). A ref that matched nothing is the+ LLM naming a column that doesn't exist (often a display label); raise an `:agent-error?`+ naming it and listing the valid columns, instead of letting it pass through into a+ schema-invalid, non-runnable query (BOT-1442).++ Pass 4, Pass 5, Pass 5.5, and Pass 5.7 require `mp` (a `MetadataProvider`); they are+ best-effort no-ops when `mp` can't resolve the relevant pieces (so the subsequent+ validate/resolve stages can surface the real error with their own, better messages). Hard FK+ errors from Pass 4 (`:no-fk-path`, `:ambiguous-fk`) are raised as `:agent-error?` ex-info so+ the tool wrapper can relay them to the LLM. Pass 5 and Pass 5.5 never throw on their own - if+ a prefix / source-card can't be resolved, they just leave the affected clauses alone; Pass+ 5.7 then raises the `:agent-error?` only for refs that survived to the end still unresolved.Note: the database-name normalisation pass (\"Pass 2.5\") that previously lived here wasremoved in `repr-plan.md` step 13, and the top-level `database:` field was removed from@@ -2737,4 +2812,46 @@(resolve-implicit-joins* mp content-store)(infer-cross-stage-field-types* mp content-store)(infer-source-card-field-types* mp content-store)+ (assert-cross-stage-refs-resolved* mp content-store)friendly-errors*)))++;;; ============================================================+;;; Post-resolve gate -- expressions the FE expression editor rejects+;;; ============================================================++(defn assert-editor-accepts-expressions!+ "Run the FE expression editor's own validation, [[metabase.lib.expression/diagnose-expression]],+ over every custom column, aggregation, and filter of the resolved `pmbql-query`, throwing a+ retryable `:agent-error?` on the first clause it rejects. Returns `pmbql-query`.++ Catches shapes that are schema-legal yet broken in the editor: `offset` in a filter, window+ functions nested inside aggregation functions, cyclic expression references, type-incompatible+ arguments, standalone constants without `:expression-literals` support.++ Unlike the rest of this namespace this runs after resolve. Call it only on a query that already+ passed the suppressed-type-check `::lib.schema/query` validation - `diagnose-expression`+ validates its query argument against that same schema, and its raw validation error would+ otherwise preempt the friendly message."+ [pmbql-query]+ (doseq [stage-idx (range (lib/stage-count pmbql-query))+ [mode clauses] [[:expression (lib/expressions pmbql-query stage-idx)]+ [:aggregation (lib/aggregations pmbql-query stage-idx)]+ [:filter (lib/filters pmbql-query stage-idx)]]+ [pos clause] (map-indexed vector clauses)]+ (when-let [{:keys [message]} (lib.expression/diagnose-expression+ pmbql-query stage-idx mode clause+ (when (#{:expression :aggregation} mode) pos))]+ (let [where (case mode+ :expression "custom column (`expressions:`)"+ :aggregation "`aggregation:`"+ :filter "`filters:`")]+ (throw (ex-info+ (tru "The query builder''s expression editor rejects a {0} clause in stage {1}: {2}. The query would build but could not be visualized or saved - rewrite the offending clause."+ where stage-idx message)+ {:agent-error? true+ :error :expression-editor-rejection+ :status-code 400+ :stage stage-idx+ :mode mode+ :clause clause})))))+ pmbql-query)diff --git a/src/metabase/metabot/tools/construct.clj b/src/metabase/metabot/tools/construct.cljindex 28de905256d..ecafcafca42 100644--- a/src/metabase/metabot/tools/construct.clj+++ b/src/metabase/metabot/tools/construct.clj@@ -1,12 +1,15 @@(ns metabase.metabot.tools.construct"Notebook query construction tool wrappers."(:require+ [malli.error :as me][metabase.agent-lib.representations :as repr][metabase.agent-lib.representations.repair :as repr.repair][metabase.agent-lib.representations.resolve :as repr.resolve][metabase.api.common :as api][metabase.lib-be.core :as lib-be][metabase.lib.core :as lib]+ [metabase.lib.schema]+ [metabase.lib.schema.expression :as lib.schema.expression][metabase.metabot.agent.links :as links][metabase.metabot.agent.streaming :as streaming][metabase.metabot.scope :as scope]@@ -24,6 +27,7 @@[metabase.util.json :as json][metabase.util.log :as log][metabase.util.malli :as mu]+ [metabase.util.malli.registry :as mr][toucan2.core :as t2]))(set! *warn-on-reflection* true)@@ -278,6 +282,14 @@(nil? (:status-code base)) (assoc :status-code 400))](ex-info (ex-message e) data e)))+(defn- query-not-runnable-explanation+ "When `pmbql-query` would make the FE's `canRun` gate return false, return a humanized Malli+ explanation of why; nil when the query is runnable."+ [pmbql-query]+ (binding [lib.schema.expression/*suppress-expression-type-check?* true]+ (when-let [explanation (mr/explain :metabase.lib.schema/query pmbql-query)]+ (me/humanize explanation))))+(defn execute-representations-query"Execute a notebook query in the canonical portable MBQL 5 representations format.@@ -302,6 +314,11 @@5. Sanity-check the post-repair shape against the portable repair schema.6. Resolve portable FKs to numeric IDs and normalize through `lib.schema/query` against themetadata-provider.+ 6.5. Backstop-gate the resolved query: mirror the FE `canRun` schema validation+ ([[query-not-runnable-explanation]]), then run the FE expression editor's own+ diagnostics over every custom column / aggregation / filter+ ([[repr.repair/assert-editor-accepts-expressions!]]). Either failure is a retryable+ `:agent-error?` - success on a query the editor rejects is BOT-1442.7. Export that final numeric pMBQL back to the portable form for the LLM-facing`:query-json` / `query-content` output.@@ -340,6 +357,18 @@(let [repaired (repr.repair/repair mp parsed permission-aware-content-store)_validated (repr/validate-query repaired)pmbql-query (repr.resolve/resolve-query mp repaired permission-aware-content-store)+ _runnable (when-let [why (query-not-runnable-explanation pmbql-query)]+ (throw (ex-info (tru "The constructed query is not runnable - it would fail the query builder''s validation, so it cannot be visualized or saved. This usually means a field reference is missing its type or names a column that does not exist, or an aggregation/window function (e.g. `offset`) was placed in `expressions:` (custom columns) where it is not allowed - move it to `aggregation:` or `order-by:`. Schema validation details: {0}"+ (pr-str why))+ {:agent-error? true+ :error :query-not-runnable+ :status-code 400})))+ ;; The FE expression editor rejects some shapes the (type-check-suppressed) canRun+ ;; gate above accepts - `offset` in a filter, window functions nested in+ ;; aggregations, cyclic expression refs, type-incompatible arguments. Must run+ ;; after the `_runnable` gate: `diagnose-expression` itself validates its query+ ;; argument against the same schema.+ _editor-ok (repr.repair/assert-editor-accepts-expressions! pmbql-query)exported-repr (repr.resolve/export-query mp pmbql-query permission-aware-content-store)_validated' (repr/validate-query exported-repr)query-id (u/generate-nano-id)]diff --git a/test/metabase/agent_lib/representations/repair_test.clj b/test/metabase/agent_lib/representations/repair_test.cljindex d70be7977ef..5cdf571b5d2 100644--- a/test/metabase/agent_lib/representations/repair_test.clj+++ b/test/metabase/agent_lib/representations/repair_test.clj@@ -1862,13 +1862,17 @@(is (not (contains? opts "base-type"))))))(deftest ^:parallel cross-stage-field-type-unknown-column-name-test- (testing "If the referenced name isn't produced by the previous stage, leave the clause- alone (the resolver will surface :unknown-column or similar with a better message)."+ (testing "A cross-stage ref to a name no previous stage produces raises an :agent-error?+ that names the bad column and lists the valid ones, rather than passing a typeless+ ref through into a non-runnable query."(let [q (assoc-in multi-stage-base-query["stages" 1 "filters" 0 2 2] "no_such_column")- out (repair/repair mp-fks q)- opts (get-in out ["stages" 1 "filters" 0 2 1])]- (is (not (contains? opts "base-type"))))))+ e (try (repair/repair mp-fks q) nil (catch clojure.lang.ExceptionInfo ex ex))]+ (is (some? e))+ (is (true? (:agent-error? (ex-data e))))+ (is (= :unresolved-cross-stage-field (:error (ex-data e))))+ (is (= ["ID" "count"] (:available (ex-data e))))+ (is (re-find #"no_such_column" (ex-message e))))))(deftest ^:parallel cross-stage-field-type-strips-surrounding-double-quotes-test(testing (str "BOT-1587: when the LLM quotes a cross-stage column name like a SQL identifier\n"@@ -1887,17 +1891,16 @@(is (= "type/Integer" (get opts "base-type")))(is (= "type/Integer" (get opts "effective-type")))))))-(deftest ^:parallel cross-stage-field-type-unmatched-quoted-name-left-alone-test+(deftest ^:parallel cross-stage-field-type-unmatched-quoted-name-raises-test(testing (str "Quote-stripping only rewrites when the stripped name matches a real column.\n"- "A quoted name whose stripped form still isn't produced by the previous stage is\n"- "left verbatim (the resolver surfaces the real error with a better message).")- (let [q (assoc-in multi-stage-base-query- ["stages" 1 "filters" 0 2 2] "\"no_such_column\"")- out (repair/repair mp-fks q)- field-clause (get-in out ["stages" 1 "filters" 0 2])- opts (nth field-clause 1)]- (is (= "\"no_such_column\"" (nth field-clause 2)) "name left untouched")- (is (not (contains? opts "base-type"))))))+ "A quoted name whose stripped form still isn't produced by the previous stage\n"+ "raises an :agent-error? listing the valid columns.")+ (let [q (assoc-in multi-stage-base-query+ ["stages" 1 "filters" 0 2 2] "\"no_such_column\"")+ e (try (repair/repair mp-fks q) nil (catch clojure.lang.ExceptionInfo ex ex))]+ (is (some? e))+ (is (true? (:agent-error? (ex-data e))))+ (is (= ["ID" "count"] (:available (ex-data e)))))))(deftest ^:parallel cross-stage-field-type-idempotent-test(testing "cross-stage field-type inference is idempotent"@@ -1933,25 +1936,57 @@(is (contains? (nth field-clause 1) "base-type")))(testing "idempotent"(is (= out (repair/repair mp-fks out))))))- (testing "a loose key matching no real column is left untouched for the resolver"- (let [out (repair/repair mp-fks- (assoc-in multi-stage-base-query- ["stages" 1 "filters" 0 2 2] "totally-unknown"))]- (is (= "totally-unknown" (get-in out ["stages" 1 "filters" 0 2 2])))))- (testing "a loose key colliding with two columns is left for the resolver (the hits>1 guard)"- ;; `normalize-col-key` folds case and hyphen/space to underscore, so `Count Where` and- ;; `count-where` both normalize to `count_where`. Real lib output names don't collide like this,- ;; so exercise the guard directly on the private matcher.+ (testing "a loose key matching no real column raises an :agent-error? listing valid names"+ (let [e (try (repair/repair mp-fks+ (assoc-in multi-stage-base-query+ ["stages" 1 "filters" 0 2 2] "totally-unknown"))+ nil (catch clojure.lang.ExceptionInfo ex ex))]+ (is (some? e))+ (is (true? (:agent-error? (ex-data e))))+ (is (= ["ID" "count"] (:available (ex-data e))))))+ (testing "a loose key colliding with two columns is left unmatched (the hits>1 guard)"+ ;; `normalize-col-key` folds case and hyphen/space to underscore, so both descriptors'+ ;; names AND display-names collide on `count_where`; neither the name nor the display-name+ ;; layer can pick a single column, so the matcher returns nil. Real lib output names don't+ ;; collide like this, so exercise the guard directly on the private matcher.(is (nil? (#'repair/match-cross-stage-column- {"Count Where" {"base-type" "type/Integer"}- "count-where" {"base-type" "type/Integer"}}+ [{:name "Count Where" :display-name "Count Where" :types {"base-type" "type/Integer"}}+ {:name "count-where" :display-name "count-where" :types {"base-type" "type/Integer"}}]"count_where")))(testing "but a single loose hit still resolves"(is (= ["count_where" {"base-type" "type/Integer"}](#'repair/match-cross-stage-column- {"count_where" {"base-type" "type/Integer"}}+ [{:name "count_where" :display-name "Count Where" :types {"base-type" "type/Integer"}}]"Count-Where"))))))+(deftest ^:parallel cross-stage-field-type-recovers-display-name-test+ (testing (str "BOT-1442: a cross-stage ref written with an aggregation column's UI display\n"+ "label (`Max of ID`) instead of its machine name (`max`) is recovered - the ref\n"+ "is rewritten to the machine name and `base-type` stamped - so the query stays\n"+ "runnable instead of silently breaking the notebook editor.")+ (let [stage0 {"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "aggregation" [["max" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "ID"]]]]+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]+ (doseq [label ["Max of ID" ; exact display label+ "max of id"]] ; folded variant, mirroring the Cynthia ref's casing+ (testing (str "ref written as " (pr-str label))+ (let [q {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [stage0+ {"lib/type" "mbql.stage/mbql"+ "breakout" [["field" {} label]]+ "aggregation" [["count" {}]]}]}+ out (repair/repair mp-fks q)+ field-clause (get-in out ["stages" 1 "breakout" 0])+ opts (nth field-clause 1)]+ (testing "the display label is rewritten to the machine name"+ (is (= "max" (nth field-clause 2))))+ (testing "base-type is stamped so the ref is schema-valid"+ (is (= "type/Integer" (get opts "base-type"))))+ (testing "idempotent"+ (is (= out (repair/repair mp-fks out))))))))))+(deftest ^:parallel cross-stage-field-type-end-to-end-resolve-test(testing (str "End-to-end: a multi-stage YAML with a stage-1 cross-stage ref lacking\n""base-type is repaired and then `resolve-query` + `lib/query` accept the\n"diff --git a/test/metabase/metabot/tools/construct_representations_test.clj b/test/metabase/metabot/tools/construct_representations_test.cljindex 6138c96fefc..806889a5a72 100644--- a/test/metabase/metabot/tools/construct_representations_test.clj+++ b/test/metabase/metabot/tools/construct_representations_test.clj@@ -83,6 +83,25 @@api/query-check allow-read-check](f)))+(def ^:private mp-coerced+ "ORDERS with a UNIX-seconds column coerced to a timestamp: `base-type :type/Integer`,+ `effective-type :type/DateTime`. Exercises the editor gate's unsuppressed expression+ type check against coerced columns (the #41122 false-positive class that+ `*suppress-expression-type-check?*` exists for)."+ (lib.tu/mock-metadata-provider+ {:database {:id 1 :name "Sample"}+ :tables [{:id 10 :name "ORDERS" :schema "PUBLIC" :db-id 1}]+ :fields [{:id 100 :name "ID" :table-id 10 :base-type :type/Integer}+ {:id 104 :name "CREATED_AT_UNIX" :table-id 10 :base-type :type/Integer+ :effective-type :type/DateTime :coercion-strategy :Coercion/UNIXSeconds->DateTime}]}))++(defn- with-coerced-mp-and-stubs! [f]+ (with-redefs [lib-be/application-database-metadata-provider (fn [_db-id] mp-coerced)+ construct/resolve-database-id-from-first-stage (fn [_] 1)+ api/read-check allow-read-check+ api/query-check allow-read-check]+ (f)))+(defn- query-data"Build the keyword-keyed external-query map the tool accepts. Tests author the query as astring-keyed map (the form documented in earlier YAML-era tests); we walk-keywordize so the@@ -486,26 +505,211 @@(deftest multi-stage-unknown-cross-stage-column-surfaces-error-test(testing (str "If the LLM references a column name that the previous stage doesn't produce,\n"- "repair leaves the clause alone (no `base-type` to infer) and the lib-schema\n"- "validator surfaces the missing-key error - reflagged as :agent-error?\n"- "by `execute-representations-query`'s catch.")+ "the cross-stage assert pass raises an :agent-error? naming the column and\n"+ "listing the valid ones, rather than passing a typeless ref through into a\n"+ "non-runnable query (BOT-1442).")(with-mp-and-stubs!(fn []- (try- (construct/execute-representations-query- (query-data- {"lib/type" "mbql/query"- "database" "Sample"- "stages" [{"lib/type" "mbql.stage/mbql"- "source-table" ["Sample" "PUBLIC" "ORDERS"]- "aggregation" [["count" {}]]- "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}- {"lib/type" "mbql.stage/mbql"- "filters" [[">" {} ["field" {} "no_such_column"] 10]]}]}))- (is false "expected throw")- (catch clojure.lang.ExceptionInfo e- (let [d (ex-data e)]- (is (true? (:agent-error? d))))))))))+ (let [e (try+ (construct/execute-representations-query+ (query-data+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "aggregation" [["count" {}]]+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}+ {"lib/type" "mbql.stage/mbql"+ "filters" [[">" {} ["field" {} "no_such_column"] 10]]}]}))+ nil (catch clojure.lang.ExceptionInfo ex ex))]+ (is (some? e))+ (let [d (ex-data e)]+ (is (true? (:agent-error? d)))+ (is (= :unresolved-cross-stage-field (:error d)))+ (is (contains? (set (:available d)) "count"))+ (is (re-find #"no_such_column" (ex-message e)))))))))++(deftest multi-stage-aggregation-display-name-recovered-end-to-end-test+ (testing (str "BOT-1442 (Cynthia): a stage-1 breakout that references a previous stage's\n"+ "aggregation column by its UI display label (`Max of Doubled`) instead of the\n"+ "machine name (`max`) is recovered end-to-end: the tool succeeds, the ref is\n"+ "rewritten to `max` with a stamped base-type, and the resolved query is runnable.")+ (with-mp-and-stubs!+ (fn []+ (let [result (construct/execute-representations-query+ (query-data+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "expressions" {"Doubled" ["*" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] 2]}+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]+ "aggregation" [["max" {} ["expression" {} "Doubled"]]]}+ {"lib/type" "mbql.stage/mbql"+ "breakout" [["field" {} "Max of Doubled"]]+ "aggregation" [["count" {}]]}]}))+ q (get-in result [:structured-output :query])+ stage1-bo (get-in q [:stages 1 :breakout 0])]+ (testing "the display label was rewritten to the aggregation's machine name"+ (is (= "max" (nth stage1-bo 2))))+ (testing "the recovered ref carries a base-type (schema-valid, runnable)"+ (is (some? (:base-type (nth stage1-bo 1))))+ (is (nil? (#'construct/query-not-runnable-explanation q)))))))))++;;; ============================================================+;;; Expression-editor diagnostics gate (post-resolve, mirrors FE `diagnose-expression`)+;;; ============================================================++(defn- editor-rejection+ "Run `query` through the tool and return the thrown ExceptionInfo, or nil on success."+ [string-keyed-query]+ (try+ (construct/execute-representations-query (query-data string-keyed-query))+ nil+ (catch clojure.lang.ExceptionInfo ex ex)))++(deftest offset-in-custom-column-surfaces-error-end-to-end-test+ (testing (str "BOT-1442 sibling: an `offset` in a custom column surfaces a loud, retryable\n"+ "agent-error through the tool, instead of building a query the editor silently\n"+ "can't run. The resolved query schema rejects any aggregation/window clause in\n"+ "`:expressions`, so this is caught by the canRun-mirror gate, whose message\n"+ "points the LLM at the misplaced-`offset` cause.")+ (with-mp-and-stubs!+ (fn []+ (doseq [exprs [{"Prev" ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]}+ ;; offset nested inside a larger expression is caught too+ {"Delta" ["-" {}+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]]+ ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]]}]]+ (let [e (editor-rejection+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "expressions" exprs+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]})]+ (is (some? e))+ (is (true? (:agent-error? (ex-data e))))+ (is (= :query-not-runnable (:error (ex-data e))))+ (is (re-find #"`offset`" (ex-message e)))+ (is (re-find #"non-aggregation expression" (ex-message e)))))))))++(deftest editor-gate-offset-in-filter-surfaces-error-end-to-end-test+ (testing (str "An `offset` anywhere inside a `filters:` clause surfaces an\n"+ ":expression-editor-rejection agent-error - the editor rejects OFFSET in\n"+ "custom filters even though the query schema accepts it.")+ (with-mp-and-stubs!+ (fn []+ (let [e (editor-rejection+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "filters" [[">" {}+ ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]+ 10]]}]})]+ (is (some? e))+ (is (true? (:agent-error? (ex-data e))))+ (is (= :expression-editor-rejection (:error (ex-data e))))+ (is (= :filter (:mode (ex-data e))))+ (is (re-find #"OFFSET is not supported in custom filters" (ex-message e))))))))++(deftest editor-gate-offset-nested-in-aggregation-function-surfaces-error-end-to-end-test+ (testing (str "A window function embedded *inside* an aggregation function\n"+ "(`sum(offset(...))`) surfaces an :expression-editor-rejection agent-error -\n"+ "only the outer position (`offset(sum(...))`) is supported.")+ (with-mp-and-stubs!+ (fn []+ (let [e (editor-rejection+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "aggregation" [["sum" {}+ ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]]]+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]})]+ (is (some? e))+ (is (true? (:agent-error? (ex-data e))))+ (is (= :expression-editor-rejection (:error (ex-data e))))+ (is (= :aggregation (:mode (ex-data e)))))))))++(deftest editor-gate-offset-in-supported-positions-passes-end-to-end-test+ (testing "`offset` in its supported positions does NOT trigger the editor gate"+ (with-mp-and-stubs!+ (fn []+ (testing "at the top of an aggregation (`offset(sum(...), -1)`)"+ (is (some? (construct/execute-representations-query+ (query-data+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "aggregation" [["offset" {}+ ["sum" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]]]+ -1]]+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]})))))+ (testing "inside an `order-by`"+ (is (some? (construct/execute-representations-query+ (query-data+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "order-by" [["asc" {}+ ["offset" {} ["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]] -1]]]}]})))))))))++(deftest editor-gate-coerced-column-expression-passes-end-to-end-test+ (testing (str "The editor gate's unsuppressed type check does not false-positive on a\n"+ "coerced column: an expression that uses the column per its effective-type\n"+ "(`get-month` on a UNIX-seconds -> DateTime coercion) builds successfully.")+ (with-coerced-mp-and-stubs!+ (fn []+ (is (some? (construct/execute-representations-query+ (query-data+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "expressions" {"Created Month" ["get-month" {}+ ["field" {} ["Sample" "PUBLIC" "ORDERS" "CREATED_AT_UNIX"]]]}}]}))))))))++(deftest editor-gate-type-incompatible-expression-surfaces-error-end-to-end-test+ (testing (str "The editor gate type-checks expressions (which the canRun-mirror schema gate\n"+ "suppresses): a non-boolean clause in `filters:` surfaces an\n"+ ":expression-editor-rejection agent-error.")+ (with-mp-and-stubs!+ (fn []+ (let [e (editor-rejection+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "filters" [["field" {} ["Sample" "PUBLIC" "ORDERS" "TOTAL"]]]}]})]+ (is (some? e))+ (is (true? (:agent-error? (ex-data e))))+ (is (= :expression-editor-rejection (:error (ex-data e))))+ (is (= :filter (:mode (ex-data e)))))))))++(deftest query-not-runnable-explanation-gate-test+ (testing (str "The runnability backstop returns nil for a runnable resolved query and a\n"+ "non-nil Malli explanation for one whose breakout field ref is missing its\n"+ "base-type (the BOT-1442 schema invalidity that disables the editor's run/save).")+ (with-mp-and-stubs!+ (fn []+ (let [result (construct/execute-representations-query+ (query-data+ {"lib/type" "mbql/query"+ "database" "Sample"+ "stages" [{"lib/type" "mbql.stage/mbql"+ "source-table" ["Sample" "PUBLIC" "ORDERS"]+ "aggregation" [["count" {}]]+ "breakout" [["field" {} ["Sample" "PUBLIC" "ORDERS" "PRODUCT_ID"]]]}]}))+ good (get-in result [:structured-output :query])+ ;; Corrupt a breakout field ref into a string-named ref with no base-type - the+ ;; exact shape that resolves structurally yet makes Lib.canRun false.+ bad (assoc-in good [:stages 0 :breakout 0] [:field {} "PRODUCT_ID"])]+ (is (nil? (#'construct/query-not-runnable-explanation good)))+ (is (some? (#'construct/query-not-runnable-explanation bad))))))));;; ============================================================;;; Step-9 contract: expressions (custom columns)