Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Let Metabot add model-based charts to documents (#82990)

Andrei Simionescuffa7eecd6c4c
diff --git a/resources/metabot/prompts/system/document-generate-content.selmer b/resources/metabot/prompts/system/document-generate-content.selmer
index d1bc3002db7..0965071b27d 100644
--- a/resources/metabot/prompts/system/document-generate-content.selmer
+++ b/resources/metabot/prompts/system/document-generate-content.selmer
@@ -14,6 +14,9 @@ IF using an existing model, identify the relevant model using the `list_availabl
THEN use the `document_construct_model_chart` tool (IF a relevant model is found) to generate the chart.
OR use the `document_schema_collect` tool (IF a relevant model is NOT found) as a first step to generating a SQL chart.
+`document_construct_model_chart` queries the model as its `source-card`: load the `construct-notebook-query-core`
+and `construct-notebook-query-advanced` skills before writing its `query`.
+
## IF using custom SQL:
IF writing custom SQL, FIRST use the `document_schema_collect` to get the schema information for the
database you will be querying against
@@ -44,3 +47,5 @@ you to `document_construct_sql_chart`.
- For date columns, extract year with `YEAR(column_name)` and month with `MONTH(column_name)` rather than EXTRACT().
{% include "metabot/prompts/shared/prompt_snippets/field-references.selmer" %}
+
+{% include "metabot/prompts/shared/skills.selmer" %}
diff --git a/resources/metabot/skills/construct-notebook-query-advanced.md b/resources/metabot/skills/construct-notebook-query-advanced.md
index ec2d0b80436..1270d3d884d 100644
--- a/resources/metabot/skills/construct-notebook-query-advanced.md
+++ b/resources/metabot/skills/construct-notebook-query-advanced.md
@@ -2,7 +2,7 @@
id: construct-notebook-query-advanced
title: Construct notebook query — advanced
description: Joins (explicit and implicit), multi-stage queries, querying saved questions/models with source-card, metrics, measures, segments, expressions, and aggregation references in construct_notebook_query — load when a query needs any of these.
-tools: [construct_notebook_query]
+tools: [construct_notebook_query, document_construct_model_chart]
priority: 50
---
# Construct Query Reference — Advanced
diff --git a/resources/metabot/skills/construct-notebook-query-core.md b/resources/metabot/skills/construct-notebook-query-core.md
index 6798d846b08..7d92e70ae9e 100644
--- a/resources/metabot/skills/construct-notebook-query-core.md
+++ b/resources/metabot/skills/construct-notebook-query-core.md
@@ -2,7 +2,7 @@
id: construct-notebook-query-core
title: Construct notebook query — core
description: Building or editing a notebook (MBQL) query with construct_notebook_query — load before your first query so you get the clause shape, field references, and the rules/anti-patterns right.
-tools: [construct_notebook_query]
+tools: [construct_notebook_query, document_construct_model_chart]
priority: 60
---
# Construct Query Reference — Core
@@ -19,6 +19,8 @@ Return:
> The Slackbot variant of this tool has a different contract: `reasoning` is **required**, `title` is optional and there's no `description`; it uses `display` (a Slack-specific visualization-type enum) instead of `visualization`. See the Slackbot system prompt for its exact argument set.
+> `document_construct_model_chart` takes the same `query`, with `name` instead of `title` and a required `viz_settings` instead of `visualization`.
+
## Minimal example — count of orders by month
```json
diff --git a/resources/metabot/skills/construct-notebook-query-operators.md b/resources/metabot/skills/construct-notebook-query-operators.md
index a9a8408ec34..92657ef32e0 100644
--- a/resources/metabot/skills/construct-notebook-query-operators.md
+++ b/resources/metabot/skills/construct-notebook-query-operators.md
@@ -2,7 +2,7 @@
id: construct-notebook-query-operators
title: Construct notebook query — operator catalogs
description: The full catalog of aggregation, filter, expression, and temporal-unit operators for construct_notebook_query — load when you need the exact name/arity of a specific operator or the list of temporal units.
-tools: [construct_notebook_query]
+tools: [construct_notebook_query, document_construct_model_chart]
priority: 40
---
# Construct Query Reference — Operator catalogs
diff --git a/src/metabase/metabot/tools/construct.clj b/src/metabase/metabot/tools/construct.clj
index 950bb5532ef..b7309d58005 100644
--- a/src/metabase/metabot/tools/construct.clj
+++ b/src/metabase/metabot/tools/construct.clj
@@ -97,6 +97,13 @@
::mr/deliberately-open true
:description "An LLM-authored MBQL 5 query, structurally unvalidated here; real validation happens at the entry-point boundaries."}])
+(def LLMQueryArgument
+ "Schema for the `:query` argument of `construct_notebook_query` and the tools that reuse it.
+
+ Validation stays a fully open, property-less `:map` (the repair layer fixes LLM shortcuts); the `:json-schema`
+ override only changes what the LLM sees. See [[construct-notebook-query-json-schema]]."
+ (mu/with LLMExternalQuery {:json-schema construct-notebook-query-json-schema}))
+
(def ^:private construct-notebook-query-args-schema
"Args schema for `construct_notebook_query`.
@@ -113,9 +120,7 @@
omits `:source_entity` and `:referenced_entities` — the query body is self-describing."
[:map {:closed true}
[:reasoning {:optional true} :string]
- ;; Validation stays a fully open, property-less `:map` (the repair layer fixes LLM shortcuts); the
- ;; `:json-schema` override only changes what the LLM sees. See [[construct-notebook-query-json-schema]].
- [:query (mu/with LLMExternalQuery {:json-schema construct-notebook-query-json-schema})]
+ [:query LLMQueryArgument]
[:visualization {:optional true} construct-visualization-schema]
[:title :string]
[:description :string]])
diff --git a/src/metabase/metabot/tools/document.clj b/src/metabase/metabot/tools/document.clj
index f9144339caa..a4c1511fca6 100644
--- a/src/metabase/metabot/tools/document.clj
+++ b/src/metabase/metabot/tools/document.clj
@@ -176,16 +176,10 @@
(metabot.tools.u/handle-agent-error e))))
(def ^:private model-chart-schema
- "Schema for `document_construct_model_chart`. Mirrors `construct_notebook_query`'s
- representations format: `:query` is a YAML string in MBQL 5 representations format.
-
- Per `repr-plan.md` step 13, `:source_entity` is no longer part of the contract — the YAML
- query is self-describing (carries `database:` at the top level and full portable FK paths
- everywhere else)."
[:map {:closed true}
[:name :string]
[:description :string]
- [:query :string]
+ [:query construct-tools/LLMQueryArgument]
[:viz_settings [:map {:closed true}
[:chart_type chart-type-enum]]]])
@@ -198,6 +192,8 @@
(let [chart-type (get viz_settings :chart_type)
result (construct-tools/construct-notebook-query-tool
{:query query
+ :title name
+ :description description
:visualization {:chart_type chart-type}})
structured (or (:structured-output result) (:structured_output result))
query-id (:query-id structured)
diff --git a/test/metabase/metabot/agent/prompts_test.clj b/test/metabase/metabot/agent/prompts_test.clj
index ae6a446cf44..81ff091669b 100644
--- a/test/metabase/metabot/agent/prompts_test.clj
+++ b/test/metabase/metabot/agent/prompts_test.clj
@@ -3,7 +3,11 @@
[clojure.java.io :as io]
[clojure.string :as str]
[clojure.test :refer :all]
- [metabase.metabot.agent.prompts :as prompts]))
+ [metabase.api-scope.core :as api-scope]
+ [metabase.metabot.agent.profiles :as profiles]
+ [metabase.metabot.agent.prompts :as prompts]
+ [metabase.metabot.scope :as scope]
+ [metabase.metabot.skills :as skills]))
(deftest ^:parallel load-system-prompt-template-test
(testing "loads internal.selmer template"
@@ -177,6 +181,16 @@
(is (not (str/includes? content "{% include"))
"unresolved {% include %} tags mean rendering failed and the raw template was returned"))))
+(deftest ^:parallel build-system-message-content-document-skills-test
+ (testing "the document prompt lists the notebook query skills its model chart tool needs"
+ (binding [scope/*current-user-scope* api-scope/unrestricted]
+ (let [profile (profiles/get-profile :document-generate-content)
+ content (prompts/build-system-message-content profile {} (profiles/profile->tools profile []) [])]
+ (doseq [skill-id [:construct-notebook-query-core
+ :construct-notebook-query-advanced
+ :construct-notebook-query-operators]]
+ (is (str/includes? content (:description (skills/get-skill skill-id)))))))))
+
(deftest ^:parallel build-system-message-content-test-9
(testing "renders sql querying template with literal model syntax"
(let [profile {:prompt-template "sql-querying-only.selmer"}
diff --git a/test/metabase/metabot/api/document_test.clj b/test/metabase/metabot/api/document_test.clj
index ba929f2b992..97953c9d71f 100644
--- a/test/metabase/metabot/api/document_test.clj
+++ b/test/metabase/metabot/api/document_test.clj
@@ -1,7 +1,10 @@
(ns metabase.metabot.api.document-test
(:require
+ [clojure.string :as str]
[clojure.test :refer :all]
[metabase.analytics.snowplow-test :as snowplow-test]
+ [metabase.lib.core :as lib]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.llm.test-util :as llm.tu]
[metabase.metabot.scope :as scope]
[metabase.metabot.self.openrouter :as openrouter]
@@ -118,6 +121,51 @@
:draft_card {:name "Orders by day"}}
response))))))))
+(deftest generate-content-model-chart-round-trip-test
+ (testing "the model loads the query skills, then charts a model by its column names"
+ (let [mp (mt/metadata-provider)]
+ (mt/with-temp [:model/Card model {:type :model
+ :dataset_query (lib/query mp (lib.metadata/table mp (mt/id :orders)))}]
+ (mt/with-temporary-setting-values [llm-providers llm.tu/default-connections
+ llm-metabot-provider test-provider]
+ (let [requests (atom [])
+ responses [[{:type :tool-input
+ :id "t1"
+ :function "load_skill"
+ :arguments {:ids ["construct-notebook-query-core" "construct-notebook-query-advanced"]}}]
+ [{:type :tool-input
+ :id "t2"
+ :function "document_construct_model_chart"
+ :arguments {:name "Orders by month"
+ :description "Count of orders per month."
+ :query {"lib/type" "mbql/query"
+ "stages" [{"lib/type" "mbql.stage/mbql"
+ "source-card" (:entity_id model)
+ "aggregation" [["count" {}]]
+ "breakout" [["field" {"temporal-unit" "month"}
+ "CREATED_AT"]]}]}
+ :viz_settings {:chart_type "line"}}}]]]
+ (mt/with-dynamic-fn-redefs [openrouter/openrouter
+ (fn [request]
+ (let [n (count (swap! requests conj request))]
+ (mut/mock-llm-response
+ (get responses (dec n) [{:type :text :text "Chart created"}]))))]
+ (let [response (mt/user-http-request :crowberto
+ :post 200 "metabot/document/generate-content"
+ {:instructions "Chart orders by month"})
+ [first-request second-request] @requests]
+ (is (some #{"load_skill"} (map :tool-name (:tools first-request))))
+ (is (some #(and (string? %)
+ (str/includes? % "<skill id=\"construct-notebook-query-advanced\">"))
+ (tree-seq coll? seq (:input second-request)))
+ "the skill body reaches the model before it writes the query")
+ (is (=? {:error nil
+ :draft_card {:name "Orders by month"
+ :display "line"
+ :database_id (mt/id)
+ :dataset_query {:stages [{:source-card (:id model)}]}}}
+ response))))))))))
+
(deftest generate-content-snowplow-test
(mt/with-temporary-setting-values [llm-providers llm.tu/default-connections
llm-metabot-provider test-provider]
diff --git a/test/metabase/metabot/skills_test.clj b/test/metabase/metabot/skills_test.clj
index d98f93cf790..3e2f5687988 100644
--- a/test/metabase/metabot/skills_test.clj
+++ b/test/metabase/metabot/skills_test.clj
@@ -60,7 +60,7 @@
(is (= "construct-notebook-query-core" (name (:id s))))
(is (string? (:title s)))
(is (str/includes? (:description s) "construct_notebook_query"))
- (is (= ["construct_notebook_query"] (:tools s)))
+ (is (= ["construct_notebook_query" "document_construct_model_chart"] (:tools s)))
(is (str/includes? (:body s) "Universal clause shape"))
;; the YAML frontmatter delimiter must not leak into the body
(is (not (str/starts-with? (:body s) "---"))))))
diff --git a/test/metabase/metabot/tools/document_test.clj b/test/metabase/metabot/tools/document_test.clj
index f1e9f42476a..d282fc377a6 100644
--- a/test/metabase/metabot/tools/document_test.clj
+++ b/test/metabase/metabot/tools/document_test.clj
@@ -1,6 +1,8 @@
(ns metabase.metabot.tools.document-test
(:require
[clojure.test :refer :all]
+ [metabase.lib.core :as lib]
+ [metabase.lib.metadata :as lib.metadata]
[metabase.metabot.table-utils :as table-utils]
[metabase.metabot.tools.construct :as construct-tools]
[metabase.metabot.tools.document :as document-tools]
@@ -118,31 +120,32 @@
(is (re-find #"missing_table" (:output result)))))))
(deftest document-construct-model-chart-tool-test
- (testing "builds chart draft payload from model query"
- (mt/with-dynamic-fn-redefs [construct-tools/construct-notebook-query-tool
- (fn [_]
- {:structured-output {:query-id "3"
- :query {:database 1
- :type "query"}}})]
- ;; `construct-notebook-query-tool` is stubbed above, so the YAML string is an opaque
- ;; placeholder — it only needs to be a string.
- (let [result (document-tools/document-construct-model-chart-tool
- {:name "Test Name"
- :description "Test Desc"
- :query ""
- :viz_settings {:chart_type "bar"}})
- structured (:structured-output result)]
- (is (some? structured)
- "structured-output present -> a successful, terminal-eligible call")
- (is (= "document_construct_model_chart" (:tool structured)))
- (is (= "Test Name" (:name structured)))
- (is (= "Test Desc" (:description structured)))
- (is (= "bar" (:display structured)))
- (is (= "bar" (:chart_type structured)))
- (is (= :chart-draft (:result-type structured)))
- (is (= {:database 1
- :type "query"}
- (:dataset_query structured)))))))
+ (testing "builds chart draft payload from a query on a model"
+ (let [mp (mt/metadata-provider)]
+ (mt/with-current-user (mt/user->id :crowberto)
+ (mt/with-temp [:model/Card model {:type :model
+ :dataset_query (lib/query mp (lib.metadata/table mp (mt/id :orders)))}]
+ (let [result (document-tools/document-construct-model-chart-tool
+ {:name "Test Name"
+ :description "Test Desc"
+ :query {:lib/type "mbql/query"
+ :stages [{:lib/type "mbql.stage/mbql"
+ :source-card (:entity_id model)
+ :aggregation [["count" {}]]
+ :breakout [["field" {:temporal-unit "month"} "CREATED_AT"]]}]}
+ :viz_settings {:chart_type "bar"}})
+ structured (:structured-output result)]
+ (is (some? structured)
+ "structured-output present -> a successful, terminal-eligible call")
+ (is (= "document_construct_model_chart" (:tool structured)))
+ (is (= "Test Name" (:name structured)))
+ (is (= "Test Desc" (:description structured)))
+ (is (= "bar" (:display structured)))
+ (is (= "bar" (:chart_type structured)))
+ (is (= :chart-draft (:result-type structured)))
+ (is (=? {:database (mt/id)
+ :stages [{:source-card (:id model)}]}
+ (:dataset_query structured)))))))))
(deftest document-construct-model-chart-new-chart-types-test
(testing "the tool schema accepts newly added chart types"
@@ -155,7 +158,7 @@
(let [result (document-tools/document-construct-model-chart-tool
{:name "Test Name"
:description "Test Desc"
- :query ""
+ :query {:lib/type "mbql/query" :stages []}
:viz_settings {:chart_type chart-type}})
structured (:structured-output result)]
(is (= chart-type (:display structured)))
diff --git a/test/metabase/metabot/tools_test.clj b/test/metabase/metabot/tools_test.clj
index fe791bbf8a5..f51934f75da 100644
--- a/test/metabase/metabot/tools_test.clj
+++ b/test/metabase/metabot/tools_test.clj
@@ -95,6 +95,8 @@
(is (contains? tools "list_available_fields"))
(is (contains? tools "get_field_values"))
(is (contains? tools "document_construct_model_chart"))
+ (is (contains? tools "load_skill")
+ "the model chart tool's query skills make load_skill available")
(testing "both SQL tools need the SQL capability, like the tools they delegate to"
(is (not (contains? tools "document_construct_sql_chart")))
(is (contains? sql-tools "document_construct_sql_chart")))