Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Explorations prompt updates (#78228)

Kevin Kelleher692050c2dbcb
diff --git a/resources/metabot/prompts/system/explorations.selmer b/resources/metabot/prompts/system/explorations.selmer
index 451d06568dd..c79d263b425 100644
--- a/resources/metabot/prompts/system/explorations.selmer
+++ b/resources/metabot/prompts/system/explorations.selmer
@@ -31,7 +31,8 @@ A research plan is made of one or more **groups**. Each group is one of:
# Important Instructions
-- You cannot create metrics for the user. If the user's instance doesn't have any metrics, explain that they can't use Research mode without metrics. You can offer to help explore the user's data, suggest relevant metrics, and walk them through the process of creating metrics.
+- You cannot create metrics for the user. If the user's instance doesn't have any metrics, explain that they can't use Research mode without metrics. You can offer to help the user by suggesting relevant metrics they can create.
+ - Don't provide specific details on how to do things in Metabase like creating metrics. Your knowledge is based on your training data and may be outdated. Instead, create a link in your response to the docs like "https://www.metabase.com/search?query=create+a+metric". You can adjust the search query string at the end of the link as needed to point the user in the right direction.
- Don't ask for confirmation before authoring groups with `add_research_groups`. The user can easily adjust the plan, and having to provide confirmation adds unnecessary friction.
- Only use metric and dimension ids that appear in `get_research_candidates`. A group that references an unknown id will fail.
diff --git a/src/metabase/metabot/agent/profiles.clj b/src/metabase/metabot/agent/profiles.clj
index ddb882cb357..48ac1b9b48e 100644
--- a/src/metabase/metabot/agent/profiles.clj
+++ b/src/metabase/metabot/agent/profiles.clj
@@ -58,6 +58,10 @@
- :always-on-skills - Optional vector of skill ids (keywords) whose bodies are inlined into this
profile's system prompt instead of being loaded on demand via `load_skill`. Always-on is a
per-profile decision: the same skill can be inlined here and on-demand elsewhere.
+ - :skills? - Optional boolean (default true). When false, the profile opts out of the skills
+ system entirely: no skill catalog, no always-on inlining, and `load_skill` is not injected —
+ even if the profile's tools would otherwise match skills (e.g. `read_resource`). Use for
+ specialized profiles that carry their own tool guidance and must not invite `load_skill` calls.
- :terminal-tools - Optional set of tool-name strings whose **successful** call ends the agent
turn for this profile. Lets a `:required-tool-call?` profile stop as soon as it produces its
answer (e.g. `:sql` after `edit_sql_query`) instead of being forced to keep calling tools.
@@ -76,6 +80,7 @@
[:max-iterations :int]
[:tools [:vector :any]]
[:always-on-skills {:optional true} [:vector :keyword]]
+ [:skills? {:optional true} :boolean]
[:terminal-tools {:optional true} [:set :string]]
[:system-prompt-context {:optional true} [:fn ifn?]]]]
(let [tool-vars (:tools profile)
@@ -87,6 +92,9 @@
(let [dups (->> (frequencies tool-name-seq)
(filter (fn [[_ cnt]] (< 1 cnt))))]
(throw (ex-info "Duplicate tool names in profile" {:tool-names (map first dups)}))))
+ (when (and (false? (:skills? profile)) (seq (:always-on-skills profile)))
+ (throw (ex-info "Profile disables skills but lists :always-on-skills"
+ {:profile (:name profile) :always-on-skills (:always-on-skills profile)})))
(when-let [unknown (seq (remove skills/get-skill (:always-on-skills profile)))]
(throw (ex-info "Profile references unknown always-on skill ids"
{:profile (:name profile) :unknown-skill-ids unknown})))
@@ -226,6 +234,7 @@
:max-iterations 10
:temperature 0.3
:system-prompt-context #'tools.explorations/research-plan-system-context
+ :skills? false
:tools [#'tools/search-tool
#'tools/read-resource-tool
#'tools/get-research-candidates-tool
@@ -301,7 +310,8 @@
Returns a map of tool-name -> tool-var.
Takes the resolved profile (not an id) so callers that also need the profile's prompt resolve it once via
[[get-profile]] — its nlq availability redirect must be probed a single time, or the prompt and tools
- could disagree. When the profile exposes any skills, `load_skill` is injected for on-demand loading."
+ could disagree. When the profile exposes any skills, `load_skill` is injected for on-demand loading.
+ Profiles with `:skills? false` never get `load_skill` (see [[metabase.metabot.skills/build-skill-manifest]])."
[profile capabilities]
(when profile
(let [base (-> profile
diff --git a/src/metabase/metabot/skills.clj b/src/metabase/metabot/skills.clj
index 859efb96825..a673bb9c209 100644
--- a/src/metabase/metabot/skills.clj
+++ b/src/metabase/metabot/skills.clj
@@ -247,26 +247,33 @@
Whether a skill is *always-on* (body inlined in the system prompt) vs *on-demand* (advertised in the
catalog, loaded via `load_skill`) is decided **per profile**, by the profile's `:always-on-skills`
set — not by the skill itself — so the same skill can be inlined for one profile and on-demand for
- another (e.g. the SQL skills are always-on for `:sql` but on-demand for `:internal`)."
+ another (e.g. the SQL skills are always-on for `:sql` but on-demand for `:internal`).
+
+ Profiles with `:skills? false` opt out entirely: empty catalog and always-on, and an empty loadable
+ set (so `load_skill` is not injected and cannot resolve guessed ids)."
[profile active-tool-names capabilities]
- (let [cap-set (capabilities/capability-set capabilities)
- always-on-ids (set (:always-on-skills profile))
- always-on? (comp always-on-ids :id)
- visible (->> (skills-for-profile profile active-tool-names)
- (filter #(skill-visible? % cap-set))
- (sort-by (juxt (comp - :priority) (comp str :id))))
- ;; Always-on skill bodies are already inlined in the system prompt, so they are neither
- ;; advertised in the catalog nor loadable on demand — only the on-demand skills are.
- ;; Recording only the loadable ids also stops `load_skill` from re-fetching an always-on
- ;; skill (a wasted iteration the model would otherwise spend).
- loadable (remove always-on? visible)]
- (record-loadable-skill-ids! loadable)
- {:always-on (filter always-on? visible)
- :catalog (mapv (fn [s]
- {:id (name (:id s))
- :title (:title s)
- :description (:description s)})
- loadable)}))
+ (if (false? (:skills? profile))
+ (do
+ (record-loadable-skill-ids! [])
+ {:always-on [] :catalog []})
+ (let [cap-set (capabilities/capability-set capabilities)
+ always-on-ids (set (:always-on-skills profile))
+ always-on? (comp always-on-ids :id)
+ visible (->> (skills-for-profile profile active-tool-names)
+ (filter #(skill-visible? % cap-set))
+ (sort-by (juxt (comp - :priority) (comp str :id))))
+ ;; Always-on skill bodies are already inlined in the system prompt, so they are neither
+ ;; advertised in the catalog nor loadable on demand — only the on-demand skills are.
+ ;; Recording only the loadable ids also stops `load_skill` from re-fetching an always-on
+ ;; skill (a wasted iteration the model would otherwise spend).
+ loadable (remove always-on? visible)]
+ (record-loadable-skill-ids! loadable)
+ {:always-on (filter always-on? visible)
+ :catalog (mapv (fn [s]
+ {:id (name (:id s))
+ :title (:title s)
+ :description (:description s)})
+ loadable)})))
;;; Dialect preload
diff --git a/test/metabase/metabot/agent/profiles_test.clj b/test/metabase/metabot/agent/profiles_test.clj
index 000402896d6..ad22dc18da9 100644
--- a/test/metabase/metabot/agent/profiles_test.clj
+++ b/test/metabase/metabot/agent/profiles_test.clj
@@ -253,4 +253,18 @@
(testing "rejects :terminal-tools the profile does not expose"
(is (thrown-with-msg? clojure.lang.ExceptionInfo #"terminal tools it does not expose"
(#'profiles/register-profile!
- (assoc base :terminal-tools #{"nonexistent_tool"})))))))
+ (assoc base :terminal-tools #{"nonexistent_tool"})))))
+ (testing "rejects :skills? false combined with :always-on-skills"
+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"disables skills but lists"
+ (#'profiles/register-profile!
+ (assoc base :skills? false :always-on-skills [:read-resource])))))))
+
+(deftest explorations-profile-disables-skills-test
+ (binding [scope/*current-user-scope* api-scope/unrestricted]
+ (testing "explorations opts out of skills so read_resource does not inject load_skill"
+ (let [profile (profiles/get-profile :explorations)
+ tools (profiles/profile->tools profile [])]
+ (is (false? (:skills? profile)))
+ (is (contains? tools "read_resource")
+ "precondition: read_resource is active (would otherwise match a skill)")
+ (is (not (contains? tools "load_skill")))))))
diff --git a/test/metabase/metabot/api_test.clj b/test/metabase/metabot/api_test.clj
index 602462fa926..0e283e4c8c5 100644
--- a/test/metabase/metabot/api_test.clj
+++ b/test/metabase/metabot/api_test.clj
@@ -252,6 +252,7 @@
(with-redefs [llm.settings/llm-openrouter-api-key (constantly "fake-key")
llm.settings/llm-openrouter-api-base-url (constantly llm-url)
scope/resolve-user-permissions (constantly scope/all-yes-permissions)
+ conversation-title/ensure-title! (constantly {:status :missing})
;; The fake LLM server doesn't gzip, but clj-http wraps with
;; GZIPInputStream by default. Closing mid-stream causes ZLIB errors.
http/post (fn [url opts]
diff --git a/test/metabase/metabot/skills_test.clj b/test/metabase/metabot/skills_test.clj
index 29e1af3935a..d98f93cf790 100644
--- a/test/metabase/metabot/skills_test.clj
+++ b/test/metabase/metabot/skills_test.clj
@@ -122,6 +122,23 @@
(is (empty? (:always-on on-demand)))
(is (some #{"read-resource"} (map :id (:catalog on-demand))))))))
+(deftest ^:parallel build-skill-manifest-skills?-false-test
+ (testing "`:skills? false` empties the manifest even when tools would match skills"
+ (binding [scope/*current-loadable-skill-ids* (atom #{:read-resource})]
+ (let [manifest (skills/build-skill-manifest
+ {:name :explorations :skills? false}
+ ["read_resource" "search"]
+ [])]
+ (is (empty? (:catalog manifest)))
+ (is (empty? (:always-on manifest)))
+ (is (empty? @scope/*current-loadable-skill-ids*)
+ "loadable set is cleared so guessed load_skill ids cannot resolve"))))
+ (testing "absent or true `:skills?` leaves normal matching intact"
+ (doseq [profile [{:name :internal}
+ {:name :internal :skills? true}]]
+ (is (some #{"read-resource"}
+ (map :id (:catalog (skills/build-skill-manifest profile ["read_resource"] []))))))))
+
(deftest ^:parallel always-on-skill-body-is-in-system-prompt-test
;; A "marker" line lifted verbatim from the read-resource skill body. If the skill is inlined,
;; this text appears in the rendered system prompt; if it is on-demand, only its one-line catalog