The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Add a Metabot query execution toggle, on by default (#84416)
Chris Truter7f4e76a76d00
diff --git a/docs/ai/privacy.md b/docs/ai/privacy.mdindex 9c8f4cf843b..7c032634d4b 100644--- a/docs/ai/privacy.md+++ b/docs/ai/privacy.md@@ -9,22 +9,51 @@ AI in Metabase is optional.If you do use AI (which you can set up in [AI settings](./settings.md)), we need to send some data outside of your Metabase. This page covers what gets sent, and to whom.-## Data sent to the Metabase AI service+## Data sent to the Metabase AI Service-When using the Metabase AI service, we don't send the _results_ of your queries to a third-party AI provider, and we don't look at them ourselves.+When using the Metabase AI service, the data we send includes:-What we send:--- Your prompts.+- Your prompts and the earlier messages in the conversation.- Metadata (like table and field names), so Metabot knows what to query.- A sampling of field values from your database. For example, if you ask Metabot to "Filter everyone from Wisconsin," it might check the values in the state field to see how the data is stored (like "WI" vs "Wisconsin"). See [syncs](../databases/sync-scan.md).-- For chart analysis, AI won't see the raw timeseries data, but it does see some metrics derived from that data (like slopes, series correlations, etc). AI may also see some categorical values. For example, if a question breaks out orders over time by a product category, the AI could see the category values ("Gadgets", "Widgets", etc).+- For chart analysis, metrics derived from the chart's data (like slopes and series correlations), and some categorical values. For example, if a question breaks out orders over time by a product category, the AI could see the category values ("Gadgets", "Widgets", etc).+- Query results, if query execution is turned on for Metabot.++How Metabase the company processes this data is covered by your agreement with Metabase. Only Metabase Cloud customers have access to the Metabase AI Service.++## Metabot sends query results to the AI provider when it runs queries++When query execution is on, Metabot can run queries and read their results, so it can answer with actual values from your data. Metabase sends those results to the AI provider.++Metabot can run queries on your behalf, like the queries it builds or the question you're looking at. These can include SQL queries.++For each query Metabot runs, Metabase sends part of the result to the AI provider, like the column names and a limited number of rows. Metabot can run many queries in one conversation.++If a query fails, Metabase also sends your database's error message. Error messages can include values from your data.++### Metabot reads results in line with your data permissions++Metabot runs queries as you, with your data permissions, including [row and column security](../permissions/row-and-column-security.md). Metabot can only read query results that you have permission to see.++### Query results are stored with the conversation++Metabase stores the query results that Metabot reads as part of the conversation, in your Metabase application database.++On Pro and Enterprise plans, admins can review stored conversations, including query results, in [AI usage auditing](./usage-auditing.md).++By default, your Metabase instance keeps stored conversations for a limited time. On self-hosted Metabase, you can change how long with [`MB_AI_USAGE_MAX_RETENTION_DAYS`](../configuring-metabase/environment-variables.md#mb_ai_usage_max_retention_days).++### Turn query execution on or off++Query execution is on by default. Admins can turn it off in [AI settings](./settings.md#query-execution).++When query execution is off, Metabot doesn't run queries to read their results, but it can still build queries and charts for you. Other features on this page, like chart analysis, can still send values from your data.-We, Metabase the company, also collect some of this metadata to gauge usage and improve the AI integration.+Metabot in Slack doesn't read query results, regardless of this setting.## Data sent to your own AI provider-If you're using your own API key, everything listed above is sent to your selected AI provider. Review your provider's data handling and privacy policies.+If you're using your own API key, everything listed above is sent to your selected AI provider, including query results when Metabot can run queries. Your provider's data handling, retention, and privacy policies apply to that data. Query execution is on by default, so review them before people start using Metabot, or turn query execution off until you have.## Data sent to the MCP client@@ -32,11 +61,13 @@ When using the [MCP server](./mcp.md), query _results_ are sent to the connected## Data sent when submitting feedback-If you [submit feedback](./metabot.md#giving-feedback-on-metabot-responses) on a Metabot response, the context for that conversation, including metadata and your prompts, might be sent to Metabase the company, and may contain sensitive data.+If you [submit feedback](./metabot.md#giving-feedback-on-metabot-responses) on a Metabot response, the context for that conversation, including metadata, your prompts, and any query results that Metabot read, might be sent to Metabase the company, and may contain sensitive data.## Further reading- [AI settings](./settings.md)- [Using Metabot](./metabot.md)+- [AI usage auditing](./usage-auditing.md)- [MCP server](./mcp.md)+- [Row and column security](../permissions/row-and-column-security.md)- [Privacy and GDPR](../installation-and-operation/privacy.md)diff --git a/docs/ai/settings.md b/docs/ai/settings.mdindex 69389574e2d..49dcbfc4b07 100644--- a/docs/ai/settings.md+++ b/docs/ai/settings.md@@ -136,6 +136,18 @@ When enabled, Metabot is available to help people create questions, analyze dataToggling off Metabot only turns off in-app Metabot features. People can still use the [MCP server](./mcp.md) and [Agent API](./agent-api.md) if those are enabled.+### Query execution++_Internal tab._++Toggle query execution on or off. It's on by default, and the toggle is hidden while Metabot is off. With query execution on, Metabot can answer questions with values from your data. When it's off, Metabot can still build queries and charts, but it can't read their results.++Metabot runs queries with the permissions of the person asking, and sends the result rows to your AI provider. See [AI privacy](./privacy.md#metabot-sends-query-results-to-the-ai-provider-when-it-runs-queries).++This setting doesn't apply to Metabot in Slack or in embedded apps.++If [`MB_METABOT_QUERY_EXECUTION_ENABLED`](../configuring-metabase/environment-variables.md#mb_metabot_query_execution_enabled) is set, the toggle shows its value and is locked.+### Enable Embedded Metabot_Embedded tab._diff --git a/docs/configuring-metabase/config-template.md b/docs/configuring-metabase/config-template.mdindex 98ad9069eed..7e2170f305b 100644--- a/docs/configuring-metabase/config-template.md+++ b/docs/configuring-metabase/config-template.md@@ -268,7 +268,7 @@ config:metabot-limit-unit: tokensmetabot-name: Metabotmetabot-nlq-system-prompt: ''- metabot-query-execution-enabled: false+ metabot-query-execution-enabled: truemetabot-quota-reached-message: You have reached your AI usage limit for the current period. Please contact your administrator.metabot-recent-views-enabled: truemetabot-show-illustrations: truediff --git a/docs/configuring-metabase/environment-variables.md b/docs/configuring-metabase/environment-variables.mdindex a3c71f72885..f1a41acfd90 100644--- a/docs/configuring-metabase/environment-variables.md+++ b/docs/configuring-metabase/environment-variables.md@@ -1939,7 +1939,7 @@ The display name for Metabot, shown throughout the Metabase UI.### `MB_METABOT_QUERY_EXECUTION_ENABLED`- Type: boolean-- Default: `false`+- Default: `true`- [Configuration file name](./config-file.md): `metabot-query-execution-enabled`Whether Metabot can run queries and read their results. Result rows are sent to the LLM provider.diff --git a/frontend/src/metabase-types/api/mocks/settings.ts b/frontend/src/metabase-types/api/mocks/settings.tsindex 3b9b32201b8..f4b43838752 100644--- a/frontend/src/metabase-types/api/mocks/settings.ts+++ b/frontend/src/metabase-types/api/mocks/settings.ts@@ -331,6 +331,7 @@ export const createMockSettings = ("loading-message": "doing-science","mcp-enabled?": true,"map-tile-server-url": "https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png",+ "metabot-query-execution-enabled?": true,"native-query-autocomplete-match-style": "substring","openai-api-key": null,"openai-organization": null,diff --git a/frontend/src/metabase-types/api/settings.ts b/frontend/src/metabase-types/api/settings.tsindex 941c62dfb70..d61b44b1540 100644--- a/frontend/src/metabase-types/api/settings.ts+++ b/frontend/src/metabase-types/api/settings.ts@@ -511,6 +511,7 @@ interface AdminSettings {"analytics-pii-retention-enabled"?: boolean;"custom-geojson-enabled": boolean;"encryption-enabled": boolean;+ "metabot-query-execution-enabled?": boolean;"deprecation-notice-version"?: string;"disable-cors-on-localhost": boolean;"embedding-secret-key"?: string;diff --git a/frontend/src/metabase/admin/ai/MetabotSettingsPanel.tsx b/frontend/src/metabase/admin/ai/MetabotSettingsPanel.tsxindex 3505c5deb11..484c6be4714 100644--- a/frontend/src/metabase/admin/ai/MetabotSettingsPanel.tsx+++ b/frontend/src/metabase/admin/ai/MetabotSettingsPanel.tsx@@ -1,5 +1,5 @@import { useDisclosure } from "@mantine/hooks";-import type { ReactNode } from "react";+import { type ReactNode, useId } from "react";import { match } from "ts-pattern";import { c, t } from "ttag";@@ -7,12 +7,13 @@ import { skipToken, useGetCollectionQuery } from "metabase/api";import { canonicalCollectionId } from "metabase/common/collections/utils";import { LoadingAndErrorWrapper } from "metabase/common/components/LoadingAndErrorWrapper";import { CollectionPickerModal } from "metabase/common/components/Pickers/CollectionPicker";+import { SetByEnvVar } from "metabase/common/components/SetByEnvVar";import { useToast } from "metabase/common/hooks";import { useGetIcon } from "metabase/hooks/use-icon";import { useUpdateMetabotMutation } from "metabase/metabot";import { FIXED_METABOT_ENTITY_IDS } from "metabase/metabot/constants";import { PLUGIN_MODERATION } from "metabase/plugins";-import { useAdminSetting } from "metabase/settings";+import { useAdminSetting, useSettingSwitch } from "metabase/settings";import { SettingHeader } from "metabase/settings-components";import {Box,@@ -55,6 +56,8 @@ export function MetabotSettingsPanel({ metabot }: { metabot: MetabotInfo }) {/><MetabotSettingsBody settingKey={enabledSettingKey}>+ {!isEmbedMetabot && <MetabotQueryExecutionToggle />}+<PLUGIN_MODERATION.MetabotVerifiedContentConfigurationPanemetabot={metabot}/>@@ -74,6 +77,57 @@ export function MetabotSettingsPanel({ metabot }: { metabot: MetabotInfo }) {);}+function MetabotQueryExecutionToggle() {+ const { checked, envName, isBusy, onChange } = useSettingSwitch(+ "metabot-query-execution-enabled?",+ );+ const label = t`Let Metabot run queries and read their results`;+ const isSetByEnv = envName != null;+ const descriptionId = useId();+ const envMessageId = useId();++ return (+ <Box>+ <SettingHeader+ id="metabot-query-execution"+ title={t`Query execution`}+ description={+ <span id={descriptionId}>+ {t`Metabot can run queries and read their results to answer questions, using the permissions of the person asking. The result rows are sent to your AI provider. This doesn't apply to Metabot in Slack or in embedded apps.`}+ </span>+ }+ />+ <Flex align="center" gap="lg" mt="lg">+ <Switch+ id="metabot-query-execution"+ aria-label={label}+ aria-describedby={+ isSetByEnv ? `${descriptionId} ${envMessageId}` : descriptionId+ }+ checked={checked}+ disabled={isSetByEnv}+ // aria-disabled, not disabled, so focus stays on the switch during a write+ aria-disabled={isBusy || undefined}+ onChange={(e) => {+ if (!isBusy) {+ onChange(e.target.checked);+ }+ }}+ w="auto"+ />+ <Text c={checked ? "text-primary" : "text-secondary"} fw="500">+ {label}+ </Text>+ </Flex>+ {isSetByEnv && (+ <Box id={envMessageId}>+ <SetByEnvVar varName={envName} />+ </Box>+ )}+ </Box>+ );+}+function MetabotEnabledToggle({settingKey,isEmbedMetabot = false,diff --git a/frontend/src/metabase/admin/ai/MetabotSettingsPanel.unit.spec.tsx b/frontend/src/metabase/admin/ai/MetabotSettingsPanel.unit.spec.tsxindex dd2b89a161a..35196c92b49 100644--- a/frontend/src/metabase/admin/ai/MetabotSettingsPanel.unit.spec.tsx+++ b/frontend/src/metabase/admin/ai/MetabotSettingsPanel.unit.spec.tsx@@ -30,10 +30,16 @@ import { waitForRequest } from "__support__/utils";import { FIXED_METABOT_ENTITY_IDS } from "metabase/metabot/constants";import { TEST_METABOT_IDS } from "metabase/metabot/tests/utils";import { reinitialize } from "metabase/plugins";-import type { MetabotId, MetabotInfo, RecentItem } from "metabase-types/api";+import type {+ MetabotId,+ MetabotInfo,+ RecentItem,+ SettingDefinition,+} from "metabase-types/api";import {createMockCollection,createMockMetabotInfo,+ createMockSettingDefinition,createMockSettings,createMockTokenFeatures,} from "metabase-types/api/mocks";@@ -92,9 +98,10 @@ const setup = async (metabots: MetabotInfo[] = defaultMetabots,seedCollections = defaultSeedCollections,settings = createMockSettings({ "llm-metabot-configured?": true }),+ settingDefinitions: SettingDefinition[] = [],) => {setupPropertiesEndpoints(settings);- setupSettingsEndpoints([]);+ setupSettingsEndpoints(settingDefinitions);setupMetabotsEndpoints(metabots);setupCollectionByIdEndpoint({collections: seedCollections.map((c: any) => ({ id: c.model_id, ...c })),@@ -136,6 +143,43 @@ const setup = async (const enabledToggle = () => screen.findByTestId("metabot-enabled-toggle");+const QUERY_EXECUTION_LABEL = "Let Metabot run queries and read their results";++const queryExecutionSwitch = () =>+ screen.findByRole("switch", { name: QUERY_EXECUTION_LABEL });++const setupQueryExecution = ({+ enabled = true,+ envName,+ metabotEnabled = true,+}: {+ enabled?: boolean;+ envName?: string;+ metabotEnabled?: boolean;+} = {}) =>+ setup(+ TEST_METABOT_IDS.INTERNAL,+ defaultMetabots,+ defaultSeedCollections,+ createMockSettings({+ "llm-metabot-configured?": true,+ "metabot-enabled?": metabotEnabled,+ "metabot-query-execution-enabled?": enabled,+ }),+ [+ createMockSettingDefinition({+ key: "metabot-enabled?",+ value: metabotEnabled,+ }),+ createMockSettingDefinition({+ key: "metabot-query-execution-enabled?",+ value: envName ? undefined : enabled,+ is_env_setting: envName != null,+ env_name: envName,+ }),+ ],+ );+const getLastSettingUpdateCall = (settingKey: string) =>fetchMock.callHistory.lastCall(`path:/api/setting/${encodeURIComponent(settingKey)}`,@@ -328,3 +372,71 @@ describe("MetabotSettingsPanel", () => {expect(putRequests[0].body).toEqual({ use_verified_content: true });});});++describe("MetabotSettingsPanel query execution", () => {+ mockGetBoundingClientRect();++ afterEach(() => {+ reinitialize();+ });++ it.each([true, false])("shows the setting's value: %s", async (enabled) => {+ await setupQueryExecution({ enabled });++ expect(+ await screen.findByRole("switch", {+ name: QUERY_EXECUTION_LABEL,+ checked: enabled,+ }),+ ).toBeInTheDocument();+ expect(await queryExecutionSwitch()).toHaveAccessibleDescription(+ /The result rows are sent to your AI provider/,+ );+ });++ it("persists turning query execution off", async () => {+ await setupQueryExecution();++ await userEvent.click(await queryExecutionSwitch());+ await waitForRequest(() =>+ getLastSettingUpdateCall("metabot-query-execution-enabled?"),+ );+ const call = getLastSettingUpdateCall("metabot-query-execution-enabled?");+ expect(call?.options?.body).toBe(JSON.stringify({ value: false }));+ });++ it("locks query execution to the value an env var sets", async () => {+ await setupQueryExecution({+ envName: "MB_METABOT_QUERY_EXECUTION_ENABLED",+ });++ const toggle = await queryExecutionSwitch();+ expect(+ await screen.findByTestId("setting-env-var-message"),+ ).toHaveTextContent("MB_METABOT_QUERY_EXECUTION_ENABLED");+ expect(toggle).toBeChecked();+ expect(toggle).toBeDisabled();+ expect(toggle).toHaveAccessibleDescription(+ /set by the MB_METABOT_QUERY_EXECUTION_ENABLED environment variable/,+ );+ });++ it("hides query execution while Metabot is off", async () => {+ await setupQueryExecution({ metabotEnabled: false });++ expect(await screen.findByText("Metabot is disabled")).toBeInTheDocument();+ expect(+ screen.queryByRole("switch", { name: QUERY_EXECUTION_LABEL }),+ ).not.toBeInTheDocument();+ });++ it("doesn't show query execution for embedded Metabot", async () => {+ setupEmbeddingPlugin();+ await setup(TEST_METABOT_IDS.EMBEDDED);++ expect(await screen.findByText("Collection Two")).toBeInTheDocument();+ expect(+ screen.queryByRole("switch", { name: QUERY_EXECUTION_LABEL }),+ ).not.toBeInTheDocument();+ });+});diff --git a/src/metabase/metabot/agent/core.clj b/src/metabase/metabot/agent/core.cljindex 8ba07fa9960..f85947d9113 100644--- a/src/metabase/metabot/agent/core.clj+++ b/src/metabase/metabot/agent/core.clj@@ -16,6 +16,7 @@[metabase.metabot.agent.profiles :as profiles][metabase.metabot.agent.streaming :as streaming][metabase.metabot.capabilities :as capabilities]+ [metabase.metabot.config :as metabot.config][metabase.metabot.context :as metabot.context][metabase.metabot.db :as metabot.db][metabase.metabot.metadata-perms :as metabot.perms]@@ -512,13 +513,14 @@profile (or (profiles/get-profile profile-id)(throw (ex-info "Unknown profile" {:profile-id profile-id})))capabilities (get context :capabilities #{})- ;; Where other people can read the conversation, or there is no conversation or user to check, run_query would- ;; refuse every call, so it is not offered and the prompt does not promise results. The tool checks again, for- ;; someone who joins mid-turn.+ ;; Where run_query would refuse every call, it is not offered and the prompt does not promise results: where+ ;; other people can read the conversation, where there is no conversation or user to check, and for the+ ;; embedded Metabot, whatever profile the request names. The tool checks again, for someone who joins mid-turn.offered (profiles/profile->tools profile capabilities)base-tools (cond-> offered(and (contains? offered "run_query")(or (not (and conversation-id api/*current-user-id*))+ (= (:entity_id metabot) metabot.config/embedded-metabot-id)(metabot.db/conversation-open-to-others? conversation-id api/*current-user-id*)))(dissoc "run_query"))seeded (-> (or state {})diff --git a/src/metabase/metabot/agent/profiles.clj b/src/metabase/metabot/agent/profiles.cljindex dff2c5c7700..b380aa2e368 100644--- a/src/metabase/metabot/agent/profiles.clj+++ b/src/metabase/metabot/agent/profiles.clj@@ -126,7 +126,8 @@;; Serves every embedded surface — the SDK `MetabotQuestion`, modular embedding, data apps, and full-app;; iframe embeds — and has no SQL tools in any of them. The prompt decides per request whether there is;; a Metabase UI around it to hand off to (see [[embedding-system-prompt-context]]).-;; TODO (Chris 2026-10-14) -- Offer run_query here once embedded Metabot has its own query execution setting.+;; TODO (Chris 2026-10-14) -- Offer run_query here once embedded Metabot has its own query execution setting,+;; and replace the embedded Metabot checks in `init-agent` and `run-query` with it.;; Queries run as the signed-in user, so sandboxing and impersonation apply,;; but the rows are the embedding customer's end users' data, and they go to the LLM provider.;; An admin should be able to show results to their own staff without showing them in every embed.diff --git a/src/metabase/metabot/settings.clj b/src/metabase/metabot/settings.cljindex e458c1b6966..556dc37728b 100644--- a/src/metabase/metabot/settings.clj+++ b/src/metabase/metabot/settings.clj@@ -111,7 +111,7 @@(deferred-tru "Whether Metabot can run queries and read their results. Result rows are sent to the LLM provider."):type :boolean:visibility :admin- :default false+ :default true:export? false)(defsetting metabot-sql-execution-enabled?diff --git a/src/metabase/metabot/tools/run_query.clj b/src/metabase/metabot/tools/run_query.cljindex 5f4abfa8576..bdd1586cbbc 100644--- a/src/metabase/metabot/tools/run_query.clj+++ b/src/metabase/metabot/tools/run_query.clj@@ -8,6 +8,7 @@[metabase.driver.util :as driver.u][metabase.lib-be.core :as lib-be][metabase.lib.core :as lib]+ [metabase.metabot.config :as metabot.config][metabase.metabot.db :as metabot.db][metabase.metabot.query-execution :as query-execution][metabase.metabot.scope :as scope]@@ -75,6 +76,13 @@{:message "Query execution is turned off for Metabot.":recovery []})+(tools/defrecoverable embedded-metabot!+ "The embedded Metabot asked to run a query. The rows would be the embedding customer's end users' data."+ {:payload [:map {:closed true}]}+ [_]+ {:message "Metabot can't run queries in embedded apps."+ :recovery []})+(tools/defrecoverable no-conversation!"The call is missing a Metabot conversation or a current user."{:payload [:map {:closed true}]}@@ -682,6 +690,8 @@[{:keys [query_id row_limit]} allow-non-sql?](when-not (metabot.settings/metabot-query-execution-enabled?)(query-execution-off! {}))+ (when (= (:entity_id shared/*metabot*) metabot.config/embedded-metabot-id)+ (embedded-metabot! {}))(check-private-conversation!)(let [{:keys [query saved-card]} (stored-query query_id allow-non-sql?)runnable (runnable-query query_id query allow-non-sql?)]diff --git a/test/metabase/metabot/agent/core_test.clj b/test/metabase/metabot/agent/core_test.cljindex be5e367dc70..b03dfafe444 100644--- a/test/metabase/metabot/agent/core_test.clj+++ b/test/metabase/metabot/agent/core_test.clj@@ -186,16 +186,27 @@(is (contains? tools "edit_sql_query"))(is (contains? tools "replace_sql_query")))))))+(defn- metabot-row+ "The Metabot whose entity ID is `entity-id`."+ [entity-id]+ (t2/select-one :model/Metabot :entity_id entity-id))++(defn- internal-metabot+ "The internal Metabot, which offers run_query."+ []+ (metabot-row metabot.config/internal-metabot-id))+(defn- run-query-offered-at-turn-start"Whether the turn starts with `run_query` among its tools, whether its system message says results can be read, andwhether it declares `show_chart` on construct_notebook_query. Reads them where the decision is made rather thanrunning a turn. The bindings stand in for the ones the agent loop makes."- [conversation-id capabilities]+ [conversation-id capabilities metabot profile-id](let [{:keys [tools system-msg]}(binding [scope/*current-user-metabot-permissions* scope/all-yes-permissionsscope/*current-user-scope* (scope/user-metabot-perms->scopes scope/all-yes-permissions)](#'agent/init-agent {:messages [{:role :user :content "How many orders?"}]- :profile-id :internal+ :metabot metabot+ :profile-id profile-id:conversation-id conversation-id:context {:capabilities capabilities}}))]{:tool? (contains? tools "run_query")@@ -209,17 +220,19 @@(deftest client-claimed-feature-capability-cannot-override-settings-test;; In a conversation the user owns, so the setting decides and not the rule for shared conversations.(mt/with-temp [:model/MetabotConversation {conversation-id :id} {:user_id (mt/user->id :crowberto)}]- (mt/with-current-user (mt/user->id :crowberto)- (testing "a request claiming feature:query_execution gets no run_query while an admin has it turned off"- (mt/with-temporary-setting-values [metabot-query-execution-enabled? false]- (is (not (:tool? (run-query-offered-at-turn-start conversation-id ["feature:query_execution"]))))))- (testing "the same request gets run_query once the admin turns it on"- (mt/with-temporary-setting-values [metabot-query-execution-enabled? true]- (is (:tool? (run-query-offered-at-turn-start conversation-id ["feature:query_execution"]))))))))+ (let [offered? #(:tool? (run-query-offered-at-turn-start conversation-id ["feature:query_execution"]+ (internal-metabot) :internal))]+ (mt/with-current-user (mt/user->id :crowberto)+ (testing "a request claiming feature:query_execution gets no run_query while an admin has it turned off"+ (mt/with-temporary-setting-values [metabot-query-execution-enabled? false]+ (is (not (offered?)))))+ (testing "the same request gets run_query once the admin turns it on"+ (mt/with-temporary-setting-values [metabot-query-execution-enabled? true]+ (is (offered?))))))))(deftest run-query-not-offered-where-others-can-read-the-conversation-test(let [rasta (mt/user->id :rasta)- offered #(run-query-offered-at-turn-start % [])]+ offered #(run-query-offered-at-turn-start % [] (internal-metabot) :internal)](mt/with-temp [:model/MetabotConversation {own :id} {:user_id rasta}:model/MetabotConversation {thread :id} {:user_id rasta, :slack_thread_ts "1.2"}:model/MetabotConversation {someone :id} {:user_id (mt/user->id :crowberto)}]@@ -235,6 +248,22 @@(testing "no conversation does none of them"(is (= {:tool? false, :prompt? false, :show-chart? false} (offered nil)))))))))+(deftest run-query-not-offered-to-embedded-metabot-test+ ;; The request names the profile, so an embedded Metabot can ask for one that offers run_query.+ (let [rasta (mt/user->id :rasta)+ embedded (metabot-row metabot.config/embedded-metabot-id)+ internal (internal-metabot)]+ (mt/with-temp [:model/MetabotConversation {own :id} {:user_id rasta}]+ (mt/with-temporary-setting-values [metabot-query-execution-enabled? true]+ (mt/with-current-user rasta+ (doseq [profile-id [:nlq :internal]]+ (testing profile-id+ (testing "the internal Metabot offers run_query"+ (is (:tool? (run-query-offered-at-turn-start own [] internal profile-id))))+ (testing "the embedded Metabot does not, and its prompt does not promise results"+ (is (=? {:tool? false, :prompt? false}+ (run-query-offered-at-turn-start own [] embedded profile-id)))))))))))+(deftest document-sql-chart-tool-requires-native-permission-test(mt/with-no-data-perms-for-all-users!(perms/set-database-permission! (perms-group/all-users) (mt/id) :perms/view-data :unrestricted)diff --git a/test/metabase/metabot/settings_test.clj b/test/metabase/metabot/settings_test.cljindex 1cc4c5240a9..97ccf566ef1 100644--- a/test/metabase/metabot/settings_test.clj+++ b/test/metabase/metabot/settings_test.clj@@ -113,6 +113,13 @@(is (boolean? (metabot.settings/embedded-metabot-enabled?)))(is (string? (metabot.settings/llm-metabot-provider))))))+(deftest query-execution-on-by-default-test+ (testing "an instance that never set it can run queries, and one that turned it off keeps it off"+ (mt/with-temporary-setting-values [metabot-query-execution-enabled? nil]+ (is (true? (metabot.settings/metabot-query-execution-enabled?))))+ (mt/with-temporary-setting-values [metabot-query-execution-enabled? false]+ (is (false? (metabot.settings/metabot-query-execution-enabled?))))))+(deftest metabot-configured-with-managed-connection-and-proxy-url-test(testing "returns true when the managed connection is selected and the LLM proxy is configured"(mt/with-premium-features #{:metabase-ai-managed}diff --git a/test/metabase/metabot/tools/run_query_test.clj b/test/metabase/metabot/tools/run_query_test.cljindex 6fd6b0e7374..5de14898927 100644--- a/test/metabase/metabot/tools/run_query_test.clj+++ b/test/metabase/metabot/tools/run_query_test.clj@@ -42,19 +42,19 @@(defn- invoke!"Call `run_query` through `tool-var` as the agent loop does.- Include `other-tools` in the profile and use `memory` as the agent memory. Return the tool's result,- or `{:output <the text the model reads> :error {:class :code}}` on failure.+ Include `other-tools` in the profile, run as `metabot`, and use `memory` as the agent memory.+ Return the tool's result, or `{:output <the text the model reads> :error {:class :code}}` on failure.Allow the runtime's scope check to pass so tests can vary the tool's SQL scope check independently."- [tool-var other-tools memory args]+ [tool-var other-tools metabot memory args](let [entries (binding [scope/*current-user-scope* api-scope/unrestricted]- (tools/->entries (cons tool-var other-tools) (atom memory) nil :internal))]+ (tools/->entries (cons tool-var other-tools) (atom memory) metabot :internal))]((:fn (get entries "run_query")) args)))(defn- call-tool!"Call Chat's `run_query` as rasta with `memory` as the agent memory."[memory args](mt/with-current-user (mt/user->id :rasta)- (invoke! #'tools/run-query-tool chat-tools memory args)))+ (invoke! #'tools/run-query-tool chat-tools nil memory args)))(defn- call-tool-in-conversation!"[[call-tool!]] in a conversation of rasta's own."@@ -179,6 +179,16 @@(mt/with-temporary-setting-values [metabot-query-execution-enabled? false](is (= (refusal ::run-query/query-execution-off "Query execution is turned off for Metabot.")(call-tool! {:state {:queries {"q1" (venues-by-id)}}} {:query_id "q1"})))))+ (testing "nothing runs for the embedded Metabot, even with query execution turned on"+ (mt/with-temp [:model/MetabotConversation {conversation-id :id} {:user_id (mt/user->id :rasta)}]+ (mt/with-temporary-setting-values [metabot-query-execution-enabled? true]+ (mt/with-current-user (mt/user->id :rasta)+ (is (= (refusal ::run-query/embedded-metabot "Metabot can't run queries in embedded apps.")+ (invoke! #'tools/run-query-tool+ chat-tools+ (t2/select-one :model/Metabot :entity_id metabot.config/embedded-metabot-id)+ {:conversation-id conversation-id, :state {:queries {"q1" (venues-by-id)}}}+ {:query_id "q1"})))))))(testing "an unknown id lists the ids the model can use"(is (= (refusal ::recoverable.common/unknown-query-id(str "Query \"nope\" was not found. Available query IDs: \"q1\". Use the ID of a query that you "@@ -451,6 +461,7 @@(is (= database-not-found(invoke! #'tools/run-query-toolchat-tools+ nil{:conversation-id conversation-id:state {:queries {"q1" {:database Integer/MAX_VALUE:type :query@@ -645,6 +656,7 @@(mt/with-current-user (mt/user->id :rasta)(invoke! #'tools/run-query-toolother-tools+ nil{:conversation-id conversation-id:state {:queries {"q1" query}}}{:query_id "q1"})))))@@ -963,6 +975,7 @@(mt/with-current-user (mt/user->id :rasta)(invoke! #'tools/run-sql-query-tool[#'tools/create-sql-query-tool]+ nil(assoc memory :conversation-id conversation-id)args))))