The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
BOT-2010: Post Metabot's Slack answer as a markdown block (#80652)
Paolo Rascunàa9f25e8645cb
diff --git a/resources/metabot/prompts/system/slackbot.selmer b/resources/metabot/prompts/system/slackbot.selmerindex 95339385d1e..155bf622578 100644--- a/resources/metabot/prompts/system/slackbot.selmer+++ b/resources/metabot/prompts/system/slackbot.selmer@@ -19,17 +19,17 @@ Be direct and concise. Get users to insights efficiently while providing transpa**Important: You cannot create inline visualizations, alerts, or dashboard subscriptions.** You do not have the ability to render visualizations or create alerts. Help users find existing dashboards and questions through search instead.{% endif %}-**Slack Markdown**: Use Slack-specific formatting exclusively:+**Slack Markdown**: Slack renders standard markdown for text styling, but keeps its own syntax for links and mentions. Follow both sets of rules below exactly.**Text Formatting**:-- `*text*` for bold text (single asterisks only)+- `**text**` for bold text (double asterisks — a single asterisk renders as italic, not bold)- `_text_` for italic text (underscores only)-- `~text~` for strikethrough text+- `~~text~~` for strikethrough text (double tildes — a single tilde does not render at all)- `` `text` `` for inline code-- ```text``` or ```language\ncode\n``` for code blocks+- ```` ```\ntext\n``` ```` or ```` ```language\ncode\n``` ```` for code blocks**Lists**:-- `• Item` or `- Item` for bullet lists+- `- Item` for bullet lists- `1. Item` for numbered lists- Indent with spaces for nested lists- NEVER add extra blank lines after list items - use single line breaks only@@ -45,8 +45,8 @@ Be direct and concise. Get users to insights efficiently while providing transpa- Use the metabase:// protocol for all Metabase entity links: `<metabase://dashboard/123|Dashboard Name>`- This applies to dashboards, questions, metrics, models, tables, charts, and queries-**Don't Use**: These standard markdown formats don't work in Slack:-- `**text**` (double asterisks for bold)+**Don't Use**: Never use these formats:+- `*text*` (single asterisks — renders as italic, not bold)- `__text__` (double underscores for bold)- `[text](url)` (standard markdown links)- `# Headers` (markdown headers)@@ -391,8 +391,8 @@ For multiple files, list each model on its own line:```Your data has been uploaded:-• 📦 <metabase://model/{id1}|{display_name1}>-• 📦 <metabase://model/{id2}|{display_name2}>+- 📦 <metabase://model/{id1}|{display_name1}>+- 📦 <metabase://model/{id2}|{display_name2}>{initial_analysis}```diff --git a/src/metabase/slackbot/channel.clj b/src/metabase/slackbot/channel.cljindex 8b4556b846f..20140218ccd 100644--- a/src/metabase/slackbot/channel.clj+++ b/src/metabase/slackbot/channel.clj@@ -29,20 +29,30 @@See https://docs.slack.dev/reference/block-kit/blocks/section-block."3000)+(def markdown-text-limit+ "Slack documents this as \"the cumulative limit for all `markdown` blocks in a single payload\" in characters.+ Today Slack enforces the figure per block rather than cumulatively.+ Budgeting cumulatively is a deliberate choice: enforcement can catch up with the documentation+ without notice, which would turn messages that fit today into rejections.+ See https://docs.slack.dev/reference/block-kit/blocks/markdown-block+ https://github.com/metabase/metabase/pull/80652#discussion_r3861967932."+ 12000)+(defn- final-text-blocks"Build the leading text block(s) for a finalized non-streaming Slack message."[text](if (str/blank? text)[]- [{:type "section"- :text {:type "mrkdwn"- :text text}}]))+ ;; Slack expands one `markdown` block into several as it renders: a heading becomes a `header`,+ ;; a thematic break a `divider`, a table a `table` -- and those count against the 50-block limit.+ [{:type "markdown"+ :text text}]))(defn- truncation-notice"Copy explaining that an answer was cut short.`url` points at the Metabase instance; nil leaves the sentence standing without a link."[url]- (str "_This answer was longer than " section-text-limit " characters. That is too long to post in "+ (str "_This answer was longer than " markdown-text-limit " characters. That is too long to post in ""Slack, so I cut it short._\n\n""Ask a narrower question so the answer comes back smaller"(if url@@ -126,7 +136,7 @@;; `final-text` doubles as the message's `:text` -- Slack's notification preview, and;; what `streaming/thread->history` replays to the model as the assistant's own words.;; The notice stays out of it, so the model is never told it said this.- final-text (u.str/elide answer section-text-limit)+ final-text (u.str/elide answer markdown-text-limit);; Derived rather than re-tested: `elide` returns `answer` itself when it fits, so;; the notice cannot disagree with whether a cut actually happened.truncated? (not= final-text answer)@@ -138,7 +148,7 @@final-text :blocks final-blocks)](when truncated?(log/infof "[slackbot] channel answer truncated (answer_length=%d limit=%d)"- (count answer) section-text-limit)+ (count answer) markdown-text-limit)(analytics/inc! :metabase-slackbot/responses-truncated))(when-let [res-ts (:ts res)](metabot.persistence/set-response-slack-msg-id! assistant-msg-id res-ts))diff --git a/test/metabase/metabot/agent/prompt_gating_test.clj b/test/metabase/metabot/agent/prompt_gating_test.cljindex 1ca2a9e62ba..727ca4146f2 100644--- a/test/metabase/metabot/agent/prompt_gating_test.clj+++ b/test/metabase/metabot/agent/prompt_gating_test.clj@@ -213,6 +213,22 @@(is (not (re-find #"You cannot build custom queries" rendered)))(is (not (re-find #"You cannot create inline visualizations" rendered))))))+(deftest ^:parallel slackbot-prompt-teaches-standard-markdown-test+ ;; The channel answer goes in a `markdown` block, which renders standard markdown rather than+ ;; Slack's `mrkdwn`. Measured against the API: `*one*` comes back italic and `~one~` is not+ ;; parsed at all, so the old dialect would silently lose every bold and strikethrough.+ (testing "the prompt teaches the dialect the answer block actually renders (BOT-2010)"+ (let [rendered (render-slackbot-template all-yes-perms)]+ (testing "standard markdown is prescribed"+ (is (re-find #"`\*\*text\*\*` for bold text" rendered))+ (is (re-find #"`~~text~~` for strikethrough text" rendered)))+ (testing "and the Slack-only dialect is not"+ (is (not (re-find #"`\*text\*` for bold text" rendered)))+ (is (not (re-find #"`~text~` for strikethrough text" rendered))))+ (testing "Slack's link syntax is kept -- it does render inside a markdown block (measured)"+ (is (re-find #"<https://example\.com\|Link Text>" rendered))+ (is (re-find #"metabase://dashboard/123\|Dashboard Name" rendered))))))+;;; ──────────────────────────────────────────────────────────────────;;; Custom system prompt injection;;; ──────────────────────────────────────────────────────────────────diff --git a/test/metabase/slackbot/api_test.clj b/test/metabase/slackbot/api_test.cljindex 33bfbff4cab..2a008d3b4fb 100644--- a/test/metabase/slackbot/api_test.clj+++ b/test/metabase/slackbot/api_test.clj@@ -225,14 +225,15 @@:timeout-ms 5000})(is (= 1 (count @post-calls)))(let [blocks (:blocks (first @post-calls))]- (testing "no section block is over the limit -- unlike the untruncated answer"- (is (nil? (tu/oversized-section-error blocks)))- (is (some? (tu/oversized-section-error [{:type "section"- :text {:type "mrkdwn" :text tu/oversized-answer}}]))+ (testing "no block is over its limit -- unlike the untruncated answer"+ (is (nil? (tu/oversized-block-error blocks)))+ ;; The control has to name the block type the answer actually goes in, or it would+ ;; assert a section rule no longer in play and pass whatever the code does.+ (is (some? (tu/oversized-block-error [{:type "markdown" :text tu/oversized-answer}]))"the answer really is past the limit, so the case under test is the real one"))(testing "the answer is cut to the limit, and the message says why"- (is (= tu/slack-section-text-limit- (count (get-in (first blocks) [:text :text]))))+ (is (= tu/slack-markdown-text-limit+ (count (:text (first blocks)))))(is (some (fn [block](and (= "context" (:type block))(str/includes? (get-in block [:elements 0 :text])diff --git a/test/metabase/slackbot/channel_test.clj b/test/metabase/slackbot/channel_test.cljindex 7736829ff00..5429a803eb7 100644--- a/test/metabase/slackbot/channel_test.clj+++ b/test/metabase/slackbot/channel_test.clj@@ -111,13 +111,15 @@(deftest ^:parallel truncation-notice-test(testing "the notice points at the instance, and says what to do about the cut"- (is (= (str "_This answer was longer than 3000 characters. That is too long to post in Slack, "+ (is (= (str "_This answer was longer than " slackbot.tu/slack-markdown-text-limit+ " characters. That is too long to post in Slack, ""so I cut it short._\n\n""Ask a narrower question so the answer comes back smaller"", or head to <https://metabase.example.com|Metabase> and try again.")(#'slackbot.channel/truncation-notice test-site-url))))(testing "an instance with no site URL still gets a sentence, just without the link"- (is (= (str "_This answer was longer than 3000 characters. That is too long to post in Slack, "+ (is (= (str "_This answer was longer than " slackbot.tu/slack-markdown-text-limit+ " characters. That is too long to post in Slack, ""so I cut it short._\n\n""Ask a narrower question so the answer comes back smaller.")(#'slackbot.channel/truncation-notice nil))))@@ -130,15 +132,14 @@(mt/with-temporary-setting-values [site-url test-site-url](let [{:keys [text blocks backfill]} (send-channel-response! slackbot.tu/oversized-answer)[answer-block notice-block] blocks- section-text (get-in answer-block [:text :text])]- (testing "no section block is over the limit -- so Slack no longer rejects the message"- (is (nil? (slackbot.tu/oversized-section-error blocks))))- (testing "the answer rides one mrkdwn section block, cut to the limit"- (is (= "section" (:type answer-block)))- (is (= "mrkdwn" (get-in answer-block [:text :type])))- (is (= slackbot.tu/slack-section-text-limit (count section-text)))- (is (str/ends-with? section-text "..."))- (is (str/starts-with? slackbot.tu/oversized-answer (str/replace section-text #"\.\.\.$" ""))+ answer-text (:text answer-block)]+ (testing "no block is over its limit -- so Slack no longer rejects the message"+ (is (nil? (slackbot.tu/oversized-block-error blocks))))+ (testing "the answer goes in one markdown block, cut to the limit (BOT-2010)"+ (is (= "markdown" (:type answer-block)))+ (is (= slackbot.tu/slack-markdown-text-limit (count answer-text)))+ (is (str/ends-with? answer-text "..."))+ (is (str/starts-with? slackbot.tu/oversized-answer (str/replace answer-text #"\.\.\.$" ""))"what survives the cut is a genuine prefix of the answer"))(testing "the notice rides in a context block of the same message, linking to the instance"(is (= "context" (:type notice-block)))@@ -148,10 +149,10 @@(is (not (str/includes? (get-in notice-block [:elements 0 :text]) "/metabot/conversation/"))"the link is the home page, not a conversation we cannot continue on the web"))(testing "the visualizations and the feedback buttons still ride the message"- (is (= ["section" "context" "section" "table" "context_actions"]+ (is (= ["markdown" "context" "section" "table" "context_actions"](mapv :type blocks))))(testing "the notice stays out of `:text`, which `thread->history` replays back to the model"- (is (= section-text text))+ (is (= answer-text text))(is (not (str/includes? text "too long to post in Slack"))))(testing "one message is posted, and its ts is persisted -- it carries the feedback buttons"(is (= {:msg-id 42 :slack-msg-id "1700000000.000002"} backfill)))))))@@ -171,10 +172,10 @@(testing "an answer within the limit is posted whole, with no notice"(let [answer "Orders peaked in March. *1,204* of them."{:keys [text blocks]} (send-channel-response! answer)]- (is (nil? (slackbot.tu/oversized-section-error blocks)))- (is (= answer (get-in (first blocks) [:text :text])))+ (is (nil? (slackbot.tu/oversized-block-error blocks)))+ (is (= answer (:text (first blocks))))(is (= answer text))- (is (= ["section" "section" "table" "context_actions"] (mapv :type blocks))+ (is (= ["markdown" "section" "table" "context_actions"] (mapv :type blocks))"no context block, because nothing was cut"))));; Not ^:parallel: `with-prometheus-system!` redefs a process-global var.diff --git a/test/metabase/slackbot/streaming_test.clj b/test/metabase/slackbot/streaming_test.cljindex a44243252b4..aabc8aa7f8a 100644--- a/test/metabase/slackbot/streaming_test.clj+++ b/test/metabase/slackbot/streaming_test.clj@@ -173,9 +173,9 @@"revenue-by-month""Revenue by month"huge-link)]- (is (nil? (tu/oversized-section-error blocks))+ (is (nil? (tu/oversized-block-error blocks))"Slack no longer rejects the whole message")- (is (some? (tu/oversized-section-error+ (is (some? (tu/oversized-block-error[{:type "section":text {:type "mrkdwn":text (str "📊 <https://metabase.example.com" huge-link "|Revenue by month>")}}]))diff --git a/test/metabase/slackbot/test_util.clj b/test/metabase/slackbot/test_util.cljindex 0f77effa041..6e00edc1a68 100644--- a/test/metabase/slackbot/test_util.clj+++ b/test/metabase/slackbot/test_util.clj@@ -52,17 +52,19 @@"Alias for [[metabase.slackbot.channel/section-text-limit]]."slackbot.channel/section-text-limit)+(def slack-markdown-text-limit+ "Alias for [[metabase.slackbot.channel/markdown-text-limit]]."+ slackbot.channel/markdown-text-limit)+(def oversized-answer- "An answer comfortably past [[slack-section-text-limit]].+ "An answer comfortably past [[slack-markdown-text-limit]].Numbered line by line, so a truncated copy can be checked against the original."- (str/join "\n" (map #(format "Line %04d of a rather long answer." %) (range 200))))+ (str/join "\n" (map #(format "Line %04d of a rather long answer." %) (range 400))))-(defn oversized-section-error- "The `chat.postMessage` rejection Slack returns for an oversized `section` block in `blocks`.- Nil when every block is within [[slack-section-text-limit]]. Models that one rule -- the one- BOT-1606 is about -- not Block Kit validation at large."+(defn- oversized-section-error+ "Returns Slack's `invalid_blocks` rejection for an over-long `section`, or nil when every one fits.+ Slack names the offending block and its limit, which the messages here reproduce."[blocks]- ;; The mocked client accepts anything, so tests that care about this rule come through here.(when-let [idx (first (keep-indexed (fn [idx block](when (and (= "section" (:type block))(> (count (get-in block [:text :text] ""))@@ -76,6 +78,28 @@(format "[ERROR] must be less than %d characters [json-pointer:/blocks/%d/text/text]"(inc slack-section-text-limit) idx)]}}))+(defn- oversized-markdown-error+ "Returns Slack's `msg_too_long` rejection for over-long `markdown` text, or nil when it fits.+ The budget is charged across every `markdown` block at once, matching what+ [[slack-markdown-text-limit]] documents rather than the per-block enforcement seen today."+ [blocks]+ (let [total (->> blocks+ (filter #(= "markdown" (:type %)))+ (map #(count (:text % "")))+ (reduce + 0))]+ (when (> total slack-markdown-text-limit)+ ;; A bare error, carrying none of the per-block detail the `section` case does (measured).+ {:ok false :error "msg_too_long"})))++(defn oversized-block-error+ "Returns the `chat.postMessage` rejection Slack would send for `blocks`, or nil when it accepts them.+ Models two rules only, the `section` and `markdown` text limits BOT-1606 and BOT-2010 are about,+ not Block Kit validation at large."+ [blocks]+ ;; The mocked client accepts anything, so tests that care about these rules come through here.+ (or (oversized-section-error blocks)+ (oversized-markdown-error blocks)))+(defmacro with-ensure-encryption"Use the existing encryption key if one is configured, otherwise set a test key.Avoids conflicts with encrypted settings in the DB that were written with the real key."