The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Show tool errors in Metabot's trace summary, remove a dead JSON check (#84629)
Andrei Simionescubccc6351c614
diff --git a/src/metabase/metabot/agent/core.clj b/src/metabase/metabot/agent/core.cljindex 5899d45c262..8edb6b7dfdb 100644--- a/src/metabase/metabot/agent/core.clj+++ b/src/metabase/metabot/agent/core.clj@@ -79,7 +79,7 @@:tool-results (mapv (fn [p] {:id (:id p):output (let [r (:result p)](or (:output r) r))- :error (:error p)})+ :error (get-in p [:result :error])})outputs)}))(defn- debug-log!diff --git a/src/metabase/metabot/self/core.clj b/src/metabase/metabot/self/core.cljindex 3fa64e98d30..2445ff108b3 100644--- a/src/metabase/metabot/self/core.clj+++ b/src/metabase/metabot/self/core.clj@@ -481,9 +481,7 @@:function (:toolName chunk):arguments (parse-tool-arguments chunks)}pm (assoc :provider-metadata pm)))- ;; No `:error` on the part. A failed call carries its `:error` inside `:result`, which is the one- ;; place anything asks — the agent loop, the SSE writer, persistence and telemetry all read it- ;; there. A second copy here is what let a failure reach the client as a success.+ ;; A failed call carries its `:error` inside `:result`, not on the part.:tool-output-available {:type :tool-output:id (:toolCallId chunk):function (:toolName chunk)diff --git a/src/metabase/metabot/tools/runtime.clj b/src/metabase/metabot/tools/runtime.cljindex 0f73c8c59f7..528d3e6ae84 100644--- a/src/metabase/metabot/tools/runtime.clj+++ b/src/metabase/metabot/tools/runtime.clj@@ -38,7 +38,6 @@Written down as data so the agent loop, the adapters and the tests share one list rather thanthree sets of literal keywords. `:repeated-error` is the agent loop's and is not raised here."{:unknown-tool :validation- :invalid-json :validation:invalid-arguments :validation:scope-denied :unrecoverable:internal :unrecoverable})@@ -92,12 +91,6 @@;;; ------------------------------------------------ Arguments ------------------------------------------------------;;; The argument-repair machinery below currently also lives in `metabase.metabot.self.core`. It is-;;; duplicated rather than shared because the dependency runs the other way once tools are migrated-;;; — `self.core/run-tool` will call [[invoke]] — and `self.core`'s copies go away with the checks-;;; they serve. Keep the wording identical until then: this is the text models have been repairing-;;; against.-(def ^:private stringified-scalar-transformer"Parses stringified numbers and booleans back into scalars, driven by the tool's own schema.Restricted to the types models get wrong — strings, keywords and enums are left alone."@@ -150,18 +143,6 @@(str "expected an object of named arguments; received "(json-type-name arguments) "."))))))-(def raw-arguments-key- "The key the stream collector leaves behind when a model's tool-call arguments were not valid JSON.-- Its *presence* is what [[invoke]] reports as `:invalid-json`; the value is the model's own- malformed output and never reaches a message. Malformed JSON and schema-valid JSON that breaks the- schema are separate codes because they are separate mistakes, and the repair differs: resend the- call as an object, versus fix a named argument."- :_raw_arguments)--(def ^:private invalid-json-message- "Invalid tool arguments: the arguments were not valid JSON. Send the call again as a JSON object.")-;;; ------------------------------------------------ Rendering -----------------------------------------------------(defn- unrecoverable-output@@ -234,7 +215,7 @@(defn- validation-ex"The exception for one of the runtime's validation errors. Thrown rather than returned so that all- seven steps of [[invoke]] leave by the same door."+ six steps of [[invoke]] leave by the same door."^Throwable [code message](ex-info message {tools.error/error-key {:class :validation :code code :message message}}))@@ -279,8 +260,6 @@"`args` coerced and checked against the declaration's `:args` schema. Throws a validation errorotherwise."[{:keys [args]} arguments]- (when (and (map? arguments) (contains? arguments raw-arguments-key))- (throw (validation-ex :invalid-json invalid-json-message)))(let [coerced (coerce-stringified-scalars args arguments)](when-let [message (invalid-arguments-message args coerced)](throw (validation-ex :invalid-arguments message)))@@ -341,12 +320,11 @@In order:1. an unknown `tool-name` is a validation error listing what this profile does have;- 2. arguments that were not valid JSON are a validation error (see [[raw-arguments-key]]);- 3. stringified scalars are coerced, then the arguments are validated against `:args`;- 4. the scope is checked, and a denial is unrecoverable;- 5. the tool runs with the dynamic vars bound from `ctx`, through `tools.core/call`;- 6. its result is validated against `tools.core/result` and its `:output` rendered to a string;- 7. any exception is classified, logged in full, and rendered for its audience."+ 2. stringified scalars are coerced, then the arguments are validated against `:args`;+ 3. the scope is checked, and a denial is unrecoverable;+ 4. the tool runs with the dynamic vars bound from `ctx`, through `tools.core/call`;+ 5. its result is validated against `tools.core/result` and its `:output` rendered to a string;+ 6. any exception is classified, logged in full, and rendered for its audience."[entries ctx tool-name args](let [tool-names (:tool-names ctx)](trydiff --git a/test/metabase/metabot/agent/core_test.clj b/test/metabase/metabot/agent/core_test.cljindex cdec9694ad8..5ae0ca2160c 100644--- a/test/metabase/metabot/agent/core_test.clj+++ b/test/metabase/metabot/agent/core_test.clj@@ -269,6 +269,11 @@(testing "should-continue? is unaffected — the gate lives in loop-step, per profile"(is (#'agent/should-continue? 0 20 #{} denial)))))+(deftest ^:parallel a-failed-call-shows-its-error-in-the-trace-summary-test+ (is (= {:class :unrecoverable :code :x}+ (-> (#'agent/summarize-tool-ios [(unrecoverable-part "a" nil)])+ :tool-results first :error))))+(defn- run-sql-denial-turn!"Run one turn whose first LLM response calls `create_sql_query` against `database-id`."[profile-id database-id]diff --git a/test/metabase/metabot/tools/runtime_test.clj b/test/metabase/metabot/tools/runtime_test.cljindex 21d9b850604..22aa8a3b749 100644--- a/test/metabase/metabot/tools/runtime_test.clj+++ b/test/metabase/metabot/tools/runtime_test.clj@@ -150,16 +150,6 @@(is (= "Invalid tool arguments: `m` is not a supported argument. `n` is required."(:output outcome)))))-(deftest ^:parallel invalid-json-is-its-own-code-test- (testing "malformed JSON and schema-invalid JSON are different mistakes with different repairs"- (let [outcome (invoke "happy" {tools.runtime/raw-arguments-key "{\"n\": "})]- (is (= {:class :validation :code :invalid-json} (:error outcome)))- (is (= (str "Invalid tool arguments: the arguments were not valid JSON. "- "Send the call again as a JSON object.")- (:output outcome)))- (testing "and the model's own malformed output is not quoted back at it"- (is (not (str/includes? (:output outcome) "{\"n\":")))))))-;;; ------------------------------------------------ Recoverable ---------------------------------------------------(deftest ^:parallel recoverable-renders-message-then-steps-test@@ -358,7 +348,6 @@(testing "the agent loop and the adapters branch on these, so the list is shared rather thanre-spelled in each place"(is (= {:unknown-tool :validation- :invalid-json :validation:invalid-arguments :validation:scope-denied :unrecoverable:internal :unrecoverable}@@ -366,12 +355,11 @@(testing "and every one of them is reachable through invoke";; The scoped case needs a ctx that grants nothing; the rest run under the default;; unrestricted one.- (is (= #{:unknown-tool :invalid-json :invalid-arguments :scope-denied :internal}+ (is (= #{:unknown-tool :invalid-arguments :scope-denied :internal}(set (keep (fn [[tool-name args overrides]](get-in (tools.runtime/invoke entries (ctx overrides) tool-name args)[:error :code]))[["nope" {}]- ["happy" {tools.runtime/raw-arguments-key "{"}]["happy" {:m 1}]["scoped" {} {:user-scope #{}}]["crash" {}]])))))))