Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Show tool errors in Metabot's trace summary, remove a dead JSON check (#84629)

Andrei Simionescubccc6351c614
diff --git a/src/metabase/metabot/agent/core.clj b/src/metabase/metabot/agent/core.clj
index 5899d45c262..8edb6b7dfdb 100644
--- a/src/metabase/metabot/agent/core.clj
+++ b/src/metabase/metabot/agent/core.clj
@@ -79,7 +79,7 @@
:tool-results (mapv (fn [p] {:id (:id p)
:output (let [r (:result p)]
(or (:output r) r))
- :error (:error p)})
+ :error (get-in p [:result :error])})
outputs)}))
(defn- debug-log!
diff --git a/src/metabase/metabot/self/core.clj b/src/metabase/metabot/self/core.clj
index 3fa64e98d30..2445ff108b3 100644
--- a/src/metabase/metabot/self/core.clj
+++ b/src/metabase/metabot/self/core.clj
@@ -481,9 +481,7 @@
:function (:toolName chunk)
:arguments (parse-tool-arguments chunks)}
pm (assoc :provider-metadata pm)))
- ;; No `:error` on the part. A failed call carries its `:error` inside `:result`, which is the one
- ;; place anything asks — the agent loop, the SSE writer, persistence and telemetry all read it
- ;; there. A second copy here is what let a failure reach the client as a success.
+ ;; A failed call carries its `:error` inside `:result`, not on the part.
:tool-output-available {:type :tool-output
:id (:toolCallId chunk)
:function (:toolName chunk)
diff --git a/src/metabase/metabot/tools/runtime.clj b/src/metabase/metabot/tools/runtime.clj
index 0f73c8c59f7..528d3e6ae84 100644
--- a/src/metabase/metabot/tools/runtime.clj
+++ b/src/metabase/metabot/tools/runtime.clj
@@ -38,7 +38,6 @@
Written down as data so the agent loop, the adapters and the tests share one list rather than
three sets of literal keywords. `:repeated-error` is the agent loop's and is not raised here."
{:unknown-tool :validation
- :invalid-json :validation
:invalid-arguments :validation
:scope-denied :unrecoverable
:internal :unrecoverable})
@@ -92,12 +91,6 @@
;;; ------------------------------------------------ Arguments -----------------------------------------------------
-;;; The argument-repair machinery below currently also lives in `metabase.metabot.self.core`. It is
-;;; duplicated rather than shared because the dependency runs the other way once tools are migrated
-;;; — `self.core/run-tool` will call [[invoke]] — and `self.core`'s copies go away with the checks
-;;; they serve. Keep the wording identical until then: this is the text models have been repairing
-;;; against.
-
(def ^:private stringified-scalar-transformer
"Parses stringified numbers and booleans back into scalars, driven by the tool's own schema.
Restricted to the types models get wrong — strings, keywords and enums are left alone."
@@ -150,18 +143,6 @@
(str "expected an object of named arguments; received "
(json-type-name arguments) "."))))))
-(def raw-arguments-key
- "The key the stream collector leaves behind when a model's tool-call arguments were not valid JSON.
-
- Its *presence* is what [[invoke]] reports as `:invalid-json`; the value is the model's own
- malformed output and never reaches a message. Malformed JSON and schema-valid JSON that breaks the
- schema are separate codes because they are separate mistakes, and the repair differs: resend the
- call as an object, versus fix a named argument."
- :_raw_arguments)
-
-(def ^:private invalid-json-message
- "Invalid tool arguments: the arguments were not valid JSON. Send the call again as a JSON object.")
-
;;; ------------------------------------------------ Rendering -----------------------------------------------------
(defn- unrecoverable-output
@@ -234,7 +215,7 @@
(defn- validation-ex
"The exception for one of the runtime's validation errors. Thrown rather than returned so that all
- seven steps of [[invoke]] leave by the same door."
+ six steps of [[invoke]] leave by the same door."
^Throwable [code message]
(ex-info message {tools.error/error-key {:class :validation :code code :message message}}))
@@ -279,8 +260,6 @@
"`args` coerced and checked against the declaration's `:args` schema. Throws a validation error
otherwise."
[{:keys [args]} arguments]
- (when (and (map? arguments) (contains? arguments raw-arguments-key))
- (throw (validation-ex :invalid-json invalid-json-message)))
(let [coerced (coerce-stringified-scalars args arguments)]
(when-let [message (invalid-arguments-message args coerced)]
(throw (validation-ex :invalid-arguments message)))
@@ -341,12 +320,11 @@
In order:
1. an unknown `tool-name` is a validation error listing what this profile does have;
- 2. arguments that were not valid JSON are a validation error (see [[raw-arguments-key]]);
- 3. stringified scalars are coerced, then the arguments are validated against `:args`;
- 4. the scope is checked, and a denial is unrecoverable;
- 5. the tool runs with the dynamic vars bound from `ctx`, through `tools.core/call`;
- 6. its result is validated against `tools.core/result` and its `:output` rendered to a string;
- 7. any exception is classified, logged in full, and rendered for its audience."
+ 2. stringified scalars are coerced, then the arguments are validated against `:args`;
+ 3. the scope is checked, and a denial is unrecoverable;
+ 4. the tool runs with the dynamic vars bound from `ctx`, through `tools.core/call`;
+ 5. its result is validated against `tools.core/result` and its `:output` rendered to a string;
+ 6. any exception is classified, logged in full, and rendered for its audience."
[entries ctx tool-name args]
(let [tool-names (:tool-names ctx)]
(try
diff --git a/test/metabase/metabot/agent/core_test.clj b/test/metabase/metabot/agent/core_test.clj
index cdec9694ad8..5ae0ca2160c 100644
--- a/test/metabase/metabot/agent/core_test.clj
+++ b/test/metabase/metabot/agent/core_test.clj
@@ -269,6 +269,11 @@
(testing "should-continue? is unaffected — the gate lives in loop-step, per profile"
(is (#'agent/should-continue? 0 20 #{} denial)))))
+(deftest ^:parallel a-failed-call-shows-its-error-in-the-trace-summary-test
+ (is (= {:class :unrecoverable :code :x}
+ (-> (#'agent/summarize-tool-ios [(unrecoverable-part "a" nil)])
+ :tool-results first :error))))
+
(defn- run-sql-denial-turn!
"Run one turn whose first LLM response calls `create_sql_query` against `database-id`."
[profile-id database-id]
diff --git a/test/metabase/metabot/tools/runtime_test.clj b/test/metabase/metabot/tools/runtime_test.clj
index 21d9b850604..22aa8a3b749 100644
--- a/test/metabase/metabot/tools/runtime_test.clj
+++ b/test/metabase/metabot/tools/runtime_test.clj
@@ -150,16 +150,6 @@
(is (= "Invalid tool arguments: `m` is not a supported argument. `n` is required."
(:output outcome)))))
-(deftest ^:parallel invalid-json-is-its-own-code-test
- (testing "malformed JSON and schema-invalid JSON are different mistakes with different repairs"
- (let [outcome (invoke "happy" {tools.runtime/raw-arguments-key "{\"n\": "})]
- (is (= {:class :validation :code :invalid-json} (:error outcome)))
- (is (= (str "Invalid tool arguments: the arguments were not valid JSON. "
- "Send the call again as a JSON object.")
- (:output outcome)))
- (testing "and the model's own malformed output is not quoted back at it"
- (is (not (str/includes? (:output outcome) "{\"n\":")))))))
-
;;; ------------------------------------------------ Recoverable ---------------------------------------------------
(deftest ^:parallel recoverable-renders-message-then-steps-test
@@ -358,7 +348,6 @@
(testing "the agent loop and the adapters branch on these, so the list is shared rather than
re-spelled in each place"
(is (= {:unknown-tool :validation
- :invalid-json :validation
:invalid-arguments :validation
:scope-denied :unrecoverable
:internal :unrecoverable}
@@ -366,12 +355,11 @@
(testing "and every one of them is reachable through invoke"
;; The scoped case needs a ctx that grants nothing; the rest run under the default
;; unrestricted one.
- (is (= #{:unknown-tool :invalid-json :invalid-arguments :scope-denied :internal}
+ (is (= #{:unknown-tool :invalid-arguments :scope-denied :internal}
(set (keep (fn [[tool-name args overrides]]
(get-in (tools.runtime/invoke entries (ctx overrides) tool-name args)
[:error :code]))
[["nope" {}]
- ["happy" {tools.runtime/raw-arguments-key "{"}]
["happy" {:m 1}]
["scoped" {} {:user-scope #{}}]
["crash" {}]])))))))