Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Keep a remote Bot answering when it asks for a learned skill that is not there (#663)

kevind12a4cdf1325
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5a67787c..5b7a9a1b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -63,6 +63,14 @@ whoever wrote a skill under that name next, and on a Bot the deployment shares t
instructions answering everybody. It is now refused with "There is no skill called …", as a grant
naming no app already was. Revoking one by hand still works.
+### A remote Bot keeps answering when it asks for a learned skill that is not there
+
+With Automatic Learning delivering skills, a Bot reached at an AG-UI endpoint (every shipped Bot
+except a built-in one) ended the whole turn with "Skill is unavailable." in place of an answer when
+its model asked for a skill by a name the snapshot does not hold, for a file the skill does not
+list, or sent arguments that were not JSON. A built-in Bot's model is handed that sentence as the
+call's result and carries on. A remote Bot's model now gets the same result and carries on too.
+
### Browser challenges can be handed to a person without losing the Bot's page
Bots pause for actionable browser challenges and resume from a fresh page snapshot after an explicit
diff --git a/server/src/learning/remote.ts b/server/src/learning/remote.ts
index 2d5b2053..4841d764 100644
--- a/server/src/learning/remote.ts
+++ b/server/src/learning/remote.ts
@@ -10,7 +10,7 @@ import {
type ToolCall,
} from "@ag-ui/client";
import { map, Observable, type Subscription } from "rxjs";
-import type { AcquireLearnedSkills } from "./runtime";
+import type { AcquireLearnedSkills, LearnedSkillInvocation } from "./runtime";
const CATALOG_ID = "openbot:learned-skills";
const CATALOG_CONTEXT = "OpenBot learned skills";
@@ -189,10 +189,7 @@ export class RemoteLearnedSkillsMiddleware extends Middleware {
const results = new Map<string, string>();
for (const call of reads) {
if (subscriber.closed) return;
- const content = await invocation.execute(
- call.function.name,
- JSON.parse(call.function.arguments || "{}"),
- );
+ const content = await answerRead(invocation, call);
if (subscriber.closed) return;
const message: Message = {
id: crypto.randomUUID(),
@@ -276,6 +273,32 @@ export class RemoteLearnedSkillsMiddleware extends Middleware {
}
}
+/**
+ * One skill read's result, including a read the snapshot cannot answer.
+ *
+ * A model asks for a skill by a name the catalog does not hold, or for a file the skill does not
+ * list, or sends arguments that are not JSON. A built-in Bot's loop (the AI SDK) hands the thrown
+ * message back to the model as that call's result and runs the next step, so the model reads
+ * "Skill is unavailable." and carries on. The reason goes back the same way here, rather than
+ * ending a remote Bot's whole turn. The snapshot is already in memory, so nothing but the call's
+ * own input can make a read fail.
+ */
+async function answerRead(
+ invocation: LearnedSkillInvocation,
+ call: ToolCall,
+): Promise<string> {
+ try {
+ return await invocation.execute(
+ call.function.name,
+ JSON.parse(call.function.arguments || "{}"),
+ );
+ } catch (error) {
+ return error instanceof Error && error.message
+ ? error.message
+ : "That learned Skill could not be read.";
+ }
+}
+
/** AG-UI accumulates from the agent's initial history, not input.messages. */
class InvocationTransport extends AbstractAgent {
constructor(
diff --git a/server/tests/learning-remote.test.ts b/server/tests/learning-remote.test.ts
index b8248ca0..05731d0f 100644
--- a/server/tests/learning-remote.test.ts
+++ b/server/tests/learning-remote.test.ts
@@ -181,6 +181,69 @@ describe("remote Automatic Learning delivery", () => {
expect(remote.inputs).toHaveLength(1);
});
+ /*
+ * A built-in Bot's loop (the AI SDK) hands a throwing tool's message back to the model as that
+ * call's result and runs the next step, so a model that asks for a skill by a name the snapshot
+ * lacks reads "Skill is unavailable." and carries on. A remote Bot has to meet the same answer,
+ * not a failed turn.
+ */
+ test.each([
+ ["a skill the snapshot lacks", '{"skill_name":"refunds"}'],
+ ["arguments that are not JSON", '{"skill_name":'],
+ ])(
+ "a read the snapshot cannot answer goes back to the model: %s",
+ async (_case, args) => {
+ const remote = new Remote((_request, index) =>
+ of(
+ start,
+ ...(index === 0
+ ? [
+ {
+ type: EventType.TOOL_CALL_START,
+ toolCallId: "call",
+ toolCallName: load.name,
+ parentMessageId: "assistant",
+ },
+ {
+ type: EventType.TOOL_CALL_ARGS,
+ toolCallId: "call",
+ delta: args,
+ },
+ { type: EventType.TOOL_CALL_END, toolCallId: "call" },
+ ]
+ : []),
+ finish,
+ ),
+ );
+ const invocation: LearnedSkillInvocation = {
+ ...snapshot(),
+ execute: async (_name, parsed) => {
+ if ((parsed as { skill_name?: unknown }).skill_name !== "refund")
+ throw new Error("Skill is unavailable.");
+ return "refund steps";
+ },
+ };
+ const output = await events(
+ new RemoteLearnedSkillsMiddleware("bot", async () => invocation),
+ remote,
+ );
+ expect(output.some((event) => event.type === EventType.RUN_ERROR)).toBe(
+ false,
+ );
+ expect(output.at(-1)?.type).toBe(EventType.RUN_FINISHED);
+ // The model is asked again, with the reason on the call it made.
+ expect(remote.inputs).toHaveLength(2);
+ const answer = remote.inputs[1]?.messages.find(
+ (message) => message.role === "tool" && message.toolCallId === "call",
+ );
+ expect(answer?.content).toBeString();
+ expect(answer?.content).not.toBe("");
+ expect(
+ output.filter((event) => event.type === EventType.TOOL_CALL_RESULT),
+ ).toMatchObject([{ toolCallId: "call", content: answer?.content }]);
+ },
+ );
+
test("rejects reserved tool collisions before contacting the remote", async () => {
const remote = new Remote(() => of(start, finish));
const output = await events(