Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Report a surface-owned tool call to the surface (#77)

Rahul Sethuramb24fdfa3082a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 285a1411..b448227a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -221,6 +221,13 @@ Sessions survive and nobody signs in again.
laptop `http://localhost` counts as one, so this never showed up in development; on a real
address it does not, and the surface did nothing at all when you pressed send. No message, no
error. Ids now come from an API with no such restriction.
+- **A framework Bot asked for a browser action and nothing happened.** `agent-langgraph` ends a run
+ when the model calls a tool the surface owns, which is how a tool that lives in the browser is
+ supposed to work: the run finishes, the surface acts, and the next run carries the result. But the
+ call was only reported to the surface from the node that executes this deployment's own tools, and
+ that node is exactly what an ending run skips. The person saw their own message, no answer under
+ it, and no explanation, because a run that finishes carrying nothing is not an error. Every Bot
+ action in the browser was affected: opening a page, filling a form, asking for help at a sign-in.
### Changed
diff --git a/agent-langgraph/src/index.ts b/agent-langgraph/src/index.ts
index c524fb0b..158fdc7d 100644
--- a/agent-langgraph/src/index.ts
+++ b/agent-langgraph/src/index.ts
@@ -436,7 +436,6 @@ async function runAgent(input: RunAgentInput): Promise<Response> {
// Accumulated rather than emitted per chunk, because a tool call's arguments arrive in
// fragments and AG-UI wants one call. The framework hands back assembled `tool_calls` on the
// final message, which is precisely the plumbing agent-bot does by hand.
- let finalMessage: AIMessage | null = null;
/** Calls seen on the way past, so a result can be paired with the arguments it answered. */
const pending = new Map<
string,
@@ -470,7 +469,6 @@ async function runAgent(input: RunAgentInput): Promise<Response> {
if (event.event === "on_chat_model_end") {
const output = event.data?.output as AIMessage | undefined;
if (output) {
- finalMessage = output;
for (const call of output.tool_calls ?? []) {
pending.set(call.id ?? call.name, {
name: call.name,
@@ -520,6 +518,31 @@ async function runAgent(input: RunAgentInput): Promise<Response> {
closeText();
+ /*
+ * Calls this process did not run, which is what a tool the surface owns looks like from here.
+ *
+ * The graph ends the run on one of those rather than inventing a result, so the `tools` node
+ * never fires and the loop above never reports the call. Without this the run is a clean
+ * RUN_STARTED/RUN_FINISHED pair carrying nothing at all: the person's message sits there with
+ * no answer under it, the surface never learns there was a browser action to execute, and
+ * because an empty run is not an error by the protocol, nothing says so. No result is sent
+ * with them; producing it is the surface's half, and it begins the next run holding it.
+ */
+ for (const [id, call] of pending) {
+ send({
+ type: "TOOL_CALL_START",
+ toolCallId: id,
+ toolCallName: call.name,
+ } as BaseEvent);
+ send({
+ type: "TOOL_CALL_ARGS",
+ toolCallId: id,
+ delta: JSON.stringify(call.args),
+ } as BaseEvent);
+ send({ type: "TOOL_CALL_END", toolCallId: id } as BaseEvent);
+ }
+ pending.clear();
+
send({
type: "RUN_FINISHED",
threadId: input.threadId,