The Open Framework for Building Agents
fix(eve): run listed tools called through eve__tool (#4562)
Andrew Barbaafe7058a3f57
diff --git a/.changeset/eve-tool-listed-tools.md b/.changeset/eve-tool-listed-tools.mdnew file mode 100644index 000000000..a9420a6e0--- /dev/null+++ b/.changeset/eve-tool-listed-tools.md@@ -0,0 +1,5 @@+---+"eve": patch+---++`eve__tool` now runs a tool from the agent's tool list, other than eve's own catalog and task tools and tools a provider or client runs, instead of returning an error, so a model that routes a listed tool through it no longer wastes a step.diff --git a/docs/concepts/built-in-tools.md b/docs/concepts/built-in-tools.mdindex 93a1cfc90..93034697f 100644--- a/docs/concepts/built-in-tools.md+++ b/docs/concepts/built-in-tools.md@@ -277,7 +277,7 @@ The decision depends only on what the agent declares, never on what its resolver- A query whose first word contains `__` searches one namespace: everything before its last `__`, without trailing underscores. The namespace only filters. `eve__search` keeps names under `<namespace>__` and anything named exactly the namespace, such as a connection's sign-in entry, then ranks them by the whole query, so `eve__search({ query: "linear__" })` returns only the `linear` connection's tools, or its sign-in result, and an exact name that contains `__` still ranks first. eve lists only the connections that can own names in that namespace, so other connections make no network call and don't appear in `unavailable`. A namespace that matches nothing fails with the closest connection names. Characters that can't appear in a name, such as a leading `^`, are ignored; `query` isn't a regular expression.- `eve__search` never asks the user to sign in. A connection whose server won't list its tools until the user signs in appears as one result named after the connection, such as `linear`; a connection whose server lists its tools without a token is searched like any other. A connection whose tools fail to load, or don't load within 10 seconds, appears under `unavailable` with its `error`, such as `"crm" did not list its tools within 10s. Try again later.` Matches from everything else are still returned, so a slow server doesn't hold up the search. An `eve__tool` call on that connection's tools fails with the same error.- `eve__tool({ name: "linear" })`, with a connection's own name, asks the user to sign in to that connection when its tools need it, and returns once they can be listed, so the next `eve__search` finds them. On a server that lists its tools without sign-in, it confirms the tools are available without asking.-- `eve__tool({ name, input? })` calls the entry named `name` with `input`, which defaults to `{}`. A connection tool's name is `<connection>__<tool>`, such as `linear__list_issues`. eve checks `input` against the entry's input schema. An invalid input fails with the entry's signature, an unknown name fails with the closest names, a skill's name fails with a reminder to load it with `eve__skill`, and a tool already in the model's tool list fails with a reminder to call it directly. When the server asks for the user's authorization, the call asks the user to sign in and parks until sign-in completes.+- `eve__tool({ name, input? })` calls the entry named `name` with `input`, which defaults to `{}`. A connection tool's name is `<connection>__<tool>`, such as `linear__list_issues`. eve checks `input` against the entry's input schema. An invalid input fails with the entry's signature, an unknown name fails with the closest names, and a skill's name fails with a reminder to load it with `eve__skill`. When the server asks for the user's authorization, the call asks the user to sign in and parks until sign-in completes.- `eve__skill({ name })` loads the named skill, deferred or not, and returns its instructions. An unknown name fails with the closest skill names, a tool's name fails with a reminder to call it with `eve__tool` or directly, and a name that matches a connection says to find its tools with `eve__search({ query: "<connection>__" })`.After `eve__tool` or `eve__skill` resolves its entry, the call runs exactly like a direct call to that entry. A skill load reports a `load-skill` action named for its skill and an `execute_tool eve:load-skill` span. Approval policies and `approvedTools`, workflow tools and tasks, agents, `endsTurn`, `toModelOutput`, hooks, and stream events all see the entry's own name and input, such as `linear__list_issues`. Only model history records the call as `eve__tool` or `eve__skill`.diff --git a/packages/eve/src/execution/catalog/step-catalog.integration.test.ts b/packages/eve/src/execution/catalog/step-catalog.integration.test.tsindex afbf48537..05aca50d9 100644--- a/packages/eve/src/execution/catalog/step-catalog.integration.test.ts+++ b/packages/eve/src/execution/catalog/step-catalog.integration.test.ts@@ -627,6 +627,85 @@ describe("step catalog in the harness (real AI SDK)", () => {]);});+ it("runs a listed tool named through eve__tool exactly as a direct call", async () => {+ const ctx = createSessionContext();+ ctx.set(BundleKey, catalogBundle());+ const driver = createDriver(+ ctx,+ toolMap(+ inlineTool("lookup_order", {+ schema: {+ type: "object",+ properties: { orderId: { type: "string" } },+ required: ["orderId"],+ },+ }),+ inlineTool("archive_account", { approval: always() }),+ workflowTool("deploy_service"),+ // eve's built-in tools and static agents are marked as framework tools.+ inlineTool("bash", { frameworkTool: true }),+ subagentTool("billing_specialist", { frameworkTool: true }),+ inlineTool("refund_invoice", { deferred: true }),+ ),+ );++ driver.reply(+ calls(+ call("lookup", CALL_TOOL_NAME, { input: { orderId: "o_1" }, name: "lookup_order" }),+ call("shell", CALL_TOOL_NAME, { name: "bash" }),+ ),+ text("Found order o_1."),+ );+ await driver.drive({ message: "Alice asks about order o_1." });+ expect(toolResult(driver.requests()[1]!, "lookup")).toEqual({+ input: { orderId: "o_1" },+ ran: "lookup_order",+ });+ expect(toolResult(driver.requests()[1]!, "shell")).toEqual({ input: {}, ran: "bash" });+ expect(+ driver.events.flatMap((event) =>+ event.type === "actions.requested" ? event.data.actions : [],+ ),+ ).toContainEqual(+ expect.objectContaining({+ callId: "lookup",+ input: { orderId: "o_1" },+ toolName: "lookup_order",+ }),+ );++ // An approval is asked for, and the call runs, under the listed tool's own name.+ driver.reply(calls(call("archive", CALL_TOOL_NAME, { name: "archive_account" })));+ const parked = await driver.drive({ message: "Alice asks to archive Bob's account." });+ const [approval] = parkedSteps(parked.session).flatMap((step) => step.requests);+ expect(approval?.action).toEqual(+ expect.objectContaining({ callId: "archive", toolName: "archive_account" }),+ );+ driver.reply(text("Archived Bob's account."));+ await driver.drive({+ inputResponses: [{ optionId: "approve", requestId: approval!.requestId }],+ });+ expect(toolResult(driver.requests().at(-1)!, "archive")).toEqual({+ input: {},+ ran: "archive_account",+ });++ // A listed workflow tool and static agent are dispatched after the step, as direct calls are.+ driver.reply(+ calls(+ call("deploy", CALL_TOOL_NAME, { input: { service: "api" }, name: "deploy_service" }),+ call("delegate", CALL_TOOL_NAME, {+ input: { message: "Review Bob's dispute." },+ name: "billing_specialist",+ }),+ ),+ );+ const deploying = await driver.drive({ message: "Alice asks to deploy and review a dispute." });+ expect(+ parkedSteps(deploying.session).flatMap((step) => step.tasks.map((task) => task.toolName)),+ ).toEqual(["deploy_service", "billing_specialist"]);+ });+it("names the entry, not eve__tool, when a deferred entry returns a result that isn't JSON", async () => {const logs = captureLogRecords();const ctx = createSessionContext();diff --git a/packages/eve/src/execution/catalog/step-catalog.test.ts b/packages/eve/src/execution/catalog/step-catalog.test.tsindex c35c68e81..52f6cfacf 100644--- a/packages/eve/src/execution/catalog/step-catalog.test.ts+++ b/packages/eve/src/execution/catalog/step-catalog.test.ts@@ -307,10 +307,12 @@ describe("buildStepCatalog", () => {).toBe("# PDF forms");});- it("resolves a direct call only to a listed tool", () => {+ it("resolves a direct call only to a listed tool, and eve__tool to a listed tool too", () => {expect(catalog.resolve({ input: {}, toolName: "add" })?.definition).toBe(add);expect(catalog.resolve({ input: {}, toolName: "refund_invoice" })).toBeUndefined();- expect(catalog.resolve({ input: { name: "add" }, toolName: CALL_TOOL_NAME })).toBeUndefined();+ expect(+ catalog.resolve({ input: { input: { a: 1 }, name: "add" }, toolName: CALL_TOOL_NAME }),+ ).toEqual({ call: { input: { a: 1 }, toolName: "add" }, definition: add });expect(catalog.resolve({ input: { name: "refund_invoice" }, toolName: SKILL_TOOL_NAME }),).toBeUndefined();@@ -338,9 +340,29 @@ describe("buildStepCatalog", () => {}),inlineTool("refund_payment", { deferred: true }),inlineTool("stripe_list_disputes", { deferred: true }),+ workflowTool("research", "task"),+ // eve's built-in tools and static agents are marked as framework tools.+ inlineTool("bash", { frameworkTool: true }),+ subagentTool("billing_specialist", { frameworkTool: true }),+ {+ ...inlineTool("web_search"),+ behavior: { availability: [], handling: { kind: "provider-tool", provider: "parallel" } },+ execute: undefined,+ },],});+ it.each([+ ["an authored tool", "add", {}],+ ["a built-in tool", "bash", {}],+ ["a static agent", "billing_specialist", { message: "Review Bob's dispute." }],+ ])(+ "eve__tool accepts %s in the model's tool list, with its own input",+ async (_case, name, input) => {+ expect(await validateTool(catalog, { input, name })).toEqual({ value: { input, name } });+ },+ );+it.each([["an unknown tool, with the closest names",@@ -352,7 +374,21 @@ describe("buildStepCatalog", () => {"weather",'No tool named "weather". Find tools with eve__search.',],- ["a tool in the model's tool list", "add", '"add" is in your tool list; call it directly.'],+ [+ "eve__search",+ SEARCH_TOOL_NAME,+ `"${SEARCH_TOOL_NAME}" is in your tool list; call it directly.`,+ ],+ [+ "a task tool",+ TASK_WAIT_TOOL_NAME,+ `"${TASK_WAIT_TOOL_NAME}" is in your tool list; call it directly.`,+ ],+ [+ "a tool its provider runs",+ "web_search",+ '"web_search" is in your tool list; call it directly.',+ ],["eve__tool itself",CALL_TOOL_NAME,diff --git a/packages/eve/src/execution/catalog/step-catalog.ts b/packages/eve/src/execution/catalog/step-catalog.tsindex ebb3205f8..e3f54e679 100644--- a/packages/eve/src/execution/catalog/step-catalog.ts+++ b/packages/eve/src/execution/catalog/step-catalog.ts@@ -31,6 +31,7 @@ import {SKILL_ENTRY_NAME,SKILL_TOOL_NAME,} from "#protocol/catalog-tools.js";+import { eveNamespaceReservation } from "#protocol/runtime-tools.js";import type { ConnectionRegistry } from "#runtime/connections/registry-types.js";import { BundleKey, type CompiledBundle } from "#runtime/sessions/runtime-context-keys.js";import type { ResolvedConnectionDefinition } from "#runtime/types.js";@@ -126,9 +127,10 @@ export interface StepCatalog extends HarnessToolLookup {describe(definition: HarnessToolDefinition): string;/*** Resolves a model tool call to the entry it runs. An `eve__tool` call that- * names a deferred entry becomes the call to that entry, and an `eve__skill`- * call becomes the call to load its skill; any other call runs the listed- * tool it names. A call that reaches none of these resolves to nothing.+ * names an entry `callableByName` accepts becomes the call to that entry, and+ * an `eve__skill` call becomes the call to load its skill; any other call runs+ * the listed tool it names. A call that reaches none of these resolves to+ * nothing.*/readonly resolve: CallResolver;}@@ -187,7 +189,9 @@ export function buildStepCatalog(input: {if (toolCall.toolName === CALL_TOOL_NAME && catalogTools.tools) {const name = targetName(toolCall.input);const definition = name === undefined ? undefined : toolEntry(name);- if (name === undefined || definition?.deferred !== true) return undefined;+ if (name === undefined || definition === undefined || !callableByName(definition)) {+ return undefined;+ }return { call: asEntryCall(toolCall, name), definition };}if (toolCall.toolName === SKILL_TOOL_NAME && catalogTools.skills) {@@ -297,8 +301,7 @@ function connectionEntryNamed(/*** The model sees one fixed schema. Validation resolves the named entry and* checks `input` against that entry's own schema, so a call that reaches- * `eve__tool` always names a deferred entry with valid input, as a direct call- * names a listed tool.+ * `eve__tool` always names an entry `callableByName` accepts, with valid input.*/function createCallTool(catalog: StepCatalog,@@ -379,6 +382,21 @@ function runResolved(return runEntryCall(resolved, options);}+/**+ * Whether `eve__tool` runs `definition`: a deferred entry, or a listed tool eve+ * runs itself. A model that routes a listed tool through `eve__tool` then gets+ * the same result as a direct call instead of a wasted step; nothing it reads+ * says this works. The tools in eve's reserved namespace, and tools a provider+ * or client runs, stay direct-only.+ */+function callableByName(definition: HarnessToolDefinition): boolean {+ return (+ definition.deferred === true ||+ (eveNamespaceReservation(definition.name) === undefined &&+ (definition.execute !== undefined || definition.workflowId !== undefined))+ );+}+interface ToolInput {readonly input?: unknown;readonly name: string;@@ -391,7 +409,7 @@ async function resolveToolInput(): Promise<StandardSchemaV1.Result<ToolInput>> {const definition = toolEntry(name);if (definition === undefined) return failure("name", unknownEntryMessage(name, catalog));- if (definition.deferred !== true) {+ if (!callableByName(definition)) {return failure("name", `"${name}" is in your tool list; call it directly.`);}const checked = await checkToolCallInput(definition, input, "");diff --git a/research/deferred-tools.md b/research/deferred-tools.mdindex a515fd48c..59cd6f1d4 100644--- a/research/deferred-tools.md+++ b/research/deferred-tools.md@@ -621,7 +621,8 @@ today.as data the model can act on:- An unknown name lists the closest tool names.- A skill's name says to load it with `eve__skill`.- - A direct tool returns `"<name>" is in your tool list; call it directly.`+ - eve's own catalog and task tools, and tools a provider or client runs,+ return `"<name>" is in your tool list; call it directly.`- Invalid input returns the entry's `signature`.- **Sign-in.** A connection's own name is an entry.`eve__tool({ name: "linear" })` finishes a pending sign-in or starts one, the@@ -1038,6 +1039,7 @@ Everything in this section lands in [3/3].| Search output | TypeScript `signature` | Raw JSON Schema (larger, and code mode needs the signature anyway) || Protocol shape | Actions carry the entry's name; only history says `eve__tool` or `eve__skill` | An outer catalog-tool action with a nested entry action || Search state | Stateless; any entry can be executed | A durable discovered set that execution checks |+| Listed tools | `eve__tool` runs them as a direct call would, unannounced; eve's own and provider-run tools still error | An error saying to call the tool directly (costs a step) || Listing | One append-only listing: the kinds present, capped namespaces, and capped connections | Nothing (the model can't tell when to search); every deferred name (defeats deferring and grows without bound); counts (churn on dynamic changes) || Old paths and tests | Deleted with no compatibility layer; old tests deleted, and new tests written fresh in [3/3] | Aliases or flags for removed tools; porting old tests to the new surface; tests spread across the implementation PRs || Loading skills | `eve__skill({ name })` for every skill, registered with any skill; `load_skill` is removed | Keeping `load_skill` beside `eve__skill` (two ways to load); `load_skill` only for skills that aren't deferred (the model must know which kind it has) |