Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

fix(eve): run listed tools called through eve__tool (#4562)

Andrew Barbaafe7058a3f57
diff --git a/.changeset/eve-tool-listed-tools.md b/.changeset/eve-tool-listed-tools.md
new file mode 100644
index 000000000..a9420a6e0
--- /dev/null
+++ b/.changeset/eve-tool-listed-tools.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+`eve__tool` now runs a tool from the agent's tool list, other than eve's own catalog and task tools and tools a provider or client runs, instead of returning an error, so a model that routes a listed tool through it no longer wastes a step.
diff --git a/docs/concepts/built-in-tools.md b/docs/concepts/built-in-tools.md
index 93a1cfc90..93034697f 100644
--- a/docs/concepts/built-in-tools.md
+++ b/docs/concepts/built-in-tools.md
@@ -277,7 +277,7 @@ The decision depends only on what the agent declares, never on what its resolver
- A query whose first word contains `__` searches one namespace: everything before its last `__`, without trailing underscores. The namespace only filters. `eve__search` keeps names under `<namespace>__` and anything named exactly the namespace, such as a connection's sign-in entry, then ranks them by the whole query, so `eve__search({ query: "linear__" })` returns only the `linear` connection's tools, or its sign-in result, and an exact name that contains `__` still ranks first. eve lists only the connections that can own names in that namespace, so other connections make no network call and don't appear in `unavailable`. A namespace that matches nothing fails with the closest connection names. Characters that can't appear in a name, such as a leading `^`, are ignored; `query` isn't a regular expression.
- `eve__search` never asks the user to sign in. A connection whose server won't list its tools until the user signs in appears as one result named after the connection, such as `linear`; a connection whose server lists its tools without a token is searched like any other. A connection whose tools fail to load, or don't load within 10 seconds, appears under `unavailable` with its `error`, such as `"crm" did not list its tools within 10s. Try again later.` Matches from everything else are still returned, so a slow server doesn't hold up the search. An `eve__tool` call on that connection's tools fails with the same error.
- `eve__tool({ name: "linear" })`, with a connection's own name, asks the user to sign in to that connection when its tools need it, and returns once they can be listed, so the next `eve__search` finds them. On a server that lists its tools without sign-in, it confirms the tools are available without asking.
-- `eve__tool({ name, input? })` calls the entry named `name` with `input`, which defaults to `{}`. A connection tool's name is `<connection>__<tool>`, such as `linear__list_issues`. eve checks `input` against the entry's input schema. An invalid input fails with the entry's signature, an unknown name fails with the closest names, a skill's name fails with a reminder to load it with `eve__skill`, and a tool already in the model's tool list fails with a reminder to call it directly. When the server asks for the user's authorization, the call asks the user to sign in and parks until sign-in completes.
+- `eve__tool({ name, input? })` calls the entry named `name` with `input`, which defaults to `{}`. A connection tool's name is `<connection>__<tool>`, such as `linear__list_issues`. eve checks `input` against the entry's input schema. An invalid input fails with the entry's signature, an unknown name fails with the closest names, and a skill's name fails with a reminder to load it with `eve__skill`. When the server asks for the user's authorization, the call asks the user to sign in and parks until sign-in completes.
- `eve__skill({ name })` loads the named skill, deferred or not, and returns its instructions. An unknown name fails with the closest skill names, a tool's name fails with a reminder to call it with `eve__tool` or directly, and a name that matches a connection says to find its tools with `eve__search({ query: "<connection>__" })`.
After `eve__tool` or `eve__skill` resolves its entry, the call runs exactly like a direct call to that entry. A skill load reports a `load-skill` action named for its skill and an `execute_tool eve:load-skill` span. Approval policies and `approvedTools`, workflow tools and tasks, agents, `endsTurn`, `toModelOutput`, hooks, and stream events all see the entry's own name and input, such as `linear__list_issues`. Only model history records the call as `eve__tool` or `eve__skill`.
diff --git a/packages/eve/src/execution/catalog/step-catalog.integration.test.ts b/packages/eve/src/execution/catalog/step-catalog.integration.test.ts
index afbf48537..05aca50d9 100644
--- a/packages/eve/src/execution/catalog/step-catalog.integration.test.ts
+++ b/packages/eve/src/execution/catalog/step-catalog.integration.test.ts
@@ -627,6 +627,85 @@ describe("step catalog in the harness (real AI SDK)", () => {
]);
});
+ it("runs a listed tool named through eve__tool exactly as a direct call", async () => {
+ const ctx = createSessionContext();
+ ctx.set(BundleKey, catalogBundle());
+ const driver = createDriver(
+ ctx,
+ toolMap(
+ inlineTool("lookup_order", {
+ schema: {
+ type: "object",
+ properties: { orderId: { type: "string" } },
+ required: ["orderId"],
+ },
+ }),
+ inlineTool("archive_account", { approval: always() }),
+ workflowTool("deploy_service"),
+ // eve's built-in tools and static agents are marked as framework tools.
+ inlineTool("bash", { frameworkTool: true }),
+ subagentTool("billing_specialist", { frameworkTool: true }),
+ inlineTool("refund_invoice", { deferred: true }),
+ ),
+ );
+
+ driver.reply(
+ calls(
+ call("lookup", CALL_TOOL_NAME, { input: { orderId: "o_1" }, name: "lookup_order" }),
+ call("shell", CALL_TOOL_NAME, { name: "bash" }),
+ ),
+ text("Found order o_1."),
+ );
+ await driver.drive({ message: "Alice asks about order o_1." });
+ expect(toolResult(driver.requests()[1]!, "lookup")).toEqual({
+ input: { orderId: "o_1" },
+ ran: "lookup_order",
+ });
+ expect(toolResult(driver.requests()[1]!, "shell")).toEqual({ input: {}, ran: "bash" });
+ expect(
+ driver.events.flatMap((event) =>
+ event.type === "actions.requested" ? event.data.actions : [],
+ ),
+ ).toContainEqual(
+ expect.objectContaining({
+ callId: "lookup",
+ input: { orderId: "o_1" },
+ toolName: "lookup_order",
+ }),
+ );
+
+ // An approval is asked for, and the call runs, under the listed tool's own name.
+ driver.reply(calls(call("archive", CALL_TOOL_NAME, { name: "archive_account" })));
+ const parked = await driver.drive({ message: "Alice asks to archive Bob's account." });
+ const [approval] = parkedSteps(parked.session).flatMap((step) => step.requests);
+ expect(approval?.action).toEqual(
+ expect.objectContaining({ callId: "archive", toolName: "archive_account" }),
+ );
+ driver.reply(text("Archived Bob's account."));
+ await driver.drive({
+ inputResponses: [{ optionId: "approve", requestId: approval!.requestId }],
+ });
+ expect(toolResult(driver.requests().at(-1)!, "archive")).toEqual({
+ input: {},
+ ran: "archive_account",
+ });
+
+ // A listed workflow tool and static agent are dispatched after the step, as direct calls are.
+ driver.reply(
+ calls(
+ call("deploy", CALL_TOOL_NAME, { input: { service: "api" }, name: "deploy_service" }),
+ call("delegate", CALL_TOOL_NAME, {
+ input: { message: "Review Bob's dispute." },
+ name: "billing_specialist",
+ }),
+ ),
+ );
+ const deploying = await driver.drive({ message: "Alice asks to deploy and review a dispute." });
+ expect(
+ parkedSteps(deploying.session).flatMap((step) => step.tasks.map((task) => task.toolName)),
+ ).toEqual(["deploy_service", "billing_specialist"]);
+ });
+
it("names the entry, not eve__tool, when a deferred entry returns a result that isn't JSON", async () => {
const logs = captureLogRecords();
const ctx = createSessionContext();
diff --git a/packages/eve/src/execution/catalog/step-catalog.test.ts b/packages/eve/src/execution/catalog/step-catalog.test.ts
index c35c68e81..52f6cfacf 100644
--- a/packages/eve/src/execution/catalog/step-catalog.test.ts
+++ b/packages/eve/src/execution/catalog/step-catalog.test.ts
@@ -307,10 +307,12 @@ describe("buildStepCatalog", () => {
).toBe("# PDF forms");
});
- it("resolves a direct call only to a listed tool", () => {
+ it("resolves a direct call only to a listed tool, and eve__tool to a listed tool too", () => {
expect(catalog.resolve({ input: {}, toolName: "add" })?.definition).toBe(add);
expect(catalog.resolve({ input: {}, toolName: "refund_invoice" })).toBeUndefined();
- expect(catalog.resolve({ input: { name: "add" }, toolName: CALL_TOOL_NAME })).toBeUndefined();
+ expect(
+ catalog.resolve({ input: { input: { a: 1 }, name: "add" }, toolName: CALL_TOOL_NAME }),
+ ).toEqual({ call: { input: { a: 1 }, toolName: "add" }, definition: add });
expect(
catalog.resolve({ input: { name: "refund_invoice" }, toolName: SKILL_TOOL_NAME }),
).toBeUndefined();
@@ -338,9 +340,29 @@ describe("buildStepCatalog", () => {
}),
inlineTool("refund_payment", { deferred: true }),
inlineTool("stripe_list_disputes", { deferred: true }),
+ workflowTool("research", "task"),
+ // eve's built-in tools and static agents are marked as framework tools.
+ inlineTool("bash", { frameworkTool: true }),
+ subagentTool("billing_specialist", { frameworkTool: true }),
+ {
+ ...inlineTool("web_search"),
+ behavior: { availability: [], handling: { kind: "provider-tool", provider: "parallel" } },
+ execute: undefined,
+ },
],
});
+ it.each([
+ ["an authored tool", "add", {}],
+ ["a built-in tool", "bash", {}],
+ ["a static agent", "billing_specialist", { message: "Review Bob's dispute." }],
+ ])(
+ "eve__tool accepts %s in the model's tool list, with its own input",
+ async (_case, name, input) => {
+ expect(await validateTool(catalog, { input, name })).toEqual({ value: { input, name } });
+ },
+ );
+
it.each([
[
"an unknown tool, with the closest names",
@@ -352,7 +374,21 @@ describe("buildStepCatalog", () => {
"weather",
'No tool named "weather". Find tools with eve__search.',
],
- ["a tool in the model's tool list", "add", '"add" is in your tool list; call it directly.'],
+ [
+ "eve__search",
+ SEARCH_TOOL_NAME,
+ `"${SEARCH_TOOL_NAME}" is in your tool list; call it directly.`,
+ ],
+ [
+ "a task tool",
+ TASK_WAIT_TOOL_NAME,
+ `"${TASK_WAIT_TOOL_NAME}" is in your tool list; call it directly.`,
+ ],
+ [
+ "a tool its provider runs",
+ "web_search",
+ '"web_search" is in your tool list; call it directly.',
+ ],
[
"eve__tool itself",
CALL_TOOL_NAME,
diff --git a/packages/eve/src/execution/catalog/step-catalog.ts b/packages/eve/src/execution/catalog/step-catalog.ts
index ebb3205f8..e3f54e679 100644
--- a/packages/eve/src/execution/catalog/step-catalog.ts
+++ b/packages/eve/src/execution/catalog/step-catalog.ts
@@ -31,6 +31,7 @@ import {
SKILL_ENTRY_NAME,
SKILL_TOOL_NAME,
} from "#protocol/catalog-tools.js";
+import { eveNamespaceReservation } from "#protocol/runtime-tools.js";
import type { ConnectionRegistry } from "#runtime/connections/registry-types.js";
import { BundleKey, type CompiledBundle } from "#runtime/sessions/runtime-context-keys.js";
import type { ResolvedConnectionDefinition } from "#runtime/types.js";
@@ -126,9 +127,10 @@ export interface StepCatalog extends HarnessToolLookup {
describe(definition: HarnessToolDefinition): string;
/**
* Resolves a model tool call to the entry it runs. An `eve__tool` call that
- * names a deferred entry becomes the call to that entry, and an `eve__skill`
- * call becomes the call to load its skill; any other call runs the listed
- * tool it names. A call that reaches none of these resolves to nothing.
+ * names an entry `callableByName` accepts becomes the call to that entry, and
+ * an `eve__skill` call becomes the call to load its skill; any other call runs
+ * the listed tool it names. A call that reaches none of these resolves to
+ * nothing.
*/
readonly resolve: CallResolver;
}
@@ -187,7 +189,9 @@ export function buildStepCatalog(input: {
if (toolCall.toolName === CALL_TOOL_NAME && catalogTools.tools) {
const name = targetName(toolCall.input);
const definition = name === undefined ? undefined : toolEntry(name);
- if (name === undefined || definition?.deferred !== true) return undefined;
+ if (name === undefined || definition === undefined || !callableByName(definition)) {
+ return undefined;
+ }
return { call: asEntryCall(toolCall, name), definition };
}
if (toolCall.toolName === SKILL_TOOL_NAME && catalogTools.skills) {
@@ -297,8 +301,7 @@ function connectionEntryNamed(
/**
* The model sees one fixed schema. Validation resolves the named entry and
* checks `input` against that entry's own schema, so a call that reaches
- * `eve__tool` always names a deferred entry with valid input, as a direct call
- * names a listed tool.
+ * `eve__tool` always names an entry `callableByName` accepts, with valid input.
*/
function createCallTool(
catalog: StepCatalog,
@@ -379,6 +382,21 @@ function runResolved(
return runEntryCall(resolved, options);
}
+/**
+ * Whether `eve__tool` runs `definition`: a deferred entry, or a listed tool eve
+ * runs itself. A model that routes a listed tool through `eve__tool` then gets
+ * the same result as a direct call instead of a wasted step; nothing it reads
+ * says this works. The tools in eve's reserved namespace, and tools a provider
+ * or client runs, stay direct-only.
+ */
+function callableByName(definition: HarnessToolDefinition): boolean {
+ return (
+ definition.deferred === true ||
+ (eveNamespaceReservation(definition.name) === undefined &&
+ (definition.execute !== undefined || definition.workflowId !== undefined))
+ );
+}
+
interface ToolInput {
readonly input?: unknown;
readonly name: string;
@@ -391,7 +409,7 @@ async function resolveToolInput(
): Promise<StandardSchemaV1.Result<ToolInput>> {
const definition = toolEntry(name);
if (definition === undefined) return failure("name", unknownEntryMessage(name, catalog));
- if (definition.deferred !== true) {
+ if (!callableByName(definition)) {
return failure("name", `"${name}" is in your tool list; call it directly.`);
}
const checked = await checkToolCallInput(definition, input, "");
diff --git a/research/deferred-tools.md b/research/deferred-tools.md
index a515fd48c..59cd6f1d4 100644
--- a/research/deferred-tools.md
+++ b/research/deferred-tools.md
@@ -621,7 +621,8 @@ today.
as data the model can act on:
- An unknown name lists the closest tool names.
- A skill's name says to load it with `eve__skill`.
- - A direct tool returns `"<name>" is in your tool list; call it directly.`
+ - eve's own catalog and task tools, and tools a provider or client runs,
+ return `"<name>" is in your tool list; call it directly.`
- Invalid input returns the entry's `signature`.
- **Sign-in.** A connection's own name is an entry.
`eve__tool({ name: "linear" })` finishes a pending sign-in or starts one, the
@@ -1038,6 +1039,7 @@ Everything in this section lands in [3/3].
| Search output | TypeScript `signature` | Raw JSON Schema (larger, and code mode needs the signature anyway) |
| Protocol shape | Actions carry the entry's name; only history says `eve__tool` or `eve__skill` | An outer catalog-tool action with a nested entry action |
| Search state | Stateless; any entry can be executed | A durable discovered set that execution checks |
+| Listed tools | `eve__tool` runs them as a direct call would, unannounced; eve's own and provider-run tools still error | An error saying to call the tool directly (costs a step) |
| Listing | One append-only listing: the kinds present, capped namespaces, and capped connections | Nothing (the model can't tell when to search); every deferred name (defeats deferring and grows without bound); counts (churn on dynamic changes) |
| Old paths and tests | Deleted with no compatibility layer; old tests deleted, and new tests written fresh in [3/3] | Aliases or flags for removed tools; porting old tests to the new surface; tests spread across the implementation PRs |
| Loading skills | `eve__skill({ name })` for every skill, registered with any skill; `load_skill` is removed | Keeping `load_skill` beside `eve__skill` (two ways to load); `load_skill` only for skills that aren't deferred (the model must know which kind it has) |