The Open Framework for Building Agents
fix(eve): add catalog tools only when reached, and report search coverage (#4698)
Andrew Barbadb71190b2c3a
diff --git a/.changeset/catalog-tools-from-resolved-entries.md b/.changeset/catalog-tools-from-resolved-entries.mdnew file mode 100644index 000000000..191a23fa2--- /dev/null+++ b/.changeset/catalog-tools-from-resolved-entries.md@@ -0,0 +1,5 @@+---+"eve": patch+---++A dynamic resolver no longer adds the catalog tools (`eve__search`, `eve__tool`, and `eve__skill`) on its own. eve adds the catalog tools once a session reaches something they find, call, or load, such as a deferred entry a resolver returns, and keeps them for the rest of the session. Agents whose resolvers return only listed tools no longer see these tools or spend steps searching.diff --git a/.changeset/search-result-counts.md b/.changeset/search-result-counts.mdnew file mode 100644index 000000000..756c457a2--- /dev/null+++ b/.changeset/search-result-counts.md@@ -0,0 +1,5 @@+---+"eve": patch+---++`eve__search` results now include `searched`, the number of entries the query covered, and `remaining`, the number of matches past `limit`, so the model can tell after one search how much is left to find. Its description now tells the model to search only when nothing in its tool list fits the request.diff --git a/docs/concepts/built-in-tools.md b/docs/concepts/built-in-tools.mdindex dc1b32a63..dd9393d2f 100644--- a/docs/concepts/built-in-tools.md+++ b/docs/concepts/built-in-tools.md@@ -24,7 +24,7 @@ export default defineAgent({This turns off the optional defaults described below. Add back only the tools the agent needs with the command in each tool's section. Existing files under `agent/tools/` remain available, including same-name replacements such as `agent/tools/bash.ts`.-`defaultTools` doesn't affect [the catalog tools](#eve__search-eve__tool-and-eve__skill): eve adds them from what the agent declares, as listed in that section.+`defaultTools` doesn't affect [the catalog tools](#eve__search-eve__tool-and-eve__skill): eve adds them from what the session can reach, as listed in that section.### `bash`@@ -301,16 +301,16 @@ export default disableTool();### `eve__search`, `eve__tool`, and `eve__skill`-These tools let the model reach what isn't in its tool list. `eve__search` finds tools defined with `deferred: true`, agents defined with `tool: "deferred"`, every tool from the agent's [connections](../connections), and deferred [skills](../skills). `eve__tool` calls a deferred tool, agent, or connection tool by name, and `eve__skill` loads a skill, deferred or not. There is no add command; eve adds each one from what the agent declares, even when `defaultTools` is `false`:+These tools let the model reach what isn't in its tool list. `eve__search` finds tools defined with `deferred: true`, agents defined with `tool: "deferred"`, every tool from the agent's [connections](../connections), and deferred [skills](../skills). `eve__tool` calls a deferred tool, agent, or connection tool by name, and `eve__skill` loads a skill, deferred or not. There is no add command; eve adds each one when the session has something it reaches, even when `defaultTools` is `false`:-- `eve__tool` comes with anything it can call: a deferred tool or agent, a connection, or a dynamic resolver in `agent/tools/`, `agent/subagents/`, or `agent/connections/` that may add one at runtime.-- `eve__skill` comes with any skill, static or from a dynamic skill resolver, so an agent whose only catalog entries are listed skills gets `eve__skill` alone.-- `eve__search` comes with anything it could find: whatever `eve__tool` can call, plus deferred skills and dynamic skill resolvers.-- An agent with none of these gets no catalog tools.+- `eve__tool` comes with anything it can call: a deferred tool or agent, or a connection.+- `eve__skill` comes with any skill, so an agent whose only catalog entries are listed skills gets `eve__skill` alone.+- `eve__search` comes with anything it could find: whatever `eve__tool` can call, plus deferred skills.+- A session with none of these gets no catalog tools.-The decision depends only on what the agent declares, never on what its resolvers return, so the tool list stays the same for a deployment and never changes within a session.+Entries the agent declares count from the session's first step. An entry a dynamic resolver returns counts from the step where it first appears, so a resolver in `agent/tools/` that returns only tools without `deferred: true` adds no catalog tools. Once a session's tool list includes a catalog tool, the session keeps it, even after a resolver stops returning the entry that brought it. The catalog tools change only when the session first reaches a new kind of entry, such as its first deferred tool or its first connection.-- `eve__search({ query, limit? })` returns the best matches, up to `limit` (default 10, at most 50). `query` is required and must contain a word. A tool match has its exact `tool` name, its `description`, and a TypeScript `signature` rendered from its schemas. A deferred skill's match has its `skill` name, its `description`, and the `path` of its `SKILL.md` when it has supporting files. A result with `tool` is called with `eve__tool`, and one with `skill` is loaded with `eve__skill`. Tools and skills rank together, in tiers: an entry whose name is exactly the query, then one whose name after its last `__` is the query, such as `linear__create_issue` or `sre__create_issue` for `create_issue`, then the tools of a connection named by the query, then entries whose names, or names after their last `__`, start with the query, then keyword matches in names, parameters, and descriptions. So `eve__search({ query: "linear" })` lists the `linear` connection's tools first. One-letter and filler words such as `a` and `the` don't count toward keyword matches unless the query has nothing else.+- `eve__search({ query, limit? })` returns the best matches, up to `limit` (default 10, at most 50). `query` is required and must contain a word. A tool match has its exact `tool` name, its `description`, and a TypeScript `signature` rendered from its schemas. A deferred skill's match has its `skill` name, its `description`, and the `path` of its `SKILL.md` when it has supporting files. A result with `tool` is called with `eve__tool`, and one with `skill` is loaded with `eve__skill`. Next to `results`, `searched` counts the entries the query covered, the whole catalog or one namespace, and `remaining` counts the matches past `limit`. A connection behind sign-in counts as its one sign-in result until the user signs in, and an unavailable connection counts as none. Tools and skills rank together, in tiers: an entry whose name is exactly the query, then one whose name after its last `__` is the query, such as `linear__create_issue` or `sre__create_issue` for `create_issue`, then the tools of a connection named by the query, then entries whose names, or names after their last `__`, start with the query, then keyword matches in names, parameters, and descriptions. So `eve__search({ query: "linear" })` lists the `linear` connection's tools first. One-letter and filler words such as `a` and `the` don't count toward keyword matches unless the query has nothing else.- A query whose first word contains `__` searches one namespace: everything before its last `__`, without trailing underscores. The namespace only filters. `eve__search` keeps names under `<namespace>__` and anything named exactly the namespace, such as a connection's sign-in entry, then ranks them by the whole query, so `eve__search({ query: "linear__" })` returns only the `linear` connection's tools, or its sign-in result, and an exact name that contains `__` still ranks first. eve lists only the connections that can own names in that namespace, so other connections make no network call and don't appear in `unavailable`. A namespace that matches nothing fails with the closest connection names. Characters that can't appear in a name, such as a leading `^`, are ignored; `query` isn't a regular expression.- `eve__search` never asks the user to sign in. A connection whose server won't list its tools until the user signs in appears as one result named after the connection, such as `linear`; a connection whose server lists its tools without a token is searched like any other. A connection whose tools fail to load, or don't load within 10 seconds, appears under `unavailable` with its `error`, such as `"crm" did not list its tools within 10s. Try again later.` Matches from everything else are still returned, so a slow server doesn't hold up the search. An `eve__tool` call on that connection's tools fails with the same error.- `eve__tool({ name: "linear" })`, with a connection's own name, asks the user to sign in to that connection when its tools need it, and returns once they can be listed, so the next `eve__search` finds them. On a server that lists its tools without sign-in, it confirms the tools are available without asking.@@ -319,7 +319,7 @@ The decision depends only on what the agent declares, never on what its resolverAfter `eve__tool` or `eve__skill` resolves its entry, the call runs exactly like a direct call to that entry. A skill load reports a `load-skill` action and a `load-skill-result`, both with the skill's `name`, and an `execute_tool eve:load-skill` span. Approval policies and `approvedTools`, workflow tools and tasks, agents, `endsTurn`, `toModelOutput`, hooks, and stream events all see the entry's own name and input, such as `linear__list_issues`. Only model history records the call as `eve__tool` or `eve__skill`.-eve tells the model what it can reach in an append-only context message rather than in the system prompt: which kinds of deferred entries exist, up to 20 namespaces (the first `__` segment of deferred names, such as `sre` for `sre__list_alerts`), and up to 20 connections with their descriptions. It never names a deferred entry, so deferred entries stay out of context until `eve__search` finds them. A later step appends the listing again only when that changes; a deferred entry added to a listed namespace, or without one, changes nothing. The definitions of the catalog tools never change within a deployment, so adding a deferred entry, signing in, or resolving a dynamic connection keeps the cached prompt prefix.+eve tells the model what it can reach in an append-only context message rather than in the system prompt: which kinds of deferred entries exist, up to 20 namespaces (the first `__` segment of deferred names, such as `sre` for `sre__list_alerts`), and up to 20 connections with their descriptions. It never names a deferred entry, so deferred entries stay out of context until `eve__search` finds them. A later step appends the listing again only when that changes; a deferred entry added to a listed namespace, or without one, changes nothing. After the session first reaches each kind of entry, adding or removing entries, signing in, or resolving more dynamic connections doesn't change the catalog tools, so the cached prompt prefix holds.The tools eve adds itself all live in the `eve` namespace: `eve__search`, `eve__tool`, `eve__skill`, `eve__task_wait`, `eve__task_cancel`, and `eve__reply`. Nothing you author or resolve may be named `eve` or start with `eve__`: tools, subagents, skills, connections, and extension mounts, static or dynamic, since a connection or mount named `eve` would own every `eve__` name. The compiler rejects an authored tool, skill, connection, or mount, such as `agent/tools/eve__search.ts`; eve rejects an authored subagent when the agent loads, and a dynamic entry when its resolver returns it. Any other name is free, including `search` and `execute`. A connection's own name is also the entry that signs the user in to it.diff --git a/docs/guides/dynamic-capabilities.md b/docs/guides/dynamic-capabilities.mdindex e4d169dc8..7c71ad0ed 100644--- a/docs/guides/dynamic-capabilities.md+++ b/docs/guides/dynamic-capabilities.md@@ -195,7 +195,9 @@ arguments, and approval options as static [MCP](../connections/mcp) and[OpenAPI](../connections/openapi) connections. Each resolved connection joinsthe per-step connection registry. eve announces it to the model in anappend-only context message, and the model reaches its tools through-`eve__search` and `eve__tool`, so the tool list never changes.+`eve__search` and `eve__tool`. If the session doesn't have those two tools yet,+eve adds them when its first connection resolves and keeps them for the rest of+the session, so later connections don't change the tool list.Set `instanceKey` on every authenticated dynamic connection. Use a stable,non-secret account or tenant identifier, and change it whenever the endpoint,@@ -274,6 +276,8 @@ export default defineDynamic({});```+An entry with `deferred: true` stays out of the model's tool list, like a static [deferred tool](../tools#keep-a-tool-out-of-the-tool-list). The model finds it with `eve__search` and calls it with `eve__tool`. If the session doesn't have those two tools yet, eve adds them on the first step where a resolver returns a deferred entry, and keeps them for the rest of the session. A resolver whose entries are all listed adds neither, so the model doesn't spend steps searching for tools it already has.+### Author replayable callbacksWrite callback properties as inline function expressions, arrows, method shorthand, or module-level function references. eve transforms authored modules that import `defineTool`, including helper modules outside `agent/tools/`, and stores each callback's referenced closure values independently.diff --git a/packages/eve/src/context/keys.ts b/packages/eve/src/context/keys.tsindex 5229e3730..87089736b 100644--- a/packages/eve/src/context/keys.ts+++ b/packages/eve/src/context/keys.ts@@ -121,6 +121,24 @@ export interface HistoryState {readonly announcements?: Readonly<Record<string, string>>;}export const HistoryStateKey = new ContextKey<HistoryState>("eve.historyState");+/**+ * The catalog tools a session's tool list has offered. Each stays offered for+ * the rest of the session, so the tool list never drops a catalog tool that+ * earlier calls named. Compaction keeps it, unlike {@link HistoryStateKey}.+ */+export interface OfferedCatalogTools {+ /** `eve__tool` mentions connected services, and `eve__search` their sign-in entries. */+ readonly connections: boolean;+ /** `eve__search` finds skills. */+ readonly searchableSkills: boolean;+ /** `eve__skill`. */+ readonly skills: boolean;+ /** `eve__tool`. */+ readonly tools: boolean;+}+export const OfferedCatalogToolsKey = new ContextKey<OfferedCatalogTools>(+ "eve.offeredCatalogTools",+);export interface ActiveChannelDelivery {readonly traceSessionId: string;readonly agentName?: string;diff --git a/packages/eve/src/execution/cancelled-model-call-batch.ts b/packages/eve/src/execution/cancelled-model-call-batch.tsindex 1df35fdc0..a3bb98960 100644--- a/packages/eve/src/execution/cancelled-model-call-batch.ts+++ b/packages/eve/src/execution/cancelled-model-call-batch.ts@@ -1,6 +1,6 @@import { contextStorage, type ContextContainer } from "#context/container.js";import { createLogger } from "#internal/logging.js";-import { AuthKey, SandboxKey, TurnDeliveryIdsKey } from "#context/keys.js";+import { AuthKey, OfferedCatalogToolsKey, SandboxKey, TurnDeliveryIdsKey } from "#context/keys.js";import { preserveSerializedSessionDynamicModelSelection } from "#context/serialized-dynamic-model-selection.js";import { withContextScope } from "#context/run-step.js";import { serializeContext } from "#context/serialize.js";@@ -79,6 +79,8 @@ export async function createCancelledModelCallBatchResult(input: {const checkpointContext = {...(input.checkpoint?.serializedContext ?? input.beforeBatchContext),[AuthKey.name]: interruptedContext[AuthKey.name],+ // The cut calls may name catalog tools in history, and the record only grows.+ [OfferedCatalogToolsKey.name]: interruptedContext[OfferedCatalogToolsKey.name],[TurnDeliveryIdsKey.name]: interruptedContext[TurnDeliveryIdsKey.name],};diff --git a/packages/eve/src/execution/catalog/mcp-sign-in.test.ts b/packages/eve/src/execution/catalog/mcp-sign-in.test.tsindex 74dfc4f91..2b1ef2963 100644--- a/packages/eve/src/execution/catalog/mcp-sign-in.test.ts+++ b/packages/eve/src/execution/catalog/mcp-sign-in.test.ts@@ -189,10 +189,12 @@ describe("MCP connections sign in only when the server asks", () => {const output = await kennel.search("kennel");expect(output).toEqual({+ remaining: 0,results: [expect.objectContaining({ tool: "kennel__find_pet" }),expect.objectContaining({ tool: "kennel__pet_invoice" }),],+ searched: 2,});expect(kennel.startAuthorization).not.toHaveBeenCalled();expect(kennel.seen.every((request) => request.authorization === null)).toBe(true);@@ -240,7 +242,9 @@ describe("MCP connections sign in only when the server asks", () => {it("finds a server that rejects anonymous listing as its sign-in result, which signs in", async () => {const kennel = setup("token-everywhere");+ // Until sign-in, the search counts the sign-in entry, not the tools behind it.expect(await kennel.search("kennel__")).toEqual({+ remaining: 0,results: [{description: "Sign in to use the Kennel tools: Pet boarding.",@@ -248,6 +252,7 @@ describe("MCP connections sign in only when the server asks", () => {tool: "kennel",},],+ searched: 1,});expect(kennel.startAuthorization).not.toHaveBeenCalled();@@ -258,6 +263,7 @@ describe("MCP connections sign in only when the server asks", () => {);expect(await kennel.search("kennel__")).toMatchObject({results: [{ tool: "kennel__find_pet" }, { tool: "kennel__pet_invoice" }],+ searched: 2,});});});diff --git a/packages/eve/src/execution/catalog/search.test.ts b/packages/eve/src/execution/catalog/search.test.tsindex 01612f182..c4d76a668 100644--- a/packages/eve/src/execution/catalog/search.test.ts+++ b/packages/eve/src/execution/catalog/search.test.ts@@ -15,7 +15,9 @@ import {import { captureLogRecords } from "#internal/testing/log-records.js";interface SearchOutput {+ readonly remaining: number;readonly results: readonly Record<string, unknown>[];+ readonly searched: number;readonly unavailable?: readonly Record<string, unknown>[];}@@ -114,7 +116,8 @@ describe("search", () => {it("finds only deferred entries; direct tools and listed skills are already in context", async () => {const search = searchFor(tools, { skills });- expect(await search({ query: "add" })).toEqual({ results: [] });+ // Four deferred tools and agents, and one deferred skill.+ expect(await search({ query: "add" })).toEqual({ remaining: 0, results: [], searched: 5 });expect(names(await search({ query: "refund voice" }))).not.toContain("refund-voice");});@@ -134,13 +137,20 @@ describe("search", () => {await expect(search({})).rejects.toThrow(needsWords);});- it("returns the best matches up to limit, 10 by default, and takes no paging", async () => {- const many = Array.from({ length: 60 }, (_, index) =>- inlineTool(`report_${String(index).padStart(2, "0")}`, { deferred: true }),- );+ it("returns the best matches up to limit, 10 by default, with how many it searched and left out", async () => {+ const many = [+ ...Array.from({ length: 60 }, (_, index) =>+ inlineTool(`report_${String(index).padStart(2, "0")}`, { deferred: true }),+ ),+ inlineTool("export_ledger", { deferred: true, description: "Export the ledger." }),+ ];const search = searchFor(many);- const best = names(await search({ limit: 50, query: "report" }));+ const top = await search({ limit: 50, query: "report" });+ // `searched` counts every entry, matching or not; `remaining` counts only matches.+ expect(top).toMatchObject({ remaining: 10, searched: 61 });+ expect(await search({ query: "ledger" })).toMatchObject({ remaining: 0, searched: 61 });+ const best = names(top);expect(best).toHaveLength(50);// A smaller limit keeps the best matches, in rank order.expect(names(await search({ limit: 3, query: "report" }))).toEqual(best.slice(0, 3));@@ -205,7 +215,15 @@ describe("search", () => {signature: "notion(input: {}): Promise<unknown>",tool: "notion",};- expect(await search({ query: "notion" })).toEqual({ results: [signIn] });+ // Four deferred tools and agents, Linear's two tools, and Notion's sign-in+ // entry, which stands in for the tools it can't list yet.+ expect(await search({ query: "notion" })).toEqual({+ remaining: 0,+ results: [signIn],+ searched: 7,+ });+ // A namespace query counts only that namespace.+ expect(await search({ query: "linear__" })).toMatchObject({ searched: 2 });// Found by the connection's description too, while its tools can't be listed.expect((await search({ query: "pages" })).results).toEqual([signIn]);expect(names(await search({ query: "notion__" }))).toEqual(["notion"]);diff --git a/packages/eve/src/execution/catalog/search.ts b/packages/eve/src/execution/catalog/search.tsindex 5cf875545..cf15b1b98 100644--- a/packages/eve/src/execution/catalog/search.ts+++ b/packages/eve/src/execution/catalog/search.ts@@ -37,21 +37,22 @@ export interface SearchReach {/*** `eve__search`'s description, saying which tool takes each kind of result.- * It mentions only the catalog tools the agent has, so it is fixed for a- * deployment.+ * It mentions only the catalog tools the session has, so it changes only when+ * the session first gains one.*/function searchDescription({ connections, skills, tools }: SearchReach): string {return [- "Find your own tools, agents, skills, and connected services by keyword; this searches what you can do, not the web.",+ "Search your own catalog for tools, agents, skills, and connected services that aren't in your tool list. Use it when nothing in your tool list fits the request; it doesn't search the web.",tools? `A result with \`tool\` has the exact name, description, and TypeScript signature: call it with ${CALL_TOOL_NAME}({ name, input }).`: undefined,skills? `A result with \`skill\` has the name and description: load it with ${SKILL_TOOL_NAME}({ name }).`: undefined,+ "`searched` counts the entries the query covered, and `remaining` counts the matches past `limit`: when `remaining` is 0, nothing else the query covered matched. Connections under `unavailable` weren't searched.",'When you already know a name or connection, from the catalog listing, an earlier result, or an error, search it directly: the exact name, or "<connection>__", which is faster and returns only that connection\'s tools.',connections- ? `A connection that needs sign-in appears as a tool named after the connection: calling it with ${CALL_TOOL_NAME}({ name }) asks the user to sign in.`+ ? `A connection that needs sign-in appears as a tool named after the connection: calling it with ${CALL_TOOL_NAME}({ name }) asks the user to sign in. \`searched\` counts its tools only after sign-in.`: undefined,].filter((sentence) => sentence !== undefined)@@ -95,6 +96,14 @@ interface UnavailableConnection {interface SearchOutput {readonly results: readonly SearchResult[];+ /** Matches past `limit`. */+ readonly remaining: number;+ /**+ * Entries the query covered: the whole catalog, or one namespace. An+ * unavailable connection counts as none, and one behind sign-in as its one+ * sign-in entry.+ */+ readonly searched: number;readonly unavailable?: readonly UnavailableConnection[];}@@ -189,10 +198,14 @@ async function search(}const limit = clampInteger(input.limit, 1, MAX_LIMIT, DEFAULT_LIMIT);- const results = rankCandidates(query, candidates)- .slice(0, limit)- .map((candidate) => candidate.result());- return unavailable.length > 0 ? { results, unavailable } : { results };+ const matches = rankCandidates(query, candidates);+ const results = matches.slice(0, limit).map((candidate) => candidate.result());+ const found = {+ results,+ remaining: matches.length - results.length,+ searched: candidates.length,+ };+ return unavailable.length > 0 ? { ...found, unavailable } : found;}/**diff --git a/packages/eve/src/execution/catalog/step-catalog.integration.test.ts b/packages/eve/src/execution/catalog/step-catalog.integration.test.tsindex fb00c3b3f..31a605779 100644--- a/packages/eve/src/execution/catalog/step-catalog.integration.test.ts+++ b/packages/eve/src/execution/catalog/step-catalog.integration.test.ts@@ -188,9 +188,9 @@ function toolResultParts(}/** A dynamic tool, stamped with the durable callback the bundler adds to authored resolvers. */-function dynamicTool(description: string) {+function dynamicTool(description: string, { deferred = true } = {}) {const entry = defineTool({- deferred: true,+ deferred,description,execute: async () => ({ synced: true }),inputSchema: { type: "object" },@@ -781,6 +781,68 @@ describe("step catalog in the harness (real AI SDK)", () => {expect(toolResult(requests[1]!, "load-rules")).toBe("# house-rules");});+ it("adds search and eve__tool once a resolver returns a deferred tool, then keeps them", async () => {+ const ctx = createSessionContext();+ let exportAvailable = false;+ const surfaceTools = {+ eventNames: ["session.started"],+ events: {+ "session.started": () => ({+ send_reply: dynamicTool("Reply on Alice's surface.", { deferred: false }),+ ...(exportAvailable ? { export_report: dynamicTool("Export the billing report.") } : {}),+ }),+ },+ logicalPath: "agent/tools/surface.ts",+ slug: "surface",+ sourceId: "tools/surface.ts",+ sourceKind: "module" as const,+ };+ const bundle = catalogBundle();+ ctx.set(BundleKey, {+ ...bundle,+ resolvedAgent: { ...bundle.resolvedAgent, dynamicToolResolvers: [surfaceTools] } as never,+ });+ const resolve = () =>+ dispatchDynamicToolEvent({+ ctx,+ event: createSessionStartedEvent(),+ messages: [],+ resolvers: [surfaceTools],+ });+ const driver = createDriver(ctx, toolMap(inlineTool("add")));++ // Alice's surface resolves only a listed tool, so there is nothing to search.+ await resolve();+ driver.reply(text("Hello, Alice."));+ await driver.drive({ message: "Alice says hello." });++ // The resolver adds a deferred export tool, and Alice's next request uses it.+ exportAvailable = true;+ await resolve();+ driver.reply(+ calls(call("export", CALL_TOOL_NAME, { name: "export_report" })),+ text("Exported the billing report."),+ );+ await driver.drive({ message: "Alice asks for the billing report." });++ // The export tool goes away, and the catalog tools stay.+ exportAvailable = false;+ await resolve();+ driver.reply(text("Anything else, Alice?"));+ await driver.drive({ message: "Alice says thanks." });++ const requests = driver.requests();+ expect(requests.map((request) => request.tools?.map((tool) => tool.name))).toEqual([+ ["add", "send_reply"],+ ["add", "send_reply", SEARCH_TOOL_NAME, CALL_TOOL_NAME],+ ["add", "send_reply", SEARCH_TOOL_NAME, CALL_TOOL_NAME],+ ["add", "send_reply", SEARCH_TOOL_NAME, CALL_TOOL_NAME],+ ]);+ // The tool list changes once, when the catalog tools join it.+ for (const request of requests.slice(2)) expect(request.tools).toEqual(requests[1]!.tools);+ expect(toolResult(requests[2]!, "export")).toEqual({ synced: true });+ });+it("validates a connection tool's input before asking anyone to approve the call", async () => {const ctx = createSessionContext();const crm = fakeConnection({diff --git a/packages/eve/src/execution/catalog/step-catalog.test.ts b/packages/eve/src/execution/catalog/step-catalog.test.tsindex 2b8358d91..69cb323d9 100644--- a/packages/eve/src/execution/catalog/step-catalog.test.ts+++ b/packages/eve/src/execution/catalog/step-catalog.test.ts@@ -67,18 +67,6 @@ describe("buildStepCatalog", () => {});describe("which catalog tools the agent gets", () => {- /** The tool list of an agent that declares one dynamic resolver of `kind`. */- function advertisedWithResolver(- kind: "dynamicConnectionResolvers" | "dynamicSkillResolvers" | "dynamicToolResolvers",- ) {- const bundle = catalogBundle();- const ctx = new ContextContainer();- ctx.set(BundleKey, { ...bundle, resolvedAgent: { ...bundle.resolvedAgent, [kind]: [{}] } });- return names(- buildStepCatalog({ agentTools: toolMap(), ctx, endsTurn: true, session: {} }).advertised,- );- }-it("counts only the deferred tools the session sees, so a delegated session doesn't search for root-only ones", () => {const tools = [inlineTool("archive", { availableInSubagents: false, deferred: true })];@@ -172,7 +160,6 @@ describe("buildStepCatalog", () => {["a deferred tool", { tools: [inlineTool("refund_invoice", { deferred: true })] }],["a deferred agent", { tools: [subagentTool("researcher", { deferred: true })] }],["a connection", { connections: [fakeConnection({ name: "linear", tools: [] })] }],- ["a dynamic subagent resolver", { dynamicSubagents: ["helper"] }],])("gives an agent with %s search and eve__tool", (_, declared) => {// A subagent also brings the task tools; the catalog tools come last.expect(names(catalogContext(declared).catalog.advertised).slice(-2)).toEqual([@@ -181,16 +168,54 @@ describe("buildStepCatalog", () => {]);});- it.each([- ["dynamicToolResolvers", [SEARCH_TOOL_NAME, CALL_TOOL_NAME]],- ["dynamicConnectionResolvers", [SEARCH_TOOL_NAME, CALL_TOOL_NAME]],- ["dynamicSkillResolvers", [SEARCH_TOOL_NAME, SKILL_TOOL_NAME]],- ] as const)(- "gives an agent with %s its catalog tools, whatever they resolve",- (kind, tools) => {- expect(advertisedWithResolver(kind)).toEqual(tools);- },- );+ it("gives an agent no catalog tools for dynamic resolvers that returned nothing they reach", () => {+ const bundle = catalogBundle({ dynamicSubagents: ["helper"] });+ const ctx = new ContextContainer();+ ctx.set(BundleKey, {+ ...bundle,+ resolvedAgent: {+ ...bundle.resolvedAgent,+ dynamicConnectionResolvers: [{}],+ dynamicSkillResolvers: [{}],+ dynamicToolResolvers: [{}],+ } as never,+ });+ const catalog = buildStepCatalog({+ agentTools: toolMap(inlineTool("add")),+ ctx,+ endsTurn: true,+ session: {},+ });++ // A dynamic subagent resolver still brings the task tools.+ expect(names(catalog.advertised)).toEqual([+ "add",+ TASK_WAIT_TOOL_NAME,+ TASK_CANCEL_TOOL_NAME,+ ]);+ });++ it("keeps the catalog tools earlier steps offered, unchanged, once nothing reaches them", () => {+ const reachingEverything = catalogContext({+ connections: [fakeConnection({ name: "linear", tools: [] })],+ skills: [{ deferred: true, name: "pdf-forms" }],+ tools: [inlineTool("refund_invoice", { deferred: true })],+ }).catalog;+ const later = catalogContext({+ offered: reachingEverything.offered,+ tools: [inlineTool("add")],+ }).catalog;+ const catalogTools = (catalog: StepCatalog) =>+ [SEARCH_TOOL_NAME, CALL_TOOL_NAME, SKILL_TOOL_NAME].map((tool) => shape(catalog, tool));++ expect(names(later.advertised)).toEqual([+ "add",+ SEARCH_TOOL_NAME,+ CALL_TOOL_NAME,+ SKILL_TOOL_NAME,+ ]);+ expect(catalogTools(later)).toEqual(catalogTools(reachingEverything));+ });});it("hides root-only entries, direct or deferred, from delegated sessions", async () => {diff --git a/packages/eve/src/execution/catalog/step-catalog.ts b/packages/eve/src/execution/catalog/step-catalog.tsindex fe2196d33..20afb007c 100644--- a/packages/eve/src/execution/catalog/step-catalog.ts+++ b/packages/eve/src/execution/catalog/step-catalog.ts@@ -1,6 +1,6 @@/*** The step's catalog: one table of every entry a model step can reach. Direct- * entries go into the model's tool list next to the catalog tools the agent+ * entries go into the model's tool list next to the catalog tools the session* has: `eve__search` finds entries, `eve__tool` calls deferred entries and* connection tools, and `eve__skill` loads skills. A call resolves to its entry* here and then runs exactly as a direct call would, so only model history@@ -11,7 +11,8 @@ import type { StandardSchemaV1 } from "#compiled/@standard-schema/spec/index.js"import { connectionToolName } from "#connections/ownership.js";import { buildDynamicTools } from "#context/build-dynamic-tools.js";import { buildDynamicSubagentTools } from "#context/dynamic-subagent-lifecycle.js";-import type { ContextReader } from "#context/key.js";+import type { ContextAccessor, ContextReader } from "#context/key.js";+import { OfferedCatalogToolsKey, type OfferedCatalogTools } from "#context/keys.js";import { ConnectionRegistryKey } from "#context/providers/connection-key.js";import { startsTasks } from "#execution/tasks/tool-entry-point.js";import { withTaskTools } from "#execution/tasks/model-step.js";@@ -33,7 +34,7 @@ import {} from "#protocol/catalog-tools.js";import { eveNamespaceReservation } from "#protocol/runtime-tools.js";import type { ConnectionRegistry } from "#runtime/connections/registry-types.js";-import { BundleKey, type CompiledBundle } from "#runtime/sessions/runtime-context-keys.js";+import { BundleKey } from "#runtime/sessions/runtime-context-keys.js";import type { ResolvedConnectionDefinition } from "#runtime/types.js";import { isObject } from "#shared/guards.js";import type { ToolExecuteOptions } from "#tools/definition.js";@@ -61,54 +62,45 @@ const LISTED_SKILL_DESCRIPTION ="Load a skill's instructions when the request clearly matches one of your listed skills or the user asks for it, then follow them.";const SEARCHABLE_SKILL_DESCRIPTION = `Load a skill's instructions when the request clearly matches a listed skill or one ${SEARCH_TOOL_NAME} found, or the user asks for it, then follow them.`;-/** Which catalog tools an agent gets, and what they reach. */-interface CatalogTools {- /** A static connection or a dynamic connection resolver. */- readonly connections: boolean;+/** Which catalog tools a step offers, and what they reach. */+interface CatalogTools extends OfferedCatalogTools {readonly search: boolean;- /** A deferred skill or a dynamic skill resolver, which `eve__search` can find. */- readonly searchableSkills: boolean;- /** Any skill, static or from a dynamic skill resolver: `eve__skill` loads it. */- readonly skills: boolean;- /** Anything `eve__tool` can call: a deferred tool or agent, or a connection. */- readonly tools: boolean;}/*** Each catalog tool comes with what it reaches: `eve__tool` with a deferred- * tool or agent, a connection, or a resolver that may add one; `eve__skill`- * with any skill; and `eve__search` with anything it could find, which is- * whatever `eve__tool` calls plus deferred and dynamic skills. All follow from- * what the agent declares, never from what a step's resolvers returned, so the- * tool list is fixed for a deployment and never changes within a session. An- * agent with none of these has no catalog tools, and pays nothing for them.+ * tool or agent, or a connection; `eve__skill` with any skill; and+ * `eve__search` with anything it could find, which is whatever `eve__tool`+ * calls plus deferred skills. What a step reaches includes what its dynamic+ * resolvers returned, so a resolver that returns only listed tools adds none.+ * Once a session's tool list offers a catalog tool it keeps it, so the tool+ * list changes at most once for each and never drops one that earlier calls+ * named. A session that reaches none of these has no catalog tools, and pays+ * nothing for them.*/-function catalogToolsFor(- agentTools: readonly HarnessToolDefinition[],- bundle: CompiledBundle | undefined,-): CatalogTools {- const agent = bundle?.resolvedAgent;- const skills = agent?.skills ?? [];- const dynamicSkills = (agent?.dynamicSkillResolvers.length ?? 0) > 0;- const connections =- (agent?.connections ?? []).length > 0 || (agent?.dynamicConnectionResolvers?.length ?? 0) > 0;- const searchableSkills = dynamicSkills || skills.some((skill) => skill.deferred === true);- const tools =- connections ||- agentTools.some((definition) => definition.deferred === true) ||- (agent?.dynamicToolResolvers.length ?? 0) > 0 ||- (bundle?.subagentRegistry.dynamicResolvers.length ?? 0) > 0;+function catalogToolsFor(input: {+ readonly connections: readonly ResolvedConnectionDefinition[];+ readonly deferred: ReadonlyMap<string, HarnessToolDefinition>;+ readonly offered: OfferedCatalogTools | undefined;+ readonly skills: ReadonlyMap<string, CatalogSkill>;+}): CatalogTools {+ const { offered } = input;+ const skills = [...input.skills.values()];+ const connections = offered?.connections === true || input.connections.length > 0;+ const searchableSkills =+ offered?.searchableSkills === true || skills.some((skill) => skill.deferred);+ const tools = offered?.tools === true || connections || input.deferred.size > 0;return {connections,search: tools || searchableSkills,searchableSkills,- skills: skills.length > 0 || dynamicSkills,+ skills: offered?.skills === true || skills.length > 0,tools,};}export interface StepCatalog extends HarnessToolLookup {- /** The model's tool list: direct entries, then the catalog tools the agent has. */+ /** The model's tool list: direct entries, then the catalog tools the session has. */readonly advertised: HarnessToolMap;readonly connections: readonly ResolvedConnectionDefinition[];/** Entries reached only through `eve__tool`, connection tools aside. */@@ -117,6 +109,8 @@ export interface StepCatalog extends HarnessToolLookup {readonly entries: HarnessToolMap;/** Every skill the session can load, deferred or not. */readonly skills: ReadonlyMap<string, CatalogSkill>;+ /** The catalog tools the tool list offers; see {@link recordOfferedCatalogTools}. */+ readonly offered: OfferedCatalogTools;/*** Whether the session offers tasks: one of the agent's own tools it sees* starts them, or the agent resolves subagents at runtime. Both are fixed by@@ -148,9 +142,6 @@ export function buildStepCatalog(input: {}): StepCatalog {const visible = (definition: HarnessToolDefinition) => isVisible(definition, input.session);const bundle = input.ctx?.get(BundleKey);- // Only the tools this session sees count, so a delegated session of the same- // agent doesn't search for root-only entries it can't reach.- const catalogTools = catalogToolsFor([...input.agentTools.values()].filter(visible), bundle);const offersTasks =(bundle?.subagentRegistry.dynamicResolvers.length ?? 0) > 0 ||[...input.agentTools.values()].some(@@ -168,6 +159,15 @@ export function buildStepCatalog(input: {const registry = input.ctx?.get(ConnectionRegistryKey);const connections = registry?.getConnections() ?? [];const skills = sessionSkills(input.ctx);+ // Entries hold only what this session sees, so a delegated session of the+ // same agent doesn't search for root-only entries it can't reach.+ const catalogTools = catalogToolsFor({+ connections,+ deferred,+ offered: input.ctx?.get(OfferedCatalogToolsKey),+ skills,+ });+ const { search: _search, ...offered } = catalogTools;const skillLoader = createSkillLoader(skills);const advertised = new Map(direct);// Listed tools include the catalog tools, so `eve__tool` naming one of them@@ -184,6 +184,7 @@ export function buildStepCatalog(input: {describe,entries,get: (name) => (name === SKILL_ENTRY_NAME ? skillLoader : toolEntry(name)),+ offered,offersTasks,resolve: (toolCall) => {if (toolCall.toolName === CALL_TOOL_NAME && catalogTools.tools) {@@ -228,6 +229,18 @@ export function buildStepCatalog(input: {return catalog;}+/**+ * Keeps the catalog tools `catalog` offers for every later step of the+ * session. The step that calls the model records them; a step rebuilt to+ * resume a parked call only reads them.+ */+export function recordOfferedCatalogTools(+ ctx: ContextAccessor | undefined,+ { offered }: StepCatalog,+): void {+ if (Object.values(offered).includes(true)) ctx?.set(OfferedCatalogToolsKey, offered);+}+/** The entry an `eve__tool` or `eve__skill` input names, if it names one. */function targetName(input: unknown): string | undefined {return isObject(input) && typeof input.name === "string" ? input.name : undefined;diff --git a/packages/eve/src/execution/session/turn-step.test.ts b/packages/eve/src/execution/session/turn-step.test.tsindex 670b3f33e..666f1329f 100644--- a/packages/eve/src/execution/session/turn-step.test.ts+++ b/packages/eve/src/execution/session/turn-step.test.ts@@ -29,6 +29,7 @@ import {TurnDynamicToolMetadataKey,TurnDeliveryIdsKey,HistoryStateKey,+ OfferedCatalogToolsKey,} from "#context/keys.js";import { BundleKey, ChannelKey } from "#runtime/sessions/runtime-context-keys.js";import { deserializeContext, serializeContext } from "#context/serialize.js";@@ -2095,7 +2096,13 @@ describe("turnStep", () => {expect(observedSchedule).toBeUndefined();});- it("keeps a session-scoped dynamic model selection when the first turn is cancelled", async () => {+ it("keeps the dynamic model selection and offered catalog tools a cancelled first turn set", async () => {+ const CATALOG_TOOLS = {+ connections: false,+ searchableSkills: false,+ skills: false,+ tools: true,+ };const announcement = "Available skills\n- policy: Tenant policy";const session = createStubSession({history: [{ role: "user", content: announcement, kind: "user" }],@@ -2127,6 +2134,7 @@ describe("turnStep", () => {id: "anthropic/claude-opus-4.6",contextWindowTokens: 1_000_000,});+ ctx.set(OfferedCatalogToolsKey, CATALOG_TOOLS);ctx.set(ThreadKey, "discard this turn-scoped mutation");ctx.delete(HistoryStateKey);throw new TurnCancelledError();@@ -2161,6 +2169,7 @@ describe("turnStep", () => {serializedContext: {[TurnDeliveryIdsKey.name]: ["cancelled-delivery"],[HistoryStateKey.name]: { announcements: { skills: announcement } },+ [OfferedCatalogToolsKey.name]: CATALOG_TOOLS,[SessionDynamicModelReferenceKey.name]: {id: "anthropic/claude-opus-4.6",contextWindowTokens: 1_000_000,diff --git a/packages/eve/src/harness/model-call/run.ts b/packages/eve/src/harness/model-call/run.tsindex 7f818401e..d26f464ea 100644--- a/packages/eve/src/harness/model-call/run.ts+++ b/packages/eve/src/harness/model-call/run.ts@@ -1,7 +1,11 @@import type { LanguageModel, ModelMessage } from "ai";import { HistoryStateKey } from "#context/keys.js";-import { buildStepCatalog, type StepCatalog } from "#execution/catalog/step-catalog.js";+import {+ buildStepCatalog,+ recordOfferedCatalogTools,+ type StepCatalog,+} from "#execution/catalog/step-catalog.js";import type { GenerationSteering } from "#harness/generation-steering.js";import { type HarnessModelMessage, validateHarnessModelMessages } from "#harness/messages.js";import {@@ -109,6 +113,7 @@ export async function runModelStep(endsTurn,session: step.session,});+ recordOfferedCatalogTools(step.ctx, catalog);step.frameworkToolNames = frameworkToolNames(catalog);const { approvedTools, pendingApprovalsNote } = humanInputContext(step, catalog);const caller = new ModelCaller(step, prompt, {diff --git a/packages/eve/src/internal/testing/catalog-fixtures.ts b/packages/eve/src/internal/testing/catalog-fixtures.tsindex 2da6abab9..0f55d2821 100644--- a/packages/eve/src/internal/testing/catalog-fixtures.ts+++ b/packages/eve/src/internal/testing/catalog-fixtures.ts@@ -6,7 +6,12 @@import { ConnectionAuthorizationRequiredError } from "#connections/errors.js";import { ContextContainer, contextStorage } from "#context/container.js";-import { AuthKey, SessionIdKey } from "#context/keys.js";+import {+ AuthKey,+ OfferedCatalogToolsKey,+ SessionIdKey,+ type OfferedCatalogTools,+} from "#context/keys.js";import { ConnectionRegistryKey } from "#context/providers/connection-key.js";import { buildStepCatalog } from "#execution/catalog/step-catalog.js";import { CallbackBaseUrlKey } from "#harness/authorization.js";@@ -253,6 +258,8 @@ export function catalogContext(input: {readonly connections?: readonly FakeConnection[];readonly dynamicSubagents?: readonly string[];+ /** The catalog tools earlier steps of the session offered. */+ readonly offered?: OfferedCatalogTools;readonly session?: { readonly rootSessionId?: string };readonly skills?: readonly CatalogSkillSource[];readonly tools?: readonly HarnessToolDefinition[];@@ -278,6 +285,7 @@ export function catalogContext(if (input.connections !== undefined) {ctx.set(ConnectionRegistryKey, connectionRegistry(input.connections));}+ if (input.offered !== undefined) ctx.set(OfferedCatalogToolsKey, input.offered);const catalog = buildStepCatalog({agentTools: toolMap(...(input.tools ?? [])),ctx,