The Open Framework for Building Agents
test(eve): reproduce tool signatures dropping property names past the length cap (#4685)
roboeve[bot]f6bf73ea1cb7
diff --git a/.changeset/signature-property-names.md b/.changeset/signature-property-names.mdnew file mode 100644index 000000000..c44a2d0dc--- /dev/null+++ b/.changeset/signature-property-names.md@@ -0,0 +1,5 @@+---+"eve": patch+---++Keep every property name in `eve__search` tool signatures that run past the length cap. A long signature now drops its property descriptions before any property name, so the model no longer misses fields a large MCP tool such as Linear's `save_issue` supports.diff --git a/docs/concepts/built-in-tools.md b/docs/concepts/built-in-tools.mdindex dd9393d2f..a211383d0 100644--- a/docs/concepts/built-in-tools.md+++ b/docs/concepts/built-in-tools.md@@ -310,7 +310,7 @@ These tools let the model reach what isn't in its tool list. `eve__search` findsEntries the agent declares count from the session's first step. An entry a dynamic resolver returns counts from the step where it first appears, so a resolver in `agent/tools/` that returns only tools without `deferred: true` adds no catalog tools. Once a session's tool list includes a catalog tool, the session keeps it, even after a resolver stops returning the entry that brought it. The catalog tools change only when the session first reaches a new kind of entry, such as its first deferred tool or its first connection.-- `eve__search({ query, limit? })` returns the best matches, up to `limit` (default 10, at most 50). `query` is required and must contain a word. A tool match has its exact `tool` name, its `description`, and a TypeScript `signature` rendered from its schemas. A deferred skill's match has its `skill` name, its `description`, and the `path` of its `SKILL.md` when it has supporting files. A result with `tool` is called with `eve__tool`, and one with `skill` is loaded with `eve__skill`. Next to `results`, `searched` counts the entries the query covered, the whole catalog or one namespace, and `remaining` counts the matches past `limit`. A connection behind sign-in counts as its one sign-in result until the user signs in, and an unavailable connection counts as none. Tools and skills rank together, in tiers: an entry whose name is exactly the query, then one whose name after its last `__` is the query, such as `linear__create_issue` or `sre__create_issue` for `create_issue`, then the tools of a connection named by the query, then entries whose names, or names after their last `__`, start with the query, then keyword matches in names, parameters, and descriptions. So `eve__search({ query: "linear" })` lists the `linear` connection's tools first. One-letter and filler words such as `a` and `the` don't count toward keyword matches unless the query has nothing else.+- `eve__search({ query, limit? })` returns the best matches, up to `limit` (default 10, at most 50). `query` is required and must contain a word. A tool match has its exact `tool` name, its `description`, and a TypeScript `signature` rendered from its schemas. A very long signature drops its property descriptions before any property name. A deferred skill's match has its `skill` name, its `description`, and the `path` of its `SKILL.md` when it has supporting files. A result with `tool` is called with `eve__tool`, and one with `skill` is loaded with `eve__skill`. Next to `results`, `searched` counts the entries the query covered, the whole catalog or one namespace, and `remaining` counts the matches past `limit`. A connection behind sign-in counts as its one sign-in result until the user signs in, and an unavailable connection counts as none. Tools and skills rank together, in tiers: an entry whose name is exactly the query, then one whose name after its last `__` is the query, such as `linear__create_issue` or `sre__create_issue` for `create_issue`, then the tools of a connection named by the query, then entries whose names, or names after their last `__`, start with the query, then keyword matches in names, parameters, and descriptions. So `eve__search({ query: "linear" })` lists the `linear` connection's tools first. One-letter and filler words such as `a` and `the` don't count toward keyword matches unless the query has nothing else.- A query whose first word contains `__` searches one namespace: everything before its last `__`, without trailing underscores. The namespace only filters. `eve__search` keeps names under `<namespace>__` and anything named exactly the namespace, such as a connection's sign-in entry, then ranks them by the whole query, so `eve__search({ query: "linear__" })` returns only the `linear` connection's tools, or its sign-in result, and an exact name that contains `__` still ranks first. eve lists only the connections that can own names in that namespace, so other connections make no network call and don't appear in `unavailable`. A namespace that matches nothing fails with the closest connection names. Characters that can't appear in a name, such as a leading `^`, are ignored; `query` isn't a regular expression.- `eve__search` never asks the user to sign in. A connection whose server won't list its tools until the user signs in appears as one result named after the connection, such as `linear`; a connection whose server lists its tools without a token is searched like any other. A connection whose tools fail to load, or don't load within 10 seconds, appears under `unavailable` with its `error`, such as `"crm" did not list its tools within 10s. Try again later.` Matches from everything else are still returned, so a slow server doesn't hold up the search. An `eve__tool` call on that connection's tools fails with the same error.- `eve__tool({ name: "linear" })`, with a connection's own name, asks the user to sign in to that connection when its tools need it, and returns once they can be listed, so the next `eve__search` finds them. On a server that lists its tools without sign-in, it confirms the tools are available without asking.diff --git a/packages/eve/src/tools/signature.test.ts b/packages/eve/src/tools/signature.test.tsindex bf7fb79cf..5f519472f 100644--- a/packages/eve/src/tools/signature.test.ts+++ b/packages/eve/src/tools/signature.test.ts@@ -81,6 +81,23 @@ describe("renderToolSignature", () => {expect(signature.endsWith("...")).toBe(true);});+ it("keeps every property name when long descriptions overflow the cap", () => {+ // Like Linear's `save_issue`: dozens of documented fields whose names alone fit easily.+ const names = Array.from({ length: 40 }, (_, index) => `field${index}`);+ const properties = Object.fromEntries(+ names.map((name) => [+ name,+ { type: "string", description: `Sets ${name}. ${"x".repeat(200)}` },+ ]),+ );+ const signature = renderToolSignature({+ name: "f",+ inputSchema: { type: "object", properties },+ });+ expect(signature.length).toBeLessThanOrEqual(4_000);+ expect(names.filter((name) => !signature.includes(`${name}?:`))).toEqual([]);+ });+it("bounds the work for schemas whose shared refs expand exponentially", () => {// Each definition has ten properties pointing at the next: about 10^6 nodes// within the depth limit if every reference were expanded.diff --git a/packages/eve/src/tools/signature.ts b/packages/eve/src/tools/signature.tsindex a65db2bf6..070b8a423 100644--- a/packages/eve/src/tools/signature.ts+++ b/packages/eve/src/tools/signature.ts@@ -41,30 +41,40 @@ interface RenderContext {/** Shared across one schema's render; counts down to zero. */readonly budget: { remaining: number };readonly depth: number;+ readonly descriptions: boolean;readonly refs: ReadonlySet<string>;readonly root: Schema;}-export function renderToolSignature(input: {+interface SignatureInput {readonly name: string;readonly inputSchema?: Record<string, unknown>;readonly outputSchema?: Record<string, unknown>;-}): string {- const inputType = input.inputSchema === undefined ? "{}" : renderRoot(input.inputSchema);- const outputType = input.outputSchema === undefined ? "unknown" : renderRoot(input.outputSchema);- const signature = `${input.name}(input: ${inputType}): Promise<${outputType}>`;- return signature.length > MAX_SIGNATURE_LENGTH- ? `${signature.slice(0, MAX_SIGNATURE_LENGTH - 3)}...`- : signature;}-function renderRoot(schema: Schema): string {- return renderType(schema, {- budget: { remaining: MAX_NODES },- depth: 0,- refs: new Set(),- root: schema,- });+export function renderToolSignature(input: SignatureInput): string {+ const signature = renderSignature(input, true);+ if (signature.length <= MAX_SIGNATURE_LENGTH) return signature;+ // A field missing from the signature reads as a field the tool doesn't have,+ // so property names outrank their descriptions.+ const compact = renderSignature(input, false);+ return compact.length > MAX_SIGNATURE_LENGTH+ ? `${compact.slice(0, MAX_SIGNATURE_LENGTH - 3)}...`+ : compact;+}++function renderSignature(input: SignatureInput, descriptions: boolean): string {+ const render = (schema: Schema) =>+ renderType(schema, {+ budget: { remaining: MAX_NODES },+ depth: 0,+ descriptions,+ refs: new Set(),+ root: schema,+ });+ const inputType = input.inputSchema === undefined ? "{}" : render(input.inputSchema);+ const outputType = input.outputSchema === undefined ? "unknown" : render(input.outputSchema);+ return `${input.name}(input: ${inputType}): Promise<${outputType}>`;}function renderType(schema: unknown, context: RenderContext): string {@@ -136,7 +146,7 @@ function renderObject(schema: Schema, context: RenderContext): string {const fields = Object.entries(properties).map(([key, value]) => {const name = IDENTIFIER.test(key) ? key : JSON.stringify(key);const optional = required.has(key) ? "" : "?";- const comment = isObject(value) ? propertyComment(value) : "";+ const comment = isObject(value) ? propertyComment(value, context.descriptions) : "";return `${name}${optional}: ${renderType(value, next(context))}${comment}`;});const extra = schema.additionalProperties;@@ -147,10 +157,10 @@ function renderObject(schema: Schema, context: RenderContext): string {return `{ ${fields.join("; ")} }`;}-function propertyComment(schema: Schema): string {+function propertyComment(schema: Schema, descriptions: boolean): string {const parts: string[] = [];const description =- typeof schema.description === "string"+ descriptions && typeof schema.description === "string"? schema.description.replaceAll(/\s+/gu, " ").replace(/[\s.;,:]+$/u, "")