Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

test(eve): reproduce tool signatures dropping property names past the length cap (#4685)

roboeve[bot]f6bf73ea1cb7
diff --git a/.changeset/signature-property-names.md b/.changeset/signature-property-names.md
new file mode 100644
index 000000000..c44a2d0dc
--- /dev/null
+++ b/.changeset/signature-property-names.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Keep every property name in `eve__search` tool signatures that run past the length cap. A long signature now drops its property descriptions before any property name, so the model no longer misses fields a large MCP tool such as Linear's `save_issue` supports.
diff --git a/docs/concepts/built-in-tools.md b/docs/concepts/built-in-tools.md
index dd9393d2f..a211383d0 100644
--- a/docs/concepts/built-in-tools.md
+++ b/docs/concepts/built-in-tools.md
@@ -310,7 +310,7 @@ These tools let the model reach what isn't in its tool list. `eve__search` finds
Entries the agent declares count from the session's first step. An entry a dynamic resolver returns counts from the step where it first appears, so a resolver in `agent/tools/` that returns only tools without `deferred: true` adds no catalog tools. Once a session's tool list includes a catalog tool, the session keeps it, even after a resolver stops returning the entry that brought it. The catalog tools change only when the session first reaches a new kind of entry, such as its first deferred tool or its first connection.
-- `eve__search({ query, limit? })` returns the best matches, up to `limit` (default 10, at most 50). `query` is required and must contain a word. A tool match has its exact `tool` name, its `description`, and a TypeScript `signature` rendered from its schemas. A deferred skill's match has its `skill` name, its `description`, and the `path` of its `SKILL.md` when it has supporting files. A result with `tool` is called with `eve__tool`, and one with `skill` is loaded with `eve__skill`. Next to `results`, `searched` counts the entries the query covered, the whole catalog or one namespace, and `remaining` counts the matches past `limit`. A connection behind sign-in counts as its one sign-in result until the user signs in, and an unavailable connection counts as none. Tools and skills rank together, in tiers: an entry whose name is exactly the query, then one whose name after its last `__` is the query, such as `linear__create_issue` or `sre__create_issue` for `create_issue`, then the tools of a connection named by the query, then entries whose names, or names after their last `__`, start with the query, then keyword matches in names, parameters, and descriptions. So `eve__search({ query: "linear" })` lists the `linear` connection's tools first. One-letter and filler words such as `a` and `the` don't count toward keyword matches unless the query has nothing else.
+- `eve__search({ query, limit? })` returns the best matches, up to `limit` (default 10, at most 50). `query` is required and must contain a word. A tool match has its exact `tool` name, its `description`, and a TypeScript `signature` rendered from its schemas. A very long signature drops its property descriptions before any property name. A deferred skill's match has its `skill` name, its `description`, and the `path` of its `SKILL.md` when it has supporting files. A result with `tool` is called with `eve__tool`, and one with `skill` is loaded with `eve__skill`. Next to `results`, `searched` counts the entries the query covered, the whole catalog or one namespace, and `remaining` counts the matches past `limit`. A connection behind sign-in counts as its one sign-in result until the user signs in, and an unavailable connection counts as none. Tools and skills rank together, in tiers: an entry whose name is exactly the query, then one whose name after its last `__` is the query, such as `linear__create_issue` or `sre__create_issue` for `create_issue`, then the tools of a connection named by the query, then entries whose names, or names after their last `__`, start with the query, then keyword matches in names, parameters, and descriptions. So `eve__search({ query: "linear" })` lists the `linear` connection's tools first. One-letter and filler words such as `a` and `the` don't count toward keyword matches unless the query has nothing else.
- A query whose first word contains `__` searches one namespace: everything before its last `__`, without trailing underscores. The namespace only filters. `eve__search` keeps names under `<namespace>__` and anything named exactly the namespace, such as a connection's sign-in entry, then ranks them by the whole query, so `eve__search({ query: "linear__" })` returns only the `linear` connection's tools, or its sign-in result, and an exact name that contains `__` still ranks first. eve lists only the connections that can own names in that namespace, so other connections make no network call and don't appear in `unavailable`. A namespace that matches nothing fails with the closest connection names. Characters that can't appear in a name, such as a leading `^`, are ignored; `query` isn't a regular expression.
- `eve__search` never asks the user to sign in. A connection whose server won't list its tools until the user signs in appears as one result named after the connection, such as `linear`; a connection whose server lists its tools without a token is searched like any other. A connection whose tools fail to load, or don't load within 10 seconds, appears under `unavailable` with its `error`, such as `"crm" did not list its tools within 10s. Try again later.` Matches from everything else are still returned, so a slow server doesn't hold up the search. An `eve__tool` call on that connection's tools fails with the same error.
- `eve__tool({ name: "linear" })`, with a connection's own name, asks the user to sign in to that connection when its tools need it, and returns once they can be listed, so the next `eve__search` finds them. On a server that lists its tools without sign-in, it confirms the tools are available without asking.
diff --git a/packages/eve/src/tools/signature.test.ts b/packages/eve/src/tools/signature.test.ts
index bf7fb79cf..5f519472f 100644
--- a/packages/eve/src/tools/signature.test.ts
+++ b/packages/eve/src/tools/signature.test.ts
@@ -81,6 +81,23 @@ describe("renderToolSignature", () => {
expect(signature.endsWith("...")).toBe(true);
});
+ it("keeps every property name when long descriptions overflow the cap", () => {
+ // Like Linear's `save_issue`: dozens of documented fields whose names alone fit easily.
+ const names = Array.from({ length: 40 }, (_, index) => `field${index}`);
+ const properties = Object.fromEntries(
+ names.map((name) => [
+ name,
+ { type: "string", description: `Sets ${name}. ${"x".repeat(200)}` },
+ ]),
+ );
+ const signature = renderToolSignature({
+ name: "f",
+ inputSchema: { type: "object", properties },
+ });
+ expect(signature.length).toBeLessThanOrEqual(4_000);
+ expect(names.filter((name) => !signature.includes(`${name}?:`))).toEqual([]);
+ });
+
it("bounds the work for schemas whose shared refs expand exponentially", () => {
// Each definition has ten properties pointing at the next: about 10^6 nodes
// within the depth limit if every reference were expanded.
diff --git a/packages/eve/src/tools/signature.ts b/packages/eve/src/tools/signature.ts
index a65db2bf6..070b8a423 100644
--- a/packages/eve/src/tools/signature.ts
+++ b/packages/eve/src/tools/signature.ts
@@ -41,30 +41,40 @@ interface RenderContext {
/** Shared across one schema's render; counts down to zero. */
readonly budget: { remaining: number };
readonly depth: number;
+ readonly descriptions: boolean;
readonly refs: ReadonlySet<string>;
readonly root: Schema;
}
-export function renderToolSignature(input: {
+interface SignatureInput {
readonly name: string;
readonly inputSchema?: Record<string, unknown>;
readonly outputSchema?: Record<string, unknown>;
-}): string {
- const inputType = input.inputSchema === undefined ? "{}" : renderRoot(input.inputSchema);
- const outputType = input.outputSchema === undefined ? "unknown" : renderRoot(input.outputSchema);
- const signature = `${input.name}(input: ${inputType}): Promise<${outputType}>`;
- return signature.length > MAX_SIGNATURE_LENGTH
- ? `${signature.slice(0, MAX_SIGNATURE_LENGTH - 3)}...`
- : signature;
}
-function renderRoot(schema: Schema): string {
- return renderType(schema, {
- budget: { remaining: MAX_NODES },
- depth: 0,
- refs: new Set(),
- root: schema,
- });
+export function renderToolSignature(input: SignatureInput): string {
+ const signature = renderSignature(input, true);
+ if (signature.length <= MAX_SIGNATURE_LENGTH) return signature;
+ // A field missing from the signature reads as a field the tool doesn't have,
+ // so property names outrank their descriptions.
+ const compact = renderSignature(input, false);
+ return compact.length > MAX_SIGNATURE_LENGTH
+ ? `${compact.slice(0, MAX_SIGNATURE_LENGTH - 3)}...`
+ : compact;
+}
+
+function renderSignature(input: SignatureInput, descriptions: boolean): string {
+ const render = (schema: Schema) =>
+ renderType(schema, {
+ budget: { remaining: MAX_NODES },
+ depth: 0,
+ descriptions,
+ refs: new Set(),
+ root: schema,
+ });
+ const inputType = input.inputSchema === undefined ? "{}" : render(input.inputSchema);
+ const outputType = input.outputSchema === undefined ? "unknown" : render(input.outputSchema);
+ return `${input.name}(input: ${inputType}): Promise<${outputType}>`;
}
function renderType(schema: unknown, context: RenderContext): string {
@@ -136,7 +146,7 @@ function renderObject(schema: Schema, context: RenderContext): string {
const fields = Object.entries(properties).map(([key, value]) => {
const name = IDENTIFIER.test(key) ? key : JSON.stringify(key);
const optional = required.has(key) ? "" : "?";
- const comment = isObject(value) ? propertyComment(value) : "";
+ const comment = isObject(value) ? propertyComment(value, context.descriptions) : "";
return `${name}${optional}: ${renderType(value, next(context))}${comment}`;
});
const extra = schema.additionalProperties;
@@ -147,10 +157,10 @@ function renderObject(schema: Schema, context: RenderContext): string {
return `{ ${fields.join("; ")} }`;
}
-function propertyComment(schema: Schema): string {
+function propertyComment(schema: Schema, descriptions: boolean): string {
const parts: string[] = [];
const description =
- typeof schema.description === "string"
+ descriptions && typeof schema.description === "string"
? schema.description
.replaceAll(/\s+/gu, " ")
.replace(/[\s.;,:]+$/u, "")