Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

feat(eve): evaluate - expose standalone evaluation with shared auth (#3472)

Andrew Barbadb5cee3725d6
diff --git a/.changeset/quiet-evaluations-share-auth.md b/.changeset/quiet-evaluations-share-auth.md
new file mode 100644
index 000000000..73165b249
--- /dev/null
+++ b/.changeset/quiet-evaluations-share-auth.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Add standalone `evaluate` to `eve/experimental/evaluate` for typed evaluations in tools and application code. It shares model authentication with `autoModel`, including the Gateway connection selected during `eve dev`.
diff --git a/docs/guides/evaluate.md b/docs/guides/evaluate.md
index 7805ce700..ecb5bf1c9 100644
--- a/docs/guides/evaluate.md
+++ b/docs/guides/evaluate.md
@@ -1,12 +1,13 @@
---
title: Automatic Model Selection
-description: "Choose an agent model from the current request with an AI SDK evaluation model."
+description: "Choose agent models automatically or evaluate typed questions in your tools and application code."
---
Use `autoModel` to choose an agent model from an allowlist before inference begins.
It uses the [AI SDK evaluation API](https://ai-sdk.dev/docs/ai-sdk-core/evaluation),
so the evaluator can be a Vercel AI Gateway model ID or an evaluation model from
-an installed provider.
+an installed provider. Use `evaluate` from the same entrypoint to ask typed
+questions in your own tools or application code.
`eve/experimental/evaluate` is experimental. Its API can change between eve
releases, and the AI SDK evaluation model specification can change in patch
@@ -103,6 +104,54 @@ Supported reasoning values are `"provider-default"`, `"none"`, `"minimal"`,
`"low"`, `"medium"`, `"high"`, and `"xhigh"`. An omitted value inherits the
agent's reasoning setting.
+## Evaluate inside a tool
+
+Use `evaluate` to ask choice, score, or boolean questions about the state you pass to it.
+It defaults to `typesafe-ai/jev` and uses the same authentication as `autoModel`,
+including the Gateway connection selected through `/login` during `eve dev`.
+Pass `model` to use another evaluation model ID or a provider instance. A configured
+AI SDK default provider takes precedence over the local Gateway connection.
+
+```ts title="agent/tools/classify-request.ts"
+import { evaluate } from "eve/experimental/evaluate";
+import { defineTool } from "eve/tools";
+import { z } from "zod";
+
+export default defineTool({
+ description: "Choose the team that can help with a customer request.",
+ inputSchema: z.object({ request: z.string().min(1).max(8000) }),
+ async execute({ request }, ctx) {
+ const result = await evaluate({
+ state: { request },
+ questions: {
+ team: {
+ type: "choice",
+ instructions: "Select the team best suited to handle the request.",
+ criteria: {
+ billing: "Invoices, payments, and refunds",
+ support: "Product questions and troubleshooting",
+ },
+ },
+ },
+ abortSignal: ctx.abortSignal,
+ });
+ return { team: result.answers.team.choice };
+ },
+});
+```
+
+The choice above is typed as `"billing" | "support"`. Each question appears under
+its authored key in `result.answers`. Results also include token usage, warnings,
+provider metadata, and response metadata.
+
+`evaluate` accepts AI SDK evaluation options, including `maxRetries`, `headers`,
+and `providerOptions`. Pass an `abortSignal` to cancel the request. Input and
+answer validation, retries, and provider errors follow AI SDK semantics.
+
+You can also call `evaluate` outside a tool; it does not require an active eve
+session. Each call performs its own evaluation. `autoModel` uses this function
+and adds the per-turn routing behavior described below.
+
## Runtime behavior
`autoModel` evaluates at the first `step.started` event, after the incoming prompt
diff --git a/docs/reference/typescript-api.md b/docs/reference/typescript-api.md
index e13484d45..deb344b1f 100644
--- a/docs/reference/typescript-api.md
+++ b/docs/reference/typescript-api.md
@@ -137,7 +137,7 @@ import template from "../../prompts/template.txt?raw";
| `eve/local-dev` | `getLocalDevCapability`, `LocalDevCapability` |
| `eve/models/openai` | `openai`, `chatgpt`, deprecated `experimental_chatgpt` |
| `eve/models/anthropic` | `anthropic` |
-| [`eve/experimental/evaluate`](../guides/evaluate) | Experimental `autoModel` |
+| [`eve/experimental/evaluate`](../guides/evaluate) | Experimental `autoModel` and standalone `evaluate` |
| `eve/evals` | `defineEval`, `defineEvalConfig`, `mockModel`, eval types |
| `eve/evals/expect` | `includes`, `equals`, `matches`, `similarity` |
| `eve/evals/reporters` | `Braintrust`, `JUnit`, `EvalReporter` |
diff --git a/e2e/fixtures/agent-evaluate/agent/agent.ts b/e2e/fixtures/agent-evaluate/agent/agent.ts
index 1eb2b9f14..55ac687b1 100644
--- a/e2e/fixtures/agent-evaluate/agent/agent.ts
+++ b/e2e/fixtures/agent-evaluate/agent/agent.ts
@@ -8,6 +8,21 @@ const { experimental } = e2eAgentConfig();
export default defineAgent({
experimental,
model: fixtureModel(async (request) => {
+ if (request.userMessages.some((text) => text.includes("evaluate-request"))) {
+ const result = request.toolResults.find((result) => result.name === "evaluate-request");
+ if (result) return JSON.stringify({ isError: result.isError, output: result.output });
+ return {
+ toolCalls: [
+ {
+ id: "evaluate-request-1",
+ name: "evaluate-request",
+ input: {
+ missingAnswer: request.userMessages.some((text) => text.includes("missing answer")),
+ },
+ },
+ ],
+ };
+ }
if (request.userMessages.some((text) => text.includes("parallel investigations"))) {
const completed = request.messages.filter((message) =>
message.text.includes("child-result:"),
diff --git a/e2e/fixtures/agent-evaluate/agent/testing.ts b/e2e/fixtures/agent-evaluate/agent/testing.ts
index a192ff4d7..8387e273d 100644
--- a/e2e/fixtures/agent-evaluate/agent/testing.ts
+++ b/e2e/fixtures/agent-evaluate/agent/testing.ts
@@ -10,7 +10,7 @@ export const routing = defineState("evaluate-fixture.routing", () => ({
reasoning: "unselected",
}));
-const evaluationModel = {
+export const evaluationModel: Exclude<Experimental_EvaluationModel, string> = {
specificationVersion: "v4",
provider: "fixture",
modelId: "fixture-evaluator",
@@ -37,7 +37,7 @@ const evaluationModel = {
response: { modelId: "fixture-evaluator" },
};
},
-} satisfies Exclude<Experimental_EvaluationModel, string>;
+};
/** Run the real router with deterministic evaluation and language models. */
export function fixtureModel(respond: MockModelResponder) {
diff --git a/e2e/fixtures/agent-evaluate/agent/tools/evaluate-request.ts b/e2e/fixtures/agent-evaluate/agent/tools/evaluate-request.ts
new file mode 100644
index 000000000..0c5d46951
--- /dev/null
+++ b/e2e/fixtures/agent-evaluate/agent/tools/evaluate-request.ts
@@ -0,0 +1,34 @@
+import { evaluate } from "eve/experimental/evaluate";
+import { defineTool } from "eve/tools";
+
+import { evaluationModel } from "../testing";
+
+export default defineTool({
+ description: "Evaluate Alice's request with the fixture evaluation provider.",
+ inputSchema: {
+ type: "object",
+ properties: { missingAnswer: { type: "boolean" } },
+ required: ["missingAnswer"],
+ additionalProperties: false,
+ },
+ async execute({ missingAnswer }, ctx) {
+ const result = await evaluate({
+ model: missingAnswer
+ ? { ...evaluationModel, doEvaluate: async () => ({ answers: {}, warnings: [] }) }
+ : evaluationModel,
+ state: { request: "Alice needs a routine summary." },
+ questions: {
+ route: {
+ type: "choice",
+ instructions: "Choose a model for the request.",
+ criteria: {
+ "openai/large": "Difficult investigations",
+ "openai/small": "Routine requests",
+ },
+ },
+ },
+ abortSignal: ctx.abortSignal,
+ });
+ return { choice: result.answers.route.choice, usage: result.usage };
+ },
+});
diff --git a/e2e/fixtures/agent-evaluate/evals/tool-evaluation-failure.eval.ts b/e2e/fixtures/agent-evaluate/evals/tool-evaluation-failure.eval.ts
new file mode 100644
index 000000000..ee08d77b7
--- /dev/null
+++ b/e2e/fixtures/agent-evaluate/evals/tool-evaluation-failure.eval.ts
@@ -0,0 +1,14 @@
+import { defineEval } from "eve/evals";
+
+export default defineEval({
+ description: "An invalid evaluation response is reported as a tool error.",
+ async test(t) {
+ const turn = await t.send(
+ "Bob uses evaluate-request to review a missing answer from the evaluation service.",
+ );
+ turn.expectOk();
+ turn.calledTool("evaluate-request", { status: "failed", count: 1 });
+ turn.messageIncludes('"isError":true');
+ t.succeeded();
+ },
+});
diff --git a/e2e/fixtures/agent-evaluate/evals/tool-evaluation.eval.ts b/e2e/fixtures/agent-evaluate/evals/tool-evaluation.eval.ts
new file mode 100644
index 000000000..210c66ffa
--- /dev/null
+++ b/e2e/fixtures/agent-evaluate/evals/tool-evaluation.eval.ts
@@ -0,0 +1,16 @@
+import { defineEval } from "eve/evals";
+
+export default defineEval({
+ description: "A tool evaluates structured state and returns typed answers and usage.",
+ async test(t) {
+ const turn = await t.send(
+ "Alice uses evaluate-request to choose a model for her routine summary.",
+ );
+ turn.expectOk();
+ turn.calledTool("evaluate-request");
+ turn.messageIncludes('"isError":false');
+ turn.messageIncludes('"choice":"openai/small"');
+ turn.messageIncludes('"totalTokens":45');
+ t.succeeded();
+ },
+});
diff --git a/packages/eve/extension-contracts/compatibility/tool/v46.ts b/packages/eve/extension-contracts/compatibility/tool/v46.ts
new file mode 100644
index 000000000..27fa92d63
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/tool/v46.ts
@@ -0,0 +1,9 @@
+import { defineTool } from "#public/tools/index.js";
+
+export default defineTool({
+ description: "Report whether the current tool execution was cancelled.",
+ inputSchema: { type: "object", properties: {}, additionalProperties: false },
+ execute(_input, ctx) {
+ return { cancelled: ctx.abortSignal?.aborted ?? false };
+ },
+});
diff --git a/packages/eve/extension-contracts/entrypoints/tool.ts b/packages/eve/extension-contracts/entrypoints/tool.ts
index 5849681a1..cb7611f78 100644
--- a/packages/eve/extension-contracts/entrypoints/tool.ts
+++ b/packages/eve/extension-contracts/entrypoints/tool.ts
@@ -18,3 +18,4 @@ export {
type WorkflowToolInput,
type WorkflowToolOptions,
} from "../../src/public/tools/workflow.ts";
+export { evaluate } from "../../src/public/experimental/evaluate/index.ts";
diff --git a/packages/eve/extension-contracts/reports/tool/v47.json b/packages/eve/extension-contracts/reports/tool/v47.json
new file mode 100644
index 000000000..72cde997d
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/tool/v47.json
@@ -0,0 +1,23 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "tool",
+ "epoch": 47,
+ "sha256": "c8f3dea743ed9d8c175769c1249d099bc5265beda83104a8edb9ff841c7f0766",
+ "exports": [
+ "WorkflowTool",
+ "WorkflowToolInput",
+ "WorkflowToolOptions",
+ "defaultWebSearch",
+ "defineTool",
+ "defineWorkflowTool",
+ "disableTool",
+ "evaluate",
+ "isDisabledToolSentinel",
+ "isWebSearchToolDefinition",
+ "toolOutput",
+ "toolOutputPart",
+ "toolResultFrom",
+ "webSearch",
+ "workflow"
+ ]
+}
diff --git a/packages/eve/src/compiler/extension-compatibility.ts b/packages/eve/src/compiler/extension-compatibility.ts
index 17184f9cb..2099dfbb3 100644
--- a/packages/eve/src/compiler/extension-compatibility.ts
+++ b/packages/eve/src/compiler/extension-compatibility.ts
@@ -22,8 +22,10 @@ interface ExtensionCapabilityContract {
const EXTENSION_CAPABILITY_CONTRACTS = {
extension: { current: 1, supported: [1], dropped: {} },
tool: {
- current: 46,
- supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 28, 29, 30, 31, 32, 34, 35, 44, 45, 46],
+ current: 47,
+ supported: [
+ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 28, 29, 30, 31, 32, 34, 35, 44, 45, 46, 47,
+ ],
dropped: {
14: "TaskExec.delegated was removed; migrate to workflow-backed background tools",
15: "TaskExec replaces stageEffect with send",
diff --git a/packages/eve/src/experimental/evaluate/auto-model.ts b/packages/eve/src/experimental/evaluate/auto-model.ts
index 56756fef3..489e313b2 100644
--- a/packages/eve/src/experimental/evaluate/auto-model.ts
+++ b/packages/eve/src/experimental/evaluate/auto-model.ts
@@ -1,9 +1,6 @@
import { createHash } from "node:crypto";
-import {
- experimental_evaluate as evaluate,
- type Experimental_EvaluationModel as EvaluationModel,
-} from "ai";
+import { type Experimental_EvaluationModel as EvaluationModel } from "ai";
import { loadContext } from "#context/container.js";
import { ContextKey } from "#context/key.js";
@@ -13,13 +10,14 @@ import {
type DynamicSentinel,
} from "#dynamic/definition.js";
import { isAgentReasoningDefinition, isRuntimeLanguageModel } from "#internal/runtime-model.js";
-import { localGatewayEvaluationModel } from "#internal/model-auth/transport.js";
import type {
AgentReasoningDefinition,
PublicAgentDynamicModelResult,
PublicAgentStaticModelDefinition,
} from "#shared/agent-definition.js";
+import { DEFAULT_EVALUATION_MODEL, evaluate } from "./evaluate.js";
+
type AutoModelOption =
| string
| {
@@ -36,8 +34,6 @@ interface AutoModelConfig<
readonly options: T;
}
-const DEFAULT_EVALUATION_MODEL = "typesafe-ai/jev";
-
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
@@ -169,11 +165,7 @@ export function autoModel<const T extends Readonly<Record<string, AutoModelOptio
if (previous?.turnId === currentTurnId) return models.get(previous.model)!;
const result = await evaluate({
- model:
- typeof evaluationModel === "string" &&
- Reflect.get(globalThis, "AI_SDK_DEFAULT_PROVIDER") == null
- ? (localGatewayEvaluationModel(evaluationModel) ?? evaluationModel)
- : evaluationModel,
+ model: evaluationModel,
state: routingState(ctx),
questions: {
route: {
diff --git a/packages/eve/src/experimental/evaluate/evaluate.integration.test.ts b/packages/eve/src/experimental/evaluate/evaluate.integration.test.ts
new file mode 100644
index 000000000..e0ce1fcf0
--- /dev/null
+++ b/packages/eve/src/experimental/evaluate/evaluate.integration.test.ts
@@ -0,0 +1,136 @@
+import { Experimental_EvaluationMockModelV4 } from "ai/test";
+import { afterEach, beforeEach, describe, expect, expectTypeOf, it, vi } from "vitest";
+
+import { evaluate } from "#public/experimental/evaluate/index.js";
+
+const localEvaluationModel = vi.hoisted(() => vi.fn());
+vi.mock("#internal/model-auth/transport.js", async (importOriginal) => ({
+ ...(await importOriginal<typeof import("#internal/model-auth/transport.js")>()),
+ localGatewayEvaluationModel: localEvaluationModel,
+}));
+
+const questions = {
+ category: {
+ type: "choice",
+ instructions: "Select the team that can help with the request.",
+ criteria: { billing: "Invoices and payments", support: "Product questions" },
+ },
+ urgent: { type: "boolean", instructions: "Does the request require immediate attention?" },
+ priority: { type: "score", instructions: "Rate urgency.", criteria: ["Low", "High"] },
+} as const;
+const state = { request: "Alice needs a copy of her invoice." };
+
+function evaluationModel() {
+ const doEvaluate = vi.fn(async () => ({
+ answers: {
+ category: { type: "choice" as const, choice: "billing" },
+ urgent: { type: "boolean" as const, probability: 0.1 },
+ priority: { type: "score" as const, score: 0.1 },
+ },
+ usage: { inputTokens: 42, outputTokens: 3 },
+ warnings: [],
+ providerMetadata: { fixture: { requestId: "request-1" } },
+ }));
+ return { doEvaluate, model: new Experimental_EvaluationMockModelV4({ doEvaluate }) };
+}
+
+beforeEach(() => {
+ localEvaluationModel.mockReset();
+ vi.stubGlobal("AI_SDK_DEFAULT_PROVIDER", undefined);
+});
+afterEach(() => vi.unstubAllGlobals());
+
+describe("evaluate", () => {
+ it("defaults to Jev using the local Gateway connection without a session context", async () => {
+ const evaluator = evaluationModel();
+ localEvaluationModel.mockReturnValue(evaluator.model);
+
+ const result = await evaluate({ state, questions });
+
+ expect(localEvaluationModel).toHaveBeenCalledWith("typesafe-ai/jev");
+ expect(result.answers.category.choice).toBe("billing");
+ expectTypeOf(result.answers.category.choice).toEqualTypeOf<"billing" | "support">();
+ expectTypeOf(result.answers.urgent.probability).toEqualTypeOf<number>();
+ expectTypeOf(result.answers.priority.score).toEqualTypeOf<number>();
+ expect(result.usage).toEqual({ inputTokens: 42, outputTokens: 3, totalTokens: 45 });
+ expect(result.providerMetadata).toEqual({ fixture: { requestId: "request-1" } });
+ });
+
+ it("resolves an explicit model ID through the local connection", async () => {
+ localEvaluationModel.mockReturnValue(evaluationModel().model);
+ await evaluate({ model: "typesafe-ai/custom", state, questions });
+ expect(localEvaluationModel).toHaveBeenCalledWith("typesafe-ai/custom");
+ });
+
+ it("preserves the configured default provider for defaults and aliases", async () => {
+ const evaluator = evaluationModel();
+ const factory = vi.fn(() => evaluator.model);
+ vi.stubGlobal("AI_SDK_DEFAULT_PROVIDER", { evaluationModel: factory });
+ localEvaluationModel.mockReturnValue(evaluationModel().model);
+
+ await evaluate({ state, questions });
+ await evaluate({ model: "internal-evaluator", state, questions });
+
+ expect(factory.mock.calls).toEqual([["typesafe-ai/jev"], ["internal-evaluator"]]);
+ expect(localEvaluationModel).not.toHaveBeenCalled();
+ });
+
+ it("passes explicit model instances and request options through unchanged", async () => {
+ const evaluator = evaluationModel();
+ const abortSignal = new AbortController().signal;
+ await evaluate({
+ model: evaluator.model,
+ state,
+ questions,
+ abortSignal,
+ headers: { "x-request-id": "request-1" },
+ providerOptions: { fixture: { mode: "fast" } },
+ maxRetries: 0,
+ });
+
+ expect(localEvaluationModel).not.toHaveBeenCalled();
+ expect(evaluator.doEvaluate).toHaveBeenCalledWith(
+ expect.objectContaining({
+ state,
+ questions,
+ abortSignal,
+ headers: expect.objectContaining({ "x-request-id": "request-1" }),
+ providerOptions: { fixture: { mode: "fast" } },
+ }),
+ );
+ });
+
+ it("preserves input and answer validation", async () => {
+ const evaluator = evaluationModel();
+ await expect(evaluate({ model: evaluator.model, state, questions: {} })).rejects.toThrow();
+ expect(evaluator.doEvaluate).not.toHaveBeenCalled();
+
+ const invalid = new Experimental_EvaluationMockModelV4({
+ doEvaluate: async () => ({ answers: {}, warnings: [] }),
+ });
+ await expect(evaluate({ model: invalid, state, questions })).rejects.toThrow(
+ "exactly one answer",
+ );
+ });
+
+ it("propagates provider errors and aborts before provider I/O", async () => {
+ const error = new Error("Evaluation unavailable.");
+ const failed = new Experimental_EvaluationMockModelV4({
+ doEvaluate: async () => {
+ throw error;
+ },
+ });
+ await expect(evaluate({ model: failed, state, questions, maxRetries: 0 })).rejects.toBe(error);
+
+ const evaluator = evaluationModel();
+ await expect(
+ evaluate({
+ model: evaluator.model,
+ state,
+ questions,
+ abortSignal: AbortSignal.abort(error),
+ }),
+ ).rejects.toBe(error);
+ expect(evaluator.doEvaluate).not.toHaveBeenCalled();
+ });
+});
diff --git a/packages/eve/src/experimental/evaluate/evaluate.ts b/packages/eve/src/experimental/evaluate/evaluate.ts
new file mode 100644
index 000000000..06e988dda
--- /dev/null
+++ b/packages/eve/src/experimental/evaluate/evaluate.ts
@@ -0,0 +1,26 @@
+import {
+ experimental_evaluate as evaluateWithAiSdk,
+ type Experimental_EvaluationModel as EvaluationModel,
+ type Experimental_EvaluationQuestion as EvaluationQuestion,
+} from "ai";
+
+import { localGatewayEvaluationModel } from "#internal/model-auth/transport.js";
+
+export const DEFAULT_EVALUATION_MODEL = "typesafe-ai/jev";
+
+/** Evaluate typed questions about the state you pass to it using eve's model authentication. */
+export function evaluate<const Questions extends Record<string, EvaluationQuestion>>({
+ model = DEFAULT_EVALUATION_MODEL,
+ ...options
+}: Omit<Parameters<typeof evaluateWithAiSdk<Questions>>[0], "model"> & {
+ /** Evaluation model instance or ID. Defaults to TypeSafe Jev. */
+ model?: EvaluationModel;
+}) {
+ return evaluateWithAiSdk({
+ ...options,
+ model:
+ typeof model === "string" && Reflect.get(globalThis, "AI_SDK_DEFAULT_PROVIDER") == null
+ ? (localGatewayEvaluationModel(model) ?? model)
+ : model,
+ });
+}
diff --git a/packages/eve/src/experimental/evaluate/package.scenario.test.ts b/packages/eve/src/experimental/evaluate/package.scenario.test.ts
index 9b14eb537..71b018ebe 100644
--- a/packages/eve/src/experimental/evaluate/package.scenario.test.ts
+++ b/packages/eve/src/experimental/evaluate/package.scenario.test.ts
@@ -26,6 +26,20 @@ export const evaluationModel = {
import { autoModel } from "eve/experimental/evaluate";
import { anthropic } from "eve/models/anthropic";
export default defineAgent({ model: autoModel({ options: { "openai/gpt-5.6-sol": "Investigations", my_secret_model: { model: anthropic("sonnet-5"), reasoning: "low", description: "Routine work" } } }) });`,
+ "agent/tools/classify.ts": `import { defineTool } from "eve/tools";
+import { evaluate } from "eve/experimental/evaluate";
+export default defineTool({
+ description: "Classify an incident",
+ inputSchema: { type: "object", properties: {}, additionalProperties: false },
+ async execute(_input, ctx) {
+ const result = await evaluate({
+ state: { incident: "Alice needs an export summary." },
+ questions: { category: { type: "choice", instructions: "Choose a category.", criteria: { routine: "Routine work", investigation: "Investigation" } } },
+ abortSignal: ctx.abortSignal,
+ });
+ return result.answers.category.choice;
+ },
+});`,
"agent/subagents/worker/instructions.md": "Review the assigned evidence.",
"agent/subagents/worker/agent.ts": `import { defineAgent } from "eve";
import { autoModel } from "eve/experimental/evaluate";
diff --git a/packages/eve/src/internal/authored-module-loader.ts b/packages/eve/src/internal/authored-module-loader.ts
index e4f646c50..0820e6b0d 100644
--- a/packages/eve/src/internal/authored-module-loader.ts
+++ b/packages/eve/src/internal/authored-module-loader.ts
@@ -210,7 +210,10 @@ export async function bundleExtensionDistributionGraph(input: {
readonly entries: readonly ExtensionDistributionGraphEntry[];
readonly packageRoot: string;
readonly runtimeDependencies: readonly string[];
-}): Promise<ReadonlyMap<string, string>> {
+}): Promise<{
+ readonly files: ReadonlyMap<string, string>;
+ readonly imports: readonly string[];
+}> {
const plugins = [
createAuthoredDirectiveGuardPlugin(),
createAuthoredRelativeExtensionResolverPlugin({ extensions: RESOLVE_EXTENSIONS }),
@@ -248,12 +251,14 @@ export async function bundleExtensionDistributionGraph(input: {
});
const files = new Map<string, string>();
+ const imports = new Set<string>();
for (const item of result.output) {
if (item.type === "chunk") {
files.set(item.fileName, removeRolldownModuleRegionComments(item.code));
+ for (const specifier of [...item.imports, ...item.dynamicImports]) imports.add(specifier);
}
}
- return files;
+ return { files, imports: [...imports] };
} catch (error) {
throw createAuthoredModuleBundleError(input.packageRoot, error);
}
diff --git a/packages/eve/src/internal/bundler/nitro-rolldown.ts b/packages/eve/src/internal/bundler/nitro-rolldown.ts
index f69dc308b..bf15d5d22 100644
--- a/packages/eve/src/internal/bundler/nitro-rolldown.ts
+++ b/packages/eve/src/internal/bundler/nitro-rolldown.ts
@@ -5,6 +5,8 @@ type RolldownOutputChunk = {
readonly type: "chunk";
readonly code: string;
readonly fileName: string;
+ readonly imports: readonly string[];
+ readonly dynamicImports: readonly string[];
};
type RolldownOutputAsset = {
diff --git a/packages/eve/src/internal/nitro/host/build-extension.scenario.test.ts b/packages/eve/src/internal/nitro/host/build-extension.scenario.test.ts
index d4c3c87e0..f3ddfa027 100644
--- a/packages/eve/src/internal/nitro/host/build-extension.scenario.test.ts
+++ b/packages/eve/src/internal/nitro/host/build-extension.scenario.test.ts
@@ -1,4 +1,4 @@
-import { mkdir, mkdtemp, readFile, symlink, writeFile } from "node:fs/promises";
+import { mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises";
import { createRequire } from "node:module";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
@@ -7,7 +7,9 @@ import { pathToFileURL } from "node:url";
import { describe, expect, it } from "vitest";
import {
+ EXTENSION_CAPABILITY_SUPPORT,
EXTENSION_CAPABILITY_VERSIONS,
+ findUnsupportedExtensionCapabilities,
parseExtensionCompatibilityManifest,
} from "#compiler/extension-compatibility.js";
import {
@@ -81,6 +83,101 @@ describe("extension build output", () => {
);
});
+ it.each([
+ ["static re-export", 'export { evaluate as assess } from "eve/experimental/evaluate";'],
+ [
+ "namespace import",
+ 'import * as evaluation from "eve/experimental/evaluate"; export const assess = evaluation.evaluate;',
+ ],
+ [
+ "dynamic import",
+ 'export async function assess(options: Parameters<typeof import("eve/experimental/evaluate").evaluate>[0]): Promise<void> { await (await import("eve/experimental/evaluate")).evaluate(options); }',
+ ],
+ ])(
+ "stamps evaluation capabilities for a hook-only extension using a %s",
+ async (_name, helper) => {
+ const root = await createExtensionPackage();
+ await rm(join(root, "extension", "tools"), { recursive: true });
+ await mkdir(join(root, "extension", "hooks"));
+ await mkdir(join(root, "extension", "lib"));
+ await writeFile(join(root, "extension", "lib", "evaluation.ts"), helper);
+ await writeFile(
+ join(root, "extension", "hooks", "evaluate.ts"),
+ `import { defineHook } from "eve/hooks";
+import { assess } from "../lib/evaluation";
+export default defineHook({ events: { "turn.started": async () => {
+ await assess({ state: { request: "Alice needs a summary." }, questions: {
+ routine: { type: "boolean", instructions: "Is this routine work?" }
+ } });
+} } });`,
+ );
+ const config = await tryReadExtensionBuildConfig(root);
+ const outDir = await buildExtensionPackage(root, config!);
+ const manifestPath = join(outDir, "extension", "_manifest.json");
+ const manifest = parseExtensionCompatibilityManifest(
+ await readFile(manifestPath, "utf8"),
+ manifestPath,
+ );
+
+ expect(manifest.requires).toEqual({
+ extension: EXTENSION_CAPABILITY_VERSIONS.extension,
+ hook: EXTENSION_CAPABILITY_VERSIONS.hook,
+ tool: EXTENSION_CAPABILITY_VERSIONS.tool,
+ dynamicTool: EXTENSION_CAPABILITY_VERSIONS.dynamicTool,
+ });
+ expect(findUnsupportedExtensionCapabilities(manifest)).toEqual([]);
+ expect(
+ findUnsupportedExtensionCapabilities(manifest, {
+ ...EXTENSION_CAPABILITY_SUPPORT,
+ tool: EXTENSION_CAPABILITY_SUPPORT.tool.filter((version) => version < 47),
+ }),
+ ).toEqual([
+ expect.objectContaining({
+ capability: "tool",
+ requiredVersion: EXTENSION_CAPABILITY_VERSIONS.tool,
+ }),
+ ]);
+ },
+ );
+
+ it.each([
+ [
+ "autoModel",
+ 'import { autoModel } from "eve/experimental/evaluate"; export const route: ReturnType<typeof autoModel> = autoModel({ options: { "openai/small": "Routine work" } });',
+ true,
+ ],
+ [
+ "type-only import",
+ 'import type { evaluate } from "eve/experimental/evaluate"; export type Evaluate = typeof evaluate;',
+ false,
+ ],
+ ])(
+ "tracks runtime evaluation imports in a tool-free helper: %s",
+ async (_name, helper, runtime) => {
+ const root = await createExtensionPackage();
+ await rm(join(root, "extension", "tools"), { recursive: true });
+ await mkdir(join(root, "extension", "lib"));
+ await writeFile(join(root, "extension", "lib", "evaluation.ts"), helper);
+ const config = await tryReadExtensionBuildConfig(root);
+ const outDir = await buildExtensionPackage(root, config!);
+ const manifestPath = join(outDir, "extension", "_manifest.json");
+ const manifest = parseExtensionCompatibilityManifest(
+ await readFile(manifestPath, "utf8"),
+ manifestPath,
+ );
+
+ expect(manifest.requires).toEqual({
+ extension: EXTENSION_CAPABILITY_VERSIONS.extension,
+ ...(runtime
+ ? {
+ tool: EXTENSION_CAPABILITY_VERSIONS.tool,
+ dynamicTool: EXTENSION_CAPABILITY_VERSIONS.dynamicTool,
+ }
+ : {}),
+ });
+ },
+ );
+
it("stamps extension-owned external dependencies into compatibility metadata", async () => {
const root = await createExtensionPackage({
dependencies: { "layout-sensitive-runtime": "1.0.0" },
diff --git a/packages/eve/src/internal/nitro/host/build-extension.ts b/packages/eve/src/internal/nitro/host/build-extension.ts
index 5e4f5fa2f..e119c4158 100644
--- a/packages/eve/src/internal/nitro/host/build-extension.ts
+++ b/packages/eve/src/internal/nitro/host/build-extension.ts
@@ -68,7 +68,7 @@ export async function buildExtensionPackage(
let preserveTransactionRoot = false;
try {
await mkdir(stagedDistRoot, { recursive: true });
- await emitExtensionDistribution({
+ const runtimeImports = await emitExtensionDistribution({
appRoot,
declarationModule,
declarationsRoot: join(transactionRoot, "declarations"),
@@ -88,6 +88,7 @@ export async function buildExtensionPackage(
? {}
: { build: { externalDependencies: config.externalDependencies } }),
requires: await deriveExtensionCapabilityRequirements({
+ runtimeImports,
declarationModule,
manifest,
packageName: config.packageName,
diff --git a/packages/eve/src/internal/nitro/host/extension-capability-requirements.ts b/packages/eve/src/internal/nitro/host/extension-capability-requirements.ts
index 62130d124..adc147125 100644
--- a/packages/eve/src/internal/nitro/host/extension-capability-requirements.ts
+++ b/packages/eve/src/internal/nitro/host/extension-capability-requirements.ts
@@ -21,6 +21,7 @@ export async function deriveExtensionCapabilityRequirements(input: {
readonly manifest: AgentSourceManifest;
readonly packageName: string;
readonly runtimeDependencies: readonly string[];
+ readonly runtimeImports: readonly string[];
readonly shortName: string;
readonly sourceRoot: string;
}): Promise<ExtensionCapabilityRequirements> {
@@ -89,6 +90,11 @@ export async function deriveExtensionCapabilityRequirements(input: {
required.add("config");
}
if (usesState) required.add("state");
+ // The evaluation entrypoint can be used from any contribution or shared helper.
+ if (input.runtimeImports.includes("eve/experimental/evaluate")) {
+ required.add("tool");
+ required.add("dynamicTool");
+ }
return Object.fromEntries(
(Object.keys(EXTENSION_CAPABILITY_VERSIONS) as ExtensionCapability[])
diff --git a/packages/eve/src/internal/nitro/host/extension-distribution.ts b/packages/eve/src/internal/nitro/host/extension-distribution.ts
index ca6c77ad6..f75283d55 100644
--- a/packages/eve/src/internal/nitro/host/extension-distribution.ts
+++ b/packages/eve/src/internal/nitro/host/extension-distribution.ts
@@ -23,7 +23,7 @@ export async function emitExtensionDistribution(input: {
readonly stagedDistRoot: string;
readonly stagedOutDir: string;
readonly transactionRoot: string;
-}): Promise<void> {
+}): Promise<readonly string[]> {
const sourceFiles = await collectExtensionSourceFiles(input.sourceRoot);
const skillPackageRoots = input.manifest.skills
.filter((skill) => skill.sourceKind === "skill-package")
@@ -44,7 +44,7 @@ export async function emitExtensionDistribution(input: {
packageRoot: input.appRoot,
runtimeDependencies: input.runtimeDependencies,
});
- for (const [fileName, code] of emitted) {
+ for (const [fileName, code] of emitted.files) {
const outputPath = join(input.stagedOutDir, fileName);
await mkdir(dirname(outputPath), { recursive: true });
await writeFile(outputPath, code, "utf8");
@@ -69,6 +69,7 @@ export async function emitExtensionDistribution(input: {
);
}
await emitDeclarationBarrels(input);
+ return emitted.imports;
}
/**
diff --git a/packages/eve/src/public/experimental/evaluate/index.ts b/packages/eve/src/public/experimental/evaluate/index.ts
index 77aab51a3..0de9d488c 100644
--- a/packages/eve/src/public/experimental/evaluate/index.ts
+++ b/packages/eve/src/public/experimental/evaluate/index.ts
@@ -1 +1,2 @@
export { autoModel } from "#experimental/evaluate/auto-model.js";
+export { evaluate } from "#experimental/evaluate/evaluate.js";
diff --git a/research/jev-decision-models.md b/research/jev-decision-models.md
index aa6d29ed0..0eb5542fa 100644
--- a/research/jev-decision-models.md
+++ b/research/jev-decision-models.md
@@ -1,24 +1,26 @@
---
issue: "TBD (maintainer-requested research; no matching issue found)"
status: implemented
-last_updated: "2026-09-16"
+last_updated: "2026-09-17"
---
# Evaluation model routing in eve
## Recommendation
-Add only `autoModel` at `eve/experimental/evaluate`. Build it on AI SDK's
-`experimental_evaluate` API instead of owning a TypeSafe client, AI Gateway
-transport, credential resolver, or public decision API. Default to the
-`typesafe-ai/jev` model string. AI SDK model strings use its default
-provider, normally Vercel AI Gateway; explicit model instances use the provider
-package that created them.
+Expose `autoModel` and standalone `evaluate` at `eve/experimental/evaluate`.
+Build both on AI SDK's `experimental_evaluate` API. `evaluate` accepts the SDK's
+state, typed questions, and request options, with an optional model defaulting to
+`typesafe-ai/jev`. It returns the SDK's typed answers and response metadata.
-Keep the first integration narrow: select an allowlisted language model from the
-incoming task. Defer a generic agent-facing decision tool. Asking the main model
-to generate tool arguments adds output tokens and latency before the cheaper
-evaluation can happen.
+`autoModel` calls the shared `evaluate` wrapper. Model strings use the configured
+AI SDK default provider; without an override, the wrapper resolves eve's local
+Gateway connection when available and otherwise leaves Gateway resolution to the
+SDK. Explicit provider instances retain their own authentication.
+
+Tool authors can call `evaluate` with structured state and pass `ctx.abortSignal`.
+Application code can use it without an active eve session. Standalone calls are
+not cached; durable per-turn selection remains specific to `autoModel`.
The API is experimental because AI SDK's evaluation model specification is also
experimental and can change in patch releases. The implemented authoring API is
@@ -160,12 +162,10 @@ direct provider behavior.
## Scope boundaries
-The first release intentionally does not expose `experimental_evaluate`, a
-TypeSafe-specific `decide` function, decision schemas, direct HTTP clients,
-credentials, custom retry logic, provider fallbacks, confidence thresholds, or
-an agent-callable decision tool. Applications that need arbitrary evaluations
-can call AI SDK directly. eve adds only the lifecycle and durable routing policy
-needed to use an evaluation result as an agent model.
+eve owns the `evaluate` wrapper and its default model and authentication behavior.
+AI SDK owns evaluation schemas, validation, retries, errors, and result metadata.
+The integration adds no TypeSafe-specific `decide` function, direct HTTP client,
+provider fallback, confidence threshold, or built-in agent-callable decision tool.
No live Jev inference or independent quality, latency, or cost benchmark was run
for this research. Deterministic tests use AI SDK evaluation model mocks and