The Open Framework for Building Agents
feat(eve): evaluate - expose standalone evaluation with shared auth (#3472)
Andrew Barbadb5cee3725d6
diff --git a/.changeset/quiet-evaluations-share-auth.md b/.changeset/quiet-evaluations-share-auth.mdnew file mode 100644index 000000000..73165b249--- /dev/null+++ b/.changeset/quiet-evaluations-share-auth.md@@ -0,0 +1,5 @@+---+"eve": patch+---++Add standalone `evaluate` to `eve/experimental/evaluate` for typed evaluations in tools and application code. It shares model authentication with `autoModel`, including the Gateway connection selected during `eve dev`.diff --git a/docs/guides/evaluate.md b/docs/guides/evaluate.mdindex 7805ce700..ecb5bf1c9 100644--- a/docs/guides/evaluate.md+++ b/docs/guides/evaluate.md@@ -1,12 +1,13 @@---title: Automatic Model Selection-description: "Choose an agent model from the current request with an AI SDK evaluation model."+description: "Choose agent models automatically or evaluate typed questions in your tools and application code."---Use `autoModel` to choose an agent model from an allowlist before inference begins.It uses the [AI SDK evaluation API](https://ai-sdk.dev/docs/ai-sdk-core/evaluation),so the evaluator can be a Vercel AI Gateway model ID or an evaluation model from-an installed provider.+an installed provider. Use `evaluate` from the same entrypoint to ask typed+questions in your own tools or application code.`eve/experimental/evaluate` is experimental. Its API can change between evereleases, and the AI SDK evaluation model specification can change in patch@@ -103,6 +104,54 @@ Supported reasoning values are `"provider-default"`, `"none"`, `"minimal"`,`"low"`, `"medium"`, `"high"`, and `"xhigh"`. An omitted value inherits theagent's reasoning setting.+## Evaluate inside a tool++Use `evaluate` to ask choice, score, or boolean questions about the state you pass to it.+It defaults to `typesafe-ai/jev` and uses the same authentication as `autoModel`,+including the Gateway connection selected through `/login` during `eve dev`.+Pass `model` to use another evaluation model ID or a provider instance. A configured+AI SDK default provider takes precedence over the local Gateway connection.++```ts title="agent/tools/classify-request.ts"+import { evaluate } from "eve/experimental/evaluate";+import { defineTool } from "eve/tools";+import { z } from "zod";++export default defineTool({+ description: "Choose the team that can help with a customer request.",+ inputSchema: z.object({ request: z.string().min(1).max(8000) }),+ async execute({ request }, ctx) {+ const result = await evaluate({+ state: { request },+ questions: {+ team: {+ type: "choice",+ instructions: "Select the team best suited to handle the request.",+ criteria: {+ billing: "Invoices, payments, and refunds",+ support: "Product questions and troubleshooting",+ },+ },+ },+ abortSignal: ctx.abortSignal,+ });+ return { team: result.answers.team.choice };+ },+});+```++The choice above is typed as `"billing" | "support"`. Each question appears under+its authored key in `result.answers`. Results also include token usage, warnings,+provider metadata, and response metadata.++`evaluate` accepts AI SDK evaluation options, including `maxRetries`, `headers`,+and `providerOptions`. Pass an `abortSignal` to cancel the request. Input and+answer validation, retries, and provider errors follow AI SDK semantics.++You can also call `evaluate` outside a tool; it does not require an active eve+session. Each call performs its own evaluation. `autoModel` uses this function+and adds the per-turn routing behavior described below.+## Runtime behavior`autoModel` evaluates at the first `step.started` event, after the incoming promptdiff --git a/docs/reference/typescript-api.md b/docs/reference/typescript-api.mdindex e13484d45..deb344b1f 100644--- a/docs/reference/typescript-api.md+++ b/docs/reference/typescript-api.md@@ -137,7 +137,7 @@ import template from "../../prompts/template.txt?raw";| `eve/local-dev` | `getLocalDevCapability`, `LocalDevCapability` || `eve/models/openai` | `openai`, `chatgpt`, deprecated `experimental_chatgpt` || `eve/models/anthropic` | `anthropic` |-| [`eve/experimental/evaluate`](../guides/evaluate) | Experimental `autoModel` |+| [`eve/experimental/evaluate`](../guides/evaluate) | Experimental `autoModel` and standalone `evaluate` || `eve/evals` | `defineEval`, `defineEvalConfig`, `mockModel`, eval types || `eve/evals/expect` | `includes`, `equals`, `matches`, `similarity` || `eve/evals/reporters` | `Braintrust`, `JUnit`, `EvalReporter` |diff --git a/e2e/fixtures/agent-evaluate/agent/agent.ts b/e2e/fixtures/agent-evaluate/agent/agent.tsindex 1eb2b9f14..55ac687b1 100644--- a/e2e/fixtures/agent-evaluate/agent/agent.ts+++ b/e2e/fixtures/agent-evaluate/agent/agent.ts@@ -8,6 +8,21 @@ const { experimental } = e2eAgentConfig();export default defineAgent({experimental,model: fixtureModel(async (request) => {+ if (request.userMessages.some((text) => text.includes("evaluate-request"))) {+ const result = request.toolResults.find((result) => result.name === "evaluate-request");+ if (result) return JSON.stringify({ isError: result.isError, output: result.output });+ return {+ toolCalls: [+ {+ id: "evaluate-request-1",+ name: "evaluate-request",+ input: {+ missingAnswer: request.userMessages.some((text) => text.includes("missing answer")),+ },+ },+ ],+ };+ }if (request.userMessages.some((text) => text.includes("parallel investigations"))) {const completed = request.messages.filter((message) =>message.text.includes("child-result:"),diff --git a/e2e/fixtures/agent-evaluate/agent/testing.ts b/e2e/fixtures/agent-evaluate/agent/testing.tsindex a192ff4d7..8387e273d 100644--- a/e2e/fixtures/agent-evaluate/agent/testing.ts+++ b/e2e/fixtures/agent-evaluate/agent/testing.ts@@ -10,7 +10,7 @@ export const routing = defineState("evaluate-fixture.routing", () => ({reasoning: "unselected",}));-const evaluationModel = {+export const evaluationModel: Exclude<Experimental_EvaluationModel, string> = {specificationVersion: "v4",provider: "fixture",modelId: "fixture-evaluator",@@ -37,7 +37,7 @@ const evaluationModel = {response: { modelId: "fixture-evaluator" },};},-} satisfies Exclude<Experimental_EvaluationModel, string>;+};/** Run the real router with deterministic evaluation and language models. */export function fixtureModel(respond: MockModelResponder) {diff --git a/e2e/fixtures/agent-evaluate/agent/tools/evaluate-request.ts b/e2e/fixtures/agent-evaluate/agent/tools/evaluate-request.tsnew file mode 100644index 000000000..0c5d46951--- /dev/null+++ b/e2e/fixtures/agent-evaluate/agent/tools/evaluate-request.ts@@ -0,0 +1,34 @@+import { evaluate } from "eve/experimental/evaluate";+import { defineTool } from "eve/tools";++import { evaluationModel } from "../testing";++export default defineTool({+ description: "Evaluate Alice's request with the fixture evaluation provider.",+ inputSchema: {+ type: "object",+ properties: { missingAnswer: { type: "boolean" } },+ required: ["missingAnswer"],+ additionalProperties: false,+ },+ async execute({ missingAnswer }, ctx) {+ const result = await evaluate({+ model: missingAnswer+ ? { ...evaluationModel, doEvaluate: async () => ({ answers: {}, warnings: [] }) }+ : evaluationModel,+ state: { request: "Alice needs a routine summary." },+ questions: {+ route: {+ type: "choice",+ instructions: "Choose a model for the request.",+ criteria: {+ "openai/large": "Difficult investigations",+ "openai/small": "Routine requests",+ },+ },+ },+ abortSignal: ctx.abortSignal,+ });+ return { choice: result.answers.route.choice, usage: result.usage };+ },+});diff --git a/e2e/fixtures/agent-evaluate/evals/tool-evaluation-failure.eval.ts b/e2e/fixtures/agent-evaluate/evals/tool-evaluation-failure.eval.tsnew file mode 100644index 000000000..ee08d77b7--- /dev/null+++ b/e2e/fixtures/agent-evaluate/evals/tool-evaluation-failure.eval.ts@@ -0,0 +1,14 @@+import { defineEval } from "eve/evals";++export default defineEval({+ description: "An invalid evaluation response is reported as a tool error.",+ async test(t) {+ const turn = await t.send(+ "Bob uses evaluate-request to review a missing answer from the evaluation service.",+ );+ turn.expectOk();+ turn.calledTool("evaluate-request", { status: "failed", count: 1 });+ turn.messageIncludes('"isError":true');+ t.succeeded();+ },+});diff --git a/e2e/fixtures/agent-evaluate/evals/tool-evaluation.eval.ts b/e2e/fixtures/agent-evaluate/evals/tool-evaluation.eval.tsnew file mode 100644index 000000000..210c66ffa--- /dev/null+++ b/e2e/fixtures/agent-evaluate/evals/tool-evaluation.eval.ts@@ -0,0 +1,16 @@+import { defineEval } from "eve/evals";++export default defineEval({+ description: "A tool evaluates structured state and returns typed answers and usage.",+ async test(t) {+ const turn = await t.send(+ "Alice uses evaluate-request to choose a model for her routine summary.",+ );+ turn.expectOk();+ turn.calledTool("evaluate-request");+ turn.messageIncludes('"isError":false');+ turn.messageIncludes('"choice":"openai/small"');+ turn.messageIncludes('"totalTokens":45');+ t.succeeded();+ },+});diff --git a/packages/eve/extension-contracts/compatibility/tool/v46.ts b/packages/eve/extension-contracts/compatibility/tool/v46.tsnew file mode 100644index 000000000..27fa92d63--- /dev/null+++ b/packages/eve/extension-contracts/compatibility/tool/v46.ts@@ -0,0 +1,9 @@+import { defineTool } from "#public/tools/index.js";++export default defineTool({+ description: "Report whether the current tool execution was cancelled.",+ inputSchema: { type: "object", properties: {}, additionalProperties: false },+ execute(_input, ctx) {+ return { cancelled: ctx.abortSignal?.aborted ?? false };+ },+});diff --git a/packages/eve/extension-contracts/entrypoints/tool.ts b/packages/eve/extension-contracts/entrypoints/tool.tsindex 5849681a1..cb7611f78 100644--- a/packages/eve/extension-contracts/entrypoints/tool.ts+++ b/packages/eve/extension-contracts/entrypoints/tool.ts@@ -18,3 +18,4 @@ export {type WorkflowToolInput,type WorkflowToolOptions,} from "../../src/public/tools/workflow.ts";+export { evaluate } from "../../src/public/experimental/evaluate/index.ts";diff --git a/packages/eve/extension-contracts/reports/tool/v47.json b/packages/eve/extension-contracts/reports/tool/v47.jsonnew file mode 100644index 000000000..72cde997d--- /dev/null+++ b/packages/eve/extension-contracts/reports/tool/v47.json@@ -0,0 +1,23 @@+{+ "kind": "eve-extension-capability-contract",+ "capability": "tool",+ "epoch": 47,+ "sha256": "c8f3dea743ed9d8c175769c1249d099bc5265beda83104a8edb9ff841c7f0766",+ "exports": [+ "WorkflowTool",+ "WorkflowToolInput",+ "WorkflowToolOptions",+ "defaultWebSearch",+ "defineTool",+ "defineWorkflowTool",+ "disableTool",+ "evaluate",+ "isDisabledToolSentinel",+ "isWebSearchToolDefinition",+ "toolOutput",+ "toolOutputPart",+ "toolResultFrom",+ "webSearch",+ "workflow"+ ]+}diff --git a/packages/eve/src/compiler/extension-compatibility.ts b/packages/eve/src/compiler/extension-compatibility.tsindex 17184f9cb..2099dfbb3 100644--- a/packages/eve/src/compiler/extension-compatibility.ts+++ b/packages/eve/src/compiler/extension-compatibility.ts@@ -22,8 +22,10 @@ interface ExtensionCapabilityContract {const EXTENSION_CAPABILITY_CONTRACTS = {extension: { current: 1, supported: [1], dropped: {} },tool: {- current: 46,- supported: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 28, 29, 30, 31, 32, 34, 35, 44, 45, 46],+ current: 47,+ supported: [+ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 28, 29, 30, 31, 32, 34, 35, 44, 45, 46, 47,+ ],dropped: {14: "TaskExec.delegated was removed; migrate to workflow-backed background tools",15: "TaskExec replaces stageEffect with send",diff --git a/packages/eve/src/experimental/evaluate/auto-model.ts b/packages/eve/src/experimental/evaluate/auto-model.tsindex 56756fef3..489e313b2 100644--- a/packages/eve/src/experimental/evaluate/auto-model.ts+++ b/packages/eve/src/experimental/evaluate/auto-model.ts@@ -1,9 +1,6 @@import { createHash } from "node:crypto";-import {- experimental_evaluate as evaluate,- type Experimental_EvaluationModel as EvaluationModel,-} from "ai";+import { type Experimental_EvaluationModel as EvaluationModel } from "ai";import { loadContext } from "#context/container.js";import { ContextKey } from "#context/key.js";@@ -13,13 +10,14 @@ import {type DynamicSentinel,} from "#dynamic/definition.js";import { isAgentReasoningDefinition, isRuntimeLanguageModel } from "#internal/runtime-model.js";-import { localGatewayEvaluationModel } from "#internal/model-auth/transport.js";import type {AgentReasoningDefinition,PublicAgentDynamicModelResult,PublicAgentStaticModelDefinition,} from "#shared/agent-definition.js";+import { DEFAULT_EVALUATION_MODEL, evaluate } from "./evaluate.js";+type AutoModelOption =| string| {@@ -36,8 +34,6 @@ interface AutoModelConfig<readonly options: T;}-const DEFAULT_EVALUATION_MODEL = "typesafe-ai/jev";-function isRecord(value: unknown): value is Record<string, unknown> {return typeof value === "object" && value !== null && !Array.isArray(value);}@@ -169,11 +165,7 @@ export function autoModel<const T extends Readonly<Record<string, AutoModelOptioif (previous?.turnId === currentTurnId) return models.get(previous.model)!;const result = await evaluate({- model:- typeof evaluationModel === "string" &&- Reflect.get(globalThis, "AI_SDK_DEFAULT_PROVIDER") == null- ? (localGatewayEvaluationModel(evaluationModel) ?? evaluationModel)- : evaluationModel,+ model: evaluationModel,state: routingState(ctx),questions: {route: {diff --git a/packages/eve/src/experimental/evaluate/evaluate.integration.test.ts b/packages/eve/src/experimental/evaluate/evaluate.integration.test.tsnew file mode 100644index 000000000..e0ce1fcf0--- /dev/null+++ b/packages/eve/src/experimental/evaluate/evaluate.integration.test.ts@@ -0,0 +1,136 @@+import { Experimental_EvaluationMockModelV4 } from "ai/test";+import { afterEach, beforeEach, describe, expect, expectTypeOf, it, vi } from "vitest";++import { evaluate } from "#public/experimental/evaluate/index.js";++const localEvaluationModel = vi.hoisted(() => vi.fn());+vi.mock("#internal/model-auth/transport.js", async (importOriginal) => ({+ ...(await importOriginal<typeof import("#internal/model-auth/transport.js")>()),+ localGatewayEvaluationModel: localEvaluationModel,+}));++const questions = {+ category: {+ type: "choice",+ instructions: "Select the team that can help with the request.",+ criteria: { billing: "Invoices and payments", support: "Product questions" },+ },+ urgent: { type: "boolean", instructions: "Does the request require immediate attention?" },+ priority: { type: "score", instructions: "Rate urgency.", criteria: ["Low", "High"] },+} as const;+const state = { request: "Alice needs a copy of her invoice." };++function evaluationModel() {+ const doEvaluate = vi.fn(async () => ({+ answers: {+ category: { type: "choice" as const, choice: "billing" },+ urgent: { type: "boolean" as const, probability: 0.1 },+ priority: { type: "score" as const, score: 0.1 },+ },+ usage: { inputTokens: 42, outputTokens: 3 },+ warnings: [],+ providerMetadata: { fixture: { requestId: "request-1" } },+ }));+ return { doEvaluate, model: new Experimental_EvaluationMockModelV4({ doEvaluate }) };+}++beforeEach(() => {+ localEvaluationModel.mockReset();+ vi.stubGlobal("AI_SDK_DEFAULT_PROVIDER", undefined);+});+afterEach(() => vi.unstubAllGlobals());++describe("evaluate", () => {+ it("defaults to Jev using the local Gateway connection without a session context", async () => {+ const evaluator = evaluationModel();+ localEvaluationModel.mockReturnValue(evaluator.model);++ const result = await evaluate({ state, questions });++ expect(localEvaluationModel).toHaveBeenCalledWith("typesafe-ai/jev");+ expect(result.answers.category.choice).toBe("billing");+ expectTypeOf(result.answers.category.choice).toEqualTypeOf<"billing" | "support">();+ expectTypeOf(result.answers.urgent.probability).toEqualTypeOf<number>();+ expectTypeOf(result.answers.priority.score).toEqualTypeOf<number>();+ expect(result.usage).toEqual({ inputTokens: 42, outputTokens: 3, totalTokens: 45 });+ expect(result.providerMetadata).toEqual({ fixture: { requestId: "request-1" } });+ });++ it("resolves an explicit model ID through the local connection", async () => {+ localEvaluationModel.mockReturnValue(evaluationModel().model);+ await evaluate({ model: "typesafe-ai/custom", state, questions });+ expect(localEvaluationModel).toHaveBeenCalledWith("typesafe-ai/custom");+ });++ it("preserves the configured default provider for defaults and aliases", async () => {+ const evaluator = evaluationModel();+ const factory = vi.fn(() => evaluator.model);+ vi.stubGlobal("AI_SDK_DEFAULT_PROVIDER", { evaluationModel: factory });+ localEvaluationModel.mockReturnValue(evaluationModel().model);++ await evaluate({ state, questions });+ await evaluate({ model: "internal-evaluator", state, questions });++ expect(factory.mock.calls).toEqual([["typesafe-ai/jev"], ["internal-evaluator"]]);+ expect(localEvaluationModel).not.toHaveBeenCalled();+ });++ it("passes explicit model instances and request options through unchanged", async () => {+ const evaluator = evaluationModel();+ const abortSignal = new AbortController().signal;+ await evaluate({+ model: evaluator.model,+ state,+ questions,+ abortSignal,+ headers: { "x-request-id": "request-1" },+ providerOptions: { fixture: { mode: "fast" } },+ maxRetries: 0,+ });++ expect(localEvaluationModel).not.toHaveBeenCalled();+ expect(evaluator.doEvaluate).toHaveBeenCalledWith(+ expect.objectContaining({+ state,+ questions,+ abortSignal,+ headers: expect.objectContaining({ "x-request-id": "request-1" }),+ providerOptions: { fixture: { mode: "fast" } },+ }),+ );+ });++ it("preserves input and answer validation", async () => {+ const evaluator = evaluationModel();+ await expect(evaluate({ model: evaluator.model, state, questions: {} })).rejects.toThrow();+ expect(evaluator.doEvaluate).not.toHaveBeenCalled();++ const invalid = new Experimental_EvaluationMockModelV4({+ doEvaluate: async () => ({ answers: {}, warnings: [] }),+ });+ await expect(evaluate({ model: invalid, state, questions })).rejects.toThrow(+ "exactly one answer",+ );+ });++ it("propagates provider errors and aborts before provider I/O", async () => {+ const error = new Error("Evaluation unavailable.");+ const failed = new Experimental_EvaluationMockModelV4({+ doEvaluate: async () => {+ throw error;+ },+ });+ await expect(evaluate({ model: failed, state, questions, maxRetries: 0 })).rejects.toBe(error);++ const evaluator = evaluationModel();+ await expect(+ evaluate({+ model: evaluator.model,+ state,+ questions,+ abortSignal: AbortSignal.abort(error),+ }),+ ).rejects.toBe(error);+ expect(evaluator.doEvaluate).not.toHaveBeenCalled();+ });+});diff --git a/packages/eve/src/experimental/evaluate/evaluate.ts b/packages/eve/src/experimental/evaluate/evaluate.tsnew file mode 100644index 000000000..06e988dda--- /dev/null+++ b/packages/eve/src/experimental/evaluate/evaluate.ts@@ -0,0 +1,26 @@+import {+ experimental_evaluate as evaluateWithAiSdk,+ type Experimental_EvaluationModel as EvaluationModel,+ type Experimental_EvaluationQuestion as EvaluationQuestion,+} from "ai";++import { localGatewayEvaluationModel } from "#internal/model-auth/transport.js";++export const DEFAULT_EVALUATION_MODEL = "typesafe-ai/jev";++/** Evaluate typed questions about the state you pass to it using eve's model authentication. */+export function evaluate<const Questions extends Record<string, EvaluationQuestion>>({+ model = DEFAULT_EVALUATION_MODEL,+ ...options+}: Omit<Parameters<typeof evaluateWithAiSdk<Questions>>[0], "model"> & {+ /** Evaluation model instance or ID. Defaults to TypeSafe Jev. */+ model?: EvaluationModel;+}) {+ return evaluateWithAiSdk({+ ...options,+ model:+ typeof model === "string" && Reflect.get(globalThis, "AI_SDK_DEFAULT_PROVIDER") == null+ ? (localGatewayEvaluationModel(model) ?? model)+ : model,+ });+}diff --git a/packages/eve/src/experimental/evaluate/package.scenario.test.ts b/packages/eve/src/experimental/evaluate/package.scenario.test.tsindex 9b14eb537..71b018ebe 100644--- a/packages/eve/src/experimental/evaluate/package.scenario.test.ts+++ b/packages/eve/src/experimental/evaluate/package.scenario.test.ts@@ -26,6 +26,20 @@ export const evaluationModel = {import { autoModel } from "eve/experimental/evaluate";import { anthropic } from "eve/models/anthropic";export default defineAgent({ model: autoModel({ options: { "openai/gpt-5.6-sol": "Investigations", my_secret_model: { model: anthropic("sonnet-5"), reasoning: "low", description: "Routine work" } } }) });`,+ "agent/tools/classify.ts": `import { defineTool } from "eve/tools";+import { evaluate } from "eve/experimental/evaluate";+export default defineTool({+ description: "Classify an incident",+ inputSchema: { type: "object", properties: {}, additionalProperties: false },+ async execute(_input, ctx) {+ const result = await evaluate({+ state: { incident: "Alice needs an export summary." },+ questions: { category: { type: "choice", instructions: "Choose a category.", criteria: { routine: "Routine work", investigation: "Investigation" } } },+ abortSignal: ctx.abortSignal,+ });+ return result.answers.category.choice;+ },+});`,"agent/subagents/worker/instructions.md": "Review the assigned evidence.","agent/subagents/worker/agent.ts": `import { defineAgent } from "eve";import { autoModel } from "eve/experimental/evaluate";diff --git a/packages/eve/src/internal/authored-module-loader.ts b/packages/eve/src/internal/authored-module-loader.tsindex e4f646c50..0820e6b0d 100644--- a/packages/eve/src/internal/authored-module-loader.ts+++ b/packages/eve/src/internal/authored-module-loader.ts@@ -210,7 +210,10 @@ export async function bundleExtensionDistributionGraph(input: {readonly entries: readonly ExtensionDistributionGraphEntry[];readonly packageRoot: string;readonly runtimeDependencies: readonly string[];-}): Promise<ReadonlyMap<string, string>> {+}): Promise<{+ readonly files: ReadonlyMap<string, string>;+ readonly imports: readonly string[];+}> {const plugins = [createAuthoredDirectiveGuardPlugin(),createAuthoredRelativeExtensionResolverPlugin({ extensions: RESOLVE_EXTENSIONS }),@@ -248,12 +251,14 @@ export async function bundleExtensionDistributionGraph(input: {});const files = new Map<string, string>();+ const imports = new Set<string>();for (const item of result.output) {if (item.type === "chunk") {files.set(item.fileName, removeRolldownModuleRegionComments(item.code));+ for (const specifier of [...item.imports, ...item.dynamicImports]) imports.add(specifier);}}- return files;+ return { files, imports: [...imports] };} catch (error) {throw createAuthoredModuleBundleError(input.packageRoot, error);}diff --git a/packages/eve/src/internal/bundler/nitro-rolldown.ts b/packages/eve/src/internal/bundler/nitro-rolldown.tsindex f69dc308b..bf15d5d22 100644--- a/packages/eve/src/internal/bundler/nitro-rolldown.ts+++ b/packages/eve/src/internal/bundler/nitro-rolldown.ts@@ -5,6 +5,8 @@ type RolldownOutputChunk = {readonly type: "chunk";readonly code: string;readonly fileName: string;+ readonly imports: readonly string[];+ readonly dynamicImports: readonly string[];};type RolldownOutputAsset = {diff --git a/packages/eve/src/internal/nitro/host/build-extension.scenario.test.ts b/packages/eve/src/internal/nitro/host/build-extension.scenario.test.tsindex d4c3c87e0..f3ddfa027 100644--- a/packages/eve/src/internal/nitro/host/build-extension.scenario.test.ts+++ b/packages/eve/src/internal/nitro/host/build-extension.scenario.test.ts@@ -1,4 +1,4 @@-import { mkdir, mkdtemp, readFile, symlink, writeFile } from "node:fs/promises";+import { mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises";import { createRequire } from "node:module";import { tmpdir } from "node:os";import { dirname, join } from "node:path";@@ -7,7 +7,9 @@ import { pathToFileURL } from "node:url";import { describe, expect, it } from "vitest";import {+ EXTENSION_CAPABILITY_SUPPORT,EXTENSION_CAPABILITY_VERSIONS,+ findUnsupportedExtensionCapabilities,parseExtensionCompatibilityManifest,} from "#compiler/extension-compatibility.js";import {@@ -81,6 +83,101 @@ describe("extension build output", () => {);});+ it.each([+ ["static re-export", 'export { evaluate as assess } from "eve/experimental/evaluate";'],+ [+ "namespace import",+ 'import * as evaluation from "eve/experimental/evaluate"; export const assess = evaluation.evaluate;',+ ],+ [+ "dynamic import",+ 'export async function assess(options: Parameters<typeof import("eve/experimental/evaluate").evaluate>[0]): Promise<void> { await (await import("eve/experimental/evaluate")).evaluate(options); }',+ ],+ ])(+ "stamps evaluation capabilities for a hook-only extension using a %s",+ async (_name, helper) => {+ const root = await createExtensionPackage();+ await rm(join(root, "extension", "tools"), { recursive: true });+ await mkdir(join(root, "extension", "hooks"));+ await mkdir(join(root, "extension", "lib"));+ await writeFile(join(root, "extension", "lib", "evaluation.ts"), helper);+ await writeFile(+ join(root, "extension", "hooks", "evaluate.ts"),+ `import { defineHook } from "eve/hooks";+import { assess } from "../lib/evaluation";+export default defineHook({ events: { "turn.started": async () => {+ await assess({ state: { request: "Alice needs a summary." }, questions: {+ routine: { type: "boolean", instructions: "Is this routine work?" }+ } });+} } });`,+ );+ const config = await tryReadExtensionBuildConfig(root);+ const outDir = await buildExtensionPackage(root, config!);+ const manifestPath = join(outDir, "extension", "_manifest.json");+ const manifest = parseExtensionCompatibilityManifest(+ await readFile(manifestPath, "utf8"),+ manifestPath,+ );++ expect(manifest.requires).toEqual({+ extension: EXTENSION_CAPABILITY_VERSIONS.extension,+ hook: EXTENSION_CAPABILITY_VERSIONS.hook,+ tool: EXTENSION_CAPABILITY_VERSIONS.tool,+ dynamicTool: EXTENSION_CAPABILITY_VERSIONS.dynamicTool,+ });+ expect(findUnsupportedExtensionCapabilities(manifest)).toEqual([]);+ expect(+ findUnsupportedExtensionCapabilities(manifest, {+ ...EXTENSION_CAPABILITY_SUPPORT,+ tool: EXTENSION_CAPABILITY_SUPPORT.tool.filter((version) => version < 47),+ }),+ ).toEqual([+ expect.objectContaining({+ capability: "tool",+ requiredVersion: EXTENSION_CAPABILITY_VERSIONS.tool,+ }),+ ]);+ },+ );++ it.each([+ [+ "autoModel",+ 'import { autoModel } from "eve/experimental/evaluate"; export const route: ReturnType<typeof autoModel> = autoModel({ options: { "openai/small": "Routine work" } });',+ true,+ ],+ [+ "type-only import",+ 'import type { evaluate } from "eve/experimental/evaluate"; export type Evaluate = typeof evaluate;',+ false,+ ],+ ])(+ "tracks runtime evaluation imports in a tool-free helper: %s",+ async (_name, helper, runtime) => {+ const root = await createExtensionPackage();+ await rm(join(root, "extension", "tools"), { recursive: true });+ await mkdir(join(root, "extension", "lib"));+ await writeFile(join(root, "extension", "lib", "evaluation.ts"), helper);+ const config = await tryReadExtensionBuildConfig(root);+ const outDir = await buildExtensionPackage(root, config!);+ const manifestPath = join(outDir, "extension", "_manifest.json");+ const manifest = parseExtensionCompatibilityManifest(+ await readFile(manifestPath, "utf8"),+ manifestPath,+ );++ expect(manifest.requires).toEqual({+ extension: EXTENSION_CAPABILITY_VERSIONS.extension,+ ...(runtime+ ? {+ tool: EXTENSION_CAPABILITY_VERSIONS.tool,+ dynamicTool: EXTENSION_CAPABILITY_VERSIONS.dynamicTool,+ }+ : {}),+ });+ },+ );+it("stamps extension-owned external dependencies into compatibility metadata", async () => {const root = await createExtensionPackage({dependencies: { "layout-sensitive-runtime": "1.0.0" },diff --git a/packages/eve/src/internal/nitro/host/build-extension.ts b/packages/eve/src/internal/nitro/host/build-extension.tsindex 5e4f5fa2f..e119c4158 100644--- a/packages/eve/src/internal/nitro/host/build-extension.ts+++ b/packages/eve/src/internal/nitro/host/build-extension.ts@@ -68,7 +68,7 @@ export async function buildExtensionPackage(let preserveTransactionRoot = false;try {await mkdir(stagedDistRoot, { recursive: true });- await emitExtensionDistribution({+ const runtimeImports = await emitExtensionDistribution({appRoot,declarationModule,declarationsRoot: join(transactionRoot, "declarations"),@@ -88,6 +88,7 @@ export async function buildExtensionPackage(? {}: { build: { externalDependencies: config.externalDependencies } }),requires: await deriveExtensionCapabilityRequirements({+ runtimeImports,declarationModule,manifest,packageName: config.packageName,diff --git a/packages/eve/src/internal/nitro/host/extension-capability-requirements.ts b/packages/eve/src/internal/nitro/host/extension-capability-requirements.tsindex 62130d124..adc147125 100644--- a/packages/eve/src/internal/nitro/host/extension-capability-requirements.ts+++ b/packages/eve/src/internal/nitro/host/extension-capability-requirements.ts@@ -21,6 +21,7 @@ export async function deriveExtensionCapabilityRequirements(input: {readonly manifest: AgentSourceManifest;readonly packageName: string;readonly runtimeDependencies: readonly string[];+ readonly runtimeImports: readonly string[];readonly shortName: string;readonly sourceRoot: string;}): Promise<ExtensionCapabilityRequirements> {@@ -89,6 +90,11 @@ export async function deriveExtensionCapabilityRequirements(input: {required.add("config");}if (usesState) required.add("state");+ // The evaluation entrypoint can be used from any contribution or shared helper.+ if (input.runtimeImports.includes("eve/experimental/evaluate")) {+ required.add("tool");+ required.add("dynamicTool");+ }return Object.fromEntries((Object.keys(EXTENSION_CAPABILITY_VERSIONS) as ExtensionCapability[])diff --git a/packages/eve/src/internal/nitro/host/extension-distribution.ts b/packages/eve/src/internal/nitro/host/extension-distribution.tsindex ca6c77ad6..f75283d55 100644--- a/packages/eve/src/internal/nitro/host/extension-distribution.ts+++ b/packages/eve/src/internal/nitro/host/extension-distribution.ts@@ -23,7 +23,7 @@ export async function emitExtensionDistribution(input: {readonly stagedDistRoot: string;readonly stagedOutDir: string;readonly transactionRoot: string;-}): Promise<void> {+}): Promise<readonly string[]> {const sourceFiles = await collectExtensionSourceFiles(input.sourceRoot);const skillPackageRoots = input.manifest.skills.filter((skill) => skill.sourceKind === "skill-package")@@ -44,7 +44,7 @@ export async function emitExtensionDistribution(input: {packageRoot: input.appRoot,runtimeDependencies: input.runtimeDependencies,});- for (const [fileName, code] of emitted) {+ for (const [fileName, code] of emitted.files) {const outputPath = join(input.stagedOutDir, fileName);await mkdir(dirname(outputPath), { recursive: true });await writeFile(outputPath, code, "utf8");@@ -69,6 +69,7 @@ export async function emitExtensionDistribution(input: {);}await emitDeclarationBarrels(input);+ return emitted.imports;}/**diff --git a/packages/eve/src/public/experimental/evaluate/index.ts b/packages/eve/src/public/experimental/evaluate/index.tsindex 77aab51a3..0de9d488c 100644--- a/packages/eve/src/public/experimental/evaluate/index.ts+++ b/packages/eve/src/public/experimental/evaluate/index.ts@@ -1 +1,2 @@export { autoModel } from "#experimental/evaluate/auto-model.js";+export { evaluate } from "#experimental/evaluate/evaluate.js";diff --git a/research/jev-decision-models.md b/research/jev-decision-models.mdindex aa6d29ed0..0eb5542fa 100644--- a/research/jev-decision-models.md+++ b/research/jev-decision-models.md@@ -1,24 +1,26 @@---issue: "TBD (maintainer-requested research; no matching issue found)"status: implemented-last_updated: "2026-09-16"+last_updated: "2026-09-17"---# Evaluation model routing in eve## Recommendation-Add only `autoModel` at `eve/experimental/evaluate`. Build it on AI SDK's-`experimental_evaluate` API instead of owning a TypeSafe client, AI Gateway-transport, credential resolver, or public decision API. Default to the-`typesafe-ai/jev` model string. AI SDK model strings use its default-provider, normally Vercel AI Gateway; explicit model instances use the provider-package that created them.+Expose `autoModel` and standalone `evaluate` at `eve/experimental/evaluate`.+Build both on AI SDK's `experimental_evaluate` API. `evaluate` accepts the SDK's+state, typed questions, and request options, with an optional model defaulting to+`typesafe-ai/jev`. It returns the SDK's typed answers and response metadata.-Keep the first integration narrow: select an allowlisted language model from the-incoming task. Defer a generic agent-facing decision tool. Asking the main model-to generate tool arguments adds output tokens and latency before the cheaper-evaluation can happen.+`autoModel` calls the shared `evaluate` wrapper. Model strings use the configured+AI SDK default provider; without an override, the wrapper resolves eve's local+Gateway connection when available and otherwise leaves Gateway resolution to the+SDK. Explicit provider instances retain their own authentication.++Tool authors can call `evaluate` with structured state and pass `ctx.abortSignal`.+Application code can use it without an active eve session. Standalone calls are+not cached; durable per-turn selection remains specific to `autoModel`.The API is experimental because AI SDK's evaluation model specification is alsoexperimental and can change in patch releases. The implemented authoring API is@@ -160,12 +162,10 @@ direct provider behavior.## Scope boundaries-The first release intentionally does not expose `experimental_evaluate`, a-TypeSafe-specific `decide` function, decision schemas, direct HTTP clients,-credentials, custom retry logic, provider fallbacks, confidence thresholds, or-an agent-callable decision tool. Applications that need arbitrary evaluations-can call AI SDK directly. eve adds only the lifecycle and durable routing policy-needed to use an evaluation result as an agent model.+eve owns the `evaluate` wrapper and its default model and authentication behavior.+AI SDK owns evaluation schemas, validation, retries, errors, and result metadata.+The integration adds no TypeSafe-specific `decide` function, direct HTTP client,+provider fallback, confidence threshold, or built-in agent-callable decision tool.No live Jev inference or independent quality, latency, or cost benchmark was runfor this research. Deterministic tests use AI SDK evaluation model mocks and