Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

feat(eve): evals - use evaluation models for judges (#3531)

Andrew Barba8e011906e06c
diff --git a/.changeset/evaluation-model-judges.md b/.changeset/evaluation-model-judges.md
new file mode 100644
index 000000000..1fb4daa80
--- /dev/null
+++ b/.changeset/evaluation-model-judges.md
@@ -0,0 +1,5 @@
+---
+"eve": minor
+---
+
+Replace `t.judge.autoevals.*` with `t.judge(...)`, supporting criteria, typed questions, and batches through evaluation models with a default of `typesafe-ai/jev`. Configure provider evaluation model instances instead of language model instances; autoevals is removed while deterministic similarity and Braintrust reporting remain available.
diff --git a/docs/evals/assertions.mdx b/docs/evals/assertions.mdx
index 8d9dbca50..f85eea8bb 100644
--- a/docs/evals/assertions.mdx
+++ b/docs/evals/assertions.mdx
@@ -162,7 +162,7 @@ Every assertion returns a chainable handle. Severity rides on the assertion, so
- `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).
- `.label(name)` adds a stable name when an eval records several assertions from the same family.
-The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge.*` assertion are soft. Annotate only when you deviate:
+The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge(...)` assertion are soft. Annotate only when you deviate:
```ts
t.calledTool("get_weather").soft(); // record the tool call as a metric, don't gate
diff --git a/docs/evals/cases.mdx b/docs/evals/cases.mdx
index 051b8c7e2..eae7abb3f 100644
--- a/docs/evals/cases.mdx
+++ b/docs/evals/cases.mdx
@@ -70,7 +70,7 @@ export default defineEval({
async test(t) {
const draft = await t.send("Draft the follow-up email.");
t.check(draft.message, includes("Best regards"));
- t.judge.autoevals.closedQA("professional tone", { on: draft.message }).atLeast(0.6);
+ t.judge("professional tone", { on: draft.message }).atLeast(0.6);
await draft.session.send("Now send it.");
t.calledTool("send_email");
@@ -94,11 +94,9 @@ export default defineEval({
t.succeeded();
second.messageIncludes("marigold");
- t.judge.autoevals
- .closedQA("The assistant remembers the user's favorite word across turns", {
- on: first.session.transcript,
- })
- .atLeast(0.8);
+ t.judge("The assistant remembers the user's favorite word across turns", {
+ on: first.session.transcript,
+ }).atLeast(0.8);
},
});
```
diff --git a/docs/evals/judge.mdx b/docs/evals/judge.mdx
index 57a4c84b3..47b19153a 100644
--- a/docs/evals/judge.mdx
+++ b/docs/evals/judge.mdx
@@ -1,9 +1,9 @@
---
title: "Judge"
-description: "Grade evals with an LLM judge via t.judge.autoevals, set thresholds on the assertion, and configure the judge model."
+description: "Grade evals with evaluation models using criteria, typed questions, or batches, and set thresholds on each assertion."
---
-When no deterministic [assertion](./assertions) captures what "good" means (factual correctness, summary quality, free-form criteria), grade the run with an LLM judge. The `t.judge.*` assertions are the only model-backed ones, and they use a judge model that is resolved separately from the agent under test. eve only uses it for scoring, never to swap out the agent.
+Use `t.judge(...)` when a deterministic [assertion](./assertions) cannot capture what a good response looks like. It calls [`evaluate` from `eve/ai`](../guides/evaluate#evaluate-inside-a-tool) and defaults to `typesafe-ai/jev`. The judge is separate from the agent under test and is used only for scoring.
```ts
import { defineEval } from "eve/evals";
@@ -12,110 +12,153 @@ export default defineEval({
async test(t) {
await t.send("Explain quantum tunneling to a 10-year-old.");
t.succeeded();
- t.judge.autoevals.closedQA("uses no math beyond arithmetic").atLeast(0.8);
+ t.judge("The response uses no math beyond arithmetic.").atLeast(0.8);
},
});
```
-## The graders
+No judge configuration is required. Configure credentials for the evaluator as described under [Configuring the judge model](#configuring-the-judge-model).
-The judges live under `t.judge.autoevals`. The namespace names the [Braintrust autoevals](https://github.com/braintrustdata/autoevals) grader family, so the factuality and closedQA semantics are autoevals', not eve-invented. Each grader scores the most recently settled turn’s assistant message by default and is soft by default (tracked, no gate):
+## The graders
-| Grader | Grades |
-| ---------------------------------------- | -------------------------------------------------------------------------------------- |
-| `t.judge.autoevals.factuality(expected)` | Factual consistency of the reply against an expected answer (A–E buckets) |
-| `t.judge.autoevals.summarizes(expected)` | How well the reply summarizes the expected text |
-| `t.judge.autoevals.closedQA(criteria)` | Whether the reply satisfies a free-form yes/no criterion (no expected answer to match) |
-| `t.judge.autoevals.sql(expected)` | Semantic equivalence of two SQL statements |
+`t.judge` accepts a criteria string, one typed question, or a batch of named questions. A string becomes a boolean question asking whether the response meets those criteria. A single question returns an assertion handle; a batch returns one handle per question.
-Judges receive the latest message text from the session whose turn most recently settled as the input prompt. Approval responses and stream reads preserve that prompt. For multipart messages, including `session.sendFile()`, the prompt contains the text parts.
+| Question | Assertion score |
+| ---------------------------- | --------------------------------------------------------------- |
+| Criteria string or `boolean` | The returned probability of true, from 0 to 1 |
+| `score` | The returned rubric position divided by the highest level index |
+| `choice` | 1 when the selected option equals `expected`; otherwise 0 |
-A failed `closedQA` assertion prints the input and judge explanation:
+Boolean scores are model estimates, not guaranteed calibrated confidence. A rubric needs at least two levels, ordered from worst to best. The SDK can return a fractional position on that rubric; eve normalizes it to the 0–1 scale used by assertion thresholds.
-```text
-✗ judge.autoevals.closedQA [citation] (0% < 80%): prompt: "Name the source."
- criteria: "cites a source"
- response: "It is widely believed."
- rationale: "The response does not cite a source."
- choice: "N"
+```ts
+t.judge({
+ type: "score",
+ instructions: "Grade the response's clarity.",
+ criteria: ["Unclear", "Mostly clear", "Clear and concise"],
+})
+ .label("clarity")
+ .atLeast(0.75);
```
-The reference or criteria is the positional argument. An options object follows:
+Choice questions require an `expected` option key. eve uses that expectation locally for scoring; it does not send it to the evaluator. Instructions and rubric descriptions accept strings, JSON objects, or JSON arrays. Descriptions can also be `null`.
+
+## Choose the value to grade
-- `on` is the value to grade, defaulting to the most recently settled turn’s assistant message. Pass an intermediate draft or parsed value to grade it instead.
-- `model` and `modelOptions` are a per-call judge override (see below).
+By default, the evaluator receives `{ input, output }`: the latest prompt and assistant message from the session whose turn most recently settled. Approval responses and stream reads preserve that prompt. For multipart messages, including `session.sendFile()`, the prompt contains the text parts.
+
+Pass `on` to replace the output. JSON values stay structured rather than being converted to strings:
```ts
const draft = await t.send("Draft the welcome email.");
-t.judge.autoevals.closedQA("professional tone", { on: draft.message }).atLeast(0.6);
+t.judge("The response has a professional tone.", { on: draft.message }).atLeast(0.6);
```
-For a multi-turn eval, pass `session.transcript` to grade a session's complete observed conversation instead of only its final reply:
+For a multi-turn eval, pass `session.transcript`:
```ts
const { session } = await t.send("My favorite word is marigold. Remember it.");
await session.send("What is my favorite word?");
+t.judge("The assistant remembers the user's favorite word across turns.", {
+ on: session.transcript,
+}).atLeast(0.8);
+```
+
+The transcript contains user and assistant messages in turn order, excluding reasoning, tool calls, and tool results. See [Multi-turn evals](./cases#multi-turn-evals).
-t.judge.autoevals
- .closedQA("The assistant remembers the user's favorite word across turns", {
- on: session.transcript,
- })
- .atLeast(0.8);
+## Evaluate several questions together
+
+A batch makes one evaluation request against one shared state. Pass `state` to replace the default `{ input, output }` completely. Batch calls use `state` instead of `on`.
+
+```ts
+const judgments = t.judge({
+ state: { response: turn.message ?? "", reference },
+ questions: {
+ accurate: {
+ type: "boolean",
+ instructions: "Is the response consistent with the reference?",
+ },
+ outcome: {
+ type: "choice",
+ instructions: "Classify the response.",
+ criteria: {
+ answered: "Answers the request",
+ declined: "Declines the request",
+ },
+ expected: "answered",
+ },
+ },
+});
+judgments.accurate.atLeast(0.9);
+judgments.outcome.gate();
```
-`session.transcript` contains the session's user and assistant messages in turn order. It excludes reasoning, tool calls, and tool results. See [Multi-turn evals](./cases#multi-turn-evals) for the transcript format and independent sessions.
+Keys identify questions and their assertion handles; instructions must explain what to judge. Every question sees the same state. Use separate calls for unrelated inputs. A batch must contain at least one question, and a provider or validation failure fails every assertion in the batch.
## Soft scoring and thresholds
-Judge assertions are soft, so the threshold rides on the assertion handle. There is no separate thresholds map:
+Judges are soft by default:
-- **No threshold** is tracked-only. The score lands in reports and artifacts and never fails the eval. Use it to watch a metric without gating on it.
-- `.atLeast(threshold)` is a soft bar. A below-threshold score marks the eval `scored`, fatal only under `eve eval --strict`.
-- `.gate(threshold)` promotes a judge to a hard gate that fails the eval outright.
+- Without a threshold, the score is tracked in reports and artifacts without failing the eval.
+- `.atLeast(threshold)` sets a soft bar. A lower score marks the eval `scored`, fatal only under `eve eval --strict`.
+- `.gate(threshold)` makes a lower score fail the eval outright. An omitted gate threshold is 1.
```ts
-t.judge.autoevals.closedQA("cites a source"); // tracked, never fails
-t.judge.autoevals.closedQA("cites a source").label("citation").atLeast(0.6);
-t.judge.autoevals.factuality(reference).gate(0.8); // hard gate at 0.8
+t.judge("The response cites a source.");
+t.judge("The response cites a source.").label("citation").atLeast(0.8);
+t.judge("The response cites a source.").gate(0.9);
```
-A judge runs once per assertion and burns tokens, so reach for one only when nothing deterministic will do. Judge calls start when recorded, and the runner waits for all of them during finalization; assertion handles themselves are intentionally not awaitable.
+A boolean probability need not reach exactly 1 even for a good response, so choose an explicit gate threshold for boolean judgments. Choice scores are binary.
+
+Calls capture their state and questions and start immediately. The runner waits for them during finalization; handles are not awaitable. Each standalone call makes its own request, while a batch shares one request. Eval timeouts cancel pending judgments and stop waiting even if a provider ignores cancellation.
## Configuring the judge model
-The judge model is resolved once when the runner builds `t`. It is **never** the model under test. Three levels resolve innermost-wins:
+Model settings resolve in this order:
-1. **Per-call**: `t.judge.autoevals.closedQA("…", { model, modelOptions })`.
-2. **Per-eval**: `defineEval({ judge: { model, modelOptions }, test })`.
-3. **Project default**: `defineEvalConfig({ judge: { model, modelOptions } })` in `evals.config.ts`.
+1. Per-call: `t.judge("…", { model, modelOptions })`.
+2. Per-eval: `defineEval({ judge: { model, modelOptions }, test })`.
+3. Project default: `defineEvalConfig({ judge: { model, modelOptions } })`.
+4. The default model used by `evaluate`, currently `typesafe-ai/jev`.
+
+A per-eval `judge` replaces the project's judge configuration. Per-call fields override the resolved configuration. `judge` and its `model` are optional; deterministic evals make no evaluation requests.
```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
export default defineEvalConfig({
- judge: { model: "openai/gpt-5.6-luna" }, // the default judge for every eval in this tree
+ judge: { model: "typesafe-ai/jev" },
});
```
-```ts title="evals/quantum.eval.ts"
-import { defineEval } from "eve/evals";
+String IDs resolve through the AI SDK default provider, or Vercel AI Gateway when none is configured. Gateway uses `AI_GATEWAY_API_KEY` or Vercel OIDC credentials; eve's existing local Gateway connection is also honored by `evaluate`. An explicit provider evaluation model uses that provider's credentials:
-export default defineEval({
- judge: { model: "anthropic/claude-opus-4.8" }, // a stronger judge for this eval
- async test(t) {
- await t.send("Explain quantum tunneling to a 10-year-old.");
- t.judge.autoevals.factuality(reference).atLeast(0.7);
- t.judge.autoevals.closedQA("is concise", { model: "anthropic/claude-haiku-4.5" }); // cheaper, per-call
- },
+```ts
+import { openai } from "@ai-sdk/openai";
+
+t.judge("The response addresses the request.", {
+ model: openai.evaluationModel("gpt-5.6-luna"),
+ modelOptions: { providerOptions: { openai: { reasoningEffort: "high" } } },
});
```
-`judge` in `evals.config.ts` is optional, and a tree of fully deterministic evals can omit it. Calling `t.judge.*` with no judge model resolved records a failed gate: the runner scores the assertion after the `test` function runs, the missing model throws, and the eval fails with that message.
+Use the provider's `evaluationModel` factory, not a language model instance. AI SDK supplies evaluation adapters for supported language models. Provider support and the evaluation contract are experimental; see [AI SDK evaluation](https://ai-sdk.dev/docs/ai-sdk-core/evaluation).
+
+String model IDs resolve through the default provider's evaluation API. Gateway accepts native evaluation models such as `typesafe-ai/jev`; passing a language model ID such as `openai/gpt-5.6-luna` does not automatically select the OpenAI adapter. Use an evaluation-model instance as shown above for language models.
+
+Missing credentials, invalid answers, unsupported question types, and provider errors become failed gates even for tracked-only judgments. They do not silently skip the eval or select another model. Validation, authentication, and retries use the shared `evaluate` implementation.
+
+## Diagnostics and migration
+
+Reports use `judge.boolean`, `judge.score`, and `judge.choice`; batch assertion names also include their question key. Diagnostics include the captured state, question, and raw answer. Assertion metadata retains the normalized score, model identity, token usage, warnings, and available provider and response metadata. Usage for a batch is marked as shared across its assertions, not a separate charge for each question. Evaluation models do not promise a prose rationale.
+
+The autoevals graders have been removed. Replace `t.judge.autoevals.closedQA(criteria, options)` with `t.judge(criteria, options)`. Scores now represent a probability rather than a binary yes/no answer, so review thresholds against your examples. For factuality, summaries, or SQL equivalence, write an explicit question and include the reference in shared state. No old grader buckets are preserved.
-A **string model id** (e.g. `"anthropic/claude-opus-4.8"`) routes through the Vercel AI Gateway and needs `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` in the environment. An **AI SDK `LanguageModel` instance** is used directly. With a model configured but no credentials, a judge-backed eval **skips visibly** rather than failing, so the run reports the skip instead of a spurious error. For provider-specific judge settings, use `modelOptions.providerOptions`.
+Change configured language model instances to evaluation model instances. `modelOptions.providerOptions`, assertion labels, and threshold methods remain available. [Braintrust reporting](./reporters) continues to accept judge scores independently of the grader implementation.
## What to read next
- [Assertions](./assertions): deterministic run-level and value assertions
-- [Reporters](./reporters): ship judged scores to Braintrust experiments
-- [Targets](./targets): local vs remote targets for judge-backed evals
+- [Reporters](./reporters): publish scores and inspect results
+- [Targets](./targets): local and remote eval targets
diff --git a/docs/evals/overview.mdx b/docs/evals/overview.mdx
index 3e44f5063..e95c6e382 100644
--- a/docs/evals/overview.mdx
+++ b/docs/evals/overview.mdx
@@ -51,12 +51,12 @@ import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";
export default defineEvalConfig({
- judge: { model: "openai/gpt-5.6-luna" },
+ judge: { model: "typesafe-ai/jev" },
reporters: [Braintrust({ projectName: "my-agent" })],
});
```
-Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge.*`); a tree of fully deterministic evals can omit it. `reporters`, `maxConcurrency`, and `timeoutMs` round out the defaults. Config `reporters` observe every eval in the run, so set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults.
+Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge(...)`); omit it to use the default evaluator, `typesafe-ai/jev`. Deterministic evals make no judge requests. `reporters`, `maxConcurrency`, and `timeoutMs` round out the defaults. Config `reporters` observe every eval in the run, so set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults.
## Deterministic fixture models
@@ -109,22 +109,24 @@ Each surface matches a genuinely different kind of judgment:
- **Scoped methods** read the final whole run on `t`, snapshot one independent session when invoked there, or inspect one immutable `EveEvalTurn`. See [Assertions](./assertions).
- **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect`, such as `t.check(turn.message, includes("sunny"))`. Grade `turn.message`, an intermediate draft, parsed JSON, or anything else. See [Assertions](./assertions).
-- **`t.judge.autoevals.*`** is the LLM-as-judge surface, like `t.judge.autoevals.closedQA("cites a source")`. It grades the most recently settled turn’s assistant message by default; pass `{ on: turn.session.transcript }` to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See [Judge](./judge).
+- **`t.judge(...)`** is the LLM-as-judge surface, like `t.judge("cites a source")`. It grades the most recently settled turn’s assistant message by default; pass `{ on: turn.session.transcript }` to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See [Judge](./judge).
## Gate vs soft
Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.
- **Gates** are hard. A failed gate marks the eval `failed` and `eve eval` exits non-zero. Run-level methods, `includes`, `equals`, and `matches` are gates by default.
-- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity` and every `t.judge.*` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.
+- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity` and every `t.judge(...)` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.
Override per assertion: `.gate(threshold?)` promotes to a hard gate, `.soft(threshold?)` demotes to tracked, and `.atLeast(threshold)` is a soft assertion with a bar.
```ts
t.succeeded(); // gate
t.calledTool("get_weather").soft(); // record as a metric, don't gate
-t.judge.autoevals.closedQA("cites a source"); // soft, tracked (no threshold)
-t.judge.autoevals.factuality(reference).atLeast(0.7); // soft, gated under --strict at 0.7
+t.judge("cites a source"); // soft, tracked (no threshold)
+t.judge("The response is consistent with output.reference.", {
+ on: { response: turn.message ?? "", reference },
+}).atLeast(0.7); // soft, gated under --strict at 0.7
```
Use `await t.require(value, assertion)` for a gate that must pass before the script can safely continue. Use `t.skip(reason)` as the first operation for an intentionally unsupported target capability.
diff --git a/docs/evals/reporters.mdx b/docs/evals/reporters.mdx
index 114e72c6f..6438552ff 100644
--- a/docs/evals/reporters.mdx
+++ b/docs/evals/reporters.mdx
@@ -18,7 +18,7 @@ import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";
export default defineEvalConfig({
- judge: { model: "openai/gpt-5.6-luna" },
+ judge: { model: "typesafe-ai/jev" },
reporters: [Braintrust({ projectName: "weather-agent" })],
});
```
diff --git a/docs/guides/evaluate.md b/docs/guides/evaluate.md
index 9be5603bc..8d0c362b3 100644
--- a/docs/guides/evaluate.md
+++ b/docs/guides/evaluate.md
@@ -151,6 +151,10 @@ You can also call `evaluate` outside a tool; it does not require an active eve
session. Each call performs its own evaluation. `auto` uses this function
and adds the per-turn routing behavior described below.
+## Judge eval results
+
+Eval authors can use `t.judge(...)` to turn evaluation answers into scored assertions, including batches of questions sharing one state. It uses this same `evaluate` implementation and default model. See [Judge](../evals/judge) for criteria, rubrics, and thresholds.
+
## Evaluate tool approvals
Use `auto({ model? })` when an evaluation model should decide whether a
diff --git a/e2e/README.md b/e2e/README.md
index e6aeecc22..0c7831693 100644
--- a/e2e/README.md
+++ b/e2e/README.md
@@ -30,7 +30,6 @@ Fixture agents author their harness-owned configuration through the private
without root-only settings.
- `e2eModel({ mock? })` — a bare model handle for nested slots such as
compaction models and dynamic selections.
-- `e2eJudgeModel()` — the judge model for `evals.config.ts`.
When `EVE_E2E_MODEL=mock`, all of these return a deterministic `mockModel()`
instead of a gateway model id. The default responder echoes the last user
@@ -128,8 +127,10 @@ must run against the alias — the `e2e-vercel` workflow sets
evals as a second `eve eval` invocation after the main suite. Without the
alias env (local matrix, plain `eve eval --strict`) the eval skips.
-Most fixture agents and their configured judges resolve `EVE_E2E_MODEL`
+Most fixture agents resolve `EVE_E2E_MODEL`
through `@eve-e2e/config`, defaulting to `openai/gpt-5.6-sol` for local runs.
+Fixture eval configs use the shared `e2eJudgeModel()` helper, which returns an OpenAI evaluation-model instance for `openai/gpt-5.6-luna` until CI has access to Jev. The adapter uses Gateway's Responses endpoint and `AI_GATEWAY_API_KEY`, independently of the agent matrix. A bare Luna string targets Gateway's native evaluation API and is not supported. Deterministic judge coverage in `agent-evaluate` passes a fixture evaluation model explicitly.
+
`agent-workflow-stress` uses eve's `mockModel` fixture helper so its 100-turn
runs stay fast and deterministic. Its concurrent and sequential evals cover
high-volume session execution and repeated session resumption respectively.
diff --git a/e2e/fixtures/agent-basic-runtime/evals/evals.config.ts b/e2e/fixtures/agent-basic-runtime/evals/evals.config.ts
index bae56fd2a..5c971dfbe 100644
--- a/e2e/fixtures/agent-basic-runtime/evals/evals.config.ts
+++ b/e2e/fixtures/agent-basic-runtime/evals/evals.config.ts
@@ -2,7 +2,6 @@ import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
import { evalLifecycleReporter } from "./reporter.js";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
reporters: [evalLifecycleReporter],
diff --git a/e2e/fixtures/agent-cancellation/evals/evals.config.ts b/e2e/fixtures/agent-cancellation/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-cancellation/evals/evals.config.ts
+++ b/e2e/fixtures/agent-cancellation/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-channels/evals/evals.config.ts b/e2e/fixtures/agent-channels/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-channels/evals/evals.config.ts
+++ b/e2e/fixtures/agent-channels/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-evaluate/evals/judge.eval.ts b/e2e/fixtures/agent-evaluate/evals/judge.eval.ts
new file mode 100644
index 000000000..9207d1acf
--- /dev/null
+++ b/e2e/fixtures/agent-evaluate/evals/judge.eval.ts
@@ -0,0 +1,60 @@
+import type { Experimental_EvaluationModel } from "ai";
+import { defineEval } from "eve/evals";
+
+const model: Exclude<Experimental_EvaluationModel, string> = {
+ specificationVersion: "v4",
+ provider: "fixture",
+ modelId: "fixture-judge",
+ supportedQuestionTypes: ["boolean", "score", "choice"],
+ async doEvaluate({ state, questions }) {
+ const routine = JSON.stringify(state).includes("openai/small");
+ return {
+ answers: Object.fromEntries(
+ Object.entries(questions).map(([id, question]) => {
+ if (question.type === "boolean")
+ return [id, { type: "boolean", probability: routine ? 0.9 : 0.1 }];
+ if (question.type === "score") return [id, { type: "score", score: routine ? 1.5 : 0 }];
+ return [id, { type: "choice", choice: routine ? "small" : "large" }];
+ }),
+ ),
+ usage: { inputTokens: 20, outputTokens: 3 },
+ warnings: [],
+ };
+ },
+};
+
+export default defineEval({
+ description: "Single and batched judges score a fixture response through an evaluation model.",
+ judge: { model },
+ async test(t) {
+ const turn = await t.send("Alice needs a routine summary of the selected model.");
+ turn.expectOk();
+ turn.messageIncludes("openai/small");
+ t.judge("The response selects the small model for Alice's routine request.").gate(0.8);
+ t.judge({
+ type: "score",
+ instructions: "Rate whether the response identifies the selected model.",
+ criteria: ["Missing", "Partly identified", "Fully identified"],
+ }).gate(0.75);
+ const judgments = t.judge({
+ state: { response: turn.message ?? "" },
+ questions: {
+ identifiesModel: {
+ type: "boolean",
+ instructions: "Does the response identify the small model?",
+ },
+ route: {
+ type: "choice",
+ instructions: "Which model does the response identify?",
+ criteria: { small: "The small model", large: "The large model" },
+ expected: "small",
+ },
+ },
+ });
+ judgments.identifiesModel.gate(0.8);
+ judgments.route.gate();
+ t.judge("The conversation identifies the small model.", { on: turn.session.transcript }).gate(
+ 0.8,
+ );
+ },
+});
diff --git a/e2e/fixtures/agent-model/evals/evals.config.ts b/e2e/fixtures/agent-model/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-model/evals/evals.config.ts
+++ b/e2e/fixtures/agent-model/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-openapi-swagger/evals/evals.config.ts b/e2e/fixtures/agent-openapi-swagger/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-openapi-swagger/evals/evals.config.ts
+++ b/e2e/fixtures/agent-openapi-swagger/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-schedules/evals/evals.config.ts b/e2e/fixtures/agent-schedules/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-schedules/evals/evals.config.ts
+++ b/e2e/fixtures/agent-schedules/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-self-modification/evals/self-modification/offer-repair.eval.ts b/e2e/fixtures/agent-self-modification/evals/self-modification/offer-repair.eval.ts
index 3754df3f9..bf228cce0 100644
--- a/e2e/fixtures/agent-self-modification/evals/self-modification/offer-repair.eval.ts
+++ b/e2e/fixtures/agent-self-modification/evals/self-modification/offer-repair.eval.ts
@@ -66,7 +66,7 @@ export default defineEval({
"That recommendation is wrong. Incoming stock should count toward the target, so it should recommend 50 units, not 80. What should we do?",
);
offered.notEvent("subagent.called", { data: { name: "self-modification__agent" } });
- t.judge.autoevals.closedQA(
+ t.judge(
"The response offers to delegate a repair of the incorrect tool and asks the user to confirm before starting it.",
{ on: offered.message },
);
diff --git a/e2e/fixtures/agent-session-limits/evals/evals.config.ts b/e2e/fixtures/agent-session-limits/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-session-limits/evals/evals.config.ts
+++ b/e2e/fixtures/agent-session-limits/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-skills/evals/evals.config.ts b/e2e/fixtures/agent-skills/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-skills/evals/evals.config.ts
+++ b/e2e/fixtures/agent-skills/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-subagents-hitl/evals/evals.config.ts b/e2e/fixtures/agent-subagents-hitl/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-subagents-hitl/evals/evals.config.ts
+++ b/e2e/fixtures/agent-subagents-hitl/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-subagents/agent/subagents/self-modification/agent.ts b/e2e/fixtures/agent-subagents/agent/subagents/self-modification/agent.ts
index 16314d4b5..b9917e71f 100644
--- a/e2e/fixtures/agent-subagents/agent/subagents/self-modification/agent.ts
+++ b/e2e/fixtures/agent-subagents/agent/subagents/self-modification/agent.ts
@@ -4,5 +4,7 @@ import { defineAgent } from "eve";
export default defineAgent({
...e2eSubagentConfig(),
description:
- "Delegate requests to change the eve agent, add reusable actions, or install and connect named capabilities. This acceptance-only child does not modify source or call tools.",
+ "Delegate requests to change the eve agent, add reusable actions, or install and connect named capabilities. " +
+ "Treat questions about whether you can install, add, enable, or connect to a named product or service as requests to extend this eve agent and delegate immediately. The child handles installation scope and setup clarification. " +
+ "This acceptance-only child does not modify source or call tools.",
});
diff --git a/e2e/fixtures/agent-subagents/evals/evals.config.ts b/e2e/fixtures/agent-subagents/evals/evals.config.ts
index acb8837a7..681c7fc01 100644
--- a/e2e/fixtures/agent-subagents/evals/evals.config.ts
+++ b/e2e/fixtures/agent-subagents/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
// Deterministic model turns reach nested Workflow waits immediately. Keep
diff --git a/e2e/fixtures/agent-tools-hitl-openai/evals/evals.config.ts b/e2e/fixtures/agent-tools-hitl-openai/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-tools-hitl-openai/evals/evals.config.ts
+++ b/e2e/fixtures/agent-tools-hitl-openai/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-tools-hitl/evals/evals.config.ts b/e2e/fixtures/agent-tools-hitl/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/evals.config.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/deny-then-regate.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/deny-then-regate.eval.ts
index bec12d3f6..1c695b247 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/deny-then-regate.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/deny-then-regate.eval.ts
@@ -31,14 +31,9 @@ export default defineEval({
});
// The denial returns to the model as context; real models paraphrase it,
// so judge the acknowledgment instead of matching literal wording.
- t.judge.autoevals
- .closedQA(
- "The reply acknowledges that the guarded-echo tool call was denied and did not run.",
- {
- on: denied.message,
- },
- )
- .atLeast(0.5);
+ t.judge("The reply acknowledges that the guarded-echo tool call was denied and did not run.", {
+ on: denied.message,
+ }).atLeast(0.5);
await session.send('Call the guarded-echo tool once more with note "retry-call".');
// Denial does not grant: the follow-up call must re-park.
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/stale-ask-question-selection-idle.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/stale-ask-question-selection-idle.eval.ts
index 8994eeb06..7f8e90933 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/stale-ask-question-selection-idle.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/stale-ask-question-selection-idle.eval.ts
@@ -50,12 +50,10 @@ export default defineEval({
// The stale selection reaches the model as context it may act on or
// disregard; judge that the reply engages with it instead of demanding
// a literal echo the model can rightly decline.
- t.judge.autoevals
- .closedQA(
- "The reply treats the message as a late response to an EARLIER question — either applying that earlier selection, or explaining that the earlier question was already answered or is no longer relevant (stale). Answer yes unless the reply ignores the message entirely or is unrelated to it.",
- { on: staleSelection.message },
- )
- .atLeast(0.5);
+ t.judge(
+ "The reply treats the message as a late response to an EARLIER question — either applying that earlier selection, or explaining that the earlier question was already answered or is no longer relevant (stale). Answer yes unless the reply ignores the message entirely or is unrelated to it.",
+ { on: staleSelection.message },
+ ).atLeast(0.5);
t.succeeded();
},
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/stale-ask-question-selection.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/stale-ask-question-selection.eval.ts
index 81f41e9b4..e077a629b 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/stale-ask-question-selection.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/stale-ask-question-selection.eval.ts
@@ -70,12 +70,10 @@ export default defineEval({
// The stale selection reaches the model as context it may act on or
// disregard; judge that the reply engages with it instead of demanding
// a literal echo the model can rightly decline.
- t.judge.autoevals
- .closedQA(
- "The reply treats the message as a late response to an EARLIER question — either applying that earlier selection, or explaining that the earlier question was already answered or is no longer relevant (stale). Answer yes unless the reply instead treats the message as the answer to the current pending question or ignores it entirely.",
- { on: staleSelection.message },
- )
- .atLeast(0.5);
+ t.judge(
+ "The reply treats the message as a late response to an EARLIER question — either applying that earlier selection, or explaining that the earlier question was already answered or is no longer relevant (stale). Answer yes unless the reply instead treats the message as the answer to the current pending question or ignores it entirely.",
+ { on: staleSelection.message },
+ ).atLeast(0.5);
t.succeeded();
},
diff --git a/e2e/fixtures/agent-tools-sandbox/evals/evals.config.ts b/e2e/fixtures/agent-tools-sandbox/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-tools-sandbox/evals/evals.config.ts
+++ b/e2e/fixtures/agent-tools-sandbox/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/agent-tools/evals/evals.config.ts b/e2e/fixtures/agent-tools/evals/evals.config.ts
index 15fbd91d9..7a61a5b55 100644
--- a/e2e/fixtures/agent-tools/evals/evals.config.ts
+++ b/e2e/fixtures/agent-tools/evals/evals.config.ts
@@ -1,7 +1,6 @@
import { e2eJudgeModel } from "@eve-e2e/config";
import { defineEvalConfig } from "eve/evals";
-/** Default judge model for any `t.judge.*` assertion in this fixture. */
export default defineEvalConfig({
judge: { model: e2eJudgeModel() },
});
diff --git a/e2e/fixtures/e2e-config/package.json b/e2e/fixtures/e2e-config/package.json
index c1398a659..d3f33222d 100644
--- a/e2e/fixtures/e2e-config/package.json
+++ b/e2e/fixtures/e2e-config/package.json
@@ -9,9 +9,11 @@
"./instrumentation-otel": "./src/instrumentation-otel.ts"
},
"scripts": {
+ "test:integration": "node --test test/*.integration.test.mjs",
"typecheck": "tsc --noEmit"
},
"dependencies": {
+ "@ai-sdk/openai": "catalog:",
"@opentelemetry/sdk-trace-base": "catalog:",
"@vercel/otel": "catalog:",
"eve": "workspace:*"
diff --git a/e2e/fixtures/e2e-config/src/index.ts b/e2e/fixtures/e2e-config/src/index.ts
index 9c2235156..1520fb03d 100644
--- a/e2e/fixtures/e2e-config/src/index.ts
+++ b/e2e/fixtures/e2e-config/src/index.ts
@@ -1,6 +1,8 @@
import type { AgentDefinition, AgentStaticModelDefinition } from "eve";
import { mockModel, type MockModelResponder } from "eve/evals";
+export { e2eJudgeModel } from "./judge.js";
+
/**
* Sentinel value for `EVE_E2E_MODEL` that makes fixtures author a
* deterministic `mockModel()` instead of a real gateway model. The world
@@ -92,20 +94,5 @@ export function e2eSubagentConfig(options?: E2EModelOptions): E2ESubagentConfig
return { model, modelContextWindowTokens: MOCK_MODEL_CONTEXT_WINDOW_TOKENS };
}
-/**
- * Resolves the judge model for a fixture's `evals.config.ts`. Judge
- * assertions only run in the model suite, so the mock sentinel falls back to
- * the harness default rather than leaking `"mock"` as a gateway model id.
- */
-export function e2eJudgeModel(): string {
- const requested = process.env.EVE_E2E_MODEL;
-
- if (requested === undefined || requested === MOCK_MODEL_SENTINEL) {
- return DEFAULT_MODEL;
- }
-
- return requested;
-}
-
const defaultMockResponder: MockModelResponder = ({ lastUserMessage }) =>
`Mock reply: ${lastUserMessage ?? ""}`;
diff --git a/e2e/fixtures/e2e-config/src/judge.ts b/e2e/fixtures/e2e-config/src/judge.ts
new file mode 100644
index 000000000..36ee89b77
--- /dev/null
+++ b/e2e/fixtures/e2e-config/src/judge.ts
@@ -0,0 +1,10 @@
+import { createOpenAI } from "@ai-sdk/openai";
+import type { EveEvalJudgeConfig } from "eve/evals";
+
+/** Shared fixture judge, independent of the agent matrix, until CI has access to Jev. */
+export function e2eJudgeModel(): Exclude<EveEvalJudgeConfig["model"], string | undefined> {
+ return createOpenAI({
+ apiKey: process.env.AI_GATEWAY_API_KEY,
+ baseURL: "https://ai-gateway.vercel.sh/v1",
+ }).evaluationModel("openai/gpt-5.6-luna");
+}
diff --git a/e2e/fixtures/e2e-config/test/judge.integration.test.mjs b/e2e/fixtures/e2e-config/test/judge.integration.test.mjs
new file mode 100644
index 000000000..7566d90f4
--- /dev/null
+++ b/e2e/fixtures/e2e-config/test/judge.integration.test.mjs
@@ -0,0 +1,65 @@
+import assert from "node:assert/strict";
+import { test } from "node:test";
+
+import { e2eJudgeModel } from "../src/judge.ts";
+
+test("fixture judge adapts Gateway Responses into evaluation answers", async (t) => {
+ const previousKey = process.env.AI_GATEWAY_API_KEY;
+ process.env.AI_GATEWAY_API_KEY = "fixture-gateway-key";
+ t.after(() => {
+ if (previousKey === undefined) delete process.env.AI_GATEWAY_API_KEY;
+ else process.env.AI_GATEWAY_API_KEY = previousKey;
+ });
+ const requests = [];
+ t.mock.method(globalThis, "fetch", async (url, init) => {
+ requests.push({ url, init });
+ return Response.json({
+ id: "fixture-response",
+ model: "openai/gpt-5.6-luna",
+ output: [
+ {
+ type: "message",
+ id: "fixture-message",
+ role: "assistant",
+ content: [
+ {
+ type: "output_text",
+ text: JSON.stringify({ q0: 0.9, q1: 1.5, q2: "c0" }),
+ annotations: [],
+ },
+ ],
+ },
+ ],
+ });
+ });
+
+ const result = await e2eJudgeModel().doEvaluate({
+ state: { response: "Hello Alice" },
+ questions: {
+ accurate: { type: "boolean", instructions: "Greets Alice" },
+ clarity: {
+ type: "score",
+ instructions: "Grade clarity",
+ criteria: ["Unclear", "Mostly clear", "Clear"],
+ },
+ outcome: {
+ type: "choice",
+ instructions: "Classify the response",
+ criteria: { answered: "Answers", declined: "Declines" },
+ },
+ },
+ });
+
+ assert.equal(requests.length, 1);
+ const { url, init } = requests[0];
+ assert.equal(String(url), "https://ai-gateway.vercel.sh/v1/responses");
+ assert.equal(new Headers(init.headers).get("authorization"), "Bearer fixture-gateway-key");
+ const body = JSON.parse(init.body);
+ assert.equal(body.model, "openai/gpt-5.6-luna");
+ assert.equal(body.text.format.type, "json_schema");
+ assert.deepEqual(result.answers, {
+ accurate: { type: "boolean", probability: 0.9 },
+ clarity: { type: "score", score: 1.5 },
+ outcome: { type: "choice", choice: "answered" },
+ });
+});
diff --git a/packages/eve/package.json b/packages/eve/package.json
index c2820f5ec..917dcef1a 100644
--- a/packages/eve/package.json
+++ b/packages/eve/package.json
@@ -525,7 +525,6 @@
"@workflow/world-local": "5.0.0-beta.45",
"@workflow/world-vercel": "5.0.0-beta.48",
"ai": "catalog:",
- "autoevals": "0.0.132",
"chat": "4.34.0",
"chokidar": "5.0.0",
"commander": "14.0.3",
diff --git a/packages/eve/src/evals/abort.test.ts b/packages/eve/src/evals/abort.test.ts
new file mode 100644
index 000000000..f1d24c471
--- /dev/null
+++ b/packages/eve/src/evals/abort.test.ts
@@ -0,0 +1,18 @@
+import { describe, expect, it } from "vitest";
+import { runUntilAborted } from "#evals/abort.js";
+
+describe("eval cancellation", () => {
+ it("rejects an already-aborted signal even when the task is settled", async () => {
+ const reason = new Error("Deadline reached");
+ await expect(runUntilAborted(Promise.resolve(1), AbortSignal.abort(reason))).rejects.toBe(
+ reason,
+ );
+ });
+
+ it("observes task failures when cancellation has already occurred", async () => {
+ const reason = new Error("Deadline reached");
+ await expect(
+ runUntilAborted(Promise.reject(new Error("Task failed")), AbortSignal.abort(reason)),
+ ).rejects.toBe(reason);
+ });
+});
diff --git a/packages/eve/src/evals/abort.ts b/packages/eve/src/evals/abort.ts
new file mode 100644
index 000000000..b97b2d160
--- /dev/null
+++ b/packages/eve/src/evals/abort.ts
@@ -0,0 +1,17 @@
+/** Stop waiting at the eval deadline even if a provider ignores cancellation. */
+export async function runUntilAborted<T>(
+ task: T | PromiseLike<T>,
+ signal: AbortSignal,
+): Promise<T> {
+ let onAbort: (() => void) | undefined;
+ const aborted = new Promise<never>((_resolve, reject) => {
+ onAbort = () => reject(signal.reason);
+ if (signal.aborted) onAbort();
+ else signal.addEventListener("abort", onAbort, { once: true });
+ });
+ try {
+ return await Promise.race([aborted, task]);
+ } finally {
+ if (onAbort !== undefined) signal.removeEventListener("abort", onAbort);
+ }
+}
diff --git a/packages/eve/src/evals/autoevals-client.test.ts b/packages/eve/src/evals/autoevals-client.test.ts
deleted file mode 100644
index 45b6cdac4..000000000
--- a/packages/eve/src/evals/autoevals-client.test.ts
+++ /dev/null
@@ -1,83 +0,0 @@
-import { afterEach, describe, expect, it, vi } from "vitest";
-import { Factuality } from "autoevals";
-
-const mocks = vi.hoisted(() => ({ generateText: vi.fn() }));
-
-vi.mock("ai", () => ({
- generateText: mocks.generateText,
- jsonSchema: vi.fn(),
-}));
-
-import { createAutoevalsClient } from "#evals/autoevals-client.js";
-
-type BraintrustGlobal = typeof globalThis & {
- __inherited_braintrust_wrap_openai?: (client: unknown) => unknown;
-};
-
-const braintrustGlobal = globalThis as BraintrustGlobal;
-const originalBraintrustWrapper = braintrustGlobal.__inherited_braintrust_wrap_openai;
-
-describe("createAutoevalsClient", () => {
- afterEach(() => {
- mocks.generateText.mockReset();
- if (originalBraintrustWrapper === undefined) {
- delete braintrustGlobal.__inherited_braintrust_wrap_openai;
- } else {
- braintrustGlobal.__inherited_braintrust_wrap_openai = originalBraintrustWrapper;
- }
- });
-
- it("identifies eve on Gateway-routed judge calls", async () => {
- mocks.generateText.mockResolvedValueOnce({ text: "ok", toolCalls: [] });
- const client = createAutoevalsClient({ languageModel: "openai/gpt-5.5" });
-
- await client.chat.completions.create({ messages: [], model: "openai/gpt-5.5" });
-
- expect(mocks.generateText).toHaveBeenCalledWith(
- expect.objectContaining({
- headers: { "user-agent": expect.stringMatching(/^eve\/.+/) },
- }),
- );
- });
-
- it("retains a distinct create function when autoevals reconstructs it", () => {
- const client = createAutoevalsClient({ languageModel: "openai/gpt-5.5" });
- const Constructor = Object.getPrototypeOf(client).constructor as new (config: {
- apiKey: string;
- }) => typeof client;
- const reconstructed = new Constructor({ apiKey: "dummy" });
-
- expect(typeof reconstructed.chat.completions.create).toBe("function");
- expect(String(reconstructed.chat.completions.create)).not.toBe(
- String(client.chat.completions.create),
- );
- });
-
- it("runs a bundled autoevals judge while the Braintrust global wrapper is active", async () => {
- const { wrapOpenAI } = await import("braintrust");
- expect(braintrustGlobal.__inherited_braintrust_wrap_openai).toBe(wrapOpenAI);
-
- mocks.generateText.mockResolvedValueOnce({
- text: "",
- toolCalls: [
- {
- input: { choice: "C" },
- toolCallId: "choice-1",
- toolName: "select_choice",
- },
- ],
- });
-
- const result = await Factuality({
- client: createAutoevalsClient({ languageModel: "openai/gpt-5.5" }),
- expected: "Paris",
- input: "What is the capital of France?",
- model: "openai/gpt-5.5",
- output: "Paris",
- useCoT: false,
- });
-
- expect(result.score).toBe(1);
- expect(mocks.generateText).toHaveBeenCalledTimes(1);
- });
-});
diff --git a/packages/eve/src/evals/autoevals-client.ts b/packages/eve/src/evals/autoevals-client.ts
deleted file mode 100644
index 1d8a558e2..000000000
--- a/packages/eve/src/evals/autoevals-client.ts
+++ /dev/null
@@ -1,176 +0,0 @@
-import {
- generateText,
- type LanguageModel,
- type ModelMessage,
- type ToolChoice,
- type ToolSet,
-} from "ai";
-import { Factuality } from "autoevals";
-import { resolveProviderHeaders } from "#internal/gateway.js";
-
-import { toInputSchema } from "#tools/schema.js";
-
-/**
- * The OpenAI-shaped client surface autoevals expects. Extracted from the
- * library so we don't take a direct dependency on `openai` types.
- */
-type AutoevalsClient = NonNullable<Parameters<typeof Factuality>[0]["client"]>;
-
-type ProviderOptions = Parameters<typeof generateText>[0]["providerOptions"];
-
-interface AutoevalsClientConfig {
- readonly languageModel: LanguageModel;
- readonly providerOptions?: ProviderOptions;
-}
-
-interface ChatParams {
- readonly messages?: readonly ChatMessage[];
- readonly tools?: readonly ChatTool[];
- readonly tool_choice?: ChatToolChoice;
-}
-
-interface ChatMessage {
- readonly role?: "assistant" | "developer" | "system" | "user";
- readonly content?: string | readonly ChatMessageContentPart[] | null;
-}
-
-interface ChatMessageContentPart {
- readonly text?: string;
-}
-
-interface ChatTool {
- readonly type?: string;
- readonly function?: {
- readonly name?: string;
- readonly description?: string;
- readonly parameters?: Record<string, unknown>;
- };
-}
-
-type ChatToolChoice =
- | "auto"
- | "none"
- | "required"
- | { readonly type: "function"; readonly function: { readonly name: string } };
-
-/**
- * Constructible OpenAI-shaped adapter for bundled autoevals graders.
- *
- * Braintrust installs a global wrapper that makes autoevals reconstruct the
- * supplied client through its prototype constructor. The reconstructed client
- * must retain `chat.completions.create`, and its probe function must differ
- * from the live function so autoevals does not wrap the AI SDK bridge.
- */
-class AutoevalsOpenAIClientAdapter {
- readonly chat: {
- readonly completions: {
- readonly create: (params: ChatParams) => Promise<{ readonly choices: readonly unknown[] }>;
- };
- };
-
- constructor(config: AutoevalsClientConfig | { readonly apiKey?: string }) {
- this.chat = {
- completions: {
- create: isAutoevalsClientConfig(config)
- ? (params) => createChatCompletion(params, config)
- : createProbeChatCompletion,
- },
- };
- }
-}
-
-export function createAutoevalsClient(config: AutoevalsClientConfig): AutoevalsClient {
- const adapter: unknown = new AutoevalsOpenAIClientAdapter(config);
- return adapter as AutoevalsClient;
-}
-
-function isAutoevalsClientConfig(
- config: AutoevalsClientConfig | { readonly apiKey?: string },
-): config is AutoevalsClientConfig {
- return "languageModel" in config;
-}
-
-async function createProbeChatCompletion(): Promise<{ readonly choices: readonly unknown[] }> {
- return { choices: [] };
-}
-
-async function createChatCompletion(
- params: ChatParams,
- config: AutoevalsClientConfig,
-): Promise<{ readonly choices: readonly unknown[] }> {
- const tools = convertTools(params.tools);
- const result = await generateText({
- headers: resolveProviderHeaders(config.languageModel),
- model: config.languageModel,
- messages: convertMessages(params.messages ?? []),
- tools: Object.keys(tools).length > 0 ? tools : undefined,
- toolChoice: convertToolChoice(params.tool_choice),
- providerOptions: config.providerOptions,
- });
-
- const toolCalls = result.toolCalls.map((call) => ({
- id: call.toolCallId,
- type: "function",
- function: {
- name: call.toolName,
- arguments: JSON.stringify(call.input ?? {}),
- },
- }));
-
- return {
- choices: [
- {
- index: 0,
- finish_reason: toolCalls.length > 0 ? "tool_calls" : "stop",
- message: {
- role: "assistant",
- content: result.text || null,
- tool_calls: toolCalls.length > 0 ? toolCalls : undefined,
- },
- },
- ],
- };
-}
-
-function convertMessages(messages: readonly ChatMessage[]): ModelMessage[] {
- return messages.map((message) => {
- const content = contentToText(message.content);
- switch (message.role) {
- case "assistant":
- return { role: "assistant", content };
- case "developer":
- case "system":
- return { role: "system", content };
- case "user":
- default:
- return { role: "user", content };
- }
- });
-}
-
-function contentToText(content: ChatMessage["content"]): string {
- if (content == null) return "";
- if (typeof content === "string") return content;
- return content
- .map((part) => part.text ?? "")
- .filter(Boolean)
- .join("\n");
-}
-
-function convertTools(tools: readonly ChatTool[] | undefined): ToolSet {
- const result: ToolSet = {};
- for (const item of tools ?? []) {
- if (item.type !== "function" || item.function?.name === undefined) continue;
- result[item.function.name] = {
- description: item.function.description,
- inputSchema: toInputSchema(item.function.parameters ?? {}),
- };
- }
- return result;
-}
-
-function convertToolChoice(choice: ChatToolChoice | undefined): ToolChoice<ToolSet> | undefined {
- if (choice === undefined) return undefined;
- if (typeof choice === "string") return choice;
- return { type: "tool", toolName: choice.function.name };
-}
diff --git a/packages/eve/src/evals/context.ts b/packages/eve/src/evals/context.ts
index f8dff7115..c2e829860 100644
--- a/packages/eve/src/evals/context.ts
+++ b/packages/eve/src/evals/context.ts
@@ -32,6 +32,7 @@ export function createEvalContext(deps: {
getReply: () => deps.manager.lastTurnSession()?.lastTurn?.message ?? null,
getInput: () => deps.manager.lastTurnSession()?.lastInput ?? "",
judge: deps.judge,
+ signal: deps.signal,
});
const context: EveEvalContext = {
diff --git a/packages/eve/src/evals/define-eval-config.test.ts b/packages/eve/src/evals/define-eval-config.test.ts
index 42c356443..021bb39d3 100644
--- a/packages/eve/src/evals/define-eval-config.test.ts
+++ b/packages/eve/src/evals/define-eval-config.test.ts
@@ -29,8 +29,8 @@ describe("defineEvalConfig", () => {
expect(config.judge).toBeUndefined();
});
- it("requires a model when a judge is provided", () => {
- expect(() => defineInvalidConfig({ judge: {} as never })).toThrow("`judge` requires a `model`");
+ it("allows judge settings to use the default evaluation model", () => {
+ expect(defineEvalConfig({ judge: {} }).judge).toEqual({});
});
it("rejects a non-positive maxConcurrency", () => {
diff --git a/packages/eve/src/evals/define-eval-config.ts b/packages/eve/src/evals/define-eval-config.ts
index 2652f6842..0c32a8931 100644
--- a/packages/eve/src/evals/define-eval-config.ts
+++ b/packages/eve/src/evals/define-eval-config.ts
@@ -5,12 +5,12 @@ import type { EveEvalConfig, EveEvalConfigInput } from "#evals/types.js";
* default export of `evals.config.ts` at the root of the `evals/` directory.
*
* Exactly one `evals.config.ts` is required. It supplies the optional default
- * `judge` model for `t.judge.*` assertions (so individual evals need not
+ * `judge` model for `t.judge(...)` assertions (so individual evals need not
* repeat it), optional run-level `reporters`, a default `maxConcurrency`, and a
* default `timeoutMs`. CLI flags (`--max-concurrency`, `--timeout`) and
* per-eval values take precedence over the config defaults.
*
- * Throws on invalid input: a `judge` without a `model`, a non-positive or
+ * Throws on invalid input: a non-positive or
* non-integer `maxConcurrency`, a negative or non-finite `timeoutMs`, or a
* non-array `reporters`.
*/
@@ -24,16 +24,6 @@ export function defineEvalConfig(input: EveEvalConfigInput): EveEvalConfig {
}
function validateEvalConfigInput(input: EveEvalConfigInput): void {
- if (
- input.judge !== undefined &&
- (input.judge.model === undefined || input.judge.model === null)
- ) {
- throw new Error(
- "Eval config `judge` requires a `model`. It is the default judge model for `t.judge.*` " +
- "assertions across every eval.",
- );
- }
-
if (
input.maxConcurrency !== undefined &&
(!Number.isInteger(input.maxConcurrency) || input.maxConcurrency < 1)
diff --git a/packages/eve/src/evals/define-eval.ts b/packages/eve/src/evals/define-eval.ts
index 15d323cbd..1cf8b45b6 100644
--- a/packages/eve/src/evals/define-eval.ts
+++ b/packages/eve/src/evals/define-eval.ts
@@ -4,12 +4,12 @@ import type { EveEvalDefinition, EveEvalInput } from "#evals/types.js";
* Defines one eve eval. Each eval file is exactly one case: an imperative
* `test(t)` function that drives the agent (`t.session`, `t.send`, …) and
* asserts on what it produced (`t.succeeded()`, `t.check(...)`,
- * `t.judge.autoevals.*`). Organize related evals with directory nesting under
+ * `t.judge(...)`). Organize related evals with directory nesting under
* `evals/`, or default-export an array of evals to fan one file out over a
* dataset.
*
- * A `judge` is optional: `t.judge.*` assertions fall back to the `judge`
- * declared in `evals.config.ts` unless this eval overrides it. The judge model
+ * A `judge` is optional: `t.judge(...)` assertions fall back to the `judge`
+ * declared in `evals.config.ts`, then the shared evaluation default. The judge model
* is used solely for scoring, never for the agent under test. Eval identity is
* derived from the `evals/<path>.eval.ts` file path by the discovery layer, so
* authoring `id` or `name` throws.
@@ -54,7 +54,7 @@ function validateEvalInput(input: EveEvalInput): void {
rejectLegacyKey(
input,
"scores",
- "Use soft assertions inside `test`: `t.check(...).atLeast(n)` or `t.judge.autoevals.*`.",
+ "Use soft assertions inside `test`: `t.check(...).atLeast(n)` or `t.judge(...)`.",
);
rejectLegacyKey(
input,
diff --git a/packages/eve/src/evals/expect/index.test.ts b/packages/eve/src/evals/expect/index.test.ts
index 633eca56c..3e39e52fc 100644
--- a/packages/eve/src/evals/expect/index.test.ts
+++ b/packages/eve/src/evals/expect/index.test.ts
@@ -87,6 +87,20 @@ describe("expect builders", () => {
expect(await assertion.score("hello")).toBe(1);
});
+ it.each([
+ ["", "", 1],
+ ["", "abc", 0],
+ ["abc", "", 0],
+ ["kitten", "sitting", 1 - 3 / 7],
+ ["abc", "xyz", 0],
+ ["😀", "😃", 0.5],
+ ["é", "e", 0],
+ [null, "", 1],
+ [42, "42", 1],
+ ])("preserves Levenshtein similarity for %j and %j", async (actual, expected, score) => {
+ expect(await similarity(String(expected)).score(actual)).toBe(score);
+ });
+
it("chaining overrides severity and threshold without mutating the original", () => {
const base = includes("x");
diff --git a/packages/eve/src/evals/expect/index.ts b/packages/eve/src/evals/expect/index.ts
index e47385a1e..713ee3614 100644
--- a/packages/eve/src/evals/expect/index.ts
+++ b/packages/eve/src/evals/expect/index.ts
@@ -1,4 +1,4 @@
-import { Levenshtein as AutoevalsLevenshtein } from "autoevals";
+import { levenshteinSimilarity } from "#evals/expect/levenshtein.js";
import type { StandardSchemaV1 } from "#compiled/@standard-schema/spec/index.js";
import { formatDiagnosticValue, toDiagnosticMetadataValue } from "#evals/diagnostics.js";
@@ -148,9 +148,8 @@ export function similarity(expected: string): Assertion {
severity: "soft",
evaluate: async (value) => {
const actual = String(value ?? "");
- const result = await AutoevalsLevenshtein({ output: actual, expected });
return {
- score: result.score ?? 0,
+ score: levenshteinSimilarity(actual, expected),
message: `expected similarity to ${formatDiagnosticValue(expected)}; received ${formatDiagnosticValue(actual)}`,
metadata: { actual, expected },
};
diff --git a/packages/eve/src/evals/expect/levenshtein.ts b/packages/eve/src/evals/expect/levenshtein.ts
new file mode 100644
index 000000000..e17373bd6
--- /dev/null
+++ b/packages/eve/src/evals/expect/levenshtein.ts
@@ -0,0 +1,20 @@
+/** Normalized edit similarity using UTF-16 code units, matching string.length. */
+export function levenshteinSimilarity(left: string, right: string): number {
+ if (left === right) return 1;
+ if (left.length < right.length) [left, right] = [right, left];
+ const row = Array.from({ length: right.length + 1 }, (_, index) => index);
+ for (let i = 1; i <= left.length; i++) {
+ let diagonal = row[0]!;
+ row[0] = i;
+ for (let j = 1; j <= right.length; j++) {
+ const above = row[j]!;
+ row[j] = Math.min(
+ above + 1,
+ row[j - 1]! + 1,
+ diagonal + (left[i - 1] === right[j - 1] ? 0 : 1),
+ );
+ diagonal = above;
+ }
+ }
+ return 1 - row[right.length]! / left.length;
+}
diff --git a/packages/eve/src/evals/index.ts b/packages/eve/src/evals/index.ts
index 9fd291549..a97416699 100644
--- a/packages/eve/src/evals/index.ts
+++ b/packages/eve/src/evals/index.ts
@@ -31,7 +31,6 @@ export type {
AssertionHandle,
AssertionResult,
AssertionSeverity,
- AutoevalsJudges,
EveEvalActionStatus,
EveEvalAssertions,
EveEvalContext,
@@ -59,6 +58,9 @@ export type {
EveEvalStreamEvent,
EveEvalWaitForEventOptions,
EveEvalVerdict,
+ JudgeBatch,
+ JudgeInput,
+ JudgeQuestion,
JudgeContext,
JudgeOpts,
} from "#evals/types.js";
diff --git a/packages/eve/src/evals/judge.integration.test.ts b/packages/eve/src/evals/judge.integration.test.ts
new file mode 100644
index 000000000..ec922788b
--- /dev/null
+++ b/packages/eve/src/evals/judge.integration.test.ts
@@ -0,0 +1,183 @@
+import { Experimental_EvaluationMockModelV4 } from "ai/test";
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { Client } from "#client/client.js";
+import { executeTask } from "#evals/runner/execute-task.js";
+import { computeEvalVerdict } from "#evals/runner/verdict.js";
+import { createEvalTargetHandle } from "#evals/target.js";
+import type { EveEvalContext, EveEvalJudgeConfig } from "#evals/types.js";
+
+const client = new Client({ host: "https://eve.test" });
+const target = createEvalTargetHandle({
+ capabilities: { devRoutes: true },
+ client,
+ kind: "local",
+ url: "https://eve.test",
+});
+function run(test: (t: EveEvalContext) => void, judge?: EveEvalJudgeConfig, timeoutMs?: number) {
+ return executeTask({
+ client,
+ target,
+ timeoutMs,
+ evaluation: { _tag: "EveEval", id: "judge", test, judge },
+ });
+}
+function model(probability = 0.8) {
+ const evaluator = new Experimental_EvaluationMockModelV4({
+ doEvaluate: async ({ questions }) => ({
+ answers: Object.fromEntries(
+ Object.keys(questions).map((key) => [key, { type: "boolean" as const, probability }]),
+ ),
+ warnings: [],
+ usage: { inputTokens: 10, outputTokens: 2 },
+ }),
+ });
+ vi.spyOn(evaluator, "doEvaluate");
+ return evaluator;
+}
+afterEach(() => {
+ vi.unstubAllGlobals();
+ vi.restoreAllMocks();
+});
+
+describe("evaluation-backed judge runner", () => {
+ it("uses evaluate's default model and leaves deterministic evals model-free", async () => {
+ const evaluator = model();
+ const factory = vi.fn(() => evaluator);
+ vi.stubGlobal("AI_SDK_DEFAULT_PROVIDER", { evaluationModel: factory });
+ await run(() => {});
+ expect(factory).not.toHaveBeenCalled();
+ const outcome = await run((t) => {
+ t.judge("Answers the request").atLeast(0.9);
+ });
+ expect(factory).toHaveBeenCalledWith("typesafe-ai/jev");
+ expect(outcome.assertions[0]).toMatchObject({ score: 0.8, severity: "soft", passed: false });
+ expect(computeEvalVerdict(outcome)).toBe("scored");
+ });
+
+ it("supports model instances, overrides, and one SDK call for a batch", async () => {
+ const inherited = model();
+ const overridden = model(1);
+ const outcome = await run(
+ (t) => {
+ t.judge("One");
+ const handles = t.judge(
+ {
+ state: { response: "Hello Alice" },
+ questions: {
+ first: { type: "boolean", instructions: "Greets Alice" },
+ second: { type: "boolean", instructions: "Is polite" },
+ },
+ },
+ {
+ model: overridden,
+ modelOptions: { providerOptions: { fixture: { mode: "override" } } },
+ },
+ );
+ handles.first.gate();
+ handles.second.gate();
+ },
+ { model: inherited, modelOptions: { providerOptions: { fixture: { mode: "inherited" } } } },
+ );
+ expect(vi.mocked(inherited.doEvaluate).mock.calls).toHaveLength(1);
+ expect(vi.mocked(overridden.doEvaluate).mock.calls).toHaveLength(1);
+ expect(vi.mocked(inherited.doEvaluate).mock.calls[0]?.[0].providerOptions).toEqual({
+ fixture: { mode: "inherited" },
+ });
+ expect(vi.mocked(overridden.doEvaluate).mock.calls[0]?.[0]).toMatchObject({
+ state: { response: "Hello Alice" },
+ providerOptions: { fixture: { mode: "override" } },
+ });
+ expect(outcome.assertions.map((item) => item.score)).toEqual([0.8, 1, 1]);
+ });
+
+ it.each([
+ { criteria: [], batch: false },
+ { criteria: ["Only level"], batch: false },
+ { criteria: [], batch: true },
+ { criteria: ["Only level"], batch: true },
+ ])("rejects an undersized score rubric before provider I/O: %j", async ({ criteria, batch }) => {
+ const evaluator = model();
+ const outcome = await run(
+ (t) => {
+ const question = { type: "score" as const, instructions: "Grade clarity.", criteria };
+ if (batch) {
+ const handles = t.judge({
+ questions: {
+ clarity: question,
+ accurate: { type: "boolean", instructions: "Is the response accurate?" },
+ },
+ });
+ handles.clarity.atLeast(0);
+ } else {
+ t.judge(question).atLeast(0);
+ }
+ },
+ { model: evaluator },
+ );
+ expect(evaluator.doEvaluate).not.toHaveBeenCalled();
+ expect(outcome.assertions).toHaveLength(batch ? 2 : 1);
+ for (const assertion of outcome.assertions) {
+ expect(assertion).toMatchObject({ score: 0, severity: "gate", passed: false });
+ expect(assertion.message).toContain(
+ "score criteria must contain at least two ordered levels",
+ );
+ }
+ expect(computeEvalVerdict(outcome)).toBe("failed");
+ });
+
+ it.each(["missing", "invalid", "provider", "unsupported"])(
+ "fails the whole batch for %s answers or capability",
+ async (failure) => {
+ const evaluator = new Experimental_EvaluationMockModelV4({
+ supportedQuestionTypes: failure === "unsupported" ? ["choice"] : ["boolean"],
+ doEvaluate: async (): Promise<
+ Awaited<ReturnType<Experimental_EvaluationMockModelV4["doEvaluate"]>>
+ > => {
+ if (failure === "provider") throw new Error("Evaluation credentials unavailable");
+ return {
+ answers:
+ failure === "missing"
+ ? {}
+ : {
+ a: { type: "boolean", probability: 2 },
+ b: { type: "boolean", probability: 1 },
+ },
+ warnings: [],
+ };
+ },
+ });
+ const outcome = await run(
+ (t) => {
+ t.judge({
+ questions: {
+ a: { type: "boolean", instructions: "A" },
+ b: { type: "boolean", instructions: "B" },
+ },
+ });
+ },
+ { model: evaluator },
+ );
+ expect(outcome.skipReason).toBeUndefined();
+ expect(outcome.assertions).toHaveLength(2);
+ expect(outcome.assertions.every((a) => !a.passed && a.severity === "gate")).toBe(true);
+ expect(computeEvalVerdict(outcome)).toBe("failed");
+ },
+ );
+
+ it("bounds finalization when a provider ignores the eval deadline", async () => {
+ const evaluator = new Experimental_EvaluationMockModelV4({
+ doEvaluate: () => new Promise(() => {}),
+ });
+ vi.spyOn(evaluator, "doEvaluate");
+ const outcome = await run(
+ (t) => {
+ t.judge("A");
+ },
+ { model: evaluator },
+ 20,
+ );
+ expect(vi.mocked(evaluator.doEvaluate).mock.calls[0]?.[0].abortSignal?.aborted).toBe(true);
+ expect(outcome.assertions[0]).toMatchObject({ passed: false, severity: "gate" });
+ expect(computeEvalVerdict(outcome)).toBe("failed");
+ });
+});
diff --git a/packages/eve/src/evals/judge.test.ts b/packages/eve/src/evals/judge.test.ts
index 2bacd6bb7..920c3674c 100644
--- a/packages/eve/src/evals/judge.test.ts
+++ b/packages/eve/src/evals/judge.test.ts
@@ -1,129 +1,227 @@
-import { afterEach, describe, expect, it, vi } from "vitest";
-
+import { afterEach, describe, expect, expectTypeOf, it, vi } from "vitest";
import { AssertionCollector } from "#evals/assertions/collector.js";
import { buildJudgeContext } from "#evals/judge.js";
import { createEmptyDerivedFacts } from "#evals/runner/derive-run-facts.js";
-import type { EveEvalTaskResult } from "#evals/types.js";
-
-const mocks = vi.hoisted(() => ({
- closedQA: vi.fn(),
- createAutoevalsClient: vi.fn(() => ({ kind: "autoevals-client" })),
- factuality: vi.fn(),
- sql: vi.fn(),
- summary: vi.fn(),
-}));
-
-vi.mock("autoevals", () => ({
- ClosedQA: mocks.closedQA,
- Factuality: mocks.factuality,
- Sql: mocks.sql,
- Summary: mocks.summary,
-}));
+import type { AssertionHandle, EveEvalTaskResult, JudgeContext } from "#evals/types.js";
-vi.mock("#evals/autoevals-client.js", () => ({
- createAutoevalsClient: mocks.createAutoevalsClient,
-}));
-
-afterEach(() => {
- vi.clearAllMocks();
-});
+const evaluate = vi.hoisted(() => vi.fn());
+vi.mock("#ai/evaluate.js", () => ({ evaluate }));
+afterEach(() => vi.resetAllMocks());
-describe("buildJudgeContext", () => {
- it("reports the prompt, criteria, response, and autoevals explanation", async () => {
- mocks.closedQA.mockResolvedValueOnce({
- name: "ClosedQA",
- score: 0,
- metadata: {
- choice: "N",
- rationale: "The response makes no mention of a source.",
- tokens: 42,
- },
- });
- const collector = new AssertionCollector();
- const judge = buildJudgeContext({
- collector,
- getInput: () => "Name the source for this claim.",
- getReply: () => "It is widely believed.",
- judge: { model: "openai/gpt-5.4-mini" },
- });
-
- judge.autoevals.closedQA("cites a source").label("source citation").atLeast(0.8);
- const [assertion] = await collector.finalize(emptyTaskResult());
+function setup() {
+ const collector = new AssertionCollector();
+ const controller = new AbortController();
+ const judge = buildJudgeContext({
+ collector,
+ signal: controller.signal,
+ getInput: () => "Name the source.",
+ getReply: () => "A sourced answer.",
+ judge: undefined,
+ });
+ return { judge, collector, controller };
+}
+function result(answers: Record<string, unknown>) {
+ return {
+ answers,
+ response: { modelId: "test-evaluator", timestamp: new Date(0) },
+ usage: { inputTokens: 10, outputTokens: 2, totalTokens: 12 },
+ warnings: [],
+ };
+}
+function emptyTaskResult(): EveEvalTaskResult {
+ return {
+ derived: createEmptyDerivedFacts(),
+ events: [],
+ finalMessage: null,
+ output: null,
+ status: "completed",
+ traceContexts: [],
+ };
+}
- expect(mocks.closedQA).toHaveBeenCalledWith(
+describe("judge", () => {
+ it("expands criteria and retains assertion handles and diagnostics", async () => {
+ evaluate.mockResolvedValue(result({ judgment: { type: "boolean", probability: 0.7 } }));
+ const { judge, collector, controller } = setup();
+ const handle = judge("cites a source").label("citation").atLeast(0.8);
+ expect(handle).not.toHaveProperty("then");
+ expect(evaluate).toHaveBeenCalledWith(
expect.objectContaining({
- criteria: "cites a source",
- input: "Name the source for this claim.",
- model: "openai/gpt-5.4-mini",
- output: "It is widely believed.",
+ model: undefined,
+ abortSignal: controller.signal,
+ state: { input: "Name the source.", output: "A sourced answer." },
+ questions: { judgment: { type: "boolean", instructions: "cites a source" } },
}),
);
+ const [assertion] = await collector.finalize(emptyTaskResult());
expect(assertion).toMatchObject({
- name: "judge.autoevals.closedQA [source citation]",
- score: 0,
+ name: "judge.boolean [citation]",
+ score: 0.7,
severity: "soft",
threshold: 0.8,
passed: false,
- message: [
- 'prompt: "Name the source for this claim."',
- 'criteria: "cites a source"',
- 'response: "It is widely believed."',
- 'rationale: "The response makes no mention of a source."',
- 'choice: "N"',
- ].join("\n"),
- metadata: {
- autoevalsName: "ClosedQA",
- choice: "N",
- criteria: "cites a source",
- input: "Name the source for this claim.",
- judge: "openai/gpt-5.4-mini",
- output: "It is widely believed.",
- rationale: "The response makes no mention of a source.",
- tokens: 42,
+ metadata: { judge: "test-evaluator", answer: { probability: 0.7 }, usageScope: "assertion" },
+ });
+ expect(assertion?.message).toContain('"cites a source"');
+ });
+
+ it("shares a batch request, normalizes rubrics, and keeps expectations local", async () => {
+ evaluate.mockResolvedValue(
+ result({
+ accurate: { type: "boolean", probability: 0.2 },
+ clarity: { type: "score", score: 1.5 },
+ outcome: { type: "choice", choice: "answered" },
+ }),
+ );
+ const { judge, collector } = setup();
+ const handles = judge({
+ state: { response: "hello" },
+ questions: {
+ accurate: { type: "boolean", instructions: "Is it accurate?" },
+ clarity: {
+ type: "score",
+ instructions: "Grade clarity",
+ criteria: ["Poor", "Fair", "Clear"],
+ },
+ outcome: {
+ type: "choice",
+ instructions: "Classify",
+ criteria: { answered: "Answer", declined: "Decline" },
+ expected: "answered",
+ },
},
});
+ handles.clarity.atLeast(0.8);
+ handles.outcome.gate();
+ expect(evaluate).toHaveBeenCalledOnce();
+ expect(evaluate.mock.calls[0]?.[0].questions.outcome).not.toHaveProperty("expected");
+ const assertions = await collector.finalize(emptyTaskResult());
+ expect(assertions.map(({ name, score, passed }) => ({ name, score, passed }))).toEqual([
+ { name: "judge.boolean.accurate", score: 0.2, passed: true },
+ { name: "judge.score.clarity", score: 0.75, passed: false },
+ { name: "judge.choice.outcome", score: 1, passed: true },
+ ]);
+ expect(assertions.every((item) => item.metadata?.usageScope === "batch")).toBe(true);
});
- it("reports the custom value and expected answer for comparison graders", async () => {
- mocks.factuality.mockResolvedValueOnce({
- name: "Factuality",
- score: 1,
- metadata: { choice: "A" },
+ it("scores a different selected choice as zero", async () => {
+ evaluate.mockResolvedValue(result({ judgment: { type: "choice", choice: "declined" } }));
+ const { judge, collector } = setup();
+ judge({
+ type: "choice",
+ instructions: "Classify",
+ criteria: { answered: null, declined: null },
+ expected: "answered",
+ }).gate();
+ expect((await collector.finalize(emptyTaskResult()))[0]).toMatchObject({
+ score: 0,
+ passed: false,
});
- const collector = new AssertionCollector();
- const judge = buildJudgeContext({
- collector,
- getInput: () => "What is the capital of France?",
- getReply: () => "This value is not graded.",
- judge: { model: "openai/gpt-5.4-mini" },
+ });
+
+ it("captures JSON state and questions before later mutations", async () => {
+ evaluate.mockResolvedValue(result({ judgment: { type: "boolean", probability: 1 } }));
+ const { judge, collector } = setup();
+ const on = { values: [1] };
+ const question = { type: "boolean" as const, instructions: "Original" };
+ judge(question, { on });
+ on.values.push(2);
+ question.instructions = "Changed";
+ await collector.finalize(emptyTaskResult());
+ expect(evaluate.mock.calls[0]?.[0]).toMatchObject({
+ state: { output: { values: [1] } },
+ questions: { judgment: { instructions: "Original" } },
});
+ });
- judge.autoevals.factuality("Paris", { on: "The capital is Paris." });
- const [assertion] = await collector.finalize(emptyTaskResult());
+ it("preserves explicit null and transcript values", async () => {
+ evaluate.mockResolvedValue(result({ judgment: { type: "boolean", probability: 1 } }));
+ const { judge, collector } = setup();
+ judge("check", { on: null });
+ judge("check", { on: "user: Hello\nassistant: Hi" });
+ await collector.finalize(emptyTaskResult());
+ expect(evaluate.mock.calls.map(([request]) => request.state.output)).toEqual([
+ null,
+ "user: Hello\nassistant: Hi",
+ ]);
+ });
- expect(assertion).toMatchObject({
- passed: true,
- message: [
- 'prompt: "What is the capital of France?"',
- 'expected: "Paris"',
- 'response: "The capital is Paris."',
- 'choice: "A"',
- ].join("\n"),
- metadata: {
- expected: "Paris",
- output: "The capital is Paris.",
+ it("validates choice expectations before provider I/O", async () => {
+ const { judge, collector } = setup();
+ // @ts-expect-error Expected must be an authored choice key.
+ judge({ type: "choice", instructions: "Classify", criteria: { yes: null }, expected: "no" });
+ expect((await collector.finalize(emptyTaskResult()))[0]).toMatchObject({
+ severity: "gate",
+ passed: false,
+ });
+ expect(evaluate).not.toHaveBeenCalled();
+ });
+
+ it("turns shared request errors into failed gates for every question", async () => {
+ evaluate.mockRejectedValue(new Error("Evaluation unavailable"));
+ const { judge, collector } = setup();
+ const handles = judge({
+ questions: {
+ a: { type: "boolean", instructions: "A" },
+ b: { type: "boolean", instructions: "B" },
},
});
+ handles.a.soft();
+ handles.b.atLeast(0);
+ expect(await collector.finalize(emptyTaskResult())).toEqual([
+ expect.objectContaining({
+ severity: "gate",
+ passed: false,
+ message: "Evaluation unavailable",
+ }),
+ expect.objectContaining({
+ severity: "gate",
+ passed: false,
+ message: "Evaluation unavailable",
+ }),
+ ]);
+ });
+
+ it("does not hang when a provider ignores cancellation", async () => {
+ evaluate.mockReturnValue(new Promise(() => {}));
+ const { judge, collector, controller } = setup();
+ judge("check");
+ controller.abort(new Error("Deadline reached"));
+ expect((await collector.finalize(emptyTaskResult()))[0]).toMatchObject({
+ severity: "gate",
+ passed: false,
+ message: "Deadline reached",
+ });
+ });
+
+ it("rejects empty batches", () => {
+ expect(() => setup().judge({ questions: {} })).toThrow("non-empty");
+ expect(evaluate).not.toHaveBeenCalled();
});
});
-function emptyTaskResult(): EveEvalTaskResult {
- return {
- derived: createEmptyDerivedFacts(),
- events: [],
- finalMessage: null,
- output: null,
- status: "completed",
- traceContexts: [],
- };
+function checkTypes(judge: JudgeContext) {
+ expectTypeOf(judge("criteria")).toEqualTypeOf<AssertionHandle>();
+ const batch = judge({
+ questions: {
+ choice: {
+ type: "choice",
+ instructions: "Classify",
+ criteria: { yes: null, no: null },
+ expected: "yes",
+ },
+ },
+ });
+ expectTypeOf(batch.choice).toEqualTypeOf<AssertionHandle>();
+ // @ts-expect-error Batch handle keys are inferred.
+ void batch.missing;
+ judge({
+ questions: {
+ // @ts-expect-error Expected must be an authored option key, including in batches.
+ choice: { type: "choice", instructions: "Classify", criteria: { yes: null }, expected: "no" },
+ },
+ });
+ // @ts-expect-error Batch state replaces the per-call on option.
+ judge({ questions: { check: { type: "boolean", instructions: "Check" } } }, { on: "other" });
}
+void checkTypes;
diff --git a/packages/eve/src/evals/judge.ts b/packages/eve/src/evals/judge.ts
index 7fd848e5f..bbfda02ca 100644
--- a/packages/eve/src/evals/judge.ts
+++ b/packages/eve/src/evals/judge.ts
@@ -1,165 +1,146 @@
-import { ClosedQA, Factuality, Sql, Summary, type Score } from "autoevals";
-import type { LanguageModel } from "ai";
-
-import { formatLanguageModelGatewayId } from "#internal/runtime-model.js";
-import type { AgentModelOptionsDefinition } from "#shared/agent-definition.js";
-import { createAutoevalsClient } from "#evals/autoevals-client.js";
+import { evaluate } from "#ai/evaluate.js";
+import { runUntilAborted } from "#evals/abort.js";
import type { AssertionCollector } from "#evals/assertions/collector.js";
-import { formatDiagnosticValue } from "#evals/diagnostics.js";
-import type { AssertionHandle, EveEvalJudgeConfig, JudgeContext, JudgeOpts } from "#evals/types.js";
+import { formatDiagnosticValue, toDiagnosticMetadataValue } from "#evals/diagnostics.js";
+import type {
+ AssertionHandle,
+ EveEvalJudgeConfig,
+ JudgeBatch,
+ JudgeContext,
+ JudgeInput,
+ JudgeOpts,
+ JudgeQuestion,
+ JudgeQuestionConstraint,
+} from "#evals/types.js";
-/**
- * Dependencies the judge namespace closes over: where to record assertions,
- * how to resolve the default graded value (the most recently settled turn) and the original prompt
- * (the autoevals `input`), and the eval/config judge model.
- */
export interface JudgeDeps {
readonly collector: AssertionCollector;
- readonly getReply: () => unknown;
+ readonly getReply: () => string | null;
readonly getInput: () => string;
readonly judge: EveEvalJudgeConfig | undefined;
+ readonly signal: AbortSignal;
}
-/** Common model/client fields every autoevals grader call needs. */
-type GraderCall = {
- readonly model: LanguageModel;
- readonly modelOptions: AgentModelOptionsDefinition | undefined;
-};
-
-type GraderEvidence = {
- readonly label: "criteria" | "expected";
- readonly value: string;
-};
-
-/**
- * Builds the `t.judge` namespace bound to the resolved judge model. Each
- * grader records a soft assertion (override with `.atLeast`/`.gate`) that
- * fires the model call immediately; the collector awaits it before the
- * verdict.
- */
+/** Bind evaluation-backed judgments to the existing assertion lifecycle. */
export function buildJudgeContext(deps: JudgeDeps): JudgeContext {
- function grade(
- name: string,
- opts: JudgeOpts | undefined,
- evidence: GraderEvidence,
- invoke: (
- params: { readonly input: string; readonly output: string } & GraderCall,
- ) => Score | Promise<Score>,
- ): AssertionHandle {
- const value = opts?.on ?? deps.getReply();
- const output = String(value ?? "");
- const input = deps.getInput();
+ function judge(criteria: string, opts?: JudgeOpts): AssertionHandle;
+ function judge<const Q extends JudgeQuestion>(
+ question: Q & JudgeQuestionConstraint<Q>,
+ opts?: JudgeOpts,
+ ): AssertionHandle;
+ function judge<const Questions extends Record<string, JudgeQuestion>>(
+ batch: JudgeBatch<Questions>,
+ opts?: EveEvalJudgeConfig,
+ ): { readonly [Key in keyof Questions]: AssertionHandle };
+ function judge(
+ input: string | JudgeQuestion | JudgeBatch<Record<string, JudgeQuestion>>,
+ opts?: JudgeOpts,
+ ): AssertionHandle | Record<string, AssertionHandle> {
+ const batch = typeof input === "object" && "questions" in input;
+ const questions = batch
+ ? input.questions
+ : {
+ judgment:
+ typeof input === "string" ? { type: "boolean" as const, instructions: input } : input,
+ };
+ const entries = Object.entries(questions);
+ if (entries.length === 0) throw new Error("Judge questions must be a non-empty map.");
+ const state =
+ batch && input.state !== undefined
+ ? input.state
+ : {
+ input: deps.getInput(),
+ output: !batch && opts?.on !== undefined ? opts.on : (deps.getReply() ?? ""),
+ };
const model = opts?.model ?? deps.judge?.model;
const modelOptions = opts?.modelOptions ?? deps.judge?.modelOptions;
+ const request = runUntilAborted(
+ (async () => {
+ const capturedState = structuredClone(state);
+ const capturedQuestions = structuredClone(questions);
+ const evaluationQuestions = Object.fromEntries(
+ Object.entries(capturedQuestions).map(([id, question]) => {
+ if (question.type !== "choice") return [id, question];
+ const { expected, ...evaluationQuestion } = question;
+ if (typeof expected !== "string" || !Object.hasOwn(question.criteria, expected)) {
+ throw new Error(`Judge question "${id}" expected must name an option in criteria.`);
+ }
+ return [id, evaluationQuestion];
+ }),
+ );
+ const result = await evaluate({
+ model,
+ state: capturedState,
+ questions: evaluationQuestions,
+ providerOptions: modelOptions?.providerOptions,
+ abortSignal: deps.signal,
+ });
+ return { result, state: capturedState, questions: capturedQuestions };
+ })(),
+ deps.signal,
+ );
- return deps.collector.recordValue({
- name,
- severity: "soft",
- score: async () => {
- if (model === undefined) {
- throw new Error(
- `${name} needs a judge model. Set \`judge\` on the eval or in evals.config.ts, ` +
- "or pass { model } to the call.",
- );
- }
- const result = await invoke({ input, output, model, modelOptions });
- const autoevalsError = result.error === undefined ? {} : { autoevalsError: result.error };
- const metadata = {
- ...result.metadata,
- ...autoevalsError,
- judge: formatLanguageModelGatewayId(model),
- input,
- output,
- [evidence.label]: evidence.value,
- autoevalsName: result.name,
- };
- return {
- score: result.score ?? 0,
- message: formatJudgeDetail(input, output, evidence, result),
- metadata,
- };
- },
- });
+ const handles = Object.fromEntries(
+ entries.map(([id, question]) => [
+ id,
+ deps.collector.recordValue({
+ name: `judge.${question.type}${batch ? `.${id}` : ""}`,
+ severity: "soft",
+ score: async () => {
+ const { result, state, questions } = await request;
+ const question = questions[id]!;
+ const answer = result.answers[id]!;
+ const score = judgeScore(question, answer);
+ return {
+ score,
+ message: formatJudgeDetail(state, question, answer),
+ metadata: {
+ judge: result.response.modelId,
+ questionId: id,
+ question,
+ state,
+ answer,
+ score,
+ usage: result.usage,
+ usageScope: batch ? "batch" : "assertion",
+ warnings: result.warnings,
+ rounding: result.rounding,
+ providerMetadata: result.providerMetadata,
+ response: toDiagnosticMetadataValue(result.response),
+ },
+ };
+ },
+ }),
+ ]),
+ );
+ return batch ? handles : handles.judgment!;
}
-
- return {
- autoevals: {
- factuality: (expected, opts) =>
- grade(
- "judge.autoevals.factuality",
- opts,
- { label: "expected", value: expected },
- ({ input, output, model, modelOptions }) =>
- Factuality({ input, output, expected, ...client(model, modelOptions) }),
- ),
- summarizes: (expected, opts) =>
- grade(
- "judge.autoevals.summarizes",
- opts,
- { label: "expected", value: expected },
- ({ input, output, model, modelOptions }) =>
- Summary({ input, output, expected, ...client(model, modelOptions) }),
- ),
- closedQA: (criteria, opts) =>
- grade(
- "judge.autoevals.closedQA",
- opts,
- { label: "criteria", value: criteria },
- ({ input, output, model, modelOptions }) =>
- ClosedQA({ input, output, criteria, ...client(model, modelOptions) }),
- ),
- sql: (expected, opts) =>
- grade(
- "judge.autoevals.sql",
- opts,
- { label: "expected", value: expected },
- ({ input, output, model, modelOptions }) =>
- Sql({ input, output, expected, ...client(model, modelOptions) }),
- ),
- },
- };
+ return judge;
}
-function formatJudgeDetail(
- input: string,
- output: string,
- evidence: GraderEvidence,
- result: Score,
-): string {
- const fields: [string, unknown][] = [
- ["prompt", input],
- [evidence.label, evidence.value],
- ["response", output],
- ];
- if (result.metadata?.rationale !== undefined) {
- fields.push(["rationale", result.metadata.rationale]);
- }
- if (result.metadata?.choice !== undefined) {
- fields.push(["choice", result.metadata.choice]);
- }
- if (result.error !== undefined) {
- fields.push(["autoevals error", result.error]);
- }
- return fields.map(([label, value]) => `${label}: ${formatDiagnosticValue(value)}`).join("\n");
+function formatJudgeDetail(state: JudgeInput, question: JudgeQuestion, answer: unknown): string {
+ return [
+ ["state", state],
+ ["question", question],
+ ["answer", answer],
+ ]
+ .map(([label, value]) => `${label}: ${formatDiagnosticValue(value)}`)
+ .join("\n");
}
-type AutoevalsClientFields = {
- readonly model: string;
- readonly client: ReturnType<typeof createAutoevalsClient>;
-};
-
-/** Resolves the gateway model id and the AI-SDK-backed autoevals client. */
-function client(
- model: LanguageModel,
- modelOptions: AgentModelOptionsDefinition | undefined,
-): AutoevalsClientFields {
- return {
- model: formatLanguageModelGatewayId(model),
- client: createAutoevalsClient({
- languageModel: model,
- providerOptions: modelOptions?.providerOptions as Parameters<
- typeof createAutoevalsClient
- >[0]["providerOptions"],
- }),
- };
+function judgeScore(
+ question: JudgeQuestion,
+ answer: Awaited<ReturnType<typeof evaluate>>["answers"][string],
+): number {
+ switch (answer.type) {
+ case "boolean":
+ return answer.probability;
+ case "score":
+ if (question.type === "score") return answer.score / (question.criteria.length - 1);
+ break;
+ case "choice":
+ if (question.type === "choice") return answer.choice === question.expected ? 1 : 0;
+ break;
+ }
+ throw new Error("Evaluation answer does not match the judge question.");
}
diff --git a/packages/eve/src/evals/runner/artifacts.test.ts b/packages/eve/src/evals/runner/artifacts.test.ts
index 6924e4853..a1976f925 100644
--- a/packages/eve/src/evals/runner/artifacts.test.ts
+++ b/packages/eve/src/evals/runner/artifacts.test.ts
@@ -70,13 +70,14 @@ describe("eval artifacts", () => {
{
assertions: [
{
- message: 'prompt: "Name the source."',
+ message: 'state: {"input":"Name the source.","output":"No source."}',
metadata: {
- criteria: "cites a source",
- input: "Name the source.",
- rationale: "No source was cited.",
+ question: { type: "boolean", instructions: "cites a source" },
+ state: { input: "Name the source.", output: "No source." },
+ answer: { type: "boolean", probability: 0 },
+ usageScope: "assertion",
},
- name: "judge.autoevals.closedQA",
+ name: "judge.boolean",
passed: false,
score: 0,
severity: "soft",
@@ -132,16 +133,17 @@ function judgedSummary(): EveEvalRunSummary {
id: "quality/source",
assertions: [
{
- name: "judge.autoevals.closedQA",
+ name: "judge.boolean",
score: 0,
severity: "soft",
threshold: 0.8,
passed: false,
- message: 'prompt: "Name the source."',
+ message: 'state: {"input":"Name the source.","output":"No source."}',
metadata: {
- criteria: "cites a source",
- input: "Name the source.",
- rationale: "No source was cited.",
+ question: { type: "boolean", instructions: "cites a source" },
+ state: { input: "Name the source.", output: "No source." },
+ answer: { type: "boolean", probability: 0 },
+ usageScope: "assertion",
},
},
],
diff --git a/packages/eve/src/evals/runner/discover.ts b/packages/eve/src/evals/runner/discover.ts
index dc61b1b8d..93d169f79 100644
--- a/packages/eve/src/evals/runner/discover.ts
+++ b/packages/eve/src/evals/runner/discover.ts
@@ -180,7 +180,7 @@ export async function discoverEvalConfig(appRoot: string): Promise<EveEvalConfig
throw new Error(
`Missing required eval config at evals/${EVAL_CONFIG_FILE}. Create it with ` +
"defineEvalConfig({}) (optionally `{ judge: { model } }` to set the default judge " +
- "model for `t.judge.*` assertions).",
+ "model for `t.judge(...)` assertions).",
);
}
throw error;
diff --git a/packages/eve/src/evals/runner/execute-task.ts b/packages/eve/src/evals/runner/execute-task.ts
index 561c56dd7..296b85be4 100644
--- a/packages/eve/src/evals/runner/execute-task.ts
+++ b/packages/eve/src/evals/runner/execute-task.ts
@@ -1,3 +1,4 @@
+import { runUntilAborted } from "#evals/abort.js";
import type { Client } from "#client/client.js";
import type { MessageStreamEvent, RuntimeIdentity } from "#protocol/message.js";
import { toErrorMessage } from "#shared/errors.js";
@@ -198,21 +199,3 @@ function sum<T>(entries: readonly T[], read: (entry: T) => number): number {
function neverAbortSignal(): AbortSignal {
return new AbortController().signal;
}
-
-async function runUntilAborted(task: void | Promise<void>, signal: AbortSignal): Promise<void> {
- signal.throwIfAborted();
-
- let onAbort: (() => void) | undefined;
- const aborted = new Promise<never>((_resolve, reject) => {
- onAbort = () => reject(signal.reason);
- signal.addEventListener("abort", onAbort, { once: true });
- });
-
- try {
- await Promise.race([task, aborted]);
- } finally {
- if (onAbort !== undefined) {
- signal.removeEventListener("abort", onAbort);
- }
- }
-}
diff --git a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts
index 06dbe00df..6d64d4daa 100644
--- a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts
+++ b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts
@@ -10,7 +10,7 @@ describe("assertion failure diagnostics", () => {
it("shows the score against its threshold and bounds verbose details", () => {
const assertion: AssertionResult = {
message: [`prompt: ${"x".repeat(300)}`, "one", "two", "three", "four", "five"].join("\n"),
- name: "judge.autoevals.closedQA [citation]",
+ name: "judge.boolean [citation]",
passed: false,
score: 0.42,
severity: "soft",
@@ -18,7 +18,7 @@ describe("assertion failure diagnostics", () => {
};
expect(formatAssertionFailureHeadline(assertion)).toMatch(
- /^judge\.autoevals\.closedQA \[citation\] \(42% < 80%\): prompt: .+…$/u,
+ /^judge\.boolean \[citation\] \(42% < 80%\): prompt: .+…$/u,
);
expect(formatAssertionFailureDetailLines(assertion)).toEqual([
"one",
diff --git a/packages/eve/src/evals/runner/reporters/braintrust.test.ts b/packages/eve/src/evals/runner/reporters/braintrust.test.ts
index 0c52db6ea..2ddcb1ea5 100644
--- a/packages/eve/src/evals/runner/reporters/braintrust.test.ts
+++ b/packages/eve/src/evals/runner/reporters/braintrust.test.ts
@@ -144,7 +144,7 @@ describe("Braintrust", () => {
input: "Name the source.",
rationale: "No source was cited.",
},
- name: "judge.autoevals.closedQA",
+ name: "judge.boolean",
passed: false,
score: 0,
severity: "soft",
@@ -166,7 +166,7 @@ describe("Braintrust", () => {
input: "Name the source.",
rationale: "No source was cited.",
},
- name: "judge.autoevals.closedQA",
+ name: "judge.boolean",
passed: false,
score: 0,
severity: "soft",
diff --git a/packages/eve/src/evals/runner/reporters/console.test.ts b/packages/eve/src/evals/runner/reporters/console.test.ts
index 6fdbc054a..ae7dbaa7c 100644
--- a/packages/eve/src/evals/runner/reporters/console.test.ts
+++ b/packages/eve/src/evals/runner/reporters/console.test.ts
@@ -25,11 +25,11 @@ describe("Console", () => {
assertions: [
{
message: [
- 'prompt: "Name the source."',
- 'criteria: "cites a source"',
- 'rationale: "No source was cited."',
+ 'state: {"input":"Name the source."}',
+ 'question: {"type":"boolean","instructions":"cites a source"}',
+ 'answer: {"type":"boolean","probability":0}',
].join("\n"),
- name: "judge.autoevals.closedQA",
+ name: "judge.boolean",
passed: false,
score: 0,
severity: "soft",
@@ -50,9 +50,9 @@ describe("Console", () => {
verdict: "scored",
});
- expect(lines).toContain(' ✗ judge.autoevals.closedQA (0% < 80%): prompt: "Name the source."');
- expect(lines).toContain(' criteria: "cites a source"');
- expect(lines).toContain(' rationale: "No source was cited."');
+ expect(lines).toContain(' ✗ judge.boolean (0% < 80%): state: {"input":"Name the source."}');
+ expect(lines).toContain(' question: {"type":"boolean","instructions":"cites a source"}');
+ expect(lines).toContain(' answer: {"type":"boolean","probability":0}');
});
it("does not print diagnostics for passing assertions", () => {
@@ -64,7 +64,7 @@ describe("Console", () => {
{
message: 'prompt: "Successful prompt details"',
metadata: { rationale: "Successful rationale details" },
- name: "judge.autoevals.closedQA",
+ name: "judge.boolean",
passed: true,
score: 1,
severity: "gate",
diff --git a/packages/eve/src/evals/runner/reporters/datadog.test.ts b/packages/eve/src/evals/runner/reporters/datadog.test.ts
index 2af2af047..a97f82c6b 100644
--- a/packages/eve/src/evals/runner/reporters/datadog.test.ts
+++ b/packages/eve/src/evals/runner/reporters/datadog.test.ts
@@ -68,7 +68,7 @@ function makeEvalResult(overrides: Partial<EveEvalResult> = {}): EveEvalResult {
assertions: [
{ name: "succeeded", score: 1, severity: "gate", passed: true },
{ name: "similarity", score: 0.9, severity: "soft", threshold: 0.6, passed: true },
- { name: "judge.autoevals.closedQA", score: 1, severity: "soft", passed: true },
+ { name: "judge.boolean", score: 1, severity: "soft", passed: true },
],
verdict: "passed",
startedAt: "2026-01-01T00:00:00.000Z",
@@ -204,7 +204,7 @@ describe("Datadog", () => {
expect(experiment.submitEvaluationMetrics).toHaveBeenCalledWith(span, [
expect.objectContaining({ label: "gate_succeeded", value: 1 }),
expect.objectContaining({ label: "similarity", value: 0.9 }),
- expect.objectContaining({ label: "judge_autoevals_closedQA", value: 1 }),
+ expect.objectContaining({ label: "judge_boolean", value: 1 }),
expect.objectContaining({ label: "eve_tool_call_count", value: 1 }),
expect.objectContaining({ label: "eve_subagent_call_count", value: 0 }),
expect.objectContaining({ label: "eve_message_count", value: 1 }),
diff --git a/packages/eve/src/evals/session.test.ts b/packages/eve/src/evals/session.test.ts
index 4cc1cae34..d6198dac6 100644
--- a/packages/eve/src/evals/session.test.ts
+++ b/packages/eve/src/evals/session.test.ts
@@ -10,11 +10,10 @@ import { EvalSessionManager } from "#evals/session-manager.js";
import { createEvalTargetHandle } from "#evals/target.js";
import { stampTestEvents } from "#internal/testing/events.js";
-const mocks = vi.hoisted(() => ({ closedQA: vi.fn() }));
+const mocks = vi.hoisted(() => ({ evaluate: vi.fn() }));
vi.mock("node:fs/promises", () => ({ readFile: vi.fn() }));
-vi.mock("autoevals", () => ({ ClosedQA: mocks.closedQA }));
-vi.mock("#evals/autoevals-client.js", () => ({ createAutoevalsClient: vi.fn() }));
+vi.mock("#ai/evaluate.js", () => ({ evaluate: mocks.evaluate }));
afterEach(() => {
vi.restoreAllMocks();
@@ -38,9 +37,11 @@ describe("eval judge input", () => {
await session.readTurn();
}
- context.judge.autoevals.closedQA("answers the user's request");
- expect(mocks.closedQA).toHaveBeenCalledWith(
- expect.objectContaining({ input: "Find Bob's order status instead." }),
+ context.judge("answers the user's request");
+ expect(mocks.evaluate).toHaveBeenCalledWith(
+ expect.objectContaining({
+ state: expect.objectContaining({ input: "Find Bob's order status instead." }),
+ }),
);
},
);
@@ -50,10 +51,12 @@ describe("eval judge input", () => {
vi.mocked(readFile).mockResolvedValue(Buffer.from("invoice"));
await session.sendFile("Summarize Alice's invoice.", "/invoice.txt", "text/plain");
- context.judge.autoevals.closedQA("summarizes the invoice");
+ context.judge("summarizes the invoice");
- expect(mocks.closedQA).toHaveBeenCalledWith(
- expect.objectContaining({ input: "Summarize Alice's invoice." }),
+ expect(mocks.evaluate).toHaveBeenCalledWith(
+ expect.objectContaining({
+ state: expect.objectContaining({ input: "Summarize Alice's invoice." }),
+ }),
);
});
@@ -70,19 +73,26 @@ describe("eval judge input", () => {
file,
{ type: "text", text: "List the line items." },
]);
- context.judge.autoevals.closedQA("lists the line items");
- expect(mocks.closedQA).toHaveBeenLastCalledWith(
- expect.objectContaining({ input: "Review Alice's invoice.\nList the line items." }),
+ context.judge("lists the line items");
+ expect(mocks.evaluate).toHaveBeenLastCalledWith(
+ expect.objectContaining({
+ state: expect.objectContaining({ input: "Review Alice's invoice.\nList the line items." }),
+ }),
);
await session.send([file]);
- context.judge.autoevals.closedQA("describes the attachment");
- expect(mocks.closedQA).toHaveBeenLastCalledWith(expect.objectContaining({ input: "" }));
+ context.judge("describes the attachment");
+ expect(mocks.evaluate).toHaveBeenLastCalledWith(
+ expect.objectContaining({ state: expect.objectContaining({ input: "" }) }),
+ );
});
});
async function setup() {
- mocks.closedQA.mockResolvedValue({ name: "ClosedQA", score: 1 });
+ mocks.evaluate.mockResolvedValue({
+ answers: { judgment: { type: "boolean", probability: 1 } },
+ response: { modelId: "test" },
+ });
vi.spyOn(globalThis, "fetch").mockResolvedValue(
Response.json({ sessionId: "session_1" }, { status: 202 }),
);
diff --git a/packages/eve/src/evals/types.ts b/packages/eve/src/evals/types.ts
index 14658674d..91ac7fd22 100644
--- a/packages/eve/src/evals/types.ts
+++ b/packages/eve/src/evals/types.ts
@@ -1,4 +1,4 @@
-import type { LanguageModel } from "ai";
+import type { Experimental_EvaluationModel as EvaluationModel } from "ai";
import type { StandardSchemaV1 } from "#compiled/@standard-schema/spec/index.js";
import type {
@@ -353,47 +353,66 @@ export interface EveEvalTurn extends EveEvalAssertions, EveEvalOutputAssertions
// Judge (LLM-as-judge)
// ---------------------------------------------------------------------------
-/**
- * The judge model used by `t.judge.*` assertions, configured per-eval or as
- * the run-wide default in `evals.config.ts`. Only ever used for scoring; it
- * never changes the agent under test. String model ids route through the
- * Vercel AI Gateway; provider model instances run directly.
- */
+/** Evaluation settings used only for scoring, independently of the agent under test. */
export interface EveEvalJudgeConfig {
- readonly model: LanguageModel;
+ /** Evaluation model ID or instance. Defaults to the model used by `eve/ai` evaluate. */
+ readonly model?: EvaluationModel;
readonly modelOptions?: AgentModelOptionsDefinition;
}
-/**
- * Per-call options for `t.judge.autoevals.*` assertions.
- */
-export interface JudgeOpts {
- /** Value to grade. Defaults to the most recently settled turn’s assistant message. */
- readonly on?: unknown;
- /** Judge model for this call only; overrides the eval/config judge model. */
- readonly model?: LanguageModel;
- readonly modelOptions?: AgentModelOptionsDefinition;
+/** JSON content accepted as evaluation state, instructions, or rubric descriptions. */
+export type JudgeInput = string | JsonObject | readonly JsonValue[];
+
+/** A boolean judgment, ordered rubric, or categorical judgment with an expected option. */
+export type JudgeQuestion =
+ | {
+ readonly type: "boolean";
+ readonly instructions: JudgeInput;
+ readonly criteria?: { readonly true?: JudgeInput | null; readonly false?: JudgeInput | null };
+ }
+ | {
+ readonly type: "score";
+ readonly instructions: JudgeInput;
+ readonly criteria: readonly (JudgeInput | null)[];
+ }
+ | {
+ readonly type: "choice";
+ readonly instructions: JudgeInput;
+ readonly criteria: Readonly<Record<string, JudgeInput | null>>;
+ readonly expected: string;
+ };
+
+/** Restricts a choice expectation to one of its authored option keys. */
+export type JudgeQuestionConstraint<Q extends JudgeQuestion> = Q extends { type: "choice" }
+ ? { readonly expected: NoInfer<Extract<keyof Q["criteria"], string>> }
+ : unknown;
+
+/** Named judgments evaluated together against one shared state. */
+export interface JudgeBatch<Questions extends Record<string, JudgeQuestion>> {
+ /** Replaces the default `{ input, output }` state when supplied. */
+ readonly state?: JudgeInput;
+ readonly questions: Questions & {
+ readonly [Key in keyof Questions]: JudgeQuestionConstraint<Questions[Key]>;
+ };
}
-/**
- * Braintrust autoevals graders, bound to the resolved judge model. The grader
- * family is named so its semantics are explicit: `factuality`'s consistency
- * buckets and `closedQA`'s yes/no grading are autoevals' behavior, not eve's.
- * These are eve-owned wrappers, not the raw library.
- */
-export interface AutoevalsJudges {
- factuality(expected: string, opts?: JudgeOpts): AssertionHandle;
- summarizes(expected: string, opts?: JudgeOpts): AssertionHandle;
- closedQA(criteria: string, opts?: JudgeOpts): AssertionHandle;
- sql(expected: string, opts?: JudgeOpts): AssertionHandle;
+/** Per-call settings override the resolved eval/project judge settings. */
+export interface JudgeOpts extends EveEvalJudgeConfig {
+ /** JSON value to grade instead of the latest settled turn's assistant message. */
+ readonly on?: JsonValue;
}
-/**
- * Model-backed assertion namespaces on `t.judge`. A future non-autoevals
- * engine would slot in as a sibling of `autoevals`.
- */
+/** Model-backed assertions. Calls start immediately and return non-awaitable handles. */
export interface JudgeContext {
- readonly autoevals: AutoevalsJudges;
+ (criteria: string, opts?: JudgeOpts): AssertionHandle;
+ <const Q extends JudgeQuestion>(
+ question: Q & JudgeQuestionConstraint<Q>,
+ opts?: JudgeOpts,
+ ): AssertionHandle;
+ <const Questions extends Record<string, JudgeQuestion>>(
+ batch: JudgeBatch<Questions>,
+ opts?: EveEvalJudgeConfig,
+ ): { readonly [Key in keyof Questions]: AssertionHandle };
}
/**
@@ -495,10 +514,10 @@ export interface EveEvalTargetHandle extends EveEvalTarget {
interface EveEvalBase {
readonly description?: string;
/**
- * Judge model for this eval's `t.judge.*` assertions. Optional: when
+ * Judge model for this eval's `t.judge(...)` assertions. Optional: when
* omitted, judge assertions fall back to the `judge` declared in
- * `evals.config.ts`. Only used for scoring; never changes the agent
- * under test.
+ * `evals.config.ts`, then the shared evaluation default. Only used for
+ * scoring; never changes the agent under test.
*/
readonly judge?: EveEvalJudgeConfig;
readonly timeoutMs?: number;
@@ -611,8 +630,8 @@ export interface EveEvalRunSummary {
*/
export interface EveEvalConfigInput {
/**
- * Default judge model for `t.judge.*` assertions across every eval.
- * Optional: evals that use no judge need not set it, and individual evals
+ * Default judge model for `t.judge(...)` assertions across every eval.
+ * Optional: omission uses the shared evaluation default. Individual evals
* may override it with their own `judge`. Only ever used for scoring.
*/
readonly judge?: EveEvalJudgeConfig;
diff --git a/packages/eve/src/internal/testing/scenario-apps/weather-agent.ts b/packages/eve/src/internal/testing/scenario-apps/weather-agent.ts
index 4784f5da0..054faf14c 100644
--- a/packages/eve/src/internal/testing/scenario-apps/weather-agent.ts
+++ b/packages/eve/src/internal/testing/scenario-apps/weather-agent.ts
@@ -74,9 +74,9 @@ const WEATHER_SYSTEM_SOURCE =
const WEATHER_EVAL_CONFIG_SOURCE = `import { defineEvalConfig } from "eve/evals";
// Run-wide config shared by every eval. The optional \`judge\` model is the
-// default for \`t.judge.*\` assertions, so individual evals need not repeat it.
+// default for \`t.judge(...)\` assertions, so individual evals need not repeat it.
export default defineEvalConfig({
- judge: { model: "openai/gpt-5.6-luna" },
+ judge: { model: "typesafe-ai/jev" },
});
`;
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 114672da3..6c94de3a5 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -1448,6 +1448,9 @@ importers:
e2e/fixtures/e2e-config:
dependencies:
+ '@ai-sdk/openai':
+ specifier: 'catalog:'
+ version: 4.0.69([email protected])
'@opentelemetry/sdk-trace-base':
specifier: 'catalog:'
version: 2.6.1(@opentelemetry/[email protected])
@@ -1729,9 +1732,6 @@ importers:
ai:
specifier: 'catalog:'
version: 7.0.105([email protected])
- autoevals:
- specifier: 0.0.132
chat:
specifier: 4.34.0
@@ -8736,9 +8736,6 @@ packages:
resolution: {integrity: sha512-P4w9o2dqARji6P7MHprklbfiArZAWvo07yW7qs3pdljb3BWr12FIB7W+p0zJiuiVsUpRO0iZn1kFFcpPegg0tQ==}
- resolution: {integrity: sha512-x033hXLO1Vyggbv68Y1QeoZlrdKHcNexcutPhVaDhDJ4SO6TU1rG6vME77g/zKvH4VWFVBPDWM1pRbPxAFF+sA==}
-
resolution: {integrity: sha512-cL1Qz6ADZhcEbny/8HPfe99J6HhNoYtpX2LFLIbhgGE7Q1hlQVkYFdetDN7Id3KiQxhDrHwzlHr/YQCnZ8+xSA==}
engines: {node: ^10 || ^12 || >=14}
@@ -8856,9 +8853,6 @@ packages:
resolution: {integrity: sha512-Ko0uX15oIUS7wJ3Rb30Fs6SkVbLmPBAKdlm7q9+ak9bbIeFf0MwuBsQV6z7+X768/cHsfg+WlysDWJcmthjsjQ==}
- resolution: {integrity: sha512-nbE1WxOTTrUWIfsfZ4aHGYu5DOuNkbxGokjV6Z2kxfJK3uaAb8zNK1muzOeipoLHZjInT4Br88BHpzevc681xA==}
-
resolution: {integrity: sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==}
@@ -9119,9 +9113,6 @@ packages:
resolution: {integrity: sha512-WDrybc/gKFpTYQutKIK6UvfcuxijIZfMfXaYm8NMsPQxSYvf+13fXUJ4rztGGbJcBQ/GF55gvrZ0Bc0bj/mqvg==}
engines: {node: '>=20.18.1'}
- resolution: {integrity: sha512-j0MiXFnLRVEZg3vUr7SXm6F3HuDNVE5arfQyDMiy1w2fAcuOs3QRgVsw7aOIACvr0aqbvn85cucwt+yOu64IiQ==}
-
resolution: {integrity: sha512-mxIojEAQcuEvT/lyXq+jf/3cO/KoA6z4CeNDGGevTybECPOMFCnQy3OPahluUkbqgPNGw5Bi78UC7Po6Lhy+NA==}
engines: {node: '>= 14.16.0'}
@@ -9285,15 +9276,6 @@ packages:
resolution: {integrity: sha512-6FqVXeETqWPoGcfzrXb37E50NP0LXT8kAMu5ooZayhWWdgEY4lBEEcbQNXtkuKQsGduxiIcI4gOTsxTmuq/bSg==}
engines: {node: '>= 14'}
- resolution: {integrity: sha512-FXhNx0ILLjGi9Z9+lglLzM12+0uoTnYkHm7GiadXDAr0HGVLm25OivUS1B/LPkbzzvlcXz/1EvWg9ZYyJSdhTw==}
-
- resolution: {integrity: sha512-L5Ocet4DdMrXboss13K59OK23GXjiSia7+7Ukc7q4Bl+RVpIXK2W9IHMbWDZkh+JUEvJAwOKRaJDiFUa1LTnJg==}
-
- resolution: {integrity: sha512-6EHh1Elj90eU28SXi+h2PLnTQvZmkkHWySpoFz+WOlVNLz3DQoC4ISUHSV9n5jMxPHtKGJ01F4uu2PsXBB8sSg==}
-
resolution: {integrity: sha512-VRhuHOLoKYOy4UbilLbUzbYg93XLjv2PncJC50EuTWPA3gaja1UjBsUP/D/9/juV3vQFr6XBEzn9KCAHdUvOHw==}
@@ -10612,9 +10594,6 @@ packages:
resolution: {integrity: sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA==}
- resolution: {integrity: sha512-f9c00hphOgeQTlDyavwTtu6RiK8AIFjD6+jvXkNkpeQ7rirK3uFWVpalkoS4LAwbdX7mfZ8aoBfFVQX1Re/8aw==}
-
resolution: {integrity: sha512-d+l3qxjSesT4V7v2fh+QnmFnUWv9lSpjarhShNTgBOfA0ttejbQUAlHLitbjkoRiDulW0OPoQPYIGhIC8ohejg==}
engines: {node: '>=18'}
@@ -11390,9 +11369,6 @@ packages:
resolution: {integrity: sha512-hmbYhX/9MUMF5uh7tOXyK/n0ZvWpad5caBA17GsC6vyuCqaWliRG5K1qS9inmUhEMaOBIW7/whAnSwveW/LtZw==}
- resolution: {integrity: sha512-o4h+tylWykC4BD1vaejp6gDxoM13bwW8FGuNs4yIKpj8xbBJcRxJx8vZpq0dCr7ZDEfeKjmsi/euolKhX6f/ww==}
-
resolution: {integrity: sha512-DDfANUiiG2wC1qawP66qlTugJeL5HyzMpfr8lLK+jMQirGzNod0B12cFB/9q838Ru27sBwfw78/rdoU7RERz6A==}
engines: {node: '>= 0.4'}
@@ -11696,10 +11672,6 @@ packages:
resolution: {integrity: sha512-uwYQp+VJ38FVvtim6qNbit6e9uT6dwWQ4Y1+H9TxhW5hcHjpHwoxlR0nMpqUmIFOmu4VqMxwdJA88gIVuZJQ/g==}
- resolution: {integrity: sha512-X2BB11YZtrRqY4EnQcLX5Rh373zbK4alC1FW7D7MBhL2gtcC17cTnr6DmfHZeS0s2rTHjUTMMHfG7gO8SSdw+g==}
- engines: {node: '>=0.10.0'}
-
resolution: {integrity: sha512-biOj/6M5qdgx5TKjDnFT1ymSpM5tbd3ylwDtrQvFQSu0Z7bBYko2dF+W/aUkXUPuk6IVpRxk/3Q2sHOzGlS36g==}
@@ -12059,9 +12031,6 @@ packages:
resolution: {integrity: sha512-/vlFKAoH5Cgt3Ie+JLhRbwOsCQePABiU3tJ1egGvyQ+33R/vcwM2Zl2QR/LzjsBeItPt3oSVXapn+m4nQDvpzw==}
engines: {node: '>=14'}
- resolution: {integrity: sha512-1T2Ek3sxpt2mBHeBFMRJEikiIK/yIOwf+mrxv/DkAU/5ddnCMndZL//hFH7QuHa1tbaQADzsf9t7rkGZKqoFfQ==}
-
resolution: {integrity: sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg==}
@@ -12648,24 +12617,6 @@ packages:
engines: {node: '>=10'}
hasBin: true
- resolution: {integrity: sha512-QQZ4kENwpWmyNb98UXRDFXrmtIXuXtt1+bSbda/2KA85+F+rrJP8hZk6QOkCQXM2Th9mUDYdq/PNByPdT9ID4A==}
-
- resolution: {integrity: sha512-GRj6Ky6sW9vGL6yIjgsHmXZ9YgrdmcQ8nCxPqEGeKc6dkfYg1XDYxGFxADUjNuZyoCd5PUscWAS4N+cFaX6hFg==}
-
- resolution: {integrity: sha512-2GGtKfSno94/kIloWGvpp/U5Q5vLvLrza+SAaGsLeo6Xj4mEbA6Gqx+oTfZFkxnd1grT2X007HfJNs3T5BsiVg==}
-
- resolution: {integrity: sha512-wFa1v6KP8bKp+fj0nYmRs1Pb5K4zRkXGKsOvLinvILENFIADncm4XlOI+S1M7yuACMGfI6cfk0IifDgd4j5xmw==}
-
- resolution: {integrity: sha512-yxTB9YpgNmo53U+c2tG1/uKKUl+CNO3hlp+3utuBlhAqDw3AD9c78gI+1rYz+sykOXV76QtAXuAhFZFZjMvgDw==}
-
- resolution: {integrity: sha512-Fz2q6dwgzGM8wYKGArTUTZDGa4lQFA2Vi6orjGeTVRy22ZnQFKlJuwS9n8NRviqz1KHAHAzdKJwbnYhdo38uYg==}
-
resolution: {integrity: sha512-d+ObxMQFmbt10sretNDytwt85VrbkhhUA/JBGm1MPaWJ65Cl4wOgLaB1NYvJSZ0Ef03MMEU/0xpPMXUIQ29UfA==}
@@ -15632,12 +15583,6 @@ packages:
resolution: {integrity: sha512-hVDIBwsRruT73PbK7uP5ebUt+ezEtCmzZz3F59BSr2F6OVFnJ/6h8liuvdLrQ88Xmnk6/+xGGuq+pG9WwTuy3A==}
engines: {node: ^20.17.0 || >=22.9.0}
- resolution: {integrity: sha512-DeOy7CnPEziggrOO5CZhVKJw6S3Yi7e9e65R1Nl/RTN1vTQKnzjfvks0/8kQ40FP/dsjRAOd4hxmJ7uLa6vxkg==}
-
- resolution: {integrity: sha512-LlFybRJEriSuBnUhQyG5bwglhh50EpTL2ul23MPIuR1odjO7XaMLFV8vHGwp7AZciFxtYOeiSCT5st+XSPONiQ==}
-
resolution: {integrity: sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==}
engines: {node: '>= 0.8'}
@@ -23794,23 +23739,6 @@ snapshots:
stubborn-fs: 2.0.0
when-exit: 2.1.5
- dependencies:
- ajv: 8.20.0
- compute-cosine-similarity: 1.1.0
- js-levenshtein: 1.1.6
- js-yaml: 4.3.2
- linear-sum-assignment: 1.0.9
- mustache: 4.2.0
- zod: 3.25.76
- zod-to-json-schema: 3.25.2([email protected])
- transitivePeerDependencies:
- - '@aws-sdk/credential-provider-node'
- - '@smithy/hash-node'
- - '@smithy/signature-v4'
- - ws
-
dependencies:
browserslist: 4.29.0
@@ -23910,8 +23838,6 @@ snapshots:
-
dependencies:
file-uri-to-path: 1.0.0
@@ -24260,8 +24186,6 @@ snapshots:
undici: 7.29.1
whatwg-mimetype: 4.0.0
-
dependencies:
readdirp: 4.1.2
@@ -24411,23 +24335,6 @@ snapshots:
normalize-path: 3.0.0
readable-stream: 4.7.0
- dependencies:
- compute-dot: 1.1.0
- compute-l2norm: 1.1.0
- validate.io-array: 1.0.6
- validate.io-function: 1.0.2
-
- dependencies:
- validate.io-array: 1.0.6
- validate.io-function: 1.0.2
-
- dependencies:
- validate.io-array: 1.0.6
- validate.io-function: 1.0.2
-
@@ -26102,8 +26009,6 @@ snapshots:
-
dependencies:
is-unicode-supported: 2.1.0
@@ -27179,8 +27084,6 @@ snapshots:
is-alphabetical: 2.0.1
is-decimal: 2.0.1
-
dependencies:
call-bind: 1.0.9
@@ -27450,8 +27353,6 @@ snapshots:
-
dependencies:
base64-js: 1.5.1
@@ -27763,12 +27664,6 @@ snapshots:
- dependencies:
- cheminfo-types: 1.16.0
- ml-matrix: 6.15.0
- ml-spectra-processing: 14.35.1
-
@@ -28748,36 +28643,6 @@ snapshots:
- dependencies:
- is-any-array: 3.0.0
-
- dependencies:
- is-any-array: 3.0.0
-
- dependencies:
- is-any-array: 3.0.0
- ml-array-max: 2.0.0
- ml-array-min: 2.0.0
-
- dependencies:
- is-any-array: 3.0.0
- ml-array-rescale: 2.0.0
-
- dependencies:
- binary-search: 1.3.6
- cheminfo-types: 1.16.0
- fft.js: 4.0.4
- is-any-array: 3.0.0
- ml-matrix: 6.15.0
- ml-xsadd: 3.0.1
-
-
dependencies:
acorn: 8.18.0
@@ -29682,12 +29547,6 @@ snapshots:
transitivePeerDependencies:
- encoding
- optionalDependencies:
- '@smithy/signature-v4': 5.7.3
- ws: 8.21.3([email protected])
- zod: 3.25.76
-
optionalDependencies:
'@smithy/signature-v4': 5.7.3
@@ -32647,10 +32506,6 @@ snapshots:
-
-
diff --git a/research/evaluation-model-judges.md b/research/evaluation-model-judges.md
new file mode 100644
index 000000000..8dca06151
--- /dev/null
+++ b/research/evaluation-model-judges.md
@@ -0,0 +1,40 @@
+---
+issue: "TBD (maintainer-requested research; no matching issue found)"
+status: implemented
+last_updated: "2026-09-18"
+---
+
+# Evaluation models for eval judges
+
+`t.judge(...)` uses the existing `evaluate` wrapper from `eve/ai`. A criteria
+string becomes a boolean question; an explicit question returns one assertion
+handle; `{ state?, questions }` returns named handles backed by one request.
+All calls start immediately and retain existing soft scoring, labels, thresholds,
+and gates. The default state is the latest settled turn's `{ input, output }`;
+`on` replaces output for individual questions and batch `state` replaces the whole
+state. Inputs are captured when the assertions are recorded.
+
+Boolean probabilities become scores directly. Ordered rubric positions divide
+by the maximum level index. Choice questions require an expected option and
+score selected-option equality as 0 or 1; the expectation is never sent to the
+model. Batches share one state, preserve question order, and fail together if
+SDK validation or the provider fails.
+
+`judge.model` accepts an evaluation model ID or instance and defaults to the
+shared evaluator, currently `typesafe-ai/jev`. Per-call settings override the
+resolved eval/project configuration. Provider options retain their existing
+`modelOptions.providerOptions` shape. AI SDK owns adapters, authentication,
+validation, and retries. The eval signal cancels provider I/O and bounds how long
+the runner waits during assertion finalization.
+
+The implementation reuses the assertion collector, diagnostics, verdicts, and
+reporters. Metadata records raw answers and normalized scores; batch token usage
+is explicitly shared. Evaluation does not promise prose rationales.
+
+Remove autoevals, its adapter, and its public namespace. Migrate criteria judges
+to the callable form and use explicit state/questions for specialized rubrics.
+Preserve deterministic similarity with the same normalized UTF-16 Levenshtein
+score. Braintrust reporting is independent and remains available. Fixture judges
+no longer inherit the agent model matrix. This is a breaking public API change.
+
+See [Judge](../docs/evals/judge.mdx) for authoring examples and migration guidance.