Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

feat(eve): record eval scores separately from verdicts and warn on merged duplicates (#4565)

Philipp Comansbc2be45f0226
diff --git a/.changeset/eval-scores-and-verdicts.md b/.changeset/eval-scores-and-verdicts.md
new file mode 100644
index 000000000..481e4f8fc
--- /dev/null
+++ b/.changeset/eval-scores-and-verdicts.md
@@ -0,0 +1,5 @@
+---
+"eve": minor
+---
+
+Adds `t.score(evaluation)` for recording a score from 0 to 1, named with `.label(key)` and gated with `.gate()` or `.atLeast()`. `AssertionResult` gains `key`, and `score`, `threshold`, and `passed` are now optional: `score` is absent when the scorer threw, and `threshold` and `passed` are absent when no rule applies, so custom reporters should treat `passed === false` as the failure signal. `eve eval` warns when soft assertions in one eval share a name with different scores.
diff --git a/docs/evals/assertions.mdx b/docs/evals/assertions.mdx
index a8bf20118..dc08d4b86 100644
--- a/docs/evals/assertions.mdx
+++ b/docs/evals/assertions.mdx
@@ -3,7 +3,7 @@ title: "Assertions"
description: "Scoped methods, value assertions, the matcher mini-language, and gate vs soft severity."
---
-Assertions are how an eval grades what its `test(t)` function produced. Each one records a result and returns a chainable handle. The runner reads every recorded result to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are two deterministic surfaces: scoped methods and `t.check` for grading a specific value. For model-graded assertions, see [Judge](./judge).
+Assertions are how an eval grades what its `test(t)` function produced. Each one records a result and returns a chainable handle. The runner reads every recorded result to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are three deterministic surfaces: scoped methods, `t.check` for grading a specific value, and `t.score` for recording a measurement you computed yourself. For model-graded assertions, see [Judge](./judge).
## Scoped assertions
@@ -83,6 +83,53 @@ For example, a failed labeled equality check prints:
Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground. For nuanced grading, reach for the [judge](./judge).
+## Scores with `t.score`
+
+`t.score(evaluation)` records a score from 0 to 1. The evaluation is the number or `{ score, message?, metadata? }`, or a promise of either. Chain `.label(key)` to name it; reporters and artifacts expose the label as `key`. Without a label the score reports as `score`, and several unlabeled scores share that name.
+
+```ts
+const reply = turn.message ?? "";
+t.score({
+ score: Math.max(0, 1 - reply.length / 400),
+ metadata: { length: reply.length },
+}).label("reply-brevity");
+t.score(gradeFaithfulness(reply, reference)).label("faithfulness"); // async
+```
+
+On its own a score has no verdict and does not affect the eval's outcome. Chain a rule when it should:
+
+```ts
+t.score(gradeFaithfulness(reply, reference)).label("faithfulness").gate(0.9); // fails below 0.9
+t.score(brevity).label("reply-brevity").atLeast(0.5); // marks the eval `scored`, fatal under --strict
+```
+
+The score is reported as measured. A `0.82` gated at `0.9` is reported as `score: 0.82`, `threshold: 0.9`, `passed: false`. An evaluator that throws or returns a score outside 0 to 1 records an errored outcome with the error message and no score, and fails the eval.
+
+Every [reporter](./reporters#custom-reporters) receives the entries in `onEvalComplete` and can read them by `key`:
+
+```ts
+onEvalComplete(result) {
+ for (const { key, score, threshold, passed } of result.assertions) {
+ if (key === undefined) continue;
+ store.write({ evalId: result.id, key, score, threshold, passed });
+ }
+}
+```
+
+The same entries land in the [run artifacts](./running#artifacts) under `.eve/evals/`, in `summary.json` and one line per eval in `results.jsonl`:
+
+```json
+{
+ "name": "score [reply-brevity]",
+ "key": "reply-brevity",
+ "score": 0.84,
+ "severity": "soft",
+ "errored": false
+}
+```
+
+`threshold` and `passed` are present only when a rule was chained.
+
## The matcher mini-language
`t.calledTool` and `t.calledSubagent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ callId, childSessionId, remoteUrl, output, status, count }`. Calls match `status: "completed"` by default. Tool lifecycle checks also accept `"pending"`, `"failed"`, and `"rejected"`. Subagent checks accept `"working"`, `"completed"`, `"failed"`, and `"cancelled"`. A numeric `count` requires an exact number of calls matching every supplied constraint. Use a predicate for ranges or other custom count requirements; it receives the observed number of matching calls.
@@ -164,9 +211,9 @@ const request = session.requireInputRequest({ toolName: "guarded" });
Every assertion returns a chainable handle. Severity rides on the assertion, so there is no separate thresholds map to keep in sync.
- `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
-- `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold, it is tracked-only and never fails.
+- `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold it records no verdict and never fails.
- `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).
-- `.label(name)` adds a stable name when an eval records several assertions from the same family.
+- `.label(key)` adds a stable name when an eval records several assertions from the same family. Reporters and artifacts expose it as `key`. Braintrust and Datadog keep one score per name, so `eve eval` warns when soft assertions in one eval share a name with different scores.
The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge(...)` assertion are soft. Annotate only when you deviate:
diff --git a/docs/evals/overview.mdx b/docs/evals/overview.mdx
index e3a8227d8..d36bff92e 100644
--- a/docs/evals/overview.mdx
+++ b/docs/evals/overview.mdx
@@ -154,6 +154,7 @@ Each surface matches a genuinely different kind of judgment:
- **Scoped methods** read the final whole run on `t`, snapshot one independent session when invoked there, or inspect one immutable `EveEvalTurn`. See [Assertions](./assertions).
- **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect`, such as `t.check(turn.message, includes("sunny"))`. Grade `turn.message`, an intermediate draft, parsed JSON, or anything else. See [Assertions](./assertions).
+- **`t.score(evaluation)`** records a score, such as a length ratio or an evaluator's result. Chain `.label(key)` to name it, and `.gate(threshold)` or `.atLeast(threshold)` to let it decide the eval. See [Assertions](./assertions#scores-with-tscore).
- **`t.judge(...)`** is the LLM-as-judge surface, like `t.judge("cites a source")`. It grades the most recently settled turn’s assistant message by default; pass `{ on: turn.session.transcript }` to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See [Judge](./judge).
## Gate vs soft
@@ -161,7 +162,7 @@ Each surface matches a genuinely different kind of judgment:
Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.
- **Gates** are hard. A failed gate marks the eval `failed` and `eve eval` exits non-zero. Run-level methods, `includes`, `equals`, and `matches` are gates by default.
-- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity` and every `t.judge(...)` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.
+- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity`, every `t.judge(...)` assertion, and every `t.score(...)` are soft by default. A soft assertion with no threshold records no verdict and never fails.
Override per assertion: `.gate(threshold?)` promotes to a hard gate, `.soft(threshold?)` demotes to tracked, and `.atLeast(threshold)` is a soft assertion with a bar.
diff --git a/docs/evals/reporters.mdx b/docs/evals/reporters.mdx
index 4eed6be64..7e068f7f3 100644
--- a/docs/evals/reporters.mdx
+++ b/docs/evals/reporters.mdx
@@ -38,7 +38,7 @@ export default defineEval({
});
```
-The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Assertions that share a name, such as several unlabeled judges, log one score: the lowest. Use `.label(name)` to group or separate them. Each assertion's score is stored in the Braintrust metadata under `eveAssertionScores` and failed assertion details under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.
+The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Assertions that share a name, such as several unlabeled judges, log one score: the lowest. The console warns under the eval when merged soft scores differ; use `.label(name)` to group or separate them. Errored assertions log no score. Each assertion's score is stored in the Braintrust metadata under `eveAssertionScores` and failed assertion details under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.
A reporter instance observes the evals that reference it. Share one instance across several evals (the config, a `shared.ts` export, or every entry of a dataset array) and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
@@ -97,6 +97,8 @@ interface EvalReporter {
- `onSessionStart` fires once for each session after eve receives its first trace context. It includes `sessionId`, `primary`, and `traceContext` with `traceId`, `spanId`, and `traceFlags`.
- `onEvalComplete` fires with the checks, scores, and verdict. The runner also supplies `context`, including every distinct trace context collected across the eval's sessions. The same list is available as `result.result.traceContexts`.
+Each entry in `result.assertions` is an `AssertionResult`. `name` is the display label, `key` the identifier set by `.label(key)`, and `score` the measurement. `threshold` and `passed` describe the rule and are absent when no rule applies. When a scorer threw, `errored` is `true`, `message` carries the error, and `score` is absent. The `.eve/evals/` artifacts write the same fields.
+
An eval can create several sessions, and a long session can produce several traces, so completion exposes a list instead of a single trace id. `onEvalStart` does not include a trace because the agent session has not started yet. If the target has no tracing configured, `onSessionStart` does not fire and the completed trace list is empty.
`result.result.derived` holds the facts the runner derived from the eval's sessions, such as tool calls and `models`: the distinct ids of the models the steps started with, in session order. Each entry in `result.result.sessions` holds the same facts for one session, in first-use order, so a reporter can tell the primary session's models from those of the other sessions the eval created or attached. A delegated subagent runs in its own session, so its models appear only when the eval attaches that session with `t.target.attachSession`. Compaction and `auto` routing calls are not included.
diff --git a/e2e/fixtures/agent-basic-runtime/evals/runtime/scores.eval.ts b/e2e/fixtures/agent-basic-runtime/evals/runtime/scores.eval.ts
new file mode 100644
index 000000000..7696e0fe1
--- /dev/null
+++ b/e2e/fixtures/agent-basic-runtime/evals/runtime/scores.eval.ts
@@ -0,0 +1,23 @@
+import { defineEval } from "eve/evals";
+
+/**
+ * Runs `t.score` end to end: one score without a rule, one gated score that
+ * decides the eval.
+ */
+export default defineEval({
+ description: "Scores with and without a rule.",
+
+ async test(t) {
+ const turn = await t.send('Reply with exactly the text "score ping" and nothing else.');
+ t.succeeded();
+
+ const reply = turn.message ?? "";
+ t.score({
+ score: Math.max(0, 1 - reply.length / 400),
+ metadata: { length: reply.length },
+ }).label("reply-brevity");
+ t.score(reply.includes("score ping") ? 1 : 0)
+ .label("mentions-ping")
+ .gate();
+ },
+});
diff --git a/packages/eve/src/evals/assertions/collector.test.ts b/packages/eve/src/evals/assertions/collector.test.ts
new file mode 100644
index 000000000..7d5cb69a4
--- /dev/null
+++ b/packages/eve/src/evals/assertions/collector.test.ts
@@ -0,0 +1,162 @@
+import { describe, expect, it } from "vitest";
+
+import { AssertionCollector } from "#evals/assertions/collector.js";
+import { createEmptyDerivedFacts } from "#evals/runner/derive-run-facts.js";
+import { computeEvalVerdict } from "#evals/runner/verdict.js";
+import type { EveEvalTaskResult } from "#evals/types.js";
+
+function taskResult(): EveEvalTaskResult {
+ return {
+ derived: createEmptyDerivedFacts(),
+ events: [],
+ finalMessage: null,
+ output: null,
+ status: "completed",
+ traceContexts: [],
+ };
+}
+
+describe("AssertionCollector.recordScore", () => {
+ it("records a score with no verdict that does not affect the eval", async () => {
+ const collector = new AssertionCollector();
+ collector.recordScore(0.82).label("fidelity");
+ collector
+ .recordScore({
+ score: 0,
+ message: "nothing matched",
+ metadata: { hits: 0 },
+ })
+ .label("misses");
+
+ const assertions = await collector.finalize(taskResult());
+
+ expect(assertions).toEqual([
+ expect.objectContaining({
+ name: "score [fidelity]",
+ key: "fidelity",
+ score: 0.82,
+ severity: "soft",
+ threshold: undefined,
+ passed: undefined,
+ errored: false,
+ }),
+ expect.objectContaining({
+ key: "misses",
+ score: 0,
+ passed: undefined,
+ message: "nothing matched",
+ metadata: { hits: 0 },
+ }),
+ ]);
+ expect(computeEvalVerdict({ assertions })).toBe("passed");
+ });
+
+ it("keeps the score when a gate threshold fails", async () => {
+ const collector = new AssertionCollector();
+ collector.recordScore(Promise.resolve(0.82)).label("fidelity").gate(0.9);
+
+ const assertions = await collector.finalize(taskResult());
+
+ expect(assertions[0]).toMatchObject({
+ key: "fidelity",
+ score: 0.82,
+ severity: "gate",
+ threshold: 0.9,
+ passed: false,
+ errored: false,
+ });
+ expect(computeEvalVerdict({ assertions })).toBe("failed");
+ });
+
+ it("passes exactly at the threshold and only demotes under a soft bar", async () => {
+ const collector = new AssertionCollector();
+ collector.recordScore(0.9).label("exact").gate(0.9);
+ collector.recordScore(0.5).label("wording").atLeast(0.6);
+
+ const assertions = await collector.finalize(taskResult());
+
+ expect(
+ assertions.map(({ severity, threshold, passed }) => ({ severity, threshold, passed })),
+ ).toEqual([
+ { severity: "gate", threshold: 0.9, passed: true },
+ { severity: "soft", threshold: 0.6, passed: false },
+ ]);
+ expect(computeEvalVerdict({ assertions })).toBe("scored");
+ });
+
+ it("reports the implicit gate threshold of 1", async () => {
+ const collector = new AssertionCollector();
+ collector.recordScore(1).label("full").gate();
+ collector.recordScore(0.99).label("partial").gate();
+
+ const assertions = await collector.finalize(taskResult());
+
+ expect(assertions.map(({ threshold, passed }) => ({ threshold, passed }))).toEqual([
+ { threshold: 1, passed: true },
+ { threshold: 1, passed: false },
+ ]);
+ });
+
+ it("reports a thrown evaluator as an errored outcome with no score", async () => {
+ const collector = new AssertionCollector();
+ collector
+ .recordScore(Promise.reject(new Error("model unavailable")))
+ .label("judge")
+ .atLeast(0.5);
+ collector.recordScore(0.82).label("fidelity");
+
+ const assertions = await collector.finalize(taskResult());
+
+ expect(assertions[0]).toMatchObject({
+ key: "judge",
+ severity: "gate",
+ passed: false,
+ errored: true,
+ message: "model unavailable",
+ });
+ expect(assertions[0]?.score).toBeUndefined();
+ expect(assertions[0]?.threshold).toBeUndefined();
+ expect(assertions[1]).toMatchObject({ key: "fidelity", score: 0.82, passed: undefined });
+ expect(computeEvalVerdict({ assertions })).toBe("failed");
+ });
+
+ it("treats a score outside 0 to 1 as an errored outcome", async () => {
+ const collector = new AssertionCollector();
+ collector.recordScore(Number.NaN);
+ collector.recordScore(1200);
+
+ const [nan, latency] = await collector.finalize(taskResult());
+
+ for (const assertion of [nan, latency]) {
+ expect(assertion).toMatchObject({ errored: true, passed: false });
+ expect(assertion?.score).toBeUndefined();
+ expect(assertion?.message).toContain("from 0 to 1");
+ }
+ });
+});
+
+describe("AssertionCollector labels", () => {
+ it("exposes the label as the structured key next to the display name", async () => {
+ const collector = new AssertionCollector();
+ collector
+ .recordValue({ name: "judge.boolean", severity: "soft", score: async () => ({ score: 0.4 }) })
+ .label("citation");
+
+ const [assertion] = await collector.finalize(taskResult());
+
+ expect(assertion).toMatchObject({ name: "judge.boolean [citation]", key: "citation" });
+ });
+
+ it("leaves the key absent for unlabeled assertions", async () => {
+ const collector = new AssertionCollector();
+ collector.recordValue({
+ name: "judge.boolean",
+ severity: "soft",
+ score: async () => ({ score: 1 }),
+ });
+
+ const [assertion] = await collector.finalize(taskResult());
+
+ expect(assertion?.key).toBeUndefined();
+ });
+});
diff --git a/packages/eve/src/evals/assertions/collector.ts b/packages/eve/src/evals/assertions/collector.ts
index 6cf6c3abc..783a3cdbb 100644
--- a/packages/eve/src/evals/assertions/collector.ts
+++ b/packages/eve/src/evals/assertions/collector.ts
@@ -6,6 +6,7 @@ import type {
AssertionResult,
AssertionSeverity,
EveEvalTaskResult,
+ ScoreInput,
} from "#evals/types.js";
import type { TargetTools } from "#evals/target.js";
import type { EveEvalAssertionSubject } from "#evals/assertions/run.js";
@@ -24,12 +25,14 @@ export interface RunAssertion {
interface MutableEntry {
readonly baseName: string;
name: string;
+ key: string | undefined;
severity: AssertionSeverity;
threshold: number | undefined;
readonly kind: "deferred" | "resolved";
readonly spec?: RunAssertion;
readonly selectSubject?: (result: EveEvalTaskResult) => EveEvalAssertionSubject;
- score: number;
+ /** The measurement once produced; stays `undefined` when the scorer threw. */
+ score: number | undefined;
message?: string;
metadata?: Readonly<Record<string, unknown>>;
/** A model/value assertion that threw — a hard failure regardless of severity. */
@@ -65,60 +68,39 @@ export class AssertionCollector {
selectSubject: (result: EveEvalTaskResult) => EveEvalAssertionSubject,
severity: AssertionSeverity = "gate",
): AssertionHandle {
- const entry: MutableEntry = {
- baseName: spec.name,
- name: spec.name,
- severity,
- threshold: undefined,
- kind: "deferred",
- spec,
- selectSubject,
- score: 0,
- errored: false,
- };
- this.#entries.push(entry);
+ const entry = this.#add({ name: spec.name, severity, kind: "deferred", spec, selectSubject });
return makeHandle(entry);
}
/** Register a value/judge assertion, evaluating the captured value now. */
recordValue(input: {
readonly name: string;
+ readonly key?: string;
readonly severity: AssertionSeverity;
readonly threshold?: number;
readonly score: () => Promise<AssertionOutcome>;
}): AssertionHandle {
- const entry: MutableEntry = {
- baseName: input.name,
- name: input.name,
- severity: input.severity,
- threshold: input.threshold,
- kind: "resolved",
- score: 0,
- errored: false,
- };
- this.#entries.push(entry);
-
- const pending = settleEntry(entry, input.score);
-
- this.#pending.push(pending);
+ const entry = this.#add({ ...input, kind: "resolved" });
+ this.#pending.push(settleEntry(entry, input.score));
return makeHandle(entry);
}
+ /** Record a score. It has no verdict unless the handle chains `.gate()` or `.atLeast()`. */
+ recordScore(evaluation: ScoreInput): AssertionHandle {
+ return this.recordValue({
+ name: "score",
+ severity: "soft",
+ score: async () => toOutcome(await evaluation),
+ });
+ }
+
/** Record an already-computed assertion outcome and return whether it passed. */
recordOutcome(input: { readonly name: string; readonly outcome: AssertionOutcome }): boolean {
- const entry: MutableEntry = {
- baseName: input.name,
- name: input.name,
- severity: "gate",
- threshold: undefined,
- kind: "resolved",
- score: input.outcome.score,
- message: input.outcome.message,
- metadata: input.outcome.metadata,
- errored: false,
- };
- this.#entries.push(entry);
- return computePassed(entry.severity, entry.threshold, entry.score, entry.errored);
+ const entry = this.#add({ name: input.name, severity: "gate", kind: "resolved" });
+ entry.score = input.outcome.score;
+ entry.message = input.outcome.message;
+ entry.metadata = input.outcome.metadata;
+ return computePassed(entry) === true;
}
/** Record and await a required value assertion, returning whether it passed. */
@@ -127,18 +109,9 @@ export class AssertionCollector {
readonly threshold?: number;
readonly score: () => Promise<AssertionOutcome>;
}): Promise<boolean> {
- const entry: MutableEntry = {
- baseName: input.name,
- name: input.name,
- severity: "gate",
- threshold: input.threshold,
- kind: "resolved",
- score: 0,
- errored: false,
- };
- this.#entries.push(entry);
+ const entry = this.#add({ ...input, severity: "gate", kind: "resolved" });
await settleEntry(entry, input.score);
- return computePassed(entry.severity, entry.threshold, entry.score, entry.errored);
+ return computePassed(entry) === true;
}
/**
@@ -156,35 +129,73 @@ export class AssertionCollector {
entry.message = outcome.message;
entry.metadata = outcome.metadata;
}
- results.push({
- name: entry.name,
- score: entry.score,
- severity: entry.severity,
- threshold: entry.threshold,
- passed: computePassed(entry.severity, entry.threshold, entry.score, entry.errored),
- errored: entry.errored,
- message: entry.message,
- metadata: entry.metadata,
- });
+ results.push(toResult(entry));
}
return results;
}
+
+ #add(input: {
+ readonly name: string;
+ readonly key?: string;
+ readonly severity: AssertionSeverity;
+ readonly threshold?: number;
+ readonly kind: MutableEntry["kind"];
+ readonly spec?: RunAssertion;
+ readonly selectSubject?: MutableEntry["selectSubject"];
+ }): MutableEntry {
+ const entry: MutableEntry = {
+ baseName: input.name,
+ name: input.name,
+ key: input.key,
+ severity: input.severity,
+ threshold: input.threshold,
+ kind: input.kind,
+ spec: input.spec,
+ selectSubject: input.selectSubject,
+ score: undefined,
+ errored: false,
+ };
+ this.#entries.push(entry);
+ return entry;
+ }
+}
+
+function toResult(entry: MutableEntry): AssertionResult {
+ return {
+ name: entry.name,
+ key: entry.key,
+ score: entry.score,
+ severity: entry.severity,
+ threshold: entry.errored ? undefined : ruleThreshold(entry),
+ passed: computePassed(entry),
+ errored: entry.errored,
+ message: entry.message,
+ metadata: entry.metadata,
+ };
+}
+
+/** The effective minimum passing score: a gate defaults to 1, a soft entry has none. */
+function ruleThreshold(entry: MutableEntry): number | undefined {
+ return entry.threshold ?? (entry.severity === "gate" ? 1 : undefined);
}
/**
- * Whether an assertion meets its bar. A gate defaults to threshold 1; a soft
- * assertion with no threshold is tracked-only and always "passes". A hard
- * thrown failure never passes.
+ * Verdict of the entry's rule. A thrown scorer fails; an entry without a rule
+ * (soft, no threshold) has no verdict.
*/
-function computePassed(
- severity: AssertionSeverity,
- threshold: number | undefined,
- score: number,
- errored: boolean,
-): boolean {
- if (errored) return false;
- const min = threshold ?? (severity === "gate" ? 1 : undefined);
- return min === undefined || score >= min;
+function computePassed(entry: MutableEntry): boolean | undefined {
+ if (entry.errored) return false;
+ const threshold = ruleThreshold(entry);
+ if (threshold === undefined) return undefined;
+ return entry.score !== undefined && entry.score >= threshold;
+}
+
+function toOutcome(value: number | AssertionOutcome): AssertionOutcome {
+ const outcome = typeof value === "number" ? { score: value } : value;
+ if (typeof outcome.score !== "number" || !(outcome.score >= 0 && outcome.score <= 1)) {
+ throw new TypeError(`Scores must be numbers from 0 to 1; received ${String(outcome.score)}.`);
+ }
+ return outcome;
}
async function settleEntry(
@@ -199,7 +210,7 @@ async function settleEntry(
} catch (error: unknown) {
// A judge/value assertion that throws (e.g. a judge model error) is a
// hard failure, surfaced as a failed gate rather than aborting the run.
- entry.score = 0;
+ // A thrown scorer produces no measurement, so the score stays absent.
entry.severity = "gate";
entry.threshold = undefined;
entry.message = toErrorMessage(error);
@@ -226,6 +237,7 @@ function makeHandle(entry: MutableEntry): AssertionHandle {
},
label(label) {
entry.name = formatAssertionName(entry.baseName, label);
+ entry.key = label.trim();
return handle;
},
};
diff --git a/packages/eve/src/evals/context.ts b/packages/eve/src/evals/context.ts
index af1ab0d2c..71ab868d0 100644
--- a/packages/eve/src/evals/context.ts
+++ b/packages/eve/src/evals/context.ts
@@ -50,6 +50,7 @@ export function createEvalContext(deps: {
// Value-level assertion over an explicit value.
check: (value, assertion) => recordCheck(collector, value, assertion),
+ score: (evaluation) => collector.recordScore(evaluation),
require: (value, assertion) => requireCheck(collector, value, assertion),
skip: (reason) => {
if (reason.trim().length === 0) throw new Error("skip() requires a non-empty reason.");
diff --git a/packages/eve/src/evals/index.ts b/packages/eve/src/evals/index.ts
index 24e24cb35..2737a6d8f 100644
--- a/packages/eve/src/evals/index.ts
+++ b/packages/eve/src/evals/index.ts
@@ -66,6 +66,7 @@ export type {
JudgeQuestion,
JudgeContext,
JudgeOpts,
+ ScoreInput,
} from "#evals/types.js";
export type {
diff --git a/packages/eve/src/evals/judge.integration.test.ts b/packages/eve/src/evals/judge.integration.test.ts
index 5c05b8930..7e489beaa 100644
--- a/packages/eve/src/evals/judge.integration.test.ts
+++ b/packages/eve/src/evals/judge.integration.test.ts
@@ -123,11 +123,11 @@ describe("decision-backed judge runner", () => {
expect(outcome.assertions).toHaveLength(batch ? 2 : 1);
for (const assertion of outcome.assertions) {
expect(assertion).toMatchObject({
- score: 0,
severity: "gate",
passed: false,
errored: true,
});
+ expect(assertion.score).toBeUndefined();
expect(assertion.message).toContain(
"score criteria must contain at least two ordered levels",
);
diff --git a/packages/eve/src/evals/judge.test.ts b/packages/eve/src/evals/judge.test.ts
index 4ac3a1b84..4f3fa984c 100644
--- a/packages/eve/src/evals/judge.test.ts
+++ b/packages/eve/src/evals/judge.test.ts
@@ -97,7 +97,7 @@ describe("judge", () => {
expect(decide.mock.calls[0]?.[0].questions.outcome).not.toHaveProperty("expected");
const assertions = await collector.finalize(emptyTaskResult());
expect(assertions.map(({ name, score, passed }) => ({ name, score, passed }))).toEqual([
- { name: "judge.boolean.accurate", score: 0.2, passed: true },
+ { name: "judge.boolean.accurate", score: 0.2, passed: undefined },
{ name: "judge.score.clarity", score: 0.75, passed: false },
{ name: "judge.choice.outcome", score: 1, passed: true },
]);
diff --git a/packages/eve/src/evals/runner/artifacts.ts b/packages/eve/src/evals/runner/artifacts.ts
index 8d6a85482..87487a3b7 100644
--- a/packages/eve/src/evals/runner/artifacts.ts
+++ b/packages/eve/src/evals/runner/artifacts.ts
@@ -66,6 +66,7 @@ function buildSummaryArtifact(summary: EveEvalRunSummary): Record<string, unknow
status: result.result.status,
assertions: result.assertions.map((a) => ({
name: a.name,
+ key: a.key,
score: a.score,
severity: a.severity,
threshold: a.threshold,
diff --git a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts
index 4b298431b..72cea442e 100644
--- a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts
+++ b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts
@@ -29,4 +29,16 @@ describe("assertion failure diagnostics", () => {
"…",
]);
});
+
+ it("omits the score comparison when the scorer threw", () => {
+ const assertion: AssertionResult = {
+ message: "model unavailable",
+ name: "judge.boolean",
+ passed: false,
+ errored: true,
+ severity: "gate",
+ };
+
+ expect(formatAssertionFailureHeadline(assertion)).toBe("judge.boolean: model unavailable");
+ });
});
diff --git a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.ts b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.ts
index 12135be0c..f033d9d06 100644
--- a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.ts
+++ b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.ts
@@ -5,11 +5,10 @@ const MAX_DETAIL_LINE_LENGTH = 240;
const MAX_DETAIL_LINES = 4;
export function formatAssertionFailureHeadline(assertion: AssertionResult): string {
- const threshold = assertion.threshold ?? (assertion.severity === "gate" ? 1 : undefined);
const comparison =
- threshold === undefined
+ assertion.threshold === undefined || assertion.score === undefined
? ""
- : ` (${formatPercent(assertion.score)} < ${formatPercent(threshold)})`;
+ : ` (${formatPercent(assertion.score)} < ${formatPercent(assertion.threshold)})`;
const [headline] = assertion.message?.split("\n") ?? [];
const detail =
headline === undefined ? "" : `: ${truncateDiagnostic(headline, MAX_DETAIL_LINE_LENGTH)}`;
diff --git a/packages/eve/src/evals/runner/reporters/assertion-scores.ts b/packages/eve/src/evals/runner/reporters/assertion-scores.ts
index 85ab4172a..449745d80 100644
--- a/packages/eve/src/evals/runner/reporters/assertion-scores.ts
+++ b/packages/eve/src/evals/runner/reporters/assertion-scores.ts
@@ -1,8 +1,19 @@
import type { AssertionResult } from "#evals/types.js";
+/**
+ * The name an assertion is exported under. Gates carry a `gate:` prefix so
+ * they do not share a column with a soft assertion of the same name, and so
+ * experiments diff gate regressions the same way they diff soft scores.
+ * Sinks with stricter label rules normalize this name themselves.
+ */
+export function exportedAssertionName(assertion: AssertionResult): string {
+ return assertion.severity === "gate" ? `gate:${assertion.name}` : assertion.name;
+}
+
/** Assertions reported under one name, scored together. */
export interface AssertionScoreGroup {
- readonly score: number;
+ /** Lowest completed score in the group; absent when every member errored. */
+ readonly score: number | undefined;
readonly assertions: readonly AssertionResult[];
}
@@ -21,7 +32,7 @@ export function groupAssertionScores(
const name = getName(assertion);
const group = groups.get(name);
groups.set(name, {
- score: Math.min(group?.score ?? assertion.score, assertion.score),
+ score: minScore(group?.score, assertion.score),
assertions: [...(group?.assertions ?? []), assertion],
});
}
@@ -30,5 +41,11 @@ export function groupAssertionScores(
/** Per-assertion scores retained in row metadata once repeated assertions share one score. */
export function composeAssertionScoreMetadata(assertions: readonly AssertionResult[]) {
- return assertions.map(({ name, severity, score }) => ({ name, severity, score }));
+ return assertions.map(({ key, name, severity, score }) => ({ key, name, severity, score }));
+}
+
+function minScore(current: number | undefined, next: number | undefined): number | undefined {
+ if (current === undefined) return next;
+ if (next === undefined) return current;
+ return Math.min(current, next);
}
diff --git a/packages/eve/src/evals/runner/reporters/braintrust.ts b/packages/eve/src/evals/runner/reporters/braintrust.ts
index 646640f0d..0263e62ab 100644
--- a/packages/eve/src/evals/runner/reporters/braintrust.ts
+++ b/packages/eve/src/evals/runner/reporters/braintrust.ts
@@ -1,6 +1,7 @@
import type { EveEval, EveEvalResult, EveEvalRunSummary, EveEvalTarget } from "#evals/types.js";
import {
composeAssertionScoreMetadata,
+ exportedAssertionName,
groupAssertionScores,
} from "#evals/runner/reporters/assertion-scores.js";
import { resolveLocalGitMetadata } from "#evals/runner/resolve-git-metadata.js";
@@ -133,19 +134,14 @@ class BraintrustReporter implements EvalReporter {
if (!this.#experiment) return;
const evaluation = this.#evaluations.get(result.id);
- // Soft assertions log under their own name; gate assertions log as binary
- // scores under a `gate:` prefix so experiments diff gate regressions the
- // same way they diff soft-score regressions.
const scores: Record<string, number> = {};
- const groups = groupAssertionScores(result.assertions, (assertion) =>
- assertion.severity === "gate" ? `gate:${assertion.name}` : assertion.name,
- );
+ const groups = groupAssertionScores(result.assertions, exportedAssertionName);
for (const [name, group] of groups) {
- scores[name] = group.score;
+ if (group.score !== undefined) scores[name] = group.score;
}
const failedAssertions = result.assertions
- .filter((assertion) => !assertion.passed)
+ .filter((assertion) => assertion.passed === false)
.map((assertion) => ({ ...assertion }));
const metadata: Record<string, unknown> = {
diff --git a/packages/eve/src/evals/runner/reporters/console.test.ts b/packages/eve/src/evals/runner/reporters/console.test.ts
index 2b7cdf504..24fdee0c5 100644
--- a/packages/eve/src/evals/runner/reporters/console.test.ts
+++ b/packages/eve/src/evals/runner/reporters/console.test.ts
@@ -56,6 +56,73 @@ describe("Console", () => {
expect(lines).toContain(' answer: {"type":"boolean","probability":0}');
});
+ it("warns when repeated soft assertions merge different scores and leaves gates quiet", () => {
+ const lines: string[] = [];
+ const reporter = Console({ color: false, log: (line) => lines.push(line) });
+
+ reporter.onEvalComplete({
+ assertions: [
+ { name: "judge.boolean", passed: true, errored: false, score: 0.9, severity: "soft" },
+ { name: "judge.boolean", passed: true, errored: false, score: 0.4, severity: "soft" },
+ { name: "judge.boolean", passed: true, errored: false, score: 1, severity: "gate" },
+ { name: "succeeded", passed: true, errored: false, score: 1, severity: "gate" },
+ {
+ name: "succeeded",
+ passed: false,
+ errored: false,
+ score: 0,
+ severity: "gate",
+ threshold: 1,
+ },
+ ],
+ completedAt: "2026-01-01T00:00:01.000Z",
+ id: "quality",
+ result: {
+ derived: createEmptyDerivedFacts(),
+ events: [],
+ finalMessage: "ok",
+ output: "ok",
+ status: "completed",
+ traceContexts: [],
+ },
+ startedAt: "2026-01-01T00:00:00.000Z",
+ verdict: "failed",
+ });
+
+ expect(lines).toEqual([
+ "✗ quality gates 2/3 judge.boolean: 90% judge.boolean: 40%",
+ " ✗ succeeded (0% < 100%)",
+ " ⚠ judge.boolean recorded 2 times with different scores; Braintrust and Datadog keep only the lowest. Add .label() to tell them apart.",
+ ]);
+ });
+
+ it("aggregates repeated soft scores as one value per eval in the run summary", () => {
+ const lines: string[] = [];
+ const reporter = Console({ color: false, log: (line) => lines.push(line) });
+ const result: EveEvalResult = {
+ assertions: [
+ { name: "judge.boolean", passed: true, errored: false, score: 0.9, severity: "soft" },
+ { name: "judge.boolean", passed: true, errored: false, score: 0.4, severity: "soft" },
+ ],
+ completedAt: "2026-01-01T00:00:01.000Z",
+ id: "quality",
+ result: {
+ derived: createEmptyDerivedFacts(),
+ events: [],
+ finalMessage: "ok",
+ output: "ok",
+ status: "completed",
+ traceContexts: [],
+ },
+ startedAt: "2026-01-01T00:00:00.000Z",
+ verdict: "passed",
+ };
+
+ reporter.onRunComplete(summary(result));
+
+ expect(lines).toContain(" judge.boolean: 40% (1 evals)");
+ });
+
it("does not print diagnostics for passing assertions", () => {
const lines: string[] = [];
const reporter = Console({ color: false, log: (line) => lines.push(line) });
diff --git a/packages/eve/src/evals/runner/reporters/console.ts b/packages/eve/src/evals/runner/reporters/console.ts
index 0b969fd3f..0bb4a93ad 100644
--- a/packages/eve/src/evals/runner/reporters/console.ts
+++ b/packages/eve/src/evals/runner/reporters/console.ts
@@ -11,6 +11,10 @@ import {
formatAssertionFailureDetailLines,
formatAssertionFailureHeadline,
} from "#evals/runner/reporters/assertion-diagnostics.js";
+import {
+ exportedAssertionName,
+ groupAssertionScores,
+} from "#evals/runner/reporters/assertion-scores.js";
import type { EvalReporter } from "#evals/runner/reporters/types.js";
/**
@@ -64,7 +68,9 @@ class ConsoleReporter implements EvalReporter {
? this.#formatGateCount(gates.filter((gate) => gate.passed).length, gates.length)
: "";
const scoreText = softs
- .map((assertion) => this.#formatScore(assertion.name, assertion.score))
+ .flatMap((assertion) =>
+ assertion.score === undefined ? [] : [this.#formatScore(assertion.name, assertion.score)],
+ )
.join(" ");
const line = [icon, this.#colors.dim(result.id), gateText, scoreText]
@@ -73,13 +79,26 @@ class ConsoleReporter implements EvalReporter {
this.#log(line);
for (const assertion of assertions) {
- if (assertion.passed) continue;
+ if (assertion.passed !== false) continue;
this.#log(` ${this.#colors.red(`✗ ${formatAssertionFailureHeadline(assertion)}`)}`);
for (const detailLine of formatAssertionFailureDetailLines(assertion)) {
this.#log(` ${this.#colors.red(detailLine)}`);
}
}
+ // Braintrust and Datadog keep one score per exported name, the lowest. For
+ // gates that aggregate is exact and the failure line above already names
+ // the failing member, so warn only for soft groups whose scores differ.
+ for (const group of groupAssertionScores(softs, exportedAssertionName).values()) {
+ if (group.assertions.every((assertion) => assertion.score === group.score)) continue;
+ const name = group.assertions[0]!.name;
+ this.#log(
+ ` ${this.#colors.yellow(
+ `⚠ ${name} recorded ${group.assertions.length} times with different scores; Braintrust and Datadog keep only the lowest. Add .label() to tell them apart.`,
+ )}`,
+ );
+ }
+
if (error) {
this.#log(` ${this.#colors.red(error)}`);
}
@@ -186,15 +205,18 @@ class ConsoleReporter implements EvalReporter {
): { name: string; avg: number; count: number }[] {
const totals = new Map<string, { sum: number; count: number }>();
+ // One value per name per eval, the group minimum, matching what the
+ // export reporters record; `count` is the number of evals.
for (const result of results) {
- for (const assertion of result.assertions) {
- if (assertion.severity !== "soft") continue;
- const entry = totals.get(assertion.name);
+ const softs = result.assertions.filter((assertion) => assertion.severity === "soft");
+ for (const [name, group] of groupAssertionScores(softs, (assertion) => assertion.name)) {
+ if (group.score === undefined) continue;
+ const entry = totals.get(name);
if (entry) {
- entry.sum += assertion.score;
+ entry.sum += group.score;
entry.count += 1;
} else {
- totals.set(assertion.name, { sum: assertion.score, count: 1 });
+ totals.set(name, { sum: group.score, count: 1 });
}
}
}
diff --git a/packages/eve/src/evals/runner/reporters/datadog.test.ts b/packages/eve/src/evals/runner/reporters/datadog.test.ts
index d9d3f4bf9..b7f5afce1 100644
--- a/packages/eve/src/evals/runner/reporters/datadog.test.ts
+++ b/packages/eve/src/evals/runner/reporters/datadog.test.ts
@@ -484,6 +484,23 @@ describe("Datadog", () => {
]);
});
+ it("omits assertion_passed when no member of a group has a verdict", async () => {
+ const { config, experiment } = makeConfig({ recordInputs: false });
+ const reporter = Datadog(config);
+ const result = makeEvalResult({
+ assertions: [{ name: "judge.boolean", score: 0.9, severity: "soft", errored: false }],
+ });
+
+ await reporter.onRunStart([makeEval()], makeTarget());
+ await reporter.onEvalComplete(result);
+
+ expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toContainEqual({
+ label: "judge_boolean",
+ value: 0.9,
+ tags: { assertion_severity: "soft", assertion_count: "1" },
+ });
+ });
+
it("records inputs, outputs, expected outputs, and runtime links in a shared dataset by default", async () => {
const { client, config, datasets, experiment, lines } = makeConfig({ experimentName: "run-1" });
const reporter = Datadog(config);
diff --git a/packages/eve/src/evals/runner/reporters/datadog.ts b/packages/eve/src/evals/runner/reporters/datadog.ts
index 10c024cca..201e802f4 100644
--- a/packages/eve/src/evals/runner/reporters/datadog.ts
+++ b/packages/eve/src/evals/runner/reporters/datadog.ts
@@ -3,6 +3,7 @@ import { resolveRuntimeTraceLinks } from "#evals/runner/reporters/datadog-runtim
import type { EvalReporter } from "#evals/runner/reporters/types.js";
import {
composeAssertionScoreMetadata,
+ exportedAssertionName,
groupAssertionScores,
} from "#evals/runner/reporters/assertion-scores.js";
import {
@@ -483,7 +484,7 @@ function resolveResultMetadata(
}
if (recordAssertionDetails) {
const failedAssertions = result.assertions
- .filter((assertion) => !assertion.passed)
+ .filter((assertion) => assertion.passed === false)
.map((assertion) => ({ name: assertion.name, message: assertion.message }));
if (failedAssertions.length > 0) {
metadata.eveFailedAssertions = failedAssertions;
@@ -500,16 +501,22 @@ function resolveEvaluationMetrics(result: EveEvalResult): DatadogEvaluationMetri
const metrics: DatadogEvaluationMetricInput[] = [];
const usedLabels = new Set<string>(BUILT_IN_METRIC_LABELS);
const groups = groupAssertionScores(result.assertions, (assertion) =>
- toDatadogMetricLabel(assertion.severity === "gate" ? `gate_${assertion.name}` : assertion.name),
+ toDatadogMetricLabel(exportedAssertionName(assertion)),
);
for (const [label, group] of groups) {
+ if (group.score === undefined) continue;
+ const verdicts = group.assertions.flatMap((assertion) =>
+ assertion.passed === undefined ? [] : [assertion.passed],
+ );
metrics.push({
label: reserveDatadogMetricLabel(label, usedLabels),
value: group.score,
tags: {
assertion_severity: group.assertions[0]?.severity ?? "soft",
- assertion_passed: String(group.assertions.every((assertion) => assertion.passed)),
+ ...(verdicts.length > 0 && {
+ assertion_passed: String(verdicts.every((passed) => passed)),
+ }),
assertion_count: String(group.assertions.length),
},
});
diff --git a/packages/eve/src/evals/runner/reporters/junit.test.ts b/packages/eve/src/evals/runner/reporters/junit.test.ts
index 89a08b15a..bb43c8414 100644
--- a/packages/eve/src/evals/runner/reporters/junit.test.ts
+++ b/packages/eve/src/evals/runner/reporters/junit.test.ts
@@ -73,6 +73,7 @@ function makeSummary(): EveEvalRunSummary {
name: "contains",
score: 0,
severity: "gate",
+ threshold: 1,
passed: false,
errored: false,
message: "expected hello",
diff --git a/packages/eve/src/evals/runner/reporters/junit.ts b/packages/eve/src/evals/runner/reporters/junit.ts
index 680128df1..31631a7a6 100644
--- a/packages/eve/src/evals/runner/reporters/junit.ts
+++ b/packages/eve/src/evals/runner/reporters/junit.ts
@@ -91,7 +91,7 @@ function buildFailureDetail(result: EveEvalResult): Record<string, unknown> {
function failureMessage(result: EveEvalResult): string {
if (result.error !== undefined) return result.error;
- const failed = result.assertions.filter((assertion) => !assertion.passed);
+ const failed = result.assertions.filter((assertion) => assertion.passed === false);
if (failed.length > 0) {
return truncateDiagnostic(failed.map(formatAssertionFailureHeadline).join("; "), 1_000);
}
diff --git a/packages/eve/src/evals/runner/verdict.test.ts b/packages/eve/src/evals/runner/verdict.test.ts
index 84f13b17f..e39f888fa 100644
--- a/packages/eve/src/evals/runner/verdict.test.ts
+++ b/packages/eve/src/evals/runner/verdict.test.ts
@@ -69,14 +69,24 @@ describe("computeEvalVerdict", () => {
).toBe("failed");
});
- it("passes when a tracked soft assertion has no threshold", () => {
+ it("passes when a tracked soft assertion has no verdict", () => {
expect(
computeEvalVerdict({
- assertions: [assertion({ severity: "soft", score: 0.1, passed: true, errored: false })],
+ assertions: [assertion({ severity: "soft", score: 0.1, passed: undefined })],
}),
).toBe("passed");
});
+ it("fails when a scorer errored without producing a score", () => {
+ expect(
+ computeEvalVerdict({
+ assertions: [
+ assertion({ severity: "gate", score: undefined, passed: false, errored: true }),
+ ],
+ }),
+ ).toBe("failed");
+ });
+
it("returns skipped when an eval intentionally skips without another failure", () => {
expect(
computeEvalVerdict({
diff --git a/packages/eve/src/evals/runner/verdict.ts b/packages/eve/src/evals/runner/verdict.ts
index 27ed56539..d1ef48b3b 100644
--- a/packages/eve/src/evals/runner/verdict.ts
+++ b/packages/eve/src/evals/runner/verdict.ts
@@ -5,8 +5,8 @@ import type { AssertionResult, EveEvalVerdict } from "#evals/types.js";
* assertions. An execution error or a failed gate assertion is a hard
* failure; a below-threshold soft assertion at worst demotes the eval to
* `"scored"` (which `eve eval --strict` later promotes to a failing exit
- * code). Soft assertions without a threshold are tracked-only and never
- * demote the verdict.
+ * code). Entries without an acceptance rule carry no verdict and never
+ * demote the eval.
*/
export function computeEvalVerdict(input: {
readonly error?: string;
@@ -17,7 +17,7 @@ export function computeEvalVerdict(input: {
let demoted = false;
for (const assertion of input.assertions) {
- if (assertion.passed) continue;
+ if (assertion.passed !== false) continue;
if (assertion.severity === "gate") return "failed";
demoted = true;
}
diff --git a/packages/eve/src/evals/types.ts b/packages/eve/src/evals/types.ts
index b0ebd8afb..d2e570617 100644
--- a/packages/eve/src/evals/types.ts
+++ b/packages/eve/src/evals/types.ts
@@ -202,6 +202,9 @@ export interface AssertionEvaluation {
readonly metadata?: Readonly<Record<string, unknown>>;
}
+/** A score from 0 to 1, or an evaluation carrying one, accepted by `t.score`; may be a promise. */
+export type ScoreInput = number | AssertionEvaluation | Promise<number | AssertionEvaluation>;
+
/**
* A value-level assertion produced by the builders in `eve/evals/expect`
* (e.g. `includes`, `equals`, `similarity`) and applied to an explicit value
@@ -234,21 +237,29 @@ export interface AssertionHandle {
gate(threshold?: number): this;
soft(threshold?: number): this;
atLeast(threshold: number): this;
- /** Adds a stable human-readable label to this recorded assertion. */
+ /** Sets this entry's stable `key` and appends the label to its display name. */
label(label: string): this;
}
/**
- * The recorded outcome of one assertion, consumed by the verdict, reporters,
- * and artifacts. A boolean assertion has `score` 0 or 1.
+ * The recorded outcome of one assertion or `t.score`, consumed by the verdict,
+ * reporters, and artifacts. `score` is the measurement and a rule does not
+ * change it; `threshold` and `passed` describe the rule and are absent when
+ * none applies. A boolean assertion has `score` 0 or 1.
*/
export interface AssertionResult {
+ /** Display name, e.g. `judge.boolean [citation]`. */
readonly name: string;
- readonly score: number;
+ /** Stable identifier set by `.label(key)`; absent for unlabeled assertions. */
+ readonly key?: string;
+ /** The measurement; absent when the scorer threw. */
+ readonly score?: number;
readonly severity: AssertionSeverity;
+ /** Effective minimum passing score (a gate defaults to 1); absent when no rule applies. */
readonly threshold?: number;
- readonly passed: boolean;
- /** Whether the assertion failed because its scorer threw instead of producing a score. */
+ /** Verdict of the acceptance rule; absent when no rule applies. */
+ readonly passed?: boolean;
+ /** Whether the scorer threw; an errored entry is always a failed outcome. */
readonly errored: boolean;
/** Human-readable failure detail, shown in console output and artifacts. */
readonly message?: string;
@@ -500,6 +511,12 @@ export interface EveEvalContext<TContext = unknown> extends EveEvalAssertions {
/** Apply a value-level assertion (from `eve/evals/expect`) to a value. */
check(value: unknown, assertion: Assertion): AssertionHandle;
+ /**
+ * Record a score from 0 to 1. Chain `.label(key)` to name it, `.gate(0.9)`
+ * to fail the eval below 0.9, or `.atLeast(0.9)` to mark it `scored`.
+ * Without a rule the score has no verdict.
+ */
+ score(evaluation: ScoreInput): AssertionHandle;
/** Record an immediate gate and abort dependent control flow when it fails. */
require<T>(value: T, assertion: Assertion): Promise<T>;
/** Mark this eval as intentionally skipped and stop executing its test body. */
diff --git a/packages/eve/test/scenarios/eval-command-environment.scenario.test.ts b/packages/eve/test/scenarios/eval-command-environment.scenario.test.ts
index 960e4b395..cbe11e87f 100644
--- a/packages/eve/test/scenarios/eval-command-environment.scenario.test.ts
+++ b/packages/eve/test/scenarios/eval-command-environment.scenario.test.ts
@@ -519,7 +519,14 @@ function makeEvalResult(id: string) {
id,
assertions: [
failed
- ? { name: "check", score: 0, severity: "gate" as const, passed: false, message: "nope" }
+ ? {
+ name: "check",
+ score: 0,
+ severity: "gate" as const,
+ threshold: 1,
+ passed: false,
+ message: "nope",
+ }
: { name: "check", score: 1, severity: "gate" as const, passed: true },
],
result: {
diff --git a/research/eval-scores-and-verdicts.md b/research/eval-scores-and-verdicts.md
new file mode 100644
index 000000000..17d666a9a
--- /dev/null
+++ b/research/eval-scores-and-verdicts.md
@@ -0,0 +1,61 @@
+---
+issue: https://github.com/vercel/eve/pull/4563
+status: implemented
+last_updated: "2026-10-09"
+---
+
+# Separate evaluation scores from pass/fail decisions in eve
+
+Expose each evaluator's score and optional pass/fail verdict separately, so
+consumers can persist both, including when a threshold fails.
+
+## Motivation
+
+An evaluation store needs the original measurement to compare runs, while CI
+needs an acceptance decision. A score of `0.82` must remain `0.82` when a `0.9`
+threshold makes the test fail.
+
+Before this change, [`AssertionResult`](../packages/eve/src/evals/types.ts)
+reported `passed: true` for assertions without a threshold, wrote `score: 0`
+when a scorer threw, and had no identifier other than the display `name`. A
+score is a measurement; an assertion is a rule applied to it.
+
+## Authoring contract
+
+`t.score(evaluation)` records a score and returns the ordinary
+`AssertionHandle`; `.label(key)` names it, as for every other assertion.
+`evaluation` is a number or `{ score, message?, metadata? }`, or a promise of
+either. Scores are numbers from 0 to 1; boolean assertions keep scoring 0 or 1.
+
+```ts
+const grade = await evaluator.evaluate({ input, output, reference });
+t.score(grade).label("faithfulness"); // no verdict, never fails
+
+t.score(grade).label("faithfulness").gate(0.9); // fails the eval below 0.9
+t.score(grade).label("faithfulness").atLeast(0.9); // marks it `scored`, fatal under --strict
+```
+
+A rule does not change the score or rerun the evaluator. Existing assertions
+are unchanged; `.label(key)` sets `key` on any of them.
+
+## Persistence contract
+
+Every finalized `AssertionResult`, in reporters' `onEvalComplete` and in the
+`.eve/evals/` artifacts, carries:
+
+| field | meaning |
+| ----------- | ----------------------------------------------------------------------------------- |
+| `key` | stable identifier set by `.label(key)`; else absent |
+| `score` | the measurement; absent when the scorer threw |
+| `threshold` | effective minimum passing score (a gate defaults to 1); absent when no rule applies |
+| `passed` | verdict of the rule; absent when no rule applies |
+| `errored` | the scorer threw; always a failed outcome, with the error in `message` |
+
+For `0.82` gated at `0.9`, consumers read `score: 0.82`, `threshold: 0.9`,
+`passed: false`. A failed threshold does not prevent persistence. A scorer
+that threw has no score, and the other measurements in the same eval still
+reach reporters. Reporters that need a number (Braintrust, Datadog) skip
+errored entries.
+
+Related precedent: LangSmith separates [feedback recording](https://docs.langchain.com/langsmith/feedback-data-format)
+from optional [test expectations](https://docs.langchain.com/langsmith/pytest#expectations).