The Open Framework for Building Agents
feat(eve): record eval scores separately from verdicts and warn on merged duplicates (#4565)
Philipp Comansbc2be45f0226
diff --git a/.changeset/eval-scores-and-verdicts.md b/.changeset/eval-scores-and-verdicts.mdnew file mode 100644index 000000000..481e4f8fc--- /dev/null+++ b/.changeset/eval-scores-and-verdicts.md@@ -0,0 +1,5 @@+---+"eve": minor+---++Adds `t.score(evaluation)` for recording a score from 0 to 1, named with `.label(key)` and gated with `.gate()` or `.atLeast()`. `AssertionResult` gains `key`, and `score`, `threshold`, and `passed` are now optional: `score` is absent when the scorer threw, and `threshold` and `passed` are absent when no rule applies, so custom reporters should treat `passed === false` as the failure signal. `eve eval` warns when soft assertions in one eval share a name with different scores.diff --git a/docs/evals/assertions.mdx b/docs/evals/assertions.mdxindex a8bf20118..dc08d4b86 100644--- a/docs/evals/assertions.mdx+++ b/docs/evals/assertions.mdx@@ -3,7 +3,7 @@ title: "Assertions"description: "Scoped methods, value assertions, the matcher mini-language, and gate vs soft severity."----Assertions are how an eval grades what its `test(t)` function produced. Each one records a result and returns a chainable handle. The runner reads every recorded result to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are two deterministic surfaces: scoped methods and `t.check` for grading a specific value. For model-graded assertions, see [Judge](./judge).+Assertions are how an eval grades what its `test(t)` function produced. Each one records a result and returns a chainable handle. The runner reads every recorded result to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are three deterministic surfaces: scoped methods, `t.check` for grading a specific value, and `t.score` for recording a measurement you computed yourself. For model-graded assertions, see [Judge](./judge).## Scoped assertions@@ -83,6 +83,53 @@ For example, a failed labeled equality check prints:Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground. For nuanced grading, reach for the [judge](./judge).+## Scores with `t.score`++`t.score(evaluation)` records a score from 0 to 1. The evaluation is the number or `{ score, message?, metadata? }`, or a promise of either. Chain `.label(key)` to name it; reporters and artifacts expose the label as `key`. Without a label the score reports as `score`, and several unlabeled scores share that name.++```ts+const reply = turn.message ?? "";+t.score({+ score: Math.max(0, 1 - reply.length / 400),+ metadata: { length: reply.length },+}).label("reply-brevity");+t.score(gradeFaithfulness(reply, reference)).label("faithfulness"); // async+```++On its own a score has no verdict and does not affect the eval's outcome. Chain a rule when it should:++```ts+t.score(gradeFaithfulness(reply, reference)).label("faithfulness").gate(0.9); // fails below 0.9+t.score(brevity).label("reply-brevity").atLeast(0.5); // marks the eval `scored`, fatal under --strict+```++The score is reported as measured. A `0.82` gated at `0.9` is reported as `score: 0.82`, `threshold: 0.9`, `passed: false`. An evaluator that throws or returns a score outside 0 to 1 records an errored outcome with the error message and no score, and fails the eval.++Every [reporter](./reporters#custom-reporters) receives the entries in `onEvalComplete` and can read them by `key`:++```ts+onEvalComplete(result) {+ for (const { key, score, threshold, passed } of result.assertions) {+ if (key === undefined) continue;+ store.write({ evalId: result.id, key, score, threshold, passed });+ }+}+```++The same entries land in the [run artifacts](./running#artifacts) under `.eve/evals/`, in `summary.json` and one line per eval in `results.jsonl`:++```json+{+ "name": "score [reply-brevity]",+ "key": "reply-brevity",+ "score": 0.84,+ "severity": "soft",+ "errored": false+}+```++`threshold` and `passed` are present only when a rule was chained.+## The matcher mini-language`t.calledTool` and `t.calledSubagent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ callId, childSessionId, remoteUrl, output, status, count }`. Calls match `status: "completed"` by default. Tool lifecycle checks also accept `"pending"`, `"failed"`, and `"rejected"`. Subagent checks accept `"working"`, `"completed"`, `"failed"`, and `"cancelled"`. A numeric `count` requires an exact number of calls matching every supplied constraint. Use a predicate for ranges or other custom count requirements; it receives the observed number of matching calls.@@ -164,9 +211,9 @@ const request = session.requireInputRequest({ toolName: "guarded" });Every assertion returns a chainable handle. Severity rides on the assertion, so there is no separate thresholds map to keep in sync.- `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero.-- `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold, it is tracked-only and never fails.+- `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold it records no verdict and never fails.- `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).-- `.label(name)` adds a stable name when an eval records several assertions from the same family.+- `.label(key)` adds a stable name when an eval records several assertions from the same family. Reporters and artifacts expose it as `key`. Braintrust and Datadog keep one score per name, so `eve eval` warns when soft assertions in one eval share a name with different scores.The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge(...)` assertion are soft. Annotate only when you deviate:diff --git a/docs/evals/overview.mdx b/docs/evals/overview.mdxindex e3a8227d8..d36bff92e 100644--- a/docs/evals/overview.mdx+++ b/docs/evals/overview.mdx@@ -154,6 +154,7 @@ Each surface matches a genuinely different kind of judgment:- **Scoped methods** read the final whole run on `t`, snapshot one independent session when invoked there, or inspect one immutable `EveEvalTurn`. See [Assertions](./assertions).- **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect`, such as `t.check(turn.message, includes("sunny"))`. Grade `turn.message`, an intermediate draft, parsed JSON, or anything else. See [Assertions](./assertions).+- **`t.score(evaluation)`** records a score, such as a length ratio or an evaluator's result. Chain `.label(key)` to name it, and `.gate(threshold)` or `.atLeast(threshold)` to let it decide the eval. See [Assertions](./assertions#scores-with-tscore).- **`t.judge(...)`** is the LLM-as-judge surface, like `t.judge("cites a source")`. It grades the most recently settled turn’s assistant message by default; pass `{ on: turn.session.transcript }` to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See [Judge](./judge).## Gate vs soft@@ -161,7 +162,7 @@ Each surface matches a genuinely different kind of judgment:Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.- **Gates** are hard. A failed gate marks the eval `failed` and `eve eval` exits non-zero. Run-level methods, `includes`, `equals`, and `matches` are gates by default.-- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity` and every `t.judge(...)` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.+- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity`, every `t.judge(...)` assertion, and every `t.score(...)` are soft by default. A soft assertion with no threshold records no verdict and never fails.Override per assertion: `.gate(threshold?)` promotes to a hard gate, `.soft(threshold?)` demotes to tracked, and `.atLeast(threshold)` is a soft assertion with a bar.diff --git a/docs/evals/reporters.mdx b/docs/evals/reporters.mdxindex 4eed6be64..7e068f7f3 100644--- a/docs/evals/reporters.mdx+++ b/docs/evals/reporters.mdx@@ -38,7 +38,7 @@ export default defineEval({});```-The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Assertions that share a name, such as several unlabeled judges, log one score: the lowest. Use `.label(name)` to group or separate them. Each assertion's score is stored in the Braintrust metadata under `eveAssertionScores` and failed assertion details under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.+The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Assertions that share a name, such as several unlabeled judges, log one score: the lowest. The console warns under the eval when merged soft scores differ; use `.label(name)` to group or separate them. Errored assertions log no score. Each assertion's score is stored in the Braintrust metadata under `eveAssertionScores` and failed assertion details under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.A reporter instance observes the evals that reference it. Share one instance across several evals (the config, a `shared.ts` export, or every entry of a dataset array) and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.@@ -97,6 +97,8 @@ interface EvalReporter {- `onSessionStart` fires once for each session after eve receives its first trace context. It includes `sessionId`, `primary`, and `traceContext` with `traceId`, `spanId`, and `traceFlags`.- `onEvalComplete` fires with the checks, scores, and verdict. The runner also supplies `context`, including every distinct trace context collected across the eval's sessions. The same list is available as `result.result.traceContexts`.+Each entry in `result.assertions` is an `AssertionResult`. `name` is the display label, `key` the identifier set by `.label(key)`, and `score` the measurement. `threshold` and `passed` describe the rule and are absent when no rule applies. When a scorer threw, `errored` is `true`, `message` carries the error, and `score` is absent. The `.eve/evals/` artifacts write the same fields.+An eval can create several sessions, and a long session can produce several traces, so completion exposes a list instead of a single trace id. `onEvalStart` does not include a trace because the agent session has not started yet. If the target has no tracing configured, `onSessionStart` does not fire and the completed trace list is empty.`result.result.derived` holds the facts the runner derived from the eval's sessions, such as tool calls and `models`: the distinct ids of the models the steps started with, in session order. Each entry in `result.result.sessions` holds the same facts for one session, in first-use order, so a reporter can tell the primary session's models from those of the other sessions the eval created or attached. A delegated subagent runs in its own session, so its models appear only when the eval attaches that session with `t.target.attachSession`. Compaction and `auto` routing calls are not included.diff --git a/e2e/fixtures/agent-basic-runtime/evals/runtime/scores.eval.ts b/e2e/fixtures/agent-basic-runtime/evals/runtime/scores.eval.tsnew file mode 100644index 000000000..7696e0fe1--- /dev/null+++ b/e2e/fixtures/agent-basic-runtime/evals/runtime/scores.eval.ts@@ -0,0 +1,23 @@+import { defineEval } from "eve/evals";++/**+ * Runs `t.score` end to end: one score without a rule, one gated score that+ * decides the eval.+ */+export default defineEval({+ description: "Scores with and without a rule.",++ async test(t) {+ const turn = await t.send('Reply with exactly the text "score ping" and nothing else.');+ t.succeeded();++ const reply = turn.message ?? "";+ t.score({+ score: Math.max(0, 1 - reply.length / 400),+ metadata: { length: reply.length },+ }).label("reply-brevity");+ t.score(reply.includes("score ping") ? 1 : 0)+ .label("mentions-ping")+ .gate();+ },+});diff --git a/packages/eve/src/evals/assertions/collector.test.ts b/packages/eve/src/evals/assertions/collector.test.tsnew file mode 100644index 000000000..7d5cb69a4--- /dev/null+++ b/packages/eve/src/evals/assertions/collector.test.ts@@ -0,0 +1,162 @@+import { describe, expect, it } from "vitest";++import { AssertionCollector } from "#evals/assertions/collector.js";+import { createEmptyDerivedFacts } from "#evals/runner/derive-run-facts.js";+import { computeEvalVerdict } from "#evals/runner/verdict.js";+import type { EveEvalTaskResult } from "#evals/types.js";++function taskResult(): EveEvalTaskResult {+ return {+ derived: createEmptyDerivedFacts(),+ events: [],+ finalMessage: null,+ output: null,+ status: "completed",+ traceContexts: [],+ };+}++describe("AssertionCollector.recordScore", () => {+ it("records a score with no verdict that does not affect the eval", async () => {+ const collector = new AssertionCollector();+ collector.recordScore(0.82).label("fidelity");+ collector+ .recordScore({+ score: 0,+ message: "nothing matched",+ metadata: { hits: 0 },+ })+ .label("misses");++ const assertions = await collector.finalize(taskResult());++ expect(assertions).toEqual([+ expect.objectContaining({+ name: "score [fidelity]",+ key: "fidelity",+ score: 0.82,+ severity: "soft",+ threshold: undefined,+ passed: undefined,+ errored: false,+ }),+ expect.objectContaining({+ key: "misses",+ score: 0,+ passed: undefined,+ message: "nothing matched",+ metadata: { hits: 0 },+ }),+ ]);+ expect(computeEvalVerdict({ assertions })).toBe("passed");+ });++ it("keeps the score when a gate threshold fails", async () => {+ const collector = new AssertionCollector();+ collector.recordScore(Promise.resolve(0.82)).label("fidelity").gate(0.9);++ const assertions = await collector.finalize(taskResult());++ expect(assertions[0]).toMatchObject({+ key: "fidelity",+ score: 0.82,+ severity: "gate",+ threshold: 0.9,+ passed: false,+ errored: false,+ });+ expect(computeEvalVerdict({ assertions })).toBe("failed");+ });++ it("passes exactly at the threshold and only demotes under a soft bar", async () => {+ const collector = new AssertionCollector();+ collector.recordScore(0.9).label("exact").gate(0.9);+ collector.recordScore(0.5).label("wording").atLeast(0.6);++ const assertions = await collector.finalize(taskResult());++ expect(+ assertions.map(({ severity, threshold, passed }) => ({ severity, threshold, passed })),+ ).toEqual([+ { severity: "gate", threshold: 0.9, passed: true },+ { severity: "soft", threshold: 0.6, passed: false },+ ]);+ expect(computeEvalVerdict({ assertions })).toBe("scored");+ });++ it("reports the implicit gate threshold of 1", async () => {+ const collector = new AssertionCollector();+ collector.recordScore(1).label("full").gate();+ collector.recordScore(0.99).label("partial").gate();++ const assertions = await collector.finalize(taskResult());++ expect(assertions.map(({ threshold, passed }) => ({ threshold, passed }))).toEqual([+ { threshold: 1, passed: true },+ { threshold: 1, passed: false },+ ]);+ });++ it("reports a thrown evaluator as an errored outcome with no score", async () => {+ const collector = new AssertionCollector();+ collector+ .recordScore(Promise.reject(new Error("model unavailable")))+ .label("judge")+ .atLeast(0.5);+ collector.recordScore(0.82).label("fidelity");++ const assertions = await collector.finalize(taskResult());++ expect(assertions[0]).toMatchObject({+ key: "judge",+ severity: "gate",+ passed: false,+ errored: true,+ message: "model unavailable",+ });+ expect(assertions[0]?.score).toBeUndefined();+ expect(assertions[0]?.threshold).toBeUndefined();+ expect(assertions[1]).toMatchObject({ key: "fidelity", score: 0.82, passed: undefined });+ expect(computeEvalVerdict({ assertions })).toBe("failed");+ });++ it("treats a score outside 0 to 1 as an errored outcome", async () => {+ const collector = new AssertionCollector();+ collector.recordScore(Number.NaN);+ collector.recordScore(1200);++ const [nan, latency] = await collector.finalize(taskResult());++ for (const assertion of [nan, latency]) {+ expect(assertion).toMatchObject({ errored: true, passed: false });+ expect(assertion?.score).toBeUndefined();+ expect(assertion?.message).toContain("from 0 to 1");+ }+ });+});++describe("AssertionCollector labels", () => {+ it("exposes the label as the structured key next to the display name", async () => {+ const collector = new AssertionCollector();+ collector+ .recordValue({ name: "judge.boolean", severity: "soft", score: async () => ({ score: 0.4 }) })+ .label("citation");++ const [assertion] = await collector.finalize(taskResult());++ expect(assertion).toMatchObject({ name: "judge.boolean [citation]", key: "citation" });+ });++ it("leaves the key absent for unlabeled assertions", async () => {+ const collector = new AssertionCollector();+ collector.recordValue({+ name: "judge.boolean",+ severity: "soft",+ score: async () => ({ score: 1 }),+ });++ const [assertion] = await collector.finalize(taskResult());++ expect(assertion?.key).toBeUndefined();+ });+});diff --git a/packages/eve/src/evals/assertions/collector.ts b/packages/eve/src/evals/assertions/collector.tsindex 6cf6c3abc..783a3cdbb 100644--- a/packages/eve/src/evals/assertions/collector.ts+++ b/packages/eve/src/evals/assertions/collector.ts@@ -6,6 +6,7 @@ import type {AssertionResult,AssertionSeverity,EveEvalTaskResult,+ ScoreInput,} from "#evals/types.js";import type { TargetTools } from "#evals/target.js";import type { EveEvalAssertionSubject } from "#evals/assertions/run.js";@@ -24,12 +25,14 @@ export interface RunAssertion {interface MutableEntry {readonly baseName: string;name: string;+ key: string | undefined;severity: AssertionSeverity;threshold: number | undefined;readonly kind: "deferred" | "resolved";readonly spec?: RunAssertion;readonly selectSubject?: (result: EveEvalTaskResult) => EveEvalAssertionSubject;- score: number;+ /** The measurement once produced; stays `undefined` when the scorer threw. */+ score: number | undefined;message?: string;metadata?: Readonly<Record<string, unknown>>;/** A model/value assertion that threw — a hard failure regardless of severity. */@@ -65,60 +68,39 @@ export class AssertionCollector {selectSubject: (result: EveEvalTaskResult) => EveEvalAssertionSubject,severity: AssertionSeverity = "gate",): AssertionHandle {- const entry: MutableEntry = {- baseName: spec.name,- name: spec.name,- severity,- threshold: undefined,- kind: "deferred",- spec,- selectSubject,- score: 0,- errored: false,- };- this.#entries.push(entry);+ const entry = this.#add({ name: spec.name, severity, kind: "deferred", spec, selectSubject });return makeHandle(entry);}/** Register a value/judge assertion, evaluating the captured value now. */recordValue(input: {readonly name: string;+ readonly key?: string;readonly severity: AssertionSeverity;readonly threshold?: number;readonly score: () => Promise<AssertionOutcome>;}): AssertionHandle {- const entry: MutableEntry = {- baseName: input.name,- name: input.name,- severity: input.severity,- threshold: input.threshold,- kind: "resolved",- score: 0,- errored: false,- };- this.#entries.push(entry);-- const pending = settleEntry(entry, input.score);-- this.#pending.push(pending);+ const entry = this.#add({ ...input, kind: "resolved" });+ this.#pending.push(settleEntry(entry, input.score));return makeHandle(entry);}+ /** Record a score. It has no verdict unless the handle chains `.gate()` or `.atLeast()`. */+ recordScore(evaluation: ScoreInput): AssertionHandle {+ return this.recordValue({+ name: "score",+ severity: "soft",+ score: async () => toOutcome(await evaluation),+ });+ }+/** Record an already-computed assertion outcome and return whether it passed. */recordOutcome(input: { readonly name: string; readonly outcome: AssertionOutcome }): boolean {- const entry: MutableEntry = {- baseName: input.name,- name: input.name,- severity: "gate",- threshold: undefined,- kind: "resolved",- score: input.outcome.score,- message: input.outcome.message,- metadata: input.outcome.metadata,- errored: false,- };- this.#entries.push(entry);- return computePassed(entry.severity, entry.threshold, entry.score, entry.errored);+ const entry = this.#add({ name: input.name, severity: "gate", kind: "resolved" });+ entry.score = input.outcome.score;+ entry.message = input.outcome.message;+ entry.metadata = input.outcome.metadata;+ return computePassed(entry) === true;}/** Record and await a required value assertion, returning whether it passed. */@@ -127,18 +109,9 @@ export class AssertionCollector {readonly threshold?: number;readonly score: () => Promise<AssertionOutcome>;}): Promise<boolean> {- const entry: MutableEntry = {- baseName: input.name,- name: input.name,- severity: "gate",- threshold: input.threshold,- kind: "resolved",- score: 0,- errored: false,- };- this.#entries.push(entry);+ const entry = this.#add({ ...input, severity: "gate", kind: "resolved" });await settleEntry(entry, input.score);- return computePassed(entry.severity, entry.threshold, entry.score, entry.errored);+ return computePassed(entry) === true;}/**@@ -156,35 +129,73 @@ export class AssertionCollector {entry.message = outcome.message;entry.metadata = outcome.metadata;}- results.push({- name: entry.name,- score: entry.score,- severity: entry.severity,- threshold: entry.threshold,- passed: computePassed(entry.severity, entry.threshold, entry.score, entry.errored),- errored: entry.errored,- message: entry.message,- metadata: entry.metadata,- });+ results.push(toResult(entry));}return results;}++ #add(input: {+ readonly name: string;+ readonly key?: string;+ readonly severity: AssertionSeverity;+ readonly threshold?: number;+ readonly kind: MutableEntry["kind"];+ readonly spec?: RunAssertion;+ readonly selectSubject?: MutableEntry["selectSubject"];+ }): MutableEntry {+ const entry: MutableEntry = {+ baseName: input.name,+ name: input.name,+ key: input.key,+ severity: input.severity,+ threshold: input.threshold,+ kind: input.kind,+ spec: input.spec,+ selectSubject: input.selectSubject,+ score: undefined,+ errored: false,+ };+ this.#entries.push(entry);+ return entry;+ }+}++function toResult(entry: MutableEntry): AssertionResult {+ return {+ name: entry.name,+ key: entry.key,+ score: entry.score,+ severity: entry.severity,+ threshold: entry.errored ? undefined : ruleThreshold(entry),+ passed: computePassed(entry),+ errored: entry.errored,+ message: entry.message,+ metadata: entry.metadata,+ };+}++/** The effective minimum passing score: a gate defaults to 1, a soft entry has none. */+function ruleThreshold(entry: MutableEntry): number | undefined {+ return entry.threshold ?? (entry.severity === "gate" ? 1 : undefined);}/**- * Whether an assertion meets its bar. A gate defaults to threshold 1; a soft- * assertion with no threshold is tracked-only and always "passes". A hard- * thrown failure never passes.+ * Verdict of the entry's rule. A thrown scorer fails; an entry without a rule+ * (soft, no threshold) has no verdict.*/-function computePassed(- severity: AssertionSeverity,- threshold: number | undefined,- score: number,- errored: boolean,-): boolean {- if (errored) return false;- const min = threshold ?? (severity === "gate" ? 1 : undefined);- return min === undefined || score >= min;+function computePassed(entry: MutableEntry): boolean | undefined {+ if (entry.errored) return false;+ const threshold = ruleThreshold(entry);+ if (threshold === undefined) return undefined;+ return entry.score !== undefined && entry.score >= threshold;+}++function toOutcome(value: number | AssertionOutcome): AssertionOutcome {+ const outcome = typeof value === "number" ? { score: value } : value;+ if (typeof outcome.score !== "number" || !(outcome.score >= 0 && outcome.score <= 1)) {+ throw new TypeError(`Scores must be numbers from 0 to 1; received ${String(outcome.score)}.`);+ }+ return outcome;}async function settleEntry(@@ -199,7 +210,7 @@ async function settleEntry(} catch (error: unknown) {// A judge/value assertion that throws (e.g. a judge model error) is a// hard failure, surfaced as a failed gate rather than aborting the run.- entry.score = 0;+ // A thrown scorer produces no measurement, so the score stays absent.entry.severity = "gate";entry.threshold = undefined;entry.message = toErrorMessage(error);@@ -226,6 +237,7 @@ function makeHandle(entry: MutableEntry): AssertionHandle {},label(label) {entry.name = formatAssertionName(entry.baseName, label);+ entry.key = label.trim();return handle;},};diff --git a/packages/eve/src/evals/context.ts b/packages/eve/src/evals/context.tsindex af1ab0d2c..71ab868d0 100644--- a/packages/eve/src/evals/context.ts+++ b/packages/eve/src/evals/context.ts@@ -50,6 +50,7 @@ export function createEvalContext(deps: {// Value-level assertion over an explicit value.check: (value, assertion) => recordCheck(collector, value, assertion),+ score: (evaluation) => collector.recordScore(evaluation),require: (value, assertion) => requireCheck(collector, value, assertion),skip: (reason) => {if (reason.trim().length === 0) throw new Error("skip() requires a non-empty reason.");diff --git a/packages/eve/src/evals/index.ts b/packages/eve/src/evals/index.tsindex 24e24cb35..2737a6d8f 100644--- a/packages/eve/src/evals/index.ts+++ b/packages/eve/src/evals/index.ts@@ -66,6 +66,7 @@ export type {JudgeQuestion,JudgeContext,JudgeOpts,+ ScoreInput,} from "#evals/types.js";export type {diff --git a/packages/eve/src/evals/judge.integration.test.ts b/packages/eve/src/evals/judge.integration.test.tsindex 5c05b8930..7e489beaa 100644--- a/packages/eve/src/evals/judge.integration.test.ts+++ b/packages/eve/src/evals/judge.integration.test.ts@@ -123,11 +123,11 @@ describe("decision-backed judge runner", () => {expect(outcome.assertions).toHaveLength(batch ? 2 : 1);for (const assertion of outcome.assertions) {expect(assertion).toMatchObject({- score: 0,severity: "gate",passed: false,errored: true,});+ expect(assertion.score).toBeUndefined();expect(assertion.message).toContain("score criteria must contain at least two ordered levels",);diff --git a/packages/eve/src/evals/judge.test.ts b/packages/eve/src/evals/judge.test.tsindex 4ac3a1b84..4f3fa984c 100644--- a/packages/eve/src/evals/judge.test.ts+++ b/packages/eve/src/evals/judge.test.ts@@ -97,7 +97,7 @@ describe("judge", () => {expect(decide.mock.calls[0]?.[0].questions.outcome).not.toHaveProperty("expected");const assertions = await collector.finalize(emptyTaskResult());expect(assertions.map(({ name, score, passed }) => ({ name, score, passed }))).toEqual([- { name: "judge.boolean.accurate", score: 0.2, passed: true },+ { name: "judge.boolean.accurate", score: 0.2, passed: undefined },{ name: "judge.score.clarity", score: 0.75, passed: false },{ name: "judge.choice.outcome", score: 1, passed: true },]);diff --git a/packages/eve/src/evals/runner/artifacts.ts b/packages/eve/src/evals/runner/artifacts.tsindex 8d6a85482..87487a3b7 100644--- a/packages/eve/src/evals/runner/artifacts.ts+++ b/packages/eve/src/evals/runner/artifacts.ts@@ -66,6 +66,7 @@ function buildSummaryArtifact(summary: EveEvalRunSummary): Record<string, unknowstatus: result.result.status,assertions: result.assertions.map((a) => ({name: a.name,+ key: a.key,score: a.score,severity: a.severity,threshold: a.threshold,diff --git a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.tsindex 4b298431b..72cea442e 100644--- a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts+++ b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.test.ts@@ -29,4 +29,16 @@ describe("assertion failure diagnostics", () => {"…",]);});++ it("omits the score comparison when the scorer threw", () => {+ const assertion: AssertionResult = {+ message: "model unavailable",+ name: "judge.boolean",+ passed: false,+ errored: true,+ severity: "gate",+ };++ expect(formatAssertionFailureHeadline(assertion)).toBe("judge.boolean: model unavailable");+ });});diff --git a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.ts b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.tsindex 12135be0c..f033d9d06 100644--- a/packages/eve/src/evals/runner/reporters/assertion-diagnostics.ts+++ b/packages/eve/src/evals/runner/reporters/assertion-diagnostics.ts@@ -5,11 +5,10 @@ const MAX_DETAIL_LINE_LENGTH = 240;const MAX_DETAIL_LINES = 4;export function formatAssertionFailureHeadline(assertion: AssertionResult): string {- const threshold = assertion.threshold ?? (assertion.severity === "gate" ? 1 : undefined);const comparison =- threshold === undefined+ assertion.threshold === undefined || assertion.score === undefined? ""- : ` (${formatPercent(assertion.score)} < ${formatPercent(threshold)})`;+ : ` (${formatPercent(assertion.score)} < ${formatPercent(assertion.threshold)})`;const [headline] = assertion.message?.split("\n") ?? [];const detail =headline === undefined ? "" : `: ${truncateDiagnostic(headline, MAX_DETAIL_LINE_LENGTH)}`;diff --git a/packages/eve/src/evals/runner/reporters/assertion-scores.ts b/packages/eve/src/evals/runner/reporters/assertion-scores.tsindex 85ab4172a..449745d80 100644--- a/packages/eve/src/evals/runner/reporters/assertion-scores.ts+++ b/packages/eve/src/evals/runner/reporters/assertion-scores.ts@@ -1,8 +1,19 @@import type { AssertionResult } from "#evals/types.js";+/**+ * The name an assertion is exported under. Gates carry a `gate:` prefix so+ * they do not share a column with a soft assertion of the same name, and so+ * experiments diff gate regressions the same way they diff soft scores.+ * Sinks with stricter label rules normalize this name themselves.+ */+export function exportedAssertionName(assertion: AssertionResult): string {+ return assertion.severity === "gate" ? `gate:${assertion.name}` : assertion.name;+}+/** Assertions reported under one name, scored together. */export interface AssertionScoreGroup {- readonly score: number;+ /** Lowest completed score in the group; absent when every member errored. */+ readonly score: number | undefined;readonly assertions: readonly AssertionResult[];}@@ -21,7 +32,7 @@ export function groupAssertionScores(const name = getName(assertion);const group = groups.get(name);groups.set(name, {- score: Math.min(group?.score ?? assertion.score, assertion.score),+ score: minScore(group?.score, assertion.score),assertions: [...(group?.assertions ?? []), assertion],});}@@ -30,5 +41,11 @@ export function groupAssertionScores(/** Per-assertion scores retained in row metadata once repeated assertions share one score. */export function composeAssertionScoreMetadata(assertions: readonly AssertionResult[]) {- return assertions.map(({ name, severity, score }) => ({ name, severity, score }));+ return assertions.map(({ key, name, severity, score }) => ({ key, name, severity, score }));+}++function minScore(current: number | undefined, next: number | undefined): number | undefined {+ if (current === undefined) return next;+ if (next === undefined) return current;+ return Math.min(current, next);}diff --git a/packages/eve/src/evals/runner/reporters/braintrust.ts b/packages/eve/src/evals/runner/reporters/braintrust.tsindex 646640f0d..0263e62ab 100644--- a/packages/eve/src/evals/runner/reporters/braintrust.ts+++ b/packages/eve/src/evals/runner/reporters/braintrust.ts@@ -1,6 +1,7 @@import type { EveEval, EveEvalResult, EveEvalRunSummary, EveEvalTarget } from "#evals/types.js";import {composeAssertionScoreMetadata,+ exportedAssertionName,groupAssertionScores,} from "#evals/runner/reporters/assertion-scores.js";import { resolveLocalGitMetadata } from "#evals/runner/resolve-git-metadata.js";@@ -133,19 +134,14 @@ class BraintrustReporter implements EvalReporter {if (!this.#experiment) return;const evaluation = this.#evaluations.get(result.id);- // Soft assertions log under their own name; gate assertions log as binary- // scores under a `gate:` prefix so experiments diff gate regressions the- // same way they diff soft-score regressions.const scores: Record<string, number> = {};- const groups = groupAssertionScores(result.assertions, (assertion) =>- assertion.severity === "gate" ? `gate:${assertion.name}` : assertion.name,- );+ const groups = groupAssertionScores(result.assertions, exportedAssertionName);for (const [name, group] of groups) {- scores[name] = group.score;+ if (group.score !== undefined) scores[name] = group.score;}const failedAssertions = result.assertions- .filter((assertion) => !assertion.passed)+ .filter((assertion) => assertion.passed === false).map((assertion) => ({ ...assertion }));const metadata: Record<string, unknown> = {diff --git a/packages/eve/src/evals/runner/reporters/console.test.ts b/packages/eve/src/evals/runner/reporters/console.test.tsindex 2b7cdf504..24fdee0c5 100644--- a/packages/eve/src/evals/runner/reporters/console.test.ts+++ b/packages/eve/src/evals/runner/reporters/console.test.ts@@ -56,6 +56,73 @@ describe("Console", () => {expect(lines).toContain(' answer: {"type":"boolean","probability":0}');});+ it("warns when repeated soft assertions merge different scores and leaves gates quiet", () => {+ const lines: string[] = [];+ const reporter = Console({ color: false, log: (line) => lines.push(line) });++ reporter.onEvalComplete({+ assertions: [+ { name: "judge.boolean", passed: true, errored: false, score: 0.9, severity: "soft" },+ { name: "judge.boolean", passed: true, errored: false, score: 0.4, severity: "soft" },+ { name: "judge.boolean", passed: true, errored: false, score: 1, severity: "gate" },+ { name: "succeeded", passed: true, errored: false, score: 1, severity: "gate" },+ {+ name: "succeeded",+ passed: false,+ errored: false,+ score: 0,+ severity: "gate",+ threshold: 1,+ },+ ],+ completedAt: "2026-01-01T00:00:01.000Z",+ id: "quality",+ result: {+ derived: createEmptyDerivedFacts(),+ events: [],+ finalMessage: "ok",+ output: "ok",+ status: "completed",+ traceContexts: [],+ },+ startedAt: "2026-01-01T00:00:00.000Z",+ verdict: "failed",+ });++ expect(lines).toEqual([+ "✗ quality gates 2/3 judge.boolean: 90% judge.boolean: 40%",+ " ✗ succeeded (0% < 100%)",+ " ⚠ judge.boolean recorded 2 times with different scores; Braintrust and Datadog keep only the lowest. Add .label() to tell them apart.",+ ]);+ });++ it("aggregates repeated soft scores as one value per eval in the run summary", () => {+ const lines: string[] = [];+ const reporter = Console({ color: false, log: (line) => lines.push(line) });+ const result: EveEvalResult = {+ assertions: [+ { name: "judge.boolean", passed: true, errored: false, score: 0.9, severity: "soft" },+ { name: "judge.boolean", passed: true, errored: false, score: 0.4, severity: "soft" },+ ],+ completedAt: "2026-01-01T00:00:01.000Z",+ id: "quality",+ result: {+ derived: createEmptyDerivedFacts(),+ events: [],+ finalMessage: "ok",+ output: "ok",+ status: "completed",+ traceContexts: [],+ },+ startedAt: "2026-01-01T00:00:00.000Z",+ verdict: "passed",+ };++ reporter.onRunComplete(summary(result));++ expect(lines).toContain(" judge.boolean: 40% (1 evals)");+ });+it("does not print diagnostics for passing assertions", () => {const lines: string[] = [];const reporter = Console({ color: false, log: (line) => lines.push(line) });diff --git a/packages/eve/src/evals/runner/reporters/console.ts b/packages/eve/src/evals/runner/reporters/console.tsindex 0b969fd3f..0bb4a93ad 100644--- a/packages/eve/src/evals/runner/reporters/console.ts+++ b/packages/eve/src/evals/runner/reporters/console.ts@@ -11,6 +11,10 @@ import {formatAssertionFailureDetailLines,formatAssertionFailureHeadline,} from "#evals/runner/reporters/assertion-diagnostics.js";+import {+ exportedAssertionName,+ groupAssertionScores,+} from "#evals/runner/reporters/assertion-scores.js";import type { EvalReporter } from "#evals/runner/reporters/types.js";/**@@ -64,7 +68,9 @@ class ConsoleReporter implements EvalReporter {? this.#formatGateCount(gates.filter((gate) => gate.passed).length, gates.length): "";const scoreText = softs- .map((assertion) => this.#formatScore(assertion.name, assertion.score))+ .flatMap((assertion) =>+ assertion.score === undefined ? [] : [this.#formatScore(assertion.name, assertion.score)],+ ).join(" ");const line = [icon, this.#colors.dim(result.id), gateText, scoreText]@@ -73,13 +79,26 @@ class ConsoleReporter implements EvalReporter {this.#log(line);for (const assertion of assertions) {- if (assertion.passed) continue;+ if (assertion.passed !== false) continue;this.#log(` ${this.#colors.red(`✗ ${formatAssertionFailureHeadline(assertion)}`)}`);for (const detailLine of formatAssertionFailureDetailLines(assertion)) {this.#log(` ${this.#colors.red(detailLine)}`);}}+ // Braintrust and Datadog keep one score per exported name, the lowest. For+ // gates that aggregate is exact and the failure line above already names+ // the failing member, so warn only for soft groups whose scores differ.+ for (const group of groupAssertionScores(softs, exportedAssertionName).values()) {+ if (group.assertions.every((assertion) => assertion.score === group.score)) continue;+ const name = group.assertions[0]!.name;+ this.#log(+ ` ${this.#colors.yellow(+ `⚠ ${name} recorded ${group.assertions.length} times with different scores; Braintrust and Datadog keep only the lowest. Add .label() to tell them apart.`,+ )}`,+ );+ }+if (error) {this.#log(` ${this.#colors.red(error)}`);}@@ -186,15 +205,18 @@ class ConsoleReporter implements EvalReporter {): { name: string; avg: number; count: number }[] {const totals = new Map<string, { sum: number; count: number }>();+ // One value per name per eval, the group minimum, matching what the+ // export reporters record; `count` is the number of evals.for (const result of results) {- for (const assertion of result.assertions) {- if (assertion.severity !== "soft") continue;- const entry = totals.get(assertion.name);+ const softs = result.assertions.filter((assertion) => assertion.severity === "soft");+ for (const [name, group] of groupAssertionScores(softs, (assertion) => assertion.name)) {+ if (group.score === undefined) continue;+ const entry = totals.get(name);if (entry) {- entry.sum += assertion.score;+ entry.sum += group.score;entry.count += 1;} else {- totals.set(assertion.name, { sum: assertion.score, count: 1 });+ totals.set(name, { sum: group.score, count: 1 });}}}diff --git a/packages/eve/src/evals/runner/reporters/datadog.test.ts b/packages/eve/src/evals/runner/reporters/datadog.test.tsindex d9d3f4bf9..b7f5afce1 100644--- a/packages/eve/src/evals/runner/reporters/datadog.test.ts+++ b/packages/eve/src/evals/runner/reporters/datadog.test.ts@@ -484,6 +484,23 @@ describe("Datadog", () => {]);});+ it("omits assertion_passed when no member of a group has a verdict", async () => {+ const { config, experiment } = makeConfig({ recordInputs: false });+ const reporter = Datadog(config);+ const result = makeEvalResult({+ assertions: [{ name: "judge.boolean", score: 0.9, severity: "soft", errored: false }],+ });++ await reporter.onRunStart([makeEval()], makeTarget());+ await reporter.onEvalComplete(result);++ expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toContainEqual({+ label: "judge_boolean",+ value: 0.9,+ tags: { assertion_severity: "soft", assertion_count: "1" },+ });+ });+it("records inputs, outputs, expected outputs, and runtime links in a shared dataset by default", async () => {const { client, config, datasets, experiment, lines } = makeConfig({ experimentName: "run-1" });const reporter = Datadog(config);diff --git a/packages/eve/src/evals/runner/reporters/datadog.ts b/packages/eve/src/evals/runner/reporters/datadog.tsindex 10c024cca..201e802f4 100644--- a/packages/eve/src/evals/runner/reporters/datadog.ts+++ b/packages/eve/src/evals/runner/reporters/datadog.ts@@ -3,6 +3,7 @@ import { resolveRuntimeTraceLinks } from "#evals/runner/reporters/datadog-runtimimport type { EvalReporter } from "#evals/runner/reporters/types.js";import {composeAssertionScoreMetadata,+ exportedAssertionName,groupAssertionScores,} from "#evals/runner/reporters/assertion-scores.js";import {@@ -483,7 +484,7 @@ function resolveResultMetadata(}if (recordAssertionDetails) {const failedAssertions = result.assertions- .filter((assertion) => !assertion.passed)+ .filter((assertion) => assertion.passed === false).map((assertion) => ({ name: assertion.name, message: assertion.message }));if (failedAssertions.length > 0) {metadata.eveFailedAssertions = failedAssertions;@@ -500,16 +501,22 @@ function resolveEvaluationMetrics(result: EveEvalResult): DatadogEvaluationMetriconst metrics: DatadogEvaluationMetricInput[] = [];const usedLabels = new Set<string>(BUILT_IN_METRIC_LABELS);const groups = groupAssertionScores(result.assertions, (assertion) =>- toDatadogMetricLabel(assertion.severity === "gate" ? `gate_${assertion.name}` : assertion.name),+ toDatadogMetricLabel(exportedAssertionName(assertion)),);for (const [label, group] of groups) {+ if (group.score === undefined) continue;+ const verdicts = group.assertions.flatMap((assertion) =>+ assertion.passed === undefined ? [] : [assertion.passed],+ );metrics.push({label: reserveDatadogMetricLabel(label, usedLabels),value: group.score,tags: {assertion_severity: group.assertions[0]?.severity ?? "soft",- assertion_passed: String(group.assertions.every((assertion) => assertion.passed)),+ ...(verdicts.length > 0 && {+ assertion_passed: String(verdicts.every((passed) => passed)),+ }),assertion_count: String(group.assertions.length),},});diff --git a/packages/eve/src/evals/runner/reporters/junit.test.ts b/packages/eve/src/evals/runner/reporters/junit.test.tsindex 89a08b15a..bb43c8414 100644--- a/packages/eve/src/evals/runner/reporters/junit.test.ts+++ b/packages/eve/src/evals/runner/reporters/junit.test.ts@@ -73,6 +73,7 @@ function makeSummary(): EveEvalRunSummary {name: "contains",score: 0,severity: "gate",+ threshold: 1,passed: false,errored: false,message: "expected hello",diff --git a/packages/eve/src/evals/runner/reporters/junit.ts b/packages/eve/src/evals/runner/reporters/junit.tsindex 680128df1..31631a7a6 100644--- a/packages/eve/src/evals/runner/reporters/junit.ts+++ b/packages/eve/src/evals/runner/reporters/junit.ts@@ -91,7 +91,7 @@ function buildFailureDetail(result: EveEvalResult): Record<string, unknown> {function failureMessage(result: EveEvalResult): string {if (result.error !== undefined) return result.error;- const failed = result.assertions.filter((assertion) => !assertion.passed);+ const failed = result.assertions.filter((assertion) => assertion.passed === false);if (failed.length > 0) {return truncateDiagnostic(failed.map(formatAssertionFailureHeadline).join("; "), 1_000);}diff --git a/packages/eve/src/evals/runner/verdict.test.ts b/packages/eve/src/evals/runner/verdict.test.tsindex 84f13b17f..e39f888fa 100644--- a/packages/eve/src/evals/runner/verdict.test.ts+++ b/packages/eve/src/evals/runner/verdict.test.ts@@ -69,14 +69,24 @@ describe("computeEvalVerdict", () => {).toBe("failed");});- it("passes when a tracked soft assertion has no threshold", () => {+ it("passes when a tracked soft assertion has no verdict", () => {expect(computeEvalVerdict({- assertions: [assertion({ severity: "soft", score: 0.1, passed: true, errored: false })],+ assertions: [assertion({ severity: "soft", score: 0.1, passed: undefined })],}),).toBe("passed");});+ it("fails when a scorer errored without producing a score", () => {+ expect(+ computeEvalVerdict({+ assertions: [+ assertion({ severity: "gate", score: undefined, passed: false, errored: true }),+ ],+ }),+ ).toBe("failed");+ });+it("returns skipped when an eval intentionally skips without another failure", () => {expect(computeEvalVerdict({diff --git a/packages/eve/src/evals/runner/verdict.ts b/packages/eve/src/evals/runner/verdict.tsindex 27ed56539..d1ef48b3b 100644--- a/packages/eve/src/evals/runner/verdict.ts+++ b/packages/eve/src/evals/runner/verdict.ts@@ -5,8 +5,8 @@ import type { AssertionResult, EveEvalVerdict } from "#evals/types.js";* assertions. An execution error or a failed gate assertion is a hard* failure; a below-threshold soft assertion at worst demotes the eval to* `"scored"` (which `eve eval --strict` later promotes to a failing exit- * code). Soft assertions without a threshold are tracked-only and never- * demote the verdict.+ * code). Entries without an acceptance rule carry no verdict and never+ * demote the eval.*/export function computeEvalVerdict(input: {readonly error?: string;@@ -17,7 +17,7 @@ export function computeEvalVerdict(input: {let demoted = false;for (const assertion of input.assertions) {- if (assertion.passed) continue;+ if (assertion.passed !== false) continue;if (assertion.severity === "gate") return "failed";demoted = true;}diff --git a/packages/eve/src/evals/types.ts b/packages/eve/src/evals/types.tsindex b0ebd8afb..d2e570617 100644--- a/packages/eve/src/evals/types.ts+++ b/packages/eve/src/evals/types.ts@@ -202,6 +202,9 @@ export interface AssertionEvaluation {readonly metadata?: Readonly<Record<string, unknown>>;}+/** A score from 0 to 1, or an evaluation carrying one, accepted by `t.score`; may be a promise. */+export type ScoreInput = number | AssertionEvaluation | Promise<number | AssertionEvaluation>;+/*** A value-level assertion produced by the builders in `eve/evals/expect`* (e.g. `includes`, `equals`, `similarity`) and applied to an explicit value@@ -234,21 +237,29 @@ export interface AssertionHandle {gate(threshold?: number): this;soft(threshold?: number): this;atLeast(threshold: number): this;- /** Adds a stable human-readable label to this recorded assertion. */+ /** Sets this entry's stable `key` and appends the label to its display name. */label(label: string): this;}/**- * The recorded outcome of one assertion, consumed by the verdict, reporters,- * and artifacts. A boolean assertion has `score` 0 or 1.+ * The recorded outcome of one assertion or `t.score`, consumed by the verdict,+ * reporters, and artifacts. `score` is the measurement and a rule does not+ * change it; `threshold` and `passed` describe the rule and are absent when+ * none applies. A boolean assertion has `score` 0 or 1.*/export interface AssertionResult {+ /** Display name, e.g. `judge.boolean [citation]`. */readonly name: string;- readonly score: number;+ /** Stable identifier set by `.label(key)`; absent for unlabeled assertions. */+ readonly key?: string;+ /** The measurement; absent when the scorer threw. */+ readonly score?: number;readonly severity: AssertionSeverity;+ /** Effective minimum passing score (a gate defaults to 1); absent when no rule applies. */readonly threshold?: number;- readonly passed: boolean;- /** Whether the assertion failed because its scorer threw instead of producing a score. */+ /** Verdict of the acceptance rule; absent when no rule applies. */+ readonly passed?: boolean;+ /** Whether the scorer threw; an errored entry is always a failed outcome. */readonly errored: boolean;/** Human-readable failure detail, shown in console output and artifacts. */readonly message?: string;@@ -500,6 +511,12 @@ export interface EveEvalContext<TContext = unknown> extends EveEvalAssertions {/** Apply a value-level assertion (from `eve/evals/expect`) to a value. */check(value: unknown, assertion: Assertion): AssertionHandle;+ /**+ * Record a score from 0 to 1. Chain `.label(key)` to name it, `.gate(0.9)`+ * to fail the eval below 0.9, or `.atLeast(0.9)` to mark it `scored`.+ * Without a rule the score has no verdict.+ */+ score(evaluation: ScoreInput): AssertionHandle;/** Record an immediate gate and abort dependent control flow when it fails. */require<T>(value: T, assertion: Assertion): Promise<T>;/** Mark this eval as intentionally skipped and stop executing its test body. */diff --git a/packages/eve/test/scenarios/eval-command-environment.scenario.test.ts b/packages/eve/test/scenarios/eval-command-environment.scenario.test.tsindex 960e4b395..cbe11e87f 100644--- a/packages/eve/test/scenarios/eval-command-environment.scenario.test.ts+++ b/packages/eve/test/scenarios/eval-command-environment.scenario.test.ts@@ -519,7 +519,14 @@ function makeEvalResult(id: string) {id,assertions: [failed- ? { name: "check", score: 0, severity: "gate" as const, passed: false, message: "nope" }+ ? {+ name: "check",+ score: 0,+ severity: "gate" as const,+ threshold: 1,+ passed: false,+ message: "nope",+ }: { name: "check", score: 1, severity: "gate" as const, passed: true },],result: {diff --git a/research/eval-scores-and-verdicts.md b/research/eval-scores-and-verdicts.mdnew file mode 100644index 000000000..17d666a9a--- /dev/null+++ b/research/eval-scores-and-verdicts.md@@ -0,0 +1,61 @@+---+issue: https://github.com/vercel/eve/pull/4563+status: implemented+last_updated: "2026-10-09"+---++# Separate evaluation scores from pass/fail decisions in eve++Expose each evaluator's score and optional pass/fail verdict separately, so+consumers can persist both, including when a threshold fails.++## Motivation++An evaluation store needs the original measurement to compare runs, while CI+needs an acceptance decision. A score of `0.82` must remain `0.82` when a `0.9`+threshold makes the test fail.++Before this change, [`AssertionResult`](../packages/eve/src/evals/types.ts)+reported `passed: true` for assertions without a threshold, wrote `score: 0`+when a scorer threw, and had no identifier other than the display `name`. A+score is a measurement; an assertion is a rule applied to it.++## Authoring contract++`t.score(evaluation)` records a score and returns the ordinary+`AssertionHandle`; `.label(key)` names it, as for every other assertion.+`evaluation` is a number or `{ score, message?, metadata? }`, or a promise of+either. Scores are numbers from 0 to 1; boolean assertions keep scoring 0 or 1.++```ts+const grade = await evaluator.evaluate({ input, output, reference });+t.score(grade).label("faithfulness"); // no verdict, never fails++t.score(grade).label("faithfulness").gate(0.9); // fails the eval below 0.9+t.score(grade).label("faithfulness").atLeast(0.9); // marks it `scored`, fatal under --strict+```++A rule does not change the score or rerun the evaluator. Existing assertions+are unchanged; `.label(key)` sets `key` on any of them.++## Persistence contract++Every finalized `AssertionResult`, in reporters' `onEvalComplete` and in the+`.eve/evals/` artifacts, carries:++| field | meaning |+| ----------- | ----------------------------------------------------------------------------------- |+| `key` | stable identifier set by `.label(key)`; else absent |+| `score` | the measurement; absent when the scorer threw |+| `threshold` | effective minimum passing score (a gate defaults to 1); absent when no rule applies |+| `passed` | verdict of the rule; absent when no rule applies |+| `errored` | the scorer threw; always a failed outcome, with the error in `message` |++For `0.82` gated at `0.9`, consumers read `score: 0.82`, `threshold: 0.9`,+`passed: false`. A failed threshold does not prevent persistence. A scorer+that threw has no score, and the other measurements in the same eval still+reach reporters. Reporters that need a number (Braintrust, Datadog) skip+errored entries.++Related precedent: LangSmith separates [feedback recording](https://docs.langchain.com/langsmith/feedback-data-format)+from optional [test expectations](https://docs.langchain.com/langsmith/pytest#expectations).