The Open Framework for Building Agents
feat(eve): report one score per assertion name in eval reporters (#3907)
Matan Kushner8149c720bcd4
diff --git a/.changeset/eval-reporter-assertion-metrics.md b/.changeset/eval-reporter-assertion-metrics.mdnew file mode 100644index 000000000..a57ee67ae--- /dev/null+++ b/.changeset/eval-reporter-assertion-metrics.md@@ -0,0 +1,5 @@+---+"eve": patch+---++The Braintrust and Datadog reporters now report one score per assertion name, the lowest, instead of numbering repeats like `judge_boolean_2`.diff --git a/docs/evals/reporters.mdx b/docs/evals/reporters.mdxindex deabdcfa3..b7bf1793d 100644--- a/docs/evals/reporters.mdx+++ b/docs/evals/reporters.mdx@@ -38,7 +38,7 @@ export default defineEval({});```-The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Repeated names use `#2`, `#3`, and later suffixes; use `.label(name)` to give them meaningful names. Failed assertion details are stored in the Braintrust metadata under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.+The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Assertions that share a name, such as several unlabeled judges, log one score: the lowest. Use `.label(name)` to group or separate them. Each assertion's score is stored in the Braintrust metadata under `eveAssertionScores` and failed assertion details under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.A reporter instance observes the evals that reference it. Share one instance across several evals (the config, a `shared.ts` export, or every entry of a dataset array) and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.@@ -61,7 +61,7 @@ The reporter creates one Datadog Experiment, submits one synthetic experiment spEvery run reuses the dataset named by `datasetName` (default `<projectName> evals`), so Datadog can compare experiments across runs. Records are matched by eval description, with `#2`, `#3` suffixes for duplicates, so give each case a unique description.-Assertion metrics use descriptive assertion names by default: characters outside letters, numbers, underscores, and hyphens are normalized to underscores, gate labels receive a `gate_` prefix, and repeated or reserved labels receive numeric suffixes. For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Authors can use `.label("stable name")` on an assertion handle to choose a stable metric name. Because assertion names can contain authored expectations, review them before exporting sensitive evals. Pass `recordAssertionDetails: false` to omit raw assertion-name tags and failed assertion messages.+Assertion metrics use descriptive assertion names by default: characters outside letters, numbers, underscores, and hyphens are normalized to underscores, gate labels receive a `gate_` prefix, assertions that share a label report one metric with the lowest score, and labels that collide with built-in metrics receive a numeric suffix. For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Authors can use `.label("stable name")` on an assertion handle to choose a stable metric name. Because assertion names can contain authored expectations, review them before exporting sensitive evals. Each assertion's name and score are kept in row metadata under `eveAssertionScores`; pass `recordAssertionDetails: false` to omit them and failed assertion messages.Datadog needs `dd-trace` installed in the app and Datadog credentials in the environment. The reporter is tested against `[email protected]`; install it with `npm install [email protected]` and set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as appropriate; without credentials, reporting is skipped. Like the Braintrust reporter, it records eval prompts, outputs, authored expectations, and execution error messages by default. Pass `recordInputs: false`, `recordOutputs: false`, `recordExpectedOutputs: false`, or `recordErrors: false` if your destination is not approved for that content. `recordInputs` stores the eval input in both the linked dataset record and experiment span. Input is read from the eval's first `t.send(...)` message, falling back to the eval description when no message event is available. When `recordExpectedOutputs` is enabled, expected output is read from eval `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output` and stored in both the dataset record and experiment span; when no expected output is authored, the input-only dataset record is still valid. Those expected-output keys are excluded from the general metadata payload. Target URL credentials, paths, query parameters, and fragments are never reported.diff --git a/packages/eve/src/evals/runner/reporters/assertion-scores.ts b/packages/eve/src/evals/runner/reporters/assertion-scores.tsnew file mode 100644index 000000000..85ab4172a--- /dev/null+++ b/packages/eve/src/evals/runner/reporters/assertion-scores.ts@@ -0,0 +1,34 @@+import type { AssertionResult } from "#evals/types.js";++/** Assertions reported under one name, scored together. */+export interface AssertionScoreGroup {+ readonly score: number;+ readonly assertions: readonly AssertionResult[];+}++/**+ * Groups an eval's assertions by reported name and scores each group by its lowest member.+ *+ * Repeated assertions, such as several unlabeled judges, share a name. The minimum keeps one+ * comparable score per name, and a group passes only when every member does.+ */+export function groupAssertionScores(+ assertions: readonly AssertionResult[],+ getName: (assertion: AssertionResult) => string,+): Map<string, AssertionScoreGroup> {+ const groups = new Map<string, AssertionScoreGroup>();+ for (const assertion of assertions) {+ const name = getName(assertion);+ const group = groups.get(name);+ groups.set(name, {+ score: Math.min(group?.score ?? assertion.score, assertion.score),+ assertions: [...(group?.assertions ?? []), assertion],+ });+ }+ return groups;+}++/** Per-assertion scores retained in row metadata once repeated assertions share one score. */+export function composeAssertionScoreMetadata(assertions: readonly AssertionResult[]) {+ return assertions.map(({ name, severity, score }) => ({ name, severity, score }));+}diff --git a/packages/eve/src/evals/runner/reporters/braintrust.test.ts b/packages/eve/src/evals/runner/reporters/braintrust.test.tsindex 2236c85f5..9c70aee30 100644--- a/packages/eve/src/evals/runner/reporters/braintrust.test.ts+++ b/packages/eve/src/evals/runner/reporters/braintrust.test.ts@@ -206,7 +206,7 @@ describe("Braintrust", () => {);});- it("keeps duplicate assertion scores under stable keys", async () => {+ it("scores repeated assertions by their lowest member and keeps each score in metadata", async () => {const reporter = Braintrust(makeConfig());await reporter.onRunStart([makeEval()], makeTarget());@@ -215,16 +215,21 @@ describe("Braintrust", () => {assertions: [{ name: "similarity", passed: true, score: 0.8, severity: "soft", errored: false },{ name: "similarity", passed: true, score: 0.6, severity: "soft", errored: false },+ { name: "similarity", passed: true, score: 1, severity: "gate", errored: false },],}),);expect(braintrustMocks.log).toHaveBeenCalledWith(expect.objectContaining({- scores: {- similarity: 0.8,- "similarity#2": 0.6,- },+ scores: { similarity: 0.6, "gate:similarity": 1 },+ metadata: expect.objectContaining({+ eveAssertionScores: [+ { name: "similarity", severity: "soft", score: 0.8 },+ { name: "similarity", severity: "soft", score: 0.6 },+ { name: "similarity", severity: "gate", score: 1 },+ ],+ }),}),);});diff --git a/packages/eve/src/evals/runner/reporters/braintrust.ts b/packages/eve/src/evals/runner/reporters/braintrust.tsindex fe5a1289b..4dc71b9dd 100644--- a/packages/eve/src/evals/runner/reporters/braintrust.ts+++ b/packages/eve/src/evals/runner/reporters/braintrust.ts@@ -1,4 +1,8 @@import type { EveEval, EveEvalResult, EveEvalRunSummary, EveEvalTarget } from "#evals/types.js";+import {+ composeAssertionScoreMetadata,+ groupAssertionScores,+} from "#evals/runner/reporters/assertion-scores.js";import { resolveLocalGitMetadata } from "#evals/runner/resolve-git-metadata.js";import type { EvalReporter } from "#evals/runner/reporters/types.js";@@ -133,13 +137,11 @@ class BraintrustReporter implements EvalReporter {// scores under a `gate:` prefix so experiments diff gate regressions the// same way they diff soft-score regressions.const scores: Record<string, number> = {};- const scoreNameCounts = new Map<string, number>();- for (const assertion of result.assertions) {- const baseName = assertion.severity === "gate" ? `gate:${assertion.name}` : assertion.name;- const count = (scoreNameCounts.get(baseName) ?? 0) + 1;- scoreNameCounts.set(baseName, count);- const key = count === 1 ? baseName : `${baseName}#${count}`;- scores[key] = assertion.score;+ const groups = groupAssertionScores(result.assertions, (assertion) =>+ assertion.severity === "gate" ? `gate:${assertion.name}` : assertion.name,+ );+ for (const [name, group] of groups) {+ scores[name] = group.score;}const failedAssertions = result.assertions@@ -155,6 +157,7 @@ class BraintrustReporter implements EvalReporter {eveToolCalls: result.result.derived.toolCalls.map((call) => call.name),eveSubagentCalls: result.result.derived.subagentCalls.map((call) => call.name),eveParked: result.result.derived.parked,+ eveAssertionScores: composeAssertionScoreMetadata(result.assertions),};if (result.result.traceContexts.length > 0) {diff --git a/packages/eve/src/evals/runner/reporters/datadog.test.ts b/packages/eve/src/evals/runner/reporters/datadog.test.tsindex 1f060c065..778630e90 100644--- a/packages/eve/src/evals/runner/reporters/datadog.test.ts+++ b/packages/eve/src/evals/runner/reporters/datadog.test.ts@@ -282,13 +282,7 @@ describe("Datadog", () => {expect.objectContaining({ label: "eve_message_count", value: 1 }),expect.objectContaining({ label: "eve_reasoning_block_count", value: 0 }),]);- expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).not.toEqual(- expect.arrayContaining([- expect.objectContaining({- tags: expect.objectContaining({ assertion_name: expect.anything() }),- }),- ]),- );+ expect(submittedSpan).not.toHaveProperty("metadata.eveAssertionScores");});it("redacts target URL secrets and omits execution errors when recordErrors is disabled", async () => {@@ -349,21 +343,24 @@ describe("Datadog", () => {expect(experiment.submitSpan.mock.calls[0]?.[0]).not.toHaveProperty("metadata.eveFailedAssertions",);+ expect(experiment.submitSpan.mock.calls[0]?.[0]).not.toHaveProperty(+ "metadata.eveAssertionScores",+ );expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toEqual(expect.arrayContaining([expect.objectContaining({label: "gate_messageIncludes_private_expectation",tags: {- assertion_index: "1",assertion_severity: "gate",assertion_passed: "false",+ assertion_count: "1",},}),]),);});- it("records assertion name tags and failure messages by default", async () => {+ it("records each assertion's score and failure messages by default", async () => {const { config, experiment } = makeConfig();const reporter = Datadog(config);const result = makeEvalResult({@@ -392,26 +389,20 @@ describe("Datadog", () => {},],);- expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toEqual(- expect.arrayContaining([- expect.objectContaining({- label: "gate_messageIncludes_private_expectation",- tags: expect.objectContaining({- assertion_name: "messageIncludes(private expectation)",- }),- }),- ]),- );+ expect(experiment.submitSpan.mock.calls[0]?.[0]).toHaveProperty("metadata.eveAssertionScores", [+ { name: "messageIncludes(private expectation)", severity: "gate", score: 0 },+ ]);});- it("deduplicates assertion metric labels and reserves built-in labels", async () => {+ it("combines repeated assertion names into one metric scored by the lowest member", async () => {const { config, experiment } = makeConfig({ recordInputs: false });const reporter = Datadog(config);const result = makeEvalResult({assertions: [{ name: "same label", score: 1, severity: "soft", passed: true, errored: false },- { name: "same@label", score: 1, severity: "soft", passed: true, errored: false },- { name: "same label", score: 1, severity: "soft", passed: true, errored: false },+ { name: "same@label", score: 0.5, severity: "soft", passed: false, errored: false },+ { name: "same label", score: 0.8, severity: "soft", passed: true, errored: false },+ { name: "same label", score: 1, severity: "gate", passed: true, errored: false },{ name: "eve_tool_call_count", score: 1, severity: "soft", passed: true, errored: false },],});@@ -420,9 +411,16 @@ describe("Datadog", () => {await reporter.onEvalComplete(result);expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toEqual([- expect.objectContaining({ label: "same_label" }),- expect.objectContaining({ label: "same_label_2" }),- expect.objectContaining({ label: "same_label_3" }),+ {+ label: "same_label",+ value: 0.5,+ tags: { assertion_severity: "soft", assertion_passed: "false", assertion_count: "3" },+ },+ {+ label: "gate_same_label",+ value: 1,+ tags: { assertion_severity: "gate", assertion_passed: "true", assertion_count: "1" },+ },expect.objectContaining({ label: "eve_tool_call_count_2" }),expect.objectContaining({ label: "eve_tool_call_count" }),expect.objectContaining({ label: "eve_subagent_call_count" }),diff --git a/packages/eve/src/evals/runner/reporters/datadog.ts b/packages/eve/src/evals/runner/reporters/datadog.tsindex 3809dd2e7..006e9e03a 100644--- a/packages/eve/src/evals/runner/reporters/datadog.ts+++ b/packages/eve/src/evals/runner/reporters/datadog.ts@@ -1,5 +1,9 @@import type { EveEval, EveEvalResult, EveEvalRunSummary, EveEvalTarget } from "#evals/types.js";import type { EvalReporter } from "#evals/runner/reporters/types.js";+import {+ composeAssertionScoreMetadata,+ groupAssertionScores,+} from "#evals/runner/reporters/assertion-scores.js";import {getDatadogDataset,MISSING_DATASET_API_MESSAGE,@@ -48,7 +52,7 @@ export interface DatadogReporterConfig {readonly recordOutputs?: boolean;/** Include `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output` on experiment rows. Defaults to true. */readonly recordExpectedOutputs?: boolean;- /** Include raw assertion names as metric tags and failure messages in row metadata. Defaults to true. */+ /** Include each assertion's name and score, and failure messages, in row metadata. Defaults to true. */readonly recordAssertionDetails?: boolean;/** Include execution error messages, which may contain application data. Defaults to true. */readonly recordErrors?: boolean;@@ -322,10 +326,7 @@ class DatadogReporter implements EvalReporter {}const span = await this.#experiment.submitSpan(spanInput);- await this.#experiment.submitEvaluationMetrics(- span,- resolveEvaluationMetrics(result, this.#recording.recordAssertionDetails),- );+ await this.#experiment.submitEvaluationMetrics(span, resolveEvaluationMetrics(result));}}@@ -481,6 +482,7 @@ function resolveResultMetadata(if (failedAssertions.length > 0) {metadata.eveFailedAssertions = failedAssertions;}+ metadata.eveAssertionScores = composeAssertionScoreMetadata(result.assertions);}if (result.result.derived.failureCode) {metadata.eveFailureCode = result.result.derived.failureCode;@@ -488,28 +490,22 @@ function resolveResultMetadata(return metadata;}-function resolveEvaluationMetrics(- result: EveEvalResult,- recordAssertionDetails: boolean,-): DatadogEvaluationMetricInput[] {+function resolveEvaluationMetrics(result: EveEvalResult): DatadogEvaluationMetricInput[] {const metrics: DatadogEvaluationMetricInput[] = [];const usedLabels = new Set<string>(BUILT_IN_METRIC_LABELS);+ const groups = groupAssertionScores(result.assertions, (assertion) =>+ toDatadogMetricLabel(assertion.severity === "gate" ? `gate_${assertion.name}` : assertion.name),+ );- for (const [index, assertion] of result.assertions.entries()) {- const rawLabel = assertion.severity === "gate" ? `gate_${assertion.name}` : assertion.name;- const tags: Record<string, string> = {- assertion_index: String(index + 1),- assertion_severity: assertion.severity,- assertion_passed: String(assertion.passed),- };- if (recordAssertionDetails) {- tags.assertion_name = assertion.name;- tags.assertion_label = rawLabel;- }+ for (const [label, group] of groups) {metrics.push({- label: reserveDatadogMetricLabel(toDatadogMetricLabel(rawLabel), usedLabels),- value: assertion.score,- tags,+ label: reserveDatadogMetricLabel(label, usedLabels),+ value: group.score,+ tags: {+ assertion_severity: group.assertions[0]?.severity ?? "soft",+ assertion_passed: String(group.assertions.every((assertion) => assertion.passed)),+ assertion_count: String(group.assertions.length),+ },});}