Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

feat(eve): report one score per assertion name in eval reporters (#3907)

Matan Kushner8149c720bcd4
diff --git a/.changeset/eval-reporter-assertion-metrics.md b/.changeset/eval-reporter-assertion-metrics.md
new file mode 100644
index 000000000..a57ee67ae
--- /dev/null
+++ b/.changeset/eval-reporter-assertion-metrics.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+The Braintrust and Datadog reporters now report one score per assertion name, the lowest, instead of numbering repeats like `judge_boolean_2`.
diff --git a/docs/evals/reporters.mdx b/docs/evals/reporters.mdx
index deabdcfa3..b7bf1793d 100644
--- a/docs/evals/reporters.mdx
+++ b/docs/evals/reporters.mdx
@@ -38,7 +38,7 @@ export default defineEval({
});
```
-The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Repeated names use `#2`, `#3`, and later suffixes; use `.label(name)` to give them meaningful names. Failed assertion details are stored in the Braintrust metadata under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.
+The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Assertions that share a name, such as several unlabeled judges, log one score: the lowest. Use `.label(name)` to group or separate them. Each assertion's score is stored in the Braintrust metadata under `eveAssertionScores` and failed assertion details under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.
A reporter instance observes the evals that reference it. Share one instance across several evals (the config, a `shared.ts` export, or every entry of a dataset array) and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
@@ -61,7 +61,7 @@ The reporter creates one Datadog Experiment, submits one synthetic experiment sp
Every run reuses the dataset named by `datasetName` (default `<projectName> evals`), so Datadog can compare experiments across runs. Records are matched by eval description, with `#2`, `#3` suffixes for duplicates, so give each case a unique description.
-Assertion metrics use descriptive assertion names by default: characters outside letters, numbers, underscores, and hyphens are normalized to underscores, gate labels receive a `gate_` prefix, and repeated or reserved labels receive numeric suffixes. For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Authors can use `.label("stable name")` on an assertion handle to choose a stable metric name. Because assertion names can contain authored expectations, review them before exporting sensitive evals. Pass `recordAssertionDetails: false` to omit raw assertion-name tags and failed assertion messages.
+Assertion metrics use descriptive assertion names by default: characters outside letters, numbers, underscores, and hyphens are normalized to underscores, gate labels receive a `gate_` prefix, assertions that share a label report one metric with the lowest score, and labels that collide with built-in metrics receive a numeric suffix. For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Authors can use `.label("stable name")` on an assertion handle to choose a stable metric name. Because assertion names can contain authored expectations, review them before exporting sensitive evals. Each assertion's name and score are kept in row metadata under `eveAssertionScores`; pass `recordAssertionDetails: false` to omit them and failed assertion messages.
Datadog needs `dd-trace` installed in the app and Datadog credentials in the environment. The reporter is tested against `[email protected]`; install it with `npm install [email protected]` and set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as appropriate; without credentials, reporting is skipped. Like the Braintrust reporter, it records eval prompts, outputs, authored expectations, and execution error messages by default. Pass `recordInputs: false`, `recordOutputs: false`, `recordExpectedOutputs: false`, or `recordErrors: false` if your destination is not approved for that content. `recordInputs` stores the eval input in both the linked dataset record and experiment span. Input is read from the eval's first `t.send(...)` message, falling back to the eval description when no message event is available. When `recordExpectedOutputs` is enabled, expected output is read from eval `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output` and stored in both the dataset record and experiment span; when no expected output is authored, the input-only dataset record is still valid. Those expected-output keys are excluded from the general metadata payload. Target URL credentials, paths, query parameters, and fragments are never reported.
diff --git a/packages/eve/src/evals/runner/reporters/assertion-scores.ts b/packages/eve/src/evals/runner/reporters/assertion-scores.ts
new file mode 100644
index 000000000..85ab4172a
--- /dev/null
+++ b/packages/eve/src/evals/runner/reporters/assertion-scores.ts
@@ -0,0 +1,34 @@
+import type { AssertionResult } from "#evals/types.js";
+
+/** Assertions reported under one name, scored together. */
+export interface AssertionScoreGroup {
+ readonly score: number;
+ readonly assertions: readonly AssertionResult[];
+}
+
+/**
+ * Groups an eval's assertions by reported name and scores each group by its lowest member.
+ *
+ * Repeated assertions, such as several unlabeled judges, share a name. The minimum keeps one
+ * comparable score per name, and a group passes only when every member does.
+ */
+export function groupAssertionScores(
+ assertions: readonly AssertionResult[],
+ getName: (assertion: AssertionResult) => string,
+): Map<string, AssertionScoreGroup> {
+ const groups = new Map<string, AssertionScoreGroup>();
+ for (const assertion of assertions) {
+ const name = getName(assertion);
+ const group = groups.get(name);
+ groups.set(name, {
+ score: Math.min(group?.score ?? assertion.score, assertion.score),
+ assertions: [...(group?.assertions ?? []), assertion],
+ });
+ }
+ return groups;
+}
+
+/** Per-assertion scores retained in row metadata once repeated assertions share one score. */
+export function composeAssertionScoreMetadata(assertions: readonly AssertionResult[]) {
+ return assertions.map(({ name, severity, score }) => ({ name, severity, score }));
+}
diff --git a/packages/eve/src/evals/runner/reporters/braintrust.test.ts b/packages/eve/src/evals/runner/reporters/braintrust.test.ts
index 2236c85f5..9c70aee30 100644
--- a/packages/eve/src/evals/runner/reporters/braintrust.test.ts
+++ b/packages/eve/src/evals/runner/reporters/braintrust.test.ts
@@ -206,7 +206,7 @@ describe("Braintrust", () => {
);
});
- it("keeps duplicate assertion scores under stable keys", async () => {
+ it("scores repeated assertions by their lowest member and keeps each score in metadata", async () => {
const reporter = Braintrust(makeConfig());
await reporter.onRunStart([makeEval()], makeTarget());
@@ -215,16 +215,21 @@ describe("Braintrust", () => {
assertions: [
{ name: "similarity", passed: true, score: 0.8, severity: "soft", errored: false },
{ name: "similarity", passed: true, score: 0.6, severity: "soft", errored: false },
+ { name: "similarity", passed: true, score: 1, severity: "gate", errored: false },
],
}),
);
expect(braintrustMocks.log).toHaveBeenCalledWith(
expect.objectContaining({
- scores: {
- similarity: 0.8,
- "similarity#2": 0.6,
- },
+ scores: { similarity: 0.6, "gate:similarity": 1 },
+ metadata: expect.objectContaining({
+ eveAssertionScores: [
+ { name: "similarity", severity: "soft", score: 0.8 },
+ { name: "similarity", severity: "soft", score: 0.6 },
+ { name: "similarity", severity: "gate", score: 1 },
+ ],
+ }),
}),
);
});
diff --git a/packages/eve/src/evals/runner/reporters/braintrust.ts b/packages/eve/src/evals/runner/reporters/braintrust.ts
index fe5a1289b..4dc71b9dd 100644
--- a/packages/eve/src/evals/runner/reporters/braintrust.ts
+++ b/packages/eve/src/evals/runner/reporters/braintrust.ts
@@ -1,4 +1,8 @@
import type { EveEval, EveEvalResult, EveEvalRunSummary, EveEvalTarget } from "#evals/types.js";
+import {
+ composeAssertionScoreMetadata,
+ groupAssertionScores,
+} from "#evals/runner/reporters/assertion-scores.js";
import { resolveLocalGitMetadata } from "#evals/runner/resolve-git-metadata.js";
import type { EvalReporter } from "#evals/runner/reporters/types.js";
@@ -133,13 +137,11 @@ class BraintrustReporter implements EvalReporter {
// scores under a `gate:` prefix so experiments diff gate regressions the
// same way they diff soft-score regressions.
const scores: Record<string, number> = {};
- const scoreNameCounts = new Map<string, number>();
- for (const assertion of result.assertions) {
- const baseName = assertion.severity === "gate" ? `gate:${assertion.name}` : assertion.name;
- const count = (scoreNameCounts.get(baseName) ?? 0) + 1;
- scoreNameCounts.set(baseName, count);
- const key = count === 1 ? baseName : `${baseName}#${count}`;
- scores[key] = assertion.score;
+ const groups = groupAssertionScores(result.assertions, (assertion) =>
+ assertion.severity === "gate" ? `gate:${assertion.name}` : assertion.name,
+ );
+ for (const [name, group] of groups) {
+ scores[name] = group.score;
}
const failedAssertions = result.assertions
@@ -155,6 +157,7 @@ class BraintrustReporter implements EvalReporter {
eveToolCalls: result.result.derived.toolCalls.map((call) => call.name),
eveSubagentCalls: result.result.derived.subagentCalls.map((call) => call.name),
eveParked: result.result.derived.parked,
+ eveAssertionScores: composeAssertionScoreMetadata(result.assertions),
};
if (result.result.traceContexts.length > 0) {
diff --git a/packages/eve/src/evals/runner/reporters/datadog.test.ts b/packages/eve/src/evals/runner/reporters/datadog.test.ts
index 1f060c065..778630e90 100644
--- a/packages/eve/src/evals/runner/reporters/datadog.test.ts
+++ b/packages/eve/src/evals/runner/reporters/datadog.test.ts
@@ -282,13 +282,7 @@ describe("Datadog", () => {
expect.objectContaining({ label: "eve_message_count", value: 1 }),
expect.objectContaining({ label: "eve_reasoning_block_count", value: 0 }),
]);
- expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).not.toEqual(
- expect.arrayContaining([
- expect.objectContaining({
- tags: expect.objectContaining({ assertion_name: expect.anything() }),
- }),
- ]),
- );
+ expect(submittedSpan).not.toHaveProperty("metadata.eveAssertionScores");
});
it("redacts target URL secrets and omits execution errors when recordErrors is disabled", async () => {
@@ -349,21 +343,24 @@ describe("Datadog", () => {
expect(experiment.submitSpan.mock.calls[0]?.[0]).not.toHaveProperty(
"metadata.eveFailedAssertions",
);
+ expect(experiment.submitSpan.mock.calls[0]?.[0]).not.toHaveProperty(
+ "metadata.eveAssertionScores",
+ );
expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toEqual(
expect.arrayContaining([
expect.objectContaining({
label: "gate_messageIncludes_private_expectation",
tags: {
- assertion_index: "1",
assertion_severity: "gate",
assertion_passed: "false",
+ assertion_count: "1",
},
}),
]),
);
});
- it("records assertion name tags and failure messages by default", async () => {
+ it("records each assertion's score and failure messages by default", async () => {
const { config, experiment } = makeConfig();
const reporter = Datadog(config);
const result = makeEvalResult({
@@ -392,26 +389,20 @@ describe("Datadog", () => {
},
],
);
- expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toEqual(
- expect.arrayContaining([
- expect.objectContaining({
- label: "gate_messageIncludes_private_expectation",
- tags: expect.objectContaining({
- assertion_name: "messageIncludes(private expectation)",
- }),
- }),
- ]),
- );
+ expect(experiment.submitSpan.mock.calls[0]?.[0]).toHaveProperty("metadata.eveAssertionScores", [
+ { name: "messageIncludes(private expectation)", severity: "gate", score: 0 },
+ ]);
});
- it("deduplicates assertion metric labels and reserves built-in labels", async () => {
+ it("combines repeated assertion names into one metric scored by the lowest member", async () => {
const { config, experiment } = makeConfig({ recordInputs: false });
const reporter = Datadog(config);
const result = makeEvalResult({
assertions: [
{ name: "same label", score: 1, severity: "soft", passed: true, errored: false },
- { name: "same@label", score: 1, severity: "soft", passed: true, errored: false },
- { name: "same label", score: 1, severity: "soft", passed: true, errored: false },
+ { name: "same@label", score: 0.5, severity: "soft", passed: false, errored: false },
+ { name: "same label", score: 0.8, severity: "soft", passed: true, errored: false },
+ { name: "same label", score: 1, severity: "gate", passed: true, errored: false },
{ name: "eve_tool_call_count", score: 1, severity: "soft", passed: true, errored: false },
],
});
@@ -420,9 +411,16 @@ describe("Datadog", () => {
await reporter.onEvalComplete(result);
expect(experiment.submitEvaluationMetrics.mock.calls[0]?.[1]).toEqual([
- expect.objectContaining({ label: "same_label" }),
- expect.objectContaining({ label: "same_label_2" }),
- expect.objectContaining({ label: "same_label_3" }),
+ {
+ label: "same_label",
+ value: 0.5,
+ tags: { assertion_severity: "soft", assertion_passed: "false", assertion_count: "3" },
+ },
+ {
+ label: "gate_same_label",
+ value: 1,
+ tags: { assertion_severity: "gate", assertion_passed: "true", assertion_count: "1" },
+ },
expect.objectContaining({ label: "eve_tool_call_count_2" }),
expect.objectContaining({ label: "eve_tool_call_count" }),
expect.objectContaining({ label: "eve_subagent_call_count" }),
diff --git a/packages/eve/src/evals/runner/reporters/datadog.ts b/packages/eve/src/evals/runner/reporters/datadog.ts
index 3809dd2e7..006e9e03a 100644
--- a/packages/eve/src/evals/runner/reporters/datadog.ts
+++ b/packages/eve/src/evals/runner/reporters/datadog.ts
@@ -1,5 +1,9 @@
import type { EveEval, EveEvalResult, EveEvalRunSummary, EveEvalTarget } from "#evals/types.js";
import type { EvalReporter } from "#evals/runner/reporters/types.js";
+import {
+ composeAssertionScoreMetadata,
+ groupAssertionScores,
+} from "#evals/runner/reporters/assertion-scores.js";
import {
getDatadogDataset,
MISSING_DATASET_API_MESSAGE,
@@ -48,7 +52,7 @@ export interface DatadogReporterConfig {
readonly recordOutputs?: boolean;
/** Include `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output` on experiment rows. Defaults to true. */
readonly recordExpectedOutputs?: boolean;
- /** Include raw assertion names as metric tags and failure messages in row metadata. Defaults to true. */
+ /** Include each assertion's name and score, and failure messages, in row metadata. Defaults to true. */
readonly recordAssertionDetails?: boolean;
/** Include execution error messages, which may contain application data. Defaults to true. */
readonly recordErrors?: boolean;
@@ -322,10 +326,7 @@ class DatadogReporter implements EvalReporter {
}
const span = await this.#experiment.submitSpan(spanInput);
- await this.#experiment.submitEvaluationMetrics(
- span,
- resolveEvaluationMetrics(result, this.#recording.recordAssertionDetails),
- );
+ await this.#experiment.submitEvaluationMetrics(span, resolveEvaluationMetrics(result));
}
}
@@ -481,6 +482,7 @@ function resolveResultMetadata(
if (failedAssertions.length > 0) {
metadata.eveFailedAssertions = failedAssertions;
}
+ metadata.eveAssertionScores = composeAssertionScoreMetadata(result.assertions);
}
if (result.result.derived.failureCode) {
metadata.eveFailureCode = result.result.derived.failureCode;
@@ -488,28 +490,22 @@ function resolveResultMetadata(
return metadata;
}
-function resolveEvaluationMetrics(
- result: EveEvalResult,
- recordAssertionDetails: boolean,
-): DatadogEvaluationMetricInput[] {
+function resolveEvaluationMetrics(result: EveEvalResult): DatadogEvaluationMetricInput[] {
const metrics: DatadogEvaluationMetricInput[] = [];
const usedLabels = new Set<string>(BUILT_IN_METRIC_LABELS);
+ const groups = groupAssertionScores(result.assertions, (assertion) =>
+ toDatadogMetricLabel(assertion.severity === "gate" ? `gate_${assertion.name}` : assertion.name),
+ );
- for (const [index, assertion] of result.assertions.entries()) {
- const rawLabel = assertion.severity === "gate" ? `gate_${assertion.name}` : assertion.name;
- const tags: Record<string, string> = {
- assertion_index: String(index + 1),
- assertion_severity: assertion.severity,
- assertion_passed: String(assertion.passed),
- };
- if (recordAssertionDetails) {
- tags.assertion_name = assertion.name;
- tags.assertion_label = rawLabel;
- }
+ for (const [label, group] of groups) {
metrics.push({
- label: reserveDatadogMetricLabel(toDatadogMetricLabel(rawLabel), usedLabels),
- value: assertion.score,
- tags,
+ label: reserveDatadogMetricLabel(label, usedLabels),
+ value: group.score,
+ tags: {
+ assertion_severity: group.assertions[0]?.severity ?? "soft",
+ assertion_passed: String(group.assertions.every((assertion) => assertion.passed)),
+ assertion_count: String(group.assertions.length),
+ },
});
}