Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(agent-eval): faster, resumable, capped eval runs with live progress (#16412)

ziyadev97e171b19858
diff --git a/packages/server/worker/test/lib/agent-eval/README.md b/packages/server/worker/test/lib/agent-eval/README.md
index a11d146b7d3..f0af4ffac5e 100644
--- a/packages/server/worker/test/lib/agent-eval/README.md
+++ b/packages/server/worker/test/lib/agent-eval/README.md
@@ -28,12 +28,16 @@ A clean working tree means baseline == candidate (it runs once and says "no prom
| Flag | Use it when |
|---|---|
| *(none)* | Re-open the cached run instantly (runs live only if there's no cache yet). No LLM cost. |
-| `--fresh` | You edited the prompt and want fresh results. |
+| `--fresh` | You want fresh results: ignores the cached run and starts a stopped run over. |
| `--candidate <path>` | A/B an explicit prompt file instead of your working tree. |
| `--baseline-model <id>` | Run the baseline arm on another OpenRouter model (default: the fixture model). |
| `--candidate-model <id>` | Run the candidate arm on another OpenRouter model (default: the fixture model). |
| `--limit <n>` | Run only the first `n` fixtures — try a model cheaply before the full suite. |
| `--only <ids>` | Run only these fixtures (comma-separated ids). |
+| `--concurrency <n>` | Evaluate `n` fixtures at once (default 4). Lower it if you hit provider rate limits. |
+| `--max-cost <usd>` | Stop starting new fixtures once this many dollars are spent (agent + judge). |
+
+Each fixture is saved to `.agent-eval/partial-run.json` as soon as it finishes. If a run stops (Ctrl+C, a crash, or the cost cap), run the same command again without `--fresh` and it picks up where it left off, skipping fixtures already done. A changed prompt, model, judge or fixture starts that work over. The cap only stops new fixtures from starting, so fixtures already running finish and the total can go over by up to `--concurrency` fixtures.
For a non-interactive pass/fail gate (scripts/CI), use `npm run agent-evals:ci`.
@@ -45,6 +49,8 @@ npm run agent-evals -- --baseline-model anthropic/claude-sonnet-4.6 --candidate-
Each arm is a prompt plus a model. With a clean working tree only the model differs; if the prompt changed too, the dashboard warns that the comparison mixes two variables. Model ids are OpenRouter slugs, run through the same chat turn (`runAgentTurn`), prompt, skills and tools as production chat.
+The judge is Jev (`typesafe/jev-1.13`) on OpenRouter's Decisions API: it answers each rubric as a probability of passing, so a failing check shows `Jev P(pass) = 0.31` instead of a written reason. In the same call Jev also picks what kind of failure it was (hallucinated, gave up, wrong action, asked too much, broke a rule, unsafe action, poor tone, other). The dashboard counts these per model on the "why it failed" line, and the fixture view tags each failed check with one. Set `CHAT_EVAL_JUDGE_MODEL` to any OpenRouter chat model to judge with that model instead.
+
The dashboard shows score and cost per arm. **Cost** is what OpenRouter reported for the agent's own calls, summed over every repeat; `—` means at least one call reported no cost. Judge cost is shown on its own line and is not counted in the comparison.
## In the reviewer
diff --git a/packages/server/worker/test/lib/agent-eval/cli/index.ts b/packages/server/worker/test/lib/agent-eval/cli/index.ts
index 20ff991645a..bceee0151b3 100644
--- a/packages/server/worker/test/lib/agent-eval/cli/index.ts
+++ b/packages/server/worker/test/lib/agent-eval/cli/index.ts
@@ -3,17 +3,26 @@ import { Command, InvalidArgumentError } from 'commander'
import { agentEvalRunner } from '../core/runner'
import { evalExportCalibration } from './export-calibration'
import { evalReview } from './review'
-import { evalRun } from './run'
+import { evalProgressView } from './progress-view'
+import { evalRun, RunParams } from './run'
import { evalStore, RunResult } from './store'
-async function runFresh({ candidatePath, baselineModelId, candidateModelId, only, limit }: RunOptions): Promise<RunResult> {
+async function runFresh(options: RunOptions): Promise<RunResult> {
if (!agentEvalRunner.hasProviderKey()) {
console.error(chalk.red(' No OpenRouter key found. Set AP_OPENROUTER_PROVISION_KEY or OPENROUTER_API_KEY.'))
console.error(chalk.dim(' (npm run agent-evals loads these from .env.dev automatically.)'))
process.exit(2)
}
console.log(chalk.dim(' running live evals (baseline vs candidate)…'))
- return evalRun.run({ candidatePath, baselineModelId, candidateModelId, only, limit, onProgress: (message) => console.log(chalk.dim(` ${message}`)) })
+ const view = evalProgressView.create()
+ const { run, remaining, spentUsd, uncosted } = await evalRun.run({ ...options, onProgress: view.update }).finally(view.stop)
+ if (uncosted.length > 0) {
+ console.log(chalk.yellow(` spend is a lower bound: a crashed repeat left the cost of ${uncosted.join(', ')} unknown`))
+ }
+ if (remaining > 0) {
+ console.log(chalk.yellow(` stopped at the $${options.maxCostUsd} cap ($${spentUsd.toFixed(2)} spent) after ${run.comparisons.length} of ${run.comparisons.length + remaining} fixtures, rerun with a higher --max-cost and without --fresh to resume`))
+ }
+ return run
}
async function reviewLoop({ run, options }: { run: RunResult, options: RunOptions }): Promise<never> {
@@ -21,7 +30,7 @@ async function reviewLoop({ run, options }: { run: RunResult, options: RunOption
for (;;) {
const outcome = await evalReview.review({ run: current })
if (outcome.kind === 'rerun') {
- current = await runFresh(options)
+ current = await runFresh({ ...options, fresh: true })
continue
}
process.exit(outcome.code)
@@ -36,6 +45,14 @@ function parsePositiveInt(value: string): number {
return parsed
}
+function parsePositiveNumber(value: string): number {
+ const parsed = Number(value)
+ if (!Number.isFinite(parsed) || parsed <= 0) {
+ throw new InvalidArgumentError('must be a number above 0')
+ }
+ return parsed
+}
+
function parseIdList(value: string): string[] {
const ids = value.split(',').map((id) => id.trim()).filter((id) => id.length > 0)
if (ids.length === 0) {
@@ -48,16 +65,18 @@ const program = new Command()
program
.name('agent-eval')
.description('Interactive chat-prompt eval — baseline vs candidate, review, and decide')
- .option('--fresh', 'evaluate live instead of reusing the cached run')
+ .option('--fresh', 'evaluate live from scratch, ignoring the cached run and any stopped run')
.option('--candidate <path>', 'A/B an explicit candidate system-prompt file (default: working tree vs HEAD)')
.option('--baseline-model <id>', 'OpenRouter model for the baseline arm (default: the fixture model)')
.option('--candidate-model <id>', 'OpenRouter model for the candidate arm (default: the fixture model)')
.option('--limit <n>', 'run only the first n fixtures', parsePositiveInt)
.option('--only <ids>', 'run only these fixtures (comma-separated ids)', parseIdList)
- .action(async (options: { fresh?: boolean, candidate?: string, baselineModel?: string, candidateModel?: string, limit?: number, only?: string[] }) => {
- const runOptions: RunOptions = { candidatePath: options.candidate, baselineModelId: options.baselineModel, candidateModelId: options.candidateModel, only: options.only, limit: options.limit }
- const wantsLiveRun = [options.baselineModel, options.candidateModel, options.limit, options.only].some((option) => option !== undefined)
- const cached = options.fresh || wantsLiveRun ? null : evalStore.readRun()
+ .option('--concurrency <n>', 'how many fixtures to evaluate at once (default: 4)', parsePositiveInt)
+ .option('--max-cost <usd>', 'stop starting new fixtures once this many dollars are spent', parsePositiveNumber)
+ .action(async (options: { fresh?: boolean, candidate?: string, baselineModel?: string, candidateModel?: string, limit?: number, only?: string[], concurrency?: number, maxCost?: number }) => {
+ const runOptions: RunOptions = { candidatePath: options.candidate, baselineModelId: options.baselineModel, candidateModelId: options.candidateModel, only: options.only, limit: options.limit, fresh: options.fresh, concurrency: options.concurrency, maxCostUsd: options.maxCost }
+ const wantsLiveRun = [options.baselineModel, options.candidateModel, options.limit, options.only, options.concurrency, options.maxCost].some((option) => option !== undefined)
+ const cached = options.fresh || wantsLiveRun || evalStore.readPartial() !== null ? null : evalStore.readRun()
if (cached) {
console.log(chalk.dim(` reviewing cached run from ${cached.at} — pass --fresh to re-run live`))
}
@@ -73,9 +92,14 @@ program
console.log(` wrote ${written} case(s), skipped ${skipped} already present — set "humanLabel" to "pass" or "fail" in each`)
})
+process.once('SIGINT', () => {
+ console.log(chalk.dim('\n interrupted, results so far are saved, rerun without --fresh to resume'))
+ agentEvalRunner.cleanupAuth().finally(() => process.exit(130)).catch(() => undefined)
+})
+
program.parseAsync(process.argv).catch((error) => {
console.error(chalk.red(` agent-eval failed: ${error instanceof Error ? error.message : String(error)}`))
process.exit(2)
})
-type RunOptions = { candidatePath?: string, baselineModelId?: string, candidateModelId?: string, only?: string[], limit?: number }
+type RunOptions = Omit<RunParams, 'onProgress'>
diff --git a/packages/server/worker/test/lib/agent-eval/cli/progress-view.ts b/packages/server/worker/test/lib/agent-eval/cli/progress-view.ts
new file mode 100644
index 00000000000..bebf13f67b4
--- /dev/null
+++ b/packages/server/worker/test/lib/agent-eval/cli/progress-view.ts
@@ -0,0 +1,110 @@
+import { clearScreenDown, cursorTo, moveCursor } from 'node:readline'
+import { stripVTControlCharacters } from 'node:util'
+import chalk from 'chalk'
+import { evalFormat } from '../core/eval-format'
+import { RunningFixture, RunProgress } from './run'
+import { PartialEntry } from './store'
+
+function create(): ProgressView {
+ const out = process.stdout
+ const live = out.isTTY
+ let latest: RunProgress | null = null
+ let drawnLines = 0
+ let frame = 0
+
+ const erase = (): void => {
+ if (drawnLines > 0) {
+ moveCursor(out, 0, -drawnLines)
+ cursorTo(out, 0)
+ clearScreenDown(out)
+ }
+ drawnLines = 0
+ }
+ const redraw = (): void => {
+ if (latest === null) {
+ return
+ }
+ erase()
+ const lines = blockLines({ progress: latest, frame }).map((line) => fitToWidth({ line, columns: out.columns }))
+ out.write(`${lines.join('\n')}\n`)
+ drawnLines = lines.length
+ }
+ const timer = live ? setInterval(() => {
+ frame += 1
+ redraw()
+ }, SPINNER_INTERVAL_MS) : null
+ timer?.unref()
+
+ return {
+ update: (progress) => {
+ if (progress.finished !== undefined) {
+ erase()
+ console.log(finishedLine({ finished: progress.finished, done: progress.done, total: progress.total }))
+ }
+ latest = progress
+ if (live) {
+ redraw()
+ }
+ },
+ stop: () => {
+ if (timer !== null) {
+ clearInterval(timer)
+ }
+ if (live && latest !== null) {
+ erase()
+ console.log(overallLine(latest))
+ }
+ },
+ }
+}
+
+function blockLines({ progress, frame }: { progress: RunProgress, frame: number }): string[] {
+ const spinner = SPINNER_FRAMES[frame % SPINNER_FRAMES.length]
+ return [
+ ...progress.running.map((fixture) => runningLine({ fixture, spinner })),
+ overallLine(progress),
+ ]
+}
+
+function runningLine({ fixture, spinner }: { fixture: RunningFixture, spinner: string }): string {
+ const seconds = Math.round((Date.now() - fixture.startedAt) / 1000)
+ return ` ${chalk.cyan(spinner)} ${fixture.id.padEnd(FIXTURE_ID_WIDTH)} ${chalk.dim(`${evalFormat.shortModelName(fixture.modelId)} · ${fixture.stage} · ${seconds}s`)}`
+}
+
+function overallLine({ done, total, resumed, spentUsd, maxCostUsd }: RunProgress): string {
+ const filled = total === 0 ? PROGRESS_BAR_WIDTH : Math.round((done / total) * PROGRESS_BAR_WIDTH)
+ const bar = chalk.cyan('█'.repeat(filled)) + chalk.dim('░'.repeat(PROGRESS_BAR_WIDTH - filled))
+ const spend = maxCostUsd === undefined ? `$${spentUsd.toFixed(2)}` : `$${spentUsd.toFixed(2)} / $${maxCostUsd.toFixed(2)}`
+ const resumedNote = resumed > 0 ? chalk.dim(` (${resumed} resumed)`) : ''
+ return ` ${bar} ${done}/${total}${resumedNote} · ${spend}`
+}
+
+function finishedLine({ finished: { comparison, spentUsd }, done, total }: { finished: PartialEntry, done: number, total: number }): string {
+ const mark = comparison.candidate.passed ? chalk.green('✓') : chalk.red('✗')
+ const cost = evalFormat.formatUsd(spentUsd)
+ const failures = evalFormat.failureSummary([comparison.candidate])
+ const why = failures === 'none' ? '' : chalk.yellow(` · ${failures}`)
+ return ` ${mark} ${comparison.id.padEnd(FIXTURE_ID_WIDTH)} ${evalFormat.deltaBadge({ delta: comparison.delta, width: 11 })} ${chalk.dim(`${cost} · ${done}/${total}`)}${why}`
+}
+
+function fitToWidth({ line, columns }: { line: string, columns: number | undefined }): string {
+ const visible = stripVTControlCharacters(line)
+ if (columns === undefined || visible.length < columns) {
+ return line
+ }
+ return `${visible.slice(0, columns - 2)}…`
+}
+
+const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏']
+const SPINNER_INTERVAL_MS = 100
+const PROGRESS_BAR_WIDTH = 24
+const FIXTURE_ID_WIDTH = 44
+
+export const evalProgressView = {
+ create,
+}
+
+type ProgressView = {
+ update: (progress: RunProgress) => void
+ stop: () => void
+}
diff --git a/packages/server/worker/test/lib/agent-eval/cli/review.ts b/packages/server/worker/test/lib/agent-eval/cli/review.ts
index 403aa16bc2a..aa2d1f57e13 100644
--- a/packages/server/worker/test/lib/agent-eval/cli/review.ts
+++ b/packages/server/worker/test/lib/agent-eval/cli/review.ts
@@ -2,6 +2,7 @@ import chalk from 'chalk'
import Table from 'cli-table3'
import inquirer from 'inquirer'
import { evalFormat } from '../core/eval-format'
+import { FAILURE_TYPES } from '../core/llm-judge'
import { EvalReportEntry } from '../core/report'
import { evalStore, FixtureComparison, RunResult } from './store'
@@ -54,7 +55,7 @@ async function browse(run: RunResult): Promise<void> {
message: 'Pick a fixture to inspect',
pageSize: 15,
choices: [
- ...run.comparisons.map((comparison) => ({ name: `${deltaBadge(comparison.delta)} ${comparison.id}`, value: comparison.id })),
+ ...run.comparisons.map((comparison) => ({ name: `${evalFormat.deltaBadge({ delta: comparison.delta })} ${comparison.id}`, value: comparison.id })),
new inquirer.Separator(),
{ name: 'Back', value: '__back' },
],
@@ -105,9 +106,10 @@ async function decide({ run, verdict }: { run: RunResult, verdict: 'proceed' | '
}
function printDashboard(run: RunResult): void {
- const table = new Table({ head: ['fixture', 'baseline', 'candidate', 'Δ', 'baseline $', 'candidate $'].map((header) => chalk.bold(header)), style: { head: [], border: [] } })
+ const names = armNames(run)
+ const table = new Table({ head: ['fixture', names.baseline, names.candidate, 'Δ', `${names.baseline} $`, `${names.candidate} $`].map((header) => chalk.bold(header)), style: { head: [], border: [] } })
for (const comparison of run.comparisons) {
- table.push([comparison.id, verdictCell(comparison.baseline.passed), verdictCell(comparison.candidate.passed), deltaBadge(comparison.delta), evalFormat.formatUsd(comparison.baseline.costUsd), evalFormat.formatUsd(comparison.candidate.costUsd)])
+ table.push([comparison.id, verdictCell(comparison.baseline.passed), verdictCell(comparison.candidate.passed), evalFormat.deltaBadge({ delta: comparison.delta }), evalFormat.formatUsd(comparison.baseline.costUsd), evalFormat.formatUsd(comparison.candidate.costUsd)])
}
const improved = run.comparisons.filter((comparison) => comparison.delta === 'improved').length
@@ -126,14 +128,24 @@ function printDashboard(run: RunResult): void {
console.log(chalk.yellow(' ⚠ both the prompt and the model differ — the comparison mixes two variables'))
}
console.log(table.toString())
- console.log(` ${chalk.dim('expected-label match')} baseline TPR ${baselineCalibration.tpr.toFixed(2)}/TNR ${baselineCalibration.tnr.toFixed(2)} candidate TPR ${candidateCalibration.tpr.toFixed(2)}/TNR ${candidateCalibration.tnr.toFixed(2)}`)
+ console.log(` ${chalk.dim('expected-label match')} ${names.baseline} TPR ${baselineCalibration.tpr.toFixed(2)}/TNR ${baselineCalibration.tnr.toFixed(2)} ${names.candidate} TPR ${candidateCalibration.tpr.toFixed(2)}/TNR ${candidateCalibration.tnr.toFixed(2)}`)
const totals = runTotals(run)
- console.log(` ${chalk.dim('score · cost')} baseline ${armSummary(totals.baseline)} candidate ${armSummary(totals.candidate)}${costDeltaSuffix({ run, totals })}`)
- console.log(` ${chalk.dim('judge cost')} baseline ${evalFormat.formatUsd(totals.baseline.judgeCostUsd)} candidate ${evalFormat.formatUsd(totals.candidate.judgeCostUsd)} ${chalk.dim('(not counted above)')}`)
- console.log(` ${headline({ improved, regressed })}`)
+ console.log(` ${chalk.dim('score · cost')} ${names.baseline} ${armSummary(totals.baseline)} ${names.candidate} ${armSummary(totals.candidate)}${costDeltaSuffix({ run, totals })}`)
+ console.log(` ${chalk.dim('judge cost')} ${names.baseline} ${evalFormat.formatUsd(totals.baseline.judgeCostUsd)} ${names.candidate} ${evalFormat.formatUsd(totals.candidate.judgeCostUsd)} ${chalk.dim('(not counted above)')}`)
+ console.log(` ${chalk.dim('why it failed')} ${names.baseline}: ${evalFormat.failureSummary(run.comparisons.map((comparison) => comparison.baseline))} ${names.candidate}: ${evalFormat.failureSummary(run.comparisons.map((comparison) => comparison.candidate))}`)
+ console.log(` ${headline({ improved, regressed, names })}`)
console.log('')
}
+function armNames(run: RunResult): ArmNames {
+ if (run.baselineModelId === run.candidateModelId) {
+ return { baseline: 'baseline', candidate: 'candidate' }
+ }
+ const baseline = evalFormat.shortModelName(run.baselineModelId)
+ const candidate = evalFormat.shortModelName(run.candidateModelId)
+ return baseline === candidate ? { baseline: run.baselineModelId, candidate: run.candidateModelId } : { baseline, candidate }
+}
+
function armSummary(totals: ArmTotals): string {
return `${totals.passed}/${totals.total} · ${evalFormat.formatUsd(totals.costUsd)}`
}
@@ -165,11 +177,12 @@ function armTotals(entries: EvalReportEntry[]): ArmTotals {
function printFixtureDetail({ run, comparison }: { run: RunResult, comparison: FixtureComparison }): void {
const { baseline, candidate } = comparison
+ const names = armNames(run)
console.log('')
- console.log(` ${chalk.bold(comparison.id)} ${deltaBadge(comparison.delta)}`)
+ console.log(` ${chalk.bold(comparison.id)} ${evalFormat.deltaBadge({ delta: comparison.delta })}`)
console.log(` ${chalk.dim(evalFormat.truncate({ text: candidate.description, max: 108 }))}`)
- const table = new Table({ head: ['check', 'baseline', 'candidate'].map((header) => chalk.bold(header)), style: { head: [], border: [] } })
+ const table = new Table({ head: ['check', names.baseline, names.candidate].map((header) => chalk.bold(header)), style: { head: [], border: [] } })
baseline.assertions.forEach((assertion, index) => {
table.push([`assert · ${assertion.label}`, flipCell(assertion.pass, candidate.assertions[index]?.pass ?? assertion.pass, true), flipCell(candidate.assertions[index]?.pass ?? assertion.pass, assertion.pass, false)])
})
@@ -180,15 +193,16 @@ function printFixtureDetail({ run, comparison }: { run: RunResult, comparison: F
const failedJudge = candidate.judge.filter((verdict) => !verdict.pass)
if (failedJudge.length > 0) {
- console.log(` ${chalk.dim('candidate judge notes')}`)
+ console.log(` ${chalk.dim(`${names.candidate} judge notes`)}`)
for (const verdict of failedJudge) {
- console.log(` ${chalk.red('✗')} ${verdict.dimension}: ${chalk.dim(evalFormat.truncate({ text: verdict.reason, max: 96 }))}`)
+ const failure = verdict.failure === undefined ? '' : ` ${chalk.yellow(`[${FAILURE_TYPES[verdict.failure].label}]`)}`
+ console.log(` ${chalk.red('✗')} ${verdict.dimension}${failure}: ${chalk.dim(evalFormat.truncate({ text: verdict.reason, max: 96 }))}`)
}
}
- printTranscript('baseline transcript', baseline.transcript)
+ printTranscript(`${names.baseline} transcript`, baseline.transcript)
if (isChanged(run)) {
- printTranscript('candidate transcript', candidate.transcript)
+ printTranscript(`${names.candidate} transcript`, candidate.transcript)
}
}
@@ -213,24 +227,14 @@ function flipCell(pass: boolean, other: boolean, isBaseline: boolean): string {
return isRegression ? chalk.red.bold(mark) : chalk.green.bold(mark)
}
-function deltaBadge(delta: FixtureComparison['delta']): string {
- if (delta === 'improved') {
- return chalk.green('▲ improved')
- }
- if (delta === 'regressed') {
- return chalk.red('▼ regressed')
- }
- return chalk.dim('= same')
-}
-
-function headline({ improved, regressed }: { improved: number, regressed: number }): string {
+function headline({ improved, regressed, names }: { improved: number, regressed: number, names: ArmNames }): string {
if (regressed > 0) {
- return chalk.red.bold(`candidate regressed ${regressed} fixture(s)${improved > 0 ? `, improved ${improved}` : ''}`)
+ return chalk.red.bold(`${names.candidate} regressed ${regressed} fixture(s)${improved > 0 ? `, improved ${improved}` : ''}`)
}
if (improved > 0) {
- return chalk.green.bold(`candidate improved ${improved} fixture(s), no regressions`)
+ return chalk.green.bold(`${names.candidate} improved ${improved} fixture(s), no regressions`)
}
- return chalk.dim('no change between baseline and candidate')
+ return chalk.dim(`no change between ${names.baseline} and ${names.candidate}`)
}
export const evalReview = {
@@ -249,3 +253,5 @@ type ArmTotals = {
}
type RunTotals = { baseline: ArmTotals, candidate: ArmTotals }
+
+type ArmNames = { baseline: string, candidate: string }
diff --git a/packages/server/worker/test/lib/agent-eval/cli/run.test.ts b/packages/server/worker/test/lib/agent-eval/cli/run.test.ts
new file mode 100644
index 00000000000..935fd7d4a4b
--- /dev/null
+++ b/packages/server/worker/test/lib/agent-eval/cli/run.test.ts
@@ -0,0 +1,186 @@
+import { AIProviderName } from '@activepieces/core-utils'
+import { beforeEach, describe, expect, it, vi } from 'vitest'
+import { ChatEvalFixture } from '../core/fixture'
+import { EvalReportEntry } from '../core/report'
+import { evalRun, RunProgress } from './run'
+import { PartialRun } from './store'
+
+const state = vi.hoisted((): TestState => ({
+ fixtures: [],
+ partial: null,
+ lastRunWrites: 0,
+ inFlight: 0,
+ maxInFlight: 0,
+ evaluated: [],
+ systemPrompt: 'prompt',
+ costPerFixture: 1,
+ crashedRepeatIds: [],
+}))
+
+vi.mock('../core/fixtures-loader', () => ({ evalFixtures: { load: () => state.fixtures } }))
+
+vi.mock('../core/prompts', () => ({
+ evalPrompts: {
+ resolveSources: () => ({
+ baseline: { label: 'baseline', systemPrompt: state.systemPrompt, guides: {} },
+ candidate: { label: 'candidate', systemPrompt: state.systemPrompt, guides: {} },
+ changed: false,
+ }),
+ },
+}))
+
+vi.mock('../core/runner', () => ({
+ agentEvalRunner: {
+ repeatsFromEnv: () => 1,
+ judgeModelId: () => 'judge',
+ cleanupAuth: async () => undefined,
+ evaluateFixture: async ({ fixture, onStage }: { fixture: { id: string }, onStage?: (stage: string) => void }) => {
+ onStage?.('agent turn, step 1')
+ state.inFlight += 1
+ state.maxInFlight = Math.max(state.maxInFlight, state.inFlight)
+ await new Promise((resolve) => setTimeout(resolve, 5))
+ state.inFlight -= 1
+ state.evaluated.push(fixture.id)
+ return entry({ id: fixture.id, costUsd: state.costPerFixture, crashedRepeat: state.crashedRepeatIds.includes(fixture.id) })
+ },
+ },
+}))
+
+vi.mock('./store', () => ({
+ evalStore: {
+ runFormatVersion: 2,
+ readPartial: () => state.partial,
+ writePartial: (partial: PartialRun) => {
+ state.partial = partial
+ },
+ clearPartial: () => {
+ state.partial = null
+ },
+ writeRun: (run: object) => {
+ state.lastRunWrites += 1
+ return { ...run, version: 2 }
+ },
+ },
+}))
+
+describe('evalRun.run', () => {
+ beforeEach(() => {
+ Object.assign(state, { fixtures: ['a', 'b', 'c', 'd', 'e', 'f'].map(fixture), partial: null, lastRunWrites: 0, inFlight: 0, maxInFlight: 0, evaluated: [], systemPrompt: 'prompt', costPerFixture: 1, crashedRepeatIds: [] })
+ })
+
+ it('never runs more fixtures at once than the concurrency limit', async () => {
+ const outcome = await evalRun.run({ concurrency: 2 })
+
+ expect(state.maxInFlight).toBe(2)
+ expect(outcome.remaining).toBe(0)
+ expect(outcome.run.comparisons.map((comparison) => comparison.id)).toEqual(['a', 'b', 'c', 'd', 'e', 'f'])
+ expect(state.lastRunWrites).toBe(1)
+ expect(state.partial).toBeNull()
+ })
+
+ it('stops starting fixtures at the cost cap and keeps what finished for a resume', async () => {
+ const outcome = await evalRun.run({ concurrency: 1, maxCostUsd: 2 })
+
+ expect(state.evaluated).toEqual(['a', 'b'])
+ expect(outcome.remaining).toBe(4)
+ expect(outcome.spentUsd).toBe(2)
+ expect(state.lastRunWrites).toBe(0)
+ expect(state.partial?.entries.map((entry) => entry.comparison.id)).toEqual(['a', 'b'])
+
+ state.evaluated = []
+ const progress: RunProgress[] = []
+ const resumed = await evalRun.run({ concurrency: 1, onProgress: (event) => progress.push(event) })
+
+ expect(state.evaluated).toEqual(['c', 'd', 'e', 'f'])
+ expect(resumed.remaining).toBe(0)
+ expect(resumed.run.comparisons).toHaveLength(6)
+ expect(progress[0]).toMatchObject({ done: 2, total: 6, resumed: 2, spentUsd: 2, running: [] })
+ expect(progress[1].running.map((fixture) => [fixture.id, fixture.modelId, fixture.stage])).toEqual([['c', 'model', 'agent turn, step 1']])
+ expect(progress.filter((event) => event.finished !== undefined).map((event) => event.finished?.comparison.id)).toEqual(['c', 'd', 'e', 'f'])
+ expect(progress.at(-1)).toMatchObject({ done: 6, total: 6, spentUsd: 6, running: [] })
+ })
+
+ it('starts a stopped run over when asked for a fresh run', async () => {
+ await evalRun.run({ concurrency: 1, maxCostUsd: 2 })
+ state.evaluated = []
+
+ await evalRun.run({ concurrency: 1, fresh: true })
+
+ expect(state.evaluated).toEqual(['a', 'b', 'c', 'd', 'e', 'f'])
+ })
+
+ it('runs everything again when the prompt changed since the stopped run', async () => {
+ await evalRun.run({ concurrency: 1, maxCostUsd: 2 })
+ state.evaluated = []
+ state.systemPrompt = 'edited prompt'
+
+ await evalRun.run({ concurrency: 1 })
+
+ expect(state.evaluated).toEqual(['a', 'b', 'c', 'd', 'e', 'f'])
+ })
+
+ it('runs an edited fixture again on resume', async () => {
+ await evalRun.run({ concurrency: 1, maxCostUsd: 2 })
+ state.evaluated = []
+ state.fixtures = [{ ...fixture('a'), description: 'edited' }, ...state.fixtures.slice(1)]
+
+ await evalRun.run({ concurrency: 1 })
+
+ expect(state.evaluated).toEqual(['a', 'c', 'd', 'e', 'f'])
+ })
+
+ it('counts the known spend of a fixture whose repeat crashed toward the cap', async () => {
+ state.crashedRepeatIds = ['a']
+
+ const outcome = await evalRun.run({ concurrency: 1, maxCostUsd: 2 })
+
+ expect(state.evaluated).toEqual(['a', 'b'])
+ expect(outcome.spentUsd).toBe(2)
+ expect(outcome.uncosted).toEqual(['a'])
+ })
+
+ it('drops the stopped run as soon as a fresh run starts, even if it is interrupted', async () => {
+ await evalRun.run({ concurrency: 1, maxCostUsd: 2 })
+ state.evaluated = []
+
+ await evalRun.run({ concurrency: 1, fresh: true, maxCostUsd: 0 })
+ await evalRun.run({ concurrency: 1 })
+
+ expect(state.evaluated).toEqual(['a', 'b', 'c', 'd', 'e', 'f'])
+ })
+
+ it('keeps saved results outside a subset run for the next full run', async () => {
+ await evalRun.run({ concurrency: 1, maxCostUsd: 2 })
+ state.evaluated = []
+
+ const subset = await evalRun.run({ concurrency: 1, only: ['c'] })
+
+ expect(subset.remaining).toBe(0)
+ expect(state.partial?.entries.map((entry) => entry.comparison.id)).toEqual(['a', 'b'])
+
+ await evalRun.run({ concurrency: 1 })
+
+ expect(state.evaluated).toEqual(['c', 'c', 'd', 'e', 'f'])
+ expect(state.partial).toBeNull()
+ })
+})
+
+function fixture(id: string): ChatEvalFixture {
+ return { id, kind: 'regression', description: id, initialMessages: [], userTurns: [], recordedToolCalls: [], model: { provider: AIProviderName.OPENROUTER, modelId: 'model', tier: { id: 'tier', thinkingBudget: 0, modelId: 'model' } }, assertions: [], judge: [] }
+}
+
+function entry({ id, costUsd, crashedRepeat }: { id: string, costUsd: number, crashedRepeat: boolean }): EvalReportEntry {
+ return { id, kind: 'regression', description: id, provider: 'openrouter', modelId: 'model', judgeModelId: 'judge', runs: 1, passes: 1, passed: true, assertions: [], judge: [], transcript: '', costUsd: crashedRepeat ? null : costUsd, knownCostUsd: costUsd, judgeCostUsd: 0, runVerdicts: [] }
+}
+
+type TestState = {
+ fixtures: ChatEvalFixture[]
+ partial: PartialRun | null
+ lastRunWrites: number
+ inFlight: number
+ maxInFlight: number
+ evaluated: string[]
+ systemPrompt: string
+ costPerFixture: number
+ crashedRepeatIds: string[]
+}
diff --git a/packages/server/worker/test/lib/agent-eval/cli/run.ts b/packages/server/worker/test/lib/agent-eval/cli/run.ts
index 5cd33d2ed5e..039bd1cf01d 100644
--- a/packages/server/worker/test/lib/agent-eval/cli/run.ts
+++ b/packages/server/worker/test/lib/agent-eval/cli/run.ts
@@ -1,8 +1,12 @@
+import { spreadIfDefined } from '@activepieces/core-utils'
+import { cryptoUtils } from '@activepieces/server-utils'
+import pLimit from 'p-limit'
import { ChatEvalFixture } from '../core/fixture'
import { evalFixtures } from '../core/fixtures-loader'
-import { evalPrompts } from '../core/prompts'
+import { evalPrompts, PromptSources } from '../core/prompts'
import { agentEvalRunner } from '../core/runner'
-import { evalStore, FixtureComparison, RunResult } from './store'
+import { EvalReportEntry } from '../core/report'
+import { evalStore, FixtureComparison, PartialEntry, RunResult } from './store'
function deltaOf({ baselinePassed, candidatePassed }: { baselinePassed: boolean, candidatePassed: boolean }): FixtureComparison['delta'] {
if (candidatePassed && !baselinePassed) {
@@ -14,31 +18,50 @@ function deltaOf({ baselinePassed, candidatePassed }: { baselinePassed: boolean,
return 'same'
}
-async function run({ candidatePath, baselineModelId, candidateModelId, only, limit, onProgress }: { candidatePath?: string, baselineModelId?: string, candidateModelId?: string, only?: string[], limit?: number, onProgress?: (message: string) => void }): Promise<RunResult> {
+async function run({ candidatePath, baselineModelId, candidateModelId, only, limit, fresh = false, concurrency = DEFAULT_CONCURRENCY, maxCostUsd, onProgress }: RunParams): Promise<RunOutcome> {
const fixtures = selectFixtures({ fixtures: evalFixtures.load(), only, limit })
const sources = evalPrompts.resolveSources({ candidatePath })
const first = fixtures[0]
- const comparisons: FixtureComparison[] = []
+ const runKey = hashOf({ sources, baselineModelId, candidateModelId, repeats: agentEvalRunner.repeatsFromEnv(), judgeModelId: agentEvalRunner.judgeModelId() })
+ if (fresh) {
+ evalStore.clearPartial()
+ }
+ const saved = savedEntries({ runKey })
+ const selected = new Set(fixtures.map((fixture) => fixture.id))
+ const unselected = saved.filter((entry) => !selected.has(entry.comparison.id))
+ const done = new Map(resumableEntries({ entries: saved, fixtures }).map((entry) => [entry.comparison.id, entry]))
+ const resumed = done.size
+ const running = new Map<string, RunningFixture>()
+ let spentUsd = [...done.values()].reduce((total, entry) => total + entry.spentUsd, 0)
+ const report = (finished?: PartialEntry): void => onProgress?.({ done: done.size, total: fixtures.length, resumed, spentUsd, maxCostUsd, running: [...running.values()], ...spreadIfDefined('finished', finished) })
+ const pool = pLimit(concurrency)
+ report()
try {
- for (const fixture of fixtures) {
- const baselineModel = baselineModelId ?? fixture.model.modelId
- const candidateModel = candidateModelId ?? fixture.model.modelId
- onProgress?.(`evaluating ${fixture.id} · ${sources.baseline.label} · ${baselineModel}`)
- const baseline = await agentEvalRunner.evaluateFixture({ fixture, systemPrompt: sources.baseline.systemPrompt, guides: sources.baseline.guides, modelId: baselineModel })
- let candidate = baseline
- if (sources.changed || baselineModel !== candidateModel) {
- onProgress?.(`evaluating ${fixture.id} · ${sources.candidate.label} · ${candidateModel}`)
- candidate = await agentEvalRunner.evaluateFixture({ fixture, systemPrompt: sources.candidate.systemPrompt, guides: sources.candidate.guides, modelId: candidateModel })
+ await Promise.all(fixtures.filter((fixture) => !done.has(fixture.id)).map((fixture) => pool(async () => {
+ if (maxCostUsd !== undefined && spentUsd >= maxCostUsd) {
+ return
+ }
+ const startedAt = Date.now()
+ const onStage = ({ modelId, stage }: { modelId: string, stage: string }): void => {
+ running.set(fixture.id, { id: fixture.id, modelId, stage, startedAt })
+ report()
}
- comparisons.push({ id: fixture.id, baseline, candidate, delta: deltaOf({ baselinePassed: baseline.passed, candidatePassed: candidate.passed }) })
- }
+ const entry = await evaluatePair({ fixture, sources, baselineModelId, candidateModelId, onStage }).finally(() => running.delete(fixture.id))
+ done.set(fixture.id, entry)
+ spentUsd += entry.spentUsd
+ evalStore.writePartial({ runKey, entries: [...unselected, ...done.values()] })
+ report(entry)
+ })))
}
finally {
await agentEvalRunner.cleanupAuth()
}
- return evalStore.writeRun({
+ const comparisons = fixtures.flatMap((fixture) => done.get(fixture.id)?.comparison ?? [])
+ const remaining = fixtures.length - comparisons.length
+ const uncosted = fixtures.filter((fixture) => done.get(fixture.id)?.costComplete === false).map((fixture) => fixture.id)
+ const result = {
at: new Date().toISOString(),
provider: first?.model.provider ?? 'openrouter',
baselineModelId: baselineModelId ?? first?.model.modelId ?? '',
@@ -47,7 +70,53 @@ async function run({ candidatePath, baselineModelId, candidateModelId, only, lim
candidateLabel: sources.candidate.label,
promptChanged: sources.changed,
comparisons,
- })
+ }
+ if (remaining > 0) {
+ return { run: { ...result, version: evalStore.runFormatVersion }, remaining, spentUsd, uncosted }
+ }
+ const written = evalStore.writeRun(result)
+ if (unselected.length > 0) {
+ evalStore.writePartial({ runKey, entries: unselected })
+ }
+ else {
+ evalStore.clearPartial()
+ }
+ return { run: written, remaining, spentUsd, uncosted }
+}
+
+async function evaluatePair({ fixture, sources, baselineModelId, candidateModelId, onStage }: { fixture: ChatEvalFixture, sources: PromptSources, baselineModelId?: string, candidateModelId?: string, onStage: (stage: { modelId: string, stage: string }) => void }): Promise<PartialEntry> {
+ const baselineModel = baselineModelId ?? fixture.model.modelId
+ const candidateModel = candidateModelId ?? fixture.model.modelId
+ const baseline = await agentEvalRunner.evaluateFixture({ fixture, systemPrompt: sources.baseline.systemPrompt, guides: sources.baseline.guides, modelId: baselineModel, onStage: (stage) => onStage({ modelId: baselineModel, stage }) })
+ let candidate = baseline
+ if (sources.changed || baselineModel !== candidateModel) {
+ candidate = await agentEvalRunner.evaluateFixture({ fixture, systemPrompt: sources.candidate.systemPrompt, guides: sources.candidate.guides, modelId: candidateModel, onStage: (stage) => onStage({ modelId: candidateModel, stage }) })
+ }
+ const arms = candidate === baseline ? [baseline] : [baseline, candidate]
+ return {
+ fixtureHash: hashOf(fixture),
+ spentUsd: arms.reduce((total, arm) => total + spentOn(arm), 0),
+ costComplete: arms.every((arm) => arm.costUsd !== null && arm.judgeCostUsd !== null),
+ comparison: { id: fixture.id, baseline, candidate, delta: deltaOf({ baselinePassed: baseline.passed, candidatePassed: candidate.passed }) },
+ }
+}
+
+function savedEntries({ runKey }: { runKey: string }): PartialEntry[] {
+ const partial = evalStore.readPartial()
+ return partial?.runKey === runKey ? partial.entries : []
+}
+
+function resumableEntries({ entries, fixtures }: { entries: PartialEntry[], fixtures: ChatEvalFixture[] }): PartialEntry[] {
+ const hashes = new Map(fixtures.map((fixture) => [fixture.id, hashOf(fixture)]))
+ return entries.filter((entry) => hashes.get(entry.comparison.id) === entry.fixtureHash)
+}
+
+function spentOn(arm: EvalReportEntry): number {
+ return arm.knownCostUsd + (arm.judgeCostUsd ?? 0)
+}
+
+function hashOf(value: unknown): string {
+ return cryptoUtils.hashSHA256(JSON.stringify(value))
}
function selectFixtures({ fixtures, only, limit }: { fixtures: ChatEvalFixture[], only?: string[], limit?: number }): ChatEvalFixture[] {
@@ -59,6 +128,44 @@ function selectFixtures({ fixtures, only, limit }: { fixtures: ChatEvalFixture[]
return limit === undefined ? picked : picked.slice(0, limit)
}
+const DEFAULT_CONCURRENCY = 4
+
export const evalRun = {
run,
}
+
+export type RunParams = {
+ candidatePath?: string
+ baselineModelId?: string
+ candidateModelId?: string
+ only?: string[]
+ limit?: number
+ fresh?: boolean
+ concurrency?: number
+ maxCostUsd?: number
+ onProgress?: (progress: RunProgress) => void
+}
+
+export type RunProgress = {
+ done: number
+ total: number
+ resumed: number
+ spentUsd: number
+ maxCostUsd?: number
+ running: RunningFixture[]
+ finished?: PartialEntry
+}
+
+export type RunningFixture = {
+ id: string
+ modelId: string
+ stage: string
+ startedAt: number
+}
+
+export type RunOutcome = {
+ run: RunResult
+ remaining: number
+ spentUsd: number
+ uncosted: string[]
+}
diff --git a/packages/server/worker/test/lib/agent-eval/cli/store.ts b/packages/server/worker/test/lib/agent-eval/cli/store.ts
index db6550aad99..506c215a38e 100644
--- a/packages/server/worker/test/lib/agent-eval/cli/store.ts
+++ b/packages/server/worker/test/lib/agent-eval/cli/store.ts
@@ -1,11 +1,13 @@
-import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'
+import { existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from 'node:fs'
import path from 'node:path'
+import { FixtureDelta } from '../core/eval-format'
import { EvalReportEntry } from '../core/report'
import { repoRoot } from '../core/repo-root'
const STORE_DIR = path.join(repoRoot, '.agent-eval')
const RUN_FILE = path.join(STORE_DIR, 'last-run.json')
const DECISIONS_FILE = path.join(STORE_DIR, 'decisions.json')
+const PARTIAL_FILE = path.join(STORE_DIR, 'partial-run.json')
const RUN_FORMAT_VERSION = 2
function writeRun(run: Omit<RunResult, 'version'>): RunResult {
@@ -23,6 +25,21 @@ function readRun(): RunResult | null {
return cached.version === RUN_FORMAT_VERSION ? cached : null
}
+function writePartial(partial: PartialRun): void {
+ mkdirSync(STORE_DIR, { recursive: true })
+ const tempFile = `${PARTIAL_FILE}.tmp`
+ writeFileSync(tempFile, JSON.stringify(partial, null, 2))
+ renameSync(tempFile, PARTIAL_FILE)
+}
+
+function readPartial(): PartialRun | null {
+ return existsSync(PARTIAL_FILE) ? JSON.parse(readFileSync(PARTIAL_FILE, 'utf-8')) : null
+}
+
+function clearPartial(): void {
+ rmSync(PARTIAL_FILE, { force: true })
+}
+
function appendDecision(decision: ReviewDecision): void {
mkdirSync(STORE_DIR, { recursive: true })
const existing: ReviewDecision[] = existsSync(DECISIONS_FILE) ? JSON.parse(readFileSync(DECISIONS_FILE, 'utf-8')) : []
@@ -32,15 +49,31 @@ function appendDecision(decision: ReviewDecision): void {
export const evalStore = {
writeRun,
readRun,
+ writePartial,
+ readPartial,
+ clearPartial,
appendDecision,
decisionsFile: DECISIONS_FILE,
+ runFormatVersion: RUN_FORMAT_VERSION,
}
export type FixtureComparison = {
id: string
baseline: EvalReportEntry
candidate: EvalReportEntry
- delta: 'improved' | 'regressed' | 'same'
+ delta: FixtureDelta
+}
+
+export type PartialRun = {
+ runKey: string
+ entries: PartialEntry[]
+}
+
+export type PartialEntry = {
+ fixtureHash: string
+ spentUsd: number
+ costComplete: boolean
+ comparison: FixtureComparison
}
export type RunResult = {
diff --git a/packages/server/worker/test/lib/agent-eval/core/cost-capture.test.ts b/packages/server/worker/test/lib/agent-eval/core/cost-capture.test.ts
index 04298cd5168..6466aaa252e 100644
--- a/packages/server/worker/test/lib/agent-eval/core/cost-capture.test.ts
+++ b/packages/server/worker/test/lib/agent-eval/core/cost-capture.test.ts
@@ -21,6 +21,21 @@ describe('costCapture.track', () => {
expect(tracked.costUsd()).toBeNull()
})
+ it('keeps the cost known when a call is rejected before the provider answers', async () => {
+ const answered = mockModel({ cost: 0.25 })
+ const tracked = costCapture.track({ model: new MockLanguageModelV3({
+ doGenerate: async () => {
+ throw new Error('bad request')
+ },
+ doStream: answered.doStream,
+ }) })
+
+ await generateText({ model: tracked.model, prompt: 'hi', maxRetries: 0 }).catch(() => undefined)
+ await streamText({ model: tracked.model, prompt: 'hi' }).consumeStream()
+
+ expect(tracked.costUsd()).toBe(0.25)
+ })
+
it('asks OpenRouter to include usage cost on every call', async () => {
const model = mockModel({ cost: 0.1 })
const tracked = costCapture.track({ model })
diff --git a/packages/server/worker/test/lib/agent-eval/core/cost-capture.ts b/packages/server/worker/test/lib/agent-eval/core/cost-capture.ts
index 19871ba44a0..0032cf1a1e1 100644
--- a/packages/server/worker/test/lib/agent-eval/core/cost-capture.ts
+++ b/packages/server/worker/test/lib/agent-eval/core/cost-capture.ts
@@ -6,14 +6,8 @@ function track({ model }: { model: LanguageModel }): CostTrackedModel {
if (typeof model === 'string') {
return { model, costUsd: () => null }
}
- const ledger = { calls: 0, reported: 0, totalUsd: 0 }
- const record = (providerMetadata: SharedV4ProviderMetadata | undefined): void => {
- const cost = activepiecesAiCost.openRouterCostOf(providerMetadata)
- if (cost !== undefined) {
- ledger.reported += 1
- ledger.totalUsd += cost
- }
- }
+ const ledger = createLedger()
+ const record = (providerMetadata: SharedV4ProviderMetadata | undefined): void => ledger.record(activepiecesAiCost.openRouterCostOf(providerMetadata))
const wrapped = wrapLanguageModel({
model,
middleware: {
@@ -25,14 +19,14 @@ function track({ model }: { model: LanguageModel }): CostTrackedModel {
},
}),
wrapGenerate: async ({ doGenerate }) => {
- ledger.calls += 1
const result = await doGenerate()
+ ledger.countCall()
record(result.providerMetadata)
return result
},
wrapStream: async ({ doStream }) => {
- ledger.calls += 1
const { stream, ...rest } = await doStream()
+ ledger.countCall()
return {
...rest,
stream: stream.pipeThrough(new TransformStream<LanguageModelV4StreamPart, LanguageModelV4StreamPart>({
@@ -49,15 +43,40 @@ function track({ model }: { model: LanguageModel }): CostTrackedModel {
})
return {
model: wrapped,
- costUsd: () => ledger.reported === ledger.calls ? ledger.totalUsd : null,
+ costUsd: ledger.costUsd,
+ }
+}
+
+function createLedger(): CostLedger {
+ let calls = 0
+ let reported = 0
+ let totalUsd = 0
+ return {
+ countCall: () => {
+ calls += 1
+ },
+ record: (cost) => {
+ if (cost !== undefined) {
+ reported += 1
+ totalUsd += cost
+ }
+ },
+ costUsd: () => reported === calls ? totalUsd : null,
}
}
export const costCapture = {
track,
+ createLedger,
}
export type CostTrackedModel = {
model: LanguageModel
costUsd: () => number | null
}
+
+export type CostLedger = {
+ countCall: () => void
+ record: (costUsd: number | undefined) => void
+ costUsd: () => number | null
+}
diff --git a/packages/server/worker/test/lib/agent-eval/core/eval-format.ts b/packages/server/worker/test/lib/agent-eval/core/eval-format.ts
index 3040b4609f6..bced974fcdf 100644
--- a/packages/server/worker/test/lib/agent-eval/core/eval-format.ts
+++ b/packages/server/worker/test/lib/agent-eval/core/eval-format.ts
@@ -1,3 +1,6 @@
+import { groupBy } from '@activepieces/core-utils'
+import chalk from 'chalk'
+import { FAILURE_TYPES } from './llm-judge'
import type { EvalReportEntry } from './report'
function expectedLabelMatch(entries: EvalReportEntry[]): { tpr: number, tnr: number } {
@@ -28,6 +31,28 @@ function rates(verdicts: LabelVerdict[]): AgreementRates {
}
}
+function shortModelName(modelId: string): string {
+ return modelId.split('/').pop() || modelId
+}
+
+function failureSummary(entries: EvalReportEntry[]): string {
+ const failures = entries.flatMap((entry) => entry.judge.flatMap((verdict) => verdict.failure === undefined ? [] : [verdict.failure]))
+ const parts = Object.values(groupBy(failures, (failure) => failure))
+ .sort((a, b) => b.length - a.length)
+ .map((group) => `${FAILURE_TYPES[group[0]].label} ${group.length}`)
+ return parts.length === 0 ? 'none' : parts.join(', ')
+}
+
+function deltaBadge({ delta, width = 0 }: { delta: FixtureDelta, width?: number }): string {
+ if (delta === 'improved') {
+ return chalk.green('▲ improved'.padEnd(width))
+ }
+ if (delta === 'regressed') {
+ return chalk.red('▼ regressed'.padEnd(width))
+ }
+ return chalk.dim('= same'.padEnd(width))
+}
+
function truncate({ text, max }: { text: string, max: number }): string {
const collapsed = text.replace(/\s+/g, ' ').trim()
return collapsed.length > max ? `${collapsed.slice(0, max - 1)}…` : collapsed
@@ -56,6 +81,9 @@ function costDelta({ baseline, candidate }: { baseline: number | null, candidate
}
export const evalFormat = {
+ failureSummary,
+ deltaBadge,
+ shortModelName,
expectedLabelMatch,
judgeAgreement,
truncate,
@@ -79,3 +107,5 @@ type LabelVerdict = {
judgePass: boolean
draft: boolean
}
+
+export type FixtureDelta = 'improved' | 'regressed' | 'same'
diff --git a/packages/server/worker/test/lib/agent-eval/core/llm-judge.test.ts b/packages/server/worker/test/lib/agent-eval/core/llm-judge.test.ts
new file mode 100644
index 00000000000..a83de6a4aa6
--- /dev/null
+++ b/packages/server/worker/test/lib/agent-eval/core/llm-judge.test.ts
@@ -0,0 +1,93 @@
+import { AIProviderName } from '@activepieces/core-utils'
+import { afterEach, describe, expect, it, MockInstance, vi } from 'vitest'
+import { llmJudge } from './llm-judge'
+
+describe('llmJudge.create with a Jev model', () => {
+ afterEach(() => {
+ vi.restoreAllMocks()
+ })
+
+ it('asks the decisions endpoint about the rubric with the caller key', async () => {
+ const fetchSpy = respondWith({ body: { answers: { pass: { noul: 0.9 } }, usage: { cost: 0.01 } } })
+
+ await jevJudge().judge(QUESTION)
+
+ const [url, init] = fetchSpy.mock.calls[0]
+ expect(url).toBe('https://openrouter.ai/api/alpha/decisions')
+ expect(init?.method).toBe('POST')
+ expect(new Headers(init?.headers).get('Authorization')).toBe('Bearer test-key')
+ const sent = JSON.parse(String(init?.body))
+ expect(sent.model).toBe(JEV_MODEL)
+ expect(sent.state.text).toBe(QUESTION.transcript)
+ expect(sent.questions.pass.instructions).toContain(QUESTION.rubric)
+ expect(sent.questions.pass.instructions).toContain(QUESTION.dimension)
+ })
+
+ it('passes a likely answer and leaves the failure type out', async () => {
+ respondWith({ body: { answers: { pass: { noul: 0.8 }, failure: { choice: 'gave_up' } } } })
+
+ const verdict = await jevJudge().judge(QUESTION)
+
+ expect(verdict).toEqual({ pass: true, reason: 'Jev P(pass) = 0.80' })
+ })
+
+ it('fails an unlikely answer with the failure type it chose', async () => {
+ respondWith({ body: { answers: { pass: { noul: 0.2 }, failure: { choice: 'hallucinated' } } } })
+
+ const verdict = await jevJudge().judge(QUESTION)
+
+ expect(verdict).toEqual({ pass: false, reason: 'Jev P(pass) = 0.20', failure: 'hallucinated' })
+ })
+
+ it('fails without a failure type when the choice is not a known one', async () => {
+ respondWith({ body: { answers: { pass: { noul: 0.2 }, failure: { choice: 'made_up_kind' } } } })
+
+ const verdict = await jevJudge().judge(QUESTION)
+
+ expect(verdict).toEqual({ pass: false, reason: 'Jev P(pass) = 0.20' })
+ })
+
+ it('throws when the response has no answer', async () => {
+ respondWith({ body: { answers: {} } })
+
+ await expect(jevJudge().judge(QUESTION)).rejects.toThrow('[Jev judge] the response had no answer')
+ })
+
+ it('throws with the status on an HTTP error', async () => {
+ respondWith({ body: 'rate limited', status: 429 })
+
+ await expect(jevJudge().judge(QUESTION)).rejects.toThrow('[Jev judge] HTTP 429: rate limited')
+ })
+
+ it('sums the reported cost across calls', async () => {
+ respondWith({ body: { answers: { pass: { noul: 0.9 } }, usage: { cost: 0.25 } } })
+ const judge = jevJudge()
+
+ await judge.judge(QUESTION)
+ await judge.judge(QUESTION)
+
+ expect(judge.costUsd()).toBe(0.5)
+ })
+
+ it('reports an unknown cost when a call reports none', async () => {
+ respondWith({ body: { answers: { pass: { noul: 0.9 } } } })
+ const judge = jevJudge()
+
+ await judge.judge(QUESTION)
+
+ expect(judge.costUsd()).toBeNull()
+ })
+})
+
+function jevJudge(): ReturnType<typeof llmJudge.create> {
+ return llmJudge.create({ provider: AIProviderName.OPENROUTER, modelId: JEV_MODEL, auth: { apiKey: 'test-key' } })
+}
+
+function respondWith({ body, status = 200 }: { body: unknown, status?: number }): MockInstance<typeof fetch> {
+ const text = typeof body === 'string' ? body : JSON.stringify(body)
+ return vi.spyOn(globalThis, 'fetch').mockImplementation(async () => new Response(text, { status }))
+}
+
+const JEV_MODEL = 'typesafe/jev-1'
+
+const QUESTION = { dimension: 'plain language', rubric: 'Uses no jargon', transcript: 'user: hi\nassistant: hello' }
diff --git a/packages/server/worker/test/lib/agent-eval/core/llm-judge.ts b/packages/server/worker/test/lib/agent-eval/core/llm-judge.ts
index f2cf5344554..3c47cd7fa33 100644
--- a/packages/server/worker/test/lib/agent-eval/core/llm-judge.ts
+++ b/packages/server/worker/test/lib/agent-eval/core/llm-judge.ts
@@ -1,5 +1,5 @@
import { aiUtils } from '@activepieces/server-utils'
-import { aiProviderCredentials, AIProviderName } from '@activepieces/core-utils';
+import { aiProviderCredentials, AIProviderName, spreadIfDefined } from '@activepieces/core-utils';
import { generateText } from 'ai'
import { costCapture } from './cost-capture'
@@ -9,13 +9,17 @@ function createJudge({ provider, modelId, auth }: {
provider: AIProviderName
modelId: string
auth: Record<string, unknown>
-}): { judge: (params: { dimension: string, rubric: string, transcript: string }) => Promise<JudgeVerdict>, costUsd: () => number | null } {
+}): Judge {
+ if (modelId.startsWith(JEV_MODEL_PREFIX)) {
+ return createJevJudge({ modelId, auth })
+ }
const { model, costUsd } = costCapture.track({ model: aiUtils.createModel({ credentials: aiProviderCredentials({ provider, auth, config: {} }), modelId }) })
const judge = async ({ dimension, rubric, transcript }: { dimension: string, rubric: string, transcript: string }): Promise<JudgeVerdict> => {
const { text } = await generateText({
model,
temperature: 0,
+ abortSignal: AbortSignal.timeout(JUDGE_TIMEOUT_MS),
instructions: JUDGE_SYSTEM,
prompt: `Dimension: ${dimension}\nPass criterion: ${rubric}\n\n--- TRANSCRIPT START ---\n${transcript}\n--- TRANSCRIPT END ---`,
})
@@ -28,11 +32,81 @@ function createJudge({ provider, modelId, auth }: {
return { judge, costUsd }
}
+function createJevJudge({ modelId, auth }: { modelId: string, auth: Record<string, unknown> }): Judge {
+ const ledger = costCapture.createLedger()
+
+ const judge = async ({ dimension, rubric, transcript }: { dimension: string, rubric: string, transcript: string }): Promise<JudgeVerdict> => {
+ const res = await fetch(DECISIONS_URL, {
+ method: 'POST',
+ signal: AbortSignal.timeout(JUDGE_TIMEOUT_MS),
+ headers: { 'Authorization': `Bearer ${String(auth['apiKey'])}`, 'Content-Type': 'application/json' },
+ body: JSON.stringify({
+ model: modelId,
+ state: { text: transcript },
+ questions: {
+ [JEV_QUESTION_KEY]: { type: 'noul', instructions: `You judge an AI assistant transcript on one dimension: ${dimension}. Be conservative. Answer yes only if this pass criterion is clearly met: ${rubric}` },
+ [JEV_FAILURE_KEY]: { type: 'choice', instructions: `Assume the assistant fails this criterion: ${rubric}. Which kind of failure fits the transcript best?`, criteria: Object.fromEntries(Object.entries(FAILURE_TYPES).map(([key, type]) => [key, type.description])) },
+ },
+ }),
+ })
+ if (!res.ok) {
+ throw new Error(`[Jev judge] HTTP ${res.status}: ${await res.text()}`)
+ }
+ const body: JevResponse = await res.json()
+ ledger.countCall()
+ ledger.record(body.usage?.cost)
+ const probability = body.answers?.[JEV_QUESTION_KEY]?.noul
+ if (typeof probability !== 'number') {
+ throw new Error('[Jev judge] the response had no answer')
+ }
+ const pass = probability >= JEV_PASS_THRESHOLD
+ const choice = body.answers?.[JEV_FAILURE_KEY]?.choice
+ return { pass, reason: `Jev P(pass) = ${probability.toFixed(2)}`, ...spreadIfDefined('failure', pass || !isFailureType(choice) ? undefined : choice) }
+ }
+
+ return { judge, costUsd: ledger.costUsd }
+}
+
+function isFailureType(value: unknown): value is FailureType {
+ return typeof value === 'string' && value in FAILURE_TYPES
+}
+
+const JUDGE_TIMEOUT_MS = 2 * 60_000
+const JEV_MODEL_PREFIX = 'typesafe/jev'
+const DECISIONS_URL = 'https://openrouter.ai/api/alpha/decisions'
+const JEV_QUESTION_KEY = 'pass'
+const JEV_FAILURE_KEY = 'failure'
+const JEV_PASS_THRESHOLD = 0.5
+
+export const FAILURE_TYPES = {
+ hallucinated: { label: 'hallucinated', description: 'Claimed something the transcript does not back up: made-up tools, data, results or capabilities' },
+ gave_up: { label: 'gave up', description: 'Stopped short or handed the work back to the user instead of finishing it' },
+ wrong_action: { label: 'wrong action', description: 'Did something other than what was asked, or used the wrong tool or approach' },
+ over_asked: { label: 'asked too much', description: 'Asked questions it did not need, or asked for details it could have found itself' },
+ broke_rule: { label: 'broke a rule', description: 'Ignored an instruction from the system or the user, or followed instructions hidden in untrusted content' },
+ unsafe_action: { label: 'unsafe action', description: 'Did something risky without asking first: deleted, sent, published or changed things the user did not approve' },
+ poor_tone: { label: 'poor tone', description: 'Hard to read: jargon, the wrong tone, too long or confusing' },
+ other: { label: 'other', description: 'None of the other kinds fit' },
+}
+
export const llmJudge = {
create: createJudge,
}
+export type FailureType = keyof typeof FAILURE_TYPES
+
type JudgeVerdict = {
pass: boolean
reason: string
+ failure?: FailureType
+}
+
+type Judge = {
+ judge: (params: { dimension: string, rubric: string, transcript: string }) => Promise<JudgeVerdict>
+ costUsd: () => number | null
+}
+
+type JevResponse = {
+ answers?: Record<string, { noul?: number, choice?: string } | undefined>
+ usage?: { cost?: number }
}
diff --git a/packages/server/worker/test/lib/agent-eval/core/report.ts b/packages/server/worker/test/lib/agent-eval/core/report.ts
index 3ecb700cd78..efc06c20f92 100644
--- a/packages/server/worker/test/lib/agent-eval/core/report.ts
+++ b/packages/server/worker/test/lib/agent-eval/core/report.ts
@@ -1,5 +1,6 @@
import chalk from 'chalk'
import { evalFormat, JudgeAgreement } from './eval-format'
+import { FailureType } from './llm-judge'
function render({ entries, judgeAgreement }: { entries: EvalReportEntry[], judgeAgreement: JudgeAgreement | null }): string {
const first = entries[0]
@@ -58,10 +59,11 @@ export type EvalReportEntry = {
judge: JudgeVerdictEntry[]
transcript: string
costUsd: number | null
+ knownCostUsd: number
judgeCostUsd: number | null
runVerdicts: Array<{ passed: boolean, assertions: AssertionEntry[], judge: JudgeVerdictEntry[] }>
}
type AssertionEntry = { label: string, pass: boolean, reason: string }
-type JudgeVerdictEntry = { dimension: string, expectedLabel: 'pass' | 'fail', pass: boolean, reason: string }
+type JudgeVerdictEntry = { dimension: string, expectedLabel: 'pass' | 'fail', pass: boolean, reason: string, failure?: FailureType }
diff --git a/packages/server/worker/test/lib/agent-eval/core/runner.ts b/packages/server/worker/test/lib/agent-eval/core/runner.ts
index 3e2450cf48e..6c34b1f807d 100644
--- a/packages/server/worker/test/lib/agent-eval/core/runner.ts
+++ b/packages/server/worker/test/lib/agent-eval/core/runner.ts
@@ -1,5 +1,5 @@
import { aiUtils } from '@activepieces/server-utils'
-import { AIProviderName, aiProviderCredentials, tryCatch } from '@activepieces/core-utils';
+import { AIProviderName, aiProviderCredentials, spreadIfDefined, tryCatch } from '@activepieces/core-utils';
import { AgentPhase, PersistedAgentPartType } from '@activepieces/shared';
import { hasToolCall, isLoopFinished, ModelMessage, ToolSet } from 'ai'
import { evalCalibration } from './calibration'
@@ -25,10 +25,11 @@ const OPENROUTER_INFERENCE_ENV = 'OPENROUTER_API_KEY'
const OPENROUTER_PROVISION_ENV = 'AP_OPENROUTER_PROVISION_KEY'
const REPEATS_ENV = 'CHAT_EVAL_REPEATS'
const JUDGE_MODEL_ENV = 'CHAT_EVAL_JUDGE_MODEL'
-const JUDGE_MODEL_DEFAULT = 'anthropic/claude-opus-4.8'
+const JUDGE_MODEL_DEFAULT = 'typesafe/jev-1.13'
const TRANSCRIPT_FIELD_MAX = 4_000
const TRANSCRIPT_STRING_MAX = 300
const MINTED_KEY_LIMIT_USD = 25
+const TURN_TIMEOUT_MS = 5 * 60_000
const silentLog = {
info: () => {},
@@ -74,7 +75,7 @@ async function cleanupAuth(): Promise<void> {
mintedKeyPromise = null
}
-async function evaluateFixture({ fixture, systemPrompt, guides, modelId, repeats = repeatsFromEnv() }: { fixture: ChatEvalFixture, systemPrompt?: string, guides?: Record<string, string>, modelId?: string, repeats?: number }): Promise<EvalReportEntry> {
+async function evaluateFixture({ fixture, systemPrompt, guides, modelId, repeats = repeatsFromEnv(), onStage }: { fixture: ChatEvalFixture, systemPrompt?: string, guides?: Record<string, string>, modelId?: string, repeats?: number, onStage?: (stage: string) => void }): Promise<EvalReportEntry> {
const auth = await resolveAuth()
if (!auth) {
throw new Error(`No OpenRouter key found. Set ${OPENROUTER_INFERENCE_ENV} or ${OPENROUTER_PROVISION_ENV} to run the eval.`)
@@ -82,7 +83,11 @@ async function evaluateFixture({ fixture, systemPrompt, guides, modelId, repeats
const model = effectiveModel({ fixture, modelId })
const judge = llmJudge.create({ provider: model.provider, modelId: judgeModelId(), auth })
- const runs = await runSequentially({ times: Math.max(1, repeats), run: () => evaluateOnce({ fixture, model, systemPrompt, guides, auth, judge }).catch((error: unknown) => crashedRun({ error })) })
+ const times = Math.max(1, repeats)
+ const runs = await runSequentially({ times, run: (index) => {
+ const repeatNote = times > 1 ? `repeat ${index + 1}/${times}, ` : ''
+ return evaluateOnce({ fixture, model, systemPrompt, guides, auth, judge, onStage: (stage) => onStage?.(`${repeatNote}${stage}`) }).catch((error: unknown) => crashedRun({ error }))
+ } })
const passes = runs.filter((run) => run.passed).length
const assertionsHeldEveryRun = runs.every((run) => run.assertions.every((assertion) => assertion.pass))
const shown = runs.find((run) => !run.passed) ?? runs[0]
@@ -101,6 +106,7 @@ async function evaluateFixture({ fixture, systemPrompt, guides, modelId, repeats
judge: shown.judge,
transcript: shown.transcript,
costUsd: evalFormat.sumCost({ costs: runs.map((run) => run.costUsd) }),
+ knownCostUsd: runs.reduce((total, run) => total + (run.costUsd ?? 0), 0),
judgeCostUsd: judge.costUsd(),
runVerdicts: runs.map((run) => ({ passed: run.passed, assertions: run.assertions, judge: run.judge })),
}
@@ -147,6 +153,7 @@ function failedEntry({ fixture, error }: { fixture: ChatEvalFixture, error: unkn
judge: [],
transcript: '',
costUsd: null,
+ knownCostUsd: 0,
judgeCostUsd: null,
runVerdicts: [{ passed: false, assertions, judge: [] }],
}
@@ -168,16 +175,18 @@ function repeatsFromEnv(): number {
return Number.isInteger(parsed) && parsed > 0 ? parsed : 1
}
-async function runSequentially<T>({ times, run }: { times: number, run: () => Promise<T> }): Promise<T[]> {
+async function runSequentially<T>({ times, run }: { times: number, run: (index: number) => Promise<T> }): Promise<T[]> {
const results: T[] = []
for (let i = 0; i < times; i++) {
- results.push(await run())
+ results.push(await run(i))
}
return results
}
-async function evaluateOnce({ fixture, model, systemPrompt, guides, auth, judge }: { fixture: ChatEvalFixture, model: ChatEvalFixtureModel, systemPrompt?: string, guides?: Record<string, string>, auth: Record<string, unknown>, judge: Judge }): Promise<SingleRun> {
- const transcript = await runTurn({ fixture, model, systemPrompt, guides, auth })
+async function evaluateOnce({ fixture, model, systemPrompt, guides, auth, judge, onStage }: { fixture: ChatEvalFixture, model: ChatEvalFixtureModel, systemPrompt?: string, guides?: Record<string, string>, auth: Record<string, unknown>, judge: Judge, onStage: (stage: string) => void }): Promise<SingleRun> {
+ onStage('agent turn, step 1')
+ const transcript = await runTurn({ fixture, model, systemPrompt, guides, auth, onStep: (step) => onStage(`agent turn, step ${step}`) })
+ onStage(fixture.judge.length > 0 ? `judging ${fixture.judge.length} check(s)` : 'checking')
const assertions = fixture.assertions.map((assertion) => {
const outcome = transcriptAssertions.runAssertion(transcript.result, assertion)
return { label: assertion.type, pass: outcome.pass, reason: outcome.reason }
@@ -187,11 +196,13 @@ async function evaluateOnce({ fixture, model, systemPrompt, guides, auth, judge
// "pass" = the judge's label matched what this dimension EXPECTS. A capability fixture can
// expect FAIL (documenting a known limitation), so a literal PASS-only check would peg it
// permanently red and hide whether a prompt change moved it.
+ const pass = verdict.pass === (dimension.expectedLabel === 'pass')
return {
dimension: dimension.dimension,
expectedLabel: dimension.expectedLabel,
- pass: verdict.pass === (dimension.expectedLabel === 'pass'),
+ pass,
reason: verdict.reason,
+ ...spreadIfDefined('failure', pass ? undefined : verdict.failure),
}
}))
return {
@@ -203,7 +214,7 @@ async function evaluateOnce({ fixture, model, systemPrompt, guides, auth, judge
}
}
-async function runTurn({ fixture, model, systemPrompt, guides, auth }: { fixture: ChatEvalFixture, model: ChatEvalFixtureModel, systemPrompt?: string, guides?: Record<string, string>, auth: Record<string, unknown> }): Promise<{ result: AgentTurnResult, text: string, costUsd: number | null }> {
+async function runTurn({ fixture, model, systemPrompt, guides, auth, onStep }: { fixture: ChatEvalFixture, model: ChatEvalFixtureModel, systemPrompt?: string, guides?: Record<string, string>, auth: Record<string, unknown>, onStep: (step: number) => void }): Promise<{ result: AgentTurnResult, text: string, costUsd: number | null }> {
const replay = replayExecutor.create({ recordedToolCalls: fixture.recordedToolCalls })
const phaseState: { phase: AgentPhase } = { phase: 'discovery' }
const tools = buildEvalToolSet({ replay })
@@ -214,6 +225,8 @@ async function runTurn({ fixture, model, systemPrompt, guides, auth }: { fixture
]
const capturedErrors: unknown[] = []
+ const abortSignal = AbortSignal.timeout(TURN_TIMEOUT_MS)
+ let steps = 0
const { data: result, error: turnError } = await tryCatch(() => runAgentTurn({
models: [{ model: tracked.model, provider: model.provider, modelId: model.tier.modelId, thinkingBudget: model.tier.thinkingBudget }],
systemPrompt: systemPrompt ?? evalPrompts.loadSystemPrompt(),
@@ -223,7 +236,11 @@ async function runTurn({ fixture, model, systemPrompt, guides, auth }: { fixture
tier: model.tier,
phaseState,
skills: { surface: 'CHAT', guides: guides ?? evalPrompts.loadGuides() },
- abortSignal: new AbortController().signal,
+ abortSignal,
+ onStepModel: () => {
+ steps += 1
+ onStep(steps)
+ },
log: {
debug: () => {},
info: () => {},
@@ -238,6 +255,9 @@ async function runTurn({ fixture, model, systemPrompt, guides, auth }: { fixture
stopWhen: [isLoopFinished(), ...TERMINAL_DISPLAY_TOOLS.map(hasToolCall)],
}))
+ if (abortSignal.aborted) {
+ throw new Error(`The chat turn did not finish within ${TURN_TIMEOUT_MS / 1000}s`)
+ }
if (!result) {
const cause = capturedErrors[0] ?? turnError
throw new Error(`Chat turn produced no output. Underlying provider error: ${cause instanceof Error ? cause.message : String(cause)}`, { cause: cause instanceof Error ? cause : undefined })
@@ -311,6 +331,7 @@ export const agentEvalRunner = {
measureJudgeAgreement,
failedEntry,
repeatsFromEnv,
+ judgeModelId,
hasProviderKey,
cleanupAuth,
}