An AI-powered search engine with a generative UI
TypeScript generative-aigenerative-uinextjsollamareactsearxngshadcn-uitailwindcsstavilytypescriptupstashvercel-ai-sdk
feat(evals): measure the quick search mode, not only adaptive (#992)
Yoshiki Miura04f08df42a47
diff --git a/.github/workflows/eval.yml b/.github/workflows/eval.ymlindex 8d03247f..f2fa1c74 100644--- a/.github/workflows/eval.yml+++ b/.github/workflows/eval.yml@@ -3,8 +3,14 @@ name: Evalon:workflow_dispatch:inputs:+ mode:+ description: 'Search mode to evaluate'+ required: false+ default: both+ type: choice+ options: [both, adaptive, quick]model:- description: 'provider:model to evaluate (default: the deployed adaptive model)'+ description: 'provider:model to evaluate (default: the selected mode model)'required: falsetype: stringtrials:@@ -28,8 +34,15 @@ concurrency:jobs:related-questions:- name: Related questions+ name: Related questions (${{ matrix.mode }})runs-on: ubuntu-latest+ # Both modes, because the point of splitting them is that a prompt or model+ # change can move one and leave the other alone. They are independent+ # measurements, so one failing should not cancel the other.+ strategy:+ fail-fast: false+ matrix:+ mode: [adaptive, quick]# Secrets are not available to pull requests from forks, so the job would# fail for reasons unrelated to the change. Keep it off the required checks.if: >-@@ -57,10 +70,20 @@ jobs:echo "::warning::Eval skipped: OPENAI_API_KEY / LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY are not all configured for this repository."exit 0fi+ # A dispatch may ask for one mode. The matrix still expands to both,+ # so the other leg reports itself as skipped rather than measuring+ # something nobody asked for.+ if [ -n "$SELECTED_MODE" ] && [ "$SELECTED_MODE" != both ] && [ "$SELECTED_MODE" != "$MODE" ]; then+ echo "::notice::Eval skipped: this dispatch selected mode '$SELECTED_MODE'."+ exit 0+ fibun run eval:related-questions -- \+ --mode "${MODE}" \--trials "${TRIALS}" \${MODEL:+--model "$MODEL"}env:+ MODE: ${{ matrix.mode }}+ SELECTED_MODE: ${{ inputs.mode }}TRIALS: ${{ inputs.trials || '5' }}MODEL: ${{ inputs.model }}OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}diff --git a/evals/README.md b/evals/README.mdindex 7c206148..55a78d7c 100644--- a/evals/README.md+++ b/evals/README.md@@ -19,6 +19,7 @@ PRs, where secrets are unavailable.```bashbun run eval:related-questions # deployed adaptive model, 1 trial per item+bun run eval:related-questions -- --mode quick # the quick search mode insteadbun run eval:related-questions -- --trials 3 # 3 generations per itembun run eval:related-questions -- --model openai:some-model # candidate modelbun run eval:related-questions -- --items fuji-routes # one item, for debugging@@ -37,6 +38,20 @@ Emission and well-formedness are separate metrics on purpose. A model that stopsemitting and a model that emits a broken block are different failures withdifferent fixes.+### Search modes++`--mode` selects which deployed configuration is replayed. The two differ in more+than wording: quick mode runs a different system prompt, a smaller tool set, and+its own `providerOptions`, so a result measured on one mode says nothing about+the other. The flag defaults to `adaptive`, which is only a default for a bare+local run; CI runs both modes as a matrix, because a prompt or model change can+move one and leave the other alone. That is the whole reason the modes are+separable here.++Both modes are gated against the same `thresholds.json` entry. There is no+per-mode threshold because nothing yet needs one; add one when a mode's healthy+range is genuinely different rather than to make a run go green.+### Why it is built this way- **Search results are fixed.** Live search would put retrieval variance into adiff --git a/evals/datasets/related-questions.json b/evals/datasets/related-questions.jsonindex 8191bcf8..8ac8c9ba 100644--- a/evals/datasets/related-questions.json+++ b/evals/datasets/related-questions.json@@ -1,6 +1,6 @@{"name": "related-questions",- "description": "Does the adaptive answer emit a well-formed related questions spec block on substantive answers, and skip it on trivial lookups? Search results are fixed so the measurement isolates the prompt and the model. An item is trivial only when the fixture holds exactly one unambiguous value answering it and no comparison or procedure is implied; a question whose answer naturally expands belongs in the substantive set, not the trivial one.",+ "description": "Does the answer emit a well-formed related questions spec block on substantive answers, and skip it on trivial lookups? Search results are fixed so the measurement isolates the prompt and the model. An item is trivial only when the fixture holds exactly one unambiguous value answering it and no comparison or procedure is implied; a question whose answer naturally expands belongs in the substantive set, not the trivial one.","fixtures": {"mount-fuji": [{diff --git a/evals/run-related-questions.ts b/evals/run-related-questions.tsindex 19d4140c..929b2473 100644--- a/evals/run-related-questions.ts+++ b/evals/run-related-questions.ts@@ -1,6 +1,6 @@#!/usr/bin/env bun/**- * Measure the related questions emission rate of the deployed adaptive+ * Measure the related questions emission rate of a deployed search mode* configuration against a fixed dataset.** bun run eval:related-questions@@ -12,14 +12,17 @@ import { config as dotenvConfig } from 'dotenv'dotenvConfig({ path: '.env.local' })-// The adaptive prompt branches on whether a general search provider is+// The search mode prompts branch on whether a general search provider is// configured, so an unset key would silently measure a different prompt than// the deployed one. Pinned before anything reads it. The fixture search tool// means no request ever reaches Brave.process.env.BRAVE_SEARCH_API_KEY ||= 'eval-pinned-general-provider'+type Mode = 'quick' | 'adaptive'+type Args = {model?: string+ mode: Modetrials: numberconcurrency: numberrunName?: string@@ -28,7 +31,12 @@ type Args = {}function parseArgs(argv: string[]): Args {- const args: Args = { trials: 1, concurrency: 5, gate: true }+ const args: Args = {+ trials: 1,+ concurrency: 5,+ gate: true,+ mode: 'adaptive'+ }for (let i = 0; i < argv.length; i++) {const flag = argv[i]@@ -42,6 +50,14 @@ function parseArgs(argv: string[]): Args {case '--model':args.model = next()break+ case '--mode': {+ const mode = next()+ if (mode !== 'quick' && mode !== 'adaptive') {+ throw new Error('--mode must be quick or adaptive')+ }+ args.mode = mode+ break+ }case '--trials':args.trials = Number(next())break@@ -65,12 +81,13 @@ function parseArgs(argv: string[]): Args {['Usage: bun run eval:related-questions [options]','',- ' --model <id> provider:model to evaluate (default: the deployed adaptive model)',- ' --trials <n> generations per dataset item (default: 1)',- ' --concurrency <n> parallel generations (default: 5)',- ' --run-name <name> exact Langfuse run name (default: name + timestamp)',- ' --items <a,b> run only these dataset item ids',- ' --no-gate report metrics without failing on threshold violations'+ ' --mode <quick|adaptive> search mode to evaluate (default: adaptive)',+ ' --model <id> provider:model to evaluate (default: the selected mode model)',+ ' --trials <n> generations per dataset item (default: 1)',+ ' --concurrency <n> parallel generations (default: 5)',+ ' --run-name <name> exact Langfuse run name (default: name + timestamp)',+ ' --items <a,b> run only these dataset item ids',+ ' --no-gate report metrics without failing on threshold violations'].join('\n'))process.exit(0)@@ -106,18 +123,28 @@ async function main() {relatedQuestionsRunEvaluator,summarizeRelatedQuestions} = await import('./evaluators/related-questions')- const { createAdaptiveAnswerTask, DEFAULT_MODEL } = await import(- './tasks/adaptive-answer'- )+ const adaptiveTask = await import('./tasks/adaptive-answer')+ const quickTask = await import('./tasks/quick-answer')const thresholds = (await import('./thresholds.json')).default- const model = args.model ?? DEFAULT_MODEL+ const taskConfig =+ args.mode === 'quick'+ ? {+ createTask: quickTask.createQuickAnswerTask,+ defaultModel: quickTask.DEFAULT_MODEL+ }+ : {+ createTask: adaptiveTask.createAdaptiveAnswerTask,+ defaultModel: adaptiveTask.DEFAULT_MODEL+ }+ const model = args.model ?? taskConfig.defaultModelconst data = loadRelatedQuestionsDataset({trials: args.trials,ids: args.ids})const tracing = setupTracing()+ console.log(`mode: ${args.mode}`)console.log(`model: ${model}`)console.log(`items: ${data.length} (${args.trials} trial(s) per query)`)console.log(`concurrency: ${args.concurrency}\n`)@@ -128,9 +155,9 @@ async function main() {name: 'related-questions',runName: args.runName,description: datasetDescription,- metadata: { model, trials: args.trials },+ metadata: { mode: args.mode, model, trials: args.trials },data,- task: createAdaptiveAnswerTask({ model, tracingEnabled: true }),+ task: taskConfig.createTask({ model, tracingEnabled: true }),evaluators: [relatedQuestionsEvaluator],runEvaluators: [relatedQuestionsRunEvaluator],maxConcurrency: args.concurrencydiff --git a/evals/tasks/adaptive-answer.ts b/evals/tasks/adaptive-answer.tsindex d0041c72..d9d35c15 100644--- a/evals/tasks/adaptive-answer.ts+++ b/evals/tasks/adaptive-answer.ts@@ -1,19 +1,12 @@-import { generateText, isStepCount, tool } from 'ai'-import { z } from 'zod'-import cloudConfig from '@/config/models/cloud.json'import { getAdaptiveModePrompt } from '@/lib/agents/prompts/search-mode-prompts'-import { fetchTool } from '@/lib/tools/fetch'-import { createSearchTool } from '@/lib/tools/search'-import { createTodoTools } from '@/lib/tools/todo'-import { getModel } from '@/lib/utils/registry'--import { type RelatedQuestionsInput, type SearchFixture } from '../lib/dataset'-// Narrow enough to satisfy the SDK's JSON-shaped provider options, wide enough-// to carry another provider's settings when a model swap is being evaluated.-type ProviderOptions = Record<string, Record<string, string | number | boolean>>+import {+ type AnswerOutput,+ type ProviderOptions,+ runAnswerTask+} from './run-answer-task'const ADAPTIVE = cloudConfig.models.adaptive@@ -23,7 +16,7 @@ export const DEFAULT_PROVIDER_OPTIONS = ADAPTIVE.providerOptions// Production allows 50 steps in adaptive mode. The fixture search returns the// same results every time, so a healthy run converges in a handful of steps;// this cap only bounds a loop that is going nowhere. A run that hits it is-// rejected rather than measured, see the finish reason check below.+// rejected rather than measured, see the finish reason check in the runner.const MAX_STEPS = 20// The provider intermittently returns a response the SDK cannot parse, at a few@@ -32,11 +25,7 @@ const MAX_STEPS = 20// retry does not cover this error class.const DEFAULT_RETRIES = 2-export type AdaptiveAnswerOutput = {- text: string- searchQueries: string[]- steps: number-}+export type AdaptiveAnswerOutput = AnswerOutput/*** Replay one query against the deployed adaptive configuration with the search@@ -58,132 +47,14 @@ export function createAdaptiveAnswerTask({tracingEnabled?: booleanretries?: number} = {}) {- // Built once: it reaches out to the search provider config, not the network.- const toModelOutput = createSearchTool(model).toModelOutput-- // The item type is the SDK's union of a local item and a Langfuse dataset- // item, whose input is `unknown`. Narrowing here is what lets the same task- // run against a managed dataset later without another code path.- return async function adaptiveAnswer(item: {- input?: unknown- }): Promise<AdaptiveAnswerOutput> {- const input = item.input as RelatedQuestionsInput | undefined- if (!input?.query || !Array.isArray(input.results)) {- throw new Error('Experiment item input must be { query, results }')- }-- let lastError: unknown- for (let attempt = 0; attempt <= retries; attempt++) {- try {- return await generate(input)- } catch (error) {- lastError = error- const message = error instanceof Error ? error.message : String(error)- console.warn(- `[eval] attempt ${attempt + 1}/${retries + 1} failed for "${input.query}": ${message}`- )- }- }- throw lastError- }-- async function generate(- input: RelatedQuestionsInput- ): Promise<AdaptiveAnswerOutput> {- const searchQueries: string[] = []-- const result = await generateText({- model: getModel(model),- instructions: getAdaptiveModePrompt(),- // The adaptive agent activates search, fetch and todoWrite, and the- // prompt tells the model those tools exist. Handing it only search would- // force a different trajectory and so a different answer shape, which is- // the thing being measured. todoWrite is the production tool as-is: its- // state is per-instance and never leaves the process.- tools: {- search: createFixtureSearchTool({- results: input.results,- toModelOutput,- onQuery: query => searchQueries.push(query)- }),- fetch: createFixtureFetchTool(input.results),- ...createTodoTools()- },- stopWhen: isStepCount(MAX_STEPS),- providerOptions,- prompt: input.query,- telemetry: {- isEnabled: tracingEnabled,- functionId: 'eval-adaptive-answer'- }- })-- // A loop cut off at the step cap leaves an answer the model never finished,- // and an unfinished answer carries no related block for reasons that have- // nothing to do with the prompt. Reject it so the retry runs and, if it- // keeps happening, completionRate reports it.- if (result.finishReason === 'tool-calls') {- throw new Error(`Step cap of ${MAX_STEPS} reached before an answer`)- }- if (result.text.trim() === '') {- throw new Error('Model produced no answer text')- }-- return {- text: result.text,- searchQueries,- steps: result.steps.length- }- }-}--// Mirrors the production fetch tool's contract while serving the item's own-// fixture, so the model can follow a URL out of the search results without the-// answer depending on what that page says today.-function createFixtureFetchTool(results: SearchFixture[]) {- return tool({- description: fetchTool.description,- inputSchema: fetchTool.inputSchema,- async execute({ url }: { url: string }) {- const match = results.find(result => result.url === url)- return {- state: 'complete' as const,- query: url,- results: match ? [match] : results,- images: [],- number_of_results: match ? 1 : results.length- }- }- })-}--function createFixtureSearchTool({- results,- toModelOutput,- onQuery-}: {- results: SearchFixture[]- toModelOutput: ReturnType<typeof createSearchTool>['toModelOutput']- onQuery: (query: string) => void-}) {- return tool({- description: 'Search the web for information.',- inputSchema: z.object({- query: z.string(),- type: z.enum(['optimized', 'general']).optional(),- max_results: z.number().optional()- }),- async execute({ query }) {- onQuery(query)- return {- state: 'complete' as const,- query,- results,- images: [],- number_of_results: results.length,- toolCallId: 'fixture-search'- }- },- toModelOutput: toModelOutput as never+ return runAnswerTask({+ model,+ providerOptions,+ tracingEnabled,+ retries,+ maxSteps: MAX_STEPS,+ getPrompt: getAdaptiveModePrompt,+ includeTodoWrite: true,+ telemetryFunctionId: 'eval-adaptive-answer'})}diff --git a/evals/tasks/fixture-tools.ts b/evals/tasks/fixture-tools.tsnew file mode 100644index 00000000..64b4ed12--- /dev/null+++ b/evals/tasks/fixture-tools.ts@@ -0,0 +1,58 @@+import { tool } from 'ai'+import { z } from 'zod'++import { fetchTool } from '@/lib/tools/fetch'+import { createSearchTool } from '@/lib/tools/search'++import { type SearchFixture } from '../lib/dataset'++// Mirrors the production fetch tool's contract while serving the item's own+// fixture, so the model can follow a URL out of the search results without the+// answer depending on what that page says today.+export function createFixtureFetchTool(results: SearchFixture[]) {+ return tool({+ description: fetchTool.description,+ inputSchema: fetchTool.inputSchema,+ async execute({ url }: { url: string }) {+ const match = results.find(result => result.url === url)+ return {+ state: 'complete' as const,+ query: url,+ results: match ? [match] : results,+ images: [],+ number_of_results: match ? 1 : results.length+ }+ }+ })+}++export function createFixtureSearchTool({+ results,+ toModelOutput,+ onQuery+}: {+ results: SearchFixture[]+ toModelOutput: ReturnType<typeof createSearchTool>['toModelOutput']+ onQuery: (query: string) => void+}) {+ return tool({+ description: 'Search the web for information.',+ inputSchema: z.object({+ query: z.string(),+ type: z.enum(['optimized', 'general']).optional(),+ max_results: z.number().optional()+ }),+ async execute({ query }) {+ onQuery(query)+ return {+ state: 'complete' as const,+ query,+ results,+ images: [],+ number_of_results: results.length,+ toolCallId: 'fixture-search'+ }+ },+ toModelOutput: toModelOutput as never+ })+}diff --git a/evals/tasks/quick-answer.ts b/evals/tasks/quick-answer.tsnew file mode 100644index 00000000..668a317f--- /dev/null+++ b/evals/tasks/quick-answer.ts@@ -0,0 +1,60 @@+import cloudConfig from '@/config/models/cloud.json'++import { getQuickModePrompt } from '@/lib/agents/prompts/search-mode-prompts'++import {+ type AnswerOutput,+ type ProviderOptions,+ runAnswerTask+} from './run-answer-task'++const QUICK = cloudConfig.models.quick++export const DEFAULT_MODEL = `${QUICK.providerId}:${QUICK.id}`+export const DEFAULT_PROVIDER_OPTIONS = QUICK.providerOptions++// Production allows 20 steps in quick mode. The fixture search returns the+// same results every time, so a healthy run converges in a handful of steps;+// this cap only bounds a loop that is going nowhere. A run that hits it is+// rejected rather than measured, see the finish reason check in the runner.+const MAX_STEPS = 20++// The provider intermittently returns a response the SDK cannot parse, at a few+// percent of calls. The experiment runner drops an item whose task throws, so+// without a retry that noise lands directly in the measured rates. The SDK's own+// retry does not cover this error class.+const DEFAULT_RETRIES = 2++export type QuickAnswerOutput = AnswerOutput++/**+ * Replay one query against the deployed quick configuration with the search+ * results pinned.+ *+ * Production faithfulness is the point: the same prompt builder, the same+ * providerOptions, and the same `toModelOutput` trimming the real search tool+ * applies. Reasoning effort in particular has moved emission rates before, so a+ * run without providerOptions would measure something the product does not do.+ */+export function createQuickAnswerTask({+ model = DEFAULT_MODEL,+ providerOptions = DEFAULT_PROVIDER_OPTIONS,+ tracingEnabled = true,+ retries = DEFAULT_RETRIES+}: {+ model?: string+ providerOptions?: ProviderOptions+ tracingEnabled?: boolean+ retries?: number+} = {}) {+ return runAnswerTask({+ model,+ providerOptions,+ tracingEnabled,+ retries,+ maxSteps: MAX_STEPS,+ getPrompt: getQuickModePrompt,+ includeTodoWrite: false,+ telemetryFunctionId: 'eval-quick-answer'+ })+}diff --git a/evals/tasks/run-answer-task.ts b/evals/tasks/run-answer-task.tsnew file mode 100644index 00000000..ef8fc85e--- /dev/null+++ b/evals/tasks/run-answer-task.ts@@ -0,0 +1,120 @@+import { generateText, isStepCount } from 'ai'++import { createSearchTool } from '@/lib/tools/search'+import { createTodoTools } from '@/lib/tools/todo'+import { getModel } from '@/lib/utils/registry'++import { type RelatedQuestionsInput } from '../lib/dataset'++import {+ createFixtureFetchTool,+ createFixtureSearchTool+} from './fixture-tools'++// Narrow enough to satisfy the SDK's JSON-shaped provider options, wide enough+// to carry another provider's settings when a model swap is being evaluated.+export type ProviderOptions = Record<+ string,+ Record<string, string | number | boolean>+>++export type AnswerOutput = {+ text: string+ searchQueries: string[]+ steps: number+}++export type AnswerTaskOptions = {+ model: string+ providerOptions: ProviderOptions+ tracingEnabled: boolean+ retries: number+ maxSteps: number+ getPrompt: () => string+ includeTodoWrite: boolean+ telemetryFunctionId: string+}++export function runAnswerTask({+ model,+ providerOptions,+ tracingEnabled,+ retries,+ maxSteps,+ getPrompt,+ includeTodoWrite,+ telemetryFunctionId+}: AnswerTaskOptions) {+ // Built once: it reaches out to the search provider config, not the network.+ const toModelOutput = createSearchTool(model).toModelOutput++ // The item type is the SDK's union of a local item and a Langfuse dataset+ // item, whose input is `unknown`. Narrowing here is what lets the same task+ // run against a managed dataset later without another code path.+ return async function answer(item: {+ input?: unknown+ }): Promise<AnswerOutput> {+ const input = item.input as RelatedQuestionsInput | undefined+ if (!input?.query || !Array.isArray(input.results)) {+ throw new Error('Experiment item input must be { query, results }')+ }++ let lastError: unknown+ for (let attempt = 0; attempt <= retries; attempt++) {+ try {+ return await generate(input)+ } catch (error) {+ lastError = error+ const message = error instanceof Error ? error.message : String(error)+ console.warn(+ `[eval] attempt ${attempt + 1}/${retries + 1} failed for "${input.query}": ${message}`+ )+ }+ }+ throw lastError+ }++ async function generate(input: RelatedQuestionsInput): Promise<AnswerOutput> {+ const searchQueries: string[] = []++ const tools = {+ search: createFixtureSearchTool({+ results: input.results,+ toModelOutput,+ onQuery: query => searchQueries.push(query)+ }),+ fetch: createFixtureFetchTool(input.results),+ ...(includeTodoWrite ? createTodoTools() : {})+ }++ const result = await generateText({+ model: getModel(model),+ instructions: getPrompt(),+ tools,+ stopWhen: isStepCount(maxSteps),+ providerOptions,+ prompt: input.query,+ telemetry: {+ isEnabled: tracingEnabled,+ functionId: telemetryFunctionId+ }+ })++ // A loop cut off at the step cap leaves an answer the model never finished,+ // and an unfinished answer carries no related block for reasons that have+ // nothing to do with the prompt. Reject it so the retry runs and, if it+ // keeps happening, completionRate reports it.+ if (result.finishReason === 'tool-calls') {+ throw new Error(`Step cap of ${maxSteps} reached before an answer`)+ }+ if (result.text.trim() === '') {+ throw new Error('Model produced no answer text')+ }++ return {+ text: result.text,+ searchQueries,+ steps: result.steps.length+ }+ }+}