Skip to content

An AI-powered search engine with a generative UI

fix(draft): keep editDraft off the draft for unrelated requests (#1100)

Yoshiki Miura38641395ada0
diff --git a/evals/README.md b/evals/README.md
index 8138bdd7..4a104d59 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -105,6 +105,33 @@ Fixture URLs use real-looking hosts. With `.example` hosts the adaptive model
correctly judged the results to be placeholders and declined to present them,
which measured the fixture instead of the prompt.
+## draft-edits
+
+```bash
+bun run eval:draft-edits # synthetic chats, plus local production seeds if present
+bun run eval:draft-edits -- --trials 3
+bun run eval:draft-edits -- --no-synthetic # production seeds only
+```
+
+Replays one turn in a chat that already holds a draft. Each item carries the
+chat so far and goes through the same history preparation as the chat route
+(draft compaction, then `<current_draft>` attached to the latest user message),
+with its own mode's prompt, tools, and `providerOptions`. The real `editDraft`
+tool runs against the rebuilt draft, so a call is recorded exactly as
+production would apply it. Reports:
+
+- `hijackRate`: unrelated requests (a question, a keyword list) where
+ `editDraft` was called at all
+- `newBlockRate`: requests for a new piece on another topic answered with a new
+ complete draft block and no `editDraft` call
+- `editSuccessRate`: requests to change the draft that changed it, either
+ through an applied edit or a new complete block
+
+The committed dataset is synthetic. Production turns, where the hijack was
+first observed, hold users' own text, so they are built into
+`evals/_local/draft-edits-seeds.json`, which is gitignored and loaded only when
+present. Output for production items is never printed, only ids and verdicts.
+
## Thresholds
`thresholds.json` gates CI. A `null` value means the metric is reported but not
diff --git a/evals/datasets/draft-edits.json b/evals/datasets/draft-edits.json
new file mode 100644
index 00000000..cd7b31f5
--- /dev/null
+++ b/evals/datasets/draft-edits.json
@@ -0,0 +1,93 @@
+{
+ "description": "Turns in a chat that already holds a draft. Unrelated requests must leave the draft alone, requests for a new piece must write a new draft block, and requests to change the draft must change it.",
+ "chats": {
+ "welcome-email": {
+ "mode": "quick",
+ "messages": [
+ {
+ "role": "user",
+ "text": "Write a 120-word welcome email for new members of our running club, Riverside Striders. Don't search."
+ },
+ {
+ "role": "assistant",
+ "text": "Here is a welcome email for your new members.\n\n````draft\ntitle: Welcome email for new members\n\nSubject: Welcome to Riverside Striders!\n\nHi {name},\n\nWelcome to Riverside Striders! We're thrilled to have you join our community of runners of every pace and experience level.\n\nWe meet every Saturday at 8:00 a.m. at the boathouse for a group run along the river, with routes from 5 km to 15 km. Wednesday evenings are for easy social runs followed by coffee nearby.\n\nBefore your first run, join our members' group chat, where we post route maps, weather updates, and race plans. Bring water, comfortable shoes, and your best attitude.\n\nIf you have questions, reply to this email anytime. We can't wait to hit the trail with you.\n\nSee you Saturday,\nThe Riverside Striders Team\n````"
+ }
+ ]
+ },
+ "skillet-article": {
+ "mode": "adaptive",
+ "messages": [
+ {
+ "role": "user",
+ "text": "Write an article titled \"How to Care for a Cast Iron Skillet\" with H2 headings."
+ },
+ {
+ "role": "assistant",
+ "text": "I wrote the article with H2 sections on cleaning, drying, seasoning, and storage.\n\n````draft\ntitle: How to Care for a Cast Iron Skillet\n\n# How to Care for a Cast Iron Skillet\n\nA cast iron skillet can last for generations. It heats evenly, moves from stovetop to oven, and gets better with use. The catch is that it needs a little care that stainless steel pans do not.\n\n## Clean It While It Is Warm\n\nRinse the skillet with hot water right after cooking and scrub off food with a brush or a chain-mail scrubber. A small drop of mild dish soap is fine on a well-seasoned pan. Avoid soaking it in the sink, which invites rust.\n\n## Dry It Completely\n\nWater is the main enemy of cast iron. Towel the skillet dry, then set it over low heat for a minute or two until every trace of moisture is gone.\n\n## Season It Regularly\n\nSeasoning is a thin layer of polymerized oil that protects the iron and keeps food from sticking. Rub a very thin coat of neutral oil over the warm pan, wipe off the excess, and heat it until it just starts to smoke. Repeat whenever the surface looks dull or dry.\n\n## Store It Dry\n\nKeep the skillet in a dry cupboard. If you stack pans, put a paper towel inside to absorb moisture and protect the surface.\n\n## Fixing Rust\n\nIf rust appears, scrub it off with steel wool, wash and dry the pan, and season it two or three times. The skillet will be as good as new.\n````"
+ }
+ ]
+ }
+ },
+ "items": [
+ {
+ "id": "email-half-marathon",
+ "chat": "welcome-email",
+ "category": "unrelated",
+ "query": "How long is a half marathon in miles?"
+ },
+ {
+ "id": "email-stretches",
+ "chat": "welcome-email",
+ "category": "unrelated",
+ "query": "List 5 stretches to do before a run."
+ },
+ {
+ "id": "email-instagram-post",
+ "chat": "welcome-email",
+ "category": "new",
+ "query": "Now write a short Instagram post announcing our Saturday trail run on November 8."
+ },
+ {
+ "id": "email-warmer",
+ "chat": "welcome-email",
+ "category": "edit",
+ "query": "Make the email warmer and mention our Tuesday track sessions."
+ },
+ {
+ "id": "email-sign-off",
+ "chat": "welcome-email",
+ "category": "edit",
+ "query": "Sign it from Dana, the club captain, instead of the team."
+ },
+ {
+ "id": "skillet-smoke-point",
+ "chat": "skillet-article",
+ "category": "unrelated",
+ "query": "What is the smoke point of flaxseed oil?"
+ },
+ {
+ "id": "skillet-keywords",
+ "chat": "skillet-article",
+ "category": "unrelated",
+ "query": "Give me 9 related keywords for \"cast iron skillet\", each word capitalized."
+ },
+ {
+ "id": "skillet-new-article",
+ "chat": "skillet-article",
+ "category": "new",
+ "query": "Write a 600-word article titled \"Carbon Steel vs. Cast Iron Pans\" with H2 headings."
+ },
+ {
+ "id": "skillet-catchier-title",
+ "chat": "skillet-article",
+ "category": "edit",
+ "query": "Change the title to something catchier."
+ },
+ {
+ "id": "skillet-faq",
+ "chat": "skillet-article",
+ "category": "edit",
+ "query": "Add a FAQ section at the end with 3 questions and answers."
+ }
+ ]
+}
diff --git a/evals/evaluators/draft-edits.ts b/evals/evaluators/draft-edits.ts
new file mode 100644
index 00000000..b490e620
--- /dev/null
+++ b/evals/evaluators/draft-edits.ts
@@ -0,0 +1,147 @@
+import type { Evaluation, Evaluator, RunEvaluator } from '@langfuse/client'
+
+import {
+ analyzeDraftTurn,
+ type DraftEditsCategory,
+ type DraftEditsExpected,
+ type DraftEditsInput,
+ type DraftEditsMetadata,
+ type DraftTurnOutput,
+ isDraftEditsCategory
+} from '../lib/draft-edits'
+
+type ItemResult = {
+ output?: unknown
+ metadata?: unknown
+ item?: { metadata?: unknown }
+}
+
+export type DraftEditsSummary = {
+ unrelatedCount: number
+ newCount: number
+ editCount: number
+ /** Share of unrelated requests where editDraft was called. */
+ hijackRate: number
+ /** Share of unrelated requests answered with a new draft block. */
+ unrelatedBlockRate: number
+ /** Share of new-piece requests answered with a new block and no edit. */
+ newBlockRate: number
+ /** Share of change requests that changed the draft or rewrote it. */
+ editSuccessRate: number
+}
+
+function turnOutput(output: unknown): DraftTurnOutput {
+ const value = output as Partial<DraftTurnOutput> | undefined
+ return {
+ text: typeof value?.text === 'string' ? value.text : '',
+ editCalls: Array.isArray(value?.editCalls) ? value.editCalls : []
+ }
+}
+
+function itemCategory(result: ItemResult): DraftEditsCategory | undefined {
+ const metadata = (result.metadata ?? result.item?.metadata) as
+ | { category?: unknown }
+ | undefined
+ return isDraftEditsCategory(metadata?.category)
+ ? metadata.category
+ : undefined
+}
+
+export const draftEditsEvaluator: Evaluator<
+ DraftEditsInput,
+ DraftEditsExpected,
+ DraftEditsMetadata
+> = async ({ output, expectedOutput }) => {
+ const category = expectedOutput?.category
+ if (!category) return []
+ const analysis = analyzeDraftTurn(turnOutput(output), category)
+
+ const evaluations: Evaluation[] = [
+ {
+ name: 'draft_turn_correct',
+ value: analysis.correct ? 1 : 0,
+ dataType: 'BOOLEAN',
+ comment: category
+ },
+ {
+ name: 'draft_edit_called',
+ value: analysis.calledEdit ? 1 : 0,
+ dataType: 'BOOLEAN'
+ },
+ {
+ name: 'draft_block_written',
+ value: analysis.wroteBlock ? 1 : 0,
+ dataType: 'BOOLEAN'
+ }
+ ]
+ return evaluations
+}
+
+function rate(hits: number, total: number) {
+ return total === 0 ? 0 : hits / total
+}
+
+export function summarizeDraftEdits(
+ itemResults: ItemResult[]
+): DraftEditsSummary {
+ const counts = { unrelated: 0, new: 0, edit: 0 }
+ let hijacked = 0
+ let unrelatedBlocks = 0
+ let newBlocks = 0
+ let edited = 0
+
+ for (const result of itemResults) {
+ const category = itemCategory(result)
+ if (!category) continue
+ const analysis = analyzeDraftTurn(turnOutput(result.output), category)
+ counts[category]++
+ if (category === 'unrelated') {
+ if (analysis.calledEdit) hijacked++
+ if (analysis.wroteBlock) unrelatedBlocks++
+ } else if (category === 'new') {
+ if (analysis.correct) newBlocks++
+ } else if (analysis.correct) {
+ edited++
+ }
+ }
+
+ return {
+ unrelatedCount: counts.unrelated,
+ newCount: counts.new,
+ editCount: counts.edit,
+ hijackRate: rate(hijacked, counts.unrelated),
+ unrelatedBlockRate: rate(unrelatedBlocks, counts.unrelated),
+ newBlockRate: rate(newBlocks, counts.new),
+ editSuccessRate: rate(edited, counts.edit)
+ }
+}
+
+export const draftEditsRunEvaluator: RunEvaluator<
+ DraftEditsInput,
+ DraftEditsExpected,
+ DraftEditsMetadata
+> = async ({ itemResults }) => {
+ const summary = summarizeDraftEdits(itemResults)
+ return [
+ {
+ name: 'hijackRate',
+ value: summary.hijackRate,
+ comment: `n=${summary.unrelatedCount}`
+ },
+ {
+ name: 'unrelatedBlockRate',
+ value: summary.unrelatedBlockRate,
+ comment: `n=${summary.unrelatedCount}`
+ },
+ {
+ name: 'newBlockRate',
+ value: summary.newBlockRate,
+ comment: `n=${summary.newCount}`
+ },
+ {
+ name: 'editSuccessRate',
+ value: summary.editSuccessRate,
+ comment: `n=${summary.editCount}`
+ }
+ ]
+}
diff --git a/evals/lib/__tests__/draft-edits.test.ts b/evals/lib/__tests__/draft-edits.test.ts
new file mode 100644
index 00000000..accd2394
--- /dev/null
+++ b/evals/lib/__tests__/draft-edits.test.ts
@@ -0,0 +1,95 @@
+import { describe, expect, test } from 'vitest'
+
+import { prepareDraftTurn } from '../../tasks/draft-turn'
+import { analyzeDraftTurn, loadDraftEditsDataset } from '../draft-edits'
+
+const BLOCK = [
+ 'Wrote it.',
+ '',
+ '````draft',
+ 'title: T',
+ '',
+ 'Body',
+ '````'
+].join('\n')
+const EDIT = { input: { title: 'Renamed' }, applied: 1 }
+
+describe('analyzeDraftTurn', () => {
+ test('an unrelated answer is correct only without an editDraft call', () => {
+ expect(
+ analyzeDraftTurn({ text: 'Here you go.', editCalls: [] }, 'unrelated')
+ .correct
+ ).toBe(true)
+ expect(
+ analyzeDraftTurn({ text: 'Renamed.', editCalls: [EDIT] }, 'unrelated')
+ .correct
+ ).toBe(false)
+ })
+
+ test('a rejected edit still counts as touching the draft', () => {
+ const analysis = analyzeDraftTurn(
+ { text: '', editCalls: [{ input: {}, applied: 0 }] },
+ 'unrelated'
+ )
+ expect(analysis.calledEdit).toBe(true)
+ expect(analysis.changedDraft).toBe(false)
+ })
+
+ test('a new piece needs a new block and no edit', () => {
+ expect(
+ analyzeDraftTurn({ text: BLOCK, editCalls: [] }, 'new').correct
+ ).toBe(true)
+ expect(
+ analyzeDraftTurn({ text: 'Updated.', editCalls: [EDIT] }, 'new').correct
+ ).toBe(false)
+ expect(
+ analyzeDraftTurn({ text: BLOCK, editCalls: [EDIT] }, 'new').correct
+ ).toBe(false)
+ })
+
+ test('a change request accepts an applied edit or a rewrite', () => {
+ expect(
+ analyzeDraftTurn({ text: 'Updated.', editCalls: [EDIT] }, 'edit').correct
+ ).toBe(true)
+ expect(
+ analyzeDraftTurn({ text: BLOCK, editCalls: [] }, 'edit').correct
+ ).toBe(true)
+ expect(
+ analyzeDraftTurn(
+ { text: 'Tried.', editCalls: [{ input: {}, applied: 0 }] },
+ 'edit'
+ ).correct
+ ).toBe(false)
+ })
+
+ test('an unclosed block is not a written draft', () => {
+ const open = '````draft\ntitle: T\n\nBody'
+ expect(
+ analyzeDraftTurn({ text: open, editCalls: [] }, 'new').wroteBlock
+ ).toBe(false)
+ })
+})
+
+describe('synthetic draft-edits dataset', () => {
+ const items = loadDraftEditsDataset({
+ seedsPath: '/nonexistent',
+ includeSynthetic: true
+ })
+
+ test('covers every category', () => {
+ const categories = new Set(items.map(item => item.metadata.category))
+ expect([...categories].sort()).toEqual(['edit', 'new', 'unrelated'])
+ })
+
+ test('every item reaches the model with its draft attached', () => {
+ for (const item of items) {
+ const prepared = prepareDraftTurn(item.input.messages)
+ expect(prepared.currentDraft).not.toBeNull()
+ const last = prepared.messages.at(-1)!
+ const attached = last.parts.at(-1)
+ expect(attached?.type === 'text' && attached.text).toContain(
+ '<current_draft'
+ )
+ }
+ })
+})
diff --git a/evals/lib/draft-edits.ts b/evals/lib/draft-edits.ts
new file mode 100644
index 00000000..557b32d9
--- /dev/null
+++ b/evals/lib/draft-edits.ts
@@ -0,0 +1,231 @@
+import type { UIMessage } from 'ai'
+import { existsSync, readFileSync } from 'fs'
+import path from 'path'
+
+import { parseDraftBlocks } from '@/lib/draft/draft'
+
+import datasetJson from '../datasets/draft-edits.json'
+
+import type { SearchFixture } from './dataset'
+
+export type DraftEditsMode = 'quick' | 'adaptive'
+
+export type DraftEditsCategory = 'unrelated' | 'new' | 'edit'
+
+export type DraftEditsInput = {
+ mode: DraftEditsMode
+ /** The chat so far, ending with the user turn under test. */
+ messages: UIMessage[]
+ results: SearchFixture[]
+}
+
+export type DraftEditsExpected = {
+ category: DraftEditsCategory
+}
+
+export type DraftEditsMetadata = {
+ itemId: string
+ category: DraftEditsCategory
+ source: 'synthetic' | 'production'
+ trial: number
+}
+
+export type DraftEditsItem = {
+ input: DraftEditsInput
+ expectedOutput: DraftEditsExpected
+ metadata: DraftEditsMetadata
+}
+
+type RawChat = {
+ mode: string
+ messages: { role: string; text: string }[]
+}
+
+type RawItem = {
+ id: string
+ chat: string
+ category: string
+ query: string
+}
+
+type SeedItem = {
+ id: string
+ category: string
+ mode: string
+ messages: UIMessage[]
+ results: SearchFixture[]
+}
+
+/**
+ * Production turns, built from the chats where the hijack was observed. They
+ * hold users' own text, so they live in a gitignored file and are used only
+ * when it exists.
+ */
+export const LOCAL_SEEDS_PATH = path.join(
+ __dirname,
+ '..',
+ '_local',
+ 'draft-edits-seeds.json'
+)
+
+export function isDraftEditsCategory(
+ value: unknown
+): value is DraftEditsCategory {
+ return value === 'unrelated' || value === 'new' || value === 'edit'
+}
+
+function isMode(value: unknown): value is DraftEditsMode {
+ return value === 'quick' || value === 'adaptive'
+}
+
+function textMessage(id: string, role: string, text: string): UIMessage {
+ return {
+ id,
+ role: role as UIMessage['role'],
+ parts: [{ type: 'text', text }]
+ }
+}
+
+type SourceItem = Omit<DraftEditsItem, 'metadata'> & { id: string }
+
+function syntheticItems(): SourceItem[] {
+ const chats = datasetJson.chats as Record<string, RawChat>
+ return (datasetJson.items as RawItem[]).map(raw => {
+ const chat = chats[raw.chat]
+ if (!chat) {
+ throw new Error(`Item "${raw.id}" references unknown chat "${raw.chat}"`)
+ }
+ if (!isDraftEditsCategory(raw.category) || !isMode(chat.mode)) {
+ throw new Error(`Item "${raw.id}" has an unknown category or mode`)
+ }
+ const messages = [
+ ...chat.messages.map((message, index) =>
+ textMessage(`${raw.chat}-${index}`, message.role, message.text)
+ ),
+ textMessage(`${raw.id}-query`, 'user', raw.query)
+ ]
+ return {
+ id: raw.id,
+ input: { mode: chat.mode, messages, results: [] },
+ expectedOutput: { category: raw.category }
+ }
+ })
+}
+
+function productionItems(seedsPath: string): SourceItem[] {
+ if (!existsSync(seedsPath)) return []
+ const seeds = JSON.parse(readFileSync(seedsPath, 'utf8')) as {
+ items: SeedItem[]
+ }
+ return seeds.items.map(seed => {
+ if (!isDraftEditsCategory(seed.category) || !isMode(seed.mode)) {
+ throw new Error(`Seed "${seed.id}" has an unknown category or mode`)
+ }
+ return {
+ id: seed.id,
+ input: {
+ mode: seed.mode,
+ messages: seed.messages,
+ results: seed.results
+ },
+ expectedOutput: { category: seed.category }
+ }
+ })
+}
+
+export function loadDraftEditsDataset({
+ trials = 1,
+ ids,
+ seedsPath = LOCAL_SEEDS_PATH,
+ includeSynthetic = true
+}: {
+ trials?: number
+ ids?: string[]
+ seedsPath?: string
+ includeSynthetic?: boolean
+} = {}): DraftEditsItem[] {
+ const wanted = ids && ids.length > 0 ? new Set(ids) : null
+ const sources = [
+ ...(includeSynthetic
+ ? syntheticItems().map(item => ({
+ ...item,
+ source: 'synthetic' as const
+ }))
+ : []),
+ ...productionItems(seedsPath).map(item => ({
+ ...item,
+ source: 'production' as const
+ }))
+ ]
+
+ const items: DraftEditsItem[] = []
+ for (const { id, input, expectedOutput, source } of sources) {
+ if (wanted && !wanted.has(id)) continue
+ for (let trial = 1; trial <= trials; trial++) {
+ items.push({
+ input,
+ expectedOutput,
+ metadata: {
+ itemId: id,
+ category: expectedOutput.category,
+ source,
+ trial
+ }
+ })
+ }
+ }
+
+ if (wanted) {
+ const found = new Set(items.map(item => item.metadata.itemId))
+ const missing = [...wanted].filter(id => !found.has(id))
+ if (missing.length > 0) {
+ throw new Error(`Unknown item id(s): ${missing.join(', ')}`)
+ }
+ }
+
+ return items
+}
+
+export const draftEditsDatasetDescription = datasetJson.description
+
+export type DraftEditCall = {
+ input: unknown
+ /** How many changes the tool applied; 0 when it rejected every edit. */
+ applied: number
+}
+
+export type DraftTurnOutput = {
+ text: string
+ editCalls: DraftEditCall[]
+}
+
+export type DraftTurnAnalysis = {
+ /** editDraft was called at all, whether or not it applied anything. */
+ calledEdit: boolean
+ /** editDraft changed the draft. */
+ changedDraft: boolean
+ /** The answer holds a completed draft block with content. */
+ wroteBlock: boolean
+ /** The turn did what its category asks for. */
+ correct: boolean
+}
+
+export function analyzeDraftTurn(
+ output: DraftTurnOutput,
+ category: DraftEditsCategory
+): DraftTurnAnalysis {
+ const calledEdit = output.editCalls.length > 0
+ const changedDraft = output.editCalls.some(call => call.applied > 0)
+ const wroteBlock = parseDraftBlocks(output.text).some(
+ block => block.closed && block.content.trim().length > 0
+ )
+
+ const correct =
+ category === 'unrelated'
+ ? !calledEdit
+ : category === 'new'
+ ? wroteBlock && !calledEdit
+ : changedDraft || wroteBlock
+
+ return { calledEdit, changedDraft, wroteBlock, correct }
+}
diff --git a/evals/run-draft-edits.ts b/evals/run-draft-edits.ts
new file mode 100644
index 00000000..db69daa4
--- /dev/null
+++ b/evals/run-draft-edits.ts
@@ -0,0 +1,237 @@
+#!/usr/bin/env bun
+/**
+ * Measure whether the deployed configuration leaves a chat's draft alone when
+ * the user asks for something else, and still changes it when asked to.
+ *
+ * bun run eval:draft-edits
+ * bun run eval:draft-edits -- --trials 3
+ *
+ * See evals/README.md for what this measures and why.
+ */
+import { config as dotenvConfig } from 'dotenv'
+
+dotenvConfig({ path: '.env.local' })
+
+// See run-source-cards.ts: the mode prompts branch on this key.
+process.env.BRAVE_SEARCH_API_KEY ||= 'eval-pinned-general-provider'
+
+type Args = {
+ model?: string
+ trials: number
+ concurrency: number
+ runName?: string
+ gate: boolean
+ ids?: string[]
+ synthetic: boolean
+ out?: string
+}
+
+function parseArgs(argv: string[]): Args {
+ const args: Args = { trials: 1, concurrency: 5, gate: true, synthetic: true }
+
+ for (let i = 0; i < argv.length; i++) {
+ const flag = argv[i]
+ const next = () => {
+ const value = argv[++i]
+ if (value === undefined) throw new Error(`${flag} requires a value`)
+ return value
+ }
+
+ switch (flag) {
+ case '--model':
+ args.model = next()
+ break
+ case '--trials':
+ args.trials = Number(next())
+ break
+ case '--concurrency':
+ args.concurrency = Number(next())
+ break
+ case '--run-name':
+ args.runName = next()
+ break
+ case '--items':
+ args.ids = next()
+ .split(',')
+ .map(id => id.trim())
+ .filter(Boolean)
+ break
+ case '--no-synthetic':
+ args.synthetic = false
+ break
+ case '--out':
+ args.out = next()
+ break
+ case '--no-gate':
+ args.gate = false
+ break
+ case '--help':
+ console.log(
+ [
+ 'Usage: bun run eval:draft-edits [options]',
+ '',
+ ' --model <id> provider:model for every item (default: each item mode model)',
+ ' --trials <n> generations per dataset item (default: 1)',
+ ' --concurrency <n> parallel generations (default: 5)',
+ ' --run-name <name> exact Langfuse run name (default: name + timestamp)',
+ ' --items <a,b> run only these item ids',
+ ' --no-synthetic run only the local production seeds',
+ ' --out <name> write per-item results as JSON to evals/_local/<name>',
+ ' --no-gate report metrics without failing on threshold violations'
+ ].join('\n')
+ )
+ process.exit(0)
+ break
+ default:
+ throw new Error(`Unknown argument: ${flag}`)
+ }
+ }
+
+ if (!Number.isFinite(args.trials) || args.trials < 1) {
+ throw new Error('--trials must be a positive number')
+ }
+ if (!Number.isFinite(args.concurrency) || args.concurrency < 1) {
+ throw new Error('--concurrency must be a positive number')
+ }
+
+ return args
+}
+
+async function main() {
+ const args = parseArgs(process.argv.slice(2))
+
+ // Imported after dotenv: the provider registry reads API keys at module load.
+ const { mkdirSync, writeFileSync } = await import('fs')
+ const path = await import('path')
+ const { LangfuseClient } = await import('@langfuse/client')
+ const {
+ analyzeDraftTurn,
+ draftEditsDatasetDescription,
+ LOCAL_SEEDS_PATH,
+ loadDraftEditsDataset
+ } = await import('./lib/draft-edits')
+ const { applyGate } = await import('./lib/gate')
+ type Threshold = import('./lib/gate').Threshold
+ const { setupTracing } = await import('./lib/otel')
+ const { draftEditsEvaluator, draftEditsRunEvaluator, summarizeDraftEdits } =
+ await import('./evaluators/draft-edits')
+ const { createDraftTurnTask, DEFAULT_MODEL } = await import(
+ './tasks/draft-turn'
+ )
+ const thresholds = (await import('./thresholds.json')).default
+
+ // Outputs can quote production drafts, so they only go to the gitignored
+ // folder that holds the seeds.
+ const outPath =
+ args.out &&
+ path.join(path.dirname(LOCAL_SEEDS_PATH), path.basename(args.out))
+ if (outPath === LOCAL_SEEDS_PATH) {
+ throw new Error('--out must not overwrite the production seeds')
+ }
+
+ const data = loadDraftEditsDataset({
+ trials: args.trials,
+ ids: args.ids,
+ includeSynthetic: args.synthetic
+ })
+ const production = data.filter(item => item.metadata.source === 'production')
+ const tracing = setupTracing()
+
+ console.log(`model: ${args.model ?? DEFAULT_MODEL}`)
+ console.log(`items: ${data.length} (${args.trials} trial(s) per turn)`)
+ console.log(
+ `production: ${production.length} ${production.length === 0 ? `(no seeds at ${LOCAL_SEEDS_PATH})` : ''}`
+ )
+ console.log(`concurrency: ${args.concurrency}\n`)
+
+ const langfuse = new LangfuseClient()
+
+ const result = await langfuse.experiment.run({
+ name: 'draft-edits',
+ runName: args.runName,
+ description: draftEditsDatasetDescription,
+ metadata: { model: args.model ?? 'per-mode', trials: args.trials },
+ data,
+ task: createDraftTurnTask({ model: args.model, tracingEnabled: true }),
+ evaluators: [draftEditsEvaluator],
+ runEvaluators: [draftEditsRunEvaluator],
+ maxConcurrency: args.concurrency
+ })
+
+ // Item outputs can quote production drafts, so only ids and verdicts print.
+ for (const itemResult of result.itemResults) {
+ const metadata = itemResult.item.metadata as {
+ itemId: string
+ category: 'unrelated' | 'new' | 'edit'
+ trial: number
+ }
+ const analysis = analyzeDraftTurn(
+ itemResult.output as never,
+ metadata.category
+ )
+ console.log(
+ `${analysis.correct ? 'ok ' : 'FAIL'} ${metadata.category.padEnd(9)} ${metadata.itemId}#${metadata.trial} edit=${analysis.calledEdit} block=${analysis.wroteBlock}`
+ )
+ }
+
+ if (outPath) {
+ mkdirSync(path.dirname(outPath), { recursive: true })
+ writeFileSync(
+ outPath,
+ JSON.stringify(
+ result.itemResults.map(itemResult => ({
+ metadata: itemResult.item.metadata,
+ output: itemResult.output
+ })),
+ null,
+ 1
+ )
+ )
+ }
+
+ const completionRate = result.itemResults.length / data.length
+ const summary = summarizeDraftEdits(result.itemResults)
+ console.log('\ncompletionRate:', completionRate.toFixed(3))
+ console.log('summary:', JSON.stringify(summary, null, 2))
+
+ const config = thresholds['draft-edits'] as Record<string, Threshold>
+
+ try {
+ if (args.gate) {
+ applyGate(result, [
+ {
+ metric: 'completionRate',
+ value: completionRate,
+ samples: data.length,
+ threshold: config.completionRate
+ },
+ {
+ metric: 'hijackRate',
+ value: summary.hijackRate,
+ samples: summary.unrelatedCount,
+ threshold: config.hijackRate
+ },
+ {
+ metric: 'newBlockRate',
+ value: summary.newBlockRate,
+ samples: summary.newCount,
+ threshold: config.newBlockRate
+ },
+ {
+ metric: 'editSuccessRate',
+ value: summary.editSuccessRate,
+ samples: summary.editCount,
+ threshold: config.editSuccessRate
+ }
+ ])
+ }
+ } finally {
+ await tracing.flush()
+ if (result.datasetRunUrl) console.log(`\n${result.datasetRunUrl}`)
+ }
+}
+
+main().catch(error => {
+ console.error(error instanceof Error ? error.message : error)
+ process.exitCode = 1
+})
diff --git a/evals/tasks/draft-turn.ts b/evals/tasks/draft-turn.ts
new file mode 100644
index 00000000..ec46dc88
--- /dev/null
+++ b/evals/tasks/draft-turn.ts
@@ -0,0 +1,168 @@
+import { convertToModelMessages, generateText, isStepCount } from 'ai'
+
+import cloudConfig from '@/config/models/cloud.json'
+
+import {
+ getAdaptiveModePrompt,
+ getQuickModePrompt
+} from '@/lib/agents/prompts/search-mode-prompts'
+import { buildDraftVersions, currentDraftVersion } from '@/lib/draft/draft'
+import { capHistoricalAttachments } from '@/lib/streaming/helpers/cap-historical-attachments'
+import { capHistoricalPastedContent } from '@/lib/streaming/helpers/cap-historical-pasted-content'
+import { compactHistoricalMessages } from '@/lib/streaming/helpers/compact-historical-messages'
+import { convertDataPart } from '@/lib/streaming/helpers/convert-data-part'
+import { assignDataPartNonces } from '@/lib/streaming/helpers/data-part-nonce'
+import { dedupeAttachments } from '@/lib/streaming/helpers/dedupe-attachments'
+import {
+ attachCurrentDraft,
+ compactDraftsInMessages
+} from '@/lib/streaming/helpers/draft-context'
+import { stripSpecFromMessages } from '@/lib/streaming/helpers/strip-spec-from-messages'
+import { createDraftTools } from '@/lib/tools/draft'
+import { createSearchTool } from '@/lib/tools/search'
+import { createTodoTools } from '@/lib/tools/todo'
+import { getModel } from '@/lib/utils/registry'
+
+import {
+ type DraftEditsInput,
+ type DraftEditsMode,
+ type DraftTurnOutput
+} from '../lib/draft-edits'
+
+import {
+ createFixtureFetchTool,
+ createFixtureSearchTool
+} from './fixture-tools'
+import { type ProviderOptions } from './run-answer-task'
+
+const MODES = {
+ quick: {
+ config: cloudConfig.models.quick,
+ getPrompt: getQuickModePrompt,
+ includeTodoWrite: false
+ },
+ adaptive: {
+ config: cloudConfig.models.adaptive,
+ getPrompt: getAdaptiveModePrompt,
+ includeTodoWrite: true
+ }
+} as const
+
+// Both deployed modes run the same model today; each item replays its own
+// mode's prompt, tools, and providerOptions.
+export const DEFAULT_MODEL = `${MODES.quick.config.providerId}:${MODES.quick.config.id}`
+
+const MAX_STEPS = 20
+const DEFAULT_RETRIES = 2
+
+/**
+ * Prepare the chat the way the chat route does before the model sees it:
+ * compact historical drafts and answers, then attach the current draft to the
+ * latest user message.
+ */
+export function prepareDraftTurn(messages: DraftEditsInput['messages']) {
+ const currentDraft = currentDraftVersion(buildDraftVersions(messages))
+ const history = dedupeAttachments(
+ capHistoricalAttachments(
+ compactHistoricalMessages(
+ capHistoricalPastedContent(
+ compactDraftsInMessages(
+ stripSpecFromMessages(assignDataPartNonces(messages))
+ )
+ )
+ )
+ )
+ )
+ return {
+ currentDraft,
+ messages: attachCurrentDraft(history, currentDraft)
+ }
+}
+
+export function createDraftTurnTask({
+ model,
+ tracingEnabled = true,
+ retries = DEFAULT_RETRIES
+}: {
+ /** Overrides the deployed model for every mode. */
+ model?: string
+ tracingEnabled?: boolean
+ retries?: number
+} = {}) {
+ return async function answer(item: {
+ input?: unknown
+ }): Promise<DraftTurnOutput> {
+ const input = item.input as DraftEditsInput | undefined
+ if (!input?.mode || !Array.isArray(input.messages)) {
+ throw new Error('Experiment item input must be { mode, messages }')
+ }
+
+ let lastError: unknown
+ for (let attempt = 0; attempt <= retries; attempt++) {
+ try {
+ return await generate(input)
+ } catch (error) {
+ lastError = error
+ const message = error instanceof Error ? error.message : String(error)
+ console.warn(`[eval] attempt ${attempt + 1}/${retries + 1}: ${message}`)
+ }
+ }
+ throw lastError
+ }
+
+ async function generate(input: DraftEditsInput): Promise<DraftTurnOutput> {
+ const mode = MODES[input.mode as DraftEditsMode]
+ const modelId = model ?? `${mode.config.providerId}:${mode.config.id}`
+ const prepared = prepareDraftTurn(input.messages)
+ if (!prepared.currentDraft) {
+ throw new Error('Item has no draft to protect')
+ }
+
+ const tools = {
+ search: createFixtureSearchTool({
+ results: input.results ?? [],
+ toModelOutput: createSearchTool(modelId).toModelOutput,
+ onQuery: () => {}
+ }),
+ fetch: createFixtureFetchTool(input.results ?? []),
+ ...(mode.includeTodoWrite ? createTodoTools() : {}),
+ ...createDraftTools(prepared.currentDraft)
+ }
+
+ const result = await generateText({
+ model: getModel(modelId),
+ instructions: `${mode.getPrompt()}\nCurrent date: ${new Date().toLocaleDateString()}`,
+ tools,
+ stopWhen: isStepCount(MAX_STEPS),
+ providerOptions: mode.config.providerOptions as ProviderOptions,
+ messages: await convertToModelMessages(prepared.messages, {
+ convertDataPart
+ }),
+ telemetry: { isEnabled: tracingEnabled, functionId: 'eval-draft-turn' }
+ })
+
+ if (result.finishReason === 'tool-calls') {
+ throw new Error(`Step cap of ${MAX_STEPS} reached before an answer`)
+ }
+
+ const editCalls = result.steps.flatMap(step =>
+ step.toolResults
+ .filter(toolResult => toolResult.toolName === 'editDraft')
+ .map(toolResult => ({
+ input: toolResult.input,
+ applied: Number(
+ (toolResult.output as { applied?: unknown } | undefined)?.applied ??
+ 0
+ )
+ }))
+ )
+
+ // An edit with no closing sentence is still a complete turn here: what is
+ // measured is whether the draft was touched.
+ if (result.text.trim() === '' && editCalls.length === 0) {
+ throw new Error('Model produced no answer text')
+ }
+
+ return { text: result.text, editCalls }
+ }
+}
diff --git a/evals/thresholds.json b/evals/thresholds.json
index 9233bb98..ead836b8 100644
--- a/evals/thresholds.json
+++ b/evals/thresholds.json
@@ -50,5 +50,23 @@
"direction": "max",
"value": null
}
+ },
+ "draft-edits": {
+ "completionRate": {
+ "direction": "min",
+ "value": 0.9
+ },
+ "hijackRate": {
+ "direction": "max",
+ "value": null
+ },
+ "newBlockRate": {
+ "direction": "min",
+ "value": null
+ },
+ "editSuccessRate": {
+ "direction": "min",
+ "value": null
+ }
}
}
diff --git a/lib/draft/__tests__/draft.test.ts b/lib/draft/__tests__/draft.test.ts
index cd797c91..e69eed3a 100644
--- a/lib/draft/__tests__/draft.test.ts
+++ b/lib/draft/__tests__/draft.test.ts
@@ -283,7 +283,7 @@ describe('formatCurrentDraft', () => {
messageId: 'a1'
})
expect(text).toBe(
- `<current_draft title="A 'quoted' title" version="2">\nBody\n</current_draft>`
+ `The chat draft below is attached by the app, not written by the user. Change it only if the user asks to.\n<current_draft title="A 'quoted' title" version="2">\nBody\n</current_draft>`
)
})
})
diff --git a/lib/draft/draft.ts b/lib/draft/draft.ts
index 28a977bf..864aa24e 100644
--- a/lib/draft/draft.ts
+++ b/lib/draft/draft.ts
@@ -407,6 +407,7 @@ export function compactDraftBlocks(text: string): string {
/** The current draft, as attached to the latest user message for the model. */
export function formatCurrentDraft(version: DraftVersion): string {
return [
+ 'The chat draft below is attached by the app, not written by the user. Change it only if the user asks to.',
`<current_draft title="${version.title.replace(/"/g, "'")}" version="${version.version}">`,
version.content,
'</current_draft>'
diff --git a/lib/draft/prompt.ts b/lib/draft/prompt.ts
index 1cf636d9..f9857bfe 100644
--- a/lib/draft/prompt.ts
+++ b/lib/draft/prompt.ts
@@ -31,8 +31,11 @@ DRAFT RULES:
repeat the draft's content outside the block.
2. Do not put citation markers inside the draft. If the draft relies on search results, cite them in the sentence
outside the block.
-3. One draft per chat. When a draft exists, the latest message carries it in <current_draft>; change it with editDraft,
- or for a change across the whole draft (shorten, change the tone, restructure, translate), write a new complete
- draft block.
+3. When a draft exists, the app attaches it to the latest message in <current_draft> for reference; the user did not
+ write it there. Use editDraft only when the user asks to change that draft; for a change across the whole draft
+ (shorten, change the tone, restructure, translate), write a new complete draft block instead.
+4. Answer questions, lists, explanations, and any other unrelated request normally, and leave the draft unchanged.
+ When the user asks for a new piece on another topic, write a new complete draft block. Never rename or rewrite the
+ current draft into a different piece.
`
}
diff --git a/lib/streaming/helpers/__tests__/draft-context.test.ts b/lib/streaming/helpers/__tests__/draft-context.test.ts
index a6ac9604..89ec95c0 100644
--- a/lib/streaming/helpers/__tests__/draft-context.test.ts
+++ b/lib/streaming/helpers/__tests__/draft-context.test.ts
@@ -51,6 +51,9 @@ describe('attachCurrentDraft', () => {
expect(result[0]).toBe(first)
expect(result[2].parts).toHaveLength(2)
expect(textOf(result[2], 1)).toContain('<current_draft title="T"')
+ expect(textOf(result[2], 1)).toMatch(
+ /^The chat draft below is attached by the app/
+ )
})
it('leaves messages alone without a draft', () => {
diff --git a/lib/tools/draft.ts b/lib/tools/draft.ts
index 178fcf32..de60b7e6 100644
--- a/lib/tools/draft.ts
+++ b/lib/tools/draft.ts
@@ -8,7 +8,7 @@ export const editDraftInputSchema = z.object({
.string()
.optional()
.describe(
- 'A new title, only for a draft without a top-level heading. When the draft has one, edit the heading instead.'
+ 'A new title, only for a draft without a top-level heading. When the draft has one, edit the heading instead. Never use it to relabel the draft for a new topic.'
),
edits: z
.array(
@@ -33,7 +33,8 @@ export function createDraftTools(currentDraft: DraftVersion | null) {
let current = currentDraft
const editDraft = tool({
- description: `Make local changes to the current draft (a sentence, a paragraph, the title, one section, a passage the user quoted) with exact find/replace edits.
+ description: `Use only when the user asks to change the current draft. Never use it to answer an unrelated request or to turn the draft into a different piece.
+Make local changes to the current draft (a sentence, a paragraph, the title, one section, a passage the user quoted) with exact find/replace edits.
- Copy each find exactly from <current_draft>. It must appear exactly once; include enough surrounding text to make it unique. If edits are reported as failed, retry only those with corrected find text.
- When the user's message quotes text from the draft (a quoted-context block), that text is what "this" refers to: change only it, and make sure each find contains it.
- For a change across the whole draft, or more than about five places, write a new complete draft block instead.
diff --git a/package.json b/package.json
index 9bd8c49c..f7ef7c10 100644
--- a/package.json
+++ b/package.json
@@ -16,6 +16,7 @@
"test": "NODE_ENV=test vitest run",
"test:watch": "vitest --watch",
"chat": "bun run scripts/chat-cli.ts",
+ "eval:draft-edits": "bun run evals/run-draft-edits.ts",
"eval:related-questions": "bun run evals/run-related-questions.ts",
"eval:source-cards": "bun run evals/run-source-cards.ts"
},