An AI-powered search engine with a generative UI
TypeScript generative-aigenerative-uinextjsollamareactsearxngshadcn-uitailwindcsstavilytypescriptupstashvercel-ai-sdk
fix(prompts): show an image block in the worked answer examples (#938)
Yoshiki Miurac60c6586110e
diff --git a/lib/agents/prompts/search-mode-prompts.ts b/lib/agents/prompts/search-mode-prompts.tsindex 8847a989..a5bebda8 100644--- a/lib/agents/prompts/search-mode-prompts.ts+++ b/lib/agents/prompts/search-mode-prompts.ts@@ -1,4 +1,5 @@import {+ EXAMPLE_IMAGE_SPEC_BLOCK,getImageSpecPrompt,getRelatedQuestionsSpecPrompt} from '@/lib/render/prompt'@@ -86,13 +87,13 @@ Fetch tool usage:Citation Format (MANDATORY):[number](#toolCallId) - Always use this EXACT format- **CRITICAL**: Use the EXACT tool call identifier from the search response- - Find the tool call ID in the search response (e.g., "I8NzFUKwrKX88107")- - Use it directly without adding any prefix: [1](#I8NzFUKwrKX88107)+ - Find the tool call ID in the search response (e.g., "EXAMPLE_TOOL_CALL_ID_1")+ - Use it directly without adding any prefix: [1](#EXAMPLE_TOOL_CALL_ID_1)- The format is: [number](#TOOLCALLID) where TOOLCALLID is the exact ID- **CRITICAL RULE**: Each unique toolCallId gets ONE number. Never use different numbers with the same toolCallId.- ✓ CORRECT: "Fact A [1](#abc123). Fact B from same search [1](#abc123)."- ✓ CORRECT: "Fact A [1](#abc123). Fact B from different search [2](#def456)."- ✗ WRONG: "Fact A [1](#abc123). Fact B [2](#abc123)." (Same toolCallId cannot have different numbers)+ ✓ CORRECT: "Fact A [1](#EXAMPLE_TOOL_CALL_ID_1). Fact B from same search [1](#EXAMPLE_TOOL_CALL_ID_1)."+ ✓ CORRECT: "Fact A [1](#EXAMPLE_TOOL_CALL_ID_1). Fact B from different search [2](#EXAMPLE_TOOL_CALL_ID_2)."+ ✗ WRONG: "Fact A [1](#EXAMPLE_TOOL_CALL_ID_1). Fact B [2](#EXAMPLE_TOOL_CALL_ID_1)." (Same toolCallId cannot have different numbers)- Assign numbers sequentially (1, 2, 3...) to each unique toolCallId as they appear in your response- **CRITICAL CITATION PLACEMENT RULES**:1. Write the COMPLETE sentence first@@ -102,18 +103,18 @@ Citation Format (MANDATORY):5. If using multiple sources in one sentence, place ALL citations together after the period**CORRECT PATTERN**: sentence. [citation]- ✓ CORRECT: "Nvidia's GPUs power AI models. [1](#abc123)"- ✓ CORRECT: "Nvidia leads in hardware and software. [1](#abc123) [2](#def456)"+ ✓ CORRECT: "Nvidia's GPUs power AI models. [1](#EXAMPLE_TOOL_CALL_ID_1)"+ ✓ CORRECT: "Nvidia leads in hardware and software. [1](#EXAMPLE_TOOL_CALL_ID_1) [2](#EXAMPLE_TOOL_CALL_ID_2)"**WRONG PATTERNS** (Do NOT do this):- ✗ WRONG: "Nvidia's GPUs power AI models [1](#abc123)." (citation BEFORE period)- ✗ WRONG: "Nvidia's GPUs. [1](#abc123) power AI models." (citation breaks sentence)- ✗ WRONG: "Nvidia leads in hardware and software. [1](#abc123), [2](#def456)" (comma between citations)+ ✗ WRONG: "Nvidia's GPUs power AI models [1](#EXAMPLE_TOOL_CALL_ID_1)." (citation BEFORE period)+ ✗ WRONG: "Nvidia's GPUs. [1](#EXAMPLE_TOOL_CALL_ID_1) power AI models." (citation breaks sentence)+ ✗ WRONG: "Nvidia leads in hardware and software. [1](#EXAMPLE_TOOL_CALL_ID_1), [2](#EXAMPLE_TOOL_CALL_ID_2)" (comma between citations)- Every sentence with information from search results MUST have citations at its end-Citation Example with Real Tool Call:-If tool call ID is "I8NzFUKwrKX88107", cite as: [1](#I8NzFUKwrKX88107)-If tool call ID is "ABC123xyz", cite as: [2](#ABC123xyz)+Citation Example with Placeholder Tool Call:+If tool call ID is "EXAMPLE_TOOL_CALL_ID_1", cite as: [1](#EXAMPLE_TOOL_CALL_ID_1)+If tool call ID is "EXAMPLE_TOOL_CALL_ID_2", cite as: [2](#EXAMPLE_TOOL_CALL_ID_2)Rule precedence:- The one-search limit is mandatory and overrides any instruction that could imply additional research.@@ -144,13 +145,15 @@ Emoji usage:Example approach:## **Topic Response**### Core Information-- **Key Point:** Direct answer with specific data/numbers when available [1](#I8NzFUKwrKX88107)-- **Detail:** Supporting information with concrete examples [2](#I8NzFUKwrKX88107)+- **Key Point:** Direct answer with specific data/numbers when available [1](#EXAMPLE_TOOL_CALL_ID_1)+- **Detail:** Supporting information with concrete examples [2](#EXAMPLE_TOOL_CALL_ID_1)++${EXAMPLE_IMAGE_SPEC_BLOCK}### When Comparing (use table format)| Feature | Option A | Option B ||---------|----------|----------|-| Price | $100 [1](#abc123) | $150 [2](#def456) |+| Price | $100 [1](#EXAMPLE_TOOL_CALL_ID_1) | $150 [2](#EXAMPLE_TOOL_CALL_ID_2) |### Additional Context (if relevant)- **Consideration:** Practical implications with real-world context@@ -202,7 +205,7 @@ Rule precedence:4. **If the query is ambiguous, use ask_question tool for clarification**-5. **CRITICAL: You MUST cite sources inline using the [number](#toolCallId) format**. **CITATION PLACEMENT**: Follow this pattern: sentence. [citation] - Write the complete sentence, add a period, then add citations after the period. Do NOT add period or punctuation after citations. If a sentence uses multiple sources, place ALL citations together after the period (e.g., "AI adoption has increased. [1](#I8NzFUKwrKX88107) [2](#aHvy9Vt17r3VSmnG)"). Use [1](#toolCallId), [2](#toolCallId), [3](#toolCallId), etc., where number matches the order within each search result and toolCallId is the ID of the search that provided the result. Every sentence with information from search results MUST have citations at its end.+5. **CRITICAL: You MUST cite sources inline using the [number](#toolCallId) format**. **CITATION PLACEMENT**: Follow this pattern: sentence. [citation] - Write the complete sentence, add a period, then add citations after the period. Do NOT add period or punctuation after citations. If a sentence uses multiple sources, place ALL citations together after the period (e.g., "AI adoption has increased. [1](#EXAMPLE_TOOL_CALL_ID_1) [2](#EXAMPLE_TOOL_CALL_ID_2)"). Use [1](#toolCallId), [2](#toolCallId), [3](#toolCallId), etc., where number matches the order within each search result and toolCallId is the ID of the search that provided the result. Every sentence with information from search results MUST have citations at its end.6. If results are not relevant or helpful, you may rely on your general knowledge ONLY AFTER at least one search attempt (do not add citations for general knowledge)@@ -261,10 +264,10 @@ When using the ask_question tool:- Match the language to the user's language (except option values which must be in English)Citation Format:-[number](#toolCallId) - Always use this EXACT format, e.g., [1](#I8NzFUKwrKX88107), [2](#aHvy9Vt17r3VSmnG)+[number](#toolCallId) - Always use this EXACT format, e.g., [1](#EXAMPLE_TOOL_CALL_ID_1), [2](#EXAMPLE_TOOL_CALL_ID_2)- The number corresponds to the result order within each search (1, 2, 3, etc.)- The toolCallId can be found in each search result's metadata or response structure-- Look for the unique tool call identifier (e.g., mK3pQr7sT9uV2wX4) in the search response+- Look for the unique tool call identifier (e.g., EXAMPLE_TOOL_CALL_ID_1) in the search response- The toolCallId is the EXACT unique identifier of the search tool call- Do NOT add ANY prefix (such as "toolu_", "call_", or "search-") to the toolCallId — use the exact ID exactly as it appears in the search response- Each search tool execution will have its own toolCallId@@ -276,16 +279,16 @@ Citation Format:5. If using multiple sources in one sentence, place ALL citations together after the period**CORRECT PATTERN**: sentence. [citation]- ✓ CORRECT: "Nvidia's stock has risen 200%. [1](#I8NzFUKwrKX88107)"- ✓ CORRECT: "Nvidia leads in hardware and software. [1](#abc123) [2](#def456)"+ ✓ CORRECT: "Nvidia's stock has risen 200%. [1](#EXAMPLE_TOOL_CALL_ID_1)"+ ✓ CORRECT: "Nvidia leads in hardware and software. [1](#EXAMPLE_TOOL_CALL_ID_1) [2](#EXAMPLE_TOOL_CALL_ID_2)"**WRONG PATTERNS** (Do NOT do this):- ✗ WRONG: "Nvidia's stock has risen 200% [1](#I8NzFUKwrKX88107)." (citation BEFORE period)- ✗ WRONG: "Nvidia's stock. [1](#I8NzFUKwrKX88107) has risen 200%." (citation breaks sentence)- ✗ WRONG: "Nvidia leads in hardware and software. [1](#abc123], [2](#def456)" (comma between citations)+ ✗ WRONG: "Nvidia's stock has risen 200% [1](#EXAMPLE_TOOL_CALL_ID_1)." (citation BEFORE period)+ ✗ WRONG: "Nvidia's stock. [1](#EXAMPLE_TOOL_CALL_ID_1) has risen 200%." (citation breaks sentence)+ ✗ WRONG: "Nvidia leads in hardware and software. [1](#EXAMPLE_TOOL_CALL_ID_1], [2](#EXAMPLE_TOOL_CALL_ID_2)" (comma between citations)IMPORTANT: Citations must appear INLINE within your response text, not separately.-Example: "The company reported record revenue. [1](#I8NzFUKwrKX88107) Analysts predict continued growth. [2](#I8NzFUKwrKX88107)"-Example with multiple searches: "Initial data shows positive trends. [1](#I8NzFUKwrKX88107) Recent updates indicate acceleration. [1](#aHvy9Vt17r3VSmnG)"+Example: "The company reported record revenue. [1](#EXAMPLE_TOOL_CALL_ID_1) Analysts predict continued growth. [2](#EXAMPLE_TOOL_CALL_ID_1)"+Example with multiple searches: "Initial data shows positive trends. [1](#EXAMPLE_TOOL_CALL_ID_1) Recent updates indicate acceleration. [1](#EXAMPLE_TOOL_CALL_ID_2)"TASK MANAGEMENT (todoWrite tool):**When to use todoWrite:**@@ -328,9 +331,11 @@ Emoji usage:Flexible example:## **Response Topic**### Primary Information-- **Core Answer:** Direct response with evidence [1](#I8NzFUKwrKX88107)+- **Core Answer:** Direct response with evidence [1](#EXAMPLE_TOOL_CALL_ID_1)- **Context:** Relevant supporting details+${EXAMPLE_IMAGE_SPEC_BLOCK}+Conclude with a brief synthesis that ties together the main insights into a clear overall understanding.${getImageSpecPrompt()}diff --git a/lib/render/__tests__/image.test.tsx b/lib/render/__tests__/image.test.tsxnew file mode 100644index 00000000..cc183b57--- /dev/null+++ b/lib/render/__tests__/image.test.tsx@@ -0,0 +1,31 @@+import React from 'react'++import { render } from '@testing-library/react'+import { describe, expect, test, vi } from 'vitest'++import { Image } from '../components/image'++vi.mock('@/lib/analytics/posthog-client', () => ({+ captureClient: vi.fn(),+ chatIdFromPath: vi.fn()+}))++const SpecImage = Image as unknown as React.FC<{ props: { src: string } }>++function renderImage(src: string) {+ return render(<SpecImage props={{ src }} />)+}++describe('render Image', () => {+ test('renders a web image', () => {+ const { container } = renderImage('https://example.org/photo.jpg')++ expect(container.querySelector('img')).not.toBeNull()+ })++ test('drops a source that is not a web url', () => {+ const { container } = renderImage('EXAMPLE_IMAGE_1')++ expect(container.querySelector('img')).toBeNull()+ })+})diff --git a/lib/render/__tests__/prompt.test.ts b/lib/render/__tests__/prompt.test.tsindex bbd88db1..e797ea03 100644--- a/lib/render/__tests__/prompt.test.ts+++ b/lib/render/__tests__/prompt.test.ts@@ -1,6 +1,15 @@import { describe, expect, test } from 'vitest'-import { getImageSpecPrompt, getRelatedQuestionsSpecPrompt } from '../prompt'+import {+ getAdaptiveModePrompt,+ getQuickModePrompt+} from '@/lib/agents/prompts/search-mode-prompts'++import {+ EXAMPLE_IMAGE_SPEC_BLOCK,+ getImageSpecPrompt,+ getRelatedQuestionsSpecPrompt+} from '../prompt'describe('render prompts', () => {test('includes related questions by default for substantive answers', () => {@@ -31,4 +40,38 @@ describe('render prompts', () => {test('describes the related block as optional alongside image specs', () => {expect(getImageSpecPrompt()).toContain('which is itself optional')})++ test.each([getQuickModePrompt, getAdaptiveModePrompt])(+ 'includes the canonical image block in the worked example',+ getPrompt => {+ const prompt = getPrompt()++ expect(prompt).toContain(EXAMPLE_IMAGE_SPEC_BLOCK)+ expect(prompt.split(EXAMPLE_IMAGE_SPEC_BLOCK)).toHaveLength(3)+ }+ )++ test.each([getQuickModePrompt, getAdaptiveModePrompt])(+ 'uses placeholders for example tool calls and image sources',+ getPrompt => {+ const prompt = getPrompt()+ const oldToolCallIds = [+ 'I8NzFUKwrKX88107',+ 'aHvy9Vt17r3VSmnG',+ 'abc123',+ 'def456'+ ]+ const imageSources = [...prompt.matchAll(/"src":"([^"]+)"/g)].map(+ match => match[1]+ )++ for (const toolCallId of oldToolCallIds) {+ expect(prompt).not.toContain(toolCallId)+ }+ expect(imageSources.length).toBeGreaterThan(0)+ expect(imageSources).toEqual(+ imageSources.filter(source => source.startsWith('EXAMPLE_IMAGE_'))+ )+ }+ )})diff --git a/lib/render/components/image.tsx b/lib/render/components/image.tsxindex 9263a32f..71430617 100644--- a/lib/render/components/image.tsx+++ b/lib/render/components/image.tsx@@ -30,7 +30,9 @@ export const Image: ComponentFn<CatalogType, 'Image'> = ({ props }) => {const { src, sourceUrl, title, description, aspectRatio = '4:3' } = propsconst [errored, setErrored] = useState(false)- if (!src || errored) {+ // Only real web images render. Anything else, including the prompt's+ // EXAMPLE_ placeholders if a model ever copies them, is dropped.+ if (!src || errored || !/^https?:\/\//i.test(src)) {return null}diff --git a/lib/render/prompt.ts b/lib/render/prompt.tsindex d6a1bc08..46950267 100644--- a/lib/render/prompt.ts+++ b/lib/render/prompt.ts@@ -59,6 +59,13 @@ SPEC RULES:`}+export const EXAMPLE_IMAGE_SPEC_BLOCK = `\`\`\`spec+{"op":"add","path":"/root","value":"grid"}+{"op":"add","path":"/elements/grid","value":{"type":"Grid","props":{"columns":2,"gap":"sm"},"children":["img1","img2"]}}+{"op":"add","path":"/elements/img1","value":{"type":"Image","props":{"src":"EXAMPLE_IMAGE_1","sourceUrl":"EXAMPLE_SOURCE_1","title":"Mount Fuji at sunrise","description":"Snow-capped peak at sunrise","aspectRatio":"4:3"},"children":[]}}+{"op":"add","path":"/elements/img2","value":{"type":"Image","props":{"src":"EXAMPLE_IMAGE_2","sourceUrl":"EXAMPLE_SOURCE_2","title":"Mount Fuji in spring","description":"Cherry blossoms framing Mount Fuji","aspectRatio":"4:3"},"children":[]}}+\`\`\``+/*** Returns the prompt section that instructs the LLM to optionally embed* inline image groups as ```spec fenced blocks within the response body.@@ -98,16 +105,15 @@ Example (inline image group embedded in markdown body):## Mount Fuji-Mount Fuji is Japan's tallest peak.+### Key facts+- **Elevation:** Mount Fuji rises 3,776 meters above sea level.+- **Setting:** Its snow-capped peak is a defining feature of the landscape.-\`\`\`spec-{"op":"add","path":"/root","value":"grid"}-{"op":"add","path":"/elements/grid","value":{"type":"Grid","props":{"columns":3,"gap":"sm"},"children":["img1","img2","img3"]}}-{"op":"add","path":"/elements/img1","value":{"type":"Image","props":{"src":"https://cdn.example.com/fuji-1.jpg","sourceUrl":"https://en.wikipedia.org/wiki/Mount_Fuji","title":"Mount Fuji - Wikipedia","description":"Snow-capped peak at sunrise","aspectRatio":"4:3"},"children":[]}}-{"op":"add","path":"/elements/img2","value":{"type":"Image","props":{"src":"https://cdn.example.com/fuji-2.jpg","sourceUrl":"https://travel.example.com/mount-fuji","title":"Mount Fuji Travel Guide","aspectRatio":"4:3"},"children":[]}}-{"op":"add","path":"/elements/img3","value":{"type":"Image","props":{"src":"https://cdn.example.com/fuji-3.jpg","title":"Cherry blossoms in spring","aspectRatio":"4:3"},"children":[]}}-\`\`\`+${EXAMPLE_IMAGE_SPEC_BLOCK}++### Why it matters+- **Significance:** The mountain is both a natural landmark and a cultural symbol.-It rises 3,776 meters above sea level...+Mount Fuji combines geographic prominence with enduring cultural importance.`}