Skip to content

https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/

fix(web): improve agent image search prompts and add agent integration docs (#33)

Yichuan Wang6e80426e2579
diff --git a/web/agent-server.mjs b/web/agent-server.mjs
index 0fcbc46..e66073f 100644
--- a/web/agent-server.mjs
+++ b/web/agent-server.mjs
@@ -55,7 +55,10 @@ const SYSTEM_PROMPT = `You are PixelRAG's research assistant. You answer using a
For every user question, without exception:
1. Call pixelrag_search to find relevant Wikipedia articles.
- - If the user uploaded an image, set use_uploaded_image: true to include it in the search (visual similarity) — the best way to answer "what/who is this?" or "which X does this most resemble?". You can add a text query in the SAME call to combine image + text into one joint search, and you may also run separate text searches for likely candidates to compare against. Always do the image search first when an image is present.
+ - If the user uploaded an image, you MUST set use_uploaded_image: true to search by visual similarity. Strategy depends on the query:
+ • For identification questions ("who/what is this?"): do image-only search FIRST (use_uploaded_image=true, NO text query) — the visual embedding alone gives the strongest match. Then do follow-up text searches to verify or compare candidates.
+ • For descriptive/specific questions ("what breed is this dog?", "which city is this skyline?"): combine image + a DESCRIPTIVE text query in the same call (use_uploaded_image=true AND query="dog breed" or "city skyline"). Use descriptive keywords about what you see, NOT the user's raw question.
+ • Never pass vague questions like "who is this" or "what is this" as the text query — they dilute the visual signal. Either omit text or use descriptive visual keywords.
- Otherwise pass a natural-language query.
2. Call pixelrag_tile to VIEW the screenshot tiles of the top results — this is how you read and compare. View at least 2-3 tiles.
3. Answer from what the tiles show, and cite the Wikipedia URLs. If the tiles don't contain the answer, say so honestly.
@@ -73,10 +76,10 @@ function log(...args) {
function createTools(onEvent, uploadedImage) {
const searchTool = tool(
"pixelrag_search",
- "Search the visual Wikipedia index by text OR by the image the user uploaded. Returns ranked results with article URLs, tile positions, and `pages` — the article's valid tile:chunk ranges (e.g. '0:0-7,1:0-4' = tile 0 has chunks 0-7, tile 1 has chunks 0-4). Use this first, then pixelrag_tile to view tiles.",
+ "Search the visual Wikipedia index by text, by the user's uploaded image, or BOTH combined. When the user uploaded an image, you MUST set use_uploaded_image=true AND provide a text query to get joint image+text retrieval — this gives the best results. Returns ranked results with article URLs, tile positions, and `pages` — the article's valid tile:chunk ranges (e.g. '0:0-7,1:0-4' = tile 0 has chunks 0-7, tile 1 has chunks 0-4). Use this first, then pixelrag_tile to view tiles.",
{
query: z.string().optional().describe("Natural language search query. Omit only when searching purely by an uploaded image."),
- use_uploaded_image: z.boolean().optional().describe("Set true to search the index BY the image the user uploaded (visual similarity). Requires an image in this conversation."),
+ use_uploaded_image: z.boolean().optional().describe("Set true to include the user's uploaded image in the search (visual similarity). ALWAYS combine with a text query for best results — set this AND provide a query string in the same call."),
n_results: z.number().int().min(1).max(20).optional().describe("Number of results (default 5)"),
},
async (args) => {
diff --git a/web/app/api/chat/route.ts b/web/app/api/chat/route.ts
index 728dd54..8ce8867 100644
--- a/web/app/api/chat/route.ts
+++ b/web/app/api/chat/route.ts
@@ -23,7 +23,10 @@ const SYSTEM_PROMPT = `You are PixelRAG's research assistant. You answer using a
For every user question, without exception:
1. Call pixelrag_search to find relevant Wikipedia articles.
- - If the user uploaded an image, set use_uploaded_image: true to include it in the search (visual similarity) — the best way to answer "what/who is this?" or "which X does this most resemble?". You can add a text query in the SAME call to combine image + text into one joint search, and you may also run separate text searches for likely candidates to compare against. Always do the image search first when an image is present.
+ - If the user uploaded an image, you MUST set use_uploaded_image: true to search by visual similarity. Strategy depends on the query:
+ • For identification questions ("who/what is this?"): do image-only search FIRST (use_uploaded_image=true, NO text query) — the visual embedding alone gives the strongest match. Then do follow-up text searches to verify or compare candidates.
+ • For descriptive/specific questions ("what breed is this dog?", "which city is this skyline?"): combine image + a DESCRIPTIVE text query in the same call (use_uploaded_image=true AND query="dog breed" or "city skyline"). Use descriptive keywords about what you see, NOT the user's raw question.
+ • Never pass vague questions like "who is this" or "what is this" as the text query — they dilute the visual signal. Either omit text or use descriptive visual keywords.
- Otherwise pass a natural-language query.
2. Call pixelrag_tile to VIEW the screenshot tiles of the top results — this is how you read and compare. View at least 2-3 tiles.
3. Answer from what the tiles show, and cite the Wikipedia URLs. If the tiles don't contain the answer, say so honestly.
@@ -44,7 +47,7 @@ function createTools(
) {
const searchTool = tool(
"pixelrag_search",
- "Search the visual Wikipedia index by text OR by the image the user uploaded. Returns ranked results with article URLs, tile positions, and `pages` — the article's valid tile:chunk ranges (e.g. '0:0-7,1:0-4' = tile 0 has chunks 0-7, tile 1 has chunks 0-4). Use this first, then pixelrag_tile to view tiles.",
+ "Search the visual Wikipedia index by text, by the user's uploaded image, or BOTH combined. When the user uploaded an image, you MUST set use_uploaded_image=true AND provide a text query to get joint image+text retrieval — this gives the best results. Returns ranked results with article URLs, tile positions, and `pages` — the article's valid tile:chunk ranges (e.g. '0:0-7,1:0-4' = tile 0 has chunks 0-7, tile 1 has chunks 0-4). Use this first, then pixelrag_tile to view tiles.",
{
query: z
.string()
@@ -53,7 +56,7 @@ function createTools(
use_uploaded_image: z
.boolean()
.optional()
- .describe("Set true to search the index BY the image the user uploaded (visual similarity). Requires an image in this conversation."),
+ .describe("Set true to include the user's uploaded image in the search (visual similarity). ALWAYS combine with a text query for best results — set this AND provide a query string in the same call."),
n_results: z
.number()
.int()
diff --git a/web/app/docs/page.tsx b/web/app/docs/page.tsx
index 6c1fe2d..e2b2f33 100644
--- a/web/app/docs/page.tsx
+++ b/web/app/docs/page.tsx
@@ -409,6 +409,36 @@ curl -X POST https://pixelrag.ai/api/search \\
</button>
))}
</div>
+
+ <h2 className="mt-10 text-xs font-semibold uppercase tracking-wider text-muted-foreground">
+ Connect your own agent
+ </h2>
+ <p className="mt-2 max-w-2xl text-sm leading-relaxed text-muted-foreground">
+ The search API is a plain HTTP endpoint &mdash; wire it into any agent framework
+ (Claude tool-use, OpenAI function calling, LangChain, custom harness) as a tool.
+ A typical agent loop: <strong className="font-medium text-foreground">search</strong> with
+ text and/or an image, <strong className="font-medium text-foreground">fetch tiles</strong> to
+ read the screenshots, then answer from what they show.
+ </p>
+ <ShellBlock
+ code={`# Text-only search
+curl -X POST https://pixelrag.ai/api/search \\
+ -H "Content-Type: application/json" \\
+ -d '{"queries": [{"text": "When was the Eiffel Tower built?"}], "n_docs": 5}'
+
+# Image-only search (visual similarity)
+curl -X POST https://pixelrag.ai/api/search \\
+ -H "Content-Type: application/json" \\
+ -d "{\\"queries\\": [{\\"image\\": \\"$(base64 < photo.jpg | tr -d '\\n')\\"}], \\"n_docs\\": 5}"
+
+# Joint image + text search (best for "what is this X?")
+curl -X POST https://pixelrag.ai/api/search \\
+ -H "Content-Type: application/json" \\
+ -d "{\\"queries\\": [{\\"image\\": \\"$(base64 < photo.jpg | tr -d '\\n')\\", \\"text\\": \\"landmark building\\"}], \\"n_docs\\": 5}"
+
+# Fetch a tile screenshot from the results
+curl https://pixelrag.ai/api/tile/698618/0/0 --output tile.png`}
+ />
</div>
)
}