Agent Zero AI framework
Refine document query image routing
Alessandro77552b939451
diff --git a/plugins/_document_query/AGENTS.md b/plugins/_document_query/AGENTS.mdindex c39208f94..3850d116d 100644--- a/plugins/_document_query/AGENTS.md+++ b/plugins/_document_query/AGENTS.md@@ -20,6 +20,7 @@## Work Guidance- Coordinate parser changes with timeout handling and fallback behavior.+- Keep prompt and skill guidance clear that image files, screenshots, scans, charts, photos, and diagrams should go to vision tools first when available; `document_query` image OCR is a text-focused fallback when vision is unavailable or cannot read the needed text.## Verificationdiff --git a/plugins/_document_query/prompts/agent.system.tool.document_query.md b/plugins/_document_query/prompts/agent.system.tool.document_query.mdindex cc2058ecb..d3d72078f 100644--- a/plugins/_document_query/prompts/agent.system.tool.document_query.md+++ b/plugins/_document_query/prompts/agent.system.tool.document_query.md@@ -1,7 +1,9 @@### document_query-read, extract, summarize, compare, OCR, or answer questions over local/remote documents, code files, and text-heavy document images/scans.+read, extract, summarize, compare, or answer questions over local/remote documents, PDFs, Office files, HTML/text/code files, large text-heavy files, and fallback OCR when vision tools cannot read a document image/scan.-For document Q&A, document/code analysis, multi-document comparison, or OCR/text extraction from images/scans, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.+For document Q&A, document/code analysis, multi-document comparison, PDF or large-file extraction, or fallback OCR after vision tools are unavailable or insufficient, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.++Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Call `document_query` for images only as document-style OCR fallback when vision cannot read the needed text.Minimal args after loading the skill:- `document`: one local path/URL or a list of paths/URLsdiff --git a/plugins/_document_query/skills/document-query/SKILL.md b/plugins/_document_query/skills/document-query/SKILL.mdindex b41a4d342..f82902b5a 100644--- a/plugins/_document_query/skills/document-query/SKILL.md+++ b/plugins/_document_query/skills/document-query/SKILL.md@@ -1,6 +1,6 @@---name: document-query-description: Use when reading, extracting, summarizing, comparing, OCRing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, and text-heavy document images or scans with the document_query tool.+description: Use when reading, extracting, summarizing, comparing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, large text-heavy files, and fallback OCR for document images or scans when vision tools are unavailable or insufficient.version: 1.0.0author: Agent Zero Teamtags: ["documents", "ocr", "qa", "pdf", "code", "analysis"]@@ -10,8 +10,8 @@ trigger_patterns:- ask questions about a document- summarize document- compare documents- - extract text from image- - OCR document+ - fallback image OCR+ - OCR document fallback- analyze code file---@@ -24,13 +24,15 @@ Use the `document_query` tool to read, extract, summarize, compare, or answer quUse `document_query` for:- Local files and URLs containing document text: PDF, HTML, Office files, plain text, Markdown, CSV/TSV, XML/JSON, logs, code files, and similar content.+- Large text-heavy files where ordinary reading/search would be too broad and the user needs Q&A, summarization, comparison, or extraction.- Q&A over one or more documents.- Summaries, comparisons, entity extraction, key-point extraction, and table/text extraction.- Code-file Q&A when the user points to specific files or URLs and wants answers from their contents.-- Text-heavy images, scans, screenshots, and document images when the task is OCR/text extraction or Q&A over visible text.-- Image OCR when vision tools are unavailable, when the main chat model is not multimodal, or when the user wants document text rather than visual scene understanding.+- Fallback OCR over text-heavy document images, scans, or screenshots only when vision tools are unavailable, insufficient, or unable to read the needed text.-Do not use `document_query` for purely visual questions that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.+Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Do not route images to `document_query` as a first-pass reader.++Do not use `document_query` for purely visual questions, first-pass screenshot inspection, or tasks that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.## Inputs@@ -111,14 +113,14 @@ For directories or codebases, first identify the relevant files with file/search}```-### OCR Or Q&A Over A Document Image+### Fallback OCR After Vision Cannot Read A Document Image```json{"thoughts": [- "The user wants text from a scanned document image."+ "Vision tools are unavailable or could not read the scanned document text, so document_query is the fallback OCR path."],- "headline": "Reading text from the scanned document",+ "headline": "Reading document text with fallback OCR","tool_name": "document_query","tool_args": {"document": "/a0/usr/workdir/scan.png",diff --git a/prompts/AGENTS.md b/prompts/AGENTS.mdindex 0728bbafe..646da3d6d 100644--- a/prompts/AGENTS.md+++ b/prompts/AGENTS.md@@ -23,6 +23,7 @@- Read the rendering path before changing placeholders or filenames.- Prefer small prompt additions over broad rewrites when fixing a specific behavior.+- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.- Update tests or snapshots when prompt budget, required sections, or generated system content changes.## Verificationdiff --git a/prompts/agent.system.main.tips.md b/prompts/agent.system.main.tips.mdindex 81d766263..a7b8a7689 100644--- a/prompts/agent.system.main.tips.md+++ b/prompts/agent.system.main.tips.md@@ -24,8 +24,8 @@ always use specialized subordinate agents for specialized tasks matching their p## Documents and OCR-use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs+use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs, especially PDFs, Office files, HTML/text files, logs, code files, and large files that need Q&Ause document_query for Q&A, summaries, comparisons, or extraction over specific code files when the user asks about file contents rather than asking to edit or search the codebase-use document_query for document images, screenshots, scans, and other image files when the task is text extraction/OCR or Q&A over document content-when vision tools are unavailable or the main chat model is not multimodal, use document_query for image OCR instead of asking the user to switch models+use vision_load first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when vision tools are available+use document_query for image OCR only when vision tools cannot read the image, vision tools are unavailable, or the user specifically needs document-style fallback OCR over visible textkeep parser/runtime details internal; users only need the document answerdiff --git a/tests/test_document_query_plugin.py b/tests/test_document_query_plugin.pyindex 1009edc45..e9fa25ef3 100644--- a/tests/test_document_query_plugin.py+++ b/tests/test_document_query_plugin.py@@ -559,9 +559,15 @@ def test_document_query_prompt_uses_progressive_skill_disclosure():assert skill is not Noneassert "document_query for Q&A" in main_promptassert "specific code files" in main_prompt+ assert "use vision_load first for image files" in main_prompt+ assert "document_query for image OCR only when vision tools cannot read" in main_promptassert "skills_tool:load" in promptassert "document-query" in promptassert "document_query" in prompt+ assert "Use vision tools first" in prompt+ assert "fallback OCR" in promptassert "answering questions over local or remote documents" in skill.description+ assert "fallback OCR" in skill.descriptionassert "### Answer Questions Over A Document" in skill.content- assert "### OCR Or Q&A Over A Document Image" in skill.content+ assert "Use vision tools first" in skill.content+ assert "### Fallback OCR After Vision Cannot Read A Document Image" in skill.content