Skip to content

Agent Zero AI framework

Refine document query image routing

Alessandro77552b939451
diff --git a/plugins/_document_query/AGENTS.md b/plugins/_document_query/AGENTS.md
index c39208f94..3850d116d 100644
--- a/plugins/_document_query/AGENTS.md
+++ b/plugins/_document_query/AGENTS.md
@@ -20,6 +20,7 @@
## Work Guidance
- Coordinate parser changes with timeout handling and fallback behavior.
+- Keep prompt and skill guidance clear that image files, screenshots, scans, charts, photos, and diagrams should go to vision tools first when available; `document_query` image OCR is a text-focused fallback when vision is unavailable or cannot read the needed text.
## Verification
diff --git a/plugins/_document_query/prompts/agent.system.tool.document_query.md b/plugins/_document_query/prompts/agent.system.tool.document_query.md
index cc2058ecb..d3d72078f 100644
--- a/plugins/_document_query/prompts/agent.system.tool.document_query.md
+++ b/plugins/_document_query/prompts/agent.system.tool.document_query.md
@@ -1,7 +1,9 @@
### document_query
-read, extract, summarize, compare, OCR, or answer questions over local/remote documents, code files, and text-heavy document images/scans.
+read, extract, summarize, compare, or answer questions over local/remote documents, PDFs, Office files, HTML/text/code files, large text-heavy files, and fallback OCR when vision tools cannot read a document image/scan.
-For document Q&A, document/code analysis, multi-document comparison, or OCR/text extraction from images/scans, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.
+For document Q&A, document/code analysis, multi-document comparison, PDF or large-file extraction, or fallback OCR after vision tools are unavailable or insufficient, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.
+
+Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Call `document_query` for images only as document-style OCR fallback when vision cannot read the needed text.
Minimal args after loading the skill:
- `document`: one local path/URL or a list of paths/URLs
diff --git a/plugins/_document_query/skills/document-query/SKILL.md b/plugins/_document_query/skills/document-query/SKILL.md
index b41a4d342..f82902b5a 100644
--- a/plugins/_document_query/skills/document-query/SKILL.md
+++ b/plugins/_document_query/skills/document-query/SKILL.md
@@ -1,6 +1,6 @@
---
name: document-query
-description: Use when reading, extracting, summarizing, comparing, OCRing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, and text-heavy document images or scans with the document_query tool.
+description: Use when reading, extracting, summarizing, comparing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, large text-heavy files, and fallback OCR for document images or scans when vision tools are unavailable or insufficient.
version: 1.0.0
author: Agent Zero Team
tags: ["documents", "ocr", "qa", "pdf", "code", "analysis"]
@@ -10,8 +10,8 @@ trigger_patterns:
- ask questions about a document
- summarize document
- compare documents
- - extract text from image
- - OCR document
+ - fallback image OCR
+ - OCR document fallback
- analyze code file
---
@@ -24,13 +24,15 @@ Use the `document_query` tool to read, extract, summarize, compare, or answer qu
Use `document_query` for:
- Local files and URLs containing document text: PDF, HTML, Office files, plain text, Markdown, CSV/TSV, XML/JSON, logs, code files, and similar content.
+- Large text-heavy files where ordinary reading/search would be too broad and the user needs Q&A, summarization, comparison, or extraction.
- Q&A over one or more documents.
- Summaries, comparisons, entity extraction, key-point extraction, and table/text extraction.
- Code-file Q&A when the user points to specific files or URLs and wants answers from their contents.
-- Text-heavy images, scans, screenshots, and document images when the task is OCR/text extraction or Q&A over visible text.
-- Image OCR when vision tools are unavailable, when the main chat model is not multimodal, or when the user wants document text rather than visual scene understanding.
+- Fallback OCR over text-heavy document images, scans, or screenshots only when vision tools are unavailable, insufficient, or unable to read the needed text.
-Do not use `document_query` for purely visual questions that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.
+Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Do not route images to `document_query` as a first-pass reader.
+
+Do not use `document_query` for purely visual questions, first-pass screenshot inspection, or tasks that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.
## Inputs
@@ -111,14 +113,14 @@ For directories or codebases, first identify the relevant files with file/search
}
```
-### OCR Or Q&A Over A Document Image
+### Fallback OCR After Vision Cannot Read A Document Image
```json
{
"thoughts": [
- "The user wants text from a scanned document image."
+ "Vision tools are unavailable or could not read the scanned document text, so document_query is the fallback OCR path."
],
- "headline": "Reading text from the scanned document",
+ "headline": "Reading document text with fallback OCR",
"tool_name": "document_query",
"tool_args": {
"document": "/a0/usr/workdir/scan.png",
diff --git a/prompts/AGENTS.md b/prompts/AGENTS.md
index 0728bbafe..646da3d6d 100644
--- a/prompts/AGENTS.md
+++ b/prompts/AGENTS.md
@@ -23,6 +23,7 @@
- Read the rendering path before changing placeholders or filenames.
- Prefer small prompt additions over broad rewrites when fixing a specific behavior.
+- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.
- Update tests or snapshots when prompt budget, required sections, or generated system content changes.
## Verification
diff --git a/prompts/agent.system.main.tips.md b/prompts/agent.system.main.tips.md
index 81d766263..a7b8a7689 100644
--- a/prompts/agent.system.main.tips.md
+++ b/prompts/agent.system.main.tips.md
@@ -24,8 +24,8 @@ always use specialized subordinate agents for specialized tasks matching their p
## Documents and OCR
-use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs
+use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs, especially PDFs, Office files, HTML/text files, logs, code files, and large files that need Q&A
use document_query for Q&A, summaries, comparisons, or extraction over specific code files when the user asks about file contents rather than asking to edit or search the codebase
-use document_query for document images, screenshots, scans, and other image files when the task is text extraction/OCR or Q&A over document content
-when vision tools are unavailable or the main chat model is not multimodal, use document_query for image OCR instead of asking the user to switch models
+use vision_load first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when vision tools are available
+use document_query for image OCR only when vision tools cannot read the image, vision tools are unavailable, or the user specifically needs document-style fallback OCR over visible text
keep parser/runtime details internal; users only need the document answer
diff --git a/tests/test_document_query_plugin.py b/tests/test_document_query_plugin.py
index 1009edc45..e9fa25ef3 100644
--- a/tests/test_document_query_plugin.py
+++ b/tests/test_document_query_plugin.py
@@ -559,9 +559,15 @@ def test_document_query_prompt_uses_progressive_skill_disclosure():
assert skill is not None
assert "document_query for Q&A" in main_prompt
assert "specific code files" in main_prompt
+ assert "use vision_load first for image files" in main_prompt
+ assert "document_query for image OCR only when vision tools cannot read" in main_prompt
assert "skills_tool:load" in prompt
assert "document-query" in prompt
assert "document_query" in prompt
+ assert "Use vision tools first" in prompt
+ assert "fallback OCR" in prompt
assert "answering questions over local or remote documents" in skill.description
+ assert "fallback OCR" in skill.description
assert "### Answer Questions Over A Document" in skill.content
- assert "### OCR Or Q&A Over A Document Image" in skill.content
+ assert "Use vision tools first" in skill.content
+ assert "### Fallback OCR After Vision Cannot Read A Document Image" in skill.content