Skip to content

Agent Zero AI framework

Restore route-agnostic vision tool contract

Alessandro540ef8d81543
diff --git a/helpers/parallel_tools.py b/helpers/parallel_tools.py
index 382025bbc..dbc79ef9f 100644
--- a/helpers/parallel_tools.py
+++ b/helpers/parallel_tools.py
@@ -485,6 +485,7 @@ async def _run_direct_tool_job(parent_context_id: str, job: ParallelJob) -> str:
_copy_project(parent_context, worker_context)
worker_agent = worker_context.agent0
+ worker_agent.last_user_message = parent_context.agent0.last_user_message
worker_agent.loop_data = LoopData()
return await execute_tool_call(
worker_agent,
diff --git a/helpers/parallel_tools.py.dox.md b/helpers/parallel_tools.py.dox.md
index 28fbbadcd..9fb3f6b5f 100644
--- a/helpers/parallel_tools.py.dox.md
+++ b/helpers/parallel_tools.py.dox.md
@@ -32,7 +32,7 @@
- Subordinate child chats are tagged with job metadata, remain outside the scheduler task list, and may use normal child-chat tools including `parallel`.
- Nested parallel jobs started by a parallel subordinate are registered as child `DeferredTask` instances so stopping the ancestor also stops its descendants.
- Direct tool jobs run in isolated background contexts and are blocked from recursively invoking `parallel`.
-- Direct tool jobs inherit the parent's active per-chat model override.
+- Direct tool jobs inherit the parent's active per-chat model override and current user message.
- Direct tool background context cleanup removes both the in-memory context and any transient chat folder left on disk.
- Parent-visible child log items are created for each wrapped call so the WebUI can inspect concurrent children separately while the wrapper result remains model-history-only.
- Child tool logs mirror normal tool-call visible args; job ids remain available through wrapper results and prompt extras rather than visible process-step args.
diff --git a/prompts/AGENTS.md b/prompts/AGENTS.md
index 07c4d5282..10917ba20 100644
--- a/prompts/AGENTS.md
+++ b/prompts/AGENTS.md
@@ -25,7 +25,7 @@
- Read the rendering path before changing placeholders or filenames.
- Prefer small prompt additions over broad rewrites when fixing a specific behavior.
- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.
-- Keep the single `vision_load` prompt accurate for both native and separate Vision Model routing: related images belong in one call, and a Vision Model result is text-only.
+- Keep the single `vision_load` tool prompt route-agnostic and preserve one-call loading for related images; internal image-analysis instructions belong in framework prompts, not Python strings.
- Update tests or snapshots when prompt budget, required sections, or generated system content changes.
## Verification
diff --git a/prompts/agent.system.tools_vision.md b/prompts/agent.system.tools_vision.md
index b766699cf..ee50376ad 100644
--- a/prompts/agent.system.tools_vision.md
+++ b/prompts/agent.system.tools_vision.md
@@ -1,40 +1,23 @@
## multimodal vision tools
### vision_load
-load or analyze images for visual reasoning
-args: `paths` list of absolute image paths or ephemeral image refs, `query` optional focused instruction
-Input schema for tool_args:
-```json
-{
- "type": "object",
- "properties": {
- "paths": {
- "type": "array",
- "items": {"type": "string"}
- },
- "query": {"type": "string"}
- },
- "required": ["paths"],
- "additionalProperties": false
-}
-```
+load images into the model for visual reasoning
+args: `paths` list of absolute image paths or tool-returned ephemeral image refs
rules:
-- put all images needed for one comparison or visual task in the same `paths` array
+- load all relevant images in one call when comparing screenshots or pages
- use when the task depends on screenshots, diagrams, scanned documents, charts, or photos
-- use a focused `query` when asking the configured Vision Model to inspect, compare, locate, or read something
- only bitmaps are supported; convert other formats first if needed
-- the tool result reports loaded and skipped image counts
+- the tool result includes loaded/skipped image totals and the corresponding path lists
example:
```json
{
"thoughts": [
"I need to inspect the screenshot before answering."
],
- "headline": "Comparing screenshots",
+ "headline": "Loading screenshot for visual analysis",
"tool_name": "vision_load",
"tool_args": {
- "paths": ["/path/to/before.png", "/path/to/after.png"],
- "query": "Compare the error banners and describe what changed."
+ "paths": ["/path/to/screenshot.png"]
}
}
```
diff --git a/prompts/fw.vision_load.md b/prompts/fw.vision_load.md
new file mode 100644
index 000000000..1c7320e2f
--- /dev/null
+++ b/prompts/fw.vision_load.md
@@ -0,0 +1,4 @@
+Analyze the attached image(s) for the current request. Return only relevant visible details, text, and layout.
+
+Current request:
+{{request}}
diff --git a/tests/test_parallel_tool.py b/tests/test_parallel_tool.py
index f7a218c36..1e612e43f 100644
--- a/tests/test_parallel_tool.py
+++ b/tests/test_parallel_tool.py
@@ -360,10 +360,13 @@ async def test_direct_parallel_worker_inherits_chat_model_override(monkeypatch)
)
override = {"preset_name": "Text only"}
parent.set_data("chat_model_override", override)
+ current_user_message = object()
+ parent.agent0.last_user_message = current_user_message
observed = {}
async def fake_execute_tool_call(agent, *_args, **_kwargs):
observed["override"] = agent.context.get_data("chat_model_override")
+ observed["last_user_message"] = agent.last_user_message
return "done"
async def remove_context(context_id):
@@ -383,6 +386,7 @@ async def test_direct_parallel_worker_inherits_chat_model_override(monkeypatch)
try:
assert await parallel_tools._run_direct_tool_job(parent_id, job) == "done"
assert observed["override"] == override
+ assert observed["last_user_message"] is current_user_message
finally:
AgentContext.remove(parent_id)
diff --git a/tests/test_tool_policy.py b/tests/test_tool_policy.py
index 96f0c594d..f64526e5b 100644
--- a/tests/test_tool_policy.py
+++ b/tests/test_tool_policy.py
@@ -580,7 +580,7 @@ async def test_active_vision_model_uses_canonical_vision_prompt(
assert schemas[0]["description"] == "canonical vision"
-def test_vision_prompt_declares_multi_image_native_schema() -> None:
+def test_vision_prompt_stays_route_agnostic_and_batches_paths() -> None:
source = (
Path(__file__).resolve().parents[1]
/ "prompts"
@@ -588,14 +588,15 @@ def test_vision_prompt_declares_multi_image_native_schema() -> None:
).read_text(encoding="utf-8")
schema = responses_tools._schema_from_prompt(source)
- assert schema["required"] == ["paths"]
- assert schema["properties"]["paths"] == {
- "type": "array",
- "items": {"type": "string"},
+ assert schema == {
+ "type": "object",
+ "properties": {},
+ "additionalProperties": True,
}
- assert "query" in schema["properties"]
- assert "raw" not in schema["properties"]
- assert "Vision Model" in source
+ assert "load all relevant images in one call" in source
+ assert "Input schema for tool_args" not in source
+ assert "Vision Model" not in source
+ assert "query" not in source
def test_mcp_prompt_and_native_schema_omit_blocked_tool(
diff --git a/tests/test_vision_load_image_refs.py b/tests/test_vision_load_image_refs.py
index 01b931a08..aba0ceb72 100644
--- a/tests/test_vision_load_image_refs.py
+++ b/tests/test_vision_load_image_refs.py
@@ -124,9 +124,9 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch
tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: updates.append(kwargs))
invalid = await tool.execute(paths=None)
- assert invalid.message == "vision_load error: `paths` must be an array."
+ assert invalid.message == "vision_load error: `paths` must be a string or an array."
- response = await tool.execute(paths=[str(image_path)])
+ response = await tool.execute(paths=str(image_path))
image_path.unlink()
await tool.after_execution(response)
@@ -205,6 +205,10 @@ async def test_vision_model_sends_multiple_images_once_and_keeps_history_text_on
agent = SimpleNamespace(
context=SimpleNamespace(id=""),
agent_name="Agent 0",
+ last_user_message=SimpleNamespace(
+ output_text=lambda: "Compare the login errors."
+ ),
+ read_prompt=lambda _name, request: f"Analyze: {request}",
hist_add_tool_result=lambda *args, **kwargs: tool_results.append((args, kwargs)),
hist_add_message=lambda *args, **kwargs: raw_messages.append((args, kwargs)),
)
@@ -218,16 +222,16 @@ async def test_vision_model_sends_multiple_images_once_and_keeps_history_text_on
)
tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: None)
- response = await tool.execute(
- paths=[str(path) for path in image_paths],
- query="Compare the login errors.",
- )
+ response = await tool.execute(paths=[str(path) for path in image_paths])
response.additional = {"_responses_output_item": {"output": response.message}}
await tool.after_execution(response)
assert len(calls) == 1
content = calls[0]["messages"][0].content
- assert content[0] == {"type": "text", "text": "Compare the login errors."}
+ assert content[0] == {
+ "type": "text",
+ "text": "Analyze: Compare the login errors.",
+ }
assert [item["type"] for item in content].count("image_url") == 2
assert "max_tokens" not in calls[0]
assert "explicit_caching" not in calls[0]
@@ -326,7 +330,14 @@ async def test_independent_vision_model_calls_can_run_concurrently(monkeypatch,
path.write_bytes(b"png-data")
def make_tool(index):
- agent = SimpleNamespace(context=SimpleNamespace(id=""), agent_name=f"Agent {index}")
+ agent = SimpleNamespace(
+ context=SimpleNamespace(id=""),
+ agent_name=f"Agent {index}",
+ last_user_message=SimpleNamespace(
+ output_text=lambda: f"inspection {index}"
+ ),
+ read_prompt=lambda _name, request: request,
+ )
return vision_load_module.VisionLoad(
agent=agent,
name="vision_load",
@@ -338,10 +349,7 @@ async def test_independent_vision_model_calls_can_run_concurrently(monkeypatch,
responses = await asyncio.gather(
*(
- make_tool(index).execute(
- paths=[str(path) for path in image_paths],
- query=f"inspection {index}",
- )
+ make_tool(index).execute(paths=[str(path) for path in image_paths])
for index in range(4)
)
)
diff --git a/tools/vision_load.py b/tools/vision_load.py
index b14b6fa5d..51f4f33b2 100644
--- a/tools/vision_load.py
+++ b/tools/vision_load.py
@@ -13,23 +13,20 @@ from plugins._model_config.helpers.model_config import (
# image token estimation for context window
TOKENS_ESTIMATE = 1500
-DEFAULT_VISION_QUERY = (
- "Describe the images precisely, including the key objects, visible text, and layout."
-)
class VisionLoad(Tool):
- async def execute(
- self, paths: list[str] = [], query: str = "", **kwargs
- ) -> Response:
+ async def execute(self, paths: list[str] | str = [], **kwargs) -> Response:
self.images_dict = {}
self.loaded_paths: list[str] = []
self.skipped_paths: list[str] = []
self.vision_config = get_vision_model_config(self.agent)
+ if isinstance(paths, str):
+ paths = [paths]
if not isinstance(paths, list):
return Response(
- message="vision_load error: `paths` must be an array.",
+ message="vision_load error: `paths` must be a string or an array.",
break_loop=False,
)
@@ -83,15 +80,13 @@ class VisionLoad(Tool):
message = self._summary() if self.images_dict or self.skipped_paths else "No images processed"
if self.vision_config and self.images_dict:
try:
- capsule = await self._call_vision_model(
- list(self.images_dict.values()), query
- )
+ capsule = await self._call_vision_model(list(self.images_dict.values()))
message = (
- f"Vision Model analyzed {len(self.images_dict)} image(s)"
+ f"Analyzed {len(self.images_dict)} image(s)"
f"; {len(self.skipped_paths)} skipped.\n\n{capsule.strip()}"
)
except Exception as exc:
- message = f"Vision Model error: {str(exc)[:1000]}"
+ message = f"Image analysis error: {str(exc)[:1000]}"
return Response(message=message, break_loop=False)
def _get_max_embeds(self) -> int:
@@ -104,8 +99,16 @@ class VisionLoad(Tool):
parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""
return str(parent_id or getattr(context, "id", "") or "").strip()
- async def _call_vision_model(self, image_paths: list[str], query: str) -> str:
- content = [{"type": "text", "text": str(query or "").strip() or DEFAULT_VISION_QUERY}]
+ async def _call_vision_model(self, image_paths: list[str]) -> str:
+ user_message = getattr(self.agent, "last_user_message", None)
+ output_text = getattr(user_message, "output_text", None)
+ request = str(output_text() if callable(output_text) else "").strip()
+ content = [
+ {
+ "type": "text",
+ "text": self.agent.read_prompt("fw.vision_load.md", request=request),
+ }
+ ]
content.extend(
{"type": "image_url", "image_url": {"url": path}}
for path in image_paths
diff --git a/tools/vision_load.py.dox.md b/tools/vision_load.py.dox.md
index 2001fca05..c0ea776e5 100644
--- a/tools/vision_load.py.dox.md
+++ b/tools/vision_load.py.dox.md
@@ -12,15 +12,16 @@
- `vision_load.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.
- Classes:
- `VisionLoad` (`Tool`)
- - `async execute(self, paths, query="", **kwargs) -> Response`
+ - `async execute(self, paths, **kwargs) -> Response`
- `async after_execution(self, response: Response, **kwargs)`
- Notable constants/configuration names: `TOKENS_ESTIMATE`.
## Runtime Contracts
- Tool modules must define `helpers.tool.Tool` subclasses and return `helpers.tool.Response` from `execute(...)`.
-- One call may contain multiple paths. The Vision Model route sends every selected path in one request and returns one textual capsule.
+- One call may contain multiple paths; a bare string is treated as one path. The Vision Model route sends every selected path in one request and returns one textual capsule.
- Model configuration exposes a Vision Model only when the effective preset selects that route; otherwise this tool follows Main's native vision path.
+- The public tool contract is route-agnostic. A Vision Model receives the current user request through `fw.vision_load.md`; direct parallel workers inherit that request from their parent.
- Delegation completes during `execute(...)` so native Responses function output contains the real capsule before `after_execution(...)` persists it.
- Delegated history contains the text capsule only. Native history contains the tool result followed by one raw message holding all loaded image blocks.
- Direct parallel workers inherit the parent's model override generically. This tool uses their recorded parent context only to resolve ephemeral refs and durable chat media.