Agent Zero AI framework
Restore route-agnostic vision tool contract
Alessandro540ef8d81543
diff --git a/helpers/parallel_tools.py b/helpers/parallel_tools.pyindex 382025bbc..dbc79ef9f 100644--- a/helpers/parallel_tools.py+++ b/helpers/parallel_tools.py@@ -485,6 +485,7 @@ async def _run_direct_tool_job(parent_context_id: str, job: ParallelJob) -> str:_copy_project(parent_context, worker_context)worker_agent = worker_context.agent0+ worker_agent.last_user_message = parent_context.agent0.last_user_messageworker_agent.loop_data = LoopData()return await execute_tool_call(worker_agent,diff --git a/helpers/parallel_tools.py.dox.md b/helpers/parallel_tools.py.dox.mdindex 28fbbadcd..9fb3f6b5f 100644--- a/helpers/parallel_tools.py.dox.md+++ b/helpers/parallel_tools.py.dox.md@@ -32,7 +32,7 @@- Subordinate child chats are tagged with job metadata, remain outside the scheduler task list, and may use normal child-chat tools including `parallel`.- Nested parallel jobs started by a parallel subordinate are registered as child `DeferredTask` instances so stopping the ancestor also stops its descendants.- Direct tool jobs run in isolated background contexts and are blocked from recursively invoking `parallel`.-- Direct tool jobs inherit the parent's active per-chat model override.+- Direct tool jobs inherit the parent's active per-chat model override and current user message.- Direct tool background context cleanup removes both the in-memory context and any transient chat folder left on disk.- Parent-visible child log items are created for each wrapped call so the WebUI can inspect concurrent children separately while the wrapper result remains model-history-only.- Child tool logs mirror normal tool-call visible args; job ids remain available through wrapper results and prompt extras rather than visible process-step args.diff --git a/prompts/AGENTS.md b/prompts/AGENTS.mdindex 07c4d5282..10917ba20 100644--- a/prompts/AGENTS.md+++ b/prompts/AGENTS.md@@ -25,7 +25,7 @@- Read the rendering path before changing placeholders or filenames.- Prefer small prompt additions over broad rewrites when fixing a specific behavior.- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.-- Keep the single `vision_load` prompt accurate for both native and separate Vision Model routing: related images belong in one call, and a Vision Model result is text-only.+- Keep the single `vision_load` tool prompt route-agnostic and preserve one-call loading for related images; internal image-analysis instructions belong in framework prompts, not Python strings.- Update tests or snapshots when prompt budget, required sections, or generated system content changes.## Verificationdiff --git a/prompts/agent.system.tools_vision.md b/prompts/agent.system.tools_vision.mdindex b766699cf..ee50376ad 100644--- a/prompts/agent.system.tools_vision.md+++ b/prompts/agent.system.tools_vision.md@@ -1,40 +1,23 @@## multimodal vision tools### vision_load-load or analyze images for visual reasoning-args: `paths` list of absolute image paths or ephemeral image refs, `query` optional focused instruction-Input schema for tool_args:-```json-{- "type": "object",- "properties": {- "paths": {- "type": "array",- "items": {"type": "string"}- },- "query": {"type": "string"}- },- "required": ["paths"],- "additionalProperties": false-}-```+load images into the model for visual reasoning+args: `paths` list of absolute image paths or tool-returned ephemeral image refsrules:-- put all images needed for one comparison or visual task in the same `paths` array+- load all relevant images in one call when comparing screenshots or pages- use when the task depends on screenshots, diagrams, scanned documents, charts, or photos-- use a focused `query` when asking the configured Vision Model to inspect, compare, locate, or read something- only bitmaps are supported; convert other formats first if needed-- the tool result reports loaded and skipped image counts+- the tool result includes loaded/skipped image totals and the corresponding path listsexample:```json{"thoughts": ["I need to inspect the screenshot before answering."],- "headline": "Comparing screenshots",+ "headline": "Loading screenshot for visual analysis","tool_name": "vision_load","tool_args": {- "paths": ["/path/to/before.png", "/path/to/after.png"],- "query": "Compare the error banners and describe what changed."+ "paths": ["/path/to/screenshot.png"]}}```diff --git a/prompts/fw.vision_load.md b/prompts/fw.vision_load.mdnew file mode 100644index 000000000..1c7320e2f--- /dev/null+++ b/prompts/fw.vision_load.md@@ -0,0 +1,4 @@+Analyze the attached image(s) for the current request. Return only relevant visible details, text, and layout.++Current request:+{{request}}diff --git a/tests/test_parallel_tool.py b/tests/test_parallel_tool.pyindex f7a218c36..1e612e43f 100644--- a/tests/test_parallel_tool.py+++ b/tests/test_parallel_tool.py@@ -360,10 +360,13 @@ async def test_direct_parallel_worker_inherits_chat_model_override(monkeypatch))override = {"preset_name": "Text only"}parent.set_data("chat_model_override", override)+ current_user_message = object()+ parent.agent0.last_user_message = current_user_messageobserved = {}async def fake_execute_tool_call(agent, *_args, **_kwargs):observed["override"] = agent.context.get_data("chat_model_override")+ observed["last_user_message"] = agent.last_user_messagereturn "done"async def remove_context(context_id):@@ -383,6 +386,7 @@ async def test_direct_parallel_worker_inherits_chat_model_override(monkeypatch)try:assert await parallel_tools._run_direct_tool_job(parent_id, job) == "done"assert observed["override"] == override+ assert observed["last_user_message"] is current_user_messagefinally:AgentContext.remove(parent_id)diff --git a/tests/test_tool_policy.py b/tests/test_tool_policy.pyindex 96f0c594d..f64526e5b 100644--- a/tests/test_tool_policy.py+++ b/tests/test_tool_policy.py@@ -580,7 +580,7 @@ async def test_active_vision_model_uses_canonical_vision_prompt(assert schemas[0]["description"] == "canonical vision"-def test_vision_prompt_declares_multi_image_native_schema() -> None:+def test_vision_prompt_stays_route_agnostic_and_batches_paths() -> None:source = (Path(__file__).resolve().parents[1]/ "prompts"@@ -588,14 +588,15 @@ def test_vision_prompt_declares_multi_image_native_schema() -> None:).read_text(encoding="utf-8")schema = responses_tools._schema_from_prompt(source)- assert schema["required"] == ["paths"]- assert schema["properties"]["paths"] == {- "type": "array",- "items": {"type": "string"},+ assert schema == {+ "type": "object",+ "properties": {},+ "additionalProperties": True,}- assert "query" in schema["properties"]- assert "raw" not in schema["properties"]- assert "Vision Model" in source+ assert "load all relevant images in one call" in source+ assert "Input schema for tool_args" not in source+ assert "Vision Model" not in source+ assert "query" not in sourcedef test_mcp_prompt_and_native_schema_omit_blocked_tool(diff --git a/tests/test_vision_load_image_refs.py b/tests/test_vision_load_image_refs.pyindex 01b931a08..aba0ceb72 100644--- a/tests/test_vision_load_image_refs.py+++ b/tests/test_vision_load_image_refs.py@@ -124,9 +124,9 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatchtool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: updates.append(kwargs))invalid = await tool.execute(paths=None)- assert invalid.message == "vision_load error: `paths` must be an array."+ assert invalid.message == "vision_load error: `paths` must be a string or an array."- response = await tool.execute(paths=[str(image_path)])+ response = await tool.execute(paths=str(image_path))image_path.unlink()await tool.after_execution(response)@@ -205,6 +205,10 @@ async def test_vision_model_sends_multiple_images_once_and_keeps_history_text_onagent = SimpleNamespace(context=SimpleNamespace(id=""),agent_name="Agent 0",+ last_user_message=SimpleNamespace(+ output_text=lambda: "Compare the login errors."+ ),+ read_prompt=lambda _name, request: f"Analyze: {request}",hist_add_tool_result=lambda *args, **kwargs: tool_results.append((args, kwargs)),hist_add_message=lambda *args, **kwargs: raw_messages.append((args, kwargs)),)@@ -218,16 +222,16 @@ async def test_vision_model_sends_multiple_images_once_and_keeps_history_text_on)tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: None)- response = await tool.execute(- paths=[str(path) for path in image_paths],- query="Compare the login errors.",- )+ response = await tool.execute(paths=[str(path) for path in image_paths])response.additional = {"_responses_output_item": {"output": response.message}}await tool.after_execution(response)assert len(calls) == 1content = calls[0]["messages"][0].content- assert content[0] == {"type": "text", "text": "Compare the login errors."}+ assert content[0] == {+ "type": "text",+ "text": "Analyze: Compare the login errors.",+ }assert [item["type"] for item in content].count("image_url") == 2assert "max_tokens" not in calls[0]assert "explicit_caching" not in calls[0]@@ -326,7 +330,14 @@ async def test_independent_vision_model_calls_can_run_concurrently(monkeypatch,path.write_bytes(b"png-data")def make_tool(index):- agent = SimpleNamespace(context=SimpleNamespace(id=""), agent_name=f"Agent {index}")+ agent = SimpleNamespace(+ context=SimpleNamespace(id=""),+ agent_name=f"Agent {index}",+ last_user_message=SimpleNamespace(+ output_text=lambda: f"inspection {index}"+ ),+ read_prompt=lambda _name, request: request,+ )return vision_load_module.VisionLoad(agent=agent,name="vision_load",@@ -338,10 +349,7 @@ async def test_independent_vision_model_calls_can_run_concurrently(monkeypatch,responses = await asyncio.gather(*(- make_tool(index).execute(- paths=[str(path) for path in image_paths],- query=f"inspection {index}",- )+ make_tool(index).execute(paths=[str(path) for path in image_paths])for index in range(4)))diff --git a/tools/vision_load.py b/tools/vision_load.pyindex b14b6fa5d..51f4f33b2 100644--- a/tools/vision_load.py+++ b/tools/vision_load.py@@ -13,23 +13,20 @@ from plugins._model_config.helpers.model_config import (# image token estimation for context windowTOKENS_ESTIMATE = 1500-DEFAULT_VISION_QUERY = (- "Describe the images precisely, including the key objects, visible text, and layout."-)class VisionLoad(Tool):- async def execute(- self, paths: list[str] = [], query: str = "", **kwargs- ) -> Response:+ async def execute(self, paths: list[str] | str = [], **kwargs) -> Response:self.images_dict = {}self.loaded_paths: list[str] = []self.skipped_paths: list[str] = []self.vision_config = get_vision_model_config(self.agent)+ if isinstance(paths, str):+ paths = [paths]if not isinstance(paths, list):return Response(- message="vision_load error: `paths` must be an array.",+ message="vision_load error: `paths` must be a string or an array.",break_loop=False,)@@ -83,15 +80,13 @@ class VisionLoad(Tool):message = self._summary() if self.images_dict or self.skipped_paths else "No images processed"if self.vision_config and self.images_dict:try:- capsule = await self._call_vision_model(- list(self.images_dict.values()), query- )+ capsule = await self._call_vision_model(list(self.images_dict.values()))message = (- f"Vision Model analyzed {len(self.images_dict)} image(s)"+ f"Analyzed {len(self.images_dict)} image(s)"f"; {len(self.skipped_paths)} skipped.\n\n{capsule.strip()}")except Exception as exc:- message = f"Vision Model error: {str(exc)[:1000]}"+ message = f"Image analysis error: {str(exc)[:1000]}"return Response(message=message, break_loop=False)def _get_max_embeds(self) -> int:@@ -104,8 +99,16 @@ class VisionLoad(Tool):parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""return str(parent_id or getattr(context, "id", "") or "").strip()- async def _call_vision_model(self, image_paths: list[str], query: str) -> str:- content = [{"type": "text", "text": str(query or "").strip() or DEFAULT_VISION_QUERY}]+ async def _call_vision_model(self, image_paths: list[str]) -> str:+ user_message = getattr(self.agent, "last_user_message", None)+ output_text = getattr(user_message, "output_text", None)+ request = str(output_text() if callable(output_text) else "").strip()+ content = [+ {+ "type": "text",+ "text": self.agent.read_prompt("fw.vision_load.md", request=request),+ }+ ]content.extend({"type": "image_url", "image_url": {"url": path}}for path in image_pathsdiff --git a/tools/vision_load.py.dox.md b/tools/vision_load.py.dox.mdindex 2001fca05..c0ea776e5 100644--- a/tools/vision_load.py.dox.md+++ b/tools/vision_load.py.dox.md@@ -12,15 +12,16 @@- `vision_load.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.- Classes:- `VisionLoad` (`Tool`)- - `async execute(self, paths, query="", **kwargs) -> Response`+ - `async execute(self, paths, **kwargs) -> Response`- `async after_execution(self, response: Response, **kwargs)`- Notable constants/configuration names: `TOKENS_ESTIMATE`.## Runtime Contracts- Tool modules must define `helpers.tool.Tool` subclasses and return `helpers.tool.Response` from `execute(...)`.-- One call may contain multiple paths. The Vision Model route sends every selected path in one request and returns one textual capsule.+- One call may contain multiple paths; a bare string is treated as one path. The Vision Model route sends every selected path in one request and returns one textual capsule.- Model configuration exposes a Vision Model only when the effective preset selects that route; otherwise this tool follows Main's native vision path.+- The public tool contract is route-agnostic. A Vision Model receives the current user request through `fw.vision_load.md`; direct parallel workers inherit that request from their parent.- Delegation completes during `execute(...)` so native Responses function output contains the real capsule before `after_execution(...)` persists it.- Delegated history contains the text capsule only. Native history contains the tool result followed by one raw message holding all loaded image blocks.- Direct parallel workers inherit the parent's model override generically. This tool uses their recorded parent context only to resolve ephemeral refs and durable chat media.