Agent Zero AI framework
improve computer-use screenshot refresh guidance
Alessandro603fc2064b25
diff --git a/plugins/_a0_connector/plugin.yaml b/plugins/_a0_connector/plugin.yamlindex d10facd86..cf8005b11 100644--- a/plugins/_a0_connector/plugin.yaml+++ b/plugins/_a0_connector/plugin.yaml@@ -1,7 +1,7 @@name: _a0_connectortitle: A0 Connectordescription: Current Agent Zero connector plugin for HTTP plus /ws integration, using session auth and handler activation through auth.handlers.-version: 1.5+version: "1.5"settings_sections:- external- developerdiff --git a/plugins/_a0_connector/prompts/agent.extras.computer_use_remote.md b/plugins/_a0_connector/prompts/agent.extras.computer_use_remote.mdindex fe49c4dac..9b4c67cd9 100644--- a/plugins/_a0_connector/prompts/agent.extras.computer_use_remote.md+++ b/plugins/_a0_connector/prompts/agent.extras.computer_use_remote.md@@ -7,10 +7,14 @@ Features: `{{features}}`Support note: `{{support_reason}}`- Use this for local desktop and native UI tasks on the connected machine.-- If the task is browser-only and the user is flexible, prefer `browser_agent` because it is usually more reliable and token-efficient than screenshot-driven desktop control.+- If the task is browser-only and the user is flexible, prefer the direct `browser` tool because it is usually more reliable and token-efficient than screenshot-driven desktop control.- Use `start_session` before interactive desktop actions. `status` is for inspection; `stop_session` ends the session.- Base every decision on the latest screenshot or a definitive tool result, not memory.+- The current action API uses normalized global screen coordinates. Do not assume window IDs, element indices, background-safe input, or semantic click targets unless the advertised features explicitly say they exist.+- If features include `real-cursor-may-move` or `focus-risk`, expect pointer actions to affect the visible desktop state; prefer keyboard/accessibility routes even more strongly.- Successful `start_session`, `move`, `click`, `scroll`, `key`, and `type` calls already attach a fresh screenshot.+- Each attached screenshot includes a `capture id`; treat the latest attached capture as authoritative and ignore superseded capture references.+- If an attached screenshot looks unchanged after a state-changing action, use one explicit `capture` to verify before repeating the same action.- Use `capture` only when you need a screen refresh without taking another action.- Prefer accessibility and semantic UI paths first: application shortcuts, command palettes, menu accelerators, address/search bars, focus traversal, selection shortcuts, and other keyboard-accessible controls.- Prefer `key` and `type` over pointer actions whenever there is a plausible keyboard or accessibility path. Use `tab`, `shift+tab`, arrow keys, hotkeys, text search, and submit keys before reaching for the mouse.diff --git a/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md b/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.mdindex efa14697c..78800f8b9 100644--- a/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md+++ b/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md@@ -21,7 +21,10 @@ Action-specific fields:- `type`: `text`, optional `submit` boolean## Runtime Notes+- The current action API uses normalized global screen coordinates; do not assume window IDs, element indices, background-safe input, or semantic click targets unless runtime guidance explicitly advertises them.- Successful `start_session`, `move`, `click`, `scroll`, `key`, and `type` calls automatically attach a fresh screenshot.+- Attached screenshots include a `capture id`; use the latest capture as the coordinate basis.+- If the attached screenshot appears unchanged after a state-changing action, verify once with `capture` before repeating the same action.- `status` reports the current computer-use state without starting a session.- Prefer accessibility, semantic UI controls, hotkeys, focus traversal, and other keyboard paths before pointer actions.- For viewport movement, prefer keyboard scrolling first; use `scroll` when a wheel-style scroll is the most reliable way to move an already-focused viewport or pane.diff --git a/plugins/_a0_connector/tools/computer_use_remote.py b/plugins/_a0_connector/tools/computer_use_remote.pyindex 3534b1b55..470011048 100644--- a/plugins/_a0_connector/tools/computer_use_remote.py+++ b/plugins/_a0_connector/tools/computer_use_remote.py@@ -30,9 +30,15 @@ _AUTO_CAPTURE_ACTIONS = {"type",}_SETTLE_DELAY_START_SESSION = 0.2+_SETTLE_DELAY_MOVE = 0.1+_SETTLE_DELAY_CLICK = 0.35+_SETTLE_DELAY_SCROLL = 0.35+_SETTLE_DELAY_KEY = 0.2+_SETTLE_DELAY_TYPE = 0.25_SETTLE_DELAY_GLOBAL_FOCUS = 0.45_SETTLE_DELAY_PLAIN_ENTER = 0.3_SETTLE_DELAY_SUBMIT = 0.45+_FRESH_CAPTURE_TIMEOUT = 0.45_SUPPORTED_ACTIONS = {"start_session","status",@@ -160,15 +166,15 @@ class ComputerUseRemote(Tool):if settle_seconds > 0:await asyncio.sleep(settle_seconds)- capture_result = await self._dispatch_payload(- sid=sid,- payload={- "op_id": str(uuid.uuid4()),- "context_id": context_id,- "action": "capture",- "session_id": session_id,- },- )+ capture_payload = {+ "op_id": str(uuid.uuid4()),+ "context_id": context_id,+ "action": "capture",+ "session_id": session_id,+ "fresh": True,+ "fresh_timeout_seconds": _FRESH_CAPTURE_TIMEOUT,+ }+ capture_result = await self._dispatch_payload(sid=sid, payload=capture_payload)if not bool(capture_result.get("ok")):return f"Automatic screen refresh failed: {self._format_error(capture_result)}"@@ -176,14 +182,22 @@ class ComputerUseRemote(Tool):if not isinstance(capture_data, dict):return "Automatic screen refresh failed: missing capture payload."- self._record_capture(capture_data)- return "Latest screen attached."+ summary = self._record_capture(capture_data)+ return f"Latest screen attached: {summary}"def _auto_capture_settle_seconds(self, action: str) -> float:if action == "start_session":return _SETTLE_DELAY_START_SESSION+ if action == "move":+ return _SETTLE_DELAY_MOVE+ if action == "click":+ return _SETTLE_DELAY_CLICK+ if action == "scroll":+ return _SETTLE_DELAY_SCROLLif action == "type" and self._coerce_bool(self.args.get("submit")):return _SETTLE_DELAY_SUBMIT+ if action == "type":+ return _SETTLE_DELAY_TYPEif action != "key":return 0.0@@ -192,7 +206,7 @@ class ComputerUseRemote(Tool):return _SETTLE_DELAY_GLOBAL_FOCUSif keyset == {"enter"}:return _SETTLE_DELAY_PLAIN_ENTER- return 0.0+ return _SETTLE_DELAY_KEYdef _requested_keys(self) -> list[str]:keys_value = self.args.get("keys")@@ -252,8 +266,8 @@ class ComputerUseRemote(Tool):data = {}if action == "capture":- self._record_capture(data)- return "Current screen attached."+ summary = self._record_capture(data)+ return f"Current screen attached: {summary}"if action == "status":return self._format_status(data)if action == "start_session":@@ -309,7 +323,18 @@ class ComputerUseRemote(Tool):_image_path, display_path = self._resolve_capture_path(data)width = data.get("width", "?")height = data.get("height", "?")- summary = f"Computer-use capture {width}x{height}."+ capture_id = str(data.get("capture_id") or Path(display_path).stem or "?").strip()+ coordinate_space = str(data.get("coordinate_space") or "normalized_global_screen").strip()+ summary = (+ f"Computer-use capture id={capture_id} {width}x{height}, "+ f"coordinates={coordinate_space} [0,1]."+ )+ if data.get("fresh") is True:+ if "fresh_after_satisfied" in data:+ fresh_state = "confirmed" if data.get("fresh_after_satisfied") is not False else "not confirmed"+ summary = f"{summary} Fresh frame {fresh_state}."+ else:+ summary = f"{summary} Fresh capture requested."content = [{"type": "text", "text": summary},{"type": "image_url", "image_url": {"url": display_path}},diff --git a/skills/computer-use-remote/SKILL.md b/skills/computer-use-remote/SKILL.mdindex 8675ed53c..4b545ebf0 100644--- a/skills/computer-use-remote/SKILL.md+++ b/skills/computer-use-remote/SKILL.md@@ -23,7 +23,7 @@ allowed_tools:Load this skill before using `computer_use_remote` for local desktop and native UI tasks on the connected machine.-For ordinary website browsing, search, form filling, and web downloads, prefer `browser_agent` instead. If the user is flexible and the task is browser-only, guide them toward browser tools because they are usually more reliable and token-efficient than screenshot-driven computer use.+For ordinary website browsing, search, form filling, and web downloads, prefer the direct `browser` tool instead. If the user is flexible and the task is browser-only, guide them toward browser tools because they are usually more reliable and token-efficient than screenshot-driven computer use.## Core Loop