Agent Zero AI framework
Harden remote Linux computer use targeting
Alessandro4cea5d7197b8
diff --git a/plugins/_a0_connector/AGENTS.md b/plugins/_a0_connector/AGENTS.mdindex b74c5b726..5c146ef8b 100644--- a/plugins/_a0_connector/AGENTS.md+++ b/plugins/_a0_connector/AGENTS.md@@ -52,6 +52,7 @@after all chunks for the `op_id` are assembled.- Host browser status metadata may advertise `available_browsers` entries with browser ids, labels, CDP endpoints, status, and enabled state; keep older CLI payloads without those fields compatible.- Model preset definitions exposed through v1 are global; project arguments select scope but never create project-owned definitions. Model switcher state reports the effective main, utility, and embedding models and preserves embedding-change notifications.+- Computer Use receipts describe transport success unless the connector returns explicit effect evidence. Linux target-bound typing requires a verified active/focused `window_id`; window activation uses focus, never a press action on an application or window node. Do not retry an identical failed Computer Use call.## Work Guidancediff --git a/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md b/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.mdindex 0482eb369..39de1b82b 100644--- a/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md+++ b/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md@@ -8,11 +8,15 @@ This is the only desktop-control path for the user's connected host/local computIf the tool reports no CLI/Launcher host bridge, disabled computer use, or `COMPUTER_USE_REARM_REQUIRED`, stop and tell the user to run `/computer-use on` in the A0 Launcher chat when using Launcher Host access, or in A0 CLI otherwise, and approve any host permission prompt.+If a Computer Use call returns an error, do not repeat the same action with identical arguments. Report the error, or use a materially different safe recovery only when the task still requires it.+Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. Read `backend_id`, `backend_family`, `features`, and the structured `capabilities` object in status/session results. When capabilities report native windows, window state, element indexes, and background dispatch, prefer `list_windows` -> `get_window_state` -> `element_action` with `dispatch: "background"` before using global coordinates. Interactive coordinate actions should use normalized global-screen coordinates from the most recent capture.Some actions are backend-specific and intentionally documented only in backend skills. If `status` or `start_session` reports backend-specific features or tells you to load a backend skill, load and follow that skill before using those backend-only actions. For structural targeting details, load and follow the backend-specific skill such as `host-computer-use-macos` or `host-computer-use-windows`; do not apply one backend's guidance to another backend.-State-changing actions automatically attach a fresh screen after they run unless the backend returns a definitive structural background result. Treat key presses, clicks, scrolling, and typing as attempts, not success; treat foreground fallbacks the same way. Inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result only proves keystrokes were sent; it does not prove that text landed in the intended place.+State-changing actions automatically attach a fresh screen after they run unless the backend returns a definitive structural background result. Treat key presses, clicks, scrolling, and typing as attempts, not success; treat foreground fallbacks the same way. Inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result proves only that keyboard events were sent unless it explicitly reports both `focus_verified=true` and the target `window_id`.++When Linux advertises `verified-window-focus` and `target-verified-keyboard-input`, use a real frame/window from `list_windows`, inspect it with `get_window_state`, and focus its window element with foreground `element_action`. Continue to `type` only after the focus result reports `focus_verified=true`, and pass that same `window_id` to `type`. Never press an application/frame/window node to activate it.```json{@@ -28,7 +32,7 @@ Required argument:Optional arguments by action:- `session_id`: session returned by `start_session`-- `pid`, `window_id`: target a native app/window for `get_window_state` and `element_action`+- `pid`, `window_id`: target a native window for `get_window_state`, `element_action`, scoped backend snapshots, and guarded keyboard input when supported- `element_index`: target an element from the latest `get_window_state`- `operation`: action such as `invoke`, `press`, `set_value`, `focus`, or backend-specific operations- `dispatch`: `background`, `auto`, or `foreground`; prefer `background` for `element_action`@@ -37,7 +41,7 @@ Optional arguments by action:- `count`: click count for `click`- `dx`, `dy`: scroll amounts for `scroll`- `key` or `keys`: key press value for `key`-- `text`: text to type for `type`+- `text`: text to type for `type`; Linux target-verified input also requires `window_id`- `submit`: boolean Enter-after-type flag for `type`Status/session results may include `contract_version` and `capabilities`. Treat `capabilities.identity.pid`, `capabilities.identity.window_id`, `capabilities.identity.element_index`, and `capabilities.dispatch.background` as the authoritative cross-platform contract for whether the native background loop is available. Use `features` for backend-specific refinements and skill selection.diff --git a/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md b/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.mdindex a9d38adc7..43748bee8 100644--- a/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md+++ b/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md@@ -29,13 +29,14 @@ Use `ax_snapshot` to inspect the Linux AT-SPI tree:"tool_name": "computer_use_remote","tool_args": {"action": "ax_snapshot",+ "window_id": "<window_id from list_windows>","max_depth": 4,"max_nodes": 200}}```-The snapshot returns paths, roles, names/titles, descriptions, frames, states, actions, text previews, values, and child nodes. Use it to choose a target, not as final visual proof.+Pass `window_id` whenever one is known so unrelated applications cannot consume the node budget. The snapshot returns paths, roles, names/titles, descriptions, frames, states, actions, text previews, values, and child nodes. Use it to choose a target, not as final visual proof.Use `ax_action` for structural actions:@@ -55,7 +56,7 @@ Use `ax_action` for structural actions:Supported operations are:-- `press`: activate a button, menu item, tab, checkbox, or similar action-bearing node+- `press`: activate a button, menu item, tab, checkbox, or similar action-bearing node; never use it on an application/frame/window- `focus`: focus a focusable node before typing or keyboard input- `set_value`: set text/value on editable nodes; pass `value` or `text`@@ -72,6 +73,10 @@ Use screenshots for proof after every state-changing action. AT-SPI actions andTrue background dispatch on Linux is compositor, toolkit, and app dependent. Do not claim a Linux action was background-safe unless the tool result explicitly says `actual_dispatch=background`.+To bring a Linux window forward, target its frame/window element from `get_window_state` with `element_action`, operation `focus`, and `dispatch: "foreground"` or `"auto"`. Continue only when the result says `focus_verified=true`; an accepted AT-SPI call without active/focused state is not activation proof.++Linux text injection is target-guarded. Pass the same verified active `window_id` to `type`. If the tool reports `COMPUTER_USE_WINDOW_REQUIRED` or `COMPUTER_USE_TARGET_NOT_FOCUSED`, do not type globally and do not substitute another application target.+On GNOME/Wayland, useful shortcuts include:- `Super+H`: hide the active windowdiff --git a/plugins/_a0_connector/skills/host-computer-use/SKILL.md b/plugins/_a0_connector/skills/host-computer-use/SKILL.mdindex 08bab40b7..142a21f17 100644--- a/plugins/_a0_connector/skills/host-computer-use/SKILL.md+++ b/plugins/_a0_connector/skills/host-computer-use/SKILL.md@@ -80,10 +80,12 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi4. If the backend advertises native window listing through capabilities or `native-window-list`, call `list_windows` before using coordinates.5. If the backend advertises window state and element-index targeting through capabilities or features, call `get_window_state` for the target `pid`/`window_id`, then use `element_action` with `dispatch: "background"` by default.6. If `element_action` reports `background_unavailable`, use `dispatch: "auto"` or `dispatch: "foreground"` only when foreground control is acceptable for the user/task.-7. Decide final success from the latest screenshot or a definitive structural result, not from memory.-8. Interactive actions already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.-9. Use `status` for state without starting a session.-10. Use `capture` only when you need another screenshot without taking an action.+7. When the backend advertises verified window focus, activate a window only with foreground `element_action` operation `focus`; require `focus_verified=true`. Never use `press` on an application/frame/window node as activation.+8. When the backend advertises target-verified keyboard input, pass the verified active `window_id` to `type`; a missing, inactive, or unverifiable target must fail closed.+9. Decide final success from the latest screenshot or a definitive structural result, not from memory.+10. Interactive actions already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.+11. Use `status` for state without starting a session.+12. Use `capture` only when you need another screenshot without taking an action.## Backend Skills@@ -107,7 +109,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi- If the same approach has already failed twice without visible progress, switch strategy instead of repeating it.- Do not infer focus or task completion from chat logs, sidebars, tool summaries, or status text.- Never claim a state-changing action succeeded until the latest screenshot visibly confirms it.-- A `type` tool result only confirms keystrokes were sent. It is not evidence that the text landed in the intended application.+- A `type` tool result confirms the destination only when it reports `focus_verified=true` with the intended `window_id`; otherwise it confirms only global keyboard events were sent.- For browser-navigation tasks done through this tool, only claim success if the browser content area visibly shows the destination page or result.- If the attached screenshot appears unchanged after a state-changing action, use one explicit `capture` to verify before repeating the same action.- Use `type(..., submit=true)` only for URL or navigation-style entry where Enter should fire immediately after typing.diff --git a/plugins/_a0_connector/tools/computer_use_remote.py b/plugins/_a0_connector/tools/computer_use_remote.pyindex 8de1cd57f..c0ea899e4 100644--- a/plugins/_a0_connector/tools/computer_use_remote.py+++ b/plugins/_a0_connector/tools/computer_use_remote.py@@ -332,6 +332,8 @@ class ComputerUseRemote(Tool):payload["key"] = self.args.get("key")elif action == "type":payload["text"] = self.args.get("text", "")+ if "window_id" in self.args:+ payload["window_id"] = self.args.get("window_id")if self._coerce_bool(self.args.get("submit")):payload["submit"] = Trueelif action == "list_windows":@@ -363,6 +365,9 @@ class ComputerUseRemote(Tool):if "selector" in self.args:payload["selector"] = self.args.get("selector")elif action == "ax_snapshot":+ for key in ("pid", "window_id"):+ if key in self.args:+ payload[key] = self.args.get(key)if "max_depth" in self.args:payload["max_depth"] = self._coerce_int(self.args.get("max_depth"), name="max_depth")if "max_nodes" in self.args:@@ -471,9 +476,17 @@ class ComputerUseRemote(Tool):return f"Sent keys: {keys!r}."if action == "type":text = str(data.get("text", "") or "")- if data.get("submitted"):- return f"Typed {len(text)} character(s) and submitted."- return f"Typed {len(text)} character(s)."+ window_id = str(data.get("window_id") or "").strip()+ submitted = " and submitted" if data.get("submitted") else ""+ if data.get("focus_verified") and window_id:+ return (+ f"Sent {len(text)} keyboard character(s){submitted} to verified active "+ f"window_id={window_id}."+ )+ return (+ f"Sent {len(text)} global keyboard character(s){submitted}; destination was not verified. "+ "Inspect the attached screen before claiming where the text landed."+ )return str(data)def _format_error(self, result: dict[str, Any]) -> str:@@ -639,7 +652,7 @@ class ComputerUseRemote(Tool):f"{frame.get('width', '?')}x{frame.get('height', '?')})")flags: list[str] = []- for flag in ("is_on_screen", "on_current_space", "focused", "visible"):+ for flag in ("is_on_screen", "on_current_space", "active", "focused", "visible"):if flag in item:flags.append(f"{flag}={item.get(flag)}")if flags:@@ -658,10 +671,16 @@ class ComputerUseRemote(Tool):node_count = data.get("node_count", "?")truncated = " truncated" if data.get("truncated") else ""mode = str(data.get("mode") or "auto").strip()+ state_parts = [+ f"{flag}={window.get(flag)}"+ for flag in ("active", "focused")+ if flag in window+ ]+ state_text = f" {' '.join(state_parts)}." if state_parts else ""return (f"Window state for {title!r}"f"{f' window_id={window_id}' if window_id else ''}: "- f"{node_count} element(s){truncated}, mode={mode}. "+ f"{node_count} element(s){truncated}, mode={mode}.{state_text} ""Use element_action with element_index; dispatch defaults to background."f"{self._structural_tree_outline(tree)}")@@ -688,7 +707,8 @@ class ComputerUseRemote(Tool):f"requested_dispatch={requested_dispatch}, actual_dispatch={actual_dispatch}"f"{', foreground_fallback_used=true' if fallback else ''}")- return f"Performed {operation} on element_index={index} {label}; {dispatch_text}."+ verification = ", focus_verified=true" if data.get("focus_verified") else ""+ return f"Performed {operation} on element_index={index} {label}; {dispatch_text}{verification}."def _format_ax_snapshot(self, data: dict[str, Any]) -> str:app = data.get("app") if isinstance(data.get("app"), dict) else {}@@ -697,8 +717,10 @@ class ComputerUseRemote(Tool):node_count = data.get("node_count", "?")truncated = " truncated" if data.get("truncated") else ""root_label = self._ax_target_label(tree)+ window_id = str(data.get("window_id") or "").strip()+ scope = f" scoped to window_id={window_id}" if data.get("scoped") and window_id else ""return (- f"AX snapshot for {app_name}: {node_count} node(s){truncated}. "+ f"AX snapshot for {app_name}{scope}: {node_count} node(s){truncated}. "f"Root {root_label}. Use path or semantic target fields with ax_action."f"{self._structural_tree_outline(tree)}")diff --git a/tests/test_a0_connector_prompt_gating.py b/tests/test_a0_connector_prompt_gating.pyindex 157af2cf5..7ea914efd 100644--- a/tests/test_a0_connector_prompt_gating.py+++ b/tests/test_a0_connector_prompt_gating.py@@ -283,6 +283,14 @@ def test_computer_use_remote_prompt_requires_visual_verification_after_actions()/ "host-computer-use"/ "SKILL.md").read_text(encoding="utf-8")+ linux_skill = (+ PROJECT_ROOT+ / "plugins"+ / "_a0_connector"+ / "skills"+ / "host-computer-use-linux"+ / "SKILL.md"+ ).read_text(encoding="utf-8")assert "Treat key presses, clicks, scrolling, and typing" in promptassert "attempts, not success" in prompt@@ -294,8 +302,13 @@ def test_computer_use_remote_prompt_requires_visual_verification_after_actions()assert "minimize" not in prompt.lower()assert "window-manager" not in promptassert "cannot actually see the image" in skill- assert "A `type` tool result only confirms keystrokes were sent" in skill+ assert "A `type` tool result confirms the destination only when" in skillassert "visibly confirms" in skill+ assert "target-verified-keyboard-input" in prompt+ assert "focus_verified=true" in prompt+ assert "do not repeat the same action with identical arguments" in prompt+ assert "Pass the same verified active `window_id` to `type`" in linux_skill+ assert "never use it on an application/frame/window" in linux_skillassert "hide window" not in skillassert "minimize window" not in skillassert "hide/minimize" not in skilldiff --git a/tests/test_tool_action_contracts.py b/tests/test_tool_action_contracts.pyindex a4cb49393..45defc0fb 100644--- a/tests/test_tool_action_contracts.py+++ b/tests/test_tool_action_contracts.py@@ -1140,6 +1140,84 @@ def test_computer_use_remote_start_session_reports_backend_features_and_windows_assert "host-computer-use-windows" in message+def test_computer_use_remote_forwards_linux_window_scope_and_type_guard(monkeypatch):+ module = _load_computer_use_remote_tool(monkeypatch)+ tool = object.__new__(module.ComputerUseRemote)+ window_id = "atspi-pid:57929:path:20.0"++ tool.args = {+ "action": "ax_snapshot",+ "pid": 57929,+ "window_id": window_id,+ "max_depth": 4,+ "max_nodes": 80,+ }+ snapshot = tool._build_payload(op_id="op-snapshot", context_id="ctx", action="ax_snapshot")+ tool.args = {"action": "type", "window_id": window_id, "text": "hello", "submit": True}+ typed = tool._build_payload(op_id="op-type", context_id="ctx", action="type")++ assert snapshot["pid"] == 57929+ assert snapshot["window_id"] == window_id+ assert snapshot["max_depth"] == 4+ assert snapshot["max_nodes"] == 80+ assert typed["window_id"] == window_id+ assert typed["text"] == "hello"+ assert typed["submit"] is True+++def test_computer_use_remote_receipts_separate_injection_from_verified_result(monkeypatch):+ module = _load_computer_use_remote_tool(monkeypatch)+ tool = object.__new__(module.ComputerUseRemote)+ window_id = "atspi-pid:57929:path:20.0"++ verified = tool._extract_result(+ "type",+ {+ "ok": True,+ "result": {+ "text": "hello",+ "window_id": window_id,+ "focus_verified": True,+ },+ },+ )+ unverified = tool._extract_result(+ "type",+ {"ok": True, "result": {"text": "hello"}},+ )+ focused = tool._extract_result(+ "element_action",+ {+ "ok": True,+ "result": {+ "operation": "focus",+ "target": {"element_index": 0, "role": "frame", "title": "Discord"},+ "requested_dispatch": "foreground",+ "actual_dispatch": "foreground",+ "focus_verified": True,+ },+ },+ )+ scoped = tool._extract_result(+ "ax_snapshot",+ {+ "ok": True,+ "result": {+ "app": {"name": "Discord"},+ "tree": {"role": "frame", "title": "Discord"},+ "node_count": 2,+ "window_id": window_id,+ "scoped": True,+ },+ },+ )++ assert f"verified active window_id={window_id}" in verified+ assert "destination was not verified" in unverified+ assert "focus_verified=true" in focused+ assert f"scoped to window_id={window_id}" in scoped++def test_computer_use_remote_capture_artifact_is_chat_scoped(monkeypatch, tmp_path: Path):module = _load_computer_use_remote_tool(monkeypatch)