Agent Zero AI framework
Guide computer-use remote through Linux AT-SPI
Alessandro97953db46ba0
diff --git a/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md b/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.mdnew file mode 100644index 000000000..df0fca704--- /dev/null+++ b/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md@@ -0,0 +1,86 @@+---+name: host-computer-use-linux+description: Backend-specific Linux guidance for `computer_use_remote`. Load after `status` or `start_session` reports backend_family `linux`, backend_id `wayland`, or AT-SPI features. Covers AT-SPI structural targeting, Wayland portal caveats, and screenshot verification.+---++# Host Computer Use - Linux++Use this after `host-computer-use` when the connected A0 CLI reports the Linux/Wayland computer-use backend.++Do not use this skill for macOS, Windows, Xpra, Docker, browser-only tasks, or the internal Agent Zero Desktop. If the backend is not Linux or does not advertise AT-SPI support, skip Linux structural actions and follow the generic host computer-use rules.++## Linux AT-SPI Targeting++Linux backends can advertise structural AT-SPI features:++- `atspi-tree-snapshot`+- `atspi-structural-targeting`+- `atspi-element-action`+- `atspi-set-value`++When these features are present, prefer structural targeting over pixel clicks for named buttons, menu items, text fields, dialogs, toolbar items, tab strips, and application windows.++Use `ax_snapshot` to inspect the Linux AT-SPI tree:++```json+{+ "tool_name": "computer_use_remote",+ "tool_args": {+ "action": "ax_snapshot",+ "max_depth": 4,+ "max_nodes": 200+ }+}+```++The snapshot returns paths, roles, names/titles, descriptions, frames, states, actions, text previews, values, and child nodes. Use it to choose a target, not as final visual proof.++Use `ax_action` for structural actions:++```json+{+ "tool_name": "computer_use_remote",+ "tool_args": {+ "action": "ax_action",+ "target": {+ "role": "push button",+ "title": "OK"+ },+ "operation": "press"+ }+}+```++Supported operations are:++- `press`: activate a button, menu item, tab, checkbox, or similar action-bearing node+- `focus`: focus a focusable node before typing or keyboard input+- `set_value`: set text/value on editable nodes; pass `value` or `text`++Targeting options:++- Prefer a semantic `target` when a node has a stable role plus title/name/description/text/state/action.+- Use a `path` returned by the latest `ax_snapshot` only while the UI is unchanged.+- If an action reports ambiguity, take a fresh snapshot and narrow the target with role plus title/name/description.+- If an action reports a missing target, take a fresh snapshot before trying coordinates.++## Wayland Notes++Use screenshots for proof after every state-changing action. AT-SPI actions and keyboard events are attempts, not proof, and Wayland focus can reject or redirect input when the active window changes.++On GNOME/Wayland, useful shortcuts include:++- `Super+H`: hide the active window+- `Alt+Tab`: switch applications+- `Ctrl+L`: focus a browser address bar when the browser is already focused+- `Ctrl+T`: open a new browser tab when the browser is already focused++Treat every shortcut as an attempt. Inspect the fresh screenshot before saying it worked. If text lands in the wrong app, stop and reassess from `capture` or `ax_snapshot`; do not continue typing from assumed focus.++Some apps expose shallow AT-SPI trees unless their own accessibility support is enabled. If the AT-SPI tree is too shallow for a task, fall back in this order: app-native/browser tooling, reliable keyboard paths, then normalized coordinate clicks from a fresh screenshot.++## Permissions++If `computer_use_remote` returns `COMPUTER_USE_AX_UNAVAILABLE`, `COMPUTER_USE_REARM_REQUIRED`, `COMPUTER_USE_APPROVAL_REQUIRED`, or `status=rearm required`, stop immediately and ask the user to re-arm or fix the Linux desktop accessibility/session state.++Do not bypass a permission or host-visibility failure with server screenshots, Docker commands, the built-in Linux Desktop/Xpra skill, or `code_execution_tool`.diff --git a/plugins/_a0_connector/skills/host-computer-use/SKILL.md b/plugins/_a0_connector/skills/host-computer-use/SKILL.mdindex b5a8bc0dc..d7d6f8b9d 100644--- a/plugins/_a0_connector/skills/host-computer-use/SKILL.md+++ b/plugins/_a0_connector/skills/host-computer-use/SKILL.md@@ -79,6 +79,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi## Backend Skills+- If the backend is Linux/Wayland or features include `atspi-tree-snapshot` / `atspi-structural-targeting`, load `host-computer-use-linux` before using Linux AT-SPI structural actions.- If the backend is macOS or features include `accessibility-tree-snapshot` / `accessibility-structural-targeting`, load `host-computer-use-macos` before using macOS structural Accessibility actions.- If the backend is Windows or features include `uia-tree-snapshot` / `uia-structural-targeting`, load `host-computer-use-windows` before using Windows UI Automation structural actions.- Do not use backend-specific actions just because their argument names exist in the generic contract. Treat them as unavailable unless the connected CLI advertises the matching feature.@@ -88,6 +89,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi- Only the latest screenshot or a definitive tool result counts as evidence.- If a tool result says a screenshot was attached but you cannot actually see the image, stop and report that visual verification is unavailable. Do not continue with another action from an assumed host state.- Outside advertised structural accessibility support, use normalized global screen coordinates; do not assume window ids, element indexes, background-safe input, or semantic click targets unless the runtime explicitly advertises them.+- On Linux, AT-SPI structural targeting uses backend-specific actions documented in `host-computer-use-linux`; do not apply macOS AX-specific assumptions unless the backend is macOS.- Prefer accessibility and semantic UI paths first: shortcuts, command palettes, menu accelerators, address/search bars, focus traversal, and other keyboard-accessible controls.- Prefer `key` and `type` over pointer actions whenever a reliable keyboard path exists.- When a menu or popup is open, treat it as the active UI and prefer keyboard navigation over clicking small transient rows by coordinate.diff --git a/plugins/_a0_connector/tools/computer_use_remote.py b/plugins/_a0_connector/tools/computer_use_remote.pyindex bb192b7ee..875ffec51 100644--- a/plugins/_a0_connector/tools/computer_use_remote.py+++ b/plugins/_a0_connector/tools/computer_use_remote.py@@ -480,6 +480,20 @@ class ComputerUseRemote(Tool):backend_id = str(data.get("backend_id", "") or "").strip().lower()backend_family = str(data.get("backend_family", "") or "").strip().lower()features = {feature.lower() for feature in self._backend_features(data)}+ has_linux_atspi = bool(+ features+ & {+ "atspi-tree-snapshot",+ "atspi-structural-targeting",+ "atspi-element-action",+ "atspi-set-value",+ }+ )+ if backend_id in {"wayland", "x11", "linux"} or backend_family == "linux" or has_linux_atspi:+ return (+ " Load skill `host-computer-use-linux` before using Linux AT-SPI "+ "structural actions."+ )has_macos_ax = bool(features& {@@ -548,6 +562,7 @@ class ComputerUseRemote(Tool):return (f"AX snapshot for {app_name}: {node_count} node(s){truncated}. "f"Root {root_label}. Use path or semantic target fields with ax_action."+ f"{self._structural_tree_outline(tree)}")def _format_uia_snapshot(self, data: dict[str, Any]) -> str:@@ -562,8 +577,72 @@ class ComputerUseRemote(Tool):f"Root {root_label}. Prefer node actions with uia_action; use "f"focus_window/minimize/restore/maximize for windows, and reserve click "f"for a last resort."+ f"{self._structural_tree_outline(tree)}")+ def _structural_tree_outline(self, tree: dict[str, Any], *, max_lines: int = 80) -> str:+ if not tree:+ return ""+ lines: list[str] = ["", "", "Nodes:"]+ truncated = False++ def visit(node: dict[str, Any], depth: int) -> None:+ nonlocal truncated+ if len(lines) - 3 >= max_lines:+ truncated = True+ return+ lines.append(self._structural_node_line(node, depth=depth))+ children = node.get("children")+ if not isinstance(children, list):+ return+ for child in children:+ if len(lines) - 3 >= max_lines:+ truncated = True+ break+ if isinstance(child, dict):+ visit(child, depth + 1)++ visit(tree, 0)+ if truncated:+ lines.append("... outline truncated; request a narrower max_depth/max_nodes snapshot if needed.")+ return "\n".join(lines)++ def _structural_node_line(self, node: dict[str, Any], *, depth: int) -> str:+ indent = " " * max(0, depth)+ role = str(node.get("role") or "element")+ path = node.get("path", [])+ parts = [f"{indent}- path={path} role={role}"]+ for key in ("title", "name", "description", "automation_id", "class_name", "selector"):+ value = node.get(key)+ if isinstance(value, str) and value.strip():+ parts.append(f"{key}={value.strip()[:120]!r}")+ break+ frame = node.get("frame")+ if isinstance(frame, dict):+ x = frame.get("x", "?")+ y = frame.get("y", "?")+ width = frame.get("width", "?")+ height = frame.get("height", "?")+ parts.append(f"frame=({x},{y} {width}x{height})")+ actions = node.get("actions")+ if isinstance(actions, list) and actions:+ names = [+ str(item.get("name") or "").strip()+ for item in actions+ if isinstance(item, dict) and str(item.get("name") or "").strip()+ ]+ if names:+ parts.append(f"actions={','.join(names[:6])}")+ states = node.get("states")+ if isinstance(states, list) and states:+ values = [str(item).strip() for item in states if str(item).strip()]+ if values:+ parts.append(f"states={','.join(values[:8])}")+ text = node.get("text")+ if isinstance(text, str) and text.strip():+ parts.append(f"text={text.strip()[:120]!r}")+ return " ".join(parts)+def _ax_target_label(self, target: dict[str, Any]) -> str:role = str(target.get("role") or "element")title = str(target.get("title") or target.get("description") or target.get("identifier") or "").strip()