Skip to content

Agent Zero AI framework

Guide computer-use remote through Linux AT-SPI

Alessandro97953db46ba0
diff --git a/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md b/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md
new file mode 100644
index 000000000..df0fca704
--- /dev/null
+++ b/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md
@@ -0,0 +1,86 @@
+---
+name: host-computer-use-linux
+description: Backend-specific Linux guidance for `computer_use_remote`. Load after `status` or `start_session` reports backend_family `linux`, backend_id `wayland`, or AT-SPI features. Covers AT-SPI structural targeting, Wayland portal caveats, and screenshot verification.
+---
+
+# Host Computer Use - Linux
+
+Use this after `host-computer-use` when the connected A0 CLI reports the Linux/Wayland computer-use backend.
+
+Do not use this skill for macOS, Windows, Xpra, Docker, browser-only tasks, or the internal Agent Zero Desktop. If the backend is not Linux or does not advertise AT-SPI support, skip Linux structural actions and follow the generic host computer-use rules.
+
+## Linux AT-SPI Targeting
+
+Linux backends can advertise structural AT-SPI features:
+
+- `atspi-tree-snapshot`
+- `atspi-structural-targeting`
+- `atspi-element-action`
+- `atspi-set-value`
+
+When these features are present, prefer structural targeting over pixel clicks for named buttons, menu items, text fields, dialogs, toolbar items, tab strips, and application windows.
+
+Use `ax_snapshot` to inspect the Linux AT-SPI tree:
+
+```json
+{
+ "tool_name": "computer_use_remote",
+ "tool_args": {
+ "action": "ax_snapshot",
+ "max_depth": 4,
+ "max_nodes": 200
+ }
+}
+```
+
+The snapshot returns paths, roles, names/titles, descriptions, frames, states, actions, text previews, values, and child nodes. Use it to choose a target, not as final visual proof.
+
+Use `ax_action` for structural actions:
+
+```json
+{
+ "tool_name": "computer_use_remote",
+ "tool_args": {
+ "action": "ax_action",
+ "target": {
+ "role": "push button",
+ "title": "OK"
+ },
+ "operation": "press"
+ }
+}
+```
+
+Supported operations are:
+
+- `press`: activate a button, menu item, tab, checkbox, or similar action-bearing node
+- `focus`: focus a focusable node before typing or keyboard input
+- `set_value`: set text/value on editable nodes; pass `value` or `text`
+
+Targeting options:
+
+- Prefer a semantic `target` when a node has a stable role plus title/name/description/text/state/action.
+- Use a `path` returned by the latest `ax_snapshot` only while the UI is unchanged.
+- If an action reports ambiguity, take a fresh snapshot and narrow the target with role plus title/name/description.
+- If an action reports a missing target, take a fresh snapshot before trying coordinates.
+
+## Wayland Notes
+
+Use screenshots for proof after every state-changing action. AT-SPI actions and keyboard events are attempts, not proof, and Wayland focus can reject or redirect input when the active window changes.
+
+On GNOME/Wayland, useful shortcuts include:
+
+- `Super+H`: hide the active window
+- `Alt+Tab`: switch applications
+- `Ctrl+L`: focus a browser address bar when the browser is already focused
+- `Ctrl+T`: open a new browser tab when the browser is already focused
+
+Treat every shortcut as an attempt. Inspect the fresh screenshot before saying it worked. If text lands in the wrong app, stop and reassess from `capture` or `ax_snapshot`; do not continue typing from assumed focus.
+
+Some apps expose shallow AT-SPI trees unless their own accessibility support is enabled. If the AT-SPI tree is too shallow for a task, fall back in this order: app-native/browser tooling, reliable keyboard paths, then normalized coordinate clicks from a fresh screenshot.
+
+## Permissions
+
+If `computer_use_remote` returns `COMPUTER_USE_AX_UNAVAILABLE`, `COMPUTER_USE_REARM_REQUIRED`, `COMPUTER_USE_APPROVAL_REQUIRED`, or `status=rearm required`, stop immediately and ask the user to re-arm or fix the Linux desktop accessibility/session state.
+
+Do not bypass a permission or host-visibility failure with server screenshots, Docker commands, the built-in Linux Desktop/Xpra skill, or `code_execution_tool`.
diff --git a/plugins/_a0_connector/skills/host-computer-use/SKILL.md b/plugins/_a0_connector/skills/host-computer-use/SKILL.md
index b5a8bc0dc..d7d6f8b9d 100644
--- a/plugins/_a0_connector/skills/host-computer-use/SKILL.md
+++ b/plugins/_a0_connector/skills/host-computer-use/SKILL.md
@@ -79,6 +79,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
## Backend Skills
+- If the backend is Linux/Wayland or features include `atspi-tree-snapshot` / `atspi-structural-targeting`, load `host-computer-use-linux` before using Linux AT-SPI structural actions.
- If the backend is macOS or features include `accessibility-tree-snapshot` / `accessibility-structural-targeting`, load `host-computer-use-macos` before using macOS structural Accessibility actions.
- If the backend is Windows or features include `uia-tree-snapshot` / `uia-structural-targeting`, load `host-computer-use-windows` before using Windows UI Automation structural actions.
- Do not use backend-specific actions just because their argument names exist in the generic contract. Treat them as unavailable unless the connected CLI advertises the matching feature.
@@ -88,6 +89,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
- Only the latest screenshot or a definitive tool result counts as evidence.
- If a tool result says a screenshot was attached but you cannot actually see the image, stop and report that visual verification is unavailable. Do not continue with another action from an assumed host state.
- Outside advertised structural accessibility support, use normalized global screen coordinates; do not assume window ids, element indexes, background-safe input, or semantic click targets unless the runtime explicitly advertises them.
+- On Linux, AT-SPI structural targeting uses backend-specific actions documented in `host-computer-use-linux`; do not apply macOS AX-specific assumptions unless the backend is macOS.
- Prefer accessibility and semantic UI paths first: shortcuts, command palettes, menu accelerators, address/search bars, focus traversal, and other keyboard-accessible controls.
- Prefer `key` and `type` over pointer actions whenever a reliable keyboard path exists.
- When a menu or popup is open, treat it as the active UI and prefer keyboard navigation over clicking small transient rows by coordinate.
diff --git a/plugins/_a0_connector/tools/computer_use_remote.py b/plugins/_a0_connector/tools/computer_use_remote.py
index bb192b7ee..875ffec51 100644
--- a/plugins/_a0_connector/tools/computer_use_remote.py
+++ b/plugins/_a0_connector/tools/computer_use_remote.py
@@ -480,6 +480,20 @@ class ComputerUseRemote(Tool):
backend_id = str(data.get("backend_id", "") or "").strip().lower()
backend_family = str(data.get("backend_family", "") or "").strip().lower()
features = {feature.lower() for feature in self._backend_features(data)}
+ has_linux_atspi = bool(
+ features
+ & {
+ "atspi-tree-snapshot",
+ "atspi-structural-targeting",
+ "atspi-element-action",
+ "atspi-set-value",
+ }
+ )
+ if backend_id in {"wayland", "x11", "linux"} or backend_family == "linux" or has_linux_atspi:
+ return (
+ " Load skill `host-computer-use-linux` before using Linux AT-SPI "
+ "structural actions."
+ )
has_macos_ax = bool(
features
& {
@@ -548,6 +562,7 @@ class ComputerUseRemote(Tool):
return (
f"AX snapshot for {app_name}: {node_count} node(s){truncated}. "
f"Root {root_label}. Use path or semantic target fields with ax_action."
+ f"{self._structural_tree_outline(tree)}"
)
def _format_uia_snapshot(self, data: dict[str, Any]) -> str:
@@ -562,8 +577,72 @@ class ComputerUseRemote(Tool):
f"Root {root_label}. Prefer node actions with uia_action; use "
f"focus_window/minimize/restore/maximize for windows, and reserve click "
f"for a last resort."
+ f"{self._structural_tree_outline(tree)}"
)
+ def _structural_tree_outline(self, tree: dict[str, Any], *, max_lines: int = 80) -> str:
+ if not tree:
+ return ""
+ lines: list[str] = ["", "", "Nodes:"]
+ truncated = False
+
+ def visit(node: dict[str, Any], depth: int) -> None:
+ nonlocal truncated
+ if len(lines) - 3 >= max_lines:
+ truncated = True
+ return
+ lines.append(self._structural_node_line(node, depth=depth))
+ children = node.get("children")
+ if not isinstance(children, list):
+ return
+ for child in children:
+ if len(lines) - 3 >= max_lines:
+ truncated = True
+ break
+ if isinstance(child, dict):
+ visit(child, depth + 1)
+
+ visit(tree, 0)
+ if truncated:
+ lines.append("... outline truncated; request a narrower max_depth/max_nodes snapshot if needed.")
+ return "\n".join(lines)
+
+ def _structural_node_line(self, node: dict[str, Any], *, depth: int) -> str:
+ indent = " " * max(0, depth)
+ role = str(node.get("role") or "element")
+ path = node.get("path", [])
+ parts = [f"{indent}- path={path} role={role}"]
+ for key in ("title", "name", "description", "automation_id", "class_name", "selector"):
+ value = node.get(key)
+ if isinstance(value, str) and value.strip():
+ parts.append(f"{key}={value.strip()[:120]!r}")
+ break
+ frame = node.get("frame")
+ if isinstance(frame, dict):
+ x = frame.get("x", "?")
+ y = frame.get("y", "?")
+ width = frame.get("width", "?")
+ height = frame.get("height", "?")
+ parts.append(f"frame=({x},{y} {width}x{height})")
+ actions = node.get("actions")
+ if isinstance(actions, list) and actions:
+ names = [
+ str(item.get("name") or "").strip()
+ for item in actions
+ if isinstance(item, dict) and str(item.get("name") or "").strip()
+ ]
+ if names:
+ parts.append(f"actions={','.join(names[:6])}")
+ states = node.get("states")
+ if isinstance(states, list) and states:
+ values = [str(item).strip() for item in states if str(item).strip()]
+ if values:
+ parts.append(f"states={','.join(values[:8])}")
+ text = node.get("text")
+ if isinstance(text, str) and text.strip():
+ parts.append(f"text={text.strip()[:120]!r}")
+ return " ".join(parts)
+
def _ax_target_label(self, target: dict[str, Any]) -> str:
role = str(target.get("role") or "element")
title = str(target.get("title") or target.get("description") or target.get("identifier") or "").strip()