Skip to content

Agent Zero AI framework

Harden remote Linux computer use targeting

Alessandro4cea5d7197b8
diff --git a/plugins/_a0_connector/AGENTS.md b/plugins/_a0_connector/AGENTS.md
index b74c5b726..5c146ef8b 100644
--- a/plugins/_a0_connector/AGENTS.md
+++ b/plugins/_a0_connector/AGENTS.md
@@ -52,6 +52,7 @@
after all chunks for the `op_id` are assembled.
- Host browser status metadata may advertise `available_browsers` entries with browser ids, labels, CDP endpoints, status, and enabled state; keep older CLI payloads without those fields compatible.
- Model preset definitions exposed through v1 are global; project arguments select scope but never create project-owned definitions. Model switcher state reports the effective main, utility, and embedding models and preserves embedding-change notifications.
+- Computer Use receipts describe transport success unless the connector returns explicit effect evidence. Linux target-bound typing requires a verified active/focused `window_id`; window activation uses focus, never a press action on an application or window node. Do not retry an identical failed Computer Use call.
## Work Guidance
diff --git a/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md b/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
index 0482eb369..39de1b82b 100644
--- a/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
+++ b/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
@@ -8,11 +8,15 @@ This is the only desktop-control path for the user's connected host/local comput
If the tool reports no CLI/Launcher host bridge, disabled computer use, or `COMPUTER_USE_REARM_REQUIRED`, stop and tell the user to run `/computer-use on` in the A0 Launcher chat when using Launcher Host access, or in A0 CLI otherwise, and approve any host permission prompt.
+If a Computer Use call returns an error, do not repeat the same action with identical arguments. Report the error, or use a materially different safe recovery only when the task still requires it.
+
Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. Read `backend_id`, `backend_family`, `features`, and the structured `capabilities` object in status/session results. When capabilities report native windows, window state, element indexes, and background dispatch, prefer `list_windows` -> `get_window_state` -> `element_action` with `dispatch: "background"` before using global coordinates. Interactive coordinate actions should use normalized global-screen coordinates from the most recent capture.
Some actions are backend-specific and intentionally documented only in backend skills. If `status` or `start_session` reports backend-specific features or tells you to load a backend skill, load and follow that skill before using those backend-only actions. For structural targeting details, load and follow the backend-specific skill such as `host-computer-use-macos` or `host-computer-use-windows`; do not apply one backend's guidance to another backend.
-State-changing actions automatically attach a fresh screen after they run unless the backend returns a definitive structural background result. Treat key presses, clicks, scrolling, and typing as attempts, not success; treat foreground fallbacks the same way. Inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result only proves keystrokes were sent; it does not prove that text landed in the intended place.
+State-changing actions automatically attach a fresh screen after they run unless the backend returns a definitive structural background result. Treat key presses, clicks, scrolling, and typing as attempts, not success; treat foreground fallbacks the same way. Inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result proves only that keyboard events were sent unless it explicitly reports both `focus_verified=true` and the target `window_id`.
+
+When Linux advertises `verified-window-focus` and `target-verified-keyboard-input`, use a real frame/window from `list_windows`, inspect it with `get_window_state`, and focus its window element with foreground `element_action`. Continue to `type` only after the focus result reports `focus_verified=true`, and pass that same `window_id` to `type`. Never press an application/frame/window node to activate it.
```json
{
@@ -28,7 +32,7 @@ Required argument:
Optional arguments by action:
- `session_id`: session returned by `start_session`
-- `pid`, `window_id`: target a native app/window for `get_window_state` and `element_action`
+- `pid`, `window_id`: target a native window for `get_window_state`, `element_action`, scoped backend snapshots, and guarded keyboard input when supported
- `element_index`: target an element from the latest `get_window_state`
- `operation`: action such as `invoke`, `press`, `set_value`, `focus`, or backend-specific operations
- `dispatch`: `background`, `auto`, or `foreground`; prefer `background` for `element_action`
@@ -37,7 +41,7 @@ Optional arguments by action:
- `count`: click count for `click`
- `dx`, `dy`: scroll amounts for `scroll`
- `key` or `keys`: key press value for `key`
-- `text`: text to type for `type`
+- `text`: text to type for `type`; Linux target-verified input also requires `window_id`
- `submit`: boolean Enter-after-type flag for `type`
Status/session results may include `contract_version` and `capabilities`. Treat `capabilities.identity.pid`, `capabilities.identity.window_id`, `capabilities.identity.element_index`, and `capabilities.dispatch.background` as the authoritative cross-platform contract for whether the native background loop is available. Use `features` for backend-specific refinements and skill selection.
diff --git a/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md b/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md
index a9d38adc7..43748bee8 100644
--- a/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md
+++ b/plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md
@@ -29,13 +29,14 @@ Use `ax_snapshot` to inspect the Linux AT-SPI tree:
"tool_name": "computer_use_remote",
"tool_args": {
"action": "ax_snapshot",
+ "window_id": "<window_id from list_windows>",
"max_depth": 4,
"max_nodes": 200
}
}
```
-The snapshot returns paths, roles, names/titles, descriptions, frames, states, actions, text previews, values, and child nodes. Use it to choose a target, not as final visual proof.
+Pass `window_id` whenever one is known so unrelated applications cannot consume the node budget. The snapshot returns paths, roles, names/titles, descriptions, frames, states, actions, text previews, values, and child nodes. Use it to choose a target, not as final visual proof.
Use `ax_action` for structural actions:
@@ -55,7 +56,7 @@ Use `ax_action` for structural actions:
Supported operations are:
-- `press`: activate a button, menu item, tab, checkbox, or similar action-bearing node
+- `press`: activate a button, menu item, tab, checkbox, or similar action-bearing node; never use it on an application/frame/window
- `focus`: focus a focusable node before typing or keyboard input
- `set_value`: set text/value on editable nodes; pass `value` or `text`
@@ -72,6 +73,10 @@ Use screenshots for proof after every state-changing action. AT-SPI actions and
True background dispatch on Linux is compositor, toolkit, and app dependent. Do not claim a Linux action was background-safe unless the tool result explicitly says `actual_dispatch=background`.
+To bring a Linux window forward, target its frame/window element from `get_window_state` with `element_action`, operation `focus`, and `dispatch: "foreground"` or `"auto"`. Continue only when the result says `focus_verified=true`; an accepted AT-SPI call without active/focused state is not activation proof.
+
+Linux text injection is target-guarded. Pass the same verified active `window_id` to `type`. If the tool reports `COMPUTER_USE_WINDOW_REQUIRED` or `COMPUTER_USE_TARGET_NOT_FOCUSED`, do not type globally and do not substitute another application target.
+
On GNOME/Wayland, useful shortcuts include:
- `Super+H`: hide the active window
diff --git a/plugins/_a0_connector/skills/host-computer-use/SKILL.md b/plugins/_a0_connector/skills/host-computer-use/SKILL.md
index 08bab40b7..142a21f17 100644
--- a/plugins/_a0_connector/skills/host-computer-use/SKILL.md
+++ b/plugins/_a0_connector/skills/host-computer-use/SKILL.md
@@ -80,10 +80,12 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
4. If the backend advertises native window listing through capabilities or `native-window-list`, call `list_windows` before using coordinates.
5. If the backend advertises window state and element-index targeting through capabilities or features, call `get_window_state` for the target `pid`/`window_id`, then use `element_action` with `dispatch: "background"` by default.
6. If `element_action` reports `background_unavailable`, use `dispatch: "auto"` or `dispatch: "foreground"` only when foreground control is acceptable for the user/task.
-7. Decide final success from the latest screenshot or a definitive structural result, not from memory.
-8. Interactive actions already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
-9. Use `status` for state without starting a session.
-10. Use `capture` only when you need another screenshot without taking an action.
+7. When the backend advertises verified window focus, activate a window only with foreground `element_action` operation `focus`; require `focus_verified=true`. Never use `press` on an application/frame/window node as activation.
+8. When the backend advertises target-verified keyboard input, pass the verified active `window_id` to `type`; a missing, inactive, or unverifiable target must fail closed.
+9. Decide final success from the latest screenshot or a definitive structural result, not from memory.
+10. Interactive actions already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
+11. Use `status` for state without starting a session.
+12. Use `capture` only when you need another screenshot without taking an action.
## Backend Skills
@@ -107,7 +109,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
- If the same approach has already failed twice without visible progress, switch strategy instead of repeating it.
- Do not infer focus or task completion from chat logs, sidebars, tool summaries, or status text.
- Never claim a state-changing action succeeded until the latest screenshot visibly confirms it.
-- A `type` tool result only confirms keystrokes were sent. It is not evidence that the text landed in the intended application.
+- A `type` tool result confirms the destination only when it reports `focus_verified=true` with the intended `window_id`; otherwise it confirms only global keyboard events were sent.
- For browser-navigation tasks done through this tool, only claim success if the browser content area visibly shows the destination page or result.
- If the attached screenshot appears unchanged after a state-changing action, use one explicit `capture` to verify before repeating the same action.
- Use `type(..., submit=true)` only for URL or navigation-style entry where Enter should fire immediately after typing.
diff --git a/plugins/_a0_connector/tools/computer_use_remote.py b/plugins/_a0_connector/tools/computer_use_remote.py
index 8de1cd57f..c0ea899e4 100644
--- a/plugins/_a0_connector/tools/computer_use_remote.py
+++ b/plugins/_a0_connector/tools/computer_use_remote.py
@@ -332,6 +332,8 @@ class ComputerUseRemote(Tool):
payload["key"] = self.args.get("key")
elif action == "type":
payload["text"] = self.args.get("text", "")
+ if "window_id" in self.args:
+ payload["window_id"] = self.args.get("window_id")
if self._coerce_bool(self.args.get("submit")):
payload["submit"] = True
elif action == "list_windows":
@@ -363,6 +365,9 @@ class ComputerUseRemote(Tool):
if "selector" in self.args:
payload["selector"] = self.args.get("selector")
elif action == "ax_snapshot":
+ for key in ("pid", "window_id"):
+ if key in self.args:
+ payload[key] = self.args.get(key)
if "max_depth" in self.args:
payload["max_depth"] = self._coerce_int(self.args.get("max_depth"), name="max_depth")
if "max_nodes" in self.args:
@@ -471,9 +476,17 @@ class ComputerUseRemote(Tool):
return f"Sent keys: {keys!r}."
if action == "type":
text = str(data.get("text", "") or "")
- if data.get("submitted"):
- return f"Typed {len(text)} character(s) and submitted."
- return f"Typed {len(text)} character(s)."
+ window_id = str(data.get("window_id") or "").strip()
+ submitted = " and submitted" if data.get("submitted") else ""
+ if data.get("focus_verified") and window_id:
+ return (
+ f"Sent {len(text)} keyboard character(s){submitted} to verified active "
+ f"window_id={window_id}."
+ )
+ return (
+ f"Sent {len(text)} global keyboard character(s){submitted}; destination was not verified. "
+ "Inspect the attached screen before claiming where the text landed."
+ )
return str(data)
def _format_error(self, result: dict[str, Any]) -> str:
@@ -639,7 +652,7 @@ class ComputerUseRemote(Tool):
f"{frame.get('width', '?')}x{frame.get('height', '?')})"
)
flags: list[str] = []
- for flag in ("is_on_screen", "on_current_space", "focused", "visible"):
+ for flag in ("is_on_screen", "on_current_space", "active", "focused", "visible"):
if flag in item:
flags.append(f"{flag}={item.get(flag)}")
if flags:
@@ -658,10 +671,16 @@ class ComputerUseRemote(Tool):
node_count = data.get("node_count", "?")
truncated = " truncated" if data.get("truncated") else ""
mode = str(data.get("mode") or "auto").strip()
+ state_parts = [
+ f"{flag}={window.get(flag)}"
+ for flag in ("active", "focused")
+ if flag in window
+ ]
+ state_text = f" {' '.join(state_parts)}." if state_parts else ""
return (
f"Window state for {title!r}"
f"{f' window_id={window_id}' if window_id else ''}: "
- f"{node_count} element(s){truncated}, mode={mode}. "
+ f"{node_count} element(s){truncated}, mode={mode}.{state_text} "
"Use element_action with element_index; dispatch defaults to background."
f"{self._structural_tree_outline(tree)}"
)
@@ -688,7 +707,8 @@ class ComputerUseRemote(Tool):
f"requested_dispatch={requested_dispatch}, actual_dispatch={actual_dispatch}"
f"{', foreground_fallback_used=true' if fallback else ''}"
)
- return f"Performed {operation} on element_index={index} {label}; {dispatch_text}."
+ verification = ", focus_verified=true" if data.get("focus_verified") else ""
+ return f"Performed {operation} on element_index={index} {label}; {dispatch_text}{verification}."
def _format_ax_snapshot(self, data: dict[str, Any]) -> str:
app = data.get("app") if isinstance(data.get("app"), dict) else {}
@@ -697,8 +717,10 @@ class ComputerUseRemote(Tool):
node_count = data.get("node_count", "?")
truncated = " truncated" if data.get("truncated") else ""
root_label = self._ax_target_label(tree)
+ window_id = str(data.get("window_id") or "").strip()
+ scope = f" scoped to window_id={window_id}" if data.get("scoped") and window_id else ""
return (
- f"AX snapshot for {app_name}: {node_count} node(s){truncated}. "
+ f"AX snapshot for {app_name}{scope}: {node_count} node(s){truncated}. "
f"Root {root_label}. Use path or semantic target fields with ax_action."
f"{self._structural_tree_outline(tree)}"
)
diff --git a/tests/test_a0_connector_prompt_gating.py b/tests/test_a0_connector_prompt_gating.py
index 157af2cf5..7ea914efd 100644
--- a/tests/test_a0_connector_prompt_gating.py
+++ b/tests/test_a0_connector_prompt_gating.py
@@ -283,6 +283,14 @@ def test_computer_use_remote_prompt_requires_visual_verification_after_actions()
/ "host-computer-use"
/ "SKILL.md"
).read_text(encoding="utf-8")
+ linux_skill = (
+ PROJECT_ROOT
+ / "plugins"
+ / "_a0_connector"
+ / "skills"
+ / "host-computer-use-linux"
+ / "SKILL.md"
+ ).read_text(encoding="utf-8")
assert "Treat key presses, clicks, scrolling, and typing" in prompt
assert "attempts, not success" in prompt
@@ -294,8 +302,13 @@ def test_computer_use_remote_prompt_requires_visual_verification_after_actions()
assert "minimize" not in prompt.lower()
assert "window-manager" not in prompt
assert "cannot actually see the image" in skill
- assert "A `type` tool result only confirms keystrokes were sent" in skill
+ assert "A `type` tool result confirms the destination only when" in skill
assert "visibly confirms" in skill
+ assert "target-verified-keyboard-input" in prompt
+ assert "focus_verified=true" in prompt
+ assert "do not repeat the same action with identical arguments" in prompt
+ assert "Pass the same verified active `window_id` to `type`" in linux_skill
+ assert "never use it on an application/frame/window" in linux_skill
assert "hide window" not in skill
assert "minimize window" not in skill
assert "hide/minimize" not in skill
diff --git a/tests/test_tool_action_contracts.py b/tests/test_tool_action_contracts.py
index a4cb49393..45defc0fb 100644
--- a/tests/test_tool_action_contracts.py
+++ b/tests/test_tool_action_contracts.py
@@ -1140,6 +1140,84 @@ def test_computer_use_remote_start_session_reports_backend_features_and_windows_
assert "host-computer-use-windows" in message
+def test_computer_use_remote_forwards_linux_window_scope_and_type_guard(monkeypatch):
+ module = _load_computer_use_remote_tool(monkeypatch)
+ tool = object.__new__(module.ComputerUseRemote)
+ window_id = "atspi-pid:57929:path:20.0"
+
+ tool.args = {
+ "action": "ax_snapshot",
+ "pid": 57929,
+ "window_id": window_id,
+ "max_depth": 4,
+ "max_nodes": 80,
+ }
+ snapshot = tool._build_payload(op_id="op-snapshot", context_id="ctx", action="ax_snapshot")
+ tool.args = {"action": "type", "window_id": window_id, "text": "hello", "submit": True}
+ typed = tool._build_payload(op_id="op-type", context_id="ctx", action="type")
+
+ assert snapshot["pid"] == 57929
+ assert snapshot["window_id"] == window_id
+ assert snapshot["max_depth"] == 4
+ assert snapshot["max_nodes"] == 80
+ assert typed["window_id"] == window_id
+ assert typed["text"] == "hello"
+ assert typed["submit"] is True
+
+
+def test_computer_use_remote_receipts_separate_injection_from_verified_result(monkeypatch):
+ module = _load_computer_use_remote_tool(monkeypatch)
+ tool = object.__new__(module.ComputerUseRemote)
+ window_id = "atspi-pid:57929:path:20.0"
+
+ verified = tool._extract_result(
+ "type",
+ {
+ "ok": True,
+ "result": {
+ "text": "hello",
+ "window_id": window_id,
+ "focus_verified": True,
+ },
+ },
+ )
+ unverified = tool._extract_result(
+ "type",
+ {"ok": True, "result": {"text": "hello"}},
+ )
+ focused = tool._extract_result(
+ "element_action",
+ {
+ "ok": True,
+ "result": {
+ "operation": "focus",
+ "target": {"element_index": 0, "role": "frame", "title": "Discord"},
+ "requested_dispatch": "foreground",
+ "actual_dispatch": "foreground",
+ "focus_verified": True,
+ },
+ },
+ )
+ scoped = tool._extract_result(
+ "ax_snapshot",
+ {
+ "ok": True,
+ "result": {
+ "app": {"name": "Discord"},
+ "tree": {"role": "frame", "title": "Discord"},
+ "node_count": 2,
+ "window_id": window_id,
+ "scoped": True,
+ },
+ },
+ )
+
+ assert f"verified active window_id={window_id}" in verified
+ assert "destination was not verified" in unverified
+ assert "focus_verified=true" in focused
+ assert f"scoped to window_id={window_id}" in scoped
+
+
def test_computer_use_remote_capture_artifact_is_chat_scoped(monkeypatch, tmp_path: Path):
module = _load_computer_use_remote_tool(monkeypatch)