Skip to content

Agent Zero AI framework

Split macOS computer-use backend guidance

Alessandro Fraue7cb3aa3fa42
diff --git a/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md b/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
index e5d7bb2bd..3b4ed31d7 100644
--- a/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
+++ b/plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
@@ -8,7 +8,9 @@ This is the only desktop-control path for the user's connected host/local comput
If the tool reports no CLI, disabled computer use, or `COMPUTER_USE_REARM_REQUIRED`, stop and tell the user to run `/computer-use on` in A0 CLI and approve any host permission prompt.
-Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. Interactive actions should use normalized global-screen coordinates from the most recent capture.
+Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. Interactive coordinate actions should use normalized global-screen coordinates from the most recent capture.
+
+Some actions are backend-specific. Use `ax_snapshot` and `ax_action` only when `status` or `start_session` reports a macOS backend with Accessibility-tree features such as `accessibility-tree-snapshot` or `accessibility-structural-targeting`. For macOS structural targeting details, load and follow skill `host-computer-use-macos`; do not apply AX guidance to non-macOS backends.
State-changing actions automatically attach a fresh screen after they run. Treat key presses, clicks, scrolling, typing, and window-manager shortcuts as attempts, not success: inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. For Ubuntu/GNOME/Wayland hide-window tasks, prefer `Super+H` (`{"action":"key","keys":["Super","H"]}`) for the active window; do not use `Alt+F9` as the primary hide/minimize shortcut because it often leaves the window visible. A `type` result only proves keystrokes were sent; it does not prove the window was hidden or that text landed in the intended place.
@@ -22,13 +24,18 @@ State-changing actions automatically attach a fresh screen after they run. Treat
```
Required argument:
-- `action`: one of `start_session`, `status`, `capture`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`
+- `action`: one of `start_session`, `status`, `capture`, `ax_snapshot`, `ax_action`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`
Optional arguments by action:
- `session_id`: session returned by `start_session`
- `x`, `y`: normalized `[0,1]` global-screen coordinates for `move` and `click`
- `button`: `left`, `right`, or `middle` for `click`
- `count`: click count for `click`
+- `max_depth`, `max_nodes`: optional bounds for backend-gated `ax_snapshot`
+- `path`: element path from `ax_snapshot` for backend-gated `ax_action`
+- `target`: semantic element target for backend-gated `ax_action`, for example role/title/description/value/identifier
+- `operation` or `ax_action`: backend-gated AX operation such as `press`, `focus`, or `set_value`
+- `value` or `text`: value for backend-gated `ax_action` with `set_value`
- `dx`, `dy`: scroll amounts for `scroll`
- `key` or `keys`: key press value for `key`
- `text`: text to type for `type`
diff --git a/plugins/_a0_connector/skills/host-computer-use-macos/SKILL.md b/plugins/_a0_connector/skills/host-computer-use-macos/SKILL.md
new file mode 100644
index 000000000..fdd04a835
--- /dev/null
+++ b/plugins/_a0_connector/skills/host-computer-use-macos/SKILL.md
@@ -0,0 +1,83 @@
+---
+name: host-computer-use-macos
+description: Backend-specific macOS guidance for `computer_use_remote`. Load after `status` or `start_session` reports backend_family/backend_id `macos` or Accessibility-tree features. Covers AX structural targeting, macOS window actions, and screenshot verification.
+---
+
+# Host Computer Use - macOS
+
+Use this after `host-computer-use` when the connected A0 CLI reports the macOS computer-use backend.
+
+Do not use this skill for Linux, Windows, Xpra, Docker, or browser-only tasks. If the backend is not macOS or does not advertise Accessibility-tree support, skip AX actions and follow the generic host computer-use rules.
+
+## macOS AX Targeting
+
+macOS backends can advertise structural Accessibility features:
+
+- `accessibility-tree-snapshot`
+- `accessibility-structural-targeting`
+- `accessibility-element-click`
+
+When these features are present, prefer structural targeting over pixel clicks for named controls such as buttons, menu items, text fields, sheets, alerts, toolbar items, and sidebar rows.
+
+Use `ax_snapshot` to inspect the frontmost app's bounded Accessibility tree:
+
+```json
+{
+ "tool_name": "computer_use_remote",
+ "tool_args": {
+ "action": "ax_snapshot",
+ "max_depth": 4,
+ "max_nodes": 200
+ }
+}
+```
+
+The snapshot returns element paths, roles, labels, frames, enabled/focused state, actions, and child nodes. Use it to choose an element, not as final visual proof.
+
+Use `ax_action` for a structural action:
+
+```json
+{
+ "tool_name": "computer_use_remote",
+ "tool_args": {
+ "action": "ax_action",
+ "target": {
+ "role": "AXButton",
+ "title": "OK"
+ },
+ "operation": "press"
+ }
+}
+```
+
+Supported operations are:
+
+- `press`: activate a button, menu item, checkbox, or similar control
+- `focus`: focus a text field or focusable element
+- `set_value`: set a value; pass `value` or `text`
+
+Targeting options:
+
+- Prefer a semantic `target` when the control has a stable title, description, value, identifier, role, or subrole.
+- Use a `path` returned by the latest `ax_snapshot` only while the UI is unchanged.
+- If an action reports ambiguity, take a fresh snapshot and narrow the target with role plus title/description/identifier.
+- If an action reports a missing target, take a fresh snapshot before trying coordinates.
+
+AX actions are attempts, not proof. They attach a fresh screenshot after state-changing actions; inspect that image before saying the requested outcome happened.
+
+## macOS Window Actions
+
+For active-app window tasks, macOS shortcuts are usually:
+
+- `Command+H`: hide the active app
+- `Command+M`: minimize the active window
+
+Treat these as attempts. After any hide or minimize shortcut, inspect the fresh screenshot. If the target window is still visible, say the attempt failed or switch strategy; do not type follow-up text into the focused app unless the screenshot clearly shows the intended target.
+
+When a visible close/minimize/full-screen button or menu item is accessible in the AX tree, prefer `ax_snapshot` plus `ax_action` over a small coordinate click.
+
+## Permissions
+
+macOS Screen Recording affects screenshots. Accessibility/Input Monitoring affect structural targeting and input. If `computer_use_remote` returns `COMPUTER_USE_REARM_REQUIRED`, `COMPUTER_USE_APPROVAL_REQUIRED`, or `status=rearm required`, stop immediately and ask the user to run `/computer-use on` in the A0 CLI and approve the macOS prompt if shown.
+
+Do not bypass a permission failure with server screenshots, Docker commands, `linux-desktop`, or browser fallbacks.
diff --git a/plugins/_a0_connector/skills/host-computer-use/SKILL.md b/plugins/_a0_connector/skills/host-computer-use/SKILL.md
index 20b1e7eaf..ccb501ce3 100644
--- a/plugins/_a0_connector/skills/host-computer-use/SKILL.md
+++ b/plugins/_a0_connector/skills/host-computer-use/SKILL.md
@@ -57,8 +57,9 @@ Use:
Arguments:
-- `action`: `start_session`, `status`, `capture`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`
+- `action`: `start_session`, `status`, `capture`, `ax_snapshot`, `ax_action`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`
- `session_id`: optional after `start_session`
+- `ax_snapshot`, `ax_action`: backend-gated structural accessibility actions; use only when backend metadata advertises matching support, and load the backend-specific skill first
- `move`: `x`, `y` normalized to `[0,1]`
- `click`: optional `x`, `y`, optional `button` (`left`, `right`, `middle`), optional `count`
- `scroll`: `dx`, `dy`
@@ -72,16 +73,22 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
## Core Loop
1. Call `start_session` first.
-2. Decide from the latest screenshot, not from memory.
-3. Interactive actions (`move`, `click`, `scroll`, `key`, `type`) already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
-4. Use `status` for state without starting a session.
-5. Use `capture` only when you need another screenshot without taking an action.
+2. Read the returned `backend_id`, `backend_family`, and `features`; load a backend-specific Computer Use skill when the task needs backend-only affordances.
+3. Decide final success from the latest screenshot, not from memory.
+4. Interactive actions (`ax_action`, `move`, `click`, `scroll`, `key`, `type`) already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
+5. Use `status` for state without starting a session.
+6. Use `capture` only when you need another screenshot without taking an action.
+
+## Backend Skills
+
+- If the backend is macOS or features include `accessibility-tree-snapshot` / `accessibility-structural-targeting`, load `host-computer-use-macos` before using `ax_snapshot` or `ax_action`.
+- Do not use backend-specific actions just because their argument names exist in the generic contract. Treat them as unavailable unless the connected CLI advertises the matching feature.
## Operating Rules
- Only the latest screenshot or a definitive tool result counts as evidence.
- If a tool result says a screenshot was attached but you cannot actually see the image, stop and report that visual verification is unavailable. Do not continue with another action from an assumed host state.
-- The current API uses normalized global screen coordinates; do not assume window ids, element indexes, background-safe input, or semantic click targets unless the runtime explicitly advertises them.
+- Outside advertised AX support, use normalized global screen coordinates; do not assume window ids, element indexes, background-safe input, or semantic click targets unless the runtime explicitly advertises them.
- Prefer accessibility and semantic UI paths first: shortcuts, command palettes, menu accelerators, address/search bars, focus traversal, and other keyboard-accessible controls.
- Prefer `key` and `type` over pointer actions whenever a reliable keyboard path exists.
- When a menu or popup is open, treat it as the active UI and prefer keyboard navigation over clicking small transient rows by coordinate.
@@ -101,7 +108,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
- Try keyboard scrolling first: `page_down`, `page_up`, `space`, `shift+space`, arrows, `home`, or `end`.
- Use `scroll` when the desired pane is already active or keyboard scrolling cannot target it.
-- Treat `move` and `click` as last-resort actions for controls that cannot be reached through keyboard, accessibility, browser, or app-native tooling.
+- Treat `move` and `click` as last-resort actions for controls that cannot be reached through backend-specific structural targeting, keyboard, browser, or app-native tooling.
- Before clicking, make sure the latest screenshot makes the target unambiguous. Use one deliberate click, then reassess from the fresh screenshot.
## Control Signals
diff --git a/plugins/_a0_connector/tools/computer_use_remote.py b/plugins/_a0_connector/tools/computer_use_remote.py
index f7edcee2b..0f7e3dc25 100644
--- a/plugins/_a0_connector/tools/computer_use_remote.py
+++ b/plugins/_a0_connector/tools/computer_use_remote.py
@@ -34,6 +34,7 @@ REARM_REQUIRED_DEFAULT_MESSAGE = (
)
_AUTO_CAPTURE_ACTIONS = {
"start_session",
+ "ax_action",
"move",
"click",
"scroll",
@@ -46,6 +47,7 @@ _SETTLE_DELAY_CLICK = 0.35
_SETTLE_DELAY_SCROLL = 0.35
_SETTLE_DELAY_KEY = 0.2
_SETTLE_DELAY_TYPE = 0.25
+_SETTLE_DELAY_AX_ACTION = 0.25
_SETTLE_DELAY_GLOBAL_FOCUS = 0.45
_SETTLE_DELAY_PLAIN_ENTER = 0.3
_SETTLE_DELAY_SUBMIT = 0.45
@@ -54,6 +56,8 @@ _SUPPORTED_ACTIONS = {
"start_session",
"status",
"capture",
+ "ax_snapshot",
+ "ax_action",
"move",
"click",
"scroll",
@@ -72,7 +76,8 @@ class ComputerUseRemote(Tool):
return Response(
message=(
"action is required and must be one of: "
- "start_session, status, capture, move, click, scroll, key, type, stop_session"
+ "start_session, status, capture, ax_snapshot, ax_action, "
+ "move, click, scroll, key, type, stop_session"
),
break_loop=False,
)
@@ -255,6 +260,8 @@ class ComputerUseRemote(Tool):
return _SETTLE_DELAY_CLICK
if action == "scroll":
return _SETTLE_DELAY_SCROLL
+ if action == "ax_action":
+ return _SETTLE_DELAY_AX_ACTION
if action == "type" and self._coerce_bool(self.args.get("submit")):
return _SETTLE_DELAY_SUBMIT
if action == "type":
@@ -310,6 +317,24 @@ class ComputerUseRemote(Tool):
payload["text"] = self.args.get("text", "")
if self._coerce_bool(self.args.get("submit")):
payload["submit"] = True
+ elif action == "ax_snapshot":
+ if "max_depth" in self.args:
+ payload["max_depth"] = self._coerce_int(self.args.get("max_depth"), name="max_depth")
+ if "max_nodes" in self.args:
+ payload["max_nodes"] = self._coerce_int(self.args.get("max_nodes"), name="max_nodes")
+ elif action == "ax_action":
+ target = self.args.get("target")
+ if isinstance(target, dict):
+ payload["target"] = dict(target)
+ if "path" in self.args:
+ payload["path"] = self.args.get("path")
+ operation = self.args.get("operation", self.args.get("ax_action", self.args.get("name")))
+ if operation is not None:
+ payload["operation"] = operation
+ if "value" in self.args:
+ payload["value"] = self.args.get("value")
+ if "text" in self.args:
+ payload["text"] = self.args.get("text", "")
return payload
@@ -329,13 +354,27 @@ class ComputerUseRemote(Tool):
if action == "capture":
summary = self._record_capture(data)
return f"Current screen attached: {summary} {CAPTURE_VERIFICATION_NOTE}"
+ if action == "ax_snapshot":
+ return self._format_ax_snapshot(data)
+ if action == "ax_action":
+ target = data.get("target") if isinstance(data.get("target"), dict) else {}
+ operation = str(data.get("operation") or "?")
+ path = target.get("path", "?")
+ return f"Performed AX {operation} on {self._ax_target_label(target)} path={path}."
if action == "status":
return self._format_status(data)
if action == "start_session":
- return (
+ message = (
f"Computer-use session started: session_id={data.get('session_id', '?')} "
f"size={data.get('width', '?')}x{data.get('height', '?')}"
)
+ backend_details = self._format_backend_details(data)
+ if backend_details:
+ message = f"{message}, {backend_details}"
+ skill_hint = self._backend_skill_hint(data)
+ if skill_hint:
+ return f"{message}.{skill_hint}"
+ return message
if action == "stop_session":
return "Computer-use session stopped."
if action == "move":
@@ -373,14 +412,54 @@ class ComputerUseRemote(Tool):
return f"{code}: {error}"
return error
+ def _format_backend_details(self, data: dict[str, Any]) -> str:
+ backend_id = str(data.get("backend_id", "") or "").strip()
+ backend_family = str(data.get("backend_family", "") or "").strip()
+ features = self._backend_features(data)
+ parts: list[str] = []
+ if backend_id:
+ backend_text = backend_id
+ if backend_family:
+ backend_text = f"{backend_text}/{backend_family}"
+ parts.append(f"backend={backend_text}")
+ if features:
+ parts.append(f"features={', '.join(features)}")
+ return ", ".join(parts)
+
+ def _backend_features(self, data: dict[str, Any]) -> list[str]:
+ raw_features = data.get("features") or []
+ if not isinstance(raw_features, (list, tuple, set)):
+ return []
+ features: list[str] = []
+ for feature in raw_features:
+ text = str(feature or "").strip()
+ if text:
+ features.append(text)
+ return features
+
+ def _backend_skill_hint(self, data: dict[str, Any]) -> str:
+ backend_id = str(data.get("backend_id", "") or "").strip().lower()
+ backend_family = str(data.get("backend_family", "") or "").strip().lower()
+ features = {feature.lower() for feature in self._backend_features(data)}
+ has_macos_ax = bool(
+ features
+ & {
+ "accessibility-tree-snapshot",
+ "accessibility-structural-targeting",
+ }
+ )
+ if backend_id == "macos" or backend_family == "macos" or has_macos_ax:
+ return (
+ " Load skill `host-computer-use-macos` before using macOS AX "
+ "structural actions."
+ )
+ return ""
+
def _format_status(self, data: dict[str, Any]) -> str:
status = str(data.get("status", "unknown") or "unknown")
trust_mode = str(data.get("trust_mode", "") or "")
- backend_id = str(data.get("backend_id", "") or "").strip()
- backend_family = str(data.get("backend_family", "") or "").strip()
active_contexts = data.get("active_contexts") or []
active_text = ", ".join(str(item) for item in active_contexts) if active_contexts else "none"
- backend_text = ""
rearm_guidance = ""
if status == "rearm required":
detail = str(data.get("last_error") or "").strip()
@@ -394,20 +473,37 @@ class ComputerUseRemote(Tool):
"is not armed. "
"Stop using computer_use_remote until the user re-arms it."
)
- if backend_id:
- backend_text = backend_id
- if backend_family:
- backend_text = f"{backend_text}/{backend_family}"
- if backend_text:
+ backend_details = self._format_backend_details(data)
+ if backend_details:
return (
f"Computer use status={status}, trust_mode={trust_mode or 'unknown'}, "
- f"backend={backend_text}, active_contexts={active_text}.{rearm_guidance}"
+ f"{backend_details}, active_contexts={active_text}."
+ f"{self._backend_skill_hint(data)}{rearm_guidance}"
)
return (
f"Computer use status={status}, trust_mode={trust_mode or 'unknown'}, "
f"active_contexts={active_text}.{rearm_guidance}"
)
+ def _format_ax_snapshot(self, data: dict[str, Any]) -> str:
+ app = data.get("app") if isinstance(data.get("app"), dict) else {}
+ tree = data.get("tree") if isinstance(data.get("tree"), dict) else {}
+ app_name = str(app.get("name") or app.get("bundle_id") or "frontmost app")
+ node_count = data.get("node_count", "?")
+ truncated = " truncated" if data.get("truncated") else ""
+ root_label = self._ax_target_label(tree)
+ return (
+ f"AX snapshot for {app_name}: {node_count} node(s){truncated}. "
+ f"Root {root_label}. Use path or semantic target fields with ax_action."
+ )
+
+ def _ax_target_label(self, target: dict[str, Any]) -> str:
+ role = str(target.get("role") or "element")
+ title = str(target.get("title") or target.get("description") or target.get("identifier") or "").strip()
+ if title:
+ return f"{role} {title!r}"
+ return role
+
def _record_capture(self, data: dict[str, Any]) -> str:
display_ref, resolved_capture_id = self._resolve_capture_ref(data)
width = data.get("width", "?")
diff --git a/tests/test_a0_connector_prompt_gating.py b/tests/test_a0_connector_prompt_gating.py
index 3e7c54476..044b07d69 100644
--- a/tests/test_a0_connector_prompt_gating.py
+++ b/tests/test_a0_connector_prompt_gating.py
@@ -345,6 +345,14 @@ def test_remote_affordance_skills_parse():
/ "host-computer-use"
/ "SKILL.md"
)
+ macos_computer_skill = _parse_skill_frontmatter(
+ PROJECT_ROOT
+ / "plugins"
+ / "_a0_connector"
+ / "skills"
+ / "host-computer-use-macos"
+ / "SKILL.md"
+ )
assert not legacy_connector_skill.exists()
assert text_editor_skill["name"] == "host-file-editing"
@@ -358,6 +366,8 @@ def test_remote_affordance_skills_parse():
assert "Use instead of linux-desktop" in computer_skill["description"]
assert "host computer" in computer_skill["triggers"]
assert "Ubuntu Wayland desktop" in computer_skill["triggers"]
+ assert macos_computer_skill["name"] == "host-computer-use-macos"
+ assert "macOS guidance" in macos_computer_skill["description"]
def test_remote_tool_stubs_are_self_contained_and_reference_per_tool_skills():
@@ -372,6 +382,14 @@ def test_remote_tool_stubs_are_self_contained_and_reference_per_tool_skills():
/ "host-computer-use"
/ "SKILL.md"
).read_text(encoding="utf-8")
+ macos_computer_skill = (
+ PROJECT_ROOT
+ / "plugins"
+ / "_a0_connector"
+ / "skills"
+ / "host-computer-use-macos"
+ / "SKILL.md"
+ ).read_text(encoding="utf-8")
assert "optionally load skill `host-file-editing`" in text_stub
assert "optionally load skill `host-code-execution`" in exec_stub
@@ -379,8 +397,12 @@ def test_remote_tool_stubs_are_self_contained_and_reference_per_tool_skills():
assert '"tool_name": "code_execution_remote"' in exec_stub
assert '"tool_name": "computer_use_remote"' in computer_stub
assert "load and follow skill `host-computer-use`" in computer_stub
+ assert "host-computer-use-macos" in computer_stub
assert "Do not substitute the `linux-desktop` skill" in computer_stub
assert '"tool_name": "computer_use_remote"' in computer_skill
+ assert '"tool_name": "computer_use_remote"' in macos_computer_skill
+ assert "ax_snapshot" in macos_computer_skill
+ assert "ax_snapshot`/`ax_action` are structural Accessibility targeting" not in computer_skill
assert "Availability, backend support, and trust mode are checked when the tool runs" in computer_stub
assert "not `code_execution_tool`" in exec_stub
assert "not to" in exec_stub
diff --git a/tests/test_skills_runtime.py b/tests/test_skills_runtime.py
index 1f3aea3ab..f37a2aa46 100644
--- a/tests/test_skills_runtime.py
+++ b/tests/test_skills_runtime.py
@@ -291,6 +291,7 @@ def test_renamed_skills_use_standard_frontmatter_only():
PROJECT_ROOT / "skills" / "scheduled-tasks" / "SKILL.md",
PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "host-code-execution" / "SKILL.md",
PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "host-computer-use" / "SKILL.md",
+ PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "host-computer-use-macos" / "SKILL.md",
PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "host-file-editing" / "SKILL.md",
PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "setup-a0-cli" / "SKILL.md",
PROJECT_ROOT / "plugins" / "_browser" / "skills" / "browser-automation" / "SKILL.md",
diff --git a/tests/test_tool_action_contracts.py b/tests/test_tool_action_contracts.py
index 8ded5a19b..e3289d56b 100644
--- a/tests/test_tool_action_contracts.py
+++ b/tests/test_tool_action_contracts.py
@@ -86,6 +86,50 @@ def _load_skills_tool(monkeypatch, skill_root: Path):
return importlib.import_module("tools.skills_tool")
+def _load_computer_use_remote_tool(monkeypatch):
+ _install_tool_stub(monkeypatch)
+
+ history_stub = types.ModuleType("helpers.history")
+
+ class _RawMessage(dict):
+ def __init__(self, raw_content, preview):
+ super().__init__(raw_content=raw_content, preview=preview)
+
+ history_stub.RawMessage = _RawMessage
+ monkeypatch.setitem(sys.modules, "helpers.history", history_stub)
+
+ print_style_stub = types.ModuleType("helpers.print_style")
+ print_style_stub.PrintStyle = lambda *args, **kwargs: types.SimpleNamespace(
+ print=lambda *a, **k: None
+ )
+ monkeypatch.setitem(sys.modules, "helpers.print_style", print_style_stub)
+
+ ws_stub = types.ModuleType("helpers.ws")
+ ws_stub.NAMESPACE = "/test"
+ monkeypatch.setitem(sys.modules, "helpers.ws", ws_stub)
+
+ ws_manager_stub = types.ModuleType("helpers.ws_manager")
+ ws_manager_stub.ConnectionNotFoundError = RuntimeError
+ ws_manager_stub.get_shared_ws_manager = lambda: types.SimpleNamespace(
+ emit_to=lambda *a, **k: None
+ )
+ monkeypatch.setitem(sys.modules, "helpers.ws_manager", ws_manager_stub)
+
+ ws_runtime_stub = types.ModuleType("plugins._a0_connector.helpers.ws_runtime")
+ ws_runtime_stub.clear_pending_computer_use_op = lambda *args, **kwargs: None
+ ws_runtime_stub.computer_use_metadata_for_sid = lambda *args, **kwargs: {}
+ ws_runtime_stub.select_computer_use_target_sid = lambda *args, **kwargs: "sid"
+ ws_runtime_stub.store_pending_computer_use_op = lambda *args, **kwargs: None
+ monkeypatch.setitem(
+ sys.modules,
+ "plugins._a0_connector.helpers.ws_runtime",
+ ws_runtime_stub,
+ )
+
+ sys.modules.pop("plugins._a0_connector.tools.computer_use_remote", None)
+ return importlib.import_module("plugins._a0_connector.tools.computer_use_remote")
+
+
def test_skills_tool_accepts_action_alias_for_search(monkeypatch, tmp_path: Path):
module = _load_skills_tool(monkeypatch, tmp_path)
tool = module.SkillsTool(
@@ -559,6 +603,10 @@ def test_computer_use_remote_is_runtime_checked_standard_tool():
project_root
/ "plugins/_a0_connector/skills/host-computer-use/SKILL.md"
).read_text(encoding="utf-8")
+ macos_skill_text = (
+ project_root
+ / "plugins/_a0_connector/skills/host-computer-use-macos/SKILL.md"
+ ).read_text(encoding="utf-8")
assert standard_prompt_path.exists()
assert not (
@@ -569,5 +617,36 @@ def test_computer_use_remote_is_runtime_checked_standard_tool():
assert "not scoped to a single chat context" in standard_prompt_text
assert "checked when the tool runs" in standard_prompt_text
assert "visual verification is unavailable" in standard_prompt_text
+ assert "host-computer-use-macos" in standard_prompt_text
assert '"tool_name": "computer_use_remote"' in skill_text
+ assert '"tool_name": "computer_use_remote"' in macos_skill_text
+ assert "Backend-specific macOS guidance" in macos_skill_text
assert "Beta desktop control" in skill_text
+
+
+def test_computer_use_remote_start_session_reports_backend_features_and_macos_skill(monkeypatch):
+ module = _load_computer_use_remote_tool(monkeypatch)
+ tool = object.__new__(module.ComputerUseRemote)
+
+ message = tool._extract_result(
+ "start_session",
+ {
+ "ok": True,
+ "result": {
+ "session_id": "s1",
+ "width": 1920,
+ "height": 1080,
+ "backend_id": "macos",
+ "backend_family": "macos",
+ "features": [
+ "accessibility-tree-snapshot",
+ "accessibility-structural-targeting",
+ ],
+ },
+ },
+ )
+
+ assert "session_id=s1" in message
+ assert "backend=macos/macos" in message
+ assert "features=accessibility-tree-snapshot, accessibility-structural-targeting" in message
+ assert "host-computer-use-macos" in message