👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
feat(click): 添加 SkyLight 后台点击
tisfeng67bcffaaced7
diff --git a/README.md b/README.mdindex 6758b9a..74a7819 100644--- a/README.md+++ b/README.md@@ -174,4 +174,4 @@ Cursor Motion is an open-source cursor motion system for macOS, based on public## License-[MIT](./LICENSE)+[MIT](./LICENSE). Third-party attribution is listed in [THIRD_PARTY_NOTICES.md](./THIRD_PARTY_NOTICES.md).diff --git a/README.zh-CN.md b/README.zh-CN.mdindex 1487d22..d72076a 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -158,4 +158,4 @@ Cursor Motion 是一个面向 macOS 的开源光标运动系统,基于 Softwar## License-[MIT](./LICENSE)+[MIT](./LICENSE)。第三方归属与许可说明见 [THIRD_PARTY_NOTICES.md](./THIRD_PARTY_NOTICES.md)。diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdnew file mode 100644index 0000000..4be3990--- /dev/null+++ b/THIRD_PARTY_NOTICES.md@@ -0,0 +1,61 @@+# Third-Party Notices++## Cua Driver++The macOS `sky_click` event recipe and private SkyLight bridge are derived from Cua Driver:++- Project: https://github.com/trycua/cua+- Source revision: `b8a0f32a06c75225ba24ebb5ab14f6507fa90d15`+- License: MIT++MIT License++Copyright (c) 2025 Cua AI, Inc.++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.++## yabai++The macOS focus-without-raise event-record pattern is derived from yabai:++- Project: https://github.com/asmvik/yabai+- Source revision: `dd845723416f5fe92af49fad5ebab00369e07edd`+- License: MIT++The MIT License (MIT)++Copyright (c) 2019 Åsmund Vikane++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.diff --git a/apps/OpenComputerUseLinux/main.go b/apps/OpenComputerUseLinux/main.goindex c0bc934..70b1421 100644--- a/apps/OpenComputerUseLinux/main.go+++ b/apps/OpenComputerUseLinux/main.go@@ -22,7 +22,7 @@ import (var version = "0.2.1"-var clickMethodValues = []string{"auto", "accessibility", "app_post", "global"}+var clickMethodValues = []string{"auto", "accessibility", "app_post", "sky_click", "global"}//go:embed runtime.pyvar linuxRuntimeScript string@@ -298,6 +298,9 @@ func (s *service) click(app, elementIndex string, x, y *float64, clickCount int,if clickMethod == "app_post" {return textResult("click_method 'app_post' is not supported on Linux", true)}+ if clickMethod == "sky_click" {+ return textResult("click_method 'sky_click' is not supported on Linux", true)+ }if clickMethod == "global" && !globalPointerFallbacksEnabled() {return textResult("click_method 'global' requires OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 because it may move the system pointer and change foreground focus", true)}@@ -1146,7 +1149,7 @@ func toolDefinitions() []toolDefinition {"y": numberProperty("Y coordinate in screenshot pixel coordinates"),"click_count": integerProperty("Number of clicks. Defaults to 1"),"mouse_button": enumStringProperty("Mouse button to click. Defaults to left.", []string{"left", "right", "middle"}),- "click_method": enumStringProperty("Click implementation: auto (default), accessibility, app_post, or global. Accessibility requires element_index. Linux supports global AT-SPI mouse synthesis and does not currently support app_post.", clickMethodValues),+ "click_method": enumStringProperty("Click implementation: auto (default), accessibility, app_post, sky_click, or global. Accessibility requires element_index. Linux supports global AT-SPI mouse synthesis and does not currently support app_post or sky_click.", clickMethodValues),}, []string{"app"}),},{diff --git a/apps/OpenComputerUseLinux/main_test.go b/apps/OpenComputerUseLinux/main_test.goindex d3ccd78..73ef1db 100644--- a/apps/OpenComputerUseLinux/main_test.go+++ b/apps/OpenComputerUseLinux/main_test.go@@ -21,7 +21,7 @@ func TestClickMethodSchemaAndParser(t *testing.T) {properties := tool.InputSchema["properties"].(map[string]any)method := properties["click_method"].(map[string]any)values := method["enum"].([]string)- if strings.Join(values, ",") != "auto,accessibility,app_post,global" {+ if strings.Join(values, ",") != "auto,accessibility,app_post,sky_click,global" {t.Fatalf("click_method enum = %#v", values)}@@ -30,6 +30,7 @@ func TestClickMethodSchemaAndParser(t *testing.T) {" AUTO ": "auto","Accessibility": "accessibility","app_post": "app_post",+ "SKY_CLICK": "sky_click","GLOBAL": "global",} {got, err := parseClickMethod(input)@@ -42,7 +43,7 @@ func TestClickMethodSchemaAndParser(t *testing.T) {}for _, input := range []string{"physical", "targeted"} {- if _, err := parseClickMethod(input); err == nil || !strings.Contains(err.Error(), "Expected one of: auto, accessibility, app_post, global") {+ if _, err := parseClickMethod(input); err == nil || !strings.Contains(err.Error(), "Expected one of: auto, accessibility, app_post, sky_click, global") {t.Fatalf("parseClickMethod(%s) error = %v", input, err)}}@@ -57,6 +58,11 @@ func TestLinuxClickMethodSafetyAndPlatformSupport(t *testing.T) {t.Fatalf("app_post click result = %#v", result)}+ result = service.click("Text Editor", "", &x, &y, 1, "left", "sky_click")+ if !result.IsError || result.Content[0].Text != "click_method 'sky_click' is not supported on Linux" {+ t.Fatalf("sky_click result = %#v", result)+ }+t.Setenv("OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS", "")result = service.click("Text Editor", "", &x, &y, 1, "left", "global")if !result.IsError || !strings.Contains(result.Content[0].Text, "requires OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1") {diff --git a/apps/OpenComputerUseLinux/runtime.py b/apps/OpenComputerUseLinux/runtime.pyindex 798bd8e..a323dd9 100644--- a/apps/OpenComputerUseLinux/runtime.py+++ b/apps/OpenComputerUseLinux/runtime.py@@ -800,6 +800,8 @@ def perform_operation(operation):)elif click_method == "app_post":raise RuntimeError("click_method 'app_post' is not supported on Linux")+ elif click_method == "sky_click":+ raise RuntimeError("click_method 'sky_click' is not supported on Linux")elif click_method == "global":x, y = screen_point(bounds,diff --git a/apps/OpenComputerUseWindows/main.go b/apps/OpenComputerUseWindows/main.goindex 5bb1cf0..d59ebc6 100644--- a/apps/OpenComputerUseWindows/main.go+++ b/apps/OpenComputerUseWindows/main.go@@ -19,7 +19,7 @@ import (var version = "0.2.1"-var clickMethodValues = []string{"auto", "accessibility", "app_post", "global"}+var clickMethodValues = []string{"auto", "accessibility", "app_post", "sky_click", "global"}//go:embed runtime.ps1var windowsRuntimeScript string@@ -295,6 +295,9 @@ func (s *service) click(app, elementIndex string, x, y *float64, clickCount int,if clickMethod == "global" {return textResult("click_method 'global' is not supported on Windows", true)}+ if clickMethod == "sky_click" {+ return textResult("click_method 'sky_click' is not supported on Windows", true)+ }snapshot := s.currentSnapshot(app)if snapshot == nil {return textResult("No app state is available for "+app+". Run get_app_state before action tools.", true)@@ -717,7 +720,7 @@ func toolDefinitions() []toolDefinition {"y": numberProperty("Y coordinate in screenshot pixel coordinates"),"click_count": integerProperty("Number of clicks. Defaults to 1"),"mouse_button": enumStringProperty("Mouse button to click. Defaults to left.", []string{"left", "right", "middle"}),- "click_method": enumStringProperty("Click implementation: auto (default), accessibility, app_post, or global. Accessibility requires element_index. Windows supports app_post through HWND messages and does not currently support global.", clickMethodValues),+ "click_method": enumStringProperty("Click implementation: auto (default), accessibility, app_post, sky_click, or global. Accessibility requires element_index. Windows supports app_post through HWND messages and does not currently support sky_click or global.", clickMethodValues),}, []string{"app"}),},{diff --git a/apps/OpenComputerUseWindows/main_test.go b/apps/OpenComputerUseWindows/main_test.goindex 2937941..c70def2 100644--- a/apps/OpenComputerUseWindows/main_test.go+++ b/apps/OpenComputerUseWindows/main_test.go@@ -18,7 +18,7 @@ func TestClickMethodSchemaAndParser(t *testing.T) {properties := tool.InputSchema["properties"].(map[string]any)method := properties["click_method"].(map[string]any)values := method["enum"].([]string)- if strings.Join(values, ",") != "auto,accessibility,app_post,global" {+ if strings.Join(values, ",") != "auto,accessibility,app_post,sky_click,global" {t.Fatalf("click_method enum = %#v", values)}@@ -27,6 +27,7 @@ func TestClickMethodSchemaAndParser(t *testing.T) {" AUTO ": "auto","Accessibility": "accessibility","app_post": "app_post",+ "SKY_CLICK": "sky_click","GLOBAL": "global",} {got, err := parseClickMethod(input)@@ -39,7 +40,7 @@ func TestClickMethodSchemaAndParser(t *testing.T) {}for _, input := range []string{"physical", "targeted"} {- if _, err := parseClickMethod(input); err == nil || !strings.Contains(err.Error(), "Expected one of: auto, accessibility, app_post, global") {+ if _, err := parseClickMethod(input); err == nil || !strings.Contains(err.Error(), "Expected one of: auto, accessibility, app_post, sky_click, global") {t.Fatalf("parseClickMethod(%s) error = %v", input, err)}}@@ -53,6 +54,14 @@ func TestWindowsRejectsUnsupportedGlobalClickBeforeSnapshotLookup(t *testing.T)}}+func TestWindowsRejectsUnsupportedSkyClickBeforeSnapshotLookup(t *testing.T) {+ x, y := 10.0, 20.0+ result := newService().click("Notepad", "", &x, &y, 1, "left", "sky_click")+ if !result.IsError || result.Content[0].Text != "click_method 'sky_click' is not supported on Windows" {+ t.Fatalf("sky_click result = %#v", result)+ }+}+func TestGetAppStateSchemaIncludesTextLimit(t *testing.T) {tool := findToolDefinition(t, "get_app_state")properties := tool.InputSchema["properties"].(map[string]any)diff --git a/apps/OpenComputerUseWindows/runtime.ps1 b/apps/OpenComputerUseWindows/runtime.ps1index 9d95acf..720e9d5 100644--- a/apps/OpenComputerUseWindows/runtime.ps1+++ b/apps/OpenComputerUseWindows/runtime.ps1@@ -934,6 +934,8 @@ try {Send-MouseClick $hwnd $point.x $point.y $operation.mouse_button ([int]$operation.click_count)} elseif ($clickMethod -eq "global") {throw "click_method 'global' is not supported on Windows"+ } elseif ($clickMethod -eq "sky_click") {+ throw "click_method 'sky_click' is not supported on Windows"} elseif ($clickMethod -eq "auto") {$handled = $falseif ($null -ne $element -and $operation.mouse_button -ne "right" -and $operation.mouse_button -ne "middle") {diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex e3005bf..128382b 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -78,9 +78,9 @@- 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略:- `perform_secondary_action` 只执行目标元素已经暴露出来的 AX action;无效 action 返回官方风格的 `... is not a valid secondary action for ...`,fixture 的 `Raise` 路径也不再为了测试去准备全局物理指针输入- `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口- - `click.click_method` 是开源版的可选扩展,支持 `auto`(默认)、`accessibility`、`app_post` 和 `global`。未传参数时继续使用原有自动路由;显式模式不会静默 fallback 到其他实现。`accessibility` 只接受 `element_index`;`app_post` / `global` 可以使用 `element_index` 的计算落点或原始 `x/y` 坐标。+ - `click.click_method` 是开源版的可选扩展,支持 `auto`(默认)、`accessibility`、`app_post`、`sky_click` 和 `global`。未传参数时继续使用原有自动路由;显式模式不会静默 fallback 到其他实现。`accessibility` 只接受 `element_index`;其余 mouse 路径可以使用 `element_index` 的计算落点或原始 `x/y` 坐标。- element-targeted `click` 的 `auto` 左键路径会先试原生列表的 `AXSelectedChildren` 选择,再试 `AXPress` / `AXConfirm` / `AXOpen` 这类真正语义化的激活动作;如果目标本身不可点,还会继续尝试其子孙 AX 元素(例如 Finder sidebar row 下面暴露 `AXOpen` 的 cell)和命中点附近的 AX hit-test 结果,最后进入现有 non-AX 分支:未开启全局指针环境变量时使用 `postToPid` 定向鼠标事件,开启时直接使用全局 HID 事件。`AXRaise` / `kAXMainAttribute` / `kAXFocusedAttribute` 这类 activation-only fallback 只允许窗口级元素使用,避免普通静态文本或容器把“获得焦点”误报成“点击已处理”;`click_count > 1` 也会优先重复可用的 AX action。- - 显式 `click_method=accessibility` 只执行上述 AX 语义动作;显式 `click_method=app_post` 绕过 AX 候选和后代扫描,直接通过 `CGEvent.postToPid` 向目标进程发送 `mouseMoved` / down / up;显式 `click_method=global` 同样绕过 AX,但通过 `.cghidEventTap` 发送系统级指针事件。`global` 还必须设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`,否则在移动 visual cursor 或发送输入前拒绝请求。+ - 显式 `click_method=accessibility` 只执行上述 AX 语义动作;显式 `click_method=app_post` 绕过 AX 候选和后代扫描,直接通过 `CGEvent.postToPid` 向目标进程发送 `mouseMoved` / down / up;显式 `click_method=sky_click` 只在 macOS 接受左键单击/双击,它验证当前 snapshot 的 `CGWindowID` 仍为同一 PID 所有且 on-screen,先通过 `SLPSPostEventRecordTo` 暂时切换 AppKit-active 状态但不 raise / 不改变 WindowServer frontmost,再按 Chromium-compatible recipe 发送 target move、`(-1,-1)` primer 和真实 down/up。每个鼠标事件同时经动态解析的 `SLEventPostToPid` 与公开 `postToPid` 投递,并带 window-local location、PID/window 和 click-group 字段;renderer settle 后再用 focus event record 恢复原 AppKit-active 状态。它不改变 `auto`,任一步失败也不 fallback。显式 `click_method=global` 同样绕过 AX,但通过 `.cghidEventTap` 发送系统级指针事件。`global` 还必须设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`,否则在移动 visual cursor 或发送输入前拒绝请求。- 对 renderer 合成的 summary `text` 行,`click` 不再默认点击父容器中心;这类元素使用左侧安全锚点。对普通 row/container/text 的后代候选,也会过滤右侧紧凑 hover action(例如 Lark / Electron 会话列表里的“完成”勾),避免把主行点击误操作成 side action;Electron/Lark 这类 app 的 WebArea 合成文本会优先寻找紧邻的行级 `AXPress` 祖先并静默执行,避免用物理鼠标点行。浏览器 WebArea(例如 Chrome/GitHub)不走这条 Electron-scoped 行级祖先点击优化,仍保留通用 link/container 点击路径。命中点如果只反查到覆盖整页的 Electron/WebArea 级 AX 元素,不再继续扫描整个大容器的子孙候选,避免一次行点击被远处的可点击元素截走。- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`type_text` 会把文本按 Unicode extended grapheme cluster 聚合成小批量 `keyboardSetUnicodeString` 事件,避免中文标点、emoji / 代理对和组合字符被逐个 UTF-16 code unit 拆开后在 Electron 富文本输入框里乱序或变形。如果当前 focused element 的 `AXValue` 可设置,`type_text` 会优先按可编辑内容追加并写回 `AXValue`,这覆盖 Feishu / Electron 富文本输入框不可靠接收后台键盘事件的场景,并会过滤已知占位提示,避免把 placeholder 拼进草稿;如果当前 focused element 不是可编辑文本目标,`type_text` 会报错要求先 click 文本输入区或使用 `set_value`,不再把无效果的后台键盘投递当成成功;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias@@ -112,7 +112,7 @@- Go runtime 通过 `go:embed` 带上 `runtime.ps1`,执行 tool call 时临时落盘并调用 Windows PowerShell。PowerShell bridge 使用 `System.Windows.Automation` 做 app/window/element discovery、tree rendering、UIA pattern action、ValuePattern set value 和 ScrollPattern scroll;当目标 app 不暴露对应 pattern 时,fallback 到 `PostMessage` / `SendMessage` 形式的 Win32 window message。- Windows runtime 默认只连接已经运行的 app,不会在 `get_app_state` 找不到进程时自动 `Start-Process`,也不会默认允许 `SetFocus` secondary action;这两条前台抢占路径分别需要 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_APP_LAUNCH=1` 和 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_FOCUS_ACTIONS=1` 显式打开。`type_text` 默认优先对可写文本控件的 child HWND 发送 `EM_SETSEL` / `EM_REPLACESEL`,不再默认走可能触发前台激活的 UIA `ValuePattern.SetValue` fallback;需要旧行为时必须设置 `OPEN_COMPUTER_USE_WINDOWS_ALLOW_UIA_TEXT_FALLBACK=1`。UIA pattern 和 Win32 message fallback 本身仍是 best-effort:很多控件可以在后台响应,但 Windows 没有一套对所有 GUI toolkit 都等价于 macOS AX 的后台键鼠输入模型。- 这 9 个 tool 的协议面与 macOS 主线保持一致:`list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`drag`、`type_text`、`press_key`、`set_value`。其中 element-targeted action 会优先复用上一轮 `get_app_state` 的 runtime id / automation metadata,coordinate action 使用 screenshot/window-relative 坐标。-- Windows `click_method=accessibility` 映射到 UI Automation pattern,`app_post` 映射到 HWND `PostMessage`;第一版没有新增会移动系统指针的 `SendInput` 路径,因此 `global` 会明确返回 unsupported。`auto` 仍保持 UIA 优先、window message fallback 的现有行为。+- Windows `click_method=accessibility` 映射到 UI Automation pattern,`app_post` 映射到 HWND `PostMessage`;macOS-only 的 `sky_click` 和没有实现的 `global` 都会在 snapshot lookup 前明确返回 unsupported。`auto` 仍保持 UIA 优先、window message fallback 的现有行为。- Windows UI Automation 需要运行在已登录用户的桌面 session 里。通过 SSH 作为脱离桌面的后台进程运行时,PowerShell 可以启动并返回 JSON,但系统可能不给它暴露顶层窗口;这种情况下 `list_apps` 会是空,`get_app_state` 可能返回 `appNotFound(...)`。- 当前 Windows 侧仍是功能性第一版:没有 visual cursor overlay、没有 installer/onboarding、没有 code signing,也没有独立的 Windows smoke fixture。后续 TODO 记录在 `docs/exec-plans/active/20260422-windows-computer-use-runtime.md`。@@ -125,7 +125,7 @@- Linux runtime 需要运行在已登录桌面用户 session 里。缺少 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 或 display 环境时,Go runtime 会在启动 Python AT-SPI bridge 前尝试从 `/run/user/<uid>` 和常见桌面进程自动发现当前用户的 session bus、display / Wayland 值;纯 SSH tty 如果找不到已登录桌面 session,可以启动二进制,但不能直接 inspect 或操作 GUI session。- `get_app_state` 的 accessibility tree 在 GTK/GNOME app 上可能很深,Linux bridge 使用与 macOS / Windows 一致的 1200 节点、64 层默认 tree budget,并支持显式提高 `max_tree_nodes` / `max_tree_depth`。截图通过 GDK root window best-effort capture;GNOME Wayland 可能返回黑图,bridge 会检测全黑采样并省略 image block。- 这 9 个 tool 的协议面与 macOS / Windows 保持一致:`list_apps`、`get_app_state`、`click`、`perform_secondary_action`、`scroll`、`drag`、`type_text`、`press_key`、`set_value`。其中 element-targeted action 会优先复用上一轮 `get_app_state` 的 runtime path metadata,coordinate action 使用 screenshot/window-relative 坐标。-- Linux `click_method=accessibility` 映射到 AT-SPI action,`global` 映射到 AT-SPI mouse synthesis 并要求全局指针环境变量;AT-SPI 没有等价的进程定向 mouse dispatch,因此 `app_post` 会明确返回 unsupported。`auto` 仍保持 AT-SPI action 优先、mouse synthesis fallback 的现有行为。+- Linux `click_method=accessibility` 映射到 AT-SPI action,`global` 映射到 AT-SPI mouse synthesis 并要求全局指针环境变量;AT-SPI 没有等价的进程定向 mouse dispatch,因此 `app_post` 和 macOS-only 的 `sky_click` 会在 snapshot lookup 前明确返回 unsupported。`auto` 仍保持 AT-SPI action 优先、mouse synthesis fallback 的现有行为。- 当前 Linux 侧仍是功能性第一版:没有 visual cursor overlay、没有 installer/desktop entry,也没有独立 Linux fixture。后续 TODO 记录在 `docs/exec-plans/active/20260422-linux-computer-use-runtime.md`。## 关键边界diff --git a/docs/RELIABILITY.md b/docs/RELIABILITY.mdindex a25df21..7ddedcf 100644--- a/docs/RELIABILITY.md+++ b/docs/RELIABILITY.md@@ -5,6 +5,7 @@- 构建:`swift build`- 单元测试:`swift test`- 端到端 smoke:`./scripts/run-tool-smoke-tests.sh`+- macOS SkyLight 实机回归:`OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests`- Linux runtime:`(cd apps/OpenComputerUseLinux && go test ./...)`、`./scripts/build-open-computer-use-linux.sh --arch arm64`- 本地诊断:- `open-computer-use doctor`@@ -13,6 +14,7 @@## 已知关键依赖- macOS 上必须给 `Open Computer Use.app` 授权 `Accessibility` 与 `Screen Recording`;终端本身不应该再是必需授权对象。+- macOS `click_method=sky_click` 额外依赖 SkyLight / ApplicationServices 私有符号 `SLEventPostToPid`、`SLEventSetIntegerValueField`、`CGEventSetWindowLocation`、`SLPSPostEventRecordTo`、`_SLPSGetFrontProcess` 和 `GetProcessForPID`。运行时会动态探测并 fail closed,但 macOS 更新、签名方式或目标 app 输入策略变化仍可能让后台投递失效。- smoke suite 依赖本地 GUI session,不能把它当成无头环境命令。- 普通 app 的 `get_app_state` 结果依赖 AX tree 和窗口截图,复杂 app 上输出会有差异;Electron/WebView app 的 AX tree 通常很深,当前会压缩空 wrapper 并放宽遍历深度,以优先保留可操作文本、按钮和输入框。- Linux runtime 依赖已登录桌面用户 session;缺少 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 或 display 环境时,会尝试从 `/run/user/<uid>` 和常见桌面进程自动发现当前用户的 session env。纯 SSH tty 如果找不到桌面 session 仍不能直接访问 AT-SPI GUI tree。@@ -23,8 +25,9 @@1. 先跑 `open-computer-use doctor`,确认权限状态;如果缺权限,命令会通过 `.app` app agent 拉起权限 onboarding 窗口,已全部授权则只打印状态并退出。2. 用 `open-computer-use list-apps` 确认目标 app 是否被发现。3. 用 `open-computer-use snapshot <app>` 看是 transport 问题还是 snapshot / action 问题。-4. 如果只想验证仓库基线,直接跑 fixture + smoke,不要先在复杂第三方 app 上排查。-5. 排查 Linux runtime 时,先确认目标命令是否由桌面用户运行,再用 `open-computer-use call list_apps` 和 `open-computer-use snapshot <app>` 区分 session/env 问题与 AT-SPI tree/action 问题。如果是 Codex MCP,重新执行 `open-computer-use install-codex-mcp` 后重启 Codex,确认配置仍是 `open-computer-use mcp`。+4. 如果只有 `sky_click` 失败,先重新执行 `get_app_state`,确认窗口仍为 on-screen、未隐藏/最小化且没有切换 Space;错误里出现 `missing SkyLight symbols` 时不要改用隐式 fallback,应按当前 macOS 版本重新验证私有 SPI。被遮挡的 Chromium 页面仍无效果时,再用受控页面区分 renderer 策略变化与坐标/window-local 映射问题。+5. 如果只想验证仓库基线,直接跑 fixture + smoke,不要先在复杂第三方 app 上排查。+6. 排查 Linux runtime 时,先确认目标命令是否由桌面用户运行,再用 `open-computer-use call list_apps` 和 `open-computer-use snapshot <app>` 区分 session/env 问题与 AT-SPI tree/action 问题。如果是 Codex MCP,重新执行 `open-computer-use install-codex-mcp` 后重启 Codex,确认配置仍是 `open-computer-use mcp`。## 后续补强方向diff --git a/docs/SECURITY.md b/docs/SECURITY.mdindex ed1624f..84f0fb9 100644--- a/docs/SECURITY.md+++ b/docs/SECURITY.md@@ -28,7 +28,9 @@- 本地使用场景共同提供。- `click_method=global` 是显式的系统级指针路径,可能移动真实鼠标、改变前台焦点或命中坐标处的其他窗口。调用参数本身不视为足够授权;macOS 和支持该模式的 Linux runtime 还要求进程环境中设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`。未设置时必须在任何可见 cursor 移动或真实输入事件之前拒绝请求。-- `click_method=app_post` 与 `accessibility` 不允许静默切换到 `global`。这保证调用方选择的非侵入边界在失败时仍然成立。+- `click_method=app_post`、`sky_click` 与 `accessibility` 不允许静默切换到 `global`。这保证调用方选择的非侵入边界在失败时仍然成立。+- `click_method=sky_click` 是显式 macOS 私有 SPI 能力,不进入 `auto`。它不移动系统指针、不改变 WindowServer frontmost app,也不 raise 或切换目标窗口;内部会短暂改变目标与原前台进程的 AppKit-active 状态,并在 renderer settle 后恢复。它仍会向指定 PID/window 注入真实输入语义,因此只允许使用当前 snapshot 的 on-screen、同 PID 窗口,并在窗口身份不匹配、focus record 失败或私有符号缺失时 fail closed。第一版仅支持同一 Space 内的左键单击/双击。+- SkyLight ABI、raw event field 和 Chromium 接收行为都不受 Apple 公共兼容性承诺保护。系统升级后的失败不得触发静默 global fallback;应先重新验证符号和受控目标,再决定是否更新实现。- 下一阶段应优先补:- session 级审批- 更清楚的敏感 app / 系统设置防护策略diff --git a/docs/exec-plans/completed/20260722-add-sky-click.md b/docs/exec-plans/completed/20260722-add-sky-click.mdnew file mode 100644index 0000000..ab24850--- /dev/null+++ b/docs/exec-plans/completed/20260722-add-sky-click.md@@ -0,0 +1,68 @@+# Add macOS SkyLight background click++## 目标++为 `click` 增加显式 `click_method=sky_click`,让 macOS 能够通过 SkyLight 私有 SPI 向当前 snapshot 对应的目标进程和窗口投递 Chromium-compatible 后台左键点击,同时保持真实鼠标、前台 app、窗口层级和既有 `auto` 行为不变。++## 范围++- 包含:+ - macOS SkyLight 符号动态解析、窗口定向事件字段、off-window primer 和真实点击序列。+ - macOS `click_method` 路由、参数校验、窗口身份校验和诊断错误。+ - Windows / Linux 公共枚举同步,以及显式 unsupported 结果。+ - Swift / Go 单元测试、skill usage、架构、安全、可靠性与 history 文档。+- 不包含:+ - 修改 `auto` 路由或让现有 `app_post` 自动升级为 SkyLight。+ - `SLPSSetFrontProcessWithOptions` foreground assist、跨 Space snapshot、隐藏或最小化窗口恢复。+ - Chromium 网页右键、Canvas / Unity / Blender 等只接受全局 HID 的 surface。+ - 发布、打 tag、提交或推送远端。++## 背景++- 相关文档:`docs/ARCHITECTURE.md`、`docs/SECURITY.md`、`docs/RELIABILITY.md`、`skills/open-computer-use/references/usage.md`。+- 相关代码路径:`ComputerUseService.click`、`InputSimulation.clickTargeted`、`AccessibilitySnapshot.AppSnapshot`、三平台 `click_method` parser / schema。+- 已知约束:显式方法不能静默 fallback;目标窗口必须来自当前 snapshot;私有 SPI 必须通过 `dlopen` / `dlsym` 运行时探测;第一版只承诺同一 Space 内仍为 on-screen 的遮挡窗口。++## 风险++- 风险:SkyLight 函数和 raw event field 都是未公开 ABI,可能随 macOS 更新失效或崩溃。+- 缓解方式:集中封装函数签名和字段、缺符号时 fail closed,并在 macOS 14 / 15 / 26 与签名 app 制品上验证。+- 风险:SkyLight 与公开 `postToPid` 双投递可能在某些非 Chromium surface 产生重复动作。+- 缓解方式:`sky_click` 保持显式且不进入 `auto`,用单次计数 fixture 检查重复投递;若出现重复则把内部 post policy 收敛为 SkyLight-only。+- 风险:过期 snapshot 的 window id 可能已被销毁或复用。+- 缓解方式:投递前检查 `CGWindowID` 的 owner pid 与 on-screen 状态,不匹配时要求重新执行 `get_app_state`。+- 风险:primer 坐标或不支持的鼠标类型命中意外目标。+- 缓解方式:primer 同时使用 off-window screen/local 坐标;第一版只接受左键和 1–2 次点击。++## 里程碑++1. 实现 SkyLight SPI、纯事件 recipe 与 macOS 显式路由。+2. 同步三平台协议、测试和使用文档。+3. 运行单元、跨平台、skill、smoke 与本地能力验证,完成 history 后归档计划。++## 验证方式++- 命令:`swift build`、`swift test`。+- 命令:`(cd apps/OpenComputerUseWindows && go test ./...)`。+- 命令:`(cd apps/OpenComputerUseLinux && go test ./...)`。+- 命令:`npm run package:skill`、`./scripts/run-tool-smoke-tests.sh`。+- 命令:`OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests`(隔离 Chrome profile + 本地页面)。+- 手工检查:`tools/list` 三平台均暴露 `sky_click`,Windows / Linux 在 snapshot lookup 前返回 unsupported。+- 观测检查:macOS 缺失 SkyLight capability、错误 window owner、非左键或超出双击范围时不发送事件;成功路径不调用 `.cghidEventTap` 或 app activation。+- 实机检查:Chrome 按钮在被其他窗口完全遮挡时只触发一次,前台 PID、真实 cursor position 和目标窗口 z-order 不变。++## 进度记录++- [x] 完成文章、Cua Driver、yabai 与 OCU 现有输入路径的源码调研。+- [x] 完成 SkyLight SPI 与 macOS `sky_click` 路由。+- [x] 完成跨平台协议、测试和文档。+- [x] 完成自动化验证与受控实机验证。+- [x] 完成 history 并归档 execution plan。++## 决策记录++- 2026-07-22:公共参数使用 `click_method=sky_click`,保持显式 macOS-only;Windows / Linux 返回稳定 unsupported。+- 2026-07-22:第一版不修改 `auto`,也不引入会 raise / 切 Space 的 `SLPSSetFrontProcessWithOptions`。完全遮挡 Chrome 实测证明只用 `SLEventPostToPid` 不足,因此按文章与 yabai pattern 加入可恢复的 `SLPSPostEventRecordTo` AppKit-active 切换。+- 2026-07-22:事件 recipe 以当前 Cua Driver Chromium 路径为基线,包含 window-local 坐标、PID/window 字段、move primer、off-window primer 和 click-group id。+- 2026-07-22:`CGEventSetWindowLocation` 的动态函数指针按 Cua Rust bridge 的 `(CGEventRef, double, double)` 标量 ABI 声明,并用无投递的 runtime test 验证字段写入,避免依赖 Swift `CGPoint` 聚合参数推断。+- 2026-07-22:focus-without-raise begin/restore 各自保留 40ms event-record 间隔,真实 mouse-up 后保留 100ms renderer settle;受控 Chrome 页面验证单次触发、前台 PID、真实 cursor 和 z-order 均不变。diff --git a/docs/histories/20260722-1905-add-sky-click.md b/docs/histories/20260722-1905-add-sky-click.mdnew file mode 100644index 0000000..a50e23e--- /dev/null+++ b/docs/histories/20260722-1905-add-sky-click.md@@ -0,0 +1,45 @@+## [2026-07-22 19:05] | Task: 添加 macOS SkyLight 后台点击++### 🤖 Execution Context+* **Agent ID**: `/root`+* **Base Model**: `GPT-5`+* **Runtime**: `Codex desktop / macOS arm64`++### 📥 User Query+> 阅读 Cua 的 macOS Window Internals 文章及相关开源库,为 OCU 实现 `sky_click` 后台点击方式。++### 🛠 Changes Overview+**Scope:** macOS click runtime、三平台工具协议、测试、skill 文档与第三方归属。++**Key Actions:**+- **SkyLight route**: 新增运行时 `dlopen` / `dlsym` 桥接,使用 `SLPSPostEventRecordTo` focus-without-raise、`SLEventPostToPid`、window-local event fields、off-window Chromium primer 和恢复阶段。+- **Safety boundary**: `sky_click` 保持显式 macOS-only,不进入 `auto`、不 fallback;只接受当前 on-screen 同 PID 窗口上的左键单击/双击,并串行化 AppKit-active 状态切换。+- **Cross-platform contract**: macOS、Windows、Linux 同步 `sky_click` 枚举;Windows / Linux 在 snapshot lookup 前返回稳定 unsupported。+- **Verification**: 增加事件 recipe、activation record、符号/ABI、窗口身份和平台 parser 测试,并加入默认跳过的隔离 Chrome 完全遮挡实机回归。+- **Attribution**: 固定 Cua / yabai 参考 commit,补充 MIT third-party notices 和私有 SPI 兼容性文档。++### 🧠 Design Intent (Why)+*Chrome 在被其他窗口遮挡时会丢弃普通 `CGEvent.postToPid` mouse event。完整后台路径需要先只切 AppKit event-routing active state、保持 WindowServer frontmost 与 z-order 不变,再通过带 Chromium telemetry 和 primer 的 SkyLight 事件投递;结束后恢复当前实际前台进程的 AppKit 状态。该能力风险高且依赖私有 ABI,因此保持显式 opt-in、fail closed,并用受控 Chrome 页面验证不重复触发和无前台副作用。*++### ✅ Verification+- `swift test`:147 tests,0 failures;默认跳过 1 个显式 live test。+- `OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests`:隔离 Chrome 被 Fixture 完全覆盖时触发恰好一次,前台 PID、真实 cursor 与 z-order 不变;最终实现连续通过。+- Windows / Linux:`go test ./...` 通过。+- `npm run package:skill` 通过。+- `./scripts/run-tool-smoke-tests.sh` 通过完整 tool smoke 与 cursor idle smoke。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/`+- `apps/OpenComputerUseWindows/`+- `apps/OpenComputerUseLinux/`+- `skills/open-computer-use/references/usage.md`+- `docs/ARCHITECTURE.md`+- `docs/SECURITY.md`+- `docs/RELIABILITY.md`+- `docs/references/macos-skylight-background-click.md`+- `THIRD_PARTY_NOTICES.md`diff --git a/docs/references/README.md b/docs/references/README.mdindex 4291d10..a9abe54 100644--- a/docs/references/README.md+++ b/docs/references/README.md@@ -21,3 +21,5 @@- 当抓包目录里的 `websocket/` + `local-sessions/` 仍不足以解释本地 tool / MCP 行为时,再补查 Codex 本地 `logs_2.sqlite`。- `codex-computer-use-cli.md`- 仓库内 `scripts/computer-use-cli/` 的用途、使用方法,以及为什么探测官方 bundled `computer-use` 时要优先走 `codex app-server` 代理而不是 direct stdio。+- `macos-skylight-background-click.md`+ - `click_method=sky_click` 的文章与开源实现来源、固定源码版本、Chromium primer 事件序列、未采用范围和 macOS 私有 SPI 兼容性检查。diff --git a/docs/references/macos-skylight-background-click.md b/docs/references/macos-skylight-background-click.mdnew file mode 100644index 0000000..dcee6f1--- /dev/null+++ b/docs/references/macos-skylight-background-click.md@@ -0,0 +1,54 @@+# macOS SkyLight 后台点击参考++## 用途++这份笔记记录 `click_method=sky_click` 所依赖的外部研究、固定源码版本和 OCU 采用的边界。SkyLight 是 macOS 私有 SPI;这里描述的是经过源码交叉验证的兼容实现,不是 Apple 承诺稳定的公开 API。++## 参考来源++- Cua 文章:[Inside macOS Window Internals](https://cua.ai/blog/inside-macos-window-internals)+ - 解释普通 HID 命中测试、PID 定向投递和 SkyLight 投递的差异。+ - 说明 Chromium 后台输入、focus-without-raise 和 AX remote observer 等问题域。+- Cua Driver:[trycua/cua](https://github.com/trycua/cua/tree/b8a0f32a06c75225ba24ebb5ab14f6507fa90d15/libs/cua-driver)+ - 本实现对照 commit `b8a0f32a06c75225ba24ebb5ab14f6507fa90d15`。+ - 事件序列基线来自 `rust/crates/platform-macos/src/input/mouse.rs` 的 `click_at_xy_chromium`。+ - 动态符号和私有函数签名基线来自 `rust/crates/platform-macos/src/input/skylight.rs`。+- yabai:[asmvik/yabai](https://github.com/asmvik/yabai/tree/dd845723416f5fe92af49fad5ebab00369e07edd)+ - 用于交叉检查 SkyLight 动态加载和私有窗口 API 的工程实践;OCU 没有复制 yabai 代码。++## OCU 采用的事件序列++`sky_click` 使用当前 snapshot 的 PID、`CGWindowID`、窗口内坐标和屏幕坐标,按以下顺序投递:++1. 保存当前前台 PSN/window id,向原前台发送 defocus record,再向目标发送 focus record;两次 `SLPSPostEventRecordTo` 只切 AppKit-active 状态,不调用会 raise / 切 Space 的 `SLPSSetFrontProcessWithOptions`。+2. 目标点 `mouseMoved`,gesture phase `2`。+3. `(-1, -1)` 的 off-window primer `mouseDown` / `mouseUp`,phase `1` / `2`。+4. 等待 `100ms`,把真实目标 `mouseDown` / `mouseUp` 以 phase `3` 投递。+5. 双击时等待 `80ms` 后发送第二对事件,并把 click state 从 `1` 递增到 `2`。+6. 等待 renderer 消费异步 mouse-up,再向目标发送 defocus、向原前台发送 focus record,恢复 AppKit-active 状态。++每个事件带相同 click-group id,并设置 PID、窗口 id、window-under-pointer 和 window-local location。每一步同时走 `SLEventPostToPid` 与公开 `CGEvent.postToPid`:前者覆盖 Chromium/Catalyst,后者保留 AppKit 兼容性。这是一个固定 dispatch policy,不是失败后的 retry。++## OCU 明确没有采用的部分++- 不调用 `SLPSSetFrontProcessWithOptions`,也不使用 `NSRunningApplication.activate` 操作目标 app。focus-without-raise 只切换并恢复 AppKit event routing state,WindowServer frontmost 与窗口 z-order 必须保持不变。+- 不把 `sky_click` 放进 `auto`,也不从失败的 `sky_click` 回退到 `global`。+- 不支持右键、中键、三击、跨 Space、隐藏或最小化窗口。+- 不承诺 Canvas、Unity、Blender 或其他拒绝 PID 定向事件的 surface 可用。++## 兼容性检查++运行时通过 `dlopen` / `dlsym` 探测以下符号,任一缺失都会在投递前 fail closed:++- `SLEventPostToPid`+- `SLEventSetIntegerValueField`+- `CGEventSetWindowLocation`+- `SLPSPostEventRecordTo`+- `_SLPSGetFrontProcess`+- `GetProcessForPID`++投递前还必须确认 snapshot 的 `CGWindowID` 仍由同一 PID 所有且仍为 on-screen。macOS 更新后应重新验证符号、事件字段、签名 app 制品和被完全遮挡的 Chromium 页面。++## 许可++Cua 代码使用 MIT License。本仓库对衍生事件 recipe 的归属与许可文本见根目录 `THIRD_PARTY_NOTICES.md`。diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex 4c1b659..250371e 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -12,6 +12,7 @@ public enum ClickMethod: String, CaseIterable, Sendable {case autocase accessibilitycase appPost = "app_post"+ case skyClick = "sky_click"case global}@@ -46,6 +47,28 @@ func validateClickMethod(}}+func validateSkyClickArguments(+ method: ClickMethod,+ mouseButton: String,+ clickCount: Int+) throws {+ guard method == .skyClick else {+ return+ }++ guard mouseButton.trimmingCharacters(in: .whitespacesAndNewlines).lowercased() == MouseButtonKind.left.rawValue else {+ throw ComputerUseError.message(+ "click_method 'sky_click' only supports mouse_button 'left'"+ )+ }++ guard (1...2).contains(clickCount) else {+ throw ComputerUseError.message(+ "click_method 'sky_click' supports click_count 1 or 2"+ )+ }+}+struct VisualCursorScreenMapping: Equatable {let screenStateFrame: CGRectlet appKitFrame: CGRect@@ -420,6 +443,11 @@ public final class ComputerUseService {hasElementIndex: elementIndex != nil,environment: ProcessInfo.processInfo.environment)+ try validateSkyClickArguments(+ method: clickMethod,+ mouseButton: mouseButton,+ clickCount: clickCount+ )let snapshot = try currentSnapshot(for: query)let button = MouseButtonKind(rawValue: mouseButton.lowercased()) ?? .left@@ -455,9 +483,10 @@ public final class ComputerUseService {if let elementIndex {let record = try lookupElement(snapshot: snapshot, index: elementIndex)- guard let targetPoint = try globalClickPoint(for: record, snapshot: snapshot) else {+ guard let windowPoint = clickPoint(for: record, snapshot: snapshot) else {throw ComputerUseError.stateUnavailable("element \(elementIndex) has no clickable frame")}+ let targetPoint = try windowPointToGlobalPoint(snapshot: snapshot, point: windowPoint)let cursorTarget = makeVisualCursorTarget(at: targetPoint,targetWindowID: snapshot.targetWindowID,@@ -498,10 +527,11 @@ public final class ComputerUseService {"click_method 'accessibility' could not click element_index=\(elementIndex)")}- case .appPost, .global:+ case .appPost, .skyClick, .global:try performExplicitMouseClick(method: clickMethod,at: targetPoint,+ windowPoint: windowPoint,button: button,clickCount: clickCount,targetDescription: "element_index=\(elementIndex)",@@ -556,10 +586,11 @@ public final class ComputerUseService {}case .accessibility:throw ComputerUseError.message("click_method 'accessibility' requires element_index")- case .appPost, .global:+ case .appPost, .skyClick, .global:try performExplicitMouseClick(method: clickMethod,at: targetPoint,+ windowPoint: point,button: button,clickCount: clickCount,targetDescription: "x=\(Int(screenshotPoint.x)) y=\(Int(screenshotPoint.y))",@@ -1552,12 +1583,8 @@ public final class ComputerUseService {)}- private func globalClickPoint(for record: ElementRecord, snapshot: AppSnapshot) throws -> CGPoint? {- guard let point = clickActionPoints(for: record, snapshot: snapshot).first ?? localCenter(for: record) else {- return nil- }-- return try windowPointToGlobalPoint(snapshot: snapshot, point: point)+ private func clickPoint(for record: ElementRecord, snapshot: AppSnapshot) -> CGPoint? {+ clickActionPoints(for: record, snapshot: snapshot).first ?? localCenter(for: record)}private func screenshotToGlobalPoint(snapshot: AppSnapshot, x: Double, y: Double) throws -> CGPoint {@@ -1789,6 +1816,7 @@ public final class ComputerUseService {private func performExplicitMouseClick(method: ClickMethod,at point: CGPoint,+ windowPoint: CGPoint,button: MouseButtonKind,clickCount: Int,targetDescription: String,@@ -1805,6 +1833,21 @@ public final class ComputerUseService {clickCount: clickCount,pid: snapshot.app.pid)+ case .skyClick:+ guard let windowBounds = snapshot.windowBounds, let windowID = snapshot.targetWindowID else {+ throw ComputerUseError.stateUnavailable(+ "click_method 'sky_click' requires a current on-screen target window. Run get_app_state again."+ )+ }+ debugClickDecision("requested=sky_click executed=skylight_pid_post target=\(targetDescription)")+ try InputSimulation.clickWithSkyLight(+ at: eventPoint,+ windowPoint: windowPoint,+ windowBounds: windowBounds,+ windowID: windowID,+ clickCount: clickCount,+ pid: snapshot.app.pid+ )case .global:guard globalPointerFallbacksEnabled(environment: ProcessInfo.processInfo.environment) else {throw ComputerUseError.message(diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swiftindex b7ba356..fa66f84 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift@@ -79,6 +79,26 @@ enum InputSimulation {}}+ static func clickWithSkyLight(+ at screenPoint: CGPoint,+ windowPoint: CGPoint,+ windowBounds: CGRect,+ windowID: CGWindowID,+ clickCount: Int,+ pid: pid_t+ ) throws {+ try SkyClickDispatcher.click(+ target: SkyClickTarget(+ screenPoint: screenPoint,+ windowPoint: windowPoint,+ windowBounds: windowBounds,+ windowID: windowID,+ pid: pid+ ),+ clickCount: clickCount+ )+ }+static func scrollTargeted(at point: CGPoint, direction: String, pages: Double, pid: pid_t) throws {guard let event = CGEvent(scrollWheelEvent2Source: nil, units: .line, wheelCount: 2, wheel1: wheel1(direction: direction, pages: pages), wheel2: wheel2(direction: direction, pages: pages), wheel3: 0) else {throw ComputerUseError.message("Failed to create scroll event.")diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swiftnew file mode 100644index 0000000..054447f--- /dev/null+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift@@ -0,0 +1,301 @@+import AppKit+import CoreGraphics+import Foundation++struct SkyClickTarget {+ let screenPoint: CGPoint+ let windowPoint: CGPoint+ let windowBounds: CGRect+ let windowID: CGWindowID+ let pid: pid_t+}++enum SkyClickEventKind: Equatable, Sendable {+ case moved+ case down+ case up++ var cgEventType: CGEventType {+ switch self {+ case .moved:+ return .mouseMoved+ case .down:+ return .leftMouseDown+ case .up:+ return .leftMouseUp+ }+ }+}++enum SkyClickPointKind: Equatable, Sendable {+ case target+ case primer+}++struct SkyClickEventStep: Equatable, Sendable {+ let kind: SkyClickEventKind+ let pointKind: SkyClickPointKind+ let clickState: Int64+ let phase: Int64+ let delayAfter: TimeInterval+}++func skyClickEventRecipe(clickCount: Int) throws -> [SkyClickEventStep] {+ guard (1...2).contains(clickCount) else {+ throw ComputerUseError.message(+ "click_method 'sky_click' supports click_count 1 or 2"+ )+ }++ var steps = [+ SkyClickEventStep(kind: .moved, pointKind: .target, clickState: 0, phase: 2, delayAfter: 0.015),+ SkyClickEventStep(kind: .down, pointKind: .primer, clickState: 1, phase: 1, delayAfter: 0.001),+ SkyClickEventStep(kind: .up, pointKind: .primer, clickState: 1, phase: 2, delayAfter: 0.100),+ ]++ for pairIndex in 1...clickCount {+ steps.append(+ SkyClickEventStep(+ kind: .down,+ pointKind: .target,+ clickState: Int64(pairIndex),+ phase: 3,+ delayAfter: 0.001+ )+ )+ steps.append(+ SkyClickEventStep(+ kind: .up,+ pointKind: .target,+ clickState: Int64(pairIndex),+ phase: 3,+ delayAfter: pairIndex < clickCount ? 0.080 : 0+ )+ )+ }++ return steps+}++func skyClickWindowMatchesTarget(+ windowInfo: [[String: Any]],+ windowID: CGWindowID,+ pid: pid_t+) -> Bool {+ windowInfo.contains { info in+ guard+ let number = info[kCGWindowNumber as String] as? NSNumber,+ number.uint32Value == windowID,+ let ownerPID = info[kCGWindowOwnerPID as String] as? NSNumber,+ ownerPID.int32Value == pid,+ let onScreen = info[kCGWindowIsOnscreen as String] as? NSNumber+ else {+ return false+ }++ return onScreen.boolValue+ }+}++enum SkyClickDispatcher {+ private static let primerScreenPoint = CGPoint(x: -1, y: -1)+ private static let primerWindowPoint = CGPoint(x: -1, y: -1)+ private static let dispatchLock = NSLock()++ // Raw private event fields used by the Chromium-compatible Cua Driver path.+ private enum EventField {+ static let gesturePhase: UInt32 = 0+ static let clickState: UInt32 = 1+ static let buttonNumber: UInt32 = 3+ static let subtype: UInt32 = 7+ static let targetPID: UInt32 = 40+ static let windowNumber: UInt32 = 51+ static let clickGroupID: UInt32 = 58+ static let windowUnderPointer: UInt32 = 91+ static let handlingWindowUnderPointer: UInt32 = 92+ }++ static func click(+ target: SkyClickTarget,+ clickCount: Int,+ spi: SkyLightSPI = .shared+ ) throws {+ guard spi.capability.isAvailable else {+ throw ComputerUseError.message(+ "click_method 'sky_click' is unavailable: \(spi.capability.unavailableReason)"+ )+ }++ dispatchLock.lock()+ defer {+ dispatchLock.unlock()+ }++ try validate(target: target)+ let recipe = try skyClickEventRecipe(clickCount: clickCount)+ guard let source = CGEventSource(stateID: .hidSystemState) else {+ throw ComputerUseError.message("Failed to create SkyLight HID event source.")+ }++ // Cua uses the nanosecond component of wall-clock time here, which is+ // always below one billion. Keep the same narrow raw-field range;+ // WindowServer does not publish field 58's accepted width.+ let clickGroupID = Int64(DispatchTime.now().uptimeNanoseconds % 1_000_000_000)++ let frontmostPID = NSWorkspace.shared.frontmostApplication?.processIdentifier+ let focusContext: SkyLightFocusContext?+ if frontmostPID == target.pid {+ focusContext = nil+ } else {+ focusContext = try spi.beginFocusWithoutRaise(+ targetPID: target.pid,+ targetWindowID: target.windowID,+ previousWindowID: frontmostWindowID(for: frontmostPID) ?? 0+ )+ }++ do {+ for step in recipe {+ let screenPoint = step.pointKind == .target ? target.screenPoint : primerScreenPoint+ let windowPoint = step.pointKind == .target ? target.windowPoint : primerWindowPoint+ guard let event = CGEvent(+ mouseEventSource: source,+ mouseType: step.kind.cgEventType,+ mouseCursorPosition: screenPoint,+ mouseButton: .left+ ) else {+ throw ComputerUseError.message(+ "Failed to create sky_click mouse event \(step.kind.cgEventType.rawValue)."+ )+ }++ try stamp(+ event,+ target: target,+ windowPoint: windowPoint,+ clickState: step.clickState,+ phase: step.phase,+ clickGroupID: clickGroupID,+ spi: spi+ )++ // The current Cua Driver Chromium recipe deliberately posts through+ // both channels: SkyLight reaches Chromium/Catalyst while the public+ // path preserves AppKit compatibility. This is one dispatch policy,+ // not a retry after an observed failure.+ try spi.postToPid(event, pid: target.pid)+ event.postToPid(target.pid)++ if step.delayAfter > 0 {+ Thread.sleep(forTimeInterval: step.delayAfter)+ }+ }+ } catch {+ if let focusContext {+ try? restoreFocus(context: focusContext, spi: spi)+ }+ throw error+ }++ if let focusContext {+ // SkyLight delivery is asynchronous. Keep the target's AppKit+ // active state long enough for Chromium's renderer hop to consume+ // the final mouse-up before restoring the user's active app.+ Thread.sleep(forTimeInterval: 0.100)+ try restoreFocus(context: focusContext, spi: spi)+ }+ }++ private static func validate(target: SkyClickTarget) throws {+ guard+ target.screenPoint.x.isFinite,+ target.screenPoint.y.isFinite,+ target.windowPoint.x.isFinite,+ target.windowPoint.y.isFinite,+ target.windowBounds.width > 0,+ target.windowBounds.height > 0+ else {+ throw ComputerUseError.message("sky_click requires finite target coordinates and window bounds")+ }++ let localBounds = CGRect(origin: .zero, size: target.windowBounds.size)+ guard localBounds.contains(target.windowPoint) else {+ throw ComputerUseError.message("sky_click target is outside the snapshot window bounds")+ }++ let windowInfo = CGWindowListCopyWindowInfo(+ [.optionIncludingWindow],+ target.windowID+ ) as? [[String: Any]] ?? []+ guard skyClickWindowMatchesTarget(+ windowInfo: windowInfo,+ windowID: target.windowID,+ pid: target.pid+ ) else {+ throw ComputerUseError.stateUnavailable(+ "sky_click target window is stale, off-screen, or no longer owned by the target app. Run get_app_state again."+ )+ }+ }++ private static func stamp(+ _ event: CGEvent,+ target: SkyClickTarget,+ windowPoint: CGPoint,+ clickState: Int64,+ phase: Int64,+ clickGroupID: Int64,+ spi: SkyLightSPI+ ) throws {+ let windowID = Int64(target.windowID)+ try spi.setIntegerField(event, field: EventField.gesturePhase, value: phase)+ try spi.setIntegerField(event, field: EventField.clickState, value: clickState)+ try spi.setIntegerField(event, field: EventField.buttonNumber, value: 0)+ try spi.setIntegerField(event, field: EventField.subtype, value: 3)+ try spi.setIntegerField(event, field: EventField.targetPID, value: Int64(target.pid))+ try spi.setIntegerField(event, field: EventField.windowNumber, value: windowID)+ try spi.setIntegerField(event, field: EventField.clickGroupID, value: clickGroupID)+ try spi.setIntegerField(event, field: EventField.windowUnderPointer, value: windowID)+ try spi.setIntegerField(event, field: EventField.handlingWindowUnderPointer, value: windowID)+ try spi.setWindowLocation(event, point: windowPoint)+ }++ private static func frontmostWindowID(for pid: pid_t?) -> CGWindowID? {+ guard let pid else {+ return nil+ }++ let windowInfo = CGWindowListCopyWindowInfo(+ [.optionOnScreenOnly, .excludeDesktopElements],+ kCGNullWindowID+ ) as? [[String: Any]] ?? []+ return windowInfo.first { info in+ guard+ let ownerPID = info[kCGWindowOwnerPID as String] as? NSNumber,+ ownerPID.int32Value == pid,+ let layer = info[kCGWindowLayer as String] as? NSNumber+ else {+ return false+ }+ return layer.intValue == 0+ }.flatMap { info in+ (info[kCGWindowNumber as String] as? NSNumber)?.uint32Value+ }+ }++ private static func restoreFocus(+ context: SkyLightFocusContext,+ spi: SkyLightSPI+ ) throws {+ let currentFrontPID = NSWorkspace.shared.frontmostApplication?.processIdentifier+ let currentFrontWindowID: CGWindowID? = if currentFrontPID == nil {+ nil+ } else {+ frontmostWindowID(for: currentFrontPID) ?? 0+ }+ try spi.restoreFocusWithoutRaise(+ context,+ currentFrontWindowID: currentFrontWindowID+ )+ }+}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swiftnew file mode 100644index 0000000..0db056b--- /dev/null+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift@@ -0,0 +1,270 @@+import CoreGraphics+import Darwin+import Foundation++struct SkyLightFocusContext {+ let previousPSN: [UInt8]+ let targetPSN: [UInt8]+ let previousWindowID: CGWindowID+ let targetWindowID: CGWindowID+}++func skyLightActivationRecord(windowID: CGWindowID, focused: Bool) -> [UInt8] {+ var record = [UInt8](repeating: 0, count: 0xF8)+ record[0x04] = 0xF8+ record[0x08] = 0x0D+ record[0x3C] = UInt8(truncatingIfNeeded: windowID)+ record[0x3D] = UInt8(truncatingIfNeeded: windowID >> 8)+ record[0x3E] = UInt8(truncatingIfNeeded: windowID >> 16)+ record[0x3F] = UInt8(truncatingIfNeeded: windowID >> 24)+ record[0x8A] = focused ? 0x01 : 0x02+ return record+}++struct SkyLightSPICapability: Equatable, Sendable {+ let missingSymbols: [String]++ var isAvailable: Bool {+ missingSymbols.isEmpty+ }++ var unavailableReason: String {+ if missingSymbols.isEmpty {+ return "available"+ }++ return "missing private click symbols: \(missingSymbols.joined(separator: ", "))"+ }+}++/// Runtime-only bridge for the private SkyLight functions used by `sky_click`.+///+/// The declarations and event-field recipe are derived from the MIT-licensed+/// Cua Driver and yabai implementations. Keep all undocumented ABI in this+/// file so a future macOS compatibility change has one review boundary.+final class SkyLightSPI: @unchecked Sendable {+ static let shared = SkyLightSPI()++ private static let frameworkPath = "/System/Library/PrivateFrameworks/SkyLight.framework/SkyLight"+ private static let postToPidSymbol = "SLEventPostToPid"+ private static let setIntegerFieldSymbol = "SLEventSetIntegerValueField"+ private static let setWindowLocationSymbol = "CGEventSetWindowLocation"+ private static let postEventRecordSymbol = "SLPSPostEventRecordTo"+ private static let getFrontProcessSymbol = "_SLPSGetFrontProcess"+ private static let getProcessForPIDSymbol = "GetProcessForPID"+ private static let applicationServicesPath = "/System/Library/Frameworks/ApplicationServices.framework/ApplicationServices"++ private typealias PostToPidFunction = @convention(c) (pid_t, UnsafeMutableRawPointer?) -> Void+ private typealias SetIntegerFieldFunction = @convention(c) (UnsafeMutableRawPointer?, UInt32, Int64) -> Void+ // Cua's current Rust bridge models this private ABI as+ // (CGEventRef, double x, double y). Keeping the scalar form here avoids+ // relying on Swift's aggregate CGPoint calling convention.+ private typealias SetWindowLocationFunction = @convention(c) (UnsafeMutableRawPointer?, Double, Double) -> Void+ private typealias PostEventRecordFunction = @convention(c) (UnsafeRawPointer?, UnsafePointer<UInt8>?) -> Int32+ private typealias GetFrontProcessFunction = @convention(c) (UnsafeMutableRawPointer?) -> Int32+ private typealias GetProcessForPIDFunction = @convention(c) (pid_t, UnsafeMutableRawPointer?) -> Int32++ private let frameworkHandle: UnsafeMutableRawPointer?+ private let applicationServicesHandle: UnsafeMutableRawPointer?+ private let postToPidFunction: PostToPidFunction?+ private let setIntegerFieldFunction: SetIntegerFieldFunction?+ private let setWindowLocationFunction: SetWindowLocationFunction?+ private let postEventRecordFunction: PostEventRecordFunction?+ private let getFrontProcessFunction: GetFrontProcessFunction?+ private let getProcessForPIDFunction: GetProcessForPIDFunction?++ let capability: SkyLightSPICapability++ private init() {+ let handle = dlopen(Self.frameworkPath, RTLD_LAZY | RTLD_GLOBAL)+ let appServicesHandle = dlopen(Self.applicationServicesPath, RTLD_LAZY | RTLD_GLOBAL)+ frameworkHandle = handle+ applicationServicesHandle = appServicesHandle+ postToPidFunction = Self.resolve(handle: handle, symbol: Self.postToPidSymbol)+ setIntegerFieldFunction = Self.resolve(handle: handle, symbol: Self.setIntegerFieldSymbol)+ setWindowLocationFunction = Self.resolve(handle: handle, symbol: Self.setWindowLocationSymbol)+ postEventRecordFunction = Self.resolve(handle: handle, symbol: Self.postEventRecordSymbol)+ getFrontProcessFunction = Self.resolve(handle: handle, symbol: Self.getFrontProcessSymbol)+ getProcessForPIDFunction = Self.resolve(handle: appServicesHandle, symbol: Self.getProcessForPIDSymbol)++ var missingSymbols: [String] = []+ if postToPidFunction == nil {+ missingSymbols.append(Self.postToPidSymbol)+ }+ if setIntegerFieldFunction == nil {+ missingSymbols.append(Self.setIntegerFieldSymbol)+ }+ if setWindowLocationFunction == nil {+ missingSymbols.append(Self.setWindowLocationSymbol)+ }+ if postEventRecordFunction == nil {+ missingSymbols.append(Self.postEventRecordSymbol)+ }+ if getFrontProcessFunction == nil {+ missingSymbols.append(Self.getFrontProcessSymbol)+ }+ if getProcessForPIDFunction == nil {+ missingSymbols.append(Self.getProcessForPIDSymbol)+ }+ capability = SkyLightSPICapability(missingSymbols: missingSymbols)+ }++ func postToPid(_ event: CGEvent, pid: pid_t) throws {+ guard let postToPidFunction else {+ throw unavailableError()+ }++ postToPidFunction(pid, opaquePointer(for: event))+ }++ func setIntegerField(_ event: CGEvent, field: UInt32, value: Int64) throws {+ guard let setIntegerFieldFunction else {+ throw unavailableError()+ }++ setIntegerFieldFunction(opaquePointer(for: event), field, value)+ }++ func setWindowLocation(_ event: CGEvent, point: CGPoint) throws {+ guard let setWindowLocationFunction else {+ throw unavailableError()+ }++ setWindowLocationFunction(opaquePointer(for: event), point.x, point.y)+ }++ func beginFocusWithoutRaise(+ targetPID: pid_t,+ targetWindowID: CGWindowID,+ previousWindowID: CGWindowID+ ) throws -> SkyLightFocusContext {+ guard+ let getFrontProcessFunction,+ let getProcessForPIDFunction+ else {+ throw unavailableError()+ }++ var previousPSN = [UInt8](repeating: 0, count: 8)+ let previousStatus = previousPSN.withUnsafeMutableBytes { bytes in+ getFrontProcessFunction(bytes.baseAddress)+ }+ guard previousStatus == 0 else {+ throw ComputerUseError.message(+ "click_method 'sky_click' could not resolve the front process PSN (OSStatus \(previousStatus))"+ )+ }++ var targetPSN = [UInt8](repeating: 0, count: 8)+ let targetStatus = targetPSN.withUnsafeMutableBytes { bytes in+ getProcessForPIDFunction(targetPID, bytes.baseAddress)+ }+ guard targetStatus == 0 else {+ throw ComputerUseError.message(+ "click_method 'sky_click' could not resolve target PID \(targetPID) to a PSN (OSStatus \(targetStatus))"+ )+ }++ try postActivationRecord(psn: previousPSN, windowID: previousWindowID, focused: false)+ Thread.sleep(forTimeInterval: 0.040)+ do {+ try postActivationRecord(psn: targetPSN, windowID: targetWindowID, focused: true)+ Thread.sleep(forTimeInterval: 0.040)+ } catch {+ try? postActivationRecord(psn: previousPSN, windowID: previousWindowID, focused: true)+ throw error+ }++ return SkyLightFocusContext(+ previousPSN: previousPSN,+ targetPSN: targetPSN,+ previousWindowID: previousWindowID,+ targetWindowID: targetWindowID+ )+ }++ func restoreFocusWithoutRaise(+ _ context: SkyLightFocusContext,+ currentFrontWindowID: CGWindowID?+ ) throws {+ let restorePSN: [UInt8]+ let restoreWindowID: CGWindowID+ if let getFrontProcessFunction {+ var currentFrontPSN = [UInt8](repeating: 0, count: 8)+ let status = currentFrontPSN.withUnsafeMutableBytes { bytes in+ getFrontProcessFunction(bytes.baseAddress)+ }+ if status == 0 {+ restorePSN = currentFrontPSN+ restoreWindowID = currentFrontWindowID ?? context.previousWindowID+ } else {+ restorePSN = context.previousPSN+ restoreWindowID = context.previousWindowID+ }+ } else {+ restorePSN = context.previousPSN+ restoreWindowID = context.previousWindowID+ }++ var defocusError: Error?+ do {+ try postActivationRecord(+ psn: context.targetPSN,+ windowID: context.targetWindowID,+ focused: false+ )+ } catch {+ defocusError = error+ }++ Thread.sleep(forTimeInterval: 0.040)+ try postActivationRecord(+ psn: restorePSN,+ windowID: restoreWindowID,+ focused: true+ )+ if let defocusError {+ throw defocusError+ }+ }++ private func unavailableError() -> ComputerUseError {+ ComputerUseError.message(+ "click_method 'sky_click' is unavailable: \(capability.unavailableReason)"+ )+ }++ private func opaquePointer(for event: CGEvent) -> UnsafeMutableRawPointer {+ Unmanaged.passUnretained(event).toOpaque()+ }++ private func postActivationRecord(+ psn: [UInt8],+ windowID: CGWindowID,+ focused: Bool+ ) throws {+ guard let postEventRecordFunction else {+ throw unavailableError()+ }++ let record = skyLightActivationRecord(windowID: windowID, focused: focused)+ let status = psn.withUnsafeBytes { psnBytes in+ record.withUnsafeBufferPointer { recordBytes in+ postEventRecordFunction(psnBytes.baseAddress, recordBytes.baseAddress)+ }+ }+ guard status == 0 else {+ throw ComputerUseError.message(+ "click_method 'sky_click' focus-without-raise event failed (OSStatus \(status))"+ )+ }+ }++ private static func resolve<T>(handle: UnsafeMutableRawPointer?, symbol: String) -> T? {+ guard let handle, let pointer = dlsym(handle, symbol) else {+ return nil+ }++ return unsafeBitCast(pointer, to: T.self)+ }+}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swiftindex 4db494c..ce2c8b4 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift@@ -46,7 +46,7 @@ public enum ToolDefinitions {enumValues: ["left", "right", "middle"]),"click_method": stringProperty(- description: "Click implementation: auto (default), accessibility, app_post, or global. Accessibility requires element_index. app_post sends an event directly to the target app or window. Global may move the system pointer and requires OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1.",+ description: "Click implementation: auto (default), accessibility, app_post, sky_click, or global. Accessibility requires element_index. app_post sends a public event directly to the target app. sky_click uses the macOS SkyLight background window path. Global may move the system pointer and requires OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1.",enumValues: ClickMethod.allCases.map(\.rawValue)),],diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 1ec1a09..a91c0ea 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -630,7 +630,7 @@ final class OpenComputerUseKitTests: XCTestCase {)XCTAssertEqual(((tools["click"]?.inputSchema["properties"] as? [String: [String: Any]])?["click_method"]?["enum"] as? [String]) ?? [],- ["auto", "accessibility", "app_post", "global"]+ ["auto", "accessibility", "app_post", "sky_click", "global"])let getAppStateSchema = tools["get_app_state"]?.inputSchemalet getAppStateProperties = getAppStateSchema?["properties"] as? [String: [String: Any]]@@ -1367,6 +1367,7 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(try parseClickMethod(" AUTO "), .auto)XCTAssertEqual(try parseClickMethod("Accessibility"), .accessibility)XCTAssertEqual(try parseClickMethod(" APP_POST "), .appPost)+ XCTAssertEqual(try parseClickMethod(" SKY_CLICK "), .skyClick)XCTAssertEqual(try parseClickMethod("GLOBAL"), .global)}@@ -1375,7 +1376,7 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertThrowsError(try parseClickMethod(value)) { error inXCTAssertEqual((error as? ComputerUseError)?.errorDescription,- "Invalid click_method '\(value)'. Expected one of: auto, accessibility, app_post, global"+ "Invalid click_method '\(value)'. Expected one of: auto, accessibility, app_post, sky_click, global")}}@@ -1415,6 +1416,131 @@ final class OpenComputerUseKitTests: XCTestCase {)}+ func testSkyClickArgumentsRequireLeftButtonAndSingleOrDoubleClick() throws {+ XCTAssertNoThrow(+ try validateSkyClickArguments(method: .skyClick, mouseButton: " LEFT ", clickCount: 1)+ )+ XCTAssertNoThrow(+ try validateSkyClickArguments(method: .skyClick, mouseButton: "left", clickCount: 2)+ )+ XCTAssertNoThrow(+ try validateSkyClickArguments(method: .appPost, mouseButton: "right", clickCount: 5)+ )++ XCTAssertThrowsError(+ try validateSkyClickArguments(method: .skyClick, mouseButton: "right", clickCount: 1)+ ) { error in+ XCTAssertEqual(+ (error as? ComputerUseError)?.errorDescription,+ "click_method 'sky_click' only supports mouse_button 'left'"+ )+ }+ XCTAssertThrowsError(+ try validateSkyClickArguments(method: .skyClick, mouseButton: "left", clickCount: 3)+ ) { error in+ XCTAssertEqual(+ (error as? ComputerUseError)?.errorDescription,+ "click_method 'sky_click' supports click_count 1 or 2"+ )+ }+ }++ func testSkyClickRecipeIncludesMovePrimerAndTargetPairs() throws {+ let single = try skyClickEventRecipe(clickCount: 1)+ XCTAssertEqual(single.count, 5)+ XCTAssertEqual(single.map(\.kind), [.moved, .down, .up, .down, .up])+ XCTAssertEqual(single.map(\.pointKind), [.target, .primer, .primer, .target, .target])+ XCTAssertEqual(single.map(\.phase), [2, 1, 2, 3, 3])+ XCTAssertEqual(single.map(\.clickState), [0, 1, 1, 1, 1])++ let double = try skyClickEventRecipe(clickCount: 2)+ XCTAssertEqual(double.count, 7)+ XCTAssertEqual(double.suffix(4).map(\.clickState), [1, 1, 2, 2])+ XCTAssertEqual(double[4].delayAfter, 0.080, accuracy: 0.000_001)+ XCTAssertEqual(double.last?.delayAfter, 0)+ }++ func testSkyClickRecipeRejectsUnsupportedClickCounts() {+ for count in [0, 3] {+ XCTAssertThrowsError(try skyClickEventRecipe(clickCount: count)) { error in+ XCTAssertEqual(+ (error as? ComputerUseError)?.errorDescription,+ "click_method 'sky_click' supports click_count 1 or 2"+ )+ }+ }+ }++ func testSkyClickWindowValidationRequiresMatchingOnScreenOwner() {+ let matching: [String: Any] = [+ kCGWindowNumber as String: NSNumber(value: UInt32(321)),+ kCGWindowOwnerPID as String: NSNumber(value: Int32(1234)),+ kCGWindowIsOnscreen as String: NSNumber(value: true),+ ]+ XCTAssertTrue(+ skyClickWindowMatchesTarget(windowInfo: [matching], windowID: 321, pid: 1234)+ )+ XCTAssertFalse(+ skyClickWindowMatchesTarget(windowInfo: [matching], windowID: 322, pid: 1234)+ )+ XCTAssertFalse(+ skyClickWindowMatchesTarget(windowInfo: [matching], windowID: 321, pid: 1235)+ )++ var offScreen = matching+ offScreen[kCGWindowIsOnscreen as String] = NSNumber(value: false)+ XCTAssertFalse(+ skyClickWindowMatchesTarget(windowInfo: [offScreen], windowID: 321, pid: 1234)+ )+ }++ func testSkyLightCapabilityReportsMissingSymbols() {+ XCTAssertEqual(+ SkyLightSPICapability(missingSymbols: []).unavailableReason,+ "available"+ )+ XCTAssertEqual(+ SkyLightSPICapability(missingSymbols: ["SLEventPostToPid"]).unavailableReason,+ "missing private click symbols: SLEventPostToPid"+ )+ }++ func testSkyLightActivationRecordEncodesWindowAndFocusState() {+ let focused = skyLightActivationRecord(windowID: 0x1234_5678, focused: true)+ XCTAssertEqual(focused.count, 0xF8)+ XCTAssertEqual(focused[0x04], 0xF8)+ XCTAssertEqual(focused[0x08], 0x0D)+ XCTAssertEqual(Array(focused[0x3C...0x3F]), [0x78, 0x56, 0x34, 0x12])+ XCTAssertEqual(focused[0x8A], 0x01)++ let defocused = skyLightActivationRecord(windowID: 42, focused: false)+ XCTAssertEqual(defocused[0x8A], 0x02)+ }++ func testSkyLightRuntimeSPIProbeCanStampEventWithoutPosting() throws {+ let spi = SkyLightSPI.shared+ guard spi.capability.isAvailable else {+ throw XCTSkip("SkyLight SPI unavailable: \(spi.capability.unavailableReason)")+ }+ guard+ let source = CGEventSource(stateID: .hidSystemState),+ let event = CGEvent(+ mouseEventSource: source,+ mouseType: .mouseMoved,+ mouseCursorPosition: CGPoint(x: 10, y: 20),+ mouseButton: .left+ )+ else {+ return XCTFail("Failed to create a probe CGEvent")+ }++ try spi.setIntegerField(event, field: 1, value: 2)+ XCTAssertEqual(event.getIntegerValueField(.mouseEventClickState), 2)+ XCTAssertNoThrow(+ try spi.setWindowLocation(event, point: CGPoint(x: 3, y: 4))+ )+ }+func testSetValueAttributeGateMatchesOfficialSettableBoundary() throws {XCTAssertTrue(try setValueAttributeIsSettable(result: .success, settable: true, attribute: kAXValueAttribute))XCTAssertFalse(try setValueAttributeIsSettable(result: .success, settable: false, attribute: kAXValueAttribute))diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swiftnew file mode 100644index 0000000..6aa0bdf--- /dev/null+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swift@@ -0,0 +1,302 @@+import AppKit+import CoreGraphics+import Foundation+import XCTest+@testable import OpenComputerUseKit++@MainActor+final class SkyClickLiveTests: XCTestCase {+ private struct WindowRecord {+ let id: CGWindowID+ let pid: pid_t+ let bounds: CGRect+ let name: String+ }++ func testCoveredChromeReceivesExactlyOneSkyClickWithoutForegroundSideEffects() throws {+ guard ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST"] == "1" else {+ throw XCTSkip("Set OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 to run the isolated Chrome live test")+ }+ let spi = SkyLightSPI.shared+ guard spi.capability.isAvailable else {+ throw XCTSkip("SkyLight SPI unavailable: \(spi.capability.unavailableReason)")+ }++ let chromeURL = URL(fileURLWithPath: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome")+ guard FileManager.default.isExecutableFile(atPath: chromeURL.path) else {+ throw XCTSkip("Google Chrome is not installed at the standard path")+ }+ let originalFrontApp = NSWorkspace.shared.frontmostApplication+ defer {+ if let originalFrontApp {+ _ = originalFrontApp.activate(options: [.activateAllWindows])+ }+ }++ let coverExecutable = Self.packageRoot+ .appendingPathComponent(".build/debug/OpenComputerUseFixture")+ guard FileManager.default.isExecutableFile(atPath: coverExecutable.path) else {+ throw XCTSkip("Build OpenComputerUseFixture before running the live test")+ }+ let coverProbe = try launch(executable: coverExecutable)+ defer {+ stop(coverProbe)+ }+ let coverProbeWindow = try waitForWindow(+ pid: coverProbe.processIdentifier,+ nameContaining: "OpenComputerUseFixture"+ )++ let testRoot = Self.packageRoot.appendingPathComponent(".build", isDirectory: true)+ .appendingPathComponent("ocu-sky-click-live-\(UUID().uuidString)", isDirectory: true)+ let profileURL = testRoot.appendingPathComponent("chrome-profile", isDirectory: true)+ let pageURL = testRoot.appendingPathComponent("index.html")+ try FileManager.default.createDirectory(at: profileURL, withIntermediateDirectories: true)+ // XCTest's temporary directory can reject Foundation's atomic+ // replace/remove dance after a GUI child process gets access to the+ // parent. This page is disposable, so a direct write is sufficient.+ try Self.liveTestHTML.write(to: pageURL, atomically: false, encoding: .utf8)+ defer {+ do {+ try FileManager.default.removeItem(at: testRoot)+ } catch {+ print("sky_click live test cleanup warning: \(error)")+ }+ }++ let targetWidth = Int(min(640, coverProbeWindow.bounds.width - 80))+ let targetHeight = Int(min(480, coverProbeWindow.bounds.height - 80))+ let targetX = Int(coverProbeWindow.bounds.minX + 40)+ let targetY = Int(coverProbeWindow.bounds.minY + 40)++ let chrome = Process()+ chrome.executableURL = chromeURL+ chrome.arguments = [+ "--user-data-dir=\(profileURL.path)",+ "--no-first-run",+ "--no-default-browser-check",+ "--disable-background-networking",+ "--disable-component-update",+ "--window-position=\(targetX),\(targetY)",+ "--window-size=\(targetWidth),\(targetHeight)",+ "--app=\(pageURL.absoluteString)",+ ]+ chrome.standardOutput = FileHandle.nullDevice+ chrome.standardError = FileHandle.nullDevice+ try chrome.run()+ defer {+ stop(chrome)+ }++ print("sky_click live test: Chrome launcher pid=\(chrome.processIdentifier)")+ let readyWindow = try waitForWindow(+ pid: chrome.processIdentifier,+ nameContaining: "ocu-sky-click-ready"+ )+ print("sky_click live test: target window=\(readyWindow.id) owner pid=\(readyWindow.pid)")+ stop(coverProbe)+ let cover = try launch(executable: coverExecutable)+ defer {+ stop(cover)+ }+ let coverWindow = try waitForWindow(+ pid: cover.processIdentifier,+ nameContaining: "OpenComputerUseFixture"+ )+ try waitUntil(timeout: 5, failure: "Isolated Chrome remained the frontmost app") {+ guard let frontPID = NSWorkspace.shared.frontmostApplication?.processIdentifier else {+ return false+ }+ return frontPID != readyWindow.pid+ }+ try waitUntil(timeout: 5, failure: "The cover window did not move above isolated Chrome") {+ let ordered = windows()+ guard+ let currentCoverIndex = ordered.firstIndex(where: { $0.id == coverWindow.id }),+ let currentTargetIndex = ordered.firstIndex(where: { $0.id == readyWindow.id })+ else {+ return false+ }+ return currentCoverIndex < currentTargetIndex+ }++ let orderedBefore = windows()+ guard+ let coveredWindow = orderedBefore.first(where: { $0.id == readyWindow.id }),+ let freshCoverWindow = orderedBefore.first(where: { $0.id == coverWindow.id }),+ let coverIndex = orderedBefore.firstIndex(where: { $0.id == coverWindow.id }),+ let targetIndex = orderedBefore.firstIndex(where: { $0.id == readyWindow.id })+ else {+ return XCTFail("Could not re-read the controlled Chrome and cover windows")+ }+ XCTAssertTrue(+ freshCoverWindow.bounds.contains(coveredWindow.bounds),+ "The controlled Chrome window must be fully covered before sky_click"+ )+ XCTAssertLessThan(coverIndex, targetIndex, "The cover window must be above Chrome in z-order")+ print(+ "sky_click live test: cover=\(freshCoverWindow.bounds) target=\(coveredWindow.bounds) "+ + "z-order=\(coverIndex)<\(targetIndex) front-pid="+ + "\(NSWorkspace.shared.frontmostApplication?.processIdentifier ?? 0)"+ )++ let frontPIDBefore = NSWorkspace.shared.frontmostApplication?.processIdentifier+ let cursorBefore = CGEvent(source: nil)?.location+ let screenPoint = CGPoint(x: coveredWindow.bounds.midX, y: coveredWindow.bounds.midY)+ let windowPoint = CGPoint(+ x: screenPoint.x - coveredWindow.bounds.minX,+ y: screenPoint.y - coveredWindow.bounds.minY+ )++ try SkyClickDispatcher.click(+ target: SkyClickTarget(+ screenPoint: screenPoint,+ windowPoint: windowPoint,+ windowBounds: coveredWindow.bounds,+ windowID: coveredWindow.id,+ pid: coveredWindow.pid+ ),+ clickCount: 1,+ spi: spi+ )+ print("sky_click live test: event recipe dispatched")++ let clickedWindow = try waitForWindow(pid: readyWindow.pid, nameContaining: "ocu-sky-click-clicked-")+ RunLoop.current.run(until: Date().addingTimeInterval(0.5))+ let finalWindow = windows().first(where: { $0.id == clickedWindow.id })+ XCTAssertEqual(finalWindow?.name, "ocu-sky-click-clicked-1", "sky_click must trigger exactly one DOM click")+ XCTAssertEqual(+ NSWorkspace.shared.frontmostApplication?.processIdentifier,+ frontPIDBefore,+ "sky_click must not change the frontmost app"+ )++ if let cursorBefore, let cursorAfter = CGEvent(source: nil)?.location {+ XCTAssertLessThan(hypot(cursorAfter.x - cursorBefore.x, cursorAfter.y - cursorBefore.y), 0.5)+ }++ let orderedAfter = windows()+ guard+ let coverIndexAfter = orderedAfter.firstIndex(where: { $0.id == coverWindow.id }),+ let targetIndexAfter = orderedAfter.firstIndex(where: { $0.id == clickedWindow.id })+ else {+ return XCTFail("Could not verify final window z-order")+ }+ XCTAssertLessThan(coverIndexAfter, targetIndexAfter, "sky_click must not raise the Chrome window")+ }++ private func waitForWindow(+ pid: pid_t? = nil,+ nameContaining marker: String,+ timeout: TimeInterval = 10+ ) throws -> WindowRecord {+ var match: WindowRecord?+ do {+ try waitUntil(timeout: timeout, failure: "Timed out waiting for Chrome window title containing \(marker)") {+ match = windows().first(where: { window in+ (pid == nil || window.pid == pid) && window.name.contains(marker)+ })+ return match != nil+ }+ } catch {+ let observed = windows()+ .filter { pid == nil || $0.pid == pid }+ .map { "\($0.id):\($0.name)" }+ .joined(separator: ", ")+ throw ComputerUseError.message(+ "Timed out waiting for Chrome window title containing \(marker); observed [\(observed)]"+ )+ }+ return try XCTUnwrap(match)+ }++ private func waitUntil(+ timeout: TimeInterval,+ failure: String,+ condition: () -> Bool+ ) throws {+ let deadline = Date().addingTimeInterval(timeout)+ while Date() < deadline {+ if condition() {+ return+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.05))+ }+ throw ComputerUseError.message(failure)+ }++ private func stop(_ process: Process) {+ if process.isRunning {+ process.terminate()+ let deadline = Date().addingTimeInterval(5)+ while process.isRunning, Date() < deadline {+ RunLoop.current.run(until: Date().addingTimeInterval(0.05))+ }+ }+ RunLoop.current.run(until: Date().addingTimeInterval(0.25))+ }++ private func launch(executable: URL) throws -> Process {+ let process = Process()+ process.executableURL = executable+ process.standardOutput = FileHandle.nullDevice+ process.standardError = FileHandle.nullDevice+ try process.run()+ return process+ }++ private func windows() -> [WindowRecord] {+ let rawWindows = CGWindowListCopyWindowInfo(+ [.optionOnScreenOnly, .excludeDesktopElements],+ kCGNullWindowID+ ) as? [[String: Any]] ?? []++ return rawWindows.compactMap { info in+ guard+ let number = info[kCGWindowNumber as String] as? NSNumber,+ let ownerPID = info[kCGWindowOwnerPID as String] as? NSNumber,+ let layer = info[kCGWindowLayer as String] as? NSNumber,+ layer.intValue == 0,+ let boundsDictionary = info[kCGWindowBounds as String] as? [String: Any],+ let bounds = CGRect(dictionaryRepresentation: boundsDictionary as CFDictionary),+ bounds.width > 0,+ bounds.height > 0+ else {+ return nil+ }++ return WindowRecord(+ id: number.uint32Value,+ pid: ownerPID.int32Value,+ bounds: bounds,+ name: info[kCGWindowName as String] as? String ?? ""+ )+ }+ }++ private static let liveTestHTML = #"""+ <!doctype html>+ <meta charset="utf-8">+ <title>ocu-sky-click-ready</title>+ <style>+ html, body, button { width: 100%; height: 100%; margin: 0; }+ button { border: 0; font: 24px system-ui; background: #5b8def; color: white; }+ </style>+ <button id="target">sky_click probe</button>+ <script>+ let count = 0;+ document.querySelector('#target').addEventListener('click', () => {+ count += 1;+ document.title = `ocu-sky-click-clicked-${count}`;+ });+ </script>+ """#++ private static let packageRoot: URL = {+ var url = URL(fileURLWithPath: #filePath)+ for _ in 0..<5 {+ url.deleteLastPathComponent()+ }+ return url+ }()+}diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.mdindex 6599e48..fce394b 100644--- a/skills/open-computer-use/references/usage.md+++ b/skills/open-computer-use/references/usage.md@@ -111,6 +111,7 @@ open-computer-use snapshot --max-tree-nodes 3000 --max-tree-depth 96 "Google Chr- `accessibility`: only invoke the element's accessibility action and require `element_index`.- `app_post`: bypass accessibility and post a mouse event directly to the target app/window without moving the system pointer. Supported on macOS and Windows.+- `sky_click`: use the macOS private SkyLight background-window path with focus-without-raise and a Chromium primer click. It supports left single/double click on a current, on-screen window in the same Space and does not move the system pointer, change the frontmost app, or raise the target window. Supported on macOS only.- `global`: bypass accessibility and use the desktop's global pointer path. Supported on macOS and Linux, and requires `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` because it may move the real pointer or change foreground focus.Use `app_post` for an exact blank-area or overlay click that must not be redirected to an accessibility descendant:@@ -119,6 +120,15 @@ Use `app_post` for an exact blank-area or overlay click that must not be redirecopen-computer-use call click --args '{"app":"Google Chrome","x":875,"y":375,"click_method":"app_post"}'```+Use `sky_click` when Chromium ignores `app_post` and the current target window is covered by another window:++```sh+open-computer-use call get_app_state --args '{"app":"Google Chrome"}'+open-computer-use call click --args '{"app":"Google Chrome","x":875,"y":375,"click_method":"sky_click"}'+```++Run `get_app_state` again after the target window moves, closes, changes Space, becomes hidden, or is minimized. `sky_click` is an explicit private-SPI mode: unavailable symbols, a stale window id, unsupported button/count, or failed delivery return an error without falling back to another click implementation.+Use `global` only after explicitly enabling the process-level safety gate:```sh@@ -127,13 +137,13 @@ OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 open-computer-use call clickKeep the environment override scoped as narrowly as possible. While it remains enabled, the existing `auto` route may also choose the global pointer path after accessibility cannot handle a click.-Windows returns an unsupported error for `global`; Linux returns an unsupported error for `app_post`. An unsupported or failed explicit method does not fall back to `auto`.+Windows returns an unsupported error for `sky_click` and `global`; Linux returns an unsupported error for `app_post` and `sky_click`. An unsupported or failed explicit method does not fall back to `auto`.## Platform Notes### macOS-The macOS runtime uses Accessibility, ScreenCaptureKit, and app-posted input events. It normally avoids moving the user's real pointer. The visual cursor overlay is part of the Open Computer Use experience and can be disabled by the surrounding runtime only when needed.+The macOS runtime uses Accessibility, ScreenCaptureKit, app-posted input events, and an explicit private-SkyLight `sky_click` route. It normally avoids moving the user's real pointer. The visual cursor overlay is part of the Open Computer Use experience and can be disabled by the surrounding runtime only when needed. Private SkyLight symbols and raw event fields are not API-stable; re-validate `sky_click` after macOS upgrades.### Windows