👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
fix(click): 保持 SkyLight 后台点击的前台焦点
tisfenga61c34b1475f
diff --git a/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift b/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swiftindex 1f02d00..9b9cf61 100644--- a/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift+++ b/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift@@ -88,6 +88,8 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDeleprivate let dragPadView = DragPadView(frame: NSRect(x: 0, y: 0, width: 320, height: 120))private var scrollView: NSScrollView!private var counter = 0+ private var activationLossCount = 0+ private var keyWindowLossCount = 0private weak var observedScrollView: NSScrollView?private var commandObserver: NSObjectProtocol?private let headless: Bool@@ -111,6 +113,15 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele}}+ func applicationDidBecomeActive(_ notification: Notification) {+ updateExportedState()+ }++ func applicationDidResignActive(_ notification: Notification) {+ activationLossCount += 1+ updateExportedState()+ }+private func buildWindow() {window = NSWindow(contentRect: NSRect(x: 160, y: 180, width: 640, height: 620),@@ -230,6 +241,15 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDeleupdateExportedState()}+ func windowDidBecomeKey(_ notification: Notification) {+ updateExportedState()+ }++ func windowDidResignKey(_ notification: Notification) {+ keyWindowLossCount += 1+ updateExportedState()+ }+private func makeScrollView() -> NSScrollView {let documentView = NSStackView()documentView.orientation = .vertical@@ -333,11 +353,12 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele}private func updateExportedState() {- guard let contentView = window.contentView else {+ guard let window, let contentView = window.contentView else {return}let state = FixtureAppState(+ processIdentifier: getpid(),windowTitle: window.title,windowBounds: FixtureRect(rect: windowBoundsInQuartzCoordinates()),focusedIdentifier: focusedIdentifier(),@@ -352,7 +373,11 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDeleelement(identifier: "fixture-scroll-view", index: 7, role: "scroll area", title: nil, value: nil, actions: ["Scroll Up", "Scroll Down"], rect: localRect(for: scrollView, in: contentView)),element(identifier: "fixture-drag-status", index: 8, role: "static text", title: nil, value: dragLabel.stringValue, actions: [], rect: localRect(for: dragLabel, in: contentView)),element(identifier: "fixture-drag-pad", index: 9, role: "group", title: "Drag Pad", value: nil, actions: [], rect: localRect(for: dragPadView, in: contentView)),- ]+ ],+ isActive: NSApp.isActive,+ isKeyWindow: window.isKeyWindow,+ activationLossCount: activationLossCount,+ keyWindowLossCount: keyWindowLossCount)try? FixtureBridge.writeState(state)diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 128382b..9d683c6 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -80,7 +80,7 @@- `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口- `click.click_method` 是开源版的可选扩展,支持 `auto`(默认)、`accessibility`、`app_post`、`sky_click` 和 `global`。未传参数时继续使用原有自动路由;显式模式不会静默 fallback 到其他实现。`accessibility` 只接受 `element_index`;其余 mouse 路径可以使用 `element_index` 的计算落点或原始 `x/y` 坐标。- element-targeted `click` 的 `auto` 左键路径会先试原生列表的 `AXSelectedChildren` 选择,再试 `AXPress` / `AXConfirm` / `AXOpen` 这类真正语义化的激活动作;如果目标本身不可点,还会继续尝试其子孙 AX 元素(例如 Finder sidebar row 下面暴露 `AXOpen` 的 cell)和命中点附近的 AX hit-test 结果,最后进入现有 non-AX 分支:未开启全局指针环境变量时使用 `postToPid` 定向鼠标事件,开启时直接使用全局 HID 事件。`AXRaise` / `kAXMainAttribute` / `kAXFocusedAttribute` 这类 activation-only fallback 只允许窗口级元素使用,避免普通静态文本或容器把“获得焦点”误报成“点击已处理”;`click_count > 1` 也会优先重复可用的 AX action。- - 显式 `click_method=accessibility` 只执行上述 AX 语义动作;显式 `click_method=app_post` 绕过 AX 候选和后代扫描,直接通过 `CGEvent.postToPid` 向目标进程发送 `mouseMoved` / down / up;显式 `click_method=sky_click` 只在 macOS 接受左键单击/双击,它验证当前 snapshot 的 `CGWindowID` 仍为同一 PID 所有且 on-screen,先通过 `SLPSPostEventRecordTo` 暂时切换 AppKit-active 状态但不 raise / 不改变 WindowServer frontmost,再按 Chromium-compatible recipe 发送 target move、`(-1,-1)` primer 和真实 down/up。每个鼠标事件同时经动态解析的 `SLEventPostToPid` 与公开 `postToPid` 投递,并带 window-local location、PID/window 和 click-group 字段;renderer settle 后再用 focus event record 恢复原 AppKit-active 状态。它不改变 `auto`,任一步失败也不 fallback。显式 `click_method=global` 同样绕过 AX,但通过 `.cghidEventTap` 发送系统级指针事件。`global` 还必须设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`,否则在移动 visual cursor 或发送输入前拒绝请求。+ - 显式 `click_method=accessibility` 只执行上述 AX 语义动作;显式 `click_method=app_post` 绕过 AX 候选和后代扫描,直接通过 `CGEvent.postToPid` 向目标进程发送 `mouseMoved` / down / up;显式 `click_method=sky_click` 只在 macOS 接受左键单击/双击,它验证当前 snapshot 的 `CGWindowID` 仍为同一 PID 所有且 on-screen,仅通过 `SLPSPostEventRecordTo` 让目标应用短暂进入 synthetic-active 状态,绝不向真实前台应用发送 defocus record,再按 Chromium-compatible recipe 发送 target move、`(-1,-1)` primer 和真实 down/up。每个鼠标事件同时经动态解析的 `SLEventPostToPid` 与公开 `postToPid` 投递,并带 window-local location、PID/window 和 click-group 字段;renderer settle 后只撤销目标的 synthetic-active 状态。`sky_click` 的 action-result snapshot 刷新也禁止 activate / `AXRaise` 恢复,目标状态瞬时不可读时直接失败。它不改变 `auto`,任一步失败也不 fallback。显式 `click_method=global` 同样绕过 AX,但通过 `.cghidEventTap` 发送系统级指针事件。`global` 还必须设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`,否则在移动 visual cursor 或发送输入前拒绝请求。- 对 renderer 合成的 summary `text` 行,`click` 不再默认点击父容器中心;这类元素使用左侧安全锚点。对普通 row/container/text 的后代候选,也会过滤右侧紧凑 hover action(例如 Lark / Electron 会话列表里的“完成”勾),避免把主行点击误操作成 side action;Electron/Lark 这类 app 的 WebArea 合成文本会优先寻找紧邻的行级 `AXPress` 祖先并静默执行,避免用物理鼠标点行。浏览器 WebArea(例如 Chrome/GitHub)不走这条 Electron-scoped 行级祖先点击优化,仍保留通用 link/container 点击路径。命中点如果只反查到覆盖整页的 Electron/WebArea 级 AX 元素,不再继续扫描整个大容器的子孙候选,避免一次行点击被远处的可点击元素截走。- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`type_text` 会把文本按 Unicode extended grapheme cluster 聚合成小批量 `keyboardSetUnicodeString` 事件,避免中文标点、emoji / 代理对和组合字符被逐个 UTF-16 code unit 拆开后在 Electron 富文本输入框里乱序或变形。如果当前 focused element 的 `AXValue` 可设置,`type_text` 会优先按可编辑内容追加并写回 `AXValue`,这覆盖 Feishu / Electron 富文本输入框不可靠接收后台键盘事件的场景,并会过滤已知占位提示,避免把 placeholder 拼进草稿;如果当前 focused element 不是可编辑文本目标,`type_text` 会报错要求先 click 文本输入区或使用 `set_value`,不再把无效果的后台键盘投递当成成功;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` aliasdiff --git a/docs/RELIABILITY.md b/docs/RELIABILITY.mdindex 7ddedcf..068a2cc 100644--- a/docs/RELIABILITY.md+++ b/docs/RELIABILITY.md@@ -14,7 +14,7 @@## 已知关键依赖- macOS 上必须给 `Open Computer Use.app` 授权 `Accessibility` 与 `Screen Recording`;终端本身不应该再是必需授权对象。-- macOS `click_method=sky_click` 额外依赖 SkyLight / ApplicationServices 私有符号 `SLEventPostToPid`、`SLEventSetIntegerValueField`、`CGEventSetWindowLocation`、`SLPSPostEventRecordTo`、`_SLPSGetFrontProcess` 和 `GetProcessForPID`。运行时会动态探测并 fail closed,但 macOS 更新、签名方式或目标 app 输入策略变化仍可能让后台投递失效。+- macOS `click_method=sky_click` 额外依赖 SkyLight / ApplicationServices 私有符号 `SLEventPostToPid`、`SLEventSetIntegerValueField`、`CGEventSetWindowLocation`、`SLPSPostEventRecordTo` 和 `GetProcessForPID`。运行时会动态探测并 fail closed,但 macOS 更新、签名方式或目标 app 输入策略变化仍可能让后台投递失效。受控实机回归除 DOM、前台 PID、鼠标和 z-order 外,还必须验证前台 AppKit active、key window、first responder 以及 resign/key-loss 计数。- smoke suite 依赖本地 GUI session,不能把它当成无头环境命令。- 普通 app 的 `get_app_state` 结果依赖 AX tree 和窗口截图,复杂 app 上输出会有差异;Electron/WebView app 的 AX tree 通常很深,当前会压缩空 wrapper 并放宽遍历深度,以优先保留可操作文本、按钮和输入框。- Linux runtime 依赖已登录桌面用户 session;缺少 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 或 display 环境时,会尝试从 `/run/user/<uid>` 和常见桌面进程自动发现当前用户的 session env。纯 SSH tty 如果找不到桌面 session 仍不能直接访问 AT-SPI GUI tree。diff --git a/docs/SECURITY.md b/docs/SECURITY.mdindex 84f0fb9..83ba567 100644--- a/docs/SECURITY.md+++ b/docs/SECURITY.md@@ -29,7 +29,7 @@共同提供。- `click_method=global` 是显式的系统级指针路径,可能移动真实鼠标、改变前台焦点或命中坐标处的其他窗口。调用参数本身不视为足够授权;macOS 和支持该模式的 Linux runtime 还要求进程环境中设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`。未设置时必须在任何可见 cursor 移动或真实输入事件之前拒绝请求。- `click_method=app_post`、`sky_click` 与 `accessibility` 不允许静默切换到 `global`。这保证调用方选择的非侵入边界在失败时仍然成立。-- `click_method=sky_click` 是显式 macOS 私有 SPI 能力,不进入 `auto`。它不移动系统指针、不改变 WindowServer frontmost app,也不 raise 或切换目标窗口;内部会短暂改变目标与原前台进程的 AppKit-active 状态,并在 renderer settle 后恢复。它仍会向指定 PID/window 注入真实输入语义,因此只允许使用当前 snapshot 的 on-screen、同 PID 窗口,并在窗口身份不匹配、focus record 失败或私有符号缺失时 fail closed。第一版仅支持同一 Space 内的左键单击/双击。+- `click_method=sky_click` 是显式 macOS 私有 SPI 能力,不进入 `auto`。它不移动系统指针、不改变 WindowServer frontmost app,也不 raise 或切换目标窗口;内部只让目标应用短暂进入 synthetic-active 状态,绝不向真实前台应用发送 defocus record,renderer settle 后也只撤销目标的合成状态。点击后的 action-result snapshot 禁止 activate / `AXRaise` 恢复。它仍会向指定 PID/window 注入真实输入语义,因此只允许使用当前 snapshot 的 on-screen、同 PID 窗口,并在窗口身份不匹配、target-focus record 失败或私有符号缺失时 fail closed。第一版仅支持同一 Space 内的左键单击/双击。- SkyLight ABI、raw event field 和 Chromium 接收行为都不受 Apple 公共兼容性承诺保护。系统升级后的失败不得触发静默 global fallback;应先重新验证符号和受控目标,再决定是否更新实现。- 下一阶段应优先补:- session 级审批diff --git a/docs/exec-plans/completed/20260723-preserve-sky-click-focus.md b/docs/exec-plans/completed/20260723-preserve-sky-click-focus.mdnew file mode 100644index 0000000..9ccec26--- /dev/null+++ b/docs/exec-plans/completed/20260723-preserve-sky-click-focus.md@@ -0,0 +1,61 @@+# Preserve foreground focus during SkyLight background clicks++## 目标++修正 `click_method=sky_click` 会让真实前台应用短暂失活的问题:完全遮挡的目标窗口仍能接收定向点击,同时前台应用、key window、first responder、真实鼠标和窗口层级保持不变。++## 范围++- 包含:+ - 将 SkyLight activation session 从“前台 defocus + 目标 focus + 双向恢复”收敛为只改变目标应用合成状态。+ - 为实机 Chrome 回归增加前台 AppKit active、key-window 和 first-responder 观测。+ - 让 `sky_click` 的 action-result snapshot 刷新禁止 activate / raise 恢复。+ - 同步架构、可靠性、SkyLight 参考资料和 history。+- 不包含:+ - 修改 `auto`、`app_post` 或 `global` 的公共行为。+ - 用 `NSRunningApplication.activate`、`AXRaise` 或全局 HID 补救失败的后台点击。+ - 复刻 Codex 闭源 `SyntheticAppFocusEnforcer` 的完整 event-tap 状态机。+ - 发布、提交或推送。++## 背景++- 相关文档:`docs/ARCHITECTURE.md`、`docs/RELIABILITY.md`、`docs/references/macos-skylight-background-click.md`。+- 相关代码路径:`SkyLightSPI.swift`、`SkyClickSimulation.swift`、`AccessibilitySnapshot.swift`、`ComputerUseService.click`、`SkyClickLiveTests.swift`。+- 已知约束:`SLPSPostEventRecordTo` 是私有 SPI;Chromium renderer 仍需要短暂 synthetic focus 和 off-window primer;显式模式必须 fail closed。++## 风险++- 风险:只向目标应用发送 focus record 后,Chromium 可能仍拒绝完全遮挡窗口的事件。+- 缓解方式:保留现有事件字段、primer、双通道投递和 renderer settle,用隔离 Chrome profile 做实机验证;失败时不恢复到会让前台应用失活的旧实现。+- 风险:snapshot action-result 刷新在目标 AX / CGWindow 瞬时不可见时返回错误。+- 缓解方式:只对 `sky_click` 的点击后刷新禁用恢复,首次显式 `get_app_state` 行为保持不变。+- 风险:只比较最终 AX 状态会漏掉短暂的 resign/key-loss。+- 缓解方式:让前台 fixture 记录 `applicationDidResignActive` 与 `windowDidResignKey` 累计次数,并同时检查 first responder。++## 里程碑++1. 落地 target-only synthetic focus session 和纯逻辑测试。+2. 增加非侵入 action-result refresh 与前台 focus probe。+3. 完成单元、构建、smoke 和 macOS 实机验证,更新文档并归档计划。++## 验证方式++- 命令:`swift build`。+- 命令:`swift test`。+- 命令:`./scripts/run-tool-smoke-tests.sh`。+- 命令:`OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests`。+- 手工检查:用户提供的 Chrome `sky_click` CLI 序列不再让当前应用输入焦点丢失。+- 观测检查:目标 DOM 只点击一次;前台 PID、AppKit active、key window、first responder、鼠标和 z-order 均不变;前台 fixture 的 resign/key-loss 计数不增加。++## 进度记录++- [x] 确认根因是旧实现主动向前台应用发送 `focused=false`。+- [x] 完成 target-only synthetic focus session。+- [x] 完成非侵入 snapshot refresh 和 focus probe。+- [x] 完成验证、文档与 history。++## 决策记录++- 2026-07-23:`without raise` 不再等同于“保持焦点”;新的硬约束是不得向真实前台应用发送 activation record。+- 2026-07-23:如果 target-only synthetic focus 无法满足某个目标框架,`sky_click` 应明确失败,不得静默恢复到前台 defocus 或全局 HID。+- 2026-07-23:隔离 Chrome 在完全遮挡时仍能通过 target-only synthetic focus 触发一次 DOM click;前台 fixture 的 active、key window、first responder 和 loss counters 全部保持不变。diff --git a/docs/histories/2026-07/20260723-1655-preserve-sky-click-focus.md b/docs/histories/2026-07/20260723-1655-preserve-sky-click-focus.mdnew file mode 100644index 0000000..9dcdb70--- /dev/null+++ b/docs/histories/2026-07/20260723-1655-preserve-sky-click-focus.md@@ -0,0 +1,40 @@+## [2026-07-23 16:55] | Task: 修复 sky_click 前台焦点丢失++### 🤖 Execution Context+* **Agent ID**: `/root`+* **Base Model**: `GPT-5`+* **Runtime**: `Codex desktop / macOS arm64`++### 📥 User Query+> 修复 `sky_click` 虽然不置顶目标窗口,但会让当前应用失去输入焦点的问题,并对照 Codex 的行为补强验证。++### 🛠 Changes Overview+**Scope:** macOS SkyLight click runtime、snapshot recovery、fixture focus probe、测试和文档。++**Key Actions:**+- **Target-only synthetic focus**: 移除对真实前台应用的 defocus / restore record,只让目标应用短暂进入 synthetic-active 状态,renderer settle 后只撤销目标状态。+- **Non-invasive refresh**: `sky_click` 的 action-result snapshot 使用 read-only recovery policy,禁止通过 `NSRunningApplication.activate` 或 `AXRaise` 恢复目标窗口。+- **Focus regression**: 扩展 fixture 状态,记录 AppKit active、key window、first responder、`resignActive` 和 `resignKey` 累计次数;隔离 Chrome 实机回归对这些状态建立断言。+- **Compatibility boundary**: 不改变 Chromium primer、PID/window event fields、SkyLight/public 双通道投递、`auto` 默认行为或显式 fail-closed 语义。++### 🧠 Design Intent (Why)+*旧实现把“without raise”误当成“保持焦点”:它没有改变 WindowServer frontmost PID 或 z-order,却主动向真实前台应用发送 `focused=false`,足以触发 AppKit resign/key/first-responder 副作用。新的 session 数据结构不再保存前台 PSN/window,从结构上禁止这类投递;后台目标如果需要兼容状态,只能获得独立的 target-only synthetic focus。*++### ✅ Verification+- `swift build` passes.+- `swift test` passes: 149 OpenComputerUseKit unit tests, 3 StandaloneCursorSupport tests, and 1 intentionally skipped opt-in live test.+- `OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests` passes; isolated fully covered Chrome triggers exactly one DOM click.+- Foreground fixture remains frontmost, active and key; its text-field first responder and resign/key-loss counters stay unchanged.+- Real pointer position and Chrome/cover z-order remain unchanged.+- `./scripts/run-tool-smoke-tests.sh` passes the full 9-tool sequence and visual cursor idle smoke.+- `npm run package:skill` passes.++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift`+- `apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/`+- `docs/` and `skills/open-computer-use/references/usage.md`diff --git a/docs/references/macos-skylight-background-click.md b/docs/references/macos-skylight-background-click.mdindex dcee6f1..a17280a 100644--- a/docs/references/macos-skylight-background-click.md+++ b/docs/references/macos-skylight-background-click.md@@ -20,18 +20,21 @@`sky_click` 使用当前 snapshot 的 PID、`CGWindowID`、窗口内坐标和屏幕坐标,按以下顺序投递:-1. 保存当前前台 PSN/window id,向原前台发送 defocus record,再向目标发送 focus record;两次 `SLPSPostEventRecordTo` 只切 AppKit-active 状态,不调用会 raise / 切 Space 的 `SLPSSetFrontProcessWithOptions`。+1. 用 `GetProcessForPID` 解析目标 PSN,只向目标发送 focus record,让目标短暂进入 synthetic-active 状态;不查询前台 PSN,也绝不向真实前台应用发送 defocus record。2. 目标点 `mouseMoved`,gesture phase `2`。3. `(-1, -1)` 的 off-window primer `mouseDown` / `mouseUp`,phase `1` / `2`。4. 等待 `100ms`,把真实目标 `mouseDown` / `mouseUp` 以 phase `3` 投递。5. 双击时等待 `80ms` 后发送第二对事件,并把 click state 从 `1` 递增到 `2`。-6. 等待 renderer 消费异步 mouse-up,再向目标发送 defocus、向原前台发送 focus record,恢复 AppKit-active 状态。+6. 等待 renderer 消费异步 mouse-up,再只向目标发送 defocus record,撤销本轮 synthetic-active 状态。每个事件带相同 click-group id,并设置 PID、窗口 id、window-under-pointer 和 window-local location。每一步同时走 `SLEventPostToPid` 与公开 `CGEvent.postToPid`:前者覆盖 Chromium/Catalyst,后者保留 AppKit 兼容性。这是一个固定 dispatch policy,不是失败后的 retry。+旧实现曾按 Cua / yabai 的 focus-without-raise pattern,先向真实前台应用发送 defocus,再在结束时发送 focus 恢复。该序列虽然不改变 WindowServer frontmost PID 或 z-order,却会触发 AppKit `resignActive` / `resignKey` 并破坏 first responder。受控 Chrome 验证表明只合成目标 focus 已足够,因此当前实现把“前台应用从未失活”作为硬约束。+## OCU 明确没有采用的部分-- 不调用 `SLPSSetFrontProcessWithOptions`,也不使用 `NSRunningApplication.activate` 操作目标 app。focus-without-raise 只切换并恢复 AppKit event routing state,WindowServer frontmost 与窗口 z-order 必须保持不变。+- 不调用 `SLPSSetFrontProcessWithOptions`,也不使用 `NSRunningApplication.activate` 或 `AXRaise` 操作目标 app。只改变目标应用的 synthetic event-routing 状态,真实前台应用的 AppKit active、key window 和 first responder 必须保持不变。+- `sky_click` 完成后的 action-result snapshot 使用 read-only recovery policy;目标 AX/window 瞬时不可读时返回错误,不允许 snapshot 恢复路径激活或抬升目标。- 不把 `sky_click` 放进 `auto`,也不从失败的 `sky_click` 回退到 `global`。- 不支持右键、中键、三击、跨 Space、隐藏或最小化窗口。- 不承诺 Canvas、Unity、Blender 或其他拒绝 PID 定向事件的 surface 可用。@@ -44,10 +47,9 @@- `SLEventSetIntegerValueField`- `CGEventSetWindowLocation`- `SLPSPostEventRecordTo`-- `_SLPSGetFrontProcess`- `GetProcessForPID`-投递前还必须确认 snapshot 的 `CGWindowID` 仍由同一 PID 所有且仍为 on-screen。macOS 更新后应重新验证符号、事件字段、签名 app 制品和被完全遮挡的 Chromium 页面。+投递前还必须确认 snapshot 的 `CGWindowID` 仍由同一 PID 所有且仍为 on-screen。macOS 更新后应重新验证符号、事件字段、签名 app 制品和被完全遮挡的 Chromium 页面;实机门槛同时包括前台 fixture 的 active/key/first-responder 状态和 transient resign/key-loss 计数。## 许可diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex 597c37e..3caee02 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -37,6 +37,11 @@ enum SnapshotMode {case fixture}+enum SnapshotRecoveryPolicy: Equatable {+ case allowActivation+ case readOnly+}+public struct AccessibilityTreeLimits: Equatable, Sendable {public static let defaultMaxNodeCount = 1200public static let defaultMaxDepth = 64@@ -143,7 +148,8 @@ enum SnapshotBuilder {static func build(for app: RunningAppDescriptor,textLimit: SnapshotTextLimit = .defaults,- treeLimits: AccessibilityTreeLimits = .defaults+ treeLimits: AccessibilityTreeLimits = .defaults,+ recoveryPolicy: SnapshotRecoveryPolicy = .allowActivation) throws -> AppSnapshot {if app.name == FixtureBridge.appName, let fixtureState = try FixtureBridge.readState() {return buildFixtureSnapshot(app: app, state: fixtureState)@@ -159,7 +165,9 @@ enum SnapshotBuilder {let systemWide = AXUIElementCreateSystemWide()var focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)var focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)- if focusedWindow == nil, recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: nil) {+ if focusedWindow == nil,+ recoveryPolicy == .allowActivation,+ recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: nil) {focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)}@@ -172,7 +180,9 @@ enum SnapshotBuilder {var windowTitle = stringValue(of: rootWindow, attribute: kAXTitleAttribute)var windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)- if windowCapture == nil, recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: rootWindow) {+ if windowCapture == nil,+ recoveryPolicy == .allowActivation,+ recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: rootWindow) {focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)if let recoveredWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide) {rootWindow = recoveredWindowdiff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex 250371e..afdce9b 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -16,6 +16,10 @@ public enum ClickMethod: String, CaseIterable, Sendable {case global}+func clickActionSnapshotRecoveryPolicy(for method: ClickMethod) -> SnapshotRecoveryPolicy {+ method == .skyClick ? .readOnly : .allowActivation+}+func parseClickMethod(_ rawValue: String?) throws -> ClickMethod {let normalized = rawValue?.trimmingCharacters(in: .whitespacesAndNewlines)@@ -607,7 +611,13 @@ public final class ComputerUseService {throw ComputerUseError.invalidArguments("click requires either element_index or x/y")}- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return snapshotResult(+ for: try refreshSnapshot(+ for: query,+ recoveryPolicy: clickActionSnapshotRecoveryPolicy(for: clickMethod)+ ),+ style: .actionResult+ )}public func performSecondaryAction(app query: String, elementIndex: String, action: String) throws -> ToolCallResult {@@ -790,10 +800,16 @@ public final class ComputerUseService {private func refreshSnapshot(for query: String,textLimit: SnapshotTextLimit = .defaults,- treeLimits: AccessibilityTreeLimits = .defaults+ treeLimits: AccessibilityTreeLimits = .defaults,+ recoveryPolicy: SnapshotRecoveryPolicy = .allowActivation) throws -> AppSnapshot {let app = try AppDiscovery.resolve(query)- let snapshot = try SnapshotBuilder.build(for: app, textLimit: textLimit, treeLimits: treeLimits)+ let snapshot = try SnapshotBuilder.build(+ for: app,+ textLimit: textLimit,+ treeLimits: treeLimits,+ recoveryPolicy: recoveryPolicy+ )let keys = Set([query.lowercased(),diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swiftindex 14cff5d..d9723f9 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift@@ -40,16 +40,36 @@ public struct FixtureElementState: Codable, Sendable {}public struct FixtureAppState: Codable, Sendable {+ public let processIdentifier: pid_t?public let windowTitle: Stringpublic let windowBounds: FixtureRectpublic let focusedIdentifier: String?public let elements: [FixtureElementState]+ public let isActive: Bool?+ public let isKeyWindow: Bool?+ public let activationLossCount: Int?+ public let keyWindowLossCount: Int?- public init(windowTitle: String, windowBounds: FixtureRect, focusedIdentifier: String?, elements: [FixtureElementState]) {+ public init(+ processIdentifier: pid_t? = nil,+ windowTitle: String,+ windowBounds: FixtureRect,+ focusedIdentifier: String?,+ elements: [FixtureElementState],+ isActive: Bool? = nil,+ isKeyWindow: Bool? = nil,+ activationLossCount: Int? = nil,+ keyWindowLossCount: Int? = nil+ ) {+ self.processIdentifier = processIdentifierself.windowTitle = windowTitleself.windowBounds = windowBoundsself.focusedIdentifier = focusedIdentifierself.elements = elements+ self.isActive = isActive+ self.isKeyWindow = isKeyWindow+ self.activationLossCount = activationLossCount+ self.keyWindowLossCount = keyWindowLossCount}}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swiftindex 054447f..878d357 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift@@ -143,14 +143,13 @@ enum SkyClickDispatcher {let clickGroupID = Int64(DispatchTime.now().uptimeNanoseconds % 1_000_000_000)let frontmostPID = NSWorkspace.shared.frontmostApplication?.processIdentifier- let focusContext: SkyLightFocusContext?+ let focusContext: SkyLightSyntheticFocusContext?if frontmostPID == target.pid {focusContext = nil} else {- focusContext = try spi.beginFocusWithoutRaise(+ focusContext = try spi.beginSyntheticTargetFocus(targetPID: target.pid,- targetWindowID: target.windowID,- previousWindowID: frontmostWindowID(for: frontmostPID) ?? 0+ targetWindowID: target.windowID)}@@ -192,17 +191,17 @@ enum SkyClickDispatcher {}} catch {if let focusContext {- try? restoreFocus(context: focusContext, spi: spi)+ try? spi.endSyntheticTargetFocus(focusContext)}throw error}if let focusContext {// SkyLight delivery is asynchronous. Keep the target's AppKit- // active state long enough for Chromium's renderer hop to consume- // the final mouse-up before restoring the user's active app.+ // synthetic active state long enough for Chromium's renderer hop+ // to consume the final mouse-up before deactivating only the target.Thread.sleep(forTimeInterval: 0.100)- try restoreFocus(context: focusContext, spi: spi)+ try spi.endSyntheticTargetFocus(focusContext)}}@@ -260,42 +259,4 @@ enum SkyClickDispatcher {try spi.setWindowLocation(event, point: windowPoint)}- private static func frontmostWindowID(for pid: pid_t?) -> CGWindowID? {- guard let pid else {- return nil- }-- let windowInfo = CGWindowListCopyWindowInfo(- [.optionOnScreenOnly, .excludeDesktopElements],- kCGNullWindowID- ) as? [[String: Any]] ?? []- return windowInfo.first { info in- guard- let ownerPID = info[kCGWindowOwnerPID as String] as? NSNumber,- ownerPID.int32Value == pid,- let layer = info[kCGWindowLayer as String] as? NSNumber- else {- return false- }- return layer.intValue == 0- }.flatMap { info in- (info[kCGWindowNumber as String] as? NSNumber)?.uint32Value- }- }-- private static func restoreFocus(- context: SkyLightFocusContext,- spi: SkyLightSPI- ) throws {- let currentFrontPID = NSWorkspace.shared.frontmostApplication?.processIdentifier- let currentFrontWindowID: CGWindowID? = if currentFrontPID == nil {- nil- } else {- frontmostWindowID(for: currentFrontPID) ?? 0- }- try spi.restoreFocusWithoutRaise(- context,- currentFrontWindowID: currentFrontWindowID- )- }}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swiftindex 0db056b..4f14930 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift@@ -2,11 +2,37 @@ import CoreGraphicsimport Darwinimport Foundation-struct SkyLightFocusContext {- let previousPSN: [UInt8]- let targetPSN: [UInt8]- let previousWindowID: CGWindowID- let targetWindowID: CGWindowID+struct SkyLightActivationCommand: Equatable {+ let psn: [UInt8]+ let windowID: CGWindowID+ let focused: Bool+}++struct SkyLightSyntheticFocusContext {+ let deactivateTarget: SkyLightActivationCommand+}++struct SkyLightSyntheticFocusPlan: Equatable {+ let activateTarget: SkyLightActivationCommand+ let deactivateTarget: SkyLightActivationCommand+}++func skyLightSyntheticTargetFocusPlan(+ targetPSN: [UInt8],+ targetWindowID: CGWindowID+) -> SkyLightSyntheticFocusPlan {+ SkyLightSyntheticFocusPlan(+ activateTarget: SkyLightActivationCommand(+ psn: targetPSN,+ windowID: targetWindowID,+ focused: true+ ),+ deactivateTarget: SkyLightActivationCommand(+ psn: targetPSN,+ windowID: targetWindowID,+ focused: false+ )+ )}func skyLightActivationRecord(windowID: CGWindowID, focused: Bool) -> [UInt8] {@@ -50,7 +76,6 @@ final class SkyLightSPI: @unchecked Sendable {private static let setIntegerFieldSymbol = "SLEventSetIntegerValueField"private static let setWindowLocationSymbol = "CGEventSetWindowLocation"private static let postEventRecordSymbol = "SLPSPostEventRecordTo"- private static let getFrontProcessSymbol = "_SLPSGetFrontProcess"private static let getProcessForPIDSymbol = "GetProcessForPID"private static let applicationServicesPath = "/System/Library/Frameworks/ApplicationServices.framework/ApplicationServices"@@ -61,7 +86,6 @@ final class SkyLightSPI: @unchecked Sendable {// relying on Swift's aggregate CGPoint calling convention.private typealias SetWindowLocationFunction = @convention(c) (UnsafeMutableRawPointer?, Double, Double) -> Voidprivate typealias PostEventRecordFunction = @convention(c) (UnsafeRawPointer?, UnsafePointer<UInt8>?) -> Int32- private typealias GetFrontProcessFunction = @convention(c) (UnsafeMutableRawPointer?) -> Int32private typealias GetProcessForPIDFunction = @convention(c) (pid_t, UnsafeMutableRawPointer?) -> Int32private let frameworkHandle: UnsafeMutableRawPointer?@@ -70,7 +94,6 @@ final class SkyLightSPI: @unchecked Sendable {private let setIntegerFieldFunction: SetIntegerFieldFunction?private let setWindowLocationFunction: SetWindowLocationFunction?private let postEventRecordFunction: PostEventRecordFunction?- private let getFrontProcessFunction: GetFrontProcessFunction?private let getProcessForPIDFunction: GetProcessForPIDFunction?let capability: SkyLightSPICapability@@ -84,7 +107,6 @@ final class SkyLightSPI: @unchecked Sendable {setIntegerFieldFunction = Self.resolve(handle: handle, symbol: Self.setIntegerFieldSymbol)setWindowLocationFunction = Self.resolve(handle: handle, symbol: Self.setWindowLocationSymbol)postEventRecordFunction = Self.resolve(handle: handle, symbol: Self.postEventRecordSymbol)- getFrontProcessFunction = Self.resolve(handle: handle, symbol: Self.getFrontProcessSymbol)getProcessForPIDFunction = Self.resolve(handle: appServicesHandle, symbol: Self.getProcessForPIDSymbol)var missingSymbols: [String] = []@@ -100,9 +122,6 @@ final class SkyLightSPI: @unchecked Sendable {if postEventRecordFunction == nil {missingSymbols.append(Self.postEventRecordSymbol)}- if getFrontProcessFunction == nil {- missingSymbols.append(Self.getFrontProcessSymbol)- }if getProcessForPIDFunction == nil {missingSymbols.append(Self.getProcessForPIDSymbol)}@@ -133,28 +152,14 @@ final class SkyLightSPI: @unchecked Sendable {setWindowLocationFunction(opaquePointer(for: event), point.x, point.y)}- func beginFocusWithoutRaise(+ func beginSyntheticTargetFocus(targetPID: pid_t,- targetWindowID: CGWindowID,- previousWindowID: CGWindowID- ) throws -> SkyLightFocusContext {- guard- let getFrontProcessFunction,- let getProcessForPIDFunction- else {+ targetWindowID: CGWindowID+ ) throws -> SkyLightSyntheticFocusContext {+ guard let getProcessForPIDFunction else {throw unavailableError()}- var previousPSN = [UInt8](repeating: 0, count: 8)- let previousStatus = previousPSN.withUnsafeMutableBytes { bytes in- getFrontProcessFunction(bytes.baseAddress)- }- guard previousStatus == 0 else {- throw ComputerUseError.message(- "click_method 'sky_click' could not resolve the front process PSN (OSStatus \(previousStatus))"- )- }-var targetPSN = [UInt8](repeating: 0, count: 8)let targetStatus = targetPSN.withUnsafeMutableBytes { bytes ingetProcessForPIDFunction(targetPID, bytes.baseAddress)@@ -165,67 +170,21 @@ final class SkyLightSPI: @unchecked Sendable {)}- try postActivationRecord(psn: previousPSN, windowID: previousWindowID, focused: false)- Thread.sleep(forTimeInterval: 0.040)- do {- try postActivationRecord(psn: targetPSN, windowID: targetWindowID, focused: true)- Thread.sleep(forTimeInterval: 0.040)- } catch {- try? postActivationRecord(psn: previousPSN, windowID: previousWindowID, focused: true)- throw error- }-- return SkyLightFocusContext(- previousPSN: previousPSN,+ let plan = skyLightSyntheticTargetFocusPlan(targetPSN: targetPSN,- previousWindowID: previousWindowID,targetWindowID: targetWindowID)- }-- func restoreFocusWithoutRaise(- _ context: SkyLightFocusContext,- currentFrontWindowID: CGWindowID?- ) throws {- let restorePSN: [UInt8]- let restoreWindowID: CGWindowID- if let getFrontProcessFunction {- var currentFrontPSN = [UInt8](repeating: 0, count: 8)- let status = currentFrontPSN.withUnsafeMutableBytes { bytes in- getFrontProcessFunction(bytes.baseAddress)- }- if status == 0 {- restorePSN = currentFrontPSN- restoreWindowID = currentFrontWindowID ?? context.previousWindowID- } else {- restorePSN = context.previousPSN- restoreWindowID = context.previousWindowID- }- } else {- restorePSN = context.previousPSN- restoreWindowID = context.previousWindowID- }+ try postActivationCommand(plan.activateTarget)+ Thread.sleep(forTimeInterval: 0.040)- var defocusError: Error?- do {- try postActivationRecord(- psn: context.targetPSN,- windowID: context.targetWindowID,- focused: false- )- } catch {- defocusError = error- }+ return SkyLightSyntheticFocusContext(+ deactivateTarget: plan.deactivateTarget+ )+ }+ func endSyntheticTargetFocus(_ context: SkyLightSyntheticFocusContext) throws {+ try postActivationCommand(context.deactivateTarget)Thread.sleep(forTimeInterval: 0.040)- try postActivationRecord(- psn: restorePSN,- windowID: restoreWindowID,- focused: true- )- if let defocusError {- throw defocusError- }}private func unavailableError() -> ComputerUseError {@@ -238,24 +197,23 @@ final class SkyLightSPI: @unchecked Sendable {Unmanaged.passUnretained(event).toOpaque()}- private func postActivationRecord(- psn: [UInt8],- windowID: CGWindowID,- focused: Bool- ) throws {+ private func postActivationCommand(_ command: SkyLightActivationCommand) throws {guard let postEventRecordFunction else {throw unavailableError()}- let record = skyLightActivationRecord(windowID: windowID, focused: focused)- let status = psn.withUnsafeBytes { psnBytes in+ let record = skyLightActivationRecord(+ windowID: command.windowID,+ focused: command.focused+ )+ let status = command.psn.withUnsafeBytes { psnBytes inrecord.withUnsafeBufferPointer { recordBytes inpostEventRecordFunction(psnBytes.baseAddress, recordBytes.baseAddress)}}guard status == 0 else {throw ComputerUseError.message(- "click_method 'sky_click' focus-without-raise event failed (OSStatus \(status))"+ "click_method 'sky_click' synthetic target-focus event failed (OSStatus \(status))")}}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex a91c0ea..8b227b5 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -1371,6 +1371,38 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(try parseClickMethod("GLOBAL"), .global)}+ func testOnlySkyClickUsesReadOnlyActionSnapshotRefresh() {+ XCTAssertEqual(clickActionSnapshotRecoveryPolicy(for: .skyClick), .readOnly)++ for method in ClickMethod.allCases where method != .skyClick {+ XCTAssertEqual(+ clickActionSnapshotRecoveryPolicy(for: method),+ .allowActivation+ )+ }+ }++ func testFixtureStateFocusProbeFieldsRemainBackwardCompatible() throws {+ let legacyPayload = #"""+ {+ "windowTitle": "Legacy Fixture",+ "windowBounds": {"x": 1, "y": 2, "width": 3, "height": 4},+ "focusedIdentifier": null,+ "elements": []+ }+ """#++ let state = try JSONDecoder().decode(+ FixtureAppState.self,+ from: Data(legacyPayload.utf8)+ )+ XCTAssertNil(state.processIdentifier)+ XCTAssertNil(state.isActive)+ XCTAssertNil(state.isKeyWindow)+ XCTAssertNil(state.activationLossCount)+ XCTAssertNil(state.keyWindowLossCount)+ }+func testClickMethodRejectsUnknownValues() {for value in ["physical", "targeted"] {XCTAssertThrowsError(try parseClickMethod(value)) { error in@@ -1517,6 +1549,30 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(defocused[0x8A], 0x02)}+ func testSkyLightSyntheticFocusPlanOnlyAddressesTarget() {+ let targetPSN = [UInt8](repeating: 7, count: 8)+ let plan = skyLightSyntheticTargetFocusPlan(+ targetPSN: targetPSN,+ targetWindowID: 321+ )++ XCTAssertEqual(+ plan,+ SkyLightSyntheticFocusPlan(+ activateTarget: SkyLightActivationCommand(+ psn: targetPSN,+ windowID: 321,+ focused: true+ ),+ deactivateTarget: SkyLightActivationCommand(+ psn: targetPSN,+ windowID: 321,+ focused: false+ )+ )+ )+ }+func testSkyLightRuntimeSPIProbeCanStampEventWithoutPosting() throws {let spi = SkyLightSPI.sharedguard spi.capability.isAvailable else {diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swiftindex 6aa0bdf..1d1acdb 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swift@@ -134,6 +134,17 @@ final class SkyClickLiveTests: XCTestCase {"The controlled Chrome window must be fully covered before sky_click")XCTAssertLessThan(coverIndex, targetIndex, "The cover window must be above Chrome in z-order")+ try FixtureBridge.post(+ FixtureCommand(kind: "click", identifier: "fixture-input")+ )+ let foregroundStateBefore = try waitForFixtureState(+ pid: cover.processIdentifier,+ failure: "The foreground fixture did not keep its text field focused"+ ) { state in+ state.isActive == true+ && state.isKeyWindow == true+ && state.focusedIdentifier == "fixture-input"+ }print("sky_click live test: cover=\(freshCoverWindow.bounds) target=\(coveredWindow.bounds) "+ "z-order=\(coverIndex)<\(targetIndex) front-pid="@@ -170,6 +181,27 @@ final class SkyClickLiveTests: XCTestCase {frontPIDBefore,"sky_click must not change the frontmost app")+ let foregroundStateAfter = try waitForFixtureState(+ pid: cover.processIdentifier,+ failure: "The foreground fixture state was unavailable after sky_click"+ ) { _ in true }+ XCTAssertEqual(foregroundStateAfter.isActive, true, "sky_click must keep the foreground app active")+ XCTAssertEqual(foregroundStateAfter.isKeyWindow, true, "sky_click must keep the foreground window key")+ XCTAssertEqual(+ foregroundStateAfter.focusedIdentifier,+ foregroundStateBefore.focusedIdentifier,+ "sky_click must preserve the foreground first responder"+ )+ XCTAssertEqual(+ foregroundStateAfter.activationLossCount,+ foregroundStateBefore.activationLossCount,+ "sky_click must not transiently resign the foreground app"+ )+ XCTAssertEqual(+ foregroundStateAfter.keyWindowLossCount,+ foregroundStateBefore.keyWindowLossCount,+ "sky_click must not transiently resign the foreground key window"+ )if let cursorBefore, let cursorAfter = CGEvent(source: nil)?.location {XCTAssertLessThan(hypot(cursorAfter.x - cursorBefore.x, cursorAfter.y - cursorBefore.y), 0.5)@@ -225,6 +257,26 @@ final class SkyClickLiveTests: XCTestCase {throw ComputerUseError.message(failure)}+ private func waitForFixtureState(+ pid: pid_t,+ failure: String,+ condition: (FixtureAppState) -> Bool+ ) throws -> FixtureAppState {+ var match: FixtureAppState?+ try waitUntil(timeout: 5, failure: failure) {+ guard+ let state = try? FixtureBridge.readState(),+ state.processIdentifier == pid,+ condition(state)+ else {+ return false+ }+ match = state+ return true+ }+ return try XCTUnwrap(match)+ }+private func stop(_ process: Process) {if process.isRunning {process.terminate()diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.mdindex fce394b..ea1ffb5 100644--- a/skills/open-computer-use/references/usage.md+++ b/skills/open-computer-use/references/usage.md@@ -111,7 +111,7 @@ open-computer-use snapshot --max-tree-nodes 3000 --max-tree-depth 96 "Google Chr- `accessibility`: only invoke the element's accessibility action and require `element_index`.- `app_post`: bypass accessibility and post a mouse event directly to the target app/window without moving the system pointer. Supported on macOS and Windows.-- `sky_click`: use the macOS private SkyLight background-window path with focus-without-raise and a Chromium primer click. It supports left single/double click on a current, on-screen window in the same Space and does not move the system pointer, change the frontmost app, or raise the target window. Supported on macOS only.+- `sky_click`: use the macOS private SkyLight background-window path with target-only synthetic focus and a Chromium primer click. It supports left single/double click on a current, on-screen window in the same Space and does not move the system pointer, deactivate the foreground app, change its key/first-responder state, or raise the target window. Its action-result snapshot refresh is read-only. Supported on macOS only.- `global`: bypass accessibility and use the desktop's global pointer path. Supported on macOS and Linux, and requires `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` because it may move the real pointer or change foreground focus.Use `app_post` for an exact blank-area or overlay click that must not be redirected to an accessibility descendant: