Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

fix(click): 保持 SkyLight 后台点击的前台焦点

tisfenga61c34b1475f
diff --git a/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift b/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift
index 1f02d00..9b9cf61 100644
--- a/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift
+++ b/apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift
@@ -88,6 +88,8 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele
private let dragPadView = DragPadView(frame: NSRect(x: 0, y: 0, width: 320, height: 120))
private var scrollView: NSScrollView!
private var counter = 0
+ private var activationLossCount = 0
+ private var keyWindowLossCount = 0
private weak var observedScrollView: NSScrollView?
private var commandObserver: NSObjectProtocol?
private let headless: Bool
@@ -111,6 +113,15 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele
}
}
+ func applicationDidBecomeActive(_ notification: Notification) {
+ updateExportedState()
+ }
+
+ func applicationDidResignActive(_ notification: Notification) {
+ activationLossCount += 1
+ updateExportedState()
+ }
+
private func buildWindow() {
window = NSWindow(
contentRect: NSRect(x: 160, y: 180, width: 640, height: 620),
@@ -230,6 +241,15 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele
updateExportedState()
}
+ func windowDidBecomeKey(_ notification: Notification) {
+ updateExportedState()
+ }
+
+ func windowDidResignKey(_ notification: Notification) {
+ keyWindowLossCount += 1
+ updateExportedState()
+ }
+
private func makeScrollView() -> NSScrollView {
let documentView = NSStackView()
documentView.orientation = .vertical
@@ -333,11 +353,12 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele
}
private func updateExportedState() {
- guard let contentView = window.contentView else {
+ guard let window, let contentView = window.contentView else {
return
}
let state = FixtureAppState(
+ processIdentifier: getpid(),
windowTitle: window.title,
windowBounds: FixtureRect(rect: windowBoundsInQuartzCoordinates()),
focusedIdentifier: focusedIdentifier(),
@@ -352,7 +373,11 @@ final class FixtureAppDelegate: NSObject, NSApplicationDelegate, NSTextFieldDele
element(identifier: "fixture-scroll-view", index: 7, role: "scroll area", title: nil, value: nil, actions: ["Scroll Up", "Scroll Down"], rect: localRect(for: scrollView, in: contentView)),
element(identifier: "fixture-drag-status", index: 8, role: "static text", title: nil, value: dragLabel.stringValue, actions: [], rect: localRect(for: dragLabel, in: contentView)),
element(identifier: "fixture-drag-pad", index: 9, role: "group", title: "Drag Pad", value: nil, actions: [], rect: localRect(for: dragPadView, in: contentView)),
- ]
+ ],
+ isActive: NSApp.isActive,
+ isKeyWindow: window.isKeyWindow,
+ activationLossCount: activationLossCount,
+ keyWindowLossCount: keyWindowLossCount
)
try? FixtureBridge.writeState(state)
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 128382b..9d683c6 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -80,7 +80,7 @@
- `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口
- `click.click_method` 是开源版的可选扩展,支持 `auto`(默认)、`accessibility`、`app_post`、`sky_click` 和 `global`。未传参数时继续使用原有自动路由;显式模式不会静默 fallback 到其他实现。`accessibility` 只接受 `element_index`;其余 mouse 路径可以使用 `element_index` 的计算落点或原始 `x/y` 坐标。
- element-targeted `click` 的 `auto` 左键路径会先试原生列表的 `AXSelectedChildren` 选择,再试 `AXPress` / `AXConfirm` / `AXOpen` 这类真正语义化的激活动作;如果目标本身不可点,还会继续尝试其子孙 AX 元素(例如 Finder sidebar row 下面暴露 `AXOpen` 的 cell)和命中点附近的 AX hit-test 结果,最后进入现有 non-AX 分支:未开启全局指针环境变量时使用 `postToPid` 定向鼠标事件,开启时直接使用全局 HID 事件。`AXRaise` / `kAXMainAttribute` / `kAXFocusedAttribute` 这类 activation-only fallback 只允许窗口级元素使用,避免普通静态文本或容器把“获得焦点”误报成“点击已处理”;`click_count > 1` 也会优先重复可用的 AX action。
- - 显式 `click_method=accessibility` 只执行上述 AX 语义动作;显式 `click_method=app_post` 绕过 AX 候选和后代扫描,直接通过 `CGEvent.postToPid` 向目标进程发送 `mouseMoved` / down / up;显式 `click_method=sky_click` 只在 macOS 接受左键单击/双击,它验证当前 snapshot 的 `CGWindowID` 仍为同一 PID 所有且 on-screen,先通过 `SLPSPostEventRecordTo` 暂时切换 AppKit-active 状态但不 raise / 不改变 WindowServer frontmost,再按 Chromium-compatible recipe 发送 target move、`(-1,-1)` primer 和真实 down/up。每个鼠标事件同时经动态解析的 `SLEventPostToPid` 与公开 `postToPid` 投递,并带 window-local location、PID/window 和 click-group 字段;renderer settle 后再用 focus event record 恢复原 AppKit-active 状态。它不改变 `auto`,任一步失败也不 fallback。显式 `click_method=global` 同样绕过 AX,但通过 `.cghidEventTap` 发送系统级指针事件。`global` 还必须设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`,否则在移动 visual cursor 或发送输入前拒绝请求。
+ - 显式 `click_method=accessibility` 只执行上述 AX 语义动作;显式 `click_method=app_post` 绕过 AX 候选和后代扫描,直接通过 `CGEvent.postToPid` 向目标进程发送 `mouseMoved` / down / up;显式 `click_method=sky_click` 只在 macOS 接受左键单击/双击,它验证当前 snapshot 的 `CGWindowID` 仍为同一 PID 所有且 on-screen,仅通过 `SLPSPostEventRecordTo` 让目标应用短暂进入 synthetic-active 状态,绝不向真实前台应用发送 defocus record,再按 Chromium-compatible recipe 发送 target move、`(-1,-1)` primer 和真实 down/up。每个鼠标事件同时经动态解析的 `SLEventPostToPid` 与公开 `postToPid` 投递,并带 window-local location、PID/window 和 click-group 字段;renderer settle 后只撤销目标的 synthetic-active 状态。`sky_click` 的 action-result snapshot 刷新也禁止 activate / `AXRaise` 恢复,目标状态瞬时不可读时直接失败。它不改变 `auto`,任一步失败也不 fallback。显式 `click_method=global` 同样绕过 AX,但通过 `.cghidEventTap` 发送系统级指针事件。`global` 还必须设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`,否则在移动 visual cursor 或发送输入前拒绝请求。
- 对 renderer 合成的 summary `text` 行,`click` 不再默认点击父容器中心;这类元素使用左侧安全锚点。对普通 row/container/text 的后代候选,也会过滤右侧紧凑 hover action(例如 Lark / Electron 会话列表里的“完成”勾),避免把主行点击误操作成 side action;Electron/Lark 这类 app 的 WebArea 合成文本会优先寻找紧邻的行级 `AXPress` 祖先并静默执行,避免用物理鼠标点行。浏览器 WebArea(例如 Chrome/GitHub)不走这条 Electron-scoped 行级祖先点击优化,仍保留通用 link/container 点击路径。命中点如果只反查到覆盖整页的 Electron/WebArea 级 AX 元素,不再继续扫描整个大容器的子孙候选,避免一次行点击被远处的可点击元素截走。
- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素
- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`type_text` 会把文本按 Unicode extended grapheme cluster 聚合成小批量 `keyboardSetUnicodeString` 事件,避免中文标点、emoji / 代理对和组合字符被逐个 UTF-16 code unit 拆开后在 Electron 富文本输入框里乱序或变形。如果当前 focused element 的 `AXValue` 可设置,`type_text` 会优先按可编辑内容追加并写回 `AXValue`,这覆盖 Feishu / Electron 富文本输入框不可靠接收后台键盘事件的场景,并会过滤已知占位提示,避免把 placeholder 拼进草稿;如果当前 focused element 不是可编辑文本目标,`type_text` 会报错要求先 click 文本输入区或使用 `set_value`,不再把无效果的后台键盘投递当成成功;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias
diff --git a/docs/RELIABILITY.md b/docs/RELIABILITY.md
index 7ddedcf..068a2cc 100644
--- a/docs/RELIABILITY.md
+++ b/docs/RELIABILITY.md
@@ -14,7 +14,7 @@
## 已知关键依赖
- macOS 上必须给 `Open Computer Use.app` 授权 `Accessibility` 与 `Screen Recording`;终端本身不应该再是必需授权对象。
-- macOS `click_method=sky_click` 额外依赖 SkyLight / ApplicationServices 私有符号 `SLEventPostToPid`、`SLEventSetIntegerValueField`、`CGEventSetWindowLocation`、`SLPSPostEventRecordTo`、`_SLPSGetFrontProcess` 和 `GetProcessForPID`。运行时会动态探测并 fail closed,但 macOS 更新、签名方式或目标 app 输入策略变化仍可能让后台投递失效。
+- macOS `click_method=sky_click` 额外依赖 SkyLight / ApplicationServices 私有符号 `SLEventPostToPid`、`SLEventSetIntegerValueField`、`CGEventSetWindowLocation`、`SLPSPostEventRecordTo` 和 `GetProcessForPID`。运行时会动态探测并 fail closed,但 macOS 更新、签名方式或目标 app 输入策略变化仍可能让后台投递失效。受控实机回归除 DOM、前台 PID、鼠标和 z-order 外,还必须验证前台 AppKit active、key window、first responder 以及 resign/key-loss 计数。
- smoke suite 依赖本地 GUI session,不能把它当成无头环境命令。
- 普通 app 的 `get_app_state` 结果依赖 AX tree 和窗口截图,复杂 app 上输出会有差异;Electron/WebView app 的 AX tree 通常很深,当前会压缩空 wrapper 并放宽遍历深度,以优先保留可操作文本、按钮和输入框。
- Linux runtime 依赖已登录桌面用户 session;缺少 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 或 display 环境时,会尝试从 `/run/user/<uid>` 和常见桌面进程自动发现当前用户的 session env。纯 SSH tty 如果找不到桌面 session 仍不能直接访问 AT-SPI GUI tree。
diff --git a/docs/SECURITY.md b/docs/SECURITY.md
index 84f0fb9..83ba567 100644
--- a/docs/SECURITY.md
+++ b/docs/SECURITY.md
@@ -29,7 +29,7 @@
共同提供。
- `click_method=global` 是显式的系统级指针路径,可能移动真实鼠标、改变前台焦点或命中坐标处的其他窗口。调用参数本身不视为足够授权;macOS 和支持该模式的 Linux runtime 还要求进程环境中设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1`。未设置时必须在任何可见 cursor 移动或真实输入事件之前拒绝请求。
- `click_method=app_post`、`sky_click` 与 `accessibility` 不允许静默切换到 `global`。这保证调用方选择的非侵入边界在失败时仍然成立。
-- `click_method=sky_click` 是显式 macOS 私有 SPI 能力,不进入 `auto`。它不移动系统指针、不改变 WindowServer frontmost app,也不 raise 或切换目标窗口;内部会短暂改变目标与原前台进程的 AppKit-active 状态,并在 renderer settle 后恢复。它仍会向指定 PID/window 注入真实输入语义,因此只允许使用当前 snapshot 的 on-screen、同 PID 窗口,并在窗口身份不匹配、focus record 失败或私有符号缺失时 fail closed。第一版仅支持同一 Space 内的左键单击/双击。
+- `click_method=sky_click` 是显式 macOS 私有 SPI 能力,不进入 `auto`。它不移动系统指针、不改变 WindowServer frontmost app,也不 raise 或切换目标窗口;内部只让目标应用短暂进入 synthetic-active 状态,绝不向真实前台应用发送 defocus record,renderer settle 后也只撤销目标的合成状态。点击后的 action-result snapshot 禁止 activate / `AXRaise` 恢复。它仍会向指定 PID/window 注入真实输入语义,因此只允许使用当前 snapshot 的 on-screen、同 PID 窗口,并在窗口身份不匹配、target-focus record 失败或私有符号缺失时 fail closed。第一版仅支持同一 Space 内的左键单击/双击。
- SkyLight ABI、raw event field 和 Chromium 接收行为都不受 Apple 公共兼容性承诺保护。系统升级后的失败不得触发静默 global fallback;应先重新验证符号和受控目标,再决定是否更新实现。
- 下一阶段应优先补:
- session 级审批
diff --git a/docs/exec-plans/completed/20260723-preserve-sky-click-focus.md b/docs/exec-plans/completed/20260723-preserve-sky-click-focus.md
new file mode 100644
index 0000000..9ccec26
--- /dev/null
+++ b/docs/exec-plans/completed/20260723-preserve-sky-click-focus.md
@@ -0,0 +1,61 @@
+# Preserve foreground focus during SkyLight background clicks
+
+## 目标
+
+修正 `click_method=sky_click` 会让真实前台应用短暂失活的问题:完全遮挡的目标窗口仍能接收定向点击,同时前台应用、key window、first responder、真实鼠标和窗口层级保持不变。
+
+## 范围
+
+- 包含:
+ - 将 SkyLight activation session 从“前台 defocus + 目标 focus + 双向恢复”收敛为只改变目标应用合成状态。
+ - 为实机 Chrome 回归增加前台 AppKit active、key-window 和 first-responder 观测。
+ - 让 `sky_click` 的 action-result snapshot 刷新禁止 activate / raise 恢复。
+ - 同步架构、可靠性、SkyLight 参考资料和 history。
+- 不包含:
+ - 修改 `auto`、`app_post` 或 `global` 的公共行为。
+ - 用 `NSRunningApplication.activate`、`AXRaise` 或全局 HID 补救失败的后台点击。
+ - 复刻 Codex 闭源 `SyntheticAppFocusEnforcer` 的完整 event-tap 状态机。
+ - 发布、提交或推送。
+
+## 背景
+
+- 相关文档:`docs/ARCHITECTURE.md`、`docs/RELIABILITY.md`、`docs/references/macos-skylight-background-click.md`。
+- 相关代码路径:`SkyLightSPI.swift`、`SkyClickSimulation.swift`、`AccessibilitySnapshot.swift`、`ComputerUseService.click`、`SkyClickLiveTests.swift`。
+- 已知约束:`SLPSPostEventRecordTo` 是私有 SPI;Chromium renderer 仍需要短暂 synthetic focus 和 off-window primer;显式模式必须 fail closed。
+
+## 风险
+
+- 风险:只向目标应用发送 focus record 后,Chromium 可能仍拒绝完全遮挡窗口的事件。
+- 缓解方式:保留现有事件字段、primer、双通道投递和 renderer settle,用隔离 Chrome profile 做实机验证;失败时不恢复到会让前台应用失活的旧实现。
+- 风险:snapshot action-result 刷新在目标 AX / CGWindow 瞬时不可见时返回错误。
+- 缓解方式:只对 `sky_click` 的点击后刷新禁用恢复,首次显式 `get_app_state` 行为保持不变。
+- 风险:只比较最终 AX 状态会漏掉短暂的 resign/key-loss。
+- 缓解方式:让前台 fixture 记录 `applicationDidResignActive` 与 `windowDidResignKey` 累计次数,并同时检查 first responder。
+
+## 里程碑
+
+1. 落地 target-only synthetic focus session 和纯逻辑测试。
+2. 增加非侵入 action-result refresh 与前台 focus probe。
+3. 完成单元、构建、smoke 和 macOS 实机验证,更新文档并归档计划。
+
+## 验证方式
+
+- 命令:`swift build`。
+- 命令:`swift test`。
+- 命令:`./scripts/run-tool-smoke-tests.sh`。
+- 命令:`OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests`。
+- 手工检查:用户提供的 Chrome `sky_click` CLI 序列不再让当前应用输入焦点丢失。
+- 观测检查:目标 DOM 只点击一次;前台 PID、AppKit active、key window、first responder、鼠标和 z-order 均不变;前台 fixture 的 resign/key-loss 计数不增加。
+
+## 进度记录
+
+- [x] 确认根因是旧实现主动向前台应用发送 `focused=false`。
+- [x] 完成 target-only synthetic focus session。
+- [x] 完成非侵入 snapshot refresh 和 focus probe。
+- [x] 完成验证、文档与 history。
+
+## 决策记录
+
+- 2026-07-23:`without raise` 不再等同于“保持焦点”;新的硬约束是不得向真实前台应用发送 activation record。
+- 2026-07-23:如果 target-only synthetic focus 无法满足某个目标框架,`sky_click` 应明确失败,不得静默恢复到前台 defocus 或全局 HID。
+- 2026-07-23:隔离 Chrome 在完全遮挡时仍能通过 target-only synthetic focus 触发一次 DOM click;前台 fixture 的 active、key window、first responder 和 loss counters 全部保持不变。
diff --git a/docs/histories/2026-07/20260723-1655-preserve-sky-click-focus.md b/docs/histories/2026-07/20260723-1655-preserve-sky-click-focus.md
new file mode 100644
index 0000000..9dcdb70
--- /dev/null
+++ b/docs/histories/2026-07/20260723-1655-preserve-sky-click-focus.md
@@ -0,0 +1,40 @@
+## [2026-07-23 16:55] | Task: 修复 sky_click 前台焦点丢失
+
+### 🤖 Execution Context
+* **Agent ID**: `/root`
+* **Base Model**: `GPT-5`
+* **Runtime**: `Codex desktop / macOS arm64`
+
+### 📥 User Query
+> 修复 `sky_click` 虽然不置顶目标窗口,但会让当前应用失去输入焦点的问题,并对照 Codex 的行为补强验证。
+
+### 🛠 Changes Overview
+**Scope:** macOS SkyLight click runtime、snapshot recovery、fixture focus probe、测试和文档。
+
+**Key Actions:**
+- **Target-only synthetic focus**: 移除对真实前台应用的 defocus / restore record,只让目标应用短暂进入 synthetic-active 状态,renderer settle 后只撤销目标状态。
+- **Non-invasive refresh**: `sky_click` 的 action-result snapshot 使用 read-only recovery policy,禁止通过 `NSRunningApplication.activate` 或 `AXRaise` 恢复目标窗口。
+- **Focus regression**: 扩展 fixture 状态,记录 AppKit active、key window、first responder、`resignActive` 和 `resignKey` 累计次数;隔离 Chrome 实机回归对这些状态建立断言。
+- **Compatibility boundary**: 不改变 Chromium primer、PID/window event fields、SkyLight/public 双通道投递、`auto` 默认行为或显式 fail-closed 语义。
+
+### 🧠 Design Intent (Why)
+*旧实现把“without raise”误当成“保持焦点”:它没有改变 WindowServer frontmost PID 或 z-order,却主动向真实前台应用发送 `focused=false`,足以触发 AppKit resign/key/first-responder 副作用。新的 session 数据结构不再保存前台 PSN/window,从结构上禁止这类投递;后台目标如果需要兼容状态,只能获得独立的 target-only synthetic focus。*
+
+### ✅ Verification
+- `swift build` passes.
+- `swift test` passes: 149 OpenComputerUseKit unit tests, 3 StandaloneCursorSupport tests, and 1 intentionally skipped opt-in live test.
+- `OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests` passes; isolated fully covered Chrome triggers exactly one DOM click.
+- Foreground fixture remains frontmost, active and key; its text-field first responder and resign/key-loss counters stay unchanged.
+- Real pointer position and Chrome/cover z-order remain unchanged.
+- `./scripts/run-tool-smoke-tests.sh` passes the full 9-tool sequence and visual cursor idle smoke.
+- `npm run package:skill` passes.
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift`
+- `apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/`
+- `docs/` and `skills/open-computer-use/references/usage.md`
diff --git a/docs/references/macos-skylight-background-click.md b/docs/references/macos-skylight-background-click.md
index dcee6f1..a17280a 100644
--- a/docs/references/macos-skylight-background-click.md
+++ b/docs/references/macos-skylight-background-click.md
@@ -20,18 +20,21 @@
`sky_click` 使用当前 snapshot 的 PID、`CGWindowID`、窗口内坐标和屏幕坐标,按以下顺序投递:
-1. 保存当前前台 PSN/window id,向原前台发送 defocus record,再向目标发送 focus record;两次 `SLPSPostEventRecordTo` 只切 AppKit-active 状态,不调用会 raise / 切 Space 的 `SLPSSetFrontProcessWithOptions`。
+1. 用 `GetProcessForPID` 解析目标 PSN,只向目标发送 focus record,让目标短暂进入 synthetic-active 状态;不查询前台 PSN,也绝不向真实前台应用发送 defocus record。
2. 目标点 `mouseMoved`,gesture phase `2`。
3. `(-1, -1)` 的 off-window primer `mouseDown` / `mouseUp`,phase `1` / `2`。
4. 等待 `100ms`,把真实目标 `mouseDown` / `mouseUp` 以 phase `3` 投递。
5. 双击时等待 `80ms` 后发送第二对事件,并把 click state 从 `1` 递增到 `2`。
-6. 等待 renderer 消费异步 mouse-up,再向目标发送 defocus、向原前台发送 focus record,恢复 AppKit-active 状态。
+6. 等待 renderer 消费异步 mouse-up,再只向目标发送 defocus record,撤销本轮 synthetic-active 状态。
每个事件带相同 click-group id,并设置 PID、窗口 id、window-under-pointer 和 window-local location。每一步同时走 `SLEventPostToPid` 与公开 `CGEvent.postToPid`:前者覆盖 Chromium/Catalyst,后者保留 AppKit 兼容性。这是一个固定 dispatch policy,不是失败后的 retry。
+旧实现曾按 Cua / yabai 的 focus-without-raise pattern,先向真实前台应用发送 defocus,再在结束时发送 focus 恢复。该序列虽然不改变 WindowServer frontmost PID 或 z-order,却会触发 AppKit `resignActive` / `resignKey` 并破坏 first responder。受控 Chrome 验证表明只合成目标 focus 已足够,因此当前实现把“前台应用从未失活”作为硬约束。
+
## OCU 明确没有采用的部分
-- 不调用 `SLPSSetFrontProcessWithOptions`,也不使用 `NSRunningApplication.activate` 操作目标 app。focus-without-raise 只切换并恢复 AppKit event routing state,WindowServer frontmost 与窗口 z-order 必须保持不变。
+- 不调用 `SLPSSetFrontProcessWithOptions`,也不使用 `NSRunningApplication.activate` 或 `AXRaise` 操作目标 app。只改变目标应用的 synthetic event-routing 状态,真实前台应用的 AppKit active、key window 和 first responder 必须保持不变。
+- `sky_click` 完成后的 action-result snapshot 使用 read-only recovery policy;目标 AX/window 瞬时不可读时返回错误,不允许 snapshot 恢复路径激活或抬升目标。
- 不把 `sky_click` 放进 `auto`,也不从失败的 `sky_click` 回退到 `global`。
- 不支持右键、中键、三击、跨 Space、隐藏或最小化窗口。
- 不承诺 Canvas、Unity、Blender 或其他拒绝 PID 定向事件的 surface 可用。
@@ -44,10 +47,9 @@
- `SLEventSetIntegerValueField`
- `CGEventSetWindowLocation`
- `SLPSPostEventRecordTo`
-- `_SLPSGetFrontProcess`
- `GetProcessForPID`
-投递前还必须确认 snapshot 的 `CGWindowID` 仍由同一 PID 所有且仍为 on-screen。macOS 更新后应重新验证符号、事件字段、签名 app 制品和被完全遮挡的 Chromium 页面。
+投递前还必须确认 snapshot 的 `CGWindowID` 仍由同一 PID 所有且仍为 on-screen。macOS 更新后应重新验证符号、事件字段、签名 app 制品和被完全遮挡的 Chromium 页面;实机门槛同时包括前台 fixture 的 active/key/first-responder 状态和 transient resign/key-loss 计数。
## 许可
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index 597c37e..3caee02 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -37,6 +37,11 @@ enum SnapshotMode {
case fixture
}
+enum SnapshotRecoveryPolicy: Equatable {
+ case allowActivation
+ case readOnly
+}
+
public struct AccessibilityTreeLimits: Equatable, Sendable {
public static let defaultMaxNodeCount = 1200
public static let defaultMaxDepth = 64
@@ -143,7 +148,8 @@ enum SnapshotBuilder {
static func build(
for app: RunningAppDescriptor,
textLimit: SnapshotTextLimit = .defaults,
- treeLimits: AccessibilityTreeLimits = .defaults
+ treeLimits: AccessibilityTreeLimits = .defaults,
+ recoveryPolicy: SnapshotRecoveryPolicy = .allowActivation
) throws -> AppSnapshot {
if app.name == FixtureBridge.appName, let fixtureState = try FixtureBridge.readState() {
return buildFixtureSnapshot(app: app, state: fixtureState)
@@ -159,7 +165,9 @@ enum SnapshotBuilder {
let systemWide = AXUIElementCreateSystemWide()
var focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)
var focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)
- if focusedWindow == nil, recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: nil) {
+ if focusedWindow == nil,
+ recoveryPolicy == .allowActivation,
+ recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: nil) {
focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)
focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)
}
@@ -172,7 +180,9 @@ enum SnapshotBuilder {
var windowTitle = stringValue(of: rootWindow, attribute: kAXTitleAttribute)
var windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)
- if windowCapture == nil, recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: rootWindow) {
+ if windowCapture == nil,
+ recoveryPolicy == .allowActivation,
+ recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: rootWindow) {
focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)
if let recoveredWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide) {
rootWindow = recoveredWindow
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
index 250371e..afdce9b 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
@@ -16,6 +16,10 @@ public enum ClickMethod: String, CaseIterable, Sendable {
case global
}
+func clickActionSnapshotRecoveryPolicy(for method: ClickMethod) -> SnapshotRecoveryPolicy {
+ method == .skyClick ? .readOnly : .allowActivation
+}
+
func parseClickMethod(_ rawValue: String?) throws -> ClickMethod {
let normalized = rawValue?
.trimmingCharacters(in: .whitespacesAndNewlines)
@@ -607,7 +611,13 @@ public final class ComputerUseService {
throw ComputerUseError.invalidArguments("click requires either element_index or x/y")
}
- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)
+ return snapshotResult(
+ for: try refreshSnapshot(
+ for: query,
+ recoveryPolicy: clickActionSnapshotRecoveryPolicy(for: clickMethod)
+ ),
+ style: .actionResult
+ )
}
public func performSecondaryAction(app query: String, elementIndex: String, action: String) throws -> ToolCallResult {
@@ -790,10 +800,16 @@ public final class ComputerUseService {
private func refreshSnapshot(
for query: String,
textLimit: SnapshotTextLimit = .defaults,
- treeLimits: AccessibilityTreeLimits = .defaults
+ treeLimits: AccessibilityTreeLimits = .defaults,
+ recoveryPolicy: SnapshotRecoveryPolicy = .allowActivation
) throws -> AppSnapshot {
let app = try AppDiscovery.resolve(query)
- let snapshot = try SnapshotBuilder.build(for: app, textLimit: textLimit, treeLimits: treeLimits)
+ let snapshot = try SnapshotBuilder.build(
+ for: app,
+ textLimit: textLimit,
+ treeLimits: treeLimits,
+ recoveryPolicy: recoveryPolicy
+ )
let keys = Set([
query.lowercased(),
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift
index 14cff5d..d9723f9 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/FixtureBridge.swift
@@ -40,16 +40,36 @@ public struct FixtureElementState: Codable, Sendable {
}
public struct FixtureAppState: Codable, Sendable {
+ public let processIdentifier: pid_t?
public let windowTitle: String
public let windowBounds: FixtureRect
public let focusedIdentifier: String?
public let elements: [FixtureElementState]
+ public let isActive: Bool?
+ public let isKeyWindow: Bool?
+ public let activationLossCount: Int?
+ public let keyWindowLossCount: Int?
- public init(windowTitle: String, windowBounds: FixtureRect, focusedIdentifier: String?, elements: [FixtureElementState]) {
+ public init(
+ processIdentifier: pid_t? = nil,
+ windowTitle: String,
+ windowBounds: FixtureRect,
+ focusedIdentifier: String?,
+ elements: [FixtureElementState],
+ isActive: Bool? = nil,
+ isKeyWindow: Bool? = nil,
+ activationLossCount: Int? = nil,
+ keyWindowLossCount: Int? = nil
+ ) {
+ self.processIdentifier = processIdentifier
self.windowTitle = windowTitle
self.windowBounds = windowBounds
self.focusedIdentifier = focusedIdentifier
self.elements = elements
+ self.isActive = isActive
+ self.isKeyWindow = isKeyWindow
+ self.activationLossCount = activationLossCount
+ self.keyWindowLossCount = keyWindowLossCount
}
}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift
index 054447f..878d357 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyClickSimulation.swift
@@ -143,14 +143,13 @@ enum SkyClickDispatcher {
let clickGroupID = Int64(DispatchTime.now().uptimeNanoseconds % 1_000_000_000)
let frontmostPID = NSWorkspace.shared.frontmostApplication?.processIdentifier
- let focusContext: SkyLightFocusContext?
+ let focusContext: SkyLightSyntheticFocusContext?
if frontmostPID == target.pid {
focusContext = nil
} else {
- focusContext = try spi.beginFocusWithoutRaise(
+ focusContext = try spi.beginSyntheticTargetFocus(
targetPID: target.pid,
- targetWindowID: target.windowID,
- previousWindowID: frontmostWindowID(for: frontmostPID) ?? 0
+ targetWindowID: target.windowID
)
}
@@ -192,17 +191,17 @@ enum SkyClickDispatcher {
}
} catch {
if let focusContext {
- try? restoreFocus(context: focusContext, spi: spi)
+ try? spi.endSyntheticTargetFocus(focusContext)
}
throw error
}
if let focusContext {
// SkyLight delivery is asynchronous. Keep the target's AppKit
- // active state long enough for Chromium's renderer hop to consume
- // the final mouse-up before restoring the user's active app.
+ // synthetic active state long enough for Chromium's renderer hop
+ // to consume the final mouse-up before deactivating only the target.
Thread.sleep(forTimeInterval: 0.100)
- try restoreFocus(context: focusContext, spi: spi)
+ try spi.endSyntheticTargetFocus(focusContext)
}
}
@@ -260,42 +259,4 @@ enum SkyClickDispatcher {
try spi.setWindowLocation(event, point: windowPoint)
}
- private static func frontmostWindowID(for pid: pid_t?) -> CGWindowID? {
- guard let pid else {
- return nil
- }
-
- let windowInfo = CGWindowListCopyWindowInfo(
- [.optionOnScreenOnly, .excludeDesktopElements],
- kCGNullWindowID
- ) as? [[String: Any]] ?? []
- return windowInfo.first { info in
- guard
- let ownerPID = info[kCGWindowOwnerPID as String] as? NSNumber,
- ownerPID.int32Value == pid,
- let layer = info[kCGWindowLayer as String] as? NSNumber
- else {
- return false
- }
- return layer.intValue == 0
- }.flatMap { info in
- (info[kCGWindowNumber as String] as? NSNumber)?.uint32Value
- }
- }
-
- private static func restoreFocus(
- context: SkyLightFocusContext,
- spi: SkyLightSPI
- ) throws {
- let currentFrontPID = NSWorkspace.shared.frontmostApplication?.processIdentifier
- let currentFrontWindowID: CGWindowID? = if currentFrontPID == nil {
- nil
- } else {
- frontmostWindowID(for: currentFrontPID) ?? 0
- }
- try spi.restoreFocusWithoutRaise(
- context,
- currentFrontWindowID: currentFrontWindowID
- )
- }
}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift
index 0db056b..4f14930 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift
@@ -2,11 +2,37 @@ import CoreGraphics
import Darwin
import Foundation
-struct SkyLightFocusContext {
- let previousPSN: [UInt8]
- let targetPSN: [UInt8]
- let previousWindowID: CGWindowID
- let targetWindowID: CGWindowID
+struct SkyLightActivationCommand: Equatable {
+ let psn: [UInt8]
+ let windowID: CGWindowID
+ let focused: Bool
+}
+
+struct SkyLightSyntheticFocusContext {
+ let deactivateTarget: SkyLightActivationCommand
+}
+
+struct SkyLightSyntheticFocusPlan: Equatable {
+ let activateTarget: SkyLightActivationCommand
+ let deactivateTarget: SkyLightActivationCommand
+}
+
+func skyLightSyntheticTargetFocusPlan(
+ targetPSN: [UInt8],
+ targetWindowID: CGWindowID
+) -> SkyLightSyntheticFocusPlan {
+ SkyLightSyntheticFocusPlan(
+ activateTarget: SkyLightActivationCommand(
+ psn: targetPSN,
+ windowID: targetWindowID,
+ focused: true
+ ),
+ deactivateTarget: SkyLightActivationCommand(
+ psn: targetPSN,
+ windowID: targetWindowID,
+ focused: false
+ )
+ )
}
func skyLightActivationRecord(windowID: CGWindowID, focused: Bool) -> [UInt8] {
@@ -50,7 +76,6 @@ final class SkyLightSPI: @unchecked Sendable {
private static let setIntegerFieldSymbol = "SLEventSetIntegerValueField"
private static let setWindowLocationSymbol = "CGEventSetWindowLocation"
private static let postEventRecordSymbol = "SLPSPostEventRecordTo"
- private static let getFrontProcessSymbol = "_SLPSGetFrontProcess"
private static let getProcessForPIDSymbol = "GetProcessForPID"
private static let applicationServicesPath = "/System/Library/Frameworks/ApplicationServices.framework/ApplicationServices"
@@ -61,7 +86,6 @@ final class SkyLightSPI: @unchecked Sendable {
// relying on Swift's aggregate CGPoint calling convention.
private typealias SetWindowLocationFunction = @convention(c) (UnsafeMutableRawPointer?, Double, Double) -> Void
private typealias PostEventRecordFunction = @convention(c) (UnsafeRawPointer?, UnsafePointer<UInt8>?) -> Int32
- private typealias GetFrontProcessFunction = @convention(c) (UnsafeMutableRawPointer?) -> Int32
private typealias GetProcessForPIDFunction = @convention(c) (pid_t, UnsafeMutableRawPointer?) -> Int32
private let frameworkHandle: UnsafeMutableRawPointer?
@@ -70,7 +94,6 @@ final class SkyLightSPI: @unchecked Sendable {
private let setIntegerFieldFunction: SetIntegerFieldFunction?
private let setWindowLocationFunction: SetWindowLocationFunction?
private let postEventRecordFunction: PostEventRecordFunction?
- private let getFrontProcessFunction: GetFrontProcessFunction?
private let getProcessForPIDFunction: GetProcessForPIDFunction?
let capability: SkyLightSPICapability
@@ -84,7 +107,6 @@ final class SkyLightSPI: @unchecked Sendable {
setIntegerFieldFunction = Self.resolve(handle: handle, symbol: Self.setIntegerFieldSymbol)
setWindowLocationFunction = Self.resolve(handle: handle, symbol: Self.setWindowLocationSymbol)
postEventRecordFunction = Self.resolve(handle: handle, symbol: Self.postEventRecordSymbol)
- getFrontProcessFunction = Self.resolve(handle: handle, symbol: Self.getFrontProcessSymbol)
getProcessForPIDFunction = Self.resolve(handle: appServicesHandle, symbol: Self.getProcessForPIDSymbol)
var missingSymbols: [String] = []
@@ -100,9 +122,6 @@ final class SkyLightSPI: @unchecked Sendable {
if postEventRecordFunction == nil {
missingSymbols.append(Self.postEventRecordSymbol)
}
- if getFrontProcessFunction == nil {
- missingSymbols.append(Self.getFrontProcessSymbol)
- }
if getProcessForPIDFunction == nil {
missingSymbols.append(Self.getProcessForPIDSymbol)
}
@@ -133,28 +152,14 @@ final class SkyLightSPI: @unchecked Sendable {
setWindowLocationFunction(opaquePointer(for: event), point.x, point.y)
}
- func beginFocusWithoutRaise(
+ func beginSyntheticTargetFocus(
targetPID: pid_t,
- targetWindowID: CGWindowID,
- previousWindowID: CGWindowID
- ) throws -> SkyLightFocusContext {
- guard
- let getFrontProcessFunction,
- let getProcessForPIDFunction
- else {
+ targetWindowID: CGWindowID
+ ) throws -> SkyLightSyntheticFocusContext {
+ guard let getProcessForPIDFunction else {
throw unavailableError()
}
- var previousPSN = [UInt8](repeating: 0, count: 8)
- let previousStatus = previousPSN.withUnsafeMutableBytes { bytes in
- getFrontProcessFunction(bytes.baseAddress)
- }
- guard previousStatus == 0 else {
- throw ComputerUseError.message(
- "click_method 'sky_click' could not resolve the front process PSN (OSStatus \(previousStatus))"
- )
- }
-
var targetPSN = [UInt8](repeating: 0, count: 8)
let targetStatus = targetPSN.withUnsafeMutableBytes { bytes in
getProcessForPIDFunction(targetPID, bytes.baseAddress)
@@ -165,67 +170,21 @@ final class SkyLightSPI: @unchecked Sendable {
)
}
- try postActivationRecord(psn: previousPSN, windowID: previousWindowID, focused: false)
- Thread.sleep(forTimeInterval: 0.040)
- do {
- try postActivationRecord(psn: targetPSN, windowID: targetWindowID, focused: true)
- Thread.sleep(forTimeInterval: 0.040)
- } catch {
- try? postActivationRecord(psn: previousPSN, windowID: previousWindowID, focused: true)
- throw error
- }
-
- return SkyLightFocusContext(
- previousPSN: previousPSN,
+ let plan = skyLightSyntheticTargetFocusPlan(
targetPSN: targetPSN,
- previousWindowID: previousWindowID,
targetWindowID: targetWindowID
)
- }
-
- func restoreFocusWithoutRaise(
- _ context: SkyLightFocusContext,
- currentFrontWindowID: CGWindowID?
- ) throws {
- let restorePSN: [UInt8]
- let restoreWindowID: CGWindowID
- if let getFrontProcessFunction {
- var currentFrontPSN = [UInt8](repeating: 0, count: 8)
- let status = currentFrontPSN.withUnsafeMutableBytes { bytes in
- getFrontProcessFunction(bytes.baseAddress)
- }
- if status == 0 {
- restorePSN = currentFrontPSN
- restoreWindowID = currentFrontWindowID ?? context.previousWindowID
- } else {
- restorePSN = context.previousPSN
- restoreWindowID = context.previousWindowID
- }
- } else {
- restorePSN = context.previousPSN
- restoreWindowID = context.previousWindowID
- }
+ try postActivationCommand(plan.activateTarget)
+ Thread.sleep(forTimeInterval: 0.040)
- var defocusError: Error?
- do {
- try postActivationRecord(
- psn: context.targetPSN,
- windowID: context.targetWindowID,
- focused: false
- )
- } catch {
- defocusError = error
- }
+ return SkyLightSyntheticFocusContext(
+ deactivateTarget: plan.deactivateTarget
+ )
+ }
+ func endSyntheticTargetFocus(_ context: SkyLightSyntheticFocusContext) throws {
+ try postActivationCommand(context.deactivateTarget)
Thread.sleep(forTimeInterval: 0.040)
- try postActivationRecord(
- psn: restorePSN,
- windowID: restoreWindowID,
- focused: true
- )
- if let defocusError {
- throw defocusError
- }
}
private func unavailableError() -> ComputerUseError {
@@ -238,24 +197,23 @@ final class SkyLightSPI: @unchecked Sendable {
Unmanaged.passUnretained(event).toOpaque()
}
- private func postActivationRecord(
- psn: [UInt8],
- windowID: CGWindowID,
- focused: Bool
- ) throws {
+ private func postActivationCommand(_ command: SkyLightActivationCommand) throws {
guard let postEventRecordFunction else {
throw unavailableError()
}
- let record = skyLightActivationRecord(windowID: windowID, focused: focused)
- let status = psn.withUnsafeBytes { psnBytes in
+ let record = skyLightActivationRecord(
+ windowID: command.windowID,
+ focused: command.focused
+ )
+ let status = command.psn.withUnsafeBytes { psnBytes in
record.withUnsafeBufferPointer { recordBytes in
postEventRecordFunction(psnBytes.baseAddress, recordBytes.baseAddress)
}
}
guard status == 0 else {
throw ComputerUseError.message(
- "click_method 'sky_click' focus-without-raise event failed (OSStatus \(status))"
+ "click_method 'sky_click' synthetic target-focus event failed (OSStatus \(status))"
)
}
}
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index a91c0ea..8b227b5 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -1371,6 +1371,38 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertEqual(try parseClickMethod("GLOBAL"), .global)
}
+ func testOnlySkyClickUsesReadOnlyActionSnapshotRefresh() {
+ XCTAssertEqual(clickActionSnapshotRecoveryPolicy(for: .skyClick), .readOnly)
+
+ for method in ClickMethod.allCases where method != .skyClick {
+ XCTAssertEqual(
+ clickActionSnapshotRecoveryPolicy(for: method),
+ .allowActivation
+ )
+ }
+ }
+
+ func testFixtureStateFocusProbeFieldsRemainBackwardCompatible() throws {
+ let legacyPayload = #"""
+ {
+ "windowTitle": "Legacy Fixture",
+ "windowBounds": {"x": 1, "y": 2, "width": 3, "height": 4},
+ "focusedIdentifier": null,
+ "elements": []
+ }
+ """#
+
+ let state = try JSONDecoder().decode(
+ FixtureAppState.self,
+ from: Data(legacyPayload.utf8)
+ )
+ XCTAssertNil(state.processIdentifier)
+ XCTAssertNil(state.isActive)
+ XCTAssertNil(state.isKeyWindow)
+ XCTAssertNil(state.activationLossCount)
+ XCTAssertNil(state.keyWindowLossCount)
+ }
+
func testClickMethodRejectsUnknownValues() {
for value in ["physical", "targeted"] {
XCTAssertThrowsError(try parseClickMethod(value)) { error in
@@ -1517,6 +1549,30 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertEqual(defocused[0x8A], 0x02)
}
+ func testSkyLightSyntheticFocusPlanOnlyAddressesTarget() {
+ let targetPSN = [UInt8](repeating: 7, count: 8)
+ let plan = skyLightSyntheticTargetFocusPlan(
+ targetPSN: targetPSN,
+ targetWindowID: 321
+ )
+
+ XCTAssertEqual(
+ plan,
+ SkyLightSyntheticFocusPlan(
+ activateTarget: SkyLightActivationCommand(
+ psn: targetPSN,
+ windowID: 321,
+ focused: true
+ ),
+ deactivateTarget: SkyLightActivationCommand(
+ psn: targetPSN,
+ windowID: 321,
+ focused: false
+ )
+ )
+ )
+ }
+
func testSkyLightRuntimeSPIProbeCanStampEventWithoutPosting() throws {
let spi = SkyLightSPI.shared
guard spi.capability.isAvailable else {
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swift
index 6aa0bdf..1d1acdb 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/SkyClickLiveTests.swift
@@ -134,6 +134,17 @@ final class SkyClickLiveTests: XCTestCase {
"The controlled Chrome window must be fully covered before sky_click"
)
XCTAssertLessThan(coverIndex, targetIndex, "The cover window must be above Chrome in z-order")
+ try FixtureBridge.post(
+ FixtureCommand(kind: "click", identifier: "fixture-input")
+ )
+ let foregroundStateBefore = try waitForFixtureState(
+ pid: cover.processIdentifier,
+ failure: "The foreground fixture did not keep its text field focused"
+ ) { state in
+ state.isActive == true
+ && state.isKeyWindow == true
+ && state.focusedIdentifier == "fixture-input"
+ }
print(
"sky_click live test: cover=\(freshCoverWindow.bounds) target=\(coveredWindow.bounds) "
+ "z-order=\(coverIndex)<\(targetIndex) front-pid="
@@ -170,6 +181,27 @@ final class SkyClickLiveTests: XCTestCase {
frontPIDBefore,
"sky_click must not change the frontmost app"
)
+ let foregroundStateAfter = try waitForFixtureState(
+ pid: cover.processIdentifier,
+ failure: "The foreground fixture state was unavailable after sky_click"
+ ) { _ in true }
+ XCTAssertEqual(foregroundStateAfter.isActive, true, "sky_click must keep the foreground app active")
+ XCTAssertEqual(foregroundStateAfter.isKeyWindow, true, "sky_click must keep the foreground window key")
+ XCTAssertEqual(
+ foregroundStateAfter.focusedIdentifier,
+ foregroundStateBefore.focusedIdentifier,
+ "sky_click must preserve the foreground first responder"
+ )
+ XCTAssertEqual(
+ foregroundStateAfter.activationLossCount,
+ foregroundStateBefore.activationLossCount,
+ "sky_click must not transiently resign the foreground app"
+ )
+ XCTAssertEqual(
+ foregroundStateAfter.keyWindowLossCount,
+ foregroundStateBefore.keyWindowLossCount,
+ "sky_click must not transiently resign the foreground key window"
+ )
if let cursorBefore, let cursorAfter = CGEvent(source: nil)?.location {
XCTAssertLessThan(hypot(cursorAfter.x - cursorBefore.x, cursorAfter.y - cursorBefore.y), 0.5)
@@ -225,6 +257,26 @@ final class SkyClickLiveTests: XCTestCase {
throw ComputerUseError.message(failure)
}
+ private func waitForFixtureState(
+ pid: pid_t,
+ failure: String,
+ condition: (FixtureAppState) -> Bool
+ ) throws -> FixtureAppState {
+ var match: FixtureAppState?
+ try waitUntil(timeout: 5, failure: failure) {
+ guard
+ let state = try? FixtureBridge.readState(),
+ state.processIdentifier == pid,
+ condition(state)
+ else {
+ return false
+ }
+ match = state
+ return true
+ }
+ return try XCTUnwrap(match)
+ }
+
private func stop(_ process: Process) {
if process.isRunning {
process.terminate()
diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.md
index fce394b..ea1ffb5 100644
--- a/skills/open-computer-use/references/usage.md
+++ b/skills/open-computer-use/references/usage.md
@@ -111,7 +111,7 @@ open-computer-use snapshot --max-tree-nodes 3000 --max-tree-depth 96 "Google Chr
- `accessibility`: only invoke the element's accessibility action and require `element_index`.
- `app_post`: bypass accessibility and post a mouse event directly to the target app/window without moving the system pointer. Supported on macOS and Windows.
-- `sky_click`: use the macOS private SkyLight background-window path with focus-without-raise and a Chromium primer click. It supports left single/double click on a current, on-screen window in the same Space and does not move the system pointer, change the frontmost app, or raise the target window. Supported on macOS only.
+- `sky_click`: use the macOS private SkyLight background-window path with target-only synthetic focus and a Chromium primer click. It supports left single/double click on a current, on-screen window in the same Space and does not move the system pointer, deactivate the foreground app, change its key/first-responder state, or raise the target window. Its action-result snapshot refresh is read-only. Supported on macOS only.
- `global`: bypass accessibility and use the desktop's global pointer path. Supported on macOS and Linux, and requires `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` because it may move the real pointer or change foreground focus.
Use `app_post` for an exact blank-area or overlay click that must not be redirected to an accessibility descendant: