👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Reduce intrusive focus stealing in computer use tools
Leo38cbf353ed04
diff --git a/artifacts/tool-comparisons/20260417-focus-behavior/README.md b/artifacts/tool-comparisons/20260417-focus-behavior/README.mdnew file mode 100644index 0000000..3ecbfe2--- /dev/null+++ b/artifacts/tool-comparisons/20260417-focus-behavior/README.md@@ -0,0 +1,28 @@+# 2026-04-17 Focus Behavior Comparison++这组样本用于对比官方 `computer-use` 与仓库内 `open-codex-computer-use` 在“只读取状态”和“坐标点击”场景下对用户前台焦点/鼠标的影响。++## 目录++- `computer-use/`+ - 来自已连接官方 `computer-use` MCP tool 的样本。+- `open-codex-computer-use/`+ - 来自仓库当前实现的样本。+ - `get_app_state` 通过本地打包后的 `dist/OpenCodexComputerUse.app` 走 JSON-RPC `tools/call` 采集。+ - `click` 同样通过本地 JSON-RPC 采集。++## 方法++- 目标 app:`Activity Monitor`+- 对照前台 app:`iTerm2`+- 观测项:+ - tool request+ - tool result 摘要+ - 调用前后的 frontmost app+ - 调用前后的鼠标坐标++## 已知限制++- 官方 `computer-use` 当前不能直接识别仓库里的裸可执行 fixture,因此对比目标改为双方都能解析的系统 app。+- 鼠标坐标会受到用户实时移动影响,因此这里只把它当作辅助观测;“是否切走前台 app”是更稳定的对比信号。+- `open-codex-computer-use` 的坐标点击现在会先做 AX hit-test;如果命中失败,则仍会降级到全局 HID。diff --git a/artifacts/tool-comparisons/20260417-focus-behavior/computer-use/get_app_state-activity-monitor.json b/artifacts/tool-comparisons/20260417-focus-behavior/computer-use/get_app_state-activity-monitor.jsonnew file mode 100644index 0000000..8ffdc23--- /dev/null+++ b/artifacts/tool-comparisons/20260417-focus-behavior/computer-use/get_app_state-activity-monitor.json@@ -0,0 +1,34 @@+{+ "captured_at": "2026-04-17",+ "tool": "get_app_state",+ "request": {+ "app": "Activity Monitor"+ },+ "transport": "assistant tool call: mcp__computer_use__.get_app_state",+ "observations": {+ "before": {+ "frontmost_app": "iTerm2",+ "mouse_location": "1701,702"+ },+ "after": {+ "frontmost_app": "iTerm2",+ "mouse_location": "1724,685"+ }+ },+ "result_summary": {+ "app": "com.apple.ActivityMonitor (pid 988)",+ "window": "Activity Monitor – All Processes",+ "focused_ui_element": "0 standard window",+ "screenshot_returned": true+ },+ "result_excerpt": [+ "Computer Use state (CUA App Version: 750)",+ "<app_state>",+ "App=com.apple.ActivityMonitor (pid 988)",+ "Window: \"Activity Monitor\", App: Activity Monitor.",+ "0 standard window Activity Monitor – All Processes, Secondary Actions: Raise",+ "38 search text field (settable, string) Helper",+ "The focused UI element is 0 standard window.",+ "</app_state>"+ ]+}diff --git a/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/click-activity-monitor-coordinate.json b/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/click-activity-monitor-coordinate.jsonnew file mode 100644index 0000000..8b9e0cd--- /dev/null+++ b/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/click-activity-monitor-coordinate.json@@ -0,0 +1,29 @@+{+ "captured_at": "2026-04-17",+ "tool": "click",+ "request": {+ "app": "Activity Monitor",+ "x": 850,+ "y": 18+ },+ "transport": "local JSON-RPC to dist/OpenCodexComputerUse.app",+ "observations": {+ "before": {+ "frontmost_app": "iTerm2",+ "mouse_location": "1535,694"+ },+ "after": {+ "frontmost_app": "iTerm2",+ "mouse_location": "1984,807"+ }+ },+ "result_summary": {+ "app": "com.apple.ActivityMonitor (pid 988)",+ "window": "Activity Monitor – All Processes",+ "is_error": false+ },+ "notes": [+ "该样本用于验证坐标点击不会默认抢前台焦点。",+ "当前结果至少证明这次调用结束后前台仍是 iTerm2;命中的具体控件仍需后续补更精确的目标样本继续评估。"+ ]+}diff --git a/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/get_app_state-activity-monitor.json b/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/get_app_state-activity-monitor.jsonnew file mode 100644index 0000000..0df83c7--- /dev/null+++ b/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/get_app_state-activity-monitor.json@@ -0,0 +1,29 @@+{+ "captured_at": "2026-04-17",+ "tool": "get_app_state",+ "request": {+ "app": "Activity Monitor"+ },+ "transport": "local JSON-RPC to dist/OpenCodexComputerUse.app",+ "observations": {+ "before": {+ "frontmost_app": "iTerm2",+ "mouse_location": "1408,675"+ },+ "after": {+ "frontmost_app": "iTerm2",+ "mouse_location": "1537,938"+ }+ },+ "result_summary": {+ "app": "com.apple.ActivityMonitor (pid 988)",+ "window": "Activity Monitor – All Processes",+ "bounds": "x=809, y=685, w=960, h=640",+ "focused_ui_element": "0 standard window",+ "screenshot_path_returned": true+ },+ "notes": [+ "这次样本来自修复后的 dist 制品,而不是已经断开的旧 MCP 会话。",+ "对比结果显示 get_app_state 不再为了读状态而切走前台 iTerm2。"+ ]+}diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 75f8545..6f1132d 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -47,12 +47,14 @@- `ComputerUseService` 负责把 MCP tool 请求映射到本地能力。- `list_apps` 通过 `NSWorkspace` 枚举运行中的 app。-- `get_app_state` 优先走真实 AX / 窗口截图;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。+- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。-- 动作型 tools 对普通 app 走:+- 动作型 tools 对普通 app 采用“非侵入优先,HID 兜底”策略:- `AXUIElementPerformAction`- `AXUIElementSetAttributeValue`- - `CGEvent` 键鼠事件+ - `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素+ - `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台+ - 只有 drag 或无法命中 AX 元素的鼠标路径,才退回全局 `CGEvent` 键鼠事件并显式前置目标 app### 4. Fixture Bridge@@ -72,6 +74,7 @@- 单元测试:`swift test`- 端到端 smoke:`./scripts/run-tool-smoke-tests.sh`- app 打包:`./scripts/build-open-codex-app.sh debug`+- 对比样本:`artifacts/tool-comparisons/20260417-focus-behavior/`- 手工诊断:- `.build/debug/OpenCodexComputerUse doctor`- `.build/debug/OpenCodexComputerUse snapshot <app>`diff --git a/docs/QUALITY_SCORE.md b/docs/QUALITY_SCORE.mdindex 6d2660a..8789028 100644--- a/docs/QUALITY_SCORE.md+++ b/docs/QUALITY_SCORE.md@@ -13,6 +13,6 @@| --- | --- | --- | --- || 产品面 | B | 已经有 Swift 本地 `computer-use` MCP server、默认 app 模式权限引导,以及 9 个 tools 的可运行实现。 | 继续收敛权限 UI 细节、app allowlist 和更清晰的用户错误提示。 || 架构文档 | B | 顶层结构、fixture bridge、app 模式和验证路径已经落文档。 | 后续补 release artifact、code signing / notarization 和 host 集成方式。 |-| 测试 | B | `swift test` + smoke suite 已覆盖 9 个 tools 的回归,并补了真实 `doctor` / `snapshot` spot check。 | 增加更多普通 app 的手工或录制回归,减少只依赖 fixture。 |-| 可观测性 | C | 已有 `doctor`、`snapshot` 和 smoke 输出,但还没有结构化日志。 | 补统一日志级别、失败上下文和 release artifact 里的诊断信息。 |+| 测试 | B | `swift test` + smoke suite 已覆盖 9 个 tools 的回归,并新增了针对“前台焦点是否被抢占”的手工对比样本沉淀。 | 增加更多普通 app 的录制回归,减少只依赖 fixture 和一次性手工检查。 |+| 可观测性 | C | 已有 `doctor`、`snapshot`、smoke 输出,以及一组仓库内留档的 `computer-use` / `open-codex-computer-use` 对比样本。 | 补统一日志级别、失败上下文和 release artifact 里的诊断信息,把一次性样本收敛成可重复采集流程。 || 安全 | B | 已明确本地-only、权限边界和 fixture test bridge 的作用域。 | 增加 app allowlist / session approval,避免对任意 app 默认开放。 |diff --git a/docs/exec-plans/completed/20260417-nonintrusive-computer-use.md b/docs/exec-plans/completed/20260417-nonintrusive-computer-use.mdnew file mode 100644index 0000000..2ccc9d0--- /dev/null+++ b/docs/exec-plans/completed/20260417-nonintrusive-computer-use.md@@ -0,0 +1,67 @@+# 非抢焦点 computer-use 交互改造++## 目标++让 `open-codex-computer-use` 在常见交互中尽量避免抢占用户当前焦点和真实鼠标位置,同时把与官方 `computer-use` 的对比样本沉淀到仓库里,便于后续做数据分析和 eval。++## 范围++- 包含:+- 收集一组 `computer-use` 与 `open-codex-computer-use` 对同一目标 app 的对比调用样本,并保存 tool call 与结果。+- 调整 `get_app_state` 与动作型 tools 的实现,减少不必要的 `activate` 和全局 HID 事件。+- 为键盘与点击路径补充更合适的定向投递或 AX 优先策略。+- 同步测试、架构文档、质量说明和 history。+- 不包含:+- 复刻官方闭源实现的私有 overlay、宿主集成和完整后台事件路由。+- 在本轮里解决所有第三方 app 的兼容性差异。++## 背景++- 相关文档:+- `docs/ARCHITECTURE.md`+- `docs/REPO_COLLAB_GUIDE.md`+- `docs/QUALITY_SCORE.md`+- 相关代码路径:+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift`+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift`+- 已知约束:+- 当前实现大量依赖 `NSRunningApplication.activate` 与 `CGEvent.post(tap: .cghidEventTap)`。+- 鼠标类事件若继续走全局 HID,理论上仍会移动真实鼠标指针。+- SDK 可用能力里可确认 `CGEventPostToPid` 和 `AXUIElementPostKeyboardEvent`,但需要验证行为边界。++## 风险++- 风险:去掉 `activate` 后,某些依赖前台窗口的 snapshot 或动作可能拿不到期望元素。+- 缓解方式:snapshot 改成优先读取 app 自身窗口/焦点信息,不再强依赖前台态;必要时保留显式降级路径。+- 风险:`CGEventPostToPid` 或 `AXUIElementPostKeyboardEvent` 在部分 app 上行为和全局 HID 不一致。+- 缓解方式:先在 fixture 上做样本验证;代码中保留有边界说明的 fallback。+- 风险:为了减少副作用而过度牺牲兼容性。+- 缓解方式:优先改“能不抢焦点就不抢”,不是无条件禁止所有全局输入。++## 里程碑++1. 收集双工具对比样本并固定归档结构。+2. 实现非抢焦点优先的 snapshot / 输入策略。+3. 完成验证、文档同步和 history 留痕。++## 验证方式++- 命令:+- `swift test`+- `./scripts/run-tool-smoke-tests.sh`+- 手工检查:+- 对 fixture app 分别执行 `get_app_state`、`click`、`type_text`,记录调用前后前台 app 和鼠标坐标。+- 观测检查:+- 查看 `artifacts/tool-comparisons/20260417-focus-behavior/` 下的双目录样本是否完整。++## 进度记录++- [x] 里程碑 1+- [x] 里程碑 2+- [x] 里程碑 3++## 决策记录++- 2026-04-17:本轮优先解决“抢焦点/抢鼠标”这一类强副作用问题,并把对比数据直接落仓库,后续 eval 先基于真实样本迭代,而不是只靠口头描述。+- 2026-04-17:`get_app_state` 改为不再显式激活目标 app;键盘事件改走 `CGEvent.postToPid`,点击优先走 AX action / AX hit-test,只有 drag 或无法命中 AX 元素时才退回全局 HID。diff --git a/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.md b/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.mdnew file mode 100644index 0000000..2f7907a--- /dev/null+++ b/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.md@@ -0,0 +1,33 @@+## [2026-04-17 15:10] | Task: 收敛非抢焦点交互++### 🤖 Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5 / Codex`+* **Runtime**: `Codex CLI`++### 📥 User Query+> 现在已经有官方 `computer-use` 和仓库内 `open-codex-computer-use` 两套工具,要求并行对比效果,并把两边的 tool call / 结果分别保存到同一目录下的两个子目录里;当前最明显的问题是我们的实现会抢用户鼠标和焦点,希望围绕这一点做改进。++### 🛠 Changes Overview+**Scope:** `OpenCodexComputerUseKit`, `docs/`, `artifacts/`++**Key Actions:**+- **[非侵入优先输入链路]**: 去掉 `get_app_state` 的强制 app 激活,把 `type_text` / `press_key` 改成按 PID 定向投递键盘事件。+- **[点击策略修正]**: 修复 raw AX actions 被错误过滤的问题,并为 coordinate click 增加 AX hit-test 优先路径,只有命中失败才退回全局 HID。+- **[对比样本留档]**: 新增 `artifacts/tool-comparisons/20260417-focus-behavior/`,分别保存官方 `computer-use` 和仓库实现的调用样本与前后台观测。+- **[文档同步]**: 更新架构与质量文档,补执行计划与本次 history。++### 🧠 Design Intent (Why)+“抢焦点/抢鼠标”本质上是当前实现把太多路径都建立在 `activate + cghidEventTap` 上。这个改动的目标不是假装所有鼠标路径都能彻底无副作用,而是把读状态、键盘输入和大部分可反解到 AX 元素的点击先收敛到更温和的通道,把真正需要全局 HID 的场景显式缩到最小。++### 📁 Files Modified+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift`+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift`+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift`+- `docs/ARCHITECTURE.md`+- `docs/QUALITY_SCORE.md`+- `docs/exec-plans/completed/20260417-nonintrusive-computer-use.md`+- `artifacts/tool-comparisons/20260417-focus-behavior/README.md`+- `artifacts/tool-comparisons/20260417-focus-behavior/computer-use/get_app_state-activity-monitor.json`+- `artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/get_app_state-activity-monitor.json`+- `artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/click-activity-monitor-coordinate.json`diff --git a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swiftindex 8670229..b8849c3 100644--- a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift@@ -48,9 +48,6 @@ enum SnapshotBuilder {throw ComputerUseError.permissionDenied("Accessibility permission is required. Run `OpenCodexComputerUse doctor` and grant access to the host terminal or app.")}- app.runningApplication.activate()- Thread.sleep(forTimeInterval: 0.25)-let appElement = AXUIElementCreateApplication(app.pid)let systemWide = AXUIElementCreateSystemWide()let focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)@@ -310,8 +307,10 @@ private struct TreeRenderer {let value = sanitizedValue(of: root)let axIdentifier = stringValue(of: root, attribute: kAXIdentifierAttribute)let traits = summarizeTraits(of: root)- let actions = (copyActions(root) ?? []).filter { $0 != kAXPressAction as String }- let prettyActions = actions.map(prettyActionName(_:))+ let actions = copyActions(root) ?? []+ let prettyActions = actions+ .filter { $0 != kAXPressAction as String }+ .map(prettyActionName(_:))let localFrame = resolveLocalFrame(of: root, windowBounds: context.windowBounds)let roleText = humanizeRole(role: role, subrole: subrole)diff --git a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swiftindex 319a8d6..d8262d9 100644--- a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift+++ b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift@@ -41,24 +41,27 @@ public final class ComputerUseService {return try refreshSnapshot(for: query).renderedText}- InputSimulation.activate(snapshot.app)-if let elementIndex {let record = try lookupElement(snapshot: snapshot, index: elementIndex)- if button == .left,- clickCount == 1,- let element = record.element,- record.rawActions.contains(kAXPressAction as String),- AXUIElementPerformAction(element, kAXPressAction as CFString) == .success {+ if try performPreferredClick(on: record, button: button, clickCount: clickCount) {Thread.sleep(forTimeInterval: 0.15)} else if let point = try globalPoint(for: record, snapshot: snapshot) {- try InputSimulation.click(at: point, button: button, clickCount: clickCount)+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)+ try InputSimulation.clickGlobally(at: point, button: button, clickCount: clickCount)} else {throw ComputerUseError.stateUnavailable("element \(elementIndex) has no clickable frame")}} else if let x, let y {- try InputSimulation.click(at: try screenshotToGlobalPoint(snapshot: snapshot, x: x, y: y), button: button, clickCount: clickCount)+ let point = CGPoint(x: x, y: y)++ if let record = try hitTestElement(at: point, in: snapshot) ?? bestElement(containing: point, in: snapshot),+ try performPreferredClick(on: record, button: button, clickCount: clickCount) {+ Thread.sleep(forTimeInterval: 0.15)+ } else {+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)+ try InputSimulation.clickGlobally(at: try screenshotToGlobalPoint(snapshot: snapshot, x: x, y: y), button: button, clickCount: clickCount)+ }} else {throw ComputerUseError.invalidArguments("click requires either element_index or x/y")}@@ -75,7 +78,7 @@ public final class ComputerUseService {throw ComputerUseError.invalidArguments("fixture mode only supports the Raise secondary action")}- InputSimulation.activate(snapshot.app)+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)return try refreshSnapshot(for: query).renderedText}@@ -114,15 +117,14 @@ public final class ComputerUseService {return try refreshSnapshot(for: query).renderedText}- InputSimulation.activate(snapshot.app)-if let rawAction = record.rawActions.first(where: { $0.caseInsensitiveCompare("AXScroll\(normalized.capitalized)ByPage") == .orderedSame }), let element = record.element {for _ in 0..<max(pages, 1) {_ = AXUIElementPerformAction(element, rawAction as CFString)Thread.sleep(forTimeInterval: 0.05)}} else if let point = try globalPoint(for: record, snapshot: snapshot) {- try InputSimulation.scroll(at: point, direction: normalized, pages: pages)+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)+ try InputSimulation.scrollGlobally(at: point, direction: normalized, pages: pages)} else {throw ComputerUseError.stateUnavailable("element \(elementIndex) has no scrollable frame")}@@ -138,8 +140,8 @@ public final class ComputerUseService {return try refreshSnapshot(for: query).renderedText}- InputSimulation.activate(snapshot.app)- try InputSimulation.drag(+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)+ try InputSimulation.dragGlobally(from: try screenshotToGlobalPoint(snapshot: snapshot, x: fromX, y: fromY),to: try screenshotToGlobalPoint(snapshot: snapshot, x: toX, y: toY))@@ -154,8 +156,7 @@ public final class ComputerUseService {return try refreshSnapshot(for: query).renderedText}- InputSimulation.activate(snapshot.app)- try InputSimulation.typeText(text)+ try InputSimulation.typeText(text, pid: snapshot.app.pid)return try refreshSnapshot(for: query).renderedText}@@ -167,8 +168,7 @@ public final class ComputerUseService {return try refreshSnapshot(for: query).renderedText}- InputSimulation.activate(snapshot.app)- try InputSimulation.pressKey(key)+ try InputSimulation.pressKey(key, pid: snapshot.app.pid)return try refreshSnapshot(for: query).renderedText}@@ -245,6 +245,173 @@ public final class ComputerUseService {return nil}+ private func performPreferredClick(on record: ElementRecord, button: MouseButtonKind, clickCount: Int) throws -> Bool {+ guard let element = record.element, clickCount == 1 else {+ return false+ }++ switch button {+ case .left:+ if try performAction(named: kAXPressAction as String, on: element, availableActions: record.rawActions) {+ return true+ }++ if try focus(element: element) {+ return true+ }++ if try performAction(named: kAXConfirmAction as String, on: element, availableActions: record.rawActions) {+ return true+ }+ case .right:+ if try performAction(named: kAXShowMenuAction as String, on: element, availableActions: record.rawActions) {+ return true+ }+ case .middle:+ break+ }++ return false+ }++ private func performAction(named action: String, on element: AXUIElement, availableActions: [String]) throws -> Bool {+ guard availableActions.contains(where: { $0.caseInsensitiveCompare(action) == .orderedSame }) else {+ return false+ }++ let result = AXUIElementPerformAction(element, action as CFString)+ switch result {+ case .success:+ return true+ case .actionUnsupported, .cannotComplete, .noValue:+ return false+ default:+ throw ComputerUseError.message("AXUIElementPerformAction(\(action)) failed with \(result.rawValue)")+ }+ }++ private func focus(element: AXUIElement) throws -> Bool {+ guard isSettable(element: element, attribute: kAXFocusedAttribute) else {+ return false+ }++ let result = AXUIElementSetAttributeValue(element, kAXFocusedAttribute as CFString, kCFBooleanTrue)+ switch result {+ case .success:+ return true+ case .attributeUnsupported, .cannotComplete, .noValue:+ return false+ default:+ throw ComputerUseError.message("AXUIElementSetAttributeValue(\(kAXFocusedAttribute)) failed with \(result.rawValue)")+ }+ }++ private func isSettable(element: AXUIElement, attribute: String) -> Bool {+ var settable: DarwinBoolean = false+ let result = AXUIElementIsAttributeSettable(element, attribute as CFString, &settable)+ return result == .success && settable.boolValue+ }++ private func bestElement(containing point: CGPoint, in snapshot: AppSnapshot) -> ElementRecord? {+ snapshot.elements.values+ .filter { $0.localFrame?.contains(point) ?? false }+ .sorted { lhs, rhs in+ let lhsPriority = clickPriority(for: lhs)+ let rhsPriority = clickPriority(for: rhs)+ if lhsPriority != rhsPriority {+ return lhsPriority < rhsPriority+ }++ return frameArea(of: lhs) < frameArea(of: rhs)+ }+ .first+ }++ private func hitTestElement(at point: CGPoint, in snapshot: AppSnapshot) throws -> ElementRecord? {+ let appElement = AXUIElementCreateApplication(snapshot.app.pid)+ let globalPoint = try screenshotToGlobalPoint(snapshot: snapshot, x: Double(point.x), y: Double(point.y))+ var hitElement: AXUIElement?+ let result = AXUIElementCopyElementAtPosition(appElement, Float(globalPoint.x), Float(globalPoint.y), &hitElement)+ guard result == .success, let hitElement else {+ return nil+ }++ let rawActions = copyActions(for: hitElement) ?? []+ return ElementRecord(+ index: -1,+ identifier: nil,+ element: hitElement,+ localFrame: localFrame(of: hitElement, windowBounds: snapshot.windowBounds),+ rawActions: rawActions,+ prettyActions: rawActions+ )+ }++ private func clickPriority(for record: ElementRecord) -> Int {+ if record.rawActions.contains(where: {+ $0.caseInsensitiveCompare(kAXPressAction as String) == .orderedSame ||+ $0.caseInsensitiveCompare(kAXConfirmAction as String) == .orderedSame ||+ $0.caseInsensitiveCompare(kAXShowMenuAction as String) == .orderedSame+ }) {+ return 0+ }++ if let element = record.element, isSettable(element: element, attribute: kAXFocusedAttribute) {+ return 1+ }++ return 2+ }++ private func frameArea(of record: ElementRecord) -> CGFloat {+ guard let frame = record.localFrame else {+ return .greatestFiniteMagnitude+ }++ return frame.width * frame.height+ }++ private func copyActions(for element: AXUIElement) -> [String]? {+ var actions: CFArray?+ let result = AXUIElementCopyActionNames(element, &actions)+ guard result == .success else {+ return nil+ }++ return actions as? [String]+ }++ private func localFrame(of element: AXUIElement, windowBounds: CGRect?) -> CGRect? {+ var positionValue: CFTypeRef?+ var sizeValue: CFTypeRef?+ let positionResult = AXUIElementCopyAttributeValue(element, kAXPositionAttribute as CFString, &positionValue)+ let sizeResult = AXUIElementCopyAttributeValue(element, kAXSizeAttribute as CFString, &sizeValue)++ guard+ positionResult == .success,+ sizeResult == .success,+ let positionValue,+ let sizeValue+ else {+ return nil+ }++ let positionAXValue = positionValue as! AXValue+ let sizeAXValue = sizeValue as! AXValue+ var position = CGPoint.zero+ var size = CGSize.zero+ guard AXValueGetValue(positionAXValue, .cgPoint, &position), AXValueGetValue(sizeAXValue, .cgSize, &size) else {+ return nil+ }++ let frame = CGRect(origin: position, size: size)+ guard let windowBounds else {+ return frame+ }++ return windowRelativeFrame(elementFrame: frame, windowBounds: windowBounds)+ }+private func globalPoint(for record: ElementRecord, snapshot: AppSnapshot) throws -> CGPoint? {guard let frame = record.localFrame else {return nildiff --git a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swiftindex b75a749..5c71d6b 100644--- a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift+++ b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift@@ -43,12 +43,12 @@ enum MouseButtonKind: String {}enum InputSimulation {- static func activate(_ app: RunningAppDescriptor) {- app.runningApplication.activate(options: [.activateIgnoringOtherApps])+ static func bringAppToFrontForGlobalPointerInput(_ app: RunningAppDescriptor) {+ app.runningApplication.activate()Thread.sleep(forTimeInterval: 0.25)}- static func click(at point: CGPoint, button: MouseButtonKind, clickCount: Int) throws {+ static func clickGlobally(at point: CGPoint, button: MouseButtonKind, clickCount: Int) throws {guard let source = CGEventSource(stateID: .hidSystemState) else {throw ComputerUseError.message("Failed to create HID event source.")}@@ -60,7 +60,7 @@ enum InputSimulation {}}- static func scroll(at point: CGPoint, direction: String, pages: Int) throws {+ static func scrollGlobally(at point: CGPoint, direction: String, pages: Int) throws {guard let event = CGEvent(scrollWheelEvent2Source: nil, units: .line, wheelCount: 2, wheel1: wheel1(direction: direction, pages: pages), wheel2: wheel2(direction: direction, pages: pages), wheel3: 0) else {throw ComputerUseError.message("Failed to create scroll event.")}@@ -70,7 +70,7 @@ enum InputSimulation {Thread.sleep(forTimeInterval: 0.1)}- static func drag(from start: CGPoint, to end: CGPoint) throws {+ static func dragGlobally(from start: CGPoint, to end: CGPoint) throws {guard let source = CGEventSource(stateID: .hidSystemState) else {throw ComputerUseError.message("Failed to create HID event source.")}@@ -90,7 +90,7 @@ enum InputSimulation {try postMouseEvent(type: .leftMouseUp, source: source, point: end, button: .left, clickState: 1)}- static func typeText(_ text: String) throws {+ static func typeText(_ text: String, pid: pid_t) throws {for character in text.utf16 {var mutableCharacter = characterguard let down = CGEvent(keyboardEventSource: nil, virtualKey: 0, keyDown: true),@@ -100,13 +100,13 @@ enum InputSimulation {down.keyboardSetUnicodeString(stringLength: 1, unicodeString: &mutableCharacter)up.keyboardSetUnicodeString(stringLength: 1, unicodeString: &mutableCharacter)- down.post(tap: .cghidEventTap)- up.post(tap: .cghidEventTap)+ down.postToPid(pid)+ up.postToPid(pid)Thread.sleep(forTimeInterval: 0.02)}}- static func pressKey(_ specification: String) throws {+ static func pressKey(_ specification: String, pid: pid_t) throws {let parsed = try KeyPressParser.parse(specification)var activeFlags: CGEventFlags = []@@ -117,7 +117,7 @@ enum InputSimulation {activeFlags.insert(modifier.flag)event.flags = activeFlags- event.post(tap: .cghidEventTap)+ event.postToPid(pid)}guard let keyDown = CGEvent(keyboardEventSource: nil, virtualKey: parsed.keyCode, keyDown: true),@@ -127,8 +127,8 @@ enum InputSimulation {keyDown.flags = activeFlagskeyUp.flags = activeFlags- keyDown.post(tap: .cghidEventTap)- keyUp.post(tap: .cghidEventTap)+ keyDown.postToPid(pid)+ keyUp.postToPid(pid)for modifier in parsed.modifiers.reversed() {guard let event = CGEvent(keyboardEventSource: nil, virtualKey: modifier.keyCode, keyDown: false) else {@@ -136,7 +136,7 @@ enum InputSimulation {}event.flags = activeFlags- event.post(tap: .cghidEventTap)+ event.postToPid(pid)activeFlags.remove(modifier.flag)}