Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Reduce intrusive focus stealing in computer use tools

Leo38cbf353ed04
diff --git a/artifacts/tool-comparisons/20260417-focus-behavior/README.md b/artifacts/tool-comparisons/20260417-focus-behavior/README.md
new file mode 100644
index 0000000..3ecbfe2
--- /dev/null
+++ b/artifacts/tool-comparisons/20260417-focus-behavior/README.md
@@ -0,0 +1,28 @@
+# 2026-04-17 Focus Behavior Comparison
+
+这组样本用于对比官方 `computer-use` 与仓库内 `open-codex-computer-use` 在“只读取状态”和“坐标点击”场景下对用户前台焦点/鼠标的影响。
+
+## 目录
+
+- `computer-use/`
+ - 来自已连接官方 `computer-use` MCP tool 的样本。
+- `open-codex-computer-use/`
+ - 来自仓库当前实现的样本。
+ - `get_app_state` 通过本地打包后的 `dist/OpenCodexComputerUse.app` 走 JSON-RPC `tools/call` 采集。
+ - `click` 同样通过本地 JSON-RPC 采集。
+
+## 方法
+
+- 目标 app:`Activity Monitor`
+- 对照前台 app:`iTerm2`
+- 观测项:
+ - tool request
+ - tool result 摘要
+ - 调用前后的 frontmost app
+ - 调用前后的鼠标坐标
+
+## 已知限制
+
+- 官方 `computer-use` 当前不能直接识别仓库里的裸可执行 fixture,因此对比目标改为双方都能解析的系统 app。
+- 鼠标坐标会受到用户实时移动影响,因此这里只把它当作辅助观测;“是否切走前台 app”是更稳定的对比信号。
+- `open-codex-computer-use` 的坐标点击现在会先做 AX hit-test;如果命中失败,则仍会降级到全局 HID。
diff --git a/artifacts/tool-comparisons/20260417-focus-behavior/computer-use/get_app_state-activity-monitor.json b/artifacts/tool-comparisons/20260417-focus-behavior/computer-use/get_app_state-activity-monitor.json
new file mode 100644
index 0000000..8ffdc23
--- /dev/null
+++ b/artifacts/tool-comparisons/20260417-focus-behavior/computer-use/get_app_state-activity-monitor.json
@@ -0,0 +1,34 @@
+{
+ "captured_at": "2026-04-17",
+ "tool": "get_app_state",
+ "request": {
+ "app": "Activity Monitor"
+ },
+ "transport": "assistant tool call: mcp__computer_use__.get_app_state",
+ "observations": {
+ "before": {
+ "frontmost_app": "iTerm2",
+ "mouse_location": "1701,702"
+ },
+ "after": {
+ "frontmost_app": "iTerm2",
+ "mouse_location": "1724,685"
+ }
+ },
+ "result_summary": {
+ "app": "com.apple.ActivityMonitor (pid 988)",
+ "window": "Activity Monitor – All Processes",
+ "focused_ui_element": "0 standard window",
+ "screenshot_returned": true
+ },
+ "result_excerpt": [
+ "Computer Use state (CUA App Version: 750)",
+ "<app_state>",
+ "App=com.apple.ActivityMonitor (pid 988)",
+ "Window: \"Activity Monitor\", App: Activity Monitor.",
+ "0 standard window Activity Monitor – All Processes, Secondary Actions: Raise",
+ "38 search text field (settable, string) Helper",
+ "The focused UI element is 0 standard window.",
+ "</app_state>"
+ ]
+}
diff --git a/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/click-activity-monitor-coordinate.json b/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/click-activity-monitor-coordinate.json
new file mode 100644
index 0000000..8b9e0cd
--- /dev/null
+++ b/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/click-activity-monitor-coordinate.json
@@ -0,0 +1,29 @@
+{
+ "captured_at": "2026-04-17",
+ "tool": "click",
+ "request": {
+ "app": "Activity Monitor",
+ "x": 850,
+ "y": 18
+ },
+ "transport": "local JSON-RPC to dist/OpenCodexComputerUse.app",
+ "observations": {
+ "before": {
+ "frontmost_app": "iTerm2",
+ "mouse_location": "1535,694"
+ },
+ "after": {
+ "frontmost_app": "iTerm2",
+ "mouse_location": "1984,807"
+ }
+ },
+ "result_summary": {
+ "app": "com.apple.ActivityMonitor (pid 988)",
+ "window": "Activity Monitor – All Processes",
+ "is_error": false
+ },
+ "notes": [
+ "该样本用于验证坐标点击不会默认抢前台焦点。",
+ "当前结果至少证明这次调用结束后前台仍是 iTerm2;命中的具体控件仍需后续补更精确的目标样本继续评估。"
+ ]
+}
diff --git a/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/get_app_state-activity-monitor.json b/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/get_app_state-activity-monitor.json
new file mode 100644
index 0000000..0df83c7
--- /dev/null
+++ b/artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/get_app_state-activity-monitor.json
@@ -0,0 +1,29 @@
+{
+ "captured_at": "2026-04-17",
+ "tool": "get_app_state",
+ "request": {
+ "app": "Activity Monitor"
+ },
+ "transport": "local JSON-RPC to dist/OpenCodexComputerUse.app",
+ "observations": {
+ "before": {
+ "frontmost_app": "iTerm2",
+ "mouse_location": "1408,675"
+ },
+ "after": {
+ "frontmost_app": "iTerm2",
+ "mouse_location": "1537,938"
+ }
+ },
+ "result_summary": {
+ "app": "com.apple.ActivityMonitor (pid 988)",
+ "window": "Activity Monitor – All Processes",
+ "bounds": "x=809, y=685, w=960, h=640",
+ "focused_ui_element": "0 standard window",
+ "screenshot_path_returned": true
+ },
+ "notes": [
+ "这次样本来自修复后的 dist 制品,而不是已经断开的旧 MCP 会话。",
+ "对比结果显示 get_app_state 不再为了读状态而切走前台 iTerm2。"
+ ]
+}
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 75f8545..6f1132d 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -47,12 +47,14 @@
- `ComputerUseService` 负责把 MCP tool 请求映射到本地能力。
- `list_apps` 通过 `NSWorkspace` 枚举运行中的 app。
-- `get_app_state` 优先走真实 AX / 窗口截图;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。
+- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。
- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。
-- 动作型 tools 对普通 app 走:
+- 动作型 tools 对普通 app 采用“非侵入优先,HID 兜底”策略:
- `AXUIElementPerformAction`
- `AXUIElementSetAttributeValue`
- - `CGEvent` 键鼠事件
+ - `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素
+ - `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台
+ - 只有 drag 或无法命中 AX 元素的鼠标路径,才退回全局 `CGEvent` 键鼠事件并显式前置目标 app
### 4. Fixture Bridge
@@ -72,6 +74,7 @@
- 单元测试:`swift test`
- 端到端 smoke:`./scripts/run-tool-smoke-tests.sh`
- app 打包:`./scripts/build-open-codex-app.sh debug`
+- 对比样本:`artifacts/tool-comparisons/20260417-focus-behavior/`
- 手工诊断:
- `.build/debug/OpenCodexComputerUse doctor`
- `.build/debug/OpenCodexComputerUse snapshot <app>`
diff --git a/docs/QUALITY_SCORE.md b/docs/QUALITY_SCORE.md
index 6d2660a..8789028 100644
--- a/docs/QUALITY_SCORE.md
+++ b/docs/QUALITY_SCORE.md
@@ -13,6 +13,6 @@
| --- | --- | --- | --- |
| 产品面 | B | 已经有 Swift 本地 `computer-use` MCP server、默认 app 模式权限引导,以及 9 个 tools 的可运行实现。 | 继续收敛权限 UI 细节、app allowlist 和更清晰的用户错误提示。 |
| 架构文档 | B | 顶层结构、fixture bridge、app 模式和验证路径已经落文档。 | 后续补 release artifact、code signing / notarization 和 host 集成方式。 |
-| 测试 | B | `swift test` + smoke suite 已覆盖 9 个 tools 的回归,并补了真实 `doctor` / `snapshot` spot check。 | 增加更多普通 app 的手工或录制回归,减少只依赖 fixture。 |
-| 可观测性 | C | 已有 `doctor`、`snapshot` 和 smoke 输出,但还没有结构化日志。 | 补统一日志级别、失败上下文和 release artifact 里的诊断信息。 |
+| 测试 | B | `swift test` + smoke suite 已覆盖 9 个 tools 的回归,并新增了针对“前台焦点是否被抢占”的手工对比样本沉淀。 | 增加更多普通 app 的录制回归,减少只依赖 fixture 和一次性手工检查。 |
+| 可观测性 | C | 已有 `doctor`、`snapshot`、smoke 输出,以及一组仓库内留档的 `computer-use` / `open-codex-computer-use` 对比样本。 | 补统一日志级别、失败上下文和 release artifact 里的诊断信息,把一次性样本收敛成可重复采集流程。 |
| 安全 | B | 已明确本地-only、权限边界和 fixture test bridge 的作用域。 | 增加 app allowlist / session approval,避免对任意 app 默认开放。 |
diff --git a/docs/exec-plans/completed/20260417-nonintrusive-computer-use.md b/docs/exec-plans/completed/20260417-nonintrusive-computer-use.md
new file mode 100644
index 0000000..2ccc9d0
--- /dev/null
+++ b/docs/exec-plans/completed/20260417-nonintrusive-computer-use.md
@@ -0,0 +1,67 @@
+# 非抢焦点 computer-use 交互改造
+
+## 目标
+
+让 `open-codex-computer-use` 在常见交互中尽量避免抢占用户当前焦点和真实鼠标位置,同时把与官方 `computer-use` 的对比样本沉淀到仓库里,便于后续做数据分析和 eval。
+
+## 范围
+
+- 包含:
+- 收集一组 `computer-use` 与 `open-codex-computer-use` 对同一目标 app 的对比调用样本,并保存 tool call 与结果。
+- 调整 `get_app_state` 与动作型 tools 的实现,减少不必要的 `activate` 和全局 HID 事件。
+- 为键盘与点击路径补充更合适的定向投递或 AX 优先策略。
+- 同步测试、架构文档、质量说明和 history。
+- 不包含:
+- 复刻官方闭源实现的私有 overlay、宿主集成和完整后台事件路由。
+- 在本轮里解决所有第三方 app 的兼容性差异。
+
+## 背景
+
+- 相关文档:
+- `docs/ARCHITECTURE.md`
+- `docs/REPO_COLLAB_GUIDE.md`
+- `docs/QUALITY_SCORE.md`
+- 相关代码路径:
+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift`
+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift`
+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift`
+- 已知约束:
+- 当前实现大量依赖 `NSRunningApplication.activate` 与 `CGEvent.post(tap: .cghidEventTap)`。
+- 鼠标类事件若继续走全局 HID,理论上仍会移动真实鼠标指针。
+- SDK 可用能力里可确认 `CGEventPostToPid` 和 `AXUIElementPostKeyboardEvent`,但需要验证行为边界。
+
+## 风险
+
+- 风险:去掉 `activate` 后,某些依赖前台窗口的 snapshot 或动作可能拿不到期望元素。
+- 缓解方式:snapshot 改成优先读取 app 自身窗口/焦点信息,不再强依赖前台态;必要时保留显式降级路径。
+- 风险:`CGEventPostToPid` 或 `AXUIElementPostKeyboardEvent` 在部分 app 上行为和全局 HID 不一致。
+- 缓解方式:先在 fixture 上做样本验证;代码中保留有边界说明的 fallback。
+- 风险:为了减少副作用而过度牺牲兼容性。
+- 缓解方式:优先改“能不抢焦点就不抢”,不是无条件禁止所有全局输入。
+
+## 里程碑
+
+1. 收集双工具对比样本并固定归档结构。
+2. 实现非抢焦点优先的 snapshot / 输入策略。
+3. 完成验证、文档同步和 history 留痕。
+
+## 验证方式
+
+- 命令:
+- `swift test`
+- `./scripts/run-tool-smoke-tests.sh`
+- 手工检查:
+- 对 fixture app 分别执行 `get_app_state`、`click`、`type_text`,记录调用前后前台 app 和鼠标坐标。
+- 观测检查:
+- 查看 `artifacts/tool-comparisons/20260417-focus-behavior/` 下的双目录样本是否完整。
+
+## 进度记录
+
+- [x] 里程碑 1
+- [x] 里程碑 2
+- [x] 里程碑 3
+
+## 决策记录
+
+- 2026-04-17:本轮优先解决“抢焦点/抢鼠标”这一类强副作用问题,并把对比数据直接落仓库,后续 eval 先基于真实样本迭代,而不是只靠口头描述。
+- 2026-04-17:`get_app_state` 改为不再显式激活目标 app;键盘事件改走 `CGEvent.postToPid`,点击优先走 AX action / AX hit-test,只有 drag 或无法命中 AX 元素时才退回全局 HID。
diff --git a/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.md b/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.md
new file mode 100644
index 0000000..2f7907a
--- /dev/null
+++ b/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.md
@@ -0,0 +1,33 @@
+## [2026-04-17 15:10] | Task: 收敛非抢焦点交互
+
+### 🤖 Execution Context
+* **Agent ID**: `Codex`
+* **Base Model**: `GPT-5 / Codex`
+* **Runtime**: `Codex CLI`
+
+### 📥 User Query
+> 现在已经有官方 `computer-use` 和仓库内 `open-codex-computer-use` 两套工具,要求并行对比效果,并把两边的 tool call / 结果分别保存到同一目录下的两个子目录里;当前最明显的问题是我们的实现会抢用户鼠标和焦点,希望围绕这一点做改进。
+
+### 🛠 Changes Overview
+**Scope:** `OpenCodexComputerUseKit`, `docs/`, `artifacts/`
+
+**Key Actions:**
+- **[非侵入优先输入链路]**: 去掉 `get_app_state` 的强制 app 激活,把 `type_text` / `press_key` 改成按 PID 定向投递键盘事件。
+- **[点击策略修正]**: 修复 raw AX actions 被错误过滤的问题,并为 coordinate click 增加 AX hit-test 优先路径,只有命中失败才退回全局 HID。
+- **[对比样本留档]**: 新增 `artifacts/tool-comparisons/20260417-focus-behavior/`,分别保存官方 `computer-use` 和仓库实现的调用样本与前后台观测。
+- **[文档同步]**: 更新架构与质量文档,补执行计划与本次 history。
+
+### 🧠 Design Intent (Why)
+“抢焦点/抢鼠标”本质上是当前实现把太多路径都建立在 `activate + cghidEventTap` 上。这个改动的目标不是假装所有鼠标路径都能彻底无副作用,而是把读状态、键盘输入和大部分可反解到 AX 元素的点击先收敛到更温和的通道,把真正需要全局 HID 的场景显式缩到最小。
+
+### 📁 Files Modified
+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift`
+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift`
+- `packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift`
+- `docs/ARCHITECTURE.md`
+- `docs/QUALITY_SCORE.md`
+- `docs/exec-plans/completed/20260417-nonintrusive-computer-use.md`
+- `artifacts/tool-comparisons/20260417-focus-behavior/README.md`
+- `artifacts/tool-comparisons/20260417-focus-behavior/computer-use/get_app_state-activity-monitor.json`
+- `artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/get_app_state-activity-monitor.json`
+- `artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/click-activity-monitor-coordinate.json`
diff --git a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift
index 8670229..b8849c3 100644
--- a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/AccessibilitySnapshot.swift
@@ -48,9 +48,6 @@ enum SnapshotBuilder {
throw ComputerUseError.permissionDenied("Accessibility permission is required. Run `OpenCodexComputerUse doctor` and grant access to the host terminal or app.")
}
- app.runningApplication.activate()
- Thread.sleep(forTimeInterval: 0.25)
-
let appElement = AXUIElementCreateApplication(app.pid)
let systemWide = AXUIElementCreateSystemWide()
let focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)
@@ -310,8 +307,10 @@ private struct TreeRenderer {
let value = sanitizedValue(of: root)
let axIdentifier = stringValue(of: root, attribute: kAXIdentifierAttribute)
let traits = summarizeTraits(of: root)
- let actions = (copyActions(root) ?? []).filter { $0 != kAXPressAction as String }
- let prettyActions = actions.map(prettyActionName(_:))
+ let actions = copyActions(root) ?? []
+ let prettyActions = actions
+ .filter { $0 != kAXPressAction as String }
+ .map(prettyActionName(_:))
let localFrame = resolveLocalFrame(of: root, windowBounds: context.windowBounds)
let roleText = humanizeRole(role: role, subrole: subrole)
diff --git a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift
index 319a8d6..d8262d9 100644
--- a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/ComputerUseService.swift
@@ -41,24 +41,27 @@ public final class ComputerUseService {
return try refreshSnapshot(for: query).renderedText
}
- InputSimulation.activate(snapshot.app)
-
if let elementIndex {
let record = try lookupElement(snapshot: snapshot, index: elementIndex)
- if button == .left,
- clickCount == 1,
- let element = record.element,
- record.rawActions.contains(kAXPressAction as String),
- AXUIElementPerformAction(element, kAXPressAction as CFString) == .success {
+ if try performPreferredClick(on: record, button: button, clickCount: clickCount) {
Thread.sleep(forTimeInterval: 0.15)
} else if let point = try globalPoint(for: record, snapshot: snapshot) {
- try InputSimulation.click(at: point, button: button, clickCount: clickCount)
+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)
+ try InputSimulation.clickGlobally(at: point, button: button, clickCount: clickCount)
} else {
throw ComputerUseError.stateUnavailable("element \(elementIndex) has no clickable frame")
}
} else if let x, let y {
- try InputSimulation.click(at: try screenshotToGlobalPoint(snapshot: snapshot, x: x, y: y), button: button, clickCount: clickCount)
+ let point = CGPoint(x: x, y: y)
+
+ if let record = try hitTestElement(at: point, in: snapshot) ?? bestElement(containing: point, in: snapshot),
+ try performPreferredClick(on: record, button: button, clickCount: clickCount) {
+ Thread.sleep(forTimeInterval: 0.15)
+ } else {
+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)
+ try InputSimulation.clickGlobally(at: try screenshotToGlobalPoint(snapshot: snapshot, x: x, y: y), button: button, clickCount: clickCount)
+ }
} else {
throw ComputerUseError.invalidArguments("click requires either element_index or x/y")
}
@@ -75,7 +78,7 @@ public final class ComputerUseService {
throw ComputerUseError.invalidArguments("fixture mode only supports the Raise secondary action")
}
- InputSimulation.activate(snapshot.app)
+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)
return try refreshSnapshot(for: query).renderedText
}
@@ -114,15 +117,14 @@ public final class ComputerUseService {
return try refreshSnapshot(for: query).renderedText
}
- InputSimulation.activate(snapshot.app)
-
if let rawAction = record.rawActions.first(where: { $0.caseInsensitiveCompare("AXScroll\(normalized.capitalized)ByPage") == .orderedSame }), let element = record.element {
for _ in 0..<max(pages, 1) {
_ = AXUIElementPerformAction(element, rawAction as CFString)
Thread.sleep(forTimeInterval: 0.05)
}
} else if let point = try globalPoint(for: record, snapshot: snapshot) {
- try InputSimulation.scroll(at: point, direction: normalized, pages: pages)
+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)
+ try InputSimulation.scrollGlobally(at: point, direction: normalized, pages: pages)
} else {
throw ComputerUseError.stateUnavailable("element \(elementIndex) has no scrollable frame")
}
@@ -138,8 +140,8 @@ public final class ComputerUseService {
return try refreshSnapshot(for: query).renderedText
}
- InputSimulation.activate(snapshot.app)
- try InputSimulation.drag(
+ InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)
+ try InputSimulation.dragGlobally(
from: try screenshotToGlobalPoint(snapshot: snapshot, x: fromX, y: fromY),
to: try screenshotToGlobalPoint(snapshot: snapshot, x: toX, y: toY)
)
@@ -154,8 +156,7 @@ public final class ComputerUseService {
return try refreshSnapshot(for: query).renderedText
}
- InputSimulation.activate(snapshot.app)
- try InputSimulation.typeText(text)
+ try InputSimulation.typeText(text, pid: snapshot.app.pid)
return try refreshSnapshot(for: query).renderedText
}
@@ -167,8 +168,7 @@ public final class ComputerUseService {
return try refreshSnapshot(for: query).renderedText
}
- InputSimulation.activate(snapshot.app)
- try InputSimulation.pressKey(key)
+ try InputSimulation.pressKey(key, pid: snapshot.app.pid)
return try refreshSnapshot(for: query).renderedText
}
@@ -245,6 +245,173 @@ public final class ComputerUseService {
return nil
}
+ private func performPreferredClick(on record: ElementRecord, button: MouseButtonKind, clickCount: Int) throws -> Bool {
+ guard let element = record.element, clickCount == 1 else {
+ return false
+ }
+
+ switch button {
+ case .left:
+ if try performAction(named: kAXPressAction as String, on: element, availableActions: record.rawActions) {
+ return true
+ }
+
+ if try focus(element: element) {
+ return true
+ }
+
+ if try performAction(named: kAXConfirmAction as String, on: element, availableActions: record.rawActions) {
+ return true
+ }
+ case .right:
+ if try performAction(named: kAXShowMenuAction as String, on: element, availableActions: record.rawActions) {
+ return true
+ }
+ case .middle:
+ break
+ }
+
+ return false
+ }
+
+ private func performAction(named action: String, on element: AXUIElement, availableActions: [String]) throws -> Bool {
+ guard availableActions.contains(where: { $0.caseInsensitiveCompare(action) == .orderedSame }) else {
+ return false
+ }
+
+ let result = AXUIElementPerformAction(element, action as CFString)
+ switch result {
+ case .success:
+ return true
+ case .actionUnsupported, .cannotComplete, .noValue:
+ return false
+ default:
+ throw ComputerUseError.message("AXUIElementPerformAction(\(action)) failed with \(result.rawValue)")
+ }
+ }
+
+ private func focus(element: AXUIElement) throws -> Bool {
+ guard isSettable(element: element, attribute: kAXFocusedAttribute) else {
+ return false
+ }
+
+ let result = AXUIElementSetAttributeValue(element, kAXFocusedAttribute as CFString, kCFBooleanTrue)
+ switch result {
+ case .success:
+ return true
+ case .attributeUnsupported, .cannotComplete, .noValue:
+ return false
+ default:
+ throw ComputerUseError.message("AXUIElementSetAttributeValue(\(kAXFocusedAttribute)) failed with \(result.rawValue)")
+ }
+ }
+
+ private func isSettable(element: AXUIElement, attribute: String) -> Bool {
+ var settable: DarwinBoolean = false
+ let result = AXUIElementIsAttributeSettable(element, attribute as CFString, &settable)
+ return result == .success && settable.boolValue
+ }
+
+ private func bestElement(containing point: CGPoint, in snapshot: AppSnapshot) -> ElementRecord? {
+ snapshot.elements.values
+ .filter { $0.localFrame?.contains(point) ?? false }
+ .sorted { lhs, rhs in
+ let lhsPriority = clickPriority(for: lhs)
+ let rhsPriority = clickPriority(for: rhs)
+ if lhsPriority != rhsPriority {
+ return lhsPriority < rhsPriority
+ }
+
+ return frameArea(of: lhs) < frameArea(of: rhs)
+ }
+ .first
+ }
+
+ private func hitTestElement(at point: CGPoint, in snapshot: AppSnapshot) throws -> ElementRecord? {
+ let appElement = AXUIElementCreateApplication(snapshot.app.pid)
+ let globalPoint = try screenshotToGlobalPoint(snapshot: snapshot, x: Double(point.x), y: Double(point.y))
+ var hitElement: AXUIElement?
+ let result = AXUIElementCopyElementAtPosition(appElement, Float(globalPoint.x), Float(globalPoint.y), &hitElement)
+ guard result == .success, let hitElement else {
+ return nil
+ }
+
+ let rawActions = copyActions(for: hitElement) ?? []
+ return ElementRecord(
+ index: -1,
+ identifier: nil,
+ element: hitElement,
+ localFrame: localFrame(of: hitElement, windowBounds: snapshot.windowBounds),
+ rawActions: rawActions,
+ prettyActions: rawActions
+ )
+ }
+
+ private func clickPriority(for record: ElementRecord) -> Int {
+ if record.rawActions.contains(where: {
+ $0.caseInsensitiveCompare(kAXPressAction as String) == .orderedSame ||
+ $0.caseInsensitiveCompare(kAXConfirmAction as String) == .orderedSame ||
+ $0.caseInsensitiveCompare(kAXShowMenuAction as String) == .orderedSame
+ }) {
+ return 0
+ }
+
+ if let element = record.element, isSettable(element: element, attribute: kAXFocusedAttribute) {
+ return 1
+ }
+
+ return 2
+ }
+
+ private func frameArea(of record: ElementRecord) -> CGFloat {
+ guard let frame = record.localFrame else {
+ return .greatestFiniteMagnitude
+ }
+
+ return frame.width * frame.height
+ }
+
+ private func copyActions(for element: AXUIElement) -> [String]? {
+ var actions: CFArray?
+ let result = AXUIElementCopyActionNames(element, &actions)
+ guard result == .success else {
+ return nil
+ }
+
+ return actions as? [String]
+ }
+
+ private func localFrame(of element: AXUIElement, windowBounds: CGRect?) -> CGRect? {
+ var positionValue: CFTypeRef?
+ var sizeValue: CFTypeRef?
+ let positionResult = AXUIElementCopyAttributeValue(element, kAXPositionAttribute as CFString, &positionValue)
+ let sizeResult = AXUIElementCopyAttributeValue(element, kAXSizeAttribute as CFString, &sizeValue)
+
+ guard
+ positionResult == .success,
+ sizeResult == .success,
+ let positionValue,
+ let sizeValue
+ else {
+ return nil
+ }
+
+ let positionAXValue = positionValue as! AXValue
+ let sizeAXValue = sizeValue as! AXValue
+ var position = CGPoint.zero
+ var size = CGSize.zero
+ guard AXValueGetValue(positionAXValue, .cgPoint, &position), AXValueGetValue(sizeAXValue, .cgSize, &size) else {
+ return nil
+ }
+
+ let frame = CGRect(origin: position, size: size)
+ guard let windowBounds else {
+ return frame
+ }
+
+ return windowRelativeFrame(elementFrame: frame, windowBounds: windowBounds)
+ }
+
private func globalPoint(for record: ElementRecord, snapshot: AppSnapshot) throws -> CGPoint? {
guard let frame = record.localFrame else {
return nil
diff --git a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift
index b75a749..5c71d6b 100644
--- a/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift
+++ b/packages/OpenCodexComputerUseKit/Sources/OpenCodexComputerUseKit/InputSimulation.swift
@@ -43,12 +43,12 @@ enum MouseButtonKind: String {
}
enum InputSimulation {
- static func activate(_ app: RunningAppDescriptor) {
- app.runningApplication.activate(options: [.activateIgnoringOtherApps])
+ static func bringAppToFrontForGlobalPointerInput(_ app: RunningAppDescriptor) {
+ app.runningApplication.activate()
Thread.sleep(forTimeInterval: 0.25)
}
- static func click(at point: CGPoint, button: MouseButtonKind, clickCount: Int) throws {
+ static func clickGlobally(at point: CGPoint, button: MouseButtonKind, clickCount: Int) throws {
guard let source = CGEventSource(stateID: .hidSystemState) else {
throw ComputerUseError.message("Failed to create HID event source.")
}
@@ -60,7 +60,7 @@ enum InputSimulation {
}
}
- static func scroll(at point: CGPoint, direction: String, pages: Int) throws {
+ static func scrollGlobally(at point: CGPoint, direction: String, pages: Int) throws {
guard let event = CGEvent(scrollWheelEvent2Source: nil, units: .line, wheelCount: 2, wheel1: wheel1(direction: direction, pages: pages), wheel2: wheel2(direction: direction, pages: pages), wheel3: 0) else {
throw ComputerUseError.message("Failed to create scroll event.")
}
@@ -70,7 +70,7 @@ enum InputSimulation {
Thread.sleep(forTimeInterval: 0.1)
}
- static func drag(from start: CGPoint, to end: CGPoint) throws {
+ static func dragGlobally(from start: CGPoint, to end: CGPoint) throws {
guard let source = CGEventSource(stateID: .hidSystemState) else {
throw ComputerUseError.message("Failed to create HID event source.")
}
@@ -90,7 +90,7 @@ enum InputSimulation {
try postMouseEvent(type: .leftMouseUp, source: source, point: end, button: .left, clickState: 1)
}
- static func typeText(_ text: String) throws {
+ static func typeText(_ text: String, pid: pid_t) throws {
for character in text.utf16 {
var mutableCharacter = character
guard let down = CGEvent(keyboardEventSource: nil, virtualKey: 0, keyDown: true),
@@ -100,13 +100,13 @@ enum InputSimulation {
down.keyboardSetUnicodeString(stringLength: 1, unicodeString: &mutableCharacter)
up.keyboardSetUnicodeString(stringLength: 1, unicodeString: &mutableCharacter)
- down.post(tap: .cghidEventTap)
- up.post(tap: .cghidEventTap)
+ down.postToPid(pid)
+ up.postToPid(pid)
Thread.sleep(forTimeInterval: 0.02)
}
}
- static func pressKey(_ specification: String) throws {
+ static func pressKey(_ specification: String, pid: pid_t) throws {
let parsed = try KeyPressParser.parse(specification)
var activeFlags: CGEventFlags = []
@@ -117,7 +117,7 @@ enum InputSimulation {
activeFlags.insert(modifier.flag)
event.flags = activeFlags
- event.post(tap: .cghidEventTap)
+ event.postToPid(pid)
}
guard let keyDown = CGEvent(keyboardEventSource: nil, virtualKey: parsed.keyCode, keyDown: true),
@@ -127,8 +127,8 @@ enum InputSimulation {
keyDown.flags = activeFlags
keyUp.flags = activeFlags
- keyDown.post(tap: .cghidEventTap)
- keyUp.post(tap: .cghidEventTap)
+ keyDown.postToPid(pid)
+ keyUp.postToPid(pid)
for modifier in parsed.modifiers.reversed() {
guard let event = CGEvent(keyboardEventSource: nil, virtualKey: modifier.keyCode, keyDown: false) else {
@@ -136,7 +136,7 @@ enum InputSimulation {
}
event.flags = activeFlags
- event.post(tap: .cghidEventTap)
+ event.postToPid(pid)
activeFlags.remove(modifier.flag)
}