👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Fix click global pointer fallback behavior
Leo5a88929b3e97
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex faf811d..a3ba47a 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -65,14 +65,14 @@- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。- overlay 不再依赖临时 `terminal settle` 补丁来修尾;主线现在统一改成“路径层给目标点,visual dynamics 层给可见姿态”的双层模型,所以 move 末段、pulse 和 idle 共用同一套状态,不会再出现 endpoint 锁住后只剩原地翻角的收尾。- overlay 的渲染输入也从单一 `rotation` 扩展成 `rotation + cursorBodyOffset + fogOffset + fogScale`,让速度滞后能真正体现在画面上,而不是只存在于主循环内部状态;其中 `rotation` 现在按二进制里 `SoftwareCursorStyle.angle + CursorView._animatedAngleOffsetDegrees` 的分层去近似,不再把“跟随运动方向的主朝向”和“小幅 wiggle offset”压成同一个受限小角度。-- 动作型 tools 对普通 app 采用“非侵入优先,HID 兜底”策略:+- 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略:- `AXUIElementPerformAction`- `AXUIElementSetAttributeValue`- - element-targeted `click` 的左键路径会先试 `AXPress`,再试窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute`;这里不再把 `AXUIElementIsAttributeSettable` 的结果当成硬门槛,因为像 `TextEdit` window 这类元素在 `kAXFocusedAttribute` 上会出现“`isSettable=false` 但直接 set 成功”的官方同款场景+ - element-targeted `click` 的左键路径会先试 `AXPress`,再试窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute`;这里不再把 `AXUIElementIsAttributeSettable` 的结果当成硬门槛,因为像 `TextEdit` window 这类元素在 `kAXFocusedAttribute` 上会出现“`isSettable=false` 但直接 set 成功”的官方同款场景;`click_count > 1` 会优先重复可用的 AX action,而不是直接退到全局鼠标事件- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台- - 当必须退回全局鼠标路径时,先尝试对目标窗口做 `AXRaise` / main-window 聚焦,只有这些 AX 提升失败后才调用 `NSRunningApplication.activate`- - 只有 drag 或无法命中 AX 元素的鼠标路径,才退回全局 `CGEvent` 键鼠事件并尽量缩小对用户当前焦点的影响+ - `click` 不再默认退回全局 `CGEvent.post(tap: .cghidEventTap)`,因为该路径会发送 `.mouseMoved` 并移动系统硬件光标;只有设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 时才允许这条物理指针兜底+ - `drag` 当前仍是 coordinate-only API,无法通过 AX action 完整表达时会继续使用全局鼠标事件;这条路径后续需要继续向官方的 window / event-target aware 路由收敛### 4. Fixture Bridgediff --git a/docs/histories/2026-04/20260421-2120-disable-click-global-pointer-default.md b/docs/histories/2026-04/20260421-2120-disable-click-global-pointer-default.mdnew file mode 100644index 0000000..411a3e4--- /dev/null+++ b/docs/histories/2026-04/20260421-2120-disable-click-global-pointer-default.md@@ -0,0 +1,27 @@+## [2026-04-21 21:20] | Task: 对齐 click 的非侵入默认行为++### 🤖 Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `local macOS shell`++### 📥 User Query+> 继续逆向确认官方 Computer Use 的 click/HID 逻辑,并在得出结论后修正本仓库实现,避免 click 抢用户鼠标。++### 🛠 Changes Overview+**Scope:** `OpenComputerUseKit` click 行为与架构文档++**Key Actions:**+- **[官方行为确认]**: 静态确认官方包有 AX action、EventTap/CGEvent 鼠标事件生成、`clickEventTap`、`MouseEventTarget` 和 `feature/computerUseAlwaysSimulateClick`,但没有直接导入公开 `CGEventPost` / `IOHID*` 符号;`AlwaysSimulateClick` 默认值为关闭。+- **[默认关闭全局 click 兜底]**: `click` 在 AX 路径失败后不再默认调用全局 `.cghidEventTap` 鼠标事件,必须设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 才允许物理指针兜底。+- **[修正 click_count 分支]**: `AXPress` / `AXConfirm` / `AXShowMenu` 支持按 `click_count` 重复执行,避免多次点击因为 `clickCount != 1` 直接落入全局鼠标路径。+- **[测试补充]**: 新增全局指针兜底环境变量默认关闭的单元测试。++### 🧠 Design Intent (Why)+官方实现有物理点击模拟能力,但它不是简单把所有 fallback 都发到系统级硬件光标;二进制里还能看到 focus-steal suppression、target 类型和 feature flag。当前本仓库直接用 `.cghidEventTap` 加 `.mouseMoved` 会移动用户真实光标,和工具说明里的后台交互预期冲突。先把 click 的高风险兜底改成显式 opt-in,同时保留可调试逃生口。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-04/20260421-2120-disable-click-global-pointer-default.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex 6c6cbcb..d3758df 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -50,6 +50,17 @@ func inputFallbackDebugEnabled(environment: [String: String]) -> Bool {return ["1", "true", "yes", "on"].contains(rawValue)}+func globalPointerFallbacksEnabled(environment: [String: String]) -> Bool {+ guard let rawValue = environment["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS"]?+ .trimmingCharacters(in: .whitespacesAndNewlines)+ .lowercased()+ else {+ return false+ }++ return ["1", "true", "yes", "on"].contains(rawValue)+}+public final class ComputerUseService {private var snapshotsByApp: [String: AppSnapshot] = [:]@@ -105,13 +116,13 @@ public final class ComputerUseService {if try performPreferredClick(on: record, button: button, clickCount: clickCount) {Thread.sleep(forTimeInterval: 0.15)} else {- debugInputFallback(- tool: "click",+ try performGlobalClickFallback(+ at: targetPoint,+ button: button,+ clickCount: clickCount,targetDescription: "element_index=\(elementIndex)",snapshot: snapshot)- InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)- try InputSimulation.clickGlobally(at: targetPoint, button: button, clickCount: clickCount)}} catch {settleVisualCursor(at: cursorTarget)@@ -135,13 +146,13 @@ public final class ComputerUseService {try performPreferredClick(on: record, button: button, clickCount: clickCount) {Thread.sleep(forTimeInterval: 0.15)} else {- debugInputFallback(- tool: "click",+ try performGlobalClickFallback(+ at: targetPoint,+ button: button,+ clickCount: clickCount,targetDescription: "x=\(Int(point.x)) y=\(Int(point.y))",snapshot: snapshot)- InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)- try InputSimulation.clickGlobally(at: targetPoint, button: button, clickCount: clickCount)}} catch {settleVisualCursor(at: cursorTarget)@@ -343,13 +354,13 @@ public final class ComputerUseService {}private func performPreferredClick(on record: ElementRecord, button: MouseButtonKind, clickCount: Int) throws -> Bool {- guard let element = record.element, clickCount == 1 else {+ guard let element = record.element else {return false}switch button {case .left:- if try performAction(named: kAXPressAction as String, on: element, availableActions: record.rawActions) {+ if try performAction(named: kAXPressAction as String, on: element, availableActions: record.rawActions, repeatCount: clickCount) {return true}@@ -357,11 +368,11 @@ public final class ComputerUseService {return true}- if try performAction(named: kAXConfirmAction as String, on: element, availableActions: record.rawActions) {+ if try performAction(named: kAXConfirmAction as String, on: element, availableActions: record.rawActions, repeatCount: clickCount) {return true}case .right:- if try performAction(named: kAXShowMenuAction as String, on: element, availableActions: record.rawActions) {+ if try performAction(named: kAXShowMenuAction as String, on: element, availableActions: record.rawActions, repeatCount: clickCount) {return true}case .middle:@@ -371,20 +382,27 @@ public final class ComputerUseService {return false}- private func performAction(named action: String, on element: AXUIElement, availableActions: [String]) throws -> Bool {+ private func performAction(named action: String, on element: AXUIElement, availableActions: [String], repeatCount: Int = 1) throws -> Bool {guard availableActions.contains(where: { $0.caseInsensitiveCompare(action) == .orderedSame }) else {return false}- let result = AXUIElementPerformAction(element, action as CFString)- switch result {- case .success:- return true- case .actionUnsupported, .cannotComplete, .noValue:- return false- default:- throw ComputerUseError.message("AXUIElementPerformAction(\(action)) failed with \(result.rawValue)")+ let attempts = max(repeatCount, 1)+ for index in 0..<attempts {+ let result = AXUIElementPerformAction(element, action as CFString)+ switch result {+ case .success:+ if index < attempts - 1 {+ Thread.sleep(forTimeInterval: 0.05)+ }+ case .actionUnsupported, .cannotComplete, .noValue:+ return false+ default:+ throw ComputerUseError.message("AXUIElementPerformAction(\(action)) failed with \(result.rawValue)")+ }}++ return true}private func activateClickTarget(element: AXUIElement, availableActions: [String]) throws -> Bool {@@ -615,6 +633,28 @@ public final class ComputerUseService {)}+ private func performGlobalClickFallback(+ at point: CGPoint,+ button: MouseButtonKind,+ clickCount: Int,+ targetDescription: String,+ snapshot: AppSnapshot+ ) throws {+ guard globalPointerFallbacksEnabled(environment: ProcessInfo.processInfo.environment) else {+ throw ComputerUseError.message(+ "click could not be handled through accessibility, and global pointer fallback is disabled. Set OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 to allow physical-pointer fallback for this process."+ )+ }++ debugInputFallback(+ tool: "click",+ targetDescription: targetDescription,+ snapshot: snapshot+ )+ InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)+ try InputSimulation.clickGlobally(at: point, button: button, clickCount: clickCount)+ }+private func snapshotResult(for snapshot: AppSnapshot, style: SnapshotTextStyle) -> ToolCallResult {var content = [ToolResultContentItem.text(snapshot.renderedText(style: style))]if let screenshotPNGData = snapshot.screenshotPNGData {diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex c146fc9..70c11dd 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -355,6 +355,14 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertFalse(inputFallbackDebugEnabled(environment: ["OPEN_COMPUTER_USE_DEBUG_INPUT_FALLBACKS": "off"]))}+ func testGlobalPointerFallbackFlagDefaultsToDisabled() {+ XCTAssertFalse(globalPointerFallbacksEnabled(environment: [:]))+ XCTAssertTrue(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "1"]))+ XCTAssertTrue(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "yes"]))+ XCTAssertFalse(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "0"]))+ XCTAssertFalse(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "false"]))+ }+func testMakeVisualCursorTargetUsesWindowRelativeElementCenter() {let target = makeVisualCursorTarget(localFrame: CGRect(x: 24, y: 32, width: 120, height: 48),