Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Fix click global pointer fallback behavior

Leo5a88929b3e97
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index faf811d..a3ba47a 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -65,14 +65,14 @@
- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。
- overlay 不再依赖临时 `terminal settle` 补丁来修尾;主线现在统一改成“路径层给目标点,visual dynamics 层给可见姿态”的双层模型,所以 move 末段、pulse 和 idle 共用同一套状态,不会再出现 endpoint 锁住后只剩原地翻角的收尾。
- overlay 的渲染输入也从单一 `rotation` 扩展成 `rotation + cursorBodyOffset + fogOffset + fogScale`,让速度滞后能真正体现在画面上,而不是只存在于主循环内部状态;其中 `rotation` 现在按二进制里 `SoftwareCursorStyle.angle + CursorView._animatedAngleOffsetDegrees` 的分层去近似,不再把“跟随运动方向的主朝向”和“小幅 wiggle offset”压成同一个受限小角度。
-- 动作型 tools 对普通 app 采用“非侵入优先,HID 兜底”策略:
+- 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略:
- `AXUIElementPerformAction`
- `AXUIElementSetAttributeValue`
- - element-targeted `click` 的左键路径会先试 `AXPress`,再试窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute`;这里不再把 `AXUIElementIsAttributeSettable` 的结果当成硬门槛,因为像 `TextEdit` window 这类元素在 `kAXFocusedAttribute` 上会出现“`isSettable=false` 但直接 set 成功”的官方同款场景
+ - element-targeted `click` 的左键路径会先试 `AXPress`,再试窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute`;这里不再把 `AXUIElementIsAttributeSettable` 的结果当成硬门槛,因为像 `TextEdit` window 这类元素在 `kAXFocusedAttribute` 上会出现“`isSettable=false` 但直接 set 成功”的官方同款场景;`click_count > 1` 会优先重复可用的 AX action,而不是直接退到全局鼠标事件
- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素
- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台
- - 当必须退回全局鼠标路径时,先尝试对目标窗口做 `AXRaise` / main-window 聚焦,只有这些 AX 提升失败后才调用 `NSRunningApplication.activate`
- - 只有 drag 或无法命中 AX 元素的鼠标路径,才退回全局 `CGEvent` 键鼠事件并尽量缩小对用户当前焦点的影响
+ - `click` 不再默认退回全局 `CGEvent.post(tap: .cghidEventTap)`,因为该路径会发送 `.mouseMoved` 并移动系统硬件光标;只有设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 时才允许这条物理指针兜底
+ - `drag` 当前仍是 coordinate-only API,无法通过 AX action 完整表达时会继续使用全局鼠标事件;这条路径后续需要继续向官方的 window / event-target aware 路由收敛
### 4. Fixture Bridge
diff --git a/docs/histories/2026-04/20260421-2120-disable-click-global-pointer-default.md b/docs/histories/2026-04/20260421-2120-disable-click-global-pointer-default.md
new file mode 100644
index 0000000..411a3e4
--- /dev/null
+++ b/docs/histories/2026-04/20260421-2120-disable-click-global-pointer-default.md
@@ -0,0 +1,27 @@
+## [2026-04-21 21:20] | Task: 对齐 click 的非侵入默认行为
+
+### 🤖 Execution Context
+* **Agent ID**: `Codex`
+* **Base Model**: `GPT-5`
+* **Runtime**: `local macOS shell`
+
+### 📥 User Query
+> 继续逆向确认官方 Computer Use 的 click/HID 逻辑,并在得出结论后修正本仓库实现,避免 click 抢用户鼠标。
+
+### 🛠 Changes Overview
+**Scope:** `OpenComputerUseKit` click 行为与架构文档
+
+**Key Actions:**
+- **[官方行为确认]**: 静态确认官方包有 AX action、EventTap/CGEvent 鼠标事件生成、`clickEventTap`、`MouseEventTarget` 和 `feature/computerUseAlwaysSimulateClick`,但没有直接导入公开 `CGEventPost` / `IOHID*` 符号;`AlwaysSimulateClick` 默认值为关闭。
+- **[默认关闭全局 click 兜底]**: `click` 在 AX 路径失败后不再默认调用全局 `.cghidEventTap` 鼠标事件,必须设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 才允许物理指针兜底。
+- **[修正 click_count 分支]**: `AXPress` / `AXConfirm` / `AXShowMenu` 支持按 `click_count` 重复执行,避免多次点击因为 `clickCount != 1` 直接落入全局鼠标路径。
+- **[测试补充]**: 新增全局指针兜底环境变量默认关闭的单元测试。
+
+### 🧠 Design Intent (Why)
+官方实现有物理点击模拟能力,但它不是简单把所有 fallback 都发到系统级硬件光标;二进制里还能看到 focus-steal suppression、target 类型和 feature flag。当前本仓库直接用 `.cghidEventTap` 加 `.mouseMoved` 会移动用户真实光标,和工具说明里的后台交互预期冲突。先把 click 的高风险兜底改成显式 opt-in,同时保留可调试逃生口。
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
+- `docs/histories/2026-04/20260421-2120-disable-click-global-pointer-default.md`
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
index 6c6cbcb..d3758df 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
@@ -50,6 +50,17 @@ func inputFallbackDebugEnabled(environment: [String: String]) -> Bool {
return ["1", "true", "yes", "on"].contains(rawValue)
}
+func globalPointerFallbacksEnabled(environment: [String: String]) -> Bool {
+ guard let rawValue = environment["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS"]?
+ .trimmingCharacters(in: .whitespacesAndNewlines)
+ .lowercased()
+ else {
+ return false
+ }
+
+ return ["1", "true", "yes", "on"].contains(rawValue)
+}
+
public final class ComputerUseService {
private var snapshotsByApp: [String: AppSnapshot] = [:]
@@ -105,13 +116,13 @@ public final class ComputerUseService {
if try performPreferredClick(on: record, button: button, clickCount: clickCount) {
Thread.sleep(forTimeInterval: 0.15)
} else {
- debugInputFallback(
- tool: "click",
+ try performGlobalClickFallback(
+ at: targetPoint,
+ button: button,
+ clickCount: clickCount,
targetDescription: "element_index=\(elementIndex)",
snapshot: snapshot
)
- InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)
- try InputSimulation.clickGlobally(at: targetPoint, button: button, clickCount: clickCount)
}
} catch {
settleVisualCursor(at: cursorTarget)
@@ -135,13 +146,13 @@ public final class ComputerUseService {
try performPreferredClick(on: record, button: button, clickCount: clickCount) {
Thread.sleep(forTimeInterval: 0.15)
} else {
- debugInputFallback(
- tool: "click",
+ try performGlobalClickFallback(
+ at: targetPoint,
+ button: button,
+ clickCount: clickCount,
targetDescription: "x=\(Int(point.x)) y=\(Int(point.y))",
snapshot: snapshot
)
- InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)
- try InputSimulation.clickGlobally(at: targetPoint, button: button, clickCount: clickCount)
}
} catch {
settleVisualCursor(at: cursorTarget)
@@ -343,13 +354,13 @@ public final class ComputerUseService {
}
private func performPreferredClick(on record: ElementRecord, button: MouseButtonKind, clickCount: Int) throws -> Bool {
- guard let element = record.element, clickCount == 1 else {
+ guard let element = record.element else {
return false
}
switch button {
case .left:
- if try performAction(named: kAXPressAction as String, on: element, availableActions: record.rawActions) {
+ if try performAction(named: kAXPressAction as String, on: element, availableActions: record.rawActions, repeatCount: clickCount) {
return true
}
@@ -357,11 +368,11 @@ public final class ComputerUseService {
return true
}
- if try performAction(named: kAXConfirmAction as String, on: element, availableActions: record.rawActions) {
+ if try performAction(named: kAXConfirmAction as String, on: element, availableActions: record.rawActions, repeatCount: clickCount) {
return true
}
case .right:
- if try performAction(named: kAXShowMenuAction as String, on: element, availableActions: record.rawActions) {
+ if try performAction(named: kAXShowMenuAction as String, on: element, availableActions: record.rawActions, repeatCount: clickCount) {
return true
}
case .middle:
@@ -371,20 +382,27 @@ public final class ComputerUseService {
return false
}
- private func performAction(named action: String, on element: AXUIElement, availableActions: [String]) throws -> Bool {
+ private func performAction(named action: String, on element: AXUIElement, availableActions: [String], repeatCount: Int = 1) throws -> Bool {
guard availableActions.contains(where: { $0.caseInsensitiveCompare(action) == .orderedSame }) else {
return false
}
- let result = AXUIElementPerformAction(element, action as CFString)
- switch result {
- case .success:
- return true
- case .actionUnsupported, .cannotComplete, .noValue:
- return false
- default:
- throw ComputerUseError.message("AXUIElementPerformAction(\(action)) failed with \(result.rawValue)")
+ let attempts = max(repeatCount, 1)
+ for index in 0..<attempts {
+ let result = AXUIElementPerformAction(element, action as CFString)
+ switch result {
+ case .success:
+ if index < attempts - 1 {
+ Thread.sleep(forTimeInterval: 0.05)
+ }
+ case .actionUnsupported, .cannotComplete, .noValue:
+ return false
+ default:
+ throw ComputerUseError.message("AXUIElementPerformAction(\(action)) failed with \(result.rawValue)")
+ }
}
+
+ return true
}
private func activateClickTarget(element: AXUIElement, availableActions: [String]) throws -> Bool {
@@ -615,6 +633,28 @@ public final class ComputerUseService {
)
}
+ private func performGlobalClickFallback(
+ at point: CGPoint,
+ button: MouseButtonKind,
+ clickCount: Int,
+ targetDescription: String,
+ snapshot: AppSnapshot
+ ) throws {
+ guard globalPointerFallbacksEnabled(environment: ProcessInfo.processInfo.environment) else {
+ throw ComputerUseError.message(
+ "click could not be handled through accessibility, and global pointer fallback is disabled. Set OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 to allow physical-pointer fallback for this process."
+ )
+ }
+
+ debugInputFallback(
+ tool: "click",
+ targetDescription: targetDescription,
+ snapshot: snapshot
+ )
+ InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)
+ try InputSimulation.clickGlobally(at: point, button: button, clickCount: clickCount)
+ }
+
private func snapshotResult(for snapshot: AppSnapshot, style: SnapshotTextStyle) -> ToolCallResult {
var content = [ToolResultContentItem.text(snapshot.renderedText(style: style))]
if let screenshotPNGData = snapshot.screenshotPNGData {
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index c146fc9..70c11dd 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -355,6 +355,14 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertFalse(inputFallbackDebugEnabled(environment: ["OPEN_COMPUTER_USE_DEBUG_INPUT_FALLBACKS": "off"]))
}
+ func testGlobalPointerFallbackFlagDefaultsToDisabled() {
+ XCTAssertFalse(globalPointerFallbacksEnabled(environment: [:]))
+ XCTAssertTrue(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "1"]))
+ XCTAssertTrue(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "yes"]))
+ XCTAssertFalse(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "0"]))
+ XCTAssertFalse(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "false"]))
+ }
+
func testMakeVisualCursorTargetUsesWindowRelativeElementCenter() {
let target = makeVisualCursorTarget(
localFrame: CGRect(x: 24, y: 32, width: 120, height: 48),