👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Fix targeted mouse coordinate conversion
usoon.xua5c072be460d
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex cae2b49..f414486 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -76,7 +76,7 @@- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias- `scroll.pages` 对齐官方 `1.0.755` 的 `number` schema,支持小数页数;整数页且目标暴露 `AXScroll*ByPage` 时优先走 AX action,否则用 `CGEvent.postToPid` 向目标进程定向发送 scroll event- - `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标+ - `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标;这些 pid-targeted mouse / scroll 事件在回放前会先把 screenshot 的 y-down 像素坐标转换到 AppKit-compatible 全局坐标,避免真实 app 因坐标系不一致点到错误位置- `click` / `scroll` / `drag` 只有设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 时才允许全局 `CGEvent.post(tap: .cghidEventTap)` 物理指针兜底;其中 `click` 默认会先走 AX,再走 `postToPid` 定向鼠标事件,最后才是显式开启的全局物理指针 fallback;默认路径不再为了 fallback 调用 `NSRunningApplication.activate`### 4. Fixture Bridgediff --git a/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md b/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.mdnew file mode 100644index 0000000..06b1800--- /dev/null+++ b/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md@@ -0,0 +1,27 @@+## [2026-04-22 17:05] | Task: fix targeted mouse coordinate space++### 🤖 Execution Context+* **Agent ID**: `unknown`+* **Base Model**: `GPT-5 Codex`+* **Runtime**: `Codex CLI`++### 📥 User Query+> if mcp click with x,y like `click({"app":"Calendar","x":1060,"y":790})`, it will trigger the "About Mac" page, why? fix this++### 🛠 Changes Overview+**Scope:** `OpenComputerUseKit`, `docs/ARCHITECTURE.md`, `docs/histories/`++**Key Actions:**+- **[Coordinate-space fix]**: 抽出共享的 screen-state -> AppKit 全局坐标转换,并把 `click` / `scroll` / `drag` 的 pid-targeted 鼠标事件统一走这条转换路径。+- **[Regression coverage]**: 增加针对 targeted mouse 坐标转换的单测,覆盖单屏 y 反转和带 display offset 的多屏场景。+- **[Behavior docs]**: 更新架构文档,明确 pid-targeted mouse / scroll 回放前会先做 screenshot 坐标到 AppKit 全局坐标的映射。++### 🧠 Design Intent (Why)+`get_app_state` 暴露给工具的是 screenshot 像素坐标,属于窗口截图/CG window capture 的 y-down 坐标系;而 pid-targeted 鼠标事件最终会被目标 app 按 AppKit-compatible 全局坐标解释。之前直接把 screenshot 坐标喂给 `CGEvent.postToPid`,真实 app 会在错误位置收到点击,表现成点 Calendar 却触发了别的系统 UI。修复的关键不是改 `x/y` 语义,而是在 targeted event replay 前把坐标系补齐。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex b054f67..db713d0 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -25,7 +25,7 @@ func currentVisualCursorScreenMappings() -> [VisualCursorScreenMapping] {}}-func visualCursorAppKitPoint(+func screenStatePointToAppKitGlobalPoint(fromScreenStatePoint point: CGPoint,screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()) -> CGPoint {@@ -42,6 +42,16 @@ func visualCursorAppKitPoint()}+func visualCursorAppKitPoint(+ fromScreenStatePoint point: CGPoint,+ screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()+) -> CGPoint {+ screenStatePointToAppKitGlobalPoint(+ fromScreenStatePoint: point,+ screenMappings: screenMappings+ )+}+func makeVisualCursorTarget(at point: CGPoint,targetWindowID: CGWindowID?,@@ -49,7 +59,7 @@ func makeVisualCursorTarget(screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()) -> VisualCursorTarget {VisualCursorTarget(- point: visualCursorAppKitPoint(+ point: screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: point,screenMappings: screenMappings),diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swiftindex 227af7b..260fed9 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift@@ -70,10 +70,11 @@ enum InputSimulation {throw ComputerUseError.message("Failed to create targeted event source.")}+ let eventPoint = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: point)for _ in 0..<max(clickCount, 1) {- try postMouseEventToPid(type: .mouseMoved, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)- try postMouseEventToPid(type: button.downEvent, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)- try postMouseEventToPid(type: button.upEvent, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)+ try postMouseEventToPid(type: .mouseMoved, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)+ try postMouseEventToPid(type: button.downEvent, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)+ try postMouseEventToPid(type: button.upEvent, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)}}@@ -82,7 +83,7 @@ enum InputSimulation {throw ComputerUseError.message("Failed to create scroll event.")}- event.location = point+ event.location = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: point)event.postToPid(pid)Thread.sleep(forTimeInterval: 0.1)}@@ -102,19 +103,22 @@ enum InputSimulation {throw ComputerUseError.message("Failed to create targeted event source.")}- try postMouseEventToPid(type: .mouseMoved, source: source, point: start, button: .left, clickState: 1, pid: pid)- try postMouseEventToPid(type: .leftMouseDown, source: source, point: start, button: .left, clickState: 1, pid: pid)+ let eventStart = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: start)+ let eventEnd = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: end)++ try postMouseEventToPid(type: .mouseMoved, source: source, point: eventStart, button: .left, clickState: 1, pid: pid)+ try postMouseEventToPid(type: .leftMouseDown, source: source, point: eventStart, button: .left, clickState: 1, pid: pid)for step in 1...10 {let progress = CGFloat(step) / 10let point = CGPoint(- x: start.x + ((end.x - start.x) * progress),- y: start.y + ((end.y - start.y) * progress)+ x: eventStart.x + ((eventEnd.x - eventStart.x) * progress),+ y: eventStart.y + ((eventEnd.y - eventStart.y) * progress))try postMouseEventToPid(type: .leftMouseDragged, source: source, point: point, button: .left, clickState: 1, pid: pid)}- try postMouseEventToPid(type: .leftMouseUp, source: source, point: end, button: .left, clickState: 1, pid: pid)+ try postMouseEventToPid(type: .leftMouseUp, source: source, point: eventEnd, button: .left, clickState: 1, pid: pid)}static func dragGlobally(from start: CGPoint, to end: CGPoint) throws {diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 51a42c0..deda869 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -556,6 +556,38 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(point, CGPoint(x: 2415, y: 1783))}+ func testScreenStatePointToAppKitGlobalPointConvertsTargetedMouseCoordinates() {+ let point = screenStatePointToAppKitGlobalPoint(+ fromScreenStatePoint: CGPoint(x: 2415, y: 181),+ screenMappings: [+ VisualCursorScreenMapping(+ screenStateFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964),+ appKitFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964)+ ),+ ]+ )++ XCTAssertEqual(point, CGPoint(x: 2415, y: 1783))+ }++ func testScreenStatePointToAppKitGlobalPointPreservesDisplayOffsets() {+ let point = screenStatePointToAppKitGlobalPoint(+ fromScreenStatePoint: CGPoint(x: 3320, y: 120),+ screenMappings: [+ VisualCursorScreenMapping(+ screenStateFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964),+ appKitFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964)+ ),+ VisualCursorScreenMapping(+ screenStateFrame: CGRect(x: 3024, y: 0, width: 1728, height: 1117),+ appKitFrame: CGRect(x: 3024, y: -153, width: 1728, height: 1117)+ ),+ ]+ )++ XCTAssertEqual(point, CGPoint(x: 3320, y: 844))+ }+func testCursorWindowGeometryAnchorsTipPosition() {let geometry = CursorWindowGeometry(windowSize: CGSize(width: 128, height: 128),