Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Fix targeted mouse coordinate conversion

usoon.xua5c072be460d
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index cae2b49..f414486 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -76,7 +76,7 @@
- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素
- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias
- `scroll.pages` 对齐官方 `1.0.755` 的 `number` schema,支持小数页数;整数页且目标暴露 `AXScroll*ByPage` 时优先走 AX action,否则用 `CGEvent.postToPid` 向目标进程定向发送 scroll event
- - `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标
+ - `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标;这些 pid-targeted mouse / scroll 事件在回放前会先把 screenshot 的 y-down 像素坐标转换到 AppKit-compatible 全局坐标,避免真实 app 因坐标系不一致点到错误位置
- `click` / `scroll` / `drag` 只有设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 时才允许全局 `CGEvent.post(tap: .cghidEventTap)` 物理指针兜底;其中 `click` 默认会先走 AX,再走 `postToPid` 定向鼠标事件,最后才是显式开启的全局物理指针 fallback;默认路径不再为了 fallback 调用 `NSRunningApplication.activate`
### 4. Fixture Bridge
diff --git a/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md b/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md
new file mode 100644
index 0000000..06b1800
--- /dev/null
+++ b/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md
@@ -0,0 +1,27 @@
+## [2026-04-22 17:05] | Task: fix targeted mouse coordinate space
+
+### 🤖 Execution Context
+* **Agent ID**: `unknown`
+* **Base Model**: `GPT-5 Codex`
+* **Runtime**: `Codex CLI`
+
+### 📥 User Query
+> if mcp click with x,y like `click({"app":"Calendar","x":1060,"y":790})`, it will trigger the "About Mac" page, why? fix this
+
+### 🛠 Changes Overview
+**Scope:** `OpenComputerUseKit`, `docs/ARCHITECTURE.md`, `docs/histories/`
+
+**Key Actions:**
+- **[Coordinate-space fix]**: 抽出共享的 screen-state -> AppKit 全局坐标转换,并把 `click` / `scroll` / `drag` 的 pid-targeted 鼠标事件统一走这条转换路径。
+- **[Regression coverage]**: 增加针对 targeted mouse 坐标转换的单测,覆盖单屏 y 反转和带 display offset 的多屏场景。
+- **[Behavior docs]**: 更新架构文档,明确 pid-targeted mouse / scroll 回放前会先做 screenshot 坐标到 AppKit 全局坐标的映射。
+
+### 🧠 Design Intent (Why)
+`get_app_state` 暴露给工具的是 screenshot 像素坐标,属于窗口截图/CG window capture 的 y-down 坐标系;而 pid-targeted 鼠标事件最终会被目标 app 按 AppKit-compatible 全局坐标解释。之前直接把 screenshot 坐标喂给 `CGEvent.postToPid`,真实 app 会在错误位置收到点击,表现成点 Calendar 却触发了别的系统 UI。修复的关键不是改 `x/y` 语义,而是在 targeted event replay 前把坐标系补齐。
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
+- `docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md`
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
index b054f67..db713d0 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
@@ -25,7 +25,7 @@ func currentVisualCursorScreenMappings() -> [VisualCursorScreenMapping] {
}
}
-func visualCursorAppKitPoint(
+func screenStatePointToAppKitGlobalPoint(
fromScreenStatePoint point: CGPoint,
screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()
) -> CGPoint {
@@ -42,6 +42,16 @@ func visualCursorAppKitPoint(
)
}
+func visualCursorAppKitPoint(
+ fromScreenStatePoint point: CGPoint,
+ screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()
+) -> CGPoint {
+ screenStatePointToAppKitGlobalPoint(
+ fromScreenStatePoint: point,
+ screenMappings: screenMappings
+ )
+}
+
func makeVisualCursorTarget(
at point: CGPoint,
targetWindowID: CGWindowID?,
@@ -49,7 +59,7 @@ func makeVisualCursorTarget(
screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()
) -> VisualCursorTarget {
VisualCursorTarget(
- point: visualCursorAppKitPoint(
+ point: screenStatePointToAppKitGlobalPoint(
fromScreenStatePoint: point,
screenMappings: screenMappings
),
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift
index 227af7b..260fed9 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift
@@ -70,10 +70,11 @@ enum InputSimulation {
throw ComputerUseError.message("Failed to create targeted event source.")
}
+ let eventPoint = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: point)
for _ in 0..<max(clickCount, 1) {
- try postMouseEventToPid(type: .mouseMoved, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)
- try postMouseEventToPid(type: button.downEvent, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)
- try postMouseEventToPid(type: button.upEvent, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)
+ try postMouseEventToPid(type: .mouseMoved, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)
+ try postMouseEventToPid(type: button.downEvent, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)
+ try postMouseEventToPid(type: button.upEvent, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)
}
}
@@ -82,7 +83,7 @@ enum InputSimulation {
throw ComputerUseError.message("Failed to create scroll event.")
}
- event.location = point
+ event.location = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: point)
event.postToPid(pid)
Thread.sleep(forTimeInterval: 0.1)
}
@@ -102,19 +103,22 @@ enum InputSimulation {
throw ComputerUseError.message("Failed to create targeted event source.")
}
- try postMouseEventToPid(type: .mouseMoved, source: source, point: start, button: .left, clickState: 1, pid: pid)
- try postMouseEventToPid(type: .leftMouseDown, source: source, point: start, button: .left, clickState: 1, pid: pid)
+ let eventStart = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: start)
+ let eventEnd = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: end)
+
+ try postMouseEventToPid(type: .mouseMoved, source: source, point: eventStart, button: .left, clickState: 1, pid: pid)
+ try postMouseEventToPid(type: .leftMouseDown, source: source, point: eventStart, button: .left, clickState: 1, pid: pid)
for step in 1...10 {
let progress = CGFloat(step) / 10
let point = CGPoint(
- x: start.x + ((end.x - start.x) * progress),
- y: start.y + ((end.y - start.y) * progress)
+ x: eventStart.x + ((eventEnd.x - eventStart.x) * progress),
+ y: eventStart.y + ((eventEnd.y - eventStart.y) * progress)
)
try postMouseEventToPid(type: .leftMouseDragged, source: source, point: point, button: .left, clickState: 1, pid: pid)
}
- try postMouseEventToPid(type: .leftMouseUp, source: source, point: end, button: .left, clickState: 1, pid: pid)
+ try postMouseEventToPid(type: .leftMouseUp, source: source, point: eventEnd, button: .left, clickState: 1, pid: pid)
}
static func dragGlobally(from start: CGPoint, to end: CGPoint) throws {
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 51a42c0..deda869 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -556,6 +556,38 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertEqual(point, CGPoint(x: 2415, y: 1783))
}
+ func testScreenStatePointToAppKitGlobalPointConvertsTargetedMouseCoordinates() {
+ let point = screenStatePointToAppKitGlobalPoint(
+ fromScreenStatePoint: CGPoint(x: 2415, y: 181),
+ screenMappings: [
+ VisualCursorScreenMapping(
+ screenStateFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964),
+ appKitFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964)
+ ),
+ ]
+ )
+
+ XCTAssertEqual(point, CGPoint(x: 2415, y: 1783))
+ }
+
+ func testScreenStatePointToAppKitGlobalPointPreservesDisplayOffsets() {
+ let point = screenStatePointToAppKitGlobalPoint(
+ fromScreenStatePoint: CGPoint(x: 3320, y: 120),
+ screenMappings: [
+ VisualCursorScreenMapping(
+ screenStateFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964),
+ appKitFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964)
+ ),
+ VisualCursorScreenMapping(
+ screenStateFrame: CGRect(x: 3024, y: 0, width: 1728, height: 1117),
+ appKitFrame: CGRect(x: 3024, y: -153, width: 1728, height: 1117)
+ ),
+ ]
+ )
+
+ XCTAssertEqual(point, CGPoint(x: 3320, y: 844))
+ }
+
func testCursorWindowGeometryAnchorsTipPosition() {
let geometry = CursorWindowGeometry(
windowSize: CGSize(width: 128, height: 128),