Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

fix: map screenshot pixels to window points

usoon.xub4234f4a59a3
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 4513244..3a94278 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -76,7 +76,7 @@
- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素
- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias
- `scroll.pages` 对齐官方 `1.0.755` 的 `number` schema,支持小数页数;整数页且目标暴露 `AXScroll*ByPage` 时优先走 AX action,否则用 `CGEvent.postToPid` 向目标进程定向发送 scroll event
- - `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标;这些 pid-targeted mouse / scroll 事件在回放前会先把 screenshot 的 y-down 像素坐标转换到 AppKit-compatible 全局坐标,避免真实 app 因坐标系不一致点到错误位置
+ - `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标;这些 coordinate tool 的 `x/y` 先按 screenshot pixel 坐标解释,再依据截图像素尺寸与目标 window bounds 的比例映射回 window point / Quartz global 坐标,避免 Retina 窗口上把 2x 像素误当成 1x point 导致点击落到错误位置
- `click` / `scroll` / `drag` 只有设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 时才允许全局 `CGEvent.post(tap: .cghidEventTap)` 物理指针兜底;其中 `click` 默认会先走 AX,再走 `postToPid` 定向鼠标事件,最后才是显式开启的全局物理指针 fallback;默认路径不再为了 fallback 调用 `NSRunningApplication.activate`
### 4. Fixture Bridge
diff --git a/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md b/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md
index 06b1800..0319c83 100644
--- a/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md
+++ b/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md
@@ -12,12 +12,12 @@
**Scope:** `OpenComputerUseKit`, `docs/ARCHITECTURE.md`, `docs/histories/`
**Key Actions:**
-- **[Coordinate-space fix]**: 抽出共享的 screen-state -> AppKit 全局坐标转换,并把 `click` / `scroll` / `drag` 的 pid-targeted 鼠标事件统一走这条转换路径。
-- **[Regression coverage]**: 增加针对 targeted mouse 坐标转换的单测,覆盖单屏 y 反转和带 display offset 的多屏场景。
-- **[Behavior docs]**: 更新架构文档,明确 pid-targeted mouse / scroll 回放前会先做 screenshot 坐标到 AppKit 全局坐标的映射。
+- **[Retina pixel mapping fix]**: 把 `click` / `drag` 的 screenshot `x/y` 先按截图像素坐标映射回 window points,再拼到 Quartz 全局坐标,避免把 Retina 2x 截图像素直接当成 window point。
+- **[Regression coverage]**: 增加针对 screenshot pixel -> window point 转换的单测,覆盖真实 Calendar 样本里的 `2048x1266` screenshot 对 `1024x633` window bounds 的 2x 场景。
+- **[Behavior docs]**: 更新架构文档,明确 coordinate tools 会先按 screenshot pixel 坐标解释,再依据截图尺寸和 window bounds 做比例换算。
### 🧠 Design Intent (Why)
-`get_app_state` 暴露给工具的是 screenshot 像素坐标,属于窗口截图/CG window capture 的 y-down 坐标系;而 pid-targeted 鼠标事件最终会被目标 app 按 AppKit-compatible 全局坐标解释。之前直接把 screenshot 坐标喂给 `CGEvent.postToPid`,真实 app 会在错误位置收到点击,表现成点 Calendar 却触发了别的系统 UI。修复的关键不是改 `x/y` 语义,而是在 targeted event replay 前把坐标系补齐。
+`get_app_state` 暴露给工具的是 screenshot 像素坐标。真实窗口 bounds 和 AX frame 则是 point 单位;在 Retina 屏上两者常常正好差一个 `2x`。本地 live 诊断里,Calendar screenshot 是 `2048x1266`,但窗口 bounds 只有 `1024x633`。之前直接把 screenshot 像素当成 window point 去点击,目标点会偏到窗口外,表现成点 Calendar 却触发了别的系统 UI。修复的关键不是再翻一层 y,而是先把 screenshot pixel 坐标按截图尺寸和 window bounds 的比例还原成 point。
### 📁 Files Modified
- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
index f9d3f24..6653518 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
@@ -1,6 +1,7 @@
import AppKit
import ApplicationServices
import Foundation
+import ImageIO
struct VisualCursorTarget: Equatable {
let point: CGPoint
@@ -112,6 +113,42 @@ func globalPointerFallbacksEnabled(environment: [String: String]) -> Bool {
return ["1", "true", "yes", "on"].contains(rawValue)
}
+func screenshotPixelScale(
+ screenshotPixelSize: CGSize?,
+ windowBounds: CGRect?
+) -> CGSize {
+ guard
+ let screenshotPixelSize,
+ let windowBounds,
+ windowBounds.width > 0,
+ windowBounds.height > 0,
+ screenshotPixelSize.width > 0,
+ screenshotPixelSize.height > 0
+ else {
+ return CGSize(width: 1, height: 1)
+ }
+
+ return CGSize(
+ width: screenshotPixelSize.width / windowBounds.width,
+ height: screenshotPixelSize.height / windowBounds.height
+ )
+}
+
+func screenshotPixelToWindowPoint(
+ _ point: CGPoint,
+ screenshotPixelSize: CGSize?,
+ windowBounds: CGRect?
+) -> CGPoint {
+ let scale = screenshotPixelScale(
+ screenshotPixelSize: screenshotPixelSize,
+ windowBounds: windowBounds
+ )
+ return CGPoint(
+ x: point.x / scale.width,
+ y: point.y / scale.height
+ )
+}
+
let nonSettableSetValueErrorMessage = "Cannot set a value for an element that is not settable"
func setValueAttributeIsSettable(result: AXError, settable: Bool, attribute: String) throws -> Bool {
@@ -206,8 +243,9 @@ public final class ComputerUseService {
pulseVisualCursor(at: cursorTarget, clickCount: clickCount, mouseButton: button)
} else if let x, let y {
- let point = CGPoint(x: x, y: y)
- let targetPoint = try screenshotToGlobalPoint(snapshot: snapshot, x: x, y: y)
+ let screenshotPoint = CGPoint(x: x, y: y)
+ let point = screenshotPixelToWindowPointInSnapshot(snapshot: snapshot, point: screenshotPoint)
+ let targetPoint = try windowPointToGlobalPoint(snapshot: snapshot, point: point)
let cursorTarget = makeVisualCursorTarget(
at: targetPoint,
targetWindowID: snapshot.targetWindowID,
@@ -231,7 +269,7 @@ public final class ComputerUseService {
at: targetPoint,
button: button,
clickCount: clickCount,
- targetDescription: "x=\(Int(point.x)) y=\(Int(point.y))",
+ targetDescription: "x=\(Int(screenshotPoint.x)) y=\(Int(screenshotPoint.y))",
snapshot: snapshot
)
}
@@ -793,16 +831,53 @@ public final class ComputerUseService {
return nil
}
- return try screenshotToGlobalPoint(snapshot: snapshot, x: frame.midX, y: frame.midY)
+ return try windowPointToGlobalPoint(
+ snapshot: snapshot,
+ point: CGPoint(x: frame.midX, y: frame.midY)
+ )
}
private func screenshotToGlobalPoint(snapshot: AppSnapshot, x: Double, y: Double) throws -> CGPoint {
+ try windowPointToGlobalPoint(
+ snapshot: snapshot,
+ point: screenshotPixelToWindowPointInSnapshot(
+ snapshot: snapshot,
+ point: CGPoint(x: x, y: y)
+ )
+ )
+ }
+
+ private func screenshotPixelToWindowPointInSnapshot(snapshot: AppSnapshot, point: CGPoint) -> CGPoint {
+ screenshotPixelToWindowPoint(
+ point,
+ screenshotPixelSize: screenshotPixelSize(snapshot: snapshot),
+ windowBounds: snapshot.windowBounds
+ )
+ }
+
+ private func screenshotPixelSize(snapshot: AppSnapshot) -> CGSize? {
+ guard
+ let screenshotPNGData = snapshot.screenshotPNGData,
+ let imageSource = CGImageSourceCreateWithData(screenshotPNGData as CFData, nil),
+ let properties = CGImageSourceCopyPropertiesAtIndex(imageSource, 0, nil) as? [CFString: Any],
+ let pixelWidth = properties[kCGImagePropertyPixelWidth] as? CGFloat,
+ let pixelHeight = properties[kCGImagePropertyPixelHeight] as? CGFloat,
+ pixelWidth > 0,
+ pixelHeight > 0
+ else {
+ return nil
+ }
+
+ return CGSize(width: pixelWidth, height: pixelHeight)
+ }
+
+ private func windowPointToGlobalPoint(snapshot: AppSnapshot, point: CGPoint) throws -> CGPoint {
guard let windowBounds = snapshot.windowBounds else {
let appReference = snapshot.app.bundleIdentifier ?? snapshot.app.name
throw ComputerUseError.stateUnavailable("No window bounds are available for \(appReference). Run get_app_state after bringing the app on screen.")
}
- return CGPoint(x: windowBounds.minX + x, y: windowBounds.minY + y)
+ return CGPoint(x: windowBounds.minX + point.x, y: windowBounds.minY + point.y)
}
private func fixtureIdentifier(at point: CGPoint, snapshot: AppSnapshot) throws -> String {
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift
index 260fed9..227af7b 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift
@@ -70,11 +70,10 @@ enum InputSimulation {
throw ComputerUseError.message("Failed to create targeted event source.")
}
- let eventPoint = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: point)
for _ in 0..<max(clickCount, 1) {
- try postMouseEventToPid(type: .mouseMoved, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)
- try postMouseEventToPid(type: button.downEvent, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)
- try postMouseEventToPid(type: button.upEvent, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)
+ try postMouseEventToPid(type: .mouseMoved, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)
+ try postMouseEventToPid(type: button.downEvent, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)
+ try postMouseEventToPid(type: button.upEvent, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)
}
}
@@ -83,7 +82,7 @@ enum InputSimulation {
throw ComputerUseError.message("Failed to create scroll event.")
}
- event.location = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: point)
+ event.location = point
event.postToPid(pid)
Thread.sleep(forTimeInterval: 0.1)
}
@@ -103,22 +102,19 @@ enum InputSimulation {
throw ComputerUseError.message("Failed to create targeted event source.")
}
- let eventStart = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: start)
- let eventEnd = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: end)
-
- try postMouseEventToPid(type: .mouseMoved, source: source, point: eventStart, button: .left, clickState: 1, pid: pid)
- try postMouseEventToPid(type: .leftMouseDown, source: source, point: eventStart, button: .left, clickState: 1, pid: pid)
+ try postMouseEventToPid(type: .mouseMoved, source: source, point: start, button: .left, clickState: 1, pid: pid)
+ try postMouseEventToPid(type: .leftMouseDown, source: source, point: start, button: .left, clickState: 1, pid: pid)
for step in 1...10 {
let progress = CGFloat(step) / 10
let point = CGPoint(
- x: eventStart.x + ((eventEnd.x - eventStart.x) * progress),
- y: eventStart.y + ((eventEnd.y - eventStart.y) * progress)
+ x: start.x + ((end.x - start.x) * progress),
+ y: start.y + ((end.y - start.y) * progress)
)
try postMouseEventToPid(type: .leftMouseDragged, source: source, point: point, button: .left, clickState: 1, pid: pid)
}
- try postMouseEventToPid(type: .leftMouseUp, source: source, point: eventEnd, button: .left, clickState: 1, pid: pid)
+ try postMouseEventToPid(type: .leftMouseUp, source: source, point: end, button: .left, clickState: 1, pid: pid)
}
static func dragGlobally(from start: CGPoint, to end: CGPoint) throws {
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 44458bb..c1b876c 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -556,36 +556,55 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertEqual(point, CGPoint(x: 2415, y: 1783))
}
- func testScreenStatePointToAppKitGlobalPointConvertsTargetedMouseCoordinates() {
- let point = screenStatePointToAppKitGlobalPoint(
- fromScreenStatePoint: CGPoint(x: 2415, y: 181),
- screenMappings: [
- VisualCursorScreenMapping(
- screenStateFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964),
- appKitFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964)
- ),
- ]
+ func testScreenshotPixelScaleUsesRetinaSizedImageAgainstWindowBounds() {
+ let scale = screenshotPixelScale(
+ screenshotPixelSize: CGSize(width: 2048, height: 1266),
+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633)
)
- XCTAssertEqual(point, CGPoint(x: 2415, y: 1783))
+ XCTAssertEqual(scale.width, 2, accuracy: 0.0001)
+ XCTAssertEqual(scale.height, 2, accuracy: 0.0001)
}
- func testScreenStatePointToAppKitGlobalPointPreservesDisplayOffsets() {
- let point = screenStatePointToAppKitGlobalPoint(
- fromScreenStatePoint: CGPoint(x: 3320, y: 120),
- screenMappings: [
- VisualCursorScreenMapping(
- screenStateFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964),
- appKitFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964)
- ),
- VisualCursorScreenMapping(
- screenStateFrame: CGRect(x: 3024, y: 0, width: 1728, height: 1117),
- appKitFrame: CGRect(x: 3024, y: -153, width: 1728, height: 1117)
- ),
- ]
+ func testScreenshotPixelScaleStaysAtOneForUnscaledDisplays() {
+ let scale = screenshotPixelScale(
+ screenshotPixelSize: CGSize(width: 1024, height: 633),
+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633)
+ )
+
+ XCTAssertEqual(scale.width, 1, accuracy: 0.0001)
+ XCTAssertEqual(scale.height, 1, accuracy: 0.0001)
+ }
+
+ func testScreenshotPixelToWindowPointConvertsScreenshotPixelsBackToWindowPoints() {
+ let point = screenshotPixelToWindowPoint(
+ CGPoint(x: 1060, y: 790),
+ screenshotPixelSize: CGSize(width: 2048, height: 1266),
+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633)
+ )
+
+ XCTAssertEqual(point.x, 530, accuracy: 0.0001)
+ XCTAssertEqual(point.y, 395, accuracy: 0.0001)
+ }
+
+ func testScreenshotPixelToWindowPointKeepsCoordinatesOnUnscaledDisplays() {
+ let point = screenshotPixelToWindowPoint(
+ CGPoint(x: 530, y: 395),
+ screenshotPixelSize: CGSize(width: 1024, height: 633),
+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633)
+ )
+
+ XCTAssertEqual(point, CGPoint(x: 530, y: 395))
+ }
+
+ func testScreenshotPixelToWindowPointFallsBackToIdentityWithoutImageSize() {
+ let point = screenshotPixelToWindowPoint(
+ CGPoint(x: 530, y: 395),
+ screenshotPixelSize: nil,
+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633)
)
- XCTAssertEqual(point, CGPoint(x: 3320, y: 844))
+ XCTAssertEqual(point, CGPoint(x: 530, y: 395))
}
func testCursorWindowGeometryAnchorsTipPosition() {