👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
fix: map screenshot pixels to window points
usoon.xub4234f4a59a3
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 4513244..3a94278 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -76,7 +76,7 @@- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias- `scroll.pages` 对齐官方 `1.0.755` 的 `number` schema,支持小数页数;整数页且目标暴露 `AXScroll*ByPage` 时优先走 AX action,否则用 `CGEvent.postToPid` 向目标进程定向发送 scroll event- - `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标;这些 pid-targeted mouse / scroll 事件在回放前会先把 screenshot 的 y-down 像素坐标转换到 AppKit-compatible 全局坐标,避免真实 app 因坐标系不一致点到错误位置+ - `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标;这些 coordinate tool 的 `x/y` 先按 screenshot pixel 坐标解释,再依据截图像素尺寸与目标 window bounds 的比例映射回 window point / Quartz global 坐标,避免 Retina 窗口上把 2x 像素误当成 1x point 导致点击落到错误位置- `click` / `scroll` / `drag` 只有设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 时才允许全局 `CGEvent.post(tap: .cghidEventTap)` 物理指针兜底;其中 `click` 默认会先走 AX,再走 `postToPid` 定向鼠标事件,最后才是显式开启的全局物理指针 fallback;默认路径不再为了 fallback 调用 `NSRunningApplication.activate`### 4. Fixture Bridgediff --git a/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md b/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.mdindex 06b1800..0319c83 100644--- a/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md+++ b/docs/histories/2026-04/20260422-1705-fix-targeted-mouse-coordinate-space.md@@ -12,12 +12,12 @@**Scope:** `OpenComputerUseKit`, `docs/ARCHITECTURE.md`, `docs/histories/`**Key Actions:**-- **[Coordinate-space fix]**: 抽出共享的 screen-state -> AppKit 全局坐标转换,并把 `click` / `scroll` / `drag` 的 pid-targeted 鼠标事件统一走这条转换路径。-- **[Regression coverage]**: 增加针对 targeted mouse 坐标转换的单测,覆盖单屏 y 反转和带 display offset 的多屏场景。-- **[Behavior docs]**: 更新架构文档,明确 pid-targeted mouse / scroll 回放前会先做 screenshot 坐标到 AppKit 全局坐标的映射。+- **[Retina pixel mapping fix]**: 把 `click` / `drag` 的 screenshot `x/y` 先按截图像素坐标映射回 window points,再拼到 Quartz 全局坐标,避免把 Retina 2x 截图像素直接当成 window point。+- **[Regression coverage]**: 增加针对 screenshot pixel -> window point 转换的单测,覆盖真实 Calendar 样本里的 `2048x1266` screenshot 对 `1024x633` window bounds 的 2x 场景。+- **[Behavior docs]**: 更新架构文档,明确 coordinate tools 会先按 screenshot pixel 坐标解释,再依据截图尺寸和 window bounds 做比例换算。### 🧠 Design Intent (Why)-`get_app_state` 暴露给工具的是 screenshot 像素坐标,属于窗口截图/CG window capture 的 y-down 坐标系;而 pid-targeted 鼠标事件最终会被目标 app 按 AppKit-compatible 全局坐标解释。之前直接把 screenshot 坐标喂给 `CGEvent.postToPid`,真实 app 会在错误位置收到点击,表现成点 Calendar 却触发了别的系统 UI。修复的关键不是改 `x/y` 语义,而是在 targeted event replay 前把坐标系补齐。+`get_app_state` 暴露给工具的是 screenshot 像素坐标。真实窗口 bounds 和 AX frame 则是 point 单位;在 Retina 屏上两者常常正好差一个 `2x`。本地 live 诊断里,Calendar screenshot 是 `2048x1266`,但窗口 bounds 只有 `1024x633`。之前直接把 screenshot 像素当成 window point 去点击,目标点会偏到窗口外,表现成点 Calendar 却触发了别的系统 UI。修复的关键不是再翻一层 y,而是先把 screenshot pixel 坐标按截图尺寸和 window bounds 的比例还原成 point。### 📁 Files Modified- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex f9d3f24..6653518 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -1,6 +1,7 @@import AppKitimport ApplicationServicesimport Foundation+import ImageIOstruct VisualCursorTarget: Equatable {let point: CGPoint@@ -112,6 +113,42 @@ func globalPointerFallbacksEnabled(environment: [String: String]) -> Bool {return ["1", "true", "yes", "on"].contains(rawValue)}+func screenshotPixelScale(+ screenshotPixelSize: CGSize?,+ windowBounds: CGRect?+) -> CGSize {+ guard+ let screenshotPixelSize,+ let windowBounds,+ windowBounds.width > 0,+ windowBounds.height > 0,+ screenshotPixelSize.width > 0,+ screenshotPixelSize.height > 0+ else {+ return CGSize(width: 1, height: 1)+ }++ return CGSize(+ width: screenshotPixelSize.width / windowBounds.width,+ height: screenshotPixelSize.height / windowBounds.height+ )+}++func screenshotPixelToWindowPoint(+ _ point: CGPoint,+ screenshotPixelSize: CGSize?,+ windowBounds: CGRect?+) -> CGPoint {+ let scale = screenshotPixelScale(+ screenshotPixelSize: screenshotPixelSize,+ windowBounds: windowBounds+ )+ return CGPoint(+ x: point.x / scale.width,+ y: point.y / scale.height+ )+}+let nonSettableSetValueErrorMessage = "Cannot set a value for an element that is not settable"func setValueAttributeIsSettable(result: AXError, settable: Bool, attribute: String) throws -> Bool {@@ -206,8 +243,9 @@ public final class ComputerUseService {pulseVisualCursor(at: cursorTarget, clickCount: clickCount, mouseButton: button)} else if let x, let y {- let point = CGPoint(x: x, y: y)- let targetPoint = try screenshotToGlobalPoint(snapshot: snapshot, x: x, y: y)+ let screenshotPoint = CGPoint(x: x, y: y)+ let point = screenshotPixelToWindowPointInSnapshot(snapshot: snapshot, point: screenshotPoint)+ let targetPoint = try windowPointToGlobalPoint(snapshot: snapshot, point: point)let cursorTarget = makeVisualCursorTarget(at: targetPoint,targetWindowID: snapshot.targetWindowID,@@ -231,7 +269,7 @@ public final class ComputerUseService {at: targetPoint,button: button,clickCount: clickCount,- targetDescription: "x=\(Int(point.x)) y=\(Int(point.y))",+ targetDescription: "x=\(Int(screenshotPoint.x)) y=\(Int(screenshotPoint.y))",snapshot: snapshot)}@@ -793,16 +831,53 @@ public final class ComputerUseService {return nil}- return try screenshotToGlobalPoint(snapshot: snapshot, x: frame.midX, y: frame.midY)+ return try windowPointToGlobalPoint(+ snapshot: snapshot,+ point: CGPoint(x: frame.midX, y: frame.midY)+ )}private func screenshotToGlobalPoint(snapshot: AppSnapshot, x: Double, y: Double) throws -> CGPoint {+ try windowPointToGlobalPoint(+ snapshot: snapshot,+ point: screenshotPixelToWindowPointInSnapshot(+ snapshot: snapshot,+ point: CGPoint(x: x, y: y)+ )+ )+ }++ private func screenshotPixelToWindowPointInSnapshot(snapshot: AppSnapshot, point: CGPoint) -> CGPoint {+ screenshotPixelToWindowPoint(+ point,+ screenshotPixelSize: screenshotPixelSize(snapshot: snapshot),+ windowBounds: snapshot.windowBounds+ )+ }++ private func screenshotPixelSize(snapshot: AppSnapshot) -> CGSize? {+ guard+ let screenshotPNGData = snapshot.screenshotPNGData,+ let imageSource = CGImageSourceCreateWithData(screenshotPNGData as CFData, nil),+ let properties = CGImageSourceCopyPropertiesAtIndex(imageSource, 0, nil) as? [CFString: Any],+ let pixelWidth = properties[kCGImagePropertyPixelWidth] as? CGFloat,+ let pixelHeight = properties[kCGImagePropertyPixelHeight] as? CGFloat,+ pixelWidth > 0,+ pixelHeight > 0+ else {+ return nil+ }++ return CGSize(width: pixelWidth, height: pixelHeight)+ }++ private func windowPointToGlobalPoint(snapshot: AppSnapshot, point: CGPoint) throws -> CGPoint {guard let windowBounds = snapshot.windowBounds else {let appReference = snapshot.app.bundleIdentifier ?? snapshot.app.namethrow ComputerUseError.stateUnavailable("No window bounds are available for \(appReference). Run get_app_state after bringing the app on screen.")}- return CGPoint(x: windowBounds.minX + x, y: windowBounds.minY + y)+ return CGPoint(x: windowBounds.minX + point.x, y: windowBounds.minY + point.y)}private func fixtureIdentifier(at point: CGPoint, snapshot: AppSnapshot) throws -> String {diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swiftindex 260fed9..227af7b 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift@@ -70,11 +70,10 @@ enum InputSimulation {throw ComputerUseError.message("Failed to create targeted event source.")}- let eventPoint = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: point)for _ in 0..<max(clickCount, 1) {- try postMouseEventToPid(type: .mouseMoved, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)- try postMouseEventToPid(type: button.downEvent, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)- try postMouseEventToPid(type: button.upEvent, source: source, point: eventPoint, button: button.cgButton, clickState: clickCount, pid: pid)+ try postMouseEventToPid(type: .mouseMoved, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)+ try postMouseEventToPid(type: button.downEvent, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)+ try postMouseEventToPid(type: button.upEvent, source: source, point: point, button: button.cgButton, clickState: clickCount, pid: pid)}}@@ -83,7 +82,7 @@ enum InputSimulation {throw ComputerUseError.message("Failed to create scroll event.")}- event.location = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: point)+ event.location = pointevent.postToPid(pid)Thread.sleep(forTimeInterval: 0.1)}@@ -103,22 +102,19 @@ enum InputSimulation {throw ComputerUseError.message("Failed to create targeted event source.")}- let eventStart = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: start)- let eventEnd = screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: end)-- try postMouseEventToPid(type: .mouseMoved, source: source, point: eventStart, button: .left, clickState: 1, pid: pid)- try postMouseEventToPid(type: .leftMouseDown, source: source, point: eventStart, button: .left, clickState: 1, pid: pid)+ try postMouseEventToPid(type: .mouseMoved, source: source, point: start, button: .left, clickState: 1, pid: pid)+ try postMouseEventToPid(type: .leftMouseDown, source: source, point: start, button: .left, clickState: 1, pid: pid)for step in 1...10 {let progress = CGFloat(step) / 10let point = CGPoint(- x: eventStart.x + ((eventEnd.x - eventStart.x) * progress),- y: eventStart.y + ((eventEnd.y - eventStart.y) * progress)+ x: start.x + ((end.x - start.x) * progress),+ y: start.y + ((end.y - start.y) * progress))try postMouseEventToPid(type: .leftMouseDragged, source: source, point: point, button: .left, clickState: 1, pid: pid)}- try postMouseEventToPid(type: .leftMouseUp, source: source, point: eventEnd, button: .left, clickState: 1, pid: pid)+ try postMouseEventToPid(type: .leftMouseUp, source: source, point: end, button: .left, clickState: 1, pid: pid)}static func dragGlobally(from start: CGPoint, to end: CGPoint) throws {diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 44458bb..c1b876c 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -556,36 +556,55 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(point, CGPoint(x: 2415, y: 1783))}- func testScreenStatePointToAppKitGlobalPointConvertsTargetedMouseCoordinates() {- let point = screenStatePointToAppKitGlobalPoint(- fromScreenStatePoint: CGPoint(x: 2415, y: 181),- screenMappings: [- VisualCursorScreenMapping(- screenStateFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964),- appKitFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964)- ),- ]+ func testScreenshotPixelScaleUsesRetinaSizedImageAgainstWindowBounds() {+ let scale = screenshotPixelScale(+ screenshotPixelSize: CGSize(width: 2048, height: 1266),+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633))- XCTAssertEqual(point, CGPoint(x: 2415, y: 1783))+ XCTAssertEqual(scale.width, 2, accuracy: 0.0001)+ XCTAssertEqual(scale.height, 2, accuracy: 0.0001)}- func testScreenStatePointToAppKitGlobalPointPreservesDisplayOffsets() {- let point = screenStatePointToAppKitGlobalPoint(- fromScreenStatePoint: CGPoint(x: 3320, y: 120),- screenMappings: [- VisualCursorScreenMapping(- screenStateFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964),- appKitFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964)- ),- VisualCursorScreenMapping(- screenStateFrame: CGRect(x: 3024, y: 0, width: 1728, height: 1117),- appKitFrame: CGRect(x: 3024, y: -153, width: 1728, height: 1117)- ),- ]+ func testScreenshotPixelScaleStaysAtOneForUnscaledDisplays() {+ let scale = screenshotPixelScale(+ screenshotPixelSize: CGSize(width: 1024, height: 633),+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633)+ )++ XCTAssertEqual(scale.width, 1, accuracy: 0.0001)+ XCTAssertEqual(scale.height, 1, accuracy: 0.0001)+ }++ func testScreenshotPixelToWindowPointConvertsScreenshotPixelsBackToWindowPoints() {+ let point = screenshotPixelToWindowPoint(+ CGPoint(x: 1060, y: 790),+ screenshotPixelSize: CGSize(width: 2048, height: 1266),+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633)+ )++ XCTAssertEqual(point.x, 530, accuracy: 0.0001)+ XCTAssertEqual(point.y, 395, accuracy: 0.0001)+ }++ func testScreenshotPixelToWindowPointKeepsCoordinatesOnUnscaledDisplays() {+ let point = screenshotPixelToWindowPoint(+ CGPoint(x: 530, y: 395),+ screenshotPixelSize: CGSize(width: 1024, height: 633),+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633)+ )++ XCTAssertEqual(point, CGPoint(x: 530, y: 395))+ }++ func testScreenshotPixelToWindowPointFallsBackToIdentityWithoutImageSize() {+ let point = screenshotPixelToWindowPoint(+ CGPoint(x: 530, y: 395),+ screenshotPixelSize: nil,+ windowBounds: CGRect(x: 1938, y: 236, width: 1024, height: 633))- XCTAssertEqual(point, CGPoint(x: 3320, y: 844))+ XCTAssertEqual(point, CGPoint(x: 530, y: 395))}func testCursorWindowGeometryAnchorsTipPosition() {