Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

fix(cursor): 窗口移动时按实时帧重定位软件光标

wicgee5a4515979853
diff --git a/docs/histories/2026-10/20261010-1745-cursor-live-frame-anchor.md b/docs/histories/2026-10/20261010-1745-cursor-live-frame-anchor.md
new file mode 100644
index 0000000..d35c2ea
--- /dev/null
+++ b/docs/histories/2026-10/20261010-1745-cursor-live-frame-anchor.md
@@ -0,0 +1,27 @@
+# 软件光标:窗口移动时按实时帧重定位(v1.0.0 基线)
+
+## 背景
+
+在 v1.0.0 上重做此前那条跨屏光标修复。双屏环境(内建屏给 Agent、主屏给人)下,目标窗口在动作途中被拖到另一块屏时,软件光标会停在窗口原来的屏幕上。
+
+## 根因
+
+1. `CGWindowListCopyWindowInfo` 在窗口移动后仍返回移动前的 frame(实测连续 83 帧、约 1 秒),只有 AX 元素的位置是即时的。
+2. 光标目标点是在快照时刻算出来的;窗口之后移动,落点随之失效。
+3. **v1.0.0 的点击路径不经过 `visualCursorTarget(for:snapshot:)`**:它在 `ComputerUseService` 的 click 分支里直接调 `makeVisualCursorTarget(at:...)`。因此最初把锚点接在 `visualCursorTarget(for:snapshot:)` 上,单测全过而真机完全无效——诊断日志证明 `register`/`liveFrame` 一次都没被调用。
+
+## 改动
+
+- 新增 `CursorWindowAnchor.swift`:`CursorRestingAnchor`(元素相对窗口帧的锚点,纯函数、可单测)+ `CursorWindowFrameTracker`(AX 元素优先、窗口列表兜底,带测试注入点)。
+- `VisualCursorTarget` 增加 `anchor`;点击分支用 `clickPoint` 得到的窗口内坐标构造锚点,并在构造锚点处注册快照携带的 AX 窗口元素。
+- Overlay:新增 `restingAnchor`、`liveTargetPoint()`、`anchoredWindowMoved()`、`abortTravelOntoObservedWindow()`;`moveCursor` / `settle` / `pulseClick` 均按实时帧重算,行进循环逐帧检查窗口是否移动,移动即中止并落到窗口的新位置。
+
+## 验证
+
+- 单测:`CursorWindowAnchorTests` 7 项(含用本次复现的真实数字构造的用例);全套 199 tests / 7 skipped / 0 failures;`make check-docs` 通过。
+- 真机(双屏):窗口在行进途中从内建屏移到主屏,**连续两次**光标跟随到主屏(光标 `x=1057`,窗口 `x=600`)。
+- 帧来源日志:76–79 次读取全部走 AX 元素,窗口列表兜底 **0** 次——即不再依赖那条会滞后的路径。
+
+## 教训
+
+第一次把锚点接在错误的函数上,单测全绿但真机无效。是**临时诊断**(无条件写日志,先验证调用路径)定位到真实构造点。结论:**先证明代码路径会被走到,再写实现**;单测覆盖不到调用路径本身。
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
index 179019e..cf1060a 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
@@ -6,6 +6,11 @@ import ImageIO
struct VisualCursorTarget: Equatable {
let point: CGPoint
let window: CursorTargetWindow?
+ /// The same point expressed relative to the window frame the snapshot
+ /// reported. The overlay keeps it so a window that moves before or during
+ /// the action can be re-derived instead of drawing on the display the
+ /// window just left.
+ var anchor: CursorRestingAnchor? = nil
}
public enum ClickMethod: String, CaseIterable, Sendable {
@@ -149,18 +154,43 @@ func inputEventPoint(
point
}
+/// Anchor for a cursor target built from a window-local point.
+///
+/// The click paths derive their point from the snapshot, so they carry the
+/// window frame that point belongs to; the overlay needs both to follow the
+/// window if it moves before or during the action.
+func makeCursorWindowAnchor(snapshot: AppSnapshot, windowLocalPoint: CGPoint) -> CursorRestingAnchor? {
+ guard let windowID = snapshot.targetWindowID, let windowBounds = snapshot.windowBounds else {
+ return nil
+ }
+
+ // Registering here keeps the live-frame source next to the anchor that
+ // depends on it: the window list can lag a move by about a second, so the
+ // accessibility element is what makes the re-derivation immediate.
+ CursorWindowFrameTracker.register(windowID: windowID, element: snapshot.windowElement)
+
+ return CursorRestingAnchor(
+ windowID: windowID,
+ layer: snapshot.targetWindowLayer ?? 0,
+ windowLocalPoint: windowLocalPoint,
+ windowBounds: windowBounds
+ )
+}
+
func makeVisualCursorTarget(
at point: CGPoint,
targetWindowID: CGWindowID?,
targetWindowLayer: Int?,
- screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()
+ screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings(),
+ anchor: CursorRestingAnchor? = nil
) -> VisualCursorTarget {
VisualCursorTarget(
point: screenStatePointToAppKitGlobalPoint(
fromScreenStatePoint: point,
screenMappings: screenMappings
),
- window: targetWindowID.map { CursorTargetWindow(windowID: $0, layer: targetWindowLayer ?? 0) }
+ window: targetWindowID.map { CursorTargetWindow(windowID: $0, layer: targetWindowLayer ?? 0) },
+ anchor: anchor
)
}
@@ -183,7 +213,15 @@ func makeVisualCursorTarget(
at: point,
targetWindowID: targetWindowID,
targetWindowLayer: targetWindowLayer,
- screenMappings: screenMappings
+ screenMappings: screenMappings,
+ anchor: targetWindowID.map { windowID in
+ CursorRestingAnchor(
+ windowID: windowID,
+ layer: targetWindowLayer ?? 0,
+ windowLocalPoint: CGPoint(x: localFrame.midX, y: localFrame.midY),
+ windowBounds: windowBounds
+ )
+ }
)
}
@@ -625,7 +663,8 @@ public final class ComputerUseService {
let cursorTarget = makeVisualCursorTarget(
at: targetPoint,
targetWindowID: snapshot.targetWindowID,
- targetWindowLayer: snapshot.targetWindowLayer
+ targetWindowLayer: snapshot.targetWindowLayer,
+ anchor: makeCursorWindowAnchor(snapshot: snapshot, windowLocalPoint: windowPoint)
)
moveVisualCursor(to: cursorTarget)
@@ -686,7 +725,8 @@ public final class ComputerUseService {
let cursorTarget = makeVisualCursorTarget(
at: targetPoint,
targetWindowID: snapshot.targetWindowID,
- targetWindowLayer: snapshot.targetWindowLayer
+ targetWindowLayer: snapshot.targetWindowLayer,
+ anchor: makeCursorWindowAnchor(snapshot: snapshot, windowLocalPoint: point)
)
moveVisualCursor(to: cursorTarget)
@@ -1943,7 +1983,15 @@ public final class ComputerUseService {
}
private func visualCursorTarget(for record: ElementRecord, snapshot: AppSnapshot) -> VisualCursorTarget? {
- makeVisualCursorTarget(
+ // The accessibility element is the overlay's only immediate source of
+ // truth for the window frame: the window list can keep reporting the
+ // pre-move origin for about a second.
+ CursorWindowFrameTracker.register(
+ windowID: snapshot.targetWindowID,
+ element: snapshot.windowElement
+ )
+
+ return makeVisualCursorTarget(
localFrame: record.localFrame,
windowBounds: snapshot.windowBounds,
targetWindowID: snapshot.targetWindowID,
@@ -1962,7 +2010,7 @@ public final class ComputerUseService {
}
VisualCursorSupport.performOnMain {
- SoftwareCursorOverlay.moveCursor(to: target.point, in: target.window)
+ SoftwareCursorOverlay.moveCursor(to: target.point, in: target.window, anchor: target.anchor)
}
}
@@ -1972,7 +2020,7 @@ public final class ComputerUseService {
}
VisualCursorSupport.performOnMain {
- SoftwareCursorOverlay.settle(at: target.point, in: target.window)
+ SoftwareCursorOverlay.settle(at: target.point, in: target.window, anchor: target.anchor)
}
}
@@ -1986,7 +2034,8 @@ public final class ComputerUseService {
at: target.point,
clickCount: clickCount,
mouseButton: mouseButton,
- in: target.window
+ in: target.window,
+ anchor: target.anchor
)
}
}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorWindowAnchor.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorWindowAnchor.swift
new file mode 100644
index 0000000..4903e26
--- /dev/null
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorWindowAnchor.swift
@@ -0,0 +1,147 @@
+import ApplicationServices
+import CoreGraphics
+import Foundation
+
+/// The point a cursor target was derived from, expressed relative to the
+/// window frame the snapshot reported.
+///
+/// A window that moves between the snapshot and the action keeps its
+/// window-local geometry, so this is what lets the overlay re-derive the point
+/// from the window's live frame instead of drawing on the display the window
+/// just left.
+struct CursorRestingAnchor: Equatable, Sendable {
+ let windowID: CGWindowID
+ let layer: Int
+ /// Element centre relative to the window's top-left corner, screen-state space.
+ let windowLocalPoint: CGPoint
+ /// Window frame the anchor was derived from, screen-state space.
+ let windowBounds: CGRect
+
+ /// The anchored point re-expressed against a live window frame.
+ ///
+ /// Returns `nil` when the live frame carries no usable size, so callers can
+ /// fall back to the point they already have.
+ func screenStatePoint(liveFrame: CGRect) -> CGPoint? {
+ guard liveFrame.width > 0, liveFrame.height > 0 else {
+ return nil
+ }
+
+ let localX = windowLocalPoint.x.clamped(to: 0...liveFrame.width)
+ let localY = windowLocalPoint.y.clamped(to: 0...liveFrame.height)
+ return CGPoint(x: liveFrame.minX + localX, y: liveFrame.minY + localY)
+ }
+}
+
+/// Live frames for target windows.
+///
+/// `CGWindowListCopyWindowInfo` keeps reporting the pre-move frame for about a
+/// second after a window is dragged to another display — measured while fixing
+/// this: 83 identical frames at the cursor travel's 120 Hz pump. The
+/// accessibility element the snapshot already holds reports the move
+/// immediately, so it is the source of truth here and the window list is only a
+/// fallback for windows the snapshot never registered.
+enum CursorWindowFrameTracker {
+ /// Lock-protected box so the tracker can be used from the service thread
+ /// and the overlay's main-thread pump alike.
+ private final class State: @unchecked Sendable {
+ let lock = NSLock()
+ var elements: [CGWindowID: AXUIElement] = [:]
+ /// Test seam: consulted before any real lookup.
+ var override: ((CGWindowID) -> CGRect?)?
+ }
+
+ private static let state = State()
+
+ static func register(windowID: CGWindowID?, element: AXUIElement?) {
+ guard let windowID, let element else {
+ return
+ }
+
+ state.lock.lock()
+ defer { state.lock.unlock() }
+ state.elements[windowID] = element
+ }
+
+ static func forget(windowID: CGWindowID) {
+ state.lock.lock()
+ defer { state.lock.unlock() }
+ state.elements.removeValue(forKey: windowID)
+ }
+
+ static func forgetAll() {
+ state.lock.lock()
+ defer { state.lock.unlock() }
+ state.elements.removeAll()
+ }
+
+ static func installFrameOverrideForTesting(_ body: ((CGWindowID) -> CGRect?)?) {
+ state.lock.lock()
+ defer { state.lock.unlock() }
+ state.override = body
+ }
+
+ static func liveFrame(for windowID: CGWindowID) -> CGRect? {
+ state.lock.lock()
+ let body = state.override
+ state.lock.unlock()
+
+ if let body {
+ return body(windowID)
+ }
+
+ return accessibilityFrame(for: windowID) ?? windowListFrame(for: windowID)
+ }
+
+ /// Frame read from the registered accessibility element, which reflects a
+ /// move immediately.
+ static func accessibilityFrame(for windowID: CGWindowID) -> CGRect? {
+ state.lock.lock()
+ let element = state.elements[windowID]
+ state.lock.unlock()
+
+ guard let element else {
+ return nil
+ }
+
+ var positionValue: CFTypeRef?
+ var sizeValue: CFTypeRef?
+ guard AXUIElementCopyAttributeValue(element, kAXPositionAttribute as CFString, &positionValue) == .success,
+ AXUIElementCopyAttributeValue(element, kAXSizeAttribute as CFString, &sizeValue) == .success,
+ let positionValue,
+ let sizeValue
+ else {
+ return nil
+ }
+
+ var position = CGPoint.zero
+ var size = CGSize.zero
+ guard AXValueGetValue(positionValue as! AXValue, .cgPoint, &position),
+ AXValueGetValue(sizeValue as! AXValue, .cgSize, &size),
+ size.width > 0,
+ size.height > 0
+ else {
+ return nil
+ }
+
+ return CGRect(origin: position, size: size)
+ }
+
+ /// Fallback frame from the window server. Kept last because it can report
+ /// the pre-move origin for about a second.
+ static func windowListFrame(for windowID: CGWindowID) -> CGRect? {
+ guard let info = CGWindowListCopyWindowInfo(
+ [.optionIncludingWindow, .excludeDesktopElements],
+ windowID
+ ) as? [[String: Any]],
+ let entry = info.first,
+ let boundsDictionary = entry[kCGWindowBounds as String] as? [String: Any],
+ let frame = CGRect(dictionaryRepresentation: boundsDictionary as CFDictionary),
+ frame.width > 0,
+ frame.height > 0
+ else {
+ return nil
+ }
+
+ return frame
+ }
+}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
index 2dfe3ad..791e39d 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
@@ -192,13 +192,20 @@ enum SoftwareCursorOverlay {
private static var restingTipPosition: CGPoint?
private static var displayedTipPosition: CGPoint?
private static var activeTargetWindow: CursorTargetWindow?
+ /// Window frame the current target point was derived from; lets a move
+ /// under the cursor be detected and re-derived instead of ignored.
+ private static var restingAnchor: CursorRestingAnchor?
private static var visualDynamicsState: CursorVisualDynamicsState?
private static var idleTimer: Timer?
private static var hideTimer: Timer?
private static var idlePhase: CGFloat = 0
private static var observationPhase = "hidden"
- static func moveCursor(to targetPoint: CGPoint, in targetWindow: CursorTargetWindow?) {
+ static func moveCursor(
+ to targetPoint: CGPoint,
+ in targetWindow: CursorTargetWindow?,
+ anchor: CursorRestingAnchor? = nil
+ ) {
guard VisualCursorSupport.isEnabled, canPresentOverlay else {
return
}
@@ -208,7 +215,8 @@ enum SoftwareCursorOverlay {
cancelPendingHide()
configureOrdering(relativeTo: targetWindow)
- let constrainedTarget = clampTipPosition(targetPoint)
+ restingAnchor = anchor
+ let constrainedTarget = clampTipPosition(liveTargetPoint(targetPoint, anchor: anchor))
let isFreshStart = displayedTipPosition == nil
let startPoint = displayedTipPosition ?? defaultInitialTipPosition()
let now = CACurrentMediaTime()
@@ -234,13 +242,20 @@ enum SoftwareCursorOverlay {
}
}
- static func pulseClick(at targetPoint: CGPoint, clickCount: Int, mouseButton: MouseButtonKind, in targetWindow: CursorTargetWindow?) {
+ static func pulseClick(
+ at targetPoint: CGPoint,
+ clickCount: Int,
+ mouseButton: MouseButtonKind,
+ in targetWindow: CursorTargetWindow?,
+ anchor: CursorRestingAnchor? = nil
+ ) {
guard VisualCursorSupport.isEnabled, canPresentOverlay else {
return
}
configureOrdering(relativeTo: targetWindow)
- let constrainedTarget = clampTipPosition(targetPoint)
+ restingAnchor = anchor ?? restingAnchor
+ let constrainedTarget = clampTipPosition(liveTargetPoint(targetPoint, anchor: anchor))
let now = CACurrentMediaTime()
seedVisualDynamicsIfNeeded(at: constrainedTarget, time: now)
restingTipPosition = constrainedTarget
@@ -250,13 +265,18 @@ enum SoftwareCursorOverlay {
scheduleHide(after: visualCursorPostInteractionIdleTimeout())
}
- static func settle(at targetPoint: CGPoint, in targetWindow: CursorTargetWindow?) {
+ static func settle(
+ at targetPoint: CGPoint,
+ in targetWindow: CursorTargetWindow?,
+ anchor: CursorRestingAnchor? = nil
+ ) {
guard VisualCursorSupport.isEnabled, canPresentOverlay else {
return
}
configureOrdering(relativeTo: targetWindow)
- let constrainedTarget = clampTipPosition(targetPoint)
+ restingAnchor = anchor ?? restingAnchor
+ let constrainedTarget = clampTipPosition(liveTargetPoint(targetPoint, anchor: anchor))
restingTipPosition = constrainedTarget
observationPhase = "settling"
placeCursor(
@@ -276,12 +296,67 @@ enum SoftwareCursorOverlay {
displayedTipPosition = nil
restingTipPosition = nil
activeTargetWindow = nil
+ restingAnchor = nil
visualDynamicsState = nil
observationPhase = "hidden"
writeObservationSnapshot(tipPosition: nil, rotation: nil)
panel?.orderOut(nil)
}
+ /// Re-derive a target point that was computed from a snapshot taken before
+ /// the window moved. Element frames are window-relative, so the anchored
+ /// point follows the window instead of staying on the display it just left.
+ private static func liveTargetPoint(_ targetPoint: CGPoint, anchor: CursorRestingAnchor?) -> CGPoint {
+ guard let anchor,
+ let liveFrame = CursorWindowFrameTracker.liveFrame(for: anchor.windowID),
+ liveFrame != anchor.windowBounds,
+ let screenStatePoint = anchor.screenStatePoint(liveFrame: liveFrame)
+ else {
+ return targetPoint
+ }
+
+ return screenStatePointToAppKitGlobalPoint(
+ fromScreenStatePoint: screenStatePoint,
+ screenMappings: currentVisualCursorScreenMappings()
+ )
+ }
+
+ /// Whether the anchored window has moved since the anchor was taken.
+ private static func anchoredWindowMoved() -> Bool {
+ guard let anchor = restingAnchor,
+ let liveFrame = CursorWindowFrameTracker.liveFrame(for: anchor.windowID)
+ else {
+ return false
+ }
+
+ return liveFrame != anchor.windowBounds
+ }
+
+ /// Stop a travel whose window moved under it: land the cursor on the
+ /// window's live frame instead of finishing the path towards the display
+ /// the window just left.
+ private static func abortTravelOntoObservedWindow() {
+ guard let anchor = restingAnchor,
+ let liveFrame = CursorWindowFrameTracker.liveFrame(for: anchor.windowID),
+ let screenStatePoint = anchor.screenStatePoint(liveFrame: liveFrame)
+ else {
+ return
+ }
+
+ let constrainedTarget = clampTipPosition(
+ screenStatePointToAppKitGlobalPoint(
+ fromScreenStatePoint: screenStatePoint,
+ screenMappings: currentVisualCursorScreenMappings()
+ )
+ )
+ restingTipPosition = constrainedTarget
+ observationPhase = "reanchored"
+ placeCursor(
+ using: advanceVisualDynamics(toward: constrainedTarget, at: CACurrentMediaTime()),
+ clickProgress: 0
+ )
+ }
+
private static var canPresentOverlay: Bool {
!NSScreen.screens.isEmpty
}
@@ -363,6 +438,11 @@ enum SoftwareCursorOverlay {
while true {
refreshActiveOrderingIfNeeded()
+ if anchoredWindowMoved() {
+ abortTravelOntoObservedWindow()
+ return
+ }
+
let elapsed = CGFloat(CACurrentMediaTime() - startTime)
let normalizedElapsed = (elapsed / max(duration, 0.001)).clamped(to: 0...1)
let springTime = normalizedElapsed * springTargetDuration
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowAnchorTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowAnchorTests.swift
new file mode 100644
index 0000000..8b0e1a7
--- /dev/null
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowAnchorTests.swift
@@ -0,0 +1,74 @@
+import CoreGraphics
+import XCTest
+@testable import OpenComputerUseKit
+
+/// Element frames are window-relative, so a cursor point derived from a
+/// snapshot stops being valid the moment the window moves. These numbers are
+/// the ones the bug was reproduced with: the same window on the built-in
+/// display (`x = -1400`) and on the main display (`x = 600`).
+final class CursorWindowAnchorTests: XCTestCase {
+ private let builtInDisplayFrame = CGRect(x: -1400, y: 200, width: 500, height: 400)
+ private let mainDisplayFrame = CGRect(x: 600, y: 200, width: 500, height: 400)
+
+ private func anchor() -> CursorRestingAnchor {
+ CursorRestingAnchor(
+ windowID: 4242,
+ layer: 0,
+ windowLocalPoint: CGPoint(x: 250, y: 200),
+ windowBounds: builtInDisplayFrame
+ )
+ }
+
+ func testAnchorStaysPutWhileTheWindowDoesNotMove() {
+ let point = anchor().screenStatePoint(liveFrame: builtInDisplayFrame)
+ XCTAssertEqual(point, CGPoint(x: -1150, y: 400))
+ }
+
+ /// The reported bug: the window is dragged to the main display while the
+ /// cursor is travelling, and the cursor finishes on the display the window
+ /// just left.
+ func testAnchorFollowsTheWindowOntoTheOtherDisplay() {
+ let point = anchor().screenStatePoint(liveFrame: mainDisplayFrame)
+ XCTAssertEqual(point, CGPoint(x: 850, y: 400))
+ }
+
+ /// Only the frame origin moves; the window-local offset must survive, so a
+ /// click still lands on the same control inside the window.
+ func testAnchorPreservesTheWindowLocalOffset() {
+ let moved = CGRect(x: -1400, y: -600, width: 500, height: 400)
+ let point = anchor().screenStatePoint(liveFrame: moved)
+ XCTAssertEqual(point, CGPoint(x: -1150, y: -400))
+ }
+
+ /// A window that shrank must not put the cursor outside itself.
+ func testAnchorClampsIntoASmallerLiveFrame() {
+ let shrunk = CGRect(x: 600, y: 200, width: 120, height: 80)
+ let point = anchor().screenStatePoint(liveFrame: shrunk)
+ XCTAssertEqual(point, CGPoint(x: 720, y: 280))
+ }
+
+ /// No usable size means no re-derivation: the caller keeps the point it has
+ /// rather than guessing.
+ func testAnchorRejectsALiveFrameWithoutSize() {
+ XCTAssertNil(anchor().screenStatePoint(liveFrame: .zero))
+ XCTAssertNil(anchor().screenStatePoint(liveFrame: CGRect(x: 10, y: 10, width: 0, height: 40)))
+ }
+
+ func testTrackerPrefersTheRegisteredLiveFrame() {
+ CursorWindowFrameTracker.installFrameOverrideForTesting { windowID in
+ windowID == 4242 ? self.mainDisplayFrame : nil
+ }
+ defer { CursorWindowFrameTracker.installFrameOverrideForTesting(nil) }
+
+ XCTAssertEqual(CursorWindowFrameTracker.liveFrame(for: 4242), mainDisplayFrame)
+ XCTAssertNil(CursorWindowFrameTracker.liveFrame(for: 1))
+ }
+
+ /// Registering nothing must not produce a frame out of thin air; the
+ /// window list is the only remaining source and an unknown id has no entry.
+ func testTrackerReportsNothingForAnUnknownWindow() {
+ CursorWindowFrameTracker.forgetAll()
+
+ XCTAssertNil(CursorWindowFrameTracker.liveFrame(for: CGWindowID(0)))
+ }
+}
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 443772e..564acee 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -2006,11 +2006,20 @@ final class OpenComputerUseKitTests: XCTestCase {
screenMappings: screenMappings
)
+ // The target also carries the anchor: the element centre relative to
+ // the window frame it was derived from, which is what lets the overlay
+ // follow the window if it moves before the action lands.
XCTAssertEqual(
target,
VisualCursorTarget(
point: CGPoint(x: 484, y: 724),
- window: CursorTargetWindow(windowID: 321, layer: 8)
+ window: CursorTargetWindow(windowID: 321, layer: 8),
+ anchor: CursorRestingAnchor(
+ windowID: 321,
+ layer: 8,
+ windowLocalPoint: CGPoint(x: 84, y: 56),
+ windowBounds: CGRect(x: 400, y: 220, width: 900, height: 640)
+ )
)
)
}
diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.md
index 40a896e..5c8426c 100644
--- a/skills/open-computer-use/references/usage.md
+++ b/skills/open-computer-use/references/usage.md
@@ -228,6 +228,18 @@ Every non-fixture `drag` result includes a text item that begins `Drag delivered
When the gate is not enabled, treat window-server drags as unavailable and reach the same outcome another way: copy or move files with a shell command instead of a Finder drag, use `set_value` or keyboard selection instead of drag-selecting text, and use the app's own window controls instead of dragging a title bar.
+## Software Cursor
+
+The software cursor follows the target window when it moves. Its point is derived from the snapshot, so it
+also carries an anchor: the same point expressed relative to the window frame the snapshot reported. Before
+drawing, and on every frame of a travel, the overlay re-derives the point from the window's live frame, so a
+window dragged to another display mid-action takes the cursor with it instead of leaving it on the display
+the window just left.
+
+The live frame comes from the window's accessibility element — the element the snapshot already holds — and
+only falls back to `CGWindowListCopyWindowInfo`, which can keep reporting the pre-move origin for about a
+second (measured: 83 identical frames at the travel's 120 Hz pump).
+
## Platform Notes
### macOS