Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

fix(cursor): follow live windows throughout the overlay lifecycle

ifuryst9bc2bd762d4f
diff --git a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift
index 7c9ab1d..677d935 100644
--- a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift
+++ b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift
@@ -196,8 +196,13 @@ enum OpenComputerUseSmokeSuite {
try client.initialize()
let tools = try client.listTools()
- guard tools.count == 9 else {
- throw SmokeError.message("Expected 9 tools, got \(tools.count)")
+ let expectedTools: Set<String> = [
+ "list_apps", "get_app_state", "query", "click", "perform_secondary_action",
+ "scroll", "drag", "type_text", "press_key", "set_value",
+ ]
+ let toolNames = Set(tools.compactMap { $0["name"] as? String })
+ guard tools.count == expectedTools.count, toolNames == expectedTools else {
+ throw SmokeError.message("Expected tools \(expectedTools.sorted()), got \(toolNames.sorted())")
}
print("1. list_apps")
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 85ee038..9f72683 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -75,6 +75,7 @@
- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。
- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。
- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;首次显示时按官方 binary 的 fresh state 从 AppKit 全局 `(0,0)` window origin 生成起点,后续动作继续复用上一帧 visible tip。真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse 和更明显但仍然很小的 rotate wobble,`set_value` 则只做 settle / idle,不给 pulse;两者收尾后会在目标点继续保持 idle 状态,等待下一次动作时 tip 保持 anchored、只保留可感知的小角度摆动;只有连续 30 秒没有新动作时才做 cleanup,这样连续 tool call 不会反复从 fresh `(0,0)` 起步;如果宿主在任务 / turn 结束时发出 `turn-ended`,cursor 会立即消失并清掉本轮位置状态。
+- overlay 用目标点相对窗口的锚点跟随窗口移动;行进、settle、点击反馈和 idle 都读取最新缓存帧,发现位置变化时直接重定位可见 tip 并清掉旧速度。窗口帧在后台串行读取,最多 30 Hz、同时最多一个请求,避免 AX 阻塞渲染线程;优先读取快照持有的 AX 窗口,仅在没有 AX 元素时使用窗口列表,读取失败保留最后有效帧。只保留当前窗口的追踪状态,reset / idle 超时后释放;这层跟随只调整显示位置,输入投递仍走既有路径。
- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在和 `CursorMotion` 一样优先渲染仓库里沉淀的 `official-software-cursor-window-252.png` baseline,只有资源缺失时才退回 `OpenComputerUseKit` 内部的程序化 pointer/fog fallback。命中点 anchor 仍固定在 `126x126` 画布里的同一组 tip-offset 上;glyph 自身的 neutral heading 继续沿用 `CursorMotion` / 官方 baseline 的 `-3π/4`。主 runtime overlay window 按 AppKit 全局坐标移动,因此在把 AX / `CGWindowList` 产出的 y-down screen-space 点击目标喂给 overlay 之前,会先转换成对应屏幕的 AppKit 全局坐标;路径选路用屏幕上实际可见的 AppKit forward heading,进入 visual dynamics / render state 前则把 velocity 的 y 轴翻回 CursorMotion 的 y-down screen state,再交给 AppKit 绘制层做角度和 `dy` 翻转。程序化 fallback 保留 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。
- overlay 的层级不再固定 `.floating`;现在会跟随 snapshot 命中的目标 window id / layer,把自己排到该目标 window 之上,而不是粗暴压到所有前台 app 最上层。
- overlay 的曲线路径不再只按固定 Bezier 模板生成;当前主线采用 reverse-engineering 约束下的 heading-driven candidate 族,候选只保留 `direct` / `turn` / `brake` / `orbit` 这些能稳定产出单侧主弧的 family,并继续保留 target-window 命中策略作为同类候选间的 tie-break。原始 binary lift 恢复出来的 `20` 条路径和 score 仍然保留在独立的 `StandaloneCursor` viewer / Python 重建脚本里,用于对照分析,不再直接作为 runtime 默认 chooser。
diff --git a/docs/exec-plans/completed/20261017-pr95-live-cursor-follow.md b/docs/exec-plans/completed/20261017-pr95-live-cursor-follow.md
new file mode 100644
index 0000000..ae3d9f9
--- /dev/null
+++ b/docs/exec-plans/completed/20261017-pr95-live-cursor-follow.md
@@ -0,0 +1,32 @@
+# PR #95: verify live cursor following on current main
+
+## Objective
+
+Review #95 against current main, reproduce remaining cursor-follow defects, fix them on the contributor branch and report merge readiness. Preserve the original contribution. Do not merge the PR as part of this task.
+
+## Scope
+
+- Follow the live window frame during travel, settle, click feedback and idle.
+- Keep Accessibility IPC out of the overlay render loop; bound tracking state and outstanding reads.
+- Preserve input delivery and window focus policies. This is visual cursor positioning, not a new input-routing mode.
+- Keep existing motion dynamics for stationary windows and clear tracking at cursor/session cleanup.
+
+## Progress
+
+- [x] Fetch PR head and merge current main without conflicts.
+- [x] Reproduce stale idle/click targets with production-overlay regression tests.
+- [x] Fix live placement, tracking lifecycle and bounded frame acquisition.
+- [x] Run Swift/Node checks, smoke and isolated live-window validation where available.
+- [x] Update history/docs and deliver follow-up commits on the existing PR branch.
+
+## Validation
+
+Exercise the actual overlay loop with controlled frame changes, including repeated moves, a move back to the original frame, unanchored targets and reset. Verify slow frame reads do not block rendering or accumulate work. Use only isolated test windows for live checks; do not move personal app windows or synthesize user typing. Distinguish injected tests from real Accessibility frame observations in the final evidence.
+
+## Evidence and decisions
+
+- The original overlay failed three of four new regression tests (16 assertions). Fixes now cover idle/pulse targets, the actual visible travel endpoint, stale anchors and orphan idle timers.
+- Frame reads run on a serial worker at up to 30 Hz with one outstanding request. Retain only the current source and last valid frame; discard results for replaced sources.
+- Swift: 351 tests, 8 skipped, 0 failures. Node: 36 passed. Fixture action smoke and cursor idle smoke passed after updating its old tool-list assertion for main's query tool.
+- The production overlay followed an isolated real AX window during travel, pulse and idle. Only one physical display was available. The full service-click test explicitly skips on this machine because the XCTest host does not satisfy the installed Locked Use identity check; retain that protection.
+- No input-routing changes or version bump. Preserve the contributor's original commit and update the existing PR.
diff --git a/docs/histories/2026-10/20261010-1745-cursor-live-frame-anchor.md b/docs/histories/2026-10/20261010-1745-cursor-live-frame-anchor.md
index d35c2ea..055c8c7 100644
--- a/docs/histories/2026-10/20261010-1745-cursor-live-frame-anchor.md
+++ b/docs/histories/2026-10/20261010-1745-cursor-live-frame-anchor.md
@@ -25,3 +25,26 @@
## 教训
第一次把锚点接在错误的函数上,单测全绿但真机无效。是**临时诊断**(无条件写日志,先验证调用路径)定位到真实构造点。结论:**先证明代码路径会被走到,再写实现**;单测覆盖不到调用路径本身。
+
+## 2026-10-17:基于当前 main 复查 #95
+
+用户要求在当前 main 上重新检查和测试,修复之前仍有问题的跟随行为,并补充提交到原 PR。先保留原提交,再合入 main 的 1.1.1 基线。
+
+### 补充修复
+
+- 原实现的 idle timer 和 click pulse 保存旧落点,窗口移动后仍在旧位置绘制。现在行进、settle、pulse 和 idle 共用最新缓存帧,并能处理连续移动和返回原位置。
+- 行进中止原本只更新 spring 目标,旧速度仍让可见 tip 留在旧位置。现在同时重置 dynamics 并直接绘制新位置;窗口在行进开始前已移动时,仍保留正常行进动画。
+- 新的无锚点目标会清掉旧窗口追踪。开始 idle / pulse 前停止旧计时器,避免重复 timer 在 reset 后继续绘制。
+- AX 查询改为后台串行读取:最多 30 Hz、同时一个请求,渲染只读缓存。慢窗口不阻塞主线程,也不会堆积请求。
+- 只保留当前窗口的 AX 引用;切换窗口和清理后丢弃在途旧结果。短暂读取失败保留最后有效帧,不回退到旧快照或滞后的窗口列表。非法 / 无限 frame 被拒绝。
+- 同步架构、skill 使用说明和功能记录。冒烟脚本的旧 9-tool 数量断言不兼容 main 新增的 query,改为检查完整的 10 个工具名称;现有 fixture 动作覆盖仍为原 9 个工具。
+
+### 验证与边界
+
+- 修复前在原 overlay 上跑新增回归,4 项中的 3 项失败、共 16 个断言失败,复现 idle / pulse 和可见 tip 问题。
+- 完整 Swift:351 项,8 项跳过,0 失败;原基线为 341 项、7 项跳过、0 失败。
+- Node REPL / CLI / 配置:36 项通过。完整 fixture 动作 smoke 和 cursor idle smoke 通过;文档检查与 diff whitespace 检查通过。
+- 新测试直接驱动生产 overlay,覆盖连续移动、返回原位置、双击期间移动、行进中止、无锚点目标、reset 和行进前已经移动;后台读取测试覆盖慢响应、请求不堆积、旧结果丢弃和读取失败保留缓存。
+- 真实独立 NSWindow 的 AX 测试通过,包含行进、双击反馈、idle 和当前所有显示器。当前测试机仅一块显示器,未重做物理双屏拖动;上文双屏实测属于原贡献者的验证。
+- 新增完整 ComputerUseService.click 集成测试,但本机 XCTest 宿主未通过已有 Locked Use 身份校验,明确跳过;没有绕过保护。这项跳过与 fixture smoke、真实窗口 AX / overlay 测试区分记录。
+- 该修改调整软件光标显示位置,不改变输入路由,也不处理窗口内部控件重新排版后的定位。
diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.md
index c50f580..33f366c 100644
--- a/docs/releases/feature-release-notes.md
+++ b/docs/releases/feature-release-notes.md
@@ -4,6 +4,7 @@
| 日期 | 功能域 | 用户价值 | 变更摘要 |
| --- | --- | --- | --- |
+| 2026-10-17 | macOS 软件光标窗口跟随 | 目标窗口移动后,行进、点击反馈和等待中的软件光标继续跟随,避免停在旧位置。 | 接入窗口相对锚点,修复 idle / pulse 的旧落点、行进中止后的可见位置及重复计时器;AX 帧读取改为后台串行缓存,限制为最多 30 Hz / 一个请求,并在清理时释放追踪。 |
| 2026-10-15 | 1.1.1 修复发布 | Locked Use 安装更新更可靠,降低认证与后台 IPC 开销。 | 统一版本为 1.1.1,发布授权宿主、安装加载检查、旧组件卸载与连接生命周期修复;完整双屏遮罩、自动解锁、点击验证和重锁实测通过。 |
| 2026-10-15 | Locked Use 安装与性能修复 | 减少认证和 IPC 开销,旧系统组件也能正常移除并重装。 | 非 UI 插件使用 privileged host,安装前验证真实加载;修复系统缓存卸载与转交 socket 的生命周期,降低空闲唤醒,status 提示组件版本差异。 |
| 2026-10-14 | 1.1.0 正式发布 | 用户可直接安装 Locked Use,在锁屏遮罩期间继续 GUI 操作,也可用定向查询和持久截图配置减少工具开销。 | 统一版本为 1.1.0;签名 universal macOS App 内嵌服务、Guardian、安装器和授权插件,普通 enable 请求权限,补齐冷启动握手隔离和发布制品检查;新增 app.query()、doctor --json 与 ocu config。 |
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorWindowAnchor.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorWindowAnchor.swift
index 4903e26..299ce42 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorWindowAnchor.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorWindowAnchor.swift
@@ -22,7 +22,7 @@ struct CursorRestingAnchor: Equatable, Sendable {
/// Returns `nil` when the live frame carries no usable size, so callers can
/// fall back to the point they already have.
func screenStatePoint(liveFrame: CGRect) -> CGPoint? {
- guard liveFrame.width > 0, liveFrame.height > 0 else {
+ guard Self.isUsable(liveFrame), windowLocalPoint.x.isFinite, windowLocalPoint.y.isFinite else {
return nil
}
@@ -30,6 +30,11 @@ struct CursorRestingAnchor: Equatable, Sendable {
let localY = windowLocalPoint.y.clamped(to: 0...liveFrame.height)
return CGPoint(x: liveFrame.minX + localX, y: liveFrame.minY + localY)
}
+
+ static func isUsable(_ frame: CGRect) -> Bool {
+ !frame.isInfinite && !frame.isNull && frame.origin.x.isFinite && frame.origin.y.isFinite &&
+ frame.width.isFinite && frame.height.isFinite && frame.width > 0 && frame.height > 0
+ }
}
/// Live frames for target windows.
@@ -38,40 +43,74 @@ struct CursorRestingAnchor: Equatable, Sendable {
/// second after a window is dragged to another display — measured while fixing
/// this: 83 identical frames at the cursor travel's 120 Hz pump. The
/// accessibility element the snapshot already holds reports the move
-/// immediately, so it is the source of truth here and the window list is only a
-/// fallback for windows the snapshot never registered.
+/// immediately, so it is the preferred source. Read it on a serial worker,
+/// cache the last usable frame, and limit requests to 30 Hz. The window list is
+/// used only when no accessibility element was registered; a failed AX read
+/// must not replace a good frame with the window list's stale origin.
enum CursorWindowFrameTracker {
+ private final class Source: @unchecked Sendable {
+ let windowID: CGWindowID
+ let element: AXUIElement?
+ init(windowID: CGWindowID, element: AXUIElement?) {
+ self.windowID = windowID
+ self.element = element
+ }
+ }
+
/// Lock-protected box so the tracker can be used from the service thread
/// and the overlay's main-thread pump alike.
private final class State: @unchecked Sendable {
let lock = NSLock()
- var elements: [CGWindowID: AXUIElement] = [:]
+ // Only one software cursor is displayed. Do not retain every window
+ // ever observed by a long-lived App Agent.
+ var source: Source?
+ var frame: CGRect?
+ var nextRefresh: TimeInterval = 0
+ var refreshing = false
/// Test seam: consulted before any real lookup.
var override: ((CGWindowID) -> CGRect?)?
+ var reader: (@Sendable (CGWindowID) -> CGRect?)?
}
private static let state = State()
+ private static let queue = DispatchQueue(label: "com.ifuryst.ocu.cursor-frame", qos: .userInitiated)
+ private static let refreshInterval: TimeInterval = 1 / 30
static func register(windowID: CGWindowID?, element: AXUIElement?) {
- guard let windowID, let element else {
+ guard let windowID else {
return
}
state.lock.lock()
defer { state.lock.unlock() }
- state.elements[windowID] = element
+ if let existing = state.source, existing.windowID == windowID {
+ switch (existing.element, element) {
+ case let (lhs?, rhs?) where CFEqual(lhs, rhs): return
+ case (nil, nil): return
+ default: break
+ }
+ }
+ state.source = Source(windowID: windowID, element: element)
+ state.frame = nil
+ state.nextRefresh = 0
}
static func forget(windowID: CGWindowID) {
state.lock.lock()
defer { state.lock.unlock() }
- state.elements.removeValue(forKey: windowID)
+ if state.source?.windowID == windowID {
+ state.source = nil
+ state.frame = nil
+ }
}
static func forgetAll() {
state.lock.lock()
defer { state.lock.unlock() }
- state.elements.removeAll()
+ state.source = nil
+ state.frame = nil
+ // An in-flight IPC read cannot be cancelled safely. Keep its slot busy
+ // until it returns; its result will be discarded by source identity.
}
static func installFrameOverrideForTesting(_ body: ((CGWindowID) -> CGRect?)?) {
@@ -80,35 +119,61 @@ enum CursorWindowFrameTracker {
state.override = body
}
- static func liveFrame(for windowID: CGWindowID) -> CGRect? {
+ static func installFrameReaderForTesting(_ body: (@Sendable (CGWindowID) -> CGRect?)?) {
state.lock.lock()
- let body = state.override
- state.lock.unlock()
+ defer { state.lock.unlock() }
+ state.reader = body
+ }
- if let body {
+ static func liveFrame(for windowID: CGWindowID) -> CGRect? {
+ state.lock.lock()
+ if let body = state.override {
+ state.lock.unlock()
return body(windowID)
}
-
- return accessibilityFrame(for: windowID) ?? windowListFrame(for: windowID)
+ guard let source = state.source, source.windowID == windowID else {
+ state.lock.unlock()
+ return nil
+ }
+ let frame = state.frame
+ let now = ProcessInfo.processInfo.systemUptime
+ let refresh = !state.refreshing && now >= state.nextRefresh
+ let reader = state.reader
+ if refresh {
+ state.refreshing = true
+ state.nextRefresh = now + refreshInterval
+ }
+ state.lock.unlock()
+ if refresh {
+ queue.async {
+ // Accessibility can block on an unresponsive app. It must not
+ // run on the render thread or accumulate a queue of requests.
+ let observed: CGRect?
+ if let reader { observed = reader(source.windowID) }
+ else if let element = source.element { observed = accessibilityFrame(element) }
+ else { observed = windowListFrame(for: source.windowID) }
+ state.lock.lock()
+ if state.source === source, let observed, CursorRestingAnchor.isUsable(observed) {
+ state.frame = observed
+ }
+ state.refreshing = false
+ state.lock.unlock()
+ }
+ }
+ return frame
}
/// Frame read from the registered accessibility element, which reflects a
/// move immediately.
- static func accessibilityFrame(for windowID: CGWindowID) -> CGRect? {
- state.lock.lock()
- let element = state.elements[windowID]
- state.lock.unlock()
-
- guard let element else {
- return nil
- }
-
+ private static func accessibilityFrame(_ element: AXUIElement) -> CGRect? {
var positionValue: CFTypeRef?
var sizeValue: CFTypeRef?
guard AXUIElementCopyAttributeValue(element, kAXPositionAttribute as CFString, &positionValue) == .success,
AXUIElementCopyAttributeValue(element, kAXSizeAttribute as CFString, &sizeValue) == .success,
let positionValue,
- let sizeValue
+ let sizeValue,
+ CFGetTypeID(positionValue) == AXValueGetTypeID(),
+ CFGetTypeID(sizeValue) == AXValueGetTypeID()
else {
return nil
}
@@ -117,8 +182,7 @@ enum CursorWindowFrameTracker {
var size = CGSize.zero
guard AXValueGetValue(positionValue as! AXValue, .cgPoint, &position),
AXValueGetValue(sizeValue as! AXValue, .cgSize, &size),
- size.width > 0,
- size.height > 0
+ CursorRestingAnchor.isUsable(CGRect(origin: position, size: size))
else {
return nil
}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
index 791e39d..436c1a3 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
@@ -195,6 +195,7 @@ enum SoftwareCursorOverlay {
/// Window frame the current target point was derived from; lets a move
/// under the cursor be detected and re-derived instead of ignored.
private static var restingAnchor: CursorRestingAnchor?
+ private static var lastAnchorFrame: CGRect?
private static var visualDynamicsState: CursorVisualDynamicsState?
private static var idleTimer: Timer?
private static var hideTimer: Timer?
@@ -215,7 +216,8 @@ enum SoftwareCursorOverlay {
cancelPendingHide()
configureOrdering(relativeTo: targetWindow)
- restingAnchor = anchor
+ adoptAnchor(anchor)
+ lastAnchorFrame = anchor.flatMap { CursorWindowFrameTracker.liveFrame(for: $0.windowID) } ?? anchor?.windowBounds
let constrainedTarget = clampTipPosition(liveTargetPoint(targetPoint, anchor: anchor))
let isFreshStart = displayedTipPosition == nil
let startPoint = displayedTipPosition ?? defaultInitialTipPosition()
@@ -254,7 +256,9 @@ enum SoftwareCursorOverlay {
}
configureOrdering(relativeTo: targetWindow)
- restingAnchor = anchor ?? restingAnchor
+ stopIdleAnimation()
+ adoptAnchor(anchor)
+ _ = reanchorIfWindowMoved()
let constrainedTarget = clampTipPosition(liveTargetPoint(targetPoint, anchor: anchor))
let now = CACurrentMediaTime()
seedVisualDynamicsIfNeeded(at: constrainedTarget, time: now)
@@ -275,7 +279,8 @@ enum SoftwareCursorOverlay {
}
configureOrdering(relativeTo: targetWindow)
- restingAnchor = anchor ?? restingAnchor
+ adoptAnchor(anchor)
+ _ = reanchorIfWindowMoved()
let constrainedTarget = clampTipPosition(liveTargetPoint(targetPoint, anchor: anchor))
restingTipPosition = constrainedTarget
observationPhase = "settling"
@@ -297,6 +302,8 @@ enum SoftwareCursorOverlay {
restingTipPosition = nil
activeTargetWindow = nil
restingAnchor = nil
+ lastAnchorFrame = nil
+ CursorWindowFrameTracker.forgetAll()
visualDynamicsState = nil
observationPhase = "hidden"
writeObservationSnapshot(tipPosition: nil, rotation: nil)
@@ -309,7 +316,6 @@ enum SoftwareCursorOverlay {
private static func liveTargetPoint(_ targetPoint: CGPoint, anchor: CursorRestingAnchor?) -> CGPoint {
guard let anchor,
let liveFrame = CursorWindowFrameTracker.liveFrame(for: anchor.windowID),
- liveFrame != anchor.windowBounds,
let screenStatePoint = anchor.screenStatePoint(liveFrame: liveFrame)
else {
return targetPoint
@@ -321,26 +327,24 @@ enum SoftwareCursorOverlay {
)
}
- /// Whether the anchored window has moved since the anchor was taken.
- private static func anchoredWindowMoved() -> Bool {
- guard let anchor = restingAnchor,
- let liveFrame = CursorWindowFrameTracker.liveFrame(for: anchor.windowID)
- else {
- return false
+ private static func adoptAnchor(_ anchor: CursorRestingAnchor?) {
+ if restingAnchor != anchor {
+ restingAnchor = anchor
+ lastAnchorFrame = anchor?.windowBounds
}
-
- return liveFrame != anchor.windowBounds
+ if anchor == nil { CursorWindowFrameTracker.forgetAll() }
}
- /// Stop a travel whose window moved under it: land the cursor on the
- /// window's live frame instead of finishing the path towards the display
- /// the window just left.
- private static func abortTravelOntoObservedWindow() {
+ /// Relocate the visible tip, not just its spring target. Otherwise an old
+ /// travel velocity can leave it on the previous display after reanchoring.
+ @discardableResult
+ private static func reanchorIfWindowMoved() -> Bool {
guard let anchor = restingAnchor,
let liveFrame = CursorWindowFrameTracker.liveFrame(for: anchor.windowID),
+ liveFrame != lastAnchorFrame,
let screenStatePoint = anchor.screenStatePoint(liveFrame: liveFrame)
else {
- return
+ return false
}
let constrainedTarget = clampTipPosition(
@@ -349,12 +353,15 @@ enum SoftwareCursorOverlay {
screenMappings: currentVisualCursorScreenMappings()
)
)
+ lastAnchorFrame = liveFrame
restingTipPosition = constrainedTarget
+ visualDynamicsState = CursorVisualDynamicsAnimator.state(at: constrainedTarget, time: CGFloat(CACurrentMediaTime()))
observationPhase = "reanchored"
placeCursor(
- using: advanceVisualDynamics(toward: constrainedTarget, at: CACurrentMediaTime()),
+ using: initialRenderState(at: constrainedTarget),
clickProgress: 0
)
+ return true
}
private static var canPresentOverlay: Bool {
@@ -438,8 +445,7 @@ enum SoftwareCursorOverlay {
while true {
refreshActiveOrderingIfNeeded()
- if anchoredWindowMoved() {
- abortTravelOntoObservedWindow()
+ if reanchorIfWindowMoved() {
return
}
@@ -632,13 +638,14 @@ enum SoftwareCursorOverlay {
let startTime = CACurrentMediaTime()
while true {
+ _ = reanchorIfWindowMoved()
let elapsed = CACurrentMediaTime() - startTime
let rawProgress = min(max(elapsed / duration, 0), 1)
let clickProgress = sin(rawProgress * .pi) * pulseBias
placeCursor(
using: advanceVisualDynamics(
- toward: point,
+ toward: restingTipPosition ?? point,
at: CACurrentMediaTime()
),
clickProgress: clickProgress
@@ -658,7 +665,7 @@ enum SoftwareCursorOverlay {
placeCursor(
using: advanceVisualDynamics(
- toward: point,
+ toward: restingTipPosition ?? point,
at: CACurrentMediaTime()
),
clickProgress: 0
@@ -666,6 +673,7 @@ enum SoftwareCursorOverlay {
}
private static func startIdleAnimation() {
+ stopIdleAnimation()
guard canPresentOverlay, let restingTipPosition else {
return
}
@@ -679,11 +687,13 @@ enum SoftwareCursorOverlay {
}
refreshActiveOrderingIfNeeded()
+ _ = reanchorIfWindowMoved()
+ guard let currentTipPosition = self.restingTipPosition else { return }
observationPhase = "idle"
idlePhase += 0.05
let idlePose = visualCursorIdlePose(
- restingTipPosition: restingTipPosition,
+ restingTipPosition: currentTipPosition,
phase: idlePhase
)
@@ -750,6 +760,9 @@ enum SoftwareCursorOverlay {
displayedTipPosition = nil
restingTipPosition = nil
activeTargetWindow = nil
+ restingAnchor = nil
+ lastAnchorFrame = nil
+ CursorWindowFrameTracker.forgetAll()
visualDynamicsState = nil
observationPhase = "hidden"
writeObservationSnapshot(tipPosition: nil, rotation: nil)
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowAnchorTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowAnchorTests.swift
index 8b0e1a7..a4f9960 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowAnchorTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowAnchorTests.swift
@@ -52,6 +52,8 @@ final class CursorWindowAnchorTests: XCTestCase {
func testAnchorRejectsALiveFrameWithoutSize() {
XCTAssertNil(anchor().screenStatePoint(liveFrame: .zero))
XCTAssertNil(anchor().screenStatePoint(liveFrame: CGRect(x: 10, y: 10, width: 0, height: 40)))
+ XCTAssertNil(anchor().screenStatePoint(liveFrame: .infinite))
+ XCTAssertNil(anchor().screenStatePoint(liveFrame: CGRect(x: CGFloat.nan, y: 0, width: 50, height: 50)))
}
func testTrackerPrefersTheRegisteredLiveFrame() {
@@ -64,11 +66,81 @@ final class CursorWindowAnchorTests: XCTestCase {
XCTAssertNil(CursorWindowFrameTracker.liveFrame(for: 1))
}
- /// Registering nothing must not produce a frame out of thin air; the
- /// window list is the only remaining source and an unknown id has no entry.
+ /// An unknown window must not produce a frame out of thin air.
func testTrackerReportsNothingForAnUnknownWindow() {
CursorWindowFrameTracker.forgetAll()
XCTAssertNil(CursorWindowFrameTracker.liveFrame(for: CGWindowID(0)))
}
+
+ func testFailedRefreshKeepsLastUsableFrame() throws {
+ let reads = DispatchSemaphore(value: 0), failedRead = DispatchSemaphore(value: 0)
+ let frame = mainDisplayFrame
+ CursorWindowFrameTracker.forgetAll()
+ CursorWindowFrameTracker.installFrameReaderForTesting { _ in
+ reads.signal()
+ return frame
+ }
+ defer {
+ CursorWindowFrameTracker.forgetAll()
+ CursorWindowFrameTracker.installFrameReaderForTesting(nil)
+ }
+ CursorWindowFrameTracker.register(windowID: 1, element: nil)
+ _ = CursorWindowFrameTracker.liveFrame(for: 1)
+ XCTAssertEqual(reads.wait(timeout: .now() + 1), .success)
+ let deadline = ProcessInfo.processInfo.systemUptime + 1
+ while CursorWindowFrameTracker.liveFrame(for: 1) == nil && ProcessInfo.processInfo.systemUptime < deadline {
+ Thread.sleep(forTimeInterval: 0.001)
+ }
+ XCTAssertEqual(CursorWindowFrameTracker.liveFrame(for: 1), frame)
+ CursorWindowFrameTracker.installFrameReaderForTesting { _ in
+ failedRead.signal()
+ return nil
+ }
+ Thread.sleep(forTimeInterval: 0.04)
+ XCTAssertEqual(CursorWindowFrameTracker.liveFrame(for: 1), frame)
+ XCTAssertEqual(failedRead.wait(timeout: .now() + 1), .success)
+ Thread.sleep(forTimeInterval: 0.01)
+ XCTAssertEqual(CursorWindowFrameTracker.liveFrame(for: 1), frame)
+ }
+
+ func testSlowReadDoesNotBlockOrQueueWorkAndCannotPublishAfterReplacement() throws {
+ let entered = DispatchSemaphore(value: 0), release = DispatchSemaphore(value: 0)
+ let first = builtInDisplayFrame, second = mainDisplayFrame
+ CursorWindowFrameTracker.forgetAll()
+ CursorWindowFrameTracker.installFrameReaderForTesting { id in
+ if id == 1 {
+ entered.signal()
+ _ = release.wait(timeout: .now() + 2)
+ return first
+ }
+ return second
+ }
+ defer {
+ release.signal()
+ CursorWindowFrameTracker.forgetAll()
+ CursorWindowFrameTracker.installFrameReaderForTesting(nil)
+ }
+ CursorWindowFrameTracker.register(windowID: 1, element: nil)
+ XCTAssertNil(CursorWindowFrameTracker.liveFrame(for: 1))
+ XCTAssertEqual(entered.wait(timeout: .now() + 1), .success)
+ let start = ProcessInfo.processInfo.systemUptime
+ for _ in 0..<1000 { XCTAssertNil(CursorWindowFrameTracker.liveFrame(for: 1)) }
+ XCTAssertLessThan(ProcessInfo.processInfo.systemUptime - start, 0.25)
+ CursorWindowFrameTracker.forgetAll()
+ CursorWindowFrameTracker.register(windowID: 2, element: nil)
+ XCTAssertNil(CursorWindowFrameTracker.liveFrame(for: 1))
+ XCTAssertNil(CursorWindowFrameTracker.liveFrame(for: 2))
+ release.signal()
+ let deadline = ProcessInfo.processInfo.systemUptime + 1
+ var observed: CGRect?
+ while observed == nil && ProcessInfo.processInfo.systemUptime < deadline {
+ observed = CursorWindowFrameTracker.liveFrame(for: 2)
+ Thread.sleep(forTimeInterval: 0.001)
+ }
+ XCTAssertEqual(observed, second)
+ XCTAssertNil(CursorWindowFrameTracker.liveFrame(for: 1))
+ // No stale queued requests from the 1,000 render ticks may remain.
+ XCTAssertEqual(entered.wait(timeout: .now() + 0.05), .timedOut)
+ }
}
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowFollowTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowFollowTests.swift
new file mode 100644
index 0000000..8415c8d
--- /dev/null
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/CursorWindowFollowTests.swift
@@ -0,0 +1,264 @@
+import AppKit
+import XCTest
+@testable import OpenComputerUseKit
+
+final class CursorWindowFollowTests: XCTestCase {
+ @MainActor
+ private final class Counter: NSObject {
+ var count = 0
+ @objc func increment() { count += 1 }
+ }
+ private final class Frame: @unchecked Sendable {
+ private let lock = NSLock()
+ private var stored: CGRect
+ init(_ value: CGRect) { stored = value }
+ var value: CGRect {
+ get { lock.lock(); defer { lock.unlock() }; return stored }
+ set { lock.lock(); defer { lock.unlock() }; stored = newValue }
+ }
+ }
+ @MainActor
+ private func withOverlay(_ body: (URL, CGRect) throws -> Void) throws {
+ guard let screen = currentVisualCursorScreenMappings().first?.screenStateFrame else {
+ throw XCTSkip("A WindowServer display is required for overlay rendering")
+ }
+ _ = NSApplication.shared
+ let file = FileManager.default.temporaryDirectory.appendingPathComponent("ocu-cursor-follow-\(UUID().uuidString).json")
+ let previous = ProcessInfo.processInfo.environment
+ setenv("OPEN_COMPUTER_USE_VISUAL_CURSOR", "1", 1)
+ setenv("OPEN_COMPUTER_USE_VISUAL_CURSOR_OBSERVATION_FILE", file.path, 1)
+ setenv("OPEN_COMPUTER_USE_ACTION_READ_BACK", "0", 1)
+ SoftwareCursorOverlay.moveCursor(to: defaultVisualCursorInitialTipPosition(), in: nil)
+ SoftwareCursorOverlay.reset()
+ defer {
+ SoftwareCursorOverlay.reset()
+ CursorWindowFrameTracker.installFrameOverrideForTesting(nil)
+ CursorWindowFrameTracker.forgetAll()
+ for key in ["OPEN_COMPUTER_USE_VISUAL_CURSOR", "OPEN_COMPUTER_USE_VISUAL_CURSOR_OBSERVATION_FILE", "OPEN_COMPUTER_USE_ACTION_READ_BACK"] {
+ if let value = previous[key] { setenv(key, value, 1) } else { unsetenv(key) }
+ }
+ try? FileManager.default.removeItem(at: file)
+ }
+ try body(file, CGRect(x: screen.minX + 100, y: screen.minY + 100, width: 200, height: 160))
+ }
+
+ private func anchor(_ frame: CGRect) -> CursorRestingAnchor {
+ .init(windowID: 4242, layer: 0, windowLocalPoint: CGPoint(x: 40, y: 50), windowBounds: frame)
+ }
+
+ @MainActor
+ private func point(_ frame: CGRect) -> CGPoint {
+ screenStatePointToAppKitGlobalPoint(fromScreenStatePoint: CGPoint(x: frame.minX + 40, y: frame.minY + 50))
+ }
+
+ @MainActor
+ private func assertPosition(_ file: URL, frame: CGRect, file testFile: StaticString = #filePath, line: UInt = #line) throws {
+ let state = try JSONDecoder().decode(VisualCursorObservationSnapshot.self, from: Data(contentsOf: file))
+ let expected = point(frame)
+ XCTAssertEqual(try XCTUnwrap(state.restingTipPosition).x, expected.x, accuracy: 1, file: testFile, line: line)
+ XCTAssertEqual(try XCTUnwrap(state.restingTipPosition).y, expected.y, accuracy: 1, file: testFile, line: line)
+ XCTAssertEqual(try XCTUnwrap(state.tipPosition).x, expected.x, accuracy: 1, file: testFile, line: line)
+ XCTAssertEqual(try XCTUnwrap(state.tipPosition).y, expected.y, accuracy: 1, file: testFile, line: line)
+ }
+
+ @MainActor
+ func testIdleFollowsRepeatedMovesAndReturnToOriginalFrame() throws {
+ try withOverlay { file, original in
+ let live = Frame(original)
+ CursorWindowFrameTracker.installFrameOverrideForTesting { _ in live.value }
+ SoftwareCursorOverlay.settle(at: point(original), in: nil, anchor: anchor(original))
+ for frame in [original.offsetBy(dx: 240, dy: 80), original.offsetBy(dx: 80, dy: 160), original] {
+ live.value = frame
+ RunLoop.main.run(until: Date().addingTimeInterval(0.08))
+ try assertPosition(file, frame: frame)
+ }
+ }
+ }
+
+ @MainActor
+ func testClickPulseTracksMovesDuringBothClicks() throws {
+ try withOverlay { file, original in
+ let live = Frame(original)
+ CursorWindowFrameTracker.installFrameOverrideForTesting { _ in live.value }
+ SoftwareCursorOverlay.settle(at: point(original), in: nil, anchor: anchor(original))
+ let first = Timer(timeInterval: 0.04, repeats: false) { _ in live.value = original.offsetBy(dx: 180, dy: 40) }
+ let second = Timer(timeInterval: 0.24, repeats: false) { _ in live.value = original.offsetBy(dx: 260, dy: 100) }
+ RunLoop.main.add(first, forMode: .common)
+ RunLoop.main.add(second, forMode: .common)
+ defer { first.invalidate(); second.invalidate() }
+ SoftwareCursorOverlay.pulseClick(at: point(original), clickCount: 2, mouseButton: .left, in: nil, anchor: anchor(original))
+ try assertPosition(file, frame: live.value)
+ }
+ }
+
+ @MainActor
+ func testTravelAbortActuallyPlacesVisibleTipAtMovedWindow() throws {
+ try withOverlay { file, original in
+ let live = Frame(original)
+ CursorWindowFrameTracker.installFrameOverrideForTesting { _ in live.value }
+ let timer = Timer(timeInterval: 0.04, repeats: false) { _ in live.value = original.offsetBy(dx: 240, dy: 100) }
+ RunLoop.main.add(timer, forMode: .common)
+ defer { timer.invalidate() }
+ SoftwareCursorOverlay.moveCursor(to: point(original), in: nil, anchor: anchor(original))
+ try assertPosition(file, frame: live.value)
+ // The next pipeline stage must not restore the snapshot's old point.
+ SoftwareCursorOverlay.settle(at: point(original), in: nil, anchor: anchor(original))
+ try assertPosition(file, frame: live.value)
+ }
+ }
+
+ @MainActor
+ func testUnanchoredTargetDoesNotKeepPreviousWindowTracking() throws {
+ try withOverlay { file, original in
+ let live = Frame(original)
+ CursorWindowFrameTracker.installFrameOverrideForTesting { _ in live.value }
+ SoftwareCursorOverlay.settle(at: point(original), in: nil, anchor: anchor(original))
+ SoftwareCursorOverlay.settle(at: point(original), in: nil)
+ live.value = original.offsetBy(dx: 240, dy: 100)
+ RunLoop.main.run(until: Date().addingTimeInterval(0.08))
+ try assertPosition(file, frame: original)
+ }
+ }
+
+ @MainActor
+ func testResetAfterPulseDoesNotLeaveAnIdleTimerRepainting() throws {
+ try withOverlay { file, original in
+ CursorWindowFrameTracker.installFrameOverrideForTesting { _ in original }
+ SoftwareCursorOverlay.settle(at: point(original), in: nil, anchor: anchor(original))
+ SoftwareCursorOverlay.pulseClick(at: point(original), clickCount: 1, mouseButton: .left, in: nil, anchor: anchor(original))
+ SoftwareCursorOverlay.reset()
+ RunLoop.main.run(until: Date().addingTimeInterval(0.08))
+ let state = try JSONDecoder().decode(VisualCursorObservationSnapshot.self, from: Data(contentsOf: file))
+ XCTAssertEqual(state.phase, "hidden")
+ XCTAssertNil(state.tipPosition)
+ XCTAssertNil(state.restingTipPosition)
+ }
+ }
+
+ @MainActor
+ func testWindowAlreadyMovedBeforeTravelDoesNotAbortAgainstOriginalSnapshot() throws {
+ try withOverlay { _, original in
+ let moved = original.offsetBy(dx: 240, dy: 100)
+ CursorWindowFrameTracker.installFrameOverrideForTesting { _ in moved }
+ let start = ProcessInfo.processInfo.systemUptime
+ SoftwareCursorOverlay.moveCursor(to: point(original), in: nil, anchor: anchor(original))
+ XCTAssertGreaterThan(ProcessInfo.processInfo.systemUptime - start, 0.5)
+ }
+ }
+
+ @MainActor
+ func testServiceClickRegistersLiveWindowAndFeedbackMatchesMovedButton() throws {
+ do { try requireUsableComputerUseSession() }
+ catch { throw XCTSkip("Service integration requires an unlocked session and a trusted test host: \(error)") }
+ try withOverlay { file, original in
+ let counter = Counter()
+ let window = NSWindow(contentRect: original, styleMask: [.titled], backing: .buffered, defer: false)
+ window.title = "OCU service click fixture"
+ window.isReleasedWhenClosed = false
+ let button = NSButton(title: "OCU follow click", target: counter, action: #selector(Counter.increment))
+ button.frame = CGRect(x: 20, y: 20, width: 140, height: 32)
+ window.contentView?.addSubview(button)
+ NSApplication.shared.finishLaunching()
+ window.orderFront(nil)
+ defer { window.close() }
+ RunLoop.main.run(until: Date().addingTimeInterval(0.05))
+ guard AXIsProcessTrusted(), let app = AppDiscovery.runningApps().first(where: { $0.pid == getpid() }) else {
+ throw XCTSkip("Live AX test application is unavailable")
+ }
+ let service = ComputerUseService()
+ let result = try service.query(app: app.bundleIdentifier ?? app.name, text: button.title, role: "button",
+ exact: true, windowID: CGWindowID(window.windowNumber))
+ let matches = try XCTUnwrap(result["matches"] as? [[String: Any]])
+ XCTAssertEqual(matches.count, 1)
+ let index = try XCTUnwrap(matches.first?["index"] as? Int)
+ let timer = Timer(timeInterval: 0.08, repeats: false) { _ in
+ MainActor.assumeIsolated {
+ window.setFrameOrigin(CGPoint(x: original.minX + 240, y: original.minY + 80))
+ }
+ }
+ RunLoop.main.add(timer, forMode: .common)
+ defer { timer.invalidate() }
+ _ = try service.click(app: app.bundleIdentifier ?? app.name, elementIndex: String(index),
+ x: nil, y: nil, clickCount: 1, mouseButton: "left", clickMethod: .accessibility)
+ XCTAssertEqual(counter.count, 1, "The real AX button action must still execute exactly once")
+ let state = try JSONDecoder().decode(VisualCursorObservationSnapshot.self, from: Data(contentsOf: file))
+ let expected = window.convertPoint(toScreen: CGPoint(x: button.frame.midX, y: button.frame.midY))
+ XCTAssertEqual(try XCTUnwrap(state.tipPosition).x, expected.x, accuracy: 1)
+ XCTAssertEqual(try XCTUnwrap(state.tipPosition).y, expected.y, accuracy: 1)
+ XCTAssertEqual(try XCTUnwrap(state.restingTipPosition).x, expected.x, accuracy: 1)
+ XCTAssertEqual(try XCTUnwrap(state.restingTipPosition).y, expected.y, accuracy: 1)
+ }
+ }
+
+ @MainActor
+ func testRealAccessibilityWindowFollowsTravelPulseAndIdleAcrossDisplays() throws {
+ try withOverlay { file, original in
+ let window = NSWindow(contentRect: original, styleMask: [.titled], backing: .buffered, defer: false)
+ window.title = "OCU cursor follow fixture \(UUID().uuidString)"
+ window.isReleasedWhenClosed = false
+ NSApplication.shared.finishLaunching()
+ window.orderFront(nil)
+ defer { window.close() }
+ RunLoop.main.run(until: Date().addingTimeInterval(0.05))
+ let application = AXUIElementCreateApplication(getpid())
+ var raw: CFTypeRef?
+ let status = AXUIElementCopyAttributeValue(application, kAXWindowsAttribute as CFString, &raw)
+ guard status == .success,
+ let windows = raw as? [AXUIElement],
+ let element = windows.first(where: { element in
+ var title: CFTypeRef?
+ return AXUIElementCopyAttributeValue(element, kAXTitleAttribute as CFString, &title) == .success &&
+ title as? String == window.title
+ })
+ else { throw XCTSkip("Live self-AX window access unavailable (AX status \(status.rawValue), trusted \(AXIsProcessTrusted()))") }
+ let id = CGWindowID(window.windowNumber)
+ CursorWindowFrameTracker.register(windowID: id, element: element)
+ func awaitFrame(_ predicate: (CGRect) -> Bool) throws -> CGRect {
+ let deadline = Date().addingTimeInterval(1)
+ while Date() < deadline {
+ if let frame = CursorWindowFrameTracker.liveFrame(for: id), predicate(frame) { return frame }
+ RunLoop.main.run(until: Date().addingTimeInterval(0.01))
+ }
+ XCTFail("Real AX frame was not observed before deadline")
+ throw NSError(domain: "CursorFollowTests", code: 1)
+ }
+ let initial = try awaitFrame { _ in true }
+ let targetAnchor = CursorRestingAnchor(windowID: id, layer: 0, windowLocalPoint: CGPoint(x: 40, y: 50), windowBounds: initial)
+ let move = Timer(timeInterval: 0.08, repeats: false) { _ in
+ MainActor.assumeIsolated {
+ window.setFrameOrigin(CGPoint(x: original.minX + 200, y: original.minY + 40))
+ }
+ }
+ RunLoop.main.add(move, forMode: .common)
+ defer { move.invalidate() }
+ SoftwareCursorOverlay.moveCursor(to: point(initial), in: nil, anchor: targetAnchor)
+ let afterTravel = try awaitFrame { abs($0.minX - (initial.minX + 200)) < 1 }
+ try assertPosition(file, frame: afterTravel)
+ SoftwareCursorOverlay.settle(at: point(initial), in: nil, anchor: targetAnchor)
+ try assertPosition(file, frame: afterTravel)
+ let duringPulse = Timer(timeInterval: 0.06, repeats: false) { _ in
+ MainActor.assumeIsolated {
+ window.setFrameOrigin(CGPoint(x: original.minX + 260, y: original.minY + 100))
+ }
+ }
+ RunLoop.main.add(duringPulse, forMode: .common)
+ defer { duringPulse.invalidate() }
+ SoftwareCursorOverlay.pulseClick(at: point(initial), clickCount: 2, mouseButton: .left, in: nil, anchor: targetAnchor)
+ let afterPulse = try awaitFrame { abs($0.minX - (initial.minX + 260)) < 1 }
+ try assertPosition(file, frame: afterPulse)
+ for offset in [CGPoint(x: 220, y: 60), CGPoint(x: 100, y: 140), .zero] {
+ window.setFrameOrigin(CGPoint(x: original.minX + offset.x, y: original.minY + offset.y))
+ let moved = try awaitFrame { abs($0.minX - (initial.minX + offset.x)) < 1 }
+ RunLoop.main.run(until: Date().addingTimeInterval(0.04))
+ try assertPosition(file, frame: moved)
+ }
+ for mapping in currentVisualCursorScreenMappings() {
+ window.setFrameOrigin(CGPoint(x: mapping.appKitFrame.minX + 120, y: mapping.appKitFrame.minY + 120))
+ let moved = try awaitFrame { abs($0.minX - (mapping.screenStateFrame.minX + 120)) < 1 }
+ RunLoop.main.run(until: Date().addingTimeInterval(0.04))
+ try assertPosition(file, frame: moved)
+ }
+ print("Live AX cursor following verified on \(currentVisualCursorScreenMappings().count) display(s).")
+ }
+ }
+}
diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.md
index 5c8426c..9a710b5 100644
--- a/skills/open-computer-use/references/usage.md
+++ b/skills/open-computer-use/references/usage.md
@@ -230,15 +230,18 @@ When the gate is not enabled, treat window-server drags as unavailable and reach
## Software Cursor
-The software cursor follows the target window when it moves. Its point is derived from the snapshot, so it
-also carries an anchor: the same point expressed relative to the window frame the snapshot reported. Before
-drawing, and on every frame of a travel, the overlay re-derives the point from the window's live frame, so a
-window dragged to another display mid-action takes the cursor with it instead of leaving it on the display
-the window just left.
-
-The live frame comes from the window's accessibility element — the element the snapshot already holds — and
-only falls back to `CGWindowListCopyWindowInfo`, which can keep reporting the pre-move origin for about a
-second (measured: 83 identical frames at the travel's 120 Hz pump).
+The software cursor keeps a window-relative anchor for its target point. During travel, settle, click
+feedback and idle, it follows changes to the target window's frame. If the window moves during travel,
+the visible tip lands at the new position immediately, without continuing toward the old position.
+
+Window frames are read on a background worker at up to 30 Hz, with at most one read in flight. Rendering
+uses the latest cached frame, so a slow Accessibility response does not block the cursor. The snapshot's
+AX window element is preferred; `CGWindowListCopyWindowInfo` is used only when no AX element is available,
+because it can report an old origin after a move. A failed read keeps the last usable frame. Tracking is
+released when the cursor is cleared or its idle timeout expires.
+
+This follows the window's position, not layout changes inside it. Refresh app state after a layout change
+before choosing another target. Input delivery continues to use the existing action paths.
## Platform Notes