Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Refine visual cursor path selection

Leo46d61586e3a5
diff --git a/docs/histories/2026-04/20260417-2130-window-aware-cursor-path-selection.md b/docs/histories/2026-04/20260417-2130-window-aware-cursor-path-selection.md
new file mode 100644
index 0000000..60d30b7
--- /dev/null
+++ b/docs/histories/2026-04/20260417-2130-window-aware-cursor-path-selection.md
@@ -0,0 +1,26 @@
+## [2026-04-17 21:30] | Task: 收敛 visual cursor 的窗口感知轨迹
+
+### 🤖 Execution Context
+* **Agent ID**: `Codex`
+* **Base Model**: `gpt-5.4`
+* **Runtime**: `Codex CLI`
+
+### 📥 User Query
+> 继续深挖官方 `Codex Computer Use.app` 里的 cursor / overlay 实现,因为当前开源版效果还不够好;希望把能挖出来的行为迁回我们自己的实现里。
+
+### 🛠 Changes Overview
+**Scope:** `packages/OpenComputerUseKit`, `docs/references`, `docs/histories`
+
+**Key Actions:**
+- **[Window-aware path selection]**: 给 `SoftwareCursorOverlay` 增加多候选 Bezier 路径选择;当 snapshot 带有目标 `windowID` 时,会对候选路径的控制点和关键采样点做 window hit-test,优先选择仍然落在目标 window 上的路径。
+- **[Conservative fallback]**: 增加严格直线的保守 fallback,避免所有曲线路径都偏离目标窗口时继续硬播夸张轨迹。
+- **[Ordering resilience]**: overlay 现在会在排序前校验目标 window 是否仍存在,并在移动动画与 idle sway 期间持续检查目标 window 是否失效,失效后回退到普通前置排序。
+- **[Docs sync]**: 更新逆向分析文档,补充官方实现里“绑定具体 target window id”和“轨迹窗口命中检查”的推断,并把这轮实现沉淀到 history。
+
+### 🧠 Design Intent (Why)
+这轮不是继续把 overlay 做得更花,而是补上官方实现里更关键的一层约束:cursor 轨迹和目标窗口之间的关系。仅仅“排到目标 window 上面”还不够,如果控制点和中间采样点明显飘出目标窗口,观感就会和官方差很多。把路径选择改成 window-aware 之后,能在不动输入注入链路的前提下,先把最明显的视觉偏差收回来。
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`
+- `docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md`
+- `docs/histories/2026-04/20260417-2130-window-aware-cursor-path-selection.md`
diff --git a/docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md b/docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md
index 5edc7c9..bf8aae9 100644
--- a/docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md
+++ b/docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md
@@ -159,6 +159,39 @@ Bounds: X=1949 Y=796 Width=126 Height=126
这比直接操纵系统可见光标更稳,也更容易做出平滑路径、点击高亮和 delay 控制。
+## 更细一层的实现推断
+
+继续看 `ComputerUseCursor.Window` 的静态分析后,可以再补两条对开源实现很关键的推断。
+
+### 1. 官方不仅调 window level,还会绑定具体 target window id
+
+从 `ComputerUseCursor.Window` 的 ivar 和辅助函数可以看到:
+
+- 它同时保存了 `useOverlayWindowLevel` 和 `correspondingWindowID`。
+- helper `0x10005d650` 会在 `useOverlayWindowLevel` 开启时检查 `correspondingWindowID` 是否还存在于当前窗口列表里。
+- helper `0x10005d7bc` 在目标 window 仍有效时,会继续把 cursor window 排到对应 window 之上;目标失效时会清掉当前动画状态并回退到普通前置排序。
+
+这说明官方做的不是“把 overlay 永远设到某个固定 level”,而是“尽量相对某个具体 window 维持排序;一旦目标 window 消失或失效,再切回兜底行为”。
+
+### 2. 官方在生成/接受轨迹前,会做窗口命中检查
+
+`ComputerUseCursor.Window` 的大方法里出现了一组很有辨识度的名字:
+
+- `distanceThreshold`
+- `closeEnough`
+- `control1`
+- `control2`
+- `staysInBounds`
+
+对应 helper `0x10005c388` 的行为也很说明问题:
+
+- 它会读取 `correspondingWindowID`。
+- 用当前候选点坐标去做 window hit-test。
+- 把命中的 window number 和 `correspondingWindowID` 比较。
+- 只有在关键采样点仍然落在目标 window 上时,才接受这组路径/控制点。
+
+换句话说,官方那条 Bezier 不是“先固定算出来再硬播”,而是带有一层 target-window-aware 的约束判断,避免鼠标视觉轨迹明显飘出被控窗口。
+
## 还没完全确认的点
- 还没有抓到一次动画进行中的连续窗口轨迹,目前只有离散位置点。
@@ -170,5 +203,7 @@ Bounds: X=1949 Y=796 Width=126 Height=126
- 可以把“真实输入注入”和“软件光标可视化”明确拆开。
- 软件光标可以做成独立 overlay window,而不是强依赖真实鼠标位置。
+- 如果手上有 `windowID`,Bezier 候选最好先做一轮窗口命中采样,再决定用哪组 `control1` / `control2`。
+- overlay 最好维护“相对目标 window 的排序”和“目标 window 是否还活着”的状态,而不是只在动画开始时排一次层级。
- 事件执行即使失败,overlay 也可以保持可观测,有利于调试和用户信任。
- 菜单栏状态项和软件光标都放在同一个 `LSUIElement` agent app 里,是一条已经被官方实现验证过的产品形态。
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
index 38ef282..556fdf8 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
@@ -61,24 +61,38 @@ struct CursorMotionPath {
let control1: CGPoint
let control2: CGPoint
let end: CGPoint
+ let curveScale: CGFloat
- init(start: CGPoint, end: CGPoint) {
+ init(start: CGPoint, end: CGPoint, curveDirection: CGFloat? = nil, curveScale: CGFloat = 1) {
let delta = CGVector(dx: end.x - start.x, dy: end.y - start.y)
let distance = max(hypot(delta.dx, delta.dy), 1)
let normal = normalized(CGVector(dx: -delta.dy, dy: delta.dx))
- let curveDirection: CGFloat = delta.dx >= 0 ? 1 : -1
- let curveAmount = min(max(distance * 0.22, 28), 110)
- let controlOffset = CGPoint(x: normal.dx * curveAmount * curveDirection, y: normal.dy * curveAmount * curveDirection)
+ let resolvedCurveDirection = curveDirection ?? (delta.dx >= 0 ? 1 : -1)
+ let resolvedCurveScale = max(curveScale, 0)
+ let curveAmount = min(max(distance * 0.22, 28), 110) * resolvedCurveScale
+ let controlOffset = CGPoint(
+ x: normal.dx * curveAmount * resolvedCurveDirection,
+ y: normal.dy * curveAmount * resolvedCurveDirection
+ )
+ let control1Base = CGPoint(
+ x: start.x + (delta.dx * (resolvedCurveScale == 0 ? 1.0 / 3.0 : 0.18)),
+ y: start.y + (delta.dy * (resolvedCurveScale == 0 ? 1.0 / 3.0 : 0.10))
+ )
+ let control2Base = CGPoint(
+ x: start.x + (delta.dx * (resolvedCurveScale == 0 ? 2.0 / 3.0 : 0.80)),
+ y: start.y + (delta.dy * (resolvedCurveScale == 0 ? 2.0 / 3.0 : 0.96))
+ )
self.start = start
self.end = end
+ self.curveScale = resolvedCurveScale
self.control1 = CGPoint(
- x: start.x + (delta.dx * 0.18) + controlOffset.x,
- y: start.y + (delta.dy * 0.10) + controlOffset.y
+ x: control1Base.x + controlOffset.x,
+ y: control1Base.y + controlOffset.y
)
self.control2 = CGPoint(
- x: start.x + (delta.dx * 0.80) + (controlOffset.x * 0.48),
- y: start.y + (delta.dy * 0.96) + (controlOffset.y * 0.48)
+ x: control2Base.x + (controlOffset.x * 0.48),
+ y: control2Base.y + (controlOffset.y * 0.48)
)
}
@@ -110,6 +124,17 @@ struct CursorMotionPath {
+ (3 * t * t * (end.y - control2.y))
)
}
+
+ func sampledConstraintPoints() -> [CGPoint] {
+ [
+ control1,
+ point(at: 0.28),
+ point(at: 0.5),
+ point(at: 0.72),
+ control2,
+ end,
+ ]
+ }
}
private struct ProcessedCursorImage {
@@ -347,7 +372,7 @@ enum SoftwareCursorOverlay {
placeCursor(at: startPoint, rotation: 0, clickProgress: 0)
if distanceBetween(startPoint, constrainedTarget) > 2 {
- animateMove(from: startPoint, to: constrainedTarget)
+ animateMove(from: startPoint, to: constrainedTarget, relativeTo: targetWindow)
}
}
@@ -423,27 +448,33 @@ enum SoftwareCursorOverlay {
return
}
- let desiredLevel = NSWindow.Level(rawValue: targetWindow?.layer ?? 0)
+ let effectiveTargetWindow = targetWindow.flatMap { targetWindow in
+ isWindowPresent(targetWindow.windowID) ? targetWindow : nil
+ }
+
+ let desiredLevel = NSWindow.Level(rawValue: effectiveTargetWindow?.layer ?? 0)
if panel.level != desiredLevel {
panel.level = desiredLevel
}
- if activeTargetWindow != targetWindow || panel.isVisible == false {
- if let targetWindow {
- panel.order(.above, relativeTo: Int(targetWindow.windowID))
+ if activeTargetWindow != effectiveTargetWindow || panel.isVisible == false {
+ if let effectiveTargetWindow {
+ panel.order(.above, relativeTo: Int(effectiveTargetWindow.windowID))
} else {
panel.orderFront(nil)
}
- activeTargetWindow = targetWindow
+ activeTargetWindow = effectiveTargetWindow
}
}
- private static func animateMove(from start: CGPoint, to end: CGPoint) {
- let path = CursorMotionPath(start: start, end: end)
+ private static func animateMove(from start: CGPoint, to end: CGPoint, relativeTo targetWindow: CursorTargetWindow?) {
+ let path = bestMotionPath(from: start, to: end, relativeTo: targetWindow)
let duration = min(max(TimeInterval(distanceBetween(start, end) / 1050), 0.24), 0.56)
let startTime = CACurrentMediaTime()
while true {
+ refreshActiveOrderingIfNeeded()
+
let elapsed = CACurrentMediaTime() - startTime
let rawProgress = min(max(elapsed / duration, 0), 1)
let eased = easeInOut(rawProgress)
@@ -461,6 +492,97 @@ enum SoftwareCursorOverlay {
}
}
+ private static func bestMotionPath(from start: CGPoint, to end: CGPoint, relativeTo targetWindow: CursorTargetWindow?) -> CursorMotionPath {
+ let preferredCurveDirection: CGFloat = end.x >= start.x ? 1 : -1
+ let candidates = [
+ CursorMotionPath(start: start, end: end, curveDirection: preferredCurveDirection, curveScale: 1),
+ CursorMotionPath(start: start, end: end, curveDirection: -preferredCurveDirection, curveScale: 1),
+ CursorMotionPath(start: start, end: end, curveDirection: preferredCurveDirection, curveScale: 0.72),
+ CursorMotionPath(start: start, end: end, curveDirection: -preferredCurveDirection, curveScale: 0.72),
+ CursorMotionPath(start: start, end: end, curveDirection: preferredCurveDirection, curveScale: 0.38),
+ CursorMotionPath(start: start, end: end, curveDirection: -preferredCurveDirection, curveScale: 0.38),
+ CursorMotionPath(start: start, end: end, curveDirection: 0, curveScale: 0),
+ ]
+
+ guard let targetWindow else {
+ return candidates[0]
+ }
+
+ let excludingWindowNumber = max(panel?.windowNumber ?? 0, 0)
+ let evaluations = candidates.map { candidate in
+ (
+ path: candidate,
+ hitCount: windowConstraintHitCount(
+ for: candidate,
+ relativeTo: targetWindow,
+ excludingWindowNumber: excludingWindowNumber
+ )
+ )
+ }
+
+ let totalSampleCount = candidates[0].sampledConstraintPoints().count
+ let bestHitCount = evaluations.map(\.hitCount).max() ?? 0
+
+ if bestHitCount == totalSampleCount {
+ return evaluations
+ .filter { $0.hitCount == bestHitCount }
+ .max { $0.path.curveScale < $1.path.curveScale }?
+ .path ?? candidates[0]
+ }
+
+ if bestHitCount > 0 {
+ return evaluations
+ .filter { $0.hitCount == bestHitCount }
+ .min { $0.path.curveScale < $1.path.curveScale }?
+ .path ?? candidates.last ?? candidates[0]
+ }
+
+ return candidates.last ?? candidates[0]
+ }
+
+ private static func windowConstraintHitCount(
+ for path: CursorMotionPath,
+ relativeTo targetWindow: CursorTargetWindow,
+ excludingWindowNumber: Int
+ ) -> Int {
+ path.sampledConstraintPoints().reduce(into: 0) { result, point in
+ if windowID(at: point, excludingWindowNumber: excludingWindowNumber) == targetWindow.windowID {
+ result += 1
+ }
+ }
+ }
+
+ private static func windowID(at point: CGPoint, excludingWindowNumber: Int) -> CGWindowID? {
+ let windowNumber = NSWindow.windowNumber(
+ at: NSPoint(x: point.x, y: point.y),
+ belowWindowWithWindowNumber: excludingWindowNumber
+ )
+
+ guard windowNumber > 0 else {
+ return nil
+ }
+
+ return CGWindowID(windowNumber)
+ }
+
+ private static func isWindowPresent(_ windowID: CGWindowID) -> Bool {
+ guard windowID != 0,
+ let windowInfo = CGWindowListCopyWindowInfo([.optionIncludingWindow], windowID) as? [[String: Any]]
+ else {
+ return false
+ }
+
+ return !windowInfo.isEmpty
+ }
+
+ private static func refreshActiveOrderingIfNeeded() {
+ guard let activeTargetWindow, !isWindowPresent(activeTargetWindow.windowID) else {
+ return
+ }
+
+ configureOrdering(relativeTo: nil)
+ }
+
private static func animateClickPulse(at point: CGPoint, clickCount: Int, mouseButton: MouseButtonKind) {
let pulseBias: CGFloat = mouseButton == .right ? 0.82 : 1
@@ -502,6 +624,8 @@ enum SoftwareCursorOverlay {
return
}
+ refreshActiveOrderingIfNeeded()
+
idlePhase += 0.05
let offset = CGPoint(
x: sin(idlePhase) * 1.6,