👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Refine visual cursor path selection
Leo46d61586e3a5
diff --git a/docs/histories/2026-04/20260417-2130-window-aware-cursor-path-selection.md b/docs/histories/2026-04/20260417-2130-window-aware-cursor-path-selection.mdnew file mode 100644index 0000000..60d30b7--- /dev/null+++ b/docs/histories/2026-04/20260417-2130-window-aware-cursor-path-selection.md@@ -0,0 +1,26 @@+## [2026-04-17 21:30] | Task: 收敛 visual cursor 的窗口感知轨迹++### 🤖 Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `gpt-5.4`+* **Runtime**: `Codex CLI`++### 📥 User Query+> 继续深挖官方 `Codex Computer Use.app` 里的 cursor / overlay 实现,因为当前开源版效果还不够好;希望把能挖出来的行为迁回我们自己的实现里。++### 🛠 Changes Overview+**Scope:** `packages/OpenComputerUseKit`, `docs/references`, `docs/histories`++**Key Actions:**+- **[Window-aware path selection]**: 给 `SoftwareCursorOverlay` 增加多候选 Bezier 路径选择;当 snapshot 带有目标 `windowID` 时,会对候选路径的控制点和关键采样点做 window hit-test,优先选择仍然落在目标 window 上的路径。+- **[Conservative fallback]**: 增加严格直线的保守 fallback,避免所有曲线路径都偏离目标窗口时继续硬播夸张轨迹。+- **[Ordering resilience]**: overlay 现在会在排序前校验目标 window 是否仍存在,并在移动动画与 idle sway 期间持续检查目标 window 是否失效,失效后回退到普通前置排序。+- **[Docs sync]**: 更新逆向分析文档,补充官方实现里“绑定具体 target window id”和“轨迹窗口命中检查”的推断,并把这轮实现沉淀到 history。++### 🧠 Design Intent (Why)+这轮不是继续把 overlay 做得更花,而是补上官方实现里更关键的一层约束:cursor 轨迹和目标窗口之间的关系。仅仅“排到目标 window 上面”还不够,如果控制点和中间采样点明显飘出目标窗口,观感就会和官方差很多。把路径选择改成 window-aware 之后,能在不动输入注入链路的前提下,先把最明显的视觉偏差收回来。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`+- `docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md`+- `docs/histories/2026-04/20260417-2130-window-aware-cursor-path-selection.md`diff --git a/docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md b/docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.mdindex 5edc7c9..bf8aae9 100644--- a/docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md+++ b/docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md@@ -159,6 +159,39 @@ Bounds: X=1949 Y=796 Width=126 Height=126这比直接操纵系统可见光标更稳,也更容易做出平滑路径、点击高亮和 delay 控制。+## 更细一层的实现推断++继续看 `ComputerUseCursor.Window` 的静态分析后,可以再补两条对开源实现很关键的推断。++### 1. 官方不仅调 window level,还会绑定具体 target window id++从 `ComputerUseCursor.Window` 的 ivar 和辅助函数可以看到:++- 它同时保存了 `useOverlayWindowLevel` 和 `correspondingWindowID`。+- helper `0x10005d650` 会在 `useOverlayWindowLevel` 开启时检查 `correspondingWindowID` 是否还存在于当前窗口列表里。+- helper `0x10005d7bc` 在目标 window 仍有效时,会继续把 cursor window 排到对应 window 之上;目标失效时会清掉当前动画状态并回退到普通前置排序。++这说明官方做的不是“把 overlay 永远设到某个固定 level”,而是“尽量相对某个具体 window 维持排序;一旦目标 window 消失或失效,再切回兜底行为”。++### 2. 官方在生成/接受轨迹前,会做窗口命中检查++`ComputerUseCursor.Window` 的大方法里出现了一组很有辨识度的名字:++- `distanceThreshold`+- `closeEnough`+- `control1`+- `control2`+- `staysInBounds`++对应 helper `0x10005c388` 的行为也很说明问题:++- 它会读取 `correspondingWindowID`。+- 用当前候选点坐标去做 window hit-test。+- 把命中的 window number 和 `correspondingWindowID` 比较。+- 只有在关键采样点仍然落在目标 window 上时,才接受这组路径/控制点。++换句话说,官方那条 Bezier 不是“先固定算出来再硬播”,而是带有一层 target-window-aware 的约束判断,避免鼠标视觉轨迹明显飘出被控窗口。+## 还没完全确认的点- 还没有抓到一次动画进行中的连续窗口轨迹,目前只有离散位置点。@@ -170,5 +203,7 @@ Bounds: X=1949 Y=796 Width=126 Height=126- 可以把“真实输入注入”和“软件光标可视化”明确拆开。- 软件光标可以做成独立 overlay window,而不是强依赖真实鼠标位置。+- 如果手上有 `windowID`,Bezier 候选最好先做一轮窗口命中采样,再决定用哪组 `control1` / `control2`。+- overlay 最好维护“相对目标 window 的排序”和“目标 window 是否还活着”的状态,而不是只在动画开始时排一次层级。- 事件执行即使失败,overlay 也可以保持可观测,有利于调试和用户信任。- 菜单栏状态项和软件光标都放在同一个 `LSUIElement` agent app 里,是一条已经被官方实现验证过的产品形态。diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swiftindex 38ef282..556fdf8 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift@@ -61,24 +61,38 @@ struct CursorMotionPath {let control1: CGPointlet control2: CGPointlet end: CGPoint+ let curveScale: CGFloat- init(start: CGPoint, end: CGPoint) {+ init(start: CGPoint, end: CGPoint, curveDirection: CGFloat? = nil, curveScale: CGFloat = 1) {let delta = CGVector(dx: end.x - start.x, dy: end.y - start.y)let distance = max(hypot(delta.dx, delta.dy), 1)let normal = normalized(CGVector(dx: -delta.dy, dy: delta.dx))- let curveDirection: CGFloat = delta.dx >= 0 ? 1 : -1- let curveAmount = min(max(distance * 0.22, 28), 110)- let controlOffset = CGPoint(x: normal.dx * curveAmount * curveDirection, y: normal.dy * curveAmount * curveDirection)+ let resolvedCurveDirection = curveDirection ?? (delta.dx >= 0 ? 1 : -1)+ let resolvedCurveScale = max(curveScale, 0)+ let curveAmount = min(max(distance * 0.22, 28), 110) * resolvedCurveScale+ let controlOffset = CGPoint(+ x: normal.dx * curveAmount * resolvedCurveDirection,+ y: normal.dy * curveAmount * resolvedCurveDirection+ )+ let control1Base = CGPoint(+ x: start.x + (delta.dx * (resolvedCurveScale == 0 ? 1.0 / 3.0 : 0.18)),+ y: start.y + (delta.dy * (resolvedCurveScale == 0 ? 1.0 / 3.0 : 0.10))+ )+ let control2Base = CGPoint(+ x: start.x + (delta.dx * (resolvedCurveScale == 0 ? 2.0 / 3.0 : 0.80)),+ y: start.y + (delta.dy * (resolvedCurveScale == 0 ? 2.0 / 3.0 : 0.96))+ )self.start = startself.end = end+ self.curveScale = resolvedCurveScaleself.control1 = CGPoint(- x: start.x + (delta.dx * 0.18) + controlOffset.x,- y: start.y + (delta.dy * 0.10) + controlOffset.y+ x: control1Base.x + controlOffset.x,+ y: control1Base.y + controlOffset.y)self.control2 = CGPoint(- x: start.x + (delta.dx * 0.80) + (controlOffset.x * 0.48),- y: start.y + (delta.dy * 0.96) + (controlOffset.y * 0.48)+ x: control2Base.x + (controlOffset.x * 0.48),+ y: control2Base.y + (controlOffset.y * 0.48))}@@ -110,6 +124,17 @@ struct CursorMotionPath {+ (3 * t * t * (end.y - control2.y)))}++ func sampledConstraintPoints() -> [CGPoint] {+ [+ control1,+ point(at: 0.28),+ point(at: 0.5),+ point(at: 0.72),+ control2,+ end,+ ]+ }}private struct ProcessedCursorImage {@@ -347,7 +372,7 @@ enum SoftwareCursorOverlay {placeCursor(at: startPoint, rotation: 0, clickProgress: 0)if distanceBetween(startPoint, constrainedTarget) > 2 {- animateMove(from: startPoint, to: constrainedTarget)+ animateMove(from: startPoint, to: constrainedTarget, relativeTo: targetWindow)}}@@ -423,27 +448,33 @@ enum SoftwareCursorOverlay {return}- let desiredLevel = NSWindow.Level(rawValue: targetWindow?.layer ?? 0)+ let effectiveTargetWindow = targetWindow.flatMap { targetWindow in+ isWindowPresent(targetWindow.windowID) ? targetWindow : nil+ }++ let desiredLevel = NSWindow.Level(rawValue: effectiveTargetWindow?.layer ?? 0)if panel.level != desiredLevel {panel.level = desiredLevel}- if activeTargetWindow != targetWindow || panel.isVisible == false {- if let targetWindow {- panel.order(.above, relativeTo: Int(targetWindow.windowID))+ if activeTargetWindow != effectiveTargetWindow || panel.isVisible == false {+ if let effectiveTargetWindow {+ panel.order(.above, relativeTo: Int(effectiveTargetWindow.windowID))} else {panel.orderFront(nil)}- activeTargetWindow = targetWindow+ activeTargetWindow = effectiveTargetWindow}}- private static func animateMove(from start: CGPoint, to end: CGPoint) {- let path = CursorMotionPath(start: start, end: end)+ private static func animateMove(from start: CGPoint, to end: CGPoint, relativeTo targetWindow: CursorTargetWindow?) {+ let path = bestMotionPath(from: start, to: end, relativeTo: targetWindow)let duration = min(max(TimeInterval(distanceBetween(start, end) / 1050), 0.24), 0.56)let startTime = CACurrentMediaTime()while true {+ refreshActiveOrderingIfNeeded()+let elapsed = CACurrentMediaTime() - startTimelet rawProgress = min(max(elapsed / duration, 0), 1)let eased = easeInOut(rawProgress)@@ -461,6 +492,97 @@ enum SoftwareCursorOverlay {}}+ private static func bestMotionPath(from start: CGPoint, to end: CGPoint, relativeTo targetWindow: CursorTargetWindow?) -> CursorMotionPath {+ let preferredCurveDirection: CGFloat = end.x >= start.x ? 1 : -1+ let candidates = [+ CursorMotionPath(start: start, end: end, curveDirection: preferredCurveDirection, curveScale: 1),+ CursorMotionPath(start: start, end: end, curveDirection: -preferredCurveDirection, curveScale: 1),+ CursorMotionPath(start: start, end: end, curveDirection: preferredCurveDirection, curveScale: 0.72),+ CursorMotionPath(start: start, end: end, curveDirection: -preferredCurveDirection, curveScale: 0.72),+ CursorMotionPath(start: start, end: end, curveDirection: preferredCurveDirection, curveScale: 0.38),+ CursorMotionPath(start: start, end: end, curveDirection: -preferredCurveDirection, curveScale: 0.38),+ CursorMotionPath(start: start, end: end, curveDirection: 0, curveScale: 0),+ ]++ guard let targetWindow else {+ return candidates[0]+ }++ let excludingWindowNumber = max(panel?.windowNumber ?? 0, 0)+ let evaluations = candidates.map { candidate in+ (+ path: candidate,+ hitCount: windowConstraintHitCount(+ for: candidate,+ relativeTo: targetWindow,+ excludingWindowNumber: excludingWindowNumber+ )+ )+ }++ let totalSampleCount = candidates[0].sampledConstraintPoints().count+ let bestHitCount = evaluations.map(\.hitCount).max() ?? 0++ if bestHitCount == totalSampleCount {+ return evaluations+ .filter { $0.hitCount == bestHitCount }+ .max { $0.path.curveScale < $1.path.curveScale }?+ .path ?? candidates[0]+ }++ if bestHitCount > 0 {+ return evaluations+ .filter { $0.hitCount == bestHitCount }+ .min { $0.path.curveScale < $1.path.curveScale }?+ .path ?? candidates.last ?? candidates[0]+ }++ return candidates.last ?? candidates[0]+ }++ private static func windowConstraintHitCount(+ for path: CursorMotionPath,+ relativeTo targetWindow: CursorTargetWindow,+ excludingWindowNumber: Int+ ) -> Int {+ path.sampledConstraintPoints().reduce(into: 0) { result, point in+ if windowID(at: point, excludingWindowNumber: excludingWindowNumber) == targetWindow.windowID {+ result += 1+ }+ }+ }++ private static func windowID(at point: CGPoint, excludingWindowNumber: Int) -> CGWindowID? {+ let windowNumber = NSWindow.windowNumber(+ at: NSPoint(x: point.x, y: point.y),+ belowWindowWithWindowNumber: excludingWindowNumber+ )++ guard windowNumber > 0 else {+ return nil+ }++ return CGWindowID(windowNumber)+ }++ private static func isWindowPresent(_ windowID: CGWindowID) -> Bool {+ guard windowID != 0,+ let windowInfo = CGWindowListCopyWindowInfo([.optionIncludingWindow], windowID) as? [[String: Any]]+ else {+ return false+ }++ return !windowInfo.isEmpty+ }++ private static func refreshActiveOrderingIfNeeded() {+ guard let activeTargetWindow, !isWindowPresent(activeTargetWindow.windowID) else {+ return+ }++ configureOrdering(relativeTo: nil)+ }+private static func animateClickPulse(at point: CGPoint, clickCount: Int, mouseButton: MouseButtonKind) {let pulseBias: CGFloat = mouseButton == .right ? 0.82 : 1@@ -502,6 +624,8 @@ enum SoftwareCursorOverlay {return}+ refreshActiveOrderingIfNeeded()+idlePhase += 0.05let offset = CGPoint(x: sin(idlePhase) * 1.6,