👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Fix runtime visual cursor heading
Leo98b4477917f9
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex bebc5e4..824363c 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -57,8 +57,8 @@- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。- 对真实 app 的 `get_app_state` / action tool 入口,当前新增了一层官方风格的高风险 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app,尽量贴近官方对终端、密码管理器、Chrome 与少量系统敏感组件的防护行为。- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。-- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;首次显示时 cursor 会从 resting forward 的背后侧出现,避免第一段移动看起来像倒退切入目标。真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。-- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在和 `CursorMotion` 一样优先渲染仓库里沉淀的 `official-software-cursor-window-252.png` baseline,只有资源缺失时才退回 `OpenComputerUseKit` 内部的程序化 pointer/fog fallback。命中点 anchor 仍固定在 `126x126` 画布里的同一组 tip-offset 上,runtime motion 状态继续按截图 / CGWindow 的 y-down 坐标计算,进入 AppKit 绘制前才做角度和 `dy` 翻转;程序化 fallback 保留 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。+- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;首次显示时按官方 binary 的 fresh state 从 AppKit 全局 `(0,0)` window origin 生成起点,后续动作继续复用上一帧 visible tip。真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。+- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在和 `CursorMotion` 一样优先渲染仓库里沉淀的 `official-software-cursor-window-252.png` baseline,只有资源缺失时才退回 `OpenComputerUseKit` 内部的程序化 pointer/fog fallback。命中点 anchor 仍固定在 `126x126` 画布里的同一组 tip-offset 上;glyph 自身的 neutral heading 继续沿用 `CursorMotion` / 官方 baseline 的 `-3π/4`。主 runtime overlay window 按 AppKit 全局坐标移动;路径选路用屏幕上实际可见的 AppKit forward heading,进入 visual dynamics / render state 前则把 velocity 的 y 轴翻回 CursorMotion 的 y-down screen state,再交给 AppKit 绘制层做角度和 `dy` 翻转。程序化 fallback 保留 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。- overlay 的层级不再固定 `.floating`;现在会跟随 snapshot 命中的目标 window id / layer,把自己排到该目标 window 之上,而不是粗暴压到所有前台 app 最上层。- overlay 的曲线路径不再只按固定 Bezier 模板生成;当前主线采用 reverse-engineering 约束下的 heading-driven candidate 族,候选只保留 `direct` / `turn` / `brake` / `orbit` 这些能稳定产出单侧主弧的 family,并继续保留 target-window 命中策略作为同类候选间的 tie-break。原始 binary lift 恢复出来的 `20` 条路径和 score 仍然保留在独立的 `StandaloneCursor` viewer / Python 重建脚本里,用于对照分析,不再直接作为 runtime 默认 chooser。- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。diff --git a/docs/histories/2026-04/20260421-1211-align-cursor-initial-origin.md b/docs/histories/2026-04/20260421-1211-align-cursor-initial-origin.mdnew file mode 100644index 0000000..0879edb--- /dev/null+++ b/docs/histories/2026-04/20260421-1211-align-cursor-initial-origin.md@@ -0,0 +1,31 @@+## [2026-04-21 12:11] | Task: 对齐 cursor 首次起点到官方 `(0,0)`++### 🤖 Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `local CLI`++### 📥 User Query+> 基于官方 `.app` 逆向发现 fresh cursor 从屏幕左下角 `(0,0)` 出现后,希望本仓库实现也从 `(0,0)` 开始。++### 🛠 Changes Overview+**Scope:** `packages/OpenComputerUseKit`、`docs/`++**Key Actions:**+- **[Initial Origin Alignment]**: `SoftwareCursorOverlay` 在没有上一帧 cursor tip 时,不再从目标点背后生成起点,而是按官方 fresh state 用 AppKit 全局 `(0,0)` window origin 计算首次 tip。+- **[Regression Coverage]**: 更新单测,验证默认首次 tip 对应的 cursor window origin 为 `(0,0)`。+- **[Docs Sync]**: 更新架构说明,明确首次显示从 `(0,0)` window origin 起步,后续动作继续复用上一帧 visible tip。++### 🧠 Design Intent (Why)+官方 `SkyComputerUseService` 的 `ComputerUseCursor.Window` 初始化会把 `currentInterpolatedOrigin` 和 `NSWindow` 初始 content rect 都置为 `(0,0)`。主运行时应该复用这个 fresh-session 语义,避免第一段移动从目标附近切入而偏离官方观感。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-04/20260421-1211-align-cursor-initial-origin.md`++### 🔁 Follow-up (2026-04-21 12:23)+- **[Runtime Heading Fix]**: 保留 `CursorMotion` / glyph resource 的 `-3π/4` neutral heading;主 runtime overlay 仍按 AppKit 全局坐标放置 window,路径选路用实际可见的 AppKit forward heading,但进入 visual dynamics / render state 前会把 velocity 的 y 轴翻回 CursorMotion 的 y-down screen state,避免上行时 cursor 侧边朝前。+- **[Regression Coverage]**: 补单测锁住 AppKit 上行速度到 CursorMotion screen-state velocity、render rotation、最终 AppKit forward heading 的转换关系。+- **[Docs Sync]**: 更新架构说明,明确 artwork calibration 与 runtime motion coordinate basis 分层。diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorMotionModel.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorMotionModel.swiftindex c8a0003..02d4467 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorMotionModel.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorMotionModel.swift@@ -1254,6 +1254,7 @@ enum CursorVisualDynamicsAnimator {targetTime: CGFloat,idleAngleOffset: CGFloat = 0,baseHeading: CGFloat,+ renderYAxisMultiplier: CGFloat = 1,configuration: CursorVisualDynamicsConfiguration = .officialInspired) -> (state: CursorVisualDynamicsState, renderState: CursorVisualRenderState) {var adjustedState = state@@ -1268,6 +1269,7 @@ enum CursorVisualDynamicsAnimator {targetTipPosition: targetTipPosition,idleAngleOffset: idleAngleOffset,baseHeading: baseHeading,+ renderYAxisMultiplier: renderYAxisMultiplier,configuration: configuration)}@@ -1278,6 +1280,7 @@ enum CursorVisualDynamicsAnimator {state: adjustedState,idleAngleOffset: idleAngleOffset,baseHeading: baseHeading,+ renderYAxisMultiplier: renderYAxisMultiplier,configuration: configuration))@@ -1288,6 +1291,7 @@ enum CursorVisualDynamicsAnimator {targetTipPosition: CGPoint,idleAngleOffset: CGFloat,baseHeading: CGFloat,+ renderYAxisMultiplier: CGFloat,configuration: CursorVisualDynamicsConfiguration) -> CursorVisualDynamicsState {let dt = configuration.tipSpring.dt@@ -1299,9 +1303,13 @@ enum CursorVisualDynamicsAnimator {let tipForce = tipDisplacement.scaled(by: configuration.tipSpring.stiffness)+ tipVelocityHalf.scaled(by: -configuration.tipSpring.drag)let tipVelocity = tipVelocityHalf + tipForce.scaled(by: halfDT)+ let renderVelocity = visualCursorScreenStateVelocity(+ fromRuntimeVelocity: tipVelocity,+ yAxisMultiplier: renderYAxisMultiplier+ )let targetAngle = resolvedTargetAngle(- velocity: tipVelocity,+ velocity: renderVelocity,idleAngleOffset: idleAngleOffset,baseHeading: baseHeading,configuration: configuration@@ -1328,11 +1336,16 @@ enum CursorVisualDynamicsAnimator {state: CursorVisualDynamicsState,idleAngleOffset: CGFloat,baseHeading: CGFloat,+ renderYAxisMultiplier: CGFloat,configuration: CursorVisualDynamicsConfiguration) -> CursorVisualRenderState {- let speed = state.tipVelocity.length+ let renderVelocity = visualCursorScreenStateVelocity(+ fromRuntimeVelocity: state.tipVelocity,+ yAxisMultiplier: renderYAxisMultiplier+ )+ let speed = renderVelocity.lengthlet direction = speed > 0.001- ? state.tipVelocity.normalized+ ? renderVelocity.normalized: CGVector(dx: cos(baseHeading + idleAngleOffset), dy: sin(baseHeading + idleAngleOffset))let bodyBackward = direction.scaled(by: -min(speed * configuration.bodyOffsetScale, configuration.bodyOffsetMax)diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swiftindex 8af3198..482276a 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift@@ -32,19 +32,42 @@ func visualCursorEnabled(environment: [String: String]) -> Bool {return !["0", "false", "no", "off"].contains(rawValue)}-func defaultVisualCursorAppearancePoint(- for targetPoint: CGPoint,- restingForward: CGVector,- distance: CGFloat+func defaultVisualCursorInitialTipPosition(+ windowOrigin: CGPoint = .zero,+ tipAnchor: CGPoint = SoftwareCursorGlyphMetrics.tipAnchor) -> CGPoint {- let length = max(hypot(restingForward.dx, restingForward.dy), 0.001)- let unitForward = CGVector(dx: restingForward.dx / length, dy: restingForward.dy / length)return CGPoint(- x: targetPoint.x - (unitForward.dx * distance),- y: targetPoint.y - (unitForward.dy * distance)+ x: windowOrigin.x + tipAnchor.x,+ y: windowOrigin.y + tipAnchor.y)}+func visualCursorRenderBaseHeading(+ artworkNeutralHeading: CGFloat = SoftwareCursorGlyphMetrics.targetNeutralHeading+) -> CGFloat {+ artworkNeutralHeading+}++func visualCursorAppKitForwardHeading(+ renderRotation: CGFloat,+ artworkNeutralHeading: CGFloat = SoftwareCursorGlyphMetrics.targetNeutralHeading+) -> CGFloat {+ -artworkNeutralHeading - renderRotation+}++func visualCursorRuntimeRenderYAxisMultiplier() -> CGFloat {+ // Window placement uses AppKit global coordinates, but glyph render state is+ // still interpreted as CursorMotion's y-down screen state before drawing.+ -1+}++func visualCursorScreenStateVelocity(+ fromRuntimeVelocity velocity: CGVector,+ yAxisMultiplier: CGFloat+) -> CGVector {+ CGVector(dx: velocity.dx, dy: velocity.dy * yAxisMultiplier)+}+struct CursorTargetWindow: Equatable, Sendable {let windowID: CGWindowIDlet layer: Int@@ -82,8 +105,8 @@ private struct CursorArtwork {@MainActorenum SoftwareCursorOverlay {private static let artwork = CursorArtwork.active- private static let baseHeading = -(3 * CGFloat.pi / 4)- private static let defaultAppearanceDistance: CGFloat = 90+ private static let renderBaseHeading = visualCursorRenderBaseHeading()+ private static let renderYAxisMultiplier = visualCursorRuntimeRenderYAxisMultiplier()private static var panel: CursorPanel?private static var cursorView: SoftwareCursorView?private static var restingTipPosition: CGPoint?@@ -105,18 +128,24 @@ enum SoftwareCursorOverlay {configureOrdering(relativeTo: targetWindow)let constrainedTarget = clampTipPosition(targetPoint)- let startPoint = displayedTipPosition ?? defaultAppearancePoint(for: constrainedTarget)+ let isFreshStart = displayedTipPosition == nil+ let startPoint = displayedTipPosition ?? defaultInitialTipPosition()let now = CACurrentMediaTime()panel?.alphaValue = 1- seedVisualDynamicsIfNeeded(at: startPoint, time: now)- placeCursor(- using: advanceVisualDynamics(- toward: startPoint,- at: now- ),- clickProgress: 0- )+ if isFreshStart {+ visualDynamicsState = CursorVisualDynamicsAnimator.state(at: startPoint, time: CGFloat(now))+ placeCursor(using: initialRenderState(at: startPoint), clickProgress: 0)+ } else {+ seedVisualDynamicsIfNeeded(at: startPoint, time: now)+ placeCursor(+ using: advanceVisualDynamics(+ toward: startPoint,+ at: now+ ),+ clickProgress: 0+ )+ }if distanceBetween(startPoint, constrainedTarget) > 2 {animateMove(from: startPoint, to: constrainedTarget, relativeTo: targetWindow)@@ -342,7 +371,7 @@ enum SoftwareCursorOverlay {}private static func forwardVector(renderRotation: CGFloat) -> CGVector {- let angle = baseHeading + renderRotation+ let angle = visualCursorAppKitForwardHeading(renderRotation: renderRotation)return CGVector(dx: cos(angle), dy: sin(angle))}@@ -545,13 +574,21 @@ enum SoftwareCursorOverlay {}}- private static func defaultAppearancePoint(for targetPoint: CGPoint) -> CGPoint {- clampTipPosition(- defaultVisualCursorAppearancePoint(- for: targetPoint,- restingForward: restingForwardVector(),- distance: defaultAppearanceDistance- )+ private static func defaultInitialTipPosition() -> CGPoint {+ defaultVisualCursorInitialTipPosition(+ windowOrigin: .zero,+ tipAnchor: artwork.geometry.tipAnchor+ )+ }++ private static func initialRenderState(at tipPosition: CGPoint) -> CursorVisualRenderState {+ CursorVisualRenderState(+ tipPosition: tipPosition,+ rotation: 0,+ cursorBodyOffset: CGVector(dx: 0, dy: 0),+ fogOffset: CGVector(dx: 0, dy: 0),+ fogOpacity: CursorVisualDynamicsConfiguration.officialInspired.fogOpacityBase,+ fogScale: 1)}@@ -579,7 +616,8 @@ enum SoftwareCursorOverlay {targetTipPosition: clampedTarget,targetTime: CGFloat(time),idleAngleOffset: idleAngleOffset,- baseHeading: baseHeading+ baseHeading: renderBaseHeading,+ renderYAxisMultiplier: renderYAxisMultiplier)visualDynamicsState = result.statereturn result.renderStatediff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex bc425d3..6c7c59c 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -363,21 +363,48 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(drawingState.clickProgress, 0.6)}- func testDefaultVisualCursorAppearanceStartsBehindRestingForward() {- let target = CGPoint(x: 300, y: 240)- let restingForward = CGVector(dx: -1, dy: -1)- let start = defaultVisualCursorAppearancePoint(- for: target,- restingForward: restingForward,- distance: 90+ func testDefaultVisualCursorInitialTipMatchesZeroWindowOrigin() {+ let geometry = CursorWindowGeometry(+ windowSize: CGSize(width: 126, height: 126),+ tipAnchor: CGPoint(x: 60.35, y: 70.3)+ )+ let start = defaultVisualCursorInitialTipPosition(+ windowOrigin: .zero,+ tipAnchor: geometry.tipAnchor)- let travelDirection = normalizedVector(from: start, to: target)- let expectedDirection = normalizedVector(from: .zero, to: CGPoint(x: restingForward.dx, y: restingForward.dy))- XCTAssertGreaterThan(start.x, target.x)- XCTAssertGreaterThan(start.y, target.y)- XCTAssertEqual(travelDirection.dx, expectedDirection.dx, accuracy: 0.0001)- XCTAssertEqual(travelDirection.dy, expectedDirection.dy, accuracy: 0.0001)+ XCTAssertEqual(geometry.origin(forTipPosition: start), .zero)+ XCTAssertEqual(start.x, geometry.tipAnchor.x, accuracy: 0.0001)+ XCTAssertEqual(start.y, geometry.tipAnchor.y, accuracy: 0.0001)+ }++ func testVisualCursorRuntimeMapsAppKitUpwardMotionToCursorMotionScreenState() {+ let renderBaseHeading = visualCursorRenderBaseHeading(+ artworkNeutralHeading: SoftwareCursorGlyphMetrics.targetNeutralHeading+ )+ let screenVelocity = visualCursorScreenStateVelocity(+ fromRuntimeVelocity: CGVector(dx: 0, dy: 1),+ yAxisMultiplier: visualCursorRuntimeRenderYAxisMultiplier()+ )+ let renderRotation = normalizedAngle(atan2(screenVelocity.dy, screenVelocity.dx) - renderBaseHeading)+ let appKitForwardHeading = visualCursorAppKitForwardHeading(+ renderRotation: renderRotation,+ artworkNeutralHeading: SoftwareCursorGlyphMetrics.targetNeutralHeading+ )++ XCTAssertEqual(renderBaseHeading, -(3 * CGFloat.pi / 4), accuracy: 0.0001)+ XCTAssertEqual(screenVelocity.dx, 0, accuracy: 0.0001)+ XCTAssertEqual(screenVelocity.dy, -1, accuracy: 0.0001)+ XCTAssertEqual(renderRotation, CGFloat.pi / 4, accuracy: 0.0001)+ XCTAssertEqual(normalizedAngle(appKitForwardHeading), CGFloat.pi / 2, accuracy: 0.0001)+ XCTAssertEqual(+ visualCursorAppKitForwardHeading(+ renderRotation: 0,+ artworkNeutralHeading: SoftwareCursorGlyphMetrics.targetNeutralHeading+ ),+ 3 * CGFloat.pi / 4,+ accuracy: 0.0001+ )}func testCursorMotionPathStartsAndEndsAtExpectedPoints() {@@ -605,6 +632,17 @@ final class OpenComputerUseKitTests: XCTestCase {return samples}+ private func normalizedAngle(_ angle: CGFloat) -> CGFloat {+ var value = angle+ while value > .pi {+ value -= 2 * .pi+ }+ while value < -.pi {+ value += 2 * .pi+ }+ return value+ }+private func normalizedVector(from start: CGPoint, to end: CGPoint) -> CGVector {let dx = end.x - start.xlet dy = end.y - start.y