Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Fix runtime visual cursor heading

Leo98b4477917f9
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index bebc5e4..824363c 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -57,8 +57,8 @@
- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。
- 对真实 app 的 `get_app_state` / action tool 入口,当前新增了一层官方风格的高风险 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app,尽量贴近官方对终端、密码管理器、Chrome 与少量系统敏感组件的防护行为。
- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。
-- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;首次显示时 cursor 会从 resting forward 的背后侧出现,避免第一段移动看起来像倒退切入目标。真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。
-- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在和 `CursorMotion` 一样优先渲染仓库里沉淀的 `official-software-cursor-window-252.png` baseline,只有资源缺失时才退回 `OpenComputerUseKit` 内部的程序化 pointer/fog fallback。命中点 anchor 仍固定在 `126x126` 画布里的同一组 tip-offset 上,runtime motion 状态继续按截图 / CGWindow 的 y-down 坐标计算,进入 AppKit 绘制前才做角度和 `dy` 翻转;程序化 fallback 保留 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。
+- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;首次显示时按官方 binary 的 fresh state 从 AppKit 全局 `(0,0)` window origin 生成起点,后续动作继续复用上一帧 visible tip。真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。
+- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在和 `CursorMotion` 一样优先渲染仓库里沉淀的 `official-software-cursor-window-252.png` baseline,只有资源缺失时才退回 `OpenComputerUseKit` 内部的程序化 pointer/fog fallback。命中点 anchor 仍固定在 `126x126` 画布里的同一组 tip-offset 上;glyph 自身的 neutral heading 继续沿用 `CursorMotion` / 官方 baseline 的 `-3π/4`。主 runtime overlay window 按 AppKit 全局坐标移动;路径选路用屏幕上实际可见的 AppKit forward heading,进入 visual dynamics / render state 前则把 velocity 的 y 轴翻回 CursorMotion 的 y-down screen state,再交给 AppKit 绘制层做角度和 `dy` 翻转。程序化 fallback 保留 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。
- overlay 的层级不再固定 `.floating`;现在会跟随 snapshot 命中的目标 window id / layer,把自己排到该目标 window 之上,而不是粗暴压到所有前台 app 最上层。
- overlay 的曲线路径不再只按固定 Bezier 模板生成;当前主线采用 reverse-engineering 约束下的 heading-driven candidate 族,候选只保留 `direct` / `turn` / `brake` / `orbit` 这些能稳定产出单侧主弧的 family,并继续保留 target-window 命中策略作为同类候选间的 tie-break。原始 binary lift 恢复出来的 `20` 条路径和 score 仍然保留在独立的 `StandaloneCursor` viewer / Python 重建脚本里,用于对照分析,不再直接作为 runtime 默认 chooser。
- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。
diff --git a/docs/histories/2026-04/20260421-1211-align-cursor-initial-origin.md b/docs/histories/2026-04/20260421-1211-align-cursor-initial-origin.md
new file mode 100644
index 0000000..0879edb
--- /dev/null
+++ b/docs/histories/2026-04/20260421-1211-align-cursor-initial-origin.md
@@ -0,0 +1,31 @@
+## [2026-04-21 12:11] | Task: 对齐 cursor 首次起点到官方 `(0,0)`
+
+### 🤖 Execution Context
+* **Agent ID**: `Codex`
+* **Base Model**: `GPT-5`
+* **Runtime**: `local CLI`
+
+### 📥 User Query
+> 基于官方 `.app` 逆向发现 fresh cursor 从屏幕左下角 `(0,0)` 出现后,希望本仓库实现也从 `(0,0)` 开始。
+
+### 🛠 Changes Overview
+**Scope:** `packages/OpenComputerUseKit`、`docs/`
+
+**Key Actions:**
+- **[Initial Origin Alignment]**: `SoftwareCursorOverlay` 在没有上一帧 cursor tip 时,不再从目标点背后生成起点,而是按官方 fresh state 用 AppKit 全局 `(0,0)` window origin 计算首次 tip。
+- **[Regression Coverage]**: 更新单测,验证默认首次 tip 对应的 cursor window origin 为 `(0,0)`。
+- **[Docs Sync]**: 更新架构说明,明确首次显示从 `(0,0)` window origin 起步,后续动作继续复用上一帧 visible tip。
+
+### 🧠 Design Intent (Why)
+官方 `SkyComputerUseService` 的 `ComputerUseCursor.Window` 初始化会把 `currentInterpolatedOrigin` 和 `NSWindow` 初始 content rect 都置为 `(0,0)`。主运行时应该复用这个 fresh-session 语义,避免第一段移动从目标附近切入而偏离官方观感。
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
+- `docs/histories/2026-04/20260421-1211-align-cursor-initial-origin.md`
+
+### 🔁 Follow-up (2026-04-21 12:23)
+- **[Runtime Heading Fix]**: 保留 `CursorMotion` / glyph resource 的 `-3π/4` neutral heading;主 runtime overlay 仍按 AppKit 全局坐标放置 window,路径选路用实际可见的 AppKit forward heading,但进入 visual dynamics / render state 前会把 velocity 的 y 轴翻回 CursorMotion 的 y-down screen state,避免上行时 cursor 侧边朝前。
+- **[Regression Coverage]**: 补单测锁住 AppKit 上行速度到 CursorMotion screen-state velocity、render rotation、最终 AppKit forward heading 的转换关系。
+- **[Docs Sync]**: 更新架构说明,明确 artwork calibration 与 runtime motion coordinate basis 分层。
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorMotionModel.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorMotionModel.swift
index c8a0003..02d4467 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorMotionModel.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/CursorMotionModel.swift
@@ -1254,6 +1254,7 @@ enum CursorVisualDynamicsAnimator {
targetTime: CGFloat,
idleAngleOffset: CGFloat = 0,
baseHeading: CGFloat,
+ renderYAxisMultiplier: CGFloat = 1,
configuration: CursorVisualDynamicsConfiguration = .officialInspired
) -> (state: CursorVisualDynamicsState, renderState: CursorVisualRenderState) {
var adjustedState = state
@@ -1268,6 +1269,7 @@ enum CursorVisualDynamicsAnimator {
targetTipPosition: targetTipPosition,
idleAngleOffset: idleAngleOffset,
baseHeading: baseHeading,
+ renderYAxisMultiplier: renderYAxisMultiplier,
configuration: configuration
)
}
@@ -1278,6 +1280,7 @@ enum CursorVisualDynamicsAnimator {
state: adjustedState,
idleAngleOffset: idleAngleOffset,
baseHeading: baseHeading,
+ renderYAxisMultiplier: renderYAxisMultiplier,
configuration: configuration
)
)
@@ -1288,6 +1291,7 @@ enum CursorVisualDynamicsAnimator {
targetTipPosition: CGPoint,
idleAngleOffset: CGFloat,
baseHeading: CGFloat,
+ renderYAxisMultiplier: CGFloat,
configuration: CursorVisualDynamicsConfiguration
) -> CursorVisualDynamicsState {
let dt = configuration.tipSpring.dt
@@ -1299,9 +1303,13 @@ enum CursorVisualDynamicsAnimator {
let tipForce = tipDisplacement.scaled(by: configuration.tipSpring.stiffness)
+ tipVelocityHalf.scaled(by: -configuration.tipSpring.drag)
let tipVelocity = tipVelocityHalf + tipForce.scaled(by: halfDT)
+ let renderVelocity = visualCursorScreenStateVelocity(
+ fromRuntimeVelocity: tipVelocity,
+ yAxisMultiplier: renderYAxisMultiplier
+ )
let targetAngle = resolvedTargetAngle(
- velocity: tipVelocity,
+ velocity: renderVelocity,
idleAngleOffset: idleAngleOffset,
baseHeading: baseHeading,
configuration: configuration
@@ -1328,11 +1336,16 @@ enum CursorVisualDynamicsAnimator {
state: CursorVisualDynamicsState,
idleAngleOffset: CGFloat,
baseHeading: CGFloat,
+ renderYAxisMultiplier: CGFloat,
configuration: CursorVisualDynamicsConfiguration
) -> CursorVisualRenderState {
- let speed = state.tipVelocity.length
+ let renderVelocity = visualCursorScreenStateVelocity(
+ fromRuntimeVelocity: state.tipVelocity,
+ yAxisMultiplier: renderYAxisMultiplier
+ )
+ let speed = renderVelocity.length
let direction = speed > 0.001
- ? state.tipVelocity.normalized
+ ? renderVelocity.normalized
: CGVector(dx: cos(baseHeading + idleAngleOffset), dy: sin(baseHeading + idleAngleOffset))
let bodyBackward = direction.scaled(
by: -min(speed * configuration.bodyOffsetScale, configuration.bodyOffsetMax)
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
index 8af3198..482276a 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
@@ -32,19 +32,42 @@ func visualCursorEnabled(environment: [String: String]) -> Bool {
return !["0", "false", "no", "off"].contains(rawValue)
}
-func defaultVisualCursorAppearancePoint(
- for targetPoint: CGPoint,
- restingForward: CGVector,
- distance: CGFloat
+func defaultVisualCursorInitialTipPosition(
+ windowOrigin: CGPoint = .zero,
+ tipAnchor: CGPoint = SoftwareCursorGlyphMetrics.tipAnchor
) -> CGPoint {
- let length = max(hypot(restingForward.dx, restingForward.dy), 0.001)
- let unitForward = CGVector(dx: restingForward.dx / length, dy: restingForward.dy / length)
return CGPoint(
- x: targetPoint.x - (unitForward.dx * distance),
- y: targetPoint.y - (unitForward.dy * distance)
+ x: windowOrigin.x + tipAnchor.x,
+ y: windowOrigin.y + tipAnchor.y
)
}
+func visualCursorRenderBaseHeading(
+ artworkNeutralHeading: CGFloat = SoftwareCursorGlyphMetrics.targetNeutralHeading
+) -> CGFloat {
+ artworkNeutralHeading
+}
+
+func visualCursorAppKitForwardHeading(
+ renderRotation: CGFloat,
+ artworkNeutralHeading: CGFloat = SoftwareCursorGlyphMetrics.targetNeutralHeading
+) -> CGFloat {
+ -artworkNeutralHeading - renderRotation
+}
+
+func visualCursorRuntimeRenderYAxisMultiplier() -> CGFloat {
+ // Window placement uses AppKit global coordinates, but glyph render state is
+ // still interpreted as CursorMotion's y-down screen state before drawing.
+ -1
+}
+
+func visualCursorScreenStateVelocity(
+ fromRuntimeVelocity velocity: CGVector,
+ yAxisMultiplier: CGFloat
+) -> CGVector {
+ CGVector(dx: velocity.dx, dy: velocity.dy * yAxisMultiplier)
+}
+
struct CursorTargetWindow: Equatable, Sendable {
let windowID: CGWindowID
let layer: Int
@@ -82,8 +105,8 @@ private struct CursorArtwork {
@MainActor
enum SoftwareCursorOverlay {
private static let artwork = CursorArtwork.active
- private static let baseHeading = -(3 * CGFloat.pi / 4)
- private static let defaultAppearanceDistance: CGFloat = 90
+ private static let renderBaseHeading = visualCursorRenderBaseHeading()
+ private static let renderYAxisMultiplier = visualCursorRuntimeRenderYAxisMultiplier()
private static var panel: CursorPanel?
private static var cursorView: SoftwareCursorView?
private static var restingTipPosition: CGPoint?
@@ -105,18 +128,24 @@ enum SoftwareCursorOverlay {
configureOrdering(relativeTo: targetWindow)
let constrainedTarget = clampTipPosition(targetPoint)
- let startPoint = displayedTipPosition ?? defaultAppearancePoint(for: constrainedTarget)
+ let isFreshStart = displayedTipPosition == nil
+ let startPoint = displayedTipPosition ?? defaultInitialTipPosition()
let now = CACurrentMediaTime()
panel?.alphaValue = 1
- seedVisualDynamicsIfNeeded(at: startPoint, time: now)
- placeCursor(
- using: advanceVisualDynamics(
- toward: startPoint,
- at: now
- ),
- clickProgress: 0
- )
+ if isFreshStart {
+ visualDynamicsState = CursorVisualDynamicsAnimator.state(at: startPoint, time: CGFloat(now))
+ placeCursor(using: initialRenderState(at: startPoint), clickProgress: 0)
+ } else {
+ seedVisualDynamicsIfNeeded(at: startPoint, time: now)
+ placeCursor(
+ using: advanceVisualDynamics(
+ toward: startPoint,
+ at: now
+ ),
+ clickProgress: 0
+ )
+ }
if distanceBetween(startPoint, constrainedTarget) > 2 {
animateMove(from: startPoint, to: constrainedTarget, relativeTo: targetWindow)
@@ -342,7 +371,7 @@ enum SoftwareCursorOverlay {
}
private static func forwardVector(renderRotation: CGFloat) -> CGVector {
- let angle = baseHeading + renderRotation
+ let angle = visualCursorAppKitForwardHeading(renderRotation: renderRotation)
return CGVector(dx: cos(angle), dy: sin(angle))
}
@@ -545,13 +574,21 @@ enum SoftwareCursorOverlay {
}
}
- private static func defaultAppearancePoint(for targetPoint: CGPoint) -> CGPoint {
- clampTipPosition(
- defaultVisualCursorAppearancePoint(
- for: targetPoint,
- restingForward: restingForwardVector(),
- distance: defaultAppearanceDistance
- )
+ private static func defaultInitialTipPosition() -> CGPoint {
+ defaultVisualCursorInitialTipPosition(
+ windowOrigin: .zero,
+ tipAnchor: artwork.geometry.tipAnchor
+ )
+ }
+
+ private static func initialRenderState(at tipPosition: CGPoint) -> CursorVisualRenderState {
+ CursorVisualRenderState(
+ tipPosition: tipPosition,
+ rotation: 0,
+ cursorBodyOffset: CGVector(dx: 0, dy: 0),
+ fogOffset: CGVector(dx: 0, dy: 0),
+ fogOpacity: CursorVisualDynamicsConfiguration.officialInspired.fogOpacityBase,
+ fogScale: 1
)
}
@@ -579,7 +616,8 @@ enum SoftwareCursorOverlay {
targetTipPosition: clampedTarget,
targetTime: CGFloat(time),
idleAngleOffset: idleAngleOffset,
- baseHeading: baseHeading
+ baseHeading: renderBaseHeading,
+ renderYAxisMultiplier: renderYAxisMultiplier
)
visualDynamicsState = result.state
return result.renderState
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index bc425d3..6c7c59c 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -363,21 +363,48 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertEqual(drawingState.clickProgress, 0.6)
}
- func testDefaultVisualCursorAppearanceStartsBehindRestingForward() {
- let target = CGPoint(x: 300, y: 240)
- let restingForward = CGVector(dx: -1, dy: -1)
- let start = defaultVisualCursorAppearancePoint(
- for: target,
- restingForward: restingForward,
- distance: 90
+ func testDefaultVisualCursorInitialTipMatchesZeroWindowOrigin() {
+ let geometry = CursorWindowGeometry(
+ windowSize: CGSize(width: 126, height: 126),
+ tipAnchor: CGPoint(x: 60.35, y: 70.3)
+ )
+ let start = defaultVisualCursorInitialTipPosition(
+ windowOrigin: .zero,
+ tipAnchor: geometry.tipAnchor
)
- let travelDirection = normalizedVector(from: start, to: target)
- let expectedDirection = normalizedVector(from: .zero, to: CGPoint(x: restingForward.dx, y: restingForward.dy))
- XCTAssertGreaterThan(start.x, target.x)
- XCTAssertGreaterThan(start.y, target.y)
- XCTAssertEqual(travelDirection.dx, expectedDirection.dx, accuracy: 0.0001)
- XCTAssertEqual(travelDirection.dy, expectedDirection.dy, accuracy: 0.0001)
+ XCTAssertEqual(geometry.origin(forTipPosition: start), .zero)
+ XCTAssertEqual(start.x, geometry.tipAnchor.x, accuracy: 0.0001)
+ XCTAssertEqual(start.y, geometry.tipAnchor.y, accuracy: 0.0001)
+ }
+
+ func testVisualCursorRuntimeMapsAppKitUpwardMotionToCursorMotionScreenState() {
+ let renderBaseHeading = visualCursorRenderBaseHeading(
+ artworkNeutralHeading: SoftwareCursorGlyphMetrics.targetNeutralHeading
+ )
+ let screenVelocity = visualCursorScreenStateVelocity(
+ fromRuntimeVelocity: CGVector(dx: 0, dy: 1),
+ yAxisMultiplier: visualCursorRuntimeRenderYAxisMultiplier()
+ )
+ let renderRotation = normalizedAngle(atan2(screenVelocity.dy, screenVelocity.dx) - renderBaseHeading)
+ let appKitForwardHeading = visualCursorAppKitForwardHeading(
+ renderRotation: renderRotation,
+ artworkNeutralHeading: SoftwareCursorGlyphMetrics.targetNeutralHeading
+ )
+
+ XCTAssertEqual(renderBaseHeading, -(3 * CGFloat.pi / 4), accuracy: 0.0001)
+ XCTAssertEqual(screenVelocity.dx, 0, accuracy: 0.0001)
+ XCTAssertEqual(screenVelocity.dy, -1, accuracy: 0.0001)
+ XCTAssertEqual(renderRotation, CGFloat.pi / 4, accuracy: 0.0001)
+ XCTAssertEqual(normalizedAngle(appKitForwardHeading), CGFloat.pi / 2, accuracy: 0.0001)
+ XCTAssertEqual(
+ visualCursorAppKitForwardHeading(
+ renderRotation: 0,
+ artworkNeutralHeading: SoftwareCursorGlyphMetrics.targetNeutralHeading
+ ),
+ 3 * CGFloat.pi / 4,
+ accuracy: 0.0001
+ )
}
func testCursorMotionPathStartsAndEndsAtExpectedPoints() {
@@ -605,6 +632,17 @@ final class OpenComputerUseKitTests: XCTestCase {
return samples
}
+ private func normalizedAngle(_ angle: CGFloat) -> CGFloat {
+ var value = angle
+ while value > .pi {
+ value -= 2 * .pi
+ }
+ while value < -.pi {
+ value += 2 * .pi
+ }
+ return value
+ }
+
private func normalizedVector(from start: CGPoint, to end: CGPoint) -> CGVector {
let dx = end.x - start.x
let dy = end.y - start.y