Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Align runtime cursor glyph with CursorMotion

Leo6a4b9412d2a5
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 88ff94e..bebc5e4 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -57,8 +57,8 @@
- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。
- 对真实 app 的 `get_app_state` / action tool 入口,当前新增了一层官方风格的高风险 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app,尽量贴近官方对终端、密码管理器、Chrome 与少量系统敏感组件的防护行为。
- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。
-- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。
-- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在在 `OpenComputerUseKit` 内部复刻 `CursorMotion` procedural fallback 的 pointer/fog glyph 校准,命中点 anchor 固定在 `126x126` 画布里的同一组 tip-offset 上。runtime motion 状态继续按截图 / CGWindow 的 y-down 坐标计算,进入 AppKit 绘制前才做角度和 `dy` 翻转;程序化 pointer 轮廓还会先做一层 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。
+- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;首次显示时 cursor 会从 resting forward 的背后侧出现,避免第一段移动看起来像倒退切入目标。真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。
+- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在和 `CursorMotion` 一样优先渲染仓库里沉淀的 `official-software-cursor-window-252.png` baseline,只有资源缺失时才退回 `OpenComputerUseKit` 内部的程序化 pointer/fog fallback。命中点 anchor 仍固定在 `126x126` 画布里的同一组 tip-offset 上,runtime motion 状态继续按截图 / CGWindow 的 y-down 坐标计算,进入 AppKit 绘制前才做角度和 `dy` 翻转;程序化 fallback 保留 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。
- overlay 的层级不再固定 `.floating`;现在会跟随 snapshot 命中的目标 window id / layer,把自己排到该目标 window 之上,而不是粗暴压到所有前台 app 最上层。
- overlay 的曲线路径不再只按固定 Bezier 模板生成;当前主线采用 reverse-engineering 约束下的 heading-driven candidate 族,候选只保留 `direct` / `turn` / `brake` / `orbit` 这些能稳定产出单侧主弧的 family,并继续保留 target-window 命中策略作为同类候选间的 tie-break。原始 binary lift 恢复出来的 `20` 条路径和 score 仍然保留在独立的 `StandaloneCursor` viewer / Python 重建脚本里,用于对照分析,不再直接作为 runtime 默认 chooser。
- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。
diff --git a/docs/histories/2026-04/20260421-1113-fix-initial-visual-cursor-approach.md b/docs/histories/2026-04/20260421-1113-fix-initial-visual-cursor-approach.md
new file mode 100644
index 0000000..b32f8ce
--- /dev/null
+++ b/docs/histories/2026-04/20260421-1113-fix-initial-visual-cursor-approach.md
@@ -0,0 +1,37 @@
+## [2026-04-21 11:13] | Task: Fix initial visual cursor approach
+
+### Execution Context
+* **Agent ID**: `Codex`
+* **Base Model**: `GPT-5`
+* **Runtime**: `Codex CLI`
+
+### User Query
+> 自己运行 direct call-seq 并用截图排查,修复 `set_value` 一开始看起来倒退过去的问题。
+
+### Changes Overview
+**Scope:** `OpenComputerUseKit` visual cursor runtime
+
+**Key Actions:**
+- **Initial cursor approach**: 将首次显示时的默认出现点从固定屏幕偏移改为基于 resting forward 的反向偏移,保证第一段 travel vector 和 cursor 朝向一致。
+- **Regression coverage**: 增加默认出现点单测,锁定“从 resting forward 背后侧出现”的几何约束。
+- **Documentation**: 更新架构文档里的 overlay 行为说明。
+
+### Design Intent (Why)
+固定 `target + (72, -54)` 偏移和当前 runtime resting forward 不一致,会让首次 `set_value` 的移动阶段看起来像侧向或倒退切入目标。默认出现点应由 cursor 自身的 resting forward 决定,而不是硬编码屏幕方向。
+
+### Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
+
+### Follow-up (2026-04-21, align runtime glyph with CursorMotion)
+
+- **Reference glyph**: `OpenComputerUseKit` 现在和 `CursorMotion` 一样优先加载 `official-software-cursor-window-252.png`,避免程序化 fallback 在主路径上暴露锯齿。
+- **Packaged app resource**: `scripts/build-open-computer-use-app.sh` 会把同一张 cursor baseline PNG 复制进 `Open Computer Use.app/Contents/Resources/`,并声明 `NSHighResolutionCapable`。
+- **Fallback boundary**: 程序化 pointer/fog 继续保留为资源缺失时的 fallback,但不再是 runtime 默认视觉路径。
+
+**Follow-up Files:**
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `scripts/build-open-computer-use-app.sh`
+- `docs/ARCHITECTURE.md`
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift
index da38a27..c616cb2 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift
@@ -41,6 +41,7 @@ struct SoftwareCursorGlyphRenderState {
enum SoftwareCursorGlyphMetrics {
static let windowSize = CGSize(width: 126, height: 126)
static let tipAnchor = CGPoint(x: 60.35, y: 70.3)
+ static let referenceImageResourceName = "official-software-cursor-window-252"
static let pointerSize = CGSize(width: 21, height: 21)
static let pointerOffset = CGPoint(x: 2.6, y: -3.2)
@@ -55,12 +56,25 @@ private enum SoftwareCursorGlyphColors {
}
enum SoftwareCursorGlyphRenderer {
+ private static let referenceImage = loadReferenceCursorWindowImage()
+
static func draw(
in bounds: CGRect,
context: CGContext,
state: SoftwareCursorGlyphRenderState
) {
let drawingState = state.appKitDrawingState
+
+ if let referenceImage {
+ drawReferenceImage(
+ referenceImage,
+ in: bounds,
+ context: context,
+ state: drawingState
+ )
+ return
+ }
+
let pulse = drawingState.clickProgress
let fogCenter = CGPoint(
x: bounds.midX + drawingState.fogOffset.dx,
@@ -88,6 +102,31 @@ enum SoftwareCursorGlyphRenderer {
)
}
+ private static func drawReferenceImage(
+ _ image: NSImage,
+ in bounds: CGRect,
+ context: CGContext,
+ state: SoftwareCursorGlyphRenderState
+ ) {
+ let motionCompression = min(hypot(state.cursorBodyOffset.dx, state.cursorBodyOffset.dy) * 0.008, 0.018)
+ let pulseCompression = state.clickProgress * 0.03
+
+ context.saveGState()
+ context.interpolationQuality = .high
+ context.translateBy(
+ x: bounds.midX + state.cursorBodyOffset.dx,
+ y: bounds.midY + state.cursorBodyOffset.dy
+ )
+ context.rotate(by: state.rotation)
+ context.scaleBy(
+ x: 1 - motionCompression - pulseCompression,
+ y: 1 + (pulseCompression * 0.4)
+ )
+ context.translateBy(x: -bounds.midX, y: -bounds.midY)
+ image.draw(in: bounds, from: .zero, operation: .sourceOver, fraction: 1)
+ context.restoreGState()
+ }
+
private static func drawFog(
in context: CGContext,
center: CGPoint,
@@ -230,3 +269,25 @@ enum SoftwareCursorGlyphRenderer {
return path
}
}
+
+func loadReferenceCursorWindowImage() -> NSImage? {
+ if let bundledReference = Bundle.main.url(
+ forResource: SoftwareCursorGlyphMetrics.referenceImageResourceName,
+ withExtension: "png"
+ ), let image = NSImage(contentsOf: bundledReference) {
+ return image
+ }
+
+ let fileURL = URL(fileURLWithPath: #filePath).standardizedFileURL
+ let repoRoot = fileURL
+ .deletingLastPathComponent()
+ .deletingLastPathComponent()
+ .deletingLastPathComponent()
+ .deletingLastPathComponent()
+ .deletingLastPathComponent()
+
+ let referenceURL = repoRoot
+ .appendingPathComponent("docs/references/codex-computer-use-reverse-engineering/assets/extracted-2026-04-19/\(SoftwareCursorGlyphMetrics.referenceImageResourceName).png")
+
+ return NSImage(contentsOf: referenceURL)
+}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
index 976ed0f..8af3198 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
@@ -32,6 +32,19 @@ func visualCursorEnabled(environment: [String: String]) -> Bool {
return !["0", "false", "no", "off"].contains(rawValue)
}
+func defaultVisualCursorAppearancePoint(
+ for targetPoint: CGPoint,
+ restingForward: CGVector,
+ distance: CGFloat
+) -> CGPoint {
+ let length = max(hypot(restingForward.dx, restingForward.dy), 0.001)
+ let unitForward = CGVector(dx: restingForward.dx / length, dy: restingForward.dy / length)
+ return CGPoint(
+ x: targetPoint.x - (unitForward.dx * distance),
+ y: targetPoint.y - (unitForward.dy * distance)
+ )
+}
+
struct CursorTargetWindow: Equatable, Sendable {
let windowID: CGWindowID
let layer: Int
@@ -70,6 +83,7 @@ private struct CursorArtwork {
enum SoftwareCursorOverlay {
private static let artwork = CursorArtwork.active
private static let baseHeading = -(3 * CGFloat.pi / 4)
+ private static let defaultAppearanceDistance: CGFloat = 90
private static var panel: CursorPanel?
private static var cursorView: SoftwareCursorView?
private static var restingTipPosition: CGPoint?
@@ -533,9 +547,10 @@ enum SoftwareCursorOverlay {
private static func defaultAppearancePoint(for targetPoint: CGPoint) -> CGPoint {
clampTipPosition(
- CGPoint(
- x: targetPoint.x + 72,
- y: targetPoint.y - 54
+ defaultVisualCursorAppearancePoint(
+ for: targetPoint,
+ restingForward: restingForwardVector(),
+ distance: defaultAppearanceDistance
)
)
}
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 6f990a2..df01ea8 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -318,6 +318,15 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertEqual(SoftwareCursorGlyphMetrics.windowSize, CGSize(width: 126, height: 126))
XCTAssertEqual(SoftwareCursorGlyphMetrics.tipAnchor.x, 60.35, accuracy: 0.01)
XCTAssertEqual(SoftwareCursorGlyphMetrics.tipAnchor.y, 70.3, accuracy: 0.01)
+ XCTAssertEqual(SoftwareCursorGlyphMetrics.referenceImageResourceName, "official-software-cursor-window-252")
+ }
+
+ func testSoftwareCursorGlyphLoadsCursorMotionReferenceImage() throws {
+ let image = try XCTUnwrap(loadReferenceCursorWindowImage())
+ let bitmap = try XCTUnwrap(image.representations.first)
+
+ XCTAssertEqual(bitmap.pixelsWide, 252)
+ XCTAssertEqual(bitmap.pixelsHigh, 252)
}
func testSoftwareCursorGlyphArtworkNeutralHeadingMatchesCursorMotionBaseline() {
@@ -354,6 +363,23 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertEqual(drawingState.clickProgress, 0.6)
}
+ func testDefaultVisualCursorAppearanceStartsBehindRestingForward() {
+ let target = CGPoint(x: 300, y: 240)
+ let restingForward = CGVector(dx: -1, dy: -1)
+ let start = defaultVisualCursorAppearancePoint(
+ for: target,
+ restingForward: restingForward,
+ distance: 90
+ )
+ let travelDirection = normalizedVector(from: start, to: target)
+ let expectedDirection = normalizedVector(from: .zero, to: CGPoint(x: restingForward.dx, y: restingForward.dy))
+
+ XCTAssertGreaterThan(start.x, target.x)
+ XCTAssertGreaterThan(start.y, target.y)
+ XCTAssertEqual(travelDirection.dx, expectedDirection.dx, accuracy: 0.0001)
+ XCTAssertEqual(travelDirection.dy, expectedDirection.dy, accuracy: 0.0001)
+ }
+
func testCursorMotionPathStartsAndEndsAtExpectedPoints() {
let path = CursorMotionPath(
start: CGPoint(x: 10, y: 20),
diff --git a/scripts/build-open-computer-use-app.sh b/scripts/build-open-computer-use-app.sh
index ddb010c..dd5ef9d 100755
--- a/scripts/build-open-computer-use-app.sh
+++ b/scripts/build-open-computer-use-app.sh
@@ -241,6 +241,7 @@ legacy_app_bundle_name="OpenComputerUse.app"
bundle_icon_name="OpenComputerUse.icns"
icon_master_png="${repo_root}/assets/app-icons/open-computer-use-1024.png"
iconset_build_script="${repo_root}/scripts/build-apple-iconset.sh"
+cursor_reference_source="${repo_root}/docs/references/codex-computer-use-reverse-engineering/assets/extracted-2026-04-19/official-software-cursor-window-252.png"
bundle_display_name="Open Computer Use"
bundle_identifier="com.ifuryst.opencomputeruse"
@@ -299,6 +300,11 @@ if [[ ! -f "${iconset_build_script}" ]]; then
exit 1
fi
+if [[ ! -f "${cursor_reference_source}" ]]; then
+ echo "Missing cursor reference PNG: ${cursor_reference_source}" >&2
+ exit 1
+fi
+
icon_work_dir="$(mktemp -d "${TMPDIR:-/tmp}/open-computer-use-icon.XXXXXX")"
cleanup() {
if [[ -n "${icon_work_dir:-}" ]]; then
@@ -310,6 +316,7 @@ iconset_dir="${icon_work_dir}/OpenComputerUse.iconset"
mkdir -p "${iconset_dir}"
"${iconset_build_script}" "${icon_master_png}" "${iconset_dir}"
iconutil -c icns "${iconset_dir}" -o "${resources_dir}/${bundle_icon_name}"
+cp "${cursor_reference_source}" "${resources_dir}/official-software-cursor-window-252.png"
cat > "${contents_dir}/Info.plist" <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
@@ -342,6 +349,8 @@ cat > "${contents_dir}/Info.plist" <<PLIST
<string>14.0</string>
<key>LSUIElement</key>
<true/>
+ <key>NSHighResolutionCapable</key>
+ <true/>
<key>NSPrincipalClass</key>
<string>NSApplication</string>
</dict>