👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Align runtime cursor glyph with CursorMotion
Leo6a4b9412d2a5
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 88ff94e..bebc5e4 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -57,8 +57,8 @@- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。- 对真实 app 的 `get_app_state` / action tool 入口,当前新增了一层官方风格的高风险 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app,尽量贴近官方对终端、密码管理器、Chrome 与少量系统敏感组件的防护行为。- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。-- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。-- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在在 `OpenComputerUseKit` 内部复刻 `CursorMotion` procedural fallback 的 pointer/fog glyph 校准,命中点 anchor 固定在 `126x126` 画布里的同一组 tip-offset 上。runtime motion 状态继续按截图 / CGWindow 的 y-down 坐标计算,进入 AppKit 绘制前才做角度和 `dy` 翻转;程序化 pointer 轮廓还会先做一层 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。+- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;首次显示时 cursor 会从 resting forward 的背后侧出现,避免第一段移动看起来像倒退切入目标。真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。+- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在和 `CursorMotion` 一样优先渲染仓库里沉淀的 `official-software-cursor-window-252.png` baseline,只有资源缺失时才退回 `OpenComputerUseKit` 内部的程序化 pointer/fog fallback。命中点 anchor 仍固定在 `126x126` 画布里的同一组 tip-offset 上,runtime motion 状态继续按截图 / CGWindow 的 y-down 坐标计算,进入 AppKit 绘制前才做角度和 `dy` 翻转;程序化 fallback 保留 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。- overlay 的层级不再固定 `.floating`;现在会跟随 snapshot 命中的目标 window id / layer,把自己排到该目标 window 之上,而不是粗暴压到所有前台 app 最上层。- overlay 的曲线路径不再只按固定 Bezier 模板生成;当前主线采用 reverse-engineering 约束下的 heading-driven candidate 族,候选只保留 `direct` / `turn` / `brake` / `orbit` 这些能稳定产出单侧主弧的 family,并继续保留 target-window 命中策略作为同类候选间的 tie-break。原始 binary lift 恢复出来的 `20` 条路径和 score 仍然保留在独立的 `StandaloneCursor` viewer / Python 重建脚本里,用于对照分析,不再直接作为 runtime 默认 chooser。- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。diff --git a/docs/histories/2026-04/20260421-1113-fix-initial-visual-cursor-approach.md b/docs/histories/2026-04/20260421-1113-fix-initial-visual-cursor-approach.mdnew file mode 100644index 0000000..b32f8ce--- /dev/null+++ b/docs/histories/2026-04/20260421-1113-fix-initial-visual-cursor-approach.md@@ -0,0 +1,37 @@+## [2026-04-21 11:13] | Task: Fix initial visual cursor approach++### Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `Codex CLI`++### User Query+> 自己运行 direct call-seq 并用截图排查,修复 `set_value` 一开始看起来倒退过去的问题。++### Changes Overview+**Scope:** `OpenComputerUseKit` visual cursor runtime++**Key Actions:**+- **Initial cursor approach**: 将首次显示时的默认出现点从固定屏幕偏移改为基于 resting forward 的反向偏移,保证第一段 travel vector 和 cursor 朝向一致。+- **Regression coverage**: 增加默认出现点单测,锁定“从 resting forward 背后侧出现”的几何约束。+- **Documentation**: 更新架构文档里的 overlay 行为说明。++### Design Intent (Why)+固定 `target + (72, -54)` 偏移和当前 runtime resting forward 不一致,会让首次 `set_value` 的移动阶段看起来像侧向或倒退切入目标。默认出现点应由 cursor 自身的 resting forward 决定,而不是硬编码屏幕方向。++### Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`++### Follow-up (2026-04-21, align runtime glyph with CursorMotion)++- **Reference glyph**: `OpenComputerUseKit` 现在和 `CursorMotion` 一样优先加载 `official-software-cursor-window-252.png`,避免程序化 fallback 在主路径上暴露锯齿。+- **Packaged app resource**: `scripts/build-open-computer-use-app.sh` 会把同一张 cursor baseline PNG 复制进 `Open Computer Use.app/Contents/Resources/`,并声明 `NSHighResolutionCapable`。+- **Fallback boundary**: 程序化 pointer/fog 继续保留为资源缺失时的 fallback,但不再是 runtime 默认视觉路径。++**Follow-up Files:**+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `scripts/build-open-computer-use-app.sh`+- `docs/ARCHITECTURE.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swiftindex da38a27..c616cb2 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift@@ -41,6 +41,7 @@ struct SoftwareCursorGlyphRenderState {enum SoftwareCursorGlyphMetrics {static let windowSize = CGSize(width: 126, height: 126)static let tipAnchor = CGPoint(x: 60.35, y: 70.3)+ static let referenceImageResourceName = "official-software-cursor-window-252"static let pointerSize = CGSize(width: 21, height: 21)static let pointerOffset = CGPoint(x: 2.6, y: -3.2)@@ -55,12 +56,25 @@ private enum SoftwareCursorGlyphColors {}enum SoftwareCursorGlyphRenderer {+ private static let referenceImage = loadReferenceCursorWindowImage()+static func draw(in bounds: CGRect,context: CGContext,state: SoftwareCursorGlyphRenderState) {let drawingState = state.appKitDrawingState++ if let referenceImage {+ drawReferenceImage(+ referenceImage,+ in: bounds,+ context: context,+ state: drawingState+ )+ return+ }+let pulse = drawingState.clickProgresslet fogCenter = CGPoint(x: bounds.midX + drawingState.fogOffset.dx,@@ -88,6 +102,31 @@ enum SoftwareCursorGlyphRenderer {)}+ private static func drawReferenceImage(+ _ image: NSImage,+ in bounds: CGRect,+ context: CGContext,+ state: SoftwareCursorGlyphRenderState+ ) {+ let motionCompression = min(hypot(state.cursorBodyOffset.dx, state.cursorBodyOffset.dy) * 0.008, 0.018)+ let pulseCompression = state.clickProgress * 0.03++ context.saveGState()+ context.interpolationQuality = .high+ context.translateBy(+ x: bounds.midX + state.cursorBodyOffset.dx,+ y: bounds.midY + state.cursorBodyOffset.dy+ )+ context.rotate(by: state.rotation)+ context.scaleBy(+ x: 1 - motionCompression - pulseCompression,+ y: 1 + (pulseCompression * 0.4)+ )+ context.translateBy(x: -bounds.midX, y: -bounds.midY)+ image.draw(in: bounds, from: .zero, operation: .sourceOver, fraction: 1)+ context.restoreGState()+ }+private static func drawFog(in context: CGContext,center: CGPoint,@@ -230,3 +269,25 @@ enum SoftwareCursorGlyphRenderer {return path}}++func loadReferenceCursorWindowImage() -> NSImage? {+ if let bundledReference = Bundle.main.url(+ forResource: SoftwareCursorGlyphMetrics.referenceImageResourceName,+ withExtension: "png"+ ), let image = NSImage(contentsOf: bundledReference) {+ return image+ }++ let fileURL = URL(fileURLWithPath: #filePath).standardizedFileURL+ let repoRoot = fileURL+ .deletingLastPathComponent()+ .deletingLastPathComponent()+ .deletingLastPathComponent()+ .deletingLastPathComponent()+ .deletingLastPathComponent()++ let referenceURL = repoRoot+ .appendingPathComponent("docs/references/codex-computer-use-reverse-engineering/assets/extracted-2026-04-19/\(SoftwareCursorGlyphMetrics.referenceImageResourceName).png")++ return NSImage(contentsOf: referenceURL)+}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swiftindex 976ed0f..8af3198 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift@@ -32,6 +32,19 @@ func visualCursorEnabled(environment: [String: String]) -> Bool {return !["0", "false", "no", "off"].contains(rawValue)}+func defaultVisualCursorAppearancePoint(+ for targetPoint: CGPoint,+ restingForward: CGVector,+ distance: CGFloat+) -> CGPoint {+ let length = max(hypot(restingForward.dx, restingForward.dy), 0.001)+ let unitForward = CGVector(dx: restingForward.dx / length, dy: restingForward.dy / length)+ return CGPoint(+ x: targetPoint.x - (unitForward.dx * distance),+ y: targetPoint.y - (unitForward.dy * distance)+ )+}+struct CursorTargetWindow: Equatable, Sendable {let windowID: CGWindowIDlet layer: Int@@ -70,6 +83,7 @@ private struct CursorArtwork {enum SoftwareCursorOverlay {private static let artwork = CursorArtwork.activeprivate static let baseHeading = -(3 * CGFloat.pi / 4)+ private static let defaultAppearanceDistance: CGFloat = 90private static var panel: CursorPanel?private static var cursorView: SoftwareCursorView?private static var restingTipPosition: CGPoint?@@ -533,9 +547,10 @@ enum SoftwareCursorOverlay {private static func defaultAppearancePoint(for targetPoint: CGPoint) -> CGPoint {clampTipPosition(- CGPoint(- x: targetPoint.x + 72,- y: targetPoint.y - 54+ defaultVisualCursorAppearancePoint(+ for: targetPoint,+ restingForward: restingForwardVector(),+ distance: defaultAppearanceDistance))}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 6f990a2..df01ea8 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -318,6 +318,15 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(SoftwareCursorGlyphMetrics.windowSize, CGSize(width: 126, height: 126))XCTAssertEqual(SoftwareCursorGlyphMetrics.tipAnchor.x, 60.35, accuracy: 0.01)XCTAssertEqual(SoftwareCursorGlyphMetrics.tipAnchor.y, 70.3, accuracy: 0.01)+ XCTAssertEqual(SoftwareCursorGlyphMetrics.referenceImageResourceName, "official-software-cursor-window-252")+ }++ func testSoftwareCursorGlyphLoadsCursorMotionReferenceImage() throws {+ let image = try XCTUnwrap(loadReferenceCursorWindowImage())+ let bitmap = try XCTUnwrap(image.representations.first)++ XCTAssertEqual(bitmap.pixelsWide, 252)+ XCTAssertEqual(bitmap.pixelsHigh, 252)}func testSoftwareCursorGlyphArtworkNeutralHeadingMatchesCursorMotionBaseline() {@@ -354,6 +363,23 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(drawingState.clickProgress, 0.6)}+ func testDefaultVisualCursorAppearanceStartsBehindRestingForward() {+ let target = CGPoint(x: 300, y: 240)+ let restingForward = CGVector(dx: -1, dy: -1)+ let start = defaultVisualCursorAppearancePoint(+ for: target,+ restingForward: restingForward,+ distance: 90+ )+ let travelDirection = normalizedVector(from: start, to: target)+ let expectedDirection = normalizedVector(from: .zero, to: CGPoint(x: restingForward.dx, y: restingForward.dy))++ XCTAssertGreaterThan(start.x, target.x)+ XCTAssertGreaterThan(start.y, target.y)+ XCTAssertEqual(travelDirection.dx, expectedDirection.dx, accuracy: 0.0001)+ XCTAssertEqual(travelDirection.dy, expectedDirection.dy, accuracy: 0.0001)+ }+func testCursorMotionPathStartsAndEndsAtExpectedPoints() {let path = CursorMotionPath(start: CGPoint(x: 10, y: 20),diff --git a/scripts/build-open-computer-use-app.sh b/scripts/build-open-computer-use-app.shindex ddb010c..dd5ef9d 100755--- a/scripts/build-open-computer-use-app.sh+++ b/scripts/build-open-computer-use-app.sh@@ -241,6 +241,7 @@ legacy_app_bundle_name="OpenComputerUse.app"bundle_icon_name="OpenComputerUse.icns"icon_master_png="${repo_root}/assets/app-icons/open-computer-use-1024.png"iconset_build_script="${repo_root}/scripts/build-apple-iconset.sh"+cursor_reference_source="${repo_root}/docs/references/codex-computer-use-reverse-engineering/assets/extracted-2026-04-19/official-software-cursor-window-252.png"bundle_display_name="Open Computer Use"bundle_identifier="com.ifuryst.opencomputeruse"@@ -299,6 +300,11 @@ if [[ ! -f "${iconset_build_script}" ]]; thenexit 1fi+if [[ ! -f "${cursor_reference_source}" ]]; then+ echo "Missing cursor reference PNG: ${cursor_reference_source}" >&2+ exit 1+fi+icon_work_dir="$(mktemp -d "${TMPDIR:-/tmp}/open-computer-use-icon.XXXXXX")"cleanup() {if [[ -n "${icon_work_dir:-}" ]]; then@@ -310,6 +316,7 @@ iconset_dir="${icon_work_dir}/OpenComputerUse.iconset"mkdir -p "${iconset_dir}""${iconset_build_script}" "${icon_master_png}" "${iconset_dir}"iconutil -c icns "${iconset_dir}" -o "${resources_dir}/${bundle_icon_name}"+cp "${cursor_reference_source}" "${resources_dir}/official-software-cursor-window-252.png"cat > "${contents_dir}/Info.plist" <<PLIST<?xml version="1.0" encoding="UTF-8"?>@@ -342,6 +349,8 @@ cat > "${contents_dir}/Info.plist" <<PLIST<string>14.0</string><key>LSUIElement</key><true/>+ <key>NSHighResolutionCapable</key>+ <true/><key>NSPrincipalClass</key><string>NSApplication</string></dict>