Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Re-isolate CursorMotion glyph renderer

Leoefec69e8d397
diff --git a/Package.swift b/Package.swift
index 49e3bbc..6ff8a7e 100644
--- a/Package.swift
+++ b/Package.swift
@@ -34,8 +34,13 @@ let package = Package(
),
],
targets: [
+ .target(
+ name: "SoftwareCursorGlyphKit",
+ path: "packages/SoftwareCursorGlyphKit/Sources/SoftwareCursorGlyphKit"
+ ),
.target(
name: "OpenComputerUseKit",
+ dependencies: ["SoftwareCursorGlyphKit"],
path: "packages/OpenComputerUseKit/Sources/OpenComputerUseKit"
),
.executableTarget(
@@ -55,7 +60,7 @@ let package = Package(
),
.executableTarget(
name: "CursorMotion",
- dependencies: ["OpenComputerUseKit"],
+ dependencies: ["SoftwareCursorGlyphKit"],
path: "experiments/CursorMotion/Sources/CursorMotion"
),
.target(
@@ -69,7 +74,7 @@ let package = Package(
),
.testTarget(
name: "OpenComputerUseKitTests",
- dependencies: ["OpenComputerUseKit"],
+ dependencies: ["OpenComputerUseKit", "SoftwareCursorGlyphKit"],
path: "packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests"
),
.testTarget(
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 2497d20..475a4dc 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -58,7 +58,7 @@
- 对真实 app 的 `get_app_state` / action tool 入口,当前新增了一层官方风格的高风险 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app,尽量贴近官方对终端、密码管理器、Chrome 与少量系统敏感组件的防护行为。
- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。
- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。
-- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;runtime 现在直接复用 `CursorMotion` 同款的程序化 pointer/fog glyph 校准,命中点 anchor 固定在 `126x126` 画布里的同一组 tip-offset 上,避免实验室和主 MCP runtime 各自维护一套 cursor 轮廓。
+- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;runtime 现在直接复用一个中立 `SoftwareCursorGlyphKit` target 里的程序化 pointer/fog glyph 校准,命中点 anchor 固定在 `126x126` 画布里的同一组 tip-offset 上,避免实验室和主 MCP runtime 各自维护一套 cursor 轮廓。
- overlay 的层级不再固定 `.floating`;现在会跟随 snapshot 命中的目标 window id / layer,把自己排到该目标 window 之上,而不是粗暴压到所有前台 app 最上层。
- overlay 的曲线路径不再只按固定 Bezier 模板生成;当前主线采用 reverse-engineering 约束下的 heading-driven candidate 族,候选只保留 `direct` / `turn` / `brake` / `orbit` 这些能稳定产出单侧主弧的 family,并继续保留 target-window 命中策略作为同类候选间的 tie-break。原始 binary lift 恢复出来的 `20` 条路径和 score 仍然保留在独立的 `StandaloneCursor` viewer / Python 重建脚本里,用于对照分析,不再直接作为 runtime 默认 chooser。
- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。
@@ -86,7 +86,7 @@
- 当前它刻意不引入 speculative 的 wall-clock duration 映射,也不复用 `CursorMotion` 里更偏视觉手感试验的 pose dynamics。
- `CursorMotion` 是一个单独的 SwiftUI demo target,可通过 `swift run CursorMotion` 本地启动。
- 这条线优先验证 motion model 本身:当前主线是 heading-driven 的 turn / brake / orbit / direct candidate 族、spring progress、独立 visual dynamics 和 debug UI;moving 阶段真正画出来的箭头角度会持续跟随 visual dynamics 的主 heading,接近停住后再平滑回到默认 resting pose,并在 idle 阶段保留原地小摆角。
-- lab 的 cursor 视觉当前单独参考 `scripts/render-synthesized-software-cursor.swift`:优先使用仓库里保存的官方 `252x252` runtime baseline 图,缺失时再退回脚本里的 procedural pointer/fog 近似;settle 态也改成中心固定的小幅摆角,而不是继续沿 XY 轻微漂移。
+- lab 的 cursor 视觉现在和主 runtime 一样,统一走 `SoftwareCursorGlyphKit` 里的程序化 pointer/fog glyph;独立脚本 `scripts/render-synthesized-software-cursor.swift` 仍保留 reference-vs-procedural 的逆向对照用途,但 demo/runtime 本身已经不再依赖 PNG baseline 图。settle 态也改成中心固定的小幅摆角,而不是继续沿 XY 轻微漂移。
- 当前它不接真实 tool call,也不回写主 `SoftwareCursorOverlay`,目的是把实验噪音与产品行为边界隔离开。
## 关键边界
diff --git a/docs/exec-plans/active/20260418-standalone-cursor-lab.md b/docs/exec-plans/active/20260418-standalone-cursor-lab.md
index 0319e56..8112933 100644
--- a/docs/exec-plans/active/20260418-standalone-cursor-lab.md
+++ b/docs/exec-plans/active/20260418-standalone-cursor-lab.md
@@ -86,7 +86,8 @@
- 2026-04-20:已把独立 demo 的对外命名统一收口到 `Cursor Motion` / `CursorMotion`;Swift Package product、实验目录、README 入口和相关文档现在都以 `swift run CursorMotion` 为准,不再保留 `StandaloneCursorLab` 旧名。
- 2026-04-20:已补一条 tag 驱动的 `Cursor Motion` 分发链路;本地可以通过 `scripts/build-cursor-motion-dmg.sh` 构建 DMG,GitHub Actions 在推送 release tag 后会自动生成 `CursorMotion-<version>.dmg` 并上传到 GitHub Releases。
- 2026-04-20:继续修正“打包版和 `swift run CursorMotion` 观感不一致”的问题;当前已确认 release app 之前没把官方 `252x252` baseline cursor PNG 带进 bundle,导致退回 procedural glyph。现在 `.app` 会优先从 `Bundle.main` 读官方 cursor 图,DMG 打包脚本也会把这张图复制进 `Contents/Resources`,并显式打开 `NSHighResolutionCapable`。
-- 2026-04-20:继续修正 packaged `Cursor Motion` 没有 app icon 的问题;当前先直接复用 `Open Computer Use` 现有的 icon render 流程,在 DMG 打包阶段生成 `CursorMotion.icns` 并写入 `CFBundleIconFile`,让 Finder / Dock 不再只显示通用占位图标。
+- 2026-04-20:继续修正 packaged `Cursor Motion` 的 app icon 资产链;当前已经收口到仓库内 checked-in 的 `1024x1024` master PNG,并在 DMG 打包阶段通过 `sips + iconutil` 生成 `CursorMotion.icns`,避免继续把 Dock 光学尺寸调校耦合在临时 icon render 脚本里。
+- 2026-04-20:继续收口 glyph 边界;`CursorMotion` 已不再优先走 `official-software-cursor-window-252.png`,而是固定使用代码绘制的程序化 pointer/fog glyph,并把这份 renderer 抽到中立 `SoftwareCursorGlyphKit` target,避免实验线重新直接依赖 `OpenComputerUseKit`。
## 决策记录
@@ -105,3 +106,4 @@
- 2026-04-20:对 `Cursor Motion` 的对外交付,当前采用“源码运行 + GitHub Releases 分发 ad-hoc signed DMG”的策略;先把 tag 驱动的可复现封装链路稳定下来,不在这一轮提前引入 notarization 和 Developer ID 签名复杂度。
- 2026-04-20:对 packaged `Cursor Motion` 的 glyph 资源,当前采用“bundle 内官方 baseline 图优先,仓库 reference 路径兜底”的策略;这样 release app 不再因为缺资源而静默退回低保真的 procedural glyph。
- 2026-04-20:对 packaged `Cursor Motion` 的 app icon,当前采用“先直接复用 `Open Computer Use` 的现有 `.icns` 渲染脚本”的策略;先解决 Finder / DMG 里的无图标问题,后续如果需要再单独设计专属 icon。
+- 2026-04-20:对 `CursorMotion` 与主 runtime 的 glyph 共享,当前采用“抽独立 `SoftwareCursorGlyphKit` target、两边都依赖这份程序化 renderer”的策略;这样既能统一 cursor 轮廓,又不会把实验线重新绑回 `OpenComputerUseKit`。
diff --git a/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md b/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md
index 0929c8e..a238388 100644
--- a/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md
+++ b/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md
@@ -57,3 +57,22 @@
- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
- `docs/ARCHITECTURE.md`
- `docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md`
+
+### 🔁 Follow-up (2026-04-20, remove PNG-first glyph split and re-isolate `CursorMotion`)
+
+- **[撤回 `CursorMotion` 的 PNG-first glyph]**: 继续对照当前实现后,确认 `CursorMotion` 仍然优先加载 `official-software-cursor-window-252.png`,这和用户要求的“代码绘制 cursor”不一致;现在 lab 已改成固定使用程序化 pointer/fog glyph,不再依赖这张 PNG。
+- **[解除实验线对主 runtime 的直接依赖]**: 上一轮为了复用 glyph renderer 临时让 `CursorMotion` 直接依赖了 `OpenComputerUseKit`,这和仓库里一直强调的“实验线独立于主 MCP runtime”边界相冲突;现在已把 renderer 抽到中立 `SoftwareCursorGlyphKit` target,由 runtime 和 lab 共同依赖。
+- **[清理打包脚本的过时资源链]**: `scripts/build-cursor-motion-dmg.sh` 不再要求把 `official-software-cursor-window-252.png` 复制进 `.app` bundle,因为 packaged `CursorMotion` 已经不再从 bundle 读取这张图。
+- **[同步当前文档]**: README、架构说明和 active execution plan 都更新成“程序化 glyph + 中立 target 共享”的最终状态,避免继续留下 PNG-first 和 `CursorMotion -> OpenComputerUseKit` 两套互相冲突的说法。
+
+**Follow-up Files:**
+- `Package.swift`
+- `packages/SoftwareCursorGlyphKit/Sources/SoftwareCursorGlyphKit/SoftwareCursorGlyphRenderer.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`
+- `experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `scripts/build-cursor-motion-dmg.sh`
+- `experiments/CursorMotion/README.md`
+- `docs/ARCHITECTURE.md`
+- `docs/exec-plans/active/20260418-standalone-cursor-lab.md`
+- `docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md`
diff --git a/experiments/CursorMotion/README.md b/experiments/CursorMotion/README.md
index 9cc311a..10c371b 100644
--- a/experiments/CursorMotion/README.md
+++ b/experiments/CursorMotion/README.md
@@ -9,6 +9,7 @@
- 主线 `packages/OpenComputerUseKit/.../SoftwareCursorOverlay.swift` 已经承担产品行为,不适合继续塞大量实验代码。
- 用户提供的视频和官方字符串都说明 cursor motion 有独立参数模型,适合先做一个 lab。
- 这块后续可能单独开源,先在目录边界上收口更干净。
+- 当前与主 runtime 共享的只有一个中立 `SoftwareCursorGlyphKit` target,用来承载程序化 cursor glyph;`CursorMotion` 不再直接依赖 `OpenComputerUseKit`。
## 当前模块边界
@@ -19,7 +20,7 @@
- `Sources/CursorLabRootView.swift`
- 本地 demo UI、slider 调参面板、候选路径 overlay 与点击交互
- `Sources/SynthesizedCursorGlyphView.swift`
- - 参考 `scripts/render-synthesized-software-cursor.swift` 的 baseline/procedural cursor renderer
+ - 复用共享 `SoftwareCursorGlyphKit` 的程序化 pointer/fog glyph renderer
## 当前参考
@@ -55,7 +56,7 @@ swift run CursorMotion
- 可见 cursor 不再直接贴在 path sample 上,而是经过独立 visual dynamics 状态,再输出 `rotation + cursorBodyOffset + fogOffset + fogScale`。
- 候选路径现在显式约束“先顺车头方向掉头,再沿主轴推进,再按 resting pose 收尾”;因此大多数跨向移动会呈现单侧 C 形,需要直接切入时才会退化为近直线,而不会再出现两侧乱甩的 S 形扭曲。
- 箭头的可见角度现在重新对齐官方抽帧与逆向证据:moving 阶段持续跟随当前 move heading,接近停住后再平滑回到默认 resting pose,并继续做原地小摆角。
-- cursor glyph 不再走之前那套亮白 asset;当前优先直接显示仓库里的官方 `252x252` runtime baseline 图,缺失时才退回脚本同款 procedural pointer/fog。
+- cursor glyph 现在固定走代码绘制的程序化 pointer/fog renderer,不再依赖 `252x252` PNG baseline 图或其它图片资产。
- settle 态不再做 XY 漂移;现在改成和参考脚本一致的中心固定小摆角,让“停住以后原地轻微转动”的观感先对齐。
后续实现应优先保持:
diff --git a/experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swift b/experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swift
index ea712e5..ddcaa2b 100644
--- a/experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swift
+++ b/experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swift
@@ -1,7 +1,7 @@
import AppKit
import CoreGraphics
import Foundation
-import OpenComputerUseKit
+import SoftwareCursorGlyphKit
@MainActor
final class SynthesizedCursorGlyphView: NSView {
@@ -29,8 +29,6 @@ final class SynthesizedCursorGlyphView: NSView {
didSet { needsDisplay = true }
}
- private let referenceImage = loadReferenceCursorWindowImage()
-
override init(frame frameRect: NSRect) {
super.init(frame: frameRect)
wantsLayer = true
@@ -56,34 +54,9 @@ final class SynthesizedCursorGlyphView: NSView {
return
}
- if let referenceImage {
- drawReferenceImage(referenceImage, in: context)
- return
- }
-
drawProceduralGlyph(in: context)
}
- private func drawReferenceImage(_ image: NSImage, in context: CGContext) {
- let drawingBodyOffset = drawingVector(from: cursorBodyOffset)
- let motionCompression = min(hypot(cursorBodyOffset.dx, cursorBodyOffset.dy) * 0.008, 0.018)
- let pulseCompression = clickProgress * 0.03
-
- context.saveGState()
- context.translateBy(
- x: bounds.midX + drawingBodyOffset.dx,
- y: bounds.midY + drawingBodyOffset.dy
- )
- context.rotate(by: drawingAngle(from: rotation - CursorGlyphCalibration.restingRotation))
- context.scaleBy(
- x: 1 - motionCompression - pulseCompression,
- y: 1 + (pulseCompression * 0.4)
- )
- context.translateBy(x: -bounds.midX, y: -bounds.midY)
- image.draw(in: bounds, from: .zero, operation: .sourceOver, fraction: 1)
- context.restoreGState()
- }
-
private func drawProceduralGlyph(in context: CGContext) {
SoftwareCursorGlyphRenderer.draw(
in: bounds,
@@ -107,25 +80,3 @@ final class SynthesizedCursorGlyphView: NSView {
-screenAngle
}
}
-
-private func loadReferenceCursorWindowImage() -> NSImage? {
- if let bundledReference = Bundle.main.url(
- forResource: "official-software-cursor-window-252",
- withExtension: "png"
- ), let image = NSImage(contentsOf: bundledReference) {
- return image
- }
-
- let fileURL = URL(fileURLWithPath: #filePath).standardizedFileURL
- let repoRoot = fileURL
- .deletingLastPathComponent()
- .deletingLastPathComponent()
- .deletingLastPathComponent()
- .deletingLastPathComponent()
- .deletingLastPathComponent()
-
- let referenceURL = repoRoot
- .appendingPathComponent("docs/references/codex-computer-use-reverse-engineering/assets/extracted-2026-04-19/official-software-cursor-window-252.png")
-
- return NSImage(contentsOf: referenceURL)
-}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
index ff36d99..a328089 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
@@ -2,6 +2,7 @@ import AppKit
import CoreGraphics
import Foundation
import QuartzCore
+import SoftwareCursorGlyphKit
public enum VisualCursorSupport {
public static var isEnabled: Bool {
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index a42733c..ba1cf29 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -1,4 +1,5 @@
import AppKit
+import SoftwareCursorGlyphKit
import XCTest
@testable import OpenComputerUseKit
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift b/packages/SoftwareCursorGlyphKit/Sources/SoftwareCursorGlyphKit/SoftwareCursorGlyphRenderer.swift
similarity index 100%
rename from packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift
rename to packages/SoftwareCursorGlyphKit/Sources/SoftwareCursorGlyphKit/SoftwareCursorGlyphRenderer.swift
diff --git a/scripts/build-cursor-motion-dmg.sh b/scripts/build-cursor-motion-dmg.sh
index 1a7cb1d..8a068a1 100755
--- a/scripts/build-cursor-motion-dmg.sh
+++ b/scripts/build-cursor-motion-dmg.sh
@@ -214,8 +214,8 @@ app_name="Cursor Motion.app"
bundle_identifier="com.ifuryst.cursormotion"
bundle_version="${CURSOR_MOTION_BUNDLE_VERSION:-${GITHUB_RUN_NUMBER:-$(git -C "${repo_root}" rev-list --count HEAD 2>/dev/null || echo 1)}}"
bundle_icon_name="CursorMotion.icns"
-icon_render_script="${repo_root}/scripts/render-open-computer-use-icon.swift"
-cursor_reference_source="${repo_root}/docs/references/codex-computer-use-reverse-engineering/assets/extracted-2026-04-19/official-software-cursor-window-252.png"
+icon_master_png="${repo_root}/assets/app-icons/open-computer-use-1024.png"
+iconset_build_script="${repo_root}/scripts/build-apple-iconset.sh"
app_root="${output_dir}/${app_name}"
contents_dir="${app_root}/Contents"
macos_dir="${contents_dir}/MacOS"
@@ -248,15 +248,13 @@ esac
chmod +x "${macos_dir}/CursorMotion"
-if [[ ! -f "${cursor_reference_source}" ]]; then
- echo "Missing cursor reference asset: ${cursor_reference_source}" >&2
+if [[ ! -f "${icon_master_png}" ]]; then
+ echo "Missing icon master PNG: ${icon_master_png}" >&2
exit 1
fi
-cp "${cursor_reference_source}" "${resources_dir}/official-software-cursor-window-252.png"
-
-if [[ ! -f "${icon_render_script}" ]]; then
- echo "Missing icon render script: ${icon_render_script}" >&2
+if [[ ! -f "${iconset_build_script}" ]]; then
+ echo "Missing iconset build script: ${iconset_build_script}" >&2
exit 1
fi
@@ -270,7 +268,7 @@ trap cleanup EXIT
icon_work_dir="$(mktemp -d "${TMPDIR:-/tmp}/cursor-motion-icon.XXXXXX")"
iconset_dir="${icon_work_dir}/CursorMotion.iconset"
mkdir -p "${iconset_dir}"
-swift "${icon_render_script}" "${iconset_dir}"
+"${iconset_build_script}" "${icon_master_png}" "${iconset_dir}"
iconutil -c icns "${iconset_dir}" -o "${resources_dir}/${bundle_icon_name}"
cat > "${contents_dir}/Info.plist" <<PLIST
diff --git a/scripts/computer-use-cli/README.md b/scripts/computer-use-cli/README.md
index 08e6736..cc6b280 100644
--- a/scripts/computer-use-cli/README.md
+++ b/scripts/computer-use-cli/README.md
@@ -64,6 +64,9 @@ This repo includes a ready-to-run example at
That sample covers all 9 official tools against `TextEdit`, and it intentionally
repeats `get_app_state` between state-changing actions because the official
bundled `computer-use` invalidates the cached app state after each mutation.
+Before running it, make sure `TextEdit` already has a normal document window open;
+otherwise `get_app_state` may only expose the application root and the hard-coded
+`element_index` values in the sample will no longer line up.
For a smaller cursor-comparison path, use
`examples/textedit-set-value-click-raise-seq.json`.