👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Re-isolate CursorMotion glyph renderer
Leoefec69e8d397
diff --git a/Package.swift b/Package.swiftindex 49e3bbc..6ff8a7e 100644--- a/Package.swift+++ b/Package.swift@@ -34,8 +34,13 @@ let package = Package(),],targets: [+ .target(+ name: "SoftwareCursorGlyphKit",+ path: "packages/SoftwareCursorGlyphKit/Sources/SoftwareCursorGlyphKit"+ ),.target(name: "OpenComputerUseKit",+ dependencies: ["SoftwareCursorGlyphKit"],path: "packages/OpenComputerUseKit/Sources/OpenComputerUseKit"),.executableTarget(@@ -55,7 +60,7 @@ let package = Package(),.executableTarget(name: "CursorMotion",- dependencies: ["OpenComputerUseKit"],+ dependencies: ["SoftwareCursorGlyphKit"],path: "experiments/CursorMotion/Sources/CursorMotion"),.target(@@ -69,7 +74,7 @@ let package = Package(),.testTarget(name: "OpenComputerUseKitTests",- dependencies: ["OpenComputerUseKit"],+ dependencies: ["OpenComputerUseKit", "SoftwareCursorGlyphKit"],path: "packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests"),.testTarget(diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 2497d20..475a4dc 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -58,7 +58,7 @@- 对真实 app 的 `get_app_state` / action tool 入口,当前新增了一层官方风格的高风险 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app,尽量贴近官方对终端、密码管理器、Chrome 与少量系统敏感组件的防护行为。- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。-- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;runtime 现在直接复用 `CursorMotion` 同款的程序化 pointer/fog glyph 校准,命中点 anchor 固定在 `126x126` 画布里的同一组 tip-offset 上,避免实验室和主 MCP runtime 各自维护一套 cursor 轮廓。+- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;runtime 现在直接复用一个中立 `SoftwareCursorGlyphKit` target 里的程序化 pointer/fog glyph 校准,命中点 anchor 固定在 `126x126` 画布里的同一组 tip-offset 上,避免实验室和主 MCP runtime 各自维护一套 cursor 轮廓。- overlay 的层级不再固定 `.floating`;现在会跟随 snapshot 命中的目标 window id / layer,把自己排到该目标 window 之上,而不是粗暴压到所有前台 app 最上层。- overlay 的曲线路径不再只按固定 Bezier 模板生成;当前主线采用 reverse-engineering 约束下的 heading-driven candidate 族,候选只保留 `direct` / `turn` / `brake` / `orbit` 这些能稳定产出单侧主弧的 family,并继续保留 target-window 命中策略作为同类候选间的 tie-break。原始 binary lift 恢复出来的 `20` 条路径和 score 仍然保留在独立的 `StandaloneCursor` viewer / Python 重建脚本里,用于对照分析,不再直接作为 runtime 默认 chooser。- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。@@ -86,7 +86,7 @@- 当前它刻意不引入 speculative 的 wall-clock duration 映射,也不复用 `CursorMotion` 里更偏视觉手感试验的 pose dynamics。- `CursorMotion` 是一个单独的 SwiftUI demo target,可通过 `swift run CursorMotion` 本地启动。- 这条线优先验证 motion model 本身:当前主线是 heading-driven 的 turn / brake / orbit / direct candidate 族、spring progress、独立 visual dynamics 和 debug UI;moving 阶段真正画出来的箭头角度会持续跟随 visual dynamics 的主 heading,接近停住后再平滑回到默认 resting pose,并在 idle 阶段保留原地小摆角。-- lab 的 cursor 视觉当前单独参考 `scripts/render-synthesized-software-cursor.swift`:优先使用仓库里保存的官方 `252x252` runtime baseline 图,缺失时再退回脚本里的 procedural pointer/fog 近似;settle 态也改成中心固定的小幅摆角,而不是继续沿 XY 轻微漂移。+- lab 的 cursor 视觉现在和主 runtime 一样,统一走 `SoftwareCursorGlyphKit` 里的程序化 pointer/fog glyph;独立脚本 `scripts/render-synthesized-software-cursor.swift` 仍保留 reference-vs-procedural 的逆向对照用途,但 demo/runtime 本身已经不再依赖 PNG baseline 图。settle 态也改成中心固定的小幅摆角,而不是继续沿 XY 轻微漂移。- 当前它不接真实 tool call,也不回写主 `SoftwareCursorOverlay`,目的是把实验噪音与产品行为边界隔离开。## 关键边界diff --git a/docs/exec-plans/active/20260418-standalone-cursor-lab.md b/docs/exec-plans/active/20260418-standalone-cursor-lab.mdindex 0319e56..8112933 100644--- a/docs/exec-plans/active/20260418-standalone-cursor-lab.md+++ b/docs/exec-plans/active/20260418-standalone-cursor-lab.md@@ -86,7 +86,8 @@- 2026-04-20:已把独立 demo 的对外命名统一收口到 `Cursor Motion` / `CursorMotion`;Swift Package product、实验目录、README 入口和相关文档现在都以 `swift run CursorMotion` 为准,不再保留 `StandaloneCursorLab` 旧名。- 2026-04-20:已补一条 tag 驱动的 `Cursor Motion` 分发链路;本地可以通过 `scripts/build-cursor-motion-dmg.sh` 构建 DMG,GitHub Actions 在推送 release tag 后会自动生成 `CursorMotion-<version>.dmg` 并上传到 GitHub Releases。- 2026-04-20:继续修正“打包版和 `swift run CursorMotion` 观感不一致”的问题;当前已确认 release app 之前没把官方 `252x252` baseline cursor PNG 带进 bundle,导致退回 procedural glyph。现在 `.app` 会优先从 `Bundle.main` 读官方 cursor 图,DMG 打包脚本也会把这张图复制进 `Contents/Resources`,并显式打开 `NSHighResolutionCapable`。-- 2026-04-20:继续修正 packaged `Cursor Motion` 没有 app icon 的问题;当前先直接复用 `Open Computer Use` 现有的 icon render 流程,在 DMG 打包阶段生成 `CursorMotion.icns` 并写入 `CFBundleIconFile`,让 Finder / Dock 不再只显示通用占位图标。+- 2026-04-20:继续修正 packaged `Cursor Motion` 的 app icon 资产链;当前已经收口到仓库内 checked-in 的 `1024x1024` master PNG,并在 DMG 打包阶段通过 `sips + iconutil` 生成 `CursorMotion.icns`,避免继续把 Dock 光学尺寸调校耦合在临时 icon render 脚本里。+- 2026-04-20:继续收口 glyph 边界;`CursorMotion` 已不再优先走 `official-software-cursor-window-252.png`,而是固定使用代码绘制的程序化 pointer/fog glyph,并把这份 renderer 抽到中立 `SoftwareCursorGlyphKit` target,避免实验线重新直接依赖 `OpenComputerUseKit`。## 决策记录@@ -105,3 +106,4 @@- 2026-04-20:对 `Cursor Motion` 的对外交付,当前采用“源码运行 + GitHub Releases 分发 ad-hoc signed DMG”的策略;先把 tag 驱动的可复现封装链路稳定下来,不在这一轮提前引入 notarization 和 Developer ID 签名复杂度。- 2026-04-20:对 packaged `Cursor Motion` 的 glyph 资源,当前采用“bundle 内官方 baseline 图优先,仓库 reference 路径兜底”的策略;这样 release app 不再因为缺资源而静默退回低保真的 procedural glyph。- 2026-04-20:对 packaged `Cursor Motion` 的 app icon,当前采用“先直接复用 `Open Computer Use` 的现有 `.icns` 渲染脚本”的策略;先解决 Finder / DMG 里的无图标问题,后续如果需要再单独设计专属 icon。+- 2026-04-20:对 `CursorMotion` 与主 runtime 的 glyph 共享,当前采用“抽独立 `SoftwareCursorGlyphKit` target、两边都依赖这份程序化 renderer”的策略;这样既能统一 cursor 轮廓,又不会把实验线重新绑回 `OpenComputerUseKit`。diff --git a/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md b/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.mdindex 0929c8e..a238388 100644--- a/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md+++ b/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md@@ -57,3 +57,22 @@- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`- `docs/ARCHITECTURE.md`- `docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md`++### 🔁 Follow-up (2026-04-20, remove PNG-first glyph split and re-isolate `CursorMotion`)++- **[撤回 `CursorMotion` 的 PNG-first glyph]**: 继续对照当前实现后,确认 `CursorMotion` 仍然优先加载 `official-software-cursor-window-252.png`,这和用户要求的“代码绘制 cursor”不一致;现在 lab 已改成固定使用程序化 pointer/fog glyph,不再依赖这张 PNG。+- **[解除实验线对主 runtime 的直接依赖]**: 上一轮为了复用 glyph renderer 临时让 `CursorMotion` 直接依赖了 `OpenComputerUseKit`,这和仓库里一直强调的“实验线独立于主 MCP runtime”边界相冲突;现在已把 renderer 抽到中立 `SoftwareCursorGlyphKit` target,由 runtime 和 lab 共同依赖。+- **[清理打包脚本的过时资源链]**: `scripts/build-cursor-motion-dmg.sh` 不再要求把 `official-software-cursor-window-252.png` 复制进 `.app` bundle,因为 packaged `CursorMotion` 已经不再从 bundle 读取这张图。+- **[同步当前文档]**: README、架构说明和 active execution plan 都更新成“程序化 glyph + 中立 target 共享”的最终状态,避免继续留下 PNG-first 和 `CursorMotion -> OpenComputerUseKit` 两套互相冲突的说法。++**Follow-up Files:**+- `Package.swift`+- `packages/SoftwareCursorGlyphKit/Sources/SoftwareCursorGlyphKit/SoftwareCursorGlyphRenderer.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`+- `experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `scripts/build-cursor-motion-dmg.sh`+- `experiments/CursorMotion/README.md`+- `docs/ARCHITECTURE.md`+- `docs/exec-plans/active/20260418-standalone-cursor-lab.md`+- `docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md`diff --git a/experiments/CursorMotion/README.md b/experiments/CursorMotion/README.mdindex 9cc311a..10c371b 100644--- a/experiments/CursorMotion/README.md+++ b/experiments/CursorMotion/README.md@@ -9,6 +9,7 @@- 主线 `packages/OpenComputerUseKit/.../SoftwareCursorOverlay.swift` 已经承担产品行为,不适合继续塞大量实验代码。- 用户提供的视频和官方字符串都说明 cursor motion 有独立参数模型,适合先做一个 lab。- 这块后续可能单独开源,先在目录边界上收口更干净。+- 当前与主 runtime 共享的只有一个中立 `SoftwareCursorGlyphKit` target,用来承载程序化 cursor glyph;`CursorMotion` 不再直接依赖 `OpenComputerUseKit`。## 当前模块边界@@ -19,7 +20,7 @@- `Sources/CursorLabRootView.swift`- 本地 demo UI、slider 调参面板、候选路径 overlay 与点击交互- `Sources/SynthesizedCursorGlyphView.swift`- - 参考 `scripts/render-synthesized-software-cursor.swift` 的 baseline/procedural cursor renderer+ - 复用共享 `SoftwareCursorGlyphKit` 的程序化 pointer/fog glyph renderer## 当前参考@@ -55,7 +56,7 @@ swift run CursorMotion- 可见 cursor 不再直接贴在 path sample 上,而是经过独立 visual dynamics 状态,再输出 `rotation + cursorBodyOffset + fogOffset + fogScale`。- 候选路径现在显式约束“先顺车头方向掉头,再沿主轴推进,再按 resting pose 收尾”;因此大多数跨向移动会呈现单侧 C 形,需要直接切入时才会退化为近直线,而不会再出现两侧乱甩的 S 形扭曲。- 箭头的可见角度现在重新对齐官方抽帧与逆向证据:moving 阶段持续跟随当前 move heading,接近停住后再平滑回到默认 resting pose,并继续做原地小摆角。-- cursor glyph 不再走之前那套亮白 asset;当前优先直接显示仓库里的官方 `252x252` runtime baseline 图,缺失时才退回脚本同款 procedural pointer/fog。+- cursor glyph 现在固定走代码绘制的程序化 pointer/fog renderer,不再依赖 `252x252` PNG baseline 图或其它图片资产。- settle 态不再做 XY 漂移;现在改成和参考脚本一致的中心固定小摆角,让“停住以后原地轻微转动”的观感先对齐。后续实现应优先保持:diff --git a/experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swift b/experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swiftindex ea712e5..ddcaa2b 100644--- a/experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swift+++ b/experiments/CursorMotion/Sources/CursorMotion/SynthesizedCursorGlyphView.swift@@ -1,7 +1,7 @@import AppKitimport CoreGraphicsimport Foundation-import OpenComputerUseKit+import SoftwareCursorGlyphKit@MainActorfinal class SynthesizedCursorGlyphView: NSView {@@ -29,8 +29,6 @@ final class SynthesizedCursorGlyphView: NSView {didSet { needsDisplay = true }}- private let referenceImage = loadReferenceCursorWindowImage()-override init(frame frameRect: NSRect) {super.init(frame: frameRect)wantsLayer = true@@ -56,34 +54,9 @@ final class SynthesizedCursorGlyphView: NSView {return}- if let referenceImage {- drawReferenceImage(referenceImage, in: context)- return- }-drawProceduralGlyph(in: context)}- private func drawReferenceImage(_ image: NSImage, in context: CGContext) {- let drawingBodyOffset = drawingVector(from: cursorBodyOffset)- let motionCompression = min(hypot(cursorBodyOffset.dx, cursorBodyOffset.dy) * 0.008, 0.018)- let pulseCompression = clickProgress * 0.03-- context.saveGState()- context.translateBy(- x: bounds.midX + drawingBodyOffset.dx,- y: bounds.midY + drawingBodyOffset.dy- )- context.rotate(by: drawingAngle(from: rotation - CursorGlyphCalibration.restingRotation))- context.scaleBy(- x: 1 - motionCompression - pulseCompression,- y: 1 + (pulseCompression * 0.4)- )- context.translateBy(x: -bounds.midX, y: -bounds.midY)- image.draw(in: bounds, from: .zero, operation: .sourceOver, fraction: 1)- context.restoreGState()- }-private func drawProceduralGlyph(in context: CGContext) {SoftwareCursorGlyphRenderer.draw(in: bounds,@@ -107,25 +80,3 @@ final class SynthesizedCursorGlyphView: NSView {-screenAngle}}--private func loadReferenceCursorWindowImage() -> NSImage? {- if let bundledReference = Bundle.main.url(- forResource: "official-software-cursor-window-252",- withExtension: "png"- ), let image = NSImage(contentsOf: bundledReference) {- return image- }-- let fileURL = URL(fileURLWithPath: #filePath).standardizedFileURL- let repoRoot = fileURL- .deletingLastPathComponent()- .deletingLastPathComponent()- .deletingLastPathComponent()- .deletingLastPathComponent()- .deletingLastPathComponent()-- let referenceURL = repoRoot- .appendingPathComponent("docs/references/codex-computer-use-reverse-engineering/assets/extracted-2026-04-19/official-software-cursor-window-252.png")-- return NSImage(contentsOf: referenceURL)-}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swiftindex ff36d99..a328089 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift@@ -2,6 +2,7 @@ import AppKitimport CoreGraphicsimport Foundationimport QuartzCore+import SoftwareCursorGlyphKitpublic enum VisualCursorSupport {public static var isEnabled: Bool {diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex a42733c..ba1cf29 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -1,4 +1,5 @@import AppKit+import SoftwareCursorGlyphKitimport XCTest@testable import OpenComputerUseKitdiff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift b/packages/SoftwareCursorGlyphKit/Sources/SoftwareCursorGlyphKit/SoftwareCursorGlyphRenderer.swiftsimilarity index 100%rename from packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swiftrename to packages/SoftwareCursorGlyphKit/Sources/SoftwareCursorGlyphKit/SoftwareCursorGlyphRenderer.swiftdiff --git a/scripts/build-cursor-motion-dmg.sh b/scripts/build-cursor-motion-dmg.shindex 1a7cb1d..8a068a1 100755--- a/scripts/build-cursor-motion-dmg.sh+++ b/scripts/build-cursor-motion-dmg.sh@@ -214,8 +214,8 @@ app_name="Cursor Motion.app"bundle_identifier="com.ifuryst.cursormotion"bundle_version="${CURSOR_MOTION_BUNDLE_VERSION:-${GITHUB_RUN_NUMBER:-$(git -C "${repo_root}" rev-list --count HEAD 2>/dev/null || echo 1)}}"bundle_icon_name="CursorMotion.icns"-icon_render_script="${repo_root}/scripts/render-open-computer-use-icon.swift"-cursor_reference_source="${repo_root}/docs/references/codex-computer-use-reverse-engineering/assets/extracted-2026-04-19/official-software-cursor-window-252.png"+icon_master_png="${repo_root}/assets/app-icons/open-computer-use-1024.png"+iconset_build_script="${repo_root}/scripts/build-apple-iconset.sh"app_root="${output_dir}/${app_name}"contents_dir="${app_root}/Contents"macos_dir="${contents_dir}/MacOS"@@ -248,15 +248,13 @@ esacchmod +x "${macos_dir}/CursorMotion"-if [[ ! -f "${cursor_reference_source}" ]]; then- echo "Missing cursor reference asset: ${cursor_reference_source}" >&2+if [[ ! -f "${icon_master_png}" ]]; then+ echo "Missing icon master PNG: ${icon_master_png}" >&2exit 1fi-cp "${cursor_reference_source}" "${resources_dir}/official-software-cursor-window-252.png"--if [[ ! -f "${icon_render_script}" ]]; then- echo "Missing icon render script: ${icon_render_script}" >&2+if [[ ! -f "${iconset_build_script}" ]]; then+ echo "Missing iconset build script: ${iconset_build_script}" >&2exit 1fi@@ -270,7 +268,7 @@ trap cleanup EXITicon_work_dir="$(mktemp -d "${TMPDIR:-/tmp}/cursor-motion-icon.XXXXXX")"iconset_dir="${icon_work_dir}/CursorMotion.iconset"mkdir -p "${iconset_dir}"-swift "${icon_render_script}" "${iconset_dir}"+"${iconset_build_script}" "${icon_master_png}" "${iconset_dir}"iconutil -c icns "${iconset_dir}" -o "${resources_dir}/${bundle_icon_name}"cat > "${contents_dir}/Info.plist" <<PLISTdiff --git a/scripts/computer-use-cli/README.md b/scripts/computer-use-cli/README.mdindex 08e6736..cc6b280 100644--- a/scripts/computer-use-cli/README.md+++ b/scripts/computer-use-cli/README.md@@ -64,6 +64,9 @@ This repo includes a ready-to-run example atThat sample covers all 9 official tools against `TextEdit`, and it intentionallyrepeats `get_app_state` between state-changing actions because the officialbundled `computer-use` invalidates the cached app state after each mutation.+Before running it, make sure `TextEdit` already has a normal document window open;+otherwise `get_app_state` may only expose the application root and the hard-coded+`element_index` values in the sample will no longer line up.For a smaller cursor-comparison path, use`examples/textedit-set-value-click-raise-seq.json`.