👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Align runtime pointer neutral axis
Leod9b7982ea7a5
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 6fb20d7..88ff94e 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -58,7 +58,7 @@- 对真实 app 的 `get_app_state` / action tool 入口,当前新增了一层官方风格的高风险 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app,尽量贴近官方对终端、密码管理器、Chrome 与少量系统敏感组件的防护行为。- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse、idle sway 和自动淡出,`set_value` 则只做 settle / idle / hide,不给 pulse。-- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在在 `OpenComputerUseKit` 内部复刻 `CursorMotion` procedural fallback 的 pointer/fog glyph 校准,命中点 anchor 固定在 `126x126` 画布里的同一组 tip-offset 上。runtime motion 状态继续按截图 / CGWindow 的 y-down 坐标计算,进入 AppKit 绘制前才做角度和 `dy` 翻转,保持方向语义与 `CursorMotion` 一致,但不让实验线依赖 runtime 代码。+- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在在 `OpenComputerUseKit` 内部复刻 `CursorMotion` procedural fallback 的 pointer/fog glyph 校准,命中点 anchor 固定在 `126x126` 画布里的同一组 tip-offset 上。runtime motion 状态继续按截图 / CGWindow 的 y-down 坐标计算,进入 AppKit 绘制前才做角度和 `dy` 翻转;程序化 pointer 轮廓还会先做一层 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。- overlay 的层级不再固定 `.floating`;现在会跟随 snapshot 命中的目标 window id / layer,把自己排到该目标 window 之上,而不是粗暴压到所有前台 app 最上层。- overlay 的曲线路径不再只按固定 Bezier 模板生成;当前主线采用 reverse-engineering 约束下的 heading-driven candidate 族,候选只保留 `direct` / `turn` / `brake` / `orbit` 这些能稳定产出单侧主弧的 family,并继续保留 target-window 命中策略作为同类候选间的 tie-break。原始 binary lift 恢复出来的 `20` 条路径和 score 仍然保留在独立的 `StandaloneCursor` viewer / Python 重建脚本里,用于对照分析,不再直接作为 runtime 默认 chooser。- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。diff --git a/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md b/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.mdindex f781bf9..fc8d3e8 100644--- a/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md+++ b/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md@@ -109,3 +109,16 @@- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`- `docs/ARCHITECTURE.md`- `docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md`++### 🔁 Follow-up (2026-04-21, align procedural pointer neutral axis)++- **[修正侧边朝前]**: 用户继续反馈 runtime cursor 仍然不是 tip 朝前,而是侧边在前;复查后确认 motion rotation 语义已经对齐 `CursorMotion`,偏差来自 runtime 程序化 pointer contour 自身的 neutral 轴不等于官方 baseline 的 `-3π/4`。+- **[加 artwork neutral correction]**: `SoftwareCursorGlyphRenderer` 在 pointer 自身中心上额外应用固定 artwork correction,把 procedural contour 的天然轴校准到 `CursorMotion` / 官方 baseline 的 forward 方向;motion 层的 heading / path 逻辑保持不变。+- **[保持实验线不动]**: 本次仍只改 `OpenComputerUseKit` runtime,没有修改 `experiments/CursorMotion`。+- **[补单测与文档]**: 新增 corrected neutral heading 单测,并同步架构文档,明确这是 runtime 内部的程序化轮廓校准。++**Follow-up Files:**+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swiftindex 9cf1f02..da38a27 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorGlyphRenderer.swift@@ -44,6 +44,9 @@ enum SoftwareCursorGlyphMetrics {static let pointerSize = CGSize(width: 21, height: 21)static let pointerOffset = CGPoint(x: 2.6, y: -3.2)+ static let targetNeutralHeading = -(3 * CGFloat.pi / 4)+ static let proceduralContourNeutralHeading = -(96.5 * CGFloat.pi / 180)+ static let pointerArtworkRotation = -(targetNeutralHeading - proceduralContourNeutralHeading)}private enum SoftwareCursorGlyphColors {@@ -168,6 +171,9 @@ enum SoftwareCursorGlyphRenderer {x: -(boundsMidpoint.x + cursorBodyOffset.dx),y: -(boundsMidpoint.y + cursorBodyOffset.dy))+ context.translateBy(x: center.x, y: center.y)+ context.rotate(by: SoftwareCursorGlyphMetrics.pointerArtworkRotation)+ context.translateBy(x: -center.x, y: -center.y)NSGraphicsContext.saveGraphicsState()let shadow = NSShadow()diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex d1a1088..6f990a2 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -320,6 +320,18 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(SoftwareCursorGlyphMetrics.tipAnchor.y, 70.3, accuracy: 0.01)}+ func testSoftwareCursorGlyphArtworkNeutralHeadingMatchesCursorMotionBaseline() {+ let correctedNeutralHeading = SoftwareCursorGlyphMetrics.proceduralContourNeutralHeading+ - SoftwareCursorGlyphMetrics.pointerArtworkRotation++ XCTAssertEqual(+ correctedNeutralHeading,+ SoftwareCursorGlyphMetrics.targetNeutralHeading,+ accuracy: 0.001+ )+ XCTAssertEqual(SoftwareCursorGlyphMetrics.targetNeutralHeading, -(3 * CGFloat.pi / 4), accuracy: 0.001)+ }+func testSoftwareCursorGlyphConvertsScreenStateToAppKitDrawingState() {let screenState = SoftwareCursorGlyphRenderState(rotation: .pi / 3,