👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Fix StandaloneCursorLab cursor heading rendering
Leocea2f2854885
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 736ed17..689f893 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -84,7 +84,7 @@- 这条线优先验证 Python 重建脚本已经收敛出来的核心:`20` 条候选路径、`measure + score`、`prefer in-bounds then lowest-score` 选路,以及 `response=1.4` / `dampingFraction=0.9` / `dt=1/240` 的 raw spring timeline。- 当前它刻意不引入 speculative 的 wall-clock duration 映射,也不复用 `StandaloneCursorLab` 里更偏视觉手感试验的 pose dynamics。- `StandaloneCursorLab` 是一个单独的 SwiftUI demo target,可通过 `swift run StandaloneCursorLab` 本地启动。-- 这条线优先验证 motion model 本身:当前主线是 heading-driven 的 turn / brake / orbit / direct candidate 族、spring progress、独立 visual dynamics 和 debug UI;内部 heading 仍会参与选路,但真正画出来的箭头角度已经收紧到更接近默认 cursor 的静止姿态,不再像车头一样持续指向移动方向,只在转弯时给一点 lean,并在停住后做原地小摆角。+- 这条线优先验证 motion model 本身:当前主线是 heading-driven 的 turn / brake / orbit / direct candidate 族、spring progress、独立 visual dynamics 和 debug UI;moving 阶段真正画出来的箭头角度会持续跟随 visual dynamics 的主 heading,接近停住后再平滑回到默认 resting pose,并在 idle 阶段保留原地小摆角。- lab 的 cursor 视觉当前单独参考 `scripts/render-synthesized-software-cursor.swift`:优先使用仓库里保存的官方 `252x252` runtime baseline 图,缺失时再退回脚本里的 procedural pointer/fog 近似;settle 态也改成中心固定的小幅摆角,而不是继续沿 XY 轻微漂移。- 当前它不接真实 tool call,也不回写主 `SoftwareCursorOverlay`,目的是把实验噪音与产品行为边界隔离开。diff --git a/docs/exec-plans/active/20260418-standalone-cursor-lab.md b/docs/exec-plans/active/20260418-standalone-cursor-lab.mdindex 9cda754..85655d4 100644--- a/docs/exec-plans/active/20260418-standalone-cursor-lab.md+++ b/docs/exec-plans/active/20260418-standalone-cursor-lab.md@@ -66,11 +66,14 @@- 2026-04-19:在对照官方视频后发现 guide/arc 相关常量不能直接按屏幕坐标向量使用;当前已改成先投到 start→end 的局部基底,再生成候选路径,默认样例和反向斜移都不再出现起点附近打结式的扭曲回环。- 2026-04-19:继续对照官方视频后,确认 lab 主线不能直接拿 raw reverse-engineered `20` candidates 当 chooser;当前已改成 heading-driven 选路,把当前可见朝向和最终 resting pose 一起参与选路,主路径重新收敛到“需要掉头时走单侧 C 形,不需要掉头时近直线”的分布。- 2026-04-20:在对照 `scripts/render-synthesized-software-cursor.swift` 与用户截图后,确认共享 glyph renderer 的亮白 asset 风格并不对;当前已把 lab 改成优先显示仓库里的官方 `252x252` runtime baseline 图,fallback 才走脚本同款 procedural pointer/fog,同时把 idle 从 XY 漂移收紧为中心固定的小幅摆角。-- 2026-04-20:继续按用户反馈收紧姿态后,lab 现在把“内部 heading”与“可见箭头角度”分离;选路仍可参考当前 heading,但屏幕上看到的箭头不再像车头一样沿切线持续转向,只在转弯时轻微 lean。+- 2026-04-20:曾按用户反馈把“内部 heading”与“可见箭头角度”分离,短暂把 moving 阶段的可见箭头收紧成轻微 lean;这层假设随后已在对照官方抽帧后撤回。- 2026-04-20:按用户反馈把左上角 5 个 slider 恢复回来,并重新接到 heading-driven 路径几何与 progress spring;当前 slider 明确只作为本地调参入口,不宣称是已完全确认的官方字段映射。- 2026-04-20:继续按用户反馈把左上角控件区裁成纯 slider,并把 slider label / panel accent 收回当前中性灰紫主色系;不再保留 `REPLAY` / `RESET` 按钮和额外 metrics 文案,避免信息噪音和低对比白字。- 2026-04-20:修正窗口 resize 时的状态重置 bug;`proxy.size` 变化现在只更新 canvas bounds,不再重新走 `configure + snap(to: start)`,因此调整窗口大小不会把 cursor 强行拉回起始点。- 2026-04-20:基于 `Codex Computer Use.app` 的新一轮 timing 逆向,确认默认 move 的 wall-clock endpoint-lock 是固定的 `343 / 240 = 1.4291667s`;lab 现已移除距离驱动的 travel-duration 压缩,默认档直接按官方 spring timeline 走,`SPRING` slider 只再改变 spring 本身的 response / damping 与对应时长。+- 2026-04-20:对照用户提供的官方视频抽帧后,确认 moving 阶段的可见箭头确实会持续跟随当前 move heading,而不是只保留一个轻微 lean;lab 现已撤回那层 `displayRotation` 分离,glyph 渲染重新直接使用 visual dynamics 的主 `rotation`。+- 2026-04-20:继续对照 `official-software-cursor-window-252.png` 与独立脚本的静止朝向后,确认 lab 之前额外加的 `-26.5°` glyph 补偿会让 moving heading 固定偏掉;当前已把静止基线收敛到和主运行时一致的“零旋转即左上朝向”,也就是 y-down 画布里的 `-3π/4`。+- 2026-04-20:继续排查“屁股超前”后,确认真正的问题是 lab 的 motion/heading 运行在 SwiftUI 的 y-down 坐标,但 glyph 渲染落在 AppKit 默认 y-up `NSView`;当前已在 glyph 渲染层对 angle、body offset 和 fog offset 统一做 y-down -> y-up 转换,避免 moving 姿态被垂直镜像。## 决策记录diff --git a/docs/histories/2026-04/20260418-1430-standalone-cursor-lab.md b/docs/histories/2026-04/20260418-1430-standalone-cursor-lab.mdindex 24de98d..818a0e3 100644--- a/docs/histories/2026-04/20260418-1430-standalone-cursor-lab.md+++ b/docs/histories/2026-04/20260418-1430-standalone-cursor-lab.md@@ -108,6 +108,46 @@- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift`- `experiments/StandaloneCursorLab/README.md`+### 🔁 Follow-up (2026-04-20, align glyph baseline heading with official cursor artwork)+**Scope:** `experiments/StandaloneCursorLab/`、`docs/`++**Key Actions:**+- **[回收多余的局部旋转补偿]**: 对照独立 `render-synthesized-software-cursor.swift` 脚本和官方 `252x252` runtime baseline 图后,确认 lab 里额外的 `restingRotation = -26.5°` 会把 moving 期间的可见 heading 固定偏开。+- **[静止朝向改为左上基线]**: `CursorGlyphCalibration` 现在直接把零旋转基线定义成官方箭头的天然静止朝向;在 lab 的 y-down 坐标里用 `neutralHeading = -3π/4`,`restingRotation = 0`,让 path heading 和 glyph 朝向共用同一套基准。+- **[对齐主运行时语义]**: 这次调整也把 `StandaloneCursorLab` 的 heading 标定方式收回到和 `SoftwareCursorOverlay` 一致的思路,不再额外维护一层实验性局部偏角。++### 📁 Additional Files Modified+- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swift`+- `docs/exec-plans/active/20260418-standalone-cursor-lab.md`++### 🔁 Follow-up (2026-04-20, fix y-axis mismatch in glyph rendering)+**Scope:** `experiments/StandaloneCursorLab/`、`docs/`++**Key Actions:**+- **[定位到渲染坐标系不一致]**: 继续对照用户反馈的“屁股超前”现象后,确认 `StandaloneCursorLab` 的 path / heading 在 SwiftUI y-down 坐标里推进,但 `SynthesizedCursorGlyphView` 实际使用的是 AppKit 默认 y-up 绘制坐标。+- **[统一翻转 angle 与 offset]**: 在 glyph 渲染层新增显式转换,把 screen-space 的 `rotation`、`cursorBodyOffset`、`fogOffset` 统一映射到 AppKit drawing space,避免 moving 期间可见姿态被垂直镜像。+- **[保持 motion model 不动]**: 这次没有继续改候选路径或 spring,只修复显示层坐标映射,让 heading-driven 选路和 visual dynamics 仍维持原来的 y-down 几何语义。++### 📁 Additional Files Modified+- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swift`+- `docs/exec-plans/active/20260418-standalone-cursor-lab.md`++### 🔁 Follow-up (2026-04-20, restore visible heading during moves)+**Scope:** `experiments/StandaloneCursorLab/`、`docs/`++**Key Actions:**+- **[按官方抽帧撤回轻微 lean 假设]**: 对照用户提供的官方 1-9 帧后,确认 moving 阶段箭头主朝向会持续跟随当前 move heading,而不是只保留一个很小的 turn lean。+- **[移除多余的 displayRotation 分层]**: `StandaloneCursorLab` 的 glyph 渲染重新直接使用 visual dynamics 主 `rotation`;之前额外加的 `displayRotation / visibleRotationOffset` 已移除,避免把主 heading 丢掉。+- **[保留小幅 idle wobble]**: idle 阶段仍沿用 `_animatedAngleOffsetDegrees` 对应的小摆角近似,但这层 offset 重新回到主 heading 之上,而不是替代 moving 期间的可见朝向。+- **[同步当前文档口径]**: README、架构说明和 active execution plan 统一改回“moving 阶段箭头跟随 heading,停住后再回 resting pose”的表述。++### 📁 Additional Files Modified+- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swift`+- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift`+- `experiments/StandaloneCursorLab/README.md`+- `docs/ARCHITECTURE.md`+- `docs/exec-plans/active/20260418-standalone-cursor-lab.md`+### 🔁 Follow-up (2026-04-20, restore slider tuning surface)**Scope:** `experiments/StandaloneCursorLab/`、`docs/`diff --git a/experiments/StandaloneCursorLab/README.md b/experiments/StandaloneCursorLab/README.mdindex d86c04e..ccc1dad 100644--- a/experiments/StandaloneCursorLab/README.md+++ b/experiments/StandaloneCursorLab/README.md@@ -46,7 +46,7 @@ swift run StandaloneCursorLab- 默认档的 wall-clock move 时长现在直接对齐 reverse-engineered 官方 endpoint-lock 时间 `343 / 240 = 1.4291667s`;不再额外按路径距离压缩时长。- 可见 cursor 不再直接贴在 path sample 上,而是经过独立 visual dynamics 状态,再输出 `rotation + cursorBodyOffset + fogOffset + fogScale`。- 候选路径现在显式约束“先顺车头方向掉头,再沿主轴推进,再按 resting pose 收尾”;因此大多数跨向移动会呈现单侧 C 形,需要直接切入时才会退化为近直线,而不会再出现两侧乱甩的 S 形扭曲。-- 箭头的可见角度不再像“车头”一样持续指向移动方向;当前会保留更接近默认 cursor 的静止姿态,只在转弯时给一点轻微 lean,并在停住后继续做原地小摆角。+- 箭头的可见角度现在重新对齐官方抽帧与逆向证据:moving 阶段持续跟随当前 move heading,接近停住后再平滑回到默认 resting pose,并继续做原地小摆角。- cursor glyph 不再走之前那套亮白 asset;当前优先直接显示仓库里的官方 `252x252` runtime baseline 图,缺失时才退回脚本同款 procedural pointer/fog。- settle 态不再做 XY 漂移;现在改成和参考脚本一致的中心固定小摆角,让“停住以后原地轻微转动”的观感先对齐。diff --git a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swift b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swiftindex 9440e76..1efd31a 100644--- a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swift+++ b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swift@@ -1,8 +1,10 @@import CoreGraphicsenum CursorGlyphCalibration {- static let neutralHeading = -CGFloat.pi / 2- static let restingRotation = -26.5 * CGFloat.pi / 180+ // The captured official baseline cursor artwork rests facing upper-left.+ // In the lab's y-down canvas coordinates, that is -3π/4.+ static let neutralHeading = -(3 * CGFloat.pi / 4)+ static let restingRotation: CGFloat = 0}enum SynthesizedCursorOverlayMetrics {diff --git a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swiftindex 5d9f514..03d3af9 100644--- a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift+++ b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift@@ -115,7 +115,6 @@ final class CursorLabViewModel: ObservableObject {@Published private(set) var currentState = CursorMotionState(point: CGPoint(x: 220, y: 440),rotation: CursorGlyphCalibration.restingRotation,- displayRotation: CursorGlyphCalibration.restingRotation,cursorBodyOffset: .zero,fogOffset: .zero,fogOpacity: CursorVisualDynamicsConfiguration.officialInspired.fogOpacityBase,@@ -391,7 +390,7 @@ private struct CursorLabCanvas: View {}CursorGlyph(- rotation: model.currentState.displayRotation,+ rotation: model.currentState.rotation,cursorBodyOffset: model.currentState.cursorBodyOffset,fogOffset: model.currentState.fogOffset,fogOpacity: model.currentState.fogOpacity,diff --git a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swift b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swiftindex 6afb1c6..d64bef4 100644--- a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swift+++ b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swift@@ -1233,7 +1233,6 @@ struct CursorVisualDynamicsConfiguration: Equatable {let angleSpring: CursorVisualSpringConfigurationlet headingVelocityFloor: CGFloatlet animatedAngleOffsetMax: CGFloat- let visibleTurnLeanMax: CGFloatlet bodyOffsetScale: CGFloatlet bodyOffsetMax: CGFloatlet bodyLateralScale: CGFloat@@ -1250,7 +1249,6 @@ struct CursorVisualDynamicsConfiguration: Equatable {angleSpring: CursorVisualSpringConfiguration(response: 0.24, dampingFraction: 0.82),headingVelocityFloor: 14,animatedAngleOffsetMax: 0.28,- visibleTurnLeanMax: 0.085,bodyOffsetScale: 0.0012,bodyOffsetMax: 2.4,bodyLateralScale: 0.06,@@ -1295,7 +1293,6 @@ struct CursorVisualDynamicsState: Equatable {struct CursorVisualRenderState: Equatable {let tipPosition: CGPointlet rotation: CGFloat- let visibleRotationOffset: CGFloatlet cursorBodyOffset: CGVectorlet fogOffset: CGVectorlet fogOpacity: CGFloat@@ -1414,16 +1411,6 @@ enum CursorVisualDynamicsAnimator {speed * configuration.fogScaleVelocityScale,configuration.fogScaleMaxDelta)- let visibleRotationOffset: CGFloat- if speed > configuration.headingVelocityFloor {- visibleRotationOffset = CGFloat.clamped(- (lateralAmount / max(configuration.bodyLateralMax, 0.001)) * configuration.visibleTurnLeanMax,- lower: -configuration.visibleTurnLeanMax,- upper: configuration.visibleTurnLeanMax- )- } else {- visibleRotationOffset = idleAngleOffset- }return CursorVisualRenderState(tipPosition: state.tipPosition,@@ -1432,7 +1419,6 @@ enum CursorVisualDynamicsAnimator {to: -configuration.animatedAngleOffsetMax...configuration.animatedAngleOffsetMax)),- visibleRotationOffset: visibleRotationOffset,cursorBodyOffset: cursorBodyOffset,fogOffset: fogOffset,fogOpacity: fogOpacity,@@ -1470,7 +1456,6 @@ enum CursorVisualDynamicsAnimator {struct CursorMotionState: Equatable {let point: CGPointlet rotation: CGFloat- let displayRotation: CGFloatlet cursorBodyOffset: CGVectorlet fogOffset: CGVectorlet fogOpacity: CGFloat@@ -1656,7 +1641,6 @@ final class CursorMotionSimulator {CursorMotionState(point: renderState.tipPosition,rotation: CursorGlyphCalibration.restingRotation + renderState.rotation,- displayRotation: CursorGlyphCalibration.restingRotation + renderState.visibleRotationOffset,cursorBodyOffset: renderState.cursorBodyOffset,fogOffset: renderState.fogOffset,fogOpacity: renderState.fogOpacity,diff --git a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swift b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swiftindex 047d599..3c5b766 100644--- a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swift+++ b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swift@@ -69,15 +69,16 @@ final class SynthesizedCursorGlyphView: NSView {}private func drawReferenceImage(_ image: NSImage, in context: CGContext) {+ let drawingBodyOffset = drawingVector(from: cursorBodyOffset)let motionCompression = min(hypot(cursorBodyOffset.dx, cursorBodyOffset.dy) * 0.008, 0.018)let pulseCompression = clickProgress * 0.03context.saveGState()context.translateBy(- x: bounds.midX + cursorBodyOffset.dx,- y: bounds.midY + cursorBodyOffset.dy+ x: bounds.midX + drawingBodyOffset.dx,+ y: bounds.midY + drawingBodyOffset.dy)- context.rotate(by: rotation - CursorGlyphCalibration.restingRotation)+ context.rotate(by: drawingAngle(from: rotation - CursorGlyphCalibration.restingRotation))context.scaleBy(x: 1 - motionCompression - pulseCompression,y: 1 + (pulseCompression * 0.4)@@ -89,13 +90,15 @@ final class SynthesizedCursorGlyphView: NSView {private func drawProceduralGlyph(in context: CGContext) {let pulse = clickProgress+ let drawingFogOffset = drawingVector(from: fogOffset)+ let drawingBodyOffset = drawingVector(from: cursorBodyOffset)let fogCenter = CGPoint(- x: bounds.midX + fogOffset.dx,- y: bounds.midY + fogOffset.dy+ x: bounds.midX + drawingFogOffset.dx,+ y: bounds.midY + drawingFogOffset.dy)let pointerCenter = CGPoint(- x: bounds.midX + 2.6 + cursorBodyOffset.dx,- y: bounds.midY - 3.2 + cursorBodyOffset.dy + (pulse * 0.35)+ x: bounds.midX + 2.6 + drawingBodyOffset.dx,+ y: bounds.midY - 3.2 + drawingBodyOffset.dy + (pulse * 0.35))drawFog(in: context, center: fogCenter, pulse: pulse)@@ -162,10 +165,11 @@ final class SynthesizedCursorGlyphView: NSView {let outerPath = pointerPath(in: pointerRect)context.saveGState()- context.translateBy(x: bounds.midX + cursorBodyOffset.dx, y: bounds.midY + cursorBodyOffset.dy)- context.rotate(by: rotation - CursorGlyphCalibration.restingRotation)+ let drawingBodyOffset = drawingVector(from: cursorBodyOffset)+ context.translateBy(x: bounds.midX + drawingBodyOffset.dx, y: bounds.midY + drawingBodyOffset.dy)+ context.rotate(by: drawingAngle(from: rotation - CursorGlyphCalibration.restingRotation))context.scaleBy(x: 1 - (pulse * 0.04), y: 1 + (pulse * 0.02))- context.translateBy(x: -(bounds.midX + cursorBodyOffset.dx), y: -(bounds.midY + cursorBodyOffset.dy))+ context.translateBy(x: -(bounds.midX + drawingBodyOffset.dx), y: -(bounds.midY + drawingBodyOffset.dy))NSGraphicsContext.saveGraphicsState()let shadow = NSShadow()@@ -221,6 +225,14 @@ final class SynthesizedCursorGlyphView: NSView {path.lineJoinStyle = .roundreturn path}++ private func drawingVector(from screenVector: CGVector) -> CGVector {+ CGVector(dx: screenVector.dx, dy: -screenVector.dy)+ }++ private func drawingAngle(from screenAngle: CGFloat) -> CGFloat {+ -screenAngle+ }}private func loadReferenceCursorWindowImage() -> NSImage? {