Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Fix StandaloneCursorLab cursor heading rendering

Leocea2f2854885
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 736ed17..689f893 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -84,7 +84,7 @@
- 这条线优先验证 Python 重建脚本已经收敛出来的核心:`20` 条候选路径、`measure + score`、`prefer in-bounds then lowest-score` 选路,以及 `response=1.4` / `dampingFraction=0.9` / `dt=1/240` 的 raw spring timeline。
- 当前它刻意不引入 speculative 的 wall-clock duration 映射,也不复用 `StandaloneCursorLab` 里更偏视觉手感试验的 pose dynamics。
- `StandaloneCursorLab` 是一个单独的 SwiftUI demo target,可通过 `swift run StandaloneCursorLab` 本地启动。
-- 这条线优先验证 motion model 本身:当前主线是 heading-driven 的 turn / brake / orbit / direct candidate 族、spring progress、独立 visual dynamics 和 debug UI;内部 heading 仍会参与选路,但真正画出来的箭头角度已经收紧到更接近默认 cursor 的静止姿态,不再像车头一样持续指向移动方向,只在转弯时给一点 lean,并在停住后做原地小摆角。
+- 这条线优先验证 motion model 本身:当前主线是 heading-driven 的 turn / brake / orbit / direct candidate 族、spring progress、独立 visual dynamics 和 debug UI;moving 阶段真正画出来的箭头角度会持续跟随 visual dynamics 的主 heading,接近停住后再平滑回到默认 resting pose,并在 idle 阶段保留原地小摆角。
- lab 的 cursor 视觉当前单独参考 `scripts/render-synthesized-software-cursor.swift`:优先使用仓库里保存的官方 `252x252` runtime baseline 图,缺失时再退回脚本里的 procedural pointer/fog 近似;settle 态也改成中心固定的小幅摆角,而不是继续沿 XY 轻微漂移。
- 当前它不接真实 tool call,也不回写主 `SoftwareCursorOverlay`,目的是把实验噪音与产品行为边界隔离开。
diff --git a/docs/exec-plans/active/20260418-standalone-cursor-lab.md b/docs/exec-plans/active/20260418-standalone-cursor-lab.md
index 9cda754..85655d4 100644
--- a/docs/exec-plans/active/20260418-standalone-cursor-lab.md
+++ b/docs/exec-plans/active/20260418-standalone-cursor-lab.md
@@ -66,11 +66,14 @@
- 2026-04-19:在对照官方视频后发现 guide/arc 相关常量不能直接按屏幕坐标向量使用;当前已改成先投到 start→end 的局部基底,再生成候选路径,默认样例和反向斜移都不再出现起点附近打结式的扭曲回环。
- 2026-04-19:继续对照官方视频后,确认 lab 主线不能直接拿 raw reverse-engineered `20` candidates 当 chooser;当前已改成 heading-driven 选路,把当前可见朝向和最终 resting pose 一起参与选路,主路径重新收敛到“需要掉头时走单侧 C 形,不需要掉头时近直线”的分布。
- 2026-04-20:在对照 `scripts/render-synthesized-software-cursor.swift` 与用户截图后,确认共享 glyph renderer 的亮白 asset 风格并不对;当前已把 lab 改成优先显示仓库里的官方 `252x252` runtime baseline 图,fallback 才走脚本同款 procedural pointer/fog,同时把 idle 从 XY 漂移收紧为中心固定的小幅摆角。
-- 2026-04-20:继续按用户反馈收紧姿态后,lab 现在把“内部 heading”与“可见箭头角度”分离;选路仍可参考当前 heading,但屏幕上看到的箭头不再像车头一样沿切线持续转向,只在转弯时轻微 lean。
+- 2026-04-20:曾按用户反馈把“内部 heading”与“可见箭头角度”分离,短暂把 moving 阶段的可见箭头收紧成轻微 lean;这层假设随后已在对照官方抽帧后撤回。
- 2026-04-20:按用户反馈把左上角 5 个 slider 恢复回来,并重新接到 heading-driven 路径几何与 progress spring;当前 slider 明确只作为本地调参入口,不宣称是已完全确认的官方字段映射。
- 2026-04-20:继续按用户反馈把左上角控件区裁成纯 slider,并把 slider label / panel accent 收回当前中性灰紫主色系;不再保留 `REPLAY` / `RESET` 按钮和额外 metrics 文案,避免信息噪音和低对比白字。
- 2026-04-20:修正窗口 resize 时的状态重置 bug;`proxy.size` 变化现在只更新 canvas bounds,不再重新走 `configure + snap(to: start)`,因此调整窗口大小不会把 cursor 强行拉回起始点。
- 2026-04-20:基于 `Codex Computer Use.app` 的新一轮 timing 逆向,确认默认 move 的 wall-clock endpoint-lock 是固定的 `343 / 240 = 1.4291667s`;lab 现已移除距离驱动的 travel-duration 压缩,默认档直接按官方 spring timeline 走,`SPRING` slider 只再改变 spring 本身的 response / damping 与对应时长。
+- 2026-04-20:对照用户提供的官方视频抽帧后,确认 moving 阶段的可见箭头确实会持续跟随当前 move heading,而不是只保留一个轻微 lean;lab 现已撤回那层 `displayRotation` 分离,glyph 渲染重新直接使用 visual dynamics 的主 `rotation`。
+- 2026-04-20:继续对照 `official-software-cursor-window-252.png` 与独立脚本的静止朝向后,确认 lab 之前额外加的 `-26.5°` glyph 补偿会让 moving heading 固定偏掉;当前已把静止基线收敛到和主运行时一致的“零旋转即左上朝向”,也就是 y-down 画布里的 `-3π/4`。
+- 2026-04-20:继续排查“屁股超前”后,确认真正的问题是 lab 的 motion/heading 运行在 SwiftUI 的 y-down 坐标,但 glyph 渲染落在 AppKit 默认 y-up `NSView`;当前已在 glyph 渲染层对 angle、body offset 和 fog offset 统一做 y-down -> y-up 转换,避免 moving 姿态被垂直镜像。
## 决策记录
diff --git a/docs/histories/2026-04/20260418-1430-standalone-cursor-lab.md b/docs/histories/2026-04/20260418-1430-standalone-cursor-lab.md
index 24de98d..818a0e3 100644
--- a/docs/histories/2026-04/20260418-1430-standalone-cursor-lab.md
+++ b/docs/histories/2026-04/20260418-1430-standalone-cursor-lab.md
@@ -108,6 +108,46 @@
- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift`
- `experiments/StandaloneCursorLab/README.md`
+### 🔁 Follow-up (2026-04-20, align glyph baseline heading with official cursor artwork)
+**Scope:** `experiments/StandaloneCursorLab/`、`docs/`
+
+**Key Actions:**
+- **[回收多余的局部旋转补偿]**: 对照独立 `render-synthesized-software-cursor.swift` 脚本和官方 `252x252` runtime baseline 图后,确认 lab 里额外的 `restingRotation = -26.5°` 会把 moving 期间的可见 heading 固定偏开。
+- **[静止朝向改为左上基线]**: `CursorGlyphCalibration` 现在直接把零旋转基线定义成官方箭头的天然静止朝向;在 lab 的 y-down 坐标里用 `neutralHeading = -3π/4`,`restingRotation = 0`,让 path heading 和 glyph 朝向共用同一套基准。
+- **[对齐主运行时语义]**: 这次调整也把 `StandaloneCursorLab` 的 heading 标定方式收回到和 `SoftwareCursorOverlay` 一致的思路,不再额外维护一层实验性局部偏角。
+
+### 📁 Additional Files Modified
+- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swift`
+- `docs/exec-plans/active/20260418-standalone-cursor-lab.md`
+
+### 🔁 Follow-up (2026-04-20, fix y-axis mismatch in glyph rendering)
+**Scope:** `experiments/StandaloneCursorLab/`、`docs/`
+
+**Key Actions:**
+- **[定位到渲染坐标系不一致]**: 继续对照用户反馈的“屁股超前”现象后,确认 `StandaloneCursorLab` 的 path / heading 在 SwiftUI y-down 坐标里推进,但 `SynthesizedCursorGlyphView` 实际使用的是 AppKit 默认 y-up 绘制坐标。
+- **[统一翻转 angle 与 offset]**: 在 glyph 渲染层新增显式转换,把 screen-space 的 `rotation`、`cursorBodyOffset`、`fogOffset` 统一映射到 AppKit drawing space,避免 moving 期间可见姿态被垂直镜像。
+- **[保持 motion model 不动]**: 这次没有继续改候选路径或 spring,只修复显示层坐标映射,让 heading-driven 选路和 visual dynamics 仍维持原来的 y-down 几何语义。
+
+### 📁 Additional Files Modified
+- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swift`
+- `docs/exec-plans/active/20260418-standalone-cursor-lab.md`
+
+### 🔁 Follow-up (2026-04-20, restore visible heading during moves)
+**Scope:** `experiments/StandaloneCursorLab/`、`docs/`
+
+**Key Actions:**
+- **[按官方抽帧撤回轻微 lean 假设]**: 对照用户提供的官方 1-9 帧后,确认 moving 阶段箭头主朝向会持续跟随当前 move heading,而不是只保留一个很小的 turn lean。
+- **[移除多余的 displayRotation 分层]**: `StandaloneCursorLab` 的 glyph 渲染重新直接使用 visual dynamics 主 `rotation`;之前额外加的 `displayRotation / visibleRotationOffset` 已移除,避免把主 heading 丢掉。
+- **[保留小幅 idle wobble]**: idle 阶段仍沿用 `_animatedAngleOffsetDegrees` 对应的小摆角近似,但这层 offset 重新回到主 heading 之上,而不是替代 moving 期间的可见朝向。
+- **[同步当前文档口径]**: README、架构说明和 active execution plan 统一改回“moving 阶段箭头跟随 heading,停住后再回 resting pose”的表述。
+
+### 📁 Additional Files Modified
+- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swift`
+- `experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift`
+- `experiments/StandaloneCursorLab/README.md`
+- `docs/ARCHITECTURE.md`
+- `docs/exec-plans/active/20260418-standalone-cursor-lab.md`
+
### 🔁 Follow-up (2026-04-20, restore slider tuning surface)
**Scope:** `experiments/StandaloneCursorLab/`、`docs/`
diff --git a/experiments/StandaloneCursorLab/README.md b/experiments/StandaloneCursorLab/README.md
index d86c04e..ccc1dad 100644
--- a/experiments/StandaloneCursorLab/README.md
+++ b/experiments/StandaloneCursorLab/README.md
@@ -46,7 +46,7 @@ swift run StandaloneCursorLab
- 默认档的 wall-clock move 时长现在直接对齐 reverse-engineered 官方 endpoint-lock 时间 `343 / 240 = 1.4291667s`;不再额外按路径距离压缩时长。
- 可见 cursor 不再直接贴在 path sample 上,而是经过独立 visual dynamics 状态,再输出 `rotation + cursorBodyOffset + fogOffset + fogScale`。
- 候选路径现在显式约束“先顺车头方向掉头,再沿主轴推进,再按 resting pose 收尾”;因此大多数跨向移动会呈现单侧 C 形,需要直接切入时才会退化为近直线,而不会再出现两侧乱甩的 S 形扭曲。
-- 箭头的可见角度不再像“车头”一样持续指向移动方向;当前会保留更接近默认 cursor 的静止姿态,只在转弯时给一点轻微 lean,并在停住后继续做原地小摆角。
+- 箭头的可见角度现在重新对齐官方抽帧与逆向证据:moving 阶段持续跟随当前 move heading,接近停住后再平滑回到默认 resting pose,并继续做原地小摆角。
- cursor glyph 不再走之前那套亮白 asset;当前优先直接显示仓库里的官方 `252x252` runtime baseline 图,缺失时才退回脚本同款 procedural pointer/fog。
- settle 态不再做 XY 漂移;现在改成和参考脚本一致的中心固定小摆角,让“停住以后原地轻微转动”的观感先对齐。
diff --git a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swift b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swift
index 9440e76..1efd31a 100644
--- a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swift
+++ b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorGlyphCalibration.swift
@@ -1,8 +1,10 @@
import CoreGraphics
enum CursorGlyphCalibration {
- static let neutralHeading = -CGFloat.pi / 2
- static let restingRotation = -26.5 * CGFloat.pi / 180
+ // The captured official baseline cursor artwork rests facing upper-left.
+ // In the lab's y-down canvas coordinates, that is -3π/4.
+ static let neutralHeading = -(3 * CGFloat.pi / 4)
+ static let restingRotation: CGFloat = 0
}
enum SynthesizedCursorOverlayMetrics {
diff --git a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift
index 5d9f514..03d3af9 100644
--- a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift
+++ b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorLabRootView.swift
@@ -115,7 +115,6 @@ final class CursorLabViewModel: ObservableObject {
@Published private(set) var currentState = CursorMotionState(
point: CGPoint(x: 220, y: 440),
rotation: CursorGlyphCalibration.restingRotation,
- displayRotation: CursorGlyphCalibration.restingRotation,
cursorBodyOffset: .zero,
fogOffset: .zero,
fogOpacity: CursorVisualDynamicsConfiguration.officialInspired.fogOpacityBase,
@@ -391,7 +390,7 @@ private struct CursorLabCanvas: View {
}
CursorGlyph(
- rotation: model.currentState.displayRotation,
+ rotation: model.currentState.rotation,
cursorBodyOffset: model.currentState.cursorBodyOffset,
fogOffset: model.currentState.fogOffset,
fogOpacity: model.currentState.fogOpacity,
diff --git a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swift b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swift
index 6afb1c6..d64bef4 100644
--- a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swift
+++ b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/CursorMotionModel.swift
@@ -1233,7 +1233,6 @@ struct CursorVisualDynamicsConfiguration: Equatable {
let angleSpring: CursorVisualSpringConfiguration
let headingVelocityFloor: CGFloat
let animatedAngleOffsetMax: CGFloat
- let visibleTurnLeanMax: CGFloat
let bodyOffsetScale: CGFloat
let bodyOffsetMax: CGFloat
let bodyLateralScale: CGFloat
@@ -1250,7 +1249,6 @@ struct CursorVisualDynamicsConfiguration: Equatable {
angleSpring: CursorVisualSpringConfiguration(response: 0.24, dampingFraction: 0.82),
headingVelocityFloor: 14,
animatedAngleOffsetMax: 0.28,
- visibleTurnLeanMax: 0.085,
bodyOffsetScale: 0.0012,
bodyOffsetMax: 2.4,
bodyLateralScale: 0.06,
@@ -1295,7 +1293,6 @@ struct CursorVisualDynamicsState: Equatable {
struct CursorVisualRenderState: Equatable {
let tipPosition: CGPoint
let rotation: CGFloat
- let visibleRotationOffset: CGFloat
let cursorBodyOffset: CGVector
let fogOffset: CGVector
let fogOpacity: CGFloat
@@ -1414,16 +1411,6 @@ enum CursorVisualDynamicsAnimator {
speed * configuration.fogScaleVelocityScale,
configuration.fogScaleMaxDelta
)
- let visibleRotationOffset: CGFloat
- if speed > configuration.headingVelocityFloor {
- visibleRotationOffset = CGFloat.clamped(
- (lateralAmount / max(configuration.bodyLateralMax, 0.001)) * configuration.visibleTurnLeanMax,
- lower: -configuration.visibleTurnLeanMax,
- upper: configuration.visibleTurnLeanMax
- )
- } else {
- visibleRotationOffset = idleAngleOffset
- }
return CursorVisualRenderState(
tipPosition: state.tipPosition,
@@ -1432,7 +1419,6 @@ enum CursorVisualDynamicsAnimator {
to: -configuration.animatedAngleOffsetMax...configuration.animatedAngleOffsetMax
)
),
- visibleRotationOffset: visibleRotationOffset,
cursorBodyOffset: cursorBodyOffset,
fogOffset: fogOffset,
fogOpacity: fogOpacity,
@@ -1470,7 +1456,6 @@ enum CursorVisualDynamicsAnimator {
struct CursorMotionState: Equatable {
let point: CGPoint
let rotation: CGFloat
- let displayRotation: CGFloat
let cursorBodyOffset: CGVector
let fogOffset: CGVector
let fogOpacity: CGFloat
@@ -1656,7 +1641,6 @@ final class CursorMotionSimulator {
CursorMotionState(
point: renderState.tipPosition,
rotation: CursorGlyphCalibration.restingRotation + renderState.rotation,
- displayRotation: CursorGlyphCalibration.restingRotation + renderState.visibleRotationOffset,
cursorBodyOffset: renderState.cursorBodyOffset,
fogOffset: renderState.fogOffset,
fogOpacity: renderState.fogOpacity,
diff --git a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swift b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swift
index 047d599..3c5b766 100644
--- a/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swift
+++ b/experiments/StandaloneCursorLab/Sources/StandaloneCursorLab/SynthesizedCursorGlyphView.swift
@@ -69,15 +69,16 @@ final class SynthesizedCursorGlyphView: NSView {
}
private func drawReferenceImage(_ image: NSImage, in context: CGContext) {
+ let drawingBodyOffset = drawingVector(from: cursorBodyOffset)
let motionCompression = min(hypot(cursorBodyOffset.dx, cursorBodyOffset.dy) * 0.008, 0.018)
let pulseCompression = clickProgress * 0.03
context.saveGState()
context.translateBy(
- x: bounds.midX + cursorBodyOffset.dx,
- y: bounds.midY + cursorBodyOffset.dy
+ x: bounds.midX + drawingBodyOffset.dx,
+ y: bounds.midY + drawingBodyOffset.dy
)
- context.rotate(by: rotation - CursorGlyphCalibration.restingRotation)
+ context.rotate(by: drawingAngle(from: rotation - CursorGlyphCalibration.restingRotation))
context.scaleBy(
x: 1 - motionCompression - pulseCompression,
y: 1 + (pulseCompression * 0.4)
@@ -89,13 +90,15 @@ final class SynthesizedCursorGlyphView: NSView {
private func drawProceduralGlyph(in context: CGContext) {
let pulse = clickProgress
+ let drawingFogOffset = drawingVector(from: fogOffset)
+ let drawingBodyOffset = drawingVector(from: cursorBodyOffset)
let fogCenter = CGPoint(
- x: bounds.midX + fogOffset.dx,
- y: bounds.midY + fogOffset.dy
+ x: bounds.midX + drawingFogOffset.dx,
+ y: bounds.midY + drawingFogOffset.dy
)
let pointerCenter = CGPoint(
- x: bounds.midX + 2.6 + cursorBodyOffset.dx,
- y: bounds.midY - 3.2 + cursorBodyOffset.dy + (pulse * 0.35)
+ x: bounds.midX + 2.6 + drawingBodyOffset.dx,
+ y: bounds.midY - 3.2 + drawingBodyOffset.dy + (pulse * 0.35)
)
drawFog(in: context, center: fogCenter, pulse: pulse)
@@ -162,10 +165,11 @@ final class SynthesizedCursorGlyphView: NSView {
let outerPath = pointerPath(in: pointerRect)
context.saveGState()
- context.translateBy(x: bounds.midX + cursorBodyOffset.dx, y: bounds.midY + cursorBodyOffset.dy)
- context.rotate(by: rotation - CursorGlyphCalibration.restingRotation)
+ let drawingBodyOffset = drawingVector(from: cursorBodyOffset)
+ context.translateBy(x: bounds.midX + drawingBodyOffset.dx, y: bounds.midY + drawingBodyOffset.dy)
+ context.rotate(by: drawingAngle(from: rotation - CursorGlyphCalibration.restingRotation))
context.scaleBy(x: 1 - (pulse * 0.04), y: 1 + (pulse * 0.02))
- context.translateBy(x: -(bounds.midX + cursorBodyOffset.dx), y: -(bounds.midY + cursorBodyOffset.dy))
+ context.translateBy(x: -(bounds.midX + drawingBodyOffset.dx), y: -(bounds.midY + drawingBodyOffset.dy))
NSGraphicsContext.saveGraphicsState()
let shadow = NSShadow()
@@ -221,6 +225,14 @@ final class SynthesizedCursorGlyphView: NSView {
path.lineJoinStyle = .round
return path
}
+
+ private func drawingVector(from screenVector: CGVector) -> CGVector {
+ CGVector(dx: screenVector.dx, dy: -screenVector.dy)
+ }
+
+ private func drawingAngle(from screenAngle: CGFloat) -> CGFloat {
+ -screenAngle
+ }
}
private func loadReferenceCursorWindowImage() -> NSImage? {