👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
fix: keep visual cursor above target window
usoon.xu0fdee32fa35b
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 91f2525..cae2b49 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -110,7 +110,7 @@- 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。- 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints,普通 stdio MCP client 在本机上可能被系统直接杀掉;如果要探测官方 bundled `computer-use`,`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪,外部 raw helper 即使走已签名 Codex binary,也可能返回 `Sender process is not authenticated`;需要真实使用官方工具时应走正常 Codex agent/tool 调用链,开源版则继续提供可直连的 `open-computer-use` MCP server。-- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。+- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,并且在 overlay 可见期间会持续重申“排在目标 window 之上”,避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住;但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。- screenshot 当前使用系统窗口截图 API,但默认直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录。- 会话状态现在是进程内内存态,保存每个 app 最近一次 snapshot 和 element index 映射。diff --git a/docs/histories/2026-04/20260422-1649-fix-visual-cursor-target-z-order.md b/docs/histories/2026-04/20260422-1649-fix-visual-cursor-target-z-order.mdnew file mode 100644index 0000000..9f5fb8e--- /dev/null+++ b/docs/histories/2026-04/20260422-1649-fix-visual-cursor-target-z-order.md@@ -0,0 +1,26 @@+## [2026-04-22 16:49] | Task: Fix visual cursor target z-order++### Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5.4`+* **Runtime**: `Codex CLI on macOS + SwiftPM`++### User Query+> the virtual cursor current z index is above the target app, but under the current active app, but if I invoke the target app manually, the virtual cursor will behind the target app, fix this, make the virtual cursor always on top of target app++### Changes Overview+**Scope:** `OpenComputerUseKit` visual cursor overlay ordering、测试、架构文档++**Key Actions:**+- **[Ordering refresh]**: 调整 `SoftwareCursorOverlay` 的排序逻辑,让 overlay 在可见期间持续重申自己应排在目标 window 之上,而不是只在目标窗口变化时排序一次。+- **[Regression coverage]**: 新增针对“强制重排”和“稳定窗口不重复重排”两类判断的单测,避免后续又退回成一次性排序。+- **[Docs sync]**: 更新 `docs/ARCHITECTURE.md`,把“用户手动激活目标 app 后仍保持压在目标 window 上面”的行为写回架构说明。++### Design Intent+这次不是调整 cursor 的视觉效果,而是修复 overlay 和目标窗口之间的持久层级关系。问题根因是当前实现只在 `activeTargetWindow` 变化时调用一次 `order(.above, relativeTo:)`;当用户随后手动激活目标 app,系统会重排该 app 的窗口顺序,但 overlay 没有重新声明自己的排序位置,结果就会掉到目标窗口后面。修复后,overlay 只要还可见,就会持续重申相对目标窗口的排序,从而保持“始终盖在目标 app 之上”。++### Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-04/20260422-1649-fix-visual-cursor-target-z-order.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swiftindex a7765ce..42b4bd3 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift@@ -247,6 +247,10 @@ enum SoftwareCursorOverlay {}private static func configureOrdering(relativeTo targetWindow: CursorTargetWindow?) {+ configureOrdering(relativeTo: targetWindow, forceReorder: false)+ }++ private static func configureOrdering(relativeTo targetWindow: CursorTargetWindow?, forceReorder: Bool) {guard let panel else {return}@@ -260,7 +264,12 @@ enum SoftwareCursorOverlay {panel.level = desiredLevel}- if activeTargetWindow != effectiveTargetWindow || panel.isVisible == false {+ if shouldReorderCursorPanel(+ activeTargetWindow: activeTargetWindow,+ effectiveTargetWindow: effectiveTargetWindow,+ panelIsVisible: panel.isVisible,+ forceReorder: forceReorder+ ) {if let effectiveTargetWindow {panel.order(.above, relativeTo: Int(effectiveTargetWindow.windowID))} else {@@ -457,7 +466,12 @@ enum SoftwareCursorOverlay {}private static func refreshActiveOrderingIfNeeded() {- guard let activeTargetWindow, !isWindowPresent(activeTargetWindow.windowID) else {+ guard let activeTargetWindow else {+ return+ }++ if isWindowPresent(activeTargetWindow.windowID) {+ configureOrdering(relativeTo: activeTargetWindow, forceReorder: true)return}@@ -696,6 +710,15 @@ enum SoftwareCursorOverlay {}}+func shouldReorderCursorPanel(+ activeTargetWindow: CursorTargetWindow?,+ effectiveTargetWindow: CursorTargetWindow?,+ panelIsVisible: Bool,+ forceReorder: Bool+) -> Bool {+ forceReorder || activeTargetWindow != effectiveTargetWindow || panelIsVisible == false+}+private final class CursorPanel: NSPanel {override var canBecomeKey: Bool { false }override var canBecomeMain: Bool { false }diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex e35ae0d..51a42c0 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -636,6 +636,32 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertGreaterThanOrEqual(visualCursorPostInteractionIdleTimeout(), 30)}+ func testCursorPanelReordersWhenForcedEvenIfTargetWindowDidNotChange() {+ let targetWindow = CursorTargetWindow(windowID: 42, layer: 0)++ XCTAssertTrue(+ shouldReorderCursorPanel(+ activeTargetWindow: targetWindow,+ effectiveTargetWindow: targetWindow,+ panelIsVisible: true,+ forceReorder: true+ )+ )+ }++ func testCursorPanelDoesNotReorderWhenVisibleAndTargetWindowIsStable() {+ let targetWindow = CursorTargetWindow(windowID: 42, layer: 0)++ XCTAssertFalse(+ shouldReorderCursorPanel(+ activeTargetWindow: targetWindow,+ effectiveTargetWindow: targetWindow,+ panelIsVisible: true,+ forceReorder: false+ )+ )+ }+func testVisualCursorRuntimeMapsAppKitUpwardMotionToCursorMotionScreenState() {let renderBaseHeading = visualCursorRenderBaseHeading(artworkNeutralHeading: SoftwareCursorGlyphMetrics.targetNeutralHeading