Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

fix: keep visual cursor above target window

usoon.xu0fdee32fa35b
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 91f2525..cae2b49 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -110,7 +110,7 @@
- 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。
- 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints,普通 stdio MCP client 在本机上可能被系统直接杀掉;如果要探测官方 bundled `computer-use`,`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪,外部 raw helper 即使走已签名 Codex binary,也可能返回 `Sender process is not authenticated`;需要真实使用官方工具时应走正常 Codex agent/tool 调用链,开源版则继续提供可直连的 `open-computer-use` MCP server。
-- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。
+- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,并且在 overlay 可见期间会持续重申“排在目标 window 之上”,避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住;但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。
- screenshot 当前使用系统窗口截图 API,但默认直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录。
- 会话状态现在是进程内内存态,保存每个 app 最近一次 snapshot 和 element index 映射。
diff --git a/docs/histories/2026-04/20260422-1649-fix-visual-cursor-target-z-order.md b/docs/histories/2026-04/20260422-1649-fix-visual-cursor-target-z-order.md
new file mode 100644
index 0000000..9f5fb8e
--- /dev/null
+++ b/docs/histories/2026-04/20260422-1649-fix-visual-cursor-target-z-order.md
@@ -0,0 +1,26 @@
+## [2026-04-22 16:49] | Task: Fix visual cursor target z-order
+
+### Execution Context
+* **Agent ID**: `Codex`
+* **Base Model**: `GPT-5.4`
+* **Runtime**: `Codex CLI on macOS + SwiftPM`
+
+### User Query
+> the virtual cursor current z index is above the target app, but under the current active app, but if I invoke the target app manually, the virtual cursor will behind the target app, fix this, make the virtual cursor always on top of target app
+
+### Changes Overview
+**Scope:** `OpenComputerUseKit` visual cursor overlay ordering、测试、架构文档
+
+**Key Actions:**
+- **[Ordering refresh]**: 调整 `SoftwareCursorOverlay` 的排序逻辑,让 overlay 在可见期间持续重申自己应排在目标 window 之上,而不是只在目标窗口变化时排序一次。
+- **[Regression coverage]**: 新增针对“强制重排”和“稳定窗口不重复重排”两类判断的单测,避免后续又退回成一次性排序。
+- **[Docs sync]**: 更新 `docs/ARCHITECTURE.md`,把“用户手动激活目标 app 后仍保持压在目标 window 上面”的行为写回架构说明。
+
+### Design Intent
+这次不是调整 cursor 的视觉效果,而是修复 overlay 和目标窗口之间的持久层级关系。问题根因是当前实现只在 `activeTargetWindow` 变化时调用一次 `order(.above, relativeTo:)`;当用户随后手动激活目标 app,系统会重排该 app 的窗口顺序,但 overlay 没有重新声明自己的排序位置,结果就会掉到目标窗口后面。修复后,overlay 只要还可见,就会持续重申相对目标窗口的排序,从而保持“始终盖在目标 app 之上”。
+
+### Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
+- `docs/histories/2026-04/20260422-1649-fix-visual-cursor-target-z-order.md`
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
index a7765ce..42b4bd3 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SoftwareCursorOverlay.swift
@@ -247,6 +247,10 @@ enum SoftwareCursorOverlay {
}
private static func configureOrdering(relativeTo targetWindow: CursorTargetWindow?) {
+ configureOrdering(relativeTo: targetWindow, forceReorder: false)
+ }
+
+ private static func configureOrdering(relativeTo targetWindow: CursorTargetWindow?, forceReorder: Bool) {
guard let panel else {
return
}
@@ -260,7 +264,12 @@ enum SoftwareCursorOverlay {
panel.level = desiredLevel
}
- if activeTargetWindow != effectiveTargetWindow || panel.isVisible == false {
+ if shouldReorderCursorPanel(
+ activeTargetWindow: activeTargetWindow,
+ effectiveTargetWindow: effectiveTargetWindow,
+ panelIsVisible: panel.isVisible,
+ forceReorder: forceReorder
+ ) {
if let effectiveTargetWindow {
panel.order(.above, relativeTo: Int(effectiveTargetWindow.windowID))
} else {
@@ -457,7 +466,12 @@ enum SoftwareCursorOverlay {
}
private static func refreshActiveOrderingIfNeeded() {
- guard let activeTargetWindow, !isWindowPresent(activeTargetWindow.windowID) else {
+ guard let activeTargetWindow else {
+ return
+ }
+
+ if isWindowPresent(activeTargetWindow.windowID) {
+ configureOrdering(relativeTo: activeTargetWindow, forceReorder: true)
return
}
@@ -696,6 +710,15 @@ enum SoftwareCursorOverlay {
}
}
+func shouldReorderCursorPanel(
+ activeTargetWindow: CursorTargetWindow?,
+ effectiveTargetWindow: CursorTargetWindow?,
+ panelIsVisible: Bool,
+ forceReorder: Bool
+) -> Bool {
+ forceReorder || activeTargetWindow != effectiveTargetWindow || panelIsVisible == false
+}
+
private final class CursorPanel: NSPanel {
override var canBecomeKey: Bool { false }
override var canBecomeMain: Bool { false }
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index e35ae0d..51a42c0 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -636,6 +636,32 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertGreaterThanOrEqual(visualCursorPostInteractionIdleTimeout(), 30)
}
+ func testCursorPanelReordersWhenForcedEvenIfTargetWindowDidNotChange() {
+ let targetWindow = CursorTargetWindow(windowID: 42, layer: 0)
+
+ XCTAssertTrue(
+ shouldReorderCursorPanel(
+ activeTargetWindow: targetWindow,
+ effectiveTargetWindow: targetWindow,
+ panelIsVisible: true,
+ forceReorder: true
+ )
+ )
+ }
+
+ func testCursorPanelDoesNotReorderWhenVisibleAndTargetWindowIsStable() {
+ let targetWindow = CursorTargetWindow(windowID: 42, layer: 0)
+
+ XCTAssertFalse(
+ shouldReorderCursorPanel(
+ activeTargetWindow: targetWindow,
+ effectiveTargetWindow: targetWindow,
+ panelIsVisible: true,
+ forceReorder: false
+ )
+ )
+ }
+
func testVisualCursorRuntimeMapsAppKitUpwardMotionToCursorMotionScreenState() {
let renderBaseHeading = visualCursorRenderBaseHeading(
artworkNeutralHeading: SoftwareCursorGlyphMetrics.targetNeutralHeading