Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Recover hidden windows before snapshot

Leo5f50d9f789d0
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index bfcb994..05d92ac 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -60,7 +60,7 @@
- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。
- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。
-- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`,否则返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。
+- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。
- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。
- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。
- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。
diff --git a/docs/histories/2026-05/20260507-1958-recover-hidden-electron-window.md b/docs/histories/2026-05/20260507-1958-recover-hidden-electron-window.md
new file mode 100644
index 0000000..3c37b31
--- /dev/null
+++ b/docs/histories/2026-05/20260507-1958-recover-hidden-electron-window.md
@@ -0,0 +1,24 @@
+# 恢复隐藏的 Electron 窗口后再采集状态
+
+## 用户诉求
+
+继续对比开源版 `open-computer-use` 和官方 `computer-use` 在 Lark / Electron app 上的工具返回,没达到官方行为的地方继续优化。
+
+## 主要改动
+
+- 在 `get_app_state` 找到 AX window 但找不到 on-screen CGWindow,或暂时找不到可用 focused window 时,增加 best-effort 窗口恢复流程。
+- 恢复流程会尝试 unhide、activate、`open -b <bundle-id>`、取消最小化、`AXRaise`、设置 main/focused,然后短暂等待并重试窗口匹配。
+- 保留恢复失败后的官方形态错误文本:`Apple event error -10005: cgWindowNotFound`。
+- 更新架构文档,说明隐藏 / 不可见 Electron 窗口会先恢复再采集。
+
+## 验证
+
+- 对比观察:开源版第一次请求隐藏的 Feishu 时返回 `cgWindowNotFound`,官方 `computer-use` 会拉起窗口并返回完整状态。
+- 本地回归:隐藏 Feishu 后,使用新构建的 Dev app 直接调用 `get_app_state`,返回 `isError=false`,且包含截图内容块。
+- `swift test --filter NoWindowErrorMessageMatchesOfficialShape`
+- `./scripts/build-open-computer-use-app.sh debug`
+
+## 受影响文件
+
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+- `docs/ARCHITECTURE.md`
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index 0e5f41b..1c9e67e 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -30,6 +30,7 @@ enum SnapshotMode {
let accessibilityTreeMaxNodeCount = 1200
let accessibilityTreeMaxDepth = 64
let screenshotCaptureTimeout: TimeInterval = 5
+private let windowVisibilityRecoveryDelay: TimeInterval = 0.7
private let axWebAreaRole = "AXWebArea"
public struct AppSnapshot {
@@ -90,17 +91,54 @@ enum SnapshotBuilder {
let appElement = AXUIElementCreateApplication(app.pid)
enableBestEffortAccessibilityModes(appElement)
let systemWide = AXUIElementCreateSystemWide()
- let focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)
- guard let focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide) else {
+ var focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)
+ var focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)
+ if focusedWindow == nil, recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: nil) {
+ focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)
+ focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)
+ }
+
+ var rootWindow: AXUIElement
+ guard let resolvedFocusedWindow = focusedWindow else {
throw ComputerUseError.stateUnavailable(computerUseNoWindowFoundMessage)
}
+ rootWindow = resolvedFocusedWindow
+
+ var windowTitle = stringValue(of: rootWindow, attribute: kAXTitleAttribute)
+ var windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)
+ if windowCapture == nil, recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: rootWindow) {
+ focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)
+ if let recoveredWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide) {
+ rootWindow = recoveredWindow
+ windowTitle = stringValue(of: recoveredWindow, attribute: kAXTitleAttribute)
+ windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)
+ }
+ }
- let windowTitle = stringValue(of: focusedWindow, attribute: kAXTitleAttribute)
- guard let windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle) else {
+ guard let windowCapture else {
throw ComputerUseError.stateUnavailable(computerUseNoWindowFoundMessage)
}
- let rootElement = focusedWindow
+ return buildAccessibilitySnapshot(
+ app: app,
+ appElement: appElement,
+ rootElement: rootWindow,
+ windowTitle: windowTitle,
+ windowCapture: windowCapture,
+ focusedApplication: focusedApplication,
+ systemWide: systemWide
+ )
+ }
+
+ private static func buildAccessibilitySnapshot(
+ app: RunningAppDescriptor,
+ appElement: AXUIElement,
+ rootElement: AXUIElement,
+ windowTitle: String?,
+ windowCapture: WindowCapture,
+ focusedApplication: AXUIElement?,
+ systemWide: AXUIElement
+ ) -> AppSnapshot {
let windowBounds = windowCapture.bounds
let screenshotPNGData = windowCapture.pngDataIfAvailable()
let focusedElement = preferredFocusedElement(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)
@@ -130,6 +168,48 @@ enum SnapshotBuilder {
)
}
+ private static func recoverVisibleWindow(for app: RunningAppDescriptor, appElement: AXUIElement, preferredWindow: AXUIElement?) -> Bool {
+ var recovered = false
+
+ if let runningApplication = NSRunningApplication(processIdentifier: app.pid) {
+ recovered = runningApplication.unhide() || recovered
+ recovered = runningApplication.activate(options: [.activateAllWindows]) || recovered
+ }
+
+ if let bundleIdentifier = app.bundleIdentifier {
+ recovered = openBundleIdentifier(bundleIdentifier) || recovered
+ }
+
+ if let window = preferredWindow ?? firstAnyWindow(for: appElement) {
+ recovered = unminimize(window) || recovered
+ recovered = raise(window) || recovered
+ recovered = setBoolAttribute(named: kAXMainAttribute as String, on: window) || recovered
+ recovered = setBoolAttribute(named: kAXFocusedAttribute as String, on: window) || recovered
+ }
+
+ if recovered {
+ Thread.sleep(forTimeInterval: windowVisibilityRecoveryDelay)
+ }
+
+ return recovered
+ }
+
+ private static func openBundleIdentifier(_ bundleIdentifier: String) -> Bool {
+ let process = Process()
+ process.executableURL = URL(fileURLWithPath: "/usr/bin/open")
+ process.arguments = ["-b", bundleIdentifier]
+ process.standardOutput = Pipe()
+ process.standardError = Pipe()
+
+ do {
+ try process.run()
+ process.waitUntilExit()
+ return process.terminationStatus == 0
+ } catch {
+ return false
+ }
+ }
+
private static func firstWindow(for appElement: AXUIElement) -> AXUIElement? {
guard let windows = copyArray(appElement, attribute: kAXWindowsAttribute) else {
return nil
@@ -138,6 +218,31 @@ enum SnapshotBuilder {
return windows.first(where: isUsableWindowElement(_:))
}
+ private static func firstAnyWindow(for appElement: AXUIElement) -> AXUIElement? {
+ copyElement(appElement, attribute: kAXFocusedWindowAttribute)
+ ?? copyArray(appElement, attribute: kAXWindowsAttribute)?.first(where: { stringValue(of: $0, attribute: kAXRoleAttribute) == kAXWindowRole as String })
+ }
+
+ private static func unminimize(_ window: AXUIElement) -> Bool {
+ guard boolValue(of: window, attribute: kAXMinimizedAttribute) == true else {
+ return false
+ }
+
+ return AXUIElementSetAttributeValue(window, kAXMinimizedAttribute as CFString, kCFBooleanFalse) == .success
+ }
+
+ private static func raise(_ window: AXUIElement) -> Bool {
+ guard copyActions(window)?.contains(where: { $0.caseInsensitiveCompare(kAXRaiseAction as String) == .orderedSame }) == true else {
+ return false
+ }
+
+ return AXUIElementPerformAction(window, kAXRaiseAction as CFString) == .success
+ }
+
+ private static func setBoolAttribute(named attribute: String, on element: AXUIElement) -> Bool {
+ AXUIElementSetAttributeValue(element, attribute as CFString, kCFBooleanTrue) == .success
+ }
+
private static func preferredFocusedWindow(appElement: AXUIElement, appPID: pid_t, focusedApplication: AXUIElement?, systemWide: AXUIElement) -> AXUIElement? {
if let focusedApplication, pid(of: focusedApplication) == appPID {
return usableWindowElement(from: copyElement(systemWide, attribute: kAXFocusedWindowAttribute))