👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Recover hidden windows before snapshot
Leo5f50d9f789d0
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex bfcb994..05d92ac 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -60,7 +60,7 @@- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。-- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`,否则返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。+- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。diff --git a/docs/histories/2026-05/20260507-1958-recover-hidden-electron-window.md b/docs/histories/2026-05/20260507-1958-recover-hidden-electron-window.mdnew file mode 100644index 0000000..3c37b31--- /dev/null+++ b/docs/histories/2026-05/20260507-1958-recover-hidden-electron-window.md@@ -0,0 +1,24 @@+# 恢复隐藏的 Electron 窗口后再采集状态++## 用户诉求++继续对比开源版 `open-computer-use` 和官方 `computer-use` 在 Lark / Electron app 上的工具返回,没达到官方行为的地方继续优化。++## 主要改动++- 在 `get_app_state` 找到 AX window 但找不到 on-screen CGWindow,或暂时找不到可用 focused window 时,增加 best-effort 窗口恢复流程。+- 恢复流程会尝试 unhide、activate、`open -b <bundle-id>`、取消最小化、`AXRaise`、设置 main/focused,然后短暂等待并重试窗口匹配。+- 保留恢复失败后的官方形态错误文本:`Apple event error -10005: cgWindowNotFound`。+- 更新架构文档,说明隐藏 / 不可见 Electron 窗口会先恢复再采集。++## 验证++- 对比观察:开源版第一次请求隐藏的 Feishu 时返回 `cgWindowNotFound`,官方 `computer-use` 会拉起窗口并返回完整状态。+- 本地回归:隐藏 Feishu 后,使用新构建的 Dev app 直接调用 `get_app_state`,返回 `isError=false`,且包含截图内容块。+- `swift test --filter NoWindowErrorMessageMatchesOfficialShape`+- `./scripts/build-open-computer-use-app.sh debug`++## 受影响文件++- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `docs/ARCHITECTURE.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex 0e5f41b..1c9e67e 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -30,6 +30,7 @@ enum SnapshotMode {let accessibilityTreeMaxNodeCount = 1200let accessibilityTreeMaxDepth = 64let screenshotCaptureTimeout: TimeInterval = 5+private let windowVisibilityRecoveryDelay: TimeInterval = 0.7private let axWebAreaRole = "AXWebArea"public struct AppSnapshot {@@ -90,17 +91,54 @@ enum SnapshotBuilder {let appElement = AXUIElementCreateApplication(app.pid)enableBestEffortAccessibilityModes(appElement)let systemWide = AXUIElementCreateSystemWide()- let focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)- guard let focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide) else {+ var focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)+ var focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)+ if focusedWindow == nil, recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: nil) {+ focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)+ focusedWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)+ }++ var rootWindow: AXUIElement+ guard let resolvedFocusedWindow = focusedWindow else {throw ComputerUseError.stateUnavailable(computerUseNoWindowFoundMessage)}+ rootWindow = resolvedFocusedWindow++ var windowTitle = stringValue(of: rootWindow, attribute: kAXTitleAttribute)+ var windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)+ if windowCapture == nil, recoverVisibleWindow(for: app, appElement: appElement, preferredWindow: rootWindow) {+ focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)+ if let recoveredWindow = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide) {+ rootWindow = recoveredWindow+ windowTitle = stringValue(of: recoveredWindow, attribute: kAXTitleAttribute)+ windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)+ }+ }- let windowTitle = stringValue(of: focusedWindow, attribute: kAXTitleAttribute)- guard let windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle) else {+ guard let windowCapture else {throw ComputerUseError.stateUnavailable(computerUseNoWindowFoundMessage)}- let rootElement = focusedWindow+ return buildAccessibilitySnapshot(+ app: app,+ appElement: appElement,+ rootElement: rootWindow,+ windowTitle: windowTitle,+ windowCapture: windowCapture,+ focusedApplication: focusedApplication,+ systemWide: systemWide+ )+ }++ private static func buildAccessibilitySnapshot(+ app: RunningAppDescriptor,+ appElement: AXUIElement,+ rootElement: AXUIElement,+ windowTitle: String?,+ windowCapture: WindowCapture,+ focusedApplication: AXUIElement?,+ systemWide: AXUIElement+ ) -> AppSnapshot {let windowBounds = windowCapture.boundslet screenshotPNGData = windowCapture.pngDataIfAvailable()let focusedElement = preferredFocusedElement(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)@@ -130,6 +168,48 @@ enum SnapshotBuilder {)}+ private static func recoverVisibleWindow(for app: RunningAppDescriptor, appElement: AXUIElement, preferredWindow: AXUIElement?) -> Bool {+ var recovered = false++ if let runningApplication = NSRunningApplication(processIdentifier: app.pid) {+ recovered = runningApplication.unhide() || recovered+ recovered = runningApplication.activate(options: [.activateAllWindows]) || recovered+ }++ if let bundleIdentifier = app.bundleIdentifier {+ recovered = openBundleIdentifier(bundleIdentifier) || recovered+ }++ if let window = preferredWindow ?? firstAnyWindow(for: appElement) {+ recovered = unminimize(window) || recovered+ recovered = raise(window) || recovered+ recovered = setBoolAttribute(named: kAXMainAttribute as String, on: window) || recovered+ recovered = setBoolAttribute(named: kAXFocusedAttribute as String, on: window) || recovered+ }++ if recovered {+ Thread.sleep(forTimeInterval: windowVisibilityRecoveryDelay)+ }++ return recovered+ }++ private static func openBundleIdentifier(_ bundleIdentifier: String) -> Bool {+ let process = Process()+ process.executableURL = URL(fileURLWithPath: "/usr/bin/open")+ process.arguments = ["-b", bundleIdentifier]+ process.standardOutput = Pipe()+ process.standardError = Pipe()++ do {+ try process.run()+ process.waitUntilExit()+ return process.terminationStatus == 0+ } catch {+ return false+ }+ }+private static func firstWindow(for appElement: AXUIElement) -> AXUIElement? {guard let windows = copyArray(appElement, attribute: kAXWindowsAttribute) else {return nil@@ -138,6 +218,31 @@ enum SnapshotBuilder {return windows.first(where: isUsableWindowElement(_:))}+ private static func firstAnyWindow(for appElement: AXUIElement) -> AXUIElement? {+ copyElement(appElement, attribute: kAXFocusedWindowAttribute)+ ?? copyArray(appElement, attribute: kAXWindowsAttribute)?.first(where: { stringValue(of: $0, attribute: kAXRoleAttribute) == kAXWindowRole as String })+ }++ private static func unminimize(_ window: AXUIElement) -> Bool {+ guard boolValue(of: window, attribute: kAXMinimizedAttribute) == true else {+ return false+ }++ return AXUIElementSetAttributeValue(window, kAXMinimizedAttribute as CFString, kCFBooleanFalse) == .success+ }++ private static func raise(_ window: AXUIElement) -> Bool {+ guard copyActions(window)?.contains(where: { $0.caseInsensitiveCompare(kAXRaiseAction as String) == .orderedSame }) == true else {+ return false+ }++ return AXUIElementPerformAction(window, kAXRaiseAction as CFString) == .success+ }++ private static func setBoolAttribute(named attribute: String, on element: AXUIElement) -> Bool {+ AXUIElementSetAttributeValue(element, attribute as CFString, kCFBooleanTrue) == .success+ }+private static func preferredFocusedWindow(appElement: AXUIElement, appPID: pid_t, focusedApplication: AXUIElement?, systemWide: AXUIElement) -> AXUIElement? {if let focusedApplication, pid(of: focusedApplication) == appPID {return usableWindowElement(from: copyElement(systemWide, attribute: kAXFocusedWindowAttribute))