👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Require visible window for app snapshots
Leo1a61062df68f
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 011fc43..be89942 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -60,7 +60,7 @@- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。-- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。+- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`,否则返回明确的窗口不可用错误,不再把 application 根节点或无截图窗口伪装成可操作状态;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。diff --git a/docs/histories/2026-05/20260507-1905-require-ax-window-for-snapshot.md b/docs/histories/2026-05/20260507-1905-require-ax-window-for-snapshot.mdindex e0b3745..ac5989e 100644--- a/docs/histories/2026-05/20260507-1905-require-ax-window-for-snapshot.md+++ b/docs/histories/2026-05/20260507-1905-require-ax-window-for-snapshot.md@@ -20,3 +20,19 @@ Official `computer-use` returns an unavailable-window error for the observed Lar### 📁 Files Modified- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`++### 🔁 Follow-up (2026-05-07, require matching visible CG window)++**Additional evidence:** Official `computer-use` `1.0.770` strings include `cgWindowNotFound`, `noWindowsAvailable`, `matchingWindowNotFound`, and `AXWindowMiniaturized` / `AXWindowDeminiaturized`, and current official calls return `cgWindowNotFound` for Lark, Chrome, and System Settings when the desktop exposes no usable window state.++**Additional changes:**+- Filtered window candidates to non-minimized `AXWindow` elements.+- Required a matching on-screen `CGWindow` before rendering a real-app snapshot.+- Synchronized `docs/ARCHITECTURE.md` with the stricter window precondition.++**Validation:**+- `swift test`+- `./scripts/check-docs.sh`+- `git diff --check`+- `./scripts/run-tool-smoke-tests.sh`+- Manual comparison: official `computer-use` and current source build both return unavailable-window errors for the current Lark / Chrome / System Settings no-window state.diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex e454c7a..f573e5a 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -96,10 +96,13 @@ enum SnapshotBuilder {}let windowTitle = stringValue(of: focusedWindow, attribute: kAXTitleAttribute)- let windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)+ guard let windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle) else {+ throw ComputerUseError.stateUnavailable("No visible window is available for \(app.bundleIdentifier ?? app.name). Bring the app window on screen and call get_app_state again.")+ }+let rootElement = focusedWindow- let windowBounds = windowCapture?.bounds- let screenshotPNGData = windowCapture?.pngDataIfAvailable()+ let windowBounds = windowCapture.bounds+ let screenshotPNGData = windowCapture.pngDataIfAvailable()let focusedElement = preferredFocusedElement(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)let selectedText = focusedElement.flatMap(copySelectedText(_:))let context = RenderContext(windowBounds: windowBounds, focusedElement: focusedElement)@@ -116,8 +119,8 @@ enum SnapshotBuilder {app: app,windowTitle: windowTitle,windowBounds: windowBounds,- targetWindowID: windowCapture?.windowID,- targetWindowLayer: windowCapture?.layer,+ targetWindowID: windowCapture.windowID,+ targetWindowLayer: windowCapture.layer,screenshotPNGData: screenshotPNGData,mode: .accessibility,treeLines: renderer.lines,@@ -132,31 +135,32 @@ enum SnapshotBuilder {return nil}- return windows.first(where: isWindowElement(_:))+ return windows.first(where: isUsableWindowElement(_:))}private static func preferredFocusedWindow(appElement: AXUIElement, appPID: pid_t, focusedApplication: AXUIElement?, systemWide: AXUIElement) -> AXUIElement? {if let focusedApplication, pid(of: focusedApplication) == appPID {- return windowElement(from: copyElement(systemWide, attribute: kAXFocusedWindowAttribute))- ?? windowElement(from: copyElement(focusedApplication, attribute: kAXFocusedWindowAttribute))+ return usableWindowElement(from: copyElement(systemWide, attribute: kAXFocusedWindowAttribute))+ ?? usableWindowElement(from: copyElement(focusedApplication, attribute: kAXFocusedWindowAttribute))?? firstWindow(for: focusedApplication)- ?? windowElement(from: copyElement(appElement, attribute: kAXFocusedWindowAttribute))+ ?? usableWindowElement(from: copyElement(appElement, attribute: kAXFocusedWindowAttribute))?? firstWindow(for: appElement)}- return windowElement(from: copyElement(appElement, attribute: kAXFocusedWindowAttribute)) ?? firstWindow(for: appElement)+ return usableWindowElement(from: copyElement(appElement, attribute: kAXFocusedWindowAttribute)) ?? firstWindow(for: appElement)}- private static func windowElement(from element: AXUIElement?) -> AXUIElement? {- guard let element, isWindowElement(element) else {+ private static func usableWindowElement(from element: AXUIElement?) -> AXUIElement? {+ guard let element, isUsableWindowElement(element) else {return nil}return element}- private static func isWindowElement(_ element: AXUIElement) -> Bool {+ private static func isUsableWindowElement(_ element: AXUIElement) -> Bool {stringValue(of: element, attribute: kAXRoleAttribute) == kAXWindowRole as String+ && boolValue(of: element, attribute: kAXMinimizedAttribute) != true}private static func preferredFocusedElement(appElement: AXUIElement, appPID: pid_t, focusedApplication: AXUIElement?, systemWide: AXUIElement) -> AXUIElement? {