Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Require visible window for app snapshots

Leo1a61062df68f
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 011fc43..be89942 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -60,7 +60,7 @@
- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。
- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。
-- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。
+- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`,否则返回明确的窗口不可用错误,不再把 application 根节点或无截图窗口伪装成可操作状态;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。
- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。
- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。
- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。
diff --git a/docs/histories/2026-05/20260507-1905-require-ax-window-for-snapshot.md b/docs/histories/2026-05/20260507-1905-require-ax-window-for-snapshot.md
index e0b3745..ac5989e 100644
--- a/docs/histories/2026-05/20260507-1905-require-ax-window-for-snapshot.md
+++ b/docs/histories/2026-05/20260507-1905-require-ax-window-for-snapshot.md
@@ -20,3 +20,19 @@ Official `computer-use` returns an unavailable-window error for the observed Lar
### 📁 Files Modified
- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+
+### 🔁 Follow-up (2026-05-07, require matching visible CG window)
+
+**Additional evidence:** Official `computer-use` `1.0.770` strings include `cgWindowNotFound`, `noWindowsAvailable`, `matchingWindowNotFound`, and `AXWindowMiniaturized` / `AXWindowDeminiaturized`, and current official calls return `cgWindowNotFound` for Lark, Chrome, and System Settings when the desktop exposes no usable window state.
+
+**Additional changes:**
+- Filtered window candidates to non-minimized `AXWindow` elements.
+- Required a matching on-screen `CGWindow` before rendering a real-app snapshot.
+- Synchronized `docs/ARCHITECTURE.md` with the stricter window precondition.
+
+**Validation:**
+- `swift test`
+- `./scripts/check-docs.sh`
+- `git diff --check`
+- `./scripts/run-tool-smoke-tests.sh`
+- Manual comparison: official `computer-use` and current source build both return unavailable-window errors for the current Lark / Chrome / System Settings no-window state.
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index e454c7a..f573e5a 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -96,10 +96,13 @@ enum SnapshotBuilder {
}
let windowTitle = stringValue(of: focusedWindow, attribute: kAXTitleAttribute)
- let windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle)
+ guard let windowCapture = WindowCapture.resolve(for: app.pid, titleHint: windowTitle) else {
+ throw ComputerUseError.stateUnavailable("No visible window is available for \(app.bundleIdentifier ?? app.name). Bring the app window on screen and call get_app_state again.")
+ }
+
let rootElement = focusedWindow
- let windowBounds = windowCapture?.bounds
- let screenshotPNGData = windowCapture?.pngDataIfAvailable()
+ let windowBounds = windowCapture.bounds
+ let screenshotPNGData = windowCapture.pngDataIfAvailable()
let focusedElement = preferredFocusedElement(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)
let selectedText = focusedElement.flatMap(copySelectedText(_:))
let context = RenderContext(windowBounds: windowBounds, focusedElement: focusedElement)
@@ -116,8 +119,8 @@ enum SnapshotBuilder {
app: app,
windowTitle: windowTitle,
windowBounds: windowBounds,
- targetWindowID: windowCapture?.windowID,
- targetWindowLayer: windowCapture?.layer,
+ targetWindowID: windowCapture.windowID,
+ targetWindowLayer: windowCapture.layer,
screenshotPNGData: screenshotPNGData,
mode: .accessibility,
treeLines: renderer.lines,
@@ -132,31 +135,32 @@ enum SnapshotBuilder {
return nil
}
- return windows.first(where: isWindowElement(_:))
+ return windows.first(where: isUsableWindowElement(_:))
}
private static func preferredFocusedWindow(appElement: AXUIElement, appPID: pid_t, focusedApplication: AXUIElement?, systemWide: AXUIElement) -> AXUIElement? {
if let focusedApplication, pid(of: focusedApplication) == appPID {
- return windowElement(from: copyElement(systemWide, attribute: kAXFocusedWindowAttribute))
- ?? windowElement(from: copyElement(focusedApplication, attribute: kAXFocusedWindowAttribute))
+ return usableWindowElement(from: copyElement(systemWide, attribute: kAXFocusedWindowAttribute))
+ ?? usableWindowElement(from: copyElement(focusedApplication, attribute: kAXFocusedWindowAttribute))
?? firstWindow(for: focusedApplication)
- ?? windowElement(from: copyElement(appElement, attribute: kAXFocusedWindowAttribute))
+ ?? usableWindowElement(from: copyElement(appElement, attribute: kAXFocusedWindowAttribute))
?? firstWindow(for: appElement)
}
- return windowElement(from: copyElement(appElement, attribute: kAXFocusedWindowAttribute)) ?? firstWindow(for: appElement)
+ return usableWindowElement(from: copyElement(appElement, attribute: kAXFocusedWindowAttribute)) ?? firstWindow(for: appElement)
}
- private static func windowElement(from element: AXUIElement?) -> AXUIElement? {
- guard let element, isWindowElement(element) else {
+ private static func usableWindowElement(from element: AXUIElement?) -> AXUIElement? {
+ guard let element, isUsableWindowElement(element) else {
return nil
}
return element
}
- private static func isWindowElement(_ element: AXUIElement) -> Bool {
+ private static func isUsableWindowElement(_ element: AXUIElement) -> Bool {
stringValue(of: element, attribute: kAXRoleAttribute) == kAXWindowRole as String
+ && boolValue(of: element, attribute: kAXMinimizedAttribute) != true
}
private static func preferredFocusedElement(appElement: AXUIElement, appPID: pid_t, focusedApplication: AXUIElement?, systemWide: AXUIElement) -> AXUIElement? {