👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Fix macOS open panel file clicks
Leo5208e19a6dc8
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 683b968..c9e3d13 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -60,7 +60,7 @@- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。-- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。+- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。@@ -75,7 +75,7 @@- 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略:- `perform_secondary_action` 只执行目标元素已经暴露出来的 AX action;无效 action 返回官方风格的 `... is not a valid secondary action for ...`,fixture 的 `Raise` 路径也不再为了测试去准备全局物理指针输入- `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口- - element-targeted `click` 的左键路径会先试 `AXPress` / `AXConfirm` / `AXOpen` 这类真正语义化的激活动作;如果目标本身不可点,还会继续尝试其子孙 AX 元素(例如 Finder sidebar row 下面暴露 `AXOpen` 的 cell)和命中点附近的 AX hit-test 结果,只有这些都失败后才退到窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute`,最后才会退到 `postToPid` 定向鼠标事件与显式 opt-in 的全局物理指针 fallback。这里也不再把 `AXUIElementIsAttributeSettable` 的结果当成硬门槛,因为像 `TextEdit` window 这类元素在 `kAXFocusedAttribute` 上会出现“`isSettable=false` 但直接 set 成功”的官方同款场景;`click_count > 1` 也会优先重复可用的 AX action+ - element-targeted `click` 的左键路径会先试原生列表的 `AXSelectedChildren` 选择,再试 `AXPress` / `AXConfirm` / `AXOpen` 这类真正语义化的激活动作;如果目标本身不可点,还会继续尝试其子孙 AX 元素(例如 Finder sidebar row 下面暴露 `AXOpen` 的 cell)和命中点附近的 AX hit-test 结果,只有这些都失败后才退到窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute`,最后才会退到 `postToPid` 定向鼠标事件与显式 opt-in 的全局物理指针 fallback。这里也不再把 `AXUIElementIsAttributeSettable` 的结果当成硬门槛,因为像 `TextEdit` window 这类元素在 `kAXFocusedAttribute` 上会出现“`isSettable=false` 但直接 set 成功”的官方同款场景;`click_count > 1` 也会优先重复可用的 AX action- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias- `scroll.pages` 对齐官方 `1.0.755` 的 `number` schema,支持小数页数;整数页且目标暴露 `AXScroll*ByPage` 时优先走 AX action,否则用 `CGEvent.postToPid` 向目标进程定向发送 scroll eventdiff --git a/docs/histories/2026-05/20260511-1029-fix-open-panel-file-click.md b/docs/histories/2026-05/20260511-1029-fix-open-panel-file-click.mdnew file mode 100644index 0000000..7992676--- /dev/null+++ b/docs/histories/2026-05/20260511-1029-fix-open-panel-file-click.md@@ -0,0 +1,32 @@+# Fix open panel file click++## User Request++> Investigate why `open-computer-use` could click directories in a macOS upload/open file panel but could not click files in the right-hand file list.++## Changes++- **[Open panel tree rendering]**: Added `AXContents` / `AXVisibleChildren` traversal so native open panel column views expose visible file items instead of stopping at empty scroll areas.+- **[Modal window capture]**: Updated window capture selection to prefer a frontmost overlapping modal/sheet window over a stale exact title match with the underlying main window.+- **[List item click]**: Made left click select native `AXList` items through `AXSelectedChildren`, and made coordinate clicks try the smallest containing snapshot element before generic AX hit-test results.+- **[Regression coverage]**: Added unit tests for modal window selection and visible-child list traversal.+- **[Docs]**: Updated `docs/ARCHITECTURE.md` with the open panel traversal and list selection behavior.++## Rationale++The live Nomi open panel exposed the sidebar through `AXRows`, but the file list was under `AXScrollArea -> AXContents -> AXList -> AXVisibleChildren`. The previous renderer did not traverse those attributes, so file rows were absent from the tool state. Coordinate clicks then hit generic sheet/list elements and could report success through focus-style fallbacks without selecting a file.++The fix keeps the preferred path semantic: surface the visible file items, select native list children through AX, and only use targeted mouse fallback after the more precise candidates fail.++## Files++- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`++## Validation++- `swift test`+- Live local CLI check with `OPEN_COMPUTER_USE_DISABLE_APP_AGENT_PROXY=1 .build/debug/OpenComputerUse call get_app_state --args '{"app":"Nomi"}'` showed `IMG_7605.JPG`, `IMG_7624.HEIC`, and `IMG_7753.JPG` as file-list elements.+- Live local CLI coordinate click changed the open panel preview from `IMG_7605.JPG` to `IMG_7753.JPG`, with the `Open` button enabled.diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex ab22860..c5cbd70 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -35,6 +35,8 @@ let screenshotResultMaxDimension: CGFloat = 1280let screenshotResultMinScale: CGFloat = 0.25private let windowVisibilityRecoveryDelay: TimeInterval = 0.7private let axWebAreaRole = "AXWebArea"+private let axContentsAttribute = "AXContents"+private let axVisibleChildrenAttribute = "AXVisibleChildren"public struct AppSnapshot {public let app: RunningAppDescriptor@@ -345,7 +347,7 @@ private struct WindowCapture {return nil}- let candidates = infoList.compactMap { info -> (CGWindowID, Int, CGRect, String?, Int)? in+ let candidates = infoList.enumerated().compactMap { offset, info -> WindowCaptureCandidate? inguardlet ownerPID = info[kCGWindowOwnerPID as String] as? pid_t,ownerPID == pid,@@ -359,28 +361,23 @@ private struct WindowCapture {let title = info[kCGWindowName as String] as? Stringlet area = Int(bounds.width * bounds.height)- return (CGWindowID(number.uint32Value), layer, bounds, title, area)+ return WindowCaptureCandidate(+ windowID: CGWindowID(number.uint32Value),+ layer: layer,+ bounds: bounds,+ title: title,+ area: area,+ frontToBackIndex: offset+ )}- guard let best = candidates.sorted(by: { lhs, rhs in- if let titleHint {- if lhs.3 == titleHint && rhs.3 != titleHint {- return true- }-- if rhs.3 == titleHint && lhs.3 != titleHint {- return false- }- }-- return lhs.4 > rhs.4- }).first else {+ guard let best = preferredWindowCaptureCandidate(candidates, titleHint: titleHint) else {return nil}- let image = captureImage(windowID: best.0, bounds: best.2)+ let image = captureImage(windowID: best.windowID, bounds: best.bounds)- return WindowCapture(windowID: best.0, layer: best.1, bounds: best.2, image: image)+ return WindowCapture(windowID: best.windowID, layer: best.layer, bounds: best.bounds, image: image)}private static func captureImage(windowID: CGWindowID, bounds: CGRect) -> CGImage? {@@ -419,6 +416,47 @@ private struct WindowCapture {}}+struct WindowCaptureCandidate {+ let windowID: CGWindowID+ let layer: Int+ let bounds: CGRect+ let title: String?+ let area: Int+ let frontToBackIndex: Int+}++func preferredWindowCaptureCandidate(_ candidates: [WindowCaptureCandidate], titleHint: String?) -> WindowCaptureCandidate? {+ let usable = candidates+ .filter { $0.layer == 0 && $0.area >= 20_000 }+ .sorted { lhs, rhs in+ lhs.frontToBackIndex < rhs.frontToBackIndex+ }++ guard !usable.isEmpty else {+ return candidates.sorted { lhs, rhs in+ lhs.area > rhs.area+ }.first+ }++ guard let titleHint, !titleHint.isEmpty,+ let hinted = usable.first(where: { $0.title == titleHint })+ else {+ return usable.first+ }++ guard let frontmost = usable.first else {+ return hinted+ }++ if frontmost.windowID != hinted.windowID,+ frontmost.bounds.intersects(hinted.bounds)+ {+ return frontmost+ }++ return hinted+}+func boundedScreenshotPNGData(for image: CGImage,maxBytes: Int = screenshotResultMaxPNGBytes,@@ -761,26 +799,27 @@ private struct TreeRenderer {}private func children(of element: AXUIElement) -> [AXUIElement] {- let attributes = [kAXChildrenAttribute, kAXRowsAttribute]+ let role = stringValue(of: element, attribute: kAXRoleAttribute)+ let rows = copyArray(element, attribute: kAXRowsAttribute) ?? []+ let visibleChildren = copyArray(element, attribute: axVisibleChildrenAttribute) ?? []+ let attributes = childTraversalAttributes(+ role: role,+ hasRows: !rows.isEmpty,+ hasVisibleChildren: !visibleChildren.isEmpty+ )var children: [AXUIElement] = []- let rowsArePrimaryChildren = usesRowsAsPrimaryChildren(element)for attribute in attributes {- guard let sourceValues = copyArray(element, attribute: attribute) else {- continue+ let sourceValues: [AXUIElement]+ if attribute == kAXRowsAttribute {+ sourceValues = rows+ } else if attribute == axVisibleChildrenAttribute {+ sourceValues = visibleChildren+ } else {+ sourceValues = copyArray(element, attribute: attribute) ?? []}- if attribute == kAXChildrenAttribute,- rowsArePrimaryChildren,- let rows = copyArray(element, attribute: kAXRowsAttribute),- !rows.isEmpty- {- continue- }-- let values = attribute == kAXRowsAttribute- ? visibleRows(in: sourceValues, parent: element)- : sourceValues+ let values = attribute == kAXRowsAttribute ? visibleRows(in: sourceValues, parent: element) : sourceValuesfor child in values {if shouldSkipChild(child, of: element) {@@ -797,8 +836,18 @@ private struct TreeRenderer {}}-private func usesRowsAsPrimaryChildren(_ element: AXUIElement) -> Bool {- let role = stringValue(of: element, attribute: kAXRoleAttribute)+func childTraversalAttributes(role: String?, hasRows: Bool, hasVisibleChildren: Bool) -> [String] {+ var attributes: [String] = []+ if !(hasRows && usesRowsAsPrimaryRole(role)) && !(hasVisibleChildren && usesVisibleChildrenAsPrimaryRole(role)) {+ attributes.append(kAXChildrenAttribute)+ }+ attributes.append(kAXRowsAttribute)+ attributes.append(axContentsAttribute)+ attributes.append(axVisibleChildrenAttribute)+ return attributes+}++private func usesRowsAsPrimaryRole(_ role: String?) -> Bool {return [kAXOutlineRole as String,kAXListRole as String,@@ -807,6 +856,10 @@ private func usesRowsAsPrimaryChildren(_ element: AXUIElement) -> Bool {].contains(role)}+private func usesVisibleChildrenAsPrimaryRole(_ role: String?) -> Bool {+ role == kAXListRole as String+}+private func shouldSkipChild(_ child: AXUIElement, of parent: AXUIElement) -> Bool {let parentRole = stringValue(of: parent, attribute: kAXRoleAttribute)guard parentRole == kAXMenuBarRole as String else {diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex 6653518..982e466 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -226,7 +226,8 @@ public final class ComputerUseService {snapshot: snapshot,button: button,clickCount: clickCount,- includeNearbyHitTesting: true+ includeNearbyHitTesting: true,+ allowActivationFallback: true)) {try performNonAXClickFallback(at: targetPoint,@@ -255,16 +256,23 @@ public final class ComputerUseService {moveVisualCursor(to: cursorTarget)do {- if let record = try hitTestElement(at: point, in: snapshot) ?? bestElement(containing: point, in: snapshot),- (try performAXClickSequence(- on: record,- snapshot: snapshot,- button: button,- clickCount: clickCount,- includeNearbyHitTesting: false- )) {- // handled- } else {+ let candidates = try clickCandidates(at: point, in: snapshot)+ var handled = false+ for record in candidates {+ if try performAXClickSequence(+ on: record,+ snapshot: snapshot,+ button: button,+ clickCount: clickCount,+ includeNearbyHitTesting: false,+ allowActivationFallback: false+ ) {+ handled = true+ break+ }+ }++ if !handled {try performNonAXClickFallback(at: targetPoint,button: button,@@ -502,6 +510,10 @@ public final class ComputerUseService {switch button {case .left:+ if clickCount <= 1, try selectContainingListItem(for: element) {+ return true+ }+if try performAction(named: kAXPressAction as String, on: element, availableActions: record.rawActions, repeatCount: clickCount) {return true}@@ -524,12 +536,83 @@ public final class ComputerUseService {return false}+ private func clickCandidates(at point: CGPoint, in snapshot: AppSnapshot) throws -> [ElementRecord] {+ var candidates: [ElementRecord] = []++ if let bestRecord = bestElement(containing: point, in: snapshot) {+ candidates.append(bestRecord)+ }++ if let hitRecord = try hitTestElement(at: point, in: snapshot) {+ candidates.append(hitRecord)+ }++ return candidates.reduce(into: []) { uniqueCandidates, candidate in+ if !uniqueCandidates.contains(where: { sameElement($0.element, candidate.element) }) {+ uniqueCandidates.append(candidate)+ }+ }+ }++ private func sameElement(_ lhs: AXUIElement?, _ rhs: AXUIElement?) -> Bool {+ guard let lhs, let rhs else {+ return false+ }++ return CFEqual(lhs, rhs)+ }++ private func selectContainingListItem(for element: AXUIElement) throws -> Bool {+ guard let target = selectableListItem(containing: element) else {+ return false+ }++ let result = AXUIElementSetAttributeValue(+ target.list,+ kAXSelectedChildrenAttribute as CFString,+ [target.item] as CFArray+ )++ switch result {+ case .success:+ Thread.sleep(forTimeInterval: 0.15)+ return true+ case .failure, .attributeUnsupported, .actionUnsupported, .cannotComplete, .noValue, .invalidUIElement, .illegalArgument:+ return false+ default:+ throw ComputerUseError.message("AXUIElementSetAttributeValue(\(kAXSelectedChildrenAttribute)) failed with \(result.rawValue)")+ }+ }++ private func selectableListItem(containing element: AXUIElement) -> (list: AXUIElement, item: AXUIElement)? {+ var current = element+ var directChild = element++ for _ in 0..<8 {+ guard let parent = copyParent(of: current) else {+ return nil+ }++ if stringValue(of: parent, attribute: kAXRoleAttribute) == kAXListRole as String,+ isSettable(element: parent, attribute: kAXSelectedChildrenAttribute)+ {+ return (parent, directChild)+ }++ directChild = parent+ current = parent+ }++ return nil+ }+private func performAXClickSequence(on record: ElementRecord,snapshot: AppSnapshot,button: MouseButtonKind,clickCount: Int,- includeNearbyHitTesting: Bool+ includeNearbyHitTesting: Bool,+ allowActivationFallback: Bool) throws -> Bool {if try performPreferredClick(on: record, button: button, clickCount: clickCount) {Thread.sleep(forTimeInterval: 0.15)@@ -563,7 +646,7 @@ public final class ComputerUseService {}}- guard button == .left, let element = record.element else {+ guard allowActivationFallback, button == .left, let element = record.element else {return false}@@ -795,6 +878,26 @@ public final class ComputerUseService {return value as? [AXUIElement] ?? []}+ private func copyParent(of element: AXUIElement) -> AXUIElement? {+ var value: CFTypeRef?+ let result = AXUIElementCopyAttributeValue(element, kAXParentAttribute as CFString, &value)+ guard result == .success, let value else {+ return nil+ }++ return (value as! AXUIElement)+ }++ private func stringValue(of element: AXUIElement, attribute: String) -> String? {+ var value: CFTypeRef?+ let result = AXUIElementCopyAttributeValue(element, attribute as CFString, &value)+ guard result == .success, let value else {+ return nil+ }++ return value as? String+ }+private func localFrame(of element: AXUIElement, windowBounds: CGRect?) -> CGRect? {var positionValue: CFTypeRef?var sizeValue: CFTypeRef?diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex c39307f..b8056dc 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -1026,6 +1026,64 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(point, CGPoint(x: 530, y: 395))}+ func testWindowCapturePrefersFrontmostOverHintWhenModalOverlapsHintedWindow() {+ let main = WindowCaptureCandidate(+ windowID: 1,+ layer: 0,+ bounds: CGRect(x: 100, y: 100, width: 800, height: 600),+ title: "Nomi",+ area: 480_000,+ frontToBackIndex: 1+ )+ let openPanel = WindowCaptureCandidate(+ windowID: 2,+ layer: 0,+ bounds: CGRect(x: 120, y: 180, width: 880, height: 448),+ title: "Open",+ area: 394_240,+ frontToBackIndex: 0+ )++ let selected = preferredWindowCaptureCandidate([openPanel, main], titleHint: "Nomi")++ XCTAssertEqual(selected?.windowID, openPanel.windowID)+ }++ func testWindowCaptureKeepsHintedWindowWhenFrontmostDoesNotOverlap() {+ let main = WindowCaptureCandidate(+ windowID: 1,+ layer: 0,+ bounds: CGRect(x: 100, y: 100, width: 800, height: 600),+ title: "Nomi",+ area: 480_000,+ frontToBackIndex: 1+ )+ let other = WindowCaptureCandidate(+ windowID: 2,+ layer: 0,+ bounds: CGRect(x: 1_200, y: 100, width: 400, height: 300),+ title: "Utility",+ area: 120_000,+ frontToBackIndex: 0+ )++ let selected = preferredWindowCaptureCandidate([other, main], titleHint: "Nomi")++ XCTAssertEqual(selected?.windowID, main.windowID)+ }++ func testListTraversalPrefersVisibleChildrenAndReadsContents() {+ let attributes = childTraversalAttributes(+ role: kAXListRole as String,+ hasRows: false,+ hasVisibleChildren: true+ )++ XCTAssertFalse(attributes.contains(kAXChildrenAttribute))+ XCTAssertTrue(attributes.contains("AXContents"))+ XCTAssertTrue(attributes.contains("AXVisibleChildren"))+ }+func testCursorWindowGeometryAnchorsTipPosition() {let geometry = CursorWindowGeometry(windowSize: CGSize(width: 128, height: 128),