👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
fix(snapshot): 保留匿名 Web 点击控件
tisfengc8a7758d50e1
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex dab42e5..1de0431 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -62,7 +62,7 @@- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。-- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。+- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但如果匿名通用节点暴露 `AXPress` / `AXConfirm` / `AXOpen` 且 frame 有效、尺寸紧凑,会保留为带窗口相对 `Frame` 的 `button`,让 icon-only Web 控件仍能获得可区分的 `element_index`,同时继续过滤零尺寸或覆盖大面积页面的匿名点击容器。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。diff --git a/docs/histories/2026-07/20260716-1548-preserve-anonymous-web-actions.md b/docs/histories/2026-07/20260716-1548-preserve-anonymous-web-actions.mdnew file mode 100644index 0000000..e445d14--- /dev/null+++ b/docs/histories/2026-07/20260716-1548-preserve-anonymous-web-actions.md@@ -0,0 +1,27 @@+## [2026-07-16 15:48] | Task: 保留匿名 Web 图标按钮++### 🤖 Execution Context+* **Agent ID**: `/root`+* **Base Model**: GPT-5+* **Runtime**: Codex desktop++### 📥 User Query+> 修复 Chrome 页面中一列只有图标和悬浮提示的按钮无法出现在 snapshot、因而无法使用 `element_index` 点击的问题。++### 🛠 Changes Overview+**Scope:** `OpenComputerUseKit` macOS Accessibility snapshot renderer++**Key Actions:**+- **[保留主点击节点]**: 匿名 `AXGroup` / `AXUnknown` 如果暴露 `AXPress`、`AXConfirm` 或 `AXOpen`,不再仅因主动作被隐藏于文本输出而直接裁掉。+- **[限制噪音]**: 只保留具备有效紧凑 frame 的匿名动作节点,继续过滤零尺寸节点和覆盖大面积页面的通用点击容器。+- **[可操作输出]**: 将这些 icon-only 控件渲染为带窗口相对 `Frame` 的 `button`,使每个控件获得可区分的 `element_index`。+- **[验证]**: 补充主点击动作识别、匿名按钮判定、尺寸边界测试,并在 Chrome 实际页面确认右侧连续图标按钮进入 snapshot。++### 🧠 Design Intent (Why)+Web 页面经常用无文本的 iconfont 或 SVG 容器实现按钮。Chrome 可能只为这类节点暴露通用 AX role、frame 和主点击动作;旧 renderer 会先隐藏 `AXPress` 等隐式主动作,再把节点当作无语义 wrapper 裁掉。修复在不恢复所有空 wrapper 的前提下保留紧凑可点击节点,让 Agent 能通过 frame 和 `element_index` 操作它们。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-07/20260716-1548-preserve-anonymous-web-actions.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex c7c78f5..597c37e 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -91,6 +91,8 @@ private let windowVisibilityRecoveryDelay: TimeInterval = 0.7private let axWebAreaRole = "AXWebArea"private let axContentsAttribute = "AXContents"private let axVisibleChildrenAttribute = "AXVisibleChildren"+private let anonymousActionTargetMaxWidth: CGFloat = 240+private let anonymousActionTargetMaxHeight: CGFloat = 120public struct AppSnapshot {public let app: RunningAppDescriptor@@ -715,6 +717,17 @@ private struct TreeRenderer {let linkText = role == "AXLink" ? markdownLinkText(for: root, title: title, label: label, value: value, textLimit: context.textLimit) : nillet displayTitle = linkText ?? titlelet inlineRowSummary = outlineRowSummary(for: root, role: role)+ let exposesPrimaryClickAction = hasPrimaryClickAction(actions)+ let rendersAnonymousActionTarget = shouldRenderAnonymousActionTarget(+ role: role,+ title: displayTitle,+ label: label,+ help: help,+ value: value,+ genericTextSummary: genericTextSummary,+ hasPrimaryClickAction: exposesPrimaryClickAction,+ localFrame: localFrame+ )let hidesChildren = shouldSuppressChildren(role: role,title: displayTitle,@@ -745,7 +758,8 @@ private struct TreeRenderer {actions: prettyActions,childCount: childElements.count,genericTextSummary: genericTextSummary,- webAreaDepth: webAreaDepth+ webAreaDepth: webAreaDepth,+ preservesAnonymousActionTarget: rendersAnonymousActionTarget) {for child in childElements {render(child, depth: depth, ancestors: nextAncestors)@@ -783,6 +797,9 @@ private struct TreeRenderer {value: value,precedingSegments: [labelSegment, helpSegment, urlSegment, identifierSegment, valueSegment])+ let frameSegment = rendersAnonymousActionTarget+ ? localFrame.map { " Frame: \($0.renderedLocalFrame)" } ?? ""+ : ""let actionsPrefix = shouldCommaSeparateActions(title: displayTitle,inlineRowSummary: inlineRowSummary,@@ -790,9 +807,10 @@ private struct TreeRenderer {segments: [labelSegment, helpSegment, urlSegment, identifierSegment, valueSegment, placeholderSegment]) ? ", Secondary Actions: " : " Secondary Actions: "let actionsSegment = prettyActions.isEmpty ? "" : "\(actionsPrefix)\(prettyActions.joined(separator: ", "))"- let linePrefix = roleText.isEmpty ? "\(index)" : "\(index) \(roleText)"+ let renderedRoleText = rendersAnonymousActionTarget ? "button" : roleText+ let linePrefix = renderedRoleText.isEmpty ? "\(index)" : "\(index) \(renderedRoleText)"- let lineBody = "\(linePrefix)\(traitsSegment)\(titleSegment)\(rowSummarySegment)\(labelSegment)\(helpSegment)\(urlSegment)\(identifierSegment)\(valueSegment)\(placeholderSegment)"+ let lineBody = "\(linePrefix)\(traitsSegment)\(titleSegment)\(rowSummarySegment)\(labelSegment)\(helpSegment)\(urlSegment)\(identifierSegment)\(valueSegment)\(placeholderSegment)\(frameSegment)"lines.append("\(String(repeating: "\t", count: depth))\(lineBody)\(actionsSegment)")let record = ElementRecord(@@ -1415,13 +1433,18 @@ func shouldElideNode(actions: [String],childCount: Int,genericTextSummary: String? = nil,- webAreaDepth: Int? = nil+ webAreaDepth: Int? = nil,+ preservesAnonymousActionTarget: Bool = false) -> Bool {let genericRoles = [kAXGroupRole as String, kAXUnknownRole as String]guard genericRoles.contains(role) else {return false}+ if preservesAnonymousActionTarget {+ return false+ }+if genericTextSummary != nil {return false}@@ -1461,6 +1484,52 @@ private func traitsAreNonDescriptiveWrapperTraits(_ traits: [String]) -> Bool {traits.isEmpty || traits == ["settable", "string"]}+func hasPrimaryClickAction(_ actions: [String]) -> Bool {+ let primaryActions = [+ kAXPressAction as String,+ kAXConfirmAction as String,+ "AXOpen",+ ]++ return actions.contains { action in+ primaryActions.contains { $0.caseInsensitiveCompare(action) == .orderedSame }+ }+}++func shouldRenderAnonymousActionTarget(+ role: String,+ title: String?,+ label: String?,+ help: String?,+ value: String?,+ genericTextSummary: String?,+ hasPrimaryClickAction: Bool,+ localFrame: CGRect?+) -> Bool {+ guard hasPrimaryClickAction else {+ return false+ }++ guard role == kAXGroupRole as String || role == kAXUnknownRole as String else {+ return false+ }++ guard let localFrame,+ localFrame.width > 0,+ localFrame.height > 0,+ localFrame.width <= anonymousActionTargetMaxWidth,+ localFrame.height <= anonymousActionTargetMaxHeight+ else {+ return false+ }++ return title == nil+ && label == nil+ && help == nil+ && value == nil+ && genericTextSummary == nil+}+private func shouldSuppressChildren(role: String,title: String?,diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 93deb8e..ca54cd1 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -958,6 +958,17 @@ final class OpenComputerUseKitTests: XCTestCase {actions: [],childCount: 0))+ XCTAssertFalse(shouldElideNode(+ role: kAXGroupRole as String,+ title: nil,+ label: nil,+ value: nil,+ identifier: nil,+ traits: [],+ actions: [],+ childCount: 0,+ preservesAnonymousActionTarget: true+ ))XCTAssertFalse(shouldElideNode(role: kAXGroupRole as String,title: "Send",@@ -1056,6 +1067,97 @@ final class OpenComputerUseKitTests: XCTestCase {))}+ func testAccessibilityRendererRecognizesPrimaryClickActions() {+ XCTAssertTrue(hasPrimaryClickAction([kAXPressAction as String]))+ XCTAssertTrue(hasPrimaryClickAction([kAXConfirmAction as String]))+ XCTAssertTrue(hasPrimaryClickAction(["AXOpen"]))+ XCTAssertTrue(hasPrimaryClickAction(["axpress"]))+ XCTAssertFalse(hasPrimaryClickAction(["AXShowMenu", "AXScrollToVisible", "AXRaise"]))+ }++ func testAccessibilityRendererMarksAnonymousGenericClickTargetsAsButtons() {+ XCTAssertTrue(shouldRenderAnonymousActionTarget(+ role: kAXGroupRole as String,+ title: nil,+ label: nil,+ help: nil,+ value: nil,+ genericTextSummary: nil,+ hasPrimaryClickAction: true,+ localFrame: CGRect(x: 1455, y: 218, width: 15, height: 20)+ ))+ XCTAssertTrue(shouldRenderAnonymousActionTarget(+ role: kAXUnknownRole as String,+ title: nil,+ label: nil,+ help: nil,+ value: nil,+ genericTextSummary: nil,+ hasPrimaryClickAction: true,+ localFrame: CGRect(x: 1455, y: 245, width: 15, height: 20)+ ))+ XCTAssertFalse(shouldRenderAnonymousActionTarget(+ role: kAXButtonRole as String,+ title: nil,+ label: nil,+ help: nil,+ value: nil,+ genericTextSummary: nil,+ hasPrimaryClickAction: true,+ localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)+ ))+ XCTAssertFalse(shouldRenderAnonymousActionTarget(+ role: kAXGroupRole as String,+ title: "Forward",+ label: nil,+ help: nil,+ value: nil,+ genericTextSummary: nil,+ hasPrimaryClickAction: true,+ localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)+ ))+ XCTAssertFalse(shouldRenderAnonymousActionTarget(+ role: kAXGroupRole as String,+ title: nil,+ label: nil,+ help: nil,+ value: nil,+ genericTextSummary: "Candidate details",+ hasPrimaryClickAction: true,+ localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)+ ))+ XCTAssertFalse(shouldRenderAnonymousActionTarget(+ role: kAXGroupRole as String,+ title: nil,+ label: nil,+ help: nil,+ value: nil,+ genericTextSummary: nil,+ hasPrimaryClickAction: false,+ localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)+ ))+ XCTAssertFalse(shouldRenderAnonymousActionTarget(+ role: kAXGroupRole as String,+ title: nil,+ label: nil,+ help: nil,+ value: nil,+ genericTextSummary: nil,+ hasPrimaryClickAction: true,+ localFrame: CGRect(x: 0, y: 0, width: 1920, height: 929)+ ))+ XCTAssertFalse(shouldRenderAnonymousActionTarget(+ role: kAXGroupRole as String,+ title: nil,+ label: nil,+ help: nil,+ value: nil,+ genericTextSummary: nil,+ hasPrimaryClickAction: true,+ localFrame: CGRect(x: 0, y: 0, width: 15, height: 0)+ ))+ }+func testAccessibilityRendererOnlyMergesShortTextOnlySiblingRuns() {XCTAssertTrue(shouldMergeTextOnlySiblings(["AgentSphere", "17:18", "好的,谢谢"]))XCTAssertFalse(shouldMergeTextOnlySiblings(["日期", "时间", "2026年5月7日", "晚餐", "18:00-20:00"]))