Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

fix(snapshot): 保留匿名 Web 点击控件

tisfengc8a7758d50e1
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index dab42e5..1de0431 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -62,7 +62,7 @@
- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。
- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。
-- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。
+- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但如果匿名通用节点暴露 `AXPress` / `AXConfirm` / `AXOpen` 且 frame 有效、尺寸紧凑,会保留为带窗口相对 `Frame` 的 `button`,让 icon-only Web 控件仍能获得可区分的 `element_index`,同时继续过滤零尺寸或覆盖大面积页面的匿名点击容器。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。
- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。
- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。
- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。
diff --git a/docs/histories/2026-07/20260716-1548-preserve-anonymous-web-actions.md b/docs/histories/2026-07/20260716-1548-preserve-anonymous-web-actions.md
new file mode 100644
index 0000000..e445d14
--- /dev/null
+++ b/docs/histories/2026-07/20260716-1548-preserve-anonymous-web-actions.md
@@ -0,0 +1,27 @@
+## [2026-07-16 15:48] | Task: 保留匿名 Web 图标按钮
+
+### 🤖 Execution Context
+* **Agent ID**: `/root`
+* **Base Model**: GPT-5
+* **Runtime**: Codex desktop
+
+### 📥 User Query
+> 修复 Chrome 页面中一列只有图标和悬浮提示的按钮无法出现在 snapshot、因而无法使用 `element_index` 点击的问题。
+
+### 🛠 Changes Overview
+**Scope:** `OpenComputerUseKit` macOS Accessibility snapshot renderer
+
+**Key Actions:**
+- **[保留主点击节点]**: 匿名 `AXGroup` / `AXUnknown` 如果暴露 `AXPress`、`AXConfirm` 或 `AXOpen`,不再仅因主动作被隐藏于文本输出而直接裁掉。
+- **[限制噪音]**: 只保留具备有效紧凑 frame 的匿名动作节点,继续过滤零尺寸节点和覆盖大面积页面的通用点击容器。
+- **[可操作输出]**: 将这些 icon-only 控件渲染为带窗口相对 `Frame` 的 `button`,使每个控件获得可区分的 `element_index`。
+- **[验证]**: 补充主点击动作识别、匿名按钮判定、尺寸边界测试,并在 Chrome 实际页面确认右侧连续图标按钮进入 snapshot。
+
+### 🧠 Design Intent (Why)
+Web 页面经常用无文本的 iconfont 或 SVG 容器实现按钮。Chrome 可能只为这类节点暴露通用 AX role、frame 和主点击动作;旧 renderer 会先隐藏 `AXPress` 等隐式主动作,再把节点当作无语义 wrapper 裁掉。修复在不恢复所有空 wrapper 的前提下保留紧凑可点击节点,让 Agent 能通过 frame 和 `element_index` 操作它们。
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
+- `docs/histories/2026-07/20260716-1548-preserve-anonymous-web-actions.md`
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index c7c78f5..597c37e 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -91,6 +91,8 @@ private let windowVisibilityRecoveryDelay: TimeInterval = 0.7
private let axWebAreaRole = "AXWebArea"
private let axContentsAttribute = "AXContents"
private let axVisibleChildrenAttribute = "AXVisibleChildren"
+private let anonymousActionTargetMaxWidth: CGFloat = 240
+private let anonymousActionTargetMaxHeight: CGFloat = 120
public struct AppSnapshot {
public let app: RunningAppDescriptor
@@ -715,6 +717,17 @@ private struct TreeRenderer {
let linkText = role == "AXLink" ? markdownLinkText(for: root, title: title, label: label, value: value, textLimit: context.textLimit) : nil
let displayTitle = linkText ?? title
let inlineRowSummary = outlineRowSummary(for: root, role: role)
+ let exposesPrimaryClickAction = hasPrimaryClickAction(actions)
+ let rendersAnonymousActionTarget = shouldRenderAnonymousActionTarget(
+ role: role,
+ title: displayTitle,
+ label: label,
+ help: help,
+ value: value,
+ genericTextSummary: genericTextSummary,
+ hasPrimaryClickAction: exposesPrimaryClickAction,
+ localFrame: localFrame
+ )
let hidesChildren = shouldSuppressChildren(
role: role,
title: displayTitle,
@@ -745,7 +758,8 @@ private struct TreeRenderer {
actions: prettyActions,
childCount: childElements.count,
genericTextSummary: genericTextSummary,
- webAreaDepth: webAreaDepth
+ webAreaDepth: webAreaDepth,
+ preservesAnonymousActionTarget: rendersAnonymousActionTarget
) {
for child in childElements {
render(child, depth: depth, ancestors: nextAncestors)
@@ -783,6 +797,9 @@ private struct TreeRenderer {
value: value,
precedingSegments: [labelSegment, helpSegment, urlSegment, identifierSegment, valueSegment]
)
+ let frameSegment = rendersAnonymousActionTarget
+ ? localFrame.map { " Frame: \($0.renderedLocalFrame)" } ?? ""
+ : ""
let actionsPrefix = shouldCommaSeparateActions(
title: displayTitle,
inlineRowSummary: inlineRowSummary,
@@ -790,9 +807,10 @@ private struct TreeRenderer {
segments: [labelSegment, helpSegment, urlSegment, identifierSegment, valueSegment, placeholderSegment]
) ? ", Secondary Actions: " : " Secondary Actions: "
let actionsSegment = prettyActions.isEmpty ? "" : "\(actionsPrefix)\(prettyActions.joined(separator: ", "))"
- let linePrefix = roleText.isEmpty ? "\(index)" : "\(index) \(roleText)"
+ let renderedRoleText = rendersAnonymousActionTarget ? "button" : roleText
+ let linePrefix = renderedRoleText.isEmpty ? "\(index)" : "\(index) \(renderedRoleText)"
- let lineBody = "\(linePrefix)\(traitsSegment)\(titleSegment)\(rowSummarySegment)\(labelSegment)\(helpSegment)\(urlSegment)\(identifierSegment)\(valueSegment)\(placeholderSegment)"
+ let lineBody = "\(linePrefix)\(traitsSegment)\(titleSegment)\(rowSummarySegment)\(labelSegment)\(helpSegment)\(urlSegment)\(identifierSegment)\(valueSegment)\(placeholderSegment)\(frameSegment)"
lines.append("\(String(repeating: "\t", count: depth))\(lineBody)\(actionsSegment)")
let record = ElementRecord(
@@ -1415,13 +1433,18 @@ func shouldElideNode(
actions: [String],
childCount: Int,
genericTextSummary: String? = nil,
- webAreaDepth: Int? = nil
+ webAreaDepth: Int? = nil,
+ preservesAnonymousActionTarget: Bool = false
) -> Bool {
let genericRoles = [kAXGroupRole as String, kAXUnknownRole as String]
guard genericRoles.contains(role) else {
return false
}
+ if preservesAnonymousActionTarget {
+ return false
+ }
+
if genericTextSummary != nil {
return false
}
@@ -1461,6 +1484,52 @@ private func traitsAreNonDescriptiveWrapperTraits(_ traits: [String]) -> Bool {
traits.isEmpty || traits == ["settable", "string"]
}
+func hasPrimaryClickAction(_ actions: [String]) -> Bool {
+ let primaryActions = [
+ kAXPressAction as String,
+ kAXConfirmAction as String,
+ "AXOpen",
+ ]
+
+ return actions.contains { action in
+ primaryActions.contains { $0.caseInsensitiveCompare(action) == .orderedSame }
+ }
+}
+
+func shouldRenderAnonymousActionTarget(
+ role: String,
+ title: String?,
+ label: String?,
+ help: String?,
+ value: String?,
+ genericTextSummary: String?,
+ hasPrimaryClickAction: Bool,
+ localFrame: CGRect?
+) -> Bool {
+ guard hasPrimaryClickAction else {
+ return false
+ }
+
+ guard role == kAXGroupRole as String || role == kAXUnknownRole as String else {
+ return false
+ }
+
+ guard let localFrame,
+ localFrame.width > 0,
+ localFrame.height > 0,
+ localFrame.width <= anonymousActionTargetMaxWidth,
+ localFrame.height <= anonymousActionTargetMaxHeight
+ else {
+ return false
+ }
+
+ return title == nil
+ && label == nil
+ && help == nil
+ && value == nil
+ && genericTextSummary == nil
+}
+
private func shouldSuppressChildren(
role: String,
title: String?,
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 93deb8e..ca54cd1 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -958,6 +958,17 @@ final class OpenComputerUseKitTests: XCTestCase {
actions: [],
childCount: 0
))
+ XCTAssertFalse(shouldElideNode(
+ role: kAXGroupRole as String,
+ title: nil,
+ label: nil,
+ value: nil,
+ identifier: nil,
+ traits: [],
+ actions: [],
+ childCount: 0,
+ preservesAnonymousActionTarget: true
+ ))
XCTAssertFalse(shouldElideNode(
role: kAXGroupRole as String,
title: "Send",
@@ -1056,6 +1067,97 @@ final class OpenComputerUseKitTests: XCTestCase {
))
}
+ func testAccessibilityRendererRecognizesPrimaryClickActions() {
+ XCTAssertTrue(hasPrimaryClickAction([kAXPressAction as String]))
+ XCTAssertTrue(hasPrimaryClickAction([kAXConfirmAction as String]))
+ XCTAssertTrue(hasPrimaryClickAction(["AXOpen"]))
+ XCTAssertTrue(hasPrimaryClickAction(["axpress"]))
+ XCTAssertFalse(hasPrimaryClickAction(["AXShowMenu", "AXScrollToVisible", "AXRaise"]))
+ }
+
+ func testAccessibilityRendererMarksAnonymousGenericClickTargetsAsButtons() {
+ XCTAssertTrue(shouldRenderAnonymousActionTarget(
+ role: kAXGroupRole as String,
+ title: nil,
+ label: nil,
+ help: nil,
+ value: nil,
+ genericTextSummary: nil,
+ hasPrimaryClickAction: true,
+ localFrame: CGRect(x: 1455, y: 218, width: 15, height: 20)
+ ))
+ XCTAssertTrue(shouldRenderAnonymousActionTarget(
+ role: kAXUnknownRole as String,
+ title: nil,
+ label: nil,
+ help: nil,
+ value: nil,
+ genericTextSummary: nil,
+ hasPrimaryClickAction: true,
+ localFrame: CGRect(x: 1455, y: 245, width: 15, height: 20)
+ ))
+ XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ role: kAXButtonRole as String,
+ title: nil,
+ label: nil,
+ help: nil,
+ value: nil,
+ genericTextSummary: nil,
+ hasPrimaryClickAction: true,
+ localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)
+ ))
+ XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ role: kAXGroupRole as String,
+ title: "Forward",
+ label: nil,
+ help: nil,
+ value: nil,
+ genericTextSummary: nil,
+ hasPrimaryClickAction: true,
+ localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)
+ ))
+ XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ role: kAXGroupRole as String,
+ title: nil,
+ label: nil,
+ help: nil,
+ value: nil,
+ genericTextSummary: "Candidate details",
+ hasPrimaryClickAction: true,
+ localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)
+ ))
+ XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ role: kAXGroupRole as String,
+ title: nil,
+ label: nil,
+ help: nil,
+ value: nil,
+ genericTextSummary: nil,
+ hasPrimaryClickAction: false,
+ localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)
+ ))
+ XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ role: kAXGroupRole as String,
+ title: nil,
+ label: nil,
+ help: nil,
+ value: nil,
+ genericTextSummary: nil,
+ hasPrimaryClickAction: true,
+ localFrame: CGRect(x: 0, y: 0, width: 1920, height: 929)
+ ))
+ XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ role: kAXGroupRole as String,
+ title: nil,
+ label: nil,
+ help: nil,
+ value: nil,
+ genericTextSummary: nil,
+ hasPrimaryClickAction: true,
+ localFrame: CGRect(x: 0, y: 0, width: 15, height: 0)
+ ))
+ }
+
func testAccessibilityRendererOnlyMergesShortTextOnlySiblingRuns() {
XCTAssertTrue(shouldMergeTextOnlySiblings(["AgentSphere", "17:18", "好的,谢谢"]))
XCTAssertFalse(shouldMergeTextOnlySiblings(["日期", "时间", "2026年5月7日", "晚餐", "18:00-20:00"]))