Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

fix(snapshot): 保留文本摘要中的可点击节点边界

tisfeng67904945a497
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 9d683c6..dbef39e 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -63,7 +63,7 @@
- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。
- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。
-- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但如果匿名通用节点暴露 `AXPress` / `AXConfirm` / `AXOpen` 且 frame 有效、尺寸紧凑,会保留为带窗口相对 `Frame` 的 `button`,让 icon-only Web 控件仍能获得可区分的 `element_index`,同时继续过滤零尺寸或覆盖大面积页面的匿名点击容器。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。
+- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但通用节点中的 `AXPress` / `AXConfirm` / `AXOpen` 子节点会形成文本摘要边界,避免多个可点击选项被合并成一个 container。这类动作节点如果 frame 有效、尺寸紧凑,会保留为带窗口相对 `Frame` 的 `button`,并用短文本后代作为按钮摘要,让 icon-only 和文字 Web 控件都能获得可区分的 `element_index`,同时继续过滤零尺寸或覆盖大面积页面的通用点击容器。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。
- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。
- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。
- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。
diff --git a/docs/histories/2026-07/20260730-1431-preserve-generic-action-boundaries.md b/docs/histories/2026-07/20260730-1431-preserve-generic-action-boundaries.md
new file mode 100644
index 0000000..a635c22
--- /dev/null
+++ b/docs/histories/2026-07/20260730-1431-preserve-generic-action-boundaries.md
@@ -0,0 +1,27 @@
+## [2026-07-30 14:31] | Task: 保留通用节点中的动作边界
+
+### 🤖 Execution Context
+* **Agent ID**: `/root`
+* **Base Model**: GPT-5
+* **Runtime**: Codex desktop
+
+### 📥 User Query
+> 修复 Chrome 中 BOSS 直聘转发弹窗的 snapshot 将“站内同事”“转发至其他”“邮件转发”合并成一个 container、导致目标没有独立 `element_index` 的问题。
+
+### 🛠 Changes Overview
+**Scope:** `OpenComputerUseKit` macOS Accessibility snapshot renderer
+
+**Key Actions:**
+- **[保留动作边界]**: 通用文本容器摘要遇到带 `AXPress`、`AXConfirm` 或 `AXOpen` 的 `AXGroup` / `AXUnknown` 子节点时停止合并,避免父摘要吞掉可点击后代。
+- **[输出可操作按钮]**: 将具备有效紧凑 frame 的通用主动作节点渲染为 `button`,允许短文本后代直接成为按钮摘要。
+- **[控制树规模]**: 普通纯文本容器仍使用原有摘要压缩;零尺寸和大面积通用动作容器仍不会误标为紧凑按钮。
+- **[验证]**: 补充动作摘要边界和紧凑动作节点测试;完整 `swift test` 通过 3 项 StandaloneCursor 与 151 项 OpenComputerUseKit 测试(1 项显式 live test 跳过、0 失败),并用当前本地构建在真实 Chrome 弹窗中确认三个选项分别获得独立 button 和 Frame。
+
+### 🧠 Design Intent (Why)
+Chrome Web Accessibility 树会把文字选项暴露为带主点击动作的通用节点,再在其下放置静态文本。旧摘要逻辑只检查后代角色,不检查通用子节点的动作,因此会跨越真实交互边界,把多个选项合成一个不可精确操作的 container。本次修复只阻止摘要跨越通用主动作节点,不关闭普通文本压缩,以兼顾交互语义和节点预算。
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
+- `docs/histories/2026-07/20260730-1431-preserve-generic-action-boundaries.md`
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index 3caee02..9313756 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -96,8 +96,8 @@ private let windowVisibilityRecoveryDelay: TimeInterval = 0.7
private let axWebAreaRole = "AXWebArea"
private let axContentsAttribute = "AXContents"
private let axVisibleChildrenAttribute = "AXVisibleChildren"
-private let anonymousActionTargetMaxWidth: CGFloat = 240
-private let anonymousActionTargetMaxHeight: CGFloat = 120
+private let compactGenericActionTargetMaxWidth: CGFloat = 240
+private let compactGenericActionTargetMaxHeight: CGFloat = 120
public struct AppSnapshot {
public let app: RunningAppDescriptor
@@ -698,23 +698,31 @@ private struct TreeRenderer {
let axIdentifier = displayIdentifier(stringValue(of: root, attribute: kAXIdentifierAttribute))
let traits = summarizeTraits(of: root)
let actions = copyActions(root) ?? []
+ let exposesPrimaryClickAction = hasPrimaryClickAction(actions)
let prettyActions = meaningfulActions(actions, role: role)
let placeholder = placeholderValue(of: root, textLimit: context.textLimit)
let webAreaDepth = webAreaDepth(role: role, ancestors: ancestors)
let localFrame = resolveLocalFrame(of: root, windowBounds: context.windowBounds)
+ let rendersCompactGenericActionTarget = shouldRenderCompactGenericActionTarget(
+ role: role,
+ hasPrimaryClickAction: exposesPrimaryClickAction,
+ localFrame: localFrame
+ )
let rowTexts = role == kAXRowRole as String ? flattenedRowTexts(of: root, textLimit: context.textLimit) : []
let childElements = children(of: root)
let genericTextSummary = summarizedGenericText(
of: root,
role: role,
childElements: childElements,
- textLimit: context.textLimit
+ textLimit: context.textLimit,
+ minimumTextCount: rendersCompactGenericActionTarget ? 1 : 2
)
let summaryImageChildren = genericTextSummary == nil ? [] : summaryImageDescendants(of: root)
- let rendersSummaryAsChildren = shouldRenderGenericTextSummaryAsChildren(
- genericTextSummary,
- summaryImageCount: summaryImageChildren.count
- )
+ let rendersSummaryAsChildren = !rendersCompactGenericActionTarget
+ && shouldRenderGenericTextSummaryAsChildren(
+ genericTextSummary,
+ summaryImageCount: summaryImageChildren.count
+ )
let title = preferredDisplayTitle(
for: root,
role: role,
@@ -727,17 +735,6 @@ private struct TreeRenderer {
let linkText = role == "AXLink" ? markdownLinkText(for: root, title: title, label: label, value: value, textLimit: context.textLimit) : nil
let displayTitle = linkText ?? title
let inlineRowSummary = outlineRowSummary(for: root, role: role)
- let exposesPrimaryClickAction = hasPrimaryClickAction(actions)
- let rendersAnonymousActionTarget = shouldRenderAnonymousActionTarget(
- role: role,
- title: displayTitle,
- label: label,
- help: help,
- value: value,
- genericTextSummary: genericTextSummary,
- hasPrimaryClickAction: exposesPrimaryClickAction,
- localFrame: localFrame
- )
let hidesChildren = shouldSuppressChildren(
role: role,
title: displayTitle,
@@ -769,7 +766,7 @@ private struct TreeRenderer {
childCount: childElements.count,
genericTextSummary: genericTextSummary,
webAreaDepth: webAreaDepth,
- preservesAnonymousActionTarget: rendersAnonymousActionTarget
+ preservesCompactGenericActionTarget: rendersCompactGenericActionTarget
) {
for child in childElements {
render(child, depth: depth, ancestors: nextAncestors)
@@ -807,7 +804,7 @@ private struct TreeRenderer {
value: value,
precedingSegments: [labelSegment, helpSegment, urlSegment, identifierSegment, valueSegment]
)
- let frameSegment = rendersAnonymousActionTarget
+ let frameSegment = rendersCompactGenericActionTarget
? localFrame.map { " Frame: \($0.renderedLocalFrame)" } ?? ""
: ""
let actionsPrefix = shouldCommaSeparateActions(
@@ -817,7 +814,7 @@ private struct TreeRenderer {
segments: [labelSegment, helpSegment, urlSegment, identifierSegment, valueSegment, placeholderSegment]
) ? ", Secondary Actions: " : " Secondary Actions: "
let actionsSegment = prettyActions.isEmpty ? "" : "\(actionsPrefix)\(prettyActions.joined(separator: ", "))"
- let renderedRoleText = rendersAnonymousActionTarget ? "button" : roleText
+ let renderedRoleText = rendersCompactGenericActionTarget ? "button" : roleText
let linePrefix = renderedRoleText.isEmpty ? "\(index)" : "\(index) \(renderedRoleText)"
let lineBody = "\(linePrefix)\(traitsSegment)\(titleSegment)\(rowSummarySegment)\(labelSegment)\(helpSegment)\(urlSegment)\(identifierSegment)\(valueSegment)\(placeholderSegment)\(frameSegment)"
@@ -1444,14 +1441,14 @@ func shouldElideNode(
childCount: Int,
genericTextSummary: String? = nil,
webAreaDepth: Int? = nil,
- preservesAnonymousActionTarget: Bool = false
+ preservesCompactGenericActionTarget: Bool = false
) -> Bool {
let genericRoles = [kAXGroupRole as String, kAXUnknownRole as String]
guard genericRoles.contains(role) else {
return false
}
- if preservesAnonymousActionTarget {
+ if preservesCompactGenericActionTarget {
return false
}
@@ -1506,13 +1503,8 @@ func hasPrimaryClickAction(_ actions: [String]) -> Bool {
}
}
-func shouldRenderAnonymousActionTarget(
+func shouldRenderCompactGenericActionTarget(
role: String,
- title: String?,
- label: String?,
- help: String?,
- value: String?,
- genericTextSummary: String?,
hasPrimaryClickAction: Bool,
localFrame: CGRect?
) -> Bool {
@@ -1527,17 +1519,13 @@ func shouldRenderAnonymousActionTarget(
guard let localFrame,
localFrame.width > 0,
localFrame.height > 0,
- localFrame.width <= anonymousActionTargetMaxWidth,
- localFrame.height <= anonymousActionTargetMaxHeight
+ localFrame.width <= compactGenericActionTargetMaxWidth,
+ localFrame.height <= compactGenericActionTargetMaxHeight
else {
return false
}
- return title == nil
- && label == nil
- && help == nil
- && value == nil
- && genericTextSummary == nil
+ return true
}
private func shouldSuppressChildren(
@@ -1567,7 +1555,8 @@ private func summarizedGenericText(
of element: AXUIElement,
role: String,
childElements: [AXUIElement],
- textLimit: SnapshotTextLimit = .defaults
+ textLimit: SnapshotTextLimit = .defaults,
+ minimumTextCount: Int = 2
) -> String? {
guard role == kAXGroupRole as String || role == kAXUnknownRole as String else {
return nil
@@ -1582,7 +1571,7 @@ private func summarizedGenericText(
}
let texts = descendantTextsForSummary(of: element, textLimit: textLimit)
- guard texts.count >= 2 else {
+ guard texts.count >= minimumTextCount else {
return nil
}
@@ -1667,6 +1656,14 @@ private func isPlainGenericTextContainer(_ element: AXUIElement, children: [AXUI
}
if childRole == kAXGroupRole as String || childRole == kAXUnknownRole as String {
+ // Crossing this boundary would collapse the actionable child into its parent's text summary.
+ if isGenericPrimaryActionSummaryBoundary(
+ role: childRole,
+ actions: copyActions(child) ?? []
+ ) {
+ return false
+ }
+
guard depth < 3 else {
return false
}
@@ -1682,6 +1679,11 @@ private func isPlainGenericTextContainer(_ element: AXUIElement, children: [AXUI
return true
}
+func isGenericPrimaryActionSummaryBoundary(role: String, actions: [String]) -> Bool {
+ let genericRoles = [kAXGroupRole as String, kAXUnknownRole as String]
+ return genericRoles.contains(role) && hasPrimaryClickAction(actions)
+}
+
func displayRoleText(
baseRoleText: String,
role: String,
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 0e96087..e720978 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -971,7 +971,7 @@ final class OpenComputerUseKitTests: XCTestCase {
traits: [],
actions: [],
childCount: 0,
- preservesAnonymousActionTarget: true
+ preservesCompactGenericActionTarget: true
))
XCTAssertFalse(shouldElideNode(
role: kAXGroupRole as String,
@@ -1079,87 +1079,70 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertFalse(hasPrimaryClickAction(["AXShowMenu", "AXScrollToVisible", "AXRaise"]))
}
- func testAccessibilityRendererMarksAnonymousGenericClickTargetsAsButtons() {
- XCTAssertTrue(shouldRenderAnonymousActionTarget(
+ func testAccessibilityRendererTreatsGenericPrimaryActionsAsSummaryBoundaries() {
+ XCTAssertTrue(isGenericPrimaryActionSummaryBoundary(
+ role: kAXGroupRole as String,
+ actions: [kAXPressAction as String]
+ ))
+ XCTAssertTrue(isGenericPrimaryActionSummaryBoundary(
+ role: kAXUnknownRole as String,
+ actions: ["AXOpen"]
+ ))
+ XCTAssertFalse(isGenericPrimaryActionSummaryBoundary(
+ role: kAXGroupRole as String,
+ actions: ["AXShowMenu", "AXScrollToVisible"]
+ ))
+ XCTAssertFalse(isGenericPrimaryActionSummaryBoundary(
+ role: kAXButtonRole as String,
+ actions: [kAXPressAction as String]
+ ))
+ XCTAssertFalse(isGenericPrimaryActionSummaryBoundary(
+ role: "AXLink",
+ actions: [kAXPressAction as String]
+ ))
+ }
+
+ func testAccessibilityRendererMarksCompactGenericClickTargetsAsButtons() {
+ XCTAssertTrue(shouldRenderCompactGenericActionTarget(
role: kAXGroupRole as String,
- title: nil,
- label: nil,
- help: nil,
- value: nil,
- genericTextSummary: nil,
hasPrimaryClickAction: true,
localFrame: CGRect(x: 1455, y: 218, width: 15, height: 20)
))
- XCTAssertTrue(shouldRenderAnonymousActionTarget(
+ XCTAssertTrue(shouldRenderCompactGenericActionTarget(
role: kAXUnknownRole as String,
- title: nil,
- label: nil,
- help: nil,
- value: nil,
- genericTextSummary: nil,
hasPrimaryClickAction: true,
localFrame: CGRect(x: 1455, y: 245, width: 15, height: 20)
))
- XCTAssertFalse(shouldRenderAnonymousActionTarget(
- role: kAXButtonRole as String,
- title: nil,
- label: nil,
- help: nil,
- value: nil,
- genericTextSummary: nil,
- hasPrimaryClickAction: true,
- localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)
- ))
- XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ XCTAssertTrue(shouldRenderCompactGenericActionTarget(
role: kAXGroupRole as String,
- title: "Forward",
- label: nil,
- help: nil,
- value: nil,
- genericTextSummary: nil,
hasPrimaryClickAction: true,
- localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)
+ localFrame: CGRect(x: 10, y: 10, width: 240, height: 120)
))
- XCTAssertFalse(shouldRenderAnonymousActionTarget(
- role: kAXGroupRole as String,
- title: nil,
- label: nil,
- help: nil,
- value: nil,
- genericTextSummary: "Candidate details",
+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(
+ role: kAXButtonRole as String,
hasPrimaryClickAction: true,
localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)
))
- XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(
role: kAXGroupRole as String,
- title: nil,
- label: nil,
- help: nil,
- value: nil,
- genericTextSummary: nil,
hasPrimaryClickAction: false,
localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)
))
- XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(
role: kAXGroupRole as String,
- title: nil,
- label: nil,
- help: nil,
- value: nil,
- genericTextSummary: nil,
hasPrimaryClickAction: true,
localFrame: CGRect(x: 0, y: 0, width: 1920, height: 929)
))
- XCTAssertFalse(shouldRenderAnonymousActionTarget(
+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(
role: kAXGroupRole as String,
- title: nil,
- label: nil,
- help: nil,
- value: nil,
- genericTextSummary: nil,
hasPrimaryClickAction: true,
localFrame: CGRect(x: 0, y: 0, width: 15, height: 0)
))
+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(
+ role: kAXGroupRole as String,
+ hasPrimaryClickAction: true,
+ localFrame: nil
+ ))
}
func testAccessibilityRendererOnlyMergesShortTextOnlySiblingRuns() {