👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
fix(snapshot): 保留文本摘要中的可点击节点边界
tisfeng67904945a497
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 9d683c6..dbef39e 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -63,7 +63,7 @@- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。-- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但如果匿名通用节点暴露 `AXPress` / `AXConfirm` / `AXOpen` 且 frame 有效、尺寸紧凑,会保留为带窗口相对 `Frame` 的 `button`,让 icon-only Web 控件仍能获得可区分的 `element_index`,同时继续过滤零尺寸或覆盖大面积页面的匿名点击容器。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。+- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但通用节点中的 `AXPress` / `AXConfirm` / `AXOpen` 子节点会形成文本摘要边界,避免多个可点击选项被合并成一个 container。这类动作节点如果 frame 有效、尺寸紧凑,会保留为带窗口相对 `Frame` 的 `button`,并用短文本后代作为按钮摘要,让 icon-only 和文字 Web 控件都能获得可区分的 `element_index`,同时继续过滤零尺寸或覆盖大面积页面的通用点击容器。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。diff --git a/docs/histories/2026-07/20260730-1431-preserve-generic-action-boundaries.md b/docs/histories/2026-07/20260730-1431-preserve-generic-action-boundaries.mdnew file mode 100644index 0000000..a635c22--- /dev/null+++ b/docs/histories/2026-07/20260730-1431-preserve-generic-action-boundaries.md@@ -0,0 +1,27 @@+## [2026-07-30 14:31] | Task: 保留通用节点中的动作边界++### 🤖 Execution Context+* **Agent ID**: `/root`+* **Base Model**: GPT-5+* **Runtime**: Codex desktop++### 📥 User Query+> 修复 Chrome 中 BOSS 直聘转发弹窗的 snapshot 将“站内同事”“转发至其他”“邮件转发”合并成一个 container、导致目标没有独立 `element_index` 的问题。++### 🛠 Changes Overview+**Scope:** `OpenComputerUseKit` macOS Accessibility snapshot renderer++**Key Actions:**+- **[保留动作边界]**: 通用文本容器摘要遇到带 `AXPress`、`AXConfirm` 或 `AXOpen` 的 `AXGroup` / `AXUnknown` 子节点时停止合并,避免父摘要吞掉可点击后代。+- **[输出可操作按钮]**: 将具备有效紧凑 frame 的通用主动作节点渲染为 `button`,允许短文本后代直接成为按钮摘要。+- **[控制树规模]**: 普通纯文本容器仍使用原有摘要压缩;零尺寸和大面积通用动作容器仍不会误标为紧凑按钮。+- **[验证]**: 补充动作摘要边界和紧凑动作节点测试;完整 `swift test` 通过 3 项 StandaloneCursor 与 151 项 OpenComputerUseKit 测试(1 项显式 live test 跳过、0 失败),并用当前本地构建在真实 Chrome 弹窗中确认三个选项分别获得独立 button 和 Frame。++### 🧠 Design Intent (Why)+Chrome Web Accessibility 树会把文字选项暴露为带主点击动作的通用节点,再在其下放置静态文本。旧摘要逻辑只检查后代角色,不检查通用子节点的动作,因此会跨越真实交互边界,把多个选项合成一个不可精确操作的 container。本次修复只阻止摘要跨越通用主动作节点,不关闭普通文本压缩,以兼顾交互语义和节点预算。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-07/20260730-1431-preserve-generic-action-boundaries.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex 3caee02..9313756 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -96,8 +96,8 @@ private let windowVisibilityRecoveryDelay: TimeInterval = 0.7private let axWebAreaRole = "AXWebArea"private let axContentsAttribute = "AXContents"private let axVisibleChildrenAttribute = "AXVisibleChildren"-private let anonymousActionTargetMaxWidth: CGFloat = 240-private let anonymousActionTargetMaxHeight: CGFloat = 120+private let compactGenericActionTargetMaxWidth: CGFloat = 240+private let compactGenericActionTargetMaxHeight: CGFloat = 120public struct AppSnapshot {public let app: RunningAppDescriptor@@ -698,23 +698,31 @@ private struct TreeRenderer {let axIdentifier = displayIdentifier(stringValue(of: root, attribute: kAXIdentifierAttribute))let traits = summarizeTraits(of: root)let actions = copyActions(root) ?? []+ let exposesPrimaryClickAction = hasPrimaryClickAction(actions)let prettyActions = meaningfulActions(actions, role: role)let placeholder = placeholderValue(of: root, textLimit: context.textLimit)let webAreaDepth = webAreaDepth(role: role, ancestors: ancestors)let localFrame = resolveLocalFrame(of: root, windowBounds: context.windowBounds)+ let rendersCompactGenericActionTarget = shouldRenderCompactGenericActionTarget(+ role: role,+ hasPrimaryClickAction: exposesPrimaryClickAction,+ localFrame: localFrame+ )let rowTexts = role == kAXRowRole as String ? flattenedRowTexts(of: root, textLimit: context.textLimit) : []let childElements = children(of: root)let genericTextSummary = summarizedGenericText(of: root,role: role,childElements: childElements,- textLimit: context.textLimit+ textLimit: context.textLimit,+ minimumTextCount: rendersCompactGenericActionTarget ? 1 : 2)let summaryImageChildren = genericTextSummary == nil ? [] : summaryImageDescendants(of: root)- let rendersSummaryAsChildren = shouldRenderGenericTextSummaryAsChildren(- genericTextSummary,- summaryImageCount: summaryImageChildren.count- )+ let rendersSummaryAsChildren = !rendersCompactGenericActionTarget+ && shouldRenderGenericTextSummaryAsChildren(+ genericTextSummary,+ summaryImageCount: summaryImageChildren.count+ )let title = preferredDisplayTitle(for: root,role: role,@@ -727,17 +735,6 @@ private struct TreeRenderer {let linkText = role == "AXLink" ? markdownLinkText(for: root, title: title, label: label, value: value, textLimit: context.textLimit) : nillet displayTitle = linkText ?? titlelet inlineRowSummary = outlineRowSummary(for: root, role: role)- let exposesPrimaryClickAction = hasPrimaryClickAction(actions)- let rendersAnonymousActionTarget = shouldRenderAnonymousActionTarget(- role: role,- title: displayTitle,- label: label,- help: help,- value: value,- genericTextSummary: genericTextSummary,- hasPrimaryClickAction: exposesPrimaryClickAction,- localFrame: localFrame- )let hidesChildren = shouldSuppressChildren(role: role,title: displayTitle,@@ -769,7 +766,7 @@ private struct TreeRenderer {childCount: childElements.count,genericTextSummary: genericTextSummary,webAreaDepth: webAreaDepth,- preservesAnonymousActionTarget: rendersAnonymousActionTarget+ preservesCompactGenericActionTarget: rendersCompactGenericActionTarget) {for child in childElements {render(child, depth: depth, ancestors: nextAncestors)@@ -807,7 +804,7 @@ private struct TreeRenderer {value: value,precedingSegments: [labelSegment, helpSegment, urlSegment, identifierSegment, valueSegment])- let frameSegment = rendersAnonymousActionTarget+ let frameSegment = rendersCompactGenericActionTarget? localFrame.map { " Frame: \($0.renderedLocalFrame)" } ?? "": ""let actionsPrefix = shouldCommaSeparateActions(@@ -817,7 +814,7 @@ private struct TreeRenderer {segments: [labelSegment, helpSegment, urlSegment, identifierSegment, valueSegment, placeholderSegment]) ? ", Secondary Actions: " : " Secondary Actions: "let actionsSegment = prettyActions.isEmpty ? "" : "\(actionsPrefix)\(prettyActions.joined(separator: ", "))"- let renderedRoleText = rendersAnonymousActionTarget ? "button" : roleText+ let renderedRoleText = rendersCompactGenericActionTarget ? "button" : roleTextlet linePrefix = renderedRoleText.isEmpty ? "\(index)" : "\(index) \(renderedRoleText)"let lineBody = "\(linePrefix)\(traitsSegment)\(titleSegment)\(rowSummarySegment)\(labelSegment)\(helpSegment)\(urlSegment)\(identifierSegment)\(valueSegment)\(placeholderSegment)\(frameSegment)"@@ -1444,14 +1441,14 @@ func shouldElideNode(childCount: Int,genericTextSummary: String? = nil,webAreaDepth: Int? = nil,- preservesAnonymousActionTarget: Bool = false+ preservesCompactGenericActionTarget: Bool = false) -> Bool {let genericRoles = [kAXGroupRole as String, kAXUnknownRole as String]guard genericRoles.contains(role) else {return false}- if preservesAnonymousActionTarget {+ if preservesCompactGenericActionTarget {return false}@@ -1506,13 +1503,8 @@ func hasPrimaryClickAction(_ actions: [String]) -> Bool {}}-func shouldRenderAnonymousActionTarget(+func shouldRenderCompactGenericActionTarget(role: String,- title: String?,- label: String?,- help: String?,- value: String?,- genericTextSummary: String?,hasPrimaryClickAction: Bool,localFrame: CGRect?) -> Bool {@@ -1527,17 +1519,13 @@ func shouldRenderAnonymousActionTarget(guard let localFrame,localFrame.width > 0,localFrame.height > 0,- localFrame.width <= anonymousActionTargetMaxWidth,- localFrame.height <= anonymousActionTargetMaxHeight+ localFrame.width <= compactGenericActionTargetMaxWidth,+ localFrame.height <= compactGenericActionTargetMaxHeightelse {return false}- return title == nil- && label == nil- && help == nil- && value == nil- && genericTextSummary == nil+ return true}private func shouldSuppressChildren(@@ -1567,7 +1555,8 @@ private func summarizedGenericText(of element: AXUIElement,role: String,childElements: [AXUIElement],- textLimit: SnapshotTextLimit = .defaults+ textLimit: SnapshotTextLimit = .defaults,+ minimumTextCount: Int = 2) -> String? {guard role == kAXGroupRole as String || role == kAXUnknownRole as String else {return nil@@ -1582,7 +1571,7 @@ private func summarizedGenericText(}let texts = descendantTextsForSummary(of: element, textLimit: textLimit)- guard texts.count >= 2 else {+ guard texts.count >= minimumTextCount else {return nil}@@ -1667,6 +1656,14 @@ private func isPlainGenericTextContainer(_ element: AXUIElement, children: [AXUI}if childRole == kAXGroupRole as String || childRole == kAXUnknownRole as String {+ // Crossing this boundary would collapse the actionable child into its parent's text summary.+ if isGenericPrimaryActionSummaryBoundary(+ role: childRole,+ actions: copyActions(child) ?? []+ ) {+ return false+ }+guard depth < 3 else {return false}@@ -1682,6 +1679,11 @@ private func isPlainGenericTextContainer(_ element: AXUIElement, children: [AXUIreturn true}+func isGenericPrimaryActionSummaryBoundary(role: String, actions: [String]) -> Bool {+ let genericRoles = [kAXGroupRole as String, kAXUnknownRole as String]+ return genericRoles.contains(role) && hasPrimaryClickAction(actions)+}+func displayRoleText(baseRoleText: String,role: String,diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 0e96087..e720978 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -971,7 +971,7 @@ final class OpenComputerUseKitTests: XCTestCase {traits: [],actions: [],childCount: 0,- preservesAnonymousActionTarget: true+ preservesCompactGenericActionTarget: true))XCTAssertFalse(shouldElideNode(role: kAXGroupRole as String,@@ -1079,87 +1079,70 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertFalse(hasPrimaryClickAction(["AXShowMenu", "AXScrollToVisible", "AXRaise"]))}- func testAccessibilityRendererMarksAnonymousGenericClickTargetsAsButtons() {- XCTAssertTrue(shouldRenderAnonymousActionTarget(+ func testAccessibilityRendererTreatsGenericPrimaryActionsAsSummaryBoundaries() {+ XCTAssertTrue(isGenericPrimaryActionSummaryBoundary(+ role: kAXGroupRole as String,+ actions: [kAXPressAction as String]+ ))+ XCTAssertTrue(isGenericPrimaryActionSummaryBoundary(+ role: kAXUnknownRole as String,+ actions: ["AXOpen"]+ ))+ XCTAssertFalse(isGenericPrimaryActionSummaryBoundary(+ role: kAXGroupRole as String,+ actions: ["AXShowMenu", "AXScrollToVisible"]+ ))+ XCTAssertFalse(isGenericPrimaryActionSummaryBoundary(+ role: kAXButtonRole as String,+ actions: [kAXPressAction as String]+ ))+ XCTAssertFalse(isGenericPrimaryActionSummaryBoundary(+ role: "AXLink",+ actions: [kAXPressAction as String]+ ))+ }++ func testAccessibilityRendererMarksCompactGenericClickTargetsAsButtons() {+ XCTAssertTrue(shouldRenderCompactGenericActionTarget(role: kAXGroupRole as String,- title: nil,- label: nil,- help: nil,- value: nil,- genericTextSummary: nil,hasPrimaryClickAction: true,localFrame: CGRect(x: 1455, y: 218, width: 15, height: 20)))- XCTAssertTrue(shouldRenderAnonymousActionTarget(+ XCTAssertTrue(shouldRenderCompactGenericActionTarget(role: kAXUnknownRole as String,- title: nil,- label: nil,- help: nil,- value: nil,- genericTextSummary: nil,hasPrimaryClickAction: true,localFrame: CGRect(x: 1455, y: 245, width: 15, height: 20)))- XCTAssertFalse(shouldRenderAnonymousActionTarget(- role: kAXButtonRole as String,- title: nil,- label: nil,- help: nil,- value: nil,- genericTextSummary: nil,- hasPrimaryClickAction: true,- localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)- ))- XCTAssertFalse(shouldRenderAnonymousActionTarget(+ XCTAssertTrue(shouldRenderCompactGenericActionTarget(role: kAXGroupRole as String,- title: "Forward",- label: nil,- help: nil,- value: nil,- genericTextSummary: nil,hasPrimaryClickAction: true,- localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)+ localFrame: CGRect(x: 10, y: 10, width: 240, height: 120)))- XCTAssertFalse(shouldRenderAnonymousActionTarget(- role: kAXGroupRole as String,- title: nil,- label: nil,- help: nil,- value: nil,- genericTextSummary: "Candidate details",+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(+ role: kAXButtonRole as String,hasPrimaryClickAction: true,localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)))- XCTAssertFalse(shouldRenderAnonymousActionTarget(+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(role: kAXGroupRole as String,- title: nil,- label: nil,- help: nil,- value: nil,- genericTextSummary: nil,hasPrimaryClickAction: false,localFrame: CGRect(x: 10, y: 10, width: 32, height: 32)))- XCTAssertFalse(shouldRenderAnonymousActionTarget(+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(role: kAXGroupRole as String,- title: nil,- label: nil,- help: nil,- value: nil,- genericTextSummary: nil,hasPrimaryClickAction: true,localFrame: CGRect(x: 0, y: 0, width: 1920, height: 929)))- XCTAssertFalse(shouldRenderAnonymousActionTarget(+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(role: kAXGroupRole as String,- title: nil,- label: nil,- help: nil,- value: nil,- genericTextSummary: nil,hasPrimaryClickAction: true,localFrame: CGRect(x: 0, y: 0, width: 15, height: 0)))+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(+ role: kAXGroupRole as String,+ hasPrimaryClickAction: true,+ localFrame: nil+ ))}func testAccessibilityRendererOnlyMergesShortTextOnlySiblingRuns() {