👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
fix(snapshot): 保留 Web 链接动作节点
tisfeng440807da1155
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex e4c400b..21412b8 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -63,7 +63,7 @@- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。-- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但通用节点中的 `AXPress` / `AXConfirm` / `AXOpen` 子节点会形成文本摘要边界,避免多个可点击选项被合并成一个 container。这类动作节点如果 frame 有效、尺寸紧凑,会保留为带窗口相对 `Frame` 的 `button`,并用短文本后代作为按钮摘要,让 icon-only 和文字 Web 控件都能获得可区分的 `element_index`,同时继续过滤零尺寸或覆盖大面积页面的通用点击容器。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。+- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但通用节点中的 `AXPress` / `AXConfirm` / `AXOpen` 子节点会形成文本摘要边界,避免多个可点击选项被合并成一个 container。这类动作节点如果 frame 有效、尺寸紧凑且不包含带 URL 的 `AXLink` 后代,会保留为带窗口相对 `Frame` 的 `button`,并用短文本后代作为按钮摘要,让 icon-only 和文字 Web 控件都能获得可区分的 `element_index`;包含带 URL 的 `AXLink` 后代时保留通用 wrapper 和链接子节点,避免导航链接被摘要吞掉。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。diff --git a/docs/histories/2026-08/20260828-1725-preserve-web-link-actions.md b/docs/histories/2026-08/20260828-1725-preserve-web-link-actions.mdnew file mode 100644index 0000000..1cbb60f--- /dev/null+++ b/docs/histories/2026-08/20260828-1725-preserve-web-link-actions.md@@ -0,0 +1,26 @@+## [2026-08-28 17:25] | Task: 修复 Web 链接动作节点解析++### 🤖 Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `Codex desktop`++### 📥 User Query+> 执行修复:BOSS 页面“职位管理”被解析成没有独立 link 的 button,导致无法点击跳转。++### 🛠 Changes Overview+**Scope:** `OpenComputerUseKit` macOS Accessibility snapshot renderer++**Key Actions:**+- **[保留导航语义]**: 通用动作节点包含带 URL 的 `AXLink` 后代时,不再提升为 compact `button`,也不再合并其文本摘要。+- **[保持通用按钮能力]**: 没有 URL 链接后代的紧凑 `AXGroup` / `AXUnknown` 仍按原规则保留为可操作 `button`。+- **[补回归覆盖]**: 增加带链接后代的 compact action 节点测试,防止 BOSS 导航链接再次被父摘要吞掉。++### 🧠 Design Intent (Why)+Chrome/BOSS 的 Accessibility tree 可能同时暴露通用父动作和真实 URL 链接子节点。父动作适合 icon-only 控件,但对导航链接必须保留子节点的独立 `element_index`,否则 AX 点击可能只触发父 wrapper 而不发生 URL 导航。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-08/20260828-1725-preserve-web-link-actions.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex 9313756..604213c 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -703,20 +703,33 @@ private struct TreeRenderer {let placeholder = placeholderValue(of: root, textLimit: context.textLimit)let webAreaDepth = webAreaDepth(role: role, ancestors: ancestors)let localFrame = resolveLocalFrame(of: root, windowBounds: context.windowBounds)- let rendersCompactGenericActionTarget = shouldRenderCompactGenericActionTarget(- role: role,- hasPrimaryClickAction: exposesPrimaryClickAction,- localFrame: localFrame- )let rowTexts = role == kAXRowRole as String ? flattenedRowTexts(of: root, textLimit: context.textLimit) : []let childElements = children(of: root)- let genericTextSummary = summarizedGenericText(- of: root,+ let hasActionableLinkDescendant =+ (role == kAXGroupRole as String || role == kAXUnknownRole as String)+ && exposesPrimaryClickAction+ && containsActionableLinkDescendant(+ in: childElements,+ textLimit: context.textLimit+ )+ let rendersCompactGenericActionTarget = shouldRenderCompactGenericActionTarget(role: role,- childElements: childElements,- textLimit: context.textLimit,- minimumTextCount: rendersCompactGenericActionTarget ? 1 : 2+ hasPrimaryClickAction: exposesPrimaryClickAction,+ localFrame: localFrame,+ hasActionableLinkDescendant: hasActionableLinkDescendant)+ let genericTextSummary: String?+ if hasActionableLinkDescendant {+ genericTextSummary = nil+ } else {+ genericTextSummary = summarizedGenericText(+ of: root,+ role: role,+ childElements: childElements,+ textLimit: context.textLimit,+ minimumTextCount: rendersCompactGenericActionTarget ? 1 : 2+ )+ }let summaryImageChildren = genericTextSummary == nil ? [] : summaryImageDescendants(of: root)let rendersSummaryAsChildren = !rendersCompactGenericActionTarget&& shouldRenderGenericTextSummaryAsChildren(@@ -936,6 +949,42 @@ private struct TreeRenderer {return children}++ private func containsActionableLinkDescendant(+ in elements: [AXUIElement],+ textLimit: SnapshotTextLimit,+ ancestors: [AXUIElement] = [],+ depth: Int = 0+ ) -> Bool {+ guard depth < 8 else {+ return false+ }++ for element in elements {+ guard !ancestors.contains(where: { CFEqual($0, element) }) else {+ continue+ }++ let role = stringValue(of: element, attribute: kAXRoleAttribute) ?? ""+ if role == "AXLink",+ let url = urlValue(of: element, attribute: kAXURLAttribute, textLimit: textLimit),+ !url.isEmpty+ {+ return true+ }++ if containsActionableLinkDescendant(+ in: children(of: element),+ textLimit: textLimit,+ ancestors: ancestors + [element],+ depth: depth + 1+ ) {+ return true+ }+ }++ return false+ }}func childTraversalAttributes(role: String?, hasRows: Bool, hasVisibleChildren: Bool) -> [String] {@@ -1506,12 +1555,19 @@ func hasPrimaryClickAction(_ actions: [String]) -> Bool {func shouldRenderCompactGenericActionTarget(role: String,hasPrimaryClickAction: Bool,- localFrame: CGRect?+ localFrame: CGRect?,+ hasActionableLinkDescendant: Bool = false) -> Bool {guard hasPrimaryClickAction else {return false}+ // A URL-bearing AXLink is the navigation target. Do not hide it behind a+ // generic action wrapper that happens to expose AXPress as well.+ guard !hasActionableLinkDescendant else {+ return false+ }+guard role == kAXGroupRole as String || role == kAXUnknownRole as String else {return false}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 4ab7d21..75c6ed6 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -1118,6 +1118,12 @@ final class OpenComputerUseKitTests: XCTestCase {hasPrimaryClickAction: true,localFrame: CGRect(x: 10, y: 10, width: 240, height: 120)))+ XCTAssertFalse(shouldRenderCompactGenericActionTarget(+ role: kAXGroupRole as String,+ hasPrimaryClickAction: true,+ localFrame: CGRect(x: 10, y: 10, width: 117, height: 35),+ hasActionableLinkDescendant: true+ ))XCTAssertFalse(shouldRenderCompactGenericActionTarget(role: kAXButtonRole as String,hasPrimaryClickAction: true,