👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
accessibility: preserve secondary-action mappings
Aikins Laryea9e9dcc2c09f0
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex ebff33f..14f78ad 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -76,7 +76,7 @@- overlay 不再依赖临时 `terminal settle` 补丁来修尾;主线现在统一改成“路径层给目标点,visual dynamics 层给可见姿态”的双层模型,所以 move 末段、pulse 和 idle 共用同一套状态,不会再出现 endpoint 锁住后只剩原地翻角的收尾。- overlay 的渲染输入也从单一 `rotation` 扩展成 `rotation + cursorBodyOffset + fogOffset + fogScale`,让速度滞后能真正体现在画面上,而不是只存在于主循环内部状态;其中 `rotation` 现在按二进制里 `SoftwareCursorStyle.angle + CursorView._animatedAngleOffsetDegrees` 的分层去近似,不再把“跟随运动方向的主朝向”和“小幅 wiggle offset”压成同一个受限小角度。- 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略:- - `perform_secondary_action` 只执行目标元素已经暴露出来的 AX action;无效 action 返回官方风格的 `... is not a valid secondary action for ...`,fixture 的 `Raise` 路径也不再为了测试去准备全局物理指针输入+ - `perform_secondary_action` 只执行目标元素已经暴露出来的 AX action;Safari 这类 AppKit custom action descriptor(例如 `Name:close tab ...`)会显示为短名称并准确映射回过滤后的原始 AX action,避免 raw/rendered action 错配;无效或歧义 action 返回官方风格的 `... is not a valid secondary action for ...`,fixture 的 `Raise` 路径也不再为了测试去准备全局物理指针输入- `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口- `click.click_method` 是开源版的可选扩展,支持 `auto`(默认)、`accessibility`、`app_post`、`sky_click` 和 `global`。未传参数时继续使用原有自动路由;显式模式不会静默 fallback 到其他实现。`accessibility` 只接受 `element_index`;其余 mouse 路径可以使用 `element_index` 的计算落点或原始 `x/y` 坐标。- element-targeted `click` 的 `auto` 左键路径会先试原生列表的 `AXSelectedChildren` 选择,再试 `AXPress` / `AXConfirm` / `AXOpen` 这类真正语义化的激活动作;如果目标本身不可点,还会继续尝试其子孙 AX 元素(例如 Finder sidebar row 下面暴露 `AXOpen` 的 cell)和命中点附近的 AX hit-test 结果,最后进入现有 non-AX 分支:未开启全局指针环境变量时使用 `postToPid` 定向鼠标事件,开启时直接使用全局 HID 事件。`AXRaise` / `kAXMainAttribute` / `kAXFocusedAttribute` 这类 activation-only fallback 只允许窗口级元素使用,避免普通静态文本或容器把“获得焦点”误报成“点击已处理”;`click_count > 1` 也会优先重复可用的 AX action。diff --git a/docs/histories/2026-09/20260906-1200-secondary-action-correctness.md b/docs/histories/2026-09/20260906-1200-secondary-action-correctness.mdnew file mode 100644index 0000000..3749bd7--- /dev/null+++ b/docs/histories/2026-09/20260906-1200-secondary-action-correctness.md@@ -0,0 +1,23 @@+## [2026-09-06 12:00] | Task: 修正 secondary action 映射++### 📥 User Query+> 独立提取 macOS secondary-action correctness 变更,支持 Safari custom action description,并修复过滤后的 raw/display action 错配。++### 🛠 Changes Overview+**Scope:** OpenComputerUseKit macOS accessibility snapshot 与 action execution++**Key Actions:**+- **动作渲染**: 解析 AppKit `Name:...` action descriptor,并输出有意义的短名称。+- **名称冲突**: 短名称冲突时显示原始 descriptor,保证每个输出 selector 都可唯一映射回对应动作。+- **动作执行**: 保持原始 AX action 精确匹配的兼容性;短名称基于同一组 role-filtered raw actions 匹配,拒绝歧义短名称。+- **回归测试**: 覆盖 Safari close-tab、过滤对齐、无效 descriptor 与歧义名称。++### 🧠 Design Intent (Why)+*渲染与执行必须共享过滤和命名语义,否则压缩后的 action 数组会与原始 action 索引错位并执行错误动作。*++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/releases/feature-release-notes.md`diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.mdindex fffc323..892ffc3 100644--- a/docs/releases/feature-release-notes.md+++ b/docs/releases/feature-release-notes.md@@ -5,6 +5,7 @@| 日期 | 功能域 | 用户价值 | 变更摘要 || --- | --- | --- | --- || 2026-09-06 | macOS 应用名称解析 | 按名称定位 Safari 等应用时优先匹配真实应用,避免同名后台 helper 抢占匹配结果。 | 按普通应用名称、普通应用 executable、后台应用名称、后台 executable 分级匹配;同级保持原顺序,bundle identifier 和安全过滤规则不变。 |+| 2026-09-06 | macOS secondary action | Agent 可以通过 Safari 暴露的 `close tab` secondary action 可靠关闭标签页,不会因内部动作被过滤而误执行另一项动作。 | macOS renderer 将 `Name:close tab ...` 形式的 AppKit custom action descriptor 显示为短名称,executor 从同一组过滤后的 raw actions 做等价名称匹配,并拒绝歧义匹配。 || 2026-09-01 | 内嵌 App Agent Socket 隔离 | 使用内嵌 OCU 的宿主不会再与用户全局 OCU 争用同一个 App Agent Socket,避免一个实例意外终止或替换另一个实例。 | 发布 `0.3.3`,为显式配置 namespace 的宿主生成确定性、短且不泄露原值的 Socket 文件名;未配置时继续兼容旧路径。 |## 2026-08diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex 604213c..a83b367 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -9,6 +9,7 @@ final class ElementRecord {let identifier: String?let element: AXUIElement?let localFrame: CGRect?+ let role: String?let rawActions: [String]let prettyActions: [String]let isSyntheticText: Bool@@ -18,6 +19,7 @@ final class ElementRecord {identifier: String?,element: AXUIElement?,localFrame: CGRect?,+ role: String? = nil,rawActions: [String],prettyActions: [String],isSyntheticText: Bool = false@@ -26,6 +28,7 @@ final class ElementRecord {self.identifier = identifierself.element = elementself.localFrame = localFrame+ self.role = roleself.rawActions = rawActionsself.prettyActions = prettyActionsself.isSyntheticText = isSyntheticText@@ -383,6 +386,7 @@ enum SnapshotBuilder {identifier: element.identifier,element: nil,localFrame: element.frame.cgRect,+ role: element.role,rawActions: element.actions,prettyActions: element.actions)@@ -838,6 +842,7 @@ private struct TreeRenderer {identifier: axIdentifier,element: root,localFrame: localFrame,+ role: role,rawActions: actions,prettyActions: prettyActions)@@ -1808,6 +1813,17 @@ private func roleDescription(of element: AXUIElement, role: String, subrole: Str}func meaningfulActions(_ values: [String], role: String) -> [String] {+ let rawActions = meaningfulRawActions(values, role: role)+ let names = rawActions.map(secondaryActionDisplayName(_:))+ return names.indices.map { index in+ let collides = names.indices.contains { other in+ other != index && secondaryActionNamesEquivalent(names[index], names[other])+ }+ return collides ? rawActions[index] : names[index]+ }+}++func meaningfulRawActions(_ values: [String], role: String) -> [String] {values.filter {var ignored = [@@ -1841,10 +1857,13 @@ func meaningfulActions(_ values: [String], role: String) -> [String] {return true}- .map(prettyActionName(_:))}-private func prettyActionName(_ value: String) -> String {+func secondaryActionDisplayName(_ value: String) -> String {+ if let name = accessibilityActionDescriptionName(value) {+ return name+ }+if value == "AXZoomWindow" {return "zoom the window"}@@ -1854,6 +1873,90 @@ private func prettyActionName(_ value: String) -> String {return splitCamelCase(withoutPage)}+func secondaryActionNamesEquivalent(_ lhs: String, _ rhs: String) -> Bool {+ !normalizedSecondaryActionCandidates(lhs).isDisjoint(with: normalizedSecondaryActionCandidates(rhs))+}++func accessibilityActionDescriptionName(_ value: String) -> String? {+ guard let nameStart = actionDescriptionValueStart(label: "name", in: value) else {+ return nil+ }++ let nameEnd = ["target", "selector", "button clicked"]+ .compactMap { actionDescriptionLabelStart(label: $0, in: value, from: nameStart) }+ .min() ?? value.endIndex+ let name = value[nameStart..<nameEnd].trimmingCharacters(in: .whitespacesAndNewlines)+ return name.isEmpty ? nil : name+}++private func actionDescriptionValueStart(label: String, in value: String) -> String.Index? {+ var searchStart = value.startIndex+ while let range = value.range(of: label, options: .caseInsensitive, range: searchStart..<value.endIndex) {+ if range.lowerBound != value.startIndex, !value[value.index(before: range.lowerBound)].isWhitespace {+ searchStart = range.upperBound+ continue+ }++ var cursor = range.upperBound+ while cursor < value.endIndex, value[cursor].isWhitespace {+ cursor = value.index(after: cursor)+ }+ guard cursor < value.endIndex, value[cursor] == ":" else {+ searchStart = range.upperBound+ continue+ }+ cursor = value.index(after: cursor)+ while cursor < value.endIndex, value[cursor].isWhitespace {+ cursor = value.index(after: cursor)+ }+ return cursor+ }+ return nil+}++private func actionDescriptionLabelStart(label: String, in value: String, from start: String.Index) -> String.Index? {+ var searchStart = start+ while let range = value.range(of: label, options: .caseInsensitive, range: searchStart..<value.endIndex) {+ if range.lowerBound != value.startIndex, !value[value.index(before: range.lowerBound)].isWhitespace {+ searchStart = range.upperBound+ continue+ }++ var cursor = range.upperBound+ while cursor < value.endIndex, value[cursor].isWhitespace {+ cursor = value.index(after: cursor)+ }+ guard cursor < value.endIndex, value[cursor] == ":" else {+ searchStart = range.upperBound+ continue+ }+ return range.lowerBound+ }+ return nil+}++private func normalizedSecondaryActionCandidates(_ value: String) -> Set<String> {+ var candidates = Set<String>()+ let normalized = normalizedSecondaryActionName(value)+ if !normalized.isEmpty {+ candidates.insert(normalized)+ }+ if let descriptionName = accessibilityActionDescriptionName(value) {+ candidates.insert(normalizedSecondaryActionName(descriptionName))+ }+ return candidates+}++private func normalizedSecondaryActionName(_ value: String) -> String {+ value+ .trimmingCharacters(in: .whitespacesAndNewlines)+ .replacingOccurrences(of: "_", with: " ")+ .replacingOccurrences(of: "-", with: " ")+ .split(whereSeparator: { $0.isWhitespace })+ .joined(separator: " ")+ .lowercased()+}+private func humanizeAXToken(_ value: String) -> String {let stripped = value.hasPrefix("AX") ? String(value.dropFirst(2)) : valuereturn splitCamelCase(stripped).lowercased()diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex afdce9b..1fbc48a 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -832,16 +832,15 @@ public final class ComputerUseService {return record}- private func matchingAction(requested: String, record: ElementRecord) -> String? {+ func matchingAction(requested: String, record: ElementRecord) -> String? {if let exact = record.rawActions.first(where: { $0.caseInsensitiveCompare(requested) == .orderedSame }) {return exact}-- if let pretty = zip(record.rawActions, record.prettyActions).first(where: { $0.1.caseInsensitiveCompare(requested) == .orderedSame }) {- return pretty.0+ let visibleActions = record.role.map { meaningfulRawActions(record.rawActions, role: $0) } ?? record.rawActions+ let matches = visibleActions.filter { rawAction in+ secondaryActionNamesEquivalent(requested, secondaryActionDisplayName(rawAction))}-- return nil+ return matches.count == 1 ? matches[0] : nil}private func invalidSecondaryActionMessage(action: String, record: ElementRecord) -> String {diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex bf28855..6c4b224 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -1138,6 +1138,59 @@ final class OpenComputerUseKitTests: XCTestCase {))}+ func testAccessibilityRendererUsesSafariCustomActionDescriptionName() {+ XCTAssertEqual(+ meaningfulActions(+ ["Name:close tab Target:SafariTab Selector:_close Button Clicked:"],+ role: kAXButtonRole as String+ ),+ ["close tab"]+ )+ XCTAssertNil(accessibilityActionDescriptionName("Namespace:close tab Target:SafariTab"))+ }++ func testSecondaryActionMatchingKeepsFilteredRawActionsAligned() {+ let service = ComputerUseService()+ let closeTab = "Name:close tab Target:SafariTab Selector:_close Button Clicked:"+ let record = ElementRecord(+ index: 48,+ identifier: nil,+ element: nil,+ localFrame: nil,+ role: kAXButtonRole as String,+ rawActions: [kAXPressAction as String, closeTab],+ prettyActions: ["close tab"]+ )++ XCTAssertEqual(service.matchingAction(requested: "close-tab", record: record), closeTab)+ XCTAssertEqual(service.matchingAction(requested: kAXPressAction as String, record: record), kAXPressAction as String)+ XCTAssertNil(service.matchingAction(requested: "Press", record: record))+ }++ func testSecondaryActionMatchingRejectsAmbiguousDisplayNames() {+ let service = ComputerUseService()+ let record = ElementRecord(+ index: 49,+ identifier: nil,+ element: nil,+ localFrame: nil,+ role: kAXButtonRole as String,+ rawActions: [+ "Name:close tab Target:FirstTab Selector:_close:",+ "Name:close tab Target:SecondTab Selector:_close:",+ ],+ prettyActions: ["close tab", "close tab"]+ )++ XCTAssertNil(service.matchingAction(requested: "close tab", record: record))+ XCTAssertEqual(service.matchingAction(requested: record.rawActions[1], record: record), record.rawActions[1])+ let emitted = meaningfulActions(record.rawActions, role: kAXButtonRole as String)+ XCTAssertEqual(emitted, record.rawActions)+ for (selector, rawAction) in zip(emitted, record.rawActions) {+ XCTAssertEqual(service.matchingAction(requested: selector, record: record), rawAction)+ }+ }+func testAccessibilityRendererMarksCompactGenericClickTargetsAsButtons() {XCTAssertTrue(shouldRenderCompactGenericActionTarget(role: kAXGroupRole as String,