Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

accessibility: preserve secondary-action mappings

Aikins Laryea9e9dcc2c09f0
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index ebff33f..14f78ad 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -76,7 +76,7 @@
- overlay 不再依赖临时 `terminal settle` 补丁来修尾;主线现在统一改成“路径层给目标点,visual dynamics 层给可见姿态”的双层模型,所以 move 末段、pulse 和 idle 共用同一套状态,不会再出现 endpoint 锁住后只剩原地翻角的收尾。
- overlay 的渲染输入也从单一 `rotation` 扩展成 `rotation + cursorBodyOffset + fogOffset + fogScale`,让速度滞后能真正体现在画面上,而不是只存在于主循环内部状态;其中 `rotation` 现在按二进制里 `SoftwareCursorStyle.angle + CursorView._animatedAngleOffsetDegrees` 的分层去近似,不再把“跟随运动方向的主朝向”和“小幅 wiggle offset”压成同一个受限小角度。
- 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略:
- - `perform_secondary_action` 只执行目标元素已经暴露出来的 AX action;无效 action 返回官方风格的 `... is not a valid secondary action for ...`,fixture 的 `Raise` 路径也不再为了测试去准备全局物理指针输入
+ - `perform_secondary_action` 只执行目标元素已经暴露出来的 AX action;Safari 这类 AppKit custom action descriptor(例如 `Name:close tab ...`)会显示为短名称并准确映射回过滤后的原始 AX action,避免 raw/rendered action 错配;无效或歧义 action 返回官方风格的 `... is not a valid secondary action for ...`,fixture 的 `Raise` 路径也不再为了测试去准备全局物理指针输入
- `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口
- `click.click_method` 是开源版的可选扩展,支持 `auto`(默认)、`accessibility`、`app_post`、`sky_click` 和 `global`。未传参数时继续使用原有自动路由;显式模式不会静默 fallback 到其他实现。`accessibility` 只接受 `element_index`;其余 mouse 路径可以使用 `element_index` 的计算落点或原始 `x/y` 坐标。
- element-targeted `click` 的 `auto` 左键路径会先试原生列表的 `AXSelectedChildren` 选择,再试 `AXPress` / `AXConfirm` / `AXOpen` 这类真正语义化的激活动作;如果目标本身不可点,还会继续尝试其子孙 AX 元素(例如 Finder sidebar row 下面暴露 `AXOpen` 的 cell)和命中点附近的 AX hit-test 结果,最后进入现有 non-AX 分支:未开启全局指针环境变量时使用 `postToPid` 定向鼠标事件,开启时直接使用全局 HID 事件。`AXRaise` / `kAXMainAttribute` / `kAXFocusedAttribute` 这类 activation-only fallback 只允许窗口级元素使用,避免普通静态文本或容器把“获得焦点”误报成“点击已处理”;`click_count > 1` 也会优先重复可用的 AX action。
diff --git a/docs/histories/2026-09/20260906-1200-secondary-action-correctness.md b/docs/histories/2026-09/20260906-1200-secondary-action-correctness.md
new file mode 100644
index 0000000..3749bd7
--- /dev/null
+++ b/docs/histories/2026-09/20260906-1200-secondary-action-correctness.md
@@ -0,0 +1,23 @@
+## [2026-09-06 12:00] | Task: 修正 secondary action 映射
+
+### 📥 User Query
+> 独立提取 macOS secondary-action correctness 变更,支持 Safari custom action description,并修复过滤后的 raw/display action 错配。
+
+### 🛠 Changes Overview
+**Scope:** OpenComputerUseKit macOS accessibility snapshot 与 action execution
+
+**Key Actions:**
+- **动作渲染**: 解析 AppKit `Name:...` action descriptor,并输出有意义的短名称。
+- **名称冲突**: 短名称冲突时显示原始 descriptor,保证每个输出 selector 都可唯一映射回对应动作。
+- **动作执行**: 保持原始 AX action 精确匹配的兼容性;短名称基于同一组 role-filtered raw actions 匹配,拒绝歧义短名称。
+- **回归测试**: 覆盖 Safari close-tab、过滤对齐、无效 descriptor 与歧义名称。
+
+### 🧠 Design Intent (Why)
+*渲染与执行必须共享过滤和命名语义,否则压缩后的 action 数组会与原始 action 索引错位并执行错误动作。*
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
+- `docs/releases/feature-release-notes.md`
diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.md
index fffc323..892ffc3 100644
--- a/docs/releases/feature-release-notes.md
+++ b/docs/releases/feature-release-notes.md
@@ -5,6 +5,7 @@
| 日期 | 功能域 | 用户价值 | 变更摘要 |
| --- | --- | --- | --- |
| 2026-09-06 | macOS 应用名称解析 | 按名称定位 Safari 等应用时优先匹配真实应用,避免同名后台 helper 抢占匹配结果。 | 按普通应用名称、普通应用 executable、后台应用名称、后台 executable 分级匹配;同级保持原顺序,bundle identifier 和安全过滤规则不变。 |
+| 2026-09-06 | macOS secondary action | Agent 可以通过 Safari 暴露的 `close tab` secondary action 可靠关闭标签页,不会因内部动作被过滤而误执行另一项动作。 | macOS renderer 将 `Name:close tab ...` 形式的 AppKit custom action descriptor 显示为短名称,executor 从同一组过滤后的 raw actions 做等价名称匹配,并拒绝歧义匹配。 |
| 2026-09-01 | 内嵌 App Agent Socket 隔离 | 使用内嵌 OCU 的宿主不会再与用户全局 OCU 争用同一个 App Agent Socket,避免一个实例意外终止或替换另一个实例。 | 发布 `0.3.3`,为显式配置 namespace 的宿主生成确定性、短且不泄露原值的 Socket 文件名;未配置时继续兼容旧路径。 |
## 2026-08
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index 604213c..a83b367 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -9,6 +9,7 @@ final class ElementRecord {
let identifier: String?
let element: AXUIElement?
let localFrame: CGRect?
+ let role: String?
let rawActions: [String]
let prettyActions: [String]
let isSyntheticText: Bool
@@ -18,6 +19,7 @@ final class ElementRecord {
identifier: String?,
element: AXUIElement?,
localFrame: CGRect?,
+ role: String? = nil,
rawActions: [String],
prettyActions: [String],
isSyntheticText: Bool = false
@@ -26,6 +28,7 @@ final class ElementRecord {
self.identifier = identifier
self.element = element
self.localFrame = localFrame
+ self.role = role
self.rawActions = rawActions
self.prettyActions = prettyActions
self.isSyntheticText = isSyntheticText
@@ -383,6 +386,7 @@ enum SnapshotBuilder {
identifier: element.identifier,
element: nil,
localFrame: element.frame.cgRect,
+ role: element.role,
rawActions: element.actions,
prettyActions: element.actions
)
@@ -838,6 +842,7 @@ private struct TreeRenderer {
identifier: axIdentifier,
element: root,
localFrame: localFrame,
+ role: role,
rawActions: actions,
prettyActions: prettyActions
)
@@ -1808,6 +1813,17 @@ private func roleDescription(of element: AXUIElement, role: String, subrole: Str
}
func meaningfulActions(_ values: [String], role: String) -> [String] {
+ let rawActions = meaningfulRawActions(values, role: role)
+ let names = rawActions.map(secondaryActionDisplayName(_:))
+ return names.indices.map { index in
+ let collides = names.indices.contains { other in
+ other != index && secondaryActionNamesEquivalent(names[index], names[other])
+ }
+ return collides ? rawActions[index] : names[index]
+ }
+}
+
+func meaningfulRawActions(_ values: [String], role: String) -> [String] {
values
.filter {
var ignored = [
@@ -1841,10 +1857,13 @@ func meaningfulActions(_ values: [String], role: String) -> [String] {
return true
}
- .map(prettyActionName(_:))
}
-private func prettyActionName(_ value: String) -> String {
+func secondaryActionDisplayName(_ value: String) -> String {
+ if let name = accessibilityActionDescriptionName(value) {
+ return name
+ }
+
if value == "AXZoomWindow" {
return "zoom the window"
}
@@ -1854,6 +1873,90 @@ private func prettyActionName(_ value: String) -> String {
return splitCamelCase(withoutPage)
}
+func secondaryActionNamesEquivalent(_ lhs: String, _ rhs: String) -> Bool {
+ !normalizedSecondaryActionCandidates(lhs).isDisjoint(with: normalizedSecondaryActionCandidates(rhs))
+}
+
+func accessibilityActionDescriptionName(_ value: String) -> String? {
+ guard let nameStart = actionDescriptionValueStart(label: "name", in: value) else {
+ return nil
+ }
+
+ let nameEnd = ["target", "selector", "button clicked"]
+ .compactMap { actionDescriptionLabelStart(label: $0, in: value, from: nameStart) }
+ .min() ?? value.endIndex
+ let name = value[nameStart..<nameEnd].trimmingCharacters(in: .whitespacesAndNewlines)
+ return name.isEmpty ? nil : name
+}
+
+private func actionDescriptionValueStart(label: String, in value: String) -> String.Index? {
+ var searchStart = value.startIndex
+ while let range = value.range(of: label, options: .caseInsensitive, range: searchStart..<value.endIndex) {
+ if range.lowerBound != value.startIndex, !value[value.index(before: range.lowerBound)].isWhitespace {
+ searchStart = range.upperBound
+ continue
+ }
+
+ var cursor = range.upperBound
+ while cursor < value.endIndex, value[cursor].isWhitespace {
+ cursor = value.index(after: cursor)
+ }
+ guard cursor < value.endIndex, value[cursor] == ":" else {
+ searchStart = range.upperBound
+ continue
+ }
+ cursor = value.index(after: cursor)
+ while cursor < value.endIndex, value[cursor].isWhitespace {
+ cursor = value.index(after: cursor)
+ }
+ return cursor
+ }
+ return nil
+}
+
+private func actionDescriptionLabelStart(label: String, in value: String, from start: String.Index) -> String.Index? {
+ var searchStart = start
+ while let range = value.range(of: label, options: .caseInsensitive, range: searchStart..<value.endIndex) {
+ if range.lowerBound != value.startIndex, !value[value.index(before: range.lowerBound)].isWhitespace {
+ searchStart = range.upperBound
+ continue
+ }
+
+ var cursor = range.upperBound
+ while cursor < value.endIndex, value[cursor].isWhitespace {
+ cursor = value.index(after: cursor)
+ }
+ guard cursor < value.endIndex, value[cursor] == ":" else {
+ searchStart = range.upperBound
+ continue
+ }
+ return range.lowerBound
+ }
+ return nil
+}
+
+private func normalizedSecondaryActionCandidates(_ value: String) -> Set<String> {
+ var candidates = Set<String>()
+ let normalized = normalizedSecondaryActionName(value)
+ if !normalized.isEmpty {
+ candidates.insert(normalized)
+ }
+ if let descriptionName = accessibilityActionDescriptionName(value) {
+ candidates.insert(normalizedSecondaryActionName(descriptionName))
+ }
+ return candidates
+}
+
+private func normalizedSecondaryActionName(_ value: String) -> String {
+ value
+ .trimmingCharacters(in: .whitespacesAndNewlines)
+ .replacingOccurrences(of: "_", with: " ")
+ .replacingOccurrences(of: "-", with: " ")
+ .split(whereSeparator: { $0.isWhitespace })
+ .joined(separator: " ")
+ .lowercased()
+}
+
private func humanizeAXToken(_ value: String) -> String {
let stripped = value.hasPrefix("AX") ? String(value.dropFirst(2)) : value
return splitCamelCase(stripped).lowercased()
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
index afdce9b..1fbc48a 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
@@ -832,16 +832,15 @@ public final class ComputerUseService {
return record
}
- private func matchingAction(requested: String, record: ElementRecord) -> String? {
+ func matchingAction(requested: String, record: ElementRecord) -> String? {
if let exact = record.rawActions.first(where: { $0.caseInsensitiveCompare(requested) == .orderedSame }) {
return exact
}
-
- if let pretty = zip(record.rawActions, record.prettyActions).first(where: { $0.1.caseInsensitiveCompare(requested) == .orderedSame }) {
- return pretty.0
+ let visibleActions = record.role.map { meaningfulRawActions(record.rawActions, role: $0) } ?? record.rawActions
+ let matches = visibleActions.filter { rawAction in
+ secondaryActionNamesEquivalent(requested, secondaryActionDisplayName(rawAction))
}
-
- return nil
+ return matches.count == 1 ? matches[0] : nil
}
private func invalidSecondaryActionMessage(action: String, record: ElementRecord) -> String {
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index bf28855..6c4b224 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -1138,6 +1138,59 @@ final class OpenComputerUseKitTests: XCTestCase {
))
}
+ func testAccessibilityRendererUsesSafariCustomActionDescriptionName() {
+ XCTAssertEqual(
+ meaningfulActions(
+ ["Name:close tab Target:SafariTab Selector:_close Button Clicked:"],
+ role: kAXButtonRole as String
+ ),
+ ["close tab"]
+ )
+ XCTAssertNil(accessibilityActionDescriptionName("Namespace:close tab Target:SafariTab"))
+ }
+
+ func testSecondaryActionMatchingKeepsFilteredRawActionsAligned() {
+ let service = ComputerUseService()
+ let closeTab = "Name:close tab Target:SafariTab Selector:_close Button Clicked:"
+ let record = ElementRecord(
+ index: 48,
+ identifier: nil,
+ element: nil,
+ localFrame: nil,
+ role: kAXButtonRole as String,
+ rawActions: [kAXPressAction as String, closeTab],
+ prettyActions: ["close tab"]
+ )
+
+ XCTAssertEqual(service.matchingAction(requested: "close-tab", record: record), closeTab)
+ XCTAssertEqual(service.matchingAction(requested: kAXPressAction as String, record: record), kAXPressAction as String)
+ XCTAssertNil(service.matchingAction(requested: "Press", record: record))
+ }
+
+ func testSecondaryActionMatchingRejectsAmbiguousDisplayNames() {
+ let service = ComputerUseService()
+ let record = ElementRecord(
+ index: 49,
+ identifier: nil,
+ element: nil,
+ localFrame: nil,
+ role: kAXButtonRole as String,
+ rawActions: [
+ "Name:close tab Target:FirstTab Selector:_close:",
+ "Name:close tab Target:SecondTab Selector:_close:",
+ ],
+ prettyActions: ["close tab", "close tab"]
+ )
+
+ XCTAssertNil(service.matchingAction(requested: "close tab", record: record))
+ XCTAssertEqual(service.matchingAction(requested: record.rawActions[1], record: record), record.rawActions[1])
+ let emitted = meaningfulActions(record.rawActions, role: kAXButtonRole as String)
+ XCTAssertEqual(emitted, record.rawActions)
+ for (selector, rawAction) in zip(emitted, record.rawActions) {
+ XCTAssertEqual(service.matchingAction(requested: selector, record: record), rawAction)
+ }
+ }
+
func testAccessibilityRendererMarksCompactGenericClickTargetsAsButtons() {
XCTAssertTrue(shouldRenderCompactGenericActionTarget(
role: kAXGroupRole as String,