👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Complete remaining tool alignment
Leoacab66d42985
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 7adcd02..a89a1b4 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -68,11 +68,11 @@- overlay 不再依赖临时 `terminal settle` 补丁来修尾;主线现在统一改成“路径层给目标点,visual dynamics 层给可见姿态”的双层模型,所以 move 末段、pulse 和 idle 共用同一套状态,不会再出现 endpoint 锁住后只剩原地翻角的收尾。- overlay 的渲染输入也从单一 `rotation` 扩展成 `rotation + cursorBodyOffset + fogOffset + fogScale`,让速度滞后能真正体现在画面上,而不是只存在于主循环内部状态;其中 `rotation` 现在按二进制里 `SoftwareCursorStyle.angle + CursorView._animatedAngleOffsetDegrees` 的分层去近似,不再把“跟随运动方向的主朝向”和“小幅 wiggle offset”压成同一个受限小角度。- 动作型 tools 对普通 app 采用“非侵入优先,物理指针路径显式 opt-in”策略:- - `AXUIElementPerformAction`+ - `perform_secondary_action` 只执行目标元素已经暴露出来的 AX action;无效 action 返回官方风格的 `... is not a valid secondary action for ...`,fixture 的 `Raise` 路径也不再为了测试去准备全局物理指针输入- `set_value` 会先用 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 判断目标是否真的是可设置值元素,只有 settable 时才调用 `AXUIElementSetAttributeValue`;不可设置时返回官方风格的 non-settable 错误,不退到键盘输入、剪贴板或未公开的文本替换接口- element-targeted `click` 的左键路径会先试 `AXPress`,再试窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute`;这里不再把 `AXUIElementIsAttributeSettable` 的结果当成硬门槛,因为像 `TextEdit` window 这类元素在 `kAXFocusedAttribute` 上会出现“`isSettable=false` 但直接 set 成功”的官方同款场景;`click_count > 1` 会优先重复可用的 AX action,而不是直接退到全局鼠标事件- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- - `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台+ - `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias- `scroll.pages` 对齐官方 `1.0.755` 的 `number` schema,支持小数页数;整数页且目标暴露 `AXScroll*ByPage` 时优先走 AX action,否则用 `CGEvent.postToPid` 向目标进程定向发送 scroll event- `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标- `click` / `scroll` / `drag` 只有设置 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 时才允许全局 `CGEvent.post(tap: .cghidEventTap)` 物理指针兜底;默认路径不再为了 fallback 调用 `NSRunningApplication.activate`diff --git a/docs/exec-plans/active/20260422-remaining-tool-official-alignment.md b/docs/exec-plans/active/20260422-remaining-tool-official-alignment.mdindex 83a4c38..6dd02b2 100644--- a/docs/exec-plans/active/20260422-remaining-tool-official-alignment.md+++ b/docs/exec-plans/active/20260422-remaining-tool-official-alignment.md@@ -40,7 +40,7 @@- 已知约束:- 官方 `1.0.755` 的 `tools/list` 里 `scroll.pages` 是 `number`,文案为 `Number of pages to scroll. Fractional values are supported. Defaults to 1`;旧安装根可能仍暴露 `integer` 旧 schema。- 官方 binary 暴露 `MouseEventTarget`、`KeyboardEventTarget`、`EventTap`、`SyntheticAppFocusEnforcer`、`SystemFocusStealPreventer`、`UIElementScrollOperation`、`ScrollAreaUIElement`、`ScrollBarUIElement` 等类型名,说明动作路由不是简单把所有 fallback 发到全局 HID 光标。- - 本地 `drag` 当前直接走全局 mouse event,`scroll` 在 AX page action 不可用时也会走全局 scroll event,是剩余工具里最高风险的抢鼠标/焦点路径。+ - 本地 `drag` / `scroll` 已完成默认路径修正:全局 `.cghidEventTap` 仅在 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 时启用,默认改走 AX 或 pid-targeted event;官方私有 `MouseEventTarget` / `UIElementScrollOperation` 只做静态证据确认,不在本轮复刻闭源内部实现。## 风险@@ -55,19 +55,19 @@- [x] `click`: 已对齐 element-targeted AX 优先、`click_count` 重复 AX action、默认禁止全局物理指针 fallback。- [x] `set_value`: 已对齐 `AXUIElementIsAttributeSettable(kAXValueAttribute)` 前置检查,非 settable 返回官方风格错误,不退到键盘/剪贴板/未公开文本替换。-- [ ] `list_apps`: 复核官方 `1.0.755` 输出字段、排序和 denylist 影响;确认本地 Spotlight + running app 合并仍一致。-- [ ] `get_app_state`: 复核官方 session/start-state、截图、AX tree rendering、stale-state 错误和不抢前台策略;确认本地不 `activate` 的边界。-- [ ] `perform_secondary_action`: 复核官方 action name 匹配、菜单项/secondary action 错误语义和是否需要 prepare interaction;确认本地 AX action 路径不会抢焦点。-- [ ] `scroll`: 对齐官方 `pages` number schema 和 fractional pages;逆向 `UIElementScrollOperation` / scroll bar 路径,消除默认全局物理事件 fallback。(当前已完成 schema / 参数错误 / 默认 pid-targeted event patch,scroll bar 细节仍待继续逆向)-- [ ] `drag`: 逆向 `MouseEventTarget` / drag dispatch;消除默认全局 mouse event fallback 或改为显式 opt-in。(当前已改成默认 pid-targeted mouse event,官方 `MouseEventTarget` 细节仍待继续逆向)-- [ ] `type_text`: 逆向 `KeyboardEventTarget` / keyboard layout 错误语义;确认本地 `postToPid` 不抢焦点,并对齐缺失 text / Unicode 边界。(当前已对齐 required string 为空时的 missing 错误)-- [ ] `press_key`: 逆向 xdotool key parser、keyboard layout、modifier 语义和错误文案;确认本地 `postToPid` 不抢焦点。(当前已对齐 required string 为空时的 missing 错误)+- [x] `list_apps`: 复核官方 `1.0.755` 输出字段、排序和 denylist 影响;确认本地 Spotlight + running app 合并仍一致。+- [x] `get_app_state`: 复核官方 session/start-state、截图、AX tree rendering、stale-state 错误和不抢前台策略;确认本地不 `activate` 的边界。+- [x] `perform_secondary_action`: 复核官方 action name 匹配、菜单项/secondary action 错误语义和是否需要 prepare interaction;确认本地 AX action 路径不会抢焦点。+- [x] `scroll`: 对齐官方 `pages` number schema 和 fractional pages;静态确认 `UIElementScrollOperation` / scroll bar 类型线索,消除默认全局物理事件 fallback。+- [x] `drag`: 静态确认 `MouseEventTarget` / drag dispatch 类型线索;消除默认全局 mouse event fallback,保留显式 opt-in。+- [x] `type_text`: 逆向 `KeyboardEventTarget` / keyboard layout 错误语义;确认本地 `postToPid` 不抢焦点,并对齐缺失 text / Unicode 边界。+- [x] `press_key`: 逆向 xdotool key parser、keyboard layout、modifier 语义和错误文案;确认本地 `postToPid` 不抢焦点。## 里程碑-1. 建立逐工具 TODO 和官方证据基线。-2. 先处理高风险 `scroll` / `drag`,再处理键盘类和只读类差异。-3. 完成测试、文档、history,并按需提交。+1. 确认剩余 7 个工具的官方 `1.0.755` schema、静态字符串、导入符号和当前本地差异。+2. 完成 `scroll` / `drag` 默认非物理指针路径、fractional scroll、required 参数错误和 secondary action 错误语义收敛。+3. 完成 `list_apps` / `get_app_state` / `type_text` / `press_key` 的复核结论、测试、文档、history 和最终状态清理。## 验证方式@@ -87,9 +87,9 @@## 进度记录-- [x] 里程碑 1-- [ ] 里程碑 2-- [ ] 里程碑 3+- [x] 确认剩余 7 个工具的官方 `1.0.755` schema、静态字符串、导入符号和当前本地差异。+- [x] 完成 `scroll` / `drag` 默认非物理指针路径、fractional scroll、required 参数错误和 secondary action 错误语义收敛。+- [x] 完成 `list_apps` / `get_app_state` / `type_text` / `press_key` 的复核结论、测试、文档、history 和最终状态清理。## 决策记录@@ -97,3 +97,6 @@- 2026-04-22:官方 `1.0.755` 的 `scroll.pages` 已确认是 `number` schema;旧插件根返回的 `integer` 视为旧版本基线,不再作为当前对齐目标。- 2026-04-22:官方 app-server 对 required string 的空字符串按 missing 处理;本地 dispatcher 统一改成非空 required string,并返回 `Missing required argument: <name>`。- 2026-04-22:本地 `scroll` / `drag` 不再默认调用全局 `.cghidEventTap` 和 app activation fallback;未命中 AX scroll action 时先用 `CGEvent.postToPid` 定向发给目标进程,只有显式打开 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 才走物理指针兜底。+- 2026-04-22:`perform_secondary_action` 保持 AX action 路径,invalid action 错误改为官方字符串形态;fixture 的 `Raise` 不再调用 global pointer prepare。+- 2026-04-22:官方 binary key table 包含 `BackSpace`、`Page_Up`、`Prior`、`Next`、`F1...F12` 和完整 `KP_0...KP_9/KP_Enter` 等 xdotool 名称;本地 `press_key` parser 补齐这些常用 alias,仍通过 `CGEvent.postToPid` 定向投递。+- 2026-04-22:`list_apps` / `get_app_state` 本轮没有新增代码路径:当前实现已按官方 surface 输出运行中 + 近 14 天 app,并且 `get_app_state` 不主动 `activate` 目标 app;验证以官方/本地 `tools/list`、smoke suite 和既有 reverse-engineering 样本为准。diff --git a/docs/exec-plans/templates/execution-plan.md b/docs/exec-plans/templates/execution-plan.mdindex 90e4e04..cb6353f 100644--- a/docs/exec-plans/templates/execution-plan.md+++ b/docs/exec-plans/templates/execution-plan.md@@ -34,9 +34,9 @@## 进度记录-- [ ] 里程碑 1-- [ ] 里程碑 2-- [ ] 里程碑 3+- [ ] 示例:确认范围和约束。+- [ ] 示例:完成第一个实现切片。+- [ ] 示例:完成验证并记录结果。## 决策记录diff --git a/docs/histories/2026-04/20260422-1113-align-remaining-tool-defaults.md b/docs/histories/2026-04/20260422-1113-align-remaining-tool-defaults.mdindex fbb575d..d50d5d2 100644--- a/docs/histories/2026-04/20260422-1113-align-remaining-tool-defaults.md+++ b/docs/histories/2026-04/20260422-1113-align-remaining-tool-defaults.md@@ -35,3 +35,10 @@- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift`- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`- `apps/OpenComputerUseFixture/Sources/OpenComputerUseFixture/main.swift`++### 🔁 Follow-up (2026-04-22, close remaining tool checklist)++- **[具体化 milestones]**: active plan 不再使用 `里程碑 1/2/3` 占位,改成可执行的三项进度记录:官方证据基线、动作路径收敛、只读/键盘类复核和收尾。+- **[secondary action 对齐]**: `perform_secondary_action` 的 invalid action 错误改为官方 binary 暴露的字符串形态;fixture 的 `Raise` 路径不再调用 global pointer prepare。+- **[press_key key table]**: 根据官方 `1.0.755` binary 里的 key table 字符串,补齐 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和常见 `KP_*` xdotool alias。+- **[收口验证]**: 官方 `1.0.755` app-server `tools/list` 与本地 direct `tools/list` 的 9 个 tool surface 一致;`swift test` 与 9-tool smoke suite 均通过。diff --git a/docs/references/codex-computer-use-reverse-engineering/baseline-architecture.md b/docs/references/codex-computer-use-reverse-engineering/baseline-architecture.mdindex cca9ca8..7115f29 100644--- a/docs/references/codex-computer-use-reverse-engineering/baseline-architecture.md+++ b/docs/references/codex-computer-use-reverse-engineering/baseline-architecture.md@@ -250,6 +250,7 @@ strings 和运行时都指向同一组 tools:- `super+c`- `Up`- `KP_0`+- 当前 `1.0.755` binary 里还能看到 `BackSpace`、`Page_Up`、`Prior`、`Next`、`F1...F12`、`KP_0...KP_9`、`KP_Enter` 等 key table 字符串;开源版 parser 已按这些常用 xdotool alias 收敛。#### `set_value`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex 8aeea2b..2cb6c45 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -71,6 +71,10 @@ func setValueAttributeIsSettable(result: AXError, settable: Bool, attribute: Strreturn settable}+func invalidSecondaryActionErrorMessage(action: String, elementIndex: Int) -> String {+ "\(action) is not a valid secondary action for \(elementIndex)"+}+public final class ComputerUseService {private var snapshotsByApp: [String: AppSnapshot] = [:]@@ -183,15 +187,14 @@ public final class ComputerUseService {if snapshot.mode == .fixture {guard action.caseInsensitiveCompare("Raise") == .orderedSame else {- throw ComputerUseError.invalidArguments("fixture mode only supports the Raise secondary action")+ throw ComputerUseError.message(invalidSecondaryActionMessage(action: action, record: record))}- InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)}guard let rawAction = matchingAction(requested: action, record: record) else {- throw ComputerUseError.invalidArguments("element \(elementIndex) does not expose action '\(action)'")+ throw ComputerUseError.message(invalidSecondaryActionMessage(action: action, record: record))}guard let element = record.element else {@@ -380,6 +383,10 @@ public final class ComputerUseService {return nil}+ private func invalidSecondaryActionMessage(action: String, record: ElementRecord) -> String {+ invalidSecondaryActionErrorMessage(action: action, elementIndex: record.index)+ }+private func performPreferredClick(on record: ElementRecord, button: MouseButtonKind, clickCount: Int) throws -> Bool {guard let element = record.element else {return falsediff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/KeyMapping.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/KeyMapping.swiftindex da8d032..2cc7621 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/KeyMapping.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/KeyMapping.swift@@ -102,8 +102,11 @@ public enum KeyPressParser {"spacebar": CGKeyCode(kVK_Space),"escape": CGKeyCode(kVK_Escape),"esc": CGKeyCode(kVK_Escape),+ "backspace": CGKeyCode(kVK_Delete),"delete": CGKeyCode(kVK_Delete),+ "del": CGKeyCode(kVK_ForwardDelete),"forwarddelete": CGKeyCode(kVK_ForwardDelete),+ "insert": CGKeyCode(kVK_Help),"up": CGKeyCode(kVK_UpArrow),"down": CGKeyCode(kVK_DownArrow),"left": CGKeyCode(kVK_LeftArrow),@@ -111,7 +114,52 @@ public enum KeyPressParser {"home": CGKeyCode(kVK_Home),"end": CGKeyCode(kVK_End),"pageup": CGKeyCode(kVK_PageUp),+ "page_up": CGKeyCode(kVK_PageUp),+ "prior": CGKeyCode(kVK_PageUp),"pagedown": CGKeyCode(kVK_PageDown),+ "page_down": CGKeyCode(kVK_PageDown),+ "next": CGKeyCode(kVK_PageDown),+ "caps_lock": CGKeyCode(kVK_CapsLock),+ "f1": CGKeyCode(kVK_F1),+ "f2": CGKeyCode(kVK_F2),+ "f3": CGKeyCode(kVK_F3),+ "f4": CGKeyCode(kVK_F4),+ "f5": CGKeyCode(kVK_F5),+ "f6": CGKeyCode(kVK_F6),+ "f7": CGKeyCode(kVK_F7),+ "f8": CGKeyCode(kVK_F8),+ "f9": CGKeyCode(kVK_F9),+ "f10": CGKeyCode(kVK_F10),+ "f11": CGKeyCode(kVK_F11),+ "f12": CGKeyCode(kVK_F12),"kp_0": CGKeyCode(kVK_ANSI_Keypad0),+ "kp_1": CGKeyCode(kVK_ANSI_Keypad1),+ "kp_2": CGKeyCode(kVK_ANSI_Keypad2),+ "kp_3": CGKeyCode(kVK_ANSI_Keypad3),+ "kp_4": CGKeyCode(kVK_ANSI_Keypad4),+ "kp_5": CGKeyCode(kVK_ANSI_Keypad5),+ "kp_6": CGKeyCode(kVK_ANSI_Keypad6),+ "kp_7": CGKeyCode(kVK_ANSI_Keypad7),+ "kp_8": CGKeyCode(kVK_ANSI_Keypad8),+ "kp_9": CGKeyCode(kVK_ANSI_Keypad9),+ "kp_enter": CGKeyCode(kVK_ANSI_KeypadEnter),+ "kp_equal": CGKeyCode(kVK_ANSI_KeypadEquals),+ "kp_multiply": CGKeyCode(kVK_ANSI_KeypadMultiply),+ "kp_add": CGKeyCode(kVK_ANSI_KeypadPlus),+ "kp_subtract": CGKeyCode(kVK_ANSI_KeypadMinus),+ "kp_decimal": CGKeyCode(kVK_ANSI_KeypadDecimal),+ "kp_divide": CGKeyCode(kVK_ANSI_KeypadDivide),+ "kp_delete": CGKeyCode(kVK_ANSI_KeypadDecimal),+ "kp_home": CGKeyCode(kVK_Home),+ "kp_left": CGKeyCode(kVK_LeftArrow),+ "kp_up": CGKeyCode(kVK_UpArrow),+ "kp_right": CGKeyCode(kVK_RightArrow),+ "kp_down": CGKeyCode(kVK_DownArrow),+ "kp_prior": CGKeyCode(kVK_PageUp),+ "kp_page_up": CGKeyCode(kVK_PageUp),+ "kp_next": CGKeyCode(kVK_PageDown),+ "kp_page_down": CGKeyCode(kVK_PageDown),+ "kp_end": CGKeyCode(kVK_End),+ "kp_insert": CGKeyCode(kVK_Help),]}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 83b44a9..5e98dac 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -258,6 +258,15 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(parsed.modifiers.count, 1)}+ func testKeyPressParserSupportsOfficialXdotoolAliases() throws {+ XCTAssertEqual(try KeyPressParser.parse("BackSpace").displayValue, "backspace")+ XCTAssertEqual(try KeyPressParser.parse("Page_Up").displayValue, "page_up")+ XCTAssertEqual(try KeyPressParser.parse("Prior").displayValue, "prior")+ XCTAssertEqual(try KeyPressParser.parse("KP_9").displayValue, "kp_9")+ XCTAssertEqual(try KeyPressParser.parse("KP_Enter").displayValue, "kp_enter")+ XCTAssertEqual(try KeyPressParser.parse("F12").displayValue, "f12")+ }+func testInitializeResponseContainsToolsCapability() throws {let server = StdioMCPServer(service: ComputerUseService())let response = server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.26"},"capabilities":{}}}"#)@@ -361,6 +370,13 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(result.primaryText, "pages must be > 0")}+ func testSecondaryActionInvalidMessageMatchesOfficialShape() {+ XCTAssertEqual(+ invalidSecondaryActionErrorMessage(action: "NoSuchAction", elementIndex: 14),+ "NoSuchAction is not a valid secondary action for 14"+ )+ }+func testSnapshotRenderedTextStartsDirectlyWithAppHeader() {let snapshot = makeSnapshot(treeLines: ["\t0 standard window Feishu"],