👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Bias open-computer-use toward low-intrusion actions
Leo98a8b808f5cb
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 021e388..5d1feed 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -54,7 +54,8 @@- `AXUIElementSetAttributeValue`- `AXUIElementCopyElementAtPosition` 做坐标命中,尽量把 coordinate click 反解成可操作 AX 元素- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台- - 只有 drag 或无法命中 AX 元素的鼠标路径,才退回全局 `CGEvent` 键鼠事件并显式前置目标 app+ - 当必须退回全局鼠标路径时,先尝试对目标窗口做 `AXRaise` / main-window 聚焦,只有这些 AX 提升失败后才调用 `NSRunningApplication.activate`+ - 只有 drag 或无法命中 AX 元素的鼠标路径,才退回全局 `CGEvent` 键鼠事件并尽量缩小对用户当前焦点的影响### 4. Fixture Bridgediff --git a/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.md b/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.mdindex 2f7907a..1599a3c 100644--- a/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.md+++ b/docs/histories/2026-04/20260417-1510-nonintrusive-computer-use.md@@ -31,3 +31,22 @@- `artifacts/tool-comparisons/20260417-focus-behavior/computer-use/get_app_state-activity-monitor.json`- `artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/get_app_state-activity-monitor.json`- `artifacts/tool-comparisons/20260417-focus-behavior/open-codex-computer-use/click-activity-monitor-coordinate.json`++### 🔁 Follow-up (2026-04-17 17:02)++同一任务在后续 MITM 调试里继续推进,新增两类收敛:++- **[全局鼠标前的 AX 提升]**: `InputSimulation` 不再把“需要全局 pointer”直接等价成 `activate()`;现在会先尝试 `AXRaise`、`kAXMainAttribute` 和 `kAXFocusedAttribute`,只有这些都失败后才回退到 `NSRunningApplication.activate`。+- **[Tool Intrusion Hints]**: 把 9 个 tools 的侵入性偏好直接写进 `ToolDefinitions` 和 plugin manifest,让模型更容易优先选择 `get_app_state` / `press_key` / `type_text` / `set_value` / `perform_secondary_action`,减少不必要的坐标点击和 drag。+- **[MITM 调试方法沉淀]**: 在 `docs/references/codex-network-capture.md` 增补 prompt 锚定差异和“宿主取消 vs MCP server 故障”的排查顺序,避免后续做 A/B 或 eval 时重复踩坑。++这次 follow-up 的重点不是再加一层复杂抽象,而是把“模型侧偏好”和“运行时兜底策略”一起往同一个方向推。仅靠运行时优化,模型仍可能频繁选到高副作用 tool;仅靠文案提示,真正退化到全局 pointer 时又仍会过早抢焦点。两边同时收口,才能更稳定地逼近官方 `computer-use` 那种“键盘优先、AX 优先、全局鼠标最后”的行为。++**Follow-up Files:**+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `plugins/open-computer-use/.codex-plugin/plugin.json`+- `docs/ARCHITECTURE.md`+- `docs/references/codex-network-capture.md`diff --git a/docs/references/codex-network-capture.md b/docs/references/codex-network-capture.mdindex 4fba987..ee4ff52 100644--- a/docs/references/codex-network-capture.md+++ b/docs/references/codex-network-capture.md@@ -227,3 +227,43 @@ artifacts/codex-dumps/<session-name>/### 3. 为什么默认不建议用 `ALL_PROXY``ALL_PROXY` 可能把本地 `127.0.0.1` 的 MCP 流量也一起代理走,容易干扰本地调试链路。默认只设 `HTTPS_PROXY` 更稳。++### 4. prompt 里写 `computer-use` 和 `open-computer-use`,为什么调用路径不一样++在 2026-04-17 这台机器上的真实样本里,prompt 文案本身会明显影响模型优先尝试的 tool namespace:++- prompt 直接写 `computer-use`+ 模型会先尝试官方 bundled `mcp__computer_use__*`。+- prompt 直接写 `open-computer-use`+ 模型会优先尝试仓库插件的 `mcp__open_computer_use__*`。++这意味着做 A/B 调试时,prompt 命名本身就是一个变量,不能忽略。想稳定比较两套实现时,建议:++1. 对两组 case 使用几乎相同的任务语义。+2. 只替换 tool 名称锚点,例如 `computer-use` vs `open-computer-use`。+3. 给每次实验加唯一标记,便于从 `websocket/*.jsonl` 和本地日志里精确过滤。++### 5. `open-computer-use` 调用失败时,怎么判断是插件宿主取消还是 MCP server 本身坏了++推荐先把问题拆成两层:++1. 用 MITM 或本地日志看 Codex 宿主是否真的发起了 `mcp__open_computer_use__*` 调用。+2. 直接对插件 launcher 做最小 JSON-RPC 探测,验证 server 本身能否 `initialize`、`tools/list`、`tools/call`。++例如:++```bash+printf '%s\n%s\n%s\n' \+'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \+'{"jsonrpc":"2.0","id":2,"method":"notifications/initialized","params":{}}' \+'{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"list_apps","arguments":{}}}' \+| ./plugins/open-computer-use/scripts/launch-open-computer-use.sh+```++如果 direct JSON-RPC 能正常返回,而 Codex 会话里仍然显示 tool 被取消或根本没有继续执行,优先怀疑:++- Codex host / plugin gate+- 当前会话对第三方插件的策略+- 插件缓存或安装态没有同步到最新++不要一开始就假设是 MCP server 逻辑本身坏了。先把“宿主问题”和“server 问题”分开,排查成本会低很多。diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex d8262d9..54cc739 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -47,7 +47,7 @@ public final class ComputerUseService {if try performPreferredClick(on: record, button: button, clickCount: clickCount) {Thread.sleep(forTimeInterval: 0.15)} else if let point = try globalPoint(for: record, snapshot: snapshot) {- InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)+ InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)try InputSimulation.clickGlobally(at: point, button: button, clickCount: clickCount)} else {throw ComputerUseError.stateUnavailable("element \(elementIndex) has no clickable frame")@@ -59,7 +59,7 @@ public final class ComputerUseService {try performPreferredClick(on: record, button: button, clickCount: clickCount) {Thread.sleep(forTimeInterval: 0.15)} else {- InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)+ InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)try InputSimulation.clickGlobally(at: try screenshotToGlobalPoint(snapshot: snapshot, x: x, y: y), button: button, clickCount: clickCount)}} else {@@ -78,7 +78,7 @@ public final class ComputerUseService {throw ComputerUseError.invalidArguments("fixture mode only supports the Raise secondary action")}- InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)+ InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)return try refreshSnapshot(for: query).renderedText}@@ -123,7 +123,7 @@ public final class ComputerUseService {Thread.sleep(forTimeInterval: 0.05)}} else if let point = try globalPoint(for: record, snapshot: snapshot) {- InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)+ InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)try InputSimulation.scrollGlobally(at: point, direction: normalized, pages: pages)} else {throw ComputerUseError.stateUnavailable("element \(elementIndex) has no scrollable frame")@@ -140,7 +140,7 @@ public final class ComputerUseService {return try refreshSnapshot(for: query).renderedText}- InputSimulation.bringAppToFrontForGlobalPointerInput(snapshot.app)+ InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)try InputSimulation.dragGlobally(from: try screenshotToGlobalPoint(snapshot: snapshot, x: fromX, y: fromY),to: try screenshotToGlobalPoint(snapshot: snapshot, x: toX, y: toY)diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swiftindex 5c71d6b..1373db0 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift@@ -43,8 +43,13 @@ enum MouseButtonKind: String {}enum InputSimulation {- static func bringAppToFrontForGlobalPointerInput(_ app: RunningAppDescriptor) {- app.runningApplication.activate()+ static func prepareAppForGlobalPointerInput(_ app: RunningAppDescriptor) {+ if raiseAppWindowViaAccessibility(pid: app.pid) {+ Thread.sleep(forTimeInterval: 0.12)+ return+ }++ _ = app.runningApplication.activate(options: [.activateAllWindows])Thread.sleep(forTimeInterval: 0.25)}@@ -153,6 +158,84 @@ enum InputSimulation {Thread.sleep(forTimeInterval: 0.03)}+ private static func raiseAppWindowViaAccessibility(pid: pid_t) -> Bool {+ let appElement = AXUIElementCreateApplication(pid)+ guard let window = preferredWindow(for: appElement) else {+ return false+ }++ if performAction(named: kAXRaiseAction as String, on: window) {+ return true+ }++ if setBoolAttribute(named: kAXMainAttribute as String, on: window) {+ return true+ }++ if setBoolAttribute(named: kAXFocusedAttribute as String, on: window) {+ return true+ }++ return false+ }++ private static func preferredWindow(for appElement: AXUIElement) -> AXUIElement? {+ copyElement(appElement, attribute: kAXFocusedWindowAttribute)+ ?? copyArray(appElement, attribute: kAXWindowsAttribute)?.first+ }++ private static func performAction(named action: String, on element: AXUIElement) -> Bool {+ guard availableActions(for: element).contains(where: { $0.caseInsensitiveCompare(action) == .orderedSame }) else {+ return false+ }++ return AXUIElementPerformAction(element, action as CFString) == .success+ }++ private static func setBoolAttribute(named attribute: String, on element: AXUIElement) -> Bool {+ guard isSettable(element: element, attribute: attribute) else {+ return false+ }++ return AXUIElementSetAttributeValue(element, attribute as CFString, kCFBooleanTrue) == .success+ }++ private static func isSettable(element: AXUIElement, attribute: String) -> Bool {+ var settable: DarwinBoolean = false+ let result = AXUIElementIsAttributeSettable(element, attribute as CFString, &settable)+ return result == .success && settable.boolValue+ }++ private static func availableActions(for element: AXUIElement) -> [String] {+ var actions: CFArray?+ let result = AXUIElementCopyActionNames(element, &actions)+ guard result == .success, let actions else {+ return []+ }++ return actions as? [String] ?? []+ }++ private static func copyElement(_ element: AXUIElement, attribute: String) -> AXUIElement? {+ var value: CFTypeRef?+ let result = AXUIElementCopyAttributeValue(element, attribute as CFString, &value)+ guard result == .success, let value else {+ return nil+ }++ return (value as! AXUIElement)+ }++ private static func copyArray(_ element: AXUIElement, attribute: String) -> [AXUIElement]? {+ var value: CFTypeRef?+ let result = AXUIElementCopyAttributeValue(element, attribute as CFString, &value)+ guard result == .success, let value else {+ return nil+ }++ return value as? [AXUIElement]+ }+private static func wheel1(direction: String, pages: Int) -> Int32 {switch direction {case "up":diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swiftindex 95362e5..4190746 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift@@ -18,12 +18,12 @@ public enum ToolDefinitions {public static let all: [ToolDefinition] = [ToolDefinition(name: "list_apps",- description: "List running or recently visible macOS applications.",+ description: "List running or recently visible macOS applications. Read-only and low intrusion.",inputSchema: objectSchema(properties: [:], required: [])),ToolDefinition(name: "get_app_state",- description: "Capture the current accessibility tree and screenshot metadata for an app.",+ description: "Capture the current accessibility tree and screenshot metadata for an app without activating it when possible. Usually call this first before deciding on an action tool.",inputSchema: objectSchema(properties: ["app": stringProperty(description: "App name or bundle identifier."),@@ -33,7 +33,7 @@ public enum ToolDefinitions {),ToolDefinition(name: "click",- description: "Click an accessibility element or a screenshot-relative coordinate.",+ description: "Click an accessibility element or a screenshot-relative coordinate. Prefer this only when a direct AX action is required; coordinate fallback may bring the app forward and use the real pointer.",inputSchema: objectSchema(properties: ["app": stringProperty(description: "App name or bundle identifier."),@@ -52,7 +52,7 @@ public enum ToolDefinitions {),ToolDefinition(name: "perform_secondary_action",- description: "Invoke an accessibility secondary action exposed by an element.",+ description: "Invoke a listed accessibility secondary action such as Raise or Show menu. Prefer this over coordinate clicks when get_app_state already exposed the desired AX action.",inputSchema: objectSchema(properties: ["app": stringProperty(description: "App name or bundle identifier."),@@ -64,7 +64,7 @@ public enum ToolDefinitions {),ToolDefinition(name: "scroll",- description: "Scroll an accessibility element in a direction by pages.",+ description: "Scroll an accessibility element in a direction by pages. Prefers AX page-scroll actions and only falls back to global wheel events when AX scrolling is unavailable.",inputSchema: objectSchema(properties: ["app": stringProperty(description: "App name or bundle identifier."),@@ -77,7 +77,7 @@ public enum ToolDefinitions {),ToolDefinition(name: "drag",- description: "Drag from one screenshot-relative coordinate to another.",+ description: "Drag from one screenshot-relative coordinate to another. This uses global mouse events, moves the real pointer, and may need to bring the target app forward.",inputSchema: objectSchema(properties: ["app": stringProperty(description: "App name or bundle identifier."),@@ -91,7 +91,7 @@ public enum ToolDefinitions {),ToolDefinition(name: "type_text",- description: "Type literal text into the focused control of an app.",+ description: "Type literal text into the focused control of an app. Sends keyboard events directly to the target app PID, which usually avoids moving the real pointer or activating the app first.",inputSchema: objectSchema(properties: ["app": stringProperty(description: "App name or bundle identifier."),@@ -102,7 +102,7 @@ public enum ToolDefinitions {),ToolDefinition(name: "press_key",- description: "Press a key or key combination using xdotool-style syntax.",+ description: "Press a key or key combination using xdotool-style syntax. Prefer keyboard shortcuts like super+n when available; events are sent directly to the target app PID.",inputSchema: objectSchema(properties: ["app": stringProperty(description: "App name or bundle identifier."),@@ -113,7 +113,7 @@ public enum ToolDefinitions {),ToolDefinition(name: "set_value",- description: "Set the value of a settable accessibility element.",+ description: "Set the value of a settable accessibility element. Prefer this over click plus type_text when the target control already exposes an AX value.",inputSchema: objectSchema(properties: ["app": stringProperty(description: "App name or bundle identifier."),diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex e37f3dd..8499c6b 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -29,4 +29,13 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(windowRelativeFrame(elementFrame: child, windowBounds: window), CGRect(x: 0, y: 0, width: 919, height: 644))XCTAssertEqual(windowRelativeFrame(elementFrame: textFieldGlobal, windowBounds: window), textField)}++ func testToolDescriptionsExposeIntrusionHints() {+ let tools = Dictionary(uniqueKeysWithValues: ToolDefinitions.all.map { ($0.name, $0.description) })++ XCTAssertTrue(tools["get_app_state"]?.contains("without activating") == true)+ XCTAssertTrue(tools["press_key"]?.contains("super+n") == true)+ XCTAssertTrue(tools["type_text"]?.contains("target app PID") == true)+ XCTAssertTrue(tools["drag"]?.contains("moves the real pointer") == true)+ }}diff --git a/plugins/open-computer-use/.codex-plugin/plugin.json b/plugins/open-computer-use/.codex-plugin/plugin.jsonindex 4aa8737..ea11934 100644--- a/plugins/open-computer-use/.codex-plugin/plugin.json+++ b/plugins/open-computer-use/.codex-plugin/plugin.json@@ -20,7 +20,7 @@"interface": {"displayName": "Open Computer Use","shortDescription": "Control macOS apps from Codex with the open local server",- "longDescription": "Open Computer Use packages this repository's open-source macOS automation server as a Codex plugin. It exposes the same nine desktop-control MCP tools through the locally built OpenComputerUse app bundle.",+ "longDescription": "Open Computer Use packages this repository's open-source macOS automation server as a Codex plugin. It exposes the same nine desktop-control MCP tools through the locally built OpenComputerUse app bundle, with an AX-first and keyboard-first strategy that tries to avoid stealing focus until global pointer input is truly required.","developerName": "Leo","category": "Productivity","capabilities": [@@ -33,7 +33,7 @@"termsOfServiceURL": "https://github.com/iFurySt/open-codex-computer-use/blob/main/LICENSE","defaultPrompt": ["Inspect the current UI state of this Mac app without stealing focus when possible",- "Click through a local macOS workflow and summarize what changed",+ "Open a new local document with keyboard shortcuts first, then type content without moving my real pointer unless required","Drive the fixture app and verify the nine computer-use tools still behave correctly"],"brandColor": "#0E7490",