👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
fix(drag): report delivery path and document the global pointer gate
Camille Clayton6489d9d7fbe4
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex f2dd4c4..2eeafa2 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -86,7 +86,7 @@- `CGEvent.postToPid` 定向发送键盘事件,避免为了 `type_text` / `press_key` 抢前台;`type_text` 会把文本按 Unicode extended grapheme cluster 聚合成小批量 `keyboardSetUnicodeString` 事件,避免中文标点、emoji / 代理对和组合字符被逐个 UTF-16 code unit 拆开后在 Electron 富文本输入框里乱序或变形。如果当前 focused element 的 `AXValue` 可设置,`type_text` 会优先按可编辑内容追加并写回 `AXValue`,这覆盖 Feishu / Electron 富文本输入框不可靠接收后台键盘事件的场景,并会过滤已知占位提示,避免把 placeholder 拼进草稿;如果当前 focused element 不是可编辑文本目标,`type_text` 会报错要求先 click 文本输入区或使用 `set_value`,不再把无效果的后台键盘投递当成成功;`press_key` 的 xdotool parser 覆盖官方 binary key table 里常见的 `BackSpace`、`Page_Up`、`Prior` / `Next`、`F1...F12` 和 `KP_*` alias- `scroll.pages` 对齐官方 `1.0.755` 的 `number` schema,支持小数页数;整数页且目标暴露 `AXScroll*ByPage` 时优先走 AX action,否则用 `CGEvent.postToPid` 向目标进程定向发送 scroll event- `drag` 仍是 coordinate-only API,但默认不再使用全局 `.cghidEventTap` mouse event;默认改为 `CGEvent.postToPid` 定向发送 mouse move / down / dragged / up 事件,避免移动用户真实硬件光标;这些 coordinate tool 的 `x/y` 先按 screenshot pixel 坐标解释,再依据截图像素尺寸与目标 window bounds 的比例映射回 window point / Quartz global 坐标,避免 Retina 窗口上把 2x 像素误当成 1x point 导致点击落到错误位置- - `click` / `scroll` / `drag` 在环境变量未开启的默认配置下不会走全局 `.cghidEventTap`,因此不会移动或抢占用户真实鼠标;`OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 是全局指针能力的进程级安全门,显式 `click_method=global` 仍必须通过这层授权。默认路径不再为了 fallback 调用 `NSRunningApplication.activate`+ - `click` / `scroll` / `drag` 在环境变量未开启的默认配置下不会走全局 `.cghidEventTap`,因此不会移动或抢占用户真实鼠标;`OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 是全局指针能力的进程级安全门,显式 `click_method=global` 仍必须通过这层授权。默认路径不再为了 fallback 调用 `NSRunningApplication.activate`。`drag` 的结果会在 snapshot 文本之后、截图之前附带一条 `Drag delivered via ...` 文本项,说明本次走的是 `app_post` 定向投递还是全局指针路径;默认路径下同时说明 `postToPid` 事件不经过 window server,因此无法驱动窗口移动、文本拖选、Finder 拖放这类 window server 拖拽会话。`drag` 的 tool description 也同步说明了这一限制和对应环境变量### 4. Fixture Bridgediff --git a/docs/histories/2026-09/20260908-0934-drag-delivery-path-legibility.md b/docs/histories/2026-09/20260908-0934-drag-delivery-path-legibility.mdnew file mode 100644index 0000000..a42f2fd--- /dev/null+++ b/docs/histories/2026-09/20260908-0934-drag-delivery-path-legibility.md@@ -0,0 +1,30 @@+## [2026-09-08 09:34] | Task: Make the default `drag` path legible++### 🤖 Execution Context+* **Agent ID**: `Claude Code`+* **Base Model**: `Claude Fable 5.1`+* **Runtime**: `macOS 26 / Swift 6.3.3 (Command Line Tools)`++### 📥 User Query+> Pick up the drag investigation and work on a fix or clearer docs for the open GitHub issue (#53: `drag` executes without error but performs no window move / text selection).++### 🛠 Changes Overview+**Scope:** `packages/OpenComputerUseKit`, `skills/open-computer-use`, `docs/`++**Key Actions:**+- **[Delivery path helpers]**: added `DragDeliveryPath` (`app_post` / `global`), `dragDeliveryPath(environment:)`, `dragDeliveryNote(for:)` and `appendingDragDeliveryNote(to:path:)` next to `globalPointerFallbacksEnabled`. `performDragEvent` now returns the path it used and `drag` inserts the note as a text content item after the snapshot text and before the screenshot, so `primaryText` is unchanged.+- **[Tool description]**: the `drag` MCP tool description now states that the default path posts events directly to the target app, cannot drive window-server drag sessions (window moves, text selection, Finder drag-and-drop), and that those require `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` in the server process environment. This mirrors what `click_method` already says about `global`.+- **[Tests]**: unit tests for the path decision, the note wording, and the content-item placement with and without a screenshot; the tool-definitions test now checks the `drag` description.+- **[Docs]**: new `Drag Delivery` section in `skills/open-computer-use/references/usage.md`, the drag case added to the `SKILL.md` guardrail, `docs/ARCHITECTURE.md` updated, and a feature release note row added.++### 🧠 Design Intent (Why)+The default `CGEvent.postToPid` path was chosen deliberately on 2026-04-22 so `drag` never moves the user's real pointer. Its cost is that pid-targeted events never reach the window server, so a window move, drag-select or Finder drag-and-drop returns success with no effect. Two reporters on issue #53 read that as a bug because neither the tool description nor the result said which path ran, and the rationale lived only in the Chinese architecture and security docs. This change keeps the default and the safety gate exactly as they are and makes the outcome self-describing at the two places a caller actually looks: the tool schema before the call and the result after it. A `drag_method` parameter with `click_method` parity was considered and left out to keep the change small; it can follow if wanted.++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `skills/open-computer-use/references/usage.md`+- `skills/open-computer-use/SKILL.md`+- `docs/ARCHITECTURE.md`+- `docs/releases/feature-release-notes.md`diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.mdindex 8ef2220..7d25e5f 100644--- a/docs/releases/feature-release-notes.md+++ b/docs/releases/feature-release-notes.md@@ -4,6 +4,7 @@| 日期 | 功能域 | 用户价值 | 变更摘要 || --- | --- | --- | --- |+| 2026-09-08 | drag 投递路径可见性 | 在默认安全配置下调用 `drag` 做窗口移动、文本拖选或 Finder 拖放时,结果会明确说明事件只投递给目标进程、为何没有效果,以及需要设置哪个环境变量,不再表现为“成功但无效果”。 | `drag` 结果新增一条 `Drag delivered via app_post` / `Drag delivered via global pointer path` 文本项;tool description 补充默认路径限制与 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 说明;skill 文档新增 Drag Delivery 章节;默认行为与安全门本身不变。 || 2026-09-08 | macOS 应用定位与 secondary action | 按名称定位应用时减少误选后台 helper,执行 secondary action 时避免动作错配。 | 发布 `0.3.4`:优先匹配普通应用;支持 Safari custom action 短名称,并修复过滤后的动作映射和名称冲突。 || 2026-09-06 | macOS secondary action | Agent 可以通过 Safari 暴露的 `close tab` secondary action 可靠关闭标签页,不会因内部动作被过滤而误执行另一项动作。 | macOS renderer 将 `Name:close tab ...` 形式的 AppKit custom action descriptor 显示为短名称,executor 从同一组过滤后的 raw actions 做等价名称匹配,并拒绝歧义匹配;短名称与其他原始 action 冲突时保留完整 descriptor,避免误执行同名原始动作。 || 2026-09-06 | macOS 应用名称解析 | 按名称定位 Safari 等应用时优先匹配真实应用,避免同名后台 helper 抢占匹配结果。 | 按普通应用名称、普通应用 executable、后台应用名称、后台 executable 分级匹配;同级保持原顺序,bundle identifier 和安全过滤规则不变。 |diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex 1fbc48a..ffee113 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -185,6 +185,40 @@ func globalPointerFallbacksEnabled(environment: [String: String]) -> Bool {return ["1", "true", "yes", "on"].contains(rawValue)}+/// How a `drag` is delivered. `drag` has no method argument; the path is decided+/// by the same process-level gate that authorizes `click_method=global`.+enum DragDeliveryPath: String, CaseIterable {+ /// `CGEvent.postToPid`: never moves the system pointer, but the events do not+ /// pass through the window server, so window-server drag sessions (window+ /// moves, text selection, Finder drag-and-drop) are not driven.+ case appPost = "app_post"+ /// `.cghidEventTap`: drives window-server drag sessions and may move the+ /// real pointer or change foreground focus.+ case global+}++func dragDeliveryPath(environment: [String: String]) -> DragDeliveryPath {+ globalPointerFallbacksEnabled(environment: environment) ? .global : .appPost+}++func dragDeliveryNote(for path: DragDeliveryPath) -> String {+ switch path {+ case .appPost:+ return "Drag delivered via app_post: mouse events were posted directly to the target process and the system pointer did not move. This path cannot drive window-server drag sessions such as window moves, text selection, or Finder drag-and-drop. If the drag had no effect, set OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 in the server process environment to use the global pointer path, which may move the real pointer and change foreground focus."+ case .global:+ return "Drag delivered via global pointer path: OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS is enabled, so the real pointer may have moved and foreground focus may have changed."+ }+}++/// Inserts the delivery note after the snapshot text and before any screenshot,+/// so `primaryText` remains the snapshot for existing consumers.+func appendingDragDeliveryNote(to result: ToolCallResult, path: DragDeliveryPath) -> ToolCallResult {+ var content = result.content+ let insertIndex = content.firstIndex { $0.dictionary["type"] as? String == "image" } ?? content.endIndex+ content.insert(.text(dragDeliveryNote(for: path)), at: insertIndex)+ return ToolCallResult(content: content, isError: result.isError)+}+func screenshotPixelScale(screenshotPixelSize: CGSize?,windowBounds: CGRect?@@ -702,13 +736,16 @@ public final class ComputerUseService {let start = try screenshotToGlobalPoint(snapshot: snapshot, x: fromX, y: fromY)let end = try screenshotToGlobalPoint(snapshot: snapshot, x: toX, y: toY)- try performDragEvent(+ let path = try performDragEvent(from: start,to: end,targetDescription: "from=(\(Int(fromX)), \(Int(fromY))) to=(\(Int(toX)), \(Int(toY)))",snapshot: snapshot)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return appendingDragDeliveryNote(+ to: snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult),+ path: path+ )}public func typeText(app query: String, text: String) throws -> ToolCallResult {@@ -1773,11 +1810,13 @@ public final class ComputerUseService {to end: CGPoint,targetDescription: String,snapshot: AppSnapshot- ) throws {+ ) throws -> DragDeliveryPath {let eventStart = inputEventPoint(fromScreenStatePoint: start)let eventEnd = inputEventPoint(fromScreenStatePoint: end)+ let path = dragDeliveryPath(environment: ProcessInfo.processInfo.environment)- if globalPointerFallbacksEnabled(environment: ProcessInfo.processInfo.environment) {+ switch path {+ case .global:debugInputFallback(tool: "drag",targetDescription: targetDescription,@@ -1785,10 +1824,11 @@ public final class ComputerUseService {)InputSimulation.prepareAppForGlobalPointerInput(snapshot.app)try InputSimulation.dragGlobally(from: eventStart, to: eventEnd)- return+ case .appPost:+ try InputSimulation.dragTargeted(from: eventStart, to: eventEnd, pid: snapshot.app.pid)}- try InputSimulation.dragTargeted(from: eventStart, to: eventEnd, pid: snapshot.app.pid)+ return path}private func performNonAXClickFallback(diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swiftindex ce2c8b4..35dff4b 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift@@ -55,7 +55,7 @@ public enum ToolDefinitions {),ToolDefinition(name: "drag",- description: "Drag from one point to another using pixel coordinates. This tool is part of plugin `Computer Use`.",+ description: "Drag from one point to another using pixel coordinates. By default mouse events are posted directly to the target app and the system pointer does not move; that path cannot drive window-server drag sessions such as window moves, text selection, or Finder drag-and-drop. Those require OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 in the server process environment, which may move the real pointer. The result reports which path was used. This tool is part of plugin `Computer Use`.",annotations: defaultAnnotations(),inputSchema: objectSchema(properties: [diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 77b1968..a83e625 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -668,6 +668,10 @@ final class OpenComputerUseKitTests: XCTestCase {((tools["click"]?.inputSchema["properties"] as? [String: [String: Any]])?["click_method"]?["enum"] as? [String]) ?? [],["auto", "accessibility", "app_post", "sky_click", "global"])+ let dragDescription = tools["drag"]?.description ?? ""+ XCTAssertTrue(dragDescription.contains("OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1"))+ XCTAssertTrue(dragDescription.contains("window moves, text selection, or Finder drag-and-drop"))+ XCTAssertTrue(dragDescription.hasSuffix("This tool is part of plugin `Computer Use`."))let getAppStateSchema = tools["get_app_state"]?.inputSchemalet getAppStateProperties = getAppStateSchema?["properties"] as? [String: [String: Any]]XCTAssertNil(getAppStateProperties?["show_full_text"])@@ -1471,6 +1475,49 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertFalse(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "false"]))}+ func testDragDeliveryPathFollowsGlobalPointerGate() {+ XCTAssertEqual(dragDeliveryPath(environment: [:]), .appPost)+ XCTAssertEqual(dragDeliveryPath(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "0"]), .appPost)+ XCTAssertEqual(dragDeliveryPath(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "1"]), .global)+ XCTAssertEqual(dragDeliveryPath(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": " True "]), .global)+ }++ func testDragDeliveryNoteExplainsWhyDefaultPathCannotDriveWindowServerDrags() {+ let appPostNote = dragDeliveryNote(for: .appPost)+ XCTAssertTrue(appPostNote.hasPrefix("Drag delivered via app_post"))+ XCTAssertTrue(appPostNote.contains("system pointer did not move"))+ XCTAssertTrue(appPostNote.contains("window moves"))+ XCTAssertTrue(appPostNote.contains("text selection"))+ XCTAssertTrue(appPostNote.contains("Finder drag-and-drop"))+ XCTAssertTrue(appPostNote.contains("OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1"))++ let globalNote = dragDeliveryNote(for: .global)+ XCTAssertTrue(globalNote.hasPrefix("Drag delivered via global pointer path"))+ XCTAssertTrue(globalNote.contains("real pointer may have moved"))+ }++ func testDragDeliveryNoteIsInsertedAfterSnapshotTextAndBeforeScreenshot() {+ let snapshotText = "App=com.example.app (pid 42)\nWindow: \"Example\", App: Example."+ let result = ToolCallResult(content: [.text(snapshotText), .pngImage(Data([0x89, 0x50, 0x4E, 0x47]))])++ let annotated = appendingDragDeliveryNote(to: result, path: .appPost)++ XCTAssertEqual(annotated.primaryText, snapshotText)+ XCTAssertEqual(annotated.content.count, 3)+ XCTAssertEqual(annotated.content[1].dictionary["type"] as? String, "text")+ XCTAssertEqual(annotated.content[1].dictionary["text"] as? String, dragDeliveryNote(for: .appPost))+ XCTAssertEqual(annotated.content[2].dictionary["type"] as? String, "image")+ XCTAssertFalse(annotated.isError)+ }++ func testDragDeliveryNoteIsAppendedWhenResultHasNoScreenshot() {+ let annotated = appendingDragDeliveryNote(to: .text("App=com.example.app (pid 42)"), path: .global)++ XCTAssertEqual(annotated.primaryText, "App=com.example.app (pid 42)")+ XCTAssertEqual(annotated.content.count, 2)+ XCTAssertEqual(annotated.content[1].dictionary["text"] as? String, dragDeliveryNote(for: .global))+ }+func testClickMethodDefaultsToAutoAndNormalizesExplicitValues() throws {XCTAssertEqual(try parseClickMethod(nil), .auto)XCTAssertEqual(try parseClickMethod(" AUTO "), .auto)diff --git a/skills/open-computer-use/SKILL.md b/skills/open-computer-use/SKILL.mdindex 014ef60..1120536 100644--- a/skills/open-computer-use/SKILL.md+++ b/skills/open-computer-use/SKILL.md@@ -36,7 +36,7 @@ It supports the same core tool surface across macOS, Linux, and Windows:- Do not assume Codex.app plugin helpers are available. Use the installed `open-computer-use` / `ocu` CLI or an explicit MCP config.- Always run `get_app_state` before using `element_index`; do not guess indexes across sessions or after large UI changes.- Prefer semantic actions and `set_value` for editable controls. Use coordinate `click`, `scroll`, and `drag` only when the element tree does not expose a safer target.-- On macOS, do not enable `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` unless the user explicitly requested `click_method: "global"` or other diagnostic behavior that may move the real pointer.+- On macOS, do not enable `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` unless the user explicitly requested `click_method: "global"`, a `drag` that must drive a window-server drag session (window move, drag-select text, Finder drag-and-drop), or other diagnostic behavior that may move the real pointer. Without it `drag` reports `Drag delivered via app_post` and those operations have no effect; see `references/usage.md` for alternatives.- On Windows and Linux, confirm the command is running inside the logged-in desktop session before assuming GUI automation is available.## Common CLI Actionsdiff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.mdindex ea1ffb5..a4793c5 100644--- a/skills/open-computer-use/references/usage.md+++ b/skills/open-computer-use/references/usage.md@@ -139,6 +139,17 @@ Keep the environment override scoped as narrowly as possible. While it remains eWindows returns an unsupported error for `sky_click` and `global`; Linux returns an unsupported error for `app_post` and `sky_click`. An unsupported or failed explicit method does not fall back to `auto`.+## Drag Delivery++`drag` has no method parameter. On macOS the path it takes is decided by the same process-level gate that authorizes `click_method: "global"`:++- Gate unset (default): mouse move / down / dragged / up events are posted directly to the target process with `CGEvent.postToPid`. The system pointer does not move and foreground focus is unchanged. Because the events never pass through the window server, this path cannot start a window-server drag session: window moves, drag-selecting text, and Finder drag-and-drop return without error but have no visible effect.+- `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` set for the server process: the drag uses the global pointer path, which drives window-server drag sessions but may move the real pointer and change foreground focus.++Every non-fixture `drag` result includes a text item that begins `Drag delivered via app_post` or `Drag delivered via global pointer path`, so a default drag that did nothing is legible instead of looking like success. For an MCP server the variable belongs in the server entry's `env`, not in the calling shell, and the server must be restarted afterwards.++When the gate is not enabled, treat window-server drags as unavailable and reach the same outcome another way: copy or move files with a shell command instead of a Finder drag, use `set_value` or keyboard selection instead of drag-selecting text, and use the app's own window controls instead of dragging a title bar.+## Platform Notes### macOS