👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Refine cursor call tooling and repo guidance
usoon.xuca91aa1d65ca
diff --git a/AGENTS.md b/AGENTS.mdindex 5675189..3a20bc0 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -34,6 +34,7 @@## 工作规则- 优先选择小而清晰、对仓库和 Agent 都友好的抽象。+- 回复默认跟随用户提问所使用的语言;如果用户切换语言,回复语言也随之切换。- prompt、规则、架构约束尽量都版本化落在仓库里。- 复杂任务不要只靠聊天上下文,应该落 execution plan。- 完成的代码变更要记到 `docs/histories/`。diff --git a/README.md b/README.mdindex 7d26dff..90240cb 100644--- a/README.md+++ b/README.md@@ -54,7 +54,9 @@ open-computer-use mcpopen-computer-use call list_appsopen-computer-use call get_app_state --args '{"app":"TextEdit"}'# Run a sequence in one process so element_index state can be reused+# Sequence runs sleep 1s between successful operations by defaultopen-computer-use call --calls '[{"tool":"get_app_state","args":{"app":"TextEdit"}},{"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}]'+open-computer-use call --calls-file examples/textedit-overlay-seq.json --sleep 0.5# Check permissions; onboarding only opens when something is missingopen-computer-use doctor# Show helpdiff --git a/README.zh-CN.md b/README.zh-CN.mdindex 9d6bad3..0c05cfd 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -59,7 +59,9 @@ open-computer-use call list_appsopen-computer-use call get_app_state --args '{"app":"TextEdit"}'# 在同一个进程里编排连续动作,复用 get_app_state 拿到的 element_index+# 连续动作默认会在成功的相邻操作之间 sleep 1 秒open-computer-use call --calls '[{"tool":"get_app_state","args":{"app":"TextEdit"}},{"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}]'+open-computer-use call --calls-file examples/textedit-overlay-seq.json --sleep 0.5# 检查权限;只有缺失时才会拉起引导,已全部授权则只打印状态并退出open-computer-use doctordiff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex a89a1b4..56f5ac8 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -57,11 +57,11 @@- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。-- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行遇到 `isError=true` 的 tool result 后停止。+- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。- 对真实 app 的 `get_app_state` / action tool 入口,当前新增了一层官方风格的高风险 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app,尽量贴近官方对终端、密码管理器、Chrome 与少量系统敏感组件的防护行为。- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。- `click` / `set_value` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:两者的移动阶段现在共用一条 heading-driven 的官方风格 motion 内核,显式把“当前 cursor 朝向”和“最终 resting pose”一起喂给选路器,优先生成需要时先掉头、再沿车头方向推进的 C 形/单侧大弧轨迹;首次显示时按官方 binary 的 fresh state 从 AppKit 全局 `(0,0)` window origin 生成起点,后续动作继续复用上一帧 visible tip。真正显示出来的 cursor 不再直接等于 path sample,而是经过一层独立的 visual dynamics 状态,把 visible tip、velocity、angle 和 fog/offset 持续推进。`click` 结尾会衔接 click pulse 和 idle sway,`set_value` 则只做 settle / idle,不给 pulse;两者收尾后都会像官方 service 一样保持短期 idle 状态,约 5 分钟无后续操作才做 cleanup,这样连续 tool call 不会反复从 fresh `(0,0)` 起步;如果宿主在任务 / turn 结束时发出 `turn-ended`,cursor 会立即消失并清掉本轮位置状态。-- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在和 `CursorMotion` 一样优先渲染仓库里沉淀的 `official-software-cursor-window-252.png` baseline,只有资源缺失时才退回 `OpenComputerUseKit` 内部的程序化 pointer/fog fallback。命中点 anchor 仍固定在 `126x126` 画布里的同一组 tip-offset 上;glyph 自身的 neutral heading 继续沿用 `CursorMotion` / 官方 baseline 的 `-3π/4`。主 runtime overlay window 按 AppKit 全局坐标移动;路径选路用屏幕上实际可见的 AppKit forward heading,进入 visual dynamics / render state 前则把 velocity 的 y 轴翻回 CursorMotion 的 y-down screen state,再交给 AppKit 绘制层做角度和 `dy` 翻转。程序化 fallback 保留 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。+- overlay 的 visual style 不再自己从官方 app bundle 裁 `SoftwareCursor` 小图;主 MCP runtime 现在和 `CursorMotion` 一样优先渲染仓库里沉淀的 `official-software-cursor-window-252.png` baseline,只有资源缺失时才退回 `OpenComputerUseKit` 内部的程序化 pointer/fog fallback。命中点 anchor 仍固定在 `126x126` 画布里的同一组 tip-offset 上;glyph 自身的 neutral heading 继续沿用 `CursorMotion` / 官方 baseline 的 `-3π/4`。主 runtime overlay window 按 AppKit 全局坐标移动,因此在把 AX / `CGWindowList` 产出的 y-down screen-space 点击目标喂给 overlay 之前,会先转换成对应屏幕的 AppKit 全局坐标;路径选路用屏幕上实际可见的 AppKit forward heading,进入 visual dynamics / render state 前则把 velocity 的 y 轴翻回 CursorMotion 的 y-down screen state,再交给 AppKit 绘制层做角度和 `dy` 翻转。程序化 fallback 保留 neutral artwork correction,把它的天然轮廓轴对齐到 `CursorMotion` / 官方 baseline 的 `-3π/4` forward 方向,但不让实验线依赖 runtime 代码。- overlay 的层级不再固定 `.floating`;现在会跟随 snapshot 命中的目标 window id / layer,把自己排到该目标 window 之上,而不是粗暴压到所有前台 app 最上层。- overlay 的曲线路径不再只按固定 Bezier 模板生成;当前主线采用 reverse-engineering 约束下的 heading-driven candidate 族,候选只保留 `direct` / `turn` / `brake` / `orbit` 这些能稳定产出单侧主弧的 family,并继续保留 target-window 命中策略作为同类候选间的 tie-break。原始 binary lift 恢复出来的 `20` 条路径和 score 仍然保留在独立的 `StandaloneCursor` viewer / Python 重建脚本里,用于对照分析,不再直接作为 runtime 默认 chooser。- overlay 的 progress 曲线也不再是固定 `easeInOut`;主线现在复用官方 `response=1.4`、`dampingFraction=0.9`、`dt=1/240` 的 spring/`VelocityVerlet` 形状,但 wall-clock duration 仍做本地校准,因为官方 transaction-level 时长映射还没有完全恢复。diff --git a/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md b/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.mdindex fc8d3e8..322a394 100644--- a/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md+++ b/docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md@@ -34,7 +34,7 @@- **[修正本地 click 决策]**: `ComputerUseService.performPreferredClick` 不再只会试 `AXPress` / `kAXFocusedAttribute` / `AXConfirm`;现在会把窗口/根元素常见的 `AXRaise`、`kAXMainAttribute`、`kAXFocusedAttribute` 都纳入 element-targeted 左键路径,只有这些都失败才允许 `clickGlobally(...)`。- **[移除过严 settable 门槛]**: 针对 `TextEdit` window 这类 `AXUIElementIsAttributeSettable(kAXFocusedAttribute) == false`、但直接 `AXUIElementSetAttributeValue(..., true)` 仍成功的场景,`click` 相关的布尔属性写入不再把 `isSettable` 当作硬前置条件。- **[补 fallback tracing]**: 新增默认关闭的 `OPEN_COMPUTER_USE_DEBUG_INPUT_FALLBACKS` 环境变量;打开后,只有真的命中全局 pointer fallback 才会往 stderr 打一行调试信息,便于继续做官方/本地 A/B。-- **[验证结果]**: 打开 `OPEN_COMPUTER_USE_DEBUG_INPUT_FALLBACKS=1` 后,当前 `scripts/computer-use-cli/examples/textedit-overlay-seq.json` 这条本地 direct MCP 序列已不再打印 `global pointer fallback`,同时 `swift test` 全绿。+- **[验证结果]**: 打开 `OPEN_COMPUTER_USE_DEBUG_INPUT_FALLBACKS=1` 后,当前 `examples/textedit-overlay-seq.json` 这条本地 direct MCP 序列已不再打印 `global pointer fallback`,同时 `swift test` 全绿。**Follow-up Files:**- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`diff --git a/docs/histories/2026-04/20260422-1338-fix-visual-cursor-coordinate-space.md b/docs/histories/2026-04/20260422-1338-fix-visual-cursor-coordinate-space.mdnew file mode 100644index 0000000..32c1101--- /dev/null+++ b/docs/histories/2026-04/20260422-1338-fix-visual-cursor-coordinate-space.md@@ -0,0 +1,26 @@+## [2026-04-22 13:38] | Task: Fix visual cursor coordinate space++### Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `Codex CLI`++### User Query+> When running `swift run OpenComputerUse call --calls-file examples/textedit-overlay-seq.json`, the virtual cursor does not hover over the right place.++### Changes Overview+**Scope:** `OpenComputerUseKit` visual cursor target conversion, tests, architecture docs++**Key Actions:**+- **[Coordinate conversion fix]**: Converted visual cursor target points from AX / `CGWindowList` screen-space into AppKit global coordinates before moving the overlay window.+- **[Regression coverage]**: Updated visual cursor target tests and added a focused conversion test for y-down screen-space to AppKit global placement.+- **[Docs sync]**: Clarified the overlay coordinate-space boundary in `docs/ARCHITECTURE.md`.++### Design Intent (Why)+The actual click path is mostly AX-driven, but the visible overlay is an AppKit panel. TextEdit toolbar targets exposed that these two paths were using different global coordinate conventions, so the click could succeed while the cursor animation visibly landed in the wrong vertical position.++### Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-04/20260422-1338-fix-visual-cursor-coordinate-space.md`diff --git a/docs/histories/2026-04/20260422-1403-sequence-call-inter-operation-sleep.md b/docs/histories/2026-04/20260422-1403-sequence-call-inter-operation-sleep.mdnew file mode 100644index 0000000..6a1d92b--- /dev/null+++ b/docs/histories/2026-04/20260422-1403-sequence-call-inter-operation-sleep.md@@ -0,0 +1,29 @@+## [2026-04-22 14:03] | Task: Add sequence call inter-operation sleep++### Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `Codex CLI`++### User Query+> 给 `swift run OpenComputerUse call --calls-file examples/textedit-overlay-seq.json` 这类 sequence call 加一个固定间隔,做成可选参数,默认值 1 秒。++### Changes Overview+**Scope:** `OpenComputerUseKit` CLI parsing, sequence execution, tests, docs++**Key Actions:**+- **[Sequence sleep option]**: Added a sequence-only `--sleep <seconds>` CLI option for `open-computer-use call --calls ...` / `--calls-file ...`, with a default inter-operation delay of 1 second.+- **[Execution behavior]**: Applied the delay only between successful neighboring sequence operations; single-tool calls still run without an extra wrapper sleep.+- **[Docs and coverage]**: Updated CLI help, README, architecture notes, and unit tests to lock in the default and custom-delay behavior.++### Design Intent (Why)+The sequence runner already preserves in-process app snapshot state, but it previously fired each step immediately after the previous one returned. Adding a small default gap makes visual cursor demos and real-app multi-step probes easier to observe, while `--sleep` keeps the pacing override explicit instead of hard-coding one fixed rhythm forever.++### Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `README.md`+- `README.zh-CN.md`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-04/20260422-1403-sequence-call-inter-operation-sleep.md`diff --git a/docs/histories/2026-04/20260422-1447-add-language-mirroring-rule-to-agents.md b/docs/histories/2026-04/20260422-1447-add-language-mirroring-rule-to-agents.mdnew file mode 100644index 0000000..1e6d34b--- /dev/null+++ b/docs/histories/2026-04/20260422-1447-add-language-mirroring-rule-to-agents.md@@ -0,0 +1,23 @@+## [2026-04-22 14:47] | Task: add language mirroring rule++### 🤖 Execution Context+* **Agent ID**: `019db3f1-0538-7a70-bdd4-19395299085c`+* **Base Model**: `GPT-5 Codex`+* **Runtime**: `Codex CLI`++### 📥 User Query+> add in agent.md, reply as the same language as the user query++### 🛠 Changes Overview+**Scope:** `AGENTS.md`, `docs/histories/`++**Key Actions:**+- **[Add rule]**: 在 `AGENTS.md` 的工作规则里新增“回复跟随用户提问语言”的约束。+- **[Record change]**: 新增对应 history,记录这次仓库级协作规则调整。++### 🧠 Design Intent (Why)+把语言跟随规则放进仓库入口约束里,能让后续 Agent 在协作时直接继承统一行为,减少对聊天上下文的依赖。++### 📁 Files Modified+- `AGENTS.md`+- `docs/histories/2026-04/20260422-1447-add-language-mirroring-rule-to-agents.md`diff --git a/docs/histories/2026-04/20260422-1452-sanitize-example-paths-and-sample-app-names.md b/docs/histories/2026-04/20260422-1452-sanitize-example-paths-and-sample-app-names.mdnew file mode 100644index 0000000..3970b54--- /dev/null+++ b/docs/histories/2026-04/20260422-1452-sanitize-example-paths-and-sample-app-names.md@@ -0,0 +1,29 @@+## [2026-04-22 14:52] | Task: 收口示例路径并脱敏样例应用名++### Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `Codex CLI`++### User Query+> 清理 `computer-use-cli` 相关示例和文档引用,统一改成仓库根目录样例路径,并避免继续在 README、测试和参考日志里出现真实产品名。++### Changes Overview+**Scope:** 示例路径、helper 文档、测试样例、参考日志++**Key Actions:**+- **[Example path consolidation]**: 删除 `scripts/computer-use-cli/` 下重复的 sequence JSON,统一改为引用根目录 `examples/textedit-overlay-seq.json`。+- **[Sample name sanitization]**: 将 helper README、Go 测试、Swift 测试和参考日志里的具体应用名改成 `TextEdit` 或 `Sample Chat` 这类通用样例。+- **[History sync]**: 同步修正既有 history 中已经过时的样例路径描述,避免文档继续指向已删除位置。++### Design Intent (Why)+这些样例本来只是为了说明调用形状,不应该继续绑定到某个真实产品名,也不应该在仓库里维护两份内容重复的 sequence 文件。把路径和命名统一后,后续手工验证、README 示例和测试断言会更稳定,也更适合开源仓库继续演进。++### Files Modified+- `scripts/computer-use-cli/README.md`+- `scripts/computer-use-cli/main.go`+- `scripts/computer-use-cli/main_test.go`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/references/codex-local-runtime-logs.md`+- `docs/histories/2026-04/20260420-2033-integrate-set-value-visual-cursor.md`+- `docs/histories/2026-04/20260422-1452-sanitize-example-paths-and-sample-app-names.md`diff --git a/docs/references/codex-local-runtime-logs.md b/docs/references/codex-local-runtime-logs.mdindex f0b4d1e..71e730d 100644--- a/docs/references/codex-local-runtime-logs.md+++ b/docs/references/codex-local-runtime-logs.md@@ -62,11 +62,11 @@ sqlite3 -readonly "$HOME/.codex/logs_2.sqlite" ".tables"对官方 `computer-use`,常见形态类似:```text-ToolCall: mcp__computer_use__click {"app":"com.electron.lark","x":194,"y":321}+ToolCall: mcp__computer_use__click {"app":"com.example.SampleChat","x":194,"y":321}``````text-event.name="codex.tool_result" tool_name=mcp__computer_use__click ... arguments={"app":"com.electron.lark","x":194,"y":321} ... output=...+event.name="codex.tool_result" tool_name=mcp__computer_use__click ... arguments={"app":"com.example.SampleChat","x":194,"y":321} ... output=...```当日志粒度足够时,还能看到:diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex 2cb6c45..a63c750 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -7,13 +7,52 @@ struct VisualCursorTarget: Equatable {let window: CursorTargetWindow?}+struct VisualCursorScreenMapping: Equatable {+ let screenStateFrame: CGRect+ let appKitFrame: CGRect+}++func currentVisualCursorScreenMappings() -> [VisualCursorScreenMapping] {+ NSScreen.screens.compactMap { screen in+ guard let screenNumber = screen.deviceDescription[NSDeviceDescriptionKey("NSScreenNumber")] as? NSNumber else {+ return nil+ }++ return VisualCursorScreenMapping(+ screenStateFrame: CGDisplayBounds(CGDirectDisplayID(screenNumber.uint32Value)),+ appKitFrame: screen.frame+ )+ }+}++func visualCursorAppKitPoint(+ fromScreenStatePoint point: CGPoint,+ screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()+) -> CGPoint {+ guard let mapping = screenMappings.first(where: { $0.screenStateFrame.contains(point) }) else {+ return point+ }++ let localX = point.x - mapping.screenStateFrame.minX+ let localY = point.y - mapping.screenStateFrame.minY++ return CGPoint(+ x: mapping.appKitFrame.minX + localX,+ y: mapping.appKitFrame.maxY - localY+ )+}+func makeVisualCursorTarget(at point: CGPoint,targetWindowID: CGWindowID?,- targetWindowLayer: Int?+ targetWindowLayer: Int?,+ screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()) -> VisualCursorTarget {VisualCursorTarget(- point: point,+ point: visualCursorAppKitPoint(+ fromScreenStatePoint: point,+ screenMappings: screenMappings+ ),window: targetWindowID.map { CursorTargetWindow(windowID: $0, layer: targetWindowLayer ?? 0) })}@@ -22,7 +61,8 @@ func makeVisualCursorTarget(localFrame: CGRect?,windowBounds: CGRect?,targetWindowID: CGWindowID?,- targetWindowLayer: Int?+ targetWindowLayer: Int?,+ screenMappings: [VisualCursorScreenMapping] = currentVisualCursorScreenMappings()) -> VisualCursorTarget? {guard let localFrame, let windowBounds else {return nil@@ -35,7 +75,8 @@ func makeVisualCursorTarget(return makeVisualCursorTarget(at: point,targetWindowID: targetWindowID,- targetWindowLayer: targetWindowLayer+ targetWindowLayer: targetWindowLayer,+ screenMappings: screenMappings)}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swiftindex da66df1..d327a07 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift@@ -146,9 +146,12 @@ public struct OpenComputerUseCallOutput {}}+public typealias OpenComputerUseSleepHandler = (TimeInterval) -> Void+public func runOpenComputerUseCall(_ invocation: OpenComputerUseCallInvocation,- service: ComputerUseService = ComputerUseService()+ service: ComputerUseService = ComputerUseService(),+ sleepHandler: OpenComputerUseSleepHandler = { Thread.sleep(forTimeInterval: $0) }) throws -> OpenComputerUseCallOutput {let dispatcher = ComputerUseToolDispatcher(service: service)@@ -164,12 +167,12 @@ public func runOpenComputerUseCall(hasToolError: result.isError)- case let .sequence(callsJSON, callsFile):+ case let .sequence(callsJSON, callsFile, interCallDelay):let calls = try readOpenComputerUseCallSequence(json: callsJSON, file: callsFile)var outputs: [[String: Any]] = []var hasToolError = false- for call in calls {+ for (index, call) in calls.enumerated() {let result = dispatcher.callToolAsResult(name: call.tool, arguments: call.arguments)outputs.append(["tool": call.tool,@@ -180,6 +183,10 @@ public func runOpenComputerUseCall(hasToolError = truebreak}++ if index < calls.count - 1, interCallDelay > 0 {+ sleepHandler(interCallDelay)+ }}return OpenComputerUseCallOutput(jsonObject: outputs, hasToolError: hasToolError)diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swiftindex fd5fab3..9db25f8 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift@@ -14,9 +14,11 @@ public enum OpenComputerUseCLICommand: Equatable {public enum OpenComputerUseCallInvocation: Equatable {case single(toolName: String, argumentsJSON: String?, argumentsFile: String?)- case sequence(callsJSON: String?, callsFile: String?)+ case sequence(callsJSON: String?, callsFile: String?, interCallDelay: TimeInterval)}+public let openComputerUseDefaultInterCallDelay: TimeInterval = 1+public struct OpenComputerUseCLIError: LocalizedError, Equatable {public let message: Stringpublic let helpCommand: String?@@ -138,17 +140,19 @@ public func openComputerUseHelpText(command: String? = nil) -> String {Usage:open-computer-use call <tool> [--args '<json-object>']open-computer-use call <tool> [--args-file <path>]- open-computer-use call --calls '<json-array>'- open-computer-use call --calls-file <path>+ open-computer-use call --calls '<json-array>' [--sleep <seconds>]+ open-computer-use call --calls-file <path> [--sleep <seconds>]Examples:open-computer-use call list_appsopen-computer-use call get_app_state --args '{"app":"TextEdit"}'open-computer-use call --calls '[{"tool":"get_app_state","args":{"app":"TextEdit"}},{"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}]'+ open-computer-use call --calls-file examples/textedit-overlay-seq.json --sleep 0.5The JSON array form keeps all calls in one process so follow-up actionscan reuse the app state and element indices captured by get_app_state.Sequence execution stops after the first tool result with isError=true.+ Sequence runs sleep \(formatOpenComputerUseDelay(openComputerUseDefaultInterCallDelay)) between successful operations by default."""case "turn-ended":return """@@ -263,6 +267,7 @@ private func parseCall(arguments: [String]) throws -> OpenComputerUseCLICommandvar argumentsFile: String?var callsJSON: String?var callsFile: String?+ var interCallDelay = openComputerUseDefaultInterCallDelayvar index = 0while index < arguments.count {@@ -277,6 +282,8 @@ private func parseCall(arguments: [String]) throws -> OpenComputerUseCLICommandcallsJSON = try parseOptionValue("--calls", arguments: arguments, index: &index)case "--calls-file":callsFile = try parseOptionValue("--calls-file", arguments: arguments, index: &index)+ case "--sleep":+ interCallDelay = try parseTimeIntervalOptionValue("--sleep", arguments: arguments, index: &index)case "-h", "--help":throw OpenComputerUseCLIError(message: "call help must be requested as `open-computer-use call --help`", helpCommand: "call")default:@@ -307,13 +314,24 @@ private func parseCall(arguments: [String]) throws -> OpenComputerUseCLICommand)}- return .call(.sequence(callsJSON: callsJSON, callsFile: callsFile))+ return .call(.sequence(+ callsJSON: callsJSON,+ callsFile: callsFile,+ interCallDelay: interCallDelay+ ))}if argumentsJSON != nil, argumentsFile != nil {throw OpenComputerUseCLIError(message: "Use either --args or --args-file, not both", helpCommand: "call")}+ if interCallDelay != openComputerUseDefaultInterCallDelay {+ throw OpenComputerUseCLIError(+ message: "--sleep is only supported with --calls or --calls-file",+ helpCommand: "call"+ )+ }+guard let toolName else {throw OpenComputerUseCLIError(message: "call requires a tool name or --calls/--calls-file", helpCommand: "call")}@@ -334,3 +352,27 @@ private func parseOptionValue(index = valueIndexreturn arguments[valueIndex]}++private func parseTimeIntervalOptionValue(+ _ option: String,+ arguments: [String],+ index: inout Int+) throws -> TimeInterval {+ let rawValue = try parseOptionValue(option, arguments: arguments, index: &index)+ guard let value = Double(rawValue), value.isFinite, value >= 0 else {+ throw OpenComputerUseCLIError(+ message: "\(option) requires a non-negative number of seconds",+ helpCommand: "call"+ )+ }++ return value+}++private func formatOpenComputerUseDelay(_ delay: TimeInterval) -> String {+ if delay.rounded() == delay {+ return "\(Int(delay))s"+ }++ return "\(delay)s"+}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 0a78a5c..0e1474e 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -34,7 +34,20 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(try parseOpenComputerUseCLI(arguments: ["call", "--calls", calls]),- .call(.sequence(callsJSON: calls, callsFile: nil))+ .call(.sequence(+ callsJSON: calls,+ callsFile: nil,+ interCallDelay: openComputerUseDefaultInterCallDelay+ ))+ )+ }++ func testCLIRecognizesJSONSequenceCallCommandWithCustomSleep() throws {+ let calls = #"[{"tool":"get_app_state","args":{"app":"TextEdit"}},{"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}]"#++ XCTAssertEqual(+ try parseOpenComputerUseCLI(arguments: ["call", "--calls", calls, "--sleep", "0.5"]),+ .call(.sequence(callsJSON: calls, callsFile: nil, interCallDelay: 0.5)))}@@ -73,6 +86,30 @@ final class OpenComputerUseKitTests: XCTestCase {}}+ func testCLIRejectsSleepForSingleToolCall() {+ XCTAssertThrowsError(try parseOpenComputerUseCLI(arguments: ["call", "list_apps", "--sleep", "0.5"])) { error in+ XCTAssertEqual(+ error as? OpenComputerUseCLIError,+ OpenComputerUseCLIError(+ message: "--sleep is only supported with --calls or --calls-file",+ helpCommand: "call"+ )+ )+ }+ }++ func testCLIRejectsInvalidSequenceSleepValue() {+ XCTAssertThrowsError(try parseOpenComputerUseCLI(arguments: ["call", "--calls", "[]", "--sleep", "-1"])) { error in+ XCTAssertEqual(+ error as? OpenComputerUseCLIError,+ OpenComputerUseCLIError(+ message: "--sleep requires a non-negative number of seconds",+ helpCommand: "call"+ )+ )+ }+ }+func testCLIRejectsUnknownOption() {XCTAssertThrowsError(try parseOpenComputerUseCLI(arguments: ["--verbose"])) { error inXCTAssertEqual(@@ -139,7 +176,8 @@ final class OpenComputerUseKitTests: XCTestCase {let output = try runOpenComputerUseCall(.sequence(callsJSON: #"[{"tool":"not_a_tool"},{"tool":"list_apps"}]"#,- callsFile: nil+ callsFile: nil,+ interCallDelay: openComputerUseDefaultInterCallDelay))@@ -148,6 +186,24 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertTrue(output.hasToolError)}+ func testRunCallSequenceSleepsBetweenSuccessfulOperations() throws {+ var recordedSleeps: [TimeInterval] = []++ let output = try runOpenComputerUseCall(+ .sequence(+ callsJSON: #"[{"tool":"list_apps"},{"tool":"list_apps"},{"tool":"list_apps"}]"#,+ callsFile: nil,+ interCallDelay: openComputerUseDefaultInterCallDelay+ ),+ sleepHandler: { recordedSleeps.append($0) }+ )++ let outputs = try XCTUnwrap(output.jsonObject as? [[String: Any]])+ XCTAssertEqual(outputs.count, 3)+ XCTAssertEqual(recordedSleeps, [openComputerUseDefaultInterCallDelay, openComputerUseDefaultInterCallDelay])+ XCTAssertFalse(output.hasToolError)+ }+func testPermissionDiagnosticsListsMissingPermissionsInCanonicalOrder() {let diagnostics = PermissionDiagnostics(accessibilityTrusted: false,@@ -379,15 +435,15 @@ final class OpenComputerUseKitTests: XCTestCase {func testSnapshotRenderedTextStartsDirectlyWithAppHeader() {let snapshot = makeSnapshot(- treeLines: ["\t0 standard window Feishu"],+ treeLines: ["\t0 standard window Sample Chat"],focusedSummary: "247 text entry area")let rendered = snapshot.renderedText(style: .actionResult)let lines = rendered.components(separatedBy: "\n")- XCTAssertEqual(lines.first, "App=com.electron.lark (pid 18465)")- XCTAssertEqual(lines.dropFirst().first, "Window: \"Feishu\", App: Feishu.")+ XCTAssertEqual(lines.first, "App=com.example.SampleChat (pid 18465)")+ XCTAssertEqual(lines.dropFirst().first, "Window: \"Sample Chat\", App: Sample Chat.")XCTAssertFalse(rendered.contains("Computer Use state (CUA App Version: 750)"))XCTAssertFalse(rendered.contains("<app_state>"))XCTAssertFalse(rendered.contains("</app_state>"))@@ -452,17 +508,24 @@ final class OpenComputerUseKitTests: XCTestCase {}func testMakeVisualCursorTargetUsesWindowRelativeElementCenter() {+ let screenMappings = [+ VisualCursorScreenMapping(+ screenStateFrame: CGRect(x: 0, y: 0, width: 1600, height: 1000),+ appKitFrame: CGRect(x: 0, y: 0, width: 1600, height: 1000)+ ),+ ]let target = makeVisualCursorTarget(localFrame: CGRect(x: 24, y: 32, width: 120, height: 48),windowBounds: CGRect(x: 400, y: 220, width: 900, height: 640),targetWindowID: 321,- targetWindowLayer: 8+ targetWindowLayer: 8,+ screenMappings: screenMappings)XCTAssertEqual(target,VisualCursorTarget(- point: CGPoint(x: 484, y: 276),+ point: CGPoint(x: 484, y: 724),window: CursorTargetWindow(windowID: 321, layer: 8)))@@ -479,6 +542,20 @@ final class OpenComputerUseKitTests: XCTestCase {)}+ func testVisualCursorAppKitPointConvertsScreenStateYDownCoordinates() {+ let point = visualCursorAppKitPoint(+ fromScreenStatePoint: CGPoint(x: 2415, y: 181),+ screenMappings: [+ VisualCursorScreenMapping(+ screenStateFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964),+ appKitFrame: CGRect(x: 0, y: 0, width: 3024, height: 1964)+ ),+ ]+ )++ XCTAssertEqual(point, CGPoint(x: 2415, y: 1783))+ }+func testCursorWindowGeometryAnchorsTipPosition() {let geometry = CursorWindowGeometry(windowSize: CGSize(width: 128, height: 128),@@ -764,12 +841,12 @@ final class OpenComputerUseKitTests: XCTestCase {private func makeSnapshot(treeLines: [String], focusedSummary: String?, selectedText: String? = nil) -> AppSnapshot {AppSnapshot(app: RunningAppDescriptor(- name: "Feishu",- bundleIdentifier: "com.electron.lark",+ name: "Sample Chat",+ bundleIdentifier: "com.example.SampleChat",pid: 18_465,runningApplication: NSRunningApplication.current),- windowTitle: "Feishu",+ windowTitle: "Sample Chat",windowBounds: nil,targetWindowID: nil,targetWindowLayer: nil,diff --git a/scripts/computer-use-cli/README.md b/scripts/computer-use-cli/README.mdindex b7397b1..9c51df5 100644--- a/scripts/computer-use-cli/README.md+++ b/scripts/computer-use-cli/README.md@@ -27,7 +27,7 @@ It supports two transports:go run . resolve-servergo run . list-tools --transport app-servergo run . call list_apps --transport direct --server-bin /path/to/open-computer-use-go run . call get_app_state --transport direct --server-bin /path/to/open-computer-use --args '{"app":"Feishu"}'+go run . call get_app_state --transport direct --server-bin /path/to/open-computer-use --args '{"app":"TextEdit"}'go run . call-seq --transport direct --server-bin /path/to/open-computer-use --calls-file /tmp/calls.json```@@ -60,19 +60,11 @@ followed by one or more action tools in the same official `computer-use` session```This repo includes a ready-to-run example at-`examples/textedit-overlay-seq.json`.-That sample covers all 9 official tools against `TextEdit`, and it intentionally-repeats `get_app_state` between state-changing actions because the official-bundled `computer-use` invalidates the cached app state after each mutation.-Before running it, make sure `TextEdit` already has a normal document window open;-otherwise `get_app_state` may only expose the application root and the hard-coded-`element_index` values in the sample will no longer line up.--For a smaller cursor-comparison path, use-`examples/textedit-set-value-click-raise-seq.json`.-That sample stays on the `set_value -> click -> Raise` flow so the same calls file-can be pointed at both the official bundled `computer-use` and the local-`open-computer-use` binary.+`../../examples/textedit-overlay-seq.json`.++Before running it, make sure `TextEdit` already has a normal document window+open; otherwise `get_app_state` may only expose the application root and the+hard-coded `element_index` values in the sample will no longer line up.## Default target@@ -151,7 +143,7 @@ go run . call-seq \--transport direct \--plugin-root ../.. \--server-bin ../../.build/debug/OpenComputerUse \- --calls-file examples/textedit-set-value-click-raise-seq.json+ --calls-file ../../examples/textedit-overlay-seq.json```Example inventory probe against the official bundled `computer-use`:diff --git a/scripts/computer-use-cli/examples/textedit-overlay-seq.json b/scripts/computer-use-cli/examples/textedit-overlay-seq.jsondeleted file mode 100644index 25e2c64..0000000--- a/scripts/computer-use-cli/examples/textedit-overlay-seq.json+++ /dev/null@@ -1,46 +0,0 @@-[- {- "tool": "list_apps"- },- {- "tool": "get_app_state",- "args": {- "app": "TextEdit"- }- },- {- "tool": "set_value",- "args": {- "app": "TextEdit",- "element_index": "2",- "value": "line 01\nline 02\nline 03\nline 04\nline 05\nline 06\nline 07\nline 08\nline 09\nline 10\nline 11\nline 12\nline 13\nline 14\nline 15\nline 16\nline 17\nline 18\nline 19\nline 20"- }- },- {- "tool": "get_app_state",- "args": {- "app": "TextEdit"- }- },- {- "tool": "click",- "args": {- "app": "TextEdit",- "element_index": "0"- }- },- {- "tool": "get_app_state",- "args": {- "app": "TextEdit"- }- },- {- "tool": "perform_secondary_action",- "args": {- "app": "TextEdit",- "element_index": "0",- "action": "Raise"- }- }-]diff --git a/scripts/computer-use-cli/examples/textedit-set-value-click-raise-seq.json b/scripts/computer-use-cli/examples/textedit-set-value-click-raise-seq.jsondeleted file mode 100644index 25e2c64..0000000--- a/scripts/computer-use-cli/examples/textedit-set-value-click-raise-seq.json+++ /dev/null@@ -1,46 +0,0 @@-[- {- "tool": "list_apps"- },- {- "tool": "get_app_state",- "args": {- "app": "TextEdit"- }- },- {- "tool": "set_value",- "args": {- "app": "TextEdit",- "element_index": "2",- "value": "line 01\nline 02\nline 03\nline 04\nline 05\nline 06\nline 07\nline 08\nline 09\nline 10\nline 11\nline 12\nline 13\nline 14\nline 15\nline 16\nline 17\nline 18\nline 19\nline 20"- }- },- {- "tool": "get_app_state",- "args": {- "app": "TextEdit"- }- },- {- "tool": "click",- "args": {- "app": "TextEdit",- "element_index": "0"- }- },- {- "tool": "get_app_state",- "args": {- "app": "TextEdit"- }- },- {- "tool": "perform_secondary_action",- "args": {- "app": "TextEdit",- "element_index": "0",- "action": "Raise"- }- }-]diff --git a/scripts/computer-use-cli/main.go b/scripts/computer-use-cli/main.goindex f630b55..70b8b49 100644--- a/scripts/computer-use-cli/main.go+++ b/scripts/computer-use-cli/main.go@@ -911,7 +911,7 @@ Examples:%s call list_apps%s call list_apps --transport app-server%s call list_apps --transport direct --server-bin /path/to/open-computer-use- %s call get_app_state --args '{"app":"Feishu"}'+ %s call get_app_state --args '{"app":"TextEdit"}'%s call-seq --transport app-server --calls-file /tmp/calls.jsonEnvironment:diff --git a/scripts/computer-use-cli/main_test.go b/scripts/computer-use-cli/main_test.goindex fd0f11f..8704b6f 100644--- a/scripts/computer-use-cli/main_test.go+++ b/scripts/computer-use-cli/main_test.go@@ -12,12 +12,12 @@ import (func TestReadToolArgsAcceptsJSONObject(t *testing.T) {t.Parallel()- args, err := readToolArgs(`{"app":"Feishu","clicks":2}`, "")+ args, err := readToolArgs(`{"app":"TextEdit","clicks":2}`, "")if err != nil {t.Fatalf("readToolArgs returned error: %v", err)}- if got, want := args["app"], "Feishu"; got != want {+ if got, want := args["app"], "TextEdit"; got != want {t.Fatalf("app = %#v, want %#v", got, want)}