👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
js repl: surface thrown errors, skip action read-back on macOS
ATGfb0e166b5ce9
diff --git a/docs/histories/2026-09/20260922-1330-js-repl-error-propagation-and-read-back.md b/docs/histories/2026-09/20260922-1330-js-repl-error-propagation-and-read-back.mdnew file mode 100644index 0000000..328a56c--- /dev/null+++ b/docs/histories/2026-09/20260922-1330-js-repl-error-propagation-and-read-back.md@@ -0,0 +1,34 @@+## [2026-09-22 13:30] | Task: Fix JS REPL error propagation and skip action read-back++### 🤖 Execution Context+* **Agent ID**: `Claude Code`+* **Base Model**: `Claude Opus 5`+* **Runtime**: `local macOS repository workspace`++### 📥 User Query+> Benchmarked the merged `js` adapter (#78) against an in-process runtime on Calculator: a ten-click batch took ~6.5 s, and any exception in `js` code hung until the 30 s timeout and reset the session. Fix both, minimally, within the adapter-over-native-MCP design.++### 🛠 Changes Overview+**Scope:** Node.js REPL adapter, macOS `ComputerUseService`, tests, and documentation.++**Key Actions:**+- **Error propagation**: A throw inside evaluated code (sync, or after an `await`) never reached the REPL eval callback; Node prints it and moves on, so the call waited for `timeout_ms` and then reset the session. The session now settles the active evaluation from the REPL's error path (`handleError` on Node 26, the domain `error` event on Node 22) and returns `isError` with the message; bindings survive.+- **Cross-realm error messages**: errors thrown inside the REPL context fail `instanceof Error`; `asError` now uses their `message` instead of `String(error)`, avoiding `Error: Error: ...`.+- **Read-back switch**: `OPEN_COMPUTER_USE_ACTION_READ_BACK=0` makes macOS action tools return a short status and skip the post-action settle and snapshot. All action-result sites route through one `actionResult(for:)` helper; the settles that exist only for the read-back go through `pauseBeforeReadBack(_:)`. The flag is read per call so the app agent's per-request environment forwarding applies. Default behaviour is unchanged.+- **Adapter**: launches the native runtime with the variable set, since `app.click()` and friends discard the native result and state is read explicitly with `getAXState()`.+- **Tests**: Node test for thrown errors settling the call and keeping bindings (fails with a 1 s timeout before the fix); XCTest for the flag parsing.++### 🧠 Design Intent (Why)+The adapter's value is batching actions and one final observation in a single round trip. Each native action was still paying a 150 ms settle plus a full accessibility snapshot that the adapter threw away, so a batch cost more than the discrete tools it replaced. Skipping the read-back only when a program asks for it keeps the nine-tool surface identical for direct MCP clients. The error fix keeps a model's mistake at one failed call instead of a timeout plus a lost session.++Measured on Calculator.app (one `js` call, 5 reps, median ms): clear×2 + click "7" + read 2295 → 402; clear×2 + click "7" ×10 + read 6469 → 405; thrown exception 30000 → 0 with bindings intact.++Linux and Windows runtimes still read back after each action; the same switch would go in their `actionResult` paths and runtime scripts.++### 📁 Files Modified+- `scripts/node-repl/open-computer-use-repl.mjs`+- `scripts/node-repl/open-computer-use-repl.test.mjs`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/references/js-repl.md`+- `docs/releases/feature-release-notes.md`diff --git a/docs/references/js-repl.md b/docs/references/js-repl.mdindex 9760522..3dff960 100644--- a/docs/references/js-repl.md+++ b/docs/references/js-repl.md@@ -66,6 +66,20 @@ Top-level bindings persist across calls. Prefer `var` for reusable names, or use`js_reset` when the session really needs a fresh lexical scope. Resetting theREPL does not close applications or erase their UI state.+A thrown exception ends the `js` call with `isError: true` and the message;+bindings survive. Only a timeout resets the session.++## Read-back++The native action tools normally settle for ~150 ms and return a fresh+snapshot after every action. The adapter discards that result and reads state+explicitly, so it launches the native runtime with+`OPEN_COMPUTER_USE_ACTION_READ_BACK=0`: actions return a short status with no+settle and no snapshot, and one `getAXState()` at the end of a batch reads the+outcome. On macOS a ten-click batch drops from several seconds to roughly one+state read. Linux and Windows runtimes ignore the variable for now and keep+reading back after each action.+## Entrypoints- The Codex plugin manifest uses the REPL launcher by default, so its advertiseddiff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.mdindex e0fa939..578867a 100644--- a/docs/releases/feature-release-notes.md+++ b/docs/releases/feature-release-notes.md@@ -5,6 +5,7 @@| 日期 | 功能域 | 用户价值 | 变更摘要 || --- | --- | --- | --- || 2026-09-22 | JS REPL CLI | 用户无需先配置 MCP host,也可以直接用 `ocu js` 执行一次性 Computer Use JavaScript,或用 `ocu repl` 在终端中复用持久 binding;Agent 还能先读取结构化 capability 状态。 | npm launcher 新增 positional/stdin/file 三种 `js` 输入、持久 terminal REPL、`capabilities [--json]` 和稳定 help availability;`ocu mcp` 仍保持原生 9-tool surface。 |+| 2026-09-22 | JS REPL 错误传播与 read-back | `js` 代码抛出的异常立即以 `isError` 返回且保留 bindings,不再等到 30 s 超时并丢弃 session;macOS 上一批 action 不再为每个动作等待 150 ms 并重新读取整棵 AX tree,十次点击从数秒降到约一次 state read。 | REPL adapter 从 Node 的 `handleError`(Node 26)或 domain(Node 22)路径 settle 当前 evaluation;adapter 以 `OPEN_COMPUTER_USE_ACTION_READ_BACK=0` 启动 native runtime,macOS action 在该模式下返回短状态、跳过 settle 与 snapshot;默认 9-tool 行为不变,Linux / Windows 暂时忽略该变量。 || 2026-09-21 | JS REPL Computer Use | Agent 可以用异步 JavaScript 绑定 app,并在一个 tool round trip 内完成 inspect、action、条件和最终验证,不必在多轮间搬运所有中间状态。 | Codex plugin 通过持久 Node.js Worker 和 app-bound `cua` API 暴露 `js` / `js_reset`;`open-computer-use mcp` 继续保留既有 9-tool compatibility surface。 || 2026-09-17 | DeepSeek Harness 集成 | DSH 用户可以用一条命令安装 Open Computer Use,并在写入配置前发现错误的可执行文件或不兼容的 MCP server。 | 新增 DSH MCP 兼容安装器、turn-boundary 光标清理 hook 和非破坏性 skill 安装;安装前校验 server identity 与非空工具目录但不锁死工具数量,显式安装在 MCP 启动失败时 fail loud,并明确该路径不注册 DSH 的 first-class computer-use provider。 || 2026-09-10 | macOS 拖拽修复 | 显式开启全局指针路径后,窗口移动、文本拖选与 Finder 拖放可收到启动真实拖拽所需的事件;默认路径的限制也会明确呈现。 | 发布 `0.3.5`:补齐拖拽位移、手势事件编号与 HID 投递时序,增加投递路径说明;同步升级 Go MCP SDK 到 `v1.4.1`。 |diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swiftindex ffee113..07e3376 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift@@ -185,6 +185,20 @@ func globalPointerFallbacksEnabled(environment: [String: String]) -> Bool {return ["1", "true", "yes", "on"].contains(rawValue)}+/// Whether an action's result reads the window back (settle, then a full snapshot).+/// A program driving the tools (the `js` REPL adapter) reads state explicitly and+/// sets `OPEN_COMPUTER_USE_ACTION_READ_BACK=0`, so its actions return a short status.+func actionReadBackEnabled(environment: [String: String]) -> Bool {+ guard let rawValue = environment["OPEN_COMPUTER_USE_ACTION_READ_BACK"]?+ .trimmingCharacters(in: .whitespacesAndNewlines)+ .lowercased()+ else {+ return true+ }++ return !["0", "false", "no", "off"].contains(rawValue)+}+/// How a `drag` is delivered. `drag` has no method argument; the path is decided/// by the same process-level gate that authorizes `click_method=global`.enum DragDeliveryPath: String, CaseIterable {@@ -448,9 +462,32 @@ func shouldPreferContainingWebRowAXClickCandidate(public final class ComputerUseService {private var snapshotsByApp: [String: AppSnapshot] = [:]+ // Read per call: the app agent applies the caller's OPEN_COMPUTER_USE_* variables+ // for the duration of each request.+ private var actionReadBack: Bool { actionReadBackEnabled(environment: ProcessInfo.processInfo.environment) }public init() {}+ /// An action's result reads the window back, so the app gets a beat to redraw+ /// first. Without read-back nothing is read, so there is no wait.+ private func pauseBeforeReadBack(_ interval: TimeInterval) {+ guard actionReadBack else {+ return+ }++ Thread.sleep(forTimeInterval: interval)+ }++ /// The result an action returns: the after-action snapshot, or a short status+ /// when read-back is off.+ private func actionResult(for query: String, recoveryPolicy: SnapshotRecoveryPolicy = .allowActivation) throws -> ToolCallResult {+ guard actionReadBack else {+ return .text("ok")+ }++ return snapshotResult(for: try refreshSnapshot(for: query, recoveryPolicy: recoveryPolicy), style: .actionResult)+ }+public func listApps() -> ToolCallResult {ToolCallResult.text(AppDiscovery.listCatalog()@@ -516,7 +553,7 @@ public final class ComputerUseService {Thread.sleep(forTimeInterval: 0.15)pulseVisualCursor(at: cursorTarget, clickCount: clickCount, mouseButton: button)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}if let elementIndex {@@ -645,13 +682,7 @@ public final class ComputerUseService {throw ComputerUseError.invalidArguments("click requires either element_index or x/y")}- return snapshotResult(- for: try refreshSnapshot(- for: query,- recoveryPolicy: clickActionSnapshotRecoveryPolicy(for: clickMethod)- ),- style: .actionResult- )+ return try actionResult(for: query, recoveryPolicy: clickActionSnapshotRecoveryPolicy(for: clickMethod))}public func performSecondaryAction(app query: String, elementIndex: String, action: String) throws -> ToolCallResult {@@ -663,7 +694,7 @@ public final class ComputerUseService {throw ComputerUseError.message(invalidSecondaryActionMessage(action: action, record: record))}- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}guard let rawAction = matchingAction(requested: action, record: record) else {@@ -679,8 +710,8 @@ public final class ComputerUseService {throw ComputerUseError.message("AXUIElementPerformAction failed with \(result.rawValue)")}- Thread.sleep(forTimeInterval: 0.15)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ pauseBeforeReadBack(0.15)+ return try actionResult(for: query)}public func scroll(app query: String, direction: String, elementIndex: String, pages: Double) throws -> ToolCallResult {@@ -701,7 +732,7 @@ public final class ComputerUseService {}try FixtureBridge.post(FixtureCommand(kind: "scroll", identifier: identifier, direction: normalized, pages: pages))Thread.sleep(forTimeInterval: 0.15)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}if let repeatCount = integralScrollPageCount(pages),@@ -723,7 +754,7 @@ public final class ComputerUseService {throw ComputerUseError.stateUnavailable("element \(elementIndex) has no scrollable frame")}- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}public func drag(app query: String, fromX: Double, fromY: Double, toX: Double, toY: Double) throws -> ToolCallResult {@@ -731,7 +762,7 @@ public final class ComputerUseService {if snapshot.mode == .fixture {try FixtureBridge.post(FixtureCommand(kind: "drag", identifier: "fixture-drag-pad", x: fromX, y: fromY, toX: toX, toY: toY))Thread.sleep(forTimeInterval: 0.15)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}let start = try screenshotToGlobalPoint(snapshot: snapshot, x: fromX, y: fromY)@@ -743,7 +774,7 @@ public final class ComputerUseService {snapshot: snapshot)return appendingDragDeliveryNote(- to: snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult),+ to: try actionResult(for: query),path: path)}@@ -753,12 +784,12 @@ public final class ComputerUseService {if snapshot.mode == .fixture {try FixtureBridge.post(FixtureCommand(kind: "type_text", identifier: "fixture-input", value: text))Thread.sleep(forTimeInterval: 0.15)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}if try typeTextBySettingFocusedValueIfAvailable(text, in: snapshot) {- Thread.sleep(forTimeInterval: 0.1)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ pauseBeforeReadBack(0.1)+ return try actionResult(for: query)}guard try canTypeTextUsingKeyboardFallback(in: snapshot) else {@@ -766,7 +797,7 @@ public final class ComputerUseService {}try InputSimulation.typeText(text, pid: snapshot.app.pid)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}public func pressKey(app query: String, key: String) throws -> ToolCallResult {@@ -774,11 +805,11 @@ public final class ComputerUseService {if snapshot.mode == .fixture {try FixtureBridge.post(FixtureCommand(kind: "press_key", identifier: "fixture-key-capture", value: key))Thread.sleep(forTimeInterval: 0.15)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}try InputSimulation.pressKey(key, pid: snapshot.app.pid)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}public func setValue(app query: String, elementIndex: String, value: String) throws -> ToolCallResult {@@ -795,7 +826,7 @@ public final class ComputerUseService {try FixtureBridge.post(FixtureCommand(kind: "set_value", identifier: identifier, value: value))Thread.sleep(forTimeInterval: 0.15)settleVisualCursor(at: cursorTarget)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}guard let element = record.element else {@@ -815,14 +846,14 @@ public final class ComputerUseService {throw ComputerUseError.message("AXUIElementSetAttributeValue failed with \(result.rawValue)")}- Thread.sleep(forTimeInterval: 0.1)+ pauseBeforeReadBack(0.1)} catch {settleVisualCursor(at: cursorTarget)throw error}settleVisualCursor(at: cursorTarget)- return snapshotResult(for: try refreshSnapshot(for: query), style: .actionResult)+ return try actionResult(for: query)}private func currentSnapshot(for query: String) throws -> AppSnapshot {@@ -959,7 +990,7 @@ public final class ComputerUseService {switch result {case .success:- Thread.sleep(forTimeInterval: 0.15)+ pauseBeforeReadBack(0.15)return truecase .failure, .attributeUnsupported, .actionUnsupported, .cannotComplete, .noValue, .invalidUIElement, .illegalArgument:return false@@ -1004,21 +1035,21 @@ public final class ComputerUseService {if preferContainingWebRowAXClick,try performContainingWebRowClick(for: record, snapshot: snapshot, button: button, clickCount: clickCount){- Thread.sleep(forTimeInterval: 0.15)+ pauseBeforeReadBack(0.15)return true}if !preferContainingWebRowAXClick {if try performPreferredClick(on: record, button: button, clickCount: clickCount) {debugClickDecision("handled by preferred target \(clickDebugDescription(record))")- Thread.sleep(forTimeInterval: 0.15)+ pauseBeforeReadBack(0.15)return true}for candidate in descendantClickCandidates(for: record, snapshot: snapshot) {if try performPreferredClick(on: candidate, button: button, clickCount: clickCount) {debugClickDecision("handled by descendant \(clickDebugDescription(candidate))")- Thread.sleep(forTimeInterval: 0.15)+ pauseBeforeReadBack(0.15)return true}}@@ -1033,7 +1064,7 @@ public final class ComputerUseService {try performPreferredClick(on: hitRecord, button: button, clickCount: clickCount){debugClickDecision("handled by hit record \(clickDebugDescription(hitRecord))")- Thread.sleep(forTimeInterval: 0.15)+ pauseBeforeReadBack(0.15)return true}@@ -1048,7 +1079,7 @@ public final class ComputerUseService {) {if try performPreferredClick(on: candidate, button: button, clickCount: clickCount) {debugClickDecision("handled by hit descendant \(clickDebugDescription(candidate))")- Thread.sleep(forTimeInterval: 0.15)+ pauseBeforeReadBack(0.15)return true}}@@ -1069,7 +1100,7 @@ public final class ComputerUseService {if try activateClickTarget(element: element, availableActions: record.rawActions) {debugClickDecision("handled by activation fallback \(clickDebugDescription(record))")- Thread.sleep(forTimeInterval: 0.15)+ pauseBeforeReadBack(0.15)return true}diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 250f5f2..c76bc8f 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -1475,6 +1475,13 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertFalse(globalPointerFallbacksEnabled(environment: ["OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS": "false"]))}+ func testActionReadBackFlagDefaultsToEnabled() {+ XCTAssertTrue(actionReadBackEnabled(environment: [:]))+ XCTAssertTrue(actionReadBackEnabled(environment: ["OPEN_COMPUTER_USE_ACTION_READ_BACK": "1"]))+ XCTAssertFalse(actionReadBackEnabled(environment: ["OPEN_COMPUTER_USE_ACTION_READ_BACK": "0"]))+ XCTAssertFalse(actionReadBackEnabled(environment: ["OPEN_COMPUTER_USE_ACTION_READ_BACK": " off "]))+ }+func testDragStepCountScalesWithDistanceAndClampsToBounds() {XCTAssertEqual(InputSimulation.dragStepCount(from: .zero, to: .zero), 10)XCTAssertEqual(InputSimulation.dragStepCount(from: CGPoint(x: 0, y: 0), to: CGPoint(x: 8, y: 0)), 10)diff --git a/scripts/node-repl/open-computer-use-repl.mjs b/scripts/node-repl/open-computer-use-repl.mjsindex 8317dc7..dacd360 100755--- a/scripts/node-repl/open-computer-use-repl.mjs+++ b/scripts/node-repl/open-computer-use-repl.mjs@@ -47,7 +47,9 @@ The runtime exposes an asynchronous app-bound API:After actions, call \`getAXState()\` in the same js invocation when the next decision depends on the updated UI. Re-derive element indexes from fresh state after navigation or layout changes. Prefer element indexes over coordinates. Open Computer Use keeps its existing local safety gates, including password-manager denial and explicit authorization for global pointer fallbacks.`;function asError(error) {- return error instanceof Error ? error : new Error(String(error));+ if (error instanceof Error) return error;+ // Errors thrown inside the REPL context come from another realm and fail instanceof.+ return new Error(typeof error?.message === "string" ? error.message : String(error));}function isPlainObject(value) {@@ -333,8 +335,14 @@ export class PersistentJavaScriptSession {const input = new PassThrough();const output = new PassThrough();output.resume();- this.repl = repl.start({ prompt: "", input, output, terminal: false, useGlobal: false, ignoreUndefined: true, breakEvalOnSigint: true });+ // A throw inside evaluated code (sync, or after an await) never reaches the eval+ // callback: the REPL prints it and moves on. Settle the active evaluation from the+ // REPL's error path instead. Node 26 exposes `handleError`; Node 22 routes through+ // a domain.+ const settle = error => { this.rejectActive?.(error); return "ignore"; };+ this.repl = repl.start({ prompt: "", input, output, terminal: false, useGlobal: false, ignoreUndefined: true, breakEvalOnSigint: true, handleError: settle });this.repl.on("error", () => {});+ this.repl._domain?.on("error", settle);const nodeRepl = {cwd: process.cwd(),homeDir: process.env.HOME ?? "",@@ -362,6 +370,7 @@ export class PersistentJavaScriptSession {const enforceTimeout = Number.isFinite(timeoutMs) && timeoutMs > 0;try {const evaluation = new Promise((resolve, reject) => {+ this.rejectActive = reject;this.repl.eval(code, this.repl.context, "open-computer-use-repl", (error, result) => error ? reject(error) : resolve(result));});const value = enforceTimeout@@ -377,6 +386,7 @@ export class PersistentJavaScriptSession {} finally {clearTimeout(timer);this.active = null;+ this.rejectActive = null;}}}@@ -486,7 +496,9 @@ function resolveNativeCommand(argv) {}export async function runServer({ command, args }) {- const native = new JsonLinePeer({ command, args, env: process.env });+ // Actions return a short status instead of a settle + full snapshot the adapter would+ // discard; state is read explicitly by getAXState(). Runtimes without the flag ignore it.+ const native = new JsonLinePeer({ command, args, env: { ...process.env, OPEN_COMPUTER_USE_ACTION_READ_BACK: "0" } });await native.initialize();const session = new WorkerJavaScriptSession({ native });let buffer = "";diff --git a/scripts/node-repl/open-computer-use-repl.test.mjs b/scripts/node-repl/open-computer-use-repl.test.mjsindex 552c909..e0ea570 100644--- a/scripts/node-repl/open-computer-use-repl.test.mjs+++ b/scripts/node-repl/open-computer-use-repl.test.mjs@@ -112,6 +112,19 @@ test("tool errors are catchable in JavaScript", async () => {assert.equal(result.content.at(-1).text, "caught: typing failed");});+test("thrown errors settle the call and keep bindings", async () => {+ const session = new PersistentJavaScriptSession({ native: mockNative() });+ await session.run("var kept = 1;");+ let result = await session.run("nodeRepl.write('before'); null[1];", 1000);+ assert.equal(result.isError, true);+ assert.match(result.content.at(-1).text, /^Error: Cannot read properties of null/);+ result = await session.run("await 1; throw new Error('after await');", 1000);+ assert.equal(result.isError, true);+ assert.equal(result.content.at(-1).text, "Error: after await");+ result = await session.run("nodeRepl.write(kept);");+ assert.equal(result.content.at(-1).text, "1");+});+test("screenshots emit binary image content", async () => {const session = new PersistentJavaScriptSession({ native: mockNative() });const result = await session.run(`