Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

feat(plugin): add persistent JavaScript REPL

Leo8684e7c086c9
diff --git a/Makefile b/Makefile
index 20dd4ef..8dc979f 100644
--- a/Makefile
+++ b/Makefile
@@ -17,6 +17,7 @@ app:
test:
swift test
+ node --test scripts/node-repl/open-computer-use-repl.test.mjs
smoke:
./scripts/run-tool-smoke-tests.sh
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index ac464ef..5807086 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -1,6 +1,6 @@
# 架构总览
-这个仓库当前已经从模板收敛成一个本地 `computer-use` 项目。主线仍是 Swift 实现的 macOS automation MCP server,同时新增了实验性的 Windows 和 Linux runtime,用独立 Go 二进制暴露同一组 9 个 Computer Use tools。
+这个仓库当前已经从模板收敛成一个本地 `computer-use` 项目。主线仍是 Swift 实现的 macOS automation MCP server,同时新增了实验性的 Windows 和 Linux runtime,用独立 Go 二进制暴露同一组 9 个 native Computer Use tools;Codex plugin 再在 native MCP 之上增加持久 Node.js REPL,向模型暴露 `js` / `js_reset`。
## 当前目录结构
@@ -27,7 +27,7 @@
- `experiments/StandaloneCursor`
新的独立 Swift cursor viewer,直接复用 `scripts/cursor-motion-re/official_cursor_motion.py` 里收敛出来的候选路径、score 与 raw spring timeline,用来观察更贴近 binary lift 的表现。
- `scripts/`
- 仓库级自动化命令,包括 smoke test、`.app` 打包入口、Windows `.exe` / Linux binary 构建入口、npm 分发脚本,以及 `scripts/computer-use-cli/` 这个用于探测官方 bundled `computer-use` 的 Go helper。
+ 仓库级自动化命令,包括 smoke test、`.app` 打包入口、Windows `.exe` / Linux binary 构建入口、npm 分发脚本、`scripts/node-repl/` 的 JS REPL adapter,以及 `scripts/computer-use-cli/` 这个用于探测官方 bundled `computer-use` 的 Go helper。
- `skills/`
面向 agent runtime 的可安装 skill。当前 `skills/open-computer-use/SKILL.md` 只作为轻量入口和目录,安装、MCP/CLI 使用、排障等细节拆到相邻 `references/` 文件里按需加载;`scripts/package-skill.sh` 负责校验并打包 `.zip` / `.skill` 制品。
- `docs/`
@@ -62,9 +62,13 @@
### 3. Tool Service 层
- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。
+- Codex plugin 的 `.mcp.json` 默认不再把这 9 个 tools 直接暴露给模型,而是启动 `scripts/node-repl/open-computer-use-repl.mjs`。adapter 以 child MCP client 连接同一个 native runtime,并只对外列出 `js` / `js_reset`;JavaScript 通过异步 app-bound API(`cua.getApp(...)`、`app.click(...)`、`app.getAXState()` 等)调用 native tools。
+- REPL 使用 Node.js 自带的 evaluator,因此保留 top-level `await` 和跨调用 lexical bindings;真正执行代码的 kernel 位于 Worker 中,超时会终止 Worker 并重建干净 session,避免 `while (true)` 挂死 MCP transport。
+- 这是增量迁移:`open-computer-use mcp` 仍是原生 9-tool compatibility surface,已有 CLI、其他 MCP host 和 smoke 不需要切换。
- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。
- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但通用节点中的 `AXPress` / `AXConfirm` / `AXOpen` 子节点会形成文本摘要边界,避免多个可点击选项被合并成一个 container。这类动作节点如果 frame 有效、尺寸紧凑且不包含带 URL 的 `AXLink` 后代,会保留为带窗口相对 `Frame` 的 `button`,并用短文本后代作为按钮摘要,让 icon-only 和文字 Web 控件都能获得可区分的 `element_index`;包含带 URL 的 `AXLink` 后代时保留通用 wrapper 和链接子节点,避免导航链接被摘要吞掉。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。
- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。
+- JS REPL 的 API 和当前官方 code-first 形态对齐到异步 app binding,而不是把离散 tools 简单包成同步函数。实现不复制或运行 proprietary `@oai/*` package;协议和行为记录在 `docs/references/js-repl.md`。
- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。
- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。
- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。
@@ -139,6 +143,7 @@
## 主要验证路径
- 单元测试:`swift test`
+- JS REPL contract:`node --test scripts/node-repl/open-computer-use-repl.test.mjs`
- standalone cursor 构建:`swift build --product StandaloneCursor`
- cursor lab 构建:`swift build --product CursorMotion`
- 端到端 smoke:`./scripts/run-tool-smoke-tests.sh`(标准 9-tool smoke + visual cursor idle smoke;脚本默认以 headless 模式启动内部 fixture,避免在用户桌面弹出测试窗口)
diff --git a/docs/RELIABILITY.md b/docs/RELIABILITY.md
index 068a2cc..11c34df 100644
--- a/docs/RELIABILITY.md
+++ b/docs/RELIABILITY.md
@@ -4,6 +4,7 @@
- 构建:`swift build`
- 单元测试:`swift test`
+- JS REPL:`node --test scripts/node-repl/open-computer-use-repl.test.mjs`
- 端到端 smoke:`./scripts/run-tool-smoke-tests.sh`
- macOS SkyLight 实机回归:`OPEN_COMPUTER_USE_RUN_SKY_CLICK_LIVE_TEST=1 swift test --filter SkyClickLiveTests`
- Linux runtime:`(cd apps/OpenComputerUseLinux && go test ./...)`、`./scripts/build-open-computer-use-linux.sh --arch arm64`
@@ -14,6 +15,7 @@
## 已知关键依赖
- macOS 上必须给 `Open Computer Use.app` 授权 `Accessibility` 与 `Screen Recording`;终端本身不应该再是必需授权对象。
+- Codex plugin 的 code-first surface 需要可执行的 `node`。REPL adapter 会为 native MCP 保留独立 child lifecycle,JavaScript kernel 则放在 Worker 中;脚本超时会丢弃该 Worker 的全部 bindings,并由下一次调用使用新的 kernel。
- macOS `click_method=sky_click` 额外依赖 SkyLight / ApplicationServices 私有符号 `SLEventPostToPid`、`SLEventSetIntegerValueField`、`CGEventSetWindowLocation`、`SLPSPostEventRecordTo` 和 `GetProcessForPID`。运行时会动态探测并 fail closed,但 macOS 更新、签名方式或目标 app 输入策略变化仍可能让后台投递失效。受控实机回归除 DOM、前台 PID、鼠标和 z-order 外,还必须验证前台 AppKit active、key window、first responder 以及 resign/key-loss 计数。
- smoke suite 依赖本地 GUI session,不能把它当成无头环境命令。
- 普通 app 的 `get_app_state` 结果依赖 AX tree 和窗口截图,复杂 app 上输出会有差异;Electron/WebView app 的 AX tree 通常很深,当前会压缩空 wrapper 并放宽遍历深度,以优先保留可操作文本、按钮和输入框。
diff --git a/docs/SECURITY.md b/docs/SECURITY.md
index b03b1ca..361fc92 100644
--- a/docs/SECURITY.md
+++ b/docs/SECURITY.md
@@ -3,6 +3,8 @@
## 当前实现边界
- 对 MCP host 暴露的接口仍是本地 `stdio`;macOS CLI 与 `.app` app agent 之间会使用用户临时目录下的 Unix domain socket,socket 创建后会收紧为当前用户读写,且不对外监听 TCP/HTTP 端口。未设置 `OPEN_COMPUTER_USE_AGENT_SOCKET_NAMESPACE` 时继续使用历史 Socket;设置后仅以 namespace 摘要派生私有文件名,不把宿主目录或原始 namespace 写入 Socket 路径。
+- Codex plugin 默认把一个本地 Node.js REPL 放在 native MCP 前面。`js` 是任意本地 JavaScript 执行能力,不是受限表达式语言:代码可按启动进程权限读取文件、环境、模块和网络。只应在 host 已经具备并允许 model-code execution 的信任边界内启用;不接受这条边界的 host 应继续直连 `open-computer-use mcp` 的离散 native tools。
+- REPL 里的 Computer Use 调用仍经过 native MCP,因此密码管理器 denylist、`OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS` 等 native safety gate 不会被 JavaScript adapter 绕过。JS kernel 在 Worker 中运行,超时会终止整个 Worker 并清空 bindings。
- 所有动作都必须显式带 `app` 参数;当前不会在后台自动扫描并控制任意 app。
- macOS 真实 app 路径依赖 `Open Computer Use.app` 已获得 `Accessibility` 与 `Screen Recording` 权限;终端里的 CLI / Node launcher 会把 `mcp`、`doctor`、`call`、`snapshot` 和 `list-apps` 转发给由 LaunchServices 启动的本地 app agent,避免把权限要求落到 iTerm / Terminal 身上。
- 实验性 Linux runtime 依赖已登录桌面用户的 AT-SPI2 / D-Bus session;coordinate mouse、drag、keyboard synthesis 只是 best-effort fallback,不应被视为跨 Wayland compositor 的通用后台输入授权。
diff --git a/docs/exec-plans/completed/20260921-js-repl-computer-use.md b/docs/exec-plans/completed/20260921-js-repl-computer-use.md
new file mode 100644
index 0000000..a0103ff
--- /dev/null
+++ b/docs/exec-plans/completed/20260921-js-repl-computer-use.md
@@ -0,0 +1,88 @@
+# JS REPL Computer Use 支持
+
+## 目标
+
+把 Open Computer Use 从“只暴露离散的 9 个 MCP tools”扩展为可由持久 Node.js REPL 编排的 Computer Use runtime:模型通过 `js` / `js_reset` 编写异步 JavaScript,使用 app-bound API,在一次 tool round trip 内完成多步动作、条件、循环和最终状态读取。
+
+## 范围
+
+- 包含:
+ - 调研官方 bundled `computer-use`、host `node_repl` 与 `@oai/sky` 的启动与调用链。
+ - 审阅社区 PR #65、#72、#73、#74、#75,重点复用 #75 的意图并修正协议偏差。
+ - 新增跨平台 Node REPL adapter,把现有 native MCP server 的离散 tools 转成异步 `cua` app-bound API。
+ - 让 Codex plugin 默认通过 Node REPL adapter 暴露 `js` / `js_reset`,保留原生 `mcp` 入口兼容其他 host。
+ - 覆盖持久 binding、reset、错误传播、截图输出、同一次调用内多步工具编排和 child lifecycle。
+ - 同步架构、安全、可靠性、usage、plugin metadata、release note 与 history。
+- 不包含:
+ - 直接复制或依赖官方 proprietary `@oai/*` packages。
+ - 在 Swift 中嵌入 JavaScriptCore 作为主 REPL;该方案缺少官方 Node REPL 的 async/top-level await/module/output 语义。
+ - 本轮合并 #65、#72、#73、#74;它们独立评审和交付。
+ - 本轮移除现有 9-tool MCP surface;`open-computer-use mcp` 保持兼容。
+
+## 背景
+
+- 相关文档:
+ - `docs/ARCHITECTURE.md`
+ - `docs/SECURITY.md`
+ - `docs/RELIABILITY.md`
+ - `skills/open-computer-use/references/usage.md`
+- 相关代码路径:
+ - `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift`
+ - `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift`
+ - `plugins/open-computer-use/`
+ - `scripts/npm/build-packages.mjs`
+- 已知约束:
+ - 官方 plugin 1.0.1001068 仍是 `@oai/sky` skill + native 9-tool MCP client;本机检查到的 host bundle 提供 Node 22 `node_repl` 与 `@oai/sky` 0.6.11。
+ - 已检查的官方 bundle 使用 host 提供的通用 `node_repl`,由 plugin 指引模型导入异步 `@oai/sky` API;REPL 不是 native MCP 自身新增的两个 tools。
+ - Node REPL 必须有持久顶层 binding、top-level await、显式输出、超时和 reset。
+ - 不得在 commit 或 PR 中加入禁止的 Co-authored-by trailer。
+
+## 风险
+
+- 风险:子进程 MCP 与 Node REPL 双层生命周期处理不当会泄漏 native runtime 或挂死。
+ - 缓解方式:adapter 统一接管 SIGINT/SIGTERM/SIGHUP、stdin EOF、pending JSON-RPC、默认 timeout,并在测试中验证关闭路径。
+- 风险:任意 JavaScript 比离散 tool surface 权限更宽。
+ - 缓解方式:明确这是本地 Node execution surface;plugin 默认入口只用于 Codex 场景,原生 MCP 保持可选;文档说明文件/网络能力与信任边界。
+- 风险:官方 API 仍在快速变化。
+ - 缓解方式:实现稳定的最小 app-bound API,独立 adapter 与 native dispatcher,通过 contract tests 固定协议,避免耦合 proprietary package。
+- 风险:截图二进制在 nested MCP/Node bridge 中膨胀。
+ - 缓解方式:只在 `getScreenshot` / `getAXStateAndScreenshot` 时解码并 emit,普通动作不透传 action-result screenshot。
+
+## 里程碑
+
+1. 调研官方机制并审阅社区 PR。
+2. 落跨平台 Node REPL adapter、plugin route 与 contract tests。
+3. 文档、history、release note 和完整 CI 验证。
+
+## 验证方式
+
+- 命令:
+ - `node --test scripts/node-repl/*.test.mjs`
+ - `swift test`
+ - `(cd apps/OpenComputerUseLinux && go test ./...)`
+ - `(cd apps/OpenComputerUseWindows && go test ./...)`
+ - `make check-docs`
+ - `./scripts/run-tool-smoke-tests.sh`
+ - `./scripts/ci.sh`
+- 手工检查:
+ - MCP `tools/list` 只有 `js` / `js_reset`。
+ - 连续 `js` 调用保留顶层 binding,`js_reset` 清空。
+ - `await cua.getApp(...)` 返回 app binding,动作与 snapshot 在一次 `js` 内按序执行。
+- 观测检查:
+ - native child 退出或超时给出结构化错误;adapter 退出时 child 不残留。
+
+## 进度记录
+
+- [x] 读取仓库规则与架构文档。
+- [x] 逆向当前与上一代官方 plugin/runtime。
+- [x] 审阅 #65、#72、#73、#74、#75 的范围、冲突和设计。
+- [x] 完成 Node REPL adapter 与 plugin 集成。
+- [x] 完成 contract tests 和跨平台验证。
+- [x] 完成文档、history、release note 与 plan 归档。
+
+## 决策记录
+
+- 2026-09-21:不直接合并 #75 的 JavaScriptCore 同步 runtime。保留其“代码编排减少 round trip”的产品判断,但采用 Node.js、top-level await、persistent lexical bindings、`nodeRepl.write/emitImage` 和 app-bound async `cua` API;这是适配任意 MCP host 的开源实现,不声称官方 plugin 本身也暴露这两个 MCP tools。
+- 2026-09-21:本轮采取 additive migration。Codex plugin 默认走 `js` / `js_reset` adapter;原生 `open-computer-use mcp` 继续暴露离散 tools,避免一次性破坏所有 MCP host。
+- 2026-09-21:#73 的 targeted AX query 与 #74 的 App Intent 不作为 model-facing standalone tools 并入本轮;未来若采用,应优先成为 app-bound `cua` API 的能力,而非继续扩大顶层 MCP tool surface。
+- 2026-09-22:完整验证通过:Node contract 12/12、Swift 167 tests(1 个 opt-in live test skipped)、Linux/Windows Go tests、Linux Python tests、9-tool 与 cursor smoke、npm dry-run/staged launcher、隔离的 Codex plugin cache install 和 macOS 真实 native bridge。Linux devbox 安装缺失的 `gir1.2-atspi-2.0` 后,另外验证了 Node 22 下的 contract、两工具 surface、结构化空 app 列表、持久 binding、CPU timeout/reset 和 native Linux bridge;该 SSH 环境没有可见的登录桌面窗口,所以 app 数量为 0。仓库级 `scripts/ci.sh` 仅在已有 repository-hygiene gate 失败:当前 checkout 的 `main` 本身缺少 `.editorconfig`、`.markdownlint.json` 及多份 `.github` 模板/workflow;本轮相关的其余 CI 子步骤已单独通过。
diff --git a/docs/histories/2026-09/20260921-2100-add-js-repl-computer-use.md b/docs/histories/2026-09/20260921-2100-add-js-repl-computer-use.md
new file mode 100644
index 0000000..87a0ff4
--- /dev/null
+++ b/docs/histories/2026-09/20260921-2100-add-js-repl-computer-use.md
@@ -0,0 +1,48 @@
+## [2026-09-21 21:00] | Task: Add JS REPL Computer Use
+
+### 🤖 Execution Context
+* **Agent ID**: `TRAE CLI`
+* **Base Model**: `GPT-5.4`
+* **Runtime**: `local macOS repository workspace`
+
+### 📥 User Query
+> Review the community PRs and the bundled official Computer Use plugin, determine how its JavaScript REPL works, and add JS REPL support to Open Computer Use.
+
+### 🛠 Changes Overview
+**Scope:** Codex plugin transport, Node.js adapter, native MCP bridge, tests, and repository documentation.
+
+**Key Actions:**
+- **Official runtime investigation**: Verified that the inspected official bundle uses a host-provided `node_repl` and instructs the model to import its asynchronous `@oai/sky` package.
+- **Persistent REPL**: Added a Node.js REPL with top-level await, persistent bindings, explicit text/image output, reset, and a worker timeout boundary.
+- **Native bridge**: Adapted the existing native 9-tool MCP server into an asynchronous `cua.getApp(...)` API without duplicating the platform automation implementations.
+- **Compatibility**: Switched the Codex plugin to `js` / `js_reset` while preserving `open-computer-use mcp` as the existing native compatibility surface.
+- **Packaging and docs**: Included the REPL files in local/plugin npm installation and documented architecture, security, reliability, and usage.
+- **Lifecycle hardening**: Isolated model code in a Worker, capped execution at five minutes, reset the kernel after timeouts, made shutdown idempotent, and verified copied/symlinked adapter entrypoints.
+- **Installer correction**: Flattened the plugin source into the versioned Codex cache root (matching Codex's cache layout) before adding the native payload and REPL scripts.
+- **Protocol normalization**: Converted the native human-readable macOS/Linux/Windows app catalog into a stable structured array for `cua.listApps()`.
+- **Concurrency and failure isolation**: Serialized direct Worker callers, converted invalid source input into structured errors, and let the outer JS timeout remain the single deadline for native calls so timeout recovery cannot leave a second timer behind.
+
+### 🧠 Design Intent (Why)
+The main value of a code tool is not another spelling of the same actions. It is the ability to keep deterministic sequencing, intermediate state, branching, retry, and final observation inside one tool round trip. The implementation therefore follows the official asynchronous Node orchestration direction while providing a small app-bound API, rather than merging the synchronous JavaScriptCore prototype from PR #75.
+
+### 📁 Files Modified
+- `scripts/node-repl/open-computer-use-repl.mjs`
+- `scripts/node-repl/open-computer-use-kernel.mjs`
+- `scripts/node-repl/open-computer-use-repl.test.mjs`
+- `plugins/open-computer-use/.mcp.json`
+- `plugins/open-computer-use/scripts/launch-open-computer-use-repl.sh`
+- `scripts/install-codex-plugin.sh`
+- `scripts/npm/build-packages.mjs`
+- `docs/references/js-repl.md`
+- `docs/ARCHITECTURE.md`
+- `docs/SECURITY.md`
+- `docs/RELIABILITY.md`
+
+### ✅ Validation
+- `node --test scripts/node-repl/open-computer-use-repl.test.mjs` — 12/12 passed locally and on Linux devbox with Node 22.
+- `swift test` — 167 tests, 0 failures, 1 opt-in live test skipped.
+- Linux and Windows `go test ./...`; Linux Python runtime tests — passed.
+- `./scripts/run-tool-smoke-tests.sh` — native 9-tool and cursor-idle smoke passed.
+- npm staging, `npm pack --dry-run --json`, isolated Codex plugin install, and staged/installed `tools/list` probes — passed and exposed only `js` / `js_reset`.
+- Real macOS native bridge probe verified structured `cua.listApps()`, binding persistence, CPU timeout recovery, and `js_reset`. Linux devbox verified the same REPL lifecycle and native bridge with Node 22; after installing its missing AT-SPI introspection package, the headless SSH session correctly returned a structured empty app array.
+- `make check-docs`, shell/Node syntax checks, action pinning, and `git diff --check` — passed. Full `scripts/ci.sh` remains blocked only by pre-existing repository-hygiene files absent from this checkout, unrelated to this change.
diff --git a/docs/references/js-repl.md b/docs/references/js-repl.md
new file mode 100644
index 0000000..b7ac1f5
--- /dev/null
+++ b/docs/references/js-repl.md
@@ -0,0 +1,91 @@
+# JavaScript REPL Computer Use
+
+Open Computer Use ships a Node.js adapter for hosts that want a code-first
+Computer Use surface. The adapter exposes two model-facing MCP tools, `js` and
+`js_reset`, and keeps the existing native 9-tool MCP server behind it.
+
+## Why this shape
+
+A UI flow usually needs several ordered operations: inspect, act, inspect again,
+branch, retry, and verify. Discrete model-facing tool calls pay one model round
+trip per operation and repeatedly serialize intermediate state. JavaScript makes
+the sequencing local: deterministic actions and the final state read can happen
+in one `js` call.
+
+The design follows the current official code-first workflow without copying its
+proprietary packages. In the inspected official bundle, the host provides a
+general `node_repl` and the Computer Use plugin tells the model to import
+`@oai/sky`. Open Computer Use packages the equivalent orchestration boundary as
+two plugin-local MCP tools because an arbitrary MCP host does not necessarily
+provide that general REPL:
+
+1. a persistent Node.js REPL owns top-level JavaScript bindings and top-level
+ `await`;
+2. an asynchronous app-bound `cua` API converts JavaScript calls into native MCP
+ `tools/call` requests;
+3. the existing Swift, Go, and Python-backed runtimes continue to own app
+ discovery, screenshots, accessibility, and input;
+4. a worker boundary lets the host terminate CPU-bound JavaScript on timeout and
+ start a fresh kernel.
+
+## API
+
+```js
+var app = await cua.getApp("TextEdit");
+await app.click(12);
+await app.typeText("Hello");
+await app.getAXState();
+```
+
+`cua.getApp(...)` accepts the same app name or bundle identifier as the native
+runtime and emits the initial accessibility state. The returned binding exposes:
+
+```text
+getAXState(options?)
+getScreenshot(options?)
+getAXStateAndScreenshot(options?)
+click(elementIndexOrPoint, options?)
+drag([fromX, fromY], [toX, toY])
+pressKey(key)
+scroll(elementIndex, direction, pages?)
+setValue(elementIndex, value)
+typeText(text)
+performSecondaryAction(elementIndex, action)
+```
+
+Discovery is available through `await cua.getState()` and
+`await cua.listApps()`. `listApps()` normalizes the native human-readable app
+catalog into an array of `{ id, displayName, lastUsedDate?, useCount?,
+isRunning? }` objects, while `getState()` returns `{ apps }`. Observation
+methods emit their result by default; pass
+`{ emit: false }` when code needs the value but the model does not need to see
+it. Use `nodeRepl.write(value)` for extra text and
+`await nodeRepl.emitImage(image)` for extra images.
+
+Top-level bindings persist across calls. Prefer `var` for reusable names, or use
+`js_reset` when the session really needs a fresh lexical scope. Resetting the
+REPL does not close applications or erase their UI state.
+
+## Entrypoints
+
+- The Codex plugin manifest uses the REPL launcher by default, so its advertised
+ model-facing surface is `js` plus `js_reset`.
+- `open-computer-use mcp` remains the native 9-tool compatibility surface for
+ other MCP clients.
+- The adapter can be run directly for development:
+
+```sh
+node scripts/node-repl/open-computer-use-repl.mjs -- \
+ ./dist/Open\ Computer\ Use.app/Contents/MacOS/OpenComputerUse mcp
+```
+
+## Security boundary
+
+`js` is local Node.js code execution, not a constrained expression language. It
+can access the local filesystem, environment, modules, and network with the
+permissions of the launching process. Only enable the REPL surface for a host
+whose model-code execution policy you trust. The native Computer Use calls still
+apply their own password-manager denylist and explicit global-pointer gate.
+
+The native MCP compatibility surface remains available when arbitrary
+JavaScript is not an acceptable host boundary.
diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.md
index a337710..42cee7f 100644
--- a/docs/releases/feature-release-notes.md
+++ b/docs/releases/feature-release-notes.md
@@ -4,6 +4,7 @@
| 日期 | 功能域 | 用户价值 | 变更摘要 |
| --- | --- | --- | --- |
+| 2026-09-21 | JS REPL Computer Use | Agent 可以用异步 JavaScript 绑定 app,并在一个 tool round trip 内完成 inspect、action、条件和最终验证,不必在多轮间搬运所有中间状态。 | Codex plugin 通过持久 Node.js Worker 和 app-bound `cua` API 暴露 `js` / `js_reset`;`open-computer-use mcp` 继续保留既有 9-tool compatibility surface。 |
| 2026-09-17 | DeepSeek Harness 集成 | DSH 用户可以用一条命令安装 Open Computer Use,并在写入配置前发现错误的可执行文件或不兼容的 MCP server。 | 新增 DSH MCP 兼容安装器、turn-boundary 光标清理 hook 和非破坏性 skill 安装;安装前校验 server identity 与非空工具目录但不锁死工具数量,显式安装在 MCP 启动失败时 fail loud,并明确该路径不注册 DSH 的 first-class computer-use provider。 |
| 2026-09-10 | macOS 拖拽修复 | 显式开启全局指针路径后,窗口移动、文本拖选与 Finder 拖放可收到启动真实拖拽所需的事件;默认路径的限制也会明确呈现。 | 发布 `0.3.5`:补齐拖拽位移、手势事件编号与 HID 投递时序,增加投递路径说明;同步升级 Go MCP SDK 到 `v1.4.1`。 |
| 2026-09-08 | drag 投递路径可见性 | 在默认安全配置下调用 `drag` 做窗口移动、文本拖选或 Finder 拖放时,结果会明确说明事件只投递给目标进程、为何没有效果,以及需要设置哪个环境变量,不再表现为“成功但无效果”。 | `drag` 结果新增一条 `Drag delivered via app_post` / `Drag delivered via global pointer path` 文本项;tool description 补充默认路径限制与 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 说明;skill 文档新增 Drag Delivery 章节;默认行为与安全门本身不变。 |
diff --git a/plugins/open-computer-use/.codex-plugin/plugin.json b/plugins/open-computer-use/.codex-plugin/plugin.json
index 3aae196..9033f54 100644
--- a/plugins/open-computer-use/.codex-plugin/plugin.json
+++ b/plugins/open-computer-use/.codex-plugin/plugin.json
@@ -1,7 +1,7 @@
{
"name": "open-computer-use",
"version": "0.3.5",
- "description": "Open-source cross-platform Computer Use MCP server packaged as a Codex plugin.",
+ "description": "Open-source cross-platform Computer Use with a persistent JavaScript REPL.",
"author": {
"name": "Leo",
"url": "https://github.com/iFurySt"
@@ -20,7 +20,7 @@
"interface": {
"displayName": "Open Computer Use",
"shortDescription": "Control desktop apps from Codex with the open local server",
- "longDescription": "Open Computer Use packages this repository's open-source desktop automation server as a Codex plugin. It exposes the same nine desktop-control MCP tools through a local macOS app bundle or Linux / Windows native binary, with semantic accessibility paths preferred before fallback input.",
+ "longDescription": "Open Computer Use packages this repository's open-source desktop automation server as a Codex plugin. The model-facing surface is a persistent Node.js REPL with js and js_reset, while the adapter composes the native accessibility and input actions through an asynchronous cua API on macOS, Linux, and Windows.",
"developerName": "Leo",
"category": "Productivity",
"capabilities": [
@@ -34,11 +34,12 @@
"defaultPrompt": [
"Inspect the current UI state of this desktop app without stealing focus when possible",
"Open a new local document with keyboard shortcuts first, then type content without moving my real pointer unless required",
- "Drive the fixture app and verify the nine computer-use tools still behave correctly"
+ "Drive the fixture app with JavaScript and verify a multi-step flow in one call"
],
"brandColor": "#0E7490",
"composerIcon": "./assets/open-computer-use-small.svg",
"logo": "./assets/open-computer-use.svg",
"screenshots": []
- }
+ },
+ "bundledContentVariant": "node-repl"
}
diff --git a/plugins/open-computer-use/.mcp.json b/plugins/open-computer-use/.mcp.json
index 3700cdb..2db9bb3 100644
--- a/plugins/open-computer-use/.mcp.json
+++ b/plugins/open-computer-use/.mcp.json
@@ -1,7 +1,7 @@
{
"mcpServers": {
"open-computer-use": {
- "command": "./scripts/launch-open-computer-use.sh",
+ "command": "./scripts/launch-open-computer-use-repl.sh",
"cwd": "."
}
}
diff --git a/plugins/open-computer-use/scripts/launch-open-computer-use-repl.sh b/plugins/open-computer-use/scripts/launch-open-computer-use-repl.sh
new file mode 100755
index 0000000..96c9ced
--- /dev/null
+++ b/plugins/open-computer-use/scripts/launch-open-computer-use-repl.sh
@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+plugin_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+repo_root="$(cd "${plugin_root}/../.." && pwd)"
+adapter_candidates=(
+ "${plugin_root}/scripts/open-computer-use-repl.mjs"
+ "${repo_root}/scripts/node-repl/open-computer-use-repl.mjs"
+)
+native_candidates=(
+ "${plugin_root}/Open Computer Use.app/Contents/MacOS/OpenComputerUse"
+ "${plugin_root}/Open Computer Use (Dev).app/Contents/MacOS/OpenComputerUse"
+ "${plugin_root}/OpenComputerUse.app/Contents/MacOS/OpenComputerUse"
+ "${plugin_root}/open-computer-use"
+ "${plugin_root}/open-computer-use.exe"
+ # install-codex-plugin.sh stores the plugin source and runtime payload as
+ # siblings beneath the versioned cache directory.
+ "${repo_root}/Open Computer Use.app/Contents/MacOS/OpenComputerUse"
+ "${repo_root}/Open Computer Use (Dev).app/Contents/MacOS/OpenComputerUse"
+ "${repo_root}/OpenComputerUse.app/Contents/MacOS/OpenComputerUse"
+ "${repo_root}/open-computer-use"
+ "${repo_root}/open-computer-use.exe"
+ "${repo_root}/dist/Open Computer Use (Dev).app/Contents/MacOS/OpenComputerUse"
+ "${repo_root}/dist/Open Computer Use.app/Contents/MacOS/OpenComputerUse"
+ "${repo_root}/dist/OpenComputerUse.app/Contents/MacOS/OpenComputerUse"
+ "${repo_root}/dist/linux/arm64/open-computer-use"
+ "${repo_root}/dist/linux/amd64/open-computer-use"
+ "${repo_root}/dist/windows/arm64/open-computer-use.exe"
+ "${repo_root}/dist/windows/amd64/open-computer-use.exe"
+)
+
+adapter=""
+for candidate in "${adapter_candidates[@]}"; do
+ if [[ -f "${candidate}" ]]; then adapter="${candidate}"; break; fi
+done
+
+if [[ -z "${adapter}" ]]; then
+ echo "open-computer-use could not find its Node REPL adapter." >&2
+ exit 1
+fi
+
+for native in "${native_candidates[@]}"; do
+ if [[ -x "${native}" ]]; then
+ exec node "${adapter}" -- "${native}" mcp
+ fi
+done
+
+if command -v open-computer-use >/dev/null 2>&1; then
+ exec node "${adapter}" -- "$(command -v open-computer-use)" mcp
+fi
+
+echo "open-computer-use could not find a runnable native runtime for the JS REPL adapter." >&2
+exit 1
diff --git a/scripts/ci.sh b/scripts/ci.sh
index ff3e1d2..424b21a 100755
--- a/scripts/ci.sh
+++ b/scripts/ci.sh
@@ -17,6 +17,8 @@ while IFS= read -r file; do
node --check "$file"
done < <(find "${repo_root}/scripts" -type f -name '*.mjs' | sort)
+node --test "${repo_root}/scripts/node-repl/open-computer-use-repl.test.mjs"
+
(
cd "${repo_root}/apps/OpenComputerUseLinux"
python3 -m unittest -v runtime_test.py
diff --git a/scripts/install-codex-plugin.sh b/scripts/install-codex-plugin.sh
index 1879720..8177470 100755
--- a/scripts/install-codex-plugin.sh
+++ b/scripts/install-codex-plugin.sh
@@ -10,6 +10,7 @@ marketplace_name="open-computer-use-local"
plugin_name="open-computer-use"
plugin_source_root="${repo_root}/plugins/${plugin_name}"
plugin_manifest="${plugin_source_root}/.codex-plugin/plugin.json"
+repl_source_root="${repo_root}/scripts/node-repl"
macos_build_script="${repo_root}/scripts/build-open-computer-use-app.sh"
linux_build_script="${repo_root}/scripts/build-open-computer-use-linux.sh"
windows_build_script="${repo_root}/scripts/build-open-computer-use-windows.sh"
@@ -190,6 +191,18 @@ if [[ ! -f "${plugin_manifest}" ]]; then
exit 1
fi
+# Source checkouts keep the adapter under scripts/node-repl, while staged npm
+# packages place it directly in the plugin's scripts directory.
+if [[ ! -f "${repl_source_root}/open-computer-use-repl.mjs" ]]; then
+ repl_source_root="${plugin_source_root}/scripts"
+fi
+for repl_file in open-computer-use-repl.mjs open-computer-use-kernel.mjs; do
+ if [[ ! -f "${repl_source_root}/${repl_file}" ]]; then
+ echo "Missing JS REPL runtime at ${repl_source_root}/${repl_file}" >&2
+ exit 1
+ fi
+done
+
plugin_version="$(node "${config_helper}" codex-plugin-version "${plugin_manifest}")"
if [[ -z "${plugin_version}" ]]; then
@@ -204,7 +217,10 @@ mkdir -p "${codex_home}" "${plugin_cache_root}"
rm -rf "${plugin_install_root}"
mkdir -p "${plugin_install_root}"
-node "${config_helper}" copy-into-dir "${plugin_install_root}" "${plugin_source_root}" "${payload_path}"
+cp -R "${plugin_source_root}/." "${plugin_install_root}/"
+node "${config_helper}" copy-into-dir "${plugin_install_root}" "${payload_path}"
+cp "${repl_source_root}/open-computer-use-repl.mjs" "${plugin_install_root}/scripts/open-computer-use-repl.mjs"
+cp "${repl_source_root}/open-computer-use-kernel.mjs" "${plugin_install_root}/scripts/open-computer-use-kernel.mjs"
node "${config_helper}" codex-plugin-config "${config_path}" "${repo_root}" "${marketplace_name}" "${plugin_name}"
diff --git a/scripts/node-repl/open-computer-use-kernel.mjs b/scripts/node-repl/open-computer-use-kernel.mjs
new file mode 100644
index 0000000..dcfca3b
--- /dev/null
+++ b/scripts/node-repl/open-computer-use-kernel.mjs
@@ -0,0 +1,38 @@
+import { parentPort } from "node:worker_threads";
+import { PersistentJavaScriptSession } from "./open-computer-use-repl.mjs";
+
+let nextNativeId = 1;
+let activeTimeoutMs;
+const pendingNative = new Map();
+const native = {
+ request(method, params, timeoutMs = activeTimeoutMs) {
+ const id = nextNativeId++;
+ return new Promise((resolve, reject) => {
+ pendingNative.set(id, { resolve, reject });
+ parentPort.postMessage({ type: "native_request", id, method, params, timeoutMs });
+ });
+ },
+};
+const session = new PersistentJavaScriptSession({ native });
+
+parentPort.on("message", async message => {
+ if (message.type === "native_response") {
+ const pending = pendingNative.get(message.id);
+ if (!pending) return;
+ pendingNative.delete(message.id);
+ if (message.error) pending.reject(new Error(message.error));
+ else pending.resolve(message.result);
+ return;
+ }
+ if (message.type === "exec") {
+ let result;
+ try {
+ activeTimeoutMs = message.timeoutMs;
+ result = await session.run(message.code, 0, message.requestMeta);
+ } catch (error) {
+ result = { content: [{ type: "text", text: `Error: ${error instanceof Error ? error.message : String(error)}` }], isError: true };
+ }
+ activeTimeoutMs = undefined;
+ parentPort.postMessage({ type: "result", id: message.id, result });
+ }
+});
diff --git a/scripts/node-repl/open-computer-use-repl.mjs b/scripts/node-repl/open-computer-use-repl.mjs
new file mode 100755
index 0000000..8317dc7
--- /dev/null
+++ b/scripts/node-repl/open-computer-use-repl.mjs
@@ -0,0 +1,554 @@
+#!/usr/bin/env node
+
+import { spawn } from "node:child_process";
+import { EventEmitter } from "node:events";
+import { existsSync, realpathSync } from "node:fs";
+import { createRequire } from "node:module";
+import path from "node:path";
+import process from "node:process";
+import { fileURLToPath } from "node:url";
+import { Worker } from "node:worker_threads";
+
+const require = createRequire(import.meta.url);
+const repl = require("node:repl");
+const { PassThrough } = require("node:stream");
+
+const DEFAULT_TIMEOUT_MS = 30_000;
+const MAX_TIMEOUT_MS = 300_000;
+const MAX_FRAME_BYTES = 32 * 1024 * 1024;
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+
+const SERVER_INSTRUCTIONS = `UI automation through a persistent JavaScript REPL using the initialized cua API.
+
+On the first call after startup or js_reset, bind the requested app with await cua.getApp("Example App"), or call await cua.getState() only when an app inventory is actually needed. App bindings persist across js calls. Batch deterministic actions and the resulting getAXState() in one js call, then verify the returned UI state. Use nodeRepl.write(value) for additional text and await nodeRepl.emitImage(image) for additional images. Ask the user before destructive or externally visible actions such as sending, deleting, or purchasing.`;
+
+const JS_DESCRIPTION = `Run JavaScript in a persistent Node.js REPL with top-level await and an initialized asynchronous cua API for Open Computer Use. Use this for all desktop interactions. Bind an app with let app = await cua.getApp("Example App"); then call await app.click(...), await app.typeText(...), and await app.getAXState(). Top-level bindings persist until js_reset, so prefer top-level var for names that may be assigned again. Batch deterministic actions and the resulting state read in one call to reduce round trips. Use nodeRepl.write(value) for extra text and await nodeRepl.emitImage(image) for extra images. If timeout_ms is omitted, execution times out after 30000 ms.`;
+
+const RESET_DESCRIPTION = `Reset the persistent Open Computer Use JavaScript session and discard all bindings created by prior js calls. The next js call starts with a freshly initialized cua API. This does not close native apps or erase their state.`;
+
+const COMPUTER_USE_GUIDANCE = `## Open Computer Use JavaScript API
+
+The runtime exposes an asynchronous app-bound API:
+
+- \`await cua.getState({ emit? })\`: list current apps.
+- \`await cua.listApps({ emit? })\`: list current apps.
+- \`await cua.getApp(nameOrBundleID)\`: bind an app and emit its initial accessibility state.
+- \`await app.getAXState({ emit?, textLimit?, maxTreeNodes?, maxTreeDepth? })\`
+- \`await app.getScreenshot({ emit? })\`
+- \`await app.getAXStateAndScreenshot(options?)\`
+- \`await app.click(elementIndexOrPoint, { mouseButton?, clickCount?, clickMethod? })\`
+- \`await app.scroll(elementIndex, direction, pages?)\`
+- \`await app.drag([fromX, fromY], [toX, toY])\`
+- \`await app.typeText(text)\`
+- \`await app.pressKey(key)\`
+- \`await app.setValue(elementIndex, value)\`
+- \`await app.performSecondaryAction(elementIndex, action)\`
+
+After actions, call \`getAXState()\` in the same js invocation when the next decision depends on the updated UI. Re-derive element indexes from fresh state after navigation or layout changes. Prefer element indexes over coordinates. Open Computer Use keeps its existing local safety gates, including password-manager denial and explicit authorization for global pointer fallbacks.`;
+
+function asError(error) {
+ return error instanceof Error ? error : new Error(String(error));
+}
+
+function isPlainObject(value) {
+ return value !== null && typeof value === "object" && !Array.isArray(value);
+}
+
+function formatValue(value) {
+ if (typeof value === "string") return value;
+ return require("node:util").inspect(value, { depth: 6, colors: false, breakLength: 100 });
+}
+
+function sniffMimeType(bytes) {
+ if (bytes.length >= 8 && bytes.subarray(0, 8).equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))) return "image/png";
+ if (bytes.length >= 3 && bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[2] === 0xff) return "image/jpeg";
+ if (bytes.length >= 12 && bytes.toString("ascii", 0, 4) === "RIFF" && bytes.toString("ascii", 8, 12) === "WEBP") return "image/webp";
+ throw new Error("nodeRepl.emitImage could not infer PNG, JPEG, or WebP MIME type");
+}
+
+function normalizeImage(value) {
+ if (typeof value === "string") {
+ if (!value.startsWith("data:image/")) throw new Error("nodeRepl.emitImage accepts image data URLs or bytes");
+ const match = /^data:([^;,]+);base64,(.*)$/s.exec(value);
+ if (!match) throw new Error("nodeRepl.emitImage expected a base64 image data URL");
+ return { mimeType: match[1], data: match[2] };
+ }
+ if (Buffer.isBuffer(value) || value instanceof Uint8Array || value instanceof ArrayBuffer) {
+ const bytes = Buffer.from(value instanceof ArrayBuffer ? new Uint8Array(value) : value);
+ return { mimeType: sniffMimeType(bytes), data: bytes.toString("base64") };
+ }
+ if (isPlainObject(value) && value.bytes !== undefined) {
+ const bytes = Buffer.from(value.bytes instanceof ArrayBuffer ? new Uint8Array(value.bytes) : value.bytes);
+ const mimeType = typeof value.mimeType === "string" ? value.mimeType : sniffMimeType(bytes);
+ if (!mimeType.startsWith("image/")) throw new Error("nodeRepl.emitImage expected an image MIME type");
+ return { mimeType, data: bytes.toString("base64") };
+ }
+ throw new Error("nodeRepl.emitImage received an unsupported value");
+}
+
+export class JsonLinePeer extends EventEmitter {
+ constructor({ command, args = [], cwd, env = process.env, child } = {}) {
+ super();
+ this.child = child ?? spawn(command, args, { cwd, env, stdio: ["pipe", "pipe", "pipe"] });
+ this.nextId = 1;
+ this.pending = new Map();
+ this.buffer = "";
+ this.stderr = "";
+ this.closed = false;
+ this.child.stdout.setEncoding("utf8");
+ this.child.stderr.setEncoding("utf8");
+ this.child.stdout.on("data", chunk => this.#onData(chunk));
+ this.child.stderr.on("data", chunk => { this.stderr = (this.stderr + chunk).slice(-8192); });
+ this.child.on("error", error => this.#close(error));
+ this.child.on("exit", (code, signal) => this.#close(new Error(`native MCP exited (code=${code ?? "null"}, signal=${signal ?? "null"})${this.stderr ? `: ${this.stderr.trim()}` : ""}`)));
+ }
+
+ #onData(chunk) {
+ this.buffer += chunk;
+ if (Buffer.byteLength(this.buffer, "utf8") > MAX_FRAME_BYTES) return this.#close(new Error("native MCP response buffer exceeded limit"));
+ for (;;) {
+ const newline = this.buffer.indexOf("\n");
+ if (newline < 0) break;
+ const line = this.buffer.slice(0, newline).trim();
+ this.buffer = this.buffer.slice(newline + 1);
+ if (!line) continue;
+ let message;
+ try { message = JSON.parse(line); } catch (error) { this.emit("protocolError", new Error(`invalid native MCP JSON: ${line.slice(0, 200)}`)); continue; }
+ if (message.id !== undefined && this.pending.has(String(message.id))) {
+ const pending = this.pending.get(String(message.id));
+ this.pending.delete(String(message.id));
+ clearTimeout(pending.timer);
+ if (message.error) pending.reject(new Error(message.error.message ?? JSON.stringify(message.error)));
+ else pending.resolve(message.result);
+ } else {
+ this.emit("message", message);
+ }
+ }
+ }
+
+ #close(error) {
+ if (this.closed) return;
+ this.closed = true;
+ for (const pending of this.pending.values()) { clearTimeout(pending.timer); pending.reject(error); }
+ this.pending.clear();
+ this.emit("close", error);
+ }
+
+ request(method, params = {}, timeoutMs = DEFAULT_TIMEOUT_MS) {
+ if (this.closed) return Promise.reject(new Error("native MCP is closed"));
+ const id = this.nextId++;
+ return new Promise((resolve, reject) => {
+ const timer = timeoutMs > 0
+ ? setTimeout(() => { this.pending.delete(String(id)); reject(new Error(`native MCP ${method} timed out after ${timeoutMs} ms`)); }, timeoutMs)
+ : undefined;
+ this.pending.set(String(id), { resolve, reject, timer });
+ this.child.stdin.write(`${JSON.stringify({ jsonrpc: "2.0", id, method, params })}\n`, error => {
+ if (!error) return;
+ if (timer) clearTimeout(timer);
+ this.pending.delete(String(id));
+ reject(error);
+ });
+ });
+ }
+
+ notify(method, params = {}) {
+ if (!this.closed) this.child.stdin.write(`${JSON.stringify({ jsonrpc: "2.0", method, params })}\n`);
+ }
+
+ async initialize() {
+ const result = await this.request("initialize", {
+ protocolVersion: "2025-06-18",
+ capabilities: {},
+ clientInfo: { name: "open-computer-use-node-repl", version: "1" },
+ });
+ this.notify("notifications/initialized", {});
+ return result;
+ }
+
+ close() {
+ if (this.closed) return;
+ this.child.stdin.end();
+ const timer = setTimeout(() => this.child.kill("SIGKILL"), 1_000);
+ timer.unref();
+ this.child.kill("SIGTERM");
+ }
+}
+
+function toolResultText(result) {
+ return (result?.content ?? []).filter(item => item?.type === "text").map(item => item.text ?? "").filter(Boolean).join("\n");
+}
+
+function toolResultImages(result) {
+ return (result?.content ?? []).filter(item => item?.type === "image" && typeof item.data === "string");
+}
+
+function parseListAppMarkers(markers) {
+ const app = {};
+ for (const marker of markers.split(/,\s*/)) {
+ if (marker === "running") app.isRunning = true;
+ else if (marker.startsWith("last-used=")) app.lastUsedDate = marker.slice("last-used=".length);
+ else if (marker.startsWith("uses=")) {
+ const useCount = Number.parseInt(marker.slice("uses=".length), 10);
+ if (Number.isSafeInteger(useCount)) app.useCount = useCount;
+ }
+ }
+ return app;
+}
+
+// The native compatibility surface intentionally returns human-readable text.
+// Normalize its macOS (em dash) and Linux/Windows (double hyphen) renderings
+// into the stable code-first shape used by cua.listApps(). Unknown lines are
+// retained as display names rather than making discovery fail completely.
+export function parseListApps(text) {
+ if (!text.trim() || /^No running top-level apps are visible/u.test(text.trim())) return [];
+ return text.split(/\r?\n/).map(line => {
+ const trimmed = line.trim();
+ const match = /^(.*?)\s+(?:—|--)\s+(.+?)\s+\[([^\]]*)\]$/u.exec(trimmed);
+ if (!match) return { id: trimmed, displayName: trimmed };
+ return { id: match[2], displayName: match[1], ...parseListAppMarkers(match[3]) };
+ }).filter(app => app.id);
+}
+
+function optionsToSnapshotArgs(options = {}) {
+ const args = {};
+ if (options.textLimit !== undefined) args.text_limit = options.textLimit;
+ if (options.maxTreeNodes !== undefined) args.max_tree_nodes = options.maxTreeNodes;
+ if (options.maxTreeDepth !== undefined) args.max_tree_depth = options.maxTreeDepth;
+ return args;
+}
+
+function optionsToClickArgs(options = {}) {
+ const args = {};
+ if (options.mouseButton !== undefined) args.mouse_button = options.mouseButton;
+ if (options.clickCount !== undefined) args.click_count = options.clickCount;
+ if (options.clickMethod !== undefined) args.click_method = options.clickMethod;
+ return args;
+}
+
+export function createCuaApi(native, activeOutput) {
+ let docsEmitted = false;
+ async function call(tool, args = {}) {
+ const result = await native.request("tools/call", { name: tool, arguments: args });
+ if (result?.isError) throw new Error(toolResultText(result) || `${tool} failed`);
+ return result;
+ }
+ async function emitText(text, options = {}) {
+ if (options.emit === false || !text) return;
+ activeOutput().write(text, "cua.state");
+ }
+ async function emitImages(images, options = {}) {
+ if (options.emit === false) return;
+ for (const image of images) await activeOutput().emitImage({ bytes: Buffer.from(image.data, "base64"), mimeType: image.mimeType ?? "image/png" });
+ }
+ async function emitDocs() {
+ if (docsEmitted) return;
+ activeOutput().write(COMPUTER_USE_GUIDANCE, "cua.core");
+ docsEmitted = true;
+ }
+ function appBinding(app) {
+ return Object.freeze({
+ async getAXState(options = {}) {
+ const result = await call("get_app_state", { app, ...optionsToSnapshotArgs(options) });
+ const text = toolResultText(result);
+ await emitText(text, options);
+ return text;
+ },
+ async getScreenshot(options = {}) {
+ const result = await call("get_app_state", { app, text_limit: 1 });
+ const image = toolResultImages(result)[0];
+ if (!image) throw new Error(`Screenshot unavailable for ${app}`);
+ const bytes = Buffer.from(image.data, "base64");
+ await emitImages([image], options);
+ return bytes;
+ },
+ async getAXStateAndScreenshot(options = {}) {
+ const result = await call("get_app_state", { app, ...optionsToSnapshotArgs(options) });
+ const state = toolResultText(result);
+ const images = toolResultImages(result);
+ await emitText(state, options);
+ await emitImages(images, options);
+ return images[0] ? { state, screenshot: Buffer.from(images[0].data, "base64") } : { state };
+ },
+ async click(target, options = {}) {
+ const targetArgs = Array.isArray(target) ? { x: target[0], y: target[1] } : { element_index: target };
+ await call("click", { app, ...targetArgs, ...optionsToClickArgs(options) });
+ },
+ async drag(from, to) { await call("drag", { app, from_x: from[0], from_y: from[1], to_x: to[0], to_y: to[1] }); },
+ async pressKey(key) { await call("press_key", { app, key }); },
+ async scroll(target, direction, pages = 1) {
+ if (Array.isArray(target)) throw new Error("coordinate scroll is not supported by this Open Computer Use runtime");
+ await call("scroll", { app, element_index: target, direction, pages });
+ },
+ async setValue(elementIndex, value) { await call("set_value", { app, element_index: elementIndex, value }); },
+ async typeText(text) { await call("type_text", { app, text }); },
+ async performSecondaryAction(elementIndex, action) { await call("perform_secondary_action", { app, element_index: elementIndex, action }); },
+ });
+ }
+ const api = {
+ async getState(options = {}) {
+ await emitDocs();
+ const result = await call("list_apps", {});
+ const text = toolResultText(result);
+ let apps;
+ try {
+ const parsed = JSON.parse(text);
+ apps = Array.isArray(parsed) ? parsed : parseListApps(text);
+ } catch {
+ apps = parseListApps(text);
+ }
+ if (options.emit !== false) activeOutput().write(apps, "cua.state");
+ return { apps };
+ },
+ async listApps(options = {}) {
+ await emitDocs();
+ const state = await api.getState({ emit: false });
+ if (options.emit !== false) activeOutput().write(state.apps, "cua.state");
+ return state.apps;
+ },
+ async getApp(app) {
+ await emitDocs();
+ const result = await call("get_app_state", { app, text_limit: "max" });
+ await emitText(toolResultText(result));
+ return appBinding(app);
+ },
+ async rewriteDocumentation() { activeOutput().write(COMPUTER_USE_GUIDANCE, "cua.core"); },
+ };
+ return Object.freeze(api);
+}
+
+export class PersistentJavaScriptSession {
+ constructor({ native }) {
+ this.native = native;
+ this.active = null;
+ this.reset();
+ }
+
+ #currentOutput() {
+ if (!this.active) throw new Error("nodeRepl output is only available while js is executing");
+ return this.active;
+ }
+
+ reset() {
+ this.repl?.close();
+ const input = new PassThrough();
+ const output = new PassThrough();
+ output.resume();
+ this.repl = repl.start({ prompt: "", input, output, terminal: false, useGlobal: false, ignoreUndefined: true, breakEvalOnSigint: true });
+ this.repl.on("error", () => {});
+ const nodeRepl = {
+ cwd: process.cwd(),
+ homeDir: process.env.HOME ?? "",
+ tmpDir: process.env.TMPDIR ?? "/tmp",
+ write: (value, itemId) => this.#currentOutput().write(value, itemId),
+ emitImage: async value => this.#currentOutput().emitImage(value),
+ };
+ Object.defineProperty(nodeRepl, "requestMeta", { get: () => this.requestMeta ?? {} });
+ Object.freeze(nodeRepl);
+ this.nodeRepl = nodeRepl;
+ Object.defineProperty(this.repl.context, "nodeRepl", { value: nodeRepl, configurable: false, writable: false });
+ Object.defineProperty(this.repl.context, "cua", { value: createCuaApi(this.native, () => this.#currentOutput()), configurable: false, writable: false });
+ }
+
+ async run(code, timeoutMs = DEFAULT_TIMEOUT_MS, requestMeta = {}) {
+ if (typeof code !== "string" || !code.trim()) throw new Error("js expects non-empty JavaScript source");
+ const content = [];
+ const active = {
+ write(value, itemId) { content.push({ type: "text", text: formatValue(value), ...(itemId ? { _meta: { id: itemId } } : {}) }); },
+ async emitImage(value) { const image = normalizeImage(await value); content.push({ type: "image", mimeType: image.mimeType, data: image.data }); },
+ };
+ this.active = active;
+ this.requestMeta = requestMeta;
+ let timer;
+ const enforceTimeout = Number.isFinite(timeoutMs) && timeoutMs > 0;
+ try {
+ const evaluation = new Promise((resolve, reject) => {
+ this.repl.eval(code, this.repl.context, "open-computer-use-repl", (error, result) => error ? reject(error) : resolve(result));
+ });
+ const value = enforceTimeout
+ ? await Promise.race([evaluation, new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(`js execution timed out after ${timeoutMs} ms; session reset`)), timeoutMs); })])
+ : await evaluation;
+ if (content.length === 0) content.push({ type: "text", text: "(no output)" });
+ return { content, isError: false };
+ } catch (error) {
+ const message = asError(error).message;
+ if (message.includes("timed out")) this.reset();
+ content.push({ type: "text", text: `Error: ${message}` });
+ return { content, isError: true };
+ } finally {
+ clearTimeout(timer);
+ this.active = null;
+ }
+ }
+}
+
+// Production isolation boundary. JavaScript runs in a worker so a CPU-bound
+// loop can be terminated without hanging the MCP transport. Native tool calls
+// are brokered back to the parent and still execute serially in the native MCP.
+export class WorkerJavaScriptSession {
+ constructor({ native }) {
+ this.native = native;
+ this.nextId = 1;
+ this.pending = new Map();
+ this.runQueue = Promise.resolve();
+ this.#start();
+ }
+
+ #start() {
+ const worker = new Worker(new URL("./open-computer-use-kernel.mjs", import.meta.url));
+ this.worker = worker;
+ worker.on("message", message => void this.#onMessage(worker, message));
+ worker.on("error", error => this.#failWorker(worker, error));
+ worker.on("exit", code => { if (code !== 0) this.#failWorker(worker, new Error(`JavaScript kernel exited with code ${code}`)); });
+ }
+
+ async #onMessage(worker, message) {
+ if (message.type === "result") {
+ const pending = this.pending.get(message.id);
+ if (!pending || pending.worker !== worker) return;
+ this.pending.delete(message.id);
+ clearTimeout(pending.timer);
+ pending.resolve(message.result);
+ return;
+ }
+ if (message.type === "native_request") {
+ try {
+ const result = await this.native.request(message.method, message.params, message.timeoutMs);
+ if (this.worker === worker) worker.postMessage({ type: "native_response", id: message.id, result });
+ } catch (error) {
+ if (this.worker === worker) worker.postMessage({ type: "native_response", id: message.id, error: asError(error).message });
+ }
+ }
+ }
+
+ #failWorker(worker, error) {
+ for (const [id, pending] of this.pending) {
+ if (pending.worker !== worker) continue;
+ clearTimeout(pending.timer);
+ pending.resolve({ content: [{ type: "text", text: `Error: ${asError(error).message}` }], isError: true });
+ this.pending.delete(id);
+ }
+ }
+
+ async reset() {
+ const old = this.worker;
+ this.#failWorker(old, new Error("JavaScript session reset"));
+ await old.terminate();
+ if (this.worker === old) this.#start();
+ }
+
+ async close() {
+ const old = this.worker;
+ this.#failWorker(old, new Error("JavaScript session closed"));
+ await old.terminate();
+ }
+
+ run(code, timeoutMs = DEFAULT_TIMEOUT_MS, requestMeta = {}) {
+ const queued = this.runQueue.then(() => this.#run(code, timeoutMs, requestMeta));
+ this.runQueue = queued.catch(() => {});
+ return queued;
+ }
+
+ #run(code, timeoutMs, requestMeta) {
+ const id = this.nextId++;
+ return new Promise(resolve => {
+ const worker = this.worker;
+ const timer = setTimeout(async () => {
+ const pending = this.pending.get(id);
+ if (!pending || pending.worker !== worker) return;
+ this.pending.delete(id);
+ await worker.terminate();
+ if (this.worker === worker) this.#start();
+ resolve({ content: [{ type: "text", text: `Error: js execution timed out after ${timeoutMs} ms; session reset` }], isError: true });
+ }, timeoutMs);
+ this.pending.set(id, { resolve, timer, worker });
+ worker.postMessage({ type: "exec", id, code, timeoutMs, requestMeta });
+ });
+ }
+}
+
+function resolveNativeCommand(argv) {
+ const separator = argv.indexOf("--");
+ if (separator >= 0) {
+ const command = argv[separator + 1];
+ if (!command) throw new Error("expected native MCP command after --");
+ return { command, args: argv.slice(separator + 2) };
+ }
+ const fromEnv = process.env.OPEN_COMPUTER_USE_NATIVE_COMMAND;
+ if (fromEnv) return { command: fromEnv, args: (process.env.OPEN_COMPUTER_USE_NATIVE_ARGS ?? "mcp").split(/\s+/).filter(Boolean) };
+ const candidates = [
+ path.resolve(HERE, "../../dist/Open Computer Use (Dev).app/Contents/MacOS/OpenComputerUse"),
+ path.resolve(HERE, "../../dist/Open Computer Use.app/Contents/MacOS/OpenComputerUse"),
+ path.resolve(HERE, "../../dist/linux/arm64/open-computer-use"),
+ path.resolve(HERE, "../../dist/linux/amd64/open-computer-use"),
+ ];
+ const command = candidates.find(existsSync) ?? "open-computer-use";
+ return { command, args: ["mcp"] };
+}
+
+export async function runServer({ command, args }) {
+ const native = new JsonLinePeer({ command, args, env: process.env });
+ await native.initialize();
+ const session = new WorkerJavaScriptSession({ native });
+ let buffer = "";
+ let requestQueue = Promise.resolve();
+ process.stdin.setEncoding("utf8");
+ process.stdin.on("data", chunk => {
+ buffer += chunk;
+ for (;;) {
+ const newline = buffer.indexOf("\n");
+ if (newline < 0) break;
+ const line = buffer.slice(0, newline).trim();
+ buffer = buffer.slice(newline + 1);
+ if (line) requestQueue = requestQueue.then(() => handle(line)).catch(error => console.error(asError(error).message));
+ }
+ });
+ async function handle(line) {
+ let request;
+ try { request = JSON.parse(line); } catch { return send({ jsonrpc: "2.0", id: null, error: { code: -32700, message: "Invalid JSON-RPC payload" } }); }
+ const { id, method, params = {} } = request;
+ if (id === undefined) {
+ if (method === "notifications/turn-ended") native.notify(method, params);
+ return;
+ }
+ try {
+ if (method === "initialize") return send({ jsonrpc: "2.0", id, result: { protocolVersion: "2025-06-18", capabilities: { tools: { listChanged: false } }, serverInfo: { name: "open-computer-use-repl", version: "1" }, instructions: SERVER_INSTRUCTIONS } });
+ if (method === "ping") return send({ jsonrpc: "2.0", id, result: {} });
+ if (method === "tools/list") return send({ jsonrpc: "2.0", id, result: { tools: toolDefinitions() } });
+ if (method === "tools/call") {
+ if (params.name === "js_reset") { await session.reset(); return send({ jsonrpc: "2.0", id, result: { content: [{ type: "text", text: "Open Computer Use JavaScript session reset" }], isError: false } }); }
+ if (params.name === "js") {
+ const requestedTimeoutMs = params.arguments?.timeout_ms;
+ const timeoutMs = Number.isInteger(requestedTimeoutMs) && requestedTimeoutMs > 0
+ ? Math.min(requestedTimeoutMs, MAX_TIMEOUT_MS)
+ : DEFAULT_TIMEOUT_MS;
+ const result = await session.run(params.arguments?.code, timeoutMs, request._meta ?? {});
+ return send({ jsonrpc: "2.0", id, result });
+ }
+ throw new Error(`Unsupported tool: ${params.name}`);
+ }
+ throw new Error(`Unsupported method: ${method}`);
+ } catch (error) {
+ send({ jsonrpc: "2.0", id, error: { code: -32603, message: asError(error).message } });
+ }
+ }
+ function send(message) { process.stdout.write(`${JSON.stringify(message)}\n`); }
+ let shuttingDown;
+ const shutdown = () => {
+ if (!shuttingDown) shuttingDown = session.close().finally(() => native.close());
+ return shuttingDown;
+ };
+ for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"]) process.on(signal, () => { void shutdown().finally(() => process.exit(0)); });
+ process.stdin.on("end", () => { void requestQueue.finally(shutdown); });
+}
+
+export function toolDefinitions() {
+ const annotations = { destructiveHint: false, openWorldHint: false };
+ return [
+ { name: "js", description: JS_DESCRIPTION, annotations, inputSchema: { type: "object", additionalProperties: false, properties: { code: { type: "string", minLength: 1, description: "JavaScript source to execute with top-level await and the initialized cua API." }, timeout_ms: { type: "integer", minimum: 1, maximum: MAX_TIMEOUT_MS, description: "Optional timeout in milliseconds. Defaults to 30000 and is capped at 300000." }, title: { type: "string", minLength: 1, maxLength: 80, description: "Short user-facing description of what this code is doing." } }, required: ["code"] } },
+ { name: "js_reset", description: RESET_DESCRIPTION, annotations: { ...annotations, readOnlyHint: true }, inputSchema: { type: "object", additionalProperties: false, properties: {} } },
+ ];
+}
+
+if (process.argv[1] && realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url))) {
+ runServer(resolveNativeCommand(process.argv.slice(2))).catch(error => { console.error(`open-computer-use-repl: ${asError(error).message}`); process.exit(1); });
+}
diff --git a/scripts/node-repl/open-computer-use-repl.test.mjs b/scripts/node-repl/open-computer-use-repl.test.mjs
new file mode 100644
index 0000000..552c909
--- /dev/null
+++ b/scripts/node-repl/open-computer-use-repl.test.mjs
@@ -0,0 +1,176 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { EventEmitter } from "node:events";
+import { PassThrough } from "node:stream";
+import { JsonLinePeer, PersistentJavaScriptSession, WorkerJavaScriptSession, parseListApps, toolDefinitions } from "./open-computer-use-repl.mjs";
+
+function textResult(text, isError = false, image) {
+ const content = [{ type: "text", text }];
+ if (image) content.push({ type: "image", mimeType: image.mimeType, data: image.data });
+ return { content, isError };
+}
+
+function mockNative() {
+ const calls = [];
+ const native = {
+ calls,
+ async request(method, params) {
+ assert.equal(method, "tools/call");
+ calls.push(params);
+ switch (params.name) {
+ case "list_apps": return textResult('[{"id":"com.example.Text","displayName":"Text"}]');
+ case "get_app_state": return textResult(`state:${params.arguments.app}`, false, { mimeType: "image/png", data: Buffer.from([0x89,0x50,0x4e,0x47,0x0d,0x0a,0x1a,0x0a]).toString("base64") });
+ case "click": return textResult("clicked");
+ case "type_text": return params.arguments.text === "fail" ? textResult("typing failed", true) : textResult("typed");
+ default: return textResult(params.name);
+ }
+ },
+ };
+ return native;
+}
+
+test("advertises only js and js_reset", () => {
+ assert.deepEqual(toolDefinitions().map(tool => tool.name), ["js", "js_reset"]);
+ assert.deepEqual(toolDefinitions()[0].inputSchema.required, ["code"]);
+ assert.equal(toolDefinitions()[0].inputSchema.properties.timeout_ms.maximum, 300_000);
+});
+
+test("normalizes native app-list text across macOS, Linux, and Windows", () => {
+ assert.deepEqual(parseListApps([
+ "TextEdit — com.apple.TextEdit [frontmost, running, last-used=2026-09-22, uses=42]",
+ "Google Chrome — com.google.Chrome [last-used=2026-09-21, uses=7]",
+ ].join("\n")), [
+ { id: "com.apple.TextEdit", displayName: "TextEdit", isRunning: true, lastUsedDate: "2026-09-22", useCount: 42 },
+ { id: "com.google.Chrome", displayName: "Google Chrome", lastUsedDate: "2026-09-21", useCount: 7 },
+ ]);
+ assert.deepEqual(parseListApps("gedit -- gedit [running, pid=42, window=Draft, notes]"), [
+ { id: "gedit", displayName: "gedit", isRunning: true },
+ ]);
+ assert.deepEqual(parseListApps("No running top-level apps are visible to this Linux runtime."), []);
+});
+
+test("cua.listApps returns structured apps for the native text protocol", async () => {
+ const native = mockNative();
+ native.request = async (method, params) => {
+ assert.equal(method, "tools/call");
+ native.calls.push(params);
+ return textResult("TextEdit — com.apple.TextEdit [running, last-used=2026-09-22, uses=4]");
+ };
+ const session = new PersistentJavaScriptSession({ native });
+ const result = await session.run(`
+ var apps = await cua.listApps({ emit: false });
+ nodeRepl.write(JSON.stringify(apps));
+ `);
+ assert.equal(result.isError, false);
+ assert.deepEqual(JSON.parse(result.content.at(-1).text), [
+ { id: "com.apple.TextEdit", displayName: "TextEdit", isRunning: true, lastUsedDate: "2026-09-22", useCount: 4 },
+ ]);
+});
+
+test("top-level await and lexical bindings persist until reset", async () => {
+ const session = new PersistentJavaScriptSession({ native: mockNative() });
+ let result = await session.run("var answer = await Promise.resolve(41); nodeRepl.write(answer);");
+ assert.equal(result.isError, false);
+ assert.equal(result.content.at(-1).text, "41");
+ result = await session.run("answer += 1; nodeRepl.write(answer);");
+ assert.equal(result.content.at(-1).text, "42");
+ session.reset();
+ result = await session.run("nodeRepl.write(typeof answer);");
+ assert.equal(result.content.at(-1).text, "undefined");
+});
+
+test("return values are not implicitly serialized into MCP output", async () => {
+ const session = new PersistentJavaScriptSession({ native: mockNative() });
+ const result = await session.run("({ huge: 'intermediate-only' })");
+ assert.equal(result.isError, false);
+ assert.equal(result.content.at(-1).text, "(no output)");
+});
+
+test("app-bound API composes actions and state in one js call", async () => {
+ const native = mockNative();
+ const session = new PersistentJavaScriptSession({ native });
+ const result = await session.run(`
+ var app = await cua.getApp("Text");
+ await app.click(7, { clickMethod: "accessibility" });
+ await app.typeText("hello");
+ await app.getAXState();
+ `);
+ assert.equal(result.isError, false);
+ assert.deepEqual(native.calls.map(call => call.name), ["get_app_state", "click", "type_text", "get_app_state"]);
+ assert.equal(native.calls[1].arguments.element_index, 7);
+ assert.equal(native.calls[1].arguments.click_method, "accessibility");
+ assert.ok(result.content.some(item => item.type === "text" && item.text === "state:Text"));
+});
+
+test("tool errors are catchable in JavaScript", async () => {
+ const session = new PersistentJavaScriptSession({ native: mockNative() });
+ const result = await session.run(`
+ var badApp = await cua.getApp("Text");
+ try { await badApp.typeText("fail"); } catch (error) { nodeRepl.write("caught: " + error.message); }
+ `);
+ assert.equal(result.isError, false);
+ assert.equal(result.content.at(-1).text, "caught: typing failed");
+});
+
+test("screenshots emit binary image content", async () => {
+ const session = new PersistentJavaScriptSession({ native: mockNative() });
+ const result = await session.run(`
+ var imageApp = await cua.getApp("Text");
+ var imageBytes = await imageApp.getScreenshot();
+ nodeRepl.write(imageBytes.length);
+ `);
+ assert.equal(result.isError, false);
+ assert.ok(result.content.some(item => item.type === "image" && item.mimeType === "image/png"));
+ assert.equal(result.content.at(-1).text, "8");
+});
+
+test("JsonLinePeer initializes and correlates native MCP responses", async () => {
+ const child = new EventEmitter();
+ child.stdout = new PassThrough(); child.stderr = new PassThrough(); child.stdin = new PassThrough();
+ child.kill = () => {};
+ let input = "";
+ child.stdin.setEncoding("utf8");
+ child.stdin.on("data", chunk => {
+ input += chunk;
+ for (;;) {
+ const newline = input.indexOf("\n"); if (newline < 0) break;
+ const request = JSON.parse(input.slice(0, newline)); input = input.slice(newline + 1);
+ if (request.id !== undefined) child.stdout.write(`${JSON.stringify({ jsonrpc: "2.0", id: request.id, result: request.method === "initialize" ? { ok: true } : { content: [] } })}\n`);
+ }
+ });
+ const peer = new JsonLinePeer({ child });
+ assert.deepEqual(await peer.initialize(), { ok: true });
+ assert.deepEqual(await peer.request("tools/call", { name: "list_apps" }), { content: [] });
+ peer.closed = true;
+});
+
+test("worker session terminates CPU-bound code and recovers with a fresh kernel", async () => {
+ const session = new WorkerJavaScriptSession({ native: mockNative() });
+ let result = await session.run("while (true) {}", 100);
+ assert.equal(result.isError, true);
+ assert.match(result.content[0].text, /timed out/);
+ result = await session.run("nodeRepl.write(21 * 2);", 5_000);
+ assert.equal(result.isError, false);
+ assert.equal(result.content.at(-1).text, "42");
+ await session.close();
+});
+
+test("worker session returns a structured error for invalid source input", async () => {
+ const session = new WorkerJavaScriptSession({ native: mockNative() });
+ const result = await session.run(undefined, 5_000);
+ assert.equal(result.isError, true);
+ assert.match(result.content[0].text, /non-empty JavaScript source/);
+ const recovered = await session.run("nodeRepl.write('ready');", 5_000);
+ assert.equal(recovered.isError, false);
+ assert.equal(recovered.content.at(-1).text, "ready");
+ await session.close();
+});
+
+test("worker session serializes concurrent callers", async () => {
+ const session = new WorkerJavaScriptSession({ native: mockNative() });
+ const first = session.run("await new Promise(resolve => setTimeout(resolve, 40)); var order = ['first']; nodeRepl.write(order.join(','));", 5_000);
+ const second = session.run("order.push('second'); nodeRepl.write(order.join(','));", 5_000);
+ assert.equal((await first).content.at(-1).text, "first");
+ assert.equal((await second).content.at(-1).text, "first,second");
+ await session.close();
+});
diff --git a/scripts/npm/build-packages.mjs b/scripts/npm/build-packages.mjs
index 99dbc4c..79258a7 100644
--- a/scripts/npm/build-packages.mjs
+++ b/scripts/npm/build-packages.mjs
@@ -562,6 +562,14 @@ function copyInstallerScripts(packageRoot) {
cpSync(path.join(repoRoot, "scripts", "install-dsh-mcp.sh"), path.join(packageRoot, "scripts", "install-dsh-mcp.sh"));
cpSync(path.join(repoRoot, "scripts", "install-codex-plugin.sh"), path.join(packageRoot, "scripts", "install-codex-plugin.sh"));
cpSync(path.join(repoRoot, "skills", "open-computer-use"), path.join(packageRoot, "skills", "open-computer-use"), { recursive: true });
+ cpSync(
+ path.join(repoRoot, "scripts", "node-repl", "open-computer-use-repl.mjs"),
+ path.join(packageRoot, "plugins", "open-computer-use", "scripts", "open-computer-use-repl.mjs")
+ );
+ cpSync(
+ path.join(repoRoot, "scripts", "node-repl", "open-computer-use-kernel.mjs"),
+ path.join(packageRoot, "plugins", "open-computer-use", "scripts", "open-computer-use-kernel.mjs")
+ );
for (const scriptName of [
"install-claude-mcp.sh",
diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.md
index a4793c5..37a153e 100644
--- a/skills/open-computer-use/references/usage.md
+++ b/skills/open-computer-use/references/usage.md
@@ -41,6 +41,19 @@ press_key
set_value
```
+This remains the native compatibility surface. The bundled Codex plugin uses a
+code-first adapter instead and advertises only `js` and `js_reset`. In that
+surface, bind an app and compose actions with asynchronous JavaScript:
+
+```js
+var app = await cua.getApp("TextEdit");
+await app.typeText("Hello");
+await app.getAXState();
+```
+
+Bindings persist across `js` calls until `js_reset`. See
+`docs/references/js-repl.md` for the full API and security boundary.
+
## Direct CLI Tool Calls
Use `call` for one-off checks: