👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Add open-computer-use skill
ifuryst2e678e2ca1bb
diff --git a/README.md b/README.mdindex ee4dcdd..7e8c61e 100644--- a/README.md+++ b/README.md@@ -69,6 +69,31 @@ Or add it to your own client manually:}```+### Skill++Install the skill directly:++```bash+# Install for Codex+npx skills add iFurySt/open-codex-computer-use -g -a codex --skill open-computer-use -y+npx skills ls -g -a codex | rg 'open-computer-use'+```++Install for Claude Code:++```bash+npx skills add iFurySt/open-codex-computer-use -g -a claude-code --skill open-computer-use -y+```++Update an existing global install, including the Codex install created above:++```bash+npx skills update open-computer-use -g -y+```++You can also manually download and install the+[`open-computer-use` skill](./skills/open-computer-use).+## MoreBesides the MCP JSON config above, you can also use the built-in commands:diff --git a/README.zh-CN.md b/README.zh-CN.mdindex fc5bd02..3c0fe77 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -67,6 +67,29 @@ open-computer-use install-codex-mcp}```+### Skill++一键安装skill:++```bash+# 安装到 Codex+npx skills add iFurySt/open-codex-computer-use -g -a codex --skill open-computer-use -y+npx skills ls -g -a codex | rg 'open-computer-use'+```++安装到 Claude Code+```+npx skills add iFurySt/open-codex-computer-use -g -a claude-code --skill open-computer-use -y+```++更新已有的全局安装,包括上面安装到 Codex 的那份:++```bash+npx skills update open-computer-use -g -y+```++也可以手动下载 [`open-computer-use` skill](./skills/open-computer-use) 安装+## 更多除了直接用上面的 MCP JSON 配置,你也可以用一些内置子命令:diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex e68e14f..e941d1d 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -28,6 +28,8 @@新的独立 Swift cursor viewer,直接复用 `scripts/cursor-motion-re/official_cursor_motion.py` 里收敛出来的候选路径、score 与 raw spring timeline,用来观察更贴近 binary lift 的表现。- `scripts/`仓库级自动化命令,包括 smoke test、`.app` 打包入口、Windows `.exe` / Linux binary 构建入口、npm 分发脚本,以及 `scripts/computer-use-cli/` 这个用于探测官方 bundled `computer-use` 的 Go helper。+- `skills/`+ 面向 agent runtime 的可安装 skill。当前 `skills/open-computer-use/SKILL.md` 只作为轻量入口和目录,安装、MCP/CLI 使用、排障等细节拆到相邻 `references/` 文件里按需加载;`scripts/package-skill.sh` 负责校验并打包 `.zip` / `.skill` 制品。- `docs/`逆向分析、执行计划、history 和项目约束。@@ -138,6 +140,7 @@- app 打包:`./scripts/build-open-computer-use-app.sh debug`- npm staging:`node ./scripts/npm/build-packages.mjs`- release tgz:`./scripts/release-package.sh`+- skill 打包:`npm run package:skill`- Windows runtime 单测:`(cd apps/OpenComputerUseWindows && go test ./...)`- Windows exe 构建:`./scripts/build-open-computer-use-windows.sh --arch arm64`- Linux runtime 单测:`(cd apps/OpenComputerUseLinux && go test ./...)`diff --git a/docs/histories/2026-05/20260516-1117-open-computer-use-skill.md b/docs/histories/2026-05/20260516-1117-open-computer-use-skill.mdnew file mode 100644index 0000000..bb13d72--- /dev/null+++ b/docs/histories/2026-05/20260516-1117-open-computer-use-skill.md@@ -0,0 +1,31 @@+# Open Computer Use skill++## 用户诉求++用户希望确认仓库当前是否没有 skills,并参考 `open-codex-browser-use` 的 `skills/` 结构,为本仓库补上 `open-computer-use` skill;`SKILL.md` 作为 TOC,安装、使用、排障等内容拆到按需加载的文件里;同时在中英文 README 里增加 `npx skills` 安装和升级方式。++## 改动++- 新增 `skills/open-computer-use/SKILL.md`,作为 Open Computer Use agent skill 入口和目录。+- 新增 `skills/open-computer-use/references/installation.md`、`usage.md`、`troubleshooting.md`,拆分安装、MCP/CLI 使用和排障说明。+- 新增 `skills/open-computer-use/agents/openai.yaml`,提供 agent 展示元数据和默认提示。+- 新增 `scripts/package-skill.sh` 与 `package:skill` npm script,用于校验并打包 `.zip` / `.skill` 制品。+- 更新 `README.md` 和 `README.zh-CN.md`,加入 `npx skills add` 和 `update` 的安装与升级命令。+- 更新 `docs/ARCHITECTURE.md`,补充 `skills/` 目录和 skill 打包验证路径。++## 设计动机++沿用 `open-codex-browser-use` 的 skill 组织方式,让 agent 先读取轻量入口,再按任务需要打开安装、使用或排障参考,避免把所有细节塞进单个 `SKILL.md`。同时保留打包脚本,方便后续 release 附带可下载 skill 制品。++## 影响文件++- `skills/open-computer-use/SKILL.md`+- `skills/open-computer-use/references/installation.md`+- `skills/open-computer-use/references/usage.md`+- `skills/open-computer-use/references/troubleshooting.md`+- `skills/open-computer-use/agents/openai.yaml`+- `scripts/package-skill.sh`+- `package.json`+- `README.md`+- `README.zh-CN.md`+- `docs/ARCHITECTURE.md`diff --git a/package.json b/package.jsonindex fd1f1ac..5867b9f 100644--- a/package.json+++ b/package.json@@ -5,6 +5,7 @@"scripts": {"npm:build": "node ./scripts/npm/build-packages.mjs","npm:pack": "./scripts/release-package.sh",- "npm:publish": "node ./scripts/npm/publish-packages.mjs"+ "npm:publish": "node ./scripts/npm/publish-packages.mjs",+ "package:skill": "./scripts/package-skill.sh"}}diff --git a/scripts/package-skill.sh b/scripts/package-skill.shnew file mode 100755index 0000000..ec35b19--- /dev/null+++ b/scripts/package-skill.sh@@ -0,0 +1,127 @@+#!/usr/bin/env bash++set -euo pipefail++repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+skill_name="open-computer-use"+skill_dir="${repo_root}/skills/${skill_name}"+dist_dir="${repo_root}/dist/skills"+zip_path="${dist_dir}/${skill_name}-skill.zip"+skill_path="${dist_dir}/${skill_name}.skill"+manifest_path="${dist_dir}/package-manifest.json"++if ! command -v node >/dev/null 2>&1; then+ echo "node is required to validate the skill package" >&2+ exit 1+fi++if ! command -v zip >/dev/null 2>&1; then+ echo "zip is required to package the skill" >&2+ exit 1+fi++if ! command -v unzip >/dev/null 2>&1; then+ echo "unzip is required to validate the skill package" >&2+ exit 1+fi++if [[ ! -f "${skill_dir}/SKILL.md" ]]; then+ echo "missing skill entrypoint: ${skill_dir}/SKILL.md" >&2+ exit 1+fi++node - "${skill_dir}/SKILL.md" <<'NODE'+const fs = require("fs");++const skillPath = process.argv[2];+const content = fs.readFileSync(skillPath, "utf8");+const errors = [];++if (!content.startsWith("---\n")) {+ errors.push("SKILL.md must start with YAML frontmatter");+}+if (!/^name:\s*open-computer-use\s*$/m.test(content)) {+ errors.push("SKILL.md frontmatter must include name: open-computer-use");+}+if (!/^description:\s*\S/m.test(content)) {+ errors.push("SKILL.md frontmatter must include a non-empty description");+}++if (errors.length > 0) {+ for (const error of errors) {+ console.error(error);+ }+ process.exit(1);+}+NODE++rm -rf "${dist_dir}"+mkdir -p "${dist_dir}"++(+ cd "${repo_root}/skills"+ zip -q -r "${zip_path}" "${skill_name}"+)+cp "${zip_path}" "${skill_path}"++node - "${zip_path}" "${skill_path}" "${manifest_path}" <<'NODE'+const crypto = require("crypto");+const fs = require("fs");++const zipPath = process.argv[2];+const skillPath = process.argv[3];+const manifestPath = process.argv[4];+const zip = fs.readFileSync(zipPath);+const skill = fs.readFileSync(skillPath);++const payload = {+ name: "open-computer-use",+ rootDirectory: "open-computer-use",+ artifacts: {+ zip: zipPath,+ skill: skillPath+ },+ sha256: {+ zip: crypto.createHash("sha256").update(zip).digest("hex"),+ skill: crypto.createHash("sha256").update(skill).digest("hex")+ },+ generatedAtUtc: new Date().toISOString()+};++if (payload.sha256.zip !== payload.sha256.skill) {+ throw new Error("open-computer-use-skill.zip and open-computer-use.skill must contain identical bytes");+}++fs.writeFileSync(manifestPath, `${JSON.stringify(payload, null, 2)}\n`);+NODE++if ! cmp -s "${zip_path}" "${skill_path}"; then+ echo "open-computer-use-skill.zip and open-computer-use.skill differ" >&2+ exit 1+fi++has_skill_entrypoint=0+entry_count=0+while IFS= read -r entry; do+ entry_count=$((entry_count + 1))+ if [[ "${entry}" != "${skill_name}/"* ]]; then+ echo "skill zip entry must be under ${skill_name}/: ${entry}" >&2+ exit 1+ fi+ if [[ "${entry}" == "${skill_name}/SKILL.md" ]]; then+ has_skill_entrypoint=1+ fi+done < <(unzip -Z1 "${zip_path}")++if [[ "${entry_count}" -eq 0 ]]; then+ echo "skill zip is empty" >&2+ exit 1+fi++if [[ "${has_skill_entrypoint}" -ne 1 ]]; then+ echo "skill zip is missing ${skill_name}/SKILL.md" >&2+ exit 1+fi++echo "${zip_path}"+echo "${skill_path}"diff --git a/skills/open-computer-use/SKILL.md b/skills/open-computer-use/SKILL.mdnew file mode 100644index 0000000..de51720--- /dev/null+++ b/skills/open-computer-use/SKILL.md@@ -0,0 +1,73 @@+---+name: open-computer-use+description: Platform-neutral guidance for using Open Computer Use, the open-source Computer Use MCP server and CLI for macOS, Linux, and Windows. Use when an agent needs to install, verify, troubleshoot, configure, or operate Open Computer Use through its native CLI, stdio MCP server, or direct Computer Use tool calls.+---++# Open Computer Use++## Overview++Open Computer Use exposes Computer Use as a local CLI and stdio MCP server. It is not Codex.app-specific; adapt the commands and MCP config to the agent runtime you are operating in.++It supports the same core tool surface across macOS, Linux, and Windows:+`list_apps`, `get_app_state`, `click`, `perform_secondary_action`, `scroll`,+`drag`, `type_text`, `press_key`, and `set_value`.++## Core Workflow++1. Check the CLI is installed with `open-computer-use -h`. If installation or setup is missing, read [references/installation.md](references/installation.md).+2. On macOS, run `open-computer-use doctor` before the first real GUI task. If permissions are missing, ask the user to approve Accessibility and Screen Recording in the onboarding UI.+3. Inspect available apps before acting: `open-computer-use call list_apps`.+4. Capture current UI state with `open-computer-use call get_app_state --args '{"app":"TextEdit"}'`.+5. Prefer element-targeted actions using `element_index` from the latest `get_app_state` result.+6. For multi-step CLI work, use `open-computer-use call --calls '<json-array>'` so one process can reuse the latest element index mapping.+7. For agent runtimes that support local MCP servers, configure `open-computer-use mcp` and call the exposed Computer Use tools directly. Read [references/usage.md](references/usage.md).+8. If communication, permission, or desktop-session access fails, read [references/troubleshooting.md](references/troubleshooting.md).++## Operating Rules++- Treat the target desktop as the user's real session. Do not inspect password managers, unrelated private content, or sensitive apps unless the user explicitly asked for that task.+- Ask before sending, deleting, purchasing, approving, uploading, or making other externally visible changes.+- Do not assume Codex.app plugin helpers are available. Use the installed `open-computer-use` CLI or an explicit MCP config.+- Always run `get_app_state` before using `element_index`; do not guess indexes across sessions or after large UI changes.+- Prefer semantic actions and `set_value` for editable controls. Use coordinate `click`, `scroll`, and `drag` only when the element tree does not expose a safer target.+- On macOS, do not enable `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` unless the user explicitly wants diagnostic behavior that may move the real pointer.+- On Windows and Linux, confirm the command is running inside the logged-in desktop session before assuming GUI automation is available.++## Common CLI Actions++```sh+open-computer-use -h+open-computer-use doctor+open-computer-use call list_apps+open-computer-use call get_app_state --args '{"app":"TextEdit"}'+open-computer-use call click --args '{"app":"TextEdit","element_index":0}'+open-computer-use call type_text --args '{"app":"TextEdit","text":"Hello from Open Computer Use"}'+```++For a short sequence that reuses state in one process:++```sh+open-computer-use call --calls '[+ {"tool":"get_app_state","args":{"app":"TextEdit"}},+ {"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}+]'+```++## MCP Usage++For runtimes that can launch local MCP servers over stdio, use:++```toml+[mcp_servers.open_computer_use]+command = "open-computer-use"+args = ["mcp"]+```++Read [references/usage.md](references/usage.md) for JSON config examples, direct tool-call patterns, and platform notes.++## References++- [references/installation.md](references/installation.md): one-time CLI install, agent MCP install commands, and macOS permissions.+- [references/usage.md](references/usage.md): MCP config, direct CLI calls, sequencing, and platform behavior.+- [references/troubleshooting.md](references/troubleshooting.md): permission, desktop-session, app discovery, and action failures.diff --git a/skills/open-computer-use/agents/openai.yaml b/skills/open-computer-use/agents/openai.yamlnew file mode 100644index 0000000..a5a60ec--- /dev/null+++ b/skills/open-computer-use/agents/openai.yaml@@ -0,0 +1,4 @@+interface:+ display_name: "Open Computer Use(OCU)"+ short_description: "Guide agents using Open Computer Use"+ default_prompt: "Use $open-computer-use to operate desktop apps through Open Computer Use."diff --git a/skills/open-computer-use/references/installation.md b/skills/open-computer-use/references/installation.mdnew file mode 100644index 0000000..ad93384--- /dev/null+++ b/skills/open-computer-use/references/installation.md@@ -0,0 +1,102 @@+# Open Computer Use Installation++Read this reference when the user asks to install, verify, repair, or explain Open Computer Use setup.++## Install The CLI++Use npm:++```sh+npm install -g open-computer-use+```++Verify:++```sh+open-computer-use -h+open-computer-use call list_apps+```++If the package is already installed and the user asks to update it:++```sh+npm update -g open-computer-use+```++## macOS Permissions++macOS needs Accessibility and Screen Recording permissions before real app state and actions can work.++Run:++```sh+open-computer-use doctor+```++If permissions are missing, the onboarding UI opens. Ask the user to grant the requested permissions in System Settings. Do not try to bypass TCC prompts or silently manipulate protected settings.++Windows and Linux do not use this macOS onboarding step, but they still need a logged-in desktop session.++## Install Into Agent MCP Configs++Use the built-in installers when they match the user's agent:++```sh+open-computer-use install-codex-mcp+open-computer-use install-claude-mcp+open-computer-use install-gemini-mcp+open-computer-use install-gemini-mcp --scope user+open-computer-use install-opencode-mcp+```++Codex App can also use the plugin installer:++```sh+open-computer-use install-codex-plugin+```++For any other MCP client, add a stdio server manually:++```json+{+ "mcpServers": {+ "open-computer-use": {+ "command": "open-computer-use",+ "args": ["mcp"]+ }+ }+}+```++## Install This Skill++Install the skill for Codex:++```sh+npx skills add iFurySt/open-codex-computer-use -g -a codex --skill open-computer-use -y+npx skills ls -g -a codex | rg 'open-computer-use'+```++Install the skill for Claude Code:++```sh+npx skills add iFurySt/open-codex-computer-use -g -a claude-code --skill open-computer-use -y+```++Update an existing global skill install:++```sh+npx skills update open-computer-use -g -y+npx skills upgrade open-computer-use -g -y+```++## Verification++After CLI and MCP setup:++```sh+open-computer-use call list_apps+open-computer-use call get_app_state --args '{"app":"TextEdit"}'+```++If this fails, read [troubleshooting.md](troubleshooting.md).diff --git a/skills/open-computer-use/references/troubleshooting.md b/skills/open-computer-use/references/troubleshooting.mdnew file mode 100644index 0000000..9628359--- /dev/null+++ b/skills/open-computer-use/references/troubleshooting.md@@ -0,0 +1,61 @@+# Open Computer Use Troubleshooting++Read this reference when setup, permission checks, app discovery, snapshots, or actions fail.++## First Checks++Start with:++```sh+open-computer-use -h+open-computer-use doctor+open-computer-use call list_apps+```++On macOS, `doctor` reports Accessibility and Screen Recording status. If either is missing, ask the user to approve the onboarding UI.++## App Not Found++If `get_app_state` cannot find an app:++1. Run `open-computer-use call list_apps`.+2. Use the app name or bundle identifier from that result.+3. Confirm the app is running and has a visible, non-minimized window.+4. On macOS, rerun `open-computer-use doctor`.++Do not silently switch to a different app when the requested target is not available.++## Empty Or Missing Snapshot++Common causes:++- The app has no visible window.+- The window is minimized, hidden, or on an unavailable desktop.+- macOS Screen Recording permission is missing.+- Windows or Linux commands are running outside the logged-in desktop session.+- Linux screenshot support is blocked by the compositor or desktop portal state.++Ask the user to bring the target app/window into a visible state when automation cannot do so safely.++## Element Action Fails++If an element-targeted action fails:++1. Re-run `get_app_state`.+2. Confirm the `element_index` still exists and refers to the intended UI element.+3. Prefer `set_value` for settable text/value controls.+4. Prefer `perform_secondary_action` only for actions exposed in the state result.+5. Use coordinate `click`, `scroll`, or `drag` only after the semantic route is unavailable.++## Desktop Session Issues++Windows UI Automation and Linux AT-SPI require a live user desktop. SSH sessions, CI jobs, launch daemons, or services often do not have access to the GUI session even when the CLI binary starts successfully.++If desktop access is missing, ask the user to run the command from the logged-in desktop session or start the target app visibly in that session.++## Permission And Safety Issues++- Do not bypass macOS TCC prompts.+- Do not enable global pointer fallbacks unless the user asks for low-level diagnostic behavior.+- Do not interact with password managers or sensitive apps unless the user explicitly requests it.+- Pause before submitting, sending, deleting, purchasing, or approving anything externally visible.diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.mdnew file mode 100644index 0000000..b94e924--- /dev/null+++ b/skills/open-computer-use/references/usage.md@@ -0,0 +1,91 @@+# Open Computer Use Usage++Read this reference when the task requires direct Computer Use tool calls, MCP configuration, or platform-specific behavior.++## MCP Server++For MCP clients that support stdio servers:++```toml+[mcp_servers.open_computer_use]+command = "open-computer-use"+args = ["mcp"]+```++Equivalent JSON shape:++```json+{+ "mcpServers": {+ "open-computer-use": {+ "command": "open-computer-use",+ "args": ["mcp"]+ }+ }+}+```++The MCP server exposes:++```text+list_apps+get_app_state+click+perform_secondary_action+scroll+drag+type_text+press_key+set_value+```++## Direct CLI Tool Calls++Use `call` for one-off checks:++```sh+open-computer-use call list_apps+open-computer-use call get_app_state --args '{"app":"TextEdit"}'+open-computer-use call set_value --args '{"app":"TextEdit","element_index":1,"value":"Draft"}'+```++Use `--calls` for short action sequences that need to reuse the same process state:++```sh+open-computer-use call --calls '[+ {"tool":"get_app_state","args":{"app":"TextEdit"}},+ {"tool":"click","args":{"app":"TextEdit","element_index":1}},+ {"tool":"type_text","args":{"app":"TextEdit","text":"Hello"}}+]'+```++Use `--calls-file` when the sequence is too large for a readable shell command:++```sh+open-computer-use call --calls-file examples/textedit-overlay-seq.json --sleep 0.5+```++## Choosing Targets++- Prefer app names or bundle identifiers returned by `list_apps`.+- Run `get_app_state` immediately before element-targeted actions.+- Re-run `get_app_state` after navigation, modal changes, page reloads, or failed actions.+- Use coordinate actions only when the rendered tree does not expose the target as an element.++## Platform Notes++### macOS++The macOS runtime uses Accessibility, ScreenCaptureKit, and targeted input events. It normally avoids moving the user's real pointer. The visual cursor overlay is part of the Open Computer Use experience and can be disabled by the surrounding runtime only when needed.++### Windows++The Windows runtime uses UI Automation and Win32 message fallbacks. It must run in a logged-in desktop session. A detached SSH or service context may start the CLI but fail to see top-level windows.++### Linux++The Linux runtime uses AT-SPI2 through the desktop session bus. It must run in a logged-in graphical session with usable accessibility services. Wayland screenshot and coordinate input support is compositor-dependent and best-effort.++## Safety++Pause and ask the user before actions that affect external systems or sensitive local state, including sending messages, submitting forms, deleting files, approving prompts, uploading files, or interacting with password managers.