👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
test: add agent and stress validation
JohanSardar9d9d5d90f4ba
diff --git a/Makefile b/Makefileindex 0f9c174..20dd4ef 100644--- a/Makefile+++ b/Makefile@@ -1,7 +1,9 @@PROJECT ?=SLUG ?=+AGENTS ?= claude,codex+SCENARIO ?= list-apps-.PHONY: init build app test smoke check-docs check-repo ci release-package npm-build npm-publish new-history new-plan+.PHONY: init build app test smoke stress agent-smoke check-docs check-repo ci release-package npm-build npm-publish new-history new-planinit:@if [ -z "$(PROJECT)" ]; then echo "用法: make init PROJECT=项目名"; exit 1; fi@@ -19,6 +21,12 @@ test:smoke:./scripts/run-tool-smoke-tests.sh+stress:+ ./scripts/run-tool-stress-tests.sh++agent-smoke:+ node ./scripts/run-agent-smoke-tests.mjs --agents=$(AGENTS) --scenario=$(SCENARIO)+check-docs:./scripts/check-docs.shdiff --git a/README.md b/README.mdindex 7e8c61e..f319290 100644--- a/README.md+++ b/README.md@@ -129,6 +129,19 @@ open-computer-use call --calls-file examples/textedit-overlay-seq.json --sleep 0# Check permissions; onboarding only opens when something is missingopen-computer-use doctor+# Run local validation from a source checkout+make smoke+OPEN_COMPUTER_USE_STRESS_LOOPS=20 make stress+make agent-smoke+make agent-smoke SCENARIO=fixture-full+node ./scripts/run-agent-smoke-tests.mjs --agents=claude,codex --command=open-computer-use+node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture --agents=claude,codex --command=open-computer-use+node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture-full --agents=claude,codex --command=open-computer-use+OPEN_COMPUTER_USE_HERMES_PROVIDER=anthropic OPEN_COMPUTER_USE_HERMES_MODEL=claude-opus-4-20250514 make agent-smoke AGENTS=hermes SCENARIO=fixture-full+node ./scripts/run-agent-smoke-tests.mjs --agents=hermes --hermes-provider=anthropic --hermes-model=claude-opus-4-20250514+node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture --agents=hermes --hermes-provider=anthropic --hermes-model=claude-opus-4-20250514+node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture-full --agents=hermes --hermes-provider=anthropic --hermes-model=claude-opus-4-20250514 --hermes-max-turns=12+# Show helpopen-computer-use -h```diff --git a/docs/histories/2026-06/20260601-2002-agent-stress-validation.md b/docs/histories/2026-06/20260601-2002-agent-stress-validation.mdnew file mode 100644index 0000000..02187c0--- /dev/null+++ b/docs/histories/2026-06/20260601-2002-agent-stress-validation.md@@ -0,0 +1,46 @@+## [2026-06-01 20:02] | Task: Agent stress validation++### Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `Codex desktop`++### User Query+> Add validation that proves the open computer-use MCP can be installed, stress tested, and used from Claude, Hermes, and Codex CLI.++### Changes Overview+**Scope:** repository validation scripts and docs++**Key Actions:**+- **Added deterministic stress runner**: `scripts/run-tool-stress-tests.sh` repeats the fixture-backed smoke suite and runs the visual cursor idle smoke.+- **Added optional agent smoke runner**: `scripts/run-agent-smoke-tests.mjs` can ask Claude Code, Codex CLI, and Hermes to call the `open-computer-use` MCP `list_apps` tool. The default agent set is Claude/Codex; Hermes stays explicit so provider/model routing can be supplied when needed.+- **Added fixture agent scenario**: `--scenario=fixture` launches `OpenComputerUseFixture`, asks the selected agent to call `get_app_state`, `set_value`, and `click`, then independently verifies the fixture text field and counter state.+- **Added full fixture agent scenario**: `--scenario=fixture-full` asks agents to use `get_app_state`, `set_value`, `click`, `type_text`, `press_key`, `perform_secondary_action`, `scroll`, and `drag`, then verifies exact final fixture state.+- **Kept Hermes model and turn routing configurable**: the smoke runner accepts `--hermes-provider`, `--hermes-model`, and `--hermes-max-turns` so local installations with stale default model aliases or longer tool-call plans can still validate MCP wiring.+- **Added make targets and docs**: `make stress`, `make agent-smoke`, configurable `AGENTS`/`SCENARIO`, and README examples document the validation path.++### Design Intent (Why)+The project already had a single smoke pass, but agent integrations and repeated fixture loops were manual. These scripts make the validation path reproducible without requiring live agents to perform hundreds of GUI mutations.++### Files Modified+- `scripts/run-tool-stress-tests.sh`+- `scripts/run-agent-smoke-tests.mjs`+- `Makefile`+- `README.md`++### Validation+- `OPEN_COMPUTER_USE_STRESS_LOOPS=2 OPEN_COMPUTER_USE_STRESS_CONFIGURATION=release ./scripts/run-tool-stress-tests.sh`+- `node ./scripts/run-agent-smoke-tests.mjs --agents=claude --command=open-computer-use --json`+- `node ./scripts/run-agent-smoke-tests.mjs --agents=codex --command=open-computer-use --json`+- `node ./scripts/run-agent-smoke-tests.mjs --agents=hermes --command=open-computer-use --hermes-provider=anthropic --hermes-model=claude-opus-4-20250514 --json`+- `node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture --agents=codex --command=open-computer-use --json`+- `node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture --agents=claude --command=open-computer-use --json --claude-budget-usd=3.00`+- `node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture --agents=hermes --command=open-computer-use --hermes-provider=anthropic --hermes-model=claude-opus-4-20250514 --json`+- `node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture-full --agents=codex --command=open-computer-use --json --timeout-ms=180000`+- `node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture-full --agents=claude --command=open-computer-use --json --timeout-ms=180000 --claude-budget-usd=3.00`+- `node ./scripts/run-agent-smoke-tests.mjs --scenario=fixture-full --agents=hermes --command=open-computer-use --hermes-provider=anthropic --hermes-model=claude-opus-4-20250514 --json --timeout-ms=180000`+- `swift test`+- `make check-docs`+- `OPEN_COMPUTER_USE_STRESS_LOOPS=20 OPEN_COMPUTER_USE_STRESS_CONFIGURATION=release ./scripts/run-tool-stress-tests.sh`++The latest 20-loop stress run completed 200 fixture operations plus cursor idle smoke in 76 seconds. Claude, Codex, and Hermes validated the read-only `list_apps` scenario. Claude, Codex, and Hermes also validated the fixture and full fixture scenarios with independent final-state checks. Hermes required explicit provider/model routing because this local Hermes default model alias returned HTTP 404 before tool use; the full fixture scenario also needed a 12-turn Hermes budget to complete all requested tool calls.diff --git a/scripts/run-agent-smoke-tests.mjs b/scripts/run-agent-smoke-tests.mjsnew file mode 100755index 0000000..510afe3--- /dev/null+++ b/scripts/run-agent-smoke-tests.mjs@@ -0,0 +1,389 @@+#!/usr/bin/env node+import { existsSync, rmSync } from "node:fs";+import { spawn, spawnSync } from "node:child_process";++const args = new Map();+for (const arg of process.argv.slice(2)) {+ const [key, ...rest] = arg.split("=");+ if (key.startsWith("--")) args.set(key.slice(2), rest.length ? rest.join("=") : "true");+}++const selectedAgents = new Set((args.get("agents") ?? process.env.OPEN_COMPUTER_USE_AGENT_AGENTS ?? "claude,codex")+ .split(",")+ .map((agent) => agent.trim())+ .filter(Boolean));+const json = args.get("json") === "true";+const command = args.get("command") ?? process.env.OPEN_COMPUTER_USE_AGENT_COMMAND ?? "open-computer-use";+const timeoutMs = Number(args.get("timeout-ms") ?? process.env.OPEN_COMPUTER_USE_AGENT_TIMEOUT_MS ?? 120000);+const maxClaudeBudget = args.get("claude-budget-usd") ?? process.env.OPEN_COMPUTER_USE_CLAUDE_BUDGET_USD ?? "2.00";+const hermesProvider = args.get("hermes-provider") ?? process.env.OPEN_COMPUTER_USE_HERMES_PROVIDER;+const hermesModel = args.get("hermes-model") ?? process.env.OPEN_COMPUTER_USE_HERMES_MODEL;+const hermesMaxTurns = args.get("hermes-max-turns") ?? process.env.OPEN_COMPUTER_USE_HERMES_MAX_TURNS ?? "12";+const scenario = args.get("scenario") ?? "list-apps";++if (!["list-apps", "fixture", "fixture-full"].includes(scenario)) {+ console.error(`Unsupported --scenario=${scenario}. Use list-apps, fixture, or fixture-full.`);+ process.exit(2);+}++function usesFixture() {+ return scenario === "fixture" || scenario === "fixture-full";+}++function runProcess(name, spec) {+ return new Promise((resolve) => {+ const child = spawn(spec.command, spec.args, {+ cwd: process.cwd(),+ stdio: ["ignore", "pipe", "pipe"],+ env: {+ ...process.env,+ OPEN_COMPUTER_USE_VISUAL_CURSOR: "0",+ },+ });++ let stdout = "";+ let stderr = "";+ let timedOut = false;+ const timer = setTimeout(() => {+ timedOut = true;+ child.kill("SIGTERM");+ setTimeout(() => child.kill("SIGKILL"), 3000).unref();+ }, timeoutMs);++ child.stdout.on("data", (chunk) => stdout += chunk);+ child.stderr.on("data", (chunk) => stderr += chunk);+ child.on("error", (error) => {+ clearTimeout(timer);+ resolve({ agent: name, ok: false, code: null, timedOut, stdout, stderr: String(error.message ?? error) });+ });+ child.on("close", (code) => {+ clearTimeout(timer);+ resolve({ agent: name, ok: code === 0 && !timedOut, code, timedOut, stdout: stdout.trim(), stderr: stderr.trim() });+ });+ });+}++function codexSawToolCalls(stdout, expectedTools) {+ const seen = new Set();+ const expected = new Set(expectedTools);+ for (const line of stdout.split(/\n+/).filter(Boolean)) {+ try {+ const event = JSON.parse(line);+ if (event.type === "item.completed"+ && event.item?.type === "mcp_tool_call"+ && event.item?.server === "open-computer-use"+ && event.item?.status === "completed"+ && expected.has(event.item?.tool)) {+ seen.add(event.item.tool);+ }+ } catch {}+ }+ return [...expected].every((tool) => seen.has(tool));+}++function fixtureExecutablePath() {+ for (const candidate of [".build/release/OpenComputerUseFixture", ".build/debug/OpenComputerUseFixture"]) {+ if (existsSync(candidate)) return candidate;+ }++ const build = spawnSync("swift", ["build", "-c", "release", "--product", "OpenComputerUseFixture"], {+ cwd: process.cwd(),+ stdio: "inherit",+ });+ if (build.status !== 0) {+ throw new Error("Failed to build OpenComputerUseFixture.");+ }+ if (existsSync(".build/release/OpenComputerUseFixture")) return ".build/release/OpenComputerUseFixture";+ throw new Error("OpenComputerUseFixture executable was not found after build.");+}++function resetFixtureState() {+ rmSync(`${process.env.TMPDIR ?? "/tmp"}/open-computer-use-fixture`, { recursive: true, force: true });+}++function launchFixture() {+ resetFixtureState();+ const executable = fixtureExecutablePath();+ const child = spawn(executable, [], {+ cwd: process.cwd(),+ stdio: ["ignore", "ignore", "pipe"],+ env: {+ ...process.env,+ OPEN_COMPUTER_USE_FIXTURE_HEADLESS: "1",+ },+ });+ child.fixtureStderr = "";+ child.fixtureExitCode = null;+ child.stderr.on("data", (chunk) => child.fixtureStderr += chunk);+ child.on("close", (code) => child.fixtureExitCode = code);+ return child;+}++function callOpenComputerUse(tool, toolArgs = {}) {+ return new Promise((resolve, reject) => {+ const child = spawn(command, ["call", tool, "--args", JSON.stringify(toolArgs)], {+ cwd: process.cwd(),+ stdio: ["ignore", "pipe", "pipe"],+ env: {+ ...process.env,+ OPEN_COMPUTER_USE_VISUAL_CURSOR: "0",+ },+ });+ let stdout = "";+ let stderr = "";+ child.stdout.on("data", (chunk) => stdout += chunk);+ child.stderr.on("data", (chunk) => stderr += chunk);+ child.on("error", reject);+ child.on("close", (code) => {+ if (code !== 0) {+ const details = [+ `${command} call ${tool} exited ${code}`,+ stderr.trim() ? `stderr:\n${stderr.trim()}` : "",+ stdout.trim() ? `stdout:\n${stdout.trim()}` : "",+ ].filter(Boolean).join("\n");+ reject(new Error(details.slice(0, 2000)));+ return;+ }+ try {+ resolve(JSON.parse(stdout));+ } catch (error) {+ reject(new Error(`Failed to parse ${command} call ${tool} output: ${error.message}`));+ }+ });+ });+}++function textFromToolResult(result) {+ return result?.content?.find((item) => item.type === "text")?.text ?? "";+}++function lineHasExactValue(text, id, value) {+ return text.split("\n").some((line) =>+ line.includes(`ID: ${id} `) && line.includes(` Value: ${value} Frame:`)+ );+}++async function waitForFixtureReady(fixture) {+ const deadline = Date.now() + 10000;+ let lastError;+ while (Date.now() < deadline) {+ if (fixture?.fixtureExitCode !== null) {+ throw new Error([+ `OpenComputerUseFixture exited before readiness with code ${fixture.fixtureExitCode}.`,+ fixture.fixtureStderr?.trim() ? `fixture stderr:\n${fixture.fixtureStderr.trim()}` : "",+ ].filter(Boolean).join("\n"));+ }+ try {+ const result = await callOpenComputerUse("get_app_state", { app: "OpenComputerUseFixture" });+ const text = textFromToolResult(result);+ if (text.includes("fixture-input") && text.includes("fixture-increment")) return text;+ } catch (error) {+ lastError = error;+ }+ await new Promise((resolve) => setTimeout(resolve, 250));+ }+ throw new Error([+ "OpenComputerUseFixture did not become visible to open-computer-use.",+ lastError ? `last error:\n${lastError.message ?? lastError}` : "",+ fixture?.fixtureStderr?.trim() ? `fixture stderr:\n${fixture.fixtureStderr.trim()}` : "",+ ].filter(Boolean).join("\n"));+}++async function validateFixtureState(expectedValue) {+ const result = await callOpenComputerUse("get_app_state", { app: "OpenComputerUseFixture" });+ const text = textFromToolResult(result);+ const expectedTextValue = scenario === "fixture-full" ? `${expectedValue}-typed` : expectedValue;+ return {+ ok: lineHasExactValue(text, "fixture-input", expectedTextValue)+ && lineHasExactValue(text, "fixture-counter-label", "Counter: 1")+ && (scenario !== "fixture-full" || (+ lineHasExactValue(text, "fixture-key-label", "Last key: Return")+ && !lineHasExactValue(text, "fixture-scroll-status", "Scroll offset: 0")+ && !lineHasExactValue(text, "fixture-drag-status", "Last drag: none")+ )),+ textTail: text.slice(-1200),+ };+}++function scenarioExpectedTools() {+ if (scenario === "fixture") return ["get_app_state", "set_value", "click"];+ if (scenario === "fixture-full") {+ return [+ "get_app_state",+ "set_value",+ "click",+ "type_text",+ "press_key",+ "perform_secondary_action",+ "scroll",+ "drag",+ ];+ }+ return ["list_apps"];+}++function scenarioAllowedTools() {+ return scenarioExpectedTools().map((tool) => `mcp__open-computer-use__${tool}`).join(",");+}++function makePrompt(expectedValue) {+ if (scenario === "list-apps") {+ return [+ "Use the open-computer-use MCP list_apps tool exactly once before answering.",+ "Do not use terminal, shell, browser, file, or any other tool.",+ "After the tool call, reply exactly OPEN_CU_AGENT_OK."+ ].join(" ");+ }++ return [+ "Use only open-computer-use MCP tools.",+ "Call get_app_state for app OpenComputerUseFixture.",+ ...(scenario === "fixture-full" ? [+ `Find the text field and use set_value to set it exactly to ${JSON.stringify(expectedValue)}.`,+ "Click the text field by element index, then use type_text to append exactly -typed.",+ "Click the Key Capture area by element index, then use press_key with key Return.",+ "Call perform_secondary_action on the window element with action Raise.",+ "Scroll the scroll area down by 1 page.",+ "Drag inside the Drag Pad from near its upper-left interior to near its lower-right interior.",+ "Click the Increment Counter button exactly once by element index.",+ ] : [+ `Find the text field and use set_value to set it exactly to ${JSON.stringify(expectedValue)}.`,+ "Find the Increment Counter button and click it exactly once by element index.",+ ]),+ "Do not use terminal, shell, browser, file, or any other tool.",+ "After the tool calls, reply exactly OPEN_CU_AGENT_OK."+ ].join(" ");+}++function makeSpecs(prompt) {+ return {+ claude: {+ command: "claude",+ args: [+ "-p",+ "--strict-mcp-config",+ "--mcp-config", JSON.stringify({+ mcpServers: {+ "open-computer-use": {+ type: "stdio",+ command,+ args: ["mcp"],+ env: { OPEN_COMPUTER_USE_VISUAL_CURSOR: "0" },+ },+ },+ }),+ "--permission-mode", "bypassPermissions",+ "--allowedTools", scenarioAllowedTools(),+ "--output-format", "text",+ "--max-budget-usd", maxClaudeBudget,+ prompt,+ ],+ validate: (result) => result.stdout.includes("OPEN_CU_AGENT_OK"),+ },+ codex: {+ command: "codex",+ args: [+ "exec",+ "--ignore-user-config",+ "--skip-git-repo-check",+ "--dangerously-bypass-approvals-and-sandbox",+ "--disable", "image_generation",+ "--disable", "js_repl",+ "--json",+ "-c", `mcp_servers.open-computer-use.command=${JSON.stringify(command)}`,+ "-c", 'mcp_servers.open-computer-use.args=["mcp"]',+ prompt,+ ],+ validate: (result) => result.stdout.includes("OPEN_CU_AGENT_OK") && codexSawToolCalls(result.stdout, scenarioExpectedTools()),+ },+ hermes: {+ command: "hermes",+ args: [+ "chat",+ "--query", prompt,+ "--quiet",+ "--yolo",+ "--max-turns", hermesMaxTurns,+ "--toolsets", "open-computer-use",+ ...(hermesProvider ? ["--provider", hermesProvider] : []),+ ...(hermesModel ? ["--model", hermesModel] : []),+ ],+ validate: (result) => result.stdout.includes("OPEN_CU_AGENT_OK"),+ },+ };+}++const results = [];+const expectedTools = scenarioExpectedTools();+for (const name of Object.keys(makeSpecs(""))) {+ if (!selectedAgents.has(name)) continue;+ const expectedValue = `agent-smoke-${name}-${Date.now()}`;+ const prompt = makePrompt(expectedValue);+ const spec = makeSpecs(prompt)[name];+ let fixture = null;+ let fixtureValidation = null;+ if (usesFixture()) {+ fixture = launchFixture();+ try {+ await waitForFixtureReady(fixture);+ } catch (error) {+ results.push({+ agent: name,+ processOk: false,+ code: null,+ timedOut: false,+ validated: false,+ expectedTools,+ fixtureValidation: { ok: false, error: String(error?.message ?? error) },+ stdoutTail: "",+ stderrTail: "",+ });+ fixture.kill("SIGTERM");+ continue;+ }+ }++ const result = await runProcess(name, spec);+ if (usesFixture()) {+ fixtureValidation = await validateFixtureState(expectedValue).catch((error) => ({+ ok: false,+ error: String(error?.message ?? error),+ }));+ fixture?.kill("SIGTERM");+ }+ const validated = result.ok+ && spec.validate(result)+ && (!usesFixture() || fixtureValidation?.ok === true);+ results.push({+ agent: name,+ processOk: result.ok,+ code: result.code,+ timedOut: result.timedOut,+ validated,+ expectedTools,+ fixtureValidation,+ stdoutTail: result.stdout.slice(-1200),+ stderrTail: result.stderr.slice(-1200),+ });+}++const report = {+ ok: results.length > 0 && results.every((result) => result.validated),+ command,+ results,+};++if (json) {+ console.log(JSON.stringify(report, null, 2));+} else {+ console.log(`agent smoke: ${report.ok ? "ok" : "failed"}`);+ for (const result of results) {+ console.log(`- ${result.agent}: processOk=${result.processOk} validated=${result.validated} code=${result.code}`);+ if (!result.validated) {+ if (result.stdoutTail) console.log(` stdout: ${result.stdoutTail.replace(/\n/g, " ").slice(0, 500)}`);+ if (result.stderrTail) console.log(` stderr: ${result.stderrTail.replace(/\n/g, " ").slice(0, 500)}`);+ }+ }+}++process.exit(report.ok ? 0 : 1);diff --git a/scripts/run-tool-stress-tests.sh b/scripts/run-tool-stress-tests.shnew file mode 100755index 0000000..4c7b7ed--- /dev/null+++ b/scripts/run-tool-stress-tests.sh@@ -0,0 +1,61 @@+#!/usr/bin/env bash++set -euo pipefail++repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+loops="${OPEN_COMPUTER_USE_STRESS_LOOPS:-20}"+include_cursor_idle="${OPEN_COMPUTER_USE_STRESS_CURSOR_IDLE:-1}"+configuration="${OPEN_COMPUTER_USE_STRESS_CONFIGURATION:-debug}"++case "${configuration}" in+ debug)+ build_args=()+ products_dir=".build/debug"+ ;;+ release)+ build_args=(-c release)+ products_dir=".build/release"+ ;;+ *)+ echo "Unsupported OPEN_COMPUTER_USE_STRESS_CONFIGURATION: ${configuration}" >&2+ exit 2+ ;;+esac++if ! [[ "${loops}" =~ ^[0-9]+$ ]] || [[ "${loops}" -lt 1 ]]; then+ echo "OPEN_COMPUTER_USE_STRESS_LOOPS must be a positive integer." >&2+ exit 2+fi++cd "${repo_root}"++swift build "${build_args[@]}"++started_at="$(date +%s)"+tmp_dir="$(mktemp -d "${TMPDIR:-/tmp}/open-computer-use-stress.XXXXXX")"+trap 'rm -rf "${tmp_dir}"' EXIT++for loop in $(seq 1 "${loops}"); do+ printf 'OpenComputerUseSmokeSuite stress loop %s/%s\n' "${loop}" "${loops}"+ OPEN_COMPUTER_USE_VISUAL_CURSOR=0 "${products_dir}/OpenComputerUseSmokeSuite" >"${tmp_dir}/loop-${loop}.log"+ tail -n 1 "${tmp_dir}/loop-${loop}.log"+done++cursor_idle_ran=0+if [[ "${include_cursor_idle}" == "1" || "${include_cursor_idle}" == "true" || "${include_cursor_idle}" == "yes" ]]; then+ "${products_dir}/OpenComputerUseSmokeSuite" --cursor-idle-only >"${tmp_dir}/cursor-idle.log"+ tail -n 1 "${tmp_dir}/cursor-idle.log"+ cursor_idle_ran=1+fi++ended_at="$(date +%s)"+cat <<JSON+{+ "ok": true,+ "configuration": "${configuration}",+ "loops": ${loops},+ "fixtureOperations": $((loops * 10)),+ "cursorIdleSmoke": ${cursor_idle_ran},+ "durationSeconds": $((ended_at - started_at))+}+JSON