Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Add orchestration MCP w/ support for demonstration prompting & processing (#892)
ddupont1cbeb71a6cbd
diff --git a/docs/content/docs/cua/examples/claude-code/human-demonstrations.mdx b/docs/content/docs/cua/examples/claude-code/human-demonstrations.mdxnew file mode 100644index 0000000000..3f64663230--- /dev/null+++ b/docs/content/docs/cua/examples/claude-code/human-demonstrations.mdx@@ -0,0 +1,106 @@+---+title: Prompting Claude Code with Human Demonstrations+description: Record human workflows and use them to guide Claude Code agents+---++import { Step, Steps } from 'fumadocs-ui/components/steps';+import { Callout } from 'fumadocs-ui/components/callout';++## Overview++Human demonstrations let you record a workflow once and replay it with AI assistance. This guide shows how to connect the CUA MCP server to Claude Code, record a demonstration, and use it to guide automated agents.++---++## Add the CUA MCP Server++Connect Claude Code to your CUA sandboxes:++```bash+claude mcp add cua-mcp -- cua serve-mcp+```++Verify the connection:++```bash+claude mcp list+```++<Callout type="info">+ Run `cua auth login` first if you haven't authenticated with your CUA API key.+</Callout>++---++## Prompts to Try++Once connected, try these prompts in sequence:++<Steps>++<Step>++### Create a Sandbox++```+Create a Linux sandbox in north-america with size medium+```++Claude Code uses the `create_sandbox` tool to provision a new cloud sandbox.++</Step>++<Step>++### Record a Demonstration++```+Record a demonstration on my-sandbox called "login-flow" that shows how to log into the admin panel+```++Claude Code opens the VNC interface with recording enabled. Perform your workflow, then click **Stop Recording**. The recording is processed and saved as a skill.++</Step>++<Step>++### Replay the Demonstration++```+Read the login-flow skill and replay those steps on my-sandbox+```++Claude Code reads the skill from `~/.cua/skills/login-flow/SKILL.md` and executes each step using the computer control tools (`computer_click`, `computer_type`, etc.).++</Step>++</Steps>++---++## Managing Skills++```bash+# List all skills+cua skills list++# Read a skill's full content+cua skills read my-skill++# Read as JSON (includes trajectory data)+cua skills read my-skill --format json++# Replay the recorded video+cua skills replay my-skill++# Delete a skill+cua skills delete my-skill+```++---++## Next Steps++- Learn about [Agent Loops](/cua/guide/fundamentals/agent-loops) for understanding how agents process tasks+- See [Trajectories](/cua/guide/advanced/trajectories) for debugging and analyzing agent behavior+- Explore [Custom Tools](/cua/guide/advanced/custom-tools) to extend agent capabilitiesdiff --git a/docs/content/docs/cua/examples/claude-code/meta.json b/docs/content/docs/cua/examples/claude-code/meta.jsonnew file mode 100644index 0000000000..c0e2d2d959--- /dev/null+++ b/docs/content/docs/cua/examples/claude-code/meta.json@@ -0,0 +1,5 @@+{+ "title": "Claude Code",+ "description": "Claude Code integration examples",+ "pages": ["human-demonstrations"]+}diff --git a/docs/content/docs/cua/examples/index.mdx b/docs/content/docs/cua/examples/index.mdxindex 32178856f6..bf56b7338d 100644--- a/docs/content/docs/cua/examples/index.mdx+++ b/docs/content/docs/cua/examples/index.mdx@@ -4,13 +4,19 @@ description: Real-world examples of building with Cua---import { Card, Cards } from 'fumadocs-ui/components/card';-import { Workflow, Laptop, Brain } from 'lucide-react';+import { Workflow, Laptop, Brain, Terminal } from 'lucide-react';# ExamplesExplore real-world examples of building computer-use agents with Cua. These examples demonstrate various use cases across different platforms and AI models.<Cards>+ <Card+ icon={<Terminal className="w-6 h-6" />}+ href="/cua/examples/claude-code/human-demonstrations"+ title="Claude Code"+ description="Use human demonstrations to guide Claude Code agents"+ /><Cardicon={<Workflow className="w-6 h-6" />}href="/cua/examples/automation/form-filling"@@ -33,6 +39,10 @@ Explore real-world examples of building computer-use agents with Cua. These exam## Featured Examples+### Claude Code++- **[Human Demonstrations](/cua/examples/claude-code/human-demonstrations)** - Record workflows and use them to guide Claude Code agents+### Automation- **[PDF to Form Automation](/cua/examples/automation/form-filling)** - Extract data from PDFs and automatically fill web formsdiff --git a/docs/content/docs/cua/guide/advanced/demonstration-guided-skills.mdx b/docs/content/docs/cua/guide/advanced/demonstration-guided-skills.mdxnew file mode 100644index 0000000000..7e3386fb17--- /dev/null+++ b/docs/content/docs/cua/guide/advanced/demonstration-guided-skills.mdx@@ -0,0 +1,115 @@+---+title: Demonstration-Guided Skills+description: Use human demonstrations to guide agent behavior and improve task completion+---++Human demonstrations provide powerful context for guiding agent behavior. By showing an agent how a task is performed once, it can learn the workflow pattern and generalize to similar tasks with different inputs.++## Overview++The demonstration-guided workflow consists of three stages:++1. **Record** - Capture a human performing the task via [VNC recording](/docs/cua/guide/advanced/vnc-recorder)+2. **Process** - Convert the recording into semantic action traces with VLM captions+3. **Prompt** - Use the processed trajectory to guide agent execution++This approach enables:++- **One-shot learning**: Teach a workflow once, execute it many times+- **Reduced errors**: Agents follow demonstrated patterns instead of discovering them+- **Complex workflows**: Handle multi-step tasks that require specific sequences++## Example: Using a Demonstration to Guide an Agent++### 1. Record a demonstration++```bash+cua skills record --sandbox my-sandbox+```++When prompted, name it `flight-booking` and describe it as "Book a flight from NYC to LA".++### 2. View the saved skill++The skill is saved to `~/.cua/skills/flight-booking/SKILL.md`:++```bash+cua skills read flight-booking+```++### 3. Use the skill to guide an agent++Read the SKILL.md file and prefix it to your prompt:++```python+from pathlib import Path+from agent import ComputerAgent++# Load the skill prompt from SKILL.md+skill_path = Path.home() / ".cua/skills/flight-booking/SKILL.md"+skill_prompt = skill_path.read_text()++agent = ComputerAgent(loop="anthropic")++# Prefix the skill as context, then give the actual task+messages = [+ {"role": "user", "content": skill_prompt},+ {"role": "user", "content": "Book a flight from Boston to Chicago for next Friday"}+]++async for result in agent.run(messages):+ print(result)+```++The skill contains a natural language description of the demonstrated workflow, allowing the agent to follow the same pattern while adapting to the new parameters (Boston→Chicago instead of NYC→LA).++## Skill Format++Each skill is saved to `~/.cua/skills/<name>/` with the following structure:++```+~/.cua/skills/flight-booking/+├── SKILL.md # Skill description and agent prompt+└── trajectory/+ ├── trajectory.json # Raw events and captions+ ├── events.json # Input events+ ├── flight-booking.mp4 # Video recording+ ├── step_1_full.jpg # Full screenshot for step 1+ ├── step_1_crop.jpg # Cropped screenshot with marker+ └── ...+```++The `SKILL.md` file contains the skill metadata, step descriptions, and agent prompt:++```markdown+---+name: flight-booking+description: Book a flight from NYC to LA+---++# flight-booking++Book a flight from NYC to LA++## Steps++### Step 1: Click on the departure city field++**Context:** A flight booking form with empty departure and arrival fields...+**Intent:** User wants to enter the departure city+**Expected Result:** Field becomes focused for text input++### Step 2: Type "NYC"++...++## Agent Prompt++You have been shown a demonstration of how to perform this task:+Book a flight from NYC to LA++The demonstration consisted of the following steps:+Step 1: Click on the departure city field+Step 2: Type "NYC"+...+```diff --git a/docs/content/docs/cua/guide/advanced/meta.json b/docs/content/docs/cua/guide/advanced/meta.jsonindex 5638a78b8c..ba5dd84457 100644--- a/docs/content/docs/cua/guide/advanced/meta.json+++ b/docs/content/docs/cua/guide/advanced/meta.json@@ -8,7 +8,8 @@"sandboxed-python","local-computer-server","human-in-the-loop",- "demonstration-recorder",+ "vnc-recorder",+ "demonstration-guided-skills","trajectories","tracing","chat-history",diff --git a/docs/content/docs/cua/guide/advanced/demonstration-recorder.mdx b/docs/content/docs/cua/guide/advanced/vnc-recorder.mdxsimilarity index 65%rename from docs/content/docs/cua/guide/advanced/demonstration-recorder.mdxrename to docs/content/docs/cua/guide/advanced/vnc-recorder.mdxindex 8c2c58577b..f977c710dc 100644--- a/docs/content/docs/cua/guide/advanced/demonstration-recorder.mdx+++ b/docs/content/docs/cua/guide/advanced/vnc-recorder.mdx@@ -1,5 +1,5 @@----title: Human Demonstration Recorder+title: VNC Recorderdescription: Record human demonstrations from CUA sandboxes for training and evaluation---@@ -9,6 +9,54 @@ CUA sandboxes include a [noVNC fork](https://github.com/trycua/novnc) with built<source src="/docs/videos/demonstration-recorder.mp4" type="video/mp4" /></video>+## Recording with the CUA CLI++The easiest way to record demonstrations is with the `cua skills` command.++### 1. Start recording++Connect to a sandbox and start recording:++```bash+# Record from a CUA Cloud sandbox+cua skills record --sandbox my-sandbox++# Or record from any VNC URL+cua skills record --vnc-url http://localhost:8006/vnc.html+```++This opens the VNC interface with recording enabled. You'll be prompted for a skill name and description after recording.++### 2. Perform your demonstration++Interact with the sandbox normally. All mouse and keyboard input is captured along with video.++### 3. Stop and save++Click **Stop Recording** in the VNC panel. The CLI processes the recording, generates captions using a VLM, and saves the skill to `~/.cua/skills/<name>/`.++### 4. Replay++View the recorded video:++```bash+cua skills replay my-skill+```++List all saved skills:++```bash+cua skills list+```++Read a skill's prompt:++```bash+cua skills read my-skill+```++---+## Manual Recording### 1. Start a sandbox and open the VNC UI@@ -60,7 +108,9 @@ http://localhost:6080/tests/vnc_playback.html?data=../recordings/your-recording.## Streaming to External Server-For automated pipelines, stream recordings directly to a custom server using the `record_url` parameter. Here's a simple recording server:+For automated pipelines, stream recordings directly to a custom server using the `record_url` parameter. The recording data is streamed as a websockify stream over WebSocket.++Here's a simple recording server:```pythonimport asyncio@@ -95,8 +145,10 @@ Recording data streams to disk in real-time via WebSocket, so the file is ready| `autorecord=1` | Start recording when connection is established || `record_url=ws://...` | Stream recording to external WebSocket server |-## Using Demonstrations to Prompt Agents+For more details on prompting agents with demonstrations, see [Demonstration-Guided Skills](/docs/cua/guide/advanced/demonstration-guided-skills).++## Reference-Recordings are raw noVNC data streams, but they can be processed into semantic action traces that improve agent planning and execution. An example of an agent that will support this capability is [ShowUI-Aloha](https://github.com/showlab/ShowUI-Aloha), which learns workflows from human demonstrations and generalizes to new task variants.+For related capabilities, see:-_Coming soon_+- [ShowUI-Aloha](https://github.com/showlab/ShowUI-Aloha) - Human-taught GUI agent that learns from demonstrationsdiff --git a/docs/content/docs/cua/guide/get-started/using-cloud-cli.mdx b/docs/content/docs/cua/guide/get-started/using-cloud-cli.mdxindex c192fcb2f5..aaad8ae1f9 100644--- a/docs/content/docs/cua/guide/get-started/using-cloud-cli.mdx+++ b/docs/content/docs/cua/guide/get-started/using-cloud-cli.mdx@@ -10,19 +10,13 @@ The Cua Cloud CLI lets you create, manage, and connect to cloud sandboxes direct## Installation-Install the CLI globally using npm or bun:--<Tabs items={['npm', 'bun']}>- <Tab value="npm">- ```bash- npm install -g @trycua/cli- ```- </Tab>- <Tab value="bun">- ```bash- bun install -g @trycua/cli- ```+<Tabs items={['macOS/Linux', 'Windows', 'bun']}>+ <Tab value="macOS/Linux">```bash curl -LsSf https://cua.ai/cli/install.sh | sh ```</Tab>+ <Tab value="Windows">+ ```powershell powershell -ExecutionPolicy ByPass -c "irm https://cua.ai/cli/install.ps1 | iex"+ ```</Tab>+ <Tab value="bun">```bash bun install -g @trycua/cli ```</Tab></Tabs>Verify the installation:diff --git a/libs/typescript/cua-cli/bun.lock b/libs/typescript/cua-cli/bun.lockindex b91c382e18..d0233508b1 100644--- a/libs/typescript/cua-cli/bun.lock+++ b/libs/typescript/cua-cli/bun.lock@@ -1,11 +1,12 @@{"lockfileVersion": 1,- "configVersion": 0,"workspaces": {"": {"name": "cua-cli","dependencies": {+ "@modelcontextprotocol/sdk": "^1.0.0","yargs": "^18.0.0",+ "zod": "^3.23.0",},"devDependencies": {"@types/bun": "latest",@@ -17,6 +18,10 @@},},"packages": {+ "@hono/node-server": ["@hono/[email protected]", "", { "peerDependencies": { "hono": "^4" } }, "sha512-vHL6w3ecZsky+8P5MD+eFfaGTyCeOHUIFYMGpQGbrBTSmNNoxv0if69rEZ5giu36weC5saFuznL411gRX7bJDw=="],++ "@modelcontextprotocol/sdk": ["@modelcontextprotocol/[email protected]", "", { "dependencies": { "@hono/node-server": "^1.19.9", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "content-type": "^1.0.5", "cors": "^2.8.5", "cross-spawn": "^7.0.5", "eventsource": "^3.0.2", "eventsource-parser": "^3.0.0", "express": "^5.0.1", "express-rate-limit": "^7.5.0", "jose": "^6.1.1", "json-schema-typed": "^8.0.2", "pkce-challenge": "^5.0.0", "raw-body": "^3.0.0", "zod": "^3.25 || ^4.0", "zod-to-json-schema": "^3.25.0" }, "peerDependencies": { "@cfworker/json-schema": "^4.1.1" }, "optionalPeers": ["@cfworker/json-schema"] }, "sha512-vsAMBMERybvYgKbg/l4L1rhS7VXV1c0CtyJg72vwxONVX0l4ZfKVAnZEWTQixJGTzKnELjQ59e4NbdFDALRiAQ=="],+"@types/bun": ["@types/[email protected]", "", { "dependencies": { "bun-types": "1.3.0" } }, "sha512-+lAGCYjXjip2qY375xX/scJeVRmZ5cY0wyHYyCYxNcdEXrQ4AOe3gACgd4iQ8ksOslJtW4VNxBJ8llUwc3a6AA=="],"@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-MKNwXh3seSK8WurXF7erHPJ2AONmMwkI7zAMrXZDPIru8jRqkk6rGDBVbw4mLwfqA+ZZliiDPg05JQ3uW66tKQ=="],@@ -27,6 +32,12 @@"@types/yargs-parser": ["@types/[email protected]", "", {}, "sha512-I4q9QU9MQv4oEOz4tAHJtNz1cwuLxn2F3xcc2iV5WdqLPpUnj30aUuxt1mAxYTG+oe8CZMV/+6rU4S4gRDzqtQ=="],+ "accepts": ["[email protected]", "", { "dependencies": { "mime-types": "^3.0.0", "negotiator": "^1.0.0" } }, "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng=="],++ "ajv": ["[email protected]", "", { "dependencies": { "fast-deep-equal": "^3.1.3", "fast-uri": "^3.0.1", "json-schema-traverse": "^1.0.0", "require-from-string": "^2.0.2" } }, "sha512-B/gBuNg5SiMTrPkC+A2+cW0RszwxYmn6VYxB/inlBStS5nx6xHIt/ehKRhIMhqusl7a8LjQoZnjCs5vhwxOQ1g=="],++ "ajv-formats": ["[email protected]", "", { "dependencies": { "ajv": "^8.0.0" } }, "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ=="],+"ansi-regex": ["[email protected]", "", {}, "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg=="],"ansi-styles": ["[email protected]", "", {}, "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg=="],@@ -35,14 +46,22 @@"args-tokenizer": ["[email protected]", "", {}, "sha512-xXAd7G2Mll5W8uo37GETpQ2VrE84M181Z7ugHFGQnJZ50M2mbOv0osSZ9VsSgPfJQ+LVG0prSi0th+ELMsno7Q=="],+ "body-parser": ["[email protected]", "", { "dependencies": { "bytes": "^3.1.2", "content-type": "^1.0.5", "debug": "^4.4.3", "http-errors": "^2.0.0", "iconv-lite": "^0.7.0", "on-finished": "^2.4.1", "qs": "^6.14.1", "raw-body": "^3.0.1", "type-is": "^2.0.1" } }, "sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA=="],+"bumpp": ["[email protected]", "", { "dependencies": { "ansis": "^4.2.0", "args-tokenizer": "^0.3.0", "c12": "^3.3.0", "cac": "^6.7.14", "escalade": "^3.2.0", "jsonc-parser": "^3.3.1", "package-manager-detector": "^1.3.0", "semver": "^7.7.2", "tinyexec": "^1.0.1", "tinyglobby": "^0.2.15", "yaml": "^2.8.1" }, "bin": { "bumpp": "bin/bumpp.mjs" } }, "sha512-cOKPRFCWvHcYPJQAHN6V7Jp/wAfnyqQRXQ+2fgWIL6Gao20rpu7xQ1cGGo1APOfmbQmmHngEPg9Fy7nJ3giRkQ=="],"bun-types": ["[email protected]", "", { "dependencies": { "@types/node": "*" }, "peerDependencies": { "@types/react": "^19" } }, "sha512-u8X0thhx+yJ0KmkxuEo9HAtdfgCBaM/aI9K90VQcQioAmkVp3SG3FkwWGibUFz3WdXAdcsqOcbU40lK7tbHdkQ=="],+ "bytes": ["[email protected]", "", {}, "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="],+"c12": ["[email protected]", "", { "dependencies": { "chokidar": "^4.0.3", "confbox": "^0.2.2", "defu": "^6.1.4", "dotenv": "^17.2.3", "exsolve": "^1.0.8", "giget": "^2.0.0", "jiti": "^2.6.1", "ohash": "^2.0.11", "pathe": "^2.0.3", "perfect-debounce": "^2.0.0", "pkg-types": "^2.3.0", "rc9": "^2.1.2" }, "peerDependencies": { "magicast": "*" }, "optionalPeers": ["magicast"] }, "sha512-QkikB2X5voO1okL3QsES0N690Sn/K9WokXqUsDQsWy5SnYb+psYQFGA10iy1bZHj3fjISKsI67Q90gruvWWM3A=="],"cac": ["[email protected]", "", {}, "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ=="],+ "call-bind-apply-helpers": ["[email protected]", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="],++ "call-bound": ["[email protected]", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "get-intrinsic": "^1.3.0" } }, "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg=="],+"chokidar": ["[email protected]", "", { "dependencies": { "readdirp": "^4.0.1" } }, "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA=="],"citty": ["[email protected]", "", { "dependencies": { "consola": "^3.2.3" } }, "sha512-tskPPKEs8D2KPafUypv2gxwJP8h/OaJmC82QQGGDQcHvXX43xF2VDACcJVmZ0EuSxkpO9Kc4MlrA3q0+FG58AQ=="],@@ -53,56 +72,202 @@"consola": ["[email protected]", "", {}, "sha512-5IKcdX0nnYavi6G7TtOhwkYzyjfJlatbjMjuLSfE2kYT5pMDOilZ4OvMhi637CcDICTmz3wARPoyhqyX1Y+XvA=="],+ "content-disposition": ["[email protected]", "", {}, "sha512-oIXISMynqSqm241k6kcQ5UwttDILMK4BiurCfGEREw6+X9jkkpEe5T9FZaApyLGGOnFuyMWZpdolTXMtvEJ08Q=="],++ "content-type": ["[email protected]", "", {}, "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA=="],++ "cookie": ["[email protected]", "", {}, "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w=="],++ "cookie-signature": ["[email protected]", "", {}, "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg=="],++ "cors": ["[email protected]", "", { "dependencies": { "object-assign": "^4", "vary": "^1" } }, "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw=="],++ "cross-spawn": ["[email protected]", "", { "dependencies": { "path-key": "^3.1.0", "shebang-command": "^2.0.0", "which": "^2.0.1" } }, "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA=="],+"csstype": ["[email protected]", "", {}, "sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw=="],+ "debug": ["[email protected]", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="],+"defu": ["[email protected]", "", {}, "sha512-mEQCMmwJu317oSz8CwdIOdwf3xMif1ttiM8LTufzc3g6kR+9Pe236twL8j3IYT1F7GfRgGcW6MWxzZjLIkuHIg=="],+ "depd": ["[email protected]", "", {}, "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw=="],+"destr": ["[email protected]", "", {}, "sha512-ugFTXCtDZunbzasqBxrK93Ik/DRYsO6S/fedkWEMKqt04xZ4csmnmwGDBAb07QWNaGMAmnTIemsYZCksjATwsA=="],"dotenv": ["[email protected]", "", {}, "sha512-JVUnt+DUIzu87TABbhPmNfVdBDt18BLOWjMUFJMSi/Qqg7NTYtabbvSNJGOJ7afbRuv9D/lngizHtP7QyLQ+9w=="],+ "dunder-proto": ["[email protected]", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.1", "es-errors": "^1.3.0", "gopd": "^1.2.0" } }, "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A=="],++ "ee-first": ["[email protected]", "", {}, "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="],+"emoji-regex": ["[email protected]", "", {}, "sha512-toUI84YS5YmxW219erniWD0CIVOo46xGKColeNQRgOzDorgBi1v4D71/OFzgD9GO2UGKIv1C3Sp8DAn0+j5w7A=="],+ "encodeurl": ["[email protected]", "", {}, "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg=="],++ "es-define-property": ["[email protected]", "", {}, "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g=="],++ "es-errors": ["[email protected]", "", {}, "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw=="],++ "es-object-atoms": ["[email protected]", "", { "dependencies": { "es-errors": "^1.3.0" } }, "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA=="],+"escalade": ["[email protected]", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="],+ "escape-html": ["[email protected]", "", {}, "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow=="],++ "etag": ["[email protected]", "", {}, "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg=="],++ "eventsource": ["[email protected]", "", { "dependencies": { "eventsource-parser": "^3.0.1" } }, "sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA=="],++ "eventsource-parser": ["[email protected]", "", {}, "sha512-Vo1ab+QXPzZ4tCa8SwIHJFaSzy4R6SHf7BY79rFBDf0idraZWAkYrDjDj8uWaSm3S2TK+hJ7/t1CEmZ7jXw+pg=="],++ "express": ["[email protected]", "", { "dependencies": { "accepts": "^2.0.0", "body-parser": "^2.2.1", "content-disposition": "^1.0.0", "content-type": "^1.0.5", "cookie": "^0.7.1", "cookie-signature": "^1.2.1", "debug": "^4.4.0", "depd": "^2.0.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "etag": "^1.8.1", "finalhandler": "^2.1.0", "fresh": "^2.0.0", "http-errors": "^2.0.0", "merge-descriptors": "^2.0.0", "mime-types": "^3.0.0", "on-finished": "^2.4.1", "once": "^1.4.0", "parseurl": "^1.3.3", "proxy-addr": "^2.0.7", "qs": "^6.14.0", "range-parser": "^1.2.1", "router": "^2.2.0", "send": "^1.1.0", "serve-static": "^2.2.0", "statuses": "^2.0.1", "type-is": "^2.0.1", "vary": "^1.1.2" } }, "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw=="],++ "express-rate-limit": ["[email protected]", "", { "peerDependencies": { "express": ">= 4.11" } }, "sha512-7iN8iPMDzOMHPUYllBEsQdWVB6fPDMPqwjBaFrgr4Jgr/+okjvzAy+UHlYYL/Vs0OsOrMkwS6PJDkFlJwoxUnw=="],+"exsolve": ["[email protected]", "", {}, "sha512-LmDxfWXwcTArk8fUEnOfSZpHOJ6zOMUJKOtFLFqJLoKJetuQG874Uc7/Kki7zFLzYybmZhp1M7+98pfMqeX8yA=="],+ "fast-deep-equal": ["[email protected]", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],++ "fast-uri": ["[email protected]", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="],+"fdir": ["[email protected]", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="],+ "finalhandler": ["[email protected]", "", { "dependencies": { "debug": "^4.4.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "on-finished": "^2.4.1", "parseurl": "^1.3.3", "statuses": "^2.0.1" } }, "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA=="],++ "forwarded": ["[email protected]", "", {}, "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow=="],++ "fresh": ["[email protected]", "", {}, "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A=="],++ "function-bind": ["[email protected]", "", {}, "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA=="],+"get-caller-file": ["[email protected]", "", {}, "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg=="],"get-east-asian-width": ["[email protected]", "", {}, "sha512-QZjmEOC+IT1uk6Rx0sX22V6uHWVwbdbxf1faPqJ1QhLdGgsRGCZoyaQBm/piRdJy/D2um6hM1UP7ZEeQ4EkP+Q=="],+ "get-intrinsic": ["[email protected]", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "es-define-property": "^1.0.1", "es-errors": "^1.3.0", "es-object-atoms": "^1.1.1", "function-bind": "^1.1.2", "get-proto": "^1.0.1", "gopd": "^1.2.0", "has-symbols": "^1.1.0", "hasown": "^2.0.2", "math-intrinsics": "^1.1.0" } }, "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ=="],++ "get-proto": ["[email protected]", "", { "dependencies": { "dunder-proto": "^1.0.1", "es-object-atoms": "^1.0.0" } }, "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g=="],+"giget": ["[email protected]", "", { "dependencies": { "citty": "^0.1.6", "consola": "^3.4.0", "defu": "^6.1.4", "node-fetch-native": "^1.6.6", "nypm": "^0.6.0", "pathe": "^2.0.3" }, "bin": { "giget": "dist/cli.mjs" } }, "sha512-L5bGsVkxJbJgdnwyuheIunkGatUF/zssUoxxjACCseZYAVbaqdh9Tsmmlkl8vYan09H7sbvKt4pS8GqKLBrEzA=="],+ "gopd": ["[email protected]", "", {}, "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg=="],++ "has-symbols": ["[email protected]", "", {}, "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ=="],++ "hasown": ["[email protected]", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="],++ "hono": ["[email protected]", "", {}, "sha512-WemPi9/WfyMwZs+ZUXdiwcCh9Y+m7L+8vki9MzDw3jJ+W9Lc+12HGsd368Qc1vZi1xwW8BWMMsnK5efYKPdt4g=="],++ "http-errors": ["[email protected]", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="],++ "iconv-lite": ["[email protected]", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3.0.0" } }, "sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw=="],++ "inherits": ["[email protected]", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],++ "ipaddr.js": ["[email protected]", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],++ "is-promise": ["[email protected]", "", {}, "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ=="],++ "isexe": ["[email protected]", "", {}, "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw=="],+"jiti": ["[email protected]", "", { "bin": { "jiti": "lib/jiti-cli.mjs" } }, "sha512-ekilCSN1jwRvIbgeg/57YFh8qQDNbwDb9xT/qu2DAHbFFZUicIl4ygVaAvzveMhMVr3LnpSKTNnwt8PoOfmKhQ=="],+ "jose": ["[email protected]", "", {}, "sha512-0TpaTfihd4QMNwrz/ob2Bp7X04yuxJkjRGi4aKmOqwhov54i6u79oCv7T+C7lo70MKH6BesI3vscD1yb/yzKXQ=="],++ "json-schema-traverse": ["[email protected]", "", {}, "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug=="],++ "json-schema-typed": ["[email protected]", "", {}, "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA=="],+"jsonc-parser": ["[email protected]", "", {}, "sha512-HUgH65KyejrUFPvHFPbqOY0rsFip3Bo5wb4ngvdi1EpCYWUQDC5V+Y7mZws+DLkr4M//zQJoanu1SP+87Dv1oQ=="],+ "math-intrinsics": ["[email protected]", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="],++ "media-typer": ["[email protected]", "", {}, "sha512-aisnrDP4GNe06UcKFnV5bfMNPBUw4jsLGaWwWfnH3v02GnBuXX2MCVn5RbrWo0j3pczUilYblq7fQ7Nw2t5XKw=="],++ "merge-descriptors": ["[email protected]", "", {}, "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g=="],++ "mime-db": ["[email protected]", "", {}, "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ=="],++ "mime-types": ["[email protected]", "", { "dependencies": { "mime-db": "^1.54.0" } }, "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A=="],++ "ms": ["[email protected]", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="],++ "negotiator": ["[email protected]", "", {}, "sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg=="],+"node-fetch-native": ["[email protected]", "", {}, "sha512-g9yhqoedzIUm0nTnTqAQvueMPVOuIY16bqgAJJC8XOOubYFNwz6IER9qs0Gq2Xd0+CecCKFjtdDTMA4u4xG06Q=="],"nypm": ["[email protected]", "", { "dependencies": { "citty": "^0.1.6", "consola": "^3.4.2", "pathe": "^2.0.3", "pkg-types": "^2.3.0", "tinyexec": "^1.0.1" }, "bin": { "nypm": "dist/cli.mjs" } }, "sha512-7eM+hpOtrKrBDCh7Ypu2lJ9Z7PNZBdi/8AT3AX8xoCj43BBVHD0hPSTEvMtkMpfs8FCqBGhxB+uToIQimA111g=="],+ "object-assign": ["[email protected]", "", {}, "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg=="],++ "object-inspect": ["[email protected]", "", {}, "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew=="],+"ohash": ["[email protected]", "", {}, "sha512-RdR9FQrFwNBNXAr4GixM8YaRZRJ5PUWbKYbE5eOsrwAjJW0q2REGcf79oYPsLyskQCZG1PLN+S/K1V00joZAoQ=="],+ "on-finished": ["[email protected]", "", { "dependencies": { "ee-first": "1.1.1" } }, "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg=="],++ "once": ["[email protected]", "", { "dependencies": { "wrappy": "1" } }, "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w=="],+"package-manager-detector": ["[email protected]", "", {}, "sha512-uBj69dVlYe/+wxj8JOpr97XfsxH/eumMt6HqjNTmJDf/6NO9s+0uxeOneIz3AsPt2m6y9PqzDzd3ATcU17MNfw=="],+ "parseurl": ["[email protected]", "", {}, "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ=="],++ "path-key": ["[email protected]", "", {}, "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q=="],++ "path-to-regexp": ["[email protected]", "", {}, "sha512-7jdwVIRtsP8MYpdXSwOS0YdD0Du+qOoF/AEPIt88PcCFrZCzx41oxku1jD88hZBwbNUIEfpqvuhjFaMAqMTWnA=="],+"pathe": ["[email protected]", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="],"perfect-debounce": ["[email protected]", "", {}, "sha512-fkEH/OBiKrqqI/yIgjR92lMfs2K8105zt/VT6+7eTjNwisrsh47CeIED9z58zI7DfKdH3uHAn25ziRZn3kgAow=="],"picomatch": ["[email protected]", "", {}, "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q=="],+ "pkce-challenge": ["[email protected]", "", {}, "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ=="],+"pkg-types": ["[email protected]", "", { "dependencies": { "confbox": "^0.2.2", "exsolve": "^1.0.7", "pathe": "^2.0.3" } }, "sha512-SIqCzDRg0s9npO5XQ3tNZioRY1uK06lA41ynBC1YmFTmnY6FjUjVt6s4LoADmwoig1qqD0oK8h1p/8mlMx8Oig=="],"prettier": ["[email protected]", "", { "bin": { "prettier": "bin/prettier.cjs" } }, "sha512-I7AIg5boAr5R0FFtJ6rCfD+LFsWHp81dolrFD8S79U9tb8Az2nGrJncnMSnys+bpQJfRUzqs9hnA81OAA3hCuQ=="],+ "proxy-addr": ["[email protected]", "", { "dependencies": { "forwarded": "0.2.0", "ipaddr.js": "1.9.1" } }, "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg=="],++ "qs": ["[email protected]", "", { "dependencies": { "side-channel": "^1.1.0" } }, "sha512-4EK3+xJl8Ts67nLYNwqw/dsFVnCf+qR7RgXSK9jEEm9unao3njwMDdmsdvoKBKHzxd7tCYz5e5M+SnMjdtXGQQ=="],++ "range-parser": ["[email protected]", "", {}, "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="],++ "raw-body": ["[email protected]", "", { "dependencies": { "bytes": "~3.1.2", "http-errors": "~2.0.1", "iconv-lite": "~0.7.0", "unpipe": "~1.0.0" } }, "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA=="],+"rc9": ["[email protected]", "", { "dependencies": { "defu": "^6.1.4", "destr": "^2.0.3" } }, "sha512-btXCnMmRIBINM2LDZoEmOogIZU7Qe7zn4BpomSKZ/ykbLObuBdvG+mFq11DL6fjH1DRwHhrlgtYWG96bJiC7Cg=="],"readdirp": ["[email protected]", "", {}, "sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg=="],+ "require-from-string": ["[email protected]", "", {}, "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw=="],++ "router": ["[email protected]", "", { "dependencies": { "debug": "^4.4.0", "depd": "^2.0.0", "is-promise": "^4.0.0", "parseurl": "^1.3.3", "path-to-regexp": "^8.0.0" } }, "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ=="],++ "safer-buffer": ["[email protected]", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="],+"semver": ["[email protected]", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-SdsKMrI9TdgjdweUSR9MweHA4EJ8YxHn8DFaDisvhVlUOe4BF1tLD7GAj0lIqWVl+dPb/rExr0Btby5loQm20Q=="],+ "send": ["[email protected]", "", { "dependencies": { "debug": "^4.4.3", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "etag": "^1.8.1", "fresh": "^2.0.0", "http-errors": "^2.0.1", "mime-types": "^3.0.2", "ms": "^2.1.3", "on-finished": "^2.4.1", "range-parser": "^1.2.1", "statuses": "^2.0.2" } }, "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ=="],++ "serve-static": ["[email protected]", "", { "dependencies": { "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "parseurl": "^1.3.3", "send": "^1.2.0" } }, "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw=="],++ "setprototypeof": ["[email protected]", "", {}, "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw=="],++ "shebang-command": ["[email protected]", "", { "dependencies": { "shebang-regex": "^3.0.0" } }, "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA=="],++ "shebang-regex": ["[email protected]", "", {}, "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A=="],++ "side-channel": ["[email protected]", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.3", "side-channel-list": "^1.0.0", "side-channel-map": "^1.0.1", "side-channel-weakmap": "^1.0.2" } }, "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw=="],++ "side-channel-list": ["[email protected]", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.3" } }, "sha512-FCLHtRD/gnpCiCHEiJLOwdmFP+wzCmDEkc9y7NsYxeF4u7Btsn1ZuwgwJGxImImHicJArLP4R0yX4c2KCrMrTA=="],++ "side-channel-map": ["[email protected]", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3" } }, "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA=="],++ "side-channel-weakmap": ["[email protected]", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3", "side-channel-map": "^1.0.1" } }, "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A=="],++ "statuses": ["[email protected]", "", {}, "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw=="],+"string-width": ["[email protected]", "", { "dependencies": { "emoji-regex": "^10.3.0", "get-east-asian-width": "^1.0.0", "strip-ansi": "^7.1.0" } }, "sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ=="],"strip-ansi": ["[email protected]", "", { "dependencies": { "ansi-regex": "^6.0.1" } }, "sha512-gmBGslpoQJtgnMAvOVqGZpEz9dyoKTCzy2nfz/n8aIFhN/jCE/rCmcxabB6jOOHV+0WNnylOxaxBQPSvcWklhA=="],@@ -111,12 +276,24 @@"tinyglobby": ["[email protected]", "", { "dependencies": { "fdir": "^6.5.0", "picomatch": "^4.0.3" } }, "sha512-j2Zq4NyQYG5XMST4cbs02Ak8iJUdxRM0XI5QyxXuZOzKOINmWurp3smXu3y5wDcJrptwpSjgXHzIQxR0omXljQ=="],+ "toidentifier": ["[email protected]", "", {}, "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA=="],++ "type-is": ["[email protected]", "", { "dependencies": { "content-type": "^1.0.5", "media-typer": "^1.1.0", "mime-types": "^3.0.0" } }, "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw=="],+"typescript": ["[email protected]", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],"undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],+ "unpipe": ["[email protected]", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="],++ "vary": ["[email protected]", "", {}, "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg=="],++ "which": ["[email protected]", "", { "dependencies": { "isexe": "^2.0.0" }, "bin": { "node-which": "./bin/node-which" } }, "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA=="],+"wrap-ansi": ["[email protected]", "", { "dependencies": { "ansi-styles": "^6.2.1", "string-width": "^7.0.0", "strip-ansi": "^7.1.0" } }, "sha512-42AtmgqjV+X1VpdOfyTGOYRi0/zsoLqtXQckTmqTeybT+BDIbM/Guxo7x3pE2vtpr1ok6xRqM9OpBe+Jyoqyww=="],+ "wrappy": ["[email protected]", "", {}, "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ=="],+"y18n": ["[email protected]", "", {}, "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA=="],"yaml": ["[email protected]", "", { "bin": { "yaml": "bin.mjs" } }, "sha512-lcYcMxX2PO9XMGvAJkJ3OsNMw+/7FKes7/hgerGUYWIoWu5j/+YQqcZr5JnPZWzOsEBgMbSbiSTn/dv/69Mkpw=="],@@ -124,5 +301,9 @@"yargs": ["[email protected]", "", { "dependencies": { "cliui": "^9.0.1", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "string-width": "^7.2.0", "y18n": "^5.0.5", "yargs-parser": "^22.0.0" } }, "sha512-4UEqdc2RYGHZc7Doyqkrqiln3p9X2DZVxaGbwhn2pi7MrRagKaOcIKe8L3OxYcbhXLgLFUS3zAYuQjKBQgmuNg=="],"yargs-parser": ["[email protected]", "", {}, "sha512-rwu/ClNdSMpkSrUb+d6BRsSkLUq1fmfsY6TOpYzTwvwkg1/NRG85KBy3kq++A8LKQwX6lsu+aWad+2khvuXrqw=="],++ "zod": ["[email protected]", "", {}, "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ=="],++ "zod-to-json-schema": ["[email protected]", "", { "peerDependencies": { "zod": "^3.25 || ^4" } }, "sha512-pM/SU9d3YAggzi6MtR4h7ruuQlqKtad8e9S0fmxcMi+ueAK5Korys/aWcV9LIIHTVbj01NdzxcnXSN+O74ZIVA=="],}}diff --git a/libs/typescript/cua-cli/package.json b/libs/typescript/cua-cli/package.jsonindex 275df2353d..3d6df88122 100644--- a/libs/typescript/cua-cli/package.json+++ b/libs/typescript/cua-cli/package.json@@ -33,7 +33,9 @@"prepublishOnly": "bun run typecheck"},"dependencies": {- "yargs": "^18.0.0"+ "@modelcontextprotocol/sdk": "^1.0.0",+ "yargs": "^18.0.0",+ "zod": "^3.23.0"},"devDependencies": {"@types/bun": "latest",diff --git a/libs/typescript/cua-cli/src/cli.ts b/libs/typescript/cua-cli/src/cli.tsindex 293b1cc3da..b72ab8f55a 100644--- a/libs/typescript/cua-cli/src/cli.ts+++ b/libs/typescript/cua-cli/src/cli.ts@@ -3,6 +3,8 @@ import { hideBin } from 'yargs/helpers';import { registerAuthCommands } from './commands/auth';import { registerSandboxCommands } from './commands/sandbox';import { registerImageCommands } from './commands/image';+import { registerSkillsCommands } from './commands/skills';+import { registerServeMcpCommands } from './commands/serve-mcp';import { http } from './http';import { getApiKey } from './storage';import type { SandboxItem } from './util';@@ -49,6 +51,16 @@ export async function runCli() {' pull Pull a VM image from cloud storage\n' +' delete Delete an image version\n' +'\n' ++ ' cua skills <command> Create and manage demonstration skills\n' ++ ' record Record a demonstration and create a skill\n' ++ ' list View all saved skills\n' ++ ' read Read a skill as JSON or Markdown\n' ++ ' replay Open the video recording for a skill\n' ++ ' delete Delete a skill\n' ++ ' clean Delete all skills (with confirmation)\n' ++ '\n' ++ ' cua serve-mcp Start an MCP server for Claude Code integration\n' ++ '\n' +'Documentation: https://docs.cua.ai/libraries/cua-cli/commands').completion(@@ -56,8 +68,27 @@ export async function runCli() {'Generate shell completion script',async function (current, argv) {// Provide dynamic completions for sandbox names- const commands = ['auth', 'sandbox', 'sb', 'completion'];+ const commands = [+ 'auth',+ 'sandbox',+ 'sb',+ 'image',+ 'img',+ 'skills',+ 'serve-mcp',+ 'completion',+ ];const authCommands = ['login', 'env', 'logout'];+ const imageCommands = ['list', 'ls', 'push', 'pull', 'delete'];+ const skillsCommands = [+ 'record',+ 'list',+ 'ls',+ 'read',+ 'replay',+ 'delete',+ 'clean',+ ];const sbCommands = ['list','ls',@@ -111,6 +142,12 @@ export async function runCli() {if (args[0] === 'sandbox' || args[0] === 'sb') {return sbCommands;}+ if (args[0] === 'image' || args[0] === 'img') {+ return imageCommands;+ }+ if (args[0] === 'skills') {+ return skillsCommands;+ }return [];}@@ -125,6 +162,8 @@ export async function runCli() {argv = registerAuthCommands(argv);argv = registerSandboxCommands(argv);argv = registerImageCommands(argv);+ argv = registerSkillsCommands(argv);+ argv = registerServeMcpCommands(argv);// Check for version flag before command validationconst args = process.argv.slice(2);diff --git a/libs/typescript/cua-cli/src/commands/serve-mcp.ts b/libs/typescript/cua-cli/src/commands/serve-mcp.tsnew file mode 100644index 0000000000..9ecc4a1f46--- /dev/null+++ b/libs/typescript/cua-cli/src/commands/serve-mcp.ts@@ -0,0 +1,1520 @@+import type { Argv } from 'yargs';+import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';+import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';+import { z } from 'zod';+import { getApiKey } from '../storage';+import { http } from '../http';+import type { SandboxItem } from '../util';+import { homedir } from 'os';+import { join } from 'path';+import { spawn } from 'child_process';++// Permission levels for the MCP server+type Permission =+ | 'list_sandboxes'+ | 'create_sandbox'+ | 'delete_sandbox'+ | 'start_sandbox'+ | 'stop_sandbox'+ | 'restart_sandbox'+ | 'suspend_sandbox'+ | 'get_sandbox'+ | 'computer_screenshot'+ | 'computer_click'+ | 'computer_type'+ | 'computer_key'+ | 'computer_scroll'+ | 'computer_drag'+ | 'computer_hotkey'+ | 'computer_clipboard'+ | 'computer_file'+ | 'computer_shell'+ | 'computer_window'+ | 'skill_list'+ | 'skill_read'+ | 'skill_record'+ | 'skill_delete';++const ALL_PERMISSIONS: Permission[] = [+ 'list_sandboxes',+ 'create_sandbox',+ 'delete_sandbox',+ 'start_sandbox',+ 'stop_sandbox',+ 'restart_sandbox',+ 'suspend_sandbox',+ 'get_sandbox',+ 'computer_screenshot',+ 'computer_click',+ 'computer_type',+ 'computer_key',+ 'computer_scroll',+ 'computer_drag',+ 'computer_hotkey',+ 'computer_clipboard',+ 'computer_file',+ 'computer_shell',+ 'computer_window',+ 'skill_list',+ 'skill_read',+ 'skill_record',+ 'skill_delete',+];++// Skills directory path+const SKILLS_DIR = join(homedir(), '.cua', 'skills');++// Ensure skills directory exists+async function ensureSkillsDir(): Promise<void> {+ const fs = await import('fs/promises');+ await fs.mkdir(SKILLS_DIR, { recursive: true });+}++// Get skill directory path+function getSkillDir(name: string): string {+ return join(SKILLS_DIR, name);+}++// List all skills+async function listSkills(): Promise<string[]> {+ const fs = await import('fs/promises');+ await ensureSkillsDir();++ try {+ const entries = await fs.readdir(SKILLS_DIR, { withFileTypes: true });+ const skills: string[] = [];++ for (const entry of entries) {+ if (entry.isDirectory()) {+ const skillMdPath = join(SKILLS_DIR, entry.name, 'SKILL.md');+ try {+ await fs.access(skillMdPath);+ skills.push(entry.name);+ } catch {+ // Skip directories without SKILL.md+ }+ }+ }++ return skills;+ } catch {+ return [];+ }+}++// Recursively get all files in a directory+async function getAllFiles(dir: string): Promise<string[]> {+ const fs = await import('fs/promises');+ const files: string[] = [];++ try {+ const entries = await fs.readdir(dir, { withFileTypes: true });+ for (const entry of entries) {+ const fullPath = join(dir, entry.name);+ if (entry.isDirectory()) {+ const subFiles = await getAllFiles(fullPath);+ files.push(...subFiles);+ } else {+ files.push(fullPath);+ }+ }+ } catch {+ // Directory doesn't exist or can't be read+ }++ return files;+}++// Read a specific skill - returns raw SKILL.md content with file paths prefixed+async function readSkill(name: string): Promise<string | null> {+ const fs = await import('fs/promises');+ const skillDir = getSkillDir(name);+ const skillMdPath = join(skillDir, 'SKILL.md');++ try {+ const content = await fs.readFile(skillMdPath, 'utf-8');+ const files = await getAllFiles(skillDir);++ // Prefix with file paths+ const filesList = files.map((f) => f).join('\n');+ return `## Files\n\n${filesList}\n\n---\n\n${content}`;+ } catch {+ return null;+ }+}++// Parse permissions from environment variable+function parsePermissions(permEnv: string | undefined): Set<Permission> {+ if (!permEnv) {+ return new Set(ALL_PERMISSIONS);+ }+ const perms = permEnv.split(',').map((p) => p.trim() as Permission);+ return new Set(perms.filter((p) => ALL_PERMISSIONS.includes(p)));+}++// Send command to sandbox's computer-server via /cmd endpoint+async function sendCommand(+ sandbox: string,+ command: string,+ params: Record<string, unknown>,+ token: string+): Promise<{ success: boolean; data?: unknown; error?: string }> {+ // First, get the sandbox details to find the host+ const listRes = await http('/v1/vms', { token });+ if (!listRes.ok) {+ return {+ success: false,+ error: `Failed to list sandboxes: ${listRes.status}`,+ };+ }++ const sandboxes = (await listRes.json()) as SandboxItem[];+ const sb = sandboxes.find((s) => s.name === sandbox);++ if (!sb) {+ return { success: false, error: `Sandbox not found: ${sandbox}` };+ }++ if (sb.status !== 'running') {+ return {+ success: false,+ error: `Sandbox is not running (status: ${sb.status})`,+ };+ }++ const host = sb.host || `${sb.name}.sandbox.cua.ai`;+ const cmdUrl = `https://${host}:8443/cmd`;++ try {+ const controller = new AbortController();+ const timeout = setTimeout(() => controller.abort(), 30000);++ const res = await fetch(cmdUrl, {+ method: 'POST',+ headers: {+ 'Content-Type': 'application/json',+ 'X-Container-Name': sandbox,+ 'X-API-Key': token,+ },+ body: JSON.stringify({ command, params }),+ signal: controller.signal,+ });++ clearTimeout(timeout);++ if (!res.ok) {+ const errorText = await res.text();+ return {+ success: false,+ error: `Command failed: ${res.status} - ${errorText}`,+ };+ }++ // Parse SSE response format: "data: {...}"+ const text = await res.text();+ if (text.startsWith('data: ')) {+ const jsonStr = text.slice(6);+ const data = JSON.parse(jsonStr);+ return { success: true, data };+ }++ // Try parsing as regular JSON+ try {+ const data = JSON.parse(text);+ return { success: true, data };+ } catch {+ return { success: true, data: text };+ }+ } catch (err) {+ return { success: false, error: `Command error: ${err}` };+ }+}++// Create and run the MCP server+async function runMcpServer(permissions: Set<Permission>) {+ const token = getApiKey();+ if (!token) {+ console.error('Error: No API key found. Run "cua auth login" first.');+ process.exit(1);+ }++ const server = new McpServer({+ name: 'cua-cloud',+ version: '1.0.0',+ });++ // ============================================================+ // SANDBOX MANAGEMENT TOOLS+ // ============================================================++ if (permissions.has('list_sandboxes')) {+ server.tool(+ 'list_sandboxes',+ 'List all cloud sandboxes in your account with their status and details',+ {},+ async () => {+ const res = await http('/v1/vms', { token });+ if (!res.ok) {+ return {+ content: [+ { type: 'text', text: `Failed to list sandboxes: ${res.status}` },+ ],+ };+ }+ const sandboxes = (await res.json()) as SandboxItem[];+ return {+ content: [+ {+ type: 'text',+ text: JSON.stringify(+ sandboxes.map((s) => ({+ name: s.name,+ status: s.status,+ host: s.host || `${s.name}.sandbox.cua.ai`,+ })),+ null,+ 2+ ),+ },+ ],+ };+ }+ );+ }++ if (permissions.has('create_sandbox')) {+ server.tool(+ 'create_sandbox',+ 'Create a new cloud sandbox with specified OS, size, and region',+ {+ os: z+ .enum(['linux', 'windows', 'macos'])+ .describe('Operating system for the sandbox'),+ size: z+ .enum(['small', 'medium', 'large'])+ .describe('Size/configuration of the sandbox'),+ region: z+ .enum(['north-america', 'europe', 'asia-pacific', 'south-america'])+ .describe('Geographic region for the sandbox'),+ },+ async ({ os, size, region }) => {+ const res = await http('/v1/vms', {+ token,+ method: 'POST',+ body: { os, configuration: size, region },+ });++ if (res.status === 200) {+ const data = (await res.json()) as {+ name: string;+ password: string;+ host: string;+ };+ return {+ content: [+ {+ type: 'text',+ text: `Sandbox created: ${data.name}\nHost: ${data.host}\nPassword: ${data.password}`,+ },+ ],+ };+ }++ if (res.status === 202) {+ const data = (await res.json()) as { name: string; job_id: string };+ return {+ content: [+ {+ type: 'text',+ text: `Sandbox provisioning started: ${data.name}\nJob ID: ${data.job_id}`,+ },+ ],+ };+ }++ return {+ content: [+ { type: 'text', text: `Failed to create sandbox: ${res.status}` },+ ],+ };+ }+ );+ }++ if (permissions.has('delete_sandbox')) {+ server.tool(+ 'delete_sandbox',+ 'Permanently delete a sandbox and all its data',+ {+ sandbox: z.string().describe('Name of the sandbox to delete'),+ },+ async ({ sandbox }) => {+ const res = await http(`/v1/vms/${encodeURIComponent(sandbox)}`, {+ token,+ method: 'DELETE',+ });++ if (res.status === 202) {+ return {+ content: [+ { type: 'text', text: `Sandbox deletion initiated: ${sandbox}` },+ ],+ };+ }++ if (res.status === 404) {+ return {+ content: [{ type: 'text', text: `Sandbox not found: ${sandbox}` }],+ };+ }++ return {+ content: [+ { type: 'text', text: `Failed to delete sandbox: ${res.status}` },+ ],+ };+ }+ );+ }++ if (permissions.has('start_sandbox')) {+ server.tool(+ 'start_sandbox',+ 'Start a stopped sandbox',+ {+ sandbox: z.string().describe('Name of the sandbox to start'),+ },+ async ({ sandbox }) => {+ const res = await http(`/v1/vms/${encodeURIComponent(sandbox)}/start`, {+ token,+ method: 'POST',+ });++ if (res.status === 204) {+ return {+ content: [{ type: 'text', text: `Start accepted for: ${sandbox}` }],+ };+ }++ return {+ content: [+ { type: 'text', text: `Failed to start sandbox: ${res.status}` },+ ],+ };+ }+ );+ }++ if (permissions.has('stop_sandbox')) {+ server.tool(+ 'stop_sandbox',+ 'Stop a running sandbox (data is preserved)',+ {+ sandbox: z.string().describe('Name of the sandbox to stop'),+ },+ async ({ sandbox }) => {+ const res = await http(`/v1/vms/${encodeURIComponent(sandbox)}/stop`, {+ token,+ method: 'POST',+ });++ if (res.status === 202) {+ return {+ content: [{ type: 'text', text: `Stop initiated for: ${sandbox}` }],+ };+ }++ return {+ content: [+ { type: 'text', text: `Failed to stop sandbox: ${res.status}` },+ ],+ };+ }+ );+ }++ if (permissions.has('restart_sandbox')) {+ server.tool(+ 'restart_sandbox',+ 'Restart a sandbox (reboot the system)',+ {+ sandbox: z.string().describe('Name of the sandbox to restart'),+ },+ async ({ sandbox }) => {+ const res = await http(+ `/v1/vms/${encodeURIComponent(sandbox)}/restart`,+ {+ token,+ method: 'POST',+ }+ );++ if (res.status === 202) {+ return {+ content: [+ { type: 'text', text: `Restart initiated for: ${sandbox}` },+ ],+ };+ }++ return {+ content: [+ { type: 'text', text: `Failed to restart sandbox: ${res.status}` },+ ],+ };+ }+ );+ }++ if (permissions.has('suspend_sandbox')) {+ server.tool(+ 'suspend_sandbox',+ 'Suspend a sandbox, preserving memory state',+ {+ sandbox: z.string().describe('Name of the sandbox to suspend'),+ },+ async ({ sandbox }) => {+ const res = await http(+ `/v1/vms/${encodeURIComponent(sandbox)}/suspend`,+ {+ token,+ method: 'POST',+ }+ );++ if (res.status === 202) {+ return {+ content: [+ { type: 'text', text: `Suspend initiated for: ${sandbox}` },+ ],+ };+ }++ return {+ content: [+ { type: 'text', text: `Failed to suspend sandbox: ${res.status}` },+ ],+ };+ }+ );+ }++ if (permissions.has('get_sandbox')) {+ server.tool(+ 'get_sandbox',+ 'Get detailed information about a specific sandbox',+ {+ sandbox: z.string().describe('Name of the sandbox to get info for'),+ },+ async ({ sandbox }) => {+ const res = await http('/v1/vms', { token });+ if (!res.ok) {+ return {+ content: [+ { type: 'text', text: `Failed to get sandbox: ${res.status}` },+ ],+ };+ }++ const sandboxes = (await res.json()) as SandboxItem[];+ const sb = sandboxes.find((s) => s.name === sandbox);++ if (!sb) {+ return {+ content: [{ type: 'text', text: `Sandbox not found: ${sandbox}` }],+ };+ }++ return {+ content: [+ {+ type: 'text',+ text: JSON.stringify(+ {+ name: sb.name,+ status: sb.status,+ host: sb.host || `${sb.name}.sandbox.cua.ai`,+ },+ null,+ 2+ ),+ },+ ],+ };+ }+ );+ }++ // ============================================================+ // COMPUTER CONTROL TOOLS (proxied to sandbox /cmd endpoint)+ // ============================================================++ if (permissions.has('computer_screenshot')) {+ server.tool(+ 'computer_screenshot',+ 'Take a screenshot of the sandbox screen. Returns a base64-encoded image.',+ {+ sandbox: z.string().describe('Name of the sandbox to screenshot'),+ },+ async ({ sandbox }) => {+ const result = await sendCommand(sandbox, 'screenshot', {}, token);+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Screenshot failed' },+ ],+ };+ }++ const data = result.data as { image_data?: string; success?: boolean };+ if (data?.image_data) {+ return {+ content: [+ {+ type: 'image',+ data: data.image_data,+ mimeType: 'image/png',+ },+ ],+ };+ }++ return {+ content: [{ type: 'text', text: JSON.stringify(result.data) }],+ };+ }+ );++ server.tool(+ 'computer_get_screen_size',+ 'Get the screen dimensions of the sandbox',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ },+ async ({ sandbox }) => {+ const result = await sendCommand(sandbox, 'get_screen_size', {}, token);+ if (!result.success) {+ return {+ content: [+ {+ type: 'text',+ text: result.error || 'Failed to get screen size',+ },+ ],+ };+ }+ return {+ content: [{ type: 'text', text: JSON.stringify(result.data) }],+ };+ }+ );+ }++ if (permissions.has('computer_click')) {+ server.tool(+ 'computer_click',+ 'Click at the specified screen coordinates',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ x: z.number().describe('X coordinate in pixels'),+ y: z.number().describe('Y coordinate in pixels'),+ button: z+ .enum(['left', 'right', 'middle'])+ .optional()+ .default('left')+ .describe('Mouse button to click'),+ },+ async ({ sandbox, x, y, button }) => {+ // Use the appropriate click command based on button+ let command: string;+ if (button === 'right') {+ command = 'right_click';+ } else if (button === 'middle') {+ // Middle click is mouse_down + mouse_up with middle button+ await sendCommand(+ sandbox,+ 'mouse_down',+ { x, y, button: 'middle' },+ token+ );+ const result = await sendCommand(+ sandbox,+ 'mouse_up',+ { x, y, button: 'middle' },+ token+ );+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Middle click failed' },+ ],+ };+ }+ return {+ content: [+ {+ type: 'text',+ text: `Middle-clicked at (${x}, ${y})`,+ },+ ],+ };+ } else {+ command = 'left_click';+ }++ const result = await sendCommand(sandbox, command, { x, y }, token);+ if (!result.success) {+ return {+ content: [{ type: 'text', text: result.error || 'Click failed' }],+ };+ }+ return {+ content: [+ {+ type: 'text',+ text: `Clicked at (${x}, ${y}) with ${button} button`,+ },+ ],+ };+ }+ );++ server.tool(+ 'computer_double_click',+ 'Double-click at the specified screen coordinates',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ x: z.number().describe('X coordinate in pixels'),+ y: z.number().describe('Y coordinate in pixels'),+ },+ async ({ sandbox, x, y }) => {+ const result = await sendCommand(+ sandbox,+ 'double_click',+ { x, y },+ token+ );+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Double click failed' },+ ],+ };+ }+ return {+ content: [{ type: 'text', text: `Double-clicked at (${x}, ${y})` }],+ };+ }+ );++ server.tool(+ 'computer_move',+ 'Move the cursor to the specified screen coordinates',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ x: z.number().describe('X coordinate in pixels'),+ y: z.number().describe('Y coordinate in pixels'),+ },+ async ({ sandbox, x, y }) => {+ const result = await sendCommand(+ sandbox,+ 'move_cursor',+ { x, y },+ token+ );+ if (!result.success) {+ return {+ content: [{ type: 'text', text: result.error || 'Move failed' }],+ };+ }+ return {+ content: [{ type: 'text', text: `Moved cursor to (${x}, ${y})` }],+ };+ }+ );++ server.tool(+ 'computer_mousedown',+ 'Press and hold a mouse button at the specified coordinates',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ x: z+ .number()+ .optional()+ .describe(+ 'X coordinate in pixels (uses current position if not specified)'+ ),+ y: z+ .number()+ .optional()+ .describe(+ 'Y coordinate in pixels (uses current position if not specified)'+ ),+ button: z+ .enum(['left', 'right', 'middle'])+ .optional()+ .default('left')+ .describe('Mouse button to press'),+ },+ async ({ sandbox, x, y, button }) => {+ const params: Record<string, unknown> = { button };+ if (x !== undefined) params.x = x;+ if (y !== undefined) params.y = y;+ const result = await sendCommand(sandbox, 'mouse_down', params, token);+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Mouse down failed' },+ ],+ };+ }+ const posStr =+ x !== undefined && y !== undefined ? ` at (${x}, ${y})` : '';+ return {+ content: [+ { type: 'text', text: `Mouse ${button} button pressed${posStr}` },+ ],+ };+ }+ );++ server.tool(+ 'computer_mouseup',+ 'Release a mouse button at the specified coordinates',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ x: z+ .number()+ .optional()+ .describe(+ 'X coordinate in pixels (uses current position if not specified)'+ ),+ y: z+ .number()+ .optional()+ .describe(+ 'Y coordinate in pixels (uses current position if not specified)'+ ),+ button: z+ .enum(['left', 'right', 'middle'])+ .optional()+ .default('left')+ .describe('Mouse button to release'),+ },+ async ({ sandbox, x, y, button }) => {+ const params: Record<string, unknown> = { button };+ if (x !== undefined) params.x = x;+ if (y !== undefined) params.y = y;+ const result = await sendCommand(sandbox, 'mouse_up', params, token);+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Mouse up failed' },+ ],+ };+ }+ const posStr =+ x !== undefined && y !== undefined ? ` at (${x}, ${y})` : '';+ return {+ content: [+ { type: 'text', text: `Mouse ${button} button released${posStr}` },+ ],+ };+ }+ );++ server.tool(+ 'computer_mousemove',+ 'Move the mouse cursor to the specified coordinates without clicking',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ x: z.number().describe('X coordinate in pixels'),+ y: z.number().describe('Y coordinate in pixels'),+ },+ async ({ sandbox, x, y }) => {+ const result = await sendCommand(+ sandbox,+ 'move_cursor',+ { x, y },+ token+ );+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Mouse move failed' },+ ],+ };+ }+ return {+ content: [{ type: 'text', text: `Mouse moved to (${x}, ${y})` }],+ };+ }+ );+ }++ if (permissions.has('computer_type')) {+ server.tool(+ 'computer_type',+ 'Type text at the current cursor position',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ text: z.string().describe('The text to type'),+ },+ async ({ sandbox, text }) => {+ const result = await sendCommand(sandbox, 'type_text', { text }, token);+ if (!result.success) {+ return {+ content: [{ type: 'text', text: result.error || 'Type failed' }],+ };+ }+ return { content: [{ type: 'text', text: `Typed: "${text}"` }] };+ }+ );+ }++ if (permissions.has('computer_key')) {+ server.tool(+ 'computer_press_key',+ 'Press a single key',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ key: z+ .string()+ .describe('The key to press (e.g., enter, tab, escape, a, b)'),+ },+ async ({ sandbox, key }) => {+ const result = await sendCommand(sandbox, 'press_key', { key }, token);+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Key press failed' },+ ],+ };+ }+ return { content: [{ type: 'text', text: `Pressed key: ${key}` }] };+ }+ );++ server.tool(+ 'computer_keydown',+ 'Press and hold a key (use with computer_keyup to release)',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ key: z+ .string()+ .describe('The key to hold down (e.g., shift, ctrl, alt, cmd)'),+ },+ async ({ sandbox, key }) => {+ const result = await sendCommand(sandbox, 'key_down', { key }, token);+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Key down failed' },+ ],+ };+ }+ return { content: [{ type: 'text', text: `Key held down: ${key}` }] };+ }+ );++ server.tool(+ 'computer_keyup',+ 'Release a held key',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ key: z.string().describe('The key to release'),+ },+ async ({ sandbox, key }) => {+ const result = await sendCommand(sandbox, 'key_up', { key }, token);+ if (!result.success) {+ return {+ content: [{ type: 'text', text: result.error || 'Key up failed' }],+ };+ }+ return { content: [{ type: 'text', text: `Key released: ${key}` }] };+ }+ );+ }++ if (permissions.has('computer_hotkey')) {+ server.tool(+ 'computer_hotkey',+ 'Press a key combination (hotkey)',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ keys: z+ .array(z.string())+ .describe(+ 'List of keys to press together (e.g., ["ctrl", "c"] for copy)'+ ),+ },+ async ({ sandbox, keys }) => {+ const result = await sendCommand(sandbox, 'hotkey', { keys }, token);+ if (!result.success) {+ return {+ content: [{ type: 'text', text: result.error || 'Hotkey failed' }],+ };+ }+ return {+ content: [+ { type: 'text', text: `Pressed hotkey: ${keys.join('+')}` },+ ],+ };+ }+ );+ }++ if (permissions.has('computer_scroll')) {+ server.tool(+ 'computer_scroll',+ 'Scroll at the specified position',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ x: z.number().describe('X coordinate where to scroll'),+ y: z.number().describe('Y coordinate where to scroll'),+ scroll_x: z+ .number()+ .optional()+ .default(0)+ .describe('Horizontal scroll amount (positive = right)'),+ scroll_y: z+ .number()+ .optional()+ .default(0)+ .describe('Vertical scroll amount (positive = down)'),+ },+ async ({ sandbox, x, y, scroll_x, scroll_y }) => {+ // First move cursor to position+ await sendCommand(sandbox, 'move_cursor', { x, y }, token);+ const result = await sendCommand(+ sandbox,+ 'scroll',+ { scroll_x, scroll_y },+ token+ );+ if (!result.success) {+ return {+ content: [{ type: 'text', text: result.error || 'Scroll failed' }],+ };+ }+ return {+ content: [+ {+ type: 'text',+ text: `Scrolled at (${x}, ${y}) by (${scroll_x}, ${scroll_y})`,+ },+ ],+ };+ }+ );+ }++ if (permissions.has('computer_drag')) {+ server.tool(+ 'computer_drag',+ 'Drag from start coordinates to end coordinates',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ start_x: z.number().describe('Starting X coordinate'),+ start_y: z.number().describe('Starting Y coordinate'),+ end_x: z.number().describe('Ending X coordinate'),+ end_y: z.number().describe('Ending Y coordinate'),+ button: z+ .enum(['left', 'right'])+ .optional()+ .default('left')+ .describe('Mouse button to use'),+ },+ async ({ sandbox, start_x, start_y, end_x, end_y, button }) => {+ // Move to start position first+ await sendCommand(+ sandbox,+ 'move_cursor',+ { x: start_x, y: start_y },+ token+ );+ const result = await sendCommand(+ sandbox,+ 'drag_to',+ { x: end_x, y: end_y, button },+ token+ );+ if (!result.success) {+ return {+ content: [{ type: 'text', text: result.error || 'Drag failed' }],+ };+ }+ return {+ content: [+ {+ type: 'text',+ text: `Dragged from (${start_x}, ${start_y}) to (${end_x}, ${end_y})`,+ },+ ],+ };+ }+ );+ }++ if (permissions.has('computer_clipboard')) {+ server.tool(+ 'computer_clipboard_get',+ 'Get the current clipboard content',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ },+ async ({ sandbox }) => {+ const result = await sendCommand(+ sandbox,+ 'copy_to_clipboard',+ {},+ token+ );+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Failed to get clipboard' },+ ],+ };+ }+ return {+ content: [{ type: 'text', text: JSON.stringify(result.data) }],+ };+ }+ );++ server.tool(+ 'computer_clipboard_set',+ 'Set the clipboard content',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ text: z.string().describe('Text to copy to clipboard'),+ },+ async ({ sandbox, text }) => {+ const result = await sendCommand(+ sandbox,+ 'set_clipboard',+ { text },+ token+ );+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Failed to set clipboard' },+ ],+ };+ }+ return { content: [{ type: 'text', text: 'Clipboard set' }] };+ }+ );+ }++ if (permissions.has('computer_shell')) {+ server.tool(+ 'computer_run_command',+ 'Execute a shell command and return the output',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ command: z.string().describe('The shell command to execute'),+ },+ async ({ sandbox, command }) => {+ const result = await sendCommand(+ sandbox,+ 'run_command',+ { command },+ token+ );+ if (!result.success) {+ return {+ content: [{ type: 'text', text: result.error || 'Command failed' }],+ };+ }+ return {+ content: [+ { type: 'text', text: JSON.stringify(result.data, null, 2) },+ ],+ };+ }+ );+ }++ if (permissions.has('computer_file')) {+ server.tool(+ 'computer_file_read',+ 'Read the text content of a file on the sandbox',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ path: z.string().describe('Path to the file to read'),+ },+ async ({ sandbox, path }) => {+ const result = await sendCommand(sandbox, 'read_text', { path }, token);+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Failed to read file' },+ ],+ };+ }+ return {+ content: [{ type: 'text', text: JSON.stringify(result.data) }],+ };+ }+ );++ server.tool(+ 'computer_file_write',+ 'Write text content to a file on the sandbox',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ path: z.string().describe('Path to the file to write'),+ content: z.string().describe('Text content to write'),+ },+ async ({ sandbox, path, content }) => {+ const result = await sendCommand(+ sandbox,+ 'write_text',+ { path, content },+ token+ );+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Failed to write file' },+ ],+ };+ }+ return { content: [{ type: 'text', text: `File written: ${path}` }] };+ }+ );++ server.tool(+ 'computer_list_directory',+ 'List the contents of a directory on the sandbox',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ path: z.string().describe('Path to the directory'),+ },+ async ({ sandbox, path }) => {+ const result = await sendCommand(sandbox, 'list_dir', { path }, token);+ if (!result.success) {+ return {+ content: [+ {+ type: 'text',+ text: result.error || 'Failed to list directory',+ },+ ],+ };+ }+ return {+ content: [+ { type: 'text', text: JSON.stringify(result.data, null, 2) },+ ],+ };+ }+ );+ }++ if (permissions.has('computer_window')) {+ server.tool(+ 'computer_open',+ 'Open a file or URL with the default application',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ target: z.string().describe('File path or URL to open'),+ },+ async ({ sandbox, target }) => {+ const result = await sendCommand(sandbox, 'open', { target }, token);+ if (!result.success) {+ return {+ content: [{ type: 'text', text: result.error || 'Failed to open' }],+ };+ }+ return { content: [{ type: 'text', text: `Opened: ${target}` }] };+ }+ );++ server.tool(+ 'computer_launch_app',+ 'Launch an application on the sandbox',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ app: z.string().describe('Application name or path'),+ args: z+ .array(z.string())+ .optional()+ .describe('Optional arguments to pass to the app'),+ },+ async ({ sandbox, app, args }) => {+ const result = await sendCommand(+ sandbox,+ 'launch',+ { app, args: args || [] },+ token+ );+ if (!result.success) {+ return {+ content: [+ { type: 'text', text: result.error || 'Failed to launch app' },+ ],+ };+ }+ return { content: [{ type: 'text', text: `Launched: ${app}` }] };+ }+ );++ server.tool(+ 'computer_get_active_window',+ 'Get the currently active window ID',+ {+ sandbox: z.string().describe('Name of the sandbox'),+ },+ async ({ sandbox }) => {+ const result = await sendCommand(+ sandbox,+ 'get_current_window_id',+ {},+ token+ );+ if (!result.success) {+ return {+ content: [+ {+ type: 'text',+ text: result.error || 'Failed to get active window',+ },+ ],+ };+ }+ return {+ content: [{ type: 'text', text: JSON.stringify(result.data) }],+ };+ }+ );+ }++ // ============================================================+ // SKILL MANAGEMENT TOOLS+ // ============================================================++ if (permissions.has('skill_list')) {+ server.tool(+ 'skill_list',+ 'List all saved skills (human demonstrations that can guide agent behavior)',+ {},+ async () => {+ const skills = await listSkills();+ if (skills.length === 0) {+ return {+ content: [+ {+ type: 'text',+ text: 'No skills found. Use skill_record to record a skill from a sandbox.',+ },+ ],+ };+ }+ return {+ content: [+ {+ type: 'text',+ text: skills.join('\n'),+ },+ ],+ };+ }+ );+ }++ if (permissions.has('skill_read')) {+ server.tool(+ 'skill_read',+ 'Read a skill and get its skill_prompt for guiding agent behavior. Returns all trajectory file paths followed by the SKILL.md content.',+ {+ name: z.string().describe('Name of the skill to read'),+ },+ async ({ name }) => {+ const content = await readSkill(name);+ if (!content) {+ return {+ content: [{ type: 'text', text: `Skill not found: ${name}` }],+ };+ }+ return {+ content: [+ {+ type: 'text',+ text: content,+ },+ ],+ };+ }+ );+ }++ if (permissions.has('skill_record')) {+ server.tool(+ 'skill_record',+ 'Record a human demonstration from a sandbox and save it as a reusable skill. Opens the VNC interface in a browser for recording. The user performs the task, then clicks Stop Recording. The recording is automatically processed with VLM captions and saved.',+ {+ sandbox: z.string().describe('Name of the sandbox to record from'),+ name: z+ .string()+ .regex(/^[a-z0-9_-]+$/)+ .describe(+ 'Unique name for the skill (lowercase, hyphens, underscores only)'+ ),+ description: z+ .string()+ .describe(+ 'Human-readable description of what this skill demonstrates'+ ),+ },+ async ({ sandbox, name, description }) => {+ // Invoke the cua skills record CLI command+ return new Promise((resolve) => {+ const args = [+ 'skills',+ 'record',+ '--sandbox',+ sandbox,+ '--name',+ name,+ '--description',+ description,+ ];++ const proc = spawn('cua', args, {+ stdio: ['inherit', 'pipe', 'pipe'],+ env: { ...process.env },+ });++ let stdout = '';+ let stderr = '';++ proc.stdout?.on('data', (data) => {+ stdout += data.toString();+ });++ proc.stderr?.on('data', (data) => {+ stderr += data.toString();+ });++ proc.on('close', (code) => {+ if (code !== 0) {+ resolve({+ content: [+ {+ type: 'text',+ text: `Recording failed (exit ${code}):\n${stderr || stdout}`,+ },+ ],+ });+ } else {+ const skillDir = getSkillDir(name);+ resolve({+ content: [+ {+ type: 'text',+ text: `Skill recorded successfully: ${name}\nDirectory: ${skillDir}\n\n${stdout}`,+ },+ ],+ });+ }+ });++ proc.on('error', (err) => {+ resolve({+ content: [+ {+ type: 'text',+ text: `Failed to run cua skills record: ${err.message}`,+ },+ ],+ });+ });+ });+ }+ );+ }++ if (permissions.has('skill_delete')) {+ server.tool(+ 'skill_delete',+ 'Delete a skill and all its trajectory files',+ {+ name: z.string().describe('Name of the skill to delete'),+ },+ async ({ name }) => {+ const fs = await import('fs/promises');+ const skillDir = getSkillDir(name);++ try {+ await fs.access(join(skillDir, 'SKILL.md'));+ } catch {+ return {+ content: [{ type: 'text', text: `Skill not found: ${name}` }],+ };+ }++ await fs.rm(skillDir, { recursive: true, force: true });++ return {+ content: [+ {+ type: 'text',+ text: `Skill deleted: ${name}`,+ },+ ],+ };+ }+ );+ }++ // Connect via stdio transport+ const transport = new StdioServerTransport();+ await server.connect(transport);+}++// Command handler+const serveMcpHandler = async (argv: Record<string, unknown>) => {+ const permEnv = argv.permissions as string | undefined;+ const permissions = parsePermissions(+ permEnv || process.env.CUA_MCP_PERMISSIONS+ );++ await runMcpServer(permissions);+};++// Register command+export function registerServeMcpCommands(y: Argv) {+ y.command(+ 'serve-mcp',+ 'Start an MCP server for Claude Code integration',+ (yargs) =>+ yargs+ .option('permissions', {+ alias: 'p',+ type: 'string',+ description:+ 'Comma-separated list of allowed permissions (default: all). ' ++ 'Available: list_sandboxes, create_sandbox, delete_sandbox, start_sandbox, ' ++ 'stop_sandbox, restart_sandbox, suspend_sandbox, get_sandbox, ' ++ 'computer_screenshot, computer_click, computer_type, computer_key, ' ++ 'computer_scroll, computer_drag, computer_hotkey, computer_clipboard, ' ++ 'computer_file, computer_shell, computer_window, ' ++ 'skill_list, skill_read, skill_record, skill_delete',+ })+ .example(+ 'claude mcp add cua-server -- cua serve-mcp',+ 'Add CUA as an MCP server to Claude Code'+ )+ .example(+ 'claude mcp add -e CUA_MCP_PERMISSIONS=list_sandboxes,computer_screenshot cua-server -- cua serve-mcp',+ 'Add with limited permissions'+ ),+ serveMcpHandler+ );++ return y;+}diff --git a/libs/typescript/cua-cli/src/commands/skills.ts b/libs/typescript/cua-cli/src/commands/skills.tsnew file mode 100644index 0000000000..ba1c70f878--- /dev/null+++ b/libs/typescript/cua-cli/src/commands/skills.ts@@ -0,0 +1,1532 @@+import type { Argv } from 'yargs';+import { ensureApiKeyInteractive } from '../auth';+import { http } from '../http';+import { clearApiKey } from '../storage';+import type { SandboxItem } from '../util';+import { openInBrowser } from '../util';+import * as readline from 'readline';++const SKILLS_DIR = `${process.env.HOME || process.env.USERPROFILE}/.cua/skills`;++// Helper to ensure skills directory exists+async function ensureSkillsDir(): Promise<void> {+ const fs = await import('fs/promises');+ try {+ await fs.mkdir(SKILLS_DIR, { recursive: true });+ } catch {+ // Directory might already exist+ }+}++// Helper to prompt for user input+function prompt(question: string): Promise<string> {+ const rl = readline.createInterface({+ input: process.stdin,+ output: process.stdout,+ });++ return new Promise((resolve) => {+ rl.question(question, (answer) => {+ rl.close();+ resolve(answer.trim());+ });+ });+}++// Helper to get sandbox VNC URL+async function getSandboxVncUrl(+ name: string,+ token: string+): Promise<{ url: string; host: string; password: string }> {+ const listRes = await http('/v1/vms', { token });+ if (listRes.status === 401) {+ clearApiKey();+ console.error("Unauthorized. Try 'cua login' again.");+ process.exit(1);+ }+ if (!listRes.ok) {+ console.error(`Request failed: ${listRes.status}`);+ process.exit(1);+ }++ const sandboxes = (await listRes.json()) as SandboxItem[];+ const sandbox = sandboxes.find((s) => s.name === name);++ if (!sandbox) {+ console.error('Sandbox not found');+ process.exit(1);+ }++ if (sandbox.status !== 'running') {+ console.error(`Sandbox is not running (status: ${sandbox.status})`);+ process.exit(1);+ }++ const host = sandbox.host || `${sandbox.name}.sandbox.cua.ai`;+ const url = `https://${host}/vnc.html?autoconnect=true&password=${encodeURIComponent(sandbox.password)}&show_dot=true`;++ return { url, host, password: sandbox.password };+}++// Recording format: JSON with events + optional MP4 video+// Format sent over WebSocket:+// [4 bytes JSON length (big-endian)][JSON data][optional MP4 data]+// JSON structure: { events: InputEvent[], metadata: {...} }++interface InputEvent {+ timestamp: number;+ type: string; // 'click', 'type', 'key', 'drag', 'scroll', 'paste'+ data: Record<string, unknown>;+}++interface RecordingData {+ events: InputEvent[];+ metadata?: {+ width?: number;+ height?: number;+ duration?: number;+ startTime?: number;+ };+}++interface TrajectoryStep {+ step_idx: number;+ caption: {+ observation: string;+ think: string;+ action: string;+ expectation: string;+ };+ raw_event: {+ timestamp: number;+ type: string;+ data: Record<string, unknown>;+ };+ screenshot_full?: string;+ screenshot_crop?: string;+}++interface Skill {+ name: string;+ description: string;+ trajectory: TrajectoryStep[];+ skill_prompt?: string;+ video_path?: string;+ metadata?: {+ recording_file?: string;+ task_description?: string;+ total_steps?: number;+ width?: number;+ height?: number;+ duration?: number;+ created_at?: string;+ };+}++// Default prompt for trajectory captioning (adapted from Aloha_Learn)+const DEFAULT_PROMPT = {+ basePrompt: `Objective:+You will describe a single GUI action using two images per step:+1) a ZOOMED-IN CROP (primary focus), and+2) a FULL-SCREEN image providing context.++Crop semantics:+• Click / Select → the crop has a red "X" marker centered on the target control. This marker is an annotation overlay—ignore it visually and describe only the UI element underneath.+• Scroll/Hotkey/Type → no marker; the crop frames the entire screen.+• Drag → the crop has a red line/path showing the drag route. This is an annotation—describe only the UI elements at the start and end points.++High-Priority Rules:+• NEVER mention the red X, red marker, red path, or any annotation overlay in your output. These are visual guides only—describe the actual UI elements they point to.+• Read the crop first, then the full screen.+• Never include coordinates, file paths, or timestamps.+• Observation must describe what is visually present—icons, text, glyphs, color, shape—and must be complete.+• Infer the actual name of the icon if possible. Describe the icon such that an operator with no prior knowledge can identify and select it correctly.+• For clicks: identify the UI element at the center of the crop (where the marker points).+• For drags: describe the start element, direction, and destination element.+• Respond strictly in JSON; no extra prose.++Output Fields:+{+ "Observation": "Describe the UI element being targeted, then relevant full-screen context. Do NOT mention any red markers or overlays.",+ "Think": "Explain the user's likely intention based on Observation.",+ "Action": "Concrete operation: click / type / drag, etc.",+ "Expectation": "What should happen immediately after the action."+}`,++ actionDeltas: {+ click:+ 'Identify the UI control at the center of the crop (toolbar button, menu item, canvas object, text field, etc.). Do not mention any visual markers.',+ type: 'Describe the visible text entry area and what is being typed. The operation is typing, not clicking.',+ drag: 'Describe the start element, direction, and destination element. Action must be: "Click-and-hold on <start>, drag to <destination>, then release."',+ scroll:+ 'Determine direction and purpose. Up usually moves view up; down moves view down.',+ key: 'Describe the special key being pressed and its likely purpose.',+ },+};++// Parse recording data (JSON header + optional MP4)+function parseRecordingData(data: Uint8Array): {+ recording: RecordingData;+ mp4Data?: Uint8Array;+} {+ if (data.length < 4) {+ throw new Error('Recording data too short');+ }++ // Read JSON length (4 bytes, big-endian)+ const jsonLength =+ ((data[0] ?? 0) << 24) |+ ((data[1] ?? 0) << 16) |+ ((data[2] ?? 0) << 8) |+ (data[3] ?? 0);++ if (data.length < 4 + jsonLength) {+ throw new Error(+ `Invalid recording: expected ${jsonLength} bytes of JSON, got ${data.length - 4}`+ );+ }++ // Parse JSON+ const jsonBytes = data.slice(4, 4 + jsonLength);+ const jsonStr = new TextDecoder().decode(jsonBytes);+ const recording = JSON.parse(jsonStr) as RecordingData;++ // Extract MP4 data if present+ let mp4Data: Uint8Array | undefined;+ if (data.length > 4 + jsonLength) {+ mp4Data = data.slice(4 + jsonLength);+ }++ return { recording, mp4Data };+}++// Extract a frame from MP4 video at a specific timestamp using ffmpeg+async function extractFrame(+ videoPath: string,+ timestampMs: number,+ outputPath: string+): Promise<boolean> {+ const timestampSec = Math.max(0, timestampMs / 1000 - 0.1); // Slightly before the action+ const proc = Bun.spawn({+ cmd: [+ 'ffmpeg',+ '-y',+ '-ss',+ timestampSec.toFixed(3),+ '-i',+ videoPath,+ '-frames:v',+ '1',+ '-q:v',+ '2',+ outputPath,+ ],+ stdout: 'ignore',+ stderr: 'ignore',+ });+ const exitCode = await proc.exited;+ return exitCode === 0;+}++// Create a cropped screenshot with optional red X marker+async function createCropWithMarker(+ fullImagePath: string,+ cropPath: string,+ x: number | undefined,+ y: number | undefined,+ cropSize: number = 256,+ actionType: string = 'click'+): Promise<boolean> {+ // For types without coordinates, just copy the full image+ if (+ x === undefined ||+ y === undefined ||+ actionType === 'type' ||+ actionType === 'key' ||+ actionType === 'scroll'+ ) {+ const proc = Bun.spawn({+ cmd: ['cp', fullImagePath, cropPath],+ stdout: 'ignore',+ stderr: 'ignore',+ });+ return (await proc.exited) === 0;+ }++ // Use ImageMagick to create crop with red X marker+ const halfSize = Math.floor(cropSize / 2);+ const xSize = 15; // Half-length of X arms+ const thickness = 3;++ // Create crop centered on x,y with red X overlay+ const proc = Bun.spawn({+ cmd: [+ 'convert',+ fullImagePath,+ // Crop region+ '-crop',+ `${cropSize}x${cropSize}+${Math.max(0, x - halfSize)}+${Math.max(0, y - halfSize)}`,+ '+repage',+ // Draw red X at center+ '-fill',+ 'none',+ '-stroke',+ 'red',+ '-strokewidth',+ String(thickness),+ '-draw',+ `line ${halfSize - xSize},${halfSize - xSize} ${halfSize + xSize},${halfSize + xSize}`,+ '-draw',+ `line ${halfSize - xSize},${halfSize + xSize} ${halfSize + xSize},${halfSize - xSize}`,+ // Add semi-transparency to the X+ cropPath,+ ],+ stdout: 'ignore',+ stderr: 'ignore',+ });++ return (await proc.exited) === 0;+}++// Progress bar helper+function renderProgressBar(+ currentStep: number,+ totalSteps: number,+ tokens: number,+ barWidth: number = 20+): string {+ const progress = currentStep / totalSteps;+ const filled = Math.round(progress * barWidth);+ const empty = barWidth - filled;+ const bar = '█'.repeat(filled) + '░'.repeat(empty);+ const tokenStr = tokens.toLocaleString();+ return `[${bar}] ${currentStep}/${totalSteps} │ ${tokenStr} tokens`;+}++// Streaming callback type+type TokenCallback = (tokens: number) => void;++// Call Anthropic API for captioning with streaming+async function callAnthropic(+ prompt: string,+ cropImagePath: string | null,+ fullImagePath: string | null,+ model: string,+ apiKey: string,+ onToken?: TokenCallback+): Promise<string> {+ const content: Array<{+ type: string;+ text?: string;+ source?: { type: string; media_type: string; data: string };+ }> = [{ type: 'text', text: prompt }];++ // Add cropped image first (primary focus)+ if (cropImagePath) {+ try {+ const cropData = await Bun.file(cropImagePath).arrayBuffer();+ content.push({+ type: 'image',+ source: {+ type: 'base64',+ media_type: 'image/jpeg',+ data: Buffer.from(cropData).toString('base64'),+ },+ });+ } catch {+ // Skip if image doesn't exist+ }+ }++ // Add full screenshot for context+ if (fullImagePath) {+ try {+ const fullData = await Bun.file(fullImagePath).arrayBuffer();+ content.push({+ type: 'image',+ source: {+ type: 'base64',+ media_type: 'image/jpeg',+ data: Buffer.from(fullData).toString('base64'),+ },+ });+ } catch {+ // Skip if image doesn't exist+ }+ }++ const response = await fetch('https://api.anthropic.com/v1/messages', {+ method: 'POST',+ headers: {+ 'x-api-key': apiKey,+ 'anthropic-version': '2023-06-01',+ 'content-type': 'application/json',+ },+ body: JSON.stringify({+ model,+ max_tokens: 1200,+ stream: true,+ messages: [{ role: 'user', content }],+ }),+ });++ if (!response.ok) {+ const error = await response.text();+ throw new Error(`Anthropic API error: ${response.status} - ${error}`);+ }++ // Process SSE stream+ const reader = response.body?.getReader();+ if (!reader) {+ throw new Error('No response body');+ }++ const decoder = new TextDecoder();+ let fullText = '';+ let tokenCount = 0;+ let buffer = '';++ while (true) {+ const { done, value } = await reader.read();+ if (done) break;++ buffer += decoder.decode(value, { stream: true });+ const lines = buffer.split('\n');+ buffer = lines.pop() || '';++ for (const line of lines) {+ if (!line.startsWith('data: ')) continue;+ const data = line.slice(6);+ if (data === '[DONE]') continue;++ try {+ const event = JSON.parse(data);+ if (event.type === 'content_block_delta' && event.delta?.text) {+ fullText += event.delta.text;+ // Estimate tokens (roughly 4 chars per token)+ const newTokens = Math.ceil(event.delta.text.length / 4);+ tokenCount += newTokens;+ onToken?.(tokenCount);+ } else if (+ event.type === 'message_delta' &&+ event.usage?.output_tokens+ ) {+ // Use actual token count if available+ tokenCount = event.usage.output_tokens;+ onToken?.(tokenCount);+ }+ } catch {+ // Ignore parse errors+ }+ }+ }++ return fullText;+}++// Call OpenAI API for captioning with streaming+async function callOpenAI(+ prompt: string,+ cropImagePath: string | null,+ fullImagePath: string | null,+ model: string,+ apiKey: string,+ onToken?: TokenCallback+): Promise<string> {+ const content: Array<{+ type: string;+ text?: string;+ image_url?: { url: string };+ }> = [{ type: 'text', text: prompt }];++ // Add cropped image first (primary focus)+ if (cropImagePath) {+ try {+ const cropData = await Bun.file(cropImagePath).arrayBuffer();+ content.push({+ type: 'image_url',+ image_url: {+ url: `data:image/jpeg;base64,${Buffer.from(cropData).toString('base64')}`,+ },+ });+ } catch {+ // Skip if image doesn't exist+ }+ }++ // Add full screenshot for context+ if (fullImagePath) {+ try {+ const fullData = await Bun.file(fullImagePath).arrayBuffer();+ content.push({+ type: 'image_url',+ image_url: {+ url: `data:image/jpeg;base64,${Buffer.from(fullData).toString('base64')}`,+ },+ });+ } catch {+ // Skip if image doesn't exist+ }+ }++ const response = await fetch('https://api.openai.com/v1/chat/completions', {+ method: 'POST',+ headers: {+ Authorization: `Bearer ${apiKey}`,+ 'Content-Type': 'application/json',+ },+ body: JSON.stringify({+ model,+ messages: [{ role: 'user', content }],+ temperature: 0.2,+ stream: true,+ }),+ });++ if (!response.ok) {+ const error = await response.text();+ throw new Error(`OpenAI API error: ${response.status} - ${error}`);+ }++ // Process SSE stream+ const reader = response.body?.getReader();+ if (!reader) {+ throw new Error('No response body');+ }++ const decoder = new TextDecoder();+ let fullText = '';+ let tokenCount = 0;+ let buffer = '';++ while (true) {+ const { done, value } = await reader.read();+ if (done) break;++ buffer += decoder.decode(value, { stream: true });+ const lines = buffer.split('\n');+ buffer = lines.pop() || '';++ for (const line of lines) {+ if (!line.startsWith('data: ')) continue;+ const data = line.slice(6);+ if (data === '[DONE]') continue;++ try {+ const event = JSON.parse(data);+ const delta = event.choices?.[0]?.delta?.content;+ if (delta) {+ fullText += delta;+ // Estimate tokens (roughly 4 chars per token)+ const newTokens = Math.ceil(delta.length / 4);+ tokenCount += newTokens;+ onToken?.(tokenCount);+ }+ } catch {+ // Ignore parse errors+ }+ }+ }++ return fullText;+}++// Extract JSON from LLM response+function extractJson(text: string): Record<string, string> {+ if (!text) return {};++ // Try to find JSON block+ const jsonMatch = text.match(/\{[\s\S]*\}/);+ if (jsonMatch) {+ try {+ return JSON.parse(jsonMatch[0]);+ } catch {+ // Continue to fallback+ }+ }++ // Fallback: try parsing the whole text+ try {+ return JSON.parse(text);+ } catch {+ return {};+ }+}++// Sanitize caption output+function sanitizeCaption(cap: Record<string, string>): {+ observation: string;+ think: string;+ action: string;+ expectation: string;+} {+ // Remove coordinates from output+ const scrub = (s: string): string => {+ if (!s) return '';+ return s+ .replace(/\bcoordinates?\b[^.]*\[[^\]]*\]/gi, '')+ .replace(/\[\s*\d+\s*,\s*\d+\s*\]/g, '')+ .replace(/\b(x\s*[:=]?\s*\d+|y\s*[:=]?\s*\d+)/gi, '')+ .replace(/\s{2,}/g, ' ')+ .trim();+ };++ let observation = scrub(cap.Observation || cap.observation || '');+ // Ensure observation starts with "Cropped image shows"+ if (+ observation &&+ !observation.toLowerCase().startsWith('cropped image shows')+ ) {+ observation = 'Cropped image shows ' + observation;+ }++ return {+ observation,+ think: scrub(cap.Think || cap.think || ''),+ action: scrub(cap.Action || cap.action || ''),+ expectation: scrub(cap.Expectation || cap.expectation || ''),+ };+}++// Build prompt for a step+function buildStepPrompt(+ event: InputEvent,+ stepIdx: number,+ recentSteps: Array<{ step_idx: number; caption: Record<string, string> }>,+ taskDescription: string+): string {+ const actionType = event.type.toLowerCase();+ const delta =+ DEFAULT_PROMPT.actionDeltas[+ actionType as keyof typeof DEFAULT_PROMPT.actionDeltas+ ] || '';++ const recentContext = recentSteps.slice(-3).map((s) => ({+ step_idx: s.step_idx,+ Observation: s.caption.observation || '',+ Think: s.caption.think || '',+ Action: s.caption.action || '',+ Expectation: s.caption.expectation || '',+ }));++ return `${DEFAULT_PROMPT.basePrompt}++Recent Steps (most recent first, up to 3):+${JSON.stringify(recentContext, null, 2)}++Step Index: ${stepIdx}+Action Type: ${event.type}+Action Data: ${JSON.stringify(event.data)}+Timestamp: ${event.timestamp}ms+Overall Task: ${taskDescription}++${delta ? `ActionTypeDelta: ${delta}` : ''}++Respond with JSON only.`;+}++// Process recording with LLM captioning+async function processRecordingWithLLM(+ data: Uint8Array,+ taskDescription: string,+ skillName: string,+ options: {+ provider: 'anthropic' | 'openai';+ model: string;+ apiKey: string;+ }+): Promise<{+ trajectory: TrajectoryStep[];+ skillPrompt: string;+ trajectoryDir: string;+ metadata: Skill['metadata'];+}> {+ const { recording, mp4Data } = parseRecordingData(data);+ const inputEvents = recording.events || [];++ // Create timestamp for trajectory folder (format: YYYYMMDD-HHMMSS)+ const now = new Date();+ const timestamp = now+ .toISOString()+ .replace(/[-:]/g, '')+ .replace('T', '-')+ .slice(0, 15);++ // Save MP4 video and other assets to {skillName}/trajectory/+ const trajectoryDir = `${SKILLS_DIR}/${skillName}/trajectory`;+ const tempDir = `${SKILLS_DIR}/.temp/${skillName}`;+ const fs = await import('fs/promises');+ await fs.mkdir(trajectoryDir, { recursive: true });+ await fs.mkdir(tempDir, { recursive: true });++ if (!mp4Data || mp4Data.length === 0) {+ throw new Error(+ 'No video data in recording. Video is required for trajectory captioning.'+ );+ }++ await ensureSkillsDir();+ const videoPath = `${trajectoryDir}/${skillName}.mp4`;+ await Bun.write(videoPath, mp4Data);++ // Save raw events JSON+ const eventsPath = `${trajectoryDir}/events.json`;+ await Bun.write(+ eventsPath,+ JSON.stringify(+ {+ events: inputEvents,+ metadata: recording.metadata,+ },+ null,+ 2+ )+ );++ const trajectory: TrajectoryStep[] = [];+ const recentSteps: Array<{+ step_idx: number;+ caption: Record<string, string>;+ }> = [];++ // Process each event+ const totalEvents = inputEvents.length;+ let totalTokens = 0;++ // Helper to update progress display+ const updateProgress = (currentStep: number, tokens: number) => {+ const progressBar = renderProgressBar(currentStep, totalEvents, tokens);+ process.stdout.write(`\r${progressBar}`);+ };++ for (let idx = 0; idx < inputEvents.length; idx++) {+ const event = inputEvents[idx];+ if (!event) continue;+ const stepIdx = idx + 1;++ // Show initial progress for this step+ updateProgress(stepIdx, totalTokens);++ let caption = {+ observation: '',+ think: '',+ action: '',+ expectation: '',+ };++ let screenshotFull: string | undefined;+ let screenshotCrop: string | undefined;++ const fullPath = `${tempDir}/step_${stepIdx}_full.jpg`;+ const cropPath = `${tempDir}/step_${stepIdx}_crop.jpg`;++ // Extract frame at event timestamp+ const extracted = await extractFrame(videoPath, event.timestamp, fullPath);++ if (!extracted) {+ throw new Error(+ `Failed to extract frame for step ${stepIdx} at timestamp ${event.timestamp}ms`+ );+ }++ // Get coordinates if available+ const eventData = event.data as { x?: number; y?: number };+ const x = eventData.x;+ const y = eventData.y;++ // Create cropped version with marker+ await createCropWithMarker(fullPath, cropPath, x, y, 256, event.type);++ // Build prompt and call LLM+ const stepPrompt = buildStepPrompt(+ event,+ stepIdx,+ recentSteps,+ taskDescription+ );++ // Token callback to update progress in real-time+ let stepTokens = 0;+ const onToken: TokenCallback = (tokens: number) => {+ stepTokens = tokens;+ updateProgress(stepIdx, totalTokens + stepTokens);+ };++ let response: string;+ if (options.provider === 'openai') {+ response = await callOpenAI(+ stepPrompt,+ cropPath,+ fullPath,+ options.model,+ options.apiKey,+ onToken+ );+ } else {+ response = await callAnthropic(+ stepPrompt,+ cropPath,+ fullPath,+ options.model,+ options.apiKey,+ onToken+ );+ }++ // Add this step's tokens to total+ totalTokens += stepTokens;++ const parsed = extractJson(response);+ caption = sanitizeCaption(parsed);++ // Copy screenshots to trajectory dir+ try {+ screenshotFull = `${trajectoryDir}/step_${stepIdx}_full.jpg`;+ screenshotCrop = `${trajectoryDir}/step_${stepIdx}_crop.jpg`;+ await fs.copyFile(fullPath, screenshotFull);+ await fs.copyFile(cropPath, screenshotCrop);+ } catch {+ // Ignore screenshot copy errors+ }++ const step: TrajectoryStep = {+ step_idx: stepIdx,+ caption,+ raw_event: {+ timestamp: event.timestamp,+ type: event.type,+ data: event.data,+ },+ screenshot_full: screenshotFull,+ screenshot_crop: screenshotCrop,+ };++ trajectory.push(step);+ recentSteps.push({ step_idx: stepIdx, caption });++ // Small delay to avoid rate limiting+ await new Promise((r) => setTimeout(r, 100));+ }++ // Show completed progress bar+ updateProgress(totalEvents, totalTokens);+ process.stdout.write('\n');++ // Clean up temp directory+ try {+ await fs.rm(tempDir, { recursive: true });+ } catch {+ // Ignore cleanup errors+ }++ // Generate skill prompt+ const stepsText = trajectory+ .map(+ (step) =>+ `Step ${step.step_idx}: ${step.caption.action || step.raw_event.type}`+ )+ .join('\n');++ const actionTypes = [...new Set(trajectory.map((s) => s.raw_event.type))];++ const skillPrompt = `You have been shown a demonstration of how to perform this task:+${taskDescription}++The demonstration consisted of the following steps:+${stepsText}++Follow this workflow pattern, adapting as needed for the current screen state.+Key observations from the demonstration:+- Total steps: ${trajectory.length}+- Main actions: ${actionTypes.join(', ')}++Execute each step while monitoring for expected outcomes before proceeding.`;++ // Save trajectory JSON to trajectory dir+ const trajectoryJsonPath = `${trajectoryDir}/trajectory.json`;+ await Bun.write(+ trajectoryJsonPath,+ JSON.stringify(+ {+ events: inputEvents,+ trajectory,+ metadata: {+ task_description: taskDescription,+ total_steps: trajectory.length,+ width: recording.metadata?.width,+ height: recording.metadata?.height,+ duration: recording.metadata?.duration,+ created_at: new Date().toISOString(),+ },+ },+ null,+ 2+ )+ );++ return {+ trajectory,+ skillPrompt,+ trajectoryDir,+ metadata: {+ task_description: taskDescription,+ total_steps: trajectory.length,+ width: recording.metadata?.width,+ height: recording.metadata?.height,+ duration: recording.metadata?.duration,+ created_at: new Date().toISOString(),+ },+ };+}++// Create WebSocket server to receive recording+async function startRecordingServer(): Promise<{+ port: number;+ waitForRecording: () => Promise<Uint8Array>;+ stop: () => void;+}> {+ let resolveRecording: (data: Uint8Array) => void;+ const recordingPromise = new Promise<Uint8Array>((resolve) => {+ resolveRecording = resolve;+ });++ const chunks: Uint8Array[] = [];++ const server = Bun.serve({+ port: 0, // Let OS pick an available port+ fetch(req, server) {+ // Upgrade to WebSocket+ if (server.upgrade(req)) {+ return;+ }+ return new Response('WebSocket upgrade required', { status: 426 });+ },+ websocket: {+ message(_ws, message) {+ if (message instanceof ArrayBuffer) {+ chunks.push(new Uint8Array(message));+ } else if (message instanceof Uint8Array) {+ chunks.push(message);+ } else if (Buffer.isBuffer(message)) {+ chunks.push(new Uint8Array(message));+ } else if (typeof message === 'string') {+ const encoder = new TextEncoder();+ chunks.push(encoder.encode(message));+ } else {+ try {+ chunks.push(new Uint8Array(message as unknown as ArrayBuffer));+ } catch {+ // Ignore conversion errors+ }+ }+ },+ close() {+ const totalLength = chunks.reduce(+ (sum, chunk) => sum + chunk.length,+ 0+ );+ const combined = new Uint8Array(totalLength);+ let offset = 0;+ for (const chunk of chunks) {+ combined.set(chunk, offset);+ offset += chunk.length;+ }+ resolveRecording(combined);+ },+ open() {},+ },+ });++ return {+ port: server.port as number,+ waitForRecording: () => recordingPromise,+ stop: () => server.stop(),+ };+}++// Check if a command is available+async function checkCommand(cmd: string): Promise<boolean> {+ try {+ const proc = Bun.spawn({+ cmd: ['which', cmd],+ stdout: 'ignore',+ stderr: 'ignore',+ });+ return (await proc.exited) === 0;+ } catch {+ return false;+ }+}++// Check required dependencies+async function checkDependencies(): Promise<void> {+ const missing: string[] = [];++ if (!(await checkCommand('ffmpeg'))) {+ missing.push('ffmpeg');+ }+ if (!(await checkCommand('convert'))) {+ missing.push('convert (ImageMagick)');+ }++ if (missing.length > 0) {+ console.error('Error: Missing required dependencies:');+ for (const dep of missing) {+ console.error(` - ${dep}`);+ }+ console.error('\nInstall with:');+ if (process.platform === 'darwin') {+ console.error(' brew install ffmpeg imagemagick');+ } else {+ console.error(' apt-get install ffmpeg imagemagick # Debian/Ubuntu');+ console.error(' # or your package manager equivalent');+ }+ process.exit(1);+ }+}++// Command handlers+const recordHandler = async (argv: Record<string, unknown>) => {+ // Check dependencies first+ await checkDependencies();++ const sandbox = argv.sandbox as string | undefined;+ const vncUrl = argv['vnc-url'] as string | undefined;+ const provider = ((argv.provider as string) || 'anthropic').toLowerCase() as+ | 'anthropic'+ | 'openai';+ const model =+ (argv.model as string) ||+ (provider === 'openai' ? 'gpt-5-mini' : 'claude-haiku-4-5');+ const apiKeyArg = argv['api-key'] as string | undefined;++ if (!sandbox && !vncUrl) {+ console.error('Either --sandbox or --vnc-url is required');+ process.exit(1);+ }++ // Get API key+ let apiKey = apiKeyArg;+ if (!apiKey) {+ if (provider === 'openai') {+ apiKey = process.env.OPENAI_API_KEY;+ } else {+ apiKey = process.env.ANTHROPIC_API_KEY;+ }+ }++ if (!apiKey) {+ const envVar =+ provider === 'openai' ? 'OPENAI_API_KEY' : 'ANTHROPIC_API_KEY';+ console.error(`Error: No ${provider.toUpperCase()} API key found.`);+ console.error(`Set ${envVar} environment variable or use --api-key flag.`);+ process.exit(1);+ }++ // Start recording server+ console.log('Starting recording server...');+ const { port, waitForRecording, stop } = await startRecordingServer();+ const recordUrl = `ws://localhost:${port}`;++ let recordingUrl: string;++ if (sandbox) {+ const token = await ensureApiKeyInteractive();+ const { url } = await getSandboxVncUrl(sandbox, token);+ const urlObj = new URL(url);+ urlObj.searchParams.set('autorecord', 'true');+ urlObj.searchParams.set('record_format', 'mp4');+ urlObj.searchParams.set('record_url', recordUrl);+ recordingUrl = urlObj.toString();+ } else {+ const urlObj = new URL(vncUrl!);+ urlObj.searchParams.set('autorecord', 'true');+ urlObj.searchParams.set('record_format', 'mp4');+ urlObj.searchParams.set('record_url', recordUrl);+ recordingUrl = urlObj.toString();+ }++ console.log('\nRecording will start automatically when you connect.');+ console.log('When finished, click "Stop Recording" in the VNC panel.\n');++ openInBrowser(recordingUrl);++ const timeoutMs = 30 * 60 * 1000;+ let recordingData: Uint8Array;++ try {+ const timeout = new Promise<never>((_, reject) =>+ setTimeout(+ () => reject(new Error('Recording timeout (30 minutes)')),+ timeoutMs+ )+ );+ recordingData = await Promise.race([waitForRecording(), timeout]);+ } catch (err) {+ stop();+ console.error(`Recording failed: ${err}`);+ process.exit(1);+ }++ stop();++ if (recordingData.length === 0) {+ console.error('No recording data received');+ process.exit(1);+ }++ // Get skill name (from flag or prompt)+ let skillName = (argv.name as string) || '';+ while (!skillName) {+ skillName = await prompt('Enter skill name: ');+ if (!skillName) {+ console.log('Skill name is required.');+ continue;+ }+ if (!/^[a-zA-Z0-9_-]+$/.test(skillName)) {+ console.log('Use only letters, numbers, hyphens, and underscores.');+ skillName = '';+ }+ }++ // Check for existing skill and deduplicate name+ await ensureSkillsDir();+ const fs = await import('fs/promises');+ let finalSkillName = skillName;+ let counter = 1;+ while (true) {+ try {+ await fs.access(`${SKILLS_DIR}/${finalSkillName}`);+ finalSkillName = `${skillName}-${counter}`;+ counter++;+ } catch {+ break; // Directory doesn't exist, we can use this name+ }+ }+ if (finalSkillName !== skillName) {+ console.log(`Skill "${skillName}" exists, using "${finalSkillName}"`);+ }+ skillName = finalSkillName;++ // Get skill description (from flag or prompt)+ let description = (argv.description as string) || '';+ while (!description) {+ description = await prompt('Describe what this skill demonstrates: ');+ if (!description) {+ console.log('Description is required.');+ }+ }++ console.log('\nProcessing recording...');++ const { trajectory, skillPrompt, trajectoryDir } =+ await processRecordingWithLLM(recordingData, description, skillName, {+ provider,+ model,+ apiKey,+ });++ if (trajectory.length === 0) {+ console.log('Warning: No input events captured.');+ }++ // Generate step descriptions from captions+ const stepsMarkdown = trajectory+ .map((step, idx) => {+ const caption = step.caption;+ const event = step.raw_event;+ const lines: string[] = [];++ lines.push(`### Step ${idx + 1}: ${caption.action || event.type}`);+ lines.push('');++ if (caption.observation) {+ // Trim "Cropped image shows" prefix if present+ const obs = caption.observation.replace(/^Cropped image shows\s*/i, '');+ lines.push(`**Context:** ${obs}`);+ lines.push('');+ }++ if (caption.think) {+ lines.push(`**Intent:** ${caption.think}`);+ lines.push('');+ }++ if (caption.expectation) {+ lines.push(`**Expected Result:** ${caption.expectation}`);+ lines.push('');+ }++ return lines.join('\n');+ })+ .join('\n');++ // Create skill markdown file with frontmatter+ const skillDir = `${SKILLS_DIR}/${skillName}`;+ const skillPath = `${skillDir}/SKILL.md`;+ const skillContent = `---+name: ${skillName}+description: ${description}+---++# ${skillName}++${description}++## Steps++${stepsMarkdown}++## Agent Prompt++${skillPrompt}+`;+ await Bun.write(skillPath, skillContent);++ console.log(`\nSkill saved: ${skillPath}`);+ console.log(` Steps: ${trajectory.length}`);++ // Explicitly exit+ process.exit(0);+};++// Parse skill markdown frontmatter+function parseSkillFrontmatter(+ content: string+): { name: string; description: string; body: string } | null {+ const match = content.match(/^---\n([\s\S]*?)\n---\n([\s\S]*)$/);+ if (!match || !match[1] || match[2] === undefined) return null;++ const frontmatter = match[1];+ const body = match[2].trim();++ const nameMatch = frontmatter.match(/^name:\s*(.+)$/m);+ const descMatch = frontmatter.match(/^description:\s*(.+)$/m);++ if (!nameMatch?.[1] || !descMatch?.[1]) return null;++ return {+ name: nameMatch[1].trim(),+ description: descMatch[1].trim(),+ body,+ };+}++const listHandler = async () => {+ await ensureSkillsDir();++ const glob = new Bun.Glob('*/SKILL.md');+ const files: string[] = [];++ for await (const file of glob.scan({ cwd: SKILLS_DIR })) {+ files.push(file);+ }++ if (files.length === 0) {+ console.log('No skills found');+ console.log(`\nRecord a skill with: cua skills record --sandbox <name>`);+ return;+ }++ // Print skills table+ const headers = ['NAME', 'DESCRIPTION', 'STEPS', 'CREATED'];+ const rows: string[][] = [headers];++ for (const file of files.sort()) {+ try {+ const skillDir = file.replace('/SKILL.md', '');+ const content = await Bun.file(`${SKILLS_DIR}/${file}`).text();+ const parsed = parseSkillFrontmatter(content);+ if (!parsed) continue;++ // Try to read trajectory.json for steps count and created date+ let steps = 0;+ let created = '-';+ try {+ const trajContent = await Bun.file(+ `${SKILLS_DIR}/${skillDir}/trajectory/trajectory.json`+ ).text();+ const trajData = JSON.parse(trajContent);+ steps = trajData.trajectory?.length || 0;+ if (trajData.metadata?.created_at) {+ created = new Date(trajData.metadata.created_at).toLocaleDateString();+ }+ } catch {+ // Trajectory file might not exist+ }++ rows.push([+ parsed.name,+ parsed.description.substring(0, 40) ++ (parsed.description.length > 40 ? '...' : ''),+ String(steps),+ created,+ ]);+ } catch {+ // Skip malformed files+ }+ }++ // Calculate column widths+ const widths = headers.map((_, i) =>+ Math.max(...rows.map((r) => (r[i] || '').length))+ );++ // Print table+ for (const row of rows) {+ console.log(row.map((c, i) => c.padEnd(widths[i] || 0)).join(' '));+ }+};++const readHandler = async (argv: Record<string, unknown>) => {+ const name = String(argv.name);+ const format = (argv.format as string) || 'md';++ await ensureSkillsDir();++ const skillDir = `${SKILLS_DIR}/${name}`;+ const skillPath = `${skillDir}/SKILL.md`;+ const trajectoryDir = `${skillDir}/trajectory`;+ const file = Bun.file(skillPath);++ if (!(await file.exists())) {+ console.error(`Skill not found: ${name}`);+ console.log(`\nAvailable skills:`);+ await listHandler();+ process.exit(1);+ }++ const content = await file.text();+ const parsed = parseSkillFrontmatter(content);++ if (!parsed) {+ console.error(`Invalid skill file format: ${name}`);+ process.exit(1);+ }++ if (format === 'md') {+ // Just output the raw markdown file+ console.log(content);+ } else if (format === 'json') {+ // Read trajectory.json and combine with frontmatter+ try {+ const trajContent = await Bun.file(+ `${trajectoryDir}/trajectory.json`+ ).text();+ const trajData = JSON.parse(trajContent);+ const skill = {+ name: parsed.name,+ description: parsed.description,+ trajectory: trajData.trajectory || [],+ skill_prompt: parsed.body,+ trajectory_dir: trajectoryDir,+ metadata: trajData.metadata,+ };+ console.log(JSON.stringify(skill, null, 2));+ } catch {+ console.error(`Failed to read trajectory data from: ${trajectoryDir}`);+ process.exit(1);+ }+ } else {+ console.error(`Unknown format: ${format}`);+ process.exit(1);+ }+};++const deleteHandler = async (argv: Record<string, unknown>) => {+ const name = String(argv.name);++ await ensureSkillsDir();++ const skillDir = `${SKILLS_DIR}/${name}`;+ const skillPath = `${skillDir}/SKILL.md`;+ const file = Bun.file(skillPath);++ if (!(await file.exists())) {+ console.error(`Skill not found: ${name}`);+ process.exit(1);+ }++ const fs = await import('fs/promises');++ // Delete entire skill directory (contains SKILL.md and trajectory/)+ await fs.rm(skillDir, { recursive: true });++ console.log(`Deleted skill: ${name}`);+};++const replayHandler = async (argv: Record<string, unknown>) => {+ const name = String(argv.name);++ await ensureSkillsDir();++ const skillDir = `${SKILLS_DIR}/${name}`;+ const skillPath = `${skillDir}/SKILL.md`;+ const trajectoryDir = `${skillDir}/trajectory`;+ const file = Bun.file(skillPath);++ if (!(await file.exists())) {+ console.error(`Skill not found: ${name}`);+ process.exit(1);+ }++ // Find the .mp4 file in the trajectory directory+ const glob = new Bun.Glob('*.mp4');+ let mp4File: string | null = null;++ for await (const f of glob.scan({ cwd: trajectoryDir })) {+ mp4File = `${trajectoryDir}/${f}`;+ break;+ }++ if (!mp4File) {+ console.error(`No video found in: ${trajectoryDir}`);+ process.exit(1);+ }++ openInBrowser(`file://${mp4File}`);+};++const cleanHandler = async () => {+ await ensureSkillsDir();++ const glob = new Bun.Glob('*/SKILL.md');+ const files: string[] = [];++ for await (const file of glob.scan({ cwd: SKILLS_DIR })) {+ files.push(file);+ }++ if (files.length === 0) {+ console.log('No skills to clean.');+ return;+ }++ // List skills that will be deleted+ console.log(`\nSkills to delete:`);+ const skillDirs = files.map((f) => f.replace('/SKILL.md', '')).sort();+ for (const dir of skillDirs) {+ console.log(` - ${dir}`);+ }++ const answer = await prompt(+ `\nDelete ${skillDirs.length} skill${skillDirs.length > 1 ? 's' : ''}? y/[n]: `+ );++ if (answer.toLowerCase() !== 'y') {+ console.log('Cancelled.');+ return;+ }++ const fs = await import('fs/promises');++ for (const dir of skillDirs) {+ // Delete entire skill directory+ await fs.rm(`${SKILLS_DIR}/${dir}`, { recursive: true });+ }++ console.log(+ `Deleted ${skillDirs.length} skill${skillDirs.length > 1 ? 's' : ''}.`+ );+};++// Register commands+export function registerSkillsCommands(argv: Argv): Argv {+ return argv.command('skills', 'Manage demonstration skills', (yargs) =>+ yargs+ .command(+ 'record',+ 'Record a demonstration and create a skill',+ (y) =>+ y+ .option('sandbox', {+ alias: 's',+ type: 'string',+ description: 'Sandbox name to connect to',+ })+ .option('vnc-url', {+ alias: 'u',+ type: 'string',+ description: 'Direct VNC URL to connect to',+ })+ .option('provider', {+ alias: 'p',+ type: 'string',+ description: 'LLM provider for captioning (anthropic or openai)',+ default: 'anthropic',+ choices: ['anthropic', 'openai'],+ })+ .option('model', {+ alias: 'm',+ type: 'string',+ description: 'Model to use for captioning',+ })+ .option('api-key', {+ alias: 'k',+ type: 'string',+ description: 'API key for the LLM provider',+ })+ .option('name', {+ alias: 'n',+ type: 'string',+ description: 'Skill name (skips interactive prompt)',+ })+ .option('description', {+ alias: 'd',+ type: 'string',+ description: 'Skill description (skips interactive prompt)',+ }),+ recordHandler+ )+ .command('list', 'List all saved skills', {}, listHandler)+ .command('ls', false, {}, listHandler)+ .command(+ 'read <name>',+ 'Read a skill as JSON or Markdown',+ (y) =>+ y+ .positional('name', {+ type: 'string',+ description: 'Skill name',+ demandOption: true,+ })+ .option('format', {+ alias: 'f',+ type: 'string',+ description: 'Output format (json or md)',+ default: 'md',+ choices: ['json', 'md'],+ }),+ readHandler+ )+ .command(+ 'delete <name>',+ 'Delete a skill',+ (y) =>+ y.positional('name', {+ type: 'string',+ description: 'Skill name',+ demandOption: true,+ }),+ deleteHandler+ )+ .command(+ 'replay <name>',+ 'Open the video recording for a skill',+ (y) =>+ y.positional('name', {+ type: 'string',+ description: 'Skill name',+ demandOption: true,+ }),+ replayHandler+ )+ .command(+ 'clean',+ 'Delete all skills (with confirmation)',+ {},+ cleanHandler+ )+ .demandCommand(1)+ );+}diff --git a/libs/typescript/cua-cli/src/util.ts b/libs/typescript/cua-cli/src/util.tsindex 6014704929..41c4132033 100644--- a/libs/typescript/cua-cli/src/util.ts+++ b/libs/typescript/cua-cli/src/util.ts@@ -57,7 +57,7 @@ export function printSandboxList(if (items.length === 0) console.log('No sandboxes found');}-export async function openInBrowser(url: string) {+export function openInBrowser(url: string) {const platform = process.platform;let cmd: string;let args: string[] = [];@@ -72,7 +72,8 @@ export async function openInBrowser(url: string) {args = [url];}try {- await Bun.spawn({ cmd: [cmd, ...args] }).exited;+ // Don't wait for browser process - just spawn and let it run+ Bun.spawn({ cmd: [cmd, ...args], stdout: 'ignore', stderr: 'ignore' });} catch {console.error(`Failed to open browser. Please visit: ${url}`);}