Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-driver): document local model workflows (#3054)
Francesco Bonaccif415cc9ae942
diff --git a/docs/content/docs/concepts/what-is-computer-use.mdx b/docs/content/docs/concepts/what-is-computer-use.mdxindex a114865fba..a874d77712 100644--- a/docs/content/docs/concepts/what-is-computer-use.mdx+++ b/docs/content/docs/concepts/what-is-computer-use.mdx@@ -17,6 +17,22 @@ A **computer-use agent** is the running system around that model. It supplies thThe distinction matters because a capable model alone does not provide a reliable agent. The surrounding system still has to deliver input to the intended application, preserve state between steps, recover from changed interfaces, and constrain consequential actions.+An **agent harness** is the runtime that hosts this loop. A coding agent such as Claude Code or+Codex can supply instructions, memory, and MCP connections while using either a hosted model or a+model served on the same machine. Cua Driver is the UI tool layer connected to that harness; it does+not select or run the model.++```text+model -> agent harness -> Cua Driver -> operating system and applications+```++This separation lets the same driver work with different models and harnesses. It also makes the+cost of each layer visible. Local models often have less inference throughput and a tighter usable+context budget, so tool schemas, screenshots, and accessibility trees can materially affect how+long the agent can continue. A smaller tool surface and bounded state reads leave more context for+the task itself. See [Run a local model with Cua Driver](/how-to-guides/driver/run-with-local-model)+for one tested configuration.+## Three action surfacesOn the **coding surface**, the agent writes and runs code as the action itself. This works especially well when the task is text-native, when the same operation must run more than once, or when a small program can cover a larger body of files and state than a human would want to touch by hand.diff --git a/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx b/docs/content/docs/how-to-guides/driver/connect-your-agent.mdxindex 5e78b12fe9..4d4204d7be 100644--- a/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx+++ b/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx@@ -84,6 +84,12 @@ cua-driver mcp-config --client claudeThat profile exposes the same driver tools under the compatibility server name. It still runs Cua Driver over MCP; Anthropic's native computer-use API is separate.+<Callout type="info" title="Using a local model">+ Claude Code can also act as the harness for a model served on the same machine. Follow [Run a+ local model with Cua Driver](/how-to-guides/driver/run-with-local-model) for a tested Muse Glimmer+ setup and a smaller MCP tool surface.+</Callout>+## CodexPrint the Codex command:diff --git a/docs/content/docs/how-to-guides/driver/meta.json b/docs/content/docs/how-to-guides/driver/meta.jsonindex b5b049baf7..d93de66869 100644--- a/docs/content/docs/how-to-guides/driver/meta.json+++ b/docs/content/docs/how-to-guides/driver/meta.json@@ -11,6 +11,7 @@"run-in-macos-lume-vm","run-tests-in-macos-lume-vm","connect-your-agent",+ "run-with-local-model","drive-a-web-page","record-and-render-a-trajectory","keep-running",diff --git a/docs/content/docs/how-to-guides/driver/record-and-render-a-trajectory.mdx b/docs/content/docs/how-to-guides/driver/record-and-render-a-trajectory.mdxindex dc3ab90661..ec9459c512 100644--- a/docs/content/docs/how-to-guides/driver/record-and-render-a-trajectory.mdx+++ b/docs/content/docs/how-to-guides/driver/record-and-render-a-trajectory.mdx@@ -96,4 +96,16 @@ sudo apt install ffmpeg</Step></Steps>+## Example: edit a local-model trajectory++<VideoDemo+ src="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/reminders-editorial.mp4"+ poster="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/reminders-editorial-poster.jpg"+ title="Turn a local-model run into a short product demo"+>+ Cua Driver captured the original display and action evidence while Muse Glimmer operated+ Reminders. An editorial pass added the opening prompt and summarized tool trace; those overlays+ are separate from the built-in trajectory renderer.+</VideoDemo>+For the complete recording schemas and render options, see [MCP recording tools](/reference/cua-driver/mcp-tools#recording-tools) and the [`cua-driver recording` CLI reference](/reference/cua-driver/cli-reference#trajectory-recording).diff --git a/docs/content/docs/how-to-guides/driver/run-in-macos-lume-vm.mdx b/docs/content/docs/how-to-guides/driver/run-in-macos-lume-vm.mdxindex 7bdcd15dd6..f5b1825688 100644--- a/docs/content/docs/how-to-guides/driver/run-in-macos-lume-vm.mdx+++ b/docs/content/docs/how-to-guides/driver/run-in-macos-lume-vm.mdx@@ -194,6 +194,12 @@ Follow the printed command, then restart the client. See [Connect your agent toCua Driver](/how-to-guides/driver/connect-your-agent) for other supportedclients.+To use a locally served model, keep Cua Driver inside the guest and connect it+to the model's harness through the generated MCP command. [Run a local model+with Cua Driver](/how-to-guides/driver/run-with-local-model) shows a tested Muse+Glimmer and Claude Code configuration. Running the exact driver in the guest+keeps screenshots, input, and recording attached to the guest's macOS identity.+### Run the agent on the hostThe host can expose the guest's MCP process over SSH. First, copy only your host@@ -267,6 +273,7 @@ session that owns the visible desktop.## Related guides+- [Run a local model with Cua Driver](/how-to-guides/driver/run-with-local-model)- [Manage local Lume VMs](/how-to-guides/lume/manage-vms)- [macOS permissions reference](/reference/cua-driver/macos-permissions)- [Run Cua Driver macOS tests in a Lume VM](/how-to-guides/driver/run-tests-in-macos-lume-vm)diff --git a/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx b/docs/content/docs/how-to-guides/driver/run-with-local-model.mdxnew file mode 100644index 0000000000..33eba3e3c6--- /dev/null+++ b/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx@@ -0,0 +1,251 @@+---+title: Run a local model with Cua Driver+description: Connect Muse Glimmer to Cua Driver through Claude Code using Ollama or llama.cpp while keeping the model's tool context small.+---++import { Callout } from 'fumadocs-ui/components/callout';++This guide shows you how to serve Muse Glimmer 30B locally, use Claude Code as the agent harness,+and let the model operate desktop applications through Cua Driver. Choose Ollama for the shorter+Apple Silicon setup. Choose llama.cpp with an Unsloth GGUF when you want the configuration used for+the recorded runs or need direct control over inference settings.++The configuration below reproduces the architecture used for these verified macOS runs:++<div className="not-prose my-8 grid gap-5 md:grid-cols-2">+ <VideoDemo+ src="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/notes-editorial.mp4"+ poster="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/notes-editorial-poster.jpg"+ title="Create a checklist in Apple Notes"+ >+ The model mixes accessibility and pixel-grounded actions, then verifies the checklist state.+ </VideoDemo>++ <VideoDemo+ src="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/reminders-editorial.mp4"+ poster="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/reminders-editorial-poster.jpg"+ title="Schedule an Apple Reminder"+ >+ The model uses native controls and a menu, then verifies the title, date, time, and completion state.+ </VideoDemo>+</div>++<Callout type="info" title="Tested configuration">+ The recorded runs used Meta's Muse Glimmer 30B, Unsloth's `UD-Q4_K_XL` GGUF, llama.cpp's+ Anthropic-compatible server, Claude Code, and Cua Driver inside a macOS Lume guest. This is a+ tested configuration rather than a compatibility claim for every local model or inference server.+</Callout>++## Before you start++You need:++- [Cua Driver installed](/how-to-guides/driver/install) on the computer the model will operate;+- Cua Driver's required operating-system permissions, confirmed with `cua-driver doctor`;+- Ollama or a current `llama-server` build with multimodal and tool-calling support;+- Claude Code; and+- enough memory and storage for the selected quant and context size.++This guide uses the host desktop. To keep the task in a disposable macOS VM, first [run Cua Driver+inside a Lume guest](/how-to-guides/driver/run-in-macos-lume-vm), then use the guest's MCP command as+the filter's `--driver` value.++## Install the MCP schema filter++Local models pay a context cost for every exposed tool schema. Install the small filter used by this+guide:++```bash+mkdir -p "$HOME/.local/bin"+curl -fsSL \+ https://cua.ai/docs/examples/local-models/cua-mcp-filter.py \+ -o "$HOME/.local/bin/cua-mcp-filter"+chmod +x "$HOME/.local/bin/cua-mcp-filter"+```++The filter changes the `tools/list` response so the model sees only the named tools. It does not+authorize those tools or block a caller that already knows another tool name. Use [permission+policies](/concepts/how-permission-policies-work) when you need an enforcement boundary.++Create `muse-cua-mcp.json`:++```json+{+ "mcpServers": {+ "cua-computer-use": {+ "command": "cua-mcp-filter",+ "args": [+ "--allow",+ "start_session,end_session,launch_app,list_apps,list_windows,get_window_state,get_accessibility_tree,move_cursor,click,type_text,press_key,hotkey,invoke_menu"+ ]+ }+ }+}+```++Keep the allowlist as small as the task permits. Add a tool only when the task needs it.++## Choose a serving path++Both paths connect the same filtered Cua Driver MCP server to Claude Code. The Ollama path uses the+official `muse-glimmer:30b-mlx` model. The llama.cpp path uses Unsloth's `UD-Q4_K_XL` GGUF and+matches the recorded configuration.++### Path 1: Use Ollama on Apple Silicon++Pull the official Muse Glimmer model:++```bash+ollama pull muse-glimmer:30b-mlx+```++From the directory that contains `muse-cua-mcp.json`, launch Claude Code:++```bash+CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 \+ollama launch claude \+ --model muse-glimmer:30b-mlx \+ --yes \+ -- \+ --bare \+ --strict-mcp-config \+ --mcp-config ./muse-cua-mcp.json \+ --tools ""+```++Ollama serves the model through its Anthropic-compatible API. The published model has a 128K+context window and supports images and tools.++### Path 2: Use llama.cpp with an Unsloth GGUF++Run llama.cpp's server in a separate terminal:++```bash+llama-server \+ --hf-repo "unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL" \+ --alias "muse-glimmer-local" \+ --host 127.0.0.1 \+ --port 8001 \+ --ctx-size 131072 \+ --parallel 1 \+ --temp 1.0 \+ --top-p 0.95 \+ --top-k 64 \+ --jinja \+ --mmproj-auto \+ --fit on \+ --no-webui+```++The first start downloads the selected GGUF and vision projector. Keep the endpoint bound to+`127.0.0.1` unless you have separately secured access to it.++Wait for the server to finish loading, then check its health from another terminal:++```bash+curl -fsS http://127.0.0.1:8001/health+```++From the directory that contains `muse-cua-mcp.json`, run:++```bash+ANTHROPIC_BASE_URL=http://127.0.0.1:8001 \+ANTHROPIC_API_KEY=local-no-key-required \+CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 \+claude \+ --bare \+ --strict-mcp-config \+ --mcp-config ./muse-cua-mcp.json \+ --tools "" \+ --model muse-glimmer-local+```++For either path, `--bare` avoids loading unrelated project instructions and integrations.+`--tools ""` removes Claude Code's built-in tools from this session, while the explicit MCP+configuration keeps the Cua Driver tools available.++## Run a bounded smoke task++Start with a short task that has an observable result:++```text+Use only the cua-computer-use MCP tools. Launch Calculator, calculate 2 + 3,+and verify from fresh state that the display shows 5. Take a fresh window+state before every action and verify the result after every action. Keep+get_window_state calls to max_elements=25 and max_depth=3 unless a deeper+accessibility tree is required.+```++The model should launch Calculator, act through Cua Driver, read a fresh final state, and report the+observed value. If it reports success without the final observation, ask it to verify again before+accepting the result.++## Keep context use under control++For local computer use, the largest avoidable costs are tool schemas and repeated desktop state.+Use these rules:++- expose only the tools required by the task;+- request window state instead of the entire desktop when the target window is known;+- bound accessibility reads with `max_elements` and `max_depth`;+- reuse the `pid` and `window_id` returned by fresh state; and+- batch deterministic text entry when one `type_text` action can replace several inference turns.++In the tested Calculator runs, filtering tools, bounding state, and batching input reduced uncached+input from 71,088 tokens to 12,251 and elapsed time from 660 seconds to 224 seconds. Both runs+independently verified the displayed value.++## Run the model against a macOS Lume guest++When the target is a Lume VM, run the exact Cua Driver executable inside the logged-in guest and+grant permissions to that guest identity. Use SSH only to carry MCP stdio between the harness and+the guest. This keeps screenshots, input, and recording attached to the macOS session being+operated instead of relaying clicks through a VM viewer.++Follow [Run Cua Driver in a macOS Lume VM](/how-to-guides/driver/run-in-macos-lume-vm) for the guest+setup and SSH command. Save that SSH command as an executable `guest-cua-mcp` wrapper in the current+directory, then pass the wrapper to the filter:++```bash+cua-mcp-filter \+ --driver ./guest-cua-mcp \+ --allow start_session,end_session,get_window_state,move_cursor,click,type_text,invoke_menu+```++## Record the run++Use [Record and render a Cua Driver trajectory](/how-to-guides/driver/record-and-render-a-trajectory)+to retain the raw display capture and per-action evidence. The two videos above add an editorial+intro and summarized tool trace after Cua Driver finished the original recording.++## Troubleshooting++### Claude Code sees the full Cua Driver tool catalog++Confirm that the session uses both `--strict-mcp-config` and the filtered `muse-cua-mcp.json`. A+global Cua Driver registration can otherwise load alongside the filtered server.++### The model stops after a few actions++Reduce `max_elements` and `max_depth`, remove unused tools, and shorten the task. With Ollama,+confirm the selected model has at least a 64K context window. If llama.cpp lowers the usable context+to fit memory, restart it with a context size the machine can hold reliably.++### Screenshots do not reach the model++Use a real MCP connection. Shell wrappers that flatten MCP image blocks into text remove the visual+input that a multimodal model needs for pixel grounding.++### macOS reports missing permissions++Run `cua-driver permissions status` on the machine being operated. In a Lume VM, query the guest+daemon and grant Accessibility and Screen Recording to the guest's Cua Driver identity.++## Related guides++- [Muse Glimmer model and quant documentation](https://unsloth.ai/docs/models/muse-glimmer)+- [Connect Ollama to Claude Code](https://docs.ollama.com/integrations/claude-code)+- [Connect your agent to Cua Driver](/how-to-guides/driver/connect-your-agent)+- [Run Cua Driver in a macOS Lume VM](/how-to-guides/driver/run-in-macos-lume-vm)+- [Record and render a Cua Driver trajectory](/how-to-guides/driver/record-and-render-a-trajectory)+- [Agent action policy](/reference/cua-driver/action-selection-policy)diff --git a/docs/public/examples/local-models/cua-mcp-filter.py b/docs/public/examples/local-models/cua-mcp-filter.pynew file mode 100644index 0000000000..b575f881e4--- /dev/null+++ b/docs/public/examples/local-models/cua-mcp-filter.py@@ -0,0 +1,108 @@+#!/usr/bin/env python3+"""Expose an allowlisted subset of a Cua Driver MCP server over stdio.++This reduces the tool-schema context sent to a model. It is not an authorization+boundary; use Cua Driver permission policies to enforce tool access.+"""++from __future__ import annotations++import argparse+import json+import shutil+import subprocess+import sys+import threading+++def parse_args() -> argparse.Namespace:+ parser = argparse.ArgumentParser()+ parser.add_argument(+ "--allow",+ required=True,+ help="Comma-separated Cua Driver tool names to advertise",+ )+ parser.add_argument(+ "--driver",+ default=shutil.which("cua-driver"),+ help="Path to cua-driver or a wrapper that accepts the mcp argument",+ )+ args = parser.parse_args()+ if not args.driver:+ parser.error("cua-driver was not found on PATH; pass --driver")+ return args+++def main() -> int:+ args = parse_args()+ allowed = {name.strip() for name in args.allow.split(",") if name.strip()}+ child = subprocess.Popen(+ [args.driver, "mcp"],+ stdin=subprocess.PIPE,+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ bufsize=0,+ )+ assert child.stdin is not None+ assert child.stdout is not None+ assert child.stderr is not None++ pending_tool_lists: set[object] = set()+ pending_lock = threading.Lock()++ def parent_to_child() -> None:+ try:+ for line in sys.stdin.buffer:+ try:+ message = json.loads(line)+ if message.get("method") == "tools/list" and "id" in message:+ with pending_lock:+ pending_tool_lists.add(message["id"])+ except (json.JSONDecodeError, AttributeError):+ pass+ child.stdin.write(line)+ child.stdin.flush()+ except (BrokenPipeError, OSError):+ pass+ finally:+ try:+ child.stdin.close()+ except OSError:+ pass++ def forward_stderr() -> None:+ for chunk in iter(lambda: child.stderr.read(8192), b""):+ sys.stderr.buffer.write(chunk)+ sys.stderr.buffer.flush()++ threading.Thread(target=parent_to_child, daemon=True).start()+ threading.Thread(target=forward_stderr, daemon=True).start()++ try:+ for line in child.stdout:+ output = line+ try:+ message = json.loads(line)+ message_id = message.get("id")+ with pending_lock:+ is_tool_list = message_id in pending_tool_lists+ if is_tool_list:+ pending_tool_lists.remove(message_id)+ tools = message.get("result", {}).get("tools")+ if is_tool_list and isinstance(tools, list):+ message["result"]["tools"] = [+ tool for tool in tools if tool.get("name") in allowed+ ]+ output = (json.dumps(message, separators=(",", ":")) + "\n").encode()+ except (json.JSONDecodeError, AttributeError, TypeError):+ pass+ sys.stdout.buffer.write(output)+ sys.stdout.buffer.flush()+ except KeyboardInterrupt:+ child.terminate()++ return child.wait()+++if __name__ == "__main__":+ raise SystemExit(main())