Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

docs(cua-driver): document local model workflows (#3054)

Francesco Bonaccif415cc9ae942
diff --git a/docs/content/docs/concepts/what-is-computer-use.mdx b/docs/content/docs/concepts/what-is-computer-use.mdx
index a114865fba..a874d77712 100644
--- a/docs/content/docs/concepts/what-is-computer-use.mdx
+++ b/docs/content/docs/concepts/what-is-computer-use.mdx
@@ -17,6 +17,22 @@ A **computer-use agent** is the running system around that model. It supplies th
The distinction matters because a capable model alone does not provide a reliable agent. The surrounding system still has to deliver input to the intended application, preserve state between steps, recover from changed interfaces, and constrain consequential actions.
+An **agent harness** is the runtime that hosts this loop. A coding agent such as Claude Code or
+Codex can supply instructions, memory, and MCP connections while using either a hosted model or a
+model served on the same machine. Cua Driver is the UI tool layer connected to that harness; it does
+not select or run the model.
+
+```text
+model -> agent harness -> Cua Driver -> operating system and applications
+```
+
+This separation lets the same driver work with different models and harnesses. It also makes the
+cost of each layer visible. Local models often have less inference throughput and a tighter usable
+context budget, so tool schemas, screenshots, and accessibility trees can materially affect how
+long the agent can continue. A smaller tool surface and bounded state reads leave more context for
+the task itself. See [Run a local model with Cua Driver](/how-to-guides/driver/run-with-local-model)
+for one tested configuration.
+
## Three action surfaces
On the **coding surface**, the agent writes and runs code as the action itself. This works especially well when the task is text-native, when the same operation must run more than once, or when a small program can cover a larger body of files and state than a human would want to touch by hand.
diff --git a/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx b/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx
index 5e78b12fe9..4d4204d7be 100644
--- a/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx
+++ b/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx
@@ -84,6 +84,12 @@ cua-driver mcp-config --client claude
That profile exposes the same driver tools under the compatibility server name. It still runs Cua Driver over MCP; Anthropic's native computer-use API is separate.
+<Callout type="info" title="Using a local model">
+ Claude Code can also act as the harness for a model served on the same machine. Follow [Run a
+ local model with Cua Driver](/how-to-guides/driver/run-with-local-model) for a tested Muse Glimmer
+ setup and a smaller MCP tool surface.
+</Callout>
+
## Codex
Print the Codex command:
diff --git a/docs/content/docs/how-to-guides/driver/meta.json b/docs/content/docs/how-to-guides/driver/meta.json
index b5b049baf7..d93de66869 100644
--- a/docs/content/docs/how-to-guides/driver/meta.json
+++ b/docs/content/docs/how-to-guides/driver/meta.json
@@ -11,6 +11,7 @@
"run-in-macos-lume-vm",
"run-tests-in-macos-lume-vm",
"connect-your-agent",
+ "run-with-local-model",
"drive-a-web-page",
"record-and-render-a-trajectory",
"keep-running",
diff --git a/docs/content/docs/how-to-guides/driver/record-and-render-a-trajectory.mdx b/docs/content/docs/how-to-guides/driver/record-and-render-a-trajectory.mdx
index dc3ab90661..ec9459c512 100644
--- a/docs/content/docs/how-to-guides/driver/record-and-render-a-trajectory.mdx
+++ b/docs/content/docs/how-to-guides/driver/record-and-render-a-trajectory.mdx
@@ -96,4 +96,16 @@ sudo apt install ffmpeg
</Step>
</Steps>
+## Example: edit a local-model trajectory
+
+<VideoDemo
+ src="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/reminders-editorial.mp4"
+ poster="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/reminders-editorial-poster.jpg"
+ title="Turn a local-model run into a short product demo"
+>
+ Cua Driver captured the original display and action evidence while Muse Glimmer operated
+ Reminders. An editorial pass added the opening prompt and summarized tool trace; those overlays
+ are separate from the built-in trajectory renderer.
+</VideoDemo>
+
For the complete recording schemas and render options, see [MCP recording tools](/reference/cua-driver/mcp-tools#recording-tools) and the [`cua-driver recording` CLI reference](/reference/cua-driver/cli-reference#trajectory-recording).
diff --git a/docs/content/docs/how-to-guides/driver/run-in-macos-lume-vm.mdx b/docs/content/docs/how-to-guides/driver/run-in-macos-lume-vm.mdx
index 7bdcd15dd6..f5b1825688 100644
--- a/docs/content/docs/how-to-guides/driver/run-in-macos-lume-vm.mdx
+++ b/docs/content/docs/how-to-guides/driver/run-in-macos-lume-vm.mdx
@@ -194,6 +194,12 @@ Follow the printed command, then restart the client. See [Connect your agent to
Cua Driver](/how-to-guides/driver/connect-your-agent) for other supported
clients.
+To use a locally served model, keep Cua Driver inside the guest and connect it
+to the model's harness through the generated MCP command. [Run a local model
+with Cua Driver](/how-to-guides/driver/run-with-local-model) shows a tested Muse
+Glimmer and Claude Code configuration. Running the exact driver in the guest
+keeps screenshots, input, and recording attached to the guest's macOS identity.
+
### Run the agent on the host
The host can expose the guest's MCP process over SSH. First, copy only your host
@@ -267,6 +273,7 @@ session that owns the visible desktop.
## Related guides
+- [Run a local model with Cua Driver](/how-to-guides/driver/run-with-local-model)
- [Manage local Lume VMs](/how-to-guides/lume/manage-vms)
- [macOS permissions reference](/reference/cua-driver/macos-permissions)
- [Run Cua Driver macOS tests in a Lume VM](/how-to-guides/driver/run-tests-in-macos-lume-vm)
diff --git a/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx b/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx
new file mode 100644
index 0000000000..33eba3e3c6
--- /dev/null
+++ b/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx
@@ -0,0 +1,251 @@
+---
+title: Run a local model with Cua Driver
+description: Connect Muse Glimmer to Cua Driver through Claude Code using Ollama or llama.cpp while keeping the model's tool context small.
+---
+
+import { Callout } from 'fumadocs-ui/components/callout';
+
+This guide shows you how to serve Muse Glimmer 30B locally, use Claude Code as the agent harness,
+and let the model operate desktop applications through Cua Driver. Choose Ollama for the shorter
+Apple Silicon setup. Choose llama.cpp with an Unsloth GGUF when you want the configuration used for
+the recorded runs or need direct control over inference settings.
+
+The configuration below reproduces the architecture used for these verified macOS runs:
+
+<div className="not-prose my-8 grid gap-5 md:grid-cols-2">
+ <VideoDemo
+ src="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/notes-editorial.mp4"
+ poster="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/notes-editorial-poster.jpg"
+ title="Create a checklist in Apple Notes"
+ >
+ The model mixes accessibility and pixel-grounded actions, then verifies the checklist state.
+ </VideoDemo>
+
+ <VideoDemo
+ src="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/reminders-editorial.mp4"
+ poster="https://cuademo06261324.blob.core.windows.net/artifacts/demos/muse-glimmer/reminders-editorial-poster.jpg"
+ title="Schedule an Apple Reminder"
+ >
+ The model uses native controls and a menu, then verifies the title, date, time, and completion state.
+ </VideoDemo>
+</div>
+
+<Callout type="info" title="Tested configuration">
+ The recorded runs used Meta's Muse Glimmer 30B, Unsloth's `UD-Q4_K_XL` GGUF, llama.cpp's
+ Anthropic-compatible server, Claude Code, and Cua Driver inside a macOS Lume guest. This is a
+ tested configuration rather than a compatibility claim for every local model or inference server.
+</Callout>
+
+## Before you start
+
+You need:
+
+- [Cua Driver installed](/how-to-guides/driver/install) on the computer the model will operate;
+- Cua Driver's required operating-system permissions, confirmed with `cua-driver doctor`;
+- Ollama or a current `llama-server` build with multimodal and tool-calling support;
+- Claude Code; and
+- enough memory and storage for the selected quant and context size.
+
+This guide uses the host desktop. To keep the task in a disposable macOS VM, first [run Cua Driver
+inside a Lume guest](/how-to-guides/driver/run-in-macos-lume-vm), then use the guest's MCP command as
+the filter's `--driver` value.
+
+## Install the MCP schema filter
+
+Local models pay a context cost for every exposed tool schema. Install the small filter used by this
+guide:
+
+```bash
+mkdir -p "$HOME/.local/bin"
+curl -fsSL \
+ https://cua.ai/docs/examples/local-models/cua-mcp-filter.py \
+ -o "$HOME/.local/bin/cua-mcp-filter"
+chmod +x "$HOME/.local/bin/cua-mcp-filter"
+```
+
+The filter changes the `tools/list` response so the model sees only the named tools. It does not
+authorize those tools or block a caller that already knows another tool name. Use [permission
+policies](/concepts/how-permission-policies-work) when you need an enforcement boundary.
+
+Create `muse-cua-mcp.json`:
+
+```json
+{
+ "mcpServers": {
+ "cua-computer-use": {
+ "command": "cua-mcp-filter",
+ "args": [
+ "--allow",
+ "start_session,end_session,launch_app,list_apps,list_windows,get_window_state,get_accessibility_tree,move_cursor,click,type_text,press_key,hotkey,invoke_menu"
+ ]
+ }
+ }
+}
+```
+
+Keep the allowlist as small as the task permits. Add a tool only when the task needs it.
+
+## Choose a serving path
+
+Both paths connect the same filtered Cua Driver MCP server to Claude Code. The Ollama path uses the
+official `muse-glimmer:30b-mlx` model. The llama.cpp path uses Unsloth's `UD-Q4_K_XL` GGUF and
+matches the recorded configuration.
+
+### Path 1: Use Ollama on Apple Silicon
+
+Pull the official Muse Glimmer model:
+
+```bash
+ollama pull muse-glimmer:30b-mlx
+```
+
+From the directory that contains `muse-cua-mcp.json`, launch Claude Code:
+
+```bash
+CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 \
+ollama launch claude \
+ --model muse-glimmer:30b-mlx \
+ --yes \
+ -- \
+ --bare \
+ --strict-mcp-config \
+ --mcp-config ./muse-cua-mcp.json \
+ --tools ""
+```
+
+Ollama serves the model through its Anthropic-compatible API. The published model has a 128K
+context window and supports images and tools.
+
+### Path 2: Use llama.cpp with an Unsloth GGUF
+
+Run llama.cpp's server in a separate terminal:
+
+```bash
+llama-server \
+ --hf-repo "unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL" \
+ --alias "muse-glimmer-local" \
+ --host 127.0.0.1 \
+ --port 8001 \
+ --ctx-size 131072 \
+ --parallel 1 \
+ --temp 1.0 \
+ --top-p 0.95 \
+ --top-k 64 \
+ --jinja \
+ --mmproj-auto \
+ --fit on \
+ --no-webui
+```
+
+The first start downloads the selected GGUF and vision projector. Keep the endpoint bound to
+`127.0.0.1` unless you have separately secured access to it.
+
+Wait for the server to finish loading, then check its health from another terminal:
+
+```bash
+curl -fsS http://127.0.0.1:8001/health
+```
+
+From the directory that contains `muse-cua-mcp.json`, run:
+
+```bash
+ANTHROPIC_BASE_URL=http://127.0.0.1:8001 \
+ANTHROPIC_API_KEY=local-no-key-required \
+CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 \
+claude \
+ --bare \
+ --strict-mcp-config \
+ --mcp-config ./muse-cua-mcp.json \
+ --tools "" \
+ --model muse-glimmer-local
+```
+
+For either path, `--bare` avoids loading unrelated project instructions and integrations.
+`--tools ""` removes Claude Code's built-in tools from this session, while the explicit MCP
+configuration keeps the Cua Driver tools available.
+
+## Run a bounded smoke task
+
+Start with a short task that has an observable result:
+
+```text
+Use only the cua-computer-use MCP tools. Launch Calculator, calculate 2 + 3,
+and verify from fresh state that the display shows 5. Take a fresh window
+state before every action and verify the result after every action. Keep
+get_window_state calls to max_elements=25 and max_depth=3 unless a deeper
+accessibility tree is required.
+```
+
+The model should launch Calculator, act through Cua Driver, read a fresh final state, and report the
+observed value. If it reports success without the final observation, ask it to verify again before
+accepting the result.
+
+## Keep context use under control
+
+For local computer use, the largest avoidable costs are tool schemas and repeated desktop state.
+Use these rules:
+
+- expose only the tools required by the task;
+- request window state instead of the entire desktop when the target window is known;
+- bound accessibility reads with `max_elements` and `max_depth`;
+- reuse the `pid` and `window_id` returned by fresh state; and
+- batch deterministic text entry when one `type_text` action can replace several inference turns.
+
+In the tested Calculator runs, filtering tools, bounding state, and batching input reduced uncached
+input from 71,088 tokens to 12,251 and elapsed time from 660 seconds to 224 seconds. Both runs
+independently verified the displayed value.
+
+## Run the model against a macOS Lume guest
+
+When the target is a Lume VM, run the exact Cua Driver executable inside the logged-in guest and
+grant permissions to that guest identity. Use SSH only to carry MCP stdio between the harness and
+the guest. This keeps screenshots, input, and recording attached to the macOS session being
+operated instead of relaying clicks through a VM viewer.
+
+Follow [Run Cua Driver in a macOS Lume VM](/how-to-guides/driver/run-in-macos-lume-vm) for the guest
+setup and SSH command. Save that SSH command as an executable `guest-cua-mcp` wrapper in the current
+directory, then pass the wrapper to the filter:
+
+```bash
+cua-mcp-filter \
+ --driver ./guest-cua-mcp \
+ --allow start_session,end_session,get_window_state,move_cursor,click,type_text,invoke_menu
+```
+
+## Record the run
+
+Use [Record and render a Cua Driver trajectory](/how-to-guides/driver/record-and-render-a-trajectory)
+to retain the raw display capture and per-action evidence. The two videos above add an editorial
+intro and summarized tool trace after Cua Driver finished the original recording.
+
+## Troubleshooting
+
+### Claude Code sees the full Cua Driver tool catalog
+
+Confirm that the session uses both `--strict-mcp-config` and the filtered `muse-cua-mcp.json`. A
+global Cua Driver registration can otherwise load alongside the filtered server.
+
+### The model stops after a few actions
+
+Reduce `max_elements` and `max_depth`, remove unused tools, and shorten the task. With Ollama,
+confirm the selected model has at least a 64K context window. If llama.cpp lowers the usable context
+to fit memory, restart it with a context size the machine can hold reliably.
+
+### Screenshots do not reach the model
+
+Use a real MCP connection. Shell wrappers that flatten MCP image blocks into text remove the visual
+input that a multimodal model needs for pixel grounding.
+
+### macOS reports missing permissions
+
+Run `cua-driver permissions status` on the machine being operated. In a Lume VM, query the guest
+daemon and grant Accessibility and Screen Recording to the guest's Cua Driver identity.
+
+## Related guides
+
+- [Muse Glimmer model and quant documentation](https://unsloth.ai/docs/models/muse-glimmer)
+- [Connect Ollama to Claude Code](https://docs.ollama.com/integrations/claude-code)
+- [Connect your agent to Cua Driver](/how-to-guides/driver/connect-your-agent)
+- [Run Cua Driver in a macOS Lume VM](/how-to-guides/driver/run-in-macos-lume-vm)
+- [Record and render a Cua Driver trajectory](/how-to-guides/driver/record-and-render-a-trajectory)
+- [Agent action policy](/reference/cua-driver/action-selection-policy)
diff --git a/docs/public/examples/local-models/cua-mcp-filter.py b/docs/public/examples/local-models/cua-mcp-filter.py
new file mode 100644
index 0000000000..b575f881e4
--- /dev/null
+++ b/docs/public/examples/local-models/cua-mcp-filter.py
@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Expose an allowlisted subset of a Cua Driver MCP server over stdio.
+
+This reduces the tool-schema context sent to a model. It is not an authorization
+boundary; use Cua Driver permission policies to enforce tool access.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import shutil
+import subprocess
+import sys
+import threading
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--allow",
+ required=True,
+ help="Comma-separated Cua Driver tool names to advertise",
+ )
+ parser.add_argument(
+ "--driver",
+ default=shutil.which("cua-driver"),
+ help="Path to cua-driver or a wrapper that accepts the mcp argument",
+ )
+ args = parser.parse_args()
+ if not args.driver:
+ parser.error("cua-driver was not found on PATH; pass --driver")
+ return args
+
+
+def main() -> int:
+ args = parse_args()
+ allowed = {name.strip() for name in args.allow.split(",") if name.strip()}
+ child = subprocess.Popen(
+ [args.driver, "mcp"],
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ bufsize=0,
+ )
+ assert child.stdin is not None
+ assert child.stdout is not None
+ assert child.stderr is not None
+
+ pending_tool_lists: set[object] = set()
+ pending_lock = threading.Lock()
+
+ def parent_to_child() -> None:
+ try:
+ for line in sys.stdin.buffer:
+ try:
+ message = json.loads(line)
+ if message.get("method") == "tools/list" and "id" in message:
+ with pending_lock:
+ pending_tool_lists.add(message["id"])
+ except (json.JSONDecodeError, AttributeError):
+ pass
+ child.stdin.write(line)
+ child.stdin.flush()
+ except (BrokenPipeError, OSError):
+ pass
+ finally:
+ try:
+ child.stdin.close()
+ except OSError:
+ pass
+
+ def forward_stderr() -> None:
+ for chunk in iter(lambda: child.stderr.read(8192), b""):
+ sys.stderr.buffer.write(chunk)
+ sys.stderr.buffer.flush()
+
+ threading.Thread(target=parent_to_child, daemon=True).start()
+ threading.Thread(target=forward_stderr, daemon=True).start()
+
+ try:
+ for line in child.stdout:
+ output = line
+ try:
+ message = json.loads(line)
+ message_id = message.get("id")
+ with pending_lock:
+ is_tool_list = message_id in pending_tool_lists
+ if is_tool_list:
+ pending_tool_lists.remove(message_id)
+ tools = message.get("result", {}).get("tools")
+ if is_tool_list and isinstance(tools, list):
+ message["result"]["tools"] = [
+ tool for tool in tools if tool.get("name") in allowed
+ ]
+ output = (json.dumps(message, separators=(",", ":")) + "\n").encode()
+ except (json.JSONDecodeError, AttributeError, TypeError):
+ pass
+ sys.stdout.buffer.write(output)
+ sys.stdout.buffer.flush()
+ except KeyboardInterrupt:
+ child.terminate()
+
+ return child.wait()
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())