Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-driver): add a text-only mode to the local-model MCP filter (#4380)
FenjuFue361a7f87e24
diff --git a/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx b/docs/content/docs/how-to-guides/driver/run-with-local-model.mdxindex 33eba3e3c6..196fcbd4ce 100644--- a/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx+++ b/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx@@ -67,6 +67,9 @@ The filter changes the `tools/list` response so the model sees only the named toauthorize those tools or block a caller that already knows another tool name. Use [permissionpolicies](/concepts/how-permission-policies-work) when you need an enforcement boundary.+When the model cannot accept images, add `--text-only`. The filter then keeps screenshots out of tool+results. See [Connect a text-only model](/use-cua-with/iflytek#connect-a-text-only-model).+Create `muse-cua-mcp.json`:```jsondiff --git a/docs/content/docs/use-cua-with/iflytek.mdx b/docs/content/docs/use-cua-with/iflytek.mdxnew file mode 100644index 0000000000..9238c741fb--- /dev/null+++ b/docs/content/docs/use-cua-with/iflytek.mdx@@ -0,0 +1,77 @@+---+title: iFlytek Spark+description: Use iFlytek Spark-X2.5 with Cua Driver through Claude Code and the Astron Token Plan.+icon: Sparkles+---++import { Callout } from 'fumadocs-ui/components/callout';++iFlytek serves Spark-X2.5 through the Astron Token Plan, which exposes both an Anthropic-compatible+and an OpenAI-compatible API. Configure the harness to use Spark-X2.5, then connect that same+harness to Cua Driver over MCP.++| Start here when… | Recommended path |+| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| You use Claude Code | Follow iFlytek's [Token Plan setup](https://www.xfyun.cn/doc/spark/TokenPlan.html), then connect Claude Code to Cua Driver through the [text-only filter](#connect-a-text-only-model) below |+| You use another MCP-capable harness | Configure the Token Plan's OpenAI-compatible endpoint from the same [setup guide](https://www.xfyun.cn/doc/spark/TokenPlan.html), then [add Cua Driver over MCP](/how-to-guides/driver/connect-your-agent#generic-mcp-json) |++Cua Driver supplies desktop state and actions. The harness owns the Spark-X2.5 connection, reasoning+loop, and tool selection.++## Connect a text-only model++Spark-X2.5 does not accept image input. The Token Plan rejects any request that carries an image,+including a screenshot inside a tool result, with `400 invalid request`. `get_window_state` returns a+screenshot by default, and once that screenshot is in the conversation history every later request+fails the same way.++Spark-X2.5 can still operate applications through the accessibility tree. Run Cua Driver behind the+[MCP schema filter](/how-to-guides/driver/run-with-local-model#install-the-mcp-schema-filter) with+`--text-only`. The filter requests `get_window_state` without a screenshot, replaces any remaining+image content with a short text note. The model still acts through the `element_token` values in+each `get_window_state` result.++Create `spark-cua-mcp.json`:++```json+{+ "mcpServers": {+ "cua-computer-use": {+ "command": "cua-mcp-filter",+ "args": [+ "--text-only",+ "--allow",+ "launch_app,list_windows,get_window_state,click,type_text,press_key,hotkey"+ ]+ }+ }+}+```++Leave pixel-only tools such as `screenshot` and `zoom` out of the allowlist. Their results would be+reduced to the omission note. On Windows, set `command` to `python` and pass the path of the+downloaded `cua-mcp-filter.py` as the first argument.++With Claude Code configured for the Token Plan, launch it with only the filtered server:++```bash+claude \+ --bare \+ --strict-mcp-config \+ --mcp-config ./spark-cua-mcp.json \+ --tools "" \+ --model spark-x2.5+```++<Callout type="info" title="Tested configuration">+ Claude Code 2.1.284, Cua Driver 0.30.4 on Windows 11, and `spark-x2.5` through the Token Plan's+ Anthropic-compatible endpoint. The model found Windows Calculator with `list_windows`, cleared it,+ entered a calculation through element clicks, and confirmed the displayed result from a fresh+ `get_window_state`. Without `--text-only`, the same run failed with `400 invalid request` as soon as a+ screenshot reached the model. This is a tested configuration rather than a compatibility claim+ for every application.+</Callout>++Text-only operation depends on the application's accessibility tree. When a target exposes no usable+elements, such as a canvas, video, or custom-drawn surface, use a vision model that the Token Plan+also serves, or another multimodal provider, instead.diff --git a/docs/content/docs/use-cua-with/index.mdx b/docs/content/docs/use-cua-with/index.mdxindex 5a9af780cb..6b41811108 100644--- a/docs/content/docs/use-cua-with/index.mdx+++ b/docs/content/docs/use-cua-with/index.mdx@@ -14,6 +14,7 @@ import {Orbit,PanelsTopLeft,Rocket,+ Sparkles,TerminalSquare,Waypoints,} from 'lucide-react';@@ -49,6 +50,9 @@ These pages show the verified harnesses and local runtimes that can put each mod<Card icon={<AudioWaveform />} title="MiniMax" href="/use-cua-with/minimax">Use MiniMax M-series models through Codex or Claude Code.</Card>+ <Card icon={<Sparkles />} title="iFlytek Spark" href="/use-cua-with/iflytek">+ Run the text-only Spark-X2.5 model through Claude Code and the accessibility tree.+ </Card></Cards>## Agent harnessesdiff --git a/docs/content/docs/use-cua-with/meta.json b/docs/content/docs/use-cua-with/meta.jsonindex a0d31b8c67..697f975e7f 100644--- a/docs/content/docs/use-cua-with/meta.json+++ b/docs/content/docs/use-cua-with/meta.json@@ -12,6 +12,7 @@"qwen","kimi","minimax",+ "iflytek","---Agent harnesses---","claude-code","codex",diff --git a/docs/public/examples/local-models/cua-mcp-filter.py b/docs/public/examples/local-models/cua-mcp-filter.pyindex b575f881e4..22ca1234bc 100644--- a/docs/public/examples/local-models/cua-mcp-filter.py+++ b/docs/public/examples/local-models/cua-mcp-filter.py@@ -3,6 +3,10 @@This reduces the tool-schema context sent to a model. It is not an authorizationboundary; use Cua Driver permission policies to enforce tool access.++With --text-only, tool results also stay usable by a model that cannot accept+images: get_window_state is asked for the tree without a screenshot, and image+content blocks are replaced by a short text note."""from __future__ import annotations@@ -13,6 +17,7 @@ import shutilimport subprocessimport sysimport threading+from typing import Anydef parse_args() -> argparse.Namespace:@@ -27,12 +32,63 @@ def parse_args() -> argparse.Namespace:default=shutil.which("cua-driver"),help="Path to cua-driver or a wrapper that accepts the mcp argument",)+ parser.add_argument(+ "--text-only",+ action="store_true",+ help="Keep image content out of tool results for models without image input",+ )args = parser.parse_args()if not args.driver:parser.error("cua-driver was not found on PATH; pass --driver")return args+def is_image_block(block: Any) -> bool:+ if not isinstance(block, dict):+ return False+ if block.get("type") == "image":+ return True+ if block.get("type") == "resource_link":+ mime_type = block.get("mimeType")+ elif block.get("type") == "resource" and isinstance(block.get("resource"), dict):+ mime_type = block["resource"].get("mimeType")+ else:+ return False+ return str(mime_type or "").lower().startswith("image/")+++def request_without_screenshot(message: dict[str, Any]) -> bool:+ params = message.get("params")+ if not isinstance(params, dict) or params.get("name") != "get_window_state":+ return False+ arguments = params.get("arguments")+ if not isinstance(arguments, dict):+ arguments = {}+ params["arguments"] = arguments+ if arguments.get("include_screenshot") is False:+ return False+ arguments["include_screenshot"] = False+ return True+++def text_only_result(result: dict[str, Any]) -> bool:+ content = result.get("content")+ if not isinstance(content, list):+ return False+ kept = [block for block in content if not is_image_block(block)]+ omitted = len(content) - len(kept)+ if not omitted:+ return False+ kept.append(+ {+ "type": "text",+ "text": f"[{omitted} image block(s) omitted: this session is text-only]",+ }+ )+ result["content"] = kept+ return True++def main() -> int:args = parse_args()allowed = {name.strip() for name in args.allow.split(",") if name.strip()}@@ -48,6 +104,7 @@ def main() -> int:assert child.stderr is not Nonepending_tool_lists: set[object] = set()+ pending_tool_calls: set[object] = set()pending_lock = threading.Lock()def parent_to_child() -> None:@@ -58,6 +115,11 @@ def main() -> int:if message.get("method") == "tools/list" and "id" in message:with pending_lock:pending_tool_lists.add(message["id"])+ if args.text_only and message.get("method") == "tools/call" and "id" in message:+ with pending_lock:+ pending_tool_calls.add(message["id"])+ if request_without_screenshot(message):+ line = (json.dumps(message, separators=(",", ":")) + "\n").encode()except (json.JSONDecodeError, AttributeError):passchild.stdin.write(line)@@ -88,12 +150,18 @@ def main() -> int:is_tool_list = message_id in pending_tool_listsif is_tool_list:pending_tool_lists.remove(message_id)- tools = message.get("result", {}).get("tools")+ is_tool_call = message_id in pending_tool_calls+ if is_tool_call:+ pending_tool_calls.remove(message_id)+ result = message.get("result")+ tools = result.get("tools") if isinstance(result, dict) else Noneif is_tool_list and isinstance(tools, list):message["result"]["tools"] = [tool for tool in tools if tool.get("name") in allowed]output = (json.dumps(message, separators=(",", ":")) + "\n").encode()+ elif is_tool_call and isinstance(result, dict) and text_only_result(result):+ output = (json.dumps(message, separators=(",", ":")) + "\n").encode()except (json.JSONDecodeError, AttributeError, TypeError):passsys.stdout.buffer.write(output)