Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

docs(cua-driver): add a text-only mode to the local-model MCP filter (#4380)

FenjuFue361a7f87e24
diff --git a/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx b/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx
index 33eba3e3c6..196fcbd4ce 100644
--- a/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx
+++ b/docs/content/docs/how-to-guides/driver/run-with-local-model.mdx
@@ -67,6 +67,9 @@ The filter changes the `tools/list` response so the model sees only the named to
authorize those tools or block a caller that already knows another tool name. Use [permission
policies](/concepts/how-permission-policies-work) when you need an enforcement boundary.
+When the model cannot accept images, add `--text-only`. The filter then keeps screenshots out of tool
+results. See [Connect a text-only model](/use-cua-with/iflytek#connect-a-text-only-model).
+
Create `muse-cua-mcp.json`:
```json
diff --git a/docs/content/docs/use-cua-with/iflytek.mdx b/docs/content/docs/use-cua-with/iflytek.mdx
new file mode 100644
index 0000000000..9238c741fb
--- /dev/null
+++ b/docs/content/docs/use-cua-with/iflytek.mdx
@@ -0,0 +1,77 @@
+---
+title: iFlytek Spark
+description: Use iFlytek Spark-X2.5 with Cua Driver through Claude Code and the Astron Token Plan.
+icon: Sparkles
+---
+
+import { Callout } from 'fumadocs-ui/components/callout';
+
+iFlytek serves Spark-X2.5 through the Astron Token Plan, which exposes both an Anthropic-compatible
+and an OpenAI-compatible API. Configure the harness to use Spark-X2.5, then connect that same
+harness to Cua Driver over MCP.
+
+| Start here when… | Recommended path |
+| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| You use Claude Code | Follow iFlytek's [Token Plan setup](https://www.xfyun.cn/doc/spark/TokenPlan.html), then connect Claude Code to Cua Driver through the [text-only filter](#connect-a-text-only-model) below |
+| You use another MCP-capable harness | Configure the Token Plan's OpenAI-compatible endpoint from the same [setup guide](https://www.xfyun.cn/doc/spark/TokenPlan.html), then [add Cua Driver over MCP](/how-to-guides/driver/connect-your-agent#generic-mcp-json) |
+
+Cua Driver supplies desktop state and actions. The harness owns the Spark-X2.5 connection, reasoning
+loop, and tool selection.
+
+## Connect a text-only model
+
+Spark-X2.5 does not accept image input. The Token Plan rejects any request that carries an image,
+including a screenshot inside a tool result, with `400 invalid request`. `get_window_state` returns a
+screenshot by default, and once that screenshot is in the conversation history every later request
+fails the same way.
+
+Spark-X2.5 can still operate applications through the accessibility tree. Run Cua Driver behind the
+[MCP schema filter](/how-to-guides/driver/run-with-local-model#install-the-mcp-schema-filter) with
+`--text-only`. The filter requests `get_window_state` without a screenshot, replaces any remaining
+image content with a short text note. The model still acts through the `element_token` values in
+each `get_window_state` result.
+
+Create `spark-cua-mcp.json`:
+
+```json
+{
+ "mcpServers": {
+ "cua-computer-use": {
+ "command": "cua-mcp-filter",
+ "args": [
+ "--text-only",
+ "--allow",
+ "launch_app,list_windows,get_window_state,click,type_text,press_key,hotkey"
+ ]
+ }
+ }
+}
+```
+
+Leave pixel-only tools such as `screenshot` and `zoom` out of the allowlist. Their results would be
+reduced to the omission note. On Windows, set `command` to `python` and pass the path of the
+downloaded `cua-mcp-filter.py` as the first argument.
+
+With Claude Code configured for the Token Plan, launch it with only the filtered server:
+
+```bash
+claude \
+ --bare \
+ --strict-mcp-config \
+ --mcp-config ./spark-cua-mcp.json \
+ --tools "" \
+ --model spark-x2.5
+```
+
+<Callout type="info" title="Tested configuration">
+ Claude Code 2.1.284, Cua Driver 0.30.4 on Windows 11, and `spark-x2.5` through the Token Plan's
+ Anthropic-compatible endpoint. The model found Windows Calculator with `list_windows`, cleared it,
+ entered a calculation through element clicks, and confirmed the displayed result from a fresh
+ `get_window_state`. Without `--text-only`, the same run failed with `400 invalid request` as soon as a
+ screenshot reached the model. This is a tested configuration rather than a compatibility claim
+ for every application.
+</Callout>
+
+Text-only operation depends on the application's accessibility tree. When a target exposes no usable
+elements, such as a canvas, video, or custom-drawn surface, use a vision model that the Token Plan
+also serves, or another multimodal provider, instead.
diff --git a/docs/content/docs/use-cua-with/index.mdx b/docs/content/docs/use-cua-with/index.mdx
index 5a9af780cb..6b41811108 100644
--- a/docs/content/docs/use-cua-with/index.mdx
+++ b/docs/content/docs/use-cua-with/index.mdx
@@ -14,6 +14,7 @@ import {
Orbit,
PanelsTopLeft,
Rocket,
+ Sparkles,
TerminalSquare,
Waypoints,
} from 'lucide-react';
@@ -49,6 +50,9 @@ These pages show the verified harnesses and local runtimes that can put each mod
<Card icon={<AudioWaveform />} title="MiniMax" href="/use-cua-with/minimax">
Use MiniMax M-series models through Codex or Claude Code.
</Card>
+ <Card icon={<Sparkles />} title="iFlytek Spark" href="/use-cua-with/iflytek">
+ Run the text-only Spark-X2.5 model through Claude Code and the accessibility tree.
+ </Card>
</Cards>
## Agent harnesses
diff --git a/docs/content/docs/use-cua-with/meta.json b/docs/content/docs/use-cua-with/meta.json
index a0d31b8c67..697f975e7f 100644
--- a/docs/content/docs/use-cua-with/meta.json
+++ b/docs/content/docs/use-cua-with/meta.json
@@ -12,6 +12,7 @@
"qwen",
"kimi",
"minimax",
+ "iflytek",
"---Agent harnesses---",
"claude-code",
"codex",
diff --git a/docs/public/examples/local-models/cua-mcp-filter.py b/docs/public/examples/local-models/cua-mcp-filter.py
index b575f881e4..22ca1234bc 100644
--- a/docs/public/examples/local-models/cua-mcp-filter.py
+++ b/docs/public/examples/local-models/cua-mcp-filter.py
@@ -3,6 +3,10 @@
This reduces the tool-schema context sent to a model. It is not an authorization
boundary; use Cua Driver permission policies to enforce tool access.
+
+With --text-only, tool results also stay usable by a model that cannot accept
+images: get_window_state is asked for the tree without a screenshot, and image
+content blocks are replaced by a short text note.
"""
from __future__ import annotations
@@ -13,6 +17,7 @@ import shutil
import subprocess
import sys
import threading
+from typing import Any
def parse_args() -> argparse.Namespace:
@@ -27,12 +32,63 @@ def parse_args() -> argparse.Namespace:
default=shutil.which("cua-driver"),
help="Path to cua-driver or a wrapper that accepts the mcp argument",
)
+ parser.add_argument(
+ "--text-only",
+ action="store_true",
+ help="Keep image content out of tool results for models without image input",
+ )
args = parser.parse_args()
if not args.driver:
parser.error("cua-driver was not found on PATH; pass --driver")
return args
+def is_image_block(block: Any) -> bool:
+ if not isinstance(block, dict):
+ return False
+ if block.get("type") == "image":
+ return True
+ if block.get("type") == "resource_link":
+ mime_type = block.get("mimeType")
+ elif block.get("type") == "resource" and isinstance(block.get("resource"), dict):
+ mime_type = block["resource"].get("mimeType")
+ else:
+ return False
+ return str(mime_type or "").lower().startswith("image/")
+
+
+def request_without_screenshot(message: dict[str, Any]) -> bool:
+ params = message.get("params")
+ if not isinstance(params, dict) or params.get("name") != "get_window_state":
+ return False
+ arguments = params.get("arguments")
+ if not isinstance(arguments, dict):
+ arguments = {}
+ params["arguments"] = arguments
+ if arguments.get("include_screenshot") is False:
+ return False
+ arguments["include_screenshot"] = False
+ return True
+
+
+def text_only_result(result: dict[str, Any]) -> bool:
+ content = result.get("content")
+ if not isinstance(content, list):
+ return False
+ kept = [block for block in content if not is_image_block(block)]
+ omitted = len(content) - len(kept)
+ if not omitted:
+ return False
+ kept.append(
+ {
+ "type": "text",
+ "text": f"[{omitted} image block(s) omitted: this session is text-only]",
+ }
+ )
+ result["content"] = kept
+ return True
+
+
def main() -> int:
args = parse_args()
allowed = {name.strip() for name in args.allow.split(",") if name.strip()}
@@ -48,6 +104,7 @@ def main() -> int:
assert child.stderr is not None
pending_tool_lists: set[object] = set()
+ pending_tool_calls: set[object] = set()
pending_lock = threading.Lock()
def parent_to_child() -> None:
@@ -58,6 +115,11 @@ def main() -> int:
if message.get("method") == "tools/list" and "id" in message:
with pending_lock:
pending_tool_lists.add(message["id"])
+ if args.text_only and message.get("method") == "tools/call" and "id" in message:
+ with pending_lock:
+ pending_tool_calls.add(message["id"])
+ if request_without_screenshot(message):
+ line = (json.dumps(message, separators=(",", ":")) + "\n").encode()
except (json.JSONDecodeError, AttributeError):
pass
child.stdin.write(line)
@@ -88,12 +150,18 @@ def main() -> int:
is_tool_list = message_id in pending_tool_lists
if is_tool_list:
pending_tool_lists.remove(message_id)
- tools = message.get("result", {}).get("tools")
+ is_tool_call = message_id in pending_tool_calls
+ if is_tool_call:
+ pending_tool_calls.remove(message_id)
+ result = message.get("result")
+ tools = result.get("tools") if isinstance(result, dict) else None
if is_tool_list and isinstance(tools, list):
message["result"]["tools"] = [
tool for tool in tools if tool.get("name") in allowed
]
output = (json.dumps(message, separators=(",", ":")) + "\n").encode()
+ elif is_tool_call and isinstance(result, dict) and text_only_result(result):
+ output = (json.dumps(message, separators=(",", ":")) + "\n").encode()
except (json.JSONDecodeError, AttributeError, TypeError):
pass
sys.stdout.buffer.write(output)