Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(computer-server): auto-detect MCP mode and expose HTTP+MCP simultaneously (#860)
Francesco Bonacci542cd97bca3a
diff --git a/libs/python/computer-server/README.md b/libs/python/computer-server/README.mdindex f0453f00a8..b51973aac6 100644--- a/libs/python/computer-server/README.md+++ b/libs/python/computer-server/README.md@@ -6,24 +6,31 @@ Server component for the Computer-Use Interface (CUI) framework providing low-le## Interfaces-The Computer Server supports multiple interfaces:+The Computer Server supports multiple interfaces with **automatic mode detection**:-- **HTTP/WebSocket** (default) - REST API and WebSocket for programmatic access-- **MCP** - Model Context Protocol for Claude Code, OpenCode, Cursor, and other MCP-compatible tools+- **HTTP/WebSocket** - REST API and WebSocket for programmatic access (interactive terminal)+- **MCP via HTTP** - Model Context Protocol over streamable HTTP at `/mcp` endpoint (interactive terminal)+- **MCP via stdio** - Model Context Protocol for Claude Code, OpenCode, Cursor, and other MCP-compatible tools (auto-detected when run as subprocess)++The server automatically detects how it's being run:+- **Interactive terminal** (stdin is TTY) → HTTP server with MCP at `/mcp`+- **Subprocess** (stdin is pipe, e.g., spawned by Claude Code) → MCP stdio mode## Installation```bash# Basic installation (HTTP/WebSocket only)-pip install cua-computer-server-server+pip install cua-computer-server-# With MCP support-pip install cua-computer-server-server[mcp]+# With MCP support (enables both HTTP and stdio modes)+pip install cua-computer-server[mcp]```## Usage-### HTTP/WebSocket Mode (Default)+### Interactive Mode (run from terminal)++When you run the server from a terminal, it starts HTTP + MCP:```bash# Start the server on default port 8000@@ -33,18 +40,20 @@ python -m computer_serverpython -m computer_server --port 8080```-### MCP Mode (for Claude Code)+This provides:+- HTTP API at `/ws`, `/cmd`, `/status` endpoints+- MCP server at `/mcp` endpoint (requires `fastmcp` package)-```bash-# Start in MCP mode-python -m computer_server --mcp+MCP clients can connect via streamable HTTP at `http://localhost:8000/mcp`.-# With resolution auto-detection (logs actual screen resolution)-python -m computer_server --mcp --detect-resolution+### Subprocess Mode (Claude Code integration)-# With target resolution (useful for Retina displays or VMs)-# Screenshots will be resized to target resolution and coordinates will be scaled-python -m computer_server --mcp --width 1512 --height 982+When spawned as a subprocess (e.g., by Claude Code), the server automatically uses MCP stdio mode:++```bash+# Resolution options work in both modes+python -m computer_server --detect-resolution+python -m computer_server --width 1512 --height 982```#### Resolution Scaling@@ -62,8 +71,8 @@ This ensures the AI model sees consistent coordinates between screenshots and moAdd to your Claude Code configuration:```bash-# Add the MCP server-claude mcp add cua-computer-server -- python -m computer_server --mcp+# Add the MCP server (auto-detects stdio mode when spawned as subprocess)+claude mcp add cua-computer-server -- python -m computer_server```Or manually add to `~/.claude/claude_desktop_config.json`:@@ -73,7 +82,7 @@ Or manually add to `~/.claude/claude_desktop_config.json`:"mcpServers": {"cua-computer-server": {"command": "python",- "args": ["-m", "computer_server", "--mcp"]+ "args": ["-m", "computer_server"]}}}diff --git a/libs/python/computer-server/computer_server/cli.py b/libs/python/computer-server/computer_server/cli.pyindex 73fb847612..03e99b99f1 100644--- a/libs/python/computer-server/computer_server/cli.py+++ b/libs/python/computer-server/computer_server/cli.py@@ -18,11 +18,6 @@ logger = logging.getLogger(__name__)def parse_args(args: Optional[List[str]] = None) -> argparse.Namespace:"""Parse command-line arguments."""parser = argparse.ArgumentParser(description="Start the Computer API server")- parser.add_argument(- "--mcp",- action="store_true",- help="Start in MCP (Model Context Protocol) mode for Claude Code integration",- )parser.add_argument("--width",type=int,@@ -91,9 +86,15 @@ def main() -> None:stream=sys.stderr, # Use stderr for MCP compatibility)- # Handle MCP mode- if args.mcp:- logger.info("Starting in MCP (Model Context Protocol) mode...")+ # Auto-detect mode based on stdin:+ # - If stdin is NOT a TTY (it's a pipe), we're being run as a subprocess (e.g., by Claude Code)+ # → use MCP stdio mode for direct communication+ # - If stdin IS a TTY (interactive terminal), user is running from command line+ # → use HTTP server with MCP available at /mcp endpoint+ use_mcp_stdio = not sys.stdin.isatty()++ if use_mcp_stdio:+ logger.info("Detected subprocess mode (stdin is pipe) - starting MCP stdio server...")try:from .mcp_server import run_mcp_serverrun_mcp_server(@@ -102,7 +103,7 @@ def main() -> None:detect_resolution=args.detect_resolution,)except ImportError as e:- logger.error(f"MCP mode requires fastmcp package: {e}")+ logger.error(f"MCP stdio mode requires fastmcp package: {e}")logger.error("Install with: pip install 'cua-computer-server[mcp]'")sys.exit(1)except KeyboardInterrupt:@@ -160,6 +161,8 @@ def main() -> None:# Create and start the serverlogger.info(f"Starting Cua Computer API server on {args.host}:{args.port}...")+ logger.info("HTTP API available at /ws, /cmd, /status endpoints")+ logger.info("MCP server available at /mcp endpoint (if fastmcp installed)")# Handle SSL configurationssl_args = {}diff --git a/libs/python/computer-server/computer_server/main.py b/libs/python/computer-server/computer_server/main.pyindex e5a9c9d4a8..03201ba4de 100644--- a/libs/python/computer-server/computer_server/main.py+++ b/libs/python/computer-server/computer_server/main.py@@ -27,6 +27,13 @@ from fastapi.responses import JSONResponse, StreamingResponsefrom .browser import get_browser_managerfrom .handlers.factory import OS_TYPE, HandlerFactory+# Try to import MCP server for SSE integration+try:+ from .mcp_server import create_mcp_server+ HAS_MCP = True+except ImportError:+ HAS_MCP = False+# Authentication session TTL (in seconds). Override via env var CUA_AUTH_TTL_SECONDS. Default: 60sAUTH_SESSION_TTL_SECONDS: int = int(os.environ.get("CUA_AUTH_TTL_SECONDS", "60"))@@ -62,6 +69,16 @@ app.add_middleware(allow_headers=["*"],)+# Mount MCP server at /mcp if fastmcp is available+if HAS_MCP:+ try:+ mcp_server = create_mcp_server()+ # Use http_app() which is the modern approach (sse_app is deprecated)+ app.mount("/mcp", mcp_server.http_app())+ logger.info("MCP server mounted at /mcp (streamable HTTP transport)")+ except Exception as e:+ logger.warning(f"Failed to mount MCP server: {e}")+protocol_version = 1try:from importlib.metadata import version@@ -266,6 +283,8 @@ async def status():features = []if HAS_AGENT:features.append("agent")+ if HAS_MCP:+ features.append("mcp")return {"status": "ok", "os_type": OS_TYPE, "features": features}diff --git a/libs/python/computer-server/pyproject.toml b/libs/python/computer-server/pyproject.tomlindex 1acc3c6f19..a759332c6b 100644--- a/libs/python/computer-server/pyproject.toml+++ b/libs/python/computer-server/pyproject.toml@@ -78,5 +78,5 @@ dev = [][tool.pdm.scripts]-api = "python -m computer_server"-mcp = "python -m computer_server --mcp"\ No newline at end of file+# Mode is auto-detected: TTY → HTTP server, pipe → MCP stdio+api = "python -m computer_server"\ No newline at end of filediff --git a/uv.lock b/uv.lockindex 40b9bb411d..852f8559fd 100644--- a/uv.lock+++ b/uv.lock@@ -1252,7 +1252,7 @@ provides-extras = ["lume", "lumier", "ui", "all"][[package]]name = "cua-computer-server"-version = "0.3.4"+version = "0.3.5"source = { editable = "libs/python/computer-server" }dependencies = [{ name = "aiohttp" },@@ -1274,49 +1274,47 @@ dependencies = [][package.optional-dependencies]-dev = [- { name = "mypy" },- { name = "ruff" },+linux = [+ { name = "python-xlib" },]-format = [- { name = "black" },- { name = "isort" },+macos = [+ { name = "pyobjc-framework-applicationservices" },+ { name = "pyobjc-framework-cocoa" },+ { name = "pyobjc-framework-quartz" },]mcp = [{ name = "fastmcp" },]-test = [- { name = "pytest" },- { name = "pytest-asyncio" },+windows = [+ { name = "pywin32" },][package.metadata]requires-dist = [{ name = "aiohttp", specifier = ">=3.9.1" },- { name = "black", marker = "extra == 'format'", specifier = ">=23.0.0" },{ name = "fastapi", specifier = ">=0.111.0" },{ name = "fastmcp", marker = "extra == 'mcp'", specifier = ">=2.0,<3" },- { name = "isort", marker = "extra == 'format'", specifier = ">=5.12.0" },- { name = "mypy", marker = "extra == 'dev'", specifier = ">=0.971" },{ name = "pillow", specifier = ">=10.2.0" },{ name = "playwright", specifier = ">=1.40.0" },{ name = "pydantic", specifier = ">=2.0.0" },{ name = "pynput", specifier = ">=1.8.1" },{ name = "pyobjc-framework-applicationservices", marker = "sys_platform == 'darwin'", specifier = ">=10.1" },+ { name = "pyobjc-framework-applicationservices", marker = "extra == 'macos'", specifier = ">=10.1" },{ name = "pyobjc-framework-cocoa", marker = "sys_platform == 'darwin'", specifier = ">=10.1" },+ { name = "pyobjc-framework-cocoa", marker = "extra == 'macos'", specifier = ">=10.1" },{ name = "pyobjc-framework-quartz", marker = "sys_platform == 'darwin'", specifier = ">=10.1" },+ { name = "pyobjc-framework-quartz", marker = "extra == 'macos'", specifier = ">=10.1" },{ name = "pyperclip", specifier = ">=1.9.0" },- { name = "pytest", marker = "extra == 'test'", specifier = ">=7.0.0" },- { name = "pytest-asyncio", marker = "extra == 'test'", specifier = ">=0.23.0" },{ name = "python-certifi-win32", marker = "sys_platform == 'win32'" },{ name = "python-xlib", marker = "sys_platform == 'linux'", specifier = ">=0.33" },+ { name = "python-xlib", marker = "extra == 'linux'", specifier = ">=0.33" },{ name = "pywin32", marker = "sys_platform == 'win32'", specifier = ">=310" },+ { name = "pywin32", marker = "extra == 'windows'", specifier = ">=310" },{ name = "pywinctl", specifier = ">=0.4.1" },- { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.0.241" },{ name = "uvicorn", extras = ["standard"], specifier = ">=0.27.0" },{ name = "websockets", specifier = ">=12.0" },]-provides-extras = ["dev", "format", "mcp", "test"]+provides-extras = ["mcp", "macos", "linux", "windows"][[package]]name = "cua-core"@@ -6044,10 +6042,10 @@ name = "pyobjc-framework-applicationservices"version = "12.1"source = { registry = "https://pypi.org/simple" }dependencies = [- { name = "pyobjc-core", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },- { name = "pyobjc-framework-cocoa", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },- { name = "pyobjc-framework-coretext", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },- { name = "pyobjc-framework-quartz", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },+ { name = "pyobjc-core" },+ { name = "pyobjc-framework-cocoa" },+ { name = "pyobjc-framework-coretext" },+ { name = "pyobjc-framework-quartz" },]sdist = { url = "https://files.pythonhosted.org/packages/be/6a/d4e613c8e926a5744fc47a9e9fea08384a510dc4f27d844f7ad7a2d793bd/pyobjc_framework_applicationservices-12.1.tar.gz", hash = "sha256:c06abb74f119bc27aeb41bf1aef8102c0ae1288aec1ac8665ea186a067a8945b", size = 103247, upload-time = "2025-11-14T10:08:52.18Z" }wheels = [@@ -6345,7 +6343,7 @@ name = "pyobjc-framework-cocoa"version = "12.1"source = { registry = "https://pypi.org/simple" }dependencies = [- { name = "pyobjc-core", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },+ { name = "pyobjc-core" },]sdist = { url = "https://files.pythonhosted.org/packages/02/a3/16ca9a15e77c061a9250afbae2eae26f2e1579eb8ca9462ae2d2c71e1169/pyobjc_framework_cocoa-12.1.tar.gz", hash = "sha256:5556c87db95711b985d5efdaaf01c917ddd41d148b1e52a0c66b1a2e2c5c1640", size = 2772191, upload-time = "2025-11-14T10:13:02.069Z" }wheels = [@@ -6625,9 +6623,9 @@ name = "pyobjc-framework-coretext"version = "12.1"source = { registry = "https://pypi.org/simple" }dependencies = [- { name = "pyobjc-core", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },- { name = "pyobjc-framework-cocoa", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },- { name = "pyobjc-framework-quartz", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },+ { name = "pyobjc-core" },+ { name = "pyobjc-framework-cocoa" },+ { name = "pyobjc-framework-quartz" },]sdist = { url = "https://files.pythonhosted.org/packages/29/da/682c9c92a39f713bd3c56e7375fa8f1b10ad558ecb075258ab6f1cdd4a6d/pyobjc_framework_coretext-12.1.tar.gz", hash = "sha256:e0adb717738fae395dc645c9e8a10bb5f6a4277e73cba8fa2a57f3b518e71da5", size = 90124, upload-time = "2025-11-14T10:14:38.596Z" }wheels = [@@ -7710,8 +7708,8 @@ name = "pyobjc-framework-quartz"version = "12.1"source = { registry = "https://pypi.org/simple" }dependencies = [- { name = "pyobjc-core", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },- { name = "pyobjc-framework-cocoa", marker = "(python_full_version < '3.13' and sys_platform != 'win32') or sys_platform == 'darwin'" },+ { name = "pyobjc-core" },+ { name = "pyobjc-framework-cocoa" },]sdist = { url = "https://files.pythonhosted.org/packages/94/18/cc59f3d4355c9456fc945eae7fe8797003c4da99212dd531ad1b0de8a0c6/pyobjc_framework_quartz-12.1.tar.gz", hash = "sha256:27f782f3513ac88ec9b6c82d9767eef95a5cf4175ce88a1e5a65875fee799608", size = 3159099, upload-time = "2025-11-14T10:21:24.31Z" }wheels = [