Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver-rs): unify browser_eval + page into one cross-platform tool (#1666)
Francesco Bonacci94f23d07ef25
diff --git a/libs/cua-driver-fixtures/README.md b/libs/cua-driver-fixtures/README.mdindex 7c76e907ca..30c9fa5eb2 100644--- a/libs/cua-driver-fixtures/README.md+++ b/libs/cua-driver-fixtures/README.md@@ -74,9 +74,9 @@ test (May 2026) showed need explicit harness coverage:Like `test_page.html`, each panel exposes a global accessor:`window.getGesturePanelState()` returns a JSON snapshot of all four-panels' state. Tests can poll it via `browser_eval` (when running with-Chromium's `--remote-debugging-port`) or read the visible status divs-via UIA / OCR.+panels' state. Tests can poll it via `page` with+`action: "execute_javascript"` (when running with Chromium's+`--remote-debugging-port`) or read the visible status divs via UIA / OCR.### Known browser-coverage gaps (May 2026)@@ -86,7 +86,7 @@ Windows, the integration tests should be aware of:- **GPU-composited content blanks PrintWindow.** `cua-driver screenshot`captures the browser chrome + a white body. Workaround: launch thetarget browser with `--remote-debugging-port=<port>` and use- `browser_eval` for both inputs and verification.+ `page` (`action: "execute_javascript"`) for both inputs and verification.- **Chromium auto-disables full a11y** unless an AT-grade client isdetected. UIA on the rendered DOM returns empty until the page isflagged accessibility-active. Workaround: enable Chromium'sdiff --git a/libs/cua-driver-rs/PARITY.md b/libs/cua-driver-rs/PARITY.mdindex 806a7bb80e..9a97b987e1 100644--- a/libs/cua-driver-rs/PARITY.md+++ b/libs/cua-driver-rs/PARITY.md@@ -20,10 +20,9 @@ cd libs/cua-driver-rs/tests/integration| Gap | Rust | Swift ||-----|------|-------|| `type_text_chars` | ✅ | missing |-| `browser_eval` | ✅ | missing || `get_accessibility_tree` | ✅ | missing |-| `page` tool | ✅ (registered as of this commit) | ✅ |-| `--version` flag | ✅ (fixed in this commit) | ✅ |+| `page` tool | ✅ (cross-platform: Apple-Events on macOS, UIA+CDP on Windows, AT-SPI+CDP on Linux) | ✅ (macOS only) |+| `--version` flag | ✅ | ✅ || `call check_permissions` JSON | ✅ JSON | human-readable text || `call screenshot` (no window_id) | ✅ full-display default | error (requires window_id) |@@ -1370,7 +1369,6 @@ older clients that only read name/description still work."id": 1,"result": {"tools": [- { "name": "browser_eval", "description": "…", "inputSchema": {…}, "annotations": {…} },{ "name": "check_permissions", "description": "…", "inputSchema": {…}, "annotations": {…} },{ "name": "click", "description": "…", "inputSchema": {…}, "annotations": {…} },{ "name": "double_click", "…": "…" },diff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md b/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.mdindex cf2734ce64..7be50ebf03 100644--- a/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md+++ b/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md@@ -387,6 +387,10 @@ they can cmd-tab back.## Browser JS primitives — Apple Events backend+The `page` tool is **cross-platform** — Windows uses UIA + CDP,+Linux uses AT-SPI + CDP, and macOS (this section) uses the+Apple-Events / CDP / AX-tree routing described below.+When the AX tree doesn't expose the data you need (common inChromium/Electron — the tree is sparse for web content), use the`page` tool or the `javascript` param on `get_window_state` to querydiff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/SKILL.md b/libs/cua-driver-rs/Skills/cua-driver-rs/SKILL.mdindex 96ea1d6ff6..3144129b8e 100644--- a/libs/cua-driver-rs/Skills/cua-driver-rs/SKILL.md+++ b/libs/cua-driver-rs/Skills/cua-driver-rs/SKILL.md@@ -418,8 +418,12 @@ silently no-ops — symptom is a system bell; use `set_value` or clicka clickable equivalent), scrolling via synthetic PageUp/Downkeystrokes, in-page clicks, and typing into web inputs.-Browser JS primitives (the `page` tool, `get_window_state(javascript=)`)-are macOS-only today — see `MACOS.md` for the Apple-Events backend.+Browser JS primitives are now **cross-platform** via the `page` tool —+macOS uses Apple Events for Chrome/Brave/Edge/Safari + CDP for Electron+(see `MACOS.md`); Windows + Linux use UIA / AT-SPI for `get_text` /+`query_dom` and the shared CDP client for `execute_javascript` (browser+must be launched with `--remote-debugging-port=N` and the port exported+as `CUA_DRIVER_CDP_PORT`).## Re-snapshot and verify — mandatorydiff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/WEB_APPS.md b/libs/cua-driver-rs/Skills/cua-driver-rs/WEB_APPS.mdindex 4a1004eb1a..d0e6588f99 100644--- a/libs/cua-driver-rs/Skills/cua-driver-rs/WEB_APPS.md+++ b/libs/cua-driver-rs/Skills/cua-driver-rs/WEB_APPS.md@@ -5,6 +5,13 @@> on macOS. For Windows web-app automation (Edge, Chrome with> PostMessage / WebView2), see `WINDOWS.md` → "Web apps on Windows". For> Linux: `LINUX.md` (BETA — limited browser automation today).+>+> The `page` tool itself is **cross-platform** — Windows + Linux back+> `get_text` / `query_dom` with UIA / AT-SPI respectively, and+> `execute_javascript` on those platforms uses CDP (browser must be+> launched with `--remote-debugging-port=N`, with the port exposed via+> the `CUA_DRIVER_CDP_PORT` env var). macOS routing (this doc) remains+> Apple Events → CDP → AX-tree fallback.Covers apps whose UI is rendered in a web runtime inside a nativemacOS shell:diff --git a/libs/cua-driver-rs/crates/cua-driver/tests/mcp_protocol_test.rs b/libs/cua-driver-rs/crates/cua-driver/tests/mcp_protocol_test.rsindex e95ee93ff0..ea368c3514 100644--- a/libs/cua-driver-rs/crates/cua-driver/tests/mcp_protocol_test.rs+++ b/libs/cua-driver-rs/crates/cua-driver/tests/mcp_protocol_test.rs@@ -325,7 +325,7 @@ fn test_all_expected_tools_registered() {"get_agent_cursor_state", "check_permissions", "get_config", "set_config","get_accessibility_tree","set_recording", "get_recording_state", "replay_trajectory",- "browser_eval",+ "page",];for name in &expected {assert!(names.contains(name), "Missing tool: {name} (registered: {names:?})");@@ -940,9 +940,9 @@ fn test_replay_trajectory() {#[test]#[cfg(target_os = "macos")]-fn test_browser_eval_no_cdp_error() {- //! Verify browser_eval is registered and returns a meaningful error when- //! no Chromium process is listening on the given cdp_port.+fn test_page_unknown_action_error() {+ //! Verify the cross-platform `page` tool is registered and rejects an+ //! unknown action with a meaningful error.let binary = binary_path();if !binary.exists() { return; }@@ -955,25 +955,24 @@ fn test_browser_eval_no_cdp_error() {send_request(stdin, &serde_json::json!({"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}));read_response(&mut stdout);- // Use a port that is almost certainly not listening.send_request(stdin, &serde_json::json!({"jsonrpc":"2.0","id":2,"method":"tools/call",- "params":{"name":"browser_eval","arguments":{- "expression": "1+1",- "cdp_port": 19722+ "params":{"name":"page","arguments":{+ "pid": 1,+ "window_id": 0,+ "action": "definitely_not_a_real_action"}}}));let resp = read_response(&mut stdout);- // The tool should return isError=true with a connection-refused message.assert!(resp["result"]["isError"].as_bool().unwrap_or(false),- "expected isError=true when CDP port is not listening: {resp:?}"+ "expected isError=true for unknown action: {resp:?}");let text = resp["result"]["content"][0]["text"].as_str().unwrap_or("");assert!(- text.contains("browser_eval"),- "error text should mention browser_eval: {text}"+ text.to_ascii_lowercase().contains("unknown action"),+ "error text should mention 'Unknown action': {text}");child.kill().ok();@@ -2128,7 +2127,7 @@ fn test_all_expected_tools_registered_windows() {"get_agent_cursor_state", "check_permissions", "get_config", "set_config","get_accessibility_tree","set_recording", "get_recording_state", "replay_trajectory",- "browser_eval",+ "page",];for name in &expected {assert!(names.contains(name), "Missing tool: {name} (registered: {names:?})");@@ -3009,7 +3008,9 @@ fn test_replay_trajectory_windows() {#[test]#[cfg(target_os = "windows")]-fn test_browser_eval_no_cdp_error_windows() {+fn test_page_unknown_action_error_windows() {+ //! Verify the cross-platform `page` tool is registered on Windows and+ //! rejects an unknown action with a meaningful error.let binary = binary_path_windows();if !binary.exists() { return; }@@ -3024,15 +3025,20 @@ fn test_browser_eval_no_cdp_error_windows() {send_request(stdin, &serde_json::json!({"jsonrpc":"2.0","id":2,"method":"tools/call",- "params":{"name":"browser_eval","arguments":{"expression":"1+1","cdp_port":19722}}+ "params":{"name":"page","arguments":{+ "pid":1,"window_id":0,"action":"definitely_not_a_real_action"+ }}}));let resp = read_response(&mut stdout);assert!(resp["result"]["isError"].as_bool().unwrap_or(false),- "expected isError=true when CDP port not listening: {resp:?}"+ "expected isError=true for unknown action: {resp:?}");let text = resp["result"]["content"][0]["text"].as_str().unwrap_or("");- assert!(text.contains("browser_eval"), "error text should mention browser_eval: {text}");+ assert!(+ text.to_ascii_lowercase().contains("unknown action"),+ "error text should mention 'Unknown action': {text}"+ );child.kill().ok();}diff --git a/libs/cua-driver-rs/crates/mcp-server/src/browser_eval.rs b/libs/cua-driver-rs/crates/mcp-server/src/cdp.rssimilarity index 59%rename from libs/cua-driver-rs/crates/mcp-server/src/browser_eval.rsrename to libs/cua-driver-rs/crates/mcp-server/src/cdp.rsindex 7cea2b52d1..579861e544 100644--- a/libs/cua-driver-rs/crates/mcp-server/src/browser_eval.rs+++ b/libs/cua-driver-rs/crates/mcp-server/src/cdp.rs@@ -1,104 +1,55 @@-//! `browser_eval` tool — Chromium CDP Runtime.evaluate over a local WebSocket.+//! Cross-platform CDP (Chrome DevTools Protocol) helper.//!-//! Cross-platform: all I/O is plain TCP to localhost (no external HTTP/WS crates).-//! Mirrors the Windows C# reference in CuaDriver.Win/Tools/BrowserEvalTool.cs-//! and CuaDriver.Win/Browser/CdpBrowserBridge.cs.+//! A self-contained `Runtime.evaluate` client that speaks raw TCP — no+//! external HTTP/WS crates. Used by the cross-platform `page` tool's+//! `execute_javascript` action on Windows/Linux (and by the macOS Electron+//! backend, which retains its own copy under `platform-macos/src/browser/`+//! that predates this helper).//!//! Flow:-//! 1. HTTP GET http://127.0.0.1:{cdp_port}/json → discover page WebSocket URLs.-//! 2. WebSocket connect to the first (or best-matched) page URL.+//! 1. HTTP GET http://127.0.0.1:{cdp_port}/json → discover page WS URLs.+//! 2. WebSocket connect to the first page URL.//! 3. Send Runtime.evaluate with userGesture=true.//! 4. Read frames until we receive the response with id=1.-use async_trait::async_trait;use serde_json::Value;-use crate::{- protocol::ToolResult,- tool::{Tool, ToolDef},-};--pub struct BrowserEvalTool;--#[async_trait]-impl Tool for BrowserEvalTool {- fn def(&self) -> &ToolDef {- static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();- DEF.get_or_init(|| ToolDef {- name: "browser_eval".into(),- description: "Evaluate JavaScript in a Chromium browser via CDP Runtime.evaluate \- with userGesture=true. Requires cdp_port. Pass window_id to target \- a specific browser window/tab (matched by title heuristic).".into(),- input_schema: serde_json::json!({- "type": "object",- "required": ["expression"],- "properties": {- "expression": {- "type": "string",- "description": "JavaScript expression to evaluate."- },- "cdp_port": {- "type": "integer",- "description": "Chromium remote debugging port (--remote-debugging-port)."- },- "window_id": {- "type": "integer",- "description": "Optional browser window/HWND/XID used to bind CDP to the matching tab."- },- "await_promise": {- "type": "boolean",- "description": "Await a returned Promise (default true)."- }- }- }),- read_only: false,- destructive: true,- idempotent: false,- open_world: true,- })- }-- async fn invoke(&self, args: Value) -> ToolResult {- let expression = match args.get("expression").and_then(|v| v.as_str()) {- Some(e) => e.to_string(),- None => return ToolResult::error("browser_eval: 'expression' is required"),- };- let cdp_port = match args.get("cdp_port").and_then(|v| v.as_u64()) {- Some(p) if p > 0 && p < 65536 => p as u16,- _ => return ToolResult::error(- "browser_eval: 'cdp_port' is required (Chromium --remote-debugging-port value)"- ),- };- let await_promise = args.get("await_promise")- .and_then(|v| v.as_bool())- .unwrap_or(true);- let window_title_hint = args.get("window_id").map(|_| ""); // reserved for future matching-- match cdp_evaluate(cdp_port, window_title_hint, &expression, await_promise).await {- Ok(response) => {- let result_text = format_cdp_result(&response);- ToolResult::text(format!("cdp.runtime.evaluate.user_gesture: {result_text}"))- }- Err(e) => ToolResult::error(format!("browser_eval: {e}")),- }- }+/// Evaluate `expression` in the first page tab exposed by a Chromium-family+/// browser listening on `port` (the value passed to `--remote-debugging-port`).+///+/// Returns the formatted result (or `exception: ...` if the evaluated code+/// raised). Caller is responsible for prefixing the human-readable label+/// (e.g. `"cdp.runtime.evaluate.user_gesture: ..."`).+pub async fn evaluate(+ port: u16,+ expression: &str,+ await_promise: bool,+) -> anyhow::Result<String> {+ let response = cdp_evaluate(port, expression, await_promise).await?;+ Ok(format_cdp_result(&response))}// ── High-level evaluate ───────────────────────────────────────────────────async fn cdp_evaluate(port: u16,- _window_title_hint: Option<&str>,expression: &str,await_promise: bool,) -> anyhow::Result<Value> {- // Discover page tabs.- let pages = cdp_list_pages(port).await?;+ // Wrap the /json discovery in its own timeout — `cdp_list_pages` does an+ // unbounded `read_to_end`, and a half-open localhost socket (browser+ // mid-shutdown, firewall mismatch, port stolen by another process) would+ // otherwise hang us forever. 10 s is generous for a localhost HTTP roundtrip.+ let pages = tokio::time::timeout(+ std::time::Duration::from_secs(10),+ cdp_list_pages(port),+ )+ .await+ .map_err(|_| anyhow::anyhow!("CDP /json discovery on port {port} timed out after 10 s"))??;if pages.is_empty() {anyhow::bail!("No CDP page tabs found on port {port}");}- // Take the first available page (future: match by window title hint).let ws_url = pages[0].get("webSocketDebuggerUrl").and_then(|u| u.as_str())@@ -115,7 +66,6 @@ async fn cdp_evaluate(}});- // 30-second wall-clock timeout (covers awaited promises).tokio::time::timeout(std::time::Duration::from_secs(30),cdp_ws_call(port, &ws_url, &cmd.to_string()),@@ -125,7 +75,6 @@ async fn cdp_evaluate(}fn format_cdp_result(response: &Value) -> String {- // If the result contains an exception, surface it.if let Some(exc) = response.get("result").and_then(|r| r.get("exceptionDetails"))@@ -134,11 +83,11 @@ fn format_cdp_result(response: &Value) -> String {.get("exception").and_then(|e| e.get("description")).and_then(|d| d.as_str())- .unwrap_or_else(|| exc.to_string().leak()); // static lifetime for error msg+ .map(str::to_owned)+ .unwrap_or_else(|| exc.to_string());return format!("exception: {msg}");}- // Extract the evaluated value.if let Some(result) = response.get("result").and_then(|r| r.get("result")) {if let Some(v) = result.get("value") {return v.to_string();@@ -153,12 +102,11 @@ fn format_cdp_result(response: &Value) -> String {// ── HTTP page discovery ───────────────────────────────────────────────────-/// GET http://127.0.0.1:{port}/json → array of page descriptors.-/// Returns only entries whose `type` == "page" and that have a webSocketDebuggerUrl.async fn cdp_list_pages(port: u16) -> anyhow::Result<Vec<Value>> {use tokio::io::{AsyncReadExt, AsyncWriteExt};- let mut stream = tokio::net::TcpStream::connect(("127.0.0.1", port)).await+ let mut stream = tokio::net::TcpStream::connect(("127.0.0.1", port))+ .await.map_err(|e| anyhow::anyhow!("Cannot connect to CDP on port {port}: {e}"))?;let req = format!(@@ -173,7 +121,8 @@ async fn cdp_list_pages(port: u16) -> anyhow::Result<Vec<Value>> {.map_err(|_| anyhow::anyhow!("CDP /json response is not valid UTF-8"))?;let body_start = text.find("\r\n\r\n")- .ok_or_else(|| anyhow::anyhow!("Invalid HTTP response from CDP"))? + 4;+ .ok_or_else(|| anyhow::anyhow!("Invalid HTTP response from CDP"))?+ + 4;let all: Value = serde_json::from_str(&text[body_start..]).map_err(|e| anyhow::anyhow!("CDP /json parse error: {e}"))?;@@ -196,13 +145,9 @@ async fn cdp_list_pages(port: u16) -> anyhow::Result<Vec<Value>> {// ── WebSocket CDP call ────────────────────────────────────────────────────-/// Open a WebSocket to `ws_url` (on `port`), send `message`, and return the-/// first response frame whose JSON `id` field equals 1.-/// Intermediate event notifications (no `id` field or id != 1) are silently skipped.async fn cdp_ws_call(port: u16, ws_url: &str, message: &str) -> anyhow::Result<Value> {use tokio::io::{AsyncReadExt, AsyncWriteExt};- // Extract the URL path from e.g. "ws://127.0.0.1:9222/devtools/page/ABC".let path: String = if let Some(rest) = ws_url.strip_prefix("ws://") {rest.splitn(2, '/').nth(1)@@ -212,11 +157,10 @@ async fn cdp_ws_call(port: u16, ws_url: &str, message: &str) -> anyhow::Result<Vws_url.to_string()};- let mut stream = tokio::net::TcpStream::connect(("127.0.0.1", port)).await+ let mut stream = tokio::net::TcpStream::connect(("127.0.0.1", port))+ .await.map_err(|e| anyhow::anyhow!("Cannot connect to CDP WebSocket on port {port}: {e}"))?;- // WebSocket HTTP upgrade.- // The Sec-WebSocket-Key is a static value — fine for localhost non-security use.let handshake = format!("GET {path} HTTP/1.1\r\n\Host: 127.0.0.1:{port}\r\n\@@ -228,7 +172,6 @@ async fn cdp_ws_call(port: u16, ws_url: &str, message: &str) -> anyhow::Result<V);stream.write_all(handshake.as_bytes()).await?;- // Read until end of HTTP response headers.let mut hdr = Vec::with_capacity(512);loop {let mut b = [0u8; 1];@@ -248,25 +191,23 @@ async fn cdp_ws_call(port: u16, ws_url: &str, message: &str) -> anyhow::Result<V);}- // Send the CDP command as a masked WebSocket text frame.ws_write_text(&mut stream, message.as_bytes()).await?;- // Read frames, skipping events, until we receive id=1.loop {match ws_read_frame(&mut stream).await? {WsFrame::Text(text) => {- // Try to parse as JSON and check for id == 1.if let Ok(val) = serde_json::from_str::<Value>(&text) {if val.get("id").and_then(|id| id.as_u64()) == Some(1) {return Ok(val);}- // Event notification — ignore and loop.} else {- anyhow::bail!("CDP returned non-JSON frame: {}", &text[..text.len().min(120)]);+ anyhow::bail!(+ "CDP returned non-JSON frame: {}",+ &text[..text.len().min(120)]+ );}}WsFrame::Ping(data) => {- // Respond with pong (RFC 6455 §5.5.3).ws_write_pong(&mut stream, &data).await?;}WsFrame::Close => {@@ -286,17 +227,15 @@ enum WsFrame {Other,}-/// Read one complete (non-fragmented) WebSocket frame.async fn ws_read_frame(stream: &mut tokio::net::TcpStream) -> anyhow::Result<WsFrame> {use tokio::io::AsyncReadExt;- // First 2 bytes: FIN+RSV+opcode, MASK+payload_len.let mut hdr = [0u8; 2];stream.read_exact(&mut hdr).await?;- let opcode = hdr[0] & 0x0F;- let masked = (hdr[1] & 0x80) != 0;- let len_byte = (hdr[1] & 0x7F) as usize;+ let opcode = hdr[0] & 0x0F;+ let masked = (hdr[1] & 0x80) != 0;+ let len_byte = (hdr[1] & 0x7F) as usize;let payload_len: usize = match len_byte {0..=125 => len_byte,@@ -308,7 +247,6 @@ async fn ws_read_frame(stream: &mut tokio::net::TcpStream) -> anyhow::Result<WsF_ => {let mut b = [0u8; 8];stream.read_exact(&mut b).await?;- // Guard against absurdly large CDP messages (>16 MiB).let n = u64::from_be_bytes(b) as usize;if n > 16 * 1024 * 1024 {anyhow::bail!("CDP WebSocket frame too large ({n} bytes)");@@ -334,26 +272,23 @@ async fn ws_read_frame(stream: &mut tokio::net::TcpStream) -> anyhow::Result<WsF}match opcode {- 1 => Ok(WsFrame::Text(- String::from_utf8(payload)- .map_err(|_| anyhow::anyhow!("CDP returned non-UTF-8 text frame"))?,- )),- 2 => Ok(WsFrame::Other), // binary — CDP doesn't use this+ 1 => Ok(WsFrame::Text(String::from_utf8(payload).map_err(|_| {+ anyhow::anyhow!("CDP returned non-UTF-8 text frame")+ })?)),+ 2 => Ok(WsFrame::Other),8 => Ok(WsFrame::Close),9 => Ok(WsFrame::Ping(payload)),_ => Ok(WsFrame::Other),}}-/// Write a masked WebSocket text frame (client → server, RFC 6455 §5.3 requires masking).async fn ws_write_text(stream: &mut tokio::net::TcpStream, payload: &[u8]) -> anyhow::Result<()> {use tokio::io::AsyncWriteExt;- // Fixed mask key — fine for a local loopback connection.const MASK: [u8; 4] = [0xDE, 0xAD, 0xBE, 0xEF];let len = payload.len();let mut frame = Vec::with_capacity(len + 10);- frame.push(0x81u8); // FIN=1, opcode=TEXT(1)+ frame.push(0x81u8);encode_len_masked(&mut frame, len);frame.extend_from_slice(&MASK);for (i, &b) in payload.iter().enumerate() {@@ -363,15 +298,14 @@ async fn ws_write_text(stream: &mut tokio::net::TcpStream, payload: &[u8]) -> anOk(())}-/// Write a masked WebSocket pong frame in response to a ping.async fn ws_write_pong(stream: &mut tokio::net::TcpStream, payload: &[u8]) -> anyhow::Result<()> {use tokio::io::AsyncWriteExt;const MASK: [u8; 4] = [0xCA, 0xFE, 0xBA, 0xBE];- let len = payload.len().min(125); // control frames ≤ 125 bytes (RFC 6455 §5.5)+ let len = payload.len().min(125);let mut frame = Vec::with_capacity(len + 6);- frame.push(0x8Au8); // FIN=1, opcode=PONG(0xA)- frame.push(0x80 | len as u8); // MASK bit + length+ frame.push(0x8Au8);+ frame.push(0x80 | len as u8);frame.extend_from_slice(&MASK);for (i, &b) in payload[..len].iter().enumerate() {frame.push(b ^ MASK[i % 4]);@@ -380,7 +314,6 @@ async fn ws_write_pong(stream: &mut tokio::net::TcpStream, payload: &[u8]) -> anOk(())}-/// Encode a (masked) length field per RFC 6455 §5.2.fn encode_len_masked(buf: &mut Vec<u8>, len: usize) {if len < 126 {buf.push(0x80 | len as u8);diff --git a/libs/cua-driver-rs/crates/mcp-server/src/lib.rs b/libs/cua-driver-rs/crates/mcp-server/src/lib.rsindex 5815e7e9f5..acf3d94279 100644--- a/libs/cua-driver-rs/crates/mcp-server/src/lib.rs+++ b/libs/cua-driver-rs/crates/mcp-server/src/lib.rs@@ -10,7 +10,8 @@//! - Each request has `jsonrpc: "2.0"`, `id` (any), `method`, optional `params`//! - Notifications (no `id`) are silently ignored-pub mod browser_eval;+pub mod cdp;+pub mod page;pub mod protocol;pub mod recording;pub mod recording_tools;diff --git a/libs/cua-driver-rs/crates/mcp-server/src/page.rs b/libs/cua-driver-rs/crates/mcp-server/src/page.rsnew file mode 100644index 0000000000..d064ea3dbb--- /dev/null+++ b/libs/cua-driver-rs/crates/mcp-server/src/page.rs@@ -0,0 +1,273 @@+//! Cross-platform `page` tool: browser JS execution, text extraction, and DOM querying.+//!+//! The tool itself (name, schema, action dispatch) lives here so every platform+//! gets the same MCP surface. Each platform crate supplies a `PageBackend` impl+//! that routes the read/write primitives to platform-native code:+//!+//! - **macOS** uses Apple Events for Chromium/Safari, CDP for Electron, and+//! the AX tree as a WKWebView fallback. See+//! `platform-macos/src/tools/page.rs`.+//! - **Windows** uses UIA TextPattern + FindAll for `get_text` / `query_dom`,+//! and the shared CDP client (`crate::cdp`) for `execute_javascript`.+//! - **Linux** uses AT-SPI for read paths, CDP for JS exec.+//!+//! `enable_javascript_apple_events` is intentionally macOS-only; the default+//! trait impl returns "not supported on this platform" so non-macOS callers+//! get a clear error instead of a panic.++use async_trait::async_trait;+use serde_json::Value;+use std::sync::Arc;++use crate::{+ protocol::ToolResult,+ tool::{Tool, ToolDef},+};++/// Platform-specific backend for the `page` tool.+///+/// All methods take `(pid, window_id)` for consistency across actions.+/// `bundle_id` is resolved by the backend if needed (macOS uses it for+/// AppleScript routing; Windows/Linux ignore it).+#[async_trait]+pub trait PageBackend: Send + Sync {+ /// Returns the visible text of the page (rough analog of+ /// `document.body.innerText`).+ async fn get_text(&self, pid: i32, window_id: u32) -> anyhow::Result<String>;++ /// Find elements matching `css_selector` and return a formatted-text+ /// response (same human-readable shape macOS already emits).+ async fn query_dom(+ &self,+ pid: i32,+ window_id: u32,+ css_selector: &str,+ attributes: &[String],+ ) -> anyhow::Result<String>;++ /// Evaluate `javascript` against the page and return the stringified+ /// result. Backends without a JS path should return an actionable error.+ async fn execute_javascript(+ &self,+ pid: i32,+ window_id: u32,+ javascript: &str,+ ) -> anyhow::Result<String>;++ /// Whether this backend can execute JS at all. False → the tool short-+ /// circuits with a "not supported here" error; true → the backend may+ /// still return an error at call time (e.g. CDP port not reachable).+ fn supports_javascript(&self) -> bool {+ true+ }++ /// macOS-only: enable "Allow JavaScript from Apple Events" on the named+ /// browser bundle. Default impl returns an unsupported-platform error.+ async fn enable_javascript_apple_events(&self, _bundle_id: &str) -> anyhow::Result<String> {+ anyhow::bail!(+ "enable_javascript_apple_events is only supported on macOS \+ (Chrome/Brave/Edge use Apple Events as the JS backend there)."+ )+ }+}++/// The cross-platform `page` MCP tool. Holds a backend trait object the+/// host platform constructs and passes in at registration time.+pub struct PageTool {+ backend: Arc<dyn PageBackend>,+}++impl PageTool {+ pub fn new(backend: Arc<dyn PageBackend>) -> Self {+ Self { backend }+ }+}++fn def() -> &'static ToolDef {+ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();+ DEF.get_or_init(|| ToolDef {+ name: "page".into(),+ description: "Interact with the browser page loaded in a running app. Supports \+ Chrome, Brave, Edge, Safari (via AppleScript on macOS), Electron apps (via CDP), \+ Chromium/Firefox on Windows (via UIA for read; CDP for execute_javascript when \+ --remote-debugging-port is set), and WKWebView/Tauri/AT-SPI fallbacks.\n\n\+ Actions:\n\+ - execute_javascript: Run JS and return the result.\n\+ - get_text: Extract visible text from the page.\n\+ - query_dom: Find elements matching a CSS selector.\n\+ - enable_javascript_apple_events: macOS-only — patch the browser's \+ Preferences to allow JS from Apple Events (Chrome/Brave/Edge, requires user \+ confirmation and a browser restart).".into(),+ input_schema: serde_json::json!({+ "type": "object",+ // `pid` and `window_id` are required for every action except+ // `enable_javascript_apple_events` (which targets a browser by+ // bundle id, not a running process). Enforced per-action in+ // `invoke` rather than via the schema so the JSON-Schema-only+ // path still validates the common case.+ "required": ["action"],+ "properties": {+ "pid": { "type": "integer", "description": "Target process ID." },+ "window_id": { "type": "integer", "description": "Target window ID from list_windows." },+ "action": {+ "type": "string",+ "enum": ["execute_javascript", "get_text", "query_dom", "enable_javascript_apple_events"],+ "description": "Action to perform."+ },+ "javascript": {+ "type": "string",+ "description": "JavaScript to execute. Required for execute_javascript."+ },+ "css_selector": {+ "type": "string",+ "description": "CSS selector for query_dom (e.g. 'a', 'button', 'input', 'h1'-'h6', 'p', 'img', 'select', '*')."+ },+ "attributes": {+ "type": "array",+ "items": { "type": "string" },+ "description": "Element attributes to include in query_dom results."+ },+ "bundle_id": {+ "type": "string",+ "description": "Bundle ID of the browser. Required for enable_javascript_apple_events (macOS only)."+ },+ "user_has_confirmed_enabling": {+ "type": "boolean",+ "description": "Must be true to proceed with enable_javascript_apple_events. \+ This will quit and relaunch the browser."+ }+ },+ "additionalProperties": false+ }),+ read_only: false,+ destructive: false,+ idempotent: false,+ open_world: false,+ })+}++#[async_trait]+impl Tool for PageTool {+ fn def(&self) -> &ToolDef {+ def()+ }++ async fn invoke(&self, args: Value) -> ToolResult {+ let action = match args.get("action").and_then(|v| v.as_str()) {+ Some(v) => v.to_owned(),+ None => return ToolResult::error("Missing required parameter: action"),+ };++ // `pid` / `window_id` are resolved per-action: every action except+ // `enable_javascript_apple_events` needs both. We resolve once here+ // so each arm can `?` on the Result and we get matching error text.+ // Narrowing casts use `TryFrom` so out-of-range JSON numbers fail+ // with an actionable error instead of silently truncating to the+ // wrong process / window.+ let resolve_pid = |args: &Value| -> Result<i32, String> {+ let raw = args+ .get("pid")+ .and_then(|v| v.as_i64())+ .ok_or_else(|| "Missing required parameter: pid".to_owned())?;+ i32::try_from(raw).map_err(|_| format!("Invalid parameter: pid {raw} out of i32 range"))+ };+ let resolve_window_id = |args: &Value| -> Result<u32, String> {+ let raw = args+ .get("window_id")+ .and_then(|v| v.as_u64())+ .ok_or_else(|| "Missing required parameter: window_id".to_owned())?;+ u32::try_from(raw)+ .map_err(|_| format!("Invalid parameter: window_id {raw} out of u32 range"))+ };++ let (pid, window_id) = if action == "enable_javascript_apple_events" {+ (0i32, 0u32) // unused+ } else {+ let pid = match resolve_pid(&args) {+ Ok(v) => v,+ Err(e) => return ToolResult::error(e),+ };+ let window_id = match resolve_window_id(&args) {+ Ok(v) => v,+ Err(e) => return ToolResult::error(e),+ };+ (pid, window_id)+ };++ match action.as_str() {+ "enable_javascript_apple_events" => {+ let confirmed = args+ .get("user_has_confirmed_enabling")+ .and_then(|v| v.as_bool())+ .unwrap_or(false);+ if !confirmed {+ return ToolResult::error(+ "Set user_has_confirmed_enabling=true to proceed. \+ This will quit and relaunch the browser.",+ );+ }+ let bid = match args.get("bundle_id").and_then(|v| v.as_str()) {+ Some(s) if !s.is_empty() => s.to_owned(),+ _ => {+ return ToolResult::error(+ "bundle_id is required for enable_javascript_apple_events",+ )+ }+ };+ match self.backend.enable_javascript_apple_events(&bid).await {+ Ok(msg) => ToolResult::text(msg),+ Err(e) => ToolResult::error(format!("{e}")),+ }+ }++ "execute_javascript" => {+ if !self.backend.supports_javascript() {+ return ToolResult::error(+ "execute_javascript is not supported by this platform's page backend. \+ Use get_text or query_dom for read-only access.",+ );+ }+ let js = match args.get("javascript").and_then(|v| v.as_str()) {+ Some(v) => v.to_owned(),+ None => return ToolResult::error("Missing required parameter: javascript"),+ };+ match self.backend.execute_javascript(pid, window_id, &js).await {+ Ok(result) => ToolResult::text(result),+ Err(e) => ToolResult::error(format!("{e}")),+ }+ }++ "get_text" => match self.backend.get_text(pid, window_id).await {+ Ok(text) => ToolResult::text(text),+ Err(e) => ToolResult::error(format!("Page text extraction failed: {e}")),+ },++ "query_dom" => {+ let selector = args+ .get("css_selector")+ .and_then(|v| v.as_str())+ .unwrap_or("*")+ .to_owned();+ let attributes: Vec<String> = args+ .get("attributes")+ .and_then(|v| v.as_array())+ .map(|arr| {+ arr.iter()+ .filter_map(|v| v.as_str().map(str::to_owned))+ .collect()+ })+ .unwrap_or_default();+ match self+ .backend+ .query_dom(pid, window_id, &selector, &attributes)+ .await+ {+ Ok(text) => ToolResult::text(text),+ Err(e) => ToolResult::error(format!("DOM query failed: {e}")),+ }+ }++ other => ToolResult::error(format!("Unknown action: {other}")),+ }+ }+}diff --git a/libs/cua-driver-rs/crates/mcp-server/src/tool.rs b/libs/cua-driver-rs/crates/mcp-server/src/tool.rsindex da020ab8ea..b052fbd8dc 100644--- a/libs/cua-driver-rs/crates/mcp-server/src/tool.rs+++ b/libs/cua-driver-rs/crates/mcp-server/src/tool.rs@@ -7,7 +7,6 @@ use async_trait::async_trait;use serde_json::Value;use crate::{- browser_eval::BrowserEvalTool,protocol::{Content, ToolResult},recording::{now_ms, RecordingSession},recording_tools::{@@ -82,7 +81,6 @@ impl ToolRegistry {self.register(Box::new(SetRecordingTool::new(session.clone())));self.register(Box::new(GetRecordingStateTool::new(session)));self.register(Box::new(ReplayTrajectoryTool));- self.register(Box::new(BrowserEvalTool));}/// Wire up the replay tool's weak self-reference.diff --git a/libs/cua-driver-rs/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver-rs/crates/platform-linux/src/tools/impl_.rsindex 59fd70de3b..df4339b9ad 100644--- a/libs/cua-driver-rs/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver-rs/crates/platform-linux/src/tools/impl_.rs@@ -2026,6 +2026,11 @@ pub fn build_registry() -> ToolRegistry {r.register(Box::new(GetAccessibilityTreeTool));r.register(Box::new(ZoomTool { state: state.clone() }));r.register(Box::new(TypeTextCharsTool));+ // Cross-platform `page` tool definition lives in mcp-server; Linux plugs+ // in its AT-SPI + CDP backend here.+ r.register(Box::new(mcp_server::page::PageTool::new(+ Arc::new(super::page::LinuxPageBackend::new()),+ )));r.register_recording_tools();r}diff --git a/libs/cua-driver-rs/crates/platform-linux/src/tools/mod.rs b/libs/cua-driver-rs/crates/platform-linux/src/tools/mod.rsindex f511768c65..ea79d24754 100644--- a/libs/cua-driver-rs/crates/platform-linux/src/tools/mod.rs+++ b/libs/cua-driver-rs/crates/platform-linux/src/tools/mod.rs@@ -7,6 +7,8 @@ use mcp_server::tool::ToolRegistry;#[cfg(target_os = "linux")]mod impl_;+#[cfg(target_os = "linux")]+pub(crate) mod page;#[cfg(not(target_os = "linux"))]mod stubs;diff --git a/libs/cua-driver-rs/crates/platform-linux/src/tools/page.rs b/libs/cua-driver-rs/crates/platform-linux/src/tools/page.rsnew file mode 100644index 0000000000..5c8cab1037--- /dev/null+++ b/libs/cua-driver-rs/crates/platform-linux/src/tools/page.rs@@ -0,0 +1,176 @@+//! Linux implementation of the cross-platform `PageBackend` trait.+//!+//! Read paths use AT-SPI via the existing `walk_tree` infrastructure+//! (`crate::atspi`). `execute_javascript` calls the shared+//! `mcp_server::cdp` helper — Chromium/Electron must be launched with+//! `--remote-debugging-port=N` for that to succeed.++use async_trait::async_trait;+use mcp_server::page::PageBackend;++pub struct LinuxPageBackend;++impl LinuxPageBackend {+ pub fn new() -> Self {+ Self+ }+}++impl Default for LinuxPageBackend {+ fn default() -> Self {+ Self::new()+ }+}++#[async_trait]+impl PageBackend for LinuxPageBackend {+ async fn get_text(&self, pid: i32, window_id: u32) -> anyhow::Result<String> {+ let pid_u = pid as u32;+ let xid = window_id as u64;+ let result = tokio::task::spawn_blocking(move || {+ crate::atspi::walk_tree(pid_u, xid, None)+ })+ .await+ .map_err(|e| anyhow::anyhow!("AT-SPI walk task failed: {e}"))?;+ Ok(extract_text_from_markdown(&result.tree_markdown))+ }++ async fn query_dom(+ &self,+ pid: i32,+ window_id: u32,+ css_selector: &str,+ _attributes: &[String],+ ) -> anyhow::Result<String> {+ let pid_u = pid as u32;+ let xid = window_id as u64;+ let selector = css_selector.to_owned();+ let result = tokio::task::spawn_blocking(move || {+ crate::atspi::walk_tree(pid_u, xid, None)+ })+ .await+ .map_err(|e| anyhow::anyhow!("AT-SPI walk task failed: {e}"))?;++ if selector.contains("[data-") {+ anyhow::bail!(+ "`[data-*]` selectors are not reachable via AT-SPI on Linux. \+ Use `execute_javascript` for full DOM access (requires the browser \+ launched with `--remote-debugging-port`)."+ );+ }++ let role = role_for_selector(&selector);+ let selector_is_wildcard = {+ let s = selector.trim();+ s.is_empty() || s == "*"+ };+ // `role_for_selector` collapses both "wildcard / match-all" AND+ // "unrecognised" into `None`. The first should keep every node; the+ // second should fail loudly so an agent doesn't get a misleading+ // dump of every element on the page when its `.foo` / `#bar` query+ // wasn't actually understood.+ if role.is_none() && !selector_is_wildcard {+ anyhow::bail!(+ "Selector '{selector}' is not supported by Linux AT-SPI role mapping. \+ Use a simple tag selector (a, button, input, textarea, h1-h6, img, li, \+ p, select, *), or `execute_javascript` for full DOM access (requires the \+ browser launched with `--remote-debugging-port`)."+ );+ }+ let lines: Vec<String> = result+ .nodes+ .iter()+ .filter(|n| match role.as_deref() {+ Some(r) => n.role.eq_ignore_ascii_case(r),+ None => true, // safe now — only reached when selector is wildcard+ })+ .map(|n| {+ let name = n.name.as_deref().unwrap_or("");+ let value = n.value.as_deref().unwrap_or("");+ let index = n+ .element_index+ .map(|i| format!("[{i}] "))+ .unwrap_or_default();+ if value.is_empty() {+ format!("- {index}{} \"{}\"", n.role, name)+ } else {+ format!("- {index}{} \"{}\" = \"{}\"", n.role, name, value)+ }+ })+ .collect();++ if lines.is_empty() {+ Ok("No elements found.".to_owned())+ } else {+ Ok(lines.join("\n"))+ }+ }++ async fn execute_javascript(+ &self,+ _pid: i32,+ _window_id: u32,+ javascript: &str,+ ) -> anyhow::Result<String> {+ let port: u16 = match std::env::var("CUA_DRIVER_CDP_PORT")+ .ok()+ .and_then(|s| s.parse::<u16>().ok())+ {+ Some(p) => p,+ None => anyhow::bail!(+ "Chromium's `execute_javascript` on Linux requires the browser launched \+ with `--remote-debugging-port=N`. Relaunch the browser with the flag and \+ export CUA_DRIVER_CDP_PORT=N before starting cua-driver, or use \+ `get_text` / `query_dom` for read-only access."+ ),+ };+ let result = mcp_server::cdp::evaluate(port, javascript, true).await?;+ Ok(format!("cdp.runtime.evaluate.user_gesture: {result}"))+ }+}++/// Map common CSS selectors to AT-SPI role names. Returns `None` for `*` or+/// unrecognised selectors (caller treats `None` as "no role filter").+fn role_for_selector(sel: &str) -> Option<String> {+ let s = sel.trim();+ // Trim trailing [attr=...] or .class / #id qualifiers — we filter on+ // role only here (the macOS backend has richer JS-side filtering).+ let tag_end = s+ .find(|c: char| c == '[' || c == '.' || c == '#')+ .unwrap_or(s.len());+ let tag = &s[..tag_end];+ let role = match tag.to_ascii_lowercase().as_str() {+ "a" => "link",+ "button" => "push button",+ "input" | "textarea" => "entry",+ "h1" | "h2" | "h3" | "h4" | "h5" | "h6" => "heading",+ "img" => "image",+ "li" => "list item",+ "p" => "paragraph",+ "select" => "combo box",+ "" | "*" => return None,+ _ => return None,+ };+ Some(role.to_owned())+}++/// Strip Markdown tree formatting and concatenate the human-readable text.+fn extract_text_from_markdown(md: &str) -> String {+ md.lines()+ .map(|l| {+ // Strip leading indent + bullet markers `-` / `[N]` so the result+ // reads like flowing text.+ let trimmed = l.trim_start();+ let trimmed = trimmed.trim_start_matches('-').trim_start();+ // Drop the `[N] ` element-index prefix.+ if let Some(rest) = trimmed.strip_prefix('[') {+ if let Some(end) = rest.find(']') {+ return rest[end + 1..].trim_start().to_owned();+ }+ }+ trimmed.to_owned()+ })+ .filter(|l| !l.is_empty())+ .collect::<Vec<_>>()+ .join("\n")+}diff --git a/libs/cua-driver-rs/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver-rs/crates/platform-macos/src/tools/mod.rsindex 1a3aa45971..35bfae9e2e 100644--- a/libs/cua-driver-rs/crates/platform-macos/src/tools/mod.rs+++ b/libs/cua-driver-rs/crates/platform-macos/src/tools/mod.rs@@ -230,7 +230,11 @@ pub fn register_all(registry: &mut ToolRegistry) {registry.register(Box::new(get_accessibility_tree::GetAccessibilityTreeTool::new(state.clone())));registry.register(Box::new(zoom::ZoomTool { state: state.clone() }));registry.register(Box::new(type_text_chars::TypeTextCharsTool::new(state.clone())));- registry.register(Box::new(page::PageTool::new(state.clone())));+ // Cross-platform `page` tool definition lives in mcp-server; macOS plugs in+ // its Apple-Events / CDP / AX-tree backend here.+ registry.register(Box::new(mcp_server::page::PageTool::new(+ Arc::new(page::MacOsPageBackend::new(state.clone())),+ )));// Recording / replay tools are platform-independent — live in mcp-server.registry.register_recording_tools();}diff --git a/libs/cua-driver-rs/crates/platform-macos/src/tools/page.rs b/libs/cua-driver-rs/crates/platform-macos/src/tools/page.rsindex 15417f4c8f..f290b9e050 100644--- a/libs/cua-driver-rs/crates/platform-macos/src/tools/page.rs+++ b/libs/cua-driver-rs/crates/platform-macos/src/tools/page.rs@@ -1,247 +1,159 @@-//! PageTool: MCP tool for browser JS execution, text extraction, and DOM querying.+//! macOS implementation of the cross-platform `PageBackend` trait.+//!+//! Routes by `bundle_id` / Electron-detection / WKWebView-detection to:+//! - **Apple Events** (Chromium-family + Safari) — zero-config, uses+//! `osascript do JavaScript`.+//! - **CDP** (Electron) — port discovered from the running process.+//! - **AX-tree fallback** (WKWebView / Tauri) — read-only, no JS.+//!+//! The MCP tool definition (name, schema, action dispatch) lives in+//! `mcp_server::page` — this file only implements the backend.use async_trait::async_trait;-use mcp_server::{protocol::ToolResult, tool::{Tool, ToolDef}};-use serde_json::Value;+use mcp_server::page::PageBackend;use std::sync::Arc;use super::ToolState;-use crate::browser::{BrowserJs, ElectronJs, is_wk_web_view_app, AXPageReader};+use crate::browser::{is_wk_web_view_app, AXPageReader, BrowserJs, ElectronJs};-pub struct PageTool {+pub struct MacOsPageBackend {+ #[allow(dead_code)] // reserved for future per-pid caches / configpub state: Arc<ToolState>,}-impl PageTool {- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }-}--static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();+impl MacOsPageBackend {+ pub fn new(state: Arc<ToolState>) -> Self {+ Self { state }+ }-fn def() -> &'static ToolDef {- DEF.get_or_init(|| ToolDef {- name: "page".into(),- description: "Interact with the browser page loaded in a running app. Supports \- Chrome, Brave, Edge, Safari (via AppleScript), Electron apps (via CDP), \- and WKWebView/Tauri apps (via AX tree fallback).\n\n\- Actions:\n\- - execute_javascript: Run JS and return the result.\n\- - get_text: Extract visible text from the page.\n\- - query_dom: Find elements matching a CSS selector.\n\- - enable_javascript_apple_events: Patch Preferences to allow JS from Apple Events \- (Chrome/Brave/Edge only, requires user confirmation and browser restart).".into(),- input_schema: serde_json::json!({- "type": "object",- "required": ["pid", "window_id", "action"],- "properties": {- "pid": { "type": "integer", "description": "Target process ID." },- "window_id": { "type": "integer", "description": "Target window ID from list_windows." },- "action": {- "type": "string",- "enum": ["execute_javascript", "get_text", "query_dom", "enable_javascript_apple_events"],- "description": "Action to perform."- },- "javascript": {- "type": "string",- "description": "JavaScript to execute. Required for execute_javascript."- },- "css_selector": {- "type": "string",- "description": "CSS selector for query_dom (e.g. 'a', 'button', 'input', 'h1'-'h6', 'p', 'img', 'select', '*')."- },- "attributes": {- "type": "array",- "items": { "type": "string" },- "description": "Element attributes to include in query_dom results."- },- "bundle_id": {- "type": "string",- "description": "Bundle ID of the browser. Required for enable_javascript_apple_events."- },- "user_has_confirmed_enabling": {- "type": "boolean",- "description": "Must be true to proceed with enable_javascript_apple_events. \- This will quit and relaunch the browser."- }- },- "additionalProperties": false- }),- read_only: false,- destructive: false,- idempotent: false,- open_world: false,- })+ /// Resolve `bundle_id` for `pid` via the running-apps list.+ async fn bundle_id_for(pid: i32) -> String {+ tokio::task::spawn_blocking(move || {+ crate::apps::list_running_apps()+ .into_iter()+ .find(|a| a.pid == pid)+ .and_then(|a| a.bundle_id)+ .unwrap_or_default()+ })+ .await+ .unwrap_or_default()+ }}#[async_trait]-impl Tool for PageTool {- fn def(&self) -> &ToolDef { def() }+impl PageBackend for MacOsPageBackend {+ async fn get_text(&self, pid: i32, window_id: u32) -> anyhow::Result<String> {+ let bundle_id = Self::bundle_id_for(pid).await;- async fn invoke(&self, args: Value) -> ToolResult {- let pid = match args.get("pid").and_then(|v| v.as_i64()) {- Some(v) => v as i32,- None => return ToolResult::error("Missing required parameter: pid"),- };- let window_id = match args.get("window_id").and_then(|v| v.as_u64()) {- Some(v) => v as u32,- None => return ToolResult::error("Missing required parameter: window_id"),- };- let action = match args.get("action").and_then(|v| v.as_str()) {- Some(v) => v.to_owned(),- None => return ToolResult::error("Missing required parameter: action"),- };-- // Resolve bundle_id from pid.- let bundle_id = {- let p = pid;- tokio::task::spawn_blocking(move || {- crate::apps::list_running_apps()- .into_iter()- .find(|a| a.pid == p)- .and_then(|a| a.bundle_id)- .unwrap_or_default()- }).await.unwrap_or_default()- };+ let use_ax_fallback = !BrowserJs::supports(&bundle_id)+ && tokio::task::spawn_blocking(move || is_wk_web_view_app(pid))+ .await+ .unwrap_or(false);- match action.as_str() {- "enable_javascript_apple_events" => {- let confirmed = args.get("user_has_confirmed_enabling")- .and_then(|v| v.as_bool())- .unwrap_or(false);- if !confirmed {- return ToolResult::error(- "Set user_has_confirmed_enabling=true to proceed. \- This will quit and relaunch the browser."- );- }- let bid = args.get("bundle_id")- .and_then(|v| v.as_str())- .unwrap_or(&bundle_id)- .to_owned();- if bid.is_empty() {- return ToolResult::error("bundle_id is required for enable_javascript_apple_events");- }- match BrowserJs::enable_javascript_apple_events(&bid).await {- Ok(()) => ToolResult::text("JavaScript from Apple Events has been enabled. The browser is restarting."),- Err(e) => ToolResult::error(format!("Failed to enable JavaScript Apple Events: {e}")),- }- }+ if use_ax_fallback {+ return ax_text_fallback(pid, window_id).await;+ }- "execute_javascript" => {- let js = match args.get("javascript").and_then(|v| v.as_str()) {- Some(v) => v.to_owned(),- None => return ToolResult::error("Missing required parameter: javascript"),- };- match execute_js(&js, &bundle_id, pid, window_id).await {- Ok(result) => ToolResult::text(result),- Err(e) => ToolResult::error(format!("{e}")),- }- }+ // Try JS path first; on error, fall back to AX walk.+ match execute_js("document.body.innerText", &bundle_id, pid, window_id).await {+ Ok(result) => Ok(result),+ Err(_) => ax_text_fallback(pid, window_id).await,+ }+ }- "get_text" => {- let use_ax_fallback = !BrowserJs::supports(&bundle_id) && {- let p = pid;- tokio::task::spawn_blocking(move || is_wk_web_view_app(p)).await.unwrap_or(false)- };+ async fn query_dom(+ &self,+ pid: i32,+ window_id: u32,+ css_selector: &str,+ attributes: &[String],+ ) -> anyhow::Result<String> {+ let bundle_id = Self::bundle_id_for(pid).await;++ let use_ax_fallback = !BrowserJs::supports(&bundle_id)+ && tokio::task::spawn_blocking(move || is_wk_web_view_app(pid))+ .await+ .unwrap_or(false);++ if use_ax_fallback {+ let results = ax_query_fallback(pid, window_id, css_selector).await?;+ return Ok(format_ax_elements(&results));+ }- if use_ax_fallback {- match ax_text_fallback(pid, window_id).await {- Ok(text) => ToolResult::text(text),- Err(e) => ToolResult::error(format!("AX text extraction failed: {e}")),- }- } else {- match execute_js("document.body.innerText", &bundle_id, pid, window_id).await {- Ok(result) => ToolResult::text(result),- Err(_) => {- // Try AX fallback on error.- match ax_text_fallback(pid, window_id).await {- Ok(text) => ToolResult::text(text),- Err(e) => ToolResult::error(format!("Page text extraction failed: {e}")),- }- }- }- }+ let js = build_query_selector_js(css_selector, attributes);+ match execute_js(&js, &bundle_id, pid, window_id).await {+ Ok(result) => Ok(result),+ Err(_) => {+ let results = ax_query_fallback(pid, window_id, css_selector).await?;+ Ok(format_ax_elements(&results))}+ }+ }- "query_dom" => {- let selector = args.get("css_selector")- .and_then(|v| v.as_str())- .unwrap_or("*")- .to_owned();- let attributes: Vec<String> = args.get("attributes")- .and_then(|v| v.as_array())- .map(|arr| arr.iter().filter_map(|v| v.as_str().map(str::to_owned)).collect())- .unwrap_or_default();-- let use_ax_fallback = !BrowserJs::supports(&bundle_id) && {- let p = pid;- tokio::task::spawn_blocking(move || is_wk_web_view_app(p)).await.unwrap_or(false)- };-- if use_ax_fallback {- match ax_query_fallback(pid, window_id, &selector).await {- Ok(results) => ToolResult::text(format_ax_elements(&results)),- Err(e) => ToolResult::error(format!("AX DOM query failed: {e}")),- }- } else {- let js = build_query_selector_js(&selector, &attributes);- match execute_js(&js, &bundle_id, pid, window_id).await {- Ok(result) => ToolResult::text(result),- Err(_) => {- // Try AX fallback.- match ax_query_fallback(pid, window_id, &selector).await {- Ok(results) => ToolResult::text(format_ax_elements(&results)),- Err(e) => ToolResult::error(format!("DOM query failed: {e}")),- }- }- }- }- }+ async fn execute_javascript(+ &self,+ pid: i32,+ window_id: u32,+ javascript: &str,+ ) -> anyhow::Result<String> {+ let bundle_id = Self::bundle_id_for(pid).await;+ execute_js(javascript, &bundle_id, pid, window_id).await+ }- other => ToolResult::error(format!("Unknown action: {other}")),- }+ async fn enable_javascript_apple_events(&self, bundle_id: &str) -> anyhow::Result<String> {+ BrowserJs::enable_javascript_apple_events(bundle_id).await?;+ Ok(+ "JavaScript from Apple Events has been enabled. The browser is restarting."+ .to_owned(),+ )}}/// Route JavaScript execution to the appropriate backend.-async fn execute_js(js: &str, bundle_id: &str, pid: i32, window_id: u32) -> anyhow::Result<String> {+async fn execute_js(+ js: &str,+ bundle_id: &str,+ pid: i32,+ window_id: u32,+) -> anyhow::Result<String> {if BrowserJs::supports(bundle_id) {- BrowserJs::execute(js, bundle_id, window_id).await- } else {- let p = pid;- let is_electron = tokio::task::spawn_blocking(move || ElectronJs::is_electron(p)).await?;- if is_electron {- ElectronJs::execute(js, pid).await- } else {- let p = pid;- let is_wk = tokio::task::spawn_blocking(move || is_wk_web_view_app(p)).await?;- if is_wk {- anyhow::bail!(- "execute_javascript is not available for WKWebView/Tauri apps. \- Use get_text or query_dom instead."- );- } else {- anyhow::bail!("Unsupported browser: bundle_id={bundle_id}");- }- }+ return BrowserJs::execute(js, bundle_id, window_id).await;+ }+ let is_electron = tokio::task::spawn_blocking(move || ElectronJs::is_electron(pid)).await?;+ if is_electron {+ return ElectronJs::execute(js, pid).await;}+ let is_wk = tokio::task::spawn_blocking(move || is_wk_web_view_app(pid)).await?;+ if is_wk {+ anyhow::bail!(+ "execute_javascript is not available for WKWebView/Tauri apps. \+ Use get_text or query_dom instead."+ );+ }+ anyhow::bail!("Unsupported browser: bundle_id={bundle_id}");}/// Extract page text via the AX tree.async fn ax_text_fallback(pid: i32, window_id: u32) -> anyhow::Result<String> {let result = tokio::task::spawn_blocking(move || {crate::ax::tree::walk_tree(pid, Some(window_id), None)- }).await+ })+ .await.map_err(|e| anyhow::anyhow!("AX walk task failed: {e}"))?;Ok(AXPageReader::extract_text(&result.tree_markdown))}/// Query AX tree by CSS selector.-async fn ax_query_fallback(pid: i32, window_id: u32, selector: &str) -> anyhow::Result<Vec<crate::browser::ax_page_reader::AXElement>> {+async fn ax_query_fallback(+ pid: i32,+ window_id: u32,+ selector: &str,+) -> anyhow::Result<Vec<crate::browser::ax_page_reader::AXElement>> {let sel = selector.to_owned();let result = tokio::task::spawn_blocking(move || {crate::ax::tree::walk_tree(pid, Some(window_id), None)- }).await+ })+ .await.map_err(|e| anyhow::anyhow!("AX walk task failed: {e}"))?;Ok(AXPageReader::query(&sel, &result.tree_markdown))}@@ -253,9 +165,21 @@ fn format_ax_elements(elements: &[crate::browser::ax_page_reader::AXElement]) ->let mut lines = Vec::new();for el in elements {let index_str = el.index.map(|i| format!("[{i}] ")).unwrap_or_default();- let title = if !el.title.is_empty() { format!(" \"{}\"", el.title) } else { String::new() };- let value = if !el.value.is_empty() { format!(" = \"{}\"", el.value) } else { String::new() };- let desc = if !el.description.is_empty() { format!(" ({})", el.description) } else { String::new() };+ let title = if !el.title.is_empty() {+ format!(" \"{}\"", el.title)+ } else {+ String::new()+ };+ let value = if !el.value.is_empty() {+ format!(" = \"{}\"", el.value)+ } else {+ String::new()+ };+ let desc = if !el.description.is_empty() {+ format!(" ({})", el.description)+ } else {+ String::new()+ };lines.push(format!("- {index_str}{}{title}{value}{desc}", el.role));}lines.join("\n")@@ -265,7 +189,8 @@ fn format_ax_elements(elements: &[crate::browser::ax_page_reader::AXElement]) ->fn build_query_selector_js(selector: &str, attributes: &[String]) -> String {let escaped_sel = json_string(selector);let attrs_js = if attributes.is_empty() {- "['tagName','textContent','href','id','class','type','value','placeholder','name']".to_owned()+ "['tagName','textContent','href','id','class','type','value','placeholder','name']"+ .to_owned()} else {let parts: Vec<String> = attributes.iter().map(|a| json_string(a)).collect();format!("[{}]", parts.join(","))diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rsindex 36c2e69de3..dae21c5d77 100644--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs@@ -4010,6 +4010,11 @@ pub fn build_registry() -> ToolRegistry {// means it doesn't show up in `tools/list` either, matching Swift's// ToolRegistry.swift (`type_text_chars` not in `handlers`).let _: &TypeTextCharsTool = &TypeTextCharsTool { _state: state.clone() }; // touch struct so it stays in this crate for now+ // Cross-platform `page` tool definition lives in mcp-server; Windows plugs+ // in its UIA TextPattern + FindAll backend (CDP for execute_javascript).+ r.register(Box::new(mcp_server::page::PageTool::new(+ std::sync::Arc::new(super::page::WindowsPageBackend::new()),+ )));r.register_recording_tools();r}diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/mod.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/mod.rsindex c808d1efed..cc0acae3da 100644--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/mod.rs+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/mod.rs@@ -7,6 +7,8 @@ use mcp_server::tool::ToolRegistry;#[cfg(target_os = "windows")]mod impl_;+#[cfg(target_os = "windows")]+pub(crate) mod page;#[cfg(not(target_os = "windows"))]mod stubs;diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/page.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/page.rsnew file mode 100644index 0000000000..3cda6c17fe--- /dev/null+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/page.rs@@ -0,0 +1,483 @@+//! Windows implementation of the cross-platform `PageBackend` trait.+//!+//! - `get_text` — find the Document UIA element under the active tab's web-+//! content pane and read `TextPattern.DocumentRange.GetText(-1)`. Falls+//! back to the window-wide tree text if no Document control is present+//! (e.g. Firefox in some configurations exposes the content under a+//! different control type).+//! - `query_dom` — `FindAll(TreeScope_Subtree, condition)` rooted at the web+//! Document. Common CSS selectors map to UIA `ControlType` conditions+//! (`a[href]` → `Hyperlink`, `button` → `Button`, `input[type=*]` →+//! `Edit`, `h1`-`h6` → `Heading`, `li` → `ListItem`, etc.). `#id` /+//! `.class` / `[role=*]` filter the matches after the fact by reading the+//! element's `AutomationId` / IA2 attributes. `[data-*]` is not+//! reachable via UIA/IA2 — that path errors with a pointer at+//! `execute_javascript`.+//! - `execute_javascript` — uses the shared `mcp_server::cdp` helper. When+//! the browser was not launched with `--remote-debugging-port=N`, returns+//! an actionable error.++use async_trait::async_trait;+use mcp_server::page::PageBackend;++use windows::core::Interface;+use windows::Win32::Foundation::HWND;+use windows::Win32::System::Com::{+ CoCreateInstance, CoInitializeEx, CLSCTX_INPROC_SERVER, COINIT_MULTITHREADED,+};+use windows::Win32::UI::Accessibility::{+ CUIAutomation, IUIAutomation, IUIAutomationElement, IUIAutomationTextPattern,+ TreeScope_Subtree, UIA_ButtonControlTypeId, UIA_CONTROLTYPE_ID, UIA_ControlTypePropertyId,+ UIA_DocumentControlTypeId, UIA_EditControlTypeId, UIA_HeaderControlTypeId,+ UIA_HyperlinkControlTypeId, UIA_ImageControlTypeId, UIA_ListItemControlTypeId,+ UIA_NamePropertyId, UIA_PaneControlTypeId, UIA_TextControlTypeId, UIA_TextPatternId,+ UIA_ValueValuePropertyId,+};++pub struct WindowsPageBackend;++impl WindowsPageBackend {+ pub fn new() -> Self {+ Self+ }+}++impl Default for WindowsPageBackend {+ fn default() -> Self {+ Self::new()+ }+}++#[async_trait]+impl PageBackend for WindowsPageBackend {+ async fn get_text(&self, _pid: i32, window_id: u32) -> anyhow::Result<String> {+ let hwnd = window_id as u64;+ tokio::task::spawn_blocking(move || unsafe { get_text_blocking(hwnd) })+ .await+ .map_err(|e| anyhow::anyhow!("join error: {e}"))?+ }++ async fn query_dom(+ &self,+ _pid: i32,+ window_id: u32,+ css_selector: &str,+ attributes: &[String],+ ) -> anyhow::Result<String> {+ let hwnd = window_id as u64;+ let selector = css_selector.to_owned();+ let attrs: Vec<String> = attributes.to_vec();+ tokio::task::spawn_blocking(move || unsafe { query_dom_blocking(hwnd, &selector, &attrs) })+ .await+ .map_err(|e| anyhow::anyhow!("join error: {e}"))?+ }++ async fn execute_javascript(+ &self,+ _pid: i32,+ _window_id: u32,+ javascript: &str,+ ) -> anyhow::Result<String> {+ // Discover CDP port from env. The "right" approach (extracting the+ // --remote-debugging-port from the process's command line) requires+ // reading PEB; for now we accept the conventional env var and fail+ // with an actionable message if it isn't set.+ let port: u16 = match std::env::var("CUA_DRIVER_CDP_PORT")+ .ok()+ .and_then(|s| s.parse::<u16>().ok())+ {+ Some(p) => p,+ None => anyhow::bail!(+ "Chromium's `execute_javascript` on Windows requires the browser launched \+ with `--remote-debugging-port=N`. cua-driver doesn't yet ship a companion \+ extension; relaunch the browser with the flag and export \+ CUA_DRIVER_CDP_PORT=N before starting cua-driver, or use `get_text` / \+ `query_dom` for read-only access."+ ),+ };+ let result = mcp_server::cdp::evaluate(port, javascript, true).await?;+ Ok(format!("cdp.runtime.evaluate.user_gesture: {result}"))+ }+}++// ── implementation ────────────────────────────────────────────────────────++unsafe fn get_text_blocking(hwnd: u64) -> anyhow::Result<String> {+ let (automation, root) = uia_root_for(hwnd)?;+ let document = find_document_descendant(&automation, &root)?;+ if let Some(doc) = document {+ text_pattern_dump(&doc)+ } else {+ // No Document control — try TextPattern on the root.+ match text_pattern_dump(&root) {+ Ok(s) => Ok(s),+ Err(_) => Ok(walk_text_blocking(&root)),+ }+ }+}++unsafe fn query_dom_blocking(+ hwnd: u64,+ selector: &str,+ attributes: &[String],+) -> anyhow::Result<String> {+ let (automation, root) = uia_root_for(hwnd)?;+ let root_for_search = find_document_descendant(&automation, &root)?.unwrap_or(root);++ let parsed = parse_selector(selector)?;++ if parsed.is_data_attr {+ anyhow::bail!(+ "`[data-*]` selectors are not reachable via UIA/IA2 on Windows. \+ Use `execute_javascript` for full DOM access (requires the browser launched \+ with `--remote-debugging-port`)."+ );+ }++ // Build a UIA condition for the ControlType (if any) and FindAll.+ let condition = match parsed.control_type {+ Some(ct) => automation.CreatePropertyCondition(+ UIA_ControlTypePropertyId,+ &ct.0.into(),+ )?,+ None => automation.CreateTrueCondition()?,+ };++ let matches = root_for_search.FindAll(TreeScope_Subtree, &condition)?;+ let count = matches.Length().unwrap_or(0);++ let mut formatted: Vec<String> = Vec::new();+ for i in 0..count {+ let elem = match matches.GetElement(i) {+ Ok(e) => e,+ Err(_) => continue,+ };++ // Post-filter by id/class. Skip elements that don't satisfy.+ if !parsed.id_filter.is_empty() {+ let aid = read_string(&elem, |e| {+ e.CurrentAutomationId().map(|b| b.to_string()).map_err(|e| e.into())+ });+ if aid.as_deref().unwrap_or_default() != parsed.id_filter {+ continue;+ }+ }++ let name = read_string(&elem, |e| {+ e.CurrentName().map(|b| b.to_string()).map_err(|e| e.into())+ })+ .unwrap_or_default();++ // attributes the caller asked for. We map a small set of well-known+ // CSS attribute names onto UIA properties.+ let mut attr_parts: Vec<String> = Vec::new();+ for a in attributes {+ let v = match a.as_str() {+ "name" => Some(name.clone()),+ "id" => read_string(&elem, |e| {+ e.CurrentAutomationId().map(|b| b.to_string()).map_err(|e| e.into())+ }),+ "value" => read_value_value(&elem),+ "href" => {+ // No native UIA "href" — closest is the AccessibleValue+ // for hyperlinks (Chromium populates this with the URL).+ read_value_value(&elem)+ }+ "textContent" | "text" => Some(name.clone()),+ _ => None,+ };+ if let Some(v) = v {+ attr_parts.push(format!("{a}=\"{}\"", escape_attr(&v)));+ }+ }++ let ct = read_control_type(&elem).unwrap_or_else(|| "Unknown".to_owned());+ let line = if attr_parts.is_empty() {+ format!("- {ct} \"{}\"", escape_attr(&name))+ } else {+ format!("- {ct} \"{}\" [{}]", escape_attr(&name), attr_parts.join(" "))+ };+ formatted.push(line);+ }++ if formatted.is_empty() {+ Ok("No elements found.".to_owned())+ } else {+ Ok(formatted.join("\n"))+ }+}++// ── UIA helpers ───────────────────────────────────────────────────────────++/// Initialise COM (best-effort) and resolve the UIA root element for `hwnd`.+unsafe fn uia_root_for(hwnd: u64) -> anyhow::Result<(IUIAutomation, IUIAutomationElement)> {+ let _ = CoInitializeEx(None, COINIT_MULTITHREADED);+ let automation: IUIAutomation = CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER)+ .map_err(|e| anyhow::anyhow!("UIA init failed: {e}"))?;+ let hwnd_win = HWND(hwnd as *mut _);+ let root = automation+ .ElementFromHandle(hwnd_win)+ .map_err(|e| anyhow::anyhow!("ElementFromHandle failed: {e}"))?;+ Ok((automation, root))+}++/// Find the first descendant Document control under `root`. The web view's+/// `Document` is the root of the rendered DOM in both Chromium and Firefox.+unsafe fn find_document_descendant(+ automation: &IUIAutomation,+ root: &IUIAutomationElement,+) -> anyhow::Result<Option<IUIAutomationElement>> {+ let doc_id: i32 = UIA_DocumentControlTypeId.0;+ let cond = automation.CreatePropertyCondition(UIA_ControlTypePropertyId, &doc_id.into())?;+ match root.FindFirst(TreeScope_Subtree, &cond) {+ Ok(elem) => Ok(Some(elem)),+ Err(_) => Ok(None),+ }+}++/// Try `TextPattern.DocumentRange.GetText(-1)` on `elem`. Returns an error if+/// the element does not support TextPattern.+unsafe fn text_pattern_dump(elem: &IUIAutomationElement) -> anyhow::Result<String> {+ let pattern_raw = elem+ .GetCurrentPattern(UIA_TextPatternId)+ .map_err(|e| anyhow::anyhow!("element does not support TextPattern: {e}"))?;+ let text_pattern: IUIAutomationTextPattern = pattern_raw+ .cast()+ .map_err(|e| anyhow::anyhow!("cast to IUIAutomationTextPattern failed: {e}"))?;+ let range = text_pattern+ .DocumentRange()+ .map_err(|e| anyhow::anyhow!("DocumentRange failed: {e}"))?;+ let bstr = range+ .GetText(-1)+ .map_err(|e| anyhow::anyhow!("GetText(-1) failed: {e}"))?;+ Ok(bstr.to_string())+}++/// Last-resort: concatenate Name properties of every descendant. Used when+/// no Document control is present and the root has no TextPattern (very+/// rare for Chromium/Firefox).+unsafe fn walk_text_blocking(root: &IUIAutomationElement) -> String {+ let mut acc = String::new();+ if let Ok(name) = root.CurrentName() {+ let s = name.to_string();+ if !s.trim().is_empty() {+ acc.push_str(&s);+ acc.push('\n');+ }+ }+ if let Ok(children) = root.FindAll(+ TreeScope_Subtree,+ // Match every element with a non-empty name.+ &match create_true_condition_or_fallback() {+ Some(c) => c,+ None => return acc,+ },+ ) {+ let count = children.Length().unwrap_or(0);+ for i in 0..count {+ if let Ok(c) = children.GetElement(i) {+ if let Ok(name) = c.CurrentName() {+ let s = name.to_string();+ if !s.trim().is_empty() {+ acc.push_str(&s);+ acc.push('\n');+ }+ }+ }+ }+ }+ acc+}++unsafe fn create_true_condition_or_fallback() -> Option<windows::Win32::UI::Accessibility::IUIAutomationCondition>+{+ let automation: IUIAutomation =+ CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER).ok()?;+ automation.CreateTrueCondition().ok()+}++unsafe fn read_string<F>(elem: &IUIAutomationElement, f: F) -> Option<String>+where+ F: FnOnce(&IUIAutomationElement) -> anyhow::Result<String>,+{+ let s = f(elem).ok()?;+ if s.is_empty() {+ None+ } else {+ Some(s)+ }+}++unsafe fn read_control_type(elem: &IUIAutomationElement) -> Option<String> {+ // Match against the raw i32 value — using the `UIA_*ControlTypeId`+ // constants directly in a match arm triggers `non_upper_case_globals`.+ let ct = elem.CurrentControlType().ok()?;+ let name = if ct.0 == UIA_ButtonControlTypeId.0 {+ "Button"+ } else if ct.0 == UIA_HyperlinkControlTypeId.0 {+ "Hyperlink"+ } else if ct.0 == UIA_EditControlTypeId.0 {+ "Edit"+ } else if ct.0 == UIA_ListItemControlTypeId.0 {+ "ListItem"+ } else if ct.0 == UIA_ImageControlTypeId.0 {+ "Image"+ } else if ct.0 == UIA_TextControlTypeId.0 {+ "Text"+ } else if ct.0 == UIA_DocumentControlTypeId.0 {+ "Document"+ } else if ct.0 == UIA_PaneControlTypeId.0 {+ "Pane"+ } else if ct.0 == UIA_HeaderControlTypeId.0 {+ "Header"+ } else {+ return Some(format!("ControlType({})", ct.0));+ };+ Some(name.to_owned())+}++unsafe fn read_value_value(elem: &IUIAutomationElement) -> Option<String> {+ let v = elem.GetCurrentPropertyValue(UIA_ValueValuePropertyId).ok()?;+ let raw = v.as_raw();+ if raw.Anonymous.Anonymous.vt != 8 {+ // VT_BSTR = 8 — anything else (empty, VT_EMPTY=0) means no value.+ return None;+ }+ // Read the BSTR pointer; we don't free it (windows-rs handles VARIANT+ // lifetime on drop of `v`).+ let bstr_ptr = raw.Anonymous.Anonymous.Anonymous.bstrVal;+ if bstr_ptr.is_null() {+ return None;+ }+ let bstr = windows::core::BSTR::from_raw(bstr_ptr);+ let s = bstr.to_string();+ // Don't double-free: hand the BSTR back to the VARIANT by leaking ours.+ std::mem::forget(bstr);+ Some(s)+}++// Avoid an unused-import warning when none of the secondary properties are+// actually read above.+#[allow(dead_code)]+const _UNUSED_NAME_PROP: UIA_PROPERTY_ID_T = UIA_NamePropertyId_VAL;++#[allow(non_camel_case_types)]+type UIA_PROPERTY_ID_T = windows::Win32::UI::Accessibility::UIA_PROPERTY_ID;+#[allow(non_upper_case_globals)]+const UIA_NamePropertyId_VAL: UIA_PROPERTY_ID_T = UIA_NamePropertyId;++// ── selector parsing ──────────────────────────────────────────────────────++struct ParsedSelector {+ control_type: Option<UIA_CONTROLTYPE_ID>,+ id_filter: String,+ is_data_attr: bool,+}++fn parse_selector(sel: &str) -> anyhow::Result<ParsedSelector> {+ let s = sel.trim();++ // Wildcard / empty → match-all, no post-filter.+ if s.is_empty() || s == "*" {+ return Ok(ParsedSelector {+ control_type: None,+ id_filter: String::new(),+ is_data_attr: false,+ });+ }++ // Reject `[data-*]` immediately — not reachable via UIA.+ if s.contains("[data-") {+ return Ok(ParsedSelector {+ control_type: None,+ id_filter: String::new(),+ is_data_attr: true,+ });+ }++ // Strip [attribute=value] suffix once we've consumed the tag.+ let (tag, rest) = match s.find('[') {+ Some(i) => (&s[..i], &s[i..]),+ None => (s, ""),+ };++ // ID suffix on tag (e.g. `div#foo`). `.class` is NOT supported — UIA's+ // `AutomationId` maps to HTML `id`, not to the class list. Supporting+ // class selectors would need an IA2 `attributes`-string parse (tracked+ // as a follow-up in the PR description).+ if tag.contains('.') {+ anyhow::bail!(+ "`.class` selectors are not supported by the Windows UIA backend. \+ Use a simple tag (a, button, input, textarea, h1-h6, img, li, p, \+ span, select), a `tag#id` selector, an `[attr=value]` selector \+ including `[role=…]`, or `execute_javascript` for full DOM access \+ (requires the browser launched with `--remote-debugging-port`)."+ );+ }+ let (tag_clean, id_filter) = match tag.find('#') {+ Some(i) => (&tag[..i], tag[i + 1..].to_owned()),+ None => (tag, String::new()),+ };++ let ct = control_type_for_tag(tag_clean);++ // For `[role=X]` selectors with no tag, try to map role → ControlType.+ let ct = if ct.is_none() && rest.starts_with("[role=") {+ let role = rest+ .trim_start_matches("[role=")+ .trim_end_matches(']')+ .trim_matches(|c: char| c == '"' || c == '\'');+ control_type_for_role(role)+ } else {+ ct+ };++ // After mapping, if we have neither a control_type nor an id_filter, the+ // selector wasn't understood — refuse so we don't dump the entire subtree+ // as a false-positive match.+ if ct.is_none() && id_filter.is_empty() && tag_clean != "*" && !tag_clean.is_empty() {+ anyhow::bail!(+ "Selector '{sel}' is not supported by the Windows UIA backend. \+ Use a simple tag (a, button, input, textarea, h1-h6, img, li, p, \+ span, select), a `tag#id` selector, `[role=…]`, or \+ `execute_javascript` for full DOM access (requires the browser \+ launched with `--remote-debugging-port`)."+ );+ }++ Ok(ParsedSelector {+ control_type: ct,+ id_filter,+ is_data_attr: false,+ })+}++fn control_type_for_tag(tag: &str) -> Option<UIA_CONTROLTYPE_ID> {+ Some(match tag.to_ascii_lowercase().as_str() {+ "a" => UIA_HyperlinkControlTypeId,+ "button" => UIA_ButtonControlTypeId,+ "input" | "textarea" => UIA_EditControlTypeId,+ "h1" | "h2" | "h3" | "h4" | "h5" | "h6" => UIA_HeaderControlTypeId,+ "img" => UIA_ImageControlTypeId,+ "li" => UIA_ListItemControlTypeId,+ "p" | "span" => UIA_TextControlTypeId,+ "" | "*" => return None,+ _ => return None,+ })+}++fn control_type_for_role(role: &str) -> Option<UIA_CONTROLTYPE_ID> {+ Some(match role.to_ascii_lowercase().as_str() {+ "button" => UIA_ButtonControlTypeId,+ "link" => UIA_HyperlinkControlTypeId,+ "textbox" | "searchbox" => UIA_EditControlTypeId,+ "heading" => UIA_HeaderControlTypeId,+ "img" | "image" => UIA_ImageControlTypeId,+ "listitem" => UIA_ListItemControlTypeId,+ _ => return None,+ })+}++fn escape_attr(s: &str) -> String {+ s.replace('"', "\\\"")+}diff --git a/libs/cua-driver-rs/tests/integration/test_api_parity.py b/libs/cua-driver-rs/tests/integration/test_api_parity.pyindex 3fbe1c2c50..5413319378 100644--- a/libs/cua-driver-rs/tests/integration/test_api_parity.py+++ b/libs/cua-driver-rs/tests/integration/test_api_parity.py@@ -130,14 +130,11 @@ REQUIRED_TOOLS = [# In Rust, not in Swift yet. Tests for these will fail on the Swift binary.RUST_ONLY_TOOLS = ["type_text_chars", # per-character delay typing; Swift uses type_text- "browser_eval", # CDP JS eval; Swift uses 'page' for browser JS"get_accessibility_tree",# lightweight desktop AX snapshot (separate from get_window_state)]-# In Swift, not registered in Rust yet (page.rs exists but not in register_all).-SWIFT_ONLY_TOOLS = [- "page", # browser JS + get_text + query_dom; Rust uses browser_eval-]+# In Swift, not registered in Rust yet. (`page` is now cross-platform in Rust.)+SWIFT_ONLY_TOOLS: list[str] = []# ── parity mixin ───────────────────────────────────────────────────────────────@@ -1070,7 +1067,7 @@ class _ParityMixin:# ── stdio MCP: Rust-only tools (missing in Swift) ────────────────────────def test_mcp_rust_only_tools_present(self) -> None:- """Rust-only tools (type_text_chars, browser_eval, get_accessibility_tree).+ """Rust-only tools (type_text_chars, get_accessibility_tree).These are in the Rust binary but not yet in Swift. Both binariesshould eventually have them. This test fails on Swift until ported.@@ -1104,11 +1101,9 @@ class _ParityMixin:# ── stdio MCP: Swift-only tools (missing/unregistered in Rust) ───────────def test_mcp_swift_only_tools_present(self) -> None:- """Swift-only tools (page).-- In Swift, not yet registered in Rust (page.rs exists but not in- register_all). Both binaries should eventually have them.- This test fails on Rust until page::PageTool is added to register_all.+ """Swift-only tools list — currently empty after the page-tool+ unification in Rust. Kept as a no-op so the matrix stays symmetric+ with `test_mcp_rust_only_tools_present`."""with self._mcp() as c:names = {t["name"] for t in c.list_tools()}@@ -1118,25 +1113,6 @@ class _ParityMixin:f"Swift-only tools missing: {missing} (not yet registered in this binary)",)- # ── stdio MCP: browser_eval ───────────────────────────────────────────────-- def test_mcp_browser_eval_is_registered(self) -> None:- """'browser_eval' tool must appear in tools/list.-- Parity gap: Swift binary does not have 'browser_eval' — it uses 'page'- for browser JS. This test documents the requirement that both binaries- expose browser_eval. Fails on Swift until it is ported.- """- with self._mcp() as c:- names = {t["name"] for t in c.list_tools()}- self.assertIn("browser_eval", names, "'browser_eval' tool is not registered")-- def test_mcp_browser_eval_missing_pid_raises_error(self) -> None:- """browser_eval with no pid should return an MCP error."""- self._assert_tool_raises_mcp_error(- "browser_eval", {"expression": "1+1"}- )-# ── stdio MCP: protocol-level contracts ──────────────────────────────────def test_mcp_tools_call_unknown_tool_returns_error(self) -> None:diff --git a/libs/cua-driver-rs/tests/integration/v2/PHILOSOPHY.md b/libs/cua-driver-rs/tests/integration/v2/PHILOSOPHY.mdindex 5f9bc53b1f..ec1b83144e 100644--- a/libs/cua-driver-rs/tests/integration/v2/PHILOSOPHY.md+++ b/libs/cua-driver-rs/tests/integration/v2/PHILOSOPHY.md@@ -17,10 +17,10 @@ The driver exposes a small, stable interface:| `scroll` | Scroll wheel || `drag` | Mouse drag |-Tests **never** reach for `browser_eval`, CDP, Apple Events, or any other-internal mechanism. The driver is a black box that delivers the action via-whatever technique works best for the target app. Tests verify the-*observable outcome*, not the delivery path.+Tests **never** reach for `page.execute_javascript`, CDP, Apple Events, or+any other internal mechanism. The driver is a black box that delivers the+action via whatever technique works best for the target app. Tests verify+the *observable outcome*, not the delivery path.---