Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver-rs): unify browser_eval + page into one cross-platform tool (#1666)

Francesco Bonacci94f23d07ef25
diff --git a/libs/cua-driver-fixtures/README.md b/libs/cua-driver-fixtures/README.md
index 7c76e907ca..30c9fa5eb2 100644
--- a/libs/cua-driver-fixtures/README.md
+++ b/libs/cua-driver-fixtures/README.md
@@ -74,9 +74,9 @@ test (May 2026) showed need explicit harness coverage:
Like `test_page.html`, each panel exposes a global accessor:
`window.getGesturePanelState()` returns a JSON snapshot of all four
-panels' state. Tests can poll it via `browser_eval` (when running with
-Chromium's `--remote-debugging-port`) or read the visible status divs
-via UIA / OCR.
+panels' state. Tests can poll it via `page` with
+`action: "execute_javascript"` (when running with Chromium's
+`--remote-debugging-port`) or read the visible status divs via UIA / OCR.
### Known browser-coverage gaps (May 2026)
@@ -86,7 +86,7 @@ Windows, the integration tests should be aware of:
- **GPU-composited content blanks PrintWindow.** `cua-driver screenshot`
captures the browser chrome + a white body. Workaround: launch the
target browser with `--remote-debugging-port=<port>` and use
- `browser_eval` for both inputs and verification.
+ `page` (`action: "execute_javascript"`) for both inputs and verification.
- **Chromium auto-disables full a11y** unless an AT-grade client is
detected. UIA on the rendered DOM returns empty until the page is
flagged accessibility-active. Workaround: enable Chromium's
diff --git a/libs/cua-driver-rs/PARITY.md b/libs/cua-driver-rs/PARITY.md
index 806a7bb80e..9a97b987e1 100644
--- a/libs/cua-driver-rs/PARITY.md
+++ b/libs/cua-driver-rs/PARITY.md
@@ -20,10 +20,9 @@ cd libs/cua-driver-rs/tests/integration
| Gap | Rust | Swift |
|-----|------|-------|
| `type_text_chars` | ✅ | missing |
-| `browser_eval` | ✅ | missing |
| `get_accessibility_tree` | ✅ | missing |
-| `page` tool | ✅ (registered as of this commit) | ✅ |
-| `--version` flag | ✅ (fixed in this commit) | ✅ |
+| `page` tool | ✅ (cross-platform: Apple-Events on macOS, UIA+CDP on Windows, AT-SPI+CDP on Linux) | ✅ (macOS only) |
+| `--version` flag | ✅ | ✅ |
| `call check_permissions` JSON | ✅ JSON | human-readable text |
| `call screenshot` (no window_id) | ✅ full-display default | error (requires window_id) |
@@ -1370,7 +1369,6 @@ older clients that only read name/description still work.
"id": 1,
"result": {
"tools": [
- { "name": "browser_eval", "description": "…", "inputSchema": {…}, "annotations": {…} },
{ "name": "check_permissions", "description": "…", "inputSchema": {…}, "annotations": {…} },
{ "name": "click", "description": "…", "inputSchema": {…}, "annotations": {…} },
{ "name": "double_click", "…": "…" },
diff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md b/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md
index cf2734ce64..7be50ebf03 100644
--- a/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md
+++ b/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md
@@ -387,6 +387,10 @@ they can cmd-tab back.
## Browser JS primitives — Apple Events backend
+The `page` tool is **cross-platform** — Windows uses UIA + CDP,
+Linux uses AT-SPI + CDP, and macOS (this section) uses the
+Apple-Events / CDP / AX-tree routing described below.
+
When the AX tree doesn't expose the data you need (common in
Chromium/Electron — the tree is sparse for web content), use the
`page` tool or the `javascript` param on `get_window_state` to query
diff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/SKILL.md b/libs/cua-driver-rs/Skills/cua-driver-rs/SKILL.md
index 96ea1d6ff6..3144129b8e 100644
--- a/libs/cua-driver-rs/Skills/cua-driver-rs/SKILL.md
+++ b/libs/cua-driver-rs/Skills/cua-driver-rs/SKILL.md
@@ -418,8 +418,12 @@ silently no-ops — symptom is a system bell; use `set_value` or click
a clickable equivalent), scrolling via synthetic PageUp/Down
keystrokes, in-page clicks, and typing into web inputs.
-Browser JS primitives (the `page` tool, `get_window_state(javascript=)`)
-are macOS-only today — see `MACOS.md` for the Apple-Events backend.
+Browser JS primitives are now **cross-platform** via the `page` tool —
+macOS uses Apple Events for Chrome/Brave/Edge/Safari + CDP for Electron
+(see `MACOS.md`); Windows + Linux use UIA / AT-SPI for `get_text` /
+`query_dom` and the shared CDP client for `execute_javascript` (browser
+must be launched with `--remote-debugging-port=N` and the port exported
+as `CUA_DRIVER_CDP_PORT`).
## Re-snapshot and verify — mandatory
diff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/WEB_APPS.md b/libs/cua-driver-rs/Skills/cua-driver-rs/WEB_APPS.md
index 4a1004eb1a..d0e6588f99 100644
--- a/libs/cua-driver-rs/Skills/cua-driver-rs/WEB_APPS.md
+++ b/libs/cua-driver-rs/Skills/cua-driver-rs/WEB_APPS.md
@@ -5,6 +5,13 @@
> on macOS. For Windows web-app automation (Edge, Chrome with
> PostMessage / WebView2), see `WINDOWS.md` → "Web apps on Windows". For
> Linux: `LINUX.md` (BETA — limited browser automation today).
+>
+> The `page` tool itself is **cross-platform** — Windows + Linux back
+> `get_text` / `query_dom` with UIA / AT-SPI respectively, and
+> `execute_javascript` on those platforms uses CDP (browser must be
+> launched with `--remote-debugging-port=N`, with the port exposed via
+> the `CUA_DRIVER_CDP_PORT` env var). macOS routing (this doc) remains
+> Apple Events → CDP → AX-tree fallback.
Covers apps whose UI is rendered in a web runtime inside a native
macOS shell:
diff --git a/libs/cua-driver-rs/crates/cua-driver/tests/mcp_protocol_test.rs b/libs/cua-driver-rs/crates/cua-driver/tests/mcp_protocol_test.rs
index e95ee93ff0..ea368c3514 100644
--- a/libs/cua-driver-rs/crates/cua-driver/tests/mcp_protocol_test.rs
+++ b/libs/cua-driver-rs/crates/cua-driver/tests/mcp_protocol_test.rs
@@ -325,7 +325,7 @@ fn test_all_expected_tools_registered() {
"get_agent_cursor_state", "check_permissions", "get_config", "set_config",
"get_accessibility_tree",
"set_recording", "get_recording_state", "replay_trajectory",
- "browser_eval",
+ "page",
];
for name in &expected {
assert!(names.contains(name), "Missing tool: {name} (registered: {names:?})");
@@ -940,9 +940,9 @@ fn test_replay_trajectory() {
#[test]
#[cfg(target_os = "macos")]
-fn test_browser_eval_no_cdp_error() {
- //! Verify browser_eval is registered and returns a meaningful error when
- //! no Chromium process is listening on the given cdp_port.
+fn test_page_unknown_action_error() {
+ //! Verify the cross-platform `page` tool is registered and rejects an
+ //! unknown action with a meaningful error.
let binary = binary_path();
if !binary.exists() { return; }
@@ -955,25 +955,24 @@ fn test_browser_eval_no_cdp_error() {
send_request(stdin, &serde_json::json!({"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}));
read_response(&mut stdout);
- // Use a port that is almost certainly not listening.
send_request(stdin, &serde_json::json!({
"jsonrpc":"2.0","id":2,"method":"tools/call",
- "params":{"name":"browser_eval","arguments":{
- "expression": "1+1",
- "cdp_port": 19722
+ "params":{"name":"page","arguments":{
+ "pid": 1,
+ "window_id": 0,
+ "action": "definitely_not_a_real_action"
}}
}));
let resp = read_response(&mut stdout);
- // The tool should return isError=true with a connection-refused message.
assert!(
resp["result"]["isError"].as_bool().unwrap_or(false),
- "expected isError=true when CDP port is not listening: {resp:?}"
+ "expected isError=true for unknown action: {resp:?}"
);
let text = resp["result"]["content"][0]["text"].as_str().unwrap_or("");
assert!(
- text.contains("browser_eval"),
- "error text should mention browser_eval: {text}"
+ text.to_ascii_lowercase().contains("unknown action"),
+ "error text should mention 'Unknown action': {text}"
);
child.kill().ok();
@@ -2128,7 +2127,7 @@ fn test_all_expected_tools_registered_windows() {
"get_agent_cursor_state", "check_permissions", "get_config", "set_config",
"get_accessibility_tree",
"set_recording", "get_recording_state", "replay_trajectory",
- "browser_eval",
+ "page",
];
for name in &expected {
assert!(names.contains(name), "Missing tool: {name} (registered: {names:?})");
@@ -3009,7 +3008,9 @@ fn test_replay_trajectory_windows() {
#[test]
#[cfg(target_os = "windows")]
-fn test_browser_eval_no_cdp_error_windows() {
+fn test_page_unknown_action_error_windows() {
+ //! Verify the cross-platform `page` tool is registered on Windows and
+ //! rejects an unknown action with a meaningful error.
let binary = binary_path_windows();
if !binary.exists() { return; }
@@ -3024,15 +3025,20 @@ fn test_browser_eval_no_cdp_error_windows() {
send_request(stdin, &serde_json::json!({
"jsonrpc":"2.0","id":2,"method":"tools/call",
- "params":{"name":"browser_eval","arguments":{"expression":"1+1","cdp_port":19722}}
+ "params":{"name":"page","arguments":{
+ "pid":1,"window_id":0,"action":"definitely_not_a_real_action"
+ }}
}));
let resp = read_response(&mut stdout);
assert!(
resp["result"]["isError"].as_bool().unwrap_or(false),
- "expected isError=true when CDP port not listening: {resp:?}"
+ "expected isError=true for unknown action: {resp:?}"
);
let text = resp["result"]["content"][0]["text"].as_str().unwrap_or("");
- assert!(text.contains("browser_eval"), "error text should mention browser_eval: {text}");
+ assert!(
+ text.to_ascii_lowercase().contains("unknown action"),
+ "error text should mention 'Unknown action': {text}"
+ );
child.kill().ok();
}
diff --git a/libs/cua-driver-rs/crates/mcp-server/src/browser_eval.rs b/libs/cua-driver-rs/crates/mcp-server/src/cdp.rs
similarity index 59%
rename from libs/cua-driver-rs/crates/mcp-server/src/browser_eval.rs
rename to libs/cua-driver-rs/crates/mcp-server/src/cdp.rs
index 7cea2b52d1..579861e544 100644
--- a/libs/cua-driver-rs/crates/mcp-server/src/browser_eval.rs
+++ b/libs/cua-driver-rs/crates/mcp-server/src/cdp.rs
@@ -1,104 +1,55 @@
-//! `browser_eval` tool — Chromium CDP Runtime.evaluate over a local WebSocket.
+//! Cross-platform CDP (Chrome DevTools Protocol) helper.
//!
-//! Cross-platform: all I/O is plain TCP to localhost (no external HTTP/WS crates).
-//! Mirrors the Windows C# reference in CuaDriver.Win/Tools/BrowserEvalTool.cs
-//! and CuaDriver.Win/Browser/CdpBrowserBridge.cs.
+//! A self-contained `Runtime.evaluate` client that speaks raw TCP — no
+//! external HTTP/WS crates. Used by the cross-platform `page` tool's
+//! `execute_javascript` action on Windows/Linux (and by the macOS Electron
+//! backend, which retains its own copy under `platform-macos/src/browser/`
+//! that predates this helper).
//!
//! Flow:
-//! 1. HTTP GET http://127.0.0.1:{cdp_port}/json → discover page WebSocket URLs.
-//! 2. WebSocket connect to the first (or best-matched) page URL.
+//! 1. HTTP GET http://127.0.0.1:{cdp_port}/json → discover page WS URLs.
+//! 2. WebSocket connect to the first page URL.
//! 3. Send Runtime.evaluate with userGesture=true.
//! 4. Read frames until we receive the response with id=1.
-use async_trait::async_trait;
use serde_json::Value;
-use crate::{
- protocol::ToolResult,
- tool::{Tool, ToolDef},
-};
-
-pub struct BrowserEvalTool;
-
-#[async_trait]
-impl Tool for BrowserEvalTool {
- fn def(&self) -> &ToolDef {
- static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
- DEF.get_or_init(|| ToolDef {
- name: "browser_eval".into(),
- description: "Evaluate JavaScript in a Chromium browser via CDP Runtime.evaluate \
- with userGesture=true. Requires cdp_port. Pass window_id to target \
- a specific browser window/tab (matched by title heuristic).".into(),
- input_schema: serde_json::json!({
- "type": "object",
- "required": ["expression"],
- "properties": {
- "expression": {
- "type": "string",
- "description": "JavaScript expression to evaluate."
- },
- "cdp_port": {
- "type": "integer",
- "description": "Chromium remote debugging port (--remote-debugging-port)."
- },
- "window_id": {
- "type": "integer",
- "description": "Optional browser window/HWND/XID used to bind CDP to the matching tab."
- },
- "await_promise": {
- "type": "boolean",
- "description": "Await a returned Promise (default true)."
- }
- }
- }),
- read_only: false,
- destructive: true,
- idempotent: false,
- open_world: true,
- })
- }
-
- async fn invoke(&self, args: Value) -> ToolResult {
- let expression = match args.get("expression").and_then(|v| v.as_str()) {
- Some(e) => e.to_string(),
- None => return ToolResult::error("browser_eval: 'expression' is required"),
- };
- let cdp_port = match args.get("cdp_port").and_then(|v| v.as_u64()) {
- Some(p) if p > 0 && p < 65536 => p as u16,
- _ => return ToolResult::error(
- "browser_eval: 'cdp_port' is required (Chromium --remote-debugging-port value)"
- ),
- };
- let await_promise = args.get("await_promise")
- .and_then(|v| v.as_bool())
- .unwrap_or(true);
- let window_title_hint = args.get("window_id").map(|_| ""); // reserved for future matching
-
- match cdp_evaluate(cdp_port, window_title_hint, &expression, await_promise).await {
- Ok(response) => {
- let result_text = format_cdp_result(&response);
- ToolResult::text(format!("cdp.runtime.evaluate.user_gesture: {result_text}"))
- }
- Err(e) => ToolResult::error(format!("browser_eval: {e}")),
- }
- }
+/// Evaluate `expression` in the first page tab exposed by a Chromium-family
+/// browser listening on `port` (the value passed to `--remote-debugging-port`).
+///
+/// Returns the formatted result (or `exception: ...` if the evaluated code
+/// raised). Caller is responsible for prefixing the human-readable label
+/// (e.g. `"cdp.runtime.evaluate.user_gesture: ..."`).
+pub async fn evaluate(
+ port: u16,
+ expression: &str,
+ await_promise: bool,
+) -> anyhow::Result<String> {
+ let response = cdp_evaluate(port, expression, await_promise).await?;
+ Ok(format_cdp_result(&response))
}
// ── High-level evaluate ───────────────────────────────────────────────────
async fn cdp_evaluate(
port: u16,
- _window_title_hint: Option<&str>,
expression: &str,
await_promise: bool,
) -> anyhow::Result<Value> {
- // Discover page tabs.
- let pages = cdp_list_pages(port).await?;
+ // Wrap the /json discovery in its own timeout — `cdp_list_pages` does an
+ // unbounded `read_to_end`, and a half-open localhost socket (browser
+ // mid-shutdown, firewall mismatch, port stolen by another process) would
+ // otherwise hang us forever. 10 s is generous for a localhost HTTP roundtrip.
+ let pages = tokio::time::timeout(
+ std::time::Duration::from_secs(10),
+ cdp_list_pages(port),
+ )
+ .await
+ .map_err(|_| anyhow::anyhow!("CDP /json discovery on port {port} timed out after 10 s"))??;
if pages.is_empty() {
anyhow::bail!("No CDP page tabs found on port {port}");
}
- // Take the first available page (future: match by window title hint).
let ws_url = pages[0]
.get("webSocketDebuggerUrl")
.and_then(|u| u.as_str())
@@ -115,7 +66,6 @@ async fn cdp_evaluate(
}
});
- // 30-second wall-clock timeout (covers awaited promises).
tokio::time::timeout(
std::time::Duration::from_secs(30),
cdp_ws_call(port, &ws_url, &cmd.to_string()),
@@ -125,7 +75,6 @@ async fn cdp_evaluate(
}
fn format_cdp_result(response: &Value) -> String {
- // If the result contains an exception, surface it.
if let Some(exc) = response
.get("result")
.and_then(|r| r.get("exceptionDetails"))
@@ -134,11 +83,11 @@ fn format_cdp_result(response: &Value) -> String {
.get("exception")
.and_then(|e| e.get("description"))
.and_then(|d| d.as_str())
- .unwrap_or_else(|| exc.to_string().leak()); // static lifetime for error msg
+ .map(str::to_owned)
+ .unwrap_or_else(|| exc.to_string());
return format!("exception: {msg}");
}
- // Extract the evaluated value.
if let Some(result) = response.get("result").and_then(|r| r.get("result")) {
if let Some(v) = result.get("value") {
return v.to_string();
@@ -153,12 +102,11 @@ fn format_cdp_result(response: &Value) -> String {
// ── HTTP page discovery ───────────────────────────────────────────────────
-/// GET http://127.0.0.1:{port}/json → array of page descriptors.
-/// Returns only entries whose `type` == "page" and that have a webSocketDebuggerUrl.
async fn cdp_list_pages(port: u16) -> anyhow::Result<Vec<Value>> {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
- let mut stream = tokio::net::TcpStream::connect(("127.0.0.1", port)).await
+ let mut stream = tokio::net::TcpStream::connect(("127.0.0.1", port))
+ .await
.map_err(|e| anyhow::anyhow!("Cannot connect to CDP on port {port}: {e}"))?;
let req = format!(
@@ -173,7 +121,8 @@ async fn cdp_list_pages(port: u16) -> anyhow::Result<Vec<Value>> {
.map_err(|_| anyhow::anyhow!("CDP /json response is not valid UTF-8"))?;
let body_start = text
.find("\r\n\r\n")
- .ok_or_else(|| anyhow::anyhow!("Invalid HTTP response from CDP"))? + 4;
+ .ok_or_else(|| anyhow::anyhow!("Invalid HTTP response from CDP"))?
+ + 4;
let all: Value = serde_json::from_str(&text[body_start..])
.map_err(|e| anyhow::anyhow!("CDP /json parse error: {e}"))?;
@@ -196,13 +145,9 @@ async fn cdp_list_pages(port: u16) -> anyhow::Result<Vec<Value>> {
// ── WebSocket CDP call ────────────────────────────────────────────────────
-/// Open a WebSocket to `ws_url` (on `port`), send `message`, and return the
-/// first response frame whose JSON `id` field equals 1.
-/// Intermediate event notifications (no `id` field or id != 1) are silently skipped.
async fn cdp_ws_call(port: u16, ws_url: &str, message: &str) -> anyhow::Result<Value> {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
- // Extract the URL path from e.g. "ws://127.0.0.1:9222/devtools/page/ABC".
let path: String = if let Some(rest) = ws_url.strip_prefix("ws://") {
rest.splitn(2, '/')
.nth(1)
@@ -212,11 +157,10 @@ async fn cdp_ws_call(port: u16, ws_url: &str, message: &str) -> anyhow::Result<V
ws_url.to_string()
};
- let mut stream = tokio::net::TcpStream::connect(("127.0.0.1", port)).await
+ let mut stream = tokio::net::TcpStream::connect(("127.0.0.1", port))
+ .await
.map_err(|e| anyhow::anyhow!("Cannot connect to CDP WebSocket on port {port}: {e}"))?;
- // WebSocket HTTP upgrade.
- // The Sec-WebSocket-Key is a static value — fine for localhost non-security use.
let handshake = format!(
"GET {path} HTTP/1.1\r\n\
Host: 127.0.0.1:{port}\r\n\
@@ -228,7 +172,6 @@ async fn cdp_ws_call(port: u16, ws_url: &str, message: &str) -> anyhow::Result<V
);
stream.write_all(handshake.as_bytes()).await?;
- // Read until end of HTTP response headers.
let mut hdr = Vec::with_capacity(512);
loop {
let mut b = [0u8; 1];
@@ -248,25 +191,23 @@ async fn cdp_ws_call(port: u16, ws_url: &str, message: &str) -> anyhow::Result<V
);
}
- // Send the CDP command as a masked WebSocket text frame.
ws_write_text(&mut stream, message.as_bytes()).await?;
- // Read frames, skipping events, until we receive id=1.
loop {
match ws_read_frame(&mut stream).await? {
WsFrame::Text(text) => {
- // Try to parse as JSON and check for id == 1.
if let Ok(val) = serde_json::from_str::<Value>(&text) {
if val.get("id").and_then(|id| id.as_u64()) == Some(1) {
return Ok(val);
}
- // Event notification — ignore and loop.
} else {
- anyhow::bail!("CDP returned non-JSON frame: {}", &text[..text.len().min(120)]);
+ anyhow::bail!(
+ "CDP returned non-JSON frame: {}",
+ &text[..text.len().min(120)]
+ );
}
}
WsFrame::Ping(data) => {
- // Respond with pong (RFC 6455 §5.5.3).
ws_write_pong(&mut stream, &data).await?;
}
WsFrame::Close => {
@@ -286,17 +227,15 @@ enum WsFrame {
Other,
}
-/// Read one complete (non-fragmented) WebSocket frame.
async fn ws_read_frame(stream: &mut tokio::net::TcpStream) -> anyhow::Result<WsFrame> {
use tokio::io::AsyncReadExt;
- // First 2 bytes: FIN+RSV+opcode, MASK+payload_len.
let mut hdr = [0u8; 2];
stream.read_exact(&mut hdr).await?;
- let opcode = hdr[0] & 0x0F;
- let masked = (hdr[1] & 0x80) != 0;
- let len_byte = (hdr[1] & 0x7F) as usize;
+ let opcode = hdr[0] & 0x0F;
+ let masked = (hdr[1] & 0x80) != 0;
+ let len_byte = (hdr[1] & 0x7F) as usize;
let payload_len: usize = match len_byte {
0..=125 => len_byte,
@@ -308,7 +247,6 @@ async fn ws_read_frame(stream: &mut tokio::net::TcpStream) -> anyhow::Result<WsF
_ => {
let mut b = [0u8; 8];
stream.read_exact(&mut b).await?;
- // Guard against absurdly large CDP messages (>16 MiB).
let n = u64::from_be_bytes(b) as usize;
if n > 16 * 1024 * 1024 {
anyhow::bail!("CDP WebSocket frame too large ({n} bytes)");
@@ -334,26 +272,23 @@ async fn ws_read_frame(stream: &mut tokio::net::TcpStream) -> anyhow::Result<WsF
}
match opcode {
- 1 => Ok(WsFrame::Text(
- String::from_utf8(payload)
- .map_err(|_| anyhow::anyhow!("CDP returned non-UTF-8 text frame"))?,
- )),
- 2 => Ok(WsFrame::Other), // binary — CDP doesn't use this
+ 1 => Ok(WsFrame::Text(String::from_utf8(payload).map_err(|_| {
+ anyhow::anyhow!("CDP returned non-UTF-8 text frame")
+ })?)),
+ 2 => Ok(WsFrame::Other),
8 => Ok(WsFrame::Close),
9 => Ok(WsFrame::Ping(payload)),
_ => Ok(WsFrame::Other),
}
}
-/// Write a masked WebSocket text frame (client → server, RFC 6455 §5.3 requires masking).
async fn ws_write_text(stream: &mut tokio::net::TcpStream, payload: &[u8]) -> anyhow::Result<()> {
use tokio::io::AsyncWriteExt;
- // Fixed mask key — fine for a local loopback connection.
const MASK: [u8; 4] = [0xDE, 0xAD, 0xBE, 0xEF];
let len = payload.len();
let mut frame = Vec::with_capacity(len + 10);
- frame.push(0x81u8); // FIN=1, opcode=TEXT(1)
+ frame.push(0x81u8);
encode_len_masked(&mut frame, len);
frame.extend_from_slice(&MASK);
for (i, &b) in payload.iter().enumerate() {
@@ -363,15 +298,14 @@ async fn ws_write_text(stream: &mut tokio::net::TcpStream, payload: &[u8]) -> an
Ok(())
}
-/// Write a masked WebSocket pong frame in response to a ping.
async fn ws_write_pong(stream: &mut tokio::net::TcpStream, payload: &[u8]) -> anyhow::Result<()> {
use tokio::io::AsyncWriteExt;
const MASK: [u8; 4] = [0xCA, 0xFE, 0xBA, 0xBE];
- let len = payload.len().min(125); // control frames ≤ 125 bytes (RFC 6455 §5.5)
+ let len = payload.len().min(125);
let mut frame = Vec::with_capacity(len + 6);
- frame.push(0x8Au8); // FIN=1, opcode=PONG(0xA)
- frame.push(0x80 | len as u8); // MASK bit + length
+ frame.push(0x8Au8);
+ frame.push(0x80 | len as u8);
frame.extend_from_slice(&MASK);
for (i, &b) in payload[..len].iter().enumerate() {
frame.push(b ^ MASK[i % 4]);
@@ -380,7 +314,6 @@ async fn ws_write_pong(stream: &mut tokio::net::TcpStream, payload: &[u8]) -> an
Ok(())
}
-/// Encode a (masked) length field per RFC 6455 §5.2.
fn encode_len_masked(buf: &mut Vec<u8>, len: usize) {
if len < 126 {
buf.push(0x80 | len as u8);
diff --git a/libs/cua-driver-rs/crates/mcp-server/src/lib.rs b/libs/cua-driver-rs/crates/mcp-server/src/lib.rs
index 5815e7e9f5..acf3d94279 100644
--- a/libs/cua-driver-rs/crates/mcp-server/src/lib.rs
+++ b/libs/cua-driver-rs/crates/mcp-server/src/lib.rs
@@ -10,7 +10,8 @@
//! - Each request has `jsonrpc: "2.0"`, `id` (any), `method`, optional `params`
//! - Notifications (no `id`) are silently ignored
-pub mod browser_eval;
+pub mod cdp;
+pub mod page;
pub mod protocol;
pub mod recording;
pub mod recording_tools;
diff --git a/libs/cua-driver-rs/crates/mcp-server/src/page.rs b/libs/cua-driver-rs/crates/mcp-server/src/page.rs
new file mode 100644
index 0000000000..d064ea3dbb
--- /dev/null
+++ b/libs/cua-driver-rs/crates/mcp-server/src/page.rs
@@ -0,0 +1,273 @@
+//! Cross-platform `page` tool: browser JS execution, text extraction, and DOM querying.
+//!
+//! The tool itself (name, schema, action dispatch) lives here so every platform
+//! gets the same MCP surface. Each platform crate supplies a `PageBackend` impl
+//! that routes the read/write primitives to platform-native code:
+//!
+//! - **macOS** uses Apple Events for Chromium/Safari, CDP for Electron, and
+//! the AX tree as a WKWebView fallback. See
+//! `platform-macos/src/tools/page.rs`.
+//! - **Windows** uses UIA TextPattern + FindAll for `get_text` / `query_dom`,
+//! and the shared CDP client (`crate::cdp`) for `execute_javascript`.
+//! - **Linux** uses AT-SPI for read paths, CDP for JS exec.
+//!
+//! `enable_javascript_apple_events` is intentionally macOS-only; the default
+//! trait impl returns "not supported on this platform" so non-macOS callers
+//! get a clear error instead of a panic.
+
+use async_trait::async_trait;
+use serde_json::Value;
+use std::sync::Arc;
+
+use crate::{
+ protocol::ToolResult,
+ tool::{Tool, ToolDef},
+};
+
+/// Platform-specific backend for the `page` tool.
+///
+/// All methods take `(pid, window_id)` for consistency across actions.
+/// `bundle_id` is resolved by the backend if needed (macOS uses it for
+/// AppleScript routing; Windows/Linux ignore it).
+#[async_trait]
+pub trait PageBackend: Send + Sync {
+ /// Returns the visible text of the page (rough analog of
+ /// `document.body.innerText`).
+ async fn get_text(&self, pid: i32, window_id: u32) -> anyhow::Result<String>;
+
+ /// Find elements matching `css_selector` and return a formatted-text
+ /// response (same human-readable shape macOS already emits).
+ async fn query_dom(
+ &self,
+ pid: i32,
+ window_id: u32,
+ css_selector: &str,
+ attributes: &[String],
+ ) -> anyhow::Result<String>;
+
+ /// Evaluate `javascript` against the page and return the stringified
+ /// result. Backends without a JS path should return an actionable error.
+ async fn execute_javascript(
+ &self,
+ pid: i32,
+ window_id: u32,
+ javascript: &str,
+ ) -> anyhow::Result<String>;
+
+ /// Whether this backend can execute JS at all. False → the tool short-
+ /// circuits with a "not supported here" error; true → the backend may
+ /// still return an error at call time (e.g. CDP port not reachable).
+ fn supports_javascript(&self) -> bool {
+ true
+ }
+
+ /// macOS-only: enable "Allow JavaScript from Apple Events" on the named
+ /// browser bundle. Default impl returns an unsupported-platform error.
+ async fn enable_javascript_apple_events(&self, _bundle_id: &str) -> anyhow::Result<String> {
+ anyhow::bail!(
+ "enable_javascript_apple_events is only supported on macOS \
+ (Chrome/Brave/Edge use Apple Events as the JS backend there)."
+ )
+ }
+}
+
+/// The cross-platform `page` MCP tool. Holds a backend trait object the
+/// host platform constructs and passes in at registration time.
+pub struct PageTool {
+ backend: Arc<dyn PageBackend>,
+}
+
+impl PageTool {
+ pub fn new(backend: Arc<dyn PageBackend>) -> Self {
+ Self { backend }
+ }
+}
+
+fn def() -> &'static ToolDef {
+ static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+ DEF.get_or_init(|| ToolDef {
+ name: "page".into(),
+ description: "Interact with the browser page loaded in a running app. Supports \
+ Chrome, Brave, Edge, Safari (via AppleScript on macOS), Electron apps (via CDP), \
+ Chromium/Firefox on Windows (via UIA for read; CDP for execute_javascript when \
+ --remote-debugging-port is set), and WKWebView/Tauri/AT-SPI fallbacks.\n\n\
+ Actions:\n\
+ - execute_javascript: Run JS and return the result.\n\
+ - get_text: Extract visible text from the page.\n\
+ - query_dom: Find elements matching a CSS selector.\n\
+ - enable_javascript_apple_events: macOS-only — patch the browser's \
+ Preferences to allow JS from Apple Events (Chrome/Brave/Edge, requires user \
+ confirmation and a browser restart).".into(),
+ input_schema: serde_json::json!({
+ "type": "object",
+ // `pid` and `window_id` are required for every action except
+ // `enable_javascript_apple_events` (which targets a browser by
+ // bundle id, not a running process). Enforced per-action in
+ // `invoke` rather than via the schema so the JSON-Schema-only
+ // path still validates the common case.
+ "required": ["action"],
+ "properties": {
+ "pid": { "type": "integer", "description": "Target process ID." },
+ "window_id": { "type": "integer", "description": "Target window ID from list_windows." },
+ "action": {
+ "type": "string",
+ "enum": ["execute_javascript", "get_text", "query_dom", "enable_javascript_apple_events"],
+ "description": "Action to perform."
+ },
+ "javascript": {
+ "type": "string",
+ "description": "JavaScript to execute. Required for execute_javascript."
+ },
+ "css_selector": {
+ "type": "string",
+ "description": "CSS selector for query_dom (e.g. 'a', 'button', 'input', 'h1'-'h6', 'p', 'img', 'select', '*')."
+ },
+ "attributes": {
+ "type": "array",
+ "items": { "type": "string" },
+ "description": "Element attributes to include in query_dom results."
+ },
+ "bundle_id": {
+ "type": "string",
+ "description": "Bundle ID of the browser. Required for enable_javascript_apple_events (macOS only)."
+ },
+ "user_has_confirmed_enabling": {
+ "type": "boolean",
+ "description": "Must be true to proceed with enable_javascript_apple_events. \
+ This will quit and relaunch the browser."
+ }
+ },
+ "additionalProperties": false
+ }),
+ read_only: false,
+ destructive: false,
+ idempotent: false,
+ open_world: false,
+ })
+}
+
+#[async_trait]
+impl Tool for PageTool {
+ fn def(&self) -> &ToolDef {
+ def()
+ }
+
+ async fn invoke(&self, args: Value) -> ToolResult {
+ let action = match args.get("action").and_then(|v| v.as_str()) {
+ Some(v) => v.to_owned(),
+ None => return ToolResult::error("Missing required parameter: action"),
+ };
+
+ // `pid` / `window_id` are resolved per-action: every action except
+ // `enable_javascript_apple_events` needs both. We resolve once here
+ // so each arm can `?` on the Result and we get matching error text.
+ // Narrowing casts use `TryFrom` so out-of-range JSON numbers fail
+ // with an actionable error instead of silently truncating to the
+ // wrong process / window.
+ let resolve_pid = |args: &Value| -> Result<i32, String> {
+ let raw = args
+ .get("pid")
+ .and_then(|v| v.as_i64())
+ .ok_or_else(|| "Missing required parameter: pid".to_owned())?;
+ i32::try_from(raw).map_err(|_| format!("Invalid parameter: pid {raw} out of i32 range"))
+ };
+ let resolve_window_id = |args: &Value| -> Result<u32, String> {
+ let raw = args
+ .get("window_id")
+ .and_then(|v| v.as_u64())
+ .ok_or_else(|| "Missing required parameter: window_id".to_owned())?;
+ u32::try_from(raw)
+ .map_err(|_| format!("Invalid parameter: window_id {raw} out of u32 range"))
+ };
+
+ let (pid, window_id) = if action == "enable_javascript_apple_events" {
+ (0i32, 0u32) // unused
+ } else {
+ let pid = match resolve_pid(&args) {
+ Ok(v) => v,
+ Err(e) => return ToolResult::error(e),
+ };
+ let window_id = match resolve_window_id(&args) {
+ Ok(v) => v,
+ Err(e) => return ToolResult::error(e),
+ };
+ (pid, window_id)
+ };
+
+ match action.as_str() {
+ "enable_javascript_apple_events" => {
+ let confirmed = args
+ .get("user_has_confirmed_enabling")
+ .and_then(|v| v.as_bool())
+ .unwrap_or(false);
+ if !confirmed {
+ return ToolResult::error(
+ "Set user_has_confirmed_enabling=true to proceed. \
+ This will quit and relaunch the browser.",
+ );
+ }
+ let bid = match args.get("bundle_id").and_then(|v| v.as_str()) {
+ Some(s) if !s.is_empty() => s.to_owned(),
+ _ => {
+ return ToolResult::error(
+ "bundle_id is required for enable_javascript_apple_events",
+ )
+ }
+ };
+ match self.backend.enable_javascript_apple_events(&bid).await {
+ Ok(msg) => ToolResult::text(msg),
+ Err(e) => ToolResult::error(format!("{e}")),
+ }
+ }
+
+ "execute_javascript" => {
+ if !self.backend.supports_javascript() {
+ return ToolResult::error(
+ "execute_javascript is not supported by this platform's page backend. \
+ Use get_text or query_dom for read-only access.",
+ );
+ }
+ let js = match args.get("javascript").and_then(|v| v.as_str()) {
+ Some(v) => v.to_owned(),
+ None => return ToolResult::error("Missing required parameter: javascript"),
+ };
+ match self.backend.execute_javascript(pid, window_id, &js).await {
+ Ok(result) => ToolResult::text(result),
+ Err(e) => ToolResult::error(format!("{e}")),
+ }
+ }
+
+ "get_text" => match self.backend.get_text(pid, window_id).await {
+ Ok(text) => ToolResult::text(text),
+ Err(e) => ToolResult::error(format!("Page text extraction failed: {e}")),
+ },
+
+ "query_dom" => {
+ let selector = args
+ .get("css_selector")
+ .and_then(|v| v.as_str())
+ .unwrap_or("*")
+ .to_owned();
+ let attributes: Vec<String> = args
+ .get("attributes")
+ .and_then(|v| v.as_array())
+ .map(|arr| {
+ arr.iter()
+ .filter_map(|v| v.as_str().map(str::to_owned))
+ .collect()
+ })
+ .unwrap_or_default();
+ match self
+ .backend
+ .query_dom(pid, window_id, &selector, &attributes)
+ .await
+ {
+ Ok(text) => ToolResult::text(text),
+ Err(e) => ToolResult::error(format!("DOM query failed: {e}")),
+ }
+ }
+
+ other => ToolResult::error(format!("Unknown action: {other}")),
+ }
+ }
+}
diff --git a/libs/cua-driver-rs/crates/mcp-server/src/tool.rs b/libs/cua-driver-rs/crates/mcp-server/src/tool.rs
index da020ab8ea..b052fbd8dc 100644
--- a/libs/cua-driver-rs/crates/mcp-server/src/tool.rs
+++ b/libs/cua-driver-rs/crates/mcp-server/src/tool.rs
@@ -7,7 +7,6 @@ use async_trait::async_trait;
use serde_json::Value;
use crate::{
- browser_eval::BrowserEvalTool,
protocol::{Content, ToolResult},
recording::{now_ms, RecordingSession},
recording_tools::{
@@ -82,7 +81,6 @@ impl ToolRegistry {
self.register(Box::new(SetRecordingTool::new(session.clone())));
self.register(Box::new(GetRecordingStateTool::new(session)));
self.register(Box::new(ReplayTrajectoryTool));
- self.register(Box::new(BrowserEvalTool));
}
/// Wire up the replay tool's weak self-reference.
diff --git a/libs/cua-driver-rs/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver-rs/crates/platform-linux/src/tools/impl_.rs
index 59fd70de3b..df4339b9ad 100644
--- a/libs/cua-driver-rs/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver-rs/crates/platform-linux/src/tools/impl_.rs
@@ -2026,6 +2026,11 @@ pub fn build_registry() -> ToolRegistry {
r.register(Box::new(GetAccessibilityTreeTool));
r.register(Box::new(ZoomTool { state: state.clone() }));
r.register(Box::new(TypeTextCharsTool));
+ // Cross-platform `page` tool definition lives in mcp-server; Linux plugs
+ // in its AT-SPI + CDP backend here.
+ r.register(Box::new(mcp_server::page::PageTool::new(
+ Arc::new(super::page::LinuxPageBackend::new()),
+ )));
r.register_recording_tools();
r
}
diff --git a/libs/cua-driver-rs/crates/platform-linux/src/tools/mod.rs b/libs/cua-driver-rs/crates/platform-linux/src/tools/mod.rs
index f511768c65..ea79d24754 100644
--- a/libs/cua-driver-rs/crates/platform-linux/src/tools/mod.rs
+++ b/libs/cua-driver-rs/crates/platform-linux/src/tools/mod.rs
@@ -7,6 +7,8 @@ use mcp_server::tool::ToolRegistry;
#[cfg(target_os = "linux")]
mod impl_;
+#[cfg(target_os = "linux")]
+pub(crate) mod page;
#[cfg(not(target_os = "linux"))]
mod stubs;
diff --git a/libs/cua-driver-rs/crates/platform-linux/src/tools/page.rs b/libs/cua-driver-rs/crates/platform-linux/src/tools/page.rs
new file mode 100644
index 0000000000..5c8cab1037
--- /dev/null
+++ b/libs/cua-driver-rs/crates/platform-linux/src/tools/page.rs
@@ -0,0 +1,176 @@
+//! Linux implementation of the cross-platform `PageBackend` trait.
+//!
+//! Read paths use AT-SPI via the existing `walk_tree` infrastructure
+//! (`crate::atspi`). `execute_javascript` calls the shared
+//! `mcp_server::cdp` helper — Chromium/Electron must be launched with
+//! `--remote-debugging-port=N` for that to succeed.
+
+use async_trait::async_trait;
+use mcp_server::page::PageBackend;
+
+pub struct LinuxPageBackend;
+
+impl LinuxPageBackend {
+ pub fn new() -> Self {
+ Self
+ }
+}
+
+impl Default for LinuxPageBackend {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+#[async_trait]
+impl PageBackend for LinuxPageBackend {
+ async fn get_text(&self, pid: i32, window_id: u32) -> anyhow::Result<String> {
+ let pid_u = pid as u32;
+ let xid = window_id as u64;
+ let result = tokio::task::spawn_blocking(move || {
+ crate::atspi::walk_tree(pid_u, xid, None)
+ })
+ .await
+ .map_err(|e| anyhow::anyhow!("AT-SPI walk task failed: {e}"))?;
+ Ok(extract_text_from_markdown(&result.tree_markdown))
+ }
+
+ async fn query_dom(
+ &self,
+ pid: i32,
+ window_id: u32,
+ css_selector: &str,
+ _attributes: &[String],
+ ) -> anyhow::Result<String> {
+ let pid_u = pid as u32;
+ let xid = window_id as u64;
+ let selector = css_selector.to_owned();
+ let result = tokio::task::spawn_blocking(move || {
+ crate::atspi::walk_tree(pid_u, xid, None)
+ })
+ .await
+ .map_err(|e| anyhow::anyhow!("AT-SPI walk task failed: {e}"))?;
+
+ if selector.contains("[data-") {
+ anyhow::bail!(
+ "`[data-*]` selectors are not reachable via AT-SPI on Linux. \
+ Use `execute_javascript` for full DOM access (requires the browser \
+ launched with `--remote-debugging-port`)."
+ );
+ }
+
+ let role = role_for_selector(&selector);
+ let selector_is_wildcard = {
+ let s = selector.trim();
+ s.is_empty() || s == "*"
+ };
+ // `role_for_selector` collapses both "wildcard / match-all" AND
+ // "unrecognised" into `None`. The first should keep every node; the
+ // second should fail loudly so an agent doesn't get a misleading
+ // dump of every element on the page when its `.foo` / `#bar` query
+ // wasn't actually understood.
+ if role.is_none() && !selector_is_wildcard {
+ anyhow::bail!(
+ "Selector '{selector}' is not supported by Linux AT-SPI role mapping. \
+ Use a simple tag selector (a, button, input, textarea, h1-h6, img, li, \
+ p, select, *), or `execute_javascript` for full DOM access (requires the \
+ browser launched with `--remote-debugging-port`)."
+ );
+ }
+ let lines: Vec<String> = result
+ .nodes
+ .iter()
+ .filter(|n| match role.as_deref() {
+ Some(r) => n.role.eq_ignore_ascii_case(r),
+ None => true, // safe now — only reached when selector is wildcard
+ })
+ .map(|n| {
+ let name = n.name.as_deref().unwrap_or("");
+ let value = n.value.as_deref().unwrap_or("");
+ let index = n
+ .element_index
+ .map(|i| format!("[{i}] "))
+ .unwrap_or_default();
+ if value.is_empty() {
+ format!("- {index}{} \"{}\"", n.role, name)
+ } else {
+ format!("- {index}{} \"{}\" = \"{}\"", n.role, name, value)
+ }
+ })
+ .collect();
+
+ if lines.is_empty() {
+ Ok("No elements found.".to_owned())
+ } else {
+ Ok(lines.join("\n"))
+ }
+ }
+
+ async fn execute_javascript(
+ &self,
+ _pid: i32,
+ _window_id: u32,
+ javascript: &str,
+ ) -> anyhow::Result<String> {
+ let port: u16 = match std::env::var("CUA_DRIVER_CDP_PORT")
+ .ok()
+ .and_then(|s| s.parse::<u16>().ok())
+ {
+ Some(p) => p,
+ None => anyhow::bail!(
+ "Chromium's `execute_javascript` on Linux requires the browser launched \
+ with `--remote-debugging-port=N`. Relaunch the browser with the flag and \
+ export CUA_DRIVER_CDP_PORT=N before starting cua-driver, or use \
+ `get_text` / `query_dom` for read-only access."
+ ),
+ };
+ let result = mcp_server::cdp::evaluate(port, javascript, true).await?;
+ Ok(format!("cdp.runtime.evaluate.user_gesture: {result}"))
+ }
+}
+
+/// Map common CSS selectors to AT-SPI role names. Returns `None` for `*` or
+/// unrecognised selectors (caller treats `None` as "no role filter").
+fn role_for_selector(sel: &str) -> Option<String> {
+ let s = sel.trim();
+ // Trim trailing [attr=...] or .class / #id qualifiers — we filter on
+ // role only here (the macOS backend has richer JS-side filtering).
+ let tag_end = s
+ .find(|c: char| c == '[' || c == '.' || c == '#')
+ .unwrap_or(s.len());
+ let tag = &s[..tag_end];
+ let role = match tag.to_ascii_lowercase().as_str() {
+ "a" => "link",
+ "button" => "push button",
+ "input" | "textarea" => "entry",
+ "h1" | "h2" | "h3" | "h4" | "h5" | "h6" => "heading",
+ "img" => "image",
+ "li" => "list item",
+ "p" => "paragraph",
+ "select" => "combo box",
+ "" | "*" => return None,
+ _ => return None,
+ };
+ Some(role.to_owned())
+}
+
+/// Strip Markdown tree formatting and concatenate the human-readable text.
+fn extract_text_from_markdown(md: &str) -> String {
+ md.lines()
+ .map(|l| {
+ // Strip leading indent + bullet markers `-` / `[N]` so the result
+ // reads like flowing text.
+ let trimmed = l.trim_start();
+ let trimmed = trimmed.trim_start_matches('-').trim_start();
+ // Drop the `[N] ` element-index prefix.
+ if let Some(rest) = trimmed.strip_prefix('[') {
+ if let Some(end) = rest.find(']') {
+ return rest[end + 1..].trim_start().to_owned();
+ }
+ }
+ trimmed.to_owned()
+ })
+ .filter(|l| !l.is_empty())
+ .collect::<Vec<_>>()
+ .join("\n")
+}
diff --git a/libs/cua-driver-rs/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver-rs/crates/platform-macos/src/tools/mod.rs
index 1a3aa45971..35bfae9e2e 100644
--- a/libs/cua-driver-rs/crates/platform-macos/src/tools/mod.rs
+++ b/libs/cua-driver-rs/crates/platform-macos/src/tools/mod.rs
@@ -230,7 +230,11 @@ pub fn register_all(registry: &mut ToolRegistry) {
registry.register(Box::new(get_accessibility_tree::GetAccessibilityTreeTool::new(state.clone())));
registry.register(Box::new(zoom::ZoomTool { state: state.clone() }));
registry.register(Box::new(type_text_chars::TypeTextCharsTool::new(state.clone())));
- registry.register(Box::new(page::PageTool::new(state.clone())));
+ // Cross-platform `page` tool definition lives in mcp-server; macOS plugs in
+ // its Apple-Events / CDP / AX-tree backend here.
+ registry.register(Box::new(mcp_server::page::PageTool::new(
+ Arc::new(page::MacOsPageBackend::new(state.clone())),
+ )));
// Recording / replay tools are platform-independent — live in mcp-server.
registry.register_recording_tools();
}
diff --git a/libs/cua-driver-rs/crates/platform-macos/src/tools/page.rs b/libs/cua-driver-rs/crates/platform-macos/src/tools/page.rs
index 15417f4c8f..f290b9e050 100644
--- a/libs/cua-driver-rs/crates/platform-macos/src/tools/page.rs
+++ b/libs/cua-driver-rs/crates/platform-macos/src/tools/page.rs
@@ -1,247 +1,159 @@
-//! PageTool: MCP tool for browser JS execution, text extraction, and DOM querying.
+//! macOS implementation of the cross-platform `PageBackend` trait.
+//!
+//! Routes by `bundle_id` / Electron-detection / WKWebView-detection to:
+//! - **Apple Events** (Chromium-family + Safari) — zero-config, uses
+//! `osascript do JavaScript`.
+//! - **CDP** (Electron) — port discovered from the running process.
+//! - **AX-tree fallback** (WKWebView / Tauri) — read-only, no JS.
+//!
+//! The MCP tool definition (name, schema, action dispatch) lives in
+//! `mcp_server::page` — this file only implements the backend.
use async_trait::async_trait;
-use mcp_server::{protocol::ToolResult, tool::{Tool, ToolDef}};
-use serde_json::Value;
+use mcp_server::page::PageBackend;
use std::sync::Arc;
use super::ToolState;
-use crate::browser::{BrowserJs, ElectronJs, is_wk_web_view_app, AXPageReader};
+use crate::browser::{is_wk_web_view_app, AXPageReader, BrowserJs, ElectronJs};
-pub struct PageTool {
+pub struct MacOsPageBackend {
+ #[allow(dead_code)] // reserved for future per-pid caches / config
pub state: Arc<ToolState>,
}
-impl PageTool {
- pub fn new(state: Arc<ToolState>) -> Self { Self { state } }
-}
-
-static DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+impl MacOsPageBackend {
+ pub fn new(state: Arc<ToolState>) -> Self {
+ Self { state }
+ }
-fn def() -> &'static ToolDef {
- DEF.get_or_init(|| ToolDef {
- name: "page".into(),
- description: "Interact with the browser page loaded in a running app. Supports \
- Chrome, Brave, Edge, Safari (via AppleScript), Electron apps (via CDP), \
- and WKWebView/Tauri apps (via AX tree fallback).\n\n\
- Actions:\n\
- - execute_javascript: Run JS and return the result.\n\
- - get_text: Extract visible text from the page.\n\
- - query_dom: Find elements matching a CSS selector.\n\
- - enable_javascript_apple_events: Patch Preferences to allow JS from Apple Events \
- (Chrome/Brave/Edge only, requires user confirmation and browser restart).".into(),
- input_schema: serde_json::json!({
- "type": "object",
- "required": ["pid", "window_id", "action"],
- "properties": {
- "pid": { "type": "integer", "description": "Target process ID." },
- "window_id": { "type": "integer", "description": "Target window ID from list_windows." },
- "action": {
- "type": "string",
- "enum": ["execute_javascript", "get_text", "query_dom", "enable_javascript_apple_events"],
- "description": "Action to perform."
- },
- "javascript": {
- "type": "string",
- "description": "JavaScript to execute. Required for execute_javascript."
- },
- "css_selector": {
- "type": "string",
- "description": "CSS selector for query_dom (e.g. 'a', 'button', 'input', 'h1'-'h6', 'p', 'img', 'select', '*')."
- },
- "attributes": {
- "type": "array",
- "items": { "type": "string" },
- "description": "Element attributes to include in query_dom results."
- },
- "bundle_id": {
- "type": "string",
- "description": "Bundle ID of the browser. Required for enable_javascript_apple_events."
- },
- "user_has_confirmed_enabling": {
- "type": "boolean",
- "description": "Must be true to proceed with enable_javascript_apple_events. \
- This will quit and relaunch the browser."
- }
- },
- "additionalProperties": false
- }),
- read_only: false,
- destructive: false,
- idempotent: false,
- open_world: false,
- })
+ /// Resolve `bundle_id` for `pid` via the running-apps list.
+ async fn bundle_id_for(pid: i32) -> String {
+ tokio::task::spawn_blocking(move || {
+ crate::apps::list_running_apps()
+ .into_iter()
+ .find(|a| a.pid == pid)
+ .and_then(|a| a.bundle_id)
+ .unwrap_or_default()
+ })
+ .await
+ .unwrap_or_default()
+ }
}
#[async_trait]
-impl Tool for PageTool {
- fn def(&self) -> &ToolDef { def() }
+impl PageBackend for MacOsPageBackend {
+ async fn get_text(&self, pid: i32, window_id: u32) -> anyhow::Result<String> {
+ let bundle_id = Self::bundle_id_for(pid).await;
- async fn invoke(&self, args: Value) -> ToolResult {
- let pid = match args.get("pid").and_then(|v| v.as_i64()) {
- Some(v) => v as i32,
- None => return ToolResult::error("Missing required parameter: pid"),
- };
- let window_id = match args.get("window_id").and_then(|v| v.as_u64()) {
- Some(v) => v as u32,
- None => return ToolResult::error("Missing required parameter: window_id"),
- };
- let action = match args.get("action").and_then(|v| v.as_str()) {
- Some(v) => v.to_owned(),
- None => return ToolResult::error("Missing required parameter: action"),
- };
-
- // Resolve bundle_id from pid.
- let bundle_id = {
- let p = pid;
- tokio::task::spawn_blocking(move || {
- crate::apps::list_running_apps()
- .into_iter()
- .find(|a| a.pid == p)
- .and_then(|a| a.bundle_id)
- .unwrap_or_default()
- }).await.unwrap_or_default()
- };
+ let use_ax_fallback = !BrowserJs::supports(&bundle_id)
+ && tokio::task::spawn_blocking(move || is_wk_web_view_app(pid))
+ .await
+ .unwrap_or(false);
- match action.as_str() {
- "enable_javascript_apple_events" => {
- let confirmed = args.get("user_has_confirmed_enabling")
- .and_then(|v| v.as_bool())
- .unwrap_or(false);
- if !confirmed {
- return ToolResult::error(
- "Set user_has_confirmed_enabling=true to proceed. \
- This will quit and relaunch the browser."
- );
- }
- let bid = args.get("bundle_id")
- .and_then(|v| v.as_str())
- .unwrap_or(&bundle_id)
- .to_owned();
- if bid.is_empty() {
- return ToolResult::error("bundle_id is required for enable_javascript_apple_events");
- }
- match BrowserJs::enable_javascript_apple_events(&bid).await {
- Ok(()) => ToolResult::text("JavaScript from Apple Events has been enabled. The browser is restarting."),
- Err(e) => ToolResult::error(format!("Failed to enable JavaScript Apple Events: {e}")),
- }
- }
+ if use_ax_fallback {
+ return ax_text_fallback(pid, window_id).await;
+ }
- "execute_javascript" => {
- let js = match args.get("javascript").and_then(|v| v.as_str()) {
- Some(v) => v.to_owned(),
- None => return ToolResult::error("Missing required parameter: javascript"),
- };
- match execute_js(&js, &bundle_id, pid, window_id).await {
- Ok(result) => ToolResult::text(result),
- Err(e) => ToolResult::error(format!("{e}")),
- }
- }
+ // Try JS path first; on error, fall back to AX walk.
+ match execute_js("document.body.innerText", &bundle_id, pid, window_id).await {
+ Ok(result) => Ok(result),
+ Err(_) => ax_text_fallback(pid, window_id).await,
+ }
+ }
- "get_text" => {
- let use_ax_fallback = !BrowserJs::supports(&bundle_id) && {
- let p = pid;
- tokio::task::spawn_blocking(move || is_wk_web_view_app(p)).await.unwrap_or(false)
- };
+ async fn query_dom(
+ &self,
+ pid: i32,
+ window_id: u32,
+ css_selector: &str,
+ attributes: &[String],
+ ) -> anyhow::Result<String> {
+ let bundle_id = Self::bundle_id_for(pid).await;
+
+ let use_ax_fallback = !BrowserJs::supports(&bundle_id)
+ && tokio::task::spawn_blocking(move || is_wk_web_view_app(pid))
+ .await
+ .unwrap_or(false);
+
+ if use_ax_fallback {
+ let results = ax_query_fallback(pid, window_id, css_selector).await?;
+ return Ok(format_ax_elements(&results));
+ }
- if use_ax_fallback {
- match ax_text_fallback(pid, window_id).await {
- Ok(text) => ToolResult::text(text),
- Err(e) => ToolResult::error(format!("AX text extraction failed: {e}")),
- }
- } else {
- match execute_js("document.body.innerText", &bundle_id, pid, window_id).await {
- Ok(result) => ToolResult::text(result),
- Err(_) => {
- // Try AX fallback on error.
- match ax_text_fallback(pid, window_id).await {
- Ok(text) => ToolResult::text(text),
- Err(e) => ToolResult::error(format!("Page text extraction failed: {e}")),
- }
- }
- }
- }
+ let js = build_query_selector_js(css_selector, attributes);
+ match execute_js(&js, &bundle_id, pid, window_id).await {
+ Ok(result) => Ok(result),
+ Err(_) => {
+ let results = ax_query_fallback(pid, window_id, css_selector).await?;
+ Ok(format_ax_elements(&results))
}
+ }
+ }
- "query_dom" => {
- let selector = args.get("css_selector")
- .and_then(|v| v.as_str())
- .unwrap_or("*")
- .to_owned();
- let attributes: Vec<String> = args.get("attributes")
- .and_then(|v| v.as_array())
- .map(|arr| arr.iter().filter_map(|v| v.as_str().map(str::to_owned)).collect())
- .unwrap_or_default();
-
- let use_ax_fallback = !BrowserJs::supports(&bundle_id) && {
- let p = pid;
- tokio::task::spawn_blocking(move || is_wk_web_view_app(p)).await.unwrap_or(false)
- };
-
- if use_ax_fallback {
- match ax_query_fallback(pid, window_id, &selector).await {
- Ok(results) => ToolResult::text(format_ax_elements(&results)),
- Err(e) => ToolResult::error(format!("AX DOM query failed: {e}")),
- }
- } else {
- let js = build_query_selector_js(&selector, &attributes);
- match execute_js(&js, &bundle_id, pid, window_id).await {
- Ok(result) => ToolResult::text(result),
- Err(_) => {
- // Try AX fallback.
- match ax_query_fallback(pid, window_id, &selector).await {
- Ok(results) => ToolResult::text(format_ax_elements(&results)),
- Err(e) => ToolResult::error(format!("DOM query failed: {e}")),
- }
- }
- }
- }
- }
+ async fn execute_javascript(
+ &self,
+ pid: i32,
+ window_id: u32,
+ javascript: &str,
+ ) -> anyhow::Result<String> {
+ let bundle_id = Self::bundle_id_for(pid).await;
+ execute_js(javascript, &bundle_id, pid, window_id).await
+ }
- other => ToolResult::error(format!("Unknown action: {other}")),
- }
+ async fn enable_javascript_apple_events(&self, bundle_id: &str) -> anyhow::Result<String> {
+ BrowserJs::enable_javascript_apple_events(bundle_id).await?;
+ Ok(
+ "JavaScript from Apple Events has been enabled. The browser is restarting."
+ .to_owned(),
+ )
}
}
/// Route JavaScript execution to the appropriate backend.
-async fn execute_js(js: &str, bundle_id: &str, pid: i32, window_id: u32) -> anyhow::Result<String> {
+async fn execute_js(
+ js: &str,
+ bundle_id: &str,
+ pid: i32,
+ window_id: u32,
+) -> anyhow::Result<String> {
if BrowserJs::supports(bundle_id) {
- BrowserJs::execute(js, bundle_id, window_id).await
- } else {
- let p = pid;
- let is_electron = tokio::task::spawn_blocking(move || ElectronJs::is_electron(p)).await?;
- if is_electron {
- ElectronJs::execute(js, pid).await
- } else {
- let p = pid;
- let is_wk = tokio::task::spawn_blocking(move || is_wk_web_view_app(p)).await?;
- if is_wk {
- anyhow::bail!(
- "execute_javascript is not available for WKWebView/Tauri apps. \
- Use get_text or query_dom instead."
- );
- } else {
- anyhow::bail!("Unsupported browser: bundle_id={bundle_id}");
- }
- }
+ return BrowserJs::execute(js, bundle_id, window_id).await;
+ }
+ let is_electron = tokio::task::spawn_blocking(move || ElectronJs::is_electron(pid)).await?;
+ if is_electron {
+ return ElectronJs::execute(js, pid).await;
}
+ let is_wk = tokio::task::spawn_blocking(move || is_wk_web_view_app(pid)).await?;
+ if is_wk {
+ anyhow::bail!(
+ "execute_javascript is not available for WKWebView/Tauri apps. \
+ Use get_text or query_dom instead."
+ );
+ }
+ anyhow::bail!("Unsupported browser: bundle_id={bundle_id}");
}
/// Extract page text via the AX tree.
async fn ax_text_fallback(pid: i32, window_id: u32) -> anyhow::Result<String> {
let result = tokio::task::spawn_blocking(move || {
crate::ax::tree::walk_tree(pid, Some(window_id), None)
- }).await
+ })
+ .await
.map_err(|e| anyhow::anyhow!("AX walk task failed: {e}"))?;
Ok(AXPageReader::extract_text(&result.tree_markdown))
}
/// Query AX tree by CSS selector.
-async fn ax_query_fallback(pid: i32, window_id: u32, selector: &str) -> anyhow::Result<Vec<crate::browser::ax_page_reader::AXElement>> {
+async fn ax_query_fallback(
+ pid: i32,
+ window_id: u32,
+ selector: &str,
+) -> anyhow::Result<Vec<crate::browser::ax_page_reader::AXElement>> {
let sel = selector.to_owned();
let result = tokio::task::spawn_blocking(move || {
crate::ax::tree::walk_tree(pid, Some(window_id), None)
- }).await
+ })
+ .await
.map_err(|e| anyhow::anyhow!("AX walk task failed: {e}"))?;
Ok(AXPageReader::query(&sel, &result.tree_markdown))
}
@@ -253,9 +165,21 @@ fn format_ax_elements(elements: &[crate::browser::ax_page_reader::AXElement]) ->
let mut lines = Vec::new();
for el in elements {
let index_str = el.index.map(|i| format!("[{i}] ")).unwrap_or_default();
- let title = if !el.title.is_empty() { format!(" \"{}\"", el.title) } else { String::new() };
- let value = if !el.value.is_empty() { format!(" = \"{}\"", el.value) } else { String::new() };
- let desc = if !el.description.is_empty() { format!(" ({})", el.description) } else { String::new() };
+ let title = if !el.title.is_empty() {
+ format!(" \"{}\"", el.title)
+ } else {
+ String::new()
+ };
+ let value = if !el.value.is_empty() {
+ format!(" = \"{}\"", el.value)
+ } else {
+ String::new()
+ };
+ let desc = if !el.description.is_empty() {
+ format!(" ({})", el.description)
+ } else {
+ String::new()
+ };
lines.push(format!("- {index_str}{}{title}{value}{desc}", el.role));
}
lines.join("\n")
@@ -265,7 +189,8 @@ fn format_ax_elements(elements: &[crate::browser::ax_page_reader::AXElement]) ->
fn build_query_selector_js(selector: &str, attributes: &[String]) -> String {
let escaped_sel = json_string(selector);
let attrs_js = if attributes.is_empty() {
- "['tagName','textContent','href','id','class','type','value','placeholder','name']".to_owned()
+ "['tagName','textContent','href','id','class','type','value','placeholder','name']"
+ .to_owned()
} else {
let parts: Vec<String> = attributes.iter().map(|a| json_string(a)).collect();
format!("[{}]", parts.join(","))
diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
index 36c2e69de3..dae21c5d77 100644
--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
@@ -4010,6 +4010,11 @@ pub fn build_registry() -> ToolRegistry {
// means it doesn't show up in `tools/list` either, matching Swift's
// ToolRegistry.swift (`type_text_chars` not in `handlers`).
let _: &TypeTextCharsTool = &TypeTextCharsTool { _state: state.clone() }; // touch struct so it stays in this crate for now
+ // Cross-platform `page` tool definition lives in mcp-server; Windows plugs
+ // in its UIA TextPattern + FindAll backend (CDP for execute_javascript).
+ r.register(Box::new(mcp_server::page::PageTool::new(
+ std::sync::Arc::new(super::page::WindowsPageBackend::new()),
+ )));
r.register_recording_tools();
r
}
diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/mod.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/mod.rs
index c808d1efed..cc0acae3da 100644
--- a/libs/cua-driver-rs/crates/platform-windows/src/tools/mod.rs
+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/mod.rs
@@ -7,6 +7,8 @@ use mcp_server::tool::ToolRegistry;
#[cfg(target_os = "windows")]
mod impl_;
+#[cfg(target_os = "windows")]
+pub(crate) mod page;
#[cfg(not(target_os = "windows"))]
mod stubs;
diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/page.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/page.rs
new file mode 100644
index 0000000000..3cda6c17fe
--- /dev/null
+++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/page.rs
@@ -0,0 +1,483 @@
+//! Windows implementation of the cross-platform `PageBackend` trait.
+//!
+//! - `get_text` — find the Document UIA element under the active tab's web-
+//! content pane and read `TextPattern.DocumentRange.GetText(-1)`. Falls
+//! back to the window-wide tree text if no Document control is present
+//! (e.g. Firefox in some configurations exposes the content under a
+//! different control type).
+//! - `query_dom` — `FindAll(TreeScope_Subtree, condition)` rooted at the web
+//! Document. Common CSS selectors map to UIA `ControlType` conditions
+//! (`a[href]` → `Hyperlink`, `button` → `Button`, `input[type=*]` →
+//! `Edit`, `h1`-`h6` → `Heading`, `li` → `ListItem`, etc.). `#id` /
+//! `.class` / `[role=*]` filter the matches after the fact by reading the
+//! element's `AutomationId` / IA2 attributes. `[data-*]` is not
+//! reachable via UIA/IA2 — that path errors with a pointer at
+//! `execute_javascript`.
+//! - `execute_javascript` — uses the shared `mcp_server::cdp` helper. When
+//! the browser was not launched with `--remote-debugging-port=N`, returns
+//! an actionable error.
+
+use async_trait::async_trait;
+use mcp_server::page::PageBackend;
+
+use windows::core::Interface;
+use windows::Win32::Foundation::HWND;
+use windows::Win32::System::Com::{
+ CoCreateInstance, CoInitializeEx, CLSCTX_INPROC_SERVER, COINIT_MULTITHREADED,
+};
+use windows::Win32::UI::Accessibility::{
+ CUIAutomation, IUIAutomation, IUIAutomationElement, IUIAutomationTextPattern,
+ TreeScope_Subtree, UIA_ButtonControlTypeId, UIA_CONTROLTYPE_ID, UIA_ControlTypePropertyId,
+ UIA_DocumentControlTypeId, UIA_EditControlTypeId, UIA_HeaderControlTypeId,
+ UIA_HyperlinkControlTypeId, UIA_ImageControlTypeId, UIA_ListItemControlTypeId,
+ UIA_NamePropertyId, UIA_PaneControlTypeId, UIA_TextControlTypeId, UIA_TextPatternId,
+ UIA_ValueValuePropertyId,
+};
+
+pub struct WindowsPageBackend;
+
+impl WindowsPageBackend {
+ pub fn new() -> Self {
+ Self
+ }
+}
+
+impl Default for WindowsPageBackend {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+#[async_trait]
+impl PageBackend for WindowsPageBackend {
+ async fn get_text(&self, _pid: i32, window_id: u32) -> anyhow::Result<String> {
+ let hwnd = window_id as u64;
+ tokio::task::spawn_blocking(move || unsafe { get_text_blocking(hwnd) })
+ .await
+ .map_err(|e| anyhow::anyhow!("join error: {e}"))?
+ }
+
+ async fn query_dom(
+ &self,
+ _pid: i32,
+ window_id: u32,
+ css_selector: &str,
+ attributes: &[String],
+ ) -> anyhow::Result<String> {
+ let hwnd = window_id as u64;
+ let selector = css_selector.to_owned();
+ let attrs: Vec<String> = attributes.to_vec();
+ tokio::task::spawn_blocking(move || unsafe { query_dom_blocking(hwnd, &selector, &attrs) })
+ .await
+ .map_err(|e| anyhow::anyhow!("join error: {e}"))?
+ }
+
+ async fn execute_javascript(
+ &self,
+ _pid: i32,
+ _window_id: u32,
+ javascript: &str,
+ ) -> anyhow::Result<String> {
+ // Discover CDP port from env. The "right" approach (extracting the
+ // --remote-debugging-port from the process's command line) requires
+ // reading PEB; for now we accept the conventional env var and fail
+ // with an actionable message if it isn't set.
+ let port: u16 = match std::env::var("CUA_DRIVER_CDP_PORT")
+ .ok()
+ .and_then(|s| s.parse::<u16>().ok())
+ {
+ Some(p) => p,
+ None => anyhow::bail!(
+ "Chromium's `execute_javascript` on Windows requires the browser launched \
+ with `--remote-debugging-port=N`. cua-driver doesn't yet ship a companion \
+ extension; relaunch the browser with the flag and export \
+ CUA_DRIVER_CDP_PORT=N before starting cua-driver, or use `get_text` / \
+ `query_dom` for read-only access."
+ ),
+ };
+ let result = mcp_server::cdp::evaluate(port, javascript, true).await?;
+ Ok(format!("cdp.runtime.evaluate.user_gesture: {result}"))
+ }
+}
+
+// ── implementation ────────────────────────────────────────────────────────
+
+unsafe fn get_text_blocking(hwnd: u64) -> anyhow::Result<String> {
+ let (automation, root) = uia_root_for(hwnd)?;
+ let document = find_document_descendant(&automation, &root)?;
+ if let Some(doc) = document {
+ text_pattern_dump(&doc)
+ } else {
+ // No Document control — try TextPattern on the root.
+ match text_pattern_dump(&root) {
+ Ok(s) => Ok(s),
+ Err(_) => Ok(walk_text_blocking(&root)),
+ }
+ }
+}
+
+unsafe fn query_dom_blocking(
+ hwnd: u64,
+ selector: &str,
+ attributes: &[String],
+) -> anyhow::Result<String> {
+ let (automation, root) = uia_root_for(hwnd)?;
+ let root_for_search = find_document_descendant(&automation, &root)?.unwrap_or(root);
+
+ let parsed = parse_selector(selector)?;
+
+ if parsed.is_data_attr {
+ anyhow::bail!(
+ "`[data-*]` selectors are not reachable via UIA/IA2 on Windows. \
+ Use `execute_javascript` for full DOM access (requires the browser launched \
+ with `--remote-debugging-port`)."
+ );
+ }
+
+ // Build a UIA condition for the ControlType (if any) and FindAll.
+ let condition = match parsed.control_type {
+ Some(ct) => automation.CreatePropertyCondition(
+ UIA_ControlTypePropertyId,
+ &ct.0.into(),
+ )?,
+ None => automation.CreateTrueCondition()?,
+ };
+
+ let matches = root_for_search.FindAll(TreeScope_Subtree, &condition)?;
+ let count = matches.Length().unwrap_or(0);
+
+ let mut formatted: Vec<String> = Vec::new();
+ for i in 0..count {
+ let elem = match matches.GetElement(i) {
+ Ok(e) => e,
+ Err(_) => continue,
+ };
+
+ // Post-filter by id/class. Skip elements that don't satisfy.
+ if !parsed.id_filter.is_empty() {
+ let aid = read_string(&elem, |e| {
+ e.CurrentAutomationId().map(|b| b.to_string()).map_err(|e| e.into())
+ });
+ if aid.as_deref().unwrap_or_default() != parsed.id_filter {
+ continue;
+ }
+ }
+
+ let name = read_string(&elem, |e| {
+ e.CurrentName().map(|b| b.to_string()).map_err(|e| e.into())
+ })
+ .unwrap_or_default();
+
+ // attributes the caller asked for. We map a small set of well-known
+ // CSS attribute names onto UIA properties.
+ let mut attr_parts: Vec<String> = Vec::new();
+ for a in attributes {
+ let v = match a.as_str() {
+ "name" => Some(name.clone()),
+ "id" => read_string(&elem, |e| {
+ e.CurrentAutomationId().map(|b| b.to_string()).map_err(|e| e.into())
+ }),
+ "value" => read_value_value(&elem),
+ "href" => {
+ // No native UIA "href" — closest is the AccessibleValue
+ // for hyperlinks (Chromium populates this with the URL).
+ read_value_value(&elem)
+ }
+ "textContent" | "text" => Some(name.clone()),
+ _ => None,
+ };
+ if let Some(v) = v {
+ attr_parts.push(format!("{a}=\"{}\"", escape_attr(&v)));
+ }
+ }
+
+ let ct = read_control_type(&elem).unwrap_or_else(|| "Unknown".to_owned());
+ let line = if attr_parts.is_empty() {
+ format!("- {ct} \"{}\"", escape_attr(&name))
+ } else {
+ format!("- {ct} \"{}\" [{}]", escape_attr(&name), attr_parts.join(" "))
+ };
+ formatted.push(line);
+ }
+
+ if formatted.is_empty() {
+ Ok("No elements found.".to_owned())
+ } else {
+ Ok(formatted.join("\n"))
+ }
+}
+
+// ── UIA helpers ───────────────────────────────────────────────────────────
+
+/// Initialise COM (best-effort) and resolve the UIA root element for `hwnd`.
+unsafe fn uia_root_for(hwnd: u64) -> anyhow::Result<(IUIAutomation, IUIAutomationElement)> {
+ let _ = CoInitializeEx(None, COINIT_MULTITHREADED);
+ let automation: IUIAutomation = CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER)
+ .map_err(|e| anyhow::anyhow!("UIA init failed: {e}"))?;
+ let hwnd_win = HWND(hwnd as *mut _);
+ let root = automation
+ .ElementFromHandle(hwnd_win)
+ .map_err(|e| anyhow::anyhow!("ElementFromHandle failed: {e}"))?;
+ Ok((automation, root))
+}
+
+/// Find the first descendant Document control under `root`. The web view's
+/// `Document` is the root of the rendered DOM in both Chromium and Firefox.
+unsafe fn find_document_descendant(
+ automation: &IUIAutomation,
+ root: &IUIAutomationElement,
+) -> anyhow::Result<Option<IUIAutomationElement>> {
+ let doc_id: i32 = UIA_DocumentControlTypeId.0;
+ let cond = automation.CreatePropertyCondition(UIA_ControlTypePropertyId, &doc_id.into())?;
+ match root.FindFirst(TreeScope_Subtree, &cond) {
+ Ok(elem) => Ok(Some(elem)),
+ Err(_) => Ok(None),
+ }
+}
+
+/// Try `TextPattern.DocumentRange.GetText(-1)` on `elem`. Returns an error if
+/// the element does not support TextPattern.
+unsafe fn text_pattern_dump(elem: &IUIAutomationElement) -> anyhow::Result<String> {
+ let pattern_raw = elem
+ .GetCurrentPattern(UIA_TextPatternId)
+ .map_err(|e| anyhow::anyhow!("element does not support TextPattern: {e}"))?;
+ let text_pattern: IUIAutomationTextPattern = pattern_raw
+ .cast()
+ .map_err(|e| anyhow::anyhow!("cast to IUIAutomationTextPattern failed: {e}"))?;
+ let range = text_pattern
+ .DocumentRange()
+ .map_err(|e| anyhow::anyhow!("DocumentRange failed: {e}"))?;
+ let bstr = range
+ .GetText(-1)
+ .map_err(|e| anyhow::anyhow!("GetText(-1) failed: {e}"))?;
+ Ok(bstr.to_string())
+}
+
+/// Last-resort: concatenate Name properties of every descendant. Used when
+/// no Document control is present and the root has no TextPattern (very
+/// rare for Chromium/Firefox).
+unsafe fn walk_text_blocking(root: &IUIAutomationElement) -> String {
+ let mut acc = String::new();
+ if let Ok(name) = root.CurrentName() {
+ let s = name.to_string();
+ if !s.trim().is_empty() {
+ acc.push_str(&s);
+ acc.push('\n');
+ }
+ }
+ if let Ok(children) = root.FindAll(
+ TreeScope_Subtree,
+ // Match every element with a non-empty name.
+ &match create_true_condition_or_fallback() {
+ Some(c) => c,
+ None => return acc,
+ },
+ ) {
+ let count = children.Length().unwrap_or(0);
+ for i in 0..count {
+ if let Ok(c) = children.GetElement(i) {
+ if let Ok(name) = c.CurrentName() {
+ let s = name.to_string();
+ if !s.trim().is_empty() {
+ acc.push_str(&s);
+ acc.push('\n');
+ }
+ }
+ }
+ }
+ }
+ acc
+}
+
+unsafe fn create_true_condition_or_fallback() -> Option<windows::Win32::UI::Accessibility::IUIAutomationCondition>
+{
+ let automation: IUIAutomation =
+ CoCreateInstance(&CUIAutomation, None, CLSCTX_INPROC_SERVER).ok()?;
+ automation.CreateTrueCondition().ok()
+}
+
+unsafe fn read_string<F>(elem: &IUIAutomationElement, f: F) -> Option<String>
+where
+ F: FnOnce(&IUIAutomationElement) -> anyhow::Result<String>,
+{
+ let s = f(elem).ok()?;
+ if s.is_empty() {
+ None
+ } else {
+ Some(s)
+ }
+}
+
+unsafe fn read_control_type(elem: &IUIAutomationElement) -> Option<String> {
+ // Match against the raw i32 value — using the `UIA_*ControlTypeId`
+ // constants directly in a match arm triggers `non_upper_case_globals`.
+ let ct = elem.CurrentControlType().ok()?;
+ let name = if ct.0 == UIA_ButtonControlTypeId.0 {
+ "Button"
+ } else if ct.0 == UIA_HyperlinkControlTypeId.0 {
+ "Hyperlink"
+ } else if ct.0 == UIA_EditControlTypeId.0 {
+ "Edit"
+ } else if ct.0 == UIA_ListItemControlTypeId.0 {
+ "ListItem"
+ } else if ct.0 == UIA_ImageControlTypeId.0 {
+ "Image"
+ } else if ct.0 == UIA_TextControlTypeId.0 {
+ "Text"
+ } else if ct.0 == UIA_DocumentControlTypeId.0 {
+ "Document"
+ } else if ct.0 == UIA_PaneControlTypeId.0 {
+ "Pane"
+ } else if ct.0 == UIA_HeaderControlTypeId.0 {
+ "Header"
+ } else {
+ return Some(format!("ControlType({})", ct.0));
+ };
+ Some(name.to_owned())
+}
+
+unsafe fn read_value_value(elem: &IUIAutomationElement) -> Option<String> {
+ let v = elem.GetCurrentPropertyValue(UIA_ValueValuePropertyId).ok()?;
+ let raw = v.as_raw();
+ if raw.Anonymous.Anonymous.vt != 8 {
+ // VT_BSTR = 8 — anything else (empty, VT_EMPTY=0) means no value.
+ return None;
+ }
+ // Read the BSTR pointer; we don't free it (windows-rs handles VARIANT
+ // lifetime on drop of `v`).
+ let bstr_ptr = raw.Anonymous.Anonymous.Anonymous.bstrVal;
+ if bstr_ptr.is_null() {
+ return None;
+ }
+ let bstr = windows::core::BSTR::from_raw(bstr_ptr);
+ let s = bstr.to_string();
+ // Don't double-free: hand the BSTR back to the VARIANT by leaking ours.
+ std::mem::forget(bstr);
+ Some(s)
+}
+
+// Avoid an unused-import warning when none of the secondary properties are
+// actually read above.
+#[allow(dead_code)]
+const _UNUSED_NAME_PROP: UIA_PROPERTY_ID_T = UIA_NamePropertyId_VAL;
+
+#[allow(non_camel_case_types)]
+type UIA_PROPERTY_ID_T = windows::Win32::UI::Accessibility::UIA_PROPERTY_ID;
+#[allow(non_upper_case_globals)]
+const UIA_NamePropertyId_VAL: UIA_PROPERTY_ID_T = UIA_NamePropertyId;
+
+// ── selector parsing ──────────────────────────────────────────────────────
+
+struct ParsedSelector {
+ control_type: Option<UIA_CONTROLTYPE_ID>,
+ id_filter: String,
+ is_data_attr: bool,
+}
+
+fn parse_selector(sel: &str) -> anyhow::Result<ParsedSelector> {
+ let s = sel.trim();
+
+ // Wildcard / empty → match-all, no post-filter.
+ if s.is_empty() || s == "*" {
+ return Ok(ParsedSelector {
+ control_type: None,
+ id_filter: String::new(),
+ is_data_attr: false,
+ });
+ }
+
+ // Reject `[data-*]` immediately — not reachable via UIA.
+ if s.contains("[data-") {
+ return Ok(ParsedSelector {
+ control_type: None,
+ id_filter: String::new(),
+ is_data_attr: true,
+ });
+ }
+
+ // Strip [attribute=value] suffix once we've consumed the tag.
+ let (tag, rest) = match s.find('[') {
+ Some(i) => (&s[..i], &s[i..]),
+ None => (s, ""),
+ };
+
+ // ID suffix on tag (e.g. `div#foo`). `.class` is NOT supported — UIA's
+ // `AutomationId` maps to HTML `id`, not to the class list. Supporting
+ // class selectors would need an IA2 `attributes`-string parse (tracked
+ // as a follow-up in the PR description).
+ if tag.contains('.') {
+ anyhow::bail!(
+ "`.class` selectors are not supported by the Windows UIA backend. \
+ Use a simple tag (a, button, input, textarea, h1-h6, img, li, p, \
+ span, select), a `tag#id` selector, an `[attr=value]` selector \
+ including `[role=…]`, or `execute_javascript` for full DOM access \
+ (requires the browser launched with `--remote-debugging-port`)."
+ );
+ }
+ let (tag_clean, id_filter) = match tag.find('#') {
+ Some(i) => (&tag[..i], tag[i + 1..].to_owned()),
+ None => (tag, String::new()),
+ };
+
+ let ct = control_type_for_tag(tag_clean);
+
+ // For `[role=X]` selectors with no tag, try to map role → ControlType.
+ let ct = if ct.is_none() && rest.starts_with("[role=") {
+ let role = rest
+ .trim_start_matches("[role=")
+ .trim_end_matches(']')
+ .trim_matches(|c: char| c == '"' || c == '\'');
+ control_type_for_role(role)
+ } else {
+ ct
+ };
+
+ // After mapping, if we have neither a control_type nor an id_filter, the
+ // selector wasn't understood — refuse so we don't dump the entire subtree
+ // as a false-positive match.
+ if ct.is_none() && id_filter.is_empty() && tag_clean != "*" && !tag_clean.is_empty() {
+ anyhow::bail!(
+ "Selector '{sel}' is not supported by the Windows UIA backend. \
+ Use a simple tag (a, button, input, textarea, h1-h6, img, li, p, \
+ span, select), a `tag#id` selector, `[role=…]`, or \
+ `execute_javascript` for full DOM access (requires the browser \
+ launched with `--remote-debugging-port`)."
+ );
+ }
+
+ Ok(ParsedSelector {
+ control_type: ct,
+ id_filter,
+ is_data_attr: false,
+ })
+}
+
+fn control_type_for_tag(tag: &str) -> Option<UIA_CONTROLTYPE_ID> {
+ Some(match tag.to_ascii_lowercase().as_str() {
+ "a" => UIA_HyperlinkControlTypeId,
+ "button" => UIA_ButtonControlTypeId,
+ "input" | "textarea" => UIA_EditControlTypeId,
+ "h1" | "h2" | "h3" | "h4" | "h5" | "h6" => UIA_HeaderControlTypeId,
+ "img" => UIA_ImageControlTypeId,
+ "li" => UIA_ListItemControlTypeId,
+ "p" | "span" => UIA_TextControlTypeId,
+ "" | "*" => return None,
+ _ => return None,
+ })
+}
+
+fn control_type_for_role(role: &str) -> Option<UIA_CONTROLTYPE_ID> {
+ Some(match role.to_ascii_lowercase().as_str() {
+ "button" => UIA_ButtonControlTypeId,
+ "link" => UIA_HyperlinkControlTypeId,
+ "textbox" | "searchbox" => UIA_EditControlTypeId,
+ "heading" => UIA_HeaderControlTypeId,
+ "img" | "image" => UIA_ImageControlTypeId,
+ "listitem" => UIA_ListItemControlTypeId,
+ _ => return None,
+ })
+}
+
+fn escape_attr(s: &str) -> String {
+ s.replace('"', "\\\"")
+}
diff --git a/libs/cua-driver-rs/tests/integration/test_api_parity.py b/libs/cua-driver-rs/tests/integration/test_api_parity.py
index 3fbe1c2c50..5413319378 100644
--- a/libs/cua-driver-rs/tests/integration/test_api_parity.py
+++ b/libs/cua-driver-rs/tests/integration/test_api_parity.py
@@ -130,14 +130,11 @@ REQUIRED_TOOLS = [
# In Rust, not in Swift yet. Tests for these will fail on the Swift binary.
RUST_ONLY_TOOLS = [
"type_text_chars", # per-character delay typing; Swift uses type_text
- "browser_eval", # CDP JS eval; Swift uses 'page' for browser JS
"get_accessibility_tree",# lightweight desktop AX snapshot (separate from get_window_state)
]
-# In Swift, not registered in Rust yet (page.rs exists but not in register_all).
-SWIFT_ONLY_TOOLS = [
- "page", # browser JS + get_text + query_dom; Rust uses browser_eval
-]
+# In Swift, not registered in Rust yet. (`page` is now cross-platform in Rust.)
+SWIFT_ONLY_TOOLS: list[str] = []
# ── parity mixin ───────────────────────────────────────────────────────────────
@@ -1070,7 +1067,7 @@ class _ParityMixin:
# ── stdio MCP: Rust-only tools (missing in Swift) ────────────────────────
def test_mcp_rust_only_tools_present(self) -> None:
- """Rust-only tools (type_text_chars, browser_eval, get_accessibility_tree).
+ """Rust-only tools (type_text_chars, get_accessibility_tree).
These are in the Rust binary but not yet in Swift. Both binaries
should eventually have them. This test fails on Swift until ported.
@@ -1104,11 +1101,9 @@ class _ParityMixin:
# ── stdio MCP: Swift-only tools (missing/unregistered in Rust) ───────────
def test_mcp_swift_only_tools_present(self) -> None:
- """Swift-only tools (page).
-
- In Swift, not yet registered in Rust (page.rs exists but not in
- register_all). Both binaries should eventually have them.
- This test fails on Rust until page::PageTool is added to register_all.
+ """Swift-only tools list — currently empty after the page-tool
+ unification in Rust. Kept as a no-op so the matrix stays symmetric
+ with `test_mcp_rust_only_tools_present`.
"""
with self._mcp() as c:
names = {t["name"] for t in c.list_tools()}
@@ -1118,25 +1113,6 @@ class _ParityMixin:
f"Swift-only tools missing: {missing} (not yet registered in this binary)",
)
- # ── stdio MCP: browser_eval ───────────────────────────────────────────────
-
- def test_mcp_browser_eval_is_registered(self) -> None:
- """'browser_eval' tool must appear in tools/list.
-
- Parity gap: Swift binary does not have 'browser_eval' — it uses 'page'
- for browser JS. This test documents the requirement that both binaries
- expose browser_eval. Fails on Swift until it is ported.
- """
- with self._mcp() as c:
- names = {t["name"] for t in c.list_tools()}
- self.assertIn("browser_eval", names, "'browser_eval' tool is not registered")
-
- def test_mcp_browser_eval_missing_pid_raises_error(self) -> None:
- """browser_eval with no pid should return an MCP error."""
- self._assert_tool_raises_mcp_error(
- "browser_eval", {"expression": "1+1"}
- )
-
# ── stdio MCP: protocol-level contracts ──────────────────────────────────
def test_mcp_tools_call_unknown_tool_returns_error(self) -> None:
diff --git a/libs/cua-driver-rs/tests/integration/v2/PHILOSOPHY.md b/libs/cua-driver-rs/tests/integration/v2/PHILOSOPHY.md
index 5f9bc53b1f..ec1b83144e 100644
--- a/libs/cua-driver-rs/tests/integration/v2/PHILOSOPHY.md
+++ b/libs/cua-driver-rs/tests/integration/v2/PHILOSOPHY.md
@@ -17,10 +17,10 @@ The driver exposes a small, stable interface:
| `scroll` | Scroll wheel |
| `drag` | Mouse drag |
-Tests **never** reach for `browser_eval`, CDP, Apple Events, or any other
-internal mechanism. The driver is a black box that delivers the action via
-whatever technique works best for the target app. Tests verify the
-*observable outcome*, not the delivery path.
+Tests **never** reach for `page.execute_javascript`, CDP, Apple Events, or
+any other internal mechanism. The driver is a black box that delivers the
+action via whatever technique works best for the target app. Tests verify
+the *observable outcome*, not the delivery path.
---