Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver-rs)(linux): gate native-Wayland backend behind opt-in flag (#1935)
Francesco Bonacci1e39e8bd85ff
diff --git a/docs/content/docs/cua-driver/guide/getting-started/linux.mdx b/docs/content/docs/cua-driver/guide/getting-started/linux.mdxindex 41295d3451..d4c0ed1dc4 100644--- a/docs/content/docs/cua-driver/guide/getting-started/linux.mdx+++ b/docs/content/docs/cua-driver/guide/getting-started/linux.mdx@@ -21,7 +21,7 @@ The `display server` probe surfaces:- **`DISPLAY` set, `WAYLAND_DISPLAY` unset** — pure X11 session. This is the expected path for the current pre-release backend.- **`WAYLAND_DISPLAY` set, `DISPLAY` set** — Wayland session with XWayland. cua-driver talks to XWayland (treats the session as X11). Native-Wayland-only apps that bypass XWayland aren't visible to the tools — call this out as a known limitation.-- **`WAYLAND_DISPLAY` set, `DISPLAY` unset** — pure Wayland session, no XWayland. **Not supported today**. Run an X11 session, or install / enable XWayland.+- **`WAYLAND_DISPLAY` set, `DISPLAY` unset** — pure Wayland session, no XWayland. Unsupported by default. There is an **experimental native-Wayland backend** (toplevel enumeration + virtual-pointer / virtual-keyboard input on wlroots compositors like sway and labwc) behind an opt-in flag — see [Experimental native Wayland](#experimental-native-wayland). Otherwise, run an X11 session or install / enable XWayland.- **Neither set** — headless. Window-driving tools will return errors. See [Headless workflow](#headless-workflow) for `Xvfb`.<Callout type="warn">@@ -33,6 +33,18 @@ The `display server` probe surfaces:distros let you pick X11 vs Wayland at the login screen).</Callout>+### Experimental native Wayland++A native-Wayland backend is shipping behind an opt-in flag. It enumerates toplevels via `zwlr_foreign_toplevel_manager_v1` and injects input via the wlroots virtual-pointer / virtual-keyboard protocols, so on **wlroots compositors** (sway, labwc, Hyprland, …) it can `list_windows` and drive input without XWayland. Screen capture and full AT-SPI parity are still landing, so it is **off by default** and a pure-Wayland session reports as unsupported until you opt in.++Enable it by starting the daemon with the flag set:++```bash+CUA_DRIVER_RS_ENABLE_WAYLAND=1 cua-driver serve+```++`cua-driver doctor` reflects the state — a pure-Wayland session shows the backend as `OFF` (with the flag to set) or `ENABLED`. Treat it as experimental: report gaps rather than relying on it in production.+## AT-SPI prerequisiteThe accessibility-tree tools (`get_window_state`, `click` with `element_index`, anything that walks UIA-equivalent structure) talk to the **AT-SPI 2** D-Bus bus. AT-SPI is the Linux accessibility framework — same role as macOS's AX or Windows' UI Automation. Cua Driver expects the AT-SPI bus to be reachable on your session bus.diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex e65ccef0a8..571f9eb3b5 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -2799,14 +2799,20 @@ impl Tool for CheckPermissionsTool {"X11 display: {}\nWayland: {}\nAT-SPI (D-Bus): {}\nXSendEvent injection: {}",if x11_ok { "✅ connected" } else { "❌ DISPLAY not set or X11 unavailable" },match &wayland_display {- Some(s) => format!("✅ native Wayland session (WAYLAND_DISPLAY={s})"),+ Some(s) if crate::wayland::wayland_enabled() =>+ format!("✅ native Wayland session (WAYLAND_DISPLAY={s}) — experimental backend ENABLED"),+ Some(s) => format!(+ "⚠️ native Wayland session (WAYLAND_DISPLAY={s}) — experimental backend OFF; \+ set {}=1 to enable it",+ crate::wayland::ENABLE_WAYLAND_ENV+ ),None => "❌ not a Wayland session".to_string(),},if atspi_ok { "✅ D-Bus session available" } else { "⚠️ D-Bus session not detected" },if x11_ok { "✅ available" } else { "❌ requires X11" });ToolResult::text(status_text)- .with_structured(json!({ "x11": x11_ok, "wayland": wayland_display.is_some(), "atspi": atspi_ok, "xsend_event": x11_ok }))+ .with_structured(json!({ "x11": x11_ok, "wayland": wayland_display.is_some(), "wayland_enabled": crate::wayland::wayland_enabled(), "atspi": atspi_ok, "xsend_event": x11_ok }))}}diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rsindex c5a2a8979e..5c7d0bf9ad 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs@@ -28,10 +28,37 @@ const BTN_LEFT: u32 = 0x110;use crate::x11::WindowInfo;-/// True when we should drive Wayland rather than X11: a Wayland display is-/// present and there is no X11 DISPLAY to fall back to.+/// Name of the opt-in env var that unlocks the experimental native-Wayland+/// backend.+pub const ENABLE_WAYLAND_ENV: &str = "CUA_DRIVER_RS_ENABLE_WAYLAND";++/// Whether the user has opted into the experimental native-Wayland backend.+///+/// The Wayland backend is incomplete (toplevel enumeration + virtual-pointer /+/// virtual-keyboard input via the wlroots protocols; capture and AT-SPI parity+/// are still landing), so it stays OFF by default and a pure-Wayland session is+/// reported as unsupported unless the user explicitly sets+/// `CUA_DRIVER_RS_ENABLE_WAYLAND=1`. Any value other than empty / `0` / `false`+/// enables it.+pub fn wayland_enabled() -> bool {+ match std::env::var(ENABLE_WAYLAND_ENV) {+ Ok(v) => {+ let v = v.trim();+ !v.is_empty() && v != "0" && !v.eq_ignore_ascii_case("false")+ }+ Err(_) => false,+ }+}++/// True when we should drive Wayland rather than X11: the experimental backend+/// is opted in ([`wayland_enabled`]), a Wayland display is present, and there is+/// no X11 DISPLAY to fall back to. Without the opt-in this returns false even on+/// a pure-Wayland session, so the backend treats it as unsupported rather than+/// silently engaging an incomplete code path.pub fn is_wayland() -> bool {- std::env::var_os("WAYLAND_DISPLAY").is_some() && std::env::var_os("DISPLAY").is_none()+ wayland_enabled()+ && std::env::var_os("WAYLAND_DISPLAY").is_some()+ && std::env::var_os("DISPLAY").is_none()}fn wl_sockets(dir: &str) -> std::collections::HashSet<String> {diff --git a/nix/cua-driver/tests/wayland/session.nix b/nix/cua-driver/tests/wayland/session.nixindex 72d5df3fe0..8f1fa48533 100644--- a/nix/cua-driver/tests/wayland/session.nix+++ b/nix/cua-driver/tests/wayland/session.nix@@ -199,6 +199,9 @@ letexport GDK_BACKEND=waylandexport QT_QPA_PLATFORM=waylandexport LIBGL_ALWAYS_SOFTWARE=1+ # The native-Wayland backend is opt-in (off by default); these tests exist+ # to exercise it, so turn it on for every Wayland-session driver wrapper.+ export CUA_DRIVER_RS_ENABLE_WAYLAND=1'';driverWrapper =