Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver-rs)(linux): gate native-Wayland backend behind opt-in flag (#1935)

Francesco Bonacci1e39e8bd85ff
diff --git a/docs/content/docs/cua-driver/guide/getting-started/linux.mdx b/docs/content/docs/cua-driver/guide/getting-started/linux.mdx
index 41295d3451..d4c0ed1dc4 100644
--- a/docs/content/docs/cua-driver/guide/getting-started/linux.mdx
+++ b/docs/content/docs/cua-driver/guide/getting-started/linux.mdx
@@ -21,7 +21,7 @@ The `display server` probe surfaces:
- **`DISPLAY` set, `WAYLAND_DISPLAY` unset** — pure X11 session. This is the expected path for the current pre-release backend.
- **`WAYLAND_DISPLAY` set, `DISPLAY` set** — Wayland session with XWayland. cua-driver talks to XWayland (treats the session as X11). Native-Wayland-only apps that bypass XWayland aren't visible to the tools — call this out as a known limitation.
-- **`WAYLAND_DISPLAY` set, `DISPLAY` unset** — pure Wayland session, no XWayland. **Not supported today**. Run an X11 session, or install / enable XWayland.
+- **`WAYLAND_DISPLAY` set, `DISPLAY` unset** — pure Wayland session, no XWayland. Unsupported by default. There is an **experimental native-Wayland backend** (toplevel enumeration + virtual-pointer / virtual-keyboard input on wlroots compositors like sway and labwc) behind an opt-in flag — see [Experimental native Wayland](#experimental-native-wayland). Otherwise, run an X11 session or install / enable XWayland.
- **Neither set** — headless. Window-driving tools will return errors. See [Headless workflow](#headless-workflow) for `Xvfb`.
<Callout type="warn">
@@ -33,6 +33,18 @@ The `display server` probe surfaces:
distros let you pick X11 vs Wayland at the login screen).
</Callout>
+### Experimental native Wayland
+
+A native-Wayland backend is shipping behind an opt-in flag. It enumerates toplevels via `zwlr_foreign_toplevel_manager_v1` and injects input via the wlroots virtual-pointer / virtual-keyboard protocols, so on **wlroots compositors** (sway, labwc, Hyprland, …) it can `list_windows` and drive input without XWayland. Screen capture and full AT-SPI parity are still landing, so it is **off by default** and a pure-Wayland session reports as unsupported until you opt in.
+
+Enable it by starting the daemon with the flag set:
+
+```bash
+CUA_DRIVER_RS_ENABLE_WAYLAND=1 cua-driver serve
+```
+
+`cua-driver doctor` reflects the state — a pure-Wayland session shows the backend as `OFF` (with the flag to set) or `ENABLED`. Treat it as experimental: report gaps rather than relying on it in production.
+
## AT-SPI prerequisite
The accessibility-tree tools (`get_window_state`, `click` with `element_index`, anything that walks UIA-equivalent structure) talk to the **AT-SPI 2** D-Bus bus. AT-SPI is the Linux accessibility framework — same role as macOS's AX or Windows' UI Automation. Cua Driver expects the AT-SPI bus to be reachable on your session bus.
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index e65ccef0a8..571f9eb3b5 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -2799,14 +2799,20 @@ impl Tool for CheckPermissionsTool {
"X11 display: {}\nWayland: {}\nAT-SPI (D-Bus): {}\nXSendEvent injection: {}",
if x11_ok { "✅ connected" } else { "❌ DISPLAY not set or X11 unavailable" },
match &wayland_display {
- Some(s) => format!("✅ native Wayland session (WAYLAND_DISPLAY={s})"),
+ Some(s) if crate::wayland::wayland_enabled() =>
+ format!("✅ native Wayland session (WAYLAND_DISPLAY={s}) — experimental backend ENABLED"),
+ Some(s) => format!(
+ "⚠️ native Wayland session (WAYLAND_DISPLAY={s}) — experimental backend OFF; \
+ set {}=1 to enable it",
+ crate::wayland::ENABLE_WAYLAND_ENV
+ ),
None => "❌ not a Wayland session".to_string(),
},
if atspi_ok { "✅ D-Bus session available" } else { "⚠️ D-Bus session not detected" },
if x11_ok { "✅ available" } else { "❌ requires X11" }
);
ToolResult::text(status_text)
- .with_structured(json!({ "x11": x11_ok, "wayland": wayland_display.is_some(), "atspi": atspi_ok, "xsend_event": x11_ok }))
+ .with_structured(json!({ "x11": x11_ok, "wayland": wayland_display.is_some(), "wayland_enabled": crate::wayland::wayland_enabled(), "atspi": atspi_ok, "xsend_event": x11_ok }))
}
}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
index c5a2a8979e..5c7d0bf9ad 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
@@ -28,10 +28,37 @@ const BTN_LEFT: u32 = 0x110;
use crate::x11::WindowInfo;
-/// True when we should drive Wayland rather than X11: a Wayland display is
-/// present and there is no X11 DISPLAY to fall back to.
+/// Name of the opt-in env var that unlocks the experimental native-Wayland
+/// backend.
+pub const ENABLE_WAYLAND_ENV: &str = "CUA_DRIVER_RS_ENABLE_WAYLAND";
+
+/// Whether the user has opted into the experimental native-Wayland backend.
+///
+/// The Wayland backend is incomplete (toplevel enumeration + virtual-pointer /
+/// virtual-keyboard input via the wlroots protocols; capture and AT-SPI parity
+/// are still landing), so it stays OFF by default and a pure-Wayland session is
+/// reported as unsupported unless the user explicitly sets
+/// `CUA_DRIVER_RS_ENABLE_WAYLAND=1`. Any value other than empty / `0` / `false`
+/// enables it.
+pub fn wayland_enabled() -> bool {
+ match std::env::var(ENABLE_WAYLAND_ENV) {
+ Ok(v) => {
+ let v = v.trim();
+ !v.is_empty() && v != "0" && !v.eq_ignore_ascii_case("false")
+ }
+ Err(_) => false,
+ }
+}
+
+/// True when we should drive Wayland rather than X11: the experimental backend
+/// is opted in ([`wayland_enabled`]), a Wayland display is present, and there is
+/// no X11 DISPLAY to fall back to. Without the opt-in this returns false even on
+/// a pure-Wayland session, so the backend treats it as unsupported rather than
+/// silently engaging an incomplete code path.
pub fn is_wayland() -> bool {
- std::env::var_os("WAYLAND_DISPLAY").is_some() && std::env::var_os("DISPLAY").is_none()
+ wayland_enabled()
+ && std::env::var_os("WAYLAND_DISPLAY").is_some()
+ && std::env::var_os("DISPLAY").is_none()
}
fn wl_sockets(dir: &str) -> std::collections::HashSet<String> {
diff --git a/nix/cua-driver/tests/wayland/session.nix b/nix/cua-driver/tests/wayland/session.nix
index 72d5df3fe0..8f1fa48533 100644
--- a/nix/cua-driver/tests/wayland/session.nix
+++ b/nix/cua-driver/tests/wayland/session.nix
@@ -199,6 +199,9 @@ let
export GDK_BACKEND=wayland
export QT_QPA_PLATFORM=wayland
export LIBGL_ALWAYS_SOFTWARE=1
+ # The native-Wayland backend is opt-in (off by default); these tests exist
+ # to exercise it, so turn it on for every Wayland-session driver wrapper.
+ export CUA_DRIVER_RS_ENABLE_WAYLAND=1
'';
driverWrapper =