Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): align Wayland capture, focus, and overlays (#3152)

Jacob Mink2fc59ea943a0
diff --git a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
index 686a5b5dde..c6d0cb65bc 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/LINUX.md
@@ -34,8 +34,11 @@ AT-SPI is talked to natively over D-Bus (the `atspi`/zbus crate) — no
clicked" case.
- The **agent cursor** is a synthetic overlay showing where the run is
acting; it never moves the real pointer (same model as macOS/Windows).
- It glides on clicks and `move_cursor`; issue `move_cursor` to make it
- track a field while typing advances focus across cells.
+ It glides on clicks and ordinary `move_cursor` calls; cursor-bearing and
+ keyboard actions re-show it automatically. Only the explicit
+ `move_cursor({x,y,scope:"desktop"})` escape hatch moves the compositor
+ cursor. Do not use desktop scope unless the user asked for real-pointer
+ control.
## `delivery_mode` — the background/foreground ladder
@@ -55,6 +58,12 @@ and Windows surface:
e.g. a GTK dialog button or a widget that only reads input while focused.
A brief focus swap unless the target was already active.
+Foreground is a user-visible takeover boundary. Never select it automatically.
+Use it only when the user already authorized foreground control for the
+workflow or after asking for approval. If background delivery refuses and that
+authorization is absent, return the refusal instead of changing the user's
+focus, workspace, or compositor cursor.
+
### Persistent focus-proxy exception
`bring_to_front` is not part of the normal input ladder. For an ordinary
@@ -226,8 +235,9 @@ control. Other focus-bound background pointer and keyboard shapes return an
exact `background_unavailable` result. They do not report success after a
silent drop.
-Use `delivery_mode:"foreground"` for raw Wayland input. The driver activates
-the selected target through a verified compositor adapter before dispatch. If
+Raw Wayland input requires explicitly authorized `delivery_mode:"foreground"`.
+The driver activates the selected target through a verified compositor adapter
+before dispatch. If
the compositor has no target-addressable activation or input backend, the call
refuses before sending input. Reconstructing coordinates alone does not make
raw background PX possible on a standard compositor.
diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
index 2ad9f79edf..18c0506d5a 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md
@@ -177,6 +177,13 @@ If you reach for a command that says "activate", "foreground",
"raise", or "make key", stop and translate to the cua-driver tool
that does the same intent without focus-stealing.
+`delivery_mode:"foreground"` is a user-visible takeover boundary, not an
+automatic retry. Use it only when the user already authorized foreground
+control for this workflow or after asking for approval. It may change focus,
+workspace, and the compositor cursor while the action runs; restoration is
+best-effort. If background delivery is unavailable and foreground control is
+not authorized, stop with the driver's refusal instead of silently escalating.
+
A desktop target is an explicit per-call choice to operate the visible desktop
and therefore uses foreground/system input. Use it only after the narrower
window ladder has been attempted and verified. Permission policy must still
@@ -185,17 +192,23 @@ or a public session label.
## GUI transport defaults — prefer cua-driver over GUI shell shims
-**Default transport is the `cua-driver` CLI** — `Bash` shelling out
-to `cua-driver <tool-name> '<JSON-args>'`. MCP tools (prefix
-`mcp__cua-driver__*`) only when the user explicitly asks for them.
-CLI wins because it picks up rebuilds instantly, failures are
-easier to diagnose, and there's no per-tool schema-load overhead.
+**Default transport is the `cua-driver` CLI for one-off calls** — `Bash`
+shelling out to `cua-driver <tool-name> '<JSON-args>'`. Each CLI invocation
+owns a disposable transport session that is cleaned up after its response.
+Use one persistent `cua-driver mcp` connection for a multi-call GUI workflow
+that needs shared cursor, recording, browser, or named-session state. A public
+session label is not a credential and a later one-shot process cannot adopt
+the previous process's lifecycle merely by repeating that label.
+
+CLI wins for isolated inspection and management because it picks up rebuilds
+instantly, failures are easier to diagnose, and there's no per-tool
+schema-load overhead. Persistent MCP wins for an ordered action loop.
Every reference to `click(...)`, `get_window_state(...)` etc. in this
skill means `cua-driver click '{...}'` — translate to MCP form only
when MCP is requested.
-### Claude Code computer-use compatibility mode
+### Claude Code computer-use compatibility flag
For normal Claude Code use, keep the default CLI or `cua-driver` MCP
server path above. If the user explicitly wants Claude Code's
@@ -205,20 +218,10 @@ vision/computer-use-style flow, they can register:
cua-driver mcp-config --client claude # then paste + run the printed line
```
-Observation: Claude Code vision flows appear to treat a screenshot
-MCP tool as the image-grounding anchor. This compatibility mode keeps
-the normal CuaDriver tools and changes only `screenshot`. The
-compatibility `screenshot` requires `pid` and `window_id`, captures
-only that target window, and returns the window-local pixel
-coordinate frame. Start with `launch_app` or `list_windows`, then
-call `screenshot({pid, window_id})`; do not assume desktop
-coordinates or a full-screen capture.
-
-Use MCP for this Claude Code vision/computer-use-style path. Do not
-shell out to `cua-driver screenshot` as a substitute: CLI screenshots
-still work as CuaDriver calls, but they do not expose the
-`mcp__cua-computer-use__screenshot` tool name that Claude Code
-appears to use as the image-grounding cue.
+The compatibility flag is retained for old setup snippets, but the standalone
+`screenshot` tool was removed. It does not add or replace tools. Use
+`get_window_state({pid, window_id})` for a window-local accessibility snapshot
+and PNG, or `get_desktop_state()` for an explicitly authorized desktop capture.
## Using cua-driver from the shell
@@ -237,18 +240,19 @@ Tool names are `snake_case`, management subcommands are
- `cua-driver recording start|stop|status` — see `RECORDING.md`
- `cua-driver check-update [--json] [--no-cache]` — read-only "is a newer release available?" probe. Same payload as the `check_for_update` MCP tool; pair with `cua-driver update --apply` to install.
-Canonical multi-step workflow (example shape — platform-specific
-launch idioms in the per-OS companion file):
+Canonical multi-step workflow within one persistent MCP connection (example
+shape — platform-specific launch idioms in the per-OS companion file):
```bash
-cua-driver serve
-cua-driver launch_app '{"bundle_id":"..."}'
+# Start the service once, then connect one MCP client with `cua-driver mcp`.
+# The calls below are tool calls on that same connection, not separate shell
+# invocations of `cua-driver <tool>`.
+launch_app({"bundle_id":"..."})
# → {pid: 844, windows: [{window_id: 10725, ...}]}
-cua-driver get_window_state '{"pid":844,"window_id":10725}'
+get_window_state({"pid":844,"window_id":10725})
# Use the returned structuredContent.elements[].element_token:
-cua-driver click '{"pid":844,"element_token":"s0000002a:14"}'
-cua-driver verify_state '{"pid":844,"window_id":10725,"expect":[{"element":{"selector":{"label_contains":"Saved"},"exists":true}}]}'
-cua-driver stop
+click({"pid":844,"element_token":"s0000002a:14"})
+verify_state({"pid":844,"window_id":10725,"expect":[{"element":{"selector":{"label_contains":"Saved"},"exists":true}}]})
```
For Chromium page content, keep the same native window selection but switch to
@@ -275,6 +279,15 @@ recording, and system activity. Motion knobs:
`end_handle`, `arc_size`, `arc_flow`, `spring` — tuneable at runtime,
persisted to config.
+**Agent-control safety.** Keep this overlay enabled whenever the agent is
+controlling pointer or keyboard input. Cursor-bearing and keyboard actions
+automatically re-show their session cursor, even if it was hidden while idle.
+On Linux, ordinary `move_cursor({x,y})` moves only this synthetic cursor.
+`move_cursor({x,y,scope:"desktop"})` is the explicit escape hatch that moves
+the user's compositor cursor and must not be used unless the user asked for
+desktop-pointer control. Raw Wayland input that requires
+`delivery_mode:"foreground"` crosses the same user-visible takeover boundary.
+
Delivery and target context is shown as host-owned chips inside the session
badge. Themes own the twelve action animations only. The session name and
context chips fade independently, so an active tool can show its execution
@@ -625,6 +638,8 @@ if resp.effect == "suspected_noop"
# Route 4 — background delivery was dropped (insert/click never arrived)
if resp.escalation.target == "foreground"
or the px action still did nothing:
+ require existing user authorization for visible foreground control
+ otherwise stop and ask; do not retry automatically
re-call the same action with delivery_mode:"foreground"
# on Wayland this is the ONLY escalation — px-bg can't target an
# unfocused window there; see LINUX.md
@@ -767,10 +782,10 @@ The response carries:
```bash
# write to file — stdout stays readable (AX/UIA tree / summary only, no base64)
-cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.jpg"}'
+cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.png"}'
# CLI --screenshot-out-file flag is equivalent
-cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.jpg
+cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.png
```
Pass `screenshot_out_file` when using `get_window_state` via CLI or
@@ -891,12 +906,14 @@ Two consequences for callers:
`scroll`, `type_text`, `press_key`, `hotkey` — uniformly. The
`foreground` rung briefly fronts the target, acts, then restores the
prior frontmost: the explicit last resort when a background attempt
- didn't land. **`foreground` is a reaction, never a prediction.** Always
+ didn't land. **`foreground` is a reaction and a user-authorization gate,
+ never a prediction.** Always
fire the `background` default first and let the driver tell you it
can't (a `background_unavailable` error with
`escalation.recommended == "foreground"`, or a successful action result
with `escalation.target == "foreground"`) — or observe a confirmed no-op —
- _before_ you escalate.
+ _before_ you consider escalation. Then use it only when the user already
+ authorized visible foreground control or after asking for approval.
Do **not** reason "it's a GTK/Chromium/Electron app, so background will
drop, so I'll front up-front": the toolkit lists in the tool schemas
are the _driver's_ internal detectors, not a checklist for you to front
diff --git a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
index e0526c7c91..504b5f427c 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md
@@ -823,12 +823,10 @@ typed browser tools yet.
materialized yet. Re-call `list_windows({pid: N})` after 500ms;
for chronic cases, key off the app name in `list_windows({})`
output.
-- **JPEG screenshot has more compression than expected** — default
- quality on the MCP screenshot compat path is 85; for raw
- `cua-driver call screenshot`, defaults to PNG (no compression).
- Pass `{format: "jpeg", quality: 70}` to opt into compressed
- screenshots. The `max_image_dimension` config (default 2048)
- downscales via Lanczos3 before encoding.
+- **Need a screenshot file** — use `get_window_state` with
+ `screenshot_out_file`; the result is PNG. The removed standalone screenshot
+ compatibility inputs (`format` and `quality`) are rejected by the live tool
+ schema. Downsample or convert the saved PNG outside cua-driver if needed.
## Diagnostics
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/skills.rs b/libs/cua-driver/rust/crates/cua-driver/src/skills.rs
index e2f8d701e8..e335b5867e 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/skills.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/skills.rs
@@ -1203,6 +1203,38 @@ mod tests {
}
}
+ #[test]
+ fn bundled_skill_keeps_agent_control_non_interfering_by_default() {
+ let crate_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
+ let skill = std::fs::read_to_string(crate_dir.join("../../Skills/cua-driver/SKILL.md"))
+ .expect("canonical skill must be readable");
+ let linux = std::fs::read_to_string(crate_dir.join("../../Skills/cua-driver/LINUX.md"))
+ .expect("canonical Linux skill must be readable");
+
+ for required in [
+ "`delivery_mode:\"foreground\"` is a user-visible takeover boundary",
+ "ordinary `move_cursor({x,y})` moves only this synthetic cursor",
+ "must not be used unless the user asked for",
+ "do not retry automatically",
+ "stop with the driver's refusal instead of silently escalating",
+ ] {
+ assert!(
+ skill.contains(required),
+ "skill lost required agent-control safety guidance: {required}"
+ );
+ }
+ for required in [
+ "keyboard actions re-show it automatically",
+ "Never select it automatically",
+ "explicitly authorized `delivery_mode:\"foreground\"`",
+ ] {
+ assert!(
+ linux.contains(required),
+ "Linux skill lost required non-interference guidance: {required}"
+ );
+ }
+ }
+
#[test]
fn extract_flat_tarball_v_0_2_20_plus() {
// Post-fix shape: one wrapper dir, files directly under it.
diff --git a/libs/cua-driver/rust/crates/cua-driver/tests/agent_cursor_showcase_test.rs b/libs/cua-driver/rust/crates/cua-driver/tests/agent_cursor_showcase_test.rs
index 0e21737c1d..bf8025d39c 100644
--- a/libs/cua-driver/rust/crates/cua-driver/tests/agent_cursor_showcase_test.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/tests/agent_cursor_showcase_test.rs
@@ -94,7 +94,7 @@ fn semantic_cursor_showcase_records_session_and_action_states() {
// visible for two seconds, so this settle stays inside that window.
settle(900);
- let cursor_png = capture_cursor_oracle_png(&mut driver);
+ let cursor_png = capture_cursor_oracle_png(&mut driver, width, height);
let cursor_frame = image::load_from_memory(&cursor_png)
.expect("decode cursor desktop screenshot")
.to_rgba8();
@@ -300,34 +300,49 @@ fn capture_desktop_png(driver: &mut McpDriver) -> (Vec<u8>, f64, f64) {
(png, width, height)
}
-fn capture_cursor_oracle_png(driver: &mut McpDriver) -> Vec<u8> {
+fn capture_cursor_oracle_png(driver: &mut McpDriver, width: f64, height: f64) -> Vec<u8> {
#[cfg(target_os = "linux")]
{
- // A compositor-less X11 root read can omit the overlay client's own
- // shaped window even while an independent display capture sees the
- // complete cursor and badge. The action evidence recorder is the
- // canonical external behavior oracle and has already captured the
- // move_cursor after-frame before this call returns.
- let path = driver
- .recording_dir()
- .expect("showcase recording directory")
- .join("turn-00001/after.png");
- for _ in 0..20 {
- if let Ok(png) = std::fs::read(&path) {
- if !png.is_empty() {
- return png;
- }
- }
- settle(50);
- }
- panic!(
- "action evidence did not produce the cursor after-frame at {}",
- path.display()
+ let _ = driver;
+ // XGetImage root reads can omit a shaped overlay client's pixels on a
+ // compositor-less X11 server. Capture the composed display exactly as
+ // the behavioral recording does so the oracle observes what a user sees.
+ let display = std::env::var("DISPLAY").expect("Linux cursor showcase requires DISPLAY");
+ let video_size = format!("{}x{}", width.round() as u32, height.round() as u32);
+ let output = std::process::Command::new("ffmpeg")
+ .args([
+ "-hide_banner",
+ "-loglevel",
+ "error",
+ "-f",
+ "x11grab",
+ "-draw_mouse",
+ "0",
+ "-video_size",
+ &video_size,
+ "-i",
+ &display,
+ "-frames:v",
+ "1",
+ "-f",
+ "image2pipe",
+ "-vcodec",
+ "png",
+ "pipe:1",
+ ])
+ .output()
+ .expect("launch ffmpeg X11 display capture");
+ assert!(
+ output.status.success() && !output.stdout.is_empty(),
+ "ffmpeg X11 display capture failed: {}",
+ String::from_utf8_lossy(&output.stderr)
);
+ output.stdout
}
#[cfg(not(target_os = "linux"))]
{
+ let _ = (width, height);
capture_desktop_png(driver).0
}
}
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
index 185c7f341b..be79e5b15d 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
@@ -407,4 +407,22 @@ mod pointer_tracking_tests {
assert!((x - (120.0 + heading.cos() * 16.0)).abs() < f64::EPSILON);
assert!((y - (80.0 + heading.sin() * 16.0)).abs() < f64::EPSILON);
}
+
+ #[test]
+ fn session_cleanup_removes_named_cursor_but_preserves_anonymous_default() {
+ let registry = CursorRegistry::new();
+ registry.update_position("session-a", 12.0, 34.0);
+ registry.update_position("default", 56.0, 78.0);
+
+ registry.remove("session-a");
+ registry.remove("default");
+
+ assert!(registry.get("session-a").is_none());
+ assert_eq!(
+ registry
+ .get("default")
+ .and_then(|cursor| cursor.x.zip(cursor.y)),
+ Some((56.0, 78.0))
+ );
+ }
}
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
index a5505ebc50..cb1e393962 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
@@ -990,6 +990,23 @@ mod session_badge_and_action_tests {
use super::*;
use crate::{CursorConfig, DeliveryModifier, TargetModifier};
+ #[test]
+ fn idle_hide_zero_keeps_a_positioned_session_cursor_visible() {
+ let mut core = RenderStateCore::new(CursorConfig::default());
+ core.motion.idle_hide_ms = 0.0;
+ assert!(core.apply_command_base(
+ OverlayCommand::ClickPulse { x: 40.0, y: 60.0 },
+ false,
+ false,
+ ));
+
+ core.tick_motion(2.0);
+
+ assert!(core.cursor_is_revealed());
+ assert_eq!(core.pos, (40.0, 60.0));
+ assert_eq!(core.idle_alpha, 1.0);
+ }
+
#[test]
fn session_badge_holds_then_fades_once() {
let mut core = RenderStateCore::new(CursorConfig::default());
diff --git a/libs/cua-driver/rust/crates/platform-linux/Cargo.toml b/libs/cua-driver/rust/crates/platform-linux/Cargo.toml
index 5a54b732aa..b6f74fa209 100644
--- a/libs/cua-driver/rust/crates/platform-linux/Cargo.toml
+++ b/libs/cua-driver/rust/crates/platform-linux/Cargo.toml
@@ -53,7 +53,7 @@ wayland-protocols-wlr = { version = "0.3", features = ["client"] }
# wp-viewporter (HiDPI for the layer-shell overlay). The transitive pull
# from wayland-protocols-wlr does NOT enable `staging`; declaring it here
# unifies the feature globally.
-wayland-protocols = { version = "0.32", features = ["client", "staging"] }
+wayland-protocols = { version = "0.32", features = ["client", "staging", "unstable"] }
# xdg-desktop-portal client: display-screenshot tier
# (org.freedesktop.portal.Screenshot), per-window ScreenCast
# (org.freedesktop.portal.ScreenCast), and the libei input session
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/lib.rs b/libs/cua-driver/rust/crates/platform-linux/src/lib.rs
index 6298435c3e..f7d13facb7 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/lib.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/lib.rs
@@ -149,7 +149,7 @@ pub fn register_tools_with_cursor_and_provider(
#[cfg(target_os = "linux")]
wayland::ensure_nested_session();
#[cfg(target_os = "linux")]
- wayland::overlay::set_config_enabled(cfg.enabled);
+ wayland::overlay::set_config(cfg.clone());
if cfg.enabled {
overlay::init(cfg.clone());
overlay::run_on_thread();
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs
index 710aa22e01..69262af890 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/overlay.rs
@@ -114,6 +114,48 @@ fn try_send_x11_message(
sender.is_some_and(|tx| tx.try_send(msg).is_ok())
}
+fn should_start_x11_overlay(wayland_display_present: bool) -> bool {
+ !wayland_display_present
+}
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum WaylandOverlayBackend {
+ SemanticShellHelper,
+ LayerShell,
+ LegacyShellHelper,
+ None,
+}
+
+#[cfg(target_os = "linux")]
+static WAYLAND_OVERLAY_BACKEND: OnceLock<WaylandOverlayBackend> = OnceLock::new();
+
+fn select_wayland_overlay_backend(
+ semantic_shell_helper: bool,
+ layer_shell_available: bool,
+ legacy_shell_helper: bool,
+) -> WaylandOverlayBackend {
+ if semantic_shell_helper {
+ WaylandOverlayBackend::SemanticShellHelper
+ } else if layer_shell_available {
+ WaylandOverlayBackend::LayerShell
+ } else if legacy_shell_helper {
+ WaylandOverlayBackend::LegacyShellHelper
+ } else {
+ WaylandOverlayBackend::None
+ }
+}
+
+#[cfg(target_os = "linux")]
+fn wayland_overlay_backend() -> WaylandOverlayBackend {
+ *WAYLAND_OVERLAY_BACKEND.get_or_init(|| {
+ select_wayland_overlay_backend(
+ crate::wayland::shell_helper::semantic_cursor_available(),
+ crate::wayland::overlay::available(),
+ crate::wayland::shell_helper::available(),
+ )
+ })
+}
+
#[cfg(target_os = "linux")]
struct X11OverlayThreadCleanup {
receiver: Option<std::sync::mpsc::Receiver<OverlayMsg>>,
@@ -288,9 +330,9 @@ pub fn send_command_for(key: CursorKey, cmd: OverlayCommand) {
let _ = try_send_command_for(key, cmd);
}
-/// Dispatch to every active Linux overlay backend. The result reports only
+/// Dispatch to exactly one Linux overlay backend. The result reports only
/// whether the X11 owner accepted the command and can fire `ARRIVAL_TX`; the
-/// Wayland layer-shell path does not currently publish arrival notifications.
+/// Wayland backends do not currently publish arrival notifications.
fn try_send_command_for(key: CursorKey, cmd: OverlayCommand) -> bool {
if key.is_empty() {
return false;
@@ -299,75 +341,88 @@ fn try_send_command_for(key: CursorKey, cmd: OverlayCommand) -> bool {
key: key.clone(),
cmd: cmd.clone(),
});
- let x11_queued = try_send_x11_message(CMD_TX.get(), msg.clone());
- if !x11_queued {
+ #[cfg(target_os = "linux")]
+ let native_wayland = crate::wayland::is_wayland();
+ let x11_overlay_allowed =
+ should_start_x11_overlay(std::env::var_os("WAYLAND_DISPLAY").is_some());
+ let x11_queued = x11_overlay_allowed && try_send_x11_message(CMD_TX.get(), msg.clone());
+ if x11_overlay_allowed && !x11_queued {
tracing::warn!(
key = %key,
sender_missing = CMD_TX.get().is_none(),
"overlay: X11 channel rejected command (no sender or queue full)"
);
}
- // Also forward to the native-Wayland layer-shell overlay when Wayland
- // is opted in. The wayland overlay's `forward` is a no-op when its
- // owner thread isn't started yet (which is the normal X11-only case).
#[cfg(target_os = "linux")]
{
- if crate::wayland::is_wayland() {
- if crate::wayland::shell_helper::semantic_cursor_available() {
- crate::wayland::shell_helper::set_cursor_color(&cursor_overlay::session_fill_hex(
- &key,
- ));
- // GNOME has no layer-shell. Drive only the final positioning
- // commands through the compositor helper; it performs its own
- // easing and avoids starting a worker that must fail.
- match &cmd {
- cursor_overlay::OverlayCommand::ClickPulse { x, y } => {
- crate::wayland::shell_helper::click_pulse(*x as i32, *y as i32);
- }
- cursor_overlay::OverlayCommand::MoveTo { x, y, .. } => {
- crate::wayland::shell_helper::move_cursor(*x as i32, *y as i32);
- }
- cursor_overlay::OverlayCommand::SnapTo { x, y, .. } => {
- crate::wayland::shell_helper::move_cursor(*x as i32, *y as i32);
- }
- cursor_overlay::OverlayCommand::BeginAction {
- action,
- delivery,
- target,
- } => {
- crate::wayland::shell_helper::set_cursor_state(
- action.as_str(),
- delivery.as_ref().map_or("", |value| value.as_str()),
- target.as_ref().map_or("", |value| value.as_str()),
- true,
- );
- }
- cursor_overlay::OverlayCommand::EndAction(action) => {
- crate::wayland::shell_helper::set_cursor_state(
- action.as_str(),
- "",
- "",
- false,
- );
- }
- cursor_overlay::OverlayCommand::SetSessionLabel(label) => {
- crate::wayland::shell_helper::set_session_label(
- cursor_overlay::sanitize_session_label(label)
- .as_deref()
- .unwrap_or(""),
- );
- }
- cursor_overlay::OverlayCommand::SetEnabled(false) => {
- crate::wayland::shell_helper::hide_cursor();
- }
- _ => {}
+ if native_wayland {
+ dispatch_wayland_overlay_message(&msg);
+ }
+ }
+ x11_queued
+}
+
+#[cfg(target_os = "linux")]
+fn dispatch_wayland_overlay_message(msg: &OverlayMsg) -> WaylandOverlayBackend {
+ let backend = wayland_overlay_backend();
+ match backend {
+ WaylandOverlayBackend::SemanticShellHelper | WaylandOverlayBackend::LegacyShellHelper => {
+ let semantic = backend == WaylandOverlayBackend::SemanticShellHelper;
+ match msg {
+ OverlayMsg::Cmd(command) => {
+ dispatch_shell_helper_command(&command.key, &command.cmd, semantic);
}
- } else if !crate::wayland::shell_helper::available() {
- let _ = crate::wayland::overlay::forward(&msg);
+ OverlayMsg::Remove(_) => crate::wayland::shell_helper::hide_cursor(),
+ OverlayMsg::Revive(_) => {}
}
}
+ WaylandOverlayBackend::LayerShell if !crate::wayland::overlay::forward(msg) => {
+ return WaylandOverlayBackend::None;
+ }
+ WaylandOverlayBackend::LayerShell | WaylandOverlayBackend::None => {}
+ }
+ backend
+}
+
+#[cfg(target_os = "linux")]
+fn dispatch_shell_helper_command(key: &str, cmd: &OverlayCommand, semantic: bool) {
+ if semantic {
+ crate::wayland::shell_helper::set_cursor_color(&cursor_overlay::session_fill_hex(key));
+ }
+ match cmd {
+ OverlayCommand::ClickPulse { x, y } if semantic => {
+ crate::wayland::shell_helper::click_pulse(*x as i32, *y as i32);
+ }
+ OverlayCommand::MoveTo { x, y, .. }
+ | OverlayCommand::SnapTo { x, y, .. }
+ | OverlayCommand::ClickPulse { x, y } => {
+ crate::wayland::shell_helper::move_cursor(*x as i32, *y as i32);
+ }
+ OverlayCommand::BeginAction {
+ action,
+ delivery,
+ target,
+ } if semantic => {
+ crate::wayland::shell_helper::set_cursor_state(
+ action.as_str(),
+ delivery.as_ref().map_or("", |value| value.as_str()),
+ target.as_ref().map_or("", |value| value.as_str()),
+ true,
+ );
+ }
+ OverlayCommand::EndAction(action) if semantic => {
+ crate::wayland::shell_helper::set_cursor_state(action.as_str(), "", "", false);
+ }
+ OverlayCommand::SetSessionLabel(label) if semantic => {
+ crate::wayland::shell_helper::set_session_label(
+ cursor_overlay::sanitize_session_label(label)
+ .as_deref()
+ .unwrap_or(""),
+ );
+ }
+ OverlayCommand::SetEnabled(false) => crate::wayland::shell_helper::hide_cursor(),
+ _ => {}
}
- x11_queued
}
pub fn is_enabled() -> bool {
@@ -538,14 +593,12 @@ pub fn remove_cursor(key: CursorKey) {
let _ = tx.try_send(msg.clone());
}
#[cfg(target_os = "linux")]
- if crate::wayland::is_wayland() && !crate::wayland::shell_helper::available() {
- let _ = crate::wayland::overlay::forward(&msg);
+ if crate::wayland::is_wayland() {
+ dispatch_wayland_overlay_message(&msg);
}
}
-/// Clear the X11 render-side tombstone after a successful explicit session
-/// revival. Wayland has no keyed tombstone, so forwarding this lifecycle
-/// signal there is an accepted no-op.
+/// Clear the render-side tombstone after a successful explicit session revival.
pub fn revive_cursor(key: CursorKey) {
if key.is_empty() {
return;
@@ -555,8 +608,8 @@ pub fn revive_cursor(key: CursorKey) {
let _ = tx.try_send(msg.clone());
}
#[cfg(target_os = "linux")]
- if crate::wayland::is_wayland() && !crate::wayland::shell_helper::available() {
- let _ = crate::wayland::overlay::forward(&msg);
+ if crate::wayland::is_wayland() {
+ dispatch_wayland_overlay_message(&msg);
}
}
@@ -579,6 +632,17 @@ pub fn run_on_thread() {
return;
}
+ // A Wayland session normally also exposes DISPLAY through XWayland, but
+ // that does not make the legacy full-root X11 overlay safe. This decision
+ // must be independent of the experimental native-Wayland feature opt-in:
+ // without that opt-in there is no layer-shell fallback, but showing no
+ // overlay is preferable to mapping an opaque black X11 root window over
+ // the Wayland desktop. With the opt-in enabled, commands are forwarded to
+ // the native layer-shell backend below.
+ if !should_start_x11_overlay(std::env::var_os("WAYLAND_DISPLAY").is_some()) {
+ return;
+ }
+
// Wayland layer-shell overlay is started LAZILY on the first
// send_command_for() that targets a Wayland session (see the
// wayland::overlay::forward() path). Starting it eagerly here added
@@ -2710,6 +2774,32 @@ fn bgra_and_visible_shape(
mod tests {
use super::*;
+ #[test]
+ fn wayland_display_does_not_start_legacy_x11_overlay() {
+ assert!(!should_start_x11_overlay(true));
+ assert!(should_start_x11_overlay(false));
+ }
+
+ #[test]
+ fn wayland_overlay_backend_is_selected_once_with_legacy_fallback() {
+ assert_eq!(
+ select_wayland_overlay_backend(true, true, true),
+ WaylandOverlayBackend::SemanticShellHelper
+ );
+ assert_eq!(
+ select_wayland_overlay_backend(false, true, true),
+ WaylandOverlayBackend::LayerShell
+ );
+ assert_eq!(
+ select_wayland_overlay_backend(false, false, true),
+ WaylandOverlayBackend::LegacyShellHelper
+ );
+ assert_eq!(
+ select_wayland_overlay_backend(false, false, false),
+ WaylandOverlayBackend::None
+ );
+ }
+
fn drain_x11_test_events(conn: &impl x11rb::connection::Connection) -> anyhow::Result<()> {
while conn.poll_for_event()?.is_some() {}
Ok(())
@@ -3155,6 +3245,27 @@ mod tests {
let _: fn(CursorKey, OverlayCommand) = send_command_for;
}
+ #[test]
+ fn session_removal_drops_the_cursor_and_rejects_late_commands() {
+ let mut map = default_render_map();
+ let command = || {
+ OverlayMsg::Cmd(KeyedOverlayCommand {
+ key: "session-a".to_owned(),
+ cmd: OverlayCommand::ClickPulse { x: 12.0, y: 34.0 },
+ })
+ };
+
+ assert_eq!(apply_msg(&mut map, command()).as_deref(), Some("session-a"));
+ assert!(map.cursors.contains_key("session-a"));
+
+ assert!(apply_msg(&mut map, OverlayMsg::Remove("session-a".to_owned())).is_none());
+ assert!(!map.cursors.contains_key("session-a"));
+ assert!(map.ended.contains("session-a"));
+
+ assert!(apply_msg(&mut map, command()).is_none());
+ assert!(!map.cursors.contains_key("session-a"));
+ }
+
#[test]
fn failed_x11_enqueue_is_reported_without_waiting() {
assert!(!try_send_x11_message(None, test_message()));
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index e71a9c8f34..e9b0bfdff9 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -1938,33 +1938,34 @@ fn resolve_cursor_key(args: &Value) -> String {
"default".to_owned()
}
-#[cfg(test)]
-mod cursor_key_resolution_tests {
- use super::resolve_cursor_key;
- use serde_json::json;
+/// Return the cursor key only for a lifecycle-owned session. Cursor positioning
+/// for keyboard actions deliberately does not opt anonymous calls or the
+/// legacy cursor_id-only path into session semantics. The proxy-minted
+/// `_session_id` is trusted lifecycle state and must behave like a public named
+/// session for cursor ownership.
+fn named_session_cursor_key(args: &Value) -> Option<String> {
+ ["session", "_session_id"].into_iter().find_map(|key| {
+ args.get(key)
+ .and_then(Value::as_str)
+ .filter(|session| !session.is_empty())
+ .map(str::to_owned)
+ })
+}
- #[test]
- fn trusted_implicit_session_owns_the_cursor() {
- assert_eq!(resolve_cursor_key(&json!({})), "default");
- assert_eq!(
- resolve_cursor_key(&json!({"_session_id": "implicit-lease"})),
- "implicit-lease"
- );
- assert_eq!(
- resolve_cursor_key(&json!({
- "_session_id": "implicit-lease",
- "cursor_id": "legacy"
- })),
- "implicit-lease"
- );
- assert_eq!(
- resolve_cursor_key(&json!({
- "session": "named",
- "_session_id": "implicit-lease"
- })),
- "named"
- );
- }
+fn finite_cursor_point(point: Option<(f64, f64)>) -> Option<(f64, f64)> {
+ point.filter(|(x, y)| x.is_finite() && y.is_finite())
+}
+
+fn choose_keyboard_cursor_target(
+ explicit: Option<(f64, f64)>,
+ remembered: Option<(f64, f64)>,
+ window_center: Option<(f64, f64)>,
+ current_pointer: Option<(f64, f64)>,
+) -> Option<(f64, f64)> {
+ finite_cursor_point(explicit)
+ .or_else(|| finite_cursor_point(remembered))
+ .or_else(|| finite_cursor_point(window_center))
+ .or_else(|| finite_cursor_point(current_pointer))
}
fn mouse_hold_json(cursor_id: &str, hold: Option<&MouseHoldState>) -> Value {
@@ -2044,16 +2045,18 @@ fn overlay_move_to_for(cursor_id: &str, sx: f64, sy: f64, heading: Option<f64>)
}
async fn overlay_glide_to_for(cursor_id: &str, sx: f64, sy: f64) {
- if !crate::overlay::is_enabled_for(cursor_id) {
- return;
- }
- // Wayland (Mutter/KDE, no layer-shell): glide the agent cursor via the
- // WinRects shell extension. It eases to the target itself, so send the
- // destination once here rather than the interpolated stream the X11 render
- // loop uses (which is invisible on those compositors anyway). No-op if the
- // extension isn't installed.
+ // Input always revives its agent cursor. Hiding it is useful while idle,
+ // but a hidden cursor must not make pointer or keyboard control invisible.
+ crate::overlay::send_command_for(
+ cursor_id.to_owned(),
+ cursor_overlay::OverlayCommand::SetEnabled(true),
+ );
+ // Every Wayland backend owns its animation loop. Send one destination and
+ // let the Linux overlay layer select the semantic helper, layer-shell, or
+ // older helper fallback without an interpolated stream fighting it.
if crate::wayland::is_wayland() {
- crate::wayland::shell_helper::move_cursor(sx as i32, sy as i32);
+ overlay_move_to_for(cursor_id, sx, sy, None);
+ return;
}
let pos = crate::overlay::current_position_for(cursor_id);
if pos.0 < 0.0 && pos.1 < 0.0 {
@@ -2066,6 +2069,146 @@ async fn overlay_glide_to_for(cursor_id: &str, sx: f64, sy: f64) {
crate::overlay::animate_cursor_to_for(cursor_id.to_owned(), sx, sy).await;
}
+/// Keep the logical cursor position in sync with every visibly targeted
+/// pointer action. Overlay delivery is intentionally best-effort: registry
+/// state is still updated when no renderer is running or its queue is closed.
+async fn reveal_pointer_action_for(
+ state: &ToolState,
+ cursor_id: &str,
+ sx: f64,
+ sy: f64,
+ click_pulse: bool,
+) {
+ if !sx.is_finite() || !sy.is_finite() {
+ return;
+ }
+ state.cursor_registry.set_enabled(cursor_id, true);
+ state.cursor_registry.update_position(cursor_id, sx, sy);
+ overlay_glide_to_for(cursor_id, sx, sy).await;
+ if click_pulse {
+ crate::overlay::send_command_for(
+ cursor_id.to_owned(),
+ cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
+ );
+ }
+}
+
+fn keyboard_window_center(xid: u64) -> Option<(f64, f64)> {
+ if xid == 0 {
+ return None;
+ }
+ if crate::wayland::is_wayland() {
+ return crate::wayland::window_geometry(xid).and_then(|(x, y, width, height)| {
+ (width > 0 && height > 0).then_some((
+ f64::from(x) + f64::from(width) / 2.0,
+ f64::from(y) + f64::from(height) / 2.0,
+ ))
+ });
+ }
+ window_screen_center(xid)
+ .ok()
+ .map(|(x, y)| (f64::from(x), f64::from(y)))
+}
+
+fn current_pointer_position() -> Option<(f64, f64)> {
+ if crate::wayland::is_wayland() {
+ return crate::wayland::last_synth_cursor_pos().map(|(x, y)| (f64::from(x), f64::from(y)));
+ }
+
+ use x11rb::connection::Connection;
+ use x11rb::protocol::xproto::ConnectionExt as _;
+ use x11rb::rust_connection::RustConnection;
+
+ let (connection, screen_num) = RustConnection::connect(None).ok()?;
+ let root = connection.setup().roots[screen_num].root;
+ let reply = connection.query_pointer(root).ok()?.reply().ok()?;
+ Some((f64::from(reply.root_x), f64::from(reply.root_y)))
+}
+
+fn explicit_keyboard_cursor_target(
+ pid: u32,
+ xid: u64,
+ element_index: Option<usize>,
+ pixel_target: Option<(f64, f64)>,
+) -> Option<(f64, f64)> {
+ if let Some(element_index) = element_index {
+ let (sx, sy) = element_screen_center(pid, element_index).ok()?;
+ return Some((sx, sy));
+ }
+
+ let (x, y) = pixel_target?;
+ if crate::wayland::wayland_input_enabled() {
+ return crate::wayland::window_geometry(xid)
+ .map(|(wx, wy, _, _)| (f64::from(wx) + x.round(), f64::from(wy) + y.round()));
+ }
+ window_local_to_screen(xid, x, y).ok()
+}
+
+/// Position and reveal a named session's cursor before keyboard/value input.
+/// `preserve_legacy_element_visual` retains the existing element-only feedback
+/// for type_text/set_value anonymous calls without adding session-style fallback
+/// placement to them. Geometry and overlay failures are observational and never
+/// affect the tool's actual input result.
+async fn position_named_session_keyboard_cursor(
+ state: &ToolState,
+ args: &Value,
+ pid: u32,
+ xid: u64,
+ element_index: Option<usize>,
+ pixel_target: Option<(f64, f64)>,
+ preserve_legacy_element_visual: bool,
+) {
+ let named_cursor_id = named_session_cursor_key(args);
+ let cursor_id = match named_cursor_id {
+ Some(ref cursor_id) => cursor_id.clone(),
+ None if preserve_legacy_element_visual && element_index.is_some() => {
+ resolve_cursor_key(args)
+ }
+ None => return,
+ };
+
+ let remembered = named_cursor_id.as_ref().and_then(|_| {
+ state
+ .cursor_registry
+ .get(&cursor_id)
+ .and_then(|cursor| cursor.x.zip(cursor.y))
+ });
+ let explicit = tokio::task::spawn_blocking(move || {
+ explicit_keyboard_cursor_target(pid, xid, element_index, pixel_target)
+ })
+ .await
+ .ok()
+ .flatten();
+
+ let fallback = if named_cursor_id.is_some()
+ && explicit.is_none()
+ && finite_cursor_point(remembered).is_none()
+ {
+ tokio::task::spawn_blocking(move || {
+ let center = keyboard_window_center(xid);
+ let pointer = center.is_none().then(current_pointer_position).flatten();
+ (center, pointer)
+ })
+ .await
+ .unwrap_or((None, None))
+ } else {
+ (None, None)
+ };
+
+ let Some((sx, sy)) =
+ choose_keyboard_cursor_target(explicit, remembered, fallback.0, fallback.1)
+ else {
+ return;
+ };
+ if xid != 0 {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ }
+ reveal_pointer_action_for(state, &cursor_id, sx, sy, false).await;
+}
+
async fn track_overlay_drag_for(
cursor_id: String,
from: (f64, f64),
@@ -2073,9 +2216,10 @@ async fn track_overlay_drag_for(
duration_ms: u64,
steps: usize,
) {
- if !crate::overlay::is_enabled_for(&cursor_id) {
- return;
- }
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::SetEnabled(true),
+ );
crate::overlay::send_command_for(
cursor_id.clone(),
cursor_overlay::OverlayCommand::SetPressed(true),
@@ -2317,7 +2461,8 @@ impl Tool for ClickTool {
// / Windows desktop paths already do this). Without it the overlay
// sits idle elsewhere while only the real pointer warps, so a viewer
// sees the cursor "click somewhere else."
- overlay_glide_to_for(&cursor_id, sx as f64, sy as f64).await;
+ reveal_pointer_action_for(&self.state, &cursor_id, f64::from(sx), f64::from(sy), true)
+ .await;
let r = tokio::task::spawn_blocking(move || {
if crate::wayland::wayland_input_enabled() {
if !modifiers.is_empty() {
@@ -2431,11 +2576,7 @@ impl Tool for ClickTool {
cursor_overlay::OverlayCommand::PinAbove(xid),
);
}
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
// Chromium can execute a genuine AT-SPI action without focus. Try
// that route before applying its background synthetic-input gate.
@@ -2573,11 +2714,7 @@ impl Tool for ClickTool {
.and_then(|r| r.ok())
};
if let Some((sx, sy)) = glide_target {
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
}
let (xi, yi) = (x as i32, y as i32);
@@ -2837,6 +2974,8 @@ impl Tool for TypeTextTool {
let text =
cua_driver_core::text_sanitize::strip_trailing_agent_protocol_tags(&input.text)
.into_owned();
+ position_named_session_keyboard_cursor(&self.state, &args, 0, 0, None, None, false)
+ .await;
let wayland = crate::wayland::wayland_input_enabled();
let path = if wayland { "wayland_focused" } else { "xtest" };
let result = tokio::task::spawn_blocking(move || {
@@ -2928,21 +3067,16 @@ impl Tool for TypeTextTool {
);
}
- let cursor_id = resolve_cursor_key(&args);
- if let Some(idx) = resolved_elem_idx {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(xid),
- );
- if let Ok(Ok((screen_x, screen_y))) =
- tokio::task::spawn_blocking(move || element_screen_center(pid, idx)).await
- {
- overlay_glide_to_for(&cursor_id, screen_x, screen_y).await;
- self.state
- .cursor_registry
- .update_position(&cursor_id, screen_x, screen_y);
- }
- }
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ resolved_elem_idx,
+ px.zip(py),
+ true,
+ )
+ .await;
let text_len = text.chars().count();
// Native toolkit editables have a stronger focus-free route than raw
@@ -3454,6 +3588,8 @@ impl Tool for PressKeyTool {
let key = input.key;
let display = key.clone();
let modifiers = input.modifiers.unwrap_or_default();
+ position_named_session_keyboard_cursor(&self.state, &args, 0, 0, None, None, false)
+ .await;
let wayland = crate::wayland::wayland_input_enabled();
let path = if wayland { "wayland_focused" } else { "xtest" };
let result = tokio::task::spawn_blocking(move || {
@@ -3503,18 +3639,18 @@ impl Tool for PressKeyTool {
Ok(r) => r,
Err(e) => return e,
};
+ let resolved_element_index = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
+ Some(*element_index)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => None,
+ };
let xid_opt = match &resolved {
cua_driver_core::element_token::ResolvedElement::Element { window_id, .. } => {
window_id_arg.or_else(|| window_id.map(|v| v as u64))
}
cua_driver_core::element_token::ResolvedElement::None => window_id_arg,
};
- let resolved_element_idx = match &resolved {
- cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
- Some(*element_index)
- }
- cua_driver_core::element_token::ResolvedElement::None => None,
- };
let xid = match xid_opt {
Some(x) => x,
None => {
@@ -3557,17 +3693,28 @@ impl Tool for PressKeyTool {
if px.is_some() != py.is_some() {
return ToolResult::error("Pass both x and y to press_key, or neither.");
}
- if px.is_some() && resolved_element_idx.is_some() {
+ if px.is_some() && resolved_element_index.is_some() {
return ToolResult::error(
"Pass either element_index (ax) or x,y (px) to press_key, not both.",
);
}
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ resolved_element_index,
+ px.zip(py),
+ false,
+ )
+ .await;
+
// Nested cua-compositor addresses the owning Wayland client directly.
// Preserve legacy modifiers by promoting the request to a chord.
if crate::wayland::is_inject_mode() {
if let Err(error) =
- focus_nested_inject_target(pid, xid, resolved_element_idx, px.zip(py)).await
+ focus_nested_inject_target(pid, xid, resolved_element_index, px.zip(py)).await
{
return error;
}
@@ -3625,7 +3772,7 @@ impl Tool for PressKeyTool {
if crate::wayland::wayland_input_enabled() {
let key_w = key.clone();
let chord = press_key_chord(&mods, &key);
- let idx = resolved_element_idx;
+ let idx = resolved_element_index;
let result = tokio::task::spawn_blocking(move || {
crate::wayland::with_target_foreground(pid, xid, || {
if let Some(idx) = idx {
@@ -3657,7 +3804,7 @@ impl Tool for PressKeyTool {
// click restores the prior top-level before returning.
let deliver_fg = delivery.is_foreground();
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
- if resolved_element_idx.is_none()
+ if resolved_element_index.is_none()
&& mods.is_empty()
&& key_for_task.eq_ignore_ascii_case("enter")
{
@@ -3673,7 +3820,7 @@ impl Tool for PressKeyTool {
// XSendEvent (no focus steal) for apps that accept it.
if deliver_fg {
return crate::input::with_x11_foreground(xid, 80, || {
- if let Some(element_index) = resolved_element_idx {
+ if let Some(element_index) = resolved_element_index {
if !crate::atspi::focus_element(pid, element_index)? {
anyhow::bail!(
"AT-SPI Component.GrabFocus returned false for element {element_index}"
@@ -3683,7 +3830,7 @@ impl Tool for PressKeyTool {
crate::input::send_key_xtest(&key_for_task, &m)
});
}
- if let Some(element_index) = resolved_element_idx {
+ if let Some(element_index) = resolved_element_index {
if !crate::atspi::focus_element(pid, element_index)? {
anyhow::bail!(
"AT-SPI Component.GrabFocus returned false for element {element_index}"
@@ -3784,6 +3931,8 @@ impl Tool for HotkeyTool {
return ToolResult::error("keys must include at least one non-modifier key.");
};
let display = keys.join("+");
+ position_named_session_keyboard_cursor(&self.state, &args, 0, 0, None, None, false)
+ .await;
let wayland = crate::wayland::wayland_input_enabled();
let path = if wayland { "wayland_focused" } else { "xtest" };
let result = tokio::task::spawn_blocking(move || {
@@ -3900,6 +4049,17 @@ impl Tool for HotkeyTool {
);
}
+ position_named_session_keyboard_cursor(
+ &self.state,
+ &args,
+ pid,
+ xid,
+ resolved_element_index,
+ px.zip(py),
+ false,
+ )
+ .await;
+
if crate::wayland::is_inject_mode() {
if let Err(error) =
focus_nested_inject_target(pid, xid, resolved_element_index, px.zip(py)).await
@@ -4044,7 +4204,9 @@ impl Tool for HotkeyTool {
// ── set_value ─────────────────────────────────────────────────────────────────
-pub struct SetValueTool;
+pub struct SetValueTool {
+ state: Arc<ToolState>,
+}
static SV_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
#[async_trait]
@@ -4090,32 +4252,23 @@ impl Tool for SetValueTool {
Ok(r) => r,
Err(e) => return e,
};
- let idx = match resolved {
- cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
- element_index
- }
+ let (idx, resolved_window_id) = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element {
+ element_index,
+ window_id,
+ ..
+ } => (*element_index, window_id.map(u64::from)),
cua_driver_core::element_token::ResolvedElement::None => return ToolResult::error(
"set_value requires element_index or element_token to address the target element.",
),
};
- let cursor_id = resolve_cursor_key(&args);
let value_for_task = value.clone();
- // Pulse the agent cursor onto the target element before writing, so a
- // value write gets the same visual feedback as a click — the viewer can
- // see *where* the agent is acting. No-op when the element bounds can't
- // be resolved or the overlay is disabled.
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || element_screen_center(pid, idx)).await
- {
- let window_id = args.u64_or("window_id", 0);
- if window_id != 0 {
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::PinAbove(window_id),
- );
- }
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- }
+ let xid = args
+ .opt_u64("window_id")
+ .or(resolved_window_id)
+ .unwrap_or(0);
+ position_named_session_keyboard_cursor(&self.state, &args, pid, xid, Some(idx), None, true)
+ .await;
let result =
tokio::task::spawn_blocking(move || crate::atspi::set_value(pid, idx, &value_for_task))
.await;
@@ -4129,7 +4282,9 @@ impl Tool for SetValueTool {
// ── scroll ────────────────────────────────────────────────────────────────────
-pub struct ScrollTool;
+pub struct ScrollTool {
+ state: Arc<ToolState>,
+}
static SCROLL_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
#[async_trait]
@@ -4181,6 +4336,16 @@ impl Tool for ScrollTool {
let display = direction.clone();
let wayland = crate::wayland::wayland_input_enabled();
let path = if wayland { "wayland_desktop" } else { "xtest" };
+ if named_session_cursor_key(&args).is_some() {
+ reveal_pointer_action_for(
+ &self.state,
+ &cursor_id,
+ f64::from(x),
+ f64::from(y),
+ false,
+ )
+ .await;
+ }
let result = tokio::task::spawn_blocking(move || {
if wayland {
crate::wayland::scroll_desktop(x, y, &direction, amount as u32)
@@ -4249,6 +4414,49 @@ impl Tool for ScrollTool {
}
};
+ let pixel_target = match (
+ args.get("x").and_then(|value| value.as_f64()),
+ args.get("y").and_then(|value| value.as_f64()),
+ ) {
+ (Some(x), Some(y)) => {
+ // Pixel targets use the latest screenshot's coordinate frame.
+ // Apply the same buffer-to-window ratio as click/drag before
+ // positioning either the agent cursor or the input device.
+ let ratio = self.state.resize_registry.ratio(pid).unwrap_or(1.0);
+ Some((x * ratio, y * ratio))
+ }
+ (None, None) => None,
+ _ => return ToolResult::error("Pass both x and y to pixel-target scroll."),
+ };
+ let resolved_element_index = match &resolved {
+ cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
+ Some(*element_index)
+ }
+ cua_driver_core::element_token::ResolvedElement::None => None,
+ };
+ if pixel_target.is_some() && resolved_element_index.is_some() {
+ return ToolResult::error(
+ "Pass either element_index (ax) or x,y (px) to scroll, not both.",
+ );
+ }
+
+ if named_session_cursor_key(&args).is_some() {
+ let visual_target = tokio::task::spawn_blocking(move || {
+ explicit_keyboard_cursor_target(pid, xid, resolved_element_index, pixel_target)
+ .or_else(|| keyboard_window_center(xid))
+ })
+ .await
+ .ok()
+ .flatten();
+ if let Some((sx, sy)) = visual_target {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, false).await;
+ }
+ }
+
let delivery = crate::input::delivery::DeliveryMode::from_args(&args);
if let Some(refusal) = unavailable_chromium_background(pid, delivery) {
return refusal;
@@ -4285,25 +4493,6 @@ impl Tool for ScrollTool {
}
}
- let pixel_target = match (
- args.get("x").and_then(|value| value.as_f64()),
- args.get("y").and_then(|value| value.as_f64()),
- ) {
- (Some(x), Some(y)) => Some((x, y)),
- (None, None) => None,
- _ => return ToolResult::error("Pass both x and y to pixel-target scroll."),
- };
- if pixel_target.is_some()
- && matches!(
- &resolved,
- cua_driver_core::element_token::ResolvedElement::Element { .. }
- )
- {
- return ToolResult::error(
- "Pass either element_index (ax) or x,y (px) to scroll, not both.",
- );
- }
-
if crate::wayland::is_inject_mode() {
let Some((x, y)) = pixel_target else {
return crate::input::delivery::background_unavailable_error(
@@ -4612,11 +4801,7 @@ impl Tool for DoubleClickTool {
cursor_id.clone(),
cursor_overlay::OverlayCommand::PinAbove(xid),
);
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
}
let lxi = lx as i32;
let lyi = ly as i32;
@@ -4703,11 +4888,7 @@ impl Tool for DoubleClickTool {
.and_then(|r| r.ok())
};
if let Some((sx, sy)) = glide_target {
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
}
let (xi, yi) = (x as i32, y as i32);
let cursor_id_for_task = cursor_id.clone();
@@ -4846,11 +5027,7 @@ impl Tool for RightClickTool {
cursor_id.clone(),
cursor_overlay::OverlayCommand::PinAbove(xid),
);
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
}
let lxi = lx as i32;
let lyi = ly as i32;
@@ -4937,11 +5114,7 @@ impl Tool for RightClickTool {
.and_then(|r| r.ok())
};
if let Some((sx, sy)) = glide_target {
- overlay_glide_to_for(&cursor_id, sx, sy).await;
- crate::overlay::send_command_for(
- cursor_id.clone(),
- cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy },
- );
+ reveal_pointer_action_for(&self.state, &cursor_id, sx, sy, true).await;
}
let (xi, yi) = (x as i32, y as i32);
let cursor_id_for_task = cursor_id.clone();
@@ -6316,6 +6489,46 @@ fn x11_screen_size() -> anyhow::Result<(u32, u32)> {
Ok((w, h))
}
+/// Put the desktop image in the exact coordinate frame consumed by desktop
+/// actions. Native Wayland capture buffers may use backing pixels while the
+/// compositor's pointer protocol and reported screen geometry use logical
+/// pixels. Returning the backing image alongside logical dimensions violates
+/// the screenshot-to-action contract and makes every vision-grounded action
+/// miss by the output scale.
+fn normalize_desktop_capture_for_action_frame(
+ png: Vec<u8>,
+ action_width: u32,
+ action_height: u32,
+) -> anyhow::Result<(Vec<u8>, u32, u32, f64)> {
+ if action_width == 0 || action_height == 0 {
+ anyhow::bail!("desktop action frame is empty: {action_width}x{action_height}");
+ }
+
+ let (capture_width, capture_height) = crate::capture::png_dimensions_pub(&png)?;
+ let scale_x = f64::from(capture_width) / f64::from(action_width);
+ let scale_y = f64::from(capture_height) / f64::from(action_height);
+ if (scale_x - scale_y).abs() > 0.01 {
+ anyhow::bail!(
+ "desktop capture {capture_width}x{capture_height} cannot be mapped uniformly to \
+ action frame {action_width}x{action_height} (scale {scale_x:.4}x{scale_y:.4})"
+ );
+ }
+
+ if capture_width == action_width && capture_height == action_height {
+ return Ok((png, action_width, action_height, 1.0));
+ }
+
+ let image = image::load_from_memory_with_format(&png, image::ImageFormat::Png)?;
+ let resized = image.resize_exact(
+ action_width,
+ action_height,
+ image::imageops::FilterType::Lanczos3,
+ );
+ let mut encoded = std::io::Cursor::new(Vec::new());
+ resized.write_to(&mut encoded, image::ImageFormat::Png)?;
+ Ok((encoded.into_inner(), action_width, action_height, scale_x))
+}
+
// ── get_desktop_state ─────────────────────────────────────────────────────────
pub struct GetDesktopStateTool;
@@ -6326,8 +6539,8 @@ impl Tool for GetDesktopStateTool {
fn def(&self) -> &ToolDef {
GDS_DEF.get_or_init(|| ToolDef {
name: "get_desktop_state".into(),
- description: "Capture the full display in true screen pixels with no downscale. \
- Use its native-size PNG as the coordinate source for actions whose target is \
+ description: "Capture the full display in the desktop action coordinate frame. \
+ Use the returned PNG directly as the coordinate source for actions whose target is \
{kind:\"desktop\",display_id:\"primary\"}. No AT-SPI walk.".into(),
input_schema: json!({"type":"object","properties":{
"session":{"type":"string","description":"For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session."},
@@ -6345,10 +6558,11 @@ impl Tool for GetDesktopStateTool {
let out_file = input.screenshot_out_file;
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
- // Vision-only: capture the FULL DISPLAY at native size. No downscale
- // so screen-absolute pixels land exactly.
- let png = crate::capture::screenshot_display_bytes()?;
- let (shot_w, shot_h) = crate::capture::png_dimensions_pub(&png)?;
+ // Capture the full display at native size first. When the
+ // compositor consumes logical input coordinates, normalize the
+ // image below so screenshot pixels still land exactly.
+ let native_png = crate::capture::screenshot_display_bytes()?;
+ let (native_w, native_h) = crate::capture::png_dimensions_pub(&native_png)?;
// True screen size. On a pure-Wayland session (native backend
// opted in, no X11 DISPLAY) the capture above came from the
// wlroots `zwlr_screencopy` cascade, whose full-display buffer is
@@ -6359,10 +6573,12 @@ impl Tool for GetDesktopStateTool {
// Only fall back to the X11 root-window geometry off Wayland, so
// the X11 / XWayland path is unchanged. See #2017 / Sway testing.
let (screen_w, screen_h) = if crate::wayland::is_wayland() {
- (shot_w, shot_h)
+ (native_w, native_h)
} else {
x11_screen_size()?
};
+ let (png, shot_w, shot_h, scale_factor) =
+ normalize_desktop_capture_for_action_frame(native_png, screen_w, screen_h)?;
// Optional: write PNG to disk instead of returning base64.
let written = if let Some(path) = out_file.as_deref() {
std::fs::write(path, &png)?;
@@ -6376,12 +6592,20 @@ impl Tool for GetDesktopStateTool {
} else {
Some(B64.encode(&png))
};
- Ok((b64, shot_w, shot_h, screen_w, screen_h, written))
+ Ok((
+ b64,
+ shot_w,
+ shot_h,
+ screen_w,
+ screen_h,
+ scale_factor,
+ written,
+ ))
})
.await;
match result {
- Ok(Ok((b64_opt, shot_w, shot_h, screen_w, screen_h, written))) => {
+ Ok(Ok((b64_opt, shot_w, shot_h, screen_w, screen_h, scale_factor, written))) => {
let mut content = Vec::new();
let mut structured = json!({
"platform": "linux",
@@ -6390,7 +6614,7 @@ impl Tool for GetDesktopStateTool {
"screenshot_height": shot_h,
"screen_width": screen_w,
"screen_height": screen_h,
- "scale_factor": 1.0,
+ "scale_factor": scale_factor,
"screenshot_mime_type": "image/png",
});
if let Some(b64) = b64_opt {
@@ -6490,12 +6714,25 @@ pub struct MoveCursorTool {
static MCURSOR_DEF: std::sync::OnceLock<ToolDef> = std::sync::OnceLock::new();
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum CursorControlScope {
+ Agent,
+ Desktop,
+}
+
+fn cursor_control_scope(args: &Value) -> CursorControlScope {
+ match args.get("scope").and_then(Value::as_str) {
+ Some("desktop") => CursorControlScope::Desktop,
+ _ => CursorControlScope::Agent,
+ }
+}
+
#[async_trait]
impl Tool for MoveCursorTool {
fn def(&self) -> &ToolDef {
MCURSOR_DEF.get_or_init(|| ToolDef {
name: "move_cursor".into(),
- description: "Move the agent cursor overlay, or with scope=desktop move the real OS pointer in get_desktop_state coordinates.".into(),
+ description: "Move the synthetic agent cursor without changing the user's pointer. Only an explicit scope=desktop request moves the real OS pointer in get_desktop_state coordinates.".into(),
input_schema: json!({"type":"object","required":["x","y"],"properties":{
"x":{"type":"number"},"y":{"type":"number"},"session": cua_driver_core::tool_schema::session_schema(),"cursor_id":{"type":"string"},"scope":{"type":"string","enum":["window","desktop"],"default":"window"}
},"additionalProperties":false}),
@@ -6504,7 +6741,7 @@ impl Tool for MoveCursorTool {
}
async fn invoke(&self, args: Value) -> ToolResult {
use cua_driver_core::tool_args::ArgsExt;
- if args.opt_str("scope").as_deref() == Some("desktop") {
+ if cursor_control_scope(&args) == CursorControlScope::Desktop {
let input = match parse_typed_projection::<MoveCursorInput>("move_cursor", &args) {
Ok(input) => input,
Err(result) => return result,
@@ -6540,35 +6777,15 @@ impl Tool for MoveCursorTool {
}
let x = args.f64_or("x", 0.0);
let y = args.f64_or("y", 0.0);
- let window_id = args.get("window_id").and_then(|v| v.as_u64());
let cursor_id = resolve_cursor_key(&args);
- self.state.cursor_registry.update_position(&cursor_id, x, y);
// End pointing upper-left (45°) — matches Swift's
// `AgentCursor.animateAndWait(endAngleDegrees: 45)` convention so the
// overlay arrow settles to the natural macOS-style pose.
// Use the acknowledged animation path so a first-ever move seeds and
// displays the session cursor just as reliably as a coordinate click.
- crate::overlay::animate_cursor_to_for(cursor_id.clone(), x, y).await;
- // Native Wayland: also warp the real cursor via zwlr_virtual_pointer.
- // Off-thread because the wayland-client roundtrip is blocking. Best-effort
- // — overlay update + registry write already succeeded; surface a warning
- // only if the warp itself failed.
- let real_warp_note = if crate::wayland::wayland_input_enabled() {
- let xi = x.round() as i32;
- let yi = y.round() as i32;
- match tokio::task::spawn_blocking(move || {
- crate::wayland::move_cursor_absolute(window_id, xi, yi)
- })
- .await
- {
- Ok(Ok(())) => " (real cursor warped via virtual-pointer)",
- Ok(Err(_)) | Err(_) => " (overlay updated; real-cursor warp failed)",
- }
- } else {
- ""
- };
+ reveal_pointer_action_for(&self.state, &cursor_id, x, y, false).await;
ToolResult::text(format!(
- "Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1}).{real_warp_note}"
+ "Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1}); the user pointer was unchanged."
))
}
}
@@ -7984,8 +8201,18 @@ pub fn build_registry_with_provider(
},
&pid_window_candidates,
));
- r.register(pid_window_guarded(SetValueTool, &pid_window_candidates));
- r.register(pid_window_guarded(ScrollTool, &pid_window_candidates));
+ r.register(pid_window_guarded(
+ SetValueTool {
+ state: state.clone(),
+ },
+ &pid_window_candidates,
+ ));
+ r.register(pid_window_guarded(
+ ScrollTool {
+ state: state.clone(),
+ },
+ &pid_window_candidates,
+ ));
cua_driver_core::clipboard::register_clipboard_tools(
&mut r,
Arc::new(crate::clipboard::LinuxClipboard::new()),
@@ -8155,3 +8382,120 @@ mod pid_window_target_tests {
));
}
}
+
+#[cfg(test)]
+mod session_cursor_target_tests {
+ use super::{
+ choose_keyboard_cursor_target, cursor_control_scope, named_session_cursor_key,
+ reveal_pointer_action_for, CursorControlScope, ToolState,
+ };
+ use serde_json::json;
+
+ #[test]
+ fn lifecycle_owned_sessions_opt_into_keyboard_cursor_positioning() {
+ assert_eq!(
+ named_session_cursor_key(&json!({"session": "editing-run"})).as_deref(),
+ Some("editing-run")
+ );
+ assert_eq!(
+ named_session_cursor_key(&json!({"cursor_id": "legacy"})),
+ None
+ );
+ assert_eq!(
+ named_session_cursor_key(&json!({"_session_id": "implicit"})).as_deref(),
+ Some("implicit")
+ );
+ assert_eq!(named_session_cursor_key(&json!({})), None);
+ }
+
+ #[test]
+ fn keyboard_cursor_uses_explicit_then_remembered_then_safe_seed() {
+ let explicit = Some((10.0, 20.0));
+ let remembered = Some((30.0, 40.0));
+ let window_center = Some((50.0, 60.0));
+ let current_pointer = Some((70.0, 80.0));
+
+ assert_eq!(
+ choose_keyboard_cursor_target(explicit, remembered, window_center, current_pointer),
+ explicit
+ );
+ assert_eq!(
+ choose_keyboard_cursor_target(None, remembered, window_center, current_pointer),
+ remembered
+ );
+ assert_eq!(
+ choose_keyboard_cursor_target(None, None, window_center, current_pointer),
+ window_center
+ );
+ assert_eq!(
+ choose_keyboard_cursor_target(None, None, None, current_pointer),
+ current_pointer
+ );
+ }
+
+ #[test]
+ fn invalid_coordinates_do_not_poison_session_position_reuse() {
+ assert_eq!(
+ choose_keyboard_cursor_target(Some((f64::NAN, 1.0)), Some((12.0, 34.0)), None, None,),
+ Some((12.0, 34.0))
+ );
+ }
+
+ #[test]
+ fn real_pointer_control_requires_explicit_desktop_scope() {
+ assert_eq!(cursor_control_scope(&json!({})), CursorControlScope::Agent);
+ assert_eq!(
+ cursor_control_scope(&json!({"scope": "window"})),
+ CursorControlScope::Agent
+ );
+ assert_eq!(
+ cursor_control_scope(&json!({"scope": "desktop"})),
+ CursorControlScope::Desktop
+ );
+ }
+
+ #[tokio::test]
+ async fn pointer_position_survives_an_unavailable_overlay() {
+ let state = ToolState::new();
+ reveal_pointer_action_for(&state, "no-renderer", 123.0, 456.0, true).await;
+
+ let cursor = state
+ .cursor_registry
+ .get("no-renderer")
+ .expect("pointer action records its position independently of rendering");
+ assert!(cursor.config.enabled, "input must revive its agent cursor");
+ assert_eq!(cursor.x.zip(cursor.y), Some((123.0, 456.0)));
+ }
+}
+
+#[cfg(test)]
+mod desktop_capture_frame_tests {
+ use super::normalize_desktop_capture_for_action_frame;
+
+ fn png(width: u32, height: u32) -> Vec<u8> {
+ let rgba = vec![0x7f; (width * height * 4) as usize];
+ cua_driver_core::image_utils::encode_rgba_to_png(&rgba, width, height)
+ .expect("encode fixture")
+ }
+
+ #[test]
+ fn native_wayland_capture_is_resized_to_the_reported_action_frame() {
+ let (normalized, width, height, scale) =
+ normalize_desktop_capture_for_action_frame(png(3200, 2000), 1600, 1000)
+ .expect("normalize 2x capture");
+
+ assert_eq!((width, height), (1600, 1000));
+ assert_eq!(
+ cua_driver_core::image_utils::png_dimensions(&normalized).unwrap(),
+ (1600, 1000)
+ );
+ assert_eq!(scale, 2.0);
+ }
+
+ #[test]
+ fn nonuniform_capture_mapping_fails_instead_of_distorting_coordinates() {
+ let error = normalize_desktop_capture_for_action_frame(png(3200, 2000), 1600, 1200)
+ .expect_err("nonuniform mapping must fail closed");
+ assert!(error.to_string().contains("cannot be mapped uniformly"));
+ }
+}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
index 3432293834..631a47337c 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
@@ -208,6 +208,7 @@ struct Toplevel {
title: String,
app_id: String,
closed: bool,
+ activated: bool,
}
#[derive(Clone, Debug)]
@@ -594,12 +595,21 @@ impl Dispatch<ZwlrForeignToplevelHandleV1, ()> for State {
match event {
ftl_handle::Event::Title { title } => tl.title = title,
ftl_handle::Event::AppId { app_id } => tl.app_id = app_id,
+ ftl_handle::Event::State { state } => {
+ tl.activated = foreign_toplevel_state_is_activated(&state)
+ }
ftl_handle::Event::Closed => tl.closed = true,
_ => {}
}
}
}
+fn foreign_toplevel_state_is_activated(state: &[u8]) -> bool {
+ state
+ .chunks_exact(std::mem::size_of::<u32>())
+ .any(|bytes| u32::from_ne_bytes(bytes.try_into().expect("four-byte state")) == 2)
+}
+
/// Enumerate native Wayland toplevels via wlr-foreign-toplevel-management.
/// `xid` begins as the foreign-toplevel handle's connection-scoped protocol id.
/// The dispatcher replaces it with a stable compositor or AT-SPI identity when
@@ -1327,10 +1337,23 @@ pub fn activate_window_for_input_target(
state.seat.clone(),
matching_handle(&state, window_id),
) {
+ let protocol_id = handle.id().protocol_id();
handle.activate(&seat);
- queue.roundtrip(&mut state)?;
- std::thread::sleep(std::time::Duration::from_millis(60));
- return Ok(());
+ let deadline = std::time::Instant::now() + std::time::Duration::from_millis(500);
+ loop {
+ queue.roundtrip(&mut state)?;
+ if state
+ .toplevels
+ .get(&protocol_id)
+ .is_some_and(|toplevel| toplevel.activated)
+ {
+ return Ok(());
+ }
+ if std::time::Instant::now() >= deadline {
+ break;
+ }
+ std::thread::sleep(std::time::Duration::from_millis(10));
+ }
}
if shell_helper::activate_window(window_id) {
@@ -3192,6 +3215,7 @@ fn enrich_native_windows(
title: undecorated_native_title(window).to_owned(),
app_id: window.app_name.clone(),
closed: false,
+ activated: false,
};
window.xid = candidate.xid;
remember_identity(window.xid, &toplevel);
@@ -3579,6 +3603,26 @@ mod tests {
assert_eq!((decoded.width(), decoded.height()), (3, 4));
}
+ #[test]
+ fn foreign_toplevel_state_requires_activated_value() {
+ let states = [0_u32, 2_u32, 3_u32]
+ .into_iter()
+ .flat_map(u32::to_ne_bytes)
+ .collect::<Vec<_>>();
+ assert!(foreign_toplevel_state_is_activated(&states));
+
+ let inactive = [0_u32, 1_u32, 3_u32]
+ .into_iter()
+ .flat_map(u32::to_ne_bytes)
+ .collect::<Vec<_>>();
+ assert!(!foreign_toplevel_state_is_activated(&inactive));
+ }
+
+ #[test]
+ fn foreign_toplevel_state_ignores_incomplete_wire_values() {
+ assert!(!foreign_toplevel_state_is_activated(&[2, 0, 0]));
+ }
+
#[test]
fn shell_helper_capture_failure_is_terminal() {
let result = checked_shell_helper_capture(true, || None)
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs
index 088846188e..129a51ced5 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/overlay.rs
@@ -2,7 +2,7 @@
//!
//! Replaces the X11-only `overlay.rs` render loop on wlroots compositors
//! (sway, labwc, kwin 5.27+, hyprland) by creating a full-screen,
-//! click-through, always-on-top `wl_surface` anchored to the first output
+//! click-through, always-on-top `wl_surface` on every enabled output
//! via `zwlr_layer_shell_v1`. The surface renders the same gradient-arrow
//! cursor as the X11 path by sharing `cursor_overlay::RenderStateCore` —
//! bloom, click-pulse, idle-fade, and motion all work identically.
@@ -15,9 +15,9 @@
//! owner thread (`cua-overlay-wl`) holds the wayland Connection +
//! EventQueue + layer surface; commands flow in over a `crossbeam-channel`.
//! The render core wakes at frame cadence only while pixels can change;
-//! stable or hidden state blocks on the command channel without polling.
+//! stable or hidden state performs only a cheap, one-second topology check.
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
use std::sync::{
atomic::{AtomicBool, Ordering},
OnceLock,
@@ -26,8 +26,9 @@ use std::thread;
use std::time::{Duration, Instant};
use crossbeam_channel::{bounded, Receiver, Sender};
-use cursor_overlay::{CursorConfig, OverlayCommand, OverlayMsg, RenderStateCore};
+use cursor_overlay::{CursorConfig, CursorKey, OverlayCommand, OverlayMsg, RenderStateCore};
use wayland_client::{
+ globals::{registry_queue_init, GlobalListContents},
protocol::{
wl_buffer::WlBuffer,
wl_compositor::WlCompositor,
@@ -40,6 +41,10 @@ use wayland_client::{
},
Connection, Dispatch, Proxy, QueueHandle,
};
+use wayland_protocols::xdg::xdg_output::zv1::client::{
+ zxdg_output_manager_v1::ZxdgOutputManagerV1,
+ zxdg_output_v1::{self, ZxdgOutputV1},
+};
use wayland_protocols_wlr::layer_shell::v1::client::{
zwlr_layer_shell_v1::{Layer, ZwlrLayerShellV1},
zwlr_layer_surface_v1::{self, Anchor, KeyboardInteractivity, ZwlrLayerSurfaceV1},
@@ -50,8 +55,9 @@ use wayland_protocols_wlr::layer_shell::v1::client::{
/// SetPressed) are forwarded as-is so the layer-shell overlay matches the
/// X11 visual: bloom + animated arrow + click pulse + press ring.
enum WlOverlayCmd {
- Cmd { cmd: OverlayCommand },
- Remove,
+ Cmd { key: CursorKey, cmd: OverlayCommand },
+ Remove(CursorKey),
+ Revive(CursorKey),
Shutdown,
}
@@ -60,9 +66,44 @@ static TX: OnceLock<Sender<WlOverlayCmd>> = OnceLock::new();
// opt the native Wayland overlay in with the daemon's CursorConfig. This keeps
// lazy forwarding from bypassing --no-overlay before any window exists.
static CONFIG_ENABLED: AtomicBool = AtomicBool::new(false);
+static CONFIG_TEMPLATE: OnceLock<CursorConfig> = OnceLock::new();
+static LAYER_SHELL_AVAILABLE: OnceLock<bool> = OnceLock::new();
+
+struct LayerShellProbe;
+
+impl Dispatch<wl_registry::WlRegistry, GlobalListContents> for LayerShellProbe {
+ fn event(
+ _: &mut Self,
+ _: &wl_registry::WlRegistry,
+ _: wl_registry::Event,
+ _: &GlobalListContents,
+ _: &Connection,
+ _: &QueueHandle<Self>,
+ ) {
+ }
+}
+
+/// Probe once so a queued command is never mistaken for a usable layer-shell
+/// backend. This lets an older compositor helper remain a real fallback when
+/// the compositor does not advertise `zwlr_layer_shell_v1`.
+pub fn available() -> bool {
+ *LAYER_SHELL_AVAILABLE.get_or_init(|| {
+ let Ok(conn) = Connection::connect_to_env() else {
+ return false;
+ };
+ let Ok((globals, _queue)) = registry_queue_init::<LayerShellProbe>(&conn) else {
+ return false;
+ };
+ globals.contents().with_list(|list| {
+ list.iter()
+ .any(|global| global.interface == "zwlr_layer_shell_v1")
+ })
+ })
+}
-pub fn set_config_enabled(enabled: bool) {
- CONFIG_ENABLED.store(enabled, Ordering::Release);
+pub fn set_config(config: CursorConfig) {
+ CONFIG_ENABLED.store(config.enabled, Ordering::Release);
+ let _ = CONFIG_TEMPLATE.set(config);
}
fn tx() -> Option<&'static Sender<WlOverlayCmd>> {
@@ -98,9 +139,9 @@ pub fn forward(msg: &OverlayMsg) -> bool {
if !should_forward(CONFIG_ENABLED.load(Ordering::Acquire), msg) {
return false;
}
- // The native Wayland path owns one cursor and does not keep keyed session
- // tombstones. Accept revival without starting the compositor thread.
- if matches!(msg, OverlayMsg::Revive(_)) {
+ // If the owner has never started, it cannot hold a tombstone. Accept the
+ // lifecycle transition without paying the compositor startup cost.
+ if matches!(msg, OverlayMsg::Revive(_)) && tx().is_none() {
return true;
}
// Lazy startup: spawning the layer-shell owner thread + connecting to
@@ -113,22 +154,18 @@ pub fn forward(msg: &OverlayMsg) -> bool {
return false;
}
let Some(tx) = tx() else { return false };
+ map_overlay_msg(msg).is_some_and(|cmd| tx.try_send(cmd).is_ok())
+}
+
+fn map_overlay_msg(msg: &OverlayMsg) -> Option<WlOverlayCmd> {
match msg {
- OverlayMsg::Remove(k) => {
- let _ = k;
- let _ = tx.try_send(WlOverlayCmd::Remove);
- true
- }
- OverlayMsg::Cmd(kc) => {
- if matches!(&kc.cmd, OverlayCommand::ShowFocusRect(_)) {
- return false;
- }
- let _ = tx.try_send(WlOverlayCmd::Cmd {
- cmd: kc.cmd.clone(),
- });
- true
- }
- OverlayMsg::Revive(_) => true,
+ OverlayMsg::Remove(key) => Some(WlOverlayCmd::Remove(key.clone())),
+ OverlayMsg::Cmd(kc) if matches!(&kc.cmd, OverlayCommand::ShowFocusRect(_)) => None,
+ OverlayMsg::Cmd(kc) => Some(WlOverlayCmd::Cmd {
+ key: kc.key.clone(),
+ cmd: kc.cmd.clone(),
+ }),
+ OverlayMsg::Revive(key) => Some(WlOverlayCmd::Revive(key.clone())),
}
}
@@ -154,15 +191,25 @@ struct OverlayState {
compositor: Option<WlCompositor>,
shm: Option<WlShm>,
layer_shell: Option<ZwlrLayerShellV1>,
- output: Option<WlOutput>,
- output_w: u32,
- output_h: u32,
- surface: Option<WlSurface>,
- layer_surface: Option<ZwlrLayerSurfaceV1>,
- configured: bool,
- /// Cross-platform render core: position, animation, gradient arrow,
- /// bloom, click pulse, idle-fade. Shared verbatim with the X11 path.
- core: RenderStateCore,
+ xdg_output_manager: Option<ZxdgOutputManagerV1>,
+ outputs: HashMap<u32, NativeOutput>,
+ /// Outputs whose most recently committed buffer contains cursor pixels.
+ /// Usually this contains exactly one output; keeping a set makes hide,
+ /// removal, and topology changes clear every previously painted surface.
+ painted_outputs: HashSet<u32>,
+ /// Configured layer surfaces that have received at least one wl_buffer.
+ /// This is intentionally separate from `painted_outputs`: every new or
+ /// reconfigured surface needs a one-shot transparent commit to complete
+ /// its layer-shell handshake, but stable empty surfaces must not trigger
+ /// recurring full-output SHM redraws.
+ initialized_outputs: HashSet<u32>,
+ topology_dirty: bool,
+ /// Keyed render cores mirror the X11 native overlay contract. Removing a
+ /// named key records it in `ended`, so already-queued late commands cannot
+ /// recreate a cursor after end_session.
+ cores: HashMap<CursorKey, RenderStateCore>,
+ template: CursorConfig,
+ ended: HashSet<CursorKey>,
/// In-flight wl_shm buffers awaiting `wl_buffer.release` from the
/// compositor. Keyed by `WlBuffer` object id; value is the
/// `(mmap ptr, mmap size, memfd fd)` triple that must be unmapped +
@@ -172,6 +219,107 @@ struct OverlayState {
pending_buffers: HashMap<u32, (*mut libc::c_void, usize, i32)>,
}
+struct NativeOutput {
+ output: WlOutput,
+ xdg_output: Option<ZxdgOutputV1>,
+ surface: Option<WlSurface>,
+ layer_surface: Option<ZwlrLayerSurfaceV1>,
+ wl_origin: Option<(i32, i32)>,
+ logical_origin: Option<(i32, i32)>,
+ logical_size: Option<(u32, u32)>,
+ configured_size: Option<(u32, u32)>,
+ mode_size: Option<(u32, u32)>,
+ scale: i32,
+ name: Option<String>,
+ closed: bool,
+}
+
+impl NativeOutput {
+ fn new(output: WlOutput) -> Self {
+ Self {
+ output,
+ xdg_output: None,
+ surface: None,
+ layer_surface: None,
+ wl_origin: None,
+ logical_origin: None,
+ logical_size: None,
+ configured_size: None,
+ mode_size: None,
+ scale: 1,
+ name: None,
+ closed: false,
+ }
+ }
+
+ fn layout(&self, id: u32) -> Option<OutputLayout> {
+ if self.layer_surface.is_none() || self.configured_size.is_none() {
+ return None;
+ }
+ let (origin_x, origin_y) = self.logical_origin.or(self.wl_origin).unwrap_or((0, 0));
+ let (width, height) = self.logical_size.or(self.configured_size).or_else(|| {
+ let scale = self.scale.max(1) as u32;
+ self.mode_size
+ .map(|(width, height)| ((width / scale).max(1), (height / scale).max(1)))
+ })?;
+ (width > 0 && height > 0).then_some(OutputLayout {
+ id,
+ origin_x,
+ origin_y,
+ width,
+ height,
+ })
+ }
+
+ fn destroy_surfaces(&mut self) {
+ self.close_layer();
+ if let Some(xdg_output) = self.xdg_output.take() {
+ xdg_output.destroy();
+ }
+ }
+
+ fn close_layer(&mut self) {
+ if let Some(layer_surface) = self.layer_surface.take() {
+ layer_surface.destroy();
+ }
+ if let Some(surface) = self.surface.take() {
+ surface.destroy();
+ }
+ self.configured_size = None;
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct OutputLayout {
+ id: u32,
+ origin_x: i32,
+ origin_y: i32,
+ width: u32,
+ height: u32,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+struct SelectedOutput {
+ id: u32,
+ local_x: f64,
+ local_y: f64,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct FrameTarget {
+ id: u32,
+}
+
+#[derive(Clone, Copy)]
+struct OutputData {
+ id: u32,
+}
+
+#[derive(Clone, Copy)]
+struct LayerData {
+ id: u32,
+}
+
// SAFETY: the raw pointers in pending_buffers point at mmap regions owned
// exclusively by this thread (the owner thread). OverlayState is never
// shared across threads — wayland-client's EventQueue<State> is !Send so
@@ -180,119 +328,305 @@ struct OverlayState {
// explicit assertion.
unsafe impl Send for OverlayState {}
-impl Default for OverlayState {
- fn default() -> Self {
+impl Drop for OverlayState {
+ fn drop(&mut self) {
+ for (_, (ptr, size, fd)) in std::mem::take(&mut self.pending_buffers) {
+ super::cleanup_mmap(ptr, size, fd);
+ }
+ }
+}
+
+impl OverlayState {
+ fn new(template: CursorConfig) -> Self {
+ let mut cores = HashMap::new();
+ // Match the X11 contract: the compatibility/default slot preserves the
+ // launch-time cursor id, while lazily-created named slots override it.
+ cores.insert("default".to_owned(), RenderStateCore::new(template.clone()));
Self {
compositor: None,
shm: None,
layer_shell: None,
- output: None,
- output_w: 0,
- output_h: 0,
- surface: None,
- layer_surface: None,
- configured: false,
- core: RenderStateCore::new(CursorConfig::default()),
+ xdg_output_manager: None,
+ outputs: HashMap::new(),
+ painted_outputs: HashSet::new(),
+ initialized_outputs: HashSet::new(),
+ topology_dirty: false,
+ cores,
+ template,
+ ended: HashSet::new(),
pending_buffers: HashMap::new(),
}
}
}
+fn render_core_for_key(template: &CursorConfig, key: &str) -> RenderStateCore {
+ let mut config = template.clone();
+ config.cursor_id = key.to_owned();
+ RenderStateCore::new(config)
+}
+
+fn apply_keyed_command(
+ cores: &mut HashMap<CursorKey, RenderStateCore>,
+ template: &CursorConfig,
+ ended: &HashSet<CursorKey>,
+ key: CursorKey,
+ cmd: OverlayCommand,
+) -> bool {
+ if ended.contains(&key) {
+ tracing::debug!(key = %key, cmd = ?cmd, "wayland overlay: command dropped — key was ended");
+ return false;
+ }
+
+ let core = cores
+ .entry(key.clone())
+ .or_insert_with(|| render_core_for_key(template, &key));
+ // Seed from the off-screen sentinel near the first targeted action so a
+ // spring animation begins on-screen. This mirrors the X11 renderer.
+ let seed_target = match &cmd {
+ OverlayCommand::MoveTo { x, y, .. }
+ | OverlayCommand::SnapTo { x, y, .. }
+ | OverlayCommand::ClickPulse { x, y } => Some((*x, *y)),
+ _ => None,
+ };
+ if let Some((target_x, target_y)) = seed_target {
+ if core.pos.0 < -50.0 {
+ const SEED_OFFSET: f64 = 140.0;
+ core.pos = (
+ (target_x - SEED_OFFSET).max(2.0),
+ (target_y - SEED_OFFSET).max(2.0),
+ );
+ }
+ }
+
+ let disabling = matches!(&cmd, OverlayCommand::SetEnabled(false));
+ let dirty = core.apply_command_base(cmd, false, false);
+ if disabling {
+ quiesce_hidden(core);
+ }
+ dirty
+}
+
+fn remove_keyed_core(
+ cores: &mut HashMap<CursorKey, RenderStateCore>,
+ ended: &mut HashSet<CursorKey>,
+ key: CursorKey,
+) -> bool {
+ if key == "default" {
+ return false;
+ }
+ let removed = cores.remove(&key).is_some();
+ ended.insert(key);
+ removed
+}
+
+fn revive_key(ended: &mut HashSet<CursorKey>, key: CursorKey) {
+ if key != "default" {
+ ended.remove(&key);
+ }
+}
+
fn dbg(msg: &str) {
if std::env::var_os("CUA_OVERLAY_DEBUG").is_some() {
eprintln!("[cua-overlay-wl] {msg}");
}
}
-fn owner_thread(rx: Receiver<WlOverlayCmd>) -> anyhow::Result<()> {
- let conn = Connection::connect_to_env()?;
- let mut queue = conn.new_event_queue::<OverlayState>();
- let qh = queue.handle();
- let _registry = conn.display().get_registry(&qh, ());
+fn select_output(layouts: &[OutputLayout], x: f64, y: f64) -> Option<SelectedOutput> {
+ if !x.is_finite() || !y.is_finite() {
+ return None;
+ }
+ layouts
+ .iter()
+ .filter(|layout| output_contains(layout, x, y))
+ .min_by_key(|layout| layout.id)
+ .map(|layout| SelectedOutput {
+ id: layout.id,
+ local_x: x - f64::from(layout.origin_x),
+ local_y: y - f64::from(layout.origin_y),
+ })
+}
- let mut state = OverlayState::default();
- queue.roundtrip(&mut state)?;
- for _ in 0..3 {
- queue.roundtrip(&mut state)?;
+fn output_contains(layout: &OutputLayout, x: f64, y: f64) -> bool {
+ x.is_finite()
+ && y.is_finite()
+ && x >= f64::from(layout.origin_x)
+ && y >= f64::from(layout.origin_y)
+ && x < f64::from(layout.origin_x) + f64::from(layout.width)
+ && y < f64::from(layout.origin_y) + f64::from(layout.height)
+}
+
+fn frame_plan(
+ layouts: &[OutputLayout],
+ painted_outputs: &HashSet<u32>,
+ initialized_outputs: &HashSet<u32>,
+ cursor_positions: impl IntoIterator<Item = (f64, f64)>,
+) -> (HashSet<u32>, Vec<FrameTarget>) {
+ let selected: HashSet<u32> = cursor_positions
+ .into_iter()
+ .filter_map(|(x, y)| select_output(layouts, x, y).map(|output| output.id))
+ .collect();
+ let mut ids: Vec<u32> = painted_outputs.iter().copied().collect();
+ ids.extend(
+ layouts
+ .iter()
+ .map(|layout| layout.id)
+ .filter(|id| !initialized_outputs.contains(id)),
+ );
+ for id in &selected {
+ if !ids.contains(id) {
+ ids.push(*id);
+ }
}
+ ids.sort_unstable();
+ ids.dedup();
+ let targets = ids.into_iter().map(|id| FrameTarget { id }).collect();
+ (selected, targets)
+}
- let compositor = state
- .compositor
- .clone()
- .ok_or_else(|| anyhow::anyhow!("compositor does not expose wl_compositor"))?;
- let shm = state
- .shm
- .clone()
- .ok_or_else(|| anyhow::anyhow!("compositor does not expose wl_shm"))?;
- let layer_shell = state
- .layer_shell
- .clone()
- .ok_or_else(|| anyhow::anyhow!("compositor does not expose zwlr_layer_shell_v1"))?;
- let output = state
- .output
- .clone()
- .ok_or_else(|| anyhow::anyhow!("compositor exposed no wl_output"))?;
+fn visible_cores_for_output<'a>(
+ cores: &'a HashMap<CursorKey, RenderStateCore>,
+ layouts: &[OutputLayout],
+ output_id: u32,
+) -> Vec<(&'a CursorKey, &'a RenderStateCore)> {
+ let mut visible_cores: Vec<_> = cores
+ .iter()
+ .filter(|(_, core)| {
+ core.visible
+ && core.pos.0 >= -100.0
+ && core.idle_alpha >= 0.004
+ && select_output(layouts, core.pos.0, core.pos.1)
+ .is_some_and(|selected| selected.id == output_id)
+ })
+ .collect();
+ visible_cores.sort_by(|(left, _), (right, _)| left.cmp(right));
+ visible_cores
+}
+
+fn ensure_output_resources(state: &mut OverlayState, qh: &QueueHandle<OverlayState>) {
+ let ids: Vec<u32> = state.outputs.keys().copied().collect();
+ for id in ids {
+ ensure_xdg_output(state, id, qh);
+ ensure_layer_surface(state, id, qh);
+ }
+}
+
+fn ensure_xdg_output(state: &mut OverlayState, id: u32, qh: &QueueHandle<OverlayState>) {
+ let Some(manager) = state.xdg_output_manager.clone() else {
+ return;
+ };
+ let Some(output) = state.outputs.get_mut(&id) else {
+ return;
+ };
+ if output.xdg_output.is_none() {
+ output.xdg_output = Some(manager.get_xdg_output(&output.output, qh, OutputData { id }));
+ }
+}
+
+fn ensure_layer_surface(state: &mut OverlayState, id: u32, qh: &QueueHandle<OverlayState>) {
+ let (Some(compositor), Some(layer_shell)) =
+ (state.compositor.clone(), state.layer_shell.clone())
+ else {
+ return;
+ };
+ let Some(output) = state.outputs.get_mut(&id) else {
+ return;
+ };
+ if output.layer_surface.is_some() || output.closed {
+ return;
+ }
- // Build the layer surface: fullscreen, overlay layer, click-through.
- let surface = compositor.create_surface(&qh, ());
+ let surface = compositor.create_surface(qh, ());
let layer_surface = layer_shell.get_layer_surface(
&surface,
- Some(&output),
+ Some(&output.output),
Layer::Overlay,
"cua-agent-cursor".to_string(),
- &qh,
- (),
+ qh,
+ LayerData { id },
);
- // Anchor to all four edges = full screen.
layer_surface.set_anchor(Anchor::Top | Anchor::Bottom | Anchor::Left | Anchor::Right);
layer_surface.set_size(0, 0);
layer_surface.set_exclusive_zone(-1);
layer_surface.set_keyboard_interactivity(KeyboardInteractivity::None);
- // Click-through: empty input region. Standard Wayland intentionally does
- // not expose another client's global pointer position, so this surface
- // cannot implement the macOS/Windows/X11 badge-hover reveal without a
- // compositor-owned adapter. Giving it an input region would steal the
- // user's pointer events instead of observing them.
- let region: WlRegion = compositor.create_region(&qh, ());
+ let region: WlRegion = compositor.create_region(qh, ());
surface.set_input_region(Some(&region));
region.destroy();
- state.surface = Some(surface);
- state.layer_surface = Some(layer_surface);
+ surface.commit();
+ output.surface = Some(surface);
+ output.layer_surface = Some(layer_surface);
+}
- // First commit kicks off the configure handshake.
- if let Some(s) = state.surface.as_ref() {
- s.commit();
+fn owner_thread(rx: Receiver<WlOverlayCmd>) -> anyhow::Result<()> {
+ let conn = Connection::connect_to_env()?;
+ let mut queue = conn.new_event_queue::<OverlayState>();
+ let qh = queue.handle();
+ let _registry = conn.display().get_registry(&qh, ());
+
+ let template = CONFIG_TEMPLATE.get().cloned().unwrap_or_default();
+ let mut state = OverlayState::new(template);
+ queue.roundtrip(&mut state)?;
+
+ state
+ .compositor
+ .clone()
+ .ok_or_else(|| anyhow::anyhow!("compositor does not expose wl_compositor"))?;
+ let shm = state
+ .shm
+ .clone()
+ .ok_or_else(|| anyhow::anyhow!("compositor does not expose wl_shm"))?;
+ state
+ .layer_shell
+ .clone()
+ .ok_or_else(|| anyhow::anyhow!("compositor does not expose zwlr_layer_shell_v1"))?;
+ if state.outputs.is_empty() {
+ anyhow::bail!("compositor exposed no wl_output");
}
- // Wait for the first configure event so we know the output dimensions
- // before drawing.
+ // Build one fullscreen, click-through layer surface per advertised output.
+ ensure_output_resources(&mut state, &qh);
+
+ // Give every enabled output a chance to configure. Disabled outputs may
+ // stay advertised without configuring and are excluded from selection.
for _ in 0..10 {
queue.roundtrip(&mut state)?;
- if state.configured && state.output_w > 0 && state.output_h > 0 {
+ ensure_output_resources(&mut state, &qh);
+ if state
+ .outputs
+ .values()
+ .filter(|output| output.layer_surface.is_some())
+ .all(|output| output.configured_size.is_some())
+ {
break;
}
- std::thread::sleep(std::time::Duration::from_millis(50));
}
- if !state.configured {
- anyhow::bail!("layer surface never received configure event");
+ let configured_outputs = state
+ .outputs
+ .values()
+ .filter(|output| output.configured_size.is_some())
+ .count();
+ if configured_outputs == 0 {
+ anyhow::bail!("no layer surface received a configure event");
}
- dbg(&format!(
- "configured: w={} h={}",
- state.output_w, state.output_h
- ));
+ dbg(&format!("configured outputs: {configured_outputs}"));
+ state.topology_dirty = false;
+
+ // A layer-shell surface is not fully initialized until the first buffer is
+ // attached after configure. Commit one transparent buffer to every empty
+ // output now, before entering the demand-driven loop; a cursor already
+ // selected on an output can share this same first frame.
+ redraw(&mut state, &shm, &qh)?;
+ queue.roundtrip(&mut state)?;
- // Demand-driven main loop. A stable cursor blocks on the command channel;
- // only active motion, click/fade animation, or the exact idle-fade
- // deadline schedules a wake. This avoids display-sized SHM allocation and
- // conversion at 60Hz when no pixel can change while preserving immediate
- // command wakeups.
+ // Demand-driven main loop. Stable cursors perform only a cheap Wayland
+ // maintenance roundtrip once per second so output topology changes are
+ // observed; full-display SHM work remains limited to visual changes.
let mut last_tick = Instant::now();
let mut frame_tick_needed = false;
loop {
- let wait = next_wait(&state.core, frame_tick_needed);
+ let wait = next_wait(&state.cores, frame_tick_needed, state.topology_dirty);
let (first_cmd, timed_out) = match wait_for_work(&rx, wait) {
WlWake::Command(cmd) => (Some(cmd), None),
WlWake::Timeout => (None, Some(wait)),
@@ -313,44 +647,20 @@ fn owner_thread(rx: Receiver<WlOverlayCmd>) -> anyhow::Result<()> {
shutdown = true;
break;
}
- Ok(WlOverlayCmd::Cmd { cmd }) => {
- // Seed: if the cursor is still at the off-screen sentinel
- // `(-200, -200)` from `RenderStateCore::new`, snap to a
- // point near the MoveTo / SnapTo target so the spring
- // animation starts on-screen. Mirrors X11 overlay.rs's
- // `seed_start_if_sentinel` helper — without it, the
- // spring oscillates around the sentinel and the cursor
- // never reaches the screen.
- let seed_target = match &cmd {
- OverlayCommand::MoveTo { x, y, .. }
- | OverlayCommand::SnapTo { x, y, .. }
- | OverlayCommand::ClickPulse { x, y } => Some((*x, *y)),
- _ => None,
- };
- if let Some((tx, ty)) = seed_target {
- if state.core.pos.0 < -50.0 {
- const SEED_OFFSET: f64 = 16.0;
- let sx = (tx - SEED_OFFSET).max(2.0);
- let sy = (ty - SEED_OFFSET).max(2.0);
- state.core.pos = (sx, sy);
- }
- }
- // apply_command_base consumes every variant the X11
- // path handles. `move_to_snap_sentinel` / `click_pulse
- // _sentinel_only` are both `false` here — same as X11.
- let disabling = matches!(&cmd, OverlayCommand::SetEnabled(false));
- dirty |= state.core.apply_command_base(cmd, false, false);
- if disabling {
- quiesce_hidden(&mut state.core);
- }
+ Ok(WlOverlayCmd::Cmd { key, cmd }) => {
+ dirty |= apply_keyed_command(
+ &mut state.cores,
+ &state.template,
+ &state.ended,
+ key,
+ cmd,
+ );
+ }
+ Ok(WlOverlayCmd::Remove(key)) => {
+ dirty |= remove_keyed_core(&mut state.cores, &mut state.ended, key);
}
- Ok(WlOverlayCmd::Remove) => {
- // Single-cursor overlay: removing the active cursor
- // hides it. Multi-cursor wlroots support can layer on
- // top of this in a follow-up if needed.
- dirty |= state.core.visible || state.core.pos.0 >= -100.0;
- state.core.visible = false;
- quiesce_hidden(&mut state.core);
+ Ok(WlOverlayCmd::Revive(key)) => {
+ revive_key(&mut state.ended, key);
}
Err(crossbeam_channel::TryRecvError::Empty) => break,
Err(crossbeam_channel::TryRecvError::Disconnected) => {
@@ -367,16 +677,40 @@ fn owner_thread(rx: Receiver<WlOverlayCmd>) -> anyhow::Result<()> {
// applies the new state at dt=0, avoiding a jump proportional to how
// long the loop was parked.
if let Some(timeout_kind) = timed_out {
- let dt = match timeout_kind {
- WlWait::Frame => elapsed.min(0.05),
- WlWait::Deadline(_) => elapsed,
- WlWait::Block => unreachable!("a blocking receive cannot time out"),
- };
- state.core.tick_motion(dt);
- dirty = true;
+ match timeout_kind {
+ WlWait::Frame => {
+ tick_all_cores(&mut state.cores, elapsed.min(0.05));
+ dirty = true;
+ }
+ WlWait::Deadline(_) => {
+ tick_all_cores(&mut state.cores, elapsed);
+ dirty = true;
+ }
+ WlWait::Maintenance(_) => {
+ let before: HashMap<CursorKey, f64> = state
+ .cores
+ .iter()
+ .map(|(key, core)| (key.clone(), core.idle_alpha))
+ .collect();
+ tick_all_cores(&mut state.cores, elapsed);
+ dirty |= state.cores.iter().any(|(key, core)| {
+ before
+ .get(key)
+ .is_none_or(|alpha| *alpha != core.idle_alpha)
+ || needs_frame_tick(core)
+ });
+
+ let topology_was_dirty = state.topology_dirty;
+ state.topology_dirty = false;
+ queue.roundtrip(&mut state)?;
+ ensure_output_resources(&mut state, &qh);
+ dirty |= topology_was_dirty || state.topology_dirty;
+ state.topology_dirty = false;
+ }
+ }
}
- let next_frame_tick_needed = needs_frame_tick(&state.core);
- if state.configured && (dirty || frame_tick_needed || next_frame_tick_needed) {
+ let next_frame_tick_needed = any_core_needs_frame_tick(&state.cores);
+ if dirty || frame_tick_needed || next_frame_tick_needed {
redraw(&mut state, &shm, &qh)?;
// Flush the committed frame and dispatch wl_buffer.release before
// parking. The buffer map remains authoritative until release, so
@@ -386,11 +720,8 @@ fn owner_thread(rx: Receiver<WlOverlayCmd>) -> anyhow::Result<()> {
frame_tick_needed = next_frame_tick_needed;
}
- if let Some(ls) = state.layer_surface.take() {
- ls.destroy();
- }
- if let Some(s) = state.surface.take() {
- s.destroy();
+ for output in state.outputs.values_mut() {
+ output.destroy_surfaces();
}
queue.roundtrip(&mut state)?;
Ok(())
@@ -398,9 +729,9 @@ fn owner_thread(rx: Receiver<WlOverlayCmd>) -> anyhow::Result<()> {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum WlWait {
- Block,
Frame,
Deadline(Duration),
+ Maintenance(Duration),
}
enum WlWake {
@@ -411,30 +742,52 @@ enum WlWake {
fn wait_for_work(rx: &Receiver<WlOverlayCmd>, wait: WlWait) -> WlWake {
match wait {
- WlWait::Block => match rx.recv() {
- Ok(cmd) => WlWake::Command(cmd),
- Err(_) => WlWake::Disconnected,
- },
WlWait::Frame => match rx.recv_timeout(Duration::from_millis(16)) {
Ok(cmd) => WlWake::Command(cmd),
Err(crossbeam_channel::RecvTimeoutError::Timeout) => WlWake::Timeout,
Err(crossbeam_channel::RecvTimeoutError::Disconnected) => WlWake::Disconnected,
},
- WlWait::Deadline(timeout) => match rx.recv_timeout(timeout) {
- Ok(cmd) => WlWake::Command(cmd),
- Err(crossbeam_channel::RecvTimeoutError::Timeout) => WlWake::Timeout,
- Err(crossbeam_channel::RecvTimeoutError::Disconnected) => WlWake::Disconnected,
- },
+ WlWait::Deadline(timeout) | WlWait::Maintenance(timeout) => {
+ match rx.recv_timeout(timeout) {
+ Ok(cmd) => WlWake::Command(cmd),
+ Err(crossbeam_channel::RecvTimeoutError::Timeout) => WlWake::Timeout,
+ Err(crossbeam_channel::RecvTimeoutError::Disconnected) => WlWake::Disconnected,
+ }
+ }
}
}
-fn next_wait(core: &RenderStateCore, frame_tick_needed: bool) -> WlWait {
- if frame_tick_needed || needs_frame_tick(core) {
+const TOPOLOGY_MAINTENANCE_INTERVAL: Duration = Duration::from_secs(1);
+
+fn next_wait(
+ cores: &HashMap<CursorKey, RenderStateCore>,
+ frame_tick_needed: bool,
+ topology_dirty: bool,
+) -> WlWait {
+ if topology_dirty {
+ return WlWait::Maintenance(Duration::ZERO);
+ }
+ if frame_tick_needed || any_core_needs_frame_tick(cores) {
return WlWait::Frame;
}
- idle_fade_wait(core)
- .map(WlWait::Deadline)
- .unwrap_or(WlWait::Block)
+ match earliest_idle_fade_wait(cores) {
+ Some(wait) if wait <= TOPOLOGY_MAINTENANCE_INTERVAL => WlWait::Deadline(wait),
+ _ => WlWait::Maintenance(TOPOLOGY_MAINTENANCE_INTERVAL),
+ }
+}
+
+fn any_core_needs_frame_tick(cores: &HashMap<CursorKey, RenderStateCore>) -> bool {
+ cores.values().any(needs_frame_tick)
+}
+
+fn earliest_idle_fade_wait(cores: &HashMap<CursorKey, RenderStateCore>) -> Option<Duration> {
+ cores.values().filter_map(idle_fade_wait).min()
+}
+
+fn tick_all_cores(cores: &mut HashMap<CursorKey, RenderStateCore>, dt: f64) {
+ for core in cores.values_mut() {
+ core.tick_motion(dt);
+ }
}
fn needs_frame_tick(core: &RenderStateCore) -> bool {
@@ -472,8 +825,8 @@ fn quiesce_hidden(core: &mut RenderStateCore) {
core.click_t = None;
}
-/// Render one cursor frame into a fresh wl_shm ARGB8888 buffer and attach
-/// it to the layer surface.
+/// Composite all visible cursor cores into their selected output and attach a
+/// transparent clearing frame to every previously painted output now empty.
///
/// Pipeline:
/// 1. Allocate a memfd-backed wl_shm pool sized at output_w × output_h.
@@ -485,34 +838,66 @@ fn quiesce_hidden(core: &mut RenderStateCore) {
/// in `ext_screencopy::encode_buffer_to_png`.
/// 4. Attach + damage + commit on the layer surface.
///
-/// When the cursor is hidden (`core.visible == false`, idle-faded, or
-/// off-screen sentinel) the pixmap is all zeros — the surface remains
-/// transparent and click-through.
+/// Hidden, idle-faded, or off-screen cores paint nothing.
fn redraw(
state: &mut OverlayState,
shm: &WlShm,
qh: &QueueHandle<OverlayState>,
) -> anyhow::Result<()> {
- let Some(surface) = state.surface.as_ref() else {
- return Ok(());
- };
- let w = state.output_w.max(1);
- let h = state.output_h.max(1);
- let stride = w as i32 * 4;
- let size = (stride as usize) * (h as usize);
+ let layouts: Vec<OutputLayout> = state
+ .outputs
+ .iter()
+ .filter_map(|(&id, output)| output.layout(id))
+ .collect();
+ let cursor_positions = state
+ .cores
+ .values()
+ .filter(|core| core.visible && core.pos.0 >= -100.0 && core.idle_alpha >= 0.004)
+ .map(|core| core.pos);
+ let (selected, targets) = frame_plan(
+ &layouts,
+ &state.painted_outputs,
+ &state.initialized_outputs,
+ cursor_positions,
+ );
- // Reuses the same anon_shm pattern as the screencopy path in mod.rs.
- let (fd, ptr) =
- super::anon_shm(size).map_err(|e| anyhow::anyhow!("overlay shm allocation failed: {e}"))?;
+ for target in targets {
+ redraw_output(state, shm, qh, target, &layouts)?;
+ }
- // SAFETY: ptr came from mmap of `size` bytes, lifetime bounded to this
- // function.
- let pixels: &mut [u8] = unsafe { std::slice::from_raw_parts_mut(ptr as *mut u8, size) };
+ state.painted_outputs = selected;
+ Ok(())
+}
- // Paint the cursor into a tiny_skia pixmap. paint_cursor early-returns
- // when the cursor is hidden / off-screen / idle-faded, so the pixmap
- // is left fully transparent in those cases (which is also what we want
- // for the click-through layer surface).
+fn redraw_output(
+ state: &mut OverlayState,
+ shm: &WlShm,
+ qh: &QueueHandle<OverlayState>,
+ target: FrameTarget,
+ layouts: &[OutputLayout],
+) -> anyhow::Result<()> {
+ let Some((surface, layout)) = state
+ .outputs
+ .get(&target.id)
+ .and_then(|output| Some((output.surface.clone()?, output.layout(target.id)?)))
+ else {
+ return Ok(());
+ };
+ let w = layout.width.max(1);
+ let h = layout.height.max(1);
+ let stride = w
+ .checked_mul(4)
+ .and_then(|stride| i32::try_from(stride).ok())
+ .ok_or_else(|| anyhow::anyhow!("overlay output {w}x{h} has an invalid stride"))?;
+ let size = usize::try_from(stride)
+ .ok()
+ .and_then(|stride| stride.checked_mul(h as usize))
+ .ok_or_else(|| anyhow::anyhow!("overlay output {w}x{h} buffer size overflow"))?;
+
+ // A clearing target intentionally stays transparent. Each cursor selected
+ // for this output subtracts its logical compositor origin from the global
+ // position; this is the same origin contract used by the Windows virtual
+ // desktop.
let pm_result = tiny_skia::Pixmap::new(w, h);
let mut pm = match pm_result {
Some(p) => p,
@@ -525,10 +910,31 @@ fn redraw(
"tiny_skia::Pixmap::new({w}, {h}) failed — out of memory for the overlay buffer"
),
};
- // backing_scale=1.0 matches the X11 path; per-output Wayland scale is
- // a follow-up (would consume `wl_output.scale` and `preferred_buffer
- // _scale` from wl_surface v6).
- cursor_overlay::paint_cursor(&mut pm, &state.core, 0.0, 0.0, None, 1.0);
+
+ // Reuses the same anon_shm pattern as the screencopy path in mod.rs. The
+ // pixmap is allocated first so a tiny-skia OOM cannot strand this mmap/fd.
+ let (fd, ptr) =
+ super::anon_shm(size).map_err(|e| anyhow::anyhow!("overlay shm allocation failed: {e}"))?;
+
+ // SAFETY: ptr came from mmap of `size` bytes and is transferred to
+ // `pending_buffers` before this function returns successfully.
+ let pixels: &mut [u8] = unsafe { std::slice::from_raw_parts_mut(ptr as *mut u8, size) };
+ let mut painted_positions = Vec::new();
+ {
+ // HashMap iteration is intentionally normalized by key so overlapping
+ // named cursors composite deterministically from frame to frame.
+ for (_, core) in visible_cores_for_output(&state.cores, layouts, target.id) {
+ cursor_overlay::paint_cursor(
+ &mut pm,
+ core,
+ f64::from(layout.origin_x),
+ f64::from(layout.origin_y),
+ None,
+ 1.0,
+ );
+ painted_positions.push(core.pos);
+ }
+ }
// CUA_OVERLAY_DEBUG=1 paints a 60x60 magenta square at the cursor's
// current pos on top of the gradient arrow. Useful when validating
@@ -536,22 +942,23 @@ fn redraw(
// small at native scale and easy to miss in a screenshot, while the
// magenta block is impossible to miss.
if std::env::var_os("CUA_OVERLAY_DEBUG").is_some() {
- let (cx, cy) = state.core.pos;
- let cx = cx as i32;
- let cy = cy as i32;
- let half = 30i32;
- for dy in -half..half {
- for dx in -half..half {
- let px = cx + dx;
- let py = cy + dy;
- if px < 0 || py < 0 || px >= w as i32 || py >= h as i32 {
- continue;
+ for &(cursor_x, cursor_y) in &painted_positions {
+ let cx = (cursor_x - f64::from(layout.origin_x)) as i32;
+ let cy = (cursor_y - f64::from(layout.origin_y)) as i32;
+ let half = 30i32;
+ for dy in -half..half {
+ for dx in -half..half {
+ let px = cx + dx;
+ let py = cy + dy;
+ if px < 0 || py < 0 || px >= w as i32 || py >= h as i32 {
+ continue;
+ }
+ let off = ((py as usize) * (w as usize) + (px as usize)) * 4;
+ pm.data_mut()[off] = 0xFF; // R
+ pm.data_mut()[off + 1] = 0x00; // G
+ pm.data_mut()[off + 2] = 0xFF; // B
+ pm.data_mut()[off + 3] = 0xFF; // A
}
- let off = ((py as usize) * (w as usize) + (px as usize)) * 4;
- pm.data_mut()[off] = 0xFF; // R
- pm.data_mut()[off + 1] = 0x00; // G
- pm.data_mut()[off + 2] = 0xFF; // B
- pm.data_mut()[off + 3] = 0xFF; // A
}
}
}
@@ -588,16 +995,33 @@ fn redraw(
state.pending_buffers.insert(buffer_id, (ptr, size, fd));
dbg(&format!(
- "redraw w={w} h={h} stride={stride} buf_id={buffer_id} pos=({:.1},{:.1}) visible={}",
- state.core.pos.0, state.core.pos.1, state.core.visible
+ "redraw output={} origin=({}, {}) w={w} h={h} stride={stride} buf_id={buffer_id} cursors={}",
+ output_label(state, target.id),
+ layout.origin_x,
+ layout.origin_y,
+ painted_positions.len(),
));
surface.attach(Some(&buffer), 0, 0);
+ // Damage both coordinate spaces. Some wlroots compositors otherwise leave
+ // stale transparent frames behind while an animated cursor is moving.
+ surface.damage(0, 0, w as i32, h as i32);
surface.damage_buffer(0, 0, w as i32, h as i32);
surface.commit();
+ // Mark initialized only after the buffer attach + surface commit succeed.
+ // An early return above leaves the output pending for the next plan.
+ state.initialized_outputs.insert(target.id);
pool.destroy();
Ok(())
}
+fn output_label(state: &OverlayState, id: u32) -> String {
+ state
+ .outputs
+ .get(&id)
+ .and_then(|output| output.name.clone())
+ .unwrap_or_else(|| format!("wl_output#{id}"))
+}
+
// ── Wayland Dispatch impls ───────────────────────────────────────────────
impl Dispatch<wl_registry::WlRegistry, ()> for OverlayState {
@@ -609,32 +1033,68 @@ impl Dispatch<wl_registry::WlRegistry, ()> for OverlayState {
_conn: &Connection,
qh: &QueueHandle<Self>,
) {
- if let wl_registry::Event::Global {
- name,
- interface,
- version,
- } = event
- {
- match interface.as_str() {
- "wl_compositor" => {
- state.compositor =
- Some(registry.bind::<WlCompositor, _, _>(name, version.min(6), qh, ()));
- }
- "wl_shm" => {
- state.shm = Some(registry.bind::<WlShm, _, _>(name, version.min(1), qh, ()));
- }
- "wl_output" => {
- if state.output.is_none() {
- state.output =
- Some(registry.bind::<WlOutput, _, _>(name, version.min(4), qh, ()));
+ match event {
+ wl_registry::Event::Global {
+ name,
+ interface,
+ version,
+ } => {
+ match interface.as_str() {
+ "wl_compositor" => {
+ state.compositor =
+ Some(registry.bind::<WlCompositor, _, _>(name, version.min(6), qh, ()));
+ }
+ "wl_shm" => {
+ state.shm =
+ Some(registry.bind::<WlShm, _, _>(name, version.min(1), qh, ()));
+ }
+ "wl_output" => {
+ let output = registry.bind::<WlOutput, _, _>(
+ name,
+ version.min(4),
+ qh,
+ OutputData { id: name },
+ );
+ state.outputs.insert(name, NativeOutput::new(output));
+ state.topology_dirty = true;
}
+ "zwlr_layer_shell_v1" => {
+ state.layer_shell = Some(registry.bind::<ZwlrLayerShellV1, _, _>(
+ name,
+ version.min(4),
+ qh,
+ (),
+ ));
+ }
+ "zxdg_output_manager_v1" => {
+ state.xdg_output_manager =
+ Some(registry.bind::<ZxdgOutputManagerV1, _, _>(
+ name,
+ version.min(3),
+ qh,
+ (),
+ ));
+ }
+ _ => {}
}
- "zwlr_layer_shell_v1" => {
- state.layer_shell =
- Some(registry.bind::<ZwlrLayerShellV1, _, _>(name, version.min(4), qh, ()));
+ ensure_output_resources(state, qh);
+ }
+ wl_registry::Event::GlobalRemove { name } => {
+ state.painted_outputs.remove(&name);
+ state.initialized_outputs.remove(&name);
+ if let Some(mut output) = state.outputs.remove(&name) {
+ output.destroy_surfaces();
+ // wl_output.release was added in v3. On an older generic
+ // wlroots compositor, dropping the client proxy is the only
+ // valid cleanup; issuing the newer request would be a
+ // protocol error.
+ if output.output.version() >= 3 {
+ output.output.release();
+ }
+ state.topology_dirty = true;
}
- _ => {}
}
+ _ => {}
}
}
}
@@ -663,21 +1123,77 @@ impl Dispatch<WlShm, ()> for OverlayState {
}
}
-impl Dispatch<WlOutput, ()> for OverlayState {
+impl Dispatch<WlOutput, OutputData> for OverlayState {
fn event(
state: &mut Self,
_: &WlOutput,
event: <WlOutput as wayland_client::Proxy>::Event,
- _: &(),
+ data: &OutputData,
_: &Connection,
_: &QueueHandle<Self>,
) {
use wayland_client::protocol::wl_output;
- if let wl_output::Event::Mode { width, height, .. } = event {
- if width > 0 && height > 0 {
- state.output_w = width as u32;
- state.output_h = height as u32;
+ let Some(output) = state.outputs.get_mut(&data.id) else {
+ return;
+ };
+ match event {
+ wl_output::Event::Geometry { x, y, .. } => {
+ state.topology_dirty |= output.wl_origin != Some((x, y));
+ output.wl_origin = Some((x, y));
+ }
+ wl_output::Event::Mode { width, height, .. } if width > 0 && height > 0 => {
+ state.topology_dirty |= output.mode_size != Some((width as u32, height as u32));
+ output.mode_size = Some((width as u32, height as u32));
+ }
+ wl_output::Event::Scale { factor } => {
+ state.topology_dirty |= output.scale != factor.max(1);
+ output.scale = factor.max(1);
+ }
+ wl_output::Event::Name { name } => {
+ output.name = Some(name);
+ }
+ _ => {}
+ }
+ }
+}
+
+impl Dispatch<ZxdgOutputManagerV1, ()> for OverlayState {
+ fn event(
+ _: &mut Self,
+ _: &ZxdgOutputManagerV1,
+ _: <ZxdgOutputManagerV1 as Proxy>::Event,
+ _: &(),
+ _: &Connection,
+ _: &QueueHandle<Self>,
+ ) {
+ }
+}
+
+impl Dispatch<ZxdgOutputV1, OutputData> for OverlayState {
+ fn event(
+ state: &mut Self,
+ _: &ZxdgOutputV1,
+ event: <ZxdgOutputV1 as Proxy>::Event,
+ data: &OutputData,
+ _: &Connection,
+ _: &QueueHandle<Self>,
+ ) {
+ let Some(output) = state.outputs.get_mut(&data.id) else {
+ return;
+ };
+ match event {
+ zxdg_output_v1::Event::LogicalPosition { x, y } => {
+ state.topology_dirty |= output.logical_origin != Some((x, y));
+ output.logical_origin = Some((x, y));
+ }
+ zxdg_output_v1::Event::LogicalSize { width, height } if width > 0 && height > 0 => {
+ state.topology_dirty |= output.logical_size != Some((width as u32, height as u32));
+ output.logical_size = Some((width as u32, height as u32));
+ }
+ zxdg_output_v1::Event::Name { name } => {
+ output.name = Some(name);
}
+ _ => {}
}
}
}
@@ -694,29 +1210,47 @@ impl Dispatch<ZwlrLayerShellV1, ()> for OverlayState {
}
}
-impl Dispatch<ZwlrLayerSurfaceV1, ()> for OverlayState {
+impl Dispatch<ZwlrLayerSurfaceV1, LayerData> for OverlayState {
fn event(
state: &mut Self,
layer: &ZwlrLayerSurfaceV1,
event: <ZwlrLayerSurfaceV1 as wayland_client::Proxy>::Event,
- _: &(),
+ data: &LayerData,
_: &Connection,
_: &QueueHandle<Self>,
) {
- if let zwlr_layer_surface_v1::Event::Configure {
- serial,
- width,
- height,
- } = event
- {
- layer.ack_configure(serial);
- if width > 0 {
- state.output_w = width;
+ match event {
+ zwlr_layer_surface_v1::Event::Configure {
+ serial,
+ width,
+ height,
+ } => {
+ layer.ack_configure(serial);
+ if let Some(output) = state.outputs.get_mut(&data.id) {
+ output.closed = false;
+ let fallback = output.logical_size.or(output.mode_size).unwrap_or((1, 1));
+ let configured_size = (
+ if width > 0 { width } else { fallback.0 },
+ if height > 0 { height } else { fallback.1 },
+ );
+ // Each configure starts a new layer-surface buffer cycle.
+ // Queue exactly one matching clear/paint frame even when
+ // the compositor repeats the same logical size.
+ state.initialized_outputs.remove(&data.id);
+ state.topology_dirty = true;
+ output.configured_size = Some(configured_size);
+ }
}
- if height > 0 {
- state.output_h = height;
+ zwlr_layer_surface_v1::Event::Closed => {
+ state.painted_outputs.remove(&data.id);
+ state.initialized_outputs.remove(&data.id);
+ if let Some(output) = state.outputs.get_mut(&data.id) {
+ output.closed = true;
+ output.close_layer();
+ state.topology_dirty = true;
+ }
}
- state.configured = true;
+ _ => {}
}
}
}
@@ -798,22 +1332,225 @@ mod tests {
core
}
+ fn three_monitor_layout() -> Vec<OutputLayout> {
+ vec![
+ // 1920x1080 logical laptop panel at 2x backing scale.
+ OutputLayout {
+ id: 1,
+ origin_x: 0,
+ origin_y: 0,
+ width: 1920,
+ height: 1080,
+ },
+ // 2560x1440 logical external display, raised above the laptop.
+ OutputLayout {
+ id: 2,
+ origin_x: 1920,
+ origin_y: -200,
+ width: 2560,
+ height: 1440,
+ },
+ // Fractionally scaled portrait display left of the laptop.
+ OutputLayout {
+ id: 3,
+ origin_x: -1280,
+ origin_y: 56,
+ width: 1280,
+ height: 1024,
+ },
+ ]
+ }
+
+ fn initialized(layouts: &[OutputLayout]) -> HashSet<u32> {
+ layouts.iter().map(|layout| layout.id).collect()
+ }
+
+ #[test]
+ fn selects_output_and_converts_global_to_output_local_coordinates() {
+ let layouts = three_monitor_layout();
+ assert_eq!(
+ select_output(&layouts, 2500.0, 100.0),
+ Some(SelectedOutput {
+ id: 2,
+ local_x: 580.0,
+ local_y: 300.0,
+ })
+ );
+ assert_eq!(
+ select_output(&layouts, -1000.0, 256.0),
+ Some(SelectedOutput {
+ id: 3,
+ local_x: 280.0,
+ local_y: 200.0,
+ })
+ );
+ assert_eq!(
+ select_output(&layouts, 1919.0, 500.0).map(|output| output.id),
+ Some(1)
+ );
+ assert_eq!(
+ select_output(&layouts, 1920.0, 500.0).map(|output| output.id),
+ Some(2)
+ );
+ assert_eq!(select_output(&layouts, 5000.0, 5000.0), None);
+ }
+
#[test]
- fn fresh_sentinel_overlay_blocks_without_frame_polling() {
+ fn overlapping_outputs_use_the_same_deterministic_selection_for_compositing() {
+ let layouts = vec![
+ OutputLayout {
+ id: 8,
+ origin_x: 0,
+ origin_y: 0,
+ width: 1920,
+ height: 1080,
+ },
+ OutputLayout {
+ id: 4,
+ origin_x: 0,
+ origin_y: 0,
+ width: 1920,
+ height: 1080,
+ },
+ ];
+ let mut core = positioned_core();
+ core.pos = (400.0, 300.0);
+ let cores = HashMap::from([("session".to_owned(), core)]);
+
+ assert_eq!(select_output(&layouts, 400.0, 300.0).unwrap().id, 4);
+ assert_eq!(visible_cores_for_output(&cores, &layouts, 4).len(), 1);
+ assert!(visible_cores_for_output(&cores, &layouts, 8).is_empty());
+ }
+
+ #[test]
+ fn initial_configured_outputs_each_plan_one_transparent_frame() {
+ let layouts = three_monitor_layout();
+ let (selected, targets) = frame_plan(
+ &layouts,
+ &HashSet::new(),
+ &HashSet::new(),
+ std::iter::empty::<(f64, f64)>(),
+ );
+
+ assert!(selected.is_empty());
+ assert_eq!(
+ targets.iter().map(|target| target.id).collect::<Vec<_>>(),
+ vec![1, 2, 3]
+ );
+ }
+
+ #[test]
+ fn initialized_stable_outputs_plan_no_idle_frame() {
+ let layouts = three_monitor_layout();
+ let (selected, targets) = frame_plan(
+ &layouts,
+ &HashSet::new(),
+ &initialized(&layouts),
+ std::iter::empty::<(f64, f64)>(),
+ );
+
+ assert!(selected.is_empty());
+ assert!(targets.is_empty());
+ }
+
+ #[test]
+ fn hotplugged_output_plans_one_initial_transparent_frame() {
+ let layouts = three_monitor_layout();
+ let already_initialized = HashSet::from([1, 2]);
+ let (selected, targets) = frame_plan(
+ &layouts,
+ &HashSet::new(),
+ &already_initialized,
+ std::iter::empty::<(f64, f64)>(),
+ );
+
+ assert!(selected.is_empty());
+ assert_eq!(targets, vec![FrameTarget { id: 3 }]);
+ }
+
+ #[test]
+ fn selected_output_combines_initialization_and_cursor_paint_in_one_frame() {
+ let layouts = three_monitor_layout();
+ let already_initialized = HashSet::from([1, 3]);
+ let (selected, targets) = frame_plan(
+ &layouts,
+ &HashSet::new(),
+ &already_initialized,
+ Some((2500.0, 100.0)),
+ );
+
+ assert_eq!(selected, HashSet::from([2]));
+ assert_eq!(targets, vec![FrameTarget { id: 2 }]);
+ }
+
+ #[test]
+ fn crossing_outputs_clears_the_old_surface_and_paints_the_new_one() {
+ let layouts = three_monitor_layout();
+ let painted = HashSet::from([1]);
+ let (selected, targets) = frame_plan(
+ &layouts,
+ &painted,
+ &initialized(&layouts),
+ Some((2500.0, 100.0)),
+ );
+
+ assert_eq!(selected, HashSet::from([2]));
+ assert_eq!(targets, vec![FrameTarget { id: 1 }, FrameTarget { id: 2 }]);
+ }
+
+ #[test]
+ fn hide_or_session_removal_clears_every_previously_painted_output() {
+ let layouts = three_monitor_layout();
+ let painted = HashSet::from([1, 2, 3]);
+ let (selected, targets) = frame_plan(
+ &layouts,
+ &painted,
+ &initialized(&layouts),
+ std::iter::empty::<(f64, f64)>(),
+ );
+
+ assert!(selected.is_empty());
+ assert_eq!(targets.len(), 3);
+ assert_eq!(
+ targets.iter().map(|target| target.id).collect::<Vec<_>>(),
+ vec![1, 2, 3]
+ );
+ }
+
+ #[test]
+ fn fresh_sentinel_overlay_uses_only_topology_maintenance() {
+ let layouts = three_monitor_layout();
let core = RenderStateCore::new(CursorConfig::default());
- assert_eq!(next_wait(&core, false), WlWait::Block);
- assert!(!needs_frame_tick(&core));
+ let cores = HashMap::from([("default".to_owned(), core)]);
+ assert_eq!(
+ next_wait(&cores, false, false),
+ WlWait::Maintenance(TOPOLOGY_MAINTENANCE_INTERVAL)
+ );
+ assert!(!any_core_needs_frame_tick(&cores));
+ assert_eq!(
+ next_wait(&cores, false, true),
+ WlWait::Maintenance(Duration::ZERO)
+ );
+ assert!(frame_plan(
+ &layouts,
+ &HashSet::new(),
+ &initialized(&layouts),
+ std::iter::empty::<(f64, f64)>()
+ )
+ .1
+ .is_empty());
}
#[test]
fn stable_visible_overlay_sleeps_until_idle_fade_deadline() {
let mut core = positioned_core();
core.idle_secs = 0.25;
+ let cores = HashMap::from([("cursor-a".to_owned(), core)]);
assert_eq!(
- next_wait(&core, false),
+ next_wait(&cores, false, false),
WlWait::Deadline(Duration::from_millis(750))
);
- assert!(!needs_frame_tick(&core));
+ assert!(!any_core_needs_frame_tick(&cores));
}
#[test]
@@ -821,13 +1558,15 @@ mod tests {
let mut core = positioned_core();
core.click_t = Some(0.0);
assert!(needs_frame_tick(&core));
- assert_eq!(next_wait(&core, false), WlWait::Frame);
+ let mut cores = HashMap::from([("cursor-a".to_owned(), core)]);
+ assert_eq!(next_wait(&cores, false, false), WlWait::Frame);
+ let core = cores.get_mut("cursor-a").unwrap();
core.click_t = None;
core.idle_secs = 1.0;
core.idle_alpha = 1.0;
- assert!(needs_frame_tick(&core));
- assert_eq!(next_wait(&core, false), WlWait::Frame);
+ assert!(needs_frame_tick(core));
+ assert_eq!(next_wait(&cores, false, false), WlWait::Frame);
}
#[test]
@@ -839,22 +1578,189 @@ mod tests {
assert!(core.click_t.is_none());
assert!(core.path.is_none());
assert!(core.spring.is_none());
- assert_eq!(next_wait(&core, false), WlWait::Block);
+ let cores = HashMap::from([("cursor-a".to_owned(), core)]);
+ assert_eq!(
+ next_wait(&cores, false, false),
+ WlWait::Maintenance(TOPOLOGY_MAINTENANCE_INTERVAL)
+ );
+ }
+
+ #[test]
+ fn forward_mapping_preserves_named_cursor_keys() {
+ let snap = message(OverlayCommand::SnapTo {
+ x: 10.0,
+ y: 20.0,
+ heading_radians: None,
+ });
+ assert!(matches!(
+ map_overlay_msg(&snap),
+ Some(WlOverlayCmd::Cmd { key, .. }) if key == "test"
+ ));
+ assert!(matches!(
+ map_overlay_msg(&OverlayMsg::Remove("session-a".to_owned())),
+ Some(WlOverlayCmd::Remove(key)) if key == "session-a"
+ ));
+ assert!(matches!(
+ map_overlay_msg(&OverlayMsg::Revive("session-a".to_owned())),
+ Some(WlOverlayCmd::Revive(key)) if key == "session-a"
+ ));
+ }
+
+ #[test]
+ fn named_cursors_render_on_independent_outputs_and_removal_clears_only_one() {
+ let template = CursorConfig::default();
+ let mut cores = HashMap::new();
+ let mut ended = HashSet::new();
+ assert!(apply_keyed_command(
+ &mut cores,
+ &template,
+ &ended,
+ "session-a".to_owned(),
+ OverlayCommand::SnapTo {
+ x: 100.0,
+ y: 100.0,
+ heading_radians: None,
+ },
+ ));
+ assert!(apply_keyed_command(
+ &mut cores,
+ &template,
+ &ended,
+ "session-b".to_owned(),
+ OverlayCommand::SnapTo {
+ x: 2500.0,
+ y: 100.0,
+ heading_radians: None,
+ },
+ ));
+ assert_eq!(cores["session-a"].cfg.cursor_id, "session-a");
+ assert_eq!(cores["session-b"].cfg.cursor_id, "session-b");
+
+ let layouts = three_monitor_layout();
+ assert_eq!(
+ visible_cores_for_output(&cores, &layouts, 1)
+ .into_iter()
+ .map(|(key, _)| key.as_str())
+ .collect::<Vec<_>>(),
+ vec!["session-a"]
+ );
+ assert_eq!(
+ visible_cores_for_output(&cores, &layouts, 2)
+ .into_iter()
+ .map(|(key, _)| key.as_str())
+ .collect::<Vec<_>>(),
+ vec!["session-b"]
+ );
+ let (painted, targets) = frame_plan(
+ &layouts,
+ &HashSet::new(),
+ &initialized(&layouts),
+ cores.values().map(|core| core.pos),
+ );
+ assert_eq!(painted, HashSet::from([1, 2]));
+ assert_eq!(targets, vec![FrameTarget { id: 1 }, FrameTarget { id: 2 }]);
+
+ assert!(remove_keyed_core(
+ &mut cores,
+ &mut ended,
+ "session-a".to_owned()
+ ));
+ assert!(!cores.contains_key("session-a"));
+ assert!(cores.contains_key("session-b"));
+ assert!(visible_cores_for_output(&cores, &layouts, 1).is_empty());
+ assert_eq!(
+ visible_cores_for_output(&cores, &layouts, 2)
+ .into_iter()
+ .map(|(key, _)| key.as_str())
+ .collect::<Vec<_>>(),
+ vec!["session-b"]
+ );
+ let (selected, targets) = frame_plan(
+ &layouts,
+ &painted,
+ &initialized(&layouts),
+ cores.values().map(|core| core.pos),
+ );
+ assert_eq!(selected, HashSet::from([2]));
+ assert_eq!(targets, vec![FrameTarget { id: 1 }, FrameTarget { id: 2 }]);
+
+ // A queued command cannot resurrect an ended named session.
+ assert!(!apply_keyed_command(
+ &mut cores,
+ &template,
+ &ended,
+ "session-a".to_owned(),
+ OverlayCommand::SnapTo {
+ x: 200.0,
+ y: 200.0,
+ heading_radians: None,
+ },
+ ));
+ assert!(!cores.contains_key("session-a"));
+
+ revive_key(&mut ended, "session-a".to_owned());
+ assert!(!ended.contains("session-a"));
+ assert!(apply_keyed_command(
+ &mut cores,
+ &template,
+ &ended,
+ "session-a".to_owned(),
+ OverlayCommand::SnapTo {
+ x: 200.0,
+ y: 200.0,
+ heading_radians: None,
+ },
+ ));
+ assert!(cores.contains_key("session-a"));
+ }
+
+ #[test]
+ fn named_cursors_schedule_animation_and_idle_deadlines_independently() {
+ let mut idle = positioned_core();
+ idle.idle_secs = 0.25;
+ let mut animated = positioned_core();
+ animated.motion.idle_hide_ms = 4_000.0;
+ animated.click_t = Some(0.0);
+ let mut cores =
+ HashMap::from([("idle".to_owned(), idle), ("animated".to_owned(), animated)]);
+
+ assert_eq!(next_wait(&cores, false, false), WlWait::Frame);
+ cores.get_mut("animated").unwrap().click_t = None;
+ assert_eq!(
+ next_wait(&cores, false, false),
+ WlWait::Deadline(Duration::from_millis(750))
+ );
+ }
+
+ #[test]
+ fn default_cursor_is_not_removed_or_marked_ended() {
+ let mut cores = HashMap::from([(
+ "default".to_owned(),
+ RenderStateCore::new(CursorConfig::default()),
+ )]);
+ let mut ended = HashSet::new();
+ assert!(!remove_keyed_core(
+ &mut cores,
+ &mut ended,
+ "default".to_owned()
+ ));
+ assert!(cores.contains_key("default"));
+ assert!(!ended.contains("default"));
}
#[test]
- fn blocked_scheduler_wakes_on_command_arrival() {
+ fn maintenance_scheduler_wakes_immediately_on_command_arrival() {
let (tx, rx) = bounded(1);
- tx.send(WlOverlayCmd::Remove).unwrap();
+ tx.send(WlOverlayCmd::Remove("test".to_owned())).unwrap();
assert!(matches!(
- wait_for_work(&rx, WlWait::Block),
- WlWake::Command(WlOverlayCmd::Remove)
+ wait_for_work(&rx, WlWait::Maintenance(Duration::from_secs(1))),
+ WlWake::Command(WlOverlayCmd::Remove(key)) if key == "test"
));
}
#[test]
fn disabled_config_refuses_forwarding_and_thread_startup() {
- set_config_enabled(false);
+ CONFIG_ENABLED.store(false, Ordering::Release);
let msg = message(OverlayCommand::SnapTo {
x: 10.0,
y: 20.0,