Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): capture GNOME windows from their own actor (WinRects v10) (#4882)

Francesco Bonacci78212adace95
diff --git a/libs/cua-driver/rust/THIRD_PARTY_NOTICES.md b/libs/cua-driver/rust/THIRD_PARTY_NOTICES.md
index 3037706305..8a111b2610 100644
--- a/libs/cua-driver/rust/THIRD_PARTY_NOTICES.md
+++ b/libs/cua-driver/rust/THIRD_PARTY_NOTICES.md
@@ -11,14 +11,14 @@ In `cursor-overlay`, the motion knobs, their defaults, the arc formula and the
Dubins glide originate in that Swift code; trope-cua is credited below only for
the items listed.
-| Project | Copyright | Used in | What |
-| --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [trope-cua](https://github.com/voctory/trope-cua) | Copyright (c) 2026 Victor Vannara | `crates/platform-windows` (`src/overlay.rs`, `src/input/`, `src/tools/impl_.rs`); `crates/cursor-overlay/src/motion.rs`, `crates/cursor-overlay/src/bezier.rs` (specific items only) | Windows cursor overlay, background input and layered pixel-click dispatch; in `cursor-overlay`, the `press_duration_ms` knob, the 80 ms dwell default, the former 20 s idle default, the `with_overrides` method shape and the cached 32-point arc-length helper |
-| [Interface-Agent](https://github.com/francedot/Interface-Agent) | Copyright (c) 2024 Francesco Bonacci | `crates/platform-windows` (`src/uia/`, `src/win32/`, `src/tools/impl_.rs`) | Windows UI Automation tree walk, app and window enumeration, InvokePattern click and ValuePattern set-value |
-| [yabai](https://github.com/koekeishiya/yabai) | Copyright (c) 2019 Åsmund Vikane | `crates/platform-macos/src/input/skylight.rs` (`activate_without_raise`) | The focus-without-raise SkyLight event-record sequence |
-| [QuickJS-ng](https://github.com/quickjs-ng/quickjs) | Copyright (c) 2017-2026 Fabrice Bellard; (c) 2017-2024 Charlie Gordon; (c) 2023-2026 Ben Noordhuis; (c) 2023-2026 Saúl Ibarra Corretgé | `crates/cua-driver-core/src/script_tool/` (compiled in through rquickjs-sys) | The JavaScript engine behind the experimental `run_script` tool |
-| [rquickjs](https://github.com/DelSkayn/rquickjs) | Copyright (c) 2020 Mees Delzenne; (c) 2025 Rquickjs Contributors | `crates/cua-driver-core/src/script_tool/` | Rust bindings to QuickJS-ng |
-| [T3 Code](https://github.com/pingdotgg/t3code) | Copyright (c) 2026 T3 Tools Inc. | `crates/cua-driver-sdk/src/embedded.rs` (`allowed_environment_name` and its test) | Linux fixes ported from T3 Code's patch to Cua Driver 0.34.0 ([pingdotgg/t3code#16975](https://github.com/pingdotgg/t3code/pull/16975), `patches/cua-driver-linux/0.34.0.patch`): the `CUA_DRIVER_RS_ENABLE_WAYLAND` embedded-host pass-through |
+| Project | Copyright | Used in | What |
+| --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [trope-cua](https://github.com/voctory/trope-cua) | Copyright (c) 2026 Victor Vannara | `crates/platform-windows` (`src/overlay.rs`, `src/input/`, `src/tools/impl_.rs`); `crates/cursor-overlay/src/motion.rs`, `crates/cursor-overlay/src/bezier.rs` (specific items only) | Windows cursor overlay, background input and layered pixel-click dispatch; in `cursor-overlay`, the `press_duration_ms` knob, the 80 ms dwell default, the former 20 s idle default, the `with_overrides` method shape and the cached 32-point arc-length helper |
+| [Interface-Agent](https://github.com/francedot/Interface-Agent) | Copyright (c) 2024 Francesco Bonacci | `crates/platform-windows` (`src/uia/`, `src/win32/`, `src/tools/impl_.rs`) | Windows UI Automation tree walk, app and window enumeration, InvokePattern click and ValuePattern set-value |
+| [yabai](https://github.com/koekeishiya/yabai) | Copyright (c) 2019 Åsmund Vikane | `crates/platform-macos/src/input/skylight.rs` (`activate_without_raise`) | The focus-without-raise SkyLight event-record sequence |
+| [QuickJS-ng](https://github.com/quickjs-ng/quickjs) | Copyright (c) 2017-2026 Fabrice Bellard; (c) 2017-2024 Charlie Gordon; (c) 2023-2026 Ben Noordhuis; (c) 2023-2026 Saúl Ibarra Corretgé | `crates/cua-driver-core/src/script_tool/` (compiled in through rquickjs-sys) | The JavaScript engine behind the experimental `run_script` tool |
+| [rquickjs](https://github.com/DelSkayn/rquickjs) | Copyright (c) 2020 Mees Delzenne; (c) 2025 Rquickjs Contributors | `crates/cua-driver-core/src/script_tool/` | Rust bindings to QuickJS-ng |
+| [T3 Code](https://github.com/pingdotgg/t3code) | Copyright (c) 2026 T3 Tools Inc. | `crates/cua-driver-sdk/src/embedded.rs` (`allowed_environment_name`); `crates/platform-linux/src/wayland/shell_helper.rs` and `shell_helper/transport.rs`; `crates/platform-linux/src/wayland/libei.rs` (input acknowledgement); `crates/platform-linux/src/wayland/mod.rs` and `src/tools/impl_.rs` (GNOME window capture); `../wayland-helper/winrects@cua/` (`CaptureWindow`, `CaptureWindowPreview`) | Linux fixes ported from T3 Code's patch to Cua Driver 0.34.0 ([pingdotgg/t3code#16975](https://github.com/pingdotgg/t3code/pull/16975), `patches/cua-driver-linux/0.34.0.patch`): the `CUA_DRIVER_RS_ENABLE_WAYLAND` embedded-host pass-through, the persistent session-bus transport for the GNOME helper, waiting for EIS to acknowledge libei input, and per-window GNOME capture with compositor-side preview downscaling (helper API 9 and 10) |
## MIT License
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index ed78dd43de..4ad9f79e90 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -1371,7 +1371,17 @@ impl Tool for GetWindowStateTool {
window_overlays(pid, xid)
};
let screenshot = if should_capture {
- let captured = if overlays.covers_window {
+ // A screenshot-only read with a size cap (a live preview) asks
+ // a GNOME helper with API 10 to downscale in the compositor,
+ // and keeps the window's full size for frame_scale.
+ let preview = if !want_tree && max_dim > 0 && crate::wayland::is_wayland() {
+ crate::wayland::shell_helper::screenshot_window_preview(xid, pid, max_dim)
+ } else {
+ None
+ };
+ let captured = if let Some((png, width, height)) = preview {
+ Ok((png, Some((width, height))))
+ } else if overlays.covers_window {
overlays
.window_rect
.ok_or_else(|| anyhow::anyhow!("window geometry unavailable"))
@@ -1379,12 +1389,16 @@ impl Tool for GetWindowStateTool {
crate::capture::screenshot_root_region_png(x, y, w, h)
})
.or_else(|_| crate::wayland::screenshot_dispatch_with_pid(xid, pid))
+ .map(|png| (png, None))
} else {
- crate::wayland::screenshot_dispatch_with_pid(xid, pid)
+ crate::wayland::screenshot_dispatch_with_pid(xid, pid).map(|png| (png, None))
};
match captured {
- Ok(raw) => {
- let (orig_w, orig_h) = crate::capture::png_dimensions_pub(&raw)?;
+ Ok((raw, original_size)) => {
+ let (orig_w, orig_h) = match original_size {
+ Some(size) => size,
+ None => crate::capture::png_dimensions_pub(&raw)?,
+ };
let png = crate::capture::resize_png_if_needed(&raw, max_dim)?;
let (w, h) = crate::capture::png_dimensions_pub(&png)?;
// An explicit per-call max_image_dimension (0 = native)
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
index 0909579d85..22d76aac46 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/mod.rs
@@ -1044,6 +1044,10 @@ fn wayland_window_crop(xid: u64) -> anyhow::Result<WaylandWindowCrop> {
window.visible,
);
}
+ // GNOME helpers before API 9 have no per-window capture, so the stage is
+ // cropped to the window's frame. GetRects proves geometry, not occlusion:
+ // a covering window shows in the crop. API 9+ helpers never reach this
+ // (see `screenshot_dispatch_for_pid`).
if let Some(window) = shell_helper::list_windows(None)
.and_then(|windows| windows.into_iter().find(|window| window.xid == xid))
{
@@ -1106,6 +1110,23 @@ fn screenshot_dispatch_for_pid(xid: u64, pid: Option<u32>) -> anyhow::Result<Vec
)
});
}
+ if is_wayland() {
+ // A GNOME helper with API 9+ snapshots the window's own actor, so a
+ // covering window never shows. It is then the only GNOME route: a
+ // refused capture (minimized window, locked screen) must not fall
+ // back to a stage crop that could show another app.
+ match shell_helper::screenshot_window(xid, pid) {
+ shell_helper::WindowCapture::Png(png) => return Ok(png),
+ shell_helper::WindowCapture::Unavailable => {
+ return Err(surface_identity_unproven(
+ xid,
+ "the GNOME helper could not capture this window (minimized, closed, \
+ or the screen is locked)",
+ ))
+ }
+ shell_helper::WindowCapture::Unsupported => {}
+ }
+ }
screenshot_window_bytes_with_dispatch(
is_wayland(),
xid,
@@ -3490,6 +3511,11 @@ pub fn list_windows_dispatch(filter_pid: Option<u32>) -> Vec<WindowInfo> {
}
};
}
+ // A verified GNOME helper already supplies exact IDs and ownership.
+ // Do not probe unrelated compositor protocols for every preview frame.
+ if let Some(ws) = shell_helper::list_windows(filter_pid) {
+ return listed_windows(ws);
+ }
if let Some(ws) = kwin_helper::list_window_infos() {
return listed_windows(apply_pid_filter(ws, filter_pid));
}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs
index 0f055c7d98..c7a888a748 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs
@@ -15,6 +15,9 @@
//! cursor as a Clutter actor on the compositor stage.
//! - `SetCursorState(...)` / `SetCursorColor(...)` — keep the compositor cursor
//! aligned with the shared semantic theme and active session identity.
+//! - `CaptureWindow(id,pid)` (API 9) / `CaptureWindowPreview(id,pid,max)`
+//! (API 10) — snapshot one window's own actor, so a covering window never
+//! shows; the preview variant downscales on the GPU first.
//!
//! Everything here is **best-effort**: if the extension isn't installed/enabled
//! the calls return `None` / no-op and callers keep the prior behaviour (no
@@ -35,6 +38,10 @@ const DBUS_PATH: &str = "/org/freedesktop/DBus";
const DBUS_IFACE: &str = "org.freedesktop.DBus";
const BROWSER_HELPER_API_VERSION: u32 = 4;
const SEMANTIC_CURSOR_API_VERSION: u32 = 8;
+/// `CaptureWindow(id, pid)`: per-window actor capture.
+pub const WINDOW_CAPTURE_API_VERSION: u32 = 9;
+/// `CaptureWindowPreview(id, pid, max_dimension)`: GPU-downscaled capture.
+pub const WINDOW_PREVIEW_API_VERSION: u32 = 10;
#[derive(Debug, Clone)]
struct ShellWindow {
@@ -186,6 +193,109 @@ fn is_trusted_gnome_shell(pid: u32) -> bool {
.is_some_and(|meta| meta.uid() == 0 && meta.permissions().mode() & 0o022 == 0)
}
+/// The outcome of a per-window capture through the GNOME helper.
+pub enum WindowCapture {
+ /// No verified helper, or one older than API 9: use another route.
+ Unsupported,
+ /// The helper can capture windows but refused this one (minimized,
+ /// closed, wrong owner, locked screen). Do not fall back to a stage crop.
+ Unavailable,
+ Png(Vec<u8>),
+}
+
+/// The verified helper owner and its API version.
+fn attested_owner_with_version() -> Option<(String, u32)> {
+ let owner = shell_owner_with_min_version(None)?;
+ let raw = gdbus_call_to(
+ &owner,
+ PATH,
+ &format!("{IFACE}.GetVersion"),
+ &[],
+ Duration::from_millis(800),
+ )?;
+ Some((owner, parse_first_u32(&raw)?))
+}
+
+/// Capture one window's compositor actor without activating it. Covered
+/// windows keep their own content, so no stage crop is involved.
+pub fn screenshot_window(id: u64, pid: Option<u32>) -> WindowCapture {
+ let Some((owner, version)) = attested_owner_with_version() else {
+ return WindowCapture::Unsupported;
+ };
+ if version < WINDOW_CAPTURE_API_VERSION {
+ return WindowCapture::Unsupported;
+ }
+ let Ok(id) = u32::try_from(id) else {
+ return WindowCapture::Unavailable;
+ };
+ gdbus_call_to(
+ &owner,
+ PATH,
+ &format!("{IFACE}.CaptureWindow"),
+ &[id.to_string(), pid.unwrap_or(0).to_string()],
+ Duration::from_secs(5),
+ )
+ .and_then(|raw| decode_capture(&raw))
+ .map_or(WindowCapture::Unavailable, WindowCapture::Png)
+}
+
+/// Preview-sized pixels of one window, downscaled in the compositor, plus the
+/// window's full-size dimensions. `None` when the helper is older than API 10
+/// or refuses; callers then take the full-size capture path.
+pub fn screenshot_window_preview(
+ id: u64,
+ pid: u32,
+ max_dimension: u32,
+) -> Option<(Vec<u8>, u32, u32)> {
+ let id = u32::try_from(id).ok()?;
+ let owner = shell_owner_with_min_version(Some(WINDOW_PREVIEW_API_VERSION))?;
+ let raw = gdbus_call_to(
+ &owner,
+ PATH,
+ &format!("{IFACE}.CaptureWindowPreview"),
+ &[id.to_string(), pid.to_string(), max_dimension.to_string()],
+ Duration::from_secs(5),
+ )?;
+ let png = strip_preview_timestamp(decode_capture(&raw)?)?;
+ let (width, height) = parse_preview_dimensions(&raw)?;
+ Some((png, width, height))
+}
+
+/// The `uint32 W, uint32 H` after the PNG in a `CaptureWindowPreview` reply.
+fn parse_preview_dimensions(raw: &str) -> Option<(u32, u32)> {
+ let tail = &raw[raw.rfind('\'')? + 1..];
+ let mut values = tail.split("uint32").skip(1).map(|part| {
+ part.trim_matches(|c: char| !c.is_ascii_digit())
+ .parse::<u32>()
+ });
+ let width = values.next()?.ok()?;
+ let height = values.next()?.ok()?;
+ (values.next().is_none() && width > 0 && height > 0).then_some((width, height))
+}
+
+/// GNOME embeds a wall-clock `Creation Time` text chunk even when the pixels
+/// are identical. Remove only that chunk, so a host can drop unchanged
+/// preview frames by comparing bytes, without decoding pixels.
+fn strip_preview_timestamp(mut png: Vec<u8>) -> Option<Vec<u8>> {
+ if !png.starts_with(b"\x89PNG\r\n\x1a\n") {
+ return None;
+ }
+ let mut offset = 8usize;
+ loop {
+ let header = png.get(offset..offset.checked_add(8)?)?;
+ let size = u32::from_be_bytes(header[..4].try_into().ok()?) as usize;
+ let end = offset.checked_add(12)?.checked_add(size)?;
+ let chunk = png.get(offset..end)?;
+ if &chunk[4..8] == b"tEXt" && chunk[8..8 + size].starts_with(b"Creation Time\0") {
+ png.drain(offset..end);
+ } else if &chunk[4..8] == b"IEND" {
+ return (size == 0 && end == png.len()).then_some(png);
+ } else {
+ offset = end;
+ }
+ }
+}
+
/// Capture the GNOME stage through the compositor helper.
///
/// Mutter does not expose wlroots screencopy protocols, and its one-shot
@@ -499,6 +609,41 @@ pub fn hide_cursor() {
mod tests {
use super::*;
+ #[test]
+ fn preview_timestamp_does_not_change_pixels_or_frame_identity() {
+ use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
+ let first = B64.decode("iVBORw0KGgoAAAANSUhEUgAAAAIAAAACCAYAAABytg0kAAAAE3RFWHRDcmVhdGlvbiBUaW1lAGZpcnN0muU4QwAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AAAAUSURBVHicY5SLqvjPwMDAwMQABQAczgHzr55lzgAAAABJRU5ErkJggg==").unwrap();
+ let second = B64.decode("iVBORw0KGgoAAAANSUhEUgAAAAIAAAACCAYAAABytg0kAAAAFHRFWHRDcmVhdGlvbiBUaW1lAHNlY29uZKzNUcIAAAAZdEVYdFNvZnR3YXJlAGdub21lLXNjcmVlbnNob3TvA78+AAAAFElEQVR4nGOUi6r4z8DAwMDEAAUAHM4B86+eZc4AAAAASUVORK5CYII=").unwrap();
+ let stable = strip_preview_timestamp(first.clone()).unwrap();
+ assert_eq!(stable, strip_preview_timestamp(second).unwrap());
+ assert_eq!(
+ image::load_from_memory(&stable).unwrap().to_rgba8(),
+ image::load_from_memory(&first).unwrap().to_rgba8()
+ );
+ assert!(stable.windows(8).any(|bytes| bytes == b"Software"));
+ assert_eq!(Some(stable.clone()), strip_preview_timestamp(stable));
+ assert!(strip_preview_timestamp(first[..first.len() - 1].to_vec()).is_none());
+ let mut malformed = first;
+ malformed[8..12].copy_from_slice(&u32::MAX.to_be_bytes());
+ assert!(strip_preview_timestamp(malformed).is_none());
+ }
+
+ #[test]
+ fn preview_reply_carries_full_size_dimensions() {
+ let raw = "('cG5n', uint32 1600, uint32 900)";
+ assert_eq!(decode_capture(raw), Some(b"png".to_vec()));
+ assert_eq!(parse_preview_dimensions(raw), Some((1600, 900)));
+ assert_eq!(
+ parse_preview_dimensions("('cG5n', uint32 0, uint32 900)"),
+ None
+ );
+ assert_eq!(parse_preview_dimensions("('cG5n',)"), None);
+ assert_eq!(
+ parse_preview_dimensions("('cG5n', uint32 1, uint32 2, uint32 3)"),
+ None
+ );
+ }
+
const EXTENSION_SOURCE: &str =
include_str!("../../../../../wayland-helper/winrects@cua/extension.js");
const EXTENSION_METADATA: &str =
@@ -565,9 +710,9 @@ mod tests {
}
#[test]
- fn bundled_helper_v8_uses_host_owned_modifier_badge_chips() {
+ fn bundled_helper_uses_host_owned_modifier_badge_chips() {
assert!(EXTENSION_SOURCE.contains("GetVersion()"));
- assert!(EXTENSION_SOURCE.contains("return 8;"));
+ assert!(EXTENSION_SOURCE.contains("return 10;"));
assert!(EXTENSION_SOURCE.contains("SetCursorState"));
assert!(EXTENSION_SOURCE.contains("SetCursorColor"));
assert!(EXTENSION_SOURCE.contains("SetSessionLabel"));
@@ -592,7 +737,7 @@ mod tests {
assert!(!EXTENSION_SOURCE.contains("function drawModifiers"));
let metadata: serde_json::Value =
serde_json::from_str(EXTENSION_METADATA).expect("valid bundled helper metadata");
- assert_eq!(metadata["version"], 8);
+ assert_eq!(metadata["version"], 10);
for action in [
"idle", "observe", "click", "drag", "scroll", "text", "key", "navigate", "app",
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper/transport.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper/transport.rs
index 4c1988bff0..e6e6590ef0 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper/transport.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper/transport.rs
@@ -111,6 +111,10 @@ async fn invoke(
("MoveCursor" | "ClickPulse", [x, y]) => {
send!((parse_i32(x)?, parse_i32(y)?))
}
+ ("CaptureWindow", [id, pid]) => send!((parse_u32(id)?, parse_u32(pid)?)),
+ ("CaptureWindowPreview", [id, pid, max]) => {
+ send!((parse_u32(id)?, parse_u32(pid)?, parse_u32(max)?))
+ }
("SetCursorState", [action, delivery, target, active]) => send!((
action.as_str(),
delivery.as_str(),
@@ -122,7 +126,7 @@ async fn invoke(
let body = reply.body();
let decoded = match member {
"GetRects" => body.deserialize::<String>().ok(),
- "GetNameOwner" | "Capture" => body
+ "GetNameOwner" | "Capture" | "CaptureWindow" => body
.deserialize::<String>()
.ok()
.map(|value| format!("('{value}',)")),
@@ -130,6 +134,10 @@ async fn invoke(
.deserialize::<u32>()
.ok()
.map(|value| format!("(uint32 {value},)")),
+ "CaptureWindowPreview" => body
+ .deserialize::<(String, u32, u32)>()
+ .ok()
+ .map(|(png, width, height)| format!("('{png}', uint32 {width}, uint32 {height})")),
"Activate" => body
.deserialize::<bool>()
.ok()
@@ -161,7 +169,7 @@ mod tests {
#[zbus::interface(name = "org.cua.WinRects")]
impl Helper {
fn get_version(&self) -> u32 {
- 8
+ 10
}
fn get_rects(&self) -> String {
RECTS.into()
@@ -169,6 +177,25 @@ mod tests {
fn capture(&self) -> String {
"cG5n".into()
}
+ fn capture_window(&self, id: u32, pid: u32) -> zbus::fdo::Result<String> {
+ if id != 7 || pid != 42 {
+ return Err(zbus::fdo::Error::Failed(
+ "Target window is unavailable".into(),
+ ));
+ }
+ Ok("cG5n".into())
+ }
+ fn capture_window_preview(
+ &self,
+ id: u32,
+ pid: u32,
+ max: u32,
+ ) -> zbus::fdo::Result<(String, u32, u32)> {
+ if id != 7 || pid != 42 || max != 800 {
+ return Err(zbus::fdo::Error::InvalidArgs("wrong target".into()));
+ }
+ Ok(("cG5n".into(), 1600, 900))
+ }
fn activate(&self, id: u32) -> bool {
id == 7
}
@@ -252,7 +279,7 @@ mod tests {
for _ in 0..3 {
assert_eq!(
call("GetVersion", vec![]).await.as_deref(),
- Ok("(uint32 8,)")
+ Ok("(uint32 10,)")
);
// GetRects is the raw JSON, so punctuation in one title no longer
// hides every window from the parsers.
@@ -269,6 +296,26 @@ mod tests {
assert_eq!(raw, "('cG5n',)");
assert_eq!(super::super::decode_capture(&raw), Some(b"png".to_vec()));
}
+ assert_eq!(
+ call("CaptureWindow", vec!["7".into(), "42".into()])
+ .await
+ .as_deref(),
+ Ok("('cG5n',)")
+ );
+ assert_eq!(
+ call(
+ "CaptureWindowPreview",
+ vec!["7".into(), "42".into(), "800".into()]
+ )
+ .await
+ .as_deref(),
+ Ok("('cG5n', uint32 1600, uint32 900)")
+ );
+ // The helper refusing another process's window is an error reply.
+ assert_eq!(
+ call("CaptureWindow", vec!["7".into(), "43".into()]).await,
+ Err(Failure::Reply)
+ );
assert_eq!(
call("Activate", vec!["7".into()]).await.as_deref(),
Ok("(true,)")
@@ -303,7 +350,7 @@ mod tests {
assert_eq!(call("NoSuchMethod", vec![]).await, Err(Failure::Reply));
assert_eq!(
call("GetVersion", vec![]).await.as_deref(),
- Ok("(uint32 8,)")
+ Ok("(uint32 10,)")
);
drop(service);
}
diff --git a/libs/cua-driver/scripts/_install-local-rust.sh b/libs/cua-driver/scripts/_install-local-rust.sh
index f64c3af753..e4b3384048 100755
--- a/libs/cua-driver/scripts/_install-local-rust.sh
+++ b/libs/cua-driver/scripts/_install-local-rust.sh
@@ -314,6 +314,11 @@ if [ -d "$SOURCE_SKILLS" ]; then
echo "${GREEN}staged skill pack at $STAGED_SKILLS${NORMAL}"
fi
+# The integer "version" of a GNOME helper metadata.json, or nothing.
+helper_metadata_version() {
+ sed -n 's/.*"version"[[:space:]]*:[[:space:]]*\([0-9][0-9]*\).*/\1/p' "$1" 2>/dev/null | head -n 1 || true
+}
+
# Keep an already-installed GNOME helper aligned with the source-built driver.
# Installing the helper is still opt-in. Once present, however, leaving old
# compositor artwork behind after install-local creates a misleading
@@ -327,10 +332,18 @@ if [ "$OS" = "Linux" ]; then
INSTALLED_WAYLAND_HELPER="${XDG_DATA_HOME:-$HOME/.local/share}/gnome-shell/extensions/winrects@cua"
if [ -d "$INSTALLED_WAYLAND_HELPER" ]; then
- cp "$SOURCE_WAYLAND_HELPER/winrects@cua/metadata.json" \
- "$SOURCE_WAYLAND_HELPER/winrects@cua/extension.js" \
- "$INSTALLED_WAYLAND_HELPER/"
- echo "${GREEN}updated installed GNOME helper; reload the GNOME session to activate it${NORMAL}"
+ # The helper's API is additive and shared with other apps that
+ # bundle it, so never replace a newer installed helper.
+ BUNDLED_HELPER_VERSION=$(helper_metadata_version "$SOURCE_WAYLAND_HELPER/winrects@cua/metadata.json")
+ INSTALLED_HELPER_VERSION=$(helper_metadata_version "$INSTALLED_WAYLAND_HELPER/metadata.json")
+ if [ "${INSTALLED_HELPER_VERSION:-0}" -gt "${BUNDLED_HELPER_VERSION:-0}" ]; then
+ echo "kept installed GNOME helper v$INSTALLED_HELPER_VERSION (newer than bundled v${BUNDLED_HELPER_VERSION:-?})"
+ else
+ cp "$SOURCE_WAYLAND_HELPER/winrects@cua/metadata.json" \
+ "$SOURCE_WAYLAND_HELPER/winrects@cua/extension.js" \
+ "$INSTALLED_WAYLAND_HELPER/"
+ echo "${GREEN}updated installed GNOME helper; reload the GNOME session to activate it${NORMAL}"
+ fi
fi
fi
fi
diff --git a/libs/cua-driver/scripts/_install-rust.sh b/libs/cua-driver/scripts/_install-rust.sh
index 8426310a7f..6a7c5ef664 100644
--- a/libs/cua-driver/scripts/_install-rust.sh
+++ b/libs/cua-driver/scripts/_install-rust.sh
@@ -189,6 +189,11 @@ TMP_DIR=$(mktemp -d)
log() { printf '==> %s\n' "$*"; }
err() { printf 'error: %s\n' "$*" >&2; }
+# The integer "version" of a GNOME helper metadata.json, or nothing.
+helper_metadata_version() {
+ sed -n 's/.*"version"[[:space:]]*:[[:space:]]*\([0-9][0-9]*\).*/\1/p' "$1" 2>/dev/null | head -n 1 || true
+}
+
# Return the source form of an app's designated code-signing requirement.
macos_designated_requirement() {
codesign -d -r- "$1" 2>/dev/null \
@@ -1423,10 +1428,18 @@ else
INSTALLED_WAYLAND_HELPER="${XDG_DATA_HOME:-$HOME/.local/share}/gnome-shell/extensions/winrects@cua"
if [[ -d "$INSTALLED_WAYLAND_HELPER" ]]; then
- cp "$SRC_WAYLAND_HELPER/winrects@cua/metadata.json" \
- "$SRC_WAYLAND_HELPER/winrects@cua/extension.js" \
- "$INSTALLED_WAYLAND_HELPER/"
- log "updated installed GNOME helper; reload the GNOME session to activate it"
+ # The helper's API is additive and shared with other apps that
+ # bundle it, so never replace a newer installed helper.
+ BUNDLED_HELPER_VERSION=$(helper_metadata_version "$SRC_WAYLAND_HELPER/winrects@cua/metadata.json")
+ INSTALLED_HELPER_VERSION=$(helper_metadata_version "$INSTALLED_WAYLAND_HELPER/metadata.json")
+ if (( ${INSTALLED_HELPER_VERSION:-0} > ${BUNDLED_HELPER_VERSION:-0} )); then
+ log "kept installed GNOME helper v$INSTALLED_HELPER_VERSION (newer than bundled v${BUNDLED_HELPER_VERSION:-?})"
+ else
+ cp "$SRC_WAYLAND_HELPER/winrects@cua/metadata.json" \
+ "$SRC_WAYLAND_HELPER/winrects@cua/extension.js" \
+ "$INSTALLED_WAYLAND_HELPER/"
+ log "updated installed GNOME helper; reload the GNOME session to activate it"
+ fi
fi
fi
log "installed $VERSIONED_DIR/$BINARY_NAME (version $VERSION, target $TARGET)"
diff --git a/libs/cua-driver/scripts/tests/test_install_local.py b/libs/cua-driver/scripts/tests/test_install_local.py
index bc5505d172..ee1f214b03 100644
--- a/libs/cua-driver/scripts/tests/test_install_local.py
+++ b/libs/cua-driver/scripts/tests/test_install_local.py
@@ -121,9 +121,10 @@ def test_explicit_local_signing_identity_never_falls_back(tmp_path: Path) -> Non
assert result.stdout == "-"
[email protected]("installed_helper_version", [4, 12], ids=["older-helper", "newer-helper"])
@pytest.mark.parametrize("relative_target", [False, True], ids=["absolute", "relative"])
def test_installer_stages_binary_from_custom_cargo_target(
- tmp_path: Path, relative_target: bool
+ tmp_path: Path, relative_target: bool, installed_helper_version: int
) -> None:
fixture_root = tmp_path / "cua-driver"
scripts_dir = fixture_root / "scripts"
@@ -177,7 +178,9 @@ esac
user_home = tmp_path / "home"
installed_helper = user_home / ".local/share/gnome-shell/extensions/winrects@cua"
installed_helper.mkdir(parents=True)
- (installed_helper / "metadata.json").write_text('{"version":4}\n', encoding="utf-8")
+ (installed_helper / "metadata.json").write_text(
+ f'{{"version":{installed_helper_version}}}\n', encoding="utf-8"
+ )
(installed_helper / "extension.js").write_text("// legacy cursor\n", encoding="utf-8")
install_bin = tmp_path / "install-bin"
env = os.environ.copy()
@@ -217,8 +220,17 @@ esac
assert (
local_home / "packages/current/wayland-helper/winrects@cua/metadata.json"
).read_text() == '{"version":5}\n'
- assert (installed_helper / "metadata.json").read_text() == '{"version":5}\n'
- assert (installed_helper / "extension.js").read_text() == "// semantic cursor v5\n"
+ if installed_helper_version < 5:
+ assert (installed_helper / "metadata.json").read_text() == '{"version":5}\n'
+ assert (installed_helper / "extension.js").read_text() == "// semantic cursor v5\n"
+ else:
+ # Another app installed a newer helper with the same UUID; its API is a
+ # superset, so the installer must not downgrade it.
+ assert (
+ installed_helper / "metadata.json"
+ ).read_text() == f'{{"version":{installed_helper_version}}}\n'
+ assert (installed_helper / "extension.js").read_text() == "// legacy cursor\n"
+ assert f"kept installed GNOME helper v{installed_helper_version}" in output
# No published CLI exists here, so the installer explains the migration
# after the shared hints instead of aliasing the local build to it.
diff --git a/libs/cua-driver/scripts/tests/test_wayland_helper_install.py b/libs/cua-driver/scripts/tests/test_wayland_helper_install.py
new file mode 100644
index 0000000000..15c5f5ea11
--- /dev/null
+++ b/libs/cua-driver/scripts/tests/test_wayland_helper_install.py
@@ -0,0 +1,73 @@
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+from pathlib import Path
+
+import pytest
+
+HELPER_DIR = Path(__file__).resolve().parents[2] / "wayland-helper"
+INSTALL = HELPER_DIR / "install.sh"
+BUNDLED_VERSION = json.loads((HELPER_DIR / "winrects@cua/metadata.json").read_text())["version"]
+
+
+def _run(tmp_path: Path, *args: str) -> subprocess.CompletedProcess[str]:
+ fake_bin = tmp_path / "bin"
+ fake_bin.mkdir(exist_ok=True)
+ gsettings = fake_bin / "gsettings"
+ gsettings.write_text("#!/bin/sh\n[ \"$1\" = get ] && echo \"@as []\"\nexit 0\n")
+ gsettings.chmod(0o755)
+ env = os.environ.copy()
+ env.update({"HOME": str(tmp_path / "home"), "PATH": f"{fake_bin}:/usr/bin:/bin"})
+ env.pop("XDG_DATA_HOME", None)
+ return subprocess.run(
+ ["/bin/bash", str(INSTALL), *args],
+ env=env,
+ text=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ check=False,
+ )
+
+
+def _installed(tmp_path: Path) -> Path:
+ return tmp_path / "home/.local/share/gnome-shell/extensions/winrects@cua"
+
+
+def _preinstall(tmp_path: Path, version: int) -> None:
+ dest = _installed(tmp_path)
+ dest.mkdir(parents=True)
+ (dest / "metadata.json").write_text(f'{{"uuid": "winrects@cua", "version": {version}}}\n')
+ (dest / "extension.js").write_text(f"// helper v{version}\n")
+
+
+def test_fresh_install_copies_the_bundled_helper(tmp_path: Path) -> None:
+ result = _run(tmp_path)
+ assert result.returncode == 0, result.stdout
+ installed = json.loads((_installed(tmp_path) / "metadata.json").read_text())
+ assert installed["version"] == BUNDLED_VERSION
+
+
[email protected]("delta", [-1, 0], ids=["older", "same"])
+def test_older_or_same_helper_is_replaced(tmp_path: Path, delta: int) -> None:
+ _preinstall(tmp_path, BUNDLED_VERSION + delta)
+ result = _run(tmp_path)
+ assert result.returncode == 0, result.stdout
+ assert (_installed(tmp_path) / "extension.js").read_text() == (
+ HELPER_DIR / "winrects@cua/extension.js"
+ ).read_text()
+
+
+def test_newer_helper_from_another_app_is_kept(tmp_path: Path) -> None:
+ newer = BUNDLED_VERSION + 1
+ _preinstall(tmp_path, newer)
+ result = _run(tmp_path)
+ assert result.returncode == 0, result.stdout
+ assert f"Keeping the installed winrects@cua v{newer}" in result.stdout
+ assert (_installed(tmp_path) / "extension.js").read_text() == f"// helper v{newer}\n"
+
+ forced = _run(tmp_path, "--force")
+ assert forced.returncode == 0, forced.stdout
+ installed = json.loads((_installed(tmp_path) / "metadata.json").read_text())
+ assert installed["version"] == BUNDLED_VERSION
diff --git a/libs/cua-driver/wayland-helper/README.md b/libs/cua-driver/wayland-helper/README.md
index ced12a5259..460c8039e2 100644
--- a/libs/cua-driver/wayland-helper/README.md
+++ b/libs/cua-driver/wayland-helper/README.md
@@ -31,6 +31,18 @@ It exposes `org.cua.WinRects` on the session bus:
- `SetCursorColor(fill_color)`: apply the stable per-session fill selected by
cua-driver. The helper validates the `#RRGGBB` value, updates the matching
glow, and keeps a white pointer outline.
+- `CaptureWindow(id,pid) -> png_base64` (API 9): snapshot one window's own
+ compositor actor (`paint_to_content`), not the stage, so a covering window
+ never shows and the target is not activated. `pid` 0 skips the owner check.
+ A 1x1 transparent clone keeps Mutter painting a fully covered window and
+ expires 4 s after the last capture. Refused while the screen is locked, for
+ minimized windows, and above 32 MiB.
+- `CaptureWindowPreview(id,pid,max_dimension) -> (png_base64, width, height)`
+ (API 10): the same capture, downscaled on the GPU so its long edge is at
+ most `max_dimension` (1 to 4096). `width`/`height` are the window's
+ full-size pixels, which cua-driver reports as the original size. cua-driver
+ removes GNOME's per-capture `Creation Time` PNG chunk, so identical frames
+ are byte-identical.
It runs in the shell's privileged context, so **no xdg-desktop-portal grant** is
needed (unlike libei/RemoteDesktop).
@@ -49,6 +61,12 @@ still work when it is absent, but pixel geometry, the Shell cursor, and safe
foreground portal input are unavailable. cua-driver refuses focus-bound input
instead of injecting into an unverified target.
+With helper API 9 or newer, cua-driver captures GNOME windows only through
+`CaptureWindow`; if the helper refuses a window, the screenshot fails rather
+than falling back to a stage crop that could show another app. With an older
+helper it still crops the stage to the window frame, which shows whatever
+covers the window. Re-run the installer to get per-window capture.
+
The semantic cursor requires helper v8. When an older helper is still loaded,
cua-driver does not draw its legacy cursor. Re-run the helper installer, then
reload the GNOME session so the new compositor-owned artwork becomes active.
@@ -60,6 +78,42 @@ the public name between verification and an activation request. One exact
target is activated only for the bounded operation, then the previously
focused Shell window is restored and verified.
+## Versions and other apps that bundle the helper
+
+The extension is identified by its UUID, `winrects@cua`, and serves the
+session-bus name `org.cua.WinRects`. Other apps ship the same helper: T3 Code
+bundles it with its Linux build of Cua Driver. Keeping one UUID is deliberate.
+Two copies under different UUIDs would both be enabled and race for the same
+bus name and object path, and which one answered would be undefined. With one
+UUID there is exactly one installed copy per user, and the rules below let
+any copy serve every client:
+
+- `GetVersion()` returns an integer API version, and `metadata.json`
+ `"version"` carries the same number.
+- API versions only add methods. A method never changes or disappears in a
+ later version, so a newer helper serves every older client. Clients gate
+ each feature on `GetVersion()`.
+- A version number names one method set everywhere. Versions 9 and 10 were
+ first defined in T3 Code's Linux patch to Cua Driver 0.34.0
+ ([pingdotgg/t3code#16975](https://github.com/pingdotgg/t3code/pull/16975))
+ and are adopted here unchanged, so T3 Code's v10 and this v10 are
+ interchangeable. Coordinate any new number with apps that bundle the helper.
+- Installers never downgrade. `install.sh` and the cua-driver installers keep
+ an installed helper whose version is newer than the bundled one, and say so.
+ `install.sh --force` replaces it anyway.
+
+| Version | Adds |
+| ------- | ------------------------------------------------------------------ |
+| 8 | The semantic `SetCursorState` contract; shipped by Cua Driver 0.34 |
+| 9 | `CaptureWindow(id,pid)` |
+| 10 | `CaptureWindowPreview(id,pid,max_dimension)` |
+
+Cua Driver 0.34 and earlier bundle v8 or older, and their installers overwrite
+an installed helper unconditionally. Running one of those installers after an app
+installed v10 downgrades the helper until the newer app reinstalls it.
+
+## Other compositors
+
wlroots compositors such as Sway and labwc do not need it: cua-driver uses
foreign-toplevel activation, virtual-pointer input, and layer-shell there.
diff --git a/libs/cua-driver/wayland-helper/install.sh b/libs/cua-driver/wayland-helper/install.sh
index e4a61ff5e4..63c3e9149d 100755
--- a/libs/cua-driver/wayland-helper/install.sh
+++ b/libs/cua-driver/wayland-helper/install.sh
@@ -3,12 +3,28 @@
# geometry (for AT-SPI coordinate reconstruction) and renders the agent cursor
# on GNOME Mutter Wayland, where a normal client can do neither. Best-effort:
# cua-driver works without it (just no screen coords / no cursor on Mutter).
+#
+# Other apps bundle the same helper (same UUID, shared additive API versions),
+# so a newer installed copy is kept. Pass --force to install this one anyway.
set -euo pipefail
UUID="winrects@cua"
SRC="$(cd "$(dirname "$0")" && pwd)/$UUID"
DEST="${XDG_DATA_HOME:-$HOME/.local/share}/gnome-shell/extensions/$UUID"
-mkdir -p "$DEST"
-cp -f "$SRC/metadata.json" "$SRC/extension.js" "$DEST/"
+FORCE=0
+if [ "${1:-}" = "--force" ]; then FORCE=1; fi
+metadata_version() {
+ sed -n 's/.*"version"[[:space:]]*:[[:space:]]*\([0-9][0-9]*\).*/\1/p' "$1" 2>/dev/null | head -n 1 || true
+}
+BUNDLED=$(metadata_version "$SRC/metadata.json")
+INSTALLED=$(metadata_version "$DEST/metadata.json")
+if [ "$FORCE" != 1 ] && [ "${INSTALLED:-0}" -gt "${BUNDLED:-0}" ]; then
+ echo "Keeping the installed $UUID v$INSTALLED: it is newer than this bundle's v${BUNDLED:-?}."
+ echo "Its API includes everything this cua-driver uses. Pass --force to replace it."
+else
+ mkdir -p "$DEST"
+ cp -f "$SRC/metadata.json" "$SRC/extension.js" "$DEST/"
+ echo "Installed $UUID v${BUNDLED:-?} to $DEST."
+fi
# Add to the enabled set (preserves existing).
cur=$(gsettings get org.gnome.shell enabled-extensions 2>/dev/null || echo "@as []")
python3 - "$cur" "$UUID" <<'PY'
@@ -20,6 +36,5 @@ import subprocess
subprocess.run(["gsettings","set","org.gnome.shell","enabled-extensions",str(l)])
print("enabled-extensions ->", l)
PY
-echo "Installed $UUID to $DEST."
echo "GNOME Shell scans extensions only at startup, so log out/in (or restart the"
echo "session) ONCE to load it. After that: gnome-extensions info $UUID should show State: ACTIVE."
diff --git a/libs/cua-driver/wayland-helper/winrects@cua/extension.js b/libs/cua-driver/wayland-helper/winrects@cua/extension.js
index 743acc2f5c..cbc28880e8 100644
--- a/libs/cua-driver/wayland-helper/winrects@cua/extension.js
+++ b/libs/cua-driver/wayland-helper/winrects@cua/extension.js
@@ -3,16 +3,20 @@ import GLib from 'gi://GLib';
import Shell from 'gi://Shell';
import St from 'gi://St';
import Clutter from 'gi://Clutter';
+import Cogl from 'gi://Cogl';
import Cairo from 'cairo';
import * as Main from 'resource:///org/gnome/shell/ui/main.js';
import { Extension } from 'resource:///org/gnome/shell/extensions/extension.js';
Gio._promisify(Shell.Screenshot.prototype, 'screenshot');
+Gio._promisify(Shell.Screenshot, 'composite_to_stream');
const IFACE = `<node><interface name="org.cua.WinRects">
<method name="GetVersion"><arg type="u" direction="out" name="version"/></method>
<method name="GetRects"><arg type="s" direction="out" name="json"/></method>
<method name="Capture"><arg type="s" direction="out" name="png_base64"/></method>
+<method name="CaptureWindow"><arg type="u" direction="in" name="id"/><arg type="u" direction="in" name="pid"/><arg type="s" direction="out" name="png_base64"/></method>
+<method name="CaptureWindowPreview"><arg type="u" direction="in" name="id"/><arg type="u" direction="in" name="pid"/><arg type="u" direction="in" name="max_dimension"/><arg type="s" direction="out" name="png_base64"/><arg type="u" direction="out" name="width"/><arg type="u" direction="out" name="height"/></method>
<method name="Activate"><arg type="u" direction="in" name="id"/><arg type="b" direction="out" name="activated"/></method>
<method name="MoveCursor"><arg type="i" direction="in" name="x"/><arg type="i" direction="in" name="y"/></method>
<method name="ClickPulse"><arg type="i" direction="in" name="x"/><arg type="i" direction="in" name="y"/></method>
@@ -680,6 +684,7 @@ function drawBadgeChip(cr, glyph, filled, fillColor) {
export default class WinRectsExtension extends Extension {
enable() {
+ this._captureClones = new Map();
this._impl = Gio.DBusExportedObject.wrapJSObject(IFACE, this);
this._impl.export(Gio.DBus.session, '/org/cua/WinRects');
this._nameId = Gio.bus_own_name(
@@ -822,6 +827,11 @@ export default class WinRectsExtension extends Extension {
});
}
disable() {
+ for (const entry of this._captureClones.values()) {
+ GLib.source_remove(entry.timeout);
+ entry.clone.destroy();
+ }
+ this._captureClones.clear();
if (this._frameId) {
GLib.source_remove(this._frameId);
this._frameId = 0;
@@ -851,7 +861,7 @@ export default class WinRectsExtension extends Extension {
}
}
GetVersion() {
- return 8;
+ return 10;
}
GetRects() {
const actors = global.get_window_actors();
@@ -891,6 +901,96 @@ export default class WinRectsExtension extends Extension {
}
return JSON.stringify(out);
}
+ CaptureWindowAsync([id, pid], invocation) {
+ return this._captureWindow(id, pid, 0, invocation);
+ }
+ CaptureWindowPreviewAsync([id, pid, maxDimension], invocation) {
+ if (maxDimension < 1 || maxDimension > 4096) {
+ invocation.return_dbus_error('org.cua.WinRects.InvalidSize', 'Invalid preview size');
+ return;
+ }
+ return this._captureWindow(id, pid, maxDimension, invocation);
+ }
+ async _captureWindow(id, pid, maxDimension, invocation) {
+ let stream;
+ try {
+ if (Main.sessionMode.isLocked || Main.sessionMode.isGreeter)
+ throw new Error('Desktop is unavailable');
+ const actor = global.get_window_actors().find(
+ candidate => candidate.meta_window?.get_stable_sequence() === id);
+ const window = actor?.meta_window;
+ if (!window || (pid !== 0 && window.get_pid() !== pid) || window.minimized)
+ throw new Error('Target window is unavailable');
+ // A mapped clone keeps Mutter from suspending a fully covered source.
+ // It paints no pixels and expires when capture requests stop.
+ let entry = this._captureClones.get(id);
+ if (entry) GLib.source_remove(entry.timeout);
+ else {
+ const clone = new Clutter.Clone({source: actor, opacity: 0, width: 1, height: 1, reactive: false});
+ global.stage.add_child(clone);
+ entry = {clone, timeout: 0};
+ this._captureClones.set(id, entry);
+ }
+ entry.timeout = GLib.timeout_add(GLib.PRIORITY_DEFAULT, 4000, () => {
+ entry.clone.destroy();
+ this._captureClones.delete(id);
+ return GLib.SOURCE_REMOVE;
+ });
+ // Snapshot this actor, never the stage. Covered windows retain their own
+ // content; neither activation nor a desktop crop is needed.
+ const frame = window.get_frame_rect();
+ const buffer = window.get_buffer_rect();
+ const scale = actor.get_resource_scale();
+ const content = actor.paint_to_content(null);
+ const texture = content?.get_texture();
+ if (!texture || !Number.isFinite(scale) || scale <= 0)
+ throw new Error('Window content is unavailable');
+ const x = Math.round((frame.x - buffer.x) * scale);
+ const y = Math.round((frame.y - buffer.y) * scale);
+ const width = Math.round(frame.width * scale);
+ const height = Math.round(frame.height * scale);
+ if (x < 0 || y < 0 || width <= 0 || height <= 0 ||
+ x + width > texture.get_width() || y + height > texture.get_height())
+ throw new Error('Window content does not match its frame');
+ stream = Gio.MemoryOutputStream.new_resizable();
+ let captureTexture = texture;
+ let captureX = x, captureY = y, captureWidth = width, captureHeight = height;
+ if (maxDimension > 0 && Math.max(width, height) > maxDimension) {
+ const ratio = maxDimension / Math.max(width, height);
+ captureWidth = Math.max(1, Math.round(width * ratio));
+ captureHeight = Math.max(1, Math.round(height * ratio));
+ const context = global.stage.get_context().get_backend().get_cogl_context();
+ captureTexture = Cogl.Texture2D.new_with_size(context, captureWidth, captureHeight);
+ const framebuffer = Cogl.Offscreen.new_with_texture(captureTexture);
+ framebuffer.allocate();
+ framebuffer.set_viewport(0, 0, captureWidth, captureHeight);
+ framebuffer.orthographic(0, 0, captureWidth, captureHeight, -1, 1);
+ const pipeline = Cogl.Pipeline.new(context);
+ pipeline.set_layer_texture(0, texture);
+ framebuffer.draw_textured_rectangle(pipeline, 0, 0, captureWidth, captureHeight,
+ x / texture.get_width(), y / texture.get_height(),
+ (x + width) / texture.get_width(), (y + height) / texture.get_height());
+ framebuffer.finish();
+ captureX = 0;
+ captureY = 0;
+ }
+ await Shell.Screenshot.composite_to_stream(
+ captureTexture, captureX, captureY, captureWidth, captureHeight, 1, null, 0, 0, 1, stream);
+ stream.close(null);
+ if (Main.sessionMode.isLocked || Main.sessionMode.isGreeter)
+ throw new Error('Desktop became unavailable');
+ if (stream.get_data_size() > 32 * 1024 * 1024)
+ throw new Error('Window capture is too large');
+ const encoded = GLib.base64_encode(stream.steal_as_bytes().get_data());
+ invocation.return_value(maxDimension > 0
+ ? new GLib.Variant('(suu)', [encoded, width, height])
+ : new GLib.Variant('(s)', [encoded]));
+ } catch (error) {
+ invocation.return_dbus_error('org.cua.WinRects.CaptureFailed', String(error));
+ } finally {
+ if (stream && !stream.is_closed()) stream.close(null);
+ }
+ }
async CaptureAsync(_params, invocation) {
try {
const shooter = new Shell.Screenshot();
diff --git a/libs/cua-driver/wayland-helper/winrects@cua/metadata.json b/libs/cua-driver/wayland-helper/winrects@cua/metadata.json
index 3faf86b779..a2843ca98d 100644
--- a/libs/cua-driver/wayland-helper/winrects@cua/metadata.json
+++ b/libs/cua-driver/wayland-helper/winrects@cua/metadata.json
@@ -3,5 +3,5 @@
"name": "WinRects",
"description": "Expose window geometry, capture, cursor, and verified activation for cua-driver on Wayland.",
"shell-version": ["45", "46", "47", "48", "49", "50"],
- "version": 8
+ "version": 10
}