Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

revert(cua-driver): remove native Agent View presentation (#3452)

Francesco Bonacci47452be2da82
diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
index 91187102b7..3dd5405122 100644
--- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx
+++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx
@@ -661,14 +661,14 @@ Caveats:
### `set_config`
-Update cua-driver-rs configuration. Changes to max_image_dimension take effect immediately. The agent_view keys are persisted to ~/.cua-driver/config.json and take effect on the next daemon restart (Agent View is initialised once at startup).
+Update cua-driver-rs configuration. Changes to max_image_dimension take effect immediately. The experimental_pip keys are persisted to ~/.cua-driver/config.json and take effect on the next daemon restart (the PiP backend is initialised once at startup).
Note: capture_mode is a per-call param (on get_window_state / click), not a stored setting. Capture modality is selected by each action's target; the old capture_scope config key is retired.
**Arguments:**
-- `agent_view` (boolean, optional): Enable the multi-target Agent View. Exact native windows and Chrome tabs are grouped automatically by session; no target claiming is involved. Applies on next daemon restart.
-- `agent_view_geometry` (string, optional): Agent View size + optional position in `WxH` or `WxH+X+Y` form (e.g. `640x420+24+24`). Applies on next daemon restart.
+- `experimental_pip` (boolean, optional): Enable the experimental picture-in-picture preview window. Applies on next daemon restart.
+- `experimental_pip_geometry` (string, optional): PiP window size + optional position in `WxH` or `WxH+X+Y` form (e.g. `320x200+24+24`). Applies on next daemon restart.
- `key` (string, optional): Name of a single config field to write ({key, value} shape, matching the CLI `config set` and the Windows/Linux tools). Pair with `value`. Equivalent to passing the field directly.
- `max_image_dimension` (integer, optional): Max dimension for screenshot resizing (0 = no limit).
- `value` (unknown, optional): New value for `key`. JSON type depends on the key.
diff --git a/libs/cua-driver/README.md b/libs/cua-driver/README.md
index 40d859a01b..710afcdfa7 100644
--- a/libs/cua-driver/README.md
+++ b/libs/cua-driver/README.md
@@ -131,9 +131,6 @@ Contributor documentation:
- `docs/cursor-themes.md` documents the default semantic cursor and custom
dotLottie authoring contract.
-- `docs/agent-view.md` documents the cross-platform miniature desktop (enabled
- by default, with an explicit opt-out), its selected-session model, and
- exact-target membership rules.
- `docs/test-matrix.md` maps unit and canonical harness E2E suites.
- `docs/action-support.md` is the empirical platform behavior ledger.
- `docs/test-harnesses-guide.md` explains fixture and runner ownership.
diff --git a/libs/cua-driver/docs/agent-view.md b/libs/cua-driver/docs/agent-view.md
deleted file mode 100644
index e5f1144d10..0000000000
--- a/libs/cua-driver/docs/agent-view.md
+++ /dev/null
@@ -1,79 +0,0 @@
-# Agent View
-
-Agent View is an optional, always-on-top miniature desktop that shows the exact
-application windows and browser tabs traversed by one Cua Driver agent session.
-It is disabled by default. Enable it for one daemon invocation with:
-
-```console
-cua-driver --agent-view
-```
-
-The durable config equivalent is `{"agent_view": true}` in
-`~/.cua-driver/config.json`.
-
-`--no-agent-view` remains available to override a persisted enable for one
-daemon invocation.
-
-Use `--agent-view-geometry WxH[+X+Y]` to override its initial size and optional
-top-left position. The default is `640x420` near the top-right of the main
-display. The view is resizable on macOS, Windows, and Linux X11/XWayland.
-Pure Wayland uses a native top-right layer surface when the compositor exposes
-`zwlr_layer_shell_v1`; its position and size are compositor-managed. GNOME
-sessions without layer-shell use the supported XWayland presentation path. A
-pure layer-shell view remains click-through and follows the most-recent session;
-local tab selection, dragging, and manual resizing require the X11/XWayland
-window path because standard Wayland does not expose those global window
-management operations to clients.
-
-## Session model
-
-- One Agent View window belongs to one daemon and displays one selected private
- runtime session at a time. Multiple sessions appear as local tabs in that
- same window; the tab strip is hidden when only one session has cards.
-- Until a person selects a session locally, Agent View follows the session with
- the most recent exact target activity.
-- When multiple sessions have cards, the native Agent View window exposes a
- local session switcher. Selecting a session pins the view there until that
- session ends or is removed. There is no agent-facing selection tool.
-- Public session names are display labels only. The private runtime session ID
- remains the isolation identity, so equal labels never combine sessions.
-- Ending, disconnecting, revoking, or idle-expiring a session removes its cards.
- Reusing a revived session starts with an empty view.
-
-Agent View does not claim targets and does not add claim/release semantics to
-the CLI, MCP tools, SDKs, or target schemas. Switching the presentation never
-focuses, moves, resizes, closes, or otherwise controls an underlying target.
-During physical Cua Driver actions, the always-on-top surface temporarily
-becomes input-transparent so an overlapping Agent View cannot intercept input
-intended for the represented application. Local switching and resizing resume
-as soon as the action finishes.
-
-## What becomes a card
-
-A target is added or refreshed only after Cua Driver receives exact target
-evidence through one of these paths:
-
-- `get_window_state` with both `pid` and `window_id`;
-- an exact native action carrying both `pid` and `window_id`;
-- `get_browser_state` with both `target_id` and `tab_id`; or
-- an exact browser-tab action carrying both `target_id` and `tab_id`.
-
-Broad discovery such as `list_windows`, `list_browser_tabs`, whole-desktop
-snapshots, ambient foreground activity, and actions performed outside Cua
-Driver do not populate Agent View.
-
-Browser cards use the page's stable internal CDP identity, so navigation and a
-repeated browser-window bind refresh the same card. Multiple tabs in one Chrome
-or Edge window remain distinct, while their native container window is hidden
-to avoid duplicating the browser.
-
-When a target is proven absent, only its exact card is removed. When a session
-ends, all cards belonging to that private session are removed. The underlying
-application or tab is never changed by presentation cleanup.
-
-## Public contract
-
-Agent View uses the existing daemon flag, geometry option, runtime sessions,
-and exact target arguments. This session-oriented behavior does not introduce
-new MCP tools, CLI flags, SDK types, capability IDs, claim tokens, or public
-target/session fields.
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs
index a38ada51f1..5f4d1a0760 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs
@@ -410,15 +410,6 @@ pub(crate) struct BrowserTabScreenshot {
pub pixel_to_css_scale_y: f64,
}
-pub(crate) struct BrowserTabPreview {
- pub screenshot: BrowserTabScreenshot,
- pub title: String,
- pub url: String,
- pub cdp_target_id: String,
- pub pid: i64,
- pub window_id: u64,
-}
-
#[derive(Debug, Clone, Copy)]
struct BrowserScreenshotViewport {
page_x: f64,
@@ -2023,36 +2014,6 @@ impl BrowserEngine {
})
}
- /// Capture one exact tab for a trusted local presentation surface while
- /// retaining the tab metadata already proved by the session-scoped store.
- /// Like `capture_tab_screenshot`, this never activates the tab or brings
- /// its native window to the foreground.
- pub(crate) async fn capture_tab_preview(
- &self,
- session: &str,
- target_id: &str,
- tab_id: &str,
- ) -> Result<BrowserTabPreview, BrowserRefusal> {
- let record = self.store.get_target(session, target_id)?;
- let tab = record.tabs.get(tab_id).cloned().ok_or_else(|| {
- refuse(
- BrowserRefusalCode::BrowserTabNotFound,
- format!("tab {tab_id} is not known for target {target_id}"),
- )
- })?;
- let screenshot = self
- .capture_tab_screenshot(session, target_id, tab_id)
- .await?;
- Ok(BrowserTabPreview {
- screenshot,
- title: tab.title,
- url: tab.url,
- cdp_target_id: tab.cdp_target_id,
- pid: record.pid,
- window_id: record.window_id,
- })
- }
-
/// Snapshot one tab's composed DOM (main frame + shadow DOM +
/// same-process iframes + capability-tested OOPIFs) and mint
/// `p<snap>:<index>` refs for interactive elements. Read-only
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
index 194018f72b..f16fb1793c 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs
@@ -28,7 +28,6 @@ use super::types::BindingQuality;
/// crates call this from their `register_all` after constructing the
/// engine with their adapter.
pub fn register_browser_tools(engine: &Arc<BrowserEngine>, registry: &mut ToolRegistry) {
- registry.set_browser_engine(engine.clone());
registry.register(Box::new(GetBrowserStateTool::new(engine.clone())));
registry.register(Box::new(BrowserPrepareTool::new(engine.clone())));
registry.register(Box::new(BrowserNavigateTool::new(engine.clone())));
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/pip_hook.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/pip_hook.rs
index a0f15c9423..d2c426169f 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/pip_hook.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/pip_hook.rs
@@ -1,5 +1,5 @@
-//! Agent View event hook, registered once by `main.rs` when `--agent-view`
-//! is enabled.
+//! PiP frame-push hook — registered once by `main.rs` when the
+//! `--experimental-pip` flag is on argv.
//!
//! The trait + factory live in the `pip-preview` crate so the platform
//! backends can implement them without depending on `cua-driver-core`.
@@ -19,114 +19,31 @@ use std::sync::OnceLock;
/// to `pip_preview::PipFrame` — duplicated here to keep `cua-driver-core`
/// from importing `pip-preview` (the dependency would be circular once
/// platform backends pull both crates in).
-#[derive(Clone, Copy)]
-pub enum PipHookTargetKind {
- NativeWindow,
- BrowserTab,
-}
-
-pub struct PipHookTarget {
- pub workspace_id: String,
- pub workspace_label: String,
- pub target_id: String,
- pub identity_key: String,
- pub target_kind: PipHookTargetKind,
- pub target_label: String,
- pub native_container: Option<PipHookNativeContainer>,
-}
-
-#[derive(Clone, Copy)]
-pub struct PipHookNativeContainer {
- pub pid: i64,
- pub window_id: u64,
-}
-
pub struct PipHookFrame {
- pub target: PipHookTarget,
pub png_bytes: Vec<u8>,
pub action_label: String,
pub timestamp_ms: u64,
- pub cursor_position: Option<(f64, f64)>,
-}
-
-pub enum PipHookEvent {
- Upsert(PipHookFrame),
- SetInputPassthrough {
- passthrough: bool,
- },
- RemoveTarget {
- workspace_id: String,
- identity_key: String,
- },
- RemoveWorkspace {
- workspace_id: String,
- },
-}
-
-/// Restores Agent View interactivity even when an action exits early or
-/// unwinds. Physical desktop actions are serialized by the dispatcher, so one
-/// process-global passthrough scope is sufficient.
-pub struct PipInputPassthroughGuard;
-
-impl Drop for PipInputPassthroughGuard {
- fn drop(&mut self) {
- if let Some(f) = PIP_EVENT_FN.get() {
- if let Err(error) = f(PipHookEvent::SetInputPassthrough { passthrough: false }) {
- tracing::error!(%error, "failed to restore Agent View input handling");
- }
- }
- }
}
-type PipEventFnBox = Box<dyn Fn(PipHookEvent) -> Result<(), String> + Send + Sync>;
-static PIP_EVENT_FN: OnceLock<PipEventFnBox> = OnceLock::new();
+type PipPushFnBox = Box<dyn Fn(PipHookFrame) + Send + Sync>;
+static PIP_PUSH_FN: OnceLock<PipPushFnBox> = OnceLock::new();
/// Register the platform-side push callback. `main.rs` calls this
/// once after starting the PiP backend.
-pub fn set_pip_event_fn(f: impl Fn(PipHookEvent) -> Result<(), String> + Send + Sync + 'static) {
- let _ = PIP_EVENT_FN.set(Box::new(f));
+pub fn set_pip_push_fn(f: impl Fn(PipHookFrame) + Send + Sync + 'static) {
+ let _ = PIP_PUSH_FN.set(Box::new(f));
}
/// True when a PiP backend is wired up. Tool dispatcher uses this to
/// skip the screenshot-bytes path when nothing would consume the
-/// frame (avoiding wasted capture work when Agent View is disabled).
+/// frame (avoiding wasted capture work in the common --pip-off case).
pub fn pip_enabled() -> bool {
- PIP_EVENT_FN.get().is_some()
+ PIP_PUSH_FN.get().is_some()
}
-/// Temporarily make Agent View input-transparent. The registered backend
-/// applies the native state synchronously before this returns.
-pub fn begin_pip_input_passthrough() -> Result<Option<PipInputPassthroughGuard>, String> {
- let Some(f) = PIP_EVENT_FN.get() else {
- return Ok(None);
- };
- f(PipHookEvent::SetInputPassthrough { passthrough: true })?;
- Ok(Some(PipInputPassthroughGuard))
-}
-
-/// Upsert one exact-target frame. No-op when no backend is registered.
+/// Push a frame to the PiP window. No-op when no backend is registered.
pub fn push_pip_frame(frame: PipHookFrame) {
- if let Some(f) = PIP_EVENT_FN.get() {
- let _ = f(PipHookEvent::Upsert(frame));
- }
-}
-
-/// Remove one exact target card. Presentation only; never closes the target.
-pub fn remove_pip_target(workspace_id: impl Into<String>, identity_key: impl Into<String>) {
- if let Some(f) = PIP_EVENT_FN.get() {
- let _ = f(PipHookEvent::RemoveTarget {
- workspace_id: workspace_id.into(),
- identity_key: identity_key.into(),
- });
- }
-}
-
-/// Remove presentation state for an ended workspace. This never closes the
-/// workspace's native windows or browser tabs.
-pub fn remove_pip_workspace(workspace_id: impl Into<String>) {
- if let Some(f) = PIP_EVENT_FN.get() {
- let _ = f(PipHookEvent::RemoveWorkspace {
- workspace_id: workspace_id.into(),
- });
+ if let Some(f) = PIP_PUSH_FN.get() {
+ f(frame);
}
}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/recording.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/recording.rs
index 8a0c3a7cdf..362f76118a 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/recording.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/recording.rs
@@ -646,7 +646,7 @@ fn capture_turn(window_id: Option<u64>, pid: Option<i64>) -> TurnCapture {
}
}
-pub(crate) fn resolve_click_point(
+fn resolve_click_point(
tool_name: &str,
args: &Value,
window_id: Option<u64>,
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
index fee4b0cfa1..c9d82805f0 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs
@@ -6,7 +6,6 @@ use std::sync::{Arc, Mutex, OnceLock};
use std::time::Duration;
use async_trait::async_trait;
-use base64::{engine::general_purpose::STANDARD as BASE64, Engine as _};
use serde_json::Value;
thread_local! {
@@ -641,7 +640,6 @@ pub struct ToolRegistry {
approval_broker: Arc<crate::consent::ApprovalBroker>,
protected_resource_grants: Arc<crate::consent::ProtectedResourceGrants>,
protected_resource_ownership: Arc<crate::consent::ProtectedResourceOwnershipStore>,
- browser_engine: Option<Arc<crate::browser::BrowserEngine>>,
}
impl ToolRegistry {
@@ -705,7 +703,6 @@ impl ToolRegistry {
approval_broker,
protected_resource_grants,
protected_resource_ownership,
- browser_engine: None,
}
}
@@ -793,10 +790,6 @@ impl ToolRegistry {
self.tools.insert(name, tool);
}
- pub(crate) fn set_browser_engine(&mut self, engine: Arc<crate::browser::BrowserEngine>) {
- self.browser_engine = Some(engine);
- }
-
pub fn retain_session_end_hook(
&mut self,
registration: crate::session::SessionEndHookRegistration,
@@ -1517,18 +1510,6 @@ impl ToolRegistry {
} else {
None
};
- let pip_input_passthrough = if _desktop_action.is_some() {
- match pip_hook::begin_pip_input_passthrough() {
- Ok(guard) => guard,
- Err(error) => {
- return ToolResult::error(format!(
- "Agent View could not yield pointer input before {resolved_name}: {error}"
- ));
- }
- }
- } else {
- None
- };
let pending_turn = should_record
.then(|| {
if private_consent_turn {
@@ -1542,7 +1523,6 @@ impl ToolRegistry {
.flatten();
let mut result = tool.invoke(args.clone()).await;
- drop(pip_input_passthrough);
drop(lifecycle_dispatch);
// The platform worker has exited, so another text operation for this
// pid may now start even while result projection and evidence capture
@@ -1666,122 +1646,26 @@ impl ToolRegistry {
);
}
- // Agent View keeps one latest frame per exact native window or browser
- // tab. Exact read-side snapshots seed cards as well as action tools,
- // so a user can inspect several targets before any mutation occurs.
- if pip_hook::pip_enabled()
- && !private_consent_turn
- && result.is_error == Some(true)
- && pip_result_proves_target_absent(&result)
- {
- if let (Some(workspace_id), Some(target)) =
- (runtime_session.as_deref(), pip_capture_target(&args))
- {
- pip_hook::remove_pip_target(workspace_id, pip_target_public_key(&target));
- }
- }
- if pip_hook::pip_enabled()
- && !private_consent_turn
- && (should_record || is_exact_visual_snapshot(name, &args))
- {
- if let Some(frame) = self
- .capture_pip_frame(name, &args, &public_args, runtime_session.as_deref())
- .await
- {
- pip_hook::push_pip_frame(frame);
+ // Experimental PiP push — only when --experimental-pip is on argv
+ // (otherwise `pip_enabled()` is false and we skip the screenshot
+ // entirely to avoid wasted capture work). We push for the same set
+ // of action tools the recording pipeline cares about (non-read-only,
+ // not the recording-control meta-tools) so the live view matches
+ // what the recorder would have captured for the turn.
+ if pip_hook::pip_enabled() && should_record && !private_consent_turn {
+ let window_id = args.opt_u64("window_id");
+ let pid = args.opt_i64("pid");
+ if let Some(png_bytes) = screenshot_for(window_id, pid) {
+ let label = synthesize_action_label(name, &public_args);
+ pip_hook::push_pip_frame(pip_hook::PipHookFrame {
+ png_bytes,
+ action_label: label,
+ timestamp_ms: now_ms(),
+ });
}
}
- result
- }
-
- async fn capture_pip_frame(
- &self,
- tool_name: &str,
- args: &Value,
- public_args: &Value,
- runtime_session: Option<&str>,
- ) -> Option<pip_hook::PipHookFrame> {
- let workspace_id = runtime_session?.to_owned();
- let workspace_label = pip_workspace_label(public_args, runtime_session?);
- let action_label = synthesize_action_label(tool_name, public_args);
- let timestamp_ms = now_ms();
-
- if let (PipCaptureTarget::Browser { target_id, tab_id }, Some(engine)) =
- (pip_capture_target(args)?, self.browser_engine.as_ref())
- {
- let browser_session = args
- .get("session")
- .and_then(Value::as_str)
- .or(runtime_session)?;
- let preview = match engine
- .capture_tab_preview(browser_session, &target_id, &tab_id)
- .await
- {
- Ok(preview) => preview,
- Err(error)
- if matches!(
- error.code,
- crate::browser::refusal::BrowserRefusalCode::BrowserTabNotFound
- | crate::browser::refusal::BrowserRefusalCode::BrowserBindingStale
- | crate::browser::refusal::BrowserRefusalCode::BrowserWrongTargetRefused
- ) =>
- {
- pip_hook::remove_pip_target(
- &workspace_id,
- format!("browser:{target_id}:{tab_id}"),
- );
- return None;
- }
- Err(_) => return None,
- };
- let png_bytes = BASE64.decode(preview.screenshot.data_base64).ok()?;
- let cursor_position = pip_cursor_position(tool_name, args, &png_bytes, None, None);
- let target_label = if preview.title.trim().is_empty() {
- preview.url
- } else {
- preview.title
- };
- return Some(pip_hook::PipHookFrame {
- target: pip_hook::PipHookTarget {
- workspace_id,
- workspace_label,
- target_id: format!("browser:{target_id}:{tab_id}"),
- identity_key: format!("browser-cdp:{}", preview.cdp_target_id),
- target_kind: pip_hook::PipHookTargetKind::BrowserTab,
- target_label,
- native_container: Some(pip_hook::PipHookNativeContainer {
- pid: preview.pid,
- window_id: preview.window_id,
- }),
- },
- png_bytes,
- action_label,
- timestamp_ms,
- cursor_position,
- });
- }
- let PipCaptureTarget::NativeWindow { pid, window_id } = pip_capture_target(args)? else {
- return None;
- };
- let png_bytes = screenshot_for(Some(window_id), Some(pid))?;
- let cursor_position =
- pip_cursor_position(tool_name, args, &png_bytes, Some(window_id), Some(pid));
- Some(pip_hook::PipHookFrame {
- target: pip_hook::PipHookTarget {
- workspace_id,
- workspace_label,
- target_id: format!("window:{pid}:{window_id}"),
- identity_key: format!("window:{pid}:{window_id}"),
- target_kind: pip_hook::PipHookTargetKind::NativeWindow,
- target_label: format!("Window {window_id}"),
- native_container: Some(pip_hook::PipHookNativeContainer { pid, window_id }),
- },
- png_bytes,
- action_label,
- timestamp_ms,
- cursor_position,
- })
+ result
}
async fn authorize_private_observation(
@@ -3181,7 +3065,7 @@ mod runtime_isolation_tests {
"key": {"type": "string"},
"value": {},
"max_image_dimension": {"type": "integer"},
- "agent_view": {"type": "boolean"}
+ "experimental_pip": {"type": "boolean"}
},
"additionalProperties": false
})
@@ -4847,144 +4731,7 @@ impl Default for ToolRegistry {
}
}
-#[derive(Debug, Clone, PartialEq, Eq)]
-enum PipCaptureTarget {
- NativeWindow { pid: i64, window_id: u64 },
- Browser { target_id: String, tab_id: String },
-}
-
-fn pip_capture_target(args: &Value) -> Option<PipCaptureTarget> {
- if let (Some(target_id), Some(tab_id)) = (
- args.get("target_id").and_then(Value::as_str),
- args.get("tab_id").and_then(Value::as_str),
- ) {
- return Some(PipCaptureTarget::Browser {
- target_id: target_id.to_owned(),
- tab_id: tab_id.to_owned(),
- });
- }
- Some(PipCaptureTarget::NativeWindow {
- pid: args.opt_i64("pid")?,
- window_id: args.opt_u64("window_id")?,
- })
-}
-
-fn pip_target_public_key(target: &PipCaptureTarget) -> String {
- match target {
- PipCaptureTarget::NativeWindow { pid, window_id } => format!("window:{pid}:{window_id}"),
- PipCaptureTarget::Browser { target_id, tab_id } => {
- format!("browser:{target_id}:{tab_id}")
- }
- }
-}
-
-fn pip_cursor_position(
- tool_name: &str,
- args: &Value,
- png_bytes: &[u8],
- window_id: Option<u64>,
- pid: Option<i64>,
-) -> Option<(f64, f64)> {
- use crate::tool_args::ArgsExt;
-
- let point = if tool_name == "drag" || tool_name == "browser_drag" {
- Some((args.opt_f64("to_x")?, args.opt_f64("to_y")?))
- } else if matches!(
- tool_name,
- "click"
- | "double_click"
- | "right_click"
- | "move_cursor"
- | "browser_click"
- | "browser_double_click"
- | "browser_right_click"
- ) {
- match (args.opt_f64("x"), args.opt_f64("y")) {
- (Some(x), Some(y)) => Some((x, y)),
- _ => crate::recording::resolve_click_point(
- tool_name,
- args,
- window_id,
- pid,
- args.opt_u64("element_index"),
- ),
- }
- } else {
- None
- }?;
- let (width, height) = pip_png_dimensions(png_bytes)?;
- Some((
- (point.0 / width).clamp(0.0, 1.0),
- (point.1 / height).clamp(0.0, 1.0),
- ))
-}
-
-fn pip_png_dimensions(bytes: &[u8]) -> Option<(f64, f64)> {
- const PNG_SIGNATURE: &[u8; 8] = b"\x89PNG\r\n\x1a\n";
- if bytes.len() < 24 || &bytes[..8] != PNG_SIGNATURE || &bytes[12..16] != b"IHDR" {
- return None;
- }
- let width = u32::from_be_bytes(bytes[16..20].try_into().ok()?) as f64;
- let height = u32::from_be_bytes(bytes[20..24].try_into().ok()?) as f64;
- (width > 0.0 && height > 0.0).then_some((width, height))
-}
-
-fn pip_result_proves_target_absent(result: &ToolResult) -> bool {
- let code = result
- .structured_content
- .as_ref()
- .and_then(|payload| payload.get("code"))
- .and_then(Value::as_str);
- matches!(
- code,
- Some(
- "window_target_not_found"
- | "window_not_found"
- | "browser_tab_not_found"
- | "browser_binding_stale"
- | "browser_wrong_target_refused"
- )
- )
-}
-
-fn is_exact_visual_snapshot(tool_name: &str, args: &Value) -> bool {
- matches!(
- (tool_name, pip_capture_target(args)),
- (
- "get_window_state",
- Some(PipCaptureTarget::NativeWindow { .. })
- ) | ("get_browser_state", Some(PipCaptureTarget::Browser { .. }))
- )
-}
-
-fn pip_workspace_label(public_args: &Value, runtime_session: &str) -> String {
- if let Some(label) = public_args
- .get("session")
- .and_then(Value::as_str)
- .filter(|label| !label.is_empty() && *label != "default")
- {
- return label.to_owned();
- }
- let suffix = runtime_session
- .rsplit(':')
- .next()
- .unwrap_or(runtime_session)
- .chars()
- .filter(|ch| ch.is_ascii_alphanumeric())
- .rev()
- .take(6)
- .collect::<String>()
- .chars()
- .rev()
- .collect::<String>();
- if suffix.is_empty() {
- "Agent".to_owned()
- } else {
- format!("Agent {suffix}")
- }
-}
-
-/// Build a short, human-friendly label for the Agent View from the
+/// Build a short, human-friendly label for the PiP overlay from the
/// tool name + raw args. Kept under ~60 chars so the macOS NSTextField
/// has room without truncation at default geometry.
fn synthesize_action_label(tool_name: &str, args: &Value) -> String {
@@ -5031,133 +4778,6 @@ fn synthesize_action_label(tool_name: &str, args: &Value) -> String {
}
}
-#[cfg(test)]
-mod pip_routing_tests {
- use super::*;
-
- fn png_header(width: u32, height: u32) -> Vec<u8> {
- let mut bytes = b"\x89PNG\r\n\x1a\n\0\0\0\rIHDR".to_vec();
- bytes.extend_from_slice(&width.to_be_bytes());
- bytes.extend_from_slice(&height.to_be_bytes());
- bytes
- }
-
- #[test]
- fn pointer_coordinates_are_normalized_to_the_captured_target() {
- assert_eq!(
- pip_cursor_position(
- "click",
- &serde_json::json!({"x": 50, "y": 25}),
- &png_header(200, 100),
- None,
- None,
- ),
- Some((0.25, 0.25))
- );
- assert_eq!(
- pip_cursor_position(
- "drag",
- &serde_json::json!({"to_x": 180, "to_y": 90}),
- &png_header(200, 100),
- None,
- None,
- ),
- Some((0.9, 0.9))
- );
- assert_eq!(
- pip_cursor_position(
- "type_text",
- &serde_json::json!({"x": 50, "y": 25}),
- &png_header(200, 100),
- None,
- None,
- ),
- None
- );
- }
-
- #[test]
- fn browser_tab_identity_wins_over_its_native_container_window() {
- let target = pip_capture_target(&serde_json::json!({
- "pid": 42,
- "window_id": 7,
- "target_id": "bt-one",
- "tab_id": "tab-two",
- }));
- assert_eq!(
- target,
- Some(PipCaptureTarget::Browser {
- target_id: "bt-one".to_owned(),
- tab_id: "tab-two".to_owned(),
- })
- );
- }
-
- #[test]
- fn native_window_identity_requires_the_exact_pair() {
- assert_eq!(
- pip_capture_target(&serde_json::json!({"pid": 42, "window_id": 7})),
- Some(PipCaptureTarget::NativeWindow {
- pid: 42,
- window_id: 7,
- })
- );
- assert!(pip_capture_target(&serde_json::json!({"pid": 42})).is_none());
- assert!(pip_capture_target(&serde_json::json!({"window_id": 7})).is_none());
- }
-
- #[test]
- fn only_exact_state_reads_seed_agent_view_cards() {
- assert!(is_exact_visual_snapshot(
- "get_window_state",
- &serde_json::json!({"pid": 42, "window_id": 7})
- ));
- assert!(is_exact_visual_snapshot(
- "get_browser_state",
- &serde_json::json!({"target_id": "bt-one", "tab_id": "tab-two"})
- ));
- assert!(!is_exact_visual_snapshot(
- "get_browser_state",
- &serde_json::json!({"pid": 42, "window_id": 7})
- ));
- }
-
- #[test]
- fn public_session_label_is_preferred_for_presentation() {
- assert_eq!(
- pip_workspace_label(
- &serde_json::json!({"session": "research"}),
- "__cua_runtime_x:private"
- ),
- "research"
- );
- assert_eq!(
- pip_workspace_label(&serde_json::json!({}), "__cua_runtime_x:agent-123456"),
- "Agent 123456"
- );
- }
-
- #[test]
- fn proven_absent_results_remove_only_the_exact_public_target() {
- let absent = ToolResult::error("gone").with_structured(serde_json::json!({
- "code": "window_target_not_found"
- }));
- assert!(pip_result_proves_target_absent(&absent));
- assert_eq!(
- pip_target_public_key(&PipCaptureTarget::NativeWindow {
- pid: 42,
- window_id: 7,
- }),
- "window:42:7"
- );
-
- let permission = ToolResult::error("denied").with_structured(serde_json::json!({
- "code": "permission_denied"
- }));
- assert!(!pip_result_proves_target_absent(&permission));
- }
-}
-
#[cfg(test)]
mod capability_tests {
//! Unit tests for the per-tool `capabilities` array and the
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
index baec2e6c49..f3e021b74e 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs
@@ -232,10 +232,10 @@ const VALUE_FLAGS: &[&str] = &[
"--session",
"--profile-mode",
"--profile-name",
- // Agent View value flag for the optional geometry override
- // (`--agent-view` itself is a bare flag and doesn't
+ // Experimental PiP preview — value flag for the optional geometry
+ // override (--experimental-pip itself is a bare flag and doesn't
// need to be listed here).
- "--agent-view-geometry",
+ "--experimental-pip-geometry",
];
/// Authorization selectors are trusted-daemon startup inputs. Direct MCP
@@ -634,7 +634,7 @@ pub fn parse_command() -> Command {
println!("doctor options:");
println!(" --json Emit the probe report as JSON for scripting.");
println!();
- println!("experimental options:");
+ println!("experimental options (default: off):");
println!(
" --experimental-history Admit encrypted local Computer History for this daemon."
);
@@ -643,30 +643,17 @@ pub fn parse_command() -> Command {
);
println!(" cua-driver history enable Opt in and initialize encrypted local history.");
println!(" cua-driver history status|pause|resume|flush|list|show|disable|delete");
- println!(" --agent-view Enable the optional Agent View.");
- println!(" Exact native windows and");
+ println!(" --experimental-pip Show a small always-on-top window with the latest");
println!(
- " Chrome tabs traversed through Cua Driver become cards."
+ " post-action screenshot + a 1-line label. macOS only"
);
println!(
- " Recent activity is followed until you locally select"
+ " today; Win/Linux print a not-yet-implemented notice."
);
println!(
- " another session in Agent View; sessions never mix."
+ " --experimental-pip-geometry WxH[+X+Y] Override window size (and optional top-left"
);
- println!(
- " Closed targets and ended sessions are removed. This is"
- );
- println!(
- " presentation only; no claim/release API is involved."
- );
- println!(
- " Native presentation is available on macOS, Windows,"
- );
- println!(" and Linux (X11/XWayland).");
- println!(" --no-agent-view Disable Agent View, including a persisted enable.");
- println!(" --agent-view-geometry WxH[+X+Y] Override window size (and optional top-left");
- println!(" origin). Defaults to 640x420 in the top-right");
+ println!(" origin). Defaults to 480x360 in the top-right");
println!(" corner of the main display.");
std::process::exit(0);
}
diff --git a/libs/cua-driver/rust/crates/cua-driver/src/main.rs b/libs/cua-driver/rust/crates/cua-driver/src/main.rs
index e95c1a3ba8..7ce3e7c072 100644
--- a/libs/cua-driver/rust/crates/cua-driver/src/main.rs
+++ b/libs/cua-driver/rust/crates/cua-driver/src/main.rs
@@ -216,31 +216,23 @@ fn run_cursor_theme_command(args: &[String]) -> ! {
std::process::exit(status.code().unwrap_or(1));
}
-/// Wire up the multi-target Agent View.
+/// Wire up the experimental picture-in-picture preview window.
///
/// Called from every long-running entry point (Serve and Mcp on all
/// platforms; the `Call` arm intentionally skips PiP since the
/// per-call binaries don't keep an AppKit/event loop alive long
/// enough to be useful).
///
-fn maybe_init_pip() -> bool {
+/// No-op when `--experimental-pip` is not on argv. On Windows / Linux
+/// the factory returns "not yet implemented" — we log and continue
+/// without a window so the rest of the daemon keeps working.
+fn maybe_init_pip() {
let cfg = match pip_preview::default_config_path() {
Some(p) => pip_preview::PipConfig::from_args_and_file(&p),
None => pip_preview::PipConfig::from_args(),
};
if !cfg.enabled {
- return false;
- }
-
- // AppKit's shared application registration aborts the process when the
- // daemon has no Window Server session (for example, an SSH-launched or
- // headless service). Do not select the Agent View run loop unless the
- // native surface can safely exist; the socket-serving thread must remain
- // the main lifetime owner in that case.
- #[cfg(target_os = "macos")]
- if !platform_macos::session::has_graphic_access() {
- eprintln!("⚗️ Agent View requested but unavailable: no Window Server graphic session");
- return false;
+ return;
}
// Register the platform factory. The set is idempotent so multiple
@@ -264,68 +256,24 @@ fn maybe_init_pip() -> bool {
StdMutex<Option<Box<dyn pip_preview::PipBackend>>>,
> = std::sync::OnceLock::new();
let _ = BACKEND.set(StdMutex::new(Some(backend)));
- cua_driver_core::pip_hook::set_pip_event_fn(|event| {
+ cua_driver_core::pip_hook::set_pip_push_fn(|frame| {
if let Some(slot) = BACKEND.get() {
if let Some(b) = slot.lock().unwrap().as_ref() {
- match event {
- cua_driver_core::pip_hook::PipHookEvent::Upsert(frame) => {
- let target_kind = match frame.target.target_kind {
- cua_driver_core::pip_hook::PipHookTargetKind::NativeWindow => {
- pip_preview::PipTargetKind::NativeWindow
- }
- cua_driver_core::pip_hook::PipHookTargetKind::BrowserTab => {
- pip_preview::PipTargetKind::BrowserTab
- }
- };
- b.push_frame(pip_preview::PipFrame {
- target: pip_preview::PipTarget {
- workspace_id: frame.target.workspace_id,
- workspace_label: frame.target.workspace_label,
- target_id: frame.target.target_id,
- identity_key: frame.target.identity_key,
- target_kind,
- target_label: frame.target.target_label,
- native_container: frame.target.native_container.map(
- |container| pip_preview::PipNativeContainer {
- pid: container.pid,
- window_id: container.window_id,
- },
- ),
- },
- png_bytes: frame.png_bytes,
- action_label: frame.action_label,
- timestamp_ms: frame.timestamp_ms,
- cursor_position: frame.cursor_position,
- });
- }
- cua_driver_core::pip_hook::PipHookEvent::SetInputPassthrough {
- passthrough,
- } => {
- return b
- .set_input_passthrough(passthrough)
- .map_err(|error| error.to_string());
- }
- cua_driver_core::pip_hook::PipHookEvent::RemoveWorkspace {
- workspace_id,
- } => b.remove_workspace(&workspace_id),
- cua_driver_core::pip_hook::PipHookEvent::RemoveTarget {
- workspace_id,
- identity_key,
- } => b.remove_target(&workspace_id, &identity_key),
- }
+ b.push_frame(pip_preview::PipFrame {
+ png_bytes: frame.png_bytes,
+ action_label: frame.action_label,
+ timestamp_ms: frame.timestamp_ms,
+ });
}
}
- Ok(())
});
- cua_driver_core::session::register_session_end_hook(|workspace_id| {
- cua_driver_core::pip_hook::remove_pip_workspace(workspace_id);
- });
- eprintln!("⚗️ Agent View enabled (native macOS, Windows, and Linux presentation)");
- true
+ eprintln!(
+ "⚗️ PiP preview enabled (experimental — macOS only today; \
+ see https://github.com/trycua/cua/issues for follow-up)"
+ );
}
Err(e) => {
- eprintln!("⚗️ Agent View requested but unavailable: {e}");
- false
+ eprintln!("⚗️ PiP preview requested but unavailable: {e}");
}
}
}
@@ -419,28 +367,6 @@ fn history_admission_requested(explicit: bool, persisted: bool) -> bool {
explicit || persisted
}
-#[cfg(target_os = "macos")]
-#[derive(Debug, PartialEq, Eq)]
-enum MacosMainLoopHost {
- CursorOverlay,
- AgentView,
- ServeThread,
-}
-
-#[cfg(target_os = "macos")]
-fn macos_main_loop_host(cursor_enabled: bool, agent_view_enabled: bool) -> MacosMainLoopHost {
- if cursor_enabled {
- // The cursor host also owns NSApplication's main run loop. Agent View
- // posts its window updates to that same queue, so both surfaces can
- // coexist without leaving cursor animations waiting forever.
- MacosMainLoopHost::CursorOverlay
- } else if agent_view_enabled {
- MacosMainLoopHost::AgentView
- } else {
- MacosMainLoopHost::ServeThread
- }
-}
-
#[cfg(test)]
mod history_admission_tests {
use super::history_admission_requested;
@@ -458,31 +384,6 @@ mod history_admission_tests {
}
}
-#[cfg(all(test, target_os = "macos"))]
-mod macos_main_loop_host_tests {
- use super::{macos_main_loop_host, MacosMainLoopHost};
-
- #[test]
- fn cursor_overlay_hosts_the_shared_appkit_loop_when_agent_view_is_enabled() {
- assert_eq!(
- macos_main_loop_host(true, true),
- MacosMainLoopHost::CursorOverlay
- );
- }
-
- #[test]
- fn agent_view_hosts_appkit_only_when_the_cursor_is_disabled() {
- assert_eq!(
- macos_main_loop_host(false, true),
- MacosMainLoopHost::AgentView
- );
- assert_eq!(
- macos_main_loop_host(false, false),
- MacosMainLoopHost::ServeThread
- );
- }
-}
-
fn mcp_uses_direct_runtime(socket: Option<&str>, direct: bool) -> anyhow::Result<bool> {
mcp_uses_direct_runtime_for(
cua_driver_core::embedded_mode(),
@@ -697,7 +598,11 @@ fn main() {
// before any blocking work so the banner can land on stderr
// early in the serve lifecycle.
version_check::maybe_announce_update();
- let pip_available = maybe_init_pip();
+ let pip_cfg = match pip_preview::default_config_path() {
+ Some(p) => pip_preview::PipConfig::from_args_and_file(&p),
+ None => pip_preview::PipConfig::from_args(),
+ };
+ maybe_init_pip();
// Agent-cursor overlay. The DAEMON is the process that actually
// performs clicks / AX presses, so the overlay NSWindow + render
@@ -804,24 +709,24 @@ fn main() {
// Keep the main thread alive for the daemon.
//
- // Cursor and Agent View share one AppKit application/main queue.
- // When both are enabled, the cursor host must drain its command
- // receiver as well as NSApplication events; the Agent View-only
- // loop would leave cursor-bearing actions waiting indefinitely.
- match macos_main_loop_host(cursor_cfg.enabled, pip_available) {
- MacosMainLoopHost::CursorOverlay => {
- // `run_on_main_thread` self-guards on graphic-session
- // access. If it returns, keep the daemon alive by joining
- // the serve thread.
- platform_macos::cursor::overlay::run_on_main_thread();
- let _ = serve_handle.join();
- }
- MacosMainLoopHost::AgentView => {
- platform_macos::pip::run_appkit_main_loop();
- }
- MacosMainLoopHost::ServeThread => {
- let _ = serve_handle.join();
- }
+ // PiP needs the AppKit main run loop to process the
+ // dispatch_async_f calls that push frames into NSImageView;
+ // park main in NSApplication.run() when --experimental-pip is
+ // on. Otherwise just join the serve thread so the process
+ // stays up as long as the daemon does.
+ if pip_cfg.enabled {
+ platform_macos::pip::run_appkit_main_loop();
+ } else if cursor_cfg.enabled {
+ // Render the agent-cursor overlay: park the main thread in the
+ // AppKit run loop so the overlay NSWindow draws. `run_on_main_thread`
+ // self-guards on `has_graphic_access()` and returns immediately
+ // when the daemon has no Window Server session — fall through to
+ // join so the daemon still serves headless. The serve thread runs
+ // on its background thread regardless.
+ platform_macos::cursor::overlay::run_on_main_thread();
+ let _ = serve_handle.join();
+ } else {
+ let _ = serve_handle.join();
}
}
cli::Command::Stop {
@@ -1088,7 +993,7 @@ fn main() -> anyhow::Result<()> {
claude_code_compat,
cua_driver_core::embedded_mode(),
)?;
- let _ = maybe_init_pip();
+ maybe_init_pip();
let sp = socket.unwrap_or_else(serve::default_socket_path);
let pid_path = serve::default_pid_file_path();
// run_serve_cmd builds its own runtime; must run on a fresh thread.
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
index 2943ccf4a1..be79e5b15d 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs
@@ -27,10 +27,10 @@ pub use render_state::{
SESSION_BADGE_HOLD_SECS,
};
pub use session_badge::{
- paint_session_badge, rasterize_inter_text, sanitize_session_label, session_badge_extents,
- session_badge_layout, BadgeExtents, BadgeLabelLayout, SessionBadgeInput, SessionBadgeLayout,
- TextRaster, BADGE_CHIP_GAP, BADGE_CHIP_GROUP_GAP, BADGE_CHIP_SIZE, BADGE_CURSOR_GAP,
- BADGE_HEIGHT, BADGE_MAX_WIDTH, MAX_SESSION_LABEL_CHARS,
+ paint_session_badge, sanitize_session_label, session_badge_extents, session_badge_layout,
+ BadgeExtents, BadgeLabelLayout, SessionBadgeInput, SessionBadgeLayout, BADGE_CHIP_GAP,
+ BADGE_CHIP_GROUP_GAP, BADGE_CHIP_SIZE, BADGE_CURSOR_GAP, BADGE_HEIGHT, BADGE_MAX_WIDTH,
+ MAX_SESSION_LABEL_CHARS,
};
pub use theme::{
session_fill_hex, session_fill_rgba, CursorAction, CursorVisualState, DeliveryModifier,
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs
index d2f3139ed6..b37a2c8576 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs
@@ -202,56 +202,6 @@ pub fn sanitize_session_label(input: &str) -> Option<String> {
Some(result)
}
-/// Tightly cropped 8-bit coverage mask for one line of rasterized text.
-#[derive(Debug, Clone, PartialEq)]
-pub struct TextRaster {
- pub width: u32,
- pub height: u32,
- /// Row-major coverage, exactly `width * height` bytes.
- pub coverage: Vec<u8>,
-}
-
-/// Rasterize one line of UI text with the bundled Inter face.
-///
-/// Renderers that do not draw through `tiny_skia` — the Windows Agent View
-/// canvas paints straight into a DIB — can compose the returned mask themselves
-/// and still share one typeface with the cursor badge, so no second font asset
-/// or drawing dependency is needed. Returns `None` for text that rasterizes to
-/// nothing, such as an empty or whitespace-only string.
-pub fn rasterize_inter_text(text: &str, font_size: f32) -> Option<TextRaster> {
- let font = font()?;
- let layout = text_layout(font, text, font_size.max(1.0));
- let bounds = text_bounds(&layout)?;
- let width = bounds.width.ceil().max(1.0) as u32;
- let height = bounds.height.ceil().max(1.0) as u32;
- let mut coverage = vec![0u8; width as usize * height as usize];
- for glyph in layout.glyphs() {
- let (metrics, bitmap) = font.rasterize_config(glyph.key);
- if metrics.width == 0 || metrics.height == 0 {
- continue;
- }
- let origin_x = (glyph.x - bounds.min_x).round() as i32;
- let origin_y = (glyph.y - bounds.min_y).round() as i32;
- for row in 0..metrics.height {
- for column in 0..metrics.width {
- let x = origin_x + column as i32;
- let y = origin_y + row as i32;
- if x < 0 || y < 0 || x >= width as i32 || y >= height as i32 {
- continue;
- }
- let at = y as usize * width as usize + x as usize;
- coverage[at] = coverage[at].max(bitmap[row * metrics.width + column]);
- }
- }
- }
- let has_ink = coverage.iter().any(|&sample| sample > 0);
- has_ink.then_some(TextRaster {
- width,
- height,
- coverage,
- })
-}
-
fn rounded_rect(rect: Rect, radius: f32) -> Option<Path> {
const K: f32 = 0.552_284_8;
let x = rect.x();
@@ -634,28 +584,6 @@ mod tests {
assert!(long.ends_with('…'));
}
- #[test]
- fn rasterized_ui_text_is_cropped_and_reuses_the_badge_typeface() {
- let raster = rasterize_inter_text("10:42 AM", 12.0).unwrap();
- assert_eq!(
- raster.coverage.len(),
- raster.width as usize * raster.height as usize
- );
- assert!(raster.width > raster.height);
- assert!(raster.coverage.iter().any(|&coverage| coverage > 128));
- // A tight crop means the first and last columns both carry ink.
- let column_has_ink = |column: u32| {
- (0..raster.height)
- .any(|row| raster.coverage[(row * raster.width + column) as usize] > 0)
- };
- assert!(column_has_ink(0));
- assert!((raster.width.saturating_sub(2)..raster.width).any(column_has_ink));
-
- let larger = rasterize_inter_text("10:42 AM", 20.0).unwrap();
- assert!(larger.width > raster.width);
- assert!(rasterize_inter_text(" ", 12.0).is_none());
- }
-
#[test]
fn badge_paints_at_one_and_two_x_backing_scales() {
for scale in [1.0, 2.0] {
diff --git a/libs/cua-driver/rust/crates/pip-preview/src/desktop_layout.rs b/libs/cua-driver/rust/crates/pip-preview/src/desktop_layout.rs
deleted file mode 100644
index 86ea2437b0..0000000000
--- a/libs/cua-driver/rust/crates/pip-preview/src/desktop_layout.rs
+++ /dev/null
@@ -1,631 +0,0 @@
-//! Platform-neutral geometry for the Agent View miniature desktop.
-
-#[derive(Debug, Clone, Copy, PartialEq)]
-pub struct LayoutRect {
- pub x: f64,
- pub y: f64,
- pub width: f64,
- pub height: f64,
-}
-
-impl LayoutRect {
- #[cfg(test)]
- fn right(self) -> f64 {
- self.x + self.width
- }
-
- #[cfg(test)]
- fn bottom(self) -> f64 {
- self.y + self.height
- }
-}
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub struct TargetSize {
- pub width: u32,
- pub height: u32,
-}
-
-impl TargetSize {
- fn aspect(self) -> f64 {
- if self.width == 0 || self.height == 0 {
- return 16.0 / 10.0;
- }
- (self.width as f64 / self.height as f64).clamp(0.4, 2.8)
- }
-}
-
-#[derive(Debug, Clone, Copy, PartialEq)]
-pub struct TargetLayout {
- pub window: LayoutRect,
- pub content: LayoutRect,
-}
-
-/// Shell chrome idiom for the miniature desktop.
-///
-/// Only the dock band differs between styles. Target geometry is identical
-/// under every style, so a session's cards never move when the host platform
-/// swaps its shell.
-#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
-pub enum ShellStyle {
- /// No permanent launcher or taskbar chrome.
- #[default]
- None,
- /// Centered dock floating clear of every edge (macOS and Linux).
- FloatingDock,
- /// Full-width bar flush with the bottom screen edge (Windows).
- EdgeTaskbar,
-}
-
-#[derive(Debug, Clone, PartialEq)]
-pub struct DesktopLayout {
- pub desktop: LayoutRect,
- pub dock: LayoutRect,
- pub dock_icons: Vec<LayoutRect>,
- pub targets: Vec<TargetLayout>,
- pub shell: ShellStyle,
- /// Leading launcher slot. `EdgeTaskbar` only.
- pub start_button: Option<LayoutRect>,
- /// Trailing status band reserved for a clock and tray glyphs.
- /// `EdgeTaskbar` only.
- pub tray: Option<LayoutRect>,
- /// Running-app marks, one per dock icon in the same order.
- /// `EdgeTaskbar` only.
- pub indicators: Vec<LayoutRect>,
-}
-
-#[derive(Debug, Clone, PartialEq)]
-struct ShellChrome {
- dock: LayoutRect,
- dock_icons: Vec<LayoutRect>,
- start_button: Option<LayoutRect>,
- tray: Option<LayoutRect>,
- indicators: Vec<LayoutRect>,
-}
-
-const DOCK_ICON_GAP: f64 = 8.0;
-
-/// Lay out a miniature desktop with the floating dock shell.
-///
-/// Targets keep a stable row-major order. Columns receive widths based on the
-/// average aspect ratio of their targets, which gives landscape windows more
-/// room while keeping portrait and square windows compact. Narrow containers
-/// collapse to one column instead of shrinking previews beyond usefulness.
-pub fn layout_desktop(width: f64, height: f64, targets: &[TargetSize]) -> DesktopLayout {
- layout_desktop_with_shell(width, height, targets, ShellStyle::None)
-}
-
-/// Lay out a miniature desktop for one platform shell idiom.
-///
-/// The desktop band, and therefore every `TargetLayout`, is derived from the
-/// dock's top edge alone. Both shells compute that edge identically, so the
-/// shell only decides how the dock band itself is drawn.
-pub fn layout_desktop_with_shell(
- width: f64,
- height: f64,
- targets: &[TargetSize],
- shell: ShellStyle,
-) -> DesktopLayout {
- let width = width.max(1.0);
- let height = height.max(1.0);
- let short_edge = width.min(height);
- // Keep target shadows and rounded corners visibly separated from the
- // miniature desktop frame, including at the minimum supported panel size.
- let outer_gap = (short_edge * 0.045).clamp(16.0, 26.0);
- let dock_height = (height * 0.12).clamp(38.0, 58.0).min(height * 0.22);
- let dock_bottom_margin = (height * 0.028).clamp(10.0, 16.0);
- let dock_y = if shell == ShellStyle::None {
- height - outer_gap
- } else {
- (height - dock_height - dock_bottom_margin).max(outer_gap)
- };
- let desktop_y = outer_gap;
- let desktop_height = (dock_y - outer_gap - desktop_y).max(1.0);
- let desktop = LayoutRect {
- x: outer_gap,
- y: desktop_y,
- width: (width - 2.0 * outer_gap).max(1.0),
- height: desktop_height,
- };
-
- let chrome = match shell {
- ShellStyle::None => ShellChrome {
- dock: LayoutRect {
- x: 0.0,
- y: height,
- width: 0.0,
- height: 0.0,
- },
- dock_icons: Vec::new(),
- start_button: None,
- tray: None,
- indicators: Vec::new(),
- },
- ShellStyle::FloatingDock => {
- floating_dock_chrome(width, outer_gap, dock_y, dock_height, targets.len())
- }
- ShellStyle::EdgeTaskbar => edge_taskbar_chrome(width, height, dock_y, targets.len()),
- };
-
- if targets.is_empty() {
- return DesktopLayout::assemble(desktop, shell, chrome, Vec::new());
- }
-
- let columns = if targets.len() == 1 || desktop.width < 430.0 {
- 1
- } else if targets.len() <= 4 || desktop.width < 760.0 {
- 2
- } else {
- 3
- };
- let rows = targets.len().div_ceil(columns);
- let cell_gap = (short_edge * 0.014).clamp(5.0, 10.0);
- let usable_width = (desktop.width - cell_gap * (columns - 1) as f64).max(1.0);
-
- let mut column_weights = vec![0.0; columns];
- let mut column_counts = vec![0usize; columns];
- for (index, target) in targets.iter().enumerate() {
- let column = index % columns;
- column_weights[column] += target.aspect().clamp(0.65, 2.0);
- column_counts[column] += 1;
- }
- for column in 0..columns {
- column_weights[column] = if column_counts[column] == 0 {
- 1.0
- } else {
- (column_weights[column] / column_counts[column] as f64).clamp(0.7, 1.8)
- };
- }
- let total_column_weight = column_weights.iter().sum::<f64>().max(1.0);
- let column_widths = column_weights
- .iter()
- .map(|weight| usable_width * weight / total_column_weight)
- .collect::<Vec<_>>();
-
- let mut row_weights = vec![1.0; rows];
- for (row, row_weight) in row_weights.iter_mut().enumerate() {
- let mut desired_height: f64 = 1.0;
- for (column, column_width) in column_widths.iter().enumerate() {
- let index = row * columns + column;
- let Some(target) = targets.get(index) else {
- continue;
- };
- desired_height = desired_height.max(column_width / target.aspect());
- }
- *row_weight = desired_height;
- }
- let usable_height = (desktop.height - cell_gap * (rows - 1) as f64).max(1.0);
- let total_row_weight = row_weights.iter().sum::<f64>().max(1.0);
- let row_heights = row_weights
- .iter()
- .map(|weight| usable_height * weight / total_row_weight)
- .collect::<Vec<_>>();
-
- let mut column_x = Vec::with_capacity(columns);
- let mut x = desktop.x;
- for column_width in &column_widths {
- column_x.push(x);
- x += column_width + cell_gap;
- }
- let mut row_y = Vec::with_capacity(rows);
- let mut y = desktop.y;
- for row_height in &row_heights {
- row_y.push(y);
- y += row_height + cell_gap;
- }
-
- let target_layouts = targets
- .iter()
- .enumerate()
- .map(|(index, target)| {
- let column = index % columns;
- let row = index / columns;
- fit_target(
- LayoutRect {
- x: column_x[column],
- y: row_y[row],
- width: column_widths[column],
- height: row_heights[row],
- },
- target.aspect(),
- )
- })
- .collect::<Vec<_>>();
-
- DesktopLayout::assemble(desktop, shell, chrome, target_layouts)
-}
-
-impl DesktopLayout {
- fn assemble(
- desktop: LayoutRect,
- shell: ShellStyle,
- chrome: ShellChrome,
- targets: Vec<TargetLayout>,
- ) -> Self {
- Self {
- desktop,
- dock: chrome.dock,
- dock_icons: chrome.dock_icons,
- targets,
- shell,
- start_button: chrome.start_button,
- tray: chrome.tray,
- indicators: chrome.indicators,
- }
- }
-}
-
-fn floating_dock_chrome(
- width: f64,
- outer_gap: f64,
- dock_y: f64,
- dock_height: f64,
- target_count: usize,
-) -> ShellChrome {
- let icon_count = target_count.max(1);
- let max_icon_width = ((width - 4.0 * outer_gap - DOCK_ICON_GAP * (icon_count - 1) as f64)
- / icon_count as f64)
- .max(16.0);
- let icon_size = (dock_height - 16.0).min(max_icon_width).clamp(20.0, 42.0);
- let dock_width =
- (icon_count as f64 * icon_size + (icon_count - 1) as f64 * DOCK_ICON_GAP + 28.0)
- .min(width - 2.0 * outer_gap)
- .max(icon_size + 20.0);
- let dock = LayoutRect {
- x: (width - dock_width) / 2.0,
- y: dock_y,
- width: dock_width,
- height: dock_height,
- };
- let icons_width = icon_count as f64 * icon_size + (icon_count - 1) as f64 * DOCK_ICON_GAP;
- let icon_x = dock.x + (dock.width - icons_width) / 2.0;
- let icon_y = dock.y + (dock.height - icon_size) / 2.0 - 1.0;
- let dock_icons = (0..target_count)
- .map(|index| LayoutRect {
- x: icon_x + index as f64 * (icon_size + DOCK_ICON_GAP),
- y: icon_y,
- width: icon_size,
- height: icon_size,
- })
- .collect::<Vec<_>>();
- ShellChrome {
- dock,
- dock_icons,
- start_button: None,
- tray: None,
- indicators: Vec::new(),
- }
-}
-
-/// Windows 11-style bar: full width, flush with the bottom edge, with the
-/// launcher and app icons centered as one group and the tray trailing.
-///
-/// The bar takes its top edge from the shared `dock_y` and reaches the bottom
-/// by absorbing the floating dock's bottom margin, so it gains height without
-/// moving the desktop band above it.
-fn edge_taskbar_chrome(width: f64, height: f64, dock_y: f64, target_count: usize) -> ShellChrome {
- let dock = LayoutRect {
- x: 0.0,
- y: dock_y,
- width,
- height: (height - dock_y).max(1.0),
- };
- let tray_width = (width * 0.16).clamp(56.0, 104.0).min(width * 0.34);
- let tray = LayoutRect {
- x: (width - tray_width).max(0.0),
- y: dock.y,
- width: tray_width,
- height: dock.height,
- };
-
- // Reserve the tray band on both sides so the icon group stays centered on
- // the whole bar rather than on the space left of the clock, and let icons
- // shrink instead of sliding under the tray on a very narrow panel.
- let slots = target_count + 1;
- let slot_span =
- (width - 2.0 * (tray_width + 6.0) - DOCK_ICON_GAP * (slots - 1) as f64) / slots as f64;
- let icon_size = (dock.height * 0.55).min(slot_span).clamp(8.0, 32.0);
- let group_width = slots as f64 * icon_size + (slots - 1) as f64 * DOCK_ICON_GAP;
- let group_x = ((width - group_width) / 2.0).max(4.0);
- // Sit the icons slightly high in the band to leave room for the marks.
- let icon_y = dock.y + (dock.height - icon_size) / 2.0 - 2.0;
- let slot_x = |index: usize| group_x + index as f64 * (icon_size + DOCK_ICON_GAP);
-
- let start_button = LayoutRect {
- x: slot_x(0),
- y: icon_y,
- width: icon_size,
- height: icon_size,
- };
- let dock_icons = (0..target_count)
- .map(|index| LayoutRect {
- x: slot_x(index + 1),
- y: icon_y,
- width: icon_size,
- height: icon_size,
- })
- .collect::<Vec<_>>();
-
- let indicator_height = (icon_size * 0.1).clamp(2.0, 3.0);
- let indicator_width = (icon_size * 0.42).max(4.0);
- let indicator_y = (icon_y + icon_size + 3.0).min(dock.y + dock.height - indicator_height - 1.0);
- let indicators = dock_icons
- .iter()
- .map(|icon| LayoutRect {
- x: icon.x + (icon.width - indicator_width) / 2.0,
- y: indicator_y,
- width: indicator_width,
- height: indicator_height,
- })
- .collect::<Vec<_>>();
-
- ShellChrome {
- dock,
- dock_icons,
- start_button: Some(start_button),
- tray: Some(tray),
- indicators,
- }
-}
-
-fn fit_target(cell: LayoutRect, aspect: f64) -> TargetLayout {
- let inset = 1.0;
- let available_width = (cell.width - 2.0 * inset).max(1.0);
- let available_content_height = (cell.height - 2.0 * inset).max(1.0);
- let content_width = available_width.min(available_content_height * aspect);
- let content_height = (content_width / aspect).min(available_content_height);
- let window_width = content_width;
- let window_height = content_height;
- let window = LayoutRect {
- x: cell.x + (cell.width - window_width) / 2.0,
- y: cell.y + (cell.height - window_height) / 2.0,
- width: window_width,
- height: window_height,
- };
- let content = LayoutRect {
- x: window.x,
- y: window.y,
- width: content_width,
- height: content_height,
- };
- TargetLayout { window, content }
-}
-
-pub fn png_dimensions(bytes: &[u8]) -> Option<TargetSize> {
- const PNG_SIGNATURE: &[u8; 8] = b"\x89PNG\r\n\x1a\n";
- if bytes.len() < 24 || &bytes[..8] != PNG_SIGNATURE || &bytes[12..16] != b"IHDR" {
- return None;
- }
- let width = u32::from_be_bytes(bytes[16..20].try_into().ok()?);
- let height = u32::from_be_bytes(bytes[20..24].try_into().ok()?);
- (width > 0 && height > 0).then_some(TargetSize { width, height })
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
-
- fn size(width: u32, height: u32) -> TargetSize {
- TargetSize { width, height }
- }
-
- fn intersects(a: LayoutRect, b: LayoutRect) -> bool {
- a.x < b.right() && a.right() > b.x && a.y < b.bottom() && a.bottom() > b.y
- }
-
- #[test]
- fn mixed_form_factors_make_the_landscape_column_wider() {
- let layout = layout_desktop(
- 720.0,
- 520.0,
- &[
- size(1600, 900),
- size(700, 1200),
- size(1400, 900),
- size(900, 900),
- ],
- );
- assert_eq!(layout.targets.len(), 4);
- assert!(layout.targets[0].window.width > layout.targets[1].window.width);
- assert!(layout.targets[2].window.width > layout.targets[3].window.width);
- }
-
- #[test]
- fn narrow_desktop_reflows_to_one_column() {
- let layout = layout_desktop(380.0, 620.0, &[size(1600, 900), size(700, 1200)]);
- assert!(layout.targets[1].window.y > layout.targets[0].window.y);
- assert!((layout.targets[0].window.x - layout.targets[1].window.x).abs() < 80.0);
- }
-
- #[test]
- fn target_windows_stay_inside_the_desktop_and_do_not_overlap() {
- let layout = layout_desktop(
- 840.0,
- 560.0,
- &[
- size(1600, 900),
- size(700, 1200),
- size(1400, 900),
- size(900, 900),
- size(1000, 700),
- size(800, 1100),
- ],
- );
- for target in &layout.targets {
- assert!(target.window.x >= layout.desktop.x - 0.01);
- assert!(target.window.y >= layout.desktop.y - 0.01);
- assert!(target.window.right() <= layout.desktop.right() + 0.01);
- assert!(target.window.bottom() <= layout.desktop.bottom() + 0.01);
- }
- for (index, target) in layout.targets.iter().enumerate() {
- for other in layout.targets.iter().skip(index + 1) {
- assert!(!intersects(target.window, other.window));
- }
- }
- }
-
- #[test]
- fn content_preserves_the_source_aspect_ratio() {
- let targets = [size(1600, 900), size(700, 1200), size(900, 900)];
- let layout = layout_desktop(720.0, 520.0, &targets);
- for (target, source) in layout.targets.iter().zip(targets) {
- let rendered_aspect = target.content.width / target.content.height;
- assert!((rendered_aspect - source.aspect()).abs() < 0.001);
- assert_eq!(target.window, target.content);
- }
- }
-
- #[test]
- fn layout_reserves_padding_but_no_menu_or_target_title_bars() {
- let layout = layout_desktop(720.0, 520.0, &[size(1600, 900)]);
- assert!((16.0..=26.0).contains(&layout.desktop.y));
- assert_eq!(layout.targets[0].window, layout.targets[0].content);
- }
-
- #[test]
- fn minimum_panel_keeps_targets_away_from_the_frame() {
- let layout = layout_desktop(360.0, 260.0, &[size(900, 700)]);
- let target = layout.targets[0].window;
- assert!(target.x >= 16.0);
- assert!(target.y >= 16.0);
- assert!(target.right() <= 344.0);
- }
-
- #[test]
- fn dock_stays_below_targets_and_inside_the_panel() {
- let height = 520.0;
- let layout = layout_desktop_with_shell(
- 720.0,
- height,
- &[size(1600, 900), size(700, 1200), size(900, 900)],
- ShellStyle::FloatingDock,
- );
- let lowest_target = layout
- .targets
- .iter()
- .map(|target| target.window.bottom())
- .fold(0.0, f64::max);
- assert!(layout.dock.y >= lowest_target);
- assert!(layout.dock.bottom() <= height - 10.0);
- }
-
- #[test]
- fn no_shell_is_the_default_and_carries_no_launcher_metadata() {
- let layout = layout_desktop(720.0, 520.0, &[size(1600, 900), size(900, 900)]);
- assert_eq!(layout.shell, ShellStyle::None);
- assert_eq!(layout.shell, ShellStyle::default());
- assert!(layout.start_button.is_none());
- assert!(layout.tray.is_none());
- assert!(layout.indicators.is_empty());
- assert_eq!(layout.dock.width, 0.0);
- assert!(layout.dock_icons.is_empty());
- }
-
- #[test]
- fn edge_taskbar_spans_the_full_width_and_reaches_the_bottom_edge() {
- let width = 720.0;
- let height = 520.0;
- let targets = [size(1600, 900), size(900, 900), size(700, 1200)];
- let floating = layout_desktop_with_shell(width, height, &targets, ShellStyle::FloatingDock);
- let taskbar = layout_desktop_with_shell(width, height, &targets, ShellStyle::EdgeTaskbar);
-
- assert_eq!(taskbar.shell, ShellStyle::EdgeTaskbar);
- assert_eq!(taskbar.dock.x, 0.0);
- assert_eq!(taskbar.dock.width, width);
- assert_eq!(taskbar.dock.bottom(), height);
- // The bar keeps the floating dock's top edge and only absorbs the
- // margin that used to sit below it.
- assert_eq!(taskbar.dock.y, floating.dock.y);
- assert!(taskbar.dock.height > floating.dock.height);
- assert!(taskbar.dock.height - floating.dock.height <= 16.0 + f64::EPSILON);
- }
-
- #[test]
- fn edge_taskbar_preserves_every_floating_dock_target_layout() {
- let targets = [
- size(1600, 900),
- size(700, 1200),
- size(1400, 900),
- size(900, 900),
- size(1000, 700),
- ];
- for (width, height) in [(720.0, 520.0), (380.0, 620.0), (360.0, 260.0)] {
- let floating =
- layout_desktop_with_shell(width, height, &targets, ShellStyle::FloatingDock);
- let taskbar =
- layout_desktop_with_shell(width, height, &targets, ShellStyle::EdgeTaskbar);
- assert_eq!(floating.desktop, taskbar.desktop);
- assert_eq!(floating.targets, taskbar.targets);
- }
- }
-
- #[test]
- fn edge_taskbar_centers_the_start_slot_with_the_app_icons() {
- let width = 720.0;
- let targets = [size(1600, 900), size(900, 900)];
- let layout = layout_desktop_with_shell(width, 520.0, &targets, ShellStyle::EdgeTaskbar);
- let start = layout.start_button.unwrap();
- let tray = layout.tray.unwrap();
-
- assert_eq!(layout.dock_icons.len(), targets.len());
- assert!(start.x < layout.dock_icons[0].x);
- assert_eq!(start.y, layout.dock_icons[0].y);
- assert_eq!(start.width, layout.dock_icons[0].width);
-
- let group_left = start.x;
- let group_right = layout.dock_icons.last().unwrap().right();
- assert!(((group_left + group_right) / 2.0 - width / 2.0).abs() < 1.0);
- assert!(group_right <= tray.x);
- assert_eq!(tray.right(), width);
- assert_eq!(tray.y, layout.dock.y);
- assert_eq!(tray.height, layout.dock.height);
- }
-
- #[test]
- fn edge_taskbar_marks_every_running_icon_inside_the_bar() {
- let targets = [size(1600, 900), size(900, 900), size(700, 1200)];
- let layout = layout_desktop_with_shell(720.0, 520.0, &targets, ShellStyle::EdgeTaskbar);
- assert_eq!(layout.indicators.len(), layout.dock_icons.len());
- for (indicator, icon) in layout.indicators.iter().zip(&layout.dock_icons) {
- assert!(indicator.width < icon.width);
- assert!(indicator.y >= icon.bottom());
- assert!(indicator.bottom() <= layout.dock.bottom());
- assert!(
- ((indicator.x + indicator.width / 2.0) - (icon.x + icon.width / 2.0)).abs() < 1.0
- );
- }
- }
-
- #[test]
- fn edge_taskbar_keeps_the_start_slot_when_no_target_has_a_card() {
- let layout = layout_desktop_with_shell(640.0, 420.0, &[], ShellStyle::EdgeTaskbar);
- assert!(layout.start_button.is_some());
- assert!(layout.tray.is_some());
- assert!(layout.dock_icons.is_empty());
- assert!(layout.indicators.is_empty());
- assert!(layout.targets.is_empty());
- }
-
- #[test]
- fn crowded_narrow_taskbar_shrinks_icons_instead_of_sliding_under_the_tray() {
- let targets = vec![size(1200, 800); 12];
- let layout = layout_desktop_with_shell(320.0, 260.0, &targets, ShellStyle::EdgeTaskbar);
- let tray = layout.tray.unwrap();
- assert!(layout.start_button.unwrap().x >= 0.0);
- assert!(layout.dock_icons.last().unwrap().right() <= tray.x);
- for icon in &layout.dock_icons {
- assert!(icon.width >= 8.0);
- }
- }
-
- #[test]
- fn reads_png_ihdr_dimensions_without_decoding_the_image() {
- let mut png = vec![0u8; 24];
- png[..8].copy_from_slice(b"\x89PNG\r\n\x1a\n");
- png[12..16].copy_from_slice(b"IHDR");
- png[16..20].copy_from_slice(&1440u32.to_be_bytes());
- png[20..24].copy_from_slice(&900u32.to_be_bytes());
- assert_eq!(png_dimensions(&png), Some(size(1440, 900)));
- assert_eq!(png_dimensions(b"not png"), None);
- }
-}
diff --git a/libs/cua-driver/rust/crates/pip-preview/src/lib.rs b/libs/cua-driver/rust/crates/pip-preview/src/lib.rs
index 22f453f4a8..f8efa30b7e 100644
--- a/libs/cua-driver/rust/crates/pip-preview/src/lib.rs
+++ b/libs/cua-driver/rust/crates/pip-preview/src/lib.rs
@@ -1,28 +1,21 @@
-//! Shared model and platform contract for Agent View.
+//! pip-preview — shared types + trait for the experimental
+//! picture-in-picture agent preview window.
//!
-//! Agent View is an opt-in surface rather than one process-global "latest screenshot".
-//! Frames carry an existing session/workspace identity and an exact native
-//! window or browser-tab identity. Platform backends can therefore keep
-//! several target cards visible without introducing target claims or leases.
+//! The PiP window is an opt-in, always-on-top floating window that
+//! shows what the cua-driver agent just did: a post-action screenshot
+//! of the target window plus a one-line label summarising the tool
+//! call. It mirrors the architecture used by `cursor-overlay` (shared
+//! config/types here, platform-specific renderer in each `platform-*`
+//! crate) and the registration pattern used by `cua_driver_core::video`
+//! (a `OnceLock` factory set once at startup by `main.rs`).
//!
-//! Native backends render the shared model on macOS, Windows, and Linux
-//! X11/XWayland while keeping platform-specific window-system code isolated.
+//! macOS is the first working implementation (NSWindow + NSImageView).
+//! Windows + Linux ship as compile-clean stubs whose `start()` returns
+//! a clear "not yet implemented" error so the rest of the daemon
+//! continues without a PiP window.
use std::sync::OnceLock;
-use std::collections::HashMap;
-
-mod desktop_layout;
-mod session_tabs;
-
-pub use desktop_layout::{
- layout_desktop, layout_desktop_with_shell, png_dimensions, DesktopLayout, LayoutRect,
- ShellStyle, TargetLayout, TargetSize,
-};
-pub use session_tabs::{layout_session_tabs, session_accent, SessionTab, SessionTabsLayout};
-
-pub const AGENT_VIEW_DEFAULT_ENABLED: bool = false;
-
/// Canonical `~/.cua-driver/config.json` path matching what the per-platform
/// `set_config` tools write to. Resolves `$HOME` first (Unix/macOS) and falls
/// back to `%USERPROFILE%` (Windows, where `HOME` is usually unset). Returns
@@ -51,8 +44,8 @@ pub fn read_config_value(key: &str) -> Option<serde_json::Value> {
/// Merge a single `key`/`value` into `~/.cua-driver/config.json`,
/// preserving any other keys that are already there. Used by the
-/// per-platform `set_config` tools to persist `agent_view` /
-/// `agent_view_geometry` so the next daemon restart picks them up.
+/// per-platform `set_config` tools to persist `experimental_pip` /
+/// `experimental_pip_geometry` so the next daemon restart picks them up.
pub fn write_config_key(key: &str, value: serde_json::Value) -> Result<(), String> {
let path = default_config_path().ok_or_else(|| "$HOME is not set".to_string())?;
let mut json: serde_json::Value = path
@@ -70,29 +63,29 @@ pub fn write_config_key(key: &str, value: serde_json::Value) -> Result<(), Strin
Ok(())
}
-/// Read `agent_view` + `agent_view_geometry` from the
+/// Read `experimental_pip` + `experimental_pip_geometry` from the
/// config file, falling back to defaults when missing or malformed.
/// Surfaced by the per-platform `get_config` tools alongside the
/// in-memory `DriverConfig` fields.
-pub fn read_agent_view_keys_from_file() -> (bool, Option<String>) {
+pub fn read_pip_keys_from_file() -> (bool, Option<String>) {
let path = match default_config_path() {
Some(p) => p,
- None => return (AGENT_VIEW_DEFAULT_ENABLED, None),
+ None => return (false, None),
};
let text = match std::fs::read_to_string(&path) {
Ok(t) => t,
- Err(_) => return (AGENT_VIEW_DEFAULT_ENABLED, None),
+ Err(_) => return (false, None),
};
let json: serde_json::Value = match serde_json::from_str(&text) {
Ok(v) => v,
- Err(_) => return (AGENT_VIEW_DEFAULT_ENABLED, None),
+ Err(_) => return (false, None),
};
let enabled = json
- .get("agent_view")
+ .get("experimental_pip")
.and_then(|v| v.as_bool())
- .unwrap_or(AGENT_VIEW_DEFAULT_ENABLED);
+ .unwrap_or(false);
let geometry = json
- .get("agent_view_geometry")
+ .get("experimental_pip_geometry")
.and_then(|v| v.as_str())
.map(|s| s.to_owned());
(enabled, geometry)
@@ -100,7 +93,7 @@ pub fn read_agent_view_keys_from_file() -> (bool, Option<String>) {
/// Geometry of the PiP window, in screen points (top-left origin).
///
-/// Parsed from `--agent-view-geometry WxH+X+Y`. `x` / `y` are
+/// Parsed from `--experimental-pip-geometry WxH+X+Y`. `x` / `y` are
/// optional; when `None` the platform backend picks a sensible
/// "top-right corner with a small inset" default so a user enabling
/// the feature without any geometry flags still sees a window.
@@ -115,8 +108,8 @@ pub struct PipGeometry {
impl Default for PipGeometry {
fn default() -> Self {
Self {
- width: 640,
- height: 420,
+ width: 320,
+ height: 200,
x: None,
y: None,
}
@@ -156,32 +149,33 @@ impl PipGeometry {
/// flags and handed to `PipBackendFactory::start`.
#[derive(Debug, Clone)]
pub struct PipConfig {
- /// `--agent-view` is on argv. The factory is only consulted
+ /// `--experimental-pip` is on argv. The factory is only consulted
/// when this is true; the field is kept here so backends that
/// share a `start()` path can early-return.
pub enabled: bool,
pub geometry: PipGeometry,
- /// Native window title shared by all platform backends.
+ /// Window title — kept here so the "experimental" label stays in
+ /// one place. Defaults to "cua-driver — PiP preview (experimental)".
pub title: String,
}
impl Default for PipConfig {
fn default() -> Self {
Self {
- enabled: AGENT_VIEW_DEFAULT_ENABLED,
+ enabled: false,
geometry: PipGeometry::default(),
- title: "Cua Agent View".to_owned(),
+ title: "cua-driver — PiP preview (experimental)".to_owned(),
}
}
}
impl PipConfig {
- /// Parse the Agent View CLI flags out of `std::env::args()`.
- /// Recognised flags:
+ /// Parse the PiP-related CLI flags out of `std::env::args()`.
+ /// Recognised flags (all opt-in):
/// ```text
- /// --agent-view
- /// --no-agent-view
- /// --agent-view-geometry WxH | WxH+X+Y
+ /// --experimental-pip
+ /// --pip (short alias)
+ /// --experimental-pip-geometry WxH | WxH+X+Y
/// ```
/// Unknown flags are ignored so this never conflicts with the
/// other arg-parser passes (CursorConfig, the subcommand router).
@@ -192,27 +186,13 @@ impl PipConfig {
pub fn parse(args: &[String]) -> Self {
let mut cfg = PipConfig::default();
- let (enabled, geometry) = Self::parse_overrides(args);
- if let Some(enabled) = enabled {
- cfg.enabled = enabled;
- }
- if let Some(geometry) = geometry {
- cfg.geometry = geometry;
- }
- cfg
- }
-
- fn parse_overrides(args: &[String]) -> (Option<bool>, Option<PipGeometry>) {
- let mut enabled = None;
- let mut geometry = None;
let mut i = 0usize;
while i < args.len() {
match args[i].as_str() {
- "--agent-view" => enabled = Some(true),
- "--no-agent-view" => enabled = Some(false),
- "--agent-view-geometry" => {
+ "--experimental-pip" | "--pip" => cfg.enabled = true,
+ "--experimental-pip-geometry" => {
if let Some(geom) = args.get(i + 1).and_then(|s| PipGeometry::parse(s)) {
- geometry = Some(geom);
+ cfg.geometry = geom;
i += 1;
}
}
@@ -220,32 +200,30 @@ impl PipConfig {
}
i += 1;
}
- (enabled, geometry)
+ cfg
}
/// Resolve the config from (in order of precedence, low → high):
///
/// defaults → `~/.cua-driver/config.json` keys
- /// (`agent_view` bool, `agent_view_geometry` string)
+ /// (`experimental_pip` bool, `experimental_pip_geometry` string)
/// → CLI flags
///
- /// Lets users persist `--agent-view` across daemon restarts by
+ /// Lets users persist `--experimental-pip` across daemon restarts by
/// editing `~/.cua-driver/config.json` once, instead of re-running
/// `claude mcp add` with the flag baked into the args list.
/// Malformed or missing file falls back to the next layer silently.
pub fn from_args_and_file(config_path: &std::path::Path) -> Self {
- let args: Vec<String> = std::env::args().skip(1).collect();
- Self::from_file_and_args(config_path, &args)
- }
-
- fn from_file_and_args(config_path: &std::path::Path, args: &[String]) -> Self {
let mut cfg = PipConfig::default();
if let Ok(text) = std::fs::read_to_string(config_path) {
if let Ok(json) = serde_json::from_str::<serde_json::Value>(&text) {
- if let Some(b) = json.get("agent_view").and_then(|v| v.as_bool()) {
+ if let Some(b) = json.get("experimental_pip").and_then(|v| v.as_bool()) {
cfg.enabled = b;
}
- if let Some(s) = json.get("agent_view_geometry").and_then(|v| v.as_str()) {
+ if let Some(s) = json
+ .get("experimental_pip_geometry")
+ .and_then(|v| v.as_str())
+ {
if let Some(g) = PipGeometry::parse(s) {
cfg.geometry = g;
}
@@ -253,73 +231,26 @@ impl PipConfig {
}
}
// CLI args override anything in the file.
- let (enabled, geometry) = PipConfig::parse_overrides(args);
- if let Some(enabled) = enabled {
- cfg.enabled = enabled;
+ let args: Vec<String> = std::env::args().collect();
+ let cli = PipConfig::parse(&args[1..]);
+ if cli.enabled {
+ cfg.enabled = true;
}
- if let Some(geometry) = geometry {
- cfg.geometry = geometry;
+ // CLI geometry only overrides when explicitly passed — detect by
+ // diffing against PipGeometry::default() (the parse() entry point
+ // returns the default when no flag is present).
+ if cli.geometry.width != PipGeometry::default().width
+ || cli.geometry.height != PipGeometry::default().height
+ || cli.geometry.x.is_some()
+ || cli.geometry.y.is_some()
+ {
+ cfg.geometry = cli.geometry;
}
cfg
}
}
-/// The exact surface represented by an Agent View card.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum PipTargetKind {
- NativeWindow,
- BrowserTab,
-}
-
-impl PipTargetKind {
- pub fn as_str(self) -> &'static str {
- match self {
- Self::NativeWindow => "native_window",
- Self::BrowserTab => "browser_tab",
- }
- }
-}
-
-/// Stable target metadata used to group frames in the Agent View.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct PipTarget {
- /// Runtime-unique session key. This is grouping metadata, not an ownership
- /// or authorization claim.
- pub workspace_id: String,
- /// Short public session label shown to the user.
- pub workspace_label: String,
- /// Stable exact-target key within the workspace.
- pub target_id: String,
- /// Process-local identity used to de-duplicate equivalent bindings. This
- /// may be more stable than the public target reference (for example a CDP
- /// page target across repeated browser-window binds).
- pub identity_key: String,
- pub target_kind: PipTargetKind,
- pub target_label: String,
- /// Native window containing this target, when proven. Browser tabs use it
- /// to avoid a duplicate native Chrome/Edge window card.
- pub native_container: Option<PipNativeContainer>,
-}
-
-impl PipTarget {
- pub fn view_id(&self) -> String {
- view_id(&self.workspace_id, &self.identity_key)
- }
-}
-
-fn view_id(workspace_id: &str, identity_key: &str) -> String {
- // Both identities may contain `:`. Prefixing the workspace byte length
- // prevents cross-session concatenation aliases.
- format!("{}:{workspace_id}{identity_key}", workspace_id.len())
-}
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
-pub struct PipNativeContainer {
- pub pid: i64,
- pub window_id: u64,
-}
-
-/// A single exact-target frame pushed into the Agent View after a tool call.
+/// A single frame pushed into the PiP window after a tool call lands.
///
/// `png_bytes` are the raw PNG bytes produced by the platform
/// screenshot callback — the same path that powers `screenshot.png`
@@ -327,7 +258,6 @@ pub struct PipNativeContainer {
/// sees.
#[derive(Debug, Clone)]
pub struct PipFrame {
- pub target: PipTarget,
pub png_bytes: Vec<u8>,
/// One-line summary shown overlayed on the frame, e.g.
/// `click element_index=2` or `type_text "hello world"`.
@@ -335,351 +265,6 @@ pub struct PipFrame {
/// Wall-clock timestamp (ms since Unix epoch) — used by backends
/// that want to show "last update Xs ago" in the title bar.
pub timestamp_ms: u64,
- /// Normalized location of the synthetic agent pointer within the target image.
- pub cursor_position: Option<(f64, f64)>,
-}
-
-/// Platform-neutral latest-frame model with bounded target retention.
-///
-/// Backends use this to keep one card per exact target. The oldest-updated
-/// card is evicted when the configured limit is reached; ending a workspace
-/// removes all of its cards without closing the underlying applications.
-pub struct PipViewModel {
- max_targets: usize,
- frames: HashMap<String, PipFrame>,
- workspace_activity_ms: HashMap<String, u64>,
- active_targets: HashMap<String, String>,
- selected_workspace_id: Option<String>,
- selection_pinned: bool,
-}
-
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct PipWorkspaceSummary {
- pub workspace_id: String,
- pub workspace_label: String,
- pub target_count: usize,
- pub updated_ms: u64,
-}
-
-impl PipViewModel {
- pub fn new(max_targets: usize) -> Self {
- Self {
- max_targets: max_targets.max(1),
- frames: HashMap::new(),
- workspace_activity_ms: HashMap::new(),
- active_targets: HashMap::new(),
- selected_workspace_id: None,
- selection_pinned: false,
- }
- }
-
- pub fn upsert(&mut self, frame: PipFrame) -> Option<String> {
- let mut frame = frame;
- let workspace_id = frame.target.workspace_id.clone();
- self.workspace_activity_ms
- .entry(workspace_id.clone())
- .and_modify(|timestamp| *timestamp = (*timestamp).max(frame.timestamp_ms))
- .or_insert(frame.timestamp_ms);
- if self.selected_workspace_id.is_none() || !self.selection_pinned {
- self.selected_workspace_id = Some(workspace_id.clone());
- }
- if frame.target.target_kind == PipTargetKind::NativeWindow
- && frame.target.native_container.is_some_and(|container| {
- self.frames.values().any(|existing| {
- existing.target.workspace_id == workspace_id
- && existing.target.target_kind == PipTargetKind::BrowserTab
- && existing.target.native_container == Some(container)
- })
- })
- {
- return None;
- }
- if frame.target.target_kind == PipTargetKind::BrowserTab {
- if let Some(container) = frame.target.native_container {
- self.frames.retain(|_, existing| {
- existing.target.workspace_id != workspace_id
- || existing.target.target_kind != PipTargetKind::NativeWindow
- || existing.target.native_container != Some(container)
- });
- }
- }
- let view_id = frame.target.view_id();
- if frame.cursor_position.is_none() {
- frame.cursor_position = self
- .frames
- .get(&view_id)
- .and_then(|previous| previous.cursor_position);
- }
- self.frames.insert(view_id.clone(), frame);
- self.active_targets
- .insert(workspace_id.clone(), view_id.clone());
- if self.frames.len() <= self.max_targets {
- return None;
- }
- let counts = self
- .frames
- .values()
- .fold(HashMap::new(), |mut counts, frame| {
- *counts
- .entry(frame.target.workspace_id.as_str())
- .or_insert(0usize) += 1;
- counts
- });
- let largest_workspace = counts
- .iter()
- .max_by(|(left_id, left_count), (right_id, right_count)| {
- left_count
- .cmp(right_count)
- .then_with(|| right_id.cmp(left_id))
- })
- .map(|(id, _)| *id);
- let evicted = self
- .frames
- .iter()
- .filter(|(id, _)| *id != &view_id)
- .min_by_key(|(_, candidate)| {
- (
- candidate.target.workspace_id.as_str() != largest_workspace.unwrap_or(""),
- candidate.timestamp_ms,
- )
- })
- .map(|(id, _)| id.clone());
- if let Some(id) = evicted.as_ref() {
- let workspace_id = self
- .frames
- .get(id)
- .map(|frame| frame.target.workspace_id.clone());
- self.frames.remove(id);
- self.clear_active_view(id);
- if let Some(workspace_id) = workspace_id.as_deref() {
- self.reconcile_active_target(workspace_id);
- }
- }
- self.prune_empty_workspace_activity();
- self.reconcile_selection();
- evicted
- }
-
- pub fn remove_target(&mut self, workspace_id: &str, identity_key: &str) -> bool {
- let direct = view_id(workspace_id, identity_key);
- let view_id = self
- .frames
- .contains_key(&direct)
- .then_some(direct)
- .or_else(|| {
- self.frames
- .iter()
- .find(|(_, frame)| {
- frame.target.workspace_id == workspace_id
- && frame.target.target_id == identity_key
- })
- .map(|(id, _)| id.clone())
- });
- let removed = view_id
- .as_ref()
- .is_some_and(|view_id| self.frames.remove(view_id).is_some());
- if removed {
- if let Some(view_id) = view_id.as_deref() {
- self.clear_active_view(view_id);
- }
- self.reconcile_active_target(workspace_id);
- self.prune_empty_workspace_activity();
- self.reconcile_selection();
- }
- removed
- }
-
- pub fn remove_workspace(&mut self, workspace_id: &str) -> Vec<String> {
- let removed = self
- .frames
- .iter()
- .filter(|(_, frame)| frame.target.workspace_id == workspace_id)
- .map(|(id, _)| id.clone())
- .collect::<Vec<_>>();
- for id in &removed {
- self.frames.remove(id);
- }
- self.workspace_activity_ms.remove(workspace_id);
- self.active_targets.remove(workspace_id);
- if self.selected_workspace_id.as_deref() == Some(workspace_id) {
- self.selection_pinned = false;
- }
- self.reconcile_selection();
- removed
- }
-
- pub fn ordered_frames(&self) -> Vec<&PipFrame> {
- let mut frames = self.frames.values().collect::<Vec<_>>();
- frames.sort_by(|a, b| {
- a.target
- .workspace_label
- .cmp(&b.target.workspace_label)
- .then_with(|| b.timestamp_ms.cmp(&a.timestamp_ms))
- .then_with(|| a.target.target_id.cmp(&b.target.target_id))
- });
- frames
- }
-
- pub fn selected_workspace_id(&self) -> Option<&str> {
- self.selected_workspace_id.as_deref()
- }
-
- pub fn selected_frames(&self) -> Vec<&PipFrame> {
- let Some(selected) = self.selected_workspace_id() else {
- return Vec::new();
- };
- let mut frames = self
- .frames
- .values()
- .filter(|frame| frame.target.workspace_id == selected)
- .collect::<Vec<_>>();
- frames.sort_by(|a, b| {
- b.timestamp_ms
- .cmp(&a.timestamp_ms)
- .then_with(|| a.target.identity_key.cmp(&b.target.identity_key))
- });
- frames
- }
-
- pub fn active_view_id(&self) -> Option<&str> {
- let workspace_id = self.selected_workspace_id()?;
- self.active_targets.get(workspace_id).map(String::as_str)
- }
-
- pub fn workspaces(&self) -> Vec<PipWorkspaceSummary> {
- let mut summaries = HashMap::<&str, PipWorkspaceSummary>::new();
- for frame in self.frames.values() {
- let entry = summaries
- .entry(&frame.target.workspace_id)
- .or_insert_with(|| PipWorkspaceSummary {
- workspace_id: frame.target.workspace_id.clone(),
- workspace_label: sanitize_label(&frame.target.workspace_label, 48),
- target_count: 0,
- updated_ms: self
- .workspace_activity_ms
- .get(&frame.target.workspace_id)
- .copied()
- .unwrap_or(frame.timestamp_ms),
- });
- entry.target_count += 1;
- }
- let mut summaries = summaries.into_values().collect::<Vec<_>>();
- summaries.sort_by(|a, b| {
- b.updated_ms
- .cmp(&a.updated_ms)
- .then_with(|| a.workspace_label.cmp(&b.workspace_label))
- .then_with(|| a.workspace_id.cmp(&b.workspace_id))
- });
- summaries
- }
-
- pub fn select_workspace(&mut self, workspace_id: &str) -> bool {
- if !self
- .frames
- .values()
- .any(|frame| frame.target.workspace_id == workspace_id)
- {
- return false;
- }
- self.selected_workspace_id = Some(workspace_id.to_owned());
- self.selection_pinned = true;
- true
- }
-
- pub fn select_next_workspace(&mut self) -> bool {
- let workspaces = self.workspaces();
- if workspaces.len() <= 1 {
- return false;
- }
- let selected = self.selected_workspace_id();
- let index = workspaces
- .iter()
- .position(|workspace| Some(workspace.workspace_id.as_str()) == selected)
- .unwrap_or(0);
- self.select_workspace(&workspaces[(index + 1) % workspaces.len()].workspace_id)
- }
-
- pub fn selection_is_pinned(&self) -> bool {
- self.selection_pinned
- }
-
- fn reconcile_selection(&mut self) {
- let selected_exists = self
- .selected_workspace_id
- .as_deref()
- .is_some_and(|selected| {
- self.frames
- .values()
- .any(|frame| frame.target.workspace_id == selected)
- });
- if selected_exists {
- return;
- }
- self.selection_pinned = false;
- self.selected_workspace_id = self
- .workspaces()
- .into_iter()
- .next()
- .map(|workspace| workspace.workspace_id);
- }
-
- fn prune_empty_workspace_activity(&mut self) {
- self.workspace_activity_ms.retain(|workspace_id, _| {
- self.frames
- .values()
- .any(|frame| frame.target.workspace_id == *workspace_id)
- });
- }
-
- fn clear_active_view(&mut self, view_id: &str) {
- self.active_targets.retain(|_, active| active != view_id);
- }
-
- fn reconcile_active_target(&mut self, workspace_id: &str) {
- if self
- .active_targets
- .get(workspace_id)
- .is_some_and(|active| self.frames.contains_key(active))
- {
- return;
- }
- let fallback = self
- .frames
- .iter()
- .filter(|(_, frame)| frame.target.workspace_id == workspace_id)
- .max_by(|(left_id, left), (right_id, right)| {
- left.timestamp_ms
- .cmp(&right.timestamp_ms)
- .then_with(|| right_id.cmp(left_id))
- })
- .map(|(id, _)| id.clone());
- if let Some(fallback) = fallback {
- self.active_targets
- .insert(workspace_id.to_owned(), fallback);
- } else {
- self.active_targets.remove(workspace_id);
- }
- }
-
- pub fn len(&self) -> usize {
- self.frames.len()
- }
-
- pub fn is_empty(&self) -> bool {
- self.frames.is_empty()
- }
-}
-
-fn sanitize_label(value: &str, max_chars: usize) -> String {
- let mut label = value
- .chars()
- .filter(|character| !character.is_control())
- .take(max_chars)
- .collect::<String>();
- if label.trim().is_empty() {
- label = "Agent".to_owned();
- }
- label
}
/// A live PiP window. Owned by `main.rs` for the lifetime of the
@@ -691,24 +276,6 @@ pub trait PipBackend: Send + Sync {
/// queue via `dispatch_async`).
fn push_frame(&self, frame: PipFrame);
- /// Remove every card associated with an ended session/workspace. This
- /// changes only Agent View presentation; it does not close applications.
- fn remove_workspace(&self, workspace_id: &str);
-
- /// Remove one exact target from presentation. This never closes or
- /// otherwise mutates the underlying native window or browser tab.
- fn remove_target(&self, _workspace_id: &str, _identity_key: &str) {}
-
- /// Make the floating surface ignore or resume native pointer input.
- ///
- /// Implementations must not return until the native window-system state is
- /// applied. Cua Driver uses this around physical desktop actions so an
- /// overlapping always-on-top Agent View cannot intercept the action meant
- /// for an underlying target.
- fn set_input_passthrough(&self, _passthrough: bool) -> anyhow::Result<()> {
- Ok(())
- }
-
/// Close the window and release native resources. Called from
/// `main.rs` on shutdown.
fn shutdown(self: Box<Self>);
@@ -739,494 +306,3 @@ pub fn start_pip(cfg: &PipConfig) -> anyhow::Result<Box<dyn PipBackend>> {
.ok_or_else(|| anyhow::anyhow!("no PiP backend registered for this platform"))?;
factory.start(cfg)
}
-
-#[cfg(test)]
-mod tests {
- use super::*;
-
- fn frame(workspace: &str, target: &str, timestamp_ms: u64) -> PipFrame {
- PipFrame {
- target: PipTarget {
- workspace_id: workspace.to_owned(),
- workspace_label: workspace.to_owned(),
- target_id: target.to_owned(),
- identity_key: target.to_owned(),
- target_kind: PipTargetKind::NativeWindow,
- target_label: target.to_owned(),
- native_container: None,
- },
- png_bytes: vec![1],
- action_label: "click".to_owned(),
- timestamp_ms,
- cursor_position: None,
- }
- }
-
- fn browser_frame(
- workspace: &str,
- target: &str,
- identity: &str,
- container: PipNativeContainer,
- timestamp_ms: u64,
- ) -> PipFrame {
- PipFrame {
- target: PipTarget {
- workspace_id: workspace.to_owned(),
- workspace_label: workspace.to_owned(),
- target_id: target.to_owned(),
- identity_key: identity.to_owned(),
- target_kind: PipTargetKind::BrowserTab,
- target_label: target.to_owned(),
- native_container: Some(container),
- },
- png_bytes: vec![1],
- action_label: "get_browser_state".to_owned(),
- timestamp_ms,
- cursor_position: None,
- }
- }
-
- fn native_frame(
- workspace: &str,
- target: &str,
- container: PipNativeContainer,
- timestamp_ms: u64,
- ) -> PipFrame {
- let mut frame = frame(workspace, target, timestamp_ms);
- frame.target.native_container = Some(container);
- frame
- }
-
- #[test]
- fn keeps_distinct_targets_in_the_same_workspace() {
- let mut model = PipViewModel::new(4);
- model.upsert(frame("agent-a", "window:1:2", 1));
- model.upsert(frame("agent-a", "browser:bt:tab", 2));
- assert_eq!(model.len(), 2);
- }
-
- #[test]
- fn non_pointer_updates_preserve_the_last_cursor_position() {
- let mut model = PipViewModel::new(4);
- let mut clicked = frame("agent-a", "window:1:2", 1);
- clicked.cursor_position = Some((0.25, 0.75));
- model.upsert(clicked);
- model.upsert(frame("agent-a", "window:1:2", 2));
- assert_eq!(
- model.selected_frames()[0].cursor_position,
- Some((0.25, 0.75))
- );
- }
-
- #[test]
- fn updating_one_target_does_not_replace_another() {
- let mut model = PipViewModel::new(4);
- model.upsert(frame("agent-a", "window:1:2", 1));
- model.upsert(frame("agent-a", "browser:bt:tab", 2));
- model.upsert(frame("agent-a", "window:1:2", 3));
- assert_eq!(model.len(), 2);
- assert_eq!(model.ordered_frames()[0].target.target_id, "window:1:2");
- }
-
- #[test]
- fn ending_a_workspace_removes_only_its_cards() {
- let mut model = PipViewModel::new(4);
- model.upsert(frame("agent-a", "window:1:2", 1));
- model.upsert(frame("agent-b", "window:3:4", 2));
- assert_eq!(model.remove_workspace("agent-a").len(), 1);
- assert_eq!(model.len(), 1);
- assert_eq!(model.ordered_frames()[0].target.workspace_id, "agent-b");
- }
-
- #[test]
- fn evicts_the_oldest_other_target_at_capacity() {
- let mut model = PipViewModel::new(2);
- model.upsert(frame("agent-a", "old", 1));
- model.upsert(frame("agent-a", "newer", 2));
- assert_eq!(
- model.upsert(frame("agent-b", "newest", 3)),
- Some(view_id("agent-a", "old"))
- );
- assert_eq!(model.len(), 2);
- }
-
- #[test]
- fn follows_recent_activity_until_the_user_pins_a_workspace() {
- let mut model = PipViewModel::new(4);
- model.upsert(frame("agent-a", "a", 1));
- model.upsert(frame("agent-b", "b", 2));
- assert_eq!(model.selected_workspace_id(), Some("agent-b"));
-
- assert!(model.select_workspace("agent-a"));
- model.upsert(frame("agent-b", "b", 3));
- assert_eq!(model.selected_workspace_id(), Some("agent-a"));
- assert!(model.selection_is_pinned());
- assert_eq!(model.selected_frames().len(), 1);
- }
-
- #[test]
- fn active_target_is_scoped_to_the_selected_workspace() {
- let mut model = PipViewModel::new(6);
- model.upsert(frame("agent-a", "a-old", 1));
- model.upsert(frame("agent-a", "a-active", 2));
- model.upsert(frame("agent-b", "b-old", 3));
- model.upsert(frame("agent-b", "b-active", 4));
-
- assert_eq!(
- model.active_view_id(),
- Some(view_id("agent-b", "b-active").as_str())
- );
- assert!(model.select_workspace("agent-a"));
- assert_eq!(
- model.active_view_id(),
- Some(view_id("agent-a", "a-active").as_str())
- );
- assert!(model.select_workspace("agent-b"));
- assert_eq!(
- model.active_view_id(),
- Some(view_id("agent-b", "b-active").as_str())
- );
- }
-
- #[test]
- fn removing_an_active_target_falls_back_within_its_workspace() {
- let mut model = PipViewModel::new(6);
- model.upsert(frame("agent-a", "older", 1));
- model.upsert(frame("agent-a", "newer", 2));
- model.upsert(frame("agent-b", "other", 3));
- assert!(model.select_workspace("agent-a"));
-
- assert!(model.remove_target("agent-a", "newer"));
-
- assert_eq!(
- model.active_view_id(),
- Some(view_id("agent-a", "older").as_str())
- );
- assert_eq!(model.selected_workspace_id(), Some("agent-a"));
- }
-
- #[test]
- fn evicting_an_active_target_falls_back_within_its_workspace() {
- let mut model = PipViewModel::new(3);
- model.upsert(frame("agent-a", "fallback", 10));
- model.upsert(frame("agent-a", "active", 1));
- model.upsert(frame("agent-b", "other", 20));
- assert!(model.select_workspace("agent-a"));
-
- assert_eq!(
- model.upsert(frame("agent-c", "new", 30)),
- Some(view_id("agent-a", "active"))
- );
- assert_eq!(
- model.active_view_id(),
- Some(view_id("agent-a", "fallback").as_str())
- );
- }
-
- #[test]
- fn removing_the_selected_workspace_falls_back_to_the_most_recent() {
- let mut model = PipViewModel::new(4);
- model.upsert(frame("agent-a", "a", 1));
- model.upsert(frame("agent-b", "b", 2));
- assert!(model.select_workspace("agent-a"));
- model.remove_workspace("agent-a");
- assert_eq!(model.selected_workspace_id(), Some("agent-b"));
- assert!(!model.selection_is_pinned());
- }
-
- #[test]
- fn exact_target_removal_uses_the_internal_identity_key() {
- let mut model = PipViewModel::new(4);
- model.upsert(frame("agent-a", "window:1:2", 1));
- assert!(model.remove_target("agent-a", "window:1:2"));
- assert!(model.is_empty());
- assert_eq!(model.selected_workspace_id(), None);
- }
-
- #[test]
- fn equal_public_labels_do_not_merge_private_workspaces() {
- let mut model = PipViewModel::new(4);
- let mut first = frame("private-a", "one", 1);
- first.target.workspace_label = "research".to_owned();
- let mut second = frame("private-b", "two", 2);
- second.target.workspace_label = "research".to_owned();
- model.upsert(first);
- model.upsert(second);
-
- let workspaces = model.workspaces();
- assert_eq!(workspaces.len(), 2);
- assert!(workspaces
- .iter()
- .all(|workspace| workspace.workspace_label == "research"));
- assert_eq!(model.selected_workspace_id(), Some("private-b"));
- }
-
- #[test]
- fn stable_browser_identity_updates_across_repeated_bindings() {
- let mut model = PipViewModel::new(4);
- let container = PipNativeContainer {
- pid: 42,
- window_id: 7,
- };
- model.upsert(browser_frame(
- "agent-a",
- "browser:first:tab-a",
- "browser-cdp:page-1",
- container,
- 1,
- ));
- model.upsert(browser_frame(
- "agent-a",
- "browser:second:tab-b",
- "browser-cdp:page-1",
- container,
- 2,
- ));
-
- assert_eq!(model.len(), 1);
- assert_eq!(
- model.selected_frames()[0].target.target_id,
- "browser:second:tab-b"
- );
- assert_eq!(model.selected_frames()[0].timestamp_ms, 2);
- }
-
- #[test]
- fn browser_tabs_replace_and_suppress_their_native_container_card() {
- let mut model = PipViewModel::new(6);
- let container = PipNativeContainer {
- pid: 42,
- window_id: 7,
- };
- model.upsert(native_frame("agent-a", "window:42:7", container, 1));
- model.upsert(browser_frame(
- "agent-a",
- "browser:binding:tab-a",
- "browser-cdp:page-a",
- container,
- 2,
- ));
- model.upsert(browser_frame(
- "agent-a",
- "browser:binding:tab-b",
- "browser-cdp:page-b",
- container,
- 3,
- ));
- model.upsert(native_frame("agent-a", "window:42:7", container, 4));
-
- let frames = model.selected_frames();
- assert_eq!(frames.len(), 2);
- assert!(frames
- .iter()
- .all(|frame| frame.target.target_kind == PipTargetKind::BrowserTab));
- }
-
- #[test]
- fn suppressed_native_container_activity_still_drives_auto_follow() {
- let mut model = PipViewModel::new(6);
- let first_container = PipNativeContainer {
- pid: 42,
- window_id: 7,
- };
- let second_container = PipNativeContainer {
- pid: 84,
- window_id: 9,
- };
- model.upsert(browser_frame(
- "agent-a",
- "browser:first:tab-a",
- "browser-cdp:page-a",
- first_container,
- 1,
- ));
- model.upsert(browser_frame(
- "agent-b",
- "browser:second:tab-b",
- "browser-cdp:page-b",
- second_container,
- 2,
- ));
- assert_eq!(model.selected_workspace_id(), Some("agent-b"));
-
- model.upsert(native_frame("agent-a", "window:42:7", first_container, 3));
-
- assert_eq!(model.selected_workspace_id(), Some("agent-a"));
- assert_eq!(model.workspaces()[0].workspace_id, "agent-a");
- assert_eq!(model.workspaces()[0].updated_ms, 3);
- assert!(model
- .selected_frames()
- .iter()
- .all(|frame| frame.target.target_kind == PipTargetKind::BrowserTab));
- }
-
- #[test]
- fn private_session_and_target_delimiters_cannot_alias() {
- let mut model = PipViewModel::new(4);
- model.upsert(frame("private:a", "b", 1));
- model.upsert(frame("private", "a:b", 2));
-
- assert_eq!(model.len(), 2);
- assert!(model.select_workspace("private:a"));
- assert_eq!(model.selected_frames()[0].target.target_id, "b");
- assert!(model.select_workspace("private"));
- assert_eq!(model.selected_frames()[0].target.target_id, "a:b");
- }
-
- #[test]
- fn eviction_prefers_the_most_populated_workspace() {
- let mut model = PipViewModel::new(3);
- model.upsert(frame("agent-a", "old-a", 1));
- model.upsert(frame("agent-a", "new-a", 2));
- model.upsert(frame("agent-b", "only-b", 3));
- assert_eq!(
- model.upsert(frame("agent-c", "only-c", 4)),
- Some(view_id("agent-a", "old-a"))
- );
- assert!(model
- .ordered_frames()
- .iter()
- .any(|frame| frame.target.workspace_id == "agent-b"));
- }
-
- #[test]
- fn parses_agent_view_cli_names() {
- let cfg = PipConfig::parse(&[
- "--agent-view".to_owned(),
- "--agent-view-geometry".to_owned(),
- "640x420+24+36".to_owned(),
- ]);
- assert!(cfg.enabled);
- assert_eq!(cfg.geometry.width, 640);
- assert_eq!(cfg.geometry.height, 420);
- assert_eq!(cfg.geometry.x, Some(24));
- assert_eq!(cfg.geometry.y, Some(36));
- }
-
- #[test]
- fn agent_view_is_disabled_by_default() {
- let cfg = PipConfig::parse(&[]);
- assert!(!cfg.enabled);
- assert!(!PipConfig::default().enabled);
- }
-
- #[test]
- fn ignores_removed_pip_cli_names() {
- let cfg = PipConfig::parse(&[
- "--experimental-pip".to_owned(),
- "--experimental-pip-geometry".to_owned(),
- "640x420".to_owned(),
- "--pip".to_owned(),
- ]);
- assert!(!cfg.enabled);
- assert_eq!(cfg.geometry.width, PipGeometry::default().width);
- assert_eq!(cfg.geometry.height, PipGeometry::default().height);
- }
-
- #[test]
- fn reads_agent_view_config_names() {
- let path = std::env::temp_dir().join(format!(
- "cua-agent-view-config-{}-new.json",
- std::process::id()
- ));
- std::fs::write(
- &path,
- r#"{"agent_view":true,"agent_view_geometry":"800x600+12+24"}"#,
- )
- .unwrap();
-
- let cfg = PipConfig::from_args_and_file(&path);
- assert!(cfg.enabled);
- assert_eq!(cfg.geometry.width, 800);
- assert_eq!(cfg.geometry.height, 600);
- assert_eq!(cfg.geometry.x, Some(12));
- assert_eq!(cfg.geometry.y, Some(24));
-
- let _ = std::fs::remove_file(path);
- }
-
- #[test]
- fn config_file_overrides_defaults_without_cli_flags() {
- let path = std::env::temp_dir().join(format!(
- "cua-agent-view-config-{}-file-precedence.json",
- std::process::id()
- ));
- std::fs::write(
- &path,
- r#"{"agent_view":false,"agent_view_geometry":"800x600+12+24"}"#,
- )
- .unwrap();
-
- let cfg = PipConfig::from_file_and_args(&path, &[]);
- assert!(!cfg.enabled);
- assert_eq!(cfg.geometry.width, 800);
- assert_eq!(cfg.geometry.height, 600);
- assert_eq!(cfg.geometry.x, Some(12));
- assert_eq!(cfg.geometry.y, Some(24));
-
- let _ = std::fs::remove_file(path);
- }
-
- #[test]
- fn agent_view_cli_overrides_disabled_config_file() {
- let path = std::env::temp_dir().join(format!(
- "cua-agent-view-config-{}-cli-enable.json",
- std::process::id()
- ));
- std::fs::write(&path, r#"{"agent_view":false}"#).unwrap();
-
- let cfg = PipConfig::from_file_and_args(&path, &["--agent-view".to_owned()]);
- assert!(cfg.enabled);
-
- let _ = std::fs::remove_file(path);
- }
-
- #[test]
- fn cli_flags_override_config_file_values() {
- let path = std::env::temp_dir().join(format!(
- "cua-agent-view-config-{}-cli-precedence.json",
- std::process::id()
- ));
- std::fs::write(
- &path,
- r#"{"agent_view":true,"agent_view_geometry":"800x600+12+24"}"#,
- )
- .unwrap();
-
- let cfg = PipConfig::from_file_and_args(
- &path,
- &[
- "--no-agent-view".to_owned(),
- "--agent-view-geometry".to_owned(),
- "720x480+30+40".to_owned(),
- ],
- );
- assert!(!cfg.enabled);
- assert_eq!(cfg.geometry.width, 720);
- assert_eq!(cfg.geometry.height, 480);
- assert_eq!(cfg.geometry.x, Some(30));
- assert_eq!(cfg.geometry.y, Some(40));
-
- let _ = std::fs::remove_file(path);
- }
-
- #[test]
- fn ignores_removed_pip_config_names() {
- let path = std::env::temp_dir().join(format!(
- "cua-agent-view-config-{}-old.json",
- std::process::id()
- ));
- std::fs::write(
- &path,
- r#"{"experimental_pip":true,"experimental_pip_geometry":"800x600"}"#,
- )
- .unwrap();
-
- let cfg = PipConfig::from_args_and_file(&path);
- assert!(!cfg.enabled);
- assert_eq!(cfg.geometry.width, PipGeometry::default().width);
- assert_eq!(cfg.geometry.height, PipGeometry::default().height);
-
- let _ = std::fs::remove_file(path);
- }
-}
diff --git a/libs/cua-driver/rust/crates/pip-preview/src/session_tabs.rs b/libs/cua-driver/rust/crates/pip-preview/src/session_tabs.rs
deleted file mode 100644
index fe85c8d7ba..0000000000
--- a/libs/cua-driver/rust/crates/pip-preview/src/session_tabs.rs
+++ /dev/null
@@ -1,136 +0,0 @@
-use crate::{LayoutRect, PipWorkspaceSummary};
-
-const MAX_VISIBLE_TABS: usize = 6;
-
-#[derive(Debug, Clone, PartialEq)]
-pub struct SessionTab {
- pub workspace_id: String,
- pub label: String,
- pub rect: LayoutRect,
- pub selected: bool,
- pub accent: (u8, u8, u8),
-}
-
-#[derive(Debug, Clone, PartialEq, Default)]
-pub struct SessionTabsLayout {
- pub tabs: Vec<SessionTab>,
- pub overflow: usize,
-}
-
-impl SessionTabsLayout {
- pub fn hit_test(&self, x: f64, y: f64) -> Option<&str> {
- self.tabs
- .iter()
- .find(|tab| {
- x >= tab.rect.x
- && x <= tab.rect.x + tab.rect.width
- && y >= tab.rect.y
- && y <= tab.rect.y + tab.rect.height
- })
- .map(|tab| tab.workspace_id.as_str())
- }
-}
-
-pub fn layout_session_tabs(
- panel: LayoutRect,
- workspaces: &[PipWorkspaceSummary],
- selected_workspace_id: Option<&str>,
-) -> SessionTabsLayout {
- if workspaces.len() <= 1 {
- return SessionTabsLayout::default();
- }
- let visible = workspaces.len().min(MAX_VISIBLE_TABS);
- let gap = 6.0;
- let height = (panel.height * 0.09).clamp(28.0, 42.0);
- let width = ((panel.width - gap * (visible.saturating_sub(1)) as f64) / visible as f64)
- .clamp(90.0, 190.0);
- let tabs = workspaces
- .iter()
- .take(visible)
- .enumerate()
- .map(|(index, workspace)| SessionTab {
- workspace_id: workspace.workspace_id.clone(),
- label: workspace.workspace_label.clone(),
- rect: LayoutRect {
- x: panel.x + index as f64 * (width + gap),
- y: panel.y,
- width,
- height,
- },
- selected: selected_workspace_id == Some(workspace.workspace_id.as_str()),
- accent: session_accent(&workspace.workspace_id),
- })
- .collect();
- SessionTabsLayout {
- tabs,
- overflow: workspaces.len().saturating_sub(visible),
- }
-}
-
-pub fn session_accent(workspace_id: &str) -> (u8, u8, u8) {
- let hash = workspace_id
- .bytes()
- .fold(0u32, |hash, byte| hash.wrapping_mul(16777619) ^ byte as u32);
- const COLORS: [(u8, u8, u8); 6] = [
- (55, 148, 255),
- (255, 166, 54),
- (54, 190, 160),
- (228, 96, 135),
- (139, 112, 246),
- (66, 181, 225),
- ];
- COLORS[hash as usize % COLORS.len()]
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
-
- fn workspace(id: &str, updated_ms: u64) -> PipWorkspaceSummary {
- PipWorkspaceSummary {
- workspace_id: id.into(),
- workspace_label: id.into(),
- target_count: 1,
- updated_ms,
- }
- }
-
- #[test]
- fn hides_single_session_and_directly_hit_tests_tabs() {
- let panel = LayoutRect {
- x: 10.0,
- y: 12.0,
- width: 600.0,
- height: 400.0,
- };
- assert!(layout_session_tabs(panel, &[workspace("a", 1)], Some("a"))
- .tabs
- .is_empty());
- let layout = layout_session_tabs(panel, &[workspace("b", 2), workspace("a", 1)], Some("a"));
- assert_eq!(
- layout.hit_test(layout.tabs[0].rect.x + 2.0, layout.tabs[0].rect.y + 2.0),
- Some("b")
- );
- assert!(layout.tabs[1].selected);
- }
-
- #[test]
- fn caps_visible_tabs_and_reports_overflow() {
- let workspaces = (0..8)
- .map(|i| workspace(&format!("agent-{i}"), i))
- .collect::<Vec<_>>();
- let layout = layout_session_tabs(
- LayoutRect {
- x: 0.0,
- y: 0.0,
- width: 900.0,
- height: 500.0,
- },
- &workspaces,
- None,
- );
- assert_eq!(layout.tabs.len(), 6);
- assert_eq!(layout.overflow, 2);
- assert_eq!(session_accent("stable"), session_accent("stable"));
- }
-}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/pip/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/pip/mod.rs
index df35fe672a..805c0eb4e6 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/pip/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/pip/mod.rs
@@ -1,1348 +1,23 @@
-//! Linux multi-target Agent View.
+//! Linux picture-in-picture preview stub.
//!
-//! The backend uses a regular, resizable X11 utility window (and therefore
-//! also works through XWayland) with a software-rendered GNOME-like overview.
-//! It only presents frames supplied by the shared Agent View model; it never
-//! moves, resizes, focuses, or closes the represented applications.
-
-use std::sync::mpsc::{self, Receiver, Sender};
-use std::time::Duration;
-
-#[cfg(target_os = "linux")]
-mod wayland;
-
-#[cfg(target_os = "linux")]
-use image::imageops::FilterType;
-use pip_preview::{
- layout_desktop, layout_session_tabs, png_dimensions, LayoutRect, PipBackend, PipBackendFactory,
- PipConfig, PipFrame, PipTargetKind, PipViewModel, PipWorkspaceSummary, SessionTabsLayout,
- TargetSize,
-};
-
-const MIN_WIDTH: u16 = 360;
-const MIN_HEIGHT: u16 = 260;
-
-pub(super) enum UiMessage {
- Frame(PipFrame),
- RemoveTarget(String, String),
- RemoveWorkspace(String),
- SetInputTransparent(bool, mpsc::SyncSender<anyhow::Result<()>>),
- Shutdown,
-}
-
-pub struct LinuxPipBackend {
- tx: Sender<UiMessage>,
-}
-
-impl LinuxPipBackend {
- /// Synchronously include or exclude Agent View from X11 pointer hit-testing.
- ///
- /// Cua actions should enable transparency before injecting input and restore
- /// interaction afterward. Waiting for the X11 thread to flush the shape
- /// change prevents an always-on-top Agent View from intercepting the action.
- pub fn set_input_transparent(&self, transparent: bool) -> anyhow::Result<()> {
- let (reply_tx, reply_rx) = mpsc::sync_channel(1);
- if self
- .tx
- .send(UiMessage::SetInputTransparent(transparent, reply_tx))
- .is_err()
- {
- return Ok(());
- }
- reply_rx
- .recv_timeout(Duration::from_secs(2))
- .map_err(|error| {
- anyhow::anyhow!("timed out updating Agent View input shape: {error}")
- })?
- }
-}
-
-impl PipBackend for LinuxPipBackend {
- fn push_frame(&self, frame: PipFrame) {
- tracing::debug!(
- target: "pip",
- view_id = %frame.target.view_id(),
- "queueing Linux Agent View frame"
- );
- let _ = self.tx.send(UiMessage::Frame(frame));
- }
-
- fn remove_workspace(&self, workspace_id: &str) {
- let _ = self
- .tx
- .send(UiMessage::RemoveWorkspace(workspace_id.to_owned()));
- }
-
- fn set_input_passthrough(&self, passthrough: bool) -> anyhow::Result<()> {
- self.set_input_transparent(passthrough)
- }
-
- fn remove_target(&self, workspace_id: &str, identity_key: &str) {
- let _ = self.tx.send(UiMessage::RemoveTarget(
- workspace_id.to_owned(),
- identity_key.to_owned(),
- ));
- }
+//! The Linux implementation is a follow-up to the experimental macOS
+//! drop. Plan: GTK4 always-on-top utility window under X11/XWayland,
+//! with a Wayland-native fallback via `wlr-layer-shell` where
+//! supported. Tracking issue linked in the docs.
+//!
+//! Until that lands, `start()` returns a clear error so `main.rs`
+//! can log "PiP unavailable on this platform" and continue without
+//! the window.
- fn shutdown(self: Box<Self>) {
- let _ = self.tx.send(UiMessage::Shutdown);
- }
-}
+use pip_preview::{PipBackend, PipBackendFactory, PipConfig};
pub struct LinuxPipBackendFactory;
impl PipBackendFactory for LinuxPipBackendFactory {
- fn start(&self, cfg: &PipConfig) -> anyhow::Result<Box<dyn PipBackend>> {
- let (tx, rx) = mpsc::channel();
- let (ready_tx, ready_rx) = mpsc::sync_channel(1);
- let cfg = cfg.clone();
- std::thread::Builder::new()
- .name("cua-agent-view-linux".to_owned())
- .spawn(move || {
- #[cfg(target_os = "linux")]
- if should_use_native_wayland() {
- wayland::run_window(cfg, rx, ready_tx);
- return;
- }
- run_x11_window(cfg, rx, ready_tx)
- })?;
-
- ready_rx
- .recv()
- .map_err(|_| anyhow::anyhow!("Agent View X11 thread exited during startup"))??;
- Ok(Box::new(LinuxPipBackend { tx }))
- }
-}
-
-#[cfg(target_os = "linux")]
-fn should_use_native_wayland() -> bool {
- std::env::var_os("WAYLAND_DISPLAY").is_some()
- && (crate::wayland::is_wayland() || std::env::var_os("DISPLAY").is_none())
-}
-
-#[cfg(target_os = "linux")]
-fn run_x11_window(
- cfg: PipConfig,
- rx: Receiver<UiMessage>,
- ready: mpsc::SyncSender<anyhow::Result<()>>,
-) {
- use x11rb::connection::Connection;
- use x11rb::protocol::xproto::{
- AtomEnum, ConnectionExt as _, CreateGCAux, CreateWindowAux, EventMask, PropMode,
- WindowClass,
- };
- use x11rb::protocol::Event;
- use x11rb::wrapper::ConnectionExt as _;
-
- let setup = (|| -> anyhow::Result<_> {
- let (conn, screen_num) = x11rb::connect(None)?;
- let screen = &conn.setup().roots[screen_num];
- let root = screen.root;
- let root_depth = screen.root_depth;
- let root_visual = screen.root_visual;
- let screen_width = screen.width_in_pixels;
- let width = cfg
- .geometry
- .width
- .clamp(u32::from(MIN_WIDTH), u32::from(u16::MAX)) as u16;
- let height = cfg
- .geometry
- .height
- .clamp(u32::from(MIN_HEIGHT), u32::from(u16::MAX)) as u16;
- let inset = 24i16;
- let x = cfg.geometry.x.map(clamp_i16).unwrap_or_else(|| {
- clamp_i16(i32::from(screen_width) - i32::from(width) - i32::from(inset))
- });
- let y = cfg.geometry.y.map(clamp_i16).unwrap_or(inset);
- let window = conn.generate_id()?;
- conn.create_window(
- root_depth,
- window,
- root,
- x,
- y,
- width,
- height,
- 0,
- WindowClass::INPUT_OUTPUT,
- root_visual,
- &CreateWindowAux::new()
- .background_pixel(0x1820_27)
- .border_pixel(0)
- .event_mask(
- EventMask::EXPOSURE
- | EventMask::STRUCTURE_NOTIFY
- | EventMask::PROPERTY_CHANGE
- | EventMask::BUTTON_PRESS,
- ),
- )?
- .check()?;
-
- let gc = conn.generate_id()?;
- conn.create_gc(gc, window, &CreateGCAux::new())?.check()?;
- conn.change_property8(
- PropMode::REPLACE,
- window,
- AtomEnum::WM_NAME,
- AtomEnum::STRING,
- cfg.title.as_bytes(),
- )?;
- conn.change_property8(
- PropMode::REPLACE,
- window,
- AtomEnum::WM_CLASS,
- AtomEnum::STRING,
- b"cua-agent-view\0CuaAgentView\0",
- )?;
- let net_wm_pid = conn.intern_atom(false, b"_NET_WM_PID")?.reply()?.atom;
- conn.change_property32(
- PropMode::REPLACE,
- window,
- net_wm_pid,
- AtomEnum::CARDINAL,
- &[std::process::id()],
- )?;
-
- let wm_protocols = conn.intern_atom(false, b"WM_PROTOCOLS")?.reply()?.atom;
- let wm_delete = conn.intern_atom(false, b"WM_DELETE_WINDOW")?.reply()?.atom;
- conn.change_property32(
- PropMode::REPLACE,
- window,
- wm_protocols,
- AtomEnum::ATOM,
- &[wm_delete],
- )?;
- let net_wm_state = conn.intern_atom(false, b"_NET_WM_STATE")?.reply()?.atom;
- let net_wm_state_above = conn
- .intern_atom(false, b"_NET_WM_STATE_ABOVE")?
- .reply()?
- .atom;
- conn.change_property32(
- PropMode::REPLACE,
- window,
- net_wm_state,
- AtomEnum::ATOM,
- &[net_wm_state_above],
- )?;
- let net_wm_window_type = conn
- .intern_atom(false, b"_NET_WM_WINDOW_TYPE")?
- .reply()?
- .atom;
- let net_wm_window_type_utility = conn
- .intern_atom(false, b"_NET_WM_WINDOW_TYPE_UTILITY")?
- .reply()?
- .atom;
- conn.change_property32(
- PropMode::REPLACE,
- window,
- net_wm_window_type,
- AtomEnum::ATOM,
- &[net_wm_window_type_utility],
- )?;
- conn.map_window(window)?.check()?;
- conn.flush()?;
- Ok((
- conn,
- window,
- gc,
- root_depth,
- wm_protocols,
- wm_delete,
- width,
- height,
+ fn start(&self, _cfg: &PipConfig) -> anyhow::Result<Box<dyn PipBackend>> {
+ Err(anyhow::anyhow!(
+ "PiP preview is not yet implemented on Linux — track \
+ trycua/cua follow-up issue for status"
))
- })();
-
- let (conn, window, gc, depth, wm_protocols, wm_delete, mut width, mut height) = match setup {
- Ok(values) => {
- let _ = ready.send(Ok(()));
- values
- }
- Err(error) => {
- let _ = ready.send(Err(error));
- return;
- }
- };
-
- let mut model = PipViewModel::new(12);
- let mut dirty = true;
- let mut running = true;
- while running {
- while let Ok(message) = rx.try_recv() {
- handle_x11_message(
- &conn,
- window,
- width,
- height,
- message,
- &mut model,
- &mut dirty,
- &mut running,
- );
- }
-
- loop {
- match conn.poll_for_event() {
- Ok(Some(Event::Expose(_))) => dirty = true,
- Ok(Some(Event::ConfigureNotify(event))) => {
- let next_width = event.width.max(1);
- let next_height = event.height.max(1);
- if next_width != width || next_height != height {
- width = next_width;
- height = next_height;
- dirty = true;
- }
- }
- Ok(Some(Event::ButtonPress(event))) => {
- let workspaces = model.workspaces();
- let tabs = session_tabs_for_bounds(
- width,
- height,
- &workspaces,
- model.selected_workspace_id(),
- );
- if let Some(workspace_id) =
- tabs.hit_test(f64::from(event.event_x), f64::from(event.event_y))
- {
- dirty |= model.select_workspace(workspace_id);
- }
- }
- Ok(Some(Event::ClientMessage(event)))
- if is_delete_message(&event, wm_protocols, wm_delete) =>
- {
- running = false;
- }
- Ok(Some(_)) => {}
- Ok(None) => break,
- Err(error) => {
- tracing::warn!(target: "pip", "Agent View X11 event error: {error}");
- running = false;
- break;
- }
- }
- }
-
- if dirty && running {
- let frames = model.selected_frames();
- let workspaces = model.workspaces();
- let pixels = render_agent_view(
- width,
- height,
- &frames,
- &workspaces,
- model.selected_workspace_id(),
- model.active_view_id(),
- );
- if let Err(error) = upload_image(&conn, window, gc, depth, width, height, &pixels) {
- tracing::warn!(target: "pip", "Agent View X11 paint failed: {error}");
- running = false;
- }
- dirty = false;
- }
-
- if running {
- match rx.recv_timeout(Duration::from_millis(16)) {
- Ok(message) => handle_x11_message(
- &conn,
- window,
- width,
- height,
- message,
- &mut model,
- &mut dirty,
- &mut running,
- ),
- Err(mpsc::RecvTimeoutError::Timeout) => {}
- Err(mpsc::RecvTimeoutError::Disconnected) => running = false,
- }
- }
- }
-
- let _ = conn.free_gc(gc);
- let _ = conn.destroy_window(window);
- let _ = conn.flush();
-}
-
-#[cfg(target_os = "linux")]
-fn handle_x11_message(
- conn: &impl x11rb::connection::Connection,
- window: u32,
- width: u16,
- height: u16,
- message: UiMessage,
- model: &mut PipViewModel,
- dirty: &mut bool,
- running: &mut bool,
-) {
- match message {
- UiMessage::SetInputTransparent(transparent, reply) => {
- let result = set_x11_input_transparent(conn, window, width, height, transparent);
- let _ = reply.send(result);
- }
- message => handle_message(message, model, dirty, running),
- }
-}
-
-#[cfg(target_os = "linux")]
-fn set_x11_input_transparent(
- conn: &impl x11rb::connection::Connection,
- window: u32,
- width: u16,
- height: u16,
- transparent: bool,
-) -> anyhow::Result<()> {
- use x11rb::protocol::shape::{ConnectionExt as ShapeConnectionExt, SK, SO};
- use x11rb::protocol::xproto::{ClipOrdering, Rectangle};
-
- let interactive_region = [Rectangle {
- x: 0,
- y: 0,
- width,
- height,
- }];
- let rectangles = if transparent {
- &[][..]
- } else {
- &interactive_region[..]
- };
- conn.shape_rectangles(
- SO::SET,
- SK::INPUT,
- ClipOrdering::UNSORTED,
- window,
- 0,
- 0,
- rectangles,
- )?
- .check()?;
- conn.flush()?;
- Ok(())
-}
-
-#[cfg(target_os = "linux")]
-fn upload_image(
- conn: &impl x11rb::connection::Connection,
- window: u32,
- gc: u32,
- depth: u8,
- width: u16,
- height: u16,
- pixels: &[u8],
-) -> anyhow::Result<()> {
- use x11rb::protocol::xproto::{ConnectionExt as _, ImageFormat};
-
- let stride = usize::from(width) * 4;
- // Keep each PutImage below the common 256 KiB X11 request limit.
- let rows_per_request = (200_000usize / stride.max(1)).max(1);
- for start_row in (0..usize::from(height)).step_by(rows_per_request) {
- let rows = rows_per_request.min(usize::from(height) - start_row);
- let start = start_row * stride;
- let end = start + rows * stride;
- conn.put_image(
- ImageFormat::Z_PIXMAP,
- window,
- gc,
- width,
- rows as u16,
- 0,
- start_row as i16,
- 0,
- depth,
- &pixels[start..end],
- )?
- .check()?;
- }
- conn.flush()?;
- Ok(())
-}
-
-fn handle_message(
- message: UiMessage,
- model: &mut PipViewModel,
- dirty: &mut bool,
- running: &mut bool,
-) {
- match message {
- UiMessage::Frame(frame) => {
- tracing::debug!(
- target: "pip",
- view_id = %frame.target.view_id(),
- "rendering Linux Agent View frame"
- );
- model.upsert(frame);
- *dirty = true;
- }
- UiMessage::RemoveTarget(workspace_id, identity_key) => {
- *dirty |= model.remove_target(&workspace_id, &identity_key);
- }
- UiMessage::RemoveWorkspace(workspace_id) => {
- *dirty |= !model.remove_workspace(&workspace_id).is_empty();
- }
- UiMessage::SetInputTransparent(_, reply) => {
- let _ = reply.send(Err(anyhow::anyhow!(
- "Agent View input shape update reached a non-X11 handler"
- )));
- }
- UiMessage::Shutdown => *running = false,
- }
-}
-
-#[cfg(not(target_os = "linux"))]
-fn run_x11_window(
- _cfg: PipConfig,
- _rx: Receiver<UiMessage>,
- ready: mpsc::SyncSender<anyhow::Result<()>>,
-) {
- let _ = ready.send(Err(anyhow::anyhow!(
- "Linux Agent View can only start on Linux"
- )));
-}
-
-#[cfg(target_os = "linux")]
-fn is_delete_message(
- event: &x11rb::protocol::xproto::ClientMessageEvent,
- wm_protocols: u32,
- wm_delete: u32,
-) -> bool {
- event.type_ == wm_protocols && event.data.as_data32()[0] == wm_delete
-}
-
-fn clamp_i16(value: i32) -> i16 {
- value.clamp(i32::from(i16::MIN), i32::from(i16::MAX)) as i16
-}
-
-pub(super) fn render_agent_view(
- width: u16,
- height: u16,
- frames: &[&PipFrame],
- workspaces: &[PipWorkspaceSummary],
- selected_workspace_id: Option<&str>,
- active_view_id: Option<&str>,
-) -> Vec<u8> {
- let mut canvas = Canvas::new(width, height);
- canvas.vertical_gradient(Color::rgb(45, 57, 66), Color::rgb(24, 29, 34));
- canvas.radial_glow(
- width as f64 * 0.18,
- height as f64 * 0.02,
- width as f64 * 0.72,
- Color::rgba(93, 129, 145, 70),
- );
- canvas.radial_glow(
- width as f64 * 0.88,
- height as f64 * 0.72,
- width as f64 * 0.55,
- Color::rgba(113, 90, 72, 42),
- );
- canvas.rounded_rect(
- Rect::new(1.0, 1.0, width as f64 - 2.0, height as f64 - 2.0),
- 15.0,
- Color::rgba(15, 18, 21, 142),
- );
- canvas.stroke_rounded_rect(
- Rect::new(1.5, 1.5, width as f64 - 3.0, height as f64 - 3.0),
- 14.5,
- 1.0,
- Color::rgba(210, 221, 226, 92),
- );
-
- let sizes = frames
- .iter()
- .map(|frame| {
- png_dimensions(&frame.png_bytes).unwrap_or(TargetSize {
- width: 16,
- height: 10,
- })
- })
- .collect::<Vec<_>>();
- let selector_height = if workspaces.len() > 1 { 34.0 } else { 0.0 };
- let mut layout = layout_desktop(
- width.into(),
- (f64::from(height) - selector_height).max(1.0),
- &sizes,
- );
- offset_layout_y(&mut layout, selector_height);
- if frames.is_empty() {
- render_waiting_state(&mut canvas, layout.desktop);
- } else {
- for (frame, target) in frames.iter().zip(&layout.targets) {
- render_target(&mut canvas, frame, target.window);
- if active_view_id == Some(frame.target.view_id().as_str()) {
- canvas.stroke_rounded_rect(
- Rect::from_layout(target.window).expand(3.0),
- 13.0,
- 2.0,
- Color::rgba(55, 148, 255, 242),
- );
- }
- }
- // Agent View has no permanent launcher; session tabs remain the only
- // presentation-level navigation surface.
- }
- render_session_selector(
- &mut canvas,
- width,
- height,
- workspaces,
- selected_workspace_id,
- );
- render_resize_affordance(&mut canvas, width, height);
- canvas.into_bgrx()
-}
-
-fn render_waiting_state(canvas: &mut Canvas, desktop: LayoutRect) {
- let center_x = desktop.x + desktop.width / 2.0;
- let center_y = desktop.y + desktop.height / 2.0;
- canvas.circle(center_x, center_y - 7.0, 22.0, Color::rgba(48, 59, 67, 220));
- canvas.stroke_circle(
- center_x,
- center_y - 7.0,
- 22.0,
- 1.0,
- Color::rgba(223, 230, 233, 88),
- );
- for offset in [-8.0, 0.0, 8.0] {
- canvas.circle(
- center_x + offset,
- center_y - 7.0,
- 2.1,
- Color::rgba(235, 239, 241, 170),
- );
- }
- canvas.rounded_rect(
- Rect::new(center_x - 58.0, center_y + 25.0, 116.0, 3.0),
- 1.5,
- Color::rgba(224, 230, 232, 45),
- );
-}
-
-fn render_target(canvas: &mut Canvas, frame: &PipFrame, rect: LayoutRect) {
- let rect = Rect::from_layout(rect);
- canvas.shadow(rect, 9.0, 6.0, Color::rgba(0, 0, 0, 118));
- canvas.rounded_rect(rect.expand(1.0), 8.0, Color::rgba(220, 225, 227, 68));
- canvas.rounded_rect(rect, 7.0, Color::rgb(21, 25, 28));
- if !draw_frame_image(canvas, frame, rect) {
- canvas.vertical_gradient_in(rect, Color::rgb(47, 55, 61), Color::rgb(27, 31, 35));
- let accent = target_accent(frame.target.target_kind);
- canvas.circle(
- rect.x + rect.width / 2.0,
- rect.y + rect.height / 2.0,
- 12.0,
- accent,
- );
- }
- canvas.stroke_rounded_rect(rect, 7.0, 1.0, Color::rgba(225, 230, 232, 76));
-}
-
-#[cfg(target_os = "linux")]
-fn draw_frame_image(canvas: &mut Canvas, frame: &PipFrame, rect: Rect) -> bool {
- let Ok(image) = image::load_from_memory(&frame.png_bytes) else {
- return false;
- };
- canvas.draw_image_contain(&image.to_rgba8(), rect, 6.5);
- true
-}
-
-#[cfg(not(target_os = "linux"))]
-fn draw_frame_image(_canvas: &mut Canvas, _frame: &PipFrame, _rect: Rect) -> bool {
- false
-}
-
-fn session_tabs_for_bounds(
- width: u16,
- height: u16,
- workspaces: &[PipWorkspaceSummary],
- selected_workspace_id: Option<&str>,
-) -> SessionTabsLayout {
- let visible = workspaces.len().min(6);
- let desired_width = visible as f64 * 190.0 + visible.saturating_sub(1) as f64 * 6.0;
- let panel_width = desired_width.min((f64::from(width) - 16.0).max(1.0));
- layout_session_tabs(
- LayoutRect {
- x: ((f64::from(width) - panel_width) / 2.0).max(0.0),
- y: 4.0_f64.min(f64::from(height).max(0.0)),
- width: panel_width,
- height: 34.0,
- },
- workspaces,
- selected_workspace_id,
- )
-}
-
-fn render_session_selector(
- canvas: &mut Canvas,
- width: u16,
- height: u16,
- workspaces: &[PipWorkspaceSummary],
- selected_workspace_id: Option<&str>,
-) {
- let tabs = session_tabs_for_bounds(width, height, workspaces, selected_workspace_id);
- let Some(first) = tabs.tabs.first() else {
- return;
- };
- let last = tabs.tabs.last().expect("session tabs are non-empty");
- let dock = Rect::new(
- first.rect.x - 6.0,
- first.rect.y - 3.0,
- last.rect.x + last.rect.width - first.rect.x + 12.0,
- first.rect.height + 6.0,
- );
- canvas.shadow(dock, 12.0, 5.0, Color::rgba(0, 0, 0, 100));
- canvas.rounded_rect(dock, dock.height * 0.27, Color::rgba(67, 70, 72, 195));
- canvas.stroke_rounded_rect(
- dock,
- dock.height * 0.27,
- 1.0,
- Color::rgba(231, 235, 236, 66),
- );
- for (workspace, tab) in workspaces.iter().zip(&tabs.tabs) {
- let icon = Rect::from_layout(tab.rect);
- let accent = Color::rgb(tab.accent.0, tab.accent.1, tab.accent.2);
- if tab.selected {
- canvas.rounded_rect(
- icon.expand(2.0),
- icon.width * 0.27,
- Color::rgba(245, 248, 249, 220),
- );
- }
- canvas.rounded_rect(icon, icon.width * 0.23, accent);
- let glyph = icon.inset(icon.width * 0.23);
- canvas.circle(
- glyph.x + glyph.width / 2.0,
- glyph.y + glyph.height * 0.36,
- glyph.width * 0.22,
- Color::rgba(250, 252, 253, 235),
- );
- canvas.rounded_rect(
- Rect::new(
- glyph.x + glyph.width * 0.16,
- glyph.y + glyph.height * 0.62,
- glyph.width * 0.68,
- glyph.height * 0.28,
- ),
- glyph.width * 0.14,
- Color::rgba(250, 252, 253, 225),
- );
- let indicators = workspace.target_count.min(3);
- for index in 0..indicators {
- canvas.circle(
- icon.x + icon.width / 2.0 + (index as f64 - (indicators - 1) as f64 / 2.0) * 5.0,
- dock.y + dock.height - 4.0,
- 1.4,
- Color::rgba(242, 245, 246, if tab.selected { 245 } else { 145 }),
- );
- }
- }
-}
-
-fn offset_layout_y(layout: &mut pip_preview::DesktopLayout, offset: f64) {
- layout.desktop.y += offset;
- layout.dock.y += offset;
- for icon in &mut layout.dock_icons {
- icon.y += offset;
- }
- for target in &mut layout.targets {
- target.window.y += offset;
- target.content.y += offset;
- }
-}
-
-fn render_dash(canvas: &mut Canvas, frames: &[&PipFrame], dock: &LayoutRect, icons: &[LayoutRect]) {
- let dock = Rect::from_layout(*dock);
- canvas.shadow(dock, 12.0, 5.0, Color::rgba(0, 0, 0, 100));
- canvas.rounded_rect(dock, dock.height * 0.27, Color::rgba(67, 70, 72, 195));
- canvas.stroke_rounded_rect(
- dock,
- dock.height * 0.27,
- 1.0,
- Color::rgba(231, 235, 236, 66),
- );
- for (frame, icon) in frames.iter().zip(icons) {
- let icon = Rect::from_layout(*icon);
- let accent = target_accent(frame.target.target_kind);
- canvas.rounded_rect(icon, icon.width * 0.23, accent);
- let glyph = icon.inset(icon.width * 0.23);
- match frame.target.target_kind {
- PipTargetKind::BrowserTab => {
- canvas.stroke_circle(
- glyph.x + glyph.width / 2.0,
- glyph.y + glyph.height / 2.0,
- glyph.width * 0.44,
- (glyph.width * 0.08).max(1.0),
- Color::rgba(250, 252, 253, 235),
- );
- canvas.line(
- glyph.x + glyph.width / 2.0,
- glyph.y,
- glyph.x + glyph.width / 2.0,
- glyph.y + glyph.height,
- (glyph.width * 0.07).max(1.0),
- Color::rgba(250, 252, 253, 215),
- );
- }
- PipTargetKind::NativeWindow => {
- canvas.rounded_rect(glyph, glyph.width * 0.12, Color::rgba(250, 252, 253, 228));
- canvas.rect(
- Rect::new(
- glyph.x + glyph.width * 0.12,
- glyph.y + glyph.height * 0.25,
- glyph.width * 0.76,
- (glyph.height * 0.09).max(1.0),
- ),
- accent,
- );
- }
- }
- canvas.circle(
- icon.x + icon.width / 2.0,
- dock.y + dock.height - 4.0,
- 1.6,
- Color::rgba(242, 245, 246, 225),
- );
- }
-}
-
-fn render_resize_affordance(canvas: &mut Canvas, width: u16, height: u16) {
- let x = f64::from(width) - 15.0;
- let y = f64::from(height) - 8.0;
- let color = Color::rgba(226, 231, 233, 116);
- canvas.line(x, y, x + 7.0, y - 7.0, 1.2, color);
- canvas.line(x + 5.0, y, x + 9.0, y - 4.0, 1.2, color);
-}
-
-fn target_accent(kind: PipTargetKind) -> Color {
- match kind {
- PipTargetKind::BrowserTab => Color::rgb(53, 154, 220),
- PipTargetKind::NativeWindow => Color::rgb(239, 112, 54),
- }
-}
-
-#[derive(Clone, Copy, Debug)]
-struct Rect {
- x: f64,
- y: f64,
- width: f64,
- height: f64,
-}
-
-impl Rect {
- fn new(x: f64, y: f64, width: f64, height: f64) -> Self {
- Self {
- x,
- y,
- width: width.max(0.0),
- height: height.max(0.0),
- }
- }
-
- fn from_layout(rect: LayoutRect) -> Self {
- Self::new(rect.x, rect.y, rect.width, rect.height)
- }
-
- fn inset(self, amount: f64) -> Self {
- Self::new(
- self.x + amount,
- self.y + amount,
- self.width - amount * 2.0,
- self.height - amount * 2.0,
- )
- }
-
- fn expand(self, amount: f64) -> Self {
- Self::new(
- self.x - amount,
- self.y - amount,
- self.width + amount * 2.0,
- self.height + amount * 2.0,
- )
- }
-}
-
-#[derive(Clone, Copy, Debug, PartialEq, Eq)]
-struct Color {
- r: u8,
- g: u8,
- b: u8,
- a: u8,
-}
-
-impl Color {
- const fn rgb(r: u8, g: u8, b: u8) -> Self {
- Self { r, g, b, a: 255 }
- }
-
- const fn rgba(r: u8, g: u8, b: u8, a: u8) -> Self {
- Self { r, g, b, a }
- }
-}
-
-struct Canvas {
- width: usize,
- height: usize,
- rgba: Vec<u8>,
-}
-
-impl Canvas {
- fn new(width: u16, height: u16) -> Self {
- Self {
- width: usize::from(width),
- height: usize::from(height),
- rgba: vec![0; usize::from(width) * usize::from(height) * 4],
- }
- }
-
- fn into_bgrx(mut self) -> Vec<u8> {
- for pixel in self.rgba.chunks_exact_mut(4) {
- pixel.swap(0, 2);
- pixel[3] = 0;
- }
- self.rgba
- }
-
- fn blend(&mut self, x: i32, y: i32, color: Color) {
- if x < 0 || y < 0 || x >= self.width as i32 || y >= self.height as i32 {
- return;
- }
- let index = (y as usize * self.width + x as usize) * 4;
- let alpha = u32::from(color.a);
- let inverse = 255 - alpha;
- self.rgba[index] =
- ((u32::from(color.r) * alpha + u32::from(self.rgba[index]) * inverse) / 255) as u8;
- self.rgba[index + 1] =
- ((u32::from(color.g) * alpha + u32::from(self.rgba[index + 1]) * inverse) / 255) as u8;
- self.rgba[index + 2] =
- ((u32::from(color.b) * alpha + u32::from(self.rgba[index + 2]) * inverse) / 255) as u8;
- self.rgba[index + 3] = 255;
- }
-
- fn vertical_gradient(&mut self, top: Color, bottom: Color) {
- self.vertical_gradient_in(
- Rect::new(0.0, 0.0, self.width as f64, self.height as f64),
- top,
- bottom,
- );
- }
-
- fn vertical_gradient_in(&mut self, rect: Rect, top: Color, bottom: Color) {
- let (x0, y0, x1, y1) = self.bounds(rect);
- let height = (y1 - y0).max(1) as f64;
- for y in y0..y1 {
- let t = f64::from(y - y0) / height;
- let color = lerp_color(top, bottom, t);
- for x in x0..x1 {
- self.blend(x, y, color);
- }
- }
- }
-
- fn radial_glow(&mut self, cx: f64, cy: f64, radius: f64, color: Color) {
- let rect = Rect::new(cx - radius, cy - radius, radius * 2.0, radius * 2.0);
- let (x0, y0, x1, y1) = self.bounds(rect);
- for y in y0..y1 {
- for x in x0..x1 {
- let distance = ((f64::from(x) - cx).powi(2) + (f64::from(y) - cy).powi(2)).sqrt();
- if distance < radius {
- let mut glow = color;
- glow.a = (f64::from(color.a) * (1.0 - distance / radius).powi(2)) as u8;
- self.blend(x, y, glow);
- }
- }
- }
- }
-
- fn rect(&mut self, rect: Rect, color: Color) {
- let (x0, y0, x1, y1) = self.bounds(rect);
- for y in y0..y1 {
- for x in x0..x1 {
- self.blend(x, y, color);
- }
- }
- }
-
- fn rounded_rect(&mut self, rect: Rect, radius: f64, color: Color) {
- let (x0, y0, x1, y1) = self.bounds(rect);
- for y in y0..y1 {
- for x in x0..x1 {
- if rounded_contains(rect, radius, f64::from(x) + 0.5, f64::from(y) + 0.5) {
- self.blend(x, y, color);
- }
- }
- }
- }
-
- fn stroke_rounded_rect(&mut self, rect: Rect, radius: f64, width: f64, color: Color) {
- let inner = rect.inset(width);
- let (x0, y0, x1, y1) = self.bounds(rect);
- for y in y0..y1 {
- for x in x0..x1 {
- let px = f64::from(x) + 0.5;
- let py = f64::from(y) + 0.5;
- if rounded_contains(rect, radius, px, py)
- && !rounded_contains(inner, (radius - width).max(0.0), px, py)
- {
- self.blend(x, y, color);
- }
- }
- }
- }
-
- fn shadow(&mut self, rect: Rect, radius: f64, y_offset: f64, color: Color) {
- let steps = 6;
- for step in (1..=steps).rev() {
- let spread = radius * f64::from(step) / f64::from(steps);
- let mut layer = color;
- layer.a = color.a / (steps as u8 + 1);
- self.rounded_rect(
- Rect::new(
- rect.x - spread,
- rect.y - spread + y_offset,
- rect.width + spread * 2.0,
- rect.height + spread * 2.0,
- ),
- 7.0 + spread,
- layer,
- );
- }
- }
-
- fn circle(&mut self, cx: f64, cy: f64, radius: f64, color: Color) {
- let (x0, y0, x1, y1) = self.bounds(Rect::new(
- cx - radius,
- cy - radius,
- radius * 2.0,
- radius * 2.0,
- ));
- let radius_squared = radius * radius;
- for y in y0..y1 {
- for x in x0..x1 {
- if (f64::from(x) + 0.5 - cx).powi(2) + (f64::from(y) + 0.5 - cy).powi(2)
- <= radius_squared
- {
- self.blend(x, y, color);
- }
- }
- }
- }
-
- fn stroke_circle(&mut self, cx: f64, cy: f64, radius: f64, width: f64, color: Color) {
- let inner = (radius - width).max(0.0);
- let (x0, y0, x1, y1) = self.bounds(Rect::new(
- cx - radius,
- cy - radius,
- radius * 2.0,
- radius * 2.0,
- ));
- for y in y0..y1 {
- for x in x0..x1 {
- let distance =
- ((f64::from(x) + 0.5 - cx).powi(2) + (f64::from(y) + 0.5 - cy).powi(2)).sqrt();
- if distance >= inner && distance <= radius {
- self.blend(x, y, color);
- }
- }
- }
- }
-
- fn line(&mut self, x0: f64, y0: f64, x1: f64, y1: f64, width: f64, color: Color) {
- let steps = ((x1 - x0).abs().max((y1 - y0).abs()).ceil() as usize).max(1);
- for step in 0..=steps {
- let t = step as f64 / steps as f64;
- self.circle(x0 + (x1 - x0) * t, y0 + (y1 - y0) * t, width / 2.0, color);
- }
- }
-
- #[cfg(target_os = "linux")]
- fn draw_image_contain(&mut self, image: &image::RgbaImage, rect: Rect, radius: f64) {
- let target_width = rect.width.ceil().max(1.0) as u32;
- let target_height = rect.height.ceil().max(1.0) as u32;
- let scale = (target_width as f64 / image.width() as f64)
- .min(target_height as f64 / image.height() as f64);
- let scaled_width = (image.width() as f64 * scale).floor().max(1.0) as u32;
- let scaled_height = (image.height() as f64 * scale).floor().max(1.0) as u32;
- let scaled =
- image::imageops::resize(image, scaled_width, scaled_height, FilterType::Triangle);
- let image_rect = Rect::new(
- rect.x + (rect.width - f64::from(scaled_width)) / 2.0,
- rect.y + (rect.height - f64::from(scaled_height)) / 2.0,
- f64::from(scaled_width),
- f64::from(scaled_height),
- );
- let (x0, y0, x1, y1) = self.bounds(image_rect);
- for y in y0..y1 {
- for x in x0..x1 {
- let px = f64::from(x) + 0.5;
- let py = f64::from(y) + 0.5;
- if !rounded_contains(rect, radius, px, py) {
- continue;
- }
- let source_x = (f64::from(x) + 0.5 - image_rect.x).floor().max(0.0) as u32;
- let source_y = (f64::from(y) + 0.5 - image_rect.y).floor().max(0.0) as u32;
- if source_x < scaled.width() && source_y < scaled.height() {
- let pixel = scaled.get_pixel(source_x, source_y);
- self.blend(x, y, Color::rgba(pixel[0], pixel[1], pixel[2], pixel[3]));
- }
- }
- }
- }
-
- fn bounds(&self, rect: Rect) -> (i32, i32, i32, i32) {
- (
- rect.x.floor().max(0.0) as i32,
- rect.y.floor().max(0.0) as i32,
- (rect.x + rect.width).ceil().min(self.width as f64) as i32,
- (rect.y + rect.height).ceil().min(self.height as f64) as i32,
- )
- }
-}
-
-fn rounded_contains(rect: Rect, radius: f64, x: f64, y: f64) -> bool {
- if x < rect.x || y < rect.y || x >= rect.x + rect.width || y >= rect.y + rect.height {
- return false;
- }
- let radius = radius.min(rect.width / 2.0).min(rect.height / 2.0).max(0.0);
- // Equivalent half-width bounds can differ by a sub-ULP after arithmetic;
- // max/min preserves the rounded geometry without f64::clamp panicking.
- let nearest_x = x.max(rect.x + radius).min(rect.x + rect.width - radius);
- let nearest_y = y.max(rect.y + radius).min(rect.y + rect.height - radius);
- (x - nearest_x).powi(2) + (y - nearest_y).powi(2) <= radius * radius
-}
-
-fn lerp_color(from: Color, to: Color, t: f64) -> Color {
- let channel = |a: u8, b: u8| (f64::from(a) + (f64::from(b) - f64::from(a)) * t) as u8;
- Color::rgba(
- channel(from.r, to.r),
- channel(from.g, to.g),
- channel(from.b, to.b),
- channel(from.a, to.a),
- )
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
-
- fn workspace(id: &str) -> PipWorkspaceSummary {
- PipWorkspaceSummary {
- workspace_id: id.to_owned(),
- workspace_label: id.to_owned(),
- target_count: 1,
- updated_ms: 0,
- }
- }
-
- #[test]
- fn renderer_emits_x11_bgrx_at_the_requested_size() {
- let rendered = render_agent_view(360, 260, &[], &[], None, None);
- assert_eq!(rendered.len(), 360 * 260 * 4);
- assert!(rendered.chunks_exact(4).all(|pixel| pixel[3] == 0));
- }
-
- #[test]
- fn session_selector_is_hidden_for_zero_or_one_workspace() {
- assert!(session_tabs_for_bounds(720, 520, &[], None).tabs.is_empty());
- assert!(
- session_tabs_for_bounds(720, 520, &[workspace("one")], Some("one"))
- .tabs
- .is_empty()
- );
- assert_eq!(
- session_tabs_for_bounds(720, 520, &[workspace("one"), workspace("two")], Some("one"),)
- .tabs
- .len(),
- 2
- );
- }
-
- #[test]
- fn session_selector_hit_testing_only_accepts_session_tabs() {
- let workspaces = [workspace("one"), workspace("two"), workspace("three")];
- let tabs = session_tabs_for_bounds(720, 520, &workspaces, Some("one"));
- let first = tabs.tabs[0].rect;
- assert_eq!(
- tabs.hit_test(first.x + first.width / 2.0, first.y + first.height / 2.0),
- Some("one")
- );
- assert_eq!(
- tabs.hit_test(first.x - 2.0, first.y + first.height / 2.0),
- None
- );
- }
-
- #[cfg(target_os = "linux")]
- #[test]
- #[ignore = "requires a live pure-Wayland compositor with layer-shell"]
- fn native_wayland_agent_view_starts_and_stops() {
- assert!(std::env::var_os("WAYLAND_DISPLAY").is_some());
- assert!(std::env::var_os("DISPLAY").is_none());
- let backend = LinuxPipBackendFactory
- .start(&PipConfig::default())
- .expect("native Wayland Agent View should start");
- backend.shutdown();
- }
-
- #[test]
- fn target_kind_uses_distinct_gnome_dash_accents() {
- assert_ne!(
- target_accent(PipTargetKind::NativeWindow),
- target_accent(PipTargetKind::BrowserTab)
- );
- }
-
- #[test]
- fn rounded_rect_clips_its_corner() {
- let rect = Rect::new(10.0, 10.0, 40.0, 30.0);
- assert!(!rounded_contains(rect, 8.0, 10.0, 10.0));
- assert!(rounded_contains(rect, 8.0, 18.0, 10.5));
- assert!(rounded_contains(rect, 8.0, 30.0, 25.0));
- }
-
- #[test]
- fn default_position_clamps_to_x11_coordinates() {
- assert_eq!(clamp_i16(i32::MAX), i16::MAX);
- assert_eq!(clamp_i16(i32::MIN), i16::MIN);
- assert_eq!(clamp_i16(24), 24);
- }
-
- #[cfg(target_os = "linux")]
- #[test]
- #[ignore = "requires a live X11 server (run under xvfb-run)"]
- fn x11_input_transparency_toggle_changes_input_shape() -> anyhow::Result<()> {
- use x11rb::connection::Connection;
- use x11rb::protocol::shape::{ConnectionExt as ShapeConnectionExt, SK};
- use x11rb::protocol::xproto::{ConnectionExt as _, CreateWindowAux, WindowClass};
-
- let (conn, screen_num) = x11rb::connect(None)?;
- let screen = &conn.setup().roots[screen_num];
- let window = conn.generate_id()?;
- let width = 320;
- let height = 240;
- conn.create_window(
- screen.root_depth,
- window,
- screen.root,
- 0,
- 0,
- width,
- height,
- 0,
- WindowClass::INPUT_OUTPUT,
- screen.root_visual,
- &CreateWindowAux::new(),
- )?
- .check()?;
- conn.map_window(window)?.check()?;
- conn.flush()?;
-
- set_x11_input_transparent(&conn, window, width, height, true)?;
- let transparent = conn.shape_get_rectangles(window, SK::INPUT)?.reply()?;
- assert!(transparent.rectangles.is_empty());
-
- set_x11_input_transparent(&conn, window, width, height, false)?;
- let interactive = conn.shape_get_rectangles(window, SK::INPUT)?.reply()?;
- assert_eq!(interactive.rectangles.len(), 1);
- assert_eq!(interactive.rectangles[0].width, width);
- assert_eq!(interactive.rectangles[0].height, height);
-
- conn.destroy_window(window)?.check()?;
- conn.flush()?;
- Ok(())
- }
-
- #[cfg(target_os = "linux")]
- #[test]
- #[ignore = "requires an interactive X11 display"]
- fn x11_visual_smoke_presents_multiple_targets() {
- use std::io::Cursor;
-
- use image::{DynamicImage, ImageFormat, Rgba, RgbaImage};
- use pip_preview::{PipGeometry, PipTarget};
-
- fn png(width: u32, height: u32, base: [u8; 3]) -> Vec<u8> {
- let mut image = RgbaImage::new(width, height);
- for (x, y, pixel) in image.enumerate_pixels_mut() {
- let shade = ((x * 31 / width.max(1) + y * 23 / height.max(1)) % 44) as u8;
- *pixel = Rgba([
- base[0].saturating_add(shade),
- base[1].saturating_add(shade),
- base[2].saturating_add(shade),
- 255,
- ]);
- }
- let mut bytes = Cursor::new(Vec::new());
- DynamicImage::ImageRgba8(image)
- .write_to(&mut bytes, ImageFormat::Png)
- .unwrap();
- bytes.into_inner()
- }
-
- let config = PipConfig {
- enabled: true,
- geometry: PipGeometry {
- width: 720,
- height: 520,
- x: Some(1080),
- y: Some(80),
- },
- title: "Cua Agent View Linux visual smoke".to_owned(),
- };
- let backend = LinuxPipBackendFactory.start(&config).unwrap();
- for (index, (workspace_id, workspace_label, width, height, base, kind)) in [
- (
- "linux-browser",
- "Browser session",
- 1280,
- 760,
- [20, 72, 92],
- PipTargetKind::BrowserTab,
- ),
- (
- "linux-browser",
- "Browser session",
- 760,
- 760,
- [25, 91, 67],
- PipTargetKind::BrowserTab,
- ),
- (
- "linux-native",
- "Native session",
- 840,
- 1060,
- [47, 58, 65],
- PipTargetKind::NativeWindow,
- ),
- (
- "linux-native",
- "Native session",
- 1440,
- 800,
- [91, 59, 34],
- PipTargetKind::NativeWindow,
- ),
- ]
- .into_iter()
- .enumerate()
- {
- backend.push_frame(PipFrame {
- target: PipTarget {
- workspace_id: workspace_id.to_owned(),
- workspace_label: workspace_label.to_owned(),
- target_id: format!("target-{index}"),
- identity_key: format!("target-{index}"),
- target_kind: kind,
- target_label: format!("Target {index}"),
- native_container: None,
- },
- png_bytes: png(width, height, base),
- action_label: "observe".to_owned(),
- timestamp_ms: index as u64,
- cursor_position: None,
- });
- }
- let seconds = std::env::var("CUA_AGENT_VIEW_SMOKE_SECONDS")
- .ok()
- .and_then(|value| value.parse().ok())
- .unwrap_or(3);
- std::thread::sleep(Duration::from_secs(seconds));
- backend.shutdown();
}
}
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/pip/wayland.rs b/libs/cua-driver/rust/crates/platform-linux/src/pip/wayland.rs
deleted file mode 100644
index 47271e9f4e..0000000000
--- a/libs/cua-driver/rust/crates/platform-linux/src/pip/wayland.rs
+++ /dev/null
@@ -1,307 +0,0 @@
-//! Native Wayland Agent View for compositors exposing layer-shell.
-
-use std::collections::HashMap;
-use std::sync::mpsc::{Receiver, SyncSender};
-use std::time::Duration;
-
-use pip_preview::{PipConfig, PipViewModel};
-use wayland_client::{
- protocol::{
- wl_buffer::{self, WlBuffer},
- wl_compositor::WlCompositor,
- wl_region::WlRegion,
- wl_registry,
- wl_shm::{Format, WlShm},
- wl_shm_pool::WlShmPool,
- wl_surface::WlSurface,
- },
- Connection, Dispatch, Proxy, QueueHandle,
-};
-use wayland_protocols_wlr::layer_shell::v1::client::{
- zwlr_layer_shell_v1::{Layer, ZwlrLayerShellV1},
- zwlr_layer_surface_v1::{self, Anchor, KeyboardInteractivity, ZwlrLayerSurfaceV1},
-};
-
-use super::{render_agent_view, UiMessage};
-
-struct State {
- compositor: Option<WlCompositor>,
- shm: Option<WlShm>,
- layer_shell: Option<ZwlrLayerShellV1>,
- surface: Option<WlSurface>,
- layer_surface: Option<ZwlrLayerSurfaceV1>,
- configured: bool,
- width: u32,
- height: u32,
- pending_buffers: HashMap<u32, (*mut libc::c_void, usize, i32)>,
-}
-
-unsafe impl Send for State {}
-
-impl State {
- fn new(width: u32, height: u32) -> Self {
- Self {
- compositor: None,
- shm: None,
- layer_shell: None,
- surface: None,
- layer_surface: None,
- configured: false,
- width,
- height,
- pending_buffers: HashMap::new(),
- }
- }
-}
-
-pub(super) fn run_window(
- cfg: PipConfig,
- rx: Receiver<UiMessage>,
- ready: SyncSender<anyhow::Result<()>>,
-) {
- if let Err(error) = run(cfg, rx, &ready) {
- let _ = ready.send(Err(error));
- }
-}
-
-fn run(
- cfg: PipConfig,
- rx: Receiver<UiMessage>,
- ready: &SyncSender<anyhow::Result<()>>,
-) -> anyhow::Result<()> {
- let connection = Connection::connect_to_env()?;
- let mut queue = connection.new_event_queue::<State>();
- let qh = queue.handle();
- let _registry = connection.display().get_registry(&qh, ());
- let mut state = State::new(cfg.geometry.width.max(360), cfg.geometry.height.max(260));
- queue.roundtrip(&mut state)?;
-
- let compositor = state
- .compositor
- .clone()
- .ok_or_else(|| anyhow::anyhow!("Wayland compositor does not expose wl_compositor"))?;
- let layer_shell = state.layer_shell.clone().ok_or_else(|| {
- anyhow::anyhow!(
- "native Wayland Agent View requires zwlr_layer_shell_v1; use XWayland on this compositor"
- )
- })?;
- let surface = compositor.create_surface(&qh, ());
- let layer_surface = layer_shell.get_layer_surface(
- &surface,
- None,
- Layer::Overlay,
- "cua-agent-view".to_owned(),
- &qh,
- (),
- );
- layer_surface.set_anchor(Anchor::Top | Anchor::Right);
- layer_surface.set_size(state.width, state.height);
- layer_surface.set_margin(24, 24, 24, 24);
- layer_surface.set_exclusive_zone(-1);
- layer_surface.set_keyboard_interactivity(KeyboardInteractivity::None);
- // Layer-shell cannot provide a normal draggable/resizable window contract.
- // Keep the presentation click-through so it can never block automation;
- // session selection continues to follow MRU on this native-only path.
- let input_region = compositor.create_region(&qh, ());
- surface.set_input_region(Some(&input_region));
- input_region.destroy();
- surface.commit();
- state.surface = Some(surface);
- state.layer_surface = Some(layer_surface);
-
- for _ in 0..8 {
- queue.roundtrip(&mut state)?;
- if state.configured {
- break;
- }
- }
- anyhow::ensure!(state.configured, "Wayland Agent View was never configured");
- ready
- .send(Ok(()))
- .map_err(|_| anyhow::anyhow!("Agent View startup receiver was dropped"))?;
-
- let mut model = PipViewModel::new(12);
- let mut running = true;
- let mut dirty = true;
- while running {
- match rx.recv_timeout(Duration::from_millis(16)) {
- Ok(UiMessage::Frame(frame)) => {
- model.upsert(frame);
- dirty = true;
- }
- Ok(UiMessage::RemoveTarget(workspace, target)) => {
- dirty |= model.remove_target(&workspace, &target);
- }
- Ok(UiMessage::RemoveWorkspace(workspace)) => {
- dirty |= !model.remove_workspace(&workspace).is_empty();
- }
- Ok(UiMessage::SetInputTransparent(_, reply)) => {
- // Layer-shell presentation does not cover the automated target
- // at its top-right inset, so no compositor input-region swap is
- // required for injected actions.
- let _ = reply.send(Ok(()));
- }
- Ok(UiMessage::Shutdown) => running = false,
- Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {}
- Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => break,
- }
- if dirty && running {
- redraw(&mut state, &model, &qh)?;
- dirty = false;
- }
- queue.roundtrip(&mut state)?;
- }
- Ok(())
-}
-
-fn redraw(state: &mut State, model: &PipViewModel, qh: &QueueHandle<State>) -> anyhow::Result<()> {
- let surface = state
- .surface
- .as_ref()
- .ok_or_else(|| anyhow::anyhow!("Wayland Agent View surface is unavailable"))?;
- let shm = state
- .shm
- .as_ref()
- .ok_or_else(|| anyhow::anyhow!("Wayland compositor does not expose wl_shm"))?;
- let width = state.width.max(1).min(u16::MAX as u32) as u16;
- let height = state.height.max(1).min(u16::MAX as u32) as u16;
- let frames = model.selected_frames();
- let workspaces = model.workspaces();
- let pixels = render_agent_view(
- width,
- height,
- &frames,
- &workspaces,
- model.selected_workspace_id(),
- model.active_view_id(),
- );
- let stride = i32::from(width) * 4;
- let size = pixels.len();
- let (fd, ptr) = crate::wayland::anon_shm(size)?;
- unsafe { std::ptr::copy_nonoverlapping(pixels.as_ptr(), ptr.cast::<u8>(), size) };
- use std::os::fd::AsFd as _;
- let pool_fd = unsafe { crate::wayland::borrowed_fd(fd) };
- let pool = shm.create_pool(pool_fd.as_fd(), size as i32, qh, ());
- let buffer = pool.create_buffer(
- 0,
- i32::from(width),
- i32::from(height),
- stride,
- Format::Xrgb8888,
- qh,
- (),
- );
- pool.destroy();
- state
- .pending_buffers
- .insert(buffer.id().protocol_id(), (ptr, size, fd));
- surface.attach(Some(&buffer), 0, 0);
- surface.damage_buffer(0, 0, i32::from(width), i32::from(height));
- surface.commit();
- Ok(())
-}
-
-impl Dispatch<wl_registry::WlRegistry, ()> for State {
- fn event(
- state: &mut Self,
- registry: &wl_registry::WlRegistry,
- event: wl_registry::Event,
- _: &(),
- _: &Connection,
- qh: &QueueHandle<Self>,
- ) {
- if let wl_registry::Event::Global {
- name,
- interface,
- version,
- } = event
- {
- match interface.as_str() {
- "wl_compositor" => {
- state.compositor =
- Some(registry.bind::<WlCompositor, _, _>(name, version.min(6), qh, ()))
- }
- "wl_shm" => {
- state.shm = Some(registry.bind::<WlShm, _, _>(name, version.min(1), qh, ()))
- }
- "zwlr_layer_shell_v1" => {
- state.layer_shell =
- Some(registry.bind::<ZwlrLayerShellV1, _, _>(name, version.min(4), qh, ()))
- }
- _ => {}
- }
- }
- }
-}
-
-impl Dispatch<ZwlrLayerSurfaceV1, ()> for State {
- fn event(
- state: &mut Self,
- layer: &ZwlrLayerSurfaceV1,
- event: zwlr_layer_surface_v1::Event,
- _: &(),
- _: &Connection,
- _: &QueueHandle<Self>,
- ) {
- match event {
- zwlr_layer_surface_v1::Event::Configure {
- serial,
- width,
- height,
- } => {
- layer.ack_configure(serial);
- if width > 0 {
- state.width = width;
- }
- if height > 0 {
- state.height = height;
- }
- state.configured = true;
- }
- zwlr_layer_surface_v1::Event::Closed => state.surface = None,
- _ => {}
- }
- }
-}
-
-impl Dispatch<WlBuffer, ()> for State {
- fn event(
- state: &mut Self,
- buffer: &WlBuffer,
- event: wl_buffer::Event,
- _: &(),
- _: &Connection,
- _: &QueueHandle<Self>,
- ) {
- if matches!(event, wl_buffer::Event::Release) {
- if let Some((ptr, size, fd)) = state.pending_buffers.remove(&buffer.id().protocol_id())
- {
- crate::wayland::cleanup_mmap(ptr, size, fd);
- }
- buffer.destroy();
- }
- }
-}
-
-macro_rules! empty_dispatch {
- ($ty:ty) => {
- impl Dispatch<$ty, ()> for State {
- fn event(
- _: &mut Self,
- _: &$ty,
- _: <$ty as Proxy>::Event,
- _: &(),
- _: &Connection,
- _: &QueueHandle<Self>,
- ) {
- }
- }
- };
-}
-
-empty_dispatch!(WlCompositor);
-empty_dispatch!(WlShm);
-empty_dispatch!(WlShmPool);
-empty_dispatch!(WlSurface);
-empty_dispatch!(WlRegion);
-empty_dispatch!(ZwlrLayerShellV1);
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index 9b509284a6..a148640ac3 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -7114,16 +7114,15 @@ impl Tool for GetConfigTool {
}
async fn invoke(&self, _args: Value) -> ToolResult {
let cfg = self.state.config.read().unwrap();
- let (agent_view_enabled, agent_view_geometry) =
- pip_preview::read_agent_view_keys_from_file();
+ let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
ToolResult::text("cua-driver-rs configuration").with_structured(json!({
"version": env!("CARGO_PKG_VERSION"),
"source_sha": option_env!("CUA_DRIVER_SOURCE_SHA"),
"platform": "linux",
"capture_mode": cfg.capture_mode,
"max_image_dimension": cfg.max_image_dimension,
- "agent_view": agent_view_enabled,
- "agent_view_geometry": agent_view_geometry
+ "experimental_pip": pip_enabled,
+ "experimental_pip_geometry": pip_geometry
}))
}
}
@@ -7146,15 +7145,16 @@ impl Tool for SetConfigTool {
- **{key, value}** (preferred): `{\"key\": \"max_image_dimension\", \"value\": 800}` \
— single leaf write.\n\
- **Legacy per-field**: `{\"capture_mode\": \"som\", \"max_image_dimension\": 0}`.\n\n\
- The agent_view keys persist to ~/.cua-driver/config.json and apply on next \
- daemon restart (Agent View is initialised once at startup).".into(),
+ The experimental_pip keys persist to ~/.cua-driver/config.json and apply on next \
+ daemon restart (the PiP backend is initialised once at startup; \
+ Linux ships only the trait stub today — see issue #1729).".into(),
input_schema: json!({"type":"object","properties":{
"key":{"type":"string","description":"Name of a single config field to write ({key, value} shape). Pair with `value`."},
"value":{"description":"New value for `key`. JSON type depends on the key."},
"capture_mode":{"type":"string","enum":["ax","vision"],"description":"Legacy per-field shape. Default capture mode for get_window_state. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},
"max_image_dimension":{"type":"integer","description":"Legacy per-field shape. Max dimension for screenshot resizing (0 = no limit)."},
- "agent_view":{"type":"boolean","description":"Enable the multi-target Agent View (applies next restart)."},
- "agent_view_geometry":{"type":"string","description":"Agent View size + optional position in `WxH` or `WxH+X+Y` form."}
+ "experimental_pip":{"type":"boolean","description":"Enable the experimental PiP preview window (applies next restart; Linux backend stubbed)."},
+ "experimental_pip_geometry":{"type":"string","description":"PiP window size + optional position in `WxH` or `WxH+X+Y` form."}
},"additionalProperties":false}),
read_only: false, destructive: false, idempotent: true, open_world: false,
})
@@ -7203,31 +7203,31 @@ impl Tool for SetConfigTool {
}
None => return ToolResult::error(format!("`max_image_dimension` must be an integer, got {val}.")),
},
- "agent_view" => match val.as_bool() {
+ "experimental_pip" => match val.as_bool() {
Some(b) => {
- if let Err(e) = pip_preview::write_config_key("agent_view", Value::Bool(b)) {
- return ToolResult::error(format!("failed to persist agent_view: {e}"));
+ if let Err(e) = pip_preview::write_config_key("experimental_pip", Value::Bool(b)) {
+ return ToolResult::error(format!("failed to persist experimental_pip: {e}"));
}
- parts.push(format!("agent_view={b} (next restart)"));
+ parts.push(format!("experimental_pip={b} (next restart)"));
}
- None => return ToolResult::error(format!("`agent_view` must be a boolean, got {val}.")),
+ None => return ToolResult::error(format!("`experimental_pip` must be a boolean, got {val}.")),
},
- "agent_view_geometry" => match val.as_str() {
+ "experimental_pip_geometry" => match val.as_str() {
Some(s) => {
if pip_preview::PipGeometry::parse(s).is_none() {
return ToolResult::error(format!(
- "agent_view_geometry `{s}` is not a valid WxH or WxH+X+Y string"
+ "experimental_pip_geometry `{s}` is not a valid WxH or WxH+X+Y string"
));
}
- if let Err(e) = pip_preview::write_config_key("agent_view_geometry", Value::String(s.to_owned())) {
- return ToolResult::error(format!("failed to persist agent_view_geometry: {e}"));
+ if let Err(e) = pip_preview::write_config_key("experimental_pip_geometry", Value::String(s.to_owned())) {
+ return ToolResult::error(format!("failed to persist experimental_pip_geometry: {e}"));
}
- parts.push(format!("agent_view_geometry={s} (next restart)"));
+ parts.push(format!("experimental_pip_geometry={s} (next restart)"));
}
- None => return ToolResult::error(format!("`agent_view_geometry` must be a string, got {val}.")),
+ None => return ToolResult::error(format!("`experimental_pip_geometry` must be a string, got {val}.")),
},
other => return ToolResult::error(format!(
- "Unknown config key `{other}`. Known: capture_mode, max_image_dimension, agent_view, agent_view_geometry."
+ "Unknown config key `{other}`. Known: capture_mode, max_image_dimension, experimental_pip, experimental_pip_geometry."
)),
}
}
@@ -7248,37 +7248,40 @@ impl Tool for SetConfigTool {
}
parts.push(format!("max_image_dimension={dim}"));
}
- if let Some(enabled) = args.get("agent_view").and_then(|v| v.as_bool()) {
- if let Err(e) = pip_preview::write_config_key("agent_view", Value::Bool(enabled)) {
- return ToolResult::error(format!("failed to persist agent_view: {e}"));
+ if let Some(enabled) = args.get("experimental_pip").and_then(|v| v.as_bool()) {
+ if let Err(e) = pip_preview::write_config_key("experimental_pip", Value::Bool(enabled))
+ {
+ return ToolResult::error(format!("failed to persist experimental_pip: {e}"));
}
- parts.push(format!("agent_view={enabled} (next restart)"));
+ parts.push(format!("experimental_pip={enabled} (next restart)"));
}
- if let Some(geom) = args.opt_str("agent_view_geometry") {
+ if let Some(geom) = args.opt_str("experimental_pip_geometry") {
if pip_preview::PipGeometry::parse(&geom).is_none() {
return ToolResult::error(format!(
- "agent_view_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
+ "experimental_pip_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
));
}
- if let Err(e) =
- pip_preview::write_config_key("agent_view_geometry", Value::String(geom.clone()))
- {
- return ToolResult::error(format!("failed to persist agent_view_geometry: {e}"));
+ if let Err(e) = pip_preview::write_config_key(
+ "experimental_pip_geometry",
+ Value::String(geom.clone()),
+ ) {
+ return ToolResult::error(format!(
+ "failed to persist experimental_pip_geometry: {e}"
+ ));
}
- parts.push(format!("agent_view_geometry={geom} (next restart)"));
+ parts.push(format!("experimental_pip_geometry={geom} (next restart)"));
}
let msg = if parts.is_empty() {
"Config unchanged (no known parameters).".to_owned()
} else {
format!("Config updated: {}", parts.join(", "))
};
- let (agent_view_enabled, agent_view_geometry) =
- pip_preview::read_agent_view_keys_from_file();
+ let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
ToolResult::text(msg).with_structured(json!({
"capture_mode": cfg.capture_mode,
"max_image_dimension": cfg.max_image_dimension,
- "agent_view": agent_view_enabled,
- "agent_view_geometry": agent_view_geometry
+ "experimental_pip": pip_enabled,
+ "experimental_pip_geometry": pip_geometry
}))
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/pip/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/pip/mod.rs
index 4bb349e555..bf873a403f 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/pip/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/pip/mod.rs
@@ -1,61 +1,90 @@
-//! macOS multi-target Agent View.
+//! macOS picture-in-picture preview window.
//!
-//! A floating, resizable miniature desktop that uses the host wallpaper and
-//! presents exact native windows and browser tabs as aspect-aware macOS-like
-//! windows. Existing lifecycle sessions group presentation only; Agent View
-//! never claims, moves, resizes, or closes the underlying targets.
-
-use std::ffi::{c_void, CStr};
-use std::sync::atomic::{AtomicBool, Ordering};
-use std::sync::{Arc, Mutex, OnceLock};
-
-use pip_preview::{
- layout_desktop, layout_session_tabs, png_dimensions, LayoutRect, PipBackend, PipBackendFactory,
- PipConfig, PipFrame, PipTargetKind, PipViewModel, PipWorkspaceSummary, SessionTabsLayout,
- TargetSize,
-};
+//! Floating NSWindow with an NSImageView showing the most recent
+//! post-action screenshot and an NSTextField with a one-line label
+//! describing the action that produced it.
+//!
+//! ## Threading model
+//!
+//! Mirrors `cursor/overlay.rs`:
+//!
+//! - The MCP/tokio server runs on a background thread.
+//! - AppKit MUST run on the main thread, which `cua-driver/src/main.rs`
+//! parks in `NSApplication.run()` for the cursor overlay.
+//! - `push_frame()` is called from arbitrary tokio tasks. It packages
+//! the frame into a heap-allocated `Box` and posts the actual UI
+//! update onto the main queue via `dispatch_async_f`. The block
+//! then constructs an `NSImage` from the PNG bytes and calls
+//! `[imageView setImage:]` + `[label setStringValue:]`.
+//!
+//! ## Window properties
+//!
+//! - `NSWindowCollectionBehaviorCanJoinAllSpaces | FullScreenAuxiliary |
+//! Stationary | Transient | IgnoresCycle`
+//! - `level = .floating` (kCGFloatingWindowLevel, between normal apps
+//! and dock; high enough to stay visible, low enough not to obscure
+//! menus or accessibility overlays).
+//! - `setIgnoresMouseEvents(false)` — user can click the red close
+//! button. Backend cleanup happens on `shutdown()`; closing the
+//! window manually decouples it from the session as the spec
+//! requires.
+//! - No activation: `setHidesOnDeactivate(false)` and
+//! `setBecomesKeyOnlyIfNeeded(true)` so the window never steals
+//! keyboard focus from the user's frontmost app.
+//!
+//! ## Init lifecycle
+//!
+//! Because the cursor overlay already owns the main thread when
+//! enabled, `MacosPipBackend::start` cannot block on it. Instead it
+//! posts the window-creation block onto the main queue and returns
+//! immediately. The first frame may arrive before the window exists;
+//! that's fine — the push path reads the window pointer from a
+//! `Mutex<Option<usize>>` and silently no-ops until init finishes.
+
+use std::ffi::c_void;
+use std::sync::Mutex;
+
+use pip_preview::{PipBackend, PipBackendFactory, PipConfig, PipFrame};
+
+// ── CGColor objc2 encoding shim ────────────────────────────────────────────
+//
+// `[NSColor CGColor]` returns a `CGColorRef` whose Objective-C type encoding
+// is `^{CGColor=}`. objc2's strict msg_send! enforcement rejects bare
+// `*mut c_void` (`^v`) for both sides of that call. Declare a phantom
+// struct with the matching encoding so we can typed-cast through it
+// without pulling in a wider CGColor binding crate.
#[repr(C)]
struct CGColor {
_opaque: [u8; 0],
}
+// RefEncode supplies an automatic Encode impl for `*mut CGColor` /
+// `*const CGColor` via objc2's blanket — that's the route msg_send! needs
+// for both setting layer.backgroundColor and reading [NSColor CGColor].
+// `ENCODING_REF` is the encoding for one level of indirection, so the
+// pointer wrap goes here (objc encoding `^{CGColor=}`).
unsafe impl objc2::RefEncode for CGColor {
const ENCODING_REF: objc2::Encoding =
objc2::Encoding::Pointer(&objc2::Encoding::Struct("CGColor", &[]));
}
+// ── Native AppKit pointer cell ─────────────────────────────────────────────
+//
+// Window, image view, and label pointers are stashed as `usize` so
+// `Send` works (raw `*mut AnyObject` is `!Send`). The actual deref +
+// `msg_send!` happens only on the main queue inside the dispatched
+// block, so there is no thread-safety hazard from the Send promise.
+
struct NativeHandles {
window: usize,
- canvas_view: usize,
- shell_wallpaper_container: usize,
- shell_wallpaper_view: usize,
- wallpaper_container: usize,
- wallpaper_view: usize,
- delegate: usize,
+ image_view: usize,
+ label: usize,
}
-/// Outer corner radius of the Agent View container chrome.
-const CONTAINER_RADIUS: f64 = 15.0;
-const PIP_CORNER_RADIUS: f64 = 12.0;
-/// The miniature desktop sits inside an asymmetric hardware-like glass shell.
-const SHELL_SIDE_INSET: f64 = 9.0;
-const SHELL_TOP_INSET: f64 = 20.0;
-const SHELL_BOTTOM_INSET: f64 = 10.0;
-const SESSION_SELECTOR_HEIGHT: f64 = 34.0;
-const RESIZE_HIT_INSET: f64 = 7.0;
-
-// Keep the PiP shell visually clean for now; invisible hit zones still provide
-// dragging and resizing without exposing a native or custom frame.
-const SHOW_SHELL_BORDER: bool = false;
-
-const RESIZE_LEFT: isize = 1;
-const RESIZE_RIGHT: isize = 2;
-const RESIZE_BOTTOM: isize = 4;
-const RESIZE_TOP: isize = 8;
-
static HANDLES: Mutex<Option<NativeHandles>> = Mutex::new(None);
-static VIEW_MODEL: Mutex<Option<PipViewModel>> = Mutex::new(None);
+
+// ── libdispatch glue — same shape as cursor::overlay ──────────────────────
#[link(name = "dispatch", kind = "dylib")]
extern "C" {
@@ -65,11 +94,6 @@ extern "C" {
context: *mut c_void,
work: unsafe extern "C" fn(*mut c_void),
);
- fn dispatch_sync_f(
- queue: *const c_void,
- context: *mut c_void,
- work: unsafe extern "C" fn(*mut c_void),
- );
}
fn dispatch_to_main<T: Send + 'static>(payload: T, cb: unsafe extern "C" fn(*mut c_void)) {
@@ -80,1082 +104,74 @@ fn dispatch_to_main<T: Send + 'static>(payload: T, cb: unsafe extern "C" fn(*mut
}
}
-fn dispatch_to_main_sync<T>(payload: T, cb: unsafe extern "C" fn(*mut c_void)) {
- let context = Box::into_raw(Box::new(payload)) as *mut c_void;
- unsafe {
- if libc::pthread_main_np() != 0 {
- cb(context);
- } else {
- let main_queue = &raw const _dispatch_main_q as *const c_void;
- dispatch_sync_f(main_queue, context, cb);
- }
- }
-}
+// ── Backend impl ──────────────────────────────────────────────────────────
pub struct MacosPipBackend;
-struct InputPassthroughRequest {
- passthrough: bool,
- applied: Arc<AtomicBool>,
-}
-
impl PipBackend for MacosPipBackend {
fn push_frame(&self, frame: PipFrame) {
- // init_cb was queued first on the serial main queue, so retaining the
- // frame here avoids dropping exact activity during daemon startup.
+ // No window yet? Drop the frame silently — start() dispatches
+ // the create block onto the main queue and the very first
+ // tool call can race that block.
+ if HANDLES.lock().unwrap().is_none() {
+ return;
+ }
dispatch_to_main(frame, push_frame_cb);
}
- fn remove_workspace(&self, workspace_id: &str) {
- dispatch_to_main(workspace_id.to_owned(), remove_workspace_cb);
- }
-
- fn remove_target(&self, workspace_id: &str, identity_key: &str) {
- dispatch_to_main(
- (workspace_id.to_owned(), identity_key.to_owned()),
- remove_target_cb,
- );
- }
-
- fn set_input_passthrough(&self, passthrough: bool) -> anyhow::Result<()> {
- let applied = Arc::new(AtomicBool::new(false));
- dispatch_to_main_sync(
- InputPassthroughRequest {
- passthrough,
- applied: Arc::clone(&applied),
- },
- set_input_passthrough_cb,
- );
- // The user may close the optional presentation window while the
- // daemon continues serving tools; that must never break input.
- let _ = applied.load(Ordering::Acquire);
- Ok(())
- }
-
fn shutdown(self: Box<Self>) {
dispatch_to_main((), shutdown_cb);
}
}
-unsafe extern "C" fn set_input_passthrough_cb(ctx: *mut c_void) {
- use objc2::msg_send;
- use objc2::runtime::AnyObject;
-
- let request: InputPassthroughRequest = *Box::from_raw(ctx as *mut InputPassthroughRequest);
- let handles = HANDLES.lock().unwrap();
- if let Some(handles) = handles.as_ref() {
- let window = handles.window as *mut AnyObject;
- let _: () = msg_send![window, setIgnoresMouseEvents: request.passthrough];
- request.applied.store(true, Ordering::Release);
- }
-}
-
unsafe extern "C" fn push_frame_cb(ctx: *mut c_void) {
- let frame: PipFrame = *Box::from_raw(ctx as *mut PipFrame);
- let snapshot = {
- let mut model = VIEW_MODEL.lock().unwrap();
- let model = model.get_or_insert_with(|| PipViewModel::new(12));
- model.upsert(frame);
- clone_snapshot(model)
- };
- render_snapshot(&snapshot);
-}
-
-unsafe extern "C" fn remove_workspace_cb(ctx: *mut c_void) {
- let workspace_id: String = *Box::from_raw(ctx as *mut String);
- let snapshot = {
- let mut model = VIEW_MODEL.lock().unwrap();
- let model = model.get_or_insert_with(|| PipViewModel::new(12));
- model.remove_workspace(&workspace_id);
- clone_snapshot(model)
- };
- render_snapshot(&snapshot);
-}
-
-unsafe extern "C" fn remove_target_cb(ctx: *mut c_void) {
- let (workspace_id, identity_key): (String, String) =
- *Box::from_raw(ctx as *mut (String, String));
- let snapshot = {
- let mut model = VIEW_MODEL.lock().unwrap();
- let model = model.get_or_insert_with(|| PipViewModel::new(12));
- model.remove_target(&workspace_id, &identity_key);
- clone_snapshot(model)
- };
- render_snapshot(&snapshot);
-}
-
-struct ViewSnapshot {
- frames: Vec<PipFrame>,
- workspaces: Vec<PipWorkspaceSummary>,
- selected_workspace_id: Option<String>,
- active_view_id: Option<String>,
-}
-
-fn clone_snapshot(model: &PipViewModel) -> ViewSnapshot {
- ViewSnapshot {
- frames: model.selected_frames().into_iter().cloned().collect(),
- workspaces: model.workspaces(),
- selected_workspace_id: model.selected_workspace_id().map(str::to_owned),
- active_view_id: model.active_view_id().map(str::to_owned),
- }
-}
-
-fn current_snapshot() -> ViewSnapshot {
- VIEW_MODEL
- .lock()
- .unwrap()
- .as_ref()
- .map(clone_snapshot)
- .unwrap_or_else(|| ViewSnapshot {
- frames: Vec::new(),
- workspaces: Vec::new(),
- selected_workspace_id: None,
- active_view_id: None,
- })
-}
-
-unsafe fn ns_string(value: &str) -> *mut objc2::runtime::AnyObject {
- use objc2::{class, msg_send};
-
- let sanitized = value.replace('\0', " ");
- let Ok(cstr) = std::ffi::CString::new(sanitized) else {
- return std::ptr::null_mut();
- };
- msg_send![
- class!(NSString),
- stringWithUTF8String: cstr.as_ptr() as *const u8
- ]
-}
-
-unsafe fn rust_string(value: *mut objc2::runtime::AnyObject) -> Option<String> {
- use objc2::msg_send;
-
- if value.is_null() {
- return None;
- }
- let utf8: *const std::os::raw::c_char = msg_send![value, UTF8String];
- (!utf8.is_null()).then(|| CStr::from_ptr(utf8).to_string_lossy().into_owned())
-}
-
-#[derive(Clone, Copy)]
-enum LabelTone {
- Muted,
- Dark,
-}
-
-unsafe fn color(red: f64, green: f64, blue: f64, alpha: f64) -> *mut objc2::runtime::AnyObject {
- use objc2::{class, msg_send};
-
- msg_send![
- class!(NSColor),
- colorWithCalibratedRed: red
- green: green
- blue: blue
- alpha: alpha
- ]
-}
-
-unsafe fn set_layer_background(
- layer: *mut objc2::runtime::AnyObject,
- background: *mut objc2::runtime::AnyObject,
-) {
- use objc2::msg_send;
-
- let cg_color: *mut CGColor = msg_send![background, CGColor];
- let _: () = msg_send![layer, setBackgroundColor: cg_color];
-}
-
-/// Opt a container layer into the macOS squircle corner curve so the chrome
-/// reads as continuous rather than as a circular-arc rectangle.
-unsafe fn set_continuous_corners(layer: *mut objc2::runtime::AnyObject) {
- use objc2::msg_send;
-
- let responds: bool = msg_send![layer, respondsToSelector: objc2::sel!(setCornerCurve:)];
- if !responds {
- return;
- }
- let curve = ns_string("continuous");
- if curve.is_null() {
- return;
- }
- let _: () = msg_send![layer, setCornerCurve: curve];
-}
-
-unsafe fn set_layer_border(
- layer: *mut objc2::runtime::AnyObject,
- width: f64,
- border: *mut objc2::runtime::AnyObject,
-) {
- use objc2::msg_send;
-
- let cg_color: *mut CGColor = msg_send![border, CGColor];
- let _: () = msg_send![layer, setBorderWidth: width];
- let _: () = msg_send![layer, setBorderColor: cg_color];
-}
-
-unsafe fn add_text_label(
- parent: *mut objc2::runtime::AnyObject,
- frame: objc2_foundation::NSRect,
- text: &str,
- font_size: f64,
- bold: bool,
- tone: LabelTone,
- alignment: isize,
-) {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
-
- let label: *mut AnyObject = {
- let alloc: *mut AnyObject = msg_send![class!(NSTextField), alloc];
- msg_send![alloc, initWithFrame: frame]
- };
- let _: () = msg_send![label, setBezeled: false];
- let _: () = msg_send![label, setDrawsBackground: false];
- let _: () = msg_send![label, setEditable: false];
- let _: () = msg_send![label, setSelectable: false];
- let _: () = msg_send![label, setAlignment: alignment];
- let text_color = match tone {
- LabelTone::Muted => color(1.0, 1.0, 1.0, 0.66),
- LabelTone::Dark => color(0.12, 0.13, 0.15, 0.94),
- };
- let _: () = msg_send![label, setTextColor: text_color];
- let font: *mut AnyObject = if bold {
- msg_send![class!(NSFont), boldSystemFontOfSize: font_size]
- } else {
- msg_send![class!(NSFont), systemFontOfSize: font_size]
- };
- let _: () = msg_send![label, setFont: font];
- let value = ns_string(text);
- if !value.is_null() {
- let _: () = msg_send![label, setStringValue: value];
- }
- let _: () = msg_send![parent, addSubview: label];
-}
-
-unsafe fn rounded_view(
- frame: objc2_foundation::NSRect,
- radius: f64,
- background: *mut objc2::runtime::AnyObject,
- clips: bool,
-) -> *mut objc2::runtime::AnyObject {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
-
- let view: *mut AnyObject = {
- let alloc: *mut AnyObject = msg_send![class!(NSView), alloc];
- msg_send![alloc, initWithFrame: frame]
- };
- let _: () = msg_send![view, setWantsLayer: true];
- let layer: *mut AnyObject = msg_send![view, layer];
- let _: () = msg_send![layer, setCornerRadius: radius];
- let _: () = msg_send![layer, setMasksToBounds: clips];
- set_layer_background(layer, background);
- view
-}
-
-/// A layer-hosting view whose backing layer is a vertical `CAGradientLayer`.
-///
-/// The chrome body needs a top-to-bottom falloff to read as a lit glass
-/// surface rather than as a flat outline. AppKit keeps a view-assigned layer
-/// sized to its view, so this survives live resize without a manual pass.
-/// Layer-hosting views must not take subviews; overlay strips go on the
-/// enclosing chrome view instead.
-unsafe fn gradient_view(
- frame: objc2_foundation::NSRect,
- radius: f64,
- top: *mut objc2::runtime::AnyObject,
- bottom: *mut objc2::runtime::AnyObject,
-) -> *mut objc2::runtime::AnyObject {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
-
- let view: *mut AnyObject = {
- let alloc: *mut AnyObject = msg_send![class!(NSView), alloc];
- msg_send![alloc, initWithFrame: frame]
- };
- let layer: *mut AnyObject = {
- let alloc: *mut AnyObject = msg_send![class!(CAGradientLayer), alloc];
- msg_send![alloc, init]
- };
- let top_cg: *mut CGColor = msg_send![top, CGColor];
- let bottom_cg: *mut CGColor = msg_send![bottom, CGColor];
- let stops: [*mut CGColor; 2] = [top_cg, bottom_cg];
- let colors: *mut AnyObject = msg_send![
- class!(NSArray),
- arrayWithObjects: stops.as_ptr() as *const *mut AnyObject
- count: 2usize
- ];
- let _: () = msg_send![layer, setColors: colors];
- let _: () = msg_send![layer, setStartPoint: objc2_foundation::NSPoint::new(0.5, 1.0)];
- let _: () = msg_send![layer, setEndPoint: objc2_foundation::NSPoint::new(0.5, 0.0)];
- let _: () = msg_send![layer, setCornerRadius: radius];
- let _: () = msg_send![layer, setMasksToBounds: true];
- set_continuous_corners(layer);
- let _: () = msg_send![view, setLayer: layer];
- let _: () = msg_send![view, setWantsLayer: true];
- view
-}
-
-unsafe fn visual_effect_view(
- frame: objc2_foundation::NSRect,
- radius: f64,
- material: isize,
- blending_mode: isize,
- clips: bool,
-) -> *mut objc2::runtime::AnyObject {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
-
- let view: *mut AnyObject = {
- let alloc: *mut AnyObject = msg_send![class!(NSVisualEffectView), alloc];
- msg_send![alloc, initWithFrame: frame]
- };
- let _: () = msg_send![view, setMaterial: material];
- let _: () = msg_send![view, setBlendingMode: blending_mode];
- let _: () = msg_send![view, setState: 1isize];
- let _: () = msg_send![view, setWantsLayer: true];
- let layer: *mut AnyObject = msg_send![view, layer];
- let _: () = msg_send![layer, setCornerRadius: radius];
- let _: () = msg_send![layer, setMasksToBounds: clips];
- view
-}
-
-unsafe fn add_circle(
- parent: *mut objc2::runtime::AnyObject,
- x: f64,
- y: f64,
- diameter: f64,
- fill: *mut objc2::runtime::AnyObject,
-) {
- use objc2::msg_send;
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- let circle = rounded_view(
- NSRect::new(NSPoint::new(x, y), NSSize::new(diameter, diameter)),
- diameter / 2.0,
- fill,
- true,
- );
- let _: () = msg_send![parent, addSubview: circle];
-}
-
-fn appkit_rect(rect: pip_preview::LayoutRect, bounds_height: f64) -> objc2_foundation::NSRect {
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- NSRect::new(
- NSPoint::new(rect.x, bounds_height - rect.y - rect.height),
- NSSize::new(rect.width, rect.height),
- )
-}
-
-fn parse_native_pid(target_id: &str) -> Option<i32> {
- let mut parts = target_id.split(':');
- (parts.next()? == "window")
- .then(|| parts.next()?.parse::<i32>().ok())
- .flatten()
-}
-
-fn truncate_label(value: &str, max_chars: usize) -> String {
- let mut chars = value.chars();
- let prefix = chars.by_ref().take(max_chars).collect::<String>();
- if chars.next().is_some() {
- format!("{prefix}...")
- } else {
- prefix
- }
-}
-
-unsafe fn target_identity(frame: &PipFrame) -> (String, *mut objc2::runtime::AnyObject, char) {
use objc2::runtime::AnyObject;
use objc2::{class, msg_send};
- if frame.target.target_kind == PipTargetKind::NativeWindow {
- if let Some(pid) = parse_native_pid(&frame.target.target_id) {
- let app: *mut AnyObject = msg_send![
- class!(NSRunningApplication),
- runningApplicationWithProcessIdentifier: pid
- ];
- if !app.is_null() {
- let name: *mut AnyObject = msg_send![app, localizedName];
- let icon: *mut AnyObject = msg_send![app, icon];
- if let Some(name) = rust_string(name) {
- let fallback = name.chars().next().unwrap_or('A').to_ascii_uppercase();
- return (truncate_label(&name, 28), icon, fallback);
- }
- }
- }
- }
+ let frame: PipFrame = *Box::from_raw(ctx as *mut PipFrame);
- let label = match frame.target.target_kind {
- PipTargetKind::BrowserTab => {
- if frame.target.target_label.trim().is_empty() {
- "Browser".to_owned()
- } else {
- truncate_label(&frame.target.target_label, 28)
- }
+ let (image_view_ptr, label_ptr) = {
+ let guard = HANDLES.lock().unwrap();
+ match guard.as_ref() {
+ Some(h) => (h.image_view, h.label),
+ None => return,
}
- PipTargetKind::NativeWindow => truncate_label(&frame.target.workspace_label, 28),
- };
- let fallback = match frame.target.target_kind {
- PipTargetKind::BrowserTab => 'B',
- PipTargetKind::NativeWindow => label.chars().next().unwrap_or('A').to_ascii_uppercase(),
};
- (label, std::ptr::null_mut(), fallback)
-}
-unsafe fn image_from_png(bytes: &[u8]) -> *mut objc2::runtime::AnyObject {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
-
- let data: *mut AnyObject = msg_send![
+ // Construct NSData from the PNG bytes, then NSImage from NSData.
+ // `dataWithBytes:length:` copies into a fresh NSData so the input
+ // `Vec<u8>` can be freed at the end of this block.
+ let png_ptr = frame.png_bytes.as_ptr() as *const c_void;
+ let png_len = frame.png_bytes.len();
+ let ns_data: *mut AnyObject = msg_send![
class!(NSData),
- dataWithBytes: bytes.as_ptr() as *const c_void
- length: bytes.len()
+ dataWithBytes: png_ptr
+ length: png_len
];
- if data.is_null() {
- return std::ptr::null_mut();
- }
- let alloc: *mut AnyObject = msg_send![class!(NSImage), alloc];
- msg_send![alloc, initWithData: data]
-}
-
-unsafe fn render_target_window(
- canvas: *mut objc2::runtime::AnyObject,
- frame: &PipFrame,
- layout: pip_preview::TargetLayout,
- bounds_height: f64,
- active: bool,
-) {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- let window_frame = appkit_rect(layout.window, bounds_height);
- let radius = (window_frame.size.width.min(window_frame.size.height) * 0.035).clamp(5.0, 9.0);
- let shadow = rounded_view(window_frame, radius, color(0.0, 0.0, 0.0, 0.0), false);
- let shadow_layer: *mut AnyObject = msg_send![shadow, layer];
- let _: () = msg_send![shadow_layer, setShadowOpacity: 0.30_f32];
- let _: () = msg_send![shadow_layer, setShadowRadius: 16.0_f64];
- let _: () = msg_send![shadow_layer, setShadowOffset: NSSize::new(0.0, -6.0)];
- let black = color(0.0, 0.0, 0.0, 0.90);
- let black_cg: *mut CGColor = msg_send![black, CGColor];
- let _: () = msg_send![shadow_layer, setShadowColor: black_cg];
-
- let window = rounded_view(
- NSRect::new(
- NSPoint::new(0.0, 0.0),
- NSSize::new(window_frame.size.width, window_frame.size.height),
- ),
- radius,
- color(0.0, 0.0, 0.0, 0.0),
- true,
- );
- let window_layer: *mut AnyObject = msg_send![window, layer];
- if active {
- set_layer_border(window_layer, 2.2, color(0.25, 0.65, 1.0, 0.96));
- } else if SHOW_SHELL_BORDER {
- set_layer_border(window_layer, 0.55, color(0.0, 0.0, 0.0, 0.22));
- }
- let image_view: *mut AnyObject = {
- let alloc: *mut AnyObject = msg_send![class!(NSImageView), alloc];
- msg_send![
- alloc,
- initWithFrame: NSRect::new(
- NSPoint::new(0.0, 0.0),
- NSSize::new(window_frame.size.width, window_frame.size.height),
- )
- ]
- };
- let _: () = msg_send![image_view, setImageScaling: 3u64];
- let image = image_from_png(&frame.png_bytes);
- if !image.is_null() {
- let _: () = msg_send![image_view, setImage: image];
- }
- let _: () = msg_send![window, addSubview: image_view];
- if let Some((normalized_x, normalized_y)) = frame.cursor_position {
- let pointer_size = NSSize::new(18.0, 22.0);
- let x = (normalized_x.clamp(0.0, 1.0) * window_frame.size.width)
- .clamp(0.0, (window_frame.size.width - pointer_size.width).max(0.0));
- let y_from_top = normalized_y.clamp(0.0, 1.0) * window_frame.size.height;
- let y = (window_frame.size.height - y_from_top - pointer_size.height).clamp(
- 0.0,
- (window_frame.size.height - pointer_size.height).max(0.0),
- );
- let pointer: *mut AnyObject = {
- let alloc: *mut AnyObject = msg_send![class!(NSImageView), alloc];
- msg_send![
- alloc,
- initWithFrame: NSRect::new(NSPoint::new(x, y), pointer_size)
- ]
- };
- let arrow_cursor: *mut AnyObject = msg_send![class!(NSCursor), arrowCursor];
- let arrow_image: *mut AnyObject = msg_send![arrow_cursor, image];
- let _: () = msg_send![pointer, setImage: arrow_image];
- let _: () = msg_send![pointer, setImageScaling: 0u64];
- let _: () = msg_send![pointer, setWantsLayer: true];
- let pointer_layer: *mut AnyObject = msg_send![pointer, layer];
- let _: () = msg_send![pointer_layer, setShadowOpacity: 0.72_f32];
- let _: () = msg_send![pointer_layer, setShadowRadius: 2.5_f64];
- let _: () = msg_send![pointer_layer, setShadowOffset: NSSize::new(0.0, -1.0)];
- let shadow_color = color(0.0, 0.0, 0.0, 0.92);
- let shadow_cg: *mut CGColor = msg_send![shadow_color, CGColor];
- let _: () = msg_send![pointer_layer, setShadowColor: shadow_cg];
- let _: () = msg_send![window, addSubview: pointer];
- }
- let _: () = msg_send![shadow, addSubview: window];
- let _: () = msg_send![canvas, addSubview: shadow];
-}
-
-unsafe fn render_dock(
- canvas: *mut objc2::runtime::AnyObject,
- frames: &[PipFrame],
- layout: &pip_preview::DesktopLayout,
- bounds_height: f64,
-) {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- let dock_frame = appkit_rect(layout.dock, bounds_height);
- let dock = visual_effect_view(dock_frame, dock_frame.size.height * 0.30, 6, 1, true);
- let appearance_name = ns_string("NSAppearanceNameVibrantDark");
- let dark_appearance: *mut AnyObject =
- msg_send![class!(NSAppearance), appearanceNamed: appearance_name];
- if !dark_appearance.is_null() {
- let _: () = msg_send![dock, setAppearance: dark_appearance];
- }
- let dock_layer: *mut AnyObject = msg_send![dock, layer];
- set_layer_background(dock_layer, color(0.30, 0.32, 0.34, 0.22));
- set_layer_border(dock_layer, 0.65, color(1.0, 1.0, 1.0, 0.20));
- let _: () = msg_send![dock_layer, setShadowOpacity: 0.34_f32];
- let _: () = msg_send![dock_layer, setShadowRadius: 14.0_f64];
- let _: () = msg_send![dock_layer, setShadowOffset: NSSize::new(0.0, -6.0)];
- let dock_shadow = color(0.0, 0.0, 0.0, 0.75);
- let dock_shadow_cg: *mut CGColor = msg_send![dock_shadow, CGColor];
- let _: () = msg_send![dock_layer, setShadowColor: dock_shadow_cg];
-
- for (frame, icon_layout) in frames.iter().zip(layout.dock_icons.iter()) {
- let icon_global = appkit_rect(*icon_layout, bounds_height);
- let icon_frame = NSRect::new(
- NSPoint::new(
- icon_global.origin.x - dock_frame.origin.x,
- icon_global.origin.y - dock_frame.origin.y,
- ),
- icon_global.size,
- );
- let (_, icon, fallback) = target_identity(frame);
- if icon.is_null() {
- let tile = rounded_view(
- icon_frame,
- icon_frame.size.width * 0.22,
- color(0.98, 0.98, 0.99, 0.86),
- true,
- );
- add_text_label(
- tile,
- NSRect::new(
- NSPoint::new(0.0, (icon_frame.size.height - 24.0) / 2.0),
- NSSize::new(icon_frame.size.width, 24.0),
- ),
- &fallback.to_string(),
- (icon_frame.size.width * 0.46).clamp(12.0, 22.0),
- true,
- LabelTone::Dark,
- 1,
- );
- let _: () = msg_send![dock, addSubview: tile];
- } else {
- let image_view: *mut AnyObject = {
- let alloc: *mut AnyObject = msg_send![class!(NSImageView), alloc];
- msg_send![
- alloc,
- initWithFrame: NSRect::new(
- icon_frame.origin,
- icon_frame.size,
- )
- ]
- };
- let _: () = msg_send![image_view, setImageScaling: 3u64];
- let _: () = msg_send![image_view, setImage: icon];
- let _: () = msg_send![dock, addSubview: image_view];
- }
-
- let dot_size = 3.2_f64.min(icon_frame.size.width * 0.10);
- let dot_x = icon_frame.origin.x + (icon_frame.size.width - dot_size) / 2.0;
- add_circle(dock, dot_x, 2.2, dot_size, color(0.20, 0.72, 0.38, 0.92));
- }
- let _: () = msg_send![canvas, addSubview: dock];
-}
-
-fn session_tabs_layout(
- width: f64,
- height: f64,
- workspaces: &[PipWorkspaceSummary],
- selected_workspace_id: Option<&str>,
-) -> SessionTabsLayout {
- layout_session_tabs(
- LayoutRect {
- x: (width - 600.0).max(0.0) / 2.0,
- y: (height - SESSION_SELECTOR_HEIGHT).max(0.0),
- width: width.min(600.0),
- height: SESSION_SELECTOR_HEIGHT,
- },
- workspaces,
- selected_workspace_id,
- )
-}
-
-unsafe fn render_session_selector(
- canvas: *mut objc2::runtime::AnyObject,
- bounds: objc2_foundation::NSRect,
- workspaces: &[PipWorkspaceSummary],
- selected_workspace_id: Option<&str>,
-) {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
-
- let tabs = session_tabs_layout(
- bounds.size.width,
- bounds.size.height,
- workspaces,
- selected_workspace_id,
- );
- let Some(first) = tabs.tabs.first() else {
- return;
- };
- let last = tabs.tabs.last().expect("tabs are non-empty");
- let selector_frame = objc2_foundation::NSRect::new(
- objc2_foundation::NSPoint::new(first.rect.x - 6.0, first.rect.y - 3.0),
- objc2_foundation::NSSize::new(
- last.rect.x + last.rect.width - first.rect.x + 12.0,
- first.rect.height + 6.0,
- ),
- );
- let selector = visual_effect_view(selector_frame, 14.0, 6, 1, true);
- let selector_layer: *mut AnyObject = msg_send![selector, layer];
- set_layer_background(selector_layer, color(0.10, 0.12, 0.15, 0.30));
- set_layer_border(selector_layer, 0.6, color(1.0, 1.0, 1.0, 0.26));
-
- let delegate = HANDLES
- .lock()
- .unwrap()
- .as_ref()
- .map(|handles| handles.delegate as *mut AnyObject)
- .unwrap_or(std::ptr::null_mut());
- for (index, tab) in tabs.tabs.iter().enumerate() {
- let workspace = &workspaces[index];
- let selected = tab.selected;
- let (red, green, blue) = (
- f64::from(tab.accent.0) / 255.0,
- f64::from(tab.accent.1) / 255.0,
- f64::from(tab.accent.2) / 255.0,
- );
- let icon_frame = objc2_foundation::NSRect::new(
- objc2_foundation::NSPoint::new(tab.rect.x, tab.rect.y),
- objc2_foundation::NSSize::new(tab.rect.width, tab.rect.height),
- );
- let button: *mut AnyObject = {
- let allocated: *mut AnyObject = msg_send![class!(NSButton), alloc];
- msg_send![allocated, initWithFrame: icon_frame]
- };
- let _: () = msg_send![button, setBordered: false];
- let _: () = msg_send![button, setRefusesFirstResponder: true];
- let _: () = msg_send![button, setWantsLayer: true];
- let layer: *mut AnyObject = msg_send![button, layer];
- let _: () = msg_send![layer, setCornerRadius: 11.0_f64];
- set_continuous_corners(layer);
- set_layer_background(
- layer,
- color(red, green, blue, if selected { 0.96 } else { 0.62 }),
- );
- set_layer_border(
- layer,
- if selected { 1.4 } else { 0.5 },
- color(1.0, 1.0, 1.0, if selected { 0.94 } else { 0.38 }),
- );
- let initial = workspace
- .workspace_label
- .chars()
- .find(|character| character.is_alphanumeric())
- .unwrap_or('A')
- .to_uppercase()
- .collect::<String>();
- let title = ns_string(&initial);
- let tooltip = ns_string(&workspace.workspace_label);
- if !title.is_null() {
- let _: () = msg_send![button, setTitle: title];
- }
- if !tooltip.is_null() {
- let _: () = msg_send![button, setToolTip: tooltip];
- }
- let font: *mut AnyObject = msg_send![class!(NSFont), boldSystemFontOfSize: 10.0_f64];
- let _: () = msg_send![button, setFont: font];
- let _: () = msg_send![button, setTag: index as isize];
- if !delegate.is_null() {
- let _: () = msg_send![button, setTarget: delegate];
- let _: () = msg_send![button, setAction: objc2::sel!(selectWorkspace:)];
- }
- let _: () = msg_send![selector, addSubview: button];
- }
- let _: () = msg_send![canvas, addSubview: selector];
-}
-
-fn desktop_frame(bounds: objc2_foundation::NSRect) -> objc2_foundation::NSRect {
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- if !SHOW_SHELL_BORDER {
- return bounds;
- }
-
- NSRect::new(
- NSPoint::new(SHELL_SIDE_INSET, SHELL_BOTTOM_INSET),
- NSSize::new(
- (bounds.size.width - 2.0 * SHELL_SIDE_INSET).max(1.0),
- (bounds.size.height - SHELL_TOP_INSET - SHELL_BOTTOM_INSET).max(1.0),
- ),
- )
-}
-
-unsafe fn install_shell_details(
- content_view: *mut objc2::runtime::AnyObject,
- bounds: objc2_foundation::NSRect,
-) {
- use objc2::msg_send;
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- if !SHOW_SHELL_BORDER {
- return;
- }
-
- let grip_width = 24.0;
- let grip = rounded_view(
- NSRect::new(
- NSPoint::new(
- (bounds.size.width - grip_width) / 2.0,
- bounds.size.height - 9.0,
- ),
- NSSize::new(grip_width, 3.0),
- ),
- 1.5,
- color(0.08, 0.10, 0.13, 0.52),
- true,
- );
- // Keep the grip centred and pinned to the shell's top edge while resizing.
- let _: () = msg_send![grip, setAutoresizingMask: 13u64];
- let grip_layer: *mut objc2::runtime::AnyObject = msg_send![grip, layer];
- let _: () = msg_send![grip_layer, setShadowOpacity: 0.32_f32];
- let _: () = msg_send![grip_layer, setShadowRadius: 0.8_f64];
- let _: () = msg_send![grip_layer, setShadowOffset: NSSize::new(0.0, 1.0)];
- let _: () = msg_send![content_view, addSubview: grip];
-
- let marker = color(1.0, 1.0, 1.0, 0.48);
- for (index, length) in [10.0_f64, 6.0].into_iter().enumerate() {
- let bar = rounded_view(
- NSRect::new(
- NSPoint::new(
- (bounds.size.width - 16.0 + index as f64 * 4.0).max(0.0),
- 4.5 + index as f64 * 2.0,
- ),
- NSSize::new(length, 1.2),
- ),
- 0.6,
- marker,
- false,
- );
- let _: () = msg_send![bar, setFrameCenterRotation: -45.0_f64];
- // Move with the right edge while remaining in the bottom shell rail.
- let _: () = msg_send![bar, setAutoresizingMask: 1u64];
- let _: () = msg_send![content_view, addSubview: bar];
- }
-}
-
-fn resized_window_frame(
- start: objc2_foundation::NSRect,
- delta_x: f64,
- delta_y: f64,
- direction: isize,
- minimum: objc2_foundation::NSSize,
-) -> objc2_foundation::NSRect {
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- let mut x = start.origin.x;
- let mut y = start.origin.y;
- let mut width = start.size.width;
- let mut height = start.size.height;
- if direction & RESIZE_LEFT != 0 {
- let applied = delta_x.min(width - minimum.width);
- x += applied;
- width -= applied;
- }
- if direction & RESIZE_RIGHT != 0 {
- width = (width + delta_x).max(minimum.width);
- }
- if direction & RESIZE_BOTTOM != 0 {
- let applied = delta_y.min(height - minimum.height);
- y += applied;
- height -= applied;
- }
- if direction & RESIZE_TOP != 0 {
- height = (height + delta_y).max(minimum.height);
- }
- NSRect::new(NSPoint::new(x, y), NSSize::new(width, height))
-}
-
-fn agent_view_shell_hit_class() -> &'static objc2::runtime::AnyClass {
- use objc2::class;
- use objc2::declare::ClassBuilder;
-
- static CLASS: OnceLock<&'static objc2::runtime::AnyClass> = OnceLock::new();
- CLASS.get_or_init(|| {
- let mut builder = ClassBuilder::new("CuaDriverAgentViewShellHitView", class!(NSView))
- .expect("CuaDriverAgentViewShellHitView already registered");
- unsafe {
- builder.add_method(
- objc2::sel!(acceptsFirstMouse:),
- shell_accepts_first_mouse as extern "C" fn(_, _, _) -> objc2::runtime::Bool,
- );
- builder.add_method(
- objc2::sel!(mouseDown:),
- shell_mouse_down as extern "C" fn(_, _, _),
- );
- }
- builder.register()
- })
-}
-
-extern "C" fn shell_accepts_first_mouse(
- _view: *mut objc2::runtime::AnyObject,
- _selector: objc2::runtime::Sel,
- _event: *mut objc2::runtime::AnyObject,
-) -> objc2::runtime::Bool {
- objc2::runtime::Bool::YES
-}
-
-extern "C" fn shell_mouse_down(
- view: *mut objc2::runtime::AnyObject,
- _selector: objc2::runtime::Sel,
- event: *mut objc2::runtime::AnyObject,
-) {
- use objc2::msg_send;
- use objc2::runtime::AnyObject;
-
- if view.is_null() || event.is_null() {
+ if ns_data.is_null() {
return;
}
- unsafe {
- let window: *mut AnyObject = msg_send![view, window];
- if window.is_null() {
- return;
- }
- let direction = rust_string(msg_send![view, toolTip])
- .and_then(|value| value.parse::<isize>().ok())
- .unwrap_or(0);
- if direction == 0 {
- let _: () = msg_send![window, performWindowDragWithEvent: event];
- return;
- }
-
- let start_frame: objc2_foundation::NSRect = msg_send![window, frame];
- let minimum: objc2_foundation::NSSize = msg_send![window, minSize];
- let start_mouse: objc2_foundation::NSPoint =
- msg_send![objc2::class!(NSEvent), mouseLocation];
- let event_mask: u64 = (1 << 2) | (1 << 6);
- let distant_future: *mut AnyObject = msg_send![objc2::class!(NSDate), distantFuture];
- let default_mode = ns_string("kCFRunLoopDefaultMode");
- loop {
- let next: *mut AnyObject = msg_send![
- window,
- nextEventMatchingMask: event_mask
- untilDate: distant_future
- inMode: default_mode
- dequeue: true
- ];
- if next.is_null() {
- break;
- }
- let event_type: usize = msg_send![next, type];
- if event_type == 2 {
- break;
- }
- let mouse: objc2_foundation::NSPoint = msg_send![objc2::class!(NSEvent), mouseLocation];
- let frame = resized_window_frame(
- start_frame,
- mouse.x - start_mouse.x,
- mouse.y - start_mouse.y,
- direction,
- minimum,
- );
- let _: () = msg_send![window, setFrame: frame display: true];
- }
- }
-}
-
-unsafe fn add_shell_hit_view(
- parent: *mut objc2::runtime::AnyObject,
- frame: objc2_foundation::NSRect,
- direction: isize,
- autoresizing_mask: u64,
-) {
- use objc2::msg_send;
- use objc2::runtime::AnyObject;
-
- let allocated: *mut AnyObject = msg_send![agent_view_shell_hit_class(), alloc];
- let view: *mut AnyObject = msg_send![allocated, initWithFrame: frame];
- let direction_string = ns_string(&direction.to_string());
- let _: () = msg_send![view, setToolTip: direction_string];
- let _: () = msg_send![view, setAutoresizingMask: autoresizing_mask];
- let _: () = msg_send![parent, addSubview: view];
-}
-
-unsafe fn install_shell_interaction(
- content_view: *mut objc2::runtime::AnyObject,
- bounds: objc2_foundation::NSRect,
-) {
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- let edge = RESIZE_HIT_INSET;
- let width = bounds.size.width;
- let height = bounds.size.height;
- add_shell_hit_view(
- content_view,
- NSRect::new(
- NSPoint::new(edge, height - edge),
- NSSize::new(width - 2.0 * edge, edge),
- ),
- RESIZE_TOP,
- 10,
- );
- add_shell_hit_view(
- content_view,
- NSRect::new(
- NSPoint::new(edge, 0.0),
- NSSize::new(width - 2.0 * edge, edge),
- ),
- RESIZE_BOTTOM,
- 10,
- );
- add_shell_hit_view(
- content_view,
- NSRect::new(
- NSPoint::new(0.0, edge),
- NSSize::new(edge, height - 2.0 * edge),
- ),
- RESIZE_LEFT,
- 20,
- );
- add_shell_hit_view(
- content_view,
- NSRect::new(
- NSPoint::new(width - edge, edge),
- NSSize::new(edge, height - 2.0 * edge),
- ),
- RESIZE_RIGHT,
- 17,
- );
- for (origin, direction, mask) in [
- (NSPoint::new(0.0, 0.0), RESIZE_LEFT | RESIZE_BOTTOM, 4),
- (
- NSPoint::new(width - edge, 0.0),
- RESIZE_RIGHT | RESIZE_BOTTOM,
- 1,
- ),
- (
- NSPoint::new(0.0, height - edge),
- RESIZE_LEFT | RESIZE_TOP,
- 8,
- ),
- (
- NSPoint::new(width - edge, height - edge),
- RESIZE_RIGHT | RESIZE_TOP,
- 2,
- ),
- ] {
- add_shell_hit_view(
- content_view,
- NSRect::new(origin, NSSize::new(edge, edge)),
- direction,
- mask,
- );
- }
- add_shell_hit_view(
- content_view,
- NSRect::new(
- NSPoint::new(edge, height - SHELL_TOP_INSET),
- NSSize::new(width - 2.0 * edge, SHELL_TOP_INSET - edge),
- ),
- 0,
- 10,
- );
-}
-
-unsafe fn render_snapshot(snapshot: &ViewSnapshot) {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- let canvas = {
- let guard = HANDLES.lock().unwrap();
- match guard.as_ref() {
- Some(handles) => handles.canvas_view as *mut AnyObject,
- None => return,
- }
+ let img: *mut AnyObject = {
+ let alloc: *mut AnyObject = msg_send![class!(NSImage), alloc];
+ msg_send![alloc, initWithData: ns_data]
};
- let empty: *mut AnyObject = msg_send![class!(NSArray), array];
- let _: () = msg_send![canvas, setSubviews: empty];
+ if !img.is_null() {
+ let image_view = image_view_ptr as *mut AnyObject;
+ let _: () = msg_send![image_view, setImage: img];
+ }
- let bounds: NSRect = msg_send![canvas, bounds];
- let selector_height = if snapshot.workspaces.len() > 1 {
- SESSION_SELECTOR_HEIGHT
- } else {
- 0.0
- };
- let content_height = (bounds.size.height - selector_height).max(1.0);
- let target_sizes = snapshot
- .frames
- .iter()
- .map(|frame| {
- png_dimensions(&frame.png_bytes).unwrap_or(TargetSize {
- width: 16,
- height: 10,
- })
- })
- .collect::<Vec<_>>();
- let layout = layout_desktop(bounds.size.width, content_height, &target_sizes);
- if snapshot.frames.is_empty() {
- let waiting = appkit_rect(layout.desktop, content_height);
- add_text_label(
- canvas,
- NSRect::new(
- NSPoint::new(
- waiting.origin.x + 16.0,
- waiting.origin.y + waiting.size.height / 2.0 - 12.0,
- ),
- NSSize::new((waiting.size.width - 32.0).max(40.0), 24.0),
- ),
- "Waiting for an exact window or browser tab...",
- 12.0,
- false,
- LabelTone::Muted,
- 1,
- );
- } else {
- for (frame, target_layout) in snapshot.frames.iter().zip(layout.targets.iter().copied()) {
- render_target_window(
- canvas,
- frame,
- target_layout,
- content_height,
- snapshot.active_view_id.as_deref() == Some(frame.target.view_id().as_str()),
- );
+ // Update the label. NSString::stringWithUTF8String requires NUL
+ // termination; copy into a CString so we hand a clean buffer.
+ if let Ok(cstr) = std::ffi::CString::new(frame.action_label) {
+ let ns_str: *mut AnyObject = msg_send![
+ class!(NSString),
+ stringWithUTF8String: cstr.as_ptr() as *const u8
+ ];
+ if !ns_str.is_null() {
+ let label = label_ptr as *mut AnyObject;
+ let _: () = msg_send![label, setStringValue: ns_str];
}
- // Keep the resting Agent View free of a redundant internal Dock.
}
- render_session_selector(
- canvas,
- bounds,
- &snapshot.workspaces,
- snapshot.selected_workspace_id.as_deref(),
- );
}
unsafe extern "C" fn shutdown_cb(_ctx: *mut c_void) {
@@ -1163,90 +179,27 @@ unsafe extern "C" fn shutdown_cb(_ctx: *mut c_void) {
use objc2::runtime::AnyObject;
let handles = HANDLES.lock().unwrap().take();
- VIEW_MODEL.lock().unwrap().take();
- if let Some(handles) = handles {
- let window = handles.window as *mut AnyObject;
- let _: () = msg_send![window, setDelegate: std::ptr::null_mut::<AnyObject>()];
- let _: () = msg_send![window, orderOut: std::ptr::null_mut::<AnyObject>()];
- let _: () = msg_send![window, close];
- let _ = handles.delegate;
- }
-}
-
-fn agent_view_delegate_class() -> &'static objc2::runtime::AnyClass {
- use objc2::class;
- use objc2::declare::ClassBuilder;
-
- static CLASS: OnceLock<&'static objc2::runtime::AnyClass> = OnceLock::new();
- CLASS.get_or_init(|| {
- let superclass = class!(NSObject);
- let mut builder = ClassBuilder::new("CuaDriverAgentViewDelegate", superclass)
- .expect("CuaDriverAgentViewDelegate already registered");
- unsafe {
- builder.add_method(
- objc2::sel!(windowDidResize:),
- on_window_did_resize as extern "C" fn(_, _, _),
- );
- builder.add_method(
- objc2::sel!(selectWorkspace:),
- on_select_workspace as extern "C" fn(_, _, _),
- );
+ if let Some(h) = handles {
+ let win = h.window as *mut AnyObject;
+ if !win.is_null() {
+ let _: () = msg_send![win, orderOut: std::ptr::null_mut::<AnyObject>()];
+ let _: () = msg_send![win, close];
}
- builder.register()
- })
-}
-
-unsafe fn agent_view_delegate_instance() -> *mut objc2::runtime::AnyObject {
- use objc2::msg_send;
- use objc2::runtime::AnyObject;
-
- let class = agent_view_delegate_class();
- let allocated: *mut AnyObject = msg_send![class, alloc];
- msg_send![allocated, init]
-}
-
-extern "C" fn on_window_did_resize(
- _delegate: *mut objc2::runtime::AnyObject,
- _selector: objc2::runtime::Sel,
- _notification: *mut objc2::runtime::AnyObject,
-) {
- let snapshot = current_snapshot();
- unsafe {
- update_wallpaper_frame();
- render_snapshot(&snapshot);
- };
-}
-
-extern "C" fn on_select_workspace(
- _delegate: *mut objc2::runtime::AnyObject,
- _selector: objc2::runtime::Sel,
- sender: *mut objc2::runtime::AnyObject,
-) {
- if sender.is_null() {
- return;
- }
- let index: isize = unsafe { objc2::msg_send![sender, tag] };
- if index < 0 {
- return;
}
- let snapshot = {
- let mut model = VIEW_MODEL.lock().unwrap();
- let Some(model) = model.as_mut() else {
- return;
- };
- let workspaces = model.workspaces();
- let Some(workspace) = workspaces.get(index as usize) else {
- return;
- };
- let workspace_id = workspace.workspace_id.clone();
- model.select_workspace(&workspace_id);
- clone_snapshot(model)
- };
- unsafe { render_snapshot(&snapshot) };
}
-/// Park the main thread in `NSApplication.run()` for the main-queue AppKit
-/// work used by Agent View when the cursor overlay is disabled.
+// ── AppKit main loop helper for Serve mode ───────────────────────────────
+
+/// Park the main thread in `NSApplication.run()`. Used by `cua-driver
+/// serve --experimental-pip` so the dispatch_async_f → main queue
+/// path PiP frames go through can be drained. Mirrors the cursor
+/// overlay's `run_appkit` startup (Accessory activation policy →
+/// finishLaunching → run) without installing the overlay's
+/// CALayer-backed window itself.
+///
+/// Never returns — the background `serve::run_serve_cmd` thread calls
+/// `std::process::exit` when it finishes, which tears down NSApp at
+/// the same time.
pub fn run_appkit_main_loop() {
use objc2::runtime::AnyObject;
use objc2::{class, msg_send};
@@ -1255,514 +208,218 @@ pub fn run_appkit_main_loop() {
.expect("run_appkit_main_loop must be called from the main thread");
unsafe {
let app: *mut AnyObject = msg_send![class!(NSApplication), sharedApplication];
+ // Accessory policy: no Dock icon, no menu bar. Keeps the
+ // daemon out of the user's application switcher, same as
+ // the cursor overlay's NSApp setup.
let _: bool = msg_send![app, setActivationPolicy: 1i64];
let _: () = msg_send![app, finishLaunching];
let _: () = msg_send![app, run];
}
}
+// ── Factory ──────────────────────────────────────────────────────────────
+
pub struct MacosPipBackendFactory;
impl PipBackendFactory for MacosPipBackendFactory {
fn start(&self, cfg: &PipConfig) -> anyhow::Result<Box<dyn PipBackend>> {
- dispatch_to_main(cfg.clone(), init_cb);
+ // Window construction must happen on the main thread. We hand
+ // off via dispatch_async_f and return immediately — the first
+ // few frames may be dropped while init races, which is fine
+ // for a live-preview UX.
+ let cfg_clone = cfg.clone();
+ dispatch_to_main(cfg_clone, init_cb);
Ok(Box::new(MacosPipBackend))
}
}
-unsafe fn install_wallpaper(
- content_view: *mut objc2::runtime::AnyObject,
- screen: *mut objc2::runtime::AnyObject,
- bounds: objc2_foundation::NSRect,
-) -> (
- *mut objc2::runtime::AnyObject,
- *mut objc2::runtime::AnyObject,
- *mut objc2::runtime::AnyObject,
- *mut objc2::runtime::AnyObject,
-) {
- use objc2::runtime::AnyObject;
- use objc2::{class, msg_send};
-
- // The same wallpaper continues beneath the shell rails. The centre is
- // covered by a sharp inset copy while the outer copy is blurred and tinted.
- let glass_frame = rounded_view(
- bounds,
- if SHOW_SHELL_BORDER {
- CONTAINER_RADIUS
- } else {
- PIP_CORNER_RADIUS
- },
- color(0.18, 0.21, 0.25, if SHOW_SHELL_BORDER { 0.16 } else { 0.0 }),
- true,
- );
- let _: () = msg_send![glass_frame, setAutoresizingMask: 18u64];
- let glass_layer: *mut AnyObject = msg_send![glass_frame, layer];
- set_continuous_corners(glass_layer);
- if SHOW_SHELL_BORDER {
- set_layer_border(glass_layer, 0.8, color(1.0, 1.0, 1.0, 0.50));
- }
- let _: () = msg_send![content_view, addSubview: glass_frame];
-
- let glass_bounds: objc2_foundation::NSRect = msg_send![glass_frame, bounds];
- let shell_wallpaper_view: *mut AnyObject = {
- let allocated: *mut AnyObject = msg_send![class!(NSImageView), alloc];
- msg_send![allocated, initWithFrame: glass_bounds]
- };
- let _: () = msg_send![shell_wallpaper_view, setImageScaling: 2u64];
- if SHOW_SHELL_BORDER {
- let _: () = msg_send![glass_frame, addSubview: shell_wallpaper_view];
- }
-
- let backdrop: *mut AnyObject = {
- let allocated: *mut AnyObject = msg_send![class!(NSVisualEffectView), alloc];
- msg_send![
- allocated,
- initWithFrame: objc2_foundation::NSRect::new(
- objc2_foundation::NSPoint::new(0.0, 0.0),
- bounds.size,
- )
- ]
- };
- let _: () = msg_send![backdrop, setAutoresizingMask: 18u64];
- let _: () = msg_send![backdrop, setMaterial: 13i64]; // HUD window material.
- let _: () = msg_send![backdrop, setBlendingMode: 1i64]; // Blur the wallpaper within the shell.
- let _: () = msg_send![backdrop, setState: 1i64]; // Keep the blur active while non-key.
- let _: () = msg_send![backdrop, setEmphasized: true];
- if SHOW_SHELL_BORDER {
- let _: () = msg_send![glass_frame, addSubview: backdrop];
- }
-
- // A restrained silver-to-graphite tint keeps the blurred backdrop legible
- // as a shell instead of exposing raw, high-contrast shapes behind it.
- let body = gradient_view(
- objc2_foundation::NSRect::new(objc2_foundation::NSPoint::new(0.0, 0.0), bounds.size),
- CONTAINER_RADIUS,
- color(0.68, 0.71, 0.76, 0.34),
- color(0.15, 0.18, 0.22, 0.52),
- );
- let _: () = msg_send![body, setAutoresizingMask: 18u64];
- let body_layer: *mut AnyObject = msg_send![body, layer];
- let scale: f64 = msg_send![screen, backingScaleFactor];
- if scale > 0.0 {
- // Layer-hosting views do not inherit the backing scale, and the chrome
- // hairlines are sub-point.
- let _: () = msg_send![body_layer, setContentsScale: scale];
- }
- if SHOW_SHELL_BORDER {
- let _: () = msg_send![glass_frame, addSubview: body];
- }
-
- // Light-from-above specular along the top edge of the chrome.
- let highlight = rounded_view(
- objc2_foundation::NSRect::new(
- objc2_foundation::NSPoint::new(CONTAINER_RADIUS, bounds.size.height - 1.7),
- objc2_foundation::NSSize::new(
- (bounds.size.width - 2.0 * CONTAINER_RADIUS).max(1.0),
- 1.0,
- ),
- ),
- 0.5,
- color(1.0, 1.0, 1.0, 0.38),
- true,
- );
- let _: () = msg_send![highlight, setAutoresizingMask: 10u64];
- if SHOW_SHELL_BORDER {
- let _: () = msg_send![glass_frame, addSubview: highlight];
- }
-
- let container_frame = desktop_frame(bounds);
- let container_radius = if SHOW_SHELL_BORDER {
- 8.5
- } else {
- PIP_CORNER_RADIUS
- };
- let wallpaper_shadow = rounded_view(
- container_frame,
- container_radius,
- color(0.0, 0.0, 0.0, 0.01),
- false,
- );
- let _: () = msg_send![wallpaper_shadow, setAutoresizingMask: 18u64];
- let shadow_layer: *mut AnyObject = msg_send![wallpaper_shadow, layer];
- set_continuous_corners(shadow_layer);
- // The inset shadow separates the desktop from the thicker shell rails.
- let _: () = msg_send![shadow_layer, setShadowOpacity: 0.62_f32];
- let _: () = msg_send![shadow_layer, setShadowRadius: 4.0_f64];
- let _: () = msg_send![shadow_layer, setShadowOffset: objc2_foundation::NSSize::new(0.0, -1.5)];
- let shadow_color = color(0.0, 0.0, 0.0, 0.85);
- let shadow_cg: *mut CGColor = msg_send![shadow_color, CGColor];
- let _: () = msg_send![shadow_layer, setShadowColor: shadow_cg];
- if SHOW_SHELL_BORDER {
- let _: () = msg_send![content_view, addSubview: wallpaper_shadow];
- }
-
- // Machined inner edge. It hugs the seam from the chrome side, so the
- // miniature desktop stays separated from the graphite body whether the
- // content under it is a bright wallpaper or a dark window.
- let seam_outset = 0.75;
- let seam = rounded_view(
- objc2_foundation::NSRect::new(
- objc2_foundation::NSPoint::new(
- container_frame.origin.x - seam_outset,
- container_frame.origin.y - seam_outset,
- ),
- objc2_foundation::NSSize::new(
- container_frame.size.width + 2.0 * seam_outset,
- container_frame.size.height + 2.0 * seam_outset,
- ),
- ),
- container_radius + seam_outset,
- color(0.0, 0.0, 0.0, 0.0),
- false,
- );
- let _: () = msg_send![seam, setAutoresizingMask: 18u64];
- let seam_layer: *mut AnyObject = msg_send![seam, layer];
- set_continuous_corners(seam_layer);
- if SHOW_SHELL_BORDER {
- set_layer_border(seam_layer, seam_outset, color(1.0, 1.0, 1.0, 0.20));
- }
- let _: () = msg_send![content_view, addSubview: seam];
-
- let wallpaper_container = rounded_view(
- container_frame,
- container_radius,
- color(0.02, 0.03, 0.04, if SHOW_SHELL_BORDER { 0.68 } else { 0.0 }),
- true,
- );
- let _: () = msg_send![wallpaper_container, setAutoresizingMask: 18u64];
- let container_layer: *mut AnyObject = msg_send![wallpaper_container, layer];
- set_continuous_corners(container_layer);
- // Dark hairline where the miniature desktop meets the chrome. Softer than
- // the flat-outline pass, because the graphite body now carries the seam.
- if SHOW_SHELL_BORDER {
- set_layer_border(container_layer, 0.8, color(0.0, 0.0, 0.0, 0.30));
- }
- let container_bounds: objc2_foundation::NSRect = msg_send![wallpaper_container, bounds];
- let wallpaper_view: *mut AnyObject = {
- let allocated: *mut AnyObject = msg_send![class!(NSImageView), alloc];
- msg_send![allocated, initWithFrame: container_bounds]
- };
- let _: () = msg_send![wallpaper_view, setImageScaling: 2u64];
- let workspace: *mut AnyObject = msg_send![class!(NSWorkspace), sharedWorkspace];
- let url: *mut AnyObject = msg_send![workspace, desktopImageURLForScreen: screen];
- if !url.is_null() {
- let allocated: *mut AnyObject = msg_send![class!(NSImage), alloc];
- let wallpaper: *mut AnyObject = msg_send![allocated, initWithContentsOfURL: url];
- if !wallpaper.is_null() {
- let _: () = msg_send![shell_wallpaper_view, setImage: wallpaper];
- let _: () = msg_send![wallpaper_view, setImage: wallpaper];
- }
- }
- let _: () = msg_send![wallpaper_container, addSubview: wallpaper_view];
- let _: () = msg_send![content_view, addSubview: wallpaper_container];
-
- (
- glass_frame,
- shell_wallpaper_view,
- wallpaper_container,
- wallpaper_view,
- )
-}
-
-fn aspect_fill_frame(
- bounds: objc2_foundation::NSRect,
- image_size: objc2_foundation::NSSize,
-) -> objc2_foundation::NSRect {
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- if image_size.width <= 0.0 || image_size.height <= 0.0 {
- return bounds;
- }
- let scale = (bounds.size.width / image_size.width).max(bounds.size.height / image_size.height);
- let width = image_size.width * scale;
- let height = image_size.height * scale;
- NSRect::new(
- NSPoint::new(
- bounds.origin.x + (bounds.size.width - width) / 2.0,
- bounds.origin.y + (bounds.size.height - height) / 2.0,
- ),
- NSSize::new(width, height),
- )
-}
-
-unsafe fn update_wallpaper_frame() {
- use objc2::msg_send;
- use objc2::runtime::AnyObject;
- use objc2_foundation::{NSRect, NSSize};
-
- let (shell_container, shell_view, wallpaper_container, wallpaper_view) = {
- let guard = HANDLES.lock().unwrap();
- let Some(handles) = guard.as_ref() else {
- return;
- };
- (
- handles.shell_wallpaper_container as *mut AnyObject,
- handles.shell_wallpaper_view as *mut AnyObject,
- handles.wallpaper_container as *mut AnyObject,
- handles.wallpaper_view as *mut AnyObject,
- )
- };
- for (container, view) in [
- (shell_container, shell_view),
- (wallpaper_container, wallpaper_view),
- ] {
- let bounds: NSRect = msg_send![container, bounds];
- let image: *mut AnyObject = msg_send![view, image];
- let image_size = if image.is_null() {
- NSSize::new(0.0, 0.0)
- } else {
- msg_send![image, size]
- };
- let frame = aspect_fill_frame(bounds, image_size);
- let _: () = msg_send![view, setFrame: frame];
- }
-}
-
unsafe extern "C" fn init_cb(ctx: *mut c_void) {
use objc2::runtime::AnyObject;
use objc2::{class, msg_send};
use objc2_foundation::{NSPoint, NSRect, NSSize};
let cfg: PipConfig = *Box::from_raw(ctx as *mut PipConfig);
+
+ // Idempotency guard — `start()` should only be called once per
+ // process, but cheap to defend against duplicate calls.
if HANDLES.lock().unwrap().is_some() {
return;
}
+ // ── Resolve geometry ──
+ // AppKit windows use a bottom-left origin in screen coordinates.
+ // The CLI flag uses a top-left X11-style origin (since that's the
+ // mental model agents have for screenshots). Flip Y here so a
+ // `+0+0` flag puts the window in the top-left corner.
let screen: *mut AnyObject = msg_send![class!(NSScreen), mainScreen];
if screen.is_null() {
+ // Headless environment (CI) — skip silently. The daemon keeps
+ // running without a PiP window.
return;
}
let screen_frame: NSRect = msg_send![screen, frame];
- let minimum = NSSize::new(360.0, 260.0);
- let width = (cfg.geometry.width as f64).max(minimum.width);
- let height = (cfg.geometry.height as f64).max(minimum.height);
+
+ let w = cfg.geometry.width as f64;
+ let h = cfg.geometry.height as f64;
+ // Default placement: top-right corner with a 24pt inset, mirroring
+ // the macOS conventions for floating utility windows.
let inset = 24.0_f64;
let (top_left_x, top_left_y) = match (cfg.geometry.x, cfg.geometry.y) {
(Some(x), Some(y)) => (x as f64, y as f64),
- _ => (screen_frame.size.width - width - inset, inset),
+ _ => (screen_frame.size.width - w - inset, inset),
};
- let bottom_y = screen_frame.size.height - top_left_y - height;
- let rect = NSRect::new(
- NSPoint::new(top_left_x, bottom_y),
- NSSize::new(width, height),
- );
-
- let style_mask: u64 = (1 << 3) | (1 << 7);
+ // Convert top-left → bottom-left for AppKit.
+ let bottom_y = screen_frame.size.height - top_left_y - h;
+ let rect = NSRect::new(NSPoint::new(top_left_x, bottom_y), NSSize::new(w, h));
+
+ // ── NSWindow ──
+ // Borderless so the image owns the whole rectangle. No close button
+ // / title bar — the window is owned by the daemon session lifecycle.
+ // The rounded-corner look comes from a CALayer-backed content view
+ // with cornerRadius + masksToBounds; the window itself stays
+ // transparent outside the rounded rect.
+ // NSWindowStyleMaskBorderless = 0
+ let style_mask: u64 = 0;
let backing_store_buffered: u64 = 2;
- let window: *mut AnyObject = {
- let allocated: *mut AnyObject = msg_send![class!(NSPanel), alloc];
+ let win: *mut AnyObject = {
+ let alloc: *mut AnyObject = msg_send![class!(NSWindow), alloc];
msg_send![
- allocated,
+ alloc,
initWithContentRect: rect
styleMask: style_mask
backing: backing_store_buffered
defer: false
]
};
- if window.is_null() {
+ if win.is_null() {
return;
}
+ // Transparent backing so the corners outside the CALayer-clipped
+ // content view show whatever's underneath — gives the floating-pill
+ // look. The shadow comes from AppKit's default `hasShadow: true`.
let clear: *mut AnyObject = msg_send![class!(NSColor), clearColor];
- let _: () = msg_send![window, setBackgroundColor: clear];
- let _: () = msg_send![window, setOpaque: false];
- let _: () = msg_send![window, setHasShadow: SHOW_SHELL_BORDER];
- // Screen sharing and recordings should include Agent View itself.
- let _: () = msg_send![window, setSharingType: 1u64];
- let _: () = msg_send![window, setMovableByWindowBackground: true];
- let _: () = msg_send![window, setIgnoresMouseEvents: false];
- let _: () = msg_send![window, setBecomesKeyOnlyIfNeeded: true];
- let _: () = msg_send![window, setFloatingPanel: true];
- let _: () = msg_send![window, setLevel: 3i64];
- let _: () = msg_send![window, setMinSize: minimum];
+ let _: () = msg_send![win, setBackgroundColor: clear];
+ let _: () = msg_send![win, setOpaque: false];
+ let _: () = msg_send![win, setHasShadow: true];
+ // Draggable from anywhere since there's no title bar.
+ let _: () = msg_send![win, setMovableByWindowBackground: true];
+
+ // Floating window level (NSFloatingWindowLevel = 3).
+ let _: () = msg_send![win, setLevel: 3i64];
+
+ // Collection behavior: visible across all spaces, no Mission
+ // Control affordance, never the main / key window.
+ // 1<<0 CanJoinAllSpaces | 1<<4 Stationary | 1<<8 FullScreenAuxiliary
+ // 1<<6 Transient | 1<<7 IgnoresCycle
let behavior: u64 = (1 << 0) | (1 << 4) | (1 << 8) | (1 << 6) | (1 << 7);
- let _: () = msg_send![window, setCollectionBehavior: behavior];
- let _: () = msg_send![window, setReleasedWhenClosed: false];
- let _: () = msg_send![window, setHidesOnDeactivate: false];
+ let _: () = msg_send![win, setCollectionBehavior: behavior];
- let content_view: *mut AnyObject = msg_send![window, contentView];
+ let _: () = msg_send![win, setReleasedWhenClosed: false];
+ let _: () = msg_send![win, setHidesOnDeactivate: false];
+
+ // ── Content view: rounded-corner black backing ──
+ // wantsLayer + masksToBounds clips the image view to the rounded
+ // rect. The backing CALayer color shows wherever the (proportionally
+ // scaled) image leaves gaps above/below or left/right.
+ let content_view: *mut AnyObject = msg_send![win, contentView];
let _: () = msg_send![content_view, setWantsLayer: true];
let content_layer: *mut AnyObject = msg_send![content_view, layer];
- let _: () = msg_send![
- content_layer,
- setCornerRadius: if SHOW_SHELL_BORDER {
- CONTAINER_RADIUS
- } else {
- PIP_CORNER_RADIUS
- }
+ let _: () = msg_send![content_layer, setCornerRadius: 12.0_f64];
+ let _: () = msg_send![content_layer, setMasksToBounds: true];
+ let black: *mut AnyObject = msg_send![
+ class!(NSColor),
+ colorWithCalibratedRed: 0.0_f64
+ green: 0.0_f64
+ blue: 0.0_f64
+ alpha: 1.0_f64
];
- set_continuous_corners(content_layer);
- // The chrome view clips its own children. Masking here too would clip the
- // chrome's own outer rim stroke in half and flatten the frame.
- let _: () = msg_send![content_layer, setMasksToBounds: false];
- set_layer_background(content_layer, color(0.0, 0.0, 0.0, 0.0));
- let bounds: NSRect = msg_send![content_view, bounds];
- let (shell_wallpaper_container, shell_wallpaper_view, wallpaper_container, wallpaper_view) =
- install_wallpaper(content_view, screen, bounds);
- let inner_frame = desktop_frame(bounds);
+ let black_cg: *mut CGColor = msg_send![black, CGColor];
+ let _: () = msg_send![content_layer, setBackgroundColor: black_cg];
- let canvas: *mut AnyObject = {
- let allocated: *mut AnyObject = msg_send![class!(NSView), alloc];
- msg_send![allocated, initWithFrame: inner_frame]
+ // ── NSImageView: fills the entire content view ──
+ let image_rect = NSRect::new(NSPoint::new(0.0, 0.0), NSSize::new(w, h));
+ let image_view: *mut AnyObject = {
+ let alloc: *mut AnyObject = msg_send![class!(NSImageView), alloc];
+ msg_send![alloc, initWithFrame: image_rect]
};
- let _: () = msg_send![canvas, setAutoresizingMask: 18u64];
- // The content view no longer masks, so the canvas clips its own contents to
- // the container silhouette.
- let _: () = msg_send![canvas, setWantsLayer: true];
- let canvas_layer: *mut AnyObject = msg_send![canvas, layer];
- let _: () = msg_send![
- canvas_layer,
- setCornerRadius: if SHOW_SHELL_BORDER {
- 8.5_f64
- } else {
- PIP_CORNER_RADIUS
- }
- ];
- set_continuous_corners(canvas_layer);
- let _: () = msg_send![canvas_layer, setMasksToBounds: true];
- let _: () = msg_send![content_view, addSubview: canvas];
- install_shell_details(content_view, bounds);
- install_shell_interaction(content_view, bounds);
-
- let delegate = agent_view_delegate_instance();
- let _: () = msg_send![window, setDelegate: delegate];
- *HANDLES.lock().unwrap() = Some(NativeHandles {
- window: window as usize,
- canvas_view: canvas as usize,
- shell_wallpaper_container: shell_wallpaper_container as usize,
- shell_wallpaper_view: shell_wallpaper_view as usize,
- wallpaper_container: wallpaper_container as usize,
- wallpaper_view: wallpaper_view as usize,
- delegate: delegate as usize,
- });
- *VIEW_MODEL.lock().unwrap() = Some(PipViewModel::new(12));
- update_wallpaper_frame();
- render_snapshot(&ViewSnapshot {
- frames: Vec::new(),
- workspaces: Vec::new(),
- selected_workspace_id: None,
- active_view_id: None,
- });
- let _: () = msg_send![window, orderFrontRegardless];
+ // NSImageScaleProportionallyUpOrDown = 3 (preserve aspect ratio).
+ // AppKit types this as NSUInteger — passing signed i64 triggers
+ // an objc2 type-encoding panic on macOS 26+.
+ let _: () = msg_send![image_view, setImageScaling: 3u64];
- tracing::info!(
- target: "pip",
- "Agent View miniature desktop initialised ({}x{})",
- width,
- height
+ // ── Label overlay: pill at bottom-center ──
+ // Container NSView with semi-transparent black backing + rounded
+ // corners (half its height for a fully rounded pill). NSTextField
+ // sits inside, centered, white text on the dark backing.
+ let pill_height = 22.0_f64;
+ let pill_inset_x = 16.0_f64;
+ let pill_inset_bottom = 10.0_f64;
+ let pill_w = (w - pill_inset_x * 2.0).max(60.0);
+ let pill_rect = NSRect::new(
+ NSPoint::new(pill_inset_x, pill_inset_bottom),
+ NSSize::new(pill_w, pill_height),
);
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
-
- fn workspace(id: &str, target_count: usize, updated_ms: u64) -> PipWorkspaceSummary {
- PipWorkspaceSummary {
- workspace_id: id.to_owned(),
- workspace_label: id.to_owned(),
- target_count,
- updated_ms,
- }
- }
-
- #[test]
- fn parses_native_pid_from_exact_window_target() {
- assert_eq!(parse_native_pid("window:501:90210"), Some(501));
- assert_eq!(parse_native_pid("browser:target:tab"), None);
- assert_eq!(parse_native_pid("window:not-a-pid:7"), None);
- }
-
- #[test]
- fn truncates_long_titles_without_touching_short_ones() {
- assert_eq!(truncate_label("Calculator", 28), "Calculator");
- assert_eq!(
- truncate_label("abcdefghijklmnopqrstuvwxyz", 8),
- "abcdefgh..."
- );
- }
-
- #[test]
- fn aspect_fill_centers_and_crops_without_distortion() {
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- let wide = aspect_fill_frame(
- NSRect::new(NSPoint::new(0.0, 0.0), NSSize::new(640.0, 420.0)),
- NSSize::new(1440.0, 900.0),
- );
- assert_eq!(wide.size, NSSize::new(672.0, 420.0));
- assert_eq!(wide.origin, NSPoint::new(-16.0, 0.0));
-
- let tall = aspect_fill_frame(
- NSRect::new(NSPoint::new(0.0, 0.0), NSSize::new(380.0, 660.0)),
- NSSize::new(1440.0, 900.0),
- );
- assert_eq!(tall.size, NSSize::new(1056.0, 660.0));
- assert_eq!(tall.origin, NSPoint::new(-338.0, 0.0));
- }
-
- #[test]
- fn desktop_frame_uses_full_bounds_when_shell_border_is_hidden() {
- use objc2_foundation::{NSPoint, NSRect, NSSize};
-
- let outer = NSRect::new(NSPoint::new(0.0, 0.0), NSSize::new(620.0, 420.0));
- let inner = desktop_frame(outer);
-
- assert_eq!(inner, outer);
- }
+ let pill: *mut AnyObject = {
+ let alloc: *mut AnyObject = msg_send![class!(NSView), alloc];
+ msg_send![alloc, initWithFrame: pill_rect]
+ };
+ let _: () = msg_send![pill, setWantsLayer: true];
+ let pill_layer: *mut AnyObject = msg_send![pill, layer];
+ let _: () = msg_send![pill_layer, setCornerRadius: pill_height / 2.0];
+ let _: () = msg_send![pill_layer, setMasksToBounds: true];
+ let pill_bg: *mut AnyObject = msg_send![
+ class!(NSColor),
+ colorWithCalibratedRed: 0.0_f64
+ green: 0.0_f64
+ blue: 0.0_f64
+ alpha: 0.62_f64
+ ];
+ let pill_bg_cg: *mut CGColor = msg_send![pill_bg, CGColor];
+ let _: () = msg_send![pill_layer, setBackgroundColor: pill_bg_cg];
- #[test]
- fn session_selector_is_local_compact_and_hidden_for_one_session() {
- let one = [workspace("one", 1, 1)];
- assert!(session_tabs_layout(602.0, 390.0, &one, Some("one"))
- .tabs
- .is_empty());
- let workspaces = [
- workspace("one", 1, 1),
- workspace("two", 1, 1),
- workspace("three", 1, 1),
+ // NSTextField inside the pill — horizontal padding via frame inset.
+ let label_rect = NSRect::new(
+ NSPoint::new(10.0, 0.0),
+ NSSize::new(pill_w - 20.0, pill_height),
+ );
+ let label: *mut AnyObject = {
+ let alloc: *mut AnyObject = msg_send![class!(NSTextField), alloc];
+ msg_send![alloc, initWithFrame: label_rect]
+ };
+ let _: () = msg_send![label, setBezeled: false];
+ let _: () = msg_send![label, setDrawsBackground: false];
+ let _: () = msg_send![label, setEditable: false];
+ let _: () = msg_send![label, setSelectable: false];
+ // NSTextAlignment.center = 2 (NSInteger).
+ let _: () = msg_send![label, setAlignment: 2i64];
+ let white: *mut AnyObject = msg_send![class!(NSColor), whiteColor];
+ let _: () = msg_send![label, setTextColor: white];
+ let font: *mut AnyObject = msg_send![class!(NSFont), systemFontOfSize: 11.0_f64];
+ let _: () = msg_send![label, setFont: font];
+ // Initial placeholder text — overwritten on the first frame.
+ if let Ok(cstr) = std::ffi::CString::new("waiting for first action…") {
+ let ns_str: *mut AnyObject = msg_send![
+ class!(NSString),
+ stringWithUTF8String: cstr.as_ptr() as *const u8
];
- let tabs = session_tabs_layout(602.0, 390.0, &workspaces, Some("one"));
- assert_eq!(tabs.tabs.len(), 3);
- assert!(tabs.tabs.iter().all(|tab| {
- tab.rect.x >= 0.0
- && tab.rect.y >= 0.0
- && tab.rect.x + tab.rect.width <= 602.0
- && tab.rect.y + tab.rect.height <= 390.0
- }));
+ if !ns_str.is_null() {
+ let _: () = msg_send![label, setStringValue: ns_str];
+ }
}
- #[test]
- fn every_resize_direction_keeps_the_opposite_edges_anchored() {
- use objc2_foundation::{NSPoint, NSRect, NSSize};
+ let _: () = msg_send![content_view, addSubview: image_view];
+ let _: () = msg_send![pill, addSubview: label];
+ let _: () = msg_send![content_view, addSubview: pill];
- let start = NSRect::new(NSPoint::new(100.0, 100.0), NSSize::new(600.0, 400.0));
- let minimum = NSSize::new(360.0, 260.0);
- let left = resized_window_frame(start, 40.0, 0.0, RESIZE_LEFT, minimum);
- assert_eq!(left.origin.x, 140.0);
- assert_eq!(left.size.width, 560.0);
- let right = resized_window_frame(start, 40.0, 0.0, RESIZE_RIGHT, minimum);
- assert_eq!(right.origin.x, 100.0);
- assert_eq!(right.size.width, 640.0);
- let bottom = resized_window_frame(start, 0.0, 30.0, RESIZE_BOTTOM, minimum);
- assert_eq!(bottom.origin.y, 130.0);
- assert_eq!(bottom.size.height, 370.0);
- let top = resized_window_frame(start, 0.0, 30.0, RESIZE_TOP, minimum);
- assert_eq!(top.origin.y, 100.0);
- assert_eq!(top.size.height, 430.0);
- let corner = resized_window_frame(start, -25.0, 35.0, RESIZE_LEFT | RESIZE_TOP, minimum);
- assert_eq!(corner.origin, NSPoint::new(75.0, 100.0));
- assert_eq!(corner.size, NSSize::new(625.0, 435.0));
- }
+ // Show the window without making it key or activating the app.
+ let _: () = msg_send![win, orderFrontRegardless];
- #[test]
- fn resize_geometry_enforces_the_minimum_without_moving_far_edges() {
- use objc2_foundation::{NSPoint, NSRect, NSSize};
+ *HANDLES.lock().unwrap() = Some(NativeHandles {
+ window: win as usize,
+ image_view: image_view as usize,
+ label: label as usize,
+ });
- let start = NSRect::new(NSPoint::new(100.0, 100.0), NSSize::new(600.0, 400.0));
- let minimum = NSSize::new(360.0, 260.0);
- let frame = resized_window_frame(start, 500.0, 500.0, RESIZE_LEFT | RESIZE_BOTTOM, minimum);
- assert_eq!(frame.origin, NSPoint::new(340.0, 240.0));
- assert_eq!(frame.size, minimum);
- }
+ tracing::info!(target: "pip", "PiP window initialised ({}x{})", cfg.geometry.width, cfg.geometry.height);
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs
index 03feba03ab..a1dc0cbab1 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs
@@ -63,12 +63,11 @@ impl Tool for GetConfigTool {
.or_else(|| self.state.cursor_registry.get("default"))
.map(|s| s.config.enabled)
.unwrap_or(true);
- // Agent View values aren't in DriverConfig — they're file-only since the
+ // PiP values aren't in DriverConfig — they're file-only since the
// backend is initialised once at startup. Read fresh so the
// response reflects whatever set_config (or a direct JSON edit)
// last wrote.
- let (agent_view_enabled, agent_view_geometry) =
- pip_preview::read_agent_view_keys_from_file();
+ let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
ToolResult::text("cua-driver-rs configuration").with_structured(serde_json::json!({
"version": env!("CARGO_PKG_VERSION"),
// Maintainer E2E builds set this at compile time so the
@@ -83,8 +82,8 @@ impl Tool for GetConfigTool {
"agent_cursor": {
"enabled": cursor_enabled,
},
- "agent_view": agent_view_enabled,
- "agent_view_geometry": agent_view_geometry,
+ "experimental_pip": pip_enabled,
+ "experimental_pip_geometry": pip_geometry,
}))
}
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs
index 31516025a0..dbf3d225f3 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_config.rs
@@ -25,8 +25,8 @@ fn def() -> &'static ToolDef {
name: "set_config".into(),
description: "Update cua-driver-rs configuration. Changes to \
max_image_dimension take effect immediately. The \
- agent_view keys are persisted to ~/.cua-driver/config.json and \
- take effect on the next daemon restart (Agent View is \
+ experimental_pip keys are persisted to ~/.cua-driver/config.json and \
+ take effect on the next daemon restart (the PiP backend is \
initialised once at startup).\n\nNote: capture_mode is a per-call \
param (on get_window_state / click), not a stored setting. Capture \
modality is selected by each action's target; the old \
@@ -47,16 +47,15 @@ fn def() -> &'static ToolDef {
"type": "integer",
"description": "Max dimension for screenshot resizing (0 = no limit)."
},
- "agent_view": {
+ "experimental_pip": {
"type": "boolean",
- "description": "Enable the multi-target Agent View. Exact native \
- windows and Chrome tabs are grouped automatically by session; no target \
- claiming is involved. Applies on next daemon restart."
+ "description": "Enable the experimental picture-in-picture preview window. \
+ Applies on next daemon restart."
},
- "agent_view_geometry": {
+ "experimental_pip_geometry": {
"type": "string",
- "description": "Agent View size + optional position in `WxH` or `WxH+X+Y` \
- form (e.g. `640x420+24+24`). Applies on next daemon restart."
+ "description": "PiP window size + optional position in `WxH` or `WxH+X+Y` \
+ form (e.g. `320x200+24+24`). Applies on next daemon restart."
}
},
"additionalProperties": false
@@ -105,17 +104,6 @@ impl Tool for SetConfigTool {
.filter(|(k, _)| k == name)
.and_then(|(_, v)| v.as_u64())
};
- let kv_bool = |name: &str| -> Option<bool> {
- kv.as_ref()
- .filter(|(k, _)| k == name)
- .and_then(|(_, v)| v.as_bool())
- };
- let kv_str = |name: &str| -> Option<String> {
- kv.as_ref()
- .filter(|(k, _)| k == name)
- .and_then(|(_, v)| v.as_str())
- .map(ToOwned::to_owned)
- };
// Validate max_image_dimension up front so both branches share the
// u32 check and we never half-apply.
@@ -157,38 +145,36 @@ impl Tool for SetConfigTool {
}
cfg.max_image_dimension
};
- // Agent View keys persist to the same config.json but take effect only on
+ // PiP keys persist to the same config.json but take effect only on
// next daemon restart — the backend is initialised once at startup.
- let mut agent_view_note = String::new();
- if let Some(enabled) = args
- .get("agent_view")
- .and_then(|v| v.as_bool())
- .or_else(|| kv_bool("agent_view"))
- {
- if let Err(e) = pip_preview::write_config_key("agent_view", Value::Bool(enabled)) {
- return ToolResult::error(format!("failed to persist agent_view: {e}"));
+ let mut pip_note = String::new();
+ if let Some(enabled) = args.get("experimental_pip").and_then(|v| v.as_bool()) {
+ if let Err(e) = pip_preview::write_config_key("experimental_pip", Value::Bool(enabled))
+ {
+ return ToolResult::error(format!("failed to persist experimental_pip: {e}"));
}
- agent_view_note =
- format!(" — restart cua-driver for agent_view={enabled} to take effect");
+ pip_note =
+ format!(" — restart cua-driver for experimental_pip={enabled} to take effect");
}
- if let Some(geom) = args
- .opt_str("agent_view_geometry")
- .or_else(|| kv_str("agent_view_geometry"))
- {
+ if let Some(geom) = args.opt_str("experimental_pip_geometry") {
// Validate before persisting so the user gets an immediate error.
if pip_preview::PipGeometry::parse(&geom).is_none() {
return ToolResult::error(format!(
- "agent_view_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
+ "experimental_pip_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
));
}
- if let Err(e) =
- pip_preview::write_config_key("agent_view_geometry", Value::String(geom.clone()))
- {
- return ToolResult::error(format!("failed to persist agent_view_geometry: {e}"));
+ if let Err(e) = pip_preview::write_config_key(
+ "experimental_pip_geometry",
+ Value::String(geom.clone()),
+ ) {
+ return ToolResult::error(format!(
+ "failed to persist experimental_pip_geometry: {e}"
+ ));
}
- if agent_view_note.is_empty() {
- agent_view_note =
- format!(" — restart cua-driver for agent_view_geometry={geom} to take effect");
+ if pip_note.is_empty() {
+ pip_note = format!(
+ " — restart cua-driver for experimental_pip_geometry={geom} to take effect"
+ );
}
}
let scope_note = if session_id.is_some() {
@@ -198,7 +184,7 @@ impl Tool for SetConfigTool {
};
ToolResult::text(format!(
"Config updated: max_image_dimension={}{}{}",
- effective_dim, scope_note, agent_view_note
+ effective_dim, scope_note, pip_note
))
.with_structured(serde_json::json!({
"version": env!("CARGO_PKG_VERSION"),
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/pip/mod.rs b/libs/cua-driver/rust/crates/platform-windows/src/pip/mod.rs
index 58e001ce0f..6f8666fced 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/pip/mod.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/pip/mod.rs
@@ -1,1495 +1,24 @@
-//! Windows multi-target Agent View.
+//! Windows picture-in-picture preview stub.
//!
-//! The preview owns a Win32 message-loop thread and never activates, so it can
-//! remain visible without stealing focus from the application being automated.
+//! The Win32 implementation is a follow-up to the experimental macOS
+//! drop. The plan is a layered, no-activate `WS_EX_NOACTIVATE |
+//! WS_EX_TOPMOST` HWND with a child `STATIC` showing the bitmap of
+//! the latest screenshot, fed via `SetWindowPos` for the never-key
+//! behavior the spec requires. Tracking issue is linked in the docs.
+//!
+//! Until that lands, `start()` returns a clear error so `main.rs`
+//! can log "PiP unavailable on this platform" and continue without
+//! the window.
-#[cfg(not(target_os = "windows"))]
use pip_preview::{PipBackend, PipBackendFactory, PipConfig};
-#[cfg(not(target_os = "windows"))]
-impl PipBackendFactory for WindowsPipBackendFactory {
- fn start(&self, _cfg: &PipConfig) -> anyhow::Result<Box<dyn PipBackend>> {
- Err(anyhow::anyhow!("Windows Agent View requires Windows"))
- }
-}
-
pub struct WindowsPipBackendFactory;
-#[cfg(target_os = "windows")]
-mod native {
- use std::sync::atomic::{AtomicBool, AtomicIsize, Ordering};
- use std::sync::{mpsc, Arc, Mutex};
-
- use cursor_overlay::{rasterize_inter_text, TextRaster};
- use image::imageops::FilterType;
- use pip_preview::{
- layout_desktop_with_shell, layout_session_tabs, png_dimensions, LayoutRect, PipBackend,
- PipBackendFactory, PipConfig, PipFrame, PipTargetKind, PipViewModel, PipWorkspaceSummary,
- ShellStyle, TargetSize,
- };
- use windows::core::PCWSTR;
- use windows::Win32::Foundation::{HWND, LPARAM, LRESULT, POINT, RECT, WPARAM};
- use windows::Win32::Graphics::Dwm::{
- DwmSetWindowAttribute, DWMWA_SYSTEMBACKDROP_TYPE, DWMWA_WINDOW_CORNER_PREFERENCE,
- };
- use windows::Win32::Graphics::Gdi::{
- BeginPaint, EndPaint, InvalidateRect, ScreenToClient, StretchDIBits, UpdateWindow,
- BITMAPINFO, BITMAPINFOHEADER, BI_RGB, DIB_RGB_COLORS, PAINTSTRUCT, SRCCOPY,
- };
- use windows::Win32::System::LibraryLoader::GetModuleHandleW;
- use windows::Win32::UI::WindowsAndMessaging::{
- CreateWindowExW, DefWindowProcW, DestroyWindow, DispatchMessageW, GetClientRect,
- GetMessageW, GetSystemMetrics, GetWindowLongPtrW, GetWindowRect, LoadCursorW, PostMessageW,
- PostQuitMessage, RegisterClassExW, SendMessageW, SetWindowLongPtrW, ShowWindow,
- TranslateMessage, CREATESTRUCTW, CS_HREDRAW, CS_VREDRAW, GWLP_USERDATA, GWL_EXSTYLE,
- HTBOTTOM, HTBOTTOMLEFT, HTBOTTOMRIGHT, HTCAPTION, HTCLIENT, HTLEFT, HTRIGHT, HTTOP,
- HTTOPLEFT, HTTOPRIGHT, HTTRANSPARENT, IDC_ARROW, MA_NOACTIVATE, MSG, SM_CXSCREEN,
- SM_CYSCREEN, SW_SHOWNOACTIVATE, WM_APP, WM_CLOSE, WM_DESTROY, WM_ERASEBKGND, WM_LBUTTONUP,
- WM_MOUSEACTIVATE, WM_NCCREATE, WM_NCDESTROY, WM_NCHITTEST, WM_PAINT, WM_SIZE, WNDCLASSEXW,
- WS_CLIPCHILDREN, WS_EX_NOACTIVATE, WS_EX_TOOLWINDOW, WS_EX_TOPMOST, WS_EX_TRANSPARENT,
- WS_POPUP, WS_THICKFRAME,
- };
-
- use super::WindowsPipBackendFactory;
-
- const REDRAW: u32 = WM_APP + 1;
- const SHUTDOWN: u32 = WM_APP + 2;
- const SET_INPUT_PASSTHROUGH: u32 = WM_APP + 3;
-
- struct State {
- hwnd: AtomicIsize,
- input_passthrough: AtomicBool,
- model: Mutex<PipViewModel>,
- }
-
- impl State {
- fn post(&self, message: u32) {
- let hwnd = self.hwnd.load(Ordering::Acquire);
- if hwnd != 0 {
- let _ =
- unsafe { PostMessageW(HWND(hwnd as *mut _), message, WPARAM(0), LPARAM(0)) };
- }
- }
-
- fn set_input_passthrough(&self, passthrough: bool) -> anyhow::Result<()> {
- let hwnd = self.hwnd.load(Ordering::Acquire);
- if hwnd == 0 {
- return Ok(());
- }
- let applied = unsafe {
- SendMessageW(
- HWND(hwnd as *mut _),
- SET_INPUT_PASSTHROUGH,
- WPARAM(usize::from(passthrough)),
- LPARAM(0),
- )
- };
- anyhow::ensure!(
- applied.0 == 1,
- "Agent View rejected the synchronous input-passthrough update"
- );
- Ok(())
- }
- }
-
- struct WindowsPipBackend {
- state: Arc<State>,
- }
-
- impl PipBackend for WindowsPipBackend {
- fn push_frame(&self, frame: PipFrame) {
- self.state.model.lock().unwrap().upsert(frame);
- self.state.post(REDRAW);
- }
-
- fn remove_workspace(&self, workspace_id: &str) {
- self.state
- .model
- .lock()
- .unwrap()
- .remove_workspace(workspace_id);
- self.state.post(REDRAW);
- }
-
- fn remove_target(&self, workspace_id: &str, identity_key: &str) {
- self.state
- .model
- .lock()
- .unwrap()
- .remove_target(workspace_id, identity_key);
- self.state.post(REDRAW);
- }
-
- fn set_input_passthrough(&self, passthrough: bool) -> anyhow::Result<()> {
- self.state.set_input_passthrough(passthrough)
- }
-
- fn shutdown(self: Box<Self>) {
- self.state.post(SHUTDOWN);
- }
- }
-
- impl PipBackendFactory for WindowsPipBackendFactory {
- fn start(&self, cfg: &PipConfig) -> anyhow::Result<Box<dyn PipBackend>> {
- let state = Arc::new(State {
- hwnd: AtomicIsize::new(0),
- input_passthrough: AtomicBool::new(false),
- model: Mutex::new(PipViewModel::new(12)),
- });
- let thread_state = Arc::clone(&state);
- let cfg = cfg.clone();
- let (ready_tx, ready_rx) = mpsc::sync_channel(1);
- std::thread::Builder::new()
- .name("cua-agent-view-windows".to_owned())
- .spawn(move || run_window(cfg, thread_state, ready_tx))?;
- ready_rx
- .recv()
- .map_err(|_| anyhow::anyhow!("Agent View UI thread exited during startup"))??;
- Ok(Box::new(WindowsPipBackend { state }))
- }
- }
-
- fn wide(value: &str) -> Vec<u16> {
- value.encode_utf16().chain(std::iter::once(0)).collect()
- }
-
- fn run_window(cfg: PipConfig, state: Arc<State>, ready: mpsc::SyncSender<anyhow::Result<()>>) {
- let hwnd = match unsafe { create_window(&cfg, &state) } {
- Ok(hwnd) => hwnd,
- Err(error) => {
- let _ = ready.send(Err(error));
- return;
- }
- };
- state.hwnd.store(hwnd.0 as isize, Ordering::Release);
- let _ = ready.send(Ok(()));
- unsafe {
- let _ = ShowWindow(hwnd, SW_SHOWNOACTIVATE);
- let _ = UpdateWindow(hwnd);
- let mut message = MSG::default();
- while GetMessageW(&mut message, None, 0, 0).as_bool() {
- let _ = TranslateMessage(&message);
- DispatchMessageW(&message);
- }
- }
- state.hwnd.store(0, Ordering::Release);
- }
-
- unsafe fn create_window(cfg: &PipConfig, state: &Arc<State>) -> anyhow::Result<HWND> {
- let instance = GetModuleHandleW(None)?;
- let class_name = wide("CuaAgentViewWindow");
- let class = WNDCLASSEXW {
- cbSize: std::mem::size_of::<WNDCLASSEXW>() as u32,
- style: CS_HREDRAW | CS_VREDRAW,
- lpfnWndProc: Some(window_proc),
- hInstance: instance.into(),
- hCursor: LoadCursorW(None, IDC_ARROW)?,
- lpszClassName: PCWSTR(class_name.as_ptr()),
- ..Default::default()
- };
- if RegisterClassExW(&class) == 0 {
- let error = windows::core::Error::from_win32();
- if error.code().0 != 0x8007_0582u32 as i32 {
- return Err(error.into());
- }
- }
-
- let width = cfg.geometry.width.max(320) as i32;
- let height = cfg.geometry.height.max(240) as i32;
- let x = cfg
- .geometry
- .x
- .unwrap_or_else(|| (GetSystemMetrics(SM_CXSCREEN) - width - 24).max(0));
- let y = cfg
- .geometry
- .y
- .unwrap_or(24)
- .min((GetSystemMetrics(SM_CYSCREEN) - height).max(0));
- let title = wide(&cfg.title);
- let state_ptr = Box::into_raw(Box::new(Arc::clone(state)));
- let hwnd = match CreateWindowExW(
- WS_EX_TOPMOST | WS_EX_NOACTIVATE | WS_EX_TOOLWINDOW,
- PCWSTR(class_name.as_ptr()),
- PCWSTR(title.as_ptr()),
- WS_POPUP | WS_THICKFRAME | WS_CLIPCHILDREN,
- x,
- y,
- width,
- height,
- None,
- None,
- instance,
- Some(state_ptr.cast()),
- ) {
- Ok(hwnd) => hwnd,
- Err(error) => {
- drop(Box::from_raw(state_ptr));
- return Err(error.into());
- }
- };
- let corner = 2i32; // DWMWCP_ROUND
- let _ = DwmSetWindowAttribute(
- hwnd,
- DWMWA_WINDOW_CORNER_PREFERENCE,
- (&corner as *const i32).cast(),
- std::mem::size_of_val(&corner) as u32,
- );
- let backdrop = 3i32; // DWMSBT_TRANSIENTWINDOW
- let _ = DwmSetWindowAttribute(
- hwnd,
- DWMWA_SYSTEMBACKDROP_TYPE,
- (&backdrop as *const i32).cast(),
- std::mem::size_of_val(&backdrop) as u32,
- );
- Ok(hwnd)
- }
-
- unsafe extern "system" fn window_proc(
- hwnd: HWND,
- message: u32,
- wparam: WPARAM,
- lparam: LPARAM,
- ) -> LRESULT {
- if message == WM_NCCREATE {
- let create = &*(lparam.0 as *const CREATESTRUCTW);
- SetWindowLongPtrW(hwnd, GWLP_USERDATA, create.lpCreateParams as isize);
- return LRESULT(1);
- }
- let state_ptr = GetWindowLongPtrW(hwnd, GWLP_USERDATA) as *mut Arc<State>;
- match message {
- REDRAW | WM_SIZE => {
- let _ = InvalidateRect(hwnd, None, false);
- LRESULT(0)
- }
- WM_PAINT => {
- if !state_ptr.is_null() {
- paint(hwnd, &*state_ptr);
- }
- LRESULT(0)
- }
- WM_ERASEBKGND => LRESULT(1),
- WM_MOUSEACTIVATE => LRESULT(MA_NOACTIVATE as isize),
- SET_INPUT_PASSTHROUGH => {
- let passthrough = wparam.0 != 0;
- if !state_ptr.is_null() {
- (&*state_ptr)
- .input_passthrough
- .store(passthrough, Ordering::Release);
- }
- let style = GetWindowLongPtrW(hwnd, GWL_EXSTYLE);
- SetWindowLongPtrW(
- hwnd,
- GWL_EXSTYLE,
- extended_style_with_input_passthrough(style, passthrough),
- );
- LRESULT(1)
- }
- WM_LBUTTONUP => {
- if !state_ptr.is_null() {
- let x = lparam.0 as i16 as i32;
- let y = (lparam.0 >> 16) as i16 as i32;
- let mut client = RECT::default();
- if GetClientRect(hwnd, &mut client).is_ok() {
- let state = &*state_ptr;
- let mut model = state.model.lock().unwrap();
- let workspaces = model.workspaces();
- let tabs = session_tabs_for_width(
- client.right - client.left,
- client.bottom - client.top,
- &workspaces,
- model.selected_workspace_id(),
- );
- let changed = tabs
- .hit_test(f64::from(x), f64::from(y))
- .map(|workspace_id| model.select_workspace(workspace_id))
- .unwrap_or(false);
- if changed {
- let _ = InvalidateRect(hwnd, None, false);
- }
- }
- }
- LRESULT(0)
- }
- WM_NCHITTEST => {
- if !state_ptr.is_null() && (&*state_ptr).input_passthrough.load(Ordering::Acquire) {
- return LRESULT(HTTRANSPARENT as isize);
- }
- let hit = DefWindowProcW(hwnd, message, wparam, lparam);
- if hit.0 == HTCLIENT as isize {
- let screen_x = lparam.0 as i16 as i32;
- let screen_y = (lparam.0 >> 16) as i16 as i32;
- let mut window = RECT::default();
- if GetWindowRect(hwnd, &mut window).is_ok() {
- let border = 8;
- let left = screen_x - window.left < border;
- let right = window.right - screen_x <= border;
- let top = screen_y - window.top < border;
- let bottom = window.bottom - screen_y <= border;
- let resize_hit = match (left, right, top, bottom) {
- (true, _, true, _) => Some(HTTOPLEFT),
- (_, true, true, _) => Some(HTTOPRIGHT),
- (true, _, _, true) => Some(HTBOTTOMLEFT),
- (_, true, _, true) => Some(HTBOTTOMRIGHT),
- (true, _, _, _) => Some(HTLEFT),
- (_, true, _, _) => Some(HTRIGHT),
- (_, _, true, _) => Some(HTTOP),
- (_, _, _, true) => Some(HTBOTTOM),
- _ => None,
- };
- if let Some(resize_hit) = resize_hit {
- return LRESULT(resize_hit as isize);
- }
- let mut client_point = POINT {
- x: screen_x,
- y: screen_y,
- };
- let mut client = RECT::default();
- let tabs_hit = if !state_ptr.is_null()
- && ScreenToClient(hwnd, &mut client_point).as_bool()
- && GetClientRect(hwnd, &mut client).is_ok()
- {
- let model = (&*state_ptr).model.lock().unwrap();
- let workspaces = model.workspaces();
- session_tabs_for_width(
- client.right - client.left,
- client.bottom - client.top,
- &workspaces,
- model.selected_workspace_id(),
- )
- .hit_test(f64::from(client_point.x), f64::from(client_point.y))
- .is_some()
- } else {
- false
- };
- if tabs_hit {
- return LRESULT(HTCLIENT as isize);
- }
- if screen_y - window.top < 28 {
- return LRESULT(HTCAPTION as isize);
- }
- }
- }
- hit
- }
- SHUTDOWN | WM_CLOSE => {
- let _ = DestroyWindow(hwnd);
- LRESULT(0)
- }
- WM_DESTROY => {
- PostQuitMessage(0);
- LRESULT(0)
- }
- WM_NCDESTROY => {
- SetWindowLongPtrW(hwnd, GWLP_USERDATA, 0);
- if !state_ptr.is_null() {
- drop(Box::from_raw(state_ptr));
- }
- DefWindowProcW(hwnd, message, wparam, lparam)
- }
- _ => DefWindowProcW(hwnd, message, wparam, lparam),
- }
- }
-
- unsafe fn paint(hwnd: HWND, state: &Arc<State>) {
- let mut ps = PAINTSTRUCT::default();
- let hdc = BeginPaint(hwnd, &mut ps);
- let mut rect = RECT::default();
- if GetClientRect(hwnd, &mut rect).is_err() {
- let _ = EndPaint(hwnd, &ps);
- return;
- }
- let width = (rect.right - rect.left).max(1) as u32;
- let height = (rect.bottom - rect.top).max(1) as u32;
- let (frames, workspaces, selected_workspace_id, active_view_id) = {
- let model = state.model.lock().unwrap();
- (
- model
- .selected_frames()
- .into_iter()
- .cloned()
- .collect::<Vec<_>>(),
- model.workspaces(),
- model.selected_workspace_id().map(str::to_owned),
- model.active_view_id().map(str::to_owned),
- )
- };
- let pixels = render_view(
- width,
- height,
- &frames,
- &workspaces,
- selected_workspace_id.as_deref(),
- active_view_id.as_deref(),
- );
- let info = BITMAPINFO {
- bmiHeader: BITMAPINFOHEADER {
- biSize: std::mem::size_of::<BITMAPINFOHEADER>() as u32,
- biWidth: width as i32,
- biHeight: -(height as i32),
- biPlanes: 1,
- biBitCount: 32,
- biCompression: BI_RGB.0,
- biSizeImage: pixels.len() as u32,
- ..Default::default()
- },
- ..Default::default()
- };
- StretchDIBits(
- hdc,
- 0,
- 0,
- width as i32,
- height as i32,
- 0,
- 0,
- width as i32,
- height as i32,
- Some(pixels.as_ptr().cast()),
- &info,
- DIB_RGB_COLORS,
- SRCCOPY,
- );
- let _ = EndPaint(hwnd, &ps);
- }
-
- fn render_view(
- width: u32,
- height: u32,
- frames: &[PipFrame],
- workspaces: &[PipWorkspaceSummary],
- selected_workspace_id: Option<&str>,
- active_view_id: Option<&str>,
- ) -> Vec<u8> {
- let mut canvas = Canvas::new(width, height);
- canvas.smoked_shell();
- let show_switcher = workspaces.len() > 1;
- let desktop = desktop_rect(width, height, show_switcher);
- canvas.wallpaper(desktop);
- let sizes = frames
- .iter()
- .map(|frame| {
- png_dimensions(&frame.png_bytes).unwrap_or(TargetSize {
- width: 16,
- height: 10,
- })
- })
- .collect::<Vec<_>>();
- let layout =
- layout_desktop_with_shell(desktop.2 as f64, desktop.3 as f64, &sizes, ShellStyle::None);
- canvas.border((0, 0, width as i32, height as i32), 14, [19, 12, 7, 225]);
- canvas.border(
- (1, 1, width as i32 - 2, height as i32 - 2),
- 13,
- [211, 184, 151, 118],
- );
- canvas.border(
- (2, 2, width as i32 - 4, height as i32 - 4),
- 12,
- [108, 82, 57, 105],
- );
- canvas.border(desktop, 10, [181, 151, 122, 138]);
- if show_switcher {
- canvas.switcher(
- switcher_rect(width as i32),
- workspaces,
- selected_workspace_id,
- );
- }
- for (frame, target) in frames.iter().zip(&layout.targets) {
- let rect = offset_pixels(target.content, desktop.0, desktop.1);
- canvas.shadow(rect);
- canvas.fill(rect, 7, [244, 247, 251, 255]);
- if let Ok(image) = image::load_from_memory(&frame.png_bytes) {
- let image = image
- .resize_exact(rect.2 as u32, rect.3 as u32, FilterType::Lanczos3)
- .to_rgba8();
- canvas.blit(&image, rect, 7);
- }
- canvas.border(rect, 7, [40, 52, 72, 76]);
- if active_view_id == Some(frame.target.view_id().as_str()) {
- canvas.border(
- (rect.0 - 3, rect.1 - 3, rect.2 + 6, rect.3 + 6),
- 9,
- [230, 158, 73, 232],
- );
- }
- }
- canvas.data
- }
-
- /// Index of the card whose capture is newest, which the taskbar marks as
- /// the foreground app.
- ///
- /// Ties resolve to the earliest card so one repaint never reorders the
- /// highlight for an unchanged set of frames.
- fn active_target_index(frames: &[PipFrame]) -> Option<usize> {
- frames
- .iter()
- .enumerate()
- .max_by_key(|(index, frame)| (frame.timestamp_ms, std::cmp::Reverse(*index)))
- .map(|(index, _)| index)
- }
-
- /// `HH:MM` for one epoch-millisecond capture stamp, in UTC.
- ///
- /// The clock reads the newest card's capture time rather than the wall
- /// clock so a rendered frame stays reproducible from its inputs alone.
- fn clock_label(timestamp_ms: u64) -> String {
- let minutes = timestamp_ms / 60_000;
- format!("{:02}:{:02}", minutes / 60 % 24, minutes % 60)
- }
-
- /// Separable box blur over a tightly packed 3-channel buffer.
- fn box_blur(source: &[u8], width: usize, height: usize, radius: usize) -> Vec<u8> {
- let mut pass = source.to_vec();
- let mut out = vec![0u8; source.len()];
- for vertical in [false, true] {
- let (major, minor) = if vertical {
- (width, height)
- } else {
- (height, width)
- };
- for outer in 0..major {
- for inner in 0..minor {
- let low = inner.saturating_sub(radius);
- let high = (inner + radius).min(minor - 1);
- let taps = (high - low + 1) as u32;
- for channel in 0..3 {
- let mut sum = 0u32;
- for tap in low..=high {
- let at = if vertical {
- tap * width + outer
- } else {
- outer * width + tap
- };
- sum += pass[at * 3 + channel] as u32;
- }
- let at = if vertical {
- inner * width + outer
- } else {
- outer * width + inner
- };
- out[at * 3 + channel] = (sum / taps) as u8;
- }
- }
- }
- std::mem::swap(&mut pass, &mut out);
- }
- pass
- }
-
- fn desktop_rect(width: u32, height: u32, show_switcher: bool) -> (i32, i32, i32, i32) {
- const SIDE_INSET: i32 = 8;
- const BOTTOM_INSET: i32 = 9;
- let top_inset = if show_switcher { 36 } else { 14 };
- (
- SIDE_INSET,
- top_inset,
- (width as i32 - SIDE_INSET * 2).max(1),
- (height as i32 - top_inset - BOTTOM_INSET).max(1),
- )
- }
-
- fn switcher_rect(width: i32) -> (i32, i32, i32, i32) {
- (12, 8, (width - 24).max(1), 28)
- }
-
- fn session_tabs_for_width(
- width: i32,
- height: i32,
- workspaces: &[PipWorkspaceSummary],
- selected_workspace_id: Option<&str>,
- ) -> pip_preview::SessionTabsLayout {
- layout_session_tabs(
- LayoutRect {
- x: 12.0,
- y: 8.0,
- width: f64::from((width - 24).max(1)),
- height: f64::from(height.max(1)),
- },
- workspaces,
- selected_workspace_id,
- )
- }
-
- fn contains(rect: (i32, i32, i32, i32), x: i32, y: i32) -> bool {
- x >= rect.0 && y >= rect.1 && x < rect.0 + rect.2 && y < rect.1 + rect.3
- }
-
- fn extended_style_with_input_passthrough(style: isize, passthrough: bool) -> isize {
- let transparent = WS_EX_TRANSPARENT.0 as isize;
- if passthrough {
- style | transparent
- } else {
- style & !transparent
- }
- }
-
- fn pixels(rect: LayoutRect) -> (i32, i32, i32, i32) {
- (
- rect.x.round() as i32,
- rect.y.round() as i32,
- rect.width.round().max(1.0) as i32,
- rect.height.round().max(1.0) as i32,
- )
- }
-
- fn offset_pixels(rect: LayoutRect, x: i32, y: i32) -> (i32, i32, i32, i32) {
- let rect = pixels(rect);
- (rect.0 + x, rect.1 + y, rect.2, rect.3)
- }
-
- struct Canvas {
- width: u32,
- height: u32,
- data: Vec<u8>,
- }
-
- impl Canvas {
- fn new(width: u32, height: u32) -> Self {
- Self {
- width,
- height,
- data: vec![0; width as usize * height as usize * 4],
- }
- }
-
- fn wallpaper(&mut self, rect: (i32, i32, i32, i32)) {
- let w = rect.2.max(1) as f32;
- let h = rect.3.max(1) as f32;
- for y in 0..rect.3 {
- for x in 0..rect.2 {
- if !Self::inside(x, y, rect.2, rect.3, 10) {
- continue;
- }
- let nx = x as f32 / w;
- let ny = y as f32 / h;
- let upper_glow =
- (1.0 - (((nx - 0.72).powi(2) + (ny - 0.08).powi(2)).sqrt() * 1.5)).max(0.0);
- let lower_glow =
- (1.0 - (((nx - 0.18).powi(2) + (ny - 0.88).powi(2)).sqrt() * 1.8)).max(0.0);
- let ribbon =
- (1.0 - ((ny - 0.64 + 0.08 * (nx * 5.4).sin()).abs() * 6.5)).max(0.0);
- self.set(
- rect.0 + x,
- rect.1 + y,
- [
- (35.0 + 45.0 * upper_glow + 25.0 * lower_glow + 19.0 * ribbon) as u8,
- (25.0 + 36.0 * upper_glow + 17.0 * lower_glow + 11.0 * ribbon) as u8,
- (18.0 + 22.0 * upper_glow + 10.0 * lower_glow + 5.0 * ribbon) as u8,
- 255,
- ],
- );
- }
- }
- }
-
- fn smoked_shell(&mut self) {
- let width = self.width as i32;
- let height = self.height as i32;
- self.fill((0, 0, width, height), 0, [15, 10, 7, 255]);
- self.fill((0, 0, width, height), 14, [24, 16, 10, 255]);
-
- let header_height = (height / 12).clamp(24, 42);
- self.fill(
- (2, 2, width.saturating_sub(4), header_height),
- 12,
- [66, 50, 36, 92],
- );
- self.fill((16, 3, width.saturating_sub(32), 1), 0, [235, 211, 181, 92]);
-
- // Fine, low-contrast grain keeps the painted fallback from looking
- // flat when the system backdrop is unavailable (for example RDP).
- for y in (8..height.saturating_sub(8)).step_by(4) {
- for x in ((y & 7)..width.saturating_sub(8)).step_by(8) {
- self.blend(x, y, [205, 181, 155, 7]);
- }
- }
- }
-
- fn switcher(
- &mut self,
- rect: (i32, i32, i32, i32),
- workspaces: &[PipWorkspaceSummary],
- selected_workspace_id: Option<&str>,
- ) {
- let tabs = layout_session_tabs(
- LayoutRect {
- x: f64::from(rect.0),
- y: f64::from(rect.1),
- width: f64::from(rect.2),
- height: f64::from(rect.3),
- },
- workspaces,
- selected_workspace_id,
- );
- for tab in tabs.tabs {
- let tab_rect = offset_pixels(tab.rect, 0, 0);
- self.shadow(tab_rect);
- self.fill(
- tab_rect,
- 9,
- [42, 31, 24, if tab.selected { 242 } else { 204 }],
- );
- self.border(
- tab_rect,
- 9,
- [
- tab.accent.0,
- tab.accent.1,
- tab.accent.2,
- if tab.selected { 235 } else { 128 },
- ],
- );
- }
- }
-
- fn set(&mut self, x: i32, y: i32, color: [u8; 4]) {
- if x < 0 || y < 0 || x >= self.width as i32 || y >= self.height as i32 {
- return;
- }
- let at = (y as usize * self.width as usize + x as usize) * 4;
- self.data[at..at + 4].copy_from_slice(&color);
- }
-
- fn blend(&mut self, x: i32, y: i32, color: [u8; 4]) {
- if x < 0 || y < 0 || x >= self.width as i32 || y >= self.height as i32 {
- return;
- }
- let at = (y as usize * self.width as usize + x as usize) * 4;
- let alpha = color[3] as u16;
- for channel in 0..3 {
- self.data[at + channel] = ((color[channel] as u16 * alpha
- + self.data[at + channel] as u16 * (255 - alpha))
- / 255) as u8;
- }
- self.data[at + 3] = 255;
- }
-
- fn inside(x: i32, y: i32, width: i32, height: i32, radius: i32) -> bool {
- let dx = if x < radius {
- radius - x
- } else if x >= width - radius {
- x - width + radius + 1
- } else {
- 0
- };
- let dy = if y < radius {
- radius - y
- } else if y >= height - radius {
- y - height + radius + 1
- } else {
- 0
- };
- dx == 0 || dy == 0 || dx * dx + dy * dy <= radius * radius
- }
-
- fn fill(&mut self, rect: (i32, i32, i32, i32), radius: i32, color: [u8; 4]) {
- for y in 0..rect.3 {
- for x in 0..rect.2 {
- if Self::inside(x, y, rect.2, rect.3, radius) {
- self.blend(rect.0 + x, rect.1 + y, color);
- }
- }
- }
- }
-
- fn border(&mut self, rect: (i32, i32, i32, i32), radius: i32, color: [u8; 4]) {
- for y in 0..rect.3 {
- for x in 0..rect.2 {
- let outer = Self::inside(x, y, rect.2, rect.3, radius);
- let inner = x > 0
- && y > 0
- && x < rect.2 - 1
- && y < rect.3 - 1
- && Self::inside(x - 1, y - 1, rect.2 - 2, rect.3 - 2, radius - 1);
- if outer && !inner {
- self.blend(rect.0 + x, rect.1 + y, color);
- }
- }
- }
- }
-
- fn shadow(&mut self, rect: (i32, i32, i32, i32)) {
- for spread in (1..=8).rev() {
- self.border(
- (
- rect.0 - spread,
- rect.1 - spread + 2,
- rect.2 + spread * 2,
- rect.3 + spread * 2,
- ),
- 7 + spread,
- [8, 24, 49, (26 / spread) as u8],
- );
- }
- }
-
- fn blit(&mut self, image: &image::RgbaImage, rect: (i32, i32, i32, i32), radius: i32) {
- for y in 0..rect.3 {
- for x in 0..rect.2 {
- if Self::inside(x, y, rect.2, rect.3, radius) {
- let p = image.get_pixel(x as u32, y as u32).0;
- self.blend(rect.0 + x, rect.1 + y, [p[2], p[1], p[0], p[3]]);
- }
- }
- }
- }
-
- fn icon(&mut self, rect: (i32, i32, i32, i32), kind: PipTargetKind, index: usize) {
- let colors = match kind {
- PipTargetKind::BrowserTab => ([31, 180, 210, 255], [205, 91, 28, 255]),
- PipTargetKind::NativeWindow => [
- ([205, 91, 138, 255], [131, 53, 92, 255]),
- ([78, 166, 66, 255], [35, 104, 26, 255]),
- ([43, 145, 235, 255], [21, 84, 180, 255]),
- ][index % 3],
- };
- self.fill(rect, (rect.2 / 4).max(3), colors.0);
- let inset = (rect.2 / 4).max(3);
- self.fill(
- (
- rect.0 + inset,
- rect.1 + inset,
- rect.2 - inset * 2,
- rect.3 - inset * 2,
- ),
- 3,
- colors.1,
- );
- }
-
- /// Acrylic-like backdrop for the edge-anchored taskbar band.
- ///
- /// The band is a material, not a card: it blurs whatever Agent View
- /// already composited beneath it, tints the result, and adds fine
- /// noise, which is why it runs after the wallpaper and the target
- /// cards. Painting is clipped to the miniature desktop so the bar
- /// keeps the desktop's rounded bottom corners.
- fn taskbar_material(&mut self, rect: (i32, i32, i32, i32), clip: (i32, i32, i32, i32)) {
- const BLUR_RADIUS: usize = 5;
- /// Tint strength out of 255. Leaving some backdrop through is what
- /// separates acrylic from a flat fill.
- const TINT_ALPHA: u32 = 196;
- const TINT: [u32; 3] = [44, 34, 27];
-
- let width = rect.2.max(0) as usize;
- let height = rect.3.max(0) as usize;
- if width == 0 || height == 0 {
- return;
- }
- // Sample with clamped coordinates so the blur never smears the
- // surrounding shell frame into the band.
- let mut backdrop = vec![0u8; width * height * 3];
- for y in 0..height {
- for x in 0..width {
- let sx = (rect.0 + x as i32).clamp(0, self.width as i32 - 1);
- let sy = (rect.1 + y as i32).clamp(0, self.height as i32 - 1);
- let from = (sy as usize * self.width as usize + sx as usize) * 4;
- let to = (y * width + x) * 3;
- backdrop[to..to + 3].copy_from_slice(&self.data[from..from + 3]);
- }
- }
- let blurred = box_blur(&backdrop, width, height, BLUR_RADIUS);
-
- for y in 0..height {
- for x in 0..width {
- let px = rect.0 + x as i32;
- let py = rect.1 + y as i32;
- if !Self::inside(px - clip.0, py - clip.1, clip.2, clip.3, 10) {
- continue;
- }
- let sample = (y * width + x) * 3;
- let mut color = [0u8; 4];
- for channel in 0..3 {
- color[channel] = ((blurred[sample + channel] as u32 * (255 - TINT_ALPHA)
- + TINT[channel] * TINT_ALPHA)
- / 255) as u8;
- }
- color[3] = 255;
- self.set(px, py, color);
- // Deterministic grain, the visual signature of acrylic.
- if (x ^ y) & 3 == 0 {
- self.blend(px, py, [188, 170, 150, 6]);
- }
- }
- }
-
- // A single bright hairline along the top edge reads as the lit
- // lip of the bar and keeps it separated from the wallpaper.
- for x in 0..rect.2 {
- let px = rect.0 + x;
- if Self::inside(px - clip.0, rect.1 - clip.1, clip.2, clip.3, 10) {
- self.blend(px, rect.1, [214, 190, 162, 96]);
- }
- }
- }
-
- /// Four-pane launcher glyph anchored at the head of the icon cluster.
- fn start_glyph(&mut self, rect: (i32, i32, i32, i32)) {
- let gap = (rect.2 / 9).max(1);
- let pane_width = ((rect.2 - gap) / 2).max(1);
- let pane_height = ((rect.3 - gap) / 2).max(1);
- // Center the glyph inside its slot, which is sized like an app icon.
- let x = rect.0 + (rect.2 - (pane_width * 2 + gap)) / 2;
- let y = rect.1 + (rect.3 - (pane_height * 2 + gap)) / 2;
- for row in 0..2 {
- for column in 0..2 {
- self.fill(
- (
- x + column * (pane_width + gap),
- y + row * (pane_height + gap),
- pane_width,
- pane_height,
- ),
- 1,
- [230, 158, 73, 235],
- );
- }
- }
- }
-
- /// Running-app mark beneath one taskbar icon.
- ///
- /// The foreground card gets the full accent pill; the rest get a
- /// shorter, dimmer mark, matching how a Windows taskbar distinguishes
- /// the active window from other running ones.
- fn running_indicator(&mut self, rect: (i32, i32, i32, i32), active: bool) {
- let (width, color) = if active {
- (rect.2, [230, 158, 73, 255])
- } else {
- ((rect.2 / 2).max(2), [176, 158, 138, 190])
- };
- self.fill(
- (rect.0 + (rect.2 - width) / 2, rect.1, width, rect.3),
- rect.3 / 2,
- color,
- );
- }
-
- /// Trailing status band: a chevron, a battery mark, and the clock.
- ///
- /// Every element is dropped rather than crowded when the band is too
- /// narrow, so a small Agent View degrades instead of overlapping the
- /// centered icon cluster.
- fn tray(&mut self, rect: (i32, i32, i32, i32), timestamp_ms: Option<u64>) {
- const INK: [u8; 4] = [224, 210, 190, 235];
- let padding = 8;
- let mut right = rect.0 + rect.2 - padding;
-
- let font_size = (rect.3 as f32 * 0.2).clamp(8.0, 13.0);
- let clock = timestamp_ms
- .map(clock_label)
- .and_then(|label| rasterize_inter_text(&label, font_size));
- if let Some(raster) = clock {
- if raster.width as i32 + padding * 2 <= rect.2 {
- right -= raster.width as i32;
- let y = rect.1 + (rect.3 - raster.height as i32) / 2;
- self.text(&raster, right, y, INK);
- }
- }
-
- let center_y = rect.1 + rect.3 / 2;
- // Battery: an outline pill with a nub, drawn only when it fits
- // clear of the clock.
- let battery_width = 14;
- if right - (battery_width + 10) >= rect.0 + padding {
- right -= battery_width + 10;
- self.border((right, center_y - 4, battery_width, 8), 2, INK);
- self.fill((right + battery_width, center_y - 2, 2, 4), 1, INK);
- }
- // Chevron: the "show hidden icons" affordance.
- if right - 12 >= rect.0 + padding {
- right -= 12;
- for step in 0..4 {
- self.blend(right + step, center_y + 1 - step, INK);
- self.blend(right + 6 - step, center_y + 1 - step, INK);
- }
- }
- }
-
- /// Composite one rasterized text mask in the buffer's channel order.
- fn text(&mut self, raster: &TextRaster, x: i32, y: i32, color: [u8; 4]) {
- for row in 0..raster.height {
- for column in 0..raster.width {
- let coverage = raster.coverage[(row * raster.width + column) as usize];
- if coverage == 0 {
- continue;
- }
- let alpha = (color[3] as u16 * coverage as u16 / 255) as u8;
- self.blend(
- x + column as i32,
- y + row as i32,
- [color[0], color[1], color[2], alpha],
- );
- }
- }
- }
- }
-
- #[cfg(test)]
- mod tests {
- use std::io::Cursor;
-
- use super::*;
- use image::{ImageFormat, Rgba, RgbaImage};
- use pip_preview::PipTarget;
-
- fn solid_png(width: u32, height: u32, color: [u8; 4]) -> Vec<u8> {
- let image = RgbaImage::from_pixel(width, height, Rgba(color));
- let mut bytes = Cursor::new(Vec::new());
- image.write_to(&mut bytes, ImageFormat::Png).unwrap();
- bytes.into_inner()
- }
-
- fn pixel(data: &[u8], width: u32, x: u32, y: u32) -> [u8; 4] {
- let offset = ((y * width + x) * 4) as usize;
- data[offset..offset + 4].try_into().unwrap()
- }
-
- fn frame(
- workspace: &str,
- id: &str,
- kind: PipTargetKind,
- png_bytes: Vec<u8>,
- timestamp_ms: u64,
- ) -> PipFrame {
- PipFrame {
- target: PipTarget {
- workspace_id: workspace.to_owned(),
- workspace_label: workspace.to_owned(),
- target_id: id.to_owned(),
- identity_key: id.to_owned(),
- target_kind: kind,
- target_label: id.to_owned(),
- native_container: None,
- },
- png_bytes,
- action_label: "click".to_owned(),
- timestamp_ms,
- cursor_position: None,
- }
- }
-
- fn workspace(id: &str, target_count: usize, updated_ms: u64) -> PipWorkspaceSummary {
- PipWorkspaceSummary {
- workspace_id: id.to_owned(),
- workspace_label: id.to_owned(),
- target_count,
- updated_ms,
- }
- }
-
- #[test]
- fn renderer_tracks_each_target_and_requested_size() {
- let workspaces = [workspace("workspace", 2, 1)];
- let data = render_view(
- 480,
- 320,
- &[
- frame(
- "workspace",
- "edge",
- PipTargetKind::BrowserTab,
- Vec::new(),
- 1,
- ),
- frame(
- "workspace",
- "notes",
- PipTargetKind::NativeWindow,
- Vec::new(),
- 1,
- ),
- ],
- &workspaces,
- Some("workspace"),
- None,
- );
- assert_eq!(data.len(), 480 * 320 * 4);
- assert!(data.chunks_exact(4).all(|pixel| pixel[3] == 255));
- }
-
- #[test]
- fn renderer_preserves_target_pixels_across_mixed_form_factors() {
- let frames = [
- frame(
- "workspace",
- "wide",
- PipTargetKind::BrowserTab,
- solid_png(80, 32, [220, 30, 20, 255]),
- 1,
- ),
- frame(
- "workspace",
- "tall",
- PipTargetKind::NativeWindow,
- solid_png(24, 72, [20, 90, 230, 255]),
- 1,
- ),
- ];
- let width = 520;
- let height = 360;
- let workspaces = [workspace("workspace", 2, 1)];
- let data = render_view(width, height, &frames, &workspaces, Some("workspace"), None);
- let desktop = desktop_rect(width, height, false);
- let layout = layout_desktop_with_shell(
- desktop.2 as f64,
- desktop.3 as f64,
- &[
- TargetSize {
- width: 80,
- height: 32,
- },
- TargetSize {
- width: 24,
- height: 72,
- },
- ],
- ShellStyle::EdgeTaskbar,
- );
-
- let wide = offset_pixels(layout.targets[0].content, desktop.0, desktop.1);
- let tall = offset_pixels(layout.targets[1].content, desktop.0, desktop.1);
- assert_eq!(
- pixel(
- &data,
- width,
- (wide.0 + wide.2 / 2) as u32,
- (wide.1 + wide.3 / 2) as u32,
- ),
- [20, 30, 220, 255]
- );
- assert_eq!(
- pixel(
- &data,
- width,
- (tall.0 + tall.2 / 2) as u32,
- (tall.1 + tall.3 / 2) as u32,
- ),
- [230, 90, 20, 255]
- );
- assert!(wide.2 > wide.3);
- assert!(tall.3 > tall.2);
- assert!(wide.0 >= desktop.0 && wide.1 >= desktop.1);
- assert!(tall.0 + tall.2 <= desktop.0 + desktop.2);
- assert!(tall.1 + tall.3 <= desktop.1 + desktop.3);
- }
-
- #[test]
- fn renderer_separates_dark_shell_from_inset_desktop() {
- let width = 360;
- let data = render_view(width, 240, &[], &[], None, None);
- let shell = pixel(&data, width, 4, 100);
- let desktop = pixel(&data, width, 180, 100);
- let highlight = pixel(&data, width, 180, 1);
- let brightness = |color: [u8; 4]| color[0] as u16 + color[1] as u16 + color[2] as u16;
-
- assert_ne!(shell, desktop);
- assert!(brightness(shell) < brightness(desktop));
- assert!(shell[0] > shell[1] && shell[1] > shell[2]);
- assert!(brightness(highlight) > brightness(shell));
- assert!(highlight[0] > highlight[1] && highlight[1] > highlight[2]);
- }
-
- #[test]
- fn switcher_is_hidden_for_one_workspace_and_cycles_without_target_activation() {
- let width = 420;
- let one = [workspace("agent-a", 1, 1)];
- let two = [workspace("agent-b", 1, 2), workspace("agent-a", 1, 1)];
- let single = render_view(width, 280, &[], &one, Some("agent-a"), None);
- let multiple = render_view(width, 280, &[], &two, Some("agent-a"), None);
- let switcher = switcher_rect(width as i32);
- let sample_x = (switcher.0 + switcher.2 / 2) as u32;
- let sample_y = (switcher.1 + switcher.3 / 2) as u32;
-
- assert_eq!(desktop_rect(width, 280, false).1, 14);
- assert_eq!(desktop_rect(width, 280, true).1, 36);
- assert_ne!(
- pixel(&single, width, sample_x, sample_y),
- pixel(&multiple, width, sample_x, sample_y)
- );
-
- let mut model = PipViewModel::new(4);
- model.upsert(frame(
- "agent-a",
- "window-a",
- PipTargetKind::NativeWindow,
- Vec::new(),
- 1,
- ));
- model.upsert(frame(
- "agent-b",
- "window-b",
- PipTargetKind::NativeWindow,
- Vec::new(),
- 2,
- ));
- assert!(model.select_workspace("agent-a"));
- assert_eq!(model.selected_frames()[0].target.workspace_id, "agent-a");
- assert!(model.select_next_workspace());
- assert_eq!(model.selected_frames()[0].target.workspace_id, "agent-b");
- }
-
- #[test]
- fn exact_target_removal_and_workspace_cleanup_redraw_the_selected_session() {
- let mut model = PipViewModel::new(6);
- model.upsert(frame(
- "agent-a",
- "wide",
- PipTargetKind::BrowserTab,
- Vec::new(),
- 1,
- ));
- model.upsert(frame(
- "agent-a",
- "tall",
- PipTargetKind::NativeWindow,
- Vec::new(),
- 2,
- ));
- model.upsert(frame(
- "agent-b",
- "other",
- PipTargetKind::NativeWindow,
- Vec::new(),
- 3,
- ));
- assert!(model.select_workspace("agent-a"));
- assert!(model.remove_target("agent-a", "wide"));
- assert_eq!(model.selected_frames().len(), 1);
- assert_eq!(model.selected_frames()[0].target.identity_key, "tall");
- assert_eq!(model.remove_workspace("agent-a").len(), 1);
- assert_eq!(model.selected_workspace_id(), Some("agent-b"));
- }
-
- fn taskbar_layout(width: u32, height: u32, targets: usize) -> pip_preview::DesktopLayout {
- let desktop = desktop_rect(width, height, false);
- let sizes = vec![
- TargetSize {
- width: 16,
- height: 10,
- };
- targets
- ];
- layout_desktop_with_shell(
- desktop.2 as f64,
- desktop.3 as f64,
- &sizes,
- ShellStyle::EdgeTaskbar,
- )
- }
-
- /// Mean of one channel over a small window, which averages out the
- /// material's deterministic grain.
- fn mean_channel(data: &[u8], width: u32, x: u32, y: u32, channel: usize) -> f32 {
- let mut sum = 0u32;
- for dy in 0..4 {
- for dx in 0..4 {
- sum += pixel(data, width, x + dx, y + dy)[channel] as u32;
- }
- }
- sum as f32 / 16.0
- }
-
- #[test]
- fn taskbar_is_edge_anchored_and_spans_the_miniature_desktop() {
- let (width, height) = (520u32, 360u32);
- let desktop = desktop_rect(width, height, false);
- let layout = taskbar_layout(width, height, 2);
-
- assert_eq!(layout.shell, ShellStyle::EdgeTaskbar);
- let bar = offset_pixels(layout.dock, desktop.0, desktop.1);
- assert_eq!(bar.0, desktop.0);
- assert_eq!(bar.2, desktop.2);
- assert!(((bar.1 + bar.3) - (desktop.1 + desktop.3)).abs() <= 1);
-
- // The band is a distinct material, so the wallpaper just above the
- // bar's top edge never matches the pixels just inside it.
- let data = render_view(
- width,
- height,
- &[
- frame("w", "a", PipTargetKind::BrowserTab, Vec::new(), 1),
- frame("w", "b", PipTargetKind::NativeWindow, Vec::new(), 2),
- ],
- &[workspace("w", 2, 2)],
- Some("w"),
- None,
- );
- let center_x = (desktop.0 + desktop.2 / 2) as u32;
- let above = pixel(&data, width, center_x, (bar.1 - 6) as u32);
- let inside = pixel(&data, width, center_x, (bar.1 + 6) as u32);
- assert_ne!(above, inside);
- }
-
- #[test]
- fn taskbar_material_carries_the_blurred_backdrop_instead_of_a_flat_fill() {
- let (width, height) = (520u32, 360u32);
- let desktop = desktop_rect(width, height, false);
- let layout = taskbar_layout(width, height, 1);
- let bar = offset_pixels(layout.dock, desktop.0, desktop.1);
- let data = render_view(
- width,
- height,
- &[frame("w", "a", PipTargetKind::BrowserTab, Vec::new(), 1)],
- &[workspace("w", 1, 1)],
- Some("w"),
- None,
- );
-
- // The wallpaper's lower glow sits left of center, so a material
- // that samples what it covers stays brighter on the left. A flat
- // fill would make these identical.
- let sample_y = (bar.1 + bar.3 / 3) as u32;
- let left = mean_channel(&data, width, (desktop.0 + 80) as u32, sample_y, 0);
- let right = mean_channel(
- &data,
- width,
- (desktop.0 + desktop.2 - 90) as u32,
- sample_y,
- 0,
- );
- assert!(
- left - right >= 3.0,
- "expected a blurred backdrop gradient, got {left} vs {right}"
- );
- // It is still a heavy tint, not a window into the wallpaper.
- assert!(left < 90.0);
- }
-
- #[test]
- fn taskbar_marks_the_newest_card_more_strongly_than_the_others() {
- let (width, height) = (560u32, 380u32);
- let desktop = desktop_rect(width, height, false);
- let layout = taskbar_layout(width, height, 2);
- let data = render_view(
- width,
- height,
- &[
- frame("w", "a", PipTargetKind::BrowserTab, Vec::new(), 1),
- frame("w", "b", PipTargetKind::NativeWindow, Vec::new(), 9),
- ],
- &[workspace("w", 2, 9)],
- Some("w"),
- None,
- );
- assert_eq!(layout.indicators.len(), 2);
- let brightness = |index: usize| {
- let mark = offset_pixels(layout.indicators[index], desktop.0, desktop.1);
- let color = pixel(
- &data,
- width,
- (mark.0 + mark.2 / 2) as u32,
- (mark.1 + mark.3 / 2) as u32,
- );
- color[0] as i32 + color[1] as i32 + color[2] as i32
- };
- // Index 1 is newest, so it wears the accent pill.
- assert!(brightness(1) != brightness(0));
- let active = offset_pixels(layout.indicators[1], desktop.0, desktop.1);
- let accent = pixel(
- &data,
- width,
- (active.0 + active.2 / 2) as u32,
- (active.1 + active.3 / 2) as u32,
- );
- assert!(accent[0] > accent[1] && accent[1] > accent[2]);
- }
-
- #[test]
- fn start_slot_and_tray_paint_inside_the_bar() {
- let (width, height) = (560u32, 380u32);
- let desktop = desktop_rect(width, height, false);
- let layout = taskbar_layout(width, height, 2);
- let bar = offset_pixels(layout.dock, desktop.0, desktop.1);
- let start = offset_pixels(layout.start_button.unwrap(), desktop.0, desktop.1);
- let tray = offset_pixels(layout.tray.unwrap(), desktop.0, desktop.1);
- assert!(start.1 >= bar.1 && start.1 + start.3 <= bar.1 + bar.3);
- assert_eq!(tray.0 + tray.2, bar.0 + bar.2);
-
- let data = render_view(
- width,
- height,
- &[
- frame("w", "a", PipTargetKind::BrowserTab, Vec::new(), 42 * 60_000),
- frame("w", "b", PipTargetKind::NativeWindow, Vec::new(), 1),
- ],
- &[workspace("w", 2, 42 * 60_000)],
- Some("w"),
- None,
- );
- let empty = render_view(width, height, &[], &[workspace("w", 0, 1)], Some("w"), None);
- let glyph_region = |data: &[u8], rect: (i32, i32, i32, i32)| {
- let mut ink = 0u32;
- for y in rect.1..rect.1 + rect.3 {
- for x in rect.0..rect.0 + rect.2 {
- ink += pixel(data, width, x as u32, y as u32)[0] as u32;
- }
- }
- ink
- };
- // Both the launcher glyph and the tray band paint measurably
- // brighter than the bare material behind them.
- assert!(glyph_region(&data, start) > glyph_region(&empty, start));
- assert!(glyph_region(&data, tray) > glyph_region(&empty, tray));
- }
-
- #[test]
- fn clock_and_foreground_selection_are_derived_only_from_frame_inputs() {
- assert_eq!(clock_label(0), "00:00");
- assert_eq!(clock_label(9 * 3_600_000 + 5 * 60_000), "09:05");
- assert_eq!(clock_label(23 * 3_600_000 + 59 * 60_000), "23:59");
- // Wrapping past a day keeps a valid time rather than overflowing.
- assert_eq!(clock_label(25 * 3_600_000), "01:00");
-
- let make = |stamps: [u64; 3]| {
- stamps
- .iter()
- .enumerate()
- .map(|(index, stamp)| {
- frame(
- "w",
- &index.to_string(),
- PipTargetKind::NativeWindow,
- Vec::new(),
- *stamp,
- )
- })
- .collect::<Vec<_>>()
- };
- assert_eq!(active_target_index(&[]), None);
- assert_eq!(active_target_index(&make([1, 7, 3])), Some(1));
- // Ties resolve to the earliest card so repaints stay stable.
- assert_eq!(active_target_index(&make([7, 7, 7])), Some(0));
- }
-
- #[test]
- fn box_blur_flattens_an_impulse_without_changing_a_uniform_field() {
- let uniform = vec![120u8; 9 * 9 * 3];
- assert_eq!(box_blur(&uniform, 9, 9, 2), uniform);
-
- let mut impulse = vec![0u8; 9 * 9 * 3];
- let center = (4 * 9 + 4) * 3;
- impulse[center..center + 3].copy_from_slice(&[255, 255, 255]);
- let blurred = box_blur(&impulse, 9, 9, 2);
- assert!(blurred[center] < 255);
- // Energy spreads to the neighbours the impulse never touched.
- assert!(blurred[((4 * 9 + 6) * 3)] > 0);
- assert!(blurred[((2 * 9 + 4) * 3)] > 0);
- // ...but not past the kernel radius.
- assert_eq!(blurred[((4 * 9 + 8) * 3)], 0);
- }
-
- #[test]
- fn input_passthrough_only_toggles_the_transparent_extended_style() {
- let base = (WS_EX_TOPMOST | WS_EX_NOACTIVATE | WS_EX_TOOLWINDOW).0 as isize;
- let passthrough = extended_style_with_input_passthrough(base, true);
- assert_ne!(passthrough & WS_EX_TRANSPARENT.0 as isize, 0);
- assert_eq!(passthrough & base, base);
-
- let interactive = extended_style_with_input_passthrough(passthrough, false);
- assert_eq!(interactive, base);
- }
+impl PipBackendFactory for WindowsPipBackendFactory {
+ fn start(&self, _cfg: &PipConfig) -> anyhow::Result<Box<dyn PipBackend>> {
+ Err(anyhow::anyhow!(
+ "PiP preview is not yet implemented on Windows — track \
+ trycua/cua follow-up issue for status"
+ ))
}
}
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index f4383cb3a6..ef1477e1d6 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -8031,8 +8031,7 @@ impl Tool for GetConfigTool {
// and read it from the overlay deterministically — `all_states().first()`
// was a nondeterministic HashMap read across sessions (macOS BUG 3).
let cursor_enabled = crate::overlay::is_enabled(&resolve_cursor_key(&args));
- let (agent_view_enabled, agent_view_geometry) =
- pip_preview::read_agent_view_keys_from_file();
+ let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
let payload = json!({
"schema_version": 1,
"version": env!("CARGO_PKG_VERSION"),
@@ -8041,8 +8040,8 @@ impl Tool for GetConfigTool {
"capture_mode": cfg.capture_mode,
"max_image_dimension": cfg.max_image_dimension,
"agent_cursor": { "enabled": cursor_enabled },
- "agent_view": agent_view_enabled,
- "agent_view_geometry": agent_view_geometry,
+ "experimental_pip": pip_enabled,
+ "experimental_pip_geometry": pip_geometry,
});
// Match Swift's text format 1:1: `"✅ <pretty JSON>"`.
let pretty = serde_json::to_string_pretty(&payload).unwrap_or_else(|_| payload.to_string());
@@ -8077,16 +8076,16 @@ impl Tool for SetConfigTool {
`get_window_state` always returns both the UIA tree and a screenshot. Still \
accepted/persisted for back-compat but has no effect.\n\
- `max_image_dimension` (integer)\n\
- - `agent_view` (boolean; enables the multi-target Agent View after restart)\n\
- - `agent_view_geometry` (Agent View size as `WxH` or `WxH+X+Y`; persisted; applies on next daemon restart)\n\n\
+ - `experimental_pip` (boolean; persisted to config.json, applies on next daemon restart — Windows backend stubbed today, see issue #1729)\n\
+ - `experimental_pip_geometry` (string `WxH` or `WxH+X+Y`; persisted; applies on next daemon restart)\n\n\
Returns the full updated config in the same shape as `get_config`.".into(),
input_schema: json!({"type":"object","properties":{
"key":{"type":"string","description":"Dotted snake_case path to a leaf config field (Swift-compatible shape). Pair with `value`."},
"value":{"description":"New value for `key`. JSON type depends on the key."},
"capture_mode":{"type":"string","enum":["ax","vision"],"description":"DEPRECATED and ignored — get_window_state always returns both the UIA tree and a screenshot. Still accepted/persisted for back-compat but has no effect. (\"som\"/\"screenshot\" still decode as deprecated aliases.)"},
"max_image_dimension":{"type":"integer","description":"Legacy per-field shape."},
- "agent_view":{"type":"boolean","description":"Per-field shape. Enables the multi-target Agent View (applies next restart)."},
- "agent_view_geometry":{"type":"string","description":"Per-field shape. Agent View size + optional position."}
+ "experimental_pip":{"type":"boolean","description":"Legacy per-field shape. Enables PiP preview (applies next restart)."},
+ "experimental_pip_geometry":{"type":"string","description":"Legacy per-field shape. PiP window size + optional position."}
},"additionalProperties":false}),
read_only: false, destructive: false, idempotent: true, open_world: false,
})
@@ -8131,31 +8130,31 @@ impl Tool for SetConfigTool {
}
None => return ToolResult::error(format!("`max_image_dimension` must be an integer, got {val}.")),
},
- "agent_view" => match val.as_bool() {
+ "experimental_pip" => match val.as_bool() {
Some(b) => {
- if let Err(e) = pip_preview::write_config_key("agent_view", Value::Bool(b)) {
- return ToolResult::error(format!("failed to persist agent_view: {e}"));
+ if let Err(e) = pip_preview::write_config_key("experimental_pip", Value::Bool(b)) {
+ return ToolResult::error(format!("failed to persist experimental_pip: {e}"));
}
applied = true;
}
- None => return ToolResult::error(format!("`agent_view` must be a boolean, got {val}.")),
+ None => return ToolResult::error(format!("`experimental_pip` must be a boolean, got {val}.")),
},
- "agent_view_geometry" => match val.as_str() {
+ "experimental_pip_geometry" => match val.as_str() {
Some(s) => {
if pip_preview::PipGeometry::parse(s).is_none() {
return ToolResult::error(format!(
- "agent_view_geometry `{s}` is not a valid WxH or WxH+X+Y string"
+ "experimental_pip_geometry `{s}` is not a valid WxH or WxH+X+Y string"
));
}
- if let Err(e) = pip_preview::write_config_key("agent_view_geometry", Value::String(s.to_owned())) {
- return ToolResult::error(format!("failed to persist agent_view_geometry: {e}"));
+ if let Err(e) = pip_preview::write_config_key("experimental_pip_geometry", Value::String(s.to_owned())) {
+ return ToolResult::error(format!("failed to persist experimental_pip_geometry: {e}"));
}
applied = true;
}
- None => return ToolResult::error(format!("`agent_view_geometry` must be a string, got {val}.")),
+ None => return ToolResult::error(format!("`experimental_pip_geometry` must be a string, got {val}.")),
},
other => return ToolResult::error(format!(
- "Unknown config key `{other}`. Known: capture_mode, max_image_dimension, agent_view, agent_view_geometry."
+ "Unknown config key `{other}`. Known: capture_mode, max_image_dimension, experimental_pip, experimental_pip_geometry."
)),
}
}
@@ -8176,22 +8175,29 @@ impl Tool for SetConfigTool {
}
applied = true;
}
- if let Some(enabled) = args.get("agent_view").and_then(|v| v.as_bool()) {
- if let Err(e) = pip_preview::write_config_key("agent_view", Value::Bool(enabled)) {
- return ToolResult::error(format!("failed to persist agent_view: {e}"));
+ if let Some(enabled) = args.get("experimental_pip").and_then(|v| v.as_bool()) {
+ if let Err(e) = pip_preview::write_config_key("experimental_pip", Value::Bool(enabled))
+ {
+ return ToolResult::error(format!("failed to persist experimental_pip: {e}"));
}
applied = true;
}
- if let Some(geom) = args.get("agent_view_geometry").and_then(|v| v.as_str()) {
+ if let Some(geom) = args
+ .get("experimental_pip_geometry")
+ .and_then(|v| v.as_str())
+ {
if pip_preview::PipGeometry::parse(geom).is_none() {
return ToolResult::error(format!(
- "agent_view_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
+ "experimental_pip_geometry `{geom}` is not a valid WxH or WxH+X+Y string"
));
}
- if let Err(e) =
- pip_preview::write_config_key("agent_view_geometry", Value::String(geom.to_owned()))
- {
- return ToolResult::error(format!("failed to persist agent_view_geometry: {e}"));
+ if let Err(e) = pip_preview::write_config_key(
+ "experimental_pip_geometry",
+ Value::String(geom.to_owned()),
+ ) {
+ return ToolResult::error(format!(
+ "failed to persist experimental_pip_geometry: {e}"
+ ));
}
applied = true;
}
@@ -8209,8 +8215,7 @@ impl Tool for SetConfigTool {
.first()
.map(|s| s.config.enabled)
.unwrap_or(true);
- let (agent_view_enabled, agent_view_geometry) =
- pip_preview::read_agent_view_keys_from_file();
+ let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file();
let payload = json!({
"schema_version": 1,
"version": env!("CARGO_PKG_VERSION"),
@@ -8219,8 +8224,8 @@ impl Tool for SetConfigTool {
"capture_mode": cfg.capture_mode,
"max_image_dimension": cfg.max_image_dimension,
"agent_cursor": { "enabled": cursor_enabled },
- "agent_view": agent_view_enabled,
- "agent_view_geometry": agent_view_geometry,
+ "experimental_pip": pip_enabled,
+ "experimental_pip_geometry": pip_geometry,
});
let pretty = serde_json::to_string_pretty(&payload).unwrap_or_else(|_| payload.to_string());
ToolResult::text(format!("✅ {pretty}")).with_structured(payload)