Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(cua-driver): refine session cursor feedback (#2665)
Francesco Bonaccid366af084e53
diff --git a/libs/cua-driver/docs/0.13.2-overnight-execution-journal.md b/libs/cua-driver/docs/0.13.2-overnight-execution-journal.mdnew file mode 100644index 0000000000..44d74dc83c--- /dev/null+++ b/libs/cua-driver/docs/0.13.2-overnight-execution-journal.md@@ -0,0 +1,67 @@+# Cua Driver 0.13.2 Overnight Execution Journal++**Status:** Active+**Started from:** `origin/main` at `1e77ab5536eb88ca4cb7d3b6cc1fb51eea03f0e4`+**Released baseline:** 0.13.1+**Release boundary:** No release, tag, deployment, or package publication++## Confirmed scope++- Reduce the default cursor size.+- Fade the session-name badge after a short reveal.+- Match the production badge to the session-accent gradient shown by the web+ harness.+- Replace the square badge marker with a circular session dot.+- Audit every public action and fix spatial actions whose agent cursor does not+ move to the resolved target.+- Complete and merge PR #2660 after exact-head macOS evidence.+- Complete the prioritized post-0.13.1 stabilization plan.+- Merge validated work into `main`.+- Run macOS Lume and Windows and Linux GitHub Actions validation.++## Baseline facts++- The original worktree contains existing user changes and remains untouched.+- The execution worktree is clean apart from this plan and journal.+- The baseline default cursor display size was 48 points.+- The baseline production badge used a solid dark background.+- The baseline production badge marker was painted with `fill_rect`, so it was+ square.+- Semantic action events are emitted centrally.+- Spatial cursor motion is implemented in platform tool paths, not by the+ central semantic event classifier.++## Work log++### Phase 1: cursor audit++- Reduced the shared native and GNOME cursor footprint from 48 to 42 logical+ points.+- Added an independent two-second badge hold and 400 millisecond fade clock.+- Ported the session-accent gradient badge into the production renderer and+ replaced the rectangular marker with a circular session orb.+- Updated the GNOME Shell renderer to API version 7 with the same size, badge,+ and fade behavior.+- Preserved the active semantic state when a spatial glide or click pulse is+ dispatched. Text, scroll, drag, and other states are no longer overwritten+ by `navigate` or `click`.+- Added indexed and token-addressed text movement on macOS and Windows.+- Corrected Linux text and value movement to use the declared session cursor+ rather than the anonymous default cursor, and moved indexed text before+ early native and compositor delivery returns.+- Corrected token-addressed cursor classification to use the AX modifier.+- Added the production session badge and fade to the repository cursor gallery.++## Evidence ledger++| SHA | Platform | Surface | Result | Evidence |+|---|---|---|---|---|+| Working tree | Local | Cursor contract and renderer | 32 renderer and 3 contract tests passing before the latest platform movement patch | Local cargo output |++## Merged PRs++- None yet.++## Current blockers++- None.diff --git a/libs/cua-driver/docs/0.13.2-overnight-stabilization-plan.md b/libs/cua-driver/docs/0.13.2-overnight-stabilization-plan.mdnew file mode 100644index 0000000000..01037843d1--- /dev/null+++ b/libs/cua-driver/docs/0.13.2-overnight-stabilization-plan.md@@ -0,0 +1,385 @@+# Cua Driver 0.13.2 Overnight Stabilization Plan++**Status:** Draft for review+**Baseline:** Cua Driver 0.13.1 is already released+**Source baseline:** `origin/main` at `1e77ab5536eb88ca4cb7d3b6cc1fb51eea03f0e4`+**Target:** Merge validated post-0.13.1 work into `main`+**Release boundary:** Do not create, publish, promote, or merge a 0.13.2 release. Francesco owns the 0.13.2 release tomorrow.++## 1. Goal++Leave `main` ready for a 0.13.2 release by:++1. making the default cursor smaller and correcting its session badge;+2. auditing every spatial action so the cursor moves to the action target;+3. landing the hermetic macOS certification work in PR [#2660](https://github.com/trycua/cua/pull/2660);+4. resolving the highest-impact regressions discovered around 0.13.1;+5. restoring the intended release automation without triggering a release;+6. validating the exact merged `main` SHA on macOS, Windows, and Linux;+7. updating public docs and post-install guidance for any behavior that changes;+8. preserving reviewable evidence, including videos for GUI behavior.++This is stabilization and correctness work. It is not a release operation.++## 2. Definition of done++The overnight work is complete only when:++- every accepted product or workflow fix is merged into `main` through a focused PR;+- the production cursor is smaller on macOS, Windows, Linux X11, and Linux Wayland;+- the session badge fades away after a short reveal, uses the promised+ session-accent gradient, and has a circular session dot;+- every supported spatial action either moves the agent cursor to its resolved+ target or documents why the action has no spatial target;+- the spatial-action audit has contract tests and representative E2E evidence;+- PR #2660 is synchronized with current `main`, reviewed, fully validated, and merged;+- issue #2655 has two consecutive same-SHA macOS certification runs that do not depend on stale worker state;+- canonical Cua Driver installers resolve and install the released 0.13.1 artifacts before any source-built validation begins;+- the final merged `main` SHA passes the relevant deterministic Linux and Windows CI;+- representative desktop E2E evidence exists for macOS, Windows, Linux X11, and Linux Wayland where the changed behavior applies;+- every GUI success is proven by an external effect, not only `ok: true`;+- background claims retain focus, z-order, real-cursor, occlusion, and input-isolation evidence;+- public docs, generated references, examples, and post-install instructions match the final behavior;+- PR titles and labels accurately describe release impact and pass release-metadata validation;+- no release, tag, package publication, deployment, or Release Please PR merge occurs;+- the execution journal states the final `main` SHA, merged PRs, exact test results, artifacts, limitations, and remaining owner.++## 3. Operating boundaries++- Start every new unit of work from freshly fetched `origin/main`.+- Preserve the existing dirty worktree. Use clean branches or worktrees for all implementation.+- Keep each independent regression in its own focused PR.+- Merge only after the PR's exact head has the required tests and evidence.+- Do not bypass a required human review or a protected branch rule.+- Do not overlap with independent backlog or contributor-PR review work.+- Preserve external contributor authorship when adapting submitted work.+- Do not introduce a breaking public API, CLI, SDK, protocol, or configuration change without asking first.+- Do not weaken product assertions or relabel a behavior failure as an environment failure.+- Keep public artifacts free of credentials, private infrastructure details, machine-specific paths, and raw session data.+- Use a living execution journal at `libs/cua-driver/docs/0.13.2-overnight-execution-journal.md`.++## 4. Priority and landing order++### P0. Correct the cursor size, session badge, and spatial action motion++Confirmed product requirements:++1. Make the default cursor visibly smaller while preserving legibility on+ light and dark backgrounds.+2. Reveal the session-name badge with the pointer, then fade only the badge+ away after a short readable interval. Cursor visibility and idle behavior+ remain independent.+3. Match the production badge to the repository web harness:+ - session-accent background gradient;+ - circular session-color dot;+ - centered label;+ - compact spacing and cursor gap;+ - consistent backing-scale behavior.+4. Audit every public action and fix spatial actions that update the semantic+ icon without moving the agent cursor to their resolved target.++Implementation work:++1. Treat the native Skia renderer as the production source of truth and the+ web harness as a visual contract and review surface.+2. Add a renderer-owned badge reveal clock with explicit hold and fade+ durations. Reset it when the session label is first set and whenever a new+ session becomes visibly active.+3. Keep the fade deterministic, independent per session, and disabled only+ when the badge itself is absent.+4. Render the badge accent with the session fill and bounded light and dark+ variants. Do not accept an agent-controlled arbitrary badge color.+5. Render the dot through a circular path, not a square rectangle.+6. Reduce the one canonical production display size and update all tests,+ docs, gallery exports, and platform bounds that depend on it.+7. Create a complete action-motion inventory with:+ - public tool;+ - semantic action;+ - target source;+ - expected movement;+ - platform implementation;+ - external E2E oracle.+8. For actions with resolved coordinates or element bounds, move the agent+ cursor before dispatch while preserving the real pointer for background+ delivery.+9. For actions without a spatial target, keep the cursor at its current+ location and animate the semantic state there.+10. At minimum, audit and test:+ - click, double click, right click;+ - drag;+ - scroll;+ - type text and set value;+ - press key and hotkey;+ - browser click, drag, scroll, type, fill, press key, and hotkey;+ - navigation, app, transfer, record, system, and observe actions.+11. Assert that semantic begin and end events are emitted for every classified+ public tool with an explicit session.+12. Add platform-focused tests that prove spatial actions send the correct+ keyed move command before action dispatch.+13. Run the repository cursor gallery and record the complete action and+ modifier sequence.+14. Run real Cua Driver action sequences on macOS, Windows, Linux X11, and+ Linux Wayland. Verify application effects independently and record videos.+15. Update cursor personalization, action catalog, session, and post-install+ documentation.++### P0. Make macOS certification hermetic++Tracking:++- Issue [#2655](https://github.com/trycua/cua/issues/2655)+- PR [#2660](https://github.com/trycua/cua/pull/2660)++Work:++1. Rebase or merge the latest `origin/main` into the PR branch without losing its existing evidence.+2. Re-run shell syntax, ShellCheck, focused script tests, Rust formatting, and all changed-path CI.+3. Preflight the macOS Lume host and guest:+ - adequate host and guest disk space;+ - logged-in GUI session;+ - exact source SHA installed;+ - correct TCC identity and grants;+ - no stale daemon, socket, fixture, Cargo target, or build-cache ownership;+ - sufficient evidence output space.+4. Run the canonical full macOS certification matrix twice consecutively at the same exact PR SHA.+5. Require each run to:+ - create a fresh run-owned build namespace;+ - own daemon startup, permission mode, socket, watchdog, shutdown, and restoration;+ - execute every declared row;+ - preserve the initial attempt and any allowed exact-cell retry;+ - record row-level screenshots, logs, videos, and external oracles;+ - restore the normal daemon after success or failure.+6. If a run fails, classify it before changing anything:+ - product failure: fix product behavior and restart exact-head validation;+ - harness failure: fix the harness without weakening assertions;+ - environment failure: prove it through preflight or independent system evidence and retry in a repaired representative environment.+7. Update the PR body with the final exact SHA and both consecutive run links.+8. Confirm the `test(cua-driver)` title and `no-release` label remain accurate.+9. Merge after required review and checks pass.+10. Verify issue #2655 is closed by the merged PR.++### P0. Verify the shipped installer baseline++Tracking:++- Issue [#2654](https://github.com/trycua/cua/issues/2654)++Work:++1. Verify the canonical Unix and Windows installers resolve the released 0.13.1 component tag.+2. Verify the expected macOS, Linux, and Windows assets exist under that exact component release.+3. Exercise a clean install and an upgrade from the prior supported version on each platform.+4. Verify:+ - installed executable path;+ - `cua-driver --version`;+ - source or release provenance;+ - daemon start and stop;+ - one representative MCP smoke test.+5. If the 0.13.1 release already resolves the reported 0.13.0 404, record the evidence and close #2654 as resolved.+6. If any canonical path still fails, fix it in a focused PR and rerun all installer-contract tests before merging.++### P0. Restore safe automatic publication for future releases++Tracking:++- Issue [#2662](https://github.com/trycua/cua/issues/2662)++Work:++1. Restore the intended normal release path after an approved Release Please PR merge:+ - immutable component tag;+ - required build and verification dependencies;+ - GitHub release publication;+ - version-locked PyPI and npm SDK publication.+2. Keep PR comment and checkbox controls unable to bump or publish packages.+3. Preserve manual recovery dispatch without making it the normal path.+4. Add deterministic workflow tests for:+ - valid component tag push;+ - required build failure;+ - tag and SHA mismatch;+ - disabled PR comment or checkbox path;+ - explicit recovery dispatch.+5. Validate the workflow without creating a tag, release, or package publication.+6. Merge the focused workflow PR only after release-metadata and workflow tests pass.++### P0. Fix silent Linux process-control success++Tracking:++- Issue [#2661](https://github.com/trycua/cua/issues/2661)++Work:++1. Reproduce `kill_app` reporting success while the process survives.+2. Reproduce `launch_app.launch_path` leaving a shell child hung without executing.+3. Trace process identity, signal delivery, child state, pipes, process groups, wait behavior, and timeout handling.+4. Change the contract so:+ - `kill_app` verifies the target exited or returns a structured failure;+ - PID reuse cannot produce a false success;+ - `launch_path` drains or redirects child pipes safely;+ - launch completion has a bounded, observable condition;+ - abandoned children are reaped;+ - unsupported sandbox behavior fails explicitly.+5. Add deterministic unit and integration coverage.+6. Run representative Linux X11 and Wayland E2E.+7. Run a gVisor-equivalent proof when an existing authorized environment is available. Otherwise, state that gVisor remains unproven and do not claim the reported platform fixed.+8. Merge only when the product cannot silently report a process-control success with no effect.++### P1. Stop Cua Driver from launching a screen reader on COSMIC++Tracking:++- Issue [#2630](https://github.com/trycua/cua/issues/2630)++Work:++1. Confirm the current default behavior from source and a representative Linux session.+2. Prefer the safe default for unknown desktops: enable generic AT-SPI inspection without claiming that a screen reader is active.+3. Preserve an explicit override for environments that genuinely require full advertisement.+4. Add regression tests for GNOME, COSMIC, known non-GNOME desktops, mixed desktop identifiers, and unknown desktops.+5. Verify on Linux Wayland that:+ - Orca and Speech Dispatcher do not start;+ - accessibility inspection still works;+ - a representative action still completes.+6. Update Linux troubleshooting and accessibility documentation if the override or default behavior changes.+7. Merge the focused fix after exact-head Linux validation.++### P1. Improve drag cursor animation++Tracking:++- Issue [#2625](https://github.com/trycua/cua/issues/2625)++Work:++1. Animate drag as a continuous point-to-point gesture rather than a standard cursor movement.+2. Keep the production cursor compact, session-colored, legible on light and dark backgrounds, and consistent across platforms.+3. Preserve the existing semantic action and delivery modifiers.+4. Add deterministic rendering or timeline tests for:+ - drag start;+ - continuous travel;+ - destination arrival;+ - release;+ - cancellation or refusal.+5. Validate at least one foreground and one supported background drag path per representative platform.+6. Record short macOS, Windows, Linux X11, and Linux Wayland videos where supported.+7. Update cursor personalization documentation and the repository cursor preview harness.+8. Keep this change separate from correctness and release-workflow fixes.++## 5. Cross-platform validation matrix++| Platform | Environment | Minimum proof |+|---|---|---|+| macOS | Logged-in Lume macOS VM with exact source SHA and TCC grants | Two consecutive full certification runs for #2660, installer smoke, affected product rows, screenshots, logs, and videos |+| Windows | GitHub Actions for deterministic coverage, interactive Azure VM when desktop behavior changes | Install or upgrade, daemon and MCP smoke, affected GUI rows, external oracles, and video |+| Linux X11 | GitHub Actions when representative, otherwise interactive Azure VM | Install or upgrade, daemon and MCP smoke, affected action rows, focus and input guards, and video |+| Linux Wayland | Representative GNOME or supported compositor session in Actions or Azure | Portal and accessibility preflight, affected action rows, external oracles, and video |+| Linux COSMIC | Representative session if available | No Orca launch, AT-SPI remains usable, one successful action |+| Linux gVisor | Existing authorized sandbox environment if available | Process exit and launch effects independently verified, with no silent success |++Hosted CI proves deterministic build and unit contracts. It does not substitute for a real interactive desktop when the claim depends on focus, z-order, TCC, compositor routing, or visible cursor behavior.++## 6. Documentation and examples audit++After the product PRs are stable:++1. Audit public Cua Driver tutorials, how-to guides, reference pages, generated tool schemas, examples, and post-install output.+2. Verify that:+ - 0.13.1 is described as the installed baseline until 0.13.2 is released;+ - no page claims 0.13.2 is available;+ - installer instructions use canonical component resolution;+ - permission and session examples match the shipped contract;+ - cursor customization documents required states and the drag timeline;+ - Linux accessibility guidance explains the safe advertisement default and override;+ - process-control errors and timeouts are documented where user action is possible.+3. Regenerate checked-in references with repository-owned generators.+4. Run link, formatting, generated-output, and sensitive-information scans.+5. Keep internal execution notes out of public-facing pages.++## 7. PR and merge discipline++Use one focused PR for each independent unit:++1. cursor size, badge, and action-motion correctness+2. #2660 macOS certification harness+3. #2662 release automation+4. #2661 Linux process control+5. #2630 COSMIC accessibility advertisement+6. #2625 drag cursor animation, unless the confirmed cursor PR fully supersedes it+7. any documentation-only reconciliation that cannot live accurately with its product PR++For every PR:++- synchronize with the latest `origin/main` immediately before final validation;+- inspect the final changed files and live PR title;+- preserve contributor credit;+- run the exact relevant unit, integration, generated-output, and platform checks;+- attach exact-SHA evidence;+- address review feedback;+- wait for required checks;+- merge only after the completion bar is satisfied;+- verify `main` contains the merge before starting dependent validation.++Do not combine unrelated fixes merely to reduce PR count.++## 8. Final merged-main certification++After all accepted PRs merge:++1. Fetch the final `origin/main` and record its exact SHA.+2. Run deterministic Rust, script, installer-contract, release-metadata, docs, Python SDK, and TypeScript SDK checks affected by the work.+3. Run the representative macOS, Windows, Linux X11, and Linux Wayland E2E cells affected by the merged changes.+4. Verify canonical 0.13.1 installation still works. Source-built tests must report their source SHA and must not be confused with the released binary.+5. Verify no tag, GitHub release, PyPI version, npm version, or installer baked version was changed to 0.13.2.+6. Publish a final internal readiness report containing:+ - merged PRs and issues;+ - final `main` SHA;+ - per-platform results;+ - videos and run links;+ - supported, refused, unavailable, and unproven cells;+ - exact remaining work owned by Francesco for tomorrow's 0.13.2 release.++## 9. Stop and escalation conditions++Continue autonomously through ordinary implementation choices, test failures, flaky infrastructure repair, documentation changes, and PR feedback.++Stop and ask only if:++- a public contract requires a breaking change;+- a required representative environment needs new paid infrastructure or credentials;+- a security vulnerability is discovered;+- a destructive or irreversible external action is required;+- a release, tag, deployment, or package publication would occur;+- a protected branch or required human review prevents an otherwise ready merge.++An isolated failed run, a VM crash, a full disk, a CI timeout, or a different recoverable defect is not a reason to stop. Preserve evidence, repair the environment or implementation, and retry.++## 10. Explicit exclusions++- Creating or publishing 0.13.2+- Merging a 0.13.2 Release Please PR+- General backlog cleanup+- Reviewing unrelated contributor PRs+- Refactoring the daemonless architecture without a regression requiring it+- Expanding permission policy beyond what a reproduced issue requires+- Changing public contracts without explicit approval++## 11. Additional GitHub work after confirmed scope++After all confirmed P0 and P1 work passes and merges:++1. Refresh open Cua Driver issues and PRs from current GitHub state.+2. Exclude work already owned by another active task or contributor review.+3. Rank candidates by:+ - silent false success or data-loss risk;+ - regression affecting released 0.13.1 behavior;+ - installer or release blocker for 0.13.2;+ - cross-platform user impact;+ - bounded implementation and representative testability.+4. Prefer superseding a stale PR only when:+ - its useful behavior is still needed;+ - direct landing is no longer practical;+ - authorship can be preserved through cherry-pick or coauthor credit;+ - parity is proven before closing the source PR.+5. Take only work that can be implemented, tested, reviewed, and merged without+ weakening the final merged-main certification.diff --git a/libs/cua-driver/docs/cursor-themes.md b/libs/cua-driver/docs/cursor-themes.mdindex 125535b9dd..1e05aac380 100644--- a/libs/cua-driver/docs/cursor-themes.md+++ b/libs/cua-driver/docs/cursor-themes.md@@ -16,12 +16,21 @@ uses Cua blue. Named sessions receive a stable fillfrom the built-in session palette, so concurrent agents remain visuallydistinct.-The native renderer places the sanitized public session name in a badge below-the built-in pointer. The badge follows the cursor and uses the same session-color, but it is not part of the dotLottie theme artifact. Custom theme authors+The native renderer places the sanitized public session name in a compact badge+below the built-in pointer. The badge uses a session-accent gradient and a+circular session marker. It appears when the cursor is first revealed, remains+fully visible for two seconds, then fades over 400 milliseconds while the+cursor stays available. Hiding and revealing the cursor shows the badge again.+The badge is not part of the dotLottie theme artifact, so custom theme authorsdo not need to provide badge artwork or a font. Cua Driver strips control-characters, collapses whitespace, and shortens labels longer than 28-characters before rendering the bundled Inter typeface.+characters, collapses whitespace, and shortens labels longer than 28 characters+before rendering the bundled Inter typeface.++The production pointer footprint is 42 logical points. Actions with a resolved+screen target, including indexed or token-addressed text and value operations,+move the agent cursor to that target before dispatch. Actions without a spatial+target, such as typing into the already focused desktop application, keep the+cursor at its current location and only change the semantic animation.The cursor is a visual aid for people watching an agent. It is not a securityindicator, an authorization prompt, or evidence that a tool call succeeded.@@ -232,9 +241,10 @@ From the repository root:./libs/cua-driver/scripts/cursor-gallery.sh serve```-Open `http://127.0.0.1:3001` to pause, replay, change speed, and compare the-states on light, dark, blue, or mixed backgrounds. The generated WebM files and-raw frames are ignored by Git.+Open `http://127.0.0.1:3001` to pause, replay, change speed, compare the states+on light, dark, blue, or mixed backgrounds, and inspect the exact session-badge+hold and fade behavior. The generated WebM files and raw frames are ignored by+Git.Regenerate the public documentation GIFs after changing the built-in cursor:diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rsindex 872524823b..e8ff92a855 100644--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs@@ -259,6 +259,7 @@ pub fn classify_cursor_semantics(name: &str, args: &Value) -> Option<CursorSemanSome(CursorTarget::Browser)} else if args.get("element_index")+ .or_else(|| args.get("element_token")).is_some_and(|value| !value.is_null()){Some(CursorTarget::Ax)@@ -331,6 +332,17 @@ mod tests {target: Some(CursorTarget::Browser),}));+ assert_eq!(+ classify_cursor_semantics(+ "type_text",+ &json!({"element_token":"opaque", "delivery_mode":"foreground"})+ ),+ Some(CursorSemantics {+ action: CursorAction::Text,+ delivery: Some(CursorDelivery::Foreground),+ target: Some(CursorTarget::Ax),+ })+ );}#[test]diff --git a/libs/cua-driver/rust/crates/cursor-overlay/examples/export_gallery_frames.rs b/libs/cua-driver/rust/crates/cursor-overlay/examples/export_gallery_frames.rsindex b500987fed..aedc75a857 100644--- a/libs/cua-driver/rust/crates/cursor-overlay/examples/export_gallery_frames.rs+++ b/libs/cua-driver/rust/crates/cursor-overlay/examples/export_gallery_frames.rs@@ -1,6 +1,6 @@use cursor_overlay::{- theme::paint_default_theme, CursorAction, CursorVisualState, DeliveryModifier, ReducedMotion,- TargetModifier,+ render_frame, theme::paint_default_theme, CursorAction, CursorConfig, CursorVisualState,+ DeliveryModifier, OverlayCommand, ReducedMotion, RenderStateCore, TargetModifier,};use std::{fs, path::Path};use tiny_skia::Pixmap;@@ -54,6 +54,38 @@ fn main() {scope.spawn(move || export_state(&path, action, delivery, target));}});+ export_session_badge(&output.join("session").join("badge"));+}++fn export_session_badge(output: &Path) {+ fs::create_dir_all(output).expect("create session badge frame output");+ for frame in 0..FPS * DURATION_SECS {+ let mut config = CursorConfig::default();+ config.cursor_id = "gallery-session".into();+ let mut core = RenderStateCore::new(config);+ core.motion.idle_hide_ms = 0.0;+ core.pos = (+ f64::from(SIZE) / (2.0 * f64::from(PREVIEW_BACKING_SCALE)),+ 64.0,+ );+ core.apply_command_base(+ OverlayCommand::SetSessionLabel("Research".into()),+ false,+ false,+ );+ core.tick_motion(f64::from(frame) / f64::from(FPS));+ let pixmap = render_frame(&core, SIZE, SIZE, 0.0, 0.0, None, PREVIEW_BACKING_SCALE);+ let pixels = unpremultiply_rgba(pixmap.data().to_vec());+ image::save_buffer_with_format(+ output.join(format!("{frame:04}.png")),+ &pixels,+ SIZE,+ SIZE,+ image::ColorType::Rgba8,+ image::ImageFormat::Png,+ )+ .expect("write session badge frame");+ }}fn export_state(diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rsindex 28a196d742..6955828053 100644--- a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs@@ -40,6 +40,9 @@ use crate::{};use std::sync::Arc;+pub const SESSION_BADGE_HOLD_SECS: f64 = 2.0;+pub const SESSION_BADGE_FADE_SECS: f64 = 0.4;+/// Platform-agnostic render state shared by macOS / Windows / Linux overlays.////// Each platform wraps this in its own struct that adds OS-specific fields@@ -81,6 +84,8 @@ pub struct RenderStateCore {pub pinned_wid: Option<u64>,/// Sanitized caller-facing label painted below the cursor.pub session_label: Option<String>,+ /// Elapsed time since the session label was revealed with the cursor.+ pub session_badge_secs: f64,}impl RenderStateCore {@@ -124,9 +129,33 @@ impl RenderStateCore {idle_alpha: 1.0,pinned_wid: None,session_label: None,+ session_badge_secs: SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS,+ }+ }++ fn cursor_is_revealed(&self) -> bool {+ self.visible && self.pos.0 >= -100.0 && self.idle_alpha >= 0.004+ }++ fn reveal_session_badge(&mut self) {+ if self.session_label.is_some() {+ self.session_badge_secs = 0.0;}}+ pub fn session_badge_alpha(&self) -> f32 {+ if self.session_label.is_none() {+ return 0.0;+ }+ if self.session_badge_secs <= SESSION_BADGE_HOLD_SECS {+ return 1.0;+ }+ let fade = ((self.session_badge_secs - SESSION_BADGE_HOLD_SECS) / SESSION_BADGE_FADE_SECS)+ .clamp(0.0, 1.0);+ let smooth = fade * fade * (3.0 - 2.0 * fade);+ (1.0 - smooth) as f32+ }+/// Return the theme that is actually being painted, including any/// non-fatal fallback from an unavailable launch-time selection.pub fn active_theme_metadata(&self) -> (String, String, String, Option<String>) {@@ -367,6 +396,10 @@ impl RenderStateCore {/// `motion.idle_hide_ms` has elapsed. Identical across all platforms.fn tick_idle(&mut self, dt: f64) {self.visual.tick(dt);+ if self.session_label.is_some() {+ self.session_badge_secs = (self.session_badge_secs + dt)+ .min(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);+ }let idle_hide_ms = self.motion.idle_hide_ms;if idle_hide_ms > 0.0 {let moving = self.path.is_some() || self.spring.is_some() || self.click_t.is_some();@@ -416,6 +449,7 @@ impl RenderStateCore {y,end_heading_radians,} => {+ let reveal_badge = !self.cursor_is_revealed();// Apply click offset (16 pt along end_heading) before planning,// matching Swift `moveTo(point:endAngleRadians:)`:// tx = clickPoint.x + cos(endAngle) * clickOffset@@ -439,9 +473,17 @@ impl RenderStateCore {self.dist = 0.0;self.spring = None;self.spring_tgt = None;- self.visual.begin(CursorAction::Navigate, None, None);+ if matches!(+ self.visual.resolved_action,+ CursorAction::Idle | CursorAction::Navigate+ ) {+ self.visual.begin(CursorAction::Navigate, None, None);+ }self.idle_secs = 0.0;self.idle_alpha = 1.0;+ if reveal_badge {+ self.reveal_session_badge();+ }true}OverlayCommand::SnapTo {@@ -449,6 +491,7 @@ impl RenderStateCore {y,heading_radians,} => {+ let reveal_badge = !self.cursor_is_revealed();self.pos = (x, y);if let Some(heading) = heading_radians {self.heading = heading;@@ -457,12 +500,21 @@ impl RenderStateCore {self.dist = 0.0;self.spring = None;self.spring_tgt = None;- self.visual.begin(CursorAction::Navigate, None, None);+ if matches!(+ self.visual.resolved_action,+ CursorAction::Idle | CursorAction::Navigate+ ) {+ self.visual.begin(CursorAction::Navigate, None, None);+ }self.idle_secs = 0.0;self.idle_alpha = 1.0;+ if reveal_badge {+ self.reveal_session_badge();+ }true}OverlayCommand::ClickPulse { x, y } => {+ let reveal_badge = !self.cursor_is_revealed();if click_pulse_sentinel_only {// macOS: only snap position on first placement (sentinel state).// After that the cursor stays where the animation landed.@@ -479,9 +531,17 @@ impl RenderStateCore {self.pos = (x, y);}self.click_t = Some(0.0);- self.visual.begin(CursorAction::Click, None, None);+ if matches!(+ self.visual.resolved_action,+ CursorAction::Idle | CursorAction::Navigate | CursorAction::Click+ ) {+ self.visual.begin(CursorAction::Click, None, None);+ }self.idle_secs = 0.0;self.idle_alpha = 1.0;+ if reveal_badge {+ self.reveal_session_badge();+ }true}OverlayCommand::SetPressed(v) => {@@ -496,7 +556,11 @@ impl RenderStateCore {true}OverlayCommand::SetEnabled(v) => {+ let reveal_badge = v && !self.visible;self.visible = v;+ if reveal_badge {+ self.reveal_session_badge();+ }true}OverlayCommand::SetMotion(m) => {@@ -543,6 +607,7 @@ impl RenderStateCore {}OverlayCommand::SetSessionLabel(label) => {self.session_label = crate::sanitize_session_label(&label);+ self.session_badge_secs = 0.0;true}OverlayCommand::ShowFocusRect(_) => false, // caller-specific@@ -718,13 +783,14 @@ pub fn paint_cursor(}if let Some(label) = core.session_label.as_deref() {+ let badge_alpha = core.session_badge_alpha();crate::paint_session_badge(pm,label,px as f32,py as f32,backing_scale.max(1.0),- alpha_scale,+ alpha_scale * badge_alpha,crate::session_fill_rgba(&core.cfg.cursor_id),);}@@ -790,6 +856,100 @@ mod glide_duration_tests {}}+#[cfg(test)]+mod session_badge_and_action_tests {+ use super::*;+ use crate::{CursorConfig, TargetModifier};++ #[test]+ fn session_badge_holds_then_fades_once() {+ let mut core = RenderStateCore::new(CursorConfig::default());+ assert_eq!(core.session_badge_alpha(), 0.0);+ assert!(core.apply_command_base(+ OverlayCommand::SetSessionLabel("Research".into()),+ false,+ false,+ ));+ assert_eq!(core.session_badge_alpha(), 1.0);++ core.tick_motion(SESSION_BADGE_HOLD_SECS - 0.05);+ assert_eq!(core.session_badge_alpha(), 1.0);+ core.tick_motion(SESSION_BADGE_FADE_SECS * 0.5 + 0.05);+ assert!(core.session_badge_alpha() > 0.0);+ assert!(core.session_badge_alpha() < 1.0);+ core.tick_motion(SESSION_BADGE_FADE_SECS);+ assert_eq!(core.session_badge_alpha(), 0.0);+ }++ #[test]+ fn revealing_hidden_cursor_restarts_badge_without_restarting_on_every_move() {+ let mut core = RenderStateCore::new(CursorConfig::default());+ core.apply_command_base(+ OverlayCommand::SetSessionLabel("Research".into()),+ false,+ false,+ );+ core.tick_motion(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);+ assert_eq!(core.session_badge_alpha(), 0.0);++ core.apply_command_base(+ OverlayCommand::SnapTo {+ x: 100.0,+ y: 100.0,+ heading_radians: None,+ },+ false,+ false,+ );+ assert_eq!(core.session_badge_alpha(), 1.0);+ core.tick_motion(0.5);+ let elapsed = core.session_badge_secs;+ core.apply_command_base(+ OverlayCommand::SnapTo {+ x: 120.0,+ y: 120.0,+ heading_radians: None,+ },+ false,+ false,+ );+ assert_eq!(core.session_badge_secs, elapsed);+ }++ #[test]+ fn movement_preserves_the_active_semantic_action() {+ let mut core = RenderStateCore::new(CursorConfig::default());+ core.pos = (20.0, 20.0);+ core.apply_command_base(+ OverlayCommand::BeginAction {+ action: CursorAction::Text,+ delivery: None,+ target: Some(TargetModifier::Ax),+ },+ false,+ false,+ );+ core.apply_command_base(+ OverlayCommand::MoveTo {+ x: 200.0,+ y: 100.0,+ end_heading_radians: 0.0,+ },+ false,+ false,+ );+ assert_eq!(core.visual.resolved_action, CursorAction::Text);+ assert_eq!(core.visual.target, Some(TargetModifier::Ax));+ core.apply_command_base(+ OverlayCommand::ClickPulse { x: 200.0, y: 100.0 },+ false,+ false,+ );+ assert_eq!(core.visual.resolved_action, CursorAction::Text);+ assert_eq!(core.visual.target, Some(TargetModifier::Ax));+ }+}+#[cfg(test)]mod backing_scale_tests {use super::*;diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rsindex 45db567cc9..2fd7575f22 100644--- a/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs@@ -9,18 +9,22 @@use fontdue::layout::{CoordinateSystem, Layout, LayoutSettings, TextStyle};use std::sync::OnceLock;-use tiny_skia::{Color, Paint, Path, PathBuilder, Pixmap, PixmapPaint, Rect, Stroke, Transform};+use tiny_skia::{+ Color, GradientStop, LinearGradient, Paint, Path, PathBuilder, Pixmap, PixmapPaint, Point,+ Rect, SpreadMode, Stroke, Transform,+};pub const MAX_SESSION_LABEL_CHARS: usize = 28;-pub const BADGE_MAX_WIDTH: f32 = 176.0;-pub const BADGE_HEIGHT: f32 = 24.0;-pub const BADGE_CURSOR_GAP: f32 = 30.0;+pub const BADGE_MAX_WIDTH: f32 = 188.0;+pub const BADGE_HEIGHT: f32 = 28.0;+pub const BADGE_CURSOR_GAP: f32 = 25.0;const FONT_BYTES: &[u8] = include_bytes!("../assets/Inter.ttf");-const FONT_SIZE: f32 = 11.0;+const FONT_SIZE: f32 = 11.5;const HORIZONTAL_PADDING: f32 = 10.0;-const DOT_SIZE: f32 = 5.0;-const DOT_GAP: f32 = 6.0;+const ORB_SIZE: f32 = 10.0;+const ORB_GAP: f32 = 7.0;+const TEXT_OPTICAL_Y_OFFSET: f32 = 1.0;fn font() -> Option<&'static fontdue::Font> {static FONT: OnceLock<Option<fontdue::Font>> = OnceLock::new();@@ -122,6 +126,68 @@ fn rounded_rect(rect: Rect, radius: f32) -> Option<Path> {builder.finish()}+fn mixed_channel(base: u8, accent: u8, accent_weight: f32) -> u8 {+ let weight = accent_weight.clamp(0.0, 1.0);+ ((base as f32 * (1.0 - weight)) + (accent as f32 * weight)).round() as u8+}++fn session_tinted_color(+ base: [u8; 3],+ session_fill: [u8; 4],+ accent_weight: f32,+ alpha: u8,+) -> Color {+ Color::from_rgba8(+ mixed_channel(base[0], session_fill[0], accent_weight),+ mixed_channel(base[1], session_fill[1], accent_weight),+ mixed_channel(base[2], session_fill[2], accent_weight),+ alpha,+ )+}++fn paint_diffuse_glow(+ pixmap: &mut Pixmap,+ rect: Rect,+ radius: f32,+ scale: f32,+ color: [u8; 3],+ alpha: f32,+ y_offset: f32,+) {+ for step in (1..=8).rev() {+ let spread = step as f32 * scale;+ let Some(layer_rect) = Rect::from_xywh(+ rect.x() - spread,+ rect.y() - spread * 0.55 + y_offset * scale,+ rect.width() + spread * 2.0,+ rect.height() + spread * 1.1,+ ) else {+ continue;+ };+ let Some(layer) = rounded_rect(layer_rect, radius + spread) else {+ continue;+ };+ let strength = 0.35 + (8 - step) as f32 * 0.085;+ let paint = Paint {+ shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(+ color[0],+ color[1],+ color[2],+ (alpha * strength).round().clamp(0.0, 255.0) as u8,+ )),+ anti_alias: true,+ ..Default::default()+ };+ pixmap.fill_path(+ &layer,+ &paint,+ tiny_skia::FillRule::Winding,+ Transform::identity(),+ None,+ );+ }+}+/// Paint the label below the cursor without rotating it with cursor heading.pub fn paint_session_badge(pixmap: &mut Pixmap,@@ -137,7 +203,7 @@ pub fn paint_session_badge(};let scale = backing_scale.max(1.0);let font_size = FONT_SIZE * scale;- let max_text_width = (BADGE_MAX_WIDTH - HORIZONTAL_PADDING * 2.0 - DOT_SIZE - DOT_GAP) * scale;+ let max_text_width = (BADGE_MAX_WIDTH - HORIZONTAL_PADDING * 2.0 - ORB_SIZE - ORB_GAP) * scale;let mut layout = Layout::new(CoordinateSystem::PositiveYDown);layout.reset(&LayoutSettings {@@ -147,15 +213,31 @@ pub fn paint_session_badge(});layout.append(&[font], &TextStyle::new(label, font_size, 0));let glyphs = layout.glyphs();- let text_width = glyphs+ if glyphs.is_empty() {+ return;+ }+ let glyph_min_x = glyphs+ .iter()+ .map(|glyph| glyph.x)+ .fold(f32::INFINITY, f32::min);+ let glyph_max_x = glyphs.iter().map(|glyph| glyph.x + glyph.width as f32)- .fold(0.0_f32, f32::max)- .min(max_text_width);+ .fold(f32::NEG_INFINITY, f32::max);+ let glyph_min_y = glyphs+ .iter()+ .map(|glyph| glyph.y)+ .fold(f32::INFINITY, f32::min);+ let glyph_max_y = glyphs+ .iter()+ .map(|glyph| glyph.y + glyph.height as f32)+ .fold(f32::NEG_INFINITY, f32::max);+ let text_width = (glyph_max_x - glyph_min_x).min(max_text_width);+ let text_height = glyph_max_y - glyph_min_y;let badge_width =- (HORIZONTAL_PADDING * 2.0 * scale + DOT_SIZE * scale + DOT_GAP * scale + text_width)+ (HORIZONTAL_PADDING * 2.0 * scale + ORB_SIZE * scale + ORB_GAP * scale + text_width).min(BADGE_MAX_WIDTH * scale)- .max(52.0 * scale);+ .max(72.0 * scale);let badge_height = BADGE_HEIGHT * scale;let x = (cursor_x - badge_width * 0.5).clamp(2.0 * scale,@@ -168,31 +250,59 @@ pub fn paint_session_badge(let Some(rect) = Rect::from_xywh(x, y, badge_width, badge_height) else {return;};- let Some(path) = rounded_rect(rect, badge_height * 0.5) else {+ let corner_radius = badge_height * 0.5;+ let Some(path) = rounded_rect(rect, corner_radius) else {return;};let opacity = alpha_scale.clamp(0.0, 1.0);- let shadow = Paint {- shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(0, 0, 0, (76.0 * opacity) as u8)),- anti_alias: true,- ..Default::default()- };- pixmap.fill_path(- &path,- &shadow,- tiny_skia::FillRule::Winding,- Transform::from_translate(0.0, 2.0 * scale),- None,+ if opacity <= 0.0 {+ return;+ }++ paint_diffuse_glow(+ pixmap,+ rect,+ corner_radius,+ scale,+ [0, 0, 0],+ 8.0 * opacity,+ 2.0,+ );+ paint_diffuse_glow(+ pixmap,+ rect,+ corner_radius,+ scale,+ [session_fill[0], session_fill[1], session_fill[2]],+ 10.0 * opacity,+ 0.0,);+ let background_shader = LinearGradient::new(+ Point::from_xy(x, y),+ Point::from_xy(x + badge_width, y + badge_height),+ vec![+ GradientStop::new(+ 0.0,+ session_tinted_color([94, 151, 178], session_fill, 0.52, (236.0 * opacity) as u8),+ ),+ GradientStop::new(+ 0.46,+ session_tinted_color([43, 92, 119], session_fill, 0.40, (239.0 * opacity) as u8),+ ),+ GradientStop::new(+ 1.0,+ session_tinted_color([13, 27, 38], session_fill, 0.18, (245.0 * opacity) as u8),+ ),+ ],+ SpreadMode::Pad,+ Transform::identity(),+ );let background = Paint {- shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(- 18,- 22,- 28,- (224.0 * opacity) as u8,- )),+ shader: background_shader.unwrap_or_else(|| {+ tiny_skia::Shader::SolidColor(Color::from_rgba8(18, 22, 28, (242.0 * opacity) as u8))+ }),anti_alias: true,..Default::default()};@@ -208,7 +318,7 @@ pub fn paint_session_badge(255,255,255,- (92.0 * opacity) as u8,+ (160.0 * opacity) as u8,)),anti_alias: true,..Default::default()@@ -224,10 +334,11 @@ pub fn paint_session_badge(None,);- let dot_x = x + HORIZONTAL_PADDING * scale;- let dot_y = y + (badge_height - DOT_SIZE * scale) * 0.5;- if let Some(dot) = Rect::from_xywh(dot_x, dot_y, DOT_SIZE * scale, DOT_SIZE * scale) {- let dot_paint = Paint {+ let orb_center_x = x + HORIZONTAL_PADDING * scale + ORB_SIZE * scale * 0.5;+ let orb_center_y = y + badge_height * 0.5;+ if let Some(orb) = PathBuilder::from_circle(orb_center_x, orb_center_y, ORB_SIZE * scale * 0.5)+ {+ let orb_paint = Paint {shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(session_fill[0],session_fill[1],@@ -237,11 +348,38 @@ pub fn paint_session_badge(anti_alias: true,..Default::default()};- pixmap.fill_rect(dot, &dot_paint, Transform::identity(), None);+ pixmap.fill_path(+ &orb,+ &orb_paint,+ tiny_skia::FillRule::Winding,+ Transform::identity(),+ None,+ );+ let orb_outline = Paint {+ shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(+ 255,+ 255,+ 255,+ (226.0 * opacity) as u8,+ )),+ anti_alias: true,+ ..Default::default()+ };+ pixmap.stroke_path(+ &orb,+ &orb_outline,+ &Stroke {+ width: 1.5 * scale,+ ..Default::default()+ },+ Transform::identity(),+ None,+ );}- let text_origin_x = dot_x + (DOT_SIZE + DOT_GAP) * scale;- let text_origin_y = y + (badge_height - FONT_SIZE * scale) * 0.5 - 1.0 * scale;+ let text_origin_x = x + HORIZONTAL_PADDING * scale + (ORB_SIZE + ORB_GAP) * scale - glyph_min_x;+ let text_origin_y =+ y + (badge_height - text_height) * 0.5 - glyph_min_y + TEXT_OPTICAL_Y_OFFSET * scale;for glyph in glyphs {let (metrics, bitmap) = font.rasterize_config(glyph.key);if metrics.width == 0 || metrics.height == 0 {@@ -302,4 +440,40 @@ mod tests {assert!(pixmap.data().chunks_exact(4).any(|pixel| pixel[3] > 0));}}++ #[test]+ fn badge_gradient_tracks_session_color_and_zero_alpha_paints_nothing() {+ let mut blue = Pixmap::new(240, 160).unwrap();+ let mut purple = Pixmap::new(240, 160).unwrap();+ let mut hidden = Pixmap::new(240, 160).unwrap();+ paint_session_badge(+ &mut blue,+ "Research run",+ 120.0,+ 60.0,+ 1.0,+ 1.0,+ [94, 192, 232, 255],+ );+ paint_session_badge(+ &mut purple,+ "Research run",+ 120.0,+ 60.0,+ 1.0,+ 1.0,+ [178, 132, 255, 255],+ );+ paint_session_badge(+ &mut hidden,+ "Research run",+ 120.0,+ 60.0,+ 1.0,+ 0.0,+ [94, 192, 232, 255],+ );+ assert_ne!(blue.data(), purple.data());+ assert!(hidden.data().chunks_exact(4).all(|pixel| pixel[3] == 0));+ }}diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/theme.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/theme.rsindex 12c05a5542..92d5c197fb 100644--- a/libs/cua-driver/rust/crates/cursor-overlay/src/theme.rs+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/theme.rs@@ -16,7 +16,7 @@ pub const DEFAULT_THEME_ID: &str = "cua.default";pub const DEFAULT_THEME_VERSION: &str = "1.0.0";pub const THEME_PROFILE: &str = "cua-driver-full-v1";pub const CANVAS_SIZE: f32 = 128.0;-pub const DISPLAY_SIZE: f32 = 48.0;+pub const DISPLAY_SIZE: f32 = 42.0;const FLOAT_DURATION_SECS: f32 = 4.0;pub const DEFAULT_CURSOR_FILL: [u8; 4] = [94, 192, 232, 255];@@ -259,8 +259,8 @@ mod tests {}#[test]- fn default_theme_uses_compact_48_point_footprint() {- assert_eq!(DISPLAY_SIZE, 48.0);+ fn default_theme_uses_compact_42_point_footprint() {+ assert_eq!(DISPLAY_SIZE, 42.0);}#[test]diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rsindex d434ce4ae0..7d9121bebd 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs@@ -1531,10 +1531,6 @@ fn held_target_mismatch(}}-async fn overlay_glide_to(sx: f64, sy: f64) {- overlay_glide_to_for("default", sx, sy).await;-}-fn overlay_snap_to_for(cursor_id: &str, sx: f64, sy: f64, heading: Option<f64>) {crate::overlay::send_command_for(cursor_id.to_owned(),@@ -2326,6 +2322,22 @@ impl Tool for TypeTextTool {);}+ let cursor_id = resolve_cursor_key(&args);+ if let Some(idx) = resolved_elem_idx {+ crate::overlay::send_command_for(+ cursor_id.clone(),+ cursor_overlay::OverlayCommand::PinAbove(xid),+ );+ if let Ok(Ok((screen_x, screen_y))) =+ tokio::task::spawn_blocking(move || element_screen_center(pid, idx)).await+ {+ overlay_glide_to_for(&cursor_id, screen_x, screen_y).await;+ self.state+ .cursor_registry+ .update_position(&cursor_id, screen_x, screen_y);+ }+ }+let text_len = text.chars().count();// Native toolkit editables have a stronger focus-free route than raw// compositor keyboard injection. Keep Chromium/WebKit on real key events@@ -2549,21 +2561,6 @@ impl Tool for TypeTextTool {Err(e) => ToolResult::error(format!("Task error: {e}")),};}- // Pulse the agent cursor onto the field being typed into (when an- // element_index OR element_token is supplied — token resolution- // already ran above so `resolved_elem_idx` covers both).- if let Some(idx) = resolved_elem_idx {- crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(xid));- if let Ok(Ok((sx, sy))) =- tokio::task::spawn_blocking(move || element_screen_center(pid, idx)).await- {- overlay_glide_to(sx, sy).await;- crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {- x: sx,- y: sy,- });- }- }// Foreground means the caller explicitly permits activation. Chromium// and WebKitGTK can acknowledge an accessibility write without// producing the renderer input event, so web embedders use real XTest@@ -3489,6 +3486,7 @@ impl Tool for SetValueTool {"set_value requires element_index or element_token to address the target element.",),};+ let cursor_id = resolve_cursor_key(&args);let value_for_task = value.clone();// Pulse the agent cursor onto the target element before writing, so a// value write gets the same visual feedback as a click — the viewer can@@ -3499,13 +3497,12 @@ impl Tool for SetValueTool {{let window_id = args.u64_or("window_id", 0);if window_id != 0 {- crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(window_id));+ crate::overlay::send_command_for(+ cursor_id.clone(),+ cursor_overlay::OverlayCommand::PinAbove(window_id),+ );}- overlay_glide_to(sx, sy).await;- crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {- x: sx,- y: sy,- });+ overlay_glide_to_for(&cursor_id, sx, sy).await;}let result =tokio::task::spawn_blocking(move || crate::atspi::set_value(pid, idx, &value_for_task))diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rsindex db311685cb..0f9de296dc 100644--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs@@ -35,7 +35,7 @@ const DBUS_DEST: &str = "org.freedesktop.DBus";const DBUS_PATH: &str = "/org/freedesktop/DBus";const DBUS_IFACE: &str = "org.freedesktop.DBus";const BROWSER_HELPER_API_VERSION: u32 = 4;-const SEMANTIC_CURSOR_API_VERSION: u32 = 6;+const SEMANTIC_CURSOR_API_VERSION: u32 = 7;#[derive(Debug, Clone)]struct ShellWindow {@@ -627,12 +627,12 @@ mod tests {}#[test]- fn bundled_helper_v6_uses_the_compact_semantic_cursor_and_session_badge() {- assert!(EXTENSION_SOURCE.contains("GetVersion() { return 6; }"));+ fn bundled_helper_v7_uses_the_compact_semantic_cursor_and_session_badge() {+ assert!(EXTENSION_SOURCE.contains("GetVersion() { return 7; }"));assert!(EXTENSION_SOURCE.contains("SetCursorState"));assert!(EXTENSION_SOURCE.contains("SetCursorColor"));assert!(EXTENSION_SOURCE.contains("SetSessionLabel"));- assert!(EXTENSION_SOURCE.contains("const DISPLAY_SIZE = 48;"));+ assert!(EXTENSION_SOURCE.contains("const DISPLAY_SIZE = 42;"));assert!(EXTENSION_SOURCE.contains("const GLOW_PADDING = 24;"));assert!(EXTENSION_SOURCE.contains("function drawCursorGlowShape"));assert!(EXTENSION_SOURCE.contains("function glowPath"));@@ -641,7 +641,7 @@ mod tests {assert!(EXTENSION_SOURCE.contains("width - 1"));assert!(EXTENSION_SOURCE.contains("createGlowSurface(this._fillColor)"));assert!(EXTENSION_SOURCE.contains("cr.translate(-GLOW_PADDING, -GLOW_PADDING);"));- assert!(EXTENSION_METADATA.contains("\"version\":6"));+ assert!(EXTENSION_METADATA.contains("\"version\":7"));for action in ["idle", "observe", "click", "drag", "scroll", "text", "key", "navigate", "app",diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rsindex 5cc5f550ee..3617c67e3b 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs@@ -158,6 +158,22 @@ impl Tool for SetValueTool {}};let element_ptr = element_guard.as_ptr();+ let cursor_key = super::cursor_tools::resolve_cursor_key(&args);+ let center_ptr = element_ptr as usize;+ if let Ok(Some((screen_x, screen_y))) = tokio::task::spawn_blocking(move || unsafe {+ crate::ax::bindings::element_screen_center(center_ptr as AXUIElementRef)+ })+ .await+ {+ crate::cursor::overlay::send_command(+ cursor_key.clone(),+ cursor_overlay::OverlayCommand::PinAbove(window_id as u64),+ );+ crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), screen_x, screen_y).await;+ self.state+ .cursor_registry+ .update_position(&cursor_key, screen_x, screen_y);+ }// An AXValue read-back is not ground truth for web content. Chromium,// WebKit, and Electron can echo the write through accessibility while// the renderer never observes it. Reuse type_text's bounded ancestordiff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rsindex d40cf4aa0e..d74f5df922 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs@@ -260,6 +260,25 @@ impl Tool for TypeTextTool {} else {None};+ if let (Some((element, _)), Some(wid)) = (element_guard.as_ref(), window_id) {+ let center_ptr = element.as_ptr() as usize;+ if let Ok(Some((screen_x, screen_y))) = tokio::task::spawn_blocking(move || unsafe {+ crate::ax::bindings::element_screen_center(center_ptr as AXUIElementRef)+ })+ .await+ {+ let cursor_key = super::cursor_tools::resolve_cursor_key(&args);+ crate::cursor::overlay::send_command(+ cursor_key.clone(),+ cursor_overlay::OverlayCommand::PinAbove(wid as u64),+ );+ crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), screen_x, screen_y)+ .await;+ self.state+ .cursor_registry+ .update_position(&cursor_key, screen_x, screen_y);+ }+ }let element_ptr = element_guard.as_ref().map(|(g, idx)| (g.as_ptr(), Some(*idx)));diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rsindex b599b94d05..379b79b7be 100644--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs@@ -3629,6 +3629,19 @@ impl Tool for TypeTextTool {}}};+ if let Some(idx) = elem_idx {+ if let Some((cx, cy)) =+ self.state+ .element_cache+ .get_element_center(pid, hwnd, idx as usize)+ {+ pin_overlay_above(&cursor_key, hwnd);+ overlay_glide_to(&cursor_key, cx as f64, cy as f64).await;+ self.state+ .cursor_registry+ .update_position(&cursor_key, cx as f64, cy as f64);+ }+ }let text_len = text.chars().count();// Native ConsoleHost can accept every synthesized Unicode event whilediff --git a/libs/cua-driver/scripts/cursor-gallery.sh b/libs/cua-driver/scripts/cursor-gallery.shindex f4cd46c631..7241beba08 100755--- a/libs/cua-driver/scripts/cursor-gallery.sh+++ b/libs/cua-driver/scripts/cursor-gallery.sh@@ -28,7 +28,7 @@ export_assets() {--features theme-authoring \-- "$FRAMES_DIR"- for group in actions modifiers combined; do+ for group in actions modifiers combined session; domkdir -p "$GENERATED_DIR/$group"for state_dir in "$FRAMES_DIR/$group"/*; dostate=$(basename "$state_dir")diff --git a/libs/cua-driver/tools/cursor-gallery/index.html b/libs/cua-driver/tools/cursor-gallery/index.htmlindex 3e32e951c5..bb54af3dc0 100644--- a/libs/cua-driver/tools/cursor-gallery/index.html+++ b/libs/cua-driver/tools/cursor-gallery/index.html@@ -78,11 +78,35 @@<span>□ Pixel target</span></div></article>++ <article class="combined-card">+ <div>+ <p class="section-count">Session identity</p>+ <h3>Compact label on cursor reveal</h3>+ <p>The session-tinted badge holds for two seconds, then fades while the cursor remains.</p>+ </div>+ <div class="demo">+ <video+ class="cursor-video"+ src="./generated/session/badge.webm"+ data-group="session"+ data-state="badge"+ autoplay+ loop+ muted+ playsinline+ ></video>+ </div>+ <div class="legend">+ <span>● Circular session marker</span>+ <span>◒ Session-accent gradient</span>+ </div>+ </article></section><footer><span>Preview scale is enlarged for review.</span>- <span>Production cursor size remains 48 points.</span>+ <span>Production cursor size is 42 points.</span></footer></main><script src="./app.js"></script>diff --git a/libs/cua-driver/tools/cursor-gallery/styles.css b/libs/cua-driver/tools/cursor-gallery/styles.cssindex 4dc33563c8..ba7c89dfbd 100644--- a/libs/cua-driver/tools/cursor-gallery/styles.css+++ b/libs/cua-driver/tools/cursor-gallery/styles.css@@ -175,7 +175,7 @@ h1 {/** Renderer frames preserve the full 256 px transparent capture canvas.* The frame exporter renders at 1.5× backing scale. Scale its centered- * 164 px layout box to preserve that 72 px review size after the 256 px+ * 164 px layout box to preserve the enlarged review size after the 256 px* capture canvas is fitted into the card.*/width: 164px;diff --git a/libs/cua-driver/wayland-helper/winrects@cua/extension.js b/libs/cua-driver/wayland-helper/winrects@cua/extension.jsindex c3ed8c1780..444a6c9036 100644--- a/libs/cua-driver/wayland-helper/winrects@cua/extension.js+++ b/libs/cua-driver/wayland-helper/winrects@cua/extension.js@@ -25,14 +25,16 @@ const IFACE = `<node><interface name="org.cua.WinRects">// GNOME cannot host the Rust renderer directly, so this Shell actor mirrors// the embedded cua.default vector theme and semantic state vocabulary.const CANVAS_SIZE = 128;-const DISPLAY_SIZE = 48;+const DISPLAY_SIZE = 42;const ACTOR_SIZE = 112;const ACTOR_CENTER = ACTOR_SIZE / 2;const SCALE = DISPLAY_SIZE / CANVAS_SIZE;const FLOAT_DURATION = 4.0;const GLOW_SURFACE_SCALE = 3;const GLOW_PADDING = 24;-const BADGE_Y_OFFSET = 34;+const BADGE_Y_OFFSET = 29;+const BADGE_HOLD_SECONDS = 2.0;+const BADGE_FADE_SECONDS = 0.4;const ACTIONS = new Set(['idle', 'observe', 'click', 'drag', 'scroll', 'text','key', 'navigate', 'app', 'transfer', 'record', 'system',@@ -57,6 +59,27 @@ function nowSeconds() {return GLib.get_monotonic_time() / 1_000_000;}+function mixChannel(base, accent, weight) {+ return Math.round(base * (1 - weight) + accent * weight);+}++function badgeStyle(fillColor) {+ const start = fillColor.map((channel, index) =>+ mixChannel([94, 151, 178][index], channel, 0.52));+ const end = fillColor.map((channel, index) =>+ mixChannel([13, 27, 38][index], channel, 0.18));+ return [+ 'spacing: 7px',+ 'padding: 6px 10px',+ `background-gradient-start: rgba(${start[0]}, ${start[1]}, ${start[2]}, 0.93)`,+ `background-gradient-end: rgba(${end[0]}, ${end[1]}, ${end[2]}, 0.96)`,+ 'background-gradient-direction: horizontal',+ 'border: 1px solid rgba(255, 255, 255, 0.63)',+ 'border-radius: 14px',+ 'box-shadow: 0 2px 9px rgba(0, 0, 0, 0.30)',+ ].join(';');+}+function easeInOut(value) {const t = Math.max(0, Math.min(1, value));return t * t * (3 - 2 * t);@@ -476,9 +499,9 @@ export default class WinRectsExtension extends Extension {this._cursor.set_pivot_point(0.5, 0.5);Main.layoutManager.addTopChrome(this._cursor);this._badgeDot = new St.Widget({- width: 5,- height: 5,- style: `background-color: ${this._fillColorCss}; border-radius: 3px;`,+ width: 10,+ height: 10,+ style: `background-color: ${this._fillColorCss}; border: 1px solid white; border-radius: 99px;`,y_align: Clutter.ActorAlign.CENTER,});this._badgeLabel = new St.Label({@@ -490,20 +513,14 @@ export default class WinRectsExtension extends Extension {visible: false,reactive: false,can_focus: false,- style: [- 'spacing: 6px',- 'padding: 5px 10px',- 'background-color: rgba(18, 22, 28, 0.88)',- 'border: 1px solid rgba(255, 255, 255, 0.36)',- 'border-radius: 12px',- 'box-shadow: 0 2px 8px rgba(0, 0, 0, 0.30)',- ].join(';'),+ style: badgeStyle(this._fillColor),});this._badge.add_child(this._badgeDot);this._badge.add_child(this._badgeLabel);Main.layoutManager.addTopChrome(this._badge);this._cursorX = null;this._cursorY = null;+ this._badgeRevealedAt = null;this._frameId = GLib.timeout_add(GLib.PRIORITY_DEFAULT, 33, () => {if (!this._cursor)return GLib.SOURCE_REMOVE;@@ -515,6 +532,18 @@ export default class WinRectsExtension extends Extension {this._setCursorState('idle', '', '');if (this._cursor.visible)this._cursor.queue_repaint();+ if (this._badge?.visible && this._badgeRevealedAt !== null) {+ const badgeElapsed = now - this._badgeRevealedAt;+ if (badgeElapsed <= BADGE_HOLD_SECONDS) {+ this._badge.opacity = 255;+ } else if (badgeElapsed < BADGE_HOLD_SECONDS + BADGE_FADE_SECONDS) {+ const fade = (badgeElapsed - BADGE_HOLD_SECONDS) / BADGE_FADE_SECONDS;+ this._badge.opacity = Math.round(255 * (1 - fade));+ } else {+ this._badge.hide();+ this._badgeRevealedAt = null;+ }+ }return GLib.SOURCE_CONTINUE;});}@@ -534,7 +563,7 @@ export default class WinRectsExtension extends Extension {if (this._impl) { this._impl.unexport(); this._impl = null; }if (this._nameId) { Gio.bus_unown_name(this._nameId); this._nameId = 0; }}- GetVersion() { return 6; }+ GetVersion() { return 7; }GetRects() {const actors = global.get_window_actors();const actorByWindow = new Map();@@ -613,6 +642,7 @@ export default class WinRectsExtension extends Extension {}MoveCursor(x, y) {if (!this._cursor) return;+ const revealBadge = !this._cursor.visible;this._cursorX = x;this._cursorY = y;this._cursor.show();@@ -623,16 +653,22 @@ export default class WinRectsExtension extends Extension {mode: Clutter.AnimationMode.EASE_OUT_CUBIC,});this._positionBadge(x, y, 480);+ if (revealBadge)+ this._revealBadge();}ClickPulse(x, y) {if (!this._cursor) return;+ const revealBadge = !this._cursor.visible;this._cursorX = x;this._cursorY = y;this._cursor.set_position(x - ACTOR_CENTER, y - ACTOR_CENTER);this._positionBadge(x, y, 0);- this._setCursorState('click', this._delivery, this._target);+ if (['idle', 'navigate', 'click'].includes(this._action))+ this._setCursorState('click', this._delivery, this._target);this._cursor.show();this._cursor.queue_repaint();+ if (revealBadge)+ this._revealBadge();}SetCursorColor(fillColor) {const match = /^#([0-9a-fA-F]{6})$/.exec(fillColor);@@ -643,8 +679,10 @@ export default class WinRectsExtension extends Extension {this._fillColorCss = `#${match[1].toLowerCase()}`;if (this._badgeDot)this._badgeDot.set_style(- `background-color: ${this._fillColorCss}; border-radius: 3px;`+ `background-color: ${this._fillColorCss}; border: 1px solid white; border-radius: 99px;`);+ if (this._badge)+ this._badge.set_style(badgeStyle(this._fillColor));if (this._glowSurface)this._glowSurface.finish();this._glowSurface = createGlowSurface(this._fillColor);@@ -666,10 +704,13 @@ export default class WinRectsExtension extends Extension {this._badgeLabel.set_text(label);if (label.length === 0 || !this._cursor?.visible) {this._badge.hide();+ this._badgeRevealedAt = null;return;}- if (this._cursorX !== null && this._cursorY !== null)+ if (this._cursorX !== null && this._cursorY !== null) {this._positionBadge(this._cursorX, this._cursorY, 0);+ this._revealBadge();+ }}_positionBadge(x, y, duration) {if (!this._badge || !this._badgeLabel ||@@ -678,7 +719,6 @@ export default class WinRectsExtension extends Extension {const [, naturalWidth] = this._badge.get_preferred_width(-1);const badgeX = Math.round(x - naturalWidth / 2);const badgeY = Math.round(y + BADGE_Y_OFFSET);- this._badge.show();if (duration > 0) {this._badge.ease({x: badgeX,@@ -690,6 +730,14 @@ export default class WinRectsExtension extends Extension {this._badge.set_position(badgeX, badgeY);}}+ _revealBadge() {+ if (!this._badge || !this._badgeLabel ||+ this._badgeLabel.get_text().length === 0)+ return;+ this._badgeRevealedAt = nowSeconds();+ this._badge.opacity = 255;+ this._badge.show();+ }_setCursorState(action, delivery, target) {this._action = ACTIONS.has(action) ? action : 'idle';this._delivery = delivery ?? '';@@ -702,5 +750,6 @@ export default class WinRectsExtension extends Extension {HideCursor() {if (this._cursor) this._cursor.hide();if (this._badge) this._badge.hide();+ this._badgeRevealedAt = null;}}diff --git a/libs/cua-driver/wayland-helper/winrects@cua/metadata.json b/libs/cua-driver/wayland-helper/winrects@cua/metadata.jsonindex 01b00c335e..434b8a5eae 100644--- a/libs/cua-driver/wayland-helper/winrects@cua/metadata.json+++ b/libs/cua-driver/wayland-helper/winrects@cua/metadata.json@@ -1 +1 @@-{"uuid":"winrects@cua","name":"WinRects","description":"Expose window geometry, capture, cursor, and verified activation for cua-driver on Wayland.","shell-version":["45","46","47","48","49","50"],"version":6}+{"uuid":"winrects@cua","name":"WinRects","description":"Expose window geometry, capture, cursor, and verified activation for cua-driver on Wayland.","shell-version":["45","46","47","48","49","50"],"version":7}