Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(cua-driver): refine session cursor feedback (#2665)

Francesco Bonaccid366af084e53
diff --git a/libs/cua-driver/docs/0.13.2-overnight-execution-journal.md b/libs/cua-driver/docs/0.13.2-overnight-execution-journal.md
new file mode 100644
index 0000000000..44d74dc83c
--- /dev/null
+++ b/libs/cua-driver/docs/0.13.2-overnight-execution-journal.md
@@ -0,0 +1,67 @@
+# Cua Driver 0.13.2 Overnight Execution Journal
+
+**Status:** Active
+**Started from:** `origin/main` at `1e77ab5536eb88ca4cb7d3b6cc1fb51eea03f0e4`
+**Released baseline:** 0.13.1
+**Release boundary:** No release, tag, deployment, or package publication
+
+## Confirmed scope
+
+- Reduce the default cursor size.
+- Fade the session-name badge after a short reveal.
+- Match the production badge to the session-accent gradient shown by the web
+ harness.
+- Replace the square badge marker with a circular session dot.
+- Audit every public action and fix spatial actions whose agent cursor does not
+ move to the resolved target.
+- Complete and merge PR #2660 after exact-head macOS evidence.
+- Complete the prioritized post-0.13.1 stabilization plan.
+- Merge validated work into `main`.
+- Run macOS Lume and Windows and Linux GitHub Actions validation.
+
+## Baseline facts
+
+- The original worktree contains existing user changes and remains untouched.
+- The execution worktree is clean apart from this plan and journal.
+- The baseline default cursor display size was 48 points.
+- The baseline production badge used a solid dark background.
+- The baseline production badge marker was painted with `fill_rect`, so it was
+ square.
+- Semantic action events are emitted centrally.
+- Spatial cursor motion is implemented in platform tool paths, not by the
+ central semantic event classifier.
+
+## Work log
+
+### Phase 1: cursor audit
+
+- Reduced the shared native and GNOME cursor footprint from 48 to 42 logical
+ points.
+- Added an independent two-second badge hold and 400 millisecond fade clock.
+- Ported the session-accent gradient badge into the production renderer and
+ replaced the rectangular marker with a circular session orb.
+- Updated the GNOME Shell renderer to API version 7 with the same size, badge,
+ and fade behavior.
+- Preserved the active semantic state when a spatial glide or click pulse is
+ dispatched. Text, scroll, drag, and other states are no longer overwritten
+ by `navigate` or `click`.
+- Added indexed and token-addressed text movement on macOS and Windows.
+- Corrected Linux text and value movement to use the declared session cursor
+ rather than the anonymous default cursor, and moved indexed text before
+ early native and compositor delivery returns.
+- Corrected token-addressed cursor classification to use the AX modifier.
+- Added the production session badge and fade to the repository cursor gallery.
+
+## Evidence ledger
+
+| SHA | Platform | Surface | Result | Evidence |
+|---|---|---|---|---|
+| Working tree | Local | Cursor contract and renderer | 32 renderer and 3 contract tests passing before the latest platform movement patch | Local cargo output |
+
+## Merged PRs
+
+- None yet.
+
+## Current blockers
+
+- None.
diff --git a/libs/cua-driver/docs/0.13.2-overnight-stabilization-plan.md b/libs/cua-driver/docs/0.13.2-overnight-stabilization-plan.md
new file mode 100644
index 0000000000..01037843d1
--- /dev/null
+++ b/libs/cua-driver/docs/0.13.2-overnight-stabilization-plan.md
@@ -0,0 +1,385 @@
+# Cua Driver 0.13.2 Overnight Stabilization Plan
+
+**Status:** Draft for review
+**Baseline:** Cua Driver 0.13.1 is already released
+**Source baseline:** `origin/main` at `1e77ab5536eb88ca4cb7d3b6cc1fb51eea03f0e4`
+**Target:** Merge validated post-0.13.1 work into `main`
+**Release boundary:** Do not create, publish, promote, or merge a 0.13.2 release. Francesco owns the 0.13.2 release tomorrow.
+
+## 1. Goal
+
+Leave `main` ready for a 0.13.2 release by:
+
+1. making the default cursor smaller and correcting its session badge;
+2. auditing every spatial action so the cursor moves to the action target;
+3. landing the hermetic macOS certification work in PR [#2660](https://github.com/trycua/cua/pull/2660);
+4. resolving the highest-impact regressions discovered around 0.13.1;
+5. restoring the intended release automation without triggering a release;
+6. validating the exact merged `main` SHA on macOS, Windows, and Linux;
+7. updating public docs and post-install guidance for any behavior that changes;
+8. preserving reviewable evidence, including videos for GUI behavior.
+
+This is stabilization and correctness work. It is not a release operation.
+
+## 2. Definition of done
+
+The overnight work is complete only when:
+
+- every accepted product or workflow fix is merged into `main` through a focused PR;
+- the production cursor is smaller on macOS, Windows, Linux X11, and Linux Wayland;
+- the session badge fades away after a short reveal, uses the promised
+ session-accent gradient, and has a circular session dot;
+- every supported spatial action either moves the agent cursor to its resolved
+ target or documents why the action has no spatial target;
+- the spatial-action audit has contract tests and representative E2E evidence;
+- PR #2660 is synchronized with current `main`, reviewed, fully validated, and merged;
+- issue #2655 has two consecutive same-SHA macOS certification runs that do not depend on stale worker state;
+- canonical Cua Driver installers resolve and install the released 0.13.1 artifacts before any source-built validation begins;
+- the final merged `main` SHA passes the relevant deterministic Linux and Windows CI;
+- representative desktop E2E evidence exists for macOS, Windows, Linux X11, and Linux Wayland where the changed behavior applies;
+- every GUI success is proven by an external effect, not only `ok: true`;
+- background claims retain focus, z-order, real-cursor, occlusion, and input-isolation evidence;
+- public docs, generated references, examples, and post-install instructions match the final behavior;
+- PR titles and labels accurately describe release impact and pass release-metadata validation;
+- no release, tag, package publication, deployment, or Release Please PR merge occurs;
+- the execution journal states the final `main` SHA, merged PRs, exact test results, artifacts, limitations, and remaining owner.
+
+## 3. Operating boundaries
+
+- Start every new unit of work from freshly fetched `origin/main`.
+- Preserve the existing dirty worktree. Use clean branches or worktrees for all implementation.
+- Keep each independent regression in its own focused PR.
+- Merge only after the PR's exact head has the required tests and evidence.
+- Do not bypass a required human review or a protected branch rule.
+- Do not overlap with independent backlog or contributor-PR review work.
+- Preserve external contributor authorship when adapting submitted work.
+- Do not introduce a breaking public API, CLI, SDK, protocol, or configuration change without asking first.
+- Do not weaken product assertions or relabel a behavior failure as an environment failure.
+- Keep public artifacts free of credentials, private infrastructure details, machine-specific paths, and raw session data.
+- Use a living execution journal at `libs/cua-driver/docs/0.13.2-overnight-execution-journal.md`.
+
+## 4. Priority and landing order
+
+### P0. Correct the cursor size, session badge, and spatial action motion
+
+Confirmed product requirements:
+
+1. Make the default cursor visibly smaller while preserving legibility on
+ light and dark backgrounds.
+2. Reveal the session-name badge with the pointer, then fade only the badge
+ away after a short readable interval. Cursor visibility and idle behavior
+ remain independent.
+3. Match the production badge to the repository web harness:
+ - session-accent background gradient;
+ - circular session-color dot;
+ - centered label;
+ - compact spacing and cursor gap;
+ - consistent backing-scale behavior.
+4. Audit every public action and fix spatial actions that update the semantic
+ icon without moving the agent cursor to their resolved target.
+
+Implementation work:
+
+1. Treat the native Skia renderer as the production source of truth and the
+ web harness as a visual contract and review surface.
+2. Add a renderer-owned badge reveal clock with explicit hold and fade
+ durations. Reset it when the session label is first set and whenever a new
+ session becomes visibly active.
+3. Keep the fade deterministic, independent per session, and disabled only
+ when the badge itself is absent.
+4. Render the badge accent with the session fill and bounded light and dark
+ variants. Do not accept an agent-controlled arbitrary badge color.
+5. Render the dot through a circular path, not a square rectangle.
+6. Reduce the one canonical production display size and update all tests,
+ docs, gallery exports, and platform bounds that depend on it.
+7. Create a complete action-motion inventory with:
+ - public tool;
+ - semantic action;
+ - target source;
+ - expected movement;
+ - platform implementation;
+ - external E2E oracle.
+8. For actions with resolved coordinates or element bounds, move the agent
+ cursor before dispatch while preserving the real pointer for background
+ delivery.
+9. For actions without a spatial target, keep the cursor at its current
+ location and animate the semantic state there.
+10. At minimum, audit and test:
+ - click, double click, right click;
+ - drag;
+ - scroll;
+ - type text and set value;
+ - press key and hotkey;
+ - browser click, drag, scroll, type, fill, press key, and hotkey;
+ - navigation, app, transfer, record, system, and observe actions.
+11. Assert that semantic begin and end events are emitted for every classified
+ public tool with an explicit session.
+12. Add platform-focused tests that prove spatial actions send the correct
+ keyed move command before action dispatch.
+13. Run the repository cursor gallery and record the complete action and
+ modifier sequence.
+14. Run real Cua Driver action sequences on macOS, Windows, Linux X11, and
+ Linux Wayland. Verify application effects independently and record videos.
+15. Update cursor personalization, action catalog, session, and post-install
+ documentation.
+
+### P0. Make macOS certification hermetic
+
+Tracking:
+
+- Issue [#2655](https://github.com/trycua/cua/issues/2655)
+- PR [#2660](https://github.com/trycua/cua/pull/2660)
+
+Work:
+
+1. Rebase or merge the latest `origin/main` into the PR branch without losing its existing evidence.
+2. Re-run shell syntax, ShellCheck, focused script tests, Rust formatting, and all changed-path CI.
+3. Preflight the macOS Lume host and guest:
+ - adequate host and guest disk space;
+ - logged-in GUI session;
+ - exact source SHA installed;
+ - correct TCC identity and grants;
+ - no stale daemon, socket, fixture, Cargo target, or build-cache ownership;
+ - sufficient evidence output space.
+4. Run the canonical full macOS certification matrix twice consecutively at the same exact PR SHA.
+5. Require each run to:
+ - create a fresh run-owned build namespace;
+ - own daemon startup, permission mode, socket, watchdog, shutdown, and restoration;
+ - execute every declared row;
+ - preserve the initial attempt and any allowed exact-cell retry;
+ - record row-level screenshots, logs, videos, and external oracles;
+ - restore the normal daemon after success or failure.
+6. If a run fails, classify it before changing anything:
+ - product failure: fix product behavior and restart exact-head validation;
+ - harness failure: fix the harness without weakening assertions;
+ - environment failure: prove it through preflight or independent system evidence and retry in a repaired representative environment.
+7. Update the PR body with the final exact SHA and both consecutive run links.
+8. Confirm the `test(cua-driver)` title and `no-release` label remain accurate.
+9. Merge after required review and checks pass.
+10. Verify issue #2655 is closed by the merged PR.
+
+### P0. Verify the shipped installer baseline
+
+Tracking:
+
+- Issue [#2654](https://github.com/trycua/cua/issues/2654)
+
+Work:
+
+1. Verify the canonical Unix and Windows installers resolve the released 0.13.1 component tag.
+2. Verify the expected macOS, Linux, and Windows assets exist under that exact component release.
+3. Exercise a clean install and an upgrade from the prior supported version on each platform.
+4. Verify:
+ - installed executable path;
+ - `cua-driver --version`;
+ - source or release provenance;
+ - daemon start and stop;
+ - one representative MCP smoke test.
+5. If the 0.13.1 release already resolves the reported 0.13.0 404, record the evidence and close #2654 as resolved.
+6. If any canonical path still fails, fix it in a focused PR and rerun all installer-contract tests before merging.
+
+### P0. Restore safe automatic publication for future releases
+
+Tracking:
+
+- Issue [#2662](https://github.com/trycua/cua/issues/2662)
+
+Work:
+
+1. Restore the intended normal release path after an approved Release Please PR merge:
+ - immutable component tag;
+ - required build and verification dependencies;
+ - GitHub release publication;
+ - version-locked PyPI and npm SDK publication.
+2. Keep PR comment and checkbox controls unable to bump or publish packages.
+3. Preserve manual recovery dispatch without making it the normal path.
+4. Add deterministic workflow tests for:
+ - valid component tag push;
+ - required build failure;
+ - tag and SHA mismatch;
+ - disabled PR comment or checkbox path;
+ - explicit recovery dispatch.
+5. Validate the workflow without creating a tag, release, or package publication.
+6. Merge the focused workflow PR only after release-metadata and workflow tests pass.
+
+### P0. Fix silent Linux process-control success
+
+Tracking:
+
+- Issue [#2661](https://github.com/trycua/cua/issues/2661)
+
+Work:
+
+1. Reproduce `kill_app` reporting success while the process survives.
+2. Reproduce `launch_app.launch_path` leaving a shell child hung without executing.
+3. Trace process identity, signal delivery, child state, pipes, process groups, wait behavior, and timeout handling.
+4. Change the contract so:
+ - `kill_app` verifies the target exited or returns a structured failure;
+ - PID reuse cannot produce a false success;
+ - `launch_path` drains or redirects child pipes safely;
+ - launch completion has a bounded, observable condition;
+ - abandoned children are reaped;
+ - unsupported sandbox behavior fails explicitly.
+5. Add deterministic unit and integration coverage.
+6. Run representative Linux X11 and Wayland E2E.
+7. Run a gVisor-equivalent proof when an existing authorized environment is available. Otherwise, state that gVisor remains unproven and do not claim the reported platform fixed.
+8. Merge only when the product cannot silently report a process-control success with no effect.
+
+### P1. Stop Cua Driver from launching a screen reader on COSMIC
+
+Tracking:
+
+- Issue [#2630](https://github.com/trycua/cua/issues/2630)
+
+Work:
+
+1. Confirm the current default behavior from source and a representative Linux session.
+2. Prefer the safe default for unknown desktops: enable generic AT-SPI inspection without claiming that a screen reader is active.
+3. Preserve an explicit override for environments that genuinely require full advertisement.
+4. Add regression tests for GNOME, COSMIC, known non-GNOME desktops, mixed desktop identifiers, and unknown desktops.
+5. Verify on Linux Wayland that:
+ - Orca and Speech Dispatcher do not start;
+ - accessibility inspection still works;
+ - a representative action still completes.
+6. Update Linux troubleshooting and accessibility documentation if the override or default behavior changes.
+7. Merge the focused fix after exact-head Linux validation.
+
+### P1. Improve drag cursor animation
+
+Tracking:
+
+- Issue [#2625](https://github.com/trycua/cua/issues/2625)
+
+Work:
+
+1. Animate drag as a continuous point-to-point gesture rather than a standard cursor movement.
+2. Keep the production cursor compact, session-colored, legible on light and dark backgrounds, and consistent across platforms.
+3. Preserve the existing semantic action and delivery modifiers.
+4. Add deterministic rendering or timeline tests for:
+ - drag start;
+ - continuous travel;
+ - destination arrival;
+ - release;
+ - cancellation or refusal.
+5. Validate at least one foreground and one supported background drag path per representative platform.
+6. Record short macOS, Windows, Linux X11, and Linux Wayland videos where supported.
+7. Update cursor personalization documentation and the repository cursor preview harness.
+8. Keep this change separate from correctness and release-workflow fixes.
+
+## 5. Cross-platform validation matrix
+
+| Platform | Environment | Minimum proof |
+|---|---|---|
+| macOS | Logged-in Lume macOS VM with exact source SHA and TCC grants | Two consecutive full certification runs for #2660, installer smoke, affected product rows, screenshots, logs, and videos |
+| Windows | GitHub Actions for deterministic coverage, interactive Azure VM when desktop behavior changes | Install or upgrade, daemon and MCP smoke, affected GUI rows, external oracles, and video |
+| Linux X11 | GitHub Actions when representative, otherwise interactive Azure VM | Install or upgrade, daemon and MCP smoke, affected action rows, focus and input guards, and video |
+| Linux Wayland | Representative GNOME or supported compositor session in Actions or Azure | Portal and accessibility preflight, affected action rows, external oracles, and video |
+| Linux COSMIC | Representative session if available | No Orca launch, AT-SPI remains usable, one successful action |
+| Linux gVisor | Existing authorized sandbox environment if available | Process exit and launch effects independently verified, with no silent success |
+
+Hosted CI proves deterministic build and unit contracts. It does not substitute for a real interactive desktop when the claim depends on focus, z-order, TCC, compositor routing, or visible cursor behavior.
+
+## 6. Documentation and examples audit
+
+After the product PRs are stable:
+
+1. Audit public Cua Driver tutorials, how-to guides, reference pages, generated tool schemas, examples, and post-install output.
+2. Verify that:
+ - 0.13.1 is described as the installed baseline until 0.13.2 is released;
+ - no page claims 0.13.2 is available;
+ - installer instructions use canonical component resolution;
+ - permission and session examples match the shipped contract;
+ - cursor customization documents required states and the drag timeline;
+ - Linux accessibility guidance explains the safe advertisement default and override;
+ - process-control errors and timeouts are documented where user action is possible.
+3. Regenerate checked-in references with repository-owned generators.
+4. Run link, formatting, generated-output, and sensitive-information scans.
+5. Keep internal execution notes out of public-facing pages.
+
+## 7. PR and merge discipline
+
+Use one focused PR for each independent unit:
+
+1. cursor size, badge, and action-motion correctness
+2. #2660 macOS certification harness
+3. #2662 release automation
+4. #2661 Linux process control
+5. #2630 COSMIC accessibility advertisement
+6. #2625 drag cursor animation, unless the confirmed cursor PR fully supersedes it
+7. any documentation-only reconciliation that cannot live accurately with its product PR
+
+For every PR:
+
+- synchronize with the latest `origin/main` immediately before final validation;
+- inspect the final changed files and live PR title;
+- preserve contributor credit;
+- run the exact relevant unit, integration, generated-output, and platform checks;
+- attach exact-SHA evidence;
+- address review feedback;
+- wait for required checks;
+- merge only after the completion bar is satisfied;
+- verify `main` contains the merge before starting dependent validation.
+
+Do not combine unrelated fixes merely to reduce PR count.
+
+## 8. Final merged-main certification
+
+After all accepted PRs merge:
+
+1. Fetch the final `origin/main` and record its exact SHA.
+2. Run deterministic Rust, script, installer-contract, release-metadata, docs, Python SDK, and TypeScript SDK checks affected by the work.
+3. Run the representative macOS, Windows, Linux X11, and Linux Wayland E2E cells affected by the merged changes.
+4. Verify canonical 0.13.1 installation still works. Source-built tests must report their source SHA and must not be confused with the released binary.
+5. Verify no tag, GitHub release, PyPI version, npm version, or installer baked version was changed to 0.13.2.
+6. Publish a final internal readiness report containing:
+ - merged PRs and issues;
+ - final `main` SHA;
+ - per-platform results;
+ - videos and run links;
+ - supported, refused, unavailable, and unproven cells;
+ - exact remaining work owned by Francesco for tomorrow's 0.13.2 release.
+
+## 9. Stop and escalation conditions
+
+Continue autonomously through ordinary implementation choices, test failures, flaky infrastructure repair, documentation changes, and PR feedback.
+
+Stop and ask only if:
+
+- a public contract requires a breaking change;
+- a required representative environment needs new paid infrastructure or credentials;
+- a security vulnerability is discovered;
+- a destructive or irreversible external action is required;
+- a release, tag, deployment, or package publication would occur;
+- a protected branch or required human review prevents an otherwise ready merge.
+
+An isolated failed run, a VM crash, a full disk, a CI timeout, or a different recoverable defect is not a reason to stop. Preserve evidence, repair the environment or implementation, and retry.
+
+## 10. Explicit exclusions
+
+- Creating or publishing 0.13.2
+- Merging a 0.13.2 Release Please PR
+- General backlog cleanup
+- Reviewing unrelated contributor PRs
+- Refactoring the daemonless architecture without a regression requiring it
+- Expanding permission policy beyond what a reproduced issue requires
+- Changing public contracts without explicit approval
+
+## 11. Additional GitHub work after confirmed scope
+
+After all confirmed P0 and P1 work passes and merges:
+
+1. Refresh open Cua Driver issues and PRs from current GitHub state.
+2. Exclude work already owned by another active task or contributor review.
+3. Rank candidates by:
+ - silent false success or data-loss risk;
+ - regression affecting released 0.13.1 behavior;
+ - installer or release blocker for 0.13.2;
+ - cross-platform user impact;
+ - bounded implementation and representative testability.
+4. Prefer superseding a stale PR only when:
+ - its useful behavior is still needed;
+ - direct landing is no longer practical;
+ - authorship can be preserved through cherry-pick or coauthor credit;
+ - parity is proven before closing the source PR.
+5. Take only work that can be implemented, tested, reviewed, and merged without
+ weakening the final merged-main certification.
diff --git a/libs/cua-driver/docs/cursor-themes.md b/libs/cua-driver/docs/cursor-themes.md
index 125535b9dd..1e05aac380 100644
--- a/libs/cua-driver/docs/cursor-themes.md
+++ b/libs/cua-driver/docs/cursor-themes.md
@@ -16,12 +16,21 @@ uses Cua blue. Named sessions receive a stable fill
from the built-in session palette, so concurrent agents remain visually
distinct.
-The native renderer places the sanitized public session name in a badge below
-the built-in pointer. The badge follows the cursor and uses the same session
-color, but it is not part of the dotLottie theme artifact. Custom theme authors
+The native renderer places the sanitized public session name in a compact badge
+below the built-in pointer. The badge uses a session-accent gradient and a
+circular session marker. It appears when the cursor is first revealed, remains
+fully visible for two seconds, then fades over 400 milliseconds while the
+cursor stays available. Hiding and revealing the cursor shows the badge again.
+The badge is not part of the dotLottie theme artifact, so custom theme authors
do not need to provide badge artwork or a font. Cua Driver strips control
-characters, collapses whitespace, and shortens labels longer than 28
-characters before rendering the bundled Inter typeface.
+characters, collapses whitespace, and shortens labels longer than 28 characters
+before rendering the bundled Inter typeface.
+
+The production pointer footprint is 42 logical points. Actions with a resolved
+screen target, including indexed or token-addressed text and value operations,
+move the agent cursor to that target before dispatch. Actions without a spatial
+target, such as typing into the already focused desktop application, keep the
+cursor at its current location and only change the semantic animation.
The cursor is a visual aid for people watching an agent. It is not a security
indicator, an authorization prompt, or evidence that a tool call succeeded.
@@ -232,9 +241,10 @@ From the repository root:
./libs/cua-driver/scripts/cursor-gallery.sh serve
```
-Open `http://127.0.0.1:3001` to pause, replay, change speed, and compare the
-states on light, dark, blue, or mixed backgrounds. The generated WebM files and
-raw frames are ignored by Git.
+Open `http://127.0.0.1:3001` to pause, replay, change speed, compare the states
+on light, dark, blue, or mixed backgrounds, and inspect the exact session-badge
+hold and fade behavior. The generated WebM files and raw frames are ignored by
+Git.
Regenerate the public documentation GIFs after changing the built-in cursor:
diff --git a/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs b/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs
index 872524823b..e8ff92a855 100644
--- a/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-contract/src/cursor.rs
@@ -259,6 +259,7 @@ pub fn classify_cursor_semantics(name: &str, args: &Value) -> Option<CursorSeman
Some(CursorTarget::Browser)
} else if args
.get("element_index")
+ .or_else(|| args.get("element_token"))
.is_some_and(|value| !value.is_null())
{
Some(CursorTarget::Ax)
@@ -331,6 +332,17 @@ mod tests {
target: Some(CursorTarget::Browser),
})
);
+ assert_eq!(
+ classify_cursor_semantics(
+ "type_text",
+ &json!({"element_token":"opaque", "delivery_mode":"foreground"})
+ ),
+ Some(CursorSemantics {
+ action: CursorAction::Text,
+ delivery: Some(CursorDelivery::Foreground),
+ target: Some(CursorTarget::Ax),
+ })
+ );
}
#[test]
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/examples/export_gallery_frames.rs b/libs/cua-driver/rust/crates/cursor-overlay/examples/export_gallery_frames.rs
index b500987fed..aedc75a857 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/examples/export_gallery_frames.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/examples/export_gallery_frames.rs
@@ -1,6 +1,6 @@
use cursor_overlay::{
- theme::paint_default_theme, CursorAction, CursorVisualState, DeliveryModifier, ReducedMotion,
- TargetModifier,
+ render_frame, theme::paint_default_theme, CursorAction, CursorConfig, CursorVisualState,
+ DeliveryModifier, OverlayCommand, ReducedMotion, RenderStateCore, TargetModifier,
};
use std::{fs, path::Path};
use tiny_skia::Pixmap;
@@ -54,6 +54,38 @@ fn main() {
scope.spawn(move || export_state(&path, action, delivery, target));
}
});
+ export_session_badge(&output.join("session").join("badge"));
+}
+
+fn export_session_badge(output: &Path) {
+ fs::create_dir_all(output).expect("create session badge frame output");
+ for frame in 0..FPS * DURATION_SECS {
+ let mut config = CursorConfig::default();
+ config.cursor_id = "gallery-session".into();
+ let mut core = RenderStateCore::new(config);
+ core.motion.idle_hide_ms = 0.0;
+ core.pos = (
+ f64::from(SIZE) / (2.0 * f64::from(PREVIEW_BACKING_SCALE)),
+ 64.0,
+ );
+ core.apply_command_base(
+ OverlayCommand::SetSessionLabel("Research".into()),
+ false,
+ false,
+ );
+ core.tick_motion(f64::from(frame) / f64::from(FPS));
+ let pixmap = render_frame(&core, SIZE, SIZE, 0.0, 0.0, None, PREVIEW_BACKING_SCALE);
+ let pixels = unpremultiply_rgba(pixmap.data().to_vec());
+ image::save_buffer_with_format(
+ output.join(format!("{frame:04}.png")),
+ &pixels,
+ SIZE,
+ SIZE,
+ image::ColorType::Rgba8,
+ image::ImageFormat::Png,
+ )
+ .expect("write session badge frame");
+ }
}
fn export_state(
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
index 28a196d742..6955828053 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/render_state.rs
@@ -40,6 +40,9 @@ use crate::{
};
use std::sync::Arc;
+pub const SESSION_BADGE_HOLD_SECS: f64 = 2.0;
+pub const SESSION_BADGE_FADE_SECS: f64 = 0.4;
+
/// Platform-agnostic render state shared by macOS / Windows / Linux overlays.
///
/// Each platform wraps this in its own struct that adds OS-specific fields
@@ -81,6 +84,8 @@ pub struct RenderStateCore {
pub pinned_wid: Option<u64>,
/// Sanitized caller-facing label painted below the cursor.
pub session_label: Option<String>,
+ /// Elapsed time since the session label was revealed with the cursor.
+ pub session_badge_secs: f64,
}
impl RenderStateCore {
@@ -124,9 +129,33 @@ impl RenderStateCore {
idle_alpha: 1.0,
pinned_wid: None,
session_label: None,
+ session_badge_secs: SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS,
+ }
+ }
+
+ fn cursor_is_revealed(&self) -> bool {
+ self.visible && self.pos.0 >= -100.0 && self.idle_alpha >= 0.004
+ }
+
+ fn reveal_session_badge(&mut self) {
+ if self.session_label.is_some() {
+ self.session_badge_secs = 0.0;
}
}
+ pub fn session_badge_alpha(&self) -> f32 {
+ if self.session_label.is_none() {
+ return 0.0;
+ }
+ if self.session_badge_secs <= SESSION_BADGE_HOLD_SECS {
+ return 1.0;
+ }
+ let fade = ((self.session_badge_secs - SESSION_BADGE_HOLD_SECS) / SESSION_BADGE_FADE_SECS)
+ .clamp(0.0, 1.0);
+ let smooth = fade * fade * (3.0 - 2.0 * fade);
+ (1.0 - smooth) as f32
+ }
+
/// Return the theme that is actually being painted, including any
/// non-fatal fallback from an unavailable launch-time selection.
pub fn active_theme_metadata(&self) -> (String, String, String, Option<String>) {
@@ -367,6 +396,10 @@ impl RenderStateCore {
/// `motion.idle_hide_ms` has elapsed. Identical across all platforms.
fn tick_idle(&mut self, dt: f64) {
self.visual.tick(dt);
+ if self.session_label.is_some() {
+ self.session_badge_secs = (self.session_badge_secs + dt)
+ .min(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);
+ }
let idle_hide_ms = self.motion.idle_hide_ms;
if idle_hide_ms > 0.0 {
let moving = self.path.is_some() || self.spring.is_some() || self.click_t.is_some();
@@ -416,6 +449,7 @@ impl RenderStateCore {
y,
end_heading_radians,
} => {
+ let reveal_badge = !self.cursor_is_revealed();
// Apply click offset (16 pt along end_heading) before planning,
// matching Swift `moveTo(point:endAngleRadians:)`:
// tx = clickPoint.x + cos(endAngle) * clickOffset
@@ -439,9 +473,17 @@ impl RenderStateCore {
self.dist = 0.0;
self.spring = None;
self.spring_tgt = None;
- self.visual.begin(CursorAction::Navigate, None, None);
+ if matches!(
+ self.visual.resolved_action,
+ CursorAction::Idle | CursorAction::Navigate
+ ) {
+ self.visual.begin(CursorAction::Navigate, None, None);
+ }
self.idle_secs = 0.0;
self.idle_alpha = 1.0;
+ if reveal_badge {
+ self.reveal_session_badge();
+ }
true
}
OverlayCommand::SnapTo {
@@ -449,6 +491,7 @@ impl RenderStateCore {
y,
heading_radians,
} => {
+ let reveal_badge = !self.cursor_is_revealed();
self.pos = (x, y);
if let Some(heading) = heading_radians {
self.heading = heading;
@@ -457,12 +500,21 @@ impl RenderStateCore {
self.dist = 0.0;
self.spring = None;
self.spring_tgt = None;
- self.visual.begin(CursorAction::Navigate, None, None);
+ if matches!(
+ self.visual.resolved_action,
+ CursorAction::Idle | CursorAction::Navigate
+ ) {
+ self.visual.begin(CursorAction::Navigate, None, None);
+ }
self.idle_secs = 0.0;
self.idle_alpha = 1.0;
+ if reveal_badge {
+ self.reveal_session_badge();
+ }
true
}
OverlayCommand::ClickPulse { x, y } => {
+ let reveal_badge = !self.cursor_is_revealed();
if click_pulse_sentinel_only {
// macOS: only snap position on first placement (sentinel state).
// After that the cursor stays where the animation landed.
@@ -479,9 +531,17 @@ impl RenderStateCore {
self.pos = (x, y);
}
self.click_t = Some(0.0);
- self.visual.begin(CursorAction::Click, None, None);
+ if matches!(
+ self.visual.resolved_action,
+ CursorAction::Idle | CursorAction::Navigate | CursorAction::Click
+ ) {
+ self.visual.begin(CursorAction::Click, None, None);
+ }
self.idle_secs = 0.0;
self.idle_alpha = 1.0;
+ if reveal_badge {
+ self.reveal_session_badge();
+ }
true
}
OverlayCommand::SetPressed(v) => {
@@ -496,7 +556,11 @@ impl RenderStateCore {
true
}
OverlayCommand::SetEnabled(v) => {
+ let reveal_badge = v && !self.visible;
self.visible = v;
+ if reveal_badge {
+ self.reveal_session_badge();
+ }
true
}
OverlayCommand::SetMotion(m) => {
@@ -543,6 +607,7 @@ impl RenderStateCore {
}
OverlayCommand::SetSessionLabel(label) => {
self.session_label = crate::sanitize_session_label(&label);
+ self.session_badge_secs = 0.0;
true
}
OverlayCommand::ShowFocusRect(_) => false, // caller-specific
@@ -718,13 +783,14 @@ pub fn paint_cursor(
}
if let Some(label) = core.session_label.as_deref() {
+ let badge_alpha = core.session_badge_alpha();
crate::paint_session_badge(
pm,
label,
px as f32,
py as f32,
backing_scale.max(1.0),
- alpha_scale,
+ alpha_scale * badge_alpha,
crate::session_fill_rgba(&core.cfg.cursor_id),
);
}
@@ -790,6 +856,100 @@ mod glide_duration_tests {
}
}
+#[cfg(test)]
+mod session_badge_and_action_tests {
+ use super::*;
+ use crate::{CursorConfig, TargetModifier};
+
+ #[test]
+ fn session_badge_holds_then_fades_once() {
+ let mut core = RenderStateCore::new(CursorConfig::default());
+ assert_eq!(core.session_badge_alpha(), 0.0);
+ assert!(core.apply_command_base(
+ OverlayCommand::SetSessionLabel("Research".into()),
+ false,
+ false,
+ ));
+ assert_eq!(core.session_badge_alpha(), 1.0);
+
+ core.tick_motion(SESSION_BADGE_HOLD_SECS - 0.05);
+ assert_eq!(core.session_badge_alpha(), 1.0);
+ core.tick_motion(SESSION_BADGE_FADE_SECS * 0.5 + 0.05);
+ assert!(core.session_badge_alpha() > 0.0);
+ assert!(core.session_badge_alpha() < 1.0);
+ core.tick_motion(SESSION_BADGE_FADE_SECS);
+ assert_eq!(core.session_badge_alpha(), 0.0);
+ }
+
+ #[test]
+ fn revealing_hidden_cursor_restarts_badge_without_restarting_on_every_move() {
+ let mut core = RenderStateCore::new(CursorConfig::default());
+ core.apply_command_base(
+ OverlayCommand::SetSessionLabel("Research".into()),
+ false,
+ false,
+ );
+ core.tick_motion(SESSION_BADGE_HOLD_SECS + SESSION_BADGE_FADE_SECS);
+ assert_eq!(core.session_badge_alpha(), 0.0);
+
+ core.apply_command_base(
+ OverlayCommand::SnapTo {
+ x: 100.0,
+ y: 100.0,
+ heading_radians: None,
+ },
+ false,
+ false,
+ );
+ assert_eq!(core.session_badge_alpha(), 1.0);
+ core.tick_motion(0.5);
+ let elapsed = core.session_badge_secs;
+ core.apply_command_base(
+ OverlayCommand::SnapTo {
+ x: 120.0,
+ y: 120.0,
+ heading_radians: None,
+ },
+ false,
+ false,
+ );
+ assert_eq!(core.session_badge_secs, elapsed);
+ }
+
+ #[test]
+ fn movement_preserves_the_active_semantic_action() {
+ let mut core = RenderStateCore::new(CursorConfig::default());
+ core.pos = (20.0, 20.0);
+ core.apply_command_base(
+ OverlayCommand::BeginAction {
+ action: CursorAction::Text,
+ delivery: None,
+ target: Some(TargetModifier::Ax),
+ },
+ false,
+ false,
+ );
+ core.apply_command_base(
+ OverlayCommand::MoveTo {
+ x: 200.0,
+ y: 100.0,
+ end_heading_radians: 0.0,
+ },
+ false,
+ false,
+ );
+ assert_eq!(core.visual.resolved_action, CursorAction::Text);
+ assert_eq!(core.visual.target, Some(TargetModifier::Ax));
+ core.apply_command_base(
+ OverlayCommand::ClickPulse { x: 200.0, y: 100.0 },
+ false,
+ false,
+ );
+ assert_eq!(core.visual.resolved_action, CursorAction::Text);
+ assert_eq!(core.visual.target, Some(TargetModifier::Ax));
+ }
+}
+
#[cfg(test)]
mod backing_scale_tests {
use super::*;
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs
index 45db567cc9..2fd7575f22 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/session_badge.rs
@@ -9,18 +9,22 @@
use fontdue::layout::{CoordinateSystem, Layout, LayoutSettings, TextStyle};
use std::sync::OnceLock;
-use tiny_skia::{Color, Paint, Path, PathBuilder, Pixmap, PixmapPaint, Rect, Stroke, Transform};
+use tiny_skia::{
+ Color, GradientStop, LinearGradient, Paint, Path, PathBuilder, Pixmap, PixmapPaint, Point,
+ Rect, SpreadMode, Stroke, Transform,
+};
pub const MAX_SESSION_LABEL_CHARS: usize = 28;
-pub const BADGE_MAX_WIDTH: f32 = 176.0;
-pub const BADGE_HEIGHT: f32 = 24.0;
-pub const BADGE_CURSOR_GAP: f32 = 30.0;
+pub const BADGE_MAX_WIDTH: f32 = 188.0;
+pub const BADGE_HEIGHT: f32 = 28.0;
+pub const BADGE_CURSOR_GAP: f32 = 25.0;
const FONT_BYTES: &[u8] = include_bytes!("../assets/Inter.ttf");
-const FONT_SIZE: f32 = 11.0;
+const FONT_SIZE: f32 = 11.5;
const HORIZONTAL_PADDING: f32 = 10.0;
-const DOT_SIZE: f32 = 5.0;
-const DOT_GAP: f32 = 6.0;
+const ORB_SIZE: f32 = 10.0;
+const ORB_GAP: f32 = 7.0;
+const TEXT_OPTICAL_Y_OFFSET: f32 = 1.0;
fn font() -> Option<&'static fontdue::Font> {
static FONT: OnceLock<Option<fontdue::Font>> = OnceLock::new();
@@ -122,6 +126,68 @@ fn rounded_rect(rect: Rect, radius: f32) -> Option<Path> {
builder.finish()
}
+fn mixed_channel(base: u8, accent: u8, accent_weight: f32) -> u8 {
+ let weight = accent_weight.clamp(0.0, 1.0);
+ ((base as f32 * (1.0 - weight)) + (accent as f32 * weight)).round() as u8
+}
+
+fn session_tinted_color(
+ base: [u8; 3],
+ session_fill: [u8; 4],
+ accent_weight: f32,
+ alpha: u8,
+) -> Color {
+ Color::from_rgba8(
+ mixed_channel(base[0], session_fill[0], accent_weight),
+ mixed_channel(base[1], session_fill[1], accent_weight),
+ mixed_channel(base[2], session_fill[2], accent_weight),
+ alpha,
+ )
+}
+
+fn paint_diffuse_glow(
+ pixmap: &mut Pixmap,
+ rect: Rect,
+ radius: f32,
+ scale: f32,
+ color: [u8; 3],
+ alpha: f32,
+ y_offset: f32,
+) {
+ for step in (1..=8).rev() {
+ let spread = step as f32 * scale;
+ let Some(layer_rect) = Rect::from_xywh(
+ rect.x() - spread,
+ rect.y() - spread * 0.55 + y_offset * scale,
+ rect.width() + spread * 2.0,
+ rect.height() + spread * 1.1,
+ ) else {
+ continue;
+ };
+ let Some(layer) = rounded_rect(layer_rect, radius + spread) else {
+ continue;
+ };
+ let strength = 0.35 + (8 - step) as f32 * 0.085;
+ let paint = Paint {
+ shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(
+ color[0],
+ color[1],
+ color[2],
+ (alpha * strength).round().clamp(0.0, 255.0) as u8,
+ )),
+ anti_alias: true,
+ ..Default::default()
+ };
+ pixmap.fill_path(
+ &layer,
+ &paint,
+ tiny_skia::FillRule::Winding,
+ Transform::identity(),
+ None,
+ );
+ }
+}
+
/// Paint the label below the cursor without rotating it with cursor heading.
pub fn paint_session_badge(
pixmap: &mut Pixmap,
@@ -137,7 +203,7 @@ pub fn paint_session_badge(
};
let scale = backing_scale.max(1.0);
let font_size = FONT_SIZE * scale;
- let max_text_width = (BADGE_MAX_WIDTH - HORIZONTAL_PADDING * 2.0 - DOT_SIZE - DOT_GAP) * scale;
+ let max_text_width = (BADGE_MAX_WIDTH - HORIZONTAL_PADDING * 2.0 - ORB_SIZE - ORB_GAP) * scale;
let mut layout = Layout::new(CoordinateSystem::PositiveYDown);
layout.reset(&LayoutSettings {
@@ -147,15 +213,31 @@ pub fn paint_session_badge(
});
layout.append(&[font], &TextStyle::new(label, font_size, 0));
let glyphs = layout.glyphs();
- let text_width = glyphs
+ if glyphs.is_empty() {
+ return;
+ }
+ let glyph_min_x = glyphs
+ .iter()
+ .map(|glyph| glyph.x)
+ .fold(f32::INFINITY, f32::min);
+ let glyph_max_x = glyphs
.iter()
.map(|glyph| glyph.x + glyph.width as f32)
- .fold(0.0_f32, f32::max)
- .min(max_text_width);
+ .fold(f32::NEG_INFINITY, f32::max);
+ let glyph_min_y = glyphs
+ .iter()
+ .map(|glyph| glyph.y)
+ .fold(f32::INFINITY, f32::min);
+ let glyph_max_y = glyphs
+ .iter()
+ .map(|glyph| glyph.y + glyph.height as f32)
+ .fold(f32::NEG_INFINITY, f32::max);
+ let text_width = (glyph_max_x - glyph_min_x).min(max_text_width);
+ let text_height = glyph_max_y - glyph_min_y;
let badge_width =
- (HORIZONTAL_PADDING * 2.0 * scale + DOT_SIZE * scale + DOT_GAP * scale + text_width)
+ (HORIZONTAL_PADDING * 2.0 * scale + ORB_SIZE * scale + ORB_GAP * scale + text_width)
.min(BADGE_MAX_WIDTH * scale)
- .max(52.0 * scale);
+ .max(72.0 * scale);
let badge_height = BADGE_HEIGHT * scale;
let x = (cursor_x - badge_width * 0.5).clamp(
2.0 * scale,
@@ -168,31 +250,59 @@ pub fn paint_session_badge(
let Some(rect) = Rect::from_xywh(x, y, badge_width, badge_height) else {
return;
};
- let Some(path) = rounded_rect(rect, badge_height * 0.5) else {
+ let corner_radius = badge_height * 0.5;
+ let Some(path) = rounded_rect(rect, corner_radius) else {
return;
};
let opacity = alpha_scale.clamp(0.0, 1.0);
- let shadow = Paint {
- shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(0, 0, 0, (76.0 * opacity) as u8)),
- anti_alias: true,
- ..Default::default()
- };
- pixmap.fill_path(
- &path,
- &shadow,
- tiny_skia::FillRule::Winding,
- Transform::from_translate(0.0, 2.0 * scale),
- None,
+ if opacity <= 0.0 {
+ return;
+ }
+
+ paint_diffuse_glow(
+ pixmap,
+ rect,
+ corner_radius,
+ scale,
+ [0, 0, 0],
+ 8.0 * opacity,
+ 2.0,
+ );
+ paint_diffuse_glow(
+ pixmap,
+ rect,
+ corner_radius,
+ scale,
+ [session_fill[0], session_fill[1], session_fill[2]],
+ 10.0 * opacity,
+ 0.0,
);
+ let background_shader = LinearGradient::new(
+ Point::from_xy(x, y),
+ Point::from_xy(x + badge_width, y + badge_height),
+ vec![
+ GradientStop::new(
+ 0.0,
+ session_tinted_color([94, 151, 178], session_fill, 0.52, (236.0 * opacity) as u8),
+ ),
+ GradientStop::new(
+ 0.46,
+ session_tinted_color([43, 92, 119], session_fill, 0.40, (239.0 * opacity) as u8),
+ ),
+ GradientStop::new(
+ 1.0,
+ session_tinted_color([13, 27, 38], session_fill, 0.18, (245.0 * opacity) as u8),
+ ),
+ ],
+ SpreadMode::Pad,
+ Transform::identity(),
+ );
let background = Paint {
- shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(
- 18,
- 22,
- 28,
- (224.0 * opacity) as u8,
- )),
+ shader: background_shader.unwrap_or_else(|| {
+ tiny_skia::Shader::SolidColor(Color::from_rgba8(18, 22, 28, (242.0 * opacity) as u8))
+ }),
anti_alias: true,
..Default::default()
};
@@ -208,7 +318,7 @@ pub fn paint_session_badge(
255,
255,
255,
- (92.0 * opacity) as u8,
+ (160.0 * opacity) as u8,
)),
anti_alias: true,
..Default::default()
@@ -224,10 +334,11 @@ pub fn paint_session_badge(
None,
);
- let dot_x = x + HORIZONTAL_PADDING * scale;
- let dot_y = y + (badge_height - DOT_SIZE * scale) * 0.5;
- if let Some(dot) = Rect::from_xywh(dot_x, dot_y, DOT_SIZE * scale, DOT_SIZE * scale) {
- let dot_paint = Paint {
+ let orb_center_x = x + HORIZONTAL_PADDING * scale + ORB_SIZE * scale * 0.5;
+ let orb_center_y = y + badge_height * 0.5;
+ if let Some(orb) = PathBuilder::from_circle(orb_center_x, orb_center_y, ORB_SIZE * scale * 0.5)
+ {
+ let orb_paint = Paint {
shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(
session_fill[0],
session_fill[1],
@@ -237,11 +348,38 @@ pub fn paint_session_badge(
anti_alias: true,
..Default::default()
};
- pixmap.fill_rect(dot, &dot_paint, Transform::identity(), None);
+ pixmap.fill_path(
+ &orb,
+ &orb_paint,
+ tiny_skia::FillRule::Winding,
+ Transform::identity(),
+ None,
+ );
+ let orb_outline = Paint {
+ shader: tiny_skia::Shader::SolidColor(Color::from_rgba8(
+ 255,
+ 255,
+ 255,
+ (226.0 * opacity) as u8,
+ )),
+ anti_alias: true,
+ ..Default::default()
+ };
+ pixmap.stroke_path(
+ &orb,
+ &orb_outline,
+ &Stroke {
+ width: 1.5 * scale,
+ ..Default::default()
+ },
+ Transform::identity(),
+ None,
+ );
}
- let text_origin_x = dot_x + (DOT_SIZE + DOT_GAP) * scale;
- let text_origin_y = y + (badge_height - FONT_SIZE * scale) * 0.5 - 1.0 * scale;
+ let text_origin_x = x + HORIZONTAL_PADDING * scale + (ORB_SIZE + ORB_GAP) * scale - glyph_min_x;
+ let text_origin_y =
+ y + (badge_height - text_height) * 0.5 - glyph_min_y + TEXT_OPTICAL_Y_OFFSET * scale;
for glyph in glyphs {
let (metrics, bitmap) = font.rasterize_config(glyph.key);
if metrics.width == 0 || metrics.height == 0 {
@@ -302,4 +440,40 @@ mod tests {
assert!(pixmap.data().chunks_exact(4).any(|pixel| pixel[3] > 0));
}
}
+
+ #[test]
+ fn badge_gradient_tracks_session_color_and_zero_alpha_paints_nothing() {
+ let mut blue = Pixmap::new(240, 160).unwrap();
+ let mut purple = Pixmap::new(240, 160).unwrap();
+ let mut hidden = Pixmap::new(240, 160).unwrap();
+ paint_session_badge(
+ &mut blue,
+ "Research run",
+ 120.0,
+ 60.0,
+ 1.0,
+ 1.0,
+ [94, 192, 232, 255],
+ );
+ paint_session_badge(
+ &mut purple,
+ "Research run",
+ 120.0,
+ 60.0,
+ 1.0,
+ 1.0,
+ [178, 132, 255, 255],
+ );
+ paint_session_badge(
+ &mut hidden,
+ "Research run",
+ 120.0,
+ 60.0,
+ 1.0,
+ 0.0,
+ [94, 192, 232, 255],
+ );
+ assert_ne!(blue.data(), purple.data());
+ assert!(hidden.data().chunks_exact(4).all(|pixel| pixel[3] == 0));
+ }
}
diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/theme.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/theme.rs
index 12c05a5542..92d5c197fb 100644
--- a/libs/cua-driver/rust/crates/cursor-overlay/src/theme.rs
+++ b/libs/cua-driver/rust/crates/cursor-overlay/src/theme.rs
@@ -16,7 +16,7 @@ pub const DEFAULT_THEME_ID: &str = "cua.default";
pub const DEFAULT_THEME_VERSION: &str = "1.0.0";
pub const THEME_PROFILE: &str = "cua-driver-full-v1";
pub const CANVAS_SIZE: f32 = 128.0;
-pub const DISPLAY_SIZE: f32 = 48.0;
+pub const DISPLAY_SIZE: f32 = 42.0;
const FLOAT_DURATION_SECS: f32 = 4.0;
pub const DEFAULT_CURSOR_FILL: [u8; 4] = [94, 192, 232, 255];
@@ -259,8 +259,8 @@ mod tests {
}
#[test]
- fn default_theme_uses_compact_48_point_footprint() {
- assert_eq!(DISPLAY_SIZE, 48.0);
+ fn default_theme_uses_compact_42_point_footprint() {
+ assert_eq!(DISPLAY_SIZE, 42.0);
}
#[test]
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
index d434ce4ae0..7d9121bebd 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
@@ -1531,10 +1531,6 @@ fn held_target_mismatch(
}
}
-async fn overlay_glide_to(sx: f64, sy: f64) {
- overlay_glide_to_for("default", sx, sy).await;
-}
-
fn overlay_snap_to_for(cursor_id: &str, sx: f64, sy: f64, heading: Option<f64>) {
crate::overlay::send_command_for(
cursor_id.to_owned(),
@@ -2326,6 +2322,22 @@ impl Tool for TypeTextTool {
);
}
+ let cursor_id = resolve_cursor_key(&args);
+ if let Some(idx) = resolved_elem_idx {
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(xid),
+ );
+ if let Ok(Ok((screen_x, screen_y))) =
+ tokio::task::spawn_blocking(move || element_screen_center(pid, idx)).await
+ {
+ overlay_glide_to_for(&cursor_id, screen_x, screen_y).await;
+ self.state
+ .cursor_registry
+ .update_position(&cursor_id, screen_x, screen_y);
+ }
+ }
+
let text_len = text.chars().count();
// Native toolkit editables have a stronger focus-free route than raw
// compositor keyboard injection. Keep Chromium/WebKit on real key events
@@ -2549,21 +2561,6 @@ impl Tool for TypeTextTool {
Err(e) => ToolResult::error(format!("Task error: {e}")),
};
}
- // Pulse the agent cursor onto the field being typed into (when an
- // element_index OR element_token is supplied — token resolution
- // already ran above so `resolved_elem_idx` covers both).
- if let Some(idx) = resolved_elem_idx {
- crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(xid));
- if let Ok(Ok((sx, sy))) =
- tokio::task::spawn_blocking(move || element_screen_center(pid, idx)).await
- {
- overlay_glide_to(sx, sy).await;
- crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {
- x: sx,
- y: sy,
- });
- }
- }
// Foreground means the caller explicitly permits activation. Chromium
// and WebKitGTK can acknowledge an accessibility write without
// producing the renderer input event, so web embedders use real XTest
@@ -3489,6 +3486,7 @@ impl Tool for SetValueTool {
"set_value requires element_index or element_token to address the target element.",
),
};
+ let cursor_id = resolve_cursor_key(&args);
let value_for_task = value.clone();
// Pulse the agent cursor onto the target element before writing, so a
// value write gets the same visual feedback as a click — the viewer can
@@ -3499,13 +3497,12 @@ impl Tool for SetValueTool {
{
let window_id = args.u64_or("window_id", 0);
if window_id != 0 {
- crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(window_id));
+ crate::overlay::send_command_for(
+ cursor_id.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(window_id),
+ );
}
- overlay_glide_to(sx, sy).await;
- crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse {
- x: sx,
- y: sy,
- });
+ overlay_glide_to_for(&cursor_id, sx, sy).await;
}
let result =
tokio::task::spawn_blocking(move || crate::atspi::set_value(pid, idx, &value_for_task))
diff --git a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs
index db311685cb..0f9de296dc 100644
--- a/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs
+++ b/libs/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs
@@ -35,7 +35,7 @@ const DBUS_DEST: &str = "org.freedesktop.DBus";
const DBUS_PATH: &str = "/org/freedesktop/DBus";
const DBUS_IFACE: &str = "org.freedesktop.DBus";
const BROWSER_HELPER_API_VERSION: u32 = 4;
-const SEMANTIC_CURSOR_API_VERSION: u32 = 6;
+const SEMANTIC_CURSOR_API_VERSION: u32 = 7;
#[derive(Debug, Clone)]
struct ShellWindow {
@@ -627,12 +627,12 @@ mod tests {
}
#[test]
- fn bundled_helper_v6_uses_the_compact_semantic_cursor_and_session_badge() {
- assert!(EXTENSION_SOURCE.contains("GetVersion() { return 6; }"));
+ fn bundled_helper_v7_uses_the_compact_semantic_cursor_and_session_badge() {
+ assert!(EXTENSION_SOURCE.contains("GetVersion() { return 7; }"));
assert!(EXTENSION_SOURCE.contains("SetCursorState"));
assert!(EXTENSION_SOURCE.contains("SetCursorColor"));
assert!(EXTENSION_SOURCE.contains("SetSessionLabel"));
- assert!(EXTENSION_SOURCE.contains("const DISPLAY_SIZE = 48;"));
+ assert!(EXTENSION_SOURCE.contains("const DISPLAY_SIZE = 42;"));
assert!(EXTENSION_SOURCE.contains("const GLOW_PADDING = 24;"));
assert!(EXTENSION_SOURCE.contains("function drawCursorGlowShape"));
assert!(EXTENSION_SOURCE.contains("function glowPath"));
@@ -641,7 +641,7 @@ mod tests {
assert!(EXTENSION_SOURCE.contains("width - 1"));
assert!(EXTENSION_SOURCE.contains("createGlowSurface(this._fillColor)"));
assert!(EXTENSION_SOURCE.contains("cr.translate(-GLOW_PADDING, -GLOW_PADDING);"));
- assert!(EXTENSION_METADATA.contains("\"version\":6"));
+ assert!(EXTENSION_METADATA.contains("\"version\":7"));
for action in [
"idle", "observe", "click", "drag", "scroll", "text", "key", "navigate", "app",
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
index 5cc5f550ee..3617c67e3b 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs
@@ -158,6 +158,22 @@ impl Tool for SetValueTool {
}
};
let element_ptr = element_guard.as_ptr();
+ let cursor_key = super::cursor_tools::resolve_cursor_key(&args);
+ let center_ptr = element_ptr as usize;
+ if let Ok(Some((screen_x, screen_y))) = tokio::task::spawn_blocking(move || unsafe {
+ crate::ax::bindings::element_screen_center(center_ptr as AXUIElementRef)
+ })
+ .await
+ {
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(window_id as u64),
+ );
+ crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), screen_x, screen_y).await;
+ self.state
+ .cursor_registry
+ .update_position(&cursor_key, screen_x, screen_y);
+ }
// An AXValue read-back is not ground truth for web content. Chromium,
// WebKit, and Electron can echo the write through accessibility while
// the renderer never observes it. Reuse type_text's bounded ancestor
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
index d40cf4aa0e..d74f5df922 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs
@@ -260,6 +260,25 @@ impl Tool for TypeTextTool {
} else {
None
};
+ if let (Some((element, _)), Some(wid)) = (element_guard.as_ref(), window_id) {
+ let center_ptr = element.as_ptr() as usize;
+ if let Ok(Some((screen_x, screen_y))) = tokio::task::spawn_blocking(move || unsafe {
+ crate::ax::bindings::element_screen_center(center_ptr as AXUIElementRef)
+ })
+ .await
+ {
+ let cursor_key = super::cursor_tools::resolve_cursor_key(&args);
+ crate::cursor::overlay::send_command(
+ cursor_key.clone(),
+ cursor_overlay::OverlayCommand::PinAbove(wid as u64),
+ );
+ crate::cursor::overlay::animate_cursor_to(cursor_key.clone(), screen_x, screen_y)
+ .await;
+ self.state
+ .cursor_registry
+ .update_position(&cursor_key, screen_x, screen_y);
+ }
+ }
let element_ptr = element_guard
.as_ref()
.map(|(g, idx)| (g.as_ptr(), Some(*idx)));
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
index b599b94d05..379b79b7be 100644
--- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
+++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs
@@ -3629,6 +3629,19 @@ impl Tool for TypeTextTool {
}
}
};
+ if let Some(idx) = elem_idx {
+ if let Some((cx, cy)) =
+ self.state
+ .element_cache
+ .get_element_center(pid, hwnd, idx as usize)
+ {
+ pin_overlay_above(&cursor_key, hwnd);
+ overlay_glide_to(&cursor_key, cx as f64, cy as f64).await;
+ self.state
+ .cursor_registry
+ .update_position(&cursor_key, cx as f64, cy as f64);
+ }
+ }
let text_len = text.chars().count();
// Native ConsoleHost can accept every synthesized Unicode event while
diff --git a/libs/cua-driver/scripts/cursor-gallery.sh b/libs/cua-driver/scripts/cursor-gallery.sh
index f4cd46c631..7241beba08 100755
--- a/libs/cua-driver/scripts/cursor-gallery.sh
+++ b/libs/cua-driver/scripts/cursor-gallery.sh
@@ -28,7 +28,7 @@ export_assets() {
--features theme-authoring \
-- "$FRAMES_DIR"
- for group in actions modifiers combined; do
+ for group in actions modifiers combined session; do
mkdir -p "$GENERATED_DIR/$group"
for state_dir in "$FRAMES_DIR/$group"/*; do
state=$(basename "$state_dir")
diff --git a/libs/cua-driver/tools/cursor-gallery/index.html b/libs/cua-driver/tools/cursor-gallery/index.html
index 3e32e951c5..bb54af3dc0 100644
--- a/libs/cua-driver/tools/cursor-gallery/index.html
+++ b/libs/cua-driver/tools/cursor-gallery/index.html
@@ -78,11 +78,35 @@
<span>□ Pixel target</span>
</div>
</article>
+
+ <article class="combined-card">
+ <div>
+ <p class="section-count">Session identity</p>
+ <h3>Compact label on cursor reveal</h3>
+ <p>The session-tinted badge holds for two seconds, then fades while the cursor remains.</p>
+ </div>
+ <div class="demo">
+ <video
+ class="cursor-video"
+ src="./generated/session/badge.webm"
+ data-group="session"
+ data-state="badge"
+ autoplay
+ loop
+ muted
+ playsinline
+ ></video>
+ </div>
+ <div class="legend">
+ <span>● Circular session marker</span>
+ <span>◒ Session-accent gradient</span>
+ </div>
+ </article>
</section>
<footer>
<span>Preview scale is enlarged for review.</span>
- <span>Production cursor size remains 48 points.</span>
+ <span>Production cursor size is 42 points.</span>
</footer>
</main>
<script src="./app.js"></script>
diff --git a/libs/cua-driver/tools/cursor-gallery/styles.css b/libs/cua-driver/tools/cursor-gallery/styles.css
index 4dc33563c8..ba7c89dfbd 100644
--- a/libs/cua-driver/tools/cursor-gallery/styles.css
+++ b/libs/cua-driver/tools/cursor-gallery/styles.css
@@ -175,7 +175,7 @@ h1 {
/*
* Renderer frames preserve the full 256 px transparent capture canvas.
* The frame exporter renders at 1.5× backing scale. Scale its centered
- * 164 px layout box to preserve that 72 px review size after the 256 px
+ * 164 px layout box to preserve the enlarged review size after the 256 px
* capture canvas is fitted into the card.
*/
width: 164px;
diff --git a/libs/cua-driver/wayland-helper/winrects@cua/extension.js b/libs/cua-driver/wayland-helper/winrects@cua/extension.js
index c3ed8c1780..444a6c9036 100644
--- a/libs/cua-driver/wayland-helper/winrects@cua/extension.js
+++ b/libs/cua-driver/wayland-helper/winrects@cua/extension.js
@@ -25,14 +25,16 @@ const IFACE = `<node><interface name="org.cua.WinRects">
// GNOME cannot host the Rust renderer directly, so this Shell actor mirrors
// the embedded cua.default vector theme and semantic state vocabulary.
const CANVAS_SIZE = 128;
-const DISPLAY_SIZE = 48;
+const DISPLAY_SIZE = 42;
const ACTOR_SIZE = 112;
const ACTOR_CENTER = ACTOR_SIZE / 2;
const SCALE = DISPLAY_SIZE / CANVAS_SIZE;
const FLOAT_DURATION = 4.0;
const GLOW_SURFACE_SCALE = 3;
const GLOW_PADDING = 24;
-const BADGE_Y_OFFSET = 34;
+const BADGE_Y_OFFSET = 29;
+const BADGE_HOLD_SECONDS = 2.0;
+const BADGE_FADE_SECONDS = 0.4;
const ACTIONS = new Set([
'idle', 'observe', 'click', 'drag', 'scroll', 'text',
'key', 'navigate', 'app', 'transfer', 'record', 'system',
@@ -57,6 +59,27 @@ function nowSeconds() {
return GLib.get_monotonic_time() / 1_000_000;
}
+function mixChannel(base, accent, weight) {
+ return Math.round(base * (1 - weight) + accent * weight);
+}
+
+function badgeStyle(fillColor) {
+ const start = fillColor.map((channel, index) =>
+ mixChannel([94, 151, 178][index], channel, 0.52));
+ const end = fillColor.map((channel, index) =>
+ mixChannel([13, 27, 38][index], channel, 0.18));
+ return [
+ 'spacing: 7px',
+ 'padding: 6px 10px',
+ `background-gradient-start: rgba(${start[0]}, ${start[1]}, ${start[2]}, 0.93)`,
+ `background-gradient-end: rgba(${end[0]}, ${end[1]}, ${end[2]}, 0.96)`,
+ 'background-gradient-direction: horizontal',
+ 'border: 1px solid rgba(255, 255, 255, 0.63)',
+ 'border-radius: 14px',
+ 'box-shadow: 0 2px 9px rgba(0, 0, 0, 0.30)',
+ ].join(';');
+}
+
function easeInOut(value) {
const t = Math.max(0, Math.min(1, value));
return t * t * (3 - 2 * t);
@@ -476,9 +499,9 @@ export default class WinRectsExtension extends Extension {
this._cursor.set_pivot_point(0.5, 0.5);
Main.layoutManager.addTopChrome(this._cursor);
this._badgeDot = new St.Widget({
- width: 5,
- height: 5,
- style: `background-color: ${this._fillColorCss}; border-radius: 3px;`,
+ width: 10,
+ height: 10,
+ style: `background-color: ${this._fillColorCss}; border: 1px solid white; border-radius: 99px;`,
y_align: Clutter.ActorAlign.CENTER,
});
this._badgeLabel = new St.Label({
@@ -490,20 +513,14 @@ export default class WinRectsExtension extends Extension {
visible: false,
reactive: false,
can_focus: false,
- style: [
- 'spacing: 6px',
- 'padding: 5px 10px',
- 'background-color: rgba(18, 22, 28, 0.88)',
- 'border: 1px solid rgba(255, 255, 255, 0.36)',
- 'border-radius: 12px',
- 'box-shadow: 0 2px 8px rgba(0, 0, 0, 0.30)',
- ].join(';'),
+ style: badgeStyle(this._fillColor),
});
this._badge.add_child(this._badgeDot);
this._badge.add_child(this._badgeLabel);
Main.layoutManager.addTopChrome(this._badge);
this._cursorX = null;
this._cursorY = null;
+ this._badgeRevealedAt = null;
this._frameId = GLib.timeout_add(GLib.PRIORITY_DEFAULT, 33, () => {
if (!this._cursor)
return GLib.SOURCE_REMOVE;
@@ -515,6 +532,18 @@ export default class WinRectsExtension extends Extension {
this._setCursorState('idle', '', '');
if (this._cursor.visible)
this._cursor.queue_repaint();
+ if (this._badge?.visible && this._badgeRevealedAt !== null) {
+ const badgeElapsed = now - this._badgeRevealedAt;
+ if (badgeElapsed <= BADGE_HOLD_SECONDS) {
+ this._badge.opacity = 255;
+ } else if (badgeElapsed < BADGE_HOLD_SECONDS + BADGE_FADE_SECONDS) {
+ const fade = (badgeElapsed - BADGE_HOLD_SECONDS) / BADGE_FADE_SECONDS;
+ this._badge.opacity = Math.round(255 * (1 - fade));
+ } else {
+ this._badge.hide();
+ this._badgeRevealedAt = null;
+ }
+ }
return GLib.SOURCE_CONTINUE;
});
}
@@ -534,7 +563,7 @@ export default class WinRectsExtension extends Extension {
if (this._impl) { this._impl.unexport(); this._impl = null; }
if (this._nameId) { Gio.bus_unown_name(this._nameId); this._nameId = 0; }
}
- GetVersion() { return 6; }
+ GetVersion() { return 7; }
GetRects() {
const actors = global.get_window_actors();
const actorByWindow = new Map();
@@ -613,6 +642,7 @@ export default class WinRectsExtension extends Extension {
}
MoveCursor(x, y) {
if (!this._cursor) return;
+ const revealBadge = !this._cursor.visible;
this._cursorX = x;
this._cursorY = y;
this._cursor.show();
@@ -623,16 +653,22 @@ export default class WinRectsExtension extends Extension {
mode: Clutter.AnimationMode.EASE_OUT_CUBIC,
});
this._positionBadge(x, y, 480);
+ if (revealBadge)
+ this._revealBadge();
}
ClickPulse(x, y) {
if (!this._cursor) return;
+ const revealBadge = !this._cursor.visible;
this._cursorX = x;
this._cursorY = y;
this._cursor.set_position(x - ACTOR_CENTER, y - ACTOR_CENTER);
this._positionBadge(x, y, 0);
- this._setCursorState('click', this._delivery, this._target);
+ if (['idle', 'navigate', 'click'].includes(this._action))
+ this._setCursorState('click', this._delivery, this._target);
this._cursor.show();
this._cursor.queue_repaint();
+ if (revealBadge)
+ this._revealBadge();
}
SetCursorColor(fillColor) {
const match = /^#([0-9a-fA-F]{6})$/.exec(fillColor);
@@ -643,8 +679,10 @@ export default class WinRectsExtension extends Extension {
this._fillColorCss = `#${match[1].toLowerCase()}`;
if (this._badgeDot)
this._badgeDot.set_style(
- `background-color: ${this._fillColorCss}; border-radius: 3px;`
+ `background-color: ${this._fillColorCss}; border: 1px solid white; border-radius: 99px;`
);
+ if (this._badge)
+ this._badge.set_style(badgeStyle(this._fillColor));
if (this._glowSurface)
this._glowSurface.finish();
this._glowSurface = createGlowSurface(this._fillColor);
@@ -666,10 +704,13 @@ export default class WinRectsExtension extends Extension {
this._badgeLabel.set_text(label);
if (label.length === 0 || !this._cursor?.visible) {
this._badge.hide();
+ this._badgeRevealedAt = null;
return;
}
- if (this._cursorX !== null && this._cursorY !== null)
+ if (this._cursorX !== null && this._cursorY !== null) {
this._positionBadge(this._cursorX, this._cursorY, 0);
+ this._revealBadge();
+ }
}
_positionBadge(x, y, duration) {
if (!this._badge || !this._badgeLabel ||
@@ -678,7 +719,6 @@ export default class WinRectsExtension extends Extension {
const [, naturalWidth] = this._badge.get_preferred_width(-1);
const badgeX = Math.round(x - naturalWidth / 2);
const badgeY = Math.round(y + BADGE_Y_OFFSET);
- this._badge.show();
if (duration > 0) {
this._badge.ease({
x: badgeX,
@@ -690,6 +730,14 @@ export default class WinRectsExtension extends Extension {
this._badge.set_position(badgeX, badgeY);
}
}
+ _revealBadge() {
+ if (!this._badge || !this._badgeLabel ||
+ this._badgeLabel.get_text().length === 0)
+ return;
+ this._badgeRevealedAt = nowSeconds();
+ this._badge.opacity = 255;
+ this._badge.show();
+ }
_setCursorState(action, delivery, target) {
this._action = ACTIONS.has(action) ? action : 'idle';
this._delivery = delivery ?? '';
@@ -702,5 +750,6 @@ export default class WinRectsExtension extends Extension {
HideCursor() {
if (this._cursor) this._cursor.hide();
if (this._badge) this._badge.hide();
+ this._badgeRevealedAt = null;
}
}
diff --git a/libs/cua-driver/wayland-helper/winrects@cua/metadata.json b/libs/cua-driver/wayland-helper/winrects@cua/metadata.json
index 01b00c335e..434b8a5eae 100644
--- a/libs/cua-driver/wayland-helper/winrects@cua/metadata.json
+++ b/libs/cua-driver/wayland-helper/winrects@cua/metadata.json
@@ -1 +1 @@
-{"uuid":"winrects@cua","name":"WinRects","description":"Expose window geometry, capture, cursor, and verified activation for cua-driver on Wayland.","shell-version":["45","46","47","48","49","50"],"version":6}
+{"uuid":"winrects@cua","name":"WinRects","description":"Expose window geometry, capture, cursor, and verified activation for cua-driver on Wayland.","shell-version":["45","46","47","48","49","50"],"version":7}