A personal assistant that lives on your desktop, remembers what matters, and can use your Mac for you.
TypeScript
fix(desktop): refuse Windows keys while the user holds a modifier
Marcelitocbe2db29960b
diff --git a/apps/desktop/src-tauri/src/computer/fake.rs b/apps/desktop/src-tauri/src/computer/fake.rsindex 2ca8805..164d36c 100644--- a/apps/desktop/src-tauri/src/computer/fake.rs+++ b/apps/desktop/src-tauri/src/computer/fake.rs@@ -24,6 +24,8 @@ pub struct FakeDesktop {/// The backend refuses this typed post (1-based), sending nothing of it; disabled by default.pub refuse_typed_at: Mutex<Option<(usize, TypedError)>>,pub typed_calls: Mutex<usize>,+ /// Windows key-state snapshot at a typed post (1-based); absent means all keys up.+ pub held_keys_at: Mutex<Option<(usize, Vec<u16>)>>,/// Apps that run as administrator, refused for keyboard input.pub elevated: Mutex<Vec<i32>>,/// Marks an app as elevated after this many recorded posts; disabled by default.@@ -63,6 +65,7 @@ impl Default for FakeDesktop {front_after_posts: Mutex::new(None),refuse_typed_at: Mutex::new(None),typed_calls: Mutex::new(0),+ held_keys_at: Mutex::new(None),elevated: Mutex::new(Vec::new()),elevated_after_posts: Mutex::new(None),front_during_capture: Mutex::new(None),@@ -147,7 +150,12 @@ impl Desktop for FakeDesktop {}}// Only posts that went through, so this stays in step with `events`.- self.post(event).map_err(TypedError::Failed)?;+ let held = self.held_keys_at.lock().unwrap().clone()+ .filter(|(at, _)| *at == call).map(|(_, keys)| keys).unwrap_or_default();+ super::winput::without_held_modifiers(+ |key| if held.contains(&key) { i16::MIN } else { 0 },+ || self.post(event).map_err(TypedError::Failed),+ )?;self.typed_posts.lock().unwrap().push((event.clone(), *recipient));Ok(())}diff --git a/apps/desktop/src-tauri/src/computer/mcp.rs b/apps/desktop/src-tauri/src/computer/mcp.rsindex 85e22e7..3563b05 100644--- a/apps/desktop/src-tauri/src/computer/mcp.rs+++ b/apps/desktop/src-tauri/src/computer/mcp.rs@@ -1676,6 +1676,38 @@ mod tests {}}+ #[test]+ fn held_ctrl_refuses_key_without_sending_anything() {+ let rig = rig();+ rig.ready();+ *rig.desktop.held_keys_at.lock().unwrap() = Some((1, vec![0xa2]));+ let result = rig.call("key", json!({"combo": "return"}));+ assert!(is_error(&result), "{result}");+ assert_eq!(first_text(&result), "Ctrl is held down, so nothing was sent. Release Ctrl and try again.");+ assert!(rig.desktop.events().is_empty());+ }++ #[test]+ fn held_shift_stops_type_before_the_next_return() {+ let rig = rig();+ rig.ready();+ *rig.desktop.held_keys_at.lock().unwrap() = Some((2, vec![0xa1]));+ let result = rig.call("type", json!({"text": "hello\nworld"}));+ assert!(is_error(&result), "{result}");+ assert_eq!(first_text(&result), "Shift is held down, so the rest was not sent. Release Shift and try again. Typing stopped after 5 of 11 characters.");+ assert_eq!(rig.desktop.events(), vec![InputEvent::Text("hello".into())]);+ }++ #[test]+ fn no_held_modifiers_leaves_a_combo_unchanged() {+ let rig = rig();+ rig.ready();+ *rig.desktop.held_keys_at.lock().unwrap() = Some((1, vec![]));+ let result = rig.call("key", json!({"combo": "ctrl+t"}));+ assert!(!is_error(&result), "{result}");+ assert_eq!(rig.desktop.events(), vec![InputEvent::Key(parse_combo("ctrl+t").unwrap())]);+ }+#[test]fn a_backend_refusal_of_a_key_says_the_key_was_not_sent() {// The same wording as the helper's own recheck of the key (#167).diff --git a/apps/desktop/src-tauri/src/computer/mod.rs b/apps/desktop/src-tauri/src/computer/mod.rsindex 4f5267e..d2055f8 100644--- a/apps/desktop/src-tauri/src/computer/mod.rs+++ b/apps/desktop/src-tauri/src/computer/mod.rs@@ -208,7 +208,7 @@ pub enum PermissionKind {pub const KEYS_CHANGED_WHILE_TYPING: &str = "The app receiving the keys changed while typing, so the rest was not sent.";pub const KEYS_CHANGED_BEFORE_TYPING: &str = "The app receiving the keys changed before typing, so nothing was sent.";-/// Why a typed post (`Desktop::post_typed`) did not go through. The first two are refusals+/// Why a typed post (`Desktop::post_typed`) did not go through. All but `Failed` are refusals/// before anything of that post was sent; only the helper knows how much of the text went out/// before it, so it words them (`message`).#[derive(Debug, Clone, PartialEq, Eq)]@@ -220,6 +220,8 @@ pub enum TypedError {/// The window in front has no keyboard focus, so where the keys would go is not the approved/// recipient's field.NoFocus,+ /// Windows observed held modifiers before the post pressed any keys of its own.+ HeldModifiers(String),/// Posting failed; part of the post may have been sent.Failed(String),}@@ -235,6 +237,11 @@ impl TypedError {(TypedError::FrontUnknown, true) => "Cannot tell which window is in front, so the rest was not sent.".into(),(TypedError::NoFocus, false) => "The window in front has no keyboard focus, so nothing was sent.".into(),(TypedError::NoFocus, true) => "The window in front has no keyboard focus, so the rest was not sent.".into(),+ (TypedError::HeldModifiers(keys), sent) => {+ let verb = if keys.contains(" and ") { "are" } else { "is" };+ let unsent = if sent { "the rest was not sent" } else { "nothing was sent" };+ format!("{keys} {verb} held down, so {unsent}. Release {keys} and try again.")+ }(TypedError::Failed(error), _) => error.clone(),}}diff --git a/apps/desktop/src-tauri/src/computer/win.rs b/apps/desktop/src-tauri/src/computer/win.rsindex 109da18..d3fbe38 100644--- a/apps/desktop/src-tauri/src/computer/win.rs+++ b/apps/desktop/src-tauri/src/computer/win.rs@@ -32,7 +32,7 @@ use super::winapps::{answers_to, app_pids, executable_target, fronts_same_app, hosted_page, request, shortcut_named, strip_ci, Request,TopWindow,};-use super::winput::{checks_typing_target, releases, send_batches, settles, steps, Step, VirtualScreen};+use super::winput::{checks_typing_target, releases, send_batches, settles, steps, without_held_modifiers, Step, VirtualScreen};use super::{runs_as_administrator, AppInfo, Capture, InputEvent, KeyboardRecipient, MouseButton, TypedError,KEYS_CHANGED_WHILE_TYPING,@@ -64,7 +64,7 @@ use windows_sys::Win32::Graphics::Gdi::{CAPTUREBLT, DIB_RGB_COLORS, HALFTONE, HBITMAP, HDC, HGDIOBJ, MONITORINFO, MONITOR_DEFAULTTONEAREST, SRCCOPY,};use windows_sys::Win32::UI::Input::KeyboardAndMouse::{- SendInput, INPUT, INPUT_KEYBOARD, INPUT_MOUSE, KEYBDINPUT, KEYEVENTF_EXTENDEDKEY, KEYEVENTF_KEYUP,+ GetAsyncKeyState, SendInput, INPUT, INPUT_KEYBOARD, INPUT_MOUSE, KEYBDINPUT, KEYEVENTF_EXTENDEDKEY, KEYEVENTF_KEYUP,KEYEVENTF_UNICODE, MOUSEEVENTF_ABSOLUTE, MOUSEEVENTF_HWHEEL, MOUSEEVENTF_LEFTDOWN, MOUSEEVENTF_LEFTUP,MOUSEEVENTF_MOVE, MOUSEEVENTF_RIGHTDOWN, MOUSEEVENTF_RIGHTUP, MOUSEEVENTF_VIRTUALDESK, MOUSEEVENTF_WHEEL,MOUSEINPUT,@@ -243,7 +243,7 @@ pub fn post(event: &InputEvent) -> Result<(), String> {/// Posts a typed chunk, a Return, or a key only to the recipient the shared guard approved: the/// windows it is bound to must still belong to the approved front app and focused app. A refusal-/// before anything of it was sent is `RecipientChanged` or `FrontUnknown`, which the helper words+/// before anything of it was sent is a `TypedError` refusal, which the helper words/// from what earlier chunks sent.pub fn post_typed(event: &InputEvent, recipient: &KeyboardRecipient) -> Result<(), TypedError> {post_input(event, Some(recipient))@@ -281,6 +281,16 @@ fn post_input(event: &InputEvent, approved: Option<&KeyboardRecipient>) -> Resul}}let plan = steps(event, &virtual_screen()).map_err(TypedError::Failed)?;+ // Sample after recipient queries and before any of this event's own modifiers go down.+ // Do not sample inside the step loop: ctrl+t would then observe its own injected Ctrl.+ // This is a snapshot, not an atomic fence against a user pressing a key during SendInput.+ if matches!(event, InputEvent::Key(_) | InputEvent::Text(_)) {+ without_held_modifiers(+ // SAFETY: plain key-state query using the Windows virtual-key codes in winput.+ |key| unsafe { GetAsyncKeyState(i32::from(key)) },+ || Ok(()),+ )?;+ }let mut done = 0;for batch in send_batches(&plan) {let checked = checks_typing_target(&batch[0])diff --git a/apps/desktop/src-tauri/src/computer/winput.rs b/apps/desktop/src-tauri/src/computer/winput.rsindex cc4ef71..a481c1e 100644--- a/apps/desktop/src-tauri/src/computer/winput.rs+++ b/apps/desktop/src-tauri/src/computer/winput.rs@@ -7,7 +7,70 @@use super::coords::Point;use super::keys::KeyCombo;-use super::{InputEvent, MouseButton};+use super::{InputEvent, MouseButton, TypedError};++/// Sample before the post presses its own modifiers, never between a combo's steps.+/// GetAsyncKeyState's high bit means down; its low (pressed-since-query) bit is irrelevant.+/// Pointer actions do not check held modifiers yet.+pub fn without_held_modifiers(+ mut key_state: impl FnMut(u16) -> i16,+ post: impl FnOnce() -> Result<(), TypedError>,+) -> Result<(), TypedError> {+ let mut held = Vec::new();+ for (name, left, right) in [+ ("Ctrl", 0xa2, 0xa3), ("Alt", 0xa4, 0xa5),+ ("Shift", 0xa0, 0xa1), ("Win", 0x5b, 0x5c),+ ] {+ if key_state(left) < 0 || key_state(right) < 0 {+ held.push(name);+ }+ }+ if held.is_empty() {+ post()+ } else {+ Err(TypedError::HeldModifiers(held.join(" and ")))+ }+}++#[cfg(test)]+mod held_modifier_tests {+ use super::*;+ use std::cell::Cell;++ #[test]+ fn either_side_of_every_modifier_refuses_before_posting() {+ for (name, keys) in [("Ctrl", [0xa2, 0xa3]), ("Alt", [0xa4, 0xa5]),+ ("Shift", [0xa0, 0xa1]), ("Win", [0x5b, 0x5c])] {+ for held in keys {+ assert_eq!(without_held_modifiers(+ |key| if key == held { i16::MIN } else { 0 },+ || panic!("held modifier must prevent the post")),+ Err(TypedError::HeldModifiers(name.into())));+ }+ }+ }++ #[test]+ fn multiple_modifiers_are_named_once_in_stable_order() {+ let error = without_held_modifiers(+ |key| if [0xa2, 0xa3, 0xa0].contains(&key) { i16::MIN } else { 0 },+ || panic!("must not post")).unwrap_err();+ assert_eq!(error.message(false), "Ctrl and Shift are held down, so nothing was sent. Release Ctrl and Shift and try again.");+ }++ #[test]+ fn own_combo_modifiers_are_pressed_only_after_the_snapshot() {+ let injecting = Cell::new(false);+ without_held_modifiers(|_| {+ assert!(!injecting.get(), "must not read our own injected modifier");+ 1 // Previously pressed, but no longer held.+ }, || {+ injecting.set(true);+ Ok(())+ }).unwrap();+ assert!(injecting.get());+ }+}/// One notch of the mouse wheel (`WHEEL_DELTA`).pub const WHEEL_DELTA: i32 = 120;diff --git a/docs/design.md b/docs/design.mdindex f32cbf3..ff92438 100644--- a/docs/design.md+++ b/docs/design.md@@ -173,6 +173,7 @@ The assistant can see the screen and operate apps through a built-in MCP server- S24.2 Tools: `screenshot`, `click`, `move`, `drag`, `scroll`, `type`, `key`, `open_app`, `list_apps`, `wait`, each with an optional `intent`. Screenshots of the display under the pointer are downscaled to a long edge of 1280 px; coordinates are pixels of the latest screenshot; actions return a fresh one; Gentle Dot's own windows are left out.- S24.3 Session grant: the first call without a session shows a native dialog ("Allow Gentle Dot to see and control this Mac for 30 minutes?"). A session ends on timeout, Stop, the tray item, or the panic shortcut ⌥⇧Esc, and queued actions are dropped.- S24.4 Guards: a fixed blocklist (Gentle Dot itself, System Settings, SecurityAgent, loginwindow, Keychain Access, password managers), 10 actions per second, and a native per-action confirmation for risky actions (send, pay, buy, delete, submit, confirm, transfer, and their Spanish equivalents, by Accessibility role and title, by `return` on a default button, or by the declared `intent`). Keyboard actions go only to the app that was steadily in front during the latest screenshot; their recipient is fixed when they are checked and confirmed and rechecked before each chunk of text, each Return, and the key, and a change stops the input with the count of characters already sent. After a confirmation, the app in front before the dialog must be back in front within 500 ms, or nothing is done.+ - Windows held-modifier guard (T59): before each keyboard post (text chunk, Return, or key), read both sides of Ctrl, Alt, Shift and Win with `GetAsyncKeyState`. Refuse before pressing the event's own modifiers, so an injected `ctrl+t` does not refuse itself. A held Ctrl reports exactly "Ctrl is held down, so nothing was sent. Release Ctrl and try again." Multiple held keys are named once in Ctrl/Alt/Shift/Win order, e.g. "Ctrl and Shift are held down, so nothing was sent. Release Ctrl and Shift and try again." After earlier text landed, use "the rest was not sent" and append the character count, e.g. "Shift is held down, so the rest was not sent. Release Shift and try again. Typing stopped after 5 of 11 characters." Before any text lands, the suffix is "Typing stopped after 0 of M characters." This is a pre-post snapshot, not a fence against keys pressed during the post. Pointer actions do not check held modifiers yet; macOS/Linux behavior is unchanged.- S24.5 Permissions: the app reports and requests Accessibility and Screen Recording (`computer_permissions`, `computer_request_permission {kind}`). The Connectors screen shows a built-in "Computer" entry with both statuses, Grant buttons, Check again, the note that debug builds lose the grants on every rebuild (macOS ties them to the code signature), and a note when the current model does not accept images (`ConnectorInfo.noImages`, from the model's `input` in the engine's `get_state` and `set_model` answers).- S24.6 Visible state: the app emits `computer://state {active, endsAt?, reason?}` (`computer_status` gives the current one). Between calls, the rose shows the "in control" state (`dot-in_control`, mood `control`: accent rim, steady traveling light) and the panel shows "Controlling your Mac · mm:ss left" with Stop (`computer_stop`). During a screenshot or action call (including an entire batch and its final screenshot), the Dot and panel are temporarily hidden and the top edge of the controlled display glows in the rose's neon with a light traveling along it, without text or logo. The click-through glow must be ready and hiding confirmed before capture or input; failed preparation sends no input. Completion and normal error/Stop paths restore prior visibility without changing saved preferences or taking focus. The tray's "Stop computer control" and the panic shortcut remain independent of the hidden panel; shortcut registration and full-screen tray availability must be verified on the target Mac.- S24.7 Wiring: the app registers its helper itself, over the app channel, right after it spawns the daemon (`computer_register {url, token}`; the daemon accepts only `http://127.0.0.1:<port>/mcp` and a header-safe key). A window cannot register, repoint, or remove it (L61 closed): the panel only calls `computer_endpoint` to know whether a helper exists. The daemon keeps the endpoint in memory, never in `connectors.json`, and renders `computer: {url, headers: {Authorization: "Bearer <key>"}, exposure: "direct"}` into `mcp.json`, then restarts the engine when idle. The guard's policy lists it under `builtin`, so its tools run without in-engine approval cards. It is removed on `computer_unregister` from the app or when the app's channel closes; windows coming and going do not touch it. A daemon without a channel (`pnpm dev`) gets no helper; the app never uses one (§9, Daemon lifecycle). The key is never logged or sent to a window: `connectors` lists only a built-in "Computer" entry (`builtin: true`) that the connector messages cannot change or remove, no saved or imported connector can take the name `computer`, and the import scan never offers the helper.