Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): let embedding hosts set the macOS key gap (#3489)
ATG310cdfd58963
diff --git a/docs/content/docs/cua-driver/guides/use-the-sdk.mdx b/docs/content/docs/cua-driver/guides/use-the-sdk.mdxindex 3ad58746df..141b580410 100644--- a/docs/content/docs/cua-driver/guides/use-the-sdk.mdx+++ b/docs/content/docs/cua-driver/guides/use-the-sdk.mdx@@ -297,6 +297,12 @@ Python exposes the same objects:`environment` option. On macOS the wait also holds the focus-restorelease, so a shorter wait shortens that protection. Windows ignores both.+- **Shorten macOS keystroke pacing.** The macOS adapter waits 8 ms between+ synthesized key events. Set `CUA_DRIVER_KEY_GAP_MS` (2 to 100) in the same+ launch environment to change it; `EmbeddedCuaDriverHost` admits it. Only+ macOS reads it. Some targets drop key events that arrive too close together,+ so read the field back before relying on a low value.+Without the SDK, the equivalent low-level launch is:```sh test="cli-shape" id="cua-driver-serve"diff --git a/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md b/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.mdindex 63a99eb39a..a00eba98be 100644--- a/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md+++ b/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md@@ -257,6 +257,32 @@ Measured on macOS with Calculator in the background and Terminal frontmost, abackground AX click took 1153 ms with the default, 325 ms with `200`, and 78 mswith `0`.+## Pacing synthesized keystrokes (macOS)++The macOS keyboard adapter waits a fixed gap between each key down and up and+between consecutive keys. For `type_text`, `press_key`, and `hotkey`, that gap+is most of the call's latency. A host whose targets accept faster input can+shorten it at trusted launch with one variable, in the same environment as the+window-observation variables above. `EmbeddedCuaDriverHost` admits it too:++| Variable | Meaning | Default | Accepted range |+| ----------------------- | ----------------------------------------------- | ------- | ----------------------------------------------------- |+| `CUA_DRIVER_KEY_GAP_MS` | Gap between synthesized key events, per event. | 8 | 2 to 100; smaller values are raised, larger clamped |++Unset, empty, or unparsable values keep the default. No tool argument can+change the gap. An explicit `type_text` `delay_ms` still applies on top of it.++- **macOS only.** Linux keeps its fixed 10 ms key delay and Windows its own+ pacing; neither reads this variable.+- **What a shorter gap costs.** Some targets coalesce or drop key events that+ arrive back to back, most often remote-input forwarders and apps under heavy+ load. Read the field back after typing before relying on a low value for a+ new target. The 2 ms floor exists because `0` posts events with no gap at all.++Measured on macOS 27 with TextEdit in the background, typing a 185-character+ASCII sentence through the PID-routed key path took about 3650 ms at the default+and about 980 ms at `2`. All ten read-backs (five per value) matched exactly.+## What embedded mode changes (and what it doesn't)| | Standalone | Embedded (`CUA_DRIVER_EMBEDDED=1`) |diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/key_pacing.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/key_pacing.rsnew file mode 100644index 0000000000..ed6a5cea99--- /dev/null+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/key_pacing.rs@@ -0,0 +1,85 @@+//! Host-chosen gap between synthesized key events (macOS only).+//!+//! The macOS keyboard adapter sleeps a fixed gap between each key down and+//! up and between consecutive keys. That gap is most of the latency of a+//! `type_text` or `hotkey` call. An embedding host whose targets accept+//! faster input can shorten it at trusted launch through the daemon+//! environment:+//!+//! - [`KEY_GAP_ENV`] (`CUA_DRIVER_KEY_GAP_MS`): the gap in milliseconds.+//!+//! Unset, empty, or unparsable values keep the adapter's default, so a daemon+//! launched without the variable behaves exactly as before. The value is read+//! from the daemon process environment only; no tool argument can change it.+//! Linux and Windows keep their own fixed pacing and do not read it. See+//! `Skills/cua-driver/EMBEDDING.md` for what a shorter gap costs.++use std::time::Duration;++/// Gap between synthesized key events, in milliseconds.+pub const KEY_GAP_ENV: &str = "CUA_DRIVER_KEY_GAP_MS";++/// Smallest accepted gap. Smaller values (including `0`) are raised, because+/// some targets coalesce or drop events posted back to back.+pub const MIN_KEY_GAP: Duration = Duration::from_millis(2);++/// Largest accepted gap. Larger values are clamped so a typo cannot make+/// every keystroke wait for seconds.+pub const MAX_KEY_GAP: Duration = Duration::from_millis(100);++/// Resolve the key gap from a raw environment value. Pure, so it can be+/// tested without touching the process environment.+pub fn key_gap_from_raw(raw: Option<&str>, default_gap: Duration) -> Duration {+ raw.and_then(|r| r.trim().parse::<u64>().ok())+ .map(Duration::from_millis)+ .unwrap_or(default_gap)+ .clamp(MIN_KEY_GAP, MAX_KEY_GAP)+}++/// Resolve the key gap from the daemon environment, falling back to the+/// adapter's default when unset or unparsable.+pub fn key_gap_from_env(default_gap: Duration) -> Duration {+ key_gap_from_raw(std::env::var(KEY_GAP_ENV).ok().as_deref(), default_gap)+}++#[cfg(test)]+mod tests {+ use super::*;++ const DEFAULT: Duration = Duration::from_millis(8);++ #[test]+ fn unset_keeps_adapter_default() {+ assert_eq!(key_gap_from_raw(None, DEFAULT), DEFAULT);+ }++ #[test]+ fn valid_values_are_honored() {+ assert_eq!(+ key_gap_from_raw(Some("4"), DEFAULT),+ Duration::from_millis(4)+ );+ assert_eq!(+ key_gap_from_raw(Some(" 20\n"), DEFAULT),+ Duration::from_millis(20)+ );+ }++ #[test]+ fn invalid_values_fall_back_to_default() {+ for raw in ["", " ", "abc", "-1", "1.5", "8ms", "18446744073709551616"] {+ assert_eq!(key_gap_from_raw(Some(raw), DEFAULT), DEFAULT, "{raw:?}");+ }+ }++ #[test]+ fn out_of_range_values_are_clamped() {+ assert_eq!(key_gap_from_raw(Some("0"), DEFAULT), MIN_KEY_GAP);+ assert_eq!(key_gap_from_raw(Some("1"), DEFAULT), MIN_KEY_GAP);+ assert_eq!(key_gap_from_raw(Some("60000"), DEFAULT), MAX_KEY_GAP);+ assert_eq!(+ key_gap_from_raw(Some(&u64::MAX.to_string()), DEFAULT),+ MAX_KEY_GAP+ );+ }+}diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rsindex c61406ad59..33217a6c7b 100644--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs@@ -78,6 +78,7 @@ pub mod health_report;pub mod history;pub mod image_utils;pub mod interactive_input;+pub mod key_pacing;pub mod launch_guard;pub mod mcp_result;pub mod page;diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rsindex f17daccc3b..1883e39bc0 100644--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs@@ -7,6 +7,7 @@ use cua_driver_contract::{CAPABILITY_VERSION, CONTRACT_VERSION, MCP_PROTOCOL_VERSION, TOOLS_LIST_SCHEMA_VERSION,};use cua_driver_core::daemon::{request_daemon_metadata, DaemonMetadata};+use cua_driver_core::key_pacing::KEY_GAP_ENV;use cua_driver_core::window_observation::{WINDOW_CHANGE_POLL_ENV, WINDOW_CHANGE_TIMEOUT_ENV};use std::collections::BTreeMap;use std::process::Stdio;@@ -838,6 +839,7 @@ pub(crate) fn allowed_environment_name(name: &str) -> bool {upper.starts_with("LC_")|| upper == WINDOW_CHANGE_TIMEOUT_ENV|| upper == WINDOW_CHANGE_POLL_ENV+ || upper == KEY_GAP_ENV|| matches!(upper.as_str(),"PATH"@@ -1431,4 +1433,26 @@ mod tests {assert_eq!(host.state(), EmbeddedDriverHostState::Stopped);assert!(!socket_path.exists());}++ /// `CUA_DRIVER_KEY_GAP_MS` must survive both propagation paths into a child launch —+ /// inherited from the parent environment and supplied as an explicit+ /// override. An allowlist miss silently strips the deployment's+ /// override in embedded/worker modes.+ #[test]+ fn key_gap_env_propagates() {+ assert!(allowed_environment_name("CUA_DRIVER_KEY_GAP_MS"));+ let merged =+ merge_safe_environment([("CUA_DRIVER_KEY_GAP_MS".to_owned(), "7".to_owned())], &[]);+ assert!(merged+ .iter()+ .any(|v| v.name.eq_ignore_ascii_case("CUA_DRIVER_KEY_GAP_MS") && v.value == "7"));+ let overrides = vec![EmbeddedEnvironmentVariable {+ name: "CUA_DRIVER_KEY_GAP_MS".to_owned(),+ value: "3".to_owned(),+ }];+ let merged = merge_safe_environment(std::iter::empty(), &overrides);+ assert!(merged+ .iter()+ .any(|v| v.name.eq_ignore_ascii_case("CUA_DRIVER_KEY_GAP_MS") && v.value == "3"));+ }}diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rsindex 5ddb3057de..5b8128e42e 100644--- a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs@@ -14,6 +14,21 @@ use foreign_types::ForeignType;const SCREEN_SHARING_BUNDLE_ID: &str = "com.apple.ScreenSharing";const SHIFT_KEY_CODE: u16 = 56;+/// Default key down→up gap and inter-key pacing.+const DEFAULT_KEY_GAP: std::time::Duration = std::time::Duration::from_millis(8);++/// Key down→up gap and inter-key pacing: `DEFAULT_KEY_GAP`, or the+/// host's `CUA_DRIVER_KEY_GAP_MS` clamped by `cua_driver_core::key_pacing`.+/// Read once: pacing is a launch-time knob, not a per-call one.+fn key_gap() -> std::time::Duration {+ static GAP: std::sync::OnceLock<std::time::Duration> = std::sync::OnceLock::new();+ *GAP.get_or_init(|| cua_driver_core::key_pacing::key_gap_from_env(DEFAULT_KEY_GAP))+}++fn key_gap_sleep() {+ std::thread::sleep(key_gap());+}+fn is_screen_sharing_bundle_id(bundle_id: &str) -> bool {bundle_id == SCREEN_SHARING_BUNDLE_ID}@@ -37,7 +52,7 @@ pub fn press_key(pid: i32, key: &str, modifiers: &[&str]) -> anyhow::Result<()>let flags = modifier_flags(&["shift"]);let eq_code = key_name_to_code("=")?;post_key(pid, eq_code, true, modifier_flags(modifiers) | flags)?;- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();post_key(pid, eq_code, false, modifier_flags(modifiers) | flags)?;return Ok(());}@@ -46,7 +61,7 @@ pub fn press_key(pid: i32, key: &str, modifiers: &[&str]) -> anyhow::Result<()>let flags = modifier_flags(modifiers);post_key(pid, key_code, true, flags)?;- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();post_key(pid, key_code, false, flags)?;Ok(())}@@ -66,20 +81,20 @@ pub fn type_text(pid: i32, text: &str) -> anyhow::Result<()> {// and the modifier leaks into the next character (Swift fix: event.flags = []).down.set_flags(CGEventFlags::CGEventFlagNull);post_keyboard_event(pid, &down);- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();let up = CGEvent::new_keyboard_event(source.clone(), 0, false).map_err(|_| anyhow::anyhow!("CGEvent keyboard up failed"))?;up.set_string(&ch_str);up.set_flags(CGEventFlags::CGEventFlagNull);post_keyboard_event(pid, &up);- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();}Ok(())}/// Type a string character-by-character with an extra `inter_char_delay_ms`-/// pause after each character (on top of the internal 8 ms down/up gap).+/// pause after each character (on top of the internal down/up key gap).pub fn type_text_with_delay(pid: i32, text: &str, inter_char_delay_ms: u64) -> anyhow::Result<()> {let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState).map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;@@ -91,7 +106,7 @@ pub fn type_text_with_delay(pid: i32, text: &str, inter_char_delay_ms: u64) -> adown.set_string(&ch_str);down.set_flags(CGEventFlags::CGEventFlagNull);post_keyboard_event(pid, &down);- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();let up = CGEvent::new_keyboard_event(source.clone(), 0, false).map_err(|_| anyhow::anyhow!("CGEvent keyboard up failed"))?;@@ -99,11 +114,11 @@ pub fn type_text_with_delay(pid: i32, text: &str, inter_char_delay_ms: u64) -> aup.set_flags(CGEventFlags::CGEventFlagNull);post_keyboard_event(pid, &up);- // Additional inter-character delay on top of the 8 ms internal gap.+ // Additional inter-character delay on top of the internal key gap.if inter_char_delay_ms > 0 {std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms));} else {- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();}}Ok(())@@ -124,7 +139,7 @@ pub fn hotkey_no_auth(pid: i32, key: &str, modifiers: &[&str]) -> anyhow::Resultlet key_code = key_name_to_code(key)?;let flags = modifier_flags(modifiers);post_key_no_auth(pid, key_code, true, flags)?;- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();post_key_no_auth(pid, key_code, false, flags)?;Ok(())}@@ -135,7 +150,7 @@ pub fn press_key_no_auth(pid: i32, key: &str, modifiers: &[&str]) -> anyhow::Reslet key_code = key_name_to_code(key)?;let flags = modifier_flags(modifiers);post_key_no_auth(pid, key_code, true, flags)?;- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();post_key_no_auth(pid, key_code, false, flags)?;Ok(())}@@ -186,7 +201,7 @@ pub fn press_key_global(key: &str, modifiers: &[&str]) -> anyhow::Result<()> {return Err(error);}pressed_modifiers.push((modifier_code, modifier_flag));- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();}let result = (|| {@@ -197,7 +212,7 @@ pub fn press_key_global(key: &str, modifiers: &[&str]) -> anyhow::Result<()> {active_flags,CGEventTapLocation::HID,)?;- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();post_global_key(&source,key_code,@@ -255,7 +270,7 @@ pub(super) fn with_global_modifier_keys<T>(return Err(error);}pressed.push((key_code, flag));- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();}let result = gesture(active_flags);@@ -289,7 +304,7 @@ fn release_global_modifiers(event.set_flags(active_flags);event.post(tap);}- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();}}@@ -337,7 +352,7 @@ pub(super) fn with_pid_modifier_keys<T>(return Err(error);}pressed.push((key_code, flag));- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();}let result = gesture();@@ -353,7 +368,7 @@ fn release_pid_modifiers(for &(key_code, flag) in pressed.iter().rev() {active_flags.remove(flag);let _ = post_key(pid, key_code, false, active_flags);- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();}}@@ -372,13 +387,13 @@ pub fn type_text_global(text: &str, inter_char_delay_ms: u64) -> anyhow::Result<down.set_string(&value);down.set_flags(CGEventFlags::CGEventFlagNull);down.post(CGEventTapLocation::HID);- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();let up = CGEvent::new_keyboard_event(source.clone(), 0, false).map_err(|_| anyhow::anyhow!("CGEvent keyboard up failed"))?;up.set_string(&value);up.set_flags(CGEventFlags::CGEventFlagNull);up.post(CGEventTapLocation::HID);- std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms.max(8)));+ std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms).max(key_gap()));}Ok(())}@@ -407,10 +422,11 @@ pub fn type_text_physical_global(text: &str, inter_char_delay_ms: u64) -> anyhow.collect::<anyhow::Result<Vec<_>>>()?;for cg_event in native_events {cg_event.post(CGEventTapLocation::HID);- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();}- if inter_char_delay_ms > 8 {- std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms - 8));+ let extra = std::time::Duration::from_millis(inter_char_delay_ms).saturating_sub(key_gap());+ if !extra.is_zero() {+ std::thread::sleep(extra);}}Ok(())@@ -451,7 +467,7 @@ pub fn press_key_bare_global(key: &str, modifiers: &[&str]) -> anyhow::Result<().collect::<anyhow::Result<Vec<_>>>()?;for event in events {event.post(CGEventTapLocation::HID);- std::thread::sleep(std::time::Duration::from_millis(8));+ key_gap_sleep();}Ok(())}diff --git a/libs/cua/skills/cua-driver/EMBEDDING.md b/libs/cua/skills/cua-driver/EMBEDDING.mdindex 63a99eb39a..a00eba98be 100644--- a/libs/cua/skills/cua-driver/EMBEDDING.md+++ b/libs/cua/skills/cua-driver/EMBEDDING.md@@ -257,6 +257,32 @@ Measured on macOS with Calculator in the background and Terminal frontmost, abackground AX click took 1153 ms with the default, 325 ms with `200`, and 78 mswith `0`.+## Pacing synthesized keystrokes (macOS)++The macOS keyboard adapter waits a fixed gap between each key down and up and+between consecutive keys. For `type_text`, `press_key`, and `hotkey`, that gap+is most of the call's latency. A host whose targets accept faster input can+shorten it at trusted launch with one variable, in the same environment as the+window-observation variables above. `EmbeddedCuaDriverHost` admits it too:++| Variable | Meaning | Default | Accepted range |+| ----------------------- | ----------------------------------------------- | ------- | ----------------------------------------------------- |+| `CUA_DRIVER_KEY_GAP_MS` | Gap between synthesized key events, per event. | 8 | 2 to 100; smaller values are raised, larger clamped |++Unset, empty, or unparsable values keep the default. No tool argument can+change the gap. An explicit `type_text` `delay_ms` still applies on top of it.++- **macOS only.** Linux keeps its fixed 10 ms key delay and Windows its own+ pacing; neither reads this variable.+- **What a shorter gap costs.** Some targets coalesce or drop key events that+ arrive back to back, most often remote-input forwarders and apps under heavy+ load. Read the field back after typing before relying on a low value for a+ new target. The 2 ms floor exists because `0` posts events with no gap at all.++Measured on macOS 27 with TextEdit in the background, typing a 185-character+ASCII sentence through the PID-routed key path took about 3650 ms at the default+and about 980 ms at `2`. All ten read-backs (five per value) matched exactly.+## What embedded mode changes (and what it doesn't)| | Standalone | Embedded (`CUA_DRIVER_EMBEDDED=1`) |