Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): let embedding hosts set the macOS key gap (#3489)

ATG310cdfd58963
diff --git a/docs/content/docs/cua-driver/guides/use-the-sdk.mdx b/docs/content/docs/cua-driver/guides/use-the-sdk.mdx
index 3ad58746df..141b580410 100644
--- a/docs/content/docs/cua-driver/guides/use-the-sdk.mdx
+++ b/docs/content/docs/cua-driver/guides/use-the-sdk.mdx
@@ -297,6 +297,12 @@ Python exposes the same objects:
`environment` option. On macOS the wait also holds the focus-restore
lease, so a shorter wait shortens that protection. Windows ignores both.
+- **Shorten macOS keystroke pacing.** The macOS adapter waits 8 ms between
+ synthesized key events. Set `CUA_DRIVER_KEY_GAP_MS` (2 to 100) in the same
+ launch environment to change it; `EmbeddedCuaDriverHost` admits it. Only
+ macOS reads it. Some targets drop key events that arrive too close together,
+ so read the field back before relying on a low value.
+
Without the SDK, the equivalent low-level launch is:
```sh test="cli-shape" id="cua-driver-serve"
diff --git a/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md b/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md
index 63a99eb39a..a00eba98be 100644
--- a/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md
+++ b/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md
@@ -257,6 +257,32 @@ Measured on macOS with Calculator in the background and Terminal frontmost, a
background AX click took 1153 ms with the default, 325 ms with `200`, and 78 ms
with `0`.
+## Pacing synthesized keystrokes (macOS)
+
+The macOS keyboard adapter waits a fixed gap between each key down and up and
+between consecutive keys. For `type_text`, `press_key`, and `hotkey`, that gap
+is most of the call's latency. A host whose targets accept faster input can
+shorten it at trusted launch with one variable, in the same environment as the
+window-observation variables above. `EmbeddedCuaDriverHost` admits it too:
+
+| Variable | Meaning | Default | Accepted range |
+| ----------------------- | ----------------------------------------------- | ------- | ----------------------------------------------------- |
+| `CUA_DRIVER_KEY_GAP_MS` | Gap between synthesized key events, per event. | 8 | 2 to 100; smaller values are raised, larger clamped |
+
+Unset, empty, or unparsable values keep the default. No tool argument can
+change the gap. An explicit `type_text` `delay_ms` still applies on top of it.
+
+- **macOS only.** Linux keeps its fixed 10 ms key delay and Windows its own
+ pacing; neither reads this variable.
+- **What a shorter gap costs.** Some targets coalesce or drop key events that
+ arrive back to back, most often remote-input forwarders and apps under heavy
+ load. Read the field back after typing before relying on a low value for a
+ new target. The 2 ms floor exists because `0` posts events with no gap at all.
+
+Measured on macOS 27 with TextEdit in the background, typing a 185-character
+ASCII sentence through the PID-routed key path took about 3650 ms at the default
+and about 980 ms at `2`. All ten read-backs (five per value) matched exactly.
+
## What embedded mode changes (and what it doesn't)
| | Standalone | Embedded (`CUA_DRIVER_EMBEDDED=1`) |
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/key_pacing.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/key_pacing.rs
new file mode 100644
index 0000000000..ed6a5cea99
--- /dev/null
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/key_pacing.rs
@@ -0,0 +1,85 @@
+//! Host-chosen gap between synthesized key events (macOS only).
+//!
+//! The macOS keyboard adapter sleeps a fixed gap between each key down and
+//! up and between consecutive keys. That gap is most of the latency of a
+//! `type_text` or `hotkey` call. An embedding host whose targets accept
+//! faster input can shorten it at trusted launch through the daemon
+//! environment:
+//!
+//! - [`KEY_GAP_ENV`] (`CUA_DRIVER_KEY_GAP_MS`): the gap in milliseconds.
+//!
+//! Unset, empty, or unparsable values keep the adapter's default, so a daemon
+//! launched without the variable behaves exactly as before. The value is read
+//! from the daemon process environment only; no tool argument can change it.
+//! Linux and Windows keep their own fixed pacing and do not read it. See
+//! `Skills/cua-driver/EMBEDDING.md` for what a shorter gap costs.
+
+use std::time::Duration;
+
+/// Gap between synthesized key events, in milliseconds.
+pub const KEY_GAP_ENV: &str = "CUA_DRIVER_KEY_GAP_MS";
+
+/// Smallest accepted gap. Smaller values (including `0`) are raised, because
+/// some targets coalesce or drop events posted back to back.
+pub const MIN_KEY_GAP: Duration = Duration::from_millis(2);
+
+/// Largest accepted gap. Larger values are clamped so a typo cannot make
+/// every keystroke wait for seconds.
+pub const MAX_KEY_GAP: Duration = Duration::from_millis(100);
+
+/// Resolve the key gap from a raw environment value. Pure, so it can be
+/// tested without touching the process environment.
+pub fn key_gap_from_raw(raw: Option<&str>, default_gap: Duration) -> Duration {
+ raw.and_then(|r| r.trim().parse::<u64>().ok())
+ .map(Duration::from_millis)
+ .unwrap_or(default_gap)
+ .clamp(MIN_KEY_GAP, MAX_KEY_GAP)
+}
+
+/// Resolve the key gap from the daemon environment, falling back to the
+/// adapter's default when unset or unparsable.
+pub fn key_gap_from_env(default_gap: Duration) -> Duration {
+ key_gap_from_raw(std::env::var(KEY_GAP_ENV).ok().as_deref(), default_gap)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ const DEFAULT: Duration = Duration::from_millis(8);
+
+ #[test]
+ fn unset_keeps_adapter_default() {
+ assert_eq!(key_gap_from_raw(None, DEFAULT), DEFAULT);
+ }
+
+ #[test]
+ fn valid_values_are_honored() {
+ assert_eq!(
+ key_gap_from_raw(Some("4"), DEFAULT),
+ Duration::from_millis(4)
+ );
+ assert_eq!(
+ key_gap_from_raw(Some(" 20\n"), DEFAULT),
+ Duration::from_millis(20)
+ );
+ }
+
+ #[test]
+ fn invalid_values_fall_back_to_default() {
+ for raw in ["", " ", "abc", "-1", "1.5", "8ms", "18446744073709551616"] {
+ assert_eq!(key_gap_from_raw(Some(raw), DEFAULT), DEFAULT, "{raw:?}");
+ }
+ }
+
+ #[test]
+ fn out_of_range_values_are_clamped() {
+ assert_eq!(key_gap_from_raw(Some("0"), DEFAULT), MIN_KEY_GAP);
+ assert_eq!(key_gap_from_raw(Some("1"), DEFAULT), MIN_KEY_GAP);
+ assert_eq!(key_gap_from_raw(Some("60000"), DEFAULT), MAX_KEY_GAP);
+ assert_eq!(
+ key_gap_from_raw(Some(&u64::MAX.to_string()), DEFAULT),
+ MAX_KEY_GAP
+ );
+ }
+}
diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
index c61406ad59..33217a6c7b 100644
--- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs
@@ -78,6 +78,7 @@ pub mod health_report;
pub mod history;
pub mod image_utils;
pub mod interactive_input;
+pub mod key_pacing;
pub mod launch_guard;
pub mod mcp_result;
pub mod page;
diff --git a/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs b/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs
index f17daccc3b..1883e39bc0 100644
--- a/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs
+++ b/libs/cua-driver/rust/crates/cua-driver-sdk/src/embedded.rs
@@ -7,6 +7,7 @@ use cua_driver_contract::{
CAPABILITY_VERSION, CONTRACT_VERSION, MCP_PROTOCOL_VERSION, TOOLS_LIST_SCHEMA_VERSION,
};
use cua_driver_core::daemon::{request_daemon_metadata, DaemonMetadata};
+use cua_driver_core::key_pacing::KEY_GAP_ENV;
use cua_driver_core::window_observation::{WINDOW_CHANGE_POLL_ENV, WINDOW_CHANGE_TIMEOUT_ENV};
use std::collections::BTreeMap;
use std::process::Stdio;
@@ -838,6 +839,7 @@ pub(crate) fn allowed_environment_name(name: &str) -> bool {
upper.starts_with("LC_")
|| upper == WINDOW_CHANGE_TIMEOUT_ENV
|| upper == WINDOW_CHANGE_POLL_ENV
+ || upper == KEY_GAP_ENV
|| matches!(
upper.as_str(),
"PATH"
@@ -1431,4 +1433,26 @@ mod tests {
assert_eq!(host.state(), EmbeddedDriverHostState::Stopped);
assert!(!socket_path.exists());
}
+
+ /// `CUA_DRIVER_KEY_GAP_MS` must survive both propagation paths into a child launch —
+ /// inherited from the parent environment and supplied as an explicit
+ /// override. An allowlist miss silently strips the deployment's
+ /// override in embedded/worker modes.
+ #[test]
+ fn key_gap_env_propagates() {
+ assert!(allowed_environment_name("CUA_DRIVER_KEY_GAP_MS"));
+ let merged =
+ merge_safe_environment([("CUA_DRIVER_KEY_GAP_MS".to_owned(), "7".to_owned())], &[]);
+ assert!(merged
+ .iter()
+ .any(|v| v.name.eq_ignore_ascii_case("CUA_DRIVER_KEY_GAP_MS") && v.value == "7"));
+ let overrides = vec![EmbeddedEnvironmentVariable {
+ name: "CUA_DRIVER_KEY_GAP_MS".to_owned(),
+ value: "3".to_owned(),
+ }];
+ let merged = merge_safe_environment(std::iter::empty(), &overrides);
+ assert!(merged
+ .iter()
+ .any(|v| v.name.eq_ignore_ascii_case("CUA_DRIVER_KEY_GAP_MS") && v.value == "3"));
+ }
}
diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
index 5ddb3057de..5b8128e42e 100644
--- a/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
+++ b/libs/cua-driver/rust/crates/platform-macos/src/input/keyboard.rs
@@ -14,6 +14,21 @@ use foreign_types::ForeignType;
const SCREEN_SHARING_BUNDLE_ID: &str = "com.apple.ScreenSharing";
const SHIFT_KEY_CODE: u16 = 56;
+/// Default key down→up gap and inter-key pacing.
+const DEFAULT_KEY_GAP: std::time::Duration = std::time::Duration::from_millis(8);
+
+/// Key down→up gap and inter-key pacing: `DEFAULT_KEY_GAP`, or the
+/// host's `CUA_DRIVER_KEY_GAP_MS` clamped by `cua_driver_core::key_pacing`.
+/// Read once: pacing is a launch-time knob, not a per-call one.
+fn key_gap() -> std::time::Duration {
+ static GAP: std::sync::OnceLock<std::time::Duration> = std::sync::OnceLock::new();
+ *GAP.get_or_init(|| cua_driver_core::key_pacing::key_gap_from_env(DEFAULT_KEY_GAP))
+}
+
+fn key_gap_sleep() {
+ std::thread::sleep(key_gap());
+}
+
fn is_screen_sharing_bundle_id(bundle_id: &str) -> bool {
bundle_id == SCREEN_SHARING_BUNDLE_ID
}
@@ -37,7 +52,7 @@ pub fn press_key(pid: i32, key: &str, modifiers: &[&str]) -> anyhow::Result<()>
let flags = modifier_flags(&["shift"]);
let eq_code = key_name_to_code("=")?;
post_key(pid, eq_code, true, modifier_flags(modifiers) | flags)?;
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
post_key(pid, eq_code, false, modifier_flags(modifiers) | flags)?;
return Ok(());
}
@@ -46,7 +61,7 @@ pub fn press_key(pid: i32, key: &str, modifiers: &[&str]) -> anyhow::Result<()>
let flags = modifier_flags(modifiers);
post_key(pid, key_code, true, flags)?;
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
post_key(pid, key_code, false, flags)?;
Ok(())
}
@@ -66,20 +81,20 @@ pub fn type_text(pid: i32, text: &str) -> anyhow::Result<()> {
// and the modifier leaks into the next character (Swift fix: event.flags = []).
down.set_flags(CGEventFlags::CGEventFlagNull);
post_keyboard_event(pid, &down);
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
let up = CGEvent::new_keyboard_event(source.clone(), 0, false)
.map_err(|_| anyhow::anyhow!("CGEvent keyboard up failed"))?;
up.set_string(&ch_str);
up.set_flags(CGEventFlags::CGEventFlagNull);
post_keyboard_event(pid, &up);
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
}
Ok(())
}
/// Type a string character-by-character with an extra `inter_char_delay_ms`
-/// pause after each character (on top of the internal 8 ms down/up gap).
+/// pause after each character (on top of the internal down/up key gap).
pub fn type_text_with_delay(pid: i32, text: &str, inter_char_delay_ms: u64) -> anyhow::Result<()> {
let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)
.map_err(|_| anyhow::anyhow!("CGEventSource::new failed"))?;
@@ -91,7 +106,7 @@ pub fn type_text_with_delay(pid: i32, text: &str, inter_char_delay_ms: u64) -> a
down.set_string(&ch_str);
down.set_flags(CGEventFlags::CGEventFlagNull);
post_keyboard_event(pid, &down);
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
let up = CGEvent::new_keyboard_event(source.clone(), 0, false)
.map_err(|_| anyhow::anyhow!("CGEvent keyboard up failed"))?;
@@ -99,11 +114,11 @@ pub fn type_text_with_delay(pid: i32, text: &str, inter_char_delay_ms: u64) -> a
up.set_flags(CGEventFlags::CGEventFlagNull);
post_keyboard_event(pid, &up);
- // Additional inter-character delay on top of the 8 ms internal gap.
+ // Additional inter-character delay on top of the internal key gap.
if inter_char_delay_ms > 0 {
std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms));
} else {
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
}
}
Ok(())
@@ -124,7 +139,7 @@ pub fn hotkey_no_auth(pid: i32, key: &str, modifiers: &[&str]) -> anyhow::Result
let key_code = key_name_to_code(key)?;
let flags = modifier_flags(modifiers);
post_key_no_auth(pid, key_code, true, flags)?;
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
post_key_no_auth(pid, key_code, false, flags)?;
Ok(())
}
@@ -135,7 +150,7 @@ pub fn press_key_no_auth(pid: i32, key: &str, modifiers: &[&str]) -> anyhow::Res
let key_code = key_name_to_code(key)?;
let flags = modifier_flags(modifiers);
post_key_no_auth(pid, key_code, true, flags)?;
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
post_key_no_auth(pid, key_code, false, flags)?;
Ok(())
}
@@ -186,7 +201,7 @@ pub fn press_key_global(key: &str, modifiers: &[&str]) -> anyhow::Result<()> {
return Err(error);
}
pressed_modifiers.push((modifier_code, modifier_flag));
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
}
let result = (|| {
@@ -197,7 +212,7 @@ pub fn press_key_global(key: &str, modifiers: &[&str]) -> anyhow::Result<()> {
active_flags,
CGEventTapLocation::HID,
)?;
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
post_global_key(
&source,
key_code,
@@ -255,7 +270,7 @@ pub(super) fn with_global_modifier_keys<T>(
return Err(error);
}
pressed.push((key_code, flag));
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
}
let result = gesture(active_flags);
@@ -289,7 +304,7 @@ fn release_global_modifiers(
event.set_flags(active_flags);
event.post(tap);
}
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
}
}
@@ -337,7 +352,7 @@ pub(super) fn with_pid_modifier_keys<T>(
return Err(error);
}
pressed.push((key_code, flag));
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
}
let result = gesture();
@@ -353,7 +368,7 @@ fn release_pid_modifiers(
for &(key_code, flag) in pressed.iter().rev() {
active_flags.remove(flag);
let _ = post_key(pid, key_code, false, active_flags);
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
}
}
@@ -372,13 +387,13 @@ pub fn type_text_global(text: &str, inter_char_delay_ms: u64) -> anyhow::Result<
down.set_string(&value);
down.set_flags(CGEventFlags::CGEventFlagNull);
down.post(CGEventTapLocation::HID);
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
let up = CGEvent::new_keyboard_event(source.clone(), 0, false)
.map_err(|_| anyhow::anyhow!("CGEvent keyboard up failed"))?;
up.set_string(&value);
up.set_flags(CGEventFlags::CGEventFlagNull);
up.post(CGEventTapLocation::HID);
- std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms.max(8)));
+ std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms).max(key_gap()));
}
Ok(())
}
@@ -407,10 +422,11 @@ pub fn type_text_physical_global(text: &str, inter_char_delay_ms: u64) -> anyhow
.collect::<anyhow::Result<Vec<_>>>()?;
for cg_event in native_events {
cg_event.post(CGEventTapLocation::HID);
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
}
- if inter_char_delay_ms > 8 {
- std::thread::sleep(std::time::Duration::from_millis(inter_char_delay_ms - 8));
+ let extra = std::time::Duration::from_millis(inter_char_delay_ms).saturating_sub(key_gap());
+ if !extra.is_zero() {
+ std::thread::sleep(extra);
}
}
Ok(())
@@ -451,7 +467,7 @@ pub fn press_key_bare_global(key: &str, modifiers: &[&str]) -> anyhow::Result<()
.collect::<anyhow::Result<Vec<_>>>()?;
for event in events {
event.post(CGEventTapLocation::HID);
- std::thread::sleep(std::time::Duration::from_millis(8));
+ key_gap_sleep();
}
Ok(())
}
diff --git a/libs/cua/skills/cua-driver/EMBEDDING.md b/libs/cua/skills/cua-driver/EMBEDDING.md
index 63a99eb39a..a00eba98be 100644
--- a/libs/cua/skills/cua-driver/EMBEDDING.md
+++ b/libs/cua/skills/cua-driver/EMBEDDING.md
@@ -257,6 +257,32 @@ Measured on macOS with Calculator in the background and Terminal frontmost, a
background AX click took 1153 ms with the default, 325 ms with `200`, and 78 ms
with `0`.
+## Pacing synthesized keystrokes (macOS)
+
+The macOS keyboard adapter waits a fixed gap between each key down and up and
+between consecutive keys. For `type_text`, `press_key`, and `hotkey`, that gap
+is most of the call's latency. A host whose targets accept faster input can
+shorten it at trusted launch with one variable, in the same environment as the
+window-observation variables above. `EmbeddedCuaDriverHost` admits it too:
+
+| Variable | Meaning | Default | Accepted range |
+| ----------------------- | ----------------------------------------------- | ------- | ----------------------------------------------------- |
+| `CUA_DRIVER_KEY_GAP_MS` | Gap between synthesized key events, per event. | 8 | 2 to 100; smaller values are raised, larger clamped |
+
+Unset, empty, or unparsable values keep the default. No tool argument can
+change the gap. An explicit `type_text` `delay_ms` still applies on top of it.
+
+- **macOS only.** Linux keeps its fixed 10 ms key delay and Windows its own
+ pacing; neither reads this variable.
+- **What a shorter gap costs.** Some targets coalesce or drop key events that
+ arrive back to back, most often remote-input forwarders and apps under heavy
+ load. Read the field back after typing before relying on a low value for a
+ new target. The 2 ms floor exists because `0` posts events with no gap at all.
+
+Measured on macOS 27 with TextEdit in the background, typing a 185-character
+ASCII sentence through the PID-routed key path took about 3650 ms at the default
+and about 980 ms at `2`. All ten read-backs (five per value) matched exactly.
+
## What embedded mode changes (and what it doesn't)
| | Standalone | Embedded (`CUA_DRIVER_EMBEDDED=1`) |