Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cli): let cua do target relay Spaces (#4797)
injaneityad563fef2497
diff --git a/docs/content/docs/cua-cli/reference/cli/do.mdx b/docs/content/docs/cua-cli/reference/cli/do.mdxindex 4b0a54d8f7..697319f0e6 100644--- a/docs/content/docs/cua-cli/reference/cli/do.mdx+++ b/docs/content/docs/cua-cli/reference/cli/do.mdx@@ -13,7 +13,7 @@ description: "One-shot computer actions (screenshot, click, type, windows) again| Command | Description || --- | --- || [`cua do`](#cua-do) | One-shot computer actions against the selected target. |-| [`cua do switch`](#cua-do-switch) | Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair. |+| [`cua do switch`](#cua-do-switch) | Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair. || [`cua do status`](#cua-do-status) | Show the current target and zoom state. || [`cua do ls`](#cua-do-ls) | List targets (optionally of one provider). || [`cua do zoom`](#cua-do-zoom) | Crop screenshots to a window and map coordinates into it. |@@ -72,11 +72,14 @@ cua do [OPTIONS] <COMMAND># Act on a sandbox, then take a screenshotcua do switch devcua do screenshot++# A relay Space (`cua spaces ls`)+cua do switch relay:0123abcd4567ef89```### `cua do switch`-Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair.+Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair.```text outputcua do switch [OPTIONS] <TARGET> [NAME]@@ -84,7 +87,7 @@ cua do switch [OPTIONS] <TARGET> [NAME]| Argument | Type | Default | Description || --- | --- | --- | --- |-| `<TARGET>` | string | required | Sandbox name or ref, `host`, `url`, or a legacy provider word. |+| `<TARGET>` | string | required | Sandbox name or ref, `relay:<id>`, `host`, `url`, or a legacy provider word. || `<NAME>` | string | optional | The URL after `url`, or the sandbox name after a provider word. || Flag | Type | Default | Env var | Description |@@ -97,6 +100,7 @@ cua do switch [OPTIONS] <TARGET> [NAME]```bash test="cli-shape" id="ref-cua-do-switch"cua do switch devcua do switch cloud:dev+cua do switch relay:0123abcd4567ef89# This machine (after `cua do-host-consent`)cua do switch hostdiff --git a/libs/cua/crates/cua-cli/src/do_cmd.rs b/libs/cua/crates/cua-cli/src/do_cmd.rsindex f8f35131eb..d0734be718 100644--- a/libs/cua/crates/cua-cli/src/do_cmd.rs+++ b/libs/cua/crates/cua-cli/src/do_cmd.rs@@ -4,9 +4,10 @@//! `~/.cua/do_target.json`; coordinates are in screenshot-image space and//! are mapped to screen points with the last screenshot's scale and origin.//!-//! Targets are SDK sandboxes (Fleet, local or direct). `host` is a direct-//! sandbox pointing at a cua-spacesd on this machine (default-//! `http://127.0.0.1:3211`), gated behind `cua do-host-consent`.+//! Targets are SDK sandboxes (Fleet, local or direct) and relay Spaces+//! (`relay:<machine-id>`). `host` is a direct sandbox pointing at a+//! cua-spacesd on this machine (default `http://127.0.0.1:3211`), gated+//! behind `cua do-host-consent`.use crate::{computer::{self, Computer, MAX_LENGTH, split_keys},@@ -45,15 +46,16 @@ pub struct DoArgs {#[derive(Subcommand, Debug)]pub enum DoAction {- /// Select the target: a sandbox name, `host`, `url <URL>`, or a legacy- /// `<provider> <name>` pair.+ /// Select the target: a sandbox name or ref, a relay Space (`relay:<id>`),+ /// `host`, `url <URL>`, or a legacy `<provider> <name>` pair.#[command(after_help = "Examples:cua do switch devcua do switch cloud:dev+ cua do switch relay:0123abcd4567ef89# This machine (after `cua do-host-consent`)cua do switch host")]Switch {- /// Sandbox name or ref, `host`, `url`, or a legacy provider word.+ /// Sandbox name or ref, `relay:<id>`, `host`, `url`, or a legacy provider word.target: String,/// The URL after `url`, or the sandbox name after a provider word.name: Option<String>,@@ -636,8 +638,16 @@ impl Run<'_> {(p.to_string(), n)}_ => {- self.cua.sandboxes().get(target.clone()).await?;- ("sandbox".to_string(), target)+ if let Some(id) = crate::sandbox::relay_space_id(&target) {+ // The same open later actions use. A missing or+ // unreachable relay Space fails here, as+ // `sandboxes().get` does for every other target.+ crate::sandbox::env_of(self.cua, &id).await?;+ ("relay".to_string(), id)+ } else {+ self.cua.sandboxes().get(target.clone()).await?;+ ("sandbox".to_string(), target)+ }}};self.state.provider = Some(provider.clone());diff --git a/libs/cua/crates/cua-cli/src/lib.rs b/libs/cua/crates/cua-cli/src/lib.rsindex 7873546409..1012be99ee 100644--- a/libs/cua/crates/cua-cli/src/lib.rs+++ b/libs/cua/crates/cua-cli/src/lib.rs@@ -129,7 +129,9 @@ enum Command {#[command(after_help = "Examples:# Act on a sandbox, then take a screenshotcua do switch dev- cua do screenshot")]+ cua do screenshot+ # A relay Space (`cua spaces ls`)+ cua do switch relay:0123abcd4567ef89")]Do(do_cmd::DoArgs),/// Grant consent for `cua do switch host`.#[command(diff --git a/libs/cua/crates/cua-cli/src/sandbox.rs b/libs/cua/crates/cua-cli/src/sandbox.rsindex 966ef6de34..22229e402c 100644--- a/libs/cua/crates/cua-cli/src/sandbox.rs+++ b/libs/cua/crates/cua-cli/src/sandbox.rs@@ -2062,9 +2062,34 @@ pub fn save_env_token(name: &str, token: Option<&str>) -> Result<(), CuaError> {)}-/// A spacesd client for a named sandbox. Direct sandboxes use the-/// remembered token (or `CUA_ENV_TOKEN`).+/// Canonical `relay:<machine-id>` when `name` is a relay Space, including+/// the legacy `space://relay/…` spelling. An invalid `relay:` spelling is+/// `None`, so the sandbox path reports that ref error.+pub(crate) fn relay_space_id(name: &str) -> Option<String> {+ let r = cua_sandbox_core::refs::SandboxRef::parse(name).ok()?;+ matches!(r, cua_sandbox_core::refs::SandboxRef::Relay { .. }).then(|| r.to_string())+}++/// A spacesd client for a named sandbox or a relay Space. Direct sandboxes+/// use the remembered token (or `CUA_ENV_TOKEN`). A relay Space opens+/// through the Spaces API; the sandboxes API does not attach those machines.pub async fn env_of(cua: &Cua, name: &str) -> Result<Arc<cua_sdk::SpacesdClient>, CuaError> {+ if let Some(id) = relay_space_id(name) {+ // The SDK attaches the signed-in relay account only when its `host`+ // feature is on. This binary reaches those Spaces the way `cua mcp` does.+ crate::host::attach_relay_account(cua, || {+ crate::host::daemon_relay_account(&crate::util::cua_home())+ });+ let spaces = cua.spaces();+ // `Spaces::space` looks the machine up in the relay directory cache.+ // An embedded runtime fills that cache on list, not on connect. A+ // qualified id that is not in the directory is not a Space yet.+ let listed = spaces.list().await?;+ if !listed.iter().any(|s| s.id == id) {+ return Err(CuaError::NotFound(format!("Space {id}")));+ }+ return spaces.space(id).await?.spacesd();+ }let sbx = cua.sandboxes();let info = sbx.get(name.to_string()).await?;if info.location == "direct"diff --git a/libs/cua/crates/cua-cli/tests/sandbox_do.rs b/libs/cua/crates/cua-cli/tests/sandbox_do.rsindex 3e8d194cb4..b921ed06b9 100644--- a/libs/cua/crates/cua-cli/tests/sandbox_do.rs+++ b/libs/cua/crates/cua-cli/tests/sandbox_do.rs@@ -1,11 +1,12 @@#![allow(deprecated)] // also exercises the deprecated `apply_pool` wrapper//! `cua sandbox` and `cua do` against the mock spacesd (direct-//! sandboxes) and the fake Fleet API.+//! sandboxes), relay Spaces, and the fake Fleet API.mod common;use common::*;use cua_daemon::fixtures;use cua_fleet::testing::FakeFleet;+use cua_host::testing::FakeRelay;use serde_json::json;async fn direct(h: &Home, url: &str, name: &str) {@@ -752,3 +753,153 @@ async fn default_ls_reads_a_keychain_session_only_with_the_marker() {assert_eq!(row["location"], "cloud");assert!(reads() > before);}++/// `cua do` drives a relay Space: switch, screenshot, click, type and shell+/// go through the same `env_of` path as a direct sandbox.+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn do_drives_a_relay_space() {+ const MACHINE: &str = "a30ad1ef1e4fea09";+ let relay = FakeRelay::start().await;+ relay.add_account("relay-token", "user-1", Some("[email protected]"));+ cua_host::RelayClient::new(&relay.url)+ .unwrap()+ .register(+ "relay-token",+ &cua_host::relay::RegisterRequest {+ id: MACHINE.into(),+ name: "Relay desk".into(),+ ..Default::default()+ },+ )+ .await+ .unwrap();+ let env = fixtures::start_env(None, None).await;+ relay.tunnel(MACHINE, &env.url);+ relay.set_online(MACHINE, true, "0.0.0-mock");++ let mut h = Home::new();+ h.set("CUA_RELAY_URL", &relay.url);+ let id = format!("relay:{MACHINE}");++ // A bad machine id stays a ref error and is not stored as a target.+ let o = h.run(&["--embedded", "do", "switch", "relay:short"]).await;+ assert_eq!(o.code, 2, "{o:?}");+ assert!(+ o.stderr.contains(+ "invalid argument: not a sandbox ref: \"relay:short\" (relay machine ids are [a-z0-9-]{8,64})"+ ),+ "{o:?}"+ );++ // Signed out, the directory refresh asks for the session. Previously+ // every relay id was refused as unsupported before that.+ let o = h.run(&["--embedded", "do", "switch", &id]).await;+ assert_eq!(o.code, 6, "{o:?}");+ assert!(+ o.stderr.contains(+ "unauthenticated: Relay authentication failed. Sign in again to refresh your machines."+ ),+ "{o:?}"+ );+ assert!(!o.stderr.contains("unsupported"), "{o:?}");++ std::fs::create_dir_all(h.cua_home()).unwrap();+ std::fs::write(+ h.cua_home().join("credentials.json"),+ json!({+ "access_token": "relay-token",+ "expires_at": "2999-01-01T00:00:00Z",+ "token_type": "Bearer"+ })+ .to_string(),+ )+ .unwrap();++ let o = h+ .run(&["--embedded", "do", "switch", "relay:0123abcd4567ef89"])+ .await;+ assert_eq!(o.code, 3, "{o:?}");+ assert!(+ o.stderr.contains("not found: Space relay:0123abcd4567ef89"),+ "{o:?}"+ );++ let o = h.run(&["--embedded", "do", "switch", &id]).await;+ o.ok();+ assert!(o.stdout.contains(&format!("✅ Switched to {id}")), "{o:?}");+ let state = read_json(&h.cua_home().join("do_target.json"));+ assert_eq!(state["provider"], "relay");+ assert_eq!(state["name"], id);++ // The legacy spelling stores the canonical id.+ let legacy = format!("space://relay/{MACHINE}");+ let o = h.run(&["--embedded", "do", "switch", &legacy]).await;+ o.ok();+ assert!(o.stdout.contains(&format!("✅ Switched to {id}")), "{o:?}");++ let o = h.run(&["--embedded", "do", "status"]).await;+ o.ok();+ assert!(+ o.stdout.contains(&format!("Current target: relay/{id}")),+ "{o:?}"+ );++ let shot = h.dir.path().join("relay.png");+ let o = h+ .run(&[+ "--embedded",+ "do",+ "screenshot",+ "--save",+ shot.to_str().unwrap(),+ ])+ .await;+ o.ok();+ assert!(o.stdout.contains("screenshot saved to"), "{o:?}");+ assert!(o.stdout.contains(&format!("💻 {id}")), "{o:?}");+ assert!(std::fs::read(&shot).unwrap().starts_with(b"\x89PNG"));+ let state = read_json(&h.cua_home().join("do_target.json"));+ assert!((state["zoom_scale"].as_f64().unwrap() - 1200.0 / 1280.0).abs() < 1e-9);++ // Image (600, 300) → screen (640, 320), the same mapping as a direct sandbox.+ let o = h.run(&["--embedded", "do", "click", "600", "300"]).await;+ o.ok();+ let ptr = env+ .mock+ .state+ .observed+ .pointer+ .lock()+ .unwrap()+ .last()+ .cloned()+ .unwrap();+ assert!(ptr.contains("x: 640.0, y: 320.0"), "{ptr}");++ h.run(&["--embedded", "do", "type", "relay-ok"]).await.ok();+ let kb = env.mock.state.observed.keyboard.lock().unwrap().clone();+ assert!(kb.iter().any(|k| k.contains("relay-ok")), "{kb:?}");++ let o = h+ .run(&["--embedded", "do", "shell", "echo", "relay-ok"])+ .await;+ o.ok();+ assert!(o.stdout.starts_with("✅ relay-ok"), "{o:?}");++ cua_host::RelayClient::new(&relay.url)+ .unwrap()+ .stop_sharing("relay-token", MACHINE)+ .await+ .unwrap();+ let o = h.run(&["--embedded", "do", "switch", &id]).await;+ assert_eq!(o.code, 1, "{o:?}");+ assert!(+ o.stderr.contains(+ "env: Relay desk stopped sharing: ask its owner to Resume sharing (or run `cua host start` there)"+ ),+ "{o:?}"+ );+ // A failed switch leaves the selected target in place.+ let state = read_json(&h.cua_home().join("do_target.json"));+ assert_eq!(state["name"], id);+}diff --git a/libs/cua/skills/gui-automation/SKILL.md b/libs/cua/skills/gui-automation/SKILL.mdindex 494b63ae57..3cc53bc716 100644--- a/libs/cua/skills/gui-automation/SKILL.md+++ b/libs/cua/skills/gui-automation/SKILL.md@@ -22,6 +22,7 @@ cua --version # check install; if missing: curl -fsSL https://cua.ai/in# Connect to target (pick one)cua do switch my-sandbox # any sandbox from `cua sb ls`+cua do switch relay:<machine-id> # a relay Space from `cua spaces ls`cua do switch url http://127.0.0.1:3211 --as dev # a cua-spacesd by URLcua do-host-consent && cua do switch host # local machine (one-time consent)```@@ -152,6 +153,7 @@ Tell the user where the trajectory is (`cua trajectory ls` prints the path).| Target | Example || ----------------- | -------------------------------------------- || a sandbox | `cua do switch my-sandbox` (see `cua sb ls`) |+| a relay Space | `cua do switch relay:<machine-id>` (`cua spaces ls`) || a spacesd URL | `cua do switch url http://host:3211 --as x` || this machine | `cua do switch host` |diff --git a/libs/cua/skills/gui-automation/references/command-reference.md b/libs/cua/skills/gui-automation/references/command-reference.mdindex 5bd785aa3b..f0409f8c33 100644--- a/libs/cua/skills/gui-automation/references/command-reference.md+++ b/libs/cua/skills/gui-automation/references/command-reference.md@@ -11,7 +11,7 @@ One-shot computer actions against the selected targetUsage: cua do [OPTIONS] <COMMAND>Commands:- switch Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair+ switch Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pairstatus Show the current target and zoom statels List targets (optionally of one provider)zoom Crop screenshots to a window and map coordinates into it@@ -41,7 +41,7 @@ Options:## cua do switch```text-Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair+Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pairUsage: cua do switch [OPTIONS] <TARGET> [NAME]diff --git a/scripts/docs-generators/cli-specs/cua.json b/scripts/docs-generators/cli-specs/cua.jsonindex 5d24e854af..cf8008283f 100644--- a/scripts/docs-generators/cli-specs/cua.json+++ b/scripts/docs-generators/cli-specs/cua.json@@ -4879,7 +4879,7 @@},{"abstract": "One-shot computer actions against the selected target",- "after_help": "Examples:\n # Act on a sandbox, then take a screenshot\n cua do switch dev\n cua do screenshot",+ "after_help": "Examples:\n # Act on a sandbox, then take a screenshot\n cua do switch dev\n cua do screenshot\n # A relay Space (`cua spaces ls`)\n cua do switch relay:0123abcd4567ef89","aliases": [],"arguments": [],"flags": [@@ -4902,13 +4902,13 @@"subcommand_optional": false,"subcommands": [{- "abstract": "Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair",- "after_help": "Examples:\n cua do switch dev\n cua do switch cloud:dev\n # This machine (after `cua do-host-consent`)\n cua do switch host",+ "abstract": "Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair",+ "after_help": "Examples:\n cua do switch dev\n cua do switch cloud:dev\n cua do switch relay:0123abcd4567ef89\n # This machine (after `cua do-host-consent`)\n cua do switch host","aliases": [],"arguments": [{"default_value": null,- "help": "Sandbox name or ref, `host`, `url`, or a legacy provider word",+ "help": "Sandbox name or ref, `relay:<id>`, `host`, `url`, or a legacy provider word","hidden": false,"is_optional": false,"name": "TARGET",diff --git a/skills/gui-automation/SKILL.md b/skills/gui-automation/SKILL.mdindex 494b63ae57..3cc53bc716 100644--- a/skills/gui-automation/SKILL.md+++ b/skills/gui-automation/SKILL.md@@ -22,6 +22,7 @@ cua --version # check install; if missing: curl -fsSL https://cua.ai/in# Connect to target (pick one)cua do switch my-sandbox # any sandbox from `cua sb ls`+cua do switch relay:<machine-id> # a relay Space from `cua spaces ls`cua do switch url http://127.0.0.1:3211 --as dev # a cua-spacesd by URLcua do-host-consent && cua do switch host # local machine (one-time consent)```@@ -152,6 +153,7 @@ Tell the user where the trajectory is (`cua trajectory ls` prints the path).| Target | Example || ----------------- | -------------------------------------------- || a sandbox | `cua do switch my-sandbox` (see `cua sb ls`) |+| a relay Space | `cua do switch relay:<machine-id>` (`cua spaces ls`) || a spacesd URL | `cua do switch url http://host:3211 --as x` || this machine | `cua do switch host` |diff --git a/skills/gui-automation/references/command-reference.md b/skills/gui-automation/references/command-reference.mdindex 5bd785aa3b..f0409f8c33 100644--- a/skills/gui-automation/references/command-reference.md+++ b/skills/gui-automation/references/command-reference.md@@ -11,7 +11,7 @@ One-shot computer actions against the selected targetUsage: cua do [OPTIONS] <COMMAND>Commands:- switch Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair+ switch Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pairstatus Show the current target and zoom statels List targets (optionally of one provider)zoom Crop screenshots to a window and map coordinates into it@@ -41,7 +41,7 @@ Options:## cua do switch```text-Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair+Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pairUsage: cua do switch [OPTIONS] <TARGET> [NAME]