Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cli): let cua do target relay Spaces (#4797)

injaneityad563fef2497
diff --git a/docs/content/docs/cua-cli/reference/cli/do.mdx b/docs/content/docs/cua-cli/reference/cli/do.mdx
index 4b0a54d8f7..697319f0e6 100644
--- a/docs/content/docs/cua-cli/reference/cli/do.mdx
+++ b/docs/content/docs/cua-cli/reference/cli/do.mdx
@@ -13,7 +13,7 @@ description: "One-shot computer actions (screenshot, click, type, windows) again
| Command | Description |
| --- | --- |
| [`cua do`](#cua-do) | One-shot computer actions against the selected target. |
-| [`cua do switch`](#cua-do-switch) | Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair. |
+| [`cua do switch`](#cua-do-switch) | Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair. |
| [`cua do status`](#cua-do-status) | Show the current target and zoom state. |
| [`cua do ls`](#cua-do-ls) | List targets (optionally of one provider). |
| [`cua do zoom`](#cua-do-zoom) | Crop screenshots to a window and map coordinates into it. |
@@ -72,11 +72,14 @@ cua do [OPTIONS] <COMMAND>
# Act on a sandbox, then take a screenshot
cua do switch dev
cua do screenshot
+
+# A relay Space (`cua spaces ls`)
+cua do switch relay:0123abcd4567ef89
```
### `cua do switch`
-Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair.
+Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair.
```text output
cua do switch [OPTIONS] <TARGET> [NAME]
@@ -84,7 +87,7 @@ cua do switch [OPTIONS] <TARGET> [NAME]
| Argument | Type | Default | Description |
| --- | --- | --- | --- |
-| `<TARGET>` | string | required | Sandbox name or ref, `host`, `url`, or a legacy provider word. |
+| `<TARGET>` | string | required | Sandbox name or ref, `relay:<id>`, `host`, `url`, or a legacy provider word. |
| `<NAME>` | string | optional | The URL after `url`, or the sandbox name after a provider word. |
| Flag | Type | Default | Env var | Description |
@@ -97,6 +100,7 @@ cua do switch [OPTIONS] <TARGET> [NAME]
```bash test="cli-shape" id="ref-cua-do-switch"
cua do switch dev
cua do switch cloud:dev
+cua do switch relay:0123abcd4567ef89
# This machine (after `cua do-host-consent`)
cua do switch host
diff --git a/libs/cua/crates/cua-cli/src/do_cmd.rs b/libs/cua/crates/cua-cli/src/do_cmd.rs
index f8f35131eb..d0734be718 100644
--- a/libs/cua/crates/cua-cli/src/do_cmd.rs
+++ b/libs/cua/crates/cua-cli/src/do_cmd.rs
@@ -4,9 +4,10 @@
//! `~/.cua/do_target.json`; coordinates are in screenshot-image space and
//! are mapped to screen points with the last screenshot's scale and origin.
//!
-//! Targets are SDK sandboxes (Fleet, local or direct). `host` is a direct
-//! sandbox pointing at a cua-spacesd on this machine (default
-//! `http://127.0.0.1:3211`), gated behind `cua do-host-consent`.
+//! Targets are SDK sandboxes (Fleet, local or direct) and relay Spaces
+//! (`relay:<machine-id>`). `host` is a direct sandbox pointing at a
+//! cua-spacesd on this machine (default `http://127.0.0.1:3211`), gated
+//! behind `cua do-host-consent`.
use crate::{
computer::{self, Computer, MAX_LENGTH, split_keys},
@@ -45,15 +46,16 @@ pub struct DoArgs {
#[derive(Subcommand, Debug)]
pub enum DoAction {
- /// Select the target: a sandbox name, `host`, `url <URL>`, or a legacy
- /// `<provider> <name>` pair.
+ /// Select the target: a sandbox name or ref, a relay Space (`relay:<id>`),
+ /// `host`, `url <URL>`, or a legacy `<provider> <name>` pair.
#[command(after_help = "Examples:
cua do switch dev
cua do switch cloud:dev
+ cua do switch relay:0123abcd4567ef89
# This machine (after `cua do-host-consent`)
cua do switch host")]
Switch {
- /// Sandbox name or ref, `host`, `url`, or a legacy provider word.
+ /// Sandbox name or ref, `relay:<id>`, `host`, `url`, or a legacy provider word.
target: String,
/// The URL after `url`, or the sandbox name after a provider word.
name: Option<String>,
@@ -636,8 +638,16 @@ impl Run<'_> {
(p.to_string(), n)
}
_ => {
- self.cua.sandboxes().get(target.clone()).await?;
- ("sandbox".to_string(), target)
+ if let Some(id) = crate::sandbox::relay_space_id(&target) {
+ // The same open later actions use. A missing or
+ // unreachable relay Space fails here, as
+ // `sandboxes().get` does for every other target.
+ crate::sandbox::env_of(self.cua, &id).await?;
+ ("relay".to_string(), id)
+ } else {
+ self.cua.sandboxes().get(target.clone()).await?;
+ ("sandbox".to_string(), target)
+ }
}
};
self.state.provider = Some(provider.clone());
diff --git a/libs/cua/crates/cua-cli/src/lib.rs b/libs/cua/crates/cua-cli/src/lib.rs
index 7873546409..1012be99ee 100644
--- a/libs/cua/crates/cua-cli/src/lib.rs
+++ b/libs/cua/crates/cua-cli/src/lib.rs
@@ -129,7 +129,9 @@ enum Command {
#[command(after_help = "Examples:
# Act on a sandbox, then take a screenshot
cua do switch dev
- cua do screenshot")]
+ cua do screenshot
+ # A relay Space (`cua spaces ls`)
+ cua do switch relay:0123abcd4567ef89")]
Do(do_cmd::DoArgs),
/// Grant consent for `cua do switch host`.
#[command(
diff --git a/libs/cua/crates/cua-cli/src/sandbox.rs b/libs/cua/crates/cua-cli/src/sandbox.rs
index 966ef6de34..22229e402c 100644
--- a/libs/cua/crates/cua-cli/src/sandbox.rs
+++ b/libs/cua/crates/cua-cli/src/sandbox.rs
@@ -2062,9 +2062,34 @@ pub fn save_env_token(name: &str, token: Option<&str>) -> Result<(), CuaError> {
)
}
-/// A spacesd client for a named sandbox. Direct sandboxes use the
-/// remembered token (or `CUA_ENV_TOKEN`).
+/// Canonical `relay:<machine-id>` when `name` is a relay Space, including
+/// the legacy `space://relay/…` spelling. An invalid `relay:` spelling is
+/// `None`, so the sandbox path reports that ref error.
+pub(crate) fn relay_space_id(name: &str) -> Option<String> {
+ let r = cua_sandbox_core::refs::SandboxRef::parse(name).ok()?;
+ matches!(r, cua_sandbox_core::refs::SandboxRef::Relay { .. }).then(|| r.to_string())
+}
+
+/// A spacesd client for a named sandbox or a relay Space. Direct sandboxes
+/// use the remembered token (or `CUA_ENV_TOKEN`). A relay Space opens
+/// through the Spaces API; the sandboxes API does not attach those machines.
pub async fn env_of(cua: &Cua, name: &str) -> Result<Arc<cua_sdk::SpacesdClient>, CuaError> {
+ if let Some(id) = relay_space_id(name) {
+ // The SDK attaches the signed-in relay account only when its `host`
+ // feature is on. This binary reaches those Spaces the way `cua mcp` does.
+ crate::host::attach_relay_account(cua, || {
+ crate::host::daemon_relay_account(&crate::util::cua_home())
+ });
+ let spaces = cua.spaces();
+ // `Spaces::space` looks the machine up in the relay directory cache.
+ // An embedded runtime fills that cache on list, not on connect. A
+ // qualified id that is not in the directory is not a Space yet.
+ let listed = spaces.list().await?;
+ if !listed.iter().any(|s| s.id == id) {
+ return Err(CuaError::NotFound(format!("Space {id}")));
+ }
+ return spaces.space(id).await?.spacesd();
+ }
let sbx = cua.sandboxes();
let info = sbx.get(name.to_string()).await?;
if info.location == "direct"
diff --git a/libs/cua/crates/cua-cli/tests/sandbox_do.rs b/libs/cua/crates/cua-cli/tests/sandbox_do.rs
index 3e8d194cb4..b921ed06b9 100644
--- a/libs/cua/crates/cua-cli/tests/sandbox_do.rs
+++ b/libs/cua/crates/cua-cli/tests/sandbox_do.rs
@@ -1,11 +1,12 @@
#![allow(deprecated)] // also exercises the deprecated `apply_pool` wrapper
//! `cua sandbox` and `cua do` against the mock spacesd (direct
-//! sandboxes) and the fake Fleet API.
+//! sandboxes), relay Spaces, and the fake Fleet API.
mod common;
use common::*;
use cua_daemon::fixtures;
use cua_fleet::testing::FakeFleet;
+use cua_host::testing::FakeRelay;
use serde_json::json;
async fn direct(h: &Home, url: &str, name: &str) {
@@ -752,3 +753,153 @@ async fn default_ls_reads_a_keychain_session_only_with_the_marker() {
assert_eq!(row["location"], "cloud");
assert!(reads() > before);
}
+
+/// `cua do` drives a relay Space: switch, screenshot, click, type and shell
+/// go through the same `env_of` path as a direct sandbox.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn do_drives_a_relay_space() {
+ const MACHINE: &str = "a30ad1ef1e4fea09";
+ let relay = FakeRelay::start().await;
+ relay.add_account("relay-token", "user-1", Some("[email protected]"));
+ cua_host::RelayClient::new(&relay.url)
+ .unwrap()
+ .register(
+ "relay-token",
+ &cua_host::relay::RegisterRequest {
+ id: MACHINE.into(),
+ name: "Relay desk".into(),
+ ..Default::default()
+ },
+ )
+ .await
+ .unwrap();
+ let env = fixtures::start_env(None, None).await;
+ relay.tunnel(MACHINE, &env.url);
+ relay.set_online(MACHINE, true, "0.0.0-mock");
+
+ let mut h = Home::new();
+ h.set("CUA_RELAY_URL", &relay.url);
+ let id = format!("relay:{MACHINE}");
+
+ // A bad machine id stays a ref error and is not stored as a target.
+ let o = h.run(&["--embedded", "do", "switch", "relay:short"]).await;
+ assert_eq!(o.code, 2, "{o:?}");
+ assert!(
+ o.stderr.contains(
+ "invalid argument: not a sandbox ref: \"relay:short\" (relay machine ids are [a-z0-9-]{8,64})"
+ ),
+ "{o:?}"
+ );
+
+ // Signed out, the directory refresh asks for the session. Previously
+ // every relay id was refused as unsupported before that.
+ let o = h.run(&["--embedded", "do", "switch", &id]).await;
+ assert_eq!(o.code, 6, "{o:?}");
+ assert!(
+ o.stderr.contains(
+ "unauthenticated: Relay authentication failed. Sign in again to refresh your machines."
+ ),
+ "{o:?}"
+ );
+ assert!(!o.stderr.contains("unsupported"), "{o:?}");
+
+ std::fs::create_dir_all(h.cua_home()).unwrap();
+ std::fs::write(
+ h.cua_home().join("credentials.json"),
+ json!({
+ "access_token": "relay-token",
+ "expires_at": "2999-01-01T00:00:00Z",
+ "token_type": "Bearer"
+ })
+ .to_string(),
+ )
+ .unwrap();
+
+ let o = h
+ .run(&["--embedded", "do", "switch", "relay:0123abcd4567ef89"])
+ .await;
+ assert_eq!(o.code, 3, "{o:?}");
+ assert!(
+ o.stderr.contains("not found: Space relay:0123abcd4567ef89"),
+ "{o:?}"
+ );
+
+ let o = h.run(&["--embedded", "do", "switch", &id]).await;
+ o.ok();
+ assert!(o.stdout.contains(&format!("✅ Switched to {id}")), "{o:?}");
+ let state = read_json(&h.cua_home().join("do_target.json"));
+ assert_eq!(state["provider"], "relay");
+ assert_eq!(state["name"], id);
+
+ // The legacy spelling stores the canonical id.
+ let legacy = format!("space://relay/{MACHINE}");
+ let o = h.run(&["--embedded", "do", "switch", &legacy]).await;
+ o.ok();
+ assert!(o.stdout.contains(&format!("✅ Switched to {id}")), "{o:?}");
+
+ let o = h.run(&["--embedded", "do", "status"]).await;
+ o.ok();
+ assert!(
+ o.stdout.contains(&format!("Current target: relay/{id}")),
+ "{o:?}"
+ );
+
+ let shot = h.dir.path().join("relay.png");
+ let o = h
+ .run(&[
+ "--embedded",
+ "do",
+ "screenshot",
+ "--save",
+ shot.to_str().unwrap(),
+ ])
+ .await;
+ o.ok();
+ assert!(o.stdout.contains("screenshot saved to"), "{o:?}");
+ assert!(o.stdout.contains(&format!("💻 {id}")), "{o:?}");
+ assert!(std::fs::read(&shot).unwrap().starts_with(b"\x89PNG"));
+ let state = read_json(&h.cua_home().join("do_target.json"));
+ assert!((state["zoom_scale"].as_f64().unwrap() - 1200.0 / 1280.0).abs() < 1e-9);
+
+ // Image (600, 300) → screen (640, 320), the same mapping as a direct sandbox.
+ let o = h.run(&["--embedded", "do", "click", "600", "300"]).await;
+ o.ok();
+ let ptr = env
+ .mock
+ .state
+ .observed
+ .pointer
+ .lock()
+ .unwrap()
+ .last()
+ .cloned()
+ .unwrap();
+ assert!(ptr.contains("x: 640.0, y: 320.0"), "{ptr}");
+
+ h.run(&["--embedded", "do", "type", "relay-ok"]).await.ok();
+ let kb = env.mock.state.observed.keyboard.lock().unwrap().clone();
+ assert!(kb.iter().any(|k| k.contains("relay-ok")), "{kb:?}");
+
+ let o = h
+ .run(&["--embedded", "do", "shell", "echo", "relay-ok"])
+ .await;
+ o.ok();
+ assert!(o.stdout.starts_with("✅ relay-ok"), "{o:?}");
+
+ cua_host::RelayClient::new(&relay.url)
+ .unwrap()
+ .stop_sharing("relay-token", MACHINE)
+ .await
+ .unwrap();
+ let o = h.run(&["--embedded", "do", "switch", &id]).await;
+ assert_eq!(o.code, 1, "{o:?}");
+ assert!(
+ o.stderr.contains(
+ "env: Relay desk stopped sharing: ask its owner to Resume sharing (or run `cua host start` there)"
+ ),
+ "{o:?}"
+ );
+ // A failed switch leaves the selected target in place.
+ let state = read_json(&h.cua_home().join("do_target.json"));
+ assert_eq!(state["name"], id);
+}
diff --git a/libs/cua/skills/gui-automation/SKILL.md b/libs/cua/skills/gui-automation/SKILL.md
index 494b63ae57..3cc53bc716 100644
--- a/libs/cua/skills/gui-automation/SKILL.md
+++ b/libs/cua/skills/gui-automation/SKILL.md
@@ -22,6 +22,7 @@ cua --version # check install; if missing: curl -fsSL https://cua.ai/in
# Connect to target (pick one)
cua do switch my-sandbox # any sandbox from `cua sb ls`
+cua do switch relay:<machine-id> # a relay Space from `cua spaces ls`
cua do switch url http://127.0.0.1:3211 --as dev # a cua-spacesd by URL
cua do-host-consent && cua do switch host # local machine (one-time consent)
```
@@ -152,6 +153,7 @@ Tell the user where the trajectory is (`cua trajectory ls` prints the path).
| Target | Example |
| ----------------- | -------------------------------------------- |
| a sandbox | `cua do switch my-sandbox` (see `cua sb ls`) |
+| a relay Space | `cua do switch relay:<machine-id>` (`cua spaces ls`) |
| a spacesd URL | `cua do switch url http://host:3211 --as x` |
| this machine | `cua do switch host` |
diff --git a/libs/cua/skills/gui-automation/references/command-reference.md b/libs/cua/skills/gui-automation/references/command-reference.md
index 5bd785aa3b..f0409f8c33 100644
--- a/libs/cua/skills/gui-automation/references/command-reference.md
+++ b/libs/cua/skills/gui-automation/references/command-reference.md
@@ -11,7 +11,7 @@ One-shot computer actions against the selected target
Usage: cua do [OPTIONS] <COMMAND>
Commands:
- switch Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair
+ switch Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair
status Show the current target and zoom state
ls List targets (optionally of one provider)
zoom Crop screenshots to a window and map coordinates into it
@@ -41,7 +41,7 @@ Options:
## cua do switch
```text
-Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair
+Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair
Usage: cua do switch [OPTIONS] <TARGET> [NAME]
diff --git a/scripts/docs-generators/cli-specs/cua.json b/scripts/docs-generators/cli-specs/cua.json
index 5d24e854af..cf8008283f 100644
--- a/scripts/docs-generators/cli-specs/cua.json
+++ b/scripts/docs-generators/cli-specs/cua.json
@@ -4879,7 +4879,7 @@
},
{
"abstract": "One-shot computer actions against the selected target",
- "after_help": "Examples:\n # Act on a sandbox, then take a screenshot\n cua do switch dev\n cua do screenshot",
+ "after_help": "Examples:\n # Act on a sandbox, then take a screenshot\n cua do switch dev\n cua do screenshot\n # A relay Space (`cua spaces ls`)\n cua do switch relay:0123abcd4567ef89",
"aliases": [],
"arguments": [],
"flags": [
@@ -4902,13 +4902,13 @@
"subcommand_optional": false,
"subcommands": [
{
- "abstract": "Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair",
- "after_help": "Examples:\n cua do switch dev\n cua do switch cloud:dev\n # This machine (after `cua do-host-consent`)\n cua do switch host",
+ "abstract": "Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair",
+ "after_help": "Examples:\n cua do switch dev\n cua do switch cloud:dev\n cua do switch relay:0123abcd4567ef89\n # This machine (after `cua do-host-consent`)\n cua do switch host",
"aliases": [],
"arguments": [
{
"default_value": null,
- "help": "Sandbox name or ref, `host`, `url`, or a legacy provider word",
+ "help": "Sandbox name or ref, `relay:<id>`, `host`, `url`, or a legacy provider word",
"hidden": false,
"is_optional": false,
"name": "TARGET",
diff --git a/skills/gui-automation/SKILL.md b/skills/gui-automation/SKILL.md
index 494b63ae57..3cc53bc716 100644
--- a/skills/gui-automation/SKILL.md
+++ b/skills/gui-automation/SKILL.md
@@ -22,6 +22,7 @@ cua --version # check install; if missing: curl -fsSL https://cua.ai/in
# Connect to target (pick one)
cua do switch my-sandbox # any sandbox from `cua sb ls`
+cua do switch relay:<machine-id> # a relay Space from `cua spaces ls`
cua do switch url http://127.0.0.1:3211 --as dev # a cua-spacesd by URL
cua do-host-consent && cua do switch host # local machine (one-time consent)
```
@@ -152,6 +153,7 @@ Tell the user where the trajectory is (`cua trajectory ls` prints the path).
| Target | Example |
| ----------------- | -------------------------------------------- |
| a sandbox | `cua do switch my-sandbox` (see `cua sb ls`) |
+| a relay Space | `cua do switch relay:<machine-id>` (`cua spaces ls`) |
| a spacesd URL | `cua do switch url http://host:3211 --as x` |
| this machine | `cua do switch host` |
diff --git a/skills/gui-automation/references/command-reference.md b/skills/gui-automation/references/command-reference.md
index 5bd785aa3b..f0409f8c33 100644
--- a/skills/gui-automation/references/command-reference.md
+++ b/skills/gui-automation/references/command-reference.md
@@ -11,7 +11,7 @@ One-shot computer actions against the selected target
Usage: cua do [OPTIONS] <COMMAND>
Commands:
- switch Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair
+ switch Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair
status Show the current target and zoom state
ls List targets (optionally of one provider)
zoom Crop screenshots to a window and map coordinates into it
@@ -41,7 +41,7 @@ Options:
## cua do switch
```text
-Select the target: a sandbox name, `host`, `url <URL>`, or a legacy `<provider> <name>` pair
+Select the target: a sandbox name or ref, a relay Space (`relay:<id>`), `host`, `url <URL>`, or a legacy `<provider> <name>` pair
Usage: cua do switch [OPTIONS] <TARGET> [NAME]