Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(telemetry): installer funnel events and an agent-used-a-Space event (CUA-1881) (#5457)
Francesco Bonacciba3e14fd8beb
diff --git a/docs/content/docs/cua-sdk/concepts/meta.json b/docs/content/docs/cua-sdk/concepts/meta.jsonindex a19fc7f501..6c31fe793b 100644--- a/docs/content/docs/cua-sdk/concepts/meta.json+++ b/docs/content/docs/cua-sdk/concepts/meta.json@@ -4,6 +4,7 @@"how-sandboxes-work","telemetry","telemetry-events",+ "telemetry-counting","telemetry-error-kinds"]}diff --git a/docs/content/docs/cua-sdk/concepts/telemetry-counting.mdx b/docs/content/docs/cua-sdk/concepts/telemetry-counting.mdxnew file mode 100644index 0000000000..13baa22541--- /dev/null+++ b/docs/content/docs/cua-sdk/concepts/telemetry-counting.mdx@@ -0,0 +1,61 @@+---+title: How events are counted+description: How the installers, an AI agent using a Space, agent runs, teleport callers and session deliveries are counted.+---++Details behind some rows of [Telemetry events](/cua-sdk/concepts/telemetry-events).++## Installers++The installers (`install.sh`, `install.ps1`) send three events about their+own run: `installer_started`, then `installer_completed` or+`installer_failed` with the part that was running and a fixed reason word.+They follow the same switches as every Cua program (`DO_NOT_TRACK`,+`CUA_TELEMETRY`, the config file, CI) and send nothing on a dry run. Before+the first one, the installer prints a short notice saying so. They create+the same anonymous install id, so the app's later events continue the same+install.++`installer_failed` also carries `install_stage`, the part that was running+(`platform`, `manifest`, `cli`, `runtime`, `app`, `driver`, `sign_in`, `host` or+`other`), and `error_kind`, a fixed reason (`network`, `checksum`, `signature`,+`unsupported_platform`, `permission_denied`, `missing_tool`, `cancelled` or+`other`), never the message. Both are `none` on the other two steps. The+installers send as product `cli`, with `$lib` `cua-installer`.++## An AI agent using a Space++`cua_agent_used_space` is sent when a tool call from an AI agent (through+`cua mcp` or the daemon's MCP server) acts inside a Space and works: a+command, a file, the screen, a service or an agent in the Space. It carries+only the kind of tool, at most once per install per UTC day. Calls that only+list, create or share Spaces do not count.++## Agent runs and retention++An agent run is counted by the install that started it: once when it+starts, and once when any Cua program of that install first sees its first+turn end, fail or stop. Until then a marker with the same fixed words waits+in `~/.cua/telemetry/agent_runs/` (dropped after 7 days). Cua programs+inside an agent run (`CUA_AGENT_RUN_ID` set) record no agent runs, so a+Space never counts as an install. Retention counts installs with a+`cua_app_active` event on day 1, 7 and 30 after their first one.++## Teleport caller kinds++`caller_kind` says who asked for a teleport: `cua_app_signed` (the Cua app or+CLI, verified by its code signature), `embedded_sdk` (another app that embeds+the cua SDK), `unsigned` (ad hoc or unsigned code) or `unknown` (the platform+cannot verify signatures). With `caller_kind_source: broker_verified` it comes+from the Keyvault broker's OS-verified caller identity; with `self_reported`+the calling process reported it, and an SDK process only ever reports+`embedded_sdk` or `unknown`. Paths, bundle ids and signing ids of callers are+never sent.++## Session deliveries++A modified build can remove its own telemetry. The receive-side count+(`cua_spacesd_session_deliveries`) is recorded by cua-spacesd inside the+sandbox, and only when the host enabled telemetry for that sandbox+(`CUA_SPACESD_TELEMETRY=1`, which the SDK sets on local sandboxes while+telemetry is on).diff --git a/docs/content/docs/cua-sdk/concepts/telemetry-events.mdx b/docs/content/docs/cua-sdk/concepts/telemetry-events.mdxindex 4e370a821c..a0a0ee4263 100644--- a/docs/content/docs/cua-sdk/concepts/telemetry-events.mdx+++ b/docs/content/docs/cua-sdk/concepts/telemetry-events.mdx@@ -27,6 +27,8 @@ Every event carries: `telemetry_schema_version`, `event_version`, `product`| `cua_spaces_feature_used` | `feature` (a fixed Spaces app feature) | Spaces app feature adoption. || `cua_stream_stats` | `transport`, `codec`, `decode` (`hw` or `sw`), `fps_bucket`, `latency_bucket`, `resolution`, `duration_bucket` | Streaming performance (50% sampled). || `cua_onboarding_step` | `step`, `outcome`, `onboarding_eligible` (`yes` or `no` on the Spaces app's `app_launched`: its first run was still due; `unknown` otherwise), `error_kind` (a failed `signed_in`: `timeout`, `cancelled`, `permission_denied`, `transport`, `unauthenticated` or `other`, never the message; `none` otherwise) | Install and onboarding funnel, and the Spaces checklist: `checklist_shown`, `checklist_<step>` (`add_machine`, `create_space`, `teleport`, `share`, `run_agent`), `checklist_completed`, `checklist_dismissed`, `checklist_reopened`. |+| `cua_installer_step` | `step` (`installer_started`, `installer_completed`, `installer_failed`), `outcome`, `install_stage`, `error_kind`, `installer` (`sh` or `ps1`), `items` (what was asked for: `cli`, `spaces`, `cua_driver`, `host` joined with `+`) | Install funnel: how many installs start and finish, and where and why they fail ([values](/cua-sdk/concepts/telemetry-counting#installers)). |+| `cua_agent_used_space` | `tool_group` (`shell`, `files`, `computer`, `services`, `agents` or `other`) | First success and agent retention: an AI agent's tool call acted inside a Space and worked. At most one per install per UTC day. || `cua_agent_run_started` | `harness` (cua-agents id or `other`), `location`, `entry` (`spaces_tool`, `cli`, `sdk`, `persistent`, `routine`), `outcome`, `error_kind` | Agent runs started, and failed starts. || `cua_agent_run_completed` | `harness`, `location`, `entry`, `outcome`, `error_kind`, `duration_bucket` | First agent run, and how reliably runs finish. Once per run. || `cua_bench_run_completed` | `taskset` (cua-bench dataset or `custom`), `score_bucket`, `task_count`, `outcome` | Benchmark use and coarse scores. |@@ -50,20 +52,5 @@ Every event carries: `telemetry_schema_version`, `event_version`, `product`| `cua_device_enroll` | `method` (`sign_in`, `approval`, `rekey`), `outcome` | How devices join your account's relay. || `cua_experiment` | `action` (`experiment_on` or `experiment_off`), `experiment` (`cua_volume`, `your_cloud`, `sharing`) | Which experiments people try, and turn off again. |-## Teleport caller kinds--`caller_kind` says who asked for a teleport: `cua_app_signed` (the Cua app or-CLI, verified by its code signature), `embedded_sdk` (another app that embeds-the cua SDK), `unsigned` (ad hoc or unsigned code) or `unknown` (the platform-cannot verify signatures). With `caller_kind_source: broker_verified` it comes-from the Keyvault broker's OS-verified caller identity; with `self_reported`-the calling process reported it, and an SDK process only ever reports-`embedded_sdk` or `unknown`. Paths, bundle ids and signing ids of callers are-never sent.--A modified build can remove its own telemetry. The receive-side count-(`cua_spacesd_session_deliveries`) is recorded by cua-spacesd inside the-sandbox, and only when the host enabled telemetry for that sandbox-(`CUA_SPACESD_TELEMETRY=1`, which the SDK sets on local sandboxes while-telemetry is on).-+Teleport caller kinds and how session deliveries are counted:+[How events are counted](/cua-sdk/concepts/telemetry-counting).diff --git a/docs/content/docs/cua-sdk/concepts/telemetry.mdx b/docs/content/docs/cua-sdk/concepts/telemetry.mdxindex e705f35831..4692e1132e 100644--- a/docs/content/docs/cua-sdk/concepts/telemetry.mdx+++ b/docs/content/docs/cua-sdk/concepts/telemetry.mdx@@ -81,19 +81,16 @@ These are the questions the events answer, and nothing else:- Are they still using Cua a day, a week and a month later?The activation funnel joins events on the anonymous install id, in order:-app launched (`cua_onboarding_step` `app_launched`), first run completed+installer started and finished (`cua_installer_step`), app launched+(`cua_onboarding_step` `app_launched`), first run completed(`onboarding_completed`), signed in (`signed_in`), first Space created(`first_space_created`), first Space ready (`first_space_ready`), first-teleport (`first_teleport`) and first agent run (`first_agent_run`). The-`first_*` steps are sent once per install.--An agent run is counted by the install that started it: once when it-starts, and once when any Cua program of that install first sees its first-turn end, fail or stop. Until then a marker with the same fixed words waits-in `~/.cua/telemetry/agent_runs/` (dropped after 7 days). Cua programs-inside an agent run (`CUA_AGENT_RUN_ID` set) record no agent runs, so a-Space never counts as an install. Retention counts installs with a-`cua_app_active` event on day 1, 7 and 30 after their first one.+teleport (`first_teleport`), first agent run (`first_agent_run`) and an+AI agent using a Space (`cua_agent_used_space`). The `first_*` steps are sent+once per install.++How the installers, an AI agent using a Space, agent runs and retention+are counted: [How events are counted](/cua-sdk/concepts/telemetry-counting).## Never collecteddiff --git a/docs/scripts/sandbox-navigation.test.ts b/docs/scripts/sandbox-navigation.test.tsindex 7819b7fc8f..ce0ace8dc2 100644--- a/docs/scripts/sandbox-navigation.test.ts+++ b/docs/scripts/sandbox-navigation.test.ts@@ -49,7 +49,13 @@ test('the Cua SDK has its task guides and one concept page', async () => {'omarchy','migrate-from-deprecated-packages',]);- assert.deepEqual((await metadata('cua-sdk/concepts')).pages, ['how-sandboxes-work', 'telemetry', 'telemetry-events']);+ assert.deepEqual((await metadata('cua-sdk/concepts')).pages, [+ 'how-sandboxes-work',+ 'telemetry',+ 'telemetry-events',+ 'telemetry-counting',+ 'telemetry-error-kinds',+ ]);for (const slug of guides.pages) {assert.equal(await routeExists(`/cua-sdk/guides/${slug}`), true, slug);}diff --git a/libs/cua/crates/cua-spaces/src/mcp/mod.rs b/libs/cua/crates/cua-spaces/src/mcp/mod.rsindex bf8957406c..623204613d 100644--- a/libs/cua/crates/cua-spaces/src/mcp/mod.rs+++ b/libs/cua/crates/cua-spaces/src/mcp/mod.rs@@ -391,7 +391,14 @@ impl McpServer {));}let args = params.get("arguments").cloned().unwrap_or(json!({}));- self.call_any(name, args).await.to_result()+ let out = self.call_any(name, args).await;+ // Only an MCP client (an AI agent) gets here; the app calls+ // tools through `call`. Anonymous: the tool's kind, once per+ // install per day (`cua_agent_used_space`).+ if !out.is_error && cua_telemetry::events::space_tool_group(name).is_some() {+ cua_telemetry::global().capture_agent_used_space(name);+ }+ out.to_result()}other => {return Some(error(diff --git a/libs/cua/crates/cua-spacesd-client/tests/environment_proxy.rs b/libs/cua/crates/cua-spacesd-client/tests/environment_proxy.rsindex c03b42b747..888fd70310 100644--- a/libs/cua/crates/cua-spacesd-client/tests/environment_proxy.rs+++ b/libs/cua/crates/cua-spacesd-client/tests/environment_proxy.rs@@ -43,48 +43,62 @@ fn relay_rpc_uses_environment_proxy() {"tls-web","tls-http",] {- let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();- let address = listener.local_addr().unwrap();- drop(listener);- let mut child = Command::new(std::env::current_exe().unwrap());- child.args(["--exact", "relay_rpc_uses_environment_proxy", "--nocapture"]);- for key in [- "HTTP_PROXY",- "http_proxy",- "HTTPS_PROXY",- "https_proxy",- "ALL_PROXY",- "all_proxy",- "NO_PROXY",- "no_proxy",- "REQUEST_METHOD",- ] {- child.env_remove(key);- }- if mode != "direct" {- child.env(+ // The probe port is released before the child binds it again, so another+ // process can take it in between. That is not a routing failure: retry the+ // case on a fresh port.+ let mut attempts_left = 5;+ let output = loop {+ let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();+ let address = listener.local_addr().unwrap();+ drop(listener);+ let mut child = Command::new(std::env::current_exe().unwrap());+ child.args(["--exact", "relay_rpc_uses_environment_proxy", "--nocapture"]);+ for key in [+ "HTTP_PROXY",+ "http_proxy",+ "HTTPS_PROXY",+ "https_proxy",+ "ALL_PROXY",+ "all_proxy",+ "NO_PROXY",+ "no_proxy",+ "REQUEST_METHOD",+ ] {+ child.env_remove(key);+ }+ if mode != "direct" {+ child.env(+ if mode.starts_with("tls-") {+ "HTTPS_PROXY"+ } else {+ "HTTP_PROXY"+ },+ format!("http://{address}"),+ );if mode.starts_with("tls-") {- "HTTPS_PROXY"- } else {- "HTTP_PROXY"- },- format!("http://{address}"),- );- if mode.starts_with("tls-") {- child.env("HTTP_PROXY", "http://invalid-http-proxy.invalid:9");+ child.env("HTTP_PROXY", "http://invalid-http-proxy.invalid:9");+ }}- }- if mode == "config" {- child.env("HTTP_PROXY", "http://user:[email protected]");- }- if mode == "bypass" {- child.env("NO_PROXY", "127.0.0.0/8");- }- let output = child- .env(MODE, mode)- .env(ADDRESS, address.to_string())- .output()- .unwrap();+ if mode == "config" {+ child.env("HTTP_PROXY", "http://user:[email protected]");+ }+ if mode == "bypass" {+ child.env("NO_PROXY", "127.0.0.0/8");+ }+ let output = child+ .env(MODE, mode)+ .env(ADDRESS, address.to_string())+ .output()+ .unwrap();+ attempts_left -= 1;+ if !output.status.success()+ && attempts_left > 0+ && String::from_utf8_lossy(&output.stderr).contains("AddrInUse")+ {+ continue;+ }+ break output;+ };if !output.status.success() {failures.push(format!("{mode}: {}\n{}",@@ -110,15 +124,17 @@ async fn head(socket: &mut TcpStream) -> String {async fn run_case() {let mode = std::env::var(MODE).unwrap();+ // Bind the proxy address first: the mock server's own ephemeral bind must not+ // be able to take the port the parent reserved for the proxy.+ let proxy = TcpListener::bind(std::env::var(ADDRESS).unwrap())+ .await+ .unwrap();let mock = MockServer::start(MockAuth {token: Some(TOKEN.into()),prefix: Some("/m/proxy-machine".into()),..Default::default()}).await;- let proxy = TcpListener::bind(std::env::var(ADDRESS).unwrap())- .await- .unwrap();let upstream = mock.addr;let attempts = Arc::new(AtomicUsize::new(0));let accepted = Arc::new(tokio::sync::Notify::new());diff --git a/libs/cua/crates/cua-telemetry/src/client.rs b/libs/cua/crates/cua-telemetry/src/client.rsindex 5e1187c665..21acd7b8bf 100644--- a/libs/cua/crates/cua-telemetry/src/client.rs+++ b/libs/cua/crates/cua-telemetry/src/client.rs@@ -43,6 +43,8 @@ const FIRST_RUN_FILE: &str = "first_run_recorded";const FUNNEL_DIR: &str = "funnel";/// The UTC day (days since the epoch) `cua_app_active` was last sent.const ACTIVE_DAY_FILE: &str = "active_day";+/// The UTC day of the last `cua_agent_used_space`.+const AGENT_USED_SPACE_FILE: &str = "agent_used_space_day";/// The Spaces app's experiments that are on (`cua_volume+sharing`, `none`).const EXPERIMENTS_FILE: &str = "experiments_on";@@ -634,6 +636,46 @@ impl Telemetry {r}+ /// Captures `cua_agent_used_space` when an AI agent's tool call `tool`+ /// reached a Space and worked: at most once per install per UTC day.+ /// Tools that do not act inside a Space are ignored.+ pub fn capture_agent_used_space(&self, tool: &str) -> Captured {+ let day = SystemTime::now()+ .duration_since(UNIX_EPOCH)+ .map(|d| d.as_secs() / 86_400)+ .unwrap_or(0);+ self.capture_agent_used_space_on(day, tool)+ }++ /// [`Self::capture_agent_used_space`] for a given UTC day (days since+ /// the epoch).+ pub fn capture_agent_used_space_on(&self, day: u64, tool: &str) -> Captured {+ let Some(ev) = crate::events::agent_used_space(tool) else {+ return Captured::Invalid(format!("{tool} does not act inside a Space"));+ };+ if !self.is_enabled() {+ return Captured::Disabled;+ }+ // A sandbox's cua-spacesd is not an install, and a Cua program inside+ // an agent run's Space is not the install the agent works for.+ if self.product() == "spacesd" || self.in_agent_run() {+ return Captured::Disabled;+ }+ let Some(dir) = self.state_dir() else {+ return Captured::Disabled;+ };+ let marker = dir.join(FUNNEL_DIR).join(AGENT_USED_SPACE_FILE);+ let today = format!("{day}\n");+ if std::fs::read_to_string(&marker).is_ok_and(|s| s == today) {+ return Captured::AlreadyRecorded;+ }+ let r = self.capture(ev);+ if r == Captured::Queued {+ let _ = fsutil::write_replace(&marker, today.as_bytes());+ }+ r+ }+/// The Spaces app's experiments that are on now (fixed ids; anything/// else is dropped): kept in the state directory, so the day's/// `cua_app_active` carries them whichever product sends it. Nothing isdiff --git a/libs/cua/crates/cua-telemetry/src/events.rs b/libs/cua/crates/cua-telemetry/src/events.rsindex 9895711985..a57e164e9a 100644--- a/libs/cua/crates/cua-telemetry/src/events.rs+++ b/libs/cua/crates/cua-telemetry/src/events.rs@@ -1159,3 +1159,86 @@ pub fn device_enroll(method: &str, outcome: Outcome) -> Option<Event> {.s("outcome", outcome.as_str())})}++/// `cua_installer_step`, as `install.sh` and `install.ps1` send it (they+/// build the same payload themselves; this builder is the reference the+/// tests hold them to). None for a step outside [`schema::INSTALLER_STEPS`].+/// `stage` and `error_kind` are kept only on `installer_failed`.+pub fn installer_step(+ step: &str,+ stage: &str,+ error_kind: &str,+ installer: &str,+ items: &[&str],+) -> Option<Event> {+ let s = pick(schema::INSTALLER_STEPS, step, "");+ if s.is_empty() {+ return None;+ }+ let failed = s == "installer_failed";+ let kind = if failed {+ pick(schema::INSTALL_ERROR_KINDS, error_kind, "other")+ } else {+ "none"+ };+ let outcome = match (failed, kind) {+ (false, _) => Outcome::Ok,+ (true, "cancelled") => Outcome::Cancelled,+ (true, _) => Outcome::Error,+ };+ let on: Vec<&str> = schema::INSTALL_ITEMS+ .iter()+ .copied()+ .filter(|i| items.iter().any(|x| x.trim() == *i))+ .collect();+ Some(+ Event::new(event::INSTALLER_STEP)+ .s("step", s)+ .s("outcome", outcome.as_str())+ .s(+ "install_stage",+ if failed {+ pick(schema::INSTALL_STAGES, stage, "other")+ } else {+ "none"+ },+ )+ .s("error_kind", kind)+ .s("installer", pick(schema::INSTALLERS, installer, "sh"))+ .owned(+ "items",+ if on.is_empty() {+ "none".into()+ } else {+ on.join("+")+ },+ ),+ )+}++/// The kind of Spaces tool `tool` is, when calling it means an agent used+/// a Space (it acts inside one); None for tools that only list, create,+/// share or administer Spaces.+pub fn space_tool_group(tool: &str) -> Option<&'static str> {+ Some(match tool {+ "space_bash" | "computer_shell" => "shell",+ "space_write"+ | "upload"+ | "send_file"+ | "download"+ | "computer_file_read"+ | "computer_file_write"+ | "computer_file_list" => "files",+ "list_tools" | "call_tool" => "services",+ "agent_start" | "agent_message" => "agents",+ "list_space_windows" | "teleport_app" => "other",+ t if t.starts_with("computer_") && !t.starts_with("computer_access_") => "computer",+ _ => return None,+ })+}++/// `cua_agent_used_space`: an agent's tool call reached a Space and worked.+/// None for a tool that does not act inside a Space.+pub fn agent_used_space(tool: &str) -> Option<Event> {+ space_tool_group(tool).map(|g| Event::new(event::AGENT_USED_SPACE).s("tool_group", g))+}diff --git a/libs/cua/crates/cua-telemetry/src/schema.rs b/libs/cua/crates/cua-telemetry/src/schema.rsindex 8aab946428..d790d04113 100644--- a/libs/cua/crates/cua-telemetry/src/schema.rs+++ b/libs/cua/crates/cua-telemetry/src/schema.rs@@ -303,9 +303,9 @@ pub const FEATURES: &[&str] = &["upsell_plan_card_click","upsell_account_menu_click",];+/// Install steps live in [`INSTALLER_STEPS`] (`cua_installer_step`): the+/// installers send them before any Cua program is on the machine.pub const ONBOARDING_STEPS: &[&str] = &[- "installer_started",- "installer_completed","app_launched","onboarding_shown","onboarding_completed",@@ -331,6 +331,38 @@ pub const ONBOARDING_STEPS: &[&str] = &["checklist_dismissed","checklist_reopened",];+/// What `install.sh` and `install.ps1` report about their own run+/// (`cua_installer_step`): started, finished, or stopped by an error.+pub const INSTALLER_STEPS: &[&str] = &[+ "installer_started",+ "installer_completed",+ "installer_failed",+];+/// The part of the install that was running when it stopped (`none` on+/// `installer_started` and `installer_completed`).+pub const INSTALL_STAGES: &[&str] = &[+ "none", "platform", "manifest", "cli", "runtime", "app", "driver", "sign_in", "host", "other",+];+/// Why an install stopped, as a fixed word the installer picks from its+/// own error; never the message.+pub const INSTALL_ERROR_KINDS: &[&str] = &[+ "none",+ "network",+ "checksum",+ "signature",+ "unsupported_platform",+ "permission_denied",+ "missing_tool",+ "cancelled",+ "other",+];+/// Which installer sent the event.+pub const INSTALLERS: &[&str] = &["sh", "ps1"];+/// What the install was asked to put on the machine (joined with `+`).+pub const INSTALL_ITEMS: &[&str] = &["cli", "spaces", "cua_driver", "host"];+/// The kind of Spaces tool an agent's call used (`cua_agent_used_space`).+pub const SPACE_TOOL_GROUPS: &[&str] =+ &["shell", "files", "computer", "services", "agents", "other"];/// Whether an `app_launched` opens the first run (`unknown` elsewhere).pub const ONBOARDING_ELIGIBLE: &[&str] = &["yes", "no", "unknown"];/// The Spaces app's first-run pages (`cua-spaces-app-core` `onboarding`).@@ -566,7 +598,11 @@ pub const COMMON: &[Prop] = &[Kind::ConstBool(true),"No location lookup from the IP.",),- p("$lib", Kind::Enum(&["cua-telemetry"]), "Sender library."),+ p(+ "$lib",+ Kind::Enum(&["cua-telemetry", "cua-installer"]),+ "Sender library (cua-installer: install.sh or install.ps1).",+ ),p("$lib_version", Kind::Version, "Sender library version."),];@@ -666,6 +702,8 @@ pub mod event {pub const APP_UPDATE: &str = "cua_app_update";pub const DEVICE_ENROLL: &str = "cua_device_enroll";pub const EXPERIMENT: &str = "cua_experiment";+ pub const INSTALLER_STEP: &str = "cua_installer_step";+ pub const AGENT_USED_SPACE: &str = "cua_agent_used_space";}/// Every event.@@ -908,6 +946,56 @@ pub const EVENTS: &[EventSpec] = &[],purpose: "Install and onboarding funnel: where people drop off.",},+ EventSpec {+ name: event::INSTALLER_STEP,+ version: 1,+ tier: Tier::Usage,+ sample_rate: 1.0,+ // The installers install the CLI, so they report as it.+ products: &["cli"],+ props: &[+ p(+ "step",+ Kind::Enum(INSTALLER_STEPS),+ "installer_started, installer_completed or installer_failed.",+ ),+ OUTCOME,+ p(+ "install_stage",+ Kind::Enum(INSTALL_STAGES),+ "installer_failed only: the part of the install that was running (platform, manifest, cli, runtime, app, driver, sign_in, host, other); none otherwise.",+ ),+ p(+ "error_kind",+ Kind::Enum(INSTALL_ERROR_KINDS),+ "installer_failed only: why, as a fixed word (network, checksum, signature, unsupported_platform, permission_denied, missing_tool, cancelled, other); none otherwise. Never the message.",+ ),+ p(+ "installer",+ Kind::Enum(INSTALLERS),+ "install.sh (sh) or install.ps1 (ps1).",+ ),+ p(+ "items",+ Kind::Set(INSTALL_ITEMS),+ "What the install was asked for: cli, spaces, cua_driver, host, joined with +.",+ ),+ ],+ purpose: "Install funnel: how many installs start, finish, and where and why they fail.",+ },+ EventSpec {+ name: event::AGENT_USED_SPACE,+ version: 1,+ tier: Tier::Usage,+ sample_rate: 1.0,+ products: ALL,+ props: &[p(+ "tool_group",+ Kind::Enum(SPACE_TOOL_GROUPS),+ "The kind of tool the call used: shell, files, computer, services, agents or other.",+ )],+ purpose: "First success and agent retention: an AI agent's tool call reached a Space and worked. At most one per install per UTC day.",+ },EventSpec {name: event::AGENT_RUN_STARTED,version: 1,diff --git a/libs/cua/crates/cua-telemetry/tests/privacy.rs b/libs/cua/crates/cua-telemetry/tests/privacy.rsindex adf211f62c..9cacc49f07 100644--- a/libs/cua/crates/cua-telemetry/tests/privacy.rs+++ b/libs/cua/crates/cua-telemetry/tests/privacy.rs@@ -219,6 +219,15 @@ fn all_builders(s: &str) -> Vec<Event> {v.push(events::app_update("downloaded", s, s).unwrap());v.push(events::device_enroll("rekey", Outcome::Ok).unwrap());v.push(events::experiment("experiment_off", "your_cloud").unwrap());+ v.push(events::installer_step("installer_started", s, s, s, &[s]).unwrap());+ v.push(events::installer_step("installer_failed", s, s, "ps1", &[s, "cli"]).unwrap());+ v.push(+ events::installer_step("installer_failed", "cli", "cancelled", "sh", &["host"]).unwrap(),+ );+ v.push(events::agent_used_space("space_bash").unwrap());+ v.push(events::agent_used_space("computer_click").unwrap());+ assert!(events::installer_step(s, "cli", "network", "sh", &[]).is_none());+ assert!(events::agent_used_space(s).is_none());v}@@ -904,6 +913,7 @@ fn docs_page_names_every_event_and_property() {let page = concat!(include_str!("../../../../../docs/content/docs/cua-sdk/concepts/telemetry.mdx"),include_str!("../../../../../docs/content/docs/cua-sdk/concepts/telemetry-events.mdx"),+ include_str!("../../../../../docs/content/docs/cua-sdk/concepts/telemetry-counting.mdx"),);for spec in schema::EVENTS {assert!(@@ -1450,3 +1460,129 @@ fn a_failed_host_space_never_reports_no_error_kind() {let ok = events::host_space_provided("vm", "macos", Outcome::Ok, None);assert_eq!(ok.props["error_kind"], "none");}++/// `cua_installer_step` is sent by install.sh and install.ps1, which build+/// the payload themselves. The fixture they are tested against+/// (`scripts/install/tests/test_install_telemetry.py`) must pass the same+/// schema check as every other event.+#[test]+fn installer_payloads_validate_against_the_declared_schema() {+ let text = include_str!("../../../../../scripts/install/tests/fixtures/installer-step.json");+ let doc: Value = serde_json::from_str(text).unwrap();+ let events = doc["events"].as_array().unwrap();+ assert_eq!(events.len(), 3);+ for e in events {+ let name = e["event"].as_str().unwrap();+ assert_eq!(name, schema::event::INSTALLER_STEP);+ let props = e["properties"].as_object().unwrap();+ schema::validate(name, props).unwrap_or_else(|err| panic!("{e}: {err}"));+ let id = e["distinct_id"].as_str().unwrap();+ assert!(id.len() == 32 && id.bytes().all(|b| b.is_ascii_hexdigit()));+ }+ // The builder agrees with the fixture on the event's own properties.+ let built =+ events::installer_step("installer_failed", "cli", "checksum", "sh", &["cli"]).unwrap();+ for (k, v) in &built.props {+ assert_eq!(&events[1]["properties"][k], v, "{k}");+ }+ let built = events::installer_step(+ "installer_started",+ "cli",+ "network",+ "sh",+ &["spaces", "cli"],+ )+ .unwrap();+ assert_eq!(built.props["install_stage"], "none");+ assert_eq!(built.props["error_kind"], "none");+ assert_eq!(built.props["items"], "cli+spaces");+}++/// An agent's tool call that reached a Space: one `cua_agent_used_space`+/// per install per UTC day, only for tools that act inside a Space, never+/// while telemetry is off and never from inside an agent run's Space.+#[test]+fn agent_used_space_is_sent_once_per_day_for_space_tools_only() {+ let h = tempfile::tempdir().unwrap();+ let (t, sink) = ready(h.path(), &[]);+ assert!(matches!(+ t.capture_agent_used_space_on(20_000, "list_spaces"),+ Captured::Invalid(_)+ ));+ assert!(matches!(+ t.capture_agent_used_space_on(20_000, "computer_access_grant"),+ Captured::Invalid(_)+ ));+ assert_eq!(+ t.capture_agent_used_space_on(20_000, "space_bash"),+ Captured::Queued+ );+ assert_eq!(+ t.capture_agent_used_space_on(20_000, "call_tool"),+ Captured::AlreadyRecorded+ );+ let (t2, _) = ready(h.path(), &[]);+ assert_eq!(+ t2.capture_agent_used_space_on(20_000, "upload"),+ Captured::AlreadyRecorded+ );+ assert_eq!(+ t.capture_agent_used_space_on(20_001, "call_tool"),+ Captured::Queued+ );+ t.flush(Duration::from_secs(1));+ let used: Vec<Value> = sink+ .events()+ .into_iter()+ .filter(|e| e["event"] == "cua_agent_used_space")+ .collect();+ assert_eq!(used.len(), 2);+ assert_eq!(used[0]["properties"]["tool_group"], "shell");+ assert_eq!(used[1]["properties"]["tool_group"], "services");++ for env in [+ &[("DO_NOT_TRACK", "1")][..],+ &[("CUA_TELEMETRY", "0")][..],+ &[("CI", "true")][..],+ &[("CUA_AGENT_RUN_ID", "run-1")][..],+ ] {+ let h = tempfile::tempdir().unwrap();+ let (t, sink) = ready(h.path(), env);+ assert_eq!(+ t.capture_agent_used_space_on(20_000, "space_bash"),+ Captured::Disabled,+ "{env:?}"+ );+ t.flush(Duration::from_secs(1));+ assert!(sink.events().is_empty(), "{env:?}");+ assert!(+ !h.path()+ .join("telemetry/funnel/agent_used_space_day")+ .exists()+ );+ }+}++#[test]+fn space_tool_groups_cover_the_tools_that_act_inside_a_space() {+ use events::space_tool_group as g;+ assert_eq!(g("space_bash"), Some("shell"));+ assert_eq!(g("computer_shell"), Some("shell"));+ assert_eq!(g("send_file"), Some("files"));+ assert_eq!(g("computer_file_write"), Some("files"));+ assert_eq!(g("call_tool"), Some("services"));+ assert_eq!(g("agent_start"), Some("agents"));+ assert_eq!(g("computer_screenshot"), Some("computer"));+ assert_eq!(g("teleport_app"), Some("other"));+ for admin in [+ "list_spaces",+ "create_space",+ "delete_space",+ "share_space",+ "computer_access_list",+ "volume_ls",+ "",+ ] {+ assert_eq!(g(admin), None, "{admin}");+ }+}diff --git a/scripts/install/README.md b/scripts/install/README.mdindex 465a24dd0c..59b06c29fd 100644--- a/scripts/install/README.md+++ b/scripts/install/README.md@@ -54,6 +54,10 @@ For cua-driver, `--prefix DIR` becomes `--bin-dir DIR/bin`, `--require-signaturePass install.sh options through the pipe with `sh -s --`, for example `curl -fsSL https://cua.ai/install.sh | sh -s -- --cli-only --yes`. `--only` and `--select` cannot be combined, nor can `--cli-only` with other items. Set `CUA_INSTALL_NONINTERACTIVE=1` to never prompt.+## Telemetry++Both installers send three anonymous events about their own run (`cua_installer_step`: `installer_started`, then `installer_completed` or `installer_failed` with the stage and a fixed reason word), and print a two-line notice before the first one. They follow the switches every Cua program uses: `DO_NOT_TRACK=1`, `CUA_TELEMETRY=0`, `[telemetry] enabled = "off"` in `$CUA_HOME/config.toml`, and off in CI unless `CUA_TELEMETRY=1`. `--dry-run` sends nothing. They create or reuse `$CUA_HOME/telemetry/install_id`, the anonymous id the CLI and app send with, and never send a name, path, URL or error message. The tests set `CUA_TELEMETRY_FORBID_NETWORK=1`, which allows only a loopback `CUA_TELEMETRY_ENDPOINT`. See [Telemetry and privacy](https://cua.ai/docs/cua-sdk/concepts/telemetry).+## What it printsOne plain line per step ("Installing the command-line tools to ...", "Installing the Cua Spaces app to ..."), then what was installed and what to do next: open Cua Spaces, or sign in and run `cua agents setup` to connect your AI agents. `--verbose` adds `cua-install:` lines with the platform, the manifest, each download and the signature checks.diff --git a/scripts/install/install.ps1 b/scripts/install/install.ps1index 7a643de202..a548f54919 100644--- a/scripts/install/install.ps1+++ b/scripts/install/install.ps1@@ -138,6 +138,164 @@ function Write-Step([string]$Message) { Write-Host $Message }function Write-Info([string]$Message) { Write-Verbose "cua-install: $Message" }function Stop-Install([string]$Message) { throw "cua-install: error: $Message" }+# ---------------------------------------------------------------- telemetry+# Three anonymous events about this install (cua_installer_step):+# installer_started, then installer_completed or installer_failed with the+# stage that was running and a fixed reason word. The same switches as every+# Cua program (DO_NOT_TRACK, CUA_TELEMETRY, `[telemetry] enabled = "off"` in+# $CUA_HOME\config.toml, off in CI) and the same anonymous install id. Never a+# name, a path, a URL or an error message; nothing with -DryRun. Best effort:+# a send never fails the install and waits at most 3 seconds.+$script:TelKeyProd = 'phc_eSkLnbLxsnYFaXksif1ksbrNzYlJShr35miFLDppF14'+$script:TelKeyStaging = 'phc_xDNLpzypNLVGmzQeBZEgZDZK7Ea6yYAmAnGZ6hMtHNKG'+$script:TelCiVars = @('CI', 'CONTINUOUS_INTEGRATION', 'GITHUB_ACTIONS', 'GITLAB_CI', 'JENKINS_URL', 'CIRCLECI', 'BUILDKITE',+ 'TF_BUILD', 'TEAMCITY_VERSION', 'TRAVIS', 'APPVEYOR', 'BITBUCKET_BUILD_NUMBER', 'CODEBUILD_BUILD_ID', 'DRONE', 'HUDSON_URL')+$script:Tel = $null+$script:TelStage = 'other'+$script:TelVersion = ''++# 'on', 'off' or $null for a switch word.+function ConvertTo-TelSwitch([string]$Value) {+ if ($null -eq $Value) { return $null }+ switch -Regex ($Value.Trim().ToLowerInvariant()) {+ '^(0|false|no|off|disable|disabled)$' { return 'off' }+ '^(1|true|yes|on|enable|enabled)$' { return 'on' }+ }+ return $null+}++function Get-TelConfigSwitch([string]$CuaHome) {+ $f = Join-Path $CuaHome 'config.toml'+ if (-not (Test-Path -LiteralPath $f)) { return $null }+ $section = ''+ foreach ($line in (Get-Content -LiteralPath $f -ErrorAction SilentlyContinue)) {+ if ($line -match '^\s*\[\s*([^\]]+?)\s*\]') { $section = $Matches[1]; continue }+ if ($section -eq 'telemetry' -and $line -match '^\s*enabled\s*=\s*"?([^"#\s]*)"?') { return (ConvertTo-TelSwitch $Matches[1]) }+ }+ return $null+}++# Whether install events may be sent here (the rules of cua-telemetry's config.rs).+function Test-TelemetryEnabled([string]$CuaHome) {+ $dnt = "$env:DO_NOT_TRACK".Trim().ToLowerInvariant()+ if ($dnt -and $dnt -ne '0' -and $dnt -ne 'false') { return $false }+ $sw = ConvertTo-TelSwitch $env:CUA_TELEMETRY+ if ($sw) { return ($sw -eq 'on') }+ if ((ConvertTo-TelSwitch $env:CUA_TELEMETRY_ENABLED) -eq 'off') { return $false }+ if ((ConvertTo-TelSwitch $env:CUA_TELEMETRY_DISABLED) -eq 'on') { return $false }+ $sw = Get-TelConfigSwitch $CuaHome+ if ($sw) { return ($sw -eq 'on') }+ return -not (Test-TelCi)+}++function Test-TelCi {+ foreach ($v in $script:TelCiVars) {+ $x = "$([Environment]::GetEnvironmentVariable($v))".Trim().ToLowerInvariant()+ if ($x -and $x -ne 'false' -and $x -ne '0') { return $true }+ }+ return $false+}++function New-TelHex {+ $b = New-Object byte[] 16+ [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($b)+ return (($b | ForEach-Object { $_.ToString('x2') }) -join '')+}++# Decides, shows the notice, and loads or creates the anonymous install id.+function Initialize-CuaTelemetry {+ $script:Tel = $null+ if ($DryRun) { return }+ $key = if ($env:CUA_POSTHOG_KEY) { $env:CUA_POSTHOG_KEY } elseif ("$env:CUA_ENV".Trim().ToLowerInvariant() -eq 'staging') { $script:TelKeyStaging } else { $script:TelKeyProd }+ if (-not $key) { return }+ $cuaHome = if ($env:CUA_HOME) { $env:CUA_HOME } else { Join-Path $HOME '.cua' }+ if (-not (Test-TelemetryEnabled $cuaHome)) { return }+ $endpoint = if ($env:CUA_TELEMETRY_ENDPOINT) { $env:CUA_TELEMETRY_ENDPOINT } else { 'https://eu.i.posthog.com/batch/' }+ if ((ConvertTo-TelSwitch $env:CUA_TELEMETRY_FORBID_NETWORK) -eq 'on' -and $endpoint -notmatch '^http://(127\.0\.0\.1|localhost|\[::1\])[:/]') { return }+ $dir = Join-Path $cuaHome 'telemetry'+ $idFile = Join-Path $dir 'install_id'+ $id = $null+ try { $id = (Get-Content -Raw -LiteralPath $idFile -ErrorAction Stop).Trim() } catch { }+ if ($id -notmatch '^[0-9a-f]{32}$') {+ $id = New-TelHex+ try {+ New-Item -ItemType Directory -Force -Path $dir | Out-Null+ Set-Content -LiteralPath $idFile -Value $id -NoNewline+ } catch { Write-Verbose "could not store the telemetry id: $_" }+ }+ $script:Tel = @{ Key = $key; Endpoint = $endpoint; Id = $id; Session = (New-TelHex); Ci = (Test-TelCi); Ended = $false }+ Write-Host 'Cua sends anonymous data about this install (whether it worked, and where it stopped if not). Never names, file paths or anything you type.'+ Write-Host 'To turn it off, set DO_NOT_TRACK=1. Details: https://cua.ai/docs/cua-sdk/concepts/telemetry'+}++# The fixed reason word for an install error message.+function Get-CuaInstallErrorKind([string]$Message) {+ switch -Wildcard ($Message) {+ '*checksum mismatch*' { return 'checksum' }+ '*signature*' { return 'signature' }+ '*release identity*' { return 'signature' }+ '*unsupported CPU*' { return 'unsupported_platform' }+ '*cannot create*' { return 'permission_denied' }+ '*ermission*' { return 'permission_denied' }+ '*Access*denied*' { return 'permission_denied' }+ '*cancelled*' { return 'cancelled' }+ '*download failed*' { return 'network' }+ '*unsupported release manifest*' { return 'network' }+ '*Unable to connect*' { return 'network' }+ '*could not be resolved*' { return 'network' }+ '*timed out*' { return 'network' }+ '*No such host*' { return 'network' }+ '*404*' { return 'network' }+ }+ return 'other'+}++# The cua_installer_step payload (one batch body).+function New-CuaInstallerStepBody($Sel, [string]$Step, [string]$Stage = 'none', [string]$Kind = 'none') {+ $failed = $Step -eq 'installer_failed'+ if (-not $failed) { $Stage = 'none'; $Kind = 'none' }+ $outcome = if (-not $failed) { 'ok' } elseif ($Kind -eq 'cancelled') { 'cancelled' } else { 'error' }+ $items = @()+ if ($Sel) {+ if ($Sel.cli) { $items += 'cli' }+ if ($Sel.spaces) { $items += 'spaces' }+ if ($Sel['cua-driver']) { $items += 'cua_driver' }+ if ($Sel.host) { $items += 'host' }+ }+ $pv = if ($script:TelVersion) { $script:TelVersion } elseif ($Version) { $Version.TrimStart('v') } else { '' }+ if ($pv -notmatch '^\d+\.\d+\.\d+(-[0-9A-Za-z.-]+)?$') { $pv = '0.0.0' }+ $isWin = (-not (Test-Path variable:IsWindows)) -or $IsWindows+ $os = if ($isWin) { 'windows' } elseif ($IsMacOS) { 'macos' } elseif ($IsLinux) { 'linux' } else { 'other' }+ $major = [int][Environment]::OSVersion.Version.Major+ if ($major -gt 999 -or $major -lt 0) { $major = 0 }+ $archRaw = if ($env:PROCESSOR_ARCHITEW6432) { $env:PROCESSOR_ARCHITEW6432 } elseif ($env:PROCESSOR_ARCHITECTURE) { $env:PROCESSOR_ARCHITECTURE } else { [System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture.ToString() }+ $arch = switch -Regex ($archRaw) { '^(AMD64|x64|X64|x86_64)$' { 'x86_64' } '^(ARM64|Arm64|aarch64)$' { 'arm64' } default { 'other' } }+ $u = New-TelHex+ $uuid = '{0}-{1}-4{2}-8{3}-{4}' -f $u.Substring(0, 8), $u.Substring(8, 4), $u.Substring(13, 3), $u.Substring(17, 3), $u.Substring(20, 12)+ $props = [ordered]@{+ telemetry_schema_version = 1; event_version = 1; product = 'cli'; product_version = $pv+ os_family = $os; os_major = $major; arch = $arch; is_ci = [bool]$script:Tel.Ci+ is_synthetic = ((ConvertTo-TelSwitch $env:CUA_TELEMETRY_SYNTHETIC) -eq 'on'); process_session_id = $script:Tel.Session+ sample_rate = 1; '$process_person_profile' = $false; '$geoip_disable' = $true+ '$lib' = 'cua-installer'; '$lib_version' = '1.0.0'+ step = $Step; outcome = $outcome; install_stage = $Stage; error_kind = $Kind; installer = 'ps1'+ items = $(if ($items.Count) { $items -join '+' } else { 'none' })+ }+ $event = [ordered]@{ event = 'cua_installer_step'; distinct_id = $script:Tel.Id; uuid = $uuid+ timestamp = [DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ'); properties = $props }+ return (ConvertTo-Json -Depth 6 -Compress ([ordered]@{ api_key = $script:Tel.Key; batch = @($event) }))+}++function Send-CuaInstallerStep($Sel, [string]$Step, [string]$Stage = 'none', [string]$Kind = 'none') {+ if (-not $script:Tel -or $script:Tel.Ended) { return }+ if ($Step -ne 'installer_started') { $script:Tel.Ended = $true }+ try {+ $body = New-CuaInstallerStepBody $Sel $Step $Stage $Kind+ if ((ConvertTo-TelSwitch $env:CUA_TELEMETRY_DEBUG) -eq 'on') { Write-Host "[cua telemetry] $body" }+ Invoke-RestMethod -Method Post -Uri $script:Tel.Endpoint -ContentType 'application/json' -Body $body -TimeoutSec 3 | Out-Null+ } catch { Write-Verbose "install telemetry not sent: $_" }+}+function Get-CuaPlatform {param([string]$Arch = $env:CUA_INSTALL_ARCH)if (-not $Arch) {@@ -603,13 +761,33 @@ function Invoke-CuaInstall {$ProgressPreference = 'SilentlyContinue'Test-PairArgs$sel = Get-CuaSelection+ Initialize-CuaTelemetry+ $script:TelStage = 'platform'+ try {+ Invoke-CuaInstallSteps $sel+ $script:TelStage = 'other'+ Send-CuaInstallerStep $sel 'installer_completed'+ } catch {+ Send-CuaInstallerStep $sel 'installer_failed' $script:TelStage (Get-CuaInstallErrorKind "$_")+ throw+ } finally {+ # Stopped without an error (Ctrl+C): cancelled.+ Send-CuaInstallerStep $sel 'installer_failed' $script:TelStage 'cancelled'+ }+}++function Invoke-CuaInstallSteps($sel) {if ($sel.spaces -and -not $script:SpacesSupported) {Write-Step 'Cua Spaces is not available on this platform; skipping the app'$sel.spaces = $false}if ($Version -and $Version.TrimStart('v') -notmatch '^\d+\.\d+\.\d+') { Stop-Install "-Version must look like 1.2.3, not '$Version'" }$platform = Get-CuaPlatform- if (Test-WantsChecklist) { $sel = Read-CuaSelection $sel }+ Send-CuaInstallerStep $sel 'installer_started'+ if (Test-WantsChecklist) {+ $picked = Read-CuaSelection $sel+ foreach ($k in @($picked.Keys)) { $sel[$k] = $picked[$k] }+ }if ($DryRun) { Write-Step 'Dry run: nothing will be changed' }$tmp = Join-Path ([IO.Path]::GetTempPath()) ("cua-install-" + [Guid]::NewGuid().ToString('N'))New-Item -ItemType Directory -Force -Path $tmp | Out-Null@@ -624,15 +802,19 @@ function Invoke-CuaInstall {$manifestUrl = $null# Nothing to download (-Pair on a computer that has Cua): no manifest to read.if ($sel.cli -or $sel.spaces -or $sel['cua-driver']) {+ $script:TelStage = 'manifest'$manifestUrl = Get-ManifestUrl$manifestFile = Join-Path $tmp 'release-artifacts.json'Get-Remote $manifestUrl $manifestFile$manifest = Get-Content -Raw -LiteralPath $manifestFile | ConvertFrom-Jsonif ($manifest.schema -ne 1) { Stop-Install "unsupported release manifest at $manifestUrl" }Write-Info "platform $platform, manifest $manifestUrl"+ $script:TelVersion = "$(@($manifest.artifacts | Where-Object { $_.component -eq 'cli' } | Select-Object -First 1).version)"}+ $script:TelStage = 'cli'if ($sel.cli) { $cua = Install-Cli $manifest $manifestUrl $platform $tmp }if ($sel.spaces) {+ $script:TelStage = 'app'if ($Mode) {$cuaHome = if ($env:CUA_HOME) { $env:CUA_HOME } else { Join-Path $HOME '.cua' }$modeFile = Join-Path $cuaHome 'spaces-install-mode'@@ -656,13 +838,16 @@ function Invoke-CuaInstall {} catch { Write-Verbose "could not record the install channel: $_" }}$env:CUA_INSTALL_CHANNEL = 'install_script'+ $script:TelStage = 'driver'if ($sel['cua-driver']) { Install-CuaDriver $cua $tmp $manifestUrl }+ $script:TelStage = 'sign_in'if ($cua -and -not $Pair -and -not $CliOnly -and -not $NoOnboarding -and -not $DryRun -and (Test-Interactive)) {Write-Host ''Write-Step 'Signing in to Cua'& $cua auth loginif ($LASTEXITCODE -ne 0) { Write-Warning 'Sign-in did not finish. Run "cua auth login" later (add --remote if this computer has no web browser).' } else { $script:SignedIn = $true }}+ $script:TelStage = 'host'if ($sel.host) { Invoke-HostSetup $cua }Write-CuaSummary $selif ($script:PathHint) {diff --git a/scripts/install/install.sh b/scripts/install/install.shindex 4bda6cdada..987240b0c9 100755--- a/scripts/install/install.sh+++ b/scripts/install/install.sh@@ -55,6 +55,14 @@# CUA_INSTALL_OS, CUA_INSTALL_ARCH override platform detection# CUA_INSTALL_MINISIGN_PUBKEY minisign public key for artifacts# CUA_HOME cua state dir (default ~/.cua)+# DO_NOT_TRACK=1 or CUA_TELEMETRY=0 send no install events (see below)+#+# Telemetry: the installer sends three anonymous events about its own run+# (installer started, finished, or failed with the step and a fixed reason+# word), under the same switches as every Cua program: DO_NOT_TRACK,+# CUA_TELEMETRY, `[telemetry] enabled = "off"` in $CUA_HOME/config.toml, and+# off in CI. Never names, paths or messages. Nothing on --dry-run.+# https://cua.ai/docs/cua-sdk/concepts/telemetryset -euREPO="${CUA_INSTALL_REPO:-trycua/cua}"@@ -114,6 +122,7 @@ info() { [ "$verbose" = 0 ] || printf 'cua-install: %s\n' "$*"; }warn() { printf 'cua-install: warning: %s\n' "$*" >&2; }die() {printf 'cua-install: error: %s\n' "$*" >&2+ tel_kind="$(tel_error_kind "$*")"exit 1}@@ -122,6 +131,11 @@ usage() {}cleanup() {+ rc=$?+ if [ "$rc" != 0 ] && [ "$tel_on" = 1 ] && [ "$tel_ended" = 0 ]; then+ [ "$rc" != 130 ] || tel_kind=cancelled+ tel_send installer_failed "$tel_stage" "${tel_kind:-other}"+ fiif [ -n "$tty_saved" ]; thenstty "$tty_saved" </dev/tty 2>/dev/null || truetty_saved=""@@ -136,6 +150,202 @@ cleanup() {trap cleanup EXITtrap 'exit 130' INT TERM+# ---------------------------------------------------------------- telemetry++# Three anonymous events about this install (cua_installer_step):+# installer_started, then installer_completed or installer_failed with the+# stage that was running and a fixed reason word. The same switches as+# every Cua program (libs/cua/crates/cua-telemetry/src/config.rs); the same+# anonymous install id, so the app's later events continue this install.+# Never a name, a path, a URL or an error message. Best effort: a send never+# fails or slows the install by more than a few seconds.+TEL_ENDPOINT_DEFAULT="https://eu.i.posthog.com/batch/"+TEL_KEY_PROD="phc_eSkLnbLxsnYFaXksif1ksbrNzYlJShr35miFLDppF14"+TEL_KEY_STAGING="phc_xDNLpzypNLVGmzQeBZEgZDZK7Ea6yYAmAnGZ6hMtHNKG"+tel_on=0+tel_ended=0+tel_stage=other+tel_kind=""+tel_version=""+tel_id=""+tel_session=""+tel_ci=false++tel_lower() { printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | tr -d '[:space:]'; }++# tel_bool VALUE: prints on, off, or nothing (unset or not a switch word).+tel_bool() {+ case "$(tel_lower "$1")" in+ 0 | false | no | off | disable | disabled) echo off ;;+ 1 | true | yes | on | enable | enabled) echo on ;;+ esac+}++# The [telemetry] enabled value in $CUA_HOME/config.toml (on, off, or nothing).+tel_config() {+ f="${CUA_HOME:-$HOME/.cua}/config.toml"+ [ -r "$f" ] || return 0+ v="$(awk '+ /^[[:space:]]*\[/ { sect = $0; gsub(/[[:space:]\[\]]/, "", sect); next }+ sect == "telemetry" && /^[[:space:]]*enabled[[:space:]]*=/ {+ sub(/^[^=]*=[[:space:]]*/, ""); sub(/[[:space:]]*(#.*)?$/, ""); gsub(/"/, ""); print; exit+ }' "$f" 2>/dev/null)"+ tel_bool "$v"+}++tel_is_ci() {+ for v in CI CONTINUOUS_INTEGRATION GITHUB_ACTIONS GITLAB_CI JENKINS_URL CIRCLECI BUILDKITE \+ TF_BUILD TEAMCITY_VERSION TRAVIS APPVEYOR BITBUCKET_BUILD_NUMBER CODEBUILD_BUILD_ID DRONE HUDSON_URL; do+ eval "x=\${$v:-}"+ x="$(tel_lower "$x")"+ case "$x" in "" | false | 0) ;; *) return 0 ;; esac+ done+ return 1+}++tel_hex() { # BYTES: random lowercase hex+ od -An -N"$1" -tx1 /dev/urandom 2>/dev/null | tr -d ' \n'+}++# telemetry_init: decide, show the notice, load or create the install id.+telemetry_init() {+ tel_on=0+ [ "$dry_run" = 0 ] || return 0+ if [ -n "${CUA_POSTHOG_KEY:-}" ]; then+ tel_key="$CUA_POSTHOG_KEY"+ elif [ "$(tel_lower "${CUA_ENV:-}")" = staging ]; then+ tel_key="$TEL_KEY_STAGING"+ else+ tel_key="$TEL_KEY_PROD"+ fi+ [ -n "$tel_key" ] || return 0+ case "$(tel_lower "${DO_NOT_TRACK:-}")" in "" | 0 | false) ;; *) return 0 ;; esac+ tel_is_ci && tel_ci=true+ sw="$(tel_bool "${CUA_TELEMETRY:-}")"+ if [ -z "$sw" ]; then+ [ "$(tel_bool "${CUA_TELEMETRY_ENABLED:-}")" != off ] || return 0+ [ "$(tel_bool "${CUA_TELEMETRY_DISABLED:-}")" != on ] || return 0+ sw="$(tel_config)"+ [ -n "$sw" ] || { [ "$tel_ci" = true ] && sw=off; }+ fi+ [ "$sw" != off ] || return 0+ tel_endpoint="${CUA_TELEMETRY_ENDPOINT:-$TEL_ENDPOINT_DEFAULT}"+ if [ "$(tel_bool "${CUA_TELEMETRY_FORBID_NETWORK:-}")" = on ]; then+ case "$tel_endpoint" in+ http://127.0.0.1[:/]* | http://localhost[:/]* | http://\[::1\][:/]*) ;;+ *) return 0 ;;+ esac+ fi+ has curl || has wget || return 0+ dir="${CUA_HOME:-$HOME/.cua}/telemetry"+ tel_id="$(cat "$dir/install_id" 2>/dev/null | tr -d '[:space:]')"+ case "$tel_id" in+ *[!0-9a-f]* | "") tel_id="" ;;+ esac+ [ "${#tel_id}" = 32 ] || tel_id=""+ if [ -z "$tel_id" ]; then+ tel_id="$(tel_hex 16)"+ [ "${#tel_id}" = 32 ] || return 0+ (umask 077 && mkdir -p "$dir" && printf '%s' "$tel_id" >"$dir/install_id") 2>/dev/null || true+ fi+ tel_session="$(tel_hex 16)"+ [ "${#tel_session}" = 32 ] || return 0+ tel_on=1+ say "Cua sends anonymous data about this install (whether it worked, and where it stopped if not). Never names, file paths or anything you type."+ say "To turn it off, set DO_NOT_TRACK=1. Details: https://cua.ai/docs/cua-sdk/concepts/telemetry"+}++# tel_error_kind MESSAGE: the fixed reason word for a die message.+tel_error_kind() {+ case "$1" in+ *"checksum mismatch"*) echo checksum ;;+ *signature* | *"release identity"*) echo signature ;;+ *"unsupported OS"* | *"unsupported CPU"* | *"on Windows, run"*) echo unsupported_platform ;;+ *"cannot create"* | *ermission*) echo permission_denied ;;+ *"need curl"* | *"need sha256sum"* | *"not found; cannot"* | *"needs bash"* | *"needs QEMU"*) echo missing_tool ;;+ *cancelled*) echo cancelled ;;+ *"download failed"* | *"could not read"* | *"unsupported release manifest"*) echo network ;;+ *) echo other ;;+ esac+}++tel_os() {+ case "$(uname -s 2>/dev/null)" in+ Darwin) echo macos ;;+ Linux) echo linux ;;+ MINGW* | MSYS* | CYGWIN*) echo windows ;;+ *) echo other ;;+ esac+}++tel_os_major() {+ case "$(uname -s 2>/dev/null)" in+ Darwin) v="$(sw_vers -productVersion 2>/dev/null)" ;;+ Linux) v="$(sed -n 's/^VERSION_ID=//p' /etc/os-release 2>/dev/null | tr -d '"')" ;;+ *) v="" ;;+ esac+ v="${v%%[!0-9]*}"+ v="$(printf '%s' "$v" | sed 's/^0*//')"+ case "$v" in "" | ????*) echo 0 ;; *) echo "$v" ;; esac+}++tel_arch() {+ case "$(uname -m 2>/dev/null)" in+ arm64 | aarch64) echo arm64 ;;+ x86_64 | amd64) echo x86_64 ;;+ *) echo other ;;+ esac+}++tel_items() {+ items=""+ [ "$want_cli" = 0 ] || items="cli"+ [ "$sel_spaces" = 0 ] || items="${items:+$items+}spaces"+ [ "$sel_driver" = 0 ] || items="${items:+$items+}cua_driver"+ [ "$sel_host" = 0 ] || items="${items:+$items+}host"+ echo "${items:-none}"+}++# tel_send STEP [STAGE KIND]: one cua_installer_step, at most 3 seconds.+tel_send() {+ [ "$tel_on" = 1 ] || return 0+ step_name="$1"+ stage=none+ kind=none+ outcome=ok+ case "$step_name" in+ installer_completed) tel_ended=1 ;;+ installer_failed)+ tel_ended=1+ stage="${2:-other}"+ kind="${3:-other}"+ outcome=error+ [ "$kind" != cancelled ] || outcome=cancelled+ ;;+ esac+ pv="${tel_version:-${version:-}}"+ case "$pv" in+ [0-9]*.[0-9]*.[0-9]*) ;;+ *) pv=0.0.0 ;;+ esac+ pv="${pv%%[!0-9A-Za-z.-]*}"+ synthetic=false+ [ "$(tel_bool "${CUA_TELEMETRY_SYNTHETIC:-}")" != on ] || synthetic=true+ u="$(tel_hex 16)"+ uuid="$(printf '%s' "$u" | sed 's/^\(.\{8\}\)\(.\{4\}\).\(.\{3\}\).\(.\{3\}\)\(.\{12\}\)$/\1-\2-4\3-8\4-\5/')"+ # The $-prefixed names are PostHog property names, not shell expansions.+ # shellcheck disable=SC2016+ body="$(printf '{"api_key":"%s","batch":[{"event":"cua_installer_step","distinct_id":"%s","uuid":"%s","timestamp":"%s","properties":{"telemetry_schema_version":1,"event_version":1,"product":"cli","product_version":"%s","os_family":"%s","os_major":%s,"arch":"%s","is_ci":%s,"is_synthetic":%s,"process_session_id":"%s","sample_rate":1,"$process_person_profile":false,"$geoip_disable":true,"$lib":"cua-installer","$lib_version":"1.0.0","step":"%s","outcome":"%s","install_stage":"%s","error_kind":"%s","installer":"sh","items":"%s"}}]}' \+ "$tel_key" "$tel_id" "$uuid" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$pv" "$(tel_os)" "$(tel_os_major)" "$(tel_arch)" \+ "$tel_ci" "$synthetic" "$tel_session" "$step_name" "$outcome" "$stage" "$kind" "$(tel_items)")"+ [ "$(tel_bool "${CUA_TELEMETRY_DEBUG:-}")" != on ] || printf '[cua telemetry] %s\n' "$body" >&2+ if has curl; then+ curl -s -o /dev/null --max-time 3 -H 'Content-Type: application/json' --data "$body" "$tel_endpoint" >/dev/null 2>&1 || true+ else+ wget -q -O /dev/null --timeout=3 --tries=1 --header='Content-Type: application/json' --post-data="$body" "$tel_endpoint" >/dev/null 2>&1 || true+ fi+}+# ---------------------------------------------------------------- argumentsparse_args() {@@ -641,6 +851,7 @@ load_manifest() {fetch "$manifest_url" "$manifest"grep -q '"schema" *: *1' "$manifest" || die "unsupported release manifest at $manifest_url"manifest_dir="${manifest_url%/*}"+ tel_version="$(sed -n 's/.*"version" *: *"\([0-9][0-9A-Za-z.-]*\)".*/\1/p' "$manifest" | head -n 1)"}# resolve_url VALUE: absolute URLs pass through; names are relative to the manifest.@@ -1176,9 +1387,12 @@ run_onboarding() {main() {parse_args "$@"+ telemetry_init+ tel_stage=platformdetect_platforminit_selectionreuse_installed+ tel_send installer_startedtmp="$(mktemp -d 2>/dev/null || mktemp -d -t cua-install)"if wants_checklist; thenchecklist@@ -1188,6 +1402,7 @@ main() {fi# Nothing to download (--pair on a computer that has Cua): no manifest to read.if [ "$want_cli" = 1 ] || [ "$sel_spaces" = 1 ] || [ "$sel_driver" = 1 ]; then+ tel_stage=manifestload_manifestinfo "platform $platform, manifest $manifest_url"fi@@ -1200,12 +1415,15 @@ main() {cua_bin="${existing_cua:-cua}"app_path=""if [ "$want_cli" = 1 ]; then+ tel_stage=cliinstall_clifi+ tel_stage=runtimeif [ "$os" = linux ] && [ "$sel_spaces" = 1 ] && { [ "$dry_run" = 1 ] || has "$cua_bin"; }; thenlinux_runtimefiif [ "$sel_spaces" = 1 ]; then+ tel_stage=appcase "$os" indarwin) install_app_darwin ;;linux) install_app_linux ;;@@ -1217,12 +1435,17 @@ main() {CUA_INSTALL_CHANNEL=install_scriptexport CUA_INSTALL_CHANNELif [ "$sel_driver" = 1 ]; then+ tel_stage=driverinstall_driverfi+ tel_stage=sign_inrun_onboardingif [ "$sel_host" = 1 ]; then+ tel_stage=hosthost_setupfi+ tel_stage=other+ tel_send installer_completedsummaryif [ -n "$path_hint" ]; thensay ""diff --git a/scripts/install/tests/fixtures/installer-step.json b/scripts/install/tests/fixtures/installer-step.jsonnew file mode 100644index 0000000000..0ac76678ca--- /dev/null+++ b/scripts/install/tests/fixtures/installer-step.json@@ -0,0 +1,86 @@+{+ "_comment": "What install.sh and install.ps1 send (cua_installer_step). libs/cua/crates/cua-telemetry/tests/privacy.rs validates these against the schema; scripts/install/tests/test_install_telemetry.py checks install.sh sends exactly these properties. Ids, times and host values vary per run.",+ "events": [+ {+ "event": "cua_installer_step",+ "distinct_id": "0123456789abcdef0123456789abcdef",+ "properties": {+ "telemetry_schema_version": 1,+ "event_version": 1,+ "product": "cli",+ "product_version": "0.0.0",+ "os_family": "macos",+ "os_major": 26,+ "arch": "arm64",+ "is_ci": false,+ "is_synthetic": false,+ "process_session_id": "fedcba9876543210fedcba9876543210",+ "sample_rate": 1,+ "$process_person_profile": false,+ "$geoip_disable": true,+ "$lib": "cua-installer",+ "$lib_version": "1.0.0",+ "step": "installer_started",+ "outcome": "ok",+ "install_stage": "none",+ "error_kind": "none",+ "installer": "sh",+ "items": "cli+spaces"+ }+ },+ {+ "event": "cua_installer_step",+ "distinct_id": "0123456789abcdef0123456789abcdef",+ "properties": {+ "telemetry_schema_version": 1,+ "event_version": 1,+ "product": "cli",+ "product_version": "1.2.3",+ "os_family": "linux",+ "os_major": 24,+ "arch": "x86_64",+ "is_ci": false,+ "is_synthetic": false,+ "process_session_id": "fedcba9876543210fedcba9876543210",+ "sample_rate": 1,+ "$process_person_profile": false,+ "$geoip_disable": true,+ "$lib": "cua-installer",+ "$lib_version": "1.0.0",+ "step": "installer_failed",+ "outcome": "error",+ "install_stage": "cli",+ "error_kind": "checksum",+ "installer": "sh",+ "items": "cli"+ }+ },+ {+ "event": "cua_installer_step",+ "distinct_id": "0123456789abcdef0123456789abcdef",+ "properties": {+ "telemetry_schema_version": 1,+ "event_version": 1,+ "product": "cli",+ "product_version": "1.2.3",+ "os_family": "windows",+ "os_major": 10,+ "arch": "x86_64",+ "is_ci": false,+ "is_synthetic": false,+ "process_session_id": "fedcba9876543210fedcba9876543210",+ "sample_rate": 1,+ "$process_person_profile": false,+ "$geoip_disable": true,+ "$lib": "cua-installer",+ "$lib_version": "1.0.0",+ "step": "installer_completed",+ "outcome": "ok",+ "install_stage": "none",+ "error_kind": "none",+ "installer": "ps1",+ "items": "cli+spaces+cua_driver"+ }+ }+ ]+}diff --git a/scripts/install/tests/install.Tests.ps1 b/scripts/install/tests/install.Tests.ps1index 0cb93fc47e..0566b4c3c4 100644--- a/scripts/install/tests/install.Tests.ps1+++ b/scripts/install/tests/install.Tests.ps1@@ -30,7 +30,8 @@ BeforeAll {function script:Invoke-Installer([hashtable]$Params, [hashtable]$Vars = @{}) {$saved = @{}- $all = @{ CUA_INSTALL_MANIFEST_URL = $env:CUA_TEST_MANIFEST_URL; CUA_INSTALL_ARCH = 'AMD64'; CUA_HOME = (Join-Path $env:CUA_TEST_ROOT 'cuahome') }+ # Install telemetry may only reach a loopback endpoint here.+ $all = @{ CUA_INSTALL_MANIFEST_URL = $env:CUA_TEST_MANIFEST_URL; CUA_INSTALL_ARCH = 'AMD64'; CUA_HOME = (Join-Path $env:CUA_TEST_ROOT 'cuahome'); CUA_TELEMETRY_FORBID_NETWORK = '1' }foreach ($k in $Vars.Keys) { $all[$k] = $Vars[$k] }foreach ($k in $all.Keys) { $saved[$k] = [Environment]::GetEnvironmentVariable($k); [Environment]::SetEnvironmentVariable($k, $all[$k]) }try {@@ -421,3 +422,98 @@ Describe 'install.ps1' {$out | Should -Match 'spaces-install-mode'}}++Describe 'install telemetry' {+ BeforeAll {+ $env:CUA_INSTALL_NO_RUN = '1'+ . $script:Installer+ Remove-Item Env:CUA_INSTALL_NO_RUN+ $script:Fixture = Get-Content -Raw (Join-Path $PSScriptRoot 'fixtures/installer-step.json') | ConvertFrom-Json+ $script:Switches = @('DO_NOT_TRACK', 'CUA_TELEMETRY', 'CUA_TELEMETRY_ENABLED', 'CUA_TELEMETRY_DISABLED') + $script:TelCiVars+ function script:With-Env([hashtable]$Vars, [scriptblock]$Body) {+ $saved = @{}+ foreach ($k in $script:Switches + @($Vars.Keys)) { $saved[$k] = [Environment]::GetEnvironmentVariable($k); [Environment]::SetEnvironmentVariable($k, $null) }+ foreach ($k in $Vars.Keys) { [Environment]::SetEnvironmentVariable($k, $Vars[$k]) }+ try { return (& $Body) } finally { foreach ($k in $saved.Keys) { [Environment]::SetEnvironmentVariable($k, $saved[$k]) } }+ }+ function script:Get-Events([string]$Out) {+ return @(($Out -split "`r?`n") | Where-Object { $_ -like '`[cua telemetry`] *' } | ForEach-Object { ($_.Substring(16) | ConvertFrom-Json).batch[0] })+ }+ }++ It 'follows the same switches as every Cua program' {+ $h = Join-Path $env:CUA_TEST_ROOT 'tel-home'+ New-Item -ItemType Directory -Force -Path $h | Out-Null+ With-Env @{} { Test-TelemetryEnabled $h } | Should -BeTrue+ With-Env @{ DO_NOT_TRACK = '1'; CUA_TELEMETRY = '1' } { Test-TelemetryEnabled $h } | Should -BeFalse+ With-Env @{ CUA_TELEMETRY = '0' } { Test-TelemetryEnabled $h } | Should -BeFalse+ With-Env @{ CUA_TELEMETRY_ENABLED = 'false' } { Test-TelemetryEnabled $h } | Should -BeFalse+ With-Env @{ CUA_TELEMETRY_DISABLED = '1' } { Test-TelemetryEnabled $h } | Should -BeFalse+ With-Env @{ GITHUB_ACTIONS = 'true' } { Test-TelemetryEnabled $h } | Should -BeFalse+ With-Env @{ CI = 'true'; CUA_TELEMETRY = '1' } { Test-TelemetryEnabled $h } | Should -BeTrue+ Set-Content -LiteralPath (Join-Path $h 'config.toml') -Value "[other]`nenabled = `"on`"`n[telemetry]`nenabled = `"off`""+ With-Env @{} { Test-TelemetryEnabled $h } | Should -BeFalse+ }++ It 'names a failure with a fixed word, never the message' {+ Get-CuaInstallErrorKind 'cua-install: error: checksum mismatch for cua.zip' | Should -Be 'checksum'+ Get-CuaInstallErrorKind 'cua-install: error: cosign signature check failed' | Should -Be 'signature'+ Get-CuaInstallErrorKind 'cua-install: error: unsupported CPU architecture: x86' | Should -Be 'unsupported_platform'+ Get-CuaInstallErrorKind 'Unable to connect to the remote server' | Should -Be 'network'+ Get-CuaInstallErrorKind 'C:\Users\alice\secret' | Should -Be 'other'+ }++ It 'sends nothing on a dry run, with DO_NOT_TRACK, or to a real endpoint in tests' {+ foreach ($vars in @(@{ DO_NOT_TRACK = '1' }, @{ CUA_TELEMETRY_ENDPOINT = 'https://eu.i.posthog.com/batch/' })) {+ $vars.CUA_TELEMETRY_DEBUG = '1'+ $vars.CUA_HOME = Join-Path $env:CUA_TEST_ROOT ('tel-off-' + [Guid]::NewGuid().ToString('N'))+ $out = Invoke-Installer @{ CliOnly = $true; Yes = $true; NoOnboarding = $true; NoModifyPath = $true; Prefix = (Join-Path $env:CUA_TEST_ROOT 'p-tel-off') } $vars+ $out | Should -Not -Match 'cua telemetry'+ $out | Should -Not -Match 'anonymous data about this install'+ Test-Path (Join-Path $vars.CUA_HOME 'telemetry/install_id') | Should -BeFalse+ }+ $out = Invoke-Installer @{ DryRun = $true; CliOnly = $true } @{ CUA_TELEMETRY_DEBUG = '1'; CUA_TELEMETRY_ENDPOINT = 'http://127.0.0.1:9/batch/' }+ $out | Should -Not -Match 'cua telemetry'+ }++ It 'sends started then completed with the fixture properties and the stored install id' {+ $cuaHome = Join-Path $env:CUA_TEST_ROOT 'tel-on'+ $vars = @{ CUA_TELEMETRY_DEBUG = '1'; CUA_TELEMETRY_ENDPOINT = 'http://127.0.0.1:9/batch/'; CUA_HOME = $cuaHome; CUA_POSTHOG_KEY = 'phc_test' }+ $out = Invoke-Installer @{ CliOnly = $true; Yes = $true; NoOnboarding = $true; NoModifyPath = $true; Prefix = (Join-Path $env:CUA_TEST_ROOT 'p-tel') } $vars+ $out | Should -Match 'anonymous data about this install'+ $events = Get-Events $out+ @($events | ForEach-Object { $_.properties.step }) | Should -Be @('installer_started', 'installer_completed')+ $id = (Get-Content -Raw (Join-Path $cuaHome 'telemetry/install_id')).Trim()+ $ref = $script:Fixture.events[2].properties+ foreach ($e in $events) {+ $e.distinct_id | Should -Be $id+ $e.event | Should -Be 'cua_installer_step'+ @($e.properties.PSObject.Properties.Name | Sort-Object) | Should -Be @($ref.PSObject.Properties.Name | Sort-Object)+ $e.properties.installer | Should -Be 'ps1'+ $e.properties.items | Should -Be 'cli'+ $e.properties.'$lib' | Should -Be 'cua-installer'+ }+ $events[1].properties.product_version | Should -Be '1.2.3'+ $out | Should -Not -Match ([regex]::Escape('"api_key":"phc_eSk'))+ }++ It 'sends installer_failed with the stage and kind when the install stops' {+ $vars = @{ CUA_INSTALL_MANIFEST_URL = (ConvertTo-FileUrl (Join-Path $env:CUA_TEST_ROOT 'missing/release-artifacts.json'))+ CUA_INSTALL_ARCH = 'AMD64'; CUA_HOME = (Join-Path $env:CUA_TEST_ROOT 'tel-fail'); CUA_TELEMETRY_FORBID_NETWORK = '1'+ CUA_TELEMETRY_DEBUG = '1'; CUA_TELEMETRY_ENDPOINT = 'http://127.0.0.1:9/batch/'+ }+ $out = With-Env $vars {+ $params = @{ CliOnly = $true; Yes = $true; NoOnboarding = $true; NoModifyPath = $true; Prefix = (Join-Path $env:CUA_TEST_ROOT 'p-tel-fail') }+ $lines = & { try { & $env:CUA_TEST_INSTALLER @params } catch { "threw: $_" } } *>&1+ ($lines | ForEach-Object { "$_" }) -join "`n"+ }+ $out | Should -Match 'threw: '+ $events = Get-Events $out+ @($events | ForEach-Object { $_.properties.step }) | Should -Be @('installer_started', 'installer_failed')+ $failed = $events[1].properties+ $failed.install_stage | Should -Be 'manifest'+ $failed.outcome | Should -Be 'error'+ $failed.error_kind | Should -BeIn @('network', 'other')+ ($events | ConvertTo-Json -Depth 6) | Should -Not -Match 'missing'+ }+}diff --git a/scripts/install/tests/run.sh b/scripts/install/tests/run.shindex 4062230fdf..524fc08664 100755--- a/scripts/install/tests/run.sh+++ b/scripts/install/tests/run.sh@@ -158,7 +158,8 @@ inst() {done[ "${1:-}" = -- ] && shift# shellcheck disable=SC2086- env -i PATH="$PATH" HOME="$home" SHELL=/bin/sh CUA_HOME="$home/.cua" \+ # Install telemetry may only reach a loopback endpoint here (none is set).+ env -i PATH="$PATH" HOME="$home" SHELL=/bin/sh CUA_HOME="$home/.cua" CUA_TELEMETRY_FORBID_NETWORK=1 \CUA_INSTALL_BASE_URL="$base" CUA_INSTALL_OS=linux CUA_INSTALL_ARCH=x86_64 \CUA_FAKE_LOG="$home/cua.log" $envs \"$sh_bin" "$installer" "$@" </dev/null >"$work/out" 2>&1diff --git a/scripts/install/tests/test_install_telemetry.py b/scripts/install/tests/test_install_telemetry.pynew file mode 100644index 0000000000..867e147b0e--- /dev/null+++ b/scripts/install/tests/test_install_telemetry.py@@ -0,0 +1,217 @@+"""Install telemetry in install.sh (python3 -m unittest discover scripts/install/tests).++Hermetic: a throwaway HOME and a loopback server that serves a fake release+and records every telemetry POST. CUA_TELEMETRY_FORBID_NETWORK=1 is set on+every run, so nothing can reach a real endpoint. The payload must carry+exactly the properties of fixtures/installer-step.json, which the Rust+privacy tests validate against the telemetry schema.+"""++import http.server+import json+import os+import subprocess+import tarfile+import tempfile+import threading+import unittest+from pathlib import Path++HERE = Path(__file__).resolve().parent+SCRIPT = HERE.parent / "install.sh"+MANIFEST_TOOL = HERE.parent / "release_manifest.py"+FIXTURE = json.loads((HERE / "fixtures" / "installer-step.json").read_text())+VERSION = "1.2.3"+# Values that vary per run or per host; everything else must match the fixture.+VARYING = {"product_version", "os_family", "os_major", "arch", "process_session_id",+ "step", "outcome", "install_stage", "error_kind", "items", "is_ci"}+++class Recorder(http.server.SimpleHTTPRequestHandler):+ posts = []++ def log_message(self, *args):+ pass++ def do_POST(self):+ n = int(self.headers.get("Content-Length", "0"))+ Recorder.posts.append(json.loads(self.rfile.read(n)))+ self.send_response(200)+ self.end_headers()+ self.wfile.write(b"{}")+++class InstallTelemetry(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ cls.tmp = tempfile.TemporaryDirectory()+ root = Path(cls.tmp.name)+ cls.release = root / "release"+ tag = cls.release / f"cua-sdk-v{VERSION}"+ tag.mkdir(parents=True)+ stage = root / "stage"+ stage.mkdir()+ cua = stage / "cua"+ cua.write_text(f'#!/bin/sh\n[ "$1" = --version ] && echo "cua {VERSION}"\nexit 0\n')+ cua.chmod(0o755)+ with tarfile.open(tag / f"cua-cli-{VERSION}-linux-x64.tar.gz", "w:gz") as t:+ t.add(cua, arcname="cua")+ handler = lambda *a, **k: Recorder(*a, directory=str(cls.release), **k) # noqa: E731+ cls.server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)+ threading.Thread(target=cls.server.serve_forever, daemon=True).start()+ cls.base = f"http://127.0.0.1:{cls.server.server_address[1]}"+ subprocess.run(+ ["python3", str(MANIFEST_TOOL), "--component", "cli", "--version", VERSION,+ "--dir", str(tag), "--base-url", f"{cls.base}/cua-sdk-v{VERSION}",+ "--out", str(tag / "release-artifacts.json")],+ check=True,+ )+ cls.manifest = f"{cls.base}/cua-sdk-v{VERSION}/release-artifacts.json"++ @classmethod+ def tearDownClass(cls):+ cls.server.shutdown()+ cls.tmp.cleanup()++ def setUp(self):+ Recorder.posts.clear()+ self.home = Path(tempfile.mkdtemp(dir=self.tmp.name))++ def run_installer(self, *args, env=None, manifest=None):+ e = {+ "PATH": os.environ["PATH"],+ "HOME": str(self.home),+ "CUA_HOME": str(self.home / ".cua"),+ "CUA_INSTALL_OS": "linux",+ "CUA_INSTALL_ARCH": "x86_64",+ "CUA_INSTALL_MANIFEST_URL": manifest or self.manifest,+ "CUA_TELEMETRY_FORBID_NETWORK": "1",+ "CUA_TELEMETRY_ENDPOINT": f"{self.base}/batch/",+ "CUA_POSTHOG_KEY": "phc_test",+ }+ e.update(env or {})+ p = subprocess.run(+ ["sh", str(SCRIPT), "--yes", "--no-onboarding", "--prefix", str(self.home / "p"), *args],+ env=e, stdin=subprocess.DEVNULL, capture_output=True, text=True, timeout=120,+ )+ return p++ def events(self):+ out = []+ for body in Recorder.posts:+ self.assertEqual(body["api_key"], "phc_test")+ out.extend(body["batch"])+ return out++ def assert_matches_fixture(self, ev):+ ref = FIXTURE["events"][0]+ self.assertEqual(ev["event"], ref["event"])+ self.assertEqual(set(ev["properties"]), set(ref["properties"]))+ for k, v in ref["properties"].items():+ if k not in VARYING:+ self.assertEqual(ev["properties"][k], v, k)+ self.assertRegex(ev["distinct_id"], r"^[0-9a-f]{32}$")+ self.assertRegex(ev["properties"]["process_session_id"], r"^[0-9a-f]{32}$")+ self.assertRegex(ev["uuid"], r"^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$")+ self.assertRegex(ev["timestamp"], r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$")++ def test_a_finished_install_sends_started_then_completed(self):+ p = self.run_installer("--cli-only")+ self.assertEqual(p.returncode, 0, p.stdout + p.stderr)+ evs = self.events()+ self.assertEqual([e["properties"]["step"] for e in evs], ["installer_started", "installer_completed"])+ for e in evs:+ self.assert_matches_fixture(e)+ self.assertEqual(e["properties"]["outcome"], "ok")+ self.assertEqual(e["properties"]["install_stage"], "none")+ self.assertEqual(e["properties"]["error_kind"], "none")+ self.assertEqual(e["properties"]["items"], "cli")+ self.assertEqual(evs[1]["properties"]["product_version"], VERSION)+ # The same anonymous id the Cua programs use afterwards, owner-only.+ id_file = self.home / ".cua" / "telemetry" / "install_id"+ self.assertEqual(id_file.read_text().strip(), evs[0]["distinct_id"])+ self.assertEqual(id_file.stat().st_mode & 0o077, 0)+ self.assertEqual(evs[0]["distinct_id"], evs[1]["distinct_id"])+ self.assertIn("anonymous data about this install", p.stdout)+ self.assertIn("DO_NOT_TRACK=1", p.stdout)+ # Nothing from the machine: no home path, no user name.+ raw = json.dumps(Recorder.posts)+ self.assertNotIn(str(self.home), raw)+ self.assertNotIn(os.environ.get("USER", "nobody-here"), raw)++ def test_an_existing_install_id_is_kept(self):+ d = self.home / ".cua" / "telemetry"+ d.mkdir(parents=True)+ (d / "install_id").write_text("0123456789abcdef0123456789abcdef")+ self.run_installer("--cli-only")+ self.assertTrue(self.events())+ for e in self.events():+ self.assertEqual(e["distinct_id"], "0123456789abcdef0123456789abcdef")++ def test_a_failed_install_says_where_and_why_never_the_message(self):+ p = self.run_installer("--cli-only", manifest=f"{self.base}/no-such-release/release-artifacts.json")+ self.assertNotEqual(p.returncode, 0)+ evs = self.events()+ self.assertEqual([e["properties"]["step"] for e in evs], ["installer_started", "installer_failed"])+ failed = evs[1]["properties"]+ self.assertEqual((failed["outcome"], failed["install_stage"], failed["error_kind"]),+ ("error", "manifest", "network"))+ self.assert_matches_fixture(evs[1])+ self.assertNotIn("no-such-release", json.dumps(Recorder.posts))++ def test_every_switch_turns_it_off(self):+ cfg = self.home / ".cua"+ cases = [+ {"DO_NOT_TRACK": "1"},+ {"DO_NOT_TRACK": "1", "CUA_TELEMETRY": "1"},+ {"CUA_TELEMETRY": "0"},+ {"CUA_TELEMETRY_ENABLED": "false"},+ {"CUA_TELEMETRY_DISABLED": "1"},+ {"CI": "true"},+ {"GITHUB_ACTIONS": "true"},+ "config",+ "dry-run",+ # The test guard: a real endpoint is refused before anything happens.+ {"CUA_TELEMETRY_ENDPOINT": "https://eu.i.posthog.com/batch/"},+ ]+ for case in cases:+ with self.subTest(case=case):+ Recorder.posts.clear()+ self.home = Path(tempfile.mkdtemp(dir=self.tmp.name))+ cfg = self.home / ".cua"+ args, env = ["--cli-only"], {}+ if case == "config":+ cfg.mkdir(parents=True)+ (cfg / "config.toml").write_text('[other]\nenabled = "on"\n\n[telemetry]\nenabled = "off"\n')+ elif case == "dry-run":+ args.append("--dry-run")+ else:+ env = case+ p = self.run_installer(*args, env=env)+ self.assertEqual(Recorder.posts, [], p.stdout + p.stderr)+ self.assertNotIn("anonymous data about this install", p.stdout)+ self.assertFalse((cfg / "telemetry" / "install_id").exists())++ def test_ci_with_cua_telemetry_on_sends_and_is_marked(self):+ p = self.run_installer("--cli-only", env={"CI": "true", "CUA_TELEMETRY": "1"})+ self.assertEqual(p.returncode, 0, p.stdout + p.stderr)+ evs = self.events()+ self.assertEqual(len(evs), 2)+ self.assertTrue(all(e["properties"]["is_ci"] is True for e in evs))++ def test_staging_sends_to_the_staging_project(self):+ self.run_installer("--cli-only", env={"CUA_ENV": "staging", "CUA_POSTHOG_KEY": ""})+ self.assertEqual(len(Recorder.posts), 2)+ for body in Recorder.posts:+ self.assertTrue(body["api_key"].startswith("phc_xDNL"), "staging project key")++ def test_a_config_file_on_wins_over_ci(self):+ cfg = self.home / ".cua"+ cfg.mkdir(parents=True)+ (cfg / "config.toml").write_text('[telemetry]\nenabled = true\n')+ self.run_installer("--cli-only", env={"CI": "1"})+ self.assertEqual(len(self.events()), 2)+++if __name__ == "__main__":+ unittest.main()