Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
test(spaces): add a Linux verification skill that drives a Space like a user (#6398)
injaneity194f3f6ce088
diff --git a/.cursor/skills/verify-cua-spaces/SKILL.md b/.cursor/skills/verify-cua-spaces/SKILL.mdnew file mode 100644index 0000000000..2bb97d5138--- /dev/null+++ b/.cursor/skills/verify-cua-spaces/SKILL.md@@ -0,0 +1,84 @@+---+name: verify-cua-spaces+description: Verify Cua Spaces on Linux the way a user drives a Space - register a running cua-spacesd with the source-built `cua`, open it in the HTML5 viewer, act on it with `cua do`, `cua spacesd` and `cua mcp` - with retained evidence. Use after changes to libs/cua-spacesd, the Spaces CLI/SDK (libs/cua) or the HTML5 viewer.+---++# Verify Cua Spaces++A **Space** is any machine running **cua-spacesd** (port 3211 in an image; a random loopback port here). Users reach it through the `cua` CLI (`cua spaces`, `cua sb view`, `cua do`, `cua spacesd`, `cua mcp`), the HTML5 viewer that cua-spacesd serves at `/viewer/`, and the desktop apps (Tauri, Electron, SwiftUI) that sit on the same SDK. This skill drives the CLI and the viewer against a throwaway Linux desktop. Read [the feature map](features/README.md) before picking a journey.++Everything here is source-built: `libs/cua-spacesd` (the daemon and the `cua-spacesd-x11-pad` input-echo fixture) and `libs/cua` (`cua-spaces-cli`, the Spaces build of `cua`; the apps ship it as `cua`). Never drive a cua-spacesd, cua daemon or `~/.cua` you did not start: every run gets its own display, session bus, audio server, port, token and cua home.++## Launch++Run from the repository root. Linux only. Needs Rust (pinned by the workspaces), `protoc`, Node 20+, Chromium, `Xvfb`, `openbox`, `dbus-daemon`, `curl`, `ss`, `python3`, and for audio `pipewire`, `wireplumber`, `pipewire-pulse` (Debian/Ubuntu: the build packages in `.github/workflows/ci-cua-spacesd.yml` plus `xvfb openbox dbus chromium pipewire wireplumber pipewire-pulse pulseaudio-utils`).++```bash+H=.cursor/skills/verify-cua-spaces/scripts/spaces-desk.sh+$H build # once: release cua-spacesd, cua-spacesd-x11-pad, cua-spaces-cli (~20 min cold, ~9 GB of target/)+$H up # or: $H up --no-audio+```++`up` starts, in this order and each in its own session: Xvfb on a free display (1280x800), a private D-Bus session bus, PipeWire + WirePlumber + pipewire-pulse (skipped with `--no-audio` or when not installed), openbox, the x11-pad fixture filling the screen with a red/green/blue/white grid, and cua-spacesd on a free loopback port with a fresh token and data dir (`--quic-port 0`). It then registers the Space with `cua spaces add 127.0.0.1:<port> --name verify` in an isolated `CUA_HOME`.++Ready means usable, not just listening: `up` waits for `GET /health` = 204 **and** `SystemService/Health` = `HEALTH_STATUS_SERVING` with every component (`desktop` is not serving until a window manager runs), then for the Space registration and the pad's `ready` line. It prints:++```+RUN=/tmp/cua-spaces-verify.XXXXXX+EVIDENCE=<repo>/artifacts/cua-spaces/verify/run-XXXXXXXX+READY space=direct:127.0.0.1:<port> display=:<n> audio=pipewire|none+```++Keep `RUN`; every other command takes it. Teardown is `$H down "$RUN"` (see Cleanup).++## Doctor++```bash+$H doctor "$RUN"+```++Read-only. Checks that every process `up` recorded is alive, that the port is owned by our cua-spacesd pid, prints the daemon version and `cua spacesd caps`, confirms the Space is registered in our cua home, then runs `cua-spacesd doctor --target <url>` and keeps `doctor.txt`/`doctor.json`/`doctor/screenshot.display.png` in the evidence dir. Exit 0 means worth driving. Expected warnings on this stand-in: `auth.token_file` (no `/run/cua/env-token` outside an image), `stream.quic` (QUIC is off), and `driver.health_report` AT-SPI unreachable. Any `[fail]`, a dead pid or a foreign port owner means stop and `down`.++## Drive++Three harnesses, all against the same Space:++- **CLI:** `$H cua "$RUN" -- <args>` runs the source-built `cua <args>` with the run's isolated `HOME`/`CUA_HOME`/`DISPLAY`, and writes the command, stdout, stderr and exit code to `cli-NNN.txt` in the evidence dir. The Space is `direct:127.0.0.1:<port>` (from `$RUN/state.env`, `SPACE_ID`) and is named `verify`.+- **Viewer:** `$H viewer "$RUN" <name> [extra-fragment]` gets a link from `cua sb view <SPACE_ID> --no-open` (a real viewer ticket), opens it in headless Chromium through `scripts/drive-viewer.mjs`, waits for `#app[data-status=streaming]` and a decoded frame, clicks the white (bottom-right) grid cell and types `ok`, then requires the pad journal to show that click and those keys. Toolbar buttons have stable `aria-label`s (`Send keys`, `Upload files`, `Share a folder`, `Sound on`, `Microphone off`, `Stats`, ...); `window.__cuaViewer.stats()` reports frames and audio packets.+- **MCP:** pipe JSON-RPC lines to `cua mcp --embedded` with the run's environment (recipe in [features/mcp.md](features/mcp.md)).++The x11-pad journal (`pad.jsonl`) is the independent side-effect oracle for input: one JSON line per `button`/`key`/`focus_in` the X server delivered to the fixture window.++## Evidence++Everything lands in the printed `EVIDENCE` dir under `artifacts/` (gitignored): `state.env`, `pids.txt`, each process log (`cua-spacesd.log`, `xvfb.log`, ...), `spaces-add.log`, `doctor.*`, `cli-NNN.txt` transcripts, `viewer-<name>.json` (steps, stats, redacted link), `viewer-<name>.png`, `viewer-<name>.aria.txt`, and `pad.jsonl`.++Proof standards:++- Drive the real user path: `cua` commands, the viewer page a `cua sb view` link opens, MCP over stdio. Not `cargo test`, not direct gRPC calls when a CLI verb exists, not `window.__cuaViewer.input()` (that hook exists for tests and skips the canvas).+- Capture the action and the resulting state: the CLI transcript plus a second read (`cua spacesd cp` back, `cua do screenshot`, `pad.jsonl`), the viewer screenshot plus the pad journal.+- Side effects count more than messages: a file read back with the same sha256, a pad event at the expected coordinates, a registry entry in `cua spaces ls --json`.+- Viewer tickets are credentials: `drive-viewer.mjs` redacts them; do not paste links into reports.++## Cleanup++```bash+$H down "$RUN"+```++Kills only the pids `up` recorded (newest first, TERM then KILL) and deletes `RUN` (cua home, token, data dir, Chromium's playwright-core). It never kills by name and never touches another display. The evidence dir survives; `down` prints its path. Run `down` after every failed attempt too. To reclaim disk afterwards, delete `libs/cua-spacesd/target` and `libs/cua/target` yourself.++## Helpers++- `scripts/spaces-desk.sh` (`build | up [--no-audio] | doctor RUN | cua RUN -- ARGS | viewer RUN NAME [FRAGMENT] | index RUN | down RUN`; no arguments prints usage).+- `scripts/drive-viewer.mjs LINK OUT PAD_LOG` (called by `viewer`; needs `PW_DIR` with `playwright-core` and `CHROMIUM`; `viewer` installs `[email protected]` into `RUN` the first time).++## Unverified here++- The desktop apps: Tauri (`apps/cua-spaces`), Electron (`apps/cua-spaces-desktop`) and SwiftUI (`apps/cua-spaces-macos`, `libs/spaces-*-swift`), the notch panel, menu bar item and first-run flow. None was launched; the web UI's demo mode (`apps/cua-spaces-web`, `pnpm dev`) is fixtures only.+- macOS and Windows cua-spacesd (ScreenCaptureKit/TCC grants, WASAPI, scheduled task). Linux X11 only; Wayland/Hyprland capture is not exercised.+- Creating Spaces (`cua spaces create`, `create_space`): needs a container runtime or VM image pulls; this skill attaches to a cua-spacesd it started instead.+- Sign-in, the relay (`cua-spacesd join`, `relay:<id>` Spaces), sharing and account features: need a Cua account and the hosted relay.+- QUIC media, hardware encoders, GPU: QUIC is disabled here; no GPU.++Use `/maintain-verification-skill` to keep the map honest as the CLI verbs, viewer handles or launch steps change.diff --git a/.cursor/skills/verify-cua-spaces/features/README.md b/.cursor/skills/verify-cua-spaces/features/README.mdnew file mode 100644index 0000000000..cc0d87d722--- /dev/null+++ b/.cursor/skills/verify-cua-spaces/features/README.md@@ -0,0 +1,40 @@+# Cua Spaces verification map++This directory is the maintained source for verifying what a user does with a Cua Space from Linux: register it, see it, act on it, move files and commands through it, and hand it to an agent. Read the index before driving, then use the matching feature file as the recipe.++## Baseline preconditions++- `scripts/spaces-desk.sh build` succeeded on this checkout.+- `scripts/spaces-desk.sh up` printed `READY space=direct:127.0.0.1:<port> ... audio=pipewire`, and you kept its `RUN=` path.+- `scripts/spaces-desk.sh doctor "$RUN"` exits 0.+- The Space is registered as `verify` (id `direct:127.0.0.1:<port>`, `SPACE_ID` in `$RUN/state.env`) in the run's own `CUA_HOME`.+- The x11-pad fixture fills the 1280x800 display with a red (top left), green (top right), blue (bottom left) and white (bottom right) grid, under a 19 px openbox title bar, and journals every input event to `pad.jsonl` in the evidence dir.+- Never drive a Space, cua daemon or cua home this run did not create.++## Driving conventions++- In the recipes, `H=.cursor/skills/verify-cua-spaces/scripts/spaces-desk.sh` and `. "$RUN/state.env"` has been sourced (for `SPACE_ID`, `URL`, `EVIDENCE`).+- Run CLI steps as `$H cua "$RUN" -- <args>`; quote arguments as shown.+- Run viewer steps as `$H viewer "$RUN" <capture-name> [fragment]`.+- `cua spacesd ...` verbs need the Space token: pass `--token "$(cat "$RUN/token")"`. The registered Space (`cua do`, `cua sb view`, `cua mcp`) does not; its token is in the cua home.+- Restore what you mutate in the guest (`rm` the files you wrote) but never delete evidence.++## Proof and skip reporting++- CLI proof is the `cli-NNN.txt` transcript (command, stdout, stderr, exit code) plus a second, independent read of the result.+- Viewer proof is `viewer-<name>.json`, `.png` and `.aria.txt`, plus the pad journal lines the action produced.+- Input proof is a new `pad.jsonl` line with the expected event, coordinates and key.+- Record the feature file and sub-feature id with each artifact you cite.+- An entry point you could not reach is reported with the attempted command and the unmet precondition. Do not report it verified through a different path.++## Feature entry contract++Each feature file starts with an H1 title and one paragraph describing the user-visible behavior, followed by exactly four H2 sections in this order: `Sub-features`, `How to get to it (user POV)`, `Driving it with spaces-desk`, `Gotchas`. The driving section starts with `Preconditions:` and pairs each user action with an exact command and an observable result.++## Features++- [Register and list Spaces](./register-and-list.md): `cua spaces add`, `cua spaces ls`, `cua sb ls`, `cua spaces rm`.+- [Watch and control in the viewer](./viewer.md): `cua sb view` and `cua viewer` links, the HTML5 viewer stream, mouse and keyboard, audio.+- [Act on a Space with cua do](./cua-do.md): select the Space, screenshot, click, type, key, shell.+- [Files and commands](./files-and-exec.md): `cua spacesd caps|exec|cp|targets|call`.+- [Agents over MCP](./mcp.md): `cua mcp` `list_spaces`, `space_write`, `space_bash`.diff --git a/.cursor/skills/verify-cua-spaces/features/cua-do.md b/.cursor/skills/verify-cua-spaces/features/cua-do.mdnew file mode 100644index 0000000000..1b2d761d14--- /dev/null+++ b/.cursor/skills/verify-cua-spaces/features/cua-do.md@@ -0,0 +1,35 @@+# Act on a Space with cua do++A user (or a script) points `cua do` at a Space once, then takes screenshots and clicks, types and runs commands in it one shot at a time.++## Sub-features++- `do-switch` selects the Space as the `cua do` target and `do-status` shows it.+- `do-screenshot` saves a PNG of the Space's screen.+- `do-click` clicks at screenshot coordinates, mapped onto the real display.+- `do-type` and `do-key` type text and press named keys.+- `do-shell` runs a shell command in the Space.++## How to get to it (user POV)++- `cua do switch <space id>`, `cua do status`.+- `cua do screenshot --save <file>`, `cua do click <x> <y>`, `cua do type <text>`, `cua do key <name>`, `cua do shell <command>`.++## Driving it with spaces-desk++Preconditions:++- Baseline. Note the line count of `$EVIDENCE/pad.jsonl` before the input steps.++- **Select.** Run `$H cua "$RUN" -- do switch "$SPACE_ID"`, then `$H cua "$RUN" -- do status`. Both exit 0; status prints `Current target: sandbox/<SPACE_ID>`.+- **Screenshot.** Run `$H cua "$RUN" -- do screenshot --save "$EVIDENCE/do-shot.png"`. Exit 0; the PNG (1200x750, scaled from 1280x800) shows the openbox title `verify-pad` and the four colour cells.+- **Click.** Run `$H cua "$RUN" -- do click 900 600` (the white cell in screenshot coordinates). A new `pad.jsonl` line `{"event":"button","button":1,"x":~960,"y":~621}` (pad coordinates = screen minus the 19 px title bar).+- **Type and key.** Run `$H cua "$RUN" -- do type "hi"` and `$H cua "$RUN" -- do key enter`. New `key` lines with `char` `h`, `i`, then keysym `65293` (Return).+- **Shell.** Run `$H cua "$RUN" -- do shell "echo do-shell-ok > /tmp/cua-verify-do.txt; cat /tmp/cua-verify-do.txt"`. Stdout contains `do-shell-ok`; `cat /tmp/cua-verify-do.txt` on this host (the Space is this machine) shows the same. Remove the file.++## Gotchas++- `cua do switch verify` (the display name) fails with `not found: sandbox verify` on a Space only `cua spaces add` registered; use the id. See register-and-list `sb-ls`.+- Screenshot coordinates are in the scaled image (1200x750 here), not display pixels; `cua do` maps them.+- `cua do` records a trajectory per command under the cua home (`--no-record` turns it off); `down` deletes it with `RUN`.+- The Space is this host: `do shell` runs as you, on your filesystem. Keep writes under `/tmp/cua-verify-*`.diff --git a/.cursor/skills/verify-cua-spaces/features/files-and-exec.md b/.cursor/skills/verify-cua-spaces/features/files-and-exec.mdnew file mode 100644index 0000000000..218adcb3a4--- /dev/null+++ b/.cursor/skills/verify-cua-spaces/features/files-and-exec.md@@ -0,0 +1,34 @@+# Files and commands++A user talks to a Space's cua-spacesd directly by URL: reads what it can do, runs a command, copies files both ways, lists what can be streamed, and calls any RPC.++## Sub-features++- `caps` prints the version, OS, protocol and each feature with its limitation.+- `exec` runs an argv (no shell) and returns its output and exit code.+- `cp-up` and `cp-down` copy a file into and out of the Space, sha256-checked.+- `targets` lists displays and, with `--windows`, windows.+- `call` calls any `cua.env.v1` RPC with a proto3-JSON body.++## How to get to it (user POV)++- `cua spacesd caps|exec|cp|shell|targets|call <url> ... --token <token>` (or `CUA_ENV_TOKEN`).++## Driving it with spaces-desk++Preconditions:++- Baseline. `T="$(cat "$RUN/token")"`.++- **Capabilities.** Run `$H cua "$RUN" -- spacesd caps "$URL" --token "$T"`. Exit 0; first line `cua-spacesd <version> (linux ... protocol 1.13)`; `+ desktop_stream` and `+ audio.desktop` present.+- **Exec.** Run `$H cua "$RUN" -- spacesd exec "$URL" --token "$T" sh -c 'echo exec-ok; exit 3'`. Stdout `exec-ok`, exit code `3`.+- **Upload.** `echo hello-spaces > "$RUN/up.txt"`, then `$H cua "$RUN" -- spacesd cp "$URL" "$RUN/up.txt" :/tmp/cua-verify-up.txt --token "$T"`. Prints `13 bytes, sha256 eb19...ed94f`.+- **Download.** Run `$H cua "$RUN" -- spacesd cp "$URL" :/tmp/cua-verify-up.txt "$RUN/down.txt" --token "$T"`. Same size and sha256; `cmp "$RUN/up.txt" "$RUN/down.txt"` exits 0. Remove `/tmp/cua-verify-up.txt`.+- **Targets.** Run `$H cua "$RUN" -- spacesd targets "$URL" --windows --token "$T"`. Lists `display 0 primary 1280x800` and a `window ... "verify-pad"` line.+- **Call.** Run `$H cua "$RUN" -- spacesd call "$URL" SystemService/Health --token "$T"`. JSON with `HEALTH_STATUS_SERVING` for `process`, `filesystem` and `desktop`.++## Gotchas++- Without `--token` (or `CUA_ENV_TOKEN`) every verb but `/health` is refused with `UNAUTHENTICATED`; the wrapper unsets `CUA_ENV_TOKEN` on purpose.+- `exec` takes an argv: use `sh -c '...'` for pipes and redirects.+- Guest paths start with `:`; a path without it is local.diff --git a/.cursor/skills/verify-cua-spaces/features/mcp.md b/.cursor/skills/verify-cua-spaces/features/mcp.mdnew file mode 100644index 0000000000..d82cc30164--- /dev/null+++ b/.cursor/skills/verify-cua-spaces/features/mcp.md@@ -0,0 +1,42 @@+# Agents over MCP++A coding agent (Claude Code, Codex, ...) launches `cua mcp` and uses the Spaces tools to find a Space, write files into it and run commands there.++## Sub-features++- `mcp-init` answers `initialize` as server `cua` and lists the Spaces tools.+- `mcp-list` returns the registered Space from `list_spaces`.+- `mcp-write` writes a file into the Space and verifies its sha256 there.+- `mcp-bash` runs a shell command in the Space and returns output and exit code.++## How to get to it (user POV)++- The agent's MCP config runs `cua mcp` (what `cua agents setup` registers) or `cua daemon mcp`.++## Driving it with spaces-desk++Preconditions:++- Baseline. The run's environment for the client: `. "$RUN/state.env"; export HOME="$RUN/home" CUA_HOME="$RUN/home/.cua" DO_NOT_TRACK=1; C=libs/cua/target/release/cua-spaces-cli`.++- **Initialize and call.** Run:++ ```bash+ printf '%s\n' \+ '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"verify","version":"0"}}}' \+ '{"jsonrpc":"2.0","method":"notifications/initialized"}' \+ '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \+ '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"list_spaces","arguments":{}}}' \+ '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"space_write","arguments":{"space":"verify","path":"/tmp/cua-verify-mcp.txt","content":"from-mcp\n"}}}' \+ '{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"space_bash","arguments":{"space":"verify","command":"cat /tmp/cua-verify-mcp.txt; echo $((6*7))"}}}' \+ | timeout 60 "$C" mcp --embedded >"$EVIDENCE/mcp.jsonl" 2>"$EVIDENCE/mcp.err"+ ```++ Exit 0. In `mcp.jsonl`: id 1 `serverInfo.name` `cua`; id 2 lists `list_spaces`, `space_write`, `space_bash`; id 3's text contains `$SPACE_ID`; id 4 says `wrote /tmp/cua-verify-mcp.txt (9 bytes, sha256 ... verified in Space)`; id 5's text is `from-mcp\n42\n[exit 0]`.+- **Side effect.** `cat /tmp/cua-verify-mcp.txt` on this host prints `from-mcp`. Remove it.++## Gotchas++- `--embedded` keeps the MCP server off any cua daemon; without it `cua mcp` starts or reuses one under the cua home.+- MCP tools take the Space by display name or id; the display name works here even where `cua sb` does not.+- Tools that show a Space on screen (`open_space_viewer`, `show_space_pip`, `stream_space_window`) call back into a running Cua Spaces app and fail without one; they are unverified here.diff --git a/.cursor/skills/verify-cua-spaces/features/register-and-list.md b/.cursor/skills/verify-cua-spaces/features/register-and-list.mdnew file mode 100644index 0000000000..8753f23817--- /dev/null+++ b/.cursor/skills/verify-cua-spaces/features/register-and-list.md@@ -0,0 +1,35 @@+# Register and list Spaces++A user adds a machine that already runs cua-spacesd as a Space, sees it in every list the CLI offers, and removes it again without touching the machine.++## Sub-features++- `add` registers a Space after a capabilities handshake, under a display name.+- `ls` lists it in `cua spaces ls` (table and `--json`) with provider `direct` and the daemon version.+- `sb-ls` lists the same Space in `cua sb ls`, the sandbox list `cua viewer` also uses.+- `rm` forgets it; the machine keeps running.++## How to get to it (user POV)++- `cua spaces add <host:port> --token <token> --name <name>`.+- `cua spaces ls`, `cua spaces ls --json`, `cua sb ls`.+- `cua spaces rm <id or name>`.+- The apps' "Connect by address" calls the same SDK path (unverified here).++## Driving it with spaces-desk++Preconditions:++- Baseline: `up` already ran `cua spaces add 127.0.0.1:<port> --name verify` (see `spaces-add.log`).++- **Listed with its version.** Run `$H cua "$RUN" -- spaces ls --json --no-size`. Exit 0; `spaces[]` has one object with `id` = `$SPACE_ID`, `name` `verify`, `provider` `direct`, `spacesd_version` matching `cua-spacesd --version`, and `features` including `desktop_stream`.+- **Table view.** Run `$H cua "$RUN" -- spaces ls --no-size`. One row for `$SPACE_ID` named `verify`.+- **Sandbox list.** Run `$H cua "$RUN" -- sb ls`. Expected: one row for `$SPACE_ID` with status `ready` and location `direct`.+- **Re-add updates.** Run `$H cua "$RUN" -- spaces add "127.0.0.1:$PORT" --token "$(cat "$RUN/token")" --name verify2`, then `spaces ls --json --no-size`. Still one entry for `$SPACE_ID`, now named `verify2`.+- **Remove.** Run `$H cua "$RUN" -- spaces rm "$SPACE_ID"`, then `spaces ls --json --no-size`: no entry. `curl -s -o /dev/null -w '%{http_code}' "$URL/health"` is still `204`. Re-add with `--name verify` before other journeys.++## Gotchas++- `cua sb ls` reads the sandbox state files, not the Spaces registry. On current main a Space added only with `cua spaces add` is missing from `cua sb ls`, `cua sb view verify` says `not found: sandbox verify`, and the `cua viewer` index says "No sandboxes", until something connects to it by id (`cua sb view <id>`, `cua do switch <id>` plus an action) and writes the state file. Check `sb-ls` on a fresh cua home, before any other journey, or the earlier journeys mask it.+- On Linux the table header reads `DISK ON THIS MAC` and the low-disk line reads "This Mac is almost out of space". That is the current wording, not a failure of the journey.+- `--name` is the Space's display name; `cua sb` addresses the same Space by its id or its sandbox name (the host, `127.0.0.1`), not by the display name.diff --git a/.cursor/skills/verify-cua-spaces/features/viewer.md b/.cursor/skills/verify-cua-spaces/features/viewer.mdnew file mode 100644index 0000000000..0bc202477d--- /dev/null+++ b/.cursor/skills/verify-cua-spaces/features/viewer.md@@ -0,0 +1,40 @@+# Watch and control in the viewer++A user opens a Space's desktop in the browser from a link the CLI prints, sees the live screen with sound, and works it with the mouse and keyboard.++## Sub-features++- `view-link` prints a ticketed viewer link (`cua sb view <id> --no-open`).+- `view-stream` streams the desktop (H.264, PNG fallback) and the status leaves "Connecting".+- `view-input` delivers clicks at the right place and typed keys to the app under them.+- `view-audio` plays desktop audio when the Space has a sound server.+- `view-no-audio` still streams video when the Space has no reachable sound server.+- `view-index` lists the Space in the `cua viewer` index page and opens it from there.++## How to get to it (user POV)++- `cua sb view <id>` opens the browser (`--no-open` prints the link, `--view-only` watches only).+- `cua viewer` serves an index of every sandbox on one loopback port; a row opens that Space.+- The apps' Space viewer windows use the same stream (unverified here).++## Driving it with spaces-desk++Preconditions:++- Baseline, with `audio=pipewire` in the `READY` line for `view-audio`.+- Chromium is installed (`CHROMIUM=` overrides the binary).++- **Open, stream, click and type.** Run `$H viewer "$RUN" desktop`. It prints `open {"status":"streaming",...}`, a `frames` line with `framesDecoded` >= 1 and the frame `1280x800`, and `input {"click":{"button":1,...,"x":>=640,"y":>=400},"keys":"ok"}`, then `PASS`. `viewer-desktop.png` shows the grid under the toolbar; `pad.jsonl` has the new `button` and two `key` lines.+- **Audio track.** In the same `viewer-desktop.json`, `stats.audio[0].decoded` > 0 and `audioPackets` > 0.+- **No sound server.** Stop audio with `kill` on the `pipewire-pulse`, `wireplumber` and `pipewire` pids from `$EVIDENCE/pids.txt` (only those), then run `$H viewer "$RUN" no-audio`. Expected: `PASS` with video and input as above (no audio track).+- **Audio off by the user.** Run `$H viewer "$RUN" audio-off "audio=none"`. `PASS`; `stats.audio` is empty.+- **Index page.** Run `$H index "$RUN"` (starts `cua viewer --no-open`, follows its keyed URL with a cookie jar, stops it). Expected: `PASS index lists <SPACE_ID>`; `viewer-index.html` has a row linking `/s/<SPACE_ID url-encoded>/open`.++## Gotchas++- On current main, **No sound server** fails: the viewer shows "audio: audio: pa_simple_new(@DEFAULT_MONITOR@): Connection refused" with a Reconnect button and no picture. `OpenMedia` fails as a whole when the audio capture cannot start, although the capability report still claims `audio.desktop`.+- PipeWire without a session manager (no WirePlumber) accepts the record stream but never links it: `OpenMedia` then blocks and the viewer sits at "Connecting to the sandbox…". `up` starts WirePlumber for that reason; a missing WirePlumber is a setup failure, not a product verdict.+- The link carries a viewer ticket (a credential, 1 h by default). `drive-viewer.mjs` redacts it in its JSON.+- Input is leased to one viewer at a time for 5 s after its last input. A second viewer opened right after another one gets `RateLimited ... leased to Web viewer` (in `cua-spacesd.log`) and nothing reaches the pad; `drive-viewer.mjs` waits 6 s before acting for that reason.+- The click lands in pad coordinates: 958,579 for the 75%/75% canvas point, because openbox's title bar takes the top 19 px.+- `view-index` shows nothing for a Space that only `cua spaces add` registered (see register-and-list `sb-ls`).diff --git a/.cursor/skills/verify-cua-spaces/scripts/drive-viewer.mjs b/.cursor/skills/verify-cua-spaces/scripts/drive-viewer.mjsnew file mode 100644index 0000000000..e8571f5b5c--- /dev/null+++ b/.cursor/skills/verify-cua-spaces/scripts/drive-viewer.mjs@@ -0,0 +1,85 @@+// SPDX-License-Identifier: FSL-1.1-MIT+// Copyright (c) 2026 Cua AI, Inc.++// Drives the cua-spacesd HTML5 viewer like a user: open the link `cua sb+// view` printed, wait for the stream, click the bottom-right (white) cell of+// the x11-pad grid, type "ok", then capture. Writes <out>.json, <out>.png and+// <out>.aria.txt, and checks the pad's own event log for the click and keys.+//+// PW_DIR=<dir with playwright-core> CHROMIUM=<binary> node drive-viewer.mjs LINK OUT PAD_LOG++import { createRequire } from "node:module";+import { readFileSync, writeFileSync } from "node:fs";++const { chromium } = createRequire(`${process.env.PW_DIR}/`)("playwright-core");+const [link, out, padLog] = process.argv.slice(2);+const padEvents = () =>+ readFileSync(padLog, "utf8")+ .split("\n")+ .filter(Boolean)+ .map((l) => JSON.parse(l));+const result = { link: link.replace(/ticket=[^&]*/, "ticket=<redacted>"), steps: [] };+const step = (name, data) => {+ result.steps.push({ name, ...data });+ console.log(name, JSON.stringify(data));+};++const browser = await chromium.launch({ executablePath: process.env.CHROMIUM, args: ["--no-sandbox"] });+let failed = null;+try {+ const page = await browser.newPage({ viewport: { width: 1280, height: 860 } });+ const consoleLines = [];+ page.on("console", (m) => consoleLines.push(`${m.type()}: ${m.text()}`));+ await page.goto(link);+ // Ready: the root's data-status leaves "connecting" (the viewer sets it).+ const status = await page+ .waitForFunction(+ () => {+ const s = document.querySelector("#app")?.dataset.status;+ return s === "streaming" || s === "failed" ? s : null;+ },+ null,+ { timeout: 30_000 },+ )+ .then((h) => h.jsonValue())+ .catch(() => "timeout");+ const overlay = (await page.locator(".cua-overlay").innerText().catch(() => "")).trim();+ step("open", { status, overlay, title: await page.title() });+ if (status !== "streaming") throw new Error(`viewer did not stream: ${status} ${overlay}`);+ await page.waitForFunction(() => (window.__cuaViewer?.stats().framesDecoded ?? 0) > 0, null, { timeout: 15_000 });+ step("frames", { stats: await page.evaluate(() => window.__cuaViewer.stats()) });++ // A viewer that just closed still holds the input lease for up to 5 s+ // (the server answers RateLimited "leased to Web viewer"): wait it out.+ await page.waitForTimeout(6_000);+ const before = padEvents().length;+ const canvas = page.locator("canvas.cua-screen");+ const box = await canvas.boundingBox();+ await page.mouse.click(box.x + box.width * 0.75, box.y + box.height * 0.75);+ await page.keyboard.type("ok");+ // The pad journal is the side effect: wait for a click and both keys.+ let fresh = [];+ for (let i = 0; i < 40; i++) {+ fresh = padEvents().slice(before);+ const keys = fresh.filter((e) => e.event === "key").map((e) => e.char).join("");+ if (fresh.some((e) => e.event === "button") && keys.includes("ok")) break;+ await page.waitForTimeout(250);+ }+ const click = fresh.find((e) => e.event === "button");+ const keys = fresh.filter((e) => e.event === "key").map((e) => e.char).join("");+ step("input", { click, keys });+ if (!click || click.x < 640 || click.y < 400) throw new Error(`no click in the white cell: ${JSON.stringify(click)}`);+ if (!keys.includes("ok")) throw new Error(`keys not delivered: ${JSON.stringify(keys)}`);++ writeFileSync(`${out}.aria.txt`, await page.locator("body").ariaSnapshot());+ await page.screenshot({ path: `${out}.png` });+ result.console = consoleLines.slice(-20);+} catch (error) {+ failed = String(error?.message ?? error);+ result.error = failed;+} finally {+ await browser.close();+ writeFileSync(`${out}.json`, JSON.stringify(result, null, 2));+}+console.log(failed ? `FAIL ${failed}` : `PASS evidence ${out}.{json,png,aria.txt}`);+process.exit(failed ? 1 : 0);diff --git a/.cursor/skills/verify-cua-spaces/scripts/spaces-desk.sh b/.cursor/skills/verify-cua-spaces/scripts/spaces-desk.shnew file mode 100755index 0000000000..3747a4c5b5--- /dev/null+++ b/.cursor/skills/verify-cua-spaces/scripts/spaces-desk.sh@@ -0,0 +1,226 @@+#!/usr/bin/env bash+# SPDX-License-Identifier: FSL-1.1-MIT+# Copyright (c) 2026 Cua AI, Inc.+#+# A throwaway Linux Space for verification: Xvfb + openbox + (optional)+# PipeWire audio + the x11-pad input-echo fixture + a source-built+# cua-spacesd, registered as a Space in an isolated cua home. Drive it with+# the source-built `cua` (cua-spaces-cli) and the HTML5 viewer.+#+# spaces-desk.sh build build cua-spacesd, x11-pad and cua (release)+# spaces-desk.sh up [--no-audio] start a Space; prints RUN=<state dir> and READY+# spaces-desk.sh doctor RUN read-only: is this instance worth driving?+# spaces-desk.sh cua RUN -- ARGS... run `cua ARGS` against RUN, transcript into the evidence dir+# spaces-desk.sh viewer RUN NAME [QUERY] open the Space in headless Chromium, click and type, capture+# spaces-desk.sh index RUN `cua viewer`'s index page: does it list the Space?+# spaces-desk.sh down RUN stop what `up` started; keeps the evidence+set -euo pipefail+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../.." && pwd)"+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+SPACESD="$ROOT/libs/cua-spacesd/target/release/cua-spacesd"+PAD="$ROOT/libs/cua-spacesd/target/release/cua-spacesd-x11-pad"+CUA="$ROOT/libs/cua/target/release/cua-spaces-cli"+die() { echo "spaces-desk: $*" >&2; exit 1; }++load() {+ local run="${1:-}"+ [ -n "$run" ] && [ -f "$run/state.env" ] || die "no state at '${run}' (pass the RUN that 'up' printed)"+ # shellcheck disable=SC1091+ . "$run/state.env"+}++# The isolated environment every client command runs in.+client_env() {+ export HOME="$RUN/home" CUA_HOME="$RUN/home/.cua" DISPLAY="$DISPLAY_ID" \+ XDG_RUNTIME_DIR="$RUN/xdg" DO_NOT_TRACK=1 CUA_TELEMETRY=0+ unset CUA_DAEMON CUA_ENV_TOKEN+}++cmd_build() {+ (cd "$ROOT/libs/cua-spacesd" && cargo build --locked --release -p cua-spacesd --bin cua-spacesd \+ && cargo build --locked --release -p cua-spacesd-test-apps --bin cua-spacesd-x11-pad)+ (cd "$ROOT/libs/cua" && cargo build --locked --release -p cua-spaces-cli)+}++start() { # name, command...: start in its own session, record the pid+ local name="$1"; shift+ setsid "$@" >"$EVIDENCE/$name.log" 2>&1 &+ echo "$!" >>"$RUN/pids"+ echo "$name $!" >>"$EVIDENCE/pids.txt"+}++cmd_up() {+ local audio=1+ [ "${1:-}" = --no-audio ] && audio=0+ for b in "$SPACESD" "$PAD" "$CUA"; do [ -x "$b" ] || die "missing $b: run '$0 build' first"; done+ command -v Xvfb >/dev/null && command -v openbox >/dev/null || die "needs Xvfb and openbox"+ RUN="$(mktemp -d "${TMPDIR:-/tmp}/cua-spaces-verify.XXXXXX")"+ mkdir -p "$ROOT/artifacts/cua-spaces/verify"+ EVIDENCE="$(mktemp -d "$ROOT/artifacts/cua-spaces/verify/run-XXXXXXXX")"+ mkdir -p "$RUN/home" "$RUN/xdg" "$RUN/data" && chmod 700 "$RUN/xdg"+ : >"$RUN/pids"+ # Display: Xvfb picks a free one and reports it.+ setsid Xvfb -displayfd 3 -screen 0 1280x800x24 -nolisten tcp 3>"$RUN/display" >"$EVIDENCE/xvfb.log" 2>&1 &+ echo "$!" >>"$RUN/pids"; echo "xvfb $!" >>"$EVIDENCE/pids.txt"+ for _ in $(seq 50); do [ -s "$RUN/display" ] && break; sleep 0.1; done+ [ -s "$RUN/display" ] || die "Xvfb did not start (see $EVIDENCE/xvfb.log)"+ DISPLAY_ID=":$(tr -d '\n' <"$RUN/display")"+ PORT="$(python3 -I -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])')"+ (umask 077; od -An -tx1 -N24 /dev/urandom | tr -d ' \n' >"$RUN/token")+ export DISPLAY="$DISPLAY_ID" XDG_RUNTIME_DIR="$RUN/xdg" HOME="$RUN/home"+ # A private session bus: AT-SPI and PipeWire's session manager need one.+ DBUS_SESSION_BUS_ADDRESS="$(dbus-daemon --session --fork --print-address=1 --print-pid=3 3>"$RUN/dbus.pid")"+ export DBUS_SESSION_BUS_ADDRESS+ cat "$RUN/dbus.pid" >>"$RUN/pids"; echo "dbus $(cat "$RUN/dbus.pid")" >>"$EVIDENCE/pids.txt"+ if [ "$audio" = 1 ] && command -v pipewire >/dev/null && command -v wireplumber >/dev/null \+ && command -v pipewire-pulse >/dev/null; then+ start pipewire pipewire; sleep 0.5+ start wireplumber wireplumber+ start pipewire-pulse pipewire-pulse+ AUDIO=pipewire+ else+ AUDIO=none+ fi+ start openbox openbox+ sleep 1+ start x11-pad "$PAD" --title verify-pad --log "$EVIDENCE/pad.jsonl" --x 0 --y 0 --width 1280 --height 800 --grid+ CUA_ENV_TOKEN="$(cat "$RUN/token")" start cua-spacesd "$SPACESD" --listen "127.0.0.1:$PORT" \+ --quic-port 0 --data-dir "$RUN/data" --token-file "$RUN/no-token-file"+ URL="http://127.0.0.1:$PORT"+ cat >"$RUN/state.env" <<EOF+RUN='$RUN'+EVIDENCE='$EVIDENCE'+DISPLAY_ID='$DISPLAY_ID'+PORT='$PORT'+URL='$URL'+SPACE_ID='direct:127.0.0.1:$PORT'+SPACE_NAME='verify'+AUDIO='$AUDIO'+EOF+ cp "$RUN/state.env" "$EVIDENCE/state.env"+ # Ready means usable: /health answers and Health says the desktop serves.+ local ok=0+ for _ in $(seq 60); do+ if [ "$(curl -s -o /dev/null -w '%{http_code}' "$URL/health")" = 204 ] \+ && CUA_ENV_TOKEN="$(cat "$RUN/token")" HOME="$RUN/home" DO_NOT_TRACK=1 \+ "$CUA" spacesd call "$URL" SystemService/Health 2>/dev/null | grep -q '"HEALTH_STATUS_SERVING","components"'; then+ ok=1; break+ fi+ sleep 0.5+ done+ [ "$ok" = 1 ] || die "cua-spacesd did not become healthy (see $EVIDENCE/cua-spacesd.log); run: $0 down $RUN"+ client_env+ "$CUA" spaces add "127.0.0.1:$PORT" --token "$(cat "$RUN/token")" --name verify >"$EVIDENCE/spaces-add.log" 2>&1 \+ || die "cua spaces add failed (see $EVIDENCE/spaces-add.log); run: $0 down $RUN"+ grep -q '"ready"' "$EVIDENCE/pad.jsonl" || die "x11-pad is not up (see $EVIDENCE/x11-pad.log)"+ echo "RUN=$RUN"+ echo "EVIDENCE=$EVIDENCE"+ echo "READY space=direct:127.0.0.1:$PORT display=$DISPLAY_ID audio=$AUDIO"+}++cmd_doctor() {+ load "$1"+ local bad=0 pid+ while read -r name pid; do+ if kill -0 "$pid" 2>/dev/null; then echo "ok $name pid $pid"; else echo "FAIL $name pid $pid is gone"; bad=1; fi+ done <"$EVIDENCE/pids.txt"+ local owner+ owner="$(ss -Hltnp "sport = :$PORT" 2>/dev/null | grep -o 'pid=[0-9]*' | head -1 | cut -d= -f2)"+ if [ "$owner" = "$(awk '$1=="cua-spacesd"{print $2}' "$EVIDENCE/pids.txt")" ]; then+ echo "ok port $PORT is owned by our cua-spacesd"+ else+ echo "FAIL port $PORT is owned by pid '${owner}', not ours"; bad=1+ fi+ "$SPACESD" --version+ client_env+ CUA_ENV_TOKEN="$(cat "$RUN/token")" "$CUA" spacesd caps "$URL" | head -1+ "$CUA" spaces ls --json --no-size | python3 -I -c \+ 'import json,sys; ids=[s["id"] for s in json.load(sys.stdin)["spaces"]]; print("spaces:", ids); sys.exit(0 if sys.argv[1] in ids else 1)' \+ "$SPACE_ID" || { echo "FAIL $SPACE_ID is not registered"; bad=1; }+ # The service's own doctor: read-only checks, report kept as evidence.+ echo "$(cat "$RUN/token")" >"$RUN/doctor-token" && chmod 600 "$RUN/doctor-token"+ "$SPACESD" doctor --target "$URL" --token-file "$RUN/doctor-token" --out "$EVIDENCE/doctor.json" \+ --artifacts "$EVIDENCE/doctor" >"$EVIDENCE/doctor.txt" 2>&1 || true+ tail -1 "$EVIDENCE/doctor.txt"+ grep -E '^\s+\[(fail|warn)\]' "$EVIDENCE/doctor.txt" || true+ return "$bad"+}++cmd_cua() {+ load "$1"; shift+ [ "${1:-}" = -- ] && shift+ client_env+ local n out err code+ n="$(find "$EVIDENCE" -maxdepth 1 -name 'cli-*.txt' | wc -l)"+ out="$EVIDENCE/cli-$(printf %03d "$n").txt"+ err="$(mktemp)"+ set +e+ "$CUA" "$@" >"$out.stdout" 2>"$err"+ code=$?+ set -e+ {+ printf '$ cua'; printf ' %q' "$@"; printf '\n'+ echo "--- stdout"; cat "$out.stdout"+ echo "--- stderr"; cat "$err"+ echo "--- exit $code"+ } >"$out"+ rm -f "$out.stdout" "$err"+ cat "$out"+ return "$code"+}++cmd_viewer() {+ load "$1"+ local name="${2:?name the capture}" query="${3:-}"+ client_env+ local pw="$RUN/pw"+ if [ ! -d "$pw/node_modules/playwright-core" ]; then+ mkdir -p "$pw" && (cd "$pw" && npm init -y >/dev/null && npm i --no-audit --no-fund --silent [email protected])+ fi+ local browser="${CHROMIUM:-$(command -v chromium || command -v chromium-browser || command -v google-chrome || true)}"+ [ -n "$browser" ] || die "needs Chromium (set CHROMIUM=/path/to/chrome)"+ local link+ link="$("$CUA" sb view "$SPACE_ID" --no-open 2>&1 | grep -o 'http://[^ ]*' | head -1 || true)"+ [ -n "$link" ] || die "cua sb view printed no link"+ [ -z "$query" ] || link="$link&$query"+ PW_DIR="$pw" CHROMIUM="$browser" node "$HERE/drive-viewer.mjs" "$link" "$EVIDENCE/viewer-$name" "$EVIDENCE/pad.jsonl"+}++cmd_index() {+ load "$1"+ client_env+ local out="$EVIDENCE/viewer-index" pid url jar="$RUN/index.jar"+ setsid "$CUA" viewer --listen 127.0.0.1:0 --no-open >"$out.log" 2>&1 &+ pid=$!+ for _ in $(seq 40); do url="$(grep -o 'http://[^ ]*key=[^ ]*' "$out.log" | head -1 || true)"; [ -n "$url" ] && break; sleep 0.25; done+ if [ -n "$url" ]; then curl -s -c "$jar" -b "$jar" -L "$url" >"$out.html"; fi+ kill "$pid" 2>/dev/null || true+ wait "$pid" 2>/dev/null || true+ [ -n "$url" ] || die "cua viewer printed no URL (see $out.log)"+ local want+ want="/s/$(python3 -I -c 'import sys,urllib.parse; print(urllib.parse.quote(sys.argv[1], safe=""))' "$SPACE_ID")/open"+ if grep -qF "$want" "$out.html"; then echo "PASS index lists $SPACE_ID ($out.html)"; else+ echo "FAIL index does not list $SPACE_ID: $(grep -o '<h1>.*' "$out.html" | sed 's/<[^>]*>/ /g' | head -c 200)"; return 1; fi+}++cmd_down() {+ load "$1"+ local pid+ # Kill only what `up` started, newest first.+ for pid in $(tac "$RUN/pids"); do kill "$pid" 2>/dev/null || true; done+ sleep 1+ for pid in $(tac "$RUN/pids"); do kill -9 "$pid" 2>/dev/null || true; done+ rm -rf "$RUN"+ echo "stopped; evidence kept in $EVIDENCE"+}++case "${1:-}" in+ build) cmd_build ;;+ up) shift; cmd_up "$@" ;;+ doctor) shift; cmd_doctor "$@" ;;+ cua) shift; cmd_cua "$@" ;;+ viewer) shift; cmd_viewer "$@" ;;+ index) shift; cmd_index "$@" ;;+ down) shift; cmd_down "$@" ;;+ *) sed -n '5,/^set -euo/p' "$0" | sed '$d;s/^# \{0,1\}//'; exit 2 ;;+esac