Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-sdk): use Cua in your own harness, with no agent layer (CUA-1901) (#5531)
ddupont3d71fc632e61
diff --git a/.github/workflows/docs-compile-lanes.yml b/.github/workflows/docs-compile-lanes.ymlindex a2ae431d98..dbaceceda1 100644--- a/.github/workflows/docs-compile-lanes.yml+++ b/.github/workflows/docs-compile-lanes.yml@@ -1,10 +1,12 @@-name: "Docs: Swift and Kotlin blocks compile"+name: "Docs: Swift, Kotlin and Rust blocks compile"# Docs code blocks tagged test="swift" or test="kotlin" compile against the# cua SDK's Swift package and Kotlin binding in this repository# (tests/e2e/cua-sdk/docs/compile_lane.py). Compile-only: the blocks are# wrapped in a function in a library target, so no Rust library is built or-# linked and nothing runs.+# linked and nothing runs. Blocks tagged test="rust" type-check (cargo check)+# against libs/cua/crates/cua-sdk, so a public API change that breaks a Rust+# snippet fails here.on:pull_request:@@ -12,6 +14,9 @@ on:- "docs/content/docs/**"- "libs/cua/swift/**"- "libs/cua/kotlin/**"+ - "libs/cua/crates/**"+ - "libs/cua/Cargo.toml"+ - "libs/cua/Cargo.lock"- "tests/e2e/cua-sdk/docs/compile_lane.py"- "tests/e2e/cua-sdk/docs/extract.py"- "tests/e2e/cua-sdk/docs/preludes/**"@@ -56,3 +61,26 @@ jobs:persist-credentials: false- name: Compile the Kotlin blocksrun: python3 tests/e2e/cua-sdk/docs/compile_lane.py --lane kotlin++ rust:+ name: rust blocks+ runs-on: ubuntu-latest+ timeout-minutes: 40+ env:+ CARGO_BUILD_JOBS: "4"+ CARGO_TARGET_DIR: ${{ github.workspace }}/target-docs-rust+ steps:+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4+ with:+ persist-credentials: false+ - uses: ./.github/actions/linux-rust-prep+ - name: Install protoc+ uses: ./.github/actions/setup-protoc+ - name: Install pinned Rust toolchain+ uses: dtolnay/[email protected]+ - uses: Swatinem/rust-cache@9d47c6ad4b02e050fd481d890b2ea34778fd09d6 # v2+ with:+ workspaces: ". -> target-docs-rust"+ key: docs-rust+ - name: Type-check the Rust blocks+ run: python3 tests/e2e/cua-sdk/docs/compile_lane.py --lane rustdiff --git a/README.md b/README.mdindex 26f32d502d..4c631ef24e 100644--- a/README.md+++ b/README.md@@ -166,8 +166,8 @@ async with Sandbox.ephemeral(Image.linux(), local=True) as sb:print((await sb.shell.run("uname -a")).stdout)```-- **SDK:** the same API in Python (`pip install cua`), TypeScript (`@trycua/cua`), Swift (`Cua`) and Kotlin (generated bindings), running embedded in your process or through a shared `cua daemon`.-- **Sandboxes need no agent inside.** Readiness comes from the runtime and optional port probes. Images that ship cua-spacesd (port 3211) add processes, files, screenshots, input through cua-driver, and low-latency video and audio streaming.+- **SDK:** the same API in Python (`pip install cua`), TypeScript (`@trycua/cua`) and Rust (`cua-sdk`), running embedded in your process or through a shared `cua daemon`. Swift and Kotlin bindings are not yet released.+- **Sandboxes need no agent inside.** Readiness comes from the runtime and optional port probes. Images that ship cua-spacesd (port 3211) add processes, files, screenshots, input through cua-driver, and low-latency video and audio streaming. There is no agent layer in the SDK either: drive sandboxes from your own harness ([guide](https://cua.ai/docs/cua-sdk/guides/own-harness)).- **Your own machines.** `cua host setup` makes this machine reachable through the cua.ai relay with no port forwarding.**[SDK README](libs/cua/README.md)** | **[Quickstart](https://cua.ai/docs/cua-sdk/quickstart)** | **[CLI reference](https://cua.ai/docs/cua-cli/reference/cli)** | **[Sandbox SDK reference](https://cua.ai/docs/cua-sdk/reference/sandbox)**@@ -213,7 +213,7 @@ uv tool run --from 'cua-bench[browser]' playwright install chromium| [cua SDK and CLI](libs/cua/README.md) | Rust core and the `cua` command: sandboxes, local runtimes, images, Spaces || [cua (Python)](libs/cua/python/README.md) | The cua SDK for Python (`pip install cua`) || [@trycua/cua](libs/cua/typescript/README.md) | The cua SDK for Node and the browser, plus `@trycua/cua/spaces` |-| [Cua (Swift)](libs/cua/swift/README.md) | The cua SDK for Swift (SwiftPM, XCFramework) |+| [Cua (Swift)](libs/cua/swift/README.md) | The cua SDK for Swift (SwiftPM, XCFramework; not yet released) || [cua-sandbox](libs/python/cua-sandbox/README.md) | High-level Python `Sandbox`/`Image`/`Pool` API, a thin wrapper over the cua SDK || [cua-spacesd](libs/cua-spacesd/README.md) | In-sandbox daemon on port 3211: processes, files, desktop, streaming (gRPC) || [Cua Spaces for macOS](apps/cua-spaces-macos/README.md) | The Spaces app on macOS: menu bar and notch, live streams, teleport, agent threads |diff --git a/docs/README.md b/docs/README.mdindex f525cbb36d..5177a34590 100644--- a/docs/README.md+++ b/docs/README.md@@ -47,6 +47,7 @@ are exempt because their generator's check owns them. The lanes:| `windows`, `lume` | shell blocks on a disposable Windows or macOS machine | nightly || `terraform` | `terraform init` and `validate` | PR || `swift`, `kotlin` | the block compiles against `libs/cua/swift` or `libs/cua/kotlin` (statements wrapped in a function; nothing runs) | PR (`docs-compile-lanes.yml`) |+| `rust` | the block type-checks (`cargo check`) against `libs/cua/crates/cua-sdk`; a block without `fn main` is wrapped in an async function returning `Result` | PR (`docs-compile-lanes.yml`) || `excerpt` | the block is byte-identical to `source="<path>#<region>"` in a sample CI builds | every PR (`docs:check-blocks`) |Run the Python and container lanes withdiff --git a/docs/code-block-policy.json b/docs/code-block-policy.jsonindex ab4d7819ef..a7908516f5 100644--- a/docs/code-block-policy.json+++ b/docs/code-block-policy.json@@ -48,6 +48,10 @@"kotlin": {"e2e": "kotlin","what": "Kotlin blocks compile against libs/cua/kotlin (tests/e2e/cua-sdk/docs/compile_lane.py --lane kotlin; compile-only)"+ },+ "rust": {+ "e2e": "rust",+ "what": "Rust blocks type-check against libs/cua/crates/cua-sdk (tests/e2e/cua-sdk/docs/compile_lane.py --lane rust; cargo check, nothing runs)"}},"skip": {diff --git a/docs/content/docs/cua-sdk/guides/desktop.mdx b/docs/content/docs/cua-sdk/guides/desktop.mdxindex 35953bae26..e5d97ad3f2 100644--- a/docs/content/docs/cua-sdk/guides/desktop.mdx+++ b/docs/content/docs/cua-sdk/guides/desktop.mdx@@ -17,6 +17,9 @@ the interfaces below raise `SpacesdNotAvailable`.| `sb.driver.connect()` | The full Cua Driver API: windows, accessibility, background input || `cua mcp --sandbox NAME` | Coding agents and MCP clients |+To drive a sandbox from your own agent loop or bot, with a complete program in+Python, TypeScript and Rust, see [Use Cua in your own harness](/cua-sdk/guides/own-harness).+## Screenshots and inputdiff --git a/docs/content/docs/cua-sdk/guides/meta.json b/docs/content/docs/cua-sdk/guides/meta.jsonindex ac66ad1f6f..a40e05db56 100644--- a/docs/content/docs/cua-sdk/guides/meta.json+++ b/docs/content/docs/cua-sdk/guides/meta.json@@ -1,6 +1,7 @@{"title": "Guides","pages": [+ "own-harness","images","services","lifecycle",diff --git a/docs/content/docs/cua-sdk/guides/own-harness.mdx b/docs/content/docs/cua-sdk/guides/own-harness.mdxnew file mode 100644index 0000000000..d3f7ad9605--- /dev/null+++ b/docs/content/docs/cua-sdk/guides/own-harness.mdx@@ -0,0 +1,389 @@+---+title: Use Cua in your own harness (no agent)+description: Drive a sandbox from your own agent loop or bot with no Cua agent layer. Create or connect, run commands, move files, take screenshots, click and type, from Python, TypeScript, Rust or any gRPC client.+---++import { Tab, Tabs } from 'fumadocs-ui/components/tabs';++The Cua SDK has no agent layer. It gives your code a computer, and your own+harness decides what to do with it. Nothing here calls a model, and nothing+needs to run inside the image besides cua-spacesd, which the Cua images ship.++| Language | Install | Agent code pulled in |+| --- | --- | --- |+| Python | `pip install cua-sandbox` | None. `cua-agent` comes only with the `cua[agent]` extra |+| TypeScript | `npm install @trycua/cua` | None |+| Rust | `cua-sdk` crate, a git dependency for now ([Rust crates](/cua-sdk/reference/rust)) | None |+| Anything else | gRPC client generated from the `.proto` files | None |++## The whole loop++Each program below creates a Linux desktop on this machine, runs a command,+writes and reads a file, clicks, types into a terminal, saves a screenshot,+lists windows through the raw protocol, and deletes the sandbox.++<Tabs groupId="sdk-lang" persist items={['Python', 'TypeScript', 'Rust']}>+ <Tab value="Python">++```bash skip="host-install"+pip install cua-sandbox # Python 3.11 to 3.13+python harness.py+```++```python test="container" id="harness-python"+import asyncio+import json+from pathlib import Path++from cua_sandbox import Image, Sandbox++# Open a terminal titled "harness" and focus it, so typed keys land there.+OPEN_TERMINAL = (+ "export DISPLAY=:1; setsid -f xfce4-terminal -T harness >/dev/null 2>&1; "+ "xdotool search --sync --onlyvisible --name harness windowactivate --sync"+)+++async def main():+ # A Linux desktop on this machine. Leaving the block deletes it.+ async with Sandbox.ephemeral(Image.linux()) as sb:+ # Run a command.+ result = await sb.shell.run("uname -sm")+ print("exec:", result.stdout.strip())++ # Write a file and read it back.+ await sb.files.write_text("/tmp/note.txt", "hello from my harness\n")+ print("file:", (await sb.files.read_text("/tmp/note.txt")).strip())++ # Click, then type into a terminal.+ width, height = await sb.get_dimensions()+ await sb.mouse.click(width // 2, height // 2)+ await sb.shell.run(OPEN_TERMINAL)+ await sb.keyboard.type("echo typed by my harness")+ await sb.keyboard.keypress("enter")+ await asyncio.sleep(1) # let the terminal draw++ # Screenshot (PNG bytes).+ Path("screen.png").write_bytes(await sb.screenshot())+ print("screenshot:", width, "x", height)++ # Any cua.env.v1 RPC without a typed method: proto3 JSON in and out.+ spacesd = await sb.spacesd()+ reply = await spacesd.call_json(+ "WindowsService/ListWindows",+ json.dumps({"filter": {"titleContains": "harness"}}),+ )+ for window in json.loads(reply).get("windows", []):+ print("window:", window["title"], window["bounds"])+++asyncio.run(main())+```++ </Tab>+ <Tab value="TypeScript">++```bash skip="host-install"+npm install @trycua/cua tsx+npx tsx harness.mts # .mts: an ES module, so top-level await works+```++```ts test="container" id="harness-typescript"+import { writeFile } from 'node:fs/promises';+import { embedded, SandboxCreateOptions } from '@trycua/cua';++// Open a terminal titled "harness" and focus it, so typed keys land there.+const OPEN_TERMINAL =+ 'export DISPLAY=:1; setsid -f xfce4-terminal -T harness >/dev/null 2>&1; ' ++ 'xdotool search --sync --onlyvisible --name harness windowactivate --sync';+const text = (bytes: ArrayBuffer) => new TextDecoder().decode(bytes);++// A Linux desktop on this machine.+const sb = await embedded().sandboxes().create(+ SandboxCreateOptions.create({ image: 'ghcr.io/trycua/linux:24.04' })+);+try {+ const env = await sb.spacesd(undefined);++ // Run a command.+ console.log('exec:', text((await env.sh('uname -sm', undefined)).stdout).trim());++ // Write a file and read it back.+ await env.upload('/tmp/note.txt', new TextEncoder().encode('hello from my harness\n').buffer, undefined);+ console.log('file:', text(await env.download('/tmp/note.txt')).trim());++ // Click, then type into a terminal.+ const { width, height } = await env.screenshot(undefined);+ await env.click(Math.floor(width / 2), Math.floor(height / 2));+ await env.sh(OPEN_TERMINAL, undefined);+ await env.typeText('echo typed by my harness');+ await env.press('enter');+ await new Promise((resolve) => setTimeout(resolve, 1000)); // let the terminal draw++ // Screenshot (PNG bytes).+ await writeFile('screen.png', Buffer.from((await env.screenshot(undefined)).image));+ console.log('screenshot:', width, 'x', height);++ // Any cua.env.v1 RPC without a typed method: proto3 JSON in and out.+ const reply = await env.callJson(+ 'WindowsService/ListWindows',+ JSON.stringify({ filter: { titleContains: 'harness' } })+ );+ for (const w of JSON.parse(reply).windows ?? []) console.log('window:', w.title, w.bounds);+} finally {+ // Deletes the sandbox.+ await sb.delete_();+}+```++ </Tab>+ <Tab value="Rust">++The crates are not on crates.io yet. Depend on a release tag:++```toml test="config" id="harness-cargo"+[package]+name = "harness"+version = "0.1.0"+edition = "2021"++[dependencies]+cua-sdk = { git = "https://github.com/trycua/cua", tag = "cua-sdk-v0.5.0" }+serde_json = "1"+tokio = { version = "1", features = ["full"] }+```++```rust test="rust" id="harness-rust"+use cua_sdk::{Cua, CuaConfig, SandboxCreateOptions};+use serde_json::{json, Value};++// Open a terminal titled "harness" and focus it, so typed keys land there.+const OPEN_TERMINAL: &str = "export DISPLAY=:1; setsid -f xfce4-terminal -T harness >/dev/null 2>&1; \+ xdotool search --sync --onlyvisible --name harness windowactivate --sync";++#[tokio::main]+async fn main() -> Result<(), Box<dyn std::error::Error>> {+ // A Linux desktop on this machine.+ let cua = Cua::embedded(CuaConfig::default())?;+ let sb = cua+ .sandboxes()+ .create(SandboxCreateOptions {+ image: "ghcr.io/trycua/linux:24.04".into(),+ ..Default::default()+ })+ .await?;+ let result = drive(&sb).await;+ // Deletes the sandbox, also when a step failed.+ sb.delete().await?;+ result+}++async fn drive(sb: &cua_sdk::Sandbox) -> Result<(), Box<dyn std::error::Error>> {+ let env = sb.spacesd(None).await?;++ // Run a command.+ let out = env.sh("uname -sm".into(), None).await?;+ println!("exec: {}", String::from_utf8_lossy(&out.stdout).trim());++ // Write a file and read it back.+ env.upload("/tmp/note.txt".into(), b"hello from my harness\n".to_vec(), None)+ .await?;+ let note = env.download("/tmp/note.txt".into()).await?;+ println!("file: {}", String::from_utf8_lossy(¬e).trim());++ // Click, then type into a terminal.+ let shot = env.screenshot(None).await?;+ let (width, height) = (shot.width, shot.height);+ env.click(f64::from(width / 2), f64::from(height / 2)).await?;+ env.sh(OPEN_TERMINAL.into(), None).await?;+ env.type_text("echo typed by my harness".into()).await?;+ env.press("enter".into()).await?;+ tokio::time::sleep(std::time::Duration::from_secs(1)).await; // let the terminal draw++ // Screenshot (PNG bytes).+ std::fs::write("screen.png", env.screenshot(None).await?.image)?;+ println!("screenshot: {width} x {height}");++ // Any cua.env.v1 RPC without a typed method: proto3 JSON in and out.+ let request = json!({ "filter": { "titleContains": "harness" } });+ let reply = env+ .call_json("WindowsService/ListWindows".into(), request.to_string())+ .await?;+ let reply: Value = serde_json::from_str(&reply)?;+ for w in reply["windows"].as_array().into_iter().flatten() {+ println!("window: {} {}", w["title"], w["bounds"]);+ }+ Ok(())+}+```++Run it with `cargo run`.++ </Tab>+</Tabs>++The Python program printed this on an Apple silicon Mac (the TypeScript and+Rust programs print the same lines in their own formats):++```text output+exec: Linux aarch64+file: hello from my harness+screenshot: 1280 x 800+window: harness {'x': 5.0, 'y': 29.0, 'width': 818.0, 'height': 483.0}+```++and saved this `screen.png`:++++## Connect to a sandbox you already have++To drive a machine you did not create in this program, connect to it instead.+A sandbox created earlier on this machine connects by name:+`Sandbox.connect("dev")` in Python, `embedded().sandboxes().connect('dev')` in+TypeScript, `cua.sandboxes().connect("dev".into())` in Rust. Any other machine+that runs cua-spacesd connects by address and env token+([Attach to or run cua-spacesd](/cua-sdk/guides/spacesd)). Connecting never+creates or deletes anything.++<Tabs groupId="sdk-lang" persist items={['Python', 'TypeScript', 'Rust']}>+ <Tab value="Python">++```python test="docs" id="harness-connect-python" session="harness-connect-python" prelude="space-url"+import asyncio++from cua_sandbox import Sandbox+++async def main():+ async with Sandbox.connect(url="http://10.0.0.5:3211", token="TOKEN") as sb:+ print((await sb.shell.run("uname -a")).stdout)+++asyncio.run(main())+```++ </Tab>+ <Tab value="TypeScript">++```ts test="docs" id="harness-connect-typescript" prelude="space-url"+import { embedded } from '@trycua/cua';++const sb = await embedded().sandboxes().connectUrl("http://10.0.0.5:3211", "TOKEN", undefined);+const env = await sb.spacesd(undefined);+console.log(new TextDecoder().decode((await env.sh('uname -a', undefined)).stdout));+```++ </Tab>+ <Tab value="Rust">++```rust test="rust" id="harness-connect-rust"+use cua_sdk::{Cua, CuaConfig};++let sb = Cua::embedded(CuaConfig::default())?+ .sandboxes()+ .connect_url("http://10.0.0.5:3211".into(), Some("TOKEN".into()), None)+ .await?;+let env = sb.spacesd(None).await?;+```++ </Tab>+</Tabs>++The typed methods cover processes, files, screenshots, pointer, keyboard and+clipboard ([`SpacesdClient`](/cua-sdk/reference/spacesd)). Python also has+[typed Cua Driver tools](/cua-sdk/guides/desktop#typed-cua-driver) for windows+and accessibility.++## Call any RPC as JSON++Windows, accessibility, Cua Driver tools, tunnels and every other `cua.env.v1`+RPC without a typed method go through `call_json` (`callJson` in TypeScript).+Pass the method as `Service/Method` and the request as proto3 JSON; the reply+is proto3 JSON. `json_methods()` lists the methods it accepts, and the+[protocol reference](/cua-sdk/reference/protocol) has every request and+response. For example, `AccessibilityService/GetTree`, `WindowsService/LaunchApp`+or `DriverService/CallTool`. Server-streaming RPCs, such as+`WindowsService/WatchWindows`, use `call_json_stream`.++## Other languages++cua-spacesd serves native gRPC and gRPC-Web on port `3211`, with server+reflection. Send the env token as `authorization: Bearer <token>`. The+contract is the `.proto` files in+[`libs/cua/proto`](https://github.com/trycua/cua/tree/main/libs/cua/proto)+(package `cua.env.v1`), so there is nothing to map by hand: generate a client+for your language with [buf](https://buf.build). For Go, put this+`buf.gen.yaml` in your module:++```yaml test="config" id="harness-buf-gen"+version: v2+managed:+ enabled: true+ override:+ - file_option: go_package_prefix+ value: example.com/harness/gen+plugins:+ - remote: buf.build/protocolbuffers/go+ out: gen+ opt: paths=source_relative+ - remote: buf.build/grpc/go+ out: gen+ opt: paths=source_relative+```++```bash skip="third-party-cli"+buf generate "https://github.com/trycua/cua.git#ref=main,subdir=libs/cua/proto"+```++Then call it like any gRPC service. The examples below reach a spacesd at+`127.0.0.1:3211`, such as the container in+[Attach to any spacesd](/cua-sdk/guides/spacesd#attach-to-any-spacesd):++```go skip="third-party-cli"+package main++import (+ "context"+ "fmt"+ "log"+ "os"++ envv1 "example.com/harness/gen/cua/env/v1"+ "google.golang.org/grpc"+ "google.golang.org/grpc/credentials/insecure"+ "google.golang.org/grpc/metadata"+)++func main() {+ conn, err := grpc.NewClient("127.0.0.1:3211", grpc.WithTransportCredentials(insecure.NewCredentials()))+ if err != nil {+ log.Fatal(err)+ }+ defer conn.Close()+ ctx := metadata.AppendToOutgoingContext(context.Background(),+ "authorization", "Bearer "+os.Getenv("CUA_ENV_TOKEN"))++ windows, err := envv1.NewWindowsServiceClient(conn).ListWindows(ctx, &envv1.ListWindowsRequest{})+ if err != nil {+ log.Fatal(err)+ }+ for _, w := range windows.Windows {+ fmt.Println(w.Title, w.App.GetName())+ }+}+```++Other languages work the same way with their own buf plugins.++Or call it with no generated code at all, through reflection:++```bash skip="third-party-cli"+grpcurl -plaintext -H "authorization: Bearer $CUA_ENV_TOKEN" \+ -d '{"filter": {"titleContains": "harness"}}' \+ 127.0.0.1:3211 cua.env.v1.WindowsService/ListWindows+```++`grpcurl -plaintext -H "authorization: Bearer $CUA_ENV_TOKEN" 127.0.0.1:3211 list`+lists the services. Keep the token out of URLs and logs: it grants full+control of the machine.diff --git a/docs/content/docs/cua-sdk/index.mdx b/docs/content/docs/cua-sdk/index.mdxindex f1c8741508..976c5b3cc7 100644--- a/docs/content/docs/cua-sdk/index.mdx+++ b/docs/content/docs/cua-sdk/index.mdx@@ -2,7 +2,7 @@title: Cua SDKdescription: Build agentic applications and workflows on isolated Linux, Windows and macOS desktops on your machine, or on any machine that runs cua-spacesd.landing:- summary: 'Create and drive computers from your own code. Python, TypeScript, Swift and Rust.'+ summary: 'Create and drive computers from your own code. Python, TypeScript and Rust.'cta: Explore Cua SDK---@@ -101,9 +101,13 @@ See [pricing](https://cua.ai/pricing#plans).| Python | `cua-sandbox` (`from cua_sandbox import Sandbox`), over the `cua` SDK (`import cua`) || TypeScript | `@trycua/cua` || Rust | `cua-sdk` crate (git dependency), see [Rust crates](/cua-sdk/reference/rust) |-| Swift | `Cua` (SwiftPM) |+| Swift, Kotlin | Not yet released |+| Other languages | gRPC on port 3211, from the [`.proto` files](/cua-sdk/guides/own-harness#other-languages) || CLI | `cua sb ...`, see [Cua CLI](/cua-cli) |+None of them includes an agent: your own code or agent loop drives the+sandbox. See [Use Cua in your own harness](/cua-sdk/guides/own-harness).+## How it fits togetherThe SDK is one Rust core with generated bindings, running in your process ordiff --git a/docs/content/docs/cua-sdk/reference/index.mdx b/docs/content/docs/cua-sdk/reference/index.mdxindex eb7734a808..188b853a6d 100644--- a/docs/content/docs/cua-sdk/reference/index.mdx+++ b/docs/content/docs/cua-sdk/reference/index.mdx@@ -17,8 +17,8 @@ The Cua SDK in Python, TypeScript, Swift, Kotlin and Rust, all generated from th| Python | `pip install cua` (0.5.0; `cua[sandbox]` adds the [high-level API](/cua-sdk/reference/python)) | `import cua` || TypeScript (Node) | `npm install @trycua/cua` (0.5.0) | `import { embedded } from '@trycua/cua'` || Browser | `@trycua/cua/browser` (a WebAssembly subset) | `import { Cua } from '@trycua/cua/browser'` |-| Swift | SwiftPM package `Cua` (macOS 12+, iOS 15+) | `import Cua` |-| Kotlin | `ai.cua.sdk` (JNA; compile-checked, not published yet) | `import ai.cua.sdk.*` |+| Swift | `Cua` (macOS 12+, iOS 15+; not yet released) | `import Cua` |+| Kotlin | `ai.cua.sdk` (JNA; not yet released) | `import ai.cua.sdk.*` || Rust | `cua-sdk` from git (see [Rust crates](/cua-sdk/reference/rust)) | `use cua_sdk::*;` |## Sandboxesdiff --git a/docs/content/docs/index.mdx b/docs/content/docs/index.mdxindex 6e596a64df..824f099c07 100644--- a/docs/content/docs/index.mdx+++ b/docs/content/docs/index.mdx@@ -56,7 +56,7 @@ import { Card, Cards } from 'fumadocs-ui/components/card';## More tools- [Cua CLI](/cua-cli): the `cua` command for Spaces, sandboxes, sign-in and MCP.-- [Cua SDK](/cua-sdk): create and drive computers from your own code, in Python, TypeScript, Swift and Rust.+- [Cua SDK](/cua-sdk): create and drive computers from your own code, in Python, TypeScript and Rust ([no agent layer](/cua-sdk/guides/own-harness)).- [Lume](/lume): macOS and Linux virtual machines on Apple silicon.- [Cua Bench](/cua-bench): run and score computer-use tests on your computer.- [Docs for agents](/docs-for-agents): give a coding agent these docs as Markdown or over MCP.diff --git a/docs/public/img/cua-sdk/own-harness-screen.png b/docs/public/img/cua-sdk/own-harness-screen.pngnew file mode 100644index 0000000000..9f8846f2d4Binary files /dev/null and b/docs/public/img/cua-sdk/own-harness-screen.png differdiff --git a/scripts/docs-generators/headers/cua-sdk/index.md b/scripts/docs-generators/headers/cua-sdk/index.mdindex ad8d345a22..364a3acaec 100644--- a/scripts/docs-generators/headers/cua-sdk/index.md+++ b/scripts/docs-generators/headers/cua-sdk/index.md@@ -3,6 +3,6 @@| Python | `pip install cua` ({{version}}; `cua[sandbox]` adds the [high-level API](/cua-sdk/reference/python)) | `import cua` || TypeScript (Node) | `npm install @trycua/cua` ({{version}}) | `import { embedded } from '@trycua/cua'` || Browser | `@trycua/cua/browser` (a WebAssembly subset) | `import { Cua } from '@trycua/cua/browser'` |-| Swift | SwiftPM package `Cua` (macOS 12+, iOS 15+) | `import Cua` |-| Kotlin | `ai.cua.sdk` (JNA; compile-checked, not published yet) | `import ai.cua.sdk.*` |+| Swift | `Cua` (macOS 12+, iOS 15+; not yet released) | `import Cua` |+| Kotlin | `ai.cua.sdk` (JNA; not yet released) | `import ai.cua.sdk.*` || Rust | `cua-sdk` from git (see [Rust crates](/cua-sdk/reference/rust)) | `use cua_sdk::*;` |diff --git a/tests/e2e/cua-sdk/docs/compile_lane.py b/tests/e2e/cua-sdk/docs/compile_lane.pyindex e02d880398..47e39afb6e 100644--- a/tests/e2e/cua-sdk/docs/compile_lane.py+++ b/tests/e2e/cua-sdk/docs/compile_lane.py@@ -1,6 +1,7 @@#!/usr/bin/env python3-"""swift and kotlin lanes: every docs block tagged ``test="swift"`` or-``test="kotlin"`` compiles against the cua SDK in this repository.+"""swift, kotlin and rust lanes: every docs block tagged ``test="swift"``,+``test="kotlin"`` or ``test="rust"`` compiles against the cua SDK in this+repository.Compile-only by default (nothing runs, no library is linked): each blockbecomes one library target of a generated package, its statements wrapped@@ -10,6 +11,11 @@ Hidden preludes (``prelude="a,b"``) come from ``preludes/<name>.swift`` orcompile_lane.py --lane swift # SwiftPM, depends on libs/cua/swift by pathcompile_lane.py --lane kotlin # Gradle, sources compiled with libs/cua/kotlin+ compile_lane.py --lane rust # Cargo, depends on libs/cua/crates/cua-sdk by path++A rust block with its own ``fn main`` is one binary as written; any other+block is wrapped in an async function returning ``Result`` (so ``?`` works).+Rust is type-checked (``cargo check``), so nothing is linked or run.Results go to ``$CUA_E2E_RESULTS/docs-blocks.jsonl``. Stdlib only."""@@ -28,14 +34,18 @@ from pathlib import Pathimport extract-LANGS = {"swift": ("swift",), "kotlin": ("kotlin", "kt")}+LANGS = {"swift": ("swift",), "kotlin": ("kotlin", "kt"), "rust": ("rust", "rs")}PRELUDES = Path(__file__).resolve().parent / "preludes"SWIFT_PKG = extract.REPO / "libs" / "cua" / "swift"KOTLIN_PKG = extract.REPO / "libs" / "cua" / "kotlin"+RUST_WORKSPACE = extract.REPO / "libs" / "cua"+RUST_SDK = RUST_WORKSPACE / "crates" / "cua-sdk"IMPORT = {"swift": re.compile(r"^\s*(@testable\s+)?import\s+\w"),"kotlin": re.compile(r"^\s*import\s+[\w.*]+\s*$"),+ "rust": re.compile(r"^use\s+[\w:{}, *]+;\s*$"),}+RUST_MAIN = re.compile(r"^\s*(async\s+)?fn\s+main\s*\(", re.M)def target_name(block_id: str) -> str:@@ -65,6 +75,23 @@ def wrap(code: str, lane: str, name: str, prelude: str = "") -> str:if line.strip() not in seen:seen.append(line.strip())indented = "\n".join((" " + ln) if ln.strip() else "" for ln in body)+ if lane == "rust":+ if RUST_MAIN.search(code):+ return (prelude + "\n" + code) if prelude else code+ return "\n".join(+ [+ "#![allow(unused)]",+ *seen,+ "",+ f"async fn {snake(name)}() -> Result<(), Box<dyn std::error::Error>> {{",+ indented,+ " Ok(())",+ "}",+ "",+ "fn main() {}",+ "",+ ]+ )if lane == "swift":return "\n".join([*seen, "", f"func {name[:1].lower() + name[1:]}() async throws {{", indented, "}", ""]@@ -83,6 +110,41 @@ def wrap(code: str, lane: str, name: str, prelude: str = "") -> str:)+def snake(name: str) -> str:+ return re.sub(r"(?<!^)(?=[A-Z])", "_", name).lower()+++def rust_package(root: Path, blocks: list[extract.Block]) -> list[str]:+ """One binary per block in a package outside the workspace, with the+ workspace's lockfile so dependencies resolve as in CI. The blocks name+ their crates as a reader's Cargo.toml would (cua-sdk, serde_json, tokio)."""+ names = [target_name(b.id) for b in blocks]+ root.mkdir(parents=True, exist_ok=True)+ bins = "".join(f'\n[[bin]]\nname = "{snake(n)}"\npath = "src/bin/{n}.rs"\n' for n in names)+ (root / "Cargo.toml").write_text(+ "[package]\n"+ 'name = "cua-docs-blocks"\n'+ 'version = "0.0.0"\n'+ 'edition = "2021"\n'+ "publish = false\n"+ "autobins = false\n\n"+ "[workspace]\n\n"+ "[dependencies]\n"+ f'cua-sdk = {{ path = "{RUST_SDK}" }}\n'+ 'serde_json = "1"\n'+ 'tokio = { version = "1", features = ["full"] }\n' + bins,+ encoding="utf-8",+ )+ shutil.copy(RUST_WORKSPACE / "Cargo.lock", root / "Cargo.lock")+ src = root / "src" / "bin"+ src.mkdir(parents=True, exist_ok=True)+ for b, n in zip(blocks, names):+ (src / f"{n}.rs").write_text(+ wrap(b.code, "rust", n, prelude_code(b, "rs")), encoding="utf-8"+ )+ return names++def swift_package(root: Path, blocks: list[extract.Block]) -> list[str]:names = [target_name(b.id) for b in blocks]root.mkdir(parents=True, exist_ok=True)@@ -141,7 +203,17 @@ def kotlin_project(root: Path, blocks: list[extract.Block]) -> list[str]:def attribute(output: str, names: list[str], blocks: list[extract.Block]) -> dict[str, str]:"""Compiler errors by block (errors name the generated source path)."""errs: dict[str, list[str]] = {}+ last_error = ""for line in output.splitlines():+ # rustc: `error[E...]: message`, then ` --> src/bin/<name>.rs:L:C`.+ if line.startswith("error"):+ last_error = line.strip()+ if line.lstrip().startswith("--> ") and last_error:+ for n, b in zip(names, blocks):+ if f"/{n}.rs" in line:+ errs.setdefault(b.id, []).append(f"{last_error} {line.strip()}")+ last_error = ""+ continueif "error" not in line and not line.startswith("e: "): # kotlinc: `e: file:///...`continuefor n, b in zip(names, blocks):@@ -156,6 +228,9 @@ def build(lane: str, blocks: list[extract.Block], work: Path) -> tuple[int, str,cmd = ["swift", "build", "--package-path", str(work)]for n in names:cmd += ["--target", n]+ elif lane == "rust":+ names = rust_package(work, blocks)+ cmd = ["cargo", "check", "--bins", "--manifest-path", str(work / "Cargo.toml")]else:names = kotlin_project(work, blocks)gradle = shutil.which("gradle")diff --git a/tests/e2e/cua-sdk/docs/test_compile_lane.py b/tests/e2e/cua-sdk/docs/test_compile_lane.pyindex 0c9568c858..15e2fdc4b4 100644--- a/tests/e2e/cua-sdk/docs/test_compile_lane.py+++ b/tests/e2e/cua-sdk/docs/test_compile_lane.py@@ -1,4 +1,4 @@-"""Unit tests for the swift / kotlin compile lanes (no toolchain needed)."""+"""Unit tests for the swift / kotlin / rust compile lanes (no toolchain needed)."""from __future__ import annotations@@ -61,6 +61,37 @@ class CompileLaneTest(unittest.TestCase):errs = cl.attribute(out, names, blocks)self.assertEqual(sorted(errs), ["p#a", "p#b"])+ def test_rust_wrap(self):+ whole = "use cua_sdk::Cua;\n\n#[tokio::main]\nasync fn main() {}\n"+ self.assertEqual(cl.wrap(whole, "rust", "DocsR"), whole)+ out = cl.wrap("use cua_sdk::{Cua, CuaConfig};\n\nlet c = Cua::embedded(CuaConfig::default())?;\n", "rust", "DocsR")+ lines = out.splitlines()+ self.assertEqual(lines[:2], ["#![allow(unused)]", "use cua_sdk::{Cua, CuaConfig};"])+ self.assertIn("async fn docs_r() -> Result<(), Box<dyn std::error::Error>> {", out)+ self.assertIn(" let c = Cua::embedded(CuaConfig::default())?;", out)+ self.assertIn("fn main() {}", out)++ def test_rust_package(self):+ tmp = Path(tempfile.mkdtemp())+ (tmp / "p.mdx").write_text('```rust test="rust" id="a"\nlet x = 1;\n```\n')+ blocks = extract.all_blocks(tmp)+ names = cl.rust_package(tmp / "rs", blocks)+ manifest = (tmp / "rs" / "Cargo.toml").read_text()+ self.assertIn(str(cl.RUST_SDK), manifest)+ self.assertIn("[workspace]", manifest)+ self.assertTrue((tmp / "rs" / "Cargo.lock").exists())+ self.assertTrue((tmp / "rs" / "src" / "bin" / f"{names[0]}.rs").exists())++ def test_attribute_rust_errors(self):+ tmp = Path(tempfile.mkdtemp())+ (tmp / "p.mdx").write_text(+ '```rust test="rust" id="a"\nx\n```\n```rust test="rust" id="b"\ny\n```\n'+ )+ blocks = extract.all_blocks(tmp)+ names = [cl.target_name(b.id) for b in blocks]+ out = f"error[E0599]: no method named `x`\n --> src/bin/{names[1]}.rs:3:5\n"+ self.assertEqual(sorted(cl.attribute(out, names, blocks)), ["p#b"])+if __name__ == "__main__":unittest.main()