Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

7.2 KB
Raw Blame

OpenKoalaBots (Rust + Tauri)

A desktop app where you hire named Bots, give each one its own computer (a Space), and watch it work beside the thread. Built on the cua SDK's Rust crates with a Tauri 2 shell and a React UI.

npx degit trycua/cua/examples/openkoalabot-example-tauri my-app
cd my-app && pnpm install
pnpm tauri dev

The cua crates and the image list are not published yet, so a degit copy does not build on its own today. Until they are, develop inside the cua repo (below).

Developing inside the cua repo

This example depends on the cua SDK by path, as a local workspace override:

Dependency Where it comes from today
cua-spaces, cua-sandbox-core, cua-daemon ../../../libs/cua/crates/* (path dependencies in src-tauri/Cargo.toml)
The sandbox image list ../../libs/images/sandbox-images.json, imported by the UI (ui/src/images.ts) and compiled into the core (src-tauri/core/src/plan.rs)

None of these are on crates.io or npm yet. Once they are, the path dependencies become version dependencies and the degit quick start above works as is.

cd examples/openkoalabot-example-tauri
pnpm install --frozen-lockfile
pnpm tauri dev          # Vite on :1430 plus the Rust shell

What is in it

Path What
src-tauri/core (openkoalabot-example-core) The app logic, headless, on cua-spaces: add, create and delete a Space, the desktop and per-window streams (media tickets), the window list, one agent thread per Bot, SHA-256 verified file drops, approval-gated session teleport (it ships with Cua Spaces; see below), presence, and the New Space plan (plan.rs). Also the openkoalabot-example-scenario runner.
src-tauri/app (openkoalabot-example-tauri) The Tauri shell. Every command is a one-liner over the core. Local Spaces run on the SDK's own runtime (cua_daemon::local::VmmLocal: containers, QEMU, Lume).
ui/ The webview (Vite + React): the Bots roster, one thread per Bot with step, status, approval and file cards, the docked composer, the Computer panel (live desktop over media wire v2, decoded with WebCodecs, picture in picture, window list, drop zone), the New Space wizard and the teleport approval sheet.
ui/src/assets The koala artwork (copied from examples/openkoalabots-assets).

The UI follows the system's light or dark mode.

Spaces

New Space opens a four-step wizard (System, Resources, Options, Summary). The image dropdown lists the published entries of the shared image list, and each entry names the engine it runs on in each location, so the wizard never offers a combination the image cannot run. Create makes one SDK call, Spaces::create(SpaceCreate { on, image, kind, runtime, name, .. }):

Where on kind / runtime
This machine local container / auto (gVisor when available), vm / qemu, vm / lume; plus cpus, memory_mb
Cua Cloud cloud Ended: a cloud plan fails with "Cua Cloud has ended; use a local sandbox or your own machine". Create the Space on This machine instead.

Add by address (in the wizard footer or the empty state) registers a machine that already runs cua-spacesd, for example a local container:

TOKEN=$(openssl rand -hex 16)
docker run -d --name cua-e2e-openkoalabots --runtime=runsc --memory=4g -e CUA_ENV_TOKEN=$TOKEN \
  -p 127.0.0.1:3211:3211 cua-e2e-local/linux:docker-local-arm64
# then add 127.0.0.1:3211 with $TOKEN

Delete… is a separate control that always asks first: a Space the app created (in Cua Cloud or on this machine) is deleted with its sandbox; a Space added by address is only removed from the list.

Teleport

Teleport ships with Cua Spaces (source-available), not with the MIT SDK this sample uses.

  • Session teleport (the approval sheet) calls the Spaces tools teleport_manifest and teleport_app (openkoalabot_example_core::teleport). The app runs the Spaces runtime in process with no teleport extension, so the runtime refuses with HostCapabilityMissing and the sheet shows "Session teleport ships with Cua Spaces (source-available)."
  • App teleport (the app picker, dropped app bundles and window drags) is not part of this sample. The drop zone's Teleport an app… button and a dropped app bundle show "App teleport ships with Cua Spaces (source-available)." and send nothing.

The Computer panel has one drop zone, the SDK's <cua-drop-zone>. Dropped files upload (SHA-256 verified); Send file… picks files to upload.

Picture in picture

The PiP button on the stream pops the desktop out; each row of the window list pops out that one window. Each is a small always-on-top window (open_pip) with its own media session, closed with it. Labels and routes come from @trycua/cua/spaces/pip (pipWindowLabel, encodePipRoute).

Host safety

The app keeps its Spaces registry and local sandbox state in its own data directory (never ~/.cua). Tests and the scenario use temp registries.

OPENKOALABOTS_WALKTHROUGH=<file.json> runs a scripted demo through the same commands a person uses (add Spaces, hire Bots, send messages, open a sheet); see the Walkthrough type in ui/src/api.ts.

Tests

pnpm test                    # vitest: 57 tests (image dropdown, wizard, thread cards, Space section, media wire, PiP, drop zone, error messages)
cd src-tauri
cargo test -p openkoalabot-example-core -- --test-threads=4   # 34 unit + 8 hermetic + 4 plan
cargo build -p openkoalabot-example-tauri --features custom-protocol  # the shell, serving ui/dist (run `pnpm build` first)
OKB_WEBVIEW_ISOLATION=1 cargo test -p openkoalabot-example-tauri --features custom-protocol --test webview_isolation  # macOS: a temp-HOME run writes nothing under the real ~/Library

Webview data stays in the app's data directory ($HOME/Library/Application Support/ai.cua.openkoalabot.example on macOS): a data directory under it on Windows and Linux, a non-persistent store on macOS. The page's saved state (Bots, selection, threads) is ui-state.json there, not web storage.

The hermetic tests run against the SDK's in-process mock cua-spacesd (cua_spacesd_client::testing) on loopback: processes are simulated and files live in memory, so nothing runs on or writes to the host. The mock has no file transfers, so there a drop is checked to fail loudly; the docker lane of the scenario covers real drops. Teleport is checked to refuse with the Cua Spaces message. tests/plan.rs checks that a wizard plan reaches the SDK as the right call (a recording fake LocalRuntime, no container or VM is started).

Scenario

cd src-tauri && cargo build -p openkoalabot-example-core --bins
../../openkoalabot-example-scenario/run.sh --impl tauri --lane fixture   # no docker
../../openkoalabot-example-scenario/run.sh --impl tauri --lane docker    # linux, runsc, 4 GiB

The same behaviour ships in openkoalabot-example-swift and openkoalabot-example-ts; all three pass the shared scenario in openkoalabot-example-scenario.

License

Source-available under FSL-1.1-MIT: its live stream uses the streaming client that ships with Cua Spaces (cua-spaces-ext). Each release becomes MIT two years after it ships.