Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Streaming example: Kotlin (headless)
The shared scenario (../SCENARIO.md) on the generated
UniFFI Kotlin binding of cua-sdk (libs/cua/kotlin, package ai.cua.sdk,
JNA). It does the same steps as the Swift example, headless only: stream
the desktop and the grid window (the SDK decodes to BGRA and PCM), hash the
last frame (FNV-1a 64), write desktop.wav / window.wav, click grid cell
(2, 3), and print SUMMARY {...}. It also runs as a bench lane
(CUA_BENCH_JSONL, tc_ms decoded from the frames). cpu_* in the bench
JSONL comes from the JVM (ThreadMXBean user time, and process CPU minus
that for sys) because the JVM has no getrusage.
Layout
build.gradle.kts: Kotlin/JVM +application. Dependencies are JNA, kotlinx-coroutines (used by the binding's async calls) and kotlinx-serialization-json (JSON parsing only, no compiler plugin).- The binding is compiled from
libs/cua/kotlin/src/main/kotlinafter a one-line patch copied intobuild/generated/cua-binding. As checked in, it does not compile:MediaSession,PortForwardandSpaceStreamSessionget bothsuspend fun close()(the Rust method) andAutoCloseable.close(), which Kotlin reports as "Conflicting overloads". The patch renames the async one tocloseAsync(). The FFI symbol does not change.
Build
No JDK is needed on the host. Compile inside the official Gradle image:
docker run --rm --memory=4g --memory-swap=4g \
-v "$PWD":/work -v "$HOME/.gradle-docker":/gh -e GRADLE_USER_HOME=/gh \
-w /work/examples/streaming/kotlin gradle:8.10.2-jdk17 \
gradle --no-daemon -q installDist # or compileKotlin
(run from the repo root). With a local JDK 17+ and Gradle, gradle installDist
in this directory does the same.
Run
The binding needs the cua-sdk native library for the JVM's OS and arch, on
-Djna.library.path. The start script and gradle run use
$CUA_SDK_LIB_DIR, or libs/cua/target/release by default.
- macOS host with a JDK:
cargo build --release -p cua-sdkinlibs/cua, thenCUA_SDK_LIB_DIR=libs/cua/target/release gradle run, with the scenario variables (CUA_ENV_URL,CUA_ENV_TOKEN, ...) exported. - Docker (Linux), the tested path: build
libcua_sdk.sofor Linux in a Rust image.libs/cuahas path dependencies onlibs/cua-spacesdandlibs/cua-driver, so mount the whole repo. Then run the installed distribution in the Space's network namespace. From the repo root:
docker run --rm --memory=4g --memory-swap=4g -v "$PWD":/work:ro \
-v cua-e2e-kotlin-target:/target -e CARGO_TARGET_DIR=/target -e CARGO_BUILD_JOBS=4 \
-w /work/libs/cua <rust image with clang/cmake> cargo build --locked --release -p cua-sdk
eval "$(libs/cua/bench/streaming/scripts/space.sh env)" # CUA_ENV_TOKEN
docker run --rm --memory=4g --memory-swap=4g --network container:cua-e2e-bench-space \
-v "$PWD":/work -v cua-e2e-kotlin-target:/target:ro \
-e CUA_ENV_URL=http://127.0.0.1:3211 -e CUA_ENV_TOKEN -e CUA_STREAM_SECONDS=5 \
-e CUA_OUT_DIR=/tmp/out -e JAVA_OPTS=-Djna.library.path=/target/release \
gradle:8.10.2-jdk17 /work/examples/streaming/kotlin/build/install/cua-streaming-kotlin/bin/cua-streaming-kotlin
The bench lane works the same way with -e CUA_BENCH_JSONL=... -e CUA_BENCH_TARGET=... -e CUA_BENCH_SECONDS=... -e CUA_BENCH_AUDIO=....
Notes (the same as the Swift example)
- Click. It first sends a media-plane
actionon a session opened withSESSION_POLICY_ALLOW_ACTIVATION. If the fixture log shows no press, it falls back toComputerService.PointerwithDELIVERY_FOREGROUND(auto delivery uses X11XSendEvent, which GTK3 ignores). keyframes/bytesarenull. The decoded callbacks do not carry them.- Stop the bench fixture before the scenario. The timecode window is
keep-above at (128, 96) and covers grid cell (2, 3), so the foreground
click lands on it and
loggedis false. - Audio allocations.
PcmAudio.samplesis aList<Short>in this binding (boxed). That is fine for an example, but it allocates heavily per audio frame.