Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Streaming example: Python
Implements ../SCENARIO.md with the cua Python binding
(libs/cua/python, UniFFI over the Rust cua-sdk crate). Standard library
only; the SDK decodes video (VideoToolbox on macOS, OpenH264 elsewhere) and
Opus audio, so the example receives BGRA frames and s16 PCM through
SpacesdClient.open_media_decoded_with_audio(options, DecodedFrameSink, PcmSink).
Setup (uv, Python 3.12)
From the repository root:
examples/streaming/python/setup.sh # cargo build --release -p cua-sdk + venv
examples/streaming/python/setup.sh --no-build # reuse libs/cua/target/release/libcua_sdk.*
setup.sh creates examples/streaming/python/.venv, installs
libs/cua/python into it (non-editable, no deps) and copies the freshly
built libcua_sdk library next to the installed cua/_native.py. It honours
CARGO_TARGET_DIR (default libs/cua/target) and CARGO_BUILD_JOBS
(default 4).
Run (headless)
libs/cua/bench/streaming/scripts/space.sh start
eval "$(libs/cua/bench/streaming/scripts/space.sh env)"
CUA_HEADLESS=1 CUA_STREAM_SECONDS=5 CUA_OUT_DIR=/tmp/cua-py-out \
examples/streaming/python/.venv/bin/python examples/streaming/python/stream_example.py
libs/cua/bench/streaming/scripts/space.sh stop
It prints the driver health, one line per target, per-stream stats, and a
final SUMMARY {...} line, and writes desktop.wav / window.wav to
CUA_OUT_DIR (default ./out). Exit code 0 only when both streams produced
frames and the click was logged by the grid fixture.
CUA_HEADLESS=0 renders frames with tkinter (no audio playback: the
standard library has no audio output). Without tkinter it runs headless.
Bench lane
libs/cua/bench/streaming/shims/python.sh runs the example in bench mode
from the venv (exits 3 with build instructions when the venv is missing):
libs/cua/bench/streaming/scripts/space.sh fixture timecode
CUA_BENCH_JSONL=/tmp/py.jsonl CUA_BENCH_TARGET='window:CUA Bench Timecode' \
CUA_BENCH_SECONDS=5 CUA_BENCH_AUDIO=0 libs/cua/bench/streaming/shims/python.sh
tc_ms is decoded from the fixture's 48-cell strip on every decoded frame
(syncs + checksum checked, 2³² window chosen nearest to the client clock).
Notes and SDK gaps
- The decoded callbacks do not carry the encoded access unit's size or
keyframe flag (
DecodedVideoFramehas neither), nor the encoded audio packet size (PcmAudio). Sokeyframes/bytesin the summary andkey/bytesin the bench JSONL arenull.encoded_frames/audio_packetscome fromMediaSession.stats(). last_hashis FNV-1a 64 over the last decoded BGRA frame (hash_of).StreamService.ListTargetshas no typed wrapper; it goes throughSpacesdClient.call_json.- Media sessions are view-only by default (
action_result.error.code = view_only), so the window session is opened withMediaOpenOptions.request_json = {"policy": "SESSION_POLICY_ALLOW_ACTIVATION"}.MediaOpenOptionshas no typed policy field. - The click is a media-plane
actionviaMediaSession.send_control; if no deliveredaction_resultarrives within 3 s the example falls back toSpacesdClient.clickin screen coordinates (click.viasays which).