Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Streaming example: Swift
The shared scenario (../SCENARIO.md) on the cua Swift SDK
(libs/cua/swift, import Cua): connect, start the grid fixture, list
targets, stream the desktop and the grid window (H.264 + Opus, decoded to
BGRA / PCM by the SDK), click grid cell (2, 3), print SUMMARY {...}. It is
also a bench "language lane" (CUA_BENCH_JSONL).
- Headless (default,
CUA_HEADLESS=1): counts frames, hashes the last decoded frame (FNV-1a 64 over BGRA), writesdesktop.wav/window.wav. - Window (
CUA_HEADLESS=0): the same run while an AppKit window shows the decoded frames and AVAudioEngine plays the audio. It opens a window on your desktop, so run it yourself; CI and the bench only use headless.
Build
The package depends on libs/cua/swift by path and links the cua-sdk static
library through an XCFramework (the package's CUA_SWIFT_XCFRAMEWORK mode):
# 1. cargo build --release -p cua-sdk, then wrap target/release/libcua_sdk.a as
# libs/cua/swift/build/CuaSDKFFI.xcframework (git-ignored there).
examples/streaming/swift/scripts/build-ffi.sh
# 2. Build the example (the path is relative to libs/cua/swift).
cd examples/streaming/swift
CUA_SWIFT_XCFRAMEWORK=build/CuaSDKFFI.xcframework swift build -c release
Command Line Tools are enough, no Xcode needed. The linker prints object file ... was built for newer 'macOS' version warnings for the Rust archive. They
are harmless.
Why not the dev dylib? libs/cua/swift/lib/libcua_sdk.dylib is checked in
and was older than the Rust sources when this was written: it lacks
open_media_decoded*, so linking against it fails. The XCFramework route
leaves tracked files alone.
deps/cua-swift is a symlink to libs/cua/swift. SwiftPM names a path
package after its directory, and both this package and the SDK live in a
directory called swift, so the symlink gives the dependency a different
identity.
Run
libs/cua/bench/streaming/scripts/space.sh start
eval "$(libs/cua/bench/streaming/scripts/space.sh env)"
cd examples/streaming/swift
CUA_HEADLESS=1 CUA_STREAM_SECONDS=5 CUA_OUT_DIR=out .build/release/cua-streaming
# window mode (opens a window, plays audio):
CUA_HEADLESS=0 .build/release/cua-streaming
Bench lane (the harness sets these):
libs/cua/bench/streaming/scripts/space.sh fixture timecode
CUA_BENCH_JSONL=out/bench.jsonl CUA_BENCH_TARGET="window:CUA Bench Timecode" \
CUA_BENCH_SECONDS=5 CUA_BENCH_AUDIO=1 .build/release/cua-streaming
tc_ms is decoded from every frame. On a display: target the strip is
found at the bench window's position from ListTargets.
Notes
- Click. The example first sends a media-plane
action(tool: "click", pixel basis) on a session opened withpolicy: SESSION_POLICY_ALLOW_ACTIVATION. The default policy is view-only and the driver answersview_only. If the fixture log shows no press, it falls back toComputerService.PointerwithDELIVERY_FOREGROUNDat the window origin + (200, 280). The summary'sviasays which one landed.env.click()(auto delivery) picks X11XSendEvent, which GTK3 ignores, so it is not used. keyframes/bytesarenull. The decoded callbacks (DecodedFrameSink,PcmSink) carry neither the encoded size nor the keyframe flag. The bench JSONL has"bytes":null,"key":nullfor the same reason.- Summary extras.
pcm_frames,frames_dropped,decode_errors,last_size,hash_ofand the click'saction_*/env_reportfields are extra keys beyond SCENARIO.md. - Stop the bench fixture before the scenario. Its keep-above window
covers grid cell (2, 3) (
space.sh fixturestarts it), so the click misses the grid. - Frame counts. The driver sends frames on damage, so a static desktop
gives about 1 frame in 5 s and
fpsisnull.