Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Streaming example: TypeScript on Node
The shared scenario (../SCENARIO.md) on @trycua/cua for
Node. It runs as TypeScript directly: Node 23.6 and later strip the types, so
there is no build step. It has been tested on Node 26.
What it uses from the SDK:
embedded().spacesd(url, token): a directSpacesdClient. It callshealth,sh(to runcua-fixtures start gridand read the fixture log),callJson("StreamService/ListTargets")andclick(the fallback path).openMediaDecodedWithAudio: this is the default media path. The SDK decodes on its own delivery thread: H.264 goes to packed BGRA through VideoToolbox on macOS and OpenH264 elsewhere, and Opus goes to s16 PCM. The example hashes the last frame (FNV-1a 64 over BGRA), reads the bench timecode (tc_ms), checks the clicked pixel and writes the WAV files.openMediaWithAudio(CUA_NODE_MEDIA=encoded): this path delivers raw access units and Opus packets. It gives exactbytesandkeyframesper frame, but there are no pixels, sotc_msis null, and there is no WAV.last_hashis then the hash of the last access unit.MediaSession.sendControlsends the click as a media-planeactionwith a pixel basis (the currentgeometry_epochand the last frame sequence). The window session is opened withpolicy: SESSION_POLICY_ALLOW_ACTIVATIONthroughMediaOpenOptions.requestJson, because the default view-only policy refuses input. If the action is not delivered, the example falls back toenv.clickin screen coordinates (window origin + (200, 280)).
Limitations:
- The decoded callbacks (
DecodedVideoFrame,PcmAudio) do not carry the access-unit size, the keyframe flag or the Opus packet size. In the default mode, the summary'skeyframes/bytesand the bench lines'bytes/keyare thereforenull. UseCUA_NODE_MEDIA=encodedwhen you need them. - Node has no renderer, so
CUA_HEADLESS=0is ignored and the example always runs headless.
Build the SDK (once)
Run these from the repository root:
cd libs/cua
CARGO_BUILD_JOBS=4 cargo build --release -p cua-sdk # features default to media + media-decode
(cd typescript && npm ci && npm run build)
# Stage libcua_sdk + the N-API runtime as the local @trycua/cua-<triple>
# package. Do not set CARGO_TARGET_DIR for this step (see "Known issues").
env -u CARGO_TARGET_DIR node scripts/stage-uniffi-library.mjs --only=node
cd ../../examples/streaming/typescript-node && npm install # links @trycua/cua (file:)
Run it headless
S=libs/cua/bench/streaming/scripts/space.sh
$S start && eval "$($S env)"
(cd examples/streaming/typescript-node && CUA_OUT_DIR=./out node main.ts)
$S stop
The last line of output is SUMMARY {...}. The WAV files go to
$CUA_OUT_DIR/desktop.wav and $CUA_OUT_DIR/window.wav. The exit code is 0
only when both streams produced frames and the fixture logged the click.
Bench mode
$S fixture timecode
CUA_BENCH_JSONL=/tmp/node.jsonl CUA_BENCH_TARGET='window:CUA Bench Timecode' \
CUA_BENCH_SECONDS=5 CUA_BENCH_AUDIO=1 libs/cua/bench/streaming/shims/node.sh
This appends open, frame (with tc_ms), audio and end lines. In
end, the CPU times come from process.resourceUsage(), and they cover the
whole Node process, including the SDK's Rust threads and decoders. The shim
exits 2 when the harness environment is missing and 3 (skipped: …) when a
build step has not been done.
node --test pixels.test.ts unit-tests the FNV, timecode and WAV helpers
without the native library.
Known issues (SDK and driver)
libs/cua/scripts/build-node-runtime.mjsfails with "missing built N-API runtime under …/runtime/napi/target/release" whenCARGO_TARGET_DIRis set. Cargo then writes to that directory, but the script looks in the temporary crate's owntarget/. Unset it for staging.- The click is not observed on the
linuxSpace image. The media-planeactionreturnsaction_result{delivered: true}andenv.click/env.moveToreturn OK, but the pointer does not move and the grid fixture logs no event.xdotoolinside the container on the sameDISPLAY=:1works. So the summary reportsclick.logged: falseand the exit code is 1.