Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Streaming example: TypeScript in the browser
The shared scenario (../SCENARIO.md) in a web page. The
page is plain ES modules, so there is no bundler. It uses:
@trycua/cua/browser(the wasm build ofcua-sdk, over gRPC-Web) for the env calls:health,sh,click, andcallJsonforStreamService.ListTargets,OpenMediaandCloseMedia;- the media plane (
libs/cua/proto/MEDIA.md) on a browserWebSocketto/media. The ticket goes in the subprotocolrcdp.v2, cua.ticket.<ticket>and never in the URL. Binary video packets are length-prefixed JSON headers, and audio packets areRAU2; - WebCodecs
VideoDecoderfor H.264. The input is Annex B with nodescription, and the codec string comes from the SPS. Frames are drawn onto a<canvas>; - WebCodecs
AudioDecoderfor Opus, played withAudioContext(with a 40 ms scheduling buffer) and written to WAV; - the click as a media-plane
actionwith a pixel basis on a session opened withSESSION_POLICY_ALLOW_ACTIVATION. The fallback isenv.clickin screen coordinates.
The browser SDK has no media-session API, so the page opens the WebSocket itself. That is the only part of the media plane it implements.
Build (once)
Run these from the repository root:
(cd libs/cua/typescript && npm ci && npm run build:browser) # wasm32 + wasm-bindgen 0.2.126 -> typescript/browser/
cd examples/streaming/typescript-web
npm install # links @trycua/cua (file:) and installs playwright
npm run install-browser # chromium-headless-shell into node_modules (PLAYWRIGHT_BROWSERS_PATH=0)
serve.mjs serves this directory at /, the SDK's browser/ at /sdk/,
and @ubjs/core (the generated bindings' only bare import) at /ubjs-core/
through the import map in index.html.
Interactive
S=libs/cua/bench/streaming/scripts/space.sh
$S start && eval "$($S env)"
node examples/streaming/typescript-web/serve.mjs 8765 # http://127.0.0.1:8765/
Open the page, paste CUA_ENV_URL and CUA_ENV_TOKEN, and press Run
scenario. The page draws the video, plays the audio, and offers the two WAV
files as downloads. The spacesd serves gRPC-Web with CORS on its own port,
so a page on another loopback origin can reach it.
Headless (Playwright)
cd examples/streaming/typescript-web
CUA_OUT_DIR=./out node headless.mjs
headless.mjs serves the page on an ephemeral loopback port. It launches
Playwright's headless Chromium with chromium.launch(), which gives a fresh
temporary profile that is removed on close. It never uses an installed
Chrome or its profile. It injects the configuration through
addInitScript (so the token stays out of URLs) and relays what the page
reports through exposeFunction("cuaReport"): the log, the SUMMARY line,
the WAV files (written to $CUA_OUT_DIR) and the bench JSONL.
Bench mode
$S fixture timecode
CUA_BENCH_JSONL=/tmp/web.jsonl CUA_BENCH_TARGET='window:CUA Bench Timecode' \
CUA_BENCH_SECONDS=5 CUA_BENCH_AUDIO=1 libs/cua/bench/streaming/shims/browser.sh
framelines carry the packet'sseq,bytes,keyandcap_us.unix_nsis the time of theVideoDecoderoutput callback, andtc_msis read from the canvas (getImageDataof the timecode strip) right afterdrawImage.audiolines are written at theAudioDecoderoutput.bytesis the size of the Opus packet.- The browser has no
getrusage. The runner writes theendline with the CPU time of every Chromium process (browser, renderer, GPU, utility), taken from CDPSystemInfo.getProcessInfo. That figure is user and system time combined, so it goes incpu_user_s, andcpu_sys_sis0(cpu_scopesays so). browser.shexits 3 withskipped: …when Playwright, its Chromium or the SDK browser build is missing. It exits 2 when the harness environment is missing.
Notes
last_hashis FNV-1a 64 over BGRA read back from the canvas. The canvas applies the browser's YUV-to-sRGB conversion, so the hash is not comparable with the Node/Rust hashes, which are taken from the SDK decoder's BGRA.- Like the Node example, the click is not observed on the current
linuximage. The action is delivered (action_delivered: true), but the grid fixture logs nobutton_press. See the Node README.