Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Cua Driver Bench task pack (public)
The multi-app tasks of the public Bench v2 task set, plus separately selectable
web tasks. These run GUI-only (no coding tools) under the ids in the table; the harness reads
them from here (CDB_TASKPACK, see
../automated-eval/macos_bench/cdb_adapter.py).
| Pack task | Bench id | Apps |
|---|---|---|
tasks/shared/cdb-s02 |
CDB-G02 | Copperfield Mail, Signal Quay Chat and Daymark Calendar (Electron), LibreOffice Calc |
tasks/shared/cdb-s03 |
CDB-G03 | Ledger Lantern Mail (Electron), LibreOffice Calc, GnuCash |
tasks/shared/cdb-s04 |
CDB-G04 | Rillwatch Cases and Margin Lantern Manual (Electron), the Morrowglass Archive site in Chrome, a log file |
tasks/shared/web-01 |
WEB-01 (separate) | Fern signup in Chrome |
tasks/shared/web-02 |
WEB-02 (separate) | Pebble checkout in Chrome |
WEB-01 and WEB-02 are Amendment 24.3 originals, not members of bench_v2.V2_TASKS.
Select them explicitly with --tasks WEB-01 WEB-02; run_bench.WEB_TASKS lists them.
They use a Node built-in-only loopback service, task-local routes and checks,
and a fresh workspace-local Chrome profile. No internet or external assets are used.
The task manifest, v2-taskset.json, lists every Bench v2 task with its
category, OS availability, conditions and split. A task kept out of this repository would run
from a separate pack named by CDB_TASKPACK_HELDOUT, with the same layout.
Each task directory holds:
brief.md: what the agent is told;task.cuabench.json: the manifest, with a sha256 for every input;apps/: the task's Electron apps;fixture/: the starting data;reset/:setup.pybuilds the workspace,verify.pychecks it byte for byte;platform/launch.macos.json: the launch descriptor the harness follows;evaluator/: the checker and its oracle (final state only, every required check);tests/: evaluator, reset and manifest tests;tools/: fixture and digest builders;PROVENANCE.md: origin, prior art, review and licence.
Source and changes
The three cdb-s0* tasks were written by Cua in its private benchmark repository and are
copied from revision 16a1937a79ff2ee2e48f5b5d9bb5e6f944119b5a, the revision
every registered Bench v2 run used. The copy differs from that revision only in
licence and provenance metadata:
- the
licensefield of everypackage.jsonandpackage-lock.jsonis nowFSL-1.1-MIT(it was a proprietary licence reference, andMITin #4973 and #4978); PROVENANCE.mdand theprovenanceblock oftask.cuabench.jsondescribe the origin for a public reader, and each task'sREADME.mdno longer links to documents of the private repository;- the manifest digests are refreshed for those files, and the
cdb-s04oracle's pinned digests of the archive site'spackage.jsonandpackage-lock.jsonare updated to the relicensed files.
For those three tasks, briefs, apps, fixtures, reset scripts, launch descriptors,
evaluators and the rest of the oracles are byte-identical to the source revision. The tree digests
of the registered runs are kept in pins.json cdb_pack.registered_runs_tree_sha256;
pins.json cdb_pack.tree_sha256 holds the digests of this copy, which the
preflight checks.
Checks
From a task directory:
python3 -m unittest discover -s tests # evaluator, reset and manifest tests
python3 tools/refresh_manifest_digests.py --check
(cd apps/<app> && node --test tests/*.test.js) # app store tests; no install needed
After changing a task-owned input, run tools/refresh_manifest_digests.py,
the tests above, and tools/validate_cdb.py from the harness in a VM, then
update pins.json cdb_pack.tree_sha256. A change to a task that a registered
run used needs a dated amendment before the next run.
For web tasks, shared helpers live in tasks/shared/web-common (not a task).
After changing common.py, regenerate task-local evaluator helpers with
python3 tasks/shared/web-common/vendor_evaluator.py. Run each web task's own
tools/refresh_manifest_digests.py, then its --check. Both manifests enumerate
shared runtime inputs; reset verification rehashes those inputs every trial,
because a task tree pin alone does not cover sibling files. Refresh tree pins with
python3 tools/check_pack_pins.py --write from the harness directory.
Adding tasks
New tasks go into tasks/<platform>/<id>/ with the layout above and a
PROVENANCE.md, plus a selector in ../automated-eval/macos_bench/probes/<ID>/task.json
("kind": "cdb", "pack_task", "category", "coding_tools": false, and
"browser": true for web tasks) and the
tree digest in pins.json. They join bench_v2.V2_TASKS only through an
amendment registered before their first trial.
Licence
FSL-1.1-MIT, under the benchmark's LICENSE. Copyright (c) 2026
Cua AI, Inc. The task files carry no per-file SPDX headers because their
content is pinned by digest; the folder-level LICENSE covers them. The copy
published in #4973 and #4978 was MIT; its tree digests are kept in pins.json
cdb_pack.mit_release_tree_sha256. Third-party software that the apps install is listed in
THIRD_PARTY_NOTICES.md; none of it is vendored.