Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

5.3 KB
Raw Blame

Cua Driver Bench task pack (public)

The multi-app tasks of the public Bench v2 task set, plus separately selectable web tasks. These run GUI-only (no coding tools) under the ids in the table; the harness reads them from here (CDB_TASKPACK, see ../automated-eval/macos_bench/cdb_adapter.py).

Pack task Bench id Apps
tasks/shared/cdb-s02 CDB-G02 Copperfield Mail, Signal Quay Chat and Daymark Calendar (Electron), LibreOffice Calc
tasks/shared/cdb-s03 CDB-G03 Ledger Lantern Mail (Electron), LibreOffice Calc, GnuCash
tasks/shared/cdb-s04 CDB-G04 Rillwatch Cases and Margin Lantern Manual (Electron), the Morrowglass Archive site in Chrome, a log file
tasks/shared/web-01 WEB-01 (separate) Fern signup in Chrome
tasks/shared/web-02 WEB-02 (separate) Pebble checkout in Chrome

WEB-01 and WEB-02 are Amendment 24.3 originals, not members of bench_v2.V2_TASKS. Select them explicitly with --tasks WEB-01 WEB-02; run_bench.WEB_TASKS lists them. They use a Node built-in-only loopback service, task-local routes and checks, and a fresh workspace-local Chrome profile. No internet or external assets are used.

The task manifest, v2-taskset.json, lists every Bench v2 task with its category, OS availability, conditions and split. A task kept out of this repository would run from a separate pack named by CDB_TASKPACK_HELDOUT, with the same layout.

Each task directory holds:

  • brief.md: what the agent is told;
  • task.cuabench.json: the manifest, with a sha256 for every input;
  • apps/: the task's Electron apps; fixture/: the starting data;
  • reset/: setup.py builds the workspace, verify.py checks it byte for byte;
  • platform/launch.macos.json: the launch descriptor the harness follows;
  • evaluator/: the checker and its oracle (final state only, every required check);
  • tests/: evaluator, reset and manifest tests; tools/: fixture and digest builders;
  • PROVENANCE.md: origin, prior art, review and licence.

Source and changes

The three cdb-s0* tasks were written by Cua in its private benchmark repository and are copied from revision 16a1937a79ff2ee2e48f5b5d9bb5e6f944119b5a, the revision every registered Bench v2 run used. The copy differs from that revision only in licence and provenance metadata:

  • the license field of every package.json and package-lock.json is now FSL-1.1-MIT (it was a proprietary licence reference, and MIT in #4973 and #4978);
  • PROVENANCE.md and the provenance block of task.cuabench.json describe the origin for a public reader, and each task's README.md no longer links to documents of the private repository;
  • the manifest digests are refreshed for those files, and the cdb-s04 oracle's pinned digests of the archive site's package.json and package-lock.json are updated to the relicensed files.

For those three tasks, briefs, apps, fixtures, reset scripts, launch descriptors, evaluators and the rest of the oracles are byte-identical to the source revision. The tree digests of the registered runs are kept in pins.json cdb_pack.registered_runs_tree_sha256; pins.json cdb_pack.tree_sha256 holds the digests of this copy, which the preflight checks.

Checks

From a task directory:

python3 -m unittest discover -s tests          # evaluator, reset and manifest tests
python3 tools/refresh_manifest_digests.py --check
(cd apps/<app> && node --test tests/*.test.js)  # app store tests; no install needed

After changing a task-owned input, run tools/refresh_manifest_digests.py, the tests above, and tools/validate_cdb.py from the harness in a VM, then update pins.json cdb_pack.tree_sha256. A change to a task that a registered run used needs a dated amendment before the next run.

For web tasks, shared helpers live in tasks/shared/web-common (not a task). After changing common.py, regenerate task-local evaluator helpers with python3 tasks/shared/web-common/vendor_evaluator.py. Run each web task's own tools/refresh_manifest_digests.py, then its --check. Both manifests enumerate shared runtime inputs; reset verification rehashes those inputs every trial, because a task tree pin alone does not cover sibling files. Refresh tree pins with python3 tools/check_pack_pins.py --write from the harness directory.

Adding tasks

New tasks go into tasks/<platform>/<id>/ with the layout above and a PROVENANCE.md, plus a selector in ../automated-eval/macos_bench/probes/<ID>/task.json ("kind": "cdb", "pack_task", "category", "coding_tools": false, and "browser": true for web tasks) and the tree digest in pins.json. They join bench_v2.V2_TASKS only through an amendment registered before their first trial.

Licence

FSL-1.1-MIT, under the benchmark's LICENSE. Copyright (c) 2026 Cua AI, Inc. The task files carry no per-file SPDX headers because their content is pinned by digest; the folder-level LICENSE covers them. The copy published in #4973 and #4978 was MIT; its tree digests are kept in pins.json cdb_pack.mit_release_tree_sha256. Third-party software that the apps install is listed in THIRD_PARTY_NOTICES.md; none of it is vendored.