Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
docs(cua-s1): document cua-s1-forms out-of-catalogue confident skips (#4321)
Francesco Bonaccid1736c0b6dbb
diff --git a/libs/cua-s1/MODEL_CARD.md b/libs/cua-s1/MODEL_CARD.mdindex 78503f8d9b..b8023d186c 100644--- a/libs/cua-s1/MODEL_CARD.md+++ b/libs/cua-s1/MODEL_CARD.md@@ -51,6 +51,29 @@ consequences; bypassing access controls, consent, rate limits, or servicepolicies; treating model output or apparent task completion as proof thatan action was correct or successful.+**Limitations:** the published `cua-s1-forms` checkpoint depends on its+labels. It keys mainly on the element (form) label matching the 55-concept+`form_labels` vocabulary in `cua_s1/concepts.py`, and to a lesser extent on+the document labels. Outside that vocabulary it does not abstain and does not+return a low-confidence answer. It returns a **high-confidence `skip`**. An+independent evaluation (#3978) measured:++| Evaluation set | Accuracy | Behavior |+| --- | --- | --- |+| In-distribution `cua_s1.synth` episodes (3,070 decisions) | 97.5% (2,993/3,070) | Matches the training distribution |+| Held-out forms with out-of-catalogue labels (41 decisions over 5 forms: logistics, hardware, DevOps, veterinary, and a Chinese-language form) | 29.3% (12/41) | 36 of 41 predictions were `skip`, with mean confidence 0.974 |++Relabeling the same held-out forms with in-catalogue wording restored+accuracy (11/11). Replacing only the element labels with unseen wording+dropped it to 25%, even when every document label stayed in the catalogue.+`check` and `click` decisions that do not depend on a document entity also+failed on the held-out forms. An executor that treats `skip` as "nothing to+do here" can therefore silently no-op an entire out-of-scope form. Do not+read a high-confidence `skip` as evidence that a form is in scope or+complete. Before acting, check element labels against the catalogue or+verify form completion independently. These held-out figures come from a+small external set and are not a benchmark claim.+## Checkpoint: cua-s1-4b-0.1**Base model:** a LoRA (low-rank adapter) fine-tune on top of the frozen,diff --git a/libs/cua-s1/README.md b/libs/cua-s1/README.mdindex 28c276e003..1c4d8c7649 100644--- a/libs/cua-s1/README.md+++ b/libs/cua-s1/README.md@@ -136,7 +136,12 @@ Layouts and loaders:a pickle `cua-s1-forms.pt`, which `cua_s1` refuses to load by design (see#3977). If you pin an older revision, pass `--exclude "*.pt"`. Unpickling acheckpoint can run arbitrary code, so never open a `.pt` copy with- `torch.load` or `pickle`.+ `torch.load` or `pickle`. The checkpoint depends on the 55-concept label+ catalogue in `cua_s1/concepts.py`. On forms whose labels fall outside it,+ it answers a confident `skip` instead of abstaining: 29.3% held-out+ accuracy vs 97.5% in-distribution (#3978). See the+ [model card](MODEL_CARD.md#checkpoint-cua-s1-form-v0) before relying on+ `skip`.- The Apache-2.0 licenses on `cua-s1-4b-0.2` and `cua-s1-4b-0.1` cover onlythe adapters. The base model is governed by its own license and isdownloaded from Qwen's repository, not redistributed by Cua.