Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

docs(spaces): reinforcement learning and evals: what Cua offers and its limits (#6481)

ddupont90c7bd9263a2
diff --git a/docs/content/docs/spaces/guides/reinforcement-learning.mdx b/docs/content/docs/spaces/guides/reinforcement-learning.mdx
new file mode 100644
index 0000000000..f5bbddb69d
--- /dev/null
+++ b/docs/content/docs/spaces/guides/reinforcement-learning.mdx
@@ -0,0 +1,88 @@
+---
+title: Reinforcement learning and evals
+description: 'What Cua offers for RL and evals on computers you own: Cua Bench environments with rewards, jobs on your pool, distributed training, and the limits to plan around.'
+---
+
+import { Callout } from 'fumadocs-ui/components/callout';
+
+Cua gives you two things for reinforcement learning and evals on hardware you
+already have: **environments** that score an agent, and **jobs** that run work
+on your computers. They are separate pieces that work together.
+
+## Environments: Cua Bench
+
+[Cua Bench](/cua-bench) runs verifiable computer-use tasks on desktop VMs: macOS,
+Windows and Linux. A task is a small module with four parts.
+
+| Part | What it does |
+| --- | --- |
+| Variants | The same goal with a different prompt, data, theme or OS. |
+| Setup | Prepares the environment and opens the apps. |
+| Oracle (optional) | One known way to succeed. A failing oracle means a broken task. |
+| Evaluator | Inspects the final state and returns rewards. It scores a person, an oracle or an agent the same way. |
+
+```text output
+dataset -> task -> variant -> session -> trajectory -> evaluator -> reward
+```
+
+The registry tasksets are listed at [cua.ai/cuabench/registry](https://cua.ai/cuabench/registry).
+To write your own, see [Write a task](/cua-bench/guides/write-a-task).
+
+For training, the `cua-bench-rl` package in the repository adds parallel
+environment workers (one server per environment), a multi-turn dataloader and
+replay buffer, and a Tinker GRPO trainer.
+
+## Jobs on your pool
+
+Computers you add to your pool with `cua host setup --share spaces` can run
+Python functions, each member in its own Space.
+
+```bash test="cli-shape" id="rl-run-jobs"
+cua run train.py::finetune
+cua run train.py::finetune --nodes 1:2 --accelerator gpu
+cua jobs
+cua jobs logs -f p-finetune-1a2b3c4d
+```
+
+A run keeps going when you disconnect or restart the daemon, keeps a log you can
+read from any offset, and can be cancelled or have its result saved. See
+[Your pool](/cua-cli/reference/cli/pool) for every option and
+[Long-running jobs](/spaces/guides/long-running-jobs) for jobs inside one Space.
+
+## Distributed training
+
+A run with `--nodes N` is a gang: each member is a rank, and the ranks exchange
+gradients. The repository has two examples in `examples/training`: an MLX LoRA
+fine-tune that runs on Apple silicon and on NVIDIA GPUs, and a PyTorch DDP
+example. If a computer is lost the pool restarts the gang, and the LoRA example
+resumes from its newest saved adapter.
+
+## Limits to plan around
+
+- **Distributed training depends on the link between computers.** Ranks wait for
+ each other on every gradient exchange. We measured the LoRA example on two
+ Macs over Wi-Fi: one Mac ran at 550 to 660 tokens per second, two together at
+ 880 (about 1.35 times). Exchanging gradients more often left two Macs only
+ slightly ahead of one. For small models, one computer can be as fast as
+ several. A faster link between the computers, such as Thunderbolt, should
+ help, but we have not measured it.
+- **Rollouts scale better than training.** Running many independent
+ environments, each in its own Space, does not need the computers to talk to
+ each other, so it is the better use of several machines today.
+- **Shared GPUs can abort a long step.** On Apple silicon, macOS can abort a
+ long Metal command buffer when the GPU is busy with other heavy work. The
+ attempt fails, the pool retries it, and the job resumes from its last saved
+ state.
+- **Linux and macOS Spaces only for `space_job`.** Windows environments run in
+ Cua Bench, but jobs inside a Windows Space are not supported yet.
+- **The environments, jobs and trainer are separate pieces.** There is no single
+ command that takes a Cua Bench task through to a trained model. You connect
+ them with the packages above.
+- **Scale numbers are not published yet.** We have not published results for
+ large numbers of parallel environments, so measure on your hardware first.
+
+<Callout type="info">
+ Cua does not rent computers. Everything here runs on computers you add: your
+ own Macs, Linux machines, or VMs in your cloud account. See
+ [Bring your own compute](/spaces/guides/bring-your-own-compute).
+</Callout>
diff --git a/docs/content/docs/spaces/meta.json b/docs/content/docs/spaces/meta.json
index 3cea9376a8..08925b5dbc 100644
--- a/docs/content/docs/spaces/meta.json
+++ b/docs/content/docs/spaces/meta.json
@@ -39,6 +39,7 @@
"guides/child-spaces",
"guides/keep-a-command-running",
"guides/long-running-jobs",
+ "guides/reinforcement-learning",
"guides/cua-ci",
"guides/warm-start",
"---Keep it working---",