Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Move Computer UI instructions from README to Cua documentation
James Murdzab357a9efd398
diff --git a/docs/content/docs/computer-sdk/computer-ui.mdx b/docs/content/docs/computer-sdk/computer-ui.mdxnew file mode 100644index 0000000000..e90e9f089f--- /dev/null+++ b/docs/content/docs/computer-sdk/computer-ui.mdx@@ -0,0 +1,78 @@+---+title: Computer UI+---++The computer module includes a Gradio UI for creating and sharing demonstration data. We make it easy for people to build community datasets for better computer use models with an upload to Huggingface feature.++```bash+# Install with UI support+pip install "cua-computer[ui]"+```++> **Note:** For precise control of the computer, we recommend using VNC or Screen Sharing instead of the Computer Gradio UI.++### Building and Sharing Demonstrations with Huggingface++Follow these steps to contribute your own demonstrations:++#### 1. Set up Huggingface Access++Set your HF_TOKEN in a .env file or in your environment variables:++```bash+# In .env file+HF_TOKEN=your_huggingface_token+```++#### 2. Launch the Computer UI++```python+# launch_ui.py+from computer.ui.gradio.app import create_gradio_ui+from dotenv import load_dotenv+load_dotenv('.env')++app = create_gradio_ui()+app.launch(share=False)+```++For examples, see [Computer UI Examples](https://github.com/trycua/cua/tree/main/examples/computer_ui_examples.py)++#### 3. Record Your Tasks++<details open>+<summary>View demonstration video</summary>+<video src="https://github.com/user-attachments/assets/de3c3477-62fe-413c-998d-4063e48de176" controls width="600"></video>+</details>++Record yourself performing various computer tasks using the UI.++#### 4. Save Your Demonstrations++<details open>+<summary>View demonstration video</summary>+<video src="https://github.com/user-attachments/assets/5ad1df37-026a-457f-8b49-922ae805faef" controls width="600"></video>+</details>++Save each task by picking a descriptive name and adding relevant tags (e.g., "office", "web-browsing", "coding").++#### 5. Record Additional Demonstrations++Repeat steps 3 and 4 until you have a good amount of demonstrations covering different tasks and scenarios.++#### 6. Upload to Huggingface++<details open>+<summary>View demonstration video</summary>+<video src="https://github.com/user-attachments/assets/c586d460-3877-4b5f-a736-3248886d2134" controls width="600"></video>+</details>++Upload your dataset to Huggingface by:+- Naming it as `{your_username}/{dataset_name}`+- Choosing public or private visibility+- Optionally selecting specific tags to upload only tasks with certain tags++#### Examples and Resources++- Example Dataset: [ddupont/test-dataset](https://huggingface.co/datasets/ddupont/test-dataset)+- Find Community Datasets: π [Browse CUA Datasets on Huggingface](https://huggingface.co/datasets?other=cua)\ No newline at end of filediff --git a/docs/content/docs/computer-sdk/meta.json b/docs/content/docs/computer-sdk/meta.jsonindex f632538bfa..92e1461237 100644--- a/docs/content/docs/computer-sdk/meta.json+++ b/docs/content/docs/computer-sdk/meta.json@@ -4,6 +4,7 @@"pages": ["computers","commands",+ "computer-ui","sandboxed-python"]}diff --git a/libs/python/computer/README.md b/libs/python/computer/README.mdindex a75c4fe352..78c66566ca 100644--- a/libs/python/computer/README.md+++ b/libs/python/computer/README.md@@ -63,82 +63,4 @@ The `cua-computer` PyPi package pulls automatically the latest executable versioRefer to this notebook for a step-by-step guide on how to use the Computer-Use Interface (CUI):-- [Computer-Use Interface (CUI)](https://github.com/trycua/cua/blob/main/notebooks/computer_nb.ipynb)--## Using the Gradio Computer UI--The computer module includes a Gradio UI for creating and sharing demonstration data. We make it easy for people to build community datasets for better computer use models with an upload to Huggingface feature.--```bash-# Install with UI support-pip install "cua-computer[ui]"-```--> **Note:** For precise control of the computer, we recommend using VNC or Screen Sharing instead of the Computer Gradio UI.--### Building and Sharing Demonstrations with Huggingface--Follow these steps to contribute your own demonstrations:--#### 1. Set up Huggingface Access--Set your HF_TOKEN in a .env file or in your environment variables:--```bash-# In .env file-HF_TOKEN=your_huggingface_token-```--#### 2. Launch the Computer UI--```python-# launch_ui.py-from computer.ui.gradio.app import create_gradio_ui-from dotenv import load_dotenv-load_dotenv('.env')--app = create_gradio_ui()-app.launch(share=False)-```--For examples, see [Computer UI Examples](https://github.com/trycua/cua/tree/main/examples/computer_ui_examples.py)--#### 3. Record Your Tasks--<details open>-<summary>View demonstration video</summary>-<video src="https://github.com/user-attachments/assets/de3c3477-62fe-413c-998d-4063e48de176" controls width="600"></video>-</details>--Record yourself performing various computer tasks using the UI.--#### 4. Save Your Demonstrations--<details open>-<summary>View demonstration video</summary>-<video src="https://github.com/user-attachments/assets/5ad1df37-026a-457f-8b49-922ae805faef" controls width="600"></video>-</details>--Save each task by picking a descriptive name and adding relevant tags (e.g., "office", "web-browsing", "coding").--#### 5. Record Additional Demonstrations--Repeat steps 3 and 4 until you have a good amount of demonstrations covering different tasks and scenarios.--#### 6. Upload to Huggingface--<details open>-<summary>View demonstration video</summary>-<video src="https://github.com/user-attachments/assets/c586d460-3877-4b5f-a736-3248886d2134" controls width="600"></video>-</details>--Upload your dataset to Huggingface by:-- Naming it as `{your_username}/{dataset_name}`-- Choosing public or private visibility-- Optionally selecting specific tags to upload only tasks with certain tags--#### Examples and Resources--- Example Dataset: [ddupont/test-dataset](https://huggingface.co/datasets/ddupont/test-dataset)-- Find Community Datasets: π [Browse CUA Datasets on Huggingface](https://huggingface.co/datasets?other=cua)-+- [Computer-Use Interface (CUI)](https://github.com/trycua/cua/blob/main/notebooks/computer_nb.ipynb)\ No newline at end of file