Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
More organization, create typescript nad python doc files
Morgan Dean72a7c13ab646
diff --git a/docs/content/docs/home/libraries/computer/index.mdx b/docs/content/docs/home/libraries/computer/index.mdxnew file mode 100644index 0000000000..56d337c8f4--- /dev/null+++ b/docs/content/docs/home/libraries/computer/index.mdx@@ -0,0 +1,32 @@+---+title: Computer+---++**@cua/computer** is a Computer-Use Interface (CUI) framework powering Cua for interacting with local macOS and Linux sandboxes. It's PyAutoGUI-compatible and pluggable with any AI agent systems (Cua, Langchain, CrewAI, AutoGen). Computer relies on [Lume](https://github.com/trycua/lume) for creating and managing sandbox environments.++<div align="center">+ <img src="/img/computer.png" />+</div>++## Available Implementations++The Computer library is available in multiple programming languages:++- [Python Implementation](/home/libraries/computer/python) - For Python developers+- [TypeScript Implementation](/home/libraries/computer/typescript) - For TypeScript/JavaScript developers++Both implementations offer equivalent functionality with language-specific syntax and conventions.++## Key Features++- Create and manage virtual machine sandboxes+- Take screenshots of the virtual machine+- Control mouse movements and clicks+- Simulate keyboard input+- Manage clipboard content+- Interact with the operating system interface+- Support for macOS and Linux environments++## Getting Started++Select your preferred language implementation from the links above to get started with installation instructions and usage examples.diff --git a/docs/content/docs/home/libraries/computer.mdx b/docs/content/docs/home/libraries/computer/python.mdxsimilarity index 85%rename from docs/content/docs/home/libraries/computer.mdxrename to docs/content/docs/home/libraries/computer/python.mdxindex d10cb255df..5300c86ef7 100644--- a/docs/content/docs/home/libraries/computer.mdx+++ b/docs/content/docs/home/libraries/computer/python.mdx@@ -1,5 +1,5 @@----title: Computer+title: Computer (Python)---<div@@ -38,13 +38,7 @@ title: Computer</a></div>-**cua-computer** is a Computer-Use Interface (CUI) framework powering Cua for interacting with local macOS and Linux sandboxes, PyAutoGUI-compatible, and pluggable with any AI agent systems (Cua, Langchain, CrewAI, AutoGen). Computer relies on [Lume](https://github.com/trycua/lume) for creating and managing sandbox environments.--### Get started with Computer--<div align="center">- <img src="/img/computer.png" />-</div>+## Python API```pythonfrom computer import Computer@@ -74,13 +68,13 @@ finally:## Install-To install the Computer-Use Interface (CUI):+To install the Computer-Use Interface (CUI) for Python:```bash-pip install "cua-computer[all]"+pip install "@cua/computer[all]"```-The `cua-computer` PyPi package pulls automatically the latest executable version of Lume through [pylume](https://github.com/trycua/pylume).+The `@cua/computer` PyPi package pulls automatically the latest executable version of Lume through [pylume](https://github.com/trycua/pylume).## Run@@ -94,7 +88,7 @@ The computer module includes a Gradio UI for creating and sharing demonstration```bash# Install with UI support-pip install "cua-computer[ui]"+pip install "@cua/computer[ui]"```> **Note:** For precise control of the computer, we recommend using VNC or Screen Sharing instead of the Computer Gradio UI.@@ -176,4 +170,4 @@ Upload your dataset to Huggingface by:#### Examples and Resources- Example Dataset: [ddupont/test-dataset](https://huggingface.co/datasets/ddupont/test-dataset)-- Find Community Datasets: π [Browse CUA Datasets on Huggingface](https://huggingface.co/datasets?other=cua)+- Find Community Datasets: π [Browse CUA Datasets on Huggingface](https://huggingface.co/datasets?other=cua)\ No newline at end of filediff --git a/docs/content/docs/home/libraries/computer/typescript.mdx b/docs/content/docs/home/libraries/computer/typescript.mdxnew file mode 100644index 0000000000..10a6b7cb4d--- /dev/null+++ b/docs/content/docs/home/libraries/computer/typescript.mdx@@ -0,0 +1,188 @@+---+title: Computer (TypeScript)+---++<div+ align="center"+ style={{+ display: 'flex',+ gap: '10px',+ margin: '0 auto',+ width: '100%',+ justifyContent: 'center',+ }}+>+ <a href="#">+ <img+ src="https://img.shields.io/badge/TypeScript-3178C6?logo=typescript&logoColor=white&labelColor=3178C6"+ alt="TypeScript"+ />+ </a>+ <a href="#">+ <img+ src="https://img.shields.io/badge/macOS-000000?logo=apple&logoColor=F0F0F0"+ alt="macOS"+ />+ </a>+ <a href="https://discord.com/invite/mVnXXpdE85">+ <img+ src="https://img.shields.io/badge/Discord-%235865F2.svg?&logo=discord&logoColor=white"+ alt="Discord"+ />+ </a>+ <a href="https://www.npmjs.com/package/@cua/computer">+ <img+ src="https://img.shields.io/npm/v/@cua/computer?color=3178C6"+ alt="npm"+ />+ </a>+</div>++## TypeScript API++```typescript+import { Computer } from '@cua/computer';++const main = async () => {+ const computer = new Computer({+ osType: "macos",+ display: "1024x768",+ memory: "8GB",+ cpu: "4"+ });++ try {+ await computer.run();++ const screenshot = await computer.interface.screenshot();+ // Save screenshot using fs or other method+ require('fs').writeFileSync('screenshot.png', screenshot);++ await computer.interface.moveCursor(100, 100);+ await computer.interface.leftClick();+ await computer.interface.rightClick(300, 300);+ await computer.interface.doubleClick(400, 400);++ await computer.interface.type("Hello, World!");+ await computer.interface.pressKey("enter");++ await computer.interface.setClipboard("Test clipboard");+ const content = await computer.interface.copyToClipboard();+ console.log(`Clipboard content: ${content}`);+ } finally {+ await computer.stop();+ }+};++main().catch(console.error);+```++## Install++To install the Computer-Use Interface (CUI) for TypeScript:++```bash+npm install @cua/computer+# or+yarn add @cua/computer+```++The TypeScript package automatically handles the necessary dependencies for working with Lume.++## Provider System++The TypeScript implementation includes a flexible provider system:++- **Lume Provider**: For local macOS and Linux sandboxes+- **Lumier Provider**: For Docker-based containers+- **Cloud Provider**: For remote cloud-based VMs++```typescript+// Using specific provider options+import { Computer } from '@cua/computer';++const computer = new Computer({+ osType: "macos",+ provider: "lume",+ providerOptions: {+ ephemeral: true,+ sharedDirectories: [+ { hostPath: "/path/to/share", tag: "shared", readOnly: true }+ ]+ }+});+```++## Interface System++The TypeScript implementation provides a comprehensive interface system for interacting with virtual machines:++```typescript+// Example of using the interface+import { Computer } from '@cua/computer';++const main = async () => {+ const computer = new Computer({ osType: "macos" });+ await computer.run();++ // Screenshot and image operations+ const screenshot = await computer.interface.screenshot();++ // Mouse operations+ await computer.interface.moveCursor(100, 100);+ await computer.interface.leftClick();++ // Keyboard operations+ await computer.interface.type("Hello from TypeScript!");+ await computer.interface.hotkey(["command", "a"]);++ // Clipboard operations+ await computer.interface.setClipboard("Clipboard content");+ const content = await computer.interface.copyToClipboard();++ await computer.stop();+};+```++## Advanced Usage++### Cloud Provider++For using the cloud provider with WebSocket communication:++```typescript+import { Computer } from '@cua/computer';++const computer = new Computer({+ osType: "macos",+ provider: "cloud",+ providerOptions: {+ apiKey: "your-api-key",+ endpoint: "wss://your-cloud-endpoint.com"+ }+});++await computer.run();+// Use computer.interface methods as usual+await computer.stop();+```++### Telemetry++The TypeScript implementation includes built-in telemetry tracking:++```typescript+import { TelemetryManager } from '@cua/computer';++// Disable telemetry if needed+TelemetryManager.getInstance().disable();++// Or track custom events+TelemetryManager.getInstance().trackEvent('custom_event', {+ property: 'value'+});+```++## Examples++For more examples and detailed usage, refer to the [TypeScript examples](https://github.com/trycua/cua/tree/main/examples/typescript) in the repository.\ No newline at end of file