Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Form filling usecase
Sarina Lib05528de3b3b
diff --git a/docs/content/docs/example-usecases/deep-research.mdx b/docs/content/docs/example-usecases/deep-research.mdxdeleted file mode 100644index 8081fb33f1..0000000000--- a/docs/content/docs/example-usecases/deep-research.mdx+++ /dev/null@@ -1,30 +0,0 @@-----title: Deep Research-description: bleh------import { Monitor, Code, BookOpen } from 'lucide-react';--# Welcome!--Cua is a framework for automating Windows, Mac, and Linux apps powered by computer-using agents (CUAs).--Cua makes every stage of computer-using agent development simple:--- **Development**: Use any LLM provider with liteLLM. The agent SDK makes multiple agent loop providers, trajectory tracing, caching, and budget management easy-- **Containerization**: cua offers Docker containers pre-installed with everything needed for AI-powered RPA-- **Deployment**: cua cloud gives you a production-ready cloud environment for your assistants--<div className="grid grid-cols-1 md:grid-cols-2 gap-6 mt-8">- <Card icon={<Monitor />} href="/quickstart-ui" title="Quickstart (UI)">- Try the cua Agent UI in your browser—no coding required.- </Card>- <Card icon={<Code />} href="/quickstart-devs" title="Quickstart (Developers)">- Build with Python—full SDK and agent code examples.- </Card>-</div>-<div className="grid grid-cols-1 gap-6 mt-6">- <Card icon={<BookOpen />} href="/libraries/agent" title="API Reference">- Explore the agent SDK and APIs- </Card>-</div>diff --git a/docs/content/docs/example-usecases/form-filling.mdx b/docs/content/docs/example-usecases/form-filling.mdxnew file mode 100644index 0000000000..036231812a--- /dev/null+++ b/docs/content/docs/example-usecases/form-filling.mdx@@ -0,0 +1,491 @@+---+title: Form Filling+description: Enhance and Automate Interactions Between Form Filling and Local File Systems+---++import { EditableCodeBlock, EditableValue, S } from '@/components/editable-code-block';+import { Tab, Tabs } from 'fumadocs-ui/components/tabs';++## Overview++Cua can be used to automate interactions between form filling and local file systems over any operating system. Cua let's you interact with all the elements of a web page and local file systems to integrate between the two.++This preset usecase uses [Computer Agent](/agent-sdk/computer-agent) to interact with a web page and local file systems along with [Agent Loops](/agent-sdk/agent-loops) to run the agent in a loop with message history.++## Quickstart++Create a `requirements.txt` file with the following dependencies:+```text+cua-agent+cua-computer+python-dotenv>=1.0.0+```++And install:++```bash+pip install -r requirements.txt+```++Create a `.env` file with the following environment variables:++```text+ANTHROPIC_API_KEY=your-api-key+CUA_API_KEY=sk_cua-api01...+```++Select the environment you want to run the code in (*click on the underlined values in the code to edit them directly!*):++<Tabs items={['☁️ Cloud', '🐳 Docker', '🍎 Lume', '🪟 Windows Sandbox']}>+ <Tab value="☁️ Cloud">++<EditableCodeBlock+ key="cloud-tab"+ lang="python"+ defaultValues={{+ "container-name": "m-linux-...",+ "api_key": "sk_cua-api01..."+ }}+>+{`import asyncio+import logging+import os+import signal+import traceback++from agent import ComputerAgent+from computer import Computer, VMProviderType+from dotenv import load_dotenv++logging.basicConfig(level=logging.INFO)+logger = logging.getLogger(__name__)+++def handle_sigint(sig, frame):+ print("\\n\\nExecution interrupted by user. Exiting gracefully...")+ exit(0)+++async def fill_application():+ try:+ async with Computer(+ os_type="linux",+ provider_type=VMProviderType.CLOUD,+ name="`}<EditableValue placeholder="container-name" />{`",+ api_key="`}<EditableValue placeholder="api_key" />{`",+ verbosity=logging.INFO,+ ) as computer:++ agent = ComputerAgent(+ model="anthropic/claude-3-5-sonnet-20241022",+ tools=[computer],+ only_n_most_recent_images=3,+ verbosity=logging.INFO,+ trajectory_dir="trajectories",+ use_prompt_caching=True,+ max_trajectory_budget=5.0,+ )++ tasks = [+ "Visit https://www.overleaf.com/latex/templates/jakes-resume/syzfjbzwjncs.pdf and download the pdf.",+ "Visit https://form.jotform.com/252881246782264 and fill the form from the information in the pdf."+ ]++ history = []++ for i, task in enumerate(tasks, 1):+ print(f"\\n[Task {i}/{len(tasks)}] {task}")++ # Add user message to history+ history.append({"role": "user", "content": task})++ # Run agent with conversation history+ async for result in agent.run(history, stream=False):+ history += result.get("output", [])++ # Print output for debugging+ for item in result.get("output", []):+ if item.get("type") == "message":+ content = item.get("content", [])+ for content_part in content:+ if content_part.get("text"):+ logger.info(f"Agent: {content_part.get('text')}")+ elif item.get("type") == "computer_call":+ action = item.get("action", {})+ action_type = action.get("type", "")+ logger.debug(f"Computer Action: {action_type}")++ print(f"✅ Task {i}/{len(tasks)} completed")++ print("\\n🎉 All tasks completed successfully!")++ except Exception as e:+ logger.error(f"Error in fill_application: {e}")+ traceback.print_exc()+ raise+++def main():+ try:+ load_dotenv()++ if "ANTHROPIC_API_KEY" not in os.environ:+ raise RuntimeError(+ "Please set the ANTHROPIC_API_KEY environment variable.\\n"+ "You can add it to a .env file in the project root."+ )++ if "CUA_API_KEY" not in os.environ:+ raise RuntimeError(+ "Please set the CUA_API_KEY environment variable.\\n"+ "You can add it to a .env file in the project root."+ )++ signal.signal(signal.SIGINT, handle_sigint)++ asyncio.run(fill_application())++ except Exception as e:+ logger.error(f"Error running automation: {e}")+ traceback.print_exc()+++if __name__ == "__main__":+ main()`}+</EditableCodeBlock>++ </Tab>+ <Tab value="🍎 Lume">++<EditableCodeBlock+ key="lume-tab"+ lang="python"+ defaultValues={{+ "container-name": "macos-sequoia-cua:latest"+ }}+>+{`import asyncio+import logging+import os+import signal+import traceback++from agent import ComputerAgent+from computer import Computer, VMProviderType+from dotenv import load_dotenv++logging.basicConfig(level=logging.INFO)+logger = logging.getLogger(__name__)+++def handle_sigint(sig, frame):+ print("\\n\\nExecution interrupted by user. Exiting gracefully...")+ exit(0)+++async def fill_application():+ try:+ async with Computer(+ os_type="macos",+ provider_type=VMProviderType.LUME,+ name="`}<EditableValue placeholder="container-name" />{`",+ verbosity=logging.INFO,+ ) as computer:++ agent = ComputerAgent(+ model="anthropic/claude-3-5-sonnet-20241022",+ tools=[computer],+ only_n_most_recent_images=3,+ verbosity=logging.INFO,+ trajectory_dir="trajectories",+ use_prompt_caching=True,+ max_trajectory_budget=5.0,+ )++ tasks = [+ "Visit https://www.overleaf.com/latex/templates/jakes-resume/syzfjbzwjncs.pdf and download the pdf.",+ "Visit https://form.jotform.com/252881246782264 and fill the form from the information in the pdf."+ ]++ history = []++ for i, task in enumerate(tasks, 1):+ print(f"\\n[Task {i}/{len(tasks)}] {task}")++ # Add user message to history+ history.append({"role": "user", "content": task})++ # Run agent with conversation history+ async for result in agent.run(history, stream=False):+ history += result.get("output", [])++ # Print output for debugging+ for item in result.get("output", []):+ if item.get("type") == "message":+ content = item.get("content", [])+ for content_part in content:+ if content_part.get("text"):+ logger.info(f"Agent: {content_part.get('text')}")+ elif item.get("type") == "computer_call":+ action = item.get("action", {})+ action_type = action.get("type", "")+ logger.debug(f"Computer Action: {action_type}")++ print(f"✅ Task {i}/{len(tasks)} completed")++ print("\\n🎉 All tasks completed successfully!")++ except Exception as e:+ logger.error(f"Error in fill_application: {e}")+ traceback.print_exc()+ raise+++def main():+ try:+ load_dotenv()++ if "ANTHROPIC_API_KEY" not in os.environ:+ raise RuntimeError(+ "Please set the ANTHROPIC_API_KEY environment variable.\\n"+ "You can add it to a .env file in the project root."+ )++ signal.signal(signal.SIGINT, handle_sigint)++ asyncio.run(fill_application())++ except Exception as e:+ logger.error(f"Error running automation: {e}")+ traceback.print_exc()+++if __name__ == "__main__":+ main()`}+</EditableCodeBlock>++ </Tab>+ <Tab value="🪟 Windows Sandbox">++<EditableCodeBlock+ key="windows-tab"+ lang="python"+ defaultValues={{}}+>+{`import asyncio+import logging+import os+import signal+import traceback++from agent import ComputerAgent+from computer import Computer, VMProviderType+from dotenv import load_dotenv++logging.basicConfig(level=logging.INFO)+logger = logging.getLogger(__name__)+++def handle_sigint(sig, frame):+ print("\\n\\nExecution interrupted by user. Exiting gracefully...")+ exit(0)+++async def fill_application():+ try:+ async with Computer(+ os_type="windows",+ provider_type=VMProviderType.WINDOWS_SANDBOX,+ verbosity=logging.INFO,+ ) as computer:++ agent = ComputerAgent(+ model="anthropic/claude-3-5-sonnet-20241022",+ tools=[computer],+ only_n_most_recent_images=3,+ verbosity=logging.INFO,+ trajectory_dir="trajectories",+ use_prompt_caching=True,+ max_trajectory_budget=5.0,+ )++ tasks = [+ "Visit https://www.overleaf.com/latex/templates/jakes-resume/syzfjbzwjncs.pdf and download the pdf.",+ "Visit https://form.jotform.com/252881246782264 and fill the form from the information in the pdf."+ ]++ history = []++ for i, task in enumerate(tasks, 1):+ print(f"\\n[Task {i}/{len(tasks)}] {task}")++ # Add user message to history+ history.append({"role": "user", "content": task})++ # Run agent with conversation history+ async for result in agent.run(history, stream=False):+ history += result.get("output", [])++ # Print output for debugging+ for item in result.get("output", []):+ if item.get("type") == "message":+ content = item.get("content", [])+ for content_part in content:+ if content_part.get("text"):+ logger.info(f"Agent: {content_part.get('text')}")+ elif item.get("type") == "computer_call":+ action = item.get("action", {})+ action_type = action.get("type", "")+ logger.debug(f"Computer Action: {action_type}")++ print(f"✅ Task {i}/{len(tasks)} completed")++ print("\\n🎉 All tasks completed successfully!")++ except Exception as e:+ logger.error(f"Error in fill_application: {e}")+ traceback.print_exc()+ raise+++def main():+ try:+ load_dotenv()++ if "ANTHROPIC_API_KEY" not in os.environ:+ raise RuntimeError(+ "Please set the ANTHROPIC_API_KEY environment variable.\\n"+ "You can add it to a .env file in the project root."+ )++ signal.signal(signal.SIGINT, handle_sigint)++ asyncio.run(fill_application())++ except Exception as e:+ logger.error(f"Error running automation: {e}")+ traceback.print_exc()+++if __name__ == "__main__":+ main()`}+</EditableCodeBlock>++ </Tab>+ <Tab value="🐳 Docker">++<EditableCodeBlock+ key="docker-tab"+ lang="python"+ defaultValues={{+ "container-name": "trycua/cua-ubuntu:latest"+ }}+>+{`import asyncio+import logging+import os+import signal+import traceback++from agent import ComputerAgent+from computer import Computer, VMProviderType+from dotenv import load_dotenv++logging.basicConfig(level=logging.INFO)+logger = logging.getLogger(__name__)+++def handle_sigint(sig, frame):+ print("\\n\\nExecution interrupted by user. Exiting gracefully...")+ exit(0)+++async def fill_application():+ try:+ async with Computer(+ os_type="linux",+ provider_type=VMProviderType.DOCKER,+ name="`}<EditableValue placeholder="container-name" />{`",+ verbosity=logging.INFO,+ ) as computer:++ agent = ComputerAgent(+ model="anthropic/claude-3-5-sonnet-20241022",+ tools=[computer],+ only_n_most_recent_images=3,+ verbosity=logging.INFO,+ trajectory_dir="trajectories",+ use_prompt_caching=True,+ max_trajectory_budget=5.0,+ )++ tasks = [+ "Visit https://www.overleaf.com/latex/templates/jakes-resume/syzfjbzwjncs.pdf and download the pdf.",+ "Visit https://form.jotform.com/252881246782264 and fill the form from the information in the pdf."+ ]++ history = []++ for i, task in enumerate(tasks, 1):+ print(f"\\n[Task {i}/{len(tasks)}] {task}")++ # Add user message to history+ history.append({"role": "user", "content": task})++ # Run agent with conversation history+ async for result in agent.run(history, stream=False):+ history += result.get("output", [])++ # Print output for debugging+ for item in result.get("output", []):+ if item.get("type") == "message":+ content = item.get("content", [])+ for content_part in content:+ if content_part.get("text"):+ logger.info(f"Agent: {content_part.get('text')}")+ elif item.get("type") == "computer_call":+ action = item.get("action", {})+ action_type = action.get("type", "")+ logger.debug(f"Computer Action: {action_type}")++ print(f"✅ Task {i}/{len(tasks)} completed")++ print("\\n🎉 All tasks completed successfully!")++ except Exception as e:+ logger.error(f"Error in fill_application: {e}")+ traceback.print_exc()+ raise+++def main():+ try:+ load_dotenv()++ if "ANTHROPIC_API_KEY" not in os.environ:+ raise RuntimeError(+ "Please set the ANTHROPIC_API_KEY environment variable.\\n"+ "You can add it to a .env file in the project root."+ )++ signal.signal(signal.SIGINT, handle_sigint)++ asyncio.run(fill_application())++ except Exception as e:+ logger.error(f"Error running automation: {e}")+ traceback.print_exc()+++if __name__ == "__main__":+ main()`}+</EditableCodeBlock>++ </Tab>+</Tabs>++## Next Steps++- Learn more about [Cua computers](/computer-sdk/computers) and [computer commands](/computer-sdk/commands)+- Read about [Agent loops](/agent-sdk/agent-loops), [tools](/agent-sdk/custom-tools), and [supported model providers](/agent-sdk/supported-model-providers/)+- Experiment with different [Models and Providers](/agent-sdk/supported-model-providers/)diff --git a/docs/content/docs/example-usecases/job-application-filling.mdx b/docs/content/docs/example-usecases/job-application-filling.mdxdeleted file mode 100644index 7b2550d233..0000000000--- a/docs/content/docs/example-usecases/job-application-filling.mdx+++ /dev/null@@ -1,96 +0,0 @@-----title: Job Application Filling-description: Automate job applications with CUA------import { EditableCodeBlock, EditableValue, S } from '@/components/editable-code-block';-import { Tab, Tabs } from 'fumadocs-ui/components/tabs';--# Job Application Automation--Fill out the form below to see how CUA can automate your job application process. Click on the underlined values in the code to edit them directly!--<Tabs items={["Python", "curl"]}>- <Tab value="Python">--<EditableCodeBlock- lang="python"- defaultValues={{- "api_key": "your-api-key",- "api_base": "https://api.cua.ai"- }}->-{`import os-import asyncio-from computer.providers.cloud.provider import CloudProvider--async def main():- api_key = "`}<EditableValue placeholder="api_key" />{`"- # Optional: point to a different API base- os.environ["CUA_API_BASE"] = "`}<EditableValue placeholder="api_base" />{`"-- provider = CloudProvider(api_key=api_key, verbose=False)- async with provider:- vms = await provider.list_vms()- for vm in vms:- print({- "name": vm["name"],- "status": vm["status"],- "api_url": vm.get("api_url"),- "vnc_url": vm.get("vnc_url"),- })--if __name__ == "__main__":- asyncio.run(main())`}-</EditableCodeBlock>-- </Tab>- <Tab value="curl">--<EditableCodeBlock- lang="bash"- defaultValues={{- "api_key": "your-api-key",- "api_base": "https://api.cua.ai"- }}->-{`curl -H "Authorization: Bearer `}<EditableValue placeholder="api_key" />{`" \\- "`}<EditableValue placeholder="api_base" />{`/v1/vms"`}-</EditableCodeBlock>-- Responses:- - 200: Array of minimal VM objects with fields `{ name, password, status }`- - 401: Unauthorized (missing/invalid API key)-- ```json- [- {- "name": "s-windows-x4snp46ebf",- "password": "49b8daa3",- "status": "running"- }- ]- ```-- Status values:-- - `pending`: VM deployment in progress- - `running`: VM is active and accessible- - `stopped`: VM is stopped but not terminated- - `terminated`: VM has been permanently destroyed- - `failed`: VM deployment or operation failed-- </Tab>-</Tabs>--## How It Works--1. Click on the underlined values in the code above to edit them-2. The code updates in real-time as you type-3. Copy the customized code to use with CUA--## Features--- **Dynamic Form Filling**: Automatically fills out all required fields-- **Cross-Platform**: Works on Windows, macOS, and Linux-- **Customizable**: Easily modify the code to match any job application form\ No newline at end of filediff --git a/docs/content/docs/example-usecases/meta.json b/docs/content/docs/example-usecases/meta.jsonindex 63bf27ae6c..8d3e409294 100644--- a/docs/content/docs/example-usecases/meta.json+++ b/docs/content/docs/example-usecases/meta.json@@ -2,7 +2,6 @@"title": "Example Use Cases","description": "Real-world examples of building with cua","pages": [- "deep-research",- "job-application-filling"+ "form-filling"]}diff --git a/docs/content/docs/quickstart-devs.mdx b/docs/content/docs/quickstart-devs.mdxindex 8a9adea7b6..4bd5b9ab6b 100644--- a/docs/content/docs/quickstart-devs.mdx+++ b/docs/content/docs/quickstart-devs.mdx@@ -306,3 +306,4 @@ Learn more about agents in [Agent Loops](/agent-sdk/agent-loops) and available m- Learn more about [Cua computers](/computer-sdk/computers) and [computer commands](/computer-sdk/commands)- Read about [Agent loops](/agent-sdk/agent-loops), [tools](/agent-sdk/custom-tools), and [supported model providers](/agent-sdk/supported-model-providers/)- Join our [Discord community](https://discord.com/invite/mVnXXpdE85) for help+- Try out [Form Filling](/example-usecases/form-filling) preset usecasediff --git a/docs/src/components/editable-code-block.tsx b/docs/src/components/editable-code-block.tsxindex 348d1e4ae6..e8cda746e6 100644--- a/docs/src/components/editable-code-block.tsx+++ b/docs/src/components/editable-code-block.tsx@@ -99,29 +99,68 @@ export function EditableValue({const value = values[placeholder] ?? defaultValue;const spanRef = React.useRef<HTMLSpanElement>(null);const placeholderSpanRef = React.useRef<HTMLSpanElement>(null);+ const inputRef = React.useRef<HTMLInputElement>(null);const [measuredWidth, setMeasuredWidth] = React.useState(0);const [placeholderWidth, setPlaceholderWidth] = React.useState(0);+ const [isHovered, setIsHovered] = React.useState(false);+ const [tooltipPosition, setTooltipPosition] = React.useState({ top: 0, left: 0 });+ const [isVisible, setIsVisible] = React.useState(false);++ // Observe visibility changes to trigger remeasurement+ React.useEffect(() => {+ if (!inputRef.current) return;++ const observer = new IntersectionObserver(+ (entries) => {+ entries.forEach((entry) => {+ setIsVisible(entry.isIntersecting);+ });+ },+ { threshold: 0.01 }+ );++ observer.observe(inputRef.current);++ return () => {+ observer.disconnect();+ };+ }, []);// Measure the actual text width using a hidden spanReact.useEffect(() => {- if (spanRef.current) {+ if (spanRef.current && isVisible) {setMeasuredWidth(spanRef.current.offsetWidth);}- }, [value]);+ }, [value, isVisible]);- // Measure placeholder width once on mount+ // Measure placeholder width when visibleReact.useEffect(() => {- if (placeholderSpanRef.current) {+ if (placeholderSpanRef.current && isVisible) {setPlaceholderWidth(placeholderSpanRef.current.offsetWidth);}- }, [placeholder]);+ }, [placeholder, isVisible]);++ // Update tooltip position when hovered+ React.useEffect(() => {+ if (isHovered && inputRef.current) {+ const rect = inputRef.current.getBoundingClientRect();+ setTooltipPosition({+ top: rect.top - 28,+ left: rect.left + rect.width / 2,+ });+ }+ }, [isHovered]);const inputWidth = explicitWidth? `${explicitWidth}ch`- : `${Math.max(placeholderWidth, measuredWidth, 50)}px`;+ : `${Math.max(placeholderWidth, measuredWidth, 80)}px`;return (- <span style={{ display: 'inline', whiteSpace: 'nowrap', position: 'relative' }}>+ <span+ style={{ display: 'inline', whiteSpace: 'nowrap', position: 'relative' }}+ onMouseEnter={() => setIsHovered(true)}+ onMouseLeave={() => setIsHovered(false)}+ >{/* Hidden span to measure current value width */}<spanref={spanRef}@@ -152,7 +191,30 @@ export function EditableValue({{placeholder}</span>+ {/* Tooltip */}+ <span+ style={{+ position: 'fixed',+ top: tooltipPosition.top,+ left: tooltipPosition.left,+ transform: 'translateX(-50%)',+ padding: '4px 8px',+ backgroundColor: 'rgba(0, 0, 0, 0.8)',+ color: 'white',+ fontSize: '12px',+ borderRadius: '4px',+ whiteSpace: 'nowrap',+ pointerEvents: 'none',+ opacity: isHovered ? 1 : 0,+ transition: 'opacity 0.2s ease-in-out',+ zIndex: 9999,+ }}+ >+ Edit me!+ </span>+<input+ ref={inputRef}type={type}value={value}onChange={(e) => updateValue(placeholder, e.target.value)}@@ -172,9 +234,10 @@ export function EditableValue({margin: 0,background: 'transparent',border: 'none',- borderBottom: '1px dashed rgba(96, 165, 250, 0.5)',+ borderBottom: '2px dashed rgba(96, 165, 250, 0.5)',outline: 'none',color: 'inherit',+ transition: 'border-bottom-color 0.2s ease-in-out',}}/></span>