Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
added gpt-5 + gpta1 examples
Dillon DuPontf45f6b84e94f
diff --git a/docs/content/docs/agent-sdk/integrations/hud.mdx b/docs/content/docs/agent-sdk/integrations/hud.mdxindex cebd36be0a..b517121e0f 100644--- a/docs/content/docs/agent-sdk/integrations/hud.mdx+++ b/docs/content/docs/agent-sdk/integrations/hud.mdx@@ -28,6 +28,8 @@ taskset = TaskSet(tasks=taskset[:10]) # limit to 10 tasks instead of all 370# Run benchmark jobjob = await run_job(model="openai/computer-use-preview",+ # model="anthropic/claude-3-5-sonnet-20241022",+ # model="huggingface-local/HelloKKMe/GTA1-7B+openai/gpt-5",task_or_taskset=taskset,job_name="test-computeragent-job",max_concurrent_tasks=5,diff --git a/docs/content/docs/agent-sdk/supported-agents/composed-agents.mdx b/docs/content/docs/agent-sdk/supported-agents/composed-agents.mdxindex 50160fd8cf..8040d2e541 100644--- a/docs/content/docs/agent-sdk/supported-agents/composed-agents.mdx+++ b/docs/content/docs/agent-sdk/supported-agents/composed-agents.mdx@@ -28,41 +28,41 @@ Any model that supports `predict_click()` can be used as the grounding componentAny vision-enabled LiteLLM-compatible model can be used as the thinking component:- **Anthropic**: `anthropic/claude-3-5-sonnet-20241022`, `anthropic/claude-3-opus-20240229`-- **OpenAI**: `openai/gpt-4o`, `openai/gpt-4-vision-preview`+- **OpenAI**: `openai/gpt-5`, `openai/gpt-o3`, `openai/gpt-4o`- **Google**: `gemini/gemini-1.5-pro`, `vertex_ai/gemini-pro-vision`- **Local models**: Any Hugging Face vision-language model## Usage Examples-### GTA1 + Claude 3.5 Sonnet+### GTA1 + GPT-5-Combine state-of-the-art grounding with powerful reasoning:+Use Google's Gemini for planning with specialized grounding:```pythonagent = ComputerAgent(- "huggingface-local/HelloKKMe/GTA1-7B+anthropic/claude-3-5-sonnet-20241022",+ "huggingface-local/HelloKKMe/GTA1-7B+openai/gpt-5",tools=[computer])-async for _ in agent.run("Open Firefox, navigate to github.com, and search for 'computer-use'"):+async for _ in agent.run("Take a screenshot, analyze the UI, and click on the most prominent button"):pass-# Success! π-# - Claude 3.5 Sonnet plans the sequence of actions-# - GTA1-7B provides precise click coordinates for each UI element```-### GTA1 + Gemini Pro+### GTA1 + Claude 3.5 Sonnet-Use Google's Gemini for planning with specialized grounding:+Combine state-of-the-art grounding with powerful reasoning:```pythonagent = ComputerAgent(- "huggingface-local/HelloKKMe/GTA1-7B+gemini/gemini-1.5-pro",+ "huggingface-local/HelloKKMe/GTA1-7B+anthropic/claude-3-5-sonnet-20241022",tools=[computer])-async for _ in agent.run("Take a screenshot, analyze the UI, and click on the most prominent button"):+async for _ in agent.run("Open Firefox, navigate to github.com, and search for 'computer-use'"):pass+# Success! π+# - Claude 3.5 Sonnet plans the sequence of actions+# - GTA1-7B provides precise click coordinates for each UI element```### UI-TARS + GPT-4o