Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
Added omniparser to grounding page
Dillon DuPont4eccf059e506
diff --git a/docs/content/docs/agent-sdk/supported-agents/grounding-models.mdx b/docs/content/docs/agent-sdk/supported-agents/grounding-models.mdxindex bf13d5a01e..61c9a70bf9 100644--- a/docs/content/docs/agent-sdk/supported-agents/grounding-models.mdx+++ b/docs/content/docs/agent-sdk/supported-agents/grounding-models.mdx@@ -29,23 +29,17 @@ All models that support `ComputerAgent.run()` also support `ComputerAgent.predicThese models are optimized specifically for click prediction and UI element grounding:-### GTA1-7B+### OmniParser-State-of-the-art grounding model from the [GUI Agent Grounding Leaderboard](https://gui-agent.github.io/grounding-leaderboard/):+OCR-focused set-of-marks model that requires an LLM for click prediction:-- `huggingface-local/HelloKKMe/GTA1-7B`+- `omniparser` (requires combination with any LiteLLM vision model)-```python-agent = ComputerAgent("huggingface-local/HelloKKMe/GTA1-7B", tools=[computer])+### GTA1-7B-# Predict click coordinates for UI elements-coords = agent.predict_click("find the submit button")-print(f"Click coordinates: {coords}") # (450, 320)+State-of-the-art grounding model from the [GUI Agent Grounding Leaderboard](https://gui-agent.github.io/grounding-leaderboard/):-# Note: GTA1 cannot perform autonomous task planning-# This will raise an error:-# agent.run("Fill out the form and submit it")-```+- `huggingface-local/HelloKKMe/GTA1-7B`## Usage Examples@@ -53,9 +47,6 @@ print(f"Click coordinates: {coords}") # (450, 320)# Using any grounding model for click predictionagent = ComputerAgent("claude-3-5-sonnet-20241022", tools=[computer])-# Take a screenshot first-screenshot = agent.computer.screenshot()-# Predict coordinates for specific elementslogin_coords = agent.predict_click("find the login button")search_coords = agent.predict_click("locate the search text field")@@ -66,6 +57,33 @@ print(f"Search field: {search_coords}")print(f"Menu icon: {menu_coords}")```+```python+# OmniParser is just for OCR, so it requires an LLM for predict_click+agent = ComputerAgent("omniparser+anthropic/claude-3-5-sonnet-20241022", tools=[computer])++# Predict click coordinates using composed agent+coords = agent.predict_click("find the submit button")+print(f"Click coordinates: {coords}") # (450, 320)++# Note: Cannot use omniparser alone for click prediction+# This will raise an error:+# agent = ComputerAgent("omniparser", tools=[computer])+# coords = agent.predict_click("find button") # Error!+```++```python+agent = ComputerAgent("huggingface-local/HelloKKMe/GTA1-7B", tools=[computer])++# Predict click coordinates for UI elements+coords = agent.predict_click("find the submit button")+print(f"Click coordinates: {coords}") # (450, 320)++# Note: GTA1 cannot perform autonomous task planning+# This will raise an error:+# agent.run("Fill out the form and submit it")+```++---For information on combining grounding models with planning capabilities, see [Composed Agents](./composed-agents).