Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

add to readme

Dillon DuPontf2501ee6b0f6
diff --git a/README.md b/README.md
index 53102fcb50..b063076002 100644
--- a/README.md
+++ b/README.md
@@ -80,8 +80,8 @@ If you want to use AI agents with virtualized environments:
async with Computer(verbosity=logging.DEBUG) as macos_computer:
agent = ComputerAgent(
computer=macos_computer,
- loop=AgentLoop.OPENAI, # or AgentLoop.ANTHROPIC, or AgentLoop.OMNI
- model=LLM(provider=LLMProvider.OPENAI) # or LLM(provider=LLMProvider.ANTHROPIC)
+ loop=AgentLoop.OPENAI, # or AgentLoop.UITARS, AgentLoop.OMNI, or AgentLoop.ANTHROPIC
+ model=LLM(provider=LLMProvider.OPENAI) # or LLM(provider=LLMProvider.MLXVLM, name="mlx-community/UI-TARS-1.5-7B-4bit")
)
tasks = [
diff --git a/libs/agent/README.md b/libs/agent/README.md
index e5dad869ef..bc4bce3292 100644
--- a/libs/agent/README.md
+++ b/libs/agent/README.md
@@ -136,7 +136,32 @@ The Gradio UI provides:
### Using UI-TARS
-You can use UI-TARS by first following the [deployment guide](https://github.com/bytedance/UI-TARS/blob/main/README_deploy.md). This will give you a provider URL like this: `https://**************.us-east-1.aws.endpoints.huggingface.cloud/v1` which you can use in the gradio UI.
+The UI-TARS models are available in two forms:
+
+1. **MLX UI-TARS models** (Default): These models run locally using MLXVLM provider
+ - `mlx-community/UI-TARS-1.5-7B-4bit` (default) - 4-bit quantized version
+ - `mlx-community/UI-TARS-1.5-7B-6bit` - 6-bit quantized version for higher quality
+
+ ```python
+ agent = ComputerAgent(
+ computer=macos_computer,
+ loop=AgentLoop.UITARS,
+ model=LLM(provider=LLMProvider.MLXVLM, name="mlx-community/UI-TARS-1.5-7B-4bit")
+ )
+ ```
+
+2. **OpenAI-compatible UI-TARS**: For using the original ByteDance model
+ - If you want to use the original ByteDance UI-TARS model via an OpenAI-compatible API, follow the [deployment guide](https://github.com/bytedance/UI-TARS/blob/main/README_deploy.md)
+ - This will give you a provider URL like `https://**************.us-east-1.aws.endpoints.huggingface.cloud/v1` which you can use in the code or Gradio UI:
+
+ ```python
+ agent = ComputerAgent(
+ computer=macos_computer,
+ loop=AgentLoop.UITARS,
+ model=LLM(provider=LLMProvider.OAICOMPAT, name="tgi",
+ provider_base_url="https://**************.us-east-1.aws.endpoints.huggingface.cloud/v1")
+ )
+ ```
## Agent Loops
@@ -146,7 +171,7 @@ The `cua-agent` package provides three agent loops variations, based on differen
|:-----------|:-----------------|:------------|:-------------|
| `AgentLoop.OPENAI` | • `computer_use_preview` | Use OpenAI Operator CUA model | Not Required |
| `AgentLoop.ANTHROPIC` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219` | Use Anthropic Computer-Use | Not Required |
-| `AgentLoop.UITARS` | • `ByteDance-Seed/UI-TARS-1.5-7B` | Uses ByteDance's UI-TARS 1.5 model | Not Required |
+| `AgentLoop.UITARS` | • `mlx-community/UI-TARS-1.5-7B-4bit` (default)<br>• `mlx-community/UI-TARS-1.5-7B-6bit`<br>• `ByteDance-Seed/UI-TARS-1.5-7B` (via openAI-compatible endpoint) | Uses UI-TARS models with MLXVLM (default) or OAICOMPAT providers | Not Required |
| `AgentLoop.OMNI` | • `claude-3-5-sonnet-20240620`<br>• `claude-3-7-sonnet-20250219`<br>• `gpt-4.5-preview`<br>• `gpt-4o`<br>• `gpt-4`<br>• `phi4`<br>• `phi4-mini`<br>• `gemma3`<br>• `...`<br>• `Any Ollama or OpenAI-compatible model` | Use OmniParser for element pixel-detection (SoM) and any VLMs for UI Grounding and Reasoning | OmniParser |
## AgentResponse