Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
updated docs
Dillon DuPont5902be2917dd
diff --git a/libs/python/agent/benchmarks/README.md b/libs/python/agent/benchmarks/README.mdindex 3df840c154..03d1a78973 100644--- a/libs/python/agent/benchmarks/README.md+++ b/libs/python/agent/benchmarks/README.md@@ -25,8 +25,8 @@ The benchmark system evaluates models on GUI grounding tasks, specifically click### 3. Interactive Testing (`interactive.py`)- **Real-time testing**: Take screenshots and visualize model predictions- **Commands**:- - Type instruction → screenshot + test all models- - `screenshot` → take screenshot without prediction+ - Type instruction → test all models on last screenshot+ - `screenshot` → take screenshot- `models` → list available models- `quit`/`exit` → exit tool- **Output**: Visual predictions with crosshairs for each model@@ -63,21 +63,6 @@ Model Results:- **Visualizations**: `output/` directory with prediction visualizations- **Interactive Output**: `interactive_output/` for interactive session results-## Metrics Tracked--- **Accuracy**: Percentage of clicks within bounding boxes-- **Timing**: Average, min, max prediction times-- **VRAM Usage**: Maximum and average GPU memory usage-- **Per-sample Results**: Detailed breakdown for debugging--## Architecture--The benchmark system is designed for:-- **Modularity**: Easy to add new models and benchmarks-- **Flexibility**: Works with any iterator of dicts with `image`, `bbox`, `instruction` keys-- **Performance**: VRAM tracking and timing analysis-- **Visualization**: Automatic generation of prediction visualizations-## ContributingTo add a new reference model, follow the instructions in [contrib.md](contrib.md).\ No newline at end of file