Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
added notebook
Dillon DuPontf270af30e1ca
diff --git a/notebooks/customizing_computeragent.ipynb b/notebooks/customizing_computeragent.ipynbnew file mode 100644index 0000000000..b0234d24f3--- /dev/null+++ b/notebooks/customizing_computeragent.ipynb@@ -0,0 +1,194 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# Customizing Your ComputerAgent\n\n",+ "This notebook demonstrates four practical ways to increase the capabilities and success rate of your `ComputerAgent` in the Agent SDK:\n\n",+ "1. Simple: Prompt engineering (via optional `instructions`)\n",+ "2. Easy: Tools (function tools and custom computer tools)\n",+ "3. Intermediate: Callbacks\n",+ "4. Expert: Custom `@register_agent` loops\n\n",+ "> Tip: The same patterns work in scripts and services — the notebook just makes it easy to iterate."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Setup\n\n",+ "We'll import `ComputerAgent`, a simple computer shim, and some utilities."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "import logging\n",+ "from agent.agent import ComputerAgent\n",+ "from agent.callbacks import PromptInstructionsCallback, LoggingCallback\n",+ "\n",+ "# A very small computer shim for demo purposes (for full computer handlers, see docs)\n",+ "class DummyComputer:\n",+ " async def screenshot(self):\n",+ " # Return a 1x1 transparent PNG as base64 string (placeholder)\n",+ " import base64\n",+ " png_bytes = base64.b64decode(\"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8Xw8AAr8B9k2m0oYAAAAASUVORK5CYII=\")\n",+ " return base64.b64encode(png_bytes).decode()\n",+ "\n",+ " async def click(self, x: int, y: int):\n",+ " pass\n",+ "\n",+ " async def type(self, text: str):\n",+ " pass\n",+ "\n",+ "computer = DummyComputer()\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 1) Simple: Prompt engineering\n\n",+ "You can guide your agent with system-like `instructions`.\n\n",+ "Under the hood, `ComputerAgent(instructions=...)` adds a `PromptInstructionsCallback` that prepends a user message before each LLM call.\n\n",+ "This mirrors the recommended snippet in code:\n\n",+ "```python\n",+ "effective_input = full_input\n",+ "if instructions:\n",+ " effective_input = [{\"role\": \"user\", \"content\": instructions}] + full_input\n",+ "```"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "instructions = (\n",+ " \"You are a meticulous software operator. Prefer safe, deterministic actions. \"\n",+ " \"Always confirm via on-screen text before proceeding.\"\n",+ ")\n",+ "agent = ComputerAgent(\n",+ " model=\"openai/computer-use-preview\",\n",+ " tools=[computer],\n",+ " instructions=instructions,\n",+ " callbacks=[LoggingCallback(level=logging.INFO)],\n",+ ")\n",+ "messages = [\n",+ " {\"role\": \"user\", \"content\": \"Open the settings and turn on dark mode.\"}\n",+ "]\n",+ "\n",+ "# In notebooks, you may want to consume the async generator\n",+ "import asyncio\n",+ "async def run_once():\n",+ " async for chunk in agent.run(messages):\n",+ " # Print any assistant text outputs\n",+ " for item in chunk.get(\"output\", []):\n",+ " if item.get(\"type\") == \"message\":\n",+ " for c in item.get(\"content\", []):\n",+ " if c.get(\"text\"):\n",+ " print(c.get(\"text\"))\n",+ "\n",+ "await run_once()\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 2) Easy: Tools\n\n",+ "Add function tools to expose deterministic capabilities. Tools are auto-extracted to schemas and callable by the agent."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def calculate_percentage(numerator: float, denominator: float) -> str:\n",+ " \"\"\"Calculate a percentage string.\n",+ "\n",+ " Args:\n",+ " numerator: Numerator value\n",+ " denominator: Denominator value\n",+ " Returns:\n",+ " A formatted percentage string (e.g., '75.00%').\n",+ " \"\"\"\n",+ " if denominator == 0:\n",+ " return \"0.00%\"\n",+ " return f\"{(numerator/denominator)*100:.2f}%\"\n",+ "\n",+ "agent_with_tool = ComputerAgent(\n",+ " model=\"openai/computer-use-preview\",\n",+ " tools=[computer, calculate_percentage],\n",+ " instructions=\"When doing math, prefer the `calculate_percentage` tool when relevant.\",\n",+ ")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 3) Intermediate: Callbacks\n\n",+ "Callbacks offer lifecycle hooks. For example, limit recent images or record trajectories."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from agent.callbacks import ImageRetentionCallback, TrajectorySaverCallback\n",+ "\n",+ "agent_with_callbacks = ComputerAgent(\n",+ " model=\"anthropic/claude-3-5-sonnet-20241022\",\n",+ " tools=[computer],\n",+ " callbacks=[\n",+ " ImageRetentionCallback(only_n_most_recent_images=3),\n",+ " TrajectorySaverCallback(\"./trajectories\"),\n",+ " ],\n",+ ")\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## 4) Expert: Custom `@register_agent`\n\n",+ "Register custom agent configs that implement `predict_step` (and optionally `predict_click`). This gives you full control over prompting, message shaping, and tool wiring.\n\n",+ "See: `libs/python/agent/agent/loops/` for concrete examples."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## Next steps\n\n",+ "- Start with `instructions` for fast wins.\n",+ "- Add function tools for determinism and reliability.\n",+ "- Use callbacks to manage cost, logs, and safety.\n",+ "- Build custom loops for specialized domains."+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "Python 3",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "name": "python",+ "version": "3.10"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 5+}