Agent Zero AI framework
refactor: extract browser agent into built-in plugin
Alessandrodb15bdb3bf42
diff --git a/README.md b/README.mdindex f16f4ab4f..f47f44b50 100644--- a/README.md+++ b/README.md@@ -93,13 +93,20 @@ docker run -p 50001:80 agent0ai/agent-zero+### Browser Agent++- Browser automation is provided by the built-in `_browser_agent` plugin.+- It uses the effective Main Model resolved by `_model_config`; there is no separate browser model slot.+- Browser vision follows the Main Model's vision setting.+- Playwright Chromium: **Docker** images ship the headless shell preinstalled. **Local development** installs it on first Browser Agent use via `ensure_playwright_binary()` in `plugins/_browser_agent/helpers/playwright.py` (into `tmp/playwright`); you can pre-install manually (see [Development Setup](docs/setup/dev-setup.md)) to skip the wait.+4. **Completely Customizable and Extensible**- Almost nothing in this framework is hard-coded. Nothing is hidden. Everything can be extended or changed by the user.- The whole behavior is defined by a system prompt in the **prompts/default/agent.system.md** file. Change this prompt and change the framework dramatically.- The framework does not guide or limit the agent in any way. There are no hard-coded rails that agents have to follow.- Every prompt, every small message template sent to the agent in its communication loop can be found in the **prompts/** folder and changed.-- Every default tool can be found in the **python/tools/** folder and changed or copied to create new predefined tools.+- Built-in tools live in the core **tools/** folder or in built-in plugins under **plugins/** and can be adapted or extended.- **Automated configuration** via `A0_SET_` environment variables for deployment automation and easy setup.@@ -238,7 +245,7 @@ docker run -p 50001:80 agent0ai/agent-zero- Secrets management - agent can use credentials without seeing them- Agent can copy paste messages and files without rewriting them- LiteLLM global configuration field-- Custom HTTP headers field for browser agent+- Browser agent configuration improvements- Progressive web app support- Extra model params support for JSON- Short IDs for files and memories to prevent LLM errorsdiff --git a/docs/guides/mcp-setup.md b/docs/guides/mcp-setup.mdindex 09a259bbd..3bd3ddd3c 100644--- a/docs/guides/mcp-setup.md+++ b/docs/guides/mcp-setup.md@@ -114,4 +114,4 @@ Community-tested and reliable MCP servers:- **VSCode MCP** - IDE workflows> [!TIP]-> For browser automation tasks, MCP-based browser tools are more reliable than the built-in browser agent.+> For browser automation tasks, the built-in Browser Agent plugin covers the default workflow. MCP-based browser tools are still useful when you need a different browser stack, remote browser control, or an alternative to the built-in Playwright Chromium (preinstalled in Docker; on demand via `ensure_playwright_binary()` in local dev).diff --git a/docs/guides/projects.md b/docs/guides/projects.mdindex 3a50782e0..62a893a8e 100644--- a/docs/guides/projects.md+++ b/docs/guides/projects.md@@ -228,7 +228,7 @@ SMTP_PASSWORD=email_pwd_here### Subagent Configuration-Projects can enable or disable specific subagents (like the Browser Agent). This is configured via the UI and stored in `.a0proj/agents.json`.+Projects can enable or disable specific subagents. This is configured via the UI and stored in `.a0proj/agents.json`. The Browser Agent is not a subagent; it is a built-in plugin.### Knowledge Filesdiff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.mdindex e3056927e..0c7a3b0b1 100644--- a/docs/guides/troubleshooting.md+++ b/docs/guides/troubleshooting.md@@ -26,8 +26,8 @@ Refer to the [Choosing your LLMs](../setup/installation.md#installing-and-using-**7. How can I make Agent Zero retain memory between sessions?**Use **Settings → Backup & Restore** and avoid mapping the entire `/a0` directory. See [How to update Agent Zero](../setup/installation.md#how-to-update-agent-zero).-**8. My browser agent fails or is unreliable. What now?**-The built-in browser agent is currently unstable on some systems. Use Skills or MCP alternatives such as Browser OS, Chrome DevTools, or Vercel's Agent Browser. See [MCP Setup](mcp-setup.md).+**8. My browser agent fails or says Playwright is missing. What now?**+The built-in Browser Agent is a plugin that uses the Main Model from `_model_config`. **Docker:** the Chromium headless shell is shipped preinstalled (typically under `/a0/tmp/playwright`). **Local development:** if the binary is missing, `ensure_playwright_binary()` in `plugins/_browser_agent/helpers/playwright.py` runs `playwright install chromium --only-shell` into `tmp/playwright` on first Browser Agent use (you may see UI notifications). To install ahead of time, run `PLAYWRIGHT_BROWSERS_PATH=tmp/playwright playwright install chromium --only-shell` after `pip install -r requirements.txt`. If you prefer an external browser stack, use MCP alternatives such as Browser OS, Chrome DevTools, or Playwright MCP. See [MCP Setup](mcp-setup.md).**9. My secrets disappeared after a backup restore.**Secrets are stored in `/a0/usr/secrets.env` and are not always included in backup archives. Copy them manually.@@ -36,7 +36,7 @@ Secrets are stored in `/a0/usr/secrets.env` and are not always included in backu- Join the Agent Zero [Skool](https://www.skool.com/agent-zero) or [Discord](https://discord.gg/B8KZKNsPpj) community.**11. How do I adjust API rate limits?**-Use the model rate limit fields in Settings (Chat/Utility/Browser model sections) to set request/input/output limits. These map to the model config limits (for example `limit_requests`, `limit_input`, `limit_output`).+Use the model rate limit fields in Settings (Main Model and Utility Model sections) to set request/input/output limits. The Browser Agent inherits the Main Model limits. These map to the model config limits (for example `limit_requests`, `limit_input`, `limit_output`).**12. My `code_execution_tool` doesn't work, what's wrong?**- Ensure Docker is installed and running.diff --git a/docs/guides/usage.md b/docs/guides/usage.mdindex 6650b3fc2..daa2514f8 100644--- a/docs/guides/usage.md+++ b/docs/guides/usage.md@@ -127,7 +127,9 @@ Agent Zero's power comes from its ability to use [tools](../developer/architectu- **Understand Tools:** Agent Zero includes default tools like knowledge (powered by SearXNG), code execution, and communication. Understand the capabilities of these tools and how to invoke them.### Browser Agent Status & MCP Alternatives-The built-in browser agent currently has dependency issues on some systems. If web automation is critical, prefer MCP-based browser tools instead:+The built-in Browser Agent is provided by the `_browser_agent` plugin. It uses the effective Main Model from `_model_config`, including per-chat overrides and the Main Model vision flag. Playwright Chromium is preinstalled in **Docker**; in **local development** it is installed on demand when needed via `ensure_playwright_binary()` (see [Development Setup](../setup/dev-setup.md) to pre-install).++If you need a different browser stack or want external browser tooling, MCP-based browser tools are still a strong option:- **Browser OS MCP**- **Chrome DevTools MCP**diff --git a/docs/setup/dev-setup.md b/docs/setup/dev-setup.mdindex d9894bf6e..f6099700d 100644--- a/docs/setup/dev-setup.md+++ b/docs/setup/dev-setup.md@@ -67,9 +67,9 @@ Now when you select one of the python files in the project, you should see prope3. Install dependencies. Run these two commands in the terminal:```bashpip install -r requirements.txt-playwright install chromium+PLAYWRIGHT_BROWSERS_PATH=tmp/playwright playwright install chromium --only-shell```-These will install all the python packages and browser binaries for playwright (browser agent).+The first command installs Python dependencies. The second installs the Chromium headless shell into `tmp/playwright` ahead of time (same path in Docker: `/a0/tmp/playwright`). If you skip the second command, **local development** still downloads the shell on first Browser Agent use through `ensure_playwright_binary()` in `plugins/_browser_agent/helpers/playwright.py`. Pre-installing avoids that wait. **Docker** images ship the shell preinstalled; runtime install is for local dev when the binary is missing.Errors in the code editor caused by missing packages should now be gone. If not, try reloading the window.diff --git a/docs/setup/installation.md b/docs/setup/installation.mdindex 5092f0b40..353f7333a 100644--- a/docs/setup/installation.md+++ b/docs/setup/installation.md@@ -330,17 +330,19 @@ The Settings page is the control center for selecting the Large Language Models| LLM Role | Description || --- | --- |-| `chat_llm` | This is the primary LLM used for conversations and generating responses. |+| `chat_llm` | This is the primary LLM used for conversations, agent reasoning, tool use, and the built-in browser agent. Vision support controls browser vision and image understanding. || `utility_llm` | This LLM handles internal tasks like summarizing messages, managing memory, and processing internal prompts. Using a smaller, less expensive model here can improve efficiency. |-| `browser_llm` | This LLM powers the browser agent for web navigation and interaction tasks. Vision support is recommended for better page understanding. || `embedding_llm` | The embedding model shipped with A0 runs on CPU and is responsible for generating embeddings used for memory retrieval and knowledge base lookups. Changing the `embedding_llm` will re-index all of A0's memory. |**How to Change:**1. Open Settings page in the Web UI.-2. Choose the provider for the LLM for each role (Chat model, Utility model, Browser model, Embedding model) and write the model name.+2. Choose the provider for the LLM for each role (Main Model, Utility Model, Embedding Model) and write the model name.3. Click "Save" to apply the changes.+> [!NOTE]+> The Browser Agent does not have a separate model slot. It uses the effective Main Model resolved by `_model_config`, including per-chat overrides and the Main Model vision flag.+### Important Considerations#### Model Naming by Providerdiff --git a/knowledge/main/about/configuration.md b/knowledge/main/about/configuration.mdindex 7eec5ecd2..57a49205a 100644--- a/knowledge/main/about/configuration.md+++ b/knowledge/main/about/configuration.md@@ -2,16 +2,15 @@## LLM Roles-Agent Zero uses four distinct LLM roles, each configurable independently:+Agent Zero uses three configurable LLM roles:| Role | Purpose ||------|---------|-| `chat_llm` | Primary model for all agent reasoning and tool use |+| `chat_llm` | Primary model for all agent reasoning, tool use, and the Browser Agent || `utility_llm` | Secondary model for internal framework tasks: memory summarization, query generation, history compression, memory recall filtering |-| `browser_llm` | Model used by the browser agent; vision capability recommended || `embedding_llm` | Produces vector embeddings for memory and knowledge indexing |-The utility model handles high-volume, lower-stakes operations and can be a cheaper/faster model than the chat model. Changing the embedding model invalidates the existing vector index - the entire knowledge base is re-indexed automatically.+The utility model handles high-volume, lower-stakes operations and can be a cheaper/faster model than the chat model. The Browser Agent uses the effective chat model resolved by `_model_config`, including per-chat overrides and the chat model vision flag. Changing the embedding model invalidates the existing vector index - the entire knowledge base is re-indexed automatically.## Model Providersdiff --git a/models.py b/models.pyindex 7b8905205..7be6aca9d 100644--- a/models.py+++ b/models.py@@ -25,7 +25,7 @@ from helpers.dotenv import load_dotenvfrom helpers.providers import ModelType as ProviderModelType, get_provider_configfrom helpers.rate_limiter import RateLimiterfrom helpers.tokens import approximate_tokens-from helpers import dirty_json, browser_use_monkeypatch+from helpers import dirty_jsonfrom langchain_core.language_models.chat_models import SimpleChatModelfrom langchain_core.outputs.chat_generation import ChatGenerationChunk@@ -57,9 +57,6 @@ def turn_off_logging():# initload_dotenv()turn_off_logging()-browser_use_monkeypatch.apply()--litellm.modify_params = True # helps fix anthropic tool calls by browser-useclass ModelType(Enum):CHAT = "Chat"@@ -578,101 +575,6 @@ class LiteLLMChatWrapper(SimpleChatModel):await asyncio.sleep(retry_delay_s)-class AsyncAIChatReplacement:- class _Completions:- def __init__(self, wrapper):- self._wrapper = wrapper-- async def create(self, *args, **kwargs):- # call the async _acall method on the wrapper- return await self._wrapper._acall(*args, **kwargs)-- class _Chat:- def __init__(self, wrapper):- self.completions = AsyncAIChatReplacement._Completions(wrapper)-- def __init__(self, wrapper, *args, **kwargs):- self._wrapper = wrapper- self.chat = AsyncAIChatReplacement._Chat(wrapper)---from browser_use.llm import ChatOllama, ChatOpenRouter, ChatGoogle, ChatAnthropic, ChatGroq, ChatOpenAI--class BrowserCompatibleChatWrapper(ChatOpenRouter):- """- A wrapper for browser agent that can filter/sanitize messages- before sending them to the LLM.- """-- def __init__(self, *args, **kwargs):- turn_off_logging()- # Create the underlying LiteLLM wrapper- self._wrapper = LiteLLMChatWrapper(*args, **kwargs)- # Browser-use may expect a 'model' attribute- self.model = self._wrapper.model_name- self.kwargs = self._wrapper.kwargs-- @property- def model_name(self) -> str:- return self._wrapper.model_name-- @property- def provider(self) -> str:- return self._wrapper.provider-- def get_client(self, *args, **kwargs): # type: ignore- return AsyncAIChatReplacement(self, *args, **kwargs)-- async def _acall(- self,- messages: List[BaseMessage],- stop: Optional[List[str]] = None,- run_manager: Optional[CallbackManagerForLLMRun] = None,- **kwargs: Any,- ):- # Apply rate limiting if configured- apply_rate_limiter_sync(self._wrapper.a0_model_conf, str(messages))-- # Call the model- try:- model = kwargs.pop("model", None)- kwrgs = {**self._wrapper.kwargs, **kwargs}-- # hack from browser-use to fix json schema for gemini (additionalProperties, $defs, $ref)- if "response_format" in kwrgs and "json_schema" in kwrgs["response_format"] and model.startswith("gemini/"):- kwrgs["response_format"]["json_schema"] = ChatGoogle("")._fix_gemini_schema(kwrgs["response_format"]["json_schema"])-- resp = await acompletion(- model=self._wrapper.model_name,- messages=messages,- stop=stop,- **kwrgs,- )-- # Gemini: strip triple backticks and conform schema- try:- msg = resp.choices[0].message # type: ignore- if self.provider == "gemini" and isinstance(getattr(msg, "content", None), str):- cleaned = browser_use_monkeypatch.gemini_clean_and_conform(msg.content) # type: ignore- if cleaned:- msg.content = cleaned- except Exception:- pass-- except Exception as e:- raise e-- # another hack for browser-use post process invalid jsons- try:- if "response_format" in kwrgs and "json_schema" in kwrgs["response_format"] or "json_object" in kwrgs["response_format"]:- if resp.choices[0].message.content is not None and not resp.choices[0].message.content.startswith("{"): # type: ignore- js = dirty_json.parse(resp.choices[0].message.content) # type: ignore- resp.choices[0].message.content = dirty_json.stringify(js) # type: ignore- except Exception as e:- pass-- return resp-class LiteLLMEmbeddingWrapper(Embeddings):model_name: strkwargs: dict = {}@@ -910,17 +812,6 @@ def get_chat_model(LiteLLMChatWrapper, name, provider_name, model_config, **kwargs)--def get_browser_model(- provider: str, name: str, model_config: Optional[ModelConfig] = None, **kwargs: Any-) -> BrowserCompatibleChatWrapper:- orig = provider.lower()- provider_name, kwargs = _merge_provider_defaults("chat", orig, kwargs)- return _get_litellm_chat(- BrowserCompatibleChatWrapper, name, provider_name, model_config, **kwargs- )--def get_embedding_model(provider: str, name: str, model_config: Optional[ModelConfig] = None, **kwargs: Any) -> LiteLLMEmbeddingWrapper | LocalSentenceTransformerWrapper:diff --git a/plugins/_browser_agent/api/status.py b/plugins/_browser_agent/api/status.pynew file mode 100644index 000000000..6ed1509b0--- /dev/null+++ b/plugins/_browser_agent/api/status.py@@ -0,0 +1,45 @@+import importlib.metadata++from helpers.api import ApiHandler, Request, Response+from plugins._browser_agent.helpers.playwright import (+ get_playwright_binary,+ get_playwright_cache_dir,+)+from plugins._model_config.helpers.model_config import get_chat_model_config+++class Status(ApiHandler):+ async def process(self, input: dict, request: Request) -> dict | Response:+ cfg = get_chat_model_config()+ binary = get_playwright_binary()++ browser_use_ok = False+ browser_use_error = ""+ browser_use_version = ""+ try:+ import browser_use # noqa: F401++ browser_use_ok = True+ browser_use_version = importlib.metadata.version("browser-use")+ except Exception as e:+ browser_use_error = str(e)++ return {+ "plugin": "_browser_agent",+ "model_source": "Main Model via _model_config",+ "model": {+ "provider": cfg.get("provider", ""),+ "name": cfg.get("name", ""),+ "vision": bool(cfg.get("vision", False)),+ },+ "playwright": {+ "cache_dir": get_playwright_cache_dir(),+ "binary_found": bool(binary),+ "binary_path": str(binary) if binary else "",+ },+ "browser_use": {+ "import_ok": browser_use_ok,+ "version": browser_use_version,+ "error": browser_use_error,+ },+ }diff --git a/lib/browser/init_override.js b/plugins/_browser_agent/assets/init_override.jssimilarity index 99%rename from lib/browser/init_override.jsrename to plugins/_browser_agent/assets/init_override.jsindex d7c786aa3..b04b0b98e 100644--- a/lib/browser/init_override.js+++ b/plugins/_browser_agent/assets/init_override.js@@ -243,4 +243,4 @@// }// return element;// };-// })();\ No newline at end of file+// })();diff --git a/plugins/_browser_agent/extensions/webui/get_message_handler/browser-agent-handler.js b/plugins/_browser_agent/extensions/webui/get_message_handler/browser-agent-handler.jsnew file mode 100644index 000000000..5177c1394--- /dev/null+++ b/plugins/_browser_agent/extensions/webui/get_message_handler/browser-agent-handler.js@@ -0,0 +1,54 @@+import {+ createActionButton,+ copyToClipboard,+} from "/components/messages/action-buttons/simple-action-buttons.js";+import { store as stepDetailStore } from "/components/modals/process-step-detail/step-detail-store.js";+import { store as speechStore } from "/components/chat/speech/speech-store.js";+import {+ buildDetailPayload,+ cleanStepTitle,+ drawProcessStep,+} from "/js/messages.js";++export default async function registerBrowserAgentHandler(extData) {+ if (extData?.type === "browser") {+ extData.handler = drawMessageBrowserAgent;+ }+}++function drawMessageBrowserAgent({+ id,+ type,+ heading,+ content,+ kvps,+ timestamp,+ agentno = 0,+ ...additional+}) {+ const title = cleanStepTitle(heading);+ const displayKvps = { ...kvps };+ const answerText = String(kvps?.answer ?? "");+ const actionButtons = answerText.trim()+ ? [+ createActionButton("detail", "", () =>+ stepDetailStore.showStepDetail(+ buildDetailPayload(arguments[0], { headerLabels: [] }),+ ),+ ),+ createActionButton("speak", "", () => speechStore.speak(answerText)),+ createActionButton("copy", "", () => copyToClipboard(answerText)),+ ].filter(Boolean)+ : [];++ return drawProcessStep({+ id,+ title,+ code: "WWW",+ classes: undefined,+ kvps: displayKvps,+ content,+ actionButtons,+ log: arguments[0],+ });+}diff --git a/plugins/_browser_agent/helpers/__init__.py b/plugins/_browser_agent/helpers/__init__.pynew file mode 100644index 000000000..2d4bd7f1d--- /dev/null+++ b/plugins/_browser_agent/helpers/__init__.py@@ -0,0 +1 @@+# Built-in browser agent helpers.diff --git a/plugins/_browser_agent/helpers/browser_llm.py b/plugins/_browser_agent/helpers/browser_llm.pynew file mode 100644index 000000000..90f289360--- /dev/null+++ b/plugins/_browser_agent/helpers/browser_llm.py@@ -0,0 +1,131 @@+from typing import Any, List, Optional+import litellm+from litellm import acompletion+from langchain_core.callbacks.manager import CallbackManagerForLLMRun+from langchain_core.messages import BaseMessage++import models+from browser_use.llm import ChatGoogle, ChatOpenRouter+from helpers import dirty_json++from plugins._browser_agent.helpers import browser_use_monkeypatch+++_BROWSER_USE_PATCHED = False+++def apply_browser_use_patches() -> None:+ global _BROWSER_USE_PATCHED+ if _BROWSER_USE_PATCHED:+ return++ browser_use_monkeypatch.apply()+ litellm.modify_params = True+ _BROWSER_USE_PATCHED = True+++class AsyncAIChatReplacement:+ class _Completions:+ def __init__(self, wrapper):+ self._wrapper = wrapper++ async def create(self, *args, **kwargs):+ return await self._wrapper._acall(*args, **kwargs)++ class _Chat:+ def __init__(self, wrapper):+ self.completions = AsyncAIChatReplacement._Completions(wrapper)++ def __init__(self, wrapper, *args, **kwargs):+ self._wrapper = wrapper+ self.chat = AsyncAIChatReplacement._Chat(wrapper)+++class BrowserCompatibleChatWrapper(ChatOpenRouter):+ """+ A wrapper for browser agent that can filter/sanitize messages+ before sending them to the LLM.+ """++ def __init__(self, *args, **kwargs):+ apply_browser_use_patches()+ models.turn_off_logging()+ self._wrapper = models.LiteLLMChatWrapper(*args, **kwargs)+ self.model = self._wrapper.model_name+ self.kwargs = self._wrapper.kwargs++ @property+ def model_name(self) -> str:+ return self._wrapper.model_name++ @property+ def provider(self) -> str:+ return self._wrapper.provider++ def get_client(self, *args, **kwargs): # type: ignore+ return AsyncAIChatReplacement(self, *args, **kwargs)++ async def _acall(+ self,+ messages: List[BaseMessage],+ stop: Optional[List[str]] = None,+ run_manager: Optional[CallbackManagerForLLMRun] = None,+ **kwargs: Any,+ ):+ models.apply_rate_limiter_sync(self._wrapper.a0_model_conf, str(messages))++ try:+ model = kwargs.pop("model", None)+ kwrgs = {**self._wrapper.kwargs, **kwargs}++ # hack from browser-use to fix json schema for gemini (additionalProperties, $defs, $ref)+ if "response_format" in kwrgs and "json_schema" in kwrgs["response_format"] and model and model.startswith("gemini/"):+ kwrgs["response_format"]["json_schema"] = ChatGoogle("")._fix_gemini_schema(kwrgs["response_format"]["json_schema"])++ resp = await acompletion(+ model=self._wrapper.model_name,+ messages=messages,+ stop=stop,+ **kwrgs,+ )++ # Gemini: strip triple backticks and conform schema+ try:+ msg = resp.choices[0].message # type: ignore+ if self.provider == "gemini" and isinstance(getattr(msg, "content", None), str):+ cleaned = browser_use_monkeypatch.gemini_clean_and_conform(msg.content) # type: ignore+ if cleaned:+ msg.content = cleaned+ except Exception:+ pass++ except Exception as e:+ raise e++ # another hack for browser-use post process invalid jsons+ try:+ if "response_format" in kwrgs and "json_schema" in kwrgs["response_format"] or "json_object" in kwrgs["response_format"]:+ if resp.choices[0].message.content is not None and not resp.choices[0].message.content.startswith("{"): # type: ignore+ js = dirty_json.parse(resp.choices[0].message.content) # type: ignore+ resp.choices[0].message.content = dirty_json.stringify(js) # type: ignore+ except Exception as e:+ pass++ return resp+++def build_browser_model_from_config(+ model_config: models.ModelConfig,+) -> BrowserCompatibleChatWrapper:+ apply_browser_use_patches()+ original_provider = model_config.provider.lower()+ provider_name, kwargs = models._merge_provider_defaults( # type: ignore[attr-defined]+ "chat", original_provider, model_config.build_kwargs()+ )+ return models._get_litellm_chat( # type: ignore[attr-defined]+ BrowserCompatibleChatWrapper,+ model_config.name,+ provider_name,+ model_config,+ **kwargs,+ )diff --git a/helpers/browser_use.py b/plugins/_browser_agent/helpers/browser_use.pysimilarity index 80%rename from helpers/browser_use.pyrename to plugins/_browser_agent/helpers/browser_use.pyindex 3ba097382..df4b3c7a6 100644--- a/helpers/browser_use.py+++ b/plugins/_browser_agent/helpers/browser_use.py@@ -1,4 +1,4 @@from helpers import dotenvdotenv.save_dotenv_value("ANONYMIZED_TELEMETRY", "false")import browser_use-import browser_use.utils\ No newline at end of file+import browser_use.utilsdiff --git a/helpers/browser_use_monkeypatch.py b/plugins/_browser_agent/helpers/browser_use_monkeypatch.pysimilarity index 100%rename from helpers/browser_use_monkeypatch.pyrename to plugins/_browser_agent/helpers/browser_use_monkeypatch.pydiff --git a/helpers/playwright.py b/plugins/_browser_agent/helpers/playwright.pysimilarity index 98%rename from helpers/playwright.pyrename to plugins/_browser_agent/helpers/playwright.pyindex c00deba6d..10c68d656 100644--- a/helpers/playwright.py+++ b/plugins/_browser_agent/helpers/playwright.py@@ -1,4 +1,3 @@-import osimport sysfrom pathlib import Path@@ -36,4 +35,4 @@ def ensure_playwright_binary():bin = get_playwright_binary()if not bin:raise Exception("Playwright binary not found after installation")- return bin\ No newline at end of file+ return bindiff --git a/plugins/_browser_agent/plugin.yaml b/plugins/_browser_agent/plugin.yamlnew file mode 100644index 000000000..4ce5fae4e--- /dev/null+++ b/plugins/_browser_agent/plugin.yaml@@ -0,0 +1,8 @@+name: _browser_agent+title: Browser Agent+description: Built-in browser-use automation tool.+version: 1.0.0+always_enabled: false+settings_sections: []+per_project_config: false+per_agent_config: falsediff --git a/prompts/agent.system.tool.browser.md b/plugins/_browser_agent/prompts/agent.system.tool.browser.mdsimilarity index 100%rename from prompts/agent.system.tool.browser.mdrename to plugins/_browser_agent/prompts/agent.system.tool.browser.mddiff --git a/prompts/browser_agent.system.md b/plugins/_browser_agent/prompts/browser_agent.system.mdsimilarity index 98%rename from prompts/browser_agent.system.mdrename to plugins/_browser_agent/prompts/browser_agent.system.mdindex 65823560b..70a41a44f 100644--- a/prompts/browser_agent.system.md+++ b/plugins/_browser_agent/prompts/browser_agent.system.md@@ -19,4 +19,4 @@ When you have completed the assigned task OR are waiting for further instruction- Response field is used to answer to user's task or ask additional questions- If you navigate to a website and no further actions are requested, call "Complete task" immediately- If you complete any requested interaction (clicking, typing, etc.), call "Complete task"-- Never leave a task running indefinitely - always conclude with "Complete task"\ No newline at end of file+- Never leave a task running indefinitely - always conclude with "Complete task"diff --git a/tools/browser_agent.py b/plugins/_browser_agent/tools/browser_agent.pysimilarity index 98%rename from tools/browser_agent.pyrename to plugins/_browser_agent/tools/browser_agent.pyindex b04da6701..47237f9a4 100644--- a/tools/browser_agent.py+++ b/plugins/_browser_agent/tools/browser_agent.py@@ -6,9 +6,9 @@ from pathlib import Pathfrom helpers.tool import Tool, Responsefrom helpers import files, defer, persist_chat, strings-from helpers.browser_use import browser_use # type: ignore[attr-defined]+from plugins._browser_agent.helpers.browser_use import browser_use # type: ignore[attr-defined]from helpers.print_style import PrintStyle-from helpers.playwright import ensure_playwright_binary+from plugins._browser_agent.helpers.playwright import ensure_playwright_binaryfrom helpers.secrets import get_secrets_managerfrom extensions.python.message_loop_start._10_iteration_no import get_iter_nofrom pydantic import BaseModel@@ -16,6 +16,9 @@ import uuidfrom helpers.dirty_json import DirtyJson+PLUGIN_DIR = Path(__file__).resolve().parents[1]++class State:@staticmethodasync def create(agent: Agent):@@ -105,7 +108,7 @@ class State:# Add init script to the browser sessionif self.browser_session and self.browser_session.browser_context:- js_override = files.get_abs_path("lib/browser/init_override.js")+ js_override = str(PLUGIN_DIR / "assets" / "init_override.js")await self.browser_session.browser_context.add_init_script(path=js_override) if self.browser_session else Nonedef start_task(self, task: str):diff --git a/plugins/_browser_agent/webui/main.html b/plugins/_browser_agent/webui/main.htmlnew file mode 100644index 000000000..c243be3ba--- /dev/null+++ b/plugins/_browser_agent/webui/main.html@@ -0,0 +1,204 @@+<html>+<head>+ <title>Browser Agent</title>+ <script type="module">+ import { callJsonApi } from "/js/api.js";+ import "/components/plugins/list/pluginListStore.js";++ globalThis.browserAgentStatusApi = { callJsonApi };+ </script>+</head>+<body>+ <div+ x-data="{+ loading: true,+ error: '',+ status: null,+ async init() {+ try {+ this.status = await browserAgentStatusApi.callJsonApi('/plugins/_browser_agent/status', {});+ } catch (error) {+ this.error = error instanceof Error ? error.message : String(error);+ } finally {+ this.loading = false;+ }+ }+ }"+ x-init="init()"+ class="browser-agent-page"+ >+ <div class="section-title">Browser Agent</div>+ <div class="section-description">+ Built-in browser automation plugin backed by `browser-use` and Playwright.+ Model selection stays in `_model_config`; the browser agent follows the effective Main Model.+ </div>++ <div class="browser-agent-card" x-show="loading">+ <div class="status-row">+ <span class="material-symbols-outlined spinning">progress_activity</span>+ <span>Loading browser status...</span>+ </div>+ </div>++ <div class="browser-agent-card error" x-show="!loading && error">+ <div class="field-title">Status check failed</div>+ <div class="field-description" x-text="error"></div>+ </div>++ <template x-if="!loading && status">+ <div class="browser-agent-grid">+ <div class="browser-agent-card">+ <div class="field-title">Model Source</div>+ <div class="field-description" x-text="status.model_source"></div>+ </div>++ <div class="browser-agent-card">+ <div class="field-title">Resolved Main Model</div>+ <div class="status-row">+ <span class="status-key">Provider</span>+ <span class="status-value" x-text="status.model.provider || 'Not configured'"></span>+ </div>+ <div class="status-row">+ <span class="status-key">Model</span>+ <span class="status-value" x-text="status.model.name || 'Not configured'"></span>+ </div>+ <div class="status-row">+ <span class="status-key">Vision</span>+ <span class="status-badge" :class="status.model.vision ? 'ok' : 'warn'" x-text="status.model.vision ? 'Enabled' : 'Disabled'"></span>+ </div>+ </div>++ <div class="browser-agent-card">+ <div class="field-title">Playwright Runtime</div>+ <div class="status-row">+ <span class="status-key">Binary</span>+ <span class="status-badge" :class="status.playwright.binary_found ? 'ok' : 'fail'" x-text="status.playwright.binary_found ? 'Found' : 'Missing'"></span>+ </div>+ <div class="status-row">+ <span class="status-key">Cache</span>+ <span class="status-value mono" x-text="status.playwright.cache_dir"></span>+ </div>+ <div class="status-row" x-show="status.playwright.binary_path">+ <span class="status-key">Path</span>+ <span class="status-value mono" x-text="status.playwright.binary_path"></span>+ </div>+ <div class="field-description" x-show="!status.playwright.binary_found">+ Docker images ship the Playwright Chromium shell preinstalled. In local development, the first run installs it on demand via <span class="mono">ensure_playwright_binary()</span> if missing.+ </div>+ </div>++ <div class="browser-agent-card">+ <div class="field-title">browser-use</div>+ <div class="status-row">+ <span class="status-key">Import</span>+ <span class="status-badge" :class="status.browser_use.import_ok ? 'ok' : 'fail'" x-text="status.browser_use.import_ok ? 'Ready' : 'Error'"></span>+ </div>+ <div class="status-row" x-show="status.browser_use.version">+ <span class="status-key">Version</span>+ <span class="status-value" x-text="status.browser_use.version"></span>+ </div>+ <div class="field-description mono" x-show="status.browser_use.error" x-text="status.browser_use.error"></div>+ </div>+ </div>+ </template>++ <div class="browser-agent-actions">+ <button+ class="btn btn-field"+ @click="$store.pluginListStore.openPluginConfig({ name: '_model_config', has_config_screen: true })"+ >+ Open Model Settings+ </button>+ <button class="btn btn-field" @click="openModal('/plugins/_model_config/webui/main.html')">+ Open Presets+ </button>+ <button class="btn btn-field" @click="openModal('/plugins/_model_config/webui/api-keys.html')">+ Open API Keys+ </button>+ </div>+ </div>++ <style>+ .browser-agent-page {+ display: flex;+ flex-direction: column;+ gap: 14px;+ }++ .browser-agent-grid {+ display: grid;+ gap: 12px;+ grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));+ }++ .browser-agent-card {+ display: flex;+ flex-direction: column;+ gap: 10px;+ padding: 14px;+ background: var(--color-input);+ border: 1px solid var(--color-border);+ border-radius: 10px;+ }++ .browser-agent-card.error {+ border-color: rgba(214, 40, 40, 0.35);+ }++ .browser-agent-actions {+ display: flex;+ gap: 8px;+ flex-wrap: wrap;+ }++ .status-row {+ display: flex;+ align-items: flex-start;+ justify-content: space-between;+ gap: 12px;+ font-size: 0.84rem;+ }++ .status-key {+ opacity: 0.7;+ min-width: 64px;+ }++ .status-value {+ text-align: right;+ word-break: break-word;+ }++ .status-badge {+ padding: 2px 8px;+ border-radius: 999px;+ font-size: 0.76rem;+ font-weight: 600;+ border: 1px solid transparent;+ }++ .status-badge.ok {+ color: #1b5e20;+ background: rgba(46, 125, 50, 0.14);+ border-color: rgba(46, 125, 50, 0.24);+ }++ .status-badge.warn {+ color: #8a6100;+ background: rgba(191, 144, 0, 0.14);+ border-color: rgba(191, 144, 0, 0.24);+ }++ .status-badge.fail {+ color: #9f1239;+ background: rgba(190, 24, 93, 0.12);+ border-color: rgba(190, 24, 93, 0.24);+ }++ .mono {+ font-family: var(--font-mono);+ font-size: 0.78rem;+ }+ </style>+</body>+</html>diff --git a/plugins/_browser_agent/webui/thumbnail.jpg b/plugins/_browser_agent/webui/thumbnail.jpgnew file mode 100644index 000000000..2ea014eb8Binary files /dev/null and b/plugins/_browser_agent/webui/thumbnail.jpg differdiff --git a/plugins/_model_config/helpers/model_config.py b/plugins/_model_config/helpers/model_config.pyindex 592d2deac..28d2f5e5d 100644--- a/plugins/_model_config/helpers/model_config.py+++ b/plugins/_model_config/helpers/model_config.py@@ -187,13 +187,15 @@ def build_utility_model(agent=None):def build_browser_model(agent=None):- """Build and return a BrowserCompatibleChatWrapper using chat model config."""+ """Build and return the browser-use adapter using chat model config."""cfg = get_chat_model_config(agent)mc = build_model_config(cfg, models.ModelType.CHAT)- return models.get_browser_model(- mc.provider, mc.name, model_config=mc, **mc.build_kwargs()+ from plugins._browser_agent.helpers.browser_llm import (+ build_browser_model_from_config,)+ return build_browser_model_from_config(mc)+def build_embedding_model(agent=None):"""Build and return an embedding model wrapper."""diff --git a/plugins/_model_config/webui/config.html b/plugins/_model_config/webui/config.htmlindex 39c0f948d..af27aee22 100644--- a/plugins/_model_config/webui/config.html+++ b/plugins/_model_config/webui/config.html@@ -257,19 +257,6 @@@change="config[section.key].kwargs = $store.modelConfig.textToKwargs(config[section.key]._kwargs_text)"></textarea></div></div>-- <!-- Browser HTTP Headers -->- <template x-if="section.key === 'chat_model'">- <div class="field field-full">- <div class="field-label">- <div class="field-title">Browser HTTP Headers</div>- <div class="field-description">- Custom HTTP headers sent with browser requests. The browser agent uses the main model. Format is KEY=VALUE, one per line.- </div>- </div>- </div>- </template>-</div></template>diff --git a/plugins/_model_config/webui/main.html b/plugins/_model_config/webui/main.htmlindex 196f71233..5399d67ac 100644--- a/plugins/_model_config/webui/main.html+++ b/plugins/_model_config/webui/main.html@@ -183,13 +183,6 @@@change="preset.chat.kwargs = $store.modelConfig.textToKwargs(preset.chat._kwargs_text)"></textarea></div></div>- <div class="field field-full">- <div class="field-label">- <div class="field-title">Browser HTTP Headers</div>- <div class="field-description">Custom HTTP headers sent with browser requests. The browser agent uses the main model. Format is KEY=VALUE, one per line.</div>- </div>- </div>-<div class="preset-subheader">Utility Model <span style="opacity:0.5; font-size:0.75rem;">(optional — falls back to the configured Utility Model)</span></div><div class="field"><div class="field-label">diff --git a/tests/test_browser_agent_regressions.py b/tests/test_browser_agent_regressions.pynew file mode 100644index 000000000..1864678cf--- /dev/null+++ b/tests/test_browser_agent_regressions.py@@ -0,0 +1,80 @@+import asyncio+import importlib+import json+import sys+from pathlib import Path+from types import SimpleNamespace+++PROJECT_ROOT = Path(__file__).resolve().parents[1]+if str(PROJECT_ROOT) not in sys.path:+ sys.path.insert(0, str(PROJECT_ROOT))++import plugins._browser_agent.helpers.browser_use_monkeypatch as browser_use_monkeypatch+import plugins._browser_agent.tools.browser_agent as browser_agent_module+++def test_gemini_clean_and_conform_normalizes_known_single_action_shapes():+ raw = (+ '{"action":['+ '{"complete_task":{"title":"T","response":"R","page_summary":"S"}}'+ ']}'+ )++ cleaned = browser_use_monkeypatch.gemini_clean_and_conform(raw)++ assert cleaned is not None+ parsed = json.loads(cleaned)+ assert parsed["action"] == [+ {+ "done": {+ "success": True,+ "data": {+ "title": "T",+ "response": "R",+ "page_summary": "S",+ },+ }+ },+ ]+++class DummyBrowserSession:+ def __init__(self) -> None:+ self.kill_called = False+ self.close_called = False++ async def kill(self) -> None:+ self.kill_called = True++ async def close(self) -> None:+ self.close_called = True+++class DummyAgent:+ def __init__(self) -> None:+ self.context = SimpleNamespace(id="ctx", task=None)+++def test_browser_session_teardown_prefers_kill_for_keep_alive_sessions():+ state = browser_agent_module.State(DummyAgent())+ session = DummyBrowserSession()+ state.browser_session = session++ state.kill_task()++ assert session.kill_called is True+ assert session.close_called is False+++def test_browser_cleanup_extensions_follow_new_extensible_path_layout():+ extension = importlib.import_module("helpers.extension")+ remove_classes = extension._get_extension_classes( # type: ignore[attr-defined]+ "_functions/agent/AgentContext/remove/start"+ )+ reset_classes = extension._get_extension_classes( # type: ignore[attr-defined]+ "_functions/agent/AgentContext/reset/start"+ )++ assert any(cls.__name__ == "CleanupBrowserStateOnRemove" for cls in remove_classes)+ assert any(cls.__name__ == "CleanupBrowserStateOnReset" for cls in reset_classes)diff --git a/webui/js/messages.js b/webui/js/messages.jsindex e57a0d081..89962b53b 100644--- a/webui/js/messages.js+++ b/webui/js/messages.js@@ -94,8 +94,6 @@ export async function getMessageHandler(type) {return drawMessageResponse;case "tool":return drawMessageTool;- case "browser":- return drawMessageBrowser;case "progress":return drawMessageProgress;case "mcp":@@ -1204,48 +1202,6 @@ export function drawMessageToolSimple({});}-/**- * @param {MessageHandlerArgs & Record<string, any>} param0- * @returns {MessageHandlerResult}- */-export function drawMessageBrowser({- id,- type,- heading,- content,- kvps,- timestamp,- agentno = 0,- ...additional-}) {- const title = cleanStepTitle(heading);- let displayKvps = { ...kvps };- const answerText = String(kvps?.answer ?? "");- const actionButtons = answerText.trim()- ? [- createActionButton("detail", "", () =>- stepDetailStore.showStepDetail(- buildDetailPayload(arguments[0], { headerLabels: [] }),- ),- ),- createActionButton("speak", "", () => speechStore.speak(answerText)),- createActionButton("copy", "", () => copyToClipboard(answerText)),- ].filter(Boolean)- : [];-- return drawProcessStep({- id,- title,- code: "WWW",- classes: undefined,- kvps: displayKvps,- content,- // contentClasses: [],- actionButtons,- log: arguments[0],- });-}-/*** @param {MessageHandlerArgs & Record<string, any>} param0* @returns {MessageHandlerResult}