Skip to content

A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.

feat: add Browser Automation Agent using browser-use, Playwright and Orq.ai

Tiioluwani73fae03fc89f
diff --git a/ai_agents/browser_automation_agent/.env.example b/ai_agents/browser_automation_agent/.env.example
new file mode 100644
index 0000000..b1eb84f
--- /dev/null
+++ b/ai_agents/browser_automation_agent/.env.example
@@ -0,0 +1,8 @@
+# Orq.ai API key (required). Get one from https://orq.ai
+ORQ_API_KEY=your_orq_api_key_here
+
+# Optional overrides
+# ORQ_BASE_URL=https://api.orq.ai/v3/router
+# Orq.ai uses provider/model format. See https://docs.orq.ai/docs/proxy/supported-models
+# ORQ_MODEL=alibaba/qwen3.6-flash
+# AGENT_MAX_STEPS=25
diff --git a/ai_agents/browser_automation_agent/README.md b/ai_agents/browser_automation_agent/README.md
new file mode 100644
index 0000000..7c9a737
--- /dev/null
+++ b/ai_agents/browser_automation_agent/README.md
@@ -0,0 +1,133 @@
+# Browser Automation Agent
+
+> An AI agent that takes a natural language instruction and autonomously navigates the web to complete it.
+
+## Overview
+
+Browser Automation Agent is a Gradio app that turns plain-English instructions into real browser actions. Describe a task such as _"Find the latest news about AI agents"_ and the agent plans a sequence of steps, drives a real Chromium browser to carry them out, and returns a structured summary of what it found. It is powered by `browser-use` for autonomous browsing and an LLM served through the Orq.ai AI Router.
+
+## Demo
+
+![Demo](assets/demo.gif)
+
+## Features
+
+- **Natural language tasks** — describe what you want in plain English; no scripting required.
+- **Autonomous browsing** — the agent plans and executes real browser actions (navigate, click, read, extract) step by step.
+- **Structured results** — returns a clear summary plus the pages visited and a collapsible action log.
+- **Resilient error handling** — surfaces provider/model errors in the UI and tolerates transient step retries.
+- **Configurable model** — point at any Orq.ai-supported model via a single environment variable.
+- **Clean chat UI** — a Gradio chat interface with ready-to-run example prompts.
+
+## Tech Stack
+
+Frameworks & Libraries:
+
+- [browser-use](https://github.com/browser-use/browser-use) — autonomous browser agent
+- [Playwright](https://playwright.dev/python/) — browser control / Chromium driver
+- [LangChain](https://www.langchain.com/) — LLM orchestration
+- [Gradio](https://www.gradio.app/) — chat web interface
+- [python-dotenv](https://pypi.org/project/python-dotenv/) — environment variable loading
+
+Additional Tools:
+
+- Browser Automation: browser-use
+- Web Framework: Gradio
+
+## Prerequisites
+
+- Python 3.10 or higher
+- API keys for:
+ - Orq.ai (`ORQ_API_KEY`)
+
+## Installation
+
+### 1. Clone the Repository
+
+```bash
+git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git
+cd Hands-On-AI-Engineering/ai_agents/browser_automation_agent
+```
+
+### 2. Create Virtual Environment (Recommended)
+
+```bash
+python -m venv venv
+# Windows:
+venv\Scripts\activate
+# macOS/Linux:
+source venv/bin/activate
+```
+
+### 3. Install Dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+Then install the Chromium browser that `browser-use` drives:
+
+```bash
+python -m playwright install chromium
+```
+
+### 4. Set Up Environment Variables
+
+```bash
+cp .env.example .env
+```
+
+Edit `.env` and set your Orq.ai API key:
+
+```env
+ORQ_API_KEY=your_orq_api_key_here
+```
+
+Optionally override the model (uses Orq.ai's `provider/model` format):
+
+```env
+ORQ_MODEL=alibaba/qwen3.6-flash
+```
+
+## Usage
+
+### Running the Application
+
+```bash
+gradio app.py
+```
+
+Then open the local URL printed in the terminal (usually `http://127.0.0.1:7860`) and start giving the agent tasks.
+
+### Example Instructions
+
+| Instruction | What the Agent Returns |
+| --- | --- |
+| "Find the latest news about AI agents and summarize the top 3 stories." | A short summary of the three most relevant recent articles, plus the pages it visited. |
+| "What is the current weather in Tokyo?" | The current temperature and conditions pulled from a weather source. |
+| "Find the top post on Hacker News right now and summarize it." | The title, link, and a brief summary of the current #1 Hacker News post. |
+| "Look up the price of the latest iPhone on Apple's website." | The model name and current price found on apple.com. |
+| "What is the latest model released by Anthropic?" | The most recent model name, release date, and a one-line description from Anthropic's newsroom. |
+
+## Project Structure
+
+```
+browser_automation_agent/
+├── app.py # Gradio UI and agent integration
+├── agent.py # browser-use agent and LLM (Orq.ai) setup
+├── requirements.txt # Python dependencies
+├── .env.example # Environment variable template
+├── .env # Your local secrets (git-ignored)
+├── .gitignore
+├── README.md
+└── assets/ # Demo GIF
+```
+
+## How It Works
+
+1. **User input**: The user types a natural language instruction into the Gradio chat interface (`app.py`).
+2. **LLM setup**: `agent.py` builds an OpenAI-compatible chat client pointed at the Orq.ai AI Router (`https://api.orq.ai/v3/router`) using the configured model (e.g. `alibaba/qwen3.6-flash`).
+3. **Agent planning**: The instruction is handed to a `browser-use` `Agent`, which uses the LLM to reason about the goal and plan a sequence of browser actions.
+4. **Browser execution**: `browser-use` drives a real Chromium browser via Playwright, executing each action step by step (navigating, clicking, reading, and extracting content) while observing the page state after every step.
+5. **Result extraction**: When the task is done, the agent's history is parsed into a structured result containing the final summary, the URLs visited, and an action log. Provider/model errors are surfaced if the task fails.
+6. **Response**: The structured result is rendered back into the Gradio chat as a readable markdown response.
diff --git a/ai_agents/browser_automation_agent/agent.py b/ai_agents/browser_automation_agent/agent.py
new file mode 100644
index 0000000..75a67fb
--- /dev/null
+++ b/ai_agents/browser_automation_agent/agent.py
@@ -0,0 +1,220 @@
+"""Browser automation agent powered by browser-use + LangChain.
+
+This module wires a LangChain `ChatOpenAI` client (pointed at the Orq.ai
+router) into a `browser-use` Agent. The agent receives a natural language
+instruction, autonomously plans and executes browser actions, and returns a
+structured summary of what it did and what it found.
+"""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass, field
+from typing import List, Optional
+
+from dotenv import load_dotenv
+
+try:
+ # Recent browser-use ships its own chat-model classes. Its Agent reads
+ # `llm.provider`, so it requires a browser-use ChatOpenAI rather than a raw
+ # LangChain model. browser-use's ChatOpenAI accepts the same
+ # base_url/api_key/model arguments, so the Orq.ai routing is unchanged.
+ from browser_use import Agent, ChatOpenAI
+except ImportError as exc: # pragma: no cover - surfaced to the user at runtime
+ raise ImportError(
+ "browser-use is not installed. Run `pip install -r requirements.txt`."
+ ) from exc
+
+
+load_dotenv()
+
+ORQ_API_KEY = os.getenv("ORQ_API_KEY")
+ORQ_BASE_URL = os.getenv("ORQ_BASE_URL", "https://api.orq.ai/v3/router")
+# Orq.ai routes models in `provider/model` format (e.g. alibaba/qwen3.6-flash).
+ORQ_MODEL = os.getenv("ORQ_MODEL", "alibaba/qwen3.6-flash")
+
+# Hard cap on the number of browser steps a single instruction may take.
+DEFAULT_MAX_STEPS = int(os.getenv("AGENT_MAX_STEPS", "25"))
+
+
+@dataclass
+class AgentResult:
+ """Structured result returned to the UI layer."""
+
+ instruction: str
+ success: bool
+ summary: str
+ visited_urls: List[str] = field(default_factory=list)
+ actions: List[str] = field(default_factory=list)
+ error: Optional[str] = None
+
+ def to_markdown(self) -> str:
+ """Render the result as a chat-friendly markdown block."""
+ status = "✅ Completed" if self.success else "⚠️ Did not complete"
+ lines = [f"**{status}**", "", self.summary.strip() or "_No summary produced._"]
+
+ if self.visited_urls:
+ lines += ["", "**Pages visited:**"]
+ lines += [f"- {url}" for url in self.visited_urls[:10]]
+
+ if self.actions:
+ lines += ["", "<details><summary>Action log</summary>", ""]
+ lines += [f"{i + 1}. {action}" for i, action in enumerate(self.actions[:30])]
+ lines += ["", "</details>"]
+
+ if self.error:
+ lines += ["", f"**Error:** `{self.error}`"]
+
+ return "\n".join(lines)
+
+
+def get_llm() -> ChatOpenAI:
+ """Build the LangChain chat model pointed at the Orq.ai router."""
+ if not ORQ_API_KEY:
+ raise ValueError(
+ "ORQ_API_KEY is not set. Copy .env.example to .env and add your key."
+ )
+
+ return ChatOpenAI(
+ base_url=ORQ_BASE_URL,
+ api_key=ORQ_API_KEY,
+ model=ORQ_MODEL,
+ temperature=0.0,
+ )
+
+
+def _extract_history(history) -> AgentResult:
+ """Best-effort extraction of useful fields from a browser-use history object.
+
+ The browser-use API surface has shifted across versions, so each accessor is
+ guarded to keep the agent resilient.
+ """
+ summary = ""
+ visited_urls: List[str] = []
+ actions: List[str] = []
+ success = True
+
+ # Final natural-language answer.
+ try:
+ summary = history.final_result() or ""
+ except Exception: # noqa: BLE001 - defensive across versions
+ summary = ""
+
+ # Whether the agent considered the task done.
+ try:
+ success = bool(history.is_done())
+ except Exception: # noqa: BLE001
+ success = bool(summary)
+
+ # Surface any step errors (e.g. a model 404 from the LLM provider) instead
+ # of silently reporting "no summary".
+ step_errors: List[str] = []
+ try:
+ step_errors = [str(e) for e in history.errors() if e]
+ except Exception: # noqa: BLE001
+ step_errors = []
+
+ # URLs the agent navigated to.
+ try:
+ visited_urls = [u for u in history.urls() if u]
+ except Exception: # noqa: BLE001
+ visited_urls = []
+
+ # Human-readable model "thoughts"/actions, if available.
+ try:
+ for thought in history.model_thoughts():
+ text = getattr(thought, "next_goal", None) or str(thought)
+ if text:
+ actions.append(text)
+ except Exception: # noqa: BLE001
+ actions = []
+
+ # Only surface errors when the task did NOT produce an answer. browser-use
+ # retries transient step failures (e.g. a model returning an empty/malformed
+ # response), so a recovered error alongside a valid summary is just noise.
+ error_msg: Optional[str] = None
+ task_failed = not summary and not success
+ if step_errors and task_failed:
+ # De-duplicate identical errors (browser-use retries a lot).
+ unique_errors = list(dict.fromkeys(step_errors))
+ error_msg = unique_errors[-1]
+ success = False
+
+ if not summary:
+ if error_msg:
+ summary = "The agent could not complete the task due to an error."
+ else:
+ summary = (
+ "The agent finished but did not return a textual summary. "
+ "Check the action log for details."
+ )
+
+ # De-duplicate visited URLs while preserving order.
+ seen = set()
+ deduped_urls = []
+ for url in visited_urls:
+ if url not in seen:
+ seen.add(url)
+ deduped_urls.append(url)
+
+ return AgentResult(
+ instruction="",
+ success=success,
+ summary=summary,
+ visited_urls=deduped_urls,
+ actions=actions,
+ error=error_msg,
+ )
+
+
+async def run_browser_agent(
+ instruction: str, max_steps: int = DEFAULT_MAX_STEPS
+) -> AgentResult:
+ """Run the browser-use agent for a single natural language instruction.
+
+ Args:
+ instruction: The user's natural language task.
+ max_steps: Safety cap on the number of browser steps.
+
+ Returns:
+ An :class:`AgentResult` describing the outcome.
+ """
+ instruction = (instruction or "").strip()
+ if not instruction:
+ return AgentResult(
+ instruction=instruction,
+ success=False,
+ summary="Please provide an instruction for the agent to carry out.",
+ )
+
+ try:
+ llm = get_llm()
+ except ValueError as exc:
+ return AgentResult(
+ instruction=instruction,
+ success=False,
+ summary="Configuration error.",
+ error=str(exc),
+ )
+
+ try:
+ agent = Agent(task=instruction, llm=llm)
+ history = await agent.run(max_steps=max_steps)
+ result = _extract_history(history)
+ result.instruction = instruction
+ return result
+ except Exception as exc: # noqa: BLE001 - report any runtime failure to the UI
+ return AgentResult(
+ instruction=instruction,
+ success=False,
+ summary="The agent hit an error while browsing.",
+ error=str(exc),
+ )
+
+
+if __name__ == "__main__":
+ import asyncio
+
+ demo_task = "Find the latest news about AI agents and summarize the top 3 stories."
+ outcome = asyncio.run(run_browser_agent(demo_task))
+ print(outcome.to_markdown())
diff --git a/ai_agents/browser_automation_agent/app.py b/ai_agents/browser_automation_agent/app.py
new file mode 100644
index 0000000..f47fcd7
--- /dev/null
+++ b/ai_agents/browser_automation_agent/app.py
@@ -0,0 +1,53 @@
+"""Gradio chat UI for the Browser Automation Agent.
+
+Type a natural language instruction (e.g. "Find the latest news about AI
+agents") and the agent will autonomously navigate the web to complete it,
+returning a structured summary in the chat.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+import gradio as gr
+
+from agent import ORQ_MODEL, run_browser_agent
+
+EXAMPLES = [
+ "Find the latest news about AI agents and summarize the top 3 stories.",
+ "What is the current weather in Tokyo?",
+ "Find the top post on Hacker News right now and summarize it.",
+ "Look up the price of the latest iPhone on Apple's website.",
+]
+
+DESCRIPTION = (
+ "Give the agent a task in plain English. It plans a sequence of browser "
+ f"actions, executes them step by step, and reports back. Powered by "
+ f"`browser-use` + LangChain via Orq.ai (`{ORQ_MODEL}`)."
+)
+
+
+def respond(message: str, history: list) -> str:
+ """Synchronous Gradio handler that drives the async browser agent."""
+ try:
+ result = asyncio.run(run_browser_agent(message))
+ except Exception as exc: # noqa: BLE001 - never crash the UI thread
+ return f"**Error:** `{exc}`"
+ return result.to_markdown()
+
+
+def build_demo() -> gr.ChatInterface:
+ return gr.ChatInterface(
+ fn=respond,
+ title="🌐 Browser Automation Agent",
+ description=DESCRIPTION,
+ examples=EXAMPLES,
+ fill_height=True,
+ )
+
+
+demo = build_demo()
+
+
+if __name__ == "__main__":
+ demo.launch()
diff --git a/ai_agents/browser_automation_agent/assets/demo.gif b/ai_agents/browser_automation_agent/assets/demo.gif
new file mode 100644
index 0000000..f90679c
Binary files /dev/null and b/ai_agents/browser_automation_agent/assets/demo.gif differ
diff --git a/ai_agents/browser_automation_agent/requirements.txt b/ai_agents/browser_automation_agent/requirements.txt
new file mode 100644
index 0000000..eac4117
--- /dev/null
+++ b/ai_agents/browser_automation_agent/requirements.txt
@@ -0,0 +1,6 @@
+browser-use>=0.1.40
+playwright>=1.45.0
+langchain>=0.3.0
+langchain-openai>=0.2.0
+gradio>=4.44.0
+python-dotenv>=1.0.0