A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.
feat: add Browser Automation Agent using browser-use, Playwright and Orq.ai
Tiioluwani73fae03fc89f
diff --git a/ai_agents/browser_automation_agent/.env.example b/ai_agents/browser_automation_agent/.env.examplenew file mode 100644index 0000000..b1eb84f--- /dev/null+++ b/ai_agents/browser_automation_agent/.env.example@@ -0,0 +1,8 @@+# Orq.ai API key (required). Get one from https://orq.ai+ORQ_API_KEY=your_orq_api_key_here++# Optional overrides+# ORQ_BASE_URL=https://api.orq.ai/v3/router+# Orq.ai uses provider/model format. See https://docs.orq.ai/docs/proxy/supported-models+# ORQ_MODEL=alibaba/qwen3.6-flash+# AGENT_MAX_STEPS=25diff --git a/ai_agents/browser_automation_agent/README.md b/ai_agents/browser_automation_agent/README.mdnew file mode 100644index 0000000..7c9a737--- /dev/null+++ b/ai_agents/browser_automation_agent/README.md@@ -0,0 +1,133 @@+# Browser Automation Agent++> An AI agent that takes a natural language instruction and autonomously navigates the web to complete it.++## Overview++Browser Automation Agent is a Gradio app that turns plain-English instructions into real browser actions. Describe a task such as _"Find the latest news about AI agents"_ and the agent plans a sequence of steps, drives a real Chromium browser to carry them out, and returns a structured summary of what it found. It is powered by `browser-use` for autonomous browsing and an LLM served through the Orq.ai AI Router.++## Demo++++## Features++- **Natural language tasks** — describe what you want in plain English; no scripting required.+- **Autonomous browsing** — the agent plans and executes real browser actions (navigate, click, read, extract) step by step.+- **Structured results** — returns a clear summary plus the pages visited and a collapsible action log.+- **Resilient error handling** — surfaces provider/model errors in the UI and tolerates transient step retries.+- **Configurable model** — point at any Orq.ai-supported model via a single environment variable.+- **Clean chat UI** — a Gradio chat interface with ready-to-run example prompts.++## Tech Stack++Frameworks & Libraries:++- [browser-use](https://github.com/browser-use/browser-use) — autonomous browser agent+- [Playwright](https://playwright.dev/python/) — browser control / Chromium driver+- [LangChain](https://www.langchain.com/) — LLM orchestration+- [Gradio](https://www.gradio.app/) — chat web interface+- [python-dotenv](https://pypi.org/project/python-dotenv/) — environment variable loading++Additional Tools:++- Browser Automation: browser-use+- Web Framework: Gradio++## Prerequisites++- Python 3.10 or higher+- API keys for:+ - Orq.ai (`ORQ_API_KEY`)++## Installation++### 1. Clone the Repository++```bash+git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git+cd Hands-On-AI-Engineering/ai_agents/browser_automation_agent+```++### 2. Create Virtual Environment (Recommended)++```bash+python -m venv venv+# Windows:+venv\Scripts\activate+# macOS/Linux:+source venv/bin/activate+```++### 3. Install Dependencies++```bash+pip install -r requirements.txt+```++Then install the Chromium browser that `browser-use` drives:++```bash+python -m playwright install chromium+```++### 4. Set Up Environment Variables++```bash+cp .env.example .env+```++Edit `.env` and set your Orq.ai API key:++```env+ORQ_API_KEY=your_orq_api_key_here+```++Optionally override the model (uses Orq.ai's `provider/model` format):++```env+ORQ_MODEL=alibaba/qwen3.6-flash+```++## Usage++### Running the Application++```bash+gradio app.py+```++Then open the local URL printed in the terminal (usually `http://127.0.0.1:7860`) and start giving the agent tasks.++### Example Instructions++| Instruction | What the Agent Returns |+| --- | --- |+| "Find the latest news about AI agents and summarize the top 3 stories." | A short summary of the three most relevant recent articles, plus the pages it visited. |+| "What is the current weather in Tokyo?" | The current temperature and conditions pulled from a weather source. |+| "Find the top post on Hacker News right now and summarize it." | The title, link, and a brief summary of the current #1 Hacker News post. |+| "Look up the price of the latest iPhone on Apple's website." | The model name and current price found on apple.com. |+| "What is the latest model released by Anthropic?" | The most recent model name, release date, and a one-line description from Anthropic's newsroom. |++## Project Structure++```+browser_automation_agent/+├── app.py # Gradio UI and agent integration+├── agent.py # browser-use agent and LLM (Orq.ai) setup+├── requirements.txt # Python dependencies+├── .env.example # Environment variable template+├── .env # Your local secrets (git-ignored)+├── .gitignore+├── README.md+└── assets/ # Demo GIF+```++## How It Works++1. **User input**: The user types a natural language instruction into the Gradio chat interface (`app.py`).+2. **LLM setup**: `agent.py` builds an OpenAI-compatible chat client pointed at the Orq.ai AI Router (`https://api.orq.ai/v3/router`) using the configured model (e.g. `alibaba/qwen3.6-flash`).+3. **Agent planning**: The instruction is handed to a `browser-use` `Agent`, which uses the LLM to reason about the goal and plan a sequence of browser actions.+4. **Browser execution**: `browser-use` drives a real Chromium browser via Playwright, executing each action step by step (navigating, clicking, reading, and extracting content) while observing the page state after every step.+5. **Result extraction**: When the task is done, the agent's history is parsed into a structured result containing the final summary, the URLs visited, and an action log. Provider/model errors are surfaced if the task fails.+6. **Response**: The structured result is rendered back into the Gradio chat as a readable markdown response.diff --git a/ai_agents/browser_automation_agent/agent.py b/ai_agents/browser_automation_agent/agent.pynew file mode 100644index 0000000..75a67fb--- /dev/null+++ b/ai_agents/browser_automation_agent/agent.py@@ -0,0 +1,220 @@+"""Browser automation agent powered by browser-use + LangChain.++This module wires a LangChain `ChatOpenAI` client (pointed at the Orq.ai+router) into a `browser-use` Agent. The agent receives a natural language+instruction, autonomously plans and executes browser actions, and returns a+structured summary of what it did and what it found.+"""++from __future__ import annotations++import os+from dataclasses import dataclass, field+from typing import List, Optional++from dotenv import load_dotenv++try:+ # Recent browser-use ships its own chat-model classes. Its Agent reads+ # `llm.provider`, so it requires a browser-use ChatOpenAI rather than a raw+ # LangChain model. browser-use's ChatOpenAI accepts the same+ # base_url/api_key/model arguments, so the Orq.ai routing is unchanged.+ from browser_use import Agent, ChatOpenAI+except ImportError as exc: # pragma: no cover - surfaced to the user at runtime+ raise ImportError(+ "browser-use is not installed. Run `pip install -r requirements.txt`."+ ) from exc+++load_dotenv()++ORQ_API_KEY = os.getenv("ORQ_API_KEY")+ORQ_BASE_URL = os.getenv("ORQ_BASE_URL", "https://api.orq.ai/v3/router")+# Orq.ai routes models in `provider/model` format (e.g. alibaba/qwen3.6-flash).+ORQ_MODEL = os.getenv("ORQ_MODEL", "alibaba/qwen3.6-flash")++# Hard cap on the number of browser steps a single instruction may take.+DEFAULT_MAX_STEPS = int(os.getenv("AGENT_MAX_STEPS", "25"))+++@dataclass+class AgentResult:+ """Structured result returned to the UI layer."""++ instruction: str+ success: bool+ summary: str+ visited_urls: List[str] = field(default_factory=list)+ actions: List[str] = field(default_factory=list)+ error: Optional[str] = None++ def to_markdown(self) -> str:+ """Render the result as a chat-friendly markdown block."""+ status = "✅ Completed" if self.success else "⚠️ Did not complete"+ lines = [f"**{status}**", "", self.summary.strip() or "_No summary produced._"]++ if self.visited_urls:+ lines += ["", "**Pages visited:**"]+ lines += [f"- {url}" for url in self.visited_urls[:10]]++ if self.actions:+ lines += ["", "<details><summary>Action log</summary>", ""]+ lines += [f"{i + 1}. {action}" for i, action in enumerate(self.actions[:30])]+ lines += ["", "</details>"]++ if self.error:+ lines += ["", f"**Error:** `{self.error}`"]++ return "\n".join(lines)+++def get_llm() -> ChatOpenAI:+ """Build the LangChain chat model pointed at the Orq.ai router."""+ if not ORQ_API_KEY:+ raise ValueError(+ "ORQ_API_KEY is not set. Copy .env.example to .env and add your key."+ )++ return ChatOpenAI(+ base_url=ORQ_BASE_URL,+ api_key=ORQ_API_KEY,+ model=ORQ_MODEL,+ temperature=0.0,+ )+++def _extract_history(history) -> AgentResult:+ """Best-effort extraction of useful fields from a browser-use history object.++ The browser-use API surface has shifted across versions, so each accessor is+ guarded to keep the agent resilient.+ """+ summary = ""+ visited_urls: List[str] = []+ actions: List[str] = []+ success = True++ # Final natural-language answer.+ try:+ summary = history.final_result() or ""+ except Exception: # noqa: BLE001 - defensive across versions+ summary = ""++ # Whether the agent considered the task done.+ try:+ success = bool(history.is_done())+ except Exception: # noqa: BLE001+ success = bool(summary)++ # Surface any step errors (e.g. a model 404 from the LLM provider) instead+ # of silently reporting "no summary".+ step_errors: List[str] = []+ try:+ step_errors = [str(e) for e in history.errors() if e]+ except Exception: # noqa: BLE001+ step_errors = []++ # URLs the agent navigated to.+ try:+ visited_urls = [u for u in history.urls() if u]+ except Exception: # noqa: BLE001+ visited_urls = []++ # Human-readable model "thoughts"/actions, if available.+ try:+ for thought in history.model_thoughts():+ text = getattr(thought, "next_goal", None) or str(thought)+ if text:+ actions.append(text)+ except Exception: # noqa: BLE001+ actions = []++ # Only surface errors when the task did NOT produce an answer. browser-use+ # retries transient step failures (e.g. a model returning an empty/malformed+ # response), so a recovered error alongside a valid summary is just noise.+ error_msg: Optional[str] = None+ task_failed = not summary and not success+ if step_errors and task_failed:+ # De-duplicate identical errors (browser-use retries a lot).+ unique_errors = list(dict.fromkeys(step_errors))+ error_msg = unique_errors[-1]+ success = False++ if not summary:+ if error_msg:+ summary = "The agent could not complete the task due to an error."+ else:+ summary = (+ "The agent finished but did not return a textual summary. "+ "Check the action log for details."+ )++ # De-duplicate visited URLs while preserving order.+ seen = set()+ deduped_urls = []+ for url in visited_urls:+ if url not in seen:+ seen.add(url)+ deduped_urls.append(url)++ return AgentResult(+ instruction="",+ success=success,+ summary=summary,+ visited_urls=deduped_urls,+ actions=actions,+ error=error_msg,+ )+++async def run_browser_agent(+ instruction: str, max_steps: int = DEFAULT_MAX_STEPS+) -> AgentResult:+ """Run the browser-use agent for a single natural language instruction.++ Args:+ instruction: The user's natural language task.+ max_steps: Safety cap on the number of browser steps.++ Returns:+ An :class:`AgentResult` describing the outcome.+ """+ instruction = (instruction or "").strip()+ if not instruction:+ return AgentResult(+ instruction=instruction,+ success=False,+ summary="Please provide an instruction for the agent to carry out.",+ )++ try:+ llm = get_llm()+ except ValueError as exc:+ return AgentResult(+ instruction=instruction,+ success=False,+ summary="Configuration error.",+ error=str(exc),+ )++ try:+ agent = Agent(task=instruction, llm=llm)+ history = await agent.run(max_steps=max_steps)+ result = _extract_history(history)+ result.instruction = instruction+ return result+ except Exception as exc: # noqa: BLE001 - report any runtime failure to the UI+ return AgentResult(+ instruction=instruction,+ success=False,+ summary="The agent hit an error while browsing.",+ error=str(exc),+ )+++if __name__ == "__main__":+ import asyncio++ demo_task = "Find the latest news about AI agents and summarize the top 3 stories."+ outcome = asyncio.run(run_browser_agent(demo_task))+ print(outcome.to_markdown())diff --git a/ai_agents/browser_automation_agent/app.py b/ai_agents/browser_automation_agent/app.pynew file mode 100644index 0000000..f47fcd7--- /dev/null+++ b/ai_agents/browser_automation_agent/app.py@@ -0,0 +1,53 @@+"""Gradio chat UI for the Browser Automation Agent.++Type a natural language instruction (e.g. "Find the latest news about AI+agents") and the agent will autonomously navigate the web to complete it,+returning a structured summary in the chat.+"""++from __future__ import annotations++import asyncio++import gradio as gr++from agent import ORQ_MODEL, run_browser_agent++EXAMPLES = [+ "Find the latest news about AI agents and summarize the top 3 stories.",+ "What is the current weather in Tokyo?",+ "Find the top post on Hacker News right now and summarize it.",+ "Look up the price of the latest iPhone on Apple's website.",+]++DESCRIPTION = (+ "Give the agent a task in plain English. It plans a sequence of browser "+ f"actions, executes them step by step, and reports back. Powered by "+ f"`browser-use` + LangChain via Orq.ai (`{ORQ_MODEL}`)."+)+++def respond(message: str, history: list) -> str:+ """Synchronous Gradio handler that drives the async browser agent."""+ try:+ result = asyncio.run(run_browser_agent(message))+ except Exception as exc: # noqa: BLE001 - never crash the UI thread+ return f"**Error:** `{exc}`"+ return result.to_markdown()+++def build_demo() -> gr.ChatInterface:+ return gr.ChatInterface(+ fn=respond,+ title="🌐 Browser Automation Agent",+ description=DESCRIPTION,+ examples=EXAMPLES,+ fill_height=True,+ )+++demo = build_demo()+++if __name__ == "__main__":+ demo.launch()diff --git a/ai_agents/browser_automation_agent/assets/demo.gif b/ai_agents/browser_automation_agent/assets/demo.gifnew file mode 100644index 0000000..f90679cBinary files /dev/null and b/ai_agents/browser_automation_agent/assets/demo.gif differdiff --git a/ai_agents/browser_automation_agent/requirements.txt b/ai_agents/browser_automation_agent/requirements.txtnew file mode 100644index 0000000..eac4117--- /dev/null+++ b/ai_agents/browser_automation_agent/requirements.txt@@ -0,0 +1,6 @@+browser-use>=0.1.40+playwright>=1.45.0+langchain>=0.3.0+langchain-openai>=0.2.0+gradio>=4.44.0+python-dotenv>=1.0.0