100+ AI Agents, Agent Skills and RAG Apps - Free and Open Source.
feat: YAML Based Multi-Agent Web Research Agent with Firecrawl MCP (low-code)
Shubhamsaboob4e2fdd489f8
diff --git a/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/README.md b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/README.mdnew file mode 100644index 00000000..3d7283ce--- /dev/null+++ b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/README.md@@ -0,0 +1,198 @@+# Multi-Agent Web Research System++A sophisticated multi-agent system built with Google ADK that uses Firecrawl MCP tools for web scraping and coordinates between specialized research and summary agents.++> **Location**: This example is part of the Google ADK Crash Course examples, located in `ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/`++## Architecture++This system consists of:++1. **Main Coordinator Agent** (`root_agent.yaml`) - Orchestrates the entire workflow+2. **Research Agent** (`research_agent.yaml`) - Uses Firecrawl MCP tools for web scraping and content analysis+3. **Summary Agent** (`summary_agent.yaml`) - Creates comprehensive reports and summaries+4. **Firecrawl MCP Integration** - Advanced web scraping with proper sub-agent configuration++## Features++- π **Advanced Web Scraping**: Uses Firecrawl MCP tools for reliable content extraction+- π¬ **Intelligent Content Analysis**: Research agent extracts insights, patterns, and key data+- π **Comprehensive Report Generation**: Summary agent creates structured reports and recommendations+- π€ **Multi-Agent Coordination**: Main agent orchestrates the entire workflow seamlessly+- π **Secure API Management**: Firecrawl API key managed via environment variables+- β‘ **Sub-Agent MCP Support**: Properly configured MCP tools in sub-agents++## Setup++### Prerequisites++1. Install Google ADK:+ ```bash+ pip install google-adk+ ```++2. Get Firecrawl API key from [firecrawl.dev](https://firecrawl.dev)++3. Set environment variables in `.env` file:++ **Option A: Google AI Studio (Recommended for development)**+ ```bash+ GOOGLE_GENAI_USE_VERTEXAI=0+ GOOGLE_API_KEY=<your-google-gemini-api-key>+ FIRECRAWL_API_KEY=<your-firecrawl-api-key>+ ```++ **Option B: Vertex AI (Recommended for production)**+ ```bash+ GOOGLE_GENAI_USE_VERTEXAI=1+ GOOGLE_CLOUD_PROJECT=<your-gcp-project-id>+ GOOGLE_CLOUD_LOCATION=us-central1+ FIRECRAWL_API_KEY=<your-firecrawl-api-key>+ ```++ **Getting API Keys:**+ - **Google AI Studio**: Get your API key from [Google AI Studio](https://aistudio.google.com/app/apikey)+ - **Vertex AI**: Set up authentication using [Google Cloud Authentication](https://cloud.google.com/vertex-ai/generative-ai/docs/start/api-keys)+ - **Firecrawl**: Get your API key from [Firecrawl](https://firecrawl.dev/app/api-keys)++### Installation++1. Navigate to the agent directory:+ ```bash+ cd ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher+ ```++2. Verify ADK installation:+ ```bash+ adk --version+ ```++## Usage++### Running the Agent++Choose one of these methods to run your agent:++1. **Web Interface** (Recommended for testing):+ ```bash+ adk web+ ```++2. **Command Line**:+ ```bash+ adk run+ ```++3. **API Server** (For integration):+ ```bash+ adk api_server+ ```++## Agent Configuration++### Main Agent (`root_agent.yaml`)++The coordinator agent that:+- Delegates tasks to specialized sub-agents+- Coordinates between research and summary agents+- Synthesizes final comprehensive reports+- Provides clear instructions to sub-agents++### Research Agent (`research_agent.yaml`)++Specialized for web scraping and content analysis:+- **Firecrawl MCP Tools**: Uses `firecrawl_scrape` and `firecrawl_search`+- **Content Analysis**: Extracts key findings and insights+- **Pattern Recognition**: Identifies trends and relationships+- **Data Extraction**: Highlights important quotes and data points+- **Research Suggestions**: Suggests areas for further investigation++**Available Firecrawl Tools:**+- `firecrawl_scrape`: Scrape content from single URLs+- `firecrawl_search`: Search the web for relevant content+- `firecrawl_batch_scrape`: Scrape multiple URLs efficiently+- `firecrawl_map`: Discover URLs on websites+- `firecrawl_crawl`: Comprehensive website crawling++### Summary Agent (`summary_agent.yaml`)++Specialized for report generation:+- Creates executive summaries+- Organizes information by topic+- Generates key takeaways+- Provides actionable recommendations++## Workflow++1. **Input**: User provides URL or research topic+2. **Delegation**: Main agent passes clear instructions to research_agent+3. **Web Scraping**: Research agent uses Firecrawl MCP tools to extract content+4. **Analysis**: Research agent analyzes scraped content for insights+5. **Summarization**: Summary agent creates comprehensive report+6. **Synthesis**: Main agent combines findings into final report++## Environment Variables++### Google AI Studio Configuration+| Variable | Description | Required |+|----------|-------------|----------|+| `GOOGLE_GENAI_USE_VERTEXAI` | Set to 0 for Google AI Studio | Yes |+| `GOOGLE_API_KEY` | Google Gemini API key from AI Studio | Yes |+| `FIRECRAWL_API_KEY` | Firecrawl API key for web scraping | Yes |++### Vertex AI Configuration+| Variable | Description | Required |+|----------|-------------|----------|+| `GOOGLE_GENAI_USE_VERTEXAI` | Set to 1 for Vertex AI | Yes |+| `GOOGLE_CLOUD_PROJECT` | Your Google Cloud Project ID | Yes |+| `GOOGLE_CLOUD_LOCATION` | GCP region (e.g., us-central1) | Yes |+| `FIRECRAWL_API_KEY` | Firecrawl API key for web scraping | Yes |++### Authentication Methods++**Google AI Studio:**+- Simple API key authentication+- Best for development and testing+- No Google Cloud setup required++**Vertex AI:**+- Enterprise-grade authentication+- Best for production deployments+- Requires Google Cloud Project setup+- Supports advanced features like grounding and safety settings+++## Example Usage++### Web Interface+1. Run `adk web`+2. Open browser to the provided URL+3. Enter a URL or research topic (e.g., "Scrape and analyze https://example.com" or "Research AI trends")+4. Watch the multi-agent system process your request++### Command Line+```bash+adk run+# Enter your research query when prompted+```++## Troubleshooting++### Common Issues++1. **API Key Errors**: Ensure all required API keys are set in `.env`+2. **ADK Not Found**: Make sure ADK is installed and Python environment is activated+3. **Firecrawl Errors**: Verify your Firecrawl API key is valid and has sufficient credits+4. **MCP Connection Issues**: Check that Node.js and npm are properly installed+5. **Authentication Issues**:+ - **Google AI Studio**: Verify your API key is valid and has proper permissions+ - **Vertex AI**: Ensure Google Cloud authentication is set up correctly (`gcloud auth application-default login`)+ - **Project ID**: Verify your Google Cloud Project ID is correct for Vertex AI+++## References++- [Google ADK Documentation](https://google.github.io/adk-docs/)+- [Agent Config Reference](https://google.github.io/adk-docs/agents/config/#build-an-agent)+- [Firecrawl Documentation](https://docs.firecrawl.dev/)+- [MCP Tools](https://modelcontextprotocol.io/)diff --git a/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/.env.example b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/.env.examplenew file mode 100644index 00000000..2db1a762--- /dev/null+++ b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/.env.example@@ -0,0 +1,12 @@+### Uncomment either of the two sections based on your prefrence.++## Using Google AI Studio+#GOOGLE_GENAI_USE_VERTEXAI=0+#GOOGLE_API_KEY=<your-google-gemini-api-key>+#FIRECRAWL_API_KEY=<your-firecrawl-api-key>++## Using Vertex AI+#GOOGLE_GENAI_USE_VERTEXAI=1+#GOOGLE_CLOUD_PROJECT=<your-gcp-project-id>+#GOOGLE_CLOUD_LOCATION=us-central1+#FIRECRAWL_API_KEY=<your-firecrawl-api-key>diff --git a/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/__init__.py b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/__init__.pynew file mode 100644index 00000000..8b137891--- /dev/null+++ b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/__init__.py@@ -0,0 +1 @@+diff --git a/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/research_agent.yaml b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/research_agent.yamlnew file mode 100644index 00000000..b8c22651--- /dev/null+++ b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/research_agent.yaml@@ -0,0 +1,33 @@+# yaml-language-server: $schema=https://raw.githubusercontent.com/google/adk-python/refs/heads/main/src/google/adk/agents/config_schemas/AgentConfig.json+name: research_agent+model: gemini-2.5-flash+description: 'Specialized agent for analyzing web content and extracting insights, patterns, and key information.'+instruction: |+ You are a research analysis agent with access to Firecrawl web scraping tools. Your job is to:+ 1. Use Firecrawl tools to scrape and search web content+ 2. Analyze scraped content for key insights and patterns+ 3. Identify important facts, trends, and relationships+ 4. Extract relevant quotes and data points+ 5. Provide structured analysis of the content+ 6. Highlight any inconsistencies or gaps in information+ Firecrawl Tool Usage:+ - For URLs: Use `firecrawl_scrape` with parameter: {"url": "https://example.com"}+ - For search queries: Use `firecrawl_search` with parameter: {"query": "search term"}+ - Always use simple object parameters, not arrays or complex structures++ Always provide your analysis in a structured format with clear sections for:+ - Key Findings+ - Important Data Points+ - Trends and Patterns+ - Notable Quotes+ - Areas for Further Investigation+tools:+ - name: MCPToolset+ args:+ stdio_server_params:+ command: "npx"+ args:+ - "-y"+ - "firecrawl-mcp"+ env:+ FIRECRAWL_API_KEY: "${FIRECRAWL_API_KEY}"\ No newline at end of filediff --git a/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/root_agent.yaml b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/root_agent.yamlnew file mode 100644index 00000000..3277aa1d--- /dev/null+++ b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/root_agent.yaml@@ -0,0 +1,23 @@+# yaml-language-server: $schema=https://raw.githubusercontent.com/google/adk-python/refs/heads/main/src/google/adk/agents/config_schemas/AgentConfig.json+name: web_research_coordinator+model: gemini-2.5-flash+description: 'A coordinator agent that manages web research using Firecrawl for scraping and two specialized sub-agents for research and summarization.'+instruction: |+ You are a web research coordinator agent. Your job is to:+ 1. Coordinate web research tasks using two sub-agents:+ - research_agent: Handles web search and scraping using the Firecrawl MCP tool, and analyzes content for insights and patterns+ - summary_agent: Creates comprehensive summaries and reports+ 2. Synthesize findings from both agents into actionable insights+ Important: When delegating to research_agent, provide clear, specific instructions:+ For URLs: "Please scrape and analyze the content from [URL]"+ For research topics: "Please search for and analyze information about [TOPIC]"+ Do NOT pass complex objects or arrays to the research_agent. Use simple, clear text instructions.+ When given a URL or research topic:+ - Pass a clear, simple instruction to the research_agent (e.g., "Scrape and analyze https://example.com" or "Research AI trends")+ - The research_agent will use appropriate Firecrawl tools with correct parameters+ - The research_agent will analyze the content and return key findings+ - Delegate summarization of the research_agent's analysis to the summary_agent+ - Combine outputs from both agents into a final comprehensive report+sub_agents:+ - config_path: research_agent.yaml+ - config_path: summary_agent.yaml\ No newline at end of filediff --git a/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/summary_agent.yaml b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/summary_agent.yamlnew file mode 100644index 00000000..067e7fdc--- /dev/null+++ b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/multi_agent_web_researcher/summary_agent.yaml@@ -0,0 +1,22 @@+# yaml-language-server: $schema=https://raw.githubusercontent.com/google/adk-python/refs/heads/main/src/google/adk/agents/config_schemas/AgentConfig.json+name: summary_agent+model: gemini-2.5-flash+description: 'Specialized agent for creating comprehensive summaries and reports from research findings.'+instruction: |+ You are a summarization agent. Your job is to:+ 1. Create clear, concise summaries of research findings+ 2. Organize information into logical sections+ 3. Generate executive summaries for quick understanding+ 4. Create detailed reports with proper formatting+ 5. Ensure all important information is captured and presented clearly+ Always structure your output with:+ - Executive Summary (2-3 sentences)+ - Detailed Summary (organized by topic)+ - Key Takeaways (bullet points)+ - Recommendations (if applicable)+ When creating summaries, ensure:+ - Information is accurate and well-organized+ - Key points are highlighted and easy to find+ - Complex information is simplified without losing meaning+ - Recommendations are actionable and specific+ - The summary is comprehensive yet concisediff --git a/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/requirements.txt b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/requirements.txtnew file mode 100644index 00000000..74aa2660--- /dev/null+++ b/ai_agent_framework_crash_course/google_adk_crash_course/adk_yaml_examples/multi_agent_web_research_team/requirements.txt@@ -0,0 +1,2 @@+google-adk+firecrawl-py\ No newline at end of file