Skip to content

Fully autonomous AI Agents system capable of performing complex penetration testing tasks

feat(tools): add Firecrawl web search provider

rramprakash3f68960d9810
diff --git a/.env.example b/.env.example
index 4b95dc8f..73c01748 100644
--- a/.env.example
+++ b/.env.example
@@ -185,6 +185,11 @@ TRAVERSAAL_API_KEY=
## Tavily search engine API
TAVILY_API_KEY=
+## Firecrawl search engine API
+## FIRECRAWL_API_URL is optional; leave the default for the cloud API or point it at a self-hosted instance
+FIRECRAWL_API_KEY=
+FIRECRAWL_API_URL=https://api.firecrawl.dev
+
## Perplexity search engine API
PERPLEXITY_API_KEY=
PERPLEXITY_MODEL=
diff --git a/CLAUDE.md b/CLAUDE.md
index 6c908dd0..e8e150ef 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -122,7 +122,7 @@ State is managed primarily through Apollo Client (GraphQL) with real-time update
### Key Integrations
- **LLM Providers**: OpenAI, Anthropic, Gemini, AWS Bedrock, Ollama, DeepSeek, GLM, Kimi, Qwen, and custom HTTP endpoints — configured via environment variables or the Settings UI
-- **Search**: DuckDuckGo, Google, Tavily, Traversaal, Perplexity, Searxng
+- **Search**: DuckDuckGo, Google, Tavily, Firecrawl, Traversaal, Perplexity, Searxng
- **Databases**: PostgreSQL + pgvector (required), Neo4j (optional, for knowledge graph)
- **Observability**: OpenTelemetry → VictoriaMetrics + Loki + Jaeger → Grafana; Langfuse for LLM analytics
diff --git a/README.md b/README.md
index fc5ca587..f09cd31d 100644
--- a/README.md
+++ b/README.md
@@ -62,7 +62,7 @@ You can watch the video **PentAGI overview**:
- Smart Memory System. Long-term storage of research results and successful approaches for future use.
- Knowledge Graph Integration. Graphiti-powered knowledge graph using Neo4j for semantic relationship tracking and advanced context understanding.
- Web Intelligence. Built-in browser via [scraper](https://hub.docker.com/r/vxcontrol/scraper) for gathering latest information from web sources.
-- External Search Systems. Integration with advanced search APIs including [Tavily](https://tavily.com), [Traversaal](https://traversaal.ai), [Perplexity](https://www.perplexity.ai), [DuckDuckGo](https://duckduckgo.com/), [Google Custom Search](https://programmablesearchengine.google.com/), [Sploitus Search](https://sploitus.com) and [Searxng](https://searxng.org) for comprehensive information gathering.
+- External Search Systems. Integration with advanced search APIs including [Tavily](https://tavily.com), [Firecrawl](https://www.firecrawl.dev), [Traversaal](https://traversaal.ai), [Perplexity](https://www.perplexity.ai), [DuckDuckGo](https://duckduckgo.com/), [Google Custom Search](https://programmablesearchengine.google.com/), [Sploitus Search](https://sploitus.com) and [Searxng](https://searxng.org) for comprehensive information gathering.
- Team of Specialists. Delegation system with specialized AI agents for research, development, and infrastructure tasks, enhanced with optional execution monitoring and intelligent task planning for optimal performance with smaller models.
- Comprehensive Monitoring. Detailed logging and integration with Grafana/Prometheus for real-time system observation.
- Detailed Reporting. Generation of thorough vulnerability reports with exploitation guides.
@@ -629,7 +629,7 @@ The installer will:
1. **System Checks**: Verify Docker, network connectivity, and system requirements
2. **Environment Setup**: Create and configure `.env` file with optimal defaults
3. **Provider Configuration**: Set up LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Ollama, Custom)
-4. **Search Engines**: Configure DuckDuckGo, Google, Tavily, Traversaal, Perplexity, Sploitus, Searxng
+4. **Search Engines**: Configure DuckDuckGo, Google, Tavily, Firecrawl, Traversaal, Perplexity, Sploitus, Searxng
5. **Security Hardening**: Generate secure credentials and configure SSL certificates
6. **Deployment**: Start PentAGI with docker-compose
@@ -647,7 +647,7 @@ The PentAGI web console already manages several settings areas after the server
The following configuration areas still need to be set on the server through environment variables, compose files, or mounted config files:
- **LLM credentials and connection details**: API keys, endpoints, auth modes, and provider-specific connection settings for OpenAI, Anthropic, Bedrock, Ollama, custom providers, and similar backends; config-path settings apply only where supported, such as `OLLAMA_SERVER_CONFIG_PATH` and `LLM_SERVER_CONFIG_PATH`.
-- **Search provider credentials and options**: Settings such as `DUCKDUCKGO_*`, `GOOGLE_*`, `TAVILY_API_KEY`, `TRAVERSAAL_API_KEY`, `PERPLEXITY_*`, `SEARXNG_*`, and `SPLOITUS_ENABLED`.
+- **Search provider credentials and options**: Settings such as `DUCKDUCKGO_*`, `GOOGLE_*`, `TAVILY_API_KEY`, `FIRECRAWL_API_KEY`, `FIRECRAWL_API_URL`, `TRAVERSAAL_API_KEY`, `PERPLEXITY_*`, `SEARXNG_*`, and `SPLOITUS_ENABLED`.
- **Third-party integrations**: Langfuse, Graphiti, and similar external services remain server-side configuration.
- **MCP server management**: MCP settings pages are not currently exposed as a live web-console feature.
@@ -724,6 +724,7 @@ SPLOITUS_ENABLED=true
GOOGLE_API_KEY=your_google_key
GOOGLE_CX_KEY=your_google_cx
TAVILY_API_KEY=your_tavily_key
+FIRECRAWL_API_KEY=your_firecrawl_key
TRAVERSAAL_API_KEY=your_traversaal_key
PERPLEXITY_API_KEY=your_perplexity_key
PERPLEXITY_MODEL=sonar-pro
@@ -3447,6 +3448,7 @@ go run cmd/ftester/main.go browser
- **google**: Search the web using Google Custom Search
- **duckduckgo**: Search the web using DuckDuckGo
- **tavily**: Search using Tavily AI search engine
+- **firecrawl**: Search using Firecrawl with full-page content scraping
- **traversaal**: Search using Traversaal AI search engine
- **perplexity**: Search using Perplexity AI
- **sploitus**: Search for security exploits, vulnerabilities (CVEs), and pentesting tools
diff --git a/backend/cmd/ftester/mocks/tools.go b/backend/cmd/ftester/mocks/tools.go
index c4bf63b7..e0f6f561 100644
--- a/backend/cmd/ftester/mocks/tools.go
+++ b/backend/cmd/ftester/mocks/tools.go
@@ -132,6 +132,28 @@ func MockResponse(funcName string, args json.RawMessage) (string, error) {
resultObj = builder.String()
+ case tools.FirecrawlToolName:
+ var searchArgs tools.SearchAction
+ if err := json.Unmarshal(args, &searchArgs); err != nil {
+ return "", fmt.Errorf("error unmarshaling search arguments: %w", err)
+ }
+
+ terminal.PrintMock("Firecrawl search:")
+ terminal.PrintKeyValue("Query", searchArgs.Query)
+ terminal.PrintKeyValueFormat("Max results", "%d", searchArgs.MaxResults.Int())
+
+ var builder strings.Builder
+ builder.WriteString("# Links\n\n")
+
+ for i := 1; i <= min(searchArgs.MaxResults.Int(), 3); i++ {
+ builder.WriteString(fmt.Sprintf("## %d. Mock Firecrawl Result %d\n\n", i, i))
+ builder.WriteString(fmt.Sprintf("* URL https://example.com/firecrawl/result%d\n\n", i))
+ builder.WriteString(fmt.Sprintf("### Short content\n\nHere is a brief description of the content from this search result related to '%s'.\n\n", searchArgs.Query))
+ builder.WriteString(fmt.Sprintf("### Raw content for %d. Mock Firecrawl Result %d\n\nThis is the full scraped markdown content that Firecrawl would retrieve from the URL. It contains comprehensive information about '%s' that helps answer your query with specific facts and data points relevant to your search.\n\n", i, i, searchArgs.Query))
+ }
+
+ resultObj = builder.String()
+
case tools.TraversaalToolName:
var searchArgs tools.SearchAction
if err := json.Unmarshal(args, &searchArgs); err != nil {
diff --git a/backend/cmd/ftester/worker/args.go b/backend/cmd/ftester/worker/args.go
index 475be317..b1d00d8e 100644
--- a/backend/cmd/ftester/worker/args.go
+++ b/backend/cmd/ftester/worker/args.go
@@ -293,6 +293,7 @@ func getStructTypeForFunction(funcName string) (reflect.Type, error) {
tools.GoogleToolName: &tools.SearchAction{},
tools.DuckDuckGoToolName: &tools.SearchAction{},
tools.TavilyToolName: &tools.SearchAction{},
+ tools.FirecrawlToolName: &tools.SearchAction{},
tools.TraversaalToolName: &tools.SearchAction{},
tools.PerplexityToolName: &tools.SearchAction{},
tools.SearxngToolName: &tools.SearchAction{},
diff --git a/backend/cmd/ftester/worker/executor.go b/backend/cmd/ftester/worker/executor.go
index 2d2f6b6a..ed232e45 100644
--- a/backend/cmd/ftester/worker/executor.go
+++ b/backend/cmd/ftester/worker/executor.go
@@ -196,6 +196,16 @@ func (te *toolExecutor) GetTool(ctx context.Context, funcName string) (tools.Too
te.GetSummarizer(),
), nil
+ case tools.FirecrawlToolName:
+ return tools.NewFirecrawlTool(
+ te.cfg,
+ te.flowID,
+ te.taskID,
+ te.subtaskID,
+ te.proxies.GetSearchLogProvider(),
+ te.GetSummarizer(),
+ ), nil
+
case tools.TraversaalToolName:
return tools.NewTraversaalTool(
te.cfg,
diff --git a/backend/cmd/installer/wizard/controller/controller.go b/backend/cmd/installer/wizard/controller/controller.go
index fe98d179..b019bf8d 100644
--- a/backend/cmd/installer/wizard/controller/controller.go
+++ b/backend/cmd/installer/wizard/controller/controller.go
@@ -1575,6 +1575,8 @@ type SearchEnginesConfig struct {
SploitusEnabled loader.EnvVar // SPLOITUS_ENABLED
PerplexityAPIKey loader.EnvVar // PERPLEXITY_API_KEY
TavilyAPIKey loader.EnvVar // TAVILY_API_KEY
+ FirecrawlAPIKey loader.EnvVar // FIRECRAWL_API_KEY
+ FirecrawlAPIURL loader.EnvVar // FIRECRAWL_API_URL
TraversaalAPIKey loader.EnvVar // TRAVERSAAL_API_KEY
GoogleAPIKey loader.EnvVar // GOOGLE_API_KEY
GoogleCXKey loader.EnvVar // GOOGLE_CX_KEY
@@ -1611,6 +1613,8 @@ func (c *controller) GetSearchEnginesConfig() *SearchEnginesConfig {
sploitusEnabled, _ := c.GetVar("SPLOITUS_ENABLED")
perplexityAPIKey, _ := c.GetVar("PERPLEXITY_API_KEY")
tavilyAPIKey, _ := c.GetVar("TAVILY_API_KEY")
+ firecrawlAPIKey, _ := c.GetVar("FIRECRAWL_API_KEY")
+ firecrawlAPIURL, _ := c.GetVar("FIRECRAWL_API_URL")
traversaalAPIKey, _ := c.GetVar("TRAVERSAAL_API_KEY")
googleAPIKey, _ := c.GetVar("GOOGLE_API_KEY")
googleCXKey, _ := c.GetVar("GOOGLE_CX_KEY")
@@ -1634,6 +1638,8 @@ func (c *controller) GetSearchEnginesConfig() *SearchEnginesConfig {
PerplexityModel: perplexityModel,
PerplexityContextSize: perplexityContextSize,
TavilyAPIKey: tavilyAPIKey,
+ FirecrawlAPIKey: firecrawlAPIKey,
+ FirecrawlAPIURL: firecrawlAPIURL,
TraversaalAPIKey: traversaalAPIKey,
GoogleAPIKey: googleAPIKey,
GoogleCXKey: googleCXKey,
@@ -1664,6 +1670,9 @@ func (c *controller) GetSearchEnginesConfig() *SearchEnginesConfig {
if tavilyAPIKey.Value != "" {
configuredCount++
}
+ if firecrawlAPIKey.Value != "" {
+ configuredCount++
+ }
if traversaalAPIKey.Value != "" {
configuredCount++
}
@@ -1712,6 +1721,12 @@ func (c *controller) UpdateSearchEnginesConfig(config *SearchEnginesConfig) erro
if err := c.SetVar("TAVILY_API_KEY", config.TavilyAPIKey.Value); err != nil {
return fmt.Errorf("failed to set TAVILY_API_KEY: %w", err)
}
+ if err := c.SetVar("FIRECRAWL_API_KEY", config.FirecrawlAPIKey.Value); err != nil {
+ return fmt.Errorf("failed to set FIRECRAWL_API_KEY: %w", err)
+ }
+ if err := c.SetVar("FIRECRAWL_API_URL", config.FirecrawlAPIURL.Value); err != nil {
+ return fmt.Errorf("failed to set FIRECRAWL_API_URL: %w", err)
+ }
if err := c.SetVar("TRAVERSAAL_API_KEY", config.TraversaalAPIKey.Value); err != nil {
return fmt.Errorf("failed to set TRAVERSAAL_API_KEY: %w", err)
}
@@ -1759,6 +1774,8 @@ func (c *controller) ResetSearchEnginesConfig() *SearchEnginesConfig {
"PERPLEXITY_MODEL",
"PERPLEXITY_CONTEXT_SIZE",
"TAVILY_API_KEY",
+ "FIRECRAWL_API_KEY",
+ "FIRECRAWL_API_URL",
"TRAVERSAAL_API_KEY",
"GOOGLE_API_KEY",
"GOOGLE_CX_KEY",
@@ -2247,6 +2264,8 @@ func (c *controller) getVariableDescription(varName string) string {
"SPLOITUS_ENABLED": locale.EnvDesc_SPLOITUS_ENABLED,
"PERPLEXITY_API_KEY": locale.EnvDesc_PERPLEXITY_API_KEY,
"TAVILY_API_KEY": locale.EnvDesc_TAVILY_API_KEY,
+ "FIRECRAWL_API_KEY": locale.EnvDesc_FIRECRAWL_API_KEY,
+ "FIRECRAWL_API_URL": locale.EnvDesc_FIRECRAWL_API_URL,
"TRAVERSAAL_API_KEY": locale.EnvDesc_TRAVERSAAL_API_KEY,
"GOOGLE_API_KEY": locale.EnvDesc_GOOGLE_API_KEY,
"GOOGLE_CX_KEY": locale.EnvDesc_GOOGLE_CX_KEY,
@@ -2342,6 +2361,7 @@ var maskedVariables = map[string]bool{
"LOCAL_SCRAPER_PASSWORD": true,
"PERPLEXITY_API_KEY": true,
"TAVILY_API_KEY": true,
+ "FIRECRAWL_API_KEY": true,
"TRAVERSAAL_API_KEY": true,
"GOOGLE_API_KEY": true,
"GOOGLE_CX_KEY": true,
@@ -2440,6 +2460,8 @@ var criticalVariables = map[string]bool{
"PERPLEXITY_MODEL": true,
"PERPLEXITY_CONTEXT_SIZE": true,
"TAVILY_API_KEY": true,
+ "FIRECRAWL_API_KEY": true,
+ "FIRECRAWL_API_URL": true,
"TRAVERSAAL_API_KEY": true,
"GOOGLE_API_KEY": true,
"GOOGLE_CX_KEY": true,
diff --git a/backend/cmd/installer/wizard/locale/locale.go b/backend/cmd/installer/wizard/locale/locale.go
index b42833c9..14fda6d4 100644
--- a/backend/cmd/installer/wizard/locale/locale.go
+++ b/backend/cmd/installer/wizard/locale/locale.go
@@ -1391,6 +1391,7 @@ const (
• Sploitus - Security exploits and vulnerabilities database (no API key required)
• Perplexity - AI-powered search with reasoning
• Tavily - Search API for AI applications
+• Firecrawl - Web search with full-page scraping (self-hosting supported)
• Traversaal - Web scraping and search
• Google Search - Requires API key and Custom Search Engine ID
• Searxng - Internet metasearch engine
@@ -1398,6 +1399,7 @@ const (
Get API keys from:
• Perplexity: https://www.perplexity.ai/
• Tavily: https://tavily.com/
+• Firecrawl: https://www.firecrawl.dev/
• Traversaal: https://traversaal.ai/
• Google: https://developers.google.com/custom-search/v1/introduction`
@@ -1415,6 +1417,10 @@ Get API keys from:
ToolsSearchEnginesPerplexityKeyDesc = "API key for Perplexity AI search"
ToolsSearchEnginesTavilyKey = "Tavily API Key"
ToolsSearchEnginesTavilyKeyDesc = "API key for Tavily search service"
+ ToolsSearchEnginesFirecrawlKey = "Firecrawl API Key"
+ ToolsSearchEnginesFirecrawlKeyDesc = "API key for Firecrawl search service"
+ ToolsSearchEnginesFirecrawlURL = "Firecrawl API URL"
+ ToolsSearchEnginesFirecrawlURLDesc = "Firecrawl API base URL (leave empty for cloud; set for self-hosted)"
ToolsSearchEnginesTraversaalKey = "Traversaal API Key"
ToolsSearchEnginesTraversaalKeyDesc = "API key for Traversaal web scraping"
ToolsSearchEnginesGoogleKey = "Google Search API Key"
@@ -2271,6 +2277,8 @@ const (
EnvDesc_SPLOITUS_ENABLED = "Sploitus Search"
EnvDesc_PERPLEXITY_API_KEY = "Perplexity API Key"
EnvDesc_TAVILY_API_KEY = "Tavily API Key"
+ EnvDesc_FIRECRAWL_API_KEY = "Firecrawl API Key"
+ EnvDesc_FIRECRAWL_API_URL = "Firecrawl API URL"
EnvDesc_TRAVERSAAL_API_KEY = "Traversaal API Key"
EnvDesc_GOOGLE_API_KEY = "Google Search API Key"
EnvDesc_GOOGLE_CX_KEY = "Google Search CX Key"
diff --git a/backend/cmd/installer/wizard/models/search_engines_form.go b/backend/cmd/installer/wizard/models/search_engines_form.go
index d571ce18..6b8e7504 100644
--- a/backend/cmd/installer/wizard/models/search_engines_form.go
+++ b/backend/cmd/installer/wizard/models/search_engines_form.go
@@ -113,6 +113,21 @@ func (m *SearchEnginesFormModel) BuildForm() tea.Cmd {
config.TavilyAPIKey,
))
+ // Firecrawl API Key
+ fields = append(fields, m.createAPIKeyField("firecrawl_api_key",
+ locale.ToolsSearchEnginesFirecrawlKey,
+ locale.ToolsSearchEnginesFirecrawlKeyDesc,
+ config.FirecrawlAPIKey,
+ ))
+
+ // Firecrawl API URL (optional, for self-hosted deployments)
+ fields = append(fields, m.createTextField("firecrawl_api_url",
+ locale.ToolsSearchEnginesFirecrawlURL,
+ locale.ToolsSearchEnginesFirecrawlURLDesc,
+ config.FirecrawlAPIURL,
+ false,
+ ))
+
// Traversaal API Key
fields = append(fields, m.createAPIKeyField("traversaal_api_key",
locale.ToolsSearchEnginesTraversaalKey,
@@ -325,6 +340,15 @@ func (m *SearchEnginesFormModel) GetCurrentConfiguration() string {
m.GetStyles().Warning.Render(locale.StatusNotConfigured)))
}
+ // Firecrawl
+ if config.FirecrawlAPIKey.Value != "" {
+ sections = append(sections, fmt.Sprintf("• Firecrawl: %s",
+ m.GetStyles().Success.Render(locale.StatusConfigured)))
+ } else {
+ sections = append(sections, fmt.Sprintf("• Firecrawl: %s",
+ m.GetStyles().Warning.Render(locale.StatusNotConfigured)))
+ }
+
// Traversaal
if config.TraversaalAPIKey.Value != "" {
sections = append(sections, fmt.Sprintf("• Traversaal: %s",
@@ -393,6 +417,8 @@ func (m *SearchEnginesFormModel) HandleSave() error {
PerplexityModel: config.PerplexityModel,
PerplexityContextSize: config.PerplexityContextSize,
TavilyAPIKey: config.TavilyAPIKey,
+ FirecrawlAPIKey: config.FirecrawlAPIKey,
+ FirecrawlAPIURL: config.FirecrawlAPIURL,
TraversaalAPIKey: config.TraversaalAPIKey,
GoogleAPIKey: config.GoogleAPIKey,
GoogleCXKey: config.GoogleCXKey,
@@ -436,6 +462,10 @@ func (m *SearchEnginesFormModel) HandleSave() error {
newConfig.PerplexityContextSize.Value = value
case "tavily_api_key":
newConfig.TavilyAPIKey.Value = value
+ case "firecrawl_api_key":
+ newConfig.FirecrawlAPIKey.Value = value
+ case "firecrawl_api_url":
+ newConfig.FirecrawlAPIURL.Value = value
case "traversaal_api_key":
newConfig.TraversaalAPIKey.Value = value
case "google_api_key":
diff --git a/backend/docs/analytics_api.md b/backend/docs/analytics_api.md
index 8966103d..d4966feb 100644
--- a/backend/docs/analytics_api.md
+++ b/backend/docs/analytics_api.md
@@ -107,7 +107,7 @@ The `isAgent` field in `FunctionToolcallsStats` categorizes tools into two types
**Non-Agent Tools (`isAgent: false`):**
- Direct tool execution (e.g., `terminal`, `browser`, `file`)
-- Search engines (e.g., `google`, `duckduckgo`, `tavily`, `sploitus`, `searxng`)
+- Search engines (e.g., `google`, `duckduckgo`, `tavily`, `firecrawl`, `sploitus`, `searxng`)
- Vector database operations (e.g., `search_in_memory`, `store_guide`)
- Environment operations (e.g., `terminal`, `file`)
- Execution time represents actual operation duration
diff --git a/backend/docs/config.md b/backend/docs/config.md
index 08c8d1b6..78b6b127 100644
--- a/backend/docs/config.md
+++ b/backend/docs/config.md
@@ -58,6 +58,7 @@ This document serves as a comprehensive guide to the configuration system in Pen
- [Google Search](#google-search)
- [Traversaal Search](#traversaal-search)
- [Tavily Search](#tavily-search)
+ - [Firecrawl Search](#firecrawl-search)
- [Perplexity Search](#perplexity-search)
- [Searxng Search](#searxng-search)
- [Usage Details](#usage-details-10)
@@ -1400,6 +1401,7 @@ Common built-in functions that can be disabled:
- `google` - Google Search
- `duckduckgo` - DuckDuckGo Search
- `tavily` - Tavily Search
+- `firecrawl` - Firecrawl Search
- `traversaal` - Traversaal Search
- `perplexity` - Perplexity Search
- `searxng` - SearXNG Search
@@ -1447,6 +1449,13 @@ These settings control the integration with various search engines used for web
| ------------ | -------------------- | ------------- | -------------------------------- |
| TavilyAPIKey | `TAVILY_API_KEY` | *(none)* | API key for Tavily search engine |
+### Firecrawl Search
+
+| Option | Environment Variable | Default Value | Description |
+| --------------- | -------------------- | --------------------------- | ------------------------------------------------------------------------------ |
+| FirecrawlAPIKey | `FIRECRAWL_API_KEY` | *(none)* | API key for Firecrawl search engine |
+| FirecrawlAPIURL | `FIRECRAWL_API_URL` | `https://api.firecrawl.dev` | Base URL for the Firecrawl API (override to point at a self-hosted deployment) |
+
### Perplexity Search
| Option | Environment Variable | Default Value | Description |
@@ -1492,6 +1501,14 @@ tavilySearch: &functions.TavilySearchFunc{
summarizer: cfg.Summarizer,
},
+// Firecrawl Search configuration
+firecrawlSearch: &functions.FirecrawlSearchFunc{
+ apiKey: fte.cfg.FirecrawlAPIKey,
+ apiURL: fte.cfg.FirecrawlAPIURL,
+ proxyURL: fte.cfg.ProxyURL,
+ summarizer: cfg.Summarizer,
+},
+
// Perplexity Search configuration
perplexitySearch: &functions.PerplexitySearchFunc{
apiKey: fte.cfg.PerplexityAPIKey,
diff --git a/backend/docs/database.md b/backend/docs/database.md
index c4e17fdc..fbad72ba 100644
--- a/backend/docs/database.md
+++ b/backend/docs/database.md
@@ -458,7 +458,7 @@ The package provides comprehensive logging for all system operations:
- **AgentLog**: Inter-agent communication and delegation
- **AssistantLog**: Human-assistant interactions
- **MsgLog**: General message logging (thoughts/browser/terminal/file/search/advice/ask/input/done)
-- **SearchLog**: External search operations (google/tavily/traversaal/browser/duckduckgo/perplexity/sploitus/searxng)
+- **SearchLog**: External search operations (google/tavily/firecrawl/traversaal/browser/duckduckgo/perplexity/sploitus/searxng)
- **TermLog**: Terminal command execution (stdin/stdout/stderr)
- **ToolCall**: AI function calling with duration tracking
- `duration_seconds` - pre-calculated execution duration (DOUBLE PRECISION, NOT NULL, DEFAULT 0.0)
diff --git a/backend/docs/flow_execution.md b/backend/docs/flow_execution.md
index f9769136..74f8e154 100644
--- a/backend/docs/flow_execution.md
+++ b/backend/docs/flow_execution.md
@@ -47,6 +47,7 @@ This document describes the internal architecture and execution workflow of Flow
- `google` - Google Custom Search API integration
- `duckduckgo` - Anonymous search engine
- `tavily` - Advanced research with citations
+ - `firecrawl` - Web search with full-page content scraping
- `traversaal` - Structured Q&A search
- `perplexity` - AI-powered comprehensive research
- `sploitus` - Search for security exploits and pentest tools
@@ -760,14 +761,16 @@ The Searcher Agent follows a strict hierarchy for information retrieval:
3. **Priority 5: Deep Analysis Tools** - Complex research synthesis
- `traversaal` - Structured answers for common questions
- `tavily` - Research-grade exploration of technical topics
+ - `firecrawl` - Deep research with full-page content scraped from discovered sites
- `perplexity` - Comprehensive analysis with advanced reasoning
-**Available Search Engines**: Google, DuckDuckGo, Tavily, Traversaal, Perplexity, Sploitus, Searxng
+**Available Search Engines**: Google, DuckDuckGo, Tavily, Firecrawl, Traversaal, Perplexity, Sploitus, Searxng
**Search Engine Configurations**:
- **Google** - Custom Search API with CX key and language restrictions
- **DuckDuckGo** - Anonymous search with VQD token authentication
- **Tavily** - Advanced research with raw content and citations
+- **Firecrawl** - Web search with full-page markdown content scraped from each result
- **Perplexity** - AI-powered synthesis with configurable context size
- **Traversaal** - Structured Q&A responses with web links
- **Sploitus** - Search for security exploits and pentest tools
diff --git a/backend/migrations/sql/20260716_120000_add_firecrawl_search_type.sql b/backend/migrations/sql/20260716_120000_add_firecrawl_search_type.sql
new file mode 100644
index 00000000..6b95898e
--- /dev/null
+++ b/backend/migrations/sql/20260716_120000_add_firecrawl_search_type.sql
@@ -0,0 +1,54 @@
+-- +goose Up
+-- +goose StatementBegin
+-- Add firecrawl to the searchengine_type enum
+CREATE TYPE SEARCHENGINE_TYPE_NEW AS ENUM (
+ 'google',
+ 'tavily',
+ 'firecrawl',
+ 'traversaal',
+ 'browser',
+ 'duckduckgo',
+ 'perplexity',
+ 'searxng',
+ 'sploitus'
+);
+
+-- Update the searchlogs table to use the new enum type
+ALTER TABLE searchlogs
+ ALTER COLUMN engine TYPE SEARCHENGINE_TYPE_NEW USING engine::text::SEARCHENGINE_TYPE_NEW;
+
+-- Drop the old type and rename the new one
+DROP TYPE SEARCHENGINE_TYPE;
+ALTER TYPE SEARCHENGINE_TYPE_NEW RENAME TO SEARCHENGINE_TYPE;
+
+-- Ensure NOT NULL constraint is preserved
+ALTER TABLE searchlogs
+ ALTER COLUMN engine SET NOT NULL;
+-- +goose StatementEnd
+
+-- +goose Down
+-- +goose StatementBegin
+-- Revert the changes by removing firecrawl from the enum
+CREATE TYPE SEARCHENGINE_TYPE_NEW AS ENUM (
+ 'google',
+ 'tavily',
+ 'traversaal',
+ 'browser',
+ 'duckduckgo',
+ 'perplexity',
+ 'searxng',
+ 'sploitus'
+);
+
+-- Update the searchlogs table to use the reverted enum type
+ALTER TABLE searchlogs
+ ALTER COLUMN engine TYPE SEARCHENGINE_TYPE_NEW USING engine::text::SEARCHENGINE_TYPE_NEW;
+
+-- Drop the new type and rename the reverted one
+DROP TYPE SEARCHENGINE_TYPE;
+ALTER TYPE SEARCHENGINE_TYPE_NEW RENAME TO SEARCHENGINE_TYPE;
+
+-- Ensure NOT NULL constraint is preserved
+ALTER TABLE searchlogs
+ ALTER COLUMN engine SET NOT NULL;
+-- +goose StatementEnd
diff --git a/backend/pkg/config/config.go b/backend/pkg/config/config.go
index 9ad47c97..13cf18f4 100644
--- a/backend/pkg/config/config.go
+++ b/backend/pkg/config/config.go
@@ -167,6 +167,10 @@ type Config struct {
// === Search Engine: Tavily AI ===
TavilyAPIKey string `env:"TAVILY_API_KEY"`
+ // === Search Engine: Firecrawl ===
+ FirecrawlAPIKey string `env:"FIRECRAWL_API_KEY"`
+ FirecrawlAPIURL string `env:"FIRECRAWL_API_URL" envDefault:"https://api.firecrawl.dev"`
+
// === Search Engine: Perplexity AI ===
PerplexityAPIKey string `env:"PERPLEXITY_API_KEY"`
PerplexityModel string `env:"PERPLEXITY_MODEL" envDefault:"sonar"`
@@ -330,6 +334,7 @@ func (c *Config) GetSecretPatterns() []patterns.Pattern {
{c.OAuthGithubClientSecret, "Github Client Secret"},
{c.TraversaalAPIKey, "Traversaal Key"},
{c.TavilyAPIKey, "Tavily Key"},
+ {c.FirecrawlAPIKey, "Firecrawl Key"},
{c.PerplexityAPIKey, "Perplexity Key"},
{c.ProxyURL, "Proxy URL"},
{c.LangfusePublicKey, "Langfuse Public Key"},
diff --git a/backend/pkg/database/models.go b/backend/pkg/database/models.go
index 87440446..379cfbc7 100644
--- a/backend/pkg/database/models.go
+++ b/backend/pkg/database/models.go
@@ -474,6 +474,7 @@ type SearchengineType string
const (
SearchengineTypeGoogle SearchengineType = "google"
SearchengineTypeTavily SearchengineType = "tavily"
+ SearchengineTypeFirecrawl SearchengineType = "firecrawl"
SearchengineTypeTraversaal SearchengineType = "traversaal"
SearchengineTypeBrowser SearchengineType = "browser"
SearchengineTypeDuckduckgo SearchengineType = "duckduckgo"
diff --git a/backend/pkg/server/models/searchlogs.go b/backend/pkg/server/models/searchlogs.go
index e11b0ec6..402736c3 100644
--- a/backend/pkg/server/models/searchlogs.go
+++ b/backend/pkg/server/models/searchlogs.go
@@ -13,6 +13,7 @@ const (
SearchEngineTypeGoogle SearchEngineType = "google"
SearchEngineTypeDuckduckgo SearchEngineType = "duckduckgo"
SearchEngineTypeTavily SearchEngineType = "tavily"
+ SearchEngineTypeFirecrawl SearchEngineType = "firecrawl"
SearchEngineTypeTraversaal SearchEngineType = "traversaal"
SearchEngineTypePerplexity SearchEngineType = "perplexity"
SearchEngineTypeBrowser SearchEngineType = "browser"
@@ -29,6 +30,7 @@ func (s SearchEngineType) Valid() error {
case SearchEngineTypeGoogle,
SearchEngineTypeDuckduckgo,
SearchEngineTypeTavily,
+ SearchEngineTypeFirecrawl,
SearchEngineTypeTraversaal,
SearchEngineTypePerplexity,
SearchEngineTypeBrowser,
diff --git a/backend/pkg/templates/prompts/assistant.tmpl b/backend/pkg/templates/prompts/assistant.tmpl
index a5f0ce05..9737679a 100644
--- a/backend/pkg/templates/prompts/assistant.tmpl
+++ b/backend/pkg/templates/prompts/assistant.tmpl
@@ -162,7 +162,7 @@ External search engines and the vector store are indexed in English and shared a
{{- if .UseAgents}}
- Web research and memory: delegated to specialists (see TEAM COLLABORATION & DELEGATION above) — you do NOT invoke search engines or vector-store searches directly in this mode
{{- else}}
-- Web search engines: `{{.GoogleToolName}}`, `{{.DuckDuckGoToolName}}`, `{{.TavilyToolName}}`, `{{.TraversaalToolName}}`, `{{.PerplexityToolName}}` (and Searxng/Sploitus when configured) — pick the engine matching the query complexity
+- Web search engines: `{{.GoogleToolName}}`, `{{.DuckDuckGoToolName}}`, `{{.TavilyToolName}}`, `{{.TraversaalToolName}}`, `{{.PerplexityToolName}}` (and Firecrawl/Searxng/Sploitus when configured) — pick the engine matching the query complexity
- Memory search (read-only): `{{.SearchInMemoryToolName}}`, `{{.SearchGuideToolName}}`, `{{.SearchAnswerToolName}}`, `{{.SearchCodeToolName}}` for retrieving stored knowledge from past engagements
{{- end}}
</available_tools>
diff --git a/backend/pkg/templates/prompts/searcher.tmpl b/backend/pkg/templates/prompts/searcher.tmpl
index 1f213c16..153537fb 100644
--- a/backend/pkg/templates/prompts/searcher.tmpl
+++ b/backend/pkg/templates/prompts/searcher.tmpl
@@ -20,7 +20,7 @@ You operate on two parallel channels. The channel of each tool argument is fixed
1. **Engagement log — engagement language `{{.Lang}}`.** Your running commentary on this engagement. Entries are every `message` field of every tool call you make (external search engines, browser navigation, the answer vector-store search and store, the memorist delegation, the closing call) and the `message` of your closing `{{.SearchResultToolName}}` call. The engagement coordination team reads the log in `{{.Lang}}`. Keep `message` log entries to 1-2 short sentences narrating what you are about to do or what you just produced.
2. **Technical channel — English.** The wire between you, your team, search engines, the vector store, and the web. Outgoing entries are:
- - search-engine queries you send to whichever providers are available in this engagement (Google, DuckDuckGo, Tavily, Traversaal, Perplexity, Searxng, Sploitus, etc.) — use exact technical terms, identifiers, and error codes
+ - search-engine queries you send to whichever providers are available in this engagement (Google, DuckDuckGo, Tavily, Firecrawl, Traversaal, Perplexity, Searxng, Sploitus, etc.) — use exact technical terms, identifiers, and error codes
- browser `url` for direct retrieval from known sources
- answer vector-store search queries: `{{.SearchAnswerToolName}}.questions`
- answer vector-store write payloads with `{{.StoreAnswerToolName}}` (`answer`, `question`)
@@ -106,6 +106,7 @@ External search engines and the answer vector store are indexed in English and s
<deep_analysis_tools>
<tool name="tavily" priority="5">For research-grade exploration of complex technical topics</tool>
+<tool name="firecrawl" priority="5">For deep research with full-page content scraped directly from the discovered web sites</tool>
<tool name="perplexity" priority="5">For comprehensive analysis with advanced reasoning</tool>
<tool name="traversaal" priority="4">For discovering structured answers to common questions</tool>
</deep_analysis_tools>
diff --git a/backend/pkg/tools/firecrawl.go b/backend/pkg/tools/firecrawl.go
new file mode 100644
index 00000000..5d41695e
--- /dev/null
+++ b/backend/pkg/tools/firecrawl.go
@@ -0,0 +1,369 @@
+package tools
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "strings"
+ "text/template"
+
+ "pentagi/pkg/config"
+ "pentagi/pkg/database"
+ obs "pentagi/pkg/observability"
+ "pentagi/pkg/observability/langfuse"
+ "pentagi/pkg/system"
+
+ "github.com/sirupsen/logrus"
+)
+
+const (
+ firecrawlDefaultURL = "https://api.firecrawl.dev"
+ firecrawlSearchPath = "/v2/search"
+)
+
+type firecrawlRequest struct {
+ Query string `json:"query"`
+ Limit int `json:"limit,omitempty"`
+ Sources []string `json:"sources,omitempty"`
+ ScrapeOptions *firecrawlScrapeOptions `json:"scrapeOptions,omitempty"`
+}
+
+type firecrawlScrapeOptions struct {
+ Formats []string `json:"formats"`
+ OnlyMainContent bool `json:"onlyMainContent"`
+}
+
+type firecrawlSearchResult struct {
+ Success bool `json:"success"`
+ Warning string `json:"warning"`
+ Error string `json:"error"`
+ Data firecrawlData `json:"data"`
+}
+
+type firecrawlData struct {
+ Web []firecrawlResult `json:"web"`
+}
+
+type firecrawlResult struct {
+ Title string `json:"title"`
+ Description string `json:"description"`
+ URL string `json:"url"`
+ Markdown string `json:"markdown"`
+ Metadata *firecrawlMetadata `json:"metadata"`
+}
+
+type firecrawlMetadata struct {
+ Title string `json:"title"`
+ SourceURL string `json:"sourceURL"`
+}
+
+// resolvedURL prefers the top-level URL and falls back to the metadata source
+// URL, which is where it lands when scrapeOptions turn a result into a document.
+func (r firecrawlResult) resolvedURL() string {
+ if r.URL != "" {
+ return r.URL
+ }
+ if r.Metadata != nil {
+ return r.Metadata.SourceURL
+ }
+ return ""
+}
+
+// resolvedTitle prefers the top-level title and falls back to the metadata title.
+func (r firecrawlResult) resolvedTitle() string {
+ if r.Title != "" {
+ return r.Title
+ }
+ if r.Metadata != nil {
+ return r.Metadata.Title
+ }
+ return ""
+}
+
+type firecrawl struct {
+ cfg *config.Config
+ flowID int64
+ taskID *int64
+ subtaskID *int64
+ slp SearchLogProvider
+ summarizer SummarizeHandler
+}
+
+func NewFirecrawlTool(
+ cfg *config.Config,
+ flowID int64,
+ taskID, subtaskID *int64,
+ slp SearchLogProvider,
+ summarizer SummarizeHandler,
+) Tool {
+ return &firecrawl{
+ cfg: cfg,
+ flowID: flowID,
+ taskID: taskID,
+ subtaskID: subtaskID,
+ slp: slp,
+ summarizer: summarizer,
+ }
+}
+
+func (f *firecrawl) Handle(ctx context.Context, name string, args json.RawMessage) (string, error) {
+ if !f.IsAvailable() {
+ return "", fmt.Errorf("firecrawl is not available")
+ }
+
+ var action SearchAction
+ ctx, observation := obs.Observer.NewObservation(ctx)
+ logger := logrus.WithContext(ctx).WithFields(enrichLogrusFields(f.flowID, f.taskID, f.subtaskID, logrus.Fields{
+ "tool": name,
+ "args": string(args),
+ }))
+
+ if err := json.Unmarshal(args, &action); err != nil {
+ logger.WithError(err).Error("failed to unmarshal firecrawl search action")
+ return "", fmt.Errorf("failed to unmarshal %s search action arguments: %w", name, err)
+ }
+
+ logger = logger.WithFields(logrus.Fields{
+ "query": action.Query[:min(len(action.Query), 1000)],
+ "max_results": action.MaxResults,
+ })
+
+ result, err := f.search(ctx, action.Query, action.MaxResults.Int())
+ if err != nil {
+ observation.Event(
+ langfuse.WithEventName("search engine error swallowed"),
+ langfuse.WithEventInput(action.Query),
+ langfuse.WithEventStatus(err.Error()),
+ langfuse.WithEventLevel(langfuse.ObservationLevelWarning),
+ langfuse.WithEventMetadata(langfuse.Metadata{
+ "tool_name": FirecrawlToolName,
+ "engine": "firecrawl",
+ "query": action.Query,
+ "max_results": action.MaxResults.Int(),
+ "error": err.Error(),
+ }),
+ )
+
+ logger.WithError(err).Error("failed to search in firecrawl")
+ return fmt.Sprintf("failed to search in firecrawl: %v", err), nil
+ }
+
+ if agentCtx, ok := GetAgentContext(ctx); ok {
+ _, _ = f.slp.PutLog(
+ ctx,
+ agentCtx.ParentAgentType,
+ agentCtx.CurrentAgentType,
+ database.SearchengineTypeFirecrawl,
+ action.Query,
+ result,
+ f.taskID,
+ f.subtaskID,
+ )
+ }
+
+ return result, nil
+}
+
+func (f *firecrawl) search(ctx context.Context, query string, maxResults int) (string, error) {
+ client, err := system.GetHTTPClient(f.cfg)
+ if err != nil {
+ return "", fmt.Errorf("failed to create http client: %w", err)
+ }
+
+ reqPayload := firecrawlRequest{
+ Query: query,
+ Limit: maxResults,
+ Sources: []string{"web"},
+ ScrapeOptions: &firecrawlScrapeOptions{
+ Formats: []string{"markdown"},
+ OnlyMainContent: true,
+ },
+ }
+ reqBody, err := json.Marshal(reqPayload)
+ if err != nil {
+ return "", fmt.Errorf("failed to marshal request body: %v", err)
+ }
+
+ req, err := http.NewRequest(http.MethodPost, f.searchURL(), bytes.NewBuffer(reqBody))
+ if err != nil {
+ return "", fmt.Errorf("failed to build request: %v", err)
+ }
+
+ req = req.WithContext(ctx)
+ req.Header.Set("Content-Type", "application/json")
+ req.Header.Set("Authorization", "Bearer "+f.apiKey())
+
+ resp, err := client.Do(req)
+ if err != nil {
+ return "", fmt.Errorf("failed to do request: %v", err)
+ }
+ defer resp.Body.Close()
+
+ return f.parseHTTPResponse(ctx, resp)
+}
+
+func (f *firecrawl) parseHTTPResponse(ctx context.Context, resp *http.Response) (string, error) {
+ switch resp.StatusCode {
+ case http.StatusOK:
+ var respBody firecrawlSearchResult
+ if err := json.NewDecoder(resp.Body).Decode(&respBody); err != nil {
+ return "", fmt.Errorf("failed to decode response body: %v", err)
+ }
+ if !respBody.Success {
+ if respBody.Error != "" {
+ return "", fmt.Errorf("request failed: %s", respBody.Error)
+ }
+ return "", fmt.Errorf("request failed")
+ }
+ return f.buildFirecrawlResult(ctx, &respBody), nil
+ case http.StatusBadRequest:
+ return "", fmt.Errorf("request is invalid")
+ case http.StatusUnauthorized:
+ return "", fmt.Errorf("API key is wrong")
+ case http.StatusPaymentRequired:
+ return "", fmt.Errorf("insufficient credits to perform this request")
+ case http.StatusForbidden:
+ return "", fmt.Errorf("the endpoint requested is hidden for administrators only")
+ case http.StatusNotFound:
+ return "", fmt.Errorf("the specified endpoint could not be found")
+ case http.StatusMethodNotAllowed:
+ return "", fmt.Errorf("there need to try to access an endpoint with an invalid method")
+ case http.StatusRequestTimeout:
+ return "", fmt.Errorf("the request timed out. try again later")
+ case http.StatusTooManyRequests:
+ return "", fmt.Errorf("there are requesting too many results")
+ case http.StatusInternalServerError:
+ return "", fmt.Errorf("there had a problem with our server. try again later")
+ case http.StatusBadGateway:
+ return "", fmt.Errorf("there was a problem with the server. Please try again later")
+ case http.StatusServiceUnavailable:
+ return "", fmt.Errorf("there are temporarily offline for maintenance. please try again later")
+ case http.StatusGatewayTimeout:
+ return "", fmt.Errorf("there are temporarily offline for maintenance. please try again later")
+ default:
+ return "", fmt.Errorf("unexpected status code: %d", resp.StatusCode)
+ }
+}
+
+func (f *firecrawl) buildFirecrawlResult(ctx context.Context, result *firecrawlSearchResult) string {
+ var writer strings.Builder
+ writer.WriteString("# Links\n\n")
+
+ isMarkdownExists := false
+ for i, res := range result.Data.Web {
+ writer.WriteString(fmt.Sprintf("## %d. %s\n\n", i+1, res.resolvedTitle()))
+ writer.WriteString(fmt.Sprintf("* URL %s\n\n", res.resolvedURL()))
+ if res.Description != "" {
+ writer.WriteString(fmt.Sprintf("### Short content\n\n%s\n\n", res.Description))
+ }
+ if res.Markdown != "" {
+ isMarkdownExists = true
+ }
+ }
+
+ if isMarkdownExists && f.summarizer != nil {
+ summarizePrompt, err := f.getSummarizePrompt(result)
+ if err != nil {
+ writer.WriteString(f.getContentFromResults(result.Data.Web))
+ } else {
+ summarizedContents, err := f.summarizer(ctx, summarizePrompt)
+ if err != nil {
+ writer.WriteString(f.getContentFromResults(result.Data.Web))
+ } else {
+ writer.WriteString(fmt.Sprintf("### Summarized Content\n\n%s\n\n", summarizedContents))
+ }
+ }
+ } else {
+ writer.WriteString(f.getContentFromResults(result.Data.Web))
+ }
+
+ return writer.String()
+}
+
+func (f *firecrawl) getContentFromResults(results []firecrawlResult) string {
+ var writer strings.Builder
+ for i, res := range results {
+ if res.Markdown != "" {
+ markdown := res.Markdown
+ markdown = markdown[:min(len(markdown), maxRawContentLength)]
+ writer.WriteString(fmt.Sprintf("### Raw content for %d. %s\n\n%s\n\n", i+1, res.resolvedTitle(), markdown))
+ }
+ }
+ return writer.String()
+}
+
+func (f *firecrawl) getSummarizePrompt(result *firecrawlSearchResult) (string, error) {
+ templateText := `<instructions>
+TASK: Summarize web search results for the following user query:
+
+DATA:
+- <raw_content> tags contain web page content with attributes: id, title, url
+- Content may include HTML, structured data, tables, or plain text
+
+REQUIREMENTS:
+1. Create concise summary (max {{.MaxLength}} chars) that DIRECTLY answers the user query
+2. Preserve ALL critical facts, statistics, technical details, and numerical data
+3. Maintain all actionable insights, procedures, or code examples exactly as presented
+4. Keep ALL query-relevant information even if reducing overall length
+5. Highlight authoritative information and note contradictions between sources
+6. Cite sources using [Source #] format when presenting specific claims
+7. Ensure the user query is fully addressed in the summary
+8. NEVER remove information that answers the user's original question
+
+FORMAT:
+- Begin with a direct answer to the user query
+- Organize thematically with clear structure using headings
+- Keep bullet points and numbered lists for clarity and steps
+- Include brief "Sources Overview" section identifying key references
+
+The summary MUST provide complete answers to the user's query, preserving all relevant information.
+</instructions>
+
+{{range $index, $result := .Results}}
+{{if $result.Markdown}}
+<raw_content id="{{$index}}" title="{{$result.Title}}" url="{{$result.URL}}">
+{{$result.Markdown}}
+</raw_content>
+{{end}}
+{{end}}`
+
+ templateContext := map[string]any{
+ "MaxLength": maxRawContentLength,
+ "Results": result.Data.Web,
+ }
+
+ tmpl, err := template.New("summarize").Parse(templateText)
+ if err != nil {
+ return "", fmt.Errorf("error creating template: %v", err)
+ }
+
+ var buf bytes.Buffer
+ if err := tmpl.Execute(&buf, templateContext); err != nil {
+ return "", fmt.Errorf("error executing template: %v", err)
+ }
+
+ return buf.String(), nil
+}
+
+func (f *firecrawl) IsAvailable() bool {
+ return f.apiKey() != ""
+}
+
+func (f *firecrawl) apiKey() string {
+ if f.cfg == nil {
+ return ""
+ }
+
+ return f.cfg.FirecrawlAPIKey
+}
+
+func (f *firecrawl) searchURL() string {
+ baseURL := firecrawlDefaultURL
+ if f.cfg != nil && f.cfg.FirecrawlAPIURL != "" {
+ baseURL = f.cfg.FirecrawlAPIURL
+ }
+
+ return strings.TrimRight(baseURL, "/") + firecrawlSearchPath
+}
diff --git a/backend/pkg/tools/firecrawl_test.go b/backend/pkg/tools/firecrawl_test.go
new file mode 100644
index 00000000..cb2f96d3
--- /dev/null
+++ b/backend/pkg/tools/firecrawl_test.go
@@ -0,0 +1,515 @@
+package tools
+
+import (
+ "context"
+ "io"
+ "net/http"
+ "strings"
+ "testing"
+
+ "pentagi/pkg/config"
+ "pentagi/pkg/database"
+)
+
+const testFirecrawlAPIKey = "test-key"
+
+func testFirecrawlConfig() *config.Config {
+ return &config.Config{FirecrawlAPIKey: testFirecrawlAPIKey}
+}
+
+func TestFirecrawlHandle(t *testing.T) {
+ var seenRequest bool
+ var receivedMethod string
+ var receivedContentType string
+ var receivedAuthorization string
+ var receivedBody []byte
+
+ mockMux := http.NewServeMux()
+ mockMux.HandleFunc("/v2/search", func(w http.ResponseWriter, r *http.Request) {
+ seenRequest = true
+ receivedMethod = r.Method
+ receivedContentType = r.Header.Get("Content-Type")
+ receivedAuthorization = r.Header.Get("Authorization")
+
+ var err error
+ receivedBody, err = io.ReadAll(r.Body)
+ if err != nil {
+ t.Errorf("failed to read request body: %v", err)
+ }
+
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(http.StatusOK)
+ w.Write([]byte(`{"success":true,"data":{"web":[{"title":"Doc","description":"short","url":"https://example.com","markdown":"long markdown content","metadata":{"title":"Doc","sourceURL":"https://example.com"}}]}}`))
+ })
+
+ proxy, err := newTestProxy("api.firecrawl.dev", mockMux)
+ if err != nil {
+ t.Fatalf("failed to create proxy: %v", err)
+ }
+ defer proxy.Close()
+
+ flowID := int64(1)
+ taskID := int64(10)
+ subtaskID := int64(20)
+ slp := &searchLogProviderMock{}
+
+ cfg := &config.Config{
+ FirecrawlAPIKey: testFirecrawlAPIKey,
+ ProxyURL: proxy.URL(),
+ ExternalSSLCAPath: proxy.CACertPath(),
+ ExternalSSLInsecure: false,
+ }
+
+ fc := NewFirecrawlTool(cfg, flowID, &taskID, &subtaskID, slp, nil)
+
+ ctx := PutAgentContext(t.Context(), database.MsgchainTypeSearcher)
+ got, err := fc.Handle(
+ ctx,
+ FirecrawlToolName,
+ []byte(`{"query":"test query","max_results":5,"message":"m"}`),
+ )
+ if err != nil {
+ t.Fatalf("Handle() unexpected error: %v", err)
+ }
+
+ // Verify mock handler was called
+ if !seenRequest {
+ t.Fatal("request was not intercepted by proxy - mock handler was not called")
+ }
+
+ // Verify request was built correctly
+ if receivedMethod != http.MethodPost {
+ t.Errorf("request method = %q, want POST", receivedMethod)
+ }
+ if receivedContentType != "application/json" {
+ t.Errorf("Content-Type = %q, want application/json", receivedContentType)
+ }
+ if receivedAuthorization != "Bearer "+testFirecrawlAPIKey {
+ t.Errorf("Authorization = %q, want %q", receivedAuthorization, "Bearer "+testFirecrawlAPIKey)
+ }
+ if !strings.Contains(string(receivedBody), `"query":"test query"`) {
+ t.Errorf("request body = %q, expected to contain query", string(receivedBody))
+ }
+ if !strings.Contains(string(receivedBody), `"limit":5`) {
+ t.Errorf("request body = %q, expected to contain limit", string(receivedBody))
+ }
+ if !strings.Contains(string(receivedBody), `"markdown"`) {
+ t.Errorf("request body = %q, expected to contain markdown format", string(receivedBody))
+ }
+
+ // Verify response was parsed correctly
+ if !strings.Contains(got, "# Links") {
+ t.Errorf("result missing '# Links' section: %q", got)
+ }
+ if !strings.Contains(got, "https://example.com") {
+ t.Errorf("result missing expected URL 'https://example.com': %q", got)
+ }
+
+ // Verify search log was written with agent context
+ if slp.calls != 1 {
+ t.Errorf("PutLog() calls = %d, want 1", slp.calls)
+ }
+ if slp.engine != database.SearchengineTypeFirecrawl {
+ t.Errorf("engine = %q, want %q", slp.engine, database.SearchengineTypeFirecrawl)
+ }
+ if slp.query != "test query" {
+ t.Errorf("logged query = %q, want %q", slp.query, "test query")
+ }
+ if slp.parentType != database.MsgchainTypeSearcher {
+ t.Errorf("parent agent type = %q, want %q", slp.parentType, database.MsgchainTypeSearcher)
+ }
+ if slp.currType != database.MsgchainTypeSearcher {
+ t.Errorf("current agent type = %q, want %q", slp.currType, database.MsgchainTypeSearcher)
+ }
+ if slp.taskID == nil || *slp.taskID != taskID {
+ t.Errorf("task ID = %v, want %d", slp.taskID, taskID)
+ }
+ if slp.subtaskID == nil || *slp.subtaskID != subtaskID {
+ t.Errorf("subtask ID = %v, want %d", slp.subtaskID, subtaskID)
+ }
+}
+
+func TestFirecrawlIsAvailable(t *testing.T) {
+ tests := []struct {
+ name string
+ cfg *config.Config
+ want bool
+ }{
+ {
+ name: "available when API key is set",
+ cfg: testFirecrawlConfig(),
+ want: true,
+ },
+ {
+ name: "unavailable when API key is empty",
+ cfg: &config.Config{},
+ want: false,
+ },
+ {
+ name: "unavailable when nil config",
+ cfg: nil,
+ want: false,
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ fc := &firecrawl{cfg: tt.cfg}
+ if got := fc.IsAvailable(); got != tt.want {
+ t.Errorf("IsAvailable() = %v, want %v", got, tt.want)
+ }
+ })
+ }
+}
+
+func TestFirecrawlSearchURL(t *testing.T) {
+ tests := []struct {
+ name string
+ cfg *config.Config
+ want string
+ }{
+ {
+ name: "default when URL is empty",
+ cfg: &config.Config{},
+ want: "https://api.firecrawl.dev/v2/search",
+ },
+ {
+ name: "custom self-hosted URL",
+ cfg: &config.Config{FirecrawlAPIURL: "https://firecrawl.internal:3002"},
+ want: "https://firecrawl.internal:3002/v2/search",
+ },
+ {
+ name: "custom URL with trailing slash",
+ cfg: &config.Config{FirecrawlAPIURL: "https://firecrawl.internal/"},
+ want: "https://firecrawl.internal/v2/search",
+ },
+ {
+ name: "nil config falls back to default",
+ cfg: nil,
+ want: "https://api.firecrawl.dev/v2/search",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ fc := &firecrawl{cfg: tt.cfg}
+ if got := fc.searchURL(); got != tt.want {
+ t.Errorf("searchURL() = %q, want %q", got, tt.want)
+ }
+ })
+ }
+}
+
+func TestFirecrawlParseHTTPResponse_StatusAndDecodeErrors(t *testing.T) {
+ fc := &firecrawl{flowID: 1}
+
+ tests := []struct {
+ name string
+ statusCode int
+ body string
+ wantErr bool
+ errContain string
+ }{
+ {
+ name: "successful response",
+ statusCode: http.StatusOK,
+ body: `{"success":true,"data":{"web":[{"title":"A","description":"c","url":"https://a.com"}]}}`,
+ wantErr: false,
+ },
+ {
+ name: "success false with error message",
+ statusCode: http.StatusOK,
+ body: `{"success":false,"error":"Request timed out"}`,
+ wantErr: true,
+ errContain: "Request timed out",
+ },
+ {
+ name: "decode error",
+ statusCode: http.StatusOK,
+ body: "{invalid json",
+ wantErr: true,
+ errContain: "failed to decode response body",
+ },
+ {
+ name: "bad request",
+ statusCode: http.StatusBadRequest,
+ body: "",
+ wantErr: true,
+ errContain: "invalid",
+ },
+ {
+ name: "unauthorized",
+ statusCode: http.StatusUnauthorized,
+ body: "",
+ wantErr: true,
+ errContain: "API key",
+ },
+ {
+ name: "payment required",
+ statusCode: http.StatusPaymentRequired,
+ body: "",
+ wantErr: true,
+ errContain: "credits",
+ },
+ {
+ name: "forbidden",
+ statusCode: http.StatusForbidden,
+ body: "",
+ wantErr: true,
+ errContain: "administrators only",
+ },
+ {
+ name: "not found",
+ statusCode: http.StatusNotFound,
+ body: "",
+ wantErr: true,
+ errContain: "could not be found",
+ },
+ {
+ name: "method not allowed",
+ statusCode: http.StatusMethodNotAllowed,
+ body: "",
+ wantErr: true,
+ errContain: "invalid method",
+ },
+ {
+ name: "request timeout",
+ statusCode: http.StatusRequestTimeout,
+ body: "",
+ wantErr: true,
+ errContain: "timed out",
+ },
+ {
+ name: "too many requests",
+ statusCode: http.StatusTooManyRequests,
+ body: "",
+ wantErr: true,
+ errContain: "too many",
+ },
+ {
+ name: "internal server error",
+ statusCode: http.StatusInternalServerError,
+ body: "",
+ wantErr: true,
+ errContain: "server",
+ },
+ {
+ name: "bad gateway",
+ statusCode: http.StatusBadGateway,
+ body: "",
+ wantErr: true,
+ errContain: "server",
+ },
+ {
+ name: "service unavailable",
+ statusCode: http.StatusServiceUnavailable,
+ body: "",
+ wantErr: true,
+ errContain: "offline",
+ },
+ {
+ name: "gateway timeout",
+ statusCode: http.StatusGatewayTimeout,
+ body: "",
+ wantErr: true,
+ errContain: "offline",
+ },
+ {
+ name: "unknown status code",
+ statusCode: 418,
+ body: "",
+ wantErr: true,
+ errContain: "unexpected status code",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ resp := &http.Response{
+ StatusCode: tt.statusCode,
+ Body: io.NopCloser(strings.NewReader(tt.body)),
+ }
+ result, err := fc.parseHTTPResponse(t.Context(), resp)
+
+ if !tt.wantErr {
+ if err != nil {
+ t.Errorf("parseHTTPResponse() unexpected error: %v", err)
+ }
+ if !strings.Contains(result, "# Links") {
+ t.Errorf("parseHTTPResponse() result missing '# Links': %q", result)
+ }
+ return
+ }
+
+ if err == nil {
+ t.Fatal("parseHTTPResponse() expected error, got nil")
+ }
+ if !strings.Contains(err.Error(), tt.errContain) {
+ t.Errorf("parseHTTPResponse() error = %q, want to contain %q", err.Error(), tt.errContain)
+ }
+ })
+ }
+}
+
+func TestFirecrawlBuildResult_WithSummarizer(t *testing.T) {
+ t.Run("uses summarizer when markdown content exists", func(t *testing.T) {
+ fc := &firecrawl{
+ summarizer: func(ctx context.Context, prompt string) (string, error) {
+ if !strings.Contains(prompt, "<raw_content") {
+ t.Fatalf("summarizer prompt must include raw content, got: %q", prompt)
+ }
+ return "short summary", nil
+ },
+ }
+
+ out := fc.buildFirecrawlResult(t.Context(), &firecrawlSearchResult{
+ Success: true,
+ Data: firecrawlData{
+ Web: []firecrawlResult{
+ {
+ Title: "Title",
+ Description: "content",
+ URL: "https://example.com",
+ Markdown: "very long markdown content",
+ },
+ },
+ },
+ })
+
+ if !strings.Contains(out, "### Summarized Content") {
+ t.Errorf("buildFirecrawlResult() missing '### Summarized Content', got: %q", out)
+ }
+ if !strings.Contains(out, "short summary") {
+ t.Errorf("buildFirecrawlResult() missing 'short summary', got: %q", out)
+ }
+ })
+
+ t.Run("falls back to markdown content when no summarizer", func(t *testing.T) {
+ fc := &firecrawl{}
+
+ out := fc.buildFirecrawlResult(t.Context(), &firecrawlSearchResult{
+ Success: true,
+ Data: firecrawlData{
+ Web: []firecrawlResult{
+ {
+ Title: "Title",
+ Description: "content",
+ URL: "https://example.com",
+ Markdown: "very long markdown content",
+ },
+ },
+ },
+ })
+
+ if !strings.Contains(out, "### Raw content for") {
+ t.Errorf("buildFirecrawlResult() missing '### Raw content for', got: %q", out)
+ }
+ if !strings.Contains(out, "very long markdown content") {
+ t.Errorf("buildFirecrawlResult() missing markdown content, got: %q", out)
+ }
+ })
+
+ t.Run("no content sections when markdown is empty", func(t *testing.T) {
+ fc := &firecrawl{}
+
+ out := fc.buildFirecrawlResult(t.Context(), &firecrawlSearchResult{
+ Success: true,
+ Data: firecrawlData{
+ Web: []firecrawlResult{
+ {
+ Title: "Title",
+ Description: "content",
+ URL: "https://example.com",
+ },
+ },
+ },
+ })
+
+ if strings.Contains(out, "### Raw content for") {
+ t.Errorf("buildFirecrawlResult() should not have raw content section, got: %q", out)
+ }
+ if strings.Contains(out, "### Summarized Content") {
+ t.Errorf("buildFirecrawlResult() should not have summarized content section, got: %q", out)
+ }
+ })
+
+ t.Run("falls back to metadata url and title", func(t *testing.T) {
+ fc := &firecrawl{}
+
+ out := fc.buildFirecrawlResult(t.Context(), &firecrawlSearchResult{
+ Success: true,
+ Data: firecrawlData{
+ Web: []firecrawlResult{
+ {
+ Description: "content",
+ Metadata: &firecrawlMetadata{
+ Title: "Meta Title",
+ SourceURL: "https://meta.example.com",
+ },
+ },
+ },
+ },
+ })
+
+ if !strings.Contains(out, "Meta Title") {
+ t.Errorf("buildFirecrawlResult() missing metadata title, got: %q", out)
+ }
+ if !strings.Contains(out, "https://meta.example.com") {
+ t.Errorf("buildFirecrawlResult() missing metadata source URL, got: %q", out)
+ }
+ })
+}
+
+func TestFirecrawlHandle_ValidationAndSwallowedError(t *testing.T) {
+ t.Run("invalid json", func(t *testing.T) {
+ fc := &firecrawl{cfg: testFirecrawlConfig()}
+ _, err := fc.Handle(t.Context(), FirecrawlToolName, []byte("{"))
+ if err == nil || !strings.Contains(err.Error(), "failed to unmarshal") {
+ t.Fatalf("expected unmarshal error, got: %v", err)
+ }
+ })
+
+ t.Run("search error swallowed", func(t *testing.T) {
+ var seenRequest bool
+ mockMux := http.NewServeMux()
+ mockMux.HandleFunc("/v2/search", func(w http.ResponseWriter, r *http.Request) {
+ seenRequest = true
+ w.WriteHeader(http.StatusBadGateway)
+ })
+
+ proxy, err := newTestProxy("api.firecrawl.dev", mockMux)
+ if err != nil {
+ t.Fatalf("failed to create proxy: %v", err)
+ }
+ defer proxy.Close()
+
+ fc := &firecrawl{
+ flowID: 1,
+ cfg: &config.Config{
+ FirecrawlAPIKey: testFirecrawlAPIKey,
+ ProxyURL: proxy.URL(),
+ ExternalSSLCAPath: proxy.CACertPath(),
+ ExternalSSLInsecure: false,
+ },
+ }
+
+ result, err := fc.Handle(
+ t.Context(),
+ FirecrawlToolName,
+ []byte(`{"query":"q","max_results":5,"message":"m"}`),
+ )
+ if err != nil {
+ t.Fatalf("Handle() unexpected error: %v", err)
+ }
+
+ // Verify mock handler was called (request was intercepted)
+ if !seenRequest {
+ t.Error("request was not intercepted by proxy - mock handler was not called")
+ }
+
+ // Verify error was swallowed and returned as string
+ if !strings.Contains(result, "failed to search in firecrawl") {
+ t.Errorf("Handle() = %q, expected swallowed error message", result)
+ }
+ })
+}
diff --git a/backend/pkg/tools/registry.go b/backend/pkg/tools/registry.go
index 8041b5da..bd21a413 100644
--- a/backend/pkg/tools/registry.go
+++ b/backend/pkg/tools/registry.go
@@ -24,6 +24,7 @@ const (
GoogleToolName = "google"
DuckDuckGoToolName = "duckduckgo"
TavilyToolName = "tavily"
+ FirecrawlToolName = "firecrawl"
TraversaalToolName = "traversaal"
PerplexityToolName = "perplexity"
SearxngToolName = "searxng"
@@ -109,6 +110,7 @@ var toolsTypeMapping = map[string]ToolType{
GoogleToolName: SearchNetworkToolType,
DuckDuckGoToolName: SearchNetworkToolType,
TavilyToolName: SearchNetworkToolType,
+ FirecrawlToolName: SearchNetworkToolType,
TraversaalToolName: SearchNetworkToolType,
PerplexityToolName: SearchNetworkToolType,
SearxngToolName: SearchNetworkToolType,
@@ -153,6 +155,7 @@ var allowedStoringInMemoryTools = []string{
GoogleToolName,
DuckDuckGoToolName,
TavilyToolName,
+ FirecrawlToolName,
TraversaalToolName,
PerplexityToolName,
SearxngToolName,
@@ -230,6 +233,13 @@ var registryDefinitions = map[string]llms.FunctionDefinition{
"with answer by query and detailed information from the web sites",
Parameters: reflector.Reflect(&SearchAction{}),
},
+ FirecrawlToolName: {
+ Name: FirecrawlToolName,
+ Description: "Search in the firecrawl search engine, it combines web search with page scraping to return " +
+ "full-page markdown content for each result, ideal for deep research on complex technical topics " +
+ "and reading documentation directly from the discovered web sites",
+ Parameters: reflector.Reflect(&SearchAction{}),
+ },
TraversaalToolName: {
Name: TraversaalToolName,
Description: "Search in the traversaal search engine, presents you answer and web-links " +
@@ -439,8 +449,8 @@ func getMessageType(name string) database.MsglogType {
return database.MsglogTypeFile
case BrowserToolName:
return database.MsglogTypeBrowser
- case MemoristToolName, SearchToolName, GoogleToolName, DuckDuckGoToolName, TavilyToolName, TraversaalToolName,
- PerplexityToolName, SearxngToolName, SploitusToolName,
+ case MemoristToolName, SearchToolName, GoogleToolName, DuckDuckGoToolName, TavilyToolName, FirecrawlToolName,
+ TraversaalToolName, PerplexityToolName, SearxngToolName, SploitusToolName,
SearchGuideToolName, SearchAnswerToolName, SearchCodeToolName, SearchInMemoryToolName, GraphitiSearchToolName:
return database.MsglogTypeSearch
case AdviceToolName:
diff --git a/backend/pkg/tools/tools.go b/backend/pkg/tools/tools.go
index 2c72884b..db6cc0e2 100644
--- a/backend/pkg/tools/tools.go
+++ b/backend/pkg/tools/tools.go
@@ -939,6 +939,17 @@ func (fte *flowToolsExecutor) GetAssistantExecutor(cfg AssistantExecutorConfig)
handlers[TavilyToolName] = tavily.Handle
}
+ firecrawl := NewFirecrawlTool(
+ fte.cfg,
+ fte.flowID, nil, nil,
+ fte.slp,
+ cfg.Summarizer,
+ )
+ if firecrawl.IsAvailable() {
+ definitions = append(definitions, registryDefinitions[FirecrawlToolName])
+ handlers[FirecrawlToolName] = firecrawl.Handle
+ }
+
traversaal := NewTraversaalTool(
fte.cfg,
fte.flowID, nil, nil,
@@ -1541,6 +1552,19 @@ func (fte *flowToolsExecutor) GetSearcherExecutor(cfg SearcherExecutorConfig) (C
ce.handlers[TavilyToolName] = tavily.Handle
}
+ firecrawl := NewFirecrawlTool(
+ fte.cfg,
+ fte.flowID,
+ cfg.TaskID,
+ cfg.SubtaskID,
+ fte.slp,
+ cfg.Summarizer,
+ )
+ if firecrawl.IsAvailable() {
+ ce.definitions = append(ce.definitions, registryDefinitions[FirecrawlToolName])
+ ce.handlers[FirecrawlToolName] = firecrawl.Handle
+ }
+
traversaal := NewTraversaalTool(
fte.cfg,
fte.flowID,
diff --git a/docker-compose.yml b/docker-compose.yml
index 20cc37d4..5a2a6f7e 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -151,6 +151,8 @@ services:
- GOOGLE_LR_KEY=${GOOGLE_LR_KEY:-}
- TRAVERSAAL_API_KEY=${TRAVERSAAL_API_KEY:-}
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
+ - FIRECRAWL_API_KEY=${FIRECRAWL_API_KEY:-}
+ - FIRECRAWL_API_URL=${FIRECRAWL_API_URL:-https://api.firecrawl.dev}
- PERPLEXITY_API_KEY=${PERPLEXITY_API_KEY:-}
- PERPLEXITY_MODEL=${PERPLEXITY_MODEL:-sonar}
- PERPLEXITY_CONTEXT_SIZE=${PERPLEXITY_CONTEXT_SIZE:-low}