Skip to content

Personal AI, On Personal Devices

Python

feat(tools): add Firecrawl as a web_search engine

Juampif2ccefa7fb27
diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md
index 45aaebf1..c7a39c14 100644
--- a/docs/getting-started/configuration.md
+++ b/docs/getting-started/configuration.md
@@ -1132,7 +1132,9 @@ OpenJarvis respects the following environment variables:
| `YOUDOTCOM_API_KEY` | API key for the You.com web search engine. Optional — raises the keyless free-tier limits and enables You.com Contents extraction for URL queries. |
| `SERPLY_API_KEY` | API key for the Serply web search engine, which proxies Google. Optional. When set and no other search key is, `auto` engine selection prefers Serply over the keyless tier. Keys: [serply.io](https://serply.io). |
| `SERPLY_PROXY_LOCATION` | Two-letter country code asking Serply for that country's Google result set, for example `DE`. Optional. Unset means the API answers from its own default region. See [serply.io/docs](https://serply.io/docs). |
-| `OPENJARVIS_WEB_SEARCH_ENGINE` | Web search engine for the `web_search` tool: `auto` (default), `youcom`, `tavily`, `serply`, or `duckduckgo`. |
+| `FIRECRAWL_API_KEY` | API key for the Firecrawl web search engine. Optional. When set and no other search key is, `auto` engine selection prefers Firecrawl over the keyless tier, and URL queries return the page or PDF as markdown. Keys: [firecrawl.dev](https://www.firecrawl.dev/app/api-keys?utm_source=openjarvis&utm_medium=integration). |
+| `FIRECRAWL_API_URL` | Base URL of the Firecrawl API, for example a [self-hosted Firecrawl](https://docs.firecrawl.dev/contributing/self-host). Optional. Defaults to `https://api.firecrawl.dev`. Any other URL enables the Firecrawl engine without a key, for instances that run without auth. |
+| `OPENJARVIS_WEB_SEARCH_ENGINE` | Web search engine for the `web_search` tool: `auto` (default), `youcom`, `tavily`, `serply`, `firecrawl`, or `duckduckgo`. |
In the desktop app, you can save `ATLASCLOUD_API_KEY` in **Cloud Models** or
**Settings → API Keys** instead of setting an environment variable.
diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md
index c44e8c26..10075d39 100644
--- a/docs/getting-started/quickstart.md
+++ b/docs/getting-started/quickstart.md
@@ -137,22 +137,24 @@ You get a ChatGPT-like interface with streaming responses, tool use, energy moni
Web search works with no configuration: queries go to the You.com keyless
free tier, which is rate limited per IP and needs no signup. To raise those
-limits, or to use Tavily or Serply instead, add `YOUDOTCOM_API_KEY`,
-`TAVILY_API_KEY` or `SERPLY_API_KEY` under **Settings → Tools → Web Search**
-after the app starts, or export one before starting quickstart:
+limits, or to use Tavily, Serply or Firecrawl instead, add `YOUDOTCOM_API_KEY`,
+`TAVILY_API_KEY`, `SERPLY_API_KEY` or `FIRECRAWL_API_KEY` under
+**Settings → Tools → Web Search** after the app starts, or export one before
+starting quickstart:
```bash
export YOUDOTCOM_API_KEY="..." # free key: https://you.com/platform
export SERPLY_API_KEY="..." # free tier: https://serply.io
+export FIRECRAWL_API_KEY="..." # free tier: https://www.firecrawl.dev/app/api-keys?utm_source=openjarvis&utm_medium=integration
./scripts/quickstart.sh
```
Set `OPENJARVIS_WEB_SEARCH_ENGINE` to pick an engine explicitly — `youcom`,
-`tavily`, `serply`, or `duckduckgo`. The default, `auto`, uses Tavily when
-`TAVILY_API_KEY` is set, then You.com, then Serply, and settles on the You.com
-keyless tier when no key is set at all. DuckDuckGo remains the fallback
-for every engine; dropping to it is logged at `WARNING`, since its scraped
-results are unranked.
+`tavily`, `serply`, `firecrawl`, or `duckduckgo`. The default, `auto`, uses
+Tavily when `TAVILY_API_KEY` is set, then You.com, then Serply, then Firecrawl,
+and settles on the You.com keyless tier when no key is set at all. DuckDuckGo
+remains the fallback for every engine; dropping to it is logged at `WARNING`,
+since its scraped results are unranked.
Serply proxies Google itself, so Google search operators reach the index
unchanged: `site:arxiv.org` restricts the corpus and `after:2026-01-01` bounds
@@ -162,6 +164,15 @@ that country's result set instead of the default region. Keys and request
options are documented at [serply.io](https://serply.io) and
[serply.io/docs](https://serply.io/docs).
+Firecrawl also changes what happens when a query is a URL: instead of the local
+HTML strip, the page comes back as markdown, including JavaScript-rendered
+pages and PDFs. Set `FIRECRAWL_API_URL` to use a
+[self-hosted Firecrawl](https://docs.firecrawl.dev/contributing/self-host)
+instead of the hosted API; no key is needed if your instance runs without
+auth. Page fetches then run on your instance, and search queries go on to the
+search backend it is configured with. Keys:
+[firecrawl.dev](https://www.firecrawl.dev/app/api-keys?utm_source=openjarvis&utm_medium=integration).
+
The script does not automatically source `.env` files. Run `source .env`
first if that is where you keep the key. Stop any existing OpenJarvis server
before restarting so it inherits the updated environment.
@@ -290,7 +301,7 @@ Agents add multi-turn reasoning and tool-calling capabilities. The `orchestrator
| `retrieval` | Search the memory store for relevant context. |
| `llm` | Make sub-queries to another model. |
| `file_read` | Read files with path validation. |
-| `web_search` | Web search via You.com (keyless by default), Tavily, or Serply, with a DuckDuckGo fallback. |
+| `web_search` | Web search via You.com (keyless by default), Tavily, Serply, or Firecrawl, with a DuckDuckGo fallback. |
### CLI Example
diff --git a/docs/user-guide/deep-research.md b/docs/user-guide/deep-research.md
index b8803f62..c9d6d988 100644
--- a/docs/user-guide/deep-research.md
+++ b/docs/user-guide/deep-research.md
@@ -160,6 +160,6 @@ Smaller chunk sizes work better for code, where each function or class is a natu
**Slow responses** -- The agent makes multiple search passes. Each turn involves a model call. Reduce `max_turns` or use a smaller model (`qwen3.5:4b`) for faster but less thorough results.
-**Web search not working** -- The `web_search` tool needs no API key by default; it uses the You.com keyless free tier, which is rate limited per IP. Set `YOUDOTCOM_API_KEY` (free key at [you.com/platform](https://you.com/platform?utm_source=open-jarvis-openjarvis&utm_medium=oss_integration&utm_campaign=2026-09-oss-integrations&utm_content=docs)) for higher limits, or install Tavily with `uv sync --extra tools-search` and set `TAVILY_API_KEY`. Set `SERPLY_API_KEY` ([serply.io](https://serply.io)) to search through Google itself, which is the only engine here that honours Google operators such as `site:` and `after:` in a research query. A `WARNING` naming the failed engine is logged whenever search drops to the DuckDuckGo fallback.
+**Web search not working** -- The `web_search` tool needs no API key by default; it uses the You.com keyless free tier, which is rate limited per IP. Set `YOUDOTCOM_API_KEY` (free key at [you.com/platform](https://you.com/platform?utm_source=open-jarvis-openjarvis&utm_medium=oss_integration&utm_campaign=2026-09-oss-integrations&utm_content=docs)) for higher limits, or install Tavily with `uv sync --extra tools-search` and set `TAVILY_API_KEY`. Set `SERPLY_API_KEY` ([serply.io](https://serply.io)) to search through Google itself, which is the only engine here that honours Google operators such as `site:` and `after:` in a research query. Set `FIRECRAWL_API_KEY` ([firecrawl.dev](https://www.firecrawl.dev/app/api-keys?utm_source=openjarvis&utm_medium=integration)) to have URL lookups, including PDFs, return markdown. A `WARNING` naming the failed engine is logged whenever search drops to the DuckDuckGo fallback.
**Wrong chunks retrieved** -- Try re-indexing with different chunk sizes. For technical documents, smaller chunks (`256`) often retrieve more precisely. For narrative text, larger chunks (`1024`) preserve more context.
diff --git a/src/openjarvis/core/credentials.py b/src/openjarvis/core/credentials.py
index c81aa1a9..c65f9bd9 100644
--- a/src/openjarvis/core/credentials.py
+++ b/src/openjarvis/core/credentials.py
@@ -33,7 +33,12 @@ def _default_path() -> Path:
TOOL_CREDENTIALS: dict[str, list[str]] = {
- "web_search": ["TAVILY_API_KEY", "YOUDOTCOM_API_KEY", "SERPLY_API_KEY"],
+ "web_search": [
+ "TAVILY_API_KEY",
+ "YOUDOTCOM_API_KEY",
+ "SERPLY_API_KEY",
+ "FIRECRAWL_API_KEY",
+ ],
"get_weather": ["OPENWEATHERMAP_API_KEY"],
"image_generate": ["OPENAI_API_KEY"],
"slack": ["SLACK_BOT_TOKEN", "SLACK_APP_TOKEN"],
@@ -74,7 +79,14 @@ TOOL_CREDENTIALS: dict[str, list[str]] = {
# and every listed key only raises limits, changes the engine, or improves
# result quality.
OPTIONAL_TOOL_CREDENTIALS: dict[str, frozenset[str]] = {
- "web_search": frozenset({"TAVILY_API_KEY", "YOUDOTCOM_API_KEY", "SERPLY_API_KEY"}),
+ "web_search": frozenset(
+ {
+ "TAVILY_API_KEY",
+ "YOUDOTCOM_API_KEY",
+ "SERPLY_API_KEY",
+ "FIRECRAWL_API_KEY",
+ }
+ ),
}
diff --git a/src/openjarvis/security/data_boundary_audit.py b/src/openjarvis/security/data_boundary_audit.py
index 82177274..64a502d6 100644
--- a/src/openjarvis/security/data_boundary_audit.py
+++ b/src/openjarvis/security/data_boundary_audit.py
@@ -13,6 +13,7 @@ import os
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Iterable, Literal
+from urllib.parse import urlparse
from openjarvis.core.credentials import TOOL_CREDENTIALS
@@ -70,6 +71,7 @@ API_KEY_ENV_VARS = {
"TAVILY_API_KEY": ("Tavily web search", {"tavily", "web_search"}),
"YOUDOTCOM_API_KEY": ("You.com web search", {"youcom", "web_search"}),
"SERPLY_API_KEY": ("Serply web search", {"serply", "web_search"}),
+ "FIRECRAWL_API_KEY": ("Firecrawl web search", {"firecrawl", "web_search"}),
}
CHANNEL_SECRET_FIELDS: tuple[tuple[str, str, str], ...] = (
@@ -953,11 +955,15 @@ def _audit_tool_surfaces(config: Any, builder: _FindingBuilder) -> None:
if tools & WEB_SEARCH_TOOLS:
destination = _web_search_destination()
+ data = "user or agent search query"
+ if destination.startswith("Firecrawl"):
+ # URL queries are scraped by Firecrawl too, not fetched locally.
+ data += " and URL lookups"
builder.add(
finding_id="web-search-tool-configured",
status="warn",
title="Web search tool is configured",
- potential_data_path=(f"user or agent search query -> {destination}"),
+ potential_data_path=(f"{data} -> {destination}"),
evidence=(
f"configured tool(s) = {_format_tools(tools & WEB_SEARCH_TOOLS)}"
),
@@ -1077,6 +1083,14 @@ def _audit_tool_surfaces(config: Any, builder: _FindingBuilder) -> None:
)
+def _firecrawl_self_hosted_url() -> str | None:
+ """Return ``FIRECRAWL_API_URL`` when it points away from the hosted API."""
+ api_url = (os.environ.get("FIRECRAWL_API_URL") or "").strip().rstrip("/")
+ if api_url and api_url != "https://api.firecrawl.dev":
+ return api_url
+ return None
+
+
def _web_search_destination() -> str:
"""Name the service ``web_search`` sends queries to, from env keys alone.
@@ -1086,7 +1100,7 @@ def _web_search_destination() -> str:
source of truth; ``test_data_boundary_audit`` asserts the two agree.
"""
engine = (os.environ.get("OPENJARVIS_WEB_SEARCH_ENGINE") or "auto").strip().lower()
- if engine not in {"auto", "youcom", "tavily", "duckduckgo", "serply"}:
+ if engine not in {"auto", "youcom", "tavily", "duckduckgo", "serply", "firecrawl"}:
engine = "auto"
if engine == "auto":
if os.environ.get("TAVILY_API_KEY"):
@@ -1095,6 +1109,8 @@ def _web_search_destination() -> str:
engine = "youcom"
elif os.environ.get("SERPLY_API_KEY"):
engine = "serply"
+ elif os.environ.get("FIRECRAWL_API_KEY") or _firecrawl_self_hosted_url():
+ engine = "firecrawl"
else:
engine = "youcom"
@@ -1107,6 +1123,16 @@ def _web_search_destination() -> str:
if location:
return f"Serply web search API (Google SERP, region {location})"
return "Serply web search API (Google SERP)"
+ if engine == "firecrawl":
+ api_url = _firecrawl_self_hosted_url()
+ if api_url:
+ # Name the host only: a self-hosted URL can carry credentials.
+ host = urlparse(api_url).hostname or "custom URL"
+ return (
+ f"Firecrawl search API (self-hosted at {host}; search queries go "
+ "on to its configured search backend)"
+ )
+ return "Firecrawl search API"
tier = "keyed" if os.environ.get("YOUDOTCOM_API_KEY") else "keyless free tier"
return f"You.com web search API ({tier})"
diff --git a/src/openjarvis/tools/web_search.py b/src/openjarvis/tools/web_search.py
index 75558e37..c6abea8d 100644
--- a/src/openjarvis/tools/web_search.py
+++ b/src/openjarvis/tools/web_search.py
@@ -8,6 +8,9 @@ API-backed engine over the DuckDuckGo HTML scrape:
* ``TAVILY_API_KEY`` set → Tavily (unchanged for existing installs)
* ``YOUDOTCOM_API_KEY`` set → You.com, keyed
* ``SERPLY_API_KEY`` set → Serply, a Google SERP proxy
+* ``FIRECRAWL_API_KEY`` set, or ``FIRECRAWL_API_URL`` pointing at a
+ self-hosted instance → Firecrawl, which also returns URL lookups (including
+ PDFs) as markdown
* none of them → You.com, keyless free tier (no signup, rate limited per IP)
The keyless tier is what makes a fresh install API-backed with zero config.
@@ -43,8 +46,29 @@ SERPLY_API_KEY_ENV = "SERPLY_API_KEY"
# set. Unset means the API answers from its own default region.
SERPLY_LOCATION_ENV = "SERPLY_PROXY_LOCATION"
+FIRECRAWL_API_KEY_ENV = "FIRECRAWL_API_KEY"
+# Base URL of the Firecrawl API. Point it at a self-hosted Firecrawl to run
+# page fetches on an instance you operate; that instance still sends search
+# queries to its own configured search backend.
+FIRECRAWL_API_URL_ENV = "FIRECRAWL_API_URL"
+FIRECRAWL_DEFAULT_API_URL = "https://api.firecrawl.dev"
+FIRECRAWL_ORIGIN = "openjarvis"
+# PDFs are billed per page, and only the first max_chars of the markdown are
+# kept, so parsing stops after a few pages.
+FIRECRAWL_PDF_MAX_PAGES = 3
+# Server-side scrape timeout, below the client timeout so a slow page fails on
+# Firecrawl's side first instead of being abandoned mid-scrape.
+FIRECRAWL_SCRAPE_TIMEOUT_MS = 45000
+# Search descriptions can run to several thousand characters of page text, so
+# each one is cut to a snippet that a small local model can take five of.
+FIRECRAWL_SNIPPET_CHARS = 500
+FIRECRAWL_KEYS_URL = (
+ "https://www.firecrawl.dev/app/api-keys"
+ "?utm_source=openjarvis&utm_medium=integration"
+)
+
ENGINE_ENV = "OPENJARVIS_WEB_SEARCH_ENGINE"
-ENGINES = ("auto", "youcom", "tavily", "duckduckgo", "serply")
+ENGINES = ("auto", "youcom", "tavily", "duckduckgo", "serply", "firecrawl")
# Identifies OpenJarvis to You.com. The keyless tier carries no API key, so the
# User-Agent is the only attribution signal; sent to You.com hosts only.
@@ -89,6 +113,8 @@ class WebSearchTool(BaseTool):
youcom_api_key: str | None = None,
serply_api_key: str | None = None,
serply_location: str | None = None,
+ firecrawl_api_key: str | None = None,
+ firecrawl_api_url: str | None = None,
):
"""Configure the search engine.
@@ -102,6 +128,19 @@ class WebSearchTool(BaseTool):
self._youcom_api_key = youcom_api_key or os.environ.get(YOUCOM_API_KEY_ENV)
self._serply_api_key = serply_api_key or os.environ.get(SERPLY_API_KEY_ENV)
self._serply_location = serply_location or os.environ.get(SERPLY_LOCATION_ENV)
+ self._firecrawl_api_key = firecrawl_api_key or os.environ.get(
+ FIRECRAWL_API_KEY_ENV
+ )
+ self._firecrawl_api_url = (
+ firecrawl_api_url
+ or os.environ.get(FIRECRAWL_API_URL_ENV)
+ or FIRECRAWL_DEFAULT_API_URL
+ ).rstrip("/")
+ # Self-hosted instances usually run without auth, so a custom URL
+ # enables the engine even when no key is set.
+ self._firecrawl_self_hosted = (
+ self._firecrawl_api_url != FIRECRAWL_DEFAULT_API_URL
+ )
self._max_results = max_results
requested = (engine or os.environ.get(ENGINE_ENV) or "auto").strip().lower()
@@ -119,9 +158,10 @@ class WebSearchTool(BaseTool):
Keyed engines come first, in the order they were added, so every
install that already had a key resolves exactly as it did before:
- Tavily, then keyed You.com, then Serply. The keyless You.com tier is
- last and remains the zero-config API-backed path, which means a Serply
- key only ever wins over having no key at all.
+ Tavily, then keyed You.com, then Serply, then Firecrawl. The keyless
+ You.com tier is last and remains the zero-config API-backed path, which
+ means a Serply or Firecrawl key only ever wins over having no key at
+ all.
"""
if self._engine != "auto":
return self._engine
@@ -131,6 +171,8 @@ class WebSearchTool(BaseTool):
return "youcom"
if self._serply_api_key:
return "serply"
+ if self._firecrawl_api_key or self._firecrawl_self_hosted:
+ return "firecrawl"
return "youcom"
@property
@@ -164,6 +206,7 @@ class WebSearchTool(BaseTool):
"TAVILY_API_KEY",
YOUCOM_API_KEY_ENV,
SERPLY_API_KEY_ENV,
+ FIRECRAWL_API_KEY_ENV,
],
"engine": engine,
"engines": list(ENGINES),
@@ -435,6 +478,139 @@ class WebSearchTool(BaseTool):
metadata=metadata,
)
+ def _firecrawl_headers(self) -> dict[str, str]:
+ headers = {"Accept": "application/json", "Content-Type": "application/json"}
+ if self._firecrawl_api_key:
+ headers["Authorization"] = f"Bearer {self._firecrawl_api_key}"
+ return headers
+
+ @staticmethod
+ def _format_firecrawl_results(payload: dict[str, Any]) -> tuple[str, int]:
+ """Render a Firecrawl search payload in the shared result format.
+
+ Only snippets are returned, each description cut to
+ :data:`FIRECRAWL_SNIPPET_CHARS`. Search can also scrape every result
+ page, but full pages per result would crowd out a small local model's
+ context, so URL lookups are the place to read a page in full.
+ """
+ data = payload.get("data") if isinstance(payload, dict) else None
+ if not isinstance(data, dict):
+ data = {}
+ parts: list[str] = []
+ count = 0
+ for item in data.get("web") or []:
+ title = item.get("title") or "Untitled"
+ url = item.get("url", "")
+ content = item.get("description") or ""
+ if len(content) > FIRECRAWL_SNIPPET_CHARS:
+ content = content[:FIRECRAWL_SNIPPET_CHARS].rstrip() + "…"
+ parts.append(f"### {title}\nSource: {url}\nSummary: {content}")
+ count += 1
+ return "\n\n---\n\n".join(parts), count
+
+ def _firecrawl_search(self, query: str, max_results: int) -> ToolResult:
+ """Search via Firecrawl, or the instance at ``FIRECRAWL_API_URL``."""
+ import httpx
+
+ if not self._firecrawl_api_key and not self._firecrawl_self_hosted:
+ raise _WebSearchEngineError(f"{FIRECRAWL_API_KEY_ENV} is not set")
+ try:
+ response = httpx.post(
+ f"{self._firecrawl_api_url}/v2/search",
+ json={
+ "query": query,
+ "limit": max_results,
+ "origin": FIRECRAWL_ORIGIN,
+ },
+ headers=self._firecrawl_headers(),
+ timeout=30.0,
+ )
+ response.raise_for_status()
+ payload = response.json()
+ except httpx.HTTPStatusError as exc:
+ status = exc.response.status_code
+ if status == 401:
+ raise _WebSearchEngineError(
+ f"Firecrawl rejected the credential in {FIRECRAWL_API_KEY_ENV}"
+ ) from exc
+ if status == 402:
+ raise _WebSearchEngineError(
+ "Firecrawl credits are exhausted for this key. Check the "
+ f"plan or create a new key at {FIRECRAWL_KEYS_URL}"
+ ) from exc
+ if status == 429:
+ raise _WebSearchEngineError(
+ "Firecrawl rate limit reached (HTTP 429)"
+ ) from exc
+ raise _WebSearchEngineError(
+ f"Firecrawl search failed with HTTP {status}"
+ ) from exc
+ except (httpx.HTTPError, ValueError) as exc:
+ raise _WebSearchEngineError(f"Firecrawl search failed: {exc}") from exc
+
+ formatted, count = self._format_firecrawl_results(payload)
+ metadata: dict[str, Any] = {"num_results": count, "engine": "firecrawl"}
+ if isinstance(payload, dict) and "creditsUsed" in payload:
+ metadata["credits"] = payload["creditsUsed"]
+ return ToolResult(
+ tool_name="web_search",
+ content=formatted or "No results found.",
+ success=True,
+ metadata=metadata,
+ )
+
+ def _firecrawl_extract(self, url: str, max_chars: int = 6000) -> str | None:
+ """Return Firecrawl-scraped markdown for ``url``, or ``None``.
+
+ Firecrawl renders JavaScript and parses PDFs, which the regex HTML
+ strip in :meth:`_fetch_url` cannot. Output is truncated at the same
+ ``max_chars`` as the local path so a long PDF cannot flood the
+ context, and PDF parsing stops after :data:`FIRECRAWL_PDF_MAX_PAGES`
+ pages since PDFs are billed per page. Any failure returns ``None`` and
+ the caller falls back to the local fetch.
+ """
+ if not self._firecrawl_api_key and not self._firecrawl_self_hosted:
+ return None
+
+ import httpx
+
+ try:
+ response = httpx.post(
+ f"{self._firecrawl_api_url}/v2/scrape",
+ json={
+ "url": url,
+ "formats": ["markdown"],
+ "onlyMainContent": True,
+ "parsers": [{"type": "pdf", "maxPages": FIRECRAWL_PDF_MAX_PAGES}],
+ "timeout": FIRECRAWL_SCRAPE_TIMEOUT_MS,
+ "origin": FIRECRAWL_ORIGIN,
+ },
+ headers=self._firecrawl_headers(),
+ timeout=60.0,
+ )
+ response.raise_for_status()
+ payload = response.json()
+ except (httpx.HTTPError, ValueError) as exc:
+ reason = type(exc).__name__
+ if isinstance(exc, httpx.HTTPStatusError):
+ reason = f"HTTP {exc.response.status_code}"
+ logger.warning(
+ "Firecrawl scrape failed for %s (%s); falling back to local "
+ "HTML extraction",
+ url,
+ reason,
+ )
+ return None
+
+ data = payload.get("data") if isinstance(payload, dict) else None
+ markdown = data.get("markdown") if isinstance(data, dict) else None
+ if not markdown:
+ return None
+ text = str(markdown)
+ if len(text) > max_chars:
+ text = text[:max_chars] + "\n\n[Content truncated]"
+ return text
+
def _duckduckgo_search(self, query: str, max_results: int) -> str:
"""Search using DuckDuckGo as fallback."""
from ddgs import DDGS
@@ -463,6 +639,26 @@ class WebSearchTool(BaseTool):
# If the query contains a URL, fetch it directly instead of searching
url = self._extract_url(query) if not self._is_url(query) else query.strip()
if url:
+ if self._resolve_engine() == "firecrawl":
+ ssrf_error = check_ssrf(self._normalize_url(url))
+ if ssrf_error:
+ return ToolResult(
+ tool_name="web_search",
+ content=f"Failed to fetch URL: {ssrf_error}",
+ success=False,
+ )
+ extracted = self._firecrawl_extract(url)
+ if extracted:
+ return ToolResult(
+ tool_name="web_search",
+ content=extracted,
+ success=True,
+ metadata={
+ "url": url,
+ "mode": "fetch",
+ "extractor": "firecrawl_scrape",
+ },
+ )
if self._resolve_engine() == "youcom":
ssrf_error = check_ssrf(self._normalize_url(url))
if ssrf_error:
@@ -509,6 +705,8 @@ class WebSearchTool(BaseTool):
return self._youcom_search(query, max_results)
if engine == "serply":
return self._serply_search(query, max_results)
+ if engine == "firecrawl":
+ return self._firecrawl_search(query, max_results)
return self._tavily_search(query, max_results)
except _WebSearchEngineError as exc:
reason = str(exc)
diff --git a/tests/core/test_credentials.py b/tests/core/test_credentials.py
index 603aaed6..fbd4e4cd 100644
--- a/tests/core/test_credentials.py
+++ b/tests/core/test_credentials.py
@@ -187,6 +187,7 @@ class TestOptionalCredentials:
assert is_credential_optional("web_search", "TAVILY_API_KEY") is True
assert is_credential_optional("web_search", "YOUDOTCOM_API_KEY") is True
assert is_credential_optional("web_search", "SERPLY_API_KEY") is True
+ assert is_credential_optional("web_search", "FIRECRAWL_API_KEY") is True
def test_other_tool_keys_stay_required(self):
from openjarvis.core.credentials import is_credential_optional
@@ -226,3 +227,15 @@ class TestOptionalCredentials:
assert load_credentials(path=path)["web_search"]["SERPLY_API_KEY"] == (
"srp-key"
)
+
+ def test_firecrawl_key_can_be_persisted(self, tmp_path, monkeypatch):
+ """Same guard for the Firecrawl key: declared, so the Settings UI can
+ store it, and still not required for web_search to run."""
+ from openjarvis.core.credentials import load_credentials, save_credential
+
+ path = tmp_path / "credentials.toml"
+ monkeypatch.delenv("FIRECRAWL_API_KEY", raising=False)
+ save_credential("web_search", "FIRECRAWL_API_KEY", "fc-test", path=path)
+ assert load_credentials(path=path)["web_search"]["FIRECRAWL_API_KEY"] == (
+ "fc-test"
+ )
diff --git a/tests/security/test_data_boundary_audit.py b/tests/security/test_data_boundary_audit.py
index 2214b4da..c700d1c2 100644
--- a/tests/security/test_data_boundary_audit.py
+++ b/tests/security/test_data_boundary_audit.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+import itertools
import os
import pytest
@@ -337,6 +338,23 @@ def test_web_search_tool_and_tavily_env_are_correlated(
assert "secret-value" not in str(report.to_dict(show_paths=True))
+def test_firecrawl_web_search_data_path_includes_url_lookups(tmp_path, monkeypatch):
+ """With Firecrawl, URL queries are scraped by the service rather than
+ fetched locally, so the finding has to say URLs leave the machine too."""
+ config = _low_noise_config()
+ config.tools.enabled = "web_search"
+ for key in ("TAVILY_API_KEY", "YOUDOTCOM_API_KEY", "SERPLY_API_KEY"):
+ monkeypatch.delenv(key, raising=False)
+ monkeypatch.delenv("OPENJARVIS_WEB_SEARCH_ENGINE", raising=False)
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+
+ report = build_data_boundary_report(config, tmp_path)
+
+ findings = {finding.id: finding for finding in report.findings}
+ path = findings["web-search-tool-configured"].potential_data_path
+ assert path == "user or agent search query and URL lookups -> Firecrawl search API"
+
+
def test_mcp_servers_are_flagged_when_non_empty(tmp_path):
config = _low_noise_config()
config.tools.mcp.enabled = True
@@ -1354,6 +1372,8 @@ class TestWebSearchDestination:
"YOUDOTCOM_API_KEY",
"SERPLY_API_KEY",
"SERPLY_PROXY_LOCATION",
+ "FIRECRAWL_API_KEY",
+ "FIRECRAWL_API_URL",
"OPENJARVIS_WEB_SEARCH_ENGINE",
):
monkeypatch.delenv(key, raising=False)
@@ -1403,6 +1423,33 @@ class TestWebSearchDestination:
monkeypatch.setenv("SERPLY_PROXY_LOCATION", "DE")
assert "region DE" in _web_search_destination()
+ def test_firecrawl_is_named(self, monkeypatch):
+ from openjarvis.security.data_boundary_audit import _web_search_destination
+
+ self._clear(monkeypatch)
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ assert _web_search_destination() == "Firecrawl search API"
+
+ def test_firecrawl_self_hosted_names_only_the_host(self, monkeypatch):
+ """A self-hosted instance keeps queries off the hosted API, so the
+ audit says so, without echoing a URL that may carry credentials."""
+ from openjarvis.security.data_boundary_audit import _web_search_destination
+
+ self._clear(monkeypatch)
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ monkeypatch.setenv("FIRECRAWL_API_URL", "https://user:[email protected]:3002/")
+ destination = _web_search_destination()
+ assert destination.startswith("Firecrawl search API (self-hosted at ")
+ assert "fc.internal;" in destination
+ assert "pw" not in destination
+
+ def test_keyless_self_hosted_firecrawl_is_named(self, monkeypatch):
+ from openjarvis.security.data_boundary_audit import _web_search_destination
+
+ self._clear(monkeypatch)
+ monkeypatch.setenv("FIRECRAWL_API_URL", "http://localhost:3002")
+ assert "self-hosted at localhost" in _web_search_destination()
+
def test_matches_the_tool_resolution(self, monkeypatch):
"""Guard against the audit's copy of the precedence rule drifting from
WebSearchTool._resolve_engine, which is the source of truth."""
@@ -1414,23 +1461,34 @@ class TestWebSearchDestination:
"tavily": "Tavily",
"duckduckgo": "DuckDuckGo",
"serply": "Serply",
+ "firecrawl": "Firecrawl",
}
- engines = (None, "auto", "youcom", "tavily", "duckduckgo", "serply")
- for tavily in (None, "tvly-key"):
- for youcom in (None, "ydc-key"):
- for serply in (None, "srp-key"):
- for engine in engines:
- self._clear(monkeypatch)
- if tavily:
- monkeypatch.setenv("TAVILY_API_KEY", tavily)
- if youcom:
- monkeypatch.setenv("YOUDOTCOM_API_KEY", youcom)
- if serply:
- monkeypatch.setenv("SERPLY_API_KEY", serply)
- if engine:
- monkeypatch.setenv("OPENJARVIS_WEB_SEARCH_ENGINE", engine)
- resolved = WebSearchTool()._resolve_engine()
- assert labels[resolved] in _web_search_destination(), (
- f"tavily={tavily} youcom={youcom} serply={serply} "
- f"engine={engine}"
- )
+ engines = (
+ None,
+ "auto",
+ "youcom",
+ "tavily",
+ "duckduckgo",
+ "serply",
+ "firecrawl",
+ )
+ keys = (
+ ("TAVILY_API_KEY", "tvly-key"),
+ ("YOUDOTCOM_API_KEY", "ydc-key"),
+ ("SERPLY_API_KEY", "srp-key"),
+ ("FIRECRAWL_API_KEY", "fc-test"),
+ ("FIRECRAWL_API_URL", "http://localhost:3002"),
+ )
+ for present in itertools.product((False, True), repeat=len(keys)):
+ for engine in engines:
+ self._clear(monkeypatch)
+ for (name, value), is_set in zip(keys, present):
+ if is_set:
+ monkeypatch.setenv(name, value)
+ if engine:
+ monkeypatch.setenv("OPENJARVIS_WEB_SEARCH_ENGINE", engine)
+ resolved = WebSearchTool()._resolve_engine()
+ set_vars = [name for (name, _), is_set in zip(keys, present) if is_set]
+ assert labels[resolved] in _web_search_destination(), (
+ f"set={set_vars} engine={engine}"
+ )
diff --git a/tests/server/test_tools_endpoint.py b/tests/server/test_tools_endpoint.py
index 697dce3d..bd1ff32d 100644
--- a/tests/server/test_tools_endpoint.py
+++ b/tests/server/test_tools_endpoint.py
@@ -86,8 +86,8 @@ def test_weather_tool_picker_recognizes_connector_credential(tmp_path, monkeypat
def test_tool_credentials_browser_lifecycle(tmp_path, monkeypatch):
"""The browser API can save, report, and remove a Tavily key.
- ``web_search`` declares You.com and Serply keys alongside it, so status
- reports all three.
+ ``web_search`` declares You.com, Serply and Firecrawl keys alongside it, so
+ status reports all four.
"""
from fastapi import FastAPI
from fastapi.testclient import TestClient
@@ -98,6 +98,7 @@ def test_tool_credentials_browser_lifecycle(tmp_path, monkeypatch):
monkeypatch.delenv("TAVILY_API_KEY", raising=False)
monkeypatch.delenv("YOUDOTCOM_API_KEY", raising=False)
monkeypatch.delenv("SERPLY_API_KEY", raising=False)
+ monkeypatch.delenv("FIRECRAWL_API_KEY", raising=False)
app = FastAPI()
tools_router = create_agent_manager_router(MagicMock())[3]
app.include_router(tools_router)
@@ -113,6 +114,7 @@ def test_tool_credentials_browser_lifecycle(tmp_path, monkeypatch):
"TAVILY_API_KEY": True,
"YOUDOTCOM_API_KEY": False,
"SERPLY_API_KEY": False,
+ "FIRECRAWL_API_KEY": False,
}
deleted = client.delete(
@@ -124,6 +126,7 @@ def test_tool_credentials_browser_lifecycle(tmp_path, monkeypatch):
"TAVILY_API_KEY": False,
"YOUDOTCOM_API_KEY": False,
"SERPLY_API_KEY": False,
+ "FIRECRAWL_API_KEY": False,
}
diff --git a/tests/tools/test_web_search.py b/tests/tools/test_web_search.py
index abd219ae..9754b58b 100644
--- a/tests/tools/test_web_search.py
+++ b/tests/tools/test_web_search.py
@@ -24,6 +24,13 @@ def _no_ambient_serply_key(monkeypatch):
monkeypatch.delenv("SERPLY_PROXY_LOCATION", raising=False)
[email protected](autouse=True)
+def _no_ambient_firecrawl_key(monkeypatch):
+ """Same guard as above for the Firecrawl env, for the same reason (#972)."""
+ monkeypatch.delenv("FIRECRAWL_API_KEY", raising=False)
+ monkeypatch.delenv("FIRECRAWL_API_URL", raising=False)
+
+
class TestWebSearchTool:
def test_spec_name_and_category(self):
tool = WebSearchTool(api_key="test-key")
@@ -805,6 +812,39 @@ class TestEngineSelection:
monkeypatch.delenv("TAVILY_API_KEY", raising=False)
assert WebSearchTool()._resolve_engine() == "youcom"
+ def test_auto_prefers_firecrawl_over_the_keyless_tier(self, monkeypatch):
+ """A Firecrawl key only ever wins over having no key at all."""
+ monkeypatch.delenv("TAVILY_API_KEY", raising=False)
+ monkeypatch.delenv("YOUDOTCOM_API_KEY", raising=False)
+ monkeypatch.delenv("OPENJARVIS_WEB_SEARCH_ENGINE", raising=False)
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ assert WebSearchTool()._resolve_engine() == "firecrawl"
+
+ def test_firecrawl_never_displaces_an_existing_keyed_engine(self, monkeypatch):
+ """Adding a Firecrawl key must not change where an install already goes."""
+ monkeypatch.delenv("OPENJARVIS_WEB_SEARCH_ENGINE", raising=False)
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+
+ monkeypatch.setenv("TAVILY_API_KEY", "tvly-x")
+ monkeypatch.setenv("YOUDOTCOM_API_KEY", "ydc-key")
+ monkeypatch.setenv("SERPLY_API_KEY", "srp-x")
+ assert WebSearchTool()._resolve_engine() == "tavily"
+
+ monkeypatch.delenv("TAVILY_API_KEY", raising=False)
+ assert WebSearchTool()._resolve_engine() == "youcom"
+
+ monkeypatch.delenv("YOUDOTCOM_API_KEY", raising=False)
+ assert WebSearchTool()._resolve_engine() == "serply"
+
+ def test_auto_uses_a_keyless_self_hosted_firecrawl(self, monkeypatch):
+ """Self-hosted instances usually run without auth, so pointing
+ FIRECRAWL_API_URL away from the hosted API is enough to opt in."""
+ monkeypatch.delenv("TAVILY_API_KEY", raising=False)
+ monkeypatch.delenv("YOUDOTCOM_API_KEY", raising=False)
+ monkeypatch.delenv("OPENJARVIS_WEB_SEARCH_ENGINE", raising=False)
+ monkeypatch.setenv("FIRECRAWL_API_URL", "http://localhost:3002")
+ assert WebSearchTool()._resolve_engine() == "firecrawl"
+
def test_explicit_engine_overrides_key_presence(self, monkeypatch):
monkeypatch.setenv("TAVILY_API_KEY", "tvly-x")
assert WebSearchTool(engine="youcom")._resolve_engine() == "youcom"
@@ -833,6 +873,8 @@ class TestEngineSelection:
assert "YOUDOTCOM_API_KEY" in spec.metadata["optional_api_keys"]
assert "SERPLY_API_KEY" in spec.metadata["optional_api_keys"]
assert "serply" in spec.metadata["engines"]
+ assert "FIRECRAWL_API_KEY" in spec.metadata["optional_api_keys"]
+ assert "firecrawl" in spec.metadata["engines"]
# Backwards compatible for readers of the old single-key field.
assert spec.metadata["requires_api_key"] == "TAVILY_API_KEY"
@@ -1141,6 +1183,332 @@ class TestSerplySearch:
assert "SERPLY_API_KEY" not in result.metadata["fallback_reason"]
+class TestFirecrawlSearch:
+ """The Firecrawl engine, exercised against mocked HTTP."""
+
+ PAYLOAD = {
+ "success": True,
+ "data": {
+ "web": [
+ {
+ "title": "Result A",
+ "url": "https://example.com/a",
+ "description": "Desc A",
+ "position": 1,
+ },
+ {
+ "title": "Result B",
+ "url": "https://example.com/b",
+ "description": "Desc B",
+ "position": 2,
+ },
+ ]
+ },
+ }
+
+ def _mock_post(self, monkeypatch, payload=None, status=200):
+ import httpx
+
+ calls = {}
+
+ def _post(url, **kwargs):
+ calls["url"] = url
+ calls["json"] = kwargs.get("json")
+ calls["headers"] = kwargs.get("headers")
+ resp = MagicMock()
+ resp.status_code = status
+ resp.json.return_value = payload if payload is not None else self.PAYLOAD
+ if status >= 400:
+ resp.text = "error body"
+ resp.raise_for_status.side_effect = httpx.HTTPStatusError(
+ f"HTTP {status}", request=MagicMock(), response=resp
+ )
+ else:
+ resp.raise_for_status = MagicMock()
+ return resp
+
+ monkeypatch.setattr(httpx, "post", _post)
+ return calls
+
+ def _mock_ddgs(self, monkeypatch):
+ mock_ddgs = MagicMock()
+ mock_ddgs.text.return_value = []
+ mock_module = MagicMock()
+ mock_module.DDGS.return_value = mock_ddgs
+ monkeypatch.setitem(sys.modules, "ddgs", mock_module)
+
+ def test_endpoint_auth_and_origin(self, monkeypatch):
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ calls = self._mock_post(monkeypatch)
+
+ result = WebSearchTool(engine="firecrawl").execute(query="test query")
+
+ assert result.success is True
+ assert calls["url"] == "https://api.firecrawl.dev/v2/search"
+ assert calls["headers"]["Authorization"] == "Bearer fc-test"
+ assert calls["json"]["origin"] == "openjarvis"
+ assert result.metadata["engine"] == "firecrawl"
+
+ def test_result_format_matches_the_other_engines(self, monkeypatch):
+ """Same labeled Source/Summary shape agents already parse (#390)."""
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ self._mock_post(monkeypatch)
+
+ result = WebSearchTool(engine="firecrawl").execute(query="test query")
+
+ assert "### Result A" in result.content
+ assert "Source: https://example.com/a" in result.content
+ assert "Summary: Desc A" in result.content
+ assert "### Result B" in result.content
+ assert result.metadata["num_results"] == 2
+
+ def test_max_results_passed_as_limit(self, monkeypatch):
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ calls = self._mock_post(monkeypatch)
+ WebSearchTool(engine="firecrawl", max_results=3).execute(
+ query="q", max_results=7
+ )
+ assert calls["json"] == {"query": "q", "limit": 7, "origin": "openjarvis"}
+
+ def test_self_hosted_api_url(self, monkeypatch):
+ """FIRECRAWL_API_URL points search at a self-hosted instance."""
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ monkeypatch.setenv("FIRECRAWL_API_URL", "http://localhost:3002/")
+ calls = self._mock_post(monkeypatch)
+
+ WebSearchTool(engine="firecrawl").execute(query="q")
+
+ assert calls["url"] == "http://localhost:3002/v2/search"
+
+ def test_keyless_self_hosted_sends_no_auth_header(self, monkeypatch):
+ monkeypatch.setenv("FIRECRAWL_API_URL", "http://localhost:3002")
+ calls = self._mock_post(monkeypatch)
+
+ result = WebSearchTool(engine="firecrawl").execute(query="q")
+
+ assert result.metadata["engine"] == "firecrawl"
+ assert "Authorization" not in calls["headers"]
+
+ def test_long_descriptions_are_capped(self, monkeypatch):
+ """Descriptions can carry thousands of characters of page text; five
+ of them must still fit a small model's context."""
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ payload = {
+ "data": {
+ "web": [
+ {
+ "title": "Long",
+ "url": "https://e.com",
+ "description": "x" * 10000,
+ }
+ ]
+ }
+ }
+ self._mock_post(monkeypatch, payload=payload)
+
+ result = WebSearchTool(engine="firecrawl").execute(query="q")
+
+ assert "x" * 500 + "…" in result.content
+ assert "x" * 501 not in result.content
+
+ def test_credits_are_reported(self, monkeypatch):
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ self._mock_post(monkeypatch, payload={**self.PAYLOAD, "creditsUsed": 2})
+ result = WebSearchTool(engine="firecrawl").execute(query="q")
+ assert result.metadata["credits"] == 2
+
+ def test_missing_key_is_named_without_a_request(self, monkeypatch):
+ """An explicit firecrawl engine with no key against the hosted API
+ should say the key is missing, not that it was rejected."""
+ import httpx
+
+ monkeypatch.setattr(
+ httpx, "post", MagicMock(side_effect=AssertionError("must not be called"))
+ )
+ self._mock_ddgs(monkeypatch)
+
+ result = WebSearchTool(engine="firecrawl").execute(query="q")
+
+ assert result.metadata["fallback_reason"] == "FIRECRAWL_API_KEY is not set"
+
+ def test_empty_results(self, monkeypatch):
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ self._mock_post(monkeypatch, payload={"success": True, "data": {"web": []}})
+ result = WebSearchTool(engine="firecrawl").execute(query="q")
+ assert result.success is True
+ assert result.content == "No results found."
+
+ def test_rejected_key_names_the_env_var(self, monkeypatch):
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-bad")
+ self._mock_post(monkeypatch, status=401)
+ self._mock_ddgs(monkeypatch)
+
+ result = WebSearchTool(engine="firecrawl").execute(query="q")
+
+ assert result.metadata["fallback_from"] == "firecrawl"
+ assert result.metadata["degraded"] is True
+ assert "FIRECRAWL_API_KEY" in result.metadata["fallback_reason"]
+
+ def test_exhausted_credits_point_to_the_key_page(self, monkeypatch):
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ self._mock_post(monkeypatch, status=402)
+ self._mock_ddgs(monkeypatch)
+
+ result = WebSearchTool(engine="firecrawl").execute(query="q")
+
+ reason = result.metadata["fallback_reason"]
+ assert "credits" in reason
+ assert "firecrawl.dev/app/api-keys" in reason
+
+ def test_rate_limit_is_named(self, monkeypatch):
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ self._mock_post(monkeypatch, status=429)
+ self._mock_ddgs(monkeypatch)
+
+ result = WebSearchTool(engine="firecrawl").execute(query="q")
+
+ assert "rate limit" in result.metadata["fallback_reason"]
+
+ def test_other_http_error_reports_the_status(self, monkeypatch):
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ self._mock_post(monkeypatch, status=500)
+ self._mock_ddgs(monkeypatch)
+
+ result = WebSearchTool(engine="firecrawl").execute(query="q")
+
+ assert "HTTP 500" in result.metadata["fallback_reason"]
+ assert "FIRECRAWL_API_KEY" not in result.metadata["fallback_reason"]
+
+
+class TestFirecrawlScrapeExtraction:
+ def _mock_ssrf(self, monkeypatch):
+ import openjarvis.tools.web_search as _ws
+
+ monkeypatch.setattr(_ws, "check_ssrf", lambda url: None)
+
+ def _mock_scrape(self, monkeypatch, markdown):
+ import httpx
+
+ resp = MagicMock()
+ resp.json.return_value = {"success": True, "data": {"markdown": markdown}}
+ resp.raise_for_status = MagicMock()
+ post = MagicMock(return_value=resp)
+ monkeypatch.setattr(httpx, "post", post)
+ return post
+
+ def test_scrape_used_for_url_queries(self, monkeypatch):
+ """URL queries come back as markdown instead of the regex HTML strip."""
+ self._mock_ssrf(monkeypatch)
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ post = self._mock_scrape(monkeypatch, "# Heading\n\nBody text.")
+
+ result = WebSearchTool(engine="firecrawl").execute(
+ query="https://example.com/a.pdf"
+ )
+
+ assert result.success is True
+ assert result.content == "# Heading\n\nBody text."
+ assert result.metadata["extractor"] == "firecrawl_scrape"
+ assert post.call_args.args[0] == "https://api.firecrawl.dev/v2/scrape"
+ assert post.call_args.kwargs["json"] == {
+ "url": "https://example.com/a.pdf",
+ "formats": ["markdown"],
+ "onlyMainContent": True,
+ "parsers": [{"type": "pdf", "maxPages": 3}],
+ "timeout": 45000,
+ "origin": "openjarvis",
+ }
+
+ def test_scrape_output_is_truncated_like_the_local_fetch(self, monkeypatch):
+ """A long PDF must not flood a small model's context."""
+ self._mock_ssrf(monkeypatch)
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ self._mock_scrape(monkeypatch, "x" * 10000)
+
+ result = WebSearchTool(engine="firecrawl").execute(
+ query="https://example.com/a"
+ )
+
+ assert result.content.startswith("x" * 6000)
+ assert result.content.endswith("[Content truncated]")
+ assert len(result.content) < 6100
+
+ def test_keyless_self_hosted_scrape(self, monkeypatch):
+ self._mock_ssrf(monkeypatch)
+ monkeypatch.setenv("FIRECRAWL_API_URL", "http://localhost:3002")
+ post = self._mock_scrape(monkeypatch, "# Self-hosted")
+
+ result = WebSearchTool(engine="firecrawl").execute(
+ query="https://example.com/a"
+ )
+
+ assert result.content == "# Self-hosted"
+ assert post.call_args.args[0] == "http://localhost:3002/v2/scrape"
+ assert "Authorization" not in post.call_args.kwargs["headers"]
+
+ def test_url_query_without_key_uses_local_fetch(self, monkeypatch):
+ """Explicit firecrawl with no key and the hosted URL never calls the
+ API for URL queries; the local fetch handles them as before."""
+ self._mock_ssrf(monkeypatch)
+ import httpx
+
+ monkeypatch.setattr(
+ httpx, "post", MagicMock(side_effect=AssertionError("must not be called"))
+ )
+ resp = MagicMock()
+ resp.text = "<html><body>Local text</body></html>"
+ resp.headers = {"content-type": "text/html"}
+ resp.raise_for_status = MagicMock()
+ monkeypatch.setattr(httpx.Client, "get", MagicMock(return_value=resp))
+
+ result = WebSearchTool(engine="firecrawl").execute(
+ query="https://example.com/a"
+ )
+
+ assert result.metadata["extractor"] == "local"
+
+ def test_scrape_failure_falls_back_to_local_fetch(self, monkeypatch):
+ self._mock_ssrf(monkeypatch)
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ import httpx
+
+ monkeypatch.setattr(
+ httpx, "post", MagicMock(side_effect=httpx.ConnectError("down"))
+ )
+ resp = MagicMock()
+ resp.text = "<html><body>Local text</body></html>"
+ resp.headers = {"content-type": "text/html"}
+ resp.raise_for_status = MagicMock()
+ monkeypatch.setattr(httpx.Client, "get", MagicMock(return_value=resp))
+
+ result = WebSearchTool(engine="firecrawl").execute(
+ query="https://example.com/a"
+ )
+
+ assert result.success is True
+ assert "Local text" in result.content
+ assert result.metadata["extractor"] == "local"
+
+ def test_url_branch_ssrf_blocked_before_scrape_call(self, monkeypatch):
+ """The scrape upgrade must not become an SSRF bypass."""
+ monkeypatch.setenv("FIRECRAWL_API_KEY", "fc-test")
+ import openjarvis.tools.web_search as _ws
+
+ monkeypatch.setattr(_ws, "check_ssrf", lambda url: "private IP blocked")
+ import httpx
+
+ monkeypatch.setattr(
+ httpx, "post", MagicMock(side_effect=AssertionError("must not be called"))
+ )
+
+ result = WebSearchTool(engine="firecrawl").execute(
+ query="http://169.254.169.254/metadata"
+ )
+
+ assert result.success is False
+ assert "private IP blocked" in result.content
+
+
class TestFallbackVisibility:
def test_fallback_is_logged_at_warning(self, monkeypatch, caplog):
"""Regression for the silent-degradation half of #923: a typo'd key