Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
firecrawl-local: make the path to search and Alexandria work for a new user (#5709)
Eric Ciarla5aed6b9307f3
diff --git a/apps/local/README.md b/apps/local/README.mdindex 146a6c612..2bb6c0e50 100644--- a/apps/local/README.md+++ b/apps/local/README.md@@ -6,6 +6,8 @@ It shares its HTML cleaning code with the Firecrawl API. Firecrawl Local compile## Build+You need git, [Rust](https://rustup.rs) and a C compiler (`build-essential` on Debian and Ubuntu, the Xcode command line tools on macOS).+```shcd apps/localcargo build --release@@ -14,7 +16,16 @@ cargo build --releaseOnce a release is on crates.io, `cargo install firecrawl-local` installs the same binary without cloning the repository.-Pages that need JavaScript are rendered in the Chrome or Chromium already installed on the machine. Without one it still handles static pages. Set `FIRECRAWL_CHROME_PATH` to point it at a specific browser.+Pages that need JavaScript are rendered in the Chrome or Chromium already installed on the machine. Without one it still handles static pages, and a page that needed a browser comes back with a `warning` saying so. Set `FIRECRAWL_CHROME_PATH` to point it at a specific browser.++Chrome keeps its sandbox on. Inside a Docker container that needs user namespaces, which Docker's default seccomp profile blocks. To render JavaScript pages there, run the container with a profile that keeps Docker's defaults and also allows `clone`, `setns` and `unshare`. Playwright [publishes one](https://playwright.dev/docs/docker) for sandboxed Chromium.++```sh+curl -fsSLO https://raw.githubusercontent.com/microsoft/playwright/main/utils/docker/seccomp_profile.json+docker run --security-opt seccomp=seccomp_profile.json your-image+```++Do not use `seccomp=unconfined` for this. It removes the whole filter from a container that is loading pages you do not control.## Use it from the command line@@ -46,7 +57,7 @@ curl -s localhost:3002/v2/scrape -H 'content-type: application/json' \`serve` answers `/v2/scrape`, `/v2/map`, `/v2/crawl` and `/v2/batch/scrape`, so existing Firecrawl SDKs work when pointed at it. The older v0 and v1 routes are not served. Crawl and batch jobs are kept in memory and expire an hour after they are created.-Every other `/v2` route, such as `/search`, `/agent` and data providers, is passed on to Firecrawl cloud when the cloud setting allows that route, and answers `CLOUD_ONLY` otherwise. With no key and no `FIRECRAWL_CLOUD`, `serve` passes nothing on. `FIRECRAWL_CLOUD=search` passes on search only, `FIRECRAWL_CLOUD=off` passes on nothing, and a value that is not recognised is read as `off`. Forwarded requests must be `application/json` or come from a tool, not a web page, and bodies may be up to 64 MB. Structured JSON output and browser actions on a local scrape are not implemented. The full self-hosted server in `apps/api` is unchanged.+Every other `/v2` route, such as `/search`, `/agent` and data providers, is passed on to Firecrawl cloud when the cloud setting allows that route, and answers `CLOUD_ONLY` otherwise. By default they are passed on, without a key until you sign in, the same as the command line and the MCP server. `FIRECRAWL_CLOUD=search` passes on search only, `FIRECRAWL_CLOUD=off` passes on nothing, and a value that is not recognised is read as `off`. Forwarded requests must be `application/json` or come from a tool, not a web page, and bodies may be up to 64 MB. Structured JSON output and browser actions on a local scrape are not implemented. The full self-hosted server in `apps/api` is unchanged.## What runs locally and what runs on Firecrawl cloud@@ -55,10 +66,12 @@ Scrape, crawl, map and document parsing run on your machine and send nothing toEverything else uses Firecrawl cloud, because it needs an index, data or a service the machine does not have. The three you will meet most are these.- **Search.** `firecrawl-local search` and the MCP server's `firecrawl_search` tool query Firecrawl's search.-- **Pages that will not load locally.** When a site refuses a plain fetch and the local browser, the CLI and MCP server ask Firecrawl cloud for the page and mark the result with `source: firecrawl-cloud`.+- **Pages that will not load locally.** When a site refuses a plain fetch and the local browser, Firecrawl Local asks Firecrawl cloud for the page and marks the result with `source: firecrawl-cloud`. If the cloud cannot take it either, the result is marked `blocked` and carries a `warning` that says why.- **Alexandria data providers.** `find-tools` and `data` reach structured data providers and need a Firecrawl account.-`firecrawl-local login` signs in through the browser and saves the key where the Firecrawl CLI keeps it. Set `FIRECRAWL_CLOUD=off` to turn off search, the cloud fallback and data providers, or `FIRECRAWL_CLOUD=search` to allow search only. `login` always talks to Firecrawl, since signing in is its whole job. `serve` stays local unless a key or `FIRECRAWL_CLOUD` is set.+`firecrawl-local login` signs in through the browser and saves the key where the Firecrawl CLI keeps it. Set `FIRECRAWL_CLOUD=off` to turn off search, the cloud fallback and data providers, or `FIRECRAWL_CLOUD=search` to allow search only. `login` always talks to Firecrawl, since signing in is its whole job. A login is picked up by `serve` and by the MCP server's search and scrape tools while they are running. The MCP server's Alexandria tools check for a key when it starts, so restart the MCP server after signing in to use them.++Without an account, Firecrawl cloud allows a limited amount of search. When that runs out, or when a call needs an account, the answer says to run `firecrawl-local login`.## Safety defaultsdiff --git a/apps/local/src/browser.rs b/apps/local/src/browser.rsindex 0408bda41..0ef9b1302 100644--- a/apps/local/src/browser.rs+++ b/apps/local/src/browser.rs@@ -316,7 +316,7 @@ impl BrowserPool {.map_err(|e| format!("browser configuration failed: {e}"))?;let (browser, mut handler) = Browser::launch(config).await- .map_err(|e| format!("could not start Chrome at {}: {e}", self.chrome.display()))?;+ .map_err(|e| chrome_start_error(&self.chrome, &e.to_string()))?;let handler = tokio::spawn(async move {while let Some(event) = handler.next().await {if event.is_err() {@@ -762,10 +762,41 @@ pub fn meta_refresh_target(raw_html: &str, base: &Url) -> Option<Url> {}}+/// Why Chrome did not start, in words someone can act on. Chrome's own+/// message runs to several hundred characters of log line.+fn chrome_start_error(chrome: &std::path::Path, detail: &str) -> String {+ if detail.contains("No usable sandbox") {+ return format!(+ "Chrome at {} could not start its sandbox, which usually means a container or system policy blocks user namespaces. In Docker, run the container with a seccomp profile that allows them; the Firecrawl Local README shows one. On Ubuntu 23.10 or newer, allow user namespaces for Chrome. Firecrawl Local does not run Chrome without its sandbox.",+ chrome.display()+ );+ }+ let short: String = detail.chars().take(300).collect();+ format!("could not start Chrome at {}: {short}", chrome.display())+}+#[cfg(test)]mod tests {use super::*;+ #[test]+ fn a_sandbox_failure_is_explained() {+ let chrome = std::path::Path::new("/usr/bin/chromium");+ let said = chrome_start_error(+ chrome,+ "Browser process exited ... No usable sandbox! If this is a Debian system ...",+ );+ assert!(+ said.contains("seccomp profile") && said.contains("/usr/bin/chromium"),+ "{said}"+ );+ // Turning the whole filter off is not what it recommends.+ assert!(!said.contains("unconfined"), "{said}");+ assert!(!said.contains("Debian system"));+ let other = chrome_start_error(chrome, &"x".repeat(2000));+ assert!(other.len() < 400);+ }+#[test]fn link_share_and_nav_only_shells() {assert!((link_text_share("[abcd](https://x.dev/long/url) ef") - 4.0 / 6.0).abs() < 1e-9);diff --git a/apps/local/src/cloud.rs b/apps/local/src/cloud.rsindex 0a89c1ee2..e23831908 100644--- a/apps/local/src/cloud.rs+++ b/apps/local/src/cloud.rs@@ -75,9 +75,63 @@ pub struct Cloud {/// The agent harness that launched the engine (from the MCP handshake).harness: Option<String>,origin: String,+ /// Whether to use the login saved by `firecrawl-local login` or the+ /// Firecrawl CLI. It is read again on every call, so signing in while an+ /// MCP server or `serve` is running takes effect without a restart.+ saved_login: bool,}-pub const SIGN_IN_HINT: &str = "Run `firecrawl login`, or set FIRECRAWL_API_KEY.";+pub const SIGN_IN_HINT: &str = "Run `firecrawl-local login`, or set FIRECRAWL_API_KEY.";++/// Reword a refusal Firecrawl cloud sent to a caller with no key, so it says+/// what to do on this machine. The cloud writes these for people calling the+/// API directly ("authenticate with: Authorization: Bearer ..."); someone+/// using a local MCP server or `serve` signs in with one command instead.+pub fn local_sign_in_message(error: &str) -> String {+ let lower = error.to_ascii_lowercase();+ let mut text = match lower.find("then authenticate with") {+ Some(at) => error[..at].trim_end().to_string(),+ None => error.trim().to_string(),+ };+ if lower.contains("alexandria") && lower.contains("api key") {+ text = "Alexandria data providers need a free Firecrawl account.".to_string();+ }+ if text.contains("firecrawl-local login") {+ return text;+ }+ if text.contains("http") {+ // Keep the sign-up link on its own line so nothing is glued to it.+ format!(+ "{text}\n\nOr run `firecrawl-local login`, which signs you in and saves the key for you."+ )+ } else {+ format!("{text} Run `firecrawl-local login` to create one or sign in. It is one step in the browser and the key is saved for you, then the same call works.")+ }+}++/// Whether a refusal is about the caller having no account, as opposed to a+/// route's own rule or an ordinary rate limit that signing in would not lift.+fn is_sign_in_refusal(status: u16, payload: &Value, error: &str) -> bool {+ if status == 401 || payload.get("signup_url").is_some() || payload.get("signupUrl").is_some() {+ return true;+ }+ let lower = error.to_ascii_lowercase();+ ["keyless", "api key", "unauthorized", "authenticate with"]+ .iter()+ .any(|sign| lower.contains(sign))+}++/// Apply `local_sign_in_message` to the `error` field of a JSON answer, when+/// the answer is a refusal for want of an account.+fn with_sign_in_step(status: u16, body: &[u8]) -> Option<bytes::Bytes> {+ let mut payload: Value = serde_json::from_slice(body).ok()?;+ let error = payload.get("error")?.as_str()?.to_string();+ if !is_sign_in_refusal(status, &payload, &error) {+ return None;+ }+ payload["error"] = Value::String(local_sign_in_message(&error));+ serde_json::to_vec(&payload).ok().map(bytes::Bytes::from)+}fn config_dir() -> Option<std::path::PathBuf> {if let Some(dir) = std::env::var_os("FIRECRAWL_LOCAL_HOME") {@@ -216,15 +270,14 @@ impl std::fmt::Display for CloudError {impl Cloud {/// Reads `FIRECRAWL_API_KEY`, `FIRECRAWL_API_URL` and `FIRECRAWL_CLOUD`,- /// falling back to the login saved by `firecrawl login`.- /// With no explicit setting: a key turns the fallback on, no key means- /// search only.+ /// falling back to the login saved by `firecrawl-local login` or the+ /// Firecrawl CLI. With no key, cloud calls go out keyless and Firecrawl+ /// cloud decides what a caller without an account may do.pub fn from_env() -> Self {- let (saved_key, saved_url) = cli_credentials();+ let (_, saved_url) = cli_credentials();let api_key = std::env::var("FIRECRAWL_API_KEY").ok()- .filter(|k| !k.trim().is_empty())- .or(saved_key);+ .filter(|k| !k.trim().is_empty());let base = std::env::var("FIRECRAWL_API_URL").ok().filter(|u| !u.trim().is_empty())@@ -237,19 +290,21 @@ impl Cloud {}}let mode = CloudMode::requested(setting.as_deref()).unwrap_or(CloudMode::Fallback);- Self::new(base, api_key, mode)+ let mut cloud = Self::new(base, api_key, mode);+ cloud.saved_login = true;+ cloud}- /// Same as `from_env`, but with the cloud off unless a key or an explicit- /// `FIRECRAWL_CLOUD` setting turns it on. For `serve`, where the people- /// calling the API did not install this engine themselves.- pub fn from_env_opt_in() -> Self {- let mut cloud = Self::from_env().with_surface(Surface::Serve, None);- let setting = std::env::var("FIRECRAWL_CLOUD").ok();- if CloudMode::requested(setting.as_deref()).is_none() && !cloud.has_key() {- cloud.mode = CloudMode::Off;- }- cloud+ /// The key cloud calls go out with: the one given explicitly, or else the+ /// saved login as it is on disk right now.+ fn key(&self) -> Option<String> {+ self.api_key.clone().or_else(|| {+ if self.saved_login {+ cli_credentials().0+ } else {+ None+ }+ })}/// Give the cloud a fixed install ID, so a test never writes one to the@@ -291,6 +346,7 @@ impl Cloud {install_id: None,harness: None,origin: origin_for(Surface::Cli, None),+ saved_login: false,}}@@ -306,7 +362,13 @@ impl Cloud {}pub fn has_key(&self) -> bool {- self.api_key.is_some()+ self.key().is_some()+ }++ /// The key in use right now, for handing to the MCP server this process+ /// starts. Never printed.+ pub fn api_key(&self) -> Option<String> {+ self.key()}/// Use the origin a client stamped on its own request, so a call the@@ -335,9 +397,10 @@ impl Cloud {}}- /// Pass a request this engine does not serve itself on to Firecrawl cloud- /// and return the answer as it came. The engine's own key is used when it- /// has one; otherwise the caller's, and with neither the call is keyless.+ /// Pass a request Firecrawl Local does not serve itself on to Firecrawl+ /// cloud and return the answer. Its own key is used when it has one;+ /// otherwise the caller's, and with neither the call is keyless. A keyless+ /// call that the cloud refuses comes back with the sign-in step added.pub async fn forward(&self,method: reqwest::Method,@@ -370,7 +433,10 @@ impl Cloud {if let Some(id) = self.install_id.clone().or_else(load_install_id) {req = req.header("x-firecrawl-install", id);}- if let Some(key) = self.api_key.as_deref().or(client_auth) {+ let own_key = self.key();+ let key = own_key.as_deref().or(client_auth);+ let keyless = key.is_none();+ if let Some(key) = key {req = req.bearer_auth(key);}let resp = req.send().await.map_err(|e| CloudError {@@ -383,10 +449,15 @@ impl Cloud {.get(reqwest::header::CONTENT_TYPE).and_then(|v| v.to_str().ok()).map(str::to_string);- let body = resp.bytes().await.map_err(|e| CloudError {+ let mut body = resp.bytes().await.map_err(|e| CloudError {status: 502,message: format!("Firecrawl cloud's answer was cut short: {e}"),})?;+ if keyless && matches!(status, 401 | 402 | 403 | 429) {+ if let Some(reworded) = with_sign_in_step(status, &body) {+ body = reworded;+ }+ }Ok(Forwarded {status,content_type,@@ -419,7 +490,9 @@ impl Cloud {if let Some(h) = &self.harness {req = req.header("x-firecrawl-harness", h);}- if let Some(key) = &self.api_key {+ let key = self.key();+ let signed_in = key.is_some();+ if let Some(key) = &key {req = req.bearer_auth(key);}let resp = req.send().await.map_err(|e| CloudError {@@ -435,7 +508,7 @@ impl Cloud {.unwrap_or("request failed").to_string();let message = match status {- 401 | 402 | 403 | 429 if !self.has_key() => {+ 401 | 402 | 403 | 429 if !signed_in => {// The cloud sends a signup link made for this caller; pass it on.let link = payload.get("signupUrl")@@ -445,8 +518,8 @@ impl Cloud {let reason = detail.lines().next().unwrap_or("").trim();format!("Firecrawl cloud needs a sign-in for this ({reason}). {SIGN_IN_HINT}{link}")}- 401 if self.has_key() => format!(- "Firecrawl rejected the saved login ({detail}). It may have been rotated or revoked. Run `firecrawl login` again, or update FIRECRAWL_API_KEY."+ 401 if signed_in => format!(+ "Firecrawl rejected the saved login ({detail}). It may have been rotated or revoked. Run `firecrawl-local login` again, or update FIRECRAWL_API_KEY."),_ => format!("Firecrawl cloud returned {status}: {detail}"),};@@ -560,9 +633,10 @@ pub async fn scrape_with_fallback(opts: &crate::scrape::ScrapeOptions,) -> Result<Value, crate::scrape::ScrapeError> {let local = engine.scrape(url, opts).await;+ let mut status = 0u16;let blocked = match &local {Ok(doc) => {- let status = doc+ status = doc.pointer("/metadata/statusCode").and_then(Value::as_u64).unwrap_or(200) as u16;@@ -571,15 +645,20 @@ pub async fn scrape_with_fallback(}Err(e) => matches!(e.code, "FETCH_FAILED" | "SCRAPE_TIMEOUT"),};+ // Why the cloud did not take over, when it was asked and could not.+ let mut cloud_refusal: Option<String> = None;if blocked && cloud.mode == CloudMode::Fallback {let cloud_opts = json!({ "formats": ["markdown"], "onlyMainContent": opts.only_main_content });- if let Ok(payload) = cloud.scrape(url, &cloud_opts).await {- if let Some(mut data) = payload.get("data").cloned() {- if let Some(obj) = data.as_object_mut() {- obj.insert("source".into(), Value::String("firecrawl-cloud".into()));+ match cloud.scrape(url, &cloud_opts).await {+ Ok(payload) => {+ if let Some(mut data) = payload.get("data").cloned() {+ if let Some(obj) = data.as_object_mut() {+ obj.insert("source".into(), Value::String("firecrawl-cloud".into()));+ }+ return Ok(data);}- return Ok(data);}+ Err(e) => cloud_refusal = Some(e.message),}}match local {@@ -588,6 +667,10 @@ pub async fn scrape_with_fallback(obj.insert("source".into(), Value::String("local".into()));if blocked {obj.insert("blocked".into(), Value::Bool(true));+ crate::scrape::add_warning(+ obj,+ blocked_warning(status, cloud.mode, cloud_refusal.as_deref()),+ );}}Ok(doc)@@ -596,6 +679,21 @@ pub async fn scrape_with_fallback(}}+/// What to tell someone whose page came back refused, so an empty result is+/// never mistaken for an empty page.+fn blocked_warning(status: u16, mode: CloudMode, cloud_refusal: Option<&str>) -> String {+ let lead = format!(+ "This site did not serve the page to this machine (status {status}), so the content is missing or is a bot check."+ );+ match (cloud_refusal, mode) {+ (Some(refusal), _) => format!("{lead} Firecrawl cloud did not fetch it either. {refusal}"),+ (None, CloudMode::Fallback) => lead,+ (None, _) => {+ format!("{lead} Firecrawl cloud can fetch pages like this when FIRECRAWL_CLOUD is auto.")+ }+ }+}+/// Mark a cloud response so callers can tell where it ran and what it cost.fn annotate(payload: &mut Value) {if let Some(obj) = payload.as_object_mut() {@@ -689,6 +787,38 @@ pub(crate) async fn echo_cloud() -> String {base}+/// A stand-in for Firecrawl cloud that refuses every call the way the+/// keyless limit does.+#[cfg(test)]+pub(crate) async fn refusing_cloud() -> String {+ use tokio::io::{AsyncReadExt, AsyncWriteExt};+ let listener = tokio::net::TcpListener::bind(("127.0.0.1", 0))+ .await+ .unwrap();+ let base = format!("http://{}", listener.local_addr().unwrap());+ tokio::spawn(async move {+ while let Ok((mut socket, _)) = listener.accept().await {+ tokio::spawn(async move {+ let mut buf = vec![0u8; 8192];+ let _ = socket.read(&mut buf).await;+ let body = serde_json::json!({+ "success": false,+ "error": "You've hit Firecrawl's keyless free tier rate limit. To continue now, create a free API key at https://firecrawl.dev/k/abc123\n\nThen authenticate with:\nAuthorization: Bearer YOUR_API_KEY",+ "signup_url": "https://firecrawl.dev/k/abc123"+ })+ .to_string();+ let answer = format!(+ "HTTP/1.1 429 Too Many Requests\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",+ body.len(),+ body+ );+ let _ = socket.write_all(answer.as_bytes()).await;+ });+ }+ });+ base+}+#[cfg(test)]mod tests {use super::*;@@ -709,6 +839,25 @@ mod tests {assert_eq!(url.as_deref(), Some("https://api.example.test"));std::fs::write(&path, "not json").unwrap();assert_eq!(cli_credentials(), (None, None));++ // A login made while the process is running is picked up on the next call.+ let mut running = Cloud::new("https://api.example.test".into(), None, CloudMode::Fallback);+ running.saved_login = true;+ assert!(!running.has_key());+ std::fs::write(&path, r#"{"apiKey":"fc-signed-in-later"}"#).unwrap();+ assert_eq!(running.key().as_deref(), Some("fc-signed-in-later"));+ // A key given explicitly wins, and a cloud that was not told to use the+ // saved login never reads it.+ assert_eq!(+ running+ .clone()+ .with_key("fc-explicit".into())+ .key()+ .as_deref(),+ Some("fc-explicit")+ );+ let plain = Cloud::new("https://api.example.test".into(), None, CloudMode::Fallback);+ assert!(!plain.has_key());std::env::remove_var("FIRECRAWL_CLI_CREDENTIALS");let _ = std::fs::remove_dir_all(&dir);}@@ -793,6 +942,102 @@ mod tests {);}+ #[test]+ fn refusals_are_reworded_for_a_local_user() {+ let limit = "You've hit Firecrawl's keyless free tier rate limit. To continue now, create a free API key at https://firecrawl.dev/k/abc123\n\nThen authenticate with:\nAuthorization: Bearer YOUR_API_KEY";+ let said = local_sign_in_message(limit);+ assert!(said.contains("https://firecrawl.dev/k/abc123\n"), "{said}");+ assert!(said.contains("firecrawl-local login"), "{said}");+ assert!(!said.contains("Authorization"), "{said}");++ let alexandria =+ local_sign_in_message("Alexandria requires an API key on a team with Alexandria access");+ assert!(alexandria.starts_with("Alexandria data providers need a free Firecrawl account."));+ assert!(alexandria.contains("firecrawl-local login"), "{alexandria}");++ // Said once, not twice.+ assert_eq!(local_sign_in_message(&said), said);++ let body = with_sign_in_step(+ 429,+ br#"{"success":false,"error":"Daily limit reached","signup_url":"x"}"#,+ )+ .unwrap();+ let payload: Value = serde_json::from_slice(&body).unwrap();+ assert!(payload["error"]+ .as_str()+ .unwrap()+ .contains("firecrawl-local login"));+ assert_eq!(payload["signup_url"], "x");+ assert!(with_sign_in_step(401, br#"{"error":"Unauthorized"}"#).is_some());+ assert!(with_sign_in_step(403, br#"{"error":"Alexandria requires an API key"}"#).is_some());++ // A route's own rule, or a plain rate limit, is not a request to sign in.+ assert!(with_sign_in_step(429, br#"{"error":"Rate limit exceeded. Retry in 10s."}"#).is_none());+ assert!(with_sign_in_step(+ 403,+ br#"{"error":"This route is not enabled for your team."}"#+ )+ .is_none());+ assert!(with_sign_in_step(402, br#"{"error":"Insufficient credits."}"#).is_none());+ assert!(with_sign_in_step(401, b"not json").is_none());+ assert!(with_sign_in_step(401, br#"{"error":{"code":1}}"#).is_none());+ }++ #[tokio::test]+ async fn a_keyless_refusal_gains_the_sign_in_step_and_a_keyed_one_does_not() {+ let base = refusing_cloud().await;+ let keyless = Cloud::new(base.clone(), None, CloudMode::Fallback).with_install_id("test");+ let answer = keyless+ .forward(+ reqwest::Method::POST,+ "/v2/search",+ Some("application/json"),+ bytes::Bytes::from_static(b"{}"),+ None,+ None,+ )+ .await+ .unwrap();+ assert_eq!(answer.status, 429);+ let text = String::from_utf8_lossy(&answer.body).to_string();+ assert!(text.contains("firecrawl-local login"), "{text}");+ assert!(!text.contains("Authorization"), "{text}");++ // A caller who sent a key gets the cloud's words as they came.+ let with_client_key = keyless+ .forward(+ reqwest::Method::POST,+ "/v2/search",+ Some("application/json"),+ bytes::Bytes::from_static(b"{}"),+ Some("fc-client"),+ None,+ )+ .await+ .unwrap();+ let text = String::from_utf8_lossy(&with_client_key.body).to_string();+ assert!(text.contains("Authorization"), "{text}");+ assert!(!text.contains("firecrawl-local login"), "{text}");+ }++ #[test]+ fn a_blocked_page_says_why_it_is_empty() {+ let off = blocked_warning(403, CloudMode::Off, None);+ assert!(+ off.contains("status 403") && off.contains("FIRECRAWL_CLOUD"),+ "{off}"+ );+ let refused = blocked_warning(+ 403,+ CloudMode::Fallback,+ Some("Firecrawl cloud needs a sign-in for this. Run `firecrawl-local login`."),+ );+ assert!(refused.contains("firecrawl-local login"), "{refused}");+ assert!(refused.contains("did not fetch it either"), "{refused}");+ assert!(!blocked_warning(503, CloudMode::Fallback, None).contains("FIRECRAWL_CLOUD"));+ }+#[test]fn modes_decide_what_is_passed_on() {let cloud = |mode| Cloud::new("https://api.example.test".into(), None, mode);diff --git a/apps/local/src/main.rs b/apps/local/src/main.rsindex 296679adf..9e220c794 100644--- a/apps/local/src/main.rs+++ b/apps/local/src/main.rs@@ -185,11 +185,18 @@ async fn launch_mcp(engine: Arc<Engine>) -> Result<(), String> {let addr = listener.local_addr().map_err(|e| format!("could not read the local port: {e}"))?;+ let cloud = Cloud::from_env().with_surface(firecrawl_local::cloud::Surface::Mcp, None);+ // The MCP server refuses Alexandria tools by itself unless it holds a key,+ // so someone who signed in with `firecrawl-local login` needs it passed on.+ let signed_in_key = cloud.api_key();let cfg = ServeConfig {addr,token: None,allowed_hosts: Vec::new(),- cloud: Cloud::from_env().with_surface(firecrawl_local::cloud::Surface::Mcp, None),+ cloud,+ // The MCP server's key is the copy handed to it below, so this process+ // stays the one that decides whether a call is signed in.+ use_client_key: false,};let (stop_tx, stop_rx) = tokio::sync::oneshot::channel::<()>();let server = tokio::spawn(serve::serve_on(listener, engine, cfg, async move {@@ -201,13 +208,17 @@ async fn launch_mcp(engine: Arc<Engine>) -> Result<(), String> {.filter(|p| !p.trim().is_empty()).unwrap_or_else(|| "firecrawl-mcp".to_string());let npx = if cfg!(windows) { "npx.cmd" } else { "npx" };- let child = tokio::process::Command::new(npx)+ let mut command = tokio::process::Command::new(npx);+ command.args(["-y", &package]).env("FIRECRAWL_API_URL", format!("http://{addr}")).stdin(std::process::Stdio::inherit()).stdout(std::process::Stdio::inherit())- .stderr(std::process::Stdio::inherit())- .spawn();+ .stderr(std::process::Stdio::inherit());+ if let Some(key) = signed_in_key {+ command.env("FIRECRAWL_API_KEY", key);+ }+ let child = command.spawn();let outcome = match child {Ok(mut child) => match child.wait().await {Ok(status) if status.success() => Ok(()),@@ -339,6 +350,11 @@ async fn run(cli: Cli) -> ExitCode {if markdown {for r in &results {+ // With only the markdown on screen, a warning is the one sign that+ // an empty or short result is not the whole page.+ if let Some(warning) = r.pointer("/data/warning").and_then(Value::as_str) {+ eprintln!("warning: {warning}");+ }match r.pointer("/data/markdown").and_then(Value::as_str) {Some(md) => println!("{md}\n"),None => eprintln!(@@ -656,7 +672,8 @@ async fn run(cli: Cli) -> ExitCode {addr: SocketAddr::new(ip, port),token: token.filter(|t| !t.is_empty()),allowed_hosts,- cloud: Cloud::from_env_opt_in(),+ cloud: Cloud::from_env().with_surface(firecrawl_local::cloud::Surface::Serve, None),+ use_client_key: true,};tokio::spawn(engine.clone().trim_memory_when_idle());let served = serve::serve(engine.clone(), cfg).await;diff --git a/apps/local/src/scrape.rs b/apps/local/src/scrape.rsindex 2c8bdbf2b..20664f69d 100644--- a/apps/local/src/scrape.rs+++ b/apps/local/src/scrape.rs@@ -164,6 +164,48 @@ impl Drop for InFlight<'_> {}}+/// Add a warning to a document, keeping any that is already there.+pub fn add_warning(data: &mut Map<String, Value>, text: impl Into<String>) {+ let text = text.into();+ let joined = match data.get("warning").and_then(Value::as_str) {+ Some(existing) if !existing.is_empty() => format!("{existing} {text}"),+ _ => text,+ };+ data.insert("warning".into(), Value::String(joined));+}++/// What to say when a page needs JavaScript and was not rendered, so a+/// near-empty result is not mistaken for the whole page. `None` when the+/// browser was turned off on purpose or did its job.+fn browser_warning(mode: BrowserMode, has_browser: bool, render: &Render) -> Option<String> {+ if mode == BrowserMode::Never {+ return None;+ }+ let lead = "This page builds its content with JavaScript, so this result is probably incomplete.";+ match (has_browser, render) {+ (false, _) => Some(format!(+ "{lead} No Chrome or Chromium was found on this machine. Install one, or point FIRECRAWL_CHROME_PATH at it."+ )),+ (true, Render::Failed(error)) => {+ Some(format!("{lead} The local Chrome could not render it. {error}"))+ }+ (true, Render::Rejected) => Some(format!(+ "{lead} The local Chrome loaded it but came back with less text than the plain fetch, as a consent wall or a half-loaded page does, so the plain fetch was kept."+ )),+ (true, Render::NotNeeded) => None,+ }+}++/// How the browser fared on a page that looked like it needed one.+enum Render {+ /// It was not asked, or its result was used.+ NotNeeded,+ /// It could not load the page.+ Failed(String),+ /// It loaded the page, and what it got was worse than the plain fetch.+ Rejected,+}+fn looks_like_html(mime: Option<&str>, body: &[u8]) -> bool {match mime {Some("text/html") | Some("application/xhtml+xml") => true,@@ -424,10 +466,10 @@ impl Engine {let challenged = can_render&& (matches!(fetched.status, 403 | 503) || crate::cloud::looks_like_bot_check(&markdown));let nav_only = looks_like_nav_only_shell(&raw_html, &markdown);- let shell = can_render- && !challenged- && (200..400).contains(&fetched.status)- && (needs_browser(&raw_html, &markdown) || nav_only);+ let needs_javascript =+ (200..400).contains(&fetched.status) && (needs_browser(&raw_html, &markdown) || nav_only);+ let shell = can_render && !challenged && needs_javascript;+ let mut render = Render::NotNeeded;let rendered = if challenged {match self.render_with_markdown(&parsed, opts, &formats, wait_for, timeout, false)@@ -453,7 +495,14 @@ impl Engine {{Some(doc)}- _ => None,+ Ok(_) => {+ render = Render::Rejected;+ None+ }+ Err(e) => {+ render = Render::Failed(e.message);+ None+ }}} else {None@@ -461,6 +510,13 @@ impl Engine {match rendered {Some(doc) => doc,None => {+ if needs_javascript && !challenged {+ if let Some(warning) =+ browser_warning(self.browser_mode, self.browser.is_some(), &render)+ {+ add_warning(&mut data, warning);+ }+ }finish_metadata(&mut metadata,&parsed,@@ -510,12 +566,12 @@ impl Engine {};if !unsupported.is_empty() {- data.insert(- "warning".into(),- Value::String(format!(+ add_warning(+ &mut data,+ format!("These formats are not produced by the local engine and were skipped: {}.",unsupported.join(", ")- )),+ ),);}Ok(Value::Object(data))@@ -1074,6 +1130,36 @@ mod tests {);}+ #[test]+ fn a_page_that_needs_a_browser_says_so() {+ let none = browser_warning(BrowserMode::Auto, false, &Render::NotNeeded).unwrap();+ assert!(none.contains("No Chrome or Chromium") && none.contains("FIRECRAWL_CHROME_PATH"));+ let failed = browser_warning(+ BrowserMode::Auto,+ true,+ &Render::Failed("Chrome could not start.".into()),+ )+ .unwrap();+ assert!(+ failed.contains("could not render it. Chrome could not start."),+ "{failed}"+ );+ // A render that was thrown away leaves the JavaScript shell, so it warns too.+ let rejected = browser_warning(BrowserMode::Auto, true, &Render::Rejected).unwrap();+ assert!(rejected.contains("plain fetch was kept"), "{rejected}");+ // Rendered fine, or turned off on purpose: nothing to say.+ assert!(browser_warning(BrowserMode::Auto, true, &Render::NotNeeded).is_none());+ assert!(browser_warning(BrowserMode::Never, false, &Render::NotNeeded).is_none());+ }++ #[test]+ fn warnings_add_up() {+ let mut data = Map::new();+ add_warning(&mut data, "First.");+ add_warning(&mut data, "Second.");+ assert_eq!(data["warning"], "First. Second.");+ }+#[test]fn bare_host_gets_https() {assert_eq!(diff --git a/apps/local/src/serve.rs b/apps/local/src/serve.rsindex 04450975b..a3ceb9a8c 100644--- a/apps/local/src/serve.rs+++ b/apps/local/src/serve.rs@@ -59,15 +59,23 @@ pub struct ServeConfig {/// Extra hostnames accepted in the Host header (for example a compose/// service name).pub allowed_hosts: Vec<String>,- /// What may be passed on to Firecrawl cloud. `serve` keeps it off unless a- /// key or setting turns it on; the MCP launcher runs with it on.+ /// What may be passed on to Firecrawl cloud. On by default, as it is for+ /// the command line and the MCP launcher; `FIRECRAWL_CLOUD=off` keeps+ /// every request on the machine.pub cloud: Cloud,+ /// Whether a caller's own Firecrawl key is used for calls passed on to+ /// Firecrawl cloud when this server has none. True for `serve`, where a+ /// caller may bring a key. False for the MCP launcher: its one caller is+ /// the MCP server it started, whose key is a copy of this process's own+ /// and would otherwise outlive a sign-out.+ pub use_client_key: bool,}struct AppState {engine: Arc<Engine>,cloud: Cloud,token: Option<String>,+ use_client_key: bool,allowed_hosts: Vec<String>,jobs: Jobs,}@@ -206,7 +214,7 @@ fn forward_refusal("success": false,"code": "CLOUD_ONLY","error": format!(- "{path} runs on Firecrawl cloud, and the cloud is off here. Set FIRECRAWL_API_KEY or FIRECRAWL_CLOUD=auto to pass these requests on."+ "{path} runs on Firecrawl cloud, and FIRECRAWL_CLOUD is set so that it is not passed on. Unset FIRECRAWL_CLOUD, or set it to auto, to pass these requests on."),"cloud": { "url": "https://www.firecrawl.dev", "docs": "https://docs.firecrawl.dev" }})),@@ -237,9 +245,10 @@ async fn pass_on(if let Some(refusal) = forward_refusal(state, &method, path, headers) {return refusal;}- // With a token set, the caller's bearer is this engine's token and must- // not travel on. Without one it is the caller's own Firecrawl key.- let client_auth = if state.token.is_none() {+ // With a token set, the caller's bearer is this server's token and must+ // not travel on. Without one it is the caller's own Firecrawl key, unless+ // this server was told not to use callers' keys.+ let client_auth = if state.token.is_none() && state.use_client_key {headers.get(header::AUTHORIZATION).and_then(|v| v.to_str().ok())@@ -697,9 +706,19 @@ pub async fn serve(engine: Arc<Engine>, cfg: ServeConfig) -> Result<(), String>.await.map_err(|e| format!("could not listen on {}: {e}", cfg.addr))?;eprintln!("firecrawl-local listening on http://{}", cfg.addr);+ eprintln!("{}", cloud_notice(&cfg.cloud));serve_on(listener, engine, cfg, shutdown_signal()).await}+/// One line at startup saying what leaves the machine.+fn cloud_notice(cloud: &Cloud) -> &'static str {+ match cloud.mode {+ cloud::CloudMode::Off => "Firecrawl cloud is off (FIRECRAWL_CLOUD). Every request stays on this machine.",+ cloud::CloudMode::SearchOnly => "Search is passed on to Firecrawl cloud. Everything else stays on this machine.",+ cloud::CloudMode::Fallback => "Scrape, crawl and map run here. Search, data providers and pages this machine cannot load go to Firecrawl cloud. FIRECRAWL_CLOUD=off keeps everything local.",+ }+}+/// Serve the API on a listener the caller already bound, until `stop`/// resolves. The MCP launcher uses this with a port the system picked.pub async fn serve_on(@@ -717,6 +736,7 @@ pub async fn serve_on(engine,cloud: cfg.cloud,token: cfg.token,+ use_client_key: cfg.use_client_key,allowed_hosts,jobs: Mutex::new(HashMap::new()),});@@ -788,6 +808,14 @@ mod tests {/// Start the server against a stand-in cloud and return its address.async fn start(mode: cloud::CloudMode) -> String {+ start_with(mode, Some("fc-own"), true).await+ }++ async fn start_with(+ mode: cloud::CloudMode,+ own_key: Option<&str>,+ use_client_key: bool,+ ) -> String {let engine = Arc::new(Engine::new(crate::scrape::EngineConfig {policy: crate::guard::NetworkPolicy::PublicOnly,@@ -804,13 +832,45 @@ mod tests {addr,token: None,allowed_hosts: Vec::new(),- cloud: Cloud::new(cloud::echo_cloud().await, Some("fc-own".into()), mode)+ cloud: Cloud::new(cloud::echo_cloud().await, own_key.map(str::to_string), mode).with_install_id("test-install"),+ use_client_key,};tokio::spawn(serve_on(listener, engine, cfg, std::future::pending()));format!("http://{addr}")}+ #[tokio::test]+ async fn a_callers_key_is_used_only_where_callers_bring_their_own() {+ let http = reqwest::Client::new();+ let search = |base: String| {+ http+ .post(format!("{base}/v2/search"))+ .header("content-type", "application/json")+ .header("authorization", "Bearer fc-callers-key")+ .body("{}")+ .send()+ };+ // `serve` with no key of its own passes the caller's key on.+ let open = start_with(cloud::CloudMode::Fallback, None, true).await;+ let seen = search(open).await.unwrap().text().await.unwrap();+ assert!(seen.contains("Bearer fc-callers-key"), "{seen}");+ // The MCP launcher's server does not: once this process is signed out,+ // the copy of the key its MCP server still holds must stop working.+ let launcher = start_with(cloud::CloudMode::Fallback, None, false).await;+ let seen = search(launcher)+ .await+ .unwrap()+ .text()+ .await+ .unwrap()+ .to_lowercase();+ assert!(+ !seen.contains("fc-callers-key") && !seen.contains("authorization:"),+ "{seen}"+ );+ }+#[tokio::test]async fn a_web_page_cannot_spend_the_key_through_a_forwarded_route() {let base = start(cloud::CloudMode::Fallback).await;