Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
docs: update README and SDK README copy (#5042)
Erik Engervall18f4d82c5d2f
diff --git a/README.md b/README.mdindex d8b347521..26e33930f 100644--- a/README.md+++ b/README.md@@ -80,6 +80,7 @@ _Pst. Hey, you, join our stargazers :)_| [**Crawl**](#crawl) | Scrape all URLs of a website with a single request || [**Map**](#map) | Discover all URLs on a website instantly || [**Batch Scrape**](#batch-scrape) | Scrape thousands of URLs asynchronously |+| [**Monitor**](https://docs.firecrawl.dev/features/monitoring) | Get notified when content on a website changes |---@@ -146,7 +147,7 @@ Output:### Scrape-Get LLM-ready data from any website — markdown, JSON, screenshots, and more.+Get LLM-ready context from any website — markdown, JSON, screenshots, and more.```pythonfrom firecrawl import Firecrawl@@ -472,7 +473,7 @@ Response:{"success": true,"links": [- {"url": "https://firecrawl.dev", "title": "Firecrawl", "description": "Turn websites into LLM-ready data"},+ {"url": "https://firecrawl.dev", "title": "Firecrawl", "description": "The context API for AI agents"},{"url": "https://firecrawl.dev/pricing", "title": "Pricing", "description": "Firecrawl pricing plans"},{"url": "https://firecrawl.dev/blog", "title": "Blog", "description": "Firecrawl blog"}]@@ -519,7 +520,7 @@ Our SDKs provide a convenient way to use all Firecrawl features and automaticallInstall the SDK:```bash-pip install firecrawl-py+pip install firecrawl``````pythonfrom firecrawl import Firecrawl@@ -859,7 +860,7 @@ print_r($results);**Agents & AI Tools**- [Firecrawl Skills Catalog](https://github.com/firecrawl/skills) — install with `npx skills add firecrawl/skills`- [Firecrawl CLI](https://docs.firecrawl.dev/sdks/cli)-- [Firecrawl MCP](https://github.com/mendableai/firecrawl-mcp-server)+- [Firecrawl MCP](https://github.com/firecrawl/firecrawl-mcp-server)The build skills (integrating Firecrawl into product code) are authored in this repo under [`skills/`](./skills) and mirrored into the catalog by CI. Contributing skills? CLI skills (including the research/developer index skills) → PR [`firecrawl/cli`](https://github.com/firecrawl/cli). Build/SDK skills → PR this repo (`skills/`). Workflow skills → PR [`firecrawl/firecrawl-workflows`](https://github.com/firecrawl/firecrawl-workflows). The catalog ([`firecrawl/skills`](https://github.com/firecrawl/skills)) is read-only — never PR it directly.@@ -870,7 +871,7 @@ The build skills (integrating Firecrawl into product code) are authored in this[View all integrations →](https://www.firecrawl.dev/integrations)-**Missing your favorite tool?** [Open an issue](https://github.com/mendableai/firecrawl/issues) and let us know!+**Missing your favorite tool?** [Open an issue](https://github.com/firecrawl/firecrawl/issues) and let us know!---diff --git a/apps/dot-net-sdk/README.md b/apps/dot-net-sdk/README.mdindex 85ec78618..0892f17c3 100644--- a/apps/dot-net-sdk/README.md+++ b/apps/dot-net-sdk/README.md@@ -1,6 +1,6 @@# Firecrawl .NET SDK-.NET SDK for the [Firecrawl API](https://firecrawl.dev) — web scraping, crawling, and data extraction.+.NET SDK for the [Firecrawl API](https://firecrawl.dev) — search, scrape, and crawl the web for AI agents.## Installationdiff --git a/apps/go-sdk/README.md b/apps/go-sdk/README.mdindex 9f5d932d1..4f4e8a319 100644--- a/apps/go-sdk/README.md+++ b/apps/go-sdk/README.md@@ -1,6 +1,6 @@# Firecrawl Go SDK-Go SDK for the [Firecrawl](https://firecrawl.dev) v2 web scraping API.+Go SDK for the [Firecrawl](https://firecrawl.dev) v2 API to search, scrape, and interact with the web for AI agents.## Requirementsdiff --git a/apps/js-sdk/firecrawl/README.md b/apps/js-sdk/firecrawl/README.mdindex 5cd6a5e48..d97928399 100644--- a/apps/js-sdk/firecrawl/README.md+++ b/apps/js-sdk/firecrawl/README.md@@ -126,7 +126,7 @@ const crawlResponse = await app.crawl('https://firecrawl.dev', {To start an asynchronous crawl, use `startCrawl`. It returns a job ID you can poll with `getCrawlStatus`.```js-const start = await app.startCrawl('https://mendable.ai', {+const start = await app.startCrawl('https://firecrawl.dev', {excludePaths: ['blog/*'],limit: 5,});@@ -323,7 +323,7 @@ await app.stopInteraction(scrapeJobId);To receive real‑time updates, start a crawl and attach a watcher.```js-const start = await app.startCrawl('https://mendable.ai', { excludePaths: ['blog/*'], limit: 5 });+const start = await app.startCrawl('https://firecrawl.dev', { excludePaths: ['blog/*'], limit: 5 });const watch = app.watcher(start.id, { kind: 'crawl', pollInterval: 2 });watch.on('document', (doc) => {@@ -346,7 +346,7 @@ await watch.start();To batch scrape multiple URLs with error handling, use the `batchScrape` method.```js-const batchScrapeResponse = await app.batchScrape(['https://firecrawl.dev', 'https://mendable.ai'], {+const batchScrapeResponse = await app.batchScrape(['https://firecrawl.dev', 'https://docs.firecrawl.dev'], {formats: ['markdown', 'html'],});```@@ -357,7 +357,7 @@ const batchScrapeResponse = await app.batchScrape(['https://firecrawl.dev', 'httTo start an asynchronous batch scrape, use `startBatchScrape` and poll with `getBatchScrapeStatus`.```js-const asyncBatchScrapeResult = await app.startBatchScrape(['https://firecrawl.dev', 'https://mendable.ai'], {+const asyncBatchScrapeResult = await app.startBatchScrape(['https://firecrawl.dev', 'https://docs.firecrawl.dev'], {formats: ['markdown', 'html'],});```@@ -367,7 +367,7 @@ const asyncBatchScrapeResult = await app.startBatchScrape(['https://firecrawl.deTo use batch scrape with real‑time updates, start the job and watch it using the watcher.```js-const start = await app.startBatchScrape(['https://firecrawl.dev', 'https://mendable.ai'], { formats: ['markdown', 'html'] });+const start = await app.startBatchScrape(['https://firecrawl.dev', 'https://docs.firecrawl.dev'], { formats: ['markdown', 'html'] });const watch = app.watcher(start.id, { kind: 'batch', pollInterval: 2 });watch.on('document', (doc) => {diff --git a/apps/ruby-sdk/README.md b/apps/ruby-sdk/README.mdindex 7435e445d..5a7c83cf0 100644--- a/apps/ruby-sdk/README.md+++ b/apps/ruby-sdk/README.md@@ -1,6 +1,6 @@# Firecrawl Ruby SDK-Ruby SDK for the [Firecrawl](https://firecrawl.dev) v2 web scraping API.+Ruby SDK for the [Firecrawl](https://firecrawl.dev) v2 API to search, scrape, and interact with the web for AI agents.## Prerequisitesdiff --git a/apps/rust-sdk/README.md b/apps/rust-sdk/README.mdindex 10f781cc0..553a449b4 100644--- a/apps/rust-sdk/README.md+++ b/apps/rust-sdk/README.md@@ -131,9 +131,9 @@ for f in formats.iter().filter(|f| f.available) {}```-### Scraping with Extract+### Scraping with JSON extraction-With Extract, you can easily extract structured data from any URL. You need to specify your schema in the JSON Schema format, using the `serde_json::json!` macro.+With the JSON format, you can easily extract structured data from any URL. You need to specify your schema in the JSON Schema format, using the `serde_json::json!` macro.```rustlet json_schema = json!({@@ -160,20 +160,23 @@ let json_schema = json!({});let llm_extraction_options = ScrapeOptions {- formats: vec![ ScrapeFormats::Extract ].into(),- extract: ExtractOptions {+ formats: vec![ Format::Json ].into(),+ json_options: JsonOptions {schema: json_schema.into(),..Default::default()}.into(),..Default::default()};-let llm_extraction_result = app- .scrape_url("https://news.ycombinator.com", llm_extraction_options)+let llm_extraction_result = client+ .scrape("https://news.ycombinator.com", llm_extraction_options).await;match llm_extraction_result {- Ok(data) => println!("LLM Extraction Result:\n{:#?}", data.extract.unwrap()),+ Ok(data) => match data.json {+ Some(json) => println!("LLM Extraction Result:\n{:#?}", json),+ None => eprintln!("LLM Extraction returned no JSON"),+ },Err(e) => eprintln!("LLM Extraction failed: {}", e),}```@@ -189,7 +192,7 @@ let crawl_options = CrawlOptions {};let crawl_result = app- .crawl_url("https://mendable.ai", crawl_options)+ .crawl_url("https://firecrawl.dev", crawl_options).await;match crawl_result {@@ -203,7 +206,7 @@ match crawl_result {To crawl without waiting for the result, use the `crawl_url_async` method. It takes the same parameters, but it returns a `CrawlAsyncRespone` struct, containing the crawl's ID. You can use that ID with the `check_crawl_status` method to check the status at any time. Do note that completed crawls are deleted after 24 hours.```rust-let crawl_id = app.crawl_url_async("https://mendable.ai", None).await?.id;+let crawl_id = app.crawl_url_async("https://firecrawl.dev", None).await?.id;// ... later ...