Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

docs: update README and SDK README copy (#5042)

Erik Engervall18f4d82c5d2f
diff --git a/README.md b/README.md
index d8b347521..26e33930f 100644
--- a/README.md
+++ b/README.md
@@ -80,6 +80,7 @@ _Pst. Hey, you, join our stargazers :)_
| [**Crawl**](#crawl) | Scrape all URLs of a website with a single request |
| [**Map**](#map) | Discover all URLs on a website instantly |
| [**Batch Scrape**](#batch-scrape) | Scrape thousands of URLs asynchronously |
+| [**Monitor**](https://docs.firecrawl.dev/features/monitoring) | Get notified when content on a website changes |
---
@@ -146,7 +147,7 @@ Output:
### Scrape
-Get LLM-ready data from any website — markdown, JSON, screenshots, and more.
+Get LLM-ready context from any website — markdown, JSON, screenshots, and more.
```python
from firecrawl import Firecrawl
@@ -472,7 +473,7 @@ Response:
{
"success": true,
"links": [
- {"url": "https://firecrawl.dev", "title": "Firecrawl", "description": "Turn websites into LLM-ready data"},
+ {"url": "https://firecrawl.dev", "title": "Firecrawl", "description": "The context API for AI agents"},
{"url": "https://firecrawl.dev/pricing", "title": "Pricing", "description": "Firecrawl pricing plans"},
{"url": "https://firecrawl.dev/blog", "title": "Blog", "description": "Firecrawl blog"}
]
@@ -519,7 +520,7 @@ Our SDKs provide a convenient way to use all Firecrawl features and automaticall
Install the SDK:
```bash
-pip install firecrawl-py
+pip install firecrawl
```
```python
from firecrawl import Firecrawl
@@ -859,7 +860,7 @@ print_r($results);
**Agents & AI Tools**
- [Firecrawl Skills Catalog](https://github.com/firecrawl/skills) — install with `npx skills add firecrawl/skills`
- [Firecrawl CLI](https://docs.firecrawl.dev/sdks/cli)
-- [Firecrawl MCP](https://github.com/mendableai/firecrawl-mcp-server)
+- [Firecrawl MCP](https://github.com/firecrawl/firecrawl-mcp-server)
The build skills (integrating Firecrawl into product code) are authored in this repo under [`skills/`](./skills) and mirrored into the catalog by CI. Contributing skills? CLI skills (including the research/developer index skills) → PR [`firecrawl/cli`](https://github.com/firecrawl/cli). Build/SDK skills → PR this repo (`skills/`). Workflow skills → PR [`firecrawl/firecrawl-workflows`](https://github.com/firecrawl/firecrawl-workflows). The catalog ([`firecrawl/skills`](https://github.com/firecrawl/skills)) is read-only — never PR it directly.
@@ -870,7 +871,7 @@ The build skills (integrating Firecrawl into product code) are authored in this
[View all integrations →](https://www.firecrawl.dev/integrations)
-**Missing your favorite tool?** [Open an issue](https://github.com/mendableai/firecrawl/issues) and let us know!
+**Missing your favorite tool?** [Open an issue](https://github.com/firecrawl/firecrawl/issues) and let us know!
---
diff --git a/apps/dot-net-sdk/README.md b/apps/dot-net-sdk/README.md
index 85ec78618..0892f17c3 100644
--- a/apps/dot-net-sdk/README.md
+++ b/apps/dot-net-sdk/README.md
@@ -1,6 +1,6 @@
# Firecrawl .NET SDK
-.NET SDK for the [Firecrawl API](https://firecrawl.dev) — web scraping, crawling, and data extraction.
+.NET SDK for the [Firecrawl API](https://firecrawl.dev) — search, scrape, and crawl the web for AI agents.
## Installation
diff --git a/apps/go-sdk/README.md b/apps/go-sdk/README.md
index 9f5d932d1..4f4e8a319 100644
--- a/apps/go-sdk/README.md
+++ b/apps/go-sdk/README.md
@@ -1,6 +1,6 @@
# Firecrawl Go SDK
-Go SDK for the [Firecrawl](https://firecrawl.dev) v2 web scraping API.
+Go SDK for the [Firecrawl](https://firecrawl.dev) v2 API to search, scrape, and interact with the web for AI agents.
## Requirements
diff --git a/apps/js-sdk/firecrawl/README.md b/apps/js-sdk/firecrawl/README.md
index 5cd6a5e48..d97928399 100644
--- a/apps/js-sdk/firecrawl/README.md
+++ b/apps/js-sdk/firecrawl/README.md
@@ -126,7 +126,7 @@ const crawlResponse = await app.crawl('https://firecrawl.dev', {
To start an asynchronous crawl, use `startCrawl`. It returns a job ID you can poll with `getCrawlStatus`.
```js
-const start = await app.startCrawl('https://mendable.ai', {
+const start = await app.startCrawl('https://firecrawl.dev', {
excludePaths: ['blog/*'],
limit: 5,
});
@@ -323,7 +323,7 @@ await app.stopInteraction(scrapeJobId);
To receive real‑time updates, start a crawl and attach a watcher.
```js
-const start = await app.startCrawl('https://mendable.ai', { excludePaths: ['blog/*'], limit: 5 });
+const start = await app.startCrawl('https://firecrawl.dev', { excludePaths: ['blog/*'], limit: 5 });
const watch = app.watcher(start.id, { kind: 'crawl', pollInterval: 2 });
watch.on('document', (doc) => {
@@ -346,7 +346,7 @@ await watch.start();
To batch scrape multiple URLs with error handling, use the `batchScrape` method.
```js
-const batchScrapeResponse = await app.batchScrape(['https://firecrawl.dev', 'https://mendable.ai'], {
+const batchScrapeResponse = await app.batchScrape(['https://firecrawl.dev', 'https://docs.firecrawl.dev'], {
formats: ['markdown', 'html'],
});
```
@@ -357,7 +357,7 @@ const batchScrapeResponse = await app.batchScrape(['https://firecrawl.dev', 'htt
To start an asynchronous batch scrape, use `startBatchScrape` and poll with `getBatchScrapeStatus`.
```js
-const asyncBatchScrapeResult = await app.startBatchScrape(['https://firecrawl.dev', 'https://mendable.ai'], {
+const asyncBatchScrapeResult = await app.startBatchScrape(['https://firecrawl.dev', 'https://docs.firecrawl.dev'], {
formats: ['markdown', 'html'],
});
```
@@ -367,7 +367,7 @@ const asyncBatchScrapeResult = await app.startBatchScrape(['https://firecrawl.de
To use batch scrape with real‑time updates, start the job and watch it using the watcher.
```js
-const start = await app.startBatchScrape(['https://firecrawl.dev', 'https://mendable.ai'], { formats: ['markdown', 'html'] });
+const start = await app.startBatchScrape(['https://firecrawl.dev', 'https://docs.firecrawl.dev'], { formats: ['markdown', 'html'] });
const watch = app.watcher(start.id, { kind: 'batch', pollInterval: 2 });
watch.on('document', (doc) => {
diff --git a/apps/ruby-sdk/README.md b/apps/ruby-sdk/README.md
index 7435e445d..5a7c83cf0 100644
--- a/apps/ruby-sdk/README.md
+++ b/apps/ruby-sdk/README.md
@@ -1,6 +1,6 @@
# Firecrawl Ruby SDK
-Ruby SDK for the [Firecrawl](https://firecrawl.dev) v2 web scraping API.
+Ruby SDK for the [Firecrawl](https://firecrawl.dev) v2 API to search, scrape, and interact with the web for AI agents.
## Prerequisites
diff --git a/apps/rust-sdk/README.md b/apps/rust-sdk/README.md
index 10f781cc0..553a449b4 100644
--- a/apps/rust-sdk/README.md
+++ b/apps/rust-sdk/README.md
@@ -131,9 +131,9 @@ for f in formats.iter().filter(|f| f.available) {
}
```
-### Scraping with Extract
+### Scraping with JSON extraction
-With Extract, you can easily extract structured data from any URL. You need to specify your schema in the JSON Schema format, using the `serde_json::json!` macro.
+With the JSON format, you can easily extract structured data from any URL. You need to specify your schema in the JSON Schema format, using the `serde_json::json!` macro.
```rust
let json_schema = json!({
@@ -160,20 +160,23 @@ let json_schema = json!({
});
let llm_extraction_options = ScrapeOptions {
- formats: vec![ ScrapeFormats::Extract ].into(),
- extract: ExtractOptions {
+ formats: vec![ Format::Json ].into(),
+ json_options: JsonOptions {
schema: json_schema.into(),
..Default::default()
}.into(),
..Default::default()
};
-let llm_extraction_result = app
- .scrape_url("https://news.ycombinator.com", llm_extraction_options)
+let llm_extraction_result = client
+ .scrape("https://news.ycombinator.com", llm_extraction_options)
.await;
match llm_extraction_result {
- Ok(data) => println!("LLM Extraction Result:\n{:#?}", data.extract.unwrap()),
+ Ok(data) => match data.json {
+ Some(json) => println!("LLM Extraction Result:\n{:#?}", json),
+ None => eprintln!("LLM Extraction returned no JSON"),
+ },
Err(e) => eprintln!("LLM Extraction failed: {}", e),
}
```
@@ -189,7 +192,7 @@ let crawl_options = CrawlOptions {
};
let crawl_result = app
- .crawl_url("https://mendable.ai", crawl_options)
+ .crawl_url("https://firecrawl.dev", crawl_options)
.await;
match crawl_result {
@@ -203,7 +206,7 @@ match crawl_result {
To crawl without waiting for the result, use the `crawl_url_async` method. It takes the same parameters, but it returns a `CrawlAsyncRespone` struct, containing the crawl's ID. You can use that ID with the `check_crawl_status` method to check the status at any time. Do note that completed crawls are deleted after 24 hours.
```rust
-let crawl_id = app.crawl_url_async("https://mendable.ai", None).await?.id;
+let crawl_id = app.crawl_url_async("https://firecrawl.dev", None).await?.id;
// ... later ...