Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
updated readme
rafaelmmiller4f904e774831
diff --git a/README.md b/README.mdindex b042290f5..8fd53adc7 100644--- a/README.md+++ b/README.md@@ -55,9 +55,10 @@ We provide an easy to use API with our hosted version. You can find the playgrouCheck out the following resources to get started:- [x] **API**: [Documentation](https://docs.firecrawl.dev/api-reference/introduction)-- [x] **SDKs**: [Python](https://docs.firecrawl.dev/sdks/python), [Node](https://docs.firecrawl.dev/sdks/node), [Go](https://docs.firecrawl.dev/sdks/go), [Rust](https://docs.firecrawl.dev/sdks/rust)+- [x] **SDKs**: [Python](https://docs.firecrawl.dev/sdks/python), [Node](https://docs.firecrawl.dev/sdks/node)- [x] **LLM Frameworks**: [Langchain (python)](https://python.langchain.com/docs/integrations/document_loaders/firecrawl/), [Langchain (js)](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/firecrawl), [Llama Index](https://docs.llamaindex.ai/en/latest/examples/data_connectors/WebPageDemo/#using-firecrawl-reader), [Crew.ai](https://docs.crewai.com/), [Composio](https://composio.dev/tools/firecrawl/all), [PraisonAI](https://docs.praison.ai/firecrawl/), [Superinterface](https://superinterface.ai/docs/assistants/functions/firecrawl), [Vectorize](https://docs.vectorize.io/integrations/source-connectors/firecrawl)- [x] **Low-code Frameworks**: [Dify](https://dify.ai/blog/dify-ai-blog-integrated-with-firecrawl), [Langflow](https://docs.langflow.org/), [Flowise AI](https://docs.flowiseai.com/integrations/langchain/document-loaders/firecrawl), [Cargo](https://docs.getcargo.io/integration/firecrawl), [Pipedream](https://pipedream.com/apps/firecrawl/)+- [x] **Community SDKs**: [Go](https://docs.firecrawl.dev/sdks/go), [Rust](https://docs.firecrawl.dev/sdks/rust)- [x] **Others**: [Zapier](https://zapier.com/apps/firecrawl/integrations), [Pabbly Connect](https://www.pabbly.com/connect/integrations/firecrawl/)- [ ] Want an SDK or Integration? Let us know by opening an issue.@@ -91,7 +92,7 @@ You can find all of Firecrawl's capabilities and how to use them in our [documenUsed to crawl a URL and all accessible subpages. This submits a crawl job and returns a job ID to check the status of the crawl.```bash-curl -X POST https://api.firecrawl.dev/v1/crawl \+curl -X POST https://api.firecrawl.dev/v2/crawl \-H 'Content-Type: application/json' \-H 'Authorization: Bearer fc-YOUR_API_KEY' \-d '{@@ -109,7 +110,7 @@ Returns a crawl job id and the url to check the status of the crawl.{"success": true,"id": "123-456-789",- "url": "https://api.firecrawl.dev/v1/crawl/123-456-789"+ "url": "https://api.firecrawl.dev/v2/crawl/123-456-789"}```@@ -118,7 +119,7 @@ Returns a crawl job id and the url to check the status of the crawl.Used to check the status of a crawl job and get its result.```bash-curl -X GET https://api.firecrawl.dev/v1/crawl/123-456-789 \+curl -X GET https://api.firecrawl.dev/v2/crawl/123-456-789 \-H 'Content-Type: application/json' \-H 'Authorization: Bearer YOUR_API_KEY'```@@ -151,7 +152,7 @@ curl -X GET https://api.firecrawl.dev/v1/crawl/123-456-789 \Used to scrape a URL and get its content in the specified formats.```bash-curl -X POST https://api.firecrawl.dev/v1/scrape \+curl -X POST https://api.firecrawl.dev/v2/scrape \-H 'Content-Type: application/json' \-H 'Authorization: Bearer YOUR_API_KEY' \-d '{@@ -192,7 +193,7 @@ Response:Used to map a URL and get urls of the website. This returns most links present on the website.```bash cURL-curl -X POST https://api.firecrawl.dev/v1/map \+curl -X POST https://api.firecrawl.dev/v2/map \-H 'Content-Type: application/json' \-H 'Authorization: Bearer YOUR_API_KEY' \-d '{@@ -204,13 +205,13 @@ Response:```json{- "status": "success",+ "success": true,"links": [- "https://firecrawl.dev",- "https://www.firecrawl.dev/pricing",- "https://www.firecrawl.dev/blog",- "https://www.firecrawl.dev/playground",- "https://www.firecrawl.dev/smart-crawl",+ { "url": "https://firecrawl.dev", "title": "Firecrawl", "description": "Firecrawl is a tool that allows you to crawl a website and get the data you need." },+ { "url": "https://www.firecrawl.dev/pricing", "title": "Firecrawl Pricing", "description": "Firecrawl Pricing" },+ { "url": "https://www.firecrawl.dev/blog", "title": "Firecrawl Blog", "description": "Firecrawl Blog" },+ { "url": "https://www.firecrawl.dev/playground", "title": "Firecrawl Playground", "description": "Firecrawl Playground" },+ { "url": "https://www.firecrawl.dev/smart-crawl", "title": "Firecrawl Smart Crawl", "description": "Firecrawl Smart Crawl" }]}```@@ -220,7 +221,7 @@ Response:Map with `search` param allows you to search for specific urls inside a website.```bash cURL-curl -X POST https://api.firecrawl.dev/v1/map \+curl -X POST https://api.firecrawl.dev/v2/map \-H 'Content-Type: application/json' \-H 'Authorization: Bearer YOUR_API_KEY' \-d '{@@ -233,11 +234,11 @@ Response will be an ordered list from the most relevant to the least relevant.```json{- "status": "success",+ "success": true,"links": [- "https://docs.firecrawl.dev",- "https://docs.firecrawl.dev/sdks/python",- "https://docs.firecrawl.dev/learn/rag-llama3",+ { "url": "https://docs.firecrawl.dev", "title": "Firecrawl Docs", "description": "Firecrawl Docs" },+ { "url": "https://docs.firecrawl.dev/sdks/python", "title": "Firecrawl Python SDK", "description": "Firecrawl Python SDK" },+ { "url": "https://docs.firecrawl.dev/learn/rag-llama3", "title": "Firecrawl RAG Llama 3", "description": "Firecrawl RAG Llama 3" }]}```@@ -254,7 +255,7 @@ Firecrawl’s search API allows you to perform web searches and optionally scrap- Control the number of results and set timeouts```bash-curl -X POST https://api.firecrawl.dev/v1/search \+curl -X POST https://api.firecrawl.dev/v2/search \-H "Content-Type: application/json" \-H "Authorization: Bearer fc-YOUR_API_KEY" \-d '{@@ -286,7 +287,7 @@ curl -X POST https://api.firecrawl.dev/v1/search \#### With content scraping```bash-curl -X POST https://api.firecrawl.dev/v1/search \+curl -X POST https://api.firecrawl.dev/v2/search \-H "Content-Type: application/json" \-H "Authorization: Bearer fc-YOUR_API_KEY" \-d '{@@ -313,7 +314,7 @@ Example: https://firecrawl.dev/*When you use /*, Firecrawl will automatically crawl and parse all URLs it can discover in that domain, then extract the requested data.```bash-curl -X POST https://api.firecrawl.dev/v1/extract \+curl -X POST https://api.firecrawl.dev/v2/extract \-H 'Content-Type: application/json' \-H 'Authorization: Bearer YOUR_API_KEY' \-d '{@@ -372,38 +373,26 @@ If you are using the sdks, it will auto pull the response for you:Used to extract structured data from scraped pages.```bash-curl -X POST https://api.firecrawl.dev/v1/scrape \- -H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY' \- -d '{- "url": "https://www.mendable.ai/",- "formats": ["json"],- "jsonOptions": {+curl -X POST https://api.firecrawl.dev/v2/scrape \+ -H 'Content-Type: application/json' \+ -H 'Authorization: Bearer YOUR_API_KEY' \+ -d '{+ "url": "https://www.mendable.ai/",+ "formats": [+ {+ "type": "json","schema": {"type": "object","properties": {- "company_mission": {- "type": "string"- },- "supports_sso": {- "type": "boolean"- },- "is_open_source": {- "type": "boolean"- },- "is_in_yc": {- "type": "boolean"- }- },- "required": [- "company_mission",- "supports_sso",- "is_open_source",- "is_in_yc"- ]+ "company_mission": { "type": "string" },+ "supports_sso": { "type": "boolean" },+ "is_open_source": { "type": "boolean" },+ "is_in_yc": { "type": "boolean" }+ }}}- }'+ ]+ }'``````json@@ -438,15 +427,17 @@ curl -X POST https://api.firecrawl.dev/v1/scrape \You can now extract without a schema by just passing a `prompt` to the endpoint. The llm chooses the structure of the data.```bash-curl -X POST https://api.firecrawl.dev/v1/scrape \+curl -X POST https://api.firecrawl.dev/v2/scrape \-H 'Content-Type: application/json' \-H 'Authorization: Bearer YOUR_API_KEY' \-d '{"url": "https://docs.firecrawl.dev/",- "formats": ["json"],- "jsonOptions": {- "prompt": "Extract the company mission from the page."- }+ "formats": [+ {+ "type": "json",+ "prompt": "Extract the company mission from the page."+ }+ ]}'```@@ -457,7 +448,7 @@ Firecrawl allows you to perform various actions on a web page before scraping itHere is an example of how to use actions to navigate to google.com, search for Firecrawl, click on the first result, and take a screenshot.```bash-curl -X POST https://api.firecrawl.dev/v1/scrape \+curl -X POST https://api.firecrawl.dev/v2/scrape \-H 'Content-Type: application/json' \-H 'Authorization: Bearer YOUR_API_KEY' \-d '{@@ -483,7 +474,7 @@ curl -X POST https://api.firecrawl.dev/v1/scrape \You can now batch scrape multiple URLs at the same time. It is very similar to how the /crawl endpoint works. It submits a batch scrape job and returns a job ID to check the status of the batch scrape.```bash-curl -X POST https://api.firecrawl.dev/v1/batch/scrape \+curl -X POST https://api.firecrawl.dev/v2/batch/scrape \-H 'Content-Type: application/json' \-H 'Authorization: Bearer YOUR_API_KEY' \-d '{@@ -505,27 +496,25 @@ pip install firecrawl-py### Crawl a website```python-from firecrawl.firecrawl import FirecrawlApp-from firecrawl.firecrawl import ScrapeOptions+from firecrawl import Firecrawl-app = FirecrawlApp(api_key="fc-YOUR_API_KEY")+firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")-# Scrape a website:-scrape_status = app.scrape_url(- 'https://firecrawl.dev',- formats=["markdown", "html"]+# Scrape a website (returns a Document)+doc = firecrawl.scrape(+ "https://firecrawl.dev",+ formats=["markdown", "html"],)-print(scrape_status)--# Crawl a website:-crawl_status = app.crawl_url(- 'https://firecrawl.dev',- limit=100,- scrape_options=ScrapeOptions(- formats=["markdown", "html"],),- poll_interval=30+print(doc.markdown)++# Crawl a website+response = firecrawl.crawl(+ "https://firecrawl.dev",+ limit=100,+ scrape_options={"formats": ["markdown", "html"]},+ poll_interval=30,)-print(crawl_status)+print(response)```### Extracting structured data from a URL@@ -533,20 +522,24 @@ print(crawl_status)With LLM extraction, you can easily extract structured data from any URL. We support pydantic schemas to make it easier for you too. Here is how you to use it:```python-class ArticleSchema(BaseModel):+from pydantic import BaseModel, Field+from typing import List++class Article(BaseModel):title: str- points: int+ points: intby: strcommentsURL: str-class TopArticlesSchema(BaseModel):- top: List[ArticleSchema] = Field(..., description="Top 5 stories")+class TopArticles(BaseModel):+ top: List[Article] = Field(..., description="Top 5 stories")-json_config = JsonConfig(schema=TopArticlesSchema.model_json_schema())--llm_extraction_result = app.scrape_url('https://news.ycombinator.com', formats=["json"], json=json_config)--print(llm_extraction_result.json)+# Use JSON format with a Pydantic schema+doc = firecrawl.scrape(+ "https://news.ycombinator.com",+ formats=[{"type": "json", "schema": TopArticles}],+)+print(doc.json)```## Using the Node SDK@@ -562,33 +555,25 @@ npm install @mendable/firecrawl-js### Usage1. Get an API key from [firecrawl.dev](https://firecrawl.dev)-2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `FirecrawlApp` class.+2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `Firecrawl` class.```js-import FirecrawlApp, { CrawlParams, CrawlStatusResponse } from '@mendable/firecrawl-js';+import Firecrawl from '@mendable/firecrawl-js';-const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});+const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR_API_KEY' });// Scrape a website-const scrapeResponse = await app.scrapeUrl('https://firecrawl.dev', {+const doc = await firecrawl.scrape('https://firecrawl.dev', {formats: ['markdown', 'html'],});--if (scrapeResponse) {- console.log(scrapeResponse)-}+console.log(doc);// Crawl a website-const crawlResponse = await app.crawlUrl('https://firecrawl.dev', {+const response = await firecrawl.crawl('https://firecrawl.dev', {limit: 100,- scrapeOptions: {- formats: ['markdown', 'html'],- }-} satisfies CrawlParams, true, 30) satisfies CrawlStatusResponse;--if (crawlResponse) {- console.log(crawlResponse)-}+ scrapeOptions: { formats: ['markdown', 'html'] },+});+console.log(response);```@@ -597,12 +582,10 @@ if (crawlResponse) {With LLM extraction, you can easily extract structured data from any URL. We support zod schema to make it easier for you too. Here is how to use it:```js-import FirecrawlApp from "@mendable/firecrawl-js";-import { z } from "zod";+import Firecrawl from '@mendable/firecrawl-js';+import { z } from 'zod';-const app = new FirecrawlApp({- apiKey: "fc-YOUR_API_KEY"-});+const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR_API_KEY' });// Define schema to extract contents intoconst schema = z.object({@@ -616,14 +599,17 @@ const schema = z.object({})).length(5)- .describe("Top 5 stories on Hacker News"),+ .describe('Top 5 stories on Hacker News'),});-const scrapeResult = await app.scrapeUrl("https://news.ycombinator.com", {- jsonOptions: { extractionSchema: schema },+// Use the v2 extract API with direct Zod schema support+const extractRes = await firecrawl.extract({+ urls: ['https://news.ycombinator.com'],+ schema,+ prompt: 'Extract the top 5 stories',});-console.log(scrapeResult.data["json"]);+console.log(extractRes);```## Open Source vs Cloud Offeringdiff --git a/img/firecrawl_logo.png b/img/firecrawl_logo.pngindex bd7232223..67976269d 100644Binary files a/img/firecrawl_logo.png and b/img/firecrawl_logo.png differdiff --git a/img/firecrawl_logo_v1.png b/img/firecrawl_logo_v1.pngnew file mode 100644index 000000000..bd7232223Binary files /dev/null and b/img/firecrawl_logo_v1.png differ