Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Update README.md
rhys-firecrawl76c42264b384
diff --git a/README.md b/README.mdindex 5373f53fb..fbc08e520 100644--- a/README.md+++ b/README.md@@ -5,20 +5,22 @@height="200"></h3>+<div align="center">- <a href="https://github.com/firecrawl/firecrawl/blob/main/LICENSE">- <img src="https://img.shields.io/github/license/firecrawl/firecrawl" alt="License">-</a>- <a href="https://pepy.tech/project/firecrawl-py">- <img src="https://static.pepy.tech/badge/firecrawl-py" alt="Downloads">-</a>-<a href="https://GitHub.com/firecrawl/firecrawl/graphs/contributors">- <img src="https://img.shields.io/github/contributors/firecrawl/firecrawl.svg" alt="GitHub Contributors">-</a>-<a href="https://firecrawl.dev">- <img src="https://img.shields.io/badge/Visit-firecrawl.dev-orange" alt="Visit firecrawl.dev">-</a>+ <a href="https://github.com/firecrawl/firecrawl/blob/main/LICENSE">+ <img src="https://img.shields.io/github/license/firecrawl/firecrawl" alt="License">+ </a>+ <a href="https://pepy.tech/project/firecrawl-py">+ <img src="https://static.pepy.tech/badge/firecrawl-py" alt="Downloads">+ </a>+ <a href="https://GitHub.com/firecrawl/firecrawl/graphs/contributors">+ <img src="https://img.shields.io/github/contributors/firecrawl/firecrawl.svg" alt="GitHub Contributors">+ </a>+ <a href="https://firecrawl.dev">+ <img src="https://img.shields.io/badge/Visit-firecrawl.dev-orange" alt="Visit firecrawl.dev">+ </a></div>+<div><p align="center"><a href="https://twitter.com/firecrawl_dev">@@ -33,626 +35,601 @@</p></div>-# 🔥 Firecrawl--Empower your AI apps with clean data from any website. Featuring advanced scraping, crawling, and data extraction capabilities.+----_This repository is in development, and we’re still integrating custom modules into the mono repo. It's not fully ready for self-hosted deployment yet, but you can run it locally._+# **🔥 Firecrawl**-## What is Firecrawl?+**Turn websites into LLM-ready data.** [**Firecrawl**](https://firecrawl.dev/?ref=github) is an API that crawls, scrapes, and extracts structured data from any website. Our Agent finds and gathers data from anywhere on the web, just describe what you need.-[Firecrawl](https://firecrawl.dev?ref=github) is an API service that takes a URL, crawls it, and converts it into clean markdown or structured data. We crawl all accessible subpages and give you clean data for each. No sitemap required. Check out our [documentation](https://docs.firecrawl.dev).+Looking for our MCP? Check out the repo [here](https://github.com/firecrawl/firecrawl-mcp-server).-Looking for our MCP? Check out the [repo here](https://github.com/firecrawl/firecrawl-mcp-server).+*This repository is in development, and we're still integrating custom modules into the mono repo. It's not fully ready for self-hosted deployment yet, but you can run it locally.*-_Pst. hey, you, join our stargazers :)_+_Pst. Hey, you, join our stargazers :)_<a href="https://github.com/firecrawl/firecrawl"><img src="https://img.shields.io/github/stars/firecrawl/firecrawl.svg?style=social&label=Star&maxAge=2592000" alt="GitHub stars"></a>-## How to use it?+----We provide an easy to use API with our hosted version. You can find the playground and documentation [here](https://firecrawl.dev/playground). You can also self host the backend if you'd like.+## Why Firecrawl?-Check out the following resources to get started:-- [x] **API**: [Documentation](https://docs.firecrawl.dev/api-reference/introduction)-- [x] **SDKs**: [Python](https://docs.firecrawl.dev/sdks/python), [Node](https://docs.firecrawl.dev/sdks/node)-- [x] **LLM Frameworks**: [Langchain (python)](https://python.langchain.com/docs/integrations/document_loaders/firecrawl/), [Langchain (js)](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/firecrawl), [Llama Index](https://docs.llamaindex.ai/en/latest/examples/data_connectors/WebPageDemo/#using-firecrawl-reader), [Crew.ai](https://docs.crewai.com/), [Composio](https://composio.dev/tools/firecrawl/all), [PraisonAI](https://docs.praison.ai/firecrawl/), [Superinterface](https://superinterface.ai/docs/assistants/functions/firecrawl), [Vectorize](https://docs.vectorize.io/integrations/source-connectors/firecrawl)-- [x] **Low-code Frameworks**: [Dify](https://dify.ai/blog/dify-ai-blog-integrated-with-firecrawl), [Langflow](https://docs.langflow.org/), [Flowise AI](https://docs.flowiseai.com/integrations/langchain/document-loaders/firecrawl), [Cargo](https://docs.getcargo.io/integration/firecrawl), [Pipedream](https://pipedream.com/apps/firecrawl/)-- [x] **Community SDKs**: [Go](https://docs.firecrawl.dev/sdks/go), [Rust](https://docs.firecrawl.dev/sdks/rust)-- [x] **Others**: [Zapier](https://zapier.com/apps/firecrawl/integrations), [Pabbly Connect](https://www.pabbly.com/connect/integrations/firecrawl/)-- [ ] Want an SDK or Integration? Let us know by opening an issue.+- **Agent**: Just describe the data you want, no URLs required+- **LLM-ready output**: Clean markdown, structured JSON, screenshots, HTML, and more+- **Industry-leading reliability**: >80% coverage on [benchmark evaluations](https://www.firecrawl.dev/blog/the-worlds-best-web-data-api-v25), outperforming every other provider tested+- **Handles the hard stuff**: Proxies, anti-bot, JavaScript rendering, authentication flows, and dynamic content that breaks other scrapers+- **Customization**: Exclude tags, crawl behind auth walls, max depth, and more+- **Media parsing**: Automatic text extraction from PDFs, DOCX, and images+- **Actions**: Click, scroll, input, wait, and more before extracting+- **Batch processing**: Scrape thousands of URLs asynchronously+- **Change tracking**: Monitor website content changes over time-To run locally, refer to guide [here](https://github.com/firecrawl/firecrawl/blob/main/CONTRIBUTING.md).+----### API Key+## Using Firecrawl-To use the API, you need to sign up on [Firecrawl](https://firecrawl.dev) and get an API key.+**Quick Start**: Use our hosted API at [firecrawl.dev](https://firecrawl.dev) - sign up and get your API key to start scraping in seconds. Try the [playground](https://firecrawl.dev/playground) to test it out.-### Features+**Integrate Directly**: Firecrawl works seamlessly with popular tools.+- **SDKs**: [Python](https://docs.firecrawl.dev/sdks/python), [Node](https://docs.firecrawl.dev/sdks/node)+- **Community SDKs**: [Go](https://github.com/mendableai/firecrawl-go), [Rust](https://docs.firecrawl.dev/sdks/rust)+- **LLM Frameworks**: [Langchain (python)](https://python.langchain.com/docs/integrations/document_loaders/firecrawl/), [Langchain (js)](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/firecrawl), [Llama Index](https://docs.firecrawl.dev/integrations/llamaindex), [Crew.ai](https://docs.firecrawl.dev/integrations/crewai), [Composio](https://composio.dev/toolkits/firecrawl), [PraisonAI](https://docs.praison.ai/docs/firecrawl), [Superinterface](https://superinterface.ai/docs/assistants/functions/firecrawl), [Vectorize](https://docs.vectorize.io/integrations/source-connectors/firecrawl)+- **Low-Code**: [Dify](https://docs.dify.ai/en/use-dify/knowledge/knowledge-pipeline/knowledge-pipeline-orchestration#firecrawl), [Langflow](https://docs.firecrawl.dev/integrations/langflow), [Flowise AI](https://docs.flowiseai.com/integrations/langchain/document-loaders/firecrawl), [Cargo](https://docs.getcargo.ai/integration/firecrawl#firecrawl), [Pipedream](https://pipedream.com/apps/firecrawl)+- **Automation**: [Zapier](https://zapier.com/apps/firecrawl/integrations), [n8n](https://n8n.io/integrations/firecrawl/), [Pabbly Connect](https://www.pabbly.com/connect/integrations/firecrawl/)-- [**Scrape**](#scraping): scrapes a URL and gets its content in LLM-ready format (markdown, structured data via [LLM Extract](#llm-extraction-beta), screenshot, html)-- [**Crawl**](#crawling): scrapes all the URLs of a web page and returns content in LLM-ready format-- [**Map**](#map): input a website and get all the website urls - extremely fast-- [**Search**](#search): search the web and get full content from results-- [**Extract**](#extract): get structured data from single page, multiple pages or entire websites with AI.+**Self-Host**: Want to run it yourself? Check out our [self-hosting guide](https://docs.firecrawl.dev/contributing/self-host).-### Powerful Capabilities-- **LLM-ready formats**: markdown, structured data, screenshot, HTML, links, metadata-- **The hard stuff**: proxies, anti-bot mechanisms, dynamic content (js-rendered), output parsing, orchestration-- **Customizability**: exclude tags, crawl behind auth walls with custom headers, max crawl depth, etc...-- **Media parsing**: pdfs, docx, images-- **Reliability first**: designed to get the data you need - no matter how hard it is-- **Actions**: click, scroll, input, wait and more before extracting data-- **Batching**: scrape thousands of URLs at the same time with a new async endpoint-- **Change Tracking**: monitor and detect changes in website content over time+**Missing your favorite tool?** Open an issue and let us know!-You can find all of Firecrawl's capabilities and how to use them in our [documentation](https://docs.firecrawl.dev)+----### Crawling+## Quick Start-Used to crawl a URL and all accessible subpages. This submits a crawl job and returns a job ID to check the status of the crawl.+Sign up at [firecrawl.dev](https://firecrawl.dev) to get your API key and start extracting data in seconds.-```bash-curl -X POST https://api.firecrawl.dev/v2/crawl \- -H 'Content-Type: application/json' \- -H 'Authorization: Bearer fc-YOUR_API_KEY' \- -d '{- "url": "https://docs.firecrawl.dev",- "limit": 10,- "scrapeOptions": {- "formats": ["markdown", "html"]- }- }'-```+----Returns a crawl job id and the url to check the status of the crawl.+### Using with AI Agents (Recommended)-```json-{- "success": true,- "id": "123-456-789",- "url": "https://api.firecrawl.dev/v2/crawl/123-456-789"-}+Install the Firecrawl skill to let AI agents like Claude Code, Codex, and OpenCode use Firecrawl automatically:+```bash+npx skills add firecrawl/cli```-### Check Crawl Job+Restart your agent after installing. See the [Skill + CLI docs](https://docs.firecrawl.dev/sdks/cli) for full setup.-Used to check the status of a crawl job and get its result.+---+### Make Your First Request```bash-curl -X GET https://api.firecrawl.dev/v2/crawl/123-456-789 \+curl -X POST 'https://api.firecrawl.dev/v2/scrape' \+ -H 'Authorization: Bearer fc-YOUR_API_KEY' \-H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY'+ -d '{"url": "https://example.com"}'```+Response:```json{- "status": "completed",- "total": 36,- "creditsUsed": 36,- "expiresAt": "2024-00-00T00:00:00.000Z",- "data": [- {- "markdown": "[Firecrawl Docs home page!...",- "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",- "metadata": {- "title": "Build a 'Chat with website' using Groq Llama 3 | Firecrawl",- "language": "en",- "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",- "description": "Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.",- "ogLocaleAlternate": [],- "statusCode": 200- }+ "success": true,+ "data": {+ "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",+ "metadata": {+ "title": "Example Domain",+ "sourceURL": "https://example.com"}- ]+ }}```-### Scraping+---++## Feature Overview++| Feature | Description |+|---------|-------------|+| [**Agent**](#agent) | Automated data gathering, just describe what you need |+| [**Scrape**](#scraping) | Convert any URL to markdown, HTML, screenshots, or structured JSON |+| [**Search**](#search) | Search the web and get full page content from results |+| [**Crawl**](#crawling) | Scrape all URLs of a website with a single request |+| [**Map**](#map) | Discover all URLs on a website instantly |++---++## Agent-Used to scrape a URL and get its content in the specified formats.+**The easiest way to get data from the web.** Describe what you need, and our AI agent searches, navigates, and extracts it. No URLs required.+Agent is the evolution of our `/extract` endpoint: faster, more reliable, and doesn't require you to know the URLs upfront.```bash-curl -X POST https://api.firecrawl.dev/v2/scrape \- -H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY' \- -d '{- "url": "https://docs.firecrawl.dev",- "formats" : ["markdown", "html"]- }'+curl -X POST 'https://api.firecrawl.dev/v2/agent' \+ -H 'Authorization: Bearer fc-YOUR_API_KEY' \+ -H 'Content-Type: application/json' \+ -d '{+ "prompt": "Find the pricing plans for Notion"+ }'```Response:-```json{"success": true,"data": {- "markdown": "Launch Week I is here! [See our Day 2 Release 🚀](https://www.firecrawl.dev/blog/launch-week-i-day-2-doubled-rate-limits)[💥 Get 2 months free...",- "html": "<!DOCTYPE html><html lang=\"en\" class=\"light\" style=\"color-scheme: light;\"><body class=\"__variable_36bd41 __variable_d7dc5d font-inter ...",- "metadata": {- "title": "Home - Firecrawl",- "description": "Firecrawl crawls and converts any website into clean markdown.",- "language": "en",- "keywords": "Firecrawl,Markdown,Data,Mendable,Langchain",- "robots": "follow, index",- "ogTitle": "Firecrawl",- "ogDescription": "Turn any website into LLM-ready data.",- "ogUrl": "https://www.firecrawl.dev/",- "ogImage": "https://www.firecrawl.dev/og.png?123",- "ogLocaleAlternate": [],- "ogSiteName": "Firecrawl",- "sourceURL": "https://firecrawl.dev",- "statusCode": 200- }+ "result": "Notion offers the following pricing plans:\n\n1. Free - $0/month...\n2. Plus - $10/seat/month...\n3. Business - $18/seat/month...",+ "sources": ["https://www.notion.so/pricing"]}}```-### Map+----Used to map a URL and get URLs of the website. This returns most links present on the website.+### Agent with Structured Output-```bash cURL-curl -X POST https://api.firecrawl.dev/v2/map \- -H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY' \- -d '{- "url": "https://firecrawl.dev"- }'-```+Use a schema to get structured data:+```python+from firecrawl import Firecrawl+from pydantic import BaseModel, Field+from typing import List, Optional-Response:+app = Firecrawl(api_key="fc-YOUR_API_KEY")+class Founder(BaseModel):+ name: str = Field(description="Full name of the founder")+ role: Optional[str] = Field(None, description="Role or position")++class FoundersSchema(BaseModel):+ founders: List[Founder] = Field(description="List of founders")++result = app.agent(+ prompt="Find the founders of Firecrawl",+ schema=FoundersSchema+)++print(result.data)+``````json{- "success": true,- "links": [- { "url": "https://firecrawl.dev", "title": "Firecrawl", "description": "Firecrawl is a tool that allows you to crawl a website and get the data you need." },- { "url": "https://www.firecrawl.dev/pricing", "title": "Firecrawl Pricing", "description": "Firecrawl Pricing" },- { "url": "https://www.firecrawl.dev/blog", "title": "Firecrawl Blog", "description": "Firecrawl Blog" },- { "url": "https://www.firecrawl.dev/playground", "title": "Firecrawl Playground", "description": "Firecrawl Playground" },- { "url": "https://www.firecrawl.dev/smart-crawl", "title": "Firecrawl Smart Crawl", "description": "Firecrawl Smart Crawl" }+ "founders": [+ {"name": "Eric Ciarla", "role": "Co-founder"},+ {"name": "Nicolas Camara", "role": "Co-founder"},+ {"name": "Caleb Peffer", "role": "Co-founder"}]}```-#### Map with search+----Map with `search` param allows you to search for specific URLs inside a website.+### Agent with URLs (Optional)-```bash cURL-curl -X POST https://api.firecrawl.dev/v2/map \- -H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY' \- -d '{- "url": "https://firecrawl.dev",- "search": "docs"- }'+Focus the agent on specific pages:+```python+result = app.agent(+ urls=["https://docs.firecrawl.dev", "https://firecrawl.dev/pricing"],+ prompt="Compare the features and pricing information"+)```-Response will be an ordered list from the most relevant to the least relevant.+----```json-{- "success": true,- "links": [- { "url": "https://docs.firecrawl.dev", "title": "Firecrawl Docs", "description": "Firecrawl Docs" },- { "url": "https://docs.firecrawl.dev/sdks/python", "title": "Firecrawl Python SDK", "description": "Firecrawl Python SDK" },- { "url": "https://docs.firecrawl.dev/learn/rag-llama3", "title": "Firecrawl RAG Llama 3", "description": "Firecrawl RAG Llama 3" }- ]-}+### Model Selection++Choose between two models based on your needs:++| Model | Cost | Best For |+|-------|------|----------|+| `spark-1-mini` (default) | 60% cheaper | Most tasks |+| `spark-1-pro` | Standard | Complex research, critical extraction |+```python+result = app.agent(+ prompt="Compare enterprise features across Firecrawl, Apify, and ScrapingBee",+ model="spark-1-pro"+)```-### Search+**When to use Pro:**+- Comparing data across multiple websites+- Extracting from sites with complex navigation or auth+- Research tasks where the agent needs to explore multiple paths+- Critical data where accuracy is paramount-Search the web and get full content from results+Learn more about Spark models in our [Agent documentation](https://docs.firecrawl.dev/features/agent).-Firecrawl’s search API allows you to perform web searches and optionally scrape the search results in one operation.+----- Choose specific output formats (markdown, HTML, links, screenshots)-- Search the web with customizable parameters (language, country, etc.)-- Optionally retrieve content from search results in various formats-- Control the number of results and set timeouts+## Scraping+Convert any URL to clean markdown, HTML, or structured data.```bash-curl -X POST https://api.firecrawl.dev/v2/search \- -H "Content-Type: application/json" \- -H "Authorization: Bearer fc-YOUR_API_KEY" \+curl -X POST 'https://api.firecrawl.dev/v2/scrape' \+ -H 'Authorization: Bearer fc-YOUR_API_KEY' \+ -H 'Content-Type: application/json' \-d '{- "query": "what is firecrawl?",- "limit": 5+ "url": "https://docs.firecrawl.dev",+ "formats": ["markdown", "html"]}'```-#### Response-+Response:```json{"success": true,- "data": [- {- "url": "https://firecrawl.dev",- "title": "Firecrawl | Home Page",- "description": "Turn websites into LLM-ready data with Firecrawl"- },- {- "url": "https://docs.firecrawl.dev",- "title": "Documentation | Firecrawl",- "description": "Learn how to use Firecrawl in your own applications"+ "data": {+ "markdown": "# Firecrawl Docs\n\nTurn websites into LLM-ready data...",+ "html": "<!DOCTYPE html><html>...",+ "metadata": {+ "title": "Quickstart | Firecrawl",+ "description": "Firecrawl allows you to turn entire websites into LLM-ready markdown",+ "sourceURL": "https://docs.firecrawl.dev",+ "statusCode": 200}- ]+ }}```-#### With content scraping+----```bash-curl -X POST https://api.firecrawl.dev/v2/search \- -H "Content-Type: application/json" \- -H "Authorization: Bearer fc-YOUR_API_KEY" \- -d '{- "query": "what is firecrawl?",- "limit": 5,- "scrapeOptions": {- "formats": ["markdown", "links"]- }- }'-```+### Extract Structured Data (JSON Mode)-### Extract (Beta)+Extract structured data using a schema:+```python+from firecrawl import Firecrawl+from pydantic import BaseModel++app = Firecrawl(api_key="fc-YOUR_API_KEY")++class CompanyInfo(BaseModel):+ company_mission: str+ is_open_source: bool+ is_in_yc: bool++result = app.scrape(+ 'https://firecrawl.dev',+ formats=[{"type": "json", "schema": CompanyInfo.model_json_schema()}]+)-Get structured data from entire websites with a prompt and/or a schema.+print(result.json)+```+```json+{"company_mission": "Turn websites into LLM-ready data", "is_open_source": true, "is_in_yc": true}+```-You can extract structured data from one or multiple URLs, including wildcards:+Or extract with just a prompt (no schema):+```python+result = app.scrape(+ 'https://firecrawl.dev',+ formats=[{"type": "json", "prompt": "Extract the company mission"}]+)+```-Single Page:-Example: https://firecrawl.dev/some-page+----Multiple Pages / Full Domain-Example: https://firecrawl.dev/*+### Scrape Formats-When you use /*, Firecrawl will automatically crawl and parse all URLs it can discover in that domain, then extract the requested data.+Available formats: `markdown`, `html`, `rawHtml`, `screenshot`, `links`, `json`, `branding`-```bash-curl -X POST https://api.firecrawl.dev/v2/extract \- -H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY' \- -d '{- "urls": [- "https://firecrawl.dev/*",- "https://docs.firecrawl.dev/",- "https://www.ycombinator.com/companies"- ],- "prompt": "Extract the company mission, whether it is open source, and whether it is in Y Combinator from the page.",- "schema": {- "type": "object",- "properties": {- "company_mission": {- "type": "string"- },- "is_open_source": {- "type": "boolean"- },- "is_in_yc": {- "type": "boolean"- }- },- "required": [- "company_mission",- "is_open_source",- "is_in_yc"- ]- }- }'+**Get a screenshot**+```python+doc = app.scrape("https://firecrawl.dev", formats=["screenshot"])+print(doc.screenshot) # Base64 encoded image```-```json-{- "success": true,- "id": "44aa536d-f1cb-4706-ab87-ed0386685740",- "urlTrace": []-}+**Extract brand identity (colors, fonts, typography)**+```python+doc = app.scrape("https://firecrawl.dev", formats=["branding"])+print(doc.branding) # {"colors": {...}, "fonts": [...], "typography": {...}}```-If you are using the SDKs, it will auto pull the response for you:+---+### Actions (Interact Before Scraping)++Click, type, scroll, and more before extracting:+```python+doc = app.scrape(+ url="https://example.com/login",+ formats=["markdown"],+ actions=[+ {"type": "write", "text": "[email protected]"},+ {"type": "press", "key": "Tab"},+ {"type": "write", "text": "password"},+ {"type": "click", "selector": 'button[type="submit"]'},+ {"type": "wait", "milliseconds": 2000},+ {"type": "screenshot"}+ ]+)+```++---++## Search++Search the web and optionally scrape the results.+```bash+curl -X POST 'https://api.firecrawl.dev/v2/search' \+ -H 'Authorization: Bearer fc-YOUR_API_KEY' \+ -H 'Content-Type: application/json' \+ -d '{+ "query": "firecrawl web scraping",+ "limit": 5+ }'+```++Response:```json{"success": true,"data": {- "company_mission": "Firecrawl is the easiest way to extract data from the web. Developers use us to reliably convert URLs into LLM-ready markdown or structured data with a single API call.",- "supports_sso": false,- "is_open_source": true,- "is_in_yc": true+ "web": [+ {+ "url": "https://www.firecrawl.dev/",+ "title": "Firecrawl - The Web Data API for AI",+ "description": "The web crawling, scraping, and search API for AI.",+ "position": 1+ }+ ],+ "images": [...],+ "news": [...]}}```-### LLM Extraction (Beta)+---++### Search with Content Scraping++Get the full content of search results:+```python+from firecrawl import Firecrawl-Used to extract structured data from scraped pages.+firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")+results = firecrawl.search(+ "firecrawl web scraping",+ limit=3,+ scrape_options={+ "formats": ["markdown", "links"]+ }+)+```++---++## Crawling++Crawl an entire website and get content from all pages.```bash-curl -X POST https://api.firecrawl.dev/v2/scrape \+curl -X POST 'https://api.firecrawl.dev/v2/crawl' \+ -H 'Authorization: Bearer fc-YOUR_API_KEY' \-H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY' \-d '{- "url": "https://www.mendable.ai/",- "formats": [- {- "type": "json",- "schema": {- "type": "object",- "properties": {- "company_mission": { "type": "string" },- "supports_sso": { "type": "boolean" },- "is_open_source": { "type": "boolean" },- "is_in_yc": { "type": "boolean" }- }- }- }- ]+ "url": "https://docs.firecrawl.dev",+ "limit": 100,+ "scrapeOptions": {+ "formats": ["markdown"]+ }}'```+Returns a job ID:```json{"success": true,- "data": {- "content": "Raw Content",- "metadata": {- "title": "Mendable",- "description": "Mendable allows you to easily build AI chat applications. Ingest, customize, then deploy with one line of code anywhere you want. Brought to you by SideGuide",- "robots": "follow, index",- "ogTitle": "Mendable",- "ogDescription": "Mendable allows you to easily build AI chat applications. Ingest, customize, then deploy with one line of code anywhere you want. Brought to you by SideGuide",- "ogUrl": "https://mendable.ai/",- "ogImage": "https://mendable.ai/mendable_new_og1.png",- "ogLocaleAlternate": [],- "ogSiteName": "Mendable",- "sourceURL": "https://mendable.ai/"- },- "json": {- "company_mission": "Train a secure AI on your technical resources that answers customer and employee questions so your team doesn't have to",- "supports_sso": true,- "is_open_source": false,- "is_in_yc": true- }- }+ "id": "123-456-789",+ "url": "https://api.firecrawl.dev/v2/crawl/123-456-789"}```-### Extracting without a schema (New)--You can now extract without a schema by just passing a `prompt` to the endpoint. The LLM chooses the structure of the data.+---+### Check Crawl Status```bash-curl -X POST https://api.firecrawl.dev/v2/scrape \- -H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY' \- -d '{- "url": "https://docs.firecrawl.dev/",- "formats": [- {- "type": "json",- "prompt": "Extract the company mission from the page."- }- ]- }'+curl -X GET 'https://api.firecrawl.dev/v2/crawl/123-456-789' \+ -H 'Authorization: Bearer fc-YOUR_API_KEY'+```+```json+{+ "status": "completed",+ "total": 50,+ "completed": 50,+ "creditsUsed": 50,+ "data": [+ {+ "markdown": "# Page Title\n\nContent...",+ "metadata": {"title": "Page Title", "sourceURL": "https://..."}+ }+ ]+}```-### Interacting with the page with Actions (Cloud-only)+**Note:** The [SDKs](#sdks) handle polling automatically for a better developer experience.-Firecrawl allows you to perform various actions on a web page before scraping its content. This is particularly useful for interacting with dynamic content, navigating through pages, or accessing content that requires user interaction.+----Here is an example of how to use actions to navigate to google.com, search for Firecrawl, click on the first result, and take a screenshot.+## Map+Discover all URLs on a website instantly.```bash-curl -X POST https://api.firecrawl.dev/v2/scrape \- -H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY' \- -d '{- "url": "google.com",- "formats": ["markdown"],- "actions": [- {"type": "wait", "milliseconds": 2000},- {"type": "click", "selector": "textarea[title=\"Search\"]"},- {"type": "wait", "milliseconds": 2000},- {"type": "write", "text": "firecrawl"},- {"type": "wait", "milliseconds": 2000},- {"type": "press", "key": "ENTER"},- {"type": "wait", "milliseconds": 3000},- {"type": "click", "selector": "h3"},- {"type": "wait", "milliseconds": 3000},- {"type": "screenshot"}- ]- }'-```--### Batch Scraping Multiple URLs (New)--You can now batch scrape multiple URLs at the same time. It is very similar to how the /crawl endpoint works. It submits a batch scrape job and returns a job ID to check the status of the batch scrape.+curl -X POST 'https://api.firecrawl.dev/v2/map' \+ -H 'Authorization: Bearer fc-YOUR_API_KEY' \+ -H 'Content-Type: application/json' \+ -d '{"url": "https://firecrawl.dev"}'+```-```bash-curl -X POST https://api.firecrawl.dev/v2/batch/scrape \- -H 'Content-Type: application/json' \- -H 'Authorization: Bearer YOUR_API_KEY' \- -d '{- "urls": ["https://docs.firecrawl.dev", "https://docs.firecrawl.dev/sdks/overview"],- "formats" : ["markdown", "html"]- }'+Response:+```json+{+ "success": true,+ "links": [+ {"url": "https://firecrawl.dev", "title": "Firecrawl", "description": "Turn websites into LLM-ready data"},+ {"url": "https://firecrawl.dev/pricing", "title": "Pricing", "description": "Firecrawl pricing plans"},+ {"url": "https://firecrawl.dev/blog", "title": "Blog", "description": "Firecrawl blog"}+ ]+}```+---+### Map with Search-## Using Python SDK+Find specific URLs within a site:+```python+from firecrawl import Firecrawl-### Installing Python SDK+app = Firecrawl(api_key="fc-YOUR_API_KEY")-```bash-pip install firecrawl-py+result = app.map("https://firecrawl.dev", search="pricing")+# Returns URLs ordered by relevance to "pricing"```-### Crawl a website+---++## Batch Scraping+Scrape multiple URLs at once:```pythonfrom firecrawl import Firecrawl-firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")+app = Firecrawl(api_key="fc-YOUR_API_KEY")-# Scrape a website (returns a Document)-doc = firecrawl.scrape(+job = app.batch_scrape(["https://firecrawl.dev",- formats=["markdown", "html"],-)-print(doc.markdown)+ "https://docs.firecrawl.dev",+ "https://firecrawl.dev/pricing"+], formats=["markdown"])-# Crawl a website-response = firecrawl.crawl(- "https://firecrawl.dev",- limit=100,- scrape_options={"formats": ["markdown", "html"]},- poll_interval=30,-)-print(response)+for doc in job.data:+ print(doc.metadata.source_url)```-### Extracting structured data from a URL+---++## SDKs-With LLM extraction, you can easily extract structured data from any URL. We support pydantic schemas to make it easier for you too. Here is how to use it:+Our SDKs provide a convenient way to interact with all Firecrawl features and automatically handle polling for async operations like crawling and batch scraping.+### Python++Install the SDK:+```bash+pip install firecrawl-py+``````python-from pydantic import BaseModel, Field-from typing import List+from firecrawl import Firecrawl-class Article(BaseModel):- title: str- points: int- by: str- commentsURL: str+app = Firecrawl(api_key="fc-YOUR_API_KEY")-class TopArticles(BaseModel):- top: List[Article] = Field(..., description="Top 5 stories")+# Scrape a single URL+doc = app.scrape("https://firecrawl.dev", formats=["markdown"])+print(doc.markdown)-# Use JSON format with a Pydantic schema-doc = firecrawl.scrape(- "https://news.ycombinator.com",- formats=[{"type": "json", "schema": TopArticles}],-)-print(doc.json)-```+# Use the Agent for autonomous data gathering+result = app.agent(prompt="Find the founders of Stripe")+print(result.data)-## Using the Node SDK+# Crawl a website (automatically waits for completion)+docs = app.crawl("https://docs.firecrawl.dev", limit=50)+for doc in docs.data:+ print(doc.metadata.source_url, doc.markdown[:100])-### Installation+# Search the web+results = app.search("best web scraping tools 2024", limit=10)+print(results)+```++----To install the Firecrawl Node SDK, you can use npm:+### Node.js+Install the SDK:```bashnpm install @mendable/firecrawl-js```+```javascript+import Firecrawl from '@mendable/firecrawl-js';-### Usage--1. Get an API key from [firecrawl.dev](https://firecrawl.dev)-2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `Firecrawl` class.+const app = new Firecrawl({ apiKey: 'fc-YOUR_API_KEY' });-```js-import Firecrawl from '@mendable/firecrawl-js';+// Scrape a single URL+const doc = await app.scrape('https://firecrawl.dev', { formats: ['markdown'] });+console.log(doc.markdown);-const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR_API_KEY' });+// Use the Agent for autonomous data gathering+const result = await app.agent({ prompt: 'Find the founders of Stripe' });+console.log(result.data);-// Scrape a website-const doc = await firecrawl.scrape('https://firecrawl.dev', {- formats: ['markdown', 'html'],+// Crawl a website (automatically waits for completion)+const docs = await app.crawl('https://docs.firecrawl.dev', { limit: 50 });+docs.data.forEach(doc => {+ console.log(doc.metadata.sourceURL, doc.markdown.substring(0, 100));});-console.log(doc);-// Crawl a website-const response = await firecrawl.crawl('https://firecrawl.dev', {- limit: 100,- scrapeOptions: { formats: ['markdown', 'html'] },+// Search the web+const results = await app.search('best web scraping tools 2024', { limit: 10 });+results.data.web.forEach(result => {+ console.log(`${result.title}: ${result.url}`);});-console.log(response);```+----### Extracting structured data from a URL+### Community SDKs-With LLM extraction, you can easily extract structured data from any URL. We support zod schema to make it easier for you too. Here is how to use it:+- [Go SDK](https://github.com/mendableai/firecrawl-go)+- [Rust SDK](https://docs.firecrawl.dev/sdks/rust)-```js-import Firecrawl from '@mendable/firecrawl-js';-import { z } from 'zod';--const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR_API_KEY' });--// Define schema to extract contents into-const schema = z.object({- top: z- .array(- z.object({- title: z.string(),- points: z.number(),- by: z.string(),- commentsURL: z.string(),- })- )- .length(5)- .describe('Top 5 stories on Hacker News'),-});+----// Use the v2 extract API with direct Zod schema support-const extractRes = await firecrawl.extract({- urls: ['https://news.ycombinator.com'],- schema,- prompt: 'Extract the top 5 stories',-});+## Integrations-console.log(extractRes);-```+- **LLM Frameworks**: [Langchain (python)](https://python.langchain.com/docs/integrations/document_loaders/firecrawl/), [Langchain (js)](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/firecrawl), [Llama Index](https://docs.firecrawl.dev/integrations/llamaindex), [Crew.ai](https://docs.firecrawl.dev/integrations/crewai), [Composio](https://composio.dev/toolkits/firecrawl), [PraisonAI](https://docs.praison.ai/docs/firecrawl), [Superinterface](https://superinterface.ai/docs/assistants/functions/firecrawl), [Vectorize](https://docs.vectorize.io/integrations/source-connectors/firecrawl)+- **Low-Code**: [Dify](https://docs.dify.ai/en/use-dify/knowledge/knowledge-pipeline/knowledge-pipeline-orchestration#firecrawl), [Langflow](https://docs.firecrawl.dev/integrations/langflow), [Flowise AI](https://docs.flowiseai.com/integrations/langchain/document-loaders/firecrawl), [Cargo](https://docs.getcargo.ai/integration/firecrawl#firecrawl), [Pipedream](https://pipedream.com/apps/firecrawl)+- **Automation**: [Zapier](https://zapier.com/apps/firecrawl/integrations), [n8n](https://n8n.io/integrations/firecrawl/), [Pabbly Connect](https://www.pabbly.com/connect/integrations/firecrawl/)+- **MCP Server**: [firecrawl-mcp-server](https://github.com/mendableai/firecrawl-mcp-server)-## Open Source vs Cloud Offering+----Firecrawl is open source available under the AGPL-3.0 license.+## Resources-To deliver the best possible product, we offer a hosted version of Firecrawl alongside our open-source offering. The cloud solution allows us to continuously innovate and maintain a high-quality, sustainable service for all users.+- [Documentation](https://docs.firecrawl.dev)+- [API Reference](https://docs.firecrawl.dev/api-reference/introduction)+- [Playground](https://firecrawl.dev/playground)+- [Changelog](https://firecrawl.dev/changelog)-Firecrawl Cloud is available at [firecrawl.dev](https://firecrawl.dev) and offers a range of features that are not available in the open source version:+---++## Open Source vs Cloud++Firecrawl is open source under the AGPL-3.0 license. The cloud version at [firecrawl.dev](https://firecrawl.dev) includes additional features:+To run locally, see the [Contributing Guide](https://github.com/firecrawl/firecrawl/blob/main/CONTRIBUTING.md). To self-host, see [Self-Hosting Guide](https://docs.firecrawl.dev/contributing/self-host).-## Contributing+----We love contributions! Please read our [contributing guide](CONTRIBUTING.md) before submitting a pull request. If you'd like to self-host, refer to the [self-hosting guide](SELF_HOST.md).+## Contributing-_It is the sole responsibility of the end users to respect websites' policies when scraping, searching and crawling with Firecrawl. Users are advised to adhere to the applicable privacy policies and terms of use of the websites prior to initiating any scraping activities. By default, Firecrawl respects the directives specified in the websites' robots.txt files when crawling. By utilizing Firecrawl, you expressly agree to comply with these conditions._+We love contributions! Please read our [Contributing Guide](https://github.com/firecrawl/firecrawl/blob/main/CONTRIBUTING.md) before submitting a pull request.-## Contributors+### Contributors<a href="https://github.com/firecrawl/firecrawl/graphs/contributors"><img alt="contributors" src="https://contrib.rocks/image?repo=firecrawl/firecrawl"/></a>-## License Disclaimer--This project is primarily licensed under the GNU Affero General Public License v3.0 (AGPL-3.0), as specified in the LICENSE file in the root directory of this repository. However, certain components of this project are licensed under the MIT License. Refer to the LICENSE files in these specific directories for details.+----Please note:+## License-- The AGPL-3.0 license applies to all parts of the project unless otherwise specified.-- The SDKs and some UI components are licensed under the MIT License. Refer to the LICENSE files in these specific directories for details.-- When using or contributing to this project, ensure you comply with the appropriate license terms for the specific component you are working with.+This project is primarily licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). The SDKs and some UI components are licensed under the MIT License. See the LICENSE files in specific directories for details.-For more details on the licensing of specific components, please refer to the LICENSE files in the respective directories or contact the project maintainers.+---+**It is the sole responsibility of end users to respect websites' policies when scraping.** Users are advised to adhere to applicable privacy policies and terms of use. By default, Firecrawl respects robots.txt directives. By using Firecrawl, you agree to comply with these conditions.<p align="right" style="font-size: 14px; color: #555; margin-top: 20px;">- <a href="#readme-top" style="text-decoration: none; color: #007bff; font-weight: bold;">- ↑ Back to Top ↑- </a>+ <a href="#readme-top" style="text-decoration: none; color: #007bff; font-weight: bold;">+ ↑ Back to Top ↑+ </a></p>