Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Update README.md
Eric Ciarladab8922ec915
diff --git a/README.md b/README.mdindex 099077643..21677807f 100644--- a/README.md+++ b/README.md@@ -41,9 +41,7 @@**Turn websites into LLM-ready data.**-[**Firecrawl**](https://firecrawl.dev/?ref=github) is an API that scrapes, crawls, and extracts structured data from any website, powering AI agents and apps with real-time context from the web.--Looking for our MCP? Check out the repo [here](https://github.com/firecrawl/firecrawl-mcp-server).+[**Firecrawl**](https://firecrawl.dev/?ref=github) is the API to search, scrape, and interact with the web at scale, powering AI agents and apps with real-time web data.*This repository is in development, and we're still integrating custom modules into the mono repo. It's not fully ready for self-hosted deployment yet, but you can run it locally.*@@ -70,299 +68,257 @@ _Pst. Hey, you, join our stargazers :)_## Quick Start-Sign up at [firecrawl.dev](https://firecrawl.dev) to get your API key and start extracting data in seconds. Try the [playground](https://firecrawl.dev/playground) to test it out.+Sign up at [firecrawl.dev](https://firecrawl.dev) to get your API key. Try the [playground](https://firecrawl.dev/playground) to test it out.-### Make Your First API Request-```bash-curl -X POST 'https://api.firecrawl.dev/v2/scrape' \- -H 'Authorization: Bearer fc-YOUR_API_KEY' \- -H 'Content-Type: application/json' \- -d '{"url": "https://example.com"}'-```+### Search-Response:-```json-{- "success": true,- "data": {- "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",- "metadata": {- "title": "Example Domain",- "sourceURL": "https://example.com"- }- }-}-```+Search the web and get full content from results.-### Install the Firecrawl Skill & CLI+```python+from firecrawl import Firecrawl-The Firecrawl Skill is an easy way for AI agents such as [Claude Code](https://claude.ai/code), [Antigravity](https://antigravity.google) and [OpenCode](https://opencode.ai) to use Firecrawl through the CLI.+app = Firecrawl(api_key="fc-YOUR_API_KEY")-Install and configure the skill for all detected AI coding agents:-```bash-npx -y firecrawl-cli@latest init --all --browser+search_result = app.search("firecrawl web scraping", limit=5)```-After installing, restart your agent for it to discover the new skill.+<details>+<summary><b>Node.js / cURL / CLI</b></summary>-You can also install the CLI globally:-```bash-npm install -g firecrawl-cli-```--Authenticate with your API key:-```bash-# Interactive login (opens browser)-firecrawl login --browser+**Node.js**+```javascript+import Firecrawl from '@mendable/firecrawl-js';-# Or login with API key directly-firecrawl login --api-key fc-YOUR_API_KEY+const app = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});-# Or set via environment variable-export FIRECRAWL_API_KEY=fc-YOUR_API_KEY+app.search("firecrawl web scraping", { limit: 5 })```-Try a quick scrape:+**cURL**```bash-firecrawl https://example.com --only-main-content+curl -X POST 'https://api.firecrawl.dev/v2/search' \+-H 'Authorization: Bearer fc-YOUR_API_KEY' \+-H 'Content-Type: application/json' \+-d '{+ "query": "firecrawl web scraping",+ "limit": 5+}'```-See the full [Skill + CLI documentation](https://docs.firecrawl.dev/sdks/cli) for all available commands including search, map, crawl, agent, and browser automation.-------## Feature Overview--| Feature | Description |-|---------|-------------|-| [**Scrape**](#scraping) | Convert any URL to markdown, HTML, screenshots, or structured JSON |-| [**Search**](#search) | Search the web and get full page content from results |-| [**Browse**](#browse) | Let agents safely interact with the web |-| [**Map**](#map) | Discover all URLs on a website instantly |-| [**Crawl**](#crawling) | Scrape all URLs of a website with a single request |-| [**Agent**](#agent) | Automated data gathering, just describe what you need |------## Scrape--Convert any URL to clean markdown, HTML, or structured data.+**CLI**```bash-curl -X POST 'https://api.firecrawl.dev/v2/scrape' \- -H 'Authorization: Bearer fc-YOUR_API_KEY' \- -H 'Content-Type: application/json' \- -d '{- "url": "https://docs.firecrawl.dev",- "formats": ["markdown", "html"]- }'+firecrawl search "firecrawl web scraping" --limit 5```+</details>-Response:+Output:```json-{- "success": true,- "data": {- "markdown": "# Firecrawl Docs\n\nTurn websites into LLM-ready data...",- "html": "<!DOCTYPE html><html>...",- "metadata": {- "title": "Quickstart | Firecrawl",- "description": "Firecrawl allows you to turn entire websites into LLM-ready markdown",- "sourceURL": "https://docs.firecrawl.dev",- "statusCode": 200- }+[+ {+ "url": "https://firecrawl.dev",+ "title": "Firecrawl",+ "markdown": "Turn websites into..."+ },+ {+ "url": "https://docs.firecrawl.dev",+ "title": "Firecrawl Docs",+ "markdown": "# Getting Started..."}-}+]```-### Extract Structured Data (JSON Mode)+### Scrape++Get LLM-ready data from any website — markdown, JSON, screenshots, and more.-Extract structured data using a schema:```pythonfrom firecrawl import Firecrawl-from pydantic import BaseModelapp = Firecrawl(api_key="fc-YOUR_API_KEY")-class CompanyInfo(BaseModel):- company_mission: str- is_open_source: bool- is_in_yc: bool+result = app.scrape('firecrawl.dev')+```-result = app.scrape(- 'https://firecrawl.dev',- formats=[{"type": "json", "schema": CompanyInfo.model_json_schema()}]-)+<details>+<summary><b>Node.js / cURL / CLI</b></summary>-print(result.json)+**Node.js**+```javascript+import Firecrawl from '@mendable/firecrawl-js';++const app = new Firecrawl({ apiKey: "fc-YOUR_API_KEY" });++app.scrape('firecrawl.dev')```-```json-{"company_mission": "Turn websites into LLM-ready data", "is_open_source": true, "is_in_yc": true}++**cURL**+```bash+curl -X POST 'https://api.firecrawl.dev/v2/scrape' \+-H 'Authorization: Bearer fc-YOUR_API_KEY' \+-H 'Content-Type: application/json' \+-d '{+ "url": "firecrawl.dev"+}'```-Or extract with just a prompt (no schema):-```python-result = app.scrape(- 'https://firecrawl.dev',- formats=[{"type": "json", "prompt": "Extract the company mission"}]-)+**CLI**+```bash+firecrawl scrape https://firecrawl.dev+firecrawl https://firecrawl.dev --only-main-content```+</details>-### Scrape Formats+Output:+```+# Firecrawl-Available formats: `markdown`, `html`, `rawHtml`, `screenshot`, `links`, `json`, `branding`+Firecrawl is a powerful web scraping tool that makes it easy+to extract clean data from any website.-**Get a screenshot**-```python-doc = app.scrape("https://firecrawl.dev", formats=["screenshot"])-print(doc.screenshot) # Base64 encoded image+## Features+- Scrape: Markdown from any page+- Search: Search + scrape the web+- Map: Discover all site URLs+- Agent: Extract with AI prompts```-**Extract brand identity (colors, fonts, typography)**+### Interact++Scrape a page, then interact with it using AI prompts or code.+```python-doc = app.scrape("https://firecrawl.dev", formats=["branding"])-print(doc.branding) # {"colors": {...}, "fonts": [...], "typography": {...}}+from firecrawl import Firecrawl++app = Firecrawl(api_key="fc-YOUR_API_KEY")++result = app.scrape("https://amazon.com")+scrape_id = result.metadata["scrapeId"]++app.interact(scrape_id, prompt="Search for 'mechanical keyboard'")+app.interact(scrape_id, prompt="Click the first result")```-### Actions (Interact Before Scraping)+<details>+<summary><b>Node.js / cURL / CLI</b></summary>-Click, type, scroll, and more before extracting:-```python-doc = app.scrape(- url="https://example.com/login",- formats=["markdown"],- actions=[- {"type": "write", "text": "[email protected]"},- {"type": "press", "key": "Tab"},- {"type": "write", "text": "password"},- {"type": "click", "selector": 'button[type="submit"]'},- {"type": "wait", "milliseconds": 2000},- {"type": "screenshot"}- ]-)+**Node.js**+```javascript+import Firecrawl from '@mendable/firecrawl-js';++const app = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});++const result = await app.scrape("https://amazon.com");++await app.interact(result.metadata.scrapeId, {+ prompt: "Search for 'mechanical keyboard'"+});+await app.interact(result.metadata.scrapeId, {+ prompt: "Click the first result"+});```----+**cURL**+```bash+# 1. Scrape the page+curl -X POST 'https://api.firecrawl.dev/v2/scrape' \+-H 'Authorization: Bearer fc-YOUR_API_KEY' \+-H 'Content-Type: application/json' \+-d '{"url": "https://amazon.com"}'-## Search+# 2. Interact with the page (use scrapeId from step 1)+curl -X POST 'https://api.firecrawl.dev/v2/scrape/SCRAPE_ID/interact' \+-H 'Authorization: Bearer fc-YOUR_API_KEY' \+-H 'Content-Type: application/json' \+-d '{"prompt": "Search for mechanical keyboard"}'+```-Search the web and optionally scrape the results.+**CLI**```bash-curl -X POST 'https://api.firecrawl.dev/v2/search' \- -H 'Authorization: Bearer fc-YOUR_API_KEY' \- -H 'Content-Type: application/json' \- -d '{- "query": "firecrawl web scraping",- "limit": 5- }'+firecrawl scrape https://amazon.com+firecrawl interact exec --prompt "Search for 'mechanical keyboard'"+firecrawl interact exec --prompt "Click the first result"```+</details>-Response:+Output:```json{"success": true,- "data": {- "web": [- {- "url": "https://www.firecrawl.dev/",- "title": "Firecrawl - The Web Data API for AI",- "description": "The web crawling, scraping, and search API for AI.",- "position": 1- }- ],- "images": [...],- "news": [...]- }+ "output": "Keyboard available at $100",+ "liveViewUrl": "https://liveview.firecrawl.dev/..."}```-### Search with Content Scraping+----Get the full content of search results:-```python-from firecrawl import Firecrawl+## Power Your Agent-firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")+Connect Firecrawl to any AI agent or MCP client in minutes.-results = firecrawl.search(- "firecrawl web scraping",- limit=3,- scrape_options={- "formats": ["markdown", "links"]- }-)-```-----+### Skill-## Browse+Give your agent easy access to real-time web data with one command.-Give your agents a secure browser environment. Let them run code safely to gather data and take action on the web.```bash-curl -X POST 'https://api.firecrawl.dev/v2/browser' \- -H 'Authorization: Bearer fc-YOUR_API_KEY' \- -H 'Content-Type: application/json'+npx -y firecrawl-cli@latest init --all --browser```-Response:+Restart your agent after installing. Works with [Claude Code](https://claude.ai/code), [Antigravity](https://antigravity.google), [OpenCode](https://opencode.ai), and more.++### MCP++Connect any MCP-compatible client to the web in seconds.+```json{- "success": true,- "id": "550e8400-e29b-41d4-a716-446655440000",- "cdpUrl": "wss://cdp-proxy.firecrawl.dev/cdp/550e8400-e29b-41d4-a716-446655440000",- "liveViewUrl": "https://liveview.firecrawl.dev/550e8400-e29b-41d4-a716-446655440000"+ "mcpServers": {+ "firecrawl-mcp": {+ "command": "npx",+ "args": ["-y", "firecrawl-mcp"],+ "env": {+ "FIRECRAWL_API_KEY": "fc-YOUR_API_KEY"+ }+ }+ }}```-### Execute Code in the Browser+### Agent Onboarding-Run Playwright code, Python, or bash commands remotely:-```javascript-import Firecrawl from '@mendable/firecrawl-js';+Are you an AI agent? Fetch this skill to sign up your user, get an API key, and start building with Firecrawl.-const firecrawl = new Firecrawl({ apiKey: "fc-YOUR_API_KEY" });+```bash+curl -s https://firecrawl.dev/agent-onboarding/SKILL.md+```-// 1. Launch a session-const session = await firecrawl.browser();+See the [Skill + CLI documentation](https://docs.firecrawl.dev/sdks/cli) for all available commands. For MCP, see [firecrawl-mcp-server](https://github.com/firecrawl/firecrawl-mcp-server).-// 2. Execute code-const result = await firecrawl.browserExecute(session.id, {- code: `- await page.goto("https://news.ycombinator.com");- const title = await page.title();- console.log(title);- `,- language: "node",-});-console.log(result.result); // "Hacker News"+----// 3. Close-await firecrawl.deleteBrowser(session.id);-```+## Feature Overview-### Persistent Sessions+**Core Endpoints**-Save and reuse browser state (cookies, localStorage) across sessions:-```javascript-const session = await firecrawl.browser({- ttl: 600,- profile: {- name: "my-profile",- saveChanges: true,- },-});-```+| Feature | Description |+|---------|-------------|+| [**Search**](#search) | Search the web and get full page content from results |+| [**Scrape**](#scrape) | Convert any URL to markdown, HTML, screenshots, or structured JSON |+| [**Interact**](#interact) | Scrape a page, then interact with it using AI prompts or code |-### agent-browser (Bash Mode)+**More**-Instead of writing Playwright code, agents can send simple bash commands via [agent-browser](https://github.com/vercel-labs/agent-browser):-```bash-firecrawl browser "open https://example.com"-firecrawl browser "snapshot"-firecrawl browser "click @e5"-```+| Feature | Description |+|---------|-------------|+| [**Agent**](#agent) | Automated data gathering, just describe what you need |+| [**Crawl**](#crawl) | Scrape all URLs of a website with a single request |+| [**Map**](#map) | Discover all URLs on a website instantly |+| [**Batch Scrape**](#batch-scraping) | Scrape thousands of URLs asynchronously |----## Agent+## More Endpoints++### Agent**The easiest way to get data from the web.** Describe what you need, and our AI agent searches, navigates, and extracts it. No URLs required.@@ -387,7 +343,7 @@ Response:}```-### Agent with Structured Output+#### Agent with Structured OutputUse a schema to get structured data:```python@@ -421,7 +377,7 @@ print(result.data)}```-### Agent with URLs (Optional)+#### Agent with URLs (Optional)Focus the agent on specific pages:```python@@ -431,7 +387,7 @@ result = app.agent()```-### Model Selection+#### Model SelectionChoose between two models based on your needs:@@ -454,18 +410,7 @@ result = app.agent(Learn more about Spark models in our [Agent documentation](https://docs.firecrawl.dev/features/agent).-### Using Firecrawl with AI agents--Install the Firecrawl skill to let AI agents like Claude Code, Codex, and OpenCode use Firecrawl automatically:-```bash-npx skills add firecrawl/cli-```--Restart your agent after installing. See the [Skill + CLI docs](https://docs.firecrawl.dev/sdks/cli) for full setup.-------## Crawling+### CrawlCrawl an entire website and get content from all pages.```bash@@ -490,7 +435,7 @@ Returns a job ID:}```-### Check Crawl Status+#### Check Crawl Status```bashcurl -X GET 'https://api.firecrawl.dev/v2/crawl/123-456-789' \-H 'Authorization: Bearer fc-YOUR_API_KEY'@@ -512,9 +457,7 @@ curl -X GET 'https://api.firecrawl.dev/v2/crawl/123-456-789' \**Note:** The [SDKs](#sdks) handle polling automatically for a better developer experience.------## Map+### MapDiscover all URLs on a website instantly.```bash@@ -536,7 +479,7 @@ Response:}```-### Map with Search+#### Map with SearchFind specific URLs within a site:```python@@ -548,9 +491,7 @@ result = app.map("https://firecrawl.dev", search="pricing")# Returns URLs ordered by relevance to "pricing"```------## Batch Scraping+### Batch ScrapeScrape multiple URLs at once:```python