Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
---
id: stagehand-crawler-guide
title: "StagehandCrawler guide"
sidebar_label: "StagehandCrawler"
description: AI-powered web crawling with natural language browser automation
---
import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
import StagehandBasicSource from '!!raw-loader!./stagehand-crawler-basic.ts';
import StagehandExtractSource from '!!raw-loader!./stagehand-crawler-extract.ts';
import StagehandCombinedSource from '!!raw-loader!./stagehand-crawler-combined.ts';
​<ApiLink to="stagehand-crawler/class/StagehandCrawler">`StagehandCrawler`</ApiLink> combines Crawlee's powerful crawling infrastructure with [Stagehand's](https://github.com/browserbase/stagehand) AI-powered browser automation. Instead of writing CSS selectors or XPath queries, you can interact with web pages using natural language instructions.
## What is Stagehand
[Stagehand](https://github.com/browserbase/stagehand) is an AI-powered browser automation library from Browserbase. It allows you to control a browser using natural language commands like "click the login button" or "extract the product price". Under the hood, Stagehand uses large language models (OpenAI, Anthropic, or Google) to understand the page structure and execute your instructions.
## How StagehandCrawler works
StagehandCrawler extends <ApiLink to="browser-crawler/class/BrowserCrawler">`BrowserCrawler`</ApiLink> and enhances each page with AI-powered methods. Here's the architecture:
1. **Stagehand launches the browser** - When a new browser is needed, Stagehand initializes and launches a Chromium browser
2. **Playwright connects via CDP** - Crawlee connects Playwright to the same browser using Chrome DevTools Protocol (CDP)
3. **Pages are enhanced with AI methods** - Each page gets `act()`, `extract()`, `observe()`, and `agent()` methods
4. **BrowserPool manages scaling** - Crawlee's BrowserPool handles browser lifecycle, retries, and concurrency
```
┌─────────────────────────────────────────────────────────┐
│ StagehandCrawler │
├─────────────────────────────────────────────────────────┤
│ BrowserPool (manages browser lifecycle & concurrency) │
├─────────────────────────────────────────────────────────┤
│ Stagehand Instance │
│ ├── Launches Chromium browser │
│ ├── Provides CDP endpoint │
│ └── Handles AI operations (act/extract/observe) │
├─────────────────────────────────────────────────────────┤
│ Playwright (connected via CDP) │
│ └── Standard page operations (goto, click, type, etc.) │
└─────────────────────────────────────────────────────────┘
```
## Key features
The enhanced page object provides four AI-powered methods:
### `page.act(instruction)` - Perform actions
Execute actions on the page using natural language. See [Stagehand act() documentation](https://docs.stagehand.dev/reference/act) for more details.
```ts
await page.act('Click the "Add to Cart" button');
await page.act('Fill in the email field with [email protected]');
await page.act('Scroll down to load more products');
```
### `page.extract(instruction, schema)` - Extract structured data
Extract data from the page using a Zod schema for type safety. See [Stagehand extract() documentation](https://docs.stagehand.dev/reference/extract) for more details.
```ts
import { z } from 'zod';
const productSchema = z.object({
title: z.string(),
price: z.number(),
description: z.string(),
});
const product = await page.extract('Get the product details', productSchema);
// product is typed as { title: string, price: number, description: string }
```
### `page.observe()` - Discover page actions
Analyze the page and get AI-suggested actions. This is useful for exploring unfamiliar pages or building adaptive scrapers. See [Stagehand observe() documentation](https://docs.stagehand.dev/reference/observe) for more details.
```ts
const actions = await page.observe();
// Returns available actions like:
// [
// { action: 'click', element: 'Load More button', selector: '.load-more' },
// { action: 'click', element: 'Next Page link', selector: 'a.pagination-next' },
// { action: 'fill', element: 'Search input', selector: '#search' },
// ]
// Use observe to find pagination dynamically
for (const action of actions) {
if (action.element?.toLowerCase().includes('next page')) {
await page.act(`Click the ${action.element}`);
break;
}
}
```
### `page.agent(config)` - Autonomous agents
Create an autonomous agent for complex multi-step workflows. Unlike `act()` which executes a single action, `agent()` can plan and execute multiple steps autonomously to achieve a goal. See [Stagehand agent() documentation](https://docs.stagehand.dev/reference/agent) for more details.
```ts
const agent = page.agent({ model: 'gpt-4.1-mini' });
const result = await agent.execute('Find the cheapest laptop and add it to cart');
```
**When to use `act()` vs `agent()`:**
- Use `act()` for single, discrete actions ("click this button", "fill this form")
- Use `agent()` for goals requiring multiple steps with decision-making ("find and purchase the cheapest item")
Note that `agent()` makes multiple LLM calls and can be slower and more expensive than sequential `act()` calls where you control the flow.
## Requirements
StagehandCrawler requires an API key for the AI model provider. The recommended way is to use the `apiKey` option:
```ts
const crawler = new StagehandCrawler({
stagehandOptions: {
model: 'openai/gpt-4.1-mini',
apiKey: 'your-api-key', // Your OpenAI API key
},
});
```
Alternatively, you can use environment variables (used as fallback when `apiKey` is not provided):
- **OpenAI**: `OPENAI_API_KEY`
- **Anthropic**: `ANTHROPIC_API_KEY`
- **Google**: `GOOGLE_API_KEY`
## Limitations
Some Crawlee features work differently or are unavailable with StagehandCrawler:
### Chromium only
Stagehand uses Chrome DevTools Protocol (CDP), so only Chromium browsers are supported. The `launcher` option is ignored - you cannot use Firefox or WebKit.
### Reduced fingerprinting control
Since Stagehand controls the browser launch process, Crawlee's advanced fingerprinting features are limited:
- **Browser fingerprints** - Basic fingerprinting (viewport, user-agent) is applied, but low-level browser properties cannot be modified
- **`launchOptions`** - Only a subset of Playwright launch options are passed through to Stagehand (`headless`, `args`, `executablePath`, `proxy`, `viewport`)
- **Browser context options** - Custom context configurations are not fully supported since Stagehand manages the browser context
Stagehand provides its own anti-detection measures, but you have less granular control compared to PlaywrightCrawler.
## When to use StagehandCrawler
**Use StagehandCrawler when:**
- Pages have complex, dynamic structures that are hard to scrape with selectors
- You need to interact with pages in ways that are difficult to express programmatically
- You want to quickly prototype scrapers without writing detailed selectors
- The target website frequently changes its structure
**Consider alternatives when:**
- You need maximum performance (use CheerioCrawler or PlaywrightCrawler)
- You need to minimize costs (LLM API calls add up)
- You need fine-grained browser control (use PlaywrightCrawler)
- You need Firefox or WebKit support (use PlaywrightCrawler)
## Basic example
Here's a simple example that extracts code examples from the Crawlee website:
<CodeBlock language="ts" title="src/main.ts">{StagehandBasicSource}</CodeBlock>
## Data extraction example
Here's an example showing structured data extraction with Zod schemas:
<CodeBlock language="ts" title="src/main.ts">{StagehandExtractSource}</CodeBlock>
## Configuration options
### Stagehand options
Configure the AI behavior through `stagehandOptions`:
```ts
const crawler = new StagehandCrawler({
stagehandOptions: {
// Environment: 'LOCAL' or 'BROWSERBASE'
env: 'LOCAL',
// AI model to use (e.g., 'openai/gpt-4.1-mini', 'anthropic/claude-sonnet-4-20250514')
model: 'openai/gpt-4.1-mini',
// API key for the LLM provider (can be overridden by environment variables)
apiKey: process.env.OPENAI_API_KEY,
// Logging verbosity: 0 (minimal), 1 (standard), 2 (debug)
verbose: 1,
// Enable automatic error recovery
selfHeal: true,
// Timeout for DOM to stabilize (ms)
domSettleTimeout: 30000,
},
});
```
### Environment variables
Stagehand options can alternatively be set via environment variables. Programmatic options always take precedence over environment variables:
| Environment variable | Option | Notes |
|---------------------|--------|-------|
| `OPENAI_API_KEY` | `apiKey` | Fallback for OpenAI models |
| `ANTHROPIC_API_KEY` | `apiKey` | Fallback for Anthropic models |
| `GOOGLE_API_KEY` | `apiKey` | Fallback for Google models |
| `STAGEHAND_ENV` | `env` | |
| `STAGEHAND_MODEL` | `model` | |
| `STAGEHAND_VERBOSE` | `verbose` | |
| `STAGEHAND_API_KEY` | `apiKey` | Browserbase API key |
| `STAGEHAND_PROJECT_ID` | `projectId` | Browserbase project ID |
## Using with Browserbase
For cloud browser infrastructure, you can use [Browserbase](https://browserbase.com/):
```ts
const crawler = new StagehandCrawler({
stagehandOptions: {
env: 'BROWSERBASE',
apiKey: process.env.BROWSERBASE_API_KEY,
projectId: process.env.BROWSERBASE_PROJECT_ID,
model: 'openai/gpt-4.1-mini',
},
});
```
## Combining AI and standard methods
You can mix AI-powered methods with standard Playwright methods:
<CodeBlock language="ts" title="src/main.ts">{StagehandCombinedSource}</CodeBlock>
## Further reading
- [Stagehand documentation](https://docs.stagehand.dev/)
- [Browserbase documentation](https://docs.browserbase.com/)
- <ApiLink to="stagehand-crawler/class/StagehandCrawler">`StagehandCrawler` API reference</ApiLink>