Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
Crawlee Scraper (crawleeScraper)
| Field | Value |
|---|---|
| Category | web_automation / tool (dual-purpose) |
| Backend handler | server/nodes/scraper/crawlee_scraper/__init__.py::CrawleeScraperNode — dispatch via BaseNode.execute() -> @Operation("scrape") |
| Tests | server/tests/nodes/test_web_automation.py |
| Skill (if any) | server/skills/web_agent/crawlee-scraper-skill/SKILL.md |
| Dual-purpose tool | yes |
Purpose
Python-based web scraping on top of the crawlee library. Supports two runtime
engines:
beautifulsoup- fast static HTML scraping viaBeautifulSoupCrawler.playwright- full headless browser for JS-rendered pages viaPlaywrightCrawler.adaptive- currently aliased tobeautifulsoupin this handler.
Two modes control how many URLs are visited: single (only the seed URL, hard
caps max_pages=1 / max_depth=0) and crawl (follow links matching
linkSelector + optional urlPattern up to maxPages / maxDepth).
Unlike httpRequest or browser, Crawlee manages concurrency, retry
back-off, request queues, and storage internally; the handler only feeds in
config and collects a page list from the route handler closure.
Inputs (handles)
| Handle | Connection type | Required | Purpose |
|---|---|---|---|
input-main |
main | no | Upstream trigger; not consumed directly |
Parameters
Params model is CrawleeScraperParams (field names are snake_case).
| Name | Type | Default | Required | displayOptions.show | Description |
|---|---|---|---|---|---|
tool_name |
string | web_scraper |
no | - | Name shown to the LLM when used as a tool (note: class tool_name is web_reader) |
tool_description |
string | (see default) | no | - | Description shown to the LLM |
url |
string | "" |
yes | - | Seed URL |
crawler_type |
options (Literal) | beautifulsoup |
no | - | beautifulsoup / playwright / adaptive |
mode |
options (Literal) | single |
no | - | single or crawl |
css_selector |
string | "" |
no | - | CSS selector used for text/HTML extraction. Empty -> whole page |
extract_links |
boolean | false |
no | - | Include links[] on each page |
output_format |
options (Literal) | text |
no | - | text / html / markdown. markdown falls back to text if html2text not installed |
max_pages |
int (1-1000) | 10 |
no | - | Clamped to 1 when mode=single |
max_concurrency |
int (1-50) | 5 |
no | - | Forwarded as max_concurrency; desired_concurrency = min(max_concurrency, 10) |
timeout |
int (1-3600) | 60 |
no | - | Per-request handler timeout AND outer asyncio.wait_for timeout |
link_selector |
string | "" |
no | mode=crawl |
Defaults to a[href] when empty |
url_pattern |
string | "" |
no | mode=crawl |
fnmatch-style pattern compiled with fnmatch.translate -> passed to enqueue_links(include=[regex]) |
max_depth |
int (0-50) | 2 |
no | mode=crawl |
Set to 0 when mode=single |
browser_type |
options (Literal) | chromium |
no | crawler_type=playwright |
chromium / firefox / webkit |
wait_for_selector |
string | "" |
no | crawler_type=playwright |
Awaits selector before extracting |
wait_timeout |
int (0-600000) | 30000 |
no | crawler_type=playwright |
Milliseconds for wait_for_selector |
take_screenshot |
boolean | false |
no | crawler_type=playwright |
Capture base64 PNG per page |
block_resources |
boolean | false |
no | crawler_type=playwright |
Declared in Params; currently not consumed by the run helpers |
use_proxy |
boolean | false |
no | - | Route through ProxyService (see Decision Logic) |
proxy_provider |
string | auto |
no | use_proxy=true |
Provider hint passed to ProxyService.get_proxy_url |
proxy_country |
string | "" |
no | use_proxy=true |
ISO country hint |
session_type |
options (Literal: rotating/sticky) | rotating |
no | use_proxy=true |
Proxy session type |
sticky_duration |
int (>=1) | 600 |
no | use_proxy=true, session_type=sticky |
Sticky session duration (s) |
Outputs (handles)
| Handle | Shape | Description |
|---|---|---|
output-main |
object | Standard envelope payload |
output-tool |
object | Same payload when wired to an AI agent's input-tools (usable_as_tool=True, class tool name web_reader) |
Output payload
{
pages: Array<{
url: string;
title: string;
content: string;
links?: string[]; // when extractLinks=true
screenshot?: string; // base64 PNG, playwright + take_screenshot=true
}>;
page_count: number;
crawler_type: 'beautifulsoup' | 'playwright' | 'adaptive';
mode: 'single' | 'crawl';
proxied: boolean;
}
Logic Flow
flowchart TD
A[BaseNode.execute -> CrawleeScraperNode.scrape] --> B{url stripped<br/>non-empty?}
B -- no --> Eurl[raise NodeUserError: URL is required]
B -- yes --> C[Read crawlerType, mode, cssSelector,<br/>extractLinks, outputFormat, maxPages,<br/>maxDepth, maxConcurrency, timeout]
C --> C1{mode == 'single'?}
C1 -- yes --> C2[Force maxPages=1, maxDepth=0]
C1 -- no --> C3[Use provided max_pages / max_depth]
C2 & C3 --> D[await _get_proxy_config parameters, url]
D --> D1{useProxy AND<br/>ProxyService.get_proxy_url<br/>returned URL?}
D1 -- yes --> D2[ProxyConfiguration proxy_urls=url]
D1 -- no / exception --> D3[proxy_config = None]
D2 & D3 --> E{crawlerType?}
E -- beautifulsoup / adaptive --> F1[_run_beautifulsoup]
E -- playwright --> F2[_run_playwright]
E -- other --> Ebad[raise NodeUserError:<br/>Unknown crawler type]
F1 --> G[BeautifulSoupCrawler<br/>max_requests_per_crawl=maxPages<br/>max_crawl_depth=maxDepth if crawl else 0<br/>request_handler_timeout=timeout<br/>ConcurrencySettings<br/>MemoryStorageClient<br/>proxy_configuration=proxy_config]
F2 --> H[PlaywrightCrawler<br/>browser_type, headless=True, proxy_configuration]
G & H --> I[register default_handler closure]
I --> I1[handler: extract title/content<br/>via _extract_text+_extract_links<br/>append to pages list<br/>if mode=crawl: ctx.enqueue_links<br/>with optional fnmatch-compiled include regex]
I1 --> J[asyncio.wait_for crawler.run URL, timeout=timeout]
J -- ImportError --> Eimp[raise NodeUserError install hint<br/>playwright vs crawlee, else re-raise]
J -- ok --> K[Return CrawleeScraperOutput<br/>pages, page_count, crawler_type, mode, proxied]
Decision Logic
- Empty URL:
url.strip() == ""->NodeUserError("URL is required"), no crawler is constructed. mode='single': hard clampsmax_pages=1andmax_depth=0, regardless of what the user set. Incrawlmode the requested values are forwarded.crawlerType='adaptive': routed to_run_beautifulsoup- there is no true adaptive detection in this handler today.- Proxy bridge:
_get_proxy_configonly runs whenuseProxy=True.ProxyService.get_proxy_urlreturningNoneor raising anyExceptionis logged at WARN and the crawl proceeds unproxied. - Link filter: when
urlPatternis set incrawlmode the pattern is passed throughfnmatch.translateand compiled to a regex forenqueue_links(include=[...]). - Content extraction:
_extract_textreturns text, html, or markdown. For markdown the handler falls back to plain text ifhtml2textis missing (ImportErrorswallowed). All output is capped at 100 000 chars (_MAX_CONTENT_LENGTH). - Error handling: only
ImportErroris caught insidescrape:- message contains
playwright->NodeUserErrorinstall hint forpip install 'crawlee[playwright]' && playwright install chromium. - message contains
crawlee->NodeUserErrorinstall hint forpip install 'crawlee[beautifulsoup]'. - any other
ImportErroris re-raised. - All non-
ImportErrorexceptions propagate toBaseNode.execute()'s generic handler (full-traceback envelope);NodeUserError(empty URL / unknown crawler type) yields a single-WARN structured envelope. The plugin does not swallow errors intostr(e).
- message contains
Side Effects
- Database writes: none.
- Broadcasts: none.
- External API calls: HTTP(S) requests to the target site(s), optionally
via a proxy returned by
ProxyService. - File I/O: none by default.
MemoryStorageClientkeeps the request queue in memory only. - Subprocess: PlaywrightCrawler launches a browser subprocess under the hood (managed by Playwright), but this handler does not spawn anything directly.
External Dependencies
- Credentials: none directly. Proxy auth, if any, is embedded in the
proxy URL returned by
ProxyService.get_proxy_url. - Services:
ProxyService(only whenuseProxy=True). - Python packages:
crawlee[beautifulsoup],crawlee[playwright](optional),bs4,html2text(optional, for markdown),playwright(for playwright mode). - Environment variables: none.
Edge cases & known limits
- Content truncation: per-page content is silently capped at 100 KB by
_MAX_CONTENT_LENGTH- longer pages are truncated without a marker. - Single mode ignores user caps:
maxPagesandmaxDepthare overridden to 1 and 0 respectively whenmode=single. adaptiveis a lie: currently aliased tobeautifulsoup; there is no runtime detection to escalate to Playwright for JS-heavy pages.- Markdown silently downgrades to text if
html2textis not installed. - Timeout is applied twice: once as
request_handler_timeoutinside the crawler and again asasyncio.wait_for(crawler.run(...), timeout=timeout)wrapping the whole run. Long multi-page crawls hit the outer limit first. - Proxy failures are swallowed: any exception from
ProxyServicecauses the crawl to proceed UN-proxied and returnsproxied=False, which may leak the user's real IP to the target. - No usage tracking: unlike
proxyRequest/httpRequest, this handler does not record cost or bytes metrics.
Related
- Skills using this as a tool:
crawlee-scraper-skill/SKILL.md - Companion nodes:
browser,apifyActor,httpRequest - Architecture docs: Proxy Service