Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

NameLast commit date
..
adaptive-crawling
crawling-context
custom-http-client
custom-logger
http-clients
impit-http-client
parallel-scraping
request-lifecycle
running-in-web-server
avoid-blocking-camoufox.ts
avoid-blocking-playwright-fingerprints-off.ts
avoid-blocking-playwright.ts
avoid-blocking-puppeteer-fingerprints-off.ts
avoid-blocking-puppeteer.ts
avoid-blocking.mdx
cheerio-crawler.mdx
configuration.mdx
cookie-modals.mdx
custom-statistics-adaptive.ts
custom-statistics-basic.ts
custom-statistics-validated.ts
custom-statistics.mdx
docker_browser_js.txt
docker_browser_ts.txt
docker_node_js.txt
docker_node_ts.txt
docker-images.mdx
http-clients.mdx
javascript-rendering-playwright-no-wait.ts
javascript-rendering-playwright.ts
javascript-rendering-puppeteer-no-wait.ts
javascript-rendering-puppeteer.ts
javascript-rendering.mdx
motivation.mdx
multiple-crawlers-cross-enqueue.ts
multiple-crawlers.mdx
proxy-management-inspection-cheerio.ts
proxy-management-inspection-http.ts
proxy-management-inspection-playwright.ts
proxy-management-inspection-puppeteer.ts
proxy-management-integration-cheerio.ts
proxy-management-integration-http.ts
proxy-management-integration-playwright.ts
proxy-management-integration-puppeteer.ts
proxy-management-session-cheerio.ts
proxy-management-session-http.ts
proxy-management-session-playwright.ts
proxy-management-session-puppeteer.ts
proxy-management-session-standalone.ts
proxy-management.mdx
public-api.mdx
remote-browser-config.ts
remote-browser-provider.ts
remote-browser-puppeteer.ts
remote-browser.mdx
request-loaders-rl-basic.ts
request-loaders-rl-tandem-explicit.ts
request-loaders-rl-tandem-helper.ts
request-loaders-sitemap-basic.ts
request-loaders-sitemap-tandem-explicit.ts
request-loaders-sitemap-tandem-helper.ts
request-loaders.mdx
request-storage-queue-basic.ts
request-storage-queue-crawler-explicit.ts
request-storage-queue-crawler.ts
request-storage-queue-only.ts
request-storage.mdx
result-storage.mdx
scaling-crawlers-concurrency-system.ts
scaling-crawlers-custom-concurrency-system.ts
scaling-crawlers-custom-load-signal.ts
scaling-crawlers-load-signals-off.ts
scaling-crawlers-max-requests-per-minute.ts
scaling-crawlers-min-max-concurrency.ts
scaling-crawlers-wrap-load-signal.ts
scaling-crawlers.mdx
send-request.mdx
session-management-basic.ts
session-management-cheerio.ts
session-management-http.ts
session-management-playwright.ts
session-management-puppeteer.ts
session-management-standalone.ts
session-management.mdx
stagehand-crawler-basic.ts
stagehand-crawler-combined.ts
stagehand-crawler-extract.ts
stagehand-crawler.mdx
trace-and-monitor-basic.ts
trace-and-monitor-crawlers.mdx
trace-and-monitor-custom.ts
trace-and-monitor-register-hook.ts
trace-and-monitor-setup.ts
trace-and-monitor-wrap-with-span.ts
typescript-project.mdx