Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

NameLast commit date
..
custom-http-client
impit-http-client
parallel-scraping
running-in-web-server
avoid_blocking_camoufox.ts
avoid_blocking_playwright_fingerprints_off.ts
avoid_blocking_playwright.ts
avoid_blocking_puppeteer_fingerprints_off.ts
avoid_blocking_puppeteer.ts
avoid_blocking.mdx
cheerio_crawler.mdx
configuration.mdx
docker_browser_js.txt
docker_browser_ts.txt
docker_images.mdx
docker_node_js.txt
docker_node_ts.txt
got_scraping.mdx
javascript-rendering-playwright-no-wait.ts
javascript-rendering-playwright.ts
javascript-rendering-puppeteer-no-wait.ts
javascript-rendering-puppeteer.ts
javascript-rendering.mdx
jsdom_crawler.mdx
motivation.mdx
proxy_management_inspection_cheerio.ts
proxy_management_inspection_http.ts
proxy_management_inspection_jsdom.ts
proxy_management_inspection_playwright.ts
proxy_management_inspection_puppeteer.ts
proxy_management_integration_cheerio.ts
proxy_management_integration_http.ts
proxy_management_integration_jsdom.ts
proxy_management_integration_playwright.ts
proxy_management_integration_puppeteer.ts
proxy_management_session_cheerio.ts
proxy_management_session_http.ts
proxy_management_session_jsdom.ts
proxy_management_session_playwright.ts
proxy_management_session_puppeteer.ts
proxy_management_session_standalone.ts
proxy_management.mdx
request_storage_queue_basic.ts
request_storage_queue_crawler_explicit.ts
request_storage_queue_crawler.ts
request_storage_queue_list.ts
request_storage_queue_only.ts
request_storage.mdx
result_storage.mdx
scaling_crawlers_autoscaledPoolOptions.ts
scaling_crawlers_maxRequestsPerMinute.ts
scaling_crawlers_minMaxConcurrency.ts
scaling_crawlers.mdx
session_management_basic.ts
session_management_cheerio.ts
session_management_http.ts
session_management_jsdom.ts
session_management_playwright.ts
session_management_puppeteer.ts
session_management_standalone.ts
session_management.mdx
typescript_project.mdx