Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

NameLast commit date
..
crawler-plugins
.eslintrc.json
accept_user_input.mdx
accept_user_input.ts
add_data_to_dataset.mdx
add_data_to_dataset.ts
basic_crawler.mdx
basic_crawler.ts
cheerio_crawler.mdx
cheerio_crawler.ts
crawl_all_links_cheerio.ts
crawl_all_links_playwright.ts
crawl_all_links_puppeteer.ts
crawl_all_links.mdx
crawl_multiple_urls_cheerio.ts
crawl_multiple_urls_playwright.ts
crawl_multiple_urls_puppeteer.ts
crawl_multiple_urls.mdx
crawl_relative_links_all.ts
crawl_relative_links_same_domain.ts
crawl_relative_links_same_hostname.ts
crawl_relative_links.mdx
crawl_single_url.mdx
crawl_single_url.ts
crawl_sitemap_cheerio.ts
crawl_sitemap_playwright.ts
crawl_sitemap_puppeteer.ts
crawl_sitemap.mdx
crawl_some_links.mdx
crawl_some_links.ts
export_entire_dataset.mdx
export_entire_dataset.ts
file_download_stream.mdx
file_download_stream.ts
file_download.mdx
file_download.ts
forms.mdx
forms.ts
http_crawler.mdx
http_crawler.ts
jsdom_crawler_react.ts
jsdom_crawler.mdx
jsdom_crawler.ts
map_and_reduce.mdx
map.ts
playwright_crawler_firefox.mdx
playwright_crawler_firefox.ts
playwright_crawler.mdx
playwright_crawler.ts
puppeteer_capture_screenshot.mdx
puppeteer_crawler_crawler_utils_snapshot.ts
puppeteer_crawler_page_screenshot.ts
puppeteer_crawler_utils_snapshot.ts
puppeteer_crawler.mdx
puppeteer_crawler.ts
puppeteer_page_screenshot.ts
puppeteer_recursive_crawl.mdx
puppeteer_recursive_crawl.ts
reduce.ts
skip-navigation.mdx
skip-navigation.ts
tsconfig.json