Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Replaced Squid-based proxy chain with proxy-chain package, launchPuppeteer() uses better default User-Agent, better unit tests, removed parseUrl/redactUrl/redactParsedUrl functions
3a4a4c1 Jan Curn committed 9 years ago
Merge pull request #13 from Apifier/feature/launch-webdriver
002b0b9 Marek Trunkát committed 9 years ago
Updated publish.sh - changed max-age for docs to 3600, removed CloudFront invalidation
844bcb2 Jan Curn committed 9 years ago