Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Add puppeteerPool.retire() function to manually retire browsers
fa13de7 Ondra Urban committed 8 years ago
Fix false positives on test (it should fail on CI for now)
50cf26f Ondra Urban committed 8 years ago
Add key validation to KeyValueStore to match Platform implementation
24ca7de Ondra Urban committed 8 years ago
Bugfix in BasicCrawler: native Promise doesn't have finally() function
d332ab9 Jan Curn committed 8 years ago
Adding maxRequestsPerCrawl parameter to BasicCrawler (#120)
d40fda2 Marek Trunkát committed 8 years ago