Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge pull request #149 from apifytech/feature/stop-crawler
0a29d4d Ondra Urban committed 8 years ago
Refactor PuppeteerCrawler, add .stop() to CheerioCrawler; update changelog
cffd6a8 Ondra Urban committed 8 years ago
Merge pull request #144 from apifytech/feature/cheerio-crawler
6fce772 Ondra Urban committed 8 years ago
Merge pull request #153 from apifytech/feature/keep-duplicates
f0dc592 Ondra Urban committed 8 years ago
Switch CheerioCrawler to async-await; fix tests; implement PR comments
afdc471 Ondra Urban committed 8 years ago