Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(release): update internal dependencies [skip ci]
bca9eab Apify Release Bot committed 3 years ago
docs: use new `crawler.exportData()` helper in the homepage example
3143586 Martin Adámek committed 3 years ago
chore(release): update internal dependencies [skip ci]
4370b95 Apify Release Bot committed 3 years ago
fix: declare missing dependency on got-scraping in the core package
cd2fd4d Martin Adámek committed 3 years ago
docs: responsive design fix (overflowing content) (#2181)
19f56b6 Jindřich Bär committed 3 years ago
fix: retry incorrect Content-Type when response has blocked status code (#2176)
b54fb8b Matěj Volf committed 3 years ago
chore(ci): get beta crawlee version from client payload too
28ec1ab Vlad Frangu committed 3 years ago