Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Better error messages in Apify.call(), Dataset.getData() throws on unsupported options
764d495 Jan Curn committed 7 years ago
RequestQueue.handledCount() function has been resurrected from deprecation
b92d5d0 Jan Curn committed 7 years ago
Bugfix in BasicCrawler - async calls in `isFinishedFunction` were not awaited
6e51536 Jan Curn committed 7 years ago
Merge pull request #390 from apifytech/fix/request-queue
5144e40 Marek Trunkát committed 7 years ago