Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Added stealth class and integrate id to the puppeteer_pool. Need a lot of work. Just an illustration of a structure.
96f9a52 Petr Patek committed 7 years ago
Merge branch 'develop' of github.com:Apifier/apify-runtime-js into feature/middlewares
8f90efc Marek Trunkat committed 7 years ago
Added param throwOnHttpError. Fixed error handling to do not throw on 400+ error codes by default. Added better contentType check.
b046730 Petr Patek committed 7 years ago
Added request better and requestLikeBrowser functions with tests.
3ac8c76 Petr Patek committed 7 years ago
Added specification for requestBetter() and requestLikeBrowser() functions
e24d47f Jan Curn committed 7 years ago