Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge branch 'develop' into feature/hiding-tricks-puppeteer
f8a0323 Petr Patek committed 7 years ago
Commented out "slowMo" in code example, some users kept it in production
3b7613c Jan Curn committed 7 years ago
Merge pull request #353 from apifytech/feature/request-like-browser
73a23e7 Ondra Urban committed 7 years ago
Merge pull request #356 from apifytech/feature/middlewares
70ecf59 Ondra Urban committed 7 years ago
Improved error handling while parsing JSON and added missing return.
2b1507c Petr Patek committed 7 years ago
removed the local testing html file since it is not working in travis.
669ee2a Petr Patek committed 7 years ago
Added hiding tricks a improved few of them. Iframe chrome mocking needs to be fix and masked.
5e2a92e Petr Patek committed 7 years ago
Added stealth class and integrate it to the puppeteer. Added tests for the hiding tricks.
bf1b4ed Petr Patek committed 7 years ago