Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Improved session error handling when setting invalid or empty cookie.
bb366b7 Petr Patek committed 6 years ago
Merge pull request #548 from apifytech/chore/remove-xregexp
8997885 Ondra Urban committed 6 years ago
Merge pull request #547 from apifytech/fix/remove-logging-from-require
bfcaea3 Ondra Urban committed 6 years ago
Merge pull request #545 from apifytech/fix/puppeteer-active-pages-counting
db460f5 Ondra Urban committed 6 years ago
Merge pull request #546 from apifytech/fix/cheerio-crawler-session
35d769a Ondra Urban committed 6 years ago
Moved the active pages decreasing into page.close event which is fired once the pages is closed or dies.
c670e3c Petr Patek committed 6 years ago