Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge branch 'master' into feature/session-pool-basic-crawler
59f8cbe Petr Patek committed 7 years ago
Merge pull request #466 from apifytech/feature/session-pool
aad245d Ondra Urban committed 7 years ago
Increase version of httpRequest. Added the timeout error. Improved throwing.
5fc4954 Petr Patek committed 7 years ago
Merge branch 'feature/session-pool' into feature/session-pool-basic-crawler
9a25e31 Petr Patek committed 7 years ago
Merge branch 'master' into feature/session-pool-basic-crawler
35a2c55 Petr Patek committed 7 years ago
fixed stealth to override the newPage on the context not on browser itself. Fixed KV test same as in #feature/session-pool.
331f6c3 Petr Patek committed 7 years ago