Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore: update lockfile and do it after stable release automatically
78d19d9 Martin Adámek committed 3 years ago
fix: add `QueueOperationInfo` export to the core package
5ec6c24 Martin Adámek committed 3 years ago
feat(MemoryStorage): read from fs if persistStorage is enabled, ram only otherwise (#1761)
e903980 Vlad Frangu committed 3 years ago
fix: Correctly compute `pendingRequestCount` in request queue (#1765)
946535f Jiří Moravčík committed 3 years ago
feat: add `forefront` option to all `enqueueLinks` variants (#1760)
a01459d Matěj Volf committed 3 years ago
Revert "fix: make `SessionOptions.sessionPool` required"
6526e1b Martin Adámek committed 3 years ago
feat(enqueueLinks): add SameOrigin strategy and relax protocol matching for the other strategies (#1748)
4ba982a Vlad Frangu committed 3 years ago