Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: wait when `body` is not available in `infiniteScroll()` from Puppeteer utils (#1277)
57987cd Martin Adámek committed 4 years ago
fix: do not use `tryCancel()` from inside sync callback (#1265)
6f951f1 Martin Adámek committed 4 years ago
fix: ignore requests that are no longer in progress (#1258)
ed9c80b Martin Adámek committed 4 years ago
chore: fix proxy rotation test (use mocked proxy servers)
f59d95b Martin Adámek committed 4 years ago
test: add test to ensure proxy IP rotation works properly
050aaa3 Martin Adámek committed 4 years ago
chore(deps): update amannn/action-semantic-pull-request action to v3.5.0
7535228 Renovate Bot committed 4 years ago
fix: automatically convert google spreadsheet URLs to CSV exports (#1255)
1c5ed76 Martin Adámek committed 4 years ago
fix: add timeouts with retries to `runTaskFunction()` (#1250)
a271862 Martin Adámek committed 4 years ago