Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge pull request #254 from apifytech/fix/puppeteer_pool_proxies
020197b Ondra Urban committed 7 years ago
Merge pull request #256 from apifytech/fix/dataset-offset-overflow
9ab36df Ondra Urban committed 7 years ago
Bugfix: PuppeteerPool was incorrectly overriding proxyUrls even if they were not defined.
f814b88 Jan Curn committed 7 years ago
Merge pull request #236 from apifytech/feature/userdata-in-enqueuelinks
d09ed81 Ondra Urban committed 7 years ago
Merge branch 'develop' into feature/userdata-in-enqueuelinks
c45fa11 Ondra Urban committed 7 years ago