Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge branch 'master' into feature/request_as_browser_update
466ed64 Petr Patek committed 7 years ago
Apify.launchPuppeteer() now supports proxyUrl with socks4, socks5 and https URL schemes.
88c692f Jan Curn committed 7 years ago
Added desiredConcurrency option to AutoscaledPool constructor
8e6e002 Jan Curn committed 7 years ago
Changed user agents according to once actually displayed in browser.
0ab3b06 Petr Patek committed 7 years ago