Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: make SessionPool queue up getSession calls to prevent overruns (#2239)
0f5665c Vlad Frangu committed 2 years ago
docs: improve warning message on empty RequestList (#2232)
f6b7e28 Jindřich Bär committed 2 years ago
docs: hide the Run on Apify button on mobile devices (#2224)
b6ece5a Jindřich Bär committed 2 years ago
fix(MemoryStorage): lock request JSON file when reading to support multiple process crawling (#2215)
eb84ce9 Vlad Frangu committed 2 years ago