Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
PlaywrightLauncher: Drafted solution of default browsers. (#963)
e4a8d7c petrpatek committed 5 years ago
docs: add link explaining the pseudo-url in quick start (#961)
d1d8090 Martin Adámek committed 5 years ago
add options to get/add session to/from the pool by id (#953)
f60fde8 Andrey Bykov committed 5 years ago
docs: fix examples in the getting started section (#957)
18949ed Martin Adámek committed 5 years ago
Fix proxy configuration to support dynamic proxy status url (#940)
7eeefed Andrey Bykov committed 5 years ago
fix inconsistency between puppeteer and sessionPool cookies, add tests (#943)
f7f38ed Andrey Bykov committed 5 years ago