Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: allow `userData` option in `enqueueLinksByClickingElements` (#1749)
736f85d Matěj Volf committed 3 years ago
fix(memory-storage): dont fail when storage already purged (#1737)
8694027 JJetmar committed 3 years ago
fix: update playwright to 1.29.2 and make peer dep. less strict (#1735)
c654fcd Martin Adámek committed 3 years ago
fix: ensure CrawlingContext interface is inferred correctly in route handlers
aa84633 Martin Adámek committed 3 years ago
fix: clone `request.userData` when creating new request object (#1728)
222ef59 Matěj Volf committed 3 years ago
chore(ci): fix lerna issues with publishing releases (#1701)
11b9565 Vlad Frangu committed 3 years ago
fix: remove memory leaks from migration event handling (#1679)
49bba25 Szymon Marczak committed 3 years ago
chore: throw different error if user accidentally imports enqueueLinks (#1681)
3692b11 Vlad Frangu committed 3 years ago
ci: always trigger docker image build from stable release workflow
cac657c Martin Adámek committed 3 years ago