Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
feat: always show error origin if inside the userland (#1677)
bbe9045 Jindřich Bär committed 3 years ago
chore: move from set-output to writing to GITHUB_OUTPUT (#1687)
f39e34c Vlad Frangu committed 3 years ago
chore: remove priority option from issue templates (#1682)
b3d3c6a Vlad Frangu committed 4 years ago
fix: make router error message more helpful for undefined routes (#1678)
ab359d8 Ondra Urban committed 4 years ago
chore: use `@crawlee/core` + `lerna.json` package for the version inference
20d574b Martin Adámek committed 4 years ago
fix(MemoryStorage): correctly respect the desc option (#1666)
b5f37f6 Vlad Frangu committed 4 years ago
fix: injectJQuery in context does not survive navs (#1661)
493a7cf Jindřich Bär committed 4 years ago
fix: shallow clone browserPoolOptions before normalization (#1665)
22467ca Szymon Marczak committed 4 years ago
test: fix puppeteer-enqueue-links and cheerio-enqueue-links e2e tests
97ee28b Martin Adámek committed 4 years ago
fix: support headfull mode in playwright js project template
ea2e61b Martin Adámek committed 4 years ago
fix: support headfull mode in puppeteer js project template
e6aceb8 Martin Adámek committed 4 years ago