Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(deps): update amannn/action-semantic-pull-request action to v5.0.2
bbf378c renovate[bot] committed 4 years ago
feat(playwright): disable animations when taking screenshots (#1601)
4e63034 Aleš Roubíček committed 4 years ago
chore(deps): update amannn/action-semantic-pull-request action to v5.0.1
fe56111 renovate[bot] committed 4 years ago
fix: build storage classes by using `this` instead of the class (#1596)
2b14eb7 Vlad Frangu committed 4 years ago
feat(core): add `forefront` option to `enqueueLinks` helper
f8755b6 Martin Adámek committed 4 years ago
test: fix flakyness of some test that used default storage (#1593)
38dc277 Martin Adámek committed 4 years ago