Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Updated docs and comments: changed "key value store" to "key-value store"
bdec22c Jan Curn committed 6 years ago
needs to include types-apify, fix log as `any` (#613) [skip ci]
b762b2f Paulo Cesar committed 6 years ago
Bump apify-client from 0.5.26 to 0.6.0 (#597) [skip ci]
08fc0a2 dependabot-preview[bot] committed 6 years ago
Bump proxy-chain from 0.3.3 to 0.4.1 (#604) [skip ci]
65d5d1e dependabot-preview[bot] committed 6 years ago
Bump semver from 6.3.0 to 7.1.3 (#595) [skip ci]
e1ff00e dependabot-preview[bot] committed 6 years ago
Bump @types/fs-extra from 7.0.0 to 8.1.0 (#596) [skip ci]
e01bbd0 dependabot-preview[bot] committed 6 years ago
Bump rimraf from 2.7.1 to 3.0.2 (#592) [skip ci]
6e0f5ec dependabot-preview[bot] committed 6 years ago
Bump htmlparser2 from 3.10.1 to 4.1.0 (#601) [skip ci]
669a57e dependabot-preview[bot] committed 6 years ago
Bump eslint from 5.16.0 to 6.8.0 (#593) [skip ci]
134dbe8 dependabot-preview[bot] committed 6 years ago
Bump sinon from 7.5.0 to 9.0.0 (#591) [skip ci]
d5aa876 dependabot-preview[bot] committed 6 years ago
Bump @types/jest from 24.9.1 to 25.1.3 (#590) [skip ci]
2605ca0 dependabot-preview[bot] committed 6 years ago