Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: investigate and temp fix for possible 0-concurrency bug in RQv2 (#2494)
4ebe820 Vlad Frangu committed 2 years ago
fix: adjust `URL_NO_COMMAS_REGEX` regexp to allow single character hostnames (#2492)
ec802e8 Martin Adámek committed 2 years ago
chore: ignore docker state when checking formatting (#2491)
fa72b4f Jan Buchar committed 2 years ago
test: Check if the proxy tier drops after an amount of successful requests (#2490)
a2f39d9 Jan Buchar committed 2 years ago
refactor: Remove redundant fields from `StatisticsPersistedState` (#2475)
72ebcd9 Jan Buchar committed 2 years ago
chore(release): update internal dependencies [skip ci]
805fc33 Apify Release Bot committed 2 years ago
feat: implement ErrorSnapshotter for error context capture (#2332)
e861dfd Hamza Alwan committed 2 years ago