Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(deps): update dependency puppeteer to v24.11.1 (#3033)
e310dba renovate[bot] committed 1 year ago
chore: remove TS build cache from git to fix build (#3035)
50858b5 Martin Adámek committed 1 year ago
chore(release): update internal dependencies [skip ci]
371eab1 Apify Release Bot committed 1 year ago
feat: Report links skipped because of various filter conditions (#3026)
5a867bc Jan Buchar committed 1 year ago
chore: bump fingerprinting tools and `impit` to latest versions (#3028)
6cbdb8a Jindřich Bär committed 1 year ago
fix: Fix link filtering in enqueueLinks in AdaptivePlaywrightCrawler (#3021)
8a3b6f8 Jan Buchar committed 1 year ago
feat: Accept (Async)Iterables in `addRequests` methods (#3013)
a4ab748 Jan Buchar committed 1 year ago
fix: Do not log 'malformed sitemap content' on network errors in `Sitemap.tryCommonNames` (#3015)
64a090f Jan Buchar committed 1 year ago
chore(release): update internal dependencies [skip ci]
039b51e Apify Release Bot committed 1 year ago
feat(dataset): add collectAllKeys option for full CSV export (#2945) (#3007)
3b629da Bhanu-Chauhan-uoa committed 1 year ago
fix: Persist rendering type detection results in `AdaptivePlaywrightCrawler` (#2987)
76431ba Jan Buchar committed 1 year ago
fix: Do not enqueue more links than what the crawler is capable of processing (#2990)
ea094c8 Jan Buchar committed 1 year ago
test: Add end-to-end test of AdaptivePlaywrightCrawler (#3003)
cf72d70 Jan Buchar committed 1 year ago
feat: support `KVS.listKeys()` `prefix` and `collection` parameters (#3001)
5c4726d Jindřich Bär committed 1 year ago
chore(release): update internal dependencies [skip ci]
f357979 Apify Release Bot committed 1 year ago