Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge branch 'feature/session-pool' into feature/session-pool-implementation
163e9c1 Petr Patek committed 7 years ago
Merge branch 'feature/session-pool' of github.com:apifytech/apify-js into feature/session-pool
55a9da8 Petr Patek committed 7 years ago
Merge pull request #468 from apifytech/feature/cheerio_crawler_content_types
c996869 Ondra Urban committed 7 years ago
Moved streams functions to shared, added option json and contentType to handlePageFunction, mine -> mine
fb26d86 drobnikj committed 7 years ago
Cheerio Scraper: Removed application/xml from default mine types
b0c7a3c drobnikj committed 7 years ago
Cheerio Scraper: use mine types instead of content types, fixes regarding PR comments
19fc738 drobnikj committed 7 years ago
Added XML to default content type for CheerioCrawler, fixed tests
f812be2 drobnikj committed 7 years ago