Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Allow follow and referrer in documentation links [skip ci]
eea53e0 Ondra Urban committed 6 years ago
Bump @types/tough-cookie from 2.3.7 to 4.0.0 (#637)
8759e86 dependabot-preview[bot] committed 6 years ago
Add option to set response encoding to CheerioCrawler (#632)
84c17ac Ondra Urban committed 6 years ago
Docs - improved formatting for the Return descriptions and other smal… (#629)
9348391 Jan Čurn committed 6 years ago