Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge pull request #233 from apifytech/feature/default-viewport
a6c51eb Ondra Urban committed 8 years ago
Updated DEFAULT_USER_AGENT and LAUNCH_PUPPETEER_DEFAULT_VIEWPORT constants
1460be3 Jan Curn committed 8 years ago
Removed disable-background-timer-throttling Chrome arg, it's added by default
6a67922 Jan Curn committed 8 years ago