Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: return true when robots.isAllowed returns undefined (#2439)
6f541f8 Gigino Chianese committed 2 years ago
chore(deps): update amannn/action-semantic-pull-request action to v5.5.2
9e3e275 renovate[bot] committed 2 years ago
chore(deps): update amannn/action-semantic-pull-request action to v5.5.0
86eb837 renovate[bot] committed 2 years ago
fix: Fixed double extension for screenshots (#2419)
e8b39c4 Robert-Andrei Damian committed 2 years ago
fix: malformed sitemap url when sitemap index child contains querystring (#2430)
e4cd41c Austin committed 2 years ago